Using BTRFS on SSD now ?

Using BTRFS on SSD now ?

[Swâmi Petaramesh]

Hi,

I just received a new laptop with a Micron 256GB SSD, and I plan to install 
Fedora 20 onto it.

I'm considering either BTRFS or ext4 (over LUKS-encrypted LVM) for this 
machine, but I'm afraid BTRFS might generate too much writes and shorten the 
SSD lifespan... Or am I mistaken ?

Is there any pro/cons currently, on a 3.14 kernel, about using BTRFS along 
with an SSD ?

Is there specific advice about leaf size, use of compression, snapshots, 
(auto-)defrag etc, that might be relevant especially for SSDs ?

TIA.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E

Re: Using BTRFS on SSD now ?

[Russell Coker]

On Thu, 5 Jun 2014 15:30:26 Swâmi Petaramesh wrote:
> I just received a new laptop with a Micron 256GB SSD, and I plan to install
> Fedora 20 onto it.
> 
> I'm considering either BTRFS or ext4 (over LUKS-encrypted LVM) for this
> machine, but I'm afraid BTRFS might generate too much writes and shorten the
> SSD lifespan... Or am I mistaken ?

http://etbe.coker.com.au/2014/04/27/swap-breaking-ssd/

I recently did some calculations based on the usage of workstations I run on 
SSD storage.  While the main focus of my post was swap and SSD I also did the 
calculations for the supposed life of SSDs and BTRFS filesystem use (all 
workstations in question run BTRFS).  My conclusion is that if the most 
conservative (pessimistic) claims about SSD life are correct then there's 
still nothing to worry about.

Also I think that laptops tend to have less use than workstations, so unless 
your laptop is your main desktop system it should get less disk use than a 
typical workstation.

http://etbe.coker.com.au/2014/05/12/btrfs-vs-lvm/

I don't believe that LVM offers any benefit if you use BTRFS.

> Is there any pro/cons currently, on a 3.14 kernel, about using BTRFS along
> with an SSD ?

Apart from BTRFS being new and still a bit buggy, not at all.

> Is there specific advice about leaf size, use of compression, snapshots,
> (auto-)defrag etc, that might be relevant especially for SSDs ?

SSDs being a lot faster than spinning media remove many of these issues.

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: Using BTRFS on SSD now ?

[Marc MERLIN]

On Thu, Jun 05, 2014 at 03:30:26PM +0200, Swâmi Petaramesh wrote:
> Hi,
> 
> I just received a new laptop with a Micron 256GB SSD, and I plan to install 
> Fedora 20 onto it.
> 
> I'm considering either BTRFS or ext4 (over LUKS-encrypted LVM) for this 
> machine, but I'm afraid BTRFS might generate too much writes and shorten the 
> SSD lifespan... Or am I mistaken ?

Yeah,
1) a lot of us use btrfs on SSDs
2) SSD write lifetime is more than you think

However SSDs, especially at least earlier models of the one you got, still
randomly die, and take your data with them. It's not because of number of
writes, but bad firmware and/or a single chip that dies and isn't redundant on
some drives.
 
> Is there any pro/cons currently, on a 3.14 kernel, about using BTRFS along 
> with an SSD ?

It's not foolproof, but works well enough for many. Have backups, but
because of your SSD, you should definitely have backups anyway.

> Is there specific advice about leaf size, use of compression, snapshots, 
> (auto-)defrag etc, that might be relevant especially for SSDs ?

compression: lzo, not zlib (zlib will slow down your SSD)
snapshots:
http://marc.merlins.org/perso/btrfs/2014-03.html#Btrfs-Tips_-How-To-Setup-Netapp-Style-Snapshots
auto-defrag: not really (read the archives)
other options: leave default
add -o discard to mount options

Do not use LVM, it's totally unnecessary and will slow you down. Dmcrypt is
ok, however add discard to cryptsetup options too:
root2		 /dev/sda2		/pwd		luks,discard

legolas:~# grep discard /proc/cmdline 
root=/dev/mapper/cryptroot ro rootflags=subvol=root cryptopts=source=/dev/sda4,keyscript=/sbin/cryptgetpw,discard

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Using BTRFS on SSD now ?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

On Thu, 5 Jun 2014 07:56:32 -0700
Marc MERLIN <marc@merlins.org> wrote:

> However SSDs, especially at least earlier models of the one you got, still
> randomly die, and take your data with them.

Yeah, a PSA of sorts: take whatever SSDs you have and use, and go to the
manufacturer's website (right *NOW* :) to check if they have a newer firmware,
or to ensure that you have the latest version. There are such horrible bugs
being found and fixed in those, that upgrading the firmware is absolutely *not*
the case of "if it's not broken, don't fix it". Also your SSD may gain niceties
such as a working TRIM support (that was sometimes disabled/broken in earlier
releases).

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Using BTRFS on SSD now ?

[Swâmi Petaramesh]

Thanks for the reply Russell :-)

Le vendredi 6 juin 2014 00:42:12 Russell Coker a écrit :
> 
> Also I think that laptops tend to have less use than workstations, so unless
> your laptop is your main desktop system it should get less disk use than a
> typical workstation.

My laptop (+ docking station) is going to be my daily main machine...

> I don't believe that LVM offers any benefit if you use BTRFS.

BTRFS cannot handle swap. A full disk encryption needs the swap to be 
encrypted as well. The only solution for getting both the swap and filesystems 
encrypted using the same key is to put both of them on an LVM itself luks-
encrypted using dm-crypt.

All my machines have been made this way for *years*, so I know it works damn 
well ;-)

...And it also allows for hibernateing the system to (encrypted) swap space...

That's a pretty fine setup :-)

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E

Re: Using BTRFS on SSD now ?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 220 bytes --]

On Thu, 2014-06-05 at 07:56 -0700, Marc MERLIN wrote: 
> Dmcrypt is
> ok, however add discard to cryptsetup options too
Be aware, that discard used with dm-crypt may have security
implications.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5165 bytes --]

Re: Using BTRFS on SSD now ?

[Martin Steigerwald]

Am Donnerstag, 5. Juni 2014, 15:30:26 schrieb Swâmi Petaramesh:
> Hi,
> 
> I just received a new laptop with a Micron 256GB SSD, and I plan to install 
> Fedora 20 onto it.
> 
> I'm considering either BTRFS or ext4 (over LUKS-encrypted LVM) for this 
> machine, but I'm afraid BTRFS might generate too much writes and shorten
> the  SSD lifespan... Or am I mistaken ?

Relax.

I don´t know why you come to the conclusion BTRFS would create more writes
than other filesystems, but here are facts:

BTRFS on Intel SSD 320, since about month or so as RAID 1 on Intel SSD 320
and Crucial m5 mSATA. BTRFS on Intel SSD 320 since I think a bit more than
three years for now.

SMART Attributes Data Structure revision number: 5
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  3 Spin_Up_Time            0x0020   100   100   000    Old_age   Offline      -       0
  4 Start_Stop_Count        0x0030   100   100   000    Old_age   Offline      -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       10289
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       2797
170 Reserve_Block_Count     0x0033   100   100   010    Pre-fail  Always       -       0
171 Program_Fail_Count      0x0032   100   100   000    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   100   100   000    Old_age   Always       -       169
183 Runtime_Bad_Block       0x0030   100   100   000    Old_age   Offline      -       1
184 End-to-End_Error        0x0032   100   100   090    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
192 Unsafe_Shutdown_Count   0x0032   100   100   000    Old_age   Always       -       241
199 UDMA_CRC_Error_Count    0x0030   100   100   000    Old_age   Offline      -       0
225 Host_Writes_32MiB       0x0032   100   100   000    Old_age   Always       -       449085
226 Workld_Media_Wear_Indic 0x0032   100   100   000    Old_age   Always       -       2204365
227 Workld_Host_Reads_Perc  0x0032   100   100   000    Old_age   Always       -       49
228 Workload_Minutes        0x0032   100   100   000    Old_age   Always       -       13212562
232 Available_Reservd_Space 0x0033   100   100   010    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   099   099   000    Old_age   Always       -       0
241 Host_Writes_32MiB       0x0032   100   100   000    Old_age   Always       -       449085
242 Host_Reads_32MiB        0x0032   100   100   000    Old_age   Always       -       1144183


Media Wearout Indicator was 100 initially, now is 99. Thus the SSDs considers
itself to be basically new. Use search machine to find PDF of Intel about
that value.

And 449085 32 MiB Host Writes with KDE session, desktop search Nepomuk, now
Baloo, Akonadi setups with insanely sized mailbox and what not. Thats:

irb(main):027:0> 449085 * 32 / 1024.0 / 1024.0
=> 13.704986572265625

13,7 TiB. This Intel SSD is specced for a useful life of 5 years with 20 GB
host writes *each day*. Thats about 7 TiB per year, even if you use 1000 as
base here. 3 years * 7 TiB = 21 TiB.


So look at your SSD endurance specification and then just relax :)

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Using BTRFS on SSD now ?

[Marc MERLIN]

On Thu, Jun 05, 2014 at 05:14:29PM +0200, Swâmi Petaramesh wrote:
> > your laptop is your main desktop system it should get less disk use than a
> > typical workstation.
> 
> My laptop (+ docking station) is going to be my daily main machine...
 
My laptop is my workstation, it's running almost 24/7 on SSD with btrfs.
The main thing is having a good SSD (right now Samsung 840 EVO 1TB for me).

See:
http://marc.merlins.org/perso/linux/post_2012-08-15_The-tale-of-SSDs_-Crucial-C300-early-Death_-Samsung-830-extreme-random-IO-slowness_-and-settling-with-OCZ-Vertex-4.html

> > I don't believe that LVM offers any benefit if you use BTRFS.
> 
> BTRFS cannot handle swap. A full disk encryption needs the swap to be 
> encrypted as well. The only solution for getting both the swap and filesystems 
> encrypted using the same key is to put both of them on an LVM itself luks-
> encrypted using dm-crypt.

Not at all.
Again, you really want to avoid the LVM layer. It will only make things
slower.
This is my drive with encrypted swap
Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1050623      524288   83  Linux
/dev/sda2         1050624    42993663    20971520   83  Linux
/dev/sda3       105908224   139462655    16777216   82  Linux swap / Solaris
/dev/sda4       139462656  1953525167   907031256   83  Linux

See
https://btrfs.wiki.kernel.org/index.php/SysadminGuide#btrfs_on_top_of_dmcrypt
and solutions like
https://github.com/gebi/keyctl_keyscript

> All my machines have been made this way for *years*, so I know it works damn 
> well ;-)

I've used LVM for years too, it makes things slower and less reliable.
 
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Using BTRFS on SSD now ?

[Swâmi Petaramesh]

Le jeudi 5 juin 2014 17:59:48 Christoph Anton Mitterer a écrit :
> Be aware, that discard used with dm-crypt may have security
> implications.

Thanks, I knew :-)

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E

Le sage n’est pas celui qui discrimine, c’est celui qui réunit
les lambeaux de lumière, d’où qu’ils proviennent.
-- Umberto Eco

Re: Using BTRFS on SSD now ?

[Chris Murphy]

On Jun 5, 2014, at 7:30 AM, Swâmi Petaramesh <swami@petaramesh.org> wrote:

> Hi,
> 
> I just received a new laptop with a Micron 256GB SSD, and I plan to install 
> Fedora 20 onto it.
> 
> I'm considering either BTRFS or ext4 (over LUKS-encrypted LVM) for this 
> machine, but I'm afraid BTRFS might generate too much writes and shorten the 
> SSD lifespan... Or am I mistaken ?

Is it certain Btrfs single profile writes more metadata than other file systems? Since the journal is the file system, I'd expect the same or less writes with Btrfs than other file systems which repetitively write a separate journal. In any case it's a really minor consideration. SSD wear is to the point you'd have to practically be abusing the drive (enterprise demand workloads on a consumer SSD).

Btrfs does work over dm layers including on LVM and dmcrypt and both. There have been some problems recently reported on the list that involve LVM and dmcrypt but I have no idea if those are contributing or merely complicating factors. Unless you have a really specific requirement offered by LVM,  or you're just looking to test with intent to break to help make the combination better for everyone, I'd advise against using it. Btrfs on dmcrypt should definitely work (I've done it for a few years and haven't had issues). There's some advantage of LVM thinp LVs, I'm not sure if Btrfs is going to offer its own thin provisioning, if not then Btrfs on LVM thinp needs to work to but I've only done a little bit of testing with this. It works, but with obscenely sized virtual LVs with totally insufficient backing, I've had some problems, ergo if you want to go this route you basically are signing up for detailed and concise problem reporting.

> Is there any pro/cons currently, on a 3.14 kernel, about using BTRFS along 
> with an SSD ?

If anything, more even wear on older devices that don't do their own wear leveling.

> Is there specific advice about leaf size, use of compression, snapshots, 
> (auto-)defrag etc, that might be relevant especially for SSDs ?

Defaults are fine. Longer term discard and autodefrag mount options will have some clearer benefits.

I pretty much gave up on discard with a Samsung 830 SSD. In both Linux and OS X I'd get these short hangs, maybe just a couple seconds or so. They were more annoying on OS X than on Linux.

Fragmentation is probably not a big performance problem on SSDs. At least on my hardware I haven't noticed a difference with even highly fragmented VM images that I haven't set xattr +C (nodatacow) on. Neverthless I've done some testing of both +C on /var/log/journal since systemd journals quickly get very fragmented on Btrfs as do VM images. I'm pretty sure all the bugs I reported about that were long ago fixed, I haven't had any recurrences. So now I'm not using +C but rather autodefrag for the past 8 months. A few months ago I had some systemd journal corruptions but I can't point the finger at either Btrfs or autodefrag because the corruptions coincided with unclean shutdowns. And it even could be an issue with systemd-journald. If I manage to reproduce the corruption, then I'll have to see about also reproducing it on either ext4 or xfs.

Compression I no longer use. I'd probably not use forced compression on an SSD because it'll probably just slow things down. 

Snapshots haven't been a problem, I'm even doing my own atomic yum/dnf updates by creating rw snapshots, chrooting them, and then running the update in the chroot. So the snapshots are what get the system update, if it fails I can just blow away the snapshots, and if it succeeds I haven't changed anything on the running OS mounted system, I can point grub to the new subvolume to boot from and reboot to the updated system whenever I want.

For what it's worth, Fedora 20's installer doesn't let you have a /boot on Btrfs. It must be on ext4. So if you want /boot on Btrfs you get to do this post-install. The current bug there being worked on (patches upstream) is kernel updates depend on grubby adding the proper entry in grub.cfg which fails with /boot is on Btrfs. As a work around to the grubby message you get when doing kernel updates, you can just run grub2-mkconfig -o <grub.cfg location> to create a new correct grub.cfg. I'm expecting this to all "just work" in Fedora 21.


Chris Murphy

Re: Using BTRFS on SSD now ?

[Chris Murphy]

On Jun 5, 2014, at 8:56 AM, Marc MERLIN <marc@merlins.org> wrote:
> 
> compression: lzo, not zlib (zlib will slow down your SSD)

I've had mixed results, where benchmarking suggests lzo is faster, but then zlib actually feels faster. For sure forced zlib compression can be dreadful, but I kinda expect that and therefore don't use it. These days I'm not using any compression. I think a lot of the benchmarks suggesting one way or the other are just the wrong benchmarks to look at for this sort of thing, as they aren't testing the actual workloads that are typical.



> snapshots:
> http://marc.merlins.org/perso/btrfs/2014-03.html#Btrfs-Tips_-How-To-Setup-Netapp-Style-Snapshots
> auto-defrag: not really (read the archives)

Definitely there are some issues, it's probably why its not yet default. So I'd either not use it, or use it with the point of testing it to make it better.


> other options: leave default
> add -o discard to mount options

I disagree. The older the SSD almost certainly the worse its TRIM behavior will be, resulting in hangs possibly long hangs. There's a whole bunch of stuff written on lkml about non-queued TRIM ugliness. I think most users are better off issuing fstrim once a week until queued TRIM devices are in the wild and well tested.

The one case were file system discard is useful is not so much to inform the physical device but the next lower layer logical block device like md raid or LVM thinp so that the right unused blocks are freed up.

> 
> Do not use LVM, it's totally unnecessary and will slow you down. Dmcrypt is
> ok, however add discard to cryptsetup options too:
> root2		 /dev/sda2		/pwd		luks,discard

So again in this case I'd say don't use discard. Or at the very least, test it first. My SSD is reportedly not so bad in this regard, but I find even 2-3 second hang of *all* read/write events when deleting a bunch of files to be abhorrent. Some people report much, much longer delays as the SSD then also does immediate and aggressive GC.

I'd say, what slight additional wear occurs from not using discard, makes the SSD die sooner in order to justify getting a new SSD that maybe (?) doesn't have this problem anymore.

Chris Murphy

Re: Using BTRFS on SSD now ?

[Chris Murphy]

On Jun 5, 2014, at 9:14 AM, Swâmi Petaramesh <swami@petaramesh.org> wrote:

>> I don't believe that LVM offers any benefit if you use BTRFS.
> 
> BTRFS cannot handle swap.

Yes but Fedora's installer doesn't support swapfiles anyway. It only creates swap partitions (encrypted or not encrypted). And it does allow you to selectively only encrypt swap if you want.

> A full disk encryption needs the swap to be 
> encrypted as well.

The Fedora installer will let you use regular partitions and encrypt them. LVM isn't necessary to achieve this.

> The only solution for getting both the swap and filesystems 
> encrypted using the same key is to put both of them on an LVM itself luks-
> encrypted using dm-crypt.

Why do they need to have the same key? The Fedora installer will let you do a Btrfs installation with swap as a separate partition, encrypt both partitions using one passphrase. Yes there are two LUKS device keys setup, but one passphrase unlocks both at the same time during boot.

> All my machines have been made this way for *years*, so I know it works damn 
> well ;-)

The Fedora installer won't even let you do this. Btrfs can't be put on LVM via anaconda.

Best practices for SSDs has been to encrypt each LV separately because dmcrypt was one thread per logical device. For an SSD this can make a big difference because it's possible to pin that single encryption thread and turn it into a bottle neck. There are multithreaded dmcrypt patches but I have no idea if it's in mainline kernels and if it gets used by default.


> 
> …And it also allows for hibernateing the system to (encrypted) swap space...

*shrug* that's hardware specific, whether it works or not. Right now hibernation in general, let alone encrypted, sucks on Linux right now. It and suspend are totally busted on all of my hardware right now, and forums are littered with people not able to get it to work, or having to become virtual experts getting it to work.


Chris Murphy

Re: Using BTRFS on SSD now ?

[Marc MERLIN]

On Thu, Jun 05, 2014 at 12:13:37PM -0600, Chris Murphy wrote:
> I'd say, what slight additional wear occurs from not using discard, makes the SSD die sooner in order to justify getting a new SSD that maybe (?) doesn't have this problem anymore.

Your points are well noted, although on the flipside I've had an SSD
perform horribly because without TRIM, it had to use its own garbage
collection which was horrible. Also, if you rely on garbage collection, you
need to make sure to make an empty partition with 5 or 10% of your space and
never use it so that yoru SSD can easily use that for garbage collection
without impacting performance too much.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Using BTRFS on SSD now ?

[Duncan]

Chris Murphy posted on Thu, 05 Jun 2014 12:13:37 -0600 as excerpted:

>> auto-defrag: not really (read the archives)
> 
> Definitely there are some issues, it's probably why its not yet default.
> So I'd either not use it, or use it with the point of testing it to make
> it better.

Usage-pattern is the big one with autodefrag.  I'm personally an 
autodefrag user and booster, but my use-case has few if any of the big 
red-flag file-access-pattern cases, and several where there's reasonable 
benefit.

Autodefrag's target: Small (up to a few hundred MiB, safely under a gig 
and preferably under half a gig) internal-rewrite-pattern files typically 
found on desktop/laptop purpose systems.  Typical example: Firefox and 
thunderbird sqlite database files.  Also: Multi-file simultaneous write, 
since this write pattern can trigger fragmentation as well, that 
autodefrag's background rewrite strategy should counter/fix.

Autodefrag sees writes that update these files and on copy-on-write (COW) 
filesystems such as btrfs trigger fragmentation, and queues the affected 
files for background sequential rewrite and defrag.  On this type of file 
this strategy works very well, neutralizing the fragmentation that would 
otherwise plague these files on btrfs, without huge ware or performance 
issues.

Autodefrag's bane: Large (gig-plus) internal-rewrite-pattern files, 
traditionally less often found on desktop/laptop systems altho they are 
becoming more common.  Typical examples: VM-images and database files 
over a gig in size.

It can be noted that these types of files are a challenge for any COW-
based filesystem, btrfs being no exception.  Like me, you may have few to 
none of these, but if you do, I strongly recommend reading up on the NOCOW 
file attribute and related strategies, as it's very likely the single 
most common performance issue seen with btrfs, autodefrag or not and ssd 
or not.  See the "Very slow filesystem" thread posted by Igor M (igork20 
at gmail) yesterday.

Autodefrag neutral: Sequentially-written files in general and append-only 
files such as the typical logfile, particularly when only one file is 
being written at once.

Autodefrag strategy hint:  If you choose to use this mount option, it's 
best to use it from THE FIRST TIME file data is written to the 
filesystem.  Unfortunately, as of a year or so ago at least, various 
distro installers did NOT use this flag by default when installing on 
btrfs, resulting in a lot of out-of-the-box installation fragmentation on 
btrfs.  Turning it on only after installation thus resulted in a 
performance drag for several days, until autodefrag caught up with the 
backlog.  So if installing to btrfs, verify that this flag is turned on 
by the installer for the install itself, if necessary using the custom 
option at this step to turn it on at initial mount, before all those 
files are copied over by the installer.

As an installation-time-autodefrag alternative, run btrfs filesystem 
defrag -r (recursive) on the filesystem after installation, before you 
start using it much and definitely before the first btrfs snapshot (or 
delete that snapshot first, do the defrag, then create a new snapshot).  
That will take care of all the fragmentation at once, and by doing it 
without snapshots taken, you avoid data duplication due to defrag being 
temporarily snapshot-aware disabled, until they fix the scaling issues 
that snapshot-awareness was triggering.

Unless you have autodefrag-bane files to worry about (and if you do, I 
strongly recommend that you do your research on that and special-case 
them with NOCOW and etc), even on ssd, autodefrag should be a small bonus 
due to saved IOPS, tho not to the extent it would be on spinning rust due 
to ssd's zero-seek-time.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Using BTRFS on SSD now ?

[Duncan]

Marc MERLIN posted on Thu, 05 Jun 2014 12:05:26 -0700 as excerpted:

> On Thu, Jun 05, 2014 at 12:13:37PM -0600, Chris Murphy wrote:
>> I'd say, what slight additional wear occurs from not using discard,
>> makes the SSD die sooner in order to justify getting a new SSD that
>> maybe (?) doesn't have this problem anymore.
> 
> Your points are well noted, although on the flipside I've had an SSD
> perform horribly because without TRIM, it had to use its own garbage
> collection which was horrible. Also, if you rely on garbage collection,
> you need to make sure to make an empty partition with 5 or 10% of your
> space and never use it so that yoru SSD can easily use that for garbage
> collection without impacting performance too much.

This subthread nicely addresses the subject next on my list to tackle in 
relation to this thread. =:^)

Here's the deal with trim/discard (two different names for the same 
feature):

Early SSDs didn't have it or had proprietary implementations, quickly 
making the community aware of the need.  Any decent and reasonably 
current SSD should offer the now standardized feature, but unfortunately, 
the initial standard made the trim instruction non-queued, so on a lot of 
hardware, issuing a trim instruction acts as an IO barrier, disrupting 
continued traffic to/from the device until the existing queue is emptied 
and the trim instruction completed, after which the queue can refill... 
until the next such operation.

As a result, while most current hardware supports trim/discard, on a lot 
of it, trim/discard in normal operation can reduce performance 
substantially. =:^(

For this sort of hardware, trim works best when used in flag-day fashion, 
at mkfs.btrfs for instance (and it /does/ issue a whole-range trim before 
setting up the filesystem), and periodically using tools such as fstrim.  
It does /not/ work so well when used routinely as part of the normal IO 
flow, when deleting a file or COWing a block, for instance, because these 
disrupt the normal hardware operations queue.

OTOH, certain high-performance hardware goes beyond the current standard 
and does a queued trim, without forcing a flush of the queue in the 
process.  But this hardware tends to be rather rare and expensive, and 
general claims to support trim can't be taken to indicate support for 
this sort of trim at all, so it tends to be a rather poor basis for a 
general recommendation or a routine-trim-by-default choice.

Then there's the encrypted device implications, which tend to favor not 
enabling discard/trim by default as well, due to the information leakage 
potential.

Thus we have the current situation, discard (aka trim) as a SEPARATE 
mount option from ssd, with ssd enabled by default where non-rotational 
storage is detected, but discard always requiring explicit invocation, as 
it simply isn't appropriate for a default option, at this point.

FWIW, the latest SATA standard (SATA 3.something) is said to explicitly 
support or perhaps even require queued trim support.  However, actual 
hardware with this support remains rare, at this point.

(FWIW, in new enough versions of smartctl, smartctl -i will have a "SATA 
Version is:" line, but even my newer Corsair Neutrons report only SATA 
2.5, so obviously they don't support queued trim by the standard, tho 
it's still possible they implement it beyond the standard, I simply don't 
know.)

Meanwhile, most authorities recommend leaving some portion of an SSD, 
typically 20-30% unformatted, thus giving the firmware plenty of room to 
manage erase-blocks as necessary, and that normally lessens the 
requirement to keep a nicely trimmed filesystem as well.

Here, it happened that when I was looking for SSDs, the 128 GB or so I 
figured I needed were either out of stock or the 256 GB or so versions 
were only a few dollars more expensive, so I ended up buying 256 GB 
versions where I had intended to buy 128 GB versions.  So in addition to 
actually putting more on the SSDs than I had originally intended (I still 
use lower cost spinning rust for my media partition, tho), I'm actually 
running only a bit over 50% partitioned, as well, with the rest of the 
SSDs being entirely unused, reserved for firmware erase-block management 
or for future use.  So I haven't actually worried much about whether I 
could efficiently turn on trim/discard or not, I just let the over-
provisioning handle it, along with doing a fresh mkfs.btrfs and restore 
from backup every few kernel cycles (and thus getting the mkfs.btrfs 
whole-filesystem trim in the process), in ordered to take advantage of 
the latest btrfs filesystem features. =:^)  I do occasional fstrims as 
well, but haven't worried about doing that in any scheduled fashion 
either, simply because what with fresh mkfs.btrfs every few kernel cycles 
and with nearly 100% overprovisioning, I've not needed to.  Tho I 
probably will once btrfs development slows down and there aren't new 
filesystem format features to be taken advantage of every few kernel 
cycles.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Using BTRFS on SSD now ?

[Martin K. Petersen]

>>>>> "Duncan" == Duncan  <1i5t5.duncan@cox.net> writes:

Duncan> OTOH, certain high-performance hardware goes beyond the current
Duncan> standard and does a queued trim, without forcing a flush of the
Duncan> queue in the process.  But this hardware tends to be rather rare
Duncan> and expensive,

Queued trim has started to appear in consumer SSDs. However, since we're
the only OS that supports it the feature has come off to a bumpy start.
We tried to enable it on a drive model that passed testing here but we
had to revert to unqueued when bug reports started rolling in this week.

Duncan> (FWIW, in new enough versions of smartctl, smartctl -i will have
Duncan> a "SATA Version is:" line, but even my newer Corsair Neutrons
Duncan> report only SATA 2.5, so obviously they don't support queued
Duncan> trim by the standard, tho it's still possible they implement it
Duncan> beyond the standard, I simply don't know.)

The reported SATA version is in no way indicative of whether a drive
supports queued trim. The capability flag was put a highly unusual place
in the protocol. I posted a patch that makes this information available
in sysfs a while back. However, the patch is currently being reworked to
support a debug override...

Until then, the following command will give you the answer:

# smartctl -l gplog,0x13 /dev/sda | grep 0000:
0000000: 01 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 |................|
         ^^          ^^
These two 01 fields indicate that the drive supports queued trim.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: Using BTRFS on SSD now ?

[Chris Murphy]

On Jun 5, 2014, at 5:58 PM, "Martin K. Petersen" <martin.petersen@oracle.com> wrote:
> 
> Queued trim has started to appear in consumer SSDs. However, since we're
> the only OS that supports it the feature has come off to a bumpy start.
> We tried to enable it on a drive model that passed testing here but we
> had to revert to unqueued when bug reports started rolling in this week.

There's a RHBZ I ran across ~6 months ago with a purportedly SATA rev 3.1 SSD, (m?)SATA, and queued trim caused some big enough of a problem that the model is black listed for queued trim. I don't remember if it was massive fs corruption or if it bricked the drive or what. So yeah they're rare enough that it's like, OK we're sorta starting over from scratch.

Chris Murphy

Re: Using BTRFS on SSD now ?

[Duncan]

Martin K. Petersen posted on Thu, 05 Jun 2014 19:58:26 -0400 as excerpted:

>>>>>> "Duncan" == Duncan  <1i5t5.duncan@cox.net> writes:
> 
> Duncan> OTOH, certain high-performance hardware goes beyond the current
> Duncan> standard and does a queued trim, without forcing a flush of the
> Duncan> queue in the process.  But this hardware tends to be rather rare
> Duncan> and expensive,
> 
> Queued trim has started to appear in consumer SSDs. However, since we're
> the only OS that supports it the feature has come off to a bumpy start.
> We tried to enable it on a drive model that passed testing here but we
> had to revert to unqueued when bug reports started rolling in this week.

[more elided]

Thanks. Fresher and better information than I had. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Using BTRFS on SSD now ?

[Pavel Volkov]

On Thursday, June 5, 2014 7:11:44 PM MSK, Roman Mamedov wrote:
> Yeah, a PSA of sorts: take whatever SSDs you have and use, and go to the
> manufacturer's website (right *NOW* :) to check if they have a 
> newer firmware,
> or to ensure that you have the latest version. There are such horrible 
bugs
> being found and fixed in those, that upgrading the firmware is 
> absolutely *not*
> the case of "if it's not broken, don't fix it". Also your SSD 
> may gain niceties
> such as a working TRIM support (that was sometimes 
> disabled/broken in earlier
> releases).
>

Sometimes the latest firmware has isuues which were not present earlier 
(like in my case with Crucial M4) so we have to be careful and listen to 
other users,
Maybe M4 is ok now, I'll have to check again,

Re: Using BTRFS on SSD now ?

[Pavel Volkov]

On Friday, June 6, 2014 3:58:26 AM MSK, Martin K. Petersen wrote:
> Until then, the following command will give you the answer:
>
> # smartctl -l gplog,0x13 /dev/sda | grep 0000:
> 0000000: 01 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 
|................|
>          ^^          ^^
> These two 01 fields indicate that the drive supports queued trim.
>

Is mine a dinosaur drive?

# smartctl -l gplog,0x13 /dev/sda 
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.14.5-gentoo-r1melf] (local 
build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

General Purpose Log 0x13 does not exist (override with '-T permissive' 
option)

Re: Using BTRFS on SSD now ?

[Martin K. Petersen]

>>>>> "Pavel" == Pavel Volkov <sailor@lists.xtsubasa.org> writes:

Pavel> Is mine a dinosaur drive?
[...]
Pavel> General Purpose Log 0x13 does not exist (override with '-T
Pavel> permissive' option)

The only flags defined in general purpose log page 0x13 are the queued
trim ones. So there is no compelling reason for a drive vendor to
implement that page unless the drive actually supports queued trim. And
consequently it's perfectly normal for that page to be absent.

-- 
Martin K. Petersen	Oracle Linux Engineering