Re: Why do we need these mount options?

[Zygo Blaxell <ce3g8jdj@umail.furryterror.org>]

On Sat, Jan 16, 2021 at 08:21:16PM +0300, Andrei Borzenkov wrote:
> 16.01.2021 18:19, Adam Borowski пишет:
> > On Sat, Jan 16, 2021 at 10:39:51AM +0300, Andrei Borzenkov wrote:
> >> 15.01.2021 06:54, Zygo Blaxell пишет:
> >>> On the other hand, I'm in favor of deprecating the whole discard option
> >>> and going with fstrim instead.  discard in its current form tends to
> >>> increase write wear rather than decrease it, especially on metadata-heavy
> >>> workloads.  discard is roughly equivalent to running fstrim thousands
> >>> of times a day, which is clearly bad for many (most?  all?) SSDs.
> >>
> >> My (probably naive) understanding so far was that trim on SSD marks
> >> areas as "unused" which means SSD need to copy less residual data from
> >> erase block when reusing it. Assuming TRIM unit is (significantly)
> >> smaller than erase block.
> >>
> >> I would appreciate if you elaborate how trim results in more write on SSD?
> > 
> > The areas are not only marked as unused, but also zeroed.  To keep the
> > zeroing semantic, every discard must be persisted, thus requiring a write
> > to the SSD's metadata (not btrfs metadata) area.
> > 
> 
> There is no requirement that TRIM did it. If device sets RZAT SUPPORTED
> bit, it should return zeroes for trimmed range, but there is no need to
> physically zero anything - simply return zeroes for areas marked as
> unallocated. Discard must be persisted in allocation table, but then
> every write must be persisted in allocation table anyway.

That is exactly the problem--the persistence is a write that counts
against total drive wear.  That is why TRIM variants that leave
the contents of the discarded LBAs undefined are better than those
which define the contents as zero.

The effect seems to be the equivalent of a small write, i.e. a 16K
write might be the same cost as any length of contiguous discard.
So it's OK to discard block-group-sized regions, but not OK to issue
one discard for every metadata free page hole.  Different drives have
different ratios between these costs, so parity might occur at 4K or
256K depending on the drive.

AIUI there is a minimum discard length filter implemented in btrfs
already, so maybe it just needs tuning?

> Moreover, to actually zero on TRIM either trim request must be issued
> for the full erase block or device must perform garbage collection.
> 
> Do you have any links that show that discards increase write load on
> physical media? I am really curious.

I have no links, it's a directly observed result.

It's fairly straightforward to replicate:  Set up a machine to do git
checkouts of each Linux kernel tag in random order, in a loop (maybe
multiple instances of this if needed to get the SSD device IO saturated).
While that happens, watch the percentage used endurance indicator reported
on the drives (smartctl -x).  Wait for the indicator to increment
twice, and measure the time between the first and second increment.
Use a low-cost consumer or OEM SSD so you get results in less than a
few hundred hours.  Then mount -o discard=async and wait for two more
increments.  Assuming the workload produces constant amounts of IO over
time, and the percentage used endurance indicator variable from SMART is
not a complete lie, the time between increments should roughly indicate
the wear rates of the different workloads.

In the field, we discovered this on CI builder workloads (lots of
tiny files created, destroyed, and created again in rapid succession).
They get almost double the SSD wear rate with discard on vs. discard off.
We have monitoring on the p-u-e-i variable, and use it to project the date
when 100% endurance will be reached.  If that date lands within the date
range when we want to be using the SSD, we get an alert.  When discard
is accidentally enabled on a CI server due to a configuration failure,
we get an alert about a week later, as it shortens our drives' projected
lifespan from more than 6 years to less than 4.

Other workloads are less sensitive to this.  If the workload has fewer
metadata updates, bigger files, and sequential writes, then discard
doesn't have a negative effect--though to be fair, it doesn't seem to
have a positive effect either, at least not by this measurement method.