[LSF/MM TOPIC] A little TLC for Discard

[LSF/MM TOPIC] A little TLC for Discard

[Dennis Zhou]

Hi everyone,

Ric proposed a similar topic already [1], but I'd like to just add a
little more to that discussion with a broader proposal for discussing
discard.

Proposal:
Discard is a topic that seems to need a little bit of TLC from the
kernel side. The two main approaches with discard are either to enable
it as a mount option or to periodically run fstrim. At Facebook, we've
seen performance wins with both enabling discard and disabling discard
in favor of fstrim. I would like to find a middle ground where we don't
have to have application developers performance tune for the hardware.

XFS has had async discard support, but it has been problematic for our
fleet. We were seeing bursts of large discard requests caused by async
discard in XFS. This resulted in degraded drive performance increasing
latency for dependent services.

An alternative to discard is a filesystem level choice of reusing
blocks. Rather than immediately issuing a discard, we can choose to
reuse a block which will cause the underlying FTL to invalidate a block
that we would have discarded. This saves the work done by the initial
discard. There will have to be a balance here though with saving discard
with hopes of reusing it and eventually issuing the discards.

Another issue is each drive has its own proprietary FTL which introduces
a wide variability of performance with discard. Some drives don't need
discard to perform well and others need discard enabled. There is also a
class of drives where discard is so not performant that it locks up the
drive. I would like to discuss how we can quantify this and help tune
discard within the kernel.

In summary, I'd like to explore discard more and understand how we can
build a more balanced discard experience where end users don't have to
be super cognizant of the implications on their applications.

[1] https://lore.kernel.org/linux-fsdevel/92ab41f7-35bc-0f56-056f-ed88526b8ea4@gmail.com/

Thanks,
Dennis

Re: [LSF/MM TOPIC] A little TLC for Discard

[Song Liu]

> On Feb 20, 2019, at 2:06 PM, Dennis Zhou <dennis@kernel.org> wrote:
> 
> Hi everyone,
> 
> Ric proposed a similar topic already [1], but I'd like to just add a
> little more to that discussion with a broader proposal for discussing
> discard.
> 
> Proposal:
> Discard is a topic that seems to need a little bit of TLC from the
> kernel side. The two main approaches with discard are either to enable
> it as a mount option or to periodically run fstrim. At Facebook, we've
> seen performance wins with both enabling discard and disabling discard
> in favor of fstrim. I would like to find a middle ground where we don't
> have to have application developers performance tune for the hardware.
> 
> XFS has had async discard support, but it has been problematic for our
> fleet. We were seeing bursts of large discard requests caused by async
> discard in XFS. This resulted in degraded drive performance increasing
> latency for dependent services.
> 
> An alternative to discard is a filesystem level choice of reusing
> blocks. Rather than immediately issuing a discard, we can choose to
> reuse a block which will cause the underlying FTL to invalidate a block
> that we would have discarded. This saves the work done by the initial
> discard. There will have to be a balance here though with saving discard
> with hopes of reusing it and eventually issuing the discards.
> 
> Another issue is each drive has its own proprietary FTL which introduces
> a wide variability of performance with discard. Some drives don't need
> discard to perform well and others need discard enabled. There is also a
> class of drives where discard is so not performant that it locks up the
> drive. I would like to discuss how we can quantify this and help tune
> discard within the kernel.
> 
> In summary, I'd like to explore discard more and understand how we can
> build a more balanced discard experience where end users don't have to
> be super cognizant of the implications on their applications.
> 
> [1] https://lore.kernel.org/linux-fsdevel/92ab41f7-35bc-0f56-056f-ed88526b8ea4@gmail.com/
> 
> Thanks,
> Dennis

Maybe we should use BPF to handle complexity and variations?

Thanks,
Song 

Re: [LSF/MM TOPIC] A little TLC for Discard

[Bryan Gurney]

On Wed, Feb 20, 2019 at 5:06 PM Dennis Zhou <dennis@kernel.org> wrote:
>
> Hi everyone,
>
> Ric proposed a similar topic already [1], but I'd like to just add a
> little more to that discussion with a broader proposal for discussing
> discard.
>
> Proposal:
> Discard is a topic that seems to need a little bit of TLC from the
> kernel side. The two main approaches with discard are either to enable
> it as a mount option or to periodically run fstrim. At Facebook, we've
> seen performance wins with both enabling discard and disabling discard
> in favor of fstrim. I would like to find a middle ground where we don't
> have to have application developers performance tune for the hardware.
>
> XFS has had async discard support, but it has been problematic for our
> fleet. We were seeing bursts of large discard requests caused by async
> discard in XFS. This resulted in degraded drive performance increasing
> latency for dependent services.
>
> An alternative to discard is a filesystem level choice of reusing
> blocks. Rather than immediately issuing a discard, we can choose to
> reuse a block which will cause the underlying FTL to invalidate a block
> that we would have discarded. This saves the work done by the initial
> discard. There will have to be a balance here though with saving discard
> with hopes of reusing it and eventually issuing the discards.
>
> Another issue is each drive has its own proprietary FTL which introduces
> a wide variability of performance with discard. Some drives don't need
> discard to perform well and others need discard enabled. There is also a
> class of drives where discard is so not performant that it locks up the
> drive. I would like to discuss how we can quantify this and help tune
> discard within the kernel.
>
> In summary, I'd like to explore discard more and understand how we can
> build a more balanced discard experience where end users don't have to
> be super cognizant of the implications on their applications.
>
> [1] https://lore.kernel.org/linux-fsdevel/92ab41f7-35bc-0f56-056f-ed88526b8ea4@gmail.com/
>
> Thanks,
> Dennis

Hi Dennis,

For your example of "bursts of large discard requests" on XFS, that
sounds almost like the "eofblocks trims" that occur after a large file
copy.  This was addressed by the following patch:

commit 13b86fc33718c6b504baa472437ae14a33abc138
Date:   Wed May 9 08:45:04 2018 -0700
    xfs: skip online discard during eofblocks trims

...which should be in Linux 4.17.  I remember this patch, because I
reported seeing a 4 GB batch of discards after a write of a couple of
32 GB files.  I noticed it from the discard counters in VDO, and then
confirmed it with blktrace / seekwatcher.

There's the potential that what you're seeing might be something else,
but out of curiosity, did the systems exhibiting the bursts of
discards have a kernel that was older than 4.17?


Thanks,

Bryan