Re: About in-band dedupe for v4.7

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 05/13/2016 08:14 PM, Austin S. Hemmelgarn wrote:
> On 2016-05-12 16:54, Mark Fasheh wrote:
>> On Wed, May 11, 2016 at 07:36:59PM +0200, David Sterba wrote:
>>> On Tue, May 10, 2016 at 07:52:11PM -0700, Mark Fasheh wrote:
>>>> Taking your history with qgroups out of this btw, my opinion does not
>>>> change.
>>>>
>>>> With respect to in-memory only dedupe, it is my honest opinion that
>>>> such a
>>>> limited feature is not worth the extra maintenance work. In particular
>>>> there's about 800 lines of code in the userspace patches which I'm sure
>>>> you'd want merged, because how could we test this then?
>>>
>>> I like the in-memory dedup backend. It's lightweight, only a heuristic,
>>> does not need any IO or persistent storage. OTOH I consider it a subpart
>>> of the in-band deduplication that does all the persistency etc. So I
>>> treat the ioctl interface from a broader aspect.
>>
>> Those are all nice qualities, but what do they all get us?
>>
>> For example, my 'large' duperemove test involves about 750 gigabytes of
>> general purpose data - quite literally /home off my workstation.
>>
>> After the run I'm usually seeing between 65-75 gigabytes saved for a
>> total
>> of only 10% duplicated data. I would expect this to be fairly 'average' -
>> /home on my machine has the usual stuff - documents, source code, media,
>> etc.
>>
>> So if you were writing your whole fs out you could expect about the same
>> from inline dedupe - 10%-ish. Let's be generous and go with that number
>> though as a general 'this is how much dedupe we get'.
>>
>> What the memory backend is doing then is providing a cache of
>> sha256/block
>> calculations. This cache is very expensive to fill, and every written
>> block
>> must go through it. On top of that, the cache does not persist between
>> mounts, and has items regularly removed from it when we run low on
>> memory.
>> All of this will drive down the amount of duplicated data we can find.
>>
>> So our best case savings is probably way below 10% - let's be _really_
>> nice
>> and say 5%.
>>
>> Now ask yourself the question - would you accept a write cache which is
>> expensive to fill and would only have a hit rate of less than 5%?
> In-band deduplication is a feature that is not used by typical desktop
> users or even many developers because it's computationally expensive,
> but it's used _all the time_ by big data-centers and similar places
> where processor time is cheap and storage efficiency is paramount.
> Deduplication is more useful in general the more data you have.  5% of 1
> TB is 20 GB, which is not much.  5% of 1 PB is 20 TB, which is at least
> 3-5 disks, which can then be used for storing more data, or providing
> better resiliency against failures.
>
> To look at it another way, deduplicating an individual's home directory
> will almost never get you decent space savings, the majority of shared
> data is usually file headers and nothing more, which can't be
> deduplicated efficiently because of block size requirements.
> Deduplicating all the home directories on a terminal server with 500
> users usually will get you decent space savings, as there very likely
> are a number of files that multiple people have exact copies of, but
> most of them are probably not big files.  Deduplicating the entirety of
> a multi-petabyte file server used for storing VM disk images will
> probably save you a very significant amount of space, because the
> probability of having data that can be deduplicated goes up as you store
> more data, and there is likely to be a lot of data shared between the
> disk images.
>
> This is exactly why I don't use deduplication on any of my personal
> systems.  On my laptop, the space saved is just not worth the time spent
> doing it, as I fall pretty solidly into the first case (most of the data
> duplication on my systems is in file headers).  On my home server, I'm
> not storing enough data with sufficient internal duplication that it
> would save more than 10-20 GB, which doesn't matter for me given that
> I'm using roughly half of the 2.2 TB of effective storage space I have.
> However, once we (eventually) get all the file servers where I work
> moved over to Linux systems running BTRFS, we will absolutely be using
> deduplication there, as we have enough duplication in our data that it
> will probably cut our storage requirements by around 20% on average.
> --

Nice to hear that.

Although the feature is not going to be merged soon, but there is some 
info which may help anyone interested with inband dedupe to plan their 
usage or tests.

1) CPU core number
    Since currently in-band dedupe balance all SHA256 calculation using
    async-thread, for default thread_pool number, it's
    max(nr_cpu + 2, 8).
    So if using default thread_pool number, and the system doesn't have
    more than 8 cores, all CPU can be easily eaten up by dedupe.

    So either use dedupe with default thread_pool setting when you have
    more  than 8 cores, or tunning thread_pool to nr_cpu - 1 or 2.

    Or your system will be less responsive then running delalloc
    (fsync/sync).

2) Dedupe speed.
    In our internal 1 core test. It's about the same speed compared to
    compress=lzo mount option or a little faster, when the data doesn't
    have a quite good compress ratio (80~100% compression ratio).

    We will do more completely performance btests ased on file contents
    (compress ratio and dedupe ratio), core numbers, and more different
    settings like dedupe bs, when writing all these things into wiki.

    But personally I recommend to think twice before choosing compress
    with dedupe.
    It's really dependent on the file contents.

3) Per-file/dir control.
    Same as compression, don't forget such useful flag (although will be
    delayed even further). It would save a lot of CPU time.

    So if tuned well (admin knows the dedupe hotspot), the space save
    can go up to 40% for specific dirs, while doesn't spend any extra CPU
    for other data.

To conclude, this feature is not intended to be used without any 
thinking/reading/checking/tuning, and not for everyone.

Although any one can try dedupe very easy on existing btrfs with patched 
kernel, and then mount it back with stable kernel after the trial.

Just as Austin said, it's some useful feature for large storage server 
where CPU time is relative cheap compared to storage or the data is 
specially patterned for dedupe.

Thanks,
Qu