Re: Why is dedup inline, not delayed (as opposed to offline)? Explain like I'm five pls.

[Rich Freeman <r-btrfs@gw.thefreemanclan.net>]

On Sat, Jan 16, 2016 at 9:10 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> Al posted on Sat, 16 Jan 2016 12:27:16 +0000 as excerpted:
>
>> Is there any urgency for dedup? What's wrong with storing the hash on
>> disk with the block and having a separate process dedup the written data
>> over time;
>
> There's actually uses for both inline and out-of-line[1] aka delayed
> dedup.  Btrfs already has a number of independent products doing various
> forms of out-of-line dedup, so what's missing and being developed now is
> the inline dedup option, which being directly in the write processing,
> must be handled by btrfs itself -- it can't be primarily done by third
> parties with just a few kernel calls, like out-of-line dedup can.

Does the out-of-line dedup option actually utilize stored hashes, or
is it forced to re-read all the data to compute hashes?  If it is
collecting checksums/etc is this done efficiently?

I think he is actually suggesting a hybrid approach where a bit of
effort is done during operations to greatly streamline out-of-line
deduplication.  I'm not sure how close we are to that already, or if
any room for improvement remains.

--
Rich