* Deduplication tools
@ 2017-04-13 11:06 Marat Khalili
2017-04-13 11:40 ` Austin S. Hemmelgarn
2017-04-13 13:19 ` Duncan
0 siblings, 2 replies; 3+ messages in thread
From: Marat Khalili @ 2017-04-13 11:06 UTC (permalink / raw)
To: linux-btrfs
After reading this maillist for a while I became a bit more cautious
about using various BTRFS features, so decided to ask just in case: is
it safe to use out-of-band deduplication tools
<https://btrfs.wiki.kernel.org/index.php/Deduplication>, and which of
them are considered more stable/mainstream? Also, won't running these
tools exacerbate often mentioned stability/performance problems with
too-many-snapshots? Any first-hand experience is very welcome.
--
With Best Regards,
Marat Khalili
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Deduplication tools
2017-04-13 11:06 Deduplication tools Marat Khalili
@ 2017-04-13 11:40 ` Austin S. Hemmelgarn
2017-04-13 13:19 ` Duncan
1 sibling, 0 replies; 3+ messages in thread
From: Austin S. Hemmelgarn @ 2017-04-13 11:40 UTC (permalink / raw)
To: Marat Khalili, linux-btrfs
On 2017-04-13 07:06, Marat Khalili wrote:
> After reading this maillist for a while I became a bit more cautious
> about using various BTRFS features, so decided to ask just in case: is
> it safe to use out-of-band deduplication tools
> <https://btrfs.wiki.kernel.org/index.php/Deduplication>, and which of
> them are considered more stable/mainstream? Also, won't running these
> tools exacerbate often mentioned stability/performance problems with
> too-many-snapshots? Any first-hand experience is very welcome.
As a general rule, as long as you're careful, you generally shouldn't
have many issues. duperemove is the tool I would suggest for just
generic deduplication on BTRFS, as I know the developer is still active
and generally does a good job of getting bugs fixed. It may well make
performance problems with large numbers of snapshots worse, but probably
not by as much as you think (unless you have huge amounts of duplicate
data). Keep in mind also that batch deduplication can take a very long
time to do.
As a general rule though, if you're storing data that's consistently
structured and organized, you may want to consider doing a custom tool
to make the process more efficient. Any generic deduplication tool is
generally going to be pretty slow on large amounts of data, but
depending on what the data is and how it's organized, it may be possible
to determine duplicate data more efficiently than the block hashing
method used by most deduplication tools.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Deduplication tools
2017-04-13 11:06 Deduplication tools Marat Khalili
2017-04-13 11:40 ` Austin S. Hemmelgarn
@ 2017-04-13 13:19 ` Duncan
1 sibling, 0 replies; 3+ messages in thread
From: Duncan @ 2017-04-13 13:19 UTC (permalink / raw)
To: linux-btrfs
Marat Khalili posted on Thu, 13 Apr 2017 14:06:07 +0300 as excerpted:
> After reading this maillist for a while I became a bit more cautious
> about using various BTRFS features, so decided to ask just in case: is
> it safe to use out-of-band deduplication tools
> <https://btrfs.wiki.kernel.org/index.php/Deduplication>, and which of
> them are considered more stable/mainstream? Also, won't running these
> tools exacerbate often mentioned stability/performance problems with
> too-many-snapshots? Any first-hand experience is very welcome.
Dedupe works by reflinking, which is how snapshots work too, so yes,
it'll trigger the same general issues.
However, while snapshots will reflink /everything/ in the snapshot, one
could hope dedupe won't find /that/ many duplicates and thus won't reflink
at the scale that snapshotting does. Tho of course if it doesn't find
at least some reasonable number of duplicates, it's hardly worth the
trouble to run.
So I'd say it may be worth doing, but do be aware of the scale of dedupe
reflinking. What I'd do here is consider that and then count it as
roughly another snapshot or two, or whatever, depending on how much
dedupeing you actually get.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2017-04-13 13:19 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-04-13 11:06 Deduplication tools Marat Khalili
2017-04-13 11:40 ` Austin S. Hemmelgarn
2017-04-13 13:19 ` Duncan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.