From: Mark Fasheh <mfasheh@suse.de>
To: Konstantinos Skarlatos <k.skarlatos@gmail.com>
Cc: Brendan Hide <brendan@swiftspirit.co.za>,
Scott Middleton <scott@assuretek.com.au>,
linux-btrfs@vger.kernel.org
Subject: Re: send/receive and bedup
Date: Tue, 20 May 2014 15:37:02 -0700 [thread overview]
Message-ID: <20140520223702.GQ27178@wotan.suse.de> (raw)
In-Reply-To: <537A80B6.9080202@gmail.com>
On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote:
>> Duperemove will be shipping as supported software in a major SUSE release so
>> it will be bug fixed, etc as you would expect. At the moment I'm very busy
>> trying to fix qgroup bugs so I haven't had much time to add features, or
>> handle external bug reports, etc. Also I'm not very good at advertising my
>> software which would be why it hasn't really been mentioned on list lately
>> :)
>>
>> I would say that state that it's in is that I've gotten the feature set to a
>> point which feels reasonable, and I've fixed enough bugs that I'd appreciate
>> folks giving it a spin and providing reasonable feedback.
> Well, after having good results with duperemove with a few gigs of data, i
> tried it on a 500gb subvolume. After it scanned all files, it is stuck at
> 100% of one cpu core for about 5 hours, and still hasn't done any deduping.
> My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats
> not the problem. So I guess the speed of duperemove drops dramatically as
> data volume increases.
Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data
sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly
have figured out. It will need a bit of work before it can handle entire
file systems. My guess is that it was spending an enormous amount of time
finding duplicates (it has a very thorough check that could probably be
optimized).
For what it's worth, handling larger data sets is the type of work I want to
be doing on it in the future.
--Mark
--
Mark Fasheh
next prev parent reply other threads:[~2014-05-20 22:37 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-05-12 12:27 send/receive and bedup Scott Middleton
2014-05-14 13:20 ` Duncan
2014-05-14 15:36 ` Scott Middleton
2014-05-19 1:07 ` Marc MERLIN
2014-05-19 13:00 ` Scott Middleton
2014-05-19 16:01 ` Brendan Hide
2014-05-19 17:12 ` Konstantinos Skarlatos
2014-05-19 17:55 ` Mark Fasheh
2014-05-19 17:59 ` Austin S Hemmelgarn
2014-05-19 18:27 ` Mark Fasheh
2014-05-19 17:38 ` Mark Fasheh
2014-05-19 22:07 ` Konstantinos Skarlatos
2014-05-20 11:12 ` Scott Middleton
2014-05-20 22:37 ` Mark Fasheh [this message]
2014-05-20 22:56 ` Konstantinos Skarlatos
2014-05-21 0:58 ` Chris Murphy
2014-05-23 15:48 ` Konstantinos Skarlatos
2014-05-23 16:24 ` Chris Murphy
2014-05-21 3:59 ` historical backups with hardlinks vs cp --reflink vs snapshots Marc MERLIN
2014-05-22 4:24 ` Russell Coker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140520223702.GQ27178@wotan.suse.de \
--to=mfasheh@suse.de \
--cc=brendan@swiftspirit.co.za \
--cc=k.skarlatos@gmail.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=scott@assuretek.com.au \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).