From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from slmp-550-94.slc.westdc.net ([50.115.112.57]:54911 "EHLO slmp-550-94.slc.westdc.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1751288AbaEUA6Z convert rfc822-to-8bit (ORCPT ); Tue, 20 May 2014 20:58:25 -0400 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) Subject: Re: send/receive and bedup From: Chris Murphy In-Reply-To: <537BDDB4.1050406@gmail.com> Date: Tue, 20 May 2014 18:58:22 -0600 Cc: Mark Fasheh , Brendan Hide , Scott Middleton , linux-btrfs@vger.kernel.org Message-Id: References: <20140519010705.GI10566@merlins.org> <537A2AD5.9050507@swiftspirit.co.za> <20140519173854.GN27178@wotan.suse.de> <537A80B6.9080202@gmail.com> <20140520223702.GQ27178@wotan.suse.de> <537BDDB4.1050406@gmail.com> To: Konstantinos Skarlatos Sender: linux-btrfs-owner@vger.kernel.org List-ID: On May 20, 2014, at 4:56 PM, Konstantinos Skarlatos wrote: > On 21/5/2014 1:37 πμ, Mark Fasheh wrote: >> On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote: >>>> Duperemove will be shipping as supported software in a major SUSE release so >>>> it will be bug fixed, etc as you would expect. At the moment I'm very busy >>>> trying to fix qgroup bugs so I haven't had much time to add features, or >>>> handle external bug reports, etc. Also I'm not very good at advertising my >>>> software which would be why it hasn't really been mentioned on list lately >>>> :) >>>> >>>> I would say that state that it's in is that I've gotten the feature set to a >>>> point which feels reasonable, and I've fixed enough bugs that I'd appreciate >>>> folks giving it a spin and providing reasonable feedback. >>> Well, after having good results with duperemove with a few gigs of data, i >>> tried it on a 500gb subvolume. After it scanned all files, it is stuck at >>> 100% of one cpu core for about 5 hours, and still hasn't done any deduping. >>> My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats >>> not the problem. So I guess the speed of duperemove drops dramatically as >>> data volume increases. >> Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data >> sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly >> have figured out. It will need a bit of work before it can handle entire >> file systems. My guess is that it was spending an enormous amount of time >> finding duplicates (it has a very thorough check that could probably be >> optimized). > It finished after 9 or so hours, so I agree it was checking for duplicates. It does a few GB in just seconds, so time probably scales exponentially with data size. I'm going to guess it ran out of memory. I wonder what happens if you take an SSD and specify a humongous swap partition on it. Like, 4x, or more, the amount of installed memory. This same trick has been mentioned on the XFS list for use with xfsrepair when memory requirements exceed system memory, and is immensely faster. Chris Murphy