send/receive and bedup

send/receive and bedup

[Scott Middleton]

Hi Everyone

History:
I just recently discovered BtrFS. Well really only just started
reading a lot about it. Starting with blogs by Jim Salters and Marc
Merlin. So, thanks for those blogs guys.

This also introduced me to ZoL (ZFS). It seemed a bit more stable but
one of the features I really wanted was deduplication and needing 20GB
RAM for 1TB of deduped data and the fact it is always on - pushed me
away. Some of those guys really don't like BtrFS BTW!

What I want to be able to do is backup Virtual images (VirtualBox and
some older VMware) over ADSL. I hoped a mixture of Dedupe and the
send/receive functions of BtrFS might accomplish this.

I am in the process of building a test BtrFS system at home now but I
hope to put it into production in the next few months.


Current server is a Dual Xeon, Intel server board, 32GB RAM, 2 x 2TB
Hardware RAID SAS (consisting of 4 x 2TB SATA drives) as / and /home
in ext4 format. I also have 2 unused 4TB SATA drives that will be
eventually be BtrFS RAID1 as /Backup. Ubuntu 14.04. Its is mainly a VM
host, small file storage for word docs etc, some large archival .pst
files and shadowprotect backups of the Terminal Server. I only just
built this server over the last weekend to replace their other aging
server and I purposely over engineered it.

Onsite there is also a backup server that is pretty basic with 2 x 4TB
HDDs and 8GB RAM. I plan on converting it to BtrFS as well. Currently
is Ubuntu 12.04 but I will be upgrading it soon to 14.04.

Offsite in a data centre I have an aging 1 RU server that I will be
upgrading. It'll probably have 8GB RAM, 1 X 60GB SSD as boot/swap and
2 X 4TB HDD BtrFS in RAID 1. Currently running 32 bit Debian 7.5. It
has had many partial hardware and OS upgrades over the years as it
originally started as Slink or even Hamm. Time to start again since I
need to move to 64bit.

What I want to do is backup the / and /home directories on the main
server to /Backup BtrFS directory, run bedup then "send" it to the
onsite backup server. The onsite backup server will "send" it to the
offsite server. I am assuming (correctly I hope) that the
deduplication will also be replicated across the machines.  I'll have
NOCOW on the VM images, Archived PST files, Shadow protect images and
some other stuff.

I guess the first question is this even possible? I don't believe that
much actual non duplicated data changes all that much mostly just word
docs that I already send offsite. I'm really hoping to backup the VMs
and the shadow protects offsite as well. I can  upgrade the broadband
to fibre but before I do that (spend a lot of money) I want to be able
to see that it would be possible.


Kind Regards

-
Scott

Re: send/receive and bedup

[Duncan]

Scott Middleton posted on Mon, 12 May 2014 20:27:13 +0800 as excerpted:

> Hi Everyone
> 
> History:
> I just recently discovered BtrFS. Well really only just started reading
> a lot about it. Starting with blogs by Jim Salters and Marc Merlin. So,
> thanks for those blogs guys.
> 
> This also introduced me to ZoL (ZFS). It seemed a bit more stable but
> one of the features I really wanted was deduplication and needing 20GB
> RAM for 1TB of deduped data and the fact it is always on - pushed me
> away. Some of those guys really don't like BtrFS BTW!
> 
> What I want to be able to do is backup Virtual images (VirtualBox and
> some older VMware) over ADSL. I hoped a mixture of Dedupe and the
> send/receive functions of BtrFS might accomplish this.
> 
> I am in the process of building a test BtrFS system at home now but I
> hope to put it into production in the next few months.
> 
> 
> Current server is a Dual Xeon, Intel server board, 32GB RAM, 2 x 2TB
> Hardware RAID SAS (consisting of 4 x 2TB SATA drives) as / and /home in
> ext4 format. I also have 2 unused 4TB SATA drives that will be
> eventually be BtrFS RAID1 as /Backup. Ubuntu 14.04. Its is mainly a VM
> host, small file storage for word docs etc, some large archival .pst
> files and shadowprotect backups of the Terminal Server. I only just
> built this server over the last weekend to replace their other aging
> server and I purposely over engineered it.
> 
> Onsite there is also a backup server that is pretty basic with 2 x 4TB
> HDDs and 8GB RAM. I plan on converting it to BtrFS as well. Currently is
> Ubuntu 12.04 but I will be upgrading it soon to 14.04.
> 
> Offsite in a data centre I have an aging 1 RU server that I will be
> upgrading. It'll probably have 8GB RAM, 1 X 60GB SSD as boot/swap and 2
> X 4TB HDD BtrFS in RAID 1. Currently running 32 bit Debian 7.5. It has
> had many partial hardware and OS upgrades over the years as it
> originally started as Slink or even Hamm. Time to start again since I
> need to move to 64bit.
> 
> What I want to do is backup the / and /home directories on the main
> server to /Backup BtrFS directory, run bedup then "send" it to the
> onsite backup server. The onsite backup server will "send" it to the
> offsite server. I am assuming (correctly I hope) that the deduplication
> will also be replicated across the machines.  I'll have NOCOW on the VM
> images, Archived PST files, Shadow protect images and some other stuff.
> 
> I guess the first question is this even possible? I don't believe that
> much actual non duplicated data changes all that much mostly just word
> docs that I already send offsite. I'm really hoping to backup the VMs
> and the shadow protects offsite as well. I can  upgrade the broadband to
> fibre but before I do that (spend a lot of money) I want to be able to
> see that it would be possible.

I left this for a couple days hoping someone else with a more directly 
similar use-case would answer, but none so far, so I'll give it a go...

First some general boilerplate.  Btrfs is still under heavy development 
and keeping current with especially the kernel is *STRONGLY* recommended, 
as every new kernel still brings lots of fixes, meaning if you're running 
an old kernel, you're running known-buggy code with fixes available in a 
current kernel.  Similarly, you probably don't want to let the btrfs-progs 
userspace tools get too outdated either, tho that's not as critical as it 
mostly means not being able to take advantage of the latest features and 
fixes for maintenance, not the risk of operational data loss if one of 
the known-fixed old-version kernel bugs hits that you have when running 
an older kernel.

Second, as you've done some research already you're likely aware of this, 
but just in case, let me give you the link to the wiki.  If you haven't 
read up there, please do, as it's likely to be quite helpful. =:^)

Memory or bookmark: https://btrfs.wiki.kernel.org

User documentation bookmark:

https://btrfs.wiki.kernel.org/index.php/Main_Page#Guides_and_usage_information

On to your proposed setup.  In general, it looks reasonable.

My first concern upon reading about the VM images was of course the 
fragmentation issues that come with the VM images territory on btrfs, but 
if you keep your working partitions as ext4 and use btrfs primarily for 
backup, that issue goes away to a large extent, since the operational 
rewriting will be happening on the ext4, which should handle it a bit 
better than btrfs does at this point, while btrfs will only be getting 
the more sequentially written backups and not have to deal with the live 
in-place updates on the VM images.

The problem with btrfs hosting operational VM images is snapshots, 
particularly when using btrfs send for backup and/or when doing frequent, 
often scripted, snapshotting.  The problem with send is that it takes a 
read-only snapshot and sends from that, so it's a snapshotting issue 
either way.  The problem with snapshots is that for NOCOW files, the 
first write to a block after a snapshot still triggers a COW write, since 
the snapshot locked the existing version in place.

So frequent snapshotting has the effect of nullifying NOCOW for big VM 
images in operation due to the operational internal-write pattern, thus 
being a big problem that the devs are still working on finding a 
reasonably satisfactory solution for.

But by keeping your operational VM images on ext4 and only using btrfs 
for originally local backups, then using btrfs send to replicate those 
backups to onsite and then offsite remote, you should be avoiding the 
biggest problem, since the backup write-pattern should be far more 
sequential and not trigger the fragmentation issue you'd have trying to 
host the operational VMs directly on btrfs.

My next concern is with bedup and its interaction with btrfs send and 
snapshots.  I simply don't have enough knowledge in that area to be able 
to comment intelligently on how all the pieces fit together there, but 
you'll need to ensure a couple things, for sure.  Btrfs send starts with 
a full send, then uses that original base, kept the same on both sides, 
as a reference for the incremental sends you do later.  Both the original 
backup ext4->btrfs and the bedup processes will need to be setup so as 
not to interfere with the efficiency of those incremental sends, or if it 
works at all, you'll be effectively resending the entire thing each time, 
which isn't what you want at all.  Unfortunately I simply don't know 
enough about the interaction between the pieces to say whether your plan 
is reasonable there or not, and if so, how to actually do it.  This is 
why I was hoping someone else with more direct experience/knowledge would 
reply, but...

Next is bedup itself.  I'm not sure of the maturity status there, tho I 
don't believe it's fully production-quality yet.  However, it's quite 
possible that to the extent that it isn't production-ready, any breakage 
simply reduces the efficiency, while still helping some.  You'd really 
need to contact the guy working on it to find out.

Finally, btrfs send/receive itself just recently had a whole host of bugs 
fixed, so again, you'll want a very recent kernel and userspace in 
ordered to get those bugfixes.  In fact, I believe some of those fixes 
are only in kernel 3.15-rc, and only in the btrfs-progs integration 
branch, not yet in stable at all.  You can of course try 3.14.x stable 
kernel and 3.14.1 btrfs-progs and hope it works for now, updating if you 
have problems, or go for the pre-releases from the get-go.  Meanwhile, 
while that round of fixes certainly means btrfs send/receive is more 
mature than it was, I'd still strongly recommend having a fallback plan, 
in case you start getting errors and it quits working for you, pending 
further fixes.  IOW, yes, I'd say use send/receive if it works for you, 
but at this point, don't count on it actually continuing to work every 
time, and have a fallback if it breaks temporarily, so you're not as they 
say left up a creek without a paddle.

Hope that helps and good luck.  Looking forward to seeing more posts as 
you experiment, and hopefully you'll then stick around to add your 
experiences to the wiki and answer questions about that use-case here as 
others may have them. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: send/receive and bedup

[Scott Middleton]

Hi



> I left this for a couple days hoping someone else with a more directly
> similar use-case would answer, but none so far, so I'll give it a go...

Thanks for getting back to me mate!


>
> First some general boilerplate.  Btrfs is still under heavy development
> and keeping current with especially the kernel is *STRONGLY* recommended,
> as every new kernel still brings lots of fixes, meaning if you're running
> an old kernel, you're running known-buggy code with fixes available in a
> current kernel.  Similarly, you probably don't want to let the btrfs-progs
> userspace tools get too outdated either, tho that's not as critical as it
> mostly means not being able to take advantage of the latest features and
> fixes for maintenance, not the risk of operational data loss if one of
> the known-fixed old-version kernel bugs hits that you have when running
> an older kernel.
>

My test environment is:
root@Ubuntu-14:~/btrfs/btrfs-progs# uname -a
Linux Ubuntu-14 3.14.1-031401-generic #201404141220 SMP Mon Apr 14
16:21:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

root@Ubuntu-14:~/btrfs/btrfs-progs# which btrfs
/usr/local/bin/btrfs
root@Ubuntu-14:~/btrfs/btrfs-progs# btrfs --version
Btrfs v3.14.1

I read so much about BtrFS that I mistaked Bedup with Duperemove.
Duperemove is actually what I am testing.

I also plan on testing SDFS - opendedup again! They have some
development since the last time I tried,  Not a fan of having to use
Java though!

> Second, as you've done some research already you're likely aware of this,
> but just in case, let me give you the link to the wiki.  If you haven't
> read up there, please do, as it's likely to be quite helpful. =:^)
>
> Memory or bookmark: https://btrfs.wiki.kernel.org


Lots of bookmarks. One of my faves is:
https://wiki.archlinux.org/index.php/Btrfs


It will be interesting on what happens. I rsync'd the clients data
from a 2TB mdadm RAID to a standard 3TB BtrFS drive, Currently running
 Duperemove on it. I reckon it will take days but it will be
interesting to see what happens.

Re: send/receive and bedup

[Marc MERLIN]

On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote:
> I read so much about BtrFS that I mistaked Bedup with Duperemove.
> Duperemove is actually what I am testing.

I'm currently using programs that find files that are the same, and
hardlink them together:
http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html

hardlink.py actually seems to be the faster (memory and CPU) one event
though it's in python.
I can get others to run out of RAM on my 8GB server easily :(

Bedup should be better, but last I tried I couldn't get it to work.
It's been updated since then, I just haven't had the chance to try it
again since then.

Please post what you find out, or if you have a hardlink maker that's
better than the ones I found :)

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: send/receive and bedup

[Scott Middleton]

On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote:
> On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote:
>> I read so much about BtrFS that I mistaked Bedup with Duperemove.
>> Duperemove is actually what I am testing.
>
> I'm currently using programs that find files that are the same, and
> hardlink them together:
> http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html
>
> hardlink.py actually seems to be the faster (memory and CPU) one event
> though it's in python.
> I can get others to run out of RAM on my 8GB server easily :(
>
> Bedup should be better, but last I tried I couldn't get it to work.
> It's been updated since then, I just haven't had the chance to try it
> again since then.
>
> Please post what you find out, or if you have a hardlink maker that's
> better than the ones I found :)
>


Thanks for that.

I may be  completely wrong in my approach.

I am not looking for a file level comparison. Bedup worked fine for
that. I have a lot of virtual images and shadow protect images where
only a few megabytes may be the difference. So a file level hash and
comparison doesn't really achieve my goals.

I thought duperemove may be on a lower level.

https://github.com/markfasheh/duperemove

"Duperemove is a simple tool for finding duplicated extents and
submitting them for deduplication. When given a list of files it will
hash their contents on a block by block basis and compare those hashes
to each other, finding and categorizing extents that match each
other. When given the -d option, duperemove will submit those
extents for deduplication using the btrfs-extent-same ioctl."

It defaults to 128k but you can make it smaller.

I hit a hurdle though. The 3TB HDD  I used seemed OK when I did a long
SMART test but seems to die every few hours. Admittedly it was part of
a failed mdadm RAID array that I pulled out of a clients machine.

The only other copy I have of the data is the original mdadm array
that was recently replaced with a new server, so I am loathe to use
that HDD yet. At least for another couple of weeks!


I am still hopeful duperemove will work.

In another month I will put the 2 X 4TB HDDs online in BtrFS RAID 1
format on the production machine and have a crack on duperemove on
that after hours. I will convert the onsite backup machine to BtrFS
with its 2 x 4TB HDDs to BtrFS not long after.

The ultimate goal is to be able to back up on a block level very large
files offsite where maybe a GB is changed on a daily basis. I realise
that I will have to make an original copy and manually take that to my
datacentre but hopefully I can backup multiple clients data after
hours, or possibly,  a trickle, constantly.

Kind Regards

Scott

Re: send/receive and bedup

[Brendan Hide]

On 19/05/14 15:00, Scott Middleton wrote:
> On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote:
>> On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote:
>>> I read so much about BtrFS that I mistaked Bedup with Duperemove.
>>> Duperemove is actually what I am testing.
>> I'm currently using programs that find files that are the same, and
>> hardlink them together:
>> http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html
>>
>> hardlink.py actually seems to be the faster (memory and CPU) one event
>> though it's in python.
>> I can get others to run out of RAM on my 8GB server easily :(

Interesting app.

An issue with hardlinking (with the backups use-case, this problem isn't likely to happen), is that if you modify a file, all the hardlinks get changed along with it - including the ones that you don't want changed.

@Marc: Since you've been using btrfs for a while now I'm sure you've already considered whether or not a reflink copy is the better/worse option.

>>
>> Bedup should be better, but last I tried I couldn't get it to work.
>> It's been updated since then, I just haven't had the chance to try it
>> again since then.
>>
>> Please post what you find out, or if you have a hardlink maker that's
>> better than the ones I found :)
>>
>
> Thanks for that.
>
> I may be  completely wrong in my approach.
>
> I am not looking for a file level comparison. Bedup worked fine for
> that. I have a lot of virtual images and shadow protect images where
> only a few megabytes may be the difference. So a file level hash and
> comparison doesn't really achieve my goals.
>
> I thought duperemove may be on a lower level.
>
> https://github.com/markfasheh/duperemove
>
> "Duperemove is a simple tool for finding duplicated extents and
> submitting them for deduplication. When given a list of files it will
> hash their contents on a block by block basis and compare those hashes
> to each other, finding and categorizing extents that match each
> other. When given the -d option, duperemove will submit those
> extents for deduplication using the btrfs-extent-same ioctl."
>
> It defaults to 128k but you can make it smaller.
>
> I hit a hurdle though. The 3TB HDD  I used seemed OK when I did a long
> SMART test but seems to die every few hours. Admittedly it was part of
> a failed mdadm RAID array that I pulled out of a clients machine.
>
> The only other copy I have of the data is the original mdadm array
> that was recently replaced with a new server, so I am loathe to use
> that HDD yet. At least for another couple of weeks!
>
>
> I am still hopeful duperemove will work.
Duperemove does look exactly like what you are looking for. The last 
traffic on the mailing list regarding that was in August last year. It 
looks like it was pulled into the main kernel repository on September 1st.

The last commit to the duperemove application was on April 20th this 
year. Maybe Mark (cc'd) can provide further insight on its current status.

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: send/receive and bedup

[Konstantinos Skarlatos]

On 19/5/2014 7:01 μμ, Brendan Hide wrote:
> On 19/05/14 15:00, Scott Middleton wrote:
>> On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote:
>>> On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote:
>>>> I read so much about BtrFS that I mistaked Bedup with Duperemove.
>>>> Duperemove is actually what I am testing.
>>> I'm currently using programs that find files that are the same, and
>>> hardlink them together:
>>> http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html 
>>>
>>>
>>> hardlink.py actually seems to be the faster (memory and CPU) one event
>>> though it's in python.
>>> I can get others to run out of RAM on my 8GB server easily :(
>
> Interesting app.
>
> An issue with hardlinking (with the backups use-case, this problem 
> isn't likely to happen), is that if you modify a file, all the 
> hardlinks get changed along with it - including the ones that you 
> don't want changed.
>
> @Marc: Since you've been using btrfs for a while now I'm sure you've 
> already considered whether or not a reflink copy is the better/worse 
> option.
>
>>>
>>> Bedup should be better, but last I tried I couldn't get it to work.
>>> It's been updated since then, I just haven't had the chance to try it
>>> again since then.
>>>
>>> Please post what you find out, or if you have a hardlink maker that's
>>> better than the ones I found :)
>>>
>>
>> Thanks for that.
>>
>> I may be  completely wrong in my approach.
>>
>> I am not looking for a file level comparison. Bedup worked fine for
>> that. I have a lot of virtual images and shadow protect images where
>> only a few megabytes may be the difference. So a file level hash and
>> comparison doesn't really achieve my goals.
>>
>> I thought duperemove may be on a lower level.
>>
>> https://github.com/markfasheh/duperemove
>>
>> "Duperemove is a simple tool for finding duplicated extents and
>> submitting them for deduplication. When given a list of files it will
>> hash their contents on a block by block basis and compare those hashes
>> to each other, finding and categorizing extents that match each
>> other. When given the -d option, duperemove will submit those
>> extents for deduplication using the btrfs-extent-same ioctl."
>>
>> It defaults to 128k but you can make it smaller.
>>
>> I hit a hurdle though. The 3TB HDD  I used seemed OK when I did a long
>> SMART test but seems to die every few hours. Admittedly it was part of
>> a failed mdadm RAID array that I pulled out of a clients machine.
>>
>> The only other copy I have of the data is the original mdadm array
>> that was recently replaced with a new server, so I am loathe to use
>> that HDD yet. At least for another couple of weeks!
>>
>>
>> I am still hopeful duperemove will work.
> Duperemove does look exactly like what you are looking for. The last 
> traffic on the mailing list regarding that was in August last year. It 
> looks like it was pulled into the main kernel repository on September 
> 1st.
>
> The last commit to the duperemove application was on April 20th this 
> year. Maybe Mark (cc'd) can provide further insight on its current 
> status.
>
I have been testing duperemove and it seems to work just fine, in 
contrast with bedup that i have been unable to install/compile/sort out 
the mess with python versions. I have 2 questions about duperemove:
1) can it use existing filesystem csums instead of calculating its own?
2) can it be included in btrfs-progs so that it becomes a standard 
feature of btrfs?
Thanks

Re: send/receive and bedup

[Mark Fasheh]

On Mon, May 19, 2014 at 06:01:25PM +0200, Brendan Hide wrote:
> On 19/05/14 15:00, Scott Middleton wrote:
>> On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote:
>> Thanks for that.
>>
>> I may be  completely wrong in my approach.
>>
>> I am not looking for a file level comparison. Bedup worked fine for
>> that. I have a lot of virtual images and shadow protect images where
>> only a few megabytes may be the difference. So a file level hash and
>> comparison doesn't really achieve my goals.
>>
>> I thought duperemove may be on a lower level.
>>
>> https://github.com/markfasheh/duperemove
>>
>> "Duperemove is a simple tool for finding duplicated extents and
>> submitting them for deduplication. When given a list of files it will
>> hash their contents on a block by block basis and compare those hashes
>> to each other, finding and categorizing extents that match each
>> other. When given the -d option, duperemove will submit those
>> extents for deduplication using the btrfs-extent-same ioctl."
>>
>> It defaults to 128k but you can make it smaller.
>>
>> I hit a hurdle though. The 3TB HDD  I used seemed OK when I did a long
>> SMART test but seems to die every few hours. Admittedly it was part of
>> a failed mdadm RAID array that I pulled out of a clients machine.
>>
>> The only other copy I have of the data is the original mdadm array
>> that was recently replaced with a new server, so I am loathe to use
>> that HDD yet. At least for another couple of weeks!
>>
>>
>> I am still hopeful duperemove will work.
> Duperemove does look exactly like what you are looking for. The last 
> traffic on the mailing list regarding that was in August last year. It 
> looks like it was pulled into the main kernel repository on September 1st.

I'm confused - you need to avoid a file scan completely? Duperemove does do
that just to be clear.

In your mind, what would be the alternative to that sort of a scan?

By the way, if you know exactly where the changes are you
could just feed the duplicate extents directly to the ioctl via a script. I
have a small tool in the duperemove repositry that can do that for you
('make btrfs-extent-same').


> The last commit to the duperemove application was on April 20th this year. 
> Maybe Mark (cc'd) can provide further insight on its current status.

Duperemove will be shipping as supported software in a major SUSE release so
it will be bug fixed, etc as you would expect. At the moment I'm very busy
trying to fix qgroup bugs so I haven't had much time to add features, or
handle external bug reports, etc. Also I'm not very good at advertising my
software which would be why it hasn't really been mentioned on list lately
:)

I would say that state that it's in is that I've gotten the feature set to a
point which feels reasonable, and I've fixed enough bugs that I'd appreciate
folks giving it a spin and providing reasonable feedback.

There's a TODO list which gives a decent idea of what's on my mind for
possible future improvements. I think what I'm most wanting to do right now
is some sort of (optional) writeout to a file of what was done during a run.
The idea is that you could feed that data back to duperemove to improve the
speed of subsequent runs. My priorities may change depending on feedback
from users of course.

I also at some point want to rewrite some of the duplicate extent finding
code as it got messy and could be a bit faster.
	--Mark

--
Mark Fasheh

Re: send/receive and bedup

[Mark Fasheh]

On Mon, May 19, 2014 at 08:12:03PM +0300, Konstantinos Skarlatos wrote:
> On 19/5/2014 7:01 μμ, Brendan Hide wrote:
>> On 19/05/14 15:00, Scott Middleton wrote:
>> Duperemove does look exactly like what you are looking for. The last 
>> traffic on the mailing list regarding that was in August last year. It 
>> looks like it was pulled into the main kernel repository on September 1st.
>>
>> The last commit to the duperemove application was on April 20th this year. 
>> Maybe Mark (cc'd) can provide further insight on its current status.
>>
> I have been testing duperemove and it seems to work just fine, in contrast 
> with bedup that i have been unable to install/compile/sort out the mess 
> with python versions. I have 2 questions about duperemove:
> 1) can it use existing filesystem csums instead of calculating its own?

Not right now, though that may be something we can feed to it in the future.

I haven't thought about this much and to be honest I don't recall *exactly*
how btrfs stores it's checksums. That said, I think feasibility of doing
this comes down to a few things:

1) how expensive is it to get at the on-disk checksums?

This might not make sense if it's simply faster to scan a file than its
checksums.


2) are they stored in a manner which makes sense for dedupe.

By that I mean, do we have a checksum for every X bytes? If so, then
theoretically life is easy - we just make our blocksize to X and load the
checksums into duperemoves internal block checksum tree. If checksums can
cover arbitrary sized extents than we might not be able to use them at all
or maybe we would have to 'fill in the blanks' so to speak.


3) what is the tradeoff of false positives?

Btrfs checksums are there for detecting bad blocks, as opposed to duplicate
data. The difference is that btrfs doesn't have to use very strong hashing
as a result. So we just want to make sure that we don't wind up passing
*so* many false positives to the kernel that it was just faster to scan the
file and checksum on our own.


Not that any of those questions are super difficult to answer by the
way, it's more about how much time I've had :)


> 2) can it be included in btrfs-progs so that it becomes a standard feature 
> of btrfs?

I have to think about this one personally as it implies some tradeoffs in my
development on duperemove that I'm not sure I want to make yet.
	--Mark

--
Mark Fasheh

Re: send/receive and bedup

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 4806 bytes --]

On 2014-05-19 13:12, Konstantinos Skarlatos wrote:
> On 19/5/2014 7:01 μμ, Brendan Hide wrote:
>> On 19/05/14 15:00, Scott Middleton wrote:
>>> On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote:
>>>> On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote:
>>>>> I read so much about BtrFS that I mistaked Bedup with Duperemove.
>>>>> Duperemove is actually what I am testing.
>>>> I'm currently using programs that find files that are the same, and
>>>> hardlink them together:
>>>> http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html
>>>>
>>>>
>>>> hardlink.py actually seems to be the faster (memory and CPU) one event
>>>> though it's in python.
>>>> I can get others to run out of RAM on my 8GB server easily :(
>>
>> Interesting app.
>>
>> An issue with hardlinking (with the backups use-case, this problem
>> isn't likely to happen), is that if you modify a file, all the
>> hardlinks get changed along with it - including the ones that you
>> don't want changed.
>>
>> @Marc: Since you've been using btrfs for a while now I'm sure you've
>> already considered whether or not a reflink copy is the better/worse
>> option.
>>
>>>>
>>>> Bedup should be better, but last I tried I couldn't get it to work.
>>>> It's been updated since then, I just haven't had the chance to try it
>>>> again since then.
>>>>
>>>> Please post what you find out, or if you have a hardlink maker that's
>>>> better than the ones I found :)
>>>>
>>>
>>> Thanks for that.
>>>
>>> I may be  completely wrong in my approach.
>>>
>>> I am not looking for a file level comparison. Bedup worked fine for
>>> that. I have a lot of virtual images and shadow protect images where
>>> only a few megabytes may be the difference. So a file level hash and
>>> comparison doesn't really achieve my goals.
>>>
>>> I thought duperemove may be on a lower level.
>>>
>>> https://github.com/markfasheh/duperemove
>>>
>>> "Duperemove is a simple tool for finding duplicated extents and
>>> submitting them for deduplication. When given a list of files it will
>>> hash their contents on a block by block basis and compare those hashes
>>> to each other, finding and categorizing extents that match each
>>> other. When given the -d option, duperemove will submit those
>>> extents for deduplication using the btrfs-extent-same ioctl."
>>>
>>> It defaults to 128k but you can make it smaller.
>>>
>>> I hit a hurdle though. The 3TB HDD  I used seemed OK when I did a long
>>> SMART test but seems to die every few hours. Admittedly it was part of
>>> a failed mdadm RAID array that I pulled out of a clients machine.
>>>
>>> The only other copy I have of the data is the original mdadm array
>>> that was recently replaced with a new server, so I am loathe to use
>>> that HDD yet. At least for another couple of weeks!
>>>
>>>
>>> I am still hopeful duperemove will work.
>> Duperemove does look exactly like what you are looking for. The last
>> traffic on the mailing list regarding that was in August last year. It
>> looks like it was pulled into the main kernel repository on September
>> 1st.
>>
>> The last commit to the duperemove application was on April 20th this
>> year. Maybe Mark (cc'd) can provide further insight on its current
>> status.
>>
> I have been testing duperemove and it seems to work just fine, in
> contrast with bedup that i have been unable to install/compile/sort out
> the mess with python versions. I have 2 questions about duperemove:
> 1) can it use existing filesystem csums instead of calculating its own?
While this might seem like a great idea at first, it really isn't.
BTRFS uses CRC32c at the moment as it's checksum algorithm, and while
that is relatively good at detecting small differences (i.e. a single
bit flipped out of every 64 or so bytes), it is known to have issues
with hash collisions.  Normally, the data on disk won't change enough
even from a media error to cause a hash collision, but when you start
using it to compare extents that aren't known to be the same to begin
with, and then try to merge those extents, you run the risk of serious
file corruption.  Also, AFAIK, BTRFS doesn't expose the block checksum
to userspace directly (although I may be wrong about this, in which case
i retract the following statement) this would therefore require some
kernelspace support.
> 2) can it be included in btrfs-progs so that it becomes a standard
> feature of btrfs?
I would definitely like to second this suggestion, I hear a lot of
people talking about how BTRFS has batch deduplication, but it's almost
impossible to make use of without extra software or writing your own code.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]

Re: send/receive and bedup

[Mark Fasheh]

On Mon, May 19, 2014 at 01:59:01PM -0400, Austin S Hemmelgarn wrote:
> On 2014-05-19 13:12, Konstantinos Skarlatos wrote:
> > I have been testing duperemove and it seems to work just fine, in
> > contrast with bedup that i have been unable to install/compile/sort out
> > the mess with python versions. I have 2 questions about duperemove:
> > 1) can it use existing filesystem csums instead of calculating its own?
> While this might seem like a great idea at first, it really isn't.
> BTRFS uses CRC32c at the moment as it's checksum algorithm, and while
> that is relatively good at detecting small differences (i.e. a single
> bit flipped out of every 64 or so bytes), it is known to have issues
> with hash collisions.  Normally, the data on disk won't change enough
> even from a media error to cause a hash collision, but when you start
> using it to compare extents that aren't known to be the same to begin
> with, and then try to merge those extents, you run the risk of serious
> file corruption.  Also, AFAIK, BTRFS doesn't expose the block checksum
> to userspace directly (although I may be wrong about this, in which case
> i retract the following statement) this would therefore require some
> kernelspace support.

I'm pretty sure you could get the checkums via ioctl. The thing about dedupe
though is that kernel is always doing a byte-by-byte comparison of the file
data before merging it so we should never corrupt just because userspace
gave us a bad range to dedupe. That said I don't necessarily disagree that
it might not be as good an idea as it sounds.
	--Mark

--
Mark Fasheh

Re: send/receive and bedup

[Konstantinos Skarlatos]

On 19/5/2014 8:38 μμ, Mark Fasheh wrote:
> On Mon, May 19, 2014 at 06:01:25PM +0200, Brendan Hide wrote:
>> On 19/05/14 15:00, Scott Middleton wrote:
>>> On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote:
>>> Thanks for that.
>>>
>>> I may be  completely wrong in my approach.
>>>
>>> I am not looking for a file level comparison. Bedup worked fine for
>>> that. I have a lot of virtual images and shadow protect images where
>>> only a few megabytes may be the difference. So a file level hash and
>>> comparison doesn't really achieve my goals.
>>>
>>> I thought duperemove may be on a lower level.
>>>
>>> https://github.com/markfasheh/duperemove
>>>
>>> "Duperemove is a simple tool for finding duplicated extents and
>>> submitting them for deduplication. When given a list of files it will
>>> hash their contents on a block by block basis and compare those hashes
>>> to each other, finding and categorizing extents that match each
>>> other. When given the -d option, duperemove will submit those
>>> extents for deduplication using the btrfs-extent-same ioctl."
>>>
>>> It defaults to 128k but you can make it smaller.
>>>
>>> I hit a hurdle though. The 3TB HDD  I used seemed OK when I did a long
>>> SMART test but seems to die every few hours. Admittedly it was part of
>>> a failed mdadm RAID array that I pulled out of a clients machine.
>>>
>>> The only other copy I have of the data is the original mdadm array
>>> that was recently replaced with a new server, so I am loathe to use
>>> that HDD yet. At least for another couple of weeks!
>>>
>>>
>>> I am still hopeful duperemove will work.
>> Duperemove does look exactly like what you are looking for. The last
>> traffic on the mailing list regarding that was in August last year. It
>> looks like it was pulled into the main kernel repository on September 1st.
> I'm confused - you need to avoid a file scan completely? Duperemove does do
> that just to be clear.
>
> In your mind, what would be the alternative to that sort of a scan?
>
> By the way, if you know exactly where the changes are you
> could just feed the duplicate extents directly to the ioctl via a script. I
> have a small tool in the duperemove repositry that can do that for you
> ('make btrfs-extent-same').
>
>
>> The last commit to the duperemove application was on April 20th this year.
>> Maybe Mark (cc'd) can provide further insight on its current status.
> Duperemove will be shipping as supported software in a major SUSE release so
> it will be bug fixed, etc as you would expect. At the moment I'm very busy
> trying to fix qgroup bugs so I haven't had much time to add features, or
> handle external bug reports, etc. Also I'm not very good at advertising my
> software which would be why it hasn't really been mentioned on list lately
> :)
>
> I would say that state that it's in is that I've gotten the feature set to a
> point which feels reasonable, and I've fixed enough bugs that I'd appreciate
> folks giving it a spin and providing reasonable feedback.
Well, after having good results with duperemove with a few gigs of data, 
i tried it on a 500gb subvolume. After it scanned all files, it is stuck 
at 100% of one cpu core for about 5 hours, and still hasn't done any 
deduping. My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i 
guess thats not the problem. So I guess the speed of duperemove drops 
dramatically as data volume increases.

>
> There's a TODO list which gives a decent idea of what's on my mind for
> possible future improvements. I think what I'm most wanting to do right now
> is some sort of (optional) writeout to a file of what was done during a run.
> The idea is that you could feed that data back to duperemove to improve the
> speed of subsequent runs. My priorities may change depending on feedback
> from users of course.
>
> I also at some point want to rewrite some of the duplicate extent finding
> code as it got messy and could be a bit faster.
> 	--Mark
>
> --
> Mark Fasheh
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: send/receive and bedup

[Scott Middleton]

On 20 May 2014 06:07, Konstantinos Skarlatos <k.skarlatos@gmail.com> wrote:
> On 19/5/2014 8:38 μμ, Mark Fasheh wrote:
>>
>
>
> Well, after having good results with duperemove with a few gigs of data, i
> tried it on a 500gb subvolume. After it scanned all files, it is stuck at
> 100% of one cpu core for about 5 hours, and still hasn't done any deduping.
> My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats not
> the problem. So I guess the speed of duperemove drops dramatically as data
> volume increases.
>
>>
>> There's a TODO list which gives a decent idea of what's on my mind for
>> possible future improvements. I think what I'm most wanting to do right
>> now
>> is some sort of (optional) writeout to a file of what was done during a
>> run.
>> The idea is that you could feed that data back to duperemove to improve
>> the
>> speed of subsequent runs. My priorities may change depending on feedback
>> from users of course.
>>
>> I also at some point want to rewrite some of the duplicate extent finding
>> code as it got messy and could be a bit faster.
>>         --Mark

I'm glad about this discussion.

While I am no where near an expert on file systems, my knowledge has
increased a lot through BtrFS.

ZFS uses RAM to store its checksum tables. Opendedup recommends a
separate HDD. Opendedup uses 4k block sizes. Both are always on.

I'm not against using a separate HDD to store csums. Cheaper than RAM,
albeit slower.

The part of duperemove I like is the ability to CHOOSE when and how I
want to dedupe.

Scott

Re: send/receive and bedup

[Mark Fasheh]

On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote:
>> Duperemove will be shipping as supported software in a major SUSE release so
>> it will be bug fixed, etc as you would expect. At the moment I'm very busy
>> trying to fix qgroup bugs so I haven't had much time to add features, or
>> handle external bug reports, etc. Also I'm not very good at advertising my
>> software which would be why it hasn't really been mentioned on list lately
>> :)
>>
>> I would say that state that it's in is that I've gotten the feature set to a
>> point which feels reasonable, and I've fixed enough bugs that I'd appreciate
>> folks giving it a spin and providing reasonable feedback.
> Well, after having good results with duperemove with a few gigs of data, i 
> tried it on a 500gb subvolume. After it scanned all files, it is stuck at 
> 100% of one cpu core for about 5 hours, and still hasn't done any deduping. 
> My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats 
> not the problem. So I guess the speed of duperemove drops dramatically as 
> data volume increases.

Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data
sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly
have figured out. It will need a bit of work before it can handle entire
file systems. My guess is that it was spending an enormous amount of time
finding duplicates (it has a very thorough check that could probably be
optimized).

For what it's worth, handling larger data sets is the type of work I want to
be doing on it in the future.
	--Mark

--
Mark Fasheh

Re: send/receive and bedup

[Konstantinos Skarlatos]

On 21/5/2014 1:37 πμ, Mark Fasheh wrote:
> On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote:
>>> Duperemove will be shipping as supported software in a major SUSE release so
>>> it will be bug fixed, etc as you would expect. At the moment I'm very busy
>>> trying to fix qgroup bugs so I haven't had much time to add features, or
>>> handle external bug reports, etc. Also I'm not very good at advertising my
>>> software which would be why it hasn't really been mentioned on list lately
>>> :)
>>>
>>> I would say that state that it's in is that I've gotten the feature set to a
>>> point which feels reasonable, and I've fixed enough bugs that I'd appreciate
>>> folks giving it a spin and providing reasonable feedback.
>> Well, after having good results with duperemove with a few gigs of data, i
>> tried it on a 500gb subvolume. After it scanned all files, it is stuck at
>> 100% of one cpu core for about 5 hours, and still hasn't done any deduping.
>> My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats
>> not the problem. So I guess the speed of duperemove drops dramatically as
>> data volume increases.
> Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data
> sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly
> have figured out. It will need a bit of work before it can handle entire
> file systems. My guess is that it was spending an enormous amount of time
> finding duplicates (it has a very thorough check that could probably be
> optimized).
It finished after 9 or so hours, so I agree it was checking for 
duplicates. It does a few GB in just seconds, so time probably scales 
exponentially with data size.
>
> For what it's worth, handling larger data sets is the type of work I want to
> be doing on it in the future.
I can help with testing :)
I would also suggest that you publish in this list any changes that you 
do, so that your program becomes better known among btrfs users. Or even 
a new announcement mail or a page in the btrfs wiki.

Finally, i would like to request the ability to do file level dedup, 
with a reflink. That has the advantage of consuming very little metadata 
compared to block level dedup. It could be done with a two pass dedup, 
first comparing all the same-sized files and after that doing your 
normal block level dedup.

Btw does anybody have a good program/script that can do file level dedup 
with reflinks and checksum comparison?

Kind regards,
Konstantinos Skarlatos
> 	--Mark
>
> --
> Mark Fasheh

Re: send/receive and bedup

[Chris Murphy]

On May 20, 2014, at 4:56 PM, Konstantinos Skarlatos <k.skarlatos@gmail.com> wrote:

> On 21/5/2014 1:37 πμ, Mark Fasheh wrote:
>> On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote:
>>>> Duperemove will be shipping as supported software in a major SUSE release so
>>>> it will be bug fixed, etc as you would expect. At the moment I'm very busy
>>>> trying to fix qgroup bugs so I haven't had much time to add features, or
>>>> handle external bug reports, etc. Also I'm not very good at advertising my
>>>> software which would be why it hasn't really been mentioned on list lately
>>>> :)
>>>> 
>>>> I would say that state that it's in is that I've gotten the feature set to a
>>>> point which feels reasonable, and I've fixed enough bugs that I'd appreciate
>>>> folks giving it a spin and providing reasonable feedback.
>>> Well, after having good results with duperemove with a few gigs of data, i
>>> tried it on a 500gb subvolume. After it scanned all files, it is stuck at
>>> 100% of one cpu core for about 5 hours, and still hasn't done any deduping.
>>> My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats
>>> not the problem. So I guess the speed of duperemove drops dramatically as
>>> data volume increases.
>> Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data
>> sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly
>> have figured out. It will need a bit of work before it can handle entire
>> file systems. My guess is that it was spending an enormous amount of time
>> finding duplicates (it has a very thorough check that could probably be
>> optimized).
> It finished after 9 or so hours, so I agree it was checking for duplicates. It does a few GB in just seconds, so time probably scales exponentially with data size.

I'm going to guess it ran out of memory. I wonder what happens if you take an SSD and specify a humongous swap partition on it. Like, 4x, or more, the amount of installed memory.

This same trick has been mentioned on the XFS list for use with xfsrepair when memory requirements exceed system memory, and is immensely faster.


Chris Murphy

Re: historical backups with hardlinks vs cp --reflink vs snapshots

[Marc MERLIN]

On Mon, May 19, 2014 at 06:01:25PM +0200, Brendan Hide wrote:
> On 19/05/14 15:00, Scott Middleton wrote:
> >On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote:
> >>On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote:
> >>>I read so much about BtrFS that I mistaked Bedup with Duperemove.
> >>>Duperemove is actually what I am testing.
> >>I'm currently using programs that find files that are the same, and
> >>hardlink them together:
> >>http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html
> >>
> >>hardlink.py actually seems to be the faster (memory and CPU) one event
> >>though it's in python.
> >>I can get others to run out of RAM on my 8GB server easily :(
> 
> Interesting app.
> 
> An issue with hardlinking (with the backups use-case, this problem isn't likely to happen), is that if you modify a file, all the hardlinks get changed along with it - including the ones that you don't want changed.
> 
> @Marc: Since you've been using btrfs for a while now I'm sure you've already considered whether or not a reflink copy is the better/worse option.

Yes, I have indeed considered it :)

I just wrote a blog post about the 3 way of doing historical snapshots:
http://marc.merlins.org/perso/btrfs/post_2014-05-20_Historical-Snapshots-With-Btrfs.html
 
I love reflink, but that forces me to use btrfs send as the only way to
copy a filesystem without losing the reflink relationship, and I have no
good way from user space to see the blocks shared to see how many are
shared or whether some just got duped in a copy.
As a result, for now I still use hardlinks.

Once bedup is a bit more ready, I may switch.

That said, duperemove is another dedup I wasn't aware of and I should
look at indeed:
https://github.com/markfasheh/duperemove/blob/master/README

Does it basically do the same work then bedup and tell btrfs to
consolidate blocks it indentified as dupes?
Does it work across subvolumes?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: historical backups with hardlinks vs cp --reflink vs snapshots

[Russell Coker]

On Tue, 20 May 2014 20:59:28 Marc MERLIN wrote:
>  just wrote a blog post about the 3 way of doing historical snapshots:
> http://marc.merlins.org/perso/btrfs/post_2014-05-20_Historical-Snapshots-Wit
> h-Btrfs.html 
> I love reflink, but that forces me to use btrfs send as the only way to
> copy a filesystem without losing the reflink relationship, and I have no
> good way from user space to see the blocks shared to see how many are
> shared or whether some just got duped in a copy.
> As a result, for now I still use hardlinks.

It would be nice if someone patched rsync to look for files with identical 
contents and use reflink or hardlinks (optionally at user request) instead of 
making multiple copies of the same data.  Also it would be nice if rsync would 
look for matching blocks in different files to save transfer.

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: send/receive and bedup

[Konstantinos Skarlatos]

On 21/5/2014 3:58 πμ, Chris Murphy wrote:
> On May 20, 2014, at 4:56 PM, Konstantinos Skarlatos <k.skarlatos@gmail.com> wrote:
>
>> On 21/5/2014 1:37 πμ, Mark Fasheh wrote:
>>> On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote:
>>>>> Duperemove will be shipping as supported software in a major SUSE release so
>>>>> it will be bug fixed, etc as you would expect. At the moment I'm very busy
>>>>> trying to fix qgroup bugs so I haven't had much time to add features, or
>>>>> handle external bug reports, etc. Also I'm not very good at advertising my
>>>>> software which would be why it hasn't really been mentioned on list lately
>>>>> :)
>>>>>
>>>>> I would say that state that it's in is that I've gotten the feature set to a
>>>>> point which feels reasonable, and I've fixed enough bugs that I'd appreciate
>>>>> folks giving it a spin and providing reasonable feedback.
>>>> Well, after having good results with duperemove with a few gigs of data, i
>>>> tried it on a 500gb subvolume. After it scanned all files, it is stuck at
>>>> 100% of one cpu core for about 5 hours, and still hasn't done any deduping.
>>>> My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats
>>>> not the problem. So I guess the speed of duperemove drops dramatically as
>>>> data volume increases.
>>> Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data
>>> sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly
>>> have figured out. It will need a bit of work before it can handle entire
>>> file systems. My guess is that it was spending an enormous amount of time
>>> finding duplicates (it has a very thorough check that could probably be
>>> optimized).
>> It finished after 9 or so hours, so I agree it was checking for duplicates. It does a few GB in just seconds, so time probably scales exponentially with data size.
> I'm going to guess it ran out of memory. I wonder what happens if you take an SSD and specify a humongous swap partition on it. Like, 4x, or more, the amount of installed memory.
Just tried it again, with 32GiB swap added on an SSD. My test files are 
633GiB.
duperemove -rv /storage/test 19537.67s user 183.86s system 89% cpu 
6:06:56.96 total

Duperemove was using about 1GiB or RAM, had one core at 100%, and I 
think swap was not touched at all.


>
> This same trick has been mentioned on the XFS list for use with xfsrepair when memory requirements exceed system memory, and is immensely faster.
>
>
> Chris Murphy
>

Re: send/receive and bedup

[Chris Murphy]

On May 23, 2014, at 9:48 AM, Konstantinos Skarlatos <k.skarlatos@gmail.com> wrote:

> On 21/5/2014 3:58 πμ, Chris Murphy wrote:
>> On May 20, 2014, at 4:56 PM, Konstantinos Skarlatos <k.skarlatos@gmail.com> wrote:
>> 
>>> On 21/5/2014 1:37 πμ, Mark Fasheh wrote:
>>>> On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote:
>>>>>> Duperemove will be shipping as supported software in a major SUSE release so
>>>>>> it will be bug fixed, etc as you would expect. At the moment I'm very busy
>>>>>> trying to fix qgroup bugs so I haven't had much time to add features, or
>>>>>> handle external bug reports, etc. Also I'm not very good at advertising my
>>>>>> software which would be why it hasn't really been mentioned on list lately
>>>>>> :)
>>>>>> 
>>>>>> I would say that state that it's in is that I've gotten the feature set to a
>>>>>> point which feels reasonable, and I've fixed enough bugs that I'd appreciate
>>>>>> folks giving it a spin and providing reasonable feedback.
>>>>> Well, after having good results with duperemove with a few gigs of data, i
>>>>> tried it on a 500gb subvolume. After it scanned all files, it is stuck at
>>>>> 100% of one cpu core for about 5 hours, and still hasn't done any deduping.
>>>>> My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats
>>>>> not the problem. So I guess the speed of duperemove drops dramatically as
>>>>> data volume increases.
>>>> Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data
>>>> sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly
>>>> have figured out. It will need a bit of work before it can handle entire
>>>> file systems. My guess is that it was spending an enormous amount of time
>>>> finding duplicates (it has a very thorough check that could probably be
>>>> optimized).
>>> It finished after 9 or so hours, so I agree it was checking for duplicates. It does a few GB in just seconds, so time probably scales exponentially with data size.
>> I'm going to guess it ran out of memory. I wonder what happens if you take an SSD and specify a humongous swap partition on it. Like, 4x, or more, the amount of installed memory.
> Just tried it again, with 32GiB swap added on an SSD. My test files are 633GiB.
> duperemove -rv /storage/test 19537.67s user 183.86s system 89% cpu 6:06:56.96 total
> 
> Duperemove was using about 1GiB or RAM, had one core at 100%, and I think swap was not touched at all.

Guess currently it's not as memory intensive as it is cpu intensive while also not threading.


Chris Murphy