* send/receive and bedup @ 2014-05-12 12:27 Scott Middleton 2014-05-14 13:20 ` Duncan 0 siblings, 1 reply; 20+ messages in thread From: Scott Middleton @ 2014-05-12 12:27 UTC (permalink / raw) To: linux-btrfs Hi Everyone History: I just recently discovered BtrFS. Well really only just started reading a lot about it. Starting with blogs by Jim Salters and Marc Merlin. So, thanks for those blogs guys. This also introduced me to ZoL (ZFS). It seemed a bit more stable but one of the features I really wanted was deduplication and needing 20GB RAM for 1TB of deduped data and the fact it is always on - pushed me away. Some of those guys really don't like BtrFS BTW! What I want to be able to do is backup Virtual images (VirtualBox and some older VMware) over ADSL. I hoped a mixture of Dedupe and the send/receive functions of BtrFS might accomplish this. I am in the process of building a test BtrFS system at home now but I hope to put it into production in the next few months. Current server is a Dual Xeon, Intel server board, 32GB RAM, 2 x 2TB Hardware RAID SAS (consisting of 4 x 2TB SATA drives) as / and /home in ext4 format. I also have 2 unused 4TB SATA drives that will be eventually be BtrFS RAID1 as /Backup. Ubuntu 14.04. Its is mainly a VM host, small file storage for word docs etc, some large archival .pst files and shadowprotect backups of the Terminal Server. I only just built this server over the last weekend to replace their other aging server and I purposely over engineered it. Onsite there is also a backup server that is pretty basic with 2 x 4TB HDDs and 8GB RAM. I plan on converting it to BtrFS as well. Currently is Ubuntu 12.04 but I will be upgrading it soon to 14.04. Offsite in a data centre I have an aging 1 RU server that I will be upgrading. It'll probably have 8GB RAM, 1 X 60GB SSD as boot/swap and 2 X 4TB HDD BtrFS in RAID 1. Currently running 32 bit Debian 7.5. It has had many partial hardware and OS upgrades over the years as it originally started as Slink or even Hamm. Time to start again since I need to move to 64bit. What I want to do is backup the / and /home directories on the main server to /Backup BtrFS directory, run bedup then "send" it to the onsite backup server. The onsite backup server will "send" it to the offsite server. I am assuming (correctly I hope) that the deduplication will also be replicated across the machines. I'll have NOCOW on the VM images, Archived PST files, Shadow protect images and some other stuff. I guess the first question is this even possible? I don't believe that much actual non duplicated data changes all that much mostly just word docs that I already send offsite. I'm really hoping to backup the VMs and the shadow protects offsite as well. I can upgrade the broadband to fibre but before I do that (spend a lot of money) I want to be able to see that it would be possible. Kind Regards - Scott ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-12 12:27 send/receive and bedup Scott Middleton @ 2014-05-14 13:20 ` Duncan 2014-05-14 15:36 ` Scott Middleton 0 siblings, 1 reply; 20+ messages in thread From: Duncan @ 2014-05-14 13:20 UTC (permalink / raw) To: linux-btrfs Scott Middleton posted on Mon, 12 May 2014 20:27:13 +0800 as excerpted: > Hi Everyone > > History: > I just recently discovered BtrFS. Well really only just started reading > a lot about it. Starting with blogs by Jim Salters and Marc Merlin. So, > thanks for those blogs guys. > > This also introduced me to ZoL (ZFS). It seemed a bit more stable but > one of the features I really wanted was deduplication and needing 20GB > RAM for 1TB of deduped data and the fact it is always on - pushed me > away. Some of those guys really don't like BtrFS BTW! > > What I want to be able to do is backup Virtual images (VirtualBox and > some older VMware) over ADSL. I hoped a mixture of Dedupe and the > send/receive functions of BtrFS might accomplish this. > > I am in the process of building a test BtrFS system at home now but I > hope to put it into production in the next few months. > > > Current server is a Dual Xeon, Intel server board, 32GB RAM, 2 x 2TB > Hardware RAID SAS (consisting of 4 x 2TB SATA drives) as / and /home in > ext4 format. I also have 2 unused 4TB SATA drives that will be > eventually be BtrFS RAID1 as /Backup. Ubuntu 14.04. Its is mainly a VM > host, small file storage for word docs etc, some large archival .pst > files and shadowprotect backups of the Terminal Server. I only just > built this server over the last weekend to replace their other aging > server and I purposely over engineered it. > > Onsite there is also a backup server that is pretty basic with 2 x 4TB > HDDs and 8GB RAM. I plan on converting it to BtrFS as well. Currently is > Ubuntu 12.04 but I will be upgrading it soon to 14.04. > > Offsite in a data centre I have an aging 1 RU server that I will be > upgrading. It'll probably have 8GB RAM, 1 X 60GB SSD as boot/swap and 2 > X 4TB HDD BtrFS in RAID 1. Currently running 32 bit Debian 7.5. It has > had many partial hardware and OS upgrades over the years as it > originally started as Slink or even Hamm. Time to start again since I > need to move to 64bit. > > What I want to do is backup the / and /home directories on the main > server to /Backup BtrFS directory, run bedup then "send" it to the > onsite backup server. The onsite backup server will "send" it to the > offsite server. I am assuming (correctly I hope) that the deduplication > will also be replicated across the machines. I'll have NOCOW on the VM > images, Archived PST files, Shadow protect images and some other stuff. > > I guess the first question is this even possible? I don't believe that > much actual non duplicated data changes all that much mostly just word > docs that I already send offsite. I'm really hoping to backup the VMs > and the shadow protects offsite as well. I can upgrade the broadband to > fibre but before I do that (spend a lot of money) I want to be able to > see that it would be possible. I left this for a couple days hoping someone else with a more directly similar use-case would answer, but none so far, so I'll give it a go... First some general boilerplate. Btrfs is still under heavy development and keeping current with especially the kernel is *STRONGLY* recommended, as every new kernel still brings lots of fixes, meaning if you're running an old kernel, you're running known-buggy code with fixes available in a current kernel. Similarly, you probably don't want to let the btrfs-progs userspace tools get too outdated either, tho that's not as critical as it mostly means not being able to take advantage of the latest features and fixes for maintenance, not the risk of operational data loss if one of the known-fixed old-version kernel bugs hits that you have when running an older kernel. Second, as you've done some research already you're likely aware of this, but just in case, let me give you the link to the wiki. If you haven't read up there, please do, as it's likely to be quite helpful. =:^) Memory or bookmark: https://btrfs.wiki.kernel.org User documentation bookmark: https://btrfs.wiki.kernel.org/index.php/Main_Page#Guides_and_usage_information On to your proposed setup. In general, it looks reasonable. My first concern upon reading about the VM images was of course the fragmentation issues that come with the VM images territory on btrfs, but if you keep your working partitions as ext4 and use btrfs primarily for backup, that issue goes away to a large extent, since the operational rewriting will be happening on the ext4, which should handle it a bit better than btrfs does at this point, while btrfs will only be getting the more sequentially written backups and not have to deal with the live in-place updates on the VM images. The problem with btrfs hosting operational VM images is snapshots, particularly when using btrfs send for backup and/or when doing frequent, often scripted, snapshotting. The problem with send is that it takes a read-only snapshot and sends from that, so it's a snapshotting issue either way. The problem with snapshots is that for NOCOW files, the first write to a block after a snapshot still triggers a COW write, since the snapshot locked the existing version in place. So frequent snapshotting has the effect of nullifying NOCOW for big VM images in operation due to the operational internal-write pattern, thus being a big problem that the devs are still working on finding a reasonably satisfactory solution for. But by keeping your operational VM images on ext4 and only using btrfs for originally local backups, then using btrfs send to replicate those backups to onsite and then offsite remote, you should be avoiding the biggest problem, since the backup write-pattern should be far more sequential and not trigger the fragmentation issue you'd have trying to host the operational VMs directly on btrfs. My next concern is with bedup and its interaction with btrfs send and snapshots. I simply don't have enough knowledge in that area to be able to comment intelligently on how all the pieces fit together there, but you'll need to ensure a couple things, for sure. Btrfs send starts with a full send, then uses that original base, kept the same on both sides, as a reference for the incremental sends you do later. Both the original backup ext4->btrfs and the bedup processes will need to be setup so as not to interfere with the efficiency of those incremental sends, or if it works at all, you'll be effectively resending the entire thing each time, which isn't what you want at all. Unfortunately I simply don't know enough about the interaction between the pieces to say whether your plan is reasonable there or not, and if so, how to actually do it. This is why I was hoping someone else with more direct experience/knowledge would reply, but... Next is bedup itself. I'm not sure of the maturity status there, tho I don't believe it's fully production-quality yet. However, it's quite possible that to the extent that it isn't production-ready, any breakage simply reduces the efficiency, while still helping some. You'd really need to contact the guy working on it to find out. Finally, btrfs send/receive itself just recently had a whole host of bugs fixed, so again, you'll want a very recent kernel and userspace in ordered to get those bugfixes. In fact, I believe some of those fixes are only in kernel 3.15-rc, and only in the btrfs-progs integration branch, not yet in stable at all. You can of course try 3.14.x stable kernel and 3.14.1 btrfs-progs and hope it works for now, updating if you have problems, or go for the pre-releases from the get-go. Meanwhile, while that round of fixes certainly means btrfs send/receive is more mature than it was, I'd still strongly recommend having a fallback plan, in case you start getting errors and it quits working for you, pending further fixes. IOW, yes, I'd say use send/receive if it works for you, but at this point, don't count on it actually continuing to work every time, and have a fallback if it breaks temporarily, so you're not as they say left up a creek without a paddle. Hope that helps and good luck. Looking forward to seeing more posts as you experiment, and hopefully you'll then stick around to add your experiences to the wiki and answer questions about that use-case here as others may have them. =:^) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-14 13:20 ` Duncan @ 2014-05-14 15:36 ` Scott Middleton 2014-05-19 1:07 ` Marc MERLIN 0 siblings, 1 reply; 20+ messages in thread From: Scott Middleton @ 2014-05-14 15:36 UTC (permalink / raw) Cc: linux-btrfs Hi > I left this for a couple days hoping someone else with a more directly > similar use-case would answer, but none so far, so I'll give it a go... Thanks for getting back to me mate! > > First some general boilerplate. Btrfs is still under heavy development > and keeping current with especially the kernel is *STRONGLY* recommended, > as every new kernel still brings lots of fixes, meaning if you're running > an old kernel, you're running known-buggy code with fixes available in a > current kernel. Similarly, you probably don't want to let the btrfs-progs > userspace tools get too outdated either, tho that's not as critical as it > mostly means not being able to take advantage of the latest features and > fixes for maintenance, not the risk of operational data loss if one of > the known-fixed old-version kernel bugs hits that you have when running > an older kernel. > My test environment is: root@Ubuntu-14:~/btrfs/btrfs-progs# uname -a Linux Ubuntu-14 3.14.1-031401-generic #201404141220 SMP Mon Apr 14 16:21:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux root@Ubuntu-14:~/btrfs/btrfs-progs# which btrfs /usr/local/bin/btrfs root@Ubuntu-14:~/btrfs/btrfs-progs# btrfs --version Btrfs v3.14.1 I read so much about BtrFS that I mistaked Bedup with Duperemove. Duperemove is actually what I am testing. I also plan on testing SDFS - opendedup again! They have some development since the last time I tried, Not a fan of having to use Java though! > Second, as you've done some research already you're likely aware of this, > but just in case, let me give you the link to the wiki. If you haven't > read up there, please do, as it's likely to be quite helpful. =:^) > > Memory or bookmark: https://btrfs.wiki.kernel.org Lots of bookmarks. One of my faves is: https://wiki.archlinux.org/index.php/Btrfs It will be interesting on what happens. I rsync'd the clients data from a 2TB mdadm RAID to a standard 3TB BtrFS drive, Currently running Duperemove on it. I reckon it will take days but it will be interesting to see what happens. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-14 15:36 ` Scott Middleton @ 2014-05-19 1:07 ` Marc MERLIN 2014-05-19 13:00 ` Scott Middleton 0 siblings, 1 reply; 20+ messages in thread From: Marc MERLIN @ 2014-05-19 1:07 UTC (permalink / raw) To: Scott Middleton; +Cc: linux-btrfs On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote: > I read so much about BtrFS that I mistaked Bedup with Duperemove. > Duperemove is actually what I am testing. I'm currently using programs that find files that are the same, and hardlink them together: http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html hardlink.py actually seems to be the faster (memory and CPU) one event though it's in python. I can get others to run out of RAM on my 8GB server easily :( Bedup should be better, but last I tried I couldn't get it to work. It's been updated since then, I just haven't had the chance to try it again since then. Please post what you find out, or if you have a hardlink maker that's better than the ones I found :) Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-19 1:07 ` Marc MERLIN @ 2014-05-19 13:00 ` Scott Middleton 2014-05-19 16:01 ` Brendan Hide 0 siblings, 1 reply; 20+ messages in thread From: Scott Middleton @ 2014-05-19 13:00 UTC (permalink / raw) Cc: linux-btrfs On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote: > On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote: >> I read so much about BtrFS that I mistaked Bedup with Duperemove. >> Duperemove is actually what I am testing. > > I'm currently using programs that find files that are the same, and > hardlink them together: > http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html > > hardlink.py actually seems to be the faster (memory and CPU) one event > though it's in python. > I can get others to run out of RAM on my 8GB server easily :( > > Bedup should be better, but last I tried I couldn't get it to work. > It's been updated since then, I just haven't had the chance to try it > again since then. > > Please post what you find out, or if you have a hardlink maker that's > better than the ones I found :) > Thanks for that. I may be completely wrong in my approach. I am not looking for a file level comparison. Bedup worked fine for that. I have a lot of virtual images and shadow protect images where only a few megabytes may be the difference. So a file level hash and comparison doesn't really achieve my goals. I thought duperemove may be on a lower level. https://github.com/markfasheh/duperemove "Duperemove is a simple tool for finding duplicated extents and submitting them for deduplication. When given a list of files it will hash their contents on a block by block basis and compare those hashes to each other, finding and categorizing extents that match each other. When given the -d option, duperemove will submit those extents for deduplication using the btrfs-extent-same ioctl." It defaults to 128k but you can make it smaller. I hit a hurdle though. The 3TB HDD I used seemed OK when I did a long SMART test but seems to die every few hours. Admittedly it was part of a failed mdadm RAID array that I pulled out of a clients machine. The only other copy I have of the data is the original mdadm array that was recently replaced with a new server, so I am loathe to use that HDD yet. At least for another couple of weeks! I am still hopeful duperemove will work. In another month I will put the 2 X 4TB HDDs online in BtrFS RAID 1 format on the production machine and have a crack on duperemove on that after hours. I will convert the onsite backup machine to BtrFS with its 2 x 4TB HDDs to BtrFS not long after. The ultimate goal is to be able to back up on a block level very large files offsite where maybe a GB is changed on a daily basis. I realise that I will have to make an original copy and manually take that to my datacentre but hopefully I can backup multiple clients data after hours, or possibly, a trickle, constantly. Kind Regards Scott ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-19 13:00 ` Scott Middleton @ 2014-05-19 16:01 ` Brendan Hide 2014-05-19 17:12 ` Konstantinos Skarlatos ` (2 more replies) 0 siblings, 3 replies; 20+ messages in thread From: Brendan Hide @ 2014-05-19 16:01 UTC (permalink / raw) To: Scott Middleton; +Cc: linux-btrfs, Mark Fasheh On 19/05/14 15:00, Scott Middleton wrote: > On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote: >> On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote: >>> I read so much about BtrFS that I mistaked Bedup with Duperemove. >>> Duperemove is actually what I am testing. >> I'm currently using programs that find files that are the same, and >> hardlink them together: >> http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html >> >> hardlink.py actually seems to be the faster (memory and CPU) one event >> though it's in python. >> I can get others to run out of RAM on my 8GB server easily :( Interesting app. An issue with hardlinking (with the backups use-case, this problem isn't likely to happen), is that if you modify a file, all the hardlinks get changed along with it - including the ones that you don't want changed. @Marc: Since you've been using btrfs for a while now I'm sure you've already considered whether or not a reflink copy is the better/worse option. >> >> Bedup should be better, but last I tried I couldn't get it to work. >> It's been updated since then, I just haven't had the chance to try it >> again since then. >> >> Please post what you find out, or if you have a hardlink maker that's >> better than the ones I found :) >> > > Thanks for that. > > I may be completely wrong in my approach. > > I am not looking for a file level comparison. Bedup worked fine for > that. I have a lot of virtual images and shadow protect images where > only a few megabytes may be the difference. So a file level hash and > comparison doesn't really achieve my goals. > > I thought duperemove may be on a lower level. > > https://github.com/markfasheh/duperemove > > "Duperemove is a simple tool for finding duplicated extents and > submitting them for deduplication. When given a list of files it will > hash their contents on a block by block basis and compare those hashes > to each other, finding and categorizing extents that match each > other. When given the -d option, duperemove will submit those > extents for deduplication using the btrfs-extent-same ioctl." > > It defaults to 128k but you can make it smaller. > > I hit a hurdle though. The 3TB HDD I used seemed OK when I did a long > SMART test but seems to die every few hours. Admittedly it was part of > a failed mdadm RAID array that I pulled out of a clients machine. > > The only other copy I have of the data is the original mdadm array > that was recently replaced with a new server, so I am loathe to use > that HDD yet. At least for another couple of weeks! > > > I am still hopeful duperemove will work. Duperemove does look exactly like what you are looking for. The last traffic on the mailing list regarding that was in August last year. It looks like it was pulled into the main kernel repository on September 1st. The last commit to the duperemove application was on April 20th this year. Maybe Mark (cc'd) can provide further insight on its current status. -- __________ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-19 16:01 ` Brendan Hide @ 2014-05-19 17:12 ` Konstantinos Skarlatos 2014-05-19 17:55 ` Mark Fasheh 2014-05-19 17:59 ` Austin S Hemmelgarn 2014-05-19 17:38 ` Mark Fasheh 2014-05-21 3:59 ` historical backups with hardlinks vs cp --reflink vs snapshots Marc MERLIN 2 siblings, 2 replies; 20+ messages in thread From: Konstantinos Skarlatos @ 2014-05-19 17:12 UTC (permalink / raw) To: Brendan Hide, Scott Middleton; +Cc: linux-btrfs, Mark Fasheh On 19/5/2014 7:01 μμ, Brendan Hide wrote: > On 19/05/14 15:00, Scott Middleton wrote: >> On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote: >>> On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote: >>>> I read so much about BtrFS that I mistaked Bedup with Duperemove. >>>> Duperemove is actually what I am testing. >>> I'm currently using programs that find files that are the same, and >>> hardlink them together: >>> http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html >>> >>> >>> hardlink.py actually seems to be the faster (memory and CPU) one event >>> though it's in python. >>> I can get others to run out of RAM on my 8GB server easily :( > > Interesting app. > > An issue with hardlinking (with the backups use-case, this problem > isn't likely to happen), is that if you modify a file, all the > hardlinks get changed along with it - including the ones that you > don't want changed. > > @Marc: Since you've been using btrfs for a while now I'm sure you've > already considered whether or not a reflink copy is the better/worse > option. > >>> >>> Bedup should be better, but last I tried I couldn't get it to work. >>> It's been updated since then, I just haven't had the chance to try it >>> again since then. >>> >>> Please post what you find out, or if you have a hardlink maker that's >>> better than the ones I found :) >>> >> >> Thanks for that. >> >> I may be completely wrong in my approach. >> >> I am not looking for a file level comparison. Bedup worked fine for >> that. I have a lot of virtual images and shadow protect images where >> only a few megabytes may be the difference. So a file level hash and >> comparison doesn't really achieve my goals. >> >> I thought duperemove may be on a lower level. >> >> https://github.com/markfasheh/duperemove >> >> "Duperemove is a simple tool for finding duplicated extents and >> submitting them for deduplication. When given a list of files it will >> hash their contents on a block by block basis and compare those hashes >> to each other, finding and categorizing extents that match each >> other. When given the -d option, duperemove will submit those >> extents for deduplication using the btrfs-extent-same ioctl." >> >> It defaults to 128k but you can make it smaller. >> >> I hit a hurdle though. The 3TB HDD I used seemed OK when I did a long >> SMART test but seems to die every few hours. Admittedly it was part of >> a failed mdadm RAID array that I pulled out of a clients machine. >> >> The only other copy I have of the data is the original mdadm array >> that was recently replaced with a new server, so I am loathe to use >> that HDD yet. At least for another couple of weeks! >> >> >> I am still hopeful duperemove will work. > Duperemove does look exactly like what you are looking for. The last > traffic on the mailing list regarding that was in August last year. It > looks like it was pulled into the main kernel repository on September > 1st. > > The last commit to the duperemove application was on April 20th this > year. Maybe Mark (cc'd) can provide further insight on its current > status. > I have been testing duperemove and it seems to work just fine, in contrast with bedup that i have been unable to install/compile/sort out the mess with python versions. I have 2 questions about duperemove: 1) can it use existing filesystem csums instead of calculating its own? 2) can it be included in btrfs-progs so that it becomes a standard feature of btrfs? Thanks ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-19 17:12 ` Konstantinos Skarlatos @ 2014-05-19 17:55 ` Mark Fasheh 2014-05-19 17:59 ` Austin S Hemmelgarn 1 sibling, 0 replies; 20+ messages in thread From: Mark Fasheh @ 2014-05-19 17:55 UTC (permalink / raw) To: Konstantinos Skarlatos; +Cc: Brendan Hide, Scott Middleton, linux-btrfs On Mon, May 19, 2014 at 08:12:03PM +0300, Konstantinos Skarlatos wrote: > On 19/5/2014 7:01 μμ, Brendan Hide wrote: >> On 19/05/14 15:00, Scott Middleton wrote: >> Duperemove does look exactly like what you are looking for. The last >> traffic on the mailing list regarding that was in August last year. It >> looks like it was pulled into the main kernel repository on September 1st. >> >> The last commit to the duperemove application was on April 20th this year. >> Maybe Mark (cc'd) can provide further insight on its current status. >> > I have been testing duperemove and it seems to work just fine, in contrast > with bedup that i have been unable to install/compile/sort out the mess > with python versions. I have 2 questions about duperemove: > 1) can it use existing filesystem csums instead of calculating its own? Not right now, though that may be something we can feed to it in the future. I haven't thought about this much and to be honest I don't recall *exactly* how btrfs stores it's checksums. That said, I think feasibility of doing this comes down to a few things: 1) how expensive is it to get at the on-disk checksums? This might not make sense if it's simply faster to scan a file than its checksums. 2) are they stored in a manner which makes sense for dedupe. By that I mean, do we have a checksum for every X bytes? If so, then theoretically life is easy - we just make our blocksize to X and load the checksums into duperemoves internal block checksum tree. If checksums can cover arbitrary sized extents than we might not be able to use them at all or maybe we would have to 'fill in the blanks' so to speak. 3) what is the tradeoff of false positives? Btrfs checksums are there for detecting bad blocks, as opposed to duplicate data. The difference is that btrfs doesn't have to use very strong hashing as a result. So we just want to make sure that we don't wind up passing *so* many false positives to the kernel that it was just faster to scan the file and checksum on our own. Not that any of those questions are super difficult to answer by the way, it's more about how much time I've had :) > 2) can it be included in btrfs-progs so that it becomes a standard feature > of btrfs? I have to think about this one personally as it implies some tradeoffs in my development on duperemove that I'm not sure I want to make yet. --Mark -- Mark Fasheh ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-19 17:12 ` Konstantinos Skarlatos 2014-05-19 17:55 ` Mark Fasheh @ 2014-05-19 17:59 ` Austin S Hemmelgarn 2014-05-19 18:27 ` Mark Fasheh 1 sibling, 1 reply; 20+ messages in thread From: Austin S Hemmelgarn @ 2014-05-19 17:59 UTC (permalink / raw) To: Konstantinos Skarlatos, Brendan Hide, Scott Middleton Cc: linux-btrfs, Mark Fasheh [-- Attachment #1: Type: text/plain, Size: 4806 bytes --] On 2014-05-19 13:12, Konstantinos Skarlatos wrote: > On 19/5/2014 7:01 μμ, Brendan Hide wrote: >> On 19/05/14 15:00, Scott Middleton wrote: >>> On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote: >>>> On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote: >>>>> I read so much about BtrFS that I mistaked Bedup with Duperemove. >>>>> Duperemove is actually what I am testing. >>>> I'm currently using programs that find files that are the same, and >>>> hardlink them together: >>>> http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html >>>> >>>> >>>> hardlink.py actually seems to be the faster (memory and CPU) one event >>>> though it's in python. >>>> I can get others to run out of RAM on my 8GB server easily :( >> >> Interesting app. >> >> An issue with hardlinking (with the backups use-case, this problem >> isn't likely to happen), is that if you modify a file, all the >> hardlinks get changed along with it - including the ones that you >> don't want changed. >> >> @Marc: Since you've been using btrfs for a while now I'm sure you've >> already considered whether or not a reflink copy is the better/worse >> option. >> >>>> >>>> Bedup should be better, but last I tried I couldn't get it to work. >>>> It's been updated since then, I just haven't had the chance to try it >>>> again since then. >>>> >>>> Please post what you find out, or if you have a hardlink maker that's >>>> better than the ones I found :) >>>> >>> >>> Thanks for that. >>> >>> I may be completely wrong in my approach. >>> >>> I am not looking for a file level comparison. Bedup worked fine for >>> that. I have a lot of virtual images and shadow protect images where >>> only a few megabytes may be the difference. So a file level hash and >>> comparison doesn't really achieve my goals. >>> >>> I thought duperemove may be on a lower level. >>> >>> https://github.com/markfasheh/duperemove >>> >>> "Duperemove is a simple tool for finding duplicated extents and >>> submitting them for deduplication. When given a list of files it will >>> hash their contents on a block by block basis and compare those hashes >>> to each other, finding and categorizing extents that match each >>> other. When given the -d option, duperemove will submit those >>> extents for deduplication using the btrfs-extent-same ioctl." >>> >>> It defaults to 128k but you can make it smaller. >>> >>> I hit a hurdle though. The 3TB HDD I used seemed OK when I did a long >>> SMART test but seems to die every few hours. Admittedly it was part of >>> a failed mdadm RAID array that I pulled out of a clients machine. >>> >>> The only other copy I have of the data is the original mdadm array >>> that was recently replaced with a new server, so I am loathe to use >>> that HDD yet. At least for another couple of weeks! >>> >>> >>> I am still hopeful duperemove will work. >> Duperemove does look exactly like what you are looking for. The last >> traffic on the mailing list regarding that was in August last year. It >> looks like it was pulled into the main kernel repository on September >> 1st. >> >> The last commit to the duperemove application was on April 20th this >> year. Maybe Mark (cc'd) can provide further insight on its current >> status. >> > I have been testing duperemove and it seems to work just fine, in > contrast with bedup that i have been unable to install/compile/sort out > the mess with python versions. I have 2 questions about duperemove: > 1) can it use existing filesystem csums instead of calculating its own? While this might seem like a great idea at first, it really isn't. BTRFS uses CRC32c at the moment as it's checksum algorithm, and while that is relatively good at detecting small differences (i.e. a single bit flipped out of every 64 or so bytes), it is known to have issues with hash collisions. Normally, the data on disk won't change enough even from a media error to cause a hash collision, but when you start using it to compare extents that aren't known to be the same to begin with, and then try to merge those extents, you run the risk of serious file corruption. Also, AFAIK, BTRFS doesn't expose the block checksum to userspace directly (although I may be wrong about this, in which case i retract the following statement) this would therefore require some kernelspace support. > 2) can it be included in btrfs-progs so that it becomes a standard > feature of btrfs? I would definitely like to second this suggestion, I hear a lot of people talking about how BTRFS has batch deduplication, but it's almost impossible to make use of without extra software or writing your own code. [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 2967 bytes --] ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-19 17:59 ` Austin S Hemmelgarn @ 2014-05-19 18:27 ` Mark Fasheh 0 siblings, 0 replies; 20+ messages in thread From: Mark Fasheh @ 2014-05-19 18:27 UTC (permalink / raw) To: Austin S Hemmelgarn Cc: Konstantinos Skarlatos, Brendan Hide, Scott Middleton, linux-btrfs On Mon, May 19, 2014 at 01:59:01PM -0400, Austin S Hemmelgarn wrote: > On 2014-05-19 13:12, Konstantinos Skarlatos wrote: > > I have been testing duperemove and it seems to work just fine, in > > contrast with bedup that i have been unable to install/compile/sort out > > the mess with python versions. I have 2 questions about duperemove: > > 1) can it use existing filesystem csums instead of calculating its own? > While this might seem like a great idea at first, it really isn't. > BTRFS uses CRC32c at the moment as it's checksum algorithm, and while > that is relatively good at detecting small differences (i.e. a single > bit flipped out of every 64 or so bytes), it is known to have issues > with hash collisions. Normally, the data on disk won't change enough > even from a media error to cause a hash collision, but when you start > using it to compare extents that aren't known to be the same to begin > with, and then try to merge those extents, you run the risk of serious > file corruption. Also, AFAIK, BTRFS doesn't expose the block checksum > to userspace directly (although I may be wrong about this, in which case > i retract the following statement) this would therefore require some > kernelspace support. I'm pretty sure you could get the checkums via ioctl. The thing about dedupe though is that kernel is always doing a byte-by-byte comparison of the file data before merging it so we should never corrupt just because userspace gave us a bad range to dedupe. That said I don't necessarily disagree that it might not be as good an idea as it sounds. --Mark -- Mark Fasheh ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-19 16:01 ` Brendan Hide 2014-05-19 17:12 ` Konstantinos Skarlatos @ 2014-05-19 17:38 ` Mark Fasheh 2014-05-19 22:07 ` Konstantinos Skarlatos 2014-05-21 3:59 ` historical backups with hardlinks vs cp --reflink vs snapshots Marc MERLIN 2 siblings, 1 reply; 20+ messages in thread From: Mark Fasheh @ 2014-05-19 17:38 UTC (permalink / raw) To: Brendan Hide; +Cc: Scott Middleton, linux-btrfs On Mon, May 19, 2014 at 06:01:25PM +0200, Brendan Hide wrote: > On 19/05/14 15:00, Scott Middleton wrote: >> On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote: >> Thanks for that. >> >> I may be completely wrong in my approach. >> >> I am not looking for a file level comparison. Bedup worked fine for >> that. I have a lot of virtual images and shadow protect images where >> only a few megabytes may be the difference. So a file level hash and >> comparison doesn't really achieve my goals. >> >> I thought duperemove may be on a lower level. >> >> https://github.com/markfasheh/duperemove >> >> "Duperemove is a simple tool for finding duplicated extents and >> submitting them for deduplication. When given a list of files it will >> hash their contents on a block by block basis and compare those hashes >> to each other, finding and categorizing extents that match each >> other. When given the -d option, duperemove will submit those >> extents for deduplication using the btrfs-extent-same ioctl." >> >> It defaults to 128k but you can make it smaller. >> >> I hit a hurdle though. The 3TB HDD I used seemed OK when I did a long >> SMART test but seems to die every few hours. Admittedly it was part of >> a failed mdadm RAID array that I pulled out of a clients machine. >> >> The only other copy I have of the data is the original mdadm array >> that was recently replaced with a new server, so I am loathe to use >> that HDD yet. At least for another couple of weeks! >> >> >> I am still hopeful duperemove will work. > Duperemove does look exactly like what you are looking for. The last > traffic on the mailing list regarding that was in August last year. It > looks like it was pulled into the main kernel repository on September 1st. I'm confused - you need to avoid a file scan completely? Duperemove does do that just to be clear. In your mind, what would be the alternative to that sort of a scan? By the way, if you know exactly where the changes are you could just feed the duplicate extents directly to the ioctl via a script. I have a small tool in the duperemove repositry that can do that for you ('make btrfs-extent-same'). > The last commit to the duperemove application was on April 20th this year. > Maybe Mark (cc'd) can provide further insight on its current status. Duperemove will be shipping as supported software in a major SUSE release so it will be bug fixed, etc as you would expect. At the moment I'm very busy trying to fix qgroup bugs so I haven't had much time to add features, or handle external bug reports, etc. Also I'm not very good at advertising my software which would be why it hasn't really been mentioned on list lately :) I would say that state that it's in is that I've gotten the feature set to a point which feels reasonable, and I've fixed enough bugs that I'd appreciate folks giving it a spin and providing reasonable feedback. There's a TODO list which gives a decent idea of what's on my mind for possible future improvements. I think what I'm most wanting to do right now is some sort of (optional) writeout to a file of what was done during a run. The idea is that you could feed that data back to duperemove to improve the speed of subsequent runs. My priorities may change depending on feedback from users of course. I also at some point want to rewrite some of the duplicate extent finding code as it got messy and could be a bit faster. --Mark -- Mark Fasheh ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-19 17:38 ` Mark Fasheh @ 2014-05-19 22:07 ` Konstantinos Skarlatos 2014-05-20 11:12 ` Scott Middleton 2014-05-20 22:37 ` Mark Fasheh 0 siblings, 2 replies; 20+ messages in thread From: Konstantinos Skarlatos @ 2014-05-19 22:07 UTC (permalink / raw) To: Mark Fasheh, Brendan Hide; +Cc: Scott Middleton, linux-btrfs On 19/5/2014 8:38 μμ, Mark Fasheh wrote: > On Mon, May 19, 2014 at 06:01:25PM +0200, Brendan Hide wrote: >> On 19/05/14 15:00, Scott Middleton wrote: >>> On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote: >>> Thanks for that. >>> >>> I may be completely wrong in my approach. >>> >>> I am not looking for a file level comparison. Bedup worked fine for >>> that. I have a lot of virtual images and shadow protect images where >>> only a few megabytes may be the difference. So a file level hash and >>> comparison doesn't really achieve my goals. >>> >>> I thought duperemove may be on a lower level. >>> >>> https://github.com/markfasheh/duperemove >>> >>> "Duperemove is a simple tool for finding duplicated extents and >>> submitting them for deduplication. When given a list of files it will >>> hash their contents on a block by block basis and compare those hashes >>> to each other, finding and categorizing extents that match each >>> other. When given the -d option, duperemove will submit those >>> extents for deduplication using the btrfs-extent-same ioctl." >>> >>> It defaults to 128k but you can make it smaller. >>> >>> I hit a hurdle though. The 3TB HDD I used seemed OK when I did a long >>> SMART test but seems to die every few hours. Admittedly it was part of >>> a failed mdadm RAID array that I pulled out of a clients machine. >>> >>> The only other copy I have of the data is the original mdadm array >>> that was recently replaced with a new server, so I am loathe to use >>> that HDD yet. At least for another couple of weeks! >>> >>> >>> I am still hopeful duperemove will work. >> Duperemove does look exactly like what you are looking for. The last >> traffic on the mailing list regarding that was in August last year. It >> looks like it was pulled into the main kernel repository on September 1st. > I'm confused - you need to avoid a file scan completely? Duperemove does do > that just to be clear. > > In your mind, what would be the alternative to that sort of a scan? > > By the way, if you know exactly where the changes are you > could just feed the duplicate extents directly to the ioctl via a script. I > have a small tool in the duperemove repositry that can do that for you > ('make btrfs-extent-same'). > > >> The last commit to the duperemove application was on April 20th this year. >> Maybe Mark (cc'd) can provide further insight on its current status. > Duperemove will be shipping as supported software in a major SUSE release so > it will be bug fixed, etc as you would expect. At the moment I'm very busy > trying to fix qgroup bugs so I haven't had much time to add features, or > handle external bug reports, etc. Also I'm not very good at advertising my > software which would be why it hasn't really been mentioned on list lately > :) > > I would say that state that it's in is that I've gotten the feature set to a > point which feels reasonable, and I've fixed enough bugs that I'd appreciate > folks giving it a spin and providing reasonable feedback. Well, after having good results with duperemove with a few gigs of data, i tried it on a 500gb subvolume. After it scanned all files, it is stuck at 100% of one cpu core for about 5 hours, and still hasn't done any deduping. My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats not the problem. So I guess the speed of duperemove drops dramatically as data volume increases. > > There's a TODO list which gives a decent idea of what's on my mind for > possible future improvements. I think what I'm most wanting to do right now > is some sort of (optional) writeout to a file of what was done during a run. > The idea is that you could feed that data back to duperemove to improve the > speed of subsequent runs. My priorities may change depending on feedback > from users of course. > > I also at some point want to rewrite some of the duplicate extent finding > code as it got messy and could be a bit faster. > --Mark > > -- > Mark Fasheh > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-19 22:07 ` Konstantinos Skarlatos @ 2014-05-20 11:12 ` Scott Middleton 2014-05-20 22:37 ` Mark Fasheh 1 sibling, 0 replies; 20+ messages in thread From: Scott Middleton @ 2014-05-20 11:12 UTC (permalink / raw) Cc: linux-btrfs On 20 May 2014 06:07, Konstantinos Skarlatos <k.skarlatos@gmail.com> wrote: > On 19/5/2014 8:38 μμ, Mark Fasheh wrote: >> > > > Well, after having good results with duperemove with a few gigs of data, i > tried it on a 500gb subvolume. After it scanned all files, it is stuck at > 100% of one cpu core for about 5 hours, and still hasn't done any deduping. > My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats not > the problem. So I guess the speed of duperemove drops dramatically as data > volume increases. > >> >> There's a TODO list which gives a decent idea of what's on my mind for >> possible future improvements. I think what I'm most wanting to do right >> now >> is some sort of (optional) writeout to a file of what was done during a >> run. >> The idea is that you could feed that data back to duperemove to improve >> the >> speed of subsequent runs. My priorities may change depending on feedback >> from users of course. >> >> I also at some point want to rewrite some of the duplicate extent finding >> code as it got messy and could be a bit faster. >> --Mark I'm glad about this discussion. While I am no where near an expert on file systems, my knowledge has increased a lot through BtrFS. ZFS uses RAM to store its checksum tables. Opendedup recommends a separate HDD. Opendedup uses 4k block sizes. Both are always on. I'm not against using a separate HDD to store csums. Cheaper than RAM, albeit slower. The part of duperemove I like is the ability to CHOOSE when and how I want to dedupe. Scott ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-19 22:07 ` Konstantinos Skarlatos 2014-05-20 11:12 ` Scott Middleton @ 2014-05-20 22:37 ` Mark Fasheh 2014-05-20 22:56 ` Konstantinos Skarlatos 1 sibling, 1 reply; 20+ messages in thread From: Mark Fasheh @ 2014-05-20 22:37 UTC (permalink / raw) To: Konstantinos Skarlatos; +Cc: Brendan Hide, Scott Middleton, linux-btrfs On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote: >> Duperemove will be shipping as supported software in a major SUSE release so >> it will be bug fixed, etc as you would expect. At the moment I'm very busy >> trying to fix qgroup bugs so I haven't had much time to add features, or >> handle external bug reports, etc. Also I'm not very good at advertising my >> software which would be why it hasn't really been mentioned on list lately >> :) >> >> I would say that state that it's in is that I've gotten the feature set to a >> point which feels reasonable, and I've fixed enough bugs that I'd appreciate >> folks giving it a spin and providing reasonable feedback. > Well, after having good results with duperemove with a few gigs of data, i > tried it on a 500gb subvolume. After it scanned all files, it is stuck at > 100% of one cpu core for about 5 hours, and still hasn't done any deduping. > My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats > not the problem. So I guess the speed of duperemove drops dramatically as > data volume increases. Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly have figured out. It will need a bit of work before it can handle entire file systems. My guess is that it was spending an enormous amount of time finding duplicates (it has a very thorough check that could probably be optimized). For what it's worth, handling larger data sets is the type of work I want to be doing on it in the future. --Mark -- Mark Fasheh ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-20 22:37 ` Mark Fasheh @ 2014-05-20 22:56 ` Konstantinos Skarlatos 2014-05-21 0:58 ` Chris Murphy 0 siblings, 1 reply; 20+ messages in thread From: Konstantinos Skarlatos @ 2014-05-20 22:56 UTC (permalink / raw) To: Mark Fasheh; +Cc: Brendan Hide, Scott Middleton, linux-btrfs On 21/5/2014 1:37 πμ, Mark Fasheh wrote: > On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote: >>> Duperemove will be shipping as supported software in a major SUSE release so >>> it will be bug fixed, etc as you would expect. At the moment I'm very busy >>> trying to fix qgroup bugs so I haven't had much time to add features, or >>> handle external bug reports, etc. Also I'm not very good at advertising my >>> software which would be why it hasn't really been mentioned on list lately >>> :) >>> >>> I would say that state that it's in is that I've gotten the feature set to a >>> point which feels reasonable, and I've fixed enough bugs that I'd appreciate >>> folks giving it a spin and providing reasonable feedback. >> Well, after having good results with duperemove with a few gigs of data, i >> tried it on a 500gb subvolume. After it scanned all files, it is stuck at >> 100% of one cpu core for about 5 hours, and still hasn't done any deduping. >> My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats >> not the problem. So I guess the speed of duperemove drops dramatically as >> data volume increases. > Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data > sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly > have figured out. It will need a bit of work before it can handle entire > file systems. My guess is that it was spending an enormous amount of time > finding duplicates (it has a very thorough check that could probably be > optimized). It finished after 9 or so hours, so I agree it was checking for duplicates. It does a few GB in just seconds, so time probably scales exponentially with data size. > > For what it's worth, handling larger data sets is the type of work I want to > be doing on it in the future. I can help with testing :) I would also suggest that you publish in this list any changes that you do, so that your program becomes better known among btrfs users. Or even a new announcement mail or a page in the btrfs wiki. Finally, i would like to request the ability to do file level dedup, with a reflink. That has the advantage of consuming very little metadata compared to block level dedup. It could be done with a two pass dedup, first comparing all the same-sized files and after that doing your normal block level dedup. Btw does anybody have a good program/script that can do file level dedup with reflinks and checksum comparison? Kind regards, Konstantinos Skarlatos > --Mark > > -- > Mark Fasheh ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-20 22:56 ` Konstantinos Skarlatos @ 2014-05-21 0:58 ` Chris Murphy 2014-05-23 15:48 ` Konstantinos Skarlatos 0 siblings, 1 reply; 20+ messages in thread From: Chris Murphy @ 2014-05-21 0:58 UTC (permalink / raw) To: Konstantinos Skarlatos Cc: Mark Fasheh, Brendan Hide, Scott Middleton, linux-btrfs On May 20, 2014, at 4:56 PM, Konstantinos Skarlatos <k.skarlatos@gmail.com> wrote: > On 21/5/2014 1:37 πμ, Mark Fasheh wrote: >> On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote: >>>> Duperemove will be shipping as supported software in a major SUSE release so >>>> it will be bug fixed, etc as you would expect. At the moment I'm very busy >>>> trying to fix qgroup bugs so I haven't had much time to add features, or >>>> handle external bug reports, etc. Also I'm not very good at advertising my >>>> software which would be why it hasn't really been mentioned on list lately >>>> :) >>>> >>>> I would say that state that it's in is that I've gotten the feature set to a >>>> point which feels reasonable, and I've fixed enough bugs that I'd appreciate >>>> folks giving it a spin and providing reasonable feedback. >>> Well, after having good results with duperemove with a few gigs of data, i >>> tried it on a 500gb subvolume. After it scanned all files, it is stuck at >>> 100% of one cpu core for about 5 hours, and still hasn't done any deduping. >>> My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats >>> not the problem. So I guess the speed of duperemove drops dramatically as >>> data volume increases. >> Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data >> sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly >> have figured out. It will need a bit of work before it can handle entire >> file systems. My guess is that it was spending an enormous amount of time >> finding duplicates (it has a very thorough check that could probably be >> optimized). > It finished after 9 or so hours, so I agree it was checking for duplicates. It does a few GB in just seconds, so time probably scales exponentially with data size. I'm going to guess it ran out of memory. I wonder what happens if you take an SSD and specify a humongous swap partition on it. Like, 4x, or more, the amount of installed memory. This same trick has been mentioned on the XFS list for use with xfsrepair when memory requirements exceed system memory, and is immensely faster. Chris Murphy ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-21 0:58 ` Chris Murphy @ 2014-05-23 15:48 ` Konstantinos Skarlatos 2014-05-23 16:24 ` Chris Murphy 0 siblings, 1 reply; 20+ messages in thread From: Konstantinos Skarlatos @ 2014-05-23 15:48 UTC (permalink / raw) To: Chris Murphy; +Cc: Mark Fasheh, Brendan Hide, Scott Middleton, linux-btrfs On 21/5/2014 3:58 πμ, Chris Murphy wrote: > On May 20, 2014, at 4:56 PM, Konstantinos Skarlatos <k.skarlatos@gmail.com> wrote: > >> On 21/5/2014 1:37 πμ, Mark Fasheh wrote: >>> On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote: >>>>> Duperemove will be shipping as supported software in a major SUSE release so >>>>> it will be bug fixed, etc as you would expect. At the moment I'm very busy >>>>> trying to fix qgroup bugs so I haven't had much time to add features, or >>>>> handle external bug reports, etc. Also I'm not very good at advertising my >>>>> software which would be why it hasn't really been mentioned on list lately >>>>> :) >>>>> >>>>> I would say that state that it's in is that I've gotten the feature set to a >>>>> point which feels reasonable, and I've fixed enough bugs that I'd appreciate >>>>> folks giving it a spin and providing reasonable feedback. >>>> Well, after having good results with duperemove with a few gigs of data, i >>>> tried it on a 500gb subvolume. After it scanned all files, it is stuck at >>>> 100% of one cpu core for about 5 hours, and still hasn't done any deduping. >>>> My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats >>>> not the problem. So I guess the speed of duperemove drops dramatically as >>>> data volume increases. >>> Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data >>> sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly >>> have figured out. It will need a bit of work before it can handle entire >>> file systems. My guess is that it was spending an enormous amount of time >>> finding duplicates (it has a very thorough check that could probably be >>> optimized). >> It finished after 9 or so hours, so I agree it was checking for duplicates. It does a few GB in just seconds, so time probably scales exponentially with data size. > I'm going to guess it ran out of memory. I wonder what happens if you take an SSD and specify a humongous swap partition on it. Like, 4x, or more, the amount of installed memory. Just tried it again, with 32GiB swap added on an SSD. My test files are 633GiB. duperemove -rv /storage/test 19537.67s user 183.86s system 89% cpu 6:06:56.96 total Duperemove was using about 1GiB or RAM, had one core at 100%, and I think swap was not touched at all. > > This same trick has been mentioned on the XFS list for use with xfsrepair when memory requirements exceed system memory, and is immensely faster. > > > Chris Murphy > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: send/receive and bedup 2014-05-23 15:48 ` Konstantinos Skarlatos @ 2014-05-23 16:24 ` Chris Murphy 0 siblings, 0 replies; 20+ messages in thread From: Chris Murphy @ 2014-05-23 16:24 UTC (permalink / raw) To: Btrfs BTRFS On May 23, 2014, at 9:48 AM, Konstantinos Skarlatos <k.skarlatos@gmail.com> wrote: > On 21/5/2014 3:58 πμ, Chris Murphy wrote: >> On May 20, 2014, at 4:56 PM, Konstantinos Skarlatos <k.skarlatos@gmail.com> wrote: >> >>> On 21/5/2014 1:37 πμ, Mark Fasheh wrote: >>>> On Tue, May 20, 2014 at 01:07:50AM +0300, Konstantinos Skarlatos wrote: >>>>>> Duperemove will be shipping as supported software in a major SUSE release so >>>>>> it will be bug fixed, etc as you would expect. At the moment I'm very busy >>>>>> trying to fix qgroup bugs so I haven't had much time to add features, or >>>>>> handle external bug reports, etc. Also I'm not very good at advertising my >>>>>> software which would be why it hasn't really been mentioned on list lately >>>>>> :) >>>>>> >>>>>> I would say that state that it's in is that I've gotten the feature set to a >>>>>> point which feels reasonable, and I've fixed enough bugs that I'd appreciate >>>>>> folks giving it a spin and providing reasonable feedback. >>>>> Well, after having good results with duperemove with a few gigs of data, i >>>>> tried it on a 500gb subvolume. After it scanned all files, it is stuck at >>>>> 100% of one cpu core for about 5 hours, and still hasn't done any deduping. >>>>> My cpu is an Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz, so i guess thats >>>>> not the problem. So I guess the speed of duperemove drops dramatically as >>>>> data volume increases. >>>> Yeah I doubt it's your CPU. Duperemove is right now targeted at smaller data >>>> sets (a few VMS, iso images, etc) than you threw it at as you undoubtedly >>>> have figured out. It will need a bit of work before it can handle entire >>>> file systems. My guess is that it was spending an enormous amount of time >>>> finding duplicates (it has a very thorough check that could probably be >>>> optimized). >>> It finished after 9 or so hours, so I agree it was checking for duplicates. It does a few GB in just seconds, so time probably scales exponentially with data size. >> I'm going to guess it ran out of memory. I wonder what happens if you take an SSD and specify a humongous swap partition on it. Like, 4x, or more, the amount of installed memory. > Just tried it again, with 32GiB swap added on an SSD. My test files are 633GiB. > duperemove -rv /storage/test 19537.67s user 183.86s system 89% cpu 6:06:56.96 total > > Duperemove was using about 1GiB or RAM, had one core at 100%, and I think swap was not touched at all. Guess currently it's not as memory intensive as it is cpu intensive while also not threading. Chris Murphy ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: historical backups with hardlinks vs cp --reflink vs snapshots 2014-05-19 16:01 ` Brendan Hide 2014-05-19 17:12 ` Konstantinos Skarlatos 2014-05-19 17:38 ` Mark Fasheh @ 2014-05-21 3:59 ` Marc MERLIN 2014-05-22 4:24 ` Russell Coker 2 siblings, 1 reply; 20+ messages in thread From: Marc MERLIN @ 2014-05-21 3:59 UTC (permalink / raw) To: Brendan Hide; +Cc: Scott Middleton, linux-btrfs, Mark Fasheh On Mon, May 19, 2014 at 06:01:25PM +0200, Brendan Hide wrote: > On 19/05/14 15:00, Scott Middleton wrote: > >On 19 May 2014 09:07, Marc MERLIN <marc@merlins.org> wrote: > >>On Wed, May 14, 2014 at 11:36:03PM +0800, Scott Middleton wrote: > >>>I read so much about BtrFS that I mistaked Bedup with Duperemove. > >>>Duperemove is actually what I am testing. > >>I'm currently using programs that find files that are the same, and > >>hardlink them together: > >>http://marc.merlins.org/perso/linux/post_2012-05-01_Handy-tip-to-save-on-inodes-and-disk-space_-finddupes_-fdupes_-and-hardlink_py.html > >> > >>hardlink.py actually seems to be the faster (memory and CPU) one event > >>though it's in python. > >>I can get others to run out of RAM on my 8GB server easily :( > > Interesting app. > > An issue with hardlinking (with the backups use-case, this problem isn't likely to happen), is that if you modify a file, all the hardlinks get changed along with it - including the ones that you don't want changed. > > @Marc: Since you've been using btrfs for a while now I'm sure you've already considered whether or not a reflink copy is the better/worse option. Yes, I have indeed considered it :) I just wrote a blog post about the 3 way of doing historical snapshots: http://marc.merlins.org/perso/btrfs/post_2014-05-20_Historical-Snapshots-With-Btrfs.html I love reflink, but that forces me to use btrfs send as the only way to copy a filesystem without losing the reflink relationship, and I have no good way from user space to see the blocks shared to see how many are shared or whether some just got duped in a copy. As a result, for now I still use hardlinks. Once bedup is a bit more ready, I may switch. That said, duperemove is another dedup I wasn't aware of and I should look at indeed: https://github.com/markfasheh/duperemove/blob/master/README Does it basically do the same work then bedup and tell btrfs to consolidate blocks it indentified as dupes? Does it work across subvolumes? Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: historical backups with hardlinks vs cp --reflink vs snapshots 2014-05-21 3:59 ` historical backups with hardlinks vs cp --reflink vs snapshots Marc MERLIN @ 2014-05-22 4:24 ` Russell Coker 0 siblings, 0 replies; 20+ messages in thread From: Russell Coker @ 2014-05-22 4:24 UTC (permalink / raw) To: Marc MERLIN, linux-btrfs On Tue, 20 May 2014 20:59:28 Marc MERLIN wrote: > just wrote a blog post about the 3 way of doing historical snapshots: > http://marc.merlins.org/perso/btrfs/post_2014-05-20_Historical-Snapshots-Wit > h-Btrfs.html > I love reflink, but that forces me to use btrfs send as the only way to > copy a filesystem without losing the reflink relationship, and I have no > good way from user space to see the blocks shared to see how many are > shared or whether some just got duped in a copy. > As a result, for now I still use hardlinks. It would be nice if someone patched rsync to look for files with identical contents and use reflink or hardlinks (optionally at user request) instead of making multiple copies of the same data. Also it would be nice if rsync would look for matching blocks in different files to save transfer. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/ ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2014-05-23 16:24 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-05-12 12:27 send/receive and bedup Scott Middleton 2014-05-14 13:20 ` Duncan 2014-05-14 15:36 ` Scott Middleton 2014-05-19 1:07 ` Marc MERLIN 2014-05-19 13:00 ` Scott Middleton 2014-05-19 16:01 ` Brendan Hide 2014-05-19 17:12 ` Konstantinos Skarlatos 2014-05-19 17:55 ` Mark Fasheh 2014-05-19 17:59 ` Austin S Hemmelgarn 2014-05-19 18:27 ` Mark Fasheh 2014-05-19 17:38 ` Mark Fasheh 2014-05-19 22:07 ` Konstantinos Skarlatos 2014-05-20 11:12 ` Scott Middleton 2014-05-20 22:37 ` Mark Fasheh 2014-05-20 22:56 ` Konstantinos Skarlatos 2014-05-21 0:58 ` Chris Murphy 2014-05-23 15:48 ` Konstantinos Skarlatos 2014-05-23 16:24 ` Chris Murphy 2014-05-21 3:59 ` historical backups with hardlinks vs cp --reflink vs snapshots Marc MERLIN 2014-05-22 4:24 ` Russell Coker
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).