Hi, I'm working on a service/program for Linux which takes care of managing backupping of folders and the snapshots of those. The user can assign forlders he/she want to be backupped (corect English?), and create a profile for this folder. This profile is about the files which will be taken into process, and which not. As tool to backup and manage the snapshots I use btrfs. Now in my construction I get a lot of situations like: /path/to/snapshots/201501261603/path/to/some/file is another version of the same file at: /path/to/snapshots/201501252252/path/to/some/file is there a way to get the difference between these two files by making use of btrfs? I know there are "high level" tools like diff, and tools that fit with the mimetype of the file (for ODT files you need another tool than say plaintext files) I know there is a tool like btrfs send / receive, but that count for whole snapshots. If there is such a tool, the next step would be whether it's possible to provide the user comprehensive information what has changed in the file, when the mimetype is something else than plaintext, but that's another question. Stef Bon
On Mon, Jan 26, 2015 at 1:52 PM, Stef Bon <stefbon@gmail.com> wrote: > Hi, > > I'm working on a service/program for Linux which takes care of > managing backupping of folders and the snapshots of those. > > The user can assign forlders he/she want to be backupped (corect > English?), and create a profile for this folder. This profile is about > the files which will be taken into process, and which not. > > As tool to backup and manage the snapshots I use btrfs. > Now in my construction I get a lot of situations like: > > /path/to/snapshots/201501261603/path/to/some/file > > is another version of the same file at: > > /path/to/snapshots/201501252252/path/to/some/file > > is there a way to get the difference between these two files by making > use of btrfs? Snapper has this functionality built into it. I'm not sure if it uses diff or something else. But the Snapper GUI in yast2 on opensuse will let you see what files have changed between snapshots, click on the file and it shows a color coded diff. https://github.com/openSUSE/snapper -- Chris Murphy
2015-01-26 22:14 GMT+01:00 Chris Murphy <lists@colorremedies.com>:
>> is there a way to get the difference between these two files by making
>> use of btrfs?
>
> Snapper has this functionality built into it. I'm not sure if it uses
> diff or something else. But the Snapper GUI in yast2 on opensuse will
> let you see what files have changed between snapshots, click on the
> file and it shows a color coded diff.
Hi Chris,
I think (I'm not sure) it's using a high level tool like diff/rdiff.
I will contact the developers of snapper anyway, since my project is
simular to theirs.
But what I'm asking is is there a lowlevel tool to give you a list of
blocks where two versions
of one file differ. I'm asking since btrfs keeps track of the blocks
where files differ, and where they
are the same.
So a lowlevel call to the btrfs filesystem.
Stef
[-- Attachment #1: Type: text/plain, Size: 2168 bytes --] On Tue, Jan 27, 2015 at 12:43:52AM +0100, Stef Bon wrote: > 2015-01-26 22:14 GMT+01:00 Chris Murphy <lists@colorremedies.com>: > >> is there a way to get the difference between these two files by making > >> use of btrfs? > > > > Snapper has this functionality built into it. I'm not sure if it uses > > diff or something else. But the Snapper GUI in yast2 on opensuse will > > let you see what files have changed between snapshots, click on the > > file and it shows a color coded diff. > > Hi Chris, > > I think (I'm not sure) it's using a high level tool like diff/rdiff. > > I will contact the developers of snapper anyway, since my project is > simular to theirs. > > But what I'm asking is is there a lowlevel tool to give you a list of > blocks where two versions > of one file differ. I'm asking since btrfs keeps track of the blocks > where files differ, and where they > are the same. > > So a lowlevel call to the btrfs filesystem. You could try the FIEMAP ioctl, which is not btrfs-specific. The same ioctl also works on e.g. ext4; however, since ext4 can't share blocks between files and btrfs can, it's much more interesting on btrfs. ;) FIEMAP will give you a list of physical block ranges (btrfs logical extent addresses). Run it on both files and analyse the results. Any blocks with the same physical addresses are trivially identical, since they refer to the same physical storage. If your system is full of huge lightly modified files (e.g. VM images or large databases) that can save you an awful lot of I/O because you'd only have to compare blocks from a pair of files if they had different physical addresses. You can also look up blocks by logical address (see btrfs-tools 'inspect logical-resolve') and find out what files they belong to. This gives you a "which files are similar to this one" sort of query, especially after you've run dedup over the files in question. > Stef > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --]
Aha, thanks a lot Zygo for the hint about fiemap. I will try this, it could indeed save an awful lot of I/O. Stef Bon