Get the diff from a file in two snapshots.

Get the diff from a file in two snapshots.

[Stef Bon]

Hi,

I'm working on a service/program for Linux which takes care of
managing backupping of folders and the snapshots of those.

The user can assign forlders he/she want to be backupped (corect
English?), and create a profile for this folder. This profile is about
the files which will be taken into process, and which not.

As tool to backup and manage the snapshots I use btrfs.
Now in my construction I get a lot of situations like:

/path/to/snapshots/201501261603/path/to/some/file

is another version of the same file at:

/path/to/snapshots/201501252252/path/to/some/file

is there a way to get the difference between these two files by making
use of btrfs?

I know there are "high level" tools like diff, and tools that fit with
the mimetype of the file
(for ODT files you need another tool than say plaintext files)

I know there is a tool like btrfs send / receive, but that count for
whole snapshots.

If there is such a tool, the next step would be whether it's possible
to provide the user comprehensive information what has changed in the
file, when the mimetype is something else than plaintext, but that's
another question.

Stef Bon

Re: Get the diff from a file in two snapshots.

[Chris Murphy]

On Mon, Jan 26, 2015 at 1:52 PM, Stef Bon <stefbon@gmail.com> wrote:
> Hi,
>
> I'm working on a service/program for Linux which takes care of
> managing backupping of folders and the snapshots of those.
>
> The user can assign forlders he/she want to be backupped (corect
> English?), and create a profile for this folder. This profile is about
> the files which will be taken into process, and which not.
>
> As tool to backup and manage the snapshots I use btrfs.
> Now in my construction I get a lot of situations like:
>
> /path/to/snapshots/201501261603/path/to/some/file
>
> is another version of the same file at:
>
> /path/to/snapshots/201501252252/path/to/some/file
>
> is there a way to get the difference between these two files by making
> use of btrfs?

Snapper has this functionality built into it. I'm not sure if it uses
diff or something else. But the Snapper GUI in yast2 on opensuse will
let you see what files have changed between snapshots, click on the
file and it shows a color coded diff.

https://github.com/openSUSE/snapper



-- 
Chris Murphy

Re: Get the diff from a file in two snapshots.

[Stef Bon]

2015-01-26 22:14 GMT+01:00 Chris Murphy <lists@colorremedies.com>:
>> is there a way to get the difference between these two files by making
>> use of btrfs?
>
> Snapper has this functionality built into it. I'm not sure if it uses
> diff or something else. But the Snapper GUI in yast2 on opensuse will
> let you see what files have changed between snapshots, click on the
> file and it shows a color coded diff.

Hi Chris,

I think (I'm not sure) it's using a high level tool like diff/rdiff.

I will contact the developers of snapper anyway, since my project is
simular to theirs.

But what I'm asking is is there a lowlevel tool to give you a list of
blocks where two versions
of one file differ. I'm asking since btrfs keeps track of the blocks
where files differ, and where they
are the same.

So a lowlevel call to the btrfs filesystem.

Stef

Re: Get the diff from a file in two snapshots.

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 2168 bytes --]

On Tue, Jan 27, 2015 at 12:43:52AM +0100, Stef Bon wrote:
> 2015-01-26 22:14 GMT+01:00 Chris Murphy <lists@colorremedies.com>:
> >> is there a way to get the difference between these two files by making
> >> use of btrfs?
> >
> > Snapper has this functionality built into it. I'm not sure if it uses
> > diff or something else. But the Snapper GUI in yast2 on opensuse will
> > let you see what files have changed between snapshots, click on the
> > file and it shows a color coded diff.
> 
> Hi Chris,
> 
> I think (I'm not sure) it's using a high level tool like diff/rdiff.
> 
> I will contact the developers of snapper anyway, since my project is
> simular to theirs.
> 
> But what I'm asking is is there a lowlevel tool to give you a list of
> blocks where two versions
> of one file differ. I'm asking since btrfs keeps track of the blocks
> where files differ, and where they
> are the same.
> 
> So a lowlevel call to the btrfs filesystem.

You could try the FIEMAP ioctl, which is not btrfs-specific.  The same
ioctl also works on e.g. ext4; however, since ext4 can't share blocks
between files and btrfs can, it's much more interesting on btrfs.  ;)

FIEMAP will give you a list of physical block ranges (btrfs logical
extent addresses).  Run it on both files and analyse the results.
Any blocks with the same physical addresses are trivially identical,
since they refer to the same physical storage.  If your system is full of
huge lightly modified files (e.g. VM images or large databases) that can
save you an awful lot of I/O because you'd only have to compare blocks
from a pair of files if they had different physical addresses.

You can also look up blocks by logical address (see btrfs-tools 'inspect
logical-resolve') and find out what files they belong to.  This gives
you a "which files are similar to this one" sort of query, especially
after you've run dedup over the files in question.

> Stef
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Get the diff from a file in two snapshots.

[Stef Bon]

Aha,

thanks a lot Zygo for the hint about fiemap. I will try this, it could
indeed save an awful lot of I/O.

Stef Bon