Btrfs file content missmatch incrementally sending subvolumes containing systemd journal files

Btrfs file content missmatch incrementally sending subvolumes containing systemd journal files

[Marc MERLIN]

I had someone asking me about this bug:
Btrfs file content missmatch incrementally sending subvolumes containing systemd journal files
https://bugzilla.kernel.org/show_bug.cgi?id=66941

Specifically:
Just to note, I have also this issue with other files:
jkarlson/.config/chromium/Default/Archived History
jkarlson/.local/share/Trash/files/wheezy-x86_64.img
jkarlson/.local/share/evolution/addressbook/system/log.0000000001
jkarlson/.local/storage/vm-images/wheezy-x86_32.img
jkarlson/.local/storage/vm-images/wheezy-x86_64.img
jkarlson/.local/storage/vm-images/windows-2008-server.img

This is a bit worrisome since it affects sqlite3 too apparently
(firefox/chrome).

Do others know the status of this?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Btrfs file content missmatch incrementally sending subvolumes containing systemd journal files

[Filipe David Manana]

On Sun, Mar 30, 2014 at 12:28 AM, Marc MERLIN <marc@merlins.org> wrote:
> I had someone asking me about this bug:
> Btrfs file content missmatch incrementally sending subvolumes containing systemd journal files
> https://bugzilla.kernel.org/show_bug.cgi?id=66941
>
> Specifically:
> Just to note, I have also this issue with other files:
> jkarlson/.config/chromium/Default/Archived History
> jkarlson/.local/share/Trash/files/wheezy-x86_64.img
> jkarlson/.local/share/evolution/addressbook/system/log.0000000001
> jkarlson/.local/storage/vm-images/wheezy-x86_32.img
> jkarlson/.local/storage/vm-images/wheezy-x86_64.img
> jkarlson/.local/storage/vm-images/windows-2008-server.img
>
> This is a bit worrisome since it affects sqlite3 too apparently
> (firefox/chrome).
>
> Do others know the status of this?

I think I might have found and fixed this today.
See the patch I just sent to the list:

https://patchwork.kernel.org/patch/3910081/

It affects only the 3.14 codebase (any of the RCs).

>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

determining snapshot size

[Marc MERLIN]

I had a look at
http://bj0z.wordpress.com/2011/04/27/determining-snapshot-size-in-btrfs/#comment-35
but it's quite old and does not work anymore since userland became
incompatible with it.

Has anyone seen something newer or have a newer fixed version of this?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: determining snapshot size -> adding "work to do" info to btrfs send

[Marc MERLIN]

On Sat, Mar 29, 2014 at 05:21:23PM -0700, Marc MERLIN wrote:
> I had a look at
> http://bj0z.wordpress.com/2011/04/27/determining-snapshot-size-in-btrfs/#comment-35
> but it's quite old and does not work anymore since userland became
> incompatible with it.
> 
> Has anyone seen something newer or have a newer fixed version of this?

While watching a btrfs send|receive going for hours, keeping the
backup disk array spinning in my living room, and my wondering "how far
is it from being done", I was thinking:

Would it be reasonably simple for btrfs send -p to have a few more
features?
1) don't read all the data from disk, just read the metadata and tell me
how many megabytes it will take to send.
I can do this with 
btrfs send | wc -c 
I believe, but it would be better if it could do this without reading all
the data blocks to send when I'm only caring about the byte output

In turn this could be used to easily compute snapshot size diffs at
least from one another.


2) output a list of files added/changed/removed, maybe with how much
data is related to each.
Arguably this would supersede 1) above even if it would be a little bit
more work to do


3) when a real btrfs send is running, just like dd, I could send it a
USR1 signal and it would output some kind of progress report.
The Ted T'so motto (used for e2fsck) is "everything with a progress bar
is faster" :)
Note, the progress wouldn't have to be perfect, it could be by number of
blocks, number of files, anything reasonably easy to implement it on.

Does that sound reasonable?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: determining snapshot size -> adding "work to do" info to btrfs send

[Filipe David Manana]

On Mon, Mar 31, 2014 at 5:35 PM, Marc MERLIN <marc@merlins.org> wrote:
> On Sat, Mar 29, 2014 at 05:21:23PM -0700, Marc MERLIN wrote:
>> I had a look at
>> http://bj0z.wordpress.com/2011/04/27/determining-snapshot-size-in-btrfs/#comment-35
>> but it's quite old and does not work anymore since userland became
>> incompatible with it.
>>
>> Has anyone seen something newer or have a newer fixed version of this?
>
> While watching a btrfs send|receive going for hours, keeping the
> backup disk array spinning in my living room, and my wondering "how far
> is it from being done", I was thinking:
>
> Would it be reasonably simple for btrfs send -p to have a few more
> features?
> 1) don't read all the data from disk, just read the metadata and tell me
> how many megabytes it will take to send.
> I can do this with
> btrfs send | wc -c
> I believe, but it would be better if it could do this without reading all
> the data blocks to send when I'm only caring about the byte output
>
> In turn this could be used to easily compute snapshot size diffs at
> least from one another.

Totally agree.
A progress indicator makes the feature much more user friendly.
In fact this is something I had been thinking for a while, but didn't
start implementation until a couple days ago after your mail. Here's a
couple RFC patches (kernel and btrfs-progs) with a prototype:

https://patchwork.kernel.org/patch/3938801/
https://patchwork.kernel.org/patch/3938811/

>
>
> 2) output a list of files added/changed/removed, maybe with how much
> data is related to each.
> Arguably this would supersede 1) above even if it would be a little bit
> more work to do

That would be cool I think. Most of what is needed is already
implemented in the send code.
You can get it somehow, in a not very user friendly way by passing the
flag BTRFS_SEND_FLAG_NO_FILE_DATA to the send ioctl, which doesn't
send any data back, only information about new/modified extent's
offset and length and passing -vvv to btrfs receive.

>
>
> 3) when a real btrfs send is running, just like dd, I could send it a
> USR1 signal and it would output some kind of progress report.
> The Ted T'so motto (used for e2fsck) is "everything with a progress bar
> is faster" :)
> Note, the progress wouldn't have to be perfect, it could be by number of
> blocks, number of files, anything reasonably easy to implement it on.
>
> Does that sound reasonable?

It does.

>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."