Incremental send robustness question

Incremental send robustness question

[Sean Greenslade]

Hi, all. I have a question about a backup plan I have involving
send/receive. As far as I can tell, there's no way to to resume a send
that has been interrupted. In this case, my interruption comes from an
overbearing firewall that doesn't like long-lived connections. I'm
trying to do the initial (non-incremental) sync of the first snapshot
from my main server to my backup endpoint. The snapshot is ~900 GiB, and
the internet link is 25 Mbps, so this'll be going for quite a long time.

What I would like to do is "fake" the first snapshot transfer by
rsync-ing the files over. So my question is this: if I rsync a subvolume
(with the -a option to make all file times, permissions, ownerships,
etc. the same), is that good enough to then be used as a parent for
future incremental sends?

And while we're at it, what are the failure modes for incremental sends?
Will it throw an error if the parents don't match, or will there just be
silent failures? I would imagine receive would barf if it was told to
reference a file that didn't exist, but what if the referenced file is
there but contains different data? Are there checks for this sort of
thing, or is it always assumed that the parent subvols are identical and
if they're not, you're in undefined behavior land?

Thanks,

--Sean

Re: Incremental send robustness question

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2374 bytes --]

On Wed, Oct 12, 2016 at 06:29:55PM -0400, Sean Greenslade wrote:
> Hi, all. I have a question about a backup plan I have involving
> send/receive. As far as I can tell, there's no way to to resume a send
> that has been interrupted. In this case, my interruption comes from an
> overbearing firewall that doesn't like long-lived connections. I'm
> trying to do the initial (non-incremental) sync of the first snapshot
> from my main server to my backup endpoint. The snapshot is ~900 GiB, and
> the internet link is 25 Mbps, so this'll be going for quite a long time.
> 
> What I would like to do is "fake" the first snapshot transfer by
> rsync-ing the files over. So my question is this: if I rsync a subvolume
> (with the -a option to make all file times, permissions, ownerships,
> etc. the same), is that good enough to then be used as a parent for
> future incremental sends?

   No, it needs some additional subvol metadata to be set so that the
receiver can detect which subvol to snapshot when restoring an
incremental.

   Depending on how much local storage you have, you could do
something like write the send stream to disk (split into pieces), and
then copy each piece to the receiver, and cat those all together to
replay them. It'd be a bit hairy to get the timing right, though,
without emptying or filling the various pipes.

> And while we're at it, what are the failure modes for incremental sends?
> Will it throw an error if the parents don't match, or will there just be
> silent failures?

   It'll throw an error, as far as I know.

> I would imagine receive would barf if it was told to
> reference a file that didn't exist, but what if the referenced file is
> there but contains different data? Are there checks for this sort of
> thing, or is it always assumed that the parent subvols are identical and
> if they're not, you're in undefined behavior land?

   The parent subvols have UUIDs in them which should act as a
sufficient check for any accidental errors. (See the received-subvol
option for btrfs sub list). It's not robust against an attacker that
can control the UUIDs... but then, if they can do that, there's a
whole load more problems you've got.

   Hugo.

-- 
Hugo Mills             | Great oxymorons of the world, no. 8:
hugo@... carfax.org.uk | The Latest In Proven Technology
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Incremental send robustness question

[Chris Murphy]

On Wed, Oct 12, 2016 at 4:29 PM, Sean Greenslade
<sean@seangreenslade.com> wrote:
> Hi, all. I have a question about a backup plan I have involving
> send/receive. As far as I can tell, there's no way to to resume a send
> that has been interrupted. In this case, my interruption comes from an
> overbearing firewall that doesn't like long-lived connections. I'm
> trying to do the initial (non-incremental) sync of the first snapshot
> from my main server to my backup endpoint. The snapshot is ~900 GiB, and
> the internet link is 25 Mbps, so this'll be going for quite a long time.
>
> What I would like to do is "fake" the first snapshot transfer by
> rsync-ing the files over. So my question is this: if I rsync a subvolume
> (with the -a option to make all file times, permissions, ownerships,
> etc. the same), is that good enough to then be used as a parent for
> future incremental sends?

No. There will be allocation differences in inode and number of
extents, and whether they're shared. But even before that, incremental
receive will fail immediately because it will see the parent isn't
really on the receive side.


> And while we're at it, what are the failure modes for incremental sends?
> Will it throw an error if the parents don't match, or will there just be
> silent failures?

It will give you an error that it can't find the parent.


>I would imagine receive would barf if it was told to
> reference a file that didn't exist, but what if the referenced file is
> there but contains different data? Are there checks for this sort of
> thing, or is it always assumed that the parent subvols are identical and
> if they're not, you're in undefined behavior land?

I haven't tried to break this too much so I'm not sure.

https://btrfs.wiki.kernel.org/index.php/Design_notes_on_Send/Receive

-- 
Chris Murphy

Re: Incremental send robustness question

[Hans van Kranenburg]

On 10/13/2016 12:29 AM, Sean Greenslade wrote:
> Hi, all. I have a question about a backup plan I have involving
> send/receive. As far as I can tell, there's no way to to resume a send
> that has been interrupted. In this case, my interruption comes from an
> overbearing firewall that doesn't like long-lived connections. I'm
> trying to do the initial (non-incremental) sync of the first snapshot
> from my main server to my backup endpoint. The snapshot is ~900 GiB, and
> the internet link is 25 Mbps, so this'll be going for quite a long time.

You can't resume an interrupted send. You'll have to remove the target
subvolume on the destination and start again.

Pipe the send into a local file, and then use any tool that can reliably
resume interrupted transfers to get it to the other side.

Or, if faster, put in on a disk and drive there with your car. :)

At the receiving side, cat the file into btrfs receive.

> What I would like to do is "fake" the first snapshot transfer by
> rsync-ing the files over. So my question is this: if I rsync a subvolume
> (with the -a option to make all file times, permissions, ownerships,
> etc. the same), is that good enough to then be used as a parent for
> future incremental sends?

No. btrfs send/receive works on a different level than rsync. Because
btrfs internally tags all things that happen with a "generation" number,
it can compare two snapshots which were taken from the same subvolume at
different moments, and just walk the btrees and see which parts of it
were updated between the previous and current snapshot.

The first send needs a read-only snapshot of a subvolume and if that one
made it to the remote, you can send incrementals by taking another read
only snapshot and letting btrfs decide what changed between them very
efficiently.

That's why btrfs send -p can do very fast transfers of differences for
snapshots, while rsync would need to compare all metadata of all files
over and over again.

> And while we're at it, what are the failure modes for incremental sends?
> Will it throw an error if the parents don't match, or will there just be
> silent failures?

Create a list of possibilities, create some test filesystems, try it.

> I would imagine receive would barf if it was told to
> reference a file that didn't exist, but what if the referenced file is
> there but contains different data? Are there checks for this sort of
> thing, or is it always assumed that the parent subvols are identical and
> if they're not, you're in undefined behavior land?

btrfs send/receive is not rsync :-)

-- 
Hans van Kranenburg

Re: Incremental send robustness question

[Sean Greenslade]

On Thu, Oct 13, 2016 at 01:14:51AM +0200, Hans van Kranenburg wrote:
> On 10/13/2016 12:29 AM, Sean Greenslade wrote:
> > Hi, all. I have a question about a backup plan I have involving
> > send/receive. As far as I can tell, there's no way to to resume a send
> > that has been interrupted. In this case, my interruption comes from an
> > overbearing firewall that doesn't like long-lived connections. I'm
> > trying to do the initial (non-incremental) sync of the first snapshot
> > from my main server to my backup endpoint. The snapshot is ~900 GiB, and
> > the internet link is 25 Mbps, so this'll be going for quite a long time.
> 
> You can't resume an interrupted send. You'll have to remove the target
> subvolume on the destination and start again.
> 
> Pipe the send into a local file, and then use any tool that can reliably
> resume interrupted transfers to get it to the other side.
> 
> Or, if faster, put in on a disk and drive there with your car. :)

I may just end up doing that. Hugo's responce gave me some crazy ideas
involving a custom build of split that waits for a command after each
output file fills, which would of course require an equally weird build
of cat that would stall the pipe indefinitely until all the files showed
up. Driving the HDD over would probably be a little simpler. =P

> > And while we're at it, what are the failure modes for incremental sends?
> > Will it throw an error if the parents don't match, or will there just be
> > silent failures?
> 
> Create a list of possibilities, create some test filesystems, try it.

I may just do that, presuming I can find the spare time. Given that I'm
building a backup solution around this tech, it would definitely bolster
my confidence in it if I knew what its failure modes looked like.

Thanks to everyone for your fast responses.

--Sean

Re: Incremental send robustness question

[Hans van Kranenburg]

On 10/13/2016 01:47 AM, Sean Greenslade wrote:
> On Thu, Oct 13, 2016 at 01:14:51AM +0200, Hans van Kranenburg wrote:
>> On 10/13/2016 12:29 AM, Sean Greenslade wrote:
>>> And while we're at it, what are the failure modes for incremental sends?
>>> Will it throw an error if the parents don't match, or will there just be
>>> silent failures?
>>
>> Create a list of possibilities, create some test filesystems, try it.
> 
> I may just do that, presuming I can find the spare time. Given that I'm
> building a backup solution around this tech, it would definitely bolster
> my confidence in it if I knew what its failure modes looked like.

Exactly :) Try to break it yourself before you have to start finding out
how things behave 3AM monday morning under pressure.

-- 
Hans van Kranenburg

Re: Incremental send robustness question

[Graham Cobb]

On 13/10/16 00:47, Sean Greenslade wrote:
> I may just end up doing that. Hugo's responce gave me some crazy ideas
> involving a custom build of split that waits for a command after each
> output file fills, which would of course require an equally weird build
> of cat that would stall the pipe indefinitely until all the files showed
> up. Driving the HDD over would probably be a little simpler. =P

I am sure it is, if that is an option.  I had considered doing something
similar: doing an initial send to a big file on a spare disk, then
sending the disk to Amazon to import using their Import/Export Disk
service, then creating a server and a filesystem in AWS and doing a
btrfs receive from the imported file. The plan would then be to do
incremental sends over my home broadband line for subsequent backups.

>>> And while we're at it, what are the failure modes for incremental sends?
>>> Will it throw an error if the parents don't match, or will there just be
>>> silent failures?
>>
>> Create a list of possibilities, create some test filesystems, try it.
> 
> I may just do that, presuming I can find the spare time. Given that I'm
> building a backup solution around this tech, it would definitely bolster
> my confidence in it if I knew what its failure modes looked like.

That's a good idea.  In the end I decided that relying on btrfs send for
my offsite cold storage backups was probably not a good idea.  Btrfs is
great, and I heavily use snapshotting and send/receive locally. But
there is always a small nagging fear that a btrfs bug could introduce
some problem which can survive send/receive and mean the backup was
corrupted as well (or that an incremental won't load for some reason).
For that reason I decided to deliberately use a different technology for
backups.  I now use dar to create backups and then upload the files to a
cloud cold storage service for safekeeping.

There are other reasons as well: encryption is easier to handle, cold
storage for files is cheaper than having disk images which need to be
online to load the incrementals, no need for a virtual server, handling
security for backups from different servers with different levels of
risk is easier, etc. There are also downsides: verifying the backups are
readable/restorable is harder, bandwidth usage is less efficient (dar
sends more data than btrfs send would as it is working at the file
level, not the extent level).

By the way, to test out your various failure modes I recommend creating
some small btrfs filesystems on loop devices -- just be careful to make
sure you create each one from scratch and do not copy disk images (so
that they all have unique UIDs).

Re: Incremental send robustness question

[Duncan]

Sean Greenslade posted on Wed, 12 Oct 2016 18:29:55 -0400 as excerpted:

> Hi, all. I have a question about a backup plan I have involving
> send/receive. As far as I can tell, there's no way to to resume a send
> that has been interrupted. In this case, my interruption comes from an
> overbearing firewall that doesn't like long-lived connections. I'm
> trying to do the initial (non-incremental) sync of the first snapshot
> from my main server to my backup endpoint. The snapshot is ~900 GiB, and
> the internet link is 25 Mbps, so this'll be going for quite a long time.
> 
> What I would like to do is "fake" the first snapshot transfer by
> rsync-ing the files over. So my question is this: if I rsync a subvolume
> (with the -a option to make all file times, permissions, ownerships,
> etc. the same),
> is that good enough to then be used as a parent for future incremental
> sends?

I see the specific questions have been answered, and alternatives 
explored in one direction, but I've another alternative, in a different 
direction, to suggest.

First a disclaimer.  I'm a btrfs user/sysadmin and regular on the list, 
but I'm not a dev, and my own use-case doesn't involve send/receive, so 
what I know regarding send/receive is from the list and manpages, not 
personal experience.  With that in mind...

It's worth noting that send/receive are subvolume-specific -- a send 
won't continue down into a subvolume.

Also note that in addition to -p/parent, there's -s/clone-src.  The 
latter is more flexible than the super-strict parent option, at the 
expense of a fatter send-stream as additional metadata is sent that 
specifies which clone the instructions are relative to.

It should be possible to use the combination of these two facts to split 
and recombine your send stream in a firewall-timeout-friendly manner, as 
long as no individual files are so big that sending an individual file 
exceeds the timeout.

1) Start by taking a read-only snapshot of your intended source 
subvolume, so you have an unchanging reference.

2) Take multiple writable snapshots of it, and selectively delete subdirs 
(and files if necessary) from each writable snapshot, trimming each one 
to a size that should pass the firewall without interruption, so that the 
combination of all these smaller subvolumes contains the content of the 
single larger one.

3) Take read-only snapshots of each of these smaller snapshots, suitable 
for sending.

4) Do a non-incremental send of each of these smaller snapshots to the 
remote.

If it's practical to keep the subvolume divisions, you can simply split 
the working tree into subvolumes and send those individually instead of 
doing the snapshot splitting above, in which case you can then use -p/
parent on each as you were trying to do on the original, and you can stop 
here.

If you need/prefer the single subvolume, continue...

5) Do an incremental send of the original full snapshot, using multiple
-c <src> options to list each of the smaller snapshots.  Since all the 
data has already been transferred in the smaller snapshot sends, this 
send should be all metadata, no actual data.  It'll simply be combining 
the individual reference subvolumes into a single larger subvolume once 
again.

6) Once you have the single larger subvolume on the receive side, you can 
delete the smaller snapshots as you now have a copy of the larger 
subvolume on each side to do further incremental sends of the working 
copy against.

7) I believe the first incremental send of the full working copy against 
the original larger snapshot will still have to use -c, while incremental 
sends based on that first one will be able to use the stricter but 
slimmer send-stream -p, with each one then using the previous one as the 
parent.  However, I'm not sure on that.  It may be that you have to 
continue using the fatter send-stream -c each time.

Again, I don't have send/receive experience of my own, so hopefully 
someone who does can reply either confirming that this should work and 
whether or not -p can be used after the initial setup, or explaining why 
the idea won't work, but at this point based on my own understanding, it 
seems like it should be perfectly workable to me. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Incremental send robustness question

[Sean Greenslade]

On October 14, 2016 12:43:03 AM EDT, Duncan <1i5t5.duncan@cox.net> wrote:
>I see the specific questions have been answered, and alternatives 
>explored in one direction, but I've another alternative, in a different
>
>direction, to suggest.
>
>First a disclaimer.  I'm a btrfs user/sysadmin and regular on the list,
>
>but I'm not a dev, and my own use-case doesn't involve send/receive, so
>
>what I know regarding send/receive is from the list and manpages, not 
>personal experience.  With that in mind...
>
>It's worth noting that send/receive are subvolume-specific -- a send 
>won't continue down into a subvolume.
>
>Also note that in addition to -p/parent, there's -s/clone-src.  The 
>latter is more flexible than the super-strict parent option, at the 
>expense of a fatter send-stream as additional metadata is sent that 
>specifies which clone the instructions are relative to.
>
>It should be possible to use the combination of these two facts to
>split 
>and recombine your send stream in a firewall-timeout-friendly manner,
>as 
>long as no individual files are so big that sending an individual file 
>exceeds the timeout.
>
>1) Start by taking a read-only snapshot of your intended source 
>subvolume, so you have an unchanging reference.
>
>2) Take multiple writable snapshots of it, and selectively delete
>subdirs 
>(and files if necessary) from each writable snapshot, trimming each one
>
>to a size that should pass the firewall without interruption, so that
>the 
>combination of all these smaller subvolumes contains the content of the
>
>single larger one.
>
>3) Take read-only snapshots of each of these smaller snapshots,
>suitable 
>for sending.
>
>4) Do a non-incremental send of each of these smaller snapshots to the 
>remote.
>
>If it's practical to keep the subvolume divisions, you can simply split
>
>the working tree into subvolumes and send those individually instead of
>
>doing the snapshot splitting above, in which case you can then use -p/
>parent on each as you were trying to do on the original, and you can
>stop 
>here.
>
>If you need/prefer the single subvolume, continue...
>
>5) Do an incremental send of the original full snapshot, using multiple
>-c <src> options to list each of the smaller snapshots.  Since all the 
>data has already been transferred in the smaller snapshot sends, this 
>send should be all metadata, no actual data.  It'll simply be combining
>
>the individual reference subvolumes into a single larger subvolume once
>
>again.
>
>6) Once you have the single larger subvolume on the receive side, you
>can 
>delete the smaller snapshots as you now have a copy of the larger 
>subvolume on each side to do further incremental sends of the working 
>copy against.
>
>7) I believe the first incremental send of the full working copy
>against 
>the original larger snapshot will still have to use -c, while
>incremental 
>sends based on that first one will be able to use the stricter but 
>slimmer send-stream -p, with each one then using the previous one as
>the 
>parent.  However, I'm not sure on that.  It may be that you have to 
>continue using the fatter send-stream -c each time.
>
>Again, I don't have send/receive experience of my own, so hopefully 
>someone who does can reply either confirming that this should work and 
>whether or not -p can be used after the initial setup, or explaining
>why 
>the idea won't work, but at this point based on my own understanding,
>it 
>seems like it should be perfectly workable to me. =:^)

I was considering doing something like this, but the simple solution of "just bring the disk over" won out. If that hadn't been possible, I might have done something like that, and I'm still mulling over possible solutions to similar / related problems.

I think the biggest solution would be support for partial / resuming receives. That'll probably go on my ever-growing list of things to possibly look into when I happen upon some free time. It sounds quite complicated, though...

--Sean