[linux-lvm] LVM & snapshots

[linux-lvm] LVM & snapshots

[Gordon Henderson]

I tried LVM a few years back and "burnt my fingers" as it were - I found 
it slow and unstable, but times are changing, progress being made, etc. so 
does anyone have any input on it's current stability and usability?

Basically, I'm running out of steam on some backup servers I have - trying 
to copy several TB of data using cp -al, then rsyncing on top of it (and 
then copying the copy, etc. to give me 30 days of backups) has been 
working well for some time, but the data-set is increasing all the time, 
and the cp -al is currently taking 4-5 hours, so snapshotting via LVM is 
looking like it's something I think I ought to be looking into again.

The current data-set is 5TB, and having 150TB of storage isn't an option 
right now... Essentially I need to be able to keep 30 cycles (days) of 
snapshots on a server that is not a live server (ie. it's a huge disk 
array, in a secure bunker backing up several remote servers every day, and 
dumping the occasional snapshot to tape for archive. Data comes into the 
server via a 100Mb line & rsync from the remote servers).

So is LVM(2) up to this task these days, or should I really be looking at 
something else?

Cheers,

Gordon

Re: [linux-lvm] LVM & snapshots

[Lars Ellenberg]

/ 2007-02-20 12:58:11 +0000
\ Gordon Henderson:
> 
> I tried LVM a few years back and "burnt my fingers" as it were - I found it slow and unstable, but times are 
> changing, progress being made, etc. so does anyone have any input on it's current stability and usability?
> 
> Basically, I'm running out of steam on some backup servers I have - trying to copy several TB of data using cp -al, 
> then rsyncing on top of it (and then copying the copy, etc. to give me 30 days of backups) has been working well 
> for some time, but the data-set is increasing all the time, and the cp -al is currently taking 4-5 hours, so 
> snapshotting via LVM is looking like it's something I think I ought to be looking into again.

lvm works fine for me :-)
but it won't work with 30 snapshots on one origin. not useful, anyways.
it would work ok with two to three snapshots.

but, just in case you want to stay with what you know,
while optimizing it: to cut the time used for your cp -al:
skip it. use the "--link-dest" option to rsync (I'm not sure which
version of rsync you need, mine is 2.6.6).

so I would do a lvm snapshot on the source, then
rsync $other_rsopts_like_include_exclude_whatever \
  -POazv '--log-format=%i %9l %n%L' \
  --link-dest /previous/backup /snapshot/mountpoint backupserver:/target
(or, if you prefer to pull, pull...)
and after that throw away the snapshot on the source machine again.

so not "snapshots as backups" on the backup server,
but as a means to get a consistent fs view on the source
for the duration of the rsync.

> The current data-set is 5TB, and having 150TB of storage isn't an
> option right now... Essentially I need to be able to keep 30 cycles
> (days) of snapshots on a server that is not a live server (ie. it's a
> huge disk array, in a secure bunker backing up several remote servers
> every day, and dumping the occasional snapshot to tape for archive.
> Data comes into the server via a 100Mb line & rsync from the remote
> servers).

-- 
: Lars Ellenberg                            Tel +43-1-8178292-55 :
: LINBIT Information Technologies GmbH      Fax +43-1-8178292-82 :
: Vivenotgasse 48, A-1120 Vienna/Europe    http://www.linbit.com :

Re: [linux-lvm] LVM & snapshots

[Les Mikesell]

Gordon Henderson wrote:
> 
> I tried LVM a few years back and "burnt my fingers" as it were - I found 
> it slow and unstable, but times are changing, progress being made, etc. 
> so does anyone have any input on it's current stability and usability?
> 
> Basically, I'm running out of steam on some backup servers I have - 
> trying to copy several TB of data using cp -al, then rsyncing on top of 
> it (and then copying the copy, etc. to give me 30 days of backups) has 
> been working well for some time, but the data-set is increasing all the 
> time, and the cp -al is currently taking 4-5 hours, so snapshotting via 
> LVM is looking like it's something I think I ought to be looking into 
> again.
> 
> The current data-set is 5TB, and having 150TB of storage isn't an option 
> right now... Essentially I need to be able to keep 30 cycles (days) of 
> snapshots on a server that is not a live server (ie. it's a huge disk 
> array, in a secure bunker backing up several remote servers every day, 
> and dumping the occasional snapshot to tape for archive. Data comes into 
> the server via a 100Mb line & rsync from the remote servers).
> 
> So is LVM(2) up to this task these days, or should I really be looking 
> at something else?

You might want to look at backuppc (http://backuppc.sourceforge.net/) to 
manage the on-line backups.  It uses a scheme of compression and 
hard-linking duplicate files to greatly reduce the storage space you 
need when keeping a long history - especially if many files are 
unchanged over that time.   With something that size you might still 
have to play some tricks with snapshots or break it down into smaller 
directory runs.  It uses tar, smb, or rsync to do the actual copies, 
then has some extra overhead for the compression and linking jobs.  If 
it is too slow, you might run it against your existing uncompressed copy 
so it has all day to make the long-term archive version.

-- 
   Les Mikesell
    lesmikesell@gmail.com

Re: [linux-lvm] LVM & snapshots

[Gordon Henderson]

On Tue, 20 Feb 2007, Les Mikesell wrote:

> You might want to look at backuppc (http://backuppc.sourceforge.net/) to 
> manage the on-line backups.  It uses a scheme of compression and hard-linking 
> duplicate files to greatly reduce the storage space you need when keeping a 
> long history - especially if many files are unchanged over that time.   With 
> something that size you might still have to play some tricks with snapshots 
> or break it down into smaller directory runs.  It uses tar, smb, or rsync to 
> do the actual copies, then has some extra overhead for the compression and 
> linking jobs.  If it is too slow, you might run it against your existing 
> uncompressed copy so it has all day to make the long-term archive version.

Interesting, especially for the Win side of things, thanks!

However one of the things I may need is to use the backup server as a 
"live" server, should the original server ever catch fire (or whatever), 
so pull the backup from the bunker, drive it across town and with minimal 
changes, have it take the place of the live server (turn on nfs/samba on 
the appropriate locations) until we get a new one installed... So it's 
compression would be of minimal use, but it may have other uses...

Thanks,

Gordon

Re: [linux-lvm] LVM & snapshots

[Gordon Henderson]

On Tue, 20 Feb 2007, Lars Ellenberg wrote:

> lvm works fine for me :-)
> but it won't work with 30 snapshots on one origin. not useful, anyways.
> it would work ok with two to three snapshots.

I wondered if that was the case - performance?

> but, just in case you want to stay with what you know,
> while optimizing it: to cut the time used for your cp -al:
> skip it. use the "--link-dest" option to rsync (I'm not sure which
> version of rsync you need, mine is 2.6.6).

My platforms are Debian sarge - rsync is 2.6.4, but they support the 
--link-dest option (which would appear to be a new one for me!)

> so I would do a lvm snapshot on the source, then
> rsync $other_rsopts_like_include_exclude_whatever \
>  -POazv '--log-format=%i %9l %n%L' \
>  --link-dest /previous/backup /snapshot/mountpoint backupserver:/target
> (or, if you prefer to pull, pull...)
> and after that throw away the snapshot on the source machine again.
>
> so not "snapshots as backups" on the backup server,
> but as a means to get a consistent fs view on the source
> for the duration of the rsync.

Right. Thanks - I'll have a look into this. I actually have 2 of these 
backup boxes, one on-site with the tape monster and one off-site, so this 
may be just what's needed for the one with the tape unit.

Cheers,

Gordon

Re: [linux-lvm] LVM & snapshots

[Greg Freemyer]

On 2/20/07, Gordon Henderson <gordon@drogon.net> wrote:
> On Tue, 20 Feb 2007, Les Mikesell wrote:
>
> > You might want to look at backuppc (http://backuppc.sourceforge.net/) to
> > manage the on-line backups.  It uses a scheme of compression and hard-linking
> > duplicate files to greatly reduce the storage space you need when keeping a
> > long history - especially if many files are unchanged over that time.   With
> > something that size you might still have to play some tricks with snapshots
> > or break it down into smaller directory runs.  It uses tar, smb, or rsync to
> > do the actual copies, then has some extra overhead for the compression and
> > linking jobs.  If it is too slow, you might run it against your existing
> > uncompressed copy so it has all day to make the long-term archive version.
>
> Interesting, especially for the Win side of things, thanks!
>
> However one of the things I may need is to use the backup server as a
> "live" server, should the original server ever catch fire (or whatever),
> so pull the backup from the bunker, drive it across town and with minimal
> changes, have it take the place of the live server (turn on nfs/samba on
> the appropriate locations) until we get a new one installed... So it's
> compression would be of minimal use, but it may have other uses...
>
> Thanks,
>
> Gordon
>
You might want to look into rdiff-backup.

It keeps a uncompressed copy of all backed up data.  Then compressed
deltas to get you to older revisions.

The uncompressed copy should be easy to serve via Samba / NSF.

Suse has been including it in their distro for a while.  I don't know
about others.

It is written in python (I think), so it may not be fast enough for
you, but it only ships the deltas across the wire (like rsync), so it
is worth looking into.

Greg
-- 
Greg Freemyer
The Norcross Group
Forensics for the 21st Century