raid1 with rotating offsite disks for backup

raid1 with rotating offsite disks for backup

[Jeff Klingner]

I'm planning a backup system for my home server and have run into a question I can't find answered in the mailing list archives or the wiki.  Here's the plan:

1. Install system and valuable data on a 3-disk raid1 array (call the disks A, B, and C).
2. Remove disk C, put it offsite.  ("offsite" is moderately time-consuming to get to.)
3a. Periodically, remove disk B, take it offsite, and retrieve disk C
3b. Insert disk C, which will be re-synced to gain any changes made since it was removed.
4. Repeat steps 3a and 3b indefinitely, alternating the roles of disks B and C.

Thus I hope to get continuous protection against a single drive failure and protection back to the last offsite swap for corrupted or deleted data.

My questions are:

In step 3b, when a disk that was a member of the array in the past but has been removed for a while is re-inserted into the 3-disk array, how does the raid system know to update C with A's contents and not A with C's contents?  Is there a timestamp involved, and if so, how can I examine it before syncing?

Is it important to always rotate disks B and C, leaving one that never leaves the computer, or does it make no difference which of the two live disks I pluck out to swap with the offsite disk when I make the trip?  Can all three disks take turns offsite, so that they all have the same duty cycle?

I saw in another list message the advice to use two stacked raid1s for this application: http://marc.info/?l=linux-raid&m=126761399008775&w=2
> Also, if you want two rotating backups I would create two stacked raid1s.
> 
> mdadm -C /dev/md0 -l1 -n2 -b internal  /dev/main-device /dev/first-backup
> mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 /dev/second-backup
> mkfs -j /dev/md1


Are there important differences between the single 3-disk raid1 array I'm planning to use and this stacked configuration?

Thanks for any help you can offer.

Jeff

Re: raid1 with rotating offsite disks for backup

[NeilBrown]

On Mon, 7 Feb 2011 15:53:46 -0800 Jeff Klingner <klingner@stanford.edu> wrote:

> I'm planning a backup system for my home server and have run into a question I can't find answered in the mailing list archives or the wiki.  Here's the plan:
> 
> 1. Install system and valuable data on a 3-disk raid1 array (call the disks A, B, and C).
> 2. Remove disk C, put it offsite.  ("offsite" is moderately time-consuming to get to.)
> 3a. Periodically, remove disk B, take it offsite, and retrieve disk C
> 3b. Insert disk C, which will be re-synced to gain any changes made since it was removed.
> 4. Repeat steps 3a and 3b indefinitely, alternating the roles of disks B and C.
> 
> Thus I hope to get continuous protection against a single drive failure and protection back to the last offsite swap for corrupted or deleted data.
> 
> My questions are:
> 
> In step 3b, when a disk that was a member of the array in the past but has been removed for a while is re-inserted into the 3-disk array, how does the raid system know to update C with A's contents and not A with C's contents?  Is there a timestamp involved, and if so, how can I examine it before syncing?
> 
> Is it important to always rotate disks B and C, leaving one that never leaves the computer, or does it make no difference which of the two live disks I pluck out to swap with the offsite disk when I make the trip?  Can all three disks take turns offsite, so that they all have the same duty cycle?
> 
> I saw in another list message the advice to use two stacked raid1s for this application: http://marc.info/?l=linux-raid&m=126761399008775&w=2
> > Also, if you want two rotating backups I would create two stacked raid1s.
> > 
> > mdadm -C /dev/md0 -l1 -n2 -b internal  /dev/main-device /dev/first-backup
> > mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 /dev/second-backup
> > mkfs -j /dev/md1
> 
> 
> Are there important differences between the single 3-disk raid1 array I'm planning to use and this stacked configuration?

Yes.  The single 3-disk RAID1 array won't work, the stacked configuration
will.


md can resync 'just the changed blocks' by using the 'write intent bitmap'
and event counters.
However it only clears the bits in the bitmap when the array is not degraded.
In your suggestion the 3-drive RAID1 is always degraded so bits are never
cleared, so each resync is effectively a complete resync.

In the stacked configuration there are two bitmaps so md can track of
different differences on two different devices.

You can see the event counts on each device with

  mdadm --examine /dev/A

The bitmap knows whether a device is new enough to be safely recovered based
on the bitmap by storing the event count of the time a bit was cleared 
'Events Cleared'.

Hope that helps.

NeilBrown

Re: raid1 with rotating offsite disks for backup

[Jeff Klingner]

On Feb 7, 2011, at 4:17 PM, NeilBrown wrote:

> On Mon, 7 Feb 2011 15:53:46 -0800 Jeff Klingner <klingner@stanford.edu> wrote:
> 
>> I'm planning a backup system for my home server and have run into a question I can't find answered in the mailing list archives or the wiki.  Here's the plan:
>> 
>> 1. Install system and valuable data on a 3-disk raid1 array (call the disks A, B, and C).
>> 2. Remove disk C, put it offsite.  ("offsite" is moderately time-consuming to get to.)
>> 3a. Periodically, remove disk B, take it offsite, and retrieve disk C
>> 3b. Insert disk C, which will be re-synced to gain any changes made since it was removed.
>> 4. Repeat steps 3a and 3b indefinitely, alternating the roles of disks B and C.
>> 
>> Thus I hope to get continuous protection against a single drive failure and protection back to the last offsite swap for corrupted or deleted data.
>> 
>> I saw in another list message the advice to use two stacked raid1s for this application: http://marc.info/?l=linux-raid&m=126761399008775&w=2
>>> Also, if you want two rotating backups I would create two stacked raid1s.
>>> 
>>> mdadm -C /dev/md0 -l1 -n2 -b internal  /dev/main-device /dev/first-backup
>>> mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 /dev/second-backup
>>> mkfs -j /dev/md1
>> 
>> 
>> Are there important differences between the single 3-disk raid1 array I'm planning to use and this stacked configuration?
> 
> Yes.  The single 3-disk RAID1 array won't work, the stacked configuration
> will.
> 

Goodness, I'm glad I asked!  Thank you!  I've already done step 1.  Do you know if I can transition to the stacked configuration without erasing the disks?  That is, generally speaking, can I change the logical arrangement of the devices in raid1 array(s), then re-sync and keep the current (lvm) storage system built on top of the raid intact?

Jeff

Re: raid1 with rotating offsite disks for backup

[NeilBrown]

On Mon, 7 Feb 2011 17:03:55 -0800 Jeff Klingner <klingner@stanford.edu> wrote:

> 
> On Feb 7, 2011, at 4:17 PM, NeilBrown wrote:
> 
> > On Mon, 7 Feb 2011 15:53:46 -0800 Jeff Klingner <klingner@stanford.edu> wrote:
> > 
> >> I'm planning a backup system for my home server and have run into a question I can't find answered in the mailing list archives or the wiki.  Here's the plan:
> >> 
> >> 1. Install system and valuable data on a 3-disk raid1 array (call the disks A, B, and C).
> >> 2. Remove disk C, put it offsite.  ("offsite" is moderately time-consuming to get to.)
> >> 3a. Periodically, remove disk B, take it offsite, and retrieve disk C
> >> 3b. Insert disk C, which will be re-synced to gain any changes made since it was removed.
> >> 4. Repeat steps 3a and 3b indefinitely, alternating the roles of disks B and C.
> >> 
> >> Thus I hope to get continuous protection against a single drive failure and protection back to the last offsite swap for corrupted or deleted data.
> >> 
> >> I saw in another list message the advice to use two stacked raid1s for this application: http://marc.info/?l=linux-raid&m=126761399008775&w=2
> >>> Also, if you want two rotating backups I would create two stacked raid1s.
> >>> 
> >>> mdadm -C /dev/md0 -l1 -n2 -b internal  /dev/main-device /dev/first-backup
> >>> mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 /dev/second-backup
> >>> mkfs -j /dev/md1
> >> 
> >> 
> >> Are there important differences between the single 3-disk raid1 array I'm planning to use and this stacked configuration?
> > 
> > Yes.  The single 3-disk RAID1 array won't work, the stacked configuration
> > will.
> > 
> 
> Goodness, I'm glad I asked!  Thank you!  I've already done step 1.  Do you know if I can transition to the stacked configuration without erasing the disks?  That is, generally speaking, can I change the logical arrangement of the devices in raid1 array(s), then re-sync and keep the current (lvm) storage system built on top of the raid intact?
> 
> Jeff
> 
> 

I'm surprised that you found the correct advise, and chose not to follow
it .....


The array you will end up installing on is (slightly) smaller than than the
array you have created.  So no: you cannot re-arrange things - you have to
start again.


(well, to be honest, it is possible that you could resize whatever is on the
array and then build the secondary array and make it work.  But that path is
error-prone and this margin is not large enough to properly describe the
process).

NeilBrown

Re: raid1 with rotating offsite disks for backup

[Jeff Klingner]

>>>> Also, if you want two rotating backups I would create two stacked raid1s.
>>>> 
>>>> mdadm -C /dev/md0 -l1 -n2 -b internal  /dev/main-device /dev/first-backup
>>>> mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 /dev/second-backup
>>>> mkfs -j /dev/md1



On Feb 7, 2011, at 5:19 PM, NeilBrown wrote:
> I'm surprised that you found the correct advise, and chose not to follow
> it .....

Thanks again for your advice.  I didn't search the list until after completing my initial setup; my mistake and my lesson learned: research first.

> The array you will end up installing on is (slightly) smaller than than the
> array you have created.  So no: you cannot re-arrange things - you have to
> start again.
> 
> 
> (well, to be honest, it is possible that you could resize whatever is on the
> array and then build the secondary array and make it work.  But that path is
> error-prone and this margin is not large enough to properly describe the
> process).

Reckless fool that I am, I'm going to try it anyway.  Here's my rough plan:

1. Degrade the 3-disk raid1 array by removing two disks, leaving only one, and leaving the array running.
2. Set up the first 2-disk array (md0 in your description) de novo on the two freed disks
3. Move my data from the original raid1 array (with only 1 of three disks present) to the new 2-disk md0 via lvm mirroring
4. Free the third disk by finally taking down the original raid1 array
5. create the second 2-device array (md1) using md0 and the freed third disk.  Somehow ensure that the md0 component has primacy for syncing.

Jeff

Re: raid1 with rotating offsite disks for backup

[NeilBrown]

On Mon, 7 Feb 2011 17:32:48 -0800 Jeff Klingner <klingner@stanford.edu> wrote:

> >>>> Also, if you want two rotating backups I would create two stacked raid1s.
> >>>> 
> >>>> mdadm -C /dev/md0 -l1 -n2 -b internal  /dev/main-device /dev/first-backup
> >>>> mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 /dev/second-backup
> >>>> mkfs -j /dev/md1
> 
> 
> 
> On Feb 7, 2011, at 5:19 PM, NeilBrown wrote:
> > I'm surprised that you found the correct advise, and chose not to follow
> > it .....
> 
> Thanks again for your advice.  I didn't search the list until after completing my initial setup; my mistake and my lesson learned: research first.
> 
> > The array you will end up installing on is (slightly) smaller than than the
> > array you have created.  So no: you cannot re-arrange things - you have to
> > start again.
> > 
> > 
> > (well, to be honest, it is possible that you could resize whatever is on the
> > array and then build the secondary array and make it work.  But that path is
> > error-prone and this margin is not large enough to properly describe the
> > process).
> 
> Reckless fool that I am, I'm going to try it anyway.  Here's my rough plan:
> 
> 1. Degrade the 3-disk raid1 array by removing two disks, leaving only one, and leaving the array running.
> 2. Set up the first 2-disk array (md0 in your description) de novo on the two freed disks
> 3. Move my data from the original raid1 array (with only 1 of three disks present) to the new 2-disk md0 via lvm mirroring

Nope.  That would be wrong.  You need to create both mirrors before moving
any data.
So:

  3.  Set up the second 2-disk array (md1) as a degraded array containing
  only md0 as a member:

     mdadm --create /dev/md1 -l1 -n2 -b internal /dev/md0 missing

  3a. move my data from the original raid1 to the new md1 using pvmove

> 4. Free the third disk by finally taking down the original raid1 array
> 5. create the second 2-device array (md1) using md0 and the freed third disk.  Somehow ensure that the md0 component has primacy for syncing.

Then step 5 becomes:
  5.  Add this newly freed disk to the degraded md1:
    mdadm /dev/md1 --add /dev/whatever

NeilBrown



> 
> Jeff

RE: raid1 with rotating offsite disks for backup

[Leslie Rhorer]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of NeilBrown
> Sent: Monday, February 07, 2011 6:18 PM
> To: Jeff Klingner
> Cc: linux-raid@vger.kernel.org
> Subject: Re: raid1 with rotating offsite disks for backup
> 
> On Mon, 7 Feb 2011 15:53:46 -0800 Jeff Klingner <klingner@stanford.edu>
> wrote:
> 
> > I'm planning a backup system for my home server and have run into a
> question I can't find answered in the mailing list archives or the wiki.
> Here's the plan:
> >
> > 1. Install system and valuable data on a 3-disk raid1 array (call the
> disks A, B, and C).
> > 2. Remove disk C, put it offsite.  ("offsite" is moderately time-
> consuming to get to.)
> > 3a. Periodically, remove disk B, take it offsite, and retrieve disk C
> > 3b. Insert disk C, which will be re-synced to gain any changes made
> since it was removed.
> > 4. Repeat steps 3a and 3b indefinitely, alternating the roles of disks B
> and C.
> >
> > Thus I hope to get continuous protection against a single drive failure
> and protection back to the last offsite swap for corrupted or deleted
> data.
> >
> > My questions are:
> >
> > In step 3b, when a disk that was a member of the array in the past but
> has been removed for a while is re-inserted into the 3-disk array, how
> does the raid system know to update C with A's contents and not A with C's
> contents?  Is there a timestamp involved, and if so, how can I examine it
> before syncing?
> >
> > Is it important to always rotate disks B and C, leaving one that never
> leaves the computer, or does it make no difference which of the two live
> disks I pluck out to swap with the offsite disk when I make the trip?  Can
> all three disks take turns offsite, so that they all have the same duty
> cycle?
> >
> > I saw in another list message the advice to use two stacked raid1s for
> this application: http://marc.info/?l=linux-raid&m=126761399008775&w=2
> > > Also, if you want two rotating backups I would create two stacked
> raid1s.
> > >
> > > mdadm -C /dev/md0 -l1 -n2 -b internal  /dev/main-device /dev/first-
> backup
> > > mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 /dev/second-backup
> > > mkfs -j /dev/md1
> >
> >
> > Are there important differences between the single 3-disk raid1 array
> I'm planning to use and this stacked configuration?
> 
> Yes.  The single 3-disk RAID1 array won't work, the stacked configuration
> will.

	Oh.  I think I mis-read his original post.  When I read it the first
time, I inferred he was attempting this to do a full backup of the array.
Reading again, I think you are correct.  If he wants to just update the data
on the array, I think rsync (or maybe dar) would be a better solution.  If
he wants a full backup from scratch, I don't see why a RAID1 solution would
not work, do you?

> md can resync 'just the changed blocks' by using the 'write intent bitmap'
> and event counters.
> However it only clears the bits in the bitmap when the array is not
> degraded.
> In your suggestion the 3-drive RAID1 is always degraded so bits are never
> cleared, so each resync is effectively a complete resync.

	Yeah, to do a full backup from scratch, I think I would set the
array up as a 2 drive array, take the array off-line, remove one drive,
assemble the array, and then add the spare.  'Clumsy, though.  I still think
he would be better off with rsync or dar.

Re: raid1 with rotating offsite disks for backup

[Roberto Spadim]

i think that a 4 disks array using spare disks is better
first... if you remove 1 mirror (2 mirrors maybe 2 disks...), you have
a protection problem (you don't have redundat array with only 1 mirror
working)
with 4 disks... 2 mirrors, to start backup put a spare online, wait it
to sync, remove it
if you have problem when resync, you have a second spare (the new
backup), and consider the resync disk, as the new array disk (not more
a backup disk)

it's like a online backup... i think we can implement it as snapshot
(on filesystem level) but since we can make it at lower level, it's a
nice feature... i think 4 disks is more secure... 3 disks is just more
cheap..

2011/2/8 Leslie Rhorer <lrhorer@satx.rr.com>:
>
>
>> -----Original Message-----
>> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
>> owner@vger.kernel.org] On Behalf Of NeilBrown
>> Sent: Monday, February 07, 2011 6:18 PM
>> To: Jeff Klingner
>> Cc: linux-raid@vger.kernel.org
>> Subject: Re: raid1 with rotating offsite disks for backup
>>
>> On Mon, 7 Feb 2011 15:53:46 -0800 Jeff Klingner <klingner@stanford.edu>
>> wrote:
>>
>> > I'm planning a backup system for my home server and have run into a
>> question I can't find answered in the mailing list archives or the wiki.
>> Here's the plan:
>> >
>> > 1. Install system and valuable data on a 3-disk raid1 array (call the
>> disks A, B, and C).
>> > 2. Remove disk C, put it offsite.  ("offsite" is moderately time-
>> consuming to get to.)
>> > 3a. Periodically, remove disk B, take it offsite, and retrieve disk C
>> > 3b. Insert disk C, which will be re-synced to gain any changes made
>> since it was removed.
>> > 4. Repeat steps 3a and 3b indefinitely, alternating the roles of disks B
>> and C.
>> >
>> > Thus I hope to get continuous protection against a single drive failure
>> and protection back to the last offsite swap for corrupted or deleted
>> data.
>> >
>> > My questions are:
>> >
>> > In step 3b, when a disk that was a member of the array in the past but
>> has been removed for a while is re-inserted into the 3-disk array, how
>> does the raid system know to update C with A's contents and not A with C's
>> contents?  Is there a timestamp involved, and if so, how can I examine it
>> before syncing?
>> >
>> > Is it important to always rotate disks B and C, leaving one that never
>> leaves the computer, or does it make no difference which of the two live
>> disks I pluck out to swap with the offsite disk when I make the trip?  Can
>> all three disks take turns offsite, so that they all have the same duty
>> cycle?
>> >
>> > I saw in another list message the advice to use two stacked raid1s for
>> this application: http://marc.info/?l=linux-raid&m=126761399008775&w=2
>> > > Also, if you want two rotating backups I would create two stacked
>> raid1s.
>> > >
>> > > mdadm -C /dev/md0 -l1 -n2 -b internal  /dev/main-device /dev/first-
>> backup
>> > > mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 /dev/second-backup
>> > > mkfs -j /dev/md1
>> >
>> >
>> > Are there important differences between the single 3-disk raid1 array
>> I'm planning to use and this stacked configuration?
>>
>> Yes.  The single 3-disk RAID1 array won't work, the stacked configuration
>> will.
>
>        Oh.  I think I mis-read his original post.  When I read it the first
> time, I inferred he was attempting this to do a full backup of the array.
> Reading again, I think you are correct.  If he wants to just update the data
> on the array, I think rsync (or maybe dar) would be a better solution.  If
> he wants a full backup from scratch, I don't see why a RAID1 solution would
> not work, do you?
>
>> md can resync 'just the changed blocks' by using the 'write intent bitmap'
>> and event counters.
>> However it only clears the bits in the bitmap when the array is not
>> degraded.
>> In your suggestion the 3-drive RAID1 is always degraded so bits are never
>> cleared, so each resync is effectively a complete resync.
>
>        Yeah, to do a full backup from scratch, I think I would set the
> array up as a 2 drive array, take the array off-line, remove one drive,
> assemble the array, and then add the spare.  'Clumsy, though.  I still think
> he would be better off with rsync or dar.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid1 with rotating offsite disks for backup

[NeilBrown]

On Tue, 8 Feb 2011 03:37:40 -0200 Roberto Spadim <roberto@spadim.com.br>
wrote:

> i think that a 4 disks array using spare disks is better

If all you are saying is that 4 disks are better than 3 disks - then yes:
obviously.

If you want to rotate two of them out independently it would still be best to
have 2 stacked RAID1 arrays as then  the resync time will be lots shorter.

Or where you saying something else?

NeilBrown


> first... if you remove 1 mirror (2 mirrors maybe 2 disks...), you have
> a protection problem (you don't have redundat array with only 1 mirror
> working)
> with 4 disks... 2 mirrors, to start backup put a spare online, wait it
> to sync, remove it
> if you have problem when resync, you have a second spare (the new
> backup), and consider the resync disk, as the new array disk (not more
> a backup disk)
> 
> it's like a online backup... i think we can implement it as snapshot
> (on filesystem level) but since we can make it at lower level, it's a
> nice feature... i think 4 disks is more secure... 3 disks is just more
> cheap..
> 
> 2011/2/8 Leslie Rhorer <lrhorer@satx.rr.com>:
> >
> >
> >> -----Original Message-----
> >> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> >> owner@vger.kernel.org] On Behalf Of NeilBrown
> >> Sent: Monday, February 07, 2011 6:18 PM
> >> To: Jeff Klingner
> >> Cc: linux-raid@vger.kernel.org
> >> Subject: Re: raid1 with rotating offsite disks for backup
> >>
> >> On Mon, 7 Feb 2011 15:53:46 -0800 Jeff Klingner <klingner@stanford.edu>
> >> wrote:
> >>
> >> > I'm planning a backup system for my home server and have run into a
> >> question I can't find answered in the mailing list archives or the wiki.
> >> Here's the plan:
> >> >
> >> > 1. Install system and valuable data on a 3-disk raid1 array (call the
> >> disks A, B, and C).
> >> > 2. Remove disk C, put it offsite.  ("offsite" is moderately time-
> >> consuming to get to.)
> >> > 3a. Periodically, remove disk B, take it offsite, and retrieve disk C
> >> > 3b. Insert disk C, which will be re-synced to gain any changes made
> >> since it was removed.
> >> > 4. Repeat steps 3a and 3b indefinitely, alternating the roles of disks B
> >> and C.
> >> >
> >> > Thus I hope to get continuous protection against a single drive failure
> >> and protection back to the last offsite swap for corrupted or deleted
> >> data.
> >> >
> >> > My questions are:
> >> >
> >> > In step 3b, when a disk that was a member of the array in the past but
> >> has been removed for a while is re-inserted into the 3-disk array, how
> >> does the raid system know to update C with A's contents and not A with C's
> >> contents?  Is there a timestamp involved, and if so, how can I examine it
> >> before syncing?
> >> >
> >> > Is it important to always rotate disks B and C, leaving one that never
> >> leaves the computer, or does it make no difference which of the two live
> >> disks I pluck out to swap with the offsite disk when I make the trip?  Can
> >> all three disks take turns offsite, so that they all have the same duty
> >> cycle?
> >> >
> >> > I saw in another list message the advice to use two stacked raid1s for
> >> this application: http://marc.info/?l=linux-raid&m=126761399008775&w=2
> >> > > Also, if you want two rotating backups I would create two stacked
> >> raid1s.
> >> > >
> >> > > mdadm -C /dev/md0 -l1 -n2 -b internal  /dev/main-device /dev/first-
> >> backup
> >> > > mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 /dev/second-backup
> >> > > mkfs -j /dev/md1
> >> >
> >> >
> >> > Are there important differences between the single 3-disk raid1 array
> >> I'm planning to use and this stacked configuration?
> >>
> >> Yes.  The single 3-disk RAID1 array won't work, the stacked configuration
> >> will.
> >
> >        Oh.  I think I mis-read his original post.  When I read it the first
> > time, I inferred he was attempting this to do a full backup of the array.
> > Reading again, I think you are correct.  If he wants to just update the data
> > on the array, I think rsync (or maybe dar) would be a better solution.  If
> > he wants a full backup from scratch, I don't see why a RAID1 solution would
> > not work, do you?
> >
> >> md can resync 'just the changed blocks' by using the 'write intent bitmap'
> >> and event counters.
> >> However it only clears the bits in the bitmap when the array is not
> >> degraded.
> >> In your suggestion the 3-drive RAID1 is always degraded so bits are never
> >> cleared, so each resync is effectively a complete resync.
> >
> >        Yeah, to do a full backup from scratch, I think I would set the
> > array up as a 2 drive array, take the array off-line, remove one drive,
> > assemble the array, and then add the spare.  'Clumsy, though.  I still think
> > he would be better off with rsync or dar.
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid1 with rotating offsite disks for backup

[Roberto Spadim]

that's right, i think the point is...
with hardware raid, to 'make a backup' you just need to put disk on
array, wait the led to flash red, and get green after sync, and remove
the disk... i think he is trying to make this with md (software raid)
since we are not hardware raid, software is very better we can make
many more commands and know what's in each disk, i think rsync is
better, maybe a online backup with md is good, but i don't know if
it's really good better than a rsync, or a snapshot at filesystem
level (not md problem)
just to help with ideas =]

2011/2/8 NeilBrown <neilb@suse.de>:
> On Tue, 8 Feb 2011 03:37:40 -0200 Roberto Spadim <roberto@spadim.com.br>
> wrote:
>
>> i think that a 4 disks array using spare disks is better
>
> If all you are saying is that 4 disks are better than 3 disks - then yes:
> obviously.
>
> If you want to rotate two of them out independently it would still be best to
> have 2 stacked RAID1 arrays as then  the resync time will be lots shorter.
>
> Or where you saying something else?
>
> NeilBrown
>
>
>> first... if you remove 1 mirror (2 mirrors maybe 2 disks...), you have
>> a protection problem (you don't have redundat array with only 1 mirror
>> working)
>> with 4 disks... 2 mirrors, to start backup put a spare online, wait it
>> to sync, remove it
>> if you have problem when resync, you have a second spare (the new
>> backup), and consider the resync disk, as the new array disk (not more
>> a backup disk)
>>
>> it's like a online backup... i think we can implement it as snapshot
>> (on filesystem level) but since we can make it at lower level, it's a
>> nice feature... i think 4 disks is more secure... 3 disks is just more
>> cheap..
>>
>> 2011/2/8 Leslie Rhorer <lrhorer@satx.rr.com>:
>> >
>> >
>> >> -----Original Message-----
>> >> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
>> >> owner@vger.kernel.org] On Behalf Of NeilBrown
>> >> Sent: Monday, February 07, 2011 6:18 PM
>> >> To: Jeff Klingner
>> >> Cc: linux-raid@vger.kernel.org
>> >> Subject: Re: raid1 with rotating offsite disks for backup
>> >>
>> >> On Mon, 7 Feb 2011 15:53:46 -0800 Jeff Klingner <klingner@stanford.edu>
>> >> wrote:
>> >>
>> >> > I'm planning a backup system for my home server and have run into a
>> >> question I can't find answered in the mailing list archives or the wiki.
>> >> Here's the plan:
>> >> >
>> >> > 1. Install system and valuable data on a 3-disk raid1 array (call the
>> >> disks A, B, and C).
>> >> > 2. Remove disk C, put it offsite.  ("offsite" is moderately time-
>> >> consuming to get to.)
>> >> > 3a. Periodically, remove disk B, take it offsite, and retrieve disk C
>> >> > 3b. Insert disk C, which will be re-synced to gain any changes made
>> >> since it was removed.
>> >> > 4. Repeat steps 3a and 3b indefinitely, alternating the roles of disks B
>> >> and C.
>> >> >
>> >> > Thus I hope to get continuous protection against a single drive failure
>> >> and protection back to the last offsite swap for corrupted or deleted
>> >> data.
>> >> >
>> >> > My questions are:
>> >> >
>> >> > In step 3b, when a disk that was a member of the array in the past but
>> >> has been removed for a while is re-inserted into the 3-disk array, how
>> >> does the raid system know to update C with A's contents and not A with C's
>> >> contents?  Is there a timestamp involved, and if so, how can I examine it
>> >> before syncing?
>> >> >
>> >> > Is it important to always rotate disks B and C, leaving one that never
>> >> leaves the computer, or does it make no difference which of the two live
>> >> disks I pluck out to swap with the offsite disk when I make the trip?  Can
>> >> all three disks take turns offsite, so that they all have the same duty
>> >> cycle?
>> >> >
>> >> > I saw in another list message the advice to use two stacked raid1s for
>> >> this application: http://marc.info/?l=linux-raid&m=126761399008775&w=2
>> >> > > Also, if you want two rotating backups I would create two stacked
>> >> raid1s.
>> >> > >
>> >> > > mdadm -C /dev/md0 -l1 -n2 -b internal  /dev/main-device /dev/first-
>> >> backup
>> >> > > mdadm -C /dev/md1 -l1 -n2 -b internal /dev/md0 /dev/second-backup
>> >> > > mkfs -j /dev/md1
>> >> >
>> >> >
>> >> > Are there important differences between the single 3-disk raid1 array
>> >> I'm planning to use and this stacked configuration?
>> >>
>> >> Yes.  The single 3-disk RAID1 array won't work, the stacked configuration
>> >> will.
>> >
>> >        Oh.  I think I mis-read his original post.  When I read it the first
>> > time, I inferred he was attempting this to do a full backup of the array.
>> > Reading again, I think you are correct.  If he wants to just update the data
>> > on the array, I think rsync (or maybe dar) would be a better solution.  If
>> > he wants a full backup from scratch, I don't see why a RAID1 solution would
>> > not work, do you?
>> >
>> >> md can resync 'just the changed blocks' by using the 'write intent bitmap'
>> >> and event counters.
>> >> However it only clears the bits in the bitmap when the array is not
>> >> degraded.
>> >> In your suggestion the 3-drive RAID1 is always degraded so bits are never
>> >> cleared, so each resync is effectively a complete resync.
>> >
>> >        Yeah, to do a full backup from scratch, I think I would set the
>> > array up as a 2 drive array, take the array off-line, remove one drive,
>> > assemble the array, and then add the spare.  'Clumsy, though.  I still think
>> > he would be better off with rsync or dar.
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >
>>
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid1 with rotating offsite disks for backup

[Martin Cracauer]

Jeff Klingner wrote on Mon, Feb 07, 2011 at 03:53:46PM -0800: 
> I'm planning a backup system for my home server and have run into a question I can't find answered in the mailing list archives or the wiki.  Here's the plan:
> 
> 1. Install system and valuable data on a 3-disk raid1 array (call the disks A, B, and C).
> 2. Remove disk C, put it offsite.  ("offsite" is moderately time-consuming to get to.)
> 3a. Periodically, remove disk B, take it offsite, and retrieve disk C
> 3b. Insert disk C, which will be re-synced to gain any changes made since it was removed.
> 4. Repeat steps 3a and 3b indefinitely, alternating the roles of disks B and C.
> 
> Thus I hope to get continuous protection against a single drive failure and protection back to the last offsite swap for corrupted or deleted data.

You are aware that this will only work reliably if at the point of
time when you remove the disk the filesystem(s) on it are one of:
- mounted readonly
- unmounted
- machine is off

Linux doesn't really have a `umount -f`, so the first two options only
work if you can get rid of all processes that might want to hold on to
the filesystem at the time when you want to remove your disk.  A
possible hack is going through a NFS mount which does support forceful
operations on the filesystem in Linux.

As has been pointed out, you don't gain much from the added
complexity.  If you would just rsync to one of the spare drives you
would only copy over what actually changed, and not do a full re-sync
of all blocks.  And that works fine with the source filesystem being
mounted read-write.

Another problem is that you are temporarily screwed if disk A dies
while re-syncing B, since C isn't with you, A is hosed and B is
half-synced.

What you do lose is that the raid1 based solution would keep the new
disk up-to-date with then-new file disk writes.  But the problem with
filesystem status is hard to solve.  Overall going 4 disks with raid1
local and having two disks that are rsynced on demand is what I would
do.

Martin
-- 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Martin Cracauer <cracauer@cons.org>   http://www.cons.org/cracauer/

Re: raid1 with rotating offsite disks for backup

[Roberto Spadim]

i think you could use snapshot at filesystem or lvm level
use a backup disk (it's don't have many operations)
on raid use a write-mostly and the other just for write
the first disk to crash is the read/write (more operations)
the second: write-mostly
the third the snapshot/rsync/backup

it's a probability not a real situation... but it's nice =]
if you don't want performace :)

2011/2/8 Martin Cracauer <cracauer@cons.org>:
> Jeff Klingner wrote on Mon, Feb 07, 2011 at 03:53:46PM -0800:
>> I'm planning a backup system for my home server and have run into a question I can't find answered in the mailing list archives or the wiki.  Here's the plan:
>>
>> 1. Install system and valuable data on a 3-disk raid1 array (call the disks A, B, and C).
>> 2. Remove disk C, put it offsite.  ("offsite" is moderately time-consuming to get to.)
>> 3a. Periodically, remove disk B, take it offsite, and retrieve disk C
>> 3b. Insert disk C, which will be re-synced to gain any changes made since it was removed.
>> 4. Repeat steps 3a and 3b indefinitely, alternating the roles of disks B and C.
>>
>> Thus I hope to get continuous protection against a single drive failure and protection back to the last offsite swap for corrupted or deleted data.
>
> You are aware that this will only work reliably if at the point of
> time when you remove the disk the filesystem(s) on it are one of:
> - mounted readonly
> - unmounted
> - machine is off
>
> Linux doesn't really have a `umount -f`, so the first two options only
> work if you can get rid of all processes that might want to hold on to
> the filesystem at the time when you want to remove your disk.  A
> possible hack is going through a NFS mount which does support forceful
> operations on the filesystem in Linux.
>
> As has been pointed out, you don't gain much from the added
> complexity.  If you would just rsync to one of the spare drives you
> would only copy over what actually changed, and not do a full re-sync
> of all blocks.  And that works fine with the source filesystem being
> mounted read-write.
>
> Another problem is that you are temporarily screwed if disk A dies
> while re-syncing B, since C isn't with you, A is hosed and B is
> half-synced.
>
> What you do lose is that the raid1 based solution would keep the new
> disk up-to-date with then-new file disk writes.  But the problem with
> filesystem status is hard to solve.  Overall going 4 disks with raid1
> local and having two disks that are rsynced on demand is what I would
> do.
>
> Martin
> --
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> Martin Cracauer <cracauer@cons.org>   http://www.cons.org/cracauer/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid1 with rotating offsite disks for backup

[hansbkk]

On Tue, Feb 8, 2011 at 6:53 AM, Jeff Klingner <klingner@stanford.edu> wrote:
> I'm planning a backup system for my home server and have run into a question I can't find answered in the mailing list archives or the wiki.  Here's the plan:
>
> 1. Install system and valuable data on a 3-disk raid1 array (call the disks A, B, and C).
> 2. Remove disk C, put it offsite.  ("offsite" is moderately time-consuming to get to.)
> 3a. Periodically, remove disk B, take it offsite, and retrieve disk C
> 3b. Insert disk C, which will be re-synced to gain any changes made since it was removed.
> 4. Repeat steps 3a and 3b indefinitely, alternating the roles of disks B and C.
>
> Thus I hope to get continuous protection against a single drive failure and protection back to the last offsite swap for corrupted or deleted data.


I went through a (probably much too) long discussion here on similar
(but more complex) ideas.


http://www.issociate.de/board/goto/2050886/Q:_LVM_over_RAID,_or_plain_disks?_A:%22Yes%22_=_best_of_both_worlds?.html

Bottom line conclusions
  - mdraid isn't designed for this, and may not be as resilient as you'd expect
  - there is extra overhead involved in the resulting filesystem on
the member, that in this case doesn't actually help fault-tolerance,
makes it more difficult to recover your data from the offsite drive,
and wastes time
  - better to just mount a plain-partitioned disk and rsync for your
removable offsite backups

In my case, some of the target filesystems (backup targets with
millions of hardlinks, created by tools like rdiff-backup and
BackupPC) can't be duplicated at the file-level; I need to copy the
filesystem using block-level tools, so that's why the end of the
thread talks about COTS cloning tools as well as FOSS enhanced
versions of dd.

For regular filesystems, IMO rsync's the way to go.

Re: raid1 with rotating offsite disks for backup

[Roberto Spadim]

i agree with rsync
rsync is more filesystem related feature
if you want copy files, you should use rsync
if you want to copy the device, may be a dd with devices?
dd if=/dev/md0 of=/dev/sda

if you want a snapshot copy, you should first remount your filesystem
to readonly (there´s no flock() for /dev/md0 since filesystem don´t
use flock() on /dev/md0) or another way to block writes to filesystem
while you is read from device, some filesystems have online backup
features..
i don´t know lvm very well, but maybe they implement online backup there...
i don´t think it´s a problem to solve at md level (it could work, but
the raid1 purpose is make a fail safe device (or partially safe), not
a online backup device)


2011/2/9  <hansbkk@gmail.com>:
> On Tue, Feb 8, 2011 at 6:53 AM, Jeff Klingner <klingner@stanford.edu> wrote:
>> I'm planning a backup system for my home server and have run into a question I can't find answered in the mailing list archives or the wiki.  Here's the plan:
>>
>> 1. Install system and valuable data on a 3-disk raid1 array (call the disks A, B, and C).
>> 2. Remove disk C, put it offsite.  ("offsite" is moderately time-consuming to get to.)
>> 3a. Periodically, remove disk B, take it offsite, and retrieve disk C
>> 3b. Insert disk C, which will be re-synced to gain any changes made since it was removed.
>> 4. Repeat steps 3a and 3b indefinitely, alternating the roles of disks B and C.
>>
>> Thus I hope to get continuous protection against a single drive failure and protection back to the last offsite swap for corrupted or deleted data.
>
>
> I went through a (probably much too) long discussion here on similar
> (but more complex) ideas.
>
>
> http://www.issociate.de/board/goto/2050886/Q:_LVM_over_RAID,_or_plain_disks?_A:%22Yes%22_=_best_of_both_worlds?.html
>
> Bottom line conclusions
>  - mdraid isn't designed for this, and may not be as resilient as you'd expect
>  - there is extra overhead involved in the resulting filesystem on
> the member, that in this case doesn't actually help fault-tolerance,
> makes it more difficult to recover your data from the offsite drive,
> and wastes time
>  - better to just mount a plain-partitioned disk and rsync for your
> removable offsite backups
>
> In my case, some of the target filesystems (backup targets with
> millions of hardlinks, created by tools like rdiff-backup and
> BackupPC) can't be duplicated at the file-level; I need to copy the
> filesystem using block-level tools, so that's why the end of the
> thread talks about COTS cloning tools as well as FOSS enhanced
> versions of dd.
>
> For regular filesystems, IMO rsync's the way to go.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid1 with rotating offsite disks for backup

[John Robinson]

On 09/02/2011 19:53, Roberto Spadim wrote:
> i agree with rsync
> rsync is more filesystem related feature
> if you want copy files, you should use rsync
> if you want to copy the device, may be a dd with devices?
> dd if=/dev/md0 of=/dev/sda
>
> if you want a snapshot copy, you should first remount your filesystem
> to readonly (there´s no flock() for /dev/md0 since filesystem don´t
> use flock() on /dev/md0) or another way to block writes to filesystem
> while you is read from device, some filesystems have online backup
> features..
> i don´t know lvm very well, but maybe they implement online backup there...
> i don´t think it´s a problem to solve at md level (it could work, but
> the raid1 purpose is make a fail safe device (or partially safe), not
> a online backup device)

I do both with several of my customers: md array with LVM on top, 
suspend system services (mailstore), take snapshot, start system 
services, then rsync diff the snapshot to another drive with luks 
crypto, which gets rotated off-site once a week. Generally the backup 
drives are bigger than the system array, so I can keep at least a week's 
worth of nightlies and several months' worth of weeklies on each one.

Yes, I have my own separate securely-stored copies of their crypto keys.

Cheers,

John.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html