* Safely swapping a disk in a RAID456 @ 2011-09-30 15:42 martin f krafft 2011-09-30 16:06 ` Rudy Zijlstra ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: martin f krafft @ 2011-09-30 15:42 UTC (permalink / raw) To: linux-raid mailing list [-- Attachment #1: Type: text/plain, Size: 1012 bytes --] Dear list, I would like to swap out a drive in a RAID6 (4 drives), but ideally without letting the RAID degrade. I was thinking that it should be possible to declare the new disk a copy of the old one, let it sync, then remove the old one and let the new one take over, but MD does not seem to support that. Next, I thought that I could add the new disk as a spare, grow the array to 5 disks, and somehow shrink it back to 4 again, but of course that does not work either, since 4→5 is a reshape, and there seems to be no way to control which disk to remove from the RAID6 on a shrink (making it a spare), without degrading the array in the process. Does anyone have any ideas? This should not be so hard… -- martin | http://madduck.net/ | http://two.sentenc.es/ "courage is not the absence of fear, but the decision that something else is more important than fear." -- ambrose redmoon spamtraps: madduck.bogus@madduck.net [-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) --] [-- Type: application/pgp-signature, Size: 1124 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Safely swapping a disk in a RAID456 2011-09-30 15:42 Safely swapping a disk in a RAID456 martin f krafft @ 2011-09-30 16:06 ` Rudy Zijlstra 2011-09-30 17:07 ` Andre Noll 2011-09-30 16:28 ` Andre Noll 2011-09-30 18:02 ` John Robinson 2 siblings, 1 reply; 9+ messages in thread From: Rudy Zijlstra @ 2011-09-30 16:06 UTC (permalink / raw) To: linux-raid mailing list Hi, On 09/30/2011 05:42 PM, martin f krafft wrote: > Dear list, > > I would like to swap out a drive in a RAID6 (4 drives), but ideally > without letting the RAID degrade. I was thinking that it should be > possible to declare the new disk a copy of the old one, let it sync, > then remove the old one and let the new one take over, but MD does > not seem to support that. If you can stop the array it can be done. Otherwise you will have to degrade it while replacing Replacing without degrade: - stop the raid - do a dd from old disk to new disk - replace the disk - re-start the raid Cheers, Rudy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Safely swapping a disk in a RAID456 2011-09-30 16:06 ` Rudy Zijlstra @ 2011-09-30 17:07 ` Andre Noll 0 siblings, 0 replies; 9+ messages in thread From: Andre Noll @ 2011-09-30 17:07 UTC (permalink / raw) To: Rudy Zijlstra; +Cc: linux-raid mailing list [-- Attachment #1: Type: text/plain, Size: 517 bytes --] On Fri, Sep 30, 18:06, Rudy Zijlstra wrote: > Replacing without degrade: > - stop the raid > - do a dd from old disk to new disk > - replace the disk > - re-start the raid This works but the array needs to be stopped during the dd, which may take a long time. Creating a raid1 first allows you to restart the raid456 immediately after the raid1 has been started, no need to wait for the initialization to complete. Andre -- The only person who always got his work done by Friday was Robinson Crusoe [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Safely swapping a disk in a RAID456 2011-09-30 15:42 Safely swapping a disk in a RAID456 martin f krafft 2011-09-30 16:06 ` Rudy Zijlstra @ 2011-09-30 16:28 ` Andre Noll 2011-10-01 14:53 ` David Brown 2011-09-30 18:02 ` John Robinson 2 siblings, 1 reply; 9+ messages in thread From: Andre Noll @ 2011-09-30 16:28 UTC (permalink / raw) To: linux-raid mailing list [-- Attachment #1: Type: text/plain, Size: 358 bytes --] On Fri, Sep 30, 17:42, martin f krafft wrote: > Does anyone have any ideas? Create a 2-disk Raid1 (with external metadata) containing the disk you'd like to replace and the replacement disk. And yes, it would be great if md provided an easier way to do this. Andre -- The only person who always got his work done by Friday was Robinson Crusoe [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Safely swapping a disk in a RAID456 2011-09-30 16:28 ` Andre Noll @ 2011-10-01 14:53 ` David Brown 2011-10-01 15:51 ` Andre Noll 0 siblings, 1 reply; 9+ messages in thread From: David Brown @ 2011-10-01 14:53 UTC (permalink / raw) To: linux-raid On 30/09/11 18:28, Andre Noll wrote: > On Fri, Sep 30, 17:42, martin f krafft wrote: >> Does anyone have any ideas? > > Create a 2-disk Raid1 (with external metadata) containing the disk > you'd like to replace and the replacement disk. > > And yes, it would be great if md provided an easier way to do this. > That would be "hot replace", which is quite high up on the md/mdadm "things to do" list. <http://neil.brown.name/blog/20110216044002#2> ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Safely swapping a disk in a RAID456 2011-10-01 14:53 ` David Brown @ 2011-10-01 15:51 ` Andre Noll 0 siblings, 0 replies; 9+ messages in thread From: Andre Noll @ 2011-10-01 15:51 UTC (permalink / raw) To: David Brown; +Cc: linux-raid [-- Attachment #1: Type: text/plain, Size: 1235 bytes --] On Sat, Oct 01, 16:53, David Brown wrote: > On 30/09/11 18:28, Andre Noll wrote: > >On Fri, Sep 30, 17:42, martin f krafft wrote: > >>Does anyone have any ideas? > > > >Create a 2-disk Raid1 (with external metadata) containing the disk > >you'd like to replace and the replacement disk. > > > >And yes, it would be great if md provided an easier way to do this. > > > > That would be "hot replace", which is quite high up on the md/mdadm > "things to do" list. > > <http://neil.brown.name/blog/20110216044002#2> Thanks for the pointer. I wonder if it would be useful to have an md-aware dd command which could read from an md device and output only the contents of one component. I.e. it would give the same result as dd if=/dev/component-device of=/dev/new-disk if there are no read errors but would redirect failed reads to the working devices. This should be easy to implement as it avoids the complexity mentioned in Neil's blog posting. Moreover, it could be written purely in user space and would work on any kernel version. The price to pay is of course that the array needs to be r/o during the operation. Andre -- The only person who always got his work done by Friday was Robinson Crusoe [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Safely swapping a disk in a RAID456 2011-09-30 15:42 Safely swapping a disk in a RAID456 martin f krafft 2011-09-30 16:06 ` Rudy Zijlstra 2011-09-30 16:28 ` Andre Noll @ 2011-09-30 18:02 ` John Robinson 2011-10-01 3:38 ` Danny Rawlins 2011-10-01 10:32 ` Asdo 2 siblings, 2 replies; 9+ messages in thread From: John Robinson @ 2011-09-30 18:02 UTC (permalink / raw) To: linux-raid mailing list On 30/09/2011 16:42, martin f krafft wrote: > Dear list, > > I would like to swap out a drive in a RAID6 (4 drives), but ideally > without letting the RAID degrade. I was thinking that it should be > possible to declare the new disc a copy of the old one, let it sync, > then remove the old one and let the new one take over, but MD does > not seem to support that. Not yet, but it's in the roadmap as a hot-replace. In the mean time, you can do something very similar by hand with almost no lost of redundancy, if your RAID456 has a bitmap. > Next, I thought that I could add the new disc as a spare, grow the > array to 5 discs, and somehow shrink it back to 4 again, but of > course that does not work either, since 4→5 is a reshape, and there > seems to be no way to control which disc to remove from the RAID6 on > a shrink (making it a spare), without degrading the array in the > process. Doing a shrink while specifying which drive to remove is also on the roadmap, I believe. But doing huge reshapes like this are not the way forward for a hot-replace. > Does anyone have any ideas? This should not be so hard… I'd be glad to hear of any "real" RAID card that made it easier, or even possible. Something along the lines of: # briefly remove your disc from the array mdadm --manage /dev/md_raid456 --remove /dev/the_disc_i_want_to_remove # make a temp raid1 with only the one disc, using build so metadata is # only in RAM and nothing is written to the disc mdadm --build /dev/md_temp_raid1 --level 1 /dev/the_disc_i_want_to_remove missing # now re-add something which looks identical to your original disc, # but is actually a single-sided mirror, back into the array # the above can all be done very quickly so your raid456 only runs # without a drive for seconds mdadm --manage /dev/md_raid456 --re-add /dev/md_temp_raid1 # now get md to make a mirror (copy) to the new disc mdadm --manage /dev/md_temp_raid1 --add /dev/the_new_disc # wait for it to finish, or just wait for it yourself mdadm --wait /dev/md_temp_raid1 # and switch back again: remove the temporary raid1 mdadm --manage /dev/md_raid456 --remove /dev/md_temp_raid1 # stop it so the new disc becomes available again mdadm --stop /dev/md_temp_raid1 # and put the new disc which is now a complete copy of the old one # back in to the array mdadm --manage /dev/md_raid456 --re-add /dev/the_new_disc # and you're done This is all fine if your old disc has no faulty sectors. If it does have, you need more help from someone much more clued-up than me (because it is already possible to do partial rebuilds by manipulating /sys), or mdadm's roadmap feature of hot-replace, which will do the above and also automatically perform partial rebuilds from the rest of the array when the old disc has bad sectors. Hope this helps. Cheers, John. -- John Robinson, yuiop IT services 0131 557 9577 / 07771 784 058 46/12 Broughton Road, Edinburgh EH7 4EE -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Safely swapping a disk in a RAID456 2011-09-30 18:02 ` John Robinson @ 2011-10-01 3:38 ` Danny Rawlins 2011-10-01 10:32 ` Asdo 1 sibling, 0 replies; 9+ messages in thread From: Danny Rawlins @ 2011-10-01 3:38 UTC (permalink / raw) To: linux-raid John Robinson wrote: > On 30/09/2011 16:42, martin f krafft wrote: >> Dear list, >> >> I would like to swap out a drive in a RAID6 (4 drives), but ideally >> without letting the RAID degrade. I was thinking that it should be >> possible to declare the new disc a copy of the old one, let it sync, >> then remove the old one and let the new one take over, but MD does >> not seem to support that. > > Not yet, but it's in the roadmap as a hot-replace. In the mean time, > you can do something very similar by hand with almost no lost of > redundancy, if your RAID456 has a bitmap. > >> Next, I thought that I could add the new disc as a spare, grow the >> array to 5 discs, and somehow shrink it back to 4 again, but of >> course that does not work either, since 4→5 is a reshape, and there >> seems to be no way to control which disc to remove from the RAID6 on >> a shrink (making it a spare), without degrading the array in the >> process. > > Doing a shrink while specifying which drive to remove is also on the > roadmap, I believe. But doing huge reshapes like this are not the way > forward for a hot-replace. > >> Does anyone have any ideas? This should not be so hard… > > I'd be glad to hear of any "real" RAID card that made it easier, or > even possible. > > Something along the lines of: > > # briefly remove your disc from the array > mdadm --manage /dev/md_raid456 --remove /dev/the_disc_i_want_to_remove > # make a temp raid1 with only the one disc, using build so metadata is > # only in RAM and nothing is written to the disc > mdadm --build /dev/md_temp_raid1 --level 1 > /dev/the_disc_i_want_to_remove missing > # now re-add something which looks identical to your original disc, > # but is actually a single-sided mirror, back into the array > # the above can all be done very quickly so your raid456 only runs > # without a drive for seconds > mdadm --manage /dev/md_raid456 --re-add /dev/md_temp_raid1 > # now get md to make a mirror (copy) to the new disc > mdadm --manage /dev/md_temp_raid1 --add /dev/the_new_disc > # wait for it to finish, or just wait for it yourself > mdadm --wait /dev/md_temp_raid1 > # and switch back again: remove the temporary raid1 > mdadm --manage /dev/md_raid456 --remove /dev/md_temp_raid1 > # stop it so the new disc becomes available again > mdadm --stop /dev/md_temp_raid1 > # and put the new disc which is now a complete copy of the old one > # back in to the array > mdadm --manage /dev/md_raid456 --re-add /dev/the_new_disc > # and you're done > > This is all fine if your old disc has no faulty sectors. If it does > have, you need more help from someone much more clued-up than me > (because it is already possible to do partial rebuilds by manipulating > /sys), or mdadm's roadmap feature of hot-replace, which will do the > above and also automatically perform partial rebuilds from the rest of > the array when the old disc has bad sectors. > > Hope this helps. > > Cheers, > > John. > For bad disks I use GNU ddrescue in two pass like this. ddrescue -n -f /dev/sda /dev/sdb ddrescue.log ddrescue -d -f -r 10 /dev/sda /dev/sdb ddrescue.log I recently found partclone can make a domain-log for ddrescue to only rescue from used file system blocks (but not applicable to this situation) I should get in touch with the GNU ddrescue author to see if (s)he is willing to restructure the ddrescue binary as a library and the ddresuce front end then tings like lvm2 and mdadm could put the ddrescue library to use in recovering as much as they can from said faulty disk before discarding it. PS great idea on adding a bitmap to the raid6 removing disk making a raid1with external metadata then syncing both raid1 disks then removing the raid1 from the raid6 and then breaking the raid1 to put the new disk back into the raid1 group. Seems a little over kill for raid6 but that would make more sense for raid5, though taking less chances on raid6 hitting some sleeping bad blocks (which i like to call them landmines) is a good thing. Regards, Danny Rawlins Romster @ freenode -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Safely swapping a disk in a RAID456 2011-09-30 18:02 ` John Robinson 2011-10-01 3:38 ` Danny Rawlins @ 2011-10-01 10:32 ` Asdo 1 sibling, 0 replies; 9+ messages in thread From: Asdo @ 2011-10-01 10:32 UTC (permalink / raw) To: linux-raid On 09/30/11 20:02, John Robinson wrote: > > Not yet, but it's in the roadmap as a hot-replace. In the mean time, > you can do something very similar by hand with almost no lost of > redundancy, if your RAID456 has a bitmap. > ... > > I'd be glad to hear of any "real" RAID card that made it easier, or > even possible. > > Something along the lines of: > > # briefly remove your disc from the array > mdadm --manage /dev/md_raid456 --remove /dev/the_disc_i_want_to_remove > # make a temp raid1 with only the one disc, using build so metadata is > # only in RAM and nothing is written to the disc > mdadm --build /dev/md_temp_raid1 --level 1 > /dev/the_disc_i_want_to_remove missing > # now re-add something which looks identical to your original disc, > # but is actually a single-sided mirror, back into the array > # the above can all be done very quickly so your raid456 only runs > # without a drive for seconds > mdadm --manage /dev/md_raid456 --re-add /dev/md_temp_raid1 > # now get md to make a mirror (copy) to the new disc > mdadm --manage /dev/md_temp_raid1 --add /dev/the_new_disc > # wait for it to finish, or just wait for it yourself > mdadm --wait /dev/md_temp_raid1 > # and switch back again: remove the temporary raid1 > mdadm --manage /dev/md_raid456 --remove /dev/md_temp_raid1 > # stop it so the new disc becomes available again > mdadm --stop /dev/md_temp_raid1 > # and put the new disc which is now a complete copy of the old one > # back in to the array > mdadm --manage /dev/md_raid456 --re-add /dev/the_new_disc > # and you're done Very good, but... > This is all fine if your old disc has no faulty sectors. If it does > have, you need more help from someone much more clued-up than me > (because it is already possible to do partial rebuilds by manipulating > /sys), This is the problem. And if you want to replace a drive, it's probably because it has bad sectors. I think that during raid1 device-add (which automatically initiates rebuild of raid1), the raid1 would go down completely as soon as it hits bad sectors. So you wouldn't be in a position to use rebuilds by /sys "afterwards", because there is no "afterwards". For this to work, somebody would need to implement the bad block list also on the legacy array without metadata (--build), so that the raid1 would stay up even on bad reads during its rebuild. Actually I don't know if Neil implemented that or not, I have not tested bad blocks list yet, I am intuitively assuming that he improved only the last version of the arrays, that is 1.x . > or mdadm's roadmap feature of hot-replace, which will do the above and > also automatically perform partial rebuilds from the rest of the array > when the old disc has bad sectors. the hot-replace is the most wanted feature as of now. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-10-01 15:51 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-09-30 15:42 Safely swapping a disk in a RAID456 martin f krafft 2011-09-30 16:06 ` Rudy Zijlstra 2011-09-30 17:07 ` Andre Noll 2011-09-30 16:28 ` Andre Noll 2011-10-01 14:53 ` David Brown 2011-10-01 15:51 ` Andre Noll 2011-09-30 18:02 ` John Robinson 2011-10-01 3:38 ` Danny Rawlins 2011-10-01 10:32 ` Asdo
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.