* Copying a disk containing a btrfs filesystem
@ 2014-04-10 13:21 Michael Schuerig
2014-04-10 13:58 ` Duncan
` (3 more replies)
0 siblings, 4 replies; 15+ messages in thread
From: Michael Schuerig @ 2014-04-10 13:21 UTC (permalink / raw)
To: linux-btrfs, linux-btrfs
SMART indicates that my notebook disk may soon be failing (an
unreadable/uncorrectable sector), therefore I intend to exchange it. The
disk contains a single btrfs filesystem with several nested(!)
subvolumes, each with several read-only snapshots in a .snapshots
subdirectory.
As far as I can tell, btrfs currently does not offer a sensible way to
duplicate the entire contents of the old disk onto a new one. I can use
cp, rsync, or send/receive to copy the "main" subvolumes. But unless I'm
missing something obvious, the snapshots are effectively lost. btrfs
send optionally takes multiple clone sources, but I've never seen an
example of its usage.
If that's what "experimental" means, I'm willing to accept it. However,
I'd like to emphasize that there's still something missing. Of course,
most of all I'd like to be proved wrong.
Michael
--
Michael Schuerig
mailto:michael@schuerig.de
http://www.schuerig.de/michael/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 13:21 Copying a disk containing a btrfs filesystem Michael Schuerig
@ 2014-04-10 13:58 ` Duncan
2014-04-10 14:53 ` Michael Schuerig
2014-04-10 14:00 ` George Eleftheriou
` (2 subsequent siblings)
3 siblings, 1 reply; 15+ messages in thread
From: Duncan @ 2014-04-10 13:58 UTC (permalink / raw)
To: linux-btrfs
Michael Schuerig posted on Thu, 10 Apr 2014 15:21:01 +0200 as excerpted:
> SMART indicates that my notebook disk may soon be failing (an
> unreadable/uncorrectable sector), therefore I intend to exchange it. The
> disk contains a single btrfs filesystem with several nested(!)
> subvolumes, each with several read-only snapshots in a .snapshots
> subdirectory.
>
> As far as I can tell, btrfs currently does not offer a sensible way to
> duplicate the entire contents of the old disk onto a new one. I can use
> cp, rsync, or send/receive to copy the "main" subvolumes. But unless I'm
> missing something obvious, the snapshots are effectively lost. btrfs
> send optionally takes multiple clone sources, but I've never seen an
> example of its usage.
>
> If that's what "experimental" means, I'm willing to accept it. However,
> I'd like to emphasize that there's still something missing. Of course,
> most of all I'd like to be proved wrong.
It's not a btrfs tool, but several of the tools you mentioned aren't, and
you didn't mention dd (or ddrescue, if your source device starts giving
you issues while you're cloning). Using it you'd clone the entire raw
device, including any not yet allocated areas, in a straight-across bit-
perfect copy. Of course you'd need a target of either the same size or
larger in ordered to do so...
That should give you a bit-perfect copy including the filesystem UUIDs,
etc, which will confuse btrfs if you try to mount anything btrfs with
both devices attached, so don't. Do your clone, then umount and
disconnect the old device before trying to mount the new one.
In fact, there are entire purpose-built special live-image distributions
such as Clonezilla for managing storage devices like this.
http://clonezilla.org/
Or use a more general purpose "rescue" live-image distro such as
SysRescue:
http://www.sysresccd.org/
Both of these tools support MS Windows systems as well. Image-cloning is
of course OS-agnostic -- what's on the cloned image doesn't matter, just
whether the source device is readable and the destination writable. And
sysrescue is often used in the MS world to reset lost passwords and etc.
I used to use it for that back when I still worked on friends' MS systems
for "gratis" (well, traded for a good dinner or whatever), before I
decided that wasn't a worthwhile use of my time, tho I'd be happy to
teach them about Linux if they wanted, or might still do it if they paid
me enough, but then there's other computer repair services that will do
that if they're willing to pay.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 13:21 Copying a disk containing a btrfs filesystem Michael Schuerig
2014-04-10 13:58 ` Duncan
@ 2014-04-10 14:00 ` George Eleftheriou
2014-04-10 15:15 ` Duncan
2014-04-10 17:17 ` Jan Kouba
2014-04-11 10:39 ` Brendan Hide
3 siblings, 1 reply; 15+ messages in thread
From: George Eleftheriou @ 2014-04-10 14:00 UTC (permalink / raw)
To: Michael Schuerig, linux-btrfs
I would see one (dangerous? risky? needing more options perhaps?) solution:
dd if=/dev/sda of=/dev/sdb
/dev/sda: old disk
/dev/sdb: new disk
Maybe there is another much more complicated solution:
Plug the old disk in a USB dock/case, do the same for the new disk in
another dock/case, plug both docks/cases in another linux system
running recent kernel and btrfs-progs and convert to a
"-dconvert=raid1 -mconvert=raid1" profile with a balance. Then degrade
it by removing the old disk and rebalance-convert back to single or
DUP profile (i am not quite sure this is even possible).
Just an idea. I wouldn't trust me.
On Thu, Apr 10, 2014 at 3:21 PM, Michael Schuerig
<michael.lists@schuerig.de> wrote:
>
> SMART indicates that my notebook disk may soon be failing (an
> unreadable/uncorrectable sector), therefore I intend to exchange it. The
> disk contains a single btrfs filesystem with several nested(!)
> subvolumes, each with several read-only snapshots in a .snapshots
> subdirectory.
>
> As far as I can tell, btrfs currently does not offer a sensible way to
> duplicate the entire contents of the old disk onto a new one. I can use
> cp, rsync, or send/receive to copy the "main" subvolumes. But unless I'm
> missing something obvious, the snapshots are effectively lost. btrfs
> send optionally takes multiple clone sources, but I've never seen an
> example of its usage.
>
> If that's what "experimental" means, I'm willing to accept it. However,
> I'd like to emphasize that there's still something missing. Of course,
> most of all I'd like to be proved wrong.
>
> Michael
>
> --
> Michael Schuerig
> mailto:michael@schuerig.de
> http://www.schuerig.de/michael/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 13:58 ` Duncan
@ 2014-04-10 14:53 ` Michael Schuerig
0 siblings, 0 replies; 15+ messages in thread
From: Michael Schuerig @ 2014-04-10 14:53 UTC (permalink / raw)
To: linux-btrfs
On Thursday 10 April 2014 13:58:45 Duncan wrote:
> Michael Schuerig posted on Thu, 10 Apr 2014 15:21:01 +0200 as
excerpted:
> > SMART indicates that my notebook disk may soon be failing (an
> > unreadable/uncorrectable sector), therefore I intend to exchange it.
> > The disk contains a single btrfs filesystem with several nested(!)
> > subvolumes, each with several read-only snapshots in a .snapshots
> > subdirectory.
> >
> > As far as I can tell, btrfs currently does not offer a sensible way
> > to duplicate the entire contents of the old disk onto a new one. I
> > can use cp, rsync, or send/receive to copy the "main" subvolumes.
> > But unless I'm missing something obvious, the snapshots are
> > effectively lost. btrfs send optionally takes multiple clone
> > sources, but I've never seen an example of its usage.
> >
> > If that's what "experimental" means, I'm willing to accept it.
> > However, I'd like to emphasize that there's still something
> > missing. Of course, most of all I'd like to be proved wrong.
>
> It's not a btrfs tool, but several of the tools you mentioned aren't,
> and you didn't mention dd (or ddrescue, if your source device starts
> giving you issues while you're cloning). Using it you'd clone the
> entire raw device, including any not yet allocated areas, in a
> straight-across bit- perfect copy. Of course you'd need a target of
> either the same size or larger in ordered to do so...
>
> That should give you a bit-perfect copy including the filesystem
> UUIDs, etc, which will confuse btrfs if you try to mount anything
> btrfs with both devices attached, so don't. Do your clone, then
> umount and disconnect the old device before trying to mount the new
> one.
Thanks for pointing out dd and especially ddrescue, which I didn't know.
I regularly use dd to copy images onto USB thumb drives, but I'm a bit
wary regarding its use on hard disks. I've always been concerned that if
I get another disk, not necessarily of the same make, that has nominally
the same capacity, it still might be a few blocks smaller. Well, I just
found one that has the right size, apparently.
Thanks again,
Michael
--
Michael Schuerig
mailto:michael@schuerig.de
http://www.schuerig.de/michael/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 14:00 ` George Eleftheriou
@ 2014-04-10 15:15 ` Duncan
2014-04-10 15:51 ` Michael Schuerig
0 siblings, 1 reply; 15+ messages in thread
From: Duncan @ 2014-04-10 15:15 UTC (permalink / raw)
To: linux-btrfs
George Eleftheriou posted on Thu, 10 Apr 2014 16:00:06 +0200 as excerpted:
> Maybe there is another much more complicated solution:
>
> Plug the old disk in a USB dock/case, do the same for the new disk in
> another dock/case, plug both docks/cases in another linux system running
> recent kernel and btrfs-progs and convert to a "-dconvert=raid1
> -mconvert=raid1" profile with a balance. Then degrade it by removing the
> old disk and rebalance-convert back to single or DUP profile (i am not
> quite sure this is even possible).
>
> Just an idea. I wouldn't trust me.
Creative idea, actually, and one I've seen come up here before tho if you
read my earlier response I didn't mention it myself, as I failed to
recall it.
But yes, it is somewhat more complicated, and sort of an abuse of
existing features for a purpose they weren't intended for. It should
still work and such unintended purpose uses are a good thing, but because
it is unintended usage, it won't have been tested much if at all, and
given that btrfs isn't itself fully stable yet, there's a fair likelihood
that some unanticipated complexities would arise while doing it, as well.
Meanwhile, by the very fact that btrfs is /not/ yet entirely stable,
there's even more reason than normal to have good, tested backups. Given
that, there should be no harm in trying this, since even if it somehow
kills the working copy, one can still restore from those already tested
backups. And someone's gotta be the first to test this method and report
on how it went, right?
Meanwhile (2), given the existence of those tested backups, there's yet
another way to accomplish things. Simply restore from the backups the
same way you would if the working copy went down and you had to restore
it, only restore to the new device instead of the old one. =:^)
(Actually, given my usual insistence on this key point, that btrfs isn't
yet fully stable and that as a result by definition, you either have
tested backups or you're demonstrating by practice that you simply don't
care about losing the data anyway whatever your words might otherwise
claim, I really should have thought of and mentioned the restoring from
backups method first, instead of having it occur to me as an after-
thought. I must be getting sloppy and careless in my thinking! =:^( )
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 15:15 ` Duncan
@ 2014-04-10 15:51 ` Michael Schuerig
2014-04-10 16:01 ` George Eleftheriou
2014-04-10 18:15 ` Martin Steigerwald
0 siblings, 2 replies; 15+ messages in thread
From: Michael Schuerig @ 2014-04-10 15:51 UTC (permalink / raw)
To: linux-btrfs
On Thursday 10 April 2014 15:15:02 Duncan wrote:
> Meanwhile (2), given the existence of those tested backups, there's
> yet another way to accomplish things. Simply restore from the
> backups the same way you would if the working copy went down and you
> had to restore it, only restore to the new device instead of the old
> one. =:^)
As the OP, let me insist that I have multiple backups. However and
unfortunately, those backups do not contain the snapshots that I'd like
to preserve when exchanging the disk. What makes the case complicate is
not the question how to preserve and copy the current data; it's how to
retain the historic data embodied in snapshots.
Michael
--
Michael Schuerig
mailto:michael@schuerig.de
http://www.schuerig.de/michael/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 15:51 ` Michael Schuerig
@ 2014-04-10 16:01 ` George Eleftheriou
2014-04-10 16:24 ` Michael Schuerig
2014-04-10 18:15 ` Martin Steigerwald
1 sibling, 1 reply; 15+ messages in thread
From: George Eleftheriou @ 2014-04-10 16:01 UTC (permalink / raw)
To: Michael Schuerig; +Cc: linux-btrfs
> What makes the case complicate is
> not the question how to preserve and copy the current data; it's how to
> retain the historic data embodied in snapshots.
>
You can always rsync (incrementally with --link-dest) to "another
place" the sequence of snapshots, provided of course there is enough
space in this "other place". However I understand this may not be the
coolest thing to do especially taking into account the SMART warnings
for imminent deterioration.
On Thu, Apr 10, 2014 at 5:15 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> claim, I really should have thought of and mentioned the restoring from
> backups method first, instead of having it occur to me as an after-
> thought. I must be getting sloppy and careless in my thinking! =:^( )
Maybe this is related to the fact that btrfs is maturing, gradually
causing you a sense of complete safety ;-)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 16:01 ` George Eleftheriou
@ 2014-04-10 16:24 ` Michael Schuerig
2014-04-11 0:35 ` Duncan
0 siblings, 1 reply; 15+ messages in thread
From: Michael Schuerig @ 2014-04-10 16:24 UTC (permalink / raw)
To: linux-btrfs
On Thursday 10 April 2014 18:01:31 George Eleftheriou wrote:
> > What makes the case complicate is
> > not the question how to preserve and copy the current data; it's how
> > to retain the historic data embodied in snapshots.
>
> You can always rsync (incrementally with --link-dest) to "another
> place" the sequence of snapshots, provided of course there is enough
> space in this "other place". However I understand this may not be the
> coolest thing to do especially taking into account the SMART warnings
> for imminent deterioration.
In the case at hand, I'll go with ddrescue. As far as backups are
concerned, rsync --link-dest may be an option, however even at best it
would only give file-granularity COW as opposed to block-granularity.
I'm not sure if this would work for me in terms of disk space
consumption.
I *think* send/receive with clone sources might be the key to a
solution. I'm still hoping that someone with a far better understanding
of btrfs than me gives it a try first...
Michael
--
Michael Schuerig
mailto:michael@schuerig.de
http://www.schuerig.de/michael/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 13:21 Copying a disk containing a btrfs filesystem Michael Schuerig
2014-04-10 13:58 ` Duncan
2014-04-10 14:00 ` George Eleftheriou
@ 2014-04-10 17:17 ` Jan Kouba
2014-04-10 18:36 ` Michael Schuerig
2014-04-11 10:39 ` Brendan Hide
3 siblings, 1 reply; 15+ messages in thread
From: Jan Kouba @ 2014-04-10 17:17 UTC (permalink / raw)
To: linux-btrfs
Dne Čt 10. dubna 2014 15:21:01, Michael Schuerig napsal(a):
> SMART indicates that my notebook disk may soon be failing (an
> unreadable/uncorrectable sector), therefore I intend to exchange it. The
> disk contains a single btrfs filesystem with several nested(!)
> subvolumes, each with several read-only snapshots in a .snapshots
> subdirectory.
>
> As far as I can tell, btrfs currently does not offer a sensible way to
> duplicate the entire contents of the old disk onto a new one.
Yes it does
You can make the old disk a seeding device and use it to seed the new one like
this:
btrfstune -S 1 <old_device>
mount <old_device> /mnt
# this will be mounted read-only
btrfs dev add <new_device> /mnt
mount -o remount,rw /mnt
btrfs dev delete <old_device>
According to my experiments, the filesystem on the new device will have
different UUID, so if you are mounting using UUIDs, you must change it
everywhere, but other than that the new filesystem should have the same content
as the old one (including subvolumes).
> I can use
> cp, rsync, or send/receive to copy the "main" subvolumes. But unless I'm
> missing something obvious, the snapshots are effectively lost. btrfs
> send optionally takes multiple clone sources, but I've never seen an
> example of its usage.
>
> If that's what "experimental" means, I'm willing to accept it. However,
> I'd like to emphasize that there's still something missing. Of course,
> most of all I'd like to be proved wrong.
>
> Michael
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 15:51 ` Michael Schuerig
2014-04-10 16:01 ` George Eleftheriou
@ 2014-04-10 18:15 ` Martin Steigerwald
1 sibling, 0 replies; 15+ messages in thread
From: Martin Steigerwald @ 2014-04-10 18:15 UTC (permalink / raw)
To: Michael Schuerig; +Cc: linux-btrfs
Am Donnerstag, 10. April 2014, 17:51:26 schrieb Michael Schuerig:
> On Thursday 10 April 2014 15:15:02 Duncan wrote:
> > Meanwhile (2), given the existence of those tested backups, there's
> > yet another way to accomplish things. Simply restore from the
> > backups the same way you would if the working copy went down and you
> > had to restore it, only restore to the new device instead of the old
> > one. =:^)
>
> As the OP, let me insist that I have multiple backups. However and
> unfortunately, those backups do not contain the snapshots that I'd like
> to preserve when exchanging the disk. What makes the case complicate is
> not the question how to preserve and copy the current data; it's how to
> retain the historic data embodied in snapshots.
Well I think this is scriptable – although this would be some work.
Like this – pseudo shell code without verifying exact argument order:
- for each subvolume create according subvolume in destination
- for each snapshot – from oldest to newest:
- rsync / btrfs send snapshot
- snapshot with same name
- rsync next newer snapshot
I have snapshots on backup drive… so I´d probably ditch snapshots on
production side… but if you want them… thats another idea.
Although I like the balance idea as well. I used it to RAID-1 my /home to dual
SSD setup. But I didn´t to the degrading step.
Ciao,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 17:17 ` Jan Kouba
@ 2014-04-10 18:36 ` Michael Schuerig
2014-04-10 22:25 ` Jan Kouba
0 siblings, 1 reply; 15+ messages in thread
From: Michael Schuerig @ 2014-04-10 18:36 UTC (permalink / raw)
To: linux-btrfs
On Thursday 10 April 2014 19:17:13 Jan Kouba wrote:
> Dne Čt 10. dubna 2014 15:21:01, Michael Schuerig napsal(a):
> > SMART indicates that my notebook disk may soon be failing (an
> > unreadable/uncorrectable sector), therefore I intend to exchange it.
> > The disk contains a single btrfs filesystem with several nested(!)
> > subvolumes, each with several read-only snapshots in a .snapshots
> > subdirectory.
> >
> > As far as I can tell, btrfs currently does not offer a sensible way
> > to duplicate the entire contents of the old disk onto a new one.
> Yes it does
>
> You can make the old disk a seeding device and use it to seed the new
> one like this:
Intriguing. I didn't find much on the topic, but these pages seem
pertinent
http://en.wikipedia.org/wiki/Btrfs#Seed_devices
https://btrfs.wiki.kernel.org/index.php/Seed-device
http://unix.stackexchange.com/a/97819/64935
> btrfstune -S 1 <old_device>
Does the "1" mean "true" or is this indeed an integer-value arg? I've
only seen the "1" in examples, are there other possible values?
> mount <old_device> /mnt
> # this will be mounted read-only
>
> btrfs dev add <new_device> /mnt
>
> mount -o remount,rw /mnt
>
> btrfs dev delete <old_device>
Is the last one the step where data is copied from the seed device to
the second device? No balance required before deleting the device?
Thanks!
Michael
--
Michael Schuerig
mailto:michael@schuerig.de
http://www.schuerig.de/michael/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 18:36 ` Michael Schuerig
@ 2014-04-10 22:25 ` Jan Kouba
0 siblings, 0 replies; 15+ messages in thread
From: Jan Kouba @ 2014-04-10 22:25 UTC (permalink / raw)
To: linux-btrfs
Dne Čt 10. dubna 2014 20:36:32, Michael Schuerig napsal(a):
> On Thursday 10 April 2014 19:17:13 Jan Kouba wrote:
> > Dne Čt 10. dubna 2014 15:21:01, Michael Schuerig napsal(a):
> > > SMART indicates that my notebook disk may soon be failing (an
> > > unreadable/uncorrectable sector), therefore I intend to exchange it.
> > > The disk contains a single btrfs filesystem with several nested(!)
> > > subvolumes, each with several read-only snapshots in a .snapshots
> > > subdirectory.
> > >
> > > As far as I can tell, btrfs currently does not offer a sensible way
> > > to duplicate the entire contents of the old disk onto a new one.
> >
> > Yes it does
> >
> > You can make the old disk a seeding device and use it to seed the new
>
> > one like this:
> Intriguing. I didn't find much on the topic, but these pages seem
> pertinent
>
> http://en.wikipedia.org/wiki/Btrfs#Seed_devices
> https://btrfs.wiki.kernel.org/index.php/Seed-device
> http://unix.stackexchange.com/a/97819/64935
>
> > btrfstune -S 1 <old_device>
>
> Does the "1" mean "true" or is this indeed an integer-value arg? I've
> only seen the "1" in examples, are there other possible values?
Yes, "1" means "true", "0" false. If you run it twice with the same argument,
it says something like "Seeding flag already set/unset.".
>
> > mount <old_device> /mnt
> > # this will be mounted read-only
> >
> > btrfs dev add <new_device> /mnt
> >
> > mount -o remount,rw /mnt
> >
> > btrfs dev delete <old_device>
>
> Is the last one the step where data is copied from the seed device to
> the second device? No balance required before deleting the device?
Yes, everything is copied, no balance needed.
Well actually if you don't need to preserve the filesystem on the old device,
you can just add the new device into the filesystem and remove the old one like
this:
mount <old_device> /mnt
btrfs dev add <new_device> /mnt
btrfs dev delete <old_device> /mnt
The last command will copy everything from <old_device> to the <new_device>
and somehow wipe the <old_device> so it can't be mounted anymore.
This method will also preserve the UUID of the filesystem so there is no need
to change any config files.
Jan Kouba
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 16:24 ` Michael Schuerig
@ 2014-04-11 0:35 ` Duncan
0 siblings, 0 replies; 15+ messages in thread
From: Duncan @ 2014-04-11 0:35 UTC (permalink / raw)
To: linux-btrfs
Michael Schuerig posted on Thu, 10 Apr 2014 18:24:00 +0200 as excerpted:
> I *think* send/receive with clone sources might be the key to a
> solution. I'm still hoping that someone with a far better understanding
> of btrfs than me gives it a try first...
Well, there's both that (which I don't know enough about to discuss in
any intelligent way at all), and the fact that at least until very (very)
recently (like recent enough I'm not sure it'll all be in 2.15 yet),
btrfs send/receive still had some serious corner-case bugs, such that to
my knowledge if it worked without error it was reliable, there were
numbers of cases (like where it was originally subdir /a/b/, and then
switched to /b/a, reversing the nesting) where it would spit out errors,
leaving the receive in a half-received state.
So it worked for some people but not others, and for some it would work
for awhile, then would start erroring out, whenever they hit whatever
corner-case.
Regulars on this list will have seen all the recent work going into send/
receive bug tracing and fixing, however, which will of course ultimately
make it far more reliable than it had been. Surely it's already far more
reliable, as a fair number of those corner cases have now been found and
dealt with.
But what I do NOT know yet, is where in the process we are, in terms of
/reliable/ send/receive. Relatively, we're certainly closer than we
were, but did that move use from say 80% to 90% reliable, or from 80% to
97 or 98% reliable (which is about where I'd put general btrfs at this
point), or from 90% to 98-ish% reliable, or... ?
Without that knowledge, even if you did know the specific CL options you
needed, I'd still consider it very much a "great if it works, but don't
count on it until you actually see it complete without errors, and even
then, be aware that the next time you try it could error out, because
it's still very much in bug-fixing-phase" solution.
Meanwhile, the ddrescue solution is mature technology, proven to work
with many different types of filesystems over many years, now, so that's
the basket I'd be putting my resource eggs into at this point. =:^)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-10 13:21 Copying a disk containing a btrfs filesystem Michael Schuerig
` (2 preceding siblings ...)
2014-04-10 17:17 ` Jan Kouba
@ 2014-04-11 10:39 ` Brendan Hide
2014-04-16 11:12 ` Michael Schuerig
3 siblings, 1 reply; 15+ messages in thread
From: Brendan Hide @ 2014-04-11 10:39 UTC (permalink / raw)
To: Michael Schuerig, linux-btrfs
Hi, Michael
Btrfs send/receive can transfer incremental snapshots as well - you're
looking for the "-p" or "parent" parameter. On the other hand, it might
not be the right tool for the job.
If you're 100% happy with your old disk's *content*/layout/etc (just not
happy with the disk's reliability), try an overnight/over-weekend
ddrescue instead:
http://www.forensicswiki.org/wiki/Ddrescue
What I've done in the past is scripted ddrescue to recover as much data
as possible. Its like using dd between two disks except that it can keep
a log of bad sectors that can be retried later. The log also helps by
ensuring that if you cancel the operation, you can start it again and it
will continue where it left off.
Additionally, you can have it skip big sections of the disk when it
comes across bad sectors - and it can "trim" these sections on
subsequent runs.
Btrfs send/receive has the advantage that you can run it while your
system is still active. DDrescue has the advantage that it is very good
at recovering 99% of your data where a disk has lots of bad sectors.
For btrfs, send/receive the main subvolumes then, afterward,
send/receive the snapshots using the "parent" parameter, "-p". There
*is* the possibility that this needs to be reversed - as in, the backup
should be treated as the parent instead of the other way around:
btrfs send /home | btrfs receive /mnt/new-disk/home
btrfs send -p /home /backups/home-2014-04-08 | btrfs receive
/mnt/new-disk/backups/.
____
Below is from the last scriptlet I made when I last did the ddrescue
route (in that case I was recovering a failing NTFS drive). It was
particularly bad and took a whole weekend to recover. The new disk
worked flawlessly however. :)
How I've used ddrescue in the past is to connect the failing and new
disk to a server. Alternatively, using USB, you could boot from a rescue
CD/flash drive and do the rescue there.
a) Identify the disks in /dev/disk/by-<whatever-you-choose> and put
those values into the bash script below. ensure that it refers to the
disk as a whole (not a partition for example). This ensures that
re-ordering of the drives after a reboot won't affect the process.
b) Set up a log file location on a separate filesystem - a flash drive
is ideal unless you've gone the "server" route, where I normally just
put the log into a path on the the server as so:
/root/brendan-laptop.recovery.log
#!/bin/bash
src_disk=/dev/disk/by-id/ata-ST3250410AS_6RYF5NP7
dst_disk=/dev/disk/by-id/ata-ST3500418AS_9VM2ZZQS
#log=/path/to/log
log=~/brendan-laptop.recovery.log
#Sector size (default is 512 - newer disks should probably be 4096)
sector_size=4096
#Force writing to a block device - disable if you're backing up to an
image file
#force=""
force="-f"
# We want to skip bigger chunks to get as much data as possible before
the source disk really dies
# For the same reason, we also want to start with (attempting) to
maintain a high read rate
#Minimum read rate in MB/s before skipping
min_readrate=10485760
#Default skip size is 64k.
for skip_size in 65536 16384; do
#Going forward
ddrescue -r 1 -a $min_readrate -b $sector_size -d $force -K $skip_size
$src_disk $dst_disk $log
#Going in reverse
ddrescue -r 1 -a $min_readrate -b $sector_size -d $force -K $skip_size
-R $src_disk $dst_disk $log
done
#Default skip size is 64k.
#This time re-trying all failed/skipped sections
for skip_size in 4096; do
#Going forward
ddrescue -r 1 -a $min_readrate -b $sector_size -d $force -K $skip_size
-A $src_disk $dst_disk $log
#Going in reverse
ddrescue -r 1 -a $min_readrate -b $sector_size -d $force -K $skip_size
-R -A $src_disk $dst_disk $log
done
#Default skip size is 64k.
for skip_size in 1024 256 64 ; do
#Going forward
ddrescue -r 1 -b $sector_size -d $force -K $skip_size $src_disk
$dst_disk $log
#Going in reverse
ddrescue -r 1 -b $sector_size -d $force -K $skip_size -R $src_disk
$dst_disk $log
done
echo "Done. Run an chkdsk/fsck/whatever-might-be appropriate for the new
disk's filesystem(s)"
On 10/04/14 15:21, Michael Schuerig wrote:
> SMART indicates that my notebook disk may soon be failing (an
> unreadable/uncorrectable sector), therefore I intend to exchange it. The
> disk contains a single btrfs filesystem with several nested(!)
> subvolumes, each with several read-only snapshots in a .snapshots
> subdirectory.
>
> As far as I can tell, btrfs currently does not offer a sensible way to
> duplicate the entire contents of the old disk onto a new one. I can use
> cp, rsync, or send/receive to copy the "main" subvolumes. But unless I'm
> missing something obvious, the snapshots are effectively lost. btrfs
> send optionally takes multiple clone sources, but I've never seen an
> example of its usage.
>
> If that's what "experimental" means, I'm willing to accept it. However,
> I'd like to emphasize that there's still something missing. Of course,
> most of all I'd like to be proved wrong.
>
> Michael
>
--
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Copying a disk containing a btrfs filesystem
2014-04-11 10:39 ` Brendan Hide
@ 2014-04-16 11:12 ` Michael Schuerig
0 siblings, 0 replies; 15+ messages in thread
From: Michael Schuerig @ 2014-04-16 11:12 UTC (permalink / raw)
To: linux-btrfs
On Friday 11 April 2014 12:39:31 Brendan Hide wrote:
> If you're 100% happy with your old disk's *content*/layout/etc (just
> not happy with the disk's reliability), try an
> overnight/over-weekend ddrescue instead
Thanks again to everyone who replied and especially for suggesting
ddrescue. In the meantime, I've got a suitable new disk and copied the
data to it using ddrescue (it took about 12 hours for 1TB). The disk had
one fault extending over a few successive sectors.
For some reason, I had to repair the first partition, which I use for
swap. The large second with btrfs on it seems fine. I ran btrfs scrub on
it and it found one error:
Apr 16 09:48:27 fuchsia kernel: [ 6792.829186] btrfs: checksum error at
logical 74443923456 on dev /dev/sdb2, sector 145398288, root 1228, inode
59102093, offset 929792, length 4096, links 1 (path:
usr/lib/debug/.build-id/8f/b82df57b7b6fff7033f6abd7de914b82f98160.debug)
That file is part of a "historical" snapshot which I have deleted by now
and thus presumably dealt with the problem.
Michael
--
Michael Schuerig
mailto:michael@schuerig.de
http://www.schuerig.de/michael/
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2014-04-16 11:12 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-04-10 13:21 Copying a disk containing a btrfs filesystem Michael Schuerig
2014-04-10 13:58 ` Duncan
2014-04-10 14:53 ` Michael Schuerig
2014-04-10 14:00 ` George Eleftheriou
2014-04-10 15:15 ` Duncan
2014-04-10 15:51 ` Michael Schuerig
2014-04-10 16:01 ` George Eleftheriou
2014-04-10 16:24 ` Michael Schuerig
2014-04-11 0:35 ` Duncan
2014-04-10 18:15 ` Martin Steigerwald
2014-04-10 17:17 ` Jan Kouba
2014-04-10 18:36 ` Michael Schuerig
2014-04-10 22:25 ` Jan Kouba
2014-04-11 10:39 ` Brendan Hide
2014-04-16 11:12 ` Michael Schuerig
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.