[linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Stirling Westrup]

On Sun, Nov 14, 2010 at 6:52 PM, Stirling Westrup <swestrup@gmail.com> wrote:
>>
> Thanks for going through all those steps. It does make the procedure a
> lot clearer in my mind, and it does look like dd_rescue is the way to
> go then. I'm going to head off to try it now.
>

Okay, I've tried the dd_rescue method that was outlined for me, and it
failed, although not for any reasons inherent in the method. It seems
that what is wrong with my 'flakey' drive is not that it has bad
sectors, but that it has a tendency to heat up when used, and then
fail all operations until its cooled down.

So, I went out and managed to borrow a sata card for my server so that
I could hook up all five drives at once, and actually have an active
system while working on it, and now I would like to pvmove all of the
PE's from the old flakey drive to the replacement.

pvmove typically reports getting somewhere around 7% done before the
drive fails, but I would like to know what that represents in terms of
checkpointed data. The man pages are frustratingly vague on a large
number of points:

1) how do you get a list of PEs on a PV?
2) how often are checkpoints made, and can you control that in any way?
3) can you request a given number of PEs to be moved? (I googled and
found someone who claimed to do that in a similar situation, but I
could find no further details).
4) the man page for pvmove says that you can reference a physical
volume with PhysicalVolume[:PE[-PE]..., but it doesn't say what those
suffixes mean, nor could I find any man page which explained it.

Basically, I want to attempt to optimize my uses of pvmove to transfer
as much as possible in as few attempts as possible. Any help would be
appreciated.





-- 
Stirling Westrup
Programmer, Entrepreneur.
https://www.linkedin.com/e/fpf/77228
http://www.linkedin.com/in/swestrup
http://technaut.livejournal.com

Re: [linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Lars Ellenberg]

On Mon, Nov 15, 2010 at 02:53:29PM -0500, Stirling Westrup wrote:
> On Sun, Nov 14, 2010 at 6:52 PM, Stirling Westrup <swestrup@gmail.com> wrote:
> >>
> > Thanks for going through all those steps. It does make the procedure a
> > lot clearer in my mind, and it does look like dd_rescue is the way to
> > go then. I'm going to head off to try it now.
> >
> 
> Okay, I've tried the dd_rescue method that was outlined for me, and it
> failed, although not for any reasons inherent in the method. It seems
> that what is wrong with my 'flakey' drive is not that it has bad
> sectors, but that it has a tendency to heat up when used, and then
> fail all operations until its cooled down.

You can hook up your old drive to the external sata,
and point a fan right at it,
or even use a long eSATA cable and put it in the fridge.  No joke, this
has been done to successfully recover data from failing drives.

> So, I went out and managed to borrow a sata card for my server so that
> I could hook up all five drives at once, and actually have an active
> system while working on it, and now I would like to pvmove all of the
> PE's from the old flakey drive to the replacement.
> 
> pvmove typically reports getting somewhere around 7% done before the
> drive fails, but I would like to know what that represents in terms of
> checkpointed data. The man pages are frustratingly vague on a large
> number of points:
> 
> 1) how do you get a list of PEs on a PV?

I find useful:
# lvs -o +seg_pe_ranges

or, if your lvs does not support that yet,
# lvs --unit 64m --segments -o +devices

use your PE size with lower case unit letter for --units
to have the segment size reported in PE

You may want to also add -O ... to sort the output.


> 2) how often are checkpoints made, and can you control that in any way?

IIRC, pvmove does one PE at a time,
and will checkpoint each of these.
Depending on wether or not you set the PE size explicitly on vgcreate
time, this frequent checkpointing every few MB may slow down things.

> 3) can you request a given number of PEs to be moved? (I googled and
> found someone who claimed to do that in a similar situation, but I
> could find no further details).
> 4) the man page for pvmove says that you can reference a physical
> volume with PhysicalVolume[:PE[-PE]..., but it doesn't say what those
> suffixes mean, nor could I find any man page which explained it.

man page synopsis says:
pvmove  [--abort]  [--alloc  AllocationPolicy]  [-b|--background]
	[-d|--debug]  [-h|--help]  [-i|--interval  Seconds] [--noudevsync]
	[-v|--verbose] [-n|--name LogicalVolume]
	[SourcePhysicalVolume[:PE[-PE]...]
	[DestinationPhysicalVolume[:PE[-PE]...]...]]

So: yes, you can. Like in
pvmove /dev/sda:7-9 /dev/sde:7-9

Ranges given are inclusive, so the above moves the three physical
extents 7,8,9 of PV sda to PV sde. Keep that in mind if you deduce
these ranges from segment start PE number and segment size in PE units:

Both PVs have to be part of the same VG already,
unallocated PEs on source are skipped,
allocated PEs on dest are ignored,
resulting to-be-moved number source PEs must not exceed
resulting available number of dest PEs.

If you give it too much rope, and the destination range contains
allocated blocks, the result may not always match your expectations,
as it usually tries to keep things continguous, even if you don't want
it to, to if you know exactly what you want, don't give it any rope ;-)
(if you are "defragmenting" the physical layout, e.g.)

If you want to pvmove within the same PV, you have to add
"--alloc anywhere".

Does that make sense?

> Basically, I want to attempt to optimize my uses of pvmove to transfer
> as much as possible in as few attempts as possible. Any help would be
> appreciated.

hth,

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD� and LINBIT� are registered trademarks of LINBIT, Austria.

Re: [linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Stirling Westrup]

On Mon, Nov 15, 2010 at 4:47 PM, Lars Ellenberg
<lars.ellenberg@linbit.com> wrote:
> On Mon, Nov 15, 2010 at 02:53:29PM -0500, Stirling Westrup wrote:
>> On Sun, Nov 14, 2010 at 6:52 PM, Stirling Westrup <swestrup@gmail.com> wrote:
>> >>
>> > Thanks for going through all those steps. It does make the procedure a
>> > lot clearer in my mind, and it does look like dd_rescue is the way to
>> > go then. I'm going to head off to try it now.
>> >
>>
>> Okay, I've tried the dd_rescue method that was outlined for me, and it
>> failed, although not for any reasons inherent in the method. It seems
>> that what is wrong with my 'flakey' drive is not that it has bad
>> sectors, but that it has a tendency to heat up when used, and then
>> fail all operations until its cooled down.
>
> You can hook up your old drive to the external sata,
> and point a fan right at it,
> or even use a long eSATA cable and put it in the fridge. �No joke, this
> has been done to successfully recover data from failing drives.

I don't have a cable long enough to try the fridge trick, but I'm
going to try a fan, and maybe rest it on a metal container full of
ice. If it works, this might make the rest of my questions a bit
redundant, but still...

> I find useful:
> # lvs -o +seg_pe_ranges
>
Aha! Thanks. That's just what I was hoping for. Again, I was looking
for a pv_ option, not an lg_ one, as I assumed that PEs were the
purview of the physical volume layer.


>> 2) how often are checkpoints made, and can you control that in any way?
>
> IIRC, pvmove does one PE at a time,
> and will checkpoint each of these.
> Depending on wether or not you set the PE size explicitly on vgcreate
> time, this frequent checkpointing every few MB may slow down things.

I didn't set my PE size explicitly, so I checked in /etc/lvm/backup
and it looks like I have 4mb extents. This should mean that I'm making
significant progress every time I attempt to continue the pvmove.

Now, is there any way to monitor that progress? Every time I issued a
bare 'pvmove' to continue the operation, it starts counting from 0%
again. I'm assuming (hoping!) this is a percentage of the amount LEFT
to move, not the total amount that needs to be moved, but I'd love
some way to verify this.

> pvmove /dev/sda:7-9 /dev/sde:7-9

Okay. What if I don't care where the PE's end up? Can I just say:

pvmove /dev/sda:1000-1500 /dev/sdb

and assume it will do something reasonable? I currently don't have any
fragmentation anywhere, so I would hope this would just work.

> Does that make sense?

Yes, it does. You've been extremely helpful so far.

-- 
Stirling Westrup
Programmer, Entrepreneur.
https://www.linkedin.com/e/fpf/77228
http://www.linkedin.com/in/swestrup
http://technaut.livejournal.com

Re: [linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Stuart D. Gathman]

On Mon, 15 Nov 2010, Stirling Westrup wrote:

> Okay. What if I don't care where the PE's end up? Can I just say:
> 
> pvmove /dev/sda:1000-1500 /dev/sdb
> 
> and assume it will do something reasonable? I currently don't have any
> fragmentation anywhere, so I would hope this would just work.

Use the test option to see what it would do without actually doing it.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Luca Berra]

On Mon, Nov 15, 2010 at 02:53:29PM -0500, Stirling Westrup wrote:
>Okay, I've tried the dd_rescue method that was outlined for me, and it
>failed, although not for any reasons inherent in the method. It seems
>that what is wrong with my 'flakey' drive is not that it has bad
>sectors, but that it has a tendency to heat up when used, and then
>fail all operations until its cooled down.


you can try with http://www.gnu.org/software/ddrescue/ddrescue.html
instead of dd_rescue
make it use a logfile on a different device
when the source drive gets too hot let it rest awhile, then restart
using the log file.

L.

-- 
Luca Berra -- bluca@comedia.it

Re: [linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Stirling Westrup]

Just wanted to thank everyone for their help and advice. With a
combination of cooling the drive with ice, and moving groups of 1000
extents at a time, I was able to completely recover all of my data
from the drive. As far as I can tell there was no data loss.

I do think the fact that pvmove does no user-visible checkpointing is
a bug. In fact, from the few tests that I ran in preparation for the
final successful recovery effort, it appears that pvmove does not do
any actual checkpointing whatsoever. Certainly none that my tests
could uncover.

In any event, I am extremely happy for a successful recovery, and
wanted to share that with the forum.

-- 
Stirling Westrup
Programmer, Entrepreneur.
https://www.linkedin.com/e/fpf/77228
http://www.linkedin.com/in/swestrup
http://technaut.livejournal.com

Re: [linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Alasdair G Kergon]

On Thu, Nov 18, 2010 at 01:30:11PM -0500, Stirling Westrup wrote:
> I do think the fact that pvmove does no user-visible checkpointing is
> a bug.

The man page explains how pvmove works.  No data is removed until it
is safely available in its new location.

Checkpoints are done after each contiguous piece of data to move.
So if your data is contiguous, you'll get no checkpoints.
The code was designed to handle more-frequent checkpoints in such
cases, but nobody's seen the need to implement that feature yet.
(I don't think we even have a bugzilla requesting it.)

Alasdair

Re: [linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Stirling Westrup]

On Thu, Nov 18, 2010 at 3:12 PM, Alasdair G Kergon <agk@redhat.com> wrote:
> On Thu, Nov 18, 2010 at 01:30:11PM -0500, Stirling Westrup wrote:
>> I do think the fact that pvmove does no user-visible checkpointing is
>> a bug.
>
> The man page explains how pvmove works. �No data is removed until it
> is safely available in its new location.

pmove's man page leaves a LOT to be desired. It gives the bare syntax
of saying how to move individual extents, or ranges thereof, but gives
no semantic information whatsoever. It doesn't even go so far as to
SAY that the syntax is for moving individual extents.

It mentions checkpointing, but gives no further information. It
certainly doesn't say anything about how often they are done, or under
what circumstances. Nor is there any mention of how one would know
when the last checkpoint was taken. I'm not the first person who's
asked about this either, as I discovered while trying to google for
information that should have been in the man page.

> Checkpoints are done after each contiguous piece of data to move.
> So if your data is contiguous, you'll get no checkpoints.
> The code was designed to handle more-frequent checkpoints in such
> cases, but nobody's seen the need to implement that feature yet.
> (I don't think we even have a bugzilla requesting it.)

Perhaps no one has made a request, because there isn't even enough
information available for someone to KNOW what to request?

-- 
Stirling Westrup
Programmer, Entrepreneur.
https://www.linkedin.com/e/fpf/77228
http://www.linkedin.com/in/swestrup
http://technaut.livejournal.com

Re: [linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Alasdair G Kergon]

On Thu, Nov 18, 2010 at 03:40:25PM -0500, Stirling Westrup wrote:
> It mentions checkpointing, but gives no further information. It
> certainly doesn't say anything about how often they are done, or under
> what circumstances. 

It's in there, but the man page would indeed benefit from more repetition
and examples.  Patches welcome!

       Every logical volume in the volume group is searched for *contiguous
       data* that need moving according to the  command  line  arguments.   For
       each  piece  of  data  found,  a new *segment* is added to the end of the
       pvmove LV.  This segment takes the form of a *temporary mirror*  to  copy
       the data from the original location to a newly-allocated location. 

       A daemon repeatedly checks progress at the specified time interval.
       When it detects that the first *temporary mirror* is in-sync,  it  breaks
       that  mirror  so that only the new location for that data gets used and
       writes a *checkpoint* into the volume group metadata on  disk. 

So checkpoint at first daemon check interval after temp mirror in-sync, and
one temp mirror per item of contiguous data to be moved.

Alasdair

Re: [linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Stirling Westrup]

On Thu, Nov 18, 2010 at 4:32 PM, Alasdair G Kergon <agk@redhat.com> wrote:
> On Thu, Nov 18, 2010 at 03:40:25PM -0500, Stirling Westrup wrote:
>> It mentions checkpointing, but gives no further information. It
>> certainly doesn't say anything about how often they are done, or under
>> what circumstances.
>
> It's in there, but the man page would indeed benefit from more repetition
> and examples. �Patches welcome!
>
> � � � Every logical volume in the volume group is searched for *contiguous
> � � � data* that need moving according to the �command �line �arguments. � For
> � � � each �piece �of �data �found, �a new *segment* is added to the end of the
> � � � pvmove LV. �This segment takes the form of a *temporary mirror* �to �copy
> � � � the data from the original location to a newly-allocated location.
>
> � � � A daemon repeatedly checks progress at the specified time interval.
> � � � When it detects that the first *temporary mirror* is in-sync, �it �breaks
> � � � that �mirror �so that only the new location for that data gets used and
> � � � writes a *checkpoint* into the volume group metadata on �disk.
>
> So checkpoint at first daemon check interval after temp mirror in-sync, and
> one temp mirror per item of contiguous data to be moved.

The interpretation you give is only possible if one knows what the
man-page writer means by 'contiguous data', 'segment' and 'temporary
mirror'. As it was, I didn't and spent some time perusing the man
pages to find anywhere where these terms are defined, with no luck.

So, yes, the man page definitely needs fixing. And that, perhaps,
should be done by someone who has a heck of a lot more familiarity
with the system than I have, or else the resulting man page is likely
to end up BOTH inadequate and erroneous.

Still, I submit that if the man page needs to talk about checkpoints
AT ALL, then there needs to be a user method for determining, the
status of checkpoints during a pvmove.
-- 
Stirling Westrup
Programmer, Entrepreneur.
https://www.linkedin.com/e/fpf/77228
http://www.linkedin.com/in/swestrup
http://technaut.livejournal.com

Re: [linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Stuart D. Gathman]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 951 bytes --]

On Thu, 18 Nov 2010, Stirling Westrup wrote:

> > The man page explains how pvmove works. �No data is removed until it
> > is safely available in its new location.
> 
> pmove's man page leaves a LOT to be desired. It gives the bare syntax

... lots of valid criticism

In the New Testament, speaking in tongues had to be accompanied by
an interpretation.  In Open Source, criticism (however valid) should
be accompanied by a contribution (in this case, an edited man page)!
Actually, I would be willing to do that, I'm up to the writing, but it will
take me a while to figure out where to check out the current man page source,
unless someone gives me a pointer.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Stirling Westrup]

On Fri, Nov 19, 2010 at 4:56 PM, Stuart D. Gathman <stuart@bmsi.com> wrote:
> On Thu, 18 Nov 2010, Stirling Westrup wrote:
>
>> > The man page explains how pvmove works. �No data is removed until it
>> > is safely available in its new location.
>>
>> pmove's man page leaves a LOT to be desired. It gives the bare syntax
>
> ... lots of valid criticism
>
> In the New Testament, speaking in tongues had to be accompanied by
> an interpretation. �In Open Source, criticism (however valid) should
> be accompanied by a contribution (in this case, an edited man page)!

While I don't necessarily disagree with the point above, there ARE
only so many hours in a day, and the vast majority of Open Source
software that I use has similar problems. In fact, most have far worse
documentation than does LVM. I refuse to let the fact that I cannot
single-handedly supply documentation to the entire open source
community deter me from pointing out deficiencies where they arise.

> Actually, I would be willing to do that, I'm up to the writing, but it will
> take me a while to figure out where to check out the current man page source,
> unless someone gives me a pointer.

I tip my hat to you, sir, for being willing to tackle this task!


-- 
Stirling Westrup
Programmer, Entrepreneur.
https://www.linkedin.com/e/fpf/77228
http://www.linkedin.com/in/swestrup
http://technaut.livejournal.com

Re: [linux-lvm] Move pvmove questions Was: Need help with a particular use-case for pvmove.

[Sven Eschenberg]

Look here:

http://sources.redhat.com/cgi-bin/cvsweb.cgi/LVM2/man/?cvsroot=lvm2

or maybe even here:

http://sourceware.org/lvm2/

Which has the necessary cvs calls to get a copy of the cvs tree, as far as
I can see.


On Fri, November 19, 2010 22:56, Stuart D. Gathman wrote:
> On Thu, 18 Nov 2010, Stirling Westrup wrote:
>
>> > The man page explains how pvmove works. �No data is removed until it
>> > is safely available in its new location.
>>
>> pmove's man page leaves a LOT to be desired. It gives the bare syntax
>
> ... lots of valid criticism
>
> In the New Testament, speaking in tongues had to be accompanied by
> an interpretation.  In Open Source, criticism (however valid) should
> be accompanied by a contribution (in this case, an edited man page)!
> Actually, I would be willing to do that, I'm up to the writing, but it
> will
> take me a while to figure out where to check out the current man page
> source,
> unless someone gives me a pointer.
>
> --
> 	      Stuart D. Gathman <stuart@bmsi.com>
>     Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703
> 591-6154
> "Confutatis maledictis, flammis acribus addictis" - background song for
> a Microsoft sponsored "Where do you want to go from here?"
> commercial._______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Move pvmove questions

[Stuart D Gathman]

I see the man page has been expanded into steps already.  I added what I
think are helpful amplifications.  What do
you think?  See the original man page reader complaint below.

diff -u -r1.3 pvmove.8.in
--- pvmove.8.in    4 Aug 2009 08:09:52 -0000    1.3
+++ pvmove.8.in    18 Jan 2011 18:30:16 -0000
@@ -40,14 +40,18 @@
 details of all the data movements required.
 
 2. Every logical volume in the volume group is searched
-for contiguous data that need moving
-according to the command line arguments.
-For each piece of data found, a new segment is added to the end of the
-pvmove LV.
+for PEs that need moving according to the command line arguments.  These PEs
+are divided into "segments", runs of physically contiguous data where the PEs
+are adjacent on a PV. For each such run of data found, a new segment is added
+to the end of the pvmove LV.
 This segment takes the form of a temporary mirror to copy the data 
-from the original location to a newly-allocated location. 
+from the original location to a newly-allocated location.  A temporary
+mirror is marked as such in the on disk metadata so that system knows now to 
+finish or undo the pvmove in the event it is interrupted by a system crash.
 The original LV is updated to use the new temporary mirror segment
-in the pvmove LV instead of accessing the data directly.
+in the pvmove LV instead of accessing the data directly, so that any writes
+to the original LV are written to both segments until the temporary mirror
+is syncronized.
 
 3. The volume group metadata is updated on disk.
 
@@ -58,7 +62,8 @@
 5. A daemon repeatedly checks progress at the specified time interval.
 When it detects that the first temporary mirror is in-sync,
 it breaks that mirror so that only the new location for that data gets used 
-and writes a checkpoint into the volume group metadata on disk.
+and writes a checkpoint into the volume group metadata on disk, so that
+the step isn't repeated in the event of a system crash.
 Then it activates the mirror for the next segment of the pvmove LV.
 
 6. When there are no more segments left to be mirrored, 


On 11/18/2010 05:06 PM, Stirling Westrup wrote:
> On Thu, Nov 18, 2010 at 4:32 PM, Alasdair G Kergon <agk@redhat.com> wrote:
>> On Thu, Nov 18, 2010 at 03:40:25PM -0500, Stirling Westrup wrote:
>>> It mentions checkpointing, but gives no further information. It
>>> certainly doesn't say anything about how often they are done, or under
>>> what circumstances.
>> It's in there, but the man page would indeed benefit from more repetition
>> and examples.  Patches welcome!
>>
>>       Every logical volume in the volume group is searched for *contiguous
>>       data* that need moving according to the  command  line  arguments.   For
>>       each  piece  of  data  found,  a new *segment* is added to the end of the
>>       pvmove LV.  This segment takes the form of a *temporary mirror*  to  copy
>>       the data from the original location to a newly-allocated location.
>>
>>       A daemon repeatedly checks progress at the specified time interval.
>>       When it detects that the first *temporary mirror* is in-sync,  it  breaks
>>       that  mirror  so that only the new location for that data gets used and
>>       writes a *checkpoint* into the volume group metadata on  disk.
>>
>> So checkpoint at first daemon check interval after temp mirror in-sync, and
>> one temp mirror per item of contiguous data to be moved.
> The interpretation you give is only possible if one knows what the
> man-page writer means by 'contiguous data', 'segment' and 'temporary
> mirror'. As it was, I didn't and spent some time perusing the man
> pages to find anywhere where these terms are defined, with no luck.
>
> So, yes, the man page definitely needs fixing. And that, perhaps,
> should be done by someone who has a heck of a lot more familiarity
> with the system than I have, or else the resulting man page is likely
> to end up BOTH inadequate and erroneous.
>
> Still, I submit that if the man page needs to talk about checkpoints
> AT ALL, then there needs to be a user method for determining, the
> status of checkpoints during a pvmove.