[linux-lvm] LVM snapshot setup for mysql-zrm backups problem

[linux-lvm] LVM snapshot setup for mysql-zrm backups problem

[Philip Amadeo Saeli]

I'm not sure where is the best place to ask this so, since I am using
LVM, I am asking here in the hope that someone can at least direct me to
the appropriate place to ask this.  This is my first time setting up
LVM snapshots for mysql-zrm backups.

CentOS-6.6
MySQL-5.6.22
MySQL-zrm-3.0 (EPEL).

Short version:

I am unable to tell if the MySQL-zrm LVM snapshots are indeed taking
place, or if the backup is being taken directly from the main database
LVs.  The log file info isn't giving me enough info to tell one way or
another.

Some more info:

I am setting up a database server with all of the database files on LVM
LVs and am using mysql-zrm for logical backups using LVM snapshots to
reduce the database lock times.

I have followed all of the various setup instructions very closely, and
am using the default "lvm-snapshot.pl" plugin.  However, the only log
file lines mentioning snapshot are the following:

  mysql:backup:INFO: Running snapshot verification using command /usr/share/mysql-zrm/plugins/lvm-snapshot.pl --action verify-config 2>/home/tmp/xH8KKpsF2O

There are no other log entries following this one that seem to have
anything to do with snapshots.  There are no errors.  The backups
succeed.  Both "--verbose" and "--noquiet" are enabled.

Here are the salient entries from the mysql-zrm.conf file:

backup-level=0
backup-mode=logical
lvm-snapshot=20G
snapshot-plugin="/usr/share/mysql-zrm/plugins/lvm-snapshot.pl"
backup-type=regular
destination=/home/mysql-zrm
retention-policy=4W
compress=1

Any info or help would be much appreciated!

Thanks,

--Phil

-- 
Philip Amadeo Saeli
psaeli@zorodyne.com

Re: [linux-lvm] LVM snapshot setup for mysql-zrm backups problem

[Marian Csontos]

On 01/28/2015 06:46 AM, Philip Amadeo Saeli wrote:
> I'm not sure where is the best place to ask this so, since I am using

MySQL list, perhaps?

What does `lvs` say? Is there any snapshot of the LV where your database 
lives? Copy if to backup destination and try to check the content - I 
presume there is a tool to check the backup...

Just a word of warning:

if the script is using old-style snapshots, you want to copy and delete 
it ASAP, or you will see severely degraded performance, and much worse 
once you have multiple snapshots in the system.

Also once the space allocated for snapshot gets full, the snapshot 
becomes invalid, so hurry up!

-- Martian

> LVM, I am asking here in the hope that someone can at least direct me to
> the appropriate place to ask this.  This is my first time setting up
> LVM snapshots for mysql-zrm backups.
>
> CentOS-6.6
> MySQL-5.6.22
> MySQL-zrm-3.0 (EPEL).
>
> Short version:
>
> I am unable to tell if the MySQL-zrm LVM snapshots are indeed taking
> place, or if the backup is being taken directly from the main database
> LVs.  The log file info isn't giving me enough info to tell one way or
> another.
>
> Some more info:
>
> I am setting up a database server with all of the database files on LVM
> LVs and am using mysql-zrm for logical backups using LVM snapshots to
> reduce the database lock times.
>
> I have followed all of the various setup instructions very closely, and
> am using the default "lvm-snapshot.pl" plugin.  However, the only log
> file lines mentioning snapshot are the following:
>
>    mysql:backup:INFO: Running snapshot verification using command /usr/share/mysql-zrm/plugins/lvm-snapshot.pl --action verify-config 2>/home/tmp/xH8KKpsF2O
>
> There are no other log entries following this one that seem to have
> anything to do with snapshots.  There are no errors.  The backups
> succeed.  Both "--verbose" and "--noquiet" are enabled.
>
> Here are the salient entries from the mysql-zrm.conf file:
>
> backup-level=0
> backup-mode=logical
> lvm-snapshot=20G
> snapshot-plugin="/usr/share/mysql-zrm/plugins/lvm-snapshot.pl"
> backup-type=regular
> destination=/home/mysql-zrm
> retention-policy=4W
> compress=1
>
> Any info or help would be much appreciated!
>
> Thanks,
>
> --Phil
>

Re: [linux-lvm] LVM snapshot setup for mysql-zrm backups problem

[Philip Amadeo Saeli]

Thanks for the response.  More info and questions following.

* Marian Csontos <mcsontos@redhat.com> [2015-01-28 04:53]:
> On 01/28/2015 06:46 AM, Philip Amadeo Saeli wrote:

> What does `lvs` say? Is there any snapshot of the LV where your
> database lives? Copy if to backup destination and try to check the
> content - I presume there is a tool to check the backup...

The backups appear to be OK.  However, my suspicion is that they're
being made from the main database LVs rather than from a freshly-made
snapshot.

I am able to manually make a snapshot LV and it looks OK.  I have also
upped the LVM log level so I can better monitor any LVM errors or
warnings.

However, I have another question:

When I run any of the various LVM cmds with the "-v" flag, I get a
"DEGRADED MODE. Incomplete RAID LVs will be processed." message in the
output.  I find this surprising since none of the LVM vols on that
system are RAID vols.  They are simply single PV vols created for the
sole purpose of enabling snapshots for consistent point-in-time database
backups.  Any ideas why it is printing that?

> Just a word of warning:
> 
> if the script is using old-style snapshots, you want to copy and
> delete it ASAP, or you will see severely degraded performance, and
> much worse once you have multiple snapshots in the system.

What do you mean by "old-style" snapshots?  Does that mean non-thin
snapshots?  Or something else?  What type of degraded performance?  Any
references?

Note that the snapshots I'm using are ephemeral and are to exist only
long enough for the actual database backup operation to complete.

> Also once the space allocated for snapshot gets full, the snapshot
> becomes invalid, so hurry up!
> 
> -- Martian

Thanks!

--Phil

> >the appropriate place to ask this.  This is my first time setting up
> >LVM snapshots for mysql-zrm backups.
> >
> >CentOS-6.6
> >MySQL-5.6.22
> >MySQL-zrm-3.0 (EPEL).
> >
> >Short version:
> >
> >I am unable to tell if the MySQL-zrm LVM snapshots are indeed taking
> >place, or if the backup is being taken directly from the main database
> >LVs.  The log file info isn't giving me enough info to tell one way or
> >another.
> >
> >Some more info:
> >
> >I am setting up a database server with all of the database files on LVM
> >LVs and am using mysql-zrm for logical backups using LVM snapshots to
> >reduce the database lock times.
> >
> >I have followed all of the various setup instructions very closely, and
> >am using the default "lvm-snapshot.pl" plugin.  However, the only log
> >file lines mentioning snapshot are the following:
> >
> >   mysql:backup:INFO: Running snapshot verification using command /usr/share/mysql-zrm/plugins/lvm-snapshot.pl --action verify-config 2>/home/tmp/xH8KKpsF2O
> >
> >There are no other log entries following this one that seem to have
> >anything to do with snapshots.  There are no errors.  The backups
> >succeed.  Both "--verbose" and "--noquiet" are enabled.
> >
> >Here are the salient entries from the mysql-zrm.conf file:
> >
> >backup-level=0
> >backup-mode=logical
> >lvm-snapshot=20G
> >snapshot-plugin="/usr/share/mysql-zrm/plugins/lvm-snapshot.pl"
> >backup-type=regular
> >destination=/home/mysql-zrm
> >retention-policy=4W
> >compress=1
> >
> >Any info or help would be much appreciated!
> >
> >Thanks,
> >
> >--Phil

-- 
Philip Amadeo Saeli
psaeli@zorodyne.com

Re: [linux-lvm] LVM snapshot setup for mysql-zrm backups problem

[Marian Csontos]

On 01/29/2015 01:18 AM, Philip Amadeo Saeli wrote:
> Thanks for the response.  More info and questions following.
>
> * Marian Csontos <mcsontos@redhat.com> [2015-01-28 04:53]:
>> On 01/28/2015 06:46 AM, Philip Amadeo Saeli wrote:
>
>> What does `lvs` say? Is there any snapshot of the LV where your
>> database lives? Copy if to backup destination and try to check the
>> content - I presume there is a tool to check the backup...
>
> The backups appear to be OK.  However, my suspicion is that they're
> being made from the main database LVs rather than from a freshly-made
> snapshot.

That's exactly question for mysql-list.

You could monitor the snapshot with for example iotop or something to 
see if it is being read from but...

>
> I am able to manually make a snapshot LV and it looks OK.  I have also
> upped the LVM log level so I can better monitor any LVM errors or
> warnings.
>
> However, I have another question:
>
> When I run any of the various LVM cmds with the "-v" flag, I get a
> "DEGRADED MODE. Incomplete RAID LVs will be processed." message in the
> output.  I find this surprising since none of the LVM vols on that
> system are RAID vols.  They are simply single PV vols created for the
> sole purpose of enabling snapshots for consistent point-in-time database
> backups.  Any ideas why it is printing that?

Post the output of the command with -vvvv, please.
And may be lvmdump.

>
>> Just a word of warning:
>>
>> if the script is using old-style snapshots, you want to copy and
>> delete it ASAP, or you will see severely degraded performance, and
>> much worse once you have multiple snapshots in the system.
>
> What do you mean by "old-style" snapshots?  Does that mean non-thin
> snapshots?  Or something else?

Yes, non-thin snapshots.

 > What type of degraded performance?  Any
> references?

For example I were unable to squeeze more than few hundred IOPS on 
rather high-end hardware from old-snapshots (but it was on RHEL6 kernel 
missing few recent performance optimizations.)

And the more snapshots of the origin LV you have the write performance 
is divided by the number of them as the original value must be written 
to each of them. (And there is no optimization for that. So remember to 
remove the snapshots early.)

HTH

Marian

>
> Note that the snapshots I'm using are ephemeral and are to exist only
> long enough for the actual database backup operation to complete.
>
>> Also once the space allocated for snapshot gets full, the snapshot
>> becomes invalid, so hurry up!
>>
>> -- Martian
>
> Thanks!
>
> --Phil
>
>>> the appropriate place to ask this.  This is my first time setting up
>>> LVM snapshots for mysql-zrm backups.
>>>
>>> CentOS-6.6
>>> MySQL-5.6.22
>>> MySQL-zrm-3.0 (EPEL).
>>>
>>> Short version:
>>>
>>> I am unable to tell if the MySQL-zrm LVM snapshots are indeed taking
>>> place, or if the backup is being taken directly from the main database
>>> LVs.  The log file info isn't giving me enough info to tell one way or
>>> another.
>>>
>>> Some more info:
>>>
>>> I am setting up a database server with all of the database files on LVM
>>> LVs and am using mysql-zrm for logical backups using LVM snapshots to
>>> reduce the database lock times.
>>>
>>> I have followed all of the various setup instructions very closely, and
>>> am using the default "lvm-snapshot.pl" plugin.  However, the only log
>>> file lines mentioning snapshot are the following:
>>>
>>>    mysql:backup:INFO: Running snapshot verification using command /usr/share/mysql-zrm/plugins/lvm-snapshot.pl --action verify-config 2>/home/tmp/xH8KKpsF2O
>>>
>>> There are no other log entries following this one that seem to have
>>> anything to do with snapshots.  There are no errors.  The backups
>>> succeed.  Both "--verbose" and "--noquiet" are enabled.
>>>
>>> Here are the salient entries from the mysql-zrm.conf file:
>>>
>>> backup-level=0
>>> backup-mode=logical
>>> lvm-snapshot=20G
>>> snapshot-plugin="/usr/share/mysql-zrm/plugins/lvm-snapshot.pl"
>>> backup-type=regular
>>> destination=/home/mysql-zrm
>>> retention-policy=4W
>>> compress=1
>>>
>>> Any info or help would be much appreciated!
>>>
>>> Thanks,
>>>
>>> --Phil
>