[linux-lvm] e2fsadm and snapshots?

[linux-lvm] e2fsadm and snapshots?

[Jim King]

I'm trying to set up snapshots on my LVM volumes sort of like the way
Network Appliance does it... with hourly rotating snapshots. The obvious
problem is that the snapshots do not dynamically size. This is a real
issue, because snapshots no longer protect us from someone (for example)
deleting a huge directory accidently.

To solve that I'm trying to figure a way to have them dynamically size
-- with a script that checks how full they are and grows them
automatically if they start to fill up.

Problem:
--------
If I do e2fsadm on a normal LVM volume it works, but if I do e2fsadm on
a snapshot I get something like:

  /sbin/e2fsck: Permission denied while trying to open
     /dev/data1/.mer_dev.hourly.3
  You must have r/w access to the filesystem or be root

Now I'm logged in as root, and the filesystem is unmounted. Ideas? I'm
suspecting it's because the snapshot is ro, but is there a way to work
around this?



In the long run it would be better to have them just dynamically size
without intervention, but obviously that's not a feature yet.


-- 
------------------------------------------------------
Jim King
Multimission Image Processing Lab  (MIPL) Engineering Group
Science Data Processing Systems Section,  Jet Propulsion Lab
James.King@jpl.nasa.gov  
------------------------------------------------------

Re: [linux-lvm] e2fsadm and snapshots?

[Andreas Dilger]

On Oct 31, 2002  14:57 -0800, Jim King wrote:
> I'm trying to set up snapshots on my LVM volumes sort of like the way
> Network Appliance does it... with hourly rotating snapshots. The obvious
> problem is that the snapshots do not dynamically size. This is a real
> issue, because snapshots no longer protect us from someone (for example)
> deleting a huge directory accidently.
> 
> To solve that I'm trying to figure a way to have them dynamically size
> -- with a script that checks how full they are and grows them
> automatically if they start to fill up.
> 
> Problem:
> --------
> If I do e2fsadm on a normal LVM volume it works, but if I do e2fsadm on
> a snapshot I get something like:
> 
>   /sbin/e2fsck: Permission denied while trying to open
>      /dev/data1/.mer_dev.hourly.3
>   You must have r/w access to the filesystem or be root
> 
> Now I'm logged in as root, and the filesystem is unmounted. Ideas? I'm
> suspecting it's because the snapshot is ro, but is there a way to work
> around this?

The LVM snapshot is the whole filesystem (with only the parts that
actually change being written to the snapshot LV) - no need to change
the filesystem size at all.  All you need to change is the LV size.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: [linux-lvm] e2fsadm and snapshots?

[Jim King]

On Thu, 2002-10-31 at 15:25, Andreas Dilger wrote:
> The LVM snapshot is the whole filesystem (with only the parts that
> actually change being written to the snapshot LV) - no need to change
> the filesystem size at all.  All you need to change is the LV size.

Thanks -- that makes a lot of sense!


-- 
------------------------------------------------------
Jim King
Multimission Image Processing Lab  (MIPL) Engineering Group
Science Data Processing Systems Section,  Jet Propulsion Lab
James.King@jpl.nasa.gov  
------------------------------------------------------

Re: [linux-lvm] e2fsadm and snapshots?

[Jim King]

On Thu, 2002-10-31 at 15:25, Andreas Dilger wrote:
> The LVM snapshot is the whole filesystem (with only the parts that
> actually change being written to the snapshot LV) - no need to change
> the filesystem size at all.  All you need to change is the LV size.

OK -- so using lvextend I can extend it. But once it's extended, I can
no longer mount it. I get:

  mount: wrong fs type, bad option, bad superblock on
    /dev/data1/.mer_dev.hourly.3, or too many mounted file systems

Is there something I'm missing here? (ok -- obviously yes...)


-- 
------------------------------------------------------
Jim King
Multimission Image Processing Lab  (MIPL) Engineering Group
Science Data Processing Systems Section,  Jet Propulsion Lab
James.King@jpl.nasa.gov  
------------------------------------------------------

Re: [linux-lvm] e2fsadm and snapshots?

[Andreas Dilger]

On Oct 31, 2002  17:08 -0800, Jim King wrote:
> On Thu, 2002-10-31 at 15:25, Andreas Dilger wrote:
> > The LVM snapshot is the whole filesystem (with only the parts that
> > actually change being written to the snapshot LV) - no need to change
> > the filesystem size at all.  All you need to change is the LV size.
> 
> OK -- so using lvextend I can extend it. But once it's extended, I can
> no longer mount it. I get:
> 
>   mount: wrong fs type, bad option, bad superblock on
>     /dev/data1/.mer_dev.hourly.3, or too many mounted file systems
> 
> Is there something I'm missing here? (ok -- obviously yes...)

You must have gotten some sort of error in the syslog after this
(dmesg will show you).

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: [linux-lvm] e2fsadm and snapshots?

[Jon Bendtsen]

Jim King wrote:
> 
> On Thu, 2002-10-31 at 15:25, Andreas Dilger wrote:
> > The LVM snapshot is the whole filesystem (with only the parts that
> > actually change being written to the snapshot LV) - no need to change
> > the filesystem size at all.  All you need to change is the LV size.
> 
> OK -- so using lvextend I can extend it. But once it's extended, I can
> no longer mount it. I get:

You dont need to unmount it to extend it.
May i suggest mounting the filesystems as ext2, if you are running ext3?


I run 5 scripts for the snapshots as well.
One is run every minute, and checks if there is less than 512M left on
the snapshot, then it extends it. (this is actualy 3 scripts, one using
find to execute another script, this one is called by cron, the 2. does
a lvdisplay on the snapshot, and pipes the data into a perl script, that
does checking and resizing + email sysadm (me))
The 4 other scripts are run every hour, every 8 hour, every day, and
every
sunday. They take a new snapshot, unmounts the old, mounts the new, and
removes the old.

Ontop of all this, i take backup using the day snapshot, and i'll soon
(setting it up today) get another server, that i will use rsync to
take a backup of the hourly snapshot to another server, which is
identical.



JonB

Re: [linux-lvm] e2fsadm and snapshots?

[Jim King]

On Thu, 2002-10-31 at 20:47, Andreas Dilger wrote:
> > OK -- so using lvextend I can extend it. But once it's extended, I can
> > no longer mount it. I get:
> > 
> >   mount: wrong fs type, bad option, bad superblock on
> >     /dev/data1/.mer_dev.hourly.3, or too many mounted file systems
> > 
> > Is there something I'm missing here? (ok -- obviously yes...)
> 
> You must have gotten some sort of error in the syslog after this
> (dmesg will show you).

  EXT2-fs: lvm(58,5): couldn't mount because of unsupported optional
  features (4).

I'm running RedHat 7.3 with 2.4.18-10smp and the RedHat lvm-1.0.3-4 RPM
package. Tried mounting both with no options, with -t ext2, and with -t
ext3. Same results in all cases. Doing the mount in verbose mode doesn't
give any extra info.




...I'd like to figure this out, but in truth there's a more serious
problem with snapshots that means I can't use it. When the file-system
is under load, having a snapshot increases the system load by many
multiples. Example:

  - 250 GB filesystem, copying data to it at a rate of aobut 56 Mbs
  - Load is constantly less than .2, usually down around .05
  - Create one snapshot on the volume: load is now over 1 continuously.
  - Create a 2nd snapshot: system load now jumps over 4.5 continuously.
  - Same experiment with the filesystem used sparsely sees no
    significant system load increase... so it has to do with 
    usage.

I'd have expected performance loss by creating the snapshots, but I
expect it to be a linear loss (ie: 2 snapshots is twice the load of 1
snapshot).

Is this sort of performance loss a known issue? I've got another way of
doing pseudo-snapshots without LVM, but it'd be really nice to use the
built-in snapshotting for a number of reasons. I can't find any mention
of this sort of thing anywhere, and don't see it addressed in any of the
changelogs.


This is on the above RedHat system, which is a dual-processor Athlon
1800 system with 1GB of memory that's doing nothing except holding
volumes and NFS serving them out. VGs are on two 3-ware ide-RAID RAID 5
setups of 960 GB apiece.


-- 
------------------------------------------------------
Jim King
Multimission Image Processing Lab  (MIPL) Engineering Group
Science Data Processing Systems Section,  Jet Propulsion Lab
James.King@jpl.nasa.gov  
------------------------------------------------------

Re: [linux-lvm] e2fsadm and snapshots?

[Andreas Dilger]

On Nov 04, 2002  13:31 -0800, Jim King wrote:
>   EXT2-fs: lvm(58,5): couldn't mount because of unsupported optional
>   features (4).
> 
> I'm running RedHat 7.3 with 2.4.18-10smp and the RedHat lvm-1.0.3-4 RPM
> package. Tried mounting both with no options, with -t ext2, and with -t
> ext3. Same results in all cases. Doing the mount in verbose mode doesn't
> give any extra info.

Two word "vfs-locking patch".  It is in LVM CVS, and _still_ has not made
it into the (a?) kernel.

> ...I'd like to figure this out, but in truth there's a more serious
> problem with snapshots that means I can't use it. When the file-system
> is under load, having a snapshot increases the system load by many
> multiples. Example:
> 
>   - 250 GB filesystem, copying data to it at a rate of aobut 56 Mbs
>   - Load is constantly less than .2, usually down around .05
>   - Create one snapshot on the volume: load is now over 1 continuously.
>   - Create a 2nd snapshot: system load now jumps over 4.5 continuously.
>   - Same experiment with the filesystem used sparsely sees no
>     significant system load increase... so it has to do with 
>     usage.
> 
> I'd have expected performance loss by creating the snapshots, but I
> expect it to be a linear loss (ie: 2 snapshots is twice the load of 1
> snapshot).

Load is only an indicator of processes that are runnable, and not necessarily
an indication of how busy a system is.  If your disk is fairly busy, and
you have lots of processes writing to it, then even a small increase in
disk traffic could result in a huge jump in load as processes block waiting
for their disk I/O to finish.

I would suggest putting your snapshot LV onto a separate disk and see how
that does.

Cheers, Andreas
--
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] e2fsadm and snapshots?

[Joe Thornber]

On Mon, Nov 04, 2002 at 04:29:15PM -0700, Andreas Dilger wrote:
> > ...I'd like to figure this out, but in truth there's a more serious
> > problem with snapshots that means I can't use it. When the file-system
> > is under load, having a snapshot increases the system load by many
> > multiples. Example:

LVM1 snapshot performance is known to be horrible, use LVM2.

- Joe

Re: [linux-lvm] e2fsadm and snapshots?

[Andreas Dilger]

On Nov 05, 2002  12:24 -0500, Greg Freemyer wrote:
> Andreas Dilger wrote:
> >  I would suggest putting your snapshot LV onto a separate disk and see how
> >  that does.
>
> Can that be done?
> 
> I thought the snapshot LV had to share the same VG/PV as the base LV.

Same VG, but it can be on any PV therein.

> Is there some way to force the snapshot on to separate physical drives.

You just specify the PV on which to create the snapshot LV.  This is no
different than any other LV creation, for which you can optionally specify
the PV(s) on which it is created.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/