[linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Davis, Matthew]

Hi,

I want to restore a snapshot without deleting the snapshot.

My use case is that I'm experimenting with a lot of different drivers, kernel modules, and file modifications all over my machine.
I want to 
1. take a snapshot of the working system
2. make changes
3. restore the snapshot (` sudo lvconvert --merge /dev/ubuntu-vg/$SNAPSHOT` then reboot)
4. make new changes
5. restore to the snapshot again

The problem is that step 3 deletes the snapshot, so step 5 fails.

My current workaround is:
1. take a snapshot of the working system
2. make changes
3. restore the snapshot (` sudo lvconvert --merge /dev/ubuntu-vg/$SNAPSHOT` then reboot)
4. Wait 1.5 hours, without making any changes to the machine
5. Take a new snapshot, with the same name as the original
6. make new changes
7. restore to the snapshot

This is not great because:
* I sometimes forget to do step 5
* I can't take a snapshot of the volume while it is still merging. This takes 1.5 hours. I want to be able to restore my snapshots multiple times per day


Is there a flag I can add to `lvconvert` to make it not delete the snapshot?
Alternatively, is there a way I can make a copy of the snapshot before I restore it?

It looks like someone else asked this question 10 years ago. 
https://www.redhat.com/archives/linux-lvm/2008-November/msg00000.html
Has this problem been solved since then?

Thanks,
Matt Davis

Technical Specialist
Telstra | Product Strategy & Innovation - Telstra Labs | Programmable Infrastructure
E �Matthew.Davis.2@team.telstra.com 

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Tomas Dalebjörk]

[-- Attachment #1: Type: text/plain, Size: 2037 bytes --]

Hi,

Drop me an email; there are other ways to perform this without "destroying"
the old snapshot.

Regards Tomas

Den tors 3 jan. 2019 kl 10:29 skrev Davis, Matthew <
Matthew.Davis.2@team.telstra.com>:

> Hi,
>
> I want to restore a snapshot without deleting the snapshot.
>
> My use case is that I'm experimenting with a lot of different drivers,
> kernel modules, and file modifications all over my machine.
> I want to
> 1. take a snapshot of the working system
> 2. make changes
> 3. restore the snapshot (` sudo lvconvert --merge
> /dev/ubuntu-vg/$SNAPSHOT` then reboot)
> 4. make new changes
> 5. restore to the snapshot again
>
> The problem is that step 3 deletes the snapshot, so step 5 fails.
>
> My current workaround is:
> 1. take a snapshot of the working system
> 2. make changes
> 3. restore the snapshot (` sudo lvconvert --merge
> /dev/ubuntu-vg/$SNAPSHOT` then reboot)
> 4. Wait 1.5 hours, without making any changes to the machine
> 5. Take a new snapshot, with the same name as the original
> 6. make new changes
> 7. restore to the snapshot
>
> This is not great because:
> * I sometimes forget to do step 5
> * I can't take a snapshot of the volume while it is still merging. This
> takes 1.5 hours. I want to be able to restore my snapshots multiple times
> per day
>
>
> Is there a flag I can add to `lvconvert` to make it not delete the
> snapshot?
> Alternatively, is there a way I can make a copy of the snapshot before I
> restore it?
>
> It looks like someone else asked this question 10 years ago.
> https://www.redhat.com/archives/linux-lvm/2008-November/msg00000.html
> Has this problem been solved since then?
>
> Thanks,
> Matt Davis
>
> Technical Specialist
> Telstra | Product Strategy & Innovation - Telstra Labs | Programmable
> Infrastructure
> E  Matthew.Davis.2@team.telstra.com
>
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 2962 bytes --]

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Marian Csontos]

On 1/3/19 5:46 AM, Davis, Matthew wrote:
> Hi,
> 
> I want to restore a snapshot without deleting the snapshot.

Hi, I think this should be possible using thin snapshots - snapshot the 
snapshot you want to restore, and merge the snapshot - may be not 
perfect, but it is at least possible. This is fast, as there is no need 
to write huge amounts of data, it is just a switching tree's root away 
(or very close to it.)

Perhaps more convenient alternative is to use boom boot manager 
(boom-boot package available on rhel8+ and fedora27+). Not sure it is 
available in ubuntu/debian.

NOTE:

*Thin pools*

When using thin pools make sure you have enough space and you do not run 
out of space in data and metadata devices, or you risk serious trouble.

1. you must enable monitoring and threshold for extending thin pool.
2. using recent lvm2 2.02.* releases is recommended.

*Boom*

Boom also requires a minor change in initramfs to pass `-K` option to 
lvchange to allow activation of volumes with skip activation flag.

-- Martian


> 
> My use case is that I'm experimenting with a lot of different drivers, kernel modules, and file modifications all over my machine.
> I want to
> 1. take a snapshot of the working system
> 2. make changes
> 3. restore the snapshot (` sudo lvconvert --merge /dev/ubuntu-vg/$SNAPSHOT` then reboot)
> 4. make new changes
> 5. restore to the snapshot again
> 
> The problem is that step 3 deletes the snapshot, so step 5 fails.
> 
> My current workaround is:
> 1. take a snapshot of the working system
> 2. make changes
> 3. restore the snapshot (` sudo lvconvert --merge /dev/ubuntu-vg/$SNAPSHOT` then reboot)
> 4. Wait 1.5 hours, without making any changes to the machine
> 5. Take a new snapshot, with the same name as the original
> 6. make new changes
> 7. restore to the snapshot
> 
> This is not great because:
> * I sometimes forget to do step 5
> * I can't take a snapshot of the volume while it is still merging. This takes 1.5 hours. I want to be able to restore my snapshots multiple times per day
> 
> 
> Is there a flag I can add to `lvconvert` to make it not delete the snapshot?
> Alternatively, is there a way I can make a copy of the snapshot before I restore it?
> 
> It looks like someone else asked this question 10 years ago.
> https://www.redhat.com/archives/linux-lvm/2008-November/msg00000.html
> Has this problem been solved since then?
> 
> Thanks,
> Matt Davis
> 
> Technical Specialist
> Telstra | Product Strategy & Innovation - Telstra Labs | Programmable Infrastructure
> E  Matthew.Davis.2@team.telstra.com
> 
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Davis, Matthew]

Hi Marian,

I'm trying to do it with thin snapshots now. It's all very confusing, and I can't get it to work.
I've read a lot of the documentation about thin stuff, and it isn't clear what's happening.

I took a snapshot with

    sudo lvcreate -s --name mySnap1 centos/root

Then I copied that snapshot with

    sudo lvcreate -s --name mySnap2 centos/mySnap1

When I try to restore mySnap1 it says:

    $sudo lvconvert --merge centos/mySnap1
    centos/mySnap1 is not a mergeable logical volume

The same happens if I use `--mergethin`.
When I try to restore mySnap2 it just "merges" it into mySnap1. I'm trying to merge it into root. I've looked through the man pages and can't find any destination argument.

    $ sudo lvconvert --merge centos/fresh2
    Volume centos/fresh2 replaced origin centos/fresh.

When I merged mySnap2 into mySnap1 and then merged that into root, and rebooted, it didn't actually merge. The files are not as they were when I made mySnap1, and `lvs` shows mySnap1 still exists. And if I try to run the merge command again it says 

    $ sudo lvconvert --merge centos/mySnap1
    Command on LV centos/mySnap1 is invalid on LV with properties: lv_is_merging_origin .
    Command not permitted on LV centos/mySnap1.

What is `lv_is_merging_origin` ? It doesn't appear when I run `lvs`, `lvdisplay`, `pvdisplay`, `man lvmthin` or `man lvm`.
The snapshot should have taken about 20 bytes of differences, so I don't expect it would take very long to restore.
How can I check on the status of the merge? For thick volumes too, I couldn't find a way to check, other than just polling the command to create a new snapshot.

What is a "thin snapshot"? I thought the whole point of LVM snapshots was that they only store the changes in the data.
e.g. If I snapshot a 100GB volume and then add a 1GB file, my snapshot is only 1GB big, not 100GB. So aren't snapshots already thin?

Also, when I take any snapshot on my thinly provisioned system, I get some warning about thresholds and auto-extending.
What is "auto-extending"? I thought the whole point of thin volumes/pools is that the volume only takes up as much space as it needs. If more data is added, it takes up more space automatically. 
I think I've already allocated all my physical disks to this thin pool, so what could "auto-extend" mean?

Thanks,
Matt

-----Original Message-----
From: Marian Csontos [mailto:mcsontos@redhat.com] 
Sent: Friday, 4 January 2019 1:09 AM
To: LVM general discussion and development <linux-lvm@redhat.com>; Davis, Matthew <Matthew.Davis.2@team.telstra.com>
Subject: Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

On 1/3/19 5:46 AM, Davis, Matthew wrote:
> Hi,
> 
> I want to restore a snapshot without deleting the snapshot.

Hi, I think this should be possible using thin snapshots - snapshot the snapshot you want to restore, and merge the snapshot - may be not perfect, but it is at least possible. This is fast, as there is no need to write huge amounts of data, it is just a switching tree's root away (or very close to it.)

Perhaps more convenient alternative is to use boom boot manager (boom-boot package available on rhel8+ and fedora27+). Not sure it is available in ubuntu/debian.

NOTE:

*Thin pools*

When using thin pools make sure you have enough space and you do not run out of space in data and metadata devices, or you risk serious trouble.

1. you must enable monitoring and threshold for extending thin pool.
2. using recent lvm2 2.02.* releases is recommended.

*Boom*

Boom also requires a minor change in initramfs to pass `-K` option to lvchange to allow activation of volumes with skip activation flag.

-- Martian


> 
> My use case is that I'm experimenting with a lot of different drivers, kernel modules, and file modifications all over my machine.
> I want to
> 1. take a snapshot of the working system 2. make changes 3. restore 
> the snapshot (` sudo lvconvert --merge /dev/ubuntu-vg/$SNAPSHOT` then 
> reboot) 4. make new changes 5. restore to the snapshot again
> 
> The problem is that step 3 deletes the snapshot, so step 5 fails.
> 
> My current workaround is:
> 1. take a snapshot of the working system 2. make changes 3. restore 
> the snapshot (` sudo lvconvert --merge /dev/ubuntu-vg/$SNAPSHOT` then 
> reboot) 4. Wait 1.5 hours, without making any changes to the machine 
> 5. Take a new snapshot, with the same name as the original 6. make new 
> changes 7. restore to the snapshot
> 
> This is not great because:
> * I sometimes forget to do step 5
> * I can't take a snapshot of the volume while it is still merging. 
> This takes 1.5 hours. I want to be able to restore my snapshots 
> multiple times per day
> 
> 
> Is there a flag I can add to `lvconvert` to make it not delete the snapshot?
> Alternatively, is there a way I can make a copy of the snapshot before I restore it?
> 
> It looks like someone else asked this question 10 years ago.
> https://www.redhat.com/archives/linux-lvm/2008-November/msg00000.html
> Has this problem been solved since then?
> 
> Thanks,
> Matt Davis
> 
> Technical Specialist
> Telstra | Product Strategy & Innovation - Telstra Labs | Programmable 
> Infrastructure E  Matthew.Davis.2@team.telstra.com
> 
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Zdenek Kabelac]

Dne 10. 01. 19 v 7:23 Davis, Matthew napsal(a):
> Hi Marian,
> 
> I'm trying to do it with thin snapshots now. It's all very confusing, and I can't get it to work.
> I've read a lot of the documentation about thin stuff, and it isn't clear what's happening.
> 
> I took a snapshot with
> 
>      sudo lvcreate -s --name mySnap1 centos/root
> 
> Then I copied that snapshot with
> 
>      sudo lvcreate -s --name mySnap2 centos/mySnap1
> 
> When I try to restore mySnap1 it says:
> 
>      $sudo lvconvert --merge centos/mySnap1
>      centos/mySnap1 is not a mergeable logical volume
> 

Hi

Aren't you using some quite ancient version of lvm2 ?
You need to use lvm2 which does support thin volume merging

Regards

Zdenek

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Davis, Matthew]

Hi Zdenek,

`sudo lvs --version` says :

  LVM version:     2.02.180(2)-RHEL7 (2018-07-20)
  Library version: 1.02.149-RHEL7 (2018-07-20)
  Driver version:  4.37.1

So that means it's version 2, right?

(I'm running the latest version of CentOS.)

Regards,
Matt

-----Original Message-----
From: Zdenek Kabelac [mailto:zkabelac@redhat.com] 
Sent: Thursday, 10 January 2019 8:45 PM
To: LVM general discussion and development <linux-lvm@redhat.com>; Davis, Matthew <Matthew.Davis.2@team.telstra.com>
Subject: Re: how to copy a snapshot, or restore snapshot without deleting it

Dne 10. 01. 19 v 7:23 Davis, Matthew napsal(a):
> Hi Marian,
> 
> I'm trying to do it with thin snapshots now. It's all very confusing, and I can't get it to work.
> I've read a lot of the documentation about thin stuff, and it isn't clear what's happening.
> 
> I took a snapshot with
> 
>      sudo lvcreate -s --name mySnap1 centos/root
> 
> Then I copied that snapshot with
> 
>      sudo lvcreate -s --name mySnap2 centos/mySnap1
> 
> When I try to restore mySnap1 it says:
> 
>      $sudo lvconvert --merge centos/mySnap1
>      centos/mySnap1 is not a mergeable logical volume
> 

Hi

Aren't you using some quite ancient version of lvm2 ?
You need to use lvm2 which does support thin volume merging

Regards

Zdenek

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Zdenek Kabelac]

Dne 14. 01. 19 v 23:44 Davis, Matthew napsal(a):
> Hi Zdenek,
> 
> `sudo lvs --version` says :
> 
>    LVM version:     2.02.180(2)-RHEL7 (2018-07-20)
>    Library version: 1.02.149-RHEL7 (2018-07-20)
>    Driver version:  4.37.1
> 
> So that means it's version 2, right?
> 
> (I'm running the latest version of CentOS.)

Hmm - version looks fine

>>       $sudo lvconvert --merge centos/mySnap1
>>       centos/mySnap1 is not a mergeable logical volume
>>
> 
> Hi
> 
> Aren't you using some quite ancient version of lvm2 ?
> You need to use lvm2 which does support thin volume merging
> 


Can you paste 'lvs -a' prior this operation ?

If the lvm2 tells you that 'mySnap1' is not mergable LV - then
it likely has no 'origin' to merge into.


Zdenek

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Davis, Matthew]

Hi Zdenek,

Here's what I see with `sudo lvs -a`. (My snapshots are actually called `fresh` and `fresh2` not `mySnap`)

```
  LV              VG     Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert
  fresh           centos Vwi---tz-k  <1.46t pool00
  fresh2          centos Vwi---tz-k  <1.46t pool00 fresh
  [lvol0_pmspare] centos ewi------- 760.00m
  pool00          centos twi-aotz--  <1.46t               0.18   3.14
  [pool00_tdata]  centos Twi-ao----  <1.46t
  [pool00_tmeta]  centos ewi-ao---- 760.00m
  root            centos Vwi-aotz--  <1.46t pool00        0.16
  swap            centos -wi-ao----   4.00g
  v01             centos Vwi---tz-k  <1.46t pool00 root
```

Regards,
Matt

-----Original Message-----
From: Zdenek Kabelac [mailto:zkabelac@redhat.com] 
Sent: Tuesday, 15 January 2019 9:50 PM
To: LVM general discussion and development <linux-lvm@redhat.com>; Davis, Matthew <Matthew.Davis.2@team.telstra.com>; Zdenek Kabelac <zkabelac@redhat.com>
Subject: Re: how to copy a snapshot, or restore snapshot without deleting it

Dne 14. 01. 19 v 23:44 Davis, Matthew napsal(a):
> Hi Zdenek,
> 
> `sudo lvs --version` says :
> 
>    LVM version:     2.02.180(2)-RHEL7 (2018-07-20)
>    Library version: 1.02.149-RHEL7 (2018-07-20)
>    Driver version:  4.37.1
> 
> So that means it's version 2, right?
> 
> (I'm running the latest version of CentOS.)

Hmm - version looks fine

>>       $sudo lvconvert --merge centos/mySnap1
>>       centos/mySnap1 is not a mergeable logical volume
>>
> 
> Hi
> 
> Aren't you using some quite ancient version of lvm2 ?
> You need to use lvm2 which does support thin volume merging
> 


Can you paste 'lvs -a' prior this operation ?

If the lvm2 tells you that 'mySnap1' is not mergable LV - then it likely has no 'origin' to merge into.


Zdenek

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Zdenek Kabelac]

Dne 16. 01. 19 v 0:03 Davis, Matthew napsal(a):
> Hi Zdenek,
> 
> Here's what I see with `sudo lvs -a`. (My snapshots are actually called `fresh` and `fresh2` not `mySnap`)
> 
> ```
>    LV              VG     Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert
>    fresh           centos Vwi---tz-k  <1.46t pool00
>    fresh2          centos Vwi---tz-k  <1.46t pool00 fresh

So these both LVs are snapshots  (since they were created with  s(k)ipped 
activation - however 'fresh' LV cannot be merged anymore  as it's origin is 
already gone.

But  'lvconvert --merge  centos/fresh2'  should work - yet you need to 
manually drop 'k' flag (via lvchange).

So to have 'working' merge - you have to see something in the 'Origin' field.
If the Origin field is already 'empty' you can't be merging such LV even when 
it was originally created as 'snapshot'.

Also note that thin snapshot merging is nothing else then a bit more 'smart' 
rename.

Zdenek

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Davis, Matthew]

Hi Zdenek,

What do you mean "it's origin is already gone"?
It is a snapshot of centos/root, which still is there. (I have since modified centos/root)
So the origin is still there.

I think it's because I merged the copy of the snapshot. If I merge the original snapshot, the copy remains and it's origin changes.
That seems to work.

* make snapshot fresh1
* change contents of root
* copy fresh1 to fresh2
* merge fresh1
* reboot
* root is now restored to how it was originally
* fresh 2 now has origin root

That works.
(Although I get a daunting error message every time I create a snapshot, and sometimes I get error messages on merge, but it works most of the time.)

Regards,
Matt

________________________________________
From: Zdenek Kabelac <zkabelac@redhat.com>
Sent: Thursday, 17 January 2019 12:55:59 AM
To: Davis, Matthew; LVM general discussion and development
Subject: Re: how to copy a snapshot, or restore snapshot without deleting it 
�
Dne 16. 01. 19 v 0:03 Davis, Matthew napsal(a):
> Hi Zdenek,
> 
> Here's what I see with `sudo lvs -a`. (My snapshots are actually called `fresh` and `fresh2` not `mySnap`)
> 
> ```
>��� LV������������� VG���� Attr������ LSize�� Pool�� Origin Data%� Meta%� Move Log Cpy%Sync Convert
>��� fresh���������� centos Vwi---tz-k� <1.46t pool00
>��� fresh2��������� centos Vwi---tz-k� <1.46t pool00 fresh

So these both LVs are snapshots� (since they were created with� s(k)ipped 
activation - however 'fresh' LV cannot be merged anymore� as it's origin is 
already gone.

But� 'lvconvert --merge� centos/fresh2'� should work - yet you need to 
manually drop 'k' flag (via lvchange).

So to have 'working' merge - you have to see something in the 'Origin' field.
If the Origin field is already 'empty' you can't be merging such LV even when 
it was originally created as 'snapshot'.

Also note that thin snapshot merging is nothing else then a bit more 'smart' 
rename.

Zdenek

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Zdenek Kabelac]

Dne 17. 01. 19 v 2:12 Davis, Matthew napsal(a):
> Hi Zdenek,
> 
> What do you mean "it's origin is already gone"?

Hi

Your field 'Origin' in your 'lvs -a' was empty - so the actual origin used for 
taking 'fresh' LV snapshot is simply no longer existing.

lvm2 is (ATM) not a database tool trying to resolve/guess what can or cannot 
be still considered as the origin -  so i.e. if you take multiple snapshots of 
a single origin and then you merge one snapshot back to origin -  'the 
original origin' used for all other snapshots is 'gone' -  as lvm2 is not 
resolving here the history and relation of data content whether the meaning of 
origin still applies.

What can make sense in your case is to extend probably  'lvconvert' logic and 
provide  operation  i.e. --replace - which would be working mostly like merge 
- but with thins you would be able to specify  which thinLV should replace 
some other thinLV so basically specifying replace LV1 with LV2 - which you can 
do by lvremove + lvrename ATM - but I can see usefulness for supporting this 
for i.e. root LV which is typically in-use all the time.

I'm not seeing possible to extended the internal logic in lvm2 that would be 
deciding which NEW origin should be replacing removed/merged origin in all 
related snapshots - I'm pretty sure every user would expect a different one...

Zdenek

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Davis, Matthew]

Hi Zdenek,

I assumed that LVM thin snapshots would work like git branches. 
Since git also uses diffs on the backend, and git is popular with developers, the same kind of behaviour seems reasonable to me.

e.g.

```
git checkout master
git branch branchA # equivalent to creating snapshotA
echo "modify" > file.txt # modify origin
git checkout branchA ; git branch branchB # copy snapshotA to snapshotB
git checkout master ; git merge branchA # restore snapshotA
git branch # snapshotB still exists
echo "modify2" > file.txt # modify origin
git checkout branchB # restoring snapshot to root
```

Oh well. It doesn't matter too much, since I've figured out how to get it to work. I just need to restore the oldest snapshot each time. Then LVM updates the origin accordingly.

Regards,
Matt

-----Original Message-----
From: Zdenek Kabelac [mailto:zkabelac@redhat.com] 
Sent: Thursday, 17 January 2019 8:22 PM
To: Davis, Matthew <Matthew.Davis.2@team.telstra.com>; LVM general discussion and development <linux-lvm@redhat.com>
Subject: Re: how to copy a snapshot, or restore snapshot without deleting it

Dne 17. 01. 19 v 2:12 Davis, Matthew napsal(a):
> Hi Zdenek,
> 
> What do you mean "it's origin is already gone"?

Hi

Your field 'Origin' in your 'lvs -a' was empty - so the actual origin used for taking 'fresh' LV snapshot is simply no longer existing.

lvm2 is (ATM) not a database tool trying to resolve/guess what can or cannot be still considered as the origin -  so i.e. if you take multiple snapshots of a single origin and then you merge one snapshot back to origin -  'the original origin' used for all other snapshots is 'gone' -  as lvm2 is not resolving here the history and relation of data content whether the meaning of origin still applies.

What can make sense in your case is to extend probably  'lvconvert' logic and provide  operation  i.e. --replace - which would be working mostly like merge
- but with thins you would be able to specify  which thinLV should replace some other thinLV so basically specifying replace LV1 with LV2 - which you can do by lvremove + lvrename ATM - but I can see usefulness for supporting this for i.e. root LV which is typically in-use all the time.

I'm not seeing possible to extended the internal logic in lvm2 that would be deciding which NEW origin should be replacing removed/merged origin in all related snapshots - I'm pretty sure every user would expect a different one...

Zdenek

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Zdenek Kabelac]

Dne 18. 01. 19 v 1:53 Davis, Matthew napsal(a):
> Hi Zdenek,
> 
> I assumed that LVM thin snapshots would work like git branches.
> Since git also uses diffs on the backend, and git is popular with developers, the same kind of behaviour seems reasonable to me.
> 
> e.g.
> 
> ```
> git checkout master
> git branch branchA # equivalent to creating snapshotA
> echo "modify" > file.txt # modify origin
> git checkout branchA ; git branch branchB # copy snapshotA to snapshotB
> git checkout master ; git merge branchA # restore snapshotA
> git branch # snapshotB still exists
> echo "modify2" > file.txt # modify origin
> git checkout branchB # restoring snapshot to root
> ```
> 
> Oh well. It doesn't matter too much, since I've figured out how to get it to work. I just need to restore the oldest snapshot each time. Then LVM updates the origin accordingly.

Hi

As already said lvm2 is NOT DB engine, but let's look on it this way:

OriginLV ->  snap1

hour later

OriginLV ->  snap2

hour later

OriginLV ->  snap3

hour later

OriginLV ->  snap4

- then you decide to merge  snap2 into  OriginLV

So now there is *THE* question - whether snap2 can be now seen as origin for 
'snap1',  'snap3',  'snap4' ?

The user might have used and update all volumes meanwhile in some way.

After long discussion - we concluded it's hard question and we decided to drop 
  the connection for now.

There is some possibility to analyze what is the relation between all the 
orphans snaps - but it's quite a lot of work - and not directly related to 
volume management.

And I also want to point out you cannot really compare this with 'git' - as 
with git all your steps are mostly easily reversible as all branches are still 
there till prune point, while with this thin merging - one bad merge may 
present quite significant data lose. Also think about the difference in 
behavior between thin snaps and old thick snaps, where the merging worked 
differently and way way way more slower...

IMHO the correct name of such operation is replace/swap with thins.

Regards

Zdenek

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Zdenek Kabelac]

Dne 18. 01. 19 v 1:53 Davis, Matthew napsal(a):
> Hi Zdenek,
> 
> I assumed that LVM thin snapshots would work like git branches.
> Since git also uses diffs on the backend, and git is popular with developers, the same kind of behaviour seems reasonable to me.
> 

Hi

There is very good reason why the git is not really a good tool for storing 
binary data...

Your use-case is 'very specific' sub-case of many different usability 
scenarios you can do with lvm2 -  so while you might see some potential 
benefit if the lvm2 would work more closely to git logic,
it would look terrible in many other situations.

Zdenek

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Zdenek Kabelac]

Dne 21. 01. 19 v 11:32 Zdenek Kabelac napsal(a):
> Dne 18. 01. 19 v 1:53 Davis, Matthew napsal(a):
>> Hi Zdenek,
>>
>> I assumed that LVM thin snapshots would work like git branches.
>> Since git also uses diffs on the backend, and git is popular with 
>> developers, the same kind of behaviour seems reasonable to me.
>>
> 
> Hi
> 
> There is very good reason why the git is not really a good tool for storing 
> binary data...
> 
> Your use-case is 'very specific' sub-case of many different usability 
> scenarios you can do with lvm2 -� so while you might see some potential 
> benefit if the lvm2 would work more closely to git logic,
> it would look terrible in many other situations.
> 

Hi

Adding some more thoughts here - we can probably give user an option to 
control this - particularly in this case  - how about something like:

lvconvert --mergesnapshot --keeporigin y|n

so if user would specify 'y' - lvm2 would relink/preserve merged snapshot as 
an origin for all other existing snapshots of the old origin - this would make 
more obvious what is going to happen when you start creating and merging lots 
of them into a single origin.

(yeah - maybe there could be better fitting option name - so it's more about
idea here for now)

Zdenek

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Davis, Matthew]

Yes that is exactly the option which I would love to have.

Regards,
Matt

-----Original Message-----
From: Zdenek Kabelac [mailto:zkabelac@redhat.com] 
Sent: Monday, 28 January 2019 10:50 PM
To: LVM general discussion and development <linux-lvm@redhat.com>; Zdenek Kabelac <zkabelac@redhat.com>; Davis, Matthew <Matthew.Davis.2@team.telstra.com>
Subject: Re: how to copy a snapshot, or restore snapshot without deleting it

Dne 21. 01. 19 v 11:32 Zdenek Kabelac napsal(a):
> Dne 18. 01. 19 v 1:53 Davis, Matthew napsal(a):
>> Hi Zdenek,
>>
>> I assumed that LVM thin snapshots would work like git branches.
>> Since git also uses diffs on the backend, and git is popular with 
>> developers, the same kind of behaviour seems reasonable to me.
>>
> 
> Hi
> 
> There is very good reason why the git is not really a good tool for 
> storing binary data...
> 
> Your use-case is 'very specific' sub-case of many different usability 
> scenarios you can do with lvm2 -� so while you might see some 
> potential benefit if the lvm2 would work more closely to git logic, it 
> would look terrible in many other situations.
> 

Hi

Adding some more thoughts here - we can probably give user an option to control this - particularly in this case  - how about something like:

lvconvert --mergesnapshot --keeporigin y|n

so if user would specify 'y' - lvm2 would relink/preserve merged snapshot as an origin for all other existing snapshots of the old origin - this would make more obvious what is going to happen when you start creating and merging lots of them into a single origin.

(yeah - maybe there could be better fitting option name - so it's more about idea here for now)

Zdenek

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Tomas Dalebjörk]

[-- Attachment #1: Type: text/plain, Size: 6118 bytes --]

Checkout this
https://github.com/mpalmer/lvmsync

This can probably do what you are asking for.

Regards Tomas

Den tors 10 jan. 2019 kl 10:12 skrev Davis, Matthew <
Matthew.Davis.2@team.telstra.com>:

> Hi Marian,
>
> I'm trying to do it with thin snapshots now. It's all very confusing, and
> I can't get it to work.
> I've read a lot of the documentation about thin stuff, and it isn't clear
> what's happening.
>
> I took a snapshot with
>
>     sudo lvcreate -s --name mySnap1 centos/root
>
> Then I copied that snapshot with
>
>     sudo lvcreate -s --name mySnap2 centos/mySnap1
>
> When I try to restore mySnap1 it says:
>
>     $sudo lvconvert --merge centos/mySnap1
>     centos/mySnap1 is not a mergeable logical volume
>
> The same happens if I use `--mergethin`.
> When I try to restore mySnap2 it just "merges" it into mySnap1. I'm trying
> to merge it into root. I've looked through the man pages and can't find any
> destination argument.
>
>     $ sudo lvconvert --merge centos/fresh2
>     Volume centos/fresh2 replaced origin centos/fresh.
>
> When I merged mySnap2 into mySnap1 and then merged that into root, and
> rebooted, it didn't actually merge. The files are not as they were when I
> made mySnap1, and `lvs` shows mySnap1 still exists. And if I try to run the
> merge command again it says
>
>     $ sudo lvconvert --merge centos/mySnap1
>     Command on LV centos/mySnap1 is invalid on LV with properties:
> lv_is_merging_origin .
>     Command not permitted on LV centos/mySnap1.
>
> What is `lv_is_merging_origin` ? It doesn't appear when I run `lvs`,
> `lvdisplay`, `pvdisplay`, `man lvmthin` or `man lvm`.
> The snapshot should have taken about 20 bytes of differences, so I don't
> expect it would take very long to restore.
> How can I check on the status of the merge? For thick volumes too, I
> couldn't find a way to check, other than just polling the command to create
> a new snapshot.
>
> What is a "thin snapshot"? I thought the whole point of LVM snapshots was
> that they only store the changes in the data.
> e.g. If I snapshot a 100GB volume and then add a 1GB file, my snapshot is
> only 1GB big, not 100GB. So aren't snapshots already thin?
>
> Also, when I take any snapshot on my thinly provisioned system, I get some
> warning about thresholds and auto-extending.
> What is "auto-extending"? I thought the whole point of thin volumes/pools
> is that the volume only takes up as much space as it needs. If more data is
> added, it takes up more space automatically.
> I think I've already allocated all my physical disks to this thin pool, so
> what could "auto-extend" mean?
>
> Thanks,
> Matt
>
> -----Original Message-----
> From: Marian Csontos [mailto:mcsontos@redhat.com]
> Sent: Friday, 4 January 2019 1:09 AM
> To: LVM general discussion and development <linux-lvm@redhat.com>; Davis,
> Matthew <Matthew.Davis.2@team.telstra.com>
> Subject: Re: [linux-lvm] how to copy a snapshot, or restore snapshot
> without deleting it
>
> On 1/3/19 5:46 AM, Davis, Matthew wrote:
> > Hi,
> >
> > I want to restore a snapshot without deleting the snapshot.
>
> Hi, I think this should be possible using thin snapshots - snapshot the
> snapshot you want to restore, and merge the snapshot - may be not perfect,
> but it is at least possible. This is fast, as there is no need to write
> huge amounts of data, it is just a switching tree's root away (or very
> close to it.)
>
> Perhaps more convenient alternative is to use boom boot manager (boom-boot
> package available on rhel8+ and fedora27+). Not sure it is available in
> ubuntu/debian.
>
> NOTE:
>
> *Thin pools*
>
> When using thin pools make sure you have enough space and you do not run
> out of space in data and metadata devices, or you risk serious trouble.
>
> 1. you must enable monitoring and threshold for extending thin pool.
> 2. using recent lvm2 2.02.* releases is recommended.
>
> *Boom*
>
> Boom also requires a minor change in initramfs to pass `-K` option to
> lvchange to allow activation of volumes with skip activation flag.
>
> -- Martian
>
>
> >
> > My use case is that I'm experimenting with a lot of different drivers,
> kernel modules, and file modifications all over my machine.
> > I want to
> > 1. take a snapshot of the working system 2. make changes 3. restore
> > the snapshot (` sudo lvconvert --merge /dev/ubuntu-vg/$SNAPSHOT` then
> > reboot) 4. make new changes 5. restore to the snapshot again
> >
> > The problem is that step 3 deletes the snapshot, so step 5 fails.
> >
> > My current workaround is:
> > 1. take a snapshot of the working system 2. make changes 3. restore
> > the snapshot (` sudo lvconvert --merge /dev/ubuntu-vg/$SNAPSHOT` then
> > reboot) 4. Wait 1.5 hours, without making any changes to the machine
> > 5. Take a new snapshot, with the same name as the original 6. make new
> > changes 7. restore to the snapshot
> >
> > This is not great because:
> > * I sometimes forget to do step 5
> > * I can't take a snapshot of the volume while it is still merging.
> > This takes 1.5 hours. I want to be able to restore my snapshots
> > multiple times per day
> >
> >
> > Is there a flag I can add to `lvconvert` to make it not delete the
> snapshot?
> > Alternatively, is there a way I can make a copy of the snapshot before I
> restore it?
> >
> > It looks like someone else asked this question 10 years ago.
> > https://www.redhat.com/archives/linux-lvm/2008-November/msg00000.html
> > Has this problem been solved since then?
> >
> > Thanks,
> > Matt Davis
> >
> > Technical Specialist
> > Telstra | Product Strategy & Innovation - Telstra Labs | Programmable
> > Infrastructure E  Matthew.Davis.2@team.telstra.com
> >
> >
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> >
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[-- Attachment #2: Type: text/html, Size: 8040 bytes --]

Re: [linux-lvm] how to copy a snapshot, or restore snapshot without deleting it

[Sarah Newman]

On 1/10/19 6:34 AM, Tomas Dalebjörk wrote:
> Checkout this
> https://github.com/mpalmer/lvmsync
> 
> This can probably do what you are asking for.
> 
> Regards Tomas

This does not address the original issue of what to do with thin snapshots, but FYI there is also https://github.com/yurytaranov/lvsync which deals
with non-thin snapshots with much fewer dependencies.

--Sarah