From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.45]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2A82D5D6A9 for ; Tue, 22 Oct 2019 17:02:27 +0000 (UTC) Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3C8513082A98 for ; Tue, 22 Oct 2019 17:02:26 +0000 (UTC) Received: by mail-lf1-f41.google.com with SMTP id y127so13732278lfc.0 for ; Tue, 22 Oct 2019 10:02:26 -0700 (PDT) MIME-Version: 1.0 References: <909d4cae-ddd2-3951-eee8-8dec8faa6f22@redhat.com> In-Reply-To: From: =?UTF-8?Q?Tomas_Dalebj=C3=B6rk?= Date: Tue, 22 Oct 2019 19:02:14 +0200 Message-ID: Content-Type: multipart/alternative; boundary="000000000000ab96cb059582c3a1" Subject: Re: [linux-lvm] exposing snapshot block device Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: To: "Stuart D. Gathman" Cc: LVM general discussion and development --000000000000ab96cb059582c3a1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks for feedbak. Think of lvmsync as a tool, which reads the block changes from the cow device. ... Lets assume that I am able to recreate this cow format instantly back to the server, and present this as a file with the name "cowfile" on the file system for simplicity. Is it possible than in some way, to use this cowfile in someway to inform LVM about the location of the snapshot area, so that lvconvert --merge can be used to restore the data quicker, using this cowfile. The cowfile will include all blocks for the logical volume. Regards Tomas Den tis 22 okt. 2019 kl 18:15 skrev Stuart D. Gathman : > On Tue, 22 Oct 2019, Zdenek Kabelac wrote: > > > Dne 22. 10. 19 v 17:29 Dalebj=C3=B6rk, Tomas napsal(a): > >> But, it would be better if the cow device could be recreated in a > faster > >> way, mentioning that all blocks are present on an external device, so > that > >> the LV volume can be restored much quicker using "lvconvert --merge" > >> command. > > > I do not want to break your imagination here, but that is exactly the > thing > > you can do with thin provisioning and thin_delta tool. > > lvconvert --merge does a "rollback" to the point at which the snapshot > was taken. The master LV already has current data. What Tomas wants to > be able to do a "rollforward" from the point at which the snapshot was > taken. He also wants to be able to put the cow volume on an > extern/remote medium, and add a snapshot using an already existing cow. > > This way, restoring means copying the full volume from backup, creating > a snapshot using existing external cow, then lvconvert --merge > instantly logically applies the cow changes while updating the master > LV. > > Pros: > > "Old" snapshots are exactly as efficient as thin when there is exactly > one. They only get inefficient with multiple snapshots. On the other > hand, thin volumes are as inefficient as an old LV with one snapshot. > An old LV is as efficient, and as anti-fragile, as a partition. Thin > volumes are much more flexible, but depend on much more fragile database > like meta-data. > > For this reason, I always prefer "old" LVs when the functionality of > thin LVs are not actually needed. I can even manually recover from > trashed meta data by editing it, as it is human readable text. > > Updates to the external cow can be pipelined (but then properly > handling reads becomes non trivial - there are mature remote block > device implementations for linux that will do the job). > > Cons: > > For the external cow to be useful, updates to it must be *strictly* > serialized. This is doable, but not as obvious or trivial as it might > seem at first glance. (Remote block device software will take care > of this as well.) > > The "rollforward" must be applied to the backup image of the snapshot. > If the admin gets it paired with the wrong backup, massive corruption > ensues. This could be automated. E.g. the full image backup and > external cow would have unique matching names. Or the full image backup > could compute an md5 in parallel, which would be store with the cow. > But none of those tools currently exist. > > -- > Stuart D. Gathman > "Confutatis maledictis, flamis acribus addictis" - background song for > a Microsoft sponsored "Where do you want to go from here?" commercial. --000000000000ab96cb059582c3a1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks for feedbak.

Think of= lvmsync as a tool, which reads the block changes from the cow device.
<offset><chunk_size>= ;<data>...
Lets assume that I am able to recreate th= is cow format instantly back to the server,
and present this as a= file with the name "cowfile" on the file system for simplicity.<= br>

Is it possible than in some way, to use this c= owfile in someway to inform LVM about the location of the snapshot area, so= that lvconvert --merge can be= used to restore the data quicker, using this cowfile.

=
The cowfile will include all blocks for the logical volume.
=
Regards Tomas

=
Den tis 22 okt. 2019 kl 18:15 skrev S= tuart D. Gathman <stuart@gathman.o= rg>:
On T= ue, 22 Oct 2019, Zdenek Kabelac wrote:

> Dne 22. 10. 19 v 17:29 Dalebj=C3=B6rk, Tomas napsal(a):
>> But, it would be better if the cow device could be recreated in a = faster
>> way, mentioning that all blocks are present on an external device,= so that
>> the LV volume can be restored much quicker using "lvconvert -= -merge"
>> command.

> I do not want to break your imagination here, but that is exactly the = thing
> you can do with thin provisioning and thin_delta tool.

lvconvert --merge does a "rollback" to the point at which the sna= pshot
was taken.=C2=A0 The master LV already has current data.=C2=A0 What Tomas w= ants to
be able to do a "rollforward" from the point at which the snapsho= t was
taken.=C2=A0 He also wants to be able to put the cow volume on an
extern/remote medium, and add a snapshot using an already existing cow.

This way, restoring means copying the full volume from backup, creating
a snapshot using existing external cow, then lvconvert --merge
instantly logically applies the cow changes while updating the master
LV.

Pros:

"Old" snapshots are exactly as efficient as thin when there is ex= actly
one.=C2=A0 They only get inefficient with multiple snapshots.=C2=A0 On the = other
hand, thin volumes are as inefficient as an old LV with one snapshot.
An old LV is as efficient, and as anti-fragile, as a partition.=C2=A0 Thin<= br> volumes are much more flexible, but depend on much more fragile database like meta-data.

For this reason, I always prefer "old" LVs when the functionality= of
thin LVs are not actually needed.=C2=A0 I can even manually recover from trashed meta data by editing it, as it is human readable text.

Updates to the external cow can be pipelined (but then properly
handling reads becomes non trivial - there are mature remote block
device implementations for linux that will do the job).

Cons:

For the external cow to be useful, updates to it must be *strictly*
serialized.=C2=A0 This is doable, but not as obvious or trivial as it might=
seem at first glance.=C2=A0 (Remote block device software will take care of this as well.)

The "rollforward" must be applied to the backup image of the snap= shot.
If the admin gets it paired with the wrong backup, massive corruption
ensues.=C2=A0 This could be automated.=C2=A0 E.g. the full image backup and=
external cow would have unique matching names.=C2=A0 Or the full image back= up
could compute an md5 in parallel, which would be store with the cow.
But none of those tools currently exist.

--
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Stuart D. Gathman <stuart@gathman.org&= gt;
"Confutatis maledictis, flamis acribus addictis" - background son= g for
a Microsoft sponsored "Where do you want to go from here?" commer= cial.
--000000000000ab96cb059582c3a1--