* Internal Qemu snapshots with RBD and libvirt
@ 2013-07-18 15:21 Wido den Hollander
2013-07-19 22:30 ` Josh Durgin
0 siblings, 1 reply; 6+ messages in thread
From: Wido den Hollander @ 2013-07-18 15:21 UTC (permalink / raw)
To: ceph-devel
Hi,
I'm working on the RBD integration for CloudStack 4.2 and now I got to
the point snapshotting.
The "problem" is that CloudStack uses libvirt for snapshotting
Instances, but Qemu/libvirt also tries to store the memory contents of
the domain to assure the snapshot is consistent.
So the way libvirt tries to do it is not possible with RBD right now,
since there is no way to store the internal memory.
I was thinking about using the Java librbd bindings to create the
snapshot, but that will not be consistent thus not 100% safe, so I'd
rather avoid that.
How is this done in OpenStack? Or are you facing similar issues?
P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team archive
with packages for OpenStack Havana.
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Internal Qemu snapshots with RBD and libvirt
2013-07-18 15:21 Internal Qemu snapshots with RBD and libvirt Wido den Hollander
@ 2013-07-19 22:30 ` Josh Durgin
2013-07-19 22:41 ` Sage Weil
0 siblings, 1 reply; 6+ messages in thread
From: Josh Durgin @ 2013-07-19 22:30 UTC (permalink / raw)
To: Wido den Hollander; +Cc: ceph-devel
On 07/18/2013 08:21 AM, Wido den Hollander wrote:
> Hi,
>
> I'm working on the RBD integration for CloudStack 4.2 and now I got to
> the point snapshotting.
>
> The "problem" is that CloudStack uses libvirt for snapshotting
> Instances, but Qemu/libvirt also tries to store the memory contents of
> the domain to assure the snapshot is consistent.
>
> So the way libvirt tries to do it is not possible with RBD right now,
> since there is no way to store the internal memory.
>
> I was thinking about using the Java librbd bindings to create the
> snapshot, but that will not be consistent thus not 100% safe, so I'd
> rather avoid that.
>
> How is this done in OpenStack? Or are you facing similar issues?
OpenStack doesn't store the memory contents of a domain. For volume
snapshots, it requires that the volume is detached, so there can be
no inconsistency, and the actual snapshot handling is done by the volume
driver in cinder, so libvirt is not involved at all. It just uses the
rbd command (or now the python bindings).
> P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team archive
> with packages for OpenStack Havana.
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Internal Qemu snapshots with RBD and libvirt
2013-07-19 22:30 ` Josh Durgin
@ 2013-07-19 22:41 ` Sage Weil
2013-07-19 22:47 ` Marcus Sorensen
0 siblings, 1 reply; 6+ messages in thread
From: Sage Weil @ 2013-07-19 22:41 UTC (permalink / raw)
To: Josh Durgin; +Cc: Wido den Hollander, ceph-devel
On Fri, 19 Jul 2013, Josh Durgin wrote:
> On 07/18/2013 08:21 AM, Wido den Hollander wrote:
> > Hi,
> >
> > I'm working on the RBD integration for CloudStack 4.2 and now I got to
> > the point snapshotting.
> >
> > The "problem" is that CloudStack uses libvirt for snapshotting
> > Instances, but Qemu/libvirt also tries to store the memory contents of
> > the domain to assure the snapshot is consistent.
> >
> > So the way libvirt tries to do it is not possible with RBD right now,
> > since there is no way to store the internal memory.
It seems like the way to view this is that to snapshot a VM, we need to
snapshot all N block devices attached to it, plus the internal memory.
It's not that there is something missing from the RBD block device
snapshot function, but that it is not clear where to put the memory at
all.
Maybe the libvirt or qemu VM metadata should specify a separate image
target for the RAM? How is this normally done when you're using, say,
qcow2? It is assumed that it can be somehow stored with the first block
device or something?
sage
> >
> > I was thinking about using the Java librbd bindings to create the
> > snapshot, but that will not be consistent thus not 100% safe, so I'd
> > rather avoid that.
> >
> > How is this done in OpenStack? Or are you facing similar issues?
>
> OpenStack doesn't store the memory contents of a domain. For volume
> snapshots, it requires that the volume is detached, so there can be
> no inconsistency, and the actual snapshot handling is done by the volume
> driver in cinder, so libvirt is not involved at all. It just uses the
> rbd command (or now the python bindings).
>
> > P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team archive
> > with packages for OpenStack Havana.
> >
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Internal Qemu snapshots with RBD and libvirt
2013-07-19 22:41 ` Sage Weil
@ 2013-07-19 22:47 ` Marcus Sorensen
2013-07-20 0:48 ` Josh Durgin
0 siblings, 1 reply; 6+ messages in thread
From: Marcus Sorensen @ 2013-07-19 22:47 UTC (permalink / raw)
To: Sage Weil; +Cc: Josh Durgin, Wido den Hollander, ceph-devel
Does RBD not honor barriers and do proper sync flushes? Or does this
have to do with RBD caching? Just wondering why online snapshots
aren't safe.
Qcow2 can keep snapshots internally, but qemu is also capable of doing
external dumps for other backing stores. I was thinking about this,
and it seems like you'd put the memory dump on secondary storage, like
a rados gateway or nfs share, so it can be read wherever the VM is
restored to. It would require some work in tracking that location,
however.
On Fri, Jul 19, 2013 at 4:41 PM, Sage Weil <sage@inktank.com> wrote:
> On Fri, 19 Jul 2013, Josh Durgin wrote:
>> On 07/18/2013 08:21 AM, Wido den Hollander wrote:
>> > Hi,
>> >
>> > I'm working on the RBD integration for CloudStack 4.2 and now I got to
>> > the point snapshotting.
>> >
>> > The "problem" is that CloudStack uses libvirt for snapshotting
>> > Instances, but Qemu/libvirt also tries to store the memory contents of
>> > the domain to assure the snapshot is consistent.
>> >
>> > So the way libvirt tries to do it is not possible with RBD right now,
>> > since there is no way to store the internal memory.
>
> It seems like the way to view this is that to snapshot a VM, we need to
> snapshot all N block devices attached to it, plus the internal memory.
> It's not that there is something missing from the RBD block device
> snapshot function, but that it is not clear where to put the memory at
> all.
>
> Maybe the libvirt or qemu VM metadata should specify a separate image
> target for the RAM? How is this normally done when you're using, say,
> qcow2? It is assumed that it can be somehow stored with the first block
> device or something?
>
> sage
>
>> >
>> > I was thinking about using the Java librbd bindings to create the
>> > snapshot, but that will not be consistent thus not 100% safe, so I'd
>> > rather avoid that.
>> >
>> > How is this done in OpenStack? Or are you facing similar issues?
>>
>> OpenStack doesn't store the memory contents of a domain. For volume
>> snapshots, it requires that the volume is detached, so there can be
>> no inconsistency, and the actual snapshot handling is done by the volume
>> driver in cinder, so libvirt is not involved at all. It just uses the
>> rbd command (or now the python bindings).
>>
>> > P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team archive
>> > with packages for OpenStack Havana.
>> >
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Internal Qemu snapshots with RBD and libvirt
2013-07-19 22:47 ` Marcus Sorensen
@ 2013-07-20 0:48 ` Josh Durgin
2013-07-20 8:42 ` Wido den Hollander
0 siblings, 1 reply; 6+ messages in thread
From: Josh Durgin @ 2013-07-20 0:48 UTC (permalink / raw)
To: Marcus Sorensen; +Cc: Sage Weil, Wido den Hollander, ceph-devel
On 07/19/2013 03:47 PM, Marcus Sorensen wrote:
> Does RBD not honor barriers and do proper sync flushes? Or does this
> have to do with RBD caching? Just wondering why online snapshots
> aren't safe.
They're safe at the filesystem level, but I think Wido's after
more application level consistency. If the fs journaled the metadata
for a file but didn't save the data yet, it'd be nice to be able to
restore the complete file.
> Qcow2 can keep snapshots internally, but qemu is also capable of doing
> external dumps for other backing stores. I was thinking about this,
> and it seems like you'd put the memory dump on secondary storage, like
> a rados gateway or nfs share, so it can be read wherever the VM is
> restored to. It would require some work in tracking that location,
> however.
This sounds like a good idea to me.
Josh
> On Fri, Jul 19, 2013 at 4:41 PM, Sage Weil <sage@inktank.com> wrote:
>> On Fri, 19 Jul 2013, Josh Durgin wrote:
>>> On 07/18/2013 08:21 AM, Wido den Hollander wrote:
>>>> Hi,
>>>>
>>>> I'm working on the RBD integration for CloudStack 4.2 and now I got to
>>>> the point snapshotting.
>>>>
>>>> The "problem" is that CloudStack uses libvirt for snapshotting
>>>> Instances, but Qemu/libvirt also tries to store the memory contents of
>>>> the domain to assure the snapshot is consistent.
>>>>
>>>> So the way libvirt tries to do it is not possible with RBD right now,
>>>> since there is no way to store the internal memory.
>>
>> It seems like the way to view this is that to snapshot a VM, we need to
>> snapshot all N block devices attached to it, plus the internal memory.
>> It's not that there is something missing from the RBD block device
>> snapshot function, but that it is not clear where to put the memory at
>> all.
>>
>> Maybe the libvirt or qemu VM metadata should specify a separate image
>> target for the RAM? How is this normally done when you're using, say,
>> qcow2? It is assumed that it can be somehow stored with the first block
>> device or something?
>>
>> sage
>>
>>>>
>>>> I was thinking about using the Java librbd bindings to create the
>>>> snapshot, but that will not be consistent thus not 100% safe, so I'd
>>>> rather avoid that.
>>>>
>>>> How is this done in OpenStack? Or are you facing similar issues?
>>>
>>> OpenStack doesn't store the memory contents of a domain. For volume
>>> snapshots, it requires that the volume is detached, so there can be
>>> no inconsistency, and the actual snapshot handling is done by the volume
>>> driver in cinder, so libvirt is not involved at all. It just uses the
>>> rbd command (or now the python bindings).
>>>
>>>> P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team archive
>>>> with packages for OpenStack Havana.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Internal Qemu snapshots with RBD and libvirt
2013-07-20 0:48 ` Josh Durgin
@ 2013-07-20 8:42 ` Wido den Hollander
0 siblings, 0 replies; 6+ messages in thread
From: Wido den Hollander @ 2013-07-20 8:42 UTC (permalink / raw)
To: Josh Durgin; +Cc: Marcus Sorensen, Sage Weil, ceph-devel
On 07/20/2013 02:48 AM, Josh Durgin wrote:
> On 07/19/2013 03:47 PM, Marcus Sorensen wrote:
>> Does RBD not honor barriers and do proper sync flushes? Or does this
>> have to do with RBD caching? Just wondering why online snapshots
>> aren't safe.
>
> They're safe at the filesystem level, but I think Wido's after
> more application level consistency. If the fs journaled the metadata
> for a file but didn't save the data yet, it'd be nice to be able to
> restore the complete file.
>
Indeed, I'm about application level consistency. I'm now implementing a
PoC where I simply snapshot the RBD image while the Instance is running.
Since CloudStack uses cache=none and RBD cache isn't enabled either it
shouldn't hurt that much.
>> Qcow2 can keep snapshots internally, but qemu is also capable of doing
>> external dumps for other backing stores. I was thinking about this,
>> and it seems like you'd put the memory dump on secondary storage, like
>> a rados gateway or nfs share, so it can be read wherever the VM is
>> restored to. It would require some work in tracking that location,
>> however.
>
> This sounds like a good idea to me.
>
I haven't looked into that, but that seems like a libvirt thing other
then CloudStack since libvirt is the one talking to Qemu.
I however think it should be generic, somebody wanting to snapshot a
running RBD guest via libvirt shouldn't have to go through all kinds of
trouble to get the Instance snapshotted.
But the question then indeed, where to store the memory contents? Create
a new RBD image?
<orig rbd img name>.memory.<snapshot name>
Like that?
Wido
> Josh
>
>> On Fri, Jul 19, 2013 at 4:41 PM, Sage Weil <sage@inktank.com> wrote:
>>> On Fri, 19 Jul 2013, Josh Durgin wrote:
>>>> On 07/18/2013 08:21 AM, Wido den Hollander wrote:
>>>>> Hi,
>>>>>
>>>>> I'm working on the RBD integration for CloudStack 4.2 and now I got to
>>>>> the point snapshotting.
>>>>>
>>>>> The "problem" is that CloudStack uses libvirt for snapshotting
>>>>> Instances, but Qemu/libvirt also tries to store the memory contents of
>>>>> the domain to assure the snapshot is consistent.
>>>>>
>>>>> So the way libvirt tries to do it is not possible with RBD right now,
>>>>> since there is no way to store the internal memory.
>>>
>>> It seems like the way to view this is that to snapshot a VM, we need to
>>> snapshot all N block devices attached to it, plus the internal memory.
>>> It's not that there is something missing from the RBD block device
>>> snapshot function, but that it is not clear where to put the memory at
>>> all.
>>>
>>> Maybe the libvirt or qemu VM metadata should specify a separate image
>>> target for the RAM? How is this normally done when you're using, say,
>>> qcow2? It is assumed that it can be somehow stored with the first block
>>> device or something?
>>>
>>> sage
>>>
>>>>>
>>>>> I was thinking about using the Java librbd bindings to create the
>>>>> snapshot, but that will not be consistent thus not 100% safe, so I'd
>>>>> rather avoid that.
>>>>>
>>>>> How is this done in OpenStack? Or are you facing similar issues?
>>>>
>>>> OpenStack doesn't store the memory contents of a domain. For volume
>>>> snapshots, it requires that the volume is detached, so there can be
>>>> no inconsistency, and the actual snapshot handling is done by the
>>>> volume
>>>> driver in cinder, so libvirt is not involved at all. It just uses the
>>>> rbd command (or now the python bindings).
>>>>
>>>>> P.S.: I'm testing with libvirt 1.0.6 from the Ubuntu Cloud Team
>>>>> archive
>>>>> with packages for OpenStack Havana.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-07-20 8:42 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-07-18 15:21 Internal Qemu snapshots with RBD and libvirt Wido den Hollander
2013-07-19 22:30 ` Josh Durgin
2013-07-19 22:41 ` Sage Weil
2013-07-19 22:47 ` Marcus Sorensen
2013-07-20 0:48 ` Josh Durgin
2013-07-20 8:42 ` Wido den Hollander
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.