[RFC V10 0/4] domain snapshot document

[RFC V10 0/4] domain snapshot document

[Chunyan Liu]

Changes to V9:
  - add details and background knowledge in overview (2/4).
  - update xl snapshot-create syntax and cfg syntax (3/4)
  - update libxl_disk_snapshot structure, add functions for
    deleting external disk snapshots, more description about
    deleting disk snapshot (4/4)

V9:
  http://comments.gmane.org/gmane.comp.emulators.xen.devel/225919

[RFC V10 1/4] domain snapshot terms

[Chunyan Liu]

=====================================================================

Terminology

* Active domain: domain created and started
                 (also referred as "live")

* Inactive domain: domain created but not started
                 (also referred as "offline")

* Domain snapshot:

  Domain snapshot is a system checkpoint of a domain. It contains
  the memory status at the checkpoint and the disk status.

* Disk-only snapshot:

  Disk-only snapshot only keeps the status of disk, not saving
  memory status.

  Contents of disks (whether a subset or all disks associated with
  the domain) are saved at a given point of time, and can be restored
  back to that state. On a running guest, a disk-only snapshot is
  likely to be only crash-consistent rather than clean (that is, it
  represents the state of the disk on a sudden power outage); on an
  inactive guest, a disk-only snapshot is clean if the disks were
  clean when the guest was last shut down.

* Live Snapshot:

  Like live migration, it will increase size of the memory dump file,
  but reducess downtime of the guest.

* Internal Disk Snapshot

  File formats such as qcow2 track both the snapshot and changes
  since the snapshot in a single file.

* External Disk Snapshot

  The snapshot is one file, and the changes since the snapshot
  are in another file.

[RFC V10 2/4] domain snapshot overview

[Chunyan Liu]

Changes to V9:
  * add description in details about domain snapshot and disk snapshot:
    possible user cases, background knowledge about disk snapshots
    implementation, domain snapshot cases, libvirt toolstack support
    status, xl support TODO, etc.

=====================================================================

1. Introduction

There are several types of snapshots:

 disk snapshot
     Contents of disks are saved at a given point of time, and can be
     restored back to that state.

     On a running guest, a disk snapshot is likely to be only
     crash-consistent rather than clean (that is, it represents the
     state of the disk on a sudden power outage, and may need fsck or
     journal replays to be made consistent).

     On a paused guest, with mechanism of quiesing disks (that is,
     all cached data written to disk), a disk snapshot is clean.

     On an inactive guest, a disk snapshot is clean if the disks were
     clean when the guest was last shut down.

     Disk snapshots exist in two forms: internal (file formats such as
     qcow2 track both the snapshot and changes since the snapshot in a
     single file) and external (the snapshot is one file, and the
     changes since the snapshot are in another file).

 memory state (or VM state)
     Tracks only the state of RAM and all other resources in use by
     the VM. If the disks are unmodified between the time a VM state
     snapshot is taken and restored, then the guest will resume in a
     consistent state; but if the disks are modified externally in
     the meantime, this is likely to lead to data corruption.

 system checkpoint (domain snapshot)
     A combination of disk snapshots for all disks as well as VM
     memory state, which can be used to resume the guest from where
     it left off with symptoms similar to hibernation (that is, TCP
     connections in the guest may have timed out, but no files or
     processes are lost).

     A system checkpoint can contain disk snapshots + VM state; or
     contains disk snapshots only without VM state, in this case,
     it should quiesce all disks before taking disk snapshots. The
     latter case is also referred as 'disk-only domain snapshot'.

VM state (memory) snapshots are created by 'domain save', and restore
via 'domain restore'.

Disk snapshot can be created by many external tools, like qemu-img,
vhd-util and lvm, etc.

Domain snapshot (including disk-only domain snapshot) will be handled
by 'domain snapshot' functionality.

Domain snapshot with memory state (as VM state) includes live and
non-live mode according to the VM downtime difference. Live mode will
try best to reduce downtime of the guest, but as a result will increase
size of the memory dump file.


2. Domain Snapshot User Cases

Domain snapshot can be used in following cases:

* Domain snapshot can be used as a domain backup. It can preserve the
  VM status at a certain point and able to roll back to it.

* Domain snapshot can support 'gold image' type deployments, i.e.
  where you create one baseline single disk image and then clone it
  multiple times to deploy lots of guests; when you create a domain
  snapshot, with it as gold domain snapshot (duplicate multiple times),
  one can restore from the gold domain snapshot mulitple times for
  different reasons.

* Disk-only domain snapshot can be used as backup out of domain,
  i.e. taking a disk-only domain snapshot and then run you usual backup
  software on the disk snapshots (which is now unchanging, which
  is handy); one can backup that static version of the disk out of band
  from the domain itself (e.g. can attach it to a separate backup VM).

3. Domain Snapshot Operations

Generally, domain snapshot includes 4 kinds of operations:

* create a domain snapshot

   create domain snapshot under different conditions:
   - domain is live, save vm state (live), disk snapshot
   - domain is live, save vm state (non-live), disk snapshot
   - domain is live, disk-only snapshot (need quiecing disks)
   - domain is offline, disk-only snapshot
   (under each above condition, disk snapshot can be
    internal/external.)

* revert (roll back to) a domain snapshot

   revert domain snapshot under different conditions:
   - domain is live, has vm state, all internal disk snapshots
   - domain is live, has vm state, has external disk snapshots
   - domain is live, no vm state, all internal disk snapshots
   - domain is live, no vm state, has external disk snapshots
   - domain is offline, has vm state, all internal disk snapshots
   - domain is offline, has vm state, has external disk snapshots
   - domain is offline, no vm state, all internal disk snapshots
   - domain is offline, no vm state, has external disk snapshots

* delete a domain snapshot

   delete domain snapshot under following conditions:
   - domain is live, not in a snapshot chain
   - domain is live, in a snapshot chain
   - domain is offline, not in a snapshot chain
   - domain is offline, in a snapshot chain

* list domain snapshot(s)
   list domain snapshot(s) contains:
   - list a single domain snapshot
   - list all domain snapshots
   - list snapshot(s) in details

Since domain snapshot includes disk snapshot, expand disk snapshot
operations here for later understanding.

# Disk Snapshot Operations

 * Create disk snapshot
 * Delete disk snapshot
 * Revert (apply) disk snapshot
 * List disk snapshots

 Tools:
   - Internal disk snapshot
     - 'qcow2' format
       * domain is live, through qmp command
       * domain is offline, qemu-img
     - other formats, not supported
   - External disk snapshot
     - 'lvm' format, lvm tool
     - 'vhd' format, vhd-util or qmp/qemu-img command
     - other formats
       * domain is live, qmp command
       * domain is offline, qemu-img command

# Existing internal/external disk snapshot implementation

  Add a few words to explain the implementation of internal
  and external disk snapshots, to help understanding.

   * Internal disk snapshot:

     Supported by 'qcow2' format only. The implementation is in qemu.
     According to qemu code, creating a snapshot will add a copy of
     cluster table, and increase refcounts of the clusters, e.g. if
     refcount > 1, will COW on a write operation; when deleting data,
     it will decrease refcount, not delete data until refcount = 0.
     [1]

     So, deleting/reverting an internal snapshot (even in a snapshot
     chain) is very straight forward, won't affect other snapshots.
     [2]

   * External disk snapshot:

     'lvm' format:
        implemented by lvm tool, using backing file mechanism.
        After snapshot command, original disk image becomes
        backing file and the new image will be created to track
        the new changes.

        If it is used in a domain, after the snapshot, domain
        should use the new image to replace the original image
        for further operations.

        e.g.:
        Before snapshot:
        domain --> disk A

        take disk snapshot:
        disk A (Base) <- disk B (Active)

        after snapshot:
        domain --> disk B

        Lvm tool doesn't support snapshot of snapshot.

     'vhd' format:
        implemented by 'vhd-util', qemu qmp command can also do
        the work. As lvm, both vhd-util and qemu qmp command
        use backing file mechanism too.

     Other formats:
        supported by qemu qmp command. Same as above,
        using backing file mechanism too.


     In summary, all external disk snapshot tools are implemented
     with backing file mechanism.

     snapshot of snapshot: becomes a backing file chain, like:

     RootBase <-sn1 <- sn2 <- sn3  <- new image A (active)
                 |
                  <--sn4 <- new image B (active)

     Compared with internal disk snapshot, deleting/reverting an
     external disk snapshot is more tricky.

     Deleting involves complicated process dealing with additional
     files, merge 'base' to 'top' (or say, 'parent' to 'children').
     Qemu supplies method to do that, refer to qmp commands:
     block-stream, block-commit, or qemu-img rebase, qemu-img commit.

     example:
     RootBase <- sn1 <- sn2 <- Active

     To delete sn1:
     [online]
     # use QMP 'block-stream' to populate data of sn1 to sn2
     # rm sn1

     [offline]
     # qemu-img rebase -b RootBase sn2 (sn1 data will be merged to sn2)
     # rm sn1

     Reverting involves duplicate/copy backing file, since backing
     file is read-only and could not be changed.

     Might because of the complexity, currently libvirt qemu driver
     doesn't support deleting or reverting to a domain snapshot
     containg external disk snapshot, but it supplies commands like:
     'virsh block-pull', 'virsh block-commit', 'virsh block-copy' to
     help users dealing with external snapshots.


4. Libvirt toolstack support status

Now, libvirt qemu driver supports domain snapshot functionality, but
not all cases. Following is the support status:

[support: Y; not support: N]

* create a domain snapshot

    [Y] - domain is live, save vm state (live), disk snapshot
    [Y] - domain is live, save vm state (non-live), disk snapshot
    [Y] - domain is live, disk-only snapshot
    [Y] - domain is offline, disk-only snapshot
          (under each above condition, disk snapshot can be
          internal/external.)

    libvirt toolstack maintains snapshot info and the chain
    relationship.

* revert (roll back to) a domain snapshot

    [Y] - domain is live/offline, has/no vm state, all internal disk snapshots
    [N] - domain is live/offline, has/no vm state, has external disk snapshots

* delete a domain snapshot

    [Y] - domain is live/offline, not in a snapshot chain,
          all internal disk snapshots
    [N] - domain is live/offline, not in a snapshot chain,
          has external disk snapshots
    [Y] - domain is live/offline, in a snapshot chain,
          no one in the chain contains external disk snapshot.
          delete this snapshot or together with children snapshots.
    [N] - domain is live/offline, in a snapshot chain,
          someone in the chain contains external disk snapshot.

* list domain snapshot(s)
    [Y] - list a single domain snapshot
    [Y] - list all domain snapshots
    [Y] - list snapshot(s) in details


5. xl toolstack planning support

Following the existing xl idioms of managing storage and saved
VM images via existing CLI command (qemu-img, lvcreate, ls, mv,
cp etc), xl snapshot functionality would be kept as simple as
possible.

xl will support creating snapshot and reverting to a snapshot.

But it won't manage snapshots information, as xl doesn't maintain
saved images created by 'xl save'. xl will have no idea of
the existence of domain snapshots and the chain relationship
among snapshots, so it depends on user to take care of the
snapshots and the snapshot chain relationship, and delete
snapshots.

xl won't support deleting snapshot and listing snapshots.

And talking about disk-only snapshot, since xl only concerns
active (live) domains, even when domain is paused, no mechanism
to quiesce disks, that means, taking a disk-only snapshot and
then resume, it is as if the guest had crashed. For this reason,
currently xl taking disk-only snapshot is useless.

Following will be the planning support:

[support: Y; not support: N]

* create a domain snapshot

    [Y] - domain is live, save vm state (live), disk snapshot
    [Y] - domain is live, save vm state (non-live), disk snapshot
    [N] - domain is live, disk-only snapshot
          (under each above condition, disk snapshot can be
          internal/external.)

* revert (roll back to) a domain snapshot

    [Y] - domain is live, has/no vm state, all internal disk snapshots
    [Y?] - domain is live, has/no vm state, has external disk snapshots

* delete a domain snapshot
    [N]

* list domain snapshots
    [N]


# Interaction with other operations:

  Don't support creating domain snapshot when domain is shutdowning
  or dying.


6. xl toolstack planning implementation outline

# General Requirements:

 * ability to save/restore domain memory
 * ability to create/delete/revert disk snapshot
 * ability to parse user config file

# xl toolstack general workflow

 Create a snapshot:
 - parse user cfg file if passed in
 - check snapshot operation is allowed or not
 - save domain, saving memory status to file (refer to: save_domain)
   [if quiecing disks mechanism implemented in future, then can
    support disk-only snapshot, in that case, instead of saving
    memory, it will be:
    - pause domain, quiecing all disks]
 - take disk snapshot (e.g. call qmp command)
 - unpause domain

 Revert to snapshot:
 - parse use cfg file (xl doesn't manage snapshots, so it has no
   idea of snapshot existence. User MUST supply configuration file)
 - destroy this domain
 - create a new domain from snapshot info
   - apply disk snapshot (e.g. call qemu-img)
   - a process like restore domain

[1] https://people.gnome.org/~markmc/qcow-image-format.html
[2] https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html

[RFC V10 3/4] domain snapshot design: xl

[Chunyan Liu]

Changes to V9:
  * add --internal|--external option to snapshot-create.
  * add @path to snapshot-create
  * remove --name to snapshot-create
  * add detailed description for snapshot-create default behaviors
    and behaviors with options
  * add description about deleting domain snapshot for a full picture
    and as background knowledge for a possible discussion later.
  * add 'memory' in cfgfile syntax for future disk-only snapshot usage

===========================================================================

XL Design

1. User Interface

xl snapshot-create:
  Create a snapshot (disk and RAM) of a domain.

  SYNOPSIS:
    snapshot-create [--live] [--internal|--external] <domain> [path] [<cfgfile>]
  OPTIONS:
    --live           take a live snapshot
    --internal       take internal disk snapshots to all disks
    --external       take external disk snapshots to all disks

    If no options specified and no @cfgfile and no @path specified:
    e.g. # xl snapshot-create domain
    By default, it will create a domain snapshot with default name
    generated according to creation time. This name will be used to
    generate default RAM snaphsot name and disk snapshot name, and
    generate the default directory to store all the snapshot data
    (RAM snapshot file, external disk snapshot files, etc.)
    e.g. result of above command would be:
    default snapshot root directory:
        /var/lib/xen/snapshots/
    default snapshot name generated :
        20150122xxx
    default subdirectory to save data of this snapshot:
        /var/lib/xen/snapshots/domain_uuid/20150122xxx/
    RAM snapshot file:
        By default, it will save memory. Location is here:
        /var/lib/xen/snapshots/domain_uuid/20150122xxx/20150122xxx.save
    disk snapshots:
        By default, to each domain disk, take internal disk snapshot if
        that disk supports, otherwise, take external disk snapshot.

        Internal disk snapshot: take disk snapshot with name 20150122xxx
        External disk snapshot: external file is:
        /var/lib/xen/snapshots/domain_uuid/20150122xxx/vda_20150122xxx.qcow2
        /var/lib/xen/snapshots/domain_uuid/20150122xxx/vdb_20150122xxx.qcow2

    If option includes --live, then the domain is not paused while creating
    the snapshot, like live migration. This increases size of the memory
    dump file, but reducess downtime of the guest.

    If @path is given, all snapshot data will be saved in this @path.
    If no @cfgfile:name specified, then use the basename of @path as snapshot
    name.

    User could specify snapshot information in details through @cfgfile, see
    following cfgfile syntax. If configuration in @cfgfile conflicts with
    other options, @cfgfile will supercede other options. (e.g. if --internal
    conflicts with disk spec in cfgfile, use disk spec in @cfgfile.)


xl snapshot-revert:
  Revert domain to status of a snapshot.

  SYNOPSIS:
      snapshot-revert <domain> <cfgfile> [--running] [--force]

  OPTIONS:
    --running        after reverting, change state to running
    --force          try harder on risky reverts

    Normally, the domain will revert to the same state the domain was in while
    the snapshot was taken (whether running, or paused).

    If option includes --running, then overrides the snapshot state to
    guarantee a running domain after the revert.


About domain snapshot delete:
    xl doesn't have snapshot chain information, so it couldn't do the full work.
    If supply:
      xl snapshot-delete <domain> <cfgfile>
    For internal disk snapshot, deleting disk snapshot doesn't need snapshot chain
    info, this commands can finish the work. But for external disk snapshot,
    deleting disk snapshot will need to merge backing file chain, then will need
    the backing file chain information, this xl command can not finish that.

    So, deleting domain snapshots will be left to user:

    user could delete RAM snapshots and disk snapshots by themselves:
    RAM snapshot file: user could remove it directly.
    Disk snapshots:
      - Internal disk snapshot, issue 'qemu-img snapshot -d'
      - External disk snapshot,


2. cfgfile syntax

# snapshot name. If user doesn't provide a VM snapshot name, xl will generate
# a name automatically by creation time or by @path basename.
name=""

# save memory or disk-only.
# If memory is '0', doesn't save memory, take disk-only domain snapshot.
# If memory is '1', domain memory is saved.
# Default if 1.
memory=1

# memory location. This field is valid when memory=1.
# If it is set to "", xl will generate a path by creation time or by @path
# basename.
memory_path=""

# disk snapshot specification
# For easier parsing config work, reuse disk configuration in xl.cfg, but
# with different meanings.
# disk syntax meaning: 'external path, external format, target device'
#
# By default, if no disks is specified here, it will take disk snapshot
# to all disks: take internal disk snapshot if disk support internal disk
# snapshot; and external disk snapshot to other disks.

#disks=['/tmp/hda_snapshot.qcow2,qcow2,hda', ',,hdb',]

3. xl snapshot-xxx implementation

"xl snapshot-create"

    1), parse args or user configuration file.
    2), if saveing memory: save domain (store saved memory to memory_path)
        if taking disk-only snapshot: pause domain, quiece disks. (not
        supported now, maybe in future.)
    3), create disk snapshots according to disk snapshot configuration
    4), unpause domain

"xl snapshot-revert"

    1), parse user configuration file
    2), destroy current domain
    3), revert disk snapshots according to disk snapshot configuration
    4), restore domain from saved memory.

4. Notes

* user should take care of snapshot data: saved memory file, disk
  snapshots info (internal, external, etc.), snapshot chain relationship
* user should delete snapshots by themselves with CLI commands like: rm,
  qemu-img, etc.

Re: [RFC V10 3/4] domain snapshot design: xl

[Ian Campbell]

On Mon, 2015-01-26 at 11:25 +0800, Chunyan Liu wrote:

This is all looking good (including the previous two patches where I
didn't have any substantial comments).

>     User could specify snapshot information in details through @cfgfile, see
>     following cfgfile syntax. If configuration in @cfgfile conflicts with
>     other options, @cfgfile will supercede other options. (e.g. if --internal
>     conflicts with disk spec in cfgfile, use disk spec in @cfgfile.)

Usually I think one would expect the command line to take precedence
over a config file.

> 
> 
> xl snapshot-revert:
>   Revert domain to status of a snapshot.
> 
>   SYNOPSIS:
>       snapshot-revert <domain> <cfgfile> [--running] [--force]
> 
>   OPTIONS:
>     --running        after reverting, change state to running

We usually talk about paused in the xl interface, e.g. --paused with the
default being to run.

>     --force          try harder on risky reverts
> 
>     Normally, the domain will revert to the same state the domain was in while
>     the snapshot was taken (whether running, or paused).

So you need both --paused and --unpause to override both cases?

>     If option includes --running, then overrides the snapshot state to
>     guarantee a running domain after the revert.
> 
> 
> About domain snapshot delete:
>     xl doesn't have snapshot chain information, so it couldn't do the full work.
>     If supply:
>       xl snapshot-delete <domain> <cfgfile>
>     For internal disk snapshot, deleting disk snapshot doesn't need snapshot chain
>     info, this commands can finish the work. But for external disk snapshot,
>     deleting disk snapshot will need to merge backing file chain, then will need
>     the backing file chain information, this xl command can not finish that.
> 
>     So, deleting domain snapshots will be left to user:
> 
>     user could delete RAM snapshots and disk snapshots by themselves:
>     RAM snapshot file: user could remove it directly.
>     Disk snapshots:
>       - Internal disk snapshot, issue 'qemu-img snapshot -d'
>       - External disk snapshot,
> 
> 
> 2. cfgfile syntax
> 
> # snapshot name. If user doesn't provide a VM snapshot name, xl will generate
> # a name automatically by creation time or by @path basename.
> name=""
> 
> # save memory or disk-only.
> # If memory is '0', doesn't save memory, take disk-only domain snapshot.
> # If memory is '1', domain memory is saved.
> # Default if 1.
> memory=1
> 
> # memory location. This field is valid when memory=1.
> # If it is set to "", xl will generate a path by creation time or by @path
> # basename.
> memory_path=""
> 
> # disk snapshot specification
> # For easier parsing config work, reuse disk configuration in xl.cfg, but
> # with different meanings.
> # disk syntax meaning: 'external path, external format, target device'

Note that the actual xl syntax is more complex than this and what you
appear to be reusing is the legacy syntax. See
docs/misc/xl-disk-configuration.txt

Re: [RFC V10 3/4] domain snapshot design: xl

[Chun Yan Liu]

>>> On 1/30/2015 at 12:33 AM, in message <1422549191.5198.32.camel@citrix.com>, Ian
Campbell <Ian.Campbell@citrix.com> wrote: 
> On Mon, 2015-01-26 at 11:25 +0800, Chunyan Liu wrote: 
>  
> This is all looking good (including the previous two patches where I 
> didn't have any substantial comments). 
>  
> >     User could specify snapshot information in details through @cfgfile,  
> see 
> >     following cfgfile syntax. If configuration in @cfgfile conflicts with 
> >     other options, @cfgfile will supercede other options. (e.g. if  
> --internal 
> >     conflicts with disk spec in cfgfile, use disk spec in @cfgfile.) 
>  
> Usually I think one would expect the command line to take precedence 
> over a config file. 

OK. That works too. Will change.

>  
> >  
> >  
> > xl snapshot-revert: 
> >   Revert domain to status of a snapshot. 
> >  
> >   SYNOPSIS: 
> >       snapshot-revert <domain> <cfgfile> [--running] [--force] 
> >  
> >   OPTIONS: 
> >     --running        after reverting, change state to running 
>  
> We usually talk about paused in the xl interface, e.g. --paused with the 
> default being to run. 
>  
> >     --force          try harder on risky reverts 
> >  
> >     Normally, the domain will revert to the same state the domain was in  
> while 
> >     the snapshot was taken (whether running, or paused). 
>  
> So you need both --paused and --unpause to override both cases? 

Oh, no, maybe the description is confusing. It should be:
if domain is paused when taking snapshot, keep paused after restoreing.
if domain is running when taking snapshot, unpause to let it be running
after restoring. Libvirt keeps domain->state when taking snapshot, so it
can do that automatically; Xl doesn't keep state, so to be simple, we could
just follow xl common behavior:
Normally, the domain will restored from snapshot to a running state.
If -p|--pause specified, do not unpause domain after restoring it.

I'll update. Thanks.

- Chunyan

>  
> >     If option includes --running, then overrides the snapshot state to 
> >     guarantee a running domain after the revert. 
> >  
> >  
> > About domain snapshot delete: 
> >     xl doesn't have snapshot chain information, so it couldn't do the full  
> work. 
> >     If supply: 
> >       xl snapshot-delete <domain> <cfgfile> 
> >     For internal disk snapshot, deleting disk snapshot doesn't need  
> snapshot chain 
> >     info, this commands can finish the work. But for external disk  
> snapshot, 
> >     deleting disk snapshot will need to merge backing file chain, then will  
> need 
> >     the backing file chain information, this xl command can not finish  
> that. 
> >  
> >     So, deleting domain snapshots will be left to user: 
> >  
> >     user could delete RAM snapshots and disk snapshots by themselves: 
> >     RAM snapshot file: user could remove it directly. 
> >     Disk snapshots: 
> >       - Internal disk snapshot, issue 'qemu-img snapshot -d' 
> >       - External disk snapshot, 
> >  
> >  
> > 2. cfgfile syntax 
> >  
> > # snapshot name. If user doesn't provide a VM snapshot name, xl will  
> generate 
> > # a name automatically by creation time or by @path basename. 
> > name="" 
> >  
> > # save memory or disk-only. 
> > # If memory is '0', doesn't save memory, take disk-only domain snapshot. 
> > # If memory is '1', domain memory is saved. 
> > # Default if 1. 
> > memory=1 
> >  
> > # memory location. This field is valid when memory=1. 
> > # If it is set to "", xl will generate a path by creation time or by @path 
> > # basename. 
> > memory_path="" 
> >  
> > # disk snapshot specification 
> > # For easier parsing config work, reuse disk configuration in xl.cfg, but 
> > # with different meanings. 
> > # disk syntax meaning: 'external path, external format, target device' 
>  
> Note that the actual xl syntax is more complex than this and what you 
> appear to be reusing is the legacy syntax. See 
> docs/misc/xl-disk-configuration.txt 
>  
>  
>  
>  

[RFC V10 4/4] domain snapshot design: libxl

[Chunyan Liu]

Changes to V9:
  * update libxl_disk_snapshot structures as Ian suggests
  * add more descriptions for libxl_disk_snapshot_delete
  * add two functions for deleting external disk snapshot:
    libxl_domain_block_rebase and libxl_domain_block_commit

===========================================================================

Libxl Design

1. New Structures

libxl_disk_snapshot_type = Struct("disk_snapshot_type", [
    (0, "default"),
    (1, "internal"),
    (2, "external"),
    ])

libxl_disk_snapshot = Struct("disk_snapshot",[
    # target disk
    ("disk",            libxl_device_disk),

    # disk snapshot name
    ("name",            string),

    ("u", KeyedUnion(None, libxl_disk_snapshot_type, "type",
         [("external", Struct(None, [

            # disk format for external files. Since external disk snapshot is
            # implemented with backing file mechanism, the external file disk
            # format must support backing file. This field can be NULL, then
            # a proper disk format will be used by default according to the
            # orignal disk format.
            ("external_format", libxl_disk_format),

            # external file path. This field should be non-NULL and a new path.
            ("external_path",   string),
            ]))
         ]))
    ])


2. New Functions

Since there're already APIs for saving memory (libxl_domain_suspend)
and restoring domain from saved memory (libxl_domain_create_restore), to
xl domain snapshot tasks, the missing part is disk snapshot functionality.
And the disk snapshot functionality would be used by libvirt too.

Considering there is qmp handling in creating/deleting disk snapshot,
will add following new functions to libxl:

/**
 * libxl_disk_snaphost_create:
 * @ctx: libxl context
 * @domid: domain id
 * @snapshot: array of disk snapshot configuration. Has "nb" members.
 *     - libxl_device_disk:
 *         structure to represent which disk.
 *     - name:
 *         snapshot name.
 *     - type:
 *        disk snapshot type: internal or external.
 *     - u.external.external_format:
 *         Format of external file.
 *         After disk snapshot, original file will become a backing
 *         file, while external file will keep the delta, so
 *         external_format should support backing file, like: cow,
 *         qcow, qcow2, etc.
 *         If it is NULL, then it will use proper format by default
 *         according to original disk format.
 *     - u.external.external_path:
 *         path to external file. non-NULL.
 * @nb: number of disks that need to take disk snapshot.
 *
 * createing internal/external disk snapshot
 *
 * Taking disk snapshots to a group of domain disks according to
 * configuration. Support both internal disk snapshot and external
 * disk snapshot. For qcow2 disk backend type, it will call qmp
 * "transaction" command to do the work. For other disk backend types,
 * might call other external commands.
 *
 * Returns 0 on success, <0 on failure.
 */
int libxl_disk_snapshot_create(libxl_ctx *ctx, uint32_t domid,
                               libxl_disk_snapshot *snapshot, int nb);


/**
 * libxl_disk_snaphost_delete:
 * @ctx: libxl context
 * @domid: domain id
 * @snapshot: array of disk snapshot configuration. Has "nb" members.
 * @nb: number of disks.
 *
 * Delete disk snapshot, deal with internal disk snapshot only.
 *
 * Delete disk snapshot of a group of domain disks according to
 * configuration. Can only deal with internal disk snapshot (currently
 * only 'qcow2', by calling qmp command). Can safely delete disk
 * snapshot even when the snapshot is in a snapshot chain, won't affect
 * other snapshots. Don't need to know snapshot chain info.
 *
 * To delete external disk snapshots, means shorten backing file chain
 * and merge snapshot data, must have snapshot chain info. Functions
 * libxl_domain_block_rebase and libxl_domain_block_commit would help.
 *
 * Returns 0 on success, <0 on failure.
 */
int libxl_disk_snapshot_delete(libxl_ctx *ctx, uint32_t domid,
                               libxl_disk_snapshot *snapshot, int nb);


Following two functions would be used to delete external disk snapshots.
They are essentially two directions to shorten backing file chain. One
is from "base" to "top" merge, the other is from "top" to "base" merge.
Both need caller to have the backing file chain information.

(This is active mode, when domain is live.
 When domain is offline, can call 'qemu-img rebase|commit' to do the work

 Since xl won't maintain snapshot chain information, either xl implement
 some commands for users so that users could call xl commands to do merge
 work, or users could do merge work by qemu-img when domain is offline.)

/**
 * libxl_domain_block_rebase:
 * @ctx: libxl context
 * @domid: domain id
 * @disk: path to the block device
 * @base: path to backing file to keep, or NULL for no backing file
 * @bandwidth: (optional) bandwidth limit in B/s, 0 for no limit.
 *
 * Merge data from base to top
 *
 * Populate a disk image with data from its backing image chain, and
 * setting the backing image to @base, or alternatively copy an entire
 * backing chain to a new file @base.
 *
 * @base must be the absolute path of one of the backing images further
 * up the chain, or NULL to convert the disk image so that it has no
 * backing image.  Once all data from its backing image chain has been
 * pulled, the disk no longer depends on those intermediate backing
 * images.
 *
 * The maximum bandwidth that will be used to do the copy can be
 * specified with the @bandwidth parameter, unit is B/s.  If set to 0,
 * there is no limit.
 *
 * e.g.:
 * backing file chain:
 *     RootBase <- A <- B <- disk
 * after libxl_domain_block_rebase(ctx, domid, disk, A, 0):
 *     RootBase <- A <- disk
 * delta in B is merged into disk.
 *
 * Returns 0 on success, <0 on failure.
 */
int libxl_domain_block_rebase(libxl_ctx *ctx, uint32_t domid,
                              const char *disk,
                              const char *base,
                              unsigned long long bandwidth);


/**
 * libxl_domain_block_commit:
 * @ctx: libxl context
 * @domid: domain id
 * @disk: path to the block device
 * @base: path to backing file to merge into, or NULL for default
 * @top: path to file within backing chain that contains data to be merged,
 *       or NULL to merge all possible data
 * @bandwidth: (optional) bandwidth limit in B/s; 0 for no limit
 *
 * Merge data from top to base
 *
 * Commit changes that were made to temporary top-level files within a disk
 * image backing file chain into a lower-level base file.  In other words,
 * take all the difference between @base and @top, and update @base to contain
 * that difference; after the commit, any portion of the chain that previously
 * depended on @top will now depend on @base, and all files after @base up
 * to and including @top will now be invalidated.  A typical use of this
 * command is to reduce the length of a backing file chain after taking an
 * external disk snapshot. To move data in the opposite direction, see
 * libxl_domain_block_commit().
 *
 * e.g.:
 * backing file chain:
 *     RootBase <- A <- B <- disk
 * after libxl_domain_block_commit(ctx, domid, disk, RootBase, B, 0):
 *     RootBase <- B <- disk
 * delta in A is merged to RootBase.
 *
 * Returns 0 on success, <0 on failure.
 */
int libxl_domain_block_commit(libxl_ctx *ctx, uint32_t domid,
                              const char *disk,
                              const char *base,
                              const char *top,
                              unsigned long bandwidth);



For disk snapshot revert, since domain snapshot revert is essentially
destroy, revert disks and restore from RAM. There is no qemu process
to speak to during reverting disks. So, it always calls external
commands to finish the work:

/**
 * libxl_disk_snapshot_revert:
 * @snapshot: array of disk snapshot configuration. Has "nb" members.
 * @nb: number of disks.
 *
 * Revert disks to specified snapshot according to configuration. To
 * different disk backend types, call different external commands to do
 * the work.
 *
 * Returns 0 on success, <0 on failure.
 */
int libxl_disk_snapshot_revert(libxl_disk_snapshot *snapshot, int nb);


/**
 * libxl_disk_to_snapshot:
 * @snapshot: array of disk snapshot configuration. Has "nb" members.
 * @nb: number of disks.
 * @ctx: libxl context
 * @domid: domain id
 * @snapshot (OUTPUT): array of disk snapshot configuration.
 * @num (OUTPUT): number of disks.
 *
 * Helper function to produce an array of libxl_disk_snapshot.
 *
 * Could be used when user doesn't specify disk snapshot information
 * in creating a domain snapshot, it will by default fill disk snapshot
 * information according to domain disks:
 *   By default it will take internal disk snapshot to each domain
 *   disk that could take internal disk snapshot, otherwise, take
 *   external disk snapshot.
 * Then the result libxl_disk_snapshot array could be passed to
 * libxl_disk_snapshot_create to do the disk snapshot creating work.
 *
 * Returns 0 on success, <0 on failure.
 */
int libxl_disk_to_snapshot(libxl_ctx *ctx, uint32_t domid,
                           libxl_disk_snapshot **snapshot, int *num);



3. Simple Architecture View

Creating domain snapshot:
(* means new functions we will need in libxl)

  "xl snapshot-create"
         |
  parse configuration ----> libxl_disk_to_snapshot (*)
         |
  saving memory ----> libxl_domain_suspend
         |
 taking disk snapshot ----> libxl_disk_snapshot_create (*)
         |                     |
         |                     --> libxl_qmp_disk_snapshot_transaction (*)
         |
    unpause domain ---->libxl_domain_unpause
         |
        End


Reverting to a snapshot:
(* means new functions we will need in libxl)

  "xl snapshot-revert"
         |
   parse configuration
         |
   destroy domain ---->libxl_domain_destroy
         |
 reverting disk snapshot ----> libxl_disk_snapshot_revert (*)
         |                       |
         |                       --> call 'qemu-img' to apply disk snapshot
         |
 restore domain from saved memory ----> libxl_domain_create_restore
         |
        End

Re: [RFC V10 4/4] domain snapshot design: libxl

[Ian Campbell]

On Mon, 2015-01-26 at 11:25 +0800, Chunyan Liu wrote:
> Changes to V9:
>   * update libxl_disk_snapshot structures as Ian suggests
>   * add more descriptions for libxl_disk_snapshot_delete
>   * add two functions for deleting external disk snapshot:
>     libxl_domain_block_rebase and libxl_domain_block_commit
> 
> ===========================================================================
> 
> Libxl Design
> 
> 1. New Structures
> 
> libxl_disk_snapshot_type = Struct("disk_snapshot_type", [
>     (0, "default"),

How is default handled wrt the keyed union? In particualr if default
turns out to mean external, then external_path would need to be non-null
-- how is that fabricated?

Or did you mean s/default/unknown/ suggesting this would be an illegal
input?

>     (1, "internal"),
>     (2, "external"),
>     ])
> 
> libxl_disk_snapshot = Struct("disk_snapshot",[
>     # target disk
>     ("disk",            libxl_device_disk),
> 
>     # disk snapshot name
>     ("name",            string),
> 
>     ("u", KeyedUnion(None, libxl_disk_snapshot_type, "type",
>          [("external", Struct(None, [
> 
>             # disk format for external files. Since external disk snapshot is
>             # implemented with backing file mechanism, the external file disk
>             # format must support backing file. This field can be NULL, then
>             # a proper disk format will be used by default according to the
>             # orignal disk format.
>             ("external_format", libxl_disk_format),
> 
>             # external file path. This field should be non-NULL and a new path.
>             ("external_path",   string),
>             ]))
>          ]))
>     ])
> 
> 
> 2. New Functions
[...]

> /**
>  * libxl_disk_to_snapshot:
>  * @snapshot: array of disk snapshot configuration. Has "nb" members.
>  * @nb: number of disks.
>  * @ctx: libxl context
>  * @domid: domain id
>  * @snapshot (OUTPUT): array of disk snapshot configuration.
>  * @num (OUTPUT): number of disks.

@nb must equal @num, perhaps you actually meant for this to be an IN/OUT
parameter along with @snapshot instead of passing an IN array and an OUT
array?

>  *
>  * Helper function to produce an array of libxl_disk_snapshot.
>  *
>  * Could be used when user doesn't specify disk snapshot information
>  * in creating a domain snapshot, it will by default fill disk snapshot
>  * information according to domain disks:
>  *   By default it will take internal disk snapshot to each domain
>  *   disk that could take internal disk snapshot, otherwise, take
>  *   external disk snapshot.

How does this function determine the external path?

Re: [RFC V10 4/4] domain snapshot design: libxl

[Chun Yan Liu]

>>> On 1/30/2015 at 12:41 AM, in message <1422549669.10225.0.camel@citrix.com>, Ian
Campbell <Ian.Campbell@citrix.com> wrote: 
> On Mon, 2015-01-26 at 11:25 +0800, Chunyan Liu wrote: 
> > Changes to V9: 
> >   * update libxl_disk_snapshot structures as Ian suggests 
> >   * add more descriptions for libxl_disk_snapshot_delete 
> >   * add two functions for deleting external disk snapshot: 
> >     libxl_domain_block_rebase and libxl_domain_block_commit 
> >  
> > =========================================================================== 
> >  
> > Libxl Design 
> >  
> > 1. New Structures 
> >  
> > libxl_disk_snapshot_type = Struct("disk_snapshot_type", [ 
> >     (0, "default"), 
>  
> How is default handled wrt the keyed union? In particualr if default 
> turns out to mean external, then external_path would need to be non-null 
> -- how is that fabricated? 
>  
> Or did you mean s/default/unknown/ suggesting this would be an illegal 
> input? 

Yeah, it means invalid input. Should be 'unknown'. Will change.

>  
> >     (1, "internal"), 
> >     (2, "external"), 
> >     ]) 
> >  
> > libxl_disk_snapshot = Struct("disk_snapshot",[ 
> >     # target disk 
> >     ("disk",            libxl_device_disk), 
> >  
> >     # disk snapshot name 
> >     ("name",            string), 
> >  
> >     ("u", KeyedUnion(None, libxl_disk_snapshot_type, "type", 
> >          [("external", Struct(None, [ 
> >  
> >             # disk format for external files. Since external disk snapshot  
> is 
> >             # implemented with backing file mechanism, the external file  
> disk 
> >             # format must support backing file. This field can be NULL,  
> then 
> >             # a proper disk format will be used by default according to the 
> >             # orignal disk format. 
> >             ("external_format", libxl_disk_format), 
> >  
> >             # external file path. This field should be non-NULL and a new  
> path. 
> >             ("external_path",   string), 
> >             ])) 
> >          ])) 
> >     ]) 
> >  
> >  
> > 2. New Functions 
> [...] 
>  
> > /** 
> >  * libxl_disk_to_snapshot: 
> >  * @snapshot: array of disk snapshot configuration. Has "nb" members. 
> >  * @nb: number of disks. 
> >  * @ctx: libxl context 
> >  * @domid: domain id 
> >  * @snapshot (OUTPUT): array of disk snapshot configuration. 
> >  * @num (OUTPUT): number of disks. 
>  
> @nb must equal @num, perhaps you actually meant for this to be an IN/OUT 
> parameter along with @snapshot instead of passing an IN array and an OUT 
> array? 
>  
> >  * 
> >  * Helper function to produce an array of libxl_disk_snapshot. 
> >  * 
> >  * Could be used when user doesn't specify disk snapshot information 
> >  * in creating a domain snapshot, it will by default fill disk snapshot 
> >  * information according to domain disks: 
> >  *   By default it will take internal disk snapshot to each domain 
> >  *   disk that could take internal disk snapshot, otherwise, take 
> >  *   external disk snapshot. 
>  
> How does this function determine the external path? 

Meant to let user to fill in snapshot name and external path after calling this
to set default. But yes, it's better to do all work in this function. Will add
two parameters 'path' and 'name'.
Then it will be:
int libxl_disk_to_snapshot(libxl_ctx *ctx, uint32_t domid,
                           const char *path, const char *name,
                           libxl_disk_snapshot **snapshot, int *num)

@path: path to store vm snapshot data. @path can be NULL if all disks
           support internal disk snapshots; otherwise, @path shouldn't be
           NULL.
@name: vm snapshot name. All disks will use this name as disk snapshot
           name, external disk snapshot will use this name to generate external
           file name.

Thanks!

- Chunyan

>  
>  
>  
>  

Re: [RFC V10 0/4] domain snapshot document

[Ian Campbell]

On Mon, 2015-01-26 at 11:25 +0800, Chunyan Liu wrote:
> Changes to V9:

This looks good to me, thanks.

Ian/Wei do you have any comments? xapi folks?

>   - add details and background knowledge in overview (2/4).
>   - update xl snapshot-create syntax and cfg syntax (3/4)
>   - update libxl_disk_snapshot structure, add functions for
>     deleting external disk snapshots, more description about
>     deleting disk snapshot (4/4)
> 
> V9:
>   http://comments.gmane.org/gmane.comp.emulators.xen.devel/225919

Re: [RFC V10 0/4] domain snapshot document

[Wei Liu]

On Thu, Jan 29, 2015 at 04:36:43PM +0000, Ian Campbell wrote:
> On Mon, 2015-01-26 at 11:25 +0800, Chunyan Liu wrote:
> > Changes to V9:
> 
> This looks good to me, thanks.
> 
> Ian/Wei do you have any comments? xapi folks?
> 

Looks good to me.

Wei.

Re: [RFC V10 0/4] domain snapshot document

[Fabio Fantoni]

Il 30/01/2015 14:38, Wei Liu ha scritto:
> On Thu, Jan 29, 2015 at 04:36:43PM +0000, Ian Campbell wrote:
>> On Mon, 2015-01-26 at 11:25 +0800, Chunyan Liu wrote:
>>> Changes to V9:
>> This looks good to me, thanks.
>>
>> Ian/Wei do you have any comments? xapi folks?
>>
> Looks good to me.
>
> Wei.

I looked to the project of domain snapshot and seems good.

I have only a question.
I saw this:
> Disk snapshot can be created by many external tools, like qemu-img,
> vhd-util and lvm, etc.
Wouldn't it be good to also use btrfs for external disks snapshot?
Even if btrfs snapshots works on subvolumes (directories) it is possible 
to atomically creates a copy-on-write snapshot of a file (with domUs 
disks on file, including raw) with btrfs clone operation (reflink 
systemcall, eg: cp --reflink source dest)


>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [RFC V10 0/4] domain snapshot document

[Wei Liu]

On Fri, Jan 30, 2015 at 03:01:11PM +0100, Fabio Fantoni wrote:
> Il 30/01/2015 14:38, Wei Liu ha scritto:
> >On Thu, Jan 29, 2015 at 04:36:43PM +0000, Ian Campbell wrote:
> >>On Mon, 2015-01-26 at 11:25 +0800, Chunyan Liu wrote:
> >>>Changes to V9:
> >>This looks good to me, thanks.
> >>
> >>Ian/Wei do you have any comments? xapi folks?
> >>
> >Looks good to me.
> >
> >Wei.
> 
> I looked to the project of domain snapshot and seems good.
> 
> I have only a question.
> I saw this:
> >Disk snapshot can be created by many external tools, like qemu-img,
> >vhd-util and lvm, etc.
> Wouldn't it be good to also use btrfs for external disks snapshot?
> Even if btrfs snapshots works on subvolumes (directories) it is possible to
> atomically creates a copy-on-write snapshot of a file (with domUs disks on
> file, including raw) with btrfs clone operation (reflink systemcall, eg: cp
> --reflink source dest)
> 

I think that "etc." potentially covers all other external tools,
including btrfs. You can list all possible external tools in the world.

Wei.

> 
> >
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@lists.xen.org
> >http://lists.xen.org/xen-devel

Re: [RFC V10 0/4] domain snapshot document

[Wei Liu]

On Fri, Jan 30, 2015 at 02:05:05PM +0000, Wei Liu wrote:
> On Fri, Jan 30, 2015 at 03:01:11PM +0100, Fabio Fantoni wrote:
> > Il 30/01/2015 14:38, Wei Liu ha scritto:
> > >On Thu, Jan 29, 2015 at 04:36:43PM +0000, Ian Campbell wrote:
> > >>On Mon, 2015-01-26 at 11:25 +0800, Chunyan Liu wrote:
> > >>>Changes to V9:
> > >>This looks good to me, thanks.
> > >>
> > >>Ian/Wei do you have any comments? xapi folks?
> > >>
> > >Looks good to me.
> > >
> > >Wei.
> > 
> > I looked to the project of domain snapshot and seems good.
> > 
> > I have only a question.
> > I saw this:
> > >Disk snapshot can be created by many external tools, like qemu-img,
> > >vhd-util and lvm, etc.
> > Wouldn't it be good to also use btrfs for external disks snapshot?
> > Even if btrfs snapshots works on subvolumes (directories) it is possible to
> > atomically creates a copy-on-write snapshot of a file (with domUs disks on
> > file, including raw) with btrfs clone operation (reflink systemcall, eg: cp
> > --reflink source dest)
> > 
> 
> I think that "etc." potentially covers all other external tools,
> including btrfs. You can list all possible external tools in the world.
                       ^^^
                       can't
> 
> Wei.
> 
> > 
> > >
> > >_______________________________________________
> > >Xen-devel mailing list
> > >Xen-devel@lists.xen.org
> > >http://lists.xen.org/xen-devel

Re: [RFC V10 0/4] domain snapshot document

[Dave Scott]

> On 29 Jan 2015, at 16:36, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> 
> On Mon, 2015-01-26 at 11:25 +0800, Chunyan Liu wrote:
>> Changes to V9:
> 
> This looks good to me, thanks.
> 
> Ian/Wei do you have any comments? xapi folks?

The API looks sensible to me.

Xapi normally manages disk snapshots through it’s own plug-in mechanism. Will it still be possible to

* libxl_domain_suspend
* do custom snapshot stuff involving tapdisk/ ceph/ ...
* libxl_domain_create_restore
?

If so then everything is fine for us. We might use the built-in support for libxl snapshots for some cases when it arrives.

Cheers,
Dave


> 
>>  - add details and background knowledge in overview (2/4).
>>  - update xl snapshot-create syntax and cfg syntax (3/4)
>>  - update libxl_disk_snapshot structure, add functions for
>>    deleting external disk snapshots, more description about
>>    deleting disk snapshot (4/4)
>> 
>> V9:
>>  http://comments.gmane.org/gmane.comp.emulators.xen.devel/225919
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [RFC V10 0/4] domain snapshot document

[Ian Campbell]

On Sun, 2015-02-01 at 20:55 +0000, Dave Scott wrote:
> 
> > On 29 Jan 2015, at 16:36, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > 
> > On Mon, 2015-01-26 at 11:25 +0800, Chunyan Liu wrote:
> >> Changes to V9:
> > 
> > This looks good to me, thanks.
> > 
> > Ian/Wei do you have any comments? xapi folks?
> 
> The API looks sensible to me.
> 
> Xapi normally manages disk snapshots through it’s own plug-in mechanism. Will it still be possible to
> 
> * libxl_domain_suspend
> * do custom snapshot stuff involving tapdisk/ ceph/ ...
> * libxl_domain_create_restore
> ?
> 
> If so then everything is fine for us.

I think so, but I'll let Chunyan confirm for sure.

The main thing which you cannot do without going through libxl is
operations which involve talking to a live domain, i.e. anything which
might talk QMP to the associate qemu process.

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel