On Thu, Feb 18, 2016 at 12:11:14PM +0000, Daniel P. Berrange wrote:
> On Wed, Feb 17, 2016 at 08:47:11PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > On 16.02.2016 20:09, Stefan Hajnoczi wrote:
> > >On Wed, Feb 10, 2016 at 10:10:04AM +0000, Stefan Hajnoczi wrote:
> > >>On Tue, Feb 09, 2016 at 05:41:50PM +0300, Denis V. Lunev wrote:
> > >>>On 02/09/2016 05:28 PM, Stefan Hajnoczi wrote:
> > >>>>On Fri, Feb 05, 2016 at 11:28:42AM +0300, Denis V. Lunev wrote:
> > >>>>>On 02/03/2016 11:14 AM, Fam Zheng wrote:
> > >>>>>>On Sat, 01/30 13:56, Vladimir Sementsov-Ogievskiy wrote:
> > >>>>>>>Hi all.
> > >>>>>>>
> > >>>>>>>These series which aims to add external backup api. This is needed to allow
> > >>>>>>>backup software use our dirty bitmaps.
> > >>>>>>>
> > >>>>>>>Vmware and Parallels Cloud Server have this feature.
> > >>>>>>What is the advantage of this appraoch over "drive-backup sync=incremental
> > >>>>>>..."?
> > >>>>>This will allow third-party vendors to backup QEMU VMs into
> > >>>>>their own formats or to the cloud etc.
> > >>>>As an example, there is a third-party backup format called VMA from
> > >>>>Proxmox.  A few years ago I posted a proof-of-concept external backup
> > >>>>tool in Python:
> > >>>>
> > >>>>https://lists.gnu.org/archive/html/qemu-devel/2013-03/msg01536.html
> > >>>>
> > >>>>It takes a full backup using drive-backup NBD (plus RAM/device state)
> > >>>>but the same can be done with incremental backups.
> > >>>>
> > >>>>Does this NBD approach meet your requirements?
> > >>>>
> > >>>>Stefan
> > >>>for us we should somehow provide implementation of
> > >>>calls posted by Vladimir. They are available in Parallels Server
> > >>>version 6 and should be available in the next QEMU based
> > >>>release using "Parallels SDK to libvirt" convertor. The problem
> > >>>for us is that this old approach is used in the other side
> > >>>of the product - in containers implementation while this
> > >>>SDK is a universal access tool to both things.
> > >>Point taken.  I think many other backup applications will expect a
> > >>similar API, so it's pragmatic to provide something compatible.
> > >Kevin Wolf and Daniel Berrange proposed an elegant way to avoid the
> > >concerns about the QMP monitor:
> > >
> > >Previously I described incremental backup in "push" mode (already
> > >supported today with drive-backup).  QEMU connects to a remote NBD
> > >server and writes out the contents of all dirty blocks:
> > >
> > >   QEMU ---Write dirty blocks--> Backup appliance (server)
> > >
> > >This doesn't lend itself well to existing backup applications that
> > >expect to iterate the dirty bitmap manually.
> > >
> > >Let's add a "pull" mode where the connection of the NBD connection is
> > >reversed.  The backup application connects to QEMU's NBD server (image
> > >fleecing).  The NBD protocol is extended to support the SCSI Get LBA
> > >Status command for querying block provisioning information.  Now the
> > >backup application can use Get LBA Status to fetch the dirty block
> > >information from QEMU.
> > >
> > >   QEMU (server) <--Get LBA Status or Read dirty blocks-- Backup appliance
> > >
> > >The dirty block information goes over the same NBD connection used to
> > >read the contents of the dirty blocks.  The QMP monitor is not used to
> > >transfer dirty block information.
> > >
> > >It may be necessary to extend the nbd-server-add command so that a
> > >bitmap name can be passed.  This bitmap will be used to answer Get LBA
> > >Status queries instead of using on bdrv_co_get_block_status().  This
> > >would be necessary if image fleecing (point in time snapshot) is used.
> > >
> > >Stefan
> > 
> > There are no such commands in nbd spec here:
> > 
> > https://github.com/yoe/nbd/blob/master/doc/proto.md
> > 
> > 
> > So, I'm not sure, that adding something qemu-specific to this external
> > protocol will be simple or even true way. Is Qemu already extending original
> > nbd?
> 
> No, we don't do any QEMU specific extensions. The point of the approach
> Stefan suggests here though, is that it is *not* an inherantly QEMU-specific
> concept, it is relevant to any NBD server implementation.
> 
> For example, consider you were using a regular NBD server to export a
> sparse LVM volume. This proposed extension would be relevant to such
> a use case. As such this proposed extension is something that is likely
> to be acceptable for the generic NBD specification.

Yes, Get LBA Status could be useful for non-QEMU NBD users too.

NBD already supports a TRIM command so the ability to query the
allocation status is a natural feature to add.

Stefan