From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37172) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aWaUn-0005bP-05 for qemu-devel@nongnu.org; Thu, 18 Feb 2016 21:08:38 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aWaUi-0008Q0-UI for qemu-devel@nongnu.org; Thu, 18 Feb 2016 21:08:36 -0500 Received: from mx1.redhat.com ([209.132.183.28]:43977) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aWaUi-0008Pw-Mw for qemu-devel@nongnu.org; Thu, 18 Feb 2016 21:08:32 -0500 Date: Fri, 19 Feb 2016 10:08:26 +0800 From: Fam Zheng Message-ID: <20160219020826.GA23506@ad.usersys.redhat.com> References: <1454151394-52320-1-git-send-email-vsementsov@virtuozzo.com> <20160203081418.GC25746@ad.usersys.redhat.com> <56B45D3A.405@virtuozzo.com> <20160209142852.GA13149@stefanha-x1.localdomain> <56B9FAAE.8040503@virtuozzo.com> <20160210101004.GB7317@stefanha-x1.localdomain> <20160216170943.GA31393@stefanha-x1.localdomain> <56C4B21F.9030006@virtuozzo.com> <20160218121114.GC12470@redhat.com> <20160218164148.GB13271@stefanha-x1.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160218164148.GB13271@stefanha-x1.localdomain> Subject: Re: [Qemu-devel] [PATCH v2 0/6] external backup api List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefan Hajnoczi Cc: kwolf@redhat.com, Vladimir Sementsov-Ogievskiy , "Denis V. Lunev" , qemu-devel@nongnu.org, armbru@redhat.com, jsnow@redhat.com On Thu, 02/18 16:41, Stefan Hajnoczi wrote: > On Thu, Feb 18, 2016 at 12:11:14PM +0000, Daniel P. Berrange wrote: > > On Wed, Feb 17, 2016 at 08:47:11PM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > On 16.02.2016 20:09, Stefan Hajnoczi wrote: > > > >On Wed, Feb 10, 2016 at 10:10:04AM +0000, Stefan Hajnoczi wrote: > > > >>On Tue, Feb 09, 2016 at 05:41:50PM +0300, Denis V. Lunev wrote: > > > >>>On 02/09/2016 05:28 PM, Stefan Hajnoczi wrote: > > > >>>>On Fri, Feb 05, 2016 at 11:28:42AM +0300, Denis V. Lunev wrote: > > > >>>>>On 02/03/2016 11:14 AM, Fam Zheng wrote: > > > >>>>>>On Sat, 01/30 13:56, Vladimir Sementsov-Ogievskiy wrote: > > > >>>>>>>Hi all. > > > >>>>>>> > > > >>>>>>>These series which aims to add external backup api. This is needed to allow > > > >>>>>>>backup software use our dirty bitmaps. > > > >>>>>>> > > > >>>>>>>Vmware and Parallels Cloud Server have this feature. > > > >>>>>>What is the advantage of this appraoch over "drive-backup sync=incremental > > > >>>>>>..."? > > > >>>>>This will allow third-party vendors to backup QEMU VMs into > > > >>>>>their own formats or to the cloud etc. > > > >>>>As an example, there is a third-party backup format called VMA from > > > >>>>Proxmox. A few years ago I posted a proof-of-concept external backup > > > >>>>tool in Python: > > > >>>> > > > >>>>https://lists.gnu.org/archive/html/qemu-devel/2013-03/msg01536.html > > > >>>> > > > >>>>It takes a full backup using drive-backup NBD (plus RAM/device state) > > > >>>>but the same can be done with incremental backups. > > > >>>> > > > >>>>Does this NBD approach meet your requirements? > > > >>>> > > > >>>>Stefan > > > >>>for us we should somehow provide implementation of > > > >>>calls posted by Vladimir. They are available in Parallels Server > > > >>>version 6 and should be available in the next QEMU based > > > >>>release using "Parallels SDK to libvirt" convertor. The problem > > > >>>for us is that this old approach is used in the other side > > > >>>of the product - in containers implementation while this > > > >>>SDK is a universal access tool to both things. > > > >>Point taken. I think many other backup applications will expect a > > > >>similar API, so it's pragmatic to provide something compatible. > > > >Kevin Wolf and Daniel Berrange proposed an elegant way to avoid the > > > >concerns about the QMP monitor: > > > > > > > >Previously I described incremental backup in "push" mode (already > > > >supported today with drive-backup). QEMU connects to a remote NBD > > > >server and writes out the contents of all dirty blocks: > > > > > > > > QEMU ---Write dirty blocks--> Backup appliance (server) > > > > > > > >This doesn't lend itself well to existing backup applications that > > > >expect to iterate the dirty bitmap manually. > > > > > > > >Let's add a "pull" mode where the connection of the NBD connection is > > > >reversed. The backup application connects to QEMU's NBD server (image > > > >fleecing). The NBD protocol is extended to support the SCSI Get LBA > > > >Status command for querying block provisioning information. Now the > > > >backup application can use Get LBA Status to fetch the dirty block > > > >information from QEMU. > > > > > > > > QEMU (server) <--Get LBA Status or Read dirty blocks-- Backup appliance > > > > > > > >The dirty block information goes over the same NBD connection used to > > > >read the contents of the dirty blocks. The QMP monitor is not used to > > > >transfer dirty block information. > > > > > > > >It may be necessary to extend the nbd-server-add command so that a > > > >bitmap name can be passed. This bitmap will be used to answer Get LBA > > > >Status queries instead of using on bdrv_co_get_block_status(). This > > > >would be necessary if image fleecing (point in time snapshot) is used. > > > > > > > >Stefan > > > > > > There are no such commands in nbd spec here: > > > > > > https://github.com/yoe/nbd/blob/master/doc/proto.md > > > > > > > > > So, I'm not sure, that adding something qemu-specific to this external > > > protocol will be simple or even true way. Is Qemu already extending original > > > nbd? > > > > No, we don't do any QEMU specific extensions. The point of the approach > > Stefan suggests here though, is that it is *not* an inherantly QEMU-specific > > concept, it is relevant to any NBD server implementation. > > > > For example, consider you were using a regular NBD server to export a > > sparse LVM volume. This proposed extension would be relevant to such > > a use case. As such this proposed extension is something that is likely > > to be acceptable for the generic NBD specification. > > Yes, Get LBA Status could be useful for non-QEMU NBD users too. > > NBD already supports a TRIM command so the ability to query the > allocation status is a natural feature to add. Is it an abuse to "Get LBA Status" to return dirty information? Because in SCSI the command reports "mapped", "allocated" and "anchored" statuses. Does that mean NBD will use a different status set? Fam