From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59059) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aVjJ9-000750-Li for qemu-devel@nongnu.org; Tue, 16 Feb 2016 12:21:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aVjJ5-0002Y4-JD for qemu-devel@nongnu.org; Tue, 16 Feb 2016 12:21:03 -0500 Received: from mx2.parallels.com ([199.115.105.18]:46059) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aVjJ5-0002Xj-Cn for qemu-devel@nongnu.org; Tue, 16 Feb 2016 12:20:59 -0500 References: <1454151394-52320-1-git-send-email-vsementsov@virtuozzo.com> <20160203081418.GC25746@ad.usersys.redhat.com> <56B45D3A.405@virtuozzo.com> <20160209142852.GA13149@stefanha-x1.localdomain> <56B9FAAE.8040503@virtuozzo.com> <20160210101004.GB7317@stefanha-x1.localdomain> <20160216170943.GA31393@stefanha-x1.localdomain> From: "Denis V. Lunev" Message-ID: <56C35A6C.8090200@virtuozzo.com> Date: Tue, 16 Feb 2016 20:20:44 +0300 MIME-Version: 1.0 In-Reply-To: <20160216170943.GA31393@stefanha-x1.localdomain> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v2 0/6] external backup api List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefan Hajnoczi Cc: kwolf@redhat.com, Vladimir Sementsov-Ogievskiy , Fam Zheng , qemu-devel@nongnu.org, armbru@redhat.com, jsnow@redhat.com On 02/16/2016 08:09 PM, Stefan Hajnoczi wrote: > On Wed, Feb 10, 2016 at 10:10:04AM +0000, Stefan Hajnoczi wrote: >> On Tue, Feb 09, 2016 at 05:41:50PM +0300, Denis V. Lunev wrote: >>> On 02/09/2016 05:28 PM, Stefan Hajnoczi wrote: >>>> On Fri, Feb 05, 2016 at 11:28:42AM +0300, Denis V. Lunev wrote: >>>>> On 02/03/2016 11:14 AM, Fam Zheng wrote: >>>>>> On Sat, 01/30 13:56, Vladimir Sementsov-Ogievskiy wrote: >>>>>>> Hi all. >>>>>>> >>>>>>> These series which aims to add external backup api. This is needed to allow >>>>>>> backup software use our dirty bitmaps. >>>>>>> >>>>>>> Vmware and Parallels Cloud Server have this feature. >>>>>> What is the advantage of this appraoch over "drive-backup sync=incremental >>>>>> ..."? >>>>> This will allow third-party vendors to backup QEMU VMs into >>>>> their own formats or to the cloud etc. >>>> As an example, there is a third-party backup format called VMA from >>>> Proxmox. A few years ago I posted a proof-of-concept external backup >>>> tool in Python: >>>> >>>> https://lists.gnu.org/archive/html/qemu-devel/2013-03/msg01536.html >>>> >>>> It takes a full backup using drive-backup NBD (plus RAM/device state) >>>> but the same can be done with incremental backups. >>>> >>>> Does this NBD approach meet your requirements? >>>> >>>> Stefan >>> for us we should somehow provide implementation of >>> calls posted by Vladimir. They are available in Parallels Server >>> version 6 and should be available in the next QEMU based >>> release using "Parallels SDK to libvirt" convertor. The problem >>> for us is that this old approach is used in the other side >>> of the product - in containers implementation while this >>> SDK is a universal access tool to both things. >> Point taken. I think many other backup applications will expect a >> similar API, so it's pragmatic to provide something compatible. > Kevin Wolf and Daniel Berrange proposed an elegant way to avoid the > concerns about the QMP monitor: > > Previously I described incremental backup in "push" mode (already > supported today with drive-backup). QEMU connects to a remote NBD > server and writes out the contents of all dirty blocks: > > QEMU ---Write dirty blocks--> Backup appliance (server) > > This doesn't lend itself well to existing backup applications that > expect to iterate the dirty bitmap manually. > > Let's add a "pull" mode where the connection of the NBD connection is > reversed. The backup application connects to QEMU's NBD server (image > fleecing). The NBD protocol is extended to support the SCSI Get LBA > Status command for querying block provisioning information. Now the > backup application can use Get LBA Status to fetch the dirty block > information from QEMU. > > QEMU (server) <--Get LBA Status or Read dirty blocks-- Backup appliance > > The dirty block information goes over the same NBD connection used to > read the contents of the dirty blocks. The QMP monitor is not used to > transfer dirty block information. > > It may be necessary to extend the nbd-server-add command so that a > bitmap name can be passed. This bitmap will be used to answer Get LBA > Status queries instead of using on bdrv_co_get_block_status(). This > would be necessary if image fleecing (point in time snapshot) is used. > > Stefan the idea seems feasible at the first glance. I'll need to check the amount of the overhead of this approach. If we'll need to perform an additional request for each block - the overhead is toooooo much. Do you have the link to the spec/implementation? Den