From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54568) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aWReO-0000AQ-8z for qemu-devel@nongnu.org; Thu, 18 Feb 2016 11:41:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aWReJ-00022e-Lc for qemu-devel@nongnu.org; Thu, 18 Feb 2016 11:41:56 -0500 Received: from mail-wm0-x230.google.com ([2a00:1450:400c:c09::230]:37463) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aWReJ-00022T-Be for qemu-devel@nongnu.org; Thu, 18 Feb 2016 11:41:51 -0500 Received: by mail-wm0-x230.google.com with SMTP id g62so34080717wme.0 for ; Thu, 18 Feb 2016 08:41:51 -0800 (PST) Date: Thu, 18 Feb 2016 16:41:48 +0000 From: Stefan Hajnoczi Message-ID: <20160218164148.GB13271@stefanha-x1.localdomain> References: <1454151394-52320-1-git-send-email-vsementsov@virtuozzo.com> <20160203081418.GC25746@ad.usersys.redhat.com> <56B45D3A.405@virtuozzo.com> <20160209142852.GA13149@stefanha-x1.localdomain> <56B9FAAE.8040503@virtuozzo.com> <20160210101004.GB7317@stefanha-x1.localdomain> <20160216170943.GA31393@stefanha-x1.localdomain> <56C4B21F.9030006@virtuozzo.com> <20160218121114.GC12470@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KFztAG8eRSV9hGtP" Content-Disposition: inline In-Reply-To: <20160218121114.GC12470@redhat.com> Subject: Re: [Qemu-devel] [PATCH v2 0/6] external backup api List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Daniel P. Berrange" Cc: kwolf@redhat.com, Vladimir Sementsov-Ogievskiy , Fam Zheng , "Denis V. Lunev" , qemu-devel@nongnu.org, armbru@redhat.com, jsnow@redhat.com --KFztAG8eRSV9hGtP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 18, 2016 at 12:11:14PM +0000, Daniel P. Berrange wrote: > On Wed, Feb 17, 2016 at 08:47:11PM +0300, Vladimir Sementsov-Ogievskiy wr= ote: > > On 16.02.2016 20:09, Stefan Hajnoczi wrote: > > >On Wed, Feb 10, 2016 at 10:10:04AM +0000, Stefan Hajnoczi wrote: > > >>On Tue, Feb 09, 2016 at 05:41:50PM +0300, Denis V. Lunev wrote: > > >>>On 02/09/2016 05:28 PM, Stefan Hajnoczi wrote: > > >>>>On Fri, Feb 05, 2016 at 11:28:42AM +0300, Denis V. Lunev wrote: > > >>>>>On 02/03/2016 11:14 AM, Fam Zheng wrote: > > >>>>>>On Sat, 01/30 13:56, Vladimir Sementsov-Ogievskiy wrote: > > >>>>>>>Hi all. > > >>>>>>> > > >>>>>>>These series which aims to add external backup api. This is need= ed to allow > > >>>>>>>backup software use our dirty bitmaps. > > >>>>>>> > > >>>>>>>Vmware and Parallels Cloud Server have this feature. > > >>>>>>What is the advantage of this appraoch over "drive-backup sync=3D= incremental > > >>>>>>..."? > > >>>>>This will allow third-party vendors to backup QEMU VMs into > > >>>>>their own formats or to the cloud etc. > > >>>>As an example, there is a third-party backup format called VMA from > > >>>>Proxmox. A few years ago I posted a proof-of-concept external back= up > > >>>>tool in Python: > > >>>> > > >>>>https://lists.gnu.org/archive/html/qemu-devel/2013-03/msg01536.html > > >>>> > > >>>>It takes a full backup using drive-backup NBD (plus RAM/device stat= e) > > >>>>but the same can be done with incremental backups. > > >>>> > > >>>>Does this NBD approach meet your requirements? > > >>>> > > >>>>Stefan > > >>>for us we should somehow provide implementation of > > >>>calls posted by Vladimir. They are available in Parallels Server > > >>>version 6 and should be available in the next QEMU based > > >>>release using "Parallels SDK to libvirt" convertor. The problem > > >>>for us is that this old approach is used in the other side > > >>>of the product - in containers implementation while this > > >>>SDK is a universal access tool to both things. > > >>Point taken. I think many other backup applications will expect a > > >>similar API, so it's pragmatic to provide something compatible. > > >Kevin Wolf and Daniel Berrange proposed an elegant way to avoid the > > >concerns about the QMP monitor: > > > > > >Previously I described incremental backup in "push" mode (already > > >supported today with drive-backup). QEMU connects to a remote NBD > > >server and writes out the contents of all dirty blocks: > > > > > > QEMU ---Write dirty blocks--> Backup appliance (server) > > > > > >This doesn't lend itself well to existing backup applications that > > >expect to iterate the dirty bitmap manually. > > > > > >Let's add a "pull" mode where the connection of the NBD connection is > > >reversed. The backup application connects to QEMU's NBD server (image > > >fleecing). The NBD protocol is extended to support the SCSI Get LBA > > >Status command for querying block provisioning information. Now the > > >backup application can use Get LBA Status to fetch the dirty block > > >information from QEMU. > > > > > > QEMU (server) <--Get LBA Status or Read dirty blocks-- Backup appli= ance > > > > > >The dirty block information goes over the same NBD connection used to > > >read the contents of the dirty blocks. The QMP monitor is not used to > > >transfer dirty block information. > > > > > >It may be necessary to extend the nbd-server-add command so that a > > >bitmap name can be passed. This bitmap will be used to answer Get LBA > > >Status queries instead of using on bdrv_co_get_block_status(). This > > >would be necessary if image fleecing (point in time snapshot) is used. > > > > > >Stefan > >=20 > > There are no such commands in nbd spec here: > >=20 > > https://github.com/yoe/nbd/blob/master/doc/proto.md > >=20 > >=20 > > So, I'm not sure, that adding something qemu-specific to this external > > protocol will be simple or even true way. Is Qemu already extending ori= ginal > > nbd? >=20 > No, we don't do any QEMU specific extensions. The point of the approach > Stefan suggests here though, is that it is *not* an inherantly QEMU-speci= fic > concept, it is relevant to any NBD server implementation. >=20 > For example, consider you were using a regular NBD server to export a > sparse LVM volume. This proposed extension would be relevant to such > a use case. As such this proposed extension is something that is likely > to be acceptable for the generic NBD specification. Yes, Get LBA Status could be useful for non-QEMU NBD users too. NBD already supports a TRIM command so the ability to query the allocation status is a natural feature to add. Stefan --KFztAG8eRSV9hGtP Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJWxfRMAAoJEJykq7OBq3PIRKMIAL8eW6Kab2x1sw9I7ygofUtv bPN50SuwuEmlBO03nmUNd1Am7DLNTPICD08V8UQqU3Mulj8wf7c/g9E2eQcre6o1 dnqWdYx4USSFlT1rnuL5SAma4uF7JU+I8NFlZLImeh24uRC/FXvBtUr453nEIqQ9 xSsKz9+JXVNMPmAVFAX9PWjfw55PTVvoFOa05iAh29gh2poWKVrWnFLaixO9MET5 Vbo0J+JCBZCLN/s6PCIiKa/bwDZVJPwRXD8LEabBj94fucM8hgI7/U6cOmop0u2e 1BeHPwXeA4X+bdqkD5tsTH5u1JVpNQKVF2Ek6Mgxqbozp5wdtQJBmv8qgBC7rhA= =bvRR -----END PGP SIGNATURE----- --KFztAG8eRSV9hGtP--