From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48058) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aZQ4v-0008Ey-Fv for qemu-devel@nongnu.org; Fri, 26 Feb 2016 16:37:38 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aZQ4s-0007tm-9J for qemu-devel@nongnu.org; Fri, 26 Feb 2016 16:37:37 -0500 Received: from mx1.redhat.com ([209.132.183.28]:39430) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aZQ4s-0007tN-1u for qemu-devel@nongnu.org; Fri, 26 Feb 2016 16:37:34 -0500 References: <1454151394-52320-1-git-send-email-vsementsov@virtuozzo.com> <20160203081418.GC25746@ad.usersys.redhat.com> <56B45D3A.405@virtuozzo.com> <20160209142852.GA13149@stefanha-x1.localdomain> <56B9FAAE.8040503@virtuozzo.com> <20160210101004.GB7317@stefanha-x1.localdomain> <20160216170943.GA31393@stefanha-x1.localdomain> <56C4B21F.9030006@virtuozzo.com> <20160218121114.GC12470@redhat.com> <20160218164148.GB13271@stefanha-x1.localdomain> <20160219020826.GA23506@ad.usersys.redhat.com> <87povtulpt.fsf@blackfin.pond.sub.org> <56D0ADBD.20000@redhat.com> <56D0AF92.7000509@redhat.com> From: John Snow Message-ID: <56D0C59B.4060900@redhat.com> Date: Fri, 26 Feb 2016 16:37:31 -0500 MIME-Version: 1.0 In-Reply-To: <56D0AF92.7000509@redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v2 0/6] external backup api List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini , Markus Armbruster , Fam Zheng Cc: kwolf@redhat.com, Stefan Hajnoczi , Vladimir Sementsov-Ogievskiy , "Denis V. Lunev" , qemu-devel@nongnu.org On 02/26/2016 03:03 PM, Paolo Bonzini wrote: > > > On 26/02/2016 20:55, Paolo Bonzini wrote: >> >> >> On 19/02/2016 09:51, Markus Armbruster wrote: >>>> Is it an abuse to "Get LBA Status" to return dirty information? Because in SCSI >>>> the command reports "mapped", "allocated" and "anchored" statuses. Does that >>>> mean NBD will use a different status set? >>> >>> Perhaps some conceptual gymnastics can get us to standard semantics. >>> >>> Incremental backup wants to copy out an image's "dirty" blocks. >>> >>> We can view this as a bitmap telling us which of the image's blocks are >>> dirty. >>> >>> An alternative view would be base image + dirty delta image. In the the >>> dirty delta, exactly the dirty blocks are allocated. The delta image >>> may be conceptual. >> >> I see a problem here. On one hand I agree that the "GET LBA STATUS" is >> a natural extension to the NBD protocol. >> >> On the other hand, the Get LBA Status command in SCSI reflects the >> state over the whole chain, not only the top element. It is the >> equivalent of bdrv_get_block_status_above(bs, NULL, ...), rather than >> bdrv_get_block_status(bs, ...). My understanding is that the dirty >> block information would require the latter, especially in the >> "conceptual delta image" model that Markus describes above. >> This is unfortunate. >> Having NBD implement subtly different semantics compared to SCSI is a >> bad idea in my opinion. >> >> Of course if we call it "READ DIRTY BLOCKS" that would work, but I >> don't think such a command would be something that it makes sense to >> have in the general purpose NBD spec, so you would need a mechanism >> for vendor-specific extensions. > > Trying to be constructive: shall we instead have a simple mini-protocol > with commands like "read bitmap slice", "clear bitmap slice", "query > next 0 bit", "query next 1 bit"? > > Not necessarily QMP, just a little socket thing. It could be JSON or > ASCII or binary. It sucks to implement a new protocol, but perhaps it > could be something compatible with VMware or Parallels. > > Sorry if this has already been proposed, I'm late to the game and I only > read this subthread. > > Paolo > The solution I proposed prior to the LBA status idea (which I did like, pending its suitability which looks... compromised, now. sigh.) was a simple binary protocol. Something akin to this: { "execute": "dirty-bitmap-export", "arguments": { "node": "drive0", "bitmap": "bitmap0", "uri": "tcp:127.0.0.1:31337" } } We'll open up a socket to the URI provided, and push data akin to the following: Bytes 0-3: "QBMP" (Qemu Bit Map Protocol) Bytes 4-11: number of dirty bits (As a lazy checksum) Bytes 12-19: granularity of bits Bytes 20+: raw bitmap data EOF A simple, non-glorious format that just gets the data out there. Then the client has the info it needs to poll dirty blocks as necessary. The other suggestion is to literally fetch the bitmap over NBD. How would this work -- would the export just be sized to the number of bytes of the bitmap data, and the client queries it? I can't help but feel like this is slightly hacky, but I can't think of any truly solid reasons against it either. --js