From: John Snow <jsnow@redhat.com>
To: qemu-devel@nongnu.org, qemu-block@nongnu.org
Cc: Fam Zheng <fam@euphon.net>, Kevin Wolf <kwolf@redhat.com>,
vsementsov@virtuozzo.com, Juan Quintela <quintela@redhat.com>,
Wen Congyang <wencongyang2@huawei.com>,
Xie Changlong <xiechanglong.d@gmail.com>,
"Dr. David Alan Gilbert" <dgilbert@redhat.com>,
Markus Armbruster <armbru@redhat.com>,
John Snow <jsnow@redhat.com>,
Stefan Hajnoczi <stefanha@redhat.com>,
Max Reitz <mreitz@redhat.com>
Subject: [Qemu-devel] [PATCH v3 00/18] bitmaps: introduce 'bitmap' sync mode
Date: Fri, 5 Jul 2019 16:16:13 -0400 [thread overview]
Message-ID: <20190705201631.26266-1-jsnow@redhat.com> (raw)
This series adds a new "BITMAP" sync mode that is meant to replace the
existing "INCREMENTAL" sync mode.
This mode can have its behavior modified by issuing any of three bitmap sync
modes, passed as arguments to the job.
The three bitmap sync modes are:
- ON-SUCCESS: This is an alias for the old incremental mode. The bitmap is
conditionally synchronized based on the return code of the job
upon completion.
- NEVER: This is, effectively, the differential backup mode. It never clears
the bitmap, as the name suggests.
- ALWAYS: Here is the new, exciting thing. The bitmap is always synchronized,
even on failure. On success, this is identical to incremental, but
on failure it clears only the bits that were copied successfully.
This can be used to "resume" incremental backups from later points
in times.
I wrote this series by accident on my way to implement incremental mode
for mirror, but this happened first -- the problem is that Mirror mode
uses its existing modes in a very particular way; and this was the best
way to add bitmap support into the mirror job properly.
Summary:
- 01-03: refactor blockdev-backup and drive-backup to share more interface code
- 04-05: add the new 'bitmap' sync mode with sync policy 'conditional',
which is functionally identical to 'incremental' sync mode.
- 06: add sync policy 'never' ("Differential" backups.)
- 07-11: rework some merging code to facilite patch 12;
- 12: add sync policy 'always' ("Resumable" backups)
- 13-16: test infrastructure changes to support patch 16:
- 17: new iotest!
- 18: minor policy loosening as a QOL improvement
Future work:
- Update bitmaps.rst to explain these. (WIP, it's hard, sorry!)
- Add these modes to Mirror. (Done*, but needs tests.)
- Allow the use of bitmaps and bitmap sync modes with non-BITMAP modes;
This will allow for resumable/re-tryable full backups.
===
V3:
===
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/18:[0006] [FC] 'qapi/block-core: Introduce BackupCommon'
002/18:[0032] [FC] 'drive-backup: create do_backup_common'
003/18:[0020] [FC] 'blockdev-backup: utilize do_backup_common'
004/18:[0006] [FC] 'qapi: add BitmapSyncMode enum'
005/18:[0019] [FC] 'block/backup: Add mirror sync mode 'bitmap''
006/18:[0010] [FC] 'block/backup: add 'never' policy to bitmap sync mode'
007/18:[----] [--] 'hbitmap: Fix merge when b is empty, and result is not an alias of a'
008/18:[----] [--] 'hbitmap: enable merging across granularities'
009/18:[0005] [FC] 'block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal'
010/18:[0003] [FC] 'block/dirty-bitmap: add bdrv_dirty_bitmap_get'
011/18:[0021] [FC] 'block/backup: upgrade copy_bitmap to BdrvDirtyBitmap'
012/18:[0020] [FC] 'block/backup: add 'always' bitmap sync policy'
013/18:[----] [--] 'iotests: add testing shim for script-style python tests'
014/18:[----] [--] 'iotests: teach run_job to cancel pending jobs'
015/18:[0005] [FC] 'iotests: teach FilePath to produce multiple paths'
016/18:[----] [--] 'iotests: Add virtio-scsi device helper'
017/18:[0038] [FC] 'iotests: add test 257 for bitmap-mode backups'
018/18:[----] [-C] 'block/backup: loosen restriction on readonly bitmaps'
001: Made suggested doc fixes.
Changed 'since' to 4.2.
002: Added bds and aio_context to backup_common
Removed accidental extraneous unref on target_bs
Removed local_err propagation
003: Fallout from #002; hoist aio_context acquisition up into do_blockdev_backup
004: 'conditional' --> 'on-success'
005: Rediscover the lost stanza that ensures a bitmap mode was given
Fallout from 2, 3, 4.
006: Block comment fix for patchew
Fallout from #4
009: Fix assert() style issue. Why'd they let a macro be lowercase like that?
Probably to make specifically my life difficult.
010: Fix style issue {
011: Fix long lines
rename "bs" --> "target_bs" where appropriate
Free copy_bitmap from the right node
012: Multiline comment changes for patchew
Fallout from #4
015: Fix long line for patchew
Reinstate that second newline that Max likes
017: Fallout from #4.
===
V2:
===
Changes:
004: Fixed typo
Change @conditional docstring
005: Moved desugaring code into blockdev.c, facilitated by patches 1-3.
006: Change @never docstring slightly.
007: Merge will clear the target bitmap when both components bitmaps are empty,
and the target bitmap is not an alias of either component bitmap.
008: Check orig_size (logical size) instead of size (actual size) to enable
cross-granularity merging.
Fix the sparse merge itself, based on the block/backup code.
Clear the target bitmap before cross-granularity merge.
Assert the size is itself equal when logical size and granularities are
equal.
---: Dropped bdrv_dirty_bitmap_claim.
012: Rewrote the cleanup logic to hopefully be clearer.
use merge intsead of dropped reclaim.
015: Changed docstring
factored out filename pattern generation.
017: Fix mkpattern indent.
Use integer division!!!
Add parenthesis to boolean assignment
Change test run ordering; update output to reflect this
Use virtio-scsi-ccw when appropriate
Update test output to reflect new test running order
018: Fallout from patches 1-3; restrictions only need loosened in one place
instead of two.
Changes not made:
- Allowing 'cancel' to skip synchronization on cancel:
Decided against it, opting for consistency. The user asked for a sync,
and it's simpler logistically to execute on that desire.
Use the new mode carefully, please!
John Snow (18):
qapi/block-core: Introduce BackupCommon
drive-backup: create do_backup_common
blockdev-backup: utilize do_backup_common
qapi: add BitmapSyncMode enum
block/backup: Add mirror sync mode 'bitmap'
block/backup: add 'never' policy to bitmap sync mode
hbitmap: Fix merge when b is empty, and result is not an alias of a
hbitmap: enable merging across granularities
block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal
block/dirty-bitmap: add bdrv_dirty_bitmap_get
block/backup: upgrade copy_bitmap to BdrvDirtyBitmap
block/backup: add 'always' bitmap sync policy
iotests: add testing shim for script-style python tests
iotests: teach run_job to cancel pending jobs
iotests: teach FilePath to produce multiple paths
iotests: Add virtio-scsi device helper
iotests: add test 257 for bitmap-mode backups
block/backup: loosen restriction on readonly bitmaps
block/backup.c | 135 +-
block/dirty-bitmap.c | 69 +-
block/mirror.c | 8 +-
block/replication.c | 2 +-
blockdev.c | 208 ++--
include/block/block_int.h | 7 +-
include/block/dirty-bitmap.h | 4 +-
migration/block.c | 5 +-
nbd/server.c | 2 +-
qapi/block-core.json | 138 +--
tests/qemu-iotests/040 | 6 +-
tests/qemu-iotests/093 | 6 +-
tests/qemu-iotests/139 | 7 +-
tests/qemu-iotests/238 | 5 +-
tests/qemu-iotests/257 | 409 ++++++
tests/qemu-iotests/257.out | 2199 +++++++++++++++++++++++++++++++++
tests/qemu-iotests/group | 1 +
tests/qemu-iotests/iotests.py | 100 +-
util/hbitmap.c | 49 +-
19 files changed, 3048 insertions(+), 312 deletions(-)
create mode 100755 tests/qemu-iotests/257
create mode 100644 tests/qemu-iotests/257.out
--
2.21.0
next reply other threads:[~2019-07-05 20:19 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-07-05 20:16 John Snow [this message]
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 01/18] qapi/block-core: Introduce BackupCommon John Snow
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 02/18] drive-backup: create do_backup_common John Snow
2019-07-09 18:28 ` Max Reitz
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 03/18] blockdev-backup: utilize do_backup_common John Snow
2019-07-09 18:33 ` Max Reitz
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 04/18] qapi: add BitmapSyncMode enum John Snow
2019-07-09 18:34 ` Max Reitz
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 05/18] block/backup: Add mirror sync mode 'bitmap' John Snow
2019-07-09 18:38 ` Max Reitz
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 06/18] block/backup: add 'never' policy to bitmap sync mode John Snow
2019-07-09 18:38 ` Max Reitz
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 07/18] hbitmap: Fix merge when b is empty, and result is not an alias of a John Snow
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 08/18] hbitmap: enable merging across granularities John Snow
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 09/18] block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal John Snow
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 10/18] block/dirty-bitmap: add bdrv_dirty_bitmap_get John Snow
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 11/18] block/backup: upgrade copy_bitmap to BdrvDirtyBitmap John Snow
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 12/18] block/backup: add 'always' bitmap sync policy John Snow
2019-07-09 18:46 ` Max Reitz
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 13/18] iotests: add testing shim for script-style python tests John Snow
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 14/18] iotests: teach run_job to cancel pending jobs John Snow
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 15/18] iotests: teach FilePath to produce multiple paths John Snow
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 16/18] iotests: Add virtio-scsi device helper John Snow
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 17/18] iotests: add test 257 for bitmap-mode backups John Snow
2019-07-09 18:49 ` Max Reitz
2019-07-09 23:14 ` John Snow
2019-07-05 20:16 ` [Qemu-devel] [PATCH v3 18/18] block/backup: loosen restriction on readonly bitmaps John Snow
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190705201631.26266-1-jsnow@redhat.com \
--to=jsnow@redhat.com \
--cc=armbru@redhat.com \
--cc=dgilbert@redhat.com \
--cc=fam@euphon.net \
--cc=kwolf@redhat.com \
--cc=mreitz@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
--cc=quintela@redhat.com \
--cc=stefanha@redhat.com \
--cc=vsementsov@virtuozzo.com \
--cc=wencongyang2@huawei.com \
--cc=xiechanglong.d@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.