QEMU-Devel Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH v9 00/34] Add subcluster allocation to qcow2
@ 2020-06-28 11:02 Alberto Garcia
  2020-06-28 11:02 ` [PATCH v9 01/34] qcow2: Make Qcow2AioTask store the full host offset Alberto Garcia
                   ` (33 more replies)
  0 siblings, 34 replies; 71+ messages in thread
From: Alberto Garcia @ 2020-06-28 11:02 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Vladimir Sementsov-Ogievskiy, Alberto Garcia,
	qemu-block, Derek Su, Max Reitz

Hi,

here's the new version of the patches to add subcluster allocation
support to qcow2.

Please refer to the cover letter of the first version for a full
description of the patches:

   https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html

The important change from v8 is patch 24. This patch fixes unaligned
offsets in ZERO_ALLOC clusters, but the previous implementation was
not working for images with extended L2 entries and was simply adding
dead code. The new implementation works properly.

If you want to test this series make sure to apply this patch first:

   https://lists.gnu.org/archive/html/qemu-block/2020-06/msg00504.html

Berto

Based-on: <20200610094600.4029-1-berto@igalia.com>

v9:
- Patch 24: Change the algorithm so it actually does something in
  images with extended L2 entries.
- Patch 31: Update expected test results after rebase
- Patch 34: Fix a couple of typos.
            Add a new test for the changes in patch 24.
            Test concurrent requests on different subclusters of the
            same cluster.

v8: https://lists.gnu.org/archive/html/qemu-block/2020-06/msg00546.html
v7: https://lists.gnu.org/archive/html/qemu-block/2020-05/msg01683.html
v6: https://lists.gnu.org/archive/html/qemu-block/2020-05/msg01583.html
v5: https://lists.gnu.org/archive/html/qemu-block/2020-05/msg00251.html
v4: https://lists.gnu.org/archive/html/qemu-block/2020-03/msg00966.html
v3: https://lists.gnu.org/archive/html/qemu-block/2019-12/msg00587.html
v2: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg01642.html
v1: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html

Output of git backport-diff against v8:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/34:[----] [--] 'qcow2: Make Qcow2AioTask store the full host offset'
002/34:[----] [--] 'qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()'
003/34:[----] [--] 'qcow2: Add calculate_l2_meta()'
004/34:[----] [--] 'qcow2: Split cluster_needs_cow() out of count_cow_clusters()'
005/34:[----] [--] 'qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()'
006/34:[----] [--] 'qcow2: Add get_l2_entry() and set_l2_entry()'
007/34:[----] [--] 'qcow2: Document the Extended L2 Entries feature'
008/34:[----] [--] 'qcow2: Add dummy has_subclusters() function'
009/34:[----] [--] 'qcow2: Add subcluster-related fields to BDRVQcow2State'
010/34:[----] [--] 'qcow2: Add offset_to_sc_index()'
011/34:[----] [--] 'qcow2: Add offset_into_subcluster() and size_to_subclusters()'
012/34:[----] [--] 'qcow2: Add l2_entry_size()'
013/34:[----] [--] 'qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()'
014/34:[----] [--] 'qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()'
015/34:[----] [--] 'qcow2: Add qcow2_get_subcluster_range_type()'
016/34:[----] [--] 'qcow2: Add qcow2_cluster_is_allocated()'
017/34:[----] [--] 'qcow2: Add cluster type parameter to qcow2_get_host_offset()'
018/34:[----] [--] 'qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*'
019/34:[----] [--] 'qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC'
020/34:[----] [--] 'qcow2: Add subcluster support to calculate_l2_meta()'
021/34:[----] [--] 'qcow2: Add subcluster support to qcow2_get_host_offset()'
022/34:[----] [--] 'qcow2: Add subcluster support to zero_in_l2_slice()'
023/34:[----] [--] 'qcow2: Add subcluster support to discard_in_l2_slice()'
024/34:[0025] [FC] 'qcow2: Add subcluster support to check_refcounts_l2()'
025/34:[----] [--] 'qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()'
026/34:[----] [--] 'qcow2: Clear the L2 bitmap when allocating a compressed cluster'
027/34:[----] [--] 'qcow2: Add subcluster support to handle_alloc_space()'
028/34:[----] [--] 'qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()'
029/34:[----] [--] 'qcow2: Add subcluster support to qcow2_measure()'
030/34:[----] [--] 'qcow2: Add prealloc field to QCowL2Meta'
031/34:[0004] [FC] 'qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit'
032/34:[----] [--] 'qcow2: Allow preallocation and backing files if extended_l2 is set'
033/34:[----] [--] 'qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters'
034/34:[0152] [FC] 'iotests: Add tests for qcow2 images with extended L2 entries'

Alberto Garcia (34):
  qcow2: Make Qcow2AioTask store the full host offset
  qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()
  qcow2: Add calculate_l2_meta()
  qcow2: Split cluster_needs_cow() out of count_cow_clusters()
  qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()
  qcow2: Add get_l2_entry() and set_l2_entry()
  qcow2: Document the Extended L2 Entries feature
  qcow2: Add dummy has_subclusters() function
  qcow2: Add subcluster-related fields to BDRVQcow2State
  qcow2: Add offset_to_sc_index()
  qcow2: Add offset_into_subcluster() and size_to_subclusters()
  qcow2: Add l2_entry_size()
  qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()
  qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()
  qcow2: Add qcow2_get_subcluster_range_type()
  qcow2: Add qcow2_cluster_is_allocated()
  qcow2: Add cluster type parameter to qcow2_get_host_offset()
  qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*
  qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
  qcow2: Add subcluster support to calculate_l2_meta()
  qcow2: Add subcluster support to qcow2_get_host_offset()
  qcow2: Add subcluster support to zero_in_l2_slice()
  qcow2: Add subcluster support to discard_in_l2_slice()
  qcow2: Add subcluster support to check_refcounts_l2()
  qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()
  qcow2: Clear the L2 bitmap when allocating a compressed cluster
  qcow2: Add subcluster support to handle_alloc_space()
  qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()
  qcow2: Add subcluster support to qcow2_measure()
  qcow2: Add prealloc field to QCowL2Meta
  qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit
  qcow2: Allow preallocation and backing files if extended_l2 is set
  qcow2: Assert that expand_zero_clusters_in_l1() does not support
    subclusters
  iotests: Add tests for qcow2 images with extended L2 entries

 docs/interop/qcow2.txt           |  68 ++-
 docs/qcow2-cache.txt             |  19 +-
 qapi/block-core.json             |   7 +
 block/qcow2.h                    | 211 ++++++-
 include/block/block_int.h        |   1 +
 block/qcow2-cluster.c            | 912 +++++++++++++++++++++----------
 block/qcow2-refcount.c           |  47 +-
 block/qcow2.c                    | 304 +++++++----
 block/trace-events               |   2 +-
 tests/qemu-iotests/031.out       |   8 +-
 tests/qemu-iotests/036.out       |   4 +-
 tests/qemu-iotests/049.out       | 102 ++--
 tests/qemu-iotests/060.out       |   3 +-
 tests/qemu-iotests/061           |   6 +
 tests/qemu-iotests/061.out       |  25 +-
 tests/qemu-iotests/065           |  12 +-
 tests/qemu-iotests/082.out       |  48 +-
 tests/qemu-iotests/085.out       |  38 +-
 tests/qemu-iotests/144.out       |   4 +-
 tests/qemu-iotests/182.out       |   2 +-
 tests/qemu-iotests/185.out       |   8 +-
 tests/qemu-iotests/198.out       |   2 +
 tests/qemu-iotests/206.out       |   6 +-
 tests/qemu-iotests/242.out       |   5 +
 tests/qemu-iotests/255.out       |   8 +-
 tests/qemu-iotests/271           | 901 ++++++++++++++++++++++++++++++
 tests/qemu-iotests/271.out       | 724 ++++++++++++++++++++++++
 tests/qemu-iotests/274.out       |  49 +-
 tests/qemu-iotests/280.out       |   2 +-
 tests/qemu-iotests/291.out       |   6 +-
 tests/qemu-iotests/common.filter |   1 +
 tests/qemu-iotests/group         |   1 +
 32 files changed, 2963 insertions(+), 573 deletions(-)
 create mode 100755 tests/qemu-iotests/271
 create mode 100644 tests/qemu-iotests/271.out

-- 
2.20.1



^ permalink raw reply	[flat|nested] 71+ messages in thread

end of thread, back to index

Thread overview: 71+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-28 11:02 [PATCH v9 00/34] Add subcluster allocation to qcow2 Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 01/34] qcow2: Make Qcow2AioTask store the full host offset Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 02/34] qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset() Alberto Garcia
2020-06-30 10:19   ` Max Reitz
2020-06-30 10:27     ` Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 03/34] qcow2: Add calculate_l2_meta() Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 04/34] qcow2: Split cluster_needs_cow() out of count_cow_clusters() Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 05/34] qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied() Alberto Garcia
2020-06-30 10:38   ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 06/34] qcow2: Add get_l2_entry() and set_l2_entry() Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 07/34] qcow2: Document the Extended L2 Entries feature Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 08/34] qcow2: Add dummy has_subclusters() function Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 09/34] qcow2: Add subcluster-related fields to BDRVQcow2State Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 10/34] qcow2: Add offset_to_sc_index() Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 11/34] qcow2: Add offset_into_subcluster() and size_to_subclusters() Alberto Garcia
2020-07-01 12:23   ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 12/34] qcow2: Add l2_entry_size() Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 13/34] qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap() Alberto Garcia
2020-07-01 12:28   ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 14/34] qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type() Alberto Garcia
2020-07-01 12:52   ` Max Reitz
2020-07-01 16:26     ` Alberto Garcia
2020-07-02  9:57       ` Max Reitz
2020-07-02 22:00         ` Alberto Garcia
2020-07-03  7:17           ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 15/34] qcow2: Add qcow2_get_subcluster_range_type() Alberto Garcia
2020-07-01 13:37   ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 16/34] qcow2: Add qcow2_cluster_is_allocated() Alberto Garcia
2020-07-01 13:55   ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 17/34] qcow2: Add cluster type parameter to qcow2_get_host_offset() Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 18/34] qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_* Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 19/34] qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 20/34] qcow2: Add subcluster support to calculate_l2_meta() Alberto Garcia
2020-07-02 11:30   ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 21/34] qcow2: Add subcluster support to qcow2_get_host_offset() Alberto Garcia
2020-07-02 12:46   ` Max Reitz
2020-07-02 22:04     ` Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 22/34] qcow2: Add subcluster support to zero_in_l2_slice() Alberto Garcia
2020-07-02 12:56   ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 23/34] qcow2: Add subcluster support to discard_in_l2_slice() Alberto Garcia
2020-07-02 13:24   ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 24/34] qcow2: Add subcluster support to check_refcounts_l2() Alberto Garcia
2020-07-02 13:32   ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 25/34] qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2() Alberto Garcia
2020-07-02 14:01   ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 26/34] qcow2: Clear the L2 bitmap when allocating a compressed cluster Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 27/34] qcow2: Add subcluster support to handle_alloc_space() Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 28/34] qcow2: Add subcluster support to qcow2_co_pwrite_zeroes() Alberto Garcia
2020-07-02 14:28   ` Max Reitz
2020-07-02 22:40     ` Alberto Garcia
2020-07-03  7:18       ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 29/34] qcow2: Add subcluster support to qcow2_measure() Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 30/34] qcow2: Add prealloc field to QCowL2Meta Alberto Garcia
2020-07-02 14:50   ` Max Reitz
2020-07-02 14:58     ` Eric Blake
2020-07-02 15:09       ` Max Reitz
2020-07-02 23:05         ` Alberto Garcia
2020-07-03  7:22           ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 31/34] qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit Alberto Garcia
2020-07-02 15:13   ` Max Reitz
2020-07-03 12:43     ` Alberto Garcia
2020-06-28 11:02 ` [PATCH v9 32/34] qcow2: Allow preallocation and backing files if extended_l2 is set Alberto Garcia
2020-07-03  7:45   ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 33/34] qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters Alberto Garcia
2020-07-03  7:46   ` Max Reitz
2020-06-28 11:02 ` [PATCH v9 34/34] iotests: Add tests for qcow2 images with extended L2 entries Alberto Garcia
2020-07-03  9:49   ` Max Reitz
2020-07-03 13:06     ` Alberto Garcia
2020-07-03 13:47       ` Max Reitz
2020-07-03 15:20         ` Alberto Garcia
2020-07-06 13:57       ` Eric Blake

QEMU-Devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/qemu-devel/0 qemu-devel/git/0.git
	git clone --mirror https://lore.kernel.org/qemu-devel/1 qemu-devel/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 qemu-devel qemu-devel/ https://lore.kernel.org/qemu-devel \
		qemu-devel@nongnu.org
	public-inbox-index qemu-devel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.nongnu.qemu-devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git