* [Qemu-devel] [PULL 01/33] iotests: use iotests.VM in 238
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
@ 2019-03-08 12:57 ` Kevin Wolf
2019-03-08 12:57 ` [Qemu-devel] [PULL 02/33] qcow2: Default to 4KB for the qcow2 cache entry size Kevin Wolf
` (32 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:57 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
From: Stefan Hajnoczi <stefanha@redhat.com>
Test 238 does not require the kvm accelerator. Using the qtest
accelerator allows the test to run in both non-kvm and non-tcg
environments.
iotests.VM implicitly uses the qtest accelerator and is really the class
that this test should be using. Switch to that instead of
qemu.QEMUMachine.
Suggested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/qemu-iotests/238 | 7 +------
1 file changed, 1 insertion(+), 6 deletions(-)
diff --git a/tests/qemu-iotests/238 b/tests/qemu-iotests/238
index 688abc9acb..1c0a46fa90 100755
--- a/tests/qemu-iotests/238
+++ b/tests/qemu-iotests/238
@@ -23,17 +23,12 @@ import os
import iotests
from iotests import log
-sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-
-from qemu import QEMUMachine
-
if iotests.qemu_default_machine == 's390-ccw-virtio':
virtio_scsi_device = 'virtio-scsi-ccw'
else:
virtio_scsi_device = 'virtio-scsi-pci'
-vm = QEMUMachine(iotests.qemu_prog)
-vm.add_args('-machine', 'accel=kvm')
+vm = iotests.VM()
vm.launch()
log(vm.qmp('blockdev-add', node_name='hd0', driver='null-co'))
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 02/33] qcow2: Default to 4KB for the qcow2 cache entry size
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
2019-03-08 12:57 ` [Qemu-devel] [PULL 01/33] iotests: use iotests.VM in 238 Kevin Wolf
@ 2019-03-08 12:57 ` Kevin Wolf
2019-03-08 12:57 ` [Qemu-devel] [PULL 03/33] iotests: open notrun files in text mode Kevin Wolf
` (31 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:57 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
From: Alberto Garcia <berto@igalia.com>
QEMU 2.12 (commit 1221fe6f636754ab5f2c1c87caa77633e9123622) introduced
a new setting called l2-cache-entry-size that allows making entries on
the qcow2 L2 cache smaller than the cluster size.
I have been performing several tests with different cluster and entry
sizes and all of them show that reducing the entry size (aka L2 slice)
consistently improves I/O performance, notably during random I/O (all
tests done with sequential I/O show similar results). This is to be
expected because loading and evicting an L2 slice is more expensive
the larger the slice is.
Here are some numbers on fully populated 40GB qcow2 images. The
rightmost column represents the maximum L2 cache size in both cases.
Cluster size = 64 KB
|-------------+--------------+--------------+--------------|
| | 1MB L2 cache | 3MB L2 cache | 5MB L2 cache |
|-------------+--------------+--------------+--------------|
| 4KB slices | 6545 IOPS | 12045 IOPS | 55680 IOPS |
| 16KB slices | 5177 IOPS | 9798 IOPS | 56278 IOPS |
| 64KB slices | 2718 IOPS | 5326 IOPS | 57355 IOPS |
|-------------+--------------+--------------+--------------|
Cluster size = 256 KB
|--------------+----------------+--------------+-----------------|
| | 512KB L2 cache | 1MB L2 cache | 1280KB L2 cache |
|--------------+----------------+--------------+-----------------|
| 4KB slices | 8539 IOPS | 21071 IOPS | 55417 IOPS |
| 64KB slices | 3598 IOPS | 9772 IOPS | 57687 IOPS |
| 256KB slices | 1415 IOPS | 4120 IOPS | 58001 IOPS |
|--------------+----------------+--------------+-----------------|
As can be seen in the numbers, the only exception to the rule is when
the cache is large enough to hold all L2 tables. This is also to be
expected because in this case no cache entry is ever evicted so
reducing its size doesn't bring any benefit.
This patch sets the default L2 cache entry size to 4KB except when the
cache is large enough for the whole disk.
Signed-off-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
docs/qcow2-cache.txt | 17 +++++++++++------
block/qcow2.c | 12 ++++++++++++
2 files changed, 23 insertions(+), 6 deletions(-)
diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt
index c1e7751fea..d57f409861 100644
--- a/docs/qcow2-cache.txt
+++ b/docs/qcow2-cache.txt
@@ -158,10 +158,10 @@ refcount cache is as small as possible unless overridden by the user.
Using smaller cache entries
---------------------------
-The qcow2 L2 cache stores complete tables by default. This means that
-if QEMU needs an entry from an L2 table then the whole table is read
-from disk and is kept in the cache. If the cache is full then a
-complete table needs to be evicted first.
+The qcow2 L2 cache can store complete tables. This means that if QEMU
+needs an entry from an L2 table then the whole table is read from disk
+and is kept in the cache. If the cache is full then a complete table
+needs to be evicted first.
This can be inefficient with large cluster sizes since it results in
more disk I/O and wastes more cache memory.
@@ -172,6 +172,9 @@ it smaller than the cluster size. This can be configured using the
-drive file=hd.qcow2,l2-cache-size=2097152,l2-cache-entry-size=4096
+Since QEMU 4.0 the value of l2-cache-entry-size defaults to 4KB (or
+the cluster size if it's smaller).
+
Some things to take into account:
- The L2 cache entry size has the same restrictions as the cluster
@@ -185,7 +188,8 @@ Some things to take into account:
- Try different entry sizes to see which one gives faster performance
in your case. The block size of the host filesystem is generally a
- good default (usually 4096 bytes in the case of ext4).
+ good default (usually 4096 bytes in the case of ext4, hence the
+ default).
- Only the L2 cache can be configured this way. The refcount cache
always uses the cluster size as the entry size.
@@ -194,7 +198,8 @@ Some things to take into account:
(as explained in the "Choosing the right cache sizes" and "How to
configure the cache sizes" sections in this document) then none of
this is necessary and you can omit the "l2-cache-entry-size"
- parameter altogether.
+ parameter altogether. In this case QEMU makes the entry size
+ equal to the cluster size by default.
Reducing the memory usage
diff --git a/block/qcow2.c b/block/qcow2.c
index 7fb2730f09..48d22f48d5 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -788,6 +788,7 @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts,
BDRVQcow2State *s = bs->opaque;
uint64_t combined_cache_size, l2_cache_max_setting;
bool l2_cache_size_set, refcount_cache_size_set, combined_cache_size_set;
+ bool l2_cache_entry_size_set;
int min_refcount_cache = MIN_REFCOUNT_CACHE_SIZE * s->cluster_size;
uint64_t virtual_disk_size = bs->total_sectors * BDRV_SECTOR_SIZE;
uint64_t max_l2_cache = virtual_disk_size / (s->cluster_size / 8);
@@ -795,6 +796,7 @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts,
combined_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_CACHE_SIZE);
l2_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_L2_CACHE_SIZE);
refcount_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_REFCOUNT_CACHE_SIZE);
+ l2_cache_entry_size_set = qemu_opt_get(opts, QCOW2_OPT_L2_CACHE_ENTRY_SIZE);
combined_cache_size = qemu_opt_get_size(opts, QCOW2_OPT_CACHE_SIZE, 0);
l2_cache_max_setting = qemu_opt_get_size(opts, QCOW2_OPT_L2_CACHE_SIZE,
@@ -841,6 +843,16 @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts,
}
}
}
+
+ /*
+ * If the L2 cache is not enough to cover the whole disk then
+ * default to 4KB entries. Smaller entries reduce the cost of
+ * loads and evictions and increase I/O performance.
+ */
+ if (*l2_cache_size < max_l2_cache && !l2_cache_entry_size_set) {
+ *l2_cache_entry_size = MIN(s->cluster_size, 4096);
+ }
+
/* l2_cache_size and refcount_cache_size are ensured to have at least
* their minimum values in qcow2_update_options_prepare() */
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 03/33] iotests: open notrun files in text mode
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
2019-03-08 12:57 ` [Qemu-devel] [PULL 01/33] iotests: use iotests.VM in 238 Kevin Wolf
2019-03-08 12:57 ` [Qemu-devel] [PULL 02/33] qcow2: Default to 4KB for the qcow2 cache entry size Kevin Wolf
@ 2019-03-08 12:57 ` Kevin Wolf
2019-03-08 12:57 ` [Qemu-devel] [PULL 04/33] block: iterate_format with account of whitelisting Kevin Wolf
` (30 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:57 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Replace the binary mode with the default text one when *.notrun
files are opened for skipped tests. That change is made for the
compatibility with Python 3 which returns error otherwise.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/qemu-iotests/iotests.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 3d15571688..46fad4ce81 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -712,7 +712,7 @@ def notrun(reason):
# Each test in qemu-iotests has a number ("seq")
seq = os.path.basename(sys.argv[0])
- open('%s/%s.notrun' % (output_dir, seq), 'wb').write(reason + '\n')
+ open('%s/%s.notrun' % (output_dir, seq), 'w').write(reason + '\n')
print('%s not run: %s' % (seq, reason))
sys.exit(0)
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 04/33] block: iterate_format with account of whitelisting
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (2 preceding siblings ...)
2019-03-08 12:57 ` [Qemu-devel] [PULL 03/33] iotests: open notrun files in text mode Kevin Wolf
@ 2019-03-08 12:57 ` Kevin Wolf
2019-03-08 12:57 ` [Qemu-devel] [PULL 05/33] iotests: ask QEMU for supported formats Kevin Wolf
` (29 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:57 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
bdrv_iterate_format (which is currently only used for printing out the
formats supported by the block layer) doesn't take format whitelisting
into account.
This creates a problem for tests: they enumerate supported formats to
decide which tests to enable, but then discover that QEMU doesn't let
them actually use some of those formats.
To avoid that, exclude formats that are not whitelisted from
enumeration, if whitelisting is in use. Since we have separate
whitelists for r/w and r/o, take this a parameter to
bdrv_iterate_format, and print two lists of supported formats (r/w and
r/o) in main qemu.
Signed-off-by: Roman Kagan <rkagan@virtuozzo.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
include/block/block.h | 2 +-
block.c | 23 +++++++++++++++++++----
blockdev.c | 4 +++-
qemu-img.c | 2 +-
4 files changed, 24 insertions(+), 7 deletions(-)
diff --git a/include/block/block.h b/include/block/block.h
index 5b5cf868df..6a758a76f8 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -472,7 +472,7 @@ void bdrv_next_cleanup(BdrvNextIterator *it);
BlockDriverState *bdrv_next_monitor_owned(BlockDriverState *bs);
bool bdrv_is_encrypted(BlockDriverState *bs);
void bdrv_iterate_format(void (*it)(void *opaque, const char *name),
- void *opaque);
+ void *opaque, bool read_only);
const char *bdrv_get_node_name(const BlockDriverState *bs);
const char *bdrv_get_device_name(const BlockDriverState *bs);
const char *bdrv_get_device_or_node_name(const BlockDriverState *bs);
diff --git a/block.c b/block.c
index 35e78e2172..ccf008c177 100644
--- a/block.c
+++ b/block.c
@@ -426,7 +426,7 @@ BlockDriver *bdrv_find_format(const char *format_name)
return bdrv_do_find_format(format_name);
}
-int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
+static int bdrv_format_is_whitelisted(const char *format_name, bool read_only)
{
static const char *whitelist_rw[] = {
CONFIG_BDRV_RW_WHITELIST
@@ -441,13 +441,13 @@ int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
}
for (p = whitelist_rw; *p; p++) {
- if (!strcmp(drv->format_name, *p)) {
+ if (!strcmp(format_name, *p)) {
return 1;
}
}
if (read_only) {
for (p = whitelist_ro; *p; p++) {
- if (!strcmp(drv->format_name, *p)) {
+ if (!strcmp(format_name, *p)) {
return 1;
}
}
@@ -455,6 +455,11 @@ int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
return 0;
}
+int bdrv_is_whitelisted(BlockDriver *drv, bool read_only)
+{
+ return bdrv_format_is_whitelisted(drv->format_name, read_only);
+}
+
bool bdrv_uses_whitelist(void)
{
return use_bdrv_whitelist;
@@ -4147,7 +4152,7 @@ static int qsort_strcmp(const void *a, const void *b)
}
void bdrv_iterate_format(void (*it)(void *opaque, const char *name),
- void *opaque)
+ void *opaque, bool read_only)
{
BlockDriver *drv;
int count = 0;
@@ -4158,6 +4163,11 @@ void bdrv_iterate_format(void (*it)(void *opaque, const char *name),
if (drv->format_name) {
bool found = false;
int i = count;
+
+ if (use_bdrv_whitelist && !bdrv_is_whitelisted(drv, read_only)) {
+ continue;
+ }
+
while (formats && i && !found) {
found = !strcmp(formats[--i], drv->format_name);
}
@@ -4176,6 +4186,11 @@ void bdrv_iterate_format(void (*it)(void *opaque, const char *name),
bool found = false;
int j = count;
+ if (use_bdrv_whitelist &&
+ !bdrv_format_is_whitelisted(format_name, read_only)) {
+ continue;
+ }
+
while (formats && j && !found) {
found = !strcmp(formats[--j], format_name);
}
diff --git a/blockdev.c b/blockdev.c
index 7e6bf9955c..871966ca13 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -531,7 +531,9 @@ static BlockBackend *blockdev_init(const char *file, QDict *bs_opts,
if ((buf = qemu_opt_get(opts, "format")) != NULL) {
if (is_help_option(buf)) {
error_printf("Supported formats:");
- bdrv_iterate_format(bdrv_format_print, NULL);
+ bdrv_iterate_format(bdrv_format_print, NULL, false);
+ error_printf("\nSupported formats (read-only):");
+ bdrv_iterate_format(bdrv_format_print, NULL, true);
error_printf("\n");
goto early_err;
}
diff --git a/qemu-img.c b/qemu-img.c
index 660c01898e..5fac840742 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -198,7 +198,7 @@ static void QEMU_NORETURN help(void)
" 'skip=N' skip N bs-sized blocks at the start of input\n";
printf("%s\nSupported formats:", help_msg);
- bdrv_iterate_format(format_print, NULL);
+ bdrv_iterate_format(format_print, NULL, false);
printf("\n\n" QEMU_HELP_BOTTOM "\n");
exit(EXIT_SUCCESS);
}
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 05/33] iotests: ask QEMU for supported formats
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (3 preceding siblings ...)
2019-03-08 12:57 ` [Qemu-devel] [PULL 04/33] block: iterate_format with account of whitelisting Kevin Wolf
@ 2019-03-08 12:57 ` Kevin Wolf
2019-03-08 12:57 ` [Qemu-devel] [PULL 06/33] iotests: check whitelisted formats Kevin Wolf
` (28 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:57 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Supported formats listed by 'qemu' may differ from those listed by
'qemu-img' due to whitelists. Some test cases require specific formats
that may be used with qemu. They can be inquired directly by running
'qemu -drive format=help'. The response takes whitelists into account.
The method supported_formats() serves for that. The method decorator
skip_if_unsupported() checks if all requested formats are whitelisted.
If not, the test case will be skipped. That has been implemented in
the 'check' file in the way similar to the 'test notrun' mechanism.
Suggested-by: Roman Kagan <rkagan@virtuozzo.com>
Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Suggested-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/qemu-iotests/check | 13 ++++++++++-
tests/qemu-iotests/iotests.py | 43 +++++++++++++++++++++++++++++++++++
2 files changed, 55 insertions(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index 895e1e3dcb..1016887438 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -25,6 +25,7 @@ try=0
n_bad=0
bad=""
notrun=""
+casenotrun=""
interrupt=true
# by default don't output timestamps
@@ -664,6 +665,11 @@ END { if (NR > 0) {
echo "Not run:$notrun"
echo "Not run:$notrun" >>check.log
fi
+ if [ ! -z "$casenotrun" ]
+ then
+ echo "Some cases not run in:$casenotrun"
+ echo "Some cases not run in:$casenotrun" >>check.log
+ fi
if [ ! -z "$n_bad" -a $n_bad != 0 ]
then
echo "Failures:$bad"
@@ -743,6 +749,7 @@ do
printf " " # prettier output with timestamps.
fi
rm -f core $seq.notrun
+ rm -f $seq.casenotrun
start=$(_wallclock)
$timestamp && printf %s " [$(date "+%T")]"
@@ -823,7 +830,11 @@ do
fi
fi
fi
-
+ if [ -f $seq.casenotrun ]
+ then
+ cat $seq.casenotrun
+ casenotrun="$casenotrun $seq"
+ fi
fi
# come here for each test, except when $showme is true
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 46fad4ce81..997dc910cb 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -716,6 +716,14 @@ def notrun(reason):
print('%s not run: %s' % (seq, reason))
sys.exit(0)
+def case_notrun(reason):
+ '''Skip this test case'''
+ # Each test in qemu-iotests has a number ("seq")
+ seq = os.path.basename(sys.argv[0])
+
+ open('%s/%s.casenotrun' % (output_dir, seq), 'a').write(
+ ' [case not run] ' + reason + '\n')
+
def verify_image_format(supported_fmts=[], unsupported_fmts=[]):
assert not (supported_fmts and unsupported_fmts)
@@ -756,6 +764,41 @@ def verify_quorum():
if not supports_quorum():
notrun('quorum support missing')
+def qemu_pipe(*args):
+ '''Run qemu with an option to print something and exit (e.g. a help option),
+ and return its output'''
+ args = [qemu_prog] + qemu_opts + list(args)
+ subp = subprocess.Popen(args, stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ universal_newlines=True)
+ exitcode = subp.wait()
+ if exitcode < 0:
+ sys.stderr.write('qemu received signal %i: %s\n' % (-exitcode,
+ ' '.join(args)))
+ return subp.communicate()[0]
+
+def supported_formats(read_only=False):
+ '''Set 'read_only' to True to check ro-whitelist
+ Otherwise, rw-whitelist is checked'''
+ format_message = qemu_pipe("-drive", "format=help")
+ line = 1 if read_only else 0
+ return format_message.splitlines()[line].split(":")[1].split()
+
+def skip_if_unsupported(required_formats=[], read_only=False):
+ '''Skip Test Decorator
+ Runs the test if all the required formats are whitelisted'''
+ def skip_test_decorator(func):
+ def func_wrapper(*args, **kwargs):
+ usf_list = list(set(required_formats) -
+ set(supported_formats(read_only)))
+ if usf_list:
+ case_notrun('{}: formats {} are not whitelisted'.format(
+ args[0], usf_list))
+ else:
+ return func(*args, **kwargs)
+ return func_wrapper
+ return skip_test_decorator
+
def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
unsupported_fmts=[]):
'''Run tests'''
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 06/33] iotests: check whitelisted formats
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (4 preceding siblings ...)
2019-03-08 12:57 ` [Qemu-devel] [PULL 05/33] iotests: ask QEMU for supported formats Kevin Wolf
@ 2019-03-08 12:57 ` Kevin Wolf
2019-03-08 12:57 ` [Qemu-devel] [PULL 07/33] tests/multiboot: Improve portability by searching bash in the $PATH Kevin Wolf
` (27 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:57 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Some test cases require specific formats. The method decorator
skip_if_unsupported() checks if requested formats are whitelisted.
The test #139 was selected for a sample output, after running
$ ./check -qcow2 131-140
137 3s ...
138 0s ...
139 2s ...
[case not run] testBlkDebug (__main__.TestBlockdevDel): formats ['blkdebug'] are not whitelisted
[case not run] testBlkVerify (__main__.TestBlockdevDel): formats ['blkverify'] are not whitelisted
[case not run] testQuorum (__main__.TestBlockdevDel): formats ['quorum'] are not whitelisted
140 0s ...
Not run: 131 135 136
Some cases not run in: 139
Passed all 7 tests
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/qemu-iotests/139 | 3 +++
1 file changed, 3 insertions(+)
diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139
index 62402c1c35..933b45121a 100755
--- a/tests/qemu-iotests/139
+++ b/tests/qemu-iotests/139
@@ -325,6 +325,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
# FIXME mirror0 disappears, drive-mirror doesn't take a reference
#self.delBlockDriverState('mirror0')
+ @iotests.skip_if_unsupported(['blkdebug'])
def testBlkDebug(self):
self.addBlkDebug('debug0', 'node0')
# 'node0' is used by the blkdebug node
@@ -333,6 +334,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
self.delBlockDriverState('debug0')
self.checkBlockDriverState('node0', False)
+ @iotests.skip_if_unsupported(['blkverify'])
def testBlkVerify(self):
self.addBlkVerify('verify0', 'node0', 'node1')
# We cannot remove the children of a blkverify device
@@ -343,6 +345,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
self.checkBlockDriverState('node0', False)
self.checkBlockDriverState('node1', False)
+ @iotests.skip_if_unsupported(['quorum'])
def testQuorum(self):
if not iotests.supports_quorum():
return
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 07/33] tests/multiboot: Improve portability by searching bash in the $PATH
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (5 preceding siblings ...)
2019-03-08 12:57 ` [Qemu-devel] [PULL 06/33] iotests: check whitelisted formats Kevin Wolf
@ 2019-03-08 12:57 ` Kevin Wolf
2019-03-08 12:57 ` [Qemu-devel] [PULL 08/33] tests/bios-tables: " Kevin Wolf
` (26 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:57 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
From: Philippe Mathieu-Daudé <philmd@redhat.com>
Bash is not always installed as /bin/bash. In particular on OpenBSD,
the package installs it in /usr/local/bin.
Use the 'env' shebang to search bash in the $PATH.
Reviewed-by: Kamil Rytarowski <n54@gmx.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/multiboot/run_test.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/multiboot/run_test.sh b/tests/multiboot/run_test.sh
index 6c33003e71..98df91e6af 100755
--- a/tests/multiboot/run_test.sh
+++ b/tests/multiboot/run_test.sh
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
# Copyright (c) 2013 Kevin Wolf <kwolf@redhat.com>
#
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 08/33] tests/bios-tables: Improve portability by searching bash in the $PATH
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (6 preceding siblings ...)
2019-03-08 12:57 ` [Qemu-devel] [PULL 07/33] tests/multiboot: Improve portability by searching bash in the $PATH Kevin Wolf
@ 2019-03-08 12:57 ` Kevin Wolf
2019-03-08 12:57 ` [Qemu-devel] [PULL 09/33] qemu-iotests: " Kevin Wolf
` (25 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:57 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
From: Philippe Mathieu-Daudé <philmd@redhat.com>
Bash is not always installed as /bin/bash. In particular on OpenBSD,
the package installs it in /usr/local/bin.
Use the 'env' shebang to search bash in the $PATH.
Reviewed-by: Kamil Rytarowski <n54@gmx.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/data/acpi/rebuild-expected-aml.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/data/acpi/rebuild-expected-aml.sh b/tests/data/acpi/rebuild-expected-aml.sh
index bf9ba242ad..abdff70a0d 100755
--- a/tests/data/acpi/rebuild-expected-aml.sh
+++ b/tests/data/acpi/rebuild-expected-aml.sh
@@ -1,4 +1,4 @@
-#! /bin/bash
+#!/usr/bin/env bash
#
# Rebuild expected AML files for acpi unit-test
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 09/33] qemu-iotests: Improve portability by searching bash in the $PATH
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (7 preceding siblings ...)
2019-03-08 12:57 ` [Qemu-devel] [PULL 08/33] tests/bios-tables: " Kevin Wolf
@ 2019-03-08 12:57 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 10/33] qemu-iotests: Ensure GNU sed is used Kevin Wolf
` (24 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:57 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
From: Philippe Mathieu-Daudé <philmd@redhat.com>
Bash is not always installed as /bin/bash. In particular on OpenBSD,
the package installs it in /usr/local/bin.
Use the 'env' shebang to search bash in the $PATH.
Patch created mechanically by running:
$ git grep -lE '#! ?/bin/bash' -- tests/qemu-iotests \
| while read f; do \
sed -i 's|^#!.\?/bin/bash$|#!/usr/bin/env bash|' $f; \
done
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/qemu-iotests/001 | 2 +-
tests/qemu-iotests/002 | 2 +-
tests/qemu-iotests/003 | 2 +-
tests/qemu-iotests/004 | 2 +-
tests/qemu-iotests/005 | 2 +-
tests/qemu-iotests/007 | 2 +-
tests/qemu-iotests/008 | 2 +-
tests/qemu-iotests/009 | 2 +-
tests/qemu-iotests/010 | 2 +-
tests/qemu-iotests/011 | 2 +-
tests/qemu-iotests/012 | 2 +-
tests/qemu-iotests/013 | 2 +-
tests/qemu-iotests/014 | 2 +-
tests/qemu-iotests/015 | 2 +-
tests/qemu-iotests/017 | 2 +-
tests/qemu-iotests/018 | 2 +-
tests/qemu-iotests/019 | 2 +-
tests/qemu-iotests/020 | 2 +-
tests/qemu-iotests/021 | 2 +-
tests/qemu-iotests/022 | 2 +-
tests/qemu-iotests/023 | 2 +-
tests/qemu-iotests/024 | 2 +-
tests/qemu-iotests/025 | 2 +-
tests/qemu-iotests/026 | 2 +-
tests/qemu-iotests/027 | 2 +-
tests/qemu-iotests/028 | 2 +-
tests/qemu-iotests/029 | 2 +-
tests/qemu-iotests/031 | 2 +-
tests/qemu-iotests/032 | 2 +-
tests/qemu-iotests/033 | 2 +-
tests/qemu-iotests/034 | 2 +-
tests/qemu-iotests/035 | 2 +-
tests/qemu-iotests/036 | 2 +-
tests/qemu-iotests/037 | 2 +-
tests/qemu-iotests/038 | 2 +-
tests/qemu-iotests/039 | 2 +-
tests/qemu-iotests/042 | 2 +-
tests/qemu-iotests/043 | 2 +-
tests/qemu-iotests/046 | 2 +-
tests/qemu-iotests/047 | 2 +-
tests/qemu-iotests/048 | 2 +-
tests/qemu-iotests/049 | 2 +-
tests/qemu-iotests/050 | 2 +-
tests/qemu-iotests/051 | 2 +-
tests/qemu-iotests/052 | 2 +-
tests/qemu-iotests/053 | 2 +-
tests/qemu-iotests/054 | 2 +-
tests/qemu-iotests/058 | 2 +-
tests/qemu-iotests/059 | 2 +-
tests/qemu-iotests/060 | 2 +-
tests/qemu-iotests/061 | 2 +-
tests/qemu-iotests/062 | 2 +-
tests/qemu-iotests/063 | 2 +-
tests/qemu-iotests/064 | 2 +-
tests/qemu-iotests/066 | 2 +-
tests/qemu-iotests/067 | 2 +-
tests/qemu-iotests/068 | 2 +-
tests/qemu-iotests/069 | 2 +-
tests/qemu-iotests/070 | 2 +-
tests/qemu-iotests/071 | 2 +-
tests/qemu-iotests/072 | 2 +-
tests/qemu-iotests/073 | 2 +-
tests/qemu-iotests/074 | 2 +-
tests/qemu-iotests/075 | 2 +-
tests/qemu-iotests/076 | 2 +-
tests/qemu-iotests/077 | 2 +-
tests/qemu-iotests/078 | 2 +-
tests/qemu-iotests/079 | 2 +-
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/081 | 2 +-
tests/qemu-iotests/082 | 2 +-
tests/qemu-iotests/083 | 2 +-
tests/qemu-iotests/084 | 2 +-
tests/qemu-iotests/085 | 2 +-
tests/qemu-iotests/086 | 2 +-
tests/qemu-iotests/087 | 2 +-
tests/qemu-iotests/088 | 2 +-
tests/qemu-iotests/089 | 2 +-
tests/qemu-iotests/090 | 2 +-
tests/qemu-iotests/091 | 2 +-
tests/qemu-iotests/092 | 2 +-
tests/qemu-iotests/094 | 2 +-
tests/qemu-iotests/095 | 2 +-
tests/qemu-iotests/097 | 2 +-
tests/qemu-iotests/098 | 2 +-
tests/qemu-iotests/099 | 2 +-
tests/qemu-iotests/101 | 2 +-
tests/qemu-iotests/102 | 2 +-
tests/qemu-iotests/103 | 2 +-
tests/qemu-iotests/104 | 2 +-
tests/qemu-iotests/105 | 2 +-
tests/qemu-iotests/106 | 2 +-
tests/qemu-iotests/107 | 2 +-
tests/qemu-iotests/108 | 2 +-
tests/qemu-iotests/109 | 2 +-
tests/qemu-iotests/110 | 2 +-
tests/qemu-iotests/111 | 2 +-
tests/qemu-iotests/112 | 2 +-
tests/qemu-iotests/113 | 2 +-
tests/qemu-iotests/114 | 2 +-
tests/qemu-iotests/115 | 2 +-
tests/qemu-iotests/116 | 2 +-
tests/qemu-iotests/117 | 2 +-
tests/qemu-iotests/119 | 2 +-
tests/qemu-iotests/120 | 2 +-
tests/qemu-iotests/121 | 2 +-
tests/qemu-iotests/122 | 2 +-
tests/qemu-iotests/123 | 2 +-
tests/qemu-iotests/125 | 2 +-
tests/qemu-iotests/126 | 2 +-
tests/qemu-iotests/127 | 2 +-
tests/qemu-iotests/128 | 2 +-
tests/qemu-iotests/130 | 2 +-
tests/qemu-iotests/131 | 2 +-
tests/qemu-iotests/133 | 2 +-
tests/qemu-iotests/134 | 2 +-
tests/qemu-iotests/135 | 2 +-
tests/qemu-iotests/137 | 2 +-
tests/qemu-iotests/138 | 2 +-
tests/qemu-iotests/140 | 2 +-
tests/qemu-iotests/141 | 2 +-
tests/qemu-iotests/142 | 2 +-
tests/qemu-iotests/143 | 2 +-
tests/qemu-iotests/144 | 2 +-
tests/qemu-iotests/145 | 2 +-
tests/qemu-iotests/146 | 2 +-
tests/qemu-iotests/150 | 2 +-
tests/qemu-iotests/153 | 2 +-
tests/qemu-iotests/154 | 2 +-
tests/qemu-iotests/156 | 2 +-
tests/qemu-iotests/157 | 2 +-
tests/qemu-iotests/158 | 2 +-
tests/qemu-iotests/159 | 2 +-
tests/qemu-iotests/160 | 2 +-
tests/qemu-iotests/161 | 2 +-
tests/qemu-iotests/162 | 2 +-
tests/qemu-iotests/170 | 2 +-
tests/qemu-iotests/171 | 2 +-
tests/qemu-iotests/172 | 2 +-
tests/qemu-iotests/173 | 2 +-
tests/qemu-iotests/174 | 2 +-
tests/qemu-iotests/175 | 2 +-
tests/qemu-iotests/176 | 2 +-
tests/qemu-iotests/177 | 2 +-
tests/qemu-iotests/178 | 2 +-
tests/qemu-iotests/179 | 2 +-
tests/qemu-iotests/181 | 2 +-
tests/qemu-iotests/182 | 2 +-
tests/qemu-iotests/183 | 2 +-
tests/qemu-iotests/184 | 2 +-
tests/qemu-iotests/185 | 2 +-
tests/qemu-iotests/186 | 2 +-
tests/qemu-iotests/187 | 2 +-
tests/qemu-iotests/188 | 2 +-
tests/qemu-iotests/189 | 2 +-
tests/qemu-iotests/190 | 2 +-
tests/qemu-iotests/191 | 2 +-
tests/qemu-iotests/192 | 2 +-
tests/qemu-iotests/195 | 2 +-
tests/qemu-iotests/197 | 2 +-
tests/qemu-iotests/198 | 2 +-
tests/qemu-iotests/200 | 2 +-
tests/qemu-iotests/201 | 2 +-
tests/qemu-iotests/204 | 2 +-
tests/qemu-iotests/214 | 2 +-
tests/qemu-iotests/215 | 2 +-
tests/qemu-iotests/217 | 2 +-
tests/qemu-iotests/220 | 2 +-
tests/qemu-iotests/221 | 2 +-
tests/qemu-iotests/223 | 2 +-
tests/qemu-iotests/225 | 2 +-
tests/qemu-iotests/226 | 2 +-
tests/qemu-iotests/227 | 2 +-
tests/qemu-iotests/229 | 2 +-
tests/qemu-iotests/231 | 2 +-
tests/qemu-iotests/232 | 2 +-
tests/qemu-iotests/233 | 2 +-
tests/qemu-iotests/check | 2 +-
tests/qemu-iotests/common.config | 2 +-
tests/qemu-iotests/common.filter | 2 +-
tests/qemu-iotests/common.nbd | 2 +-
tests/qemu-iotests/common.pattern | 2 +-
tests/qemu-iotests/common.qemu | 2 +-
tests/qemu-iotests/common.rc | 2 +-
tests/qemu-iotests/common.tls | 2 +-
185 files changed, 185 insertions(+), 185 deletions(-)
diff --git a/tests/qemu-iotests/001 b/tests/qemu-iotests/001
index 55dcbb71d9..5d266e170a 100755
--- a/tests/qemu-iotests/001
+++ b/tests/qemu-iotests/001
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test simple read/write using plain bdrv_read/bdrv_write
#
diff --git a/tests/qemu-iotests/002 b/tests/qemu-iotests/002
index 74572b4711..7fb85084a1 100755
--- a/tests/qemu-iotests/002
+++ b/tests/qemu-iotests/002
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test simple read/write using plain bdrv_pread/bdrv_pwrite
#
diff --git a/tests/qemu-iotests/003 b/tests/qemu-iotests/003
index bf2595559b..f008c57cdc 100755
--- a/tests/qemu-iotests/003
+++ b/tests/qemu-iotests/003
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test simple read/write using bdrv_aio_readv/bdrv_aio_writev
#
diff --git a/tests/qemu-iotests/004 b/tests/qemu-iotests/004
index 841b15dfac..64fab3e714 100755
--- a/tests/qemu-iotests/004
+++ b/tests/qemu-iotests/004
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Make sure we can't read and write outside of the image size.
#
diff --git a/tests/qemu-iotests/005 b/tests/qemu-iotests/005
index 8aa4283a4d..2fef63af88 100755
--- a/tests/qemu-iotests/005
+++ b/tests/qemu-iotests/005
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Make sure qemu-img can create 5TB images
#
diff --git a/tests/qemu-iotests/007 b/tests/qemu-iotests/007
index b983022a7f..3ab5490db3 100755
--- a/tests/qemu-iotests/007
+++ b/tests/qemu-iotests/007
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Check for one possible case of qcow2 refcount corruption.
#
diff --git a/tests/qemu-iotests/008 b/tests/qemu-iotests/008
index 8dfa10bcb8..75067e36ad 100755
--- a/tests/qemu-iotests/008
+++ b/tests/qemu-iotests/008
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test simple asynchronous read/write operations.
#
diff --git a/tests/qemu-iotests/009 b/tests/qemu-iotests/009
index 73ae09db69..bc4b461122 100755
--- a/tests/qemu-iotests/009
+++ b/tests/qemu-iotests/009
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Nolan I qcow2 corruption - incorrectly reports free clusters
#
diff --git a/tests/qemu-iotests/010 b/tests/qemu-iotests/010
index 751aca9813..6920408d28 100755
--- a/tests/qemu-iotests/010
+++ b/tests/qemu-iotests/010
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Nolan II qcow2 corruption - wrong used cluster
#
diff --git a/tests/qemu-iotests/011 b/tests/qemu-iotests/011
index 35909564a9..b4c7e8f799 100755
--- a/tests/qemu-iotests/011
+++ b/tests/qemu-iotests/011
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test for AIO allocation on the same cluster
#
diff --git a/tests/qemu-iotests/012 b/tests/qemu-iotests/012
index de9a5fb4d5..2c3b42d9dd 100755
--- a/tests/qemu-iotests/012
+++ b/tests/qemu-iotests/012
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Make sure we can open read-only images
#
diff --git a/tests/qemu-iotests/013 b/tests/qemu-iotests/013
index 5e1efcee28..5cb9032f16 100755
--- a/tests/qemu-iotests/013
+++ b/tests/qemu-iotests/013
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# qcow2 pattern test, empty and compressed image - 4k cluster patterns
#
diff --git a/tests/qemu-iotests/014 b/tests/qemu-iotests/014
index 9ade571a95..2f728a1956 100755
--- a/tests/qemu-iotests/014
+++ b/tests/qemu-iotests/014
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# qcow2 pattern test, complex patterns including compression and snapshots
# Using patterns for 4k cluster size.
diff --git a/tests/qemu-iotests/015 b/tests/qemu-iotests/015
index 21f7d42c84..5a4063e4f5 100755
--- a/tests/qemu-iotests/015
+++ b/tests/qemu-iotests/015
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Combined test to grow the refcount table and test snapshots.
#
diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
index 1ac6f74502..83744f29a3 100755
--- a/tests/qemu-iotests/017
+++ b/tests/qemu-iotests/017
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Simple backing file reads
#
diff --git a/tests/qemu-iotests/018 b/tests/qemu-iotests/018
index bba30a1be2..78169838ba 100755
--- a/tests/qemu-iotests/018
+++ b/tests/qemu-iotests/018
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Merge backing file into test image when converting the image
#
diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019
index 8f911a79c1..a56dd30bed 100755
--- a/tests/qemu-iotests/019
+++ b/tests/qemu-iotests/019
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# When using a backing file for the output image in qemu-img convert,
# the backing file clusters must not copied. The data must still be
diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
index 6b972d082f..71fa753b4e 100755
--- a/tests/qemu-iotests/020
+++ b/tests/qemu-iotests/020
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Commit changes to backing file
#
diff --git a/tests/qemu-iotests/021 b/tests/qemu-iotests/021
index c15ebf9eb8..f6555f3b74 100755
--- a/tests/qemu-iotests/021
+++ b/tests/qemu-iotests/021
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test handling of invalid patterns arguments to qemu-io
#
diff --git a/tests/qemu-iotests/022 b/tests/qemu-iotests/022
index 44765c7b7a..b68cd64b33 100755
--- a/tests/qemu-iotests/022
+++ b/tests/qemu-iotests/022
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test bdrv_load/save_vmstate using the usual patterns
#
diff --git a/tests/qemu-iotests/023 b/tests/qemu-iotests/023
index c8e1b9a761..02ed047820 100755
--- a/tests/qemu-iotests/023
+++ b/tests/qemu-iotests/023
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# qcow2 pattern test with various cluster sizes
#
diff --git a/tests/qemu-iotests/024 b/tests/qemu-iotests/024
index 428b5c815d..23298c6f59 100755
--- a/tests/qemu-iotests/024
+++ b/tests/qemu-iotests/024
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Rebasing COW images
#
diff --git a/tests/qemu-iotests/025 b/tests/qemu-iotests/025
index fcd4d97c17..d9a4ebc5e7 100755
--- a/tests/qemu-iotests/025
+++ b/tests/qemu-iotests/025
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Resizing images
#
diff --git a/tests/qemu-iotests/026 b/tests/qemu-iotests/026
index 31276d9027..ca89ad7048 100755
--- a/tests/qemu-iotests/026
+++ b/tests/qemu-iotests/026
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# qcow2 error path testing
#
diff --git a/tests/qemu-iotests/027 b/tests/qemu-iotests/027
index 2c46ae1457..b7df9701f7 100755
--- a/tests/qemu-iotests/027
+++ b/tests/qemu-iotests/027
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test that sub-cluster allocating writes zero the rest of the cluster
#
diff --git a/tests/qemu-iotests/028 b/tests/qemu-iotests/028
index a2a7c93bcd..01f495912f 100755
--- a/tests/qemu-iotests/028
+++ b/tests/qemu-iotests/028
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test that backing files can be smaller than the image
#
diff --git a/tests/qemu-iotests/029 b/tests/qemu-iotests/029
index cf0fe0f6a6..5f42f76cc6 100755
--- a/tests/qemu-iotests/029
+++ b/tests/qemu-iotests/029
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# qcow2 internal snapshots/VM state tests
#
diff --git a/tests/qemu-iotests/031 b/tests/qemu-iotests/031
index ac0dfaed7d..ef92d8eee3 100755
--- a/tests/qemu-iotests/031
+++ b/tests/qemu-iotests/031
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test that all qcow2 header extensions survive a header rewrite
#
diff --git a/tests/qemu-iotests/032 b/tests/qemu-iotests/032
index 3e86bb0111..a1757bb15e 100755
--- a/tests/qemu-iotests/032
+++ b/tests/qemu-iotests/032
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test that AIO requests are drained before an image is closed. This used
# to segfault because the request coroutine kept running even after the
diff --git a/tests/qemu-iotests/033 b/tests/qemu-iotests/033
index 46b91388ef..cfdf1ec2ba 100755
--- a/tests/qemu-iotests/033
+++ b/tests/qemu-iotests/033
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test aligned and misaligned write zeroes operations.
#
diff --git a/tests/qemu-iotests/034 b/tests/qemu-iotests/034
index 62812cd53c..324bed28c6 100755
--- a/tests/qemu-iotests/034
+++ b/tests/qemu-iotests/034
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test bdrv_pwrite_zeroes with backing files (see also 154)
#
diff --git a/tests/qemu-iotests/035 b/tests/qemu-iotests/035
index 85d9ef7f8e..46aa835936 100755
--- a/tests/qemu-iotests/035
+++ b/tests/qemu-iotests/035
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Let a few AIO requests run in parallel and have them access different L2
# tables so that the cache has a chance to get used up.
diff --git a/tests/qemu-iotests/036 b/tests/qemu-iotests/036
index 4e76602a93..1b56394129 100755
--- a/tests/qemu-iotests/036
+++ b/tests/qemu-iotests/036
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test qcow2 feature bits
#
diff --git a/tests/qemu-iotests/037 b/tests/qemu-iotests/037
index a11992dad2..0781bebefe 100755
--- a/tests/qemu-iotests/037
+++ b/tests/qemu-iotests/037
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test COW from backing files
#
diff --git a/tests/qemu-iotests/038 b/tests/qemu-iotests/038
index 575093e8cf..707e2d72e9 100755
--- a/tests/qemu-iotests/038
+++ b/tests/qemu-iotests/038
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test COW from backing files with AIO
#
diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
index b3c344cb27..0d4e963bd4 100755
--- a/tests/qemu-iotests/039
+++ b/tests/qemu-iotests/039
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test qcow2 lazy refcounts
#
diff --git a/tests/qemu-iotests/042 b/tests/qemu-iotests/042
index beaa339000..a9a7fc3041 100755
--- a/tests/qemu-iotests/042
+++ b/tests/qemu-iotests/042
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test qemu-img operation on zero size images
#
diff --git a/tests/qemu-iotests/043 b/tests/qemu-iotests/043
index fc9005b28f..9894b154ec 100755
--- a/tests/qemu-iotests/043
+++ b/tests/qemu-iotests/043
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test that qemu-img info --backing-chain detects infinite loops
#
diff --git a/tests/qemu-iotests/046 b/tests/qemu-iotests/046
index 5e41d96daa..95160bea4c 100755
--- a/tests/qemu-iotests/046
+++ b/tests/qemu-iotests/046
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test concurrent cluster allocations
#
diff --git a/tests/qemu-iotests/047 b/tests/qemu-iotests/047
index 6e776d2ce5..ce81fc6fa7 100755
--- a/tests/qemu-iotests/047
+++ b/tests/qemu-iotests/047
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Regression test for commit b7ab0fea (which was a corruption fix,
# despite the commit message claiming otherwise)
diff --git a/tests/qemu-iotests/048 b/tests/qemu-iotests/048
index 9ed04a068d..bde408ca92 100755
--- a/tests/qemu-iotests/048
+++ b/tests/qemu-iotests/048
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
##
## qemu-img compare test
##
diff --git a/tests/qemu-iotests/049 b/tests/qemu-iotests/049
index 97d8a64697..bc09cd6717 100755
--- a/tests/qemu-iotests/049
+++ b/tests/qemu-iotests/049
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Check qemu-img option parsing
#
diff --git a/tests/qemu-iotests/050 b/tests/qemu-iotests/050
index 963a0db97f..dd7b2c72eb 100755
--- a/tests/qemu-iotests/050
+++ b/tests/qemu-iotests/050
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test qemu-img rebase with zero clusters
#
diff --git a/tests/qemu-iotests/051 b/tests/qemu-iotests/051
index 32741d7efd..3b50c7f188 100755
--- a/tests/qemu-iotests/051
+++ b/tests/qemu-iotests/051
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test command line configuration of block devices and driver-specific options
#
diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052
index b992adf4ff..b3a2dc1143 100755
--- a/tests/qemu-iotests/052
+++ b/tests/qemu-iotests/052
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test bdrv_read/bdrv_write using BDRV_O_SNAPSHOT
#
diff --git a/tests/qemu-iotests/053 b/tests/qemu-iotests/053
index afa109c950..50c62f0f56 100755
--- a/tests/qemu-iotests/053
+++ b/tests/qemu-iotests/053
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test qemu-img convert when image length is not a multiple of cluster size
#
diff --git a/tests/qemu-iotests/054 b/tests/qemu-iotests/054
index cf88a7c76e..0d5e14f847 100755
--- a/tests/qemu-iotests/054
+++ b/tests/qemu-iotests/054
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test huge qcow2 images
#
diff --git a/tests/qemu-iotests/058 b/tests/qemu-iotests/058
index d6d4f94d5d..8c3212a72f 100755
--- a/tests/qemu-iotests/058
+++ b/tests/qemu-iotests/058
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test export internal snapshot by qemu-nbd, convert it by qemu-img.
#
diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
index 54d5567acc..279aee6815 100755
--- a/tests/qemu-iotests/059
+++ b/tests/qemu-iotests/059
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for vmdk
#
diff --git a/tests/qemu-iotests/060 b/tests/qemu-iotests/060
index af0588ae9a..89e911400c 100755
--- a/tests/qemu-iotests/060
+++ b/tests/qemu-iotests/060
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for image corruption (overlapping data structures) in qcow2
#
diff --git a/tests/qemu-iotests/061 b/tests/qemu-iotests/061
index 1a50163419..d2e68c0203 100755
--- a/tests/qemu-iotests/061
+++ b/tests/qemu-iotests/061
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for image option amendment in qcow2.
#
diff --git a/tests/qemu-iotests/062 b/tests/qemu-iotests/062
index 985fbef41e..ed7400fed2 100755
--- a/tests/qemu-iotests/062
+++ b/tests/qemu-iotests/062
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for snapshotting images with unallocated zero clusters in
# qcow2
diff --git a/tests/qemu-iotests/063 b/tests/qemu-iotests/063
index 041fb5c1ac..2d5c0ce9fb 100755
--- a/tests/qemu-iotests/063
+++ b/tests/qemu-iotests/063
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# test of qemu-img convert -n - convert without creation
#
diff --git a/tests/qemu-iotests/064 b/tests/qemu-iotests/064
index f55ff37ca7..90673186ec 100755
--- a/tests/qemu-iotests/064
+++ b/tests/qemu-iotests/064
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test VHDX read/write from a sample image created with Hyper-V
#
diff --git a/tests/qemu-iotests/066 b/tests/qemu-iotests/066
index 26c043711b..f480986e35 100755
--- a/tests/qemu-iotests/066
+++ b/tests/qemu-iotests/066
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for preallocated zero clusters in qcow2
#
diff --git a/tests/qemu-iotests/067 b/tests/qemu-iotests/067
index 342b2b0a30..fda16a6b0d 100755
--- a/tests/qemu-iotests/067
+++ b/tests/qemu-iotests/067
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test automatic deletion of BDSes created by -drive/drive_add
#
diff --git a/tests/qemu-iotests/068 b/tests/qemu-iotests/068
index f0583d52ae..881a022107 100755
--- a/tests/qemu-iotests/068
+++ b/tests/qemu-iotests/068
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for loading a saved VM state from a qcow2 image
#
diff --git a/tests/qemu-iotests/069 b/tests/qemu-iotests/069
index fdee121f43..6a8e4aa22e 100755
--- a/tests/qemu-iotests/069
+++ b/tests/qemu-iotests/069
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for deleting a backing file
#
diff --git a/tests/qemu-iotests/070 b/tests/qemu-iotests/070
index 78e0390f5f..cb0f927c16 100755
--- a/tests/qemu-iotests/070
+++ b/tests/qemu-iotests/070
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test VHDX log replay from an image with a journal that needs to be
# replayed
diff --git a/tests/qemu-iotests/071 b/tests/qemu-iotests/071
index 6e467dc1da..7f3e5abd57 100755
--- a/tests/qemu-iotests/071
+++ b/tests/qemu-iotests/071
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for the QMP blkdebug and blkverify interfaces
#
diff --git a/tests/qemu-iotests/072 b/tests/qemu-iotests/072
index 08ef29f5b4..6f9f247fa5 100755
--- a/tests/qemu-iotests/072
+++ b/tests/qemu-iotests/072
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for nested image formats
#
diff --git a/tests/qemu-iotests/073 b/tests/qemu-iotests/073
index 5e7f76cb7f..990f90acbd 100755
--- a/tests/qemu-iotests/073
+++ b/tests/qemu-iotests/073
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test count_contiguous_clusters in qcow2
#
diff --git a/tests/qemu-iotests/074 b/tests/qemu-iotests/074
index b17866bd34..bb4ad1cc08 100755
--- a/tests/qemu-iotests/074
+++ b/tests/qemu-iotests/074
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
##
## qemu-img compare test (qcow2 only ones)
##
diff --git a/tests/qemu-iotests/075 b/tests/qemu-iotests/075
index 45b8901ef0..389d5675fa 100755
--- a/tests/qemu-iotests/075
+++ b/tests/qemu-iotests/075
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# cloop format input validation tests
#
diff --git a/tests/qemu-iotests/076 b/tests/qemu-iotests/076
index 3b5ab3fd08..0d405ef3f2 100755
--- a/tests/qemu-iotests/076
+++ b/tests/qemu-iotests/076
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# parallels format input validation tests
#
diff --git a/tests/qemu-iotests/077 b/tests/qemu-iotests/077
index 58fe8932b3..c284952082 100755
--- a/tests/qemu-iotests/077
+++ b/tests/qemu-iotests/077
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test concurrent pread/pwrite
#
diff --git a/tests/qemu-iotests/078 b/tests/qemu-iotests/078
index 68d0ea8802..54fc654d8e 100755
--- a/tests/qemu-iotests/078
+++ b/tests/qemu-iotests/078
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# bochs format input validation tests
#
diff --git a/tests/qemu-iotests/079 b/tests/qemu-iotests/079
index fca2f77d37..1b6594ebef 100755
--- a/tests/qemu-iotests/079
+++ b/tests/qemu-iotests/079
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test qcow2 preallocation with different cluster_sizes
#
diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080
index cec2376f59..4bcb5021e8 100755
--- a/tests/qemu-iotests/080
+++ b/tests/qemu-iotests/080
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# qcow2 format input validation tests
#
diff --git a/tests/qemu-iotests/081 b/tests/qemu-iotests/081
index edf6e6172a..c418bab093 100755
--- a/tests/qemu-iotests/081
+++ b/tests/qemu-iotests/081
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test Quorum block driver
#
diff --git a/tests/qemu-iotests/082 b/tests/qemu-iotests/082
index 61eec63797..d0afa46e9a 100755
--- a/tests/qemu-iotests/082
+++ b/tests/qemu-iotests/082
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test qemu-img command line parsing
#
diff --git a/tests/qemu-iotests/083 b/tests/qemu-iotests/083
index 89f67db70f..b270550d3e 100755
--- a/tests/qemu-iotests/083
+++ b/tests/qemu-iotests/083
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test NBD client unexpected disconnect
#
diff --git a/tests/qemu-iotests/084 b/tests/qemu-iotests/084
index e131fa9642..c29d7395e9 100755
--- a/tests/qemu-iotests/084
+++ b/tests/qemu-iotests/084
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for VDI header corruption; image too large, and too many blocks.
# Also simple test for creating dynamic and static VDI images.
diff --git a/tests/qemu-iotests/085 b/tests/qemu-iotests/085
index ade68ef853..68cb665987 100755
--- a/tests/qemu-iotests/085
+++ b/tests/qemu-iotests/085
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Live snapshot tests
#
diff --git a/tests/qemu-iotests/086 b/tests/qemu-iotests/086
index 3cca3687ea..fea1a7bd8a 100755
--- a/tests/qemu-iotests/086
+++ b/tests/qemu-iotests/086
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test qemu-img progress output
#
diff --git a/tests/qemu-iotests/087 b/tests/qemu-iotests/087
index f625887082..d6c8613419 100755
--- a/tests/qemu-iotests/087
+++ b/tests/qemu-iotests/087
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test unsupported blockdev-add cases
#
diff --git a/tests/qemu-iotests/088 b/tests/qemu-iotests/088
index c5e9ab42c7..b44edd0cf9 100755
--- a/tests/qemu-iotests/088
+++ b/tests/qemu-iotests/088
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# vpc (VHD) format input validation tests
#
diff --git a/tests/qemu-iotests/089 b/tests/qemu-iotests/089
index 3165d79e2a..6609954908 100755
--- a/tests/qemu-iotests/089
+++ b/tests/qemu-iotests/089
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for support of JSON filenames
#
diff --git a/tests/qemu-iotests/090 b/tests/qemu-iotests/090
index 1450993e15..193bae7d77 100755
--- a/tests/qemu-iotests/090
+++ b/tests/qemu-iotests/090
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test for discarding compressed clusters on qcow2 images
#
diff --git a/tests/qemu-iotests/091 b/tests/qemu-iotests/091
index 2f2f98ee64..d62ef18a02 100755
--- a/tests/qemu-iotests/091
+++ b/tests/qemu-iotests/091
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Live migration test
#
diff --git a/tests/qemu-iotests/092 b/tests/qemu-iotests/092
index 8e318f10b9..e2e0726de1 100755
--- a/tests/qemu-iotests/092
+++ b/tests/qemu-iotests/092
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# qcow1 format input validation tests
#
diff --git a/tests/qemu-iotests/094 b/tests/qemu-iotests/094
index 7adc9b9138..0bcca77261 100755
--- a/tests/qemu-iotests/094
+++ b/tests/qemu-iotests/094
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for drive-mirror to NBD
#
diff --git a/tests/qemu-iotests/095 b/tests/qemu-iotests/095
index 9fc47f6b87..18505b7181 100755
--- a/tests/qemu-iotests/095
+++ b/tests/qemu-iotests/095
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test for commit of larger active layer
#
diff --git a/tests/qemu-iotests/097 b/tests/qemu-iotests/097
index 7234b16053..690f3d3ce1 100755
--- a/tests/qemu-iotests/097
+++ b/tests/qemu-iotests/097
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Commit changes into backing chains and empty the top image if the
# backing image is not explicitly specified
diff --git a/tests/qemu-iotests/098 b/tests/qemu-iotests/098
index c7977da99a..461144c831 100755
--- a/tests/qemu-iotests/098
+++ b/tests/qemu-iotests/098
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test qcow2's bdrv_make_empty for images without internal snapshots
#
diff --git a/tests/qemu-iotests/099 b/tests/qemu-iotests/099
index 578808b747..ae02f27afe 100755
--- a/tests/qemu-iotests/099
+++ b/tests/qemu-iotests/099
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test valid filenames for blkdebug and blkverify representatively for
# other protocols (such as NBD) when queried
diff --git a/tests/qemu-iotests/101 b/tests/qemu-iotests/101
index 3001ba3c0a..a4c1b6366a 100755
--- a/tests/qemu-iotests/101
+++ b/tests/qemu-iotests/101
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test short file I/O
#
diff --git a/tests/qemu-iotests/102 b/tests/qemu-iotests/102
index 29a6a940e2..cedd2b25dc 100755
--- a/tests/qemu-iotests/102
+++ b/tests/qemu-iotests/102
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for qemu-io -c map and qemu-img map
#
diff --git a/tests/qemu-iotests/103 b/tests/qemu-iotests/103
index 66f8167f02..6773e94d9f 100755
--- a/tests/qemu-iotests/103
+++ b/tests/qemu-iotests/103
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for qcow2 metadata cache size specification
#
diff --git a/tests/qemu-iotests/104 b/tests/qemu-iotests/104
index 34bb0d23ba..390167bad4 100755
--- a/tests/qemu-iotests/104
+++ b/tests/qemu-iotests/104
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test image creation with aligned and unaligned sizes
#
diff --git a/tests/qemu-iotests/105 b/tests/qemu-iotests/105
index 943bda2f4f..3b5a596844 100755
--- a/tests/qemu-iotests/105
+++ b/tests/qemu-iotests/105
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Create, read, write big image
#
diff --git a/tests/qemu-iotests/106 b/tests/qemu-iotests/106
index 4129fee6bc..ac47eaa0f5 100755
--- a/tests/qemu-iotests/106
+++ b/tests/qemu-iotests/106
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test preallocated resize of raw images
#
diff --git a/tests/qemu-iotests/107 b/tests/qemu-iotests/107
index 5d70ad2007..fcd5a24dfe 100755
--- a/tests/qemu-iotests/107
+++ b/tests/qemu-iotests/107
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Tests updates of the qcow2 L1 table
#
diff --git a/tests/qemu-iotests/108 b/tests/qemu-iotests/108
index 58e8ad7636..9c08172237 100755
--- a/tests/qemu-iotests/108
+++ b/tests/qemu-iotests/108
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for repairing qcow2 images which cannot be repaired using
# the on-disk refcount structures
diff --git a/tests/qemu-iotests/109 b/tests/qemu-iotests/109
index b51e4616c6..9897ceb6cd 100755
--- a/tests/qemu-iotests/109
+++ b/tests/qemu-iotests/109
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test writing image headers of other formats into raw images
#
diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110
index 185ad5437e..fad672c1ae 100755
--- a/tests/qemu-iotests/110
+++ b/tests/qemu-iotests/110
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for relative backing file names in complex BDS trees
#
diff --git a/tests/qemu-iotests/111 b/tests/qemu-iotests/111
index e15e66ac5d..57395be64c 100755
--- a/tests/qemu-iotests/111
+++ b/tests/qemu-iotests/111
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for non-existing backing file when creating a qcow2 image
# and not specifying the size
diff --git a/tests/qemu-iotests/112 b/tests/qemu-iotests/112
index d67e6ebe9c..6d81c75a9c 100755
--- a/tests/qemu-iotests/112
+++ b/tests/qemu-iotests/112
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test cases for different refcount_bits values
#
diff --git a/tests/qemu-iotests/113 b/tests/qemu-iotests/113
index d8d78c46dc..f2703a2c50 100755
--- a/tests/qemu-iotests/113
+++ b/tests/qemu-iotests/113
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for accessing creation options on image formats and
# protocols not supporting image creation
diff --git a/tests/qemu-iotests/114 b/tests/qemu-iotests/114
index e17fb514cb..f36b88f3f3 100755
--- a/tests/qemu-iotests/114
+++ b/tests/qemu-iotests/114
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test invalid backing file format in qcow2 images
#
diff --git a/tests/qemu-iotests/115 b/tests/qemu-iotests/115
index 0581e03c26..7ed347010f 100755
--- a/tests/qemu-iotests/115
+++ b/tests/qemu-iotests/115
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for non-self-referential qcow2 refcount blocks
#
diff --git a/tests/qemu-iotests/116 b/tests/qemu-iotests/116
index f8a27b9c02..941b07a1a9 100755
--- a/tests/qemu-iotests/116
+++ b/tests/qemu-iotests/116
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test error code paths for invalid QED images
#
diff --git a/tests/qemu-iotests/117 b/tests/qemu-iotests/117
index e533e230a3..0af0f31c5a 100755
--- a/tests/qemu-iotests/117
+++ b/tests/qemu-iotests/117
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for shared BDS between backend trees
#
diff --git a/tests/qemu-iotests/119 b/tests/qemu-iotests/119
index 32810d52c9..ea6770a484 100755
--- a/tests/qemu-iotests/119
+++ b/tests/qemu-iotests/119
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# NBD test case for overriding BDRV_O_PROTOCOL by explicitly specifying
# a driver
diff --git a/tests/qemu-iotests/120 b/tests/qemu-iotests/120
index 76afdf449b..ca95b9276e 100755
--- a/tests/qemu-iotests/120
+++ b/tests/qemu-iotests/120
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Non-NBD test cases for overriding BDRV_O_PROTOCOL by explicitly
# specifying a driver
diff --git a/tests/qemu-iotests/121 b/tests/qemu-iotests/121
index d2885c700f..90a0424edb 100755
--- a/tests/qemu-iotests/121
+++ b/tests/qemu-iotests/121
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test cases for qcow2 refcount table growth
#
diff --git a/tests/qemu-iotests/122 b/tests/qemu-iotests/122
index eab3399dd6..85c3a8d047 100755
--- a/tests/qemu-iotests/122
+++ b/tests/qemu-iotests/122
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test some qemu-img convert cases
#
diff --git a/tests/qemu-iotests/123 b/tests/qemu-iotests/123
index 168b985c8b..d33950eb54 100755
--- a/tests/qemu-iotests/123
+++ b/tests/qemu-iotests/123
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for qemu-img convert to NBD
#
diff --git a/tests/qemu-iotests/125 b/tests/qemu-iotests/125
index 778c874933..212dcd8f0d 100755
--- a/tests/qemu-iotests/125
+++ b/tests/qemu-iotests/125
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test preallocated growth of qcow2 images
#
diff --git a/tests/qemu-iotests/126 b/tests/qemu-iotests/126
index 91148383ad..96dc048d59 100755
--- a/tests/qemu-iotests/126
+++ b/tests/qemu-iotests/126
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Tests handling of colons in filenames (which may be confused with protocol
# prefixes)
diff --git a/tests/qemu-iotests/127 b/tests/qemu-iotests/127
index c9139ed5e6..3e941f74d4 100755
--- a/tests/qemu-iotests/127
+++ b/tests/qemu-iotests/127
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for mirroring with dataplane
#
diff --git a/tests/qemu-iotests/128 b/tests/qemu-iotests/128
index 925f5c7e98..3606c41760 100755
--- a/tests/qemu-iotests/128
+++ b/tests/qemu-iotests/128
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test that opening O_DIRECT succeeds when image file I/O produces EIO
#
diff --git a/tests/qemu-iotests/130 b/tests/qemu-iotests/130
index f2f2706b28..77ad2aa13a 100755
--- a/tests/qemu-iotests/130
+++ b/tests/qemu-iotests/130
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test that temporary backing file overrides (on the command line or in
# blockdev-add) don't replace the original path stored in the image during
diff --git a/tests/qemu-iotests/131 b/tests/qemu-iotests/131
index 58c25f7abe..27870231cf 100755
--- a/tests/qemu-iotests/131
+++ b/tests/qemu-iotests/131
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# parallels format validation tests (created by QEMU)
#
diff --git a/tests/qemu-iotests/133 b/tests/qemu-iotests/133
index 565e0b1b6e..1f6056d144 100755
--- a/tests/qemu-iotests/133
+++ b/tests/qemu-iotests/133
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test for reopen
#
diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134
index cacabcd28b..e9e3e84c2a 100755
--- a/tests/qemu-iotests/134
+++ b/tests/qemu-iotests/134
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test encrypted read/write using plain bdrv_read/bdrv_write
#
diff --git a/tests/qemu-iotests/135 b/tests/qemu-iotests/135
index a18a0c7230..3b3d1dc2a5 100755
--- a/tests/qemu-iotests/135
+++ b/tests/qemu-iotests/135
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test VPC open of image with large Max Table Entries value.
#
diff --git a/tests/qemu-iotests/137 b/tests/qemu-iotests/137
index 09cd4450ca..0c3d2a1cf0 100755
--- a/tests/qemu-iotests/137
+++ b/tests/qemu-iotests/137
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test qcow2 reopen
#
diff --git a/tests/qemu-iotests/138 b/tests/qemu-iotests/138
index eccbcae3a6..f353ac8219 100755
--- a/tests/qemu-iotests/138
+++ b/tests/qemu-iotests/138
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# General test case for qcow2's image check
#
diff --git a/tests/qemu-iotests/140 b/tests/qemu-iotests/140
index d4623b5a5d..b965b1dd5d 100755
--- a/tests/qemu-iotests/140
+++ b/tests/qemu-iotests/140
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for ejecting a BlockBackend with an NBD server attached to it
#
diff --git a/tests/qemu-iotests/141 b/tests/qemu-iotests/141
index e2408c7988..2197a82d45 100755
--- a/tests/qemu-iotests/141
+++ b/tests/qemu-iotests/141
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for ejecting BDSs with block jobs still running on them
#
diff --git a/tests/qemu-iotests/142 b/tests/qemu-iotests/142
index 5fc488f5d2..d9b98cf60a 100755
--- a/tests/qemu-iotests/142
+++ b/tests/qemu-iotests/142
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test for configuring cache modes of arbitrary nodes (requires O_DIRECT)
#
diff --git a/tests/qemu-iotests/143 b/tests/qemu-iotests/143
index d6302cc06d..c223867cb3 100755
--- a/tests/qemu-iotests/143
+++ b/tests/qemu-iotests/143
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for connecting to a non-existing NBD export name
#
diff --git a/tests/qemu-iotests/144 b/tests/qemu-iotests/144
index 118c099994..15157f33d7 100755
--- a/tests/qemu-iotests/144
+++ b/tests/qemu-iotests/144
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
# Check live snapshot, followed by active commit, and another snapshot.
#
# This test is to catch the error case of BZ #1300209:
diff --git a/tests/qemu-iotests/145 b/tests/qemu-iotests/145
index 6ce8a46f92..28878dc8a1 100755
--- a/tests/qemu-iotests/145
+++ b/tests/qemu-iotests/145
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test the combination of -incoming and snapshot=on
#
diff --git a/tests/qemu-iotests/146 b/tests/qemu-iotests/146
index 3f61351ffe..2e43abddfc 100755
--- a/tests/qemu-iotests/146
+++ b/tests/qemu-iotests/146
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test VHD image format creator detection and override
#
diff --git a/tests/qemu-iotests/150 b/tests/qemu-iotests/150
index 955b877efa..3b1f32197a 100755
--- a/tests/qemu-iotests/150
+++ b/tests/qemu-iotests/150
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test that qemu-img convert -S 0 fully allocates the target image
#
diff --git a/tests/qemu-iotests/153 b/tests/qemu-iotests/153
index 3120a61da4..c989c2495f 100755
--- a/tests/qemu-iotests/153
+++ b/tests/qemu-iotests/153
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test image locking
#
diff --git a/tests/qemu-iotests/154 b/tests/qemu-iotests/154
index 4a4abf0589..d68f66b9e0 100755
--- a/tests/qemu-iotests/154
+++ b/tests/qemu-iotests/154
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# qcow2 specific bdrv_pwrite_zeroes tests with backing files (complements 034)
#
diff --git a/tests/qemu-iotests/156 b/tests/qemu-iotests/156
index f97f96f666..8d134029c6 100755
--- a/tests/qemu-iotests/156
+++ b/tests/qemu-iotests/156
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Tests oVirt-like storage migration:
# - Create snapshot
diff --git a/tests/qemu-iotests/157 b/tests/qemu-iotests/157
index 6fb26596ad..69b25cab30 100755
--- a/tests/qemu-iotests/157
+++ b/tests/qemu-iotests/157
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test command line configuration of block devices with qdev
#
diff --git a/tests/qemu-iotests/158 b/tests/qemu-iotests/158
index d277ddcc94..8c0928a7f9 100755
--- a/tests/qemu-iotests/158
+++ b/tests/qemu-iotests/158
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test encrypted read/write using backing files
#
diff --git a/tests/qemu-iotests/159 b/tests/qemu-iotests/159
index e74b2739de..29066eebde 100755
--- a/tests/qemu-iotests/159
+++ b/tests/qemu-iotests/159
@@ -1,4 +1,4 @@
-#! /bin/bash
+#!/usr/bin/env bash
#
# qemu-img dd test with different block sizes
#
diff --git a/tests/qemu-iotests/160 b/tests/qemu-iotests/160
index 92fff45d10..df89d3864b 100755
--- a/tests/qemu-iotests/160
+++ b/tests/qemu-iotests/160
@@ -1,4 +1,4 @@
-#! /bin/bash
+#!/usr/bin/env bash
#
# qemu-img dd test for the skip option
#
diff --git a/tests/qemu-iotests/161 b/tests/qemu-iotests/161
index 180df17ad6..456a4bd8c4 100755
--- a/tests/qemu-iotests/161
+++ b/tests/qemu-iotests/161
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test reopening a backing image after block-stream and block-commit
#
diff --git a/tests/qemu-iotests/162 b/tests/qemu-iotests/162
index ef02d844a2..2e9947fd9a 100755
--- a/tests/qemu-iotests/162
+++ b/tests/qemu-iotests/162
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for specifying runtime options of the wrong type to some
# block drivers
diff --git a/tests/qemu-iotests/170 b/tests/qemu-iotests/170
index 861eabf5cc..7deb7563c9 100755
--- a/tests/qemu-iotests/170
+++ b/tests/qemu-iotests/170
@@ -1,4 +1,4 @@
-#! /bin/bash
+#!/usr/bin/env bash
#
# qemu-img dd test
#
diff --git a/tests/qemu-iotests/171 b/tests/qemu-iotests/171
index 5b46069fde..341064a1c6 100755
--- a/tests/qemu-iotests/171
+++ b/tests/qemu-iotests/171
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test 'offset' and 'size' options of the raw driver. Make sure we can't
# (or can) read and write outside of the image size.
diff --git a/tests/qemu-iotests/172 b/tests/qemu-iotests/172
index 1e60a7e3d6..ba7dad9057 100755
--- a/tests/qemu-iotests/172
+++ b/tests/qemu-iotests/172
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test floppy configuration
#
diff --git a/tests/qemu-iotests/173 b/tests/qemu-iotests/173
index 1fe8c5d738..47036a5564 100755
--- a/tests/qemu-iotests/173
+++ b/tests/qemu-iotests/173
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test QAPI commands looking up protocol based images with relative
# filename backing strings
diff --git a/tests/qemu-iotests/174 b/tests/qemu-iotests/174
index d8bb05c4e2..0a952a73fd 100755
--- a/tests/qemu-iotests/174
+++ b/tests/qemu-iotests/174
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test that qemu-io fail with non-zero exit code
#
diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index ebbeb6e74c..d0ffc495c2 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test creating raw image preallocation mode
#
diff --git a/tests/qemu-iotests/176 b/tests/qemu-iotests/176
index 4ecd5894a3..50df4c00fa 100755
--- a/tests/qemu-iotests/176
+++ b/tests/qemu-iotests/176
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Commit changes into backing chains and empty the top image if the
# backing image is not explicitly specified.
diff --git a/tests/qemu-iotests/177 b/tests/qemu-iotests/177
index f0c1155e80..752d29f8ad 100755
--- a/tests/qemu-iotests/177
+++ b/tests/qemu-iotests/177
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test corner cases with unusual block geometries
#
diff --git a/tests/qemu-iotests/178 b/tests/qemu-iotests/178
index 927bf06e4d..21231cadd3 100755
--- a/tests/qemu-iotests/178
+++ b/tests/qemu-iotests/178
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# qemu-img measure sub-command tests
#
diff --git a/tests/qemu-iotests/179 b/tests/qemu-iotests/179
index 3040631636..9372dc30ef 100755
--- a/tests/qemu-iotests/179
+++ b/tests/qemu-iotests/179
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for write zeroes with unmap
#
diff --git a/tests/qemu-iotests/181 b/tests/qemu-iotests/181
index 0c44108dac..e317e63422 100755
--- a/tests/qemu-iotests/181
+++ b/tests/qemu-iotests/181
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test postcopy live migration with shared storage
#
diff --git a/tests/qemu-iotests/182 b/tests/qemu-iotests/182
index 9e078c5484..ff3d7e7ec1 100755
--- a/tests/qemu-iotests/182
+++ b/tests/qemu-iotests/182
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test image locking for POSIX locks
#
diff --git a/tests/qemu-iotests/183 b/tests/qemu-iotests/183
index ebb5e304ac..93b7bd798a 100755
--- a/tests/qemu-iotests/183
+++ b/tests/qemu-iotests/183
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test old-style block migration (migrate -b)
#
diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184
index 0af7a73aca..cb0c181228 100755
--- a/tests/qemu-iotests/184
+++ b/tests/qemu-iotests/184
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test I/O throttle block filter driver interface
#
diff --git a/tests/qemu-iotests/185 b/tests/qemu-iotests/185
index d8f1505cd8..454ff600cc 100755
--- a/tests/qemu-iotests/185
+++ b/tests/qemu-iotests/185
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test exiting qemu while jobs are still running
#
diff --git a/tests/qemu-iotests/186 b/tests/qemu-iotests/186
index c27dc953b6..5dd2177b89 100755
--- a/tests/qemu-iotests/186
+++ b/tests/qemu-iotests/186
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test 'info block' with all kinds of configurations
#
diff --git a/tests/qemu-iotests/187 b/tests/qemu-iotests/187
index 1feddca508..a45addde09 100755
--- a/tests/qemu-iotests/187
+++ b/tests/qemu-iotests/187
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test switching between read-only and read-write
#
diff --git a/tests/qemu-iotests/188 b/tests/qemu-iotests/188
index af40e496ee..be7278aa65 100755
--- a/tests/qemu-iotests/188
+++ b/tests/qemu-iotests/188
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test encrypted read/write using plain bdrv_read/bdrv_write
#
diff --git a/tests/qemu-iotests/189 b/tests/qemu-iotests/189
index 222bec133b..c9ce9d3bed 100755
--- a/tests/qemu-iotests/189
+++ b/tests/qemu-iotests/189
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test encrypted read/write using backing files
#
diff --git a/tests/qemu-iotests/190 b/tests/qemu-iotests/190
index 95ba06d8f4..e1c1d407f0 100755
--- a/tests/qemu-iotests/190
+++ b/tests/qemu-iotests/190
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# qemu-img measure sub-command tests on huge qcow2 files
#
diff --git a/tests/qemu-iotests/191 b/tests/qemu-iotests/191
index 198272ea3b..1ea908ce3d 100755
--- a/tests/qemu-iotests/191
+++ b/tests/qemu-iotests/191
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test commit block job where top has two parents
#
diff --git a/tests/qemu-iotests/192 b/tests/qemu-iotests/192
index 415c706db5..158086f9d2 100755
--- a/tests/qemu-iotests/192
+++ b/tests/qemu-iotests/192
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test NBD export with -incoming (non-shared storage migration use case from
# libvirt)
diff --git a/tests/qemu-iotests/195 b/tests/qemu-iotests/195
index a977c9798e..bd1b71ae5e 100755
--- a/tests/qemu-iotests/195
+++ b/tests/qemu-iotests/195
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test change-backing-file command
#
diff --git a/tests/qemu-iotests/197 b/tests/qemu-iotests/197
index 8170f5d4ab..2c664793f4 100755
--- a/tests/qemu-iotests/197
+++ b/tests/qemu-iotests/197
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for copy-on-read into qcow2
#
diff --git a/tests/qemu-iotests/198 b/tests/qemu-iotests/198
index 4d961f4f3a..c8f824cfae 100755
--- a/tests/qemu-iotests/198
+++ b/tests/qemu-iotests/198
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test commit of encrypted qcow2 files
#
diff --git a/tests/qemu-iotests/200 b/tests/qemu-iotests/200
index b9ebd5a8c7..12d25f4a1c 100755
--- a/tests/qemu-iotests/200
+++ b/tests/qemu-iotests/200
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Block job co-routine race condition test.
#
diff --git a/tests/qemu-iotests/201 b/tests/qemu-iotests/201
index c1a1e00077..7abf740fe4 100755
--- a/tests/qemu-iotests/201
+++ b/tests/qemu-iotests/201
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test savevm and loadvm after live migration with postcopy flag
#
diff --git a/tests/qemu-iotests/204 b/tests/qemu-iotests/204
index 30f0653ce9..abb73dc381 100755
--- a/tests/qemu-iotests/204
+++ b/tests/qemu-iotests/204
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test corner cases with unusual block geometries
#
diff --git a/tests/qemu-iotests/214 b/tests/qemu-iotests/214
index 7a2d5391bb..c1a452ff9a 100755
--- a/tests/qemu-iotests/214
+++ b/tests/qemu-iotests/214
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test qcow2 image compression
#
diff --git a/tests/qemu-iotests/215 b/tests/qemu-iotests/215
index 230fd2551a..7b063d7cfa 100755
--- a/tests/qemu-iotests/215
+++ b/tests/qemu-iotests/215
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test case for copy-on-read into qcow2, using the COR filter driver
#
diff --git a/tests/qemu-iotests/217 b/tests/qemu-iotests/217
index d3ab5d72be..f5482bb669 100755
--- a/tests/qemu-iotests/217
+++ b/tests/qemu-iotests/217
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# I/O errors when working with internal qcow2 snapshots, and repairing
# the result
diff --git a/tests/qemu-iotests/220 b/tests/qemu-iotests/220
index 0c5682bda0..2d62c5dcac 100755
--- a/tests/qemu-iotests/220
+++ b/tests/qemu-iotests/220
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# max limits on compression in huge qcow2 files
#
diff --git a/tests/qemu-iotests/221 b/tests/qemu-iotests/221
index 06f48f1f23..808cd9a289 100755
--- a/tests/qemu-iotests/221
+++ b/tests/qemu-iotests/221
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test qemu-img vs. unaligned images
#
diff --git a/tests/qemu-iotests/223 b/tests/qemu-iotests/223
index f120a01646..98b6cc73af 100755
--- a/tests/qemu-iotests/223
+++ b/tests/qemu-iotests/223
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test reading dirty bitmap over NBD
#
diff --git a/tests/qemu-iotests/225 b/tests/qemu-iotests/225
index e42ee94ff0..fbd7404791 100755
--- a/tests/qemu-iotests/225
+++ b/tests/qemu-iotests/225
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test vmdk backing file correlation
#
diff --git a/tests/qemu-iotests/226 b/tests/qemu-iotests/226
index aec413b23c..c1e1fb2b1c 100755
--- a/tests/qemu-iotests/226
+++ b/tests/qemu-iotests/226
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# This test covers expected filetypes for the file, host_cdrom and
# host_device drivers.
diff --git a/tests/qemu-iotests/227 b/tests/qemu-iotests/227
index be1b636af0..10cf144eb0 100755
--- a/tests/qemu-iotests/227
+++ b/tests/qemu-iotests/227
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test query-blockstats with different ways to create a BB
#
diff --git a/tests/qemu-iotests/229 b/tests/qemu-iotests/229
index b0d4885fa6..e18a464fe0 100755
--- a/tests/qemu-iotests/229
+++ b/tests/qemu-iotests/229
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test for force canceling a running blockjob that is paused in
# an error state.
diff --git a/tests/qemu-iotests/231 b/tests/qemu-iotests/231
index e9f8aaacd3..5b2cbab9ac 100755
--- a/tests/qemu-iotests/231
+++ b/tests/qemu-iotests/231
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test legacy and modern option parsing for rbd/ceph. This will not
# actually connect to a ceph server, but rather looks for the appropriate
diff --git a/tests/qemu-iotests/232 b/tests/qemu-iotests/232
index e48bc8f5db..71fd48eff0 100755
--- a/tests/qemu-iotests/232
+++ b/tests/qemu-iotests/232
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test for auto-read-only
#
diff --git a/tests/qemu-iotests/233 b/tests/qemu-iotests/233
index adb742fafb..876cd5997b 100755
--- a/tests/qemu-iotests/233
+++ b/tests/qemu-iotests/233
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Test NBD TLS certificate / authorization integration
#
diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index 1016887438..f9c24b6753 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Copyright (C) 2009 Red Hat, Inc.
# Copyright (c) 2000-2002,2006 Silicon Graphics, Inc. All Rights Reserved.
diff --git a/tests/qemu-iotests/common.config b/tests/qemu-iotests/common.config
index 9f460f203d..9bd1a5a6fc 100644
--- a/tests/qemu-iotests/common.config
+++ b/tests/qemu-iotests/common.config
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Copyright (C) 2009 Red Hat, Inc.
# Copyright (c) 2000-2003,2006 Silicon Graphics, Inc. All Rights Reserved.
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index 1aa7d57140..c282bc24f0 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Copyright (C) 2009 Red Hat, Inc.
# Copyright (c) 2000-2001 Silicon Graphics, Inc. All Rights Reserved.
diff --git a/tests/qemu-iotests/common.nbd b/tests/qemu-iotests/common.nbd
index 233187a25c..25fc9ffaa4 100644
--- a/tests/qemu-iotests/common.nbd
+++ b/tests/qemu-iotests/common.nbd
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
# -*- shell-script-mode -*-
#
# Helpers for NBD server related config
diff --git a/tests/qemu-iotests/common.pattern b/tests/qemu-iotests/common.pattern
index b67bb34136..25aa0d01c1 100644
--- a/tests/qemu-iotests/common.pattern
+++ b/tests/qemu-iotests/common.pattern
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Copyright (C) 2009 Red Hat, Inc.
#
diff --git a/tests/qemu-iotests/common.qemu b/tests/qemu-iotests/common.qemu
index 7c87b897fa..8d2021a7eb 100644
--- a/tests/qemu-iotests/common.qemu
+++ b/tests/qemu-iotests/common.qemu
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# This allows for launching of multiple QEMU instances, with independent
# communication possible to each instance.
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index 09a27f02d0..f21020eba6 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Copyright (C) 2009 Red Hat, Inc.
# Copyright (c) 2000-2006 Silicon Graphics, Inc. All Rights Reserved.
diff --git a/tests/qemu-iotests/common.tls b/tests/qemu-iotests/common.tls
index 3caf989d28..54c331d7a5 100644
--- a/tests/qemu-iotests/common.tls
+++ b/tests/qemu-iotests/common.tls
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
#
# Helpers for TLS related config
#
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 10/33] qemu-iotests: Ensure GNU sed is used
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (8 preceding siblings ...)
2019-03-08 12:57 ` [Qemu-devel] [PULL 09/33] qemu-iotests: " Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 11/33] qemu-iotests: Test qcow2 preallocation modes Kevin Wolf
` (23 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
From: Philippe Mathieu-Daudé <philmd@redhat.com>
Various sed regexp from common.filter use sed GNU extensions.
Instead of spending time to write these regex to be POSIX compliant,
verify the GNU sed is available and use it.
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/qemu-iotests/common.filter | 36 ++++++++++++++++----------------
tests/qemu-iotests/common.rc | 13 ++++++++++++
2 files changed, 31 insertions(+), 18 deletions(-)
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index c282bc24f0..35fddc746f 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -23,37 +23,37 @@
#
_filter_date()
{
- sed \
+ $SED \
-e 's/[A-Z][a-z][a-z] [A-z][a-z][a-z] *[0-9][0-9]* [0-9][0-9]:[0-9][0-9]:[0-9][0-9] [0-9][0-9][0-9][0-9]$/DATE/'
}
_filter_generated_node_ids()
{
- sed -re 's/\#block[0-9]{3,}/NODE_NAME/'
+ $SED -re 's/\#block[0-9]{3,}/NODE_NAME/'
}
_filter_qom_path()
{
- sed -e 's#\(Attached to: *\) /.*#\1 PATH#'
+ $SED -e 's#\(Attached to: *\) /.*#\1 PATH#'
}
# replace occurrences of the actual TEST_DIR value with TEST_DIR
_filter_testdir()
{
- sed -e "s#$TEST_DIR/#TEST_DIR/#g"
+ $SED -e "s#$TEST_DIR/#TEST_DIR/#g"
}
# replace occurrences of the actual IMGFMT value with IMGFMT
_filter_imgfmt()
{
- sed -e "s#$IMGFMT#IMGFMT#g"
+ $SED -e "s#$IMGFMT#IMGFMT#g"
}
# Replace error message when the format is not supported and delete
# the output lines after the first one
_filter_qemu_img_check()
{
- sed -e '/allocated.*fragmented.*compressed clusters/d' \
+ $SED -e '/allocated.*fragmented.*compressed clusters/d' \
-e 's/qemu-img: This image format does not support checks/No errors were found on the image./' \
-e '/Image end offset: [0-9]\+/d'
}
@@ -61,13 +61,13 @@ _filter_qemu_img_check()
# Removes \r from messages
_filter_win32()
{
- sed -e 's/\r//g'
+ $SED -e 's/\r//g'
}
# sanitize qemu-io output
_filter_qemu_io()
{
- _filter_win32 | sed -e "s/[0-9]* ops\; [0-9/:. sec]* ([0-9/.inf]* [EPTGMKiBbytes]*\/sec and [0-9/.inf]* ops\/sec)/X ops\; XX:XX:XX.X (XXX YYY\/sec and XXX ops\/sec)/" \
+ _filter_win32 | $SED -e "s/[0-9]* ops\; [0-9/:. sec]* ([0-9/.inf]* [EPTGMKiBbytes]*\/sec and [0-9/.inf]* ops\/sec)/X ops\; XX:XX:XX.X (XXX YYY\/sec and XXX ops\/sec)/" \
-e "s/: line [0-9][0-9]*: *[0-9][0-9]*\( Aborted\| Killed\)/:\1/" \
-e "s/qemu-io> //g"
}
@@ -75,7 +75,7 @@ _filter_qemu_io()
# replace occurrences of QEMU_PROG with "qemu"
_filter_qemu()
{
- sed -e "s#\\(^\\|(qemu) \\)$(basename $QEMU_PROG):#\1QEMU_PROG:#" \
+ $SED -e "s#\\(^\\|(qemu) \\)$(basename $QEMU_PROG):#\1QEMU_PROG:#" \
-e 's#^QEMU [0-9]\+\.[0-9]\+\.[0-9]\+ monitor#QEMU X.Y.Z monitor#' \
-e $'s#\r##' # QEMU monitor uses \r\n line endings
}
@@ -84,7 +84,7 @@ _filter_qemu()
_filter_qmp()
{
_filter_win32 | \
- sed -e 's#\("\(micro\)\?seconds": \)[0-9]\+#\1 TIMESTAMP#g' \
+ $SED -e 's#\("\(micro\)\?seconds": \)[0-9]\+#\1 TIMESTAMP#g' \
-e 's#^{"QMP":.*}$#QMP_VERSION#' \
-e '/^ "QMP": {\s*$/, /^ }\s*$/ c\' \
-e ' QMP_VERSION'
@@ -93,32 +93,32 @@ _filter_qmp()
# readline makes HMP command strings so long that git complains
_filter_hmp()
{
- sed -e $'s/^\\((qemu) \\)\\?.*\e\\[D/\\1/g' \
+ $SED -e $'s/^\\((qemu) \\)\\?.*\e\\[D/\\1/g' \
-e $'s/\e\\[K//g'
}
# replace block job offset
_filter_block_job_offset()
{
- sed -e 's/, "offset": [0-9]\+,/, "offset": OFFSET,/'
+ $SED -e 's/, "offset": [0-9]\+,/, "offset": OFFSET,/'
}
# replace block job len
_filter_block_job_len()
{
- sed -e 's/, "len": [0-9]\+,/, "len": LEN,/g'
+ $SED -e 's/, "len": [0-9]\+,/, "len": LEN,/g'
}
# replace actual image size (depends on the host filesystem)
_filter_actual_image_size()
{
- sed -s 's/\("actual-size":\s*\)[0-9]\+/\1SIZE/g'
+ $SED -s 's/\("actual-size":\s*\)[0-9]\+/\1SIZE/g'
}
# replace driver-specific options in the "Formatting..." line
_filter_img_create()
{
- sed -e "s#$REMOTE_TEST_DIR#TEST_DIR#g" \
+ $SED -e "s#$REMOTE_TEST_DIR#TEST_DIR#g" \
-e "s#$IMGPROTO:$TEST_DIR#TEST_DIR#g" \
-e "s#$TEST_DIR#TEST_DIR#g" \
-e "s#$IMGFMT#IMGFMT#g" \
@@ -154,7 +154,7 @@ _filter_img_info()
discard=0
regex_json_spec_start='^ *"format-specific": \{'
- sed -e "s#$REMOTE_TEST_DIR#TEST_DIR#g" \
+ $SED -e "s#$REMOTE_TEST_DIR#TEST_DIR#g" \
-e "s#$IMGPROTO:$TEST_DIR#TEST_DIR#g" \
-e "s#$TEST_DIR#TEST_DIR#g" \
-e "s#$IMGFMT#IMGFMT#g" \
@@ -201,7 +201,7 @@ _filter_img_info()
# human and json output
_filter_qemu_img_map()
{
- sed -e 's/\([0-9a-fx]* *[0-9a-fx]* *\)[0-9a-fx]* */\1/g' \
+ $SED -e 's/\([0-9a-fx]* *[0-9a-fx]* *\)[0-9a-fx]* */\1/g' \
-e 's/"offset": [0-9]\+/"offset": OFFSET/g' \
-e 's/Mapped to *//' | _filter_testdir | _filter_imgfmt
}
@@ -213,7 +213,7 @@ _filter_nbd()
# receive callbacks sometimes, making them unreliable.
#
# Filter out the TCP port number since this changes between runs.
- sed -e '/nbd\/.*\.c:/d' \
+ $SED -e '/nbd\/.*\.c:/d' \
-e 's#127\.0\.0\.1:[0-9]*#127.0.0.1:PORT#g' \
-e "s#?socket=$TEST_DIR#?socket=TEST_DIR#g" \
-e 's#\(foo\|PORT/\?\|.sock\): Failed to .*$#\1#'
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index f21020eba6..a543e546c2 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -17,6 +17,19 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
+SED=
+for sed in sed gsed; do
+ ($sed --version | grep 'GNU sed') > /dev/null 2>&1
+ if [ "$?" -eq 0 ]; then
+ SED=$sed
+ break
+ fi
+done
+if [ -z "$SED" ]; then
+ echo "$0: GNU sed not found"
+ exit 1
+fi
+
dd()
{
if [ "$HOSTOS" == "Linux" ]
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 11/33] qemu-iotests: Test qcow2 preallocation modes
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (9 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 10/33] qemu-iotests: Ensure GNU sed is used Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 12/33] qcow2: Simplify preallocation code Kevin Wolf
` (22 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/243 | 63 ++++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/243.out | 26 ++++++++++++++++
tests/qemu-iotests/group | 1 +
3 files changed, 90 insertions(+)
create mode 100755 tests/qemu-iotests/243
create mode 100644 tests/qemu-iotests/243.out
diff --git a/tests/qemu-iotests/243 b/tests/qemu-iotests/243
new file mode 100755
index 0000000000..6a19919add
--- /dev/null
+++ b/tests/qemu-iotests/243
@@ -0,0 +1,63 @@
+#!/bin/bash
+#
+# Test qcow2 preallocation
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=kwolf@redhat.com
+
+seq=$(basename $0)
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+for mode in off metadata falloc full; do
+
+ echo
+ echo "=== preallocation=$mode ==="
+ echo
+
+ IMGOPTS="preallocation=$mode" _make_test_img 64M
+
+ printf "File size: "
+ du -b $TEST_IMG | cut -f1
+
+ # Can't use precise numbers here because they differ between filesystems
+ printf "Disk usage: "
+ [ $(du -B1 $TEST_IMG | cut -f1) -lt 1048576 ] && echo "low" || echo "high"
+
+done
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/243.out b/tests/qemu-iotests/243.out
new file mode 100644
index 0000000000..e531753ad1
--- /dev/null
+++ b/tests/qemu-iotests/243.out
@@ -0,0 +1,26 @@
+QA output created by 243
+
+=== preallocation=off ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=off
+File size: 196616
+Disk usage: low
+
+=== preallocation=metadata ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=metadata
+File size: 67436544
+Disk usage: low
+
+=== preallocation=falloc ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=falloc
+File size: 67436544
+Disk usage: high
+
+=== preallocation=full ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=full
+File size: 67436544
+Disk usage: high
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index b5ca63cf72..03aa93dbf5 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -241,3 +241,4 @@
239 rw auto quick
240 auto quick
242 rw auto quick
+243 rw auto quick
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 12/33] qcow2: Simplify preallocation code
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (10 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 11/33] qemu-iotests: Test qcow2 preallocation modes Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 13/33] qcow2: Extend spec for external data files Kevin Wolf
` (21 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Image creation already involves a bdrv_co_truncate() call, which allows
to specify a preallocation mode. Just pass the right mode there and
remove the code that is made redundant by this.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
block/qcow2.c | 28 +---------------------------
1 file changed, 1 insertion(+), 27 deletions(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index 48d22f48d5..9489d795e5 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -2952,19 +2952,6 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
goto out;
}
- if (qcow2_opts->preallocation == PREALLOC_MODE_FULL ||
- qcow2_opts->preallocation == PREALLOC_MODE_FALLOC)
- {
- int64_t prealloc_size =
- qcow2_calc_prealloc_size(qcow2_opts->size, cluster_size,
- refcount_order);
-
- ret = blk_truncate(blk, prealloc_size, qcow2_opts->preallocation, errp);
- if (ret < 0) {
- goto out;
- }
- }
-
/* Write the header */
QEMU_BUILD_BUG_ON((1 << MIN_CLUSTER_BITS) < sizeof(*header));
header = g_malloc0(cluster_size);
@@ -3046,7 +3033,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
}
/* Okay, now that we have a valid image, let's give it the right size */
- ret = blk_truncate(blk, qcow2_opts->size, PREALLOC_MODE_OFF, errp);
+ ret = blk_truncate(blk, qcow2_opts->size, qcow2_opts->preallocation, errp);
if (ret < 0) {
error_prepend(errp, "Could not resize image: ");
goto out;
@@ -3078,19 +3065,6 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
}
}
- /* And if we're supposed to preallocate metadata, do that now */
- if (qcow2_opts->preallocation != PREALLOC_MODE_OFF) {
- BDRVQcow2State *s = blk_bs(blk)->opaque;
- qemu_co_mutex_lock(&s->lock);
- ret = preallocate_co(blk_bs(blk), 0, qcow2_opts->size);
- qemu_co_mutex_unlock(&s->lock);
-
- if (ret < 0) {
- error_setg_errno(errp, -ret, "Could not preallocate metadata");
- goto out;
- }
- }
-
blk_unref(blk);
blk = NULL;
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 13/33] qcow2: Extend spec for external data files
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (11 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 12/33] qcow2: Simplify preallocation code Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 14/33] qcow2: Basic definitions " Kevin Wolf
` (20 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
This adds external data file to the qcow2 spec as a new incompatible
feature.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
docs/interop/qcow2.txt | 42 ++++++++++++++++++++++++++++++++++++++----
1 file changed, 38 insertions(+), 4 deletions(-)
diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index fb5cb47245..bfb97cfde3 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -97,7 +97,19 @@ in the description of a field.
be written to (unless for regaining
consistency).
- Bits 2-63: Reserved (set to 0)
+ Bit 2: External data file bit. If this bit is set, an
+ external data file is used. Guest clusters are
+ then stored in the external data file. For such
+ images, clusters in the external data file are
+ not refcounted. The offset field in the
+ Standard Cluster Descriptor must match the
+ guest offset and neither compressed clusters
+ nor internal snapshots are supported.
+
+ An External Data File Name header extension may
+ be present if this bit is set.
+
+ Bits 3-63: Reserved (set to 0)
80 - 87: compatible_features
Bitmask of compatible features. An implementation can
@@ -126,7 +138,21 @@ in the description of a field.
bit is unset, the bitmaps extension data must be
considered inconsistent.
- Bits 1-63: Reserved (set to 0)
+ Bit 1: If this bit is set, the external data file can
+ be read as a consistent standalone raw image
+ without looking at the qcow2 metadata.
+
+ Setting this bit has a performance impact for
+ some operations on the image (e.g. writing
+ zeros requires writing to the data file instead
+ of only setting the zero flag in the L2 table
+ entry) and conflicts with backing files.
+
+ This bit may only be set if the External Data
+ File bit (incompatible feature bit 1) is also
+ set.
+
+ Bits 2-63: Reserved (set to 0)
96 - 99: refcount_order
Describes the width of a reference count block entry (width
@@ -148,6 +174,7 @@ be stored. Each extension has a structure like the following:
0x6803f857 - Feature name table
0x23852875 - Bitmaps extension
0x0537be77 - Full disk encryption header pointer
+ 0x44415441 - External data file name
other - Unknown header extension, can be safely
ignored
@@ -437,6 +464,11 @@ L2 table entry:
This information is only accurate in L2 tables
that are reachable from the active L1 table.
+ With external data files, all guest clusters have an
+ implicit refcount of 1 (because of the fixed host = guest
+ mapping for guest cluster offsets), so this bit should be 1
+ for all allocated clusters.
+
Standard Cluster Descriptor:
Bit 0: If set to 1, the cluster reads as all zeros. The host
@@ -450,8 +482,10 @@ Standard Cluster Descriptor:
1 - 8: Reserved (set to 0)
9 - 55: Bits 9-55 of host cluster offset. Must be aligned to a
- cluster boundary. If the offset is 0, the cluster is
- unallocated.
+ cluster boundary. If the offset is 0 and bit 63 is clear,
+ the cluster is unallocated. The offset may only be 0 with
+ bit 63 set (indicating a host cluster offset of 0) when an
+ external data file is used.
56 - 61: Reserved (set to 0)
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 14/33] qcow2: Basic definitions for external data files
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (12 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 13/33] qcow2: Extend spec for external data files Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 15/33] qcow2: Pass bs to qcow2_get_cluster_type() Kevin Wolf
` (19 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
This adds basic constants, struct fields and helper function for
external data file support to the implementation.
QCOW2_INCOMPAT_MASK and QCOW2_AUTOCLEAR_MASK are not updated yet so that
opening images with an external data file still fails (we don't handle
them correctly yet).
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
block/qcow2.h | 32 ++++++++++++++++++++++----------
block/qcow2.c | 9 +++++++++
tests/qemu-iotests/031.out | 8 ++++----
tests/qemu-iotests/036.out | 4 ++--
tests/qemu-iotests/061.out | 14 +++++++-------
5 files changed, 44 insertions(+), 23 deletions(-)
diff --git a/block/qcow2.h b/block/qcow2.h
index 9dd02df831..c63c3959f7 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -197,13 +197,15 @@ enum {
/* Incompatible feature bits */
enum {
- QCOW2_INCOMPAT_DIRTY_BITNR = 0,
- QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
- QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
- QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
-
- QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
- | QCOW2_INCOMPAT_CORRUPT,
+ QCOW2_INCOMPAT_DIRTY_BITNR = 0,
+ QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
+ QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
+ QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
+ QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
+ QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+
+ QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
+ | QCOW2_INCOMPAT_CORRUPT,
};
/* Compatible feature bits */
@@ -216,10 +218,12 @@ enum {
/* Autoclear feature bits */
enum {
- QCOW2_AUTOCLEAR_BITMAPS_BITNR = 0,
- QCOW2_AUTOCLEAR_BITMAPS = 1 << QCOW2_AUTOCLEAR_BITMAPS_BITNR,
+ QCOW2_AUTOCLEAR_BITMAPS_BITNR = 0,
+ QCOW2_AUTOCLEAR_DATA_FILE_RAW_BITNR = 1,
+ QCOW2_AUTOCLEAR_BITMAPS = 1 << QCOW2_AUTOCLEAR_BITMAPS_BITNR,
+ QCOW2_AUTOCLEAR_DATA_FILE_RAW = 1 << QCOW2_AUTOCLEAR_DATA_FILE_RAW_BITNR,
- QCOW2_AUTOCLEAR_MASK = QCOW2_AUTOCLEAR_BITMAPS,
+ QCOW2_AUTOCLEAR_MASK = QCOW2_AUTOCLEAR_BITMAPS,
};
enum qcow2_discard_type {
@@ -340,6 +344,8 @@ typedef struct BDRVQcow2State {
CoQueue compress_wait_queue;
int nb_compress_threads;
+
+ BdrvChild *data_file;
} BDRVQcow2State;
typedef struct Qcow2COWRegion {
@@ -457,6 +463,12 @@ typedef enum QCow2MetadataOverlap {
#define REFT_OFFSET_MASK 0xfffffffffffffe00ULL
+static inline bool has_data_file(BlockDriverState *bs)
+{
+ BDRVQcow2State *s = bs->opaque;
+ return (s->data_file != bs->file);
+}
+
static inline int64_t start_of_cluster(BDRVQcow2State *s, int64_t offset)
{
return offset & ~(s->cluster_size - 1);
diff --git a/block/qcow2.c b/block/qcow2.c
index 9489d795e5..59cf706dc2 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -73,6 +73,7 @@ typedef struct {
#define QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857
#define QCOW2_EXT_MAGIC_CRYPTO_HEADER 0x0537be77
#define QCOW2_EXT_MAGIC_BITMAPS 0x23852875
+#define QCOW2_EXT_MAGIC_DATA_FILE 0x44415441
static int coroutine_fn
qcow2_co_preadv_compressed(BlockDriverState *bs,
@@ -1452,6 +1453,9 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
goto fail;
}
+ /* TODO Open external data file */
+ s->data_file = bs->file;
+
/* qcow2_read_extension may have set up the crypto context
* if the crypt method needs a header region, some methods
* don't need header extensions, so must check here
@@ -2440,6 +2444,11 @@ int qcow2_update_header(BlockDriverState *bs)
.bit = QCOW2_INCOMPAT_CORRUPT_BITNR,
.name = "corrupt bit",
},
+ {
+ .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
+ .bit = QCOW2_INCOMPAT_DATA_FILE_BITNR,
+ .name = "external data file",
+ },
{
.type = QCOW2_FEAT_TYPE_COMPATIBLE,
.bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
diff --git a/tests/qemu-iotests/031.out b/tests/qemu-iotests/031.out
index 7f5050b816..68a74d03b9 100644
--- a/tests/qemu-iotests/031.out
+++ b/tests/qemu-iotests/031.out
@@ -117,7 +117,7 @@ header_length 104
Header extension:
magic 0x6803f857
-length 144
+length 192
data <binary>
Header extension:
@@ -150,7 +150,7 @@ header_length 104
Header extension:
magic 0x6803f857
-length 144
+length 192
data <binary>
Header extension:
@@ -164,7 +164,7 @@ No errors were found on the image.
magic 0x514649fb
version 3
-backing_file_offset 0x148
+backing_file_offset 0x178
backing_file_size 0x17
cluster_bits 16
size 67108864
@@ -188,7 +188,7 @@ data 'host_device'
Header extension:
magic 0x6803f857
-length 144
+length 192
data <binary>
Header extension:
diff --git a/tests/qemu-iotests/036.out b/tests/qemu-iotests/036.out
index 9b009b8c15..e489b44386 100644
--- a/tests/qemu-iotests/036.out
+++ b/tests/qemu-iotests/036.out
@@ -58,7 +58,7 @@ header_length 104
Header extension:
magic 0x6803f857
-length 144
+length 192
data <binary>
@@ -86,7 +86,7 @@ header_length 104
Header extension:
magic 0x6803f857
-length 144
+length 192
data <binary>
*** done
diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
index 183f7dd690..758284011b 100644
--- a/tests/qemu-iotests/061.out
+++ b/tests/qemu-iotests/061.out
@@ -26,7 +26,7 @@ header_length 104
Header extension:
magic 0x6803f857
-length 144
+length 192
data <binary>
magic 0x514649fb
@@ -84,7 +84,7 @@ header_length 104
Header extension:
magic 0x6803f857
-length 144
+length 192
data <binary>
magic 0x514649fb
@@ -144,7 +144,7 @@ header_length 104
Header extension:
magic 0x6803f857
-length 144
+length 192
data <binary>
ERROR cluster 5 refcount=0 reference=1
@@ -199,7 +199,7 @@ header_length 104
Header extension:
magic 0x6803f857
-length 144
+length 192
data <binary>
magic 0x514649fb
@@ -268,7 +268,7 @@ header_length 104
Header extension:
magic 0x6803f857
-length 144
+length 192
data <binary>
read 65536/65536 bytes at offset 44040192
@@ -306,7 +306,7 @@ header_length 104
Header extension:
magic 0x6803f857
-length 144
+length 192
data <binary>
ERROR cluster 5 refcount=0 reference=1
@@ -335,7 +335,7 @@ header_length 104
Header extension:
magic 0x6803f857
-length 144
+length 192
data <binary>
read 131072/131072 bytes at offset 0
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 15/33] qcow2: Pass bs to qcow2_get_cluster_type()
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (13 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 14/33] qcow2: Basic definitions " Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 16/33] qcow2: Prepare qcow2_get_cluster_type() for external data file Kevin Wolf
` (18 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
block/qcow2.h | 3 ++-
block/qcow2-cluster.c | 37 +++++++++++++++++++------------------
block/qcow2-refcount.c | 10 +++++-----
3 files changed, 26 insertions(+), 24 deletions(-)
diff --git a/block/qcow2.h b/block/qcow2.h
index c63c3959f7..7a34bd0c53 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -510,7 +510,8 @@ static inline int64_t qcow2_vm_state_offset(BDRVQcow2State *s)
return (int64_t)s->l1_vm_state_index << (s->cluster_bits + s->l2_bits);
}
-static inline QCow2ClusterType qcow2_get_cluster_type(uint64_t l2_entry)
+static inline QCow2ClusterType qcow2_get_cluster_type(BlockDriverState *bs,
+ uint64_t l2_entry)
{
if (l2_entry & QCOW_OFLAG_COMPRESSED) {
return QCOW2_CLUSTER_COMPRESSED;
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 179aa2c728..9cc8f0f3e4 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -380,8 +380,8 @@ fail:
* as contiguous. (This allows it, for example, to stop at the first compressed
* cluster which may require a different handling)
*/
-static int count_contiguous_clusters(int nb_clusters, int cluster_size,
- uint64_t *l2_slice, uint64_t stop_flags)
+static int count_contiguous_clusters(BlockDriverState *bs, int nb_clusters,
+ int cluster_size, uint64_t *l2_slice, uint64_t stop_flags)
{
int i;
QCow2ClusterType first_cluster_type;
@@ -394,7 +394,7 @@ static int count_contiguous_clusters(int nb_clusters, int cluster_size,
}
/* must be allocated */
- first_cluster_type = qcow2_get_cluster_type(first_entry);
+ first_cluster_type = qcow2_get_cluster_type(bs, first_entry);
assert(first_cluster_type == QCOW2_CLUSTER_NORMAL ||
first_cluster_type == QCOW2_CLUSTER_ZERO_ALLOC);
@@ -412,7 +412,8 @@ static int count_contiguous_clusters(int nb_clusters, int cluster_size,
* Checks how many consecutive unallocated clusters in a given L2
* slice have the same cluster type.
*/
-static int count_contiguous_clusters_unallocated(int nb_clusters,
+static int count_contiguous_clusters_unallocated(BlockDriverState *bs,
+ int nb_clusters,
uint64_t *l2_slice,
QCow2ClusterType wanted_type)
{
@@ -422,7 +423,7 @@ static int count_contiguous_clusters_unallocated(int nb_clusters,
wanted_type == QCOW2_CLUSTER_UNALLOCATED);
for (i = 0; i < nb_clusters; i++) {
uint64_t entry = be64_to_cpu(l2_slice[i]);
- QCow2ClusterType type = qcow2_get_cluster_type(entry);
+ QCow2ClusterType type = qcow2_get_cluster_type(bs, entry);
if (type != wanted_type) {
break;
@@ -595,7 +596,7 @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset,
* true */
assert(nb_clusters <= INT_MAX);
- type = qcow2_get_cluster_type(*cluster_offset);
+ type = qcow2_get_cluster_type(bs, *cluster_offset);
if (s->qcow_version < 3 && (type == QCOW2_CLUSTER_ZERO_PLAIN ||
type == QCOW2_CLUSTER_ZERO_ALLOC)) {
qcow2_signal_corruption(bs, true, -1, -1, "Zero cluster entry found"
@@ -613,14 +614,14 @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset,
case QCOW2_CLUSTER_ZERO_PLAIN:
case QCOW2_CLUSTER_UNALLOCATED:
/* how many empty clusters ? */
- c = count_contiguous_clusters_unallocated(nb_clusters,
+ c = count_contiguous_clusters_unallocated(bs, nb_clusters,
&l2_slice[l2_index], type);
*cluster_offset = 0;
break;
case QCOW2_CLUSTER_ZERO_ALLOC:
case QCOW2_CLUSTER_NORMAL:
/* how many allocated clusters ? */
- c = count_contiguous_clusters(nb_clusters, s->cluster_size,
+ c = count_contiguous_clusters(bs, nb_clusters, s->cluster_size,
&l2_slice[l2_index], QCOW_OFLAG_ZERO);
*cluster_offset &= L2E_OFFSET_MASK;
if (offset_into_cluster(s, *cluster_offset)) {
@@ -1013,14 +1014,14 @@ void qcow2_alloc_cluster_abort(BlockDriverState *bs, QCowL2Meta *m)
* write, but require COW to be performed (this includes yet unallocated space,
* which must copy from the backing file)
*/
-static int count_cow_clusters(BDRVQcow2State *s, int nb_clusters,
+static int count_cow_clusters(BlockDriverState *bs, int nb_clusters,
uint64_t *l2_slice, int l2_index)
{
int i;
for (i = 0; i < nb_clusters; i++) {
uint64_t l2_entry = be64_to_cpu(l2_slice[l2_index + i]);
- QCow2ClusterType cluster_type = qcow2_get_cluster_type(l2_entry);
+ QCow2ClusterType cluster_type = qcow2_get_cluster_type(bs, l2_entry);
switch(cluster_type) {
case QCOW2_CLUSTER_NORMAL:
@@ -1165,7 +1166,7 @@ static int handle_copied(BlockDriverState *bs, uint64_t guest_offset,
cluster_offset = be64_to_cpu(l2_slice[l2_index]);
/* Check how many clusters are already allocated and don't need COW */
- if (qcow2_get_cluster_type(cluster_offset) == QCOW2_CLUSTER_NORMAL
+ if (qcow2_get_cluster_type(bs, cluster_offset) == QCOW2_CLUSTER_NORMAL
&& (cluster_offset & QCOW_OFLAG_COPIED))
{
/* If a specific host_offset is required, check it */
@@ -1189,7 +1190,7 @@ static int handle_copied(BlockDriverState *bs, uint64_t guest_offset,
/* We keep all QCOW_OFLAG_COPIED clusters */
keep_clusters =
- count_contiguous_clusters(nb_clusters, s->cluster_size,
+ count_contiguous_clusters(bs, nb_clusters, s->cluster_size,
&l2_slice[l2_index],
QCOW_OFLAG_COPIED | QCOW_OFLAG_ZERO);
assert(keep_clusters <= nb_clusters);
@@ -1324,7 +1325,7 @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset,
if (entry & QCOW_OFLAG_COMPRESSED) {
nb_clusters = 1;
} else {
- nb_clusters = count_cow_clusters(s, nb_clusters, l2_slice, l2_index);
+ nb_clusters = count_cow_clusters(bs, nb_clusters, l2_slice, l2_index);
}
/* This function is only called when there were no non-COW clusters, so if
@@ -1332,7 +1333,7 @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset,
* wrong with our code. */
assert(nb_clusters > 0);
- if (qcow2_get_cluster_type(entry) == QCOW2_CLUSTER_ZERO_ALLOC &&
+ if (qcow2_get_cluster_type(bs, entry) == QCOW2_CLUSTER_ZERO_ALLOC &&
(entry & QCOW_OFLAG_COPIED) &&
(!*host_offset ||
start_of_cluster(s, *host_offset) == (entry & L2E_OFFSET_MASK)))
@@ -1352,7 +1353,7 @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset,
* would be fine, too, but count_cow_clusters() above has limited
* nb_clusters already to a range of COW clusters */
preallocated_nb_clusters =
- count_contiguous_clusters(nb_clusters, s->cluster_size,
+ count_contiguous_clusters(bs, nb_clusters, s->cluster_size,
&l2_slice[l2_index], QCOW_OFLAG_COPIED);
assert(preallocated_nb_clusters > 0);
@@ -1616,7 +1617,7 @@ static int discard_in_l2_slice(BlockDriverState *bs, uint64_t offset,
* If full_discard is true, the sector should not read back as zeroes,
* but rather fall through to the backing file.
*/
- switch (qcow2_get_cluster_type(old_l2_entry)) {
+ switch (qcow2_get_cluster_type(bs, old_l2_entry)) {
case QCOW2_CLUSTER_UNALLOCATED:
if (full_discard || !bs->backing) {
continue;
@@ -1729,7 +1730,7 @@ static int zero_in_l2_slice(BlockDriverState *bs, uint64_t offset,
* Minimize L2 changes if the cluster already reads back as
* zeroes with correct allocation.
*/
- cluster_type = qcow2_get_cluster_type(old_offset);
+ cluster_type = qcow2_get_cluster_type(bs, old_offset);
if (cluster_type == QCOW2_CLUSTER_ZERO_PLAIN ||
(cluster_type == QCOW2_CLUSTER_ZERO_ALLOC && !unmap)) {
continue;
@@ -1871,7 +1872,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,
uint64_t l2_entry = be64_to_cpu(l2_slice[j]);
int64_t offset = l2_entry & L2E_OFFSET_MASK;
QCow2ClusterType cluster_type =
- qcow2_get_cluster_type(l2_entry);
+ qcow2_get_cluster_type(bs, l2_entry);
if (cluster_type != QCOW2_CLUSTER_ZERO_PLAIN &&
cluster_type != QCOW2_CLUSTER_ZERO_ALLOC) {
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index 6f13d470d3..05e7974d7e 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -1157,7 +1157,7 @@ void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry,
{
BDRVQcow2State *s = bs->opaque;
- switch (qcow2_get_cluster_type(l2_entry)) {
+ switch (qcow2_get_cluster_type(bs, l2_entry)) {
case QCOW2_CLUSTER_COMPRESSED:
{
int nb_csectors;
@@ -1300,7 +1300,7 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
entry &= ~QCOW_OFLAG_COPIED;
offset = entry & L2E_OFFSET_MASK;
- switch (qcow2_get_cluster_type(entry)) {
+ switch (qcow2_get_cluster_type(bs, entry)) {
case QCOW2_CLUSTER_COMPRESSED:
nb_csectors = ((entry >> s->csize_shift) &
s->csize_mask) + 1;
@@ -1582,7 +1582,7 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res,
for(i = 0; i < s->l2_size; i++) {
l2_entry = be64_to_cpu(l2_table[i]);
- switch (qcow2_get_cluster_type(l2_entry)) {
+ switch (qcow2_get_cluster_type(bs, l2_entry)) {
case QCOW2_CLUSTER_COMPRESSED:
/* Compressed clusters don't have QCOW_OFLAG_COPIED */
if (l2_entry & QCOW_OFLAG_COPIED) {
@@ -1633,7 +1633,7 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res,
/* Correct offsets are cluster aligned */
if (offset_into_cluster(s, offset)) {
- if (qcow2_get_cluster_type(l2_entry) ==
+ if (qcow2_get_cluster_type(bs, l2_entry) ==
QCOW2_CLUSTER_ZERO_ALLOC)
{
fprintf(stderr, "%s offset=%" PRIx64 ": Preallocated zero "
@@ -1868,7 +1868,7 @@ static int check_oflag_copied(BlockDriverState *bs, BdrvCheckResult *res,
for (j = 0; j < s->l2_size; j++) {
uint64_t l2_entry = be64_to_cpu(l2_table[j]);
uint64_t data_offset = l2_entry & L2E_OFFSET_MASK;
- QCow2ClusterType cluster_type = qcow2_get_cluster_type(l2_entry);
+ QCow2ClusterType cluster_type = qcow2_get_cluster_type(bs, l2_entry);
if (cluster_type == QCOW2_CLUSTER_NORMAL ||
cluster_type == QCOW2_CLUSTER_ZERO_ALLOC) {
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 16/33] qcow2: Prepare qcow2_get_cluster_type() for external data file
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (14 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 15/33] qcow2: Pass bs to qcow2_get_cluster_type() Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 17/33] qcow2: Prepare count_contiguous_clusters() " Kevin Wolf
` (17 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
block/qcow2.h | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/block/qcow2.h b/block/qcow2.h
index 7a34bd0c53..8fe2d55005 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -521,7 +521,15 @@ static inline QCow2ClusterType qcow2_get_cluster_type(BlockDriverState *bs,
}
return QCOW2_CLUSTER_ZERO_PLAIN;
} else if (!(l2_entry & L2E_OFFSET_MASK)) {
- return QCOW2_CLUSTER_UNALLOCATED;
+ /* Offset 0 generally means unallocated, but it is ambiguous with
+ * external data files because 0 is a valid offset there. However, all
+ * clusters in external data files always have refcount 1, so we can
+ * rely on QCOW_OFLAG_COPIED to disambiguate. */
+ if (has_data_file(bs) && (l2_entry & QCOW_OFLAG_COPIED)) {
+ return QCOW2_CLUSTER_NORMAL;
+ } else {
+ return QCOW2_CLUSTER_UNALLOCATED;
+ }
} else {
return QCOW2_CLUSTER_NORMAL;
}
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 17/33] qcow2: Prepare count_contiguous_clusters() for external data file
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (15 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 16/33] qcow2: Prepare qcow2_get_cluster_type() for external data file Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 18/33] qcow2: Don't assume 0 is an invalid cluster offset Kevin Wolf
` (16 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Offset 0 can be valid for normal (allocated) clusters now, so use
qcow2_get_cluster_type() instead.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
block/qcow2-cluster.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 9cc8f0f3e4..660161bf00 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -389,12 +389,12 @@ static int count_contiguous_clusters(BlockDriverState *bs, int nb_clusters,
uint64_t first_entry = be64_to_cpu(l2_slice[0]);
uint64_t offset = first_entry & mask;
- if (!offset) {
+ first_cluster_type = qcow2_get_cluster_type(bs, first_entry);
+ if (first_cluster_type == QCOW2_CLUSTER_UNALLOCATED) {
return 0;
}
/* must be allocated */
- first_cluster_type = qcow2_get_cluster_type(bs, first_entry);
assert(first_cluster_type == QCOW2_CLUSTER_NORMAL ||
first_cluster_type == QCOW2_CLUSTER_ZERO_ALLOC);
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 18/33] qcow2: Don't assume 0 is an invalid cluster offset
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (16 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 17/33] qcow2: Prepare count_contiguous_clusters() " Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 19/33] qcow2: Return 0/-errno in qcow2_alloc_compressed_cluster_offset() Kevin Wolf
` (15 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
The cluster allocation code uses 0 as an invalid offset that is used in
case of errors or as "offset not yet determined". With external data
files, a host cluster offset of 0 becomes valid, though.
Define a constant INV_OFFSET (which is not cluster aligned and will
therefore never be a valid offset) that can be used for such purposes.
This removes the additional host_offset == 0 check that commit
ff52aab2df5 introduced; the confusion between an invalid offset and
(erroneous) allocation at offset 0 is removed with this change.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
block/qcow2.h | 2 ++
block/qcow2-cluster.c | 59 ++++++++++++++++++++-----------------------
2 files changed, 29 insertions(+), 32 deletions(-)
diff --git a/block/qcow2.h b/block/qcow2.h
index 8fe2d55005..e3bf322be1 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -463,6 +463,8 @@ typedef enum QCow2MetadataOverlap {
#define REFT_OFFSET_MASK 0xfffffffffffffe00ULL
+#define INV_OFFSET (-1ULL)
+
static inline bool has_data_file(BlockDriverState *bs)
{
BDRVQcow2State *s = bs->opaque;
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 660161bf00..1d09f5454e 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -1109,9 +1109,9 @@ static int handle_dependencies(BlockDriverState *bs, uint64_t guest_offset,
/*
* Checks how many already allocated clusters that don't require a copy on
- * write there are at the given guest_offset (up to *bytes). If
- * *host_offset is not zero, only physically contiguous clusters beginning at
- * this host offset are counted.
+ * write there are at the given guest_offset (up to *bytes). If *host_offset is
+ * not INV_OFFSET, only physically contiguous clusters beginning at this host
+ * offset are counted.
*
* Note that guest_offset may not be cluster aligned. In this case, the
* returned *host_offset points to exact byte referenced by guest_offset and
@@ -1143,8 +1143,8 @@ static int handle_copied(BlockDriverState *bs, uint64_t guest_offset,
trace_qcow2_handle_copied(qemu_coroutine_self(), guest_offset, *host_offset,
*bytes);
- assert(*host_offset == 0 || offset_into_cluster(s, guest_offset)
- == offset_into_cluster(s, *host_offset));
+ assert(*host_offset == INV_OFFSET || offset_into_cluster(s, guest_offset)
+ == offset_into_cluster(s, *host_offset));
/*
* Calculate the number of clusters to look for. We stop at L2 slice
@@ -1182,7 +1182,7 @@ static int handle_copied(BlockDriverState *bs, uint64_t guest_offset,
goto out;
}
- if (*host_offset != 0 && !offset_matches) {
+ if (*host_offset != INV_OFFSET && !offset_matches) {
*bytes = 0;
ret = 0;
goto out;
@@ -1225,10 +1225,10 @@ out:
* contain the number of clusters that have been allocated and are contiguous
* in the image file.
*
- * If *host_offset is non-zero, it specifies the offset in the image file at
- * which the new clusters must start. *nb_clusters can be 0 on return in this
- * case if the cluster at host_offset is already in use. If *host_offset is
- * zero, the clusters can be allocated anywhere in the image file.
+ * If *host_offset is not INV_OFFSET, it specifies the offset in the image file
+ * at which the new clusters must start. *nb_clusters can be 0 on return in
+ * this case if the cluster at host_offset is already in use. If *host_offset
+ * is INV_OFFSET, the clusters can be allocated anywhere in the image file.
*
* *host_offset is updated to contain the offset into the image file at which
* the first allocated cluster starts.
@@ -1247,7 +1247,7 @@ static int do_alloc_cluster_offset(BlockDriverState *bs, uint64_t guest_offset,
/* Allocate new clusters */
trace_qcow2_cluster_alloc_phys(qemu_coroutine_self());
- if (*host_offset == 0) {
+ if (*host_offset == INV_OFFSET) {
int64_t cluster_offset =
qcow2_alloc_clusters(bs, *nb_clusters * s->cluster_size);
if (cluster_offset < 0) {
@@ -1267,8 +1267,8 @@ static int do_alloc_cluster_offset(BlockDriverState *bs, uint64_t guest_offset,
/*
* Allocates new clusters for an area that either is yet unallocated or needs a
- * copy on write. If *host_offset is non-zero, clusters are only allocated if
- * the new allocation can match the specified host offset.
+ * copy on write. If *host_offset is not INV_OFFSET, clusters are only
+ * allocated if the new allocation can match the specified host offset.
*
* Note that guest_offset may not be cluster aligned. In this case, the
* returned *host_offset points to exact byte referenced by guest_offset and
@@ -1296,7 +1296,7 @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset,
int ret;
bool keep_old_clusters = false;
- uint64_t alloc_cluster_offset = 0;
+ uint64_t alloc_cluster_offset = INV_OFFSET;
trace_qcow2_handle_alloc(qemu_coroutine_self(), guest_offset, *host_offset,
*bytes);
@@ -1335,7 +1335,7 @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset,
if (qcow2_get_cluster_type(bs, entry) == QCOW2_CLUSTER_ZERO_ALLOC &&
(entry & QCOW_OFLAG_COPIED) &&
- (!*host_offset ||
+ (*host_offset == INV_OFFSET ||
start_of_cluster(s, *host_offset) == (entry & L2E_OFFSET_MASK)))
{
int preallocated_nb_clusters;
@@ -1367,9 +1367,10 @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset,
qcow2_cache_put(s->l2_table_cache, (void **) &l2_slice);
- if (!alloc_cluster_offset) {
+ if (alloc_cluster_offset == INV_OFFSET) {
/* Allocate, if necessary at a given offset in the image file */
- alloc_cluster_offset = start_of_cluster(s, *host_offset);
+ alloc_cluster_offset = *host_offset == INV_OFFSET ? INV_OFFSET :
+ start_of_cluster(s, *host_offset);
ret = do_alloc_cluster_offset(bs, guest_offset, &alloc_cluster_offset,
&nb_clusters);
if (ret < 0) {
@@ -1382,16 +1383,7 @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset,
return 0;
}
- /* !*host_offset would overwrite the image header and is reserved for
- * "no host offset preferred". If 0 was a valid host offset, it'd
- * trigger the following overlap check; do that now to avoid having an
- * invalid value in *host_offset. */
- if (!alloc_cluster_offset) {
- ret = qcow2_pre_write_overlap_check(bs, 0, alloc_cluster_offset,
- nb_clusters * s->cluster_size);
- assert(ret < 0);
- goto fail;
- }
+ assert(alloc_cluster_offset != INV_OFFSET);
}
/*
@@ -1483,14 +1475,14 @@ int qcow2_alloc_cluster_offset(BlockDriverState *bs, uint64_t offset,
again:
start = offset;
remaining = *bytes;
- cluster_offset = 0;
- *host_offset = 0;
+ cluster_offset = INV_OFFSET;
+ *host_offset = INV_OFFSET;
cur_bytes = 0;
*m = NULL;
while (true) {
- if (!*host_offset) {
+ if (*host_offset == INV_OFFSET && cluster_offset != INV_OFFSET) {
*host_offset = start_of_cluster(s, cluster_offset);
}
@@ -1498,7 +1490,10 @@ again:
start += cur_bytes;
remaining -= cur_bytes;
- cluster_offset += cur_bytes;
+
+ if (cluster_offset != INV_OFFSET) {
+ cluster_offset += cur_bytes;
+ }
if (remaining == 0) {
break;
@@ -1570,7 +1565,7 @@ again:
*bytes -= remaining;
assert(*bytes > 0);
- assert(*host_offset != 0);
+ assert(*host_offset != INV_OFFSET);
return 0;
}
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 19/33] qcow2: Return 0/-errno in qcow2_alloc_compressed_cluster_offset()
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (17 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 18/33] qcow2: Don't assume 0 is an invalid cluster offset Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 20/33] qcow2: Prepare qcow2_co_block_status() for data file Kevin Wolf
` (14 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
qcow2_alloc_compressed_cluster_offset() used to return the cluster
offset for success and 0 for error. This doesn't only conflict with 0 as
a valid host offset, but also loses the error code.
Similar to the change made to qcow2_alloc_cluster_offset() for
uncompressed clusters in commit 148da7ea9d6, make the function return
0/-errno and return the allocated cluster offset in a by-reference
parameter.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
block/qcow2.h | 7 ++++---
block/qcow2-cluster.c | 28 +++++++++++++---------------
block/qcow2.c | 19 ++++++++-----------
tests/qemu-iotests/220.out | 2 +-
4 files changed, 26 insertions(+), 30 deletions(-)
diff --git a/block/qcow2.h b/block/qcow2.h
index e3bf322be1..fad6abf602 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -647,9 +647,10 @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset,
int qcow2_alloc_cluster_offset(BlockDriverState *bs, uint64_t offset,
unsigned int *bytes, uint64_t *host_offset,
QCowL2Meta **m);
-uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
- uint64_t offset,
- int compressed_size);
+int qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
+ uint64_t offset,
+ int compressed_size,
+ uint64_t *host_offset);
int qcow2_alloc_cluster_link_l2(BlockDriverState *bs, QCowL2Meta *m);
void qcow2_alloc_cluster_abort(BlockDriverState *bs, QCowL2Meta *m);
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 1d09f5454e..8c4b4005ff 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -736,19 +736,16 @@ static int get_cluster_table(BlockDriverState *bs, uint64_t offset,
/*
* alloc_compressed_cluster_offset
*
- * For a given offset of the disk image, return cluster offset in
- * qcow2 file.
- *
- * If the offset is not found, allocate a new compressed cluster.
- *
- * Return the cluster offset if successful,
- * Return 0, otherwise.
+ * For a given offset on the virtual disk, allocate a new compressed cluster
+ * and put the host offset of the cluster into *host_offset. If a cluster is
+ * already allocated at the offset, return an error.
*
+ * Return 0 on success and -errno in error cases
*/
-
-uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
- uint64_t offset,
- int compressed_size)
+int qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
+ uint64_t offset,
+ int compressed_size,
+ uint64_t *host_offset)
{
BDRVQcow2State *s = bs->opaque;
int l2_index, ret;
@@ -758,7 +755,7 @@ uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
ret = get_cluster_table(bs, offset, &l2_slice, &l2_index);
if (ret < 0) {
- return 0;
+ return ret;
}
/* Compression can't overwrite anything. Fail if the cluster was already
@@ -766,13 +763,13 @@ uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
cluster_offset = be64_to_cpu(l2_slice[l2_index]);
if (cluster_offset & L2E_OFFSET_MASK) {
qcow2_cache_put(s->l2_table_cache, (void **) &l2_slice);
- return 0;
+ return -EIO;
}
cluster_offset = qcow2_alloc_bytes(bs, compressed_size);
if (cluster_offset < 0) {
qcow2_cache_put(s->l2_table_cache, (void **) &l2_slice);
- return 0;
+ return cluster_offset;
}
nb_csectors = ((cluster_offset + compressed_size - 1) >> 9) -
@@ -790,7 +787,8 @@ uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
l2_slice[l2_index] = cpu_to_be64(cluster_offset);
qcow2_cache_put(s->l2_table_cache, (void **) &l2_slice);
- return cluster_offset;
+ *host_offset = cluster_offset & s->cluster_offset_mask;
+ return 0;
}
static int perform_cow(BlockDriverState *bs, QCowL2Meta *m)
diff --git a/block/qcow2.c b/block/qcow2.c
index 59cf706dc2..eaccd1c11a 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -3896,17 +3896,16 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uint64_t offset,
int ret;
size_t out_len;
uint8_t *buf, *out_buf;
- int64_t cluster_offset;
+ uint64_t cluster_offset;
if (bytes == 0) {
/* align end of file to a sector boundary to ease reading with
sector based I/Os */
- cluster_offset = bdrv_getlength(bs->file->bs);
- if (cluster_offset < 0) {
- return cluster_offset;
+ int64_t len = bdrv_getlength(bs->file->bs);
+ if (len < 0) {
+ return len;
}
- return bdrv_co_truncate(bs->file, cluster_offset, PREALLOC_MODE_OFF,
- NULL);
+ return bdrv_co_truncate(bs->file, len, PREALLOC_MODE_OFF, NULL);
}
if (offset_into_cluster(s, offset)) {
@@ -3943,14 +3942,12 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uint64_t offset,
}
qemu_co_mutex_lock(&s->lock);
- cluster_offset =
- qcow2_alloc_compressed_cluster_offset(bs, offset, out_len);
- if (!cluster_offset) {
+ ret = qcow2_alloc_compressed_cluster_offset(bs, offset, out_len,
+ &cluster_offset);
+ if (ret < 0) {
qemu_co_mutex_unlock(&s->lock);
- ret = -EIO;
goto fail;
}
- cluster_offset &= s->cluster_offset_mask;
ret = qcow2_pre_write_overlap_check(bs, 0, cluster_offset, out_len);
qemu_co_mutex_unlock(&s->lock);
diff --git a/tests/qemu-iotests/220.out b/tests/qemu-iotests/220.out
index af3021fd88..33b994b8a1 100644
--- a/tests/qemu-iotests/220.out
+++ b/tests/qemu-iotests/220.out
@@ -38,7 +38,7 @@ wrote 2097152/2097152 bytes at offset 37748736
No errors were found on the image.
image size 39845888
== Trying to write compressed cluster ==
-write failed: Input/output error
+write failed: File too large
image size 562949957615616
== Writing normal cluster ==
wrote 2097152/2097152 bytes at offset 0
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 20/33] qcow2: Prepare qcow2_co_block_status() for data file
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (18 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 19/33] qcow2: Return 0/-errno in qcow2_alloc_compressed_cluster_offset() Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 21/33] qcow2: External file I/O Kevin Wolf
` (13 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Offset 0 cannot be assumed to mean an unallocated cluster any more.
Instead, the cluster type needs to be checked.
*file must refer to the data file instead of the image file if a valid
offset is returned from qcow2_co_block_status().
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
block/qcow2.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index eaccd1c11a..8d1f667e91 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1829,11 +1829,11 @@ static int coroutine_fn qcow2_co_block_status(BlockDriverState *bs,
*pnum = bytes;
- if (cluster_offset != 0 && ret != QCOW2_CLUSTER_COMPRESSED &&
+ if ((ret == QCOW2_CLUSTER_NORMAL || ret == QCOW2_CLUSTER_ZERO_ALLOC) &&
!s->crypto) {
index_in_cluster = offset & (s->cluster_size - 1);
*map = cluster_offset | index_in_cluster;
- *file = bs->file->bs;
+ *file = s->data_file->bs;
status |= BDRV_BLOCK_OFFSET_VALID;
}
if (ret == QCOW2_CLUSTER_ZERO_PLAIN || ret == QCOW2_CLUSTER_ZERO_ALLOC) {
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 21/33] qcow2: External file I/O
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (19 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 20/33] qcow2: Prepare qcow2_co_block_status() for data file Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 22/33] qcow2: Return error for snapshot operation with data file Kevin Wolf
` (12 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
This changes the qcow2 implementation to direct all guest data I/O to
s->data_file rather than bs->file, while metadata I/O still uses
bs->file. At the moment, this is still always the same, but soon we'll
add options to set s->data_file to an external data file.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
block/qcow2.h | 2 +-
block/qcow2-bitmap.c | 7 +++---
block/qcow2-cache.c | 6 ++---
block/qcow2-cluster.c | 46 +++++++++++++++++++++++++++++++------
block/qcow2-refcount.c | 39 +++++++++++++++++++++++--------
block/qcow2-snapshot.c | 7 +++---
block/qcow2.c | 52 +++++++++++++++++++++++++++++++++---------
7 files changed, 122 insertions(+), 37 deletions(-)
diff --git a/block/qcow2.h b/block/qcow2.h
index fad6abf602..aac7fc4348 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -622,7 +622,7 @@ void qcow2_process_discards(BlockDriverState *bs, int ret);
int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,
int64_t size);
int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset,
- int64_t size);
+ int64_t size, bool data_file);
int qcow2_inc_refcounts_imrt(BlockDriverState *bs, BdrvCheckResult *res,
void **refcount_table,
int64_t *refcount_table_size,
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 3ee524da4b..9d968bdcda 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -778,7 +778,8 @@ static int bitmap_list_store(BlockDriverState *bs, Qcow2BitmapList *bm_list,
* directory in-place (actually, turn-off the extension), which is checked
* in qcow2_check_metadata_overlap() */
ret = qcow2_pre_write_overlap_check(
- bs, in_place ? QCOW2_OL_BITMAP_DIRECTORY : 0, dir_offset, dir_size);
+ bs, in_place ? QCOW2_OL_BITMAP_DIRECTORY : 0, dir_offset, dir_size,
+ false);
if (ret < 0) {
goto fail;
}
@@ -1224,7 +1225,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
memset(buf + write_size, 0, s->cluster_size - write_size);
}
- ret = qcow2_pre_write_overlap_check(bs, 0, off, s->cluster_size);
+ ret = qcow2_pre_write_overlap_check(bs, 0, off, s->cluster_size, false);
if (ret < 0) {
error_setg_errno(errp, -ret, "Qcow2 overlap check failed");
goto fail;
@@ -1292,7 +1293,7 @@ static int store_bitmap(BlockDriverState *bs, Qcow2Bitmap *bm, Error **errp)
}
ret = qcow2_pre_write_overlap_check(bs, 0, tb_offset,
- tb_size * sizeof(tb[0]));
+ tb_size * sizeof(tb[0]), false);
if (ret < 0) {
error_setg_errno(errp, -ret, "Qcow2 overlap check failed");
goto fail;
diff --git a/block/qcow2-cache.c b/block/qcow2-cache.c
index d9dafa31e5..df02e7b20a 100644
--- a/block/qcow2-cache.c
+++ b/block/qcow2-cache.c
@@ -205,13 +205,13 @@ static int qcow2_cache_entry_flush(BlockDriverState *bs, Qcow2Cache *c, int i)
if (c == s->refcount_block_cache) {
ret = qcow2_pre_write_overlap_check(bs, QCOW2_OL_REFCOUNT_BLOCK,
- c->entries[i].offset, c->table_size);
+ c->entries[i].offset, c->table_size, false);
} else if (c == s->l2_table_cache) {
ret = qcow2_pre_write_overlap_check(bs, QCOW2_OL_ACTIVE_L2,
- c->entries[i].offset, c->table_size);
+ c->entries[i].offset, c->table_size, false);
} else {
ret = qcow2_pre_write_overlap_check(bs, 0,
- c->entries[i].offset, c->table_size);
+ c->entries[i].offset, c->table_size, false);
}
if (ret < 0) {
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 8c4b4005ff..7579f5a5ae 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -153,7 +153,7 @@ int qcow2_grow_l1_table(BlockDriverState *bs, uint64_t min_size,
/* the L1 position has not yet been updated, so these clusters must
* indeed be completely free */
ret = qcow2_pre_write_overlap_check(bs, 0, new_l1_table_offset,
- new_l1_size2);
+ new_l1_size2, false);
if (ret < 0) {
goto fail;
}
@@ -238,7 +238,7 @@ int qcow2_write_l1_entry(BlockDriverState *bs, int l1_index)
}
ret = qcow2_pre_write_overlap_check(bs, QCOW2_OL_ACTIVE_L1,
- s->l1_table_offset + 8 * l1_start_index, sizeof(buf));
+ s->l1_table_offset + 8 * l1_start_index, sizeof(buf), false);
if (ret < 0) {
return ret;
}
@@ -490,6 +490,7 @@ static int coroutine_fn do_perform_cow_write(BlockDriverState *bs,
unsigned offset_in_cluster,
QEMUIOVector *qiov)
{
+ BDRVQcow2State *s = bs->opaque;
int ret;
if (qiov->size == 0) {
@@ -497,13 +498,13 @@ static int coroutine_fn do_perform_cow_write(BlockDriverState *bs,
}
ret = qcow2_pre_write_overlap_check(bs, 0,
- cluster_offset + offset_in_cluster, qiov->size);
+ cluster_offset + offset_in_cluster, qiov->size, true);
if (ret < 0) {
return ret;
}
BLKDBG_EVENT(bs->file, BLKDBG_COW_WRITE);
- ret = bdrv_co_pwritev(bs->file, cluster_offset + offset_in_cluster,
+ ret = bdrv_co_pwritev(s->data_file, cluster_offset + offset_in_cluster,
qiov->size, qiov, 0);
if (ret < 0) {
return ret;
@@ -607,6 +608,14 @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset,
}
switch (type) {
case QCOW2_CLUSTER_COMPRESSED:
+ if (has_data_file(bs)) {
+ qcow2_signal_corruption(bs, true, -1, -1, "Compressed cluster "
+ "entry found in image with external data "
+ "file (L2 offset: %#" PRIx64 ", L2 index: "
+ "%#x)", l2_offset, l2_index);
+ ret = -EIO;
+ goto fail;
+ }
/* Compressed clusters can only be processed one by one */
c = 1;
*cluster_offset &= L2E_COMPRESSED_OFFSET_SIZE_MASK;
@@ -633,6 +642,17 @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset,
ret = -EIO;
goto fail;
}
+ if (has_data_file(bs) && *cluster_offset != offset - offset_in_cluster)
+ {
+ qcow2_signal_corruption(bs, true, -1, -1,
+ "External data file host cluster offset %#"
+ PRIx64 " does not match guest cluster "
+ "offset: %#" PRIx64
+ ", L2 index: %#x)", *cluster_offset,
+ offset - offset_in_cluster, l2_index);
+ ret = -EIO;
+ goto fail;
+ }
break;
default:
abort();
@@ -753,6 +773,10 @@ int qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs,
int64_t cluster_offset;
int nb_csectors;
+ if (has_data_file(bs)) {
+ return 0;
+ }
+
ret = get_cluster_table(bs, offset, &l2_slice, &l2_index);
if (ret < 0) {
return ret;
@@ -1243,6 +1267,13 @@ static int do_alloc_cluster_offset(BlockDriverState *bs, uint64_t guest_offset,
trace_qcow2_do_alloc_clusters_offset(qemu_coroutine_self(), guest_offset,
*host_offset, *nb_clusters);
+ if (has_data_file(bs)) {
+ assert(*host_offset == INV_OFFSET ||
+ *host_offset == start_of_cluster(s, guest_offset));
+ *host_offset = start_of_cluster(s, guest_offset);
+ return 0;
+ }
+
/* Allocate new clusters */
trace_qcow2_cluster_alloc_phys(qemu_coroutine_self());
if (*host_offset == INV_OFFSET) {
@@ -1919,7 +1950,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,
}
ret = qcow2_pre_write_overlap_check(bs, 0, offset,
- s->cluster_size);
+ s->cluster_size, true);
if (ret < 0) {
if (cluster_type == QCOW2_CLUSTER_ZERO_PLAIN) {
qcow2_free_clusters(bs, offset, s->cluster_size,
@@ -1928,7 +1959,8 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,
goto fail;
}
- ret = bdrv_pwrite_zeroes(bs->file, offset, s->cluster_size, 0);
+ ret = bdrv_pwrite_zeroes(s->data_file, offset,
+ s->cluster_size, 0);
if (ret < 0) {
if (cluster_type == QCOW2_CLUSTER_ZERO_PLAIN) {
qcow2_free_clusters(bs, offset, s->cluster_size,
@@ -1955,7 +1987,7 @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table,
if (l2_dirty) {
ret = qcow2_pre_write_overlap_check(
bs, QCOW2_OL_INACTIVE_L2 | QCOW2_OL_ACTIVE_L2,
- slice_offset, slice_size2);
+ slice_offset, slice_size2, false);
if (ret < 0) {
goto fail;
}
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index 05e7974d7e..df73580e5d 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -1156,8 +1156,20 @@ void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry,
int nb_clusters, enum qcow2_discard_type type)
{
BDRVQcow2State *s = bs->opaque;
+ QCow2ClusterType ctype = qcow2_get_cluster_type(bs, l2_entry);
- switch (qcow2_get_cluster_type(bs, l2_entry)) {
+ if (has_data_file(bs)) {
+ if (s->discard_passthrough[type] &&
+ (ctype == QCOW2_CLUSTER_NORMAL ||
+ ctype == QCOW2_CLUSTER_ZERO_ALLOC))
+ {
+ bdrv_pdiscard(s->data_file, l2_entry & L2E_OFFSET_MASK,
+ nb_clusters << s->cluster_bits);
+ }
+ return;
+ }
+
+ switch (ctype) {
case QCOW2_CLUSTER_COMPRESSED:
{
int nb_csectors;
@@ -1649,7 +1661,7 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res,
l2_table[i] = cpu_to_be64(l2_entry);
ret = qcow2_pre_write_overlap_check(bs,
QCOW2_OL_ACTIVE_L2 | QCOW2_OL_INACTIVE_L2,
- l2e_offset, sizeof(uint64_t));
+ l2e_offset, sizeof(uint64_t), false);
if (ret < 0) {
fprintf(stderr, "ERROR: Overlap check failed\n");
res->check_errors++;
@@ -1898,7 +1910,8 @@ static int check_oflag_copied(BlockDriverState *bs, BdrvCheckResult *res,
if (l2_dirty) {
ret = qcow2_pre_write_overlap_check(bs, QCOW2_OL_ACTIVE_L2,
- l2_offset, s->cluster_size);
+ l2_offset, s->cluster_size,
+ false);
if (ret < 0) {
fprintf(stderr, "ERROR: Could not write L2 table; metadata "
"overlap check failed: %s\n", strerror(-ret));
@@ -2366,7 +2379,7 @@ write_refblocks:
}
ret = qcow2_pre_write_overlap_check(bs, 0, refblock_offset,
- s->cluster_size);
+ s->cluster_size, false);
if (ret < 0) {
fprintf(stderr, "ERROR writing refblock: %s\n", strerror(-ret));
goto fail;
@@ -2417,7 +2430,8 @@ write_refblocks:
}
ret = qcow2_pre_write_overlap_check(bs, 0, reftable_offset,
- reftable_size * sizeof(uint64_t));
+ reftable_size * sizeof(uint64_t),
+ false);
if (ret < 0) {
fprintf(stderr, "ERROR writing reftable: %s\n", strerror(-ret));
goto fail;
@@ -2751,10 +2765,15 @@ QEMU_BUILD_BUG_ON(QCOW2_OL_MAX_BITNR != ARRAY_SIZE(metadata_ol_names));
* overlaps; or a negative value (-errno) on error.
*/
int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset,
- int64_t size)
+ int64_t size, bool data_file)
{
- int ret = qcow2_check_metadata_overlap(bs, ign, offset, size);
+ int ret;
+
+ if (data_file && has_data_file(bs)) {
+ return 0;
+ }
+ ret = qcow2_check_metadata_overlap(bs, ign, offset, size);
if (ret < 0) {
return ret;
} else if (ret > 0) {
@@ -2855,7 +2874,8 @@ static int flush_refblock(BlockDriverState *bs, uint64_t **reftable,
if (reftable_index < *reftable_size && (*reftable)[reftable_index]) {
offset = (*reftable)[reftable_index];
- ret = qcow2_pre_write_overlap_check(bs, 0, offset, s->cluster_size);
+ ret = qcow2_pre_write_overlap_check(bs, 0, offset, s->cluster_size,
+ false);
if (ret < 0) {
error_setg_errno(errp, -ret, "Overlap check failed");
return ret;
@@ -3121,7 +3141,8 @@ int qcow2_change_refcount_order(BlockDriverState *bs, int refcount_order,
/* Write the new reftable */
ret = qcow2_pre_write_overlap_check(bs, 0, new_reftable_offset,
- new_reftable_size * sizeof(uint64_t));
+ new_reftable_size * sizeof(uint64_t),
+ false);
if (ret < 0) {
error_setg_errno(errp, -ret, "Overlap check failed");
goto done;
diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index 20e8472191..5ae3407f68 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -184,7 +184,7 @@ static int qcow2_write_snapshots(BlockDriverState *bs)
/* The snapshot list position has not yet been updated, so these clusters
* must indeed be completely free */
- ret = qcow2_pre_write_overlap_check(bs, 0, offset, snapshots_size);
+ ret = qcow2_pre_write_overlap_check(bs, 0, offset, snapshots_size, false);
if (ret < 0) {
goto fail;
}
@@ -389,7 +389,7 @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
}
ret = qcow2_pre_write_overlap_check(bs, 0, sn->l1_table_offset,
- s->l1_size * sizeof(uint64_t));
+ s->l1_size * sizeof(uint64_t), false);
if (ret < 0) {
goto fail;
}
@@ -528,7 +528,8 @@ int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id)
}
ret = qcow2_pre_write_overlap_check(bs, QCOW2_OL_ACTIVE_L1,
- s->l1_table_offset, cur_l1_bytes);
+ s->l1_table_offset, cur_l1_bytes,
+ false);
if (ret < 0) {
goto fail;
}
diff --git a/block/qcow2.c b/block/qcow2.c
index 8d1f667e91..f048763e10 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -140,7 +140,7 @@ static ssize_t qcow2_crypto_hdr_init_func(QCryptoBlock *block, size_t headerlen,
/* Zero fill remaining space in cluster so it has predictable
* content in case of future spec changes */
clusterlen = size_to_clusters(s, headerlen) * s->cluster_size;
- assert(qcow2_pre_write_overlap_check(bs, 0, ret, clusterlen) == 0);
+ assert(qcow2_pre_write_overlap_check(bs, 0, ret, clusterlen, false) == 0);
ret = bdrv_pwrite_zeroes(bs->file,
ret + headerlen,
clusterlen - headerlen, 0);
@@ -1965,7 +1965,7 @@ static coroutine_fn int qcow2_co_preadv(BlockDriverState *bs, uint64_t offset,
*/
if (!cluster_data) {
cluster_data =
- qemu_try_blockalign(bs->file->bs,
+ qemu_try_blockalign(s->data_file->bs,
QCOW_MAX_CRYPT_CLUSTERS
* s->cluster_size);
if (cluster_data == NULL) {
@@ -1981,7 +1981,7 @@ static coroutine_fn int qcow2_co_preadv(BlockDriverState *bs, uint64_t offset,
BLKDBG_EVENT(bs->file, BLKDBG_READ_AIO);
qemu_co_mutex_unlock(&s->lock);
- ret = bdrv_co_preadv(bs->file,
+ ret = bdrv_co_preadv(s->data_file,
cluster_offset + offset_in_cluster,
cur_bytes, &hd_qiov, 0);
qemu_co_mutex_lock(&s->lock);
@@ -2140,7 +2140,7 @@ static coroutine_fn int qcow2_co_pwritev(BlockDriverState *bs, uint64_t offset,
}
ret = qcow2_pre_write_overlap_check(bs, 0,
- cluster_offset + offset_in_cluster, cur_bytes);
+ cluster_offset + offset_in_cluster, cur_bytes, true);
if (ret < 0) {
goto fail;
}
@@ -2154,7 +2154,7 @@ static coroutine_fn int qcow2_co_pwritev(BlockDriverState *bs, uint64_t offset,
BLKDBG_EVENT(bs->file, BLKDBG_WRITE_AIO);
trace_qcow2_writev_data(qemu_coroutine_self(),
cluster_offset + offset_in_cluster);
- ret = bdrv_co_pwritev(bs->file,
+ ret = bdrv_co_pwritev(s->data_file,
cluster_offset + offset_in_cluster,
cur_bytes, &hd_qiov, 0);
qemu_co_mutex_lock(&s->lock);
@@ -3356,7 +3356,7 @@ qcow2_co_copy_range_from(BlockDriverState *bs,
goto out;
case QCOW2_CLUSTER_NORMAL:
- child = bs->file;
+ child = s->data_file;
copy_offset += offset_into_cluster(s, src_offset);
if ((copy_offset & 511) != 0) {
ret = -EIO;
@@ -3426,14 +3426,14 @@ qcow2_co_copy_range_to(BlockDriverState *bs,
assert((cluster_offset & 511) == 0);
ret = qcow2_pre_write_overlap_check(bs, 0,
- cluster_offset + offset_in_cluster, cur_bytes);
+ cluster_offset + offset_in_cluster, cur_bytes, true);
if (ret < 0) {
goto fail;
}
qemu_co_mutex_unlock(&s->lock);
ret = bdrv_co_copy_range_to(src, src_offset,
- bs->file,
+ s->data_file,
cluster_offset + offset_in_cluster,
cur_bytes, read_flags, write_flags);
qemu_co_mutex_lock(&s->lock);
@@ -3588,6 +3588,17 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,
int64_t old_file_size, new_file_size;
uint64_t nb_new_data_clusters, nb_new_l2_tables;
+ /* With a data file, preallocation means just allocating the metadata
+ * and forwarding the truncate request to the data file */
+ if (has_data_file(bs)) {
+ ret = preallocate_co(bs, old_length, offset);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "Preallocation failed");
+ goto fail;
+ }
+ break;
+ }
+
old_file_size = bdrv_getlength(bs->file->bs);
if (old_file_size < 0) {
error_setg_errno(errp, -old_file_size,
@@ -3696,6 +3707,16 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,
bs->total_sectors = offset / BDRV_SECTOR_SIZE;
+ if (has_data_file(bs)) {
+ if (prealloc == PREALLOC_MODE_METADATA) {
+ prealloc = PREALLOC_MODE_OFF;
+ }
+ ret = bdrv_co_truncate(s->data_file, offset, prealloc, errp);
+ if (ret < 0) {
+ goto fail;
+ }
+ }
+
/* write updated header.size */
offset = cpu_to_be64(offset);
ret = bdrv_pwrite_sync(bs->file, offsetof(QCowHeader, size),
@@ -3898,6 +3919,10 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uint64_t offset,
uint8_t *buf, *out_buf;
uint64_t cluster_offset;
+ if (has_data_file(bs)) {
+ return -ENOTSUP;
+ }
+
if (bytes == 0) {
/* align end of file to a sector boundary to ease reading with
sector based I/Os */
@@ -3949,7 +3974,7 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uint64_t offset,
goto fail;
}
- ret = qcow2_pre_write_overlap_check(bs, 0, cluster_offset, out_len);
+ ret = qcow2_pre_write_overlap_check(bs, 0, cluster_offset, out_len, true);
qemu_co_mutex_unlock(&s->lock);
if (ret < 0) {
goto fail;
@@ -3957,8 +3982,8 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uint64_t offset,
qemu_iovec_init_buf(&hd_qiov, out_buf, out_len);
- BLKDBG_EVENT(bs->file, BLKDBG_WRITE_COMPRESSED);
- ret = bdrv_co_pwritev(bs->file, cluster_offset, out_len, &hd_qiov, 0);
+ BLKDBG_EVENT(s->data_file, BLKDBG_WRITE_COMPRESSED);
+ ret = bdrv_co_pwritev(s->data_file, cluster_offset, out_len, &hd_qiov, 0);
if (ret < 0) {
goto fail;
}
@@ -4547,6 +4572,11 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,
return -ENOTSUP;
}
+ if (has_data_file(bs)) {
+ error_setg(errp, "Cannot downgrade an image with a data file");
+ return -ENOTSUP;
+ }
+
/* clear incompatible features */
if (s->incompatible_features & QCOW2_INCOMPAT_DIRTY) {
ret = qcow2_mark_clean(bs);
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 22/33] qcow2: Return error for snapshot operation with data file
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (20 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 21/33] qcow2: External file I/O Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 23/33] qcow2: Support external data file in qemu-img check Kevin Wolf
` (11 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Internal snapshots and an external data file are incompatible because
snapshots require refcounting and non-linear mapping. Return an error
for all of the snapshot operations if an external data file is in use.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
block/qcow2-snapshot.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index 5ae3407f68..a6ffae89a6 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -353,6 +353,10 @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
return -EFBIG;
}
+ if (has_data_file(bs)) {
+ return -ENOTSUP;
+ }
+
memset(sn, 0, sizeof(*sn));
/* Generate an ID */
@@ -466,6 +470,10 @@ int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id)
int ret;
uint64_t *sn_l1_table = NULL;
+ if (has_data_file(bs)) {
+ return -ENOTSUP;
+ }
+
/* Search the snapshot */
snapshot_index = find_snapshot_by_id_or_name(bs, snapshot_id);
if (snapshot_index < 0) {
@@ -599,6 +607,10 @@ int qcow2_snapshot_delete(BlockDriverState *bs,
QCowSnapshot sn;
int snapshot_index, ret;
+ if (has_data_file(bs)) {
+ return -ENOTSUP;
+ }
+
/* Search the snapshot */
snapshot_index = find_snapshot_by_id_and_name(bs, snapshot_id, name);
if (snapshot_index < 0) {
@@ -670,6 +682,9 @@ int qcow2_snapshot_list(BlockDriverState *bs, QEMUSnapshotInfo **psn_tab)
QCowSnapshot *sn;
int i;
+ if (has_data_file(bs)) {
+ return -ENOTSUP;
+ }
if (!s->nb_snapshots) {
*psn_tab = NULL;
return s->nb_snapshots;
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 23/33] qcow2: Support external data file in qemu-img check
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (21 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 22/33] qcow2: Return error for snapshot operation with data file Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 24/33] qcow2: Add basic data-file infrastructure Kevin Wolf
` (10 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
For external data files, data clusters must be excluded from the
refcount calculations. Instead, an implicit refcount of 1 is assumed for
the COPIED flag.
Compressed clusters and internal snapshots are incompatible with
external data files, so print an error if they are in use for images
with an external data file.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
block/qcow2-refcount.c | 41 ++++++++++++++++++++++++++++++-----------
1 file changed, 30 insertions(+), 11 deletions(-)
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index df73580e5d..e0fe322500 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -1605,6 +1605,13 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res,
res->corruptions++;
}
+ if (has_data_file(bs)) {
+ fprintf(stderr, "ERROR compressed cluster %d with data file, "
+ "entry=0x%" PRIx64 "\n", i, l2_entry);
+ res->corruptions++;
+ break;
+ }
+
/* Mark cluster as used */
nb_csectors = ((l2_entry >> s->csize_shift) &
s->csize_mask) + 1;
@@ -1695,11 +1702,13 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res,
}
/* Mark cluster as used */
- ret = qcow2_inc_refcounts_imrt(bs, res,
- refcount_table, refcount_table_size,
- offset, s->cluster_size);
- if (ret < 0) {
- goto fail;
+ if (!has_data_file(bs)) {
+ ret = qcow2_inc_refcounts_imrt(bs, res, refcount_table,
+ refcount_table_size,
+ offset, s->cluster_size);
+ if (ret < 0) {
+ goto fail;
+ }
}
break;
}
@@ -1884,12 +1893,16 @@ static int check_oflag_copied(BlockDriverState *bs, BdrvCheckResult *res,
if (cluster_type == QCOW2_CLUSTER_NORMAL ||
cluster_type == QCOW2_CLUSTER_ZERO_ALLOC) {
- ret = qcow2_get_refcount(bs,
- data_offset >> s->cluster_bits,
- &refcount);
- if (ret < 0) {
- /* don't print message nor increment check_errors */
- continue;
+ if (has_data_file(bs)) {
+ refcount = 1;
+ } else {
+ ret = qcow2_get_refcount(bs,
+ data_offset >> s->cluster_bits,
+ &refcount);
+ if (ret < 0) {
+ /* don't print message nor increment check_errors */
+ continue;
+ }
}
if ((refcount == 1) != ((l2_entry & QCOW_OFLAG_COPIED) != 0)) {
fprintf(stderr, "%s OFLAG_COPIED data cluster: "
@@ -2083,6 +2096,12 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
}
/* snapshots */
+ if (has_data_file(bs) && s->nb_snapshots) {
+ fprintf(stderr, "ERROR %d snapshots in image with data file\n",
+ s->nb_snapshots);
+ res->corruptions++;
+ }
+
for (i = 0; i < s->nb_snapshots; i++) {
sn = s->snapshots + i;
if (offset_into_cluster(s, sn->l1_table_offset)) {
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 24/33] qcow2: Add basic data-file infrastructure
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (22 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 23/33] qcow2: Support external data file in qemu-img check Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 25/33] qcow2: Creating images with external data file Kevin Wolf
` (9 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
This adds a .bdrv_open option to specify the external data file node.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
qapi/block-core.json | 7 ++++++-
block/qcow2.h | 4 +++-
block/qcow2.c | 34 ++++++++++++++++++++++++++++++++--
3 files changed, 41 insertions(+), 4 deletions(-)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 2b8afbb924..de4d4fd0e4 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -3080,6 +3080,10 @@
# encrypted images, except when doing a metadata-only
# probe of the image. (since 2.10)
#
+# @data-file: reference to or definition of the external data file.
+# This may only be specified for images that require an
+# external data file. (since 4.0)
+#
# Since: 2.9
##
{ 'struct': 'BlockdevOptionsQcow2',
@@ -3094,7 +3098,8 @@
'*l2-cache-entry-size': 'int',
'*refcount-cache-size': 'int',
'*cache-clean-interval': 'int',
- '*encrypt': 'BlockdevQcow2Encryption' } }
+ '*encrypt': 'BlockdevQcow2Encryption',
+ '*data-file': 'BlockdevRef' } }
##
# @SshHostKeyCheckMode:
diff --git a/block/qcow2.h b/block/qcow2.h
index aac7fc4348..f23c003a46 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -91,6 +91,7 @@
#define DEFAULT_CLUSTER_SIZE 65536
+#define QCOW2_OPT_DATA_FILE "data-file"
#define QCOW2_OPT_LAZY_REFCOUNTS "lazy-refcounts"
#define QCOW2_OPT_DISCARD_REQUEST "pass-discard-request"
#define QCOW2_OPT_DISCARD_SNAPSHOT "pass-discard-snapshot"
@@ -205,7 +206,8 @@ enum {
QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
- | QCOW2_INCOMPAT_CORRUPT,
+ | QCOW2_INCOMPAT_CORRUPT
+ | QCOW2_INCOMPAT_DATA_FILE,
};
/* Compatible feature bits */
diff --git a/block/qcow2.c b/block/qcow2.c
index f048763e10..ddbf8731bd 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1453,8 +1453,31 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
goto fail;
}
- /* TODO Open external data file */
- s->data_file = bs->file;
+ /* Open external data file */
+ s->data_file = bdrv_open_child(NULL, options, "data-file", bs, &child_file,
+ true, &local_err);
+ if (local_err) {
+ error_propagate(errp, local_err);
+ ret = -EINVAL;
+ goto fail;
+ }
+
+ if (s->incompatible_features & QCOW2_INCOMPAT_DATA_FILE) {
+ if (!s->data_file) {
+ error_setg(errp, "'data-file' is required for this image");
+ ret = -EINVAL;
+ goto fail;
+ }
+ } else {
+ if (s->data_file) {
+ error_setg(errp, "'data-file' can only be set for images with an "
+ "external data file");
+ ret = -EINVAL;
+ goto fail;
+ } else {
+ s->data_file = bs->file;
+ }
+ }
/* qcow2_read_extension may have set up the crypto context
* if the crypt method needs a header region, some methods
@@ -1627,6 +1650,9 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
return ret;
fail:
+ if (has_data_file(bs)) {
+ bdrv_unref_child(bs, s->data_file);
+ }
g_free(s->unknown_header_fields);
cleanup_unknown_header_ext(bs);
qcow2_free_snapshots(bs);
@@ -2246,6 +2272,10 @@ static void qcow2_close(BlockDriverState *bs)
g_free(s->image_backing_file);
g_free(s->image_backing_format);
+ if (has_data_file(bs)) {
+ bdrv_unref_child(bs, s->data_file);
+ }
+
qcow2_refcount_close(bs);
qcow2_free_snapshots(bs);
}
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 25/33] qcow2: Creating images with external data file
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (23 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 24/33] qcow2: Add basic data-file infrastructure Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 26/33] qcow2: Store data file name in the image Kevin Wolf
` (8 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
This adds a .bdrv_create option to use an external data file.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
qapi/block-core.json | 4 ++++
include/block/block_int.h | 1 +
block/qcow2.c | 26 ++++++++++++++++++++++++++
3 files changed, 31 insertions(+)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index de4d4fd0e4..2303266bc4 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4135,6 +4135,9 @@
# Driver specific image creation options for qcow2.
#
# @file Node to create the image format on
+# @data-file Node to use as an external data file in which all guest
+# data is stored so that only metadata remains in the qcow2
+# file (since: 4.0)
# @size Size of the virtual disk in bytes
# @version Compatibility level (default: v3)
# @backing-file File name of the backing file if a backing file
@@ -4150,6 +4153,7 @@
##
{ 'struct': 'BlockdevCreateOptionsQcow2',
'data': { 'file': 'BlockdevRef',
+ '*data-file': 'BlockdevRef',
'size': 'size',
'*version': 'BlockdevQcow2Version',
'*backing-file': 'str',
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 836d67c1ae..acd29ce521 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -56,6 +56,7 @@
#define BLOCK_OPT_NOCOW "nocow"
#define BLOCK_OPT_OBJECT_SIZE "object_size"
#define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"
+#define BLOCK_OPT_DATA_FILE "data_file"
#define BLOCK_PROBE_BUF_SIZE 512
diff --git a/block/qcow2.c b/block/qcow2.c
index ddbf8731bd..c20141b4e7 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -2880,6 +2880,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
*/
BlockBackend *blk = NULL;
BlockDriverState *bs = NULL;
+ BlockDriverState *data_bs = NULL;
QCowHeader *header;
size_t cluster_size;
int version;
@@ -2976,6 +2977,20 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
}
refcount_order = ctz32(qcow2_opts->refcount_bits);
+ if (qcow2_opts->data_file) {
+ if (version < 3) {
+ error_setg(errp, "External data files are only supported with "
+ "compatibility level 1.1 and above (use version=v3 or "
+ "greater)");
+ ret = -EINVAL;
+ goto out;
+ }
+ data_bs = bdrv_open_blockdev_ref(qcow2_opts->data_file, errp);
+ if (bs == NULL) {
+ ret = -EIO;
+ goto out;
+ }
+ }
/* Create BlockBackend to write to the image */
blk = blk_new(BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL);
@@ -3014,6 +3029,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
header->compatible_features |=
cpu_to_be64(QCOW2_COMPAT_LAZY_REFCOUNTS);
}
+ if (data_bs) {
+ header->incompatible_features |=
+ cpu_to_be64(QCOW2_INCOMPAT_DATA_FILE);
+ }
ret = blk_pwrite(blk, 0, header, cluster_size, 0);
g_free(header);
@@ -3044,6 +3063,9 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
options = qdict_new();
qdict_put_str(options, "driver", "qcow2");
qdict_put_str(options, "file", bs->node_name);
+ if (data_bs) {
+ qdict_put_str(options, "data-file", data_bs->node_name);
+ }
blk = blk_new_open(NULL, NULL, options,
BDRV_O_RDWR | BDRV_O_RESIZE | BDRV_O_NO_FLUSH,
&local_err);
@@ -3116,6 +3138,9 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
options = qdict_new();
qdict_put_str(options, "driver", "qcow2");
qdict_put_str(options, "file", bs->node_name);
+ if (data_bs) {
+ qdict_put_str(options, "data-file", data_bs->node_name);
+ }
blk = blk_new_open(NULL, NULL, options,
BDRV_O_RDWR | BDRV_O_NO_BACKING | BDRV_O_NO_IO,
&local_err);
@@ -3129,6 +3154,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
out:
blk_unref(blk);
bdrv_unref(bs);
+ bdrv_unref(data_bs);
return ret;
}
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 26/33] qcow2: Store data file name in the image
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (24 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 25/33] qcow2: Creating images with external data file Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 27/33] qcow2: Implement data-file-raw create option Kevin Wolf
` (7 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Rather than requiring that the external data file node is passed
explicitly when creating the qcow2 node, store the filename in the
designated header extension during .bdrv_create and read it from there
as a default during .bdrv_open.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
qapi/block-core.json | 8 +++-
block/qcow2.h | 1 +
block/qcow2.c | 94 +++++++++++++++++++++++++++++++++++++-
tests/qemu-iotests/082.out | 27 +++++++++++
4 files changed, 128 insertions(+), 2 deletions(-)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 2303266bc4..e6faa94fa2 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -59,6 +59,9 @@
#
# @compat: compatibility level
#
+# @data-file: the filename of the external data file that is stored in the
+# image and used as a default for opening the image (since: 4.0)
+#
# @lazy-refcounts: on or off; only valid for compat >= 1.1
#
# @corrupt: true if the image has been marked corrupt; only valid for
@@ -76,6 +79,7 @@
{ 'struct': 'ImageInfoSpecificQCow2',
'data': {
'compat': 'str',
+ '*data-file': 'str',
'*lazy-refcounts': 'bool',
'*corrupt': 'bool',
'refcount-bits': 'int',
@@ -3082,7 +3086,9 @@
#
# @data-file: reference to or definition of the external data file.
# This may only be specified for images that require an
-# external data file. (since 4.0)
+# external data file. If it is not specified for such
+# an image, the data file name is loaded from the image
+# file. (since 4.0)
#
# Since: 2.9
##
diff --git a/block/qcow2.h b/block/qcow2.h
index f23c003a46..a9c9cb4a26 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -343,6 +343,7 @@ typedef struct BDRVQcow2State {
* override) */
char *image_backing_file;
char *image_backing_format;
+ char *image_data_file;
CoQueue compress_wait_queue;
int nb_compress_threads;
diff --git a/block/qcow2.c b/block/qcow2.c
index c20141b4e7..f32ddda2a7 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -398,6 +398,21 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset,
#endif
break;
+ case QCOW2_EXT_MAGIC_DATA_FILE:
+ {
+ s->image_data_file = g_malloc0(ext.len + 1);
+ ret = bdrv_pread(bs->file, offset, s->image_data_file, ext.len);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret,
+ "ERROR: Could not read data file name");
+ return ret;
+ }
+#ifdef DEBUG_EXT
+ printf("Qcow2: Got external data file %s\n", s->image_data_file);
+#endif
+ break;
+ }
+
default:
/* unknown magic - save it in case we need to rewrite the header */
/* If you add a new feature, make sure to also update the fast
@@ -1463,6 +1478,15 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
}
if (s->incompatible_features & QCOW2_INCOMPAT_DATA_FILE) {
+ if (!s->data_file && s->image_data_file) {
+ s->data_file = bdrv_open_child(s->image_data_file, options,
+ "data-file", bs, &child_file,
+ false, errp);
+ if (!s->data_file) {
+ ret = -EINVAL;
+ goto fail;
+ }
+ }
if (!s->data_file) {
error_setg(errp, "'data-file' is required for this image");
ret = -EINVAL;
@@ -1650,6 +1674,7 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
return ret;
fail:
+ g_free(s->image_data_file);
if (has_data_file(bs)) {
bdrv_unref_child(bs, s->data_file);
}
@@ -2269,6 +2294,7 @@ static void qcow2_close(BlockDriverState *bs)
g_free(s->unknown_header_fields);
cleanup_unknown_header_ext(bs);
+ g_free(s->image_data_file);
g_free(s->image_backing_file);
g_free(s->image_backing_format);
@@ -2445,6 +2471,19 @@ int qcow2_update_header(BlockDriverState *bs)
buflen -= ret;
}
+ /* External data file header extension */
+ if (has_data_file(bs) && s->image_data_file) {
+ ret = header_ext_add(buf, QCOW2_EXT_MAGIC_DATA_FILE,
+ s->image_data_file, strlen(s->image_data_file),
+ buflen);
+ if (ret < 0) {
+ goto fail;
+ }
+
+ buf += ret;
+ buflen -= ret;
+ }
+
/* Full disk encryption header pointer extension */
if (s->crypto_header.offset != 0) {
s->crypto_header.offset = cpu_to_be64(s->crypto_header.offset);
@@ -3086,6 +3125,12 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
abort();
}
+ /* Set the external data file if necessary */
+ if (data_bs) {
+ BDRVQcow2State *s = blk_bs(blk)->opaque;
+ s->image_data_file = g_strdup(data_bs->filename);
+ }
+
/* Create a full header (including things like feature table) */
ret = qcow2_update_header(blk_bs(blk));
if (ret < 0) {
@@ -3165,6 +3210,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
QDict *qdict;
Visitor *v;
BlockDriverState *bs = NULL;
+ BlockDriverState *data_bs = NULL;
Error *local_err = NULL;
const char *val;
int ret;
@@ -3228,6 +3274,26 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
goto finish;
}
+ /* Create and open an external data file (protocol layer) */
+ val = qdict_get_try_str(qdict, BLOCK_OPT_DATA_FILE);
+ if (val) {
+ ret = bdrv_create_file(val, opts, errp);
+ if (ret < 0) {
+ goto finish;
+ }
+
+ data_bs = bdrv_open(val, NULL, NULL,
+ BDRV_O_RDWR | BDRV_O_RESIZE | BDRV_O_PROTOCOL,
+ errp);
+ if (data_bs == NULL) {
+ ret = -EIO;
+ goto finish;
+ }
+
+ qdict_del(qdict, BLOCK_OPT_DATA_FILE);
+ qdict_put_str(qdict, "data-file", data_bs->node_name);
+ }
+
/* Set 'driver' and 'node' options */
qdict_put_str(qdict, "driver", "qcow2");
qdict_put_str(qdict, "file", bs->node_name);
@@ -3262,6 +3328,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
finish:
qobject_unref(qdict);
bdrv_unref(bs);
+ bdrv_unref(data_bs);
qapi_free_BlockdevCreateOptions(create_options);
return ret;
}
@@ -4552,6 +4619,8 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs,
.refcount_bits = s->refcount_bits,
.has_bitmaps = !!bitmaps,
.bitmaps = bitmaps,
+ .has_data_file = !!s->image_data_file,
+ .data_file = g_strdup(s->image_data_file),
};
} else {
/* if this assertion fails, this probably means a new version was
@@ -4754,7 +4823,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
BDRVQcow2State *s = bs->opaque;
int old_version = s->qcow_version, new_version = old_version;
uint64_t new_size = 0;
- const char *backing_file = NULL, *backing_format = NULL;
+ const char *backing_file = NULL, *backing_format = NULL, *data_file = NULL;
bool lazy_refcounts = s->use_lazy_refcounts;
const char *compat = NULL;
uint64_t cluster_size = s->cluster_size;
@@ -4836,6 +4905,13 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
"may not exceed 64 bits");
return -EINVAL;
}
+ } else if (!strcmp(desc->name, BLOCK_OPT_DATA_FILE)) {
+ data_file = qemu_opt_get(opts, BLOCK_OPT_DATA_FILE);
+ if (data_file && !has_data_file(bs)) {
+ error_setg(errp, "data-file can only be set for images that "
+ "use an external data file");
+ return -EINVAL;
+ }
} else {
/* if this point is reached, this probably means a new option was
* added without having it covered here */
@@ -4882,6 +4958,17 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
}
}
+ if (data_file) {
+ g_free(s->image_data_file);
+ s->image_data_file = *data_file ? g_strdup(data_file) : NULL;
+ }
+
+ ret = qcow2_update_header(bs);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "Failed to update the image header");
+ return ret;
+ }
+
if (backing_file || backing_format) {
ret = qcow2_change_backing_file(bs,
backing_file ?: s->image_backing_file,
@@ -5029,6 +5116,11 @@ static QemuOptsList qcow2_create_opts = {
.type = QEMU_OPT_STRING,
.help = "Image format of the base image"
},
+ {
+ .name = BLOCK_OPT_DATA_FILE,
+ .type = QEMU_OPT_STRING,
+ .help = "File name of an external data file"
+ },
{
.name = BLOCK_OPT_ENCRYPT,
.type = QEMU_OPT_BOOL,
diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out
index 0ce18c075b..7dc59f6075 100644
--- a/tests/qemu-iotests/082.out
+++ b/tests/qemu-iotests/082.out
@@ -48,6 +48,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -69,6 +70,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -90,6 +92,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -111,6 +114,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -132,6 +136,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -153,6 +158,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -174,6 +180,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -195,6 +202,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -231,6 +239,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -304,6 +313,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -325,6 +335,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -346,6 +357,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -367,6 +379,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -388,6 +401,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -409,6 +423,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -430,6 +445,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -451,6 +467,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -487,6 +504,7 @@ Supported options:
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -568,6 +586,7 @@ Creation options for 'qcow2':
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -590,6 +609,7 @@ Creation options for 'qcow2':
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -612,6 +632,7 @@ Creation options for 'qcow2':
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -634,6 +655,7 @@ Creation options for 'qcow2':
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -656,6 +678,7 @@ Creation options for 'qcow2':
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -678,6 +701,7 @@ Creation options for 'qcow2':
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -700,6 +724,7 @@ Creation options for 'qcow2':
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -722,6 +747,7 @@ Creation options for 'qcow2':
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -761,6 +787,7 @@ Creation options for 'qcow2':
backing_fmt=<str> - Image format of the base image
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
+ data_file=<str> - File name of an external data file
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 27/33] qcow2: Implement data-file-raw create option
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (25 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 26/33] qcow2: Store data file name in the image Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 28/33] qemu-iotests: Preallocation with external data file Kevin Wolf
` (6 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Provide an option to force QEMU to always keep the external data file
consistent as a standalone read-only raw image.
At the moment, this means making sure that write_zeroes requests are
forwarded to the data file instead of just updating the metadata, and
checking that no backing file is used.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
qapi/block-core.json | 9 ++++++
block/qcow2.h | 9 +++++-
include/block/block_int.h | 1 +
block/qcow2-cluster.c | 10 +++++++
block/qcow2.c | 56 ++++++++++++++++++++++++++++++++++++--
tests/qemu-iotests/082.out | 27 ++++++++++++++++++
6 files changed, 109 insertions(+), 3 deletions(-)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index e6faa94fa2..919d0530b2 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -62,6 +62,10 @@
# @data-file: the filename of the external data file that is stored in the
# image and used as a default for opening the image (since: 4.0)
#
+# @data-file-raw: True if the external data file must stay valid as a
+# standalone (read-only) raw image without looking at qcow2
+# metadata (since: 4.0)
+#
# @lazy-refcounts: on or off; only valid for compat >= 1.1
#
# @corrupt: true if the image has been marked corrupt; only valid for
@@ -80,6 +84,7 @@
'data': {
'compat': 'str',
'*data-file': 'str',
+ '*data-file-raw': 'bool',
'*lazy-refcounts': 'bool',
'*corrupt': 'bool',
'refcount-bits': 'int',
@@ -4144,6 +4149,9 @@
# @data-file Node to use as an external data file in which all guest
# data is stored so that only metadata remains in the qcow2
# file (since: 4.0)
+# @data-file-raw True if the external data file must stay valid as a
+# standalone (read-only) raw image without looking at qcow2
+# metadata (default: false; since: 4.0)
# @size Size of the virtual disk in bytes
# @version Compatibility level (default: v3)
# @backing-file File name of the backing file if a backing file
@@ -4160,6 +4168,7 @@
{ 'struct': 'BlockdevCreateOptionsQcow2',
'data': { 'file': 'BlockdevRef',
'*data-file': 'BlockdevRef',
+ '*data-file-raw': 'bool',
'size': 'size',
'*version': 'BlockdevQcow2Version',
'*backing-file': 'str',
diff --git a/block/qcow2.h b/block/qcow2.h
index a9c9cb4a26..de2a3bdfc5 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -225,7 +225,8 @@ enum {
QCOW2_AUTOCLEAR_BITMAPS = 1 << QCOW2_AUTOCLEAR_BITMAPS_BITNR,
QCOW2_AUTOCLEAR_DATA_FILE_RAW = 1 << QCOW2_AUTOCLEAR_DATA_FILE_RAW_BITNR,
- QCOW2_AUTOCLEAR_MASK = QCOW2_AUTOCLEAR_BITMAPS,
+ QCOW2_AUTOCLEAR_MASK = QCOW2_AUTOCLEAR_BITMAPS
+ | QCOW2_AUTOCLEAR_DATA_FILE_RAW,
};
enum qcow2_discard_type {
@@ -474,6 +475,12 @@ static inline bool has_data_file(BlockDriverState *bs)
return (s->data_file != bs->file);
}
+static inline bool data_file_is_raw(BlockDriverState *bs)
+{
+ BDRVQcow2State *s = bs->opaque;
+ return !!(s->autoclear_features & QCOW2_AUTOCLEAR_DATA_FILE_RAW);
+}
+
static inline int64_t start_of_cluster(BDRVQcow2State *s, int64_t offset)
{
return offset & ~(s->cluster_size - 1);
diff --git a/include/block/block_int.h b/include/block/block_int.h
index acd29ce521..a23cabaddd 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -57,6 +57,7 @@
#define BLOCK_OPT_OBJECT_SIZE "object_size"
#define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"
#define BLOCK_OPT_DATA_FILE "data_file"
+#define BLOCK_OPT_DATA_FILE_RAW "data_file_raw"
#define BLOCK_PROBE_BUF_SIZE 512
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 7579f5a5ae..974a4e8656 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -1783,6 +1783,16 @@ int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset,
int64_t cleared;
int ret;
+ /* If we have to stay in sync with an external data file, zero out
+ * s->data_file first. */
+ if (data_file_is_raw(bs)) {
+ assert(has_data_file(bs));
+ ret = bdrv_co_pwrite_zeroes(s->data_file, offset, bytes, flags);
+ if (ret < 0) {
+ return ret;
+ }
+ }
+
/* Caller must pass aligned values, except at image end */
assert(QEMU_IS_ALIGNED(offset, s->cluster_size));
assert(QEMU_IS_ALIGNED(end_offset, s->cluster_size) ||
diff --git a/block/qcow2.c b/block/qcow2.c
index f32ddda2a7..c4dd876fb4 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1498,8 +1498,14 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
"external data file");
ret = -EINVAL;
goto fail;
- } else {
- s->data_file = bs->file;
+ }
+
+ s->data_file = bs->file;
+
+ if (data_file_is_raw(bs)) {
+ error_setg(errp, "data-file-raw requires a data file");
+ ret = -EINVAL;
+ goto fail;
}
}
@@ -2606,6 +2612,12 @@ static int qcow2_change_backing_file(BlockDriverState *bs,
{
BDRVQcow2State *s = bs->opaque;
+ /* Adding a backing file means that the external data file alone won't be
+ * enough to make sense of the content */
+ if (backing_file && data_file_is_raw(bs)) {
+ return -EINVAL;
+ }
+
if (backing_file && strlen(backing_file) > 1023) {
return -EINVAL;
}
@@ -3016,6 +3028,18 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
}
refcount_order = ctz32(qcow2_opts->refcount_bits);
+ if (qcow2_opts->data_file_raw && !qcow2_opts->data_file) {
+ error_setg(errp, "data-file-raw requires data-file");
+ ret = -EINVAL;
+ goto out;
+ }
+ if (qcow2_opts->data_file_raw && qcow2_opts->has_backing_file) {
+ error_setg(errp, "Backing file and data-file-raw cannot be used at "
+ "the same time");
+ ret = -EINVAL;
+ goto out;
+ }
+
if (qcow2_opts->data_file) {
if (version < 3) {
error_setg(errp, "External data files are only supported with "
@@ -3072,6 +3096,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
header->incompatible_features |=
cpu_to_be64(QCOW2_INCOMPAT_DATA_FILE);
}
+ if (qcow2_opts->data_file_raw) {
+ header->autoclear_features |=
+ cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
+ }
ret = blk_pwrite(blk, 0, header, cluster_size, 0);
g_free(header);
@@ -3253,6 +3281,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
{ BLOCK_OPT_REFCOUNT_BITS, "refcount-bits" },
{ BLOCK_OPT_ENCRYPT, BLOCK_OPT_ENCRYPT_FORMAT },
{ BLOCK_OPT_COMPAT_LEVEL, "version" },
+ { BLOCK_OPT_DATA_FILE_RAW, "data-file-raw" },
{ NULL, NULL },
};
@@ -4621,6 +4650,8 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs,
.bitmaps = bitmaps,
.has_data_file = !!s->image_data_file,
.data_file = g_strdup(s->image_data_file),
+ .has_data_file_raw = has_data_file(bs),
+ .data_file_raw = data_file_is_raw(bs),
};
} else {
/* if this assertion fails, this probably means a new version was
@@ -4825,6 +4856,7 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
uint64_t new_size = 0;
const char *backing_file = NULL, *backing_format = NULL, *data_file = NULL;
bool lazy_refcounts = s->use_lazy_refcounts;
+ bool data_file_raw = data_file_is_raw(bs);
const char *compat = NULL;
uint64_t cluster_size = s->cluster_size;
bool encrypt;
@@ -4912,6 +4944,14 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
"use an external data file");
return -EINVAL;
}
+ } else if (!strcmp(desc->name, BLOCK_OPT_DATA_FILE_RAW)) {
+ data_file_raw = qemu_opt_get_bool(opts, BLOCK_OPT_DATA_FILE_RAW,
+ data_file_raw);
+ if (data_file_raw && !data_file_is_raw(bs)) {
+ error_setg(errp, "data-file-raw cannot be set on existing "
+ "images");
+ return -EINVAL;
+ }
} else {
/* if this point is reached, this probably means a new option was
* added without having it covered here */
@@ -4958,6 +4998,13 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
}
}
+ /* data-file-raw blocks backing files, so clear it first if requested */
+ if (data_file_raw) {
+ s->autoclear_features |= QCOW2_AUTOCLEAR_DATA_FILE_RAW;
+ } else {
+ s->autoclear_features &= ~QCOW2_AUTOCLEAR_DATA_FILE_RAW;
+ }
+
if (data_file) {
g_free(s->image_data_file);
s->image_data_file = *data_file ? g_strdup(data_file) : NULL;
@@ -5121,6 +5168,11 @@ static QemuOptsList qcow2_create_opts = {
.type = QEMU_OPT_STRING,
.help = "File name of an external data file"
},
+ {
+ .name = BLOCK_OPT_DATA_FILE_RAW,
+ .type = QEMU_OPT_BOOL,
+ .help = "The external data file must stay valid as a raw image"
+ },
{
.name = BLOCK_OPT_ENCRYPT,
.type = QEMU_OPT_BOOL,
diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out
index 7dc59f6075..915640613f 100644
--- a/tests/qemu-iotests/082.out
+++ b/tests/qemu-iotests/082.out
@@ -49,6 +49,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -71,6 +72,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -93,6 +95,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -115,6 +118,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -137,6 +141,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -159,6 +164,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -181,6 +187,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -203,6 +210,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -240,6 +248,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -314,6 +323,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -336,6 +346,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -358,6 +369,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -380,6 +392,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -402,6 +415,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -424,6 +438,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -446,6 +461,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -468,6 +484,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -505,6 +522,7 @@ Supported options:
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -587,6 +605,7 @@ Creation options for 'qcow2':
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -610,6 +629,7 @@ Creation options for 'qcow2':
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -633,6 +653,7 @@ Creation options for 'qcow2':
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -656,6 +677,7 @@ Creation options for 'qcow2':
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -679,6 +701,7 @@ Creation options for 'qcow2':
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -702,6 +725,7 @@ Creation options for 'qcow2':
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -725,6 +749,7 @@ Creation options for 'qcow2':
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -748,6 +773,7 @@ Creation options for 'qcow2':
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
@@ -788,6 +814,7 @@ Creation options for 'qcow2':
cluster_size=<size> - qcow2 cluster size
compat=<str> - Compatibility level (0.10 or 1.1)
data_file=<str> - File name of an external data file
+ data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
encrypt.cipher-mode=<str> - Name of encryption cipher mode
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 28/33] qemu-iotests: Preallocation with external data file
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (26 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 27/33] qcow2: Implement data-file-raw create option Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 29/33] qemu-iotests: General tests for qcow2 " Kevin Wolf
` (5 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Test that preallocating metadata results in a somewhat larger qcow2
file, but preallocating data only affects the disk usage of the data
file and the qcow2 file stays small.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/qemu-iotests/243 | 22 ++++++++++++++++++++++
tests/qemu-iotests/243.out | 32 ++++++++++++++++++++++++++++++++
2 files changed, 54 insertions(+)
diff --git a/tests/qemu-iotests/243 b/tests/qemu-iotests/243
index 6a19919add..5838c6e89c 100755
--- a/tests/qemu-iotests/243
+++ b/tests/qemu-iotests/243
@@ -29,6 +29,7 @@ status=1 # failure is the default!
_cleanup()
{
_cleanup_test_img
+ rm -f $TEST_IMG.data
}
trap "_cleanup; exit \$status" 0 1 2 3 15
@@ -57,6 +58,27 @@ for mode in off metadata falloc full; do
done
+for mode in off metadata falloc full; do
+
+ echo
+ echo "=== External data file: preallocation=$mode ==="
+ echo
+
+ IMGOPTS="data_file=$TEST_IMG.data,preallocation=$mode" _make_test_img 64M
+
+ echo -n "qcow2 file size: "
+ du -b $TEST_IMG | cut -f1
+ echo -n "data file size: "
+ du -b $TEST_IMG.data | cut -f1
+
+ # Can't use precise numbers here because they differ between filesystems
+ echo -n "qcow2 disk usage: "
+ [ $(du -B1 $TEST_IMG | cut -f1) -lt 1048576 ] && echo "low" || echo "high"
+ echo -n "data disk usage: "
+ [ $(du -B1 $TEST_IMG.data | cut -f1) -lt 1048576 ] && echo "low" || echo "high"
+
+done
+
# success, all done
echo "*** done"
rm -f $seq.full
diff --git a/tests/qemu-iotests/243.out b/tests/qemu-iotests/243.out
index e531753ad1..dcb33fac32 100644
--- a/tests/qemu-iotests/243.out
+++ b/tests/qemu-iotests/243.out
@@ -23,4 +23,36 @@ Disk usage: high
Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=full
File size: 67436544
Disk usage: high
+
+=== External data file: preallocation=off ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=off
+qcow2 file size: 196616
+data file size: 67108864
+qcow2 disk usage: low
+data disk usage: low
+
+=== External data file: preallocation=metadata ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=metadata
+qcow2 file size: 327680
+data file size: 67108864
+qcow2 disk usage: low
+data disk usage: low
+
+=== External data file: preallocation=falloc ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=falloc
+qcow2 file size: 327680
+data file size: 67108864
+qcow2 disk usage: low
+data disk usage: high
+
+=== External data file: preallocation=full ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=full
+qcow2 file size: 327680
+data file size: 67108864
+qcow2 disk usage: low
+data disk usage: high
*** done
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 29/33] qemu-iotests: General tests for qcow2 with external data file
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (27 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 28/33] qemu-iotests: Preallocation with external data file Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 30/33] qemu-iotests: amend " Kevin Wolf
` (4 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/qemu-iotests/244 | 200 +++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/244.out | 125 +++++++++++++++++++++++
tests/qemu-iotests/group | 1 +
3 files changed, 326 insertions(+)
create mode 100755 tests/qemu-iotests/244
create mode 100644 tests/qemu-iotests/244.out
diff --git a/tests/qemu-iotests/244 b/tests/qemu-iotests/244
new file mode 100755
index 0000000000..d8e7122305
--- /dev/null
+++ b/tests/qemu-iotests/244
@@ -0,0 +1,200 @@
+#!/bin/bash
+#
+# Test qcow2 with external data files
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=kwolf@redhat.com
+
+seq=$(basename $0)
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+ rm -f $TEST_IMG.data
+ rm -f $TEST_IMG.src
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+echo
+echo "=== Create and open image with external data file ==="
+echo
+
+echo "With data file name in the image:"
+IMGOPTS="data_file=$TEST_IMG.data" _make_test_img 64M
+_check_test_img
+
+$QEMU_IO -c "open $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir
+$QEMU_IO -c "open -odata-file.filename=$TEST_IMG.data $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir
+$QEMU_IO -c "open -odata-file.filename=inexistent $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir
+
+echo
+echo "Data file required, but without data file name in the image:"
+$QEMU_IMG amend -odata_file= $TEST_IMG
+
+$QEMU_IO -c "open $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir
+$QEMU_IO -c "open -odata-file.filename=$TEST_IMG.data $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir
+$QEMU_IO -c "open -odata-file.filename=inexistent $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir
+
+echo
+echo "Setting data-file for an image with internal data:"
+_make_test_img 64M
+
+$QEMU_IO -c "open -odata-file.filename=$TEST_IMG.data $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir
+$QEMU_IO -c "open -odata-file.filename=inexistent $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir
+
+echo
+echo "=== Conflicting features ==="
+echo
+
+echo "Convert to compressed target with data file:"
+TEST_IMG="$TEST_IMG.src" _make_test_img 64M
+
+$QEMU_IO -c 'write -P 0x11 0 1M' \
+ -f $IMGFMT "$TEST_IMG.src" |
+ _filter_qemu_io
+
+$QEMU_IMG convert -f $IMGFMT -O $IMGFMT -c -odata_file="$TEST_IMG.data" \
+ "$TEST_IMG.src" "$TEST_IMG"
+
+echo
+echo "Convert uncompressed, then write compressed data manually:"
+$QEMU_IMG convert -f $IMGFMT -O $IMGFMT -odata_file="$TEST_IMG.data" \
+ "$TEST_IMG.src" "$TEST_IMG"
+$QEMU_IMG compare "$TEST_IMG.src" "$TEST_IMG"
+
+$QEMU_IO -c 'write -c -P 0x22 0 1M' \
+ -f $IMGFMT "$TEST_IMG" |
+ _filter_qemu_io
+_check_test_img
+
+echo
+echo "Take an internal snapshot:"
+
+$QEMU_IMG snapshot -c test "$TEST_IMG"
+_check_test_img
+
+echo
+echo "=== Standalone image with external data file (efficient) ==="
+echo
+
+IMGOPTS="data_file=$TEST_IMG.data" _make_test_img 64M
+
+echo -n "qcow2 file size before I/O: "
+du -b $TEST_IMG | cut -f1
+
+# Create image with the following layout
+# 0-1 MB: Unallocated
+# 1-2 MB: Written (pattern 0x11)
+# 2-3 MB: Discarded
+# 3-4 MB: Zero write over discarded space
+# 4-5 MB: Zero write over written space
+# 5-6 MB: Zero write over unallocated space
+
+echo
+$QEMU_IO -c 'write -P 0x11 1M 4M' \
+ -c 'discard 2M 2M' \
+ -c 'write -z 3M 3M' \
+ -f $IMGFMT "$TEST_IMG" |
+ _filter_qemu_io
+_check_test_img
+
+echo
+$QEMU_IMG map --output=json "$TEST_IMG"
+
+echo
+$QEMU_IO -c 'read -P 0 0 1M' \
+ -c 'read -P 0x11 1M 1M' \
+ -c 'read -P 0 2M 4M' \
+ -f $IMGFMT "$TEST_IMG" |
+ _filter_qemu_io
+
+# Zero clusters are only marked as such in the qcow2 metadata, but contain
+# stale data in the external data file
+echo
+$QEMU_IO -c 'read -P 0 0 1M' \
+ -c 'read -P 0x11 1M 1M' \
+ -c 'read -P 0 2M 2M' \
+ -c 'read -P 0x11 4M 1M' \
+ -c 'read -P 0 5M 1M' \
+ -f raw "$TEST_IMG.data" |
+ _filter_qemu_io
+
+
+echo -n "qcow2 file size after I/O: "
+du -b $TEST_IMG | cut -f1
+
+echo
+echo "=== Standalone image with external data file (valid raw) ==="
+echo
+
+IMGOPTS="data_file=$TEST_IMG.data,data_file_raw=on" _make_test_img 64M
+
+echo -n "qcow2 file size before I/O: "
+du -b $TEST_IMG | cut -f1
+
+echo
+$QEMU_IO -c 'write -P 0x11 1M 4M' \
+ -c 'discard 2M 2M' \
+ -c 'write -z 3M 3M' \
+ -f $IMGFMT "$TEST_IMG" |
+ _filter_qemu_io
+_check_test_img
+
+echo
+$QEMU_IMG map --output=json "$TEST_IMG"
+
+echo
+$QEMU_IO -c 'read -P 0 0 1M' \
+ -c 'read -P 0x11 1M 1M' \
+ -c 'read -P 0 2M 4M' \
+ -f $IMGFMT "$TEST_IMG" |
+ _filter_qemu_io
+
+echo
+$QEMU_IMG compare "$TEST_IMG" "$TEST_IMG.data"
+
+echo -n "qcow2 file size after I/O: "
+du -b $TEST_IMG | cut -f1
+
+echo
+echo "=== bdrv_co_block_status test for file and offset=0 ==="
+echo
+
+IMGOPTS="data_file=$TEST_IMG.data" _make_test_img 64M
+
+$QEMU_IO -c 'write -P 0x11 0 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c 'read -P 0x11 0 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
+$QEMU_IMG map --output=human "$TEST_IMG" | _filter_testdir
+$QEMU_IMG map --output=json "$TEST_IMG"
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/244.out b/tests/qemu-iotests/244.out
new file mode 100644
index 0000000000..98e5946976
--- /dev/null
+++ b/tests/qemu-iotests/244.out
@@ -0,0 +1,125 @@
+QA output created by 244
+
+=== Create and open image with external data file ===
+
+With data file name in the image:
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data
+No errors were found on the image.
+read 65536/65536 bytes at offset 0
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 65536/65536 bytes at offset 0
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+can't open device TEST_DIR/t.qcow2: Could not open 'inexistent': No such file or directory
+no file open, try 'help open'
+
+Data file required, but without data file name in the image:
+can't open device TEST_DIR/t.qcow2: 'data-file' is required for this image
+no file open, try 'help open'
+read 65536/65536 bytes at offset 0
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+can't open device TEST_DIR/t.qcow2: Could not open 'inexistent': No such file or directory
+no file open, try 'help open'
+
+Setting data-file for an image with internal data:
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+can't open device TEST_DIR/t.qcow2: 'data-file' can only be set for images with an external data file
+no file open, try 'help open'
+can't open device TEST_DIR/t.qcow2: Could not open 'inexistent': No such file or directory
+no file open, try 'help open'
+
+=== Conflicting features ===
+
+Convert to compressed target with data file:
+Formatting 'TEST_DIR/t.IMGFMT.src', fmt=IMGFMT size=67108864
+wrote 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+qemu-img: error while writing sector 0: Operation not supported
+
+Convert uncompressed, then write compressed data manually:
+Images are identical.
+write failed: Operation not supported
+No errors were found on the image.
+
+Take an internal snapshot:
+qemu-img: Could not create snapshot 'test': -95 (Operation not supported)
+No errors were found on the image.
+
+=== Standalone image with external data file (efficient) ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data
+qcow2 file size before I/O: 196616
+
+wrote 4194304/4194304 bytes at offset 1048576
+4 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+discard 2097152/2097152 bytes at offset 2097152
+2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 3145728/3145728 bytes at offset 3145728
+3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+No errors were found on the image.
+
+[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false},
+{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 1048576},
+{ "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false},
+{ "start": 4194304, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": 4194304},
+{ "start": 5242880, "length": 61865984, "depth": 0, "zero": true, "data": false}]
+
+read 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 1048576/1048576 bytes at offset 1048576
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 4194304/4194304 bytes at offset 2097152
+4 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+read 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 1048576/1048576 bytes at offset 1048576
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 2097152/2097152 bytes at offset 2097152
+2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 1048576/1048576 bytes at offset 4194304
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 1048576/1048576 bytes at offset 5242880
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+qcow2 file size after I/O: 327680
+
+=== Standalone image with external data file (valid raw) ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data data_file_raw=on
+qcow2 file size before I/O: 196616
+
+wrote 4194304/4194304 bytes at offset 1048576
+4 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+discard 2097152/2097152 bytes at offset 2097152
+2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 3145728/3145728 bytes at offset 3145728
+3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+No errors were found on the image.
+
+[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false},
+{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 1048576},
+{ "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false},
+{ "start": 4194304, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": 4194304},
+{ "start": 5242880, "length": 61865984, "depth": 0, "zero": true, "data": false}]
+
+read 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 1048576/1048576 bytes at offset 1048576
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 4194304/4194304 bytes at offset 2097152
+4 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+Images are identical.
+qcow2 file size after I/O: 327680
+
+=== bdrv_co_block_status test for file and offset=0 ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data
+wrote 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Offset Length Mapped to File
+0 0x100000 0 TEST_DIR/t.qcow2.data
+[{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 0},
+{ "start": 1048576, "length": 66060288, "depth": 0, "zero": true, "data": false}]
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 03aa93dbf5..36100b803c 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -242,3 +242,4 @@
240 auto quick
242 rw auto quick
243 rw auto quick
+244 rw auto quick
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 30/33] qemu-iotests: amend with external data file
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (28 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 29/33] qemu-iotests: General tests for qcow2 " Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 31/33] ahci-test: Add dependency to qemu-img tool Kevin Wolf
` (3 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/qemu-iotests/061 | 45 ++++++++++++++++++-
tests/qemu-iotests/061.out | 89 ++++++++++++++++++++++++++++++++++++++
2 files changed, 133 insertions(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/061 b/tests/qemu-iotests/061
index d2e68c0203..d7dbd7e2c7 100755
--- a/tests/qemu-iotests/061
+++ b/tests/qemu-iotests/061
@@ -28,7 +28,8 @@ status=1 # failure is the default!
_cleanup()
{
- _cleanup_test_img
+ _cleanup_test_img
+ rm -f $TEST_IMG.data
}
trap "_cleanup; exit \$status" 0 1 2 3 15
@@ -250,6 +251,48 @@ $QEMU_IMG snapshot -c foo "$TEST_IMG"
$QEMU_IMG amend -p -o "compat=0.10" "$TEST_IMG"
_check_test_img
+echo
+echo "=== Testing version downgrade with external data file ==="
+echo
+IMGOPTS="compat=1.1,data_file=$TEST_IMG.data" _make_test_img 64M
+$QEMU_IMG amend -o "compat=0.10" "$TEST_IMG"
+_img_info --format-specific
+_check_test_img
+
+echo
+echo "=== Try changing the external data file ==="
+echo
+IMGOPTS="compat=1.1" _make_test_img 64M
+$QEMU_IMG amend -o "data_file=foo" "$TEST_IMG"
+
+echo
+IMGOPTS="compat=1.1,data_file=$TEST_IMG.data" _make_test_img 64M
+$QEMU_IMG amend -o "data_file=foo" "$TEST_IMG"
+_img_info --format-specific
+TEST_IMG="data-file.filename=$TEST_IMG.data,file.filename=$TEST_IMG" _img_info --format-specific --image-opts
+
+echo
+$QEMU_IMG amend -o "data_file=" --image-opts "data-file.filename=$TEST_IMG.data,file.filename=$TEST_IMG"
+_img_info --format-specific
+TEST_IMG="data-file.filename=$TEST_IMG.data,file.filename=$TEST_IMG" _img_info --format-specific --image-opts
+
+echo
+echo "=== Clearing and setting data-file-raw ==="
+echo
+IMGOPTS="compat=1.1,data_file=$TEST_IMG.data,data_file_raw=on" _make_test_img 64M
+$QEMU_IMG amend -o "data_file_raw=on" "$TEST_IMG"
+_img_info --format-specific
+_check_test_img
+
+$QEMU_IMG amend -o "data_file_raw=off" "$TEST_IMG"
+_img_info --format-specific
+_check_test_img
+
+$QEMU_IMG amend -o "data_file_raw=on" "$TEST_IMG"
+_img_info --format-specific
+_check_test_img
+
+
# success, all done
echo "*** done"
rm -f $seq.full
diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
index 758284011b..9fe1ec702f 100644
--- a/tests/qemu-iotests/061.out
+++ b/tests/qemu-iotests/061.out
@@ -488,4 +488,93 @@ wrote 65536/65536 bytes at offset 3221225472
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
(0.00/100%)\r (6.25/100%)\r (12.50/100%)\r (18.75/100%)\r (25.00/100%)\r (31.25/100%)\r (37.50/100%)\r (43.75/100%)\r (50.00/100%)\r (56.25/100%)\r (62.50/100%)\r (68.75/100%)\r (75.00/100%)\r (81.25/100%)\r (87.50/100%)\r (93.75/100%)\r (100.00/100%)\r (100.00/100%)
No errors were found on the image.
+
+=== Testing version downgrade with external data file ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data
+qemu-img: Cannot downgrade an image with a data file
+image: TEST_DIR/t.IMGFMT
+file format: IMGFMT
+virtual size: 64M (67108864 bytes)
+cluster_size: 65536
+Format specific information:
+ compat: 1.1
+ lazy refcounts: false
+ refcount bits: 16
+ data file: TEST_DIR/t.IMGFMT.data
+ data file raw: false
+ corrupt: false
+No errors were found on the image.
+
+=== Try changing the external data file ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+qemu-img: data-file can only be set for images that use an external data file
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data
+qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Could not open 'foo': No such file or directory
+image: TEST_DIR/t.IMGFMT
+file format: IMGFMT
+virtual size: 64M (67108864 bytes)
+cluster_size: 65536
+Format specific information:
+ compat: 1.1
+ lazy refcounts: false
+ refcount bits: 16
+ data file: foo
+ data file raw: false
+ corrupt: false
+
+qemu-img: Could not open 'TEST_DIR/t.IMGFMT': 'data-file' is required for this image
+image: TEST_DIR/t.IMGFMT
+file format: IMGFMT
+virtual size: 64M (67108864 bytes)
+cluster_size: 65536
+Format specific information:
+ compat: 1.1
+ lazy refcounts: false
+ refcount bits: 16
+ data file raw: false
+ corrupt: false
+
+=== Clearing and setting data-file-raw ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data data_file_raw=on
+image: TEST_DIR/t.IMGFMT
+file format: IMGFMT
+virtual size: 64M (67108864 bytes)
+cluster_size: 65536
+Format specific information:
+ compat: 1.1
+ lazy refcounts: false
+ refcount bits: 16
+ data file: TEST_DIR/t.IMGFMT.data
+ data file raw: true
+ corrupt: false
+No errors were found on the image.
+image: TEST_DIR/t.IMGFMT
+file format: IMGFMT
+virtual size: 64M (67108864 bytes)
+cluster_size: 65536
+Format specific information:
+ compat: 1.1
+ lazy refcounts: false
+ refcount bits: 16
+ data file: TEST_DIR/t.IMGFMT.data
+ data file raw: false
+ corrupt: false
+No errors were found on the image.
+qemu-img: data-file-raw cannot be set on existing images
+image: TEST_DIR/t.IMGFMT
+file format: IMGFMT
+virtual size: 64M (67108864 bytes)
+cluster_size: 65536
+Format specific information:
+ compat: 1.1
+ lazy refcounts: false
+ refcount bits: 16
+ data file: TEST_DIR/t.IMGFMT.data
+ data file raw: false
+ corrupt: false
+No errors were found on the image.
*** done
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 31/33] ahci-test: Add dependency to qemu-img tool
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (29 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 30/33] qemu-iotests: amend " Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 32/33] qemu-iotests: Add dependency to qemu-nbd tool Kevin Wolf
` (2 subsequent siblings)
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
From: Philippe Mathieu-Daudé <philmd@redhat.com>
Since the ahci-test uses qemu-img, add a dependency to build it
before using it.
This fixes:
$ gmake check-qtest V=1
QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/ahci-test
Failed to execute child process "/tmp/qemu-test.19tMRF/qemu-img" (No such file or directory)
ERROR:tests/libqos/libqos.c:192:mkimg: assertion failed: (ret && !err)
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/Makefile.include | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 97e1cb90a3..b6bd0a5ed8 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -758,7 +758,7 @@ tests/prom-env-test$(EXESUF): tests/prom-env-test.o $(libqos-obj-y)
tests/rtas-test$(EXESUF): tests/rtas-test.o $(libqos-spapr-obj-y)
tests/fdc-test$(EXESUF): tests/fdc-test.o
tests/ide-test$(EXESUF): tests/ide-test.o $(libqos-pc-obj-y)
-tests/ahci-test$(EXESUF): tests/ahci-test.o $(libqos-pc-obj-y)
+tests/ahci-test$(EXESUF): tests/ahci-test.o $(libqos-pc-obj-y) qemu-img$(EXESUF)
tests/ipmi-kcs-test$(EXESUF): tests/ipmi-kcs-test.o
tests/ipmi-bt-test$(EXESUF): tests/ipmi-bt-test.o
tests/hd-geo-test$(EXESUF): tests/hd-geo-test.o
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 32/33] qemu-iotests: Add dependency to qemu-nbd tool
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (30 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 31/33] ahci-test: Add dependency to qemu-img tool Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-08 12:58 ` [Qemu-devel] [PULL 33/33] qcow2 spec: Describe string header extensions Kevin Wolf
2019-03-09 17:35 ` [Qemu-devel] [PULL 00/33] Block layer patches Peter Maydell
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
From: Philippe Mathieu-Daudé <philmd@redhat.com>
Since a9660664fde, some iotests use qemu-nbd.
Add a dependency to build it before using it.
This fixes:
$ make check-block
GEN qemu-img-cmds.h
CC qemu-img.o
LINK qemu-img
CC qemu-io.o
LINK qemu-io
CC tests/qemu-iotests/socket_scm_helper.o
LINK tests/qemu-iotests/socket_scm_helper
tests/qemu-iotests-quick.sh
check: qemu-nbd not found
make: *** [tests/Makefile.include:1059: check-tests/qemu-iotests-quick.sh] Error 1
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/Makefile.include | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/Makefile.include b/tests/Makefile.include
index b6bd0a5ed8..45a4302103 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -1104,7 +1104,7 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
QEMU_IOTESTS_HELPERS-$(call land,$(CONFIG_SOFTMMU),$(CONFIG_LINUX)) = tests/qemu-iotests/socket_scm_helper$(EXESUF)
.PHONY: check-tests/qemu-iotests-quick.sh
-check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) qemu-io$(EXESUF) $(QEMU_IOTESTS_HELPERS-y)
+check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) qemu-io$(EXESUF) qemu-nbd$(EXESUF) $(QEMU_IOTESTS_HELPERS-y)
$<
.PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Qemu-devel] [PULL 33/33] qcow2 spec: Describe string header extensions
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (31 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 32/33] qemu-iotests: Add dependency to qemu-nbd tool Kevin Wolf
@ 2019-03-08 12:58 ` Kevin Wolf
2019-03-09 17:35 ` [Qemu-devel] [PULL 00/33] Block layer patches Peter Maydell
33 siblings, 0 replies; 38+ messages in thread
From: Kevin Wolf @ 2019-03-08 12:58 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, peter.maydell, qemu-devel
Be more specific about the string representation in header extensions.
Suggested-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
---
docs/interop/qcow2.txt | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index bfb97cfde3..8c3098d8d9 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -170,11 +170,11 @@ be stored. Each extension has a structure like the following:
Byte 0 - 3: Header extension type:
0x00000000 - End of the header extension area
- 0xE2792ACA - Backing file format name
+ 0xE2792ACA - Backing file format name string
0x6803f857 - Feature name table
0x23852875 - Bitmaps extension
0x0537be77 - Full disk encryption header pointer
- 0x44415441 - External data file name
+ 0x44415441 - External data file name string
other - Unknown header extension, can be safely
ignored
@@ -196,6 +196,16 @@ data of compatible features that it doesn't support. Compatible features that
need space for additional data can use a header extension.
+== String header extensions ==
+
+Some header extensions (such as the backing file format name and the external
+data file name) are just a single string. In this case, the header extension
+length is the string length and the string is not '\0' terminated. (The header
+extension padding can make it look like a string is '\0' terminated, but
+neither is padding always necessary nor is there a guarantee that zero bytes
+are used for padding.)
+
+
== Feature name table ==
The feature name table is an optional header extension that contains the name
--
2.20.1
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [Qemu-devel] [PULL 00/33] Block layer patches
2019-03-08 12:57 [Qemu-devel] [PULL 00/33] Block layer patches Kevin Wolf
` (32 preceding siblings ...)
2019-03-08 12:58 ` [Qemu-devel] [PULL 33/33] qcow2 spec: Describe string header extensions Kevin Wolf
@ 2019-03-09 17:35 ` Peter Maydell
33 siblings, 0 replies; 38+ messages in thread
From: Peter Maydell @ 2019-03-09 17:35 UTC (permalink / raw)
To: Kevin Wolf; +Cc: Qemu-block, QEMU Developers
On Fri, 8 Mar 2019 at 12:58, Kevin Wolf <kwolf@redhat.com> wrote:
>
> The following changes since commit c4e0780ed1ffd056f205348d387a61b4136a45df:
>
> Merge remote-tracking branch 'remotes/vivier2/tags/linux-user-for-4.0-pull-request' into staging (2019-03-07 18:40:43 +0000)
>
> are available in the Git repository at:
>
> git://repo.or.cz/qemu/kevin.git tags/for-upstream
>
> for you to fetch changes up to e88153ea9a40009a8ae7648282c0eac1b7f5494f:
>
> qcow2 spec: Describe string header extensions (2019-03-08 12:26:46 +0100)
>
> ----------------------------------------------------------------
> Block layer patches:
>
> - qcow2: Support for external data files
> - qcow2: Default to 4KB for the qcow2 cache entry size
> - Apply block driver whitelist for -drive format=help
> - Several qemu-iotests improvements
>
Applied, thanks.
Please update the changelog at https://wiki.qemu.org/ChangeLog/4.0
for any user-visible changes.
-- PMM
^ permalink raw reply [flat|nested] 38+ messages in thread