* [PULL 00/11] Block patches
@ 2019-11-05 15:43 Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 01/11] image-fuzzer: Open image files in binary mode Stefan Hajnoczi
` (11 more replies)
0 siblings, 12 replies; 13+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:43 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell, qemu-block, Stefan Hajnoczi
The following changes since commit 36609b4fa36f0ac934874371874416f7533a5408:
Merge remote-tracking branch 'remotes/palmer/tags/palmer-for-master-4.2-sf1' into staging (2019-11-02 17:59:03 +0000)
are available in the Git repository at:
https://github.com/stefanha/qemu.git tags/block-pull-request
for you to fetch changes up to 9fdd7860adec188ed50d2530e9a819e8d953f9bb:
image-fuzzer: Use OSerror.strerror instead of tuple subscript (2019-11-05 16:36:11 +0100)
----------------------------------------------------------------
Pull request
Let's get the image fuzzer Python 3 changes merged in QEMU 4.2.
----------------------------------------------------------------
Eduardo Habkost (11):
image-fuzzer: Open image files in binary mode
image-fuzzer: Write bytes instead of string to image file
image-fuzzer: Explicitly use integer division operator
image-fuzzer: Use io.StringIO
image-fuzzer: Use %r for all fiels at Field.__repr__()
image-fuzzer: Return bytes objects on string fuzzing functions
image-fuzzer: Use bytes constant for field values
image-fuzzer: Encode file name and file format to bytes
image-fuzzer: Run using python3
image-fuzzer: Use errors parameter of subprocess.Popen()
image-fuzzer: Use OSerror.strerror instead of tuple subscript
tests/image-fuzzer/qcow2/__init__.py | 1 -
tests/image-fuzzer/qcow2/fuzz.py | 54 +++++++++++++-------------
tests/image-fuzzer/qcow2/layout.py | 57 ++++++++++++++--------------
tests/image-fuzzer/runner.py | 16 ++++----
4 files changed, 63 insertions(+), 65 deletions(-)
--
2.23.0
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PULL 01/11] image-fuzzer: Open image files in binary mode
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
@ 2019-11-05 15:43 ` Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 02/11] image-fuzzer: Write bytes instead of string to image file Stefan Hajnoczi
` (10 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Eduardo Habkost, qemu-block, John Snow,
Stefan Hajnoczi, Philippe Mathieu-Daudé
From: Eduardo Habkost <ehabkost@redhat.com>
This probably never caused problems because on Linux there's no
actual newline conversion happening, but on Python 3 the
binary/text distinction is stronger and we must explicitly open
the image file in binary mode.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-2-ehabkost@redhat.com
Message-Id: <20191016192430.25098-2-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
tests/image-fuzzer/qcow2/layout.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index 675877da96..c57418fa15 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -503,7 +503,7 @@ class Image(object):
def write(self, filename):
"""Write an entire image to the file."""
- image_file = open(filename, 'w')
+ image_file = open(filename, 'wb')
for field in self:
image_file.seek(field.offset)
image_file.write(struct.pack(field.fmt, field.value))
--
2.23.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PULL 02/11] image-fuzzer: Write bytes instead of string to image file
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 01/11] image-fuzzer: Open image files in binary mode Stefan Hajnoczi
@ 2019-11-05 15:43 ` Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 03/11] image-fuzzer: Explicitly use integer division operator Stefan Hajnoczi
` (9 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Eduardo Habkost, qemu-block, John Snow,
Stefan Hajnoczi, Philippe Mathieu-Daudé
From: Eduardo Habkost <ehabkost@redhat.com>
This is necessary for Python 3 compatibility.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-3-ehabkost@redhat.com
Message-Id: <20191016192430.25098-3-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
tests/image-fuzzer/qcow2/layout.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index c57418fa15..fe273d4143 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -518,7 +518,7 @@ class Image(object):
rounded = (size + self.cluster_size - 1) & ~(self.cluster_size - 1)
if rounded > size:
image_file.seek(rounded - 1)
- image_file.write("\0")
+ image_file.write(b'\x00')
image_file.close()
@staticmethod
--
2.23.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PULL 03/11] image-fuzzer: Explicitly use integer division operator
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 01/11] image-fuzzer: Open image files in binary mode Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 02/11] image-fuzzer: Write bytes instead of string to image file Stefan Hajnoczi
@ 2019-11-05 15:43 ` Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 04/11] image-fuzzer: Use io.StringIO Stefan Hajnoczi
` (8 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Eduardo Habkost, qemu-block, John Snow,
Stefan Hajnoczi, Philippe Mathieu-Daudé
From: Eduardo Habkost <ehabkost@redhat.com>
Most of the division expressions in image-fuzzer assume integer
division. Use the // operator to keep the same behavior when we
move to Python 3.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-4-ehabkost@redhat.com
Message-Id: <20191016192430.25098-4-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
tests/image-fuzzer/qcow2/fuzz.py | 12 ++++-----
tests/image-fuzzer/qcow2/layout.py | 40 +++++++++++++++---------------
2 files changed, 26 insertions(+), 26 deletions(-)
diff --git a/tests/image-fuzzer/qcow2/fuzz.py b/tests/image-fuzzer/qcow2/fuzz.py
index abc4f0635d..154dc06cc0 100644
--- a/tests/image-fuzzer/qcow2/fuzz.py
+++ b/tests/image-fuzzer/qcow2/fuzz.py
@@ -27,14 +27,14 @@ UINT64 = 0xffffffffffffffff
UINT32_M = 31
UINT64_M = 63
# Fuzz vectors
-UINT8_V = [0, 0x10, UINT8/4, UINT8/2 - 1, UINT8/2, UINT8/2 + 1, UINT8 - 1,
+UINT8_V = [0, 0x10, UINT8//4, UINT8//2 - 1, UINT8//2, UINT8//2 + 1, UINT8 - 1,
UINT8]
-UINT16_V = [0, 0x100, 0x1000, UINT16/4, UINT16/2 - 1, UINT16/2, UINT16/2 + 1,
+UINT16_V = [0, 0x100, 0x1000, UINT16//4, UINT16//2 - 1, UINT16//2, UINT16//2 + 1,
UINT16 - 1, UINT16]
-UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32/4, UINT32/2 - 1,
- UINT32/2, UINT32/2 + 1, UINT32 - 1, UINT32]
-UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64/4,
- UINT64/2 - 1, UINT64/2, UINT64/2 + 1, UINT64 - 1,
+UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32//4, UINT32//2 - 1,
+ UINT32//2, UINT32//2 + 1, UINT32 - 1, UINT32]
+UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64//4,
+ UINT64//2 - 1, UINT64//2, UINT64//2 + 1, UINT64 - 1,
UINT64]
STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x',
'%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n',
diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index fe273d4143..6501c9fd4b 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -253,7 +253,7 @@ class Image(object):
['>I', self.ext_offset, 0x6803f857, 'ext_magic'],
# One feature table contains 3 fields and takes 48 bytes
['>I', self.ext_offset + UINT32_S,
- len(feature_tables) / 3 * 48, 'ext_length']
+ len(feature_tables) // 3 * 48, 'ext_length']
] + feature_tables)
self.ext_offset = inner_offset
@@ -271,7 +271,7 @@ class Image(object):
def create_l2_entry(host, guest, l2_cluster):
"""Generate one L2 entry."""
offset = l2_cluster * self.cluster_size
- l2_size = self.cluster_size / UINT64_S
+ l2_size = self.cluster_size // UINT64_S
entry_offset = offset + UINT64_S * (guest % l2_size)
cluster_descriptor = host * self.cluster_size
if not self.header['version'][0].value == 2:
@@ -283,8 +283,8 @@ class Image(object):
def create_l1_entry(l2_cluster, l1_offset, guest):
"""Generate one L1 entry."""
- l2_size = self.cluster_size / UINT64_S
- entry_offset = l1_offset + UINT64_S * (guest / l2_size)
+ l2_size = self.cluster_size // UINT64_S
+ entry_offset = l1_offset + UINT64_S * (guest // l2_size)
# While snapshots are not supported bit #63 = 1
entry_val = (1 << 63) + l2_cluster * self.cluster_size
return ['>Q', entry_offset, entry_val, 'l1_entry']
@@ -298,11 +298,11 @@ class Image(object):
l2 = []
else:
meta_data = self._get_metadata()
- guest_clusters = random.sample(range(self.image_size /
+ guest_clusters = random.sample(range(self.image_size //
self.cluster_size),
len(self.data_clusters))
# Number of entries in a L1/L2 table
- l_size = self.cluster_size / UINT64_S
+ l_size = self.cluster_size // UINT64_S
# Number of clusters necessary for L1 table
l1_size = int(ceil((max(guest_clusters) + 1) / float(l_size**2)))
l1_start = self._get_adjacent_clusters(self.data_clusters |
@@ -318,7 +318,7 @@ class Image(object):
# L2 entries
l2 = []
for host, guest in zip(self.data_clusters, guest_clusters):
- l2_id = guest / l_size
+ l2_id = guest // l_size
if l2_id not in l2_ids:
l2_ids.append(l2_id)
l2_clusters.append(self._get_adjacent_clusters(
@@ -339,14 +339,14 @@ class Image(object):
def allocate_rfc_blocks(data, size):
"""Return indices of clusters allocated for refcount blocks."""
cluster_ids = set()
- diff = block_ids = set([x / size for x in data])
+ diff = block_ids = set([x // size for x in data])
while len(diff) != 0:
# Allocate all yet not allocated clusters
new = self._get_available_clusters(data | cluster_ids,
len(diff))
# Indices of new refcount blocks necessary to cover clusters
# in 'new'
- diff = set([x / size for x in new]) - block_ids
+ diff = set([x // size for x in new]) - block_ids
cluster_ids |= new
block_ids |= diff
return cluster_ids, block_ids
@@ -359,7 +359,7 @@ class Image(object):
blocks = set(init_blocks)
clusters = set()
# Number of entries in one cluster of the refcount table
- size = self.cluster_size / UINT64_S
+ size = self.cluster_size // UINT64_S
# Number of clusters necessary for the refcount table based on
# the current number of refcount blocks
table_size = int(ceil((max(blocks) + 1) / float(size)))
@@ -373,7 +373,7 @@ class Image(object):
table_size + 1))
# New refcount blocks necessary for clusters occupied by the
# refcount table
- diff = set([c / block_size for c in table_clusters]) - blocks
+ diff = set([c // block_size for c in table_clusters]) - blocks
blocks |= diff
while len(diff) != 0:
# Allocate clusters for new refcount blocks
@@ -382,12 +382,12 @@ class Image(object):
len(diff))
# Indices of new refcount blocks necessary to cover
# clusters in 'new'
- diff = set([x / block_size for x in new]) - blocks
+ diff = set([x // block_size for x in new]) - blocks
clusters |= new
blocks |= diff
# Check if the refcount table needs one more cluster
if int(ceil((max(blocks) + 1) / float(size))) > table_size:
- new_block_id = (table_start + table_size) / block_size
+ new_block_id = (table_start + table_size) // block_size
# Check if the additional table cluster needs
# one more refcount block
if new_block_id not in blocks:
@@ -399,13 +399,13 @@ class Image(object):
def create_table_entry(table_offset, block_cluster, block_size,
cluster):
"""Generate a refcount table entry."""
- offset = table_offset + UINT64_S * (cluster / block_size)
+ offset = table_offset + UINT64_S * (cluster // block_size)
return ['>Q', offset, block_cluster * self.cluster_size,
'refcount_table_entry']
def create_block_entry(block_cluster, block_size, cluster):
"""Generate a list of entries for the current block."""
- entry_size = self.cluster_size / block_size
+ entry_size = self.cluster_size // block_size
offset = block_cluster * self.cluster_size
entry_offset = offset + entry_size * (cluster % block_size)
# While snapshots are not supported all refcounts are set to 1
@@ -415,7 +415,7 @@ class Image(object):
# Number of refcount entries per refcount block
# Convert self.cluster_size from bytes to bits to have the same
# base for the numerator and denominator
- block_size = self.cluster_size * 8 / refcount_bits
+ block_size = self.cluster_size * 8 // refcount_bits
meta_data = self._get_metadata()
if len(self.data_clusters) == 0:
# All metadata for an empty guest image needs 4 clusters:
@@ -452,8 +452,8 @@ class Image(object):
rfc_blocks = []
for cluster in sorted(self.data_clusters | meta_data):
- if cluster / block_size != block_id:
- block_id = cluster / block_size
+ if cluster // block_size != block_id:
+ block_id = cluster // block_size
block_cluster = block_clusters[block_ids.index(block_id)]
rfc_table.append(create_table_entry(table_offset,
block_cluster,
@@ -587,7 +587,7 @@ class Image(object):
def _alloc_data(img_size, cluster_size):
"""Return a set of random indices of clusters allocated for guest data.
"""
- num_of_cls = img_size/cluster_size
+ num_of_cls = img_size // cluster_size
return set(random.sample(range(1, num_of_cls + 1),
random.randint(0, num_of_cls)))
@@ -595,7 +595,7 @@ class Image(object):
"""Return indices of clusters allocated for image metadata."""
ids = set()
for x in self:
- ids.add(x.offset/self.cluster_size)
+ ids.add(x.offset // self.cluster_size)
return ids
--
2.23.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PULL 04/11] image-fuzzer: Use io.StringIO
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
` (2 preceding siblings ...)
2019-11-05 15:43 ` [PULL 03/11] image-fuzzer: Explicitly use integer division operator Stefan Hajnoczi
@ 2019-11-05 15:43 ` Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 05/11] image-fuzzer: Use %r for all fiels at Field.__repr__() Stefan Hajnoczi
` (7 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Eduardo Habkost, qemu-block, John Snow,
Stefan Hajnoczi, Philippe Mathieu-Daudé
From: Eduardo Habkost <ehabkost@redhat.com>
StringIO.StringIO is not available on Python 3, but io.StringIO
is available on both Python 2 and 3. io.StringIO is slightly
different from the Python 2 StringIO module, though, so we need
bytes coming from subprocess.Popen() to be explicitly decoded.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-5-ehabkost@redhat.com
Message-Id: <20191016192430.25098-5-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
tests/image-fuzzer/runner.py | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index 95d84f38f3..94cab5bd93 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -28,7 +28,7 @@ import shutil
from itertools import count
import time
import getopt
-import StringIO
+import io
import resource
try:
@@ -84,8 +84,12 @@ def run_app(fd, q_args):
try:
out, err = process.communicate()
signal.alarm(0)
- fd.write(out)
- fd.write(err)
+ # fd is a text file, so we need to decode the process output before
+ # writing to it.
+ # We could be simply using the `errors` parameter of subprocess.Popen(),
+ # but this will be possible only after migrating to Python 3
+ fd.write(out.decode(errors='replace'))
+ fd.write(err.decode(errors='replace'))
fd.flush()
return process.returncode
@@ -183,7 +187,7 @@ class TestEnv(object):
MAX_BACKING_FILE_SIZE) * (1 << 20)
cmd = self.qemu_img + ['create', '-f', backing_file_fmt,
backing_file_name, str(backing_file_size)]
- temp_log = StringIO.StringIO()
+ temp_log = io.StringIO()
retcode = run_app(temp_log, cmd)
if retcode == 0:
temp_log.close()
@@ -240,7 +244,7 @@ class TestEnv(object):
"Backing file: %s\n" \
% (self.seed, " ".join(current_cmd),
self.current_dir, backing_file_name)
- temp_log = StringIO.StringIO()
+ temp_log = io.StringIO()
try:
retcode = run_app(temp_log, current_cmd)
except OSError as e:
--
2.23.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PULL 05/11] image-fuzzer: Use %r for all fiels at Field.__repr__()
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
` (3 preceding siblings ...)
2019-11-05 15:43 ` [PULL 04/11] image-fuzzer: Use io.StringIO Stefan Hajnoczi
@ 2019-11-05 15:43 ` Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 06/11] image-fuzzer: Return bytes objects on string fuzzing functions Stefan Hajnoczi
` (6 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Eduardo Habkost, qemu-block, John Snow,
Stefan Hajnoczi, Philippe Mathieu-Daudé
From: Eduardo Habkost <ehabkost@redhat.com>
This makes the formatting code simpler, and safer if we change
the type of self.value from str to bytes.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-6-ehabkost@redhat.com
Message-Id: <20191016192430.25098-6-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
tests/image-fuzzer/qcow2/layout.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index 6501c9fd4b..0adcbd448d 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -53,8 +53,8 @@ class Field(object):
return iter([self.fmt, self.offset, self.value, self.name])
def __repr__(self):
- return "Field(fmt='%s', offset=%d, value=%s, name=%s)" % \
- (self.fmt, self.offset, str(self.value), self.name)
+ return "Field(fmt=%r, offset=%r, value=%r, name=%r)" % \
+ (self.fmt, self.offset, self.value, self.name)
class FieldsList(object):
--
2.23.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PULL 06/11] image-fuzzer: Return bytes objects on string fuzzing functions
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
` (4 preceding siblings ...)
2019-11-05 15:43 ` [PULL 05/11] image-fuzzer: Use %r for all fiels at Field.__repr__() Stefan Hajnoczi
@ 2019-11-05 15:43 ` Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 07/11] image-fuzzer: Use bytes constant for field values Stefan Hajnoczi
` (5 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Eduardo Habkost, qemu-block, John Snow,
Stefan Hajnoczi, Philippe Mathieu-Daudé
From: Eduardo Habkost <ehabkost@redhat.com>
No caller of fuzzer functions is interested in unicode string values,
so replace them with bytes sequences.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-7-ehabkost@redhat.com
Message-Id: <20191016192430.25098-7-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
tests/image-fuzzer/qcow2/fuzz.py | 42 ++++++++++++++++----------------
1 file changed, 21 insertions(+), 21 deletions(-)
diff --git a/tests/image-fuzzer/qcow2/fuzz.py b/tests/image-fuzzer/qcow2/fuzz.py
index 154dc06cc0..c58bf11005 100644
--- a/tests/image-fuzzer/qcow2/fuzz.py
+++ b/tests/image-fuzzer/qcow2/fuzz.py
@@ -36,11 +36,11 @@ UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32//4, UINT32//2 - 1,
UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64//4,
UINT64//2 - 1, UINT64//2, UINT64//2 + 1, UINT64 - 1,
UINT64]
-STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x',
- '%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n',
- '%%20x', '%%20s', '%s%s%s%s%s%s%s%s%s%s', '%p%p%p%p%p%p%p%p%p%p',
- '%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%',
- '%s x 129', '%x x 257']
+BYTES_V = [b'%s%p%x%d', b'.1024d', b'%.2049d', b'%p%p%p%p', b'%x%x%x%x',
+ b'%d%d%d%d', b'%s%s%s%s', b'%99999999999s', b'%08x', b'%%20d', b'%%20n',
+ b'%%20x', b'%%20s', b'%s%s%s%s%s%s%s%s%s%s', b'%p%p%p%p%p%p%p%p%p%p',
+ b'%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%',
+ b'%s x 129', b'%x x 257']
def random_from_intervals(intervals):
@@ -76,12 +76,12 @@ def random_bits(bit_ranges):
return val
-def truncate_string(strings, length):
- """Return strings truncated to specified length."""
- if type(strings) == list:
- return [s[:length] for s in strings]
+def truncate_bytes(sequences, length):
+ """Return sequences truncated to specified length."""
+ if type(sequences) == list:
+ return [s[:length] for s in sequences]
else:
- return strings[:length]
+ return sequences[:length]
def validator(current, pick, choices):
@@ -110,12 +110,12 @@ def bit_validator(current, bit_ranges):
return validator(current, random_bits, bit_ranges)
-def string_validator(current, strings):
- """Return a random string value from the list not equal to the current.
+def bytes_validator(current, sequences):
+ """Return a random bytes value from the list not equal to the current.
This function is useful for selection from valid values except current one.
"""
- return validator(current, random.choice, strings)
+ return validator(current, random.choice, sequences)
def selector(current, constraints, validate=int_validator):
@@ -283,9 +283,9 @@ def header_length(current):
def bf_name(current):
"""Fuzz the backing file name."""
constraints = [
- truncate_string(STRING_V, len(current))
+ truncate_bytes(BYTES_V, len(current))
]
- return selector(current, constraints, string_validator)
+ return selector(current, constraints, bytes_validator)
def ext_magic(current):
@@ -303,10 +303,10 @@ def ext_length(current):
def bf_format(current):
"""Fuzz backing file format in the corresponding header extension."""
constraints = [
- truncate_string(STRING_V, len(current)),
- truncate_string(STRING_V, (len(current) + 7) & ~7) # Fuzz padding
+ truncate_bytes(BYTES_V, len(current)),
+ truncate_bytes(BYTES_V, (len(current) + 7) & ~7) # Fuzz padding
]
- return selector(current, constraints, string_validator)
+ return selector(current, constraints, bytes_validator)
def feature_type(current):
@@ -324,10 +324,10 @@ def feature_bit_number(current):
def feature_name(current):
"""Fuzz feature name field of a feature name table header extension."""
constraints = [
- truncate_string(STRING_V, len(current)),
- truncate_string(STRING_V, 46) # Fuzz padding (field length = 46)
+ truncate_bytes(BYTES_V, len(current)),
+ truncate_bytes(BYTES_V, 46) # Fuzz padding (field length = 46)
]
- return selector(current, constraints, string_validator)
+ return selector(current, constraints, bytes_validator)
def l1_entry(current):
--
2.23.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PULL 07/11] image-fuzzer: Use bytes constant for field values
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
` (5 preceding siblings ...)
2019-11-05 15:43 ` [PULL 06/11] image-fuzzer: Return bytes objects on string fuzzing functions Stefan Hajnoczi
@ 2019-11-05 15:43 ` Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 08/11] image-fuzzer: Encode file name and file format to bytes Stefan Hajnoczi
` (4 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Eduardo Habkost, qemu-block, John Snow,
Stefan Hajnoczi, Philippe Mathieu-Daudé
From: Eduardo Habkost <ehabkost@redhat.com>
Field values are supposed to be bytes objects, not unicode
strings. Change two constants that were declared as strings.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-8-ehabkost@redhat.com
Message-Id: <20191016192430.25098-8-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
tests/image-fuzzer/qcow2/layout.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index 0adcbd448d..a0fd53c7ad 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -122,7 +122,7 @@ class Image(object):
def create_header(self, cluster_bits, backing_file_name=None):
"""Generate a random valid header."""
meta_header = [
- ['>4s', 0, "QFI\xfb", 'magic'],
+ ['>4s', 0, b"QFI\xfb", 'magic'],
['>I', 4, random.randint(2, 3), 'version'],
['>Q', 8, 0, 'backing_file_offset'],
['>I', 16, 0, 'backing_file_size'],
@@ -231,7 +231,7 @@ class Image(object):
feature_tables = []
feature_ids = []
inner_offset = self.ext_offset + ext_header_len
- feat_name = 'some cool feature'
+ feat_name = b'some cool feature'
while len(feature_tables) < num_fnt_entries * 3:
feat_type, feat_bit = gen_feat_ids()
# Remove duplicates
--
2.23.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PULL 08/11] image-fuzzer: Encode file name and file format to bytes
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
` (6 preceding siblings ...)
2019-11-05 15:43 ` [PULL 07/11] image-fuzzer: Use bytes constant for field values Stefan Hajnoczi
@ 2019-11-05 15:43 ` Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 09/11] image-fuzzer: Run using python3 Stefan Hajnoczi
` (3 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Eduardo Habkost, qemu-block, John Snow,
Stefan Hajnoczi, Philippe Mathieu-Daudé
From: Eduardo Habkost <ehabkost@redhat.com>
Callers of create_image() will pass strings as arguments, but the
Image class will expect bytes objects to be provided. Encode
them inside create_image().
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-9-ehabkost@redhat.com
Message-Id: <20191016192430.25098-9-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
tests/image-fuzzer/qcow2/layout.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index a0fd53c7ad..01bff4d05e 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -602,8 +602,8 @@ class Image(object):
def create_image(test_img_path, backing_file_name=None, backing_file_fmt=None,
fields_to_fuzz=None):
"""Create a fuzzed image and write it to the specified file."""
- image = Image(backing_file_name)
- image.set_backing_file_format(backing_file_fmt)
+ image = Image(backing_file_name.encode())
+ image.set_backing_file_format(backing_file_fmt.encode())
image.create_feature_name_table()
image.set_end_of_extension_area()
image.create_l_structures()
--
2.23.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PULL 09/11] image-fuzzer: Run using python3
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
` (7 preceding siblings ...)
2019-11-05 15:43 ` [PULL 08/11] image-fuzzer: Encode file name and file format to bytes Stefan Hajnoczi
@ 2019-11-05 15:43 ` Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 10/11] image-fuzzer: Use errors parameter of subprocess.Popen() Stefan Hajnoczi
` (2 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Eduardo Habkost, qemu-block, John Snow,
Stefan Hajnoczi, Philippe Mathieu-Daudé
From: Eduardo Habkost <ehabkost@redhat.com>
image-fuzzer is now supposed to be ready to run using Python 3.
Remove the __future__ imports and change the interpreter line to
"#!/usr/bin/env python3".
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-10-ehabkost@redhat.com
Message-Id: <20191016192430.25098-10-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
tests/image-fuzzer/qcow2/__init__.py | 1 -
tests/image-fuzzer/qcow2/layout.py | 1 -
tests/image-fuzzer/runner.py | 3 +--
3 files changed, 1 insertion(+), 4 deletions(-)
diff --git a/tests/image-fuzzer/qcow2/__init__.py b/tests/image-fuzzer/qcow2/__init__.py
index 09ef59821b..ed3af5da86 100644
--- a/tests/image-fuzzer/qcow2/__init__.py
+++ b/tests/image-fuzzer/qcow2/__init__.py
@@ -1,2 +1 @@
-from __future__ import absolute_import
from .layout import create_image
diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index 01bff4d05e..57ebe86e9a 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -16,7 +16,6 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
-from __future__ import absolute_import
import random
import struct
from . import fuzz
diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index 94cab5bd93..0793234815 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
# Tool for running fuzz tests
#
@@ -18,7 +18,6 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
-from __future__ import print_function
import sys
import os
import signal
--
2.23.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PULL 10/11] image-fuzzer: Use errors parameter of subprocess.Popen()
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
` (8 preceding siblings ...)
2019-11-05 15:43 ` [PULL 09/11] image-fuzzer: Run using python3 Stefan Hajnoczi
@ 2019-11-05 15:43 ` Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 11/11] image-fuzzer: Use OSerror.strerror instead of tuple subscript Stefan Hajnoczi
2019-11-06 16:01 ` [PULL 00/11] Block patches Peter Maydell
11 siblings, 0 replies; 13+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Eduardo Habkost, qemu-block, John Snow,
Stefan Hajnoczi, Philippe Mathieu-Daudé
From: Eduardo Habkost <ehabkost@redhat.com>
Instead of manually encoding stderr and stdout output, use
`errors` parameter of subprocess.Popen(). This will make
process.communicate() return unicode strings instead of bytes
objects.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-11-ehabkost@redhat.com
Message-Id: <20191016192430.25098-11-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
tests/image-fuzzer/runner.py | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index 0793234815..4ba5c79e13 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -79,16 +79,13 @@ def run_app(fd, q_args):
devnull = open('/dev/null', 'r+')
process = subprocess.Popen(q_args, stdin=devnull,
stdout=subprocess.PIPE,
- stderr=subprocess.PIPE)
+ stderr=subprocess.PIPE,
+ errors='replace')
try:
out, err = process.communicate()
signal.alarm(0)
- # fd is a text file, so we need to decode the process output before
- # writing to it.
- # We could be simply using the `errors` parameter of subprocess.Popen(),
- # but this will be possible only after migrating to Python 3
- fd.write(out.decode(errors='replace'))
- fd.write(err.decode(errors='replace'))
+ fd.write(out)
+ fd.write(err)
fd.flush()
return process.returncode
--
2.23.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PULL 11/11] image-fuzzer: Use OSerror.strerror instead of tuple subscript
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
` (9 preceding siblings ...)
2019-11-05 15:43 ` [PULL 10/11] image-fuzzer: Use errors parameter of subprocess.Popen() Stefan Hajnoczi
@ 2019-11-05 15:43 ` Stefan Hajnoczi
2019-11-06 16:01 ` [PULL 00/11] Block patches Peter Maydell
11 siblings, 0 replies; 13+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, John Snow, qemu-block, Stefan Hajnoczi, Eduardo Habkost
From: Eduardo Habkost <ehabkost@redhat.com>
OSError can't be used like a tuple on Python 3, so change the
code to use `e.sterror` instead of `e[1]`.
Reported-by: John Snow <jsnow@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20191021214117.18091-1-ehabkost@redhat.com
Message-Id: <20191021214117.18091-1-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
tests/image-fuzzer/runner.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index 4ba5c79e13..2fc010fd9d 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -159,7 +159,7 @@ class TestEnv(object):
os.makedirs(self.current_dir)
except OSError as e:
print("Error: The working directory '%s' cannot be used. Reason: %s"\
- % (self.work_dir, e[1]), file=sys.stderr)
+ % (self.work_dir, e.strerror), file=sys.stderr)
raise TestException
self.log = open(os.path.join(self.current_dir, "test.log"), "w")
self.parent_log = open(run_log, "a")
@@ -246,7 +246,7 @@ class TestEnv(object):
except OSError as e:
multilog("%sError: Start of '%s' failed. Reason: %s\n\n"
% (test_summary, os.path.basename(current_cmd[0]),
- e[1]),
+ e.strerror),
sys.stderr, self.log, self.parent_log)
raise TestException
--
2.23.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PULL 00/11] Block patches
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
` (10 preceding siblings ...)
2019-11-05 15:43 ` [PULL 11/11] image-fuzzer: Use OSerror.strerror instead of tuple subscript Stefan Hajnoczi
@ 2019-11-06 16:01 ` Peter Maydell
11 siblings, 0 replies; 13+ messages in thread
From: Peter Maydell @ 2019-11-06 16:01 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: QEMU Developers, Qemu-block
On Tue, 5 Nov 2019 at 15:43, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> The following changes since commit 36609b4fa36f0ac934874371874416f7533a5408:
>
> Merge remote-tracking branch 'remotes/palmer/tags/palmer-for-master-4.2-sf1' into staging (2019-11-02 17:59:03 +0000)
>
> are available in the Git repository at:
>
> https://github.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to 9fdd7860adec188ed50d2530e9a819e8d953f9bb:
>
> image-fuzzer: Use OSerror.strerror instead of tuple subscript (2019-11-05 16:36:11 +0100)
>
> ----------------------------------------------------------------
> Pull request
>
> Let's get the image fuzzer Python 3 changes merged in QEMU 4.2.
>
> ----------------------------------------------------------------
Applied, thanks.
Please update the changelog at https://wiki.qemu.org/ChangeLog/4.2
for any user-visible changes.
-- PMM
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2019-11-06 16:04 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-05 15:43 [PULL 00/11] Block patches Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 01/11] image-fuzzer: Open image files in binary mode Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 02/11] image-fuzzer: Write bytes instead of string to image file Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 03/11] image-fuzzer: Explicitly use integer division operator Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 04/11] image-fuzzer: Use io.StringIO Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 05/11] image-fuzzer: Use %r for all fiels at Field.__repr__() Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 06/11] image-fuzzer: Return bytes objects on string fuzzing functions Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 07/11] image-fuzzer: Use bytes constant for field values Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 08/11] image-fuzzer: Encode file name and file format to bytes Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 09/11] image-fuzzer: Run using python3 Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 10/11] image-fuzzer: Use errors parameter of subprocess.Popen() Stefan Hajnoczi
2019-11-05 15:43 ` [PULL 11/11] image-fuzzer: Use OSerror.strerror instead of tuple subscript Stefan Hajnoczi
2019-11-06 16:01 ` [PULL 00/11] Block patches Peter Maydell
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).