[PULL 00/11] Block patches

[PULL 00/11] Block patches

[Stefan Hajnoczi]

The following changes since commit 36609b4fa36f0ac934874371874416f7533a5408:

  Merge remote-tracking branch 'remotes/palmer/tags/palmer-for-master-4.2-sf1' into staging (2019-11-02 17:59:03 +0000)

are available in the Git repository at:

  https://github.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to 9fdd7860adec188ed50d2530e9a819e8d953f9bb:

  image-fuzzer: Use OSerror.strerror instead of tuple subscript (2019-11-05 16:36:11 +0100)

----------------------------------------------------------------
Pull request

Let's get the image fuzzer Python 3 changes merged in QEMU 4.2.

----------------------------------------------------------------

Eduardo Habkost (11):
  image-fuzzer: Open image files in binary mode
  image-fuzzer: Write bytes instead of string to image file
  image-fuzzer: Explicitly use integer division operator
  image-fuzzer: Use io.StringIO
  image-fuzzer: Use %r for all fiels at Field.__repr__()
  image-fuzzer: Return bytes objects on string fuzzing functions
  image-fuzzer: Use bytes constant for field values
  image-fuzzer: Encode file name and file format to bytes
  image-fuzzer: Run using python3
  image-fuzzer: Use errors parameter of subprocess.Popen()
  image-fuzzer: Use OSerror.strerror instead of tuple subscript

 tests/image-fuzzer/qcow2/__init__.py |  1 -
 tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
 tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
 tests/image-fuzzer/runner.py         | 16 ++++----
 4 files changed, 63 insertions(+), 65 deletions(-)

-- 
2.23.0

[PULL 01/11] image-fuzzer: Open image files in binary mode

[Stefan Hajnoczi]

From: Eduardo Habkost <ehabkost@redhat.com>

This probably never caused problems because on Linux there's no
actual newline conversion happening, but on Python 3 the
binary/text distinction is stronger and we must explicitly open
the image file in binary mode.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-2-ehabkost@redhat.com
Message-Id: <20191016192430.25098-2-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/image-fuzzer/qcow2/layout.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index 675877da96..c57418fa15 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -503,7 +503,7 @@ class Image(object):
 
     def write(self, filename):
         """Write an entire image to the file."""
-        image_file = open(filename, 'w')
+        image_file = open(filename, 'wb')
         for field in self:
             image_file.seek(field.offset)
             image_file.write(struct.pack(field.fmt, field.value))
-- 
2.23.0

[PULL 02/11] image-fuzzer: Write bytes instead of string to image file

[Stefan Hajnoczi]

From: Eduardo Habkost <ehabkost@redhat.com>

This is necessary for Python 3 compatibility.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-3-ehabkost@redhat.com
Message-Id: <20191016192430.25098-3-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/image-fuzzer/qcow2/layout.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index c57418fa15..fe273d4143 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -518,7 +518,7 @@ class Image(object):
         rounded = (size + self.cluster_size - 1) & ~(self.cluster_size - 1)
         if rounded > size:
             image_file.seek(rounded - 1)
-            image_file.write("\0")
+            image_file.write(b'\x00')
         image_file.close()
 
     @staticmethod
-- 
2.23.0

[PULL 03/11] image-fuzzer: Explicitly use integer division operator

[Stefan Hajnoczi]

From: Eduardo Habkost <ehabkost@redhat.com>

Most of the division expressions in image-fuzzer assume integer
division.  Use the // operator to keep the same behavior when we
move to Python 3.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-4-ehabkost@redhat.com
Message-Id: <20191016192430.25098-4-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/image-fuzzer/qcow2/fuzz.py   | 12 ++++-----
 tests/image-fuzzer/qcow2/layout.py | 40 +++++++++++++++---------------
 2 files changed, 26 insertions(+), 26 deletions(-)

diff --git a/tests/image-fuzzer/qcow2/fuzz.py b/tests/image-fuzzer/qcow2/fuzz.py
index abc4f0635d..154dc06cc0 100644
--- a/tests/image-fuzzer/qcow2/fuzz.py
+++ b/tests/image-fuzzer/qcow2/fuzz.py
@@ -27,14 +27,14 @@ UINT64 = 0xffffffffffffffff
 UINT32_M = 31
 UINT64_M = 63
 # Fuzz vectors
-UINT8_V = [0, 0x10, UINT8/4, UINT8/2 - 1, UINT8/2, UINT8/2 + 1, UINT8 - 1,
+UINT8_V = [0, 0x10, UINT8//4, UINT8//2 - 1, UINT8//2, UINT8//2 + 1, UINT8 - 1,
            UINT8]
-UINT16_V = [0, 0x100, 0x1000, UINT16/4, UINT16/2 - 1, UINT16/2, UINT16/2 + 1,
+UINT16_V = [0, 0x100, 0x1000, UINT16//4, UINT16//2 - 1, UINT16//2, UINT16//2 + 1,
             UINT16 - 1, UINT16]
-UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32/4, UINT32/2 - 1,
-            UINT32/2, UINT32/2 + 1, UINT32 - 1, UINT32]
-UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64/4,
-                       UINT64/2 - 1, UINT64/2, UINT64/2 + 1, UINT64 - 1,
+UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32//4, UINT32//2 - 1,
+            UINT32//2, UINT32//2 + 1, UINT32 - 1, UINT32]
+UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64//4,
+                       UINT64//2 - 1, UINT64//2, UINT64//2 + 1, UINT64 - 1,
                        UINT64]
 STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x',
             '%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n',
diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index fe273d4143..6501c9fd4b 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -253,7 +253,7 @@ class Image(object):
                 ['>I', self.ext_offset, 0x6803f857, 'ext_magic'],
                 # One feature table contains 3 fields and takes 48 bytes
                 ['>I', self.ext_offset + UINT32_S,
-                 len(feature_tables) / 3 * 48, 'ext_length']
+                 len(feature_tables) // 3 * 48, 'ext_length']
             ] + feature_tables)
             self.ext_offset = inner_offset
 
@@ -271,7 +271,7 @@ class Image(object):
         def create_l2_entry(host, guest, l2_cluster):
             """Generate one L2 entry."""
             offset = l2_cluster * self.cluster_size
-            l2_size = self.cluster_size / UINT64_S
+            l2_size = self.cluster_size // UINT64_S
             entry_offset = offset + UINT64_S * (guest % l2_size)
             cluster_descriptor = host * self.cluster_size
             if not self.header['version'][0].value == 2:
@@ -283,8 +283,8 @@ class Image(object):
 
         def create_l1_entry(l2_cluster, l1_offset, guest):
             """Generate one L1 entry."""
-            l2_size = self.cluster_size / UINT64_S
-            entry_offset = l1_offset + UINT64_S * (guest / l2_size)
+            l2_size = self.cluster_size // UINT64_S
+            entry_offset = l1_offset + UINT64_S * (guest // l2_size)
             # While snapshots are not supported bit #63 = 1
             entry_val = (1 << 63) + l2_cluster * self.cluster_size
             return ['>Q', entry_offset, entry_val, 'l1_entry']
@@ -298,11 +298,11 @@ class Image(object):
             l2 = []
         else:
             meta_data = self._get_metadata()
-            guest_clusters = random.sample(range(self.image_size /
+            guest_clusters = random.sample(range(self.image_size //
                                                  self.cluster_size),
                                            len(self.data_clusters))
             # Number of entries in a L1/L2 table
-            l_size = self.cluster_size / UINT64_S
+            l_size = self.cluster_size // UINT64_S
             # Number of clusters necessary for L1 table
             l1_size = int(ceil((max(guest_clusters) + 1) / float(l_size**2)))
             l1_start = self._get_adjacent_clusters(self.data_clusters |
@@ -318,7 +318,7 @@ class Image(object):
             # L2 entries
             l2 = []
             for host, guest in zip(self.data_clusters, guest_clusters):
-                l2_id = guest / l_size
+                l2_id = guest // l_size
                 if l2_id not in l2_ids:
                     l2_ids.append(l2_id)
                     l2_clusters.append(self._get_adjacent_clusters(
@@ -339,14 +339,14 @@ class Image(object):
         def allocate_rfc_blocks(data, size):
             """Return indices of clusters allocated for refcount blocks."""
             cluster_ids = set()
-            diff = block_ids = set([x / size for x in data])
+            diff = block_ids = set([x // size for x in data])
             while len(diff) != 0:
                 # Allocate all yet not allocated clusters
                 new = self._get_available_clusters(data | cluster_ids,
                                                    len(diff))
                 # Indices of new refcount blocks necessary to cover clusters
                 # in 'new'
-                diff = set([x / size for x in new]) - block_ids
+                diff = set([x // size for x in new]) - block_ids
                 cluster_ids |= new
                 block_ids |= diff
             return cluster_ids, block_ids
@@ -359,7 +359,7 @@ class Image(object):
             blocks = set(init_blocks)
             clusters = set()
             # Number of entries in one cluster of the refcount table
-            size = self.cluster_size / UINT64_S
+            size = self.cluster_size // UINT64_S
             # Number of clusters necessary for the refcount table based on
             # the current number of refcount blocks
             table_size = int(ceil((max(blocks) + 1) / float(size)))
@@ -373,7 +373,7 @@ class Image(object):
                                                  table_size + 1))
             # New refcount blocks necessary for clusters occupied by the
             # refcount table
-            diff = set([c / block_size for c in table_clusters]) - blocks
+            diff = set([c // block_size for c in table_clusters]) - blocks
             blocks |= diff
             while len(diff) != 0:
                 # Allocate clusters for new refcount blocks
@@ -382,12 +382,12 @@ class Image(object):
                                                    len(diff))
                 # Indices of new refcount blocks necessary to cover
                 # clusters in 'new'
-                diff = set([x / block_size for x in new]) - blocks
+                diff = set([x // block_size for x in new]) - blocks
                 clusters |= new
                 blocks |= diff
                 # Check if the refcount table needs one more cluster
                 if int(ceil((max(blocks) + 1) / float(size))) > table_size:
-                    new_block_id = (table_start + table_size) / block_size
+                    new_block_id = (table_start + table_size) // block_size
                     # Check if the additional table cluster needs
                     # one more refcount block
                     if new_block_id not in blocks:
@@ -399,13 +399,13 @@ class Image(object):
         def create_table_entry(table_offset, block_cluster, block_size,
                                cluster):
             """Generate a refcount table entry."""
-            offset = table_offset + UINT64_S * (cluster / block_size)
+            offset = table_offset + UINT64_S * (cluster // block_size)
             return ['>Q', offset, block_cluster * self.cluster_size,
                     'refcount_table_entry']
 
         def create_block_entry(block_cluster, block_size, cluster):
             """Generate a list of entries for the current block."""
-            entry_size = self.cluster_size / block_size
+            entry_size = self.cluster_size // block_size
             offset = block_cluster * self.cluster_size
             entry_offset = offset + entry_size * (cluster % block_size)
             # While snapshots are not supported all refcounts are set to 1
@@ -415,7 +415,7 @@ class Image(object):
         # Number of refcount entries per refcount block
         # Convert self.cluster_size from bytes to bits to have the same
         # base for the numerator and denominator
-        block_size = self.cluster_size * 8 / refcount_bits
+        block_size = self.cluster_size * 8 // refcount_bits
         meta_data = self._get_metadata()
         if len(self.data_clusters) == 0:
             # All metadata for an empty guest image needs 4 clusters:
@@ -452,8 +452,8 @@ class Image(object):
         rfc_blocks = []
 
         for cluster in sorted(self.data_clusters | meta_data):
-            if cluster / block_size != block_id:
-                block_id = cluster / block_size
+            if cluster // block_size != block_id:
+                block_id = cluster // block_size
                 block_cluster = block_clusters[block_ids.index(block_id)]
                 rfc_table.append(create_table_entry(table_offset,
                                                     block_cluster,
@@ -587,7 +587,7 @@ class Image(object):
     def _alloc_data(img_size, cluster_size):
         """Return a set of random indices of clusters allocated for guest data.
         """
-        num_of_cls = img_size/cluster_size
+        num_of_cls = img_size // cluster_size
         return set(random.sample(range(1, num_of_cls + 1),
                                  random.randint(0, num_of_cls)))
 
@@ -595,7 +595,7 @@ class Image(object):
         """Return indices of clusters allocated for image metadata."""
         ids = set()
         for x in self:
-            ids.add(x.offset/self.cluster_size)
+            ids.add(x.offset // self.cluster_size)
         return ids
 
 
-- 
2.23.0

[PULL 04/11] image-fuzzer: Use io.StringIO

[Stefan Hajnoczi]

From: Eduardo Habkost <ehabkost@redhat.com>

StringIO.StringIO is not available on Python 3, but io.StringIO
is available on both Python 2 and 3.  io.StringIO is slightly
different from the Python 2 StringIO module, though, so we need
bytes coming from subprocess.Popen() to be explicitly decoded.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-5-ehabkost@redhat.com
Message-Id: <20191016192430.25098-5-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/image-fuzzer/runner.py | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index 95d84f38f3..94cab5bd93 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -28,7 +28,7 @@ import shutil
 from itertools import count
 import time
 import getopt
-import StringIO
+import io
 import resource
 
 try:
@@ -84,8 +84,12 @@ def run_app(fd, q_args):
     try:
         out, err = process.communicate()
         signal.alarm(0)
-        fd.write(out)
-        fd.write(err)
+        # fd is a text file, so we need to decode the process output before
+        # writing to it.
+        # We could be simply using the `errors` parameter of subprocess.Popen(),
+        # but this will be possible only after migrating to Python 3
+        fd.write(out.decode(errors='replace'))
+        fd.write(err.decode(errors='replace'))
         fd.flush()
         return process.returncode
 
@@ -183,7 +187,7 @@ class TestEnv(object):
                                            MAX_BACKING_FILE_SIZE) * (1 << 20)
         cmd = self.qemu_img + ['create', '-f', backing_file_fmt,
                                backing_file_name, str(backing_file_size)]
-        temp_log = StringIO.StringIO()
+        temp_log = io.StringIO()
         retcode = run_app(temp_log, cmd)
         if retcode == 0:
             temp_log.close()
@@ -240,7 +244,7 @@ class TestEnv(object):
                            "Backing file: %s\n" \
                            % (self.seed, " ".join(current_cmd),
                               self.current_dir, backing_file_name)
-            temp_log = StringIO.StringIO()
+            temp_log = io.StringIO()
             try:
                 retcode = run_app(temp_log, current_cmd)
             except OSError as e:
-- 
2.23.0

[PULL 05/11] image-fuzzer: Use %r for all fiels at Field.__repr__()

[Stefan Hajnoczi]

From: Eduardo Habkost <ehabkost@redhat.com>

This makes the formatting code simpler, and safer if we change
the type of self.value from str to bytes.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-6-ehabkost@redhat.com
Message-Id: <20191016192430.25098-6-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/image-fuzzer/qcow2/layout.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index 6501c9fd4b..0adcbd448d 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -53,8 +53,8 @@ class Field(object):
         return iter([self.fmt, self.offset, self.value, self.name])
 
     def __repr__(self):
-        return "Field(fmt='%s', offset=%d, value=%s, name=%s)" % \
-            (self.fmt, self.offset, str(self.value), self.name)
+        return "Field(fmt=%r, offset=%r, value=%r, name=%r)" % \
+            (self.fmt, self.offset, self.value, self.name)
 
 
 class FieldsList(object):
-- 
2.23.0

[PULL 06/11] image-fuzzer: Return bytes objects on string fuzzing functions

[Stefan Hajnoczi]

From: Eduardo Habkost <ehabkost@redhat.com>

No caller of fuzzer functions is interested in unicode string values,
so replace them with bytes sequences.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-7-ehabkost@redhat.com
Message-Id: <20191016192430.25098-7-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/image-fuzzer/qcow2/fuzz.py | 42 ++++++++++++++++----------------
 1 file changed, 21 insertions(+), 21 deletions(-)

diff --git a/tests/image-fuzzer/qcow2/fuzz.py b/tests/image-fuzzer/qcow2/fuzz.py
index 154dc06cc0..c58bf11005 100644
--- a/tests/image-fuzzer/qcow2/fuzz.py
+++ b/tests/image-fuzzer/qcow2/fuzz.py
@@ -36,11 +36,11 @@ UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32//4, UINT32//2 - 1,
 UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64//4,
                        UINT64//2 - 1, UINT64//2, UINT64//2 + 1, UINT64 - 1,
                        UINT64]
-STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x',
-            '%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n',
-            '%%20x', '%%20s', '%s%s%s%s%s%s%s%s%s%s', '%p%p%p%p%p%p%p%p%p%p',
-            '%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%',
-            '%s x 129', '%x x 257']
+BYTES_V = [b'%s%p%x%d', b'.1024d', b'%.2049d', b'%p%p%p%p', b'%x%x%x%x',
+           b'%d%d%d%d', b'%s%s%s%s', b'%99999999999s', b'%08x', b'%%20d', b'%%20n',
+           b'%%20x', b'%%20s', b'%s%s%s%s%s%s%s%s%s%s', b'%p%p%p%p%p%p%p%p%p%p',
+           b'%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%',
+           b'%s x 129', b'%x x 257']
 
 
 def random_from_intervals(intervals):
@@ -76,12 +76,12 @@ def random_bits(bit_ranges):
     return val
 
 
-def truncate_string(strings, length):
-    """Return strings truncated to specified length."""
-    if type(strings) == list:
-        return [s[:length] for s in strings]
+def truncate_bytes(sequences, length):
+    """Return sequences truncated to specified length."""
+    if type(sequences) == list:
+        return [s[:length] for s in sequences]
     else:
-        return strings[:length]
+        return sequences[:length]
 
 
 def validator(current, pick, choices):
@@ -110,12 +110,12 @@ def bit_validator(current, bit_ranges):
     return validator(current, random_bits, bit_ranges)
 
 
-def string_validator(current, strings):
-    """Return a random string value from the list not equal to the current.
+def bytes_validator(current, sequences):
+    """Return a random bytes value from the list not equal to the current.
 
     This function is useful for selection from valid values except current one.
     """
-    return validator(current, random.choice, strings)
+    return validator(current, random.choice, sequences)
 
 
 def selector(current, constraints, validate=int_validator):
@@ -283,9 +283,9 @@ def header_length(current):
 def bf_name(current):
     """Fuzz the backing file name."""
     constraints = [
-        truncate_string(STRING_V, len(current))
+        truncate_bytes(BYTES_V, len(current))
     ]
-    return selector(current, constraints, string_validator)
+    return selector(current, constraints, bytes_validator)
 
 
 def ext_magic(current):
@@ -303,10 +303,10 @@ def ext_length(current):
 def bf_format(current):
     """Fuzz backing file format in the corresponding header extension."""
     constraints = [
-        truncate_string(STRING_V, len(current)),
-        truncate_string(STRING_V, (len(current) + 7) & ~7)  # Fuzz padding
+        truncate_bytes(BYTES_V, len(current)),
+        truncate_bytes(BYTES_V, (len(current) + 7) & ~7)  # Fuzz padding
     ]
-    return selector(current, constraints, string_validator)
+    return selector(current, constraints, bytes_validator)
 
 
 def feature_type(current):
@@ -324,10 +324,10 @@ def feature_bit_number(current):
 def feature_name(current):
     """Fuzz feature name field of a feature name table header extension."""
     constraints = [
-        truncate_string(STRING_V, len(current)),
-        truncate_string(STRING_V, 46)  # Fuzz padding (field length = 46)
+        truncate_bytes(BYTES_V, len(current)),
+        truncate_bytes(BYTES_V, 46)  # Fuzz padding (field length = 46)
     ]
-    return selector(current, constraints, string_validator)
+    return selector(current, constraints, bytes_validator)
 
 
 def l1_entry(current):
-- 
2.23.0

[PULL 07/11] image-fuzzer: Use bytes constant for field values

[Stefan Hajnoczi]

From: Eduardo Habkost <ehabkost@redhat.com>

Field values are supposed to be bytes objects, not unicode
strings.  Change two constants that were declared as strings.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-8-ehabkost@redhat.com
Message-Id: <20191016192430.25098-8-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/image-fuzzer/qcow2/layout.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index 0adcbd448d..a0fd53c7ad 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -122,7 +122,7 @@ class Image(object):
     def create_header(self, cluster_bits, backing_file_name=None):
         """Generate a random valid header."""
         meta_header = [
-            ['>4s', 0, "QFI\xfb", 'magic'],
+            ['>4s', 0, b"QFI\xfb", 'magic'],
             ['>I', 4, random.randint(2, 3), 'version'],
             ['>Q', 8, 0, 'backing_file_offset'],
             ['>I', 16, 0, 'backing_file_size'],
@@ -231,7 +231,7 @@ class Image(object):
             feature_tables = []
             feature_ids = []
             inner_offset = self.ext_offset + ext_header_len
-            feat_name = 'some cool feature'
+            feat_name = b'some cool feature'
             while len(feature_tables) < num_fnt_entries * 3:
                 feat_type, feat_bit = gen_feat_ids()
                 # Remove duplicates
-- 
2.23.0

[PULL 08/11] image-fuzzer: Encode file name and file format to bytes

[Stefan Hajnoczi]

From: Eduardo Habkost <ehabkost@redhat.com>

Callers of create_image() will pass strings as arguments, but the
Image class will expect bytes objects to be provided.  Encode
them inside create_image().

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-9-ehabkost@redhat.com
Message-Id: <20191016192430.25098-9-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/image-fuzzer/qcow2/layout.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index a0fd53c7ad..01bff4d05e 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -602,8 +602,8 @@ class Image(object):
 def create_image(test_img_path, backing_file_name=None, backing_file_fmt=None,
                  fields_to_fuzz=None):
     """Create a fuzzed image and write it to the specified file."""
-    image = Image(backing_file_name)
-    image.set_backing_file_format(backing_file_fmt)
+    image = Image(backing_file_name.encode())
+    image.set_backing_file_format(backing_file_fmt.encode())
     image.create_feature_name_table()
     image.set_end_of_extension_area()
     image.create_l_structures()
-- 
2.23.0

[PULL 09/11] image-fuzzer: Run using python3

[Stefan Hajnoczi]

From: Eduardo Habkost <ehabkost@redhat.com>

image-fuzzer is now supposed to be ready to run using Python 3.
Remove the __future__ imports and change the interpreter line to
"#!/usr/bin/env python3".

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-10-ehabkost@redhat.com
Message-Id: <20191016192430.25098-10-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/image-fuzzer/qcow2/__init__.py | 1 -
 tests/image-fuzzer/qcow2/layout.py   | 1 -
 tests/image-fuzzer/runner.py         | 3 +--
 3 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/tests/image-fuzzer/qcow2/__init__.py b/tests/image-fuzzer/qcow2/__init__.py
index 09ef59821b..ed3af5da86 100644
--- a/tests/image-fuzzer/qcow2/__init__.py
+++ b/tests/image-fuzzer/qcow2/__init__.py
@@ -1,2 +1 @@
-from __future__ import absolute_import
 from .layout import create_image
diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index 01bff4d05e..57ebe86e9a 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -16,7 +16,6 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from __future__ import absolute_import
 import random
 import struct
 from . import fuzz
diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index 94cab5bd93..0793234815 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Tool for running fuzz tests
 #
@@ -18,7 +18,6 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from __future__ import print_function
 import sys
 import os
 import signal
-- 
2.23.0

[PULL 10/11] image-fuzzer: Use errors parameter of subprocess.Popen()

[Stefan Hajnoczi]

From: Eduardo Habkost <ehabkost@redhat.com>

Instead of manually encoding stderr and stdout output, use
`errors` parameter of subprocess.Popen().  This will make
process.communicate() return unicode strings instead of bytes
objects.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-11-ehabkost@redhat.com
Message-Id: <20191016192430.25098-11-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/image-fuzzer/runner.py | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index 0793234815..4ba5c79e13 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -79,16 +79,13 @@ def run_app(fd, q_args):
     devnull = open('/dev/null', 'r+')
     process = subprocess.Popen(q_args, stdin=devnull,
                                stdout=subprocess.PIPE,
-                               stderr=subprocess.PIPE)
+                               stderr=subprocess.PIPE,
+                               errors='replace')
     try:
         out, err = process.communicate()
         signal.alarm(0)
-        # fd is a text file, so we need to decode the process output before
-        # writing to it.
-        # We could be simply using the `errors` parameter of subprocess.Popen(),
-        # but this will be possible only after migrating to Python 3
-        fd.write(out.decode(errors='replace'))
-        fd.write(err.decode(errors='replace'))
+        fd.write(out)
+        fd.write(err)
         fd.flush()
         return process.returncode
 
-- 
2.23.0

[PULL 11/11] image-fuzzer: Use OSerror.strerror instead of tuple subscript

[Stefan Hajnoczi]

From: Eduardo Habkost <ehabkost@redhat.com>

OSError can't be used like a tuple on Python 3, so change the
code to use `e.sterror` instead of `e[1]`.

Reported-by: John Snow <jsnow@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20191021214117.18091-1-ehabkost@redhat.com
Message-Id: <20191021214117.18091-1-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/image-fuzzer/runner.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index 4ba5c79e13..2fc010fd9d 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -159,7 +159,7 @@ class TestEnv(object):
             os.makedirs(self.current_dir)
         except OSError as e:
             print("Error: The working directory '%s' cannot be used. Reason: %s"\
-                % (self.work_dir, e[1]), file=sys.stderr)
+                % (self.work_dir, e.strerror), file=sys.stderr)
             raise TestException
         self.log = open(os.path.join(self.current_dir, "test.log"), "w")
         self.parent_log = open(run_log, "a")
@@ -246,7 +246,7 @@ class TestEnv(object):
             except OSError as e:
                 multilog("%sError: Start of '%s' failed. Reason: %s\n\n"
                          % (test_summary, os.path.basename(current_cmd[0]),
-                            e[1]),
+                            e.strerror),
                          sys.stderr, self.log, self.parent_log)
                 raise TestException
 
-- 
2.23.0

Re: [PULL 00/11] Block patches

[Peter Maydell]

On Tue, 5 Nov 2019 at 15:43, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> The following changes since commit 36609b4fa36f0ac934874371874416f7533a5408:
>
>   Merge remote-tracking branch 'remotes/palmer/tags/palmer-for-master-4.2-sf1' into staging (2019-11-02 17:59:03 +0000)
>
> are available in the Git repository at:
>
>   https://github.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to 9fdd7860adec188ed50d2530e9a819e8d953f9bb:
>
>   image-fuzzer: Use OSerror.strerror instead of tuple subscript (2019-11-05 16:36:11 +0100)
>
> ----------------------------------------------------------------
> Pull request
>
> Let's get the image fuzzer Python 3 changes merged in QEMU 4.2.
>
> ----------------------------------------------------------------


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/4.2
for any user-visible changes.

-- PMM

[PULL 00/11] Block patches

[Stefan Hajnoczi]

The following changes since commit 66234fee9c2d37bfbc523aa8d0ae5300a14cc10e:

  Merge remote-tracking branch 'remotes/alistair/tags/pull-riscv-to-apply-202=
00603' into staging (2020-06-04 11:38:48 +0100)

are available in the Git repository at:

  https://github.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to 7d2410cea154bf915fb30179ebda3b17ac36e70e:

  block: Factor out bdrv_run_co() (2020-06-05 09:54:48 +0100)

----------------------------------------------------------------
Pull request

----------------------------------------------------------------

Alexander Bulekov (4):
  fuzz: add datadir for oss-fuzz compatability
  fuzz: fix typo in i440fx-qtest-reboot arguments
  fuzz: add mangled object name to linker script
  fuzz: run the main-loop in fork-server process

Philippe Mathieu-Daud=C3=A9 (4):
  memory: Rename memory_region_do_writeback -> memory_region_writeback
  memory: Extract memory_region_msync() from memory_region_writeback()
  hw/block: Let the NVMe emulated device be target-agnostic
  exec: Rename qemu_ram_writeback() as qemu_ram_msync()

Stefano Garzarella (2):
  io_uring: retry io_uring_submit() if it fails with errno=3DEINTR
  io_uring: use io_uring_cq_ready() to check for ready cqes

Vladimir Sementsov-Ogievskiy (1):
  block: Factor out bdrv_run_co()

 hw/block/Makefile.objs              |   2 +-
 include/exec/memory.h               |  15 ++-
 include/exec/ram_addr.h             |   4 +-
 include/sysemu/sysemu.h             |   2 +
 block/io.c                          | 193 +++++++++++-----------------
 block/io_uring.c                    |  11 +-
 exec.c                              |   2 +-
 hw/block/nvme.c                     |   6 +-
 memory.c                            |  12 +-
 softmmu/vl.c                        |   2 +-
 target/arm/helper.c                 |   2 +-
 tests/qtest/fuzz/fuzz.c             |  15 +++
 tests/qtest/fuzz/i440fx_fuzz.c      |   3 +-
 tests/qtest/fuzz/virtio_net_fuzz.c  |   2 +
 tests/qtest/fuzz/virtio_scsi_fuzz.c |   2 +
 tests/qtest/fuzz/fork_fuzz.ld       |   5 +
 16 files changed, 134 insertions(+), 144 deletions(-)

--=20
2.25.4

Re: [PULL 00/11] Block patches

[Peter Maydell]

On Fri, 5 Jun 2020 at 10:48, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> The following changes since commit 66234fee9c2d37bfbc523aa8d0ae5300a14cc10e:
>
>   Merge remote-tracking branch 'remotes/alistair/tags/pull-riscv-to-apply-202=
> 00603' into staging (2020-06-04 11:38:48 +0100)
>
> are available in the Git repository at:
>
>   https://github.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to 7d2410cea154bf915fb30179ebda3b17ac36e70e:
>
>   block: Factor out bdrv_run_co() (2020-06-05 09:54:48 +0100)
>
> ----------------------------------------------------------------
> Pull request
>
> ----------------------------------------------------------------


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/5.1
for any user-visible changes.

-- PMM

[PULL 00/11] Block patches

[Stefan Hajnoczi]

The following changes since commit 88afdc92b644120e0182c8567e1b1d236e471b12:

  Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging (2021-09-05 15:48:42 +0100)

are available in the Git repository at:

  https://gitlab.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to 9bd2788f49c331b02372cc257b11e4c984d39708:

  block/nvme: Only report VFIO error on failed retry (2021-09-07 09:08:24 +0100)

----------------------------------------------------------------
Pull request

Userspace NVMe driver patches.

----------------------------------------------------------------

Philippe Mathieu-Daudé (11):
  block/nvme: Use safer trace format string
  util/vfio-helpers: Let qemu_vfio_verify_mappings() use error_report()
  util/vfio-helpers: Replace qemu_mutex_lock() calls with
    QEMU_LOCK_GUARD
  util/vfio-helpers: Remove unreachable code in qemu_vfio_dma_map()
  block/nvme: Have nvme_create_queue_pair() report errors consistently
  util/vfio-helpers: Pass Error handle to qemu_vfio_dma_map()
  util/vfio-helpers: Extract qemu_vfio_water_mark_reached()
  util/vfio-helpers: Use error_setg in qemu_vfio_find_[fixed/temp]_iova
  util/vfio-helpers: Simplify qemu_vfio_dma_map() returning directly
  util/vfio-helpers: Let qemu_vfio_do_mapping() propagate Error
  block/nvme: Only report VFIO error on failed retry

 include/qemu/vfio-helpers.h |  2 +-
 block/nvme.c                | 29 +++++++----
 util/vfio-helpers.c         | 99 ++++++++++++++++++++-----------------
 block/trace-events          |  2 +-
 4 files changed, 76 insertions(+), 56 deletions(-)

-- 
2.31.1

Re: [PULL 00/11] Block patches

[Peter Maydell]

On Tue, 7 Sept 2021 at 09:14, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> The following changes since commit 88afdc92b644120e0182c8567e1b1d236e471b12:
>
>   Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging (2021-09-05 15:48:42 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to 9bd2788f49c331b02372cc257b11e4c984d39708:
>
>   block/nvme: Only report VFIO error on failed retry (2021-09-07 09:08:24 +0100)
>
> ----------------------------------------------------------------
> Pull request
>
> Userspace NVMe driver patches.


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/6.2
for any user-visible changes.

-- PMM