[PATCH 00/10] image-fuzzer: Port to Python 3

[PATCH 00/10] image-fuzzer: Port to Python 3

[Eduardo Habkost]

This series ports image-fuzzer to Python 3.

Eduardo Habkost (10):
  image-fuzzer: Open image files in binary mode
  image-fuzzer: Write bytes instead of string to image file
  image-fuzzer: Explicitly use integer division operator
  image-fuzzer: Use io.StringIO
  image-fuzzer: Use %r for all fiels at Field.__repr__()
  image-fuzzer: Return bytes objects on string fuzzing functions
  image-fuzzer: Use bytes constant for field values
  image-fuzzer: Encode file name and file format to bytes
  image-fuzzer: Run using python3
  image-fuzzer: Use errors parameter of subprocess.Popen()

 tests/image-fuzzer/qcow2/__init__.py |  1 -
 tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
 tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
 tests/image-fuzzer/runner.py         | 12 +++---
 4 files changed, 61 insertions(+), 63 deletions(-)

-- 
2.21.0

[PATCH 01/10] image-fuzzer: Open image files in binary mode

[Eduardo Habkost]

This probably never caused problems because on Linux there's no
actual newline conversion happening, but on Python 3 the
binary/text distinction is stronger and we must explicitly open
the image file in binary mode.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/image-fuzzer/qcow2/layout.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index 675877da96..c57418fa15 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -503,7 +503,7 @@ class Image(object):
 
     def write(self, filename):
         """Write an entire image to the file."""
-        image_file = open(filename, 'w')
+        image_file = open(filename, 'wb')
         for field in self:
             image_file.seek(field.offset)
             image_file.write(struct.pack(field.fmt, field.value))
-- 
2.21.0

Re: [PATCH 01/10] image-fuzzer: Open image files in binary mode

[Philippe Mathieu-Daudé]

On 10/16/19 9:24 PM, Eduardo Habkost wrote:
> This probably never caused problems because on Linux there's no
> actual newline conversion happening, but on Python 3 the
> binary/text distinction is stronger and we must explicitly open
> the image file in binary mode.
> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>   tests/image-fuzzer/qcow2/layout.py | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
> index 675877da96..c57418fa15 100644
> --- a/tests/image-fuzzer/qcow2/layout.py
> +++ b/tests/image-fuzzer/qcow2/layout.py
> @@ -503,7 +503,7 @@ class Image(object):
>   
>       def write(self, filename):
>           """Write an entire image to the file."""
> -        image_file = open(filename, 'w')
> +        image_file = open(filename, 'wb')
>           for field in self:
>               image_file.seek(field.offset)
>               image_file.write(struct.pack(field.fmt, field.value))
> 

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

[PATCH 02/10] image-fuzzer: Write bytes instead of string to image file

[Eduardo Habkost]

This is necessary for Python 3 compatibility.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/image-fuzzer/qcow2/layout.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index c57418fa15..fe273d4143 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -518,7 +518,7 @@ class Image(object):
         rounded = (size + self.cluster_size - 1) & ~(self.cluster_size - 1)
         if rounded > size:
             image_file.seek(rounded - 1)
-            image_file.write("\0")
+            image_file.write(b'\x00')
         image_file.close()
 
     @staticmethod
-- 
2.21.0

Re: [PATCH 02/10] image-fuzzer: Write bytes instead of string to image file

[Philippe Mathieu-Daudé]

On 10/16/19 9:24 PM, Eduardo Habkost wrote:
> This is necessary for Python 3 compatibility.
> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>   tests/image-fuzzer/qcow2/layout.py | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
> index c57418fa15..fe273d4143 100644
> --- a/tests/image-fuzzer/qcow2/layout.py
> +++ b/tests/image-fuzzer/qcow2/layout.py
> @@ -518,7 +518,7 @@ class Image(object):
>           rounded = (size + self.cluster_size - 1) & ~(self.cluster_size - 1)
>           if rounded > size:
>               image_file.seek(rounded - 1)
> -            image_file.write("\0")
> +            image_file.write(b'\x00')
>           image_file.close()
>   
>       @staticmethod
> 

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

[PATCH 03/10] image-fuzzer: Explicitly use integer division operator

[Eduardo Habkost]

Most of the division expressions in image-fuzzer assume integer
division.  Use the // operator to keep the same behavior when we
move to Python 3.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/image-fuzzer/qcow2/fuzz.py   | 12 ++++-----
 tests/image-fuzzer/qcow2/layout.py | 40 +++++++++++++++---------------
 2 files changed, 26 insertions(+), 26 deletions(-)

diff --git a/tests/image-fuzzer/qcow2/fuzz.py b/tests/image-fuzzer/qcow2/fuzz.py
index abc4f0635d..154dc06cc0 100644
--- a/tests/image-fuzzer/qcow2/fuzz.py
+++ b/tests/image-fuzzer/qcow2/fuzz.py
@@ -27,14 +27,14 @@ UINT64 = 0xffffffffffffffff
 UINT32_M = 31
 UINT64_M = 63
 # Fuzz vectors
-UINT8_V = [0, 0x10, UINT8/4, UINT8/2 - 1, UINT8/2, UINT8/2 + 1, UINT8 - 1,
+UINT8_V = [0, 0x10, UINT8//4, UINT8//2 - 1, UINT8//2, UINT8//2 + 1, UINT8 - 1,
            UINT8]
-UINT16_V = [0, 0x100, 0x1000, UINT16/4, UINT16/2 - 1, UINT16/2, UINT16/2 + 1,
+UINT16_V = [0, 0x100, 0x1000, UINT16//4, UINT16//2 - 1, UINT16//2, UINT16//2 + 1,
             UINT16 - 1, UINT16]
-UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32/4, UINT32/2 - 1,
-            UINT32/2, UINT32/2 + 1, UINT32 - 1, UINT32]
-UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64/4,
-                       UINT64/2 - 1, UINT64/2, UINT64/2 + 1, UINT64 - 1,
+UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32//4, UINT32//2 - 1,
+            UINT32//2, UINT32//2 + 1, UINT32 - 1, UINT32]
+UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64//4,
+                       UINT64//2 - 1, UINT64//2, UINT64//2 + 1, UINT64 - 1,
                        UINT64]
 STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x',
             '%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n',
diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index fe273d4143..6501c9fd4b 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -253,7 +253,7 @@ class Image(object):
                 ['>I', self.ext_offset, 0x6803f857, 'ext_magic'],
                 # One feature table contains 3 fields and takes 48 bytes
                 ['>I', self.ext_offset + UINT32_S,
-                 len(feature_tables) / 3 * 48, 'ext_length']
+                 len(feature_tables) // 3 * 48, 'ext_length']
             ] + feature_tables)
             self.ext_offset = inner_offset
 
@@ -271,7 +271,7 @@ class Image(object):
         def create_l2_entry(host, guest, l2_cluster):
             """Generate one L2 entry."""
             offset = l2_cluster * self.cluster_size
-            l2_size = self.cluster_size / UINT64_S
+            l2_size = self.cluster_size // UINT64_S
             entry_offset = offset + UINT64_S * (guest % l2_size)
             cluster_descriptor = host * self.cluster_size
             if not self.header['version'][0].value == 2:
@@ -283,8 +283,8 @@ class Image(object):
 
         def create_l1_entry(l2_cluster, l1_offset, guest):
             """Generate one L1 entry."""
-            l2_size = self.cluster_size / UINT64_S
-            entry_offset = l1_offset + UINT64_S * (guest / l2_size)
+            l2_size = self.cluster_size // UINT64_S
+            entry_offset = l1_offset + UINT64_S * (guest // l2_size)
             # While snapshots are not supported bit #63 = 1
             entry_val = (1 << 63) + l2_cluster * self.cluster_size
             return ['>Q', entry_offset, entry_val, 'l1_entry']
@@ -298,11 +298,11 @@ class Image(object):
             l2 = []
         else:
             meta_data = self._get_metadata()
-            guest_clusters = random.sample(range(self.image_size /
+            guest_clusters = random.sample(range(self.image_size //
                                                  self.cluster_size),
                                            len(self.data_clusters))
             # Number of entries in a L1/L2 table
-            l_size = self.cluster_size / UINT64_S
+            l_size = self.cluster_size // UINT64_S
             # Number of clusters necessary for L1 table
             l1_size = int(ceil((max(guest_clusters) + 1) / float(l_size**2)))
             l1_start = self._get_adjacent_clusters(self.data_clusters |
@@ -318,7 +318,7 @@ class Image(object):
             # L2 entries
             l2 = []
             for host, guest in zip(self.data_clusters, guest_clusters):
-                l2_id = guest / l_size
+                l2_id = guest // l_size
                 if l2_id not in l2_ids:
                     l2_ids.append(l2_id)
                     l2_clusters.append(self._get_adjacent_clusters(
@@ -339,14 +339,14 @@ class Image(object):
         def allocate_rfc_blocks(data, size):
             """Return indices of clusters allocated for refcount blocks."""
             cluster_ids = set()
-            diff = block_ids = set([x / size for x in data])
+            diff = block_ids = set([x // size for x in data])
             while len(diff) != 0:
                 # Allocate all yet not allocated clusters
                 new = self._get_available_clusters(data | cluster_ids,
                                                    len(diff))
                 # Indices of new refcount blocks necessary to cover clusters
                 # in 'new'
-                diff = set([x / size for x in new]) - block_ids
+                diff = set([x // size for x in new]) - block_ids
                 cluster_ids |= new
                 block_ids |= diff
             return cluster_ids, block_ids
@@ -359,7 +359,7 @@ class Image(object):
             blocks = set(init_blocks)
             clusters = set()
             # Number of entries in one cluster of the refcount table
-            size = self.cluster_size / UINT64_S
+            size = self.cluster_size // UINT64_S
             # Number of clusters necessary for the refcount table based on
             # the current number of refcount blocks
             table_size = int(ceil((max(blocks) + 1) / float(size)))
@@ -373,7 +373,7 @@ class Image(object):
                                                  table_size + 1))
             # New refcount blocks necessary for clusters occupied by the
             # refcount table
-            diff = set([c / block_size for c in table_clusters]) - blocks
+            diff = set([c // block_size for c in table_clusters]) - blocks
             blocks |= diff
             while len(diff) != 0:
                 # Allocate clusters for new refcount blocks
@@ -382,12 +382,12 @@ class Image(object):
                                                    len(diff))
                 # Indices of new refcount blocks necessary to cover
                 # clusters in 'new'
-                diff = set([x / block_size for x in new]) - blocks
+                diff = set([x // block_size for x in new]) - blocks
                 clusters |= new
                 blocks |= diff
                 # Check if the refcount table needs one more cluster
                 if int(ceil((max(blocks) + 1) / float(size))) > table_size:
-                    new_block_id = (table_start + table_size) / block_size
+                    new_block_id = (table_start + table_size) // block_size
                     # Check if the additional table cluster needs
                     # one more refcount block
                     if new_block_id not in blocks:
@@ -399,13 +399,13 @@ class Image(object):
         def create_table_entry(table_offset, block_cluster, block_size,
                                cluster):
             """Generate a refcount table entry."""
-            offset = table_offset + UINT64_S * (cluster / block_size)
+            offset = table_offset + UINT64_S * (cluster // block_size)
             return ['>Q', offset, block_cluster * self.cluster_size,
                     'refcount_table_entry']
 
         def create_block_entry(block_cluster, block_size, cluster):
             """Generate a list of entries for the current block."""
-            entry_size = self.cluster_size / block_size
+            entry_size = self.cluster_size // block_size
             offset = block_cluster * self.cluster_size
             entry_offset = offset + entry_size * (cluster % block_size)
             # While snapshots are not supported all refcounts are set to 1
@@ -415,7 +415,7 @@ class Image(object):
         # Number of refcount entries per refcount block
         # Convert self.cluster_size from bytes to bits to have the same
         # base for the numerator and denominator
-        block_size = self.cluster_size * 8 / refcount_bits
+        block_size = self.cluster_size * 8 // refcount_bits
         meta_data = self._get_metadata()
         if len(self.data_clusters) == 0:
             # All metadata for an empty guest image needs 4 clusters:
@@ -452,8 +452,8 @@ class Image(object):
         rfc_blocks = []
 
         for cluster in sorted(self.data_clusters | meta_data):
-            if cluster / block_size != block_id:
-                block_id = cluster / block_size
+            if cluster // block_size != block_id:
+                block_id = cluster // block_size
                 block_cluster = block_clusters[block_ids.index(block_id)]
                 rfc_table.append(create_table_entry(table_offset,
                                                     block_cluster,
@@ -587,7 +587,7 @@ class Image(object):
     def _alloc_data(img_size, cluster_size):
         """Return a set of random indices of clusters allocated for guest data.
         """
-        num_of_cls = img_size/cluster_size
+        num_of_cls = img_size // cluster_size
         return set(random.sample(range(1, num_of_cls + 1),
                                  random.randint(0, num_of_cls)))
 
@@ -595,7 +595,7 @@ class Image(object):
         """Return indices of clusters allocated for image metadata."""
         ids = set()
         for x in self:
-            ids.add(x.offset/self.cluster_size)
+            ids.add(x.offset // self.cluster_size)
         return ids
 
 
-- 
2.21.0

[PATCH 04/10] image-fuzzer: Use io.StringIO

[Eduardo Habkost]

StringIO.StringIO is not available on Python 3, but io.StringIO
is available on both Python 2 and 3.  io.StringIO is slightly
different from the Python 2 StringIO module, though, so we need
bytes coming from subprocess.Popen() to be explicitly decoded.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/image-fuzzer/runner.py | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index 95d84f38f3..94cab5bd93 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -28,7 +28,7 @@ import shutil
 from itertools import count
 import time
 import getopt
-import StringIO
+import io
 import resource
 
 try:
@@ -84,8 +84,12 @@ def run_app(fd, q_args):
     try:
         out, err = process.communicate()
         signal.alarm(0)
-        fd.write(out)
-        fd.write(err)
+        # fd is a text file, so we need to decode the process output before
+        # writing to it.
+        # We could be simply using the `errors` parameter of subprocess.Popen(),
+        # but this will be possible only after migrating to Python 3
+        fd.write(out.decode(errors='replace'))
+        fd.write(err.decode(errors='replace'))
         fd.flush()
         return process.returncode
 
@@ -183,7 +187,7 @@ class TestEnv(object):
                                            MAX_BACKING_FILE_SIZE) * (1 << 20)
         cmd = self.qemu_img + ['create', '-f', backing_file_fmt,
                                backing_file_name, str(backing_file_size)]
-        temp_log = StringIO.StringIO()
+        temp_log = io.StringIO()
         retcode = run_app(temp_log, cmd)
         if retcode == 0:
             temp_log.close()
@@ -240,7 +244,7 @@ class TestEnv(object):
                            "Backing file: %s\n" \
                            % (self.seed, " ".join(current_cmd),
                               self.current_dir, backing_file_name)
-            temp_log = StringIO.StringIO()
+            temp_log = io.StringIO()
             try:
                 retcode = run_app(temp_log, current_cmd)
             except OSError as e:
-- 
2.21.0

[PATCH 05/10] image-fuzzer: Use %r for all fiels at Field.__repr__()

[Eduardo Habkost]

This makes the formatting code simpler, and safer if we change
the type of self.value from str to bytes.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/image-fuzzer/qcow2/layout.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index 6501c9fd4b..0adcbd448d 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -53,8 +53,8 @@ class Field(object):
         return iter([self.fmt, self.offset, self.value, self.name])
 
     def __repr__(self):
-        return "Field(fmt='%s', offset=%d, value=%s, name=%s)" % \
-            (self.fmt, self.offset, str(self.value), self.name)
+        return "Field(fmt=%r, offset=%r, value=%r, name=%r)" % \
+            (self.fmt, self.offset, self.value, self.name)
 
 
 class FieldsList(object):
-- 
2.21.0

Re: [PATCH 05/10] image-fuzzer: Use %r for all fiels at Field.__repr__()

[John Snow]

"fields" in the commit message.

On 10/16/19 3:24 PM, Eduardo Habkost wrote:
> This makes the formatting code simpler, and safer if we change
> the type of self.value from str to bytes.
> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>  tests/image-fuzzer/qcow2/layout.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
> index 6501c9fd4b..0adcbd448d 100644
> --- a/tests/image-fuzzer/qcow2/layout.py
> +++ b/tests/image-fuzzer/qcow2/layout.py
> @@ -53,8 +53,8 @@ class Field(object):
>          return iter([self.fmt, self.offset, self.value, self.name])
>  
>      def __repr__(self):
> -        return "Field(fmt='%s', offset=%d, value=%s, name=%s)" % \
> -            (self.fmt, self.offset, str(self.value), self.name)
> +        return "Field(fmt=%r, offset=%r, value=%r, name=%r)" % \
> +            (self.fmt, self.offset, self.value, self.name)
>  
>  
>  class FieldsList(object):
> 

[PATCH 06/10] image-fuzzer: Return bytes objects on string fuzzing functions

[Eduardo Habkost]

No caller of fuzzer functions is interested in unicode string values,
so replace them with bytes sequences.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/image-fuzzer/qcow2/fuzz.py | 42 ++++++++++++++++----------------
 1 file changed, 21 insertions(+), 21 deletions(-)

diff --git a/tests/image-fuzzer/qcow2/fuzz.py b/tests/image-fuzzer/qcow2/fuzz.py
index 154dc06cc0..c58bf11005 100644
--- a/tests/image-fuzzer/qcow2/fuzz.py
+++ b/tests/image-fuzzer/qcow2/fuzz.py
@@ -36,11 +36,11 @@ UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32//4, UINT32//2 - 1,
 UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64//4,
                        UINT64//2 - 1, UINT64//2, UINT64//2 + 1, UINT64 - 1,
                        UINT64]
-STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x',
-            '%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n',
-            '%%20x', '%%20s', '%s%s%s%s%s%s%s%s%s%s', '%p%p%p%p%p%p%p%p%p%p',
-            '%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%',
-            '%s x 129', '%x x 257']
+BYTES_V = [b'%s%p%x%d', b'.1024d', b'%.2049d', b'%p%p%p%p', b'%x%x%x%x',
+           b'%d%d%d%d', b'%s%s%s%s', b'%99999999999s', b'%08x', b'%%20d', b'%%20n',
+           b'%%20x', b'%%20s', b'%s%s%s%s%s%s%s%s%s%s', b'%p%p%p%p%p%p%p%p%p%p',
+           b'%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%',
+           b'%s x 129', b'%x x 257']
 
 
 def random_from_intervals(intervals):
@@ -76,12 +76,12 @@ def random_bits(bit_ranges):
     return val
 
 
-def truncate_string(strings, length):
-    """Return strings truncated to specified length."""
-    if type(strings) == list:
-        return [s[:length] for s in strings]
+def truncate_bytes(sequences, length):
+    """Return sequences truncated to specified length."""
+    if type(sequences) == list:
+        return [s[:length] for s in sequences]
     else:
-        return strings[:length]
+        return sequences[:length]
 
 
 def validator(current, pick, choices):
@@ -110,12 +110,12 @@ def bit_validator(current, bit_ranges):
     return validator(current, random_bits, bit_ranges)
 
 
-def string_validator(current, strings):
-    """Return a random string value from the list not equal to the current.
+def bytes_validator(current, sequences):
+    """Return a random bytes value from the list not equal to the current.
 
     This function is useful for selection from valid values except current one.
     """
-    return validator(current, random.choice, strings)
+    return validator(current, random.choice, sequences)
 
 
 def selector(current, constraints, validate=int_validator):
@@ -283,9 +283,9 @@ def header_length(current):
 def bf_name(current):
     """Fuzz the backing file name."""
     constraints = [
-        truncate_string(STRING_V, len(current))
+        truncate_bytes(BYTES_V, len(current))
     ]
-    return selector(current, constraints, string_validator)
+    return selector(current, constraints, bytes_validator)
 
 
 def ext_magic(current):
@@ -303,10 +303,10 @@ def ext_length(current):
 def bf_format(current):
     """Fuzz backing file format in the corresponding header extension."""
     constraints = [
-        truncate_string(STRING_V, len(current)),
-        truncate_string(STRING_V, (len(current) + 7) & ~7)  # Fuzz padding
+        truncate_bytes(BYTES_V, len(current)),
+        truncate_bytes(BYTES_V, (len(current) + 7) & ~7)  # Fuzz padding
     ]
-    return selector(current, constraints, string_validator)
+    return selector(current, constraints, bytes_validator)
 
 
 def feature_type(current):
@@ -324,10 +324,10 @@ def feature_bit_number(current):
 def feature_name(current):
     """Fuzz feature name field of a feature name table header extension."""
     constraints = [
-        truncate_string(STRING_V, len(current)),
-        truncate_string(STRING_V, 46)  # Fuzz padding (field length = 46)
+        truncate_bytes(BYTES_V, len(current)),
+        truncate_bytes(BYTES_V, 46)  # Fuzz padding (field length = 46)
     ]
-    return selector(current, constraints, string_validator)
+    return selector(current, constraints, bytes_validator)
 
 
 def l1_entry(current):
-- 
2.21.0

[PATCH 07/10] image-fuzzer: Use bytes constant for field values

[Eduardo Habkost]

Field values are supposed to be bytes objects, not unicode
strings.  Change two constants that were declared as strings.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/image-fuzzer/qcow2/layout.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index 0adcbd448d..a0fd53c7ad 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -122,7 +122,7 @@ class Image(object):
     def create_header(self, cluster_bits, backing_file_name=None):
         """Generate a random valid header."""
         meta_header = [
-            ['>4s', 0, "QFI\xfb", 'magic'],
+            ['>4s', 0, b"QFI\xfb", 'magic'],
             ['>I', 4, random.randint(2, 3), 'version'],
             ['>Q', 8, 0, 'backing_file_offset'],
             ['>I', 16, 0, 'backing_file_size'],
@@ -231,7 +231,7 @@ class Image(object):
             feature_tables = []
             feature_ids = []
             inner_offset = self.ext_offset + ext_header_len
-            feat_name = 'some cool feature'
+            feat_name = b'some cool feature'
             while len(feature_tables) < num_fnt_entries * 3:
                 feat_type, feat_bit = gen_feat_ids()
                 # Remove duplicates
-- 
2.21.0

Re: [PATCH 07/10] image-fuzzer: Use bytes constant for field values

[Philippe Mathieu-Daudé]

On 10/16/19 9:24 PM, Eduardo Habkost wrote:
> Field values are supposed to be bytes objects, not unicode
> strings.  Change two constants that were declared as strings.
> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>   tests/image-fuzzer/qcow2/layout.py | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
> index 0adcbd448d..a0fd53c7ad 100644
> --- a/tests/image-fuzzer/qcow2/layout.py
> +++ b/tests/image-fuzzer/qcow2/layout.py
> @@ -122,7 +122,7 @@ class Image(object):
>       def create_header(self, cluster_bits, backing_file_name=None):
>           """Generate a random valid header."""
>           meta_header = [
> -            ['>4s', 0, "QFI\xfb", 'magic'],
> +            ['>4s', 0, b"QFI\xfb", 'magic'],
>               ['>I', 4, random.randint(2, 3), 'version'],
>               ['>Q', 8, 0, 'backing_file_offset'],
>               ['>I', 16, 0, 'backing_file_size'],
> @@ -231,7 +231,7 @@ class Image(object):
>               feature_tables = []
>               feature_ids = []
>               inner_offset = self.ext_offset + ext_header_len
> -            feat_name = 'some cool feature'
> +            feat_name = b'some cool feature'
>               while len(feature_tables) < num_fnt_entries * 3:
>                   feat_type, feat_bit = gen_feat_ids()
>                   # Remove duplicates
> 

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

[PATCH 08/10] image-fuzzer: Encode file name and file format to bytes

[Eduardo Habkost]

Callers of create_image() will pass strings as arguments, but the
Image class will expect bytes objects to be provided.  Encode
them inside create_image().

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/image-fuzzer/qcow2/layout.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index a0fd53c7ad..01bff4d05e 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -602,8 +602,8 @@ class Image(object):
 def create_image(test_img_path, backing_file_name=None, backing_file_fmt=None,
                  fields_to_fuzz=None):
     """Create a fuzzed image and write it to the specified file."""
-    image = Image(backing_file_name)
-    image.set_backing_file_format(backing_file_fmt)
+    image = Image(backing_file_name.encode())
+    image.set_backing_file_format(backing_file_fmt.encode())
     image.create_feature_name_table()
     image.set_end_of_extension_area()
     image.create_l_structures()
-- 
2.21.0

[PATCH 09/10] image-fuzzer: Run using python3

[Eduardo Habkost]

image-fuzzer is now supposed to be ready to run using Python 3.
Remove the __future__ imports and change the interpreter line to
"#!/usr/bin/env python3".

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/image-fuzzer/qcow2/__init__.py | 1 -
 tests/image-fuzzer/qcow2/layout.py   | 1 -
 tests/image-fuzzer/runner.py         | 3 +--
 3 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/tests/image-fuzzer/qcow2/__init__.py b/tests/image-fuzzer/qcow2/__init__.py
index 09ef59821b..ed3af5da86 100644
--- a/tests/image-fuzzer/qcow2/__init__.py
+++ b/tests/image-fuzzer/qcow2/__init__.py
@@ -1,2 +1 @@
-from __future__ import absolute_import
 from .layout import create_image
diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
index 01bff4d05e..57ebe86e9a 100644
--- a/tests/image-fuzzer/qcow2/layout.py
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -16,7 +16,6 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from __future__ import absolute_import
 import random
 import struct
 from . import fuzz
diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index 94cab5bd93..0793234815 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Tool for running fuzz tests
 #
@@ -18,7 +18,6 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from __future__ import print_function
 import sys
 import os
 import signal
-- 
2.21.0

[PATCH 10/10] image-fuzzer: Use errors parameter of subprocess.Popen()

[Eduardo Habkost]

Instead of manually encoding stderr and stdout output, use
`errors` parameter of subprocess.Popen().  This will make
process.communicate() return unicode strings instead of bytes
objects.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/image-fuzzer/runner.py | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index 0793234815..4ba5c79e13 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -79,16 +79,13 @@ def run_app(fd, q_args):
     devnull = open('/dev/null', 'r+')
     process = subprocess.Popen(q_args, stdin=devnull,
                                stdout=subprocess.PIPE,
-                               stderr=subprocess.PIPE)
+                               stderr=subprocess.PIPE,
+                               errors='replace')
     try:
         out, err = process.communicate()
         signal.alarm(0)
-        # fd is a text file, so we need to decode the process output before
-        # writing to it.
-        # We could be simply using the `errors` parameter of subprocess.Popen(),
-        # but this will be possible only after migrating to Python 3
-        fd.write(out.decode(errors='replace'))
-        fd.write(err.decode(errors='replace'))
+        fd.write(out)
+        fd.write(err)
         fd.flush()
         return process.returncode
 
-- 
2.21.0

Re: [PATCH 00/10] image-fuzzer: Port to Python 3

[John Snow]

On 10/16/19 3:24 PM, Eduardo Habkost wrote:
> This series ports image-fuzzer to Python 3.
> 
> Eduardo Habkost (10):
>   image-fuzzer: Open image files in binary mode
>   image-fuzzer: Write bytes instead of string to image file
>   image-fuzzer: Explicitly use integer division operator
>   image-fuzzer: Use io.StringIO
>   image-fuzzer: Use %r for all fiels at Field.__repr__()
>   image-fuzzer: Return bytes objects on string fuzzing functions
>   image-fuzzer: Use bytes constant for field values
>   image-fuzzer: Encode file name and file format to bytes
>   image-fuzzer: Run using python3
>   image-fuzzer: Use errors parameter of subprocess.Popen()
> 
>  tests/image-fuzzer/qcow2/__init__.py |  1 -
>  tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
>  tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
>  tests/image-fuzzer/runner.py         | 12 +++---
>  4 files changed, 61 insertions(+), 63 deletions(-)
> 

When I gave my try at converting this to python3 I noticed that the
"except OSError as e" segments used e[1] in a way that was not seemingly
supported.

Did you fix that in this series or did I miss it?

--js

Re: [PATCH 00/10] image-fuzzer: Port to Python 3

[Eduardo Habkost]

On Thu, Oct 17, 2019 at 05:11:29PM -0400, John Snow wrote:
> 
> 
> On 10/16/19 3:24 PM, Eduardo Habkost wrote:
> > This series ports image-fuzzer to Python 3.
> > 
> > Eduardo Habkost (10):
> >   image-fuzzer: Open image files in binary mode
> >   image-fuzzer: Write bytes instead of string to image file
> >   image-fuzzer: Explicitly use integer division operator
> >   image-fuzzer: Use io.StringIO
> >   image-fuzzer: Use %r for all fiels at Field.__repr__()
> >   image-fuzzer: Return bytes objects on string fuzzing functions
> >   image-fuzzer: Use bytes constant for field values
> >   image-fuzzer: Encode file name and file format to bytes
> >   image-fuzzer: Run using python3
> >   image-fuzzer: Use errors parameter of subprocess.Popen()
> > 
> >  tests/image-fuzzer/qcow2/__init__.py |  1 -
> >  tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
> >  tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
> >  tests/image-fuzzer/runner.py         | 12 +++---
> >  4 files changed, 61 insertions(+), 63 deletions(-)
> > 
> 
> When I gave my try at converting this to python3 I noticed that the
> "except OSError as e" segments used e[1] in a way that was not seemingly
> supported.
> 
> Did you fix that in this series or did I miss it?

Good catch, I hadn't noticed that.  I didn't fix it.

-- 
Eduardo

Re: [PATCH 00/10] image-fuzzer: Port to Python 3

[John Snow]

On 10/17/19 5:29 PM, Eduardo Habkost wrote:
> On Thu, Oct 17, 2019 at 05:11:29PM -0400, John Snow wrote:
>>
>>
>> On 10/16/19 3:24 PM, Eduardo Habkost wrote:
>>> This series ports image-fuzzer to Python 3.
>>>
>>> Eduardo Habkost (10):
>>>   image-fuzzer: Open image files in binary mode
>>>   image-fuzzer: Write bytes instead of string to image file
>>>   image-fuzzer: Explicitly use integer division operator
>>>   image-fuzzer: Use io.StringIO
>>>   image-fuzzer: Use %r for all fiels at Field.__repr__()
>>>   image-fuzzer: Return bytes objects on string fuzzing functions
>>>   image-fuzzer: Use bytes constant for field values
>>>   image-fuzzer: Encode file name and file format to bytes
>>>   image-fuzzer: Run using python3
>>>   image-fuzzer: Use errors parameter of subprocess.Popen()
>>>
>>>  tests/image-fuzzer/qcow2/__init__.py |  1 -
>>>  tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
>>>  tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
>>>  tests/image-fuzzer/runner.py         | 12 +++---
>>>  4 files changed, 61 insertions(+), 63 deletions(-)
>>>
>>
>> When I gave my try at converting this to python3 I noticed that the
>> "except OSError as e" segments used e[1] in a way that was not seemingly
>> supported.
>>
>> Did you fix that in this series or did I miss it?
> 
> Good catch, I hadn't noticed that.  I didn't fix it.
> 

I recommend using pylint(3) with a bunch of the style issues turned off,
e.g.;

--disable=missing-docstring --disable=invalid-name

it will still whine about a lot of reasonably harmless stuff, but
sometimes it has a few errors to show.

--js

Re: [PATCH 00/10] image-fuzzer: Port to Python 3

[Eduardo Habkost]

On Thu, Oct 17, 2019 at 06:29:27PM -0300, Eduardo Habkost wrote:
> On Thu, Oct 17, 2019 at 05:11:29PM -0400, John Snow wrote:
> > 
> > 
> > On 10/16/19 3:24 PM, Eduardo Habkost wrote:
> > > This series ports image-fuzzer to Python 3.
> > > 
> > > Eduardo Habkost (10):
> > >   image-fuzzer: Open image files in binary mode
> > >   image-fuzzer: Write bytes instead of string to image file
> > >   image-fuzzer: Explicitly use integer division operator
> > >   image-fuzzer: Use io.StringIO
> > >   image-fuzzer: Use %r for all fiels at Field.__repr__()
> > >   image-fuzzer: Return bytes objects on string fuzzing functions
> > >   image-fuzzer: Use bytes constant for field values
> > >   image-fuzzer: Encode file name and file format to bytes
> > >   image-fuzzer: Run using python3
> > >   image-fuzzer: Use errors parameter of subprocess.Popen()
> > > 
> > >  tests/image-fuzzer/qcow2/__init__.py |  1 -
> > >  tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
> > >  tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
> > >  tests/image-fuzzer/runner.py         | 12 +++---
> > >  4 files changed, 61 insertions(+), 63 deletions(-)
> > > 
> > 
> > When I gave my try at converting this to python3 I noticed that the
> > "except OSError as e" segments used e[1] in a way that was not seemingly
> > supported.
> > 
> > Did you fix that in this series or did I miss it?
> 
> Good catch, I hadn't noticed that.  I didn't fix it.

Separate patch sent for that issue:
https://lore.kernel.org/qemu-devel/20191021214117.18091-1-ehabkost@redhat.com/

-- 
Eduardo

Re: [PATCH 00/10] image-fuzzer: Port to Python 3

[John Snow]

On 10/22/19 4:26 PM, Eduardo Habkost wrote:
> On Thu, Oct 17, 2019 at 06:29:27PM -0300, Eduardo Habkost wrote:
>> On Thu, Oct 17, 2019 at 05:11:29PM -0400, John Snow wrote:
>>>
>>>
>>> On 10/16/19 3:24 PM, Eduardo Habkost wrote:
>>>> This series ports image-fuzzer to Python 3.
>>>>
>>>> Eduardo Habkost (10):
>>>>   image-fuzzer: Open image files in binary mode
>>>>   image-fuzzer: Write bytes instead of string to image file
>>>>   image-fuzzer: Explicitly use integer division operator
>>>>   image-fuzzer: Use io.StringIO
>>>>   image-fuzzer: Use %r for all fiels at Field.__repr__()
>>>>   image-fuzzer: Return bytes objects on string fuzzing functions
>>>>   image-fuzzer: Use bytes constant for field values
>>>>   image-fuzzer: Encode file name and file format to bytes
>>>>   image-fuzzer: Run using python3
>>>>   image-fuzzer: Use errors parameter of subprocess.Popen()
>>>>
>>>>  tests/image-fuzzer/qcow2/__init__.py |  1 -
>>>>  tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
>>>>  tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
>>>>  tests/image-fuzzer/runner.py         | 12 +++---
>>>>  4 files changed, 61 insertions(+), 63 deletions(-)
>>>>
>>>
>>> When I gave my try at converting this to python3 I noticed that the
>>> "except OSError as e" segments used e[1] in a way that was not seemingly
>>> supported.
>>>
>>> Did you fix that in this series or did I miss it?
>>
>> Good catch, I hadn't noticed that.  I didn't fix it.
> 
> Separate patch sent for that issue:
> https://lore.kernel.org/qemu-devel/20191021214117.18091-1-ehabkost@redhat.com/
> 

With that squished in somewhere into this series:

Reviewed-by: John Snow <jsnow@redhat.com>

Re: [PATCH 00/10] image-fuzzer: Port to Python 3

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1126 bytes --]

On Wed, Oct 16, 2019 at 04:24:20PM -0300, Eduardo Habkost wrote:
> This series ports image-fuzzer to Python 3.
> 
> Eduardo Habkost (10):
>   image-fuzzer: Open image files in binary mode
>   image-fuzzer: Write bytes instead of string to image file
>   image-fuzzer: Explicitly use integer division operator
>   image-fuzzer: Use io.StringIO
>   image-fuzzer: Use %r for all fiels at Field.__repr__()
>   image-fuzzer: Return bytes objects on string fuzzing functions
>   image-fuzzer: Use bytes constant for field values
>   image-fuzzer: Encode file name and file format to bytes
>   image-fuzzer: Run using python3
>   image-fuzzer: Use errors parameter of subprocess.Popen()
> 
>  tests/image-fuzzer/qcow2/__init__.py |  1 -
>  tests/image-fuzzer/qcow2/fuzz.py     | 54 +++++++++++++-------------
>  tests/image-fuzzer/qcow2/layout.py   | 57 ++++++++++++++--------------
>  tests/image-fuzzer/runner.py         | 12 +++---
>  4 files changed, 61 insertions(+), 63 deletions(-)
> 
> -- 
> 2.21.0
> 
> 

Thanks, applied to my block tree:
https://github.com/stefanha/qemu/commits/block

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]