[Fuego] [PATCH 0/8] Python 2 & 3 compatible work

[Fuego] [PATCH 0/8] Python 2 & 3 compatible work

[sireesha.nakkala]

From: sireesha <sireesha.nakkala@toshiba-tsip.com>

Dear Tim,

I'm sending below series of patches which are compatible for python 2 &
3.

Kindly review the changes and provide feedback.


Shivanand Kunijadar (5):
  Create mylong function as a compatible replacement for long()
  Use file descriptor instead of file.write()
  Use decode() to convert bytes to string in python3
  Use actual string name with replace() instead of string.replace()
  Replace cmp() function with equivalent expression

sireesha (3):
  ftc: Use text mode in file operation to make compatible with python3
  Fix TypeError: sort() takes no positional arguments in python3
  python2to3: use list() for dict.keys(), dict.values() and dict.items()

 scripts/check-dependencies           |  2 +-
 scripts/deorphan-runs.py             | 10 +++++++++-
 scripts/ftc                          | 16 ++++++++--------
 scripts/jdiff                        |  4 ++--
 scripts/ovgen.py                     | 20 ++++++++++----------
 scripts/parser/common.py             | 17 +++++++++--------
 scripts/parser/prepare_chart_data.py | 22 +++++++++++-----------
 tests/Functional.LTP/ltp_process.py  |  2 +-
 8 files changed, 51 insertions(+), 42 deletions(-)

-- 
2.20.1

[Fuego] [PATCH 1/8] Create mylong function as a compatible replacement for long()

[sireesha.nakkala]

From: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>

Python3 removed the long() variable type and cast.
Define mylong() in try and except block which replaces mylong()
with long() in python2 and returns same passed value in python3.

Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
---
 scripts/deorphan-runs.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/scripts/deorphan-runs.py b/scripts/deorphan-runs.py
index c81ed24..82dcacb 100755
--- a/scripts/deorphan-runs.py
+++ b/scripts/deorphan-runs.py
@@ -77,6 +77,14 @@ class data_class:
         else:
             return item
 
+# define mylong to work in python2 and 3
+try:
+    val = long(4)
+    mylong = long
+except:
+    def mylong(x):
+        return x
+
 def populate_build_data():
     # construct the build_data map to hold information about this build
     build_data = data_class()
@@ -88,7 +96,7 @@ def populate_build_data():
     test_name = os.path.basename(test_name)
     build_data.job_name = "%s.%s.%s" % (board_name, test.spec, test_name)
     build_data.workspace = conf.FUEGO_RW+"/buildzone"
-    build_data.start_time = long(time.time() * 1000)
+    build_data.start_time = mylong(time.time() * 1000)
     build_data.reboot_flag = test.reboot
     build_data.rebuild_flag = test.rebuild
     build_data.precleanup_flag = test.precleanup
-- 
2.20.1

Re: [Fuego] [PATCH 1/8] Create mylong function as a compatible replacement for long()

[Bird, Tim]

This fix was interesting... :-)

> -----Original Message-----
> From: sireesha.nakkala@toshiba-tsip.com <sireesha.nakkala@toshiba-tsip.com>
> 
> From: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> 
> Python3 removed the long() variable type and cast.
> Define mylong() in try and except block which replaces mylong()
> with long() in python2 and returns same passed value in python3.
> 
> Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> ---
>  scripts/deorphan-runs.py | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/deorphan-runs.py b/scripts/deorphan-runs.py
> index c81ed24..82dcacb 100755
> --- a/scripts/deorphan-runs.py
> +++ b/scripts/deorphan-runs.py
> @@ -77,6 +77,14 @@ class data_class:
>          else:
>              return item
> 
> +# define mylong to work in python2 and 3
> +try:
> +    val = long(4)
> +    mylong = long
> +except:
> +    def mylong(x):
> +        return x

Interesting use of exception to define mylong function in Python3!

Since long() in python2 can also convert strings, it would be better,
in the general case, if the last line here was:
return int(x)
to get equivalent functionality.

However, since this is only used one place, with well-defined, numeric
inputs, this is fine as written.

> +
>  def populate_build_data():
>      # construct the build_data map to hold information about this build
>      build_data = data_class()
> @@ -88,7 +96,7 @@ def populate_build_data():
>      test_name = os.path.basename(test_name)
>      build_data.job_name = "%s.%s.%s" % (board_name, test.spec, test_name)
>      build_data.workspace = conf.FUEGO_RW+"/buildzone"
> -    build_data.start_time = long(time.time() * 1000)
> +    build_data.start_time = mylong(time.time() * 1000)
>      build_data.reboot_flag = test.reboot
>      build_data.rebuild_flag = test.rebuild
>      build_data.precleanup_flag = test.precleanup
> --
> 2.20.1
> 

Applied.

Just a question - the top of deorphan-runs.py has this invocation line:
#!/usr/bin/python

Are you running this on a system where /usr/bin/python is python3 and not python2?
I'm just curious.  If so, what Linux distro is that that?

Thanks!
 -- Tim

[Fuego] [PATCH 2/8] ftc: Use text mode in file operation to make compatible with python3

[sireesha.nakkala]

From: sireesha <sireesha.nakkala@toshiba-tsip.com>

In python 2 the str type can be used for both text an binary data, and
it works well even the file is opened in binary mode.

Whereas in python 3 it has made the text and binary data in distinct types,
so use text mode that supports both in python 2 and 3

Signed-off-by: sireesha <sireesha.nakkala@toshiba-tsip.com>
Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 scripts/ftc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/ftc b/scripts/ftc
index 3d61c4b..ce34b08 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -2974,7 +2974,7 @@ def gen_csv_report(header_data, report_data, report_file):
     import csv
 
     # generate header
-    with open(report_file, "wb") as report:
+    with open(report_file, "w") as report:
         writer = csv.writer(report)
         writer.writerows(header_data)
 
-- 
2.20.1

Re: [Fuego] [PATCH 2/8] ftc: Use text mode in file operation to make compatible with python3

[Bird, Tim]

> -----Original Message-----
> From: sireesha.nakkala@toshiba-tsip.com <sireesha.nakkala@toshiba-tsip.com>
> 
> From: sireesha <sireesha.nakkala@toshiba-tsip.com>
> 
> In python 2 the str type can be used for both text an binary data, and
> it works well even the file is opened in binary mode.
> 
> Whereas in python 3 it has made the text and binary data in distinct types,
> so use text mode that supports both in python 2 and 3
> 
> Signed-off-by: sireesha <sireesha.nakkala@toshiba-tsip.com>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  scripts/ftc | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/ftc b/scripts/ftc
> index 3d61c4b..ce34b08 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -2974,7 +2974,7 @@ def gen_csv_report(header_data, report_data, report_file):
>      import csv
> 
>      # generate header
> -    with open(report_file, "wb") as report:
> +    with open(report_file, "w") as report:
>          writer = csv.writer(report)
>          writer.writerows(header_data)
> 
> --
> 2.20.1
> 

Looks good.  Applied. 
 -- Tim

[Fuego] [PATCH 3/8] Use file descriptor instead of file.write()

[sireesha.nakkala]

From: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>

Use of file descriptor to write a file is compatible with python2 and
python3.

Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
---
 scripts/ovgen.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/scripts/ovgen.py b/scripts/ovgen.py
index faabc62..a6e0f13 100755
--- a/scripts/ovgen.py
+++ b/scripts/ovgen.py
@@ -375,27 +375,27 @@ def generateProlog(logdir, ofcls, classes, testdir, testspec):
         name = ofc.name
         debug_print ("\nwriting %s base class" % (name))
 
-        file.write(outfile, "#class: %s\n" % (name))
+        outfile.write("#class: %s\n" % (name))
 
         for var in ofc.vars:
             outStr = "%s=\"%s\"" % (var, ofc.vars[var])
             outStr = outStr.replace('"', '\"')
             debug_print("%s <- %s" % (outFilePath, outStr))
-            file.write(outfile, outStr+"\n")
+            outfile.write(outStr+"\n")
 
         for cap in ofc.cap_list:
             outStr = "CAP_%s=\"yes\"" % (cap)
             debug_print("%s <- %s" % (outFilePath, outStr))
-            file.write(outfile, outStr+"\n")
+            outfile.write(outStr+"\n")
 
-        file.write(outfile, "\n")
+        outfile.write("\n")
 
         for func in ofc.funcs:
             body = ofc.funcs[func]
             debug_print("%s <- %s()" % (outFilePath, func))
-            file.write(outfile, body+"\n")
+            outfile.write(body+"\n")
 
-        file.write(outfile, "\n")
+        outfile.write("\n")
 
     ts = parseSpec(logdir, testdir, testspec)
     generateSpec(ts, outfile)
-- 
2.20.1

Re: [Fuego] [PATCH 3/8] Use file descriptor instead of file.write()

[Bird, Tim]

> -----Original Message-----
> From: sireesha.nakkala@toshiba-tsip.com <sireesha.nakkala@toshiba-tsip.com>
> 
> From: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> 
> Use of file descriptor to write a file is compatible with python2 and
> python3.
> 
> Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> ---
>  scripts/ovgen.py | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/scripts/ovgen.py b/scripts/ovgen.py
> index faabc62..a6e0f13 100755
> --- a/scripts/ovgen.py
> +++ b/scripts/ovgen.py
> @@ -375,27 +375,27 @@ def generateProlog(logdir, ofcls, classes, testdir, testspec):
>          name = ofc.name
>          debug_print ("\nwriting %s base class" % (name))
> 
> -        file.write(outfile, "#class: %s\n" % (name))
> +        outfile.write("#class: %s\n" % (name))
> 
>          for var in ofc.vars:
>              outStr = "%s=\"%s\"" % (var, ofc.vars[var])
>              outStr = outStr.replace('"', '\"')
>              debug_print("%s <- %s" % (outFilePath, outStr))
> -            file.write(outfile, outStr+"\n")
> +            outfile.write(outStr+"\n")
> 
>          for cap in ofc.cap_list:
>              outStr = "CAP_%s=\"yes\"" % (cap)
>              debug_print("%s <- %s" % (outFilePath, outStr))
> -            file.write(outfile, outStr+"\n")
> +            outfile.write(outStr+"\n")
> 
> -        file.write(outfile, "\n")
> +        outfile.write("\n")
> 
>          for func in ofc.funcs:
>              body = ofc.funcs[func]
>              debug_print("%s <- %s()" % (outFilePath, func))
> -            file.write(outfile, body+"\n")
> +            outfile.write(body+"\n")
> 
> -        file.write(outfile, "\n")
> +        outfile.write("\n")
> 
>      ts = parseSpec(logdir, testdir, testspec)
>      generateSpec(ts, outfile)
> --
> 2.20.1
> 

Looks good.  Tested (with python2) and applied.

Thanks.
 -- Tim

[Fuego] [PATCH 4/8] Use decode() to convert bytes to string in python3

[sireesha.nakkala]

From: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>

The file read operation in python3 converts data to bytes.
Use decode() to convert bytes to string in python3 which is
compatible with python2 and python3.

Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
---
 scripts/ftc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/ftc b/scripts/ftc
index ce34b08..3723e7e 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -3960,7 +3960,7 @@ def ftc_exec_command(command, timeout):
             log.flush()
             data = os.read(tail_fd, 4096)
             if data:
-                sys.stdout.write(data)
+                sys.stdout.write(data.decode("utf8"))
                 sys.stdout.flush()
 
             time.sleep(.1)
@@ -4813,7 +4813,7 @@ def do_run_test(conf, options):
     # drain the log, in case there's more there
     data = os.read(tail_fd, 4096)
     while data:
-        sys.stdout.write(data)
+        sys.stdout.write(data.decode("utf8"))
         data = os.read(tail_fd, 4096)
 
     # record success or failure
-- 
2.20.1

Re: [Fuego] [PATCH 4/8] Use decode() to convert bytes to string in python3

[Bird, Tim]

See comments inline below.

> -----Original Message-----
> From: sireesha.nakkala@toshiba-tsip.com <sireesha.nakkala@toshiba-tsip.com>
> 
> From: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> 
> The file read operation in python3 converts data to bytes.
> Use decode() to convert bytes to string in python3 which is
> compatible with python2 and python3.
> 
> Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> ---
>  scripts/ftc | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/ftc b/scripts/ftc
> index ce34b08..3723e7e 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -3960,7 +3960,7 @@ def ftc_exec_command(command, timeout):
>              log.flush()
>              data = os.read(tail_fd, 4096)
>              if data:
> -                sys.stdout.write(data)
> +                sys.stdout.write(data.decode("utf8"))

I didn't like this solution, because it invokes a 'decode' operation on data
that may not actually be in 'utf' format.  If any of the test programs
(executed by the chain of processes under 'ftc_exec_command') produce
non-UTF-compatible output, then this might cause a UnicodeDecode exception.

Really, I just want the data to pass from the test program stdout to ftc stdout
unmodified in any way.  The decode might modify the data, or raise an
exception.

Instead of this, I changed the code to use this sequence:
stdout_fd = sys.stdout.fileno()
os.write(stdout_fd, data)

os.write() takes bytes, which is what os.read() returns, and this appears to work
for both python3 and python2.

>                  sys.stdout.flush()
> 
>              time.sleep(.1)
> @@ -4813,7 +4813,7 @@ def do_run_test(conf, options):
>      # drain the log, in case there's more there
>      data = os.read(tail_fd, 4096)
>      while data:
> -        sys.stdout.write(data)
> +        sys.stdout.write(data.decode("utf8"))
>          data = os.read(tail_fd, 4096)
> 
>      # record success or failure
> --
> 2.20.1
> 

The locations you pointed out have been changed with a commit that I made.
It should be compatible with both Python2 and Python3, but please give it
a test.

I gave you a "Reported by" credit in the commit.

Thanks for the report.
 -- Tim

[Fuego] [PATCH 5/8] Use actual string name with replace() instead of string.replace()

[sireesha.nakkala]

From: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>

string.replace() is deprecated function and it is not supported in
python3. Use actual string name with replace() which is compatible
with python2 and python3.

Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
---
 scripts/ovgen.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/ovgen.py b/scripts/ovgen.py
index a6e0f13..1386ff5 100755
--- a/scripts/ovgen.py
+++ b/scripts/ovgen.py
@@ -408,15 +408,15 @@ def generateSpec(ts, fout):
 
     for var in ts.variables:
         varname = "%s_%s" % (ts.name, var)
-        varname = string.replace(varname, ".", "_").upper()
+        varname = varname.replace(".", "_").upper()
         value = "%s" % (ts.variables[var])
         outStr = '%s="%s"' % (varname, value)
         debug_print(outStr, 3)
         fout.write(outStr + "\n")
 
     if ts.fail_case:
-        tNameUp = string.replace(ts.name, ".", "_").upper()
-        tNameUp = string.replace(tNameUp, "-", "_").upper()
+        tNameUp = ts.name.replace(".", "_").upper()
+        tNameUp = tNameUp.replace("-", "_").upper()
         fc_num = len(ts.fail_case)
         outNum = "%s_FAIL_CASE_COUNT=%s" % (tNameUp, fc_num)
         fout.write(outNum + "\n")
-- 
2.20.1

Re: [Fuego] [PATCH 5/8] Use actual string name with replace() instead of string.replace()

[Bird, Tim]

> -----Original Message-----
> From: sireesha.nakkala@toshiba-tsip.com <sireesha.nakkala@toshiba-tsip.com>
> 
> From: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> 
> string.replace() is deprecated function and it is not supported in
> python3. Use actual string name with replace() which is compatible
> with python2 and python3.
> 
> Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> ---
>  scripts/ovgen.py | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/ovgen.py b/scripts/ovgen.py
> index a6e0f13..1386ff5 100755
> --- a/scripts/ovgen.py
> +++ b/scripts/ovgen.py
> @@ -408,15 +408,15 @@ def generateSpec(ts, fout):
> 
>      for var in ts.variables:
>          varname = "%s_%s" % (ts.name, var)
> -        varname = string.replace(varname, ".", "_").upper()
> +        varname = varname.replace(".", "_").upper()
>          value = "%s" % (ts.variables[var])
>          outStr = '%s="%s"' % (varname, value)
>          debug_print(outStr, 3)
>          fout.write(outStr + "\n")
> 
>      if ts.fail_case:
> -        tNameUp = string.replace(ts.name, ".", "_").upper()
> -        tNameUp = string.replace(tNameUp, "-", "_").upper()
> +        tNameUp = ts.name.replace(".", "_").upper()
> +        tNameUp = tNameUp.replace("-", "_").upper()
>          fc_num = len(ts.fail_case)
>          outNum = "%s_FAIL_CASE_COUNT=%s" % (tNameUp, fc_num)
>          fout.write(outNum + "\n")
> --
> 2.20.1
> 

Looks good. Applied.

I also did a followup commit to remove 'import string' from the module,
(and I renamed a an argument from 'string' to 'msg' in one function).
This should help prevent accidental use of this module in the future.

These were the only references to the string module in the code, so the
import is no longer needed, and it's good form to remove no-longer-needed
imports, to avoid future confusion.

Thanks,
 -- Tim

[Fuego] [PATCH 6/8] Replace cmp() function with equivalent expression

[sireesha.nakkala]

From: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>

The function cmp() is not supported in python3. Replace cmp() function
with equivalent expression and the expression works well in python2 and 3

Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
---
 scripts/parser/common.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/parser/common.py b/scripts/parser/common.py
index d7dcfbd..a04047a 100644
--- a/scripts/parser/common.py
+++ b/scripts/parser/common.py
@@ -547,7 +547,7 @@ def name_compare(a, b):
         b_name = b["name"]
     except:
         b_name = b
-    return cmp(a_name, b_name)
+    return (a_name > b_name) - (a_name < b_name)
 
 def dump_ordered_data(data, indent=""):
     if type(data)==type({}):
-- 
2.20.1

Re: [Fuego] [PATCH 6/8] Replace cmp() function with equivalent expression

[Bird, Tim]

> -----Original Message-----
> From: sireesha.nakkala@toshiba-tsip.com <sireesha.nakkala@toshiba-tsip.com>> 
> From: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> 
> The function cmp() is not supported in python3. Replace cmp() function
> with equivalent expression and the expression works well in python2 and 3
> 
> Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> ---
>  scripts/parser/common.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/parser/common.py b/scripts/parser/common.py
> index d7dcfbd..a04047a 100644
> --- a/scripts/parser/common.py
> +++ b/scripts/parser/common.py
> @@ -547,7 +547,7 @@ def name_compare(a, b):
>          b_name = b["name"]
>      except:
>          b_name = b
> -    return cmp(a_name, b_name)
> +    return (a_name > b_name) - (a_name < b_name)
> 
>  def dump_ordered_data(data, indent=""):
>      if type(data)==type({}):
> --
> 2.20.1
> 

Looks good. Applied.
 -- Tim

[Fuego] [PATCH 7/8] Fix TypeError: sort() takes no positional arguments in python3

[sireesha.nakkala]

From: sireesha <sireesha.nakkala@toshiba-tsip.com>

Transform an old-style comparison function to a key function.
use functools.cmp_to_key(func) with sort() function to fix the issue.
This change is compatible with python2 and python3.

Signed-off-by: sireesha <sireesha.nakkala@toshiba-tsip.com>
Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
---
 scripts/parser/common.py             | 5 +++--
 scripts/parser/prepare_chart_data.py | 6 +++---
 2 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/scripts/parser/common.py b/scripts/parser/common.py
index a04047a..8c307f1 100644
--- a/scripts/parser/common.py
+++ b/scripts/parser/common.py
@@ -26,7 +26,8 @@ common.py - This library contains parsing functions
 By Daniel Sangorrin (July 2017)
 """
 
-import sys, os, re, json, time, collections
+import sys, os, re, json, time, collections, functools
+
 from fuego_parser_utils import hls, split_test_id, get_test_case
 
 loglevel = "info"
@@ -562,7 +563,7 @@ def dump_ordered_data(data, indent=""):
     if type(data)==type([]):
         print("%s[" % indent)
         item_list = data[:]
-        item_list.sort(name_compare)
+        item_list.sort(key=functools.cmp_to_key(name_compare))
         for item in item_list:
             dump_ordered_data(item,indent+"    ")
         print("%s]" % indent)
diff --git a/scripts/parser/prepare_chart_data.py b/scripts/parser/prepare_chart_data.py
index 9aea666..d85315e 100644
--- a/scripts/parser/prepare_chart_data.py
+++ b/scripts/parser/prepare_chart_data.py
@@ -32,7 +32,7 @@ the results of tests in the Jenkins interface.
 By Tim Bird (September 2017)
 """
 
-import sys, os, re, json, collections
+import sys, os, re, json, collections, functools
 from filelock import FileLock
 from operator import itemgetter
 from fuego_parser_utils import split_test_id, get_test_case
@@ -507,7 +507,7 @@ def make_measure_tables(test_name, chart_config, entries):
 
         # one row per test case
         tg_list = result_map.keys()
-        tg_list.sort(cmp_alpha_num)
+        tg_list.sort(key=functools.cmp_to_key(cmp_alpha_num))
 
         for tg in tg_list:
             # break apart tguid(tc) and divide into test set and test case
@@ -730,7 +730,7 @@ def make_testcase_table(test_name, chart_config, entries):
 
         # one row per test case
         tc_list = result_map.keys()
-        tc_list.sort(cmp_alpha_num)
+        tc_list.sort(key=functools.cmp_to_key(cmp_alpha_num))
 
         for tc in tc_list:
             row = '<tr><td>' + tc + '</td>'
-- 
2.20.1

Re: [Fuego] [PATCH 7/8] Fix TypeError: sort() takes no positional arguments in python3

[Bird, Tim]

> -----Original Message-----
> From: sireesha.nakkala@toshiba-tsip.com <sireesha.nakkala@toshiba-tsip.com>
> 
> From: sireesha <sireesha.nakkala@toshiba-tsip.com>
> 
> Transform an old-style comparison function to a key function.
> use functools.cmp_to_key(func) with sort() function to fix the issue.
> This change is compatible with python2 and python3.
> 
> Signed-off-by: sireesha <sireesha.nakkala@toshiba-tsip.com>
> Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> ---
>  scripts/parser/common.py             | 5 +++--
>  scripts/parser/prepare_chart_data.py | 6 +++---
>  2 files changed, 6 insertions(+), 5 deletions(-)
> 
> diff --git a/scripts/parser/common.py b/scripts/parser/common.py
> index a04047a..8c307f1 100644
> --- a/scripts/parser/common.py
> +++ b/scripts/parser/common.py
> @@ -26,7 +26,8 @@ common.py - This library contains parsing functions
>  By Daniel Sangorrin (July 2017)
>  """
> 
> -import sys, os, re, json, time, collections
> +import sys, os, re, json, time, collections, functools
> +
>  from fuego_parser_utils import hls, split_test_id, get_test_case
> 
>  loglevel = "info"
> @@ -562,7 +563,7 @@ def dump_ordered_data(data, indent=""):
>      if type(data)==type([]):
>          print("%s[" % indent)
>          item_list = data[:]
> -        item_list.sort(name_compare)
> +        item_list.sort(key=functools.cmp_to_key(name_compare))
>          for item in item_list:
>              dump_ordered_data(item,indent+"    ")
>          print("%s]" % indent)
> diff --git a/scripts/parser/prepare_chart_data.py b/scripts/parser/prepare_chart_data.py
> index 9aea666..d85315e 100644
> --- a/scripts/parser/prepare_chart_data.py
> +++ b/scripts/parser/prepare_chart_data.py
> @@ -32,7 +32,7 @@ the results of tests in the Jenkins interface.
>  By Tim Bird (September 2017)
>  """
> 
> -import sys, os, re, json, collections
> +import sys, os, re, json, collections, functools
>  from filelock import FileLock
>  from operator import itemgetter
>  from fuego_parser_utils import split_test_id, get_test_case
> @@ -507,7 +507,7 @@ def make_measure_tables(test_name, chart_config, entries):
> 
>          # one row per test case
>          tg_list = result_map.keys()
> -        tg_list.sort(cmp_alpha_num)
> +        tg_list.sort(key=functools.cmp_to_key(cmp_alpha_num))
> 
>          for tg in tg_list:
>              # break apart tguid(tc) and divide into test set and test case
> @@ -730,7 +730,7 @@ def make_testcase_table(test_name, chart_config, entries):
> 
>          # one row per test case
>          tc_list = result_map.keys()
> -        tc_list.sort(cmp_alpha_num)
> +        tc_list.sort(key=functools.cmp_to_key(cmp_alpha_num))
> 
>          for tc in tc_list:
>              row = '<tr><td>' + tc + '</td>'
> --
> 2.20.1
> 

Looks good.  I don't particularly like that the Python3 language designers dropped support
for cmp functions.  If you look under the hood, this 'key function' nonsense just implements
a cmp under several more layers of abstraction.  It's inexcusable to break backwards 
compatibility like this (removing sort()'s positional arguments), especially when it creates
worthless abstraction layers as a requirement for forward-porting.

However, none of that is your fault.  Thanks for the patch.  Sorry for the rant.

Sigh - have I told anyone lately how much I dislike Python3 and their language development policies?

 -- Tim

[Fuego] [PATCH 8/8] python2to3: use list() for dict.keys(), dict.values() and dict.items()

[sireesha.nakkala]

From: sireesha <sireesha.nakkala@toshiba-tsip.com>

python3 returns dict_keys type for dict.keys() which doesn't support
indexing where as python2 returns a list type which does.
Use list() on the output of dict.keys() to make the code work with
python3.

Signed-off-by: sireesha <sireesha.nakkala@toshiba-tsip.com>
Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
---
 scripts/check-dependencies           |  2 +-
 scripts/ftc                          | 10 +++++-----
 scripts/jdiff                        |  4 ++--
 scripts/ovgen.py                     |  2 +-
 scripts/parser/common.py             | 10 +++++-----
 scripts/parser/prepare_chart_data.py | 16 ++++++++--------
 tests/Functional.LTP/ltp_process.py  |  2 +-
 7 files changed, 23 insertions(+), 23 deletions(-)

diff --git a/scripts/check-dependencies b/scripts/check-dependencies
index 72a5f31..583012e 100755
--- a/scripts/check-dependencies
+++ b/scripts/check-dependencies
@@ -218,7 +218,7 @@ def main():
 
     # print the testcases skipped
     # this is the main output from the program
-    testcases_to_skip = reasons.keys()
+    testcases_to_skip = list(reasons.keys())
     testcases_to_skip.sort()
     for testcase in testcases_to_skip:
         print(testcase)
diff --git a/scripts/ftc b/scripts/ftc
index 3723e7e..f32d9cc 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -905,7 +905,7 @@ class test_class:
     }
     def __init__(self, conf, test_dict, test_flags={}):
         merged_defaults = dict(self.DEFAULT_DEFAULTS)
-        for key, value in test_flags.items():
+        for key, value in list(test_flags.items()):
             merged_defaults[key] = value
         self.name = str(test_dict["testName"])
         self.test_type = self.name.split(".")[0]
@@ -1642,7 +1642,7 @@ def parse_testplan(conf, plan, test_dict):
             test_flags['postcleanup'] = plan['default_postcleanup']
 
         # override with testplan per-test flags
-        for key, value in plan_test_dict.items():
+        for key, value in list(plan_test_dict.items()):
             if key == "testName":
                 test_dict[key] = value
                 continue
@@ -2584,7 +2584,7 @@ def parse_where(where_string):
 
 def filter_runs(run_map, where_list):
     new_run_list = []
-    for run_id, run in run_map.items():
+    for run_id, run in list(run_map.items()):
         match = True
         for where in where_list:
             if not where.match(run):
@@ -2915,7 +2915,7 @@ def pts_set_style(ws):
         for cell in row:
             if cell.value:
                 dims[cell.column] = max((dims.get(cell.column, 0), len(str(cell.value)) + 2))
-    for col, value in dims.items():
+    for col, value in list(dims.items()):
         ws.column_dimensions[col].width = value
 
 
@@ -4767,7 +4767,7 @@ def do_run_test(conf, options):
                 test_spec_data['specs'][test.spec][key] = dyn_vars[key]
 
         # track what variables where modified
-        test_spec_data['specs'][test.spec]['dyn_vars'] = dyn_vars.keys()
+        test_spec_data['specs'][test.spec]['dyn_vars'] = list(dyn_vars.keys())
     dvar("test_spec_data")
 
     if os.path.isdir(build_data.test_logdir):
diff --git a/scripts/jdiff b/scripts/jdiff
index b267afd..39ea1db 100755
--- a/scripts/jdiff
+++ b/scripts/jdiff
@@ -94,9 +94,9 @@ def diff_list(prefix, l1, l2):
 def diff_map(prefix, m1, m2):
     dprint("in diff_map, m1=%s, m2=%s" % (m1, m2))
 
-    l1s = m1.keys()
+    l1s = list(m1.keys())
     l1s.sort()
-    l2s = m2.keys()
+    l2s = list(m2.keys())
     l2s.sort()
     dprint("in diff_map, l1s=%s, l2s=%s" % (l1s, l2s))
 
diff --git a/scripts/ovgen.py b/scripts/ovgen.py
index 1386ff5..f47a384 100755
--- a/scripts/ovgen.py
+++ b/scripts/ovgen.py
@@ -453,7 +453,7 @@ def parseSpec(logdir, testdir, testspec):
                 test_spec_data = json.load(f)
             except:
                 error_out("Error parsing spec file %s" % specpath)
-        for key in test_spec_data['specs'].keys():
+        for key in list(test_spec_data['specs'].keys()):
             if key != testspec:
                 del test_spec_data['specs'][key]
         debug_print("test spec data:" + str(test_spec_data))
diff --git a/scripts/parser/common.py b/scripts/parser/common.py
index 8c307f1..a3d71ca 100644
--- a/scripts/parser/common.py
+++ b/scripts/parser/common.py
@@ -178,7 +178,7 @@ def add_results(results, run_data):
     dprint("in add_results")
     if not results:
         return
-    for test_case_id in results.keys():
+    for test_case_id in list(results.keys()):
         test_case = get_test_case(test_case_id, run_data)
         if not test_case:
             continue
@@ -516,7 +516,7 @@ def apply_criteria(run_data, criteria_data):
 def create_default_ref(results):
     dprint("in create_default_ref")
     ref = {'test_sets': []}
-    for test_case_id in results.keys():
+    for test_case_id in list(results.keys()):
         test_set_name, test_case_name = split_test_id(test_case_id)
         item = results[test_case_id]
         if isinstance(item, list):
@@ -553,7 +553,7 @@ def name_compare(a, b):
 def dump_ordered_data(data, indent=""):
     if type(data)==type({}):
         print("%s{" % indent)
-        keylist = data.keys()
+        keylist = list(data.keys())
         keylist.sort()
         for key in keylist:
             print('%s "%s":' % (indent+"  ", key),
@@ -734,7 +734,7 @@ def process_data(ref_section_pat, test_results, plot_type, label):
     test_name = TESTDIR.split(".")[1]
 
     # convert old-style cur_dict into measurements structure
-    for (old_id, value) in test_results.items():
+    for (old_id, value) in list(test_results.items()):
         ts_name, tc_name, measure = split_old_id(old_id)
         test_case_id = "%s.%s" % (ts_name, tc_name)
         new_measure = {"name":measure, "measure": float(value)}
@@ -819,7 +819,7 @@ def split_output_per_testcase (regex_string, measurements, info_follows_regex=0)
     fd.close()
 
     # note that measurements is an OrderedDict, so keys comes out ordered
-    testcase_names = measurements.keys()
+    testcase_names = list(measurements.keys())
 
     if info_follows_regex:
         # You can have stuff before the first testcase delimiter that
diff --git a/scripts/parser/prepare_chart_data.py b/scripts/parser/prepare_chart_data.py
index d85315e..61b55f5 100644
--- a/scripts/parser/prepare_chart_data.py
+++ b/scripts/parser/prepare_chart_data.py
@@ -386,7 +386,7 @@ def make_measure_plots(test_name, chart_config, entries):
                 point = [entry.build_number, value]
                 series_map[ref_series_key]["data"].append(point)
 
-        flot_data = series_map.values()
+        flot_data = list(series_map.values())
         flot_data.sort(key=itemgetter('label'))
 
         flot_options = {
@@ -425,7 +425,7 @@ def make_measure_tables(test_name, chart_config, entries):
     for entry in entries:
         bsp_key = entry.board + "." + entry.spec + "." + entry.kernel
         bsp_map[bsp_key] = ((entry.board, entry.spec, entry.kernel))
-    bsp_list = bsp_map.values()
+    bsp_list = list(bsp_map.values())
 
     # now make a chart for each one:
     for board, spec, kver in bsp_list:
@@ -473,7 +473,7 @@ def make_measure_tables(test_name, chart_config, entries):
             else:
                 build_num_map[entry.build_number][3] += 1
 
-        bn_list = build_num_map.keys()
+        bn_list = list(build_num_map.keys())
         bn_list.sort()
 
         # FIXTHIS - should read col_limit from chart_config
@@ -506,7 +506,7 @@ def make_measure_tables(test_name, chart_config, entries):
         html += row
 
         # one row per test case
-        tg_list = result_map.keys()
+        tg_list = list(result_map.keys())
         tg_list.sort(key=functools.cmp_to_key(cmp_alpha_num))
 
         for tg in tg_list:
@@ -617,7 +617,7 @@ def make_testcase_table(test_name, chart_config, entries):
     for entry in entries:
         bts_key = entry.board + "." + entry.test_set
         bts_map[bts_key] = ((entry.board, entry.test_set))
-    bts_list = bts_map.values()
+    bts_list = list(bts_map.values())
 
     # now make a chart for each one:
     for board, ts in bts_list:
@@ -697,7 +697,7 @@ def make_testcase_table(test_name, chart_config, entries):
             else:
                 build_num_map[entry.build_number][3] += 1
 
-        bn_list = build_num_map.keys()
+        bn_list = list(build_num_map.keys())
         bn_list.sort()
 
         # FIXTHIS - should read col_limit from chart_config
@@ -729,7 +729,7 @@ def make_testcase_table(test_name, chart_config, entries):
         html += row
 
         # one row per test case
-        tc_list = result_map.keys()
+        tc_list = list(result_map.keys())
         tc_list.sort(key=functools.cmp_to_key(cmp_alpha_num))
 
         for tc in tc_list:
@@ -885,7 +885,7 @@ def make_testset_summary_table(test_name, chart_config, entries):
         html += row
 
         # one row per spec/build_number/test set
-        ssb_list = ssb_map.keys()
+        ssb_list = list(ssb_map.keys())
         ssb_list.sort(ssb_cmp)
 
         # calculate rowspan for each spec and spec/build_number combo
diff --git a/tests/Functional.LTP/ltp_process.py b/tests/Functional.LTP/ltp_process.py
index 8a29830..8e2b637 100644
--- a/tests/Functional.LTP/ltp_process.py
+++ b/tests/Functional.LTP/ltp_process.py
@@ -269,7 +269,7 @@ def pts_set_style(ws):
         for cell in row:
             if cell.value:
                 dims[cell.column] = max((dims.get(cell.column, 0), len(cell.value) + 2))
-    for col, value in dims.items():
+    for col, value in list(dims.items()):
         ws.column_dimensions[col].width = value
 
 if os.path.exists('pts.log'):
-- 
2.20.1

Re: [Fuego] [PATCH 8/8] python2to3: use list() for dict.keys(), dict.values() and dict.items()

[Bird, Tim]

> -----Original Message-----
> From: sireesha.nakkala@toshiba-tsip.com <sireesha.nakkala@toshiba-tsip.com>
> Sent: Wednesday, April 20, 2022 4:12 AM
> To: Bird, Tim <Tim.Bird@sony.com>
> Cc: sireesha <sireesha.nakkala@toshiba-tsip.com>; fuego@lists.linuxfoundation.org; dinesh.kumar@toshiba-tsip.com;
> daichi1.fukui@toshiba.co.jp; kazuhiro3.hayashi@toshiba.co.jp; Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> Subject: [PATCH 8/8] python2to3: use list() for dict.keys(), dict.values() and dict.items()
> 
> From: sireesha <sireesha.nakkala@toshiba-tsip.com>
> 
> python3 returns dict_keys type for dict.keys() which doesn't support
> indexing where as python2 returns a list type which does.
> Use list() on the output of dict.keys() to make the code work with
> python3.

Well, strictly speaking some of the places you changed with this patch
don't really need to be changed.  The Python3 dict.keys() function now
returns an iterator instead of a list (as it did in Python2).  In places where
the only usage of the dict.keys() value was in a for loop, the change to
use list() is not needed, and in fact might hurt performance.  However,
some of these are real compatibility issues for using the code with Python3.

Overall, this seems like an easy way to shut up linter software (like 2to3)
to ignore any messages about these code sequences in the future.  So even
if they are not needed for code compatibility, these changes do make the
code easier to verify with the linter.

 
> Signed-off-by: sireesha <sireesha.nakkala@toshiba-tsip.com>
> Signed-off-by: Shivanand Kunijadar <Shivanand.Kunijadar@toshiba-tsip.com>
> ---
>  scripts/check-dependencies           |  2 +-
>  scripts/ftc                          | 10 +++++-----
>  scripts/jdiff                        |  4 ++--
>  scripts/ovgen.py                     |  2 +-
>  scripts/parser/common.py             | 10 +++++-----
>  scripts/parser/prepare_chart_data.py | 16 ++++++++--------
>  tests/Functional.LTP/ltp_process.py  |  2 +-
>  7 files changed, 23 insertions(+), 23 deletions(-)
> 
> diff --git a/scripts/check-dependencies b/scripts/check-dependencies
> index 72a5f31..583012e 100755
> --- a/scripts/check-dependencies
> +++ b/scripts/check-dependencies
> @@ -218,7 +218,7 @@ def main():
> 
>      # print the testcases skipped
>      # this is the main output from the program
> -    testcases_to_skip = reasons.keys()
> +    testcases_to_skip = list(reasons.keys())
>      testcases_to_skip.sort()
>      for testcase in testcases_to_skip:
>          print(testcase)
> diff --git a/scripts/ftc b/scripts/ftc
> index 3723e7e..f32d9cc 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -905,7 +905,7 @@ class test_class:
>      }
>      def __init__(self, conf, test_dict, test_flags={}):
>          merged_defaults = dict(self.DEFAULT_DEFAULTS)
> -        for key, value in test_flags.items():
> +        for key, value in list(test_flags.items()):

list() is not needed here
>              merged_defaults[key] = value
>          self.name = str(test_dict["testName"])
>          self.test_type = self.name.split(".")[0]
> @@ -1642,7 +1642,7 @@ def parse_testplan(conf, plan, test_dict):
>              test_flags['postcleanup'] = plan['default_postcleanup']
> 
>          # override with testplan per-test flags
> -        for key, value in plan_test_dict.items():
> +        for key, value in list(plan_test_dict.items()):
list() is not needed here

>              if key == "testName":
>                  test_dict[key] = value
>                  continue
> @@ -2584,7 +2584,7 @@ def parse_where(where_string):
> 
>  def filter_runs(run_map, where_list):
>      new_run_list = []
> -    for run_id, run in run_map.items():
> +    for run_id, run in list(run_map.items()):
list() is not needed here

>          match = True
>          for where in where_list:
>              if not where.match(run):
> @@ -2915,7 +2915,7 @@ def pts_set_style(ws):
>          for cell in row:
>              if cell.value:
>                  dims[cell.column] = max((dims.get(cell.column, 0), len(str(cell.value)) + 2))
> -    for col, value in dims.items():
> +    for col, value in list(dims.items()):
list() is not needed here()

>          ws.column_dimensions[col].width = value
> 
> 
> @@ -4767,7 +4767,7 @@ def do_run_test(conf, options):
>                  test_spec_data['specs'][test.spec][key] = dyn_vars[key]
> 
>          # track what variables where modified
> -        test_spec_data['specs'][test.spec]['dyn_vars'] = dyn_vars.keys()
> +        test_spec_data['specs'][test.spec]['dyn_vars'] = list(dyn_vars.keys())
>      dvar("test_spec_data")
> 
>      if os.path.isdir(build_data.test_logdir):
> diff --git a/scripts/jdiff b/scripts/jdiff
> index b267afd..39ea1db 100755
> --- a/scripts/jdiff
> +++ b/scripts/jdiff
> @@ -94,9 +94,9 @@ def diff_list(prefix, l1, l2):
>  def diff_map(prefix, m1, m2):
>      dprint("in diff_map, m1=%s, m2=%s" % (m1, m2))
> 
> -    l1s = m1.keys()
> +    l1s = list(m1.keys())
>      l1s.sort()
> -    l2s = m2.keys()
> +    l2s = list(m2.keys())
>      l2s.sort()
>      dprint("in diff_map, l1s=%s, l2s=%s" % (l1s, l2s))
> 
> diff --git a/scripts/ovgen.py b/scripts/ovgen.py
> index 1386ff5..f47a384 100755
> --- a/scripts/ovgen.py
> +++ b/scripts/ovgen.py
> @@ -453,7 +453,7 @@ def parseSpec(logdir, testdir, testspec):
>                  test_spec_data = json.load(f)
>              except:
>                  error_out("Error parsing spec file %s" % specpath)
> -        for key in test_spec_data['specs'].keys():
> +        for key in list(test_spec_data['specs'].keys()):
list() is not needed here

>              if key != testspec:
>                  del test_spec_data['specs'][key]
>          debug_print("test spec data:" + str(test_spec_data))
> diff --git a/scripts/parser/common.py b/scripts/parser/common.py
> index 8c307f1..a3d71ca 100644
> --- a/scripts/parser/common.py
> +++ b/scripts/parser/common.py
> @@ -178,7 +178,7 @@ def add_results(results, run_data):
>      dprint("in add_results")
>      if not results:
>          return
> -    for test_case_id in results.keys():
> +    for test_case_id in list(results.keys()):
list() is not needed here.

>          test_case = get_test_case(test_case_id, run_data)
>          if not test_case:
>              continue
> @@ -516,7 +516,7 @@ def apply_criteria(run_data, criteria_data):
>  def create_default_ref(results):
>      dprint("in create_default_ref")
>      ref = {'test_sets': []}
> -    for test_case_id in results.keys():
> +    for test_case_id in list(results.keys()):
>          test_set_name, test_case_name = split_test_id(test_case_id)
>          item = results[test_case_id]
>          if isinstance(item, list):
> @@ -553,7 +553,7 @@ def name_compare(a, b):
>  def dump_ordered_data(data, indent=""):
>      if type(data)==type({}):
>          print("%s{" % indent)
> -        keylist = data.keys()
> +        keylist = list(data.keys())
>          keylist.sort()
>          for key in keylist:
>              print('%s "%s":' % (indent+"  ", key),
> @@ -734,7 +734,7 @@ def process_data(ref_section_pat, test_results, plot_type, label):
>      test_name = TESTDIR.split(".")[1]
> 
>      # convert old-style cur_dict into measurements structure
> -    for (old_id, value) in test_results.items():
> +    for (old_id, value) in list(test_results.items()):
list() is not needed here.

>          ts_name, tc_name, measure = split_old_id(old_id)
>          test_case_id = "%s.%s" % (ts_name, tc_name)
>          new_measure = {"name":measure, "measure": float(value)}
> @@ -819,7 +819,7 @@ def split_output_per_testcase (regex_string, measurements, info_follows_regex=0)
>      fd.close()
> 
>      # note that measurements is an OrderedDict, so keys comes out ordered
> -    testcase_names = measurements.keys()
> +    testcase_names = list(measurements.keys())
> 
>      if info_follows_regex:
>          # You can have stuff before the first testcase delimiter that
> diff --git a/scripts/parser/prepare_chart_data.py b/scripts/parser/prepare_chart_data.py
> index d85315e..61b55f5 100644
> --- a/scripts/parser/prepare_chart_data.py
> +++ b/scripts/parser/prepare_chart_data.py
> @@ -386,7 +386,7 @@ def make_measure_plots(test_name, chart_config, entries):
>                  point = [entry.build_number, value]
>                  series_map[ref_series_key]["data"].append(point)
> 
> -        flot_data = series_map.values()
> +        flot_data = list(series_map.values())
>          flot_data.sort(key=itemgetter('label'))
> 
>          flot_options = {
> @@ -425,7 +425,7 @@ def make_measure_tables(test_name, chart_config, entries):
>      for entry in entries:
>          bsp_key = entry.board + "." + entry.spec + "." + entry.kernel
>          bsp_map[bsp_key] = ((entry.board, entry.spec, entry.kernel))
> -    bsp_list = bsp_map.values()
> +    bsp_list = list(bsp_map.values())
> 
>      # now make a chart for each one:
>      for board, spec, kver in bsp_list:
> @@ -473,7 +473,7 @@ def make_measure_tables(test_name, chart_config, entries):
>              else:
>                  build_num_map[entry.build_number][3] += 1
> 
> -        bn_list = build_num_map.keys()
> +        bn_list = list(build_num_map.keys())
>          bn_list.sort()
> 
>          # FIXTHIS - should read col_limit from chart_config
> @@ -506,7 +506,7 @@ def make_measure_tables(test_name, chart_config, entries):
>          html += row
> 
>          # one row per test case
> -        tg_list = result_map.keys()
> +        tg_list = list(result_map.keys())
>          tg_list.sort(key=functools.cmp_to_key(cmp_alpha_num))
> 
>          for tg in tg_list:
> @@ -617,7 +617,7 @@ def make_testcase_table(test_name, chart_config, entries):
>      for entry in entries:
>          bts_key = entry.board + "." + entry.test_set
>          bts_map[bts_key] = ((entry.board, entry.test_set))
> -    bts_list = bts_map.values()
> +    bts_list = list(bts_map.values())
> 
>      # now make a chart for each one:
>      for board, ts in bts_list:
> @@ -697,7 +697,7 @@ def make_testcase_table(test_name, chart_config, entries):
>              else:
>                  build_num_map[entry.build_number][3] += 1
> 
> -        bn_list = build_num_map.keys()
> +        bn_list = list(build_num_map.keys())
>          bn_list.sort()
> 
>          # FIXTHIS - should read col_limit from chart_config
> @@ -729,7 +729,7 @@ def make_testcase_table(test_name, chart_config, entries):
>          html += row
> 
>          # one row per test case
> -        tc_list = result_map.keys()
> +        tc_list = list(result_map.keys())
>          tc_list.sort(key=functools.cmp_to_key(cmp_alpha_num))
> 
>          for tc in tc_list:
> @@ -885,7 +885,7 @@ def make_testset_summary_table(test_name, chart_config, entries):
>          html += row
> 
>          # one row per spec/build_number/test set
> -        ssb_list = ssb_map.keys()
> +        ssb_list = list(ssb_map.keys())
>          ssb_list.sort(ssb_cmp)
> 
>          # calculate rowspan for each spec and spec/build_number combo
> diff --git a/tests/Functional.LTP/ltp_process.py b/tests/Functional.LTP/ltp_process.py
> index 8a29830..8e2b637 100644
> --- a/tests/Functional.LTP/ltp_process.py
> +++ b/tests/Functional.LTP/ltp_process.py
> @@ -269,7 +269,7 @@ def pts_set_style(ws):
>          for cell in row:
>              if cell.value:
>                  dims[cell.column] = max((dims.get(cell.column, 0), len(cell.value) + 2))
> -    for col, value in dims.items():
> +    for col, value in list(dims.items()):
list() is not needed here.

>          ws.column_dimensions[col].width = value
> 
>  if os.path.exists('pts.log'):
> --
> 2.20.1
> 

OK - overall this looks good.  It's applied.

Sorry it took so long to get to these.  Thanks very much for the effort to fix
these incompatibilities with Python3 submit the patches!!

 -- Tim