[PATCH for-next RFC 0/4] tools: Python 3 compatibility

[PATCH for-next RFC 0/4] tools: Python 3 compatibility

[Wei Liu]

This series makes tools build with Python 3.

Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able
to give people some idea what sort of work is involved.

You will also need Andrew's "tools/xen-foreign: Update python scripts to be
Py3 compatible".

Wei.

Wei Liu (4):
  build/m4: make python_devel.m4 work with both python 2 and 3
  libxl: make python scripts work with python 2 and 3
  pygrub: convert python files with 2to3
  pygrub: make it build with python 3

 m4/python_devel.m4                 | 27 ++++++-----
 tools/configure                    | 34 ++++++++------
 tools/libxl/gentest.py             |  4 +-
 tools/libxl/gentypes.py            | 12 ++---
 tools/libxl/idl.py                 | 21 ++++-----
 tools/pygrub/src/ExtLinuxConf.py   | 16 +++----
 tools/pygrub/src/GrubConf.py       | 36 +++++++--------
 tools/pygrub/src/LiloConf.py       | 16 +++----
 tools/pygrub/src/fsimage/fsimage.c | 92 ++++++++++++++++++++++++++++++++++----
 9 files changed, 175 insertions(+), 83 deletions(-)

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH for-next RFC 1/4] build/m4: make python_devel.m4 work with both python 2 and 3

[Wei Liu]

Do the following:

1. Change the form of "print".
2. Check for ABI flags -- this is complicated because it is only
   introduced in 3.2.
3. Fix library name in AC_CHECK_LIB.
4. Remove other-libs in AC_CHECK_LIB.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
I doubt the non python-pkg branch works, because the paths generated
seem rather off. It definitely doesn't work on my machine, but I
don't know how other systems could possibly be configured before the
existence of python-config.
---
 m4/python_devel.m4 | 27 ++++++++++++++++-----------
 tools/configure    | 34 ++++++++++++++++++++--------------
 2 files changed, 36 insertions(+), 25 deletions(-)

diff --git a/m4/python_devel.m4 b/m4/python_devel.m4
index 05ea4ef7e2..1e2f41b6aa 100644
--- a/m4/python_devel.m4
+++ b/m4/python_devel.m4
@@ -2,37 +2,42 @@ AC_DEFUN([AX_CHECK_PYTHON_DEVEL], [
 ac_previous_cppflags=$CPPFLAGS
 ac_previous_ldflags=$LDFLAGS
 ac_python_version=`$PYTHON -c 'import distutils.sysconfig; \
-    print distutils.sysconfig.get_config_var("VERSION")'`
+    print(distutils.sysconfig.get_config_var("VERSION"))'`
+ac_python_abiflags=
 AC_PATH_PROG([pyconfig], [$PYTHON-config], [no])
 AS_IF([test x"$pyconfig" = x"no"], [
     dnl For those that don't have python-config
     CPPFLAGS="$CFLAGS `$PYTHON -c 'import distutils.sysconfig; \
         print "-I" + distutils.sysconfig.get_config_var("INCLUDEPY")'`"
     CPPFLAGS="$CPPFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("CFLAGS")'`"
+        print(distutils.sysconfig.get_config_var("CFLAGS"))'`"
     PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LIBS")'`"
+        print(distutils.sysconfig.get_config_var("LIBS"))'`"
     PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("SYSLIBS")'`"
+        print(distutils.sysconfig.get_config_var("SYSLIBS"))'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print "-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
-        standard_lib=1) + "/config"'`"
+        print("-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
+        standard_lib=1) + "/config")'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LINKFORSHARED")'`"
+        print(distutils.sysconfig.get_config_var("LINKFORSHARED"))'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LDFLAGS")'`"
+        print(distutils.sysconfig.get_config_var("LDFLAGS"))'`"
 ], [
     dnl If python-config is found use it
     CPPFLAGS="$CFLAGS `$PYTHON-config --cflags`"
     LDFLAGS="$LDFLAGS `$PYTHON-config --ldflags`"
     PYTHON_LIBS="$LIBS `$PYTHON-config --libs`"
+    abiflags="`$PYTHON-config --abiflags`"
+    if test "$?" == "0"
+    then
+        ac_python_abiflags="$abiflags"
+    fi
 ])
 
 AC_CHECK_HEADER([Python.h], [],
     [AC_MSG_ERROR([Unable to find Python development headers])],)
-AC_CHECK_LIB(python$ac_python_version, PyArg_ParseTuple, [],
-    [AC_MSG_ERROR([Unable to find a suitable python development library])],
-    [$PYTHON_LIBS])
+AC_CHECK_LIB(python$ac_python_version$ac_python_abiflags, PyArg_ParseTuple, [],
+    [AC_MSG_ERROR([Unable to find a suitable python development library])])
 CPPFLAGS=$ac_previous_cppflags
 LDFLAGS=$ac_previous_ldflags
 ])
diff --git a/tools/configure b/tools/configure
index acc857510e..8cc216b669 100755
--- a/tools/configure
+++ b/tools/configure
@@ -7419,7 +7419,8 @@ if test "$cross_compiling" != yes; then :
 ac_previous_cppflags=$CPPFLAGS
 ac_previous_ldflags=$LDFLAGS
 ac_python_version=`$PYTHON -c 'import distutils.sysconfig; \
-    print distutils.sysconfig.get_config_var("VERSION")'`
+    print(distutils.sysconfig.get_config_var("VERSION"))'`
+ac_python_abiflags=
 # Extract the first word of "$PYTHON-config", so it can be a program name with args.
 set dummy $PYTHON-config; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
@@ -7466,24 +7467,29 @@ if test x"$pyconfig" = x"no"; then :
         CPPFLAGS="$CFLAGS `$PYTHON -c 'import distutils.sysconfig; \
         print "-I" + distutils.sysconfig.get_config_var("INCLUDEPY")'`"
     CPPFLAGS="$CPPFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("CFLAGS")'`"
+        print(distutils.sysconfig.get_config_var("CFLAGS"))'`"
     PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LIBS")'`"
+        print(distutils.sysconfig.get_config_var("LIBS"))'`"
     PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("SYSLIBS")'`"
+        print(distutils.sysconfig.get_config_var("SYSLIBS"))'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print "-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
-        standard_lib=1) + "/config"'`"
+        print("-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
+        standard_lib=1) + "/config")'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LINKFORSHARED")'`"
+        print(distutils.sysconfig.get_config_var("LINKFORSHARED"))'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LDFLAGS")'`"
+        print(distutils.sysconfig.get_config_var("LDFLAGS"))'`"
 
 else
 
         CPPFLAGS="$CFLAGS `$PYTHON-config --cflags`"
     LDFLAGS="$LDFLAGS `$PYTHON-config --ldflags`"
     PYTHON_LIBS="$LIBS `$PYTHON-config --libs`"
+    abiflags="`$PYTHON-config --abiflags`"
+    if test "$?" == "0"
+    then
+        ac_python_abiflags="$abiflags"
+    fi
 
 fi
 
@@ -7495,14 +7501,14 @@ else
 fi
 
 
-as_ac_Lib=`$as_echo "ac_cv_lib_python$ac_python_version''_PyArg_ParseTuple" | $as_tr_sh`
-{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for PyArg_ParseTuple in -lpython$ac_python_version" >&5
-$as_echo_n "checking for PyArg_ParseTuple in -lpython$ac_python_version... " >&6; }
+as_ac_Lib=`$as_echo "ac_cv_lib_python$ac_python_version$ac_python_abiflags''_PyArg_ParseTuple" | $as_tr_sh`
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for PyArg_ParseTuple in -lpython$ac_python_version$ac_python_abiflags" >&5
+$as_echo_n "checking for PyArg_ParseTuple in -lpython$ac_python_version$ac_python_abiflags... " >&6; }
 if eval \${$as_ac_Lib+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
-LIBS="-lpython$ac_python_version $PYTHON_LIBS $LIBS"
+LIBS="-lpython$ac_python_version$ac_python_abiflags  $LIBS"
 cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
 
@@ -7535,10 +7541,10 @@ eval ac_res=\$$as_ac_Lib
 $as_echo "$ac_res" >&6; }
 if eval test \"x\$"$as_ac_Lib"\" = x"yes"; then :
   cat >>confdefs.h <<_ACEOF
-#define `$as_echo "HAVE_LIBpython$ac_python_version" | $as_tr_cpp` 1
+#define `$as_echo "HAVE_LIBpython$ac_python_version$ac_python_abiflags" | $as_tr_cpp` 1
 _ACEOF
 
-  LIBS="-lpython$ac_python_version $LIBS"
+  LIBS="-lpython$ac_python_version$ac_python_abiflags $LIBS"
 
 else
   as_fn_error $? "Unable to find a suitable python development library" "$LINENO" 5
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH for-next RFC 2/4] libxl: make python scripts work with python 2 and 3

[Wei Liu]

All scripts are transformed by 2to3.

The only addition is "from __future__ import print_function" so that
print("BLAH", file=sys.stderr) can work.

https://python-future.org/compatible_idioms.html

Tested with 2.7 and 3.5.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
I don't have environment to test 2.4 -- it is almost 15 years old. We
may want to consider bumping the minimum requirement to 2.7?
---
 tools/libxl/gentest.py  |  4 +++-
 tools/libxl/gentypes.py | 12 +++++++-----
 tools/libxl/idl.py      | 21 +++++++++++----------
 3 files changed, 21 insertions(+), 16 deletions(-)

diff --git a/tools/libxl/gentest.py b/tools/libxl/gentest.py
index 989959fc68..1cc7eebc82 100644
--- a/tools/libxl/gentest.py
+++ b/tools/libxl/gentest.py
@@ -1,5 +1,7 @@
 #!/usr/bin/python
 
+from __future__ import print_function
+
 import os
 import sys
 import re
@@ -86,7 +88,7 @@ def gen_rand_init(ty, v, indent = "    ", parent = None):
 
 if __name__ == '__main__':
     if len(sys.argv) < 3:
-        print >>sys.stderr, "Usage: gentest.py <idl> <implementation>"
+        print("Usage: gentest.py <idl> <implementation>", file=sys.stderr)
         sys.exit(1)
 
     random.seed(os.getenv('LIBXL_TESTIDL_SEED'))
diff --git a/tools/libxl/gentypes.py b/tools/libxl/gentypes.py
index 88e5c5f30e..6417c9dd8c 100644
--- a/tools/libxl/gentypes.py
+++ b/tools/libxl/gentypes.py
@@ -1,5 +1,7 @@
 #!/usr/bin/python
 
+from __future__ import print_function
+
 import sys
 import re
 
@@ -576,14 +578,14 @@ def libxl_C_enum_from_string(ty, str, e, indent = "    "):
 
 if __name__ == '__main__':
     if len(sys.argv) != 6:
-        print >>sys.stderr, "Usage: gentypes.py <idl> <header> <header-private> <header-json> <implementation>"
+        print("Usage: gentypes.py <idl> <header> <header-private> <header-json> <implementation>", file=sys.stderr)
         sys.exit(1)
 
     (_, idlname, header, header_private, header_json, impl) = sys.argv
 
     (builtins,types) = idl.parse(idlname)
 
-    print "outputting libxl type definitions to %s" % header
+    print("outputting libxl type definitions to %s" % header)
 
     f = open(header, "w")
 
@@ -633,7 +635,7 @@ if __name__ == '__main__':
     f.write("""#endif /* %s */\n""" % (header_define))
     f.close()
 
-    print "outputting libxl JSON definitions to %s" % header_json
+    print("outputting libxl JSON definitions to %s" % header_json)
 
     f = open(header_json, "w")
 
@@ -657,7 +659,7 @@ if __name__ == '__main__':
     f.write("""#endif /* %s */\n""" % header_json_define)
     f.close()
 
-    print "outputting libxl type internal definitions to %s" % header_private
+    print("outputting libxl type internal definitions to %s" % header_private)
 
     f = open(header_private, "w")
 
@@ -683,7 +685,7 @@ if __name__ == '__main__':
     f.write("""#endif /* %s */\n""" % header_json_define)
     f.close()
 
-    print "outputting libxl type implementations to %s" % impl
+    print("outputting libxl type implementations to %s" % impl)
 
     f = open(impl, "w")
     f.write("""
diff --git a/tools/libxl/idl.py b/tools/libxl/idl.py
index 2a7f3c44fe..27bd1053a6 100644
--- a/tools/libxl/idl.py
+++ b/tools/libxl/idl.py
@@ -1,3 +1,5 @@
+from __future__ import print_function
+
 import sys
 
 PASS_BY_VALUE = 1
@@ -11,7 +13,7 @@ DIR_BOTH = 3
 _default_namespace = ""
 def namespace(s):
     if type(s) != str:
-        raise TypeError, "Require a string for the default namespace."
+        raise TypeError("Require a string for the default namespace.")
     global _default_namespace
     _default_namespace = s
 
@@ -269,7 +271,7 @@ class KeyedUnion(Aggregate):
         if not isinstance(keyvar_type, Enumeration):
             raise ValueError
 
-        kv_kwargs = dict([(x.lstrip('keyvar_'),y) for (x,y) in kwargs.items() if x.startswith('keyvar_')])
+        kv_kwargs = dict([(x.lstrip('keyvar_'),y) for (x,y) in list(kwargs.items()) if x.startswith('keyvar_')])
         
         self.keyvar = Field(keyvar_type, keyvar_name, **kv_kwargs)
 
@@ -315,7 +317,7 @@ class Array(Type):
         kwargs.setdefault('json_parse_type', 'JSON_ARRAY')
         Type.__init__(self, namespace=elem_type.namespace, typename=elem_type.rawname + " *", **kwargs)
 
-        lv_kwargs = dict([(x.lstrip('lenvar_'),y) for (x,y) in kwargs.items() if x.startswith('lenvar_')])
+        lv_kwargs = dict([(x.lstrip('lenvar_'),y) for (x,y) in list(kwargs.items()) if x.startswith('lenvar_')])
 
         self.lenvar = Field(integer, lenvar_name, **lv_kwargs)
         self.elem_type = elem_type
@@ -346,12 +348,12 @@ class OrderedDict(dict):
         return [(x,self[x]) for x in self.__ordered]
 
 def parse(f):
-    print >>sys.stderr, "Parsing %s" % f
+    print("Parsing %s" % f, file=sys.stderr)
 
     globs = {}
     locs = OrderedDict()
 
-    for n,t in globals().items():
+    for n,t in list(globals().items()):
         if isinstance(t, Type):
             globs[n] = t
         elif isinstance(t,type(object)) and issubclass(t, Type):
@@ -362,11 +364,10 @@ def parse(f):
             globs[n] = t
 
     try:
-        execfile(f, globs, locs)
-    except SyntaxError,e:
-        raise SyntaxError, \
-              "Errors were found at line %d while processing %s:\n\t%s"\
-              %(e.lineno,f,e.text)
+        exec(compile(open(f).read(), f, 'exec'), globs, locs)
+    except SyntaxError as e:
+        raise SyntaxError("Errors were found at line %d while processing %s:\n\t%s"\
+              %(e.lineno,f,e.text))
 
     types = [t for t in locs.ordered_values() if isinstance(t,Type)]
 
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH for-next RFC 3/4] pygrub: convert python files with 2to3

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
Not sure this works with python 2.4, but it should work with 2.7 since
the changes look more or less in the same vein as the changes in
libxl.

The conversion of the import is interesting. This definitely needs
some testing.
---
 tools/pygrub/src/ExtLinuxConf.py | 16 ++++++++--------
 tools/pygrub/src/GrubConf.py     | 36 ++++++++++++++++++------------------
 tools/pygrub/src/LiloConf.py     | 16 ++++++++--------
 3 files changed, 34 insertions(+), 34 deletions(-)

diff --git a/tools/pygrub/src/ExtLinuxConf.py b/tools/pygrub/src/ExtLinuxConf.py
index d1789bf020..60da960c4b 100644
--- a/tools/pygrub/src/ExtLinuxConf.py
+++ b/tools/pygrub/src/ExtLinuxConf.py
@@ -12,7 +12,7 @@
 
 import sys, re, os
 import logging
-import GrubConf
+from . import GrubConf
 
 class ExtLinuxImage(object):
     def __init__(self, lines, path):
@@ -32,7 +32,7 @@ class ExtLinuxImage(object):
         self.lines = []
         self.path = path
         self.root = ""
-        map(self.set_from_line, lines)
+        list(map(self.set_from_line, lines))
 
     def set_from_line(self, line, replace = None):
         (com, arg) = GrubConf.grub_exact_split(line, 2)
@@ -67,7 +67,7 @@ class ExtLinuxImage(object):
                         setattr(self, "initrd", a.replace("initrd=", ""))
                         arg = arg.replace(a, "")
 
-        if com is not None and self.commands.has_key(com):
+        if com is not None and com in self.commands:
             if self.commands[com] is not None:
                 setattr(self, self.commands[com], re.sub('^"(.+)"$', r"\1", arg.strip()))
             else:
@@ -136,7 +136,7 @@ class ExtLinuxConfigFile(object):
     def parse(self, buf = None):
         if buf is None:
             if self.filename is None:
-                raise ValueError, "No config file defined to parse!"
+                raise ValueError("No config file defined to parse!")
 
             f = open(self.filename, 'r')
             lines = f.readlines()
@@ -167,7 +167,7 @@ class ExtLinuxConfigFile(object):
 
             (com, arg) = GrubConf.grub_exact_split(l, 2)
             com = com.lower()
-            if self.commands.has_key(com):
+            if com in self.commands:
                 if self.commands[com] is not None:
                     setattr(self, self.commands[com], arg.strip())
                 else:
@@ -207,8 +207,8 @@ class ExtLinuxConfigFile(object):
         
 if __name__ == "__main__":
     if len(sys.argv) < 2:
-        raise RuntimeError, "Need a configuration file to read"
+        raise RuntimeError("Need a configuration file to read")
     g = ExtLinuxConfigFile(sys.argv[1])
     for i in g.images:
-        print i
-    print g.default
+        print(i)
+    print(g.default)
diff --git a/tools/pygrub/src/GrubConf.py b/tools/pygrub/src/GrubConf.py
index dc810d55cb..17f6e686cc 100644
--- a/tools/pygrub/src/GrubConf.py
+++ b/tools/pygrub/src/GrubConf.py
@@ -44,7 +44,7 @@ def get_path(s):
         return (None, s)
     idx = s.find(')')
     if idx == -1:
-        raise ValueError, "Unable to find matching ')'"
+        raise ValueError("Unable to find matching ')'")
     d = s[:idx]
     return (GrubDiskPart(d), s[idx + 1:])
 
@@ -100,7 +100,7 @@ class _GrubImage(object):
                 "  initrd: %s\n" %(self.title, self.root, self.kernel,
                                    self.args, self.initrd))
     def _parse(self, lines):
-        map(self.set_from_line, lines)
+        list(map(self.set_from_line, lines))
 
     def reset(self, lines):
         self._root = self._initrd = self._kernel = self._args = None
@@ -141,7 +141,7 @@ class GrubImage(_GrubImage):
     def set_from_line(self, line, replace = None):
         (com, arg) = grub_exact_split(line, 2)
 
-        if self.commands.has_key(com):
+        if com in self.commands:
             if self.commands[com] is not None:
                 setattr(self, self.commands[com], arg.strip())
             else:
@@ -177,7 +177,7 @@ class _GrubConfigFile(object):
             self.parse()
 
     def parse(self, buf = None):
-        raise RuntimeError, "unimplemented parse function"   
+        raise RuntimeError("unimplemented parse function")   
 
     def hasPasswordAccess(self):
         return self.passwordAccess
@@ -201,7 +201,7 @@ class _GrubConfigFile(object):
                 import crypt
                 if crypt.crypt(password, pwd[1]) == pwd[1]:
                     return True
-            except Exception, e:
+            except Exception as e:
                 self.passExc = "Can't verify password: %s" % str(e)
                 return False
 
@@ -213,7 +213,7 @@ class _GrubConfigFile(object):
 
     def set(self, line):
         (com, arg) = grub_exact_split(line, 2)
-        if self.commands.has_key(com):
+        if com in self.commands:
             if self.commands[com] is not None:
                 setattr(self, self.commands[com], arg.strip())
             else:
@@ -233,7 +233,7 @@ class _GrubConfigFile(object):
             self._default = val
 
         if self._default < 0:
-            raise ValueError, "default must be positive number"
+            raise ValueError("default must be positive number")
     default = property(_get_default, _set_default)
 
     def set_splash(self, val):
@@ -265,7 +265,7 @@ class GrubConfigFile(_GrubConfigFile):
     def parse(self, buf = None):
         if buf is None:
             if self.filename is None:
-                raise ValueError, "No config file defined to parse!"
+                raise ValueError("No config file defined to parse!")
 
             f = open(self.filename, 'r')
             lines = f.readlines()
@@ -296,7 +296,7 @@ class GrubConfigFile(_GrubConfigFile):
                 continue
 
             (com, arg) = grub_exact_split(l, 2)
-            if self.commands.has_key(com):
+            if com in self.commands:
                 if self.commands[com] is not None:
                     setattr(self, self.commands[com], arg.strip())
                 else:
@@ -328,7 +328,7 @@ class Grub2Image(_GrubImage):
         if com == "set":
             (com,arg) = grub2_handle_set(arg)
             
-        if self.commands.has_key(com):
+        if com in self.commands:
             if self.commands[com] is not None:
                 setattr(self, self.commands[com], arg.strip())
             else:
@@ -364,7 +364,7 @@ class Grub2ConfigFile(_GrubConfigFile):
     def parse(self, buf = None):
         if buf is None:
             if self.filename is None:
-                raise ValueError, "No config file defined to parse!"
+                raise ValueError("No config file defined to parse!")
 
             f = open(self.filename, 'r')
             lines = f.readlines()
@@ -398,7 +398,7 @@ class Grub2ConfigFile(_GrubConfigFile):
             title_match = re.match('^menuentry ["\'](.*?)["\'] (.*){', l)
             if title_match:
                 if img is not None:
-                    raise RuntimeError, "syntax error: cannot nest menuentry (%d %s)" % (len(img),img)
+                    raise RuntimeError("syntax error: cannot nest menuentry (%d %s)" % (len(img),img))
                 img = []
                 title = title_match.group(1)
                 continue
@@ -413,7 +413,7 @@ class Grub2ConfigFile(_GrubConfigFile):
                         menu_level -= 1
                         continue
                     else:
-                        raise RuntimeError, "syntax error: closing brace without menuentry"
+                        raise RuntimeError("syntax error: closing brace without menuentry")
 
                 self.add_image(Grub2Image(title, img))
                 img = None
@@ -428,7 +428,7 @@ class Grub2ConfigFile(_GrubConfigFile):
             if com == "set":
                 (com,arg) = grub2_handle_set(arg)
                 
-            if self.commands.has_key(com):
+            if com in self.commands:
                 if self.commands[com] is not None:
                     arg_strip = arg.strip()
                     if arg_strip == "${saved_entry}" or arg_strip == "${next_entry}":
@@ -443,7 +443,7 @@ class Grub2ConfigFile(_GrubConfigFile):
                 logging.warning("Unknown directive %s" %(com,))
             
         if img is not None:
-            raise RuntimeError, "syntax error: end of file with open menuentry(%d %s)" % (len(img),img)
+            raise RuntimeError("syntax error: end of file with open menuentry(%d %s)" % (len(img),img))
 
         if self.hasPassword():
             self.setPasswordAccess(False)
@@ -462,12 +462,12 @@ class Grub2ConfigFile(_GrubConfigFile):
         
 if __name__ == "__main__":
     if len(sys.argv) < 3:
-        raise RuntimeError, "Need a grub version (\"grub\" or \"grub2\") and a grub.conf or grub.cfg to read"
+        raise RuntimeError("Need a grub version (\"grub\" or \"grub2\") and a grub.conf or grub.cfg to read")
     if sys.argv[1] == "grub":
         g = GrubConfigFile(sys.argv[2])
     elif sys.argv[1] == "grub2":
         g = Grub2ConfigFile(sys.argv[2])
     else:
-        raise RuntimeError, "Unknown config type %s" % sys.argv[1]
+        raise RuntimeError("Unknown config type %s" % sys.argv[1])
     for i in g.images:
-        print i #, i.title, i.root, i.kernel, i.args, i.initrd
+        print(i) #, i.title, i.root, i.kernel, i.args, i.initrd
diff --git a/tools/pygrub/src/LiloConf.py b/tools/pygrub/src/LiloConf.py
index 2cb649f115..8c080e697c 100644
--- a/tools/pygrub/src/LiloConf.py
+++ b/tools/pygrub/src/LiloConf.py
@@ -4,7 +4,7 @@
 
 import sys, re, os
 import logging
-import GrubConf
+from . import GrubConf
 
 class LiloImage(object):
     def __init__(self, lines, path):
@@ -24,12 +24,12 @@ class LiloImage(object):
         self.lines = []
         self.path = path
         self.root = ""
-        map(self.set_from_line, lines)
+        list(map(self.set_from_line, lines))
 
     def set_from_line(self, line, replace = None):
         (com, arg) = GrubConf.grub_exact_split(line, 2)
 
-        if self.commands.has_key(com):
+        if com in self.commands:
             if self.commands[com] is not None:
                 setattr(self, self.commands[com], re.sub('^"(.+)"$', r"\1", arg.strip()))
             else:
@@ -97,7 +97,7 @@ class LiloConfigFile(object):
     def parse(self, buf = None):
         if buf is None:
             if self.filename is None:
-                raise ValueError, "No config file defined to parse!"
+                raise ValueError("No config file defined to parse!")
 
             f = open(self.filename, 'r')
             lines = f.readlines()
@@ -127,7 +127,7 @@ class LiloConfigFile(object):
                 continue
 
             (com, arg) = GrubConf.grub_exact_split(l, 2)
-            if self.commands.has_key(com):
+            if com in self.commands:
                 if self.commands[com] is not None:
                     setattr(self, self.commands[com], arg.strip())
                 else:
@@ -170,8 +170,8 @@ class LiloConfigFile(object):
 
 if __name__ == "__main__":
     if len(sys.argv) < 2:
-        raise RuntimeError, "Need a lilo.conf to read"
+        raise RuntimeError("Need a lilo.conf to read")
     g = LiloConfigFile(sys.argv[1])
     for i in g.images:
-        print i #, i.title, i.root, i.kernel, i.args, i.initrd
-    print g.default
+        print(i) #, i.title, i.root, i.kernel, i.args, i.initrd
+    print(g.default)
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH for-next RFC 4/4] pygrub: make it build with python 3

[Wei Liu]

With the help of two porting guides and cpython source code:

1. Use PyUnicode to replace PyString counterparts.
2. Use PyVarObject_HEAD_INIT and provide compatibility for 2.5 and
   earlier.
3. Remove usage of Py_FindMethod.
4. Use new module initialisation routine.

For #3, Py_FindMethod was removed, yet an alternative wasn't
documented.  The code is the result of reverse-engineering cpython
commit 6116d4a1d1

https://docs.python.org/3/howto/cporting.html
http://python3porting.com/cextensions.html

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
XXX Build test only!
---
 tools/pygrub/src/fsimage/fsimage.c | 92 ++++++++++++++++++++++++++++++++++----
 1 file changed, 84 insertions(+), 8 deletions(-)

diff --git a/tools/pygrub/src/fsimage/fsimage.c b/tools/pygrub/src/fsimage/fsimage.c
index 743a3fb7b8..c914253741 100644
--- a/tools/pygrub/src/fsimage/fsimage.c
+++ b/tools/pygrub/src/fsimage/fsimage.c
@@ -26,11 +26,15 @@
 #include <xenfsimage.h>
 #include <stdlib.h>
 
+#if PY_MAJOR_VERSION < 3
 #if (PYTHON_API_VERSION >= 1011)
 #define PY_PAD 0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L
 #else
 #define PY_PAD 0L,0L,0L,0L
 #endif
+#else
+#define PY_PAD 0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L
+#endif
 
 typedef struct fsimage_fs {
 	PyObject_HEAD
@@ -66,12 +70,24 @@ fsimage_file_read(fsimage_file_t *file, PyObject *args, PyObject *kwargs)
 
 	bufsize = size ? size : 4096;
 
-	if ((buffer = PyString_FromStringAndSize(NULL, bufsize)) == NULL)
+	buffer =
+#if PY_MAJOR_VERSION < 3
+		PyString_FromStringAndSize(NULL, bufsize);
+#else
+		PyUnicode_FromStringAndSize(NULL, bufsize);
+#endif
+
+	if (buffer == NULL)
 		return (NULL);
  
 	while (1) {
 		int err;
-		void *buf = PyString_AS_STRING(buffer) + bytesread;
+		void *buf =
+#if PY_MAJOR_VERSION < 3
+			PyString_AS_STRING(buffer) + bytesread;
+#else
+			PyUnicode_AS_UNICODE(buffer) + bytesread;
+#endif
 
 		err = fsi_pread_file(file->file, buf, bufsize,
 		    bytesread + offset);
@@ -91,12 +107,20 @@ fsimage_file_read(fsimage_file_t *file, PyObject *args, PyObject *kwargs)
 			if (bufsize == 0)
 				break;
 		} else {
+#if PY_MAJOR_VERSION < 3
 			if (_PyString_Resize(&buffer, bytesread + bufsize) < 0)
+#else
+			if (PyUnicode_Resize(&buffer, bytesread + bufsize) < 0)
+#endif
 				return (NULL);
 		}
 	}
 
+#if PY_MAJOR_VERSION < 3
 	_PyString_Resize(&buffer, bytesread);
+#else
+	PyUnicode_Resize(&buffer, bytesread);
+#endif
 	return (buffer);
 }
 
@@ -113,11 +137,13 @@ static struct PyMethodDef fsimage_file_methods[] = {
 	{ NULL, NULL, 0, NULL }	
 };
 
+#if PY_MAJOR_VERSION < 3
 static PyObject *
 fsimage_file_getattr(fsimage_file_t *file, char *name)
 {
 	return (Py_FindMethod(fsimage_file_methods, (PyObject *)file, name));
 }
+#endif
 
 static void
 fsimage_file_dealloc(fsimage_file_t *file)
@@ -128,16 +154,25 @@ fsimage_file_dealloc(fsimage_file_t *file)
 	PyObject_DEL(file);
 }
 
+/* Compatibility for 2.5 and earlier */
+#ifndef PyVarObject_HEAD_INIT
+#define PyVarObject_HEAD_INIT(type, size) \
+	PyObject_HEAD_INIT(type) size,
+#endif
+
 static char fsimage_file_type__doc__[] = "Filesystem image file";
 PyTypeObject fsimage_file_type = {
-	PyObject_HEAD_INIT(&PyType_Type)
-	0,					/* ob_size */
+	PyVarObject_HEAD_INIT(&PyType_Type, 0)
 	"xenfsimage.file",			/* tp_name */
 	sizeof(fsimage_file_t),			/* tp_size */
 	0,					/* tp_itemsize */
 	(destructor) fsimage_file_dealloc, 	/* tp_dealloc */
 	0,					/* tp_print */
+#if PY_MAJOR_VERSION < 3
 	(getattrfunc) fsimage_file_getattr, 	/* tp_getattr */
+#else
+	0,					/* tp_getattr */
+#endif
 	0,					/* tp_setattr */
 	0,					/* tp_compare */
 	0,					/* tp_repr */
@@ -151,7 +186,16 @@ PyTypeObject fsimage_file_type = {
 	0,					/* tp_setattro */
 	0,					/* tp_as_buffer */
 	Py_TPFLAGS_DEFAULT,			/* tp_flags */
-	fsimage_file_type__doc__,
+	fsimage_file_type__doc__,		/* tp_doc */
+#if PY_MAJOR_VERSION >= 3
+	0,					/* tp_traverse */
+	0,					/* tp_clear */
+	0,					/* tp_richcompare */
+	0,					/* tp_weaklistoffset */
+	0,					/* tp_iter */
+	0,					/* tp_iternext */
+	fsimage_file_methods,			/* tp_methods */
+#endif
 	PY_PAD
 };
 
@@ -215,11 +259,13 @@ static struct PyMethodDef fsimage_fs_methods[] = {
 	{ NULL, NULL, 0, NULL }	
 };
 
+#if PY_MAJOR_VERSION < 3
 static PyObject *
 fsimage_fs_getattr(fsimage_fs_t *fs, char *name)
 {
 	return (Py_FindMethod(fsimage_fs_methods, (PyObject *)fs, name));
 }
+#endif
 
 static void
 fsimage_fs_dealloc (fsimage_fs_t *fs)
@@ -232,14 +278,17 @@ fsimage_fs_dealloc (fsimage_fs_t *fs)
 PyDoc_STRVAR(fsimage_fs_type__doc__, "Filesystem image");
 
 PyTypeObject fsimage_fs_type = {
-	PyObject_HEAD_INIT(&PyType_Type)
-	0,					/* ob_size */
+	PyVarObject_HEAD_INIT(&PyType_Type, 0)
 	"xenfsimage.fs",			/* tp_name */
 	sizeof(fsimage_fs_t),			/* tp_size */
 	0,					/* tp_itemsize */
 	(destructor) fsimage_fs_dealloc, 	/* tp_dealloc */
 	0,					/* tp_print */
+#if PY_MAJOR_VERSION < 3
 	(getattrfunc) fsimage_fs_getattr, 	/* tp_getattr */
+#else
+	0,					/* tp_getattr */
+#endif
 	0,					/* tp_setattr */
 	0,					/* tp_compare */
 	0,					/* tp_repr */
@@ -253,7 +302,16 @@ PyTypeObject fsimage_fs_type = {
 	0,					/* tp_setattro */
 	0,					/* tp_as_buffer */
 	Py_TPFLAGS_DEFAULT,			/* tp_flags */
-	fsimage_fs_type__doc__,
+	fsimage_fs_type__doc__,			/* tp_doc */
+#if PY_MAJOR_VERSION >= 3
+	0,					/* tp_traverse */
+	0,					/* tp_clear */
+	0,					/* tp_richcompare */
+	0,					/* tp_weaklistoffset */
+	0,					/* tp_iter */
+	0,					/* tp_iternext */
+	fsimage_fs_methods,			/* tp_methods */
+#endif
 	PY_PAD
 };
 
@@ -316,8 +374,26 @@ static struct PyMethodDef fsimage_module_methods[] = {
 	{ NULL, NULL, 0, NULL }
 };
 
+#if PY_MAJOR_VERSION >= 3
+static struct PyModuleDef fsimage_module_def = {
+	PyModuleDef_HEAD_INIT,
+	"xenfsimage",		/* m_name */
+	"",			/* m_doc */
+	-1,			/* m_size */
+	fsimage_module_methods,	/* m_methods */
+	NULL,			/* m_reload */
+	NULL,			/* m_traverse */
+	NULL,			/* m_clear */
+	NULL			/* m_free */
+};
+#endif
+
 PyMODINIT_FUNC
 initxenfsimage(void)
 {
+#if PY_MAJOR_VERSION < 3
 	Py_InitModule("xenfsimage", fsimage_module_methods);
+#else
+	return PyModule_Create(&fsimage_module_def);
+#endif
 }
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 2/4] libxl: make python scripts work with python 2 and 3

[Andrew Cooper]

On 05/03/2019 16:42, Wei Liu wrote:
> All scripts are transformed by 2to3.
>
> The only addition is "from __future__ import print_function" so that
> print("BLAH", file=sys.stderr) can work.
>
> https://python-future.org/compatible_idioms.html
>
> Tested with 2.7 and 3.5.
>
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
> I don't have environment to test 2.4 -- it is almost 15 years old. We
> may want to consider bumping the minimum requirement to 2.7?

The compatible way to do this is  sys.stderr.write(msg + "\n") and using
print() without the future import.

> @@ -269,7 +271,7 @@ class KeyedUnion(Aggregate):
>          if not isinstance(keyvar_type, Enumeration):
>              raise ValueError
>  
> -        kv_kwargs = dict([(x.lstrip('keyvar_'),y) for (x,y) in kwargs.items() if x.startswith('keyvar_')])
> +        kv_kwargs = dict([(x.lstrip('keyvar_'),y) for (x,y) in list(kwargs.items()) if x.startswith('keyvar_')])

This shouldn't need changing.  List comprehensions are one of the few
uses of .items() which is compatible with older versions of python IIRC.

> @@ -362,11 +364,10 @@ def parse(f):
>              globs[n] = t
>  
>      try:
> -        execfile(f, globs, locs)
> -    except SyntaxError,e:
> -        raise SyntaxError, \
> -              "Errors were found at line %d while processing %s:\n\t%s"\
> -              %(e.lineno,f,e.text)
> +        exec(compile(open(f).read(), f, 'exec'), globs, locs)
> +    except SyntaxError as e:

This is the only really awkward bit, and isn't Py 2.4 compatible.

The only option here to retain pre 2.6 compatibility is:

try:
    ...
except SyntaxError:
    _, e = sys.exc_info()[:2]
    ...

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 4/4] pygrub: make it build with python 3

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 1661 bytes --]

On 05/03/2019 16:42, Wei Liu wrote:
> With the help of two porting guides and cpython source code:
>
> 1. Use PyUnicode to replace PyString counterparts.
> 2. Use PyVarObject_HEAD_INIT and provide compatibility for 2.5 and
>    earlier.
> 3. Remove usage of Py_FindMethod.
> 4. Use new module initialisation routine.
>
> For #3, Py_FindMethod was removed, yet an alternative wasn't
> documented.  The code is the result of reverse-engineering cpython
> commit 6116d4a1d1
>
> https://docs.python.org/3/howto/cporting.html
> http://python3porting.com/cextensions.html
>
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>

Marek already made the tools/python/* libraries compatible with Py2 and Py3

The following commits are the relevant ones:

* be6b316 - python: handle long type in scripts (2 years ago) <Marek Marczykowski-Górecki>
* e16c705 - python: adjust module initalization for Python3 (2 years ago) <Marek Marczykowski-Górecki>
* dd986cd - python: use PyLong_* for constructing 'int' type in Python3 (2 years ago) <Marek Marczykowski-Górecki>
* 121d9d4 - python: use PyBytes/PyUnicode instead of PyString (2 years ago) <Marek Marczykowski-Górecki>
* 0c8981f - python: initialize specific fields of PyTypeObject (2 years ago) <Marek Marczykowski-Górecki>
* 7b1e5f7 - python: use Py_TYPE instead of looking directly into PyObject_HEAD (2 years ago) <Marek Marczykowski-Górecki>
* 96d1ee6 - python: drop tp_getattr implementation (2 years ago) <Marek Marczykowski-Górecki>
* 6b28df3 - python: check return value of PyErr_NewException (2 years ago) <Marek Marczykowski-Górecki>

Which in particular handle strings differently in the Py2 case.

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 2378 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 4/4] pygrub: make it build with python 3

[Wei Liu]

On Tue, Mar 05, 2019 at 05:42:07PM +0000, Andrew Cooper wrote:
> On 05/03/2019 16:42, Wei Liu wrote:
> > With the help of two porting guides and cpython source code:
> >
> > 1. Use PyUnicode to replace PyString counterparts.
> > 2. Use PyVarObject_HEAD_INIT and provide compatibility for 2.5 and
> >    earlier.
> > 3. Remove usage of Py_FindMethod.
> > 4. Use new module initialisation routine.
> >
> > For #3, Py_FindMethod was removed, yet an alternative wasn't
> > documented.  The code is the result of reverse-engineering cpython
> > commit 6116d4a1d1
> >
> > https://docs.python.org/3/howto/cporting.html
> > http://python3porting.com/cextensions.html
> >
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> 
> Marek already made the tools/python/* libraries compatible with Py2 and Py3
> 
> The following commits are the relevant ones:
> 
> * be6b316 - python: handle long type in scripts (2 years ago) <Marek Marczykowski-Górecki>
> * e16c705 - python: adjust module initalization for Python3 (2 years ago) <Marek Marczykowski-Górecki>
> * dd986cd - python: use PyLong_* for constructing 'int' type in Python3 (2 years ago) <Marek Marczykowski-Górecki>
> * 121d9d4 - python: use PyBytes/PyUnicode instead of PyString (2 years ago) <Marek Marczykowski-Górecki>
> * 0c8981f - python: initialize specific fields of PyTypeObject (2 years ago) <Marek Marczykowski-Górecki>
> * 7b1e5f7 - python: use Py_TYPE instead of looking directly into PyObject_HEAD (2 years ago) <Marek Marczykowski-Górecki>
> * 96d1ee6 - python: drop tp_getattr implementation (2 years ago) <Marek Marczykowski-Górecki>
> * 6b28df3 - python: check return value of PyErr_NewException (2 years ago) <Marek Marczykowski-Górecki>

I knew.

> 
> Which in particular handle strings differently in the Py2 case.


I am not sure his changes for the string APIs are correct -- they seem
to deviate from the official porting guide. But hey, I don't use these
bindings myself, so he probably knows better.

CC Marek.

Wei.

> 
> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 3/4] pygrub: convert python files with 2to3

[Andrew Cooper]

On 05/03/2019 16:42, Wei Liu wrote:
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
> Not sure this works with python 2.4, but it should work with 2.7 since
> the changes look more or less in the same vein as the changes in
> libxl.
>
> The conversion of the import is interesting. This definitely needs
> some testing.
> ---
>  tools/pygrub/src/ExtLinuxConf.py | 16 ++++++++--------
>  tools/pygrub/src/GrubConf.py     | 36 ++++++++++++++++++------------------
>  tools/pygrub/src/LiloConf.py     | 16 ++++++++--------
>  3 files changed, 34 insertions(+), 34 deletions(-)
>
> diff --git a/tools/pygrub/src/ExtLinuxConf.py b/tools/pygrub/src/ExtLinuxConf.py
> index d1789bf020..60da960c4b 100644
> --- a/tools/pygrub/src/ExtLinuxConf.py
> +++ b/tools/pygrub/src/ExtLinuxConf.py
> @@ -12,7 +12,7 @@
>  
>  import sys, re, os
>  import logging
> -import GrubConf
> +from . import GrubConf

Relative imports definitely don't exist in Py 2.4

>  
>  class ExtLinuxImage(object):
>      def __init__(self, lines, path):
> @@ -32,7 +32,7 @@ class ExtLinuxImage(object):
>          self.lines = []
>          self.path = path
>          self.root = ""
> -        map(self.set_from_line, lines)
> +        list(map(self.set_from_line, lines))

This an abuse of map() in the first place, but the automatic
transformation makes the result even more confusing.

Instead, I'd rewrite the code as

for line in lines:
    self.set_from_line(line)

which makes it clear what is actually going on.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 4/4] pygrub: make it build with python 3

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 3294 bytes --]

On Tue, Mar 05, 2019 at 05:48:10PM +0000, Wei Liu wrote:
> On Tue, Mar 05, 2019 at 05:42:07PM +0000, Andrew Cooper wrote:
> > On 05/03/2019 16:42, Wei Liu wrote:
> > > With the help of two porting guides and cpython source code:
> > >
> > > 1. Use PyUnicode to replace PyString counterparts.
> > > 2. Use PyVarObject_HEAD_INIT and provide compatibility for 2.5 and
> > >    earlier.
> > > 3. Remove usage of Py_FindMethod.
> > > 4. Use new module initialisation routine.
> > >
> > > For #3, Py_FindMethod was removed, yet an alternative wasn't
> > > documented.  The code is the result of reverse-engineering cpython
> > > commit 6116d4a1d1
> > >
> > > https://docs.python.org/3/howto/cporting.html
> > > http://python3porting.com/cextensions.html
> > >
> > > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> > 
> > Marek already made the tools/python/* libraries compatible with Py2 and Py3
> > 
> > The following commits are the relevant ones:
> > 
> > * be6b316 - python: handle long type in scripts (2 years ago) <Marek Marczykowski-Górecki>
> > * e16c705 - python: adjust module initalization for Python3 (2 years ago) <Marek Marczykowski-Górecki>
> > * dd986cd - python: use PyLong_* for constructing 'int' type in Python3 (2 years ago) <Marek Marczykowski-Górecki>
> > * 121d9d4 - python: use PyBytes/PyUnicode instead of PyString (2 years ago) <Marek Marczykowski-Górecki>
> > * 0c8981f - python: initialize specific fields of PyTypeObject (2 years ago) <Marek Marczykowski-Górecki>
> > * 7b1e5f7 - python: use Py_TYPE instead of looking directly into PyObject_HEAD (2 years ago) <Marek Marczykowski-Górecki>
> > * 96d1ee6 - python: drop tp_getattr implementation (2 years ago) <Marek Marczykowski-Górecki>
> > * 6b28df3 - python: check return value of PyErr_NewException (2 years ago) <Marek Marczykowski-Górecki>
> 
> I knew.
> 
> > 
> > Which in particular handle strings differently in the Py2 case.
> 
> 
> I am not sure his changes for the string APIs are correct -- they seem
> to deviate from the official porting guide. But hey, I don't use these
> bindings myself, so he probably knows better.

That was intentional, because in py2 str type is the same as bytes
types, and in fact some of those str should really be bytes. It's in the
commit message:

    python: use PyBytes/PyUnicode instead of PyString
    
    In Python2 PyBytes is the same as PyString, but in Python3 PyString is
    gone and 'str' is really PyUnicode in C-API.
    When handling arbitrary data, use PyBytes - which is the right thing to
    do in Python3, and pose no API change in Python2. When handling
    xenstore paths and transaction ids, which have well defined format, use
    PyUnicode - to ease API usage - no need to prefix all xenstore paths
    with 'b' when migrating scripts to Python3.

I'm not sure if the same reasoning applies to pygrub, but I guess it
may. For example fsimage_file_read sounds like handling binary data, not
really UTF-8 strings. Using PyUnicode for arbitrary binary data may lead
to various UnicodeDecodeErrors.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 3/4] pygrub: convert python files with 2to3

[Wei Liu]

On Tue, Mar 05, 2019 at 05:51:04PM +0000, Andrew Cooper wrote:
> On 05/03/2019 16:42, Wei Liu wrote:
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> > ---
> > Not sure this works with python 2.4, but it should work with 2.7 since
> > the changes look more or less in the same vein as the changes in
> > libxl.
> >
> > The conversion of the import is interesting. This definitely needs
> > some testing.
> > ---
> >  tools/pygrub/src/ExtLinuxConf.py | 16 ++++++++--------
> >  tools/pygrub/src/GrubConf.py     | 36 ++++++++++++++++++------------------
> >  tools/pygrub/src/LiloConf.py     | 16 ++++++++--------
> >  3 files changed, 34 insertions(+), 34 deletions(-)
> >
> > diff --git a/tools/pygrub/src/ExtLinuxConf.py b/tools/pygrub/src/ExtLinuxConf.py
> > index d1789bf020..60da960c4b 100644
> > --- a/tools/pygrub/src/ExtLinuxConf.py
> > +++ b/tools/pygrub/src/ExtLinuxConf.py
> > @@ -12,7 +12,7 @@
> >  
> >  import sys, re, os
> >  import logging
> > -import GrubConf
> > +from . import GrubConf
> 
> Relative imports definitely don't exist in Py 2.4
> 
> >  
> >  class ExtLinuxImage(object):
> >      def __init__(self, lines, path):
> > @@ -32,7 +32,7 @@ class ExtLinuxImage(object):
> >          self.lines = []
> >          self.path = path
> >          self.root = ""
> > -        map(self.set_from_line, lines)
> > +        list(map(self.set_from_line, lines))
> 
> This an abuse of map() in the first place, but the automatic
> transformation makes the result even more confusing.

Right. I tried to find the justification for this transformation but the
document doesn't provide that.

I will drop this and the relative import -- I _think_ the original code
should still work with python 3.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 3/4] pygrub: convert python files with 2to3

[Andrew Cooper]

On 06/03/2019 11:31, Wei Liu wrote:
> On Tue, Mar 05, 2019 at 05:51:04PM +0000, Andrew Cooper wrote:
>> On 05/03/2019 16:42, Wei Liu wrote:
>>> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
>>> ---
>>> Not sure this works with python 2.4, but it should work with 2.7 since
>>> the changes look more or less in the same vein as the changes in
>>> libxl.
>>>
>>> The conversion of the import is interesting. This definitely needs
>>> some testing.
>>> ---
>>>  tools/pygrub/src/ExtLinuxConf.py | 16 ++++++++--------
>>>  tools/pygrub/src/GrubConf.py     | 36 ++++++++++++++++++------------------
>>>  tools/pygrub/src/LiloConf.py     | 16 ++++++++--------
>>>  3 files changed, 34 insertions(+), 34 deletions(-)
>>>
>>> diff --git a/tools/pygrub/src/ExtLinuxConf.py b/tools/pygrub/src/ExtLinuxConf.py
>>> index d1789bf020..60da960c4b 100644
>>> --- a/tools/pygrub/src/ExtLinuxConf.py
>>> +++ b/tools/pygrub/src/ExtLinuxConf.py
>>> @@ -12,7 +12,7 @@
>>>  
>>>  import sys, re, os
>>>  import logging
>>> -import GrubConf
>>> +from . import GrubConf
>> Relative imports definitely don't exist in Py 2.4
>>
>>>  
>>>  class ExtLinuxImage(object):
>>>      def __init__(self, lines, path):
>>> @@ -32,7 +32,7 @@ class ExtLinuxImage(object):
>>>          self.lines = []
>>>          self.path = path
>>>          self.root = ""
>>> -        map(self.set_from_line, lines)
>>> +        list(map(self.set_from_line, lines))
>> This an abuse of map() in the first place, but the automatic
>> transformation makes the result even more confusing.
> Right. I tried to find the justification for this transformation but the
> document doesn't provide that.

The expected use of map is in the form:

x = map(fn, y)

which would leave x as a list in Py2, and a generator in Py3.

In most code, wrapping map with list() is the correct transformation to
make, because a) a lot of code written for Py2 expects it to be a list
and b) you cant programmatically evaluate whether leaving it in its
generator form is safe in context.

For this piece of code (and the other similar examples), map() is not
the correct construct to use in the first place, and probably wants
fixing for clarity alone, irrespective of the Py3 transformation.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 4/4] pygrub: make it build with python 3

[Wei Liu]

On Tue, Mar 05, 2019 at 07:17:07PM +0100, Marek Marczykowski-Górecki wrote:
> On Tue, Mar 05, 2019 at 05:48:10PM +0000, Wei Liu wrote:
> > On Tue, Mar 05, 2019 at 05:42:07PM +0000, Andrew Cooper wrote:
> > > On 05/03/2019 16:42, Wei Liu wrote:
> > > > With the help of two porting guides and cpython source code:
> > > >
> > > > 1. Use PyUnicode to replace PyString counterparts.
> > > > 2. Use PyVarObject_HEAD_INIT and provide compatibility for 2.5 and
> > > >    earlier.
> > > > 3. Remove usage of Py_FindMethod.
> > > > 4. Use new module initialisation routine.
> > > >
> > > > For #3, Py_FindMethod was removed, yet an alternative wasn't
> > > > documented.  The code is the result of reverse-engineering cpython
> > > > commit 6116d4a1d1
> > > >
> > > > https://docs.python.org/3/howto/cporting.html
> > > > http://python3porting.com/cextensions.html
> > > >
> > > > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> > > 
> > > Marek already made the tools/python/* libraries compatible with Py2 and Py3
> > > 
> > > The following commits are the relevant ones:
> > > 
> > > * be6b316 - python: handle long type in scripts (2 years ago) <Marek Marczykowski-Górecki>
> > > * e16c705 - python: adjust module initalization for Python3 (2 years ago) <Marek Marczykowski-Górecki>
> > > * dd986cd - python: use PyLong_* for constructing 'int' type in Python3 (2 years ago) <Marek Marczykowski-Górecki>
> > > * 121d9d4 - python: use PyBytes/PyUnicode instead of PyString (2 years ago) <Marek Marczykowski-Górecki>
> > > * 0c8981f - python: initialize specific fields of PyTypeObject (2 years ago) <Marek Marczykowski-Górecki>
> > > * 7b1e5f7 - python: use Py_TYPE instead of looking directly into PyObject_HEAD (2 years ago) <Marek Marczykowski-Górecki>
> > > * 96d1ee6 - python: drop tp_getattr implementation (2 years ago) <Marek Marczykowski-Górecki>
> > > * 6b28df3 - python: check return value of PyErr_NewException (2 years ago) <Marek Marczykowski-Górecki>
> > 
> > I knew.
> > 
> > > 
> > > Which in particular handle strings differently in the Py2 case.
> > 
> > 
> > I am not sure his changes for the string APIs are correct -- they seem
> > to deviate from the official porting guide. But hey, I don't use these
> > bindings myself, so he probably knows better.
> 
> That was intentional, because in py2 str type is the same as bytes
> types, and in fact some of those str should really be bytes. It's in the
> commit message:
> 
>     python: use PyBytes/PyUnicode instead of PyString
>     
>     In Python2 PyBytes is the same as PyString, but in Python3 PyString is
>     gone and 'str' is really PyUnicode in C-API.
>     When handling arbitrary data, use PyBytes - which is the right thing to
>     do in Python3, and pose no API change in Python2. When handling
>     xenstore paths and transaction ids, which have well defined format, use
>     PyUnicode - to ease API usage - no need to prefix all xenstore paths
>     with 'b' when migrating scripts to Python3.
> 
> I'm not sure if the same reasoning applies to pygrub, but I guess it
> may. For example fsimage_file_read sounds like handling binary data, not
> really UTF-8 strings. Using PyUnicode for arbitrary binary data may lead
> to various UnicodeDecodeErrors.

Good point. Let me dig into this a bit more. Frankly I trust you more
than I trust myself on this matter. :-)

Wei.

> 
> -- 
> Best Regards,
> Marek Marczykowski-Górecki
> Invisible Things Lab
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 3/4] pygrub: convert python files with 2to3

[Wei Liu]

On Wed, Mar 06, 2019 at 11:46:24AM +0000, Andrew Cooper wrote:
> On 06/03/2019 11:31, Wei Liu wrote:
> > On Tue, Mar 05, 2019 at 05:51:04PM +0000, Andrew Cooper wrote:
> >> On 05/03/2019 16:42, Wei Liu wrote:
> >>> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> >>> ---
> >>> Not sure this works with python 2.4, but it should work with 2.7 since
> >>> the changes look more or less in the same vein as the changes in
> >>> libxl.
> >>>
> >>> The conversion of the import is interesting. This definitely needs
> >>> some testing.
> >>> ---
> >>>  tools/pygrub/src/ExtLinuxConf.py | 16 ++++++++--------
> >>>  tools/pygrub/src/GrubConf.py     | 36 ++++++++++++++++++------------------
> >>>  tools/pygrub/src/LiloConf.py     | 16 ++++++++--------
> >>>  3 files changed, 34 insertions(+), 34 deletions(-)
> >>>
> >>> diff --git a/tools/pygrub/src/ExtLinuxConf.py b/tools/pygrub/src/ExtLinuxConf.py
> >>> index d1789bf020..60da960c4b 100644
> >>> --- a/tools/pygrub/src/ExtLinuxConf.py
> >>> +++ b/tools/pygrub/src/ExtLinuxConf.py
> >>> @@ -12,7 +12,7 @@
> >>>  
> >>>  import sys, re, os
> >>>  import logging
> >>> -import GrubConf
> >>> +from . import GrubConf
> >> Relative imports definitely don't exist in Py 2.4
> >>
> >>>  
> >>>  class ExtLinuxImage(object):
> >>>      def __init__(self, lines, path):
> >>> @@ -32,7 +32,7 @@ class ExtLinuxImage(object):
> >>>          self.lines = []
> >>>          self.path = path
> >>>          self.root = ""
> >>> -        map(self.set_from_line, lines)
> >>> +        list(map(self.set_from_line, lines))
> >> This an abuse of map() in the first place, but the automatic
> >> transformation makes the result even more confusing.
> > Right. I tried to find the justification for this transformation but the
> > document doesn't provide that.
> 
> The expected use of map is in the form:
> 
> x = map(fn, y)
> 
> which would leave x as a list in Py2, and a generator in Py3.

Oh, so we should indeed force it to evaluate.

> 
> In most code, wrapping map with list() is the correct transformation to
> make, because a) a lot of code written for Py2 expects it to be a list
> and b) you cant programmatically evaluate whether leaving it in its
> generator form is safe in context.
> 
> For this piece of code (and the other similar examples), map() is not
> the correct construct to use in the first place, and probably wants
> fixing for clarity alone, irrespective of the Py3 transformation.
> 

Sure.

Wei.

> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 0/4] tools: Python 3 compatibility

[George Dunlap]

On 3/5/19 4:42 PM, Wei Liu wrote:
> This series makes tools build with Python 3.
> 
> Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able
> to give people some idea what sort of work is involved.
> 
> You will also need Andrew's "tools/xen-foreign: Update python scripts to be
> Py3 compatible".

Tossing this out there: Given that python2 is (theoretically) EOL in
less than a year, is it worth maintaining compatibility with python2, or
would it be better to just go whole-hog into python3?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 0/4] tools: Python 3 compatibility

[George Dunlap]

On 3/6/19 1:49 PM, George Dunlap wrote:
> On 3/5/19 4:42 PM, Wei Liu wrote:
>> This series makes tools build with Python 3.
>>
>> Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able
>> to give people some idea what sort of work is involved.
>>
>> You will also need Andrew's "tools/xen-foreign: Update python scripts to be
>> Py3 compatible".
> 
> Tossing this out there: Given that python2 is (theoretically) EOL in
> less than a year, is it worth maintaining compatibility with python2, or
> would it be better to just go whole-hog into python3?

I mean -- looking at some of the discussions about how differently
certain kinds of things are interpreted between python2 and python3, it
seems a bit mad to try to write code that works in both: we're just
asking for trouble.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 0/4] tools: Python 3 compatibility

[Wei Liu]

On Wed, Mar 06, 2019 at 01:49:18PM +0000, George Dunlap wrote:
> On 3/5/19 4:42 PM, Wei Liu wrote:
> > This series makes tools build with Python 3.
> > 
> > Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able
> > to give people some idea what sort of work is involved.
> > 
> > You will also need Andrew's "tools/xen-foreign: Update python scripts to be
> > Py3 compatible".
> 
> Tossing this out there: Given that python2 is (theoretically) EOL in
> less than a year, is it worth maintaining compatibility with python2, or
> would it be better to just go whole-hog into python3?

Some enterprise-y distros still ship ancient python AFAICT.

Given that the work involved seems to be manageable so far I would say
let's keep the compatibility for now.

Wei.

> 
>  -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 0/4] tools: Python 3 compatibility

[Wei Liu]

On Wed, Mar 06, 2019 at 01:53:01PM +0000, George Dunlap wrote:
> On 3/6/19 1:49 PM, George Dunlap wrote:
> > On 3/5/19 4:42 PM, Wei Liu wrote:
> >> This series makes tools build with Python 3.
> >>
> >> Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able
> >> to give people some idea what sort of work is involved.
> >>
> >> You will also need Andrew's "tools/xen-foreign: Update python scripts to be
> >> Py3 compatible".
> > 
> > Tossing this out there: Given that python2 is (theoretically) EOL in
> > less than a year, is it worth maintaining compatibility with python2, or
> > would it be better to just go whole-hog into python3?
> 
> I mean -- looking at some of the discussions about how differently
> certain kinds of things are interpreted between python2 and python3, it
> seems a bit mad to try to write code that works in both: we're just
> asking for trouble.

I think Python is not very consistent even within a major version. New
syntax and constructs get added to point releases. Some of the
discussions re code isn't about the differences between 2 and 3.  There
is at least one about 2.4 vs 2.7.

I am all for moving away from 2.4.

Wei.

> 
>  -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 0/4] tools: Python 3 compatibility

[Andrew Cooper]

On 06/03/2019 13:49, George Dunlap wrote:
> On 3/5/19 4:42 PM, Wei Liu wrote:
>> This series makes tools build with Python 3.
>>
>> Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able
>> to give people some idea what sort of work is involved.
>>
>> You will also need Andrew's "tools/xen-foreign: Update python scripts to be
>> Py3 compatible".
> Tossing this out there: Given that python2 is (theoretically) EOL in
> less than a year, is it worth maintaining compatibility with python2, or
> would it be better to just go whole-hog into python3?

Not when we've got supported distros which only ship Py2 by default. 
(CentOS 6 - I'm looking at you in particular...)

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 0/4] tools: Python 3 compatibility

[Anthony PERARD]

On Wed, Mar 06, 2019 at 01:57:55PM +0000, Wei Liu wrote:
> On Wed, Mar 06, 2019 at 01:53:01PM +0000, George Dunlap wrote:
> > On 3/6/19 1:49 PM, George Dunlap wrote:
> > > On 3/5/19 4:42 PM, Wei Liu wrote:
> > >> This series makes tools build with Python 3.
> > >>
> > >> Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able
> > >> to give people some idea what sort of work is involved.
> > >>
> > >> You will also need Andrew's "tools/xen-foreign: Update python scripts to be
> > >> Py3 compatible".
> > > 
> > > Tossing this out there: Given that python2 is (theoretically) EOL in
> > > less than a year, is it worth maintaining compatibility with python2, or
> > > would it be better to just go whole-hog into python3?
> > 
> > I mean -- looking at some of the discussions about how differently
> > certain kinds of things are interpreted between python2 and python3, it
> > seems a bit mad to try to write code that works in both: we're just
> > asking for trouble.
> 
> I think Python is not very consistent even within a major version. New
> syntax and constructs get added to point releases. Some of the
> discussions re code isn't about the differences between 2 and 3.  There
> is at least one about 2.4 vs 2.7.

I think it's because they backport syntax changes from 3.X to 2.6 and
2.7. Also all the _future_X imports. All of that makes it easier to have
scripts compatible with both 2.7 and 3.X.

> I am all for moving away from 2.4.

Moving to a min of 2.6 would be nice. CentOS 6 comes it.

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 0/4] tools: Python 3 compatibility

[Wei Liu]

On Wed, Mar 06, 2019 at 02:17:31PM +0000, Anthony PERARD wrote:
> On Wed, Mar 06, 2019 at 01:57:55PM +0000, Wei Liu wrote:
> > On Wed, Mar 06, 2019 at 01:53:01PM +0000, George Dunlap wrote:
> > > On 3/6/19 1:49 PM, George Dunlap wrote:
> > > > On 3/5/19 4:42 PM, Wei Liu wrote:
> > > >> This series makes tools build with Python 3.
> > > >>
> > > >> Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able
> > > >> to give people some idea what sort of work is involved.
> > > >>
> > > >> You will also need Andrew's "tools/xen-foreign: Update python scripts to be
> > > >> Py3 compatible".
> > > > 
> > > > Tossing this out there: Given that python2 is (theoretically) EOL in
> > > > less than a year, is it worth maintaining compatibility with python2, or
> > > > would it be better to just go whole-hog into python3?
> > > 
> > > I mean -- looking at some of the discussions about how differently
> > > certain kinds of things are interpreted between python2 and python3, it
> > > seems a bit mad to try to write code that works in both: we're just
> > > asking for trouble.
> > 
> > I think Python is not very consistent even within a major version. New
> > syntax and constructs get added to point releases. Some of the
> > discussions re code isn't about the differences between 2 and 3.  There
> > is at least one about 2.4 vs 2.7.
> 
> I think it's because they backport syntax changes from 3.X to 2.6 and
> 2.7. Also all the _future_X imports. All of that makes it easier to have
> scripts compatible with both 2.7 and 3.X.
> 
> > I am all for moving away from 2.4.
> 
> Moving to a min of 2.6 would be nice. CentOS 6 comes it.

My next version will have a patch to do that. Any objection?

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 1/4] build/m4: make python_devel.m4 work with both python 2 and 3

[Anthony PERARD]

On Tue, Mar 05, 2019 at 04:42:03PM +0000, Wei Liu wrote:
> Do the following:
> 
> 1. Change the form of "print".
> 2. Check for ABI flags -- this is complicated because it is only
>    introduced in 3.2.

Is this a recommanded way of doing this? I may have a better way of
fixing this macro, see below.

> 3. Fix library name in AC_CHECK_LIB.
> 4. Remove other-libs in AC_CHECK_LIB.

Why did you remove the other libs? Also, with this change, PYTHON_LIBS
isn't used anywhere anymore, and can be removed.

> 
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
> I doubt the non python-pkg branch works, because the paths generated
> seem rather off. It definitely doesn't work on my machine, but I
> don't know how other systems could possibly be configured before the
> existence of python-config.
> ---
>  m4/python_devel.m4 | 27 ++++++++++++++++-----------
>  tools/configure    | 34 ++++++++++++++++++++--------------
>  2 files changed, 36 insertions(+), 25 deletions(-)
> 
> diff --git a/m4/python_devel.m4 b/m4/python_devel.m4
> index 05ea4ef7e2..1e2f41b6aa 100644
> --- a/m4/python_devel.m4
> +++ b/m4/python_devel.m4
> @@ -2,37 +2,42 @@ AC_DEFUN([AX_CHECK_PYTHON_DEVEL], [
>  ac_previous_cppflags=$CPPFLAGS
>  ac_previous_ldflags=$LDFLAGS
>  ac_python_version=`$PYTHON -c 'import distutils.sysconfig; \
> -    print distutils.sysconfig.get_config_var("VERSION")'`
> +    print(distutils.sysconfig.get_config_var("VERSION"))'`
> +ac_python_abiflags=
>  AC_PATH_PROG([pyconfig], [$PYTHON-config], [no])
>  AS_IF([test x"$pyconfig" = x"no"], [
>      dnl For those that don't have python-config
>      CPPFLAGS="$CFLAGS `$PYTHON -c 'import distutils.sysconfig; \
>          print "-I" + distutils.sysconfig.get_config_var("INCLUDEPY")'`"
>      CPPFLAGS="$CPPFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> -        print distutils.sysconfig.get_config_var("CFLAGS")'`"
> +        print(distutils.sysconfig.get_config_var("CFLAGS"))'`"
>      PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> -        print distutils.sysconfig.get_config_var("LIBS")'`"
> +        print(distutils.sysconfig.get_config_var("LIBS"))'`"
>      PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> -        print distutils.sysconfig.get_config_var("SYSLIBS")'`"
> +        print(distutils.sysconfig.get_config_var("SYSLIBS"))'`"
>      LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> -        print "-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
> -        standard_lib=1) + "/config"'`"
> +        print("-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
> +        standard_lib=1) + "/config")'`"
>      LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> -        print distutils.sysconfig.get_config_var("LINKFORSHARED")'`"
> +        print(distutils.sysconfig.get_config_var("LINKFORSHARED"))'`"
>      LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> -        print distutils.sysconfig.get_config_var("LDFLAGS")'`"
> +        print(distutils.sysconfig.get_config_var("LDFLAGS"))'`"
>  ], [
>      dnl If python-config is found use it
>      CPPFLAGS="$CFLAGS `$PYTHON-config --cflags`"
>      LDFLAGS="$LDFLAGS `$PYTHON-config --ldflags`"
>      PYTHON_LIBS="$LIBS `$PYTHON-config --libs`"
> +    abiflags="`$PYTHON-config --abiflags`"
> +    if test "$?" == "0"
> +    then
> +        ac_python_abiflags="$abiflags"
> +    fi
>  ])
>  
>  AC_CHECK_HEADER([Python.h], [],
>      [AC_MSG_ERROR([Unable to find Python development headers])],)
> -AC_CHECK_LIB(python$ac_python_version, PyArg_ParseTuple, [],
> -    [AC_MSG_ERROR([Unable to find a suitable python development library])],
> -    [$PYTHON_LIBS])
> +AC_CHECK_LIB(python$ac_python_version$ac_python_abiflags, PyArg_ParseTuple, [],
> +    [AC_MSG_ERROR([Unable to find a suitable python development library])])

So, AC_CHECK_LIB seems to only be used to check if PyArg_ParseTuple
exist, and requires as argument the name of the lib which is now
complicated.

But, AC_CHECK_LIB do test compilation using the LDFLAGS which already
contain the python lib we want, so instead, we could only do the part of
the jobs that we need:

AC_LINK_IFELSE([AC_LANG_CALL([], [PyArg_ParseTuple])], [],
    [AC_MSG_ERROR([Unable to find a suitable python development library])])

That generate a main.c with PyArg_ParseTuple() call like AC_CHECK_LIB
do, and do build/link. If that fails, throw an error.

That avoid to use the --abiflags, which we don't need.

What do you thing?

Some progress message can be added, similair to AC_CHECK_LIB:
AC_MSG_CHECKING([for PyArg_ParseTuple])
and [AC_MSG_RESULT([yes])] on success.

(I think AC_CHECK_LIB would also update $LIBS, but I don't think our
build system is using that.)

>  CPPFLAGS=$ac_previous_cppflags
>  LDFLAGS=$ac_previous_ldflags
>  ])

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 1/4] build/m4: make python_devel.m4 work with both python 2 and 3

[Wei Liu]

On Wed, Mar 06, 2019 at 03:07:16PM +0000, Anthony PERARD wrote:
> On Tue, Mar 05, 2019 at 04:42:03PM +0000, Wei Liu wrote:
> > Do the following:
> > 
> > 1. Change the form of "print".
> > 2. Check for ABI flags -- this is complicated because it is only
> >    introduced in 3.2.
> 
> Is this a recommanded way of doing this? I may have a better way of
> fixing this macro, see below.

Nope. I figured this out myself. There doesn't seem to be a recommended
way to do it.

> 
> > 3. Fix library name in AC_CHECK_LIB.
> > 4. Remove other-libs in AC_CHECK_LIB.
> 
> Why did you remove the other libs? Also, with this change, PYTHON_LIBS
> isn't used anywhere anymore, and can be removed.
> 

Because

"The other-libraries argument should be limited to cases where it is
desirable to test for one library in the presence of another that is not
already in LIBS."

Its original usage is wrong. At least in the python-config case,
PYTHON_LIBS is "-lpython2.7 -lpthread -ldl -lutil -lm". ./configure
already knows how to use LDFLAGS as far as I can tell from the log.

Yes. PYTHON_LIBS can be removed. I think.


> > 
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> > ---
> > I doubt the non python-pkg branch works, because the paths generated
> > seem rather off. It definitely doesn't work on my machine, but I
> > don't know how other systems could possibly be configured before the
> > existence of python-config.
> > ---
> >  m4/python_devel.m4 | 27 ++++++++++++++++-----------
> >  tools/configure    | 34 ++++++++++++++++++++--------------
> >  2 files changed, 36 insertions(+), 25 deletions(-)
> > 
> > diff --git a/m4/python_devel.m4 b/m4/python_devel.m4
> > index 05ea4ef7e2..1e2f41b6aa 100644
> > --- a/m4/python_devel.m4
> > +++ b/m4/python_devel.m4
> > @@ -2,37 +2,42 @@ AC_DEFUN([AX_CHECK_PYTHON_DEVEL], [
> >  ac_previous_cppflags=$CPPFLAGS
> >  ac_previous_ldflags=$LDFLAGS
> >  ac_python_version=`$PYTHON -c 'import distutils.sysconfig; \
> > -    print distutils.sysconfig.get_config_var("VERSION")'`
> > +    print(distutils.sysconfig.get_config_var("VERSION"))'`
> > +ac_python_abiflags=
> >  AC_PATH_PROG([pyconfig], [$PYTHON-config], [no])
> >  AS_IF([test x"$pyconfig" = x"no"], [
> >      dnl For those that don't have python-config
> >      CPPFLAGS="$CFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> >          print "-I" + distutils.sysconfig.get_config_var("INCLUDEPY")'`"
> >      CPPFLAGS="$CPPFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> > -        print distutils.sysconfig.get_config_var("CFLAGS")'`"
> > +        print(distutils.sysconfig.get_config_var("CFLAGS"))'`"
> >      PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> > -        print distutils.sysconfig.get_config_var("LIBS")'`"
> > +        print(distutils.sysconfig.get_config_var("LIBS"))'`"
> >      PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> > -        print distutils.sysconfig.get_config_var("SYSLIBS")'`"
> > +        print(distutils.sysconfig.get_config_var("SYSLIBS"))'`"
> >      LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> > -        print "-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
> > -        standard_lib=1) + "/config"'`"
> > +        print("-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
> > +        standard_lib=1) + "/config")'`"
> >      LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> > -        print distutils.sysconfig.get_config_var("LINKFORSHARED")'`"
> > +        print(distutils.sysconfig.get_config_var("LINKFORSHARED"))'`"
> >      LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
> > -        print distutils.sysconfig.get_config_var("LDFLAGS")'`"
> > +        print(distutils.sysconfig.get_config_var("LDFLAGS"))'`"
> >  ], [
> >      dnl If python-config is found use it
> >      CPPFLAGS="$CFLAGS `$PYTHON-config --cflags`"
> >      LDFLAGS="$LDFLAGS `$PYTHON-config --ldflags`"
> >      PYTHON_LIBS="$LIBS `$PYTHON-config --libs`"
> > +    abiflags="`$PYTHON-config --abiflags`"
> > +    if test "$?" == "0"
> > +    then
> > +        ac_python_abiflags="$abiflags"
> > +    fi
> >  ])
> >  
> >  AC_CHECK_HEADER([Python.h], [],
> >      [AC_MSG_ERROR([Unable to find Python development headers])],)
> > -AC_CHECK_LIB(python$ac_python_version, PyArg_ParseTuple, [],
> > -    [AC_MSG_ERROR([Unable to find a suitable python development library])],
> > -    [$PYTHON_LIBS])
> > +AC_CHECK_LIB(python$ac_python_version$ac_python_abiflags, PyArg_ParseTuple, [],
> > +    [AC_MSG_ERROR([Unable to find a suitable python development library])])
> 
> So, AC_CHECK_LIB seems to only be used to check if PyArg_ParseTuple
> exist, and requires as argument the name of the lib which is now
> complicated.
> 
> But, AC_CHECK_LIB do test compilation using the LDFLAGS which already
> contain the python lib we want, so instead, we could only do the part of
> the jobs that we need:
> 
> AC_LINK_IFELSE([AC_LANG_CALL([], [PyArg_ParseTuple])], [],
>     [AC_MSG_ERROR([Unable to find a suitable python development library])])
> 
> That generate a main.c with PyArg_ParseTuple() call like AC_CHECK_LIB
> do, and do build/link. If that fails, throw an error.
> 
> That avoid to use the --abiflags, which we don't need.
> 
> What do you thing?

Definitely looks better. Let me try this.

> 
> Some progress message can be added, similair to AC_CHECK_LIB:
> AC_MSG_CHECKING([for PyArg_ParseTuple])
> and [AC_MSG_RESULT([yes])] on success.
> 
> (I think AC_CHECK_LIB would also update $LIBS, but I don't think our
> build system is using that.)
> 

Yes it updates LIBS and yes xen build system doesn't need it.

Wei.

> >  CPPFLAGS=$ac_previous_cppflags
> >  LDFLAGS=$ac_previous_ldflags
> >  ])
> 
> -- 
> Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 1/4] build/m4: make python_devel.m4 work with both python 2 and 3

[Anthony PERARD]

On Wed, Mar 06, 2019 at 03:24:43PM +0000, Wei Liu wrote:
> On Wed, Mar 06, 2019 at 03:07:16PM +0000, Anthony PERARD wrote:
> > > -AC_CHECK_LIB(python$ac_python_version, PyArg_ParseTuple, [],
> > > -    [AC_MSG_ERROR([Unable to find a suitable python development library])],
> > > -    [$PYTHON_LIBS])
> > > +AC_CHECK_LIB(python$ac_python_version$ac_python_abiflags, PyArg_ParseTuple, [],
> > > +    [AC_MSG_ERROR([Unable to find a suitable python development library])])
> > 
> > So, AC_CHECK_LIB seems to only be used to check if PyArg_ParseTuple
> > exist, and requires as argument the name of the lib which is now
> > complicated.
> > 
> > But, AC_CHECK_LIB do test compilation using the LDFLAGS which already
> > contain the python lib we want, so instead, we could only do the part of
> > the jobs that we need:
> > 
> > AC_LINK_IFELSE([AC_LANG_CALL([], [PyArg_ParseTuple])], [],
> >     [AC_MSG_ERROR([Unable to find a suitable python development library])])
> > 
> > That generate a main.c with PyArg_ParseTuple() call like AC_CHECK_LIB
> > do, and do build/link. If that fails, throw an error.
> > 
> > That avoid to use the --abiflags, which we don't need.
> > 
> > What do you thing?
> 
> Definitely looks better. Let me try this.

Actually, I think I just found better, same things but using a macro
that already do what we want:

AC_CHECK_FUNC([PyArg_ParseTuple], [],
    [AC_MSG_ERROR([Unable to find a suitable python development library])])

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 2/4] libxl: make python scripts work with python 2 and 3

[Wei Liu]

On Tue, Mar 05, 2019 at 05:34:13PM +0000, Andrew Cooper wrote:
> On 05/03/2019 16:42, Wei Liu wrote:
> > All scripts are transformed by 2to3.
> >
> > The only addition is "from __future__ import print_function" so that
> > print("BLAH", file=sys.stderr) can work.
> >
> > https://python-future.org/compatible_idioms.html
> >
> > Tested with 2.7 and 3.5.
> >
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> > ---
> > I don't have environment to test 2.4 -- it is almost 15 years old. We
> > may want to consider bumping the minimum requirement to 2.7?
> 
> The compatible way to do this is  sys.stderr.write(msg + "\n") and using
> print() without the future import.
> 
> > @@ -269,7 +271,7 @@ class KeyedUnion(Aggregate):
> >          if not isinstance(keyvar_type, Enumeration):
> >              raise ValueError
> >  
> > -        kv_kwargs = dict([(x.lstrip('keyvar_'),y) for (x,y) in kwargs.items() if x.startswith('keyvar_')])
> > +        kv_kwargs = dict([(x.lstrip('keyvar_'),y) for (x,y) in list(kwargs.items()) if x.startswith('keyvar_')])
> 
> This shouldn't need changing.  List comprehensions are one of the few
> uses of .items() which is compatible with older versions of python IIRC.
> 
> > @@ -362,11 +364,10 @@ def parse(f):
> >              globs[n] = t
> >  
> >      try:
> > -        execfile(f, globs, locs)
> > -    except SyntaxError,e:
> > -        raise SyntaxError, \
> > -              "Errors were found at line %d while processing %s:\n\t%s"\
> > -              %(e.lineno,f,e.text)
> > +        exec(compile(open(f).read(), f, 'exec'), globs, locs)
> > +    except SyntaxError as e:
> 
> This is the only really awkward bit, and isn't Py 2.4 compatible.
> 
> The only option here to retain pre 2.6 compatibility is:
> 
> try:
>     ...
> except SyntaxError:
>     _, e = sys.exc_info()[:2]
>     ...

Since we will bump python requirement to 2.6, I think the transformation
made by 2to3 should be fine.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 1/4] build/m4: make python_devel.m4 work with both python 2 and 3

[Wei Liu]

On Wed, Mar 06, 2019 at 03:35:53PM +0000, Anthony PERARD wrote:
> On Wed, Mar 06, 2019 at 03:24:43PM +0000, Wei Liu wrote:
> > On Wed, Mar 06, 2019 at 03:07:16PM +0000, Anthony PERARD wrote:
> > > > -AC_CHECK_LIB(python$ac_python_version, PyArg_ParseTuple, [],
> > > > -    [AC_MSG_ERROR([Unable to find a suitable python development library])],
> > > > -    [$PYTHON_LIBS])
> > > > +AC_CHECK_LIB(python$ac_python_version$ac_python_abiflags, PyArg_ParseTuple, [],
> > > > +    [AC_MSG_ERROR([Unable to find a suitable python development library])])
> > > 
> > > So, AC_CHECK_LIB seems to only be used to check if PyArg_ParseTuple
> > > exist, and requires as argument the name of the lib which is now
> > > complicated.
> > > 
> > > But, AC_CHECK_LIB do test compilation using the LDFLAGS which already
> > > contain the python lib we want, so instead, we could only do the part of
> > > the jobs that we need:
> > > 
> > > AC_LINK_IFELSE([AC_LANG_CALL([], [PyArg_ParseTuple])], [],
> > >     [AC_MSG_ERROR([Unable to find a suitable python development library])])
> > > 
> > > That generate a main.c with PyArg_ParseTuple() call like AC_CHECK_LIB
> > > do, and do build/link. If that fails, throw an error.
> > > 
> > > That avoid to use the --abiflags, which we don't need.
> > > 
> > > What do you thing?
> > 
> > Definitely looks better. Let me try this.
> 
> Actually, I think I just found better, same things but using a macro
> that already do what we want:
> 
> AC_CHECK_FUNC([PyArg_ParseTuple], [],
>     [AC_MSG_ERROR([Unable to find a suitable python development library])])
> 

This works. Thank you.

Wei.

> -- 
> Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-next RFC 4/4] pygrub: make it build with python 3

[Wei Liu]

On Tue, Mar 05, 2019 at 04:42:06PM +0000, Wei Liu wrote:
> +
>  PyMODINIT_FUNC
>  initxenfsimage(void)

So Python 3 requires the initialisation function to be called
PyInit_xenfsimage, otherwise it can't find the entry point of this
module.

I have fixed this in my next version.

Wei.

>  {
> +#if PY_MAJOR_VERSION < 3
>  	Py_InitModule("xenfsimage", fsimage_module_methods);
> +#else
> +	return PyModule_Create(&fsimage_module_def);
> +#endif
>  }
> -- 
> 2.11.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel