* [PATCH 02/22] classes/lib: Convert to use python3 octal syntax
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 21:57 ` Christopher Larson
2016-06-01 12:35 ` [PATCH 03/22] classes/lib: Update to use python3 command pipeline decoding Richard Purdie
` (22 subsequent siblings)
23 siblings, 1 reply; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
The syntax for octal values changed in python3, adapt to it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/classes/base.bbclass | 10 +++++-----
meta/classes/package_deb.bbclass | 4 ++--
meta/classes/package_ipk.bbclass | 2 +-
meta/classes/package_rpm.bbclass | 6 +++---
meta/classes/sanity.bbclass | 4 ++--
meta/classes/sstate.bbclass | 6 +++---
meta/classes/terminal.bbclass | 2 +-
meta/lib/oe/package_manager.py | 4 ++--
meta/lib/oe/patch.py | 4 ++--
meta/lib/oe/sdk.py | 4 ++--
10 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 4be0a7e..1ba1222 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -456,15 +456,15 @@ python () {
# If we're building a target package we need to use fakeroot (pseudo)
# in order to capture permissions, owners, groups and special files
if not bb.data.inherits_class('native', d) and not bb.data.inherits_class('cross', d):
- d.setVarFlag('do_unpack', 'umask', '022')
- d.setVarFlag('do_configure', 'umask', '022')
- d.setVarFlag('do_compile', 'umask', '022')
+ d.setVarFlag('do_unpack', 'umask', '0o022')
+ d.setVarFlag('do_configure', 'umask', '0o022')
+ d.setVarFlag('do_compile', 'umask', '0o022')
d.appendVarFlag('do_install', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
d.setVarFlag('do_install', 'fakeroot', '1')
- d.setVarFlag('do_install', 'umask', '022')
+ d.setVarFlag('do_install', 'umask', '0o022')
d.appendVarFlag('do_package', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
d.setVarFlag('do_package', 'fakeroot', '1')
- d.setVarFlag('do_package', 'umask', '022')
+ d.setVarFlag('do_package', 'umask', '0o022')
d.setVarFlag('do_package_setscene', 'fakeroot', '1')
d.appendVarFlag('do_package_setscene', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
d.setVarFlag('do_devshell', 'fakeroot', '1')
diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index e1d05a7..bb5220e 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -117,7 +117,7 @@ python do_package_deb () {
controldir = os.path.join(root, 'DEBIAN')
bb.utils.mkdirhier(controldir)
- os.chmod(controldir, 0755)
+ os.chmod(controldir, 0o755)
try:
import codecs
ctrlfile = codecs.open(os.path.join(controldir, 'control'), 'w', 'utf-8')
@@ -293,7 +293,7 @@ python do_package_deb () {
scriptfile.write(scriptvar[pos:])
scriptfile.write('\n')
scriptfile.close()
- os.chmod(os.path.join(controldir, script), 0755)
+ os.chmod(os.path.join(controldir, script), 0o755)
conffiles_str = ' '.join(get_conffiles(pkg, d))
if conffiles_str:
diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index f1ad1d5..08f7020 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -225,7 +225,7 @@ python do_package_ipk () {
raise bb.build.FuncFailed("unable to open %s script file for writing." % script)
scriptfile.write(scriptvar)
scriptfile.close()
- os.chmod(os.path.join(controldir, script), 0755)
+ os.chmod(os.path.join(controldir, script), 0o755)
conffiles_str = ' '.join(get_conffiles(pkg, d))
if conffiles_str:
diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 7d523a1..f9398a9 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -63,7 +63,7 @@ def write_rpm_perfiledata(srcname, d):
dump_filerdeps('RDEPENDS', dependsfile, d)
dependsfile.close()
- os.chmod(outdepends, 0755)
+ os.chmod(outdepends, 0o755)
# OE-core / RPM Provides
outprovides = workdir + "/" + srcname + ".provides"
@@ -76,7 +76,7 @@ def write_rpm_perfiledata(srcname, d):
dump_filerdeps('RPROVIDES', providesfile, d)
providesfile.close()
- os.chmod(outprovides, 0755)
+ os.chmod(outprovides, 0o755)
return (outdepends, outprovides)
@@ -702,7 +702,7 @@ python do_package_rpm () {
pkgarch = d.expand('${PACKAGE_ARCH_EXTEND}${HOST_VENDOR}-${HOST_OS}')
magicfile = d.expand('${STAGING_DIR_NATIVE}${datadir_native}/misc/magic.mgc')
bb.utils.mkdirhier(pkgwritedir)
- os.chmod(pkgwritedir, 0755)
+ os.chmod(pkgwritedir, 0o755)
cmd = rpmbuild
cmd = cmd + " --nodeps --short-circuit --target " + pkgarch + " --buildroot " + pkgd
diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 3d5ace2..b0e7451 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -842,8 +842,8 @@ def check_sanity_everybuild(status, d):
check_supported_distro(d)
- omask = os.umask(022)
- if omask & 0755:
+ omask = os.umask(0o022)
+ if omask & 0o755:
status.addresult("Please use a umask which allows a+rx and u+rwx\n")
os.umask(omask)
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 6bf94b0..994eae6 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -655,9 +655,9 @@ python sstate_task_postfunc () {
sstate_install(shared_state, d)
for intercept in shared_state['interceptfuncs']:
bb.build.exec_func(intercept, d, (d.getVar("WORKDIR", True),))
- omask = os.umask(002)
- if omask != 002:
- bb.note("Using umask 002 (not %0o) for sstate packaging" % omask)
+ omask = os.umask(0o002)
+ if omask != 0o002:
+ bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask)
sstate_package(shared_state, d)
os.umask(omask)
}
diff --git a/meta/classes/terminal.bbclass b/meta/classes/terminal.bbclass
index 9f4c24e..a94f755 100644
--- a/meta/classes/terminal.bbclass
+++ b/meta/classes/terminal.bbclass
@@ -29,7 +29,7 @@ def emit_terminal_func(command, envdata, d):
bb.data.emit_func(cmd_func, script, envdata)
script.write(cmd_func)
script.write("\n")
- os.chmod(runfile, 0755)
+ os.chmod(runfile, 0o755)
return runfile
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 79b3206..3bc4ebf 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -1090,8 +1090,8 @@ class RpmPM(PackageManager):
native_root)
open(self.scriptlet_wrapper, 'w+').write(scriptlet_content)
- bb.note("Configuring RPM cross-install scriptlet_wrapper")
- os.chmod(self.scriptlet_wrapper, 0755)
+ bb.note("configuring RPM cross-install scriptlet_wrapper")
+ os.chmod(self.scriptlet_wrapper, 0o755)
cmd = 'config --set rpm-extra-macros._cross_scriptlet_wrapper=%s' % \
self.scriptlet_wrapper
self._invoke_smart(cmd)
diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 9d36172..a25fd52 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -434,7 +434,7 @@ class GitApplyTree(PatchTree):
# change other places which read it back
f.write('echo >> $1\n')
f.write('echo "%s: $PATCHFILE" >> $1\n' % GitApplyTree.patch_line_prefix)
- os.chmod(commithook, 0755)
+ os.chmod(commithook, 0o755)
shutil.copy2(commithook, applyhook)
try:
patchfilevar = 'PATCHFILE="%s"' % os.path.basename(patch['file'])
@@ -672,7 +672,7 @@ class UserResolver(Resolver):
f.write("echo 'Run \"quilt refresh\" when patch is corrected, press CTRL+D to exit.'\n")
f.write("echo ''\n")
f.write(" ".join(patchcmd) + "\n")
- os.chmod(rcfile, 0775)
+ os.chmod(rcfile, 0o775)
self.terminal("bash --rcfile " + rcfile, 'Patch Rejects: Please fix patch rejects manually', self.patchset.d)
diff --git a/meta/lib/oe/sdk.py b/meta/lib/oe/sdk.py
index f1bbef6..4786cc5 100644
--- a/meta/lib/oe/sdk.py
+++ b/meta/lib/oe/sdk.py
@@ -251,12 +251,12 @@ class OpkgSdk(Sdk):
self.mkdirhier(target_sysconfdir)
shutil.copy(self.target_conf, target_sysconfdir)
os.chmod(os.path.join(target_sysconfdir,
- os.path.basename(self.target_conf)), 0644)
+ os.path.basename(self.target_conf)), 0o644)
self.mkdirhier(host_sysconfdir)
shutil.copy(self.host_conf, host_sysconfdir)
os.chmod(os.path.join(host_sysconfdir,
- os.path.basename(self.host_conf)), 0644)
+ os.path.basename(self.host_conf)), 0o644)
native_opkg_state_dir = os.path.join(self.sdk_output, self.sdk_native_path,
self.d.getVar('localstatedir_nativesdk', True).strip('/'),
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH 02/22] classes/lib: Convert to use python3 octal syntax
2016-06-01 12:35 ` [PATCH 02/22] classes/lib: Convert to use python3 octal syntax Richard Purdie
@ 2016-06-01 21:57 ` Christopher Larson
2016-06-02 7:11 ` Richard Purdie
0 siblings, 1 reply; 31+ messages in thread
From: Christopher Larson @ 2016-06-01 21:57 UTC (permalink / raw)
To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 2207 bytes --]
On Wed, Jun 1, 2016 at 5:35 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 4be0a7e..1ba1222 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -456,15 +456,15 @@ python () {
> # If we're building a target package we need to use fakeroot (pseudo)
> # in order to capture permissions, owners, groups and special files
> if not bb.data.inherits_class('native', d) and not bb.data.inherits_class('cross',
> d):
> - d.setVarFlag('do_unpack', 'umask', '022')
> - d.setVarFlag('do_configure', 'umask', '022')
> - d.setVarFlag('do_compile', 'umask', '022')
> + d.setVarFlag('do_unpack', 'umask', '0o022')
> + d.setVarFlag('do_configure', 'umask', '0o022')
> + d.setVarFlag('do_compile', 'umask', '0o022')
> d.appendVarFlag('do_install', 'depends', '
> virtual/fakeroot-native:do_populate_sysroot')
> d.setVarFlag('do_install', 'fakeroot', '1')
> - d.setVarFlag('do_install', 'umask', '022')
> + d.setVarFlag('do_install', 'umask', '0o022')
> d.appendVarFlag('do_package', 'depends', '
> virtual/fakeroot-native:do_populate_sysroot')
> d.setVarFlag('do_package', 'fakeroot', '1')
> - d.setVarFlag('do_package', 'umask', '022')
> + d.setVarFlag('do_package', 'umask', '0o022')
> d.setVarFlag('do_package_setscene', 'fakeroot', '1')
> d.appendVarFlag('do_package_setscene', 'depends', '
> virtual/fakeroot-native:do_populate_sysroot')
> d.setVarFlag('do_devshell', 'fakeroot', '1')
>
Rather than modifying the string values in the metadata, shouldn't we
retain compatibility by letting the umask *string* stored in the flag use
the old format and if needed convert it when we convert the string to a
number to pass to the appropriate os functions? Or was this just a sed gone
wild, since these are string values, not python source octal syntax?
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
[-- Attachment #2: Type: text/html, Size: 3334 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH 02/22] classes/lib: Convert to use python3 octal syntax
2016-06-01 21:57 ` Christopher Larson
@ 2016-06-02 7:11 ` Richard Purdie
0 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-02 7:11 UTC (permalink / raw)
To: Christopher Larson; +Cc: Patches and discussions about the oe-core layer
On Wed, 2016-06-01 at 14:57 -0700, Christopher Larson wrote:
>
> On Wed, Jun 1, 2016 at 5:35 AM, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
> > diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> > index 4be0a7e..1ba1222 100644
> > --- a/meta/classes/base.bbclass
> > +++ b/meta/classes/base.bbclass
> > @@ -456,15 +456,15 @@ python () {
> > # If we're building a target package we need to use fakeroot
> > (pseudo)
> > # in order to capture permissions, owners, groups and special
> > files
> > if not bb.data.inherits_class('native', d) and not
> > bb.data.inherits_class('cross', d):
> > - d.setVarFlag('do_unpack', 'umask', '022')
> > - d.setVarFlag('do_configure', 'umask', '022')
> > - d.setVarFlag('do_compile', 'umask', '022')
> > + d.setVarFlag('do_unpack', 'umask', '0o022')
> > + d.setVarFlag('do_configure', 'umask', '0o022')
> > + d.setVarFlag('do_compile', 'umask', '0o022')
> > d.appendVarFlag('do_install', 'depends', '
> > virtual/fakeroot-native:do_populate_sysroot')
> > d.setVarFlag('do_install', 'fakeroot', '1')
> > - d.setVarFlag('do_install', 'umask', '022')
> > + d.setVarFlag('do_install', 'umask', '0o022')
> > d.appendVarFlag('do_package', 'depends', '
> > virtual/fakeroot-native:do_populate_sysroot')
> > d.setVarFlag('do_package', 'fakeroot', '1')
> > - d.setVarFlag('do_package', 'umask', '022')
> > + d.setVarFlag('do_package', 'umask', '0o022')
> > d.setVarFlag('do_package_setscene', 'fakeroot', '1')
> > d.appendVarFlag('do_package_setscene', 'depends', '
> > virtual/fakeroot-native:do_populate_sysroot')
> > d.setVarFlag('do_devshell', 'fakeroot', '1')
> >
> Rather than modifying the string values in the metadata, shouldn't we
> retain compatibility by letting the umask *string* stored in the flag
> use the old format and if needed convert it when we convert the
> string to a number to pass to the appropriate os functions? Or was
> this just a sed gone wild, since these are string values, not python
> source octal syntax?
I'm not entirely sure why I did this but its not needed, the string
octal values continue to work just fine. This change doesn't break
anything but also doesn't help.
I'll take this piece out...
Cheers,
Richard
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH 03/22] classes/lib: Update to use python3 command pipeline decoding
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
2016-06-01 12:35 ` [PATCH 02/22] classes/lib: Convert to use python3 octal syntax Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 04/22] classes/lib: Update to match python3 iter requirements Richard Purdie
` (21 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
In python3, strings are unicode by default. We need to encode/decode
from command pipelines and other places where we interface with the
real world using the correct locales. This patch updates various
call sites to use the correct encoding/decodings.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/classes/buildhistory.bbclass | 12 +--
meta/classes/chrpath.bbclass | 2 +
meta/classes/externalsrc.bbclass | 2 +-
meta/classes/insane.bbclass | 8 +-
meta/classes/libc-package.bbclass | 3 +
meta/classes/package.bbclass | 6 +-
meta/classes/package_deb.bbclass | 26 +++----
meta/classes/toaster.bbclass | 1 +
meta/lib/oe/gpg_sign.py | 10 +--
meta/lib/oe/package.py | 4 +-
meta/lib/oe/package_manager.py | 88 +++++++++++-----------
meta/lib/oe/packagedata.py | 2 +-
meta/lib/oeqa/oetest.py | 4 +-
meta/lib/oeqa/runtime/parselogs.py | 4 +-
meta/lib/oeqa/runtime/ping.py | 2 +-
meta/lib/oeqa/utils/commands.py | 2 +-
meta/lib/oeqa/utils/qemurunner.py | 26 ++++---
meta/lib/oeqa/utils/qemutinyrunner.py | 4 +-
meta/lib/oeqa/utils/sshcontrol.py | 1 +
.../recipes-graphics/xorg-xserver/xserver-xorg.inc | 5 +-
20 files changed, 110 insertions(+), 102 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 581d532..e3b5c44 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -233,7 +233,7 @@ python buildhistory_emit_pkghistory() {
key = item[0]
if key.endswith('_' + pkg):
key = key[:-len(pkg)-1]
- pkgdata[key] = item[1].decode('utf-8').decode('string_escape')
+ pkgdata[key] = item[1]
pkge = pkgdata.get('PKGE', '0')
pkgv = pkgdata['PKGV']
@@ -288,14 +288,12 @@ python buildhistory_emit_pkghistory() {
def write_recipehistory(rcpinfo, d):
- import codecs
-
bb.debug(2, "Writing recipe history")
pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
infofile = os.path.join(pkghistdir, "latest")
- with codecs.open(infofile, "w", encoding='utf8') as f:
+ with open(infofile, "w") as f:
if rcpinfo.pe != "0":
f.write(u"PE = %s\n" % rcpinfo.pe)
f.write(u"PV = %s\n" % rcpinfo.pv)
@@ -305,8 +303,6 @@ def write_recipehistory(rcpinfo, d):
def write_pkghistory(pkginfo, d):
- import codecs
-
bb.debug(2, "Writing package history for package %s" % pkginfo.name)
pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
@@ -316,7 +312,7 @@ def write_pkghistory(pkginfo, d):
bb.utils.mkdirhier(pkgpath)
infofile = os.path.join(pkgpath, "latest")
- with codecs.open(infofile, "w", encoding='utf8') as f:
+ with open(infofile, "w") as f:
if pkginfo.pe != "0":
f.write(u"PE = %s\n" % pkginfo.pe)
f.write(u"PV = %s\n" % pkginfo.pv)
@@ -349,7 +345,7 @@ def write_pkghistory(pkginfo, d):
filevarpath = os.path.join(pkgpath, "latest.%s" % filevar)
val = pkginfo.filevars[filevar]
if val:
- with codecs.open(filevarpath, "w", encoding='utf8') as f:
+ with open(filevarpath, "w") as f:
f.write(val)
else:
if os.path.exists(filevarpath):
diff --git a/meta/classes/chrpath.bbclass b/meta/classes/chrpath.bbclass
index 9c68855..cdd7f27 100644
--- a/meta/classes/chrpath.bbclass
+++ b/meta/classes/chrpath.bbclass
@@ -10,6 +10,8 @@ def process_file_linux(cmd, fpath, rootdir, baseprefix, tmpdir, d):
if p.returncode != 0:
return
+ err = err.decode('utf-8')
+
# Handle RUNPATH as well as RPATH
err = err.replace("RUNPATH=","RPATH=")
# Throw away everything other than the rpath list
diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index da7eb478..b7140a3 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -145,7 +145,7 @@ def srctree_hash_files(d):
env = os.environ.copy()
env['GIT_INDEX_FILE'] = tmp_index.name
subprocess.check_output(['git', 'add', '.'], cwd=s_dir, env=env)
- sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env)
+ sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8")
with open(oe_hash_file, 'w') as fobj:
fobj.write(sha1)
ret = oe_hash_file + ':True'
diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 71999ad..9b2337c 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -399,7 +399,7 @@ def package_qa_check_unsafe_references_in_binaries(path, name, d, elf, messages)
sysroot_path_usr = sysroot_path + exec_prefix
try:
- ldd_output = bb.process.Popen(["prelink-rtld", "--root", sysroot_path, path], stdout=sub.PIPE).stdout.read()
+ ldd_output = bb.process.Popen(["prelink-rtld", "--root", sysroot_path, path], stdout=sub.PIPE).stdout.read().decode("utf-8")
except bb.process.CmdError:
error_msg = pn + ": prelink-rtld aborted when processing %s" % path
package_qa_handle_error("unsafe-references-in-binaries", error_msg, d)
@@ -986,12 +986,12 @@ def package_qa_check_expanded_d(path,name,d,elf,messages):
return sane
def package_qa_check_encoding(keys, encode, d):
- def check_encoding(key,enc):
+ def check_encoding(key, enc):
sane = True
value = d.getVar(key, True)
if value:
try:
- s = unicode(value, enc)
+ s = value.encode(enc)
except UnicodeDecodeError as e:
error_msg = "%s has non %s characters" % (key,enc)
sane = False
@@ -1217,7 +1217,7 @@ Missing inherit gettext?""" % (gt, config))
try:
flag = "WARNING: unrecognized options:"
log = os.path.join(d.getVar('B', True), 'config.log')
- output = subprocess.check_output(['grep', '-F', flag, log]).replace(', ', ' ')
+ output = subprocess.check_output(['grep', '-F', flag, log]).decode("utf-8").replace(', ', ' ')
options = set()
for line in output.splitlines():
options |= set(line.partition(flag)[2].split())
diff --git a/meta/classes/libc-package.bbclass b/meta/classes/libc-package.bbclass
index 467d567..70f479b 100644
--- a/meta/classes/libc-package.bbclass
+++ b/meta/classes/libc-package.bbclass
@@ -150,6 +150,7 @@ python package_do_split_gconvs () {
c_re = re.compile('^copy "(.*)"')
i_re = re.compile('^include "(\w+)".*')
for l in f.readlines():
+ l = l.decode("latin-1")
m = c_re.match(l) or i_re.match(l)
if m:
dp = legitimize_package_name('%s%s-gconv-%s' % (mlprefix, bpn, m.group(1)))
@@ -171,6 +172,7 @@ python package_do_split_gconvs () {
c_re = re.compile('^copy "(.*)"')
i_re = re.compile('^include "(\w+)".*')
for l in f.readlines():
+ l = l.decode("latin-1")
m = c_re.match(l) or i_re.match(l)
if m:
dp = legitimize_package_name('%s%s-charmap-%s' % (mlprefix, bpn, m.group(1)))
@@ -191,6 +193,7 @@ python package_do_split_gconvs () {
c_re = re.compile('^copy "(.*)"')
i_re = re.compile('^include "(\w+)".*')
for l in f.readlines():
+ l = l.decode("latin-1")
m = c_re.match(l) or i_re.match(l)
if m:
dp = legitimize_package_name(mlprefix+bpn+'-localedata-%s' % m.group(1))
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index a4125a0..501004e 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -63,7 +63,7 @@ def legitimize_package_name(s):
def fixutf(m):
cp = m.group(1)
if cp:
- return ('\u%s' % cp).decode('unicode_escape').encode('utf-8')
+ return ('\\u%s' % cp).encode('latin-1').decode('unicode_escape')
# Handle unicode codepoints encoded as <U0123>, as in glibc locale files.
s = re.sub('<U([0-9A-Fa-f]{1,4})>', fixutf, s)
@@ -1259,8 +1259,8 @@ python emit_pkgdata() {
def write_if_exists(f, pkg, var):
def encode(str):
import codecs
- c = codecs.getencoder("string_escape")
- return c(str)[0]
+ c = codecs.getencoder("unicode_escape")
+ return c(str)[0].decode("latin1")
val = d.getVar('%s_%s' % (var, pkg), True)
if val:
diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index bb5220e..e35f427 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -173,7 +173,7 @@ python do_package_deb () {
# Special behavior for description...
if 'DESCRIPTION' in fs:
summary = localdata.getVar('SUMMARY', True) or localdata.getVar('DESCRIPTION', True) or "."
- ctrlfile.write('Description: %s\n' % unicode(summary,'utf-8'))
+ ctrlfile.write('Description: %s\n' % summary)
description = localdata.getVar('DESCRIPTION', True) or "."
description = textwrap.dedent(description).strip()
if '\\n' in description:
@@ -182,29 +182,25 @@ python do_package_deb () {
# We don't limit the width when manually indent, but we do
# need the textwrap.fill() to set the initial_indent and
# subsequent_indent, so set a large width
- ctrlfile.write('%s\n' % unicode(textwrap.fill(t, width=100000, initial_indent=' ', subsequent_indent=' '),'utf-8'))
+ ctrlfile.write('%s\n' % textwrap.fill(t, width=100000, initial_indent=' ', subsequent_indent=' '))
else:
# Auto indent
- ctrlfile.write('%s\n' % unicode(textwrap.fill(description.strip(), width=74, initial_indent=' ', subsequent_indent=' '),'utf-8'))
+ ctrlfile.write('%s\n' % textwrap.fill(description.strip(), width=74, initial_indent=' ', subsequent_indent=' '))
else:
- ctrlfile.write(unicode(c % tuple(pullData(fs, localdata)),'utf-8'))
+ ctrlfile.write(c % tuple(pullData(fs, localdata)))
except KeyError:
import sys
(type, value, traceback) = sys.exc_info()
bb.utils.unlockfile(lf)
ctrlfile.close()
raise bb.build.FuncFailed("Missing field for deb generation: %s" % value)
- except UnicodeDecodeError:
- bb.utils.unlockfile(lf)
- ctrlfile.close()
- raise bb.build.FuncFailed("Non UTF-8 characters found in one of the fields")
# more fields
custom_fields_chunk = get_package_additional_metadata("deb", localdata)
if custom_fields_chunk is not None:
- ctrlfile.write(unicode(custom_fields_chunk))
+ ctrlfile.write(custom_fields_chunk)
ctrlfile.write("\n")
mapping_rename_hook(localdata)
@@ -255,17 +251,17 @@ python do_package_deb () {
rconflicts = bb.utils.explode_dep_versions2(localdata.getVar("RCONFLICTS", True) or "")
debian_cmp_remap(rconflicts)
if rdepends:
- ctrlfile.write("Depends: %s\n" % unicode(bb.utils.join_deps(rdepends)))
+ ctrlfile.write("Depends: %s\n" % bb.utils.join_deps(rdepends))
if rsuggests:
- ctrlfile.write("Suggests: %s\n" % unicode(bb.utils.join_deps(rsuggests)))
+ ctrlfile.write("Suggests: %s\n" % bb.utils.join_deps(rsuggests))
if rrecommends:
- ctrlfile.write("Recommends: %s\n" % unicode(bb.utils.join_deps(rrecommends)))
+ ctrlfile.write("Recommends: %s\n" % bb.utils.join_deps(rrecommends))
if rprovides:
- ctrlfile.write("Provides: %s\n" % unicode(bb.utils.join_deps(rprovides)))
+ ctrlfile.write("Provides: %s\n" % bb.utils.join_deps(rprovides))
if rreplaces:
- ctrlfile.write("Replaces: %s\n" % unicode(bb.utils.join_deps(rreplaces)))
+ ctrlfile.write("Replaces: %s\n" % bb.utils.join_deps(rreplaces))
if rconflicts:
- ctrlfile.write("Conflicts: %s\n" % unicode(bb.utils.join_deps(rconflicts)))
+ ctrlfile.write("Conflicts: %s\n" % bb.utils.join_deps(rconflicts))
ctrlfile.close()
for script in ["preinst", "postinst", "prerm", "postrm"]:
diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass
index 1a70f14..1878fe0 100644
--- a/meta/classes/toaster.bbclass
+++ b/meta/classes/toaster.bbclass
@@ -33,6 +33,7 @@ python toaster_layerinfo_dumpdata() {
def _get_git_branch(layer_path):
branch = subprocess.Popen("git symbolic-ref HEAD 2>/dev/null ", cwd=layer_path, shell=True, stdout=subprocess.PIPE).communicate()[0]
+ branch = branch.decode('utf-8')
branch = branch.replace('refs/heads/', '').rstrip()
return branch
diff --git a/meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_sign.py
index b83ee86..a8a478a 100644
--- a/meta/lib/oe/gpg_sign.py
+++ b/meta/lib/oe/gpg_sign.py
@@ -24,7 +24,7 @@ class LocalSigner(object):
status, output = oe.utils.getstatusoutput(cmd)
if status:
raise bb.build.FuncFailed('Failed to export gpg public key (%s): %s' %
- (keyid, output))
+ (keyid, output.decode("utf-8")))
def sign_rpms(self, files, keyid, passphrase):
"""Sign RPM files"""
@@ -39,7 +39,7 @@ class LocalSigner(object):
status, output = oe.utils.getstatusoutput(cmd)
if status:
- raise bb.build.FuncFailed("Failed to sign RPM packages: %s" % output)
+ raise bb.build.FuncFailed("Failed to sign RPM packages: %s" % output.decode("utf-8"))
def detach_sign(self, input_file, keyid, passphrase_file, passphrase=None, armor=True):
"""Create a detached signature of a file"""
@@ -71,11 +71,11 @@ class LocalSigner(object):
passphrase = fobj.readline();
job = subprocess.Popen(cmd, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
- (_, stderr) = job.communicate(passphrase)
+ (_, stderr) = job.communicate(passphrase.encode("utf-8"))
if job.returncode:
raise bb.build.FuncFailed("GPG exited with code %d: %s" %
- (job.returncode, stderr))
+ (job.returncode, stderr.decode("utf-8")))
except IOError as e:
bb.error("IO error (%s): %s" % (e.errno, e.strerror))
@@ -90,7 +90,7 @@ class LocalSigner(object):
"""Return the gpg version"""
import subprocess
try:
- return subprocess.check_output((self.gpg_bin, "--version")).split()[2]
+ return subprocess.check_output((self.gpg_bin, "--version")).split()[2].decode("utf-8")
except subprocess.CalledProcessError as e:
raise bb.build.FuncFailed("Could not get gpg version: %s" % e)
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 252e32d..5bb15bb 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -64,8 +64,8 @@ def filedeprunner(arg):
def process_deps(pipe, pkg, pkgdest, provides, requires):
for line in pipe:
- f = line.split(" ", 1)[0].strip()
- line = line.split(" ", 1)[1].strip()
+ f = line.decode("utf-8").split(" ", 1)[0].strip()
+ line = line.decode("utf-8").split(" ", 1)[1].strip()
if line.startswith("Requires:"):
i = requires
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 3bc4ebf..abe9f68 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -17,10 +17,10 @@ def create_index(arg):
try:
bb.note("Executing '%s' ..." % index_cmd)
- result = subprocess.check_output(index_cmd, stderr=subprocess.STDOUT, shell=True)
+ result = subprocess.check_output(index_cmd, stderr=subprocess.STDOUT, shell=True).decode("utf-8")
except subprocess.CalledProcessError as e:
return("Index creation command '%s' failed with return code %d:\n%s" %
- (e.cmd, e.returncode, e.output))
+ (e.cmd, e.returncode, e.output.decode("utf-8")))
if result:
bb.note(result)
@@ -367,10 +367,10 @@ class RpmPkgsList(PkgsList):
# Determine rpm version
cmd = "%s --version" % self.rpm_cmd
try:
- output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True)
+ output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).decode("utf-8")
except subprocess.CalledProcessError as e:
bb.fatal("Getting rpm version failed. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
'''
Translate the RPM/Smart format names to the OE multilib format names
@@ -411,10 +411,10 @@ class RpmPkgsList(PkgsList):
"-t", self.image_rpmlib]
try:
- output = subprocess.check_output(cmd, stderr=subprocess.STDOUT).strip()
+ output = subprocess.check_output(cmd, stderr=subprocess.STDOUT).strip().decode("utf-8")
except subprocess.CalledProcessError as e:
bb.fatal("Cannot get the package dependencies. Command '%s' "
- "returned %d:\n%s" % (' '.join(cmd), e.returncode, e.output))
+ "returned %d:\n%s" % (' '.join(cmd), e.returncode, e.output.decode("utf-8")))
return output
@@ -425,10 +425,10 @@ class RpmPkgsList(PkgsList):
try:
# bb.note(cmd)
- tmp_output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).strip()
+ tmp_output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).strip().decode("utf-8")
except subprocess.CalledProcessError as e:
bb.fatal("Cannot get the installed packages list. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
output = dict()
deps = dict()
@@ -485,6 +485,8 @@ class OpkgPkgsList(PkgsList):
# output streams separately and check for empty stderr.
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
cmd_output, cmd_stderr = p.communicate()
+ cmd_output = cmd_output.decode("utf-8")
+ cmd_stderr = cmd_stderr.decode("utf-8")
if p.returncode or cmd_stderr:
bb.fatal("Cannot get the installed packages list. Command '%s' "
"returned %d and stderr:\n%s" % (cmd, p.returncode, cmd_stderr))
@@ -502,10 +504,10 @@ class DpkgPkgsList(PkgsList):
cmd.append("-f=Package: ${Package}\nArchitecture: ${PackageArch}\nVersion: ${Version}\nFile: ${Package}_${Version}_${Architecture}.deb\nDepends: ${Depends}\nRecommends: ${Recommends}\n\n")
try:
- cmd_output = subprocess.check_output(cmd, stderr=subprocess.STDOUT).strip()
+ cmd_output = subprocess.check_output(cmd, stderr=subprocess.STDOUT).strip().decode("utf-8")
except subprocess.CalledProcessError as e:
bb.fatal("Cannot get the installed packages list. Command '%s' "
- "returned %d:\n%s" % (' '.join(cmd), e.returncode, e.output))
+ "returned %d:\n%s" % (' '.join(cmd), e.returncode, e.output.decode("utf-8")))
return opkg_query(cmd_output)
@@ -608,11 +610,11 @@ class PackageManager(object):
try:
bb.note("Installing complementary packages ...")
bb.note('Running %s' % cmd)
- complementary_pkgs = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+ complementary_pkgs = subprocess.check_output(cmd, stderr=subprocess.STDOUT).decode("utf-8")
except subprocess.CalledProcessError as e:
bb.fatal("Could not compute complementary packages list. Command "
"'%s' returned %d:\n%s" %
- (' '.join(cmd), e.returncode, e.output))
+ (' '.join(cmd), e.returncode, e.output.decode("utf-8")))
self.install(complementary_pkgs.split(), attempt_only=True)
os.remove(installed_pkgs_file)
@@ -784,12 +786,12 @@ class RpmPM(PackageManager):
try:
complementary_pkgs = subprocess.check_output(cmd,
stderr=subprocess.STDOUT,
- shell=True)
+ shell=True).decode("utf-8")
# bb.note(complementary_pkgs)
return complementary_pkgs
except subprocess.CalledProcessError as e:
bb.fatal("Could not invoke smart. Command "
- "'%s' returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "'%s' returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
def _search_pkg_name_in_feeds(self, pkg, feed_archs):
for arch in feed_archs:
@@ -808,7 +810,7 @@ class RpmPM(PackageManager):
(self.smart_cmd, self.smart_opt, pkg)
cmd += " | sed -ne 's/ *Provides://p'"
bb.note('cmd: %s' % cmd)
- output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True)
+ output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).decode("utf-8")
# Found a provider
if output:
bb.note('Found providers for %s: %s' % (pkg, output))
@@ -956,7 +958,7 @@ class RpmPM(PackageManager):
subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True)
except subprocess.CalledProcessError as e:
bb.fatal("Create rpm database failed. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
# Import GPG key to RPM database of the target system
if self.d.getVar('RPM_SIGN_PACKAGES', True) == '1':
pubkey_path = self.d.getVar('RPM_GPG_PUBKEY', True)
@@ -1203,11 +1205,11 @@ class RpmPM(PackageManager):
cmd = "%s %s install --attempt -y %s" % \
(self.smart_cmd, self.smart_opt, ' '.join(pkgs))
try:
- output = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
+ output = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT).decode("utf-8")
bb.note(output)
except subprocess.CalledProcessError as e:
bb.fatal("Unable to install packages. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
'''
Remove pkgs with smart, the pkg name is smart/rpm format
@@ -1233,11 +1235,11 @@ class RpmPM(PackageManager):
try:
bb.note(cmd)
- output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True)
+ output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).decode("utf-8")
bb.note(output)
except subprocess.CalledProcessError as e:
bb.note("Unable to remove packages. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
def upgrade(self):
bb.note('smart upgrade')
@@ -1310,7 +1312,7 @@ class RpmPM(PackageManager):
install_pkgs.append(pkg)
except subprocess.CalledProcessError as e:
bb.note("Unable to dump install packages. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
# Recovery rpmsys channel
self._invoke_smart('channel --enable rpmsys')
return install_pkgs
@@ -1352,7 +1354,7 @@ class RpmPM(PackageManager):
available_pkgs.append(pkg.strip())
except subprocess.CalledProcessError as e:
bb.note("Unable to list all available packages. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
self.fullpkglist = available_pkgs
@@ -1379,12 +1381,12 @@ class RpmPM(PackageManager):
try:
bb.note(cmd)
- output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).strip()
+ output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).strip().decode("utf-8")
bb.note(output)
- os.chmod(saved_dir, 0755)
+ os.chmod(saved_dir, 0o755)
except subprocess.CalledProcessError as e:
bb.fatal("Invoke save_rpmpostinst failed. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
'''Write common configuration for target usage'''
def rpm_setup_smart_target_config(self):
@@ -1417,7 +1419,7 @@ class RpmPM(PackageManager):
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True)
except subprocess.CalledProcessError as e:
bb.fatal("Unable to list available packages. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
# Set default values to avoid UnboundLocalError
arch = ""
@@ -1482,7 +1484,7 @@ class RpmPM(PackageManager):
except subprocess.CalledProcessError as e:
bb.utils.remove(tmp_dir, recurse=True)
bb.fatal("Unable to extract %s package. Command '%s' "
- "returned %d:\n%s" % (pkg_path, cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (pkg_path, cmd, e.returncode, e.output.decode("utf-8")))
except OSError as e:
bb.utils.remove(tmp_dir, recurse=True)
bb.fatal("Unable to extract %s package. Command '%s' "
@@ -1512,7 +1514,7 @@ class OpkgDpkgPM(PackageManager):
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True)
except subprocess.CalledProcessError as e:
bb.fatal("Unable to list available packages. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
return opkg_query(output)
"""
@@ -1544,7 +1546,7 @@ class OpkgDpkgPM(PackageManager):
except subprocess.CalledProcessError as e:
bb.utils.remove(tmp_dir, recurse=True)
bb.fatal("Unable to extract %s package. Command '%s' "
- "returned %d:\n%s" % (pkg_path, cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (pkg_path, cmd, e.returncode, e.output.decode("utf-8")))
except OSError as e:
bb.utils.remove(tmp_dir, recurse=True)
bb.fatal("Unable to extract %s package. Command '%s' "
@@ -1733,7 +1735,7 @@ class OpkgPM(OpkgDpkgPM):
except subprocess.CalledProcessError as e:
self.deploy_dir_unlock()
bb.fatal("Unable to update the package index files. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
self.deploy_dir_unlock()
@@ -1754,12 +1756,12 @@ class OpkgPM(OpkgDpkgPM):
try:
bb.note("Installing the following packages: %s" % ' '.join(pkgs))
bb.note(cmd)
- output = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
+ output = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT).decode("utf-8")
bb.note(output)
except subprocess.CalledProcessError as e:
(bb.fatal, bb.note)[attempt_only]("Unable to install packages. "
"Command '%s' returned %d:\n%s" %
- (cmd, e.returncode, e.output))
+ (cmd, e.returncode, e.output.decode("utf-8")))
def remove(self, pkgs, with_dependencies=True):
if with_dependencies:
@@ -1771,11 +1773,11 @@ class OpkgPM(OpkgDpkgPM):
try:
bb.note(cmd)
- output = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
+ output = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT).decode("utf-8")
bb.note(output)
except subprocess.CalledProcessError as e:
bb.fatal("Unable to remove packages. Command '%s' "
- "returned %d:\n%s" % (e.cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (e.cmd, e.returncode, e.output.decode("utf-8")))
def write_index(self):
self.deploy_dir_lock()
@@ -1818,10 +1820,10 @@ class OpkgPM(OpkgDpkgPM):
pkg_info = cmd + pkg
try:
- output = subprocess.check_output(pkg_info.split(), stderr=subprocess.STDOUT).strip()
+ output = subprocess.check_output(pkg_info.split(), stderr=subprocess.STDOUT).strip().decode("utf-8")
except subprocess.CalledProcessError as e:
bb.fatal("Cannot get package info. Command '%s' "
- "returned %d:\n%s" % (pkg_info, e.returncode, e.output))
+ "returned %d:\n%s" % (pkg_info, e.returncode, e.output.decode("utf-8")))
if output == "":
bb.note("Ignored bad recommendation: '%s' is "
@@ -1858,7 +1860,7 @@ class OpkgPM(OpkgDpkgPM):
subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True)
except subprocess.CalledProcessError as e:
bb.fatal("Unable to update. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
# Dummy installation
cmd = "%s %s --noaction install %s " % (self.opkg_cmd,
@@ -1868,7 +1870,7 @@ class OpkgPM(OpkgDpkgPM):
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True)
except subprocess.CalledProcessError as e:
bb.fatal("Unable to dummy install packages. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
bb.utils.remove(temp_rootfs, True)
@@ -2012,7 +2014,7 @@ class DpkgPM(OpkgDpkgPM):
subprocess.check_output(p_full, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
bb.note("%s for package %s failed with %d:\n%s" %
- (suffix[1], pkg_name, e.returncode, e.output))
+ (suffix[1], pkg_name, e.returncode, e.output.decode("utf-8")))
failed_pkgs.append(pkg_name)
break
@@ -2030,7 +2032,7 @@ class DpkgPM(OpkgDpkgPM):
subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
bb.fatal("Unable to update the package index files. Command '%s' "
- "returned %d:\n%s" % (e.cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (e.cmd, e.returncode, e.output.decode("utf-8")))
self.deploy_dir_unlock()
@@ -2049,7 +2051,7 @@ class DpkgPM(OpkgDpkgPM):
except subprocess.CalledProcessError as e:
(bb.fatal, bb.note)[attempt_only]("Unable to install packages. "
"Command '%s' returned %d:\n%s" %
- (cmd, e.returncode, e.output))
+ (cmd, e.returncode, e.output.decode("utf-8")))
# rename *.dpkg-new files/dirs
for root, dirs, files in os.walk(self.target_rootfs):
@@ -2080,7 +2082,7 @@ class DpkgPM(OpkgDpkgPM):
subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
bb.fatal("Unable to remove packages. Command '%s' "
- "returned %d:\n%s" % (e.cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (e.cmd, e.returncode, e.output.decode("utf-8")))
def write_index(self):
self.deploy_dir_lock()
@@ -2213,7 +2215,7 @@ class DpkgPM(OpkgDpkgPM):
subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
bb.fatal("Cannot fix broken dependencies. Command '%s' "
- "returned %d:\n%s" % (cmd, e.returncode, e.output))
+ "returned %d:\n%s" % (cmd, e.returncode, e.output.decode("utf-8")))
def list_installed(self):
return DpkgPkgsList(self.d, self.target_rootfs).list_pkgs()
diff --git a/meta/lib/oe/packagedata.py b/meta/lib/oe/packagedata.py
index bc0fd06..df1b4c5 100644
--- a/meta/lib/oe/packagedata.py
+++ b/meta/lib/oe/packagedata.py
@@ -8,7 +8,7 @@ def read_pkgdatafile(fn):
pkgdata = {}
def decode(str):
- c = codecs.getdecoder("string_escape")
+ c = codecs.getdecoder("unicode_escape")
return c(str)[0]
if os.access(fn, os.R_OK):
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index a50f9d8..4211ffc 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -132,7 +132,7 @@ class oeSDKTest(oeTest):
return False
def _run(self, cmd):
- return subprocess.check_output(". %s > /dev/null; %s;" % (self.tc.sdkenv, cmd), shell=True)
+ return subprocess.check_output(". %s > /dev/null; %s;" % (self.tc.sdkenv, cmd), shell=True).decode("utf-8")
class oeSDKExtTest(oeSDKTest):
def _run(self, cmd):
@@ -144,7 +144,7 @@ class oeSDKExtTest(oeSDKTest):
env['PATH'] = avoid_paths_in_environ(paths_to_avoid)
return subprocess.check_output(". %s > /dev/null;"\
- " %s;" % (self.tc.sdkenv, cmd), shell=True, env=env)
+ " %s;" % (self.tc.sdkenv, cmd), shell=True, env=env).decode("utf-8")
def getmodule(pos=2):
# stack returns a list of tuples containg frame information
diff --git a/meta/lib/oeqa/runtime/parselogs.py b/meta/lib/oeqa/runtime/parselogs.py
index a936601..242cd8c 100644
--- a/meta/lib/oeqa/runtime/parselogs.py
+++ b/meta/lib/oeqa/runtime/parselogs.py
@@ -238,7 +238,7 @@ class ParseLogsTest(oeRuntimeTest):
result = None
thegrep = self.build_grepcmd(errors, ignore_errors, log)
try:
- result = subprocess.check_output(thegrep, shell=True)
+ result = subprocess.check_output(thegrep, shell=True).decode("utf-8")
except:
pass
if (result is not None):
@@ -246,7 +246,7 @@ class ParseLogsTest(oeRuntimeTest):
rez = result.splitlines()
for xrez in rez:
try:
- grep_output = subprocess.check_output(['grep', '-F', xrez, '-B', str(lines_before), '-A', str(lines_after), log])
+ grep_output = subprocess.check_output(['grep', '-F', xrez, '-B', str(lines_before), '-A', str(lines_after), log]).decode("utf-8")
except:
pass
results[log.replace('target_logs/','')][xrez]=grep_output
diff --git a/meta/lib/oeqa/runtime/ping.py b/meta/lib/oeqa/runtime/ping.py
index 80c4601..0f27447 100644
--- a/meta/lib/oeqa/runtime/ping.py
+++ b/meta/lib/oeqa/runtime/ping.py
@@ -14,7 +14,7 @@ class PingTest(oeRuntimeTest):
endtime = time.time() + 60
while count < 5 and time.time() < endtime:
proc = subprocess.Popen("ping -c 1 %s" % self.target.ip, shell=True, stdout=subprocess.PIPE)
- output += proc.communicate()[0]
+ output += proc.communicate()[0].decode("utf-8")
if proc.poll() == 0:
count += 1
else:
diff --git a/meta/lib/oeqa/utils/commands.py b/meta/lib/oeqa/utils/commands.py
index 48f6441..9a7c1d1 100644
--- a/meta/lib/oeqa/utils/commands.py
+++ b/meta/lib/oeqa/utils/commands.py
@@ -78,7 +78,7 @@ class Command(object):
self.process.kill()
self.thread.join()
- self.output = self.output.rstrip()
+ self.output = self.output.decode("utf-8").rstrip()
self.status = self.process.poll()
self.log.debug("Command '%s' returned %d as exit code." % (self.cmd, self.status))
diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 695402f..f51de99 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -71,7 +71,8 @@ class QemuRunner:
if self.logfile:
# It is needed to sanitize the data received from qemu
# because is possible to have control characters
- msg = re_control_char.sub('', unicode(msg, 'utf-8'))
+ msg = msg.decode("utf-8")
+ msg = re_control_char.sub('', msg)
with codecs.open(self.logfile, "a", encoding="utf-8") as f:
f.write("%s" % msg)
@@ -79,7 +80,7 @@ class QemuRunner:
import fcntl
fl = fcntl.fcntl(o, fcntl.F_GETFL)
fcntl.fcntl(o, fcntl.F_SETFL, fl | os.O_NONBLOCK)
- return os.read(o.fileno(), 1000000)
+ return os.read(o.fileno(), 1000000).decode("utf-8")
def handleSIGCHLD(self, signum, frame):
@@ -229,14 +230,19 @@ class QemuRunner:
socklist.remove(self.server_socket)
logger.info("Connection from %s:%s" % addr)
else:
- data = sock.recv(1024)
+ data = data + sock.recv(1024)
if data:
- bootlog += data
- if re.search(".* login:", bootlog):
- self.server_socket = qemusock
- stopread = True
- reachedlogin = True
- logger.info("Reached login banner")
+ try:
+ data = data.decode("utf-8")
+ bootlog += data
+ data = b''
+ if re.search(".* login:", bootlog):
+ self.server_socket = qemusock
+ stopread = True
+ reachedlogin = True
+ logger.info("Reached login banner")
+ except UnicodeDecodeError:
+ continue
else:
socklist.remove(sock)
sock.close()
@@ -325,7 +331,7 @@ class QemuRunner:
# Walk the process tree from the process specified looking for a qemu-system. Return its [pid'cmd]
#
ps = subprocess.Popen(['ps', 'axww', '-o', 'pid,ppid,command'], stdout=subprocess.PIPE).communicate()[0]
- processes = ps.split('\n')
+ processes = ps.decode("utf-8").split('\n')
nfields = len(processes[0].split()) - 1
pids = {}
commands = {}
diff --git a/meta/lib/oeqa/utils/qemutinyrunner.py b/meta/lib/oeqa/utils/qemutinyrunner.py
index d6ce096..054ab0e 100644
--- a/meta/lib/oeqa/utils/qemutinyrunner.py
+++ b/meta/lib/oeqa/utils/qemutinyrunner.py
@@ -102,7 +102,7 @@ class QemuTinyRunner(QemuRunner):
bb.note("Qemu pid didn't appeared in %s seconds" % self.runqemutime)
output = self.runqemu.stdout
self.stop()
- bb.note("Output from runqemu:\n%s" % output.read())
+ bb.note("Output from runqemu:\n%s" % output.read().decode("utf-8"))
return False
return self.is_alive()
@@ -131,7 +131,7 @@ class QemuTinyRunner(QemuRunner):
# Walk the process tree from the process specified looking for a qemu-system. Return its [pid'cmd]
#
ps = subprocess.Popen(['ps', 'axww', '-o', 'pid,ppid,command'], stdout=subprocess.PIPE).communicate()[0]
- processes = ps.split('\n')
+ processes = ps.decode("utf-8").split('\n')
nfields = len(processes[0].split()) - 1
pids = {}
commands = {}
diff --git a/meta/lib/oeqa/utils/sshcontrol.py b/meta/lib/oeqa/utils/sshcontrol.py
index ff88d37..f5d46e0 100644
--- a/meta/lib/oeqa/utils/sshcontrol.py
+++ b/meta/lib/oeqa/utils/sshcontrol.py
@@ -58,6 +58,7 @@ class SSHProcess(object):
self.process.stdout.close()
eof = True
else:
+ data = data.decode("utf-8")
output += data
self.log(data)
endtime = time.time() + timeout
diff --git a/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc b/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc
index 30307bc..371c279 100644
--- a/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc
+++ b/meta/recipes-graphics/xorg-xserver/xserver-xorg.inc
@@ -168,9 +168,10 @@ python populate_packages_prepend() {
}
p = subprocess.Popen(args="pkg-config --variable=%s xorg-server" % abis[name],
shell=True, env=newenv, stdout=subprocess.PIPE)
- output = p.communicate()[0]
+ stdout, stderr = p.communicate()
+ output = stdout.decode("utf-8").split(".")[0]
mlprefix = d.getVar('MLPREFIX', True) or ''
- return "%sxorg-abi-%s-%s" % (mlprefix, name, output.split(".")[0])
+ return "%sxorg-abi-%s-%s" % (mlprefix, name, output)
pn = d.getVar("PN", True)
d.appendVar("RPROVIDES_" + pn, " " + get_abi("input"))
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 04/22] classes/lib: Update to match python3 iter requirements
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
2016-06-01 12:35 ` [PATCH 02/22] classes/lib: Convert to use python3 octal syntax Richard Purdie
2016-06-01 12:35 ` [PATCH 03/22] classes/lib: Update to use python3 command pipeline decoding Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 05/22] classes/lib: Update to explictly create lists where needed Richard Purdie
` (20 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
python3 standardises its use of iteration operations. Update
the code to match the for python3 requires.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/classes/buildhistory.bbclass | 2 +-
meta/classes/license.bbclass | 4 ++--
meta/classes/populate_sdk_ext.bbclass | 4 ++--
meta/lib/oe/buildhistory_analysis.py | 4 ++--
meta/lib/oe/copy_buildsystem.py | 2 +-
meta/lib/oe/data.py | 2 +-
meta/lib/oe/recipeutils.py | 8 ++++----
meta/lib/oe/sstatesig.py | 2 +-
meta/lib/oeqa/selftest/devtool.py | 2 +-
meta/lib/oeqa/selftest/sstatetests.py | 4 ++--
meta/lib/oeqa/utils/decorators.py | 2 +-
11 files changed, 18 insertions(+), 18 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index e3b5c44..cc233b5 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -838,7 +838,7 @@ python write_srcrev() {
f.write('# SRCREV_%s = "%s"\n' % (name, orig_srcrev))
f.write('SRCREV_%s = "%s"\n' % (name, srcrev))
else:
- f.write('SRCREV = "%s"\n' % srcrevs.itervalues().next())
+ f.write('SRCREV = "%s"\n' % srcrevs.values())
if len(tag_srcrevs) > 0:
for name, srcrev in tag_srcrevs.items():
f.write('# tag_%s = "%s"\n' % (name, srcrev))
diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index 69335d6..538ab19 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -200,7 +200,7 @@ def get_deployed_dependencies(d):
# it might contain the bootloader.
taskdata = d.getVar("BB_TASKDEPDATA", False)
depends = list(set([dep[0] for dep
- in taskdata.itervalues()
+ in list(taskdata.values())
if not dep[0].endswith("-native")]))
extra_depends = d.getVar("EXTRA_IMAGEDEPENDS", True)
boot_depends = get_boot_dependencies(d)
@@ -261,7 +261,7 @@ def get_boot_dependencies(d):
depends.append(dep)
# We need to search for the provider of the dependency
else:
- for taskdep in taskdepdata.itervalues():
+ for taskdep in taskdepdata.values():
# The fifth field contains what the task provides
if dep in taskdep[4]:
info_file = os.path.join(
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 4c3a038..a9c6fe5 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -45,7 +45,7 @@ def get_sdk_install_targets(d):
sdk_install_targets = d.getVar('SDK_TARGETS', True)
depd = d.getVar('BB_TASKDEPDATA', False)
- for v in depd.itervalues():
+ for v in depd.values():
if v[1] == 'do_image_complete':
if v[0] not in sdk_install_targets:
sdk_install_targets += ' {}'.format(v[0])
@@ -267,7 +267,7 @@ python copy_buildsystem () {
# Ensure any variables set from the external environment (by way of
# BB_ENV_EXTRAWHITE) are set in the SDK's configuration
extralines = []
- for name, value in env_whitelist_values.iteritems():
+ for name, value in env_whitelist_values.items():
actualvalue = d.getVar(name, True) or ''
if value != actualvalue:
extralines.append('%s = "%s"\n' % (name, actualvalue))
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index 5395c76..0dcd49f 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -62,7 +62,7 @@ class ChangeRecord:
def pkglist_combine(depver):
pkglist = []
- for k,v in depver.iteritems():
+ for k,v in depver.items():
if v:
pkglist.append("%s (%s)" % (k,v))
else:
@@ -220,7 +220,7 @@ def compare_file_lists(alines, blines):
adict = file_list_to_dict(alines)
bdict = file_list_to_dict(blines)
filechanges = []
- for path, splitv in adict.iteritems():
+ for path, splitv in adict.items():
newsplitv = bdict.pop(path, None)
if newsplitv:
# Check type
diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index 7b9a0ee..0589b7f 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -124,7 +124,7 @@ class BuildSystem(object):
def generate_locked_sigs(sigfile, d):
bb.utils.mkdirhier(os.path.dirname(sigfile))
depd = d.getVar('BB_TASKDEPDATA', False)
- tasks = ['%s.%s' % (v[2], v[1]) for v in depd.itervalues()]
+ tasks = ['%s.%s' % (v[2], v[1]) for v in depd.values()]
bb.parse.siggen.dump_lockedsigs(sigfile, tasks)
def prune_lockedsigs(excluded_tasks, excluded_targets, lockedsigs, pruned_output):
diff --git a/meta/lib/oe/data.py b/meta/lib/oe/data.py
index 23a9067..ee48950 100644
--- a/meta/lib/oe/data.py
+++ b/meta/lib/oe/data.py
@@ -7,7 +7,7 @@ def typed_value(key, d):
flags = d.getVarFlags(key)
if flags is not None:
flags = dict((flag, d.expand(value))
- for flag, value in flags.iteritems())
+ for flag, value in list(flags.items()))
else:
flags = {}
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 3e17873..b437720 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -336,7 +336,7 @@ def patch_recipe(d, fn, varvalues, patch=False, relpath=''):
varfiles = get_var_files(fn, varlist, d)
locs = localise_file_vars(fn, varfiles, varlist)
patches = []
- for f,v in locs.iteritems():
+ for f,v in locs.items():
vals = {k: varvalues[k] for k in v}
patchdata = patch_recipe_file(f, vals, patch, relpath)
if patch:
@@ -554,7 +554,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
bbappendlines = []
if extralines:
if isinstance(extralines, dict):
- for name, value in extralines.iteritems():
+ for name, value in extralines.items():
bbappendlines.append((name, '=', value))
else:
# Do our best to split it
@@ -594,7 +594,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
copyfiles = {}
if srcfiles:
instfunclines = []
- for newfile, origsrcfile in srcfiles.iteritems():
+ for newfile, origsrcfile in srcfiles.items():
srcfile = origsrcfile
srcurientry = None
if not srcfile:
@@ -717,7 +717,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
if copyfiles:
if machine:
destsubdir = os.path.join(destsubdir, machine)
- for newfile, srcfile in copyfiles.iteritems():
+ for newfile, srcfile in copyfiles.items():
filedest = os.path.join(appenddir, destsubdir, os.path.basename(srcfile))
if os.path.abspath(newfile) != os.path.abspath(filedest):
bb.note('Copying %s to %s' % (newfile, filedest))
diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 01dce66..500122d 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -220,7 +220,7 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
for task in range(len(sq_fn)):
if task not in ret:
for pn in self.lockedsigs:
- if sq_hash[task] in self.lockedsigs[pn].itervalues():
+ if sq_hash[task] in iter(self.lockedsigs[pn].values()):
if sq_task[task] == 'do_shared_workdir':
continue
sstate_missing_msgs.append("Locked sig is set for %s:%s (%s) yet not in sstate cache?"
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index b64f9b3..6d2417e 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -50,7 +50,7 @@ class DevtoolBase(oeSelfTest):
missingvars = {}
- for var, value in checkvars.iteritems():
+ for var, value in checkvars.items():
if value is not None:
missingvars[var] = value
self.assertEqual(missingvars, {}, 'Some expected variables not found in recipe: %s' % checkvars)
diff --git a/meta/lib/oeqa/selftest/sstatetests.py b/meta/lib/oeqa/selftest/sstatetests.py
index a1e5d33..cc64c6c 100644
--- a/meta/lib/oeqa/selftest/sstatetests.py
+++ b/meta/lib/oeqa/selftest/sstatetests.py
@@ -445,14 +445,14 @@ http_proxy = "http://example.com/"
files1 = get_files(topdir + "/tmp-sstatesamehash/stamps/")
files2 = get_files(topdir + "/tmp-sstatesamehash2/stamps/")
# Remove items that are identical in both sets
- for k,v in files1.viewitems() & files2.viewitems():
+ for k,v in files1.items() & files2.items():
del files1[k]
del files2[k]
if not files1 and not files2:
# No changes, so we're done
return
- for k in files1.viewkeys() | files2.viewkeys():
+ for k in files1.keys() | files2.keys():
if k in files1 and k in files2:
print("%s differs:" % k)
print(subprocess.check_output(("bitbake-diffsigs",
diff --git a/meta/lib/oeqa/utils/decorators.py b/meta/lib/oeqa/utils/decorators.py
index d52f326..6fb09db 100644
--- a/meta/lib/oeqa/utils/decorators.py
+++ b/meta/lib/oeqa/utils/decorators.py
@@ -215,7 +215,7 @@ def tag(*args, **kwargs):
def wrap_ob(ob):
for name in args:
setattr(ob, __tag_prefix + name, True)
- for name, value in kwargs.iteritems():
+ for name, value in kwargs.items():
setattr(ob, __tag_prefix + name, value)
return ob
return wrap_ob
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 05/22] classes/lib: Update to explictly create lists where needed
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (2 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 04/22] classes/lib: Update to match python3 iter requirements Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 06/22] classes/lib: Update xrange -> range for python3 Richard Purdie
` (19 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
Iterators now return views, not lists in python3. Where we need
lists, handle this explicitly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/classes/buildhistory.bbclass | 2 +-
meta/classes/license.bbclass | 2 +-
meta/classes/package.bbclass | 4 +--
meta/lib/oe/classutils.py | 2 +-
meta/lib/oe/copy_buildsystem.py | 2 +-
meta/lib/oe/distro_check.py | 4 +--
meta/lib/oe/license.py | 10 +++----
meta/lib/oe/manifest.py | 4 +--
meta/lib/oe/package_manager.py | 6 ++---
meta/lib/oe/packagedata.py | 2 +-
meta/lib/oe/packagegroup.py | 4 +--
meta/lib/oe/prservice.py | 4 +--
meta/lib/oe/recipeutils.py | 6 ++---
meta/lib/oe/rootfs.py | 4 +--
meta/lib/oe/sstatesig.py | 2 +-
meta/lib/oe/utils.py | 6 ++---
meta/lib/oeqa/selftest/pkgdata.py | 56 +++++++++++++++++++--------------------
meta/lib/oeqa/utils/qemurunner.py | 2 +-
scripts/lib/devtool/standard.py | 12 ++++-----
scripts/oe-selftest | 2 +-
20 files changed, 68 insertions(+), 68 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index cc233b5..1ccd9ee 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -274,7 +274,7 @@ python buildhistory_emit_pkghistory() {
# Gather information about packaged files
val = pkgdata.get('FILES_INFO', '')
dictval = json.loads(val)
- filelist = dictval.keys()
+ filelist = list(dictval.keys())
filelist.sort()
pkginfo.filelist = " ".join(filelist)
diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index 538ab19..10d6ed8 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -635,7 +635,7 @@ def check_license_format(d):
licenses = d.getVar('LICENSE', True)
from oe.license import license_operator, license_operator_chars, license_pattern
- elements = filter(lambda x: x.strip(), license_operator.split(licenses))
+ elements = list(filter(lambda x: x.strip(), license_operator.split(licenses)))
for pos, element in enumerate(elements):
if license_pattern.match(element):
if pos > 0 and license_pattern.match(elements[pos - 1]):
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 501004e..c9e2aa8 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1504,7 +1504,7 @@ python package_do_shlibs() {
m = re.match("\s+RPATH\s+([^\s]*)", l)
if m:
rpaths = m.group(1).replace("$ORIGIN", ldir).split(":")
- rpath = map(os.path.normpath, rpaths)
+ rpath = list(map(os.path.normpath, rpaths))
for l in lines:
m = re.match("\s+NEEDED\s+([^\s]*)", l)
if m:
@@ -1674,7 +1674,7 @@ python package_do_shlibs() {
bb.debug(2, '%s: Dependency %s covered by PRIVATE_LIBS' % (pkg, n[0]))
continue
if n[0] in shlib_provider.keys():
- shlib_provider_path = list()
+ shlib_provider_path = []
for k in shlib_provider[n[0]].keys():
shlib_provider_path.append(k)
match = None
diff --git a/meta/lib/oe/classutils.py b/meta/lib/oe/classutils.py
index 58188fd..98bb059 100644
--- a/meta/lib/oe/classutils.py
+++ b/meta/lib/oe/classutils.py
@@ -34,7 +34,7 @@ abstract base classes out of the registry)."""
@classmethod
def prioritized(tcls):
- return sorted(tcls.registry.values(),
+ return sorted(list(tcls.registry.values()),
key=lambda v: v.priority, reverse=True)
def unregister(cls):
diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index 0589b7f..eddf5bb 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -195,7 +195,7 @@ def merge_lockedsigs(copy_tasks, lockedsigs_main, lockedsigs_extra, merged_outpu
fulltypes.append(typename)
f.write('SIGGEN_LOCKEDSIGS_TYPES = "%s"\n' % ' '.join(fulltypes))
- write_sigs_file(copy_output, tocopy.keys(), tocopy)
+ write_sigs_file(copy_output, list(tocopy.keys()), tocopy)
if merged_output:
write_sigs_file(merged_output, arch_order, merged)
diff --git a/meta/lib/oe/distro_check.py b/meta/lib/oe/distro_check.py
index 8655a6f..ba1bba6 100644
--- a/meta/lib/oe/distro_check.py
+++ b/meta/lib/oe/distro_check.py
@@ -104,8 +104,8 @@ def get_source_package_list_from_url(url, section, d):
bb.note("Reading %s: %s" % (url, section))
links = get_links_from_url(url, d)
- srpms = filter(is_src_rpm, links)
- names_list = map(package_name_from_srpm, srpms)
+ srpms = list(filter(is_src_rpm, links))
+ names_list = list(map(package_name_from_srpm, srpms))
new_pkgs = []
for pkgs in names_list:
diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py
index f0f661c..39ef965 100644
--- a/meta/lib/oe/license.py
+++ b/meta/lib/oe/license.py
@@ -47,7 +47,7 @@ class LicenseVisitor(ast.NodeVisitor):
"""Get elements based on OpenEmbedded license strings"""
def get_elements(self, licensestr):
new_elements = []
- elements = filter(lambda x: x.strip(), license_operator.split(licensestr))
+ elements = list([x for x in license_operator.split(licensestr) if x.strip()])
for pos, element in enumerate(elements):
if license_pattern.match(element):
if pos > 0 and license_pattern.match(elements[pos-1]):
@@ -118,8 +118,8 @@ def is_included(licensestr, whitelist=None, blacklist=None):
def choose_licenses(alpha, beta):
"""Select the option in an OR which is the 'best' (has the most
included licenses)."""
- alpha_weight = len(filter(include_license, alpha))
- beta_weight = len(filter(include_license, beta))
+ alpha_weight = len(list(filter(include_license, alpha)))
+ beta_weight = len(list(filter(include_license, beta)))
if alpha_weight > beta_weight:
return alpha
else:
@@ -132,8 +132,8 @@ def is_included(licensestr, whitelist=None, blacklist=None):
blacklist = []
licenses = flattened_licenses(licensestr, choose_licenses)
- excluded = filter(lambda lic: exclude_license(lic), licenses)
- included = filter(lambda lic: include_license(lic), licenses)
+ excluded = [lic for lic in licenses if exclude_license(lic)]
+ included = [lic for lic in licenses if include_license(lic)]
if excluded:
return False, excluded
else:
diff --git a/meta/lib/oe/manifest.py b/meta/lib/oe/manifest.py
index 42832f1..ec2ef50 100644
--- a/meta/lib/oe/manifest.py
+++ b/meta/lib/oe/manifest.py
@@ -219,7 +219,7 @@ class RpmManifest(Manifest):
if var in self.vars_to_split:
split_pkgs = self._split_multilib(self.d.getVar(var, True))
if split_pkgs is not None:
- pkgs = dict(pkgs.items() + split_pkgs.items())
+ pkgs = dict(list(pkgs.items()) + list(split_pkgs.items()))
else:
pkg_list = self.d.getVar(var, True)
if pkg_list is not None:
@@ -269,7 +269,7 @@ class OpkgManifest(Manifest):
if var in self.vars_to_split:
split_pkgs = self._split_multilib(self.d.getVar(var, True))
if split_pkgs is not None:
- pkgs = dict(pkgs.items() + split_pkgs.items())
+ pkgs = dict(list(pkgs.items()) + list(split_pkgs.items()))
else:
pkg_list = self.d.getVar(var, True)
if pkg_list is not None:
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index abe9f68..54e6970 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -643,8 +643,8 @@ class PackageManager(object):
def construct_uris(self, uris, base_paths):
def _append(arr1, arr2, sep='/'):
res = []
- narr1 = map(lambda a: string.rstrip(a, sep), arr1)
- narr2 = map(lambda a: string.lstrip(string.rstrip(a, sep), sep), arr2)
+ narr1 = [string.rstrip(a, sep) for a in arr1]
+ narr2 = [string.lstrip(string.rstrip(a, sep), sep) for a in arr2]
for a1 in narr1:
if arr2:
for a2 in narr2:
@@ -1111,7 +1111,7 @@ class RpmPM(PackageManager):
sub_rdep = sub_data.get("RDEPENDS_" + pkg)
if not sub_rdep:
continue
- done = bb.utils.explode_dep_versions2(sub_rdep).keys()
+ done = list(bb.utils.explode_dep_versions2(sub_rdep).keys())
next = done
# Find all the rdepends on dependency chain
while next:
diff --git a/meta/lib/oe/packagedata.py b/meta/lib/oe/packagedata.py
index df1b4c5..21d4de9 100644
--- a/meta/lib/oe/packagedata.py
+++ b/meta/lib/oe/packagedata.py
@@ -66,7 +66,7 @@ def _pkgmap(d):
bb.warn("No files in %s?" % pkgdatadir)
files = []
- for pn in filter(lambda f: not os.path.isdir(os.path.join(pkgdatadir, f)), files):
+ for pn in [f for f in files if not os.path.isdir(os.path.join(pkgdatadir, f))]:
try:
pkgdata = read_pkgdatafile(os.path.join(pkgdatadir, pn))
except OSError:
diff --git a/meta/lib/oe/packagegroup.py b/meta/lib/oe/packagegroup.py
index a6fee5f..9781927 100644
--- a/meta/lib/oe/packagegroup.py
+++ b/meta/lib/oe/packagegroup.py
@@ -16,11 +16,11 @@ def packages(features, d):
yield pkg
def required_packages(features, d):
- req = filter(lambda feature: not is_optional(feature, d), features)
+ req = [feature for feature in features if not is_optional(feature, d)]
return packages(req, d)
def optional_packages(features, d):
- opt = filter(lambda feature: is_optional(feature, d), features)
+ opt = [feature for feature in features if is_optional(feature, d)]
return packages(opt, d)
def active_packages(features, d):
diff --git a/meta/lib/oe/prservice.py b/meta/lib/oe/prservice.py
index 9e5a0c9..0054f95 100644
--- a/meta/lib/oe/prservice.py
+++ b/meta/lib/oe/prservice.py
@@ -1,7 +1,7 @@
def prserv_make_conn(d, check = False):
import prserv.serv
- host_params = filter(None, (d.getVar("PRSERV_HOST", True) or '').split(':'))
+ host_params = list([_f for _f in (d.getVar("PRSERV_HOST", True) or '').split(':') if _f])
try:
conn = None
conn = prserv.serv.PRServerConnection(host_params[0], int(host_params[1]))
@@ -114,7 +114,7 @@ def prserv_export_tofile(d, metainfo, datainfo, lockdown, nomax=False):
bb.utils.unlockfile(lf)
def prserv_check_avail(d):
- host_params = filter(None, (d.getVar("PRSERV_HOST", True) or '').split(':'))
+ host_params = list([_f for _f in (d.getVar("PRSERV_HOST", True) or '').split(':') if _f])
try:
if len(host_params) != 2:
raise TypeError
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index b437720..146fe83 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -692,7 +692,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
varnames = [item[0] for item in bbappendlines]
if removevalues:
- varnames.extend(removevalues.keys())
+ varnames.extend(list(removevalues.keys()))
with open(appendpath, 'r') as f:
(updated, newlines) = bb.utils.edit_metadata(f, varnames, appendfile_varfunc)
@@ -743,12 +743,12 @@ def replace_dir_vars(path, d):
"""Replace common directory paths with appropriate variable references (e.g. /etc becomes ${sysconfdir})"""
dirvars = {}
# Sort by length so we get the variables we're interested in first
- for var in sorted(d.keys(), key=len):
+ for var in sorted(list(d.keys()), key=len):
if var.endswith('dir') and var.lower() == var:
value = d.getVar(var, True)
if value.startswith('/') and not '\n' in value and value not in dirvars:
dirvars[value] = var
- for dirpath in sorted(dirvars.keys(), reverse=True):
+ for dirpath in sorted(list(dirvars.keys()), reverse=True):
path = path.replace(dirpath, '${%s}' % dirvars[dirpath])
return path
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 7087b12..d934858 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -536,7 +536,7 @@ class DpkgOpkgRootfs(Rootfs):
pkg_depends = m_depends.group(1)
# remove package dependencies not in postinsts
- pkg_names = pkgs.keys()
+ pkg_names = list(pkgs.keys())
for pkg_name in pkg_names:
deps = pkgs[pkg_name][:]
@@ -569,7 +569,7 @@ class DpkgOpkgRootfs(Rootfs):
pkgs = self._get_pkgs_postinsts(status_file)
if pkgs:
root = "__packagegroup_postinst__"
- pkgs[root] = pkgs.keys()
+ pkgs[root] = list(pkgs.keys())
_dep_resolve(pkgs, root, pkg_list, [])
pkg_list.remove(root)
diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 500122d..a58f03a 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -210,7 +210,7 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
continue
f.write(" " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n")
f.write(' "\n')
- f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys())))
+ f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(list(types.keys()))))
def checkhashes(self, missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d):
warn_msgs = []
diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
index 32d6179..1bbdbb4 100644
--- a/meta/lib/oe/utils.py
+++ b/meta/lib/oe/utils.py
@@ -85,11 +85,11 @@ def prune_suffix(var, suffixes, d):
def str_filter(f, str, d):
from re import match
- return " ".join(filter(lambda x: match(f, x, 0), str.split()))
+ return " ".join([x for x in str.split() if match(f, x, 0)])
def str_filter_out(f, str, d):
from re import match
- return " ".join(filter(lambda x: not match(f, x, 0), str.split()))
+ return " ".join([x for x in str.split() if not match(f, x, 0)])
def param_bool(cfg, field, dflt = None):
"""Lookup <field> in <cfg> map and convert it to a boolean; take
@@ -134,7 +134,7 @@ def packages_filter_out_system(d):
PN-dbg PN-doc PN-locale-eb-gb removed.
"""
pn = d.getVar('PN', True)
- blacklist = map(lambda suffix: pn + suffix, ('', '-dbg', '-dev', '-doc', '-locale', '-staticdev'))
+ blacklist = [pn + suffix for suffix in ('', '-dbg', '-dev', '-doc', '-locale', '-staticdev')]
localepkg = pn + "-locale-"
pkgs = []
diff --git a/meta/lib/oeqa/selftest/pkgdata.py b/meta/lib/oeqa/selftest/pkgdata.py
index 138b03a..5a63f89 100644
--- a/meta/lib/oeqa/selftest/pkgdata.py
+++ b/meta/lib/oeqa/selftest/pkgdata.py
@@ -131,15 +131,15 @@ class OePkgdataUtilTests(oeSelfTest):
# Test recipe-space package name
result = runCmd('oe-pkgdata-util list-pkg-files zlib-dev zlib-doc')
files = splitoutput(result.output)
- self.assertIn('zlib-dev', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('zlib-doc', files.keys(), "listed pkgs. files: %s" %result.output)
+ self.assertIn('zlib-dev', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('zlib-doc', list(files.keys()), "listed pkgs. files: %s" %result.output)
self.assertIn(os.path.join(includedir, 'zlib.h'), files['zlib-dev'])
self.assertIn(os.path.join(mandir, 'man3/zlib.3'), files['zlib-doc'])
# Test runtime package name
result = runCmd('oe-pkgdata-util list-pkg-files -r libz1 libz-dev')
files = splitoutput(result.output)
- self.assertIn('libz1', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('libz-dev', files.keys(), "listed pkgs. files: %s" %result.output)
+ self.assertIn('libz1', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('libz-dev', list(files.keys()), "listed pkgs. files: %s" %result.output)
self.assertGreater(len(files['libz1']), 1)
libspec = os.path.join(base_libdir, 'libz.so.1.*')
found = False
@@ -152,12 +152,12 @@ class OePkgdataUtilTests(oeSelfTest):
# Test recipe
result = runCmd('oe-pkgdata-util list-pkg-files -p zlib')
files = splitoutput(result.output)
- self.assertIn('zlib-dbg', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('zlib-doc', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('zlib-dev', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('zlib-staticdev', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('zlib', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertNotIn('zlib-locale', files.keys(), "listed pkgs. files: %s" %result.output)
+ self.assertIn('zlib-dbg', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('zlib-doc', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('zlib-dev', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('zlib-staticdev', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('zlib', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertNotIn('zlib-locale', list(files.keys()), "listed pkgs. files: %s" %result.output)
# (ignore ptest, might not be there depending on config)
self.assertIn(os.path.join(includedir, 'zlib.h'), files['zlib-dev'])
self.assertIn(os.path.join(mandir, 'man3/zlib.3'), files['zlib-doc'])
@@ -165,36 +165,36 @@ class OePkgdataUtilTests(oeSelfTest):
# Test recipe, runtime
result = runCmd('oe-pkgdata-util list-pkg-files -p zlib -r')
files = splitoutput(result.output)
- self.assertIn('libz-dbg', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('libz-doc', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('libz-dev', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('libz-staticdev', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('libz1', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertNotIn('libz-locale', files.keys(), "listed pkgs. files: %s" %result.output)
+ self.assertIn('libz-dbg', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('libz-doc', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('libz-dev', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('libz-staticdev', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('libz1', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertNotIn('libz-locale', list(files.keys()), "listed pkgs. files: %s" %result.output)
self.assertIn(os.path.join(includedir, 'zlib.h'), files['libz-dev'])
self.assertIn(os.path.join(mandir, 'man3/zlib.3'), files['libz-doc'])
self.assertIn(os.path.join(libdir, 'libz.a'), files['libz-staticdev'])
# Test recipe, unpackaged
result = runCmd('oe-pkgdata-util list-pkg-files -p zlib -u')
files = splitoutput(result.output)
- self.assertIn('zlib-dbg', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('zlib-doc', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('zlib-dev', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('zlib-staticdev', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('zlib', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('zlib-locale', files.keys(), "listed pkgs. files: %s" %result.output) # this is the key one
+ self.assertIn('zlib-dbg', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('zlib-doc', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('zlib-dev', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('zlib-staticdev', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('zlib', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('zlib-locale', list(files.keys()), "listed pkgs. files: %s" %result.output) # this is the key one
self.assertIn(os.path.join(includedir, 'zlib.h'), files['zlib-dev'])
self.assertIn(os.path.join(mandir, 'man3/zlib.3'), files['zlib-doc'])
self.assertIn(os.path.join(libdir, 'libz.a'), files['zlib-staticdev'])
# Test recipe, runtime, unpackaged
result = runCmd('oe-pkgdata-util list-pkg-files -p zlib -r -u')
files = splitoutput(result.output)
- self.assertIn('libz-dbg', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('libz-doc', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('libz-dev', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('libz-staticdev', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('libz1', files.keys(), "listed pkgs. files: %s" %result.output)
- self.assertIn('libz-locale', files.keys(), "listed pkgs. files: %s" %result.output) # this is the key one
+ self.assertIn('libz-dbg', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('libz-doc', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('libz-dev', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('libz-staticdev', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('libz1', list(files.keys()), "listed pkgs. files: %s" %result.output)
+ self.assertIn('libz-locale', list(files.keys()), "listed pkgs. files: %s" %result.output) # this is the key one
self.assertIn(os.path.join(includedir, 'zlib.h'), files['libz-dev'])
self.assertIn(os.path.join(mandir, 'man3/zlib.3'), files['libz-doc'])
self.assertIn(os.path.join(libdir, 'libz.a'), files['libz-staticdev'])
diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index f51de99..c1d07d9 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -22,7 +22,7 @@ import logging
logger = logging.getLogger("BitBake.QemuRunner")
# Get Unicode non printable control chars
-control_range = range(0,32)+range(127,160)
+control_range = list(range(0,32))+list(range(127,160))
control_chars = [unichr(x) for x in control_range
if unichr(x) not in string.printable]
re_control_char = re.compile('[%s]' % re.escape("".join(control_chars)))
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 3be3214..18847cf 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -321,7 +321,7 @@ def _git_exclude_path(srctree, path):
# becomes greater than that.
path = os.path.normpath(path)
recurse = True if len(path.split(os.path.sep)) > 1 else False
- git_files = _git_ls_tree(srctree, 'HEAD', recurse).keys()
+ git_files = list(_git_ls_tree(srctree, 'HEAD', recurse).keys())
if path in git_files:
git_files.remove(path)
return git_files
@@ -1073,14 +1073,14 @@ def _update_recipe_srcrev(args, srctree, rd, config_data):
patches_dir)
# Remove deleted local files and "overlapping" patches
- remove_files = del_f.values() + upd_p.values()
+ remove_files = list(del_f.values()) + list(upd_p.values())
if remove_files:
removedentries = _remove_file_entries(srcuri, remove_files)[0]
update_srcuri = True
if args.append:
files = dict((os.path.join(local_files_dir, key), val) for
- key, val in upd_f.items() + new_f.items())
+ key, val in list(upd_f.items()) + list(new_f.items()))
removevalues = {}
if update_srcuri:
removevalues = {'SRC_URI': removedentries}
@@ -1142,7 +1142,7 @@ def _update_recipe_patch(args, config, workspace, srctree, rd, config_data):
upd_p, new_p, del_p = _export_patches(srctree, rd, initial_rev,
all_patches_dir)
# Remove deleted local files and patches
- remove_files = del_f.values() + del_p.values()
+ remove_files = list(del_f.values()) + list(del_p.values())
# Get updated patches from source tree
patches_dir = tempfile.mkdtemp(dir=tempdir)
@@ -1154,9 +1154,9 @@ def _update_recipe_patch(args, config, workspace, srctree, rd, config_data):
srcuri = (rd.getVar('SRC_URI', False) or '').split()
if args.append:
files = dict((os.path.join(local_files_dir, key), val) for
- key, val in upd_f.items() + new_f.items())
+ key, val in list(upd_f.items()) + list(new_f.items()))
files.update(dict((os.path.join(patches_dir, key), val) for
- key, val in upd_p.items() + new_p.items()))
+ key, val in list(upd_p.items()) + list(new_p.items())))
if files or remove_files:
removevalues = None
if remove_files:
diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index 00ef51f..db132fd 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -211,7 +211,7 @@ def get_tests_from_module(tmod):
try:
import importlib
modlib = importlib.import_module(tmod)
- for mod in vars(modlib).values():
+ for mod in list(vars(modlib).values()):
if isinstance(mod, type(oeSelfTest)) and issubclass(mod, oeSelfTest) and mod is not oeSelfTest:
for test in dir(mod):
if test.startswith('test_') and hasattr(vars(mod)[test], '__call__'):
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 06/22] classes/lib: Update xrange -> range for python3
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (3 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 05/22] classes/lib: Update to explictly create lists where needed Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 07/22] sanity: Switch urlparse to urllib.parse Richard Purdie
` (18 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
xrange() no longer exists in python 3, use range()
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/classes/staging.bbclass | 2 +-
meta/lib/oe/recipeutils.py | 4 ++--
meta/recipes-core/meta/uninative-tarball.bb | 2 +-
meta/recipes-rt/rt-tests/files/rt_bmark.py | 2 +-
scripts/lib/devtool/standard.py | 4 ++--
5 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index a0f82be..30f2b50 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -187,7 +187,7 @@ def sysroot_checkhashes(covered, tasknames, fnids, fns, d, invalidtasks = None):
problems = set()
configurefnids = set()
if not invalidtasks:
- invalidtasks = xrange(len(tasknames))
+ invalidtasks = range(len(tasknames))
for task in invalidtasks:
if tasknames[task] == "do_configure" and task not in covered:
configurefnids.add(fnids[task])
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 146fe83..632184f 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -568,14 +568,14 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
raise Exception('Invalid extralines value passed')
def popline(varname):
- for i in xrange(0, len(bbappendlines)):
+ for i in range(0, len(bbappendlines)):
if bbappendlines[i][0] == varname:
line = bbappendlines.pop(i)
return line
return None
def appendline(varname, op, value):
- for i in xrange(0, len(bbappendlines)):
+ for i in range(0, len(bbappendlines)):
item = bbappendlines[i]
if item[0] == varname:
bbappendlines[i] = (item[0], item[1], item[2] + ' ' + value)
diff --git a/meta/recipes-core/meta/uninative-tarball.bb b/meta/recipes-core/meta/uninative-tarball.bb
index a983e42..fb147a1 100644
--- a/meta/recipes-core/meta/uninative-tarball.bb
+++ b/meta/recipes-core/meta/uninative-tarball.bb
@@ -50,7 +50,7 @@ fakeroot tar_sdk() {
DEST="./${SDK_ARCH}-${SDK_OS}"
mv sysroots/${SDK_SYS} $DEST
rm sysroots -rf
- patchelf --set-interpreter ${@''.join('a' for n in xrange(1024))} $DEST/usr/bin/patchelf
+ patchelf --set-interpreter ${@''.join('a' for n in range(1024))} $DEST/usr/bin/patchelf
mv $DEST/usr/bin/patchelf $DEST/usr/bin/patchelf-uninative
tar ${SDKTAROPTS} -c -j --file=${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.bz2 .
}
diff --git a/meta/recipes-rt/rt-tests/files/rt_bmark.py b/meta/recipes-rt/rt-tests/files/rt_bmark.py
index e2280e4..0e2b105 100755
--- a/meta/recipes-rt/rt-tests/files/rt_bmark.py
+++ b/meta/recipes-rt/rt-tests/files/rt_bmark.py
@@ -324,7 +324,7 @@ def run_cyclictest_suite():
t = time.time()
max_list = []
- for i in xrange(0, suite_size):
+ for i in range(0, suite_size):
tmp_min, tmp_avg, tmp_max = run_cyclictest_once()
msg = "%2d/%2d:" % (i+1, suite_size)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 18847cf..d6186b9 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -837,7 +837,7 @@ def _get_patchset_revs(args, srctree, recipe_path):
stdout, _ = bb.process.run('git rev-list --reverse %s..HEAD' %
initial_rev, cwd=srctree)
newcommits = stdout.split()
- for i in xrange(min(len(commits), len(newcommits))):
+ for i in range(min(len(commits), len(newcommits))):
if newcommits[i] == commits[i]:
update_rev = commits[i]
@@ -863,7 +863,7 @@ def _remove_file_entries(srcuri, filelist):
entries = []
for fname in filelist:
basename = os.path.basename(fname)
- for i in xrange(len(srcuri)):
+ for i in range(len(srcuri)):
if (srcuri[i].startswith('file://') and
os.path.basename(srcuri[i].split(';')[0]) == basename):
entries.append(srcuri[i])
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 07/22] sanity: Switch urlparse to urllib.parse
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (4 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 06/22] classes/lib: Update xrange -> range for python3 Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 08/22] wic: use python3 in shebang Richard Purdie
` (17 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Jeremy Puhlman <jpuhlman@mvista.com>
urlparse is replaced with urllib.parse functionality in python3
Signed-off-by: Jeremy Puhlman <jpuhlman@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/classes/sanity.bbclass | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index b0e7451..0b63d6e 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -904,13 +904,13 @@ def check_sanity_everybuild(status, d):
continue
if mirror.startswith('file://'):
- import urlparse
- check_symlink(urlparse.urlparse(mirror).path, d)
+ import urllib
+ check_symlink(urllib.parse.urlparse(mirror).path, d)
# SSTATE_MIRROR ends with a /PATH string
if mirror.endswith('/PATH'):
# remove /PATH$ from SSTATE_MIRROR to get a working
# base directory path
- mirror_base = urlparse.urlparse(mirror[:-1*len('/PATH')]).path
+ mirror_base = urllib.parse.urlparse(mirror[:-1*len('/PATH')]).path
check_symlink(mirror_base, d)
# Check that TMPDIR hasn't changed location since the last time we were run
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 08/22] wic: use python3 in shebang
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (5 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 07/22] sanity: Switch urlparse to urllib.parse Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 09/22] wic: encode help text Richard Purdie
` (16 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Ed Bartosh <ed.bartosh@linux.intel.com>
Switched scripts/wic to use python3 as a default
python interpreter.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
scripts/wic | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/wic b/scripts/wic
index 3d33430..5cc06f4 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
# ex:ts=4:sw=4:sts=4:et
# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 09/22] wic: encode help text
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (6 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 08/22] wic: use python3 in shebang Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 10/22] qemurunner: convert data when working with socket Richard Purdie
` (15 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Ed Bartosh <ed.bartosh@linux.intel.com>
Encoded help text before sending it to pager.communicate as
it expects binary.
[YOCTO #9412]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
scripts/lib/wic/help.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 6395596..e5347ec 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -45,7 +45,7 @@ def display_help(subcommand, subcommands):
if callable(hlp):
hlp = hlp()
pager = subprocess.Popen('less', stdin=subprocess.PIPE)
- pager.communicate(hlp)
+ pager.communicate(hlp.encode('utf-8'))
return True
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 10/22] qemurunner: convert data when working with socket
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (7 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 09/22] wic: encode help text Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 11/22] scripts: Fix deprecated dict methods for python3 Richard Purdie
` (14 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Ed Bartosh <ed.bartosh@linux.intel.com>
Converted str to bytes before sending to socket.
Converted bytes to str after receiving from socket.
This should fix TypeError: 'str' does not support the buffer interface
for qemurunner.run_serial method.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/lib/oeqa/utils/qemurunner.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index c1d07d9..3d60433 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -376,14 +376,14 @@ class QemuRunner:
data = ''
status = 0
- self.server_socket.sendall(command)
+ self.server_socket.sendall(command.encode('utf-8'))
keepreading = True
while keepreading:
sread, _, _ = select.select([self.server_socket],[],[],5)
if sread:
answer = self.server_socket.recv(1024)
if answer:
- data += answer
+ data += answer.decode('utf-8')
# Search the prompt to stop
if re.search("[a-zA-Z0-9]+@[a-zA-Z0-9\-]+:~#", data):
keepreading = False
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 11/22] scripts: Fix deprecated dict methods for python3
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (8 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 10/22] qemurunner: convert data when working with socket Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 12/22] scripts: Replace basestring -> str " Richard Purdie
` (13 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Ed Bartosh <ed.bartosh@linux.intel.com>
Replaced iteritems -> items, itervalues -> values,
iterkeys -> keys or 'in'
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/lib/oe/recipeutils.py | 4 ++--
scripts/buildhistory-collect-srcrevs | 2 +-
scripts/combo-layer | 2 +-
scripts/contrib/bbvars.py | 10 +++++-----
scripts/contrib/list-packageconfig-flags.py | 4 ++--
scripts/contrib/python/generate-manifest-2.7.py | 4 ++--
scripts/contrib/python/generate-manifest-3.5.py | 4 ++--
scripts/lib/devtool/__init__.py | 2 +-
scripts/lib/devtool/standard.py | 14 +++++++-------
scripts/lib/devtool/upgrade.py | 2 +-
scripts/lib/recipetool/append.py | 8 ++++----
scripts/lib/recipetool/create.py | 12 ++++++------
scripts/lib/recipetool/create_buildsys.py | 10 +++++-----
scripts/lib/recipetool/create_buildsys_python.py | 18 +++++++++---------
scripts/lib/recipetool/create_npm.py | 4 ++--
scripts/pythondeps | 2 +-
16 files changed, 51 insertions(+), 51 deletions(-)
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 632184f..1b85387 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -194,7 +194,7 @@ def patch_recipe_lines(fromlines, values, trailing_newline=True):
remainingnames = {}
for k in values.keys():
remainingnames[k] = get_recipe_pos(k)
- remainingnames = OrderedDict(sorted(remainingnames.iteritems(), key=lambda x: x[1]))
+ remainingnames = OrderedDict(sorted(remainingnames.items(), key=lambda x: x[1]))
modifying = False
@@ -234,7 +234,7 @@ def patch_recipe_lines(fromlines, values, trailing_newline=True):
if modifying:
# Insert anything that should come before this variable
pos = get_recipe_pos(varname)
- for k in remainingnames.keys()[:]:
+ for k in list(remainingnames):
if remainingnames[k] > -1 and pos >= remainingnames[k] and not k in existingnames:
outputvalue(k, newlines, rewindcomments=True)
del remainingnames[k]
diff --git a/scripts/buildhistory-collect-srcrevs b/scripts/buildhistory-collect-srcrevs
index f3eb76b..79d2657 100755
--- a/scripts/buildhistory-collect-srcrevs
+++ b/scripts/buildhistory-collect-srcrevs
@@ -103,7 +103,7 @@ def main():
if options.reportall or value != orig:
all_srcrevs[curdir].append((pn, name, srcrev))
- for curdir, srcrevs in sorted(all_srcrevs.iteritems()):
+ for curdir, srcrevs in sorted(all_srcrevs.items()):
if srcrevs:
print('# %s' % curdir)
for pn, name, srcrev in srcrevs:
diff --git a/scripts/combo-layer b/scripts/combo-layer
index e470592..52367f0 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -1178,7 +1178,7 @@ def update_with_history(conf, components, revisions, repos):
# components imported head revision.
if additional_heads:
runcmd("git reset --hard", **wargs)
- for rev, base in additional_heads.iteritems():
+ for rev, base in additional_heads.items():
apply_commit(base, rev, wargs, wargs, None)
# Commit with all component branches as parents as well as the previous head.
diff --git a/scripts/contrib/bbvars.py b/scripts/contrib/bbvars.py
index 04f5023..6f47935 100755
--- a/scripts/contrib/bbvars.py
+++ b/scripts/contrib/bbvars.py
@@ -59,8 +59,8 @@ def collect_bbvars(metadir):
for root,dirs,files in os.walk(metadir):
for name in files:
if name.find(".bb") >= 0:
- for key in recipe_bbvars(os.path.join(root,name)).iterkeys():
- if bbvars.has_key(key):
+ for key in recipe_bbvars(os.path.join(root,name)).keys():
+ if key in bbvars:
bbvars[key] = bbvars[key] + 1
else:
bbvars[key] = 1
@@ -155,15 +155,15 @@ def main():
# Collect all the variable names from the recipes in the metadirs
for m in metadirs:
- for key,cnt in collect_bbvars(m).iteritems():
- if bbvars.has_key(key):
+ for key,cnt in collect_bbvars(m).items():
+ if key in bbvars:
bbvars[key] = bbvars[key] + cnt
else:
bbvars[key] = cnt
# Check each var for documentation
varlen = 0
- for v in bbvars.iterkeys():
+ for v in bbvars.keys():
if len(v) > varlen:
varlen = len(v)
if not bbvar_is_documented(v, docfiles):
diff --git a/scripts/contrib/list-packageconfig-flags.py b/scripts/contrib/list-packageconfig-flags.py
index 5dfe796..22d0c49 100755
--- a/scripts/contrib/list-packageconfig-flags.py
+++ b/scripts/contrib/list-packageconfig-flags.py
@@ -86,7 +86,7 @@ def collect_flags(pkg_dict):
''' Collect available PACKAGECONFIG flags and all affected pkgs '''
# flag_dict = {'flag': ['pkg1', 'pkg2',...]}
flag_dict = {}
- for pkgname, flaglist in pkg_dict.iteritems():
+ for pkgname, flaglist in pkg_dict.items():
for flag in flaglist:
if flag in flag_dict:
flag_dict[flag].append(pkgname)
@@ -132,7 +132,7 @@ def display_all(data_dict):
packageconfig = 'None'
print('PACKAGECONFIG %s' % packageconfig)
- for flag,flag_val in data_dict[fn].getVarFlags("PACKAGECONFIG").iteritems():
+ for flag,flag_val in data_dict[fn].getVarFlags("PACKAGECONFIG").items():
if flag == "doc":
continue
print('PACKAGECONFIG[%s] %s' % (flag, flag_val))
diff --git a/scripts/contrib/python/generate-manifest-2.7.py b/scripts/contrib/python/generate-manifest-2.7.py
index 48be3a6..f2ecf8d 100755
--- a/scripts/contrib/python/generate-manifest-2.7.py
+++ b/scripts/contrib/python/generate-manifest-2.7.py
@@ -97,7 +97,7 @@ class MakefileMaker:
# generate package variables
#
- for name, data in sorted(self.packages.iteritems()):
+ for name, data in sorted(self.packages.items()):
desc, deps, files = data
#
@@ -130,7 +130,7 @@ class MakefileMaker:
self.out( 'SUMMARY_${PN}-modules="All Python modules"' )
line = 'RDEPENDS_${PN}-modules="'
- for name, data in sorted(self.packages.iteritems()):
+ for name, data in sorted(self.packages.items()):
if name not in ['${PN}-dev', '${PN}-distutils-staticdev']:
line += "%s " % name
diff --git a/scripts/contrib/python/generate-manifest-3.5.py b/scripts/contrib/python/generate-manifest-3.5.py
index a7b38d2..41f209b 100755
--- a/scripts/contrib/python/generate-manifest-3.5.py
+++ b/scripts/contrib/python/generate-manifest-3.5.py
@@ -100,7 +100,7 @@ class MakefileMaker:
# generate package variables
#
- for name, data in sorted(self.packages.iteritems()):
+ for name, data in sorted(self.packages.items()):
desc, deps, files = data
#
@@ -133,7 +133,7 @@ class MakefileMaker:
self.out( 'SUMMARY_${PN}-modules="All Python modules"' )
line = 'RDEPENDS_${PN}-modules="'
- for name, data in sorted(self.packages.iteritems()):
+ for name, data in sorted(self.packages.items()):
if name not in ['${PN}-dev', '${PN}-distutils-staticdev']:
line += "%s " % name
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index ff97dfc..506d95b 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -155,7 +155,7 @@ def check_workspace_recipe(workspace, pn, checksrc=True, bbclassextend=False):
workspacepn = pn
- for recipe, value in workspace.iteritems():
+ for recipe, value in workspace.items():
if recipe == pn:
break
if bbclassextend:
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index d6186b9..08153c6 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1091,11 +1091,11 @@ def _update_recipe_srcrev(args, srctree, rd, config_data):
else:
files_dir = os.path.join(os.path.dirname(recipefile),
rd.getVar('BPN', True))
- for basepath, path in upd_f.iteritems():
+ for basepath, path in upd_f.items():
logger.info('Updating file %s' % basepath)
_move_file(os.path.join(local_files_dir, basepath), path)
update_srcuri= True
- for basepath, path in new_f.iteritems():
+ for basepath, path in new_f.items():
logger.info('Adding new file %s' % basepath)
_move_file(os.path.join(local_files_dir, basepath),
os.path.join(files_dir, basepath))
@@ -1173,11 +1173,11 @@ def _update_recipe_patch(args, config, workspace, srctree, rd, config_data):
logger.info('No patches or local source files needed updating')
else:
# Update existing files
- for basepath, path in upd_f.iteritems():
+ for basepath, path in upd_f.items():
logger.info('Updating file %s' % basepath)
_move_file(os.path.join(local_files_dir, basepath), path)
updatefiles = True
- for basepath, path in upd_p.iteritems():
+ for basepath, path in upd_p.items():
patchfn = os.path.join(patches_dir, basepath)
if changed_revs is not None:
# Avoid updating patches that have not actually changed
@@ -1192,13 +1192,13 @@ def _update_recipe_patch(args, config, workspace, srctree, rd, config_data):
# Add any new files
files_dir = os.path.join(os.path.dirname(recipefile),
rd.getVar('BPN', True))
- for basepath, path in new_f.iteritems():
+ for basepath, path in new_f.items():
logger.info('Adding new file %s' % basepath)
_move_file(os.path.join(local_files_dir, basepath),
os.path.join(files_dir, basepath))
srcuri.append('file://%s' % basepath)
updaterecipe = True
- for basepath, path in new_p.iteritems():
+ for basepath, path in new_p.items():
logger.info('Adding new patch %s' % basepath)
_move_file(os.path.join(patches_dir, basepath),
os.path.join(files_dir, basepath))
@@ -1285,7 +1285,7 @@ def update_recipe(args, config, basepath, workspace):
def status(args, config, basepath, workspace):
"""Entry point for the devtool 'status' subcommand"""
if workspace:
- for recipe, value in workspace.iteritems():
+ for recipe, value in workspace.items():
recipefile = value['recipefile']
if recipefile:
recipestr = ' (%s)' % recipefile
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 07a9018..66e5f59 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -70,7 +70,7 @@ def _remove_patch_dirs(recipefolder):
def _recipe_contains(rd, var):
rf = rd.getVar('FILE', True)
varfiles = oe.recipeutils.get_var_files(rf, [var], rd)
- for var, fn in varfiles.iteritems():
+ for var, fn in varfiles.items():
if fn and fn.startswith(os.path.dirname(rf) + os.sep):
return True
return False
diff --git a/scripts/lib/recipetool/append.py b/scripts/lib/recipetool/append.py
index 558fd25..35756b08 100644
--- a/scripts/lib/recipetool/append.py
+++ b/scripts/lib/recipetool/append.py
@@ -61,7 +61,7 @@ def find_target_file(targetpath, d, pkglist=None):
'/etc/gshadow': '/etc/gshadow should be managed through the useradd and extrausers classes',
'${sysconfdir}/hostname': '${sysconfdir}/hostname contents should be set by setting hostname_pn-base-files = "value" in configuration',}
- for pthspec, message in invalidtargets.iteritems():
+ for pthspec, message in invalidtargets.items():
if fnmatch.fnmatchcase(targetpath, d.expand(pthspec)):
raise InvalidTargetFileError(d.expand(message))
@@ -152,7 +152,7 @@ def determine_file_source(targetpath, rd):
# Check patches
srcpatches = []
patchedfiles = oe.recipeutils.get_recipe_patched_files(rd)
- for patch, filelist in patchedfiles.iteritems():
+ for patch, filelist in patchedfiles.items():
for fileitem in filelist:
if fileitem[0] == srcpath:
srcpatches.append((patch, fileitem[1]))
@@ -270,7 +270,7 @@ def appendfile(args):
postinst_pns = []
selectpn = None
- for targetpath, pnlist in recipes.iteritems():
+ for targetpath, pnlist in recipes.items():
for pn in pnlist:
if pn.startswith('?'):
alternative_pns.append(pn[1:])
@@ -351,7 +351,7 @@ def appendsrc(args, files, rd, extralines=None):
copyfiles = {}
extralines = extralines or []
- for newfile, srcfile in files.iteritems():
+ for newfile, srcfile in files.items():
src_destdir = os.path.dirname(srcfile)
if not args.use_workdir:
if rd.getVar('S', True) == rd.getVar('STAGING_KERNEL_DIR', True):
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 4a59363..5a37d18 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -61,8 +61,8 @@ class RecipeHandler(object):
libpaths = list(set([base_libdir, libdir]))
libname_re = re.compile('^lib(.+)\.so.*$')
pkglibmap = {}
- for lib, item in shlib_providers.iteritems():
- for path, pkg in item.iteritems():
+ for lib, item in shlib_providers.items():
+ for path, pkg in item.items():
if path in libpaths:
res = libname_re.match(lib)
if res:
@@ -74,7 +74,7 @@ class RecipeHandler(object):
# Now turn it into a library->recipe mapping
pkgdata_dir = d.getVar('PKGDATA_DIR', True)
- for libname, pkg in pkglibmap.iteritems():
+ for libname, pkg in pkglibmap.items():
try:
with open(os.path.join(pkgdata_dir, 'runtime', pkg)) as f:
for line in f:
@@ -663,7 +663,7 @@ def create_recipe(args):
else:
extraoutdir = os.path.join(os.path.dirname(outfile), pn)
bb.utils.mkdirhier(extraoutdir)
- for destfn, extrafile in extrafiles.iteritems():
+ for destfn, extrafile in extrafiles.items():
shutil.move(extrafile, os.path.join(extraoutdir, destfn))
lines = lines_before
@@ -901,7 +901,7 @@ def split_pkg_licenses(licvalues, packages, outlines, fallback_licenses=None, pn
"""
pkglicenses = {pn: []}
for license, licpath, _ in licvalues:
- for pkgname, pkgpath in packages.iteritems():
+ for pkgname, pkgpath in packages.items():
if licpath.startswith(pkgpath + '/'):
if pkgname in pkglicenses:
pkglicenses[pkgname].append(license)
@@ -928,7 +928,7 @@ def read_pkgconfig_provides(d):
for line in f:
pkgmap[os.path.basename(line.rstrip())] = os.path.splitext(os.path.basename(fn))[0]
recipemap = {}
- for pc, pkg in pkgmap.iteritems():
+ for pc, pkg in pkgmap.items():
pkgdatafile = os.path.join(pkgdatadir, 'runtime', pkg)
if os.path.exists(pkgdatafile):
with open(pkgdatafile, 'r') as f:
diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index 37d161e..8fc33e7 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -44,7 +44,7 @@ class CmakeRecipeHandler(RecipeHandler):
classes.append('cmake')
values = CmakeRecipeHandler.extract_cmake_deps(lines_before, srctree, extravalues)
classes.extend(values.pop('inherit', '').split())
- for var, value in values.iteritems():
+ for var, value in values.items():
lines_before.append('%s = "%s"' % (var, value))
lines_after.append('# Specify any options you want to pass to cmake using EXTRA_OECMAKE:')
lines_after.append('EXTRA_OECMAKE = ""')
@@ -159,7 +159,7 @@ class CmakeRecipeHandler(RecipeHandler):
def find_cmake_package(pkg):
RecipeHandler.load_devel_filemap(tinfoil.config_data)
- for fn, pn in RecipeHandler.recipecmakefilemap.iteritems():
+ for fn, pn in RecipeHandler.recipecmakefilemap.items():
splitname = fn.split('/')
if len(splitname) > 1:
if splitname[0].lower().startswith(pkg.lower()):
@@ -348,7 +348,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
autoconf = True
values = AutotoolsRecipeHandler.extract_autotools_deps(lines_before, srctree, extravalues)
classes.extend(values.pop('inherit', '').split())
- for var, value in values.iteritems():
+ for var, value in values.items():
lines_before.append('%s = "%s"' % (var, value))
else:
conffile = RecipeHandler.checkfiles(srctree, ['configure'])
@@ -446,7 +446,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
defines = {}
def subst_defines(value):
newvalue = value
- for define, defval in defines.iteritems():
+ for define, defval in defines.items():
newvalue = newvalue.replace(define, defval)
if newvalue != value:
return subst_defines(newvalue)
@@ -753,7 +753,7 @@ class MakefileRecipeHandler(RecipeHandler):
if scanfile and os.path.exists(scanfile):
values = AutotoolsRecipeHandler.extract_autotools_deps(lines_before, srctree, acfile=scanfile)
classes.extend(values.pop('inherit', '').split())
- for var, value in values.iteritems():
+ for var, value in values.items():
if var == 'DEPENDS':
lines_before.append('# NOTE: some of these dependencies may be optional, check the Makefile and/or upstream documentation')
lines_before.append('%s = "%s"' % (var, value))
diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index c382330..cc3b5a4 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -238,7 +238,7 @@ class PythonRecipeHandler(RecipeHandler):
# Map PKG-INFO & setup.py fields to bitbake variables
bbinfo = {}
- for field, values in info.iteritems():
+ for field, values in info.items():
if field in self.excluded_fields:
continue
@@ -294,8 +294,8 @@ class PythonRecipeHandler(RecipeHandler):
lines_after.append('# The upstream names may not correspond exactly to bitbake package names.')
lines_after.append('#')
lines_after.append('# Uncomment this line to enable all the optional features.')
- lines_after.append('#PACKAGECONFIG ?= "{}"'.format(' '.join(k.lower() for k in extras_req.iterkeys())))
- for feature, feature_reqs in extras_req.iteritems():
+ lines_after.append('#PACKAGECONFIG ?= "{}"'.format(' '.join(k.lower() for k in extras_req)))
+ for feature, feature_reqs in extras_req.items():
unmapped_deps.difference_update(feature_reqs)
feature_req_deps = ('python-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
@@ -442,8 +442,8 @@ class PythonRecipeHandler(RecipeHandler):
del info[variable]
elif new_value != value:
info[variable] = new_value
- elif hasattr(value, 'iteritems'):
- for dkey, dvalue in value.iteritems():
+ elif hasattr(value, 'items'):
+ for dkey, dvalue in value.items():
new_list = []
for pos, a_value in enumerate(dvalue):
new_value = replace_value(search, replace, a_value)
@@ -558,7 +558,7 @@ class PythonRecipeHandler(RecipeHandler):
else:
continue
- for fn in files_info.iterkeys():
+ for fn in files_info:
for suffix in suffixes:
if fn.endswith(suffix):
break
@@ -640,7 +640,7 @@ class SetupScriptVisitor(ast.NodeVisitor):
def visit_setup(self, node):
call = LiteralAstTransform().visit(node)
self.keywords = call.keywords
- for k, v in self.keywords.iteritems():
+ for k, v in self.keywords.items():
if has_non_literals(v):
self.non_literals.append(k)
@@ -708,8 +708,8 @@ def has_non_literals(value):
return True
elif isinstance(value, basestring):
return False
- elif hasattr(value, 'itervalues'):
- return any(has_non_literals(v) for v in value.itervalues())
+ elif hasattr(value, 'values'):
+ return any(has_non_literals(v) for v in value.values())
elif hasattr(value, '__iter__'):
return any(has_non_literals(v) for v in value)
diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index ffbcb49..fcc0172 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -128,7 +128,7 @@ class NpmRecipeHandler(RecipeHandler):
license = self._handle_license(data)
if license:
licenses['${PN}'] = license
- for pkgname, pkgitem in npmpackages.iteritems():
+ for pkgname, pkgitem in npmpackages.items():
_, pdata = pkgitem
license = self._handle_license(pdata)
if license:
@@ -136,7 +136,7 @@ class NpmRecipeHandler(RecipeHandler):
# Now write out the package-specific license values
# We need to strip out the json data dicts for this since split_pkg_licenses
# isn't expecting it
- packages = OrderedDict((x,y[0]) for x,y in npmpackages.iteritems())
+ packages = OrderedDict((x,y[0]) for x,y in npmpackages.items())
packages['${PN}'] = ''
pkglicenses = split_pkg_licenses(licvalues, packages, lines_after, licenses)
all_licenses = list(set([item for pkglicense in pkglicenses.values() for item in pkglicense]))
diff --git a/scripts/pythondeps b/scripts/pythondeps
index ff92e74..f1e6452 100755
--- a/scripts/pythondeps
+++ b/scripts/pythondeps
@@ -187,7 +187,7 @@ def get_depends_recursive(directory):
directory = os.path.realpath(directory)
provides = dict((v, k) for k, v in get_provides(directory))
- for filename, provide in provides.iteritems():
+ for filename, provide in provides.items():
if os.path.isdir(filename):
filename = os.path.join(filename, '__init__.py')
ispkg = True
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 12/22] scripts: Replace basestring -> str for python3
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (9 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 11/22] scripts: Fix deprecated dict methods for python3 Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 13/22] scripts: Fix encoding errors " Richard Purdie
` (12 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Ed Bartosh <ed.bartosh@linux.intel.com>
Python 3 doesn't have basestring type as all string
are unicode strings.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/files/ext-sdk-prepare.py | 2 +-
scripts/lib/devtool/__init__.py | 2 +-
scripts/lib/recipetool/create.py | 2 +-
scripts/lib/recipetool/create_buildsys_python.py | 6 +++---
scripts/lib/scriptutils.py | 2 +-
5 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/meta/files/ext-sdk-prepare.py b/meta/files/ext-sdk-prepare.py
index 605e2eb..bf0d033 100644
--- a/meta/files/ext-sdk-prepare.py
+++ b/meta/files/ext-sdk-prepare.py
@@ -8,7 +8,7 @@ import subprocess
def exec_watch(cmd, **options):
"""Run program with stdout shown on sys.stdout"""
- if isinstance(cmd, basestring) and not "shell" in options:
+ if isinstance(cmd, str) and not "shell" in options:
options["shell"] = True
process = subprocess.Popen(
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 506d95b..6f5bb8d 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -59,7 +59,7 @@ def exec_build_env_command(init_path, builddir, cmd, watch=False, **options):
def exec_watch(cmd, **options):
"""Run program with stdout shown on sys.stdout"""
import bb
- if isinstance(cmd, basestring) and not "shell" in options:
+ if isinstance(cmd, str) and not "shell" in options:
options["shell"] = True
process = subprocess.Popen(
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 5a37d18..e3e01e3 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -167,7 +167,7 @@ class RecipeHandler(object):
unmappedpc = []
pcdeps = list(set(pcdeps))
for pcdep in pcdeps:
- if isinstance(pcdep, basestring):
+ if isinstance(pcdep, str):
recipe = recipemap.get(pcdep, None)
if recipe:
deps.append(recipe)
diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index cc3b5a4..5cbea2a 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -245,7 +245,7 @@ class PythonRecipeHandler(RecipeHandler):
if field not in self.bbvar_map:
continue
- if isinstance(values, basestring):
+ if isinstance(values, str):
value = values
else:
value = ' '.join(str(v) for v in values if v)
@@ -436,7 +436,7 @@ class PythonRecipeHandler(RecipeHandler):
return value
value = info[variable]
- if isinstance(value, basestring):
+ if isinstance(value, str):
new_value = replace_value(search, replace, value)
if new_value is None:
del info[variable]
@@ -706,7 +706,7 @@ class LiteralAstTransform(ast.NodeTransformer):
def has_non_literals(value):
if isinstance(value, ast.AST):
return True
- elif isinstance(value, basestring):
+ elif isinstance(value, str):
return False
elif hasattr(value, 'values'):
return any(has_non_literals(v) for v in value.values())
diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index aef19d3..bd082d8 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -103,7 +103,7 @@ def fetch_uri(d, uri, destdir, srcrev=None):
return ret
def run_editor(fn):
- if isinstance(fn, basestring):
+ if isinstance(fn, str):
params = '"%s"' % fn
else:
params = ''
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 13/22] scripts: Fix encoding errors for python3
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (10 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 12/22] scripts: Replace basestring -> str " Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 14/22] scripts: Rename ConfigParser -> configparser " Richard Purdie
` (11 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Ed Bartosh <ed.bartosh@linux.intel.com>
Moved call of decode('utf-8') as close as possible to
call of subprocess API to avoid calling it in a lot of
other places.
Decoded binary data to utf-8 where appropriate to fix devtool
and recipetool tests in python 3 environment.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/lib/oe/patch.py | 5 ++++-
scripts/lib/devtool/__init__.py | 1 +
scripts/lib/recipetool/append.py | 4 ++--
scripts/lib/recipetool/create.py | 4 ++--
scripts/lib/recipetool/create_buildsys_python.py | 2 +-
5 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index a25fd52..4a0d3f7 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -292,7 +292,10 @@ class GitApplyTree(PatchTree):
def decodeAuthor(line):
from email.header import decode_header
authorval = line.split(':', 1)[1].strip().replace('"', '')
- return decode_header(authorval)[0][0]
+ result = decode_header(authorval)[0][0]
+ if hasattr(result, 'decode'):
+ result = result.decode('utf-8')
+ return result
@staticmethod
def interpretPatchHeader(headerlines):
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 6f5bb8d..7005363 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -69,6 +69,7 @@ def exec_watch(cmd, **options):
buf = ''
while True:
out = process.stdout.read(1)
+ out = out.decode('utf-8')
if out:
sys.stdout.write(out)
sys.stdout.flush()
diff --git a/scripts/lib/recipetool/append.py b/scripts/lib/recipetool/append.py
index 35756b08..5d73d30 100644
--- a/scripts/lib/recipetool/append.py
+++ b/scripts/lib/recipetool/append.py
@@ -90,7 +90,7 @@ def find_target_file(targetpath, d, pkglist=None):
if fnmatch.fnmatchcase(fullpth, targetpath):
recipes[targetpath].append(pn)
elif line.startswith('pkg_preinst_') or line.startswith('pkg_postinst_'):
- scriptval = line.split(':', 1)[1].strip().decode('string_escape')
+ scriptval = line.split(':', 1)[1].strip().encode('utf-8').decode('unicode_escape')
if 'update-alternatives --install %s ' % targetpath in scriptval:
recipes[targetpath].append('?%s' % pn)
elif targetpath_re.search(scriptval):
@@ -172,7 +172,7 @@ def get_source_path(cmdelements):
"""Find the source path specified within a command"""
command = cmdelements[0]
if command in ['install', 'cp']:
- helptext = subprocess.check_output('LC_ALL=C %s --help' % command, shell=True)
+ helptext = subprocess.check_output('LC_ALL=C %s --help' % command, shell=True).decode('utf-8')
argopts = ''
argopt_line_re = re.compile('^-([a-zA-Z0-9]), --[a-z-]+=')
for line in helptext.splitlines():
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index e3e01e3..744ff1b 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -851,14 +851,14 @@ def crunch_license(licfile):
continue
# Squash spaces, and replace smart quotes, double quotes
# and backticks with single quotes
- line = oe.utils.squashspaces(line.strip()).decode("utf-8")
+ line = oe.utils.squashspaces(line.strip())
line = line.replace(u"\u2018", "'").replace(u"\u2019", "'").replace(u"\u201c","'").replace(u"\u201d", "'").replace('"', '\'').replace('`', '\'')
if line:
lictext.append(line)
m = hashlib.md5()
try:
- m.update(' '.join(lictext))
+ m.update(' '.join(lictext).encode('utf-8'))
md5val = m.hexdigest()
except UnicodeEncodeError:
md5val = None
diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index 5cbea2a..55cce0e 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -516,7 +516,7 @@ class PythonRecipeHandler(RecipeHandler):
except (OSError, subprocess.CalledProcessError):
pass
else:
- for line in dep_output.splitlines():
+ for line in dep_output.decode('utf-8').splitlines():
line = line.rstrip()
dep, filename = line.split('\t', 1)
if filename.endswith('/setup.py'):
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 14/22] scripts: Rename ConfigParser -> configparser for python3
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (11 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 13/22] scripts: Fix encoding errors " Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 15/22] scripts: Fix urlparse imports " Richard Purdie
` (10 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Ed Bartosh <ed.bartosh@linux.intel.com>
The ConfigParser API was renamed to configparser in python 3.
Renamed ConfigParser -> configparser in scripts/ to make the
code working in python 3.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
scripts/combo-layer | 6 +++---
scripts/contrib/python/generate-manifest-3.5.py | 2 +-
scripts/devtool | 6 +++---
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/scripts/combo-layer b/scripts/combo-layer
index 52367f0..234d9e4 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -26,7 +26,7 @@ import optparse
import logging
import subprocess
import tempfile
-import ConfigParser
+import configparser
import re
import copy
import pipes
@@ -87,7 +87,7 @@ class Configuration(object):
self.commit_msg_template = value
logger.debug("Loading config file %s" % self.conffile)
- self.parser = ConfigParser.ConfigParser()
+ self.parser = configparser.ConfigParser()
with open(self.conffile) as f:
self.parser.readfp(f)
@@ -116,7 +116,7 @@ class Configuration(object):
self.localconffile = lcfile
logger.debug("Loading local config file %s" % self.localconffile)
- self.localparser = ConfigParser.ConfigParser()
+ self.localparser = configparser.ConfigParser()
with open(self.localconffile) as f:
self.localparser.readfp(f)
diff --git a/scripts/contrib/python/generate-manifest-3.5.py b/scripts/contrib/python/generate-manifest-3.5.py
index 41f209b..e04da1d 100755
--- a/scripts/contrib/python/generate-manifest-3.5.py
+++ b/scripts/contrib/python/generate-manifest-3.5.py
@@ -167,7 +167,7 @@ if __name__ == "__main__":
#
m.addPackage( "${PN}-core", "Python interpreter and core modules", "${PN}-lang ${PN}-re ${PN}-reprlib ${PN}-codecs ${PN}-io ${PN}-math",
- "__future__.* _abcoll.* abc.* ast.* copy.* copyreg.* ConfigParser.* " +
+ "__future__.* _abcoll.* abc.* ast.* copy.* copyreg.* configparser.* " +
"genericpath.* getopt.* linecache.* new.* " +
"os.* posixpath.* struct.* " +
"warnings.* site.* stat.* " +
diff --git a/scripts/devtool b/scripts/devtool
index 9ac6e79..4dbb527 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -22,7 +22,7 @@ import os
import argparse
import glob
import re
-import ConfigParser
+import configparser
import subprocess
import logging
@@ -51,12 +51,12 @@ class ConfigHandler(object):
def __init__(self, filename):
self.config_file = filename
- self.config_obj = ConfigParser.SafeConfigParser()
+ self.config_obj = configparser.SafeConfigParser()
def get(self, section, option, default=None):
try:
ret = self.config_obj.get(section, option)
- except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+ except (configparser.NoOptionError, configparser.NoSectionError):
if default != None:
ret = default
else:
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 15/22] scripts: Fix urlparse imports for python3
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (12 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 14/22] scripts: Rename ConfigParser -> configparser " Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 16/22] scripts: use python3 in shebang Richard Purdie
` (9 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Ed Bartosh <ed.bartosh@linux.intel.com>
Used urllib.parse instead of urlparse to make code
working in python 3.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
scripts/lib/recipetool/create.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 744ff1b..1297428 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -24,7 +24,7 @@ import re
import json
import logging
import scriptutils
-import urlparse
+from urllib.parse import urlparse, urldefrag, urlsplit
import hashlib
logger = logging.getLogger('recipetool')
@@ -283,7 +283,7 @@ def determine_from_url(srcuri):
"""Determine name and version from a URL"""
pn = None
pv = None
- parseres = urlparse.urlparse(srcuri.lower().split(';', 1)[0])
+ parseres = urlparse(srcuri.lower().split(';', 1)[0])
if parseres.path:
if 'github.com' in parseres.netloc:
res = re.search(r'.*/(.*?)/archive/(.*)-final\.(tar|zip)', parseres.path)
@@ -355,7 +355,7 @@ def create_recipe(args):
srcrev = '${AUTOREV}'
if '://' in args.source:
# Fetch a URL
- fetchuri = reformat_git_uri(urlparse.urldefrag(args.source)[0])
+ fetchuri = reformat_git_uri(urldefrag(args.source)[0])
if args.binary:
# Assume the archive contains the directory structure verbatim
# so we need to extract to a subdirectory
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 16/22] scripts: use python3 in shebang
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (13 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 15/22] scripts: Fix urlparse imports " Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 17/22] classes/lib: Complete transition to python3 Richard Purdie
` (8 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Ed Bartosh <ed.bartosh@linux.intel.com>
As most of oe-test cases work for devtool and recipetool
it makes sense to switch both tools to python 3 by
explicitly referring to python3 in their shebangs.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
scripts/devtool | 2 +-
scripts/oe-pkgdata-util | 2 +-
scripts/recipetool | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/scripts/devtool b/scripts/devtool
index 4dbb527..ff368dc 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
# OpenEmbedded Development tool
#
diff --git a/scripts/oe-pkgdata-util b/scripts/oe-pkgdata-util
index a04e44d..b39d9b5 100755
--- a/scripts/oe-pkgdata-util
+++ b/scripts/oe-pkgdata-util
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
# OpenEmbedded pkgdata utility
#
diff --git a/scripts/recipetool b/scripts/recipetool
index 6c66487..0e8bffb 100755
--- a/scripts/recipetool
+++ b/scripts/recipetool
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
# Recipe creation tool
#
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 17/22] classes/lib: Complete transition to python3
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (14 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 16/22] scripts: use python3 in shebang Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 18/22] selftest/devtool: Compare sets instead of arrays on AssertEqual Richard Purdie
` (7 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
This patch contains all the other misc pieces of the transition to
python3 which didn't make sense to be broken into individual patches.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/classes/base.bbclass | 2 +-
meta/classes/buildhistory.bbclass | 2 +-
meta/classes/populate_sdk_ext.bbclass | 4 ++--
meta/classes/report-error.bbclass | 2 +-
meta/lib/oe/classutils.py | 15 ++++++++-------
meta/lib/oe/distro_check.py | 4 ++--
meta/lib/oe/maketype.py | 9 ++++++---
meta/lib/oe/manifest.py | 3 +--
meta/lib/oe/package.py | 2 +-
meta/lib/oe/package_manager.py | 11 +++--------
meta/lib/oe/qa.py | 28 ++++++++++++++--------------
meta/lib/oe/recipeutils.py | 4 ++--
meta/lib/oe/rootfs.py | 3 +--
meta/lib/oe/sdk.py | 4 +---
meta/lib/oe/terminal.py | 6 ++----
meta/lib/oe/tests/test_path.py | 2 +-
meta/lib/oe/types.py | 4 ++--
meta/lib/oe/utils.py | 6 +++---
meta/lib/oeqa/controllers/masterimage.py | 4 +---
meta/lib/oeqa/oetest.py | 10 +++++++++-
meta/lib/oeqa/selftest/_toaster.py | 4 ++--
meta/lib/oeqa/selftest/recipetool.py | 4 ++--
meta/lib/oeqa/selftest/sstatetests.py | 6 +++---
meta/lib/oeqa/targetcontrol.py | 4 +---
meta/lib/oeqa/utils/commands.py | 4 ++--
meta/lib/oeqa/utils/decorators.py | 17 ++++++++++++++++-
meta/lib/oeqa/utils/dump.py | 2 +-
meta/lib/oeqa/utils/httpserver.py | 6 +++---
meta/lib/oeqa/utils/logparser.py | 2 +-
meta/lib/oeqa/utils/qemurunner.py | 22 ++++++++++++----------
meta/lib/oeqa/utils/qemutinyrunner.py | 2 +-
meta/lib/oeqa/utils/targetbuild.py | 4 +---
meta/lib/oeqa/utils/testexport.py | 2 +-
scripts/oe-selftest | 10 ++++++----
34 files changed, 114 insertions(+), 100 deletions(-)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 1ba1222..1bbe812 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -10,7 +10,7 @@ inherit utility-tasks
inherit metadata_scm
inherit logging
-OE_IMPORTS += "os sys time oe.path oe.utils oe.data oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath"
+OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath"
OE_IMPORTS[type] = "list"
def oe_import(d):
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 1ccd9ee..6995d06 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -145,7 +145,7 @@ python buildhistory_emit_pkghistory() {
elif name == "RCONFLICTS":
pkginfo.rconflicts = value
elif name == "PKGSIZE":
- pkginfo.size = long(value)
+ pkginfo.size = int(value)
elif name == "FILES":
pkginfo.files = value
elif name == "FILELIST":
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index a9c6fe5..b9d9543 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -129,8 +129,8 @@ python copy_buildsystem () {
d.setVar('scriptrelpath', scriptrelpath)
# Write out config file for devtool
- import ConfigParser
- config = ConfigParser.SafeConfigParser()
+ import configparser
+ config = configparser.SafeConfigParser()
config.add_section('General')
config.set('General', 'bitbake_subdir', conf_bbpath)
config.set('General', 'init_path', conf_initpath)
diff --git a/meta/classes/report-error.bbclass b/meta/classes/report-error.bbclass
index 20d2bef..5cbf8f0 100644
--- a/meta/classes/report-error.bbclass
+++ b/meta/classes/report-error.bbclass
@@ -43,7 +43,7 @@ python errorreport_handler () {
data['target_sys'] = e.data.getVar("TARGET_SYS", True)
data['failures'] = []
data['component'] = e.getPkgs()[0]
- data['branch_commit'] = base_detect_branch(e.data) + ": " + base_detect_revision(e.data)
+ data['branch_commit'] = str(base_detect_branch(e.data)) + ": " + str(base_detect_revision(e.data))
lock = bb.utils.lockfile(datafile + '.lock')
errorreport_savedata(e, data, "error-report.txt")
bb.utils.unlockfile(lock)
diff --git a/meta/lib/oe/classutils.py b/meta/lib/oe/classutils.py
index 98bb059..e7856c8 100644
--- a/meta/lib/oe/classutils.py
+++ b/meta/lib/oe/classutils.py
@@ -1,4 +1,11 @@
-class ClassRegistry(type):
+
+class ClassRegistryMeta(type):
+ """Give each ClassRegistry their own registry"""
+ def __init__(cls, name, bases, attrs):
+ cls.registry = {}
+ type.__init__(cls, name, bases, attrs)
+
+class ClassRegistry(type, metaclass=ClassRegistryMeta):
"""Maintain a registry of classes, indexed by name.
Note that this implementation requires that the names be unique, as it uses
@@ -12,12 +19,6 @@ Subclasses of ClassRegistry may define an 'implemented' property to exert
control over whether the class will be added to the registry (e.g. to keep
abstract base classes out of the registry)."""
priority = 0
- class __metaclass__(type):
- """Give each ClassRegistry their own registry"""
- def __init__(cls, name, bases, attrs):
- cls.registry = {}
- type.__init__(cls, name, bases, attrs)
-
def __init__(cls, name, bases, attrs):
super(ClassRegistry, cls).__init__(name, bases, attrs)
try:
diff --git a/meta/lib/oe/distro_check.py b/meta/lib/oe/distro_check.py
index ba1bba6..746e242 100644
--- a/meta/lib/oe/distro_check.py
+++ b/meta/lib/oe/distro_check.py
@@ -1,8 +1,8 @@
from contextlib import contextmanager
@contextmanager
def create_socket(url, d):
- import urllib
- socket = urllib.urlopen(url, proxies=get_proxies(d))
+ import urllib.request, urllib.parse, urllib.error
+ socket = urllib.request.urlopen(url, proxies=get_proxies(d))
try:
yield socket
finally:
diff --git a/meta/lib/oe/maketype.py b/meta/lib/oe/maketype.py
index 139f333..f88981d 100644
--- a/meta/lib/oe/maketype.py
+++ b/meta/lib/oe/maketype.py
@@ -6,7 +6,8 @@ the arguments of the type's factory for details.
"""
import inspect
-import types
+import oe.types as types
+import collections
available_types = {}
@@ -53,7 +54,9 @@ def get_callable_args(obj):
if type(obj) is type:
obj = obj.__init__
- args, varargs, keywords, defaults = inspect.getargspec(obj)
+ sig = inspect.signature(obj)
+ args = list(sig.parameters.keys())
+ defaults = list(s for s in sig.parameters.keys() if sig.parameters[s].default != inspect.Parameter.empty)
flaglist = []
if args:
if len(args) > 1 and args[0] == 'self':
@@ -93,7 +96,7 @@ for name in dir(types):
continue
obj = getattr(types, name)
- if not callable(obj):
+ if not isinstance(obj, collections.Callable):
continue
register(name, obj)
diff --git a/meta/lib/oe/manifest.py b/meta/lib/oe/manifest.py
index ec2ef50..95f8eb2 100644
--- a/meta/lib/oe/manifest.py
+++ b/meta/lib/oe/manifest.py
@@ -4,11 +4,10 @@ import re
import bb
-class Manifest(object):
+class Manifest(object, metaclass=ABCMeta):
"""
This is an abstract class. Do not instantiate this directly.
"""
- __metaclass__ = ABCMeta
PKG_TYPE_MUST_INSTALL = "mip"
PKG_TYPE_MULTILIB = "mlp"
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 5bb15bb..faa0ab2 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -8,7 +8,7 @@ def runstrip(arg):
# 8 - shared library
# 16 - kernel module
- import commands, stat, subprocess
+ import stat, subprocess
(file, elftype, strip) = arg
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 54e6970..71e5b50 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -89,9 +89,7 @@ def opkg_query(cmd_output):
return output
-class Indexer(object):
- __metaclass__ = ABCMeta
-
+class Indexer(object, metaclass=ABCMeta):
def __init__(self, d, deploy_dir):
self.d = d
self.deploy_dir = deploy_dir
@@ -342,9 +340,7 @@ class DpkgIndexer(Indexer):
-class PkgsList(object):
- __metaclass__ = ABCMeta
-
+class PkgsList(object, metaclass=ABCMeta):
def __init__(self, d, rootfs_dir):
self.d = d
self.rootfs_dir = rootfs_dir
@@ -512,11 +508,10 @@ class DpkgPkgsList(PkgsList):
return opkg_query(cmd_output)
-class PackageManager(object):
+class PackageManager(object, metaclass=ABCMeta):
"""
This is an abstract class. Do not instantiate this directly.
"""
- __metaclass__ = ABCMeta
def __init__(self, d):
self.d = d
diff --git a/meta/lib/oe/qa.py b/meta/lib/oe/qa.py
index 2c30141..75e7df8 100644
--- a/meta/lib/oe/qa.py
+++ b/meta/lib/oe/qa.py
@@ -50,41 +50,41 @@ class ELFFile:
if len(self.data) < ELFFile.EI_NIDENT + 4:
raise NotELFFileError("%s is not an ELF" % self.name)
- self.my_assert(self.data[0], chr(0x7f) )
- self.my_assert(self.data[1], 'E')
- self.my_assert(self.data[2], 'L')
- self.my_assert(self.data[3], 'F')
+ self.my_assert(self.data[0], 0x7f)
+ self.my_assert(self.data[1], ord('E'))
+ self.my_assert(self.data[2], ord('L'))
+ self.my_assert(self.data[3], ord('F'))
if self.bits == 0:
- if self.data[ELFFile.EI_CLASS] == chr(ELFFile.ELFCLASS32):
+ if self.data[ELFFile.EI_CLASS] == ELFFile.ELFCLASS32:
self.bits = 32
- elif self.data[ELFFile.EI_CLASS] == chr(ELFFile.ELFCLASS64):
+ elif self.data[ELFFile.EI_CLASS] == ELFFile.ELFCLASS64:
self.bits = 64
else:
# Not 32-bit or 64.. lets assert
raise NotELFFileError("ELF but not 32 or 64 bit.")
elif self.bits == 32:
- self.my_assert(self.data[ELFFile.EI_CLASS], chr(ELFFile.ELFCLASS32))
+ self.my_assert(self.data[ELFFile.EI_CLASS], ELFFile.ELFCLASS32)
elif self.bits == 64:
- self.my_assert(self.data[ELFFile.EI_CLASS], chr(ELFFile.ELFCLASS64))
+ self.my_assert(self.data[ELFFile.EI_CLASS], ELFFile.ELFCLASS64)
else:
raise NotELFFileError("Must specify unknown, 32 or 64 bit size.")
- self.my_assert(self.data[ELFFile.EI_VERSION], chr(ELFFile.EV_CURRENT) )
+ self.my_assert(self.data[ELFFile.EI_VERSION], ELFFile.EV_CURRENT)
self.sex = self.data[ELFFile.EI_DATA]
- if self.sex == chr(ELFFile.ELFDATANONE):
+ if self.sex == ELFFile.ELFDATANONE:
raise NotELFFileError("self.sex == ELFDATANONE")
- elif self.sex == chr(ELFFile.ELFDATA2LSB):
+ elif self.sex == ELFFile.ELFDATA2LSB:
self.sex = "<"
- elif self.sex == chr(ELFFile.ELFDATA2MSB):
+ elif self.sex == ELFFile.ELFDATA2MSB:
self.sex = ">"
else:
raise NotELFFileError("Unknown self.sex")
def osAbi(self):
- return ord(self.data[ELFFile.EI_OSABI])
+ return self.data[ELFFile.EI_OSABI]
def abiVersion(self):
- return ord(self.data[ELFFile.EI_ABIVERSION])
+ return self.data[ELFFile.EI_ABIVERSION]
def abiSize(self):
return self.bits
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 1b85387..e3c4b8a 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -11,7 +11,7 @@ import os.path
import tempfile
import textwrap
import difflib
-import utils
+from . import utils
import shutil
import re
import fnmatch
@@ -662,7 +662,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
if removevar in removevalues:
remove = removevalues[removevar]
- if isinstance(remove, basestring):
+ if isinstance(remove, str):
if remove in splitval:
splitval.remove(remove)
changed = True
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index d934858..1fc35bd 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -10,11 +10,10 @@ import subprocess
import re
-class Rootfs(object):
+class Rootfs(object, metaclass=ABCMeta):
"""
This is an abstract class. Do not instantiate this directly.
"""
- __metaclass__ = ABCMeta
def __init__(self, d):
self.d = d
diff --git a/meta/lib/oe/sdk.py b/meta/lib/oe/sdk.py
index 4786cc5..c74525f 100644
--- a/meta/lib/oe/sdk.py
+++ b/meta/lib/oe/sdk.py
@@ -8,9 +8,7 @@ import glob
import traceback
-class Sdk(object):
- __metaclass__ = ABCMeta
-
+class Sdk(object, metaclass=ABCMeta):
def __init__(self, d, manifest_dir):
self.d = d
self.sdk_output = self.d.getVar('SDK_OUTPUT', True)
diff --git a/meta/lib/oe/terminal.py b/meta/lib/oe/terminal.py
index 634daa9..dc25d14 100644
--- a/meta/lib/oe/terminal.py
+++ b/meta/lib/oe/terminal.py
@@ -25,9 +25,7 @@ class Registry(oe.classutils.ClassRegistry):
return bool(cls.command)
-class Terminal(Popen):
- __metaclass__ = Registry
-
+class Terminal(Popen, metaclass=Registry):
def __init__(self, sh_cmd, title=None, env=None, d=None):
fmt_sh_cmd = self.format_command(sh_cmd, title)
try:
@@ -41,7 +39,7 @@ class Terminal(Popen):
def format_command(self, sh_cmd, title):
fmt = {'title': title or 'Terminal', 'command': sh_cmd}
- if isinstance(self.command, basestring):
+ if isinstance(self.command, str):
return shlex.split(self.command.format(**fmt))
else:
return [element.format(**fmt) for element in self.command]
diff --git a/meta/lib/oe/tests/test_path.py b/meta/lib/oe/tests/test_path.py
index 5fa2448..44d0681 100644
--- a/meta/lib/oe/tests/test_path.py
+++ b/meta/lib/oe/tests/test_path.py
@@ -85,5 +85,5 @@ class TestRealPath(unittest.TestCase):
def test_loop(self):
for e in self.EXCEPTIONS:
- self.assertRaisesRegexp(OSError, r'\[Errno %u\]' % e[1],
+ self.assertRaisesRegex(OSError, r'\[Errno %u\]' % e[1],
self.__realpath, e[0], False, False)
diff --git a/meta/lib/oe/types.py b/meta/lib/oe/types.py
index 7f47c17..4ae58ac 100644
--- a/meta/lib/oe/types.py
+++ b/meta/lib/oe/types.py
@@ -33,7 +33,7 @@ def choice(value, choices):
Acts as a multiple choice for the user. To use this, set the variable
type flag to 'choice', and set the 'choices' flag to a space separated
list of valid values."""
- if not isinstance(value, basestring):
+ if not isinstance(value, str):
raise TypeError("choice accepts a string, not '%s'" % type(value))
value = value.lower()
@@ -106,7 +106,7 @@ def boolean(value):
Valid values for false: 'no', 'n', 'false', 'f', '0'
"""
- if not isinstance(value, basestring):
+ if not isinstance(value, str):
raise TypeError("boolean accepts a string, not '%s'" % type(value))
value = value.lower()
diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
index 1bbdbb4..cecddc6 100644
--- a/meta/lib/oe/utils.py
+++ b/meta/lib/oe/utils.py
@@ -46,7 +46,7 @@ def both_contain(variable1, variable2, checkvalue, d):
val2 = d.getVar(variable2, True)
val1 = set(val1.split())
val2 = set(val2.split())
- if isinstance(checkvalue, basestring):
+ if isinstance(checkvalue, str):
checkvalue = set(checkvalue.split())
else:
checkvalue = set(checkvalue)
@@ -235,7 +235,7 @@ def format_pkg_list(pkg_dict, ret_format=None):
# so implement a version here
#
-from Queue import Queue
+from queue import Queue
from threading import Thread
class ThreadedWorker(Thread):
@@ -249,7 +249,7 @@ class ThreadedWorker(Thread):
self.worker_end = worker_end
def run(self):
- from Queue import Empty
+ from queue import Empty
if self.worker_init is not None:
self.worker_init(self)
diff --git a/meta/lib/oeqa/controllers/masterimage.py b/meta/lib/oeqa/controllers/masterimage.py
index 522f9eb..4cb7553 100644
--- a/meta/lib/oeqa/controllers/masterimage.py
+++ b/meta/lib/oeqa/controllers/masterimage.py
@@ -24,9 +24,7 @@ from oeqa.utils import CommandError
from abc import ABCMeta, abstractmethod
-class MasterImageHardwareTarget(oeqa.targetcontrol.BaseTarget):
-
- __metaclass__ = ABCMeta
+class MasterImageHardwareTarget(oeqa.targetcontrol.BaseTarget, metaclass=ABCMeta):
supported_image_fstypes = ['tar.gz', 'tar.bz2']
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 4211ffc..b4cf34b 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -13,6 +13,7 @@ import inspect
import subprocess
import signal
import shutil
+import functools
try:
import bb
except ImportError:
@@ -340,7 +341,14 @@ class TestContext(object):
for index, suite in enumerate(suites):
set_suite_depth(suite)
suite.index = index
- suites.sort(cmp=lambda a,b: cmp((a.depth, a.index), (b.depth, b.index)))
+
+ def cmp(a, b):
+ return (a > b) - (a < b)
+
+ def cmpfunc(a, b):
+ return cmp((a.depth, a.index), (b.depth, b.index))
+
+ suites.sort(key=functools.cmp_to_key(cmpfunc))
self.suite = testloader.suiteClass(suites)
diff --git a/meta/lib/oeqa/selftest/_toaster.py b/meta/lib/oeqa/selftest/_toaster.py
index c424659..15ea9df 100644
--- a/meta/lib/oeqa/selftest/_toaster.py
+++ b/meta/lib/oeqa/selftest/_toaster.py
@@ -2,7 +2,7 @@ import unittest
import os
import sys
import shlex, subprocess
-import urllib, commands, time, getpass, re, json, shlex
+import urllib.request, urllib.parse, urllib.error, subprocess, time, getpass, re, json, shlex
import oeqa.utils.ftools as ftools
from oeqa.selftest.base import oeSelfTest
@@ -290,7 +290,7 @@ class Toaster_DB_Tests(ToasterSetup):
layers = Layer.objects.values('id', 'layer_index_url')
cnt_err = []
for layer in layers:
- resp = urllib.urlopen(layer['layer_index_url'])
+ resp = urllib.request.urlopen(layer['layer_index_url'])
if (resp.getcode() != 200):
cnt_err.append(layer['id'])
self.assertEqual(len(cnt_err), 0, msg = 'Errors for layer id: %s' % cnt_err)
diff --git a/meta/lib/oeqa/selftest/recipetool.py b/meta/lib/oeqa/selftest/recipetool.py
index e72911b..a93d18e 100644
--- a/meta/lib/oeqa/selftest/recipetool.py
+++ b/meta/lib/oeqa/selftest/recipetool.py
@@ -1,7 +1,7 @@
import os
import logging
import tempfile
-import urlparse
+import urllib.parse
from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer
from oeqa.utils.decorators import testcase
@@ -471,7 +471,7 @@ class RecipetoolAppendsrcBase(RecipetoolBase):
'''Return the first file:// in SRC_URI for the specified recipe.'''
src_uri = get_bb_var('SRC_URI', recipe).split()
for uri in src_uri:
- p = urlparse.urlparse(uri)
+ p = urllib.parse.urlparse(uri)
if p.scheme == 'file':
return p.netloc + p.path
diff --git a/meta/lib/oeqa/selftest/sstatetests.py b/meta/lib/oeqa/selftest/sstatetests.py
index cc64c6c..07212ac 100644
--- a/meta/lib/oeqa/selftest/sstatetests.py
+++ b/meta/lib/oeqa/selftest/sstatetests.py
@@ -264,7 +264,7 @@ SDKMACHINE = "i686"
files2 = get_files(topdir + "/tmp-sstatesamehash2/stamps/")
files2 = [x.replace("tmp-sstatesamehash2", "tmp-sstatesamehash").replace("i686-linux", "x86_64-linux").replace("i686" + targetvendor + "-linux", "x86_64" + targetvendor + "-linux", ) for x in files2]
self.maxDiff = None
- self.assertItemsEqual(files1, files2)
+ self.assertCountEqual(files1, files2)
@testcase(1271)
@@ -298,7 +298,7 @@ NATIVELSBSTRING = \"DistroB\"
files2 = get_files(topdir + "/tmp-sstatesamehash2/stamps/")
files2 = [x.replace("tmp-sstatesamehash2", "tmp-sstatesamehash") for x in files2]
self.maxDiff = None
- self.assertItemsEqual(files1, files2)
+ self.assertCountEqual(files1, files2)
@testcase(1368)
def test_sstate_allarch_samesigs(self):
@@ -393,7 +393,7 @@ DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
files2 = get_files(topdir + "/tmp-sstatesamehash2/stamps")
files2 = [x.replace("tmp-sstatesamehash2", "tmp-sstatesamehash") for x in files2]
self.maxDiff = None
- self.assertItemsEqual(files1, files2)
+ self.assertCountEqual(files1, files2)
def test_sstate_noop_samesigs(self):
diff --git a/meta/lib/oeqa/targetcontrol.py b/meta/lib/oeqa/targetcontrol.py
index 5422a61..1c57efa 100644
--- a/meta/lib/oeqa/targetcontrol.py
+++ b/meta/lib/oeqa/targetcontrol.py
@@ -43,9 +43,7 @@ def get_target_controller(d):
return controller(d)
-class BaseTarget(object):
-
- __metaclass__ = ABCMeta
+class BaseTarget(object, metaclass=ABCMeta):
supported_image_fstypes = []
diff --git a/meta/lib/oeqa/utils/commands.py b/meta/lib/oeqa/utils/commands.py
index 9a7c1d1..18fe39e 100644
--- a/meta/lib/oeqa/utils/commands.py
+++ b/meta/lib/oeqa/utils/commands.py
@@ -41,7 +41,7 @@ class Command(object):
self.data = data
self.options = dict(self.defaultopts)
- if isinstance(self.cmd, basestring):
+ if isinstance(self.cmd, str):
self.options["shell"] = True
if self.data:
self.options['stdin'] = subprocess.PIPE
@@ -123,7 +123,7 @@ def bitbake(command, ignore_status=False, timeout=None, postconfig=None, **optio
else:
extra_args = ""
- if isinstance(command, basestring):
+ if isinstance(command, str):
cmd = "bitbake " + extra_args + " " + command
else:
cmd = [ "bitbake" ] + [a for a in (command + extra_args.split(" ")) if a not in [""]]
diff --git a/meta/lib/oeqa/utils/decorators.py b/meta/lib/oeqa/utils/decorators.py
index 6fb09db..0b23565 100644
--- a/meta/lib/oeqa/utils/decorators.py
+++ b/meta/lib/oeqa/utils/decorators.py
@@ -115,6 +115,8 @@ class NoParsingFilter(logging.Filter):
def filter(self, record):
return record.levelno == 100
+import inspect
+
def LogResults(original_class):
orig_method = original_class.run
@@ -124,6 +126,19 @@ def LogResults(original_class):
logfile = os.path.join(os.getcwd(),'results-'+caller+'.'+timestamp+'.log')
linkfile = os.path.join(os.getcwd(),'results-'+caller+'.log')
+ def get_class_that_defined_method(meth):
+ if inspect.ismethod(meth):
+ for cls in inspect.getmro(meth.__self__.__class__):
+ if cls.__dict__.get(meth.__name__) is meth:
+ return cls
+ meth = meth.__func__ # fallback to __qualname__ parsing
+ if inspect.isfunction(meth):
+ cls = getattr(inspect.getmodule(meth),
+ meth.__qualname__.split('.<locals>', 1)[0].rsplit('.', 1)[0])
+ if isinstance(cls, type):
+ return cls
+ return None
+
#rewrite the run method of unittest.TestCase to add testcase logging
def run(self, result, *args, **kws):
orig_method(self, result, *args, **kws)
@@ -135,7 +150,7 @@ def LogResults(original_class):
except AttributeError:
test_case = self._testMethodName
- class_name = str(testMethod.im_class).split("'")[1]
+ class_name = str(get_class_that_defined_method(testMethod)).split("'")[1]
#create custom logging level for filtering.
custom_log_level = 100
diff --git a/meta/lib/oeqa/utils/dump.py b/meta/lib/oeqa/utils/dump.py
index 63a591d..71422a9 100644
--- a/meta/lib/oeqa/utils/dump.py
+++ b/meta/lib/oeqa/utils/dump.py
@@ -3,7 +3,7 @@ import sys
import errno
import datetime
import itertools
-from commands import runCmd
+from .commands import runCmd
def get_host_dumper(d):
cmds = d.getVar("testimage_dump_host", True)
diff --git a/meta/lib/oeqa/utils/httpserver.py b/meta/lib/oeqa/utils/httpserver.py
index 76518d8..bd76f36 100644
--- a/meta/lib/oeqa/utils/httpserver.py
+++ b/meta/lib/oeqa/utils/httpserver.py
@@ -1,8 +1,8 @@
-import SimpleHTTPServer
+import http.server
import multiprocessing
import os
-class HTTPServer(SimpleHTTPServer.BaseHTTPServer.HTTPServer):
+class HTTPServer(http.server.HTTPServer):
def server_start(self, root_dir):
import signal
@@ -10,7 +10,7 @@ class HTTPServer(SimpleHTTPServer.BaseHTTPServer.HTTPServer):
os.chdir(root_dir)
self.serve_forever()
-class HTTPRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
+class HTTPRequestHandler(http.server.SimpleHTTPRequestHandler):
def log_message(self, format_str, *args):
pass
diff --git a/meta/lib/oeqa/utils/logparser.py b/meta/lib/oeqa/utils/logparser.py
index 87b5035..b377dcd 100644
--- a/meta/lib/oeqa/utils/logparser.py
+++ b/meta/lib/oeqa/utils/logparser.py
@@ -3,7 +3,7 @@
import sys
import os
import re
-import ftools
+from . import ftools
# A parser that can be used to identify weather a line is a test result or a section statement.
diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 3d60433..e408fbb 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -23,8 +23,8 @@ logger = logging.getLogger("BitBake.QemuRunner")
# Get Unicode non printable control chars
control_range = list(range(0,32))+list(range(127,160))
-control_chars = [unichr(x) for x in control_range
- if unichr(x) not in string.printable]
+control_chars = [chr(x) for x in control_range
+ if chr(x) not in string.printable]
re_control_char = re.compile('[%s]' % re.escape("".join(control_chars)))
class QemuRunner:
@@ -220,6 +220,7 @@ class QemuRunner:
stopread = False
qemusock = None
bootlog = ''
+ data = b''
while time.time() < endtime and not stopread:
sread, swrite, serror = select.select(socklist, [], [], 5)
for sock in sread:
@@ -283,13 +284,14 @@ class QemuRunner:
if hasattr(self, "origchldhandler"):
signal.signal(signal.SIGCHLD, self.origchldhandler)
if self.runqemu:
- os.kill(self.monitorpid, signal.SIGKILL)
- logger.info("Sending SIGTERM to runqemu")
- try:
- os.killpg(os.getpgid(self.runqemu.pid), signal.SIGTERM)
- except OSError as e:
- if e.errno != errno.ESRCH:
- raise
+ if hasattr(self, "monitorpid"):
+ os.kill(self.monitorpid, signal.SIGKILL)
+ logger.info("Sending SIGTERM to runqemu")
+ try:
+ os.killpg(os.getpgid(self.runqemu.pid), signal.SIGTERM)
+ except OSError as e:
+ if e.errno != errno.ESRCH:
+ raise
endtime = time.time() + self.runqemutime
while self.runqemu.poll() is None and time.time() < endtime:
time.sleep(1)
@@ -448,7 +450,7 @@ class LoggingThread(threading.Thread):
def stop(self):
self.logger.info("Stopping logging thread")
if self.running:
- os.write(self.writepipe, "stop")
+ os.write(self.writepipe, bytes("stop", "utf-8"))
def teardown(self):
self.logger.info("Tearing down logging thread")
diff --git a/meta/lib/oeqa/utils/qemutinyrunner.py b/meta/lib/oeqa/utils/qemutinyrunner.py
index 054ab0e..c823157 100644
--- a/meta/lib/oeqa/utils/qemutinyrunner.py
+++ b/meta/lib/oeqa/utils/qemutinyrunner.py
@@ -13,7 +13,7 @@ import re
import socket
import select
import bb
-from qemurunner import QemuRunner
+from .qemurunner import QemuRunner
class QemuTinyRunner(QemuRunner):
diff --git a/meta/lib/oeqa/utils/targetbuild.py b/meta/lib/oeqa/utils/targetbuild.py
index f850d78..d538f6b 100644
--- a/meta/lib/oeqa/utils/targetbuild.py
+++ b/meta/lib/oeqa/utils/targetbuild.py
@@ -10,9 +10,7 @@ import bb.utils
import subprocess
from abc import ABCMeta, abstractmethod
-class BuildProject():
-
- __metaclass__ = ABCMeta
+class BuildProject(metaclass=ABCMeta):
def __init__(self, d, uri, foldername=None, tmpdir="/tmp/"):
self.d = d
diff --git a/meta/lib/oeqa/utils/testexport.py b/meta/lib/oeqa/utils/testexport.py
index 4fbf4bd..57be2ca 100644
--- a/meta/lib/oeqa/utils/testexport.py
+++ b/meta/lib/oeqa/utils/testexport.py
@@ -6,7 +6,7 @@
import os, re, glob as g, shutil as sh,sys
from time import sleep
-from commands import runCmd
+from .commands import runCmd
from difflib import SequenceMatcher as SM
try:
diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index db132fd..b1ecf7f 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
# Copyright (c) 2013 Intel Corporation
#
@@ -34,6 +34,8 @@ import subprocess
import time as t
import re
import fnmatch
+import collections
+import imp
sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + '/lib')
import scriptpath
@@ -414,7 +416,7 @@ def coverage_report():
# Coverage under version 4 uses coverage.coverage
from coverage import coverage as Coverage
- import cStringIO as StringIO
+ import io as StringIO
from coverage.misc import CoverageException
cov_output = StringIO.StringIO()
@@ -442,7 +444,7 @@ def main():
bbpath = get_bb_var('BBPATH').split(':')
layer_libdirs = [p for p in (os.path.join(l, 'lib') for l in bbpath) if os.path.exists(p)]
sys.path.extend(layer_libdirs)
- reload(oeqa.selftest)
+ imp.reload(oeqa.selftest)
if args.run_tests_by and len(args.run_tests_by) >= 2:
valid_options = ['name', 'class', 'module', 'id', 'tag']
@@ -491,7 +493,7 @@ def main():
if isinstance(t, type(oeSelfTest)) and issubclass(t, oeSelfTest) and t!=oeSelfTest:
print(" --", v)
for method in dir(t):
- if method.startswith("test_") and callable(vars(t)[method]):
+ if method.startswith("test_") and isinstance(vars(t)[method], collections.Callable):
print(" -- --", method)
except (AttributeError, ImportError) as e:
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 18/22] selftest/devtool: Compare sets instead of arrays on AssertEqual
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (15 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 17/22] classes/lib: Complete transition to python3 Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 19/22] buildhistory_analysis: Convert stream data (bytes) to strings Richard Purdie
` (6 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Sets are safer when comparing internal elements and positions are not
important. This commit avoid errors observed on python3 builds as reported
on the below bugzilla entry.
[YOCTO #9661]
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/lib/oeqa/selftest/devtool.py | 32 ++++++++++++++++----------------
1 file changed, 16 insertions(+), 16 deletions(-)
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 6d2417e..deb3090 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -818,28 +818,28 @@ class DevtoolTests(DevtoolBase):
# Check bbappend contents
result = runCmd('git rev-parse HEAD', cwd=tempsrcdir)
- expectedlines = ['SRCREV = "%s"\n' % result.output,
- '\n',
- 'SRC_URI = "%s"\n' % git_uri,
- '\n']
+ expectedlines = set(['SRCREV = "%s"\n' % result.output,
+ '\n',
+ 'SRC_URI = "%s"\n' % git_uri,
+ '\n'])
with open(bbappendfile, 'r') as f:
- self.assertEqual(expectedlines, f.readlines())
+ self.assertEqual(expectedlines, set(f.readlines()))
# Check we can run it again and bbappend isn't modified
result = runCmd('devtool update-recipe -m srcrev %s -a %s' % (testrecipe, templayerdir))
with open(bbappendfile, 'r') as f:
- self.assertEqual(expectedlines, f.readlines())
+ self.assertEqual(expectedlines, set(f.readlines()))
# Drop new commit and check SRCREV changes
result = runCmd('git reset HEAD^', cwd=tempsrcdir)
result = runCmd('devtool update-recipe -m srcrev %s -a %s' % (testrecipe, templayerdir))
self.assertFalse(os.path.exists(os.path.join(appenddir, testrecipe)), 'Patch directory should not be created')
result = runCmd('git rev-parse HEAD', cwd=tempsrcdir)
- expectedlines = ['SRCREV = "%s"\n' % result.output,
- '\n',
- 'SRC_URI = "%s"\n' % git_uri,
- '\n']
+ expectedlines = set(['SRCREV = "%s"\n' % result.output,
+ '\n',
+ 'SRC_URI = "%s"\n' % git_uri,
+ '\n'])
with open(bbappendfile, 'r') as f:
- self.assertEqual(expectedlines, f.readlines())
+ self.assertEqual(expectedlines, set(f.readlines()))
# Put commit back and check we can run it if layer isn't in bblayers.conf
os.remove(bbappendfile)
result = runCmd('git commit -a -m "Change the Makefile"', cwd=tempsrcdir)
@@ -848,12 +848,12 @@ class DevtoolTests(DevtoolBase):
self.assertIn('WARNING: Specified layer is not currently enabled in bblayers.conf', result.output)
self.assertFalse(os.path.exists(os.path.join(appenddir, testrecipe)), 'Patch directory should not be created')
result = runCmd('git rev-parse HEAD', cwd=tempsrcdir)
- expectedlines = ['SRCREV = "%s"\n' % result.output,
- '\n',
- 'SRC_URI = "%s"\n' % git_uri,
- '\n']
+ expectedlines = set(['SRCREV = "%s"\n' % result.output,
+ '\n',
+ 'SRC_URI = "%s"\n' % git_uri,
+ '\n'])
with open(bbappendfile, 'r') as f:
- self.assertEqual(expectedlines, f.readlines())
+ self.assertEqual(expectedlines, set(f.readlines()))
# Deleting isn't expected to work under these circumstances
@testcase(1370)
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 19/22] buildhistory_analysis: Convert stream data (bytes) to strings
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (16 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 18/22] selftest/devtool: Compare sets instead of arrays on AssertEqual Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 20/22] devtool.py: Fix parsing of bitbake-layers' output Richard Purdie
` (5 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
The bytes type variables are threat different as strings variables in python3,
so convert bytes to strings. This was found when using the
script buildhistory-diff, where the interpreter was yielding this error
TypeError: Type str doesn't support the buffer API
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/lib/oe/buildhistory_analysis.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index 0dcd49f..16491a9 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -190,7 +190,7 @@ class FileChange:
def blob_to_dict(blob):
- alines = blob.data_stream.read().splitlines()
+ alines = [line.decode() for line in blob.data_stream.read().splitlines()]
adict = {}
for line in alines:
splitv = [i.strip() for i in line.split('=',1)]
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 20/22] devtool.py: Fix parsing of bitbake-layers' output
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (17 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 19/22] buildhistory_analysis: Convert stream data (bytes) to strings Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 15:38 ` Christopher Larson
2016-06-01 12:35 ` [PATCH 21/22] oeqa/selftest/recipetool.py: Use regex to match warning line Richard Purdie
` (4 subsequent siblings)
23 siblings, 1 reply; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Current parsing was picking wrong targets, leading to the following problem:
AssertionError: Command 'bitbake Parsing recipes..done. -e' returned non-zero exit status 1:
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/lib/oeqa/selftest/devtool.py | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index deb3090..0b305c8 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -467,12 +467,11 @@ class DevtoolTests(DevtoolBase):
testrecipes = 'perf kernel-devsrc package-index core-image-minimal meta-toolchain packagegroup-core-sdk meta-ide-support'.split()
# Find actual name of gcc-source since it now includes the version - crude, but good enough for this purpose
result = runCmd('bitbake-layers show-recipes gcc-source*')
- reading = False
for line in result.output.splitlines():
- if line.startswith('=='):
- reading = True
- elif reading and not line.startswith(' '):
- testrecipes.append(line.split(':')[0])
+ # just match those lines that contain a real target
+ m = re.match('(?P<recipe>^[a-zA-Z0-9.-]+)(?P<colon>:$)', line)
+ if m:
+ testrecipes.append(m.group('recipe'))
for testrecipe in testrecipes:
# Check it's a valid recipe
bitbake('%s -e' % testrecipe)
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH 20/22] devtool.py: Fix parsing of bitbake-layers' output
2016-06-01 12:35 ` [PATCH 20/22] devtool.py: Fix parsing of bitbake-layers' output Richard Purdie
@ 2016-06-01 15:38 ` Christopher Larson
0 siblings, 0 replies; 31+ messages in thread
From: Christopher Larson @ 2016-06-01 15:38 UTC (permalink / raw)
To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 854 bytes --]
On Wed, Jun 1, 2016 at 5:35 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
>
> Current parsing was picking wrong targets, leading to the following
> problem:
>
> AssertionError: Command 'bitbake Parsing recipes..done. -e'
> returned non-zero exit status 1:
>
> Signed-off-by: Leonardo Sandoval <
> leonardo.sandoval.gonzalez@linux.intel.com>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
We should probably think about redirecting parsing messages to stderr, at
the least in the standalone scripts, since that's out of band from the
intended output.
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
[-- Attachment #2: Type: text/html, Size: 1513 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH 21/22] oeqa/selftest/recipetool.py: Use regex to match warning line
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (18 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 20/22] devtool.py: Fix parsing of bitbake-layers' output Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 12:35 ` [PATCH 22/22] qemurunner: Use surrogateescape decoding Richard Purdie
` (3 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
We are observing cases (see below) where the 'WARNING:' prefix is not at the
beginning of a line, so instead of expecting it in the beginning, match it with
a regex.
======================================================================
FAIL: test_recipetool_appendfile_patch (oeqa.selftest.recipetool.RecipetoolTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/decorators.py", line 109, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/recipetool.py", line 285, in test_recipetool_appendfile_patch
self.fail('Patch warning not found in output:\n%s' % output)
AssertionError: Patch warning not found in output:
Parsing recipes..WARNING: File /etc/selftest-replaceme-patched is added by the patch /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta-selftest/recipes-test/recipetool/files/add-file.patch - you may need to remove or replace this patch in order to replace the file.
NOTE: Writing append file /tmp/recipetoolqaug0kdb4x/recipes-test/recipetool/selftest-recipetool-appendfile.bbappend
NOTE: Copying /tmp/recipetoolqagci9tita/testfile to /tmp/recipetoolqaug0kdb4x/recipes-test/recipetool/selftest-recipetool-appendfile/testfile
done.
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/lib/oeqa/selftest/recipetool.py | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oeqa/selftest/recipetool.py b/meta/lib/oeqa/selftest/recipetool.py
index a93d18e..7c60699 100644
--- a/meta/lib/oeqa/selftest/recipetool.py
+++ b/meta/lib/oeqa/selftest/recipetool.py
@@ -2,6 +2,7 @@ import os
import logging
import tempfile
import urllib.parse
+import re
from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer
from oeqa.utils.decorators import testcase
@@ -278,8 +279,9 @@ class RecipetoolTests(RecipetoolBase):
'}\n']
_, output = self._try_recipetool_appendfile('selftest-recipetool-appendfile', '/etc/selftest-replaceme-patched', self.testfile, '', expectedlines, ['testfile'])
for line in output.splitlines():
- if line.startswith('WARNING: '):
- self.assertIn('add-file.patch', line, 'Unexpected warning found in output:\n%s' % line)
+ # match the line containing the warning and the patch
+ match = re.match("(WARNING:).*(add-file.patch)", line)
+ if match:
break
else:
self.fail('Patch warning not found in output:\n%s' % output)
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 22/22] qemurunner: Use surrogateescape decoding
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (19 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 21/22] oeqa/selftest/recipetool.py: Use regex to match warning line Richard Purdie
@ 2016-06-01 12:35 ` Richard Purdie
2016-06-01 15:20 ` [PATCH 01/22] oe-buildenv-internal: Update to python3 Randle, William C
` (2 subsequent siblings)
23 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 12:35 UTC (permalink / raw)
To: openembedded-core
Since the stream can contain invalid binary characters (e.g. from
ppc's bootloader) use surrogateescape decoding to ensure we do process
the character stream, else it can hang/timeout.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/lib/oeqa/utils/qemurunner.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index e408fbb..b8ac3f0 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -234,7 +234,7 @@ class QemuRunner:
data = data + sock.recv(1024)
if data:
try:
- data = data.decode("utf-8")
+ data = data.decode("utf-8", errors="surrogateescape")
bootlog += data
data = b''
if re.search(".* login:", bootlog):
--
2.5.0
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH 01/22] oe-buildenv-internal: Update to python3
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (20 preceding siblings ...)
2016-06-01 12:35 ` [PATCH 22/22] qemurunner: Use surrogateescape decoding Richard Purdie
@ 2016-06-01 15:20 ` Randle, William C
2016-06-01 15:35 ` Christopher Larson
2016-06-02 7:31 ` Python3 for bitbake/OE merged Richard Purdie
23 siblings, 0 replies; 31+ messages in thread
From: Randle, William C @ 2016-06-01 15:20 UTC (permalink / raw)
To: richard.purdie, openembedded-core
On Wed, 2016-06-01 at 13:35 +0100, Richard Purdie wrote:
> Check that 'python' is a python v2 since that is what we assume everywhere
> and upstream python devs recommend. We can need both python2 and python3
> available since we don't control the software we might download and run.
>
> Also check that python 3 is >= 3.4.0, our minimum version for bitbake.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> scripts/oe-buildenv-internal | 38 +++++++++++++++++++++-----------------
> 1 file changed, 21 insertions(+), 17 deletions(-)
>
> diff --git a/scripts/oe-buildenv-internal b/scripts/oe-buildenv-internal
> index 81ee784..56d3419 100755
> --- a/scripts/oe-buildenv-internal
> +++ b/scripts/oe-buildenv-internal
> @@ -29,27 +29,31 @@ if [ -z "$OE_SKIP_SDK_CHECK" ] && [ -n
> "$OECORE_SDK_VERSION" ]; then
> return 1
> fi
>
> -# Make sure we're not using python v3.x. This check can't go into
> -# sanity.bbclass because bitbake's source code doesn't even pass
> -# parsing stage when used with python v3, so we catch it here so we
> -# can offer a meaningful error message.
> -py_v3_check=$(/usr/bin/env python --version 2>&1 | grep "Python 3")
> -if [ -n "$py_v3_check" ]; then
> - echo >&2 "Bitbake is not compatible with python v3"
> - echo >&2 "Please set up python v2 as your default python interpreter"
> +# Make sure we're not using python v3.x as 'python', we don't support it.
> +py_v2_check=$(/usr/bin/env python --version 2>&1 | grep "Python 3")
> +if [ -n "$py_v2_check" ]; then
> + echo >&2 "OpenEmbedded requires 'python' to be python v2 (>= 2.7.3), not
> python v3."
> + echo >&2 "Please set up python v2 as your default 'python' interpreter."
> return 1
> fi
> -unset py_v3_check
> -
> -# Similarly, we now have code that doesn't parse correctly with older
> -# versions of Python, and rather than fixing that and being eternally
> -# vigilant for any other new feature use, just check the version here.
> -py_v26_check=$(python -c 'import sys; print sys.version_info >= (2,7,3)')
> -if [ "$py_v26_check" != "True" ]; then
> - echo >&2 "BitBake requires Python 2.7.3 or later"
> +unset py_v2_check
> +
> +py_v27_check=$(python -c 'import sys; print sys.version_info >= (2,7,3)')
> +if [ "$py_v27_check" != "True" ]; then
> + echo >&2 "OpenEmbedded requires 'python' to be python v2 (>= 2.7.3), not
> python v3."
> + echo >&2 "Please upgrade your python v2."
> +fi
> +unset py_v27_check
> +
> +# We potentially have code that doesn't parse correctly with older versions
> +# of Python, and rather than fixing that and being eternally vigilant for
> +# any other new feature use, just check the version here.
> +py_v34_check=$(python3 -c 'import sys; print(sys.version_info >= (3,4,0))')
> +if [ "$py_v34_check" != "True" ]; then
> + echo >&2 "BitBake requires Python 3.4.0 or later as 'python3'"
> return 1
> fi
> -unset py_v26_check
> +unset py_v34_check
>
> if [ -z "$BDIR" ]; then
> if [ -z "$1" ]; then
> --
> 2.5.0
It seems to me the comment about older Python versions should remain before the
2.7.3 check so that it applies to both the 2.7.3 and 3.4.0 checks.
-Bill
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH 01/22] oe-buildenv-internal: Update to python3
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (21 preceding siblings ...)
2016-06-01 15:20 ` [PATCH 01/22] oe-buildenv-internal: Update to python3 Randle, William C
@ 2016-06-01 15:35 ` Christopher Larson
2016-06-01 20:57 ` Richard Purdie
2016-06-02 7:31 ` Python3 for bitbake/OE merged Richard Purdie
23 siblings, 1 reply; 31+ messages in thread
From: Christopher Larson @ 2016-06-01 15:35 UTC (permalink / raw)
To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 868 bytes --]
On Wed, Jun 1, 2016 at 5:35 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> Check that 'python' is a python v2 since that is what we assume everywhere
> and upstream python devs recommend. We can need both python2 and python3
> available since we don't control the software we might download and run.
>
> Also check that python 3 is >= 3.4.0, our minimum version for bitbake.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
I'm curious, why don't we just have a bit of logic to figure out what the
names are of the binaries, create a subdir and prepend it to the path, and
add symlinks or wrapper scripts there to the real binaries on the system?
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
[-- Attachment #2: Type: text/html, Size: 1424 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH 01/22] oe-buildenv-internal: Update to python3
2016-06-01 15:35 ` Christopher Larson
@ 2016-06-01 20:57 ` Richard Purdie
2016-06-01 20:59 ` Christopher Larson
0 siblings, 1 reply; 31+ messages in thread
From: Richard Purdie @ 2016-06-01 20:57 UTC (permalink / raw)
To: Christopher Larson; +Cc: Patches and discussions about the oe-core layer
On Wed, 2016-06-01 at 08:35 -0700, Christopher Larson wrote:
>
> On Wed, Jun 1, 2016 at 5:35 AM, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
> > Check that 'python' is a python v2 since that is what we assume
> > everywhere
> > and upstream python devs recommend. We can need both python2 and
> > python3
> > available since we don't control the software we might download and
> > run.
> >
> > Also check that python 3 is >= 3.4.0, our minimum version for
> > bitbake.
> >
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> >
> I'm curious, why don't we just have a bit of logic to figure out what
> the names are of the binaries, create a subdir and prepend it to the
> path, and add symlinks or wrapper scripts there to the real binaries
> on the system?
We could do this I guess, I'm kind of hoping we don't have to as it
just adds more complexity.
Tempted to leave the patch as it stands for initial merging and then we
can see how many people have issues...
Cheers,
Richard
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 01/22] oe-buildenv-internal: Update to python3
2016-06-01 20:57 ` Richard Purdie
@ 2016-06-01 20:59 ` Christopher Larson
0 siblings, 0 replies; 31+ messages in thread
From: Christopher Larson @ 2016-06-01 20:59 UTC (permalink / raw)
To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 1531 bytes --]
On Wed, Jun 1, 2016 at 1:57 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> On Wed, 2016-06-01 at 08:35 -0700, Christopher Larson wrote:
> >
> > On Wed, Jun 1, 2016 at 5:35 AM, Richard Purdie <
> > richard.purdie@linuxfoundation.org> wrote:
> > > Check that 'python' is a python v2 since that is what we assume
> > > everywhere
> > > and upstream python devs recommend. We can need both python2 and
> > > python3
> > > available since we don't control the software we might download and
> > > run.
> > >
> > > Also check that python 3 is >= 3.4.0, our minimum version for
> > > bitbake.
> > >
> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > >
> > I'm curious, why don't we just have a bit of logic to figure out what
> > the names are of the binaries, create a subdir and prepend it to the
> > path, and add symlinks or wrapper scripts there to the real binaries
> > on the system?
>
> We could do this I guess, I'm kind of hoping we don't have to as it
> just adds more complexity.
>
> Tempted to leave the patch as it stands for initial merging and then we
> can see how many people have issues...
That's reasonable. I think the main distro that'll run into this is arch,
and arch users should be used to having to hack around 'python' not being
python 2 for other projects by now :)
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
[-- Attachment #2: Type: text/html, Size: 2294 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread
* Python3 for bitbake/OE merged
2016-06-01 12:35 [PATCH 01/22] oe-buildenv-internal: Update to python3 Richard Purdie
` (22 preceding siblings ...)
2016-06-01 15:35 ` Christopher Larson
@ 2016-06-02 7:31 ` Richard Purdie
2016-06-02 15:23 ` Christopher Larson
23 siblings, 1 reply; 31+ messages in thread
From: Richard Purdie @ 2016-06-02 7:31 UTC (permalink / raw)
To: openembedded-core
I just watched things break on the autobuilder as trying to keep the
python3 patches in sync with all the other changes coming in was
proving tricky and not really a productive use of my time.
I've therefore merged the python3 patches and bitbake is now using
python3. If there are any further issues, we can fix in master. This
may mean some other patchsets need to rebase.
Cheers,
Richard
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: Python3 for bitbake/OE merged
2016-06-02 7:31 ` Python3 for bitbake/OE merged Richard Purdie
@ 2016-06-02 15:23 ` Christopher Larson
0 siblings, 0 replies; 31+ messages in thread
From: Christopher Larson @ 2016-06-02 15:23 UTC (permalink / raw)
To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 668 bytes --]
On Thu, Jun 2, 2016 at 12:31 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> I just watched things break on the autobuilder as trying to keep the
> python3 patches in sync with all the other changes coming in was
> proving tricky and not really a productive use of my time.
>
> I've therefore merged the python3 patches and bitbake is now using
> python3. If there are any further issues, we can fix in master. This
> may mean some other patchsets need to rebase.
>
Huzzah!
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
[-- Attachment #2: Type: text/html, Size: 1119 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread