[opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse

[opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse

[Martin Jansa]

First 5 patches are taken from oe-core.

Then there are some cleanups and fixes for issues I've found while looking 
for cause of very long package-index calls.

Some patches uses compatible changes from 2to3 (tested with python2.7), but
python3 support is not 100% complete, but were closer.

And in the end I've found the cause.. printing long fields (e. g. description)
with opkg.py was producing empty lines in Packages file (which are Packages entry 
separators) so only first entries till first empty line were processed from old
Packages file and the rest was always extracted from control files in packaged .ipk.

And this also fixes Packages.filelist generation.

I don't use python very much, so please review.

And the speedup? Almost 60 times :)
before:
real    20m10.484s
user    18m32.335s
sys     0m11.885s

after
real    0m24.492s
user    0m23.748s
sys     0m0.565s

The following changes since commit 002d29bc605d7c2d02e4cf20a43c5277c15f5597:

  [opkg-utils] fix install fail problem  Thanks for khorben's patch :-)  https://docs.openmoko.org/trac/attachment/ticket/2072/patch-opkg-utils_Makefile_install_path.diff (2008-11-03 03:59:59 +0000)

are available in the git repository at:
  git://github.com/shr-project/opkg-utils jansa/pull
  https://github.com/shr-project/opkg-utils/tree/jansa/pull

Christopher Larson (1):
  Use python via the PATH, rather than hardcoding /usr/bin/python

Enrico Scholz (1):
  opkg-make-index: convert mtime to int before comparing it

Khem Raj (1):
  opkg.py: Add knowledge about License field in ipk headers

Martin Jansa (17):
  opkg.py: use hashlib instead of old md5 module
  opkg.py, arfile.py: report which file has wrong format and use the
    same test
  arfile: decode read lines as ascii string before getting fields from
    it
  2to3: use subprocess instead of commands
  2to3: print fixes
  2to3: dictionary fixes
  2to3: exception handling fixes
  opkg-make-index: show OSError/IOError
  opkg.py: use string funtcions directly on string variable
  opkg.py: catch TypeError when reading control file, to show which one
    is failing
  arfile: fix test
  opkg-make-index: don't use stdout for Packages output and use __str__
    instead of __repr__
  opkg.py: cast lines from controlfile as string
  opkg.py: computeFileMD5 only when we have fn, otherwise fails to read
    None file
  opkg.py: fix write_package when called from main test
  opkg.py: use textwrap for description writing
  opkg.py: improve test so it prints temporary control file with long
    description and then reads it back

Richard Purdie (1):
  opkg-make-index: don't error out when some package disappears

Scott Anderson (1):
  arfile.py: handle six digit UIDs

 arfile.py            |   55 +++++++++++++---------
 makePackage          |    2 +-
 opkg-compare-indexes |   22 ++++----
 opkg-list-fields     |    4 +-
 opkg-make-index      |   74 +++++++++++++++++++----------
 opkg-show-deps       |   28 +++++------
 opkg-unbuild         |    4 +-
 opkg-update-index    |    5 +-
 opkg.py              |  125 +++++++++++++++++++++++++++++--------------------
 9 files changed, 186 insertions(+), 133 deletions(-)

-- 
1.7.8.5

[opkg-utils][PATCH 01/22] opkg-make-index: don't error out when some package disappears

[Martin Jansa]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

* If we're building an image and some package rebuilds while this is
  happening some package can be removed/added to the ipk deploy
  directory. The image will not depend on this package so we can
  safely ignore these cases rather than error out.

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg-make-index |   25 ++++++++++++++++++++++++-
 1 files changed, 24 insertions(+), 1 deletions(-)

diff --git a/opkg-make-index b/opkg-make-index
index ae829e6..2f1ae17 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -96,6 +96,7 @@ if (verbose):
 files=glob(pkg_dir + '/*.opk') + glob(pkg_dir + '/*.deb') + glob(pkg_dir + '/*.ipk')
 files.sort()
 for filename in files:
+  try:
      basename = os.path.basename(filename)
      pkg = None
      fnameStat = os.stat(filename)
@@ -130,6 +131,12 @@ for filename in files:
                to_morgue(basename)
           if opt_s:
                print filename
+  except OSError:
+      sys.stderr.write("Package %s disappeared on us!\n" % (filename))
+      continue
+  except IOError:
+      sys.stderr.write("Package %s disappeared on us!\n" % (filename))
+      continue
 
 pkgsStampsFile = open(stamplist_filename, "w")
 for f in pkgsStamps.keys():
@@ -148,6 +155,7 @@ if packages_filename:
 names = packages.packages.keys()
 names.sort()
 for name in names:
+  try:
      pkg = packages.packages[name]
      if locales_dir and pkg.depends:
          depends = string.split(pkg.depends, ',')
@@ -165,6 +173,13 @@ for name in names:
      if (verbose):
           sys.stderr.write("Writing info for package %s\n" % (pkg.package,))
      print pkg
+  except OSError:
+      sys.stderr.write("Package %s disappeared on us!\n" % (name))
+      continue
+  except IOError:
+      sys.stderr.write("Package %s disappeared on us!\n" % (name))
+      continue
+
 if packages_filename:
      sys.stdout.close()
      sys.stdout = old_stdout
@@ -182,7 +197,15 @@ files = {}
 names = packages.packages.keys()
 names.sort()
 for name in names:
-     for fn in packages[name].get_file_list():
+     try:
+          fnlist = packages[name].get_file_list()
+     except OSError, e:
+          sys.stderr.write("Package %s disappeared on us!\n" % (name))
+          continue
+     except IOError, e:
+          sys.stderr.write("Package %s disappeared on us!\n" % (name))
+          continue
+     for fn in fnlist:
           (h,t) = os.path.split(fn)
           if not t: continue
           if not files.has_key(t): files[t] = name+':'+fn
-- 
1.7.8.5

[opkg-utils][PATCH 02/22] opkg-make-index: convert mtime to int before comparing it

[Martin Jansa]

From: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>

* The st_mtime attribute (which is a float) is compared against a value
  from the timestamp database, which was stored as an integer there.

* When working on a filesystem with precise timestamps the comparision
  will fail nearly everytime hence.

* Although it might be possible to enhance the database to store the
  fractional part too, this will complicate things more than we would
  gain by this change.

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg-make-index |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/opkg-make-index b/opkg-make-index
index 2f1ae17..dc98c63 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -101,7 +101,7 @@ for filename in files:
      pkg = None
      fnameStat = os.stat(filename)
      if old_pkg_hash.has_key(basename):
-          if pkgsStamps.has_key(basename) and fnameStat.st_mtime == pkgsStamps[basename]:
+          if pkgsStamps.has_key(basename) and int(fnameStat.st_mtime) == pkgsStamps[basename]:
             if (verbose):
                sys.stderr.write("Found %s in Packages\n" % (filename,))
             pkg = old_pkg_hash[basename]
-- 
1.7.8.5

[opkg-utils][PATCH 03/22] opkg.py: Add knowledge about License field in ipk headers

[Martin Jansa]

From: Khem Raj <raj.khem@gmail.com>

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg.py |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/opkg.py b/opkg.py
index 3fda9b5..8ddc8b8 100644
--- a/opkg.py
+++ b/opkg.py
@@ -145,6 +145,7 @@ class Package:
         self.priority = None
         self.tags = None
         self.fn = fn
+        self.license = None
 
         if fn:
             # see if it is deb format
@@ -319,6 +320,12 @@ class Package:
     def get_section(self, section):
         return self.section
 
+    def set_license(self, license):
+        self.license = license
+
+    def get_license(self, license):
+        return self.license
+
     def get_file_list(self):
         if not self.fn:
             return []
@@ -425,6 +432,7 @@ class Package:
         if self.description: out = out + "Description: %s\n" % (self.description)
         if self.oe: out = out + "OE: %s\n" % (self.oe)
         if self.homepage: out = out + "HomePage: %s\n" % (self.homepage)
+        if self.license: out = out + "License: %s\n" % (self.license)
         if self.priority: out = out + "Priority: %s\n" % (self.priority)
         if self.tags: out = out + "Tags: %s\n" % (self.tags)
         out = out + "\n"
-- 
1.7.8.5

[opkg-utils][PATCH 04/22] arfile.py: handle six digit UIDs

[Martin Jansa]

From: Scott Anderson <o2e@saaworld.com>

* Essentially, the problem is that arfile.py is splitting the ar header with
  white-space instead of fixed-width fields, so two fields would get treated
  as a single field.  This makes things better than before as it now honors
  the fixed field widths.

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 arfile.py |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/arfile.py b/arfile.py
index 22548af..8291a2d 100644
--- a/arfile.py
+++ b/arfile.py
@@ -75,7 +75,12 @@ class ArFile:
                 l = self.f.readline()
                 if not l: break
             l = l.replace('`', '')
-            descriptor = l.split()
+            # Field lengths from /usr/include/ar.h:
+            ar_field_lens = [ 16, 12, 6, 6, 8, 10, 2 ]
+            descriptor = []
+            for field_len in ar_field_lens:
+                descriptor.append(l[:field_len].strip())
+                l = l[field_len:]
 #            print descriptor
             size = int(descriptor[5])
             memberName = descriptor[0][:-1]
-- 
1.7.8.5

[opkg-utils][PATCH 05/22] Use python via the PATH, rather than hardcoding /usr/bin/python

[Martin Jansa]

From: Christopher Larson <kergoth@gmail.com>

Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg-list-fields  |    2 +-
 opkg-make-index   |    2 +-
 opkg-show-deps    |    2 +-
 opkg-unbuild      |    2 +-
 opkg-update-index |    2 +-
 5 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/opkg-list-fields b/opkg-list-fields
index d263b90..da78d53 100755
--- a/opkg-list-fields
+++ b/opkg-list-fields
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 
 import sys, opkg
 
diff --git a/opkg-make-index b/opkg-make-index
index dc98c63..b65dc6e 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 
 import sys, os, posixpath
 from glob import glob
diff --git a/opkg-show-deps b/opkg-show-deps
index a6681f4..9de1aac 100755
--- a/opkg-show-deps
+++ b/opkg-show-deps
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 
 import sys, os, posixpath
 from glob import glob
diff --git a/opkg-unbuild b/opkg-unbuild
index eff604b..b5c5227 100755
--- a/opkg-unbuild
+++ b/opkg-unbuild
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 
 import sys, os, re
 
diff --git a/opkg-update-index b/opkg-update-index
index 807f8f4..3864fa5 100755
--- a/opkg-update-index
+++ b/opkg-update-index
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2.1
+#!/usr/bin/env python
 
 import sys, os
 from glob import glob
-- 
1.7.8.5

[opkg-utils][PATCH 06/22] opkg.py: use hashlib instead of old md5 module

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg.py |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/opkg.py b/opkg.py
index 8ddc8b8..56e774d 100644
--- a/opkg.py
+++ b/opkg.py
@@ -36,7 +36,7 @@ import tempfile
 import os
 import sys
 import glob
-import md5
+import hashlib
 import re
 import string
 import commands
@@ -188,8 +188,8 @@ class Package:
     def _computeFileMD5(self):
         # compute the MD5.
         f = open(self.fn, "rb")
-        sum = md5.new()
-        while 1:
+        sum = hashlib.md5()
+        while True:
             data = f.read(1024)
             if not data: break
             sum.update(data)
-- 
1.7.8.5

[opkg-utils][PATCH 07/22] opkg.py, arfile.py: report which file has wrong format and use the same test

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 arfile.py |   16 +++++++++-------
 opkg.py   |   11 ++---------
 2 files changed, 11 insertions(+), 16 deletions(-)

diff --git a/arfile.py b/arfile.py
index 8291a2d..7a695c6 100644
--- a/arfile.py
+++ b/arfile.py
@@ -40,13 +40,13 @@ class FileSection:
 
 class ArFile:
 
-    def __init__(self, f):
+    def __init__(self, f, fn):
         self.f = f
         self.directory = {}
         self.directoryRead = False
 
         signature = self.f.readline()
-        assert signature == "!<arch>\n"
+        assert signature == "!<arch>\n" or signature == b"!<arch>\n", "Old ipk format (non-deb) is unsupported, file: %s, magic: %s, expected %s" % (fn, signature, "!<arch>")
         self.directoryOffset = self.f.tell()
 
     def open(self, fname):
@@ -100,9 +100,10 @@ class ArFile:
 
 if __name__ == "__main__":
     if None:
-        f = open(sys.argv[1], "rb")
+        fn = sys.argv[1]
+        f = open(fn, "rb")
 
-        ar = ArFile(f)
+        ar = ArFile(f, fn)
         tarStream = ar.open("data.tar.gz")
         print "--------"
         tarStream = ar.open("data.tar.gz")
@@ -120,10 +121,11 @@ if __name__ == "__main__":
     for f in os.listdir(dir):
         if not f.endswith(".opk") and not f.endswith(".ipk"): continue
 
-        print "=== %s ===" % f
-        f = open(dir + "/" + f, "rb")
+        print("=== %s ===" % f)
+        fn = "%s/%s" % (dir, f)
+        f = open(fn, "rb")
 
-        ar = ArFile(f)
+        ar = ArFile(f, fn)
         tarStream = ar.open("control.tar.gz")
         tarf = tarfile.open("control.tar.gz", "r", tarStream)
         #tarf.list()
diff --git a/opkg.py b/opkg.py
index 56e774d..31b847c 100644
--- a/opkg.py
+++ b/opkg.py
@@ -138,7 +138,6 @@ class Package:
         #self.size = None
         self.installed_size = None
         self.filename = None
-        self.isdeb = 0
         self.file_ext_opk = "ipk"
         self.homepage = None
         self.oe = None
@@ -150,18 +149,12 @@ class Package:
         if fn:
             # see if it is deb format
             f = open(fn, "rb")
-            magic = f.read(4)
-            f.seek(0, 0)
-            if (magic == "!<ar"):
-                self.isdeb = 1
-
 
             self.filename = os.path.basename(fn)
-            assert self.isdeb == 1, "Old ipk format (non-deb) is unsupported"
 
             ## sys.stderr.write("  extracting control.tar.gz from %s\n"% (fn,)) 
 
-            ar = arfile.ArFile(f)
+            ar = arfile.ArFile(f, fn)
             tarStream = ar.open("control.tar.gz")
             tarf = tarfile.open("control.tar.gz", "r", tarStream)
 
@@ -330,7 +323,7 @@ class Package:
         if not self.fn:
             return []
         f = open(self.fn, "rb")
-        ar = arfile.ArFile(f)
+        ar = arfile.ArFile(f, self.fn)
         tarStream = ar.open("data.tar.gz")
         tarf = tarfile.open("data.tar.gz", "r", tarStream)
         self.file_list = tarf.getnames()
-- 
1.7.8.5

[opkg-utils][PATCH 08/22] arfile: decode read lines as ascii string before getting fields from it

[Martin Jansa]

* python3 returns them as byte sequence

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 arfile.py |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/arfile.py b/arfile.py
index 7a695c6..92ffee0 100644
--- a/arfile.py
+++ b/arfile.py
@@ -74,6 +74,7 @@ class ArFile:
             if l == "\n":
                 l = self.f.readline()
                 if not l: break
+            l = l.decode('ascii')
             l = l.replace('`', '')
             # Field lengths from /usr/include/ar.h:
             ar_field_lens = [ 16, 12, 6, 6, 8, 10, 2 ]
-- 
1.7.8.5

[opkg-utils][PATCH 09/22] 2to3: use subprocess instead of commands

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg-compare-indexes |    6 +++---
 opkg-make-index      |    5 ++---
 opkg-show-deps       |    1 -
 opkg-update-index    |    1 -
 opkg.py              |    2 +-
 5 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/opkg-compare-indexes b/opkg-compare-indexes
index 6c68125..2610987 100755
--- a/opkg-compare-indexes
+++ b/opkg-compare-indexes
@@ -2,7 +2,7 @@
 
 import sys, os
 from glob import glob
-import commands
+import subprocess
 import opkg
 
 pkg_dir1 = sys.argv[1]
@@ -40,8 +40,8 @@ for name in names:
     if pkg1 and pkg2 and pkg1.version != pkg2.version:
         print "CHANGED: %s from version %s to %s (%s)" % (pkg1.package, pkg1.version, pkg2.version, pkg2.maintainer)
         cmd = "opkg-diff %s %s > %s " % ((pkg_dir1 + pkg1.filename),  (pkg_dir2 + pkg2.filename), (pkg1.package + '.diff'))
-        print cmd
-	commands.getstatusoutput(cmd)
+        print(cmd)
+        subprocess.call(cmd)
     if not pkg1:
         print "NEW: %s version %s (%s)"% (pkg2.package, pkg2.version, pkg2.maintainer)
     if not pkg2:
diff --git a/opkg-make-index b/opkg-make-index
index b65dc6e..7923f1e 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -2,7 +2,7 @@
 
 import sys, os, posixpath
 from glob import glob
-import commands
+import subprocess
 import opkg
 import getopt
 import string
@@ -186,8 +186,7 @@ if packages_filename:
      gzip_filename = ("%s.gz" % packages_filename)
      tmp_gzip_filename = ("%s.%d" % (gzip_filename, os.getpid()))
      gzip_cmd = "gzip -9c < %s > %s" % (tmp_packages_filename, tmp_gzip_filename)
-     (rc, outtext) = commands.getstatusoutput(gzip_cmd)
-     print outtext
+     rc = subprocess.check_output(gzip_cmd, shell=True)
      os.rename(tmp_packages_filename, packages_filename)
      os.rename(tmp_gzip_filename, gzip_filename)
 
diff --git a/opkg-show-deps b/opkg-show-deps
index 9de1aac..5ab5b4f 100755
--- a/opkg-show-deps
+++ b/opkg-show-deps
@@ -2,7 +2,6 @@
 
 import sys, os, posixpath
 from glob import glob
-import commands
 import opkg
 import getopt
 import string
diff --git a/opkg-update-index b/opkg-update-index
index 3864fa5..d9c9b43 100755
--- a/opkg-update-index
+++ b/opkg-update-index
@@ -2,7 +2,6 @@
 
 import sys, os
 from glob import glob
-import commands
 import opkg
 
 pkg_dir=sys.argv[1]
diff --git a/opkg.py b/opkg.py
index 31b847c..f37a68d 100644
--- a/opkg.py
+++ b/opkg.py
@@ -39,7 +39,7 @@ import glob
 import hashlib
 import re
 import string
-import commands
+import subprocess
 from stat import ST_SIZE
 import arfile
 import tarfile
-- 
1.7.8.5

[opkg-utils][PATCH 10/22] 2to3: print fixes

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 arfile.py            |   20 ++++++++++----------
 makePackage          |    2 +-
 opkg-compare-indexes |    6 +++---
 opkg-list-fields     |    2 +-
 opkg-make-index      |    8 ++++----
 opkg-show-deps       |    2 +-
 opkg-unbuild         |    2 +-
 opkg.py              |   24 ++++++++++++------------
 8 files changed, 33 insertions(+), 33 deletions(-)

diff --git a/arfile.py b/arfile.py
index 92ffee0..320c9ca 100644
--- a/arfile.py
+++ b/arfile.py
@@ -20,7 +20,7 @@ class FileSection:
         self.seek(0, 0)
 
     def seek(self, offset, whence = 0):
-#        print "seek(%x, %d)" % (offset, whence)
+#        print("seek(%x, %d)" % (offset, whence))
         if whence == 0:
             return self.f.seek(offset + self.offset, whence)
         elif whence == 1:
@@ -31,11 +31,11 @@ class FileSection:
             assert False
 
     def tell(self):
-#        print "tell()"
+#        print("tell()")
         return self.f.tell() - self.offset
 
     def read(self, size = -1):
-#        print "read(%d)" % size
+#        print("read(%d)" % size)
         return self.f.read(size)
 
 class ArFile:
@@ -82,11 +82,11 @@ class ArFile:
             for field_len in ar_field_lens:
                 descriptor.append(l[:field_len].strip())
                 l = l[field_len:]
-#            print descriptor
+#            print(descriptor)
             size = int(descriptor[5])
             memberName = descriptor[0][:-1]
             self.directory[memberName] = descriptor + [self.f.tell()]
-#            print "read:", memberName
+#            print(("read:", memberName))
             if memberName == fname:
                 # Record directory offset to start from next time
                 self.directoryOffset = self.f.tell() + size
@@ -96,7 +96,7 @@ class ArFile:
             if size % 2:
                 size = size + 1
             data = self.f.seek(size, 1)
-#            print hex(f.tell())
+#            print(hex(self.f.tell()))
 
 
 if __name__ == "__main__":
@@ -106,11 +106,11 @@ if __name__ == "__main__":
 
         ar = ArFile(f, fn)
         tarStream = ar.open("data.tar.gz")
-        print "--------"
+        print("--------")
         tarStream = ar.open("data.tar.gz")
-        print "--------"
+        print("--------")
         tarStream = ar.open("control.tar.gz")
-        print "--------"
+        print("--------")
         tarStream = ar.open("control.tar.gz2")
 
         sys.exit(0)
@@ -132,4 +132,4 @@ if __name__ == "__main__":
         #tarf.list()
 
         f2 = tarf.extractfile("control")
-        print f2.read()
+        print(f2.read())
diff --git a/makePackage b/makePackage
index 082a81f..ec76338 100755
--- a/makePackage
+++ b/makePackage
@@ -11,4 +11,4 @@ import opkg
 
 fn = sys.argv[1]
 pkg = opkg.Package(fn)
-print pkg
+print(pkg)
diff --git a/opkg-compare-indexes b/opkg-compare-indexes
index 2610987..e0933ce 100755
--- a/opkg-compare-indexes
+++ b/opkg-compare-indexes
@@ -38,12 +38,12 @@ for name in names:
     if pkgs2.packages.has_key(name):
         pkg2 = pkgs2.packages[name]
     if pkg1 and pkg2 and pkg1.version != pkg2.version:
-        print "CHANGED: %s from version %s to %s (%s)" % (pkg1.package, pkg1.version, pkg2.version, pkg2.maintainer)
+        print("CHANGED: %s from version %s to %s (%s)" % (pkg1.package, pkg1.version, pkg2.version, pkg2.maintainer))
         cmd = "opkg-diff %s %s > %s " % ((pkg_dir1 + pkg1.filename),  (pkg_dir2 + pkg2.filename), (pkg1.package + '.diff'))
         print(cmd)
         subprocess.call(cmd)
     if not pkg1:
-        print "NEW: %s version %s (%s)"% (pkg2.package, pkg2.version, pkg2.maintainer)
+        print("NEW: %s version %s (%s)"% (pkg2.package, pkg2.version, pkg2.maintainer))
     if not pkg2:
-        print "DELETE: %s version %s (%s)"% (pkg1.package, pkg1.version, pkg1.maintainer)
+        print("DELETE: %s version %s (%s)"% (pkg1.package, pkg1.version, pkg1.maintainer))
     
diff --git a/opkg-list-fields b/opkg-list-fields
index da78d53..1fb7fd1 100755
--- a/opkg-list-fields
+++ b/opkg-list-fields
@@ -9,5 +9,5 @@ def usage():
 if (len(sys.argv) < 2):
      usage()
 
-print opkg.Package(sys.argv[1])
+print(opkg.Package(sys.argv[1]))
 
diff --git a/opkg-make-index b/opkg-make-index
index 7923f1e..c3a292b 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -125,12 +125,12 @@ for filename in files:
                if opt_m:
                     to_morgue(old_filename)
                if opt_s:
-                    print pkg_dir + "/" + old_filename
+                    print(("%s/%s" % (pkg_dir, old_filename)))
      else:
           if opt_m:
                to_morgue(basename)
           if opt_s:
-               print filename
+               print(filename)
   except OSError:
       sys.stderr.write("Package %s disappeared on us!\n" % (filename))
       continue
@@ -172,7 +172,7 @@ for name in names:
               continue
      if (verbose):
           sys.stderr.write("Writing info for package %s\n" % (pkg.package,))
-     print pkg
+     print(pkg)
   except OSError:
       sys.stderr.write("Package %s disappeared on us!\n" % (name))
       continue
@@ -216,7 +216,7 @@ if filelist_filename:
      names = files.keys()
      names.sort()
      for name in names:
-          print name,files[name]
+          print((name,files[name]))
      sys.stdout.close()
      if posixpath.exists(filelist_filename):
           os.unlink(filelist_filename)
diff --git a/opkg-show-deps b/opkg-show-deps
index 5ab5b4f..524d247 100755
--- a/opkg-show-deps
+++ b/opkg-show-deps
@@ -83,5 +83,5 @@ for root in remaining_args:
           recurse(p)
      
 for pkg in required.keys():
-     print pkg
+     print(pkg)
 
diff --git a/opkg-unbuild b/opkg-unbuild
index b5c5227..35a387f 100755
--- a/opkg-unbuild
+++ b/opkg-unbuild
@@ -3,7 +3,7 @@
 import sys, os, re
 
 if (len(sys.argv) == 0):
-    print 'usage: %s: package.opk' % sys.argv[0]
+    print('usage: %s: package.opk' % sys.argv[0])
     sys.exit(1)
 
 for filename in sys.argv[1:]:
diff --git a/opkg.py b/opkg.py
index f37a68d..54060c3 100644
--- a/opkg.py
+++ b/opkg.py
@@ -56,10 +56,10 @@ class Version:
         while 1:
             ## first look for non-numeric version component
             selfm = re.match('([^0-9]*)(.*)', selfversion)
-            #print 'selfm', selfm.groups()
+            #print(('selfm', selfm.groups()))
             (selfalpha, selfversion) = selfm.groups()
             refm = re.match('([^0-9]*)(.*)', refversion)
-            #print 'refm', refm.groups()
+            #print(('refm', refm.groups())
             (refalpha, refversion) = refm.groups()
             if (selfalpha > refalpha):
                 return 1
@@ -68,8 +68,8 @@ class Version:
             ## now look for numeric version component
             (selfnum, selfversion) = re.match('([0-9]*)(.*)', selfversion).groups()
             (refnum, refversion) = re.match('([0-9]*)(.*)', refversion).groups()
-            #print 'selfnum', selfnum, selfversion
-            #print 'refnum', refnum, refversion
+            #print(('selfnum', selfnum, selfversion)
+            #print(('refnum', refnum, refversion)
             if (selfnum != ''):
                 selfnum = int(selfnum)
             else:
@@ -93,12 +93,12 @@ class Version:
         else:
             self_ver_comps = re.match(r"(.+?)(-r.+)?$", self.version)
             ref_ver_comps = re.match(r"(.+?)(-r.+)?$", ref.version)
-            #print (self_ver_comps.group(1), self_ver_comps.group(2))
-            #print (ref_ver_comps.group(1), ref_ver_comps.group(2))
+            #print((self_ver_comps.group(1), self_ver_comps.group(2)))
+            #print((ref_ver_comps.group(1), ref_ver_comps.group(2)))
             r = self._versioncompare(self_ver_comps.group(1), ref_ver_comps.group(1))
             if r == 0:
                 r = self._versioncompare(self_ver_comps.group(2), ref_ver_comps.group(2))
-            #print "compare: %s vs %s = %d" % (self, ref, r)
+            #print("compare: %s vs %s = %d" % (self, ref, r))
             return r
 
     def __str__(self):
@@ -220,7 +220,7 @@ class Package:
                 elif self.__dict__.has_key(name):
                     self.__dict__[name] = value
                 else:
-                    print "Lost field %s, %s" % (name,value)
+                    print("Lost field %s, %s" % (name,value))
                     pass
 
                 if line and line[0] == '\n':
@@ -391,9 +391,9 @@ class Package:
     def compare_version(self, ref):
         """Compare package versions of self and ref"""
         if not self.version:
-            print 'No version for package %s' % self.package
+            print('No version for package %s' % self.package)
         if not ref.version:
-            print 'No version for package %s' % ref.package
+            print('No version for package %s' % ref.package)
         if not self.parsed_version:
             self.parsed_version = parse_version(self.version)
         if not ref.parsed_version:
@@ -499,9 +499,9 @@ if __name__ == "__main__":
     package.set_depends("libc")
     package.set_description("A test of the APIs.")
 
-    print "<"
+    print("<")
     sys.stdout.write(package)
-    print ">"
+    print(">")
 
     package.write_package("/tmp")
 
-- 
1.7.8.5

[opkg-utils][PATCH 11/22] 2to3: dictionary fixes

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 arfile.py            |    2 +-
 opkg-compare-indexes |   10 +++++-----
 opkg-make-index      |   18 +++++++++---------
 opkg-show-deps       |   12 ++++++------
 opkg-update-index    |    2 +-
 opkg.py              |    8 ++++----
 6 files changed, 26 insertions(+), 26 deletions(-)

diff --git a/arfile.py b/arfile.py
index 320c9ca..9149387 100644
--- a/arfile.py
+++ b/arfile.py
@@ -50,7 +50,7 @@ class ArFile:
         self.directoryOffset = self.f.tell()
 
     def open(self, fname):
-        if self.directory.has_key(fname):
+        if fname in self.directory:
             return FileSection(self.f, self.directory[fname][-1], int(self.directory[fname][5]))
 
         if self.directoryRead:
diff --git a/opkg-compare-indexes b/opkg-compare-indexes
index e0933ce..0c119e3 100755
--- a/opkg-compare-indexes
+++ b/opkg-compare-indexes
@@ -18,8 +18,8 @@ pkgs1.read_packages_file(pkg_dir1 + '/Packages')
 pkgs2 = opkg.Packages()
 pkgs2.read_packages_file(pkg_dir2 + '/Packages')
 
-names1 = pkgs1.packages.keys()
-names2 = pkgs2.packages.keys()
+names1 = list(pkgs1.packages.keys())
+names2 = list(pkgs2.packages.keys())
 
 ## union of the two names lists
 pkgs = {}
@@ -28,14 +28,14 @@ for name in names1:
 for name in names2:
     pkgs[name] = pkgs2.packages[name]
 
-names = pkgs.keys()
+names = list(pkgs.keys())
 names.sort() 
 for name in names:
     pkg1 = None
     pkg2 = None
-    if pkgs1.packages.has_key(name):
+    if name in pkgs1.packages:
         pkg1 = pkgs1.packages[name]
-    if pkgs2.packages.has_key(name):
+    if name in pkgs2.packages:
         pkg2 = pkgs2.packages[name]
     if pkg1 and pkg2 and pkg1.version != pkg2.version:
         print("CHANGED: %s from version %s to %s (%s)" % (pkg1.package, pkg1.version, pkg2.version, pkg2.maintainer))
diff --git a/opkg-make-index b/opkg-make-index
index c3a292b..2fc8a69 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -79,7 +79,7 @@ if old_filename:
           sys.stderr.write("Reading package list from " + old_filename + "\n")
      old_packages = opkg.Packages()
      old_packages.read_packages_file(old_filename)
-     for k in old_packages.packages.keys():
+     for k in list(old_packages.packages.keys()):
           p = old_packages.packages[k]
           old_pkg_hash[p.filename] = p
      try:
@@ -100,8 +100,8 @@ for filename in files:
      basename = os.path.basename(filename)
      pkg = None
      fnameStat = os.stat(filename)
-     if old_pkg_hash.has_key(basename):
-          if pkgsStamps.has_key(basename) and int(fnameStat.st_mtime) == pkgsStamps[basename]:
+     if basename in old_pkg_hash:
+          if basename in pkgsStamps and int(fnameStat.st_mtime) == pkgsStamps[basename]:
             if (verbose):
                sys.stderr.write("Found %s in Packages\n" % (filename,))
             pkg = old_pkg_hash[basename]
@@ -113,7 +113,7 @@ for filename in files:
                sys.stderr.write("Reading info for package %s\n" % (filename,))
           pkg = opkg.Package(filename)
      pkg_key = ("%s:%s" % (pkg.package, pkg.architecture))
-     if (packages.packages.has_key(pkg_key)):
+     if (pkg_key in packages.packages):
           old_filename = packages.packages[pkg_key].filename
      else:
           old_filename = ""
@@ -139,7 +139,7 @@ for filename in files:
       continue
 
 pkgsStampsFile = open(stamplist_filename, "w")
-for f in pkgsStamps.keys():
+for f in list(pkgsStamps.keys()):
     pkgsStampsFile.write("%d %s\n" % (pkgsStamps[f], f))
 pkgsStampsFile.close()
 
@@ -152,7 +152,7 @@ if packages_filename:
      old_stdout = sys.stdout
      tmp_packages_filename = ("%s.%d" % (packages_filename, os.getpid()))
      sys.stdout = open(tmp_packages_filename, "w")
-names = packages.packages.keys()
+names = list(packages.packages.keys())
 names.sort()
 for name in names:
   try:
@@ -193,7 +193,7 @@ if packages_filename:
 if verbose:
      sys.stderr.write("Generate Packages.filelist file\n")
 files = {}
-names = packages.packages.keys()
+names = list(packages.packages.keys())
 names.sort()
 for name in names:
      try:
@@ -207,13 +207,13 @@ for name in names:
      for fn in fnlist:
           (h,t) = os.path.split(fn)
           if not t: continue
-          if not files.has_key(t): files[t] = name+':'+fn
+          if t not in files: files[t] = name+':'+fn
           else: files[t] = files[t] + ',' + name+':'+fn
 
 if filelist_filename:
      tmp_filelist_filename = ("%s.%d" % (filelist_filename, os.getpid()))
      sys.stdout = open(tmp_filelist_filename, "w")
-     names = files.keys()
+     names = list(files.keys())
      names.sort()
      for name in names:
           print((name,files[name]))
diff --git a/opkg-show-deps b/opkg-show-deps
index 524d247..3f5899d 100755
--- a/opkg-show-deps
+++ b/opkg-show-deps
@@ -40,21 +40,21 @@ def split_list(str):
                r.append(ii)
      return r
 
-for i in packages.packages.keys():
+for i in list(packages.packages.keys()):
      p = packages.packages[i]
-     if not provider_hash.has_key(p.package):
+     if p.package not in provider_hash:
           provider_hash[p.package] = []
      provider_hash[p.package].append(p)
      if p.provides:
           provides = string.split(p.provides, ",")
           for prov in provides:
                prov = string.strip(prov)
-               if not provider_hash.has_key(prov):
+               if prov not in provider_hash:
                     provider_hash[prov] = []
                provider_hash[prov].append(p)
 
 def find_package(name):
-     if provider_hash.has_key(name):
+     if name in provider_hash:
           return provider_hash[name]
      return None
 
@@ -68,7 +68,7 @@ def recurse(pkg):
                newpkgs = find_package(dep)
                if newpkgs:
                     for newpkg in newpkgs:
-	                 if required.has_key(newpkg.package):
+	                 if newpkg.package in required:
 			      return
                     recurse(newpkgs[0])
                else:
@@ -82,6 +82,6 @@ for root in remaining_args:
      for p in pkgs:
           recurse(p)
      
-for pkg in required.keys():
+for pkg in list(required.keys()):
      print(pkg)
 
diff --git a/opkg-update-index b/opkg-update-index
index d9c9b43..1b05875 100755
--- a/opkg-update-index
+++ b/opkg-update-index
@@ -15,7 +15,7 @@ packages = opkg.Packages()
 
 packages.read_packages_file(pkg_dir + '/Packages')
 
-names = packages.packages.keys()
+names = list(packages.packages.keys())
 
 packages.add_package(opkg.Package(pkg_filename))
 
diff --git a/opkg.py b/opkg.py
index 54060c3..75ad3d3 100644
--- a/opkg.py
+++ b/opkg.py
@@ -217,7 +217,7 @@ class Package:
                     self.size = int(value)
                 elif name == 'md5sum':
                     self.md5 = value
-                elif self.__dict__.has_key(name):
+                elif name in self.__dict__:
                     self.__dict__[name] = value
                 else:
                     print("Lost field %s, %s" % (name,value))
@@ -447,7 +447,7 @@ class Packages:
         package = pkg.package
         arch = pkg.architecture
         name = ("%s:%s" % (package, arch))
-        if (not self.packages.has_key(name)):
+        if (name not in self.packages):
             self.packages[name] = pkg
         
         if pkg.compare_version(self.packages[name]) >= 0:
@@ -470,14 +470,14 @@ class Packages:
 
     def write_packages_file(self, fn):
         f = open(fn, "w")
-        names = self.packages.keys()
+        names = list(self.packages.keys())
         names.sort()
         for name in names:
             f.write(self.packages[name].__repr__())
         return    
 
     def keys(self):
-        return self.packages.keys()
+        return list(self.packages.keys())
 
     def __getitem__(self, key):
         return self.packages[key]
-- 
1.7.8.5

[opkg-utils][PATCH 12/22] 2to3: exception handling fixes

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 arfile.py       |    4 ++--
 opkg-make-index |    4 ++--
 opkg.py         |    2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/arfile.py b/arfile.py
index 9149387..3f797a8 100644
--- a/arfile.py
+++ b/arfile.py
@@ -54,11 +54,11 @@ class ArFile:
             return FileSection(self.f, self.directory[fname][-1], int(self.directory[fname][5]))
 
         if self.directoryRead:
-            raise IOError, (2, "AR member not found: " + fname)
+            raise IOError("AR member not found: " + fname)
 
         f = self._scan(fname)
         if f == None:
-            raise IOError, (2, "AR member not found: " + fname)
+            raise IOError("AR member not found: " + fname)
         return f
 
 
diff --git a/opkg-make-index b/opkg-make-index
index 2fc8a69..997f286 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -198,10 +198,10 @@ names.sort()
 for name in names:
      try:
           fnlist = packages[name].get_file_list()
-     except OSError, e:
+     except OSError as e:
           sys.stderr.write("Package %s disappeared on us!\n" % (name))
           continue
-     except IOError, e:
+     except IOError as e:
           sys.stderr.write("Package %s disappeared on us!\n" % (name))
           continue
      for fn in fnlist:
diff --git a/opkg.py b/opkg.py
index 75ad3d3..ae31794 100644
--- a/opkg.py
+++ b/opkg.py
@@ -176,7 +176,7 @@ class Package:
         elif name == 'size':
             return self._get_file_size()
         else:
-            raise AttributeError, name
+            raise AttributeError(name)
 
     def _computeFileMD5(self):
         # compute the MD5.
-- 
1.7.8.5

[opkg-utils][PATCH 13/22] opkg-make-index: show OSError/IOError

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg-make-index |   20 ++++++++++----------
 1 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/opkg-make-index b/opkg-make-index
index 997f286..e0844a7 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -131,11 +131,11 @@ for filename in files:
                to_morgue(basename)
           if opt_s:
                print(filename)
-  except OSError:
-      sys.stderr.write("Package %s disappeared on us!\n" % (filename))
+  except OSError as e:
+      sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (filename, e))
       continue
-  except IOError:
-      sys.stderr.write("Package %s disappeared on us!\n" % (filename))
+  except IOError as e:
+      sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (filename, e))
       continue
 
 pkgsStampsFile = open(stamplist_filename, "w")
@@ -173,11 +173,11 @@ for name in names:
      if (verbose):
           sys.stderr.write("Writing info for package %s\n" % (pkg.package,))
      print(pkg)
-  except OSError:
-      sys.stderr.write("Package %s disappeared on us!\n" % (name))
+  except OSError as e:
+      sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e))
       continue
-  except IOError:
-      sys.stderr.write("Package %s disappeared on us!\n" % (name))
+  except IOError as e:
+      sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e))
       continue
 
 if packages_filename:
@@ -199,10 +199,10 @@ for name in names:
      try:
           fnlist = packages[name].get_file_list()
      except OSError as e:
-          sys.stderr.write("Package %s disappeared on us!\n" % (name))
+          sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e))
           continue
      except IOError as e:
-          sys.stderr.write("Package %s disappeared on us!\n" % (name))
+          sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e))
           continue
      for fn in fnlist:
           (h,t) = os.path.split(fn)
-- 
1.7.8.5

[opkg-utils][PATCH 14/22] opkg.py: use string funtcions directly on string variable

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg-make-index |    3 +--
 opkg-show-deps  |   11 +++++------
 opkg.py         |    5 ++---
 3 files changed, 8 insertions(+), 11 deletions(-)

diff --git a/opkg-make-index b/opkg-make-index
index e0844a7..898a2f6 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -5,7 +5,6 @@ from glob import glob
 import subprocess
 import opkg
 import getopt
-import string
 import re
 
 verbose = 0
@@ -158,7 +157,7 @@ for name in names:
   try:
      pkg = packages.packages[name]
      if locales_dir and pkg.depends:
-         depends = string.split(pkg.depends, ',')
+         depends = pkg.depends.split(',')
          locale = None
          for d in depends:
               m = re.match('.*virtual-locale-([a-zA-Z]+).*', d)
diff --git a/opkg-show-deps b/opkg-show-deps
index 3f5899d..d188aad 100755
--- a/opkg-show-deps
+++ b/opkg-show-deps
@@ -4,7 +4,6 @@ import sys, os, posixpath
 from glob import glob
 import opkg
 import getopt
-import string
 import re
 
 verbose = 0
@@ -32,11 +31,11 @@ provider_hash = {}
 
 def split_list(str):
      r = []
-     l = string.split(str, ",")
+     l = str.split(",")
      for i in l:
-          ll = string.split(i, "|")
+          ll = i.split("|")
           for ii in ll:
-               ii = string.strip(ii)
+               ii = ii.strip()
                r.append(ii)
      return r
 
@@ -46,9 +45,9 @@ for i in list(packages.packages.keys()):
           provider_hash[p.package] = []
      provider_hash[p.package].append(p)
      if p.provides:
-          provides = string.split(p.provides, ",")
+          provides = p.provides.split(",")
           for prov in provides:
-               prov = string.strip(prov)
+               prov = prov.strip()
                if prov not in provider_hash:
                     provider_hash[prov] = []
                provider_hash[prov].append(p)
diff --git a/opkg.py b/opkg.py
index ae31794..856e057 100644
--- a/opkg.py
+++ b/opkg.py
@@ -38,7 +38,6 @@ import sys
 import glob
 import hashlib
 import re
-import string
 import subprocess
 from stat import ST_SIZE
 import arfile
@@ -203,10 +202,10 @@ class Package:
         line = control.readline()
         while 1:
             if not line: break
-            line = string.rstrip(line)
+            line = line.rstrip()
             lineparts = re.match(r'([\w-]*?):\s*(.*)', line)
             if lineparts:
-                name = string.lower(lineparts.group(1))
+                name = lineparts.group(1).lower()
                 value = lineparts.group(2)
                 while 1:
                     line = control.readline()
-- 
1.7.8.5

[opkg-utils][PATCH 15/22] opkg.py: catch TypeError when reading control file, to show which one is failing

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg.py |   13 ++++++++++---
 1 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/opkg.py b/opkg.py
index 856e057..299e84e 100644
--- a/opkg.py
+++ b/opkg.py
@@ -161,7 +161,10 @@ class Package:
                 control = tarf.extractfile("control")
             except KeyError:
                 control = tarf.extractfile("./control")
-            self.read_control(control)
+            try:
+                self.read_control(control)
+            except TypeError as e:
+                sys.stderr.write("Cannot read control file '%s' - %s\n" % (fn, e))
             control.close()
 
         self.scratch_dir = None
@@ -457,9 +460,13 @@ class Packages:
 
     def read_packages_file(self, fn):
         f = open(fn, "r")
-        while 1:
+        while True:
             pkg = Package()
-            pkg.read_control(f)
+            try:
+                pkg.read_control(f)
+            except TypeError as e:
+                sys.stderr.write("Cannot read control file '%s' - %s\n" % (fn, e))
+                continue
             if pkg.get_package():
                 self.add_package(pkg)
             else:
-- 
1.7.8.5

[opkg-utils][PATCH 16/22] arfile: fix test

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 arfile.py |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/arfile.py b/arfile.py
index 3f797a8..88ace46 100644
--- a/arfile.py
+++ b/arfile.py
@@ -131,5 +131,8 @@ if __name__ == "__main__":
         tarf = tarfile.open("control.tar.gz", "r", tarStream)
         #tarf.list()
 
-        f2 = tarf.extractfile("control")
+        try:
+            f2 = tarf.extractfile("control")
+        except KeyError:
+            f2 = tarf.extractfile("./control")
         print(f2.read())
-- 
1.7.8.5

[opkg-utils][PATCH 17/22] opkg-make-index: don't use stdout for Packages output and use __str__ instead of __repr__

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg-make-index |   17 +++++++++--------
 opkg.py         |    4 ++--
 2 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/opkg-make-index b/opkg-make-index
index 898a2f6..02c425f 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -148,9 +148,8 @@ if opt_s:
 if verbose:
      sys.stderr.write("Generating Packages file\n")
 if packages_filename:
-     old_stdout = sys.stdout
      tmp_packages_filename = ("%s.%d" % (packages_filename, os.getpid()))
-     sys.stdout = open(tmp_packages_filename, "w")
+     pkgsFile = open(tmp_packages_filename, "w")
 names = list(packages.packages.keys())
 names.sort()
 for name in names:
@@ -171,7 +170,10 @@ for name in names:
               continue
      if (verbose):
           sys.stderr.write("Writing info for package %s\n" % (pkg.package,))
-     print(pkg)
+     if packages_filename:
+          pkgsFile.write(str(pkg))
+     else:
+          print(pkg)
   except OSError as e:
       sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e))
       continue
@@ -180,8 +182,7 @@ for name in names:
       continue
 
 if packages_filename:
-     sys.stdout.close()
-     sys.stdout = old_stdout
+     pkgsFile.close()
      gzip_filename = ("%s.gz" % packages_filename)
      tmp_gzip_filename = ("%s.%d" % (gzip_filename, os.getpid()))
      gzip_cmd = "gzip -9c < %s > %s" % (tmp_packages_filename, tmp_gzip_filename)
@@ -211,12 +212,12 @@ for name in names:
 
 if filelist_filename:
      tmp_filelist_filename = ("%s.%d" % (filelist_filename, os.getpid()))
-     sys.stdout = open(tmp_filelist_filename, "w")
+     f = open(tmp_filelist_filename, "w")
      names = list(files.keys())
      names.sort()
      for name in names:
-          print((name,files[name]))
-     sys.stdout.close()
+          f.write("%s %s\n" % (name, files[name]))
+     f.close()
      if posixpath.exists(filelist_filename):
           os.unlink(filelist_filename)
      os.rename(tmp_filelist_filename, filelist_filename)
diff --git a/opkg.py b/opkg.py
index 299e84e..84583be 100644
--- a/opkg.py
+++ b/opkg.py
@@ -402,7 +402,7 @@ class Package:
             ref.parsed_version = parse_version(ref.version)
         return self.parsed_version.compare(ref.parsed_version)
 
-    def __repr__(self):
+    def __str__(self):
         out = ""
 
         # XXX - Some checks need to be made, and some exceptions
@@ -506,7 +506,7 @@ if __name__ == "__main__":
     package.set_description("A test of the APIs.")
 
     print("<")
-    sys.stdout.write(package)
+    sys.stdout.write(str(package))
     print(">")
 
     package.write_package("/tmp")
-- 
1.7.8.5

[opkg-utils][PATCH 18/22] opkg.py: cast lines from controlfile as string

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg.py |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/opkg.py b/opkg.py
index 84583be..6c39616 100644
--- a/opkg.py
+++ b/opkg.py
@@ -206,7 +206,7 @@ class Package:
         while 1:
             if not line: break
             line = line.rstrip()
-            lineparts = re.match(r'([\w-]*?):\s*(.*)', line)
+            lineparts = re.match(r'([\w-]*?):\s*(.*)', str(line))
             if lineparts:
                 name = lineparts.group(1).lower()
                 value = lineparts.group(2)
-- 
1.7.8.5

[opkg-utils][PATCH 19/22] opkg.py: computeFileMD5 only when we have fn, otherwise fails to read None file

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg.py |   19 +++++++++++--------
 1 files changed, 11 insertions(+), 8 deletions(-)

diff --git a/opkg.py b/opkg.py
index 6c39616..9daff9e 100644
--- a/opkg.py
+++ b/opkg.py
@@ -182,14 +182,17 @@ class Package:
 
     def _computeFileMD5(self):
         # compute the MD5.
-        f = open(self.fn, "rb")
-        sum = hashlib.md5()
-        while True:
-            data = f.read(1024)
-            if not data: break
-            sum.update(data)
-        f.close()
-        self.md5 = sum.hexdigest()
+        if not self.fn:
+            self.md5 = 'Unknown'
+        else:
+            f = open(self.fn, "rb")
+            sum = hashlib.md5()
+            while True:
+               data = f.read(1024)
+               if not data: break
+               sum.update(data)
+            f.close()
+            self.md5 = sum.hexdigest()
 
     def _get_file_size(self):
         if not self.fn:
-- 
1.7.8.5

[opkg-utils][PATCH 20/22] opkg.py: fix write_package when called from main test

[Martin Jansa]

* there is no self.meta_dir before self._setup_scratch_area
* control file needs to be closed before packing it with tar, otherwise it could be empty

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg.py |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/opkg.py b/opkg.py
index 9daff9e..ad3a695 100644
--- a/opkg.py
+++ b/opkg.py
@@ -344,11 +344,11 @@ class Package:
         return self.file_ext_opk
 
     def write_package(self, dirname):
-        buf = self.render_control()
+        self._setup_scratch_area()
         file = open("%s/control" % self.meta_dir, 'w')
-        file.write(buf)
+        file.write(str(self))
+        file.close()
 
-        self._setup_scratch_area()
         cmd = "cd %s ; tar cvfz %s/control.tar.gz control" % (self.meta_dir,
                                                               self.scratch_dir)
 
-- 
1.7.8.5

[opkg-utils][PATCH 21/22] opkg.py: use textwrap for description writing

[Martin Jansa]

* description field in oe-core sometimes has extra line-feeds and is long
* extra line-feeds breaks read_control, because empty line means next Package
* long descriptions should be wrapped and properly indented, so they are parsed back properly

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg.py |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/opkg.py b/opkg.py
index ad3a695..2294160 100644
--- a/opkg.py
+++ b/opkg.py
@@ -42,6 +42,7 @@ import subprocess
 from stat import ST_SIZE
 import arfile
 import tarfile
+import textwrap
 
 class Version:
     """A class for holding parsed package version information."""
@@ -427,7 +428,9 @@ class Package:
         if self.installed_size: out = out + "InstalledSize: %d\n" % int(self.installed_size)
         if self.filename: out = out + "Filename: %s\n" % (self.filename)
         if self.source: out = out + "Source: %s\n" % (self.source)
-        if self.description: out = out + "Description: %s\n" % (self.description)
+        if self.description:
+            printable_description = textwrap.dedent(self.description).strip()
+            out = out + "Description: %s\n" % textwrap.fill(printable_description, width=74, initial_indent=' ', subsequent_indent=' ')
         if self.oe: out = out + "OE: %s\n" % (self.oe)
         if self.homepage: out = out + "HomePage: %s\n" % (self.homepage)
         if self.license: out = out + "License: %s\n" % (self.license)
-- 
1.7.8.5

[opkg-utils][PATCH 22/22] opkg.py: improve test so it prints temporary control file with long description and then reads it back

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg.py |   12 +++++++++++-
 1 files changed, 11 insertions(+), 1 deletions(-)

diff --git a/opkg.py b/opkg.py
index 2294160..27cbc2a 100644
--- a/opkg.py
+++ b/opkg.py
@@ -509,11 +509,21 @@ if __name__ == "__main__":
     package.set_architecture("arm")
     package.set_maintainer("Testing <testing@testing.testing>")
     package.set_depends("libc")
-    package.set_description("A test of the APIs.")
+    package.set_description("A test of the APIs. And very long descriptions so often used in oe-core\nfoo\n\n\nbar")
 
     print("<")
     sys.stdout.write(str(package))
     print(">")
+    f = open("/tmp/control", "w")
+    f.write(str(package))
+    f.close()
+
+    f = open("/tmp/control", "r")
+    package2 = Package()
+    package2.read_control(f)
+    print("<")
+    sys.stdout.write(str(package2))
+    print(">")
 
     package.write_package("/tmp")
 
-- 
1.7.8.5

[opkg-utils][PATCH 23/23] opkg-make-index: generate complete filelist

[martin.jansa]

From: Martin Jansa <Martin.Jansa@gmail.com>

* when '-l Packages.filelist' option is used together with '-r Packages.old',
  then only 'new' packages are processed to create Packages.filelist
  packages found in Packages.old doesn't have fn set so get_file_list()
  was returning empty
* now added get_file_list_dir() looks for filename in pkg_dir and if it
  finds correct .ipk file it will use that to read its filelist, but it
  also means that it will always unpack *all* Packages - very slow.
* it would be nice to add new param for Packages.filelist.old and then
  filter filelist for packages used from Packages.old and merge it
  together with new Packages.filelist, but that's more difficult because
  of files structure.

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 opkg-make-index |   64 +++++++++++++++++++++++++++++-------------------------
 opkg.py         |   16 +++++++++++++
 2 files changed, 50 insertions(+), 30 deletions(-)

diff --git a/opkg-make-index b/opkg-make-index
index 02c425f..4425107 100755
--- a/opkg-make-index
+++ b/opkg-make-index
@@ -190,34 +190,38 @@ if packages_filename:
      os.rename(tmp_packages_filename, packages_filename)
      os.rename(tmp_gzip_filename, gzip_filename)
 
-if verbose:
-     sys.stderr.write("Generate Packages.filelist file\n")
-files = {}
-names = list(packages.packages.keys())
-names.sort()
-for name in names:
-     try:
-          fnlist = packages[name].get_file_list()
-     except OSError as e:
-          sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e))
-          continue
-     except IOError as e:
-          sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e))
-          continue
-     for fn in fnlist:
-          (h,t) = os.path.split(fn)
-          if not t: continue
-          if t not in files: files[t] = name+':'+fn
-          else: files[t] = files[t] + ',' + name+':'+fn
-
 if filelist_filename:
-     tmp_filelist_filename = ("%s.%d" % (filelist_filename, os.getpid()))
-     f = open(tmp_filelist_filename, "w")
-     names = list(files.keys())
-     names.sort()
-     for name in names:
-          f.write("%s %s\n" % (name, files[name]))
-     f.close()
-     if posixpath.exists(filelist_filename):
-          os.unlink(filelist_filename)
-     os.rename(tmp_filelist_filename, filelist_filename)
+    if verbose:
+        sys.stderr.write("Generate Packages.filelist file\n")
+    files = {}
+    names = list(packages.packages.keys())
+    names.sort()
+    for name in names:
+        try:
+            if verbose:
+                sys.stderr.write("Reading filelist for package '%s'\n" % name)
+#            sys.stderr.write("Package for name '%s':\n'%s'\n" % (name, packages[name]))
+            fnlist = packages[name].get_file_list_dir(pkg_dir)
+#            sys.stderr.write("Filelist for package '%s': '%s'\n" % (name, fnlist))
+        except OSError as e:
+            sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e))
+            continue
+        except IOError as e:
+            sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e))
+            continue
+        for fn in fnlist:
+            (h,t) = os.path.split(fn)
+            if not t: continue
+            if t not in files: files[t] = name+':'+fn
+        else: files[t] = files[t] + ',' + name+':'+fn
+
+    tmp_filelist_filename = ("%s.%d" % (filelist_filename, os.getpid()))
+    f = open(tmp_filelist_filename, "w")
+    names = list(files.keys())
+    names.sort()
+    for name in names:
+         f.write("%s %s\n" % (name, files[name]))
+    f.close()
+    if posixpath.exists(filelist_filename):
+        os.unlink(filelist_filename)
+    os.rename(tmp_filelist_filename, filelist_filename)
diff --git a/opkg.py b/opkg.py
index 27cbc2a..707a882 100644
--- a/opkg.py
+++ b/opkg.py
@@ -325,8 +325,24 @@ class Package:
     def get_license(self, license):
         return self.license
 
+    def get_file_list_dir(self, directory):
+        if not self.fn:
+            try:
+                cmd = "find %s -name %s | head -n 1" % (directory, self.filename)
+                rc = subprocess.check_output(cmd, shell=True)
+                newfn = str(rc).split()[0]
+#                sys.stderr.write("Package '%s' with empty fn and filename is '%s' was found in '%s', updating fn\n" % (self.package, self.filename, newfn))
+                self.fn = newfn
+            except OSError as e:
+                sys.stderr.write("Cannot find current fn for package '%s' filename '%s' in dir '%s'\n(%s)\n" % (self.package, self.filename, directory, e))
+            except IOError as e:
+                sys.stderr.write("Cannot find current fn for package '%s' filename '%s' in dir '%s'\n(%s)\n" % (self.package, self.filename, directory, e))
+        return self.get_file_list()
+
+
     def get_file_list(self):
         if not self.fn:
+            sys.stderr.write("Package '%s' has empty fn returning empty filelist\n" % (self.package))
             return []
         f = open(self.fn, "rb")
         ar = arfile.ArFile(f, self.fn)
-- 
1.7.8.5

Re: [opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse

[Saul Wold]

Martin,

Just wanted to let you know that this was received, I am holding off 
doing anything about it right now until we get through the current 
release process, this will be build and tested with early in the 1.3 as 
appropriate.

Thanks
	Sau!


On 03/29/2012 03:21 PM, Martin Jansa wrote:
> First 5 patches are taken from oe-core.
>
> Then there are some cleanups and fixes for issues I've found while looking
> for cause of very long package-index calls.
>
> Some patches uses compatible changes from 2to3 (tested with python2.7), but
> python3 support is not 100% complete, but were closer.
>
> And in the end I've found the cause.. printing long fields (e. g. description)
> with opkg.py was producing empty lines in Packages file (which are Packages entry
> separators) so only first entries till first empty line were processed from old
> Packages file and the rest was always extracted from control files in packaged .ipk.
>
> And this also fixes Packages.filelist generation.
>
> I don't use python very much, so please review.
>
> And the speedup? Almost 60 times :)
> before:
> real    20m10.484s
> user    18m32.335s
> sys     0m11.885s
>
> after
> real    0m24.492s
> user    0m23.748s
> sys     0m0.565s
>
> The following changes since commit 002d29bc605d7c2d02e4cf20a43c5277c15f5597:
>
>    [opkg-utils] fix install fail problem  Thanks for khorben's patch :-)  https://docs.openmoko.org/trac/attachment/ticket/2072/patch-opkg-utils_Makefile_install_path.diff (2008-11-03 03:59:59 +0000)
>
> are available in the git repository at:
>    git://github.com/shr-project/opkg-utils jansa/pull
>    https://github.com/shr-project/opkg-utils/tree/jansa/pull
>
> Christopher Larson (1):
>    Use python via the PATH, rather than hardcoding /usr/bin/python
>
> Enrico Scholz (1):
>    opkg-make-index: convert mtime to int before comparing it
>
> Khem Raj (1):
>    opkg.py: Add knowledge about License field in ipk headers
>
> Martin Jansa (17):
>    opkg.py: use hashlib instead of old md5 module
>    opkg.py, arfile.py: report which file has wrong format and use the
>      same test
>    arfile: decode read lines as ascii string before getting fields from
>      it
>    2to3: use subprocess instead of commands
>    2to3: print fixes
>    2to3: dictionary fixes
>    2to3: exception handling fixes
>    opkg-make-index: show OSError/IOError
>    opkg.py: use string funtcions directly on string variable
>    opkg.py: catch TypeError when reading control file, to show which one
>      is failing
>    arfile: fix test
>    opkg-make-index: don't use stdout for Packages output and use __str__
>      instead of __repr__
>    opkg.py: cast lines from controlfile as string
>    opkg.py: computeFileMD5 only when we have fn, otherwise fails to read
>      None file
>    opkg.py: fix write_package when called from main test
>    opkg.py: use textwrap for description writing
>    opkg.py: improve test so it prints temporary control file with long
>      description and then reads it back
>
> Richard Purdie (1):
>    opkg-make-index: don't error out when some package disappears
>
> Scott Anderson (1):
>    arfile.py: handle six digit UIDs
>
>   arfile.py            |   55 +++++++++++++---------
>   makePackage          |    2 +-
>   opkg-compare-indexes |   22 ++++----
>   opkg-list-fields     |    4 +-
>   opkg-make-index      |   74 +++++++++++++++++++----------
>   opkg-show-deps       |   28 +++++------
>   opkg-unbuild         |    4 +-
>   opkg-update-index    |    5 +-
>   opkg.py              |  125 +++++++++++++++++++++++++++++--------------------
>   9 files changed, 186 insertions(+), 133 deletions(-)
>

Re: [opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 4461 bytes --]

On Thu, Apr 05, 2012 at 04:54:50PM -0700, Saul Wold wrote:
> 
> Martin,
> 
> Just wanted to let you know that this was received, I am holding off 
> doing anything about it right now until we get through the current 
> release process, this will be build and tested with early in the 1.3 as 
> appropriate.

Hi,

can you please apply at least first patchset (just import of patches we
already had), someone wanted to switch opkg-utils in oe-classic from svn
to this git repo, so it would be nice to just use correct SRCREV instead
of importing those local patches.

Or import everything as it won't be used until we change SRCREV in
recipe and we'll change SRCREV in 1.3 cycle.

Thanks!

> 
> Thanks
> 	Sau!
> 
> 
> On 03/29/2012 03:21 PM, Martin Jansa wrote:
> > First 5 patches are taken from oe-core.
> >
> > Then there are some cleanups and fixes for issues I've found while looking
> > for cause of very long package-index calls.
> >
> > Some patches uses compatible changes from 2to3 (tested with python2.7), but
> > python3 support is not 100% complete, but were closer.
> >
> > And in the end I've found the cause.. printing long fields (e. g. description)
> > with opkg.py was producing empty lines in Packages file (which are Packages entry
> > separators) so only first entries till first empty line were processed from old
> > Packages file and the rest was always extracted from control files in packaged .ipk.
> >
> > And this also fixes Packages.filelist generation.
> >
> > I don't use python very much, so please review.
> >
> > And the speedup? Almost 60 times :)
> > before:
> > real    20m10.484s
> > user    18m32.335s
> > sys     0m11.885s
> >
> > after
> > real    0m24.492s
> > user    0m23.748s
> > sys     0m0.565s
> >
> > The following changes since commit 002d29bc605d7c2d02e4cf20a43c5277c15f5597:
> >
> >    [opkg-utils] fix install fail problem  Thanks for khorben's patch :-)  https://docs.openmoko.org/trac/attachment/ticket/2072/patch-opkg-utils_Makefile_install_path.diff (2008-11-03 03:59:59 +0000)
> >
> > are available in the git repository at:
> >    git://github.com/shr-project/opkg-utils jansa/pull
> >    https://github.com/shr-project/opkg-utils/tree/jansa/pull
> >
> > Christopher Larson (1):
> >    Use python via the PATH, rather than hardcoding /usr/bin/python
> >
> > Enrico Scholz (1):
> >    opkg-make-index: convert mtime to int before comparing it
> >
> > Khem Raj (1):
> >    opkg.py: Add knowledge about License field in ipk headers
> >
> > Martin Jansa (17):
> >    opkg.py: use hashlib instead of old md5 module
> >    opkg.py, arfile.py: report which file has wrong format and use the
> >      same test
> >    arfile: decode read lines as ascii string before getting fields from
> >      it
> >    2to3: use subprocess instead of commands
> >    2to3: print fixes
> >    2to3: dictionary fixes
> >    2to3: exception handling fixes
> >    opkg-make-index: show OSError/IOError
> >    opkg.py: use string funtcions directly on string variable
> >    opkg.py: catch TypeError when reading control file, to show which one
> >      is failing
> >    arfile: fix test
> >    opkg-make-index: don't use stdout for Packages output and use __str__
> >      instead of __repr__
> >    opkg.py: cast lines from controlfile as string
> >    opkg.py: computeFileMD5 only when we have fn, otherwise fails to read
> >      None file
> >    opkg.py: fix write_package when called from main test
> >    opkg.py: use textwrap for description writing
> >    opkg.py: improve test so it prints temporary control file with long
> >      description and then reads it back
> >
> > Richard Purdie (1):
> >    opkg-make-index: don't error out when some package disappears
> >
> > Scott Anderson (1):
> >    arfile.py: handle six digit UIDs
> >
> >   arfile.py            |   55 +++++++++++++---------
> >   makePackage          |    2 +-
> >   opkg-compare-indexes |   22 ++++----
> >   opkg-list-fields     |    4 +-
> >   opkg-make-index      |   74 +++++++++++++++++++----------
> >   opkg-show-deps       |   28 +++++------
> >   opkg-unbuild         |    4 +-
> >   opkg-update-index    |    5 +-
> >   opkg.py              |  125 +++++++++++++++++++++++++++++--------------------
> >   9 files changed, 186 insertions(+), 133 deletions(-)
> >

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse

[Richard Purdie]

On Fri, 2012-03-30 at 00:21 +0200, Martin Jansa wrote:
> First 5 patches are taken from oe-core.
> 
> Then there are some cleanups and fixes for issues I've found while looking 
> for cause of very long package-index calls.
> 
> Some patches uses compatible changes from 2to3 (tested with python2.7), but
> python3 support is not 100% complete, but were closer.
> 
> And in the end I've found the cause.. printing long fields (e. g. description)
> with opkg.py was producing empty lines in Packages file (which are Packages entry 
> separators) so only first entries till first empty line were processed from old
> Packages file and the rest was always extracted from control files in packaged .ipk.
> 
> And this also fixes Packages.filelist generation.
> 
> I don't use python very much, so please review.
> 
> And the speedup? Almost 60 times :)
> before:
> real    20m10.484s
> user    18m32.335s
> sys     0m11.885s
> 
> after
> real    0m24.492s
> user    0m23.748s
> sys     0m0.565s
> 
> The following changes since commit 002d29bc605d7c2d02e4cf20a43c5277c15f5597:
> 
>   [opkg-utils] fix install fail problem  Thanks for khorben's patch :-)  https://docs.openmoko.org/trac/attachment/ticket/2072/patch-opkg-utils_Makefile_install_path.diff (2008-11-03 03:59:59 +0000)
> 
> are available in the git repository at:
>   git://github.com/shr-project/opkg-utils jansa/pull
>   https://github.com/shr-project/opkg-utils/tree/jansa/pull
> 
> Christopher Larson (1):
>   Use python via the PATH, rather than hardcoding /usr/bin/python
> 
> Enrico Scholz (1):
>   opkg-make-index: convert mtime to int before comparing it
> 
> Khem Raj (1):
>   opkg.py: Add knowledge about License field in ipk headers
> 
> Martin Jansa (17):
>   opkg.py: use hashlib instead of old md5 module
>   opkg.py, arfile.py: report which file has wrong format and use the
>     same test
>   arfile: decode read lines as ascii string before getting fields from
>     it
>   2to3: use subprocess instead of commands
>   2to3: print fixes
>   2to3: dictionary fixes
>   2to3: exception handling fixes
>   opkg-make-index: show OSError/IOError
>   opkg.py: use string funtcions directly on string variable
>   opkg.py: catch TypeError when reading control file, to show which one
>     is failing
>   arfile: fix test
>   opkg-make-index: don't use stdout for Packages output and use __str__
>     instead of __repr__
>   opkg.py: cast lines from controlfile as string
>   opkg.py: computeFileMD5 only when we have fn, otherwise fails to read
>     None file
>   opkg.py: fix write_package when called from main test
>   opkg.py: use textwrap for description writing
>   opkg.py: improve test so it prints temporary control file with long
>     description and then reads it back
> 
> Richard Purdie (1):
>   opkg-make-index: don't error out when some package disappears
> 
> Scott Anderson (1):
>   arfile.py: handle six digit UIDs

I reviewed these and they all look like good improvements. I've merged
it into master, thanks!

Richard

Re: [opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse

[Koen Kooi]

Op 26 apr. 2012, om 12:43 heeft Richard Purdie het volgende geschreven:

> On Fri, 2012-03-30 at 00:21 +0200, Martin Jansa wrote:
>> First 5 patches are taken from oe-core.
>> 
>> Then there are some cleanups and fixes for issues I've found while looking 
>> for cause of very long package-index calls.
>> 
>> Some patches uses compatible changes from 2to3 (tested with python2.7), but
>> python3 support is not 100% complete, but were closer.
>> 
>> And in the end I've found the cause.. printing long fields (e. g. description)
>> with opkg.py was producing empty lines in Packages file (which are Packages entry 
>> separators) so only first entries till first empty line were processed from old
>> Packages file and the rest was always extracted from control files in packaged .ipk.
>> 
>> And this also fixes Packages.filelist generation.
>> 
>> I don't use python very much, so please review.
>> 
>> And the speedup? Almost 60 times :)

[..]

> I reviewed these and they all look like good improvements. I've merged
> it into master, thanks!

After some more time in oe-core master for testing we should seriously consider cherry-picking the SRCREV bump to the denzil branch.

regards,

Koen

Re: [opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1574 bytes --]

On Thu, Apr 26, 2012 at 06:21:45PM +0200, Koen Kooi wrote:
> 
> Op 26 apr. 2012, om 12:43 heeft Richard Purdie het volgende geschreven:
> 
> > On Fri, 2012-03-30 at 00:21 +0200, Martin Jansa wrote:
> >> First 5 patches are taken from oe-core.
> >> 
> >> Then there are some cleanups and fixes for issues I've found while looking 
> >> for cause of very long package-index calls.
> >> 
> >> Some patches uses compatible changes from 2to3 (tested with python2.7), but
> >> python3 support is not 100% complete, but were closer.
> >> 
> >> And in the end I've found the cause.. printing long fields (e. g. description)
> >> with opkg.py was producing empty lines in Packages file (which are Packages entry 
> >> separators) so only first entries till first empty line were processed from old
> >> Packages file and the rest was always extracted from control files in packaged .ipk.
> >> 
> >> And this also fixes Packages.filelist generation.
> >> 
> >> I don't use python very much, so please review.
> >> 
> >> And the speedup? Almost 60 times :)
> 
> [..]
> 
> > I reviewed these and they all look like good improvements. I've merged
> > it into master, thanks!
> 
> After some more time in oe-core master for testing we should seriously consider cherry-picking the SRCREV bump to the denzil branch.

Agreed, saves about 4 hours per day (13 images each with own package-index run 
and extra run for feed if built after iamges) on my buildhost for last month or so :)

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse

[Koen Kooi]

Op 26 apr. 2012, om 20:52 heeft Martin Jansa het volgende geschreven:

> On Thu, Apr 26, 2012 at 06:21:45PM +0200, Koen Kooi wrote:
>> 
>> Op 26 apr. 2012, om 12:43 heeft Richard Purdie het volgende geschreven:
>> 
>>> On Fri, 2012-03-30 at 00:21 +0200, Martin Jansa wrote:
>>>> First 5 patches are taken from oe-core.
>>>> 
>>>> Then there are some cleanups and fixes for issues I've found while looking 
>>>> for cause of very long package-index calls.
>>>> 
>>>> Some patches uses compatible changes from 2to3 (tested with python2.7), but
>>>> python3 support is not 100% complete, but were closer.
>>>> 
>>>> And in the end I've found the cause.. printing long fields (e. g. description)
>>>> with opkg.py was producing empty lines in Packages file (which are Packages entry 
>>>> separators) so only first entries till first empty line were processed from old
>>>> Packages file and the rest was always extracted from control files in packaged .ipk.
>>>> 
>>>> And this also fixes Packages.filelist generation.
>>>> 
>>>> I don't use python very much, so please review.
>>>> 
>>>> And the speedup? Almost 60 times :)
>> 
>> [..]
>> 
>>> I reviewed these and they all look like good improvements. I've merged
>>> it into master, thanks!
>> 
>> After some more time in oe-core master for testing we should seriously consider cherry-picking the SRCREV bump to the denzil branch.
> 
> Agreed, saves about 4 hours per day (13 images each with own package-index run 
> and extra run for feed if built after iamges) on my buildhost for last month or so :)

I merged in denzil-next locally and grabbed 26465 from patchwork. Let's see how much time it saves on my machine :)

Re: [opkg-utils][PATCH 00/22] Partially prepare for python3 and fix old Packages file reuse

[Koen Kooi]

Op 27 apr. 2012, om 08:39 heeft Koen Kooi het volgende geschreven:

> 
> Op 26 apr. 2012, om 20:52 heeft Martin Jansa het volgende geschreven:
> 
>> On Thu, Apr 26, 2012 at 06:21:45PM +0200, Koen Kooi wrote:
>>> 
>>> Op 26 apr. 2012, om 12:43 heeft Richard Purdie het volgende geschreven:
>>> 
>>>> On Fri, 2012-03-30 at 00:21 +0200, Martin Jansa wrote:
>>>>> First 5 patches are taken from oe-core.
>>>>> 
>>>>> Then there are some cleanups and fixes for issues I've found while looking 
>>>>> for cause of very long package-index calls.
>>>>> 
>>>>> Some patches uses compatible changes from 2to3 (tested with python2.7), but
>>>>> python3 support is not 100% complete, but were closer.
>>>>> 
>>>>> And in the end I've found the cause.. printing long fields (e. g. description)
>>>>> with opkg.py was producing empty lines in Packages file (which are Packages entry 
>>>>> separators) so only first entries till first empty line were processed from old
>>>>> Packages file and the rest was always extracted from control files in packaged .ipk.
>>>>> 
>>>>> And this also fixes Packages.filelist generation.
>>>>> 
>>>>> I don't use python very much, so please review.
>>>>> 
>>>>> And the speedup? Almost 60 times :)
>>> 
>>> [..]
>>> 
>>>> I reviewed these and they all look like good improvements. I've merged
>>>> it into master, thanks!
>>> 
>>> After some more time in oe-core master for testing we should seriously consider cherry-picking the SRCREV bump to the denzil branch.
>> 
>> Agreed, saves about 4 hours per day (13 images each with own package-index run 
>> and extra run for feed if built after iamges) on my buildhost for last month or so :)
> 
> I merged in denzil-next locally and grabbed 26465 from patchwork. Let's see how much time it saves on my machine :)

With a relatively small deploy area: 7 minutes saved per image build!