[PATCH 1/2] python3: fix create_manifest to handle pycache folders

[PATCH 1/2] python3: fix create_manifest to handle pycache folders

[Alejandro Enedino Hernandez Samaniego]

We have a couple of python modules that contain folders themselves,
for that reason they also contain a __pycache__ folder inside those
directories, since we include the whole folder in the manifest, the
pycache directories end up with the files and not the cache files.

This patch catches that and adds the directories to the correct
structure.

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
---
 .../python/python3/create_manifest3.py               |  5 ++++-
 .../python/python3/python3-manifest.json             | 20 +++++++++++---------
 2 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/meta/recipes-devtools/python/python3/create_manifest3.py b/meta/recipes-devtools/python/python3/create_manifest3.py
index ead27e9..43e95ce 100644
--- a/meta/recipes-devtools/python/python3/create_manifest3.py
+++ b/meta/recipes-devtools/python/python3/create_manifest3.py
@@ -186,7 +186,10 @@ for key in old_manifest:
         # Ignore folders, since we don't import those, difficult to handle multilib
         if isFolder(value):
             # Pass folders directly
-            new_manifest[key]['files'].append(value)
+            if isCached(value):
+                new_manifest[key]['cached'].append(value)
+            else:
+                new_manifest[key]['files'].append(value)
         # Ignore binaries, since we don't import those
         if '${bindir}' in value:
             # Pass it directly to the new manifest data structure
diff --git a/meta/recipes-devtools/python/python3/python3-manifest.json b/meta/recipes-devtools/python/python3/python3-manifest.json
index 95071c5..911be8e 100644
--- a/meta/recipes-devtools/python/python3/python3-manifest.json
+++ b/meta/recipes-devtools/python/python3/python3-manifest.json
@@ -737,21 +737,21 @@
             "${libdir}/python3.5/__pycache__/poplib.*.pyc",
             "${libdir}/python3.5/__pycache__/smtplib.*.pyc",
             "${libdir}/python3.5/__pycache__/telnetlib.*.pyc",
-            "${libdir}/python3.5/__pycache__/uuid.*.pyc"
+            "${libdir}/python3.5/__pycache__/uuid.*.pyc",
+            "${libdir}/python3.5/http/__pycache__",
+            "${libdir}/python3.5/urllib/__pycache__"
         ],
         "files": [
             "${libdir}/python3.5/base64.py",
             "${libdir}/python3.5/ftplib.py",
             "${libdir}/python3.5/hmac.py",
             "${libdir}/python3.5/http",
-            "${libdir}/python3.5/http/__pycache__",
             "${libdir}/python3.5/mimetypes.py",
             "${libdir}/python3.5/nntplib.py",
             "${libdir}/python3.5/poplib.py",
             "${libdir}/python3.5/smtplib.py",
             "${libdir}/python3.5/telnetlib.py",
             "${libdir}/python3.5/urllib",
-            "${libdir}/python3.5/urllib/__pycache__",
             "${libdir}/python3.5/uuid.py"
         ],
         "rdepends": [
@@ -1088,11 +1088,12 @@
         "summary": "Python typing support"
     },
     "unittest": {
-        "cached": [],
+        "cached": [
+            "${libdir}/python3.5/unittest/__pycache__"
+        ],
         "files": [
             "${libdir}/python3.5/unittest",
-            "${libdir}/python3.5/unittest/",
-            "${libdir}/python3.5/unittest/__pycache__"
+            "${libdir}/python3.5/unittest/"
         ],
         "rdepends": [
             "core",
@@ -1132,11 +1133,12 @@
         "summary": "Python basic XML support"
     },
     "xmlrpc": {
-        "cached": [],
-        "files": [
-            "${libdir}/python3.5/xmlrpc",
+        "cached": [
             "${libdir}/python3.5/xmlrpc/__pycache__"
         ],
+        "files": [
+            "${libdir}/python3.5/xmlrpc"
+        ],
         "rdepends": [
             "core",
             "xml"
--
2.7.4

This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

[PATCH 2/2] python3: Fix do_create_manifest for python3-sqlite3

[Alejandro Enedino Hernandez Samaniego]

Some of the sqlite3 files ended up in python3-misc incorrectly,
this is caused becuse we couldnt add the whole ${libdir}/python3/sqlite3
folder on the package because we also have another sqlite3-tests
package that needs to include another folder from that directory.

This patch not only fixes the do_create_manifest script to handle this
situation, but also patches the manifest (created using the script)
which also fixes a hiddn runtime dependency that we wouldn't have seen.

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
---
 .../python/python3/create_manifest3.py             | 43 ++++++++++++++--------
 .../python/python3/python3-manifest.json           | 15 +++++---
 2 files changed, 37 insertions(+), 21 deletions(-)

diff --git a/meta/recipes-devtools/python/python3/create_manifest3.py b/meta/recipes-devtools/python/python3/create_manifest3.py
index 43e95ce..212ddd4 100644
--- a/meta/recipes-devtools/python/python3/create_manifest3.py
+++ b/meta/recipes-devtools/python/python3/create_manifest3.py
@@ -124,7 +124,6 @@ for value in old_manifest['core']['files']:
   # Get module name , shouldnt be affected by libdir/bindir
   value = os.path.splitext(os.path.basename(os.path.normpath(value)))[0]

-
   # Launch separate task for each module for deterministic behavior
   # Each module will only import what is necessary for it to work in specific
   print ('Getting dependencies for module: %s' % value)
@@ -203,8 +202,20 @@ for key in old_manifest:
             if value not in new_manifest[key]['files']:
                 new_manifest[key]['files'].append(value)
             continue
+
         # Get module name , shouldnt be affected by libdir/bindir
-        value = os.path.splitext(os.path.basename(os.path.normpath(value)))[0]
+        # We need to check if the imported module comes from another (e.g. sqlite3.dump)
+        path,value = os.path.split(value)
+        path = os.path.basename(path)
+        value = os.path.splitext(os.path.basename(value))[0]
+
+        # If this condition is met, it means we need to import it from another module
+        # or its the folder itself (e.g. unittest)
+        if path == key:
+          if value:
+            value = path + '.' + value
+          else:
+            value = path

         # Launch separate task for each module for deterministic behavior
         # Each module will only import what is necessary for it to work in specific
@@ -292,19 +303,20 @@ for key in old_manifest:
                                        new_manifest[key]['rdepends'].append(newkey)
                                     break
                     else:
-                      # Debug
-                      print('Adding %s to %s FILES' % (item, key))
-                      # Since it wasnt found on another package, its not an RDEP, so add it to FILES for this package
-                      if isCached(item):
-                          new_manifest[key]['cached'].append(item)
-                      else:
-                          new_manifest[key]['files'].append(item)
-                      if item.endswith('*'):
-                          wildcards.append(item)
-                      if item not in allfiles:
-                          allfiles.append(item)
-                      else:
-                          repeated.append(item)
+                      # A module shouldn't contain itself (${libdir}/python3/sqlite3 shouldnt be on sqlite3 files)
+                      if os.path.basename(item) != key:
+                        print('Adding %s to %s FILES' % (item, key))
+                        # Since it wasnt found on another package, its not an RDEP, so add it to FILES for this package
+                        if isCached(item):
+                            new_manifest[key]['cached'].append(item)
+                        else:
+                            new_manifest[key]['files'].append(item)
+                        if item.endswith('*'):
+                            wildcards.append(item)
+                        if item not in allfiles:
+                            allfiles.append(item)
+                        else:
+                            repeated.append(item)

 print ('The following files are repeated (contained in more than one package), please check which package should get it:')
 print (repeated)
@@ -322,3 +334,4 @@ for key in new_manifest:
 # Create the manifest from the data structure that was built
 with open('python3-manifest.json.new','w') as outfile:
     json.dump(new_manifest,outfile,sort_keys=True, indent=4)
+    outfile.write("\n")
diff --git a/meta/recipes-devtools/python/python3/python3-manifest.json b/meta/recipes-devtools/python/python3/python3-manifest.json
index 911be8e..26fa613 100644
--- a/meta/recipes-devtools/python/python3/python3-manifest.json
+++ b/meta/recipes-devtools/python/python3/python3-manifest.json
@@ -591,8 +591,7 @@
         ],
         "rdepends": [
             "core",
-            "stringold",
-            "netserver"
+            "stringold"
         ],
         "summary": "Python logging support"
     },
@@ -978,14 +977,18 @@
     },
     "sqlite3": {
         "cached": [
-            "${libdir}/python3.5/sqlite3/__pycache__/*.pyc"
+            "${libdir}/python3.5/sqlite3/__pycache__",
+            "${libdir}/python3.5/sqlite3/__pycache__/dbapi2.*.pyc",
+            "${libdir}/python3.5/sqlite3/__pycache__/dump.*.pyc"
         ],
         "files": [
             "${libdir}/python3.5/lib-dynload/_sqlite3.*.so",
-            "${libdir}/python3.5/sqlite3/*.py"
+            "${libdir}/python3.5/sqlite3/dbapi2.py",
+            "${libdir}/python3.5/sqlite3/dump.py"
         ],
         "rdepends": [
-            "core"
+            "core",
+            "datetime"
         ],
         "summary": "Python Sqlite3 database support"
     },
@@ -1145,4 +1148,4 @@
         ],
         "summary": "Python XML-RPC support"
     }
-}
+}
\ No newline at end of file
--
2.7.4

This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.