[PATCH 0/4] Update cleanup-workdir

[PATCH 0/4] Update cleanup-workdir

[Kang Kai]

Hi,

This series of patches update the cleanup-workdir:
1 update the help text
2 update the way to check obsolete directories
3 only clean the directories related to current arch
4 replace module commands with subprocess

Regards,
Kai

The following changes since commit 5ed855d12cddf2de535c3f6d05d3dfe85d69d99d:

  openjade-native: Ensure we reautoconf the package (2012-06-12 16:34:30 +0100)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib kangkai/distro
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kangkai/distro

Kang Kai (4):
  cleanup-workdir: update the way to check obsolete dirs
  cleanup-workdir: replace commands with subprocess
  cleanup-workdir: only deal dirs related to current arch
  cleanup-workdir: update help text

 scripts/cleanup-workdir |  128 ++++++++++++++++++++++++++++++-----------------
 1 files changed, 82 insertions(+), 46 deletions(-)

-- 
1.7.5.4

[PATCH 1/4] cleanup-workdir: update the way to check obsolete dirs

[Kang Kai]

Update the way to check obsolete directories.

According to package and its version construct a list of all packages'
current build directory. If any directory under $WORKDIR/*/ is not in
the list will be removed.

At same time, all the files(vs. directory) under $WORKDIR and
$WORKDIR/*/ will be removed because they are not created by poky.

Signed-off-by: Kang Kai <kai.kang@windriver.com>
---
 scripts/cleanup-workdir |   59 ++++++++++++++++++----------------------------
 1 files changed, 23 insertions(+), 36 deletions(-)

diff --git a/scripts/cleanup-workdir b/scripts/cleanup-workdir
index b77e8c6..3739a00 100755
--- a/scripts/cleanup-workdir
+++ b/scripts/cleanup-workdir
@@ -22,7 +22,7 @@ import re
 import commands
 import shutil
 
-versions = {}
+pkg_cur_dirs = []
 obsolete_dirs = []
 parser = None
 
@@ -39,15 +39,6 @@ def parse_version(verstr):
     else:
         return epoch + '_' + elems[1]
 
-def parse_dir(match, pkgabsdir):
-    pkg_name = match.group(1)
-    pkg_version = match.group(2)
-    if pkg_name in versions:
-        if pkg_version != versions[pkg_name]:
-            obsolete_dirs.append(pkgabsdir)
-        return True
-    return False
-
 def main():
     global parser
     parser = optparse.OptionParser(
@@ -89,7 +80,7 @@ This script must be ran under BUILDDIR after source file \"oe-init-build-env\"."
             version = parse_version(elems[1])
         else:
             version = parse_version(elems[2])
-        versions[elems[0]] = version
+        pkg_cur_dirs.append(elems[0] + '-' + version)
 
     cmd = "bitbake -e | grep ^TMPDIR"
     (ret, output) = commands.getstatusoutput(cmd)
@@ -103,31 +94,27 @@ This script must be ran under BUILDDIR after source file \"oe-init-build-env\"."
         print "WORKDIR %s does NOT exist. Quit." % workdir
         return 1
 
-    for archdir in os.listdir(workdir):
-        archdir = os.path.join(workdir, archdir)
-        if not os.path.isdir(archdir):
-            pass
-
-        for pkgdir in sorted(os.listdir(archdir)):
-            pkgabsdir = os.path.join(archdir, pkgdir)
-            if not os.path.isdir(pkgabsdir):
-                pass
-
-            # parse the package directory names
-            # parse native/nativesdk packages first
-            match = re.match('(.*?-native.*?)-(.*)', pkgdir)
-            if match and parse_dir(match, pkgabsdir):
-                continue
-
-            # parse package names which ends with numbers such as 'glib-2.0'
-            match = re.match('(.*?-[\.\d]+)-(\d.*)', pkgdir)
-            if match and parse_dir(match, pkgabsdir):
-                continue
-
-            # other packages
-            match = re.match('(.*?)-(\d.*)', pkgdir)
-            if match and parse_dir(match, pkgabsdir):
-                continue
+    for workroot, dirs, files in os.walk(workdir):
+        # For the files, they should NOT exist in WORKDIR. Romve them.
+        for f in files:
+            obsolete_dirs.append(os.path.join(workroot, f))
+
+        for d in dirs:
+            for pkgroot, pkgdirs, filenames in os.walk(os.path.join(workroot, d)):
+                for f in filenames:
+                    obsolete_dirs.append(os.path.join(pkgroot, f))
+
+                for pkgdir in sorted(pkgdirs):
+                    if pkgdir not in pkg_cur_dirs:
+                        obsolete_dirs.append(os.path.join(pkgroot, pkgdir))
+
+                # just process the top dir of every package under tmp/work/*/,
+                # then jump out of the above os.walk()
+                break
+
+        # it is convenient to use os.walk() to get dirs and files at same time
+        # both of them have been dealed in the loop, so jump out
+        break
 
     for d in obsolete_dirs:
         print "Deleleting %s" % d
-- 
1.7.5.4

[PATCH 2/4] cleanup-workdir: replace commands with subprocess

[Kang Kai]

Use modules subprocess to run command instead of module commands.

Signed-off-by: Kang Kai <kai.kang@windriver.com>
---
 scripts/cleanup-workdir |   22 ++++++++++++----------
 1 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/scripts/cleanup-workdir b/scripts/cleanup-workdir
index 3739a00..1e9c56d 100755
--- a/scripts/cleanup-workdir
+++ b/scripts/cleanup-workdir
@@ -19,7 +19,7 @@ import os
 import sys
 import optparse
 import re
-import commands
+import subprocess
 import shutil
 
 pkg_cur_dirs = []
@@ -39,6 +39,14 @@ def parse_version(verstr):
     else:
         return epoch + '_' + elems[1]
 
+def run_command(cmd):
+    pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
+    output = pipe.communicate()[0]
+    if pipe.returncode != 0:
+        print "Execute command '%s' failed." % cmd
+        sys.exit(1)
+    return output
+
 def main():
     global parser
     parser = optparse.OptionParser(
@@ -49,7 +57,7 @@ This script must be ran under BUILDDIR after source file \"oe-init-build-env\"."
 
     options, args = parser.parse_args(sys.argv)
 
-    builddir = commands.getoutput('echo $BUILDDIR')
+    builddir = run_command('echo $BUILDDIR').strip()
     if len(builddir) == 0:
         err_quit("Please source file \"oe-init-build-env\" first.\n")
 
@@ -58,10 +66,7 @@ This script must be ran under BUILDDIR after source file \"oe-init-build-env\"."
 
     print 'Updating bitbake caches...'
     cmd = "bitbake -s"
-    (ret, output) = commands.getstatusoutput(cmd)
-    if ret != 0:
-        print "Execute 'bitbake -s' failed. Can't get packages' versions."
-        return 1
+    output = run_command(cmd)
 
     output = output.split('\n')
     index = 0
@@ -83,10 +88,7 @@ This script must be ran under BUILDDIR after source file \"oe-init-build-env\"."
         pkg_cur_dirs.append(elems[0] + '-' + version)
 
     cmd = "bitbake -e | grep ^TMPDIR"
-    (ret, output) = commands.getstatusoutput(cmd)
-    if ret != 0:
-        print "Execute 'bitbke -e' failed. Can't get TMPDIR."
-        return 1
+    output = run_command(cmd)
 
     tmpdir = output.split('"')[1]
     workdir = os.path.join(tmpdir, 'work')
-- 
1.7.5.4

[PATCH 3/4] cleanup-workdir: only deal dirs related to current arch

[Kang Kai]

Some users may build for different archs under same workdir, so they
don't want to clean the dirs not related to current arch.

Run command 'bitbake -e' with selected packages to get the dirs related
to current arch then clean them.

Update the way to get the WORKDIR by parsing the IMAGE_ROOTFS by the
way.

Signed-off-by: Kang Kai <kai.kang@windriver.com>
---
 scripts/cleanup-workdir |   54 ++++++++++++++++++++++++++++++++++++++++++----
 1 files changed, 49 insertions(+), 5 deletions(-)

diff --git a/scripts/cleanup-workdir b/scripts/cleanup-workdir
index 1e9c56d..156a259 100755
--- a/scripts/cleanup-workdir
+++ b/scripts/cleanup-workdir
@@ -47,6 +47,19 @@ def run_command(cmd):
         sys.exit(1)
     return output
 
+def get_cur_arch_dirs(workdir, arch_dirs):
+    pattern = workdir + '/(.*?)/'
+
+    # select thest 5 packages to get the dirs of current arch
+    pkgs = ['hicolor-icon-theme', 'base-files', 'acl-native', 'binutils-crosssdk', 'autoconf-nativesdk']
+
+    for pkg in pkgs:
+        cmd = "bitbake -e " + pkg + " | grep ^IMAGE_ROOTFS="
+        output = run_command(cmd)
+        output = output.split('"')[1]
+        m = re.match(pattern, output)
+        arch_dirs.append(m.group(1))
+
 def main():
     global parser
     parser = optparse.OptionParser(
@@ -87,21 +100,52 @@ This script must be ran under BUILDDIR after source file \"oe-init-build-env\"."
             version = parse_version(elems[2])
         pkg_cur_dirs.append(elems[0] + '-' + version)
 
-    cmd = "bitbake -e | grep ^TMPDIR"
+    cmd = "bitbake -e"
     output = run_command(cmd)
 
-    tmpdir = output.split('"')[1]
-    workdir = os.path.join(tmpdir, 'work')
-    if not os.path.exists(workdir):
-        print "WORKDIR %s does NOT exist. Quit." % workdir
+    tmpdir = None
+    image_rootfs = None
+    output = output.split('\n')
+    for line in output:
+        if tmpdir and image_rootfs:
+            break
+
+        if not tmpdir:
+            m = re.match('TMPDIR="(.*)"', line)
+            if m:
+                tmpdir = m.group(1)
+
+        if not image_rootfs:
+            m = re.match('IMAGE_ROOTFS="(.*)"', line)
+            if m:
+                image_rootfs = m.group(1)
+
+    # won't fail just in case
+    if not tmpdir or not image_rootfs:
+        print "Can't get TMPDIR or IMAGE_ROOTFS."
+        return 1
+
+    pattern = tmpdir + '/(.*?)/(.*?)/'
+    m = re.match(pattern, image_rootfs)
+    if not m:
+        print "Can't get WORKDIR."
         return 1
 
+    workdir = os.path.join(tmpdir, m.group(1))
+
+    # we only deal the dirs of current arch, total numbers of dirs are 6
+    cur_arch_dirs = [m.group(2)]
+    get_cur_arch_dirs(workdir, cur_arch_dirs)
+
     for workroot, dirs, files in os.walk(workdir):
         # For the files, they should NOT exist in WORKDIR. Romve them.
         for f in files:
             obsolete_dirs.append(os.path.join(workroot, f))
 
         for d in dirs:
+            if d not in cur_arch_dirs:
+                continue
+
             for pkgroot, pkgdirs, filenames in os.walk(os.path.join(workroot, d)):
                 for f in filenames:
                     obsolete_dirs.append(os.path.join(pkgroot, f))
-- 
1.7.5.4

[PATCH 4/4] cleanup-workdir: update help text

[Kang Kai]

Update the help text to tell user that the files and dirs under WORKDIR
which are not created by Yocto will be deleted.

Signed-off-by: Kang Kai <kai.kang@windriver.com>
---
 scripts/cleanup-workdir |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/cleanup-workdir b/scripts/cleanup-workdir
index 156a259..c748846 100755
--- a/scripts/cleanup-workdir
+++ b/scripts/cleanup-workdir
@@ -65,8 +65,11 @@ def main():
     parser = optparse.OptionParser(
         usage = """%prog
 
-Remove the obsolete packages' build directories in WORKDIR.
-This script must be ran under BUILDDIR after source file \"oe-init-build-env\".""")
+%prog removes the obsolete packages' build directories in WORKDIR.
+This script must be ran under BUILDDIR after source file \"oe-init-build-env\".
+
+Any file or directory under WORKDIR which is not created by Yocto
+will be deleted. Be CAUTIOUS.""")
 
     options, args = parser.parse_args(sys.argv)
 
-- 
1.7.5.4

Re: [PATCH 0/4] Update cleanup-workdir

[Saul Wold]

On 06/14/2012 10:56 PM, Kang Kai wrote:
> Hi,
>
> This series of patches update the cleanup-workdir:
> 1 update the help text
> 2 update the way to check obsolete directories
> 3 only clean the directories related to current arch
> 4 replace module commands with subprocess
>
> Regards,
> Kai
>
> The following changes since commit 5ed855d12cddf2de535c3f6d05d3dfe85d69d99d:
>
>    openjade-native: Ensure we reautoconf the package (2012-06-12 16:34:30 +0100)
>
> are available in the git repository at:
>    git://git.pokylinux.org/poky-contrib kangkai/distro
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kangkai/distro
>
> Kang Kai (4):
>    cleanup-workdir: update the way to check obsolete dirs
>    cleanup-workdir: replace commands with subprocess
>    cleanup-workdir: only deal dirs related to current arch
>    cleanup-workdir: update help text
>
>   scripts/cleanup-workdir |  128 ++++++++++++++++++++++++++++++-----------------
>   1 files changed, 82 insertions(+), 46 deletions(-)
>

Merged into OE-Core

Thanks
	Sau!