[PATCH 1/2] cleanup-workdir: add a method for getting the build dir

[PATCH 1/2] cleanup-workdir: add a method for getting the build dir

[Lucas Dutra Nunes]

Signed-off-by: Lucas Dutra Nunes <ldnunes@ossystems.com.br>
---
 scripts/cleanup-workdir | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/cleanup-workdir b/scripts/cleanup-workdir
index a7f5a3a..bf37f90 100755
--- a/scripts/cleanup-workdir
+++ b/scripts/cleanup-workdir
@@ -64,6 +64,9 @@ def get_cur_arch_dirs(workdir, arch_dirs):
         m = re.match(pattern, output)
         arch_dirs.append(m.group(1))
 
+def get_build_dir():
+    return run_command('echo $BUILDDIR').strip()
+
 def main():
     global parser
     parser = optparse.OptionParser(
@@ -77,7 +80,7 @@ will be deleted. Be CAUTIOUS.""")
 
     options, args = parser.parse_args(sys.argv)
 
-    builddir = run_command('echo $BUILDDIR').strip()
+    builddir = get_build_dir()
     if len(builddir) == 0:
         err_quit("Please source file \"oe-init-build-env\" first.\n")
 
-- 
2.1.4

[PATCH 2/2] cleanup-workdir: wait for bitbake instances to finish

[Lucas Dutra Nunes]

bitbake uses a lock file on the build dir, "bitbake.lock", to prevent it
from running before an instance has exited. And sometimes the
cleanup-workdir script can call bibake too many times, too fast, before
the lock has been released.

By simply waiting that the lock has been released solves this problem.

Signed-off-by: Lucas Dutra Nunes <ldnunes@ossystems.com.br>
---
 scripts/cleanup-workdir | 31 +++++++++++++++++++++++++++----
 1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/scripts/cleanup-workdir b/scripts/cleanup-workdir
index bf37f90..3e1df1f 100755
--- a/scripts/cleanup-workdir
+++ b/scripts/cleanup-workdir
@@ -21,6 +21,8 @@ import optparse
 import re
 import subprocess
 import shutil
+import fcntl
+from contextlib import contextmanager
 
 pkg_cur_dirs = {}
 obsolete_dirs = []
@@ -51,7 +53,7 @@ def get_cur_arch_dirs(workdir, arch_dirs):
     pattern = workdir + '/(.*?)/'
 
     cmd = "bitbake -e | grep ^SDK_ARCH="
-    output = run_command(cmd)
+    output = run_bitbake_command(cmd)
     sdk_arch = output.split('"')[1]
 
     # select thest 5 packages to get the dirs of current arch
@@ -59,7 +61,7 @@ def get_cur_arch_dirs(workdir, arch_dirs):
 
     for pkg in pkgs:
         cmd = "bitbake -e " + pkg + " | grep ^IMAGE_ROOTFS="
-        output = run_command(cmd)
+        output = run_bitbake_command(cmd)
         output = output.split('"')[1]
         m = re.match(pattern, output)
         arch_dirs.append(m.group(1))
@@ -67,6 +69,27 @@ def get_cur_arch_dirs(workdir, arch_dirs):
 def get_build_dir():
     return run_command('echo $BUILDDIR').strip()
 
+@contextmanager
+def wait_for_bitbake():
+    builddir = get_build_dir()
+    bitbake_lock_file = os.path.join(builddir, 'bitbake.lock')
+
+    with open(bitbake_lock_file, 'w+') as f:
+        fd = f.fileno()
+        try:
+            # Lock and unlock the lock file, to be sure that there are no other
+            # instances of bitbake running at the moment:
+            fcntl.flock(fd, fcntl.LOCK_EX)
+            fcntl.flock(fd, fcntl.LOCK_UN)
+            yield
+        finally:
+            # And unlock again, just to be safe:
+            fcntl.flock(fd, fcntl.LOCK_UN)
+
+def run_bitbake_command(cmd):
+    with wait_for_bitbake():
+        return run_command(cmd)
+
 def main():
     global parser
     parser = optparse.OptionParser(
@@ -89,7 +112,7 @@ will be deleted. Be CAUTIOUS.""")
 
     print 'Updating bitbake caches...'
     cmd = "bitbake -s"
-    output = run_command(cmd)
+    output = run_bitbake_command(cmd)
 
     output = output.split('\n')
     index = 0
@@ -111,7 +134,7 @@ will be deleted. Be CAUTIOUS.""")
         pkg_cur_dirs[elems[0]] = version
 
     cmd = "bitbake -e"
-    output = run_command(cmd)
+    output = run_bitbake_command(cmd)
 
     tmpdir = None
     image_rootfs = None
-- 
2.1.4

Re: [PATCH 2/2] cleanup-workdir: wait for bitbake instances to finish

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 883 bytes --]

On Mon, May 18, 2015 at 1:08 PM, Lucas Dutra Nunes <ldnunes@ossystems.com.br
> wrote:

> bitbake uses a lock file on the build dir, "bitbake.lock", to prevent it
> from running before an instance has exited. And sometimes the
> cleanup-workdir script can call bibake too many times, too fast, before
> the lock has been released.
>
> By simply waiting that the lock has been released solves this problem.
>
> Signed-off-by: Lucas Dutra Nunes <ldnunes@ossystems.com.br>


The fact that the bitbake UI can exit before the lock has been released is
a bug. Not being able to assume bitbake is done by the time the bitbake
command exits is not ideal, no one should have to block directly o the lock
like this.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1342 bytes --]

Re: [PATCH 2/2] cleanup-workdir: wait for bitbake instances to finish

[Otavio Salvador]

On Mon, May 18, 2015 at 7:23 PM, Christopher Larson <clarson@kergoth.com> wrote:
>
>
> On Mon, May 18, 2015 at 1:08 PM, Lucas Dutra Nunes <ldnunes@ossystems.com.br> wrote:
>>
>> bitbake uses a lock file on the build dir, "bitbake.lock", to prevent it
>> from running before an instance has exited. And sometimes the
>> cleanup-workdir script can call bibake too many times, too fast, before
>> the lock has been released.
>>
>> By simply waiting that the lock has been released solves this problem.
>>
>> Signed-off-by: Lucas Dutra Nunes <ldnunes@ossystems.com.br>
>
>
> The fact that the bitbake UI can exit before the lock has been released is a bug. Not being able to assume bitbake is done by the time the bitbake command exits is not ideal, no one should have to block directly o the lock like this.

Agreed so you would prefer this to be fixed there?

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 2/2] cleanup-workdir: wait for bitbake instances to finish

[Anders Darander]

* Otavio Salvador <otavio@ossystems.com.br> [150519 00:27]:

> On Mon, May 18, 2015 at 7:23 PM, Christopher Larson <clarson@kergoth.com> wrote:

> > On Mon, May 18, 2015 at 1:08 PM, Lucas Dutra Nunes <ldnunes@ossystems.com.br> wrote:

> >> bitbake uses a lock file on the build dir, "bitbake.lock", to prevent it
> >> from running before an instance has exited. And sometimes the
> >> cleanup-workdir script can call bibake too many times, too fast, before
> >> the lock has been released.

> >> By simply waiting that the lock has been released solves this problem.

> >> Signed-off-by: Lucas Dutra Nunes <ldnunes@ossystems.com.br>

> > The fact that the bitbake UI can exit before the lock has been
> > released is a bug. Not being able to assume bitbake is done by the
> > time the bitbake command exits is not ideal, no one should have to
> > block directly o the lock like this.

> Agreed so you would prefer this to be fixed there?

Sure, that would benefit everyone, not just this use-case.

Cheers,
Anders
-- 
Anders Darander
ChargeStorm AB / eStorm AB

Re: [PATCH 2/2] cleanup-workdir: wait for bitbake instances to finish

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 449 bytes --]

On 18 May 2015 at 23:23, Christopher Larson <clarson@kergoth.com> wrote:

> The fact that the bitbake UI can exit before the lock has been released is
> a bug. Not being able to assume bitbake is done by the time the bitbake
> command exits is not ideal, no one should have to block directly o the lock
> like this.
>

Agreed and I personally seem to hit this often, when doing things like
bitbake foo -ccleansstate ; bitbake foo.

Ross

[-- Attachment #2: Type: text/html, Size: 876 bytes --]