[bitbake][dunfell][1.46][PATCH 0/8] Patch review

[bitbake][dunfell][1.46][PATCH 0/8] Patch review

[Steve Sakoman]

Please review this next set of patches for 1.46 (dunfell) and have comments back
by end of day Friday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1167

The following changes since commit 1d3892d93ee9688d69f5bb5ea6aef8b1152bd1e3:

  msg: Avoid issues where paths have relative components (2020-07-01 05:24:29 -1000)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Chris Laplante (1):
  ui/teamcity: don't use removed logging classes

David Khouya (2):
  lib/ui/taskexp: Validate gi import
  lib/ui/taskexp: Fix missing Gtk import

Konrad Weihmann (1):
  pyshyacc: allow double COMMA statements

Richard Purdie (4):
  server/process: Increase timeout for commands
  fetch2: Change git fetcher not to destroy old references
  server/process: Fix a rare lockfile race
  server/process: Ensure UI-less servers don't sit in infinite loops

 lib/bb/fetch2/git.py     |  2 +-
 lib/bb/pysh/pyshyacc.py  |  1 +
 lib/bb/server/process.py | 25 +++++++++++++++++++------
 lib/bb/ui/taskexp.py     | 13 ++++++++++---
 lib/bb/ui/teamcity.py    |  2 --
 5 files changed, 31 insertions(+), 12 deletions(-)

-- 
2.17.1

[bitbake][dunfell][1.46][PATCH 1/8] lib/ui/taskexp: Validate gi import

[Steve Sakoman]

From: David Khouya <dakhouya@gmail.com>

When running bitbake -g -u taskexp without having gi python module or
and invalid gtk version, bitbake fails with a stack trace.

In case of import or version error, bitbake should exit with an error
message instead of a stack trace.

Signed-off-by: David Khouya <dakhouya@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2a2c507f239b047f34765312df4168030e38b90d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/ui/taskexp.py | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/lib/bb/ui/taskexp.py b/lib/bb/ui/taskexp.py
index 8fff2442..336d197d 100644
--- a/lib/bb/ui/taskexp.py
+++ b/lib/bb/ui/taskexp.py
@@ -8,9 +8,15 @@
 #
 
 import sys
-import gi
-gi.require_version('Gtk', '3.0')
-from gi.repository import Gtk, Gdk, GObject
+
+try:
+    import gi
+    gi.require_version('Gtk', '3.0')
+except ValueError:
+    sys.exit("FATAL: Gtk version needs to be 3.0")
+except ImportError:
+    sys.exit("FATAL: Gtk ui could not load the required gi python module")
+
 import threading
 from xmlrpc import client
 import bb
-- 
2.17.1

[bitbake][dunfell][1.46][PATCH 2/8] lib/ui/taskexp: Fix missing Gtk import

[Steve Sakoman]

From: David Khouya <dakhouya@gmail.com>

Adding back gtk objects import.

Fix bug introduce when adding validation on gtk import.

Signed-off-by: David Khouya <dakhouya@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 765be5ef60668f8a1cfbcba248f4995725807196)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/ui/taskexp.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/bb/ui/taskexp.py b/lib/bb/ui/taskexp.py
index 336d197d..05e32338 100644
--- a/lib/bb/ui/taskexp.py
+++ b/lib/bb/ui/taskexp.py
@@ -12,6 +12,7 @@ import sys
 try:
     import gi
     gi.require_version('Gtk', '3.0')
+    from gi.repository import Gtk, Gdk, GObject
 except ValueError:
     sys.exit("FATAL: Gtk version needs to be 3.0")
 except ImportError:
-- 
2.17.1

[bitbake][dunfell][1.46][PATCH 3/8] ui/teamcity: don't use removed logging classes

[Steve Sakoman]

From: Chris Laplante <mostthingsweb@gmail.com>

Allows the TeamCity frontend to be used again.

Signed-off-by: Chris Laplante <mostthingsweb@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c5477ba79fcad4a887808dd0df9cfe3554e2c17a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/ui/teamcity.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/lib/bb/ui/teamcity.py b/lib/bb/ui/teamcity.py
index 1854292f..fca46c28 100644
--- a/lib/bb/ui/teamcity.py
+++ b/lib/bb/ui/teamcity.py
@@ -167,8 +167,6 @@ def main(server, eventHandler, params):
         forcelevel = bb.msg.BBLogFormatter.ERROR
     else:
         forcelevel = bb.msg.BBLogFormatter.WARNING
-    bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut, forcelevel)
-    bb.msg.addDefaultlogFilter(errconsole, bb.msg.BBLogFilterStdErr)
     console.setFormatter(format)
     errconsole.setFormatter(format)
     if not bb.msg.has_console_handler(logger):
-- 
2.17.1

[bitbake][dunfell][1.46][PATCH 4/8] server/process: Increase timeout for commands

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

We're running into this timeout on loaded autobuilders in situations
where things should otherwise succeed. Log a note in these
cases and continue to try for longer.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e567743e70f426786ae54dcb5ab550748d9266e4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/server/process.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
index 69aae626..83385baf 100644
--- a/lib/bb/server/process.py
+++ b/lib/bb/server/process.py
@@ -331,7 +331,9 @@ class ServerCommunicator():
     def runCommand(self, command):
         self.connection.send(command)
         if not self.recv.poll(30):
-            raise ProcessTimeout("Timeout while waiting for a reply from the bitbake server")
+            logger.note("No reply from server in 30s")
+            if not self.recv.poll(30):
+                raise ProcessTimeout("Timeout while waiting for a reply from the bitbake server (60s)")
         return self.recv.get()
 
     def updateFeatureSet(self, featureset):
-- 
2.17.1

[bitbake][dunfell][1.46][PATCH 5/8] fetch2: Change git fetcher not to destroy old references

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

It looks like we're about to see a lot of changes in branch names in repos. If
we have the prune option here, those old names are lost, the changes propagate
to our source mirrors and our old releases break.

We have the force option so any replaced references should be replaced, its only
orphaned branches which will now be preserved.

I believe this behaviour will cause us fewer problems given the changes that
look likely to happen.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 820ab886e79eea516560c0c008e4cf059c6e11a3)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/fetch2/git.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 5b3793a7..2ce9395f 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -355,7 +355,7 @@ class Git(FetchMethod):
               runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
 
             runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, repourl), d, workdir=ud.clonedir)
-            fetch_cmd = "LANG=C %s fetch -f --prune --progress %s refs/*:refs/*" % (ud.basecmd, repourl)
+            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, repourl)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
-- 
2.17.1

[bitbake][dunfell][1.46][PATCH 6/8] pyshyacc: allow double COMMA statements

[Steve Sakoman]

From: Konrad Weihmann <kweihmann@outlook.com>

this allows shell statements like '; ;' to pass the parser.
As it may be bad code but still valid enough to execute

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b7732b1b5085bea73e17d112e1bd9ac3d4dc34fb)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/pysh/pyshyacc.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/bb/pysh/pyshyacc.py b/lib/bb/pysh/pyshyacc.py
index de565dc9..924860a6 100644
--- a/lib/bb/pysh/pyshyacc.py
+++ b/lib/bb/pysh/pyshyacc.py
@@ -570,6 +570,7 @@ def p_linebreak(p):
 
 def p_separator_op(p):                 
     """separator_op : COMMA
+                    | COMMA COMMA
                     | AMP"""
     p[0] = p[1]
 
-- 
2.17.1

[bitbake][dunfell][1.46][PATCH 7/8] server/process: Fix a rare lockfile race

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

We're seeing rare occasional races on the autobuilder as if two server
processes have the lockfile at the same time. We need to be extremely
careful this does not happen.

I think there is a potential race in this shutdown code since we delete
the lockfile, then call unlockfile() which also tries to delete it.

This means we may remove a lock file now held by another process if we're
unlucky. Since unlockfile removes the lockfile when it can, just rely on
that and remove any possible race window.

An example cooker-deamonlog:

--- Starting bitbake server pid 2266 at 2020-07-11 06:17:18.210777 ---
Started bitbake server pid 2266
Entering server connection loop
Accepting [<socket.socket fd=20, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([])
Processing Client
Connecting Client
Running command ['setFeatures', [2]]
Running command ['updateConfig', XXX]
Running command ['getVariable', 'BBINCLUDELOGS']
Running command ['getVariable', 'BBINCLUDELOGS_LINES']
Running command ['getSetVariable', 'BB_CONSOLELOG']
Running command ['getSetVariable', 'BB_LOGCONFIG']
Running command ['getUIHandlerNum']
Running command ['setEventMask', XXXX]
Running command ['getVariable', 'BB_DEFAULT_TASK']
Running command ['setConfig', 'cmd', 'build']
Running command ['getVariable', 'BBTARGETS']
Running command ['parseFiles']
--- Starting bitbake server pid 8252 at 2020-07-11 06:17:28.584514 ---
Started bitbake server pid 8252
--- Starting bitbake server pid 13278 at 2020-07-11 06:17:31.330635 ---
Started bitbake server pid 13278
Running command ['dataStoreConnectorCmd', 0, 'getVar', ('BBMULTICONFIG',), {}]
Running command ['getRecipes', '']
Running command ['clientComplete']
Processing Client
Disconnecting Client
No timeout, exiting.
Exiting

where it looks like there are two server processes running which should not be.
In that build there was a process left sitting in memory with its bitbake.sock file
missing but holding the lock (not sure why it wouldn't timeout/exit).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

(cherry picked from commit e1a7c1821483031b224a1570bfe834da755219cc)

Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/server/process.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
index 83385baf..475931a7 100644
--- a/lib/bb/server/process.py
+++ b/lib/bb/server/process.py
@@ -243,7 +243,7 @@ class ProcessServer(multiprocessing.Process):
                 lock = bb.utils.lockfile(lockfile, shared=False, retry=False, block=True)
                 if lock:
                     # We hold the lock so we can remove the file (hide stale pid data)
-                    bb.utils.remove(lockfile)
+                    # via unlockfile.
                     bb.utils.unlockfile(lock)
                     return
 
-- 
2.17.1

[bitbake][dunfell][1.46][PATCH 8/8] server/process: Ensure UI-less servers don't sit in infinite loops

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If server startup is broken for some reason (e.g. lockfile issues)
and no UI connection is made, the server will just sit inifinitely
waiting.

Add a timeout upon startup in the non-memory resident case so that
such infinite waits are avoided. In the memory resident case, the
server wouldn't have shut down in the first place or will timeout
according to configuration.

Since any race may mean the socket file is no longer present, ensure
the unlink doesn't fault upon exit, thus ensuring any hashequiv or
PRServ is removed from memory, allowing all processes to exit
cleanly in such scenarios.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 39888b750df12478e8bdea6727cca112dce1df85)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/server/process.py | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
index 475931a7..d4beac43 100644
--- a/lib/bb/server/process.py
+++ b/lib/bb/server/process.py
@@ -48,7 +48,7 @@ class ProcessServer(multiprocessing.Process):
 
         self.event_handle = None
         self.haveui = False
-        self.lastui = False
+        self.maxuiwait = 30
         self.xmlrpc = False
 
         self._idlefuns = {}
@@ -155,6 +155,7 @@ class ProcessServer(multiprocessing.Process):
                 print("No timeout, exiting.")
                 self.quit = True
 
+        self.lastui = time.time()
         while not self.quit:
             if self.sock in ready:
                 while select.select([self.sock],[],[],0)[0]:
@@ -191,11 +192,18 @@ class ProcessServer(multiprocessing.Process):
                 except (EOFError, OSError):
                     disconnect_client(self, fds)
 
-            if not self.timeout == -1.0 and not self.haveui and self.lastui and self.timeout and \
+            if not self.timeout == -1.0 and not self.haveui and self.timeout and \
                     (self.lastui + self.timeout) < time.time():
                 print("Server timeout, exiting.")
                 self.quit = True
 
+            # If we don't see a UI connection within maxuiwait, its unlikely we're going to see
+            # one. We have had issue with processes hanging indefinitely so timing out UI-less
+            # servers is useful.
+            if not self.haveui and not self.timeout and (self.lastui + self.maxuiwait) < time.time():
+                print("No UI connection within max timeout, exiting to avoid infinite loop.")
+                self.quit = True
+
             if self.command_channel in ready:
                 try:
                     command = self.command_channel.get()
@@ -220,10 +228,13 @@ class ProcessServer(multiprocessing.Process):
 
         print("Exiting")
         # Remove the socket file so we don't get any more connections to avoid races
-        os.unlink(self.sockname)
+        try:
+            os.unlink(self.sockname)
+        except:
+            pass
         self.sock.close()
 
-        try: 
+        try:
             self.cooker.shutdown(True)
             self.cooker.notifier.stop()
             self.cooker.confignotifier.stop()
-- 
2.17.1