[PATCH] bitbake: Add task timeout support

* [PATCH] bitbake: Add task timeout support
@ 2023-05-25 10:21 Michal Sieron
  2023-05-25 10:44 ` [bitbake-devel] " Mikko Rapeli
  0 siblings, 1 reply; 10+ messages in thread
From: Michal Sieron @ 2023-05-25 10:21 UTC (permalink / raw)
  To: bitbake-devel; +Cc: Michal Sieron, Tomasz Dziendzielski, Mateusz Marciniec

By setting `BB_TASK_TIMEOUT` you can control timeout of for tasks.
Using flags `BB_TASK_TIMEOUT[do_mytask]` you can override timeout
settings for specific tasks.

This is especially useful when some server doesn't work properly and
`do_fetch` task takes too long. We may want to kill it early instead
of waiting for a fetch that won't happen.

Signed-off-by: Michal Sieron <michalwsieron@gmail.com>
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Mateusz Marciniec <mateuszmar2@gmail.com>
---
 .../bitbake-user-manual-ref-variables.rst     |  7 +++
 bitbake/lib/bb/build.py                       | 43 ++++++++++++++++++-
 2 files changed, 49 insertions(+), 1 deletion(-)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
index 01d4f8d14a..eaaa307960 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
@@ -686,6 +686,13 @@ overview of their function and contents.
       in images is given a higher priority as compared to build tasks to
       ensure that images do not suffer timeouts on loaded systems.
 
+   :term:`BB_TASK_TIMEOUT`
+      Specifies time after which still running tasks will be terminated.
+
+      Time is provided as an integer number of seconds.
+      You can set timeout for specific task using ``BB_TASK_TIMEOUT[do_mytask]``.
+      To remove timeout use an empty value ``BB_TASK_TIMEOUT = ""``.
+
    :term:`BB_TASKHASH`
       Within an executing task, this variable holds the hash of the task as
       returned by the currently enabled signature generator.
diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 44d08f5c55..2057dd9415 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -17,6 +17,7 @@ import sys
 import logging
 import glob
 import itertools
+import multiprocessing
 import time
 import re
 import stat
@@ -594,7 +595,7 @@ def _task_data(fn, task, d):
     bb.data.expandKeys(localdata)
     return localdata
 
-def _exec_task(fn, task, d, quieterr):
+def __exec_task(fn, task, d, quieterr):
     """Execute a BB 'task'
 
     Execution of a task involves a bit more setup than executing a function,
@@ -761,6 +762,46 @@ def _exec_task(fn, task, d, quieterr):
 
     return 0
 
+def get_task_timeout(d, task):
+    # task specific timeout
+    timeout = d.getVarFlag("BB_TASK_TIMEOUT", task)
+
+    if timeout is None:
+        # any task timeout
+        timeout = d.getVar("BB_TASK_TIMEOUT")
+
+    if timeout is None or timeout == "":
+        return None
+    else:
+        try:
+            return int(timeout)
+        except ValueError as e:
+            raise ValueError("invalid `BB_TASK_TIMEOUT` value '%s'" % timeout)
+
+def _exec_task(fn, task, d, quieterr):
+    """Intermediate function between exec_task and __exec_task, to support task timeouts."""
+
+    def targetFunc(func):
+        def inner(*args, **kwargs):
+            sys.exit(func(*args, **kwargs))
+
+        return inner
+
+    try:
+        timeout = get_task_timeout(d, task)
+    except ValueError as e:
+        logger.error("%s: %s" % (d.getVar("PF"), e.args[0]))
+        return 1
+
+    task_proc = multiprocessing.Process(target=targetFunc(__exec_task), args=(fn, task, d, quieterr))
+    task_proc.start()
+    task_proc.join(timeout)
+    if task_proc.exitcode is None:
+        logger.error("timeout exceeded (%s s)" % (str(timeout)))
+        return 1
+    else:
+        return task_proc.exitcode
+
 def exec_task(fn, task, d, profile = False):
     try:
         quieterr = False
-- 
2.40.1



^ permalink raw reply related	[flat|nested] 10+ messages in thread