Pipe error on bitbake server startup on centos7-ty-4

Pipe error on bitbake server startup on centos7-ty-4

[Richard Purdie]

I've cc'd a few people who may be interested in different aspects of this.

I noticed an intermittent build failure in Alex's build:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1192

the key part of the failure being:

2020-07-26 13:32:25,274 - oe-selftest - INFO - runtime_test.Postinst.test_postinst_rootfs_and_boot_systemd (subunit.RemotedTestCase)
2020-07-26 13:32:27,651 - oe-selftest - INFO -  ... ERROR
2020-07-26 13:32:28,272 - oe-selftest - INFO - 2: 36/37 352/413 (4478.73s) (runtime_test.Postinst.test_postinst_rootfs_and_boot_systemd)
2020-07-26 13:32:28,272 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 322, in test_postinst_rootfs_and_boot_systemd
    self.init_manager_loop("systemd")
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 273, in init_manager_loop
    with runqemu('core-image-minimal') as qemu:
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 316, in runqemu
    tinfoil.prepare(config_only=False, quiet=True)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/tinfoil.py", line 393, in prepare
    self.server_connection, ui_module = setup_bitbake(config_params,
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/main.py", line 438, in setup_bitbake
    server = bb.server.process.BitBakeServer(lock, sockname, configuration, featureset)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/server/process.py", line 460, in __init__
    raise SystemExit(1)
SystemExit: 1

tinfoil should really do a better job of logging here as there was
logging output but tinfoil didn't display it. Logging onto that worker,
we could recover more info, the bitbake-cookerdaemon log file has:

--- Starting bitbake server pid 22694 at 2020-07-26 13:30:34.930639 ---
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/daemonize.py", line 87, in createDaemon
    function()
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/server/process.py", line 476, in _startServer
    writer.send("r")
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/lib/bb/server/process.py", line 666, in send
    self.writer.send_bytes(obj)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
--- Starting bitbake server pid 46703 at 2020-07-26 13:32:37.519727 ---
Started bitbake server pid 46703
DEBUG: BBCooker starting 1595770357.5207798

bitbake created a pipe with os.pipe() to know when "r" was written by
the server to know it was ready. It didn't see the "r" value within the
allowed 90s so it closed the pipe. We don't know how long it had tried
to make the write for. I'd also note there was janitor build tree
removal active without the recent nice level changes yet.

Suggests the worker may be struggling with load :(

For now I at least wanted this all written down somewhere.

Richard