* [PATCH 1/3] oeqa/qemurunner: Improve logging thread exit handling for qemu shutdown test
@ 2021-05-06 7:50 Richard Purdie
2021-05-06 7:50 ` [PATCH 2/3] oeqa/qemurunner: Handle path length issues for qmp socket Richard Purdie
2021-05-06 7:51 ` [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts Richard Purdie
0 siblings, 2 replies; 6+ messages in thread
From: Richard Purdie @ 2021-05-06 7:50 UTC (permalink / raw)
To: openembedded-core
Rather than totally disabling the logging, inform it we're about to exit
so we can log messages over the exit cleanly too. This aids debugging. It
also avoids a race where the logging handler could still error whilst
shutting down.
Also remove a race window by notificing the handler of the shutdown
first, before triggering it. This removes a race window I watched in
local testing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/lib/oeqa/selftest/cases/runqemu.py | 9 ++++-----
meta/lib/oeqa/utils/qemurunner.py | 12 +++++++++++-
2 files changed, 15 insertions(+), 6 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/runqemu.py b/meta/lib/oeqa/selftest/cases/runqemu.py
index 7e676bcb416..da22f77b276 100644
--- a/meta/lib/oeqa/selftest/cases/runqemu.py
+++ b/meta/lib/oeqa/selftest/cases/runqemu.py
@@ -163,12 +163,11 @@ class QemuTest(OESelftestTestCase):
bitbake(cls.recipe)
def _start_qemu_shutdown_check_if_shutdown_succeeded(self, qemu, timeout):
+ # Allow the runner's LoggingThread instance to exit without errors
+ # (such as the exception "Console connection closed unexpectedly")
+ # as qemu will disappear when we shut it down
+ qemu.runner.allowexit()
qemu.run_serial("shutdown -h now")
- # Stop thread will stop the LoggingThread instance used for logging
- # qemu through serial console, stop thread will prevent this code
- # from facing exception (Console connection closed unexpectedly)
- # when qemu was shutdown by the above shutdown command
- qemu.runner.stop_thread()
time_track = 0
try:
while True:
diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index f6e10072887..cb0be5603c0 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -534,6 +534,10 @@ class QemuRunner:
self.thread.stop()
self.thread.join()
+ def allowexit(self):
+ if self.thread:
+ self.thread.allowexit()
+
def restart(self, qemuparams = None):
self.logger.warning("Restarting qemu process")
if self.runqemu.poll() is None:
@@ -630,6 +634,7 @@ class LoggingThread(threading.Thread):
self.logger = logger
self.readsock = None
self.running = False
+ self.canexit = False
self.errorevents = select.POLLERR | select.POLLHUP | select.POLLNVAL
self.readevents = select.POLLIN | select.POLLPRI
@@ -663,6 +668,9 @@ class LoggingThread(threading.Thread):
self.close_ignore_error(self.writepipe)
self.running = False
+ def allowexit(self):
+ self.canexit = True
+
def eventloop(self):
poll = select.poll()
event_read_mask = self.errorevents | self.readevents
@@ -719,7 +727,9 @@ class LoggingThread(threading.Thread):
# happened. But for this code it counts as an
# error since the connection shouldn't go away
# until qemu exits.
- raise Exception("Console connection closed unexpectedly")
+ if not self.canexit:
+ raise Exception("Console connection closed unexpectedly")
+ return ''
return data
--
2.30.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/3] oeqa/qemurunner: Handle path length issues for qmp socket
2021-05-06 7:50 [PATCH 1/3] oeqa/qemurunner: Improve logging thread exit handling for qemu shutdown test Richard Purdie
@ 2021-05-06 7:50 ` Richard Purdie
2021-05-06 7:51 ` [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts Richard Purdie
1 sibling, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2021-05-06 7:50 UTC (permalink / raw)
To: openembedded-core
After the addition of the qmp socket, runqemu started failing:
ERROR - Failed to run qemu: qemu-system-aarch64: -qmp unix:/home/yocto/actions-runner-meta-openembedded/_work/meta-openembedded/meta-openembedded/yoe/build/tmp/.3eg5fiid,server,wait:
UNIX socket path '/home/yocto/actions-runner-meta-openembedded/_work/meta-openembedded/meta-openembedded/yoe/build/tmp/.3eg5fiid' is too long
Path must be less than 108 bytes
To avoid this, run qemu within tmpdir and use a relative path to the socket.
This avoids having to patch the socket code within qemu.
Update the client code to chdir and only use a relative path to the socket
to match.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/lib/oeqa/utils/qemurunner.py | 56 ++++++++++++++++++-------------
1 file changed, 32 insertions(+), 24 deletions(-)
diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index cb0be5603c0..3d3213d3d31 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -186,8 +186,10 @@ class QemuRunner:
except:
self.logger.error("qemurunner: qmp.py missing, please ensure it's installed")
return False
- qmp_port = self.tmpdir + "/." + next(tempfile._get_candidate_names())
- qmp_param = ' -S -qmp unix:%s,server,wait' % (qmp_port)
+ # Path relative to tmpdir used as cwd for qemu below to avoid unix socket path length issues
+ qmp_file = "." + next(tempfile._get_candidate_names())
+ qmp_param = ' -S -qmp unix:./%s,server,wait' % (qmp_file)
+ qmp_port = self.tmpdir + "/" + qmp_file
try:
if self.serial_ports >= 2:
@@ -224,7 +226,7 @@ class QemuRunner:
# blocking at the end of the runqemu script when using this within
# oe-selftest (this makes stty error out immediately). There ought
# to be a proper fix but this will suffice for now.
- self.runqemu = subprocess.Popen(launch_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE, preexec_fn=os.setpgrp, env=env)
+ self.runqemu = subprocess.Popen(launch_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE, preexec_fn=os.setpgrp, env=env, cwd=self.tmpdir)
output = self.runqemu.stdout
#
@@ -287,31 +289,37 @@ class QemuRunner:
# This will allow us to read status from Qemu if the the process
# is still alive
self.logger.debug("QMP Initializing to %s" % (qmp_port))
+ # chdir dance for path length issues with unix sockets
+ origpath = os.getcwd()
try:
- self.qmp = qmp.QEMUMonitorProtocol(qmp_port)
- except OSError as msg:
- self.logger.warning("Failed to initialize qemu monitor socket: %s File: %s" % (msg, msg.filename))
- return False
+ os.chdir(os.path.dirname(qmp_port))
+ try:
+ self.qmp = qmp.QEMUMonitorProtocol(os.path.basename(qmp_port))
+ except OSError as msg:
+ self.logger.warning("Failed to initialize qemu monitor socket: %s File: %s" % (msg, msg.filename))
+ return False
- self.logger.debug("QMP Connecting to %s" % (qmp_port))
- if not os.path.exists(qmp_port) and self.is_alive():
- self.logger.debug("QMP Port does not exist waiting for it to be created")
- endtime = time.time() + self.runqemutime
- while not os.path.exists(qmp_port) and self.is_alive() and time.time() < endtime:
- self.logger.warning("QMP port does not exist yet!")
- time.sleep(0.5)
+ self.logger.debug("QMP Connecting to %s" % (qmp_port))
if not os.path.exists(qmp_port) and self.is_alive():
- self.logger.warning("QMP Port still does not exist but QEMU is alive")
- return False
+ self.logger.debug("QMP Port does not exist waiting for it to be created")
+ endtime = time.time() + self.runqemutime
+ while not os.path.exists(qmp_port) and self.is_alive() and time.time() < endtime:
+ self.logger.warning("QMP port does not exist yet!")
+ time.sleep(0.5)
+ if not os.path.exists(qmp_port) and self.is_alive():
+ self.logger.warning("QMP Port still does not exist but QEMU is alive")
+ return False
- try:
- self.qmp.connect()
- except OSError as msg:
- self.logger.warning("Failed to connect qemu monitor socket: %s File: %s" % (msg, msg.filename))
- return False
- except qmp.QMPConnectError as msg:
- self.logger.warning("Failed to communicate with qemu monitor: %s" % (msg))
- return False
+ try:
+ self.qmp.connect()
+ except OSError as msg:
+ self.logger.warning("Failed to connect qemu monitor socket: %s File: %s" % (msg, msg.filename))
+ return False
+ except qmp.QMPConnectError as msg:
+ self.logger.warning("Failed to communicate with qemu monitor: %s" % (msg))
+ return False
+ finally:
+ os.chdir(origpath)
# Release the qemu porcess to continue running
self.run_monitor('cont')
--
2.30.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts
2021-05-06 7:50 [PATCH 1/3] oeqa/qemurunner: Improve logging thread exit handling for qemu shutdown test Richard Purdie
2021-05-06 7:50 ` [PATCH 2/3] oeqa/qemurunner: Handle path length issues for qmp socket Richard Purdie
@ 2021-05-06 7:51 ` Richard Purdie
2021-08-23 12:31 ` [OE-core] " Sean Nyekjaer
1 sibling, 1 reply; 6+ messages in thread
From: Richard Purdie @ 2021-05-06 7:51 UTC (permalink / raw)
To: openembedded-core
If the scripts/postinst-intercepts is owned by root/root then the copyfile() calls
will fail due to chown issues. We don't care about ownership of these files so
use shutil.copy() instead which won't perform any chown.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/lib/oe/package_manager/__init__.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oe/package_manager/__init__.py b/meta/lib/oe/package_manager/__init__.py
index 8e7128b1958..4d22bc0296c 100644
--- a/meta/lib/oe/package_manager/__init__.py
+++ b/meta/lib/oe/package_manager/__init__.py
@@ -189,7 +189,7 @@ class PackageManager(object, metaclass=ABCMeta):
bb.utils.remove(self.intercepts_dir, True)
bb.utils.mkdirhier(self.intercepts_dir)
for intercept in postinst_intercepts:
- bb.utils.copyfile(intercept, os.path.join(self.intercepts_dir, os.path.basename(intercept)))
+ shutil.copy(intercept, os.path.join(self.intercepts_dir, os.path.basename(intercept)))
@abstractmethod
def _handle_intercept_failure(self, failed_script):
--
2.30.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [OE-core] [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts
2021-05-06 7:51 ` [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts Richard Purdie
@ 2021-08-23 12:31 ` Sean Nyekjaer
2021-08-24 8:14 ` Anuj Mittal
0 siblings, 1 reply; 6+ messages in thread
From: Sean Nyekjaer @ 2021-08-23 12:31 UTC (permalink / raw)
To: Richard Purdie, Anuj Mittal; +Cc: openembedded-core
On Thu, May 06, 2021 at 08:51:00AM +0100, Richard Purdie wrote:
> If the scripts/postinst-intercepts is owned by root/root then the copyfile() calls
> will fail due to chown issues. We don't care about ownership of these files so
> use shutil.copy() instead which won't perform any chown.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Anuj will you backport this to gatesgarth?
/Sean
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts
2021-08-23 12:31 ` [OE-core] " Sean Nyekjaer
@ 2021-08-24 8:14 ` Anuj Mittal
2021-08-24 8:43 ` Sean Nyekjaer
0 siblings, 1 reply; 6+ messages in thread
From: Anuj Mittal @ 2021-08-24 8:14 UTC (permalink / raw)
To: sean, richard.purdie; +Cc: openembedded-core
Hello,
On Mon, 2021-08-23 at 14:31 +0200, Sean Nyekjaer wrote:
> On Thu, May 06, 2021 at 08:51:00AM +0100, Richard Purdie wrote:
> > If the scripts/postinst-intercepts is owned by root/root then the
> > copyfile() calls
> > will fail due to chown issues. We don't care about ownership of these
> > files so
> > use shutil.copy() instead which won't perform any chown.
> >
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> Anuj will you backport this to gatesgarth?
gatesgarth isn't being maintained any more.
https://wiki.yoctoproject.org/wiki/Releases
Thanks,
Anuj
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts
2021-08-24 8:14 ` Anuj Mittal
@ 2021-08-24 8:43 ` Sean Nyekjaer
0 siblings, 0 replies; 6+ messages in thread
From: Sean Nyekjaer @ 2021-08-24 8:43 UTC (permalink / raw)
To: Mittal, Anuj; +Cc: richard.purdie, openembedded-core
On Tue, Aug 24, 2021 at 08:14:58AM +0000, Mittal, Anuj wrote:
> Hello,
>
> On Mon, 2021-08-23 at 14:31 +0200, Sean Nyekjaer wrote:
> > On Thu, May 06, 2021 at 08:51:00AM +0100, Richard Purdie wrote:
> > > If the scripts/postinst-intercepts is owned by root/root then the
> > > copyfile() calls
> > > will fail due to chown issues. We don't care about ownership of these
> > > files so
> > > use shutil.copy() instead which won't perform any chown.
> > >
> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> >
> > Anuj will you backport this to gatesgarth?
>
> gatesgarth isn't being maintained any more.
>
> https://wiki.yoctoproject.org/wiki/Releases
>
Hi Anuj,
I know, but gatesgarth is left in a broken state because of
edc8051bc0 image: Add directories to PSEUDO_IGNORE_PATHS
was merged ;)
/Sean
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-08-24 8:43 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-06 7:50 [PATCH 1/3] oeqa/qemurunner: Improve logging thread exit handling for qemu shutdown test Richard Purdie
2021-05-06 7:50 ` [PATCH 2/3] oeqa/qemurunner: Handle path length issues for qmp socket Richard Purdie
2021-05-06 7:51 ` [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts Richard Purdie
2021-08-23 12:31 ` [OE-core] " Sean Nyekjaer
2021-08-24 8:14 ` Anuj Mittal
2021-08-24 8:43 ` Sean Nyekjaer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.