All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 1/3] oeqa/qemurunner: Improve logging thread exit handling for qemu shutdown test
@ 2021-05-06  7:50 Richard Purdie
  2021-05-06  7:50 ` [PATCH 2/3] oeqa/qemurunner: Handle path length issues for qmp socket Richard Purdie
  2021-05-06  7:51 ` [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts Richard Purdie
  0 siblings, 2 replies; 6+ messages in thread
From: Richard Purdie @ 2021-05-06  7:50 UTC (permalink / raw)
  To: openembedded-core

Rather than totally disabling the logging, inform it we're about to exit
so we can log messages over the exit cleanly too. This aids debugging. It
also avoids a race where the logging handler could still error whilst
shutting down.

Also remove a race window by notificing the handler of the shutdown
first, before triggering it. This removes a race window I watched in
local testing.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/selftest/cases/runqemu.py |  9 ++++-----
 meta/lib/oeqa/utils/qemurunner.py       | 12 +++++++++++-
 2 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/runqemu.py b/meta/lib/oeqa/selftest/cases/runqemu.py
index 7e676bcb416..da22f77b276 100644
--- a/meta/lib/oeqa/selftest/cases/runqemu.py
+++ b/meta/lib/oeqa/selftest/cases/runqemu.py
@@ -163,12 +163,11 @@ class QemuTest(OESelftestTestCase):
         bitbake(cls.recipe)
 
     def _start_qemu_shutdown_check_if_shutdown_succeeded(self, qemu, timeout):
+        # Allow the runner's LoggingThread instance to exit without errors
+        # (such as the exception "Console connection closed unexpectedly")
+        # as qemu will disappear when we shut it down
+        qemu.runner.allowexit()
         qemu.run_serial("shutdown -h now")
-        # Stop thread will stop the LoggingThread instance used for logging
-        # qemu through serial console, stop thread will prevent this code
-        # from facing exception (Console connection closed unexpectedly)
-        # when qemu was shutdown by the above shutdown command
-        qemu.runner.stop_thread()
         time_track = 0
         try:
             while True:
diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index f6e10072887..cb0be5603c0 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -534,6 +534,10 @@ class QemuRunner:
             self.thread.stop()
             self.thread.join()
 
+    def allowexit(self):
+        if self.thread:
+            self.thread.allowexit()
+
     def restart(self, qemuparams = None):
         self.logger.warning("Restarting qemu process")
         if self.runqemu.poll() is None:
@@ -630,6 +634,7 @@ class LoggingThread(threading.Thread):
         self.logger = logger
         self.readsock = None
         self.running = False
+        self.canexit = False
 
         self.errorevents = select.POLLERR | select.POLLHUP | select.POLLNVAL
         self.readevents = select.POLLIN | select.POLLPRI
@@ -663,6 +668,9 @@ class LoggingThread(threading.Thread):
         self.close_ignore_error(self.writepipe)
         self.running = False
 
+    def allowexit(self):
+        self.canexit = True
+
     def eventloop(self):
         poll = select.poll()
         event_read_mask = self.errorevents | self.readevents
@@ -719,7 +727,9 @@ class LoggingThread(threading.Thread):
             # happened. But for this code it counts as an
             # error since the connection shouldn't go away
             # until qemu exits.
-            raise Exception("Console connection closed unexpectedly")
+            if not self.canexit:
+                raise Exception("Console connection closed unexpectedly")
+            return ''
 
         return data
 
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* [PATCH 2/3] oeqa/qemurunner: Handle path length issues for qmp socket
  2021-05-06  7:50 [PATCH 1/3] oeqa/qemurunner: Improve logging thread exit handling for qemu shutdown test Richard Purdie
@ 2021-05-06  7:50 ` Richard Purdie
  2021-05-06  7:51 ` [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts Richard Purdie
  1 sibling, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2021-05-06  7:50 UTC (permalink / raw)
  To: openembedded-core

After the addition of the qmp socket, runqemu started failing:

ERROR - Failed to run qemu: qemu-system-aarch64: -qmp unix:/home/yocto/actions-runner-meta-openembedded/_work/meta-openembedded/meta-openembedded/yoe/build/tmp/.3eg5fiid,server,wait:
UNIX socket path '/home/yocto/actions-runner-meta-openembedded/_work/meta-openembedded/meta-openembedded/yoe/build/tmp/.3eg5fiid' is too long
Path must be less than 108 bytes

To avoid this, run qemu within tmpdir and use a relative path to the socket.
This avoids having to patch the socket code within qemu.

Update the client code to chdir and only use a relative path to the socket
to match.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/utils/qemurunner.py | 56 ++++++++++++++++++-------------
 1 file changed, 32 insertions(+), 24 deletions(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index cb0be5603c0..3d3213d3d31 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -186,8 +186,10 @@ class QemuRunner:
         except:
             self.logger.error("qemurunner: qmp.py missing, please ensure it's installed")
             return False
-        qmp_port = self.tmpdir + "/." + next(tempfile._get_candidate_names())
-        qmp_param = ' -S -qmp unix:%s,server,wait' % (qmp_port)
+        # Path relative to tmpdir used as cwd for qemu below to avoid unix socket path length issues
+        qmp_file = "." + next(tempfile._get_candidate_names())
+        qmp_param = ' -S -qmp unix:./%s,server,wait' % (qmp_file)
+        qmp_port = self.tmpdir + "/" + qmp_file
 
         try:
             if self.serial_ports >= 2:
@@ -224,7 +226,7 @@ class QemuRunner:
         # blocking at the end of the runqemu script when using this within
         # oe-selftest (this makes stty error out immediately). There ought
         # to be a proper fix but this will suffice for now.
-        self.runqemu = subprocess.Popen(launch_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE, preexec_fn=os.setpgrp, env=env)
+        self.runqemu = subprocess.Popen(launch_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE, preexec_fn=os.setpgrp, env=env, cwd=self.tmpdir)
         output = self.runqemu.stdout
 
         #
@@ -287,31 +289,37 @@ class QemuRunner:
         # This will allow us to read status from Qemu if the the process
         # is still alive
         self.logger.debug("QMP Initializing to %s" % (qmp_port))
+        # chdir dance for path length issues with unix sockets
+        origpath = os.getcwd()
         try:
-            self.qmp = qmp.QEMUMonitorProtocol(qmp_port)
-        except OSError as msg:
-            self.logger.warning("Failed to initialize qemu monitor socket: %s File: %s" % (msg, msg.filename))
-            return False
+            os.chdir(os.path.dirname(qmp_port))
+            try:
+               self.qmp = qmp.QEMUMonitorProtocol(os.path.basename(qmp_port))
+            except OSError as msg:
+                self.logger.warning("Failed to initialize qemu monitor socket: %s File: %s" % (msg, msg.filename))
+                return False
 
-        self.logger.debug("QMP Connecting to %s" % (qmp_port))
-        if not os.path.exists(qmp_port) and self.is_alive():
-            self.logger.debug("QMP Port does not exist waiting for it to be created")
-            endtime = time.time() + self.runqemutime
-            while not os.path.exists(qmp_port) and self.is_alive() and time.time() < endtime:
-               self.logger.warning("QMP port does not exist yet!")
-               time.sleep(0.5)
+            self.logger.debug("QMP Connecting to %s" % (qmp_port))
             if not os.path.exists(qmp_port) and self.is_alive():
-                self.logger.warning("QMP Port still does not exist but QEMU is alive")
-                return False
+                self.logger.debug("QMP Port does not exist waiting for it to be created")
+                endtime = time.time() + self.runqemutime
+                while not os.path.exists(qmp_port) and self.is_alive() and time.time() < endtime:
+                   self.logger.warning("QMP port does not exist yet!")
+                   time.sleep(0.5)
+                if not os.path.exists(qmp_port) and self.is_alive():
+                    self.logger.warning("QMP Port still does not exist but QEMU is alive")
+                    return False
 
-        try:
-            self.qmp.connect()
-        except OSError as msg:
-            self.logger.warning("Failed to connect qemu monitor socket: %s File: %s" % (msg, msg.filename))
-            return False
-        except qmp.QMPConnectError as msg:
-            self.logger.warning("Failed to communicate with qemu monitor: %s" % (msg))
-            return False
+            try:
+                self.qmp.connect()
+            except OSError as msg:
+                self.logger.warning("Failed to connect qemu monitor socket: %s File: %s" % (msg, msg.filename))
+                return False
+            except qmp.QMPConnectError as msg:
+                self.logger.warning("Failed to communicate with qemu monitor: %s" % (msg))
+                return False
+        finally:
+            os.chdir(origpath)
 
         # Release the qemu porcess to continue running
         self.run_monitor('cont')
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts
  2021-05-06  7:50 [PATCH 1/3] oeqa/qemurunner: Improve logging thread exit handling for qemu shutdown test Richard Purdie
  2021-05-06  7:50 ` [PATCH 2/3] oeqa/qemurunner: Handle path length issues for qmp socket Richard Purdie
@ 2021-05-06  7:51 ` Richard Purdie
  2021-08-23 12:31   ` [OE-core] " Sean Nyekjaer
  1 sibling, 1 reply; 6+ messages in thread
From: Richard Purdie @ 2021-05-06  7:51 UTC (permalink / raw)
  To: openembedded-core

If the scripts/postinst-intercepts is owned by root/root then the copyfile() calls
will fail due to chown issues. We don't care about ownership of these files so
use shutil.copy() instead which won't perform any chown.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oe/package_manager/__init__.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager/__init__.py b/meta/lib/oe/package_manager/__init__.py
index 8e7128b1958..4d22bc0296c 100644
--- a/meta/lib/oe/package_manager/__init__.py
+++ b/meta/lib/oe/package_manager/__init__.py
@@ -189,7 +189,7 @@ class PackageManager(object, metaclass=ABCMeta):
         bb.utils.remove(self.intercepts_dir, True)
         bb.utils.mkdirhier(self.intercepts_dir)
         for intercept in postinst_intercepts:
-            bb.utils.copyfile(intercept, os.path.join(self.intercepts_dir, os.path.basename(intercept)))
+            shutil.copy(intercept, os.path.join(self.intercepts_dir, os.path.basename(intercept)))
 
     @abstractmethod
     def _handle_intercept_failure(self, failed_script):
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [OE-core] [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts
  2021-05-06  7:51 ` [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts Richard Purdie
@ 2021-08-23 12:31   ` Sean Nyekjaer
  2021-08-24  8:14     ` Anuj Mittal
  0 siblings, 1 reply; 6+ messages in thread
From: Sean Nyekjaer @ 2021-08-23 12:31 UTC (permalink / raw)
  To: Richard Purdie, Anuj Mittal; +Cc: openembedded-core

On Thu, May 06, 2021 at 08:51:00AM +0100, Richard Purdie wrote:
> If the scripts/postinst-intercepts is owned by root/root then the copyfile() calls
> will fail due to chown issues. We don't care about ownership of these files so
> use shutil.copy() instead which won't perform any chown.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Anuj will you backport this to gatesgarth?

/Sean

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [OE-core] [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts
  2021-08-23 12:31   ` [OE-core] " Sean Nyekjaer
@ 2021-08-24  8:14     ` Anuj Mittal
  2021-08-24  8:43       ` Sean Nyekjaer
  0 siblings, 1 reply; 6+ messages in thread
From: Anuj Mittal @ 2021-08-24  8:14 UTC (permalink / raw)
  To: sean, richard.purdie; +Cc: openembedded-core

Hello,

On Mon, 2021-08-23 at 14:31 +0200, Sean Nyekjaer wrote:
> On Thu, May 06, 2021 at 08:51:00AM +0100, Richard Purdie wrote:
> > If the scripts/postinst-intercepts is owned by root/root then the
> > copyfile() calls
> > will fail due to chown issues. We don't care about ownership of these
> > files so
> > use shutil.copy() instead which won't perform any chown.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> Anuj will you backport this to gatesgarth?

gatesgarth isn't being maintained any more.

https://wiki.yoctoproject.org/wiki/Releases

Thanks,

Anuj

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [OE-core] [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts
  2021-08-24  8:14     ` Anuj Mittal
@ 2021-08-24  8:43       ` Sean Nyekjaer
  0 siblings, 0 replies; 6+ messages in thread
From: Sean Nyekjaer @ 2021-08-24  8:43 UTC (permalink / raw)
  To: Mittal, Anuj; +Cc: richard.purdie, openembedded-core

On Tue, Aug 24, 2021 at 08:14:58AM +0000, Mittal, Anuj wrote:
> Hello,
> 
> On Mon, 2021-08-23 at 14:31 +0200, Sean Nyekjaer wrote:
> > On Thu, May 06, 2021 at 08:51:00AM +0100, Richard Purdie wrote:
> > > If the scripts/postinst-intercepts is owned by root/root then the
> > > copyfile() calls
> > > will fail due to chown issues. We don't care about ownership of these
> > > files so
> > > use shutil.copy() instead which won't perform any chown.
> > > 
> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > 
> > Anuj will you backport this to gatesgarth?
> 
> gatesgarth isn't being maintained any more.
> 
> https://wiki.yoctoproject.org/wiki/Releases
> 

Hi Anuj,

I know, but gatesgarth is left in a broken state because of
edc8051bc0 image: Add directories to PSEUDO_IGNORE_PATHS
was merged ;)

/Sean

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2021-08-24  8:43 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-06  7:50 [PATCH 1/3] oeqa/qemurunner: Improve logging thread exit handling for qemu shutdown test Richard Purdie
2021-05-06  7:50 ` [PATCH 2/3] oeqa/qemurunner: Handle path length issues for qmp socket Richard Purdie
2021-05-06  7:51 ` [PATCH 3/3] lib/package_manager: Use shutil.copy instead of bb.utils.copyfile for intercepts Richard Purdie
2021-08-23 12:31   ` [OE-core] " Sean Nyekjaer
2021-08-24  8:14     ` Anuj Mittal
2021-08-24  8:43       ` Sean Nyekjaer

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.