[PATCH] arm/lib: Improve FVPRunner shutdown logic

* [PATCH] arm/lib: Improve FVPRunner shutdown logic
@ 2022-07-18 12:15 Peter Hoyes
  2022-07-18 17:00 ` Jon Mason
  2022-07-18 19:05 ` Jon Mason
  0 siblings, 2 replies; 4+ messages in thread
From: Peter Hoyes @ 2022-07-18 12:15 UTC (permalink / raw)
  To: meta-arm; +Cc: diego.sueiro, Peter Hoyes

From: Peter Hoyes <Peter.Hoyes@arm.com>

Please can this be applied to master + kirkstone.

We have encountered intermittent hanging during FVP shutdown, so improve
the termination logic by first issuing a terminate(), waiting a bit
then, if necessary, issuing a kill().

Move returncode logic to after the telnet/pexpect cleanup so it
actually runs.

Move pexpect.EOF logic into FVPRunner.stop so that it executes before
closing the pexpect handle.

Issue-Id: SCM-4957
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Change-Id: Iebb3c3c89367256b1e116e66ffdb6b742358bce4
---
 meta-arm/lib/fvp/runner.py           | 34 +++++++++++++++++++---------
 meta-arm/lib/oeqa/controllers/fvp.py |  6 -----
 2 files changed, 23 insertions(+), 17 deletions(-)

diff --git a/meta-arm/lib/fvp/runner.py b/meta-arm/lib/fvp/runner.py
index 3b3fd00..7641cd6 100644
--- a/meta-arm/lib/fvp/runner.py
+++ b/meta-arm/lib/fvp/runner.py
@@ -72,24 +72,36 @@ class FVPRunner:
 
     async def stop(self):
         if self._fvp_process:
-            self._logger.debug(f"Killing FVP PID {self._fvp_process.pid}")
+            self._logger.debug(f"Terminating FVP PID {self._fvp_process.pid}")
             try:
                 self._fvp_process.terminate()
+                await asyncio.wait_for(self._fvp_process.wait(), 10.0)
+            except asyncio.TimeoutError:
+                self._logger.debug(f"Killing FVP PID {self._fvp_process.pid}")
+                self._fvp_process.kill()
             except ProcessLookupError:
                 pass
 
-            if await self._fvp_process.wait() != 0:
-                self._logger.info(f"FVP quit with code {self._fvp_process.returncode}")
-                return self._fvp_process.returncode
-            else:
-                return 0
-
         for telnet in self._telnets:
-            await telnet.terminate()
-            await telnet.wait()
+            try:
+                telnet.terminate()
+                await asyncio.wait_for(telnet.wait(), 10.0)
+            except asyncio.TimeoutError:
+                telnet.kill()
+            except ProcessLookupError:
+                pass
+
+        for console in self._pexpects:
+            import pexpect
+            # Ensure pexpect logs all remaining output to the logfile
+            console.expect(pexpect.EOF, timeout=5.0)
+            console.close()
 
-        for pexpect in self._pexpects:
-            pexpect.close()
+        if self._fvp_process and self._fvp_process.returncode:
+            self._logger.info(f"FVP quit with code {self._fvp_process.returncode}")
+            return self._fvp_process.returncode
+        else:
+            return 0
 
     async def run(self, until=None):
         if until and until():
diff --git a/meta-arm/lib/oeqa/controllers/fvp.py b/meta-arm/lib/oeqa/controllers/fvp.py
index 30b6296..c8dcf29 100644
--- a/meta-arm/lib/oeqa/controllers/fvp.py
+++ b/meta-arm/lib/oeqa/controllers/fvp.py
@@ -127,12 +127,6 @@ class OEFVPSerialTarget(OEFVPSSHTarget):
                 default_test_file = f"{name}_log{self.test_log_suffix}"
                 os.symlink(default_test_file, self.bootlog)
 
-    async def _after_stop(self):
-        # Ensure pexpect logs all remaining output to the logfile
-        for terminal in self.terminals.values():
-            terminal.expect(pexpect.EOF, timeout=5)
-            terminal.close()
-
     def _get_terminal(self, name):
         return self.terminals[name]
 
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 4+ messages in thread