[PATCH 1/7] arm/oeqa/fvp: wrap bootlog output correctly

[PATCH 1/7] arm/oeqa/fvp: wrap bootlog output correctly

[Ross Burton]

Join the list of boot log lines with newlines, so it displays properly.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta-arm/lib/oeqa/controllers/fvp.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta-arm/lib/oeqa/controllers/fvp.py b/meta-arm/lib/oeqa/controllers/fvp.py
index 79d2f2ef..87e1b941 100644
--- a/meta-arm/lib/oeqa/controllers/fvp.py
+++ b/meta-arm/lib/oeqa/controllers/fvp.py
@@ -64,7 +64,7 @@ class OEFVPTarget(oeqa.core.target.ssh.OESSHTarget):
                 return
         except asyncio.TimeoutError:
             self.logger.info("Timed out waiting for login prompt.")
-        self.logger.info(b"".join(bootlog.splitlines()[-20:]).decode("utf-8", errors="replace"))
+        self.logger.info(b"\n".join(bootlog.splitlines()[-20:]).decode("utf-8", errors="replace"))
         raise RuntimeError("Failed to start FVP.")
 
     def start(self, **kwargs):
-- 
2.25.1

[PATCH 2/7] runfvp: add an asyncio TODO

[Ross Burton]

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 scripts/runfvp | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/runfvp b/scripts/runfvp
index 66a76cc5..524de24d 100755
--- a/scripts/runfvp
+++ b/scripts/runfvp
@@ -205,6 +205,7 @@ def runfvp(cli_args):
             telnet = await asyncio.create_subprocess_exec("telnet", "localhost", str(port), stdin=sys.stdin, stdout=sys.stdout)
             await telnet.wait()
             logger.debug(f"Telnet quit, cancelling tasks")
+            # TODO: this is 3.7+
             for t in asyncio.all_tasks():
                 logger.debug(f"Cancelling {t}")
                 t.cancel()
-- 
2.25.1

[PATCH 3/7] runfvp: handle the fvp quitting before we kill it

[Ross Burton]

Don't raise an exception if the FVP has quit before we get around to
killing it.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 scripts/runfvp | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/runfvp b/scripts/runfvp
index 524de24d..877e80cb 100755
--- a/scripts/runfvp
+++ b/scripts/runfvp
@@ -179,7 +179,10 @@ async def start_fvp(cli, console_cb):
     finally:
         # If we get cancelled or throw an exception, kill the FVP
         logger.debug(f"Killing FVP PID {fvp_process.pid}")
-        fvp_process.terminate()
+        try:
+            fvp_process.terminate()
+        except ProcessLookupError:
+            pass
 
     if await fvp_process.wait() != 0:
         logger.info(f"{cli[0]} quit with code {fvp_process.returncode}")
-- 
2.25.1

[PATCH 4/7] runfvp: don't hide output when using terminals

[Ross Burton]

Only pass a console_cb if we're hooking up a console, so that the output
from the FVP is visible on the terminal.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 scripts/runfvp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/runfvp b/scripts/runfvp
index 877e80cb..77668248 100755
--- a/scripts/runfvp
+++ b/scripts/runfvp
@@ -216,7 +216,8 @@ def runfvp(cli_args):
     try:
         # When we can assume Py3.7+, this can simply be asyncio.run()
         loop = asyncio.get_event_loop()
-        loop.run_until_complete(asyncio.gather(start_fvp(cli, console_cb=console_started)))
+        console_cb = expected_terminal and console_started or None
+        loop.run_until_complete(asyncio.gather(start_fvp(cli, console_cb=console_cb)))
     except asyncio.CancelledError:
         pass
 
-- 
2.25.1

[PATCH 5/7] runfvp: don't use f-string when there's no expressions

[Ross Burton]

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 scripts/runfvp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/runfvp b/scripts/runfvp
index 77668248..fb334795 100755
--- a/scripts/runfvp
+++ b/scripts/runfvp
@@ -169,7 +169,7 @@ async def start_fvp(cli, console_cb):
 
             # Look for serial connections opening
             if console_cb:
-                m = re.match(fr"^(\S+): Listening for serial connection on port (\d+)$", line)
+                m = re.match(r"^(\S+): Listening for serial connection on port (\d+)$", line)
                 if m:
                     terminal = m.group(1)
                     port = int(m.group(2))
-- 
2.25.1

[PATCH 6/7] runfvp: refactor terminal selection

[Ross Burton]

Rewrite the terminal code to have a priority list of terminals when
selecting a default, allow the user to pick a default with a
configuration file, and add gnome-terminal to the list.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 scripts/runfvp | 69 +++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 52 insertions(+), 17 deletions(-)

diff --git a/scripts/runfvp b/scripts/runfvp
index fb334795..ca1e797c 100755
--- a/scripts/runfvp
+++ b/scripts/runfvp
@@ -1,6 +1,7 @@
 #! /usr/bin/env python3
 
 import asyncio
+import collections
 import json
 import os
 import re
@@ -12,25 +13,59 @@ import pathlib
 import logging
 logger = logging.getLogger("RunFVP")
 
-# TODO: option to switch between telnet and netcat
-connect_command = "telnet localhost %port"
+from typing import List, Optional
+
+def get_config_dir() -> pathlib.Path:
+    value = os.environ.get("XDG_CONFIG_HOME")
+    if value and os.path.isabs(value):
+        return pathlib.Path(value)
+    else:
+        return pathlib.Path.home() / ".config"
+
+class Terminals:
+    Terminal = collections.namedtuple("Terminal", ["priority", "name", "command"])
+
+    def __init__(self):
+        self.terminals = []
+
+    def add_terminal(self, priority, name, command):
+        self.terminals.append(Terminals.Terminal(priority, name, command))
+        # Keep this list sorted by priority
+        self.terminals.sort(reverse=True, key=lambda t: t.priority)
+        self.name_map = {t.name: t for t in self.terminals}
 
-terminal_map = {
-    "tmux": f"tmux new-window -n \"%title\" \"{connect_command}\"",
-    "xterm": f"xterm -title \"%title\" -e {connect_command}",
-    "none": ""
-    # TODO more terminals
-}
+    def configured_terminal(self) -> Optional[str]:
+        import configparser
 
-def get_default_terminal():
-    import shutil
-    import shlex
+        config = configparser.ConfigParser()
+        config.read(get_config_dir() / "runfvp.conf")
+        return config.get("RunFVP", "Terminal", fallback=None)
 
-    for terminal in "xterm",:
-        command = shlex.split(terminal_map[terminal])[0]
-        if shutil.which(command):
-            return terminal
-    return "none"
+    def preferred_terminal(self) -> str:
+        import shlex, shutil
+
+        preferred = self.configured_terminal()
+        if preferred:
+            return preferred
+
+        for t in self.terminals:
+            if t.command and shutil.which(shlex.split(t.command)[0]):
+                return t.name
+        return self.terminals[-1].name
+
+    def all_terminals(self) -> List[str]:
+        return self.name_map.keys()
+
+    def __getitem__(self, name: str):
+        return self.name_map[name]
+
+terminals = Terminals()
+# TODO: option to switch between telnet and netcat
+connect_command = "telnet localhost %port"
+terminals.add_terminal(2, "tmux", f"tmux new-window -n \"%title\" \"{connect_command}\""),
+terminals.add_terminal(2, "gnome-terminal", f"gnome-terminal --window --title \"%title\" --command \"{connect_command}\""),
+terminals.add_terminal(1, "xterm", f"xterm -title \"%title\" -e {connect_command}"),
+terminals.add_terminal(0, "none", None)
 
 def get_image_directory(machine=None):
     """
@@ -59,7 +94,7 @@ def parse_args(arguments):
     parser = argparse.ArgumentParser(description="Run images in a FVP")
     parser.add_argument("config", nargs="?", help="Machine name or path to .fvpconf file")
     group = parser.add_mutually_exclusive_group()
-    group.add_argument("-t", "--terminals", choices=terminal_map.keys(), default=get_default_terminal(), help="Automatically start terminals (default: %(default)s)")
+    group.add_argument("-t", "--terminals", choices=terminals.all_terminals(), default=terminals.preferred_terminal(), help="Automatically start terminals (default: %(default)s)")
     group.add_argument("-c", "--console", action="store_true", help="Attach the first uart to stdin/stdout")
     parser.add_argument("--verbose", action="store_true", help="Output verbose logging")
     parser.usage = f"{parser.format_usage().strip()} -- [ arguments passed to FVP ]"
-- 
2.25.1

[PATCH 7/7] runfvp: add basic documentation

[Ross Burton]

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 documentation/runfvp.md | 120 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 120 insertions(+)
 create mode 100644 documentation/runfvp.md

diff --git a/documentation/runfvp.md b/documentation/runfvp.md
new file mode 100644
index 00000000..b9e007b1
--- /dev/null
+++ b/documentation/runfvp.md
@@ -0,0 +1,120 @@
+# Running Images with a FVP
+
+The `runfvp` tool in meta-arm makes it easy to run Yocto Project disk images inside a [Fixed Virtual Platform (FVP)][FVP].  Some FVPs, such as the [Arm Architecture Models][AEM], are available free to download, but others need registration or are only available commercially.  The `fvp-base` machine in meta-arm-bsp uses one of these AEM models.
+
+## Running images with `runfvp`
+
+To build images with the FVP integration, the `fvpboot` class needs to be inherited.  If the machine does not do this explicitly it can be done in `local.conf`:
+
+```
+INHERIT += "fvpboot"
+```
+
+The class will download the correct FVP and write a `.fvpconf` configuration file when an image is built.
+
+To run an image in a FVP, pass either a machine name or a `.fvpconf` path to `runfvp`.
+
+```
+$ ./meta-arm/scripts/runfvp tmp/deploy/images/fvp-base/core-image-minimal-fvp-base.fvpconf
+```
+
+When a machine name is passed, `runfvp` will start the latest image that has been built for that machine. This requires that the BitBake environment has been initialized (using `oe-init-build-env` or similar) as it will start BitBake to determine where the images are.
+
+```
+$ ./meta-arm/scripts/runfvp fvp-base
+```
+
+Note that currently meta-arm's `scripts` directory isn't in `PATH`, so a full path needs to be used.
+
+`runfvp` will automatically start terminals connected to each of the serial ports that the machine specifies.  This can be controlled by using the `--terminals` option, for example `--terminals=none` will mean no terminals are started, and `--terminals=tmux` will start the terminals in [`tmux`][tmux] sessions.  Alternatively, passing `--console` will connect the serial port directly to the current session, without needing to open further windows.
+
+The default terminal can also be configured by writing a [INI-style][INI] configuration file to `~/.config/runfvp.conf`:
+
+```
+[RunFVP]
+Terminal=tmux
+```
+
+Arbitrary options can be passed directly to the FVP by specifying them after a double dash, for example this will list all of the FVP parameters:
+
+```
+$ runfvp fvp-base -- --list-params
+```
+
+## Configuring machines with `fvpboot`
+
+To configure a machine so that it can be ran inside `runfvp`, a number of variables need to be set in the machine configuration file (such as `meta-arm-bsp/conf/machine/fvp-base.conf`).
+
+Note that at present these variables are not stable and their behaviour may be changed in the future.
+
+### `FVP_EXE`
+
+The name of the FVP binary itself, for example `fvp-base` uses `FVP_Base_RevC-2xAEMvA`.
+
+### `FVP_PROVIDER`
+
+The name of the recipe that provides the FVP executable set in `FVP_EXE`, for example `fvp-base` uses `fvp-base-a-aem-native`.  This *must* be a `-native` recipe as the binary will be executed on the build host.
+
+There are recipes for common FVPs in meta-arm already, and writing new recipes is trivial.  For FVPs which are free to download `fvp-base-a-aem.bb` is a good example. Some FVPs must be downloaded separately as they need an account on Arm's website, `fvp-base-r-aem.bb` is a good example of those.
+
+If `FVP_PROVIDER` is not set then it is assumed that `FVP_EXE` is installed on the host already.
+
+### `FVP_CONFIG`
+
+Parameters passed to the FVP with the `--parameter`/`-C` option.  These are expressed as variable flags so individual parameters can be altered easily. For example:
+
+```
+FVP_CONFIG[bp.flashloader0.fname] = "${DEPLOY_DIR_IMAGE}/fip-fvp.bin"
+```
+
+### `FVP_DATA`
+
+Specify raw data to load at the specified address, passed to the FVP with the `--data` option.  This is a space-separated list of parameters in the format `[INST=]FILE@[MEMSPACE:]ADDRESS`. For example:
+
+```
+FVP_DATA = "cluster0.cpu0=${DEPLOY_DIR_IMAGE}/Image@0x80080000 \
+            cluster0.cpu0=${DEPLOY_DIR_IMAGE}/fvp-base-revc.dtb@0x83000000"
+```
+
+### `FVP_APPLICATIONS`
+
+Applications to load on the cores, passed to the FVP with the `--application` option.  These are expressed as variable flags with the flag name being the instance and flag value the filename, for example:
+
+```
+FVP_APPLICATIONS[cluster0] = "${DEPLOY_DIR_IMAGE}/linux-system.axf"
+```
+
+Note that symbols are not allowed in flag names, so if you need to use a wildcard in the instance then you'll need to use `FVP_EXTRA_ARGS` and `--application` directly.
+
+### `FVP_TERMINALS`
+
+Map hardware serial ports to abstract names. For example the `FVP_Base_RevC-2xAEMvA` FVP exposes four serial ports, `terminal_0` to `terminal_3`.  Typically only `terminal_0` is used in the `fvp-base` machine so this can be named `"Console"` and the others `""`.  When runfvp starts terminals it will only start named serial ports, so instead of opening four windows where only one is useful, it will only open one.
+
+For example:
+```
+FVP_TERMINALS[bp.terminal_0] = "Console"
+FVP_TERMINALS[bp.terminal_1] = ""
+FVP_TERMINALS[bp.terminal_2] = ""
+FVP_TERMINALS[bp.terminal_3] = ""
+```
+
+### `FVP_CONSOLE`
+
+This specifies what serial port is used when `--console` is passed to runfvp. Note that this has to be the FVP identifier but without the board prefix, for example:
+```
+FVP_CONSOLE = "terminal_0"
+```
+
+### `FVP_EXTRA_ARGS`
+
+Arbitrary extra arguments that are passed directly to the FVP.  For example:
+
+```
+FVP_EXTRA_ARGS = "--simlimit 60"
+```
+
+
+[AEM]: https://developer.arm.com/tools-and-software/simulation-models/fixed-virtual-platforms/arm-ecosystem-models
+[FVP]: https://developer.arm.com/tools-and-software/simulation-models/fixed-virtual-platforms
+[tmux]: https://tmux.github.io/
+[INI]: https://docs.python.org/3/library/configparser.html
-- 
2.25.1

Re: [PATCH 1/7] arm/oeqa/fvp: wrap bootlog output correctly

[Jon Mason]

On Mon, 20 Dec 2021 17:13:31 +0000, Ross Burton wrote:
> Join the list of boot log lines with newlines, so it displays properly.
> 
> 

Applied, thanks!

[1/7] arm/oeqa/fvp: wrap bootlog output correctly
      commit: 17f7371ca85974422a752a8bea63aaacb9fb58c3
[2/7] runfvp: add an asyncio TODO
      commit: 12501b717f68e7b8f3d04fc89c71c86d055cd8cf
[3/7] runfvp: handle the fvp quitting before we kill it
      commit: 1512bf2aef5931c94c9a5bb996133a90ab31f454
[4/7] runfvp: don't hide output when using terminals
      commit: 1ccccbf6037c9ce3e41162e86f6372e0e082b1ce
[5/7] runfvp: don't use f-string when there's no expressions
      commit: 778037450047d1b6a3b6592a3c1b2ddb3731ecf0
[6/7] runfvp: refactor terminal selection
      commit: 10e60cc8ef6584c6a76e1b6eded5de28ae7af41c
[7/7] runfvp: add basic documentation
      commit: 31a520838392c1834684a1a5f027a1b4b65e241d

Best regards,
-- 
Jon Mason <jon.mason@arm.com>