[PATCH 0/4] devtool ide-sdk source mappings

[PATCH 0/4] devtool ide-sdk source mappings

[Adrian Freihofer]

This is an improved version of the patch from Enguerrand de Ribaucourt
which was originally published here:
https://lists.openembedded.org/g/openembedded-core/message/196037

The basic idea is still the same but the implementation is a bit
different and the commit message and some comments explain how this is
supposed to work.

The tests have been improved to check whether GDB really uses the
correct source files. The tests still use the cpp example project, which
only provides local files. Problems in relation to source files that are
retrieved from a Git repository, for example, are therefore not yet
detected.

The commit "devtool: ide-sdk launch.json per recipe only" is a fix for
an issue which was discoverd during testing.

Adrian Freihofer (3):
  devtool: ide-sdk launch.json per recipe only
  devtool: ide-sdk prefer sources from workspace
  oe-selftest devtool: ide-sdk tests

Enguerrand de Ribaucourt (1):
  devtool: ide-sdk source mapping for vscode

 meta/lib/oeqa/selftest/cases/devtool.py     | 42 ++++++++++++++++-----
 scripts/lib/devtool/ide_plugins/__init__.py | 19 +++++++++-
 scripts/lib/devtool/ide_plugins/ide_code.py | 25 ++++++++++--
 scripts/lib/devtool/ide_sdk.py              |  2 +
 4 files changed, 73 insertions(+), 15 deletions(-)

-- 
2.43.1

[PATCH 1/4] devtool: ide-sdk launch.json per recipe only

[Adrian Freihofer]

If multiple recipes are processed at once, the launch.json and the
tasks.json of the second recipe contains also the configurations for the
binaries of the first recipe.

Example:
  devtool ide-sdk powertop cmake-example oe-selftest-image
generated a launch and a tasks configuration for the cmake-example
recipe which also offers debugging the powertop binary.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 scripts/lib/devtool/ide_plugins/ide_code.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/ide_plugins/ide_code.py b/scripts/lib/devtool/ide_plugins/ide_code.py
index d5e3a4f4215..d58d0bdc539 100644
--- a/scripts/lib/devtool/ide_plugins/ide_code.py
+++ b/scripts/lib/devtool/ide_plugins/ide_code.py
@@ -255,8 +255,10 @@ class IdeVSCode(IdeBase):
     def vscode_launch(self, modified_recipe):
         """GDB Launch configuration for binaries (elf files)"""
 
-        configurations = [self.vscode_launch_bin_dbg(
-            gdb_cross_config) for gdb_cross_config in self.gdb_cross_configs]
+        configurations = []
+        for gdb_cross_config in self.gdb_cross_configs:
+            if gdb_cross_config.modified_recipe is modified_recipe:
+                configurations.append(self.vscode_launch_bin_dbg(gdb_cross_config))
         launch_dict = {
             "version": "0.2.0",
             "configurations": configurations
@@ -280,6 +282,8 @@ class IdeVSCode(IdeBase):
             ]
         }
         for gdb_cross_config in self.gdb_cross_configs:
+            if gdb_cross_config.modified_recipe is not modified_recipe:
+                continue
             tasks_dict['tasks'].append(
                 {
                     "label": gdb_cross_config.id_pretty,
@@ -394,6 +398,8 @@ class IdeVSCode(IdeBase):
         }
         if modified_recipe.gdb_cross:
             for gdb_cross_config in self.gdb_cross_configs:
+                if gdb_cross_config.modified_recipe is not modified_recipe:
+                    continue
                 tasks_dict['tasks'].append(
                     {
                         "label": gdb_cross_config.id_pretty,
-- 
2.43.1

[PATCH 2/4] devtool: ide-sdk source mapping for vscode

[Adrian Freihofer]

From: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>

When launching the debug configuration, the source files from the debug
rootfs were openened in the editor instead of the local workspace files.
We add an exception to properly map them to the file being developed and
compiled by the IDE integration. This also more closely matches what the
user would expect compared to native development.

This is also true for the devtool fallback mode.

Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
---
 scripts/lib/devtool/ide_plugins/ide_code.py | 1 +
 scripts/lib/devtool/ide_sdk.py              | 1 +
 2 files changed, 2 insertions(+)

diff --git a/scripts/lib/devtool/ide_plugins/ide_code.py b/scripts/lib/devtool/ide_plugins/ide_code.py
index d58d0bdc539..09f0bb2a07e 100644
--- a/scripts/lib/devtool/ide_plugins/ide_code.py
+++ b/scripts/lib/devtool/ide_plugins/ide_code.py
@@ -242,6 +242,7 @@ class IdeVSCode(IdeBase):
         if gdb_cross_config.image_recipe.rootfs_dbg:
             launch_config['additionalSOLibSearchPath'] = modified_recipe.solib_search_path_str(
                 gdb_cross_config.image_recipe)
+            src_file_map[os.path.join("/usr/src/debug", modified_recipe.pn, modified_recipe.pv)] = "${workspaceFolder}"
             src_file_map["/usr/src/debug"] = os.path.join(
                 gdb_cross_config.image_recipe.rootfs_dbg, "usr", "src", "debug")
         else:
diff --git a/scripts/lib/devtool/ide_sdk.py b/scripts/lib/devtool/ide_sdk.py
index 3bfeb5a3fce..289bbbf47c2 100755
--- a/scripts/lib/devtool/ide_sdk.py
+++ b/scripts/lib/devtool/ide_sdk.py
@@ -357,6 +357,7 @@ class RecipeModified:
             'PACKAGE_DEBUG_SPLIT_STYLE')
         self.path = recipe_d.getVar('PATH')
         self.pn = recipe_d.getVar('PN')
+        self.pv = recipe_d.getVar('PV')
         self.recipe_sysroot = os.path.realpath(
             recipe_d.getVar('RECIPE_SYSROOT'))
         self.recipe_sysroot_native = os.path.realpath(
-- 
2.43.1

[PATCH 3/4] devtool: ide-sdk prefer sources from workspace

[Adrian Freihofer]

Improve the previous commit:
- log an error if some assumptions are not true
- Use TARGET_DBGSRC_DIR variable
- Do the same for ide none

Why the additional source mapping is required:

For example the cmake-example recipe refers to sources like this:
./recipe-sysroot-native/usr/bin/x86_64-poky-linux/x86_64-poky-linux-readelf \
  -wi image/usr/bin/cmake-example | grep -B1 DW_AT_comp_dir
    ...
    <560>   DW_AT_name        : (indirect line string, offset: 0x1da):
    /usr/src/debug/cmake-example/1.0/oe-local-files/cpp-example.cpp
    ...

Another example is powertop:
./recipe-sysroot-native/usr/bin/x86_64-poky-linux/x86_64-poky-linux-readelf \
  -wi image/usr/sbin/powertop | grep -B1 DW_AT_comp_dir
    ...
    <561>   DW_AT_name        : (indirect line string, offset: 0x1da):
    /usr/src/debug/powertop/2.15/src/devlist.cpp
    ...

For recipes with local files this works. The oe-local-files folder is
not available in the rootfs-dbg and therefore the sources are first
found in the workspace folder. GDB searches for source files in various
places:
https://sourceware.org/gdb/current/onlinedocs/gdb.html/Source-Path.html
However, for the powertop example the sources opened in the editor are
from the rootfs-dbg instead of from the workspace.

Bitbake calls the compiler with
  -fmacro-prefix-map=${S}=${TARGET_DBGSRC_DIR}
where TARGET_DBGSRC_DIR defaults to "/usr/src/debug/${PN}/${PV}".

A source map which maps the recipe specific path from TARGET_DBGSRC_DIR
to the workspace fixes this.
The already existing source map for /usr/src/debug applies for all other
recipes. It finds the sources (read only) in the rootfs-dbg folder.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 scripts/lib/devtool/ide_plugins/__init__.py | 19 +++++++++++++++++--
 scripts/lib/devtool/ide_plugins/ide_code.py | 16 +++++++++++++---
 scripts/lib/devtool/ide_sdk.py              |  3 ++-
 3 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/scripts/lib/devtool/ide_plugins/__init__.py b/scripts/lib/devtool/ide_plugins/__init__.py
index 02707bfc37b..19c2f61c5fd 100644
--- a/scripts/lib/devtool/ide_plugins/__init__.py
+++ b/scripts/lib/devtool/ide_plugins/__init__.py
@@ -134,8 +134,23 @@ class GdbCrossConfig:
         if self.image_recipe.rootfs_dbg:
             gdbinit_lines.append(
                 'set solib-search-path "' + self.modified_recipe.solib_search_path_str(self.image_recipe) + '"')
-            gdbinit_lines.append('set substitute-path "/usr/src/debug" "' + os.path.join(
-                self.image_recipe.rootfs_dbg, 'usr', 'src', 'debug') + '"')
+            # First: Search for sources of this recipe in the workspace folder
+            if self.modified_recipe.pn in self.modified_recipe.target_dbgsrc_dir:
+                gdbinit_lines.append('set substitute-path "%s" "%s"' %
+                                     (self.modified_recipe.target_dbgsrc_dir, self.modified_recipe.real_srctree))
+            else:
+                logger.error(
+                    "TARGET_DBGSRC_DIR must contain the recipe name PN.")
+            # Second: Search for sources of other recipes in the rootfs-dbg
+            if self.modified_recipe.target_dbgsrc_dir.startswith("/usr/src/debug"):
+                gdbinit_lines.append('set substitute-path "/usr/src/debug" "%s"' % os.path.join(
+                    self.image_recipe.rootfs_dbg, "usr", "src", "debug"))
+            else:
+                logger.error(
+                    "TARGET_DBGSRC_DIR must start with /usr/src/debug.")
+        else:
+            logger.warning(
+                "Cannot setup debug symbols configuration for GDB. IMAGE_GEN_DEBUGFS is not enabled.")
         gdbinit_lines.append(
             '%s %s:%d' % (remote_cmd, self.gdb_cross.host, self.gdbserver_port))
         gdbinit_lines.append('set remote exec-file ' + self.binary)
diff --git a/scripts/lib/devtool/ide_plugins/ide_code.py b/scripts/lib/devtool/ide_plugins/ide_code.py
index 09f0bb2a07e..a62b93224ee 100644
--- a/scripts/lib/devtool/ide_plugins/ide_code.py
+++ b/scripts/lib/devtool/ide_plugins/ide_code.py
@@ -242,9 +242,19 @@ class IdeVSCode(IdeBase):
         if gdb_cross_config.image_recipe.rootfs_dbg:
             launch_config['additionalSOLibSearchPath'] = modified_recipe.solib_search_path_str(
                 gdb_cross_config.image_recipe)
-            src_file_map[os.path.join("/usr/src/debug", modified_recipe.pn, modified_recipe.pv)] = "${workspaceFolder}"
-            src_file_map["/usr/src/debug"] = os.path.join(
-                gdb_cross_config.image_recipe.rootfs_dbg, "usr", "src", "debug")
+            # First: Search for sources of this recipe in the workspace folder
+            if modified_recipe.pn in modified_recipe.target_dbgsrc_dir:
+                src_file_map[modified_recipe.target_dbgsrc_dir] = "${workspaceFolder}"
+            else:
+                logger.error(
+                    "TARGET_DBGSRC_DIR must contain the recipe name PN.")
+            # Second: Search for sources of other recipes in the rootfs-dbg
+            if modified_recipe.target_dbgsrc_dir.startswith("/usr/src/debug"):
+                src_file_map["/usr/src/debug"] = os.path.join(
+                    gdb_cross_config.image_recipe.rootfs_dbg, "usr", "src", "debug")
+            else:
+                logger.error(
+                    "TARGET_DBGSRC_DIR must start with /usr/src/debug.")
         else:
             logger.warning(
                 "Cannot setup debug symbols configuration for GDB. IMAGE_GEN_DEBUGFS is not enabled.")
diff --git a/scripts/lib/devtool/ide_sdk.py b/scripts/lib/devtool/ide_sdk.py
index 289bbbf47c2..7807b322b31 100755
--- a/scripts/lib/devtool/ide_sdk.py
+++ b/scripts/lib/devtool/ide_sdk.py
@@ -301,6 +301,7 @@ class RecipeModified:
         self.staging_incdir = None
         self.strip_cmd = None
         self.target_arch = None
+        self.target_dbgsrc_dir = None
         self.topdir = None
         self.workdir = None
         self.recipe_id = None
@@ -357,7 +358,6 @@ class RecipeModified:
             'PACKAGE_DEBUG_SPLIT_STYLE')
         self.path = recipe_d.getVar('PATH')
         self.pn = recipe_d.getVar('PN')
-        self.pv = recipe_d.getVar('PV')
         self.recipe_sysroot = os.path.realpath(
             recipe_d.getVar('RECIPE_SYSROOT'))
         self.recipe_sysroot_native = os.path.realpath(
@@ -368,6 +368,7 @@ class RecipeModified:
             recipe_d.getVar('STAGING_INCDIR'))
         self.strip_cmd = recipe_d.getVar('STRIP')
         self.target_arch = recipe_d.getVar('TARGET_ARCH')
+        self.target_dbgsrc_dir = recipe_d.getVar('TARGET_DBGSRC_DIR')
         self.topdir = recipe_d.getVar('TOPDIR')
         self.workdir = os.path.realpath(recipe_d.getVar('WORKDIR'))
 
-- 
2.43.1

[PATCH 4/4] oe-selftest devtool: ide-sdk tests

[Adrian Freihofer]

Improve the GDB related tests. Verify GDB finds the correct source
files.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 meta/lib/oeqa/selftest/cases/devtool.py | 42 +++++++++++++++++++------
 1 file changed, 33 insertions(+), 9 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index d37848bdefc..ecff3a9ebdf 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -2544,6 +2544,10 @@ class DevtoolIdeSdkTests(DevtoolBase):
         self.assertEqual(status, 0, msg="%s failed: %s" % (ptest_cmd, output))
         self.assertIn("PASS: cpp-example-lib", output)
 
+        # Verify remote debugging works
+        self._gdb_cross_debugging(
+            qemu, recipe_name, example_exe, MAGIC_STRING_ORIG)
+
         # Replace the Magic String in the code, compile and deploy to Qemu
         cpp_example_lib_hpp = os.path.join(tempdir, 'cpp-example-lib.hpp')
         with open(cpp_example_lib_hpp, 'r') as file:
@@ -2565,6 +2569,10 @@ class DevtoolIdeSdkTests(DevtoolBase):
         self.assertEqual(status, 0, msg="%s failed: %s" % (ptest_cmd, output))
         self.assertIn("PASS: cpp-example-lib", output)
 
+        # Verify remote debugging works wit the modified magic string
+        self._gdb_cross_debugging(
+            qemu, recipe_name, example_exe, MAGIC_STRING_NEW)
+
     def _gdb_cross(self):
         """Verify gdb-cross is provided by devtool ide-sdk"""
         target_arch = self.td["TARGET_ARCH"]
@@ -2578,13 +2586,22 @@ class DevtoolIdeSdkTests(DevtoolBase):
         self.assertEqual(r.status, 0)
         self.assertIn("GNU gdb", r.output)
 
-    def _gdb_cross_debugging(self, qemu, recipe_name, example_exe):
+    def _gdb_cross_debugging(self, qemu, recipe_name, example_exe, magic_string):
         """Verify gdb-cross is working
 
         Test remote debugging:
         break main
         run
         continue
+        break CppExample::print_json()
+        continue
+        print CppExample::test_string.compare("cpp-example-lib Magic: 123456789")
+        $1 = 0
+        print CppExample::test_string.compare("cpp-example-lib Magic: 123456789aaa")
+        $2 = -3
+        list cpp-example-lib.hpp:13,13
+        13	    inline static const std::string test_string = "cpp-example-lib Magic: 123456789";
+        continue
         """
         sshargs = '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
         gdbserver_script = os.path.join(self._workspace_scripts_dir(
@@ -2609,10 +2626,21 @@ class DevtoolIdeSdkTests(DevtoolBase):
         self.assertIn("gdbserver", r.output)
 
         # Test remote debugging works
-        r = runCmd(
-            gdb_script + " --batch -ex 'break main' --ex 'run' -ex 'continue'")
+        gdb_batch_cmd = " --batch -ex 'break main' -ex 'run'"
+        gdb_batch_cmd += " -ex 'break CppExample::print_json()' -ex 'continue'"
+        gdb_batch_cmd += " -ex 'print CppExample::test_string.compare(\"cpp-example-lib %s\")'" % magic_string
+        gdb_batch_cmd += " -ex 'print CppExample::test_string.compare(\"cpp-example-lib %saaa\")'" % magic_string
+        gdb_batch_cmd += " -ex 'list cpp-example-lib.hpp:13,13'"
+        gdb_batch_cmd += " -ex 'continue'"
+        r = runCmd(gdb_script + gdb_batch_cmd)
+        self.logger.debug("%s %s returned: %s", gdb_script,
+                          gdb_batch_cmd, r.output)
         self.assertEqual(r.status, 0)
         self.assertIn("Breakpoint 1, main", r.output)
+        self.assertIn("$1 = 0", r.output)  # test.string.compare equal
+        self.assertIn("$2 = -3", r.output)  # test.string.compare longer
+        self.assertIn(
+            'inline static const std::string test_string = "cpp-example-lib %s";' % magic_string, r.output)
         self.assertIn("exited normally", r.output)
 
         # Stop the gdbserver
@@ -2689,14 +2717,12 @@ class DevtoolIdeSdkTests(DevtoolBase):
             bitbake_sdk_cmd = 'devtool ide-sdk %s %s -t root@%s -c --ide=none' % (
                 recipe_name, testimage, qemu.ip)
             runCmd(bitbake_sdk_cmd)
+            self._gdb_cross()
             self._verify_cmake_preset(tempdir)
             self._devtool_ide_sdk_qemu(tempdir, qemu, recipe_name, example_exe)
             # Verify the oe-scripts sym-link is valid
             self.assertEqual(self._workspace_scripts_dir(
                 recipe_name), self._sources_scripts_dir(tempdir))
-            # Verify GDB is working after devtool ide-sdk
-            self._gdb_cross()
-            self._gdb_cross_debugging(qemu, recipe_name, example_exe)
 
             # meson-example recipe
             recipe_name = "meson-example"
@@ -2707,13 +2733,11 @@ class DevtoolIdeSdkTests(DevtoolBase):
             bitbake_sdk_cmd = 'devtool ide-sdk %s %s -t root@%s -c --ide=none' % (
                 recipe_name, testimage, qemu.ip)
             runCmd(bitbake_sdk_cmd)
+            self._gdb_cross()
             self._devtool_ide_sdk_qemu(tempdir, qemu, recipe_name, example_exe)
             # Verify the oe-scripts sym-link is valid
             self.assertEqual(self._workspace_scripts_dir(
                 recipe_name), self._sources_scripts_dir(tempdir))
-            # Verify GDB is working after devtool ide-sdk
-            self._gdb_cross()
-            self._gdb_cross_debugging(qemu, recipe_name, example_exe)
 
     def test_devtool_ide_sdk_code_cmake(self):
         """Verify a cmake recipe works with ide=code mode"""
-- 
2.43.1

Re: [PATCH 3/4] devtool: ide-sdk prefer sources from workspace

[Enguerrand de Ribaucourt]

Great explanation and fix!