[PATCH 1/2] kernel-dev: working with kernel using devtool does not require building and installing eSDK

[PATCH 1/2] kernel-dev: working with kernel using devtool does not require building and installing eSDK

[Alexander Kanavin]

In fact, all the same tasks can be done directly from a Yocto build. Let's just
drop the steps to produce and install the eSDK - if someone wants it, the SDK
manual covers everything.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 documentation/kernel-dev/common.rst | 84 +++++------------------------
 documentation/kernel-dev/intro.rst  |  8 +--
 2 files changed, 16 insertions(+), 76 deletions(-)

diff --git a/documentation/kernel-dev/common.rst b/documentation/kernel-dev/common.rst
index dc3345a52..16ef6453b 100644
--- a/documentation/kernel-dev/common.rst
+++ b/documentation/kernel-dev/common.rst
@@ -52,8 +52,8 @@ image and ready to make modifications as described in the
 ":ref:`kernel-dev/common:using \`\`devtool\`\` to patch the kernel`"
 section:
 
-1. *Initialize the BitBake Environment:* Before building an extensible
-   SDK, you need to initialize the BitBake build environment by sourcing
+1. *Initialize the BitBake Environment:*
+   you need to initialize the BitBake build environment by sourcing
    the build environment script (i.e. :ref:`structure-core-script`)::
 
       $ cd poky
@@ -120,67 +120,10 @@ section:
       NOTE: Starting bitbake server...
       $
 
-5. *Build the Extensible SDK:* Use BitBake to build the extensible SDK
-   specifically for use with images to be run using QEMU::
+5. *Build the Clean Image:* The final step in preparing to work on the
+   kernel is to build an initial image using ``bitbake``::
 
-      $ cd poky/build
-      $ bitbake core-image-minimal -c populate_sdk_ext
-
-   Once
-   the build finishes, you can find the SDK installer file (i.e.
-   ``*.sh`` file) in the following directory::
-
-      poky/build/tmp/deploy/sdk
-
-   For this example, the installer file is named
-   ``poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-&DISTRO;.sh``.
-
-6. *Install the Extensible SDK:* Use the following command to install
-   the SDK. For this example, install the SDK in the default
-   ``poky_sdk`` directory::
-
-      $ cd poky/build/tmp/deploy/sdk
-      $ ./poky-glibc-x86_64-core-image-minimal-i586-toolchain-ext-&DISTRO;.sh
-      Poky (Yocto Project Reference Distro) Extensible SDK installer version &DISTRO;
-      ============================================================================
-      Enter target directory for SDK (default: poky_sdk):
-      You are about to install the SDK to "/home/scottrif/poky_sdk". Proceed [Y/n]? Y
-      Extracting SDK......................................done
-      Setting it up...
-      Extracting buildtools...
-      Preparing build system...
-      Parsing recipes: 100% |#################################################################| Time: 0:00:52
-      Initializing tasks: 100% |############## ###############################################| Time: 0:00:04
-      Checking sstate mirror object availability: 100% |######################################| Time: 0:00:00
-      Parsing recipes: 100% |#################################################################| Time: 0:00:33
-      Initializing tasks: 100% |##############################################################| Time: 0:00:00
-      done
-      SDK has been successfully set up and is ready to be used.
-      Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g.
-       $ . /home/scottrif/poky_sdk/environment-setup-i586-poky-linux
-
-
-7. *Set Up a New Terminal to Work With the Extensible SDK:* You must set
-   up a new terminal to work with the SDK. You cannot use the same
-   BitBake shell used to build the installer.
-
-   After opening a new shell, run the SDK environment setup script as
-   directed by the output from installing the SDK::
-
-      $ source poky_sdk/environment-setup-i586-poky-linux
-      "SDK environment now set up; additionally you may now run devtool to perform development tasks.
-      Run devtool --help for further details.
-
-   .. note::
-
-      If you get a warning about attempting to use the extensible SDK in
-      an environment set up to run BitBake, you did not use a new shell.
-
-8. *Build the Clean Image:* The final step in preparing to work on the
-   kernel is to build an initial image using ``devtool`` in the new
-   terminal you just set up and initialized for SDK work::
-
-      $ devtool build-image
+      $ bitbake core-image-minimal
       Parsing recipes: 100% |##########################################| Time: 0:00:05
       Parsing of 830 .bb files complete (0 cached, 830 parsed). 1299 targets, 47 skipped, 0 masked, 0 errors.
       WARNING: No packages to add, building image core-image-minimal unmodified
@@ -192,7 +135,6 @@ section:
       NOTE: Executing SetScene Tasks
       NOTE: Executing RunQueue Tasks
       NOTE: Tasks Summary: Attempted 2866 tasks of which 2604 didn't need to be rerun and all succeeded.
-      NOTE: Successfully built core-image-minimal. You can find output files in /home/scottrif/poky_sdk/tmp/deploy/images/qemux86
 
    If you were
    building for actual hardware and not for emulation, you could flash
@@ -202,7 +144,7 @@ section:
    Wiki page.
 
 At this point you have set up to start making modifications to the
-kernel by using the extensible SDK. For a continued example, see the
+kernel. For a continued example, see the
 ":ref:`kernel-dev/common:using \`\`devtool\`\` to patch the kernel`"
 section.
 
@@ -744,7 +686,7 @@ Using ``devtool`` to Patch the Kernel
 =====================================
 
 The steps in this procedure show you how you can patch the kernel using
-the extensible SDK and ``devtool``.
+``devtool``.
 
 .. note::
 
@@ -766,8 +708,7 @@ console. The example is a continuation of the setup procedure found in
 the ":ref:`kernel-dev/common:getting ready to develop using \`\`devtool\`\``" Section.
 
 1. *Check Out the Kernel Source Files:* First you must use ``devtool``
-   to checkout the kernel source code in its workspace. Be sure you are
-   in the terminal set up to do work with the extensible SDK.
+   to checkout the kernel source code in its workspace.
 
    .. note::
 
@@ -867,7 +808,7 @@ the ":ref:`kernel-dev/common:getting ready to develop using \`\`devtool\`\``" Se
       the results of your ``printk`` statements as part of the output
       when you scroll down the console window.
 
-6. *Stage and commit your changes*: Within your eSDK terminal, change
+6. *Stage and commit your changes*: Change
    your working directory to where you modified the ``calibrate.c`` file
    and use these Git commands to stage and commit your changes::
 
@@ -878,8 +819,7 @@ the ":ref:`kernel-dev/common:getting ready to develop using \`\`devtool\`\``" Se
 
 7. *Export the Patches and Create an Append File:* To export your
    commits as patches and create a ``.bbappend`` file, use the following
-   command in the terminal used to work with the extensible SDK. This
-   example uses the previously established layer named ``meta-mylayer``.
+   command. This example uses the previously established layer named ``meta-mylayer``.
    ::
 
       $ devtool finish linux-yocto ~/meta-mylayer
@@ -907,8 +847,8 @@ Using Traditional Kernel Development to Patch the Kernel
 ========================================================
 
 The steps in this procedure show you how you can patch the kernel using
-traditional kernel development (i.e. not using ``devtool`` and the
-extensible SDK as described in the
+traditional kernel development (i.e. not using ``devtool``
+as described in the
 ":ref:`kernel-dev/common:using \`\`devtool\`\` to patch the kernel`"
 section).
 
diff --git a/documentation/kernel-dev/intro.rst b/documentation/kernel-dev/intro.rst
index b9ce7f241..4ff4dc7d3 100644
--- a/documentation/kernel-dev/intro.rst
+++ b/documentation/kernel-dev/intro.rst
@@ -114,13 +114,13 @@ general information and references for further information.
    a build host ready to use the Yocto Project.
 
 2. *Set Up Your Host Development System for Kernel Development:* It is
-   recommended that you use ``devtool`` and an extensible SDK for kernel
+   recommended that you use ``devtool`` for kernel
    development. Alternatively, you can use traditional kernel
    development methods with the Yocto Project. Either way, there are
    steps you need to take to get the development environment ready.
 
-   Using ``devtool`` and the eSDK requires that you have a clean build
-   of the image and that you are set up with the appropriate eSDK. For
+   Using ``devtool`` requires that you have a clean build
+   of the image. For
    more information, see the
    ":ref:`kernel-dev/common:getting ready to develop using \`\`devtool\`\``"
    section.
@@ -134,7 +134,7 @@ general information and references for further information.
 3. *Make Changes to the Kernel Source Code if applicable:* Modifying the
    kernel does not always mean directly changing source files. However,
    if you have to do this, you make the changes to the files in the
-   eSDK's Build Directory if you are using ``devtool``. For more
+   Yocto's Build Directory if you are using ``devtool``. For more
    information, see the
    ":ref:`kernel-dev/common:using \`\`devtool\`\` to patch the kernel`"
    section.
-- 
2.30.2

[PATCH 2/2] sdk-manual: describe how to use extensible SDK functionality directly in a Yocto build

[Alexander Kanavin]

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 .../sdk-manual/appendix-customizing.rst       | 14 ++--
 documentation/sdk-manual/appendix-obtain.rst  | 18 +++++-
 documentation/sdk-manual/extensible.rst       | 64 ++++++++++++++++++-
 documentation/sdk-manual/working-projects.rst |  8 +++
 4 files changed, 96 insertions(+), 8 deletions(-)

diff --git a/documentation/sdk-manual/appendix-customizing.rst b/documentation/sdk-manual/appendix-customizing.rst
index 9a76cc59d..23a437eb9 100644
--- a/documentation/sdk-manual/appendix-customizing.rst
+++ b/documentation/sdk-manual/appendix-customizing.rst
@@ -1,11 +1,17 @@
 .. SPDX-License-Identifier: CC-BY-SA-2.0-UK
 
-******************************
-Customizing the Extensible SDK
-******************************
+***************************************************
+Customizing the Extensible SDK standalone installer
+***************************************************
 
 This appendix describes customizations you can apply to the extensible
-SDK.
+SDK when using in the standalone installer version.
+
+.. note::
+
+   It is also possible to use the Extensible SDK functionality directly in a
+   Yocto build, avoiding separate installer artefacts. Please refer to
+   ":ref:`sdk-manual/extensible:Installing the Extensible SDK`"
 
 Configuring the Extensible SDK
 ==============================
diff --git a/documentation/sdk-manual/appendix-obtain.rst b/documentation/sdk-manual/appendix-obtain.rst
index ece378c75..9b7717a4c 100644
--- a/documentation/sdk-manual/appendix-obtain.rst
+++ b/documentation/sdk-manual/appendix-obtain.rst
@@ -4,8 +4,22 @@
 Obtaining the SDK
 *****************
 
+Working with the SDK components directly in a Yocto build
+=========================================================
+
+Please refer to section
+":ref:`sdk-manual/extensible:Setting up the Extensible SDK environment directly in a Yocto build`"
+
+Note that to use this feature effectively either a powerful build
+machine, or a well-functioning sstate cache infrastructure is required:
+otherwise significant time could be spent waiting for components to be built
+by bitbake from source code.
+
+Working with standalone SDK Installers
+======================================
+
 Locating Pre-Built SDK Installers
-=================================
+---------------------------------
 
 You can use existing, pre-built toolchains by locating and running an
 SDK installer script that ships with the Yocto Project. Using this
@@ -72,7 +86,7 @@ Follow these steps to locate and hand-install the toolchain:
    section for more information.
 
 Building an SDK Installer
-=========================
+-------------------------
 
 As an alternative to locating and downloading an SDK installer, you can
 build the SDK installer. Follow these steps:
diff --git a/documentation/sdk-manual/extensible.rst b/documentation/sdk-manual/extensible.rst
index ed9e43a2d..ae0587fec 100644
--- a/documentation/sdk-manual/extensible.rst
+++ b/documentation/sdk-manual/extensible.rst
@@ -41,6 +41,42 @@ functionality.
 Installing the Extensible SDK
 =============================
 
+Two ways to install the Extensible SDK
+--------------------------------------
+
+Extensible SDK can be installed in two different ways, and both have
+their own pros and cons:
+
+1. *Setting up the Extensible SDK environment directly in a Yocto build*. This
+avoids having to produce, test, distribute and maintain separate SDK installer
+archives, which can get very large. There is only one environment for the regular
+yocto build and the SDK and less code paths where things can go not according to plan.
+It's easier to update the SDK: it simply means updating the yocto layers with
+git fetch or layer management tooling. The SDK extensibility is better than in the
+second option: just run ``bitbake`` again to add more things to the sysroot, or add layers
+if even more things are required.
+
+2. *Setting up the Extensible SDK from a standalone installer*. This has the benefit of
+having a single, self-contained archive that includes all the needed binary artifacts.
+So nothing needs to be rebuild, and there is no need to provide a well-functioning
+binary artefact cache over the network for developers with underpowered laptops.
+
+Setting up the Extensible SDK environment directly in a Yocto build
+-------------------------------------------------------------------
+
+1. Set up all the needed layers and a yocto build directory, e.g. a regular yocto
+   build where ``bitbake`` can be executed.
+
+2. Run:
+    $ bitbake meta-ide-support
+    $ bitbake -c populate_sysroot gtk+3
+    (or any other target or native item that the application developer would need)
+    $ bitbake populate-sysroots
+
+
+Setting up the Extensible SDK from a standalone installer
+---------------------------------------------------------
+
 The first thing you need to do is install the SDK on your :term:`Build
 Host` by running the ``*.sh`` installation script.
 
@@ -136,7 +172,12 @@ Running the Extensible SDK Environment Setup Script
 ===================================================
 
 Once you have the SDK installed, you must run the SDK environment setup
-script before you can actually use the SDK. This setup script resides in
+script before you can actually use the SDK.
+
+When using a SDK directly in a Yocto build, you will find the script in
+``tmp/deploy/images/qemux86-64/`` in your build directory.
+
+When using a standalone SDK installer, this setup script resides in
 the directory you chose when you installed the SDK, which is either the
 default ``poky_sdk`` directory or the directory you chose during
 installation.
@@ -154,6 +195,11 @@ script is for an IA-based target machine using i586 tuning::
    SDK environment now set up; additionally you may now run devtool to perform development tasks.
    Run devtool --help for further details.
 
+When using the environment script directly in a Yocto build, it can
+be run similarly:
+
+   $ source tmp/deploy/images/qemux86-64/environment-setup-core2-64-poky-linux
+
 Running the setup script defines many environment variables needed in
 order to use the SDK (e.g. ``PATH``,
 :term:`CC`,
@@ -1215,10 +1261,24 @@ need to link to libGL but you are not sure which recipe provides libGL.
 You can use the following command to find out::
 
    $ devtool search libGL mesa
+   A free implementation of the OpenGL API
 
-A free implementation of the OpenGL API Once you know the recipe
+Once you know the recipe
 (i.e. ``mesa`` in this example), you can install it::
 
+When using the extensible SDK directly in a Yocto build
+-------------------------------------------------------
+
+In this scenario, the Yocto build tooling, e.g. ``bitbake``
+is directly accessible to build additional items, and it
+can simply be executed directly:
+
+   $ bitbake mesa
+   $ bitbake populate-sysroots
+
+When using a standalone installer for the Extensible SDK
+--------------------------------------------------------
+
    $ devtool sdk-install mesa
 
 By default, the ``devtool sdk-install`` command assumes
diff --git a/documentation/sdk-manual/working-projects.rst b/documentation/sdk-manual/working-projects.rst
index 7f8d9b849..12cb2efee 100644
--- a/documentation/sdk-manual/working-projects.rst
+++ b/documentation/sdk-manual/working-projects.rst
@@ -88,6 +88,10 @@ project:
 
       $ source /opt/poky/&DISTRO;/environment-setup-i586-poky-linux
 
+   Another example is sourcing the environment setup directly in a yocto
+   build::
+      $ source tmp/deploy/images/qemux86-64/environment-setup-core2-64-poky-linux
+
 3. *Create the configure Script:* Use the ``autoreconf`` command to
    generate the ``configure`` script.
    ::
@@ -279,6 +283,10 @@ example:
 
       $ source /opt/poky/&DISTRO;/environment-setup-i586-poky-linux
 
+   Another example is sourcing the environment setup directly in a yocto
+   build::
+      $ source tmp/deploy/images/qemux86-64/environment-setup-core2-64-poky-linux
+
 3. *Create the Makefile:* For this example, the Makefile contains
    two lines that can be used to set the :term:`CC` variable. One line is
    identical to the value that is set when you run the SDK environment
-- 
2.30.2

Re: [docs] [PATCH 1/2] kernel-dev: working with kernel using devtool does not require building and installing eSDK

[Michael Opdenacker]

Hi Alex,

On 8/2/22 10:38, Alexander Kanavin wrote:
> In fact, all the same tasks can be done directly from a Yocto build. Let's just
> drop the steps to produce and install the eSDK - if someone wants it, the SDK
> manual covers everything.
>
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>   documentation/kernel-dev/common.rst | 84 +++++------------------------
>   documentation/kernel-dev/intro.rst  |  8 +--
>   2 files changed, 16 insertions(+), 76 deletions(-)


Thanks for the patch.
Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
... and merged into master-next.

Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH 2/2] sdk-manual: describe how to use extensible SDK functionality directly in a Yocto build

[Michael Opdenacker]

Hi Alex,

On 8/2/22 10:38, Alexander Kanavin wrote:
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>   .../sdk-manual/appendix-customizing.rst       | 14 ++--
>   documentation/sdk-manual/appendix-obtain.rst  | 18 +++++-
>   documentation/sdk-manual/extensible.rst       | 64 ++++++++++++++++++-
>   documentation/sdk-manual/working-projects.rst |  8 +++
>   4 files changed, 96 insertions(+), 8 deletions(-)


Thanks for the patch!

I had a few issues indentation issues but I fixed them:

/home/mike/work/git/git.yoctoproject.org/yocto-docs/documentation/sdk-manual/extensible.rst:1269: 
WARNING: Literal block expected; none found.
/home/mike/work/git/git.yoctoproject.org/yocto-docs/documentation/sdk-manual/working-projects.rst:93: 
ERROR: Unexpected indentation.
/home/mike/work/git/git.yoctoproject.org/yocto-docs/documentation/sdk-manual/working-projects.rst:288: 
ERROR: Unexpected indentation.

> diff --git a/documentation/sdk-manual/appendix-obtain.rst b/documentation/sdk-manual/appendix-obtain.rst
> index ece378c75..9b7717a4c 100644
> --- a/documentation/sdk-manual/appendix-obtain.rst
> +++ b/documentation/sdk-manual/appendix-obtain.rst
> @@ -4,8 +4,22 @@
>   Obtaining the SDK
>   *****************
>   
> +Working with the SDK components directly in a Yocto build
> +=========================================================
> +
> +Please refer to section
> +":ref:`sdk-manual/extensible:Setting up the Extensible SDK environment directly in a Yocto build`"
> +
> +Note that to use this feature effectively either a powerful build
> +machine, or a well-functioning sstate cache infrastructure is required:
> +otherwise significant time could be spent waiting for components to be built
> +by bitbake from source code.


I replaced "bitbake" by "BitBake" here. That's how we usually refer to 
the tool.

>
> diff --git a/documentation/sdk-manual/extensible.rst b/documentation/sdk-manual/extensible.rst
> index ed9e43a2d..ae0587fec 100644
> --- a/documentation/sdk-manual/extensible.rst
> +++ b/documentation/sdk-manual/extensible.rst
> @@ -41,6 +41,42 @@ functionality.
>   Installing the Extensible SDK
>   =============================
>   
> +Two ways to install the Extensible SDK
> +--------------------------------------
> +
> +Extensible SDK can be installed in two different ways, and both have
> +their own pros and cons:
> +
> +1. *Setting up the Extensible SDK environment directly in a Yocto build*. This
> +avoids having to produce, test, distribute and maintain separate SDK installer
> +archives, which can get very large. There is only one environment for the regular
> +yocto build and the SDK and less code paths where things can go not according to plan.
> +It's easier to update the SDK: it simply means updating the yocto layers with

Here, I replaced "yocto" by "Yocto" too (4 times in this file).

> +git fetch or layer management tooling. The SDK extensibility is better than in the
> +second option: just run ``bitbake`` again to add more things to the sysroot, or add layers
> +if even more things are required.
> +
> +2. *Setting up the Extensible SDK from a standalone installer*. This has the benefit of
> +having a single, self-contained archive that includes all the needed binary artifacts.
> +So nothing needs to be rebuild, and there is no need to provide a well-functioning

s/rebuild/rebuilt/
Fixed

> diff --git a/documentation/sdk-manual/working-projects.rst b/documentation/sdk-manual/working-projects.rst
> index 7f8d9b849..12cb2efee 100644
> --- a/documentation/sdk-manual/working-projects.rst
> +++ b/documentation/sdk-manual/working-projects.rst
> @@ -88,6 +88,10 @@ project:
>   
>         $ source /opt/poky/&DISTRO;/environment-setup-i586-poky-linux
>   
> +   Another example is sourcing the environment setup directly in a yocto
> +   build::
> +      $ source tmp/deploy/images/qemux86-64/environment-setup-core2-64-poky-linux


Missing an empty line after "::" here. Fixed.

> +
>   3. *Create the configure Script:* Use the ``autoreconf`` command to
>      generate the ``configure`` script.
>      ::
> @@ -279,6 +283,10 @@ example:
>   
>         $ source /opt/poky/&DISTRO;/environment-setup-i586-poky-linux
>   
> +   Another example is sourcing the environment setup directly in a yocto
> +   build::
> +      $ source tmp/deploy/images/qemux86-64/environment-setup-core2-64-poky-linux


Same here. Fixed too.

After the fixes: Reviewed-by: Michael Opdenacker 
<michael.opdenacker@bootlin.com>
... and merged into master-next.

Thanks again!
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com