Re: [OE-core] [PATCH] migration-guides: update 4.2 migration and release notes

Re: [OE-core] [PATCH] migration-guides: update 4.2 migration and release notes

[Michael Opdenacker]

Greetings,

I copied the OE-core mailing list to get the attention of contributors. 
If you want your new features to be featured in the release and 
migration notes for 4.2, it's time to speak!

What I wrote so far is a very early draft meant to help you detect 
what's missing and correct what I've written so far.

See my questions and comments below...

On 05.04.23 at 19:46, Michael Opdenacker via lists.openembedded.org wrote:
> diff --git a/documentation/migration-guides/migration-4.2.rst b/documentation/migration-guides/migration-4.2.rst
> index 1339411b3e..3b04e072cb 100644
> ...
> +Rust: mandatory checksums for crates
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +This release now supports checksums for Rust crates and make
> +them mandatory for each crate in a recipe. See :yocto_git:`python3_bcrypt recipe changes
> +</poky/commit/?h=mickledore&id=0dcb5ab3462fdaaf1646b05a00c7150eea711a9a>`
> +for example.
> +
> +The ``cargo-update-recipe-crates`` utility
> +:yocto_git:`has been extended </poky/commit/?h=mickledore&id=eef7fbea2c5bf59369390be4d5efa915591b7b22>`
> +to include such checksums.


Any more Rust changes we should mention in the migration notes?


> diff --git a/documentation/migration-guides/release-notes-4.2.rst b/documentation/migration-guides/release-notes-4.2.rst
> index edafe378b0..1a60fe5490 100644
> --- a/documentation/migration-guides/release-notes-4.2.rst
> +++ b/documentation/migration-guides/release-notes-4.2.rst


Here in the release notes, I didn't work on the detailed changes, such 
as version changes for the kernel and many recipes. I'm trying to help 
with the first part showing the big picture.

>   
> -     addpylib ${LAYERDIR}/lib oe
> +      addpylib ${LAYERDIR}/lib oe
> +
> +-  BitBake has seen multiple internal changes that may impact
> +   memory and disk usage as well as parsing time, in particular:
> +
> +   -  BitBake's Cooker server is now multithreaded.
> +
> +   -  BitBake's cache has been extended to include more hash
> +      debugging data, but has also been optimized to :yocto_git:`compress
> +      cache data <https://git.yoctoproject.org/poky/commit/?h=mickledore&id=7d010055e2af3294e17db862f42664ca689a9356>`.
> +
> +   -  BitBake's Cooker server :yocto_git:`can now be pinged
> +      https://git.yoctoproject.org/poky/commit/?h=mickledore&id=26f255da09>`
> +      from the UI.


More BitBake changes we should mention?

> +
> +-  Rust improvements:
> +
> +   -  This release adds Cargo support on the target, and includes
> +      automated QA tests for this functionality.
> +
> +   -  It also supports checksums for Rust crates and makes
> +      them mandatory for each crate in a recipe.


More Rust changes we should highlight?

> +-  Miscellaneous changes:
> +
> +   -  Supporting 64 bit dates on 32 bit platforms: several packages have been
> +      updated to pass Y2038 tests.


By the way, what's the status of Y2038 support in Mickledore? Are all 
tests passing now?

To make the drafts easier to review, they have been compiled in the 
master-next branch:
https://docs.yoctoproject.org/next/migration-guides/release-4.2.html

Thanks in advance
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH] migration-guides: update 4.2 migration and release notes

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 2782 bytes --]

On Wed, 5 Apr 2023 at 20:40, Michael Opdenacker <
michael.opdenacker@bootlin.com> wrote:

> Greetings,
>
> I copied the OE-core mailing list to get the attention of contributors.
> If you want your new features to be featured in the release and
> migration notes for 4.2, it's time to speak!
>
> What I wrote so far is a very early draft meant to help you detect
> what's missing and correct what I've written so far.
>
> See my questions and comments below...
>
> On 05.04.23 at 19:46, Michael Opdenacker via lists.openembedded.org wrote:
> > diff --git a/documentation/migration-guides/migration-4.2.rst
> b/documentation/migration-guides/migration-4.2.rst
> > index 1339411b3e..3b04e072cb 100644
> > ...
> > +Rust: mandatory checksums for crates
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +This release now supports checksums for Rust crates and make
> > +them mandatory for each crate in a recipe. See
> :yocto_git:`python3_bcrypt recipe changes
> >
> +</poky/commit/?h=mickledore&id=0dcb5ab3462fdaaf1646b05a00c7150eea711a9a>`
> > +for example.
> > +
> > +The ``cargo-update-recipe-crates`` utility
> > +:yocto_git:`has been extended
> </poky/commit/?h=mickledore&id=eef7fbea2c5bf59369390be4d5efa915591b7b22>`
> > +to include such checksums.
>
>
> Any more Rust changes we should mention in the migration notes?
>
>
Maybe mention a specific procedure to update a cargo based recipe from
scratch (i.e. a recipe that
didn't use the cargo-update-recipe-crates inheritance):

To bootstrap a cargo base recipe which have not the checksum list, below are
the steps to make:
  - make the recipe inherit cargo-update-recipe-crates
  - remove all crate:// from the recipe
  - create an empty ${BPN}-crates.inc file and make your recipe requires it
  - execute "bitbake -c update_crates your_recipe"
  - copy paste the output of bitbake about the missing checksums in your
${BPN}-crates.inc

The last step requires a bitbake patch to show all the missing checksum
which I don't know
if it will be in 4.2.


> > +
> > +-  Rust improvements:
> > +
> > +   -  This release adds Cargo support on the target, and includes
> > +      automated QA tests for this functionality.
> > +
> > +   -  It also supports checksums for Rust crates and makes
> > +      them mandatory for each crate in a recipe.
>
>
> More Rust changes we should highlight?
>

This one seems redundant with the previously mentioned rust changes, isn't
it ?
 Nevertheless, I'd maybe add a line about the fact that now we can develop
rust project which contains "private" crate (i.e. crates that would be
hosted
in a git repository instead of crates.io registry), and by develop, it
means we
can of course build such a package but perform devtool operation on it (at
least devtool modify and devtool deploy-target)

[-- Attachment #2: Type: text/html, Size: 3883 bytes --]

Re: [docs] [OE-core] [PATCH] migration-guides: update 4.2 migration and release notes

[Michael Opdenacker]

Hi Frederic

On 05.04.23 at 21:04, Frederic Martinsons wrote:
>
>
>
>     Any more Rust changes we should mention in the migration notes?
>
>
> Maybe mention a specific procedure to update a cargo based recipe from 
> scratch (i.e. a recipe that
> didn't use the cargo-update-recipe-crates inheritance):
>
> To bootstrap a cargo base recipe which have not the checksum list, 
> below are
> the steps to make:
>   - make the recipe inherit cargo-update-recipe-crates
>   - remove all crate:// from the recipe
>   - create an empty ${BPN}-crates.inc file and make your recipe 
> requires it
>   - execute "bitbake -c update_crates your_recipe"
>   - copy paste the output of bitbake about the missing checksums in 
> your ${BPN}-crates.inc
>
> The last step requires a bitbake patch to show all the missing 
> checksum which I don't know
> if it will be in 4.2.


Do you mean this one?
https://git.openembedded.org/bitbake/commit/?h=2.4&id=dafa07c080e05975b6319b5adf78a9691c6b6643

It just got merged.
Thanks for the tip.

Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [OE-core] [PATCH] migration-guides: update 4.2 migration and release notes

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 1374 bytes --]

Le jeu. 6 avr. 2023, 10:23, Michael Opdenacker <
michael.opdenacker@bootlin.com> a écrit :

> Hi Frederic
>
> On 05.04.23 at 21:04, Frederic Martinsons wrote:
> >
> >
> >
> >     Any more Rust changes we should mention in the migration notes?
> >
> >
> > Maybe mention a specific procedure to update a cargo based recipe from
> > scratch (i.e. a recipe that
> > didn't use the cargo-update-recipe-crates inheritance):
> >
> > To bootstrap a cargo base recipe which have not the checksum list,
> > below are
> > the steps to make:
> >   - make the recipe inherit cargo-update-recipe-crates
> >   - remove all crate:// from the recipe
> >   - create an empty ${BPN}-crates.inc file and make your recipe
> > requires it
> >   - execute "bitbake -c update_crates your_recipe"
> >   - copy paste the output of bitbake about the missing checksums in
> > your ${BPN}-crates.inc
> >
> > The last step requires a bitbake patch to show all the missing
> > checksum which I don't know
> > if it will be in 4.2.
>
>
> Do you mean this one?
>
> https://git.openembedded.org/bitbake/commit/?h=2.4&id=dafa07c080e05975b6319b5adf78a9691c6b6643
>
> It just got merged.
> Thanks for the tip.
>
> Cheers
> Michael.
>
> --
> Michael Opdenacker, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>

Yes that's the one. Great.

[-- Attachment #2: Type: text/html, Size: 2240 bytes --]

[PATCH v2] migration-guides: update 4.2 migration and release notes

[michael.opdenacker]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
CC: Frederic Martinsons <frederic.martinsons@gmail.com>
CC: Marta Rybczynska <rybczynska@gmail.com>
CC: Paul Eggleton <paul.eggleton@linux.microsoft.com>
CC: Alex Kiernan <alex.kiernan@gmail.com>

---

Changes in V2:
- Add details about filling checksums using the 'cargo-update-recipe-crates'
  class, as suggested by Frederic Martinsons.
---
 .../migration-guides/migration-4.2.rst        | 84 ++++++++++++++++++-
 .../migration-guides/release-notes-4.2.rst    | 68 +++++++++++++--
 2 files changed, 143 insertions(+), 9 deletions(-)

diff --git a/documentation/migration-guides/migration-4.2.rst b/documentation/migration-guides/migration-4.2.rst
index 1339411b3e..3b60b9a807 100644
--- a/documentation/migration-guides/migration-4.2.rst
+++ b/documentation/migration-guides/migration-4.2.rst
@@ -18,11 +18,78 @@ BitBake and OpenEmbedded-Core are now relying on Python 3.8,
 making it a requirement to use a distribution providing at least this
 version, or to use :term:`buildtools`.
 
+.. _migration-4.2-gcc-8.0:
+
+gcc 8.0 is now the minumum required GNU C compiler version
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This version, released in 2018, is a minimum requirement
+to build the ``mesa-native`` recipe.
+
+.. _migration-4.2-new-nvd-api:
+
+Fetching the NVD vulnerability database through the 2.0 API
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This new version adds a new fetcher for the NVD database using the 2.0 API,
+as the 1.0 API will be retired in 2023.
+
+The implementation changes as little as possible, keeping the current
+database format (but using a different database file for the transition
+period), with a notable exception of not using the META table.
+
+Here are minor changes that you may notice:
+
+-  The database starts in 1999 instead of 2002
+-  The complete fetch is longer (30 minutes typically)
+
+.. _migration-4.2-rust-crate-checksums:
+
+Rust: mandatory checksums for crates
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This release now supports checksums for Rust crates and make
+them mandatory for each crate in a recipe. See :yocto_git:`python3_bcrypt recipe changes
+</poky/commit/?h=mickledore&id=0dcb5ab3462fdaaf1646b05a00c7150eea711a9a>`
+for example.
+
+The ``cargo-update-recipe-crates`` utility
+:yocto_git:`has been extended </poky/commit/?h=mickledore&id=eef7fbea2c5bf59369390be4d5efa915591b7b22>`
+to include such checksums. So, in case you need to add the list of checksums
+to a recipe just inheriting the :ref:`ref-classes-cargo` class so far, you can
+follow these steps:
+
+#.  Make the recipe inherit :ref:`ref-classes-cargo-update-recipe-crates`
+#.  Remove all ``crate://`` lines from the recipe
+#.  Create an empty ``${BPN}-crates.inc`` file and make your recipe require it
+#.  Execute ``bitbake -c update_crates your_recipe``
+#.  Copy and paste the output of BitBake about the missing checksums into the
+    ``${BPN}-crates.inc`` file.
+
 .. _migration-4.2-qa-checks:
 
 QA check changes
 ~~~~~~~~~~~~~~~~
 
+.. _migration-4.2-supported-distributions:
+
+Supported distributions
+~~~~~~~~~~~~~~~~~~~~~~~
+
+This release supports running BitBake on new GNU/Linux distributions:
+
+-  Fedora 36 and 37
+-  AlmaLinux 8.7 and 9.1
+-  OpenSuse 15.4
+
+On the other hand, some earlier distributions are no longer supported:
+
+-  Debian 10.x
+-  Fedora 34 and 35
+-  AlmaLinux 8.5
+
+See :ref:`all supported distributions <system-requirements-supported-distros>`.
+
 .. _migration-4.2-misc-changes:
 
 Miscellaneous changes
@@ -30,6 +97,7 @@ Miscellaneous changes
 
 -  The ``OEBasic`` signature handler (see :term:`BB_SIGNATURE_HANDLER`) has been
    removed.
+  
 
 .. _migration-4.2-removed-variables:
 
@@ -38,7 +106,7 @@ Removed variables
 
 The following variables have been removed:
 
--  ``SERIAL_CONSOLE``, deprecated since version 2.6, replaced by :term:``SERIAL_CONSOLES``.
+-  ``SERIAL_CONSOLE``, deprecated since version 2.6, replaced by :term:`SERIAL_CONSOLES`.
 
 .. _migration-4.2-removed-recipes:
 
@@ -47,3 +115,17 @@ Removed recipes
 
 The following recipes have been removed in this release:
 
+-  ``python3-picobuild``: after switching to ``python3-build``
+-  ``python3-strict-rfc3339``: unmaintained and not needed by anything in
+   :oe_git:`openembedded-core </openembedded-core>`
+   or :oe_git:`meta-openembedded </meta-openembedded>`.
+
+.. _migration-4.2-removed-classes:
+
+Removed classes
+~~~~~~~~~~~~~~~
+
+The following classes have been removed in this release:
+
+-  ``rust-bin``: no longer used in Poky
+
diff --git a/documentation/migration-guides/release-notes-4.2.rst b/documentation/migration-guides/release-notes-4.2.rst
index edafe378b0..1a60fe5490 100644
--- a/documentation/migration-guides/release-notes-4.2.rst
+++ b/documentation/migration-guides/release-notes-4.2.rst
@@ -6,23 +6,75 @@ Release notes for 4.2 (mickledore)
 New Features / Enhancements in 4.2
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-- Python 3.8 is the minimum Python version required on the build host.
-  For host distributions that do not provide it, this is included as part of the
-  :term:`buildtools` tarball.
+-  Python 3.8 is the minimum Python version required on the build host.
+   For host distributions that do not provide it, this is included as part of the
+   :term:`buildtools` tarball.
 
-- This release now supports a new ``addpylib`` directive to enable
-  Python libraries within layers.
+-  BitBake in this release now supports a new ``addpylib`` directive to enable
+   Python libraries within layers.
 
-  This directive should be added to your layer configuration,
-  as in the below example from ``meta/conf/layer.conf``::
+   This directive should be added to your layer configuration
+   as in the below example from ``meta/conf/layer.conf``::
 
-     addpylib ${LAYERDIR}/lib oe
+      addpylib ${LAYERDIR}/lib oe
+
+-  BitBake has seen multiple internal changes that may impact
+   memory and disk usage as well as parsing time, in particular:
+
+   -  BitBake's Cooker server is now multithreaded.
+
+   -  BitBake's cache has been extended to include more hash
+      debugging data, but has also been optimized to :yocto_git:`compress
+      cache data <https://git.yoctoproject.org/poky/commit/?h=mickledore&id=7d010055e2af3294e17db862f42664ca689a9356>`.
+
+   -  BitBake's Cooker server :yocto_git:`can now be pinged
+      https://git.yoctoproject.org/poky/commit/?h=mickledore&id=26f255da09>`
+      from the UI.
 
 -  Architecture-specific enhancements:
 
+   -  This release adds initial support for the
+      :wikipedia:`LoongArch <Loongson#LoongArch>`
+      (``loongarch64``) architecture, though there is no testing for it yet.
+
 -  Kernel-related enhancements:
 
 -  QEMU/runqemu enhancements:
 
 -  Image-related enhancements:
 
+-  New variables:
+
+   - :term:`VOLATILE_TMP_DIR` allows to specify
+     whether ``/tmp`` should be on persistent storage
+     or in RAM.
+
+-  Rust improvements:
+
+   -  This release adds Cargo support on the target, and includes
+      automated QA tests for this functionality.
+
+   -  It also supports checksums for Rust crates and makes
+      them mandatory for each crate in a recipe.
+
+-  Testing:
+
+   -  The ``test-results`` script has been substantially improved
+      to better process and help to interpret test results.
+      See `this example <https://autobuilder.yocto.io/pub/non-release/20230405-6/testresults/buildperf-alma8/perf-alma8_master_20230405030047_38e3769a72.html>`__.
+
+   -  It is now possible to track regression changes between releases using
+      ``resulttool``. Here is an example command, which allowed to spot and fix
+      a regression in the ``quilt`` ptest::
+
+         yocto_testresults_query.py regression-report 4.2_M1 4.2_M2
+
+   -  This release adds support for parallel ptest execution with a ptest per image.
+      This takes ptest execution time from 3.5 hours to around 45 minutes on the autobuilder.
+
+-  Miscellaneous changes:
+
+   -  Supporting 64 bit dates on 32 bit platforms: several packages have been
+      updated to pass Y2038 tests.
+
+   -  Many packages were updated to add large file support.
-- 
2.34.1

Re: [OE-core] [PATCH v2] migration-guides: update 4.2 migration and release notes

[Richard Purdie]

On Thu, 2023-04-06 at 11:12 +0200, Michael Opdenacker via
lists.openembedded.org wrote:
> From: Michael Opdenacker <michael.opdenacker@bootlin.com>
> 
> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> CC: Frederic Martinsons <frederic.martinsons@gmail.com>
> CC: Marta Rybczynska <rybczynska@gmail.com>
> CC: Paul Eggleton <paul.eggleton@linux.microsoft.com>
> CC: Alex Kiernan <alex.kiernan@gmail.com>
> 
> ---
> 
> Changes in V2:
> - Add details about filling checksums using the 'cargo-update-recipe-crates'
>   class, as suggested by Frederic Martinsons.

We should add something about ptest changes:


"""
The ptest images have changed structure in this release. The underlying
core-image-ptest recipe now uses BBCLASSEXTEND to create a variant for
each ptest enabled recipe in OE-Core. 

For example, this means that "core-image-ptest-bzip2", "core-image-
ptest-lttng-tools" and many more image targets now exist and can be
built/tested individually. 

The core-image-ptest-all and core-image-ptest-fast targets are now
wrappers that target groups of individual images and means that the
tests can be executed in parallel during our automated testing. This
also means the dependencies are more accurately tested.
"""

Cheers,

Richard

Re: [OE-core] [PATCH v2] migration-guides: update 4.2 migration and release notes

[Michael Opdenacker]

On 06.04.23 at 12:12, Richard Purdie wrote:
> The ptest images have changed structure in this release. The underlying
> core-image-ptest recipe now uses BBCLASSEXTEND to create a variant for
> each ptest enabled recipe in OE-Core.
>
> For example, this means that "core-image-ptest-bzip2", "core-image-
> ptest-lttng-tools" and many more image targets now exist and can be
> built/tested individually.
>
> The core-image-ptest-all and core-image-ptest-fast targets are now
> wrappers that target groups of individual images and means that the
> tests can be executed in parallel during our automated testing. This
> also means the dependencies are more accurately tested.


Thanks Richard. This will be included in the next update I send.
Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com