[PATCH 0/2] Doc updates for the 3.1 release

[PATCH 0/2] Doc updates for the 3.1 release

[Paul Eggleton]

Migration guide for 3.1, and since I noticed it was missing there's an
update for the npm fetcher changes.


The following changes since commit 7f391a0656442e7ee2f3dd48b848befb7f461aec:

  dev-manual: use "git", not "apmd", to explain checksum names (2020-04-06 16:52:58 +0100)

are available in the Git repository at:

  git://git.yoctoproject.org/poky-contrib paule/doc-updates-3.1
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/doc-updates-3.1

Paul Eggleton (2):
  dev-manual: basic updates for npm fetcher changes
  ref-manual: add migration section for 3.1

 .../dev-manual/dev-manual-common-tasks.xml    |  52 ++---
 documentation/ref-manual/migration.xml        | 201 ++++++++++++++++++
 2 files changed, 222 insertions(+), 31 deletions(-)

-- 
2.20.1

[PATCH 1/2] dev-manual: basic updates for npm fetcher changes

[Paul Eggleton]

From: Paul Eggleton <paul.eggleton@microsoft.com>

Update the npm:// URL and the generated npm recipe, and remove
references to NPM_SHRINKWRAP, NPM_LOCKDOWN and lockdown functionality in
general.

Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
---
 .../dev-manual/dev-manual-common-tasks.xml    | 52 ++++++++-----------
 1 file changed, 21 insertions(+), 31 deletions(-)

diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml
index 2b8868b62..185e1d769 100644
--- a/documentation/dev-manual/dev-manual-common-tasks.xml
+++ b/documentation/dev-manual/dev-manual-common-tasks.xml
@@ -10590,7 +10590,7 @@
                     <filename>devtool</filename> and the NPM fetcher to
                     create the recipe:
                     <literallayout class='monospaced'>
-     $ devtool add "npm://registry.npmjs.org;name=cute-files;version=1.0.2"
+     $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
                     </literallayout>
                     The <filename>devtool add</filename> command runs
                     <filename>recipetool create</filename> and uses the
@@ -10615,25 +10615,13 @@
                 </para>
 
                 <para>
-                    <filename>recipetool</filename> creates "shrinkwrap" and
-                    "lockdown" files for your recipe.
+                    <filename>recipetool</filename> creates a "shrinkwrap" file
+                    for your recipe.
                     Shrinkwrap files capture the version of all dependent
                     modules.
                     Many packages do not provide shrinkwrap files.
                     <filename>recipetool</filename> create a shrinkwrap
                     file as it runs.
-                    You can replace the shrinkwrap file with your own file
-                    by setting the <filename>NPM_SHRINKWRAP</filename>
-                    variable.
-                </para>
-
-                <para>
-                    Lockdown files contain the checksum for each module
-                    to determine if your users download the same files when
-                    building with a recipe.
-                    Lockdown files ensure that dependencies have not been
-                    changed and that your NPM registry is still providing
-                    the same file.
                     <note>
                         A package is created for each sub-module.
                         This policy is the only practical way to have the
@@ -10648,23 +10636,26 @@
                     <literallayout class='monospaced'>
      $ devtool edit-recipe cute-files
      SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
-     LICENSE = "BSD-3-Clause &amp; Unknown &amp; MIT &amp; ISC"
+     LICENSE = "MIT &amp; ISC &amp; Unknown"
      LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
-                         file://node_modules/content-disposition/LICENSE;md5=c6e0ce1e688c5ff16db06b7259e9cd20 \
-                         file://node_modules/express/LICENSE;md5=5513c00a5c36cd361da863dd9aa8875d \
+                         file://node_modules/toidentifier/LICENSE;md5=1a261071a044d02eb6f2bb47f51a3502 \
+                         file://node_modules/debug/LICENSE;md5=ddd815a475e7338b0be7a14d8ee35a99 \
                          ...
 
-     SRC_URI = "npm://registry.npmjs.org;name=cute-files;version=${PV}"
-     NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"
-     NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"
+     SRC_URI = " \
+         npm://registry.npmjs.org/;package=cute-files;version=${PV} \
+         npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
+     "
+
+     S = "${WORKDIR}/npm"
+
      inherit npm
-     # Must be set after inherit npm since that itself sets S
-     S = "${WORKDIR}/npmpkg"
 
-     LICENSE_${PN}-content-disposition = "MIT"
-     ...
-     LICENSE_${PN}-express = "MIT"
      LICENSE_${PN} = "MIT"
+     LICENSE_${PN}-accepts = "MIT"
+     LICENSE_${PN}-array-flatten = "MIT"
+     ...
+     LICENSE_${PN}-vary = "MIT"
                     </literallayout>
                     Three key points exist in the previous example:
                     <itemizedlist>
@@ -10757,11 +10748,10 @@
                     However, the <filename>SRC_URI</filename> looks like the
                     following:
                     <literallayout class='monospaced'>
-     SRC_URI = "git://github.com/martinaglv/cute-files.git;protocol=https \
-               npm://registry.npmjs.org;name=commander;version=2.9.0;subdir=node_modules/commander \
-               npm://registry.npmjs.org;name=express;version=4.14.0;subdir=node_modules/express \
-               npm://registry.npmjs.org;name=content-disposition;version=0.3.0;subdir=node_modules/content-disposition \
-               "
+     SRC_URI = " \
+         git://github.com/martinaglv/cute-files.git;protocol=https \
+         npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
+         "
                     </literallayout>
                     In this example, the main module is taken from the Git
                     repository and dependents are taken from the NPM registry.
-- 
2.20.1

[PATCH 2/2] ref-manual: add migration section for 3.1

[Paul Eggleton]

From: Paul Eggleton <paul.eggleton@microsoft.com>

Migration information for the 3.1 release.

Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
---
 documentation/ref-manual/migration.xml | 201 +++++++++++++++++++++++++
 1 file changed, 201 insertions(+)

diff --git a/documentation/ref-manual/migration.xml b/documentation/ref-manual/migration.xml
index 03efc7be2..9422b5a27 100644
--- a/documentation/ref-manual/migration.xml
+++ b/documentation/ref-manual/migration.xml
@@ -7095,6 +7095,207 @@ id=f4d4f99cfbc2396e49c1613a7d237b9e57f06f81'>commit message</ulink>.
     </section>
 </section>
 
+
+<section id='moving-to-the-yocto-project-3.1-release'>
+    <title>Moving to the Yocto Project 3.1 Release</title>
+
+    <para>
+        This section provides migration information for moving to the
+        Yocto Project 3.1 Release from the prior release.
+    </para>
+
+    <section id='migration-3.1-minimum-system-requirements'>
+        <title>Minimum system requirements</title>
+
+        <para>
+            The following versions / requirements of build host components have been updated:
+            <itemizedlist>
+                <listitem><para>gcc 5.0</para></listitem>
+                <listitem><para>python 3.5</para></listitem>
+                <listitem><para>tar 1.28</para></listitem>
+                <listitem><para><filename>rpcgen</filename> is now required on the host (part of the <filename>libc-dev-bin</filename> package on Ubuntu, Debian and related distributions, and the <filename>glibc</filename> package on RPM-based distributions).</para></listitem>
+            </itemizedlist>
+            
+            Additionally, the <filename>makeinfo</filename> and <filename>pod2man</filename>
+            tools are <emphasis>no longer</emphasis> required on the host.
+        </para>
+    </section>
+
+    <section id='migration-3.1-mpc8315e-rdb-removed'>
+        <title>mpc8315e-rdb machine removed</title>
+
+        <para>
+            The MPC8315E-RDB machine is old/obsolete and unobtainable, thus given the maintenance burden
+            the <filename>mpc8315e-rdb</filename> machine configuration that supported it has been removed
+            in this release. The removal does leave a gap in official PowerPC reference hardware
+            support; this may change in future if a suitable machine with accompanying support resources
+            is found.
+        </para>
+    </section>
+    
+    <section id='migration-3.1-python-2-removed'>
+        <title>Python 2 removed</title>
+
+        <para>
+            Due to the expiration of upstream support in January 2020, support for Python 2 has now been removed; it is recommended that you use Python 3 instead. If absolutely needed there is a meta-python2 community layer containing Python 2, related classes and various Python 2-based modules, however it should not be considered as supported.
+        </para>
+    </section>
+
+    <section id='migration-3.1-reproducible-builds'>
+        <title>Reproducible builds now enabled by default</title>
+
+        <para>
+            In order to avoid unnecessary differences in output files (aiding binary reproducibility), the Poky distribution configuration (<filename><link linkend='var-DISTRO'>DISTRO</link> = "poky"</filename>) now inherits the <filename>reproducible_build</filename> class by default.
+        </para>
+    </section>
+    
+    <section id='migration-3.1-ptest-feature-impact'>
+        <title>Impact of ptest feature is now more significant</title>
+
+        <para>
+            The Poky distribution configuration (<filename><link linkend='var-DISTRO'>DISTRO</link> = "poky"</filename>) enables ptests by default to enable runtime testing of various components. In this release, a dependency needed to be added that has resulted in a significant increase in the number of components that will be built just when building a simple image such as core-image-minimal. If you do not need runtime tests enabled for core components, then it is recommended that you remove "ptest" from <link linkend='var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></link> to save a significant amount of build time e.g. by adding the following in your configuration:
+
+            <literallayout class='monospaced'>
+      DISTRO_FEATURES_remove = "ptest"
+            </literallayout>
+        </para>
+    </section>
+
+    <section id='migration-3.1-removed-recipes'>
+        <title>Removed recipes</title>
+
+        <para>
+            The following recipes have been removed:
+
+            <itemizedlist>
+                <listitem><para><filename>chkconfig</filename>: obsolete</para></listitem>
+                <listitem><para><filename>console-tools</filename>: obsolete</para></listitem>
+                <listitem><para><filename>enchant</filename>: replaced by <filename>enchant2</filename></para></listitem>
+                <listitem><para><filename>foomatic-filters</filename>: obsolete</para></listitem>
+                <listitem><para><filename>libidn</filename>: no longer needed, moved to meta-oe</para></listitem>
+                <listitem><para><filename>libmodulemd</filename>: replaced by <filename>libmodulemd-v1</filename></para></listitem>
+                <listitem><para><filename>linux-yocto</filename>: drop 4.19, 5.2 version recipes (5.4 now provided)</para></listitem>
+                <listitem><para><filename>nspr</filename>: no longer needed, moved to meta-oe</para></listitem>
+                <listitem><para><filename>nss</filename>: no longer needed, moved to meta-oe</para></listitem>
+                <listitem><para><filename>python</filename>: Python 2 removed (Python 3 preferred)</para></listitem>
+                <listitem><para><filename>python-setuptools</filename>: Python 2 version removed (python3-setuptools preferred)</para></listitem>
+                <listitem><para><filename>sysprof</filename>: no longer needed, moved to meta-oe</para></listitem>
+                <listitem><para><filename>texi2html</filename>: obsolete</para></listitem>
+                <listitem><para><filename>u-boot-fw-utils</filename>: functionally replaced by <filename>libubootenv</filename></para></listitem>
+            </itemizedlist>
+        </para>
+    </section>
+
+    <section id='migration-3.1-features-check'>
+        <title>features_check class replaces distro_features_check</title>
+
+        <para>
+            The <filename>distro_features_check</filename> class has had its functionality expanded, now supporting <filename>ANY_OF_MACHINE_FEATURES</filename>, <filename>REQUIRED_MACHINE_FEATURES</filename>, <filename>CONFLICT_MACHINE_FEATURES</filename>, <filename>ANY_OF_COMBINED_FEATURES</filename>, <filename>REQUIRED_COMBINED_FEATURES</filename>, <filename>CONFLICT_COMBINED_FEATURES</filename>. As a result the class has now been renamed to <filename>features_check</filename>; the <filename>distro_features_check</filename> class still exists but generates a warning and redirects to the new class. In preparation for a future removal of the old class it is recommended that you update recipes currently inheriting <filename>distro_features_check</filename> to inherit <filename>features_check</filename> instead.
+        </para>
+    </section>
+
+    <section id='migration-3.1-removed-classes'>
+        <title>Removed classes</title>
+
+        <para>
+            The following classes have been removed:
+
+            <itemizedlist>
+                <listitem><para><filename>distutils-base</filename>: moved to meta-python2</para></listitem>
+                <listitem><para><filename>distutils</filename>: moved to meta-python2</para></listitem>
+                <listitem><para><filename>libc-common</filename>: merged into the glibc recipe as nothing else used it.</para></listitem>
+                <listitem><para><filename>python-dir</filename>: moved to meta-python2</para></listitem>
+                <listitem><para><filename>pythonnative</filename>: moved to meta-python2</para></listitem>
+                <listitem><para><filename>setuptools</filename>: moved to meta-python2</para></listitem>
+                <listitem><para><filename>tinderclient</filename>: dropped as it was obsolete.</para></listitem>
+            </itemizedlist>
+        </para>
+    </section>
+
+    <section id='migration-3.1-src-uri-checksums'>
+        <title>SRC_URI checksum behaviour</title>
+
+        <para>
+            Previously, recipes by tradition included both SHA256 and MD5 checksums for remotely fetched files in <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>, even though only one is actually mandated. However, the MD5 checksum does not add much given its inherent weakness; thus when a checksum fails only the SHA256 sum will now be printed. The md5sum will still be verified if it is specified.
+        </para>
+    </section>
+
+
+    <section id='migration-3.1-npm'>
+        <title>npm fetcher changes</title>
+
+        <para>
+            The npm fetcher has been completely reworked in this release. The npm fetcher now only fetches the package source itself and no longer the dependencies; there is now also an npmsw fetcher which explicitly fetches the shrinkwrap file and the dependencies. This removes the slightly awkward <filename>NPM_LOCKDOWN</filename> and <filename>NPM_SHRINKWRAP</filename> variables which pointed to local files; the lockdown file is no longer needed at all. Additionally, the package name in <filename>npm://</filename> entries in <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link> is now specified using a <filename>package</filename> parameter instead of the earlier <filename>name</filename> which overlapped with the generic <filename>name</filename> parameter. All recipes using the npm fetcher will need to be changed as a result. 
+        </para>
+        <para>
+            An example of the new scheme:
+            <literallayout class='monospaced'>
+SRC_URI = "npm://registry.npmjs.org;package=array-flatten;version=1.1.1 \
+           npmsw://${THISDIR}/npm-shrinkwrap.json"
+            </literallayout>
+            Another example where the sources are fetched from git rather than an npm repository:
+            <literallayout class='monospaced'>
+SRC_URI = "git://github.com/foo/bar.git;protocol=https \
+           npmsw://${THISDIR}/npm-shrinkwrap.json"
+            </literallayout>
+        </para>
+        <para>
+            devtool and recipetool have also been updated to match with the npm fetcher changes. Other than producing working and more complete recipes for npm sources, there is also a minor change to the command line for devtool: the <filename>--fetch-dev</filename> option has been renamed to <filename>--npm-dev</filename> as it is npm-specific.
+        </para>
+    </section>
+    
+    
+    <section id='migration-3.1-packaging-changes'>
+        <title>Packaging changes</title>
+
+        <para>
+            <itemizedlist>
+                <listitem><para><filename>intltool</filename> has been removed from <filename>packagegroup-core-sdk</filename> as it is rarely needed to build modern software - gettext can do most of the things it used to be needed for. <filename>intltool</filename> has also been removed from <filename>packagegroup-core-self-hosted</filename> as it is not needed to for standard builds.</para></listitem>
+                <listitem><para>git: <filename>git-am</filename>, <filename>git-difftool</filename>, <filename>git-submodule</filename>, and <filename>git-request-pull</filename> are no longer perl-based, so are now installed with the main <filename>git</filename> package instead of within <filename>git-perltools</filename>.</para></listitem>
+                <listitem><para>The <filename>ldconfig</filename> binary built as part of glibc has now been moved to its own <filename>ldconfig</filename> package (note no <filename>glibc-</filename> prefix). This package is in the <link linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link> of the main <filename>glibc</filename> package if <filename>ldconfig</filename> is present in <link linkend='var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></link>.</para></listitem>
+                <listitem><para><filename>libevent</filename> now splits each shared library into its own package (as Debian does). Since these are shared libraries and will be pulled in through the normal shared library dependency handling, there should be no impact to existing configurations other than less unnecessary libraries being installed in some cases.</para></listitem>
+                <listitem><para>linux-firmware now has a new package for <filename>bcm4366c</filename> and includes available NVRAM config files into the <filename>bcm43340</filename>, <filename>bcm43362</filename>, <filename>bcm43430</filename> and <filename>bcm4356-pcie</filename> packages.</para></listitem>
+                <listitem><para><filename>harfbuzz</filename> now splits the new <filename>libharfbuzz-subset.so</filename> library into its own package to reduce the main package size in cases where <filename>libharfbuzz-subset.so</filename> is not needed.</para></listitem>
+            </itemizedlist>
+        </para>
+    </section>
+    
+    <section id='migration-3.1-package-qa-warnings'>
+        <title>Additional warnings</title>
+
+        <para>
+            Warnings will now be shown at <filename>do_package_qa</filename> time in the following circumstances:
+            
+            <itemizedlist>
+                <listitem><para>A recipe installs <filename>.desktop</filename> files containing <filename>MimeType</filename> keys but does not inherit the new <filename>mime-xdg</filename> class</para></listitem>
+                <listitem><para>A recipe installs <filename>.xml</filename> files into <filename>${datadir}/mime/packages</filename> but does not inherit the <filename>mime</filename> class</para></listitem>
+            </itemizedlist>
+        </para>
+    </section>
+        
+    <section id='migration-3.1-x86-live-wic'>
+        <title><filename>wic</filename> image type now used instead of <filename>live</filename> by default for x86</title>
+
+        <para>
+            <filename>conf/machine/include/x86-base.inc</filename> (inherited by most x86 machine configurations) now specifies <filename>wic</filename> instead of <filename>live</filename> by default in <link linkend='var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></link>. The <filename>live</filename> image type will likely be removed in a future release so it is recommended that you use <filename>wic</filename> instead.
+        </para>
+    </section>
+                
+    <section id='migration-3.1-misc'>
+        <title>Miscellaneous changes</title>
+
+        <para>
+            <itemizedlist>
+                <listitem><para>The undocumented <filename>SRC_DISTRIBUTE_LICENSES</filename> variable has now been removed in favour of a new <filename>AVAILABLE_LICENSES</filename> variable which is dynamically set based upon license files found in <filename>${COMMON_LICENSE_DIR}</filename> and <filename>${LICENSE_PATH}</filename>.</para></listitem>
+                <listitem><para>The tune definition for big-endian microblaze machines is now <filename>microblaze</filename> instead of <filename>microblazeeb</filename>.</para></listitem>
+                <listitem><para><filename>newlib</filename> no longer has built-in syscalls. <filename>libgloss</filename> should then provide the syscalls, <filename>crt0.o</filename> and other functions that are no longer part of <filename>newlib</filename> itself. If you are using <filename>TCLIBC = "newlib"</filename> this now means that you must link applications with both <filename>newlib</filename> and <filename>libgloss</filename>, whereas before <filename>newlib</filename> would run in many configurations by itself.</para></listitem>
+            </itemizedlist>
+        </para>
+    </section>
+
+</section>
+
+
 </chapter>
 <!--
 vim: expandtab tw=80 ts=4
-- 
2.20.1

Re: [docs] [PATCH 2/2] ref-manual: add migration section for 3.1

[akuster]

[-- Attachment #1: Type: text/plain, Size: 17383 bytes --]

Paul,

On 4/15/20 2:49 PM, Paul Eggleton wrote:
> From: Paul Eggleton <paul.eggleton@microsoft.com>
>
> Migration information for the 3.1 release.

Thanks for pulling this together. I have in-lined a few questions below.
>
> Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
> ---
>  documentation/ref-manual/migration.xml | 201 +++++++++++++++++++++++++
>  1 file changed, 201 insertions(+)
>
> diff --git a/documentation/ref-manual/migration.xml b/documentation/ref-manual/migration.xml
> index 03efc7be2..9422b5a27 100644
> --- a/documentation/ref-manual/migration.xml
> +++ b/documentation/ref-manual/migration.xml
> @@ -7095,6 +7095,207 @@ id=f4d4f99cfbc2396e49c1613a7d237b9e57f06f81'>commit message</ulink>.
>      </section>
>  </section>
>
> +
> +<section id='moving-to-the-yocto-project-3.1-release'>
> +    <title>Moving to the Yocto Project 3.1 Release</title>
> +
> +    <para>
> +        This section provides migration information for moving to the
> +        Yocto Project 3.1 Release from the prior release.
> +    </para>
> +
> +    <section id='migration-3.1-minimum-system-requirements'>
> +        <title>Minimum system requirements</title>
> +
> +        <para>
> +            The following versions / requirements of build host components have been updated:
> +            <itemizedlist>
> +                <listitem><para>gcc 5.0</para></listitem>
> +                <listitem><para>python 3.5</para></listitem>
> +                <listitem><para>tar 1.28</para></listitem>
> +                <listitem><para><filename>rpcgen</filename> is now required on the host (part of the <filename>libc-dev-bin</filename> package on Ubuntu, Debian and related distributions, and the <filename>glibc</filename> package on RPM-based distributions).</para></listitem>
> +            </itemizedlist>
> +
> +            Additionally, the <filename>makeinfo</filename> and <filename>pod2man</filename>
> +            tools are <emphasis>no longer</emphasis> required on the host.
> +        </para>
> +    </section>
> +
> +    <section id='migration-3.1-mpc8315e-rdb-removed'>
> +        <title>mpc8315e-rdb machine removed</title>
> +
> +        <para>
> +            The MPC8315E-RDB machine is old/obsolete and unobtainable, thus given the maintenance burden
> +            the <filename>mpc8315e-rdb</filename> machine configuration that supported it has been removed
> +            in this release. The removal does leave a gap in official PowerPC reference hardware
> +            support; this may change in future if a suitable machine with accompanying support resources
> +            is found.
> +        </para>
> +    </section>
> +
> +    <section id='migration-3.1-python-2-removed'>
> +        <title>Python 2 removed</title>
> +
> +        <para>
> +            Due to the expiration of upstream support in January 2020, support for Python 2 has now been removed; it is recommended that you use Python 3 instead. If absolutely needed there is a meta-python2 community layer containing Python 2, related classes and various Python 2-based modules, however it should not be considered as supported.
> +        </para>
> +    </section>
> +
> +    <section id='migration-3.1-reproducible-builds'>
> +        <title>Reproducible builds now enabled by default</title>
> +
> +        <para>
> +            In order to avoid unnecessary differences in output files (aiding binary reproducibility), the Poky distribution configuration (<filename><link linkend='var-DISTRO'>DISTRO</link> = "poky"</filename>) now inherits the <filename>reproducible_build</filename> class by default.
> +        </para>
> +    </section>

AK] I think the l hash equivalence server is also enabled by default:

commit 1fb4aa42f9c3b9738d78d198ad695fe4fcb6dc2f
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Wed Aug 7 21:56:01 2019 +0100

    poky: Default to enabling a local hash equivalence server
   

> +
> +    <section id='migration-3.1-ptest-feature-impact'>
> +        <title>Impact of ptest feature is now more significant</title>
> +
> +        <para>
> +            The Poky distribution configuration (<filename><link linkend='var-DISTRO'>DISTRO</link> = "poky"</filename>) enables ptests by default to enable runtime testing of various components. In this release, a dependency needed to be added that has resulted in a significant increase in the number of components that will be built just when building a simple image such as core-image-minimal. If you do not need runtime tests enabled for core components, then it is recommended that you remove "ptest" from <link linkend='var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></link> to save a significant amount of build time e.g. by adding the following in your configuration:
> +
> +            <literallayout class='monospaced'>
> +      DISTRO_FEATURES_remove = "ptest"
> +            </literallayout>
> +        </para>
> +    </section>
> +
> +    <section id='migration-3.1-removed-recipes'>
> +        <title>Removed recipes</title>
> +
> +        <para>
> +            The following recipes have been removed:
> +
> +            <itemizedlist>
> +                <listitem><para><filename>chkconfig</filename>: obsolete</para></listitem>
> +                <listitem><para><filename>console-tools</filename>: obsolete</para></listitem>
> +                <listitem><para><filename>enchant</filename>: replaced by <filename>enchant2</filename></para></listitem>
> +                <listitem><para><filename>foomatic-filters</filename>: obsolete</para></listitem>
> +                <listitem><para><filename>libidn</filename>: no longer needed, moved to meta-oe</para></listitem>
> +                <listitem><para><filename>libmodulemd</filename>: replaced by <filename>libmodulemd-v1</filename></para></listitem>
> +                <listitem><para><filename>linux-yocto</filename>: drop 4.19, 5.2 version recipes (5.4 now provided)</para></listitem>
> +                <listitem><para><filename>nspr</filename>: no longer needed, moved to meta-oe</para></listitem>
> +                <listitem><para><filename>nss</filename>: no longer needed, moved to meta-oe</para></listitem>
> +                <listitem><para><filename>python</filename>: Python 2 removed (Python 3 preferred)</para></listitem>
> +                <listitem><para><filename>python-setuptools</filename>: Python 2 version removed (python3-setuptools preferred)</para></listitem>
> +                <listitem><para><filename>sysprof</filename>: no longer needed, moved to meta-oe</para></listitem>
> +                <listitem><para><filename>texi2html</filename>: obsolete</para></listitem>
> +                <listitem><para><filename>u-boot-fw-utils</filename>: functionally replaced by <filename>libubootenv</filename></para></listitem>
> +            </itemizedlist>
> +        </para>
> +    </section>
> +
> +    <section id='migration-3.1-features-check'>
> +        <title>features_check class replaces distro_features_check</title>
> +
> +        <para>
> +            The <filename>distro_features_check</filename> class has had its functionality expanded, now supporting <filename>ANY_OF_MACHINE_FEATURES</filename>, <filename>REQUIRED_MACHINE_FEATURES</filename>, <filename>CONFLICT_MACHINE_FEATURES</filename>, <filename>ANY_OF_COMBINED_FEATURES</filename>, <filename>REQUIRED_COMBINED_FEATURES</filename>, <filename>CONFLICT_COMBINED_FEATURES</filename>. As a result the class has now been renamed to <filename>features_check</filename>; the <filename>distro_features_check</filename> class still exists but generates a warning and redirects to the new class. In preparation for a future removal of the old class it is recommended that you update recipes currently inheriting <filename>distro_features_check</filename> to inherit <filename>features_check</filename> instead.
> +        </para>
> +    </section>
> +
> +    <section id='migration-3.1-removed-classes'>
> +        <title>Removed classes</title>
> +
> +        <para>
> +            The following classes have been removed:
> +
> +            <itemizedlist>
> +                <listitem><para><filename>distutils-base</filename>: moved to meta-python2</para></listitem>
> +                <listitem><para><filename>distutils</filename>: moved to meta-python2</para></listitem>
> +                <listitem><para><filename>libc-common</filename>: merged into the glibc recipe as nothing else used it.</para></listitem>
> +                <listitem><para><filename>python-dir</filename>: moved to meta-python2</para></listitem>
> +                <listitem><para><filename>pythonnative</filename>: moved to meta-python2</para></listitem>
> +                <listitem><para><filename>setuptools</filename>: moved to meta-python2</para></listitem>
> +                <listitem><para><filename>tinderclient</filename>: dropped as it was obsolete.</para></listitem>
> +            </itemizedlist>
> +        </para>
> +    </section>
> +
> +    <section id='migration-3.1-src-uri-checksums'>
> +        <title>SRC_URI checksum behaviour</title>
> +
> +        <para>
> +            Previously, recipes by tradition included both SHA256 and MD5 checksums for remotely fetched files in <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>, even though only one is actually mandated. However, the MD5 checksum does not add much given its inherent weakness; thus when a checksum fails only the SHA256 sum will now be printed. The md5sum will still be verified if it is specified.

AK] If I am not mistaken, we can support all these in a recipe:  "md5",
"sha256", "sha1", "sha384", "sha512"

ref: bitbake/lib/bb/fetch2/__init__.py ; CHECKSUM_LIST

Not sure if we want to mention that too??? not saying it has to include
this.

> +        </para>
> +    </section>
> +
> +
> +    <section id='migration-3.1-npm'>
> +        <title>npm fetcher changes</title>
> +
> +        <para>
> +            The npm fetcher has been completely reworked in this release. The npm fetcher now only fetches the package source itself and no longer the dependencies; there is now also an npmsw fetcher which explicitly fetches the shrinkwrap file and the dependencies. This removes the slightly awkward <filename>NPM_LOCKDOWN</filename> and <filename>NPM_SHRINKWRAP</filename> variables which pointed to local files; the lockdown file is no longer needed at all. Additionally, the package name in <filename>npm://</filename> entries in <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link> is now specified using a <filename>package</filename> parameter instead of the earlier <filename>name</filename> which overlapped with the generic <filename>name</filename> parameter. All recipes using the npm fetcher will need to be changed as a result. 
> +        </para>
> +        <para>
> +            An example of the new scheme:
> +            <literallayout class='monospaced'>
> +SRC_URI = "npm://registry.npmjs.org;package=array-flatten;version=1.1.1 \
> +           npmsw://${THISDIR}/npm-shrinkwrap.json"
> +            </literallayout>
> +            Another example where the sources are fetched from git rather than an npm repository:
> +            <literallayout class='monospaced'>
> +SRC_URI = "git://github.com/foo/bar.git;protocol=https \
> +           npmsw://${THISDIR}/npm-shrinkwrap.json"
> +            </literallayout>
> +        </para>
> +        <para>
> +            devtool and recipetool have also been updated to match with the npm fetcher changes. Other than producing working and more complete recipes for npm sources, there is also a minor change to the command line for devtool: the <filename>--fetch-dev</filename> option has been renamed to <filename>--npm-dev</filename> as it is npm-specific.
> +        </para>
> +    </section>
> +
> +
> +    <section id='migration-3.1-packaging-changes'>
> +        <title>Packaging changes</title>
> +
> +        <para>
> +            <itemizedlist>
> +                <listitem><para><filename>intltool</filename> has been removed from <filename>packagegroup-core-sdk</filename> as it is rarely needed to build modern software - gettext can do most of the things it used to be needed for. <filename>intltool</filename> has also been removed from <filename>packagegroup-core-self-hosted</filename> as it is not needed to for standard builds.</para></listitem>
> +                <listitem><para>git: <filename>git-am</filename>, <filename>git-difftool</filename>, <filename>git-submodule</filename>, and <filename>git-request-pull</filename> are no longer perl-based, so are now installed with the main <filename>git</filename> package instead of within <filename>git-perltools</filename>.</para></listitem>
> +                <listitem><para>The <filename>ldconfig</filename> binary built as part of glibc has now been moved to its own <filename>ldconfig</filename> package (note no <filename>glibc-</filename> prefix). This package is in the <link linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link> of the main <filename>glibc</filename> package if <filename>ldconfig</filename> is present in <link linkend='var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></link>.</para></listitem>
> +                <listitem><para><filename>libevent</filename> now splits each shared library into its own package (as Debian does). Since these are shared libraries and will be pulled in through the normal shared library dependency handling, there should be no impact to existing configurations other than less unnecessary libraries being installed in some cases.</para></listitem>
> +                <listitem><para>linux-firmware now has a new package for <filename>bcm4366c</filename> and includes available NVRAM config files into the <filename>bcm43340</filename>, <filename>bcm43362</filename>, <filename>bcm43430</filename> and <filename>bcm4356-pcie</filename> packages.</para></listitem>
> +                <listitem><para><filename>harfbuzz</filename> now splits the new <filename>libharfbuzz-subset.so</filename> library into its own package to reduce the main package size in cases where <filename>libharfbuzz-subset.so</filename> is not needed.</para></listitem>
> +            </itemizedlist>
> +        </para>
> +    </section>
> +
> +    <section id='migration-3.1-package-qa-warnings'>
> +        <title>Additional warnings</title>
> +
> +        <para>
> +            Warnings will now be shown at <filename>do_package_qa</filename> time in the following circumstances:
> +
> +            <itemizedlist>
> +                <listitem><para>A recipe installs <filename>.desktop</filename> files containing <filename>MimeType</filename> keys but does not inherit the new <filename>mime-xdg</filename> class</para></listitem>
> +                <listitem><para>A recipe installs <filename>.xml</filename> files into <filename>${datadir}/mime/packages</filename> but does not inherit the <filename>mime</filename> class</para></listitem>
> +            </itemizedlist>
> +        </para>
> +    </section>
> +
> +    <section id='migration-3.1-x86-live-wic'>
> +        <title><filename>wic</filename> image type now used instead of <filename>live</filename> by default for x86</title>
> +
> +        <para>
> +            <filename>conf/machine/include/x86-base.inc</filename> (inherited by most x86 machine configurations) now specifies <filename>wic</filename> instead of <filename>live</filename> by default in <link linkend='var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></link>. The <filename>live</filename> image type will likely be removed in a future release so it is recommended that you use <filename>wic</filename> instead.
> +        </para>
> +    </section>
> +
> +    <section id='migration-3.1-misc'>
> +        <title>Miscellaneous changes</title>
> +
> +        <para>
> +            <itemizedlist>
> +                <listitem><para>The undocumented <filename>SRC_DISTRIBUTE_LICENSES</filename> variable has now been removed in favour of a new <filename>AVAILABLE_LICENSES</filename> variable which is dynamically set based upon license files found in <filename>${COMMON_LICENSE_DIR}</filename> and <filename>${LICENSE_PATH}</filename>.</para></listitem>
> +                <listitem><para>The tune definition for big-endian microblaze machines is now <filename>microblaze</filename> instead of <filename>microblazeeb</filename>.</para></listitem>
> +                <listitem><para><filename>newlib</filename> no longer has built-in syscalls. <filename>libgloss</filename> should then provide the syscalls, <filename>crt0.o</filename> and other functions that are no longer part of <filename>newlib</filename> itself. If you are using <filename>TCLIBC = "newlib"</filename> this now means that you must link applications with both <filename>newlib</filename> and <filename>libgloss</filename>, whereas before <filename>newlib</filename> would run in many configurations by itself.</para></listitem>
> +            </itemizedlist>
> +        </para>
> +    </section>
> +
> +</section>
> +
> +
>  </chapter>
>  <!--
>  vim: expandtab tw=80 ts=4
>
> 


[-- Attachment #2: Type: text/html, Size: 21560 bytes --]

Re: [docs] [PATCH 2/2] ref-manual: add migration section for 3.1

[Paul Eggleton]

Hi Armin

On Thursday, 16 April 2020 10:39:26 AM NZST akuster808 wrote:
> On 4/15/20 2:49 PM, Paul Eggleton wrote:
> > +        <para>
> > +            In order to avoid unnecessary differences in output files
> > (aiding binary reproducibility), the Poky distribution configuration
> > (<filename><link linkend='var-DISTRO'>DISTRO</link> = "poky"</filename>)
> > now inherits the <filename>reproducible_build</filename> class by
> > default. +        </para>
> > +    </section>
> 
> AK] I think the l hash equivalence server is also enabled by default:
> 
> commit 1fb4aa42f9c3b9738d78d198ad695fe4fcb6dc2f
> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> Date:   Wed Aug 7 21:56:01 2019 +0100
> 
>     poky: Default to enabling a local hash equivalence server

That is true. The purpose of this document though is more to highlight changes 
that the user will need to take some sort of action in response to when they 
upgrade. You might perhaps very reasonably argue that the reproducible builds 
change  doesn't qualify either :)

> > +        <para>
> > +            Previously, recipes by tradition included both SHA256 and MD5
> > checksums for remotely fetched files in <link
> > linkend='var-SRC_URI'><filename>SRC_URI</filename></link>, even though
> > only one is actually mandated. However, the MD5 checksum does not add
> > much given its inherent weakness; thus when a checksum fails only the
> > SHA256 sum will now be printed. The md5sum will still be verified if it
> > is specified.
>
> AK] If I am not mistaken, we can support all these in a recipe:  "md5",
> "sha256", "sha1", "sha384", "sha512"
> 
> ref: bitbake/lib/bb/fetch2/__init__.py ; CHECKSUM_LIST
> 
> Not sure if we want to mention that too???

We certainly could, but it's along similar lines as above. It wouldn't hurt to 
add it on to what we've got though I guess.

Cheers,
Paul