[oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>

Add a recipe for repo, prerequisite for the repo fetcher.

Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
---
 .../repo/files/0001-python3-shebang.patch     | 21 ++++++++++++
 .../0001-Set-REPO_REV-to-v2.17.3.patch        | 33 +++++++++++++++++++
 meta/recipes-devtools/repo/repo.inc           | 25 ++++++++++++++
 meta/recipes-devtools/repo/repo_2.17.3.bb     |  7 ++++
 4 files changed, 86 insertions(+)
 create mode 100644 meta/recipes-devtools/repo/files/0001-python3-shebang.patch
 create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
 create mode 100644 meta/recipes-devtools/repo/repo.inc
 create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb

diff --git a/meta/recipes-devtools/repo/files/0001-python3-shebang.patch b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
new file mode 100644
index 0000000000..09ccf58264
--- /dev/null
+++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
@@ -0,0 +1,21 @@
+From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
+Date: Tue, 14 Sep 2021 16:46:51 +0200
+Subject: [PATCH] python3 shebang
+
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index b13e34c..205e0e5 100755
+--- a/repo
++++ b/repo
+@@ -1,4 +1,4 @@
+-#!/usr/bin/env python
++#!/usr/bin/env python3
+ # -*- coding:utf-8 -*-
+ #
+ # Copyright (C) 2008 The Android Open Source Project
+--
+2.33.0
diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
new file mode 100644
index 0000000000..294a3af53a
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
@@ -0,0 +1,33 @@
+From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
+From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
+Date: Thu, 4 Nov 2021 16:55:12 +0100
+Subject: [PATCH] Set REPO_REV to v2.17.3
+
+repo is an unusual tool because it downloads all of its own Python modules
+using GPG-signed git tags, and stores those files as part of the project
+that it is working with.
+
+So in order to have a reproducible repo installation within the project
+folders, we hardcode the REPO_REV variable to this recipes PV.
+
+Upstream-Status: Inappropriate [configuration]
+Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
+---
+ repo | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/repo b/repo
+index 4cddbf1..cf5f6b1 100755
+--- a/repo
++++ b/repo
+@@ -142,7 +142,7 @@ if __name__ == '__main__':
+ REPO_URL = os.environ.get('REPO_URL', None)
+ if not REPO_URL:
+   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
+-REPO_REV = os.environ.get('REPO_REV')
++REPO_REV = 'v2.17.3'
+ if not REPO_REV:
+   REPO_REV = 'stable'
+ # URL to file bug reports for repo tool issues.
+--
+2.33.1
diff --git a/meta/recipes-devtools/repo/repo.inc b/meta/recipes-devtools/repo/repo.inc
new file mode 100644
index 0000000000..60b32e4d74
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo.inc
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
+
+SUMMARY = "Tool for managing many Git repositories"
+DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git repositories, does the uploads to revision control systems, and automates parts of the development workflow."
+HOMEPAGE = "https://android.googlesource.com/tools/repo"
+SECTION = "console/utils"
+
+LICENSE = "Apache-2.0"
+
+SRC_URI = "git://git@gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
+MIRRORS = "git://git@gerrit.googlesource.com/git-repo.git git://github.com/GerritCodeReview/git-repo.git \n"
+
+SRC_URI += "file://0001-python3-shebang.patch"
+
+S = "${WORKDIR}/git"
+
+RDEPENDS_${PN} = "python3"
+
+do_install() {
+    install -d ${D}${bindir}
+    install -m 755 ${WORKDIR}/git/repo ${D}${bindir}
+}
+
+BBCLASSEXTEND = "native nativesdk"
diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb b/meta/recipes-devtools/repo/repo_2.17.3.bb
new file mode 100644
index 0000000000..c26264b9e9
--- /dev/null
+++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: MIT
+# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
+
+require recipes-devtools/repo/repo.inc
+
+SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
-- 
2.33.1

[oe-core][PATCH 2/2] base.bbclass: Add sysroot deps for repo fetcher

[Jasper Orschulko]

From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>

Add git-native and repo-native as prerequisite for the repo fetcher.

Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
---
 meta/classes/base.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index a65fcc6c1d..3298bd1952 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -665,6 +665,11 @@ python () {
         elif uri.scheme == "npm":
             d.appendVarFlag('do_fetch', 'depends', ' nodejs-native:do_populate_sysroot')
 
+        elif uri.scheme == "repo":
+            needsrcrev = True
+            d.appendVarFlag('do_fetch', 'depends', ' repo-native:do_populate_sysroot')
+            d.appendVarFlag('do_fetch', 'depends', ' git-native:do_populate_sysroot')
+
         # *.lz4 should DEPEND on lz4-native for unpacking
         if path.endswith('.lz4'):
             d.appendVarFlag('do_unpack', 'depends', ' lz4-native:do_populate_sysroot')
-- 
2.33.1

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Konrad Weihmann]

Is this implementation of repo being patched against phoning home to 
google everytime and trying to do that pesky auto-update of the tool?

Otherwise this will become an issue when at one point in time an (then) 
outdated version of the tool is used in a BB_NO_NETWORK build.

On 05.11.21 14:31, Jasper Orschulko via lists.openembedded.org wrote:
> From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> 
> Add a recipe for repo, prerequisite for the repo fetcher.
> 
> Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> ---
>   .../repo/files/0001-python3-shebang.patch     | 21 ++++++++++++
>   .../0001-Set-REPO_REV-to-v2.17.3.patch        | 33 +++++++++++++++++++
>   meta/recipes-devtools/repo/repo.inc           | 25 ++++++++++++++
>   meta/recipes-devtools/repo/repo_2.17.3.bb     |  7 ++++
>   4 files changed, 86 insertions(+)
>   create mode 100644 meta/recipes-devtools/repo/files/0001-python3-shebang.patch
>   create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
>   create mode 100644 meta/recipes-devtools/repo/repo.inc
>   create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
> 
> diff --git a/meta/recipes-devtools/repo/files/0001-python3-shebang.patch b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
> new file mode 100644
> index 0000000000..09ccf58264
> --- /dev/null
> +++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
> @@ -0,0 +1,21 @@
> +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001
> +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> +Date: Tue, 14 Sep 2021 16:46:51 +0200
> +Subject: [PATCH] python3 shebang
> +
> +---
> + repo | 2 +-
> + 1 file changed, 1 insertion(+), 1 deletion(-)
> +
> +diff --git a/repo b/repo
> +index b13e34c..205e0e5 100755
> +--- a/repo
> ++++ b/repo
> +@@ -1,4 +1,4 @@
> +-#!/usr/bin/env python
> ++#!/usr/bin/env python3
> + # -*- coding:utf-8 -*-
> + #
> + # Copyright (C) 2008 The Android Open Source Project
> +--
> +2.33.0
> diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> new file mode 100644
> index 0000000000..294a3af53a
> --- /dev/null
> +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> @@ -0,0 +1,33 @@
> +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
> +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> +Date: Thu, 4 Nov 2021 16:55:12 +0100
> +Subject: [PATCH] Set REPO_REV to v2.17.3
> +
> +repo is an unusual tool because it downloads all of its own Python modules
> +using GPG-signed git tags, and stores those files as part of the project
> +that it is working with.
> +
> +So in order to have a reproducible repo installation within the project
> +folders, we hardcode the REPO_REV variable to this recipes PV.
> +
> +Upstream-Status: Inappropriate [configuration]
> +Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> +---
> + repo | 2 +-
> + 1 file changed, 1 insertion(+), 1 deletion(-)
> +
> +diff --git a/repo b/repo
> +index 4cddbf1..cf5f6b1 100755
> +--- a/repo
> ++++ b/repo
> +@@ -142,7 +142,7 @@ if __name__ == '__main__':
> + REPO_URL = os.environ.get('REPO_URL', None)
> + if not REPO_URL:
> +   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> +-REPO_REV = os.environ.get('REPO_REV')
> ++REPO_REV = 'v2.17.3'
> + if not REPO_REV:
> +   REPO_REV = 'stable'
> + # URL to file bug reports for repo tool issues.
> +--
> +2.33.1
> diff --git a/meta/recipes-devtools/repo/repo.inc b/meta/recipes-devtools/repo/repo.inc
> new file mode 100644
> index 0000000000..60b32e4d74
> --- /dev/null
> +++ b/meta/recipes-devtools/repo/repo.inc
> @@ -0,0 +1,25 @@
> +# SPDX-License-Identifier: MIT
> +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> +
> +SUMMARY = "Tool for managing many Git repositories"
> +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git repositories, does the uploads to revision control systems, and automates parts of the development workflow."
> +HOMEPAGE = "https://android.googlesource.com/tools/repo"
> +SECTION = "console/utils"
> +
> +LICENSE = "Apache-2.0"
> +
> +SRC_URI = "git://git@gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
> +MIRRORS = "git://git@gerrit.googlesource.com/git-repo.git git://github.com/GerritCodeReview/git-repo.git \n"
> +
> +SRC_URI += "file://0001-python3-shebang.patch"
> +
> +S = "${WORKDIR}/git"
> +
> +RDEPENDS_${PN} = "python3"
> +
> +do_install() {
> +    install -d ${D}${bindir}
> +    install -m 755 ${WORKDIR}/git/repo ${D}${bindir}
> +}
> +
> +BBCLASSEXTEND = "native nativesdk"
> diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb b/meta/recipes-devtools/repo/repo_2.17.3.bb
> new file mode 100644
> index 0000000000..c26264b9e9
> --- /dev/null
> +++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
> @@ -0,0 +1,7 @@
> +# SPDX-License-Identifier: MIT
> +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> +
> +require recipes-devtools/repo/repo.inc
> +
> +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
> +LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#12893): https://lists.openembedded.org/g/bitbake-devel/message/12893
> Mute This Topic: https://lists.openembedded.org/mt/86840271/3647476
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [kweihmann@outlook.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Konrad,

that might be a good point. I patched the REPO_REV to a fixed refspec
within the wrapper tool, as to stop repo from using the latest stable
release. But you are right, and this might not be enough. Will do some
more digging on this.

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Fri, 2021-11-05 at 14:35 +0100, Konrad Weihmann wrote:
> Is this implementation of repo being patched against phoning home to 
> google everytime and trying to do that pesky auto-update of the tool?
> 
> Otherwise this will become an issue when at one point in time an
> (then) 
> outdated version of the tool is used in a BB_NO_NETWORK build.
> 
> On 05.11.21 14:31, Jasper Orschulko via lists.openembedded.org wrote:
> > From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > 
> > Add a recipe for repo, prerequisite for the repo fetcher.
> > 
> > Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > ---
> >   .../repo/files/0001-python3-shebang.patch     | 21 ++++++++++++
> >   .../0001-Set-REPO_REV-to-v2.17.3.patch        | 33
> > +++++++++++++++++++
> >   meta/recipes-devtools/repo/repo.inc           | 25 ++++++++++++++
> >   meta/recipes-devtools/repo/repo_2.17.3.bb     |  7 ++++
> >   4 files changed, 86 insertions(+)
> >   create mode 100644 meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch
> >   create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-
> > Set-REPO_REV-to-v2.17.3.patch
> >   create mode 100644 meta/recipes-devtools/repo/repo.inc
> >   create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
> > 
> > diff --git a/meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch b/meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch
> > new file mode 100644
> > index 0000000000..09ccf58264
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
> > @@ -0,0 +1,21 @@
> > +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > +Date: Tue, 14 Sep 2021 16:46:51 +0200
> > +Subject: [PATCH] python3 shebang
> > +
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index b13e34c..205e0e5 100755
> > +--- a/repo
> > ++++ b/repo
> > +@@ -1,4 +1,4 @@
> > +-#!/usr/bin/env python
> > ++#!/usr/bin/env python3
> > + # -*- coding:utf-8 -*-
> > + #
> > + # Copyright (C) 2008 The Android Open Source Project
> > +--
> > +2.33.0
> > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
> > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> > new file mode 100644
> > index 0000000000..294a3af53a
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-
> > v2.17.3.patch
> > @@ -0,0 +1,33 @@
> > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > +Date: Thu, 4 Nov 2021 16:55:12 +0100
> > +Subject: [PATCH] Set REPO_REV to v2.17.3
> > +
> > +repo is an unusual tool because it downloads all of its own Python
> > modules
> > +using GPG-signed git tags, and stores those files as part of the
> > project
> > +that it is working with.
> > +
> > +So in order to have a reproducible repo installation within the
> > project
> > +folders, we hardcode the REPO_REV variable to this recipes PV.
> > +
> > +Upstream-Status: Inappropriate [configuration]
> > +Signed-off-by: Jasper Orschulko
> > <Jasper.Orschulko@iris-sensing.com>
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index 4cddbf1..cf5f6b1 100755
> > +--- a/repo
> > ++++ b/repo
> > +@@ -142,7 +142,7 @@ if __name__ == '__main__':
> > + REPO_URL = os.environ.get('REPO_URL', None)
> > + if not REPO_URL:
> > +   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> > +-REPO_REV = os.environ.get('REPO_REV')
> > ++REPO_REV = 'v2.17.3'
> > + if not REPO_REV:
> > +   REPO_REV = 'stable'
> > + # URL to file bug reports for repo tool issues.
> > +--
> > +2.33.1
> > diff --git a/meta/recipes-devtools/repo/repo.inc b/meta/recipes-
> > devtools/repo/repo.inc
> > new file mode 100644
> > index 0000000000..60b32e4d74
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo.inc
> > @@ -0,0 +1,25 @@
> > +# SPDX-License-Identifier: MIT
> > +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> > +
> > +SUMMARY = "Tool for managing many Git repositories"
> > +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps
> > manage many Git repositories, does the uploads to revision control
> > systems, and automates parts of the development workflow."
> > +HOMEPAGE = "https://android.googlesource.com/tools/repo"
> > +SECTION = "console/utils"
> > +
> > +LICENSE = "Apache-2.0"
> > +
> > +SRC_URI = "git://git@gerrit.googlesource.com/git-
> > repo.git;protocol=https;branch=main"
> > +MIRRORS = "git://git@gerrit.googlesource.com/git-repo.git
> > git://github.com/GerritCodeReview/git-repo.git \n"
> > +
> > +SRC_URI += "file://0001-python3-shebang.patch"
> > +
> > +S = "${WORKDIR}/git"
> > +
> > +RDEPENDS_${PN} = "python3"
> > +
> > +do_install() {
> > +    install -d ${D}${bindir}
> > +    install -m 755 ${WORKDIR}/git/repo ${D}${bindir}
> > +}
> > +
> > +BBCLASSEXTEND = "native nativesdk"
> > diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
> > b/meta/recipes-devtools/repo/repo_2.17.3.bb
> > new file mode 100644
> > index 0000000000..c26264b9e9
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
> > @@ -0,0 +1,7 @@
> > +# SPDX-License-Identifier: MIT
> > +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> > +
> > +require recipes-devtools/repo/repo.inc
> > +
> > +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
> > +LIC_FILES_CHKSUM =
> > "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
> > 
> > 
> > 
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#12893):
> > https://lists.openembedded.org/g/bitbake-devel/message/12893
> > Mute This Topic: https://lists.openembedded.org/mt/86840271/3647476
> > Group Owner: bitbake-devel+owner@lists.openembedded.org
> > Unsubscribe:
> > https://lists.openembedded.org/g/bitbake-devel/unsub [kweihmann@outlook.com
> > ]
> > -=-=-=-=-=-=-=-=-=-=-=-
> > 
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFNegACgkQYgqew07V
MNVw6Af/WN3h3P19RWdnE7gncdbOgMbS/FBX2viw8HucAfjXovlTQyUzCyN5bROG
CyeSwOp5Fvblyg52CzPqkVuXvAoj01mqtHWhLdkzqqB+MUDR1p3Y0qPFJo7BBspP
vN8pY4S5dEW3np6zg+9nSkH5nrrZ6AjU+wZipMEGeaGLuEOGvCh5pwDK8d9VzAAu
mXq5i8f+ZTr75LNnOpmWz0LgkN32PUCD0VS42/aQFloxvTkssdVKupyCwjUHttxN
Fn2J+BNIXbMkJLFiMY6dYaqJKai25l8YH6l8bMVWQLPpJg1d2DpPf1FuGGG7aidh
/KNnb8B/tYMF9MNrKXduBayNIj01Vg==
=THWb
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Actually, I don't believe this to be an issue for the following
reasons:

1. The recipe does nothing more than taking the wrapper script as is
and copying it to the sysroot. As the repo binary is never called when
building, it does not have the "chance" to look for an update during
the yocto offline build process. The outcome of an offline build is
therefore predictable.

2. When using repo within the yocto build itself (with the repo
fetcher), the repo binary is only ever called during the do_fetch step,
which obviously has no issues with any network access.

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Fri, 2021-11-05 at 14:47 +0100, Jasper Orschulko wrote:
> Hi Konrad,
> 
> that might be a good point. I patched the REPO_REV to a fixed refspec
> within the wrapper tool, as to stop repo from using the latest stable
> release. But you are right, and this might not be enough. Will do
> some
> more digging on this.
> 
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFOvsACgkQYgqew07V
MNVfxQf/VZLnjKv7To1VaqM6dMIX/R3P9cWQtuTxOg6G05Ziooizao5VgQDUXEIy
Zs3h5lCFMuXdJlEZc9q2AJQC55BIGqtW8Njo6lNQDPI0Iz51HpXhNgsWdoC3E7qB
tphfw+H0WNnIhtDMkK/+hUAZWAsHHUb7I3rAyruE34GKqvEmbBwHo45B0NpgeCqF
rNPH7FeI9VvehloZB/5+jnJbrnmyWWogXCNx3PXPlgyrnxyRDuibOUivW6R/MRGd
RkSdYvllMgS9ajBLyK/PEyeA6rupEVQquKv0n9ghxZDOzVm6mboCvBMcjxsaSCpR
UlFP+h+4sYw0DTfAN9J4UIGWq4VmJA==
=P/xG
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Konrad Weihmann]

On 05.11.21 15:09, Jasper Orschulko wrote:
> Actually, I don't believe this to be an issue for the following
> reasons:
> 
> 1. The recipe does nothing more than taking the wrapper script as is
> and copying it to the sysroot. As the repo binary is never called when
> building, it does not have the "chance" to look for an update during
> the yocto offline build process. The outcome of an offline build is
> therefore predictable.
> 
> 2. When using repo within the yocto build itself (with the repo
> fetcher), the repo binary is only ever called during the do_fetch step,
> which obviously has no issues with any network access.
> 
 >

Okay fine that makes sense, even if it is still phoning home.
Another thing, could you please exchange "+REPO_REV = 'v2.17.3'" for a 
SHA revision, as I wouldn't bet my life on google not deleting/removing 
tags from their repos - and unpleasantly we wouldn't even notice that easily

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

> Okay fine that makes sense, even if it is still phoning home.
True. Bad software design at best. But luckily not an issue from a
build system perspective. :)

> Another thing, could you please exchange "+REPO_REV = 'v2.17.3'" for
> a 
> SHA revision, as I wouldn't bet my life on google not
> deleting/removing 
> tags from their repos - and unpleasantly we wouldn't even notice that
> easily

Fair point, will do!

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Fri, 2021-11-05 at 15:20 +0100, Konrad Weihmann wrote:
> 
> 
> On 05.11.21 15:09, Jasper Orschulko wrote:
> > Actually, I don't believe this to be an issue for the following
> > reasons:
> > 
> > 1. The recipe does nothing more than taking the wrapper script as is
> > and copying it to the sysroot. As the repo binary is never called
> > when
> > building, it does not have the "chance" to look for an update during
> > the yocto offline build process. The outcome of an offline build is
> > therefore predictable.
> > 
> > 2. When using repo within the yocto build itself (with the repo
> > fetcher), the repo binary is only ever called during the do_fetch
> > step,
> > which obviously has no issues with any network access.
> > 
>  >
> 
> Okay fine that makes sense, even if it is still phoning home.
> Another thing, could you please exchange "+REPO_REV = 'v2.17.3'" for a 
> SHA revision, as I wouldn't bet my life on google not deleting/removing
> tags from their repos - and unpleasantly we wouldn't even notice that
> easily
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFRU4ACgkQYgqew07V
MNX3ywf+MQAwmH9TVF0izfVVIYhwiTASgZLlG0pwMBXfQw6iIeWAXmO8LccJNrwb
fceVMFFXPi1/2SjAdjhOzzh5X7uSGbcVT0G13RpeuYfsnu7HyJ3n6TAdzoXiQtNa
PqJqPVsZBEIpHilTYEZqPu/ENHNq74Sj/sQXqM0XjNMTaJKKW7Syc9jbGdEdn+MV
gU9iNIQmYWEPSUS2Tp+lgFh9xC3qQOQJZPpcJ5V3hWfAIibaRMed4kGdd3mlJsYj
/Aoq+7gXe1iZOXB7OxXX+bk7ueoHCnp7QUPyWFVxslKx/r/faH+f+PwHmJHZp0VD
gOrzbjaIK2E2CVnRobaOkSknIffJzQ==
=Tfz+
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

SHA rev set in v2 of the patch series :)

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Fri, 2021-11-05 at 15:20 +0100, Konrad Weihmann wrote:
> 
> 
> On 05.11.21 15:09, Jasper Orschulko wrote:
> > Actually, I don't believe this to be an issue for the following
> > reasons:
> > 
> > 1. The recipe does nothing more than taking the wrapper script as
> > is
> > and copying it to the sysroot. As the repo binary is never called
> > when
> > building, it does not have the "chance" to look for an update
> > during
> > the yocto offline build process. The outcome of an offline build is
> > therefore predictable.
> > 
> > 2. When using repo within the yocto build itself (with the repo
> > fetcher), the repo binary is only ever called during the do_fetch
> > step,
> > which obviously has no issues with any network access.
> > 
>  >
> 
> Okay fine that makes sense, even if it is still phoning home.
> Another thing, could you please exchange "+REPO_REV = 'v2.17.3'" for
> a 
> SHA revision, as I wouldn't bet my life on google not
> deleting/removing 
> tags from their repos - and unpleasantly we wouldn't even notice that
> easily
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFTxsACgkQYgqew07V
MNW2ugf/RS4zCitkPe9ZEDJvApdZGie1dJmc3gGvcweNv6d6kd1bFd2bS4BYdbiK
BOhHF3kMtdznaJNyQlAHBSmysfvTUWymqUahRT3f7+kEuHqrwQcYJFoDqM3tXR4s
4VKYpbRo8klw62XjuBHdwJ8LxJiz5EWQdQF8gdQTl3Tiojp6y2KGDrjo3j/MYjAk
6KfwP0bAca6o/vMF7Jjhbs9sIlCqOkUXrNRbktEonO7hPZxdpgiEC6QPHCq9ONiN
3hHPhV13gd+BTa/7k1GNPSLW4XV6ye7qc26CpqMDing8dxz4RJnzpk2IxTNFHQ0m
95mLdHAygo7o9RJYdz8NIO4hXVqb7Q==
=nQVK
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Richard Purdie]

On Fri, 2021-11-05 at 14:09 +0000, Jasper Orschulko wrote:
> Actually, I don't believe this to be an issue for the following
> reasons:
> 
> [...]
>
> 2. When using repo within the yocto build itself (with the repo
> fetcher), the repo binary is only ever called during the do_fetch step,
> which obviously has no issues with any network access.

This is not true. The system may be configured for no network access and is
meant just to be building from MIRRORS/PREMIRRORS (see also BB_NO_NETWORK).

One of the reasons for the fetcher abstraction is to allow us to support
features like that. do_fetch is where network access should happen, if the
system is configured for it.

I guess the question becomes whether the usages of repo when using DL_DIR as a
cache will still touch the network?

Cheers,

Richard

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Richard,

I think our implementation of the repo fetcher checks the (what I
believe to be) most relevant parts of your checklist (thanks for the
write-up). Martin, please corrent me if I'm missing something:

> a) network access for sources is only expected to happen in the
> do_fetch step.
> This is not enforced or tested but is required so that we can:
> 
>  i) audit the sources used (i.e. for license/manifest reasons)
>  ii) support offline builds with a suitable cache
>  iii) allow work to continue even with downtime upstream
>  iv) allow for changes upstream in incompatible ways
>  v) allow rebuilding of the software in X years time

check

> b) network access is not expected in do_unpack

check 

> c) you can take DL_DIR and use it as a mirror for offline builds

check

> d) access to the network is only made when explicitly configured in
> recipes
>  (e.g. use of AUTOREV, or use of git tags which change revision)

check

> e) fetcher output is deterministic
>  (i.e. if you fetch configuration XXX now it will match in future
> exactly in 
>  a clean build with a new DL_DIR)

check. When a fixed refspec is set within the recipe, the fetcher will
check, if all repositories in the repo manifest are set to a fixed
refspec as well and otherwise throw an error.

> f) network access is expected to work with the standard linux proxy
> variables
>  environment but only in the do_fetch tasks)

this should work, as we keep the GIT_PROXY_COMMAND environment and run
repo via runfetchcmd(). Not further tested though.

> g) access during parsing has to be minimal, a "git ls-remote" for an
> AUTOREV 
>  git recipe might be ok but you can't expect to checkout a git tree

unfortunately, do to the nature of repo, we need to clone the manifest
repo, so that we can run "git ls-remote" on the referenced git repos
and therefore ensure that 

> h) we need to provide revision information during parsing such that a
> version
>  for the recipe can be constructed.

check. We create a hash value from git ls-remote of the recipe file, as
well as the git ls-remote of each repo referenced in the manifest.

> i) versions are expected to be able to increase in a way which sorts
> allowing 
>  package feeds to operate (see PR server required for git revisions to
> sort)

check. We use the PR server and didn't notice any issues.

> j) API to query for possible version upgrades of a url is highly
> desireable to 
>  allow out automated upgrage code to function (it is implied this does
> always 
>  have network access)

check (if I understand you correctly). We support AUTOINC.

> k) Where fixes or changes to behaviour in the fetcher are made, we ask
> that 
>  test cases are added (run with "bitbake-selftest bb.tests.fetch"). We
> do 
>  have fairly extensive test coverage of the fetcher as it is the only
> way
>  to track all of it's corner cases, it still doesn't give entire
> coverage 
>  though sadly.

We are currently still missing any tests for the new fetcher. We will
add them in the course of the next days. Meanwhile, I have prepared a
demo environment so if anymeone is interested in doing some further
testing, feel free: https://github.com/Jasper-Ben/demo-kas :)

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/
> 
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGKWtQACgkQYgqew07V
MNU3Lgf+PlDNyHxoCive3HL8V00KlfrsjTcuQVYiODav39n/h/cRvq8F8Wh7NiyM
Hjrr9O7MCOZLNGQiW9vBN/R6FbWESZ+sxl2nLdGH98eCYPKBurcHzvIXlJgincc+
dL1HtxDqjdk9TOHfaKAD2zc8U0tY6xu6Hgj7QIAqELt8HAwOQeIherAuIQw7g+KH
eWY5xhn8+KrY//RlnnRmDMO20LBY68bdkCOCH3kAIBfcSIzj8fzf1A4PZhGFi8AM
dydVCBMrKvPg3bfJnenZrVPijWnb82pOwjWzoZltzYxtCjqAhwevoCr8twGNDbx4
IVPcEqtMWRrf8rpQiHsZFTheap8BdA==
=hdir
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Richard Purdie]

On Tue, 2021-11-09 at 11:26 +0000, Jasper Orschulko wrote:
> 
> 
> > e) fetcher output is deterministic
> >  (i.e. if you fetch configuration XXX now it will match in future
> > exactly in 
> >  a clean build with a new DL_DIR)
> 
> check. When a fixed refspec is set within the recipe, the fetcher will
> check, if all repositories in the repo manifest are set to a fixed
> refspec as well and otherwise throw an error.

When you say "fixed refspec", will that be a definitive sha revision or a tag?
We always force resolution of tags as they tend to cause problems and can change
even if it is bad form.

> > f) network access is expected to work with the standard linux proxy
> > variables
> >  environment but only in the do_fetch tasks)
> 
> this should work, as we keep the GIT_PROXY_COMMAND environment and run
> repo via runfetchcmd(). Not further tested though.

If you're using runfetchcmd that should work.

> 
> > g) access during parsing has to be minimal, a "git ls-remote" for an
> > AUTOREV 
> >  git recipe might be ok but you can't expect to checkout a git tree
> 
> unfortunately, do to the nature of repo, we need to clone the manifest
> repo, so that we can run "git ls-remote" on the referenced git repos
> and therefore ensure that 

This is potentially a big issue. Cloning operations during parsing is pretty
horrible. We'd not expect any thing being written out like that during a parse.
It would probably work "ok" for one recipe but if you start getting the hundreds
of git recipes we have in some layers, it wouldn't scale if we allowed that :(.

Not sure what to recommend here but it is definitely problematic.

> 
Cheers,

Richard

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Richard,

> When you say "fixed refspec", will that be a definitive sha revision
> or a tag?
> We always force resolution of tags as they tend to cause problems and
> can change
> even if it is bad form.

that's a good point. Actually, Martin and I have just been discussing
this, as we noticed that this point actually got "lost" during our
implementation. We are currently working on fixing this. Good to know
how you handle this. I will keep you posted.

> This is potentially a big issue. Cloning operations during parsing is
> pretty
> horrible. We'd not expect any thing being written out like that
> during a parse.
> It would probably work "ok" for one recipe but if you start getting
> the hundreds
> of git recipes we have in some layers, it wouldn't scale if we
> allowed that :(.
> 
> Not sure what to recommend here but it is definitely problematic.

Just to make sure that we are on the same page: This ONLY affects
recipes which use the repo fetcher. And it ONLY clones the repository
containing the repo manifest (which tend to be small in size). So
unless developers start using hundreds of repo-based recipes, which I
find a very unlikely scenario, this should not be an issue.

Unfortunately, I don't see any other way to access the repo manifest
file, as we need to calculate the commit hashes of the git repos
referenced in the repo manifest file. Otherwise, it is impossible for
us to determinate the necessity of an update when SRCREV =
"${AUTOREV}". 

However, I see one potential improvement here. Currently the cloning of
the manifest repo is done on a per-recipe basis. E.g. this means if we
have 10 recipes inheriting a bbclass containing a repo fetcher, we will
clone 10 identical manifest repos. We'll work on improving this.

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Wed, 2021-11-10 at 12:46 +0000, Richard Purdie wrote:
> On Tue, 2021-11-09 at 11:26 +0000, Jasper Orschulko wrote:
> > 
> > 
> > > e) fetcher output is deterministic
> > >  (i.e. if you fetch configuration XXX now it will match in future
> > > exactly in 
> > >  a clean build with a new DL_DIR)
> > 
> > check. When a fixed refspec is set within the recipe, the fetcher
> > will
> > check, if all repositories in the repo manifest are set to a fixed
> > refspec as well and otherwise throw an error.
> 
> When you say "fixed refspec", will that be a definitive sha revision
> or a tag?
> We always force resolution of tags as they tend to cause problems and
> can change
> even if it is bad form.
> 
> > > f) network access is expected to work with the standard linux
> > > proxy
> > > variables
> > >  environment but only in the do_fetch tasks)
> > 
> > this should work, as we keep the GIT_PROXY_COMMAND environment and
> > run
> > repo via runfetchcmd(). Not further tested though.
> 
> If you're using runfetchcmd that should work.
> 
> > 
> > > g) access during parsing has to be minimal, a "git ls-remote" for
> > > anin
> > > AUTOREV 
> > >  git recipe might be ok but you can't expect to checkout a git
> > > tree
> > 
> > unfortunately, do to the nature of repo, we need to clone the
> > manifest
> > repo, so that we can run "git ls-remote" on the referenced git
> > repos
> > and therefore ensure that 
> 
> This is potentially a big issue. Cloning operations during parsing is
> pretty
> horrible. We'd not expect any thing being written out like that
> during a parse.
> It would probably work "ok" for one recipe but if you start getting
> the hundreds
> of git recipes we have in some layers, it wouldn't scale if we
> allowed that :(.
> 
> Not sure what to recommend here but it is definitely problematic.
> 
> > 
> Cheers,
> 
> Richard
> 
> 
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGLzpwACgkQYgqew07V
MNV3XQf9FDzIq3yihsO5FCEn/QOm7v48VuuOZF/85K6dxRgexCfdHHxkn7zJ0luE
ZxgPDmcXM83HcFh6B5XKis88/vnkU2R+YITgWe9+81l1foQryKTP9u7E2giIHW/F
JpGzxTtTb5F3N0+xjmqnyR7OYEB3TqJ1VFsaLlYdYs/sWaDYbt/9AEtcD39ynCr5
dEEqEgiIk05X03kiNnyUd2jDpy0bAbihqJu7OPzU4zSvNn/+zXRM0CMKDemyONb1
u2lINROQpk98qaVzBTX+uOskQTkMrkRRuuncUY0ggq6pHGz3TRubv15mePYIeLlJ
y2lBBB6O8f/iPesjbvKFa85EeNhdAg==
=4Ewh
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Richard Purdie]

On Wed, 2021-11-10 at 13:52 +0000, Jasper Orschulko wrote:
> Hi Richard,
> 
> > When you say "fixed refspec", will that be a definitive sha revision
> > or a tag?
> > We always force resolution of tags as they tend to cause problems and
> > can change even if it is bad form.
> 
> that's a good point. Actually, Martin and I have just been discussing
> this, as we noticed that this point actually got "lost" during our
> implementation. We are currently working on fixing this. Good to know
> how you handle this. I will keep you posted.

Ok, it is good to be clear on that one. I know the fact we hit the network for
tags does concern some but it really is the only way to handle them.

> > This is potentially a big issue. Cloning operations during parsing is
> > pretty
> > horrible. We'd not expect any thing being written out like that
> > during a parse.
> > It would probably work "ok" for one recipe but if you start getting
> > the hundreds
> > of git recipes we have in some layers, it wouldn't scale if we
> > allowed that :(.
> > 
> > Not sure what to recommend here but it is definitely problematic.
> 
> Just to make sure that we are on the same page: This ONLY affects
> recipes which use the repo fetcher. And it ONLY clones the repository
> containing the repo manifest (which tend to be small in size).

Correct, we are on the same page. This is still quite problematic as the recipes
are meant to parse quickly and a repository clone is definitely not expected.

> So unless developers start using hundreds of repo-based recipes, which I
> find a very unlikely scenario, this should not be an issue.

Even ten recipes using this will show a degradation in parsing speed and I do
get a lot of complaints when parsing slows down for any reason. The user doesn't
expect this and it won't be visible what bitbake is doing (sitting at 99% parsed
for a period).

Also, the "tend to be small" implies someone will create a huge one at some
point even if that is a bad idea for whatever reasons, I just know how these
things end up going :(.

> Unfortunately, I don't see any other way to access the repo manifest
> file, as we need to calculate the commit hashes of the git repos
> referenced in the repo manifest file. Otherwise, it is impossible for
> us to determinate the necessity of an update when SRCREV =
> "${AUTOREV}". 

Some further questions:

* Does it only clone a repo in the AUTOREV case?
* Could it only obtain the manifest file somehow without a clone of the repo?

> However, I see one potential improvement here. Currently the cloning of
> the manifest repo is done on a per-recipe basis. E.g. this means if we
> have 10 recipes inheriting a bbclass containing a repo fetcher, we will
> clone 10 identical manifest repos. We'll work on improving this.

At least for wget or git, it is assumed that for a given url, there would be one
tarball/clone and that there is locking in place to share it between them. This
means you'll see do_fetch tasks for binutils, binutils-cross-XXX, nativesdk-
binutils and binutils-native and one will block the others but the fetch will
happen once and be shared between them. I guess with repo it may not be as
simple as that but we should try and share what we can if possible.

Cheers,

Richard

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Richard,

> Even ten recipes using this will show a degradation in parsing speed
> and I do
> get a lot of complaints when parsing slows down for any reason. The
> user doesn't
> expect this and it won't be visible what bitbake is doing (sitting at
> 99% parsed
> for a period).
> 
> Also, the "tend to be small" implies someone will create a huge one
> at some
> point even if that is a bad idea for whatever reasons, I just know
> how these
> things end up going :(.

I believe this is something that could be addressed in the
documentation of the repo fetcher? Something along the lines of "Using
the repo fetcher can increase the time spent in the parsing phase, as
we need to clone and inspect the manifest repository. To avoid this,
keep the size of your manifest repository at a minimum."

When this is done, the dalay is barely noticable. We have a testsetup
with 24 recipes and reasonable parsing speeds.

> * Does it only clone a repo in the AUTOREV case?

No, it also clones with fixed refspecs, as we want to ensure that each
of the git repos referenced in the manifest also uses fixed refspecs.
Otherwise, the sources in the background might change, while the recipe
still (correctly) references the same revision of the manifest repo.

> * Could it only obtain the manifest file somehow without a clone of
> the repo?

Unfortunately not really. git does not offer a way to list the content
of a file without cloning it beforehand.

The only other way I could imagine this to work would require us to
keep the manifest file within the meta layer and do a somewhat "hacky"
workaround:

It is possible to run `repo init` on a local git folder.
As such, something like this could work:

1) create temp folder (e.g. /tmp/repo-XXXXXXXXX)
2) run git init in temp folder
3) install manifest file into temp folder
4) add & commit
5) run repo init -u file://$temp-folder

The repo manifest could even still be used outside of yocto, by using
the metalayer repo as basis for the repo init: `repo init -u
$metalayer-repo -m $path-to-manifest-file`. Not pretty, but it works.

So if cloning within the parser is a dealbreaker for you, this option
might be worth looking into.

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Wed, 2021-11-10 at 16:33 +0000, Richard Purdie wrote:
> On Wed, 2021-11-10 at 13:52 +0000, Jasper Orschulko wrote:
> > Hi Richard,
> > 
> > > When you say "fixed refspec", will that be a definitive sha
> > > revision
> > > or a tag?
> > > We always force resolution of tags as they tend to cause problems
> > > and
> > > can change even if it is bad form.
> > 
> > that's a good point. Actually, Martin and I have just been
> > discussing
> > this, as we noticed that this point actually got "lost" during our
> > implementation. We are currently working on fixing this. Good to
> > know
> > how you handle this. I will keep you posted.
> 
> Ok, it is good to be clear on that one. I know the fact we hit the
> network for
> tags does concern some but it really is the only way to handle them.
> 
> > > This is potentially a big issue. Cloning operations during
> > > parsing is
> > > pretty
> > > horrible. We'd not expect any thing being written out like that
> > > during a parse.
> > > It would probably work "ok" for one recipe but if you start
> > > getting
> > > the hundreds
> > > of git recipes we have in some layers, it wouldn't scale if we
> > > allowed that :(.
> > > 
> > > Not sure what to recommend here but it is definitely problematic.
> > 
> > Just to make sure that we are on the same page: This ONLY affects
> > recipes which use the repo fetcher. And it ONLY clones the
> > repository
> > containing the repo manifest (which tend to be small in size).
> 
> Correct, we are on the same page. This is still quite problematic as
> the recipes
> are meant to parse quickly and a repository clone is definitely not
> expected.
> 
> > So unless developers start using hundreds of repo-based recipes,
> > which I
> > find a very unlikely scenario, this should not be an issue.
> 
> Even ten recipes using this will show a degradation in parsing speed
> and I do
> get a lot of complaints when parsing slows down for any reason. The
> user doesn't
> expect this and it won't be visible what bitbake is doing (sitting at
> 99% parsed
> for a period).
> 
> Also, the "tend to be small" implies someone will create a huge one
> at some
> point even if that is a bad idea for whatever reasons, I just know
> how these
> things end up going :(.
> 
> > Unfortunately, I don't see any other way to access the repo
> > manifest
> > file, as we need to calculate the commit hashes of the git repos
> > referenced in the repo manifest file. Otherwise, it is impossible
> > for
> > us to determinate the necessity of an update when SRCREV =
> > "${AUTOREV}". 
> 
> Some further questions:
> 
> * Does it only clone a repo in the AUTOREV case?
> * Could it only obtain the manifest file somehow without a clone of
> the repo?
> 
> > However, I see one potential improvement here. Currently the
> > cloning of
> > the manifest repo is done on a per-recipe basis. E.g. this means if
> > we
> > have 10 recipes inheriting a bbclass containing a repo fetcher, we
> > will
> > clone 10 identical manifest repos. We'll work on improving this.
> 
> At least for wget or git, it is assumed that for a given url, there
> would be one
> tarball/clone and that there is locking in place to share it between
> them. This
> means you'll see do_fetch tasks for binutils, binutils-cross-XXX,
> nativesdk-
> binutils and binutils-native and one will block the others but the
> fetch will
> happen once and be shared between them. I guess with repo it may not
> be as
> simple as that but we should try and share what we can if possible.
> 
> Cheers,
> 
> Richard
> 
> 
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGNAZwACgkQYgqew07V
MNUYpAf/foJRPcTJZTmDixuz5ZmI/4x9RB47XWupJrwy//1hHNJDPyc3iHqj79te
6T8EuqIYUtYGIE2FHzkbVwAiyNQYCODhh3Uc/LmNoo5/59XypLNVgf+UH7KowytX
1YptbB477Dl/hR39ul7oDxdC9D+j0gHjClzs8KmHt0iMzzc1nLvih9G8yXkYNYVH
+0ovvlnpjHsC+WFYackEJe8CphUTFEn/EgffYtr/vl1SefRCzqs8aFB/gnhXKMjU
3zXH3DoeUZBdNahyEP9lAlbceM1AVoh9kNCXcVp/lUu1m7M3MVyxREMoeVfo22up
J1kZetA1LqX02PwDacU4Fftoc502Dw==
=BQ5W
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Richard and folks,

> The only other way I could imagine this to work would require us to
> keep the manifest file within the meta layer and do a somewhat
> "hacky"
> workaround:
> 
> It is possible to run `repo init` on a local git folder.
> As such, something like this could work:
> 
> 1) create temp folder (e.g. /tmp/repo-XXXXXXXXX)
> 2) run git init in temp folder
> 3) install manifest file into temp folder
> 4) add & commit
> 5) run repo init -u file://$temp-folder

today I discovered that repo also supports the --standalone-manifest
flag, which makes the workaround described above obsolete.
Thus, I believe supporting repo manifests from the layer repo only is
the cleanest and easiest way to implement this and we
wouldn't have to rely on cloning manifest repos during recipe parsing.

However, this would be a breaking change to the current repo fetcher
implementation, as it supports remote manifest repos.
I am pretty confident however that no one uses the arguably already
broken and unusable current repo fetcher implementation
in a production workload so this shouldn't be a major issue.

What are your thoughts on this?

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Thu, 2021-11-11 at 12:42 +0100, Jasper Orschulko wrote:
> Hi Richard,
> 
> > Even ten recipes using this will show a degradation in parsing
> > speed
> > and I do
> > get a lot of complaints when parsing slows down for any reason. The
> > user doesn't
> > expect this and it won't be visible what bitbake is doing (sitting
> > at
> > 99% parsed
> > for a period).
> > 
> > Also, the "tend to be small" implies someone will create a huge one
> > at some
> > point even if that is a bad idea for whatever reasons, I just know
> > how these
> > things end up going :(.
> 
> I believe this is something that could be addressed in the
> documentation of the repo fetcher? Something along the lines of
> "Using
> the repo fetcher can increase the time spent in the parsing phase, as
> we need to clone and inspect the manifest repository. To avoid this,
> keep the size of your manifest repository at a minimum."
> 
> When this is done, the dalay is barely noticable. We have a testsetup
> with 24 recipes and reasonable parsing speeds.
> 
> > * Does it only clone a repo in the AUTOREV case?
> 
> No, it also clones with fixed refspecs, as we want to ensure that
> each
> of the git repos referenced in the manifest also uses fixed refspecs.
> Otherwise, the sources in the background might change, while the
> recipe
> still (correctly) references the same revision of the manifest repo.
> 
> > * Could it only obtain the manifest file somehow without a clone of
> > the repo?
> 
> Unfortunately not really. git does not offer a way to list the
> content
> of a file without cloning it beforehand.
> 
> The only other way I could imagine this to work would require us to
> keep the manifest file within the meta layer and do a somewhat
> "hacky"
> workaround:
> 
> It is possible to run `repo init` on a local git folder.
> As such, something like this could work:
> 
> 1) create temp folder (e.g. /tmp/repo-XXXXXXXXX)
> 2) run git init in temp folder
> 3) install manifest file into temp folder
> 4) add & commit
> 5) run repo init -u file://$temp-folder
> 
> The repo manifest could even still be used outside of yocto, by using
> the metalayer repo as basis for the repo init: `repo init -u
> $metalayer-repo -m $path-to-manifest-file`. Not pretty, but it works.
> 
> So if cloning within the parser is a dealbreaker for you, this option
> might be worth looking into.
> 
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGeYnwACgkQYgqew07V
MNWXFAgAkZ7+CRZadUItM+Ay8uePffysuQhlwxHaAgzx1H+4Pk/HKK9QpEy3jFWz
kODr3DPOWlf6eS+AcqpJXVYdMsTucvE34eLIWb8reFkrPQKMtxp0N6U5gqz7e+Bm
Wf4lS04tqGZzC/Ld2DOs/zEnGKEwXBRHMX/9j6YvVMYxwKjxghW2LRloWpzghduH
j0FTAp1aGKLhCQjqeEuNZRMT+rn0i8k6u8Lxel1wRHi2LNhyT3VD3NThjVL64acH
FF83jhaxQ0JxpsSn9ouvJD3wMJHi72rdfGzznuQHmFT0lOxO5AjlcIOtK3KisiAI
KQ/jEV/44WN0JO7OpmHrVyOSfPeVBA==
=H0FZ
-----END PGP SIGNATURE-----

RE: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Peter Kjellerstedt]

> -----Original Message-----
> From: bitbake-devel@lists.openembedded.org <bitbake-
> devel@lists.openembedded.org> On Behalf Of Jasper Orschulko
> Sent: den 9 november 2021 12:26
> To: richard.purdie@linuxfoundation.org; openembedded-
> core@lists.openembedded.org; kweihmann@outlook.com; jasper@fancydomain.eu
> Cc: martin@mko.dev; Daniel.Baumgart@iris-sensing.net; bitbake-
> devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe
> for repo 2.17.3
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> Hi Richard,
> 
> I think our implementation of the repo fetcher checks the (what I
> believe to be) most relevant parts of your checklist (thanks for the
> write-up). Martin, please corrent me if I'm missing something:
> 
> > a) network access for sources is only expected to happen in the
> > do_fetch step.
> > This is not enforced or tested but is required so that we can:
> >
> >  i) audit the sources used (i.e. for license/manifest reasons)
> >  ii) support offline builds with a suitable cache
> >  iii) allow work to continue even with downtime upstream
> >  iv) allow for changes upstream in incompatible ways
> >  v) allow rebuilding of the software in X years time
> 
> check
> 
> > b) network access is not expected in do_unpack
> 
> check

How do you avoid the repo wrapper fetching the repo source itself 
and instead fetch it using the bitbake fetcher? I have seen nothing 
to that extent in the patches so far.

//Peter

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Peter,

> How do you avoid the repo wrapper fetching the repo source itself 
> and instead fetch it using the bitbake fetcher? I have seen nothing
> to that extent in the patches so far.

we don't. This recipe only installs the repo wrapper. The source code
is downloaded and installed when running `repo init`.

However, in our opinion this is not an issue. When installing repo as a
package to a target, this is the expected behaviour. The target would
only have the repo wrapper installed, just like on any other linux
distrobution. The actual usage of repo on the target is decoupled from
the bitbake build process.

When using the repo fetcher, the repo source is fetched during the
do_fetch stage by running `repo init` (when SRCREV = AUTOREV, changes
to the recipe or no previous sources available in DL_DIR). By executing
this within the "runfetchcmd" function, this also works with the usual
network features bitbake provides, e.g. proxy.
If the recipe has not changed and sources are already available from a
previous run, repo will not be rerun. As such, reproducing a build
offline is also possible.

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Wed, 2021-11-10 at 23:55 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: bitbake-devel@lists.openembedded.org <bitbake-
> > devel@lists.openembedded.org> On Behalf Of Jasper Orschulko
> > Sent: den 9 november 2021 12:26
> > To: richard.purdie@linuxfoundation.org; openembedded-
> > core@lists.openembedded.org; kweihmann@outlook.com;
> > jasper@fancydomain.eu
> > Cc: martin@mko.dev; Daniel.Baumgart@iris-sensing.net; bitbake-
> > devel@lists.openembedded.org
> > Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial
> > recipe
> > for repo 2.17.3
> > 
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> > 
> > Hi Richard,
> > 
> > I think our implementation of the repo fetcher checks the (what I
> > believe to be) most relevant parts of your checklist (thanks for the
> > write-up). Martin, please corrent me if I'm missing something:
> > 
> > > a) network access for sources is only expected to happen in the
> > > do_fetch step.
> > > This is not enforced or tested but is required so that we can:
> > > 
> > >  i) audit the sources used (i.e. for license/manifest reasons)
> > >  ii) support offline builds with a suitable cache
> > >  iii) allow work to continue even with downtime upstream
> > >  iv) allow for changes upstream in incompatible ways
> > >  v) allow rebuilding of the software in X years time
> > 
> > check
> > 
> > > b) network access is not expected in do_unpack
> > 
> > check
> 
> How do you avoid the repo wrapper fetching the repo source itself 
> and instead fetch it using the bitbake fetcher? I have seen nothing 
> to that extent in the patches so far.
> 
> //Peter
> 
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGM6rUACgkQYgqew07V
MNUCIAf+MXGp9Hnfxmu+8w3BasVsP2N7ttW2FN3Zroiyr/hKrJzeq/qDQwO+/K3U
zWZJ85H6L2eTjOP8fPaHbVMRD1jUoYVpYUAE/fN64FbhCBmurVuWFrLo/u6Gy2cU
DCd3SIejXidB25EQRoS4Bfl25il1wG4iMw1pCFA6ku5rGhb8q5jvfZECf0Wuhh7X
FnMQlI3VZsSk4CakF3g88AFTLFrsQYcN2vDUskdhMfTZpuRA8duIvW4rVgOvHJQT
v07rASNR/9yzPRVkzpgPUI9Vl2LA3vEZ3Rccw3jscYHFwkzj0096DO4+jeDwyza5
fFm0dDT9HjGuzTz66c5JL8sdZZi9pw==
=AOyp
-----END PGP SIGNATURE-----

RE: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Peter Kjellerstedt]

> -----Original Message-----
> From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> Sent: den 11 november 2021 11:05
> To: richard.purdie@linuxfoundation.org; openembedded-
> core@lists.openembedded.org; kweihmann@outlook.com; Peter Kjellerstedt
> <peter.kjellerstedt@axis.com>; jasper@fancydomain.eu
> Cc: martin@mko.dev; Daniel Baumgart <Daniel.Baumgart@iris-sensing.com>;
> bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe
> for repo 2.17.3
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> Hi Peter,
> 
> > How do you avoid the repo wrapper fetching the repo source itself
> > and instead fetch it using the bitbake fetcher? I have seen nothing
> > to that extent in the patches so far.
> 
> we don't. This recipe only installs the repo wrapper. The source code
> is downloaded and installed when running `repo init`.
> 
> However, in our opinion this is not an issue. When installing repo as a
> package to a target, this is the expected behaviour. The target would
> only have the repo wrapper installed, just like on any other linux
> distrobution. The actual usage of repo on the target is decoupled from
> the bitbake build process.

It might be how repo is designed, but it will still break the bitbake 
expectations. I.e., if I have an environment with the layers available, 
a populated DL_DIR and BB_NO_NETWORK = "1", and then disconnect the 
build host from the Internet, I should still be able to source 
oe-init-build-env for a new machine and build it. However, this means 
that only the source for the repo wrapper is available in DL_DIR. So 
when the repo wrapper executes it will try to go on the Internet to 
fetch the rest of the repo source, which will fail.

> When using the repo fetcher, the repo source is fetched during the
> do_fetch stage by running `repo init` (when SRCREV = AUTOREV, changes
> to the recipe or no previous sources available in DL_DIR). By executing
> this within the "runfetchcmd" function, this also works with the usual
> network features bitbake provides, e.g. proxy.
> If the recipe has not changed and sources are already available from a
> previous run, repo will not be rerun. As such, reproducing a build
> offline is also possible.
> 
> - --
> With best regards
> 
> Jasper Orschulko
> DevOps Engineer
> 
> Tel. +49 30 58 58 14 265
> Fax +49 30 58 58 14 999
> Jasper.Orschulko@iris-sensing.com
> 
> • • • • • • • • • • • • • • • • • • • • • • • • • •
> 
> iris-GmbH
> infrared & intelligent sensors
> Schnellerstraße 1-5 | 12439 Berlin
> 
> https://iris-sensing.com/
> 
> 
> 
> 
> 
> On Wed, 2021-11-10 at 23:55 +0000, Peter Kjellerstedt wrote:
> > > -----Original Message-----
> > > From: bitbake-devel@lists.openembedded.org <bitbake-
> > > devel@lists.openembedded.org> On Behalf Of Jasper Orschulko
> > > Sent: den 9 november 2021 12:26
> > > To: richard.purdie@linuxfoundation.org; openembedded-
> > > core@lists.openembedded.org; kweihmann@outlook.com;
> > > jasper@fancydomain.eu
> > > Cc: martin@mko.dev; Daniel.Baumgart@iris-sensing.net; bitbake-
> > > devel@lists.openembedded.org
> > > Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial
> > > recipe
> > > for repo 2.17.3
> > >
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA256
> > >
> > > Hi Richard,
> > >
> > > I think our implementation of the repo fetcher checks the (what I
> > > believe to be) most relevant parts of your checklist (thanks for the
> > > write-up). Martin, please corrent me if I'm missing something:
> > >
> > > > a) network access for sources is only expected to happen in the
> > > > do_fetch step.
> > > > This is not enforced or tested but is required so that we can:
> > > >
> > > >  i) audit the sources used (i.e. for license/manifest reasons)
> > > >  ii) support offline builds with a suitable cache
> > > >  iii) allow work to continue even with downtime upstream
> > > >  iv) allow for changes upstream in incompatible ways
> > > >  v) allow rebuilding of the software in X years time
> > >
> > > check
> > >
> > > > b) network access is not expected in do_unpack
> > >
> > > check
> >
> > How do you avoid the repo wrapper fetching the repo source itself
> > and instead fetch it using the bitbake fetcher? I have seen nothing
> > to that extent in the patches so far.
> >
> > //Peter
> >
> -----BEGIN PGP SIGNATURE-----
> 
> iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGM6rUACgkQYgqew07V
> MNUCIAf+MXGp9Hnfxmu+8w3BasVsP2N7ttW2FN3Zroiyr/hKrJzeq/qDQwO+/K3U
> zWZJ85H6L2eTjOP8fPaHbVMRD1jUoYVpYUAE/fN64FbhCBmurVuWFrLo/u6Gy2cU
> DCd3SIejXidB25EQRoS4Bfl25il1wG4iMw1pCFA6ku5rGhb8q5jvfZECf0Wuhh7X
> FnMQlI3VZsSk4CakF3g88AFTLFrsQYcN2vDUskdhMfTZpuRA8duIvW4rVgOvHJQT
> v07rASNR/9yzPRVkzpgPUI9Vl2LA3vEZ3Rccw3jscYHFwkzj0096DO4+jeDwyza5
> fFm0dDT9HjGuzTz66c5JL8sdZZi9pw==
> =AOyp
> -----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Peter,

> However, this means 
> that only the source for the repo wrapper is available in DL_DIR. So 
> when the repo wrapper executes it will try to go on the Internet to 
> fetch the rest of the repo source, which will fail.

just to clearify: Are we talking about installing repo on a target, or
are we talking about the repo fetcher?

When installing repo on the target, we do not call any "repo" command
within the bitbake build process. We just clone the git repo containing
the wrapper script and install the wrapper script into the sysroot. At
that point the bitbake build process is finished and the user ends up
with just the wrapper script on his device. Just as he would when
installing repo from the system repositories on debian, arch, etc.

When using the fetcher, the repo sources are fetched during the initial
do_fetch (this also works with `bitbake --runonly=do_fetch`) and placed
into the DL_DIR (in the .repo folder, next to the sources as described
in the repo manifest). So unless the user changes something fetcher
related in the recipe or deletes his DL_DIR, he can use this build
environment for an offline build.

You can try this for yourself, if you like:

1) clone https://github.com/Jasper-Ben/demo-kas
2) (with docker and docker-compose installed) run `make offline-build`

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Thu, 2021-11-11 at 11:34 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > Sent: den 11 november 2021 11:05
> > To: richard.purdie@linuxfoundation.org; openembedded-
> > core@lists.openembedded.org; kweihmann@outlook.com; Peter
> > Kjellerstedt
> > <peter.kjellerstedt@axis.com>; jasper@fancydomain.eu
> > Cc: martin@mko.dev; Daniel Baumgart
> > <Daniel.Baumgart@iris-sensing.com>;
> > bitbake-devel@lists.openembedded.org
> > Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial
> > recipe
> > for repo 2.17.3
> > 
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> > 
> > Hi Peter,
> > 
> > > How do you avoid the repo wrapper fetching the repo source itself
> > > and instead fetch it using the bitbake fetcher? I have seen nothing
> > > to that extent in the patches so far.
> > 
> > we don't. This recipe only installs the repo wrapper. The source code
> > is downloaded and installed when running `repo init`.
> > 
> > However, in our opinion this is not an issue. When installing repo as
> > a
> > package to a target, this is the expected behaviour. The target would
> > only have the repo wrapper installed, just like on any other linux
> > distrobution. The actual usage of repo on the target is decoupled
> > from
> > the bitbake build process.
> 
> It might be how repo is designed, but it will still break the bitbake
> expectations. I.e., if I have an environment with the layers available,
> a populated DL_DIR and BB_NO_NETWORK = "1", and then disconnect the 
> build host from the Internet, I should still be able to source 
> oe-init-build-env for a new machine and build it. However, this means
> that only the source for the repo wrapper is available in DL_DIR. So 
> when the repo wrapper executes it will try to go on the Internet to 
> fetch the rest of the repo source, which will fail.
> 
> > When using the repo fetcher, the repo source is fetched during the
> > do_fetch stage by running `repo init` (when SRCREV = AUTOREV, changes
> > to the recipe or no previous sources available in DL_DIR). By
> > executing
> > this within the "runfetchcmd" function, this also works with the
> > usual
> > network features bitbake provides, e.g. proxy.
> > If the recipe has not changed and sources are already available from
> > a
> > previous run, repo will not be rerun. As such, reproducing a build
> > offline is also possible.
> > 
> > - --
> > With best regards
> > 
> > Jasper Orschulko
> > DevOps Engineer
> > 
> > Tel. +49 30 58 58 14 265
> > Fax +49 30 58 58 14 999
> > Jasper.Orschulko@iris-sensing.com
> > 
> > • • • • • • • • • • • • • • • • • • • • • • • • • •
> > 
> > iris-GmbH
> > infrared & intelligent sensors
> > Schnellerstraße 1-5 | 12439 Berlin
> > 
> > https://iris-sensing.com/
> > 
> > 
> > 
> > 
> > 
> > On Wed, 2021-11-10 at 23:55 +0000, Peter Kjellerstedt wrote:
> > > > -----Original Message-----
> > > > From: bitbake-devel@lists.openembedded.org <bitbake-
> > > > devel@lists.openembedded.org> On Behalf Of Jasper Orschulko
> > > > Sent: den 9 november 2021 12:26
> > > > To: richard.purdie@linuxfoundation.org; openembedded-
> > > > core@lists.openembedded.org; kweihmann@outlook.com;
> > > > jasper@fancydomain.eu
> > > > Cc: martin@mko.dev; Daniel.Baumgart@iris-sensing.net; bitbake-
> > > > devel@lists.openembedded.org
> > > > Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools:
> > > > Initial
> > > > recipe
> > > > for repo 2.17.3
> > > > 
> > > > -----BEGIN PGP SIGNED MESSAGE-----
> > > > Hash: SHA256
> > > > 
> > > > Hi Richard,
> > > > 
> > > > I think our implementation of the repo fetcher checks the (what I
> > > > believe to be) most relevant parts of your checklist (thanks for
> > > > the
> > > > write-up). Martin, please corrent me if I'm missing something:
> > > > 
> > > > > a) network access for sources is only expected to happen in the
> > > > > do_fetch step.
> > > > > This is not enforced or tested but is required so that we can:
> > > > > 
> > > > >  i) audit the sources used (i.e. for license/manifest reasons)
> > > > >  ii) support offline builds with a suitable cache
> > > > >  iii) allow work to continue even with downtime upstream
> > > > >  iv) allow for changes upstream in incompatible ways
> > > > >  v) allow rebuilding of the software in X years time
> > > > 
> > > > check
> > > > 
> > > > > b) network access is not expected in do_unpack
> > > > 
> > > > check
> > > 
> > > How do you avoid the repo wrapper fetching the repo source itself
> > > and instead fetch it using the bitbake fetcher? I have seen nothing
> > > to that extent in the patches so far.
> > > 
> > > //Peter
> > > 
> > -----BEGIN PGP SIGNATURE-----
> > 
> > iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGM6rUACgkQYgqew07V
> > MNUCIAf+MXGp9Hnfxmu+8w3BasVsP2N7ttW2FN3Zroiyr/hKrJzeq/qDQwO+/K3U
> > zWZJ85H6L2eTjOP8fPaHbVMRD1jUoYVpYUAE/fN64FbhCBmurVuWFrLo/u6Gy2cU
> > DCd3SIejXidB25EQRoS4Bfl25il1wG4iMw1pCFA6ku5rGhb8q5jvfZECf0Wuhh7X
> > FnMQlI3VZsSk4CakF3g88AFTLFrsQYcN2vDUskdhMfTZpuRA8duIvW4rVgOvHJQT
> > v07rASNR/9yzPRVkzpgPUI9Vl2LA3vEZ3Rccw3jscYHFwkzj0096DO4+jeDwyza5
> > fFm0dDT9HjGuzTz66c5JL8sdZZi9pw==
> > =AOyp
> > -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGNCDIACgkQYgqew07V
MNXG5ggAlwuv5I3Pi3aG9typiSePABvw0cQT+GbLysr74UTU7UJNDd+ByeM5U7Vc
lmFuVO6SKRct0brPy5W7weZxpS7YxgwrFc6QliUpsd4yIZJkPTZVBzdqBxskMU/f
z0vHb9lipyXPkg4OU49TGv2k3rBBcrYGMSwG8WiqJL+f2gXHDaCbwb/1oJuI1Ie5
LXW1nkIPViaRuoPgKdlvzptFNI+vBuIRg8KvwSf4W5pfJYC03jZLdRthWj1TNQvz
653Krzyvu3hhxIKgn9YYPb/mI3ZI1NgYm/9hpEYwjtZSLq303uLMB70qYWj6fG/V
uHaBFN7b2sF4hbMffJ3jKOEIj/mgSQ==
=PARI
-----END PGP SIGNATURE-----

RE: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Peter Kjellerstedt]

> -----Original Message-----
> From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> Sent: den 11 november 2021 13:10
> To: richard.purdie@linuxfoundation.org; openembedded-
> core@lists.openembedded.org; kweihmann@outlook.com; Peter Kjellerstedt
> <peter.kjellerstedt@axis.com>; jasper@fancydomain.eu
> Cc: martin@mko.dev; Daniel Baumgart <Daniel.Baumgart@iris-sensing.com>;
> bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe
> for repo 2.17.3
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> Hi Peter,
> 
> > However, this means
> > that only the source for the repo wrapper is available in DL_DIR. So
> > when the repo wrapper executes it will try to go on the Internet to
> > fetch the rest of the repo source, which will fail.
> 
> just to clearify: Are we talking about installing repo on a target, or
> are we talking about the repo fetcher?

The repo fetcher, i.e., when repo is used by bitbake.

> When installing repo on the target, we do not call any "repo" command
> within the bitbake build process. We just clone the git repo containing
> the wrapper script and install the wrapper script into the sysroot. At
> that point the bitbake build process is finished and the user ends up
> with just the wrapper script on his device. Just as he would when
> installing repo from the system repositories on debian, arch, etc.
> 
> When using the fetcher, the repo sources are fetched during the initial
> do_fetch (this also works with `bitbake --runonly=do_fetch`) and placed
> into the DL_DIR (in the .repo folder, next to the sources as described
> in the repo manifest). So unless the user changes something fetcher
> related in the recipe or deletes his DL_DIR, he can use this build
> environment for an offline build.

How can it work? The repo source, which is fetched by the repo wrapper, 
will not be in DL_DIR or handled by bitbake at all. This means the 
repo wrapper will try to contact googlesource.com directly and fetch it 
when it is run by bitbake, which cannot work since the host is 
disconnected from the network. How or from where would it be able to 
get the sources it needs?

> You can try this for yourself, if you like:
> 
> 1) clone https://github.com/Jasper-Ben/demo-kas
> 2) (with docker and docker-compose installed) run `make offline-build`
> 
> - --
> With best regards
> 
> Jasper Orschulko
> DevOps Engineer
> 
> Tel. +49 30 58 58 14 265
> Fax +49 30 58 58 14 999
> Jasper.Orschulko@iris-sensing.com
> 
> • • • • • • • • • • • • • • • • • • • • • • • • • •
> 
> iris-GmbH
> infrared & intelligent sensors
> Schnellerstraße 1-5 | 12439 Berlin
> 
> https://iris-sensing.com/

//Peter

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Peter,

during this initial fetch it is true, that the repo fetcher has to
download the repo sources from googlesource.com. But during this first
fetch the fetcher also needs to download any sources listed in the repo
manifest anyway, so internet access is a given.

Consequential fetches will only re-run repo if:
a) the SRC_URI or SRCREV has changed within the recipe
b) the SRCREV is set to AUTOREV

Both cases do not apply when reproducing a build offline.

So if you have done this initial fetch of your sources and stash your
working dir away, you can do an offline build.

- -- 

With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Thu, 2021-11-11 at 14:11 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > Sent: den 11 november 2021 13:10
> > To: richard.purdie@linuxfoundation.org; openembedded-
> > core@lists.openembedded.org; kweihmann@outlook.com; Peter
> > Kjellerstedt
> > <peter.kjellerstedt@axis.com>; jasper@fancydomain.eu
> > Cc: martin@mko.dev; Daniel Baumgart
> > <Daniel.Baumgart@iris-sensing.com>;
> > bitbake-devel@lists.openembedded.org
> > Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial
> > recipe
> > for repo 2.17.3
> > 
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> > 
> > Hi Peter,
> > 
> > > However, this means
> > > that only the source for the repo wrapper is available in DL_DIR.
> > > So
> > > when the repo wrapper executes it will try to go on the Internet
> > > to
> > > fetch the rest of the repo source, which will fail.
> > 
> > just to clearify: Are we talking about installing repo on a target,
> > or
> > are we talking about the repo fetcher?
> 
> The repo fetcher, i.e., when repo is used by bitbake.
> 
> > When installing repo on the target, we do not call any "repo"
> > command
> > within the bitbake build process. We just clone the git repo
> > containing
> > the wrapper script and install the wrapper script into the sysroot.
> > At
> > that point the bitbake build process is finished and the user ends
> > up
> > with just the wrapper script on his device. Just as he would when
> > installing repo from the system repositories on debian, arch, etc.
> > 
> > When using the fetcher, the repo sources are fetched during the
> > initial
> > do_fetch (this also works with `bitbake --runonly=do_fetch`) and
> > placed
> > into the DL_DIR (in the .repo folder, next to the sources as
> > described
> > in the repo manifest). So unless the user changes something fetcher
> > related in the recipe or deletes his DL_DIR, he can use this build
> > environment for an offline build.
> 
> How can it work? The repo source, which is fetched by the repo
> wrapper, 
> will not be in DL_DIR or handled by bitbake at all. This means the 
> repo wrapper will try to contact googlesource.com directly and fetch
> it 
> when it is run by bitbake, which cannot work since the host is 
> disconnected from the network. How or from where would it be able to 
> get the sources it needs?
> 
> > You can try this for yourself, if you like:
> > 
> > 1) clone https://github.com/Jasper-Ben/demo-kas
> > 2) (with docker and docker-compose installed) run `make offline-
> > build`
> > 
> > - --
> > With best regards
> > 
> > Jasper Orschulko
> > DevOps Engineer
> > 
> > Tel. +49 30 58 58 14 265
> > Fax +49 30 58 58 14 999
> > Jasper.Orschulko@iris-sensing.com
> > 
> > • • • • • • • • • • • • • • • • • • • • • • • • • •
> > 
> > iris-GmbH
> > infrared & intelligent sensors
> > Schnellerstraße 1-5 | 12439 Berlin
> > 
> > https://iris-sensing.com/
> 
> //Peter
> 
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGNMesACgkQYgqew07V
MNU//Af+IFFqmbtV/k3SscMUWYRNDpnSeK6elwfpwnDi84tQHjRo66Xtrzi12D+H
zzya3H87unVY3BWqWnQIE3K4aShquz+p9tQA7zuzlARImCyqAsDQzKddD7hhK7AI
4toNobOqNoJZ2D1kJQyu7o7VsnvbiyvYz8Z8l+t4FF35+bwU26aE0iCFF63toZSi
GLvmTzLigYNAu9UUW+UgSUAhHVsaCyOolnGwBdHEJW3NliCZEOLDf8omDygG3xDp
UOKcX2Q238sGN2fCwAeS2ABC4ffMNDfrpzAS0Q1qy2WVmheWmmtGoaksW2z3sDZk
FKxLlRWNQvPWOy/tY2JLyB1QNyedyg==
=c20l
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 436 bytes --]

On Thu, 11 Nov 2021 at 16:08, Jasper Orschulko <
Jasper.Orschulko@iris-sensing.com> wrote:

> So if you have done this initial fetch of your sources and stash your
> working dir away, you can do an offline build.
>

But can you do an offline build without a prepopulated working dir? That's
the crucial point: offline build should work in a freshly set up build
directory, without sstate cache, and with prepopulated DL_DIR only.

Alex

[-- Attachment #2: Type: text/html, Size: 866 bytes --]

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Alex,

thanks for your input. You are absolutely correct, this currently does
not work. We'll look into this.

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Thu, 2021-11-11 at 20:20 +0100, Alexander Kanavin wrote:
> On Thu, 11 Nov 2021 at 16:08, Jasper Orschulko
> <Jasper.Orschulko@iris-sensing.com> wrote:
> > So if you have done this initial fetch of your sources and stash
> > your
> > working dir away, you can do an offline build.
> 
> But can you do an offline build without a prepopulated working dir?
> That's the crucial point: offline build should work in a freshly set
> up build directory, without sstate cache, and with prepopulated
> DL_DIR only.
> 
> Alex
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGOXJwACgkQYgqew07V
MNXaxwf+NDOx5TYR4dY4iCsb0IfdfoB01+H7UDZR2s5GZKKA9TyMKbEqBdZrHBdj
PTqZJ339IOXBlks1KPAUci0VOXBdwHoybR2XFMGENTnrj6XA+9Yr96ofSPHqQTWr
amD/low0cFvfqZClb+md/FcxhnVK45rFhfLJckKElPiYmKjcghs/PXAWbh+Tw6S7
m5KCq8agPqXnaU/bWDmj2eC9t6t+VT/0OnC0tQtohMbYVd8wul4cf5Ic3JilecjI
qCH4rK7EBFfzWhs1WTYDGeXX5iWNmZOsdufpXF0URJa5cpjGA5bhjfyXjoUZPzCR
WxErD1cnQSNQqysqWlnB5EgM3y+cBg==
=q0zZ
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Alexander,

"[PATCH v3] fetch2/repo: Implement AUTOREV for repo fetcher" contains a
fix for this issue. Reproducing a build offline with only the DL_DIR
should work now.

We are currently still lacking the appropriate tests for the fetcher.
These will follow soon.

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Thu, 2021-11-11 at 20:20 +0100, Alexander Kanavin wrote:
> On Thu, 11 Nov 2021 at 16:08, Jasper Orschulko
> <Jasper.Orschulko@iris-sensing.com> wrote:
> > So if you have done this initial fetch of your sources and stash
> > your
> > working dir away, you can do an offline build.
> 
> But can you do an offline build without a prepopulated working dir?
> That's the crucial point: offline build should work in a freshly set
> up build directory, without sstate cache, and with prepopulated
> DL_DIR only.
> 
> Alex
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGSWb4ACgkQYgqew07V
MNVUYQgAgA6+8XI+JxtA532FNNIMAZG5ZGZ12GKvVkjoJErnzRf0RZxUG3UyNYmm
pSDcfbuodBmBbHxDBpN8VspToKZy8dKcm/jfe9UY7f9U39bZSNXSS3hNPqkqKP5J
jCg1zQfxjbW1aVSHJkQB5MoY/EDeoQVv7RI6RmcVsLJZxozPeYaOQWSdTNG/7czg
oFjKr6jtSOYO45teMGt3+2+AtAXhlzTgKZo33rX/tknS7f1+kkyNGLv0ssSP5Jcf
Bd8YbcdG9lqXKUo0A7rn9Q83YBdpIyA6+By159zzPy7bP8j9aqMH8nNWauahhcFS
sY39fiky57aEgMe7g1OZX8oRODowPA==
=hpAK
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2019 bytes --]

Thanks, maybe you should write a test for offline builds as well then :)

Alex

On Mon, 15 Nov 2021 at 13:59, Jasper Orschulko <
Jasper.Orschulko@iris-sensing.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Hi Alexander,
>
> "[PATCH v3] fetch2/repo: Implement AUTOREV for repo fetcher" contains a
> fix for this issue. Reproducing a build offline with only the DL_DIR
> should work now.
>
> We are currently still lacking the appropriate tests for the fetcher.
> These will follow soon.
>
> - --
> With best regards
>
> Jasper Orschulko
> DevOps Engineer
>
> Tel. +49 30 58 58 14 265
> Fax +49 30 58 58 14 999
> Jasper.Orschulko@iris-sensing.com
>
> • • • • • • • • • • • • • • • • • • • • • • • • • •
>
> iris-GmbH
> infrared & intelligent sensors
> Schnellerstraße 1-5 | 12439 Berlin
>
> https://iris-sensing.com/
>
>
>
>
>
> On Thu, 2021-11-11 at 20:20 +0100, Alexander Kanavin wrote:
> > On Thu, 11 Nov 2021 at 16:08, Jasper Orschulko
> > <Jasper.Orschulko@iris-sensing.com> wrote:
> > > So if you have done this initial fetch of your sources and stash
> > > your
> > > working dir away, you can do an offline build.
> >
> > But can you do an offline build without a prepopulated working dir?
> > That's the crucial point: offline build should work in a freshly set
> > up build directory, without sstate cache, and with prepopulated
> > DL_DIR only.
> >
> > Alex
> -----BEGIN PGP SIGNATURE-----
>
> iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGSWb4ACgkQYgqew07V
> MNVUYQgAgA6+8XI+JxtA532FNNIMAZG5ZGZ12GKvVkjoJErnzRf0RZxUG3UyNYmm
> pSDcfbuodBmBbHxDBpN8VspToKZy8dKcm/jfe9UY7f9U39bZSNXSS3hNPqkqKP5J
> jCg1zQfxjbW1aVSHJkQB5MoY/EDeoQVv7RI6RmcVsLJZxozPeYaOQWSdTNG/7czg
> oFjKr6jtSOYO45teMGt3+2+AtAXhlzTgKZo33rX/tknS7f1+kkyNGLv0ssSP5Jcf
> Bd8YbcdG9lqXKUo0A7rn9Q83YBdpIyA6+By159zzPy7bP8j9aqMH8nNWauahhcFS
> sY39fiky57aEgMe7g1OZX8oRODowPA==
> =hpAK
> -----END PGP SIGNATURE-----
>

[-- Attachment #2: Type: text/html, Size: 2758 bytes --]

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Will do! :)

This only affects the other patch series though, the recipe itself
should be fine now.

It seems the only question left open is this:

> > I think we can replace the patch 0001-Set-REPO_REV-to-v2.17.3.patch
> > with a post function
> > and with it we can reuse the SRCREV of the recipe. Something like:
> > 
> > do_fix_rev(){
> >     sed -i "s/REPO_REV = 'stable'/REPO_REV = '${SRCREV}'" ${S}/repo
> > }
> > 
> > do_patch[postfuncs] += "do_fix_rev"
> 
> Yeah... was thinking about that as well. Wasn't sure though if this
> would be considered bad practise. What do the others think about
> this?

Some feedback on this would be appreciated. Thanks!

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Mon, 2021-11-15 at 14:05 +0100, Alexander Kanavin wrote:
> Thanks, maybe you should write a test for offline builds as well then
> :)
> 
> Alex
> 
> On Mon, 15 Nov 2021 at 13:59, Jasper Orschulko
> <Jasper.Orschulko@iris-sensing.com> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> > 
> > Hi Alexander,
> > 
> > "[PATCH v3] fetch2/repo: Implement AUTOREV for repo fetcher"
> > contains
> > a
> > fix for this issue. Reproducing a build offline with only the
> > DL_DIR
> > should work now.
> > 
> > We are currently still lacking the appropriate tests for the
> > fetcher.
> > These will follow soon.
> > 
> > - -- 
> > With best regards
> > 
> > Jasper Orschulko
> > DevOps Engineer
> > 
> > Tel. +49 30 58 58 14 265
> > Fax +49 30 58 58 14 999
> > Jasper.Orschulko@iris-sensing.com
> > 
> > • • • • • • • • • • • • • • • • • • • • • • • • • •
> > 
> > iris-GmbH
> > infrared & intelligent sensors
> > Schnellerstraße 1-5 | 12439 Berlin
> > 
> > https://iris-sensing.com/
> > 
> > 
> > 
> > 
> > 
> > On Thu, 2021-11-11 at 20:20 +0100, Alexander Kanavin wrote:
> > > On Thu, 11 Nov 2021 at 16:08, Jasper Orschulko
> > > <Jasper.Orschulko@iris-sensing.com> wrote:
> > > > So if you have done this initial fetch of your sources and
> > > > stash
> > > > your
> > > > working dir away, you can do an offline build.
> > > 
> > > But can you do an offline build without a prepopulated working
> > > dir?
> > > That's the crucial point: offline build should work in a freshly
> > set
> > > up build directory, without sstate cache, and with prepopulated
> > > DL_DIR only.
> > > 
> > > Alex
> > -----BEGIN PGP SIGNATURE-----
> > 
> > iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGSWb4ACgkQYgqew07V
> > MNVUYQgAgA6+8XI+JxtA532FNNIMAZG5ZGZ12GKvVkjoJErnzRf0RZxUG3UyNYmm
> > pSDcfbuodBmBbHxDBpN8VspToKZy8dKcm/jfe9UY7f9U39bZSNXSS3hNPqkqKP5J
> > jCg1zQfxjbW1aVSHJkQB5MoY/EDeoQVv7RI6RmcVsLJZxozPeYaOQWSdTNG/7czg
> > oFjKr6jtSOYO45teMGt3+2+AtAXhlzTgKZo33rX/tknS7f1+kkyNGLv0ssSP5Jcf
> > Bd8YbcdG9lqXKUo0A7rn9Q83YBdpIyA6+By159zzPy7bP8j9aqMH8nNWauahhcFS
> > sY39fiky57aEgMe7g1OZX8oRODowPA==
> > =hpAK
> > -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGSXNkACgkQYgqew07V
MNUKmQf/UYTURPKZ0iY2MKPFB60WdnmsTKJMjotQlcVBUpiwQ6LhSpkfWRqOFLpN
na9cNZbYjGFeDNdL3d0ILOEQ+oVhLY7bQpj5Xb2nDsIWspjlHg1IkkvCVGa9zFb3
XALaDrGBrlad9KNhI4C2CmXjdJZgx3yBuuvK6mSiga5fs44QB/lBb9JOQGYucRt+
QeLkIatvFo1Jxzmc6tNkRr3osgacQroagHj4CISh5W+ezWqtdWHA5ndf4HvCNwJV
hkpyXtxeQDINGVGBsMnJkOIWRSStOviL66Q1h5/dgbJ23FXQuEkM9CyD19kXWSIW
ZSI6as4bzjDqDVPzCPWHDmQdSnJyXQ==
=aNK6
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 5545 bytes --]

Wait, what is the use case for the repo fetcher?

Alex

On Fri, 5 Nov 2021 at 14:31, Jasper Orschulko via lists.openembedded.org
<jasper=fancydomain.eu@lists.openembedded.org> wrote:

> From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
>
> Add a recipe for repo, prerequisite for the repo fetcher.
>
> Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> ---
>  .../repo/files/0001-python3-shebang.patch     | 21 ++++++++++++
>  .../0001-Set-REPO_REV-to-v2.17.3.patch        | 33 +++++++++++++++++++
>  meta/recipes-devtools/repo/repo.inc           | 25 ++++++++++++++
>  meta/recipes-devtools/repo/repo_2.17.3.bb     |  7 ++++
>  4 files changed, 86 insertions(+)
>  create mode 100644
> meta/recipes-devtools/repo/files/0001-python3-shebang.patch
>  create mode 100644
> meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
>  create mode 100644 meta/recipes-devtools/repo/repo.inc
>  create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
>
> diff --git a/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
> b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
> new file mode 100644
> index 0000000000..09ccf58264
> --- /dev/null
> +++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
> @@ -0,0 +1,21 @@
> +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001
> +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> +Date: Tue, 14 Sep 2021 16:46:51 +0200
> +Subject: [PATCH] python3 shebang
> +
> +---
> + repo | 2 +-
> + 1 file changed, 1 insertion(+), 1 deletion(-)
> +
> +diff --git a/repo b/repo
> +index b13e34c..205e0e5 100755
> +--- a/repo
> ++++ b/repo
> +@@ -1,4 +1,4 @@
> +-#!/usr/bin/env python
> ++#!/usr/bin/env python3
> + # -*- coding:utf-8 -*-
> + #
> + # Copyright (C) 2008 The Android Open Source Project
> +--
> +2.33.0
> diff --git
> a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> new file mode 100644
> index 0000000000..294a3af53a
> --- /dev/null
> +++
> b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> @@ -0,0 +1,33 @@
> +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
> +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> +Date: Thu, 4 Nov 2021 16:55:12 +0100
> +Subject: [PATCH] Set REPO_REV to v2.17.3
> +
> +repo is an unusual tool because it downloads all of its own Python modules
> +using GPG-signed git tags, and stores those files as part of the project
> +that it is working with.
> +
> +So in order to have a reproducible repo installation within the project
> +folders, we hardcode the REPO_REV variable to this recipes PV.
> +
> +Upstream-Status: Inappropriate [configuration]
> +Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> +---
> + repo | 2 +-
> + 1 file changed, 1 insertion(+), 1 deletion(-)
> +
> +diff --git a/repo b/repo
> +index 4cddbf1..cf5f6b1 100755
> +--- a/repo
> ++++ b/repo
> +@@ -142,7 +142,7 @@ if __name__ == '__main__':
> + REPO_URL = os.environ.get('REPO_URL', None)
> + if not REPO_URL:
> +   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> +-REPO_REV = os.environ.get('REPO_REV')
> ++REPO_REV = 'v2.17.3'
> + if not REPO_REV:
> +   REPO_REV = 'stable'
> + # URL to file bug reports for repo tool issues.
> +--
> +2.33.1
> diff --git a/meta/recipes-devtools/repo/repo.inc
> b/meta/recipes-devtools/repo/repo.inc
> new file mode 100644
> index 0000000000..60b32e4d74
> --- /dev/null
> +++ b/meta/recipes-devtools/repo/repo.inc
> @@ -0,0 +1,25 @@
> +# SPDX-License-Identifier: MIT
> +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> +
> +SUMMARY = "Tool for managing many Git repositories"
> +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many
> Git repositories, does the uploads to revision control systems, and
> automates parts of the development workflow."
> +HOMEPAGE = "https://android.googlesource.com/tools/repo"
> +SECTION = "console/utils"
> +
> +LICENSE = "Apache-2.0"
> +
> +SRC_URI = "git://
> git@gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
> +MIRRORS = "git://git@gerrit.googlesource.com/git-repo.git git://
> github.com/GerritCodeReview/git-repo.git \n"
> +
> +SRC_URI += "file://0001-python3-shebang.patch"
> +
> +S = "${WORKDIR}/git"
> +
> +RDEPENDS_${PN} = "python3"
> +
> +do_install() {
> +    install -d ${D}${bindir}
> +    install -m 755 ${WORKDIR}/git/repo ${D}${bindir}
> +}
> +
> +BBCLASSEXTEND = "native nativesdk"
> diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
> b/meta/recipes-devtools/repo/repo_2.17.3.bb
> new file mode 100644
> index 0000000000..c26264b9e9
> --- /dev/null
> +++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
> @@ -0,0 +1,7 @@
> +# SPDX-License-Identifier: MIT
> +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> +
> +require recipes-devtools/repo/repo.inc
> +
> +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
> +LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
> --
> 2.33.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#12893):
> https://lists.openembedded.org/g/bitbake-devel/message/12893
> Mute This Topic: https://lists.openembedded.org/mt/86840271/1686489
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 8207 bytes --]

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 5934 bytes --]

Ah I didn't notice it's something that already exists. Still, I'm curious
for which items is it used, and why repo over other options?

Alex

On Fri, 5 Nov 2021 at 15:20, Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> Wait, what is the use case for the repo fetcher?
>
> Alex
>
> On Fri, 5 Nov 2021 at 14:31, Jasper Orschulko via lists.openembedded.org
> <jasper=fancydomain.eu@lists.openembedded.org> wrote:
>
>> From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
>>
>> Add a recipe for repo, prerequisite for the repo fetcher.
>>
>> Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
>> ---
>>  .../repo/files/0001-python3-shebang.patch     | 21 ++++++++++++
>>  .../0001-Set-REPO_REV-to-v2.17.3.patch        | 33 +++++++++++++++++++
>>  meta/recipes-devtools/repo/repo.inc           | 25 ++++++++++++++
>>  meta/recipes-devtools/repo/repo_2.17.3.bb     |  7 ++++
>>  4 files changed, 86 insertions(+)
>>  create mode 100644
>> meta/recipes-devtools/repo/files/0001-python3-shebang.patch
>>  create mode 100644
>> meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
>>  create mode 100644 meta/recipes-devtools/repo/repo.inc
>>  create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
>>
>> diff --git a/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
>> b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
>> new file mode 100644
>> index 0000000000..09ccf58264
>> --- /dev/null
>> +++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
>> @@ -0,0 +1,21 @@
>> +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001
>> +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
>> +Date: Tue, 14 Sep 2021 16:46:51 +0200
>> +Subject: [PATCH] python3 shebang
>> +
>> +---
>> + repo | 2 +-
>> + 1 file changed, 1 insertion(+), 1 deletion(-)
>> +
>> +diff --git a/repo b/repo
>> +index b13e34c..205e0e5 100755
>> +--- a/repo
>> ++++ b/repo
>> +@@ -1,4 +1,4 @@
>> +-#!/usr/bin/env python
>> ++#!/usr/bin/env python3
>> + # -*- coding:utf-8 -*-
>> + #
>> + # Copyright (C) 2008 The Android Open Source Project
>> +--
>> +2.33.0
>> diff --git
>> a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
>> b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
>> new file mode 100644
>> index 0000000000..294a3af53a
>> --- /dev/null
>> +++
>> b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
>> @@ -0,0 +1,33 @@
>> +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
>> +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
>> +Date: Thu, 4 Nov 2021 16:55:12 +0100
>> +Subject: [PATCH] Set REPO_REV to v2.17.3
>> +
>> +repo is an unusual tool because it downloads all of its own Python
>> modules
>> +using GPG-signed git tags, and stores those files as part of the project
>> +that it is working with.
>> +
>> +So in order to have a reproducible repo installation within the project
>> +folders, we hardcode the REPO_REV variable to this recipes PV.
>> +
>> +Upstream-Status: Inappropriate [configuration]
>> +Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
>> +---
>> + repo | 2 +-
>> + 1 file changed, 1 insertion(+), 1 deletion(-)
>> +
>> +diff --git a/repo b/repo
>> +index 4cddbf1..cf5f6b1 100755
>> +--- a/repo
>> ++++ b/repo
>> +@@ -142,7 +142,7 @@ if __name__ == '__main__':
>> + REPO_URL = os.environ.get('REPO_URL', None)
>> + if not REPO_URL:
>> +   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
>> +-REPO_REV = os.environ.get('REPO_REV')
>> ++REPO_REV = 'v2.17.3'
>> + if not REPO_REV:
>> +   REPO_REV = 'stable'
>> + # URL to file bug reports for repo tool issues.
>> +--
>> +2.33.1
>> diff --git a/meta/recipes-devtools/repo/repo.inc
>> b/meta/recipes-devtools/repo/repo.inc
>> new file mode 100644
>> index 0000000000..60b32e4d74
>> --- /dev/null
>> +++ b/meta/recipes-devtools/repo/repo.inc
>> @@ -0,0 +1,25 @@
>> +# SPDX-License-Identifier: MIT
>> +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
>> +
>> +SUMMARY = "Tool for managing many Git repositories"
>> +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage
>> many Git repositories, does the uploads to revision control systems, and
>> automates parts of the development workflow."
>> +HOMEPAGE = "https://android.googlesource.com/tools/repo"
>> +SECTION = "console/utils"
>> +
>> +LICENSE = "Apache-2.0"
>> +
>> +SRC_URI = "git://
>> git@gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
>> +MIRRORS = "git://git@gerrit.googlesource.com/git-repo.git git://
>> github.com/GerritCodeReview/git-repo.git \n"
>> +
>> +SRC_URI += "file://0001-python3-shebang.patch"
>> +
>> +S = "${WORKDIR}/git"
>> +
>> +RDEPENDS_${PN} = "python3"
>> +
>> +do_install() {
>> +    install -d ${D}${bindir}
>> +    install -m 755 ${WORKDIR}/git/repo ${D}${bindir}
>> +}
>> +
>> +BBCLASSEXTEND = "native nativesdk"
>> diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
>> b/meta/recipes-devtools/repo/repo_2.17.3.bb
>> new file mode 100644
>> index 0000000000..c26264b9e9
>> --- /dev/null
>> +++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
>> @@ -0,0 +1,7 @@
>> +# SPDX-License-Identifier: MIT
>> +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
>> +
>> +require recipes-devtools/repo/repo.inc
>> +
>> +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
>> +LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
>> --
>> 2.33.1
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#12893):
>> https://lists.openembedded.org/g/bitbake-devel/message/12893
>> Mute This Topic: https://lists.openembedded.org/mt/86840271/1686489
>> Group Owner: bitbake-devel+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
>> alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>>

[-- Attachment #2: Type: text/html, Size: 8794 bytes --]

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Yeah. Unfortunately the current repo fetcher is pretty much broken,
this patch is more or less a requirement for changes to the repo
fetcher we proposed in the bitbake mailinglist.

In our case some binary which is made up from some 50ish separately
versioned sub-components. We used to have separate recipes for each of
this components and static link them using DEPENDS. However, this does
not scale well. E.g. if you want to create another version with
different cmake flags, you would have to create copies of all 50ish
recipes. You could move all the components into a single recipe and do
50x git fetcher, but keeping this versioned is still painful.
Especially since the developers mainly don't use yocto but use the SDK
for cross-compiling. As such this used to mean, that versioning in the
development workflow and the release workflow worked differently ->
more maintenance work.

With repo fetcher we can streamline the versioning within the yocto
build process and the daily developer workflow, as both can fetch the
repo manifest from a central stored git repo. The idea is, that the
repo manifest defaults to develop branch for all component repos and if
we want to create a release we can use repos own tooling to
automatically create fixed refspecs from the development manifest.
something like `repo manifest -r -o release-1.0.0.xml`, push that as a
release to the manifest repo, set this release as refspec in the recipe
within the meta layer and proceed with the release process. So no more
manually maintaining the component versioning within the yocto recipes.

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Fri, 2021-11-05 at 15:20 +0100, Alexander Kanavin wrote:
> Wait, what is the use case for the repo fetcher? 
> 
> Alex
> 
> On Fri, 5 Nov 2021 at 14:31, Jasper Orschulko via
> lists.openembedded.org <jasper=fancydomain.eu@lists.openembedded.org>
> wrote:
> > From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > 
> > Add a recipe for repo, prerequisite for the repo fetcher.
> > 
> > Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > ---
> >  .../repo/files/0001-python3-shebang.patch     | 21 ++++++++++++
> >  .../0001-Set-REPO_REV-to-v2.17.3.patch        | 33
> > +++++++++++++++++++
> >  meta/recipes-devtools/repo/repo.inc           | 25 ++++++++++++++
> >  meta/recipes-devtools/repo/repo_2.17.3.bb     |  7 ++++
> >  4 files changed, 86 insertions(+)
> >  create mode 100644 meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch
> >  create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-
> > Set-
> > REPO_REV-to-v2.17.3.patch
> >  create mode 100644 meta/recipes-devtools/repo/repo.inc
> >  create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
> > 
> > diff --git a/meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch b/meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch
> > new file mode 100644
> > index 0000000000..09ccf58264
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
> > @@ -0,0 +1,21 @@
> > +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > +Date: Tue, 14 Sep 2021 16:46:51 +0200
> > +Subject: [PATCH] python3 shebang
> > +
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index b13e34c..205e0e5 100755
> > +--- a/repo
> > ++++ b/repo
> > +@@ -1,4 +1,4 @@
> > +-#!/usr/bin/env python
> > ++#!/usr/bin/env python3
> > + # -*- coding:utf-8 -*-
> > + #
> > + # Copyright (C) 2008 The Android Open Source Project
> > +--
> > +2.33.0
> > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
> > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> > new file mode 100644
> > index 0000000000..294a3af53a
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-
> > v2.17.3.patch
> > @@ -0,0 +1,33 @@
> > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > +Date: Thu, 4 Nov 2021 16:55:12 +0100
> > +Subject: [PATCH] Set REPO_REV to v2.17.3
> > +
> > +repo is an unusual tool because it downloads all of its own Python
> > modules
> > +using GPG-signed git tags, and stores those files as part of the
> > project
> > +that it is working with.
> > +
> > +So in order to have a reproducible repo installation within the
> > project
> > +folders, we hardcode the REPO_REV variable to this recipes PV.
> > +
> > +Upstream-Status: Inappropriate [configuration]
> > +Signed-off-by: Jasper Orschulko
> > <Jasper.Orschulko@iris-sensing.com>
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index 4cddbf1..cf5f6b1 100755
> > +--- a/repo
> > ++++ b/repo
> > +@@ -142,7 +142,7 @@ if __name__ == '__main__':
> > + REPO_URL = os.environ.get('REPO_URL', None)
> > + if not REPO_URL:
> > +   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> > +-REPO_REV = os.environ.get('REPO_REV')
> > ++REPO_REV = 'v2.17.3'
> > + if not REPO_REV:
> > +   REPO_REV = 'stable'
> > + # URL to file bug reports for repo tool issues.
> > +--
> > +2.33.1
> > diff --git a/meta/recipes-devtools/repo/repo.inc b/meta/recipes-
> > devtools/repo/repo.inc
> > new file mode 100644
> > index 0000000000..60b32e4d74
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo.inc
> > @@ -0,0 +1,25 @@
> > +# SPDX-License-Identifier: MIT
> > +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> > +
> > +SUMMARY = "Tool for managing many Git repositories"
> > +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps
> > manage
> > many Git repositories, does the uploads to revision control
> > systems,
> > and automates parts of the development workflow."
> > +HOMEPAGE = "https://android.googlesource.com/tools/repo"
> > +SECTION = "console/utils"
> > +
> > +LICENSE = "Apache-2.0"
> > +
> > +SRC_URI =
> > "git://git@gerrit.googlesource.com/git-
> > repo.git;protocol=https;branch=main
> > "
> > +MIRRORS = "git://git@gerrit.googlesource.com/git-repo.git
> > git://github.com/GerritCodeReview/git-repo.git \n"
> > +
> > +SRC_URI += "file://0001-python3-shebang.patch"
> > +
> > +S = "${WORKDIR}/git"
> > +
> > +RDEPENDS_${PN} = "python3"
> > +
> > +do_install() {
> > +    install -d ${D}${bindir}
> > +    install -m 755 ${WORKDIR}/git/repo ${D}${bindir}
> > +}
> > +
> > +BBCLASSEXTEND = "native nativesdk"
> > diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
> > b/meta/recipes-devtools/repo/repo_2.17.3.bb
> > new file mode 100644
> > index 0000000000..c26264b9e9
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
> > @@ -0,0 +1,7 @@
> > +# SPDX-License-Identifier: MIT
> > +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> > +
> > +require recipes-devtools/repo/repo.inc
> > +
> > +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
> > +LIC_FILES_CHKSUM =
> > "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFTK4ACgkQYgqew07V
MNXGKwf8CbYnBqQhNvouhixH8rqaEjhFBAcbEvK1Pgke4B0zMuSJVz2YmTKPiHgK
dFznVtBxIJS7zwhuj/DyfhrG33Ws3Rw1m8IppZ5FP2XivNbmnimH13a7Pu5hw3FH
o1ka6VEqyuR/L4pyVZ+ocSgdEIyReSOJ6E1zyIomz13j5hScQIKFdAu4tinLNLck
XnC/SCDe3lZbijo41GK1v9GhKFjwY3Hpko6ezG/CEFPHKZomjHNL9Mi8gbv4Y/50
8+pYnvlYU7agsW4aN/koeRWOm78/Qi6rlOX5pVxAaVVVArrIKnU2aG6UBwk61BtC
UIqLBYOMyZ4IXlNp5y2tGbr6QS0YrQ==
=M87Y
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1842 bytes --]

On Fri, 5 Nov 2021 at 16:24, Jasper Orschulko <
Jasper.Orschulko@iris-sensing.com> wrote:

>
> In our case some binary which is made up from some 50ish separately
> versioned sub-components. We used to have separate recipes for each of
> this components and static link them using DEPENDS. However, this does
> not scale well. E.g. if you want to create another version with
> different cmake flags, you would have to create copies of all 50ish
> recipes. You could move all the components into a single recipe and do
> 50x git fetcher, but keeping this versioned is still painful.
> Especially since the developers mainly don't use yocto but use the SDK
> for cross-compiling. As such this used to mean, that versioning in the
> development workflow and the release workflow worked differently ->
> more maintenance work.
>
> With repo fetcher we can streamline the versioning within the yocto
> build process and the daily developer workflow, as both can fetch the
> repo manifest from a central stored git repo. The idea is, that the
> repo manifest defaults to develop branch for all component repos and if
> we want to create a release we can use repos own tooling to
> automatically create fixed refspecs from the development manifest.
> something like `repo manifest -r -o release-1.0.0.xml`, push that as a
> release to the manifest repo, set this release as refspec in the recipe
> within the meta layer and proceed with the release process. So no more
> manually maintaining the component versioning within the yocto recipes.
>

I see. I don't particularly endorse that you invented a custom, proprietary
workflow that you have to support entirely by yourselves - instead of
finding ways to do what you need with standard yocto tooling, improving the
tooling where/if necessary. But if that works out ok for you, then fine :)

Alex

[-- Attachment #2: Type: text/html, Size: 2291 bytes --]

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

[-- Attachment #1: Type: text/plain, Size: 2749 bytes --]

Hi Alex,

> that you invented a custom, proprietary 
> workflow that you have to support entirely by > yourselves

We are not though. We are integrating an established tool for multi-repository management into yocto. Google repo is not proprietary by the way, it is permissive licensed. It is just a little "special" in it's modus operandi.

> instead of finding ways to do what you need
> with standard yocto tooling, improving the
> tooling where/if necessary.

But that is exactly what we are doing, by integrating the repo fetcher into the yocto workflow, thus improving the yocto tooling :)
Why reinvent the wheel, when you can reuse whats already there? You wouldn't reinvent git just for yocto, would you?

Best regards,
Jasper


On 5 November 2021 18:46:27 CET, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>On Fri, 5 Nov 2021 at 16:24, Jasper Orschulko <
>Jasper.Orschulko@iris-sensing.com> wrote:
>
>>
>> In our case some binary which is made up from some 50ish separately
>> versioned sub-components. We used to have separate recipes for each of
>> this components and static link them using DEPENDS. However, this does
>> not scale well. E.g. if you want to create another version with
>> different cmake flags, you would have to create copies of all 50ish
>> recipes. You could move all the components into a single recipe and do
>> 50x git fetcher, but keeping this versioned is still painful.
>> Especially since the developers mainly don't use yocto but use the SDK
>> for cross-compiling. As such this used to mean, that versioning in the
>> development workflow and the release workflow worked differently ->
>> more maintenance work.
>>
>> With repo fetcher we can streamline the versioning within the yocto
>> build process and the daily developer workflow, as both can fetch the
>> repo manifest from a central stored git repo. The idea is, that the
>> repo manifest defaults to develop branch for all component repos and if
>> we want to create a release we can use repos own tooling to
>> automatically create fixed refspecs from the development manifest.
>> something like `repo manifest -r -o release-1.0.0.xml`, push that as a
>> release to the manifest repo, set this release as refspec in the recipe
>> within the meta layer and proceed with the release process. So no more
>> manually maintaining the component versioning within the yocto recipes.
>>
>
>I see. I don't particularly endorse that you invented a custom, proprietary
>workflow that you have to support entirely by yourselves - instead of
>finding ways to do what you need with standard yocto tooling, improving the
>tooling where/if necessary. But if that works out ok for you, then fine :)
>
>Alex

[-- Attachment #2: Type: text/html, Size: 3394 bytes --]

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 991 bytes --]

On Fri, 5 Nov 2021 at 19:05, Jasper Orschulko <jasper@fancydomain.eu> wrote:

>
> But that is exactly what we are doing, by integrating the repo fetcher
> into the yocto workflow, thus improving the yocto tooling :)
> Why reinvent the wheel, when you can reuse whats already there? You
> wouldn't reinvent git just for yocto, would you?
>

What I mean is that I would try harder to find a workable setup while
keeping the recipes for individual items that need to be built.
For instance, devtool is capable of updating source revisions in recipes
automatically, it may not work exactly as you want, but that can be fixed.
Yes, repo itself is not proprietary, but your organizational workflow is.

Particularly this:
"if you want to create another version with different cmake flags, you
would have to create copies of all 50ish recipes"
looks odd. Why would you make copies? Just set the needed flags with a
variable that is set from a global config,
perhaps the distro or local.conf.

Alex

[-- Attachment #2: Type: text/html, Size: 1601 bytes --]

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Alex,

we are getting a bit off topic here, but whatever... ;) 

> What I mean is that I would try harder to find a workable setup while
> keeping the recipes for individual items that need to be built.

the individual recipes don't really add that much benefit for us,
mostly just maintenance overhead and complexity, as we need to
constantly ensure that the various components depend on each other in
the right order. The only beneficial aspect to the multitude of recipes
is the sstate cache. However, as the packages heavily depend on one
another, the benefit is marginal, as we have to rebuild most of the
application anyways.

Additionally, this is not how the developers build the application, so
basically we currently have to maintain two separate build workflows.
Changing to repo also improves the developer workflow outside of yocto,
e.g. thanks to the notion of "topics". The "devtool" option, whilst
duable I assume, would again limit the usability to within the Yocto
environment. We are currently trying to get to a point, where we manage
most of our (automated) workloads (e.g. testing) within Yocto but
historically this was not the case and for daily development this is
just not really practical.

> Particularly this:
> "if you want to create another version with different cmake flags,
> you would have to create copies of all 50ish recipes"
> looks odd. Why would you make copies? Just set the needed flags with
> a variable that is set from a global config,
> perhaps the distro or local.conf.

As far as I understand, this is not reasonably possible (though I might
be wrong. TBH, understanding variable scopes in Yocto is a nightmare).

To give this some context, our current base setup looks as follows:

- - We have one distro.conf
- - We have three image recipes with slightly different config +
featureset
- - AFAIK an image recipe cannot set variables within another recipe, it
just "consumes" existing recipes with their configuration.
- -> Thus we have three respective recipes with slightly different
configuration. If we where to use individual recipes in this setup,
we'd have 50ish x 3 recipes.

The approach you are describing might circumvent this particular
situation but basically just moves the issue around, as you're now
unabe to build all three configurations in one go, but need to start
juggling your local.confs. Alternatively you might think about using
multi-confs. However, each multiconf doubles the parse times for your
recipes. As we already use a couple of multiconfs for different machine
configs, we would basically increase the time it takes to parse the
recipes by ~fifteen-fold (5 pre-existing multi-confs x 3 (or more)
disto confs). Additionally, you now need to start worrying about
artifact names, as by default the distro name is not included.

So yeah... as far as I can tell, there are multiple ways to approach
this issue, but none of them seem straight forward nor pretty. Bitbake
as it is is just fundamentally not good at handling highly dynamic
configurations. The combination with KAS somewhat defuses the
situation, but there are still some situations where there is no easy
answer.

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Fri, 2021-11-05 at 19:45 +0100, Alexander Kanavin wrote:
> On Fri, 5 Nov 2021 at 19:05, Jasper Orschulko <jasper@fancydomain.eu>
> wrote:
> > 
> > But that is exactly what we are doing, by integrating the repo
> > fetcher into the yocto workflow, thus improving the yocto tooling
> > :)
> > Why reinvent the wheel, when you can reuse whats already there? You
> > wouldn't reinvent git just for yocto, would you?
> > 
> 
> 
> What I mean is that I would try harder to find a workable setup while
> keeping the recipes for individual items that need to be built. 
> For instance, devtool is capable of updating source revisions in
> recipes automatically, it may not work exactly as you want, but that
> can be fixed.
> Yes, repo itself is not proprietary, but your organizational workflow
> is.
> 
> Particularly this:
> "if you want to create another version with different cmake flags,
> you would have to create copies of all 50ish recipes"
> looks odd. Why would you make copies? Just set the needed flags with
> a variable that is set from a global config,
> perhaps the distro or local.conf.
> 
> Alex
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFlPMACgkQYgqew07V
MNXoCAf/fN4gi1dfx7r4acfbu7kaw9+3rjdMu34J6SX/ahGjCfbTZAg1Twf8N6RZ
LBuOpDkXPU77iJuJixZySve5EZc4A2/NS0hqXEpD0aWf8AXaZzxq0UbEBM78oTgH
Hk1zVEFxwLwEHpVIQyvwx5rFz/JvZ2EN4aH576ciPaw5n+bHIZBevUfhssPrjQWX
yUTuWyeUoMJlXz41E+JfSd1VIQOC3hdWKF26er152zSyhYvhCagBMsHZMTKdrahn
2jmInGMSMizXR6vXBUTfwaB48Ba0iqFoi/tr7+jNIHV5WZNx4AvU61UOu8Lca6he
bGvM5nM7xRpeHuNX2xa9xWs3SkUNEg==
=UJaE
-----END PGP SIGNATURE-----

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1170 bytes --]

On Fri, 5 Nov 2021 at 21:32, Jasper Orschulko <
Jasper.Orschulko@iris-sensing.com> wrote:

> So yeah... as far as I can tell, there are multiple ways to approach
> this issue, but none of them seem straight forward nor pretty. Bitbake
> as it is is just fundamentally not good at handling highly dynamic
> configurations. The combination with KAS somewhat defuses the
> situation, but there are still some situations where there is no easy
> answer.
>

Yes, yocto does not support 'variations' of the build that is already set
up. Once you set the distro, and the machine, there's only one way to build
a recipe; if you want another way, you need to indeed make another recipe,
or another distro. I would still say making a set of distros is still the
proper 'yocto way' to solve your problem. There are examples in meta-poky:
poky-altcfg and poky-tiny, and there has been poky-bleeding-revisions or
similar in the past.

This is a reoccurring topic; there have been proposal in the past to
introduce 'variants' as yet another abstraction, but they add so much
complexity on top of what is already very complex, that the tradeoff is not
acceptable to the project.

Alex

[-- Attachment #2: Type: text/html, Size: 1649 bytes --]

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Richard Purdie]

On Fri, 2021-11-05 at 14:31 +0100, Jasper Orschulko via lists.openembedded.org
wrote:
> From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> 
> Add a recipe for repo, prerequisite for the repo fetcher.
> 
> Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> ---
>  .../repo/files/0001-python3-shebang.patch     | 21 ++++++++++++
>  .../0001-Set-REPO_REV-to-v2.17.3.patch        | 33 +++++++++++++++++++
>  meta/recipes-devtools/repo/repo.inc           | 25 ++++++++++++++
>  meta/recipes-devtools/repo/repo_2.17.3.bb     |  7 ++++
>  4 files changed, 86 insertions(+)
>  create mode 100644 meta/recipes-devtools/repo/files/0001-python3-shebang.patch
>  create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
>  create mode 100644 meta/recipes-devtools/repo/repo.inc
>  create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb

This basically looks ok to me, I've some minor comments below.

The patch is missing adding an entry to:

meta/conf/distro/include/maintainers.inc

which is required for OE-Core recipes and will trip up automated testing if we
don't.

> 
> diff --git a/meta/recipes-devtools/repo/files/0001-python3-shebang.patch b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
> new file mode 100644
> index 0000000000..09ccf58264
> --- /dev/null
> +++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch


I'd put this in a folder called "repo" rather than files.

> @@ -0,0 +1,21 @@
> +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001
> +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> +Date: Tue, 14 Sep 2021 16:46:51 +0200
> +Subject: [PATCH] python3 shebang
> +
> +---
> + repo | 2 +-
> + 1 file changed, 1 insertion(+), 1 deletion(-)
> 

Missing Upstream-Status: (you got the second one ok though! :)

> +diff --git a/repo b/repo
> +index b13e34c..205e0e5 100755
> +--- a/repo
> ++++ b/repo
> +@@ -1,4 +1,4 @@
> +-#!/usr/bin/env python
> ++#!/usr/bin/env python3
> + # -*- coding:utf-8 -*-
> + #
> + # Copyright (C) 2008 The Android Open Source Project
> +--
> +2.33.0
> diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> new file mode 100644
> index 0000000000..294a3af53a
> --- /dev/null
> +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> @@ -0,0 +1,33 @@
> +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001
> +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> +Date: Thu, 4 Nov 2021 16:55:12 +0100
> +Subject: [PATCH] Set REPO_REV to v2.17.3
> +
> +repo is an unusual tool because it downloads all of its own Python modules
> +using GPG-signed git tags, and stores those files as part of the project
> +that it is working with.
> +
> +So in order to have a reproducible repo installation within the project
> +folders, we hardcode the REPO_REV variable to this recipes PV.
> +
> +Upstream-Status: Inappropriate [configuration]
> +Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> +---
> + repo | 2 +-
> + 1 file changed, 1 insertion(+), 1 deletion(-)
> +
> +diff --git a/repo b/repo
> +index 4cddbf1..cf5f6b1 100755
> +--- a/repo
> ++++ b/repo
> +@@ -142,7 +142,7 @@ if __name__ == '__main__':
> + REPO_URL = os.environ.get('REPO_URL', None)
> + if not REPO_URL:
> +   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> +-REPO_REV = os.environ.get('REPO_REV')
> ++REPO_REV = 'v2.17.3'
> + if not REPO_REV:
> +   REPO_REV = 'stable'
> + # URL to file bug reports for repo tool issues.
> +--
> +2.33.1
> diff --git a/meta/recipes-devtools/repo/repo.inc b/meta/recipes-devtools/repo/repo.inc
> new file mode 100644
> index 0000000000..60b32e4d74
> --- /dev/null
> +++ b/meta/recipes-devtools/repo/repo.inc
> @@ -0,0 +1,25 @@
> +# SPDX-License-Identifier: MIT
> +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> +
> +SUMMARY = "Tool for managing many Git repositories"
> +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git repositories, does the uploads to revision control systems, and automates parts of the development workflow."
> +HOMEPAGE = "https://android.googlesource.com/tools/repo"
> +SECTION = "console/utils"
> +
> +LICENSE = "Apache-2.0"
> +
> +SRC_URI = "git://git@gerrit.googlesource.com/git-repo.git;protocol=https;branch=main"
> +MIRRORS = "git://git@gerrit.googlesource.com/git-repo.git git://github.com/GerritCodeReview/git-repo.git \n"

You shouldn't be clearing MIRRORS with "=" as the user may have something set in
there the system needs to work, you probably want += here.

> +
> +SRC_URI += "file://0001-python3-shebang.patch"
> +
> +S = "${WORKDIR}/git"
> +
> +RDEPENDS_${PN} = "python3"

This is old overrides syntax, it is RDEPENDS:${PN} now as others have mentioned.

> +
> +do_install() {
> +    install -d ${D}${bindir}
> +    install -m 755 ${WORKDIR}/git/repo ${D}${bindir}
> +}
> +
> +BBCLASSEXTEND = "native nativesdk"
> diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb b/meta/recipes-devtools/repo/repo_2.17.3.bb
> new file mode 100644
> index 0000000000..c26264b9e9
> --- /dev/null
> +++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
> @@ -0,0 +1,7 @@
> +# SPDX-License-Identifier: MIT
> +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> +
> +require recipes-devtools/repo/repo.inc
> +
> +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
> +LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"

Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Richard and Peter,

I've addressed your comments and will post the v3 patches momentarily.

Quick comment regarding the new overrides syntax, or rather the
documentation. I just noticed that my go-to page for the mega-manual is
outdated
(https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html)
and that the new location now seems to be
https://docs.yoctoproject.org/3.4/singleindex.html.

There is a redirect in place, e.g. when trying to call a 3.4 version of
the mega manual like this
https://www.yoctoproject.org/docs/3.4/mega-manual/mega-manual.html.
However, this does not extend to the "latest" page, which is just a
source for confusion. Is there any chance we could change that?

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Sun, 2021-11-07 at 09:05 +0000, Richard Purdie wrote:
> On Fri, 2021-11-05 at 14:31 +0100, Jasper Orschulko via
> lists.openembedded.org
> wrote:
> > From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > 
> > Add a recipe for repo, prerequisite for the repo fetcher.
> > 
> > Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > ---
> >  .../repo/files/0001-python3-shebang.patch     | 21 ++++++++++++
> >  .../0001-Set-REPO_REV-to-v2.17.3.patch        | 33
> > +++++++++++++++++++
> >  meta/recipes-devtools/repo/repo.inc           | 25 ++++++++++++++
> >  meta/recipes-devtools/repo/repo_2.17.3.bb     |  7 ++++
> >  4 files changed, 86 insertions(+)
> >  create mode 100644 meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch
> >  create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch
> >  create mode 100644 meta/recipes-devtools/repo/repo.inc
> >  create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
> 
> This basically looks ok to me, I've some minor comments below.
> 
> The patch is missing adding an entry to:
> 
> meta/conf/distro/include/maintainers.inc
> 
> which is required for OE-Core recipes and will trip up automated
> testing if we
> don't.
> 
> > 
> > diff --git a/meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch b/meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch
> > new file mode 100644
> > index 0000000000..09ccf58264
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
> 
> 
> I'd put this in a folder called "repo" rather than files.
> 
> > @@ -0,0 +1,21 @@
> > +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > +Date: Tue, 14 Sep 2021 16:46:51 +0200
> > +Subject: [PATCH] python3 shebang
> > +
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> 
> Missing Upstream-Status: (you got the second one ok though! :)
> 
> > +diff --git a/repo b/repo
> > +index b13e34c..205e0e5 100755
> > +--- a/repo
> > ++++ b/repo
> > +@@ -1,4 +1,4 @@
> > +-#!/usr/bin/env python
> > ++#!/usr/bin/env python3
> > + # -*- coding:utf-8 -*-
> > + #
> > + # Copyright (C) 2008 The Android Open Source Project
> > +--
> > +2.33.0
> > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
> > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> > new file mode 100644
> > index 0000000000..294a3af53a
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-
> > v2.17.3.patch
> > @@ -0,0 +1,33 @@
> > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > +Date: Thu, 4 Nov 2021 16:55:12 +0100
> > +Subject: [PATCH] Set REPO_REV to v2.17.3
> > +
> > +repo is an unusual tool because it downloads all of its own Python
> > modules
> > +using GPG-signed git tags, and stores those files as part of the
> > project
> > +that it is working with.
> > +
> > +So in order to have a reproducible repo installation within the
> > project
> > +folders, we hardcode the REPO_REV variable to this recipes PV.
> > +
> > +Upstream-Status: Inappropriate [configuration]
> > +Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index 4cddbf1..cf5f6b1 100755
> > +--- a/repo
> > ++++ b/repo
> > +@@ -142,7 +142,7 @@ if __name__ == '__main__':
> > + REPO_URL = os.environ.get('REPO_URL', None)
> > + if not REPO_URL:
> > +   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> > +-REPO_REV = os.environ.get('REPO_REV')
> > ++REPO_REV = 'v2.17.3'
> > + if not REPO_REV:
> > +   REPO_REV = 'stable'
> > + # URL to file bug reports for repo tool issues.
> > +--
> > +2.33.1
> > diff --git a/meta/recipes-devtools/repo/repo.inc b/meta/recipes-
> > devtools/repo/repo.inc
> > new file mode 100644
> > index 0000000000..60b32e4d74
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo.inc
> > @@ -0,0 +1,25 @@
> > +# SPDX-License-Identifier: MIT
> > +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> > +
> > +SUMMARY = "Tool for managing many Git repositories"
> > +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage
> > many Git repositories, does the uploads to revision control systems,
> > and automates parts of the development workflow."
> > +HOMEPAGE = "https://android.googlesource.com/tools/repo"
> > +SECTION = "console/utils"
> > +
> > +LICENSE = "Apache-2.0"
> > +
> > +SRC_URI = "git://git@gerrit.googlesource.com/git-
> > repo.git;protocol=https;branch=main"
> > +MIRRORS = "git://git@gerrit.googlesource.com/git-repo.git
> > git://github.com/GerritCodeReview/git-repo.git \n"
> 
> You shouldn't be clearing MIRRORS with "=" as the user may have
> something set in
> there the system needs to work, you probably want += here.
> 
> > +
> > +SRC_URI += "file://0001-python3-shebang.patch"
> > +
> > +S = "${WORKDIR}/git"
> > +
> > +RDEPENDS_${PN} = "python3"
> 
> This is old overrides syntax, it is RDEPENDS:${PN} now as others have
> mentioned.
> 
> > +
> > +do_install() {
> > +    install -d ${D}${bindir}
> > +    install -m 755 ${WORKDIR}/git/repo ${D}${bindir}
> > +}
> > +
> > +BBCLASSEXTEND = "native nativesdk"
> > diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
> > b/meta/recipes-devtools/repo/repo_2.17.3.bb
> > new file mode 100644
> > index 0000000000..c26264b9e9
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
> > @@ -0,0 +1,7 @@
> > +# SPDX-License-Identifier: MIT
> > +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> > +
> > +require recipes-devtools/repo/repo.inc
> > +
> > +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
> > +LIC_FILES_CHKSUM =
> > "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
> 
> 
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGJEEsACgkQYgqew07V
MNUO0Qf/Rxt9hZHWTmDX6eGjEEbMxxSGyJ2You7H421dZZH9WMQxnXVHsl2n2C6B
2gzMqyRZJdnR/60tfPr97wuQbeyxlHjsxaYa5F8p23bYAyGVYaSm/uDXmLnqDmXQ
qDmliY1XYkh+UhP/FkzmjCyCrEMPul1CLEZemY++RQZhBj5ge8W0G2JMsm7GAoa1
1dRghiclvvi5aHB2caSPsz9RlDHsSAdMZ78LheULVxAXBy4tE4h7p6JFMrK/pUy5
1wCLLQGSeqqlFbmRs3W3FKavlrPIvw80lq5DaxNJauMhhKGTPMKp6yu6V+Lr0BmX
LkduXEmbmJRQ5oKxA5FFFzwXCiUe3g==
=48qv
-----END PGP SIGNATURE-----

Fwd: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Richard Purdie]

[-- Attachment #1: Type: text/plain, Size: 99 bytes --]

Hi Michael and Michael,

This sounds like something we should prioritise fixing?

Cheers,

Richard

[-- Attachment #2: Forwarded message — Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3 --]
[-- Type: message/rfc822, Size: 14893 bytes --]

From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
To: "richard.purdie@linuxfoundation.org" <richard.purdie@linuxfoundation.org>, "openembedded-core@lists.openembedded.org" <openembedded-core@lists.openembedded.org>, "peter.kjellerstedt@axis.com" <peter.kjellerstedt@axis.com>, "jasper@fancydomain.eu" <jasper@fancydomain.eu>
Cc: "martin@mko.dev" <martin@mko.dev>, "Daniel.Baumgart@iris-sensing.net" <Daniel.Baumgart@iris-sensing.net>, "bitbake-devel@lists.openembedded.org" <bitbake-devel@lists.openembedded.org>
Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3
Date: Mon, 8 Nov 2021 11:55:56 +0000
Message-ID: <a82b048c5f9f164696b1dd1e3e6c4997ede4f5ad.camel@iris-sensing.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Richard and Peter,

I've addressed your comments and will post the v3 patches momentarily.

Quick comment regarding the new overrides syntax, or rather the
documentation. I just noticed that my go-to page for the mega-manual is
outdated
(https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html)
and that the new location now seems to be
https://docs.yoctoproject.org/3.4/singleindex.html.

There is a redirect in place, e.g. when trying to call a 3.4 version of
the mega manual like this
https://www.yoctoproject.org/docs/3.4/mega-manual/mega-manual.html.
However, this does not extend to the "latest" page, which is just a
source for confusion. Is there any chance we could change that?

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Sun, 2021-11-07 at 09:05 +0000, Richard Purdie wrote:
> On Fri, 2021-11-05 at 14:31 +0100, Jasper Orschulko via
> lists.openembedded.org
> wrote:
> > From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > 
> > Add a recipe for repo, prerequisite for the repo fetcher.
> > 
> > Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > ---
> >  .../repo/files/0001-python3-shebang.patch     | 21 ++++++++++++
> >  .../0001-Set-REPO_REV-to-v2.17.3.patch        | 33
> > +++++++++++++++++++
> >  meta/recipes-devtools/repo/repo.inc           | 25 ++++++++++++++
> >  meta/recipes-devtools/repo/repo_2.17.3.bb     |  7 ++++
> >  4 files changed, 86 insertions(+)
> >  create mode 100644 meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch
> >  create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch
> >  create mode 100644 meta/recipes-devtools/repo/repo.inc
> >  create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
> 
> This basically looks ok to me, I've some minor comments below.
> 
> The patch is missing adding an entry to:
> 
> meta/conf/distro/include/maintainers.inc
> 
> which is required for OE-Core recipes and will trip up automated
> testing if we
> don't.
> 
> > 
> > diff --git a/meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch b/meta/recipes-devtools/repo/files/0001-python3-
> > shebang.patch
> > new file mode 100644
> > index 0000000000..09ccf58264
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/files/0001-python3-shebang.patch
> 
> 
> I'd put this in a folder called "repo" rather than files.
> 
> > @@ -0,0 +1,21 @@
> > +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > +Date: Tue, 14 Sep 2021 16:46:51 +0200
> > +Subject: [PATCH] python3 shebang
> > +
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> 
> Missing Upstream-Status: (you got the second one ok though! :)
> 
> > +diff --git a/repo b/repo
> > +index b13e34c..205e0e5 100755
> > +--- a/repo
> > ++++ b/repo
> > +@@ -1,4 +1,4 @@
> > +-#!/usr/bin/env python
> > ++#!/usr/bin/env python3
> > + # -*- coding:utf-8 -*-
> > + #
> > + # Copyright (C) 2008 The Android Open Source Project
> > +--
> > +2.33.0
> > diff --git a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-
> > REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
> > 2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
> > new file mode 100644
> > index 0000000000..294a3af53a
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-
> > v2.17.3.patch
> > @@ -0,0 +1,33 @@
> > +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00
> > 2001
> > +From: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > +Date: Thu, 4 Nov 2021 16:55:12 +0100
> > +Subject: [PATCH] Set REPO_REV to v2.17.3
> > +
> > +repo is an unusual tool because it downloads all of its own Python
> > modules
> > +using GPG-signed git tags, and stores those files as part of the
> > project
> > +that it is working with.
> > +
> > +So in order to have a reproducible repo installation within the
> > project
> > +folders, we hardcode the REPO_REV variable to this recipes PV.
> > +
> > +Upstream-Status: Inappropriate [configuration]
> > +Signed-off-by: Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>
> > +---
> > + repo | 2 +-
> > + 1 file changed, 1 insertion(+), 1 deletion(-)
> > +
> > +diff --git a/repo b/repo
> > +index 4cddbf1..cf5f6b1 100755
> > +--- a/repo
> > ++++ b/repo
> > +@@ -142,7 +142,7 @@ if __name__ == '__main__':
> > + REPO_URL = os.environ.get('REPO_URL', None)
> > + if not REPO_URL:
> > +   REPO_URL = 'https://gerrit.googlesource.com/git-repo'
> > +-REPO_REV = os.environ.get('REPO_REV')
> > ++REPO_REV = 'v2.17.3'
> > + if not REPO_REV:
> > +   REPO_REV = 'stable'
> > + # URL to file bug reports for repo tool issues.
> > +--
> > +2.33.1
> > diff --git a/meta/recipes-devtools/repo/repo.inc b/meta/recipes-
> > devtools/repo/repo.inc
> > new file mode 100644
> > index 0000000000..60b32e4d74
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo.inc
> > @@ -0,0 +1,25 @@
> > +# SPDX-License-Identifier: MIT
> > +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> > +
> > +SUMMARY = "Tool for managing many Git repositories"
> > +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage
> > many Git repositories, does the uploads to revision control systems,
> > and automates parts of the development workflow."
> > +HOMEPAGE = "https://android.googlesource.com/tools/repo"
> > +SECTION = "console/utils"
> > +
> > +LICENSE = "Apache-2.0"
> > +
> > +SRC_URI = "git://git@gerrit.googlesource.com/git-
> > repo.git;protocol=https;branch=main"
> > +MIRRORS = "git://git@gerrit.googlesource.com/git-repo.git
> > git://github.com/GerritCodeReview/git-repo.git \n"
> 
> You shouldn't be clearing MIRRORS with "=" as the user may have
> something set in
> there the system needs to work, you probably want += here.
> 
> > +
> > +SRC_URI += "file://0001-python3-shebang.patch"
> > +
> > +S = "${WORKDIR}/git"
> > +
> > +RDEPENDS_${PN} = "python3"
> 
> This is old overrides syntax, it is RDEPENDS:${PN} now as others have
> mentioned.
> 
> > +
> > +do_install() {
> > +    install -d ${D}${bindir}
> > +    install -m 755 ${WORKDIR}/git/repo ${D}${bindir}
> > +}
> > +
> > +BBCLASSEXTEND = "native nativesdk"
> > diff --git a/meta/recipes-devtools/repo/repo_2.17.3.bb
> > b/meta/recipes-devtools/repo/repo_2.17.3.bb
> > new file mode 100644
> > index 0000000000..c26264b9e9
> > --- /dev/null
> > +++ b/meta/recipes-devtools/repo/repo_2.17.3.bb
> > @@ -0,0 +1,7 @@
> > +# SPDX-License-Identifier: MIT
> > +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
> > +
> > +require recipes-devtools/repo/repo.inc
> > +
> > +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc"
> > +LIC_FILES_CHKSUM =
> > "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
> 
> 
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGJEEsACgkQYgqew07V
MNUO0Qf/Rxt9hZHWTmDX6eGjEEbMxxSGyJ2You7H421dZZH9WMQxnXVHsl2n2C6B
2gzMqyRZJdnR/60tfPr97wuQbeyxlHjsxaYa5F8p23bYAyGVYaSm/uDXmLnqDmXQ
qDmliY1XYkh+UhP/FkzmjCyCrEMPul1CLEZemY++RQZhBj5ge8W0G2JMsm7GAoa1
1dRghiclvvi5aHB2caSPsz9RlDHsSAdMZ78LheULVxAXBy4tE4h7p6JFMrK/pUy5
1wCLLQGSeqqlFbmRs3W3FKavlrPIvw80lq5DaxNJauMhhKGTPMKp6yu6V+Lr0BmX
LkduXEmbmJRQ5oKxA5FFFzwXCiUe3g==
=48qv
-----END PGP SIGNATURE-----

Re: [docs] [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Quentin Schulz]

On Mon, Nov 08, 2021 at 12:48:08PM +0000, Richard Purdie wrote:
> Hi Michael and Michael,
> 
> This sounds like something we should prioritise fixing?
> 

I agree.

Hijacking for a different (though related) topic:

I think we also need to add a banner stating that this doc is
out-of-date. But I just don't have the skills to do it quickly and still
have something nice enough.

Considering how long the pages in the documentation are, I was trying
to implement a banner that would stick to the top of the view (stays
visible even when scrolling or linking to somewhere in the middle of
the page).

And the same to do for the current docs.yoctoproject.org (though there's
already a header, it;s not at the top of the view).

If anyone has the skills to do it quickly, please do so :) otherwise, I
guess I'll come to it the day I can't stand it anymore :)

Cheers,
Quentin

Re: Fwd: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Michael Opdenacker]

Richard, Jasper,

On 11/8/21 1:48 PM, Richard Purdie wrote:
> Hi Michael and Michael,
>
> This sounds like something we should prioritise fixing?

Good catch, thanks!

I agree,
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html
should also redirect to https://docs.yoctoproject.org/singleindex.html

I'll ask Michael H. to change this.

Cheers
Michael

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: Fwd: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Michael Opdenacker]

On 11/9/21 11:40 AM, Michael Opdenacker wrote:
> Richard, Jasper,
>
> On 11/8/21 1:48 PM, Richard Purdie wrote:
>> Hi Michael and Michael,
>>
>> This sounds like something we should prioritise fixing?
> Good catch, thanks!
>
> I agree,
> https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html
> should also redirect to https://docs.yoctoproject.org/singleindex.html
>
> I'll ask Michael H. to change this.

Michael H. took care of this.
Thanks again
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: Fwd: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Richard Purdie]

On Wed, 2021-11-10 at 09:47 +0100, Michael Opdenacker wrote:
> On 11/9/21 11:40 AM, Michael Opdenacker wrote:
> > Richard, Jasper,
> > 
> > On 11/8/21 1:48 PM, Richard Purdie wrote:
> > > Hi Michael and Michael,
> > > 
> > > This sounds like something we should prioritise fixing?
> > Good catch, thanks!
> > 
> > I agree,
> > https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html
> > should also redirect to https://docs.yoctoproject.org/singleindex.html
> > 
> > I'll ask Michael H. to change this.
> 
> Michael H. took care of this.

Thanks all, that should be a useful fix.

Cheers,

Richard