[OE-core][PATCH 0/4] Import recipes from meta-python

[OE-core][PATCH 0/4] Import recipes from meta-python

[Joshua Watt]

Imports and upgrades pycryptodome, pycryptodomex, and pyelftools from
meta-python.  These recipes are commonly used by other layers (e.g.
meta-arm) so putting them in oe-core reduces layer dependencies.

Joshua Watt (4):
  pycryptodome: Import from meta-python
  pyelftools: Import from meta-python
  python3-pycryptodome(x): Upgrade 3.9.4 -> 3.9.7
  python3-pyelftools: Upgrade 0.25 -> 0.26

 meta/conf/distro/include/maintainers.inc      |  5 +++-
 .../python/python-pycryptodome.inc            | 26 +++++++++++++++++++
 .../python/python3-pycryptodome_3.9.7.bb      |  5 ++++
 .../python/python3-pycryptodomex_3.9.7.bb     |  9 +++++++
 .../python/python3-pyelftools_0.26.bb         | 14 ++++++++++
 5 files changed, 58 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-devtools/python/python-pycryptodome.inc
 create mode 100644 meta/recipes-devtools/python/python3-pycryptodome_3.9.7.bb
 create mode 100644 meta/recipes-devtools/python/python3-pycryptodomex_3.9.7.bb
 create mode 100644 meta/recipes-devtools/python/python3-pyelftools_0.26.bb

-- 
2.17.1

[OE-core][PATCH 1/4] pycryptodome: Import from meta-python

[Joshua Watt]

Imports the pycryptodome recipes from meta-python, as of 7c02c7d41
("gnome-themes-extra: correct the recipe name").

These recipes are commonly used by other layers, so moving them into
OE-core helps to cut down on layer dependencies.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/conf/distro/include/maintainers.inc      |  4 ++-
 .../python/python-pycryptodome.inc            | 29 +++++++++++++++++++
 .../python/python3-pycryptodome_3.9.4.bb      |  2 ++
 .../python/python3-pycryptodomex_3.9.4.bb     | 10 +++++++
 4 files changed, 44 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-devtools/python/python-pycryptodome.inc
 create mode 100644 meta/recipes-devtools/python/python3-pycryptodome_3.9.4.bb
 create mode 100644 meta/recipes-devtools/python/python3-pycryptodomex_3.9.4.bb

diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
index 256666c0ec..b87790e0e9 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -2,7 +2,7 @@
 #
 # This file contains a list of recipe maintainers.
 #
-# Please submit any patches against recipes in meta to the 
+# Please submit any patches against recipes in meta to the
 # OE-Core mail list (openembedded-core@lists.openembedded.org)
 # For recipes in meta-yocto please use the Poky list (poky@yoctoproject.org)
 #
@@ -579,6 +579,8 @@ RECIPE_MAINTAINER_pn-python3-cython = "Oleksandr Kravchuk <open.source@oleksandr
 RECIPE_MAINTAINER_pn-python3-dbus = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-dbusmock = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-docutils = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
+RECIPE_MAINTAINER_pn-python3-pycryptodome = "Joshua Watt <JPEWhacker@gmail.com>"
+RECIPE_MAINTAINER_pn-python3-pycryptodomex = "Joshua Watt <JPEWhacker@gmail.com>"
 RECIPE_MAINTAINER_pn-python3-extras = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-git = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-gitdb = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
diff --git a/meta/recipes-devtools/python/python-pycryptodome.inc b/meta/recipes-devtools/python/python-pycryptodome.inc
new file mode 100644
index 0000000000..63b4a4abb4
--- /dev/null
+++ b/meta/recipes-devtools/python/python-pycryptodome.inc
@@ -0,0 +1,29 @@
+SUMMARY = "Cryptographic library for Python"
+DESCRIPTION = "PyCryptodome is a self-contained Python package of low-level\
+ cryptographic primitives."
+HOMEPAGE = "http://www.pycryptodome.org"
+LICENSE = "PD & BSD-2-Clause"
+LIC_FILES_CHKSUM = "file://LICENSE.rst;md5=6dc0e2a13d2f25d6f123c434b761faba"
+
+SRC_URI[md5sum] = "f990716b49add7b14ea8b8a961fb3746"
+SRC_URI[sha256sum] = "a168e73879619b467072509a223282a02c8047d932a48b74fbd498f27224aa04"
+
+inherit pypi
+
+RDEPENDS_${PN} += " \
+    ${PYTHON_PN}-io \
+    ${PYTHON_PN}-math \
+"
+
+RDEPENDS_${PN}-tests += " \
+    ${PYTHON_PN}-unittest \
+"
+
+PACKAGES =+ "${PN}-tests"
+
+FILES_${PN}-tests = " \
+    ${PYTHON_SITEPACKAGES_DIR}/Crypto/SelfTest/ \
+    ${PYTHON_SITEPACKAGES_DIR}/Crypto/SelfTest/__pycache__/ \
+"
+
+BBCLASSEXTEND = "native nativesdk"
diff --git a/meta/recipes-devtools/python/python3-pycryptodome_3.9.4.bb b/meta/recipes-devtools/python/python3-pycryptodome_3.9.4.bb
new file mode 100644
index 0000000000..0c062dddf8
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-pycryptodome_3.9.4.bb
@@ -0,0 +1,2 @@
+require python-pycryptodome.inc
+inherit setuptools3
diff --git a/meta/recipes-devtools/python/python3-pycryptodomex_3.9.4.bb b/meta/recipes-devtools/python/python3-pycryptodomex_3.9.4.bb
new file mode 100644
index 0000000000..e41c14b142
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-pycryptodomex_3.9.4.bb
@@ -0,0 +1,10 @@
+require python-pycryptodome.inc
+inherit setuptools3
+
+SRC_URI[md5sum] = "46ba513d95b6e323734074d960a7d57b"
+SRC_URI[sha256sum] = "22d970cee5c096b9123415e183ae03702b2cd4d3ba3f0ced25c4e1aba3967167"
+
+FILES_${PN}-tests = " \
+    ${PYTHON_SITEPACKAGES_DIR}/Cryptodome/SelfTest/ \
+    ${PYTHON_SITEPACKAGES_DIR}/Cryptodome/SelfTest/__pycache__/ \
+"
-- 
2.17.1

[OE-core][PATCH 2/4] pyelftools: Import from meta-python

[Joshua Watt]

Imports the pyelftools recipes from meta-python, as of 7c02c7d41
("gnome-themes-extra: correct the recipe name").

This recipe is commonly used by other layers, so moving it into
OE-core helps to cut down on layer dependencies.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/conf/distro/include/maintainers.inc           |  1 +
 .../python/python3-pyelftools_0.25.bb              | 14 ++++++++++++++
 2 files changed, 15 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-pyelftools_0.25.bb

diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
index b87790e0e9..51e6da669c 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -593,6 +593,7 @@ RECIPE_MAINTAINER_pn-python3-numpy = "Oleksandr Kravchuk <open.source@oleksandr-
 RECIPE_MAINTAINER_pn-python3-pbr = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-pip = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-pycairo = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
+RECIPE_MAINTAINER_pn-python3-pyelftools = "Joshua Watt <JPEWhacker@gmail.com>"
 RECIPE_MAINTAINER_pn-python3-pygments = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-pygobject = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
 RECIPE_MAINTAINER_pn-python3-pyparsing = "Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>"
diff --git a/meta/recipes-devtools/python/python3-pyelftools_0.25.bb b/meta/recipes-devtools/python/python3-pyelftools_0.25.bb
new file mode 100644
index 0000000000..03d96db3c2
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-pyelftools_0.25.bb
@@ -0,0 +1,14 @@
+DESCRIPTION = "pyelftools is a pure-Python library for parsing and analyzing ELF files and DWARF debugging information"
+HOMEPAGE = "https://github.com/eliben/pyelftools"
+SECTION = "devel/python"
+LICENSE = "PD"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=5ce2a2b07fca326bc7c146d10105ccfc"
+
+SRC_URI[md5sum] = "c5629b9a5d19c82107a946cce52eeec2"
+SRC_URI[sha256sum] = "89c6da6f56280c37a5ff33468591ba9a124e17d71fe42de971818cbff46c1b24"
+
+PYPI_PACKAGE = "pyelftools"
+
+inherit pypi setuptools3
+
+BBCLASSEXTEND = "native"
-- 
2.17.1

[OE-core][PATCH 3/4] python3-pycryptodome(x): Upgrade 3.9.4 -> 3.9.7

[Joshua Watt]

Also splits apart the SRC_URI checksums to make automatic upgrades
easier

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/recipes-devtools/python/python-pycryptodome.inc         | 3 ---
 meta/recipes-devtools/python/python3-pycryptodome_3.9.4.bb   | 2 --
 meta/recipes-devtools/python/python3-pycryptodome_3.9.7.bb   | 5 +++++
 ...pycryptodomex_3.9.4.bb => python3-pycryptodomex_3.9.7.bb} | 3 +--
 4 files changed, 6 insertions(+), 7 deletions(-)
 delete mode 100644 meta/recipes-devtools/python/python3-pycryptodome_3.9.4.bb
 create mode 100644 meta/recipes-devtools/python/python3-pycryptodome_3.9.7.bb
 rename meta/recipes-devtools/python/{python3-pycryptodomex_3.9.4.bb => python3-pycryptodomex_3.9.7.bb} (58%)

diff --git a/meta/recipes-devtools/python/python-pycryptodome.inc b/meta/recipes-devtools/python/python-pycryptodome.inc
index 63b4a4abb4..68b084eb04 100644
--- a/meta/recipes-devtools/python/python-pycryptodome.inc
+++ b/meta/recipes-devtools/python/python-pycryptodome.inc
@@ -5,9 +5,6 @@ HOMEPAGE = "http://www.pycryptodome.org"
 LICENSE = "PD & BSD-2-Clause"
 LIC_FILES_CHKSUM = "file://LICENSE.rst;md5=6dc0e2a13d2f25d6f123c434b761faba"
 
-SRC_URI[md5sum] = "f990716b49add7b14ea8b8a961fb3746"
-SRC_URI[sha256sum] = "a168e73879619b467072509a223282a02c8047d932a48b74fbd498f27224aa04"
-
 inherit pypi
 
 RDEPENDS_${PN} += " \
diff --git a/meta/recipes-devtools/python/python3-pycryptodome_3.9.4.bb b/meta/recipes-devtools/python/python3-pycryptodome_3.9.4.bb
deleted file mode 100644
index 0c062dddf8..0000000000
--- a/meta/recipes-devtools/python/python3-pycryptodome_3.9.4.bb
+++ /dev/null
@@ -1,2 +0,0 @@
-require python-pycryptodome.inc
-inherit setuptools3
diff --git a/meta/recipes-devtools/python/python3-pycryptodome_3.9.7.bb b/meta/recipes-devtools/python/python3-pycryptodome_3.9.7.bb
new file mode 100644
index 0000000000..8f19984bed
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-pycryptodome_3.9.7.bb
@@ -0,0 +1,5 @@
+require python-pycryptodome.inc
+inherit setuptools3
+
+SRC_URI[sha256sum] = "f1add21b6d179179b3c177c33d18a2186a09cc0d3af41ff5ed3f377360b869f2"
+
diff --git a/meta/recipes-devtools/python/python3-pycryptodomex_3.9.4.bb b/meta/recipes-devtools/python/python3-pycryptodomex_3.9.7.bb
similarity index 58%
rename from meta/recipes-devtools/python/python3-pycryptodomex_3.9.4.bb
rename to meta/recipes-devtools/python/python3-pycryptodomex_3.9.7.bb
index e41c14b142..abb03b9909 100644
--- a/meta/recipes-devtools/python/python3-pycryptodomex_3.9.4.bb
+++ b/meta/recipes-devtools/python/python3-pycryptodomex_3.9.7.bb
@@ -1,8 +1,7 @@
 require python-pycryptodome.inc
 inherit setuptools3
 
-SRC_URI[md5sum] = "46ba513d95b6e323734074d960a7d57b"
-SRC_URI[sha256sum] = "22d970cee5c096b9123415e183ae03702b2cd4d3ba3f0ced25c4e1aba3967167"
+SRC_URI[sha256sum] = "50163324834edd0c9ce3e4512ded3e221c969086e10fdd5d3fdcaadac5e24a78"
 
 FILES_${PN}-tests = " \
     ${PYTHON_SITEPACKAGES_DIR}/Cryptodome/SelfTest/ \
-- 
2.17.1

[OE-core][PATCH 4/4] python3-pyelftools: Upgrade 0.25 -> 0.26

[Joshua Watt]

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 ...{python3-pyelftools_0.25.bb => python3-pyelftools_0.26.bb} | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
 rename meta/recipes-devtools/python/{python3-pyelftools_0.25.bb => python3-pyelftools_0.26.bb} (72%)

diff --git a/meta/recipes-devtools/python/python3-pyelftools_0.25.bb b/meta/recipes-devtools/python/python3-pyelftools_0.26.bb
similarity index 72%
rename from meta/recipes-devtools/python/python3-pyelftools_0.25.bb
rename to meta/recipes-devtools/python/python3-pyelftools_0.26.bb
index 03d96db3c2..575dfc4dc9 100644
--- a/meta/recipes-devtools/python/python3-pyelftools_0.25.bb
+++ b/meta/recipes-devtools/python/python3-pyelftools_0.26.bb
@@ -4,8 +4,8 @@ SECTION = "devel/python"
 LICENSE = "PD"
 LIC_FILES_CHKSUM = "file://LICENSE;md5=5ce2a2b07fca326bc7c146d10105ccfc"
 
-SRC_URI[md5sum] = "c5629b9a5d19c82107a946cce52eeec2"
-SRC_URI[sha256sum] = "89c6da6f56280c37a5ff33468591ba9a124e17d71fe42de971818cbff46c1b24"
+SRC_URI[md5sum] = "0ba0de4b47127249c4d632ae299cb0e8"
+SRC_URI[sha256sum] = "86ac6cee19f6c945e8dedf78c6ee74f1112bd14da5a658d8c9d4103aed5756a2"
 
 PYPI_PACKAGE = "pyelftools"
 
-- 
2.17.1

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Denys Dmytriyenko]

I see Richard has merged these to master-next, thanks!
And thanks Joshua for volunteering to maintain these recipes :)
I submitted removal patches to meta-python.

So, it is good we are getting this extra dependency resolved in master. The 
question is - can we get these backported to dunfell, does it qualify?

Otherwise Jon wanted to finally branch off meta-arm into dunfell today and 
we'll have to live with it until gatesgarth.

-- 
Denys


On Thu, May 14, 2020 at 04:04:54PM -0500, Joshua Watt wrote:
> Imports and upgrades pycryptodome, pycryptodomex, and pyelftools from
> meta-python.  These recipes are commonly used by other layers (e.g.
> meta-arm) so putting them in oe-core reduces layer dependencies.
> 
> Joshua Watt (4):
>   pycryptodome: Import from meta-python
>   pyelftools: Import from meta-python
>   python3-pycryptodome(x): Upgrade 3.9.4 -> 3.9.7
>   python3-pyelftools: Upgrade 0.25 -> 0.26
> 
>  meta/conf/distro/include/maintainers.inc      |  5 +++-
>  .../python/python-pycryptodome.inc            | 26 +++++++++++++++++++
>  .../python/python3-pycryptodome_3.9.7.bb      |  5 ++++
>  .../python/python3-pycryptodomex_3.9.7.bb     |  9 +++++++
>  .../python/python3-pyelftools_0.26.bb         | 14 ++++++++++
>  5 files changed, 58 insertions(+), 1 deletion(-)
>  create mode 100644 meta/recipes-devtools/python/python-pycryptodome.inc
>  create mode 100644 meta/recipes-devtools/python/python3-pycryptodome_3.9.7.bb
>  create mode 100644 meta/recipes-devtools/python/python3-pycryptodomex_3.9.7.bb
>  create mode 100644 meta/recipes-devtools/python/python3-pyelftools_0.26.bb
> 
> -- 
> 2.17.1
> 

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Richard Purdie]

On Fri, 2020-05-15 at 14:53 -0400, Denys Dmytriyenko wrote:
> I see Richard has merged these to master-next, thanks!
> And thanks Joshua for volunteering to maintain these recipes :)
> I submitted removal patches to meta-python.
> 
> So, it is good we are getting this extra dependency resolved in master. The 
> question is - can we get these backported to dunfell, does it qualify?
> 
> Otherwise Jon wanted to finally branch off meta-arm into dunfell today and 
> we'll have to live with it until gatesgarth.

I put them into -next so we could test them, see if the autobuilder had
any surprises. Good news is there weren't any.

The question still remains whether this is the best approach or not and
some people I talked to were not convinced that a consensus had been
reached.

The question I guess is how widely used is optee and does it warrant
adding this? Its a little ironic the thing people want is trusted-
firmware...

Cheers,

Richard

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Joshua Watt]

On 5/15/20 2:05 PM, Richard Purdie wrote:
> On Fri, 2020-05-15 at 14:53 -0400, Denys Dmytriyenko wrote:
>> I see Richard has merged these to master-next, thanks!
>> And thanks Joshua for volunteering to maintain these recipes :)
>> I submitted removal patches to meta-python.
>>
>> So, it is good we are getting this extra dependency resolved in master. The
>> question is - can we get these backported to dunfell, does it qualify?
>>
>> Otherwise Jon wanted to finally branch off meta-arm into dunfell today and
>> we'll have to live with it until gatesgarth.
> I put them into -next so we could test them, see if the autobuilder had
> any surprises. Good news is there weren't any.
>
> The question still remains whether this is the best approach or not and
> some people I talked to were not convinced that a consensus had been
> reached.
>
> The question I guess is how widely used is optee and does it warrant
> adding this? Its a little ironic the thing people want is trusted-
> firmware...

FWIW, after having gone through the exercise of pulling in TF-A into 
qemuarm64, I think I've been convinced that op-tee and TF-A belong 
together since they are sometimes tightly integrated together (something 
I didn't realize before). As such, having both in the same layer makes 
sense. Even if TF-A was in oe-core, you'd probably want op-tee also, 
which means the python modules would have to be there anyway. I think we 
can still have the discussion about moving the whole lot over there, but 
we don't need to do that now, and moving the python recipes at least 
cuts out the meta-python dependency.

My last concern was testing of optee, since there was no platform that 
could build it by default, but I fixed that by implemented support for a 
qemuarm64-based machine that's defined in meta-arm which uses TF-A + 
optee + u-boot and can successful boot using TF-A and passes the op-tee 
unit tests, so I don't have any concern about that anymore.

>
> Cheers,
>
> Richard
>

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Denys Dmytriyenko]

On Fri, May 15, 2020 at 02:12:55PM -0500, Joshua Watt wrote:
> 
> On 5/15/20 2:05 PM, Richard Purdie wrote:
> >On Fri, 2020-05-15 at 14:53 -0400, Denys Dmytriyenko wrote:
> >>I see Richard has merged these to master-next, thanks!
> >>And thanks Joshua for volunteering to maintain these recipes :)
> >>I submitted removal patches to meta-python.
> >>
> >>So, it is good we are getting this extra dependency resolved in master. The
> >>question is - can we get these backported to dunfell, does it qualify?
> >>
> >>Otherwise Jon wanted to finally branch off meta-arm into dunfell today and
> >>we'll have to live with it until gatesgarth.
> >I put them into -next so we could test them, see if the autobuilder had
> >any surprises. Good news is there weren't any.
> >
> >The question still remains whether this is the best approach or not and
> >some people I talked to were not convinced that a consensus had been
> >reached.
> >
> >The question I guess is how widely used is optee and does it warrant
> >adding this? Its a little ironic the thing people want is trusted-
> >firmware...
> 
> FWIW, after having gone through the exercise of pulling in TF-A into
> qemuarm64, I think I've been convinced that op-tee and TF-A belong
> together since they are sometimes tightly integrated together
> (something I didn't realize before). As such, having both in the
> same layer makes sense. Even if TF-A was in oe-core, you'd probably
> want op-tee also, which means the python modules would have to be
> there anyway. I think we can still have the discussion about moving
> the whole lot over there, but we don't need to do that now, and
> moving the python recipes at least cuts out the meta-python
> dependency.
> 
> My last concern was testing of optee, since there was no platform
> that could build it by default,

Many TI platforms do. All our recent Aarch64 platforms (K3 family) use 
TF-A and OP-TEE by default and require them both to boot.

Even many of our Armv7 platforms use OP-TEE, but you'd need a special "secure" 
version of the platform, and those are not available to the broad-market.


> but I fixed that by implemented
> support for a qemuarm64-based machine that's defined in meta-arm
> which uses TF-A + optee + u-boot and can successful boot using TF-A
> and passes the op-tee unit tests, so I don't have any concern about
> that anymore.

That's great for testing meta-arm components w/o requiring any specialized 
platforms!

-- 
Denys

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Khem Raj]

On 5/15/20 12:12 PM, Joshua Watt wrote:
> 
> On 5/15/20 2:05 PM, Richard Purdie wrote:
>> On Fri, 2020-05-15 at 14:53 -0400, Denys Dmytriyenko wrote:
>>> I see Richard has merged these to master-next, thanks!
>>> And thanks Joshua for volunteering to maintain these recipes :)
>>> I submitted removal patches to meta-python.
>>>
>>> So, it is good we are getting this extra dependency resolved in 
>>> master. The
>>> question is - can we get these backported to dunfell, does it qualify?
>>>
>>> Otherwise Jon wanted to finally branch off meta-arm into dunfell 
>>> today and
>>> we'll have to live with it until gatesgarth.
>> I put them into -next so we could test them, see if the autobuilder had
>> any surprises. Good news is there weren't any.
>>
>> The question still remains whether this is the best approach or not and
>> some people I talked to were not convinced that a consensus had been
>> reached.
>>
>> The question I guess is how widely used is optee and does it warrant
>> adding this? Its a little ironic the thing people want is trusted-
>> firmware...
> 
> FWIW, after having gone through the exercise of pulling in TF-A into 
> qemuarm64, I think I've been convinced that op-tee and TF-A belong 
> together since they are sometimes tightly integrated together (something 
> I didn't realize before). As such, having both in the same layer makes 
> sense. Even if TF-A was in oe-core, you'd probably want op-tee also, 
> which means the python modules would have to be there anyway. I think we 
> can still have the discussion about moving the whole lot over there, but 
> we don't need to do that now, and moving the python recipes at least 
> cuts out the meta-python dependency.
> 
> My last concern was testing of optee, since there was no platform that 
> could build it by default, but I fixed that by implemented support for a 
> qemuarm64-based machine that's defined in meta-arm which uses TF-A + 
> optee + u-boot and can successful boot using TF-A and passes the op-tee 
> unit tests, so I don't have any concern about that anymore.
> 

I think getting oe-core qemuarm64 boot with uboot+TF-A as an option 
sounds good, and if thats more common solution that armv8 devices are 
following then perhaps would make sense to have this tested in core too. 
But I guess that can be ironed out. As such python deps I think are good 
for dunfell too, we have to ensure meta-python removal happens in 
dunfell as well.

>>
>> Cheers,
>>
>> Richard
>>

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Andrey Zhizhikin]

On Fri, May 15, 2020 at 9:26 PM Denys Dmytriyenko <denis@denix.org> wrote:
>
> On Fri, May 15, 2020 at 02:12:55PM -0500, Joshua Watt wrote:
> >
> > On 5/15/20 2:05 PM, Richard Purdie wrote:
> > >On Fri, 2020-05-15 at 14:53 -0400, Denys Dmytriyenko wrote:
> > >>I see Richard has merged these to master-next, thanks!
> > >>And thanks Joshua for volunteering to maintain these recipes :)
> > >>I submitted removal patches to meta-python.
> > >>
> > >>So, it is good we are getting this extra dependency resolved in master. The
> > >>question is - can we get these backported to dunfell, does it qualify?
> > >>
> > >>Otherwise Jon wanted to finally branch off meta-arm into dunfell today and
> > >>we'll have to live with it until gatesgarth.
> > >I put them into -next so we could test them, see if the autobuilder had
> > >any surprises. Good news is there weren't any.
> > >
> > >The question still remains whether this is the best approach or not and
> > >some people I talked to were not convinced that a consensus had been
> > >reached.
> > >
> > >The question I guess is how widely used is optee and does it warrant
> > >adding this? Its a little ironic the thing people want is trusted-
> > >firmware...
> >
> > FWIW, after having gone through the exercise of pulling in TF-A into
> > qemuarm64, I think I've been convinced that op-tee and TF-A belong
> > together since they are sometimes tightly integrated together
> > (something I didn't realize before). As such, having both in the
> > same layer makes sense. Even if TF-A was in oe-core, you'd probably
> > want op-tee also, which means the python modules would have to be
> > there anyway. I think we can still have the discussion about moving
> > the whole lot over there, but we don't need to do that now, and
> > moving the python recipes at least cuts out the meta-python
> > dependency.
> >
> > My last concern was testing of optee, since there was no platform
> > that could build it by default,
>
> Many TI platforms do. All our recent Aarch64 platforms (K3 family) use
> TF-A and OP-TEE by default and require them both to boot.

Just FYI: NXP i.MX8[M] series uses OP-TEE and it is also required to
boot the SoC.

>
> Even many of our Armv7 platforms use OP-TEE, but you'd need a special "secure"
> version of the platform, and those are not available to the broad-market.
>
>
> > but I fixed that by implemented
> > support for a qemuarm64-based machine that's defined in meta-arm
> > which uses TF-A + optee + u-boot and can successful boot using TF-A
> > and passes the op-tee unit tests, so I don't have any concern about
> > that anymore.
>
> That's great for testing meta-arm components w/o requiring any specialized
> platforms!
>
> --
> Denys
> 



-- 
Regards,
Andrey.

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Steve Sakoman]

On Fri, May 15, 2020 at 9:06 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Fri, 2020-05-15 at 14:53 -0400, Denys Dmytriyenko wrote:
> > I see Richard has merged these to master-next, thanks!
> > And thanks Joshua for volunteering to maintain these recipes :)
> > I submitted removal patches to meta-python.
> >
> > So, it is good we are getting this extra dependency resolved in master. The
> > question is - can we get these backported to dunfell, does it qualify?
> >
> > Otherwise Jon wanted to finally branch off meta-arm into dunfell today and
> > we'll have to live with it until gatesgarth.
>
> I put them into -next so we could test them, see if the autobuilder had
> any surprises. Good news is there weren't any.
>
> The question still remains whether this is the best approach or not and
> some people I talked to were not convinced that a consensus had been
> reached.
>
> The question I guess is how widely used is optee and does it warrant
> adding this? Its a little ironic the thing people want is trusted-
> firmware...

I'm seeing a few votes of "yes for dunfell" and no opposition as yet
(at least here on the list).

I support the desire to get rid of a meta-python dependency in bsp
layers so I'm also a yes vote unless someone comes up with a good
reason why we shouldn't.

Steve

> Cheers,
>
> Richard
>
> 

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Adrian Bunk]

On Fri, May 15, 2020 at 03:21:57PM -1000, Steve Sakoman wrote:
> 
> I'm seeing a few votes of "yes for dunfell" and no opposition as yet
> (at least here on the list).
> 
> I support the desire to get rid of a meta-python dependency in bsp
> layers so I'm also a yes vote unless someone comes up with a good
> reason why we shouldn't.

meta-openembedded/meta-python has a higher layer priority than OE-core.

Adding higher upstream versions of these recipes to a lower-priority
layer in a stable series is a potential source for weird problems.

> Steve

cu
Adrian

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Adrian Bunk
> Sent: den 16 maj 2020 19:47
> To: Steve Sakoman <sakoman@gmail.com>
> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; Denys
> Dmytriyenko <denis@denix.org>; Joshua Watt <jpewhacker@gmail.com>;
> Patches and discussions about the oe-core layer <openembedded-
> core@lists.openembedded.org>; jdmason@kudzu.us; Khem Raj
> <raj.khem@gmail.com>
> Subject: Re: [OE-core][PATCH 0/4] Import recipes from meta-python
> 
> On Fri, May 15, 2020 at 03:21:57PM -1000, Steve Sakoman wrote:
> >
> > I'm seeing a few votes of "yes for dunfell" and no opposition as yet
> > (at least here on the list).
> >
> > I support the desire to get rid of a meta-python dependency in bsp
> > layers so I'm also a yes vote unless someone comes up with a good
> > reason why we shouldn't.
> 
> meta-openembedded/meta-python has a higher layer priority than OE-core.
> 
> Adding higher upstream versions of these recipes to a lower-priority
> layer in a stable series is a potential source for weird problems.
> 
> > Steve
> 
> cu
> Adrian

I would assume they'd be removed from meta-python at the same time 
they are added to meta.

//Peter

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Adrian Bunk]

On Sat, May 16, 2020 at 07:54:32PM +0000, Peter Kjellerstedt wrote:
> > 
> > meta-openembedded/meta-python has a higher layer priority than OE-core.
> > 
> > Adding higher upstream versions of these recipes to a lower-priority
> > layer in a stable series is a potential source for weird problems.
> 
> I would assume they'd be removed from meta-python at the same time 
> they are added to meta.

"at the same time" is complicated since OE-core has releases,
but meta-openembedded is just a branch.

I would also assume that not all users of stable series are updating
meta-openembedded to the latest on the dunfell branch at the same
time as OE-core, some users might end up updating one but never
updating the other one.

> //Peter

cu
Adrian

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Andrey Zhizhikin]

On Sat, May 16, 2020 at 10:10 PM Adrian Bunk <bunk@stusta.de> wrote:
>
> On Sat, May 16, 2020 at 07:54:32PM +0000, Peter Kjellerstedt wrote:
> > >
> > > meta-openembedded/meta-python has a higher layer priority than OE-core.
> > >
> > > Adding higher upstream versions of these recipes to a lower-priority
> > > layer in a stable series is a potential source for weird problems.
> >
> > I would assume they'd be removed from meta-python at the same time
> > they are added to meta.
>
> "at the same time" is complicated since OE-core has releases,
> but meta-openembedded is just a branch.
>
> I would also assume that not all users of stable series are updating
> meta-openembedded to the latest on the dunfell branch at the same
> time as OE-core, some users might end up updating one but never
> updating the other one.

That I believe would be a terrible mistake when people opt-in to take
a layer, but never care about updating it... :(

On the contrary, a quick run of "bitbake-layers show-overlayed"
exhibits the following on the [master] of both OE-Core and
meta-openembedded:
=================
python3-cython:
  meta-python          0.29.14
  meta                 0.29.16
python3-dbusmock:
  meta-python          0.16.7
  meta                 0.19
python3-docutils:
  meta-python          0.15.2
  meta                 0.16
python3-pyparsing:
  meta-python          2.4.6
  meta                 2.4.7
=================

Judging be versions, it looks like OE-Core recipes are maintained, but
in meta-openembedded they are left on the side...

>
> > //Peter
>
> cu
> Adrian
> 



-- 
Regards,
Andrey.

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Andre McCurdy]

On Sat, May 16, 2020 at 1:24 PM Andrey Zhizhikin <andrey.z@gmail.com> wrote:
> On Sat, May 16, 2020 at 10:10 PM Adrian Bunk <bunk@stusta.de> wrote:
> > On Sat, May 16, 2020 at 07:54:32PM +0000, Peter Kjellerstedt wrote:
> > > >
> > > > meta-openembedded/meta-python has a higher layer priority than OE-core.
> > > >
> > > > Adding higher upstream versions of these recipes to a lower-priority
> > > > layer in a stable series is a potential source for weird problems.
> > >
> > > I would assume they'd be removed from meta-python at the same time
> > > they are added to meta.
> >
> > "at the same time" is complicated since OE-core has releases,
> > but meta-openembedded is just a branch.
> >
> > I would also assume that not all users of stable series are updating
> > meta-openembedded to the latest on the dunfell branch at the same
> > time as OE-core, some users might end up updating one but never
> > updating the other one.
>
> That I believe would be a terrible mistake when people opt-in to take
> a layer, but never care about updating it... :(
>
> On the contrary, a quick run of "bitbake-layers show-overlayed"
> exhibits the following on the [master] of both OE-Core and
> meta-openembedded:
> =================
> python3-cython:
>   meta-python          0.29.14
>   meta                 0.29.16
> python3-dbusmock:
>   meta-python          0.16.7
>   meta                 0.19
> python3-docutils:
>   meta-python          0.15.2
>   meta                 0.16
> python3-pyparsing:
>   meta-python          2.4.6
>   meta                 2.4.7
> =================
>
> Judging be versions, it looks like OE-Core recipes are maintained, but
> in meta-openembedded they are left on the side...

Right, and that's the problem. Anyone using meta-python will continue
to use the older recipes and not see the updates (since meta-python is
a higher priority layer than oe-core).

The right way to have done this would have been to update the recipes
in meta-python before merging to oe-core, not merging to oe-core and
then immediately updating them there.

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Alejandro Hernandez]

[-- Attachment #1: Type: text/plain, Size: 2906 bytes --]

On Fri, May 15, 2020, 1:32 PM Khem Raj <raj.khem@gmail.com> wrote:

>
>
> On 5/15/20 12:12 PM, Joshua Watt wrote:
> >
> > On 5/15/20 2:05 PM, Richard Purdie wrote:
> >> On Fri, 2020-05-15 at 14:53 -0400, Denys Dmytriyenko wrote:
> >>> I see Richard has merged these to master-next, thanks!
> >>> And thanks Joshua for volunteering to maintain these recipes :)
> >>> I submitted removal patches to meta-python.
> >>>
> >>> So, it is good we are getting this extra dependency resolved in
> >>> master. The
> >>> question is - can we get these backported to dunfell, does it qualify?
> >>>
> >>> Otherwise Jon wanted to finally branch off meta-arm into dunfell
> >>> today and
> >>> we'll have to live with it until gatesgarth.
> >> I put them into -next so we could test them, see if the autobuilder had
> >> any surprises. Good news is there weren't any.
> >>
> >> The question still remains whether this is the best approach or not and
> >> some people I talked to were not convinced that a consensus had been
> >> reached.
> >>
> >> The question I guess is how widely used is optee and does it warrant
> >> adding this? Its a little ironic the thing people want is trusted-
> >> firmware...
> >
> > FWIW, after having gone through the exercise of pulling in TF-A into
> > qemuarm64, I think I've been convinced that op-tee and TF-A belong
> > together since they are sometimes tightly integrated together (something
> > I didn't realize before). As such, having both in the same layer makes
> > sense. Even if TF-A was in oe-core, you'd probably want op-tee also,
> > which means the python modules would have to be there anyway. I think we
> > can still have the discussion about moving the whole lot over there, but
> > we don't need to do that now, and moving the python recipes at least
> > cuts out the meta-python dependency.
> >
> > My last concern was testing of optee, since there was no platform that
> > could build it by default, but I fixed that by implemented support for a
> > qemuarm64-based machine that's defined in meta-arm which uses TF-A +
> > optee + u-boot and can successful boot using TF-A and passes the op-tee
> > unit tests, so I don't have any concern about that anymore.
> >
>
> I think getting oe-core qemuarm64 boot with uboot+TF-A as an option
> sounds good, and if thats more common solution that armv8 devices are
> following then perhaps would make sense to have this tested in core too.
> But I guess that can be ironed out. As such python deps I think are good
> for dunfell too, we have to ensure meta-python removal happens in
> dunfell as well.
>

+1 on keeping testing this in core as the end goal, I also agree that this
would qualify for the Dunfell changes in both oe-core and meta-python since
its the first step in that direction

Alejandro


> >>
> >> Cheers,
> >>
> >> Richard
> >>
> 
>

[-- Attachment #2: Type: text/html, Size: 4049 bytes --]

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Adrian Bunk]

On Sat, May 16, 2020 at 10:24:04PM +0200, Andrey Zhizhikin wrote:
> On Sat, May 16, 2020 at 10:10 PM Adrian Bunk <bunk@stusta.de> wrote:
> >
> > On Sat, May 16, 2020 at 07:54:32PM +0000, Peter Kjellerstedt wrote:
> > > >
> > > > meta-openembedded/meta-python has a higher layer priority than OE-core.
> > > >
> > > > Adding higher upstream versions of these recipes to a lower-priority
> > > > layer in a stable series is a potential source for weird problems.
> > >
> > > I would assume they'd be removed from meta-python at the same time
> > > they are added to meta.
> >
> > "at the same time" is complicated since OE-core has releases,
> > but meta-openembedded is just a branch.
> >
> > I would also assume that not all users of stable series are updating
> > meta-openembedded to the latest on the dunfell branch at the same
> > time as OE-core, some users might end up updating one but never
> > updating the other one.
> 
> That I believe would be a terrible mistake when people opt-in to take
> a layer, but never care about updating it... :(

The typical Yocto user starts with whatever prehistoric Yocto release 
came with the BSP distribution for the reference hardware, and then 
develops a new product on top of that.

I would not be surprised if someone will have the initial Yocto 2.7
release, but uses the latest from the corresponding  meta-openembedded 
branch on top.

> On the contrary, a quick run of "bitbake-layers show-overlayed"
> exhibits the following on the [master] of both OE-Core and

Thanks a lot for this.

> meta-openembedded:
> =================
> python3-cython:
>   meta-python          0.29.14
>   meta                 0.29.16
> python3-dbusmock:
>   meta-python          0.16.7
>   meta                 0.19
> python3-docutils:
>   meta-python          0.15.2
>   meta                 0.16
> python3-pyparsing:
>   meta-python          2.4.6
>   meta                 2.4.7
> =================

I'll take care of getting these removed from meta-openembedded.

> Judging be versions, it looks like OE-Core recipes are maintained, but
> in meta-openembedded they are left on the side...

python3-docutils is ouch, since this problem is also in dunfell.

On a more positive note 0.16 is already in OE-core in the initial
Yocto 3.1 release, so removing it from meta-openembedded will only
be like a 0.15.2 -> 0.16 upgrade for some users but cannot result
in losing the recipe in weird layer combinations.

> Regards,
> Andrey.

cu
Adrian

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1562 bytes --]

On Sun, May 17, 2020 at 3:22 PM Adrian Bunk <bunk@stusta.de> wrote:

> > That I believe would be a terrible mistake when people opt-in to take
> > a layer, but never care about updating it... :(
>
> The typical Yocto user starts with whatever prehistoric Yocto release
> came with the BSP distribution for the reference hardware, and then
> develops a new product on top of that.
>
> I would not be surprised if someone will have the initial Yocto 2.7
> release, but uses the latest from the corresponding  meta-openembedded
> branch on top.
>

And should these people really block us doing changes which require update
in multiple layers at "the same time"?

If they are using the prehistory revision of given Yocto release and not
even the latest revision in given branch, then they are already missing
much more than 3 python recipes migrated from meta-oe and they should learn
to update the layers they are using, I think we have relatively good track
of not breaking stable branches with incompatible changes, so upgrading
from e.g. initial release revision of dunfell to latest in oe-core together
with all other layers, shouldn't cause (m)any issues after 5 years when
someone will be looking at this BSP which got released with prehistory
dunfell release.

Point releases are nice, but people should really use whatever is latest
revisions in the branches they use and that's IMHO the recommendation from
Yocto Project itself as well (IIRC we discussed it in OE TSC when talking
about tags for point releases).

Regards,

[-- Attachment #2: Type: text/html, Size: 1947 bytes --]

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Adrian Bunk]

On Sun, May 17, 2020 at 03:56:20PM +0200, Martin Jansa wrote:
>...
> so upgrading
> from e.g. initial release revision of dunfell to latest in oe-core together
> with all other layers,
>...

Your "together" is an assumption that is not always true in my experience.

> Point releases are nice, but people should really use whatever is latest
> revisions in the branches they use and that's IMHO the recommendation from
> Yocto Project itself as well (IIRC we discussed it in OE TSC when talking
> about tags for point releases).

What often strikes me about OE/Yocto is how far apart what developer 
think how things should be done is from what users are actually doing.

Yocto lacks novice level documentation how users should maintain
a distribution they got from somewhere for building a product.

> Regards,

cu
Adrian

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1780 bytes --]

On Sun, May 17, 2020 at 5:45 PM Adrian Bunk <bunk@stusta.de> wrote:

> On Sun, May 17, 2020 at 03:56:20PM +0200, Martin Jansa wrote:
> >...
> > so upgrading
> > from e.g. initial release revision of dunfell to latest in oe-core
> together
> > with all other layers,
> >...
>
> Your "together" is an assumption that is not always true in my experience.
>

Yes, I might be lucky that I can influence that in projects I'm involved in.

But still we should try to improve that experience instead of blocking
updates like this.


> > Point releases are nice, but people should really use whatever is latest
> > revisions in the branches they use and that's IMHO the recommendation
> from
> > Yocto Project itself as well (IIRC we discussed it in OE TSC when talking
> > about tags for point releases).
>
> What often strikes me about OE/Yocto is how far apart what developer
> think how things should be done is from what users are actually doing.
>
> Yocto lacks novice level documentation how users should maintain
> a distribution they got from somewhere for building a product.
>

Agreed, the documentation and user education should be improved.

But IMHO carrying duplicate recipes in oe-core and meta-oe, just because
some users aren't updating meta-oe, only adds more confusion.

If they update oe-core and don't update meta-oe, then they will still get
older version of these python modules, which is bad, but they are already
missing some other fixes from meta-oe.

If they update meta-oe and don't update oe-core, then these python modules
will disappear for them, if they use them at all, then they will probably
notice and that might help them realize, that they should be updating all
layers, not only some of them.

Cheers,

[-- Attachment #2: Type: text/html, Size: 2404 bytes --]

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Adrian Bunk]

On Sun, May 17, 2020 at 06:00:58PM +0200, Martin Jansa wrote:
>...
> But IMHO carrying duplicate recipes in oe-core and meta-oe, just because
> some users aren't updating meta-oe, only adds more confusion.
>...

IMHO moving recipes from meta-openembedded (or other layers) to oe-core 
in a released stable series is not a good idea.

> Cheers,

cu
Adrian

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 1853 bytes --]



On 5/16/20 1:09 PM, Adrian Bunk wrote:
> On Sat, May 16, 2020 at 07:54:32PM +0000, Peter Kjellerstedt wrote:
>>>
>>> meta-openembedded/meta-python has a higher layer priority than OE-core.
>>>
>>> Adding higher upstream versions of these recipes to a lower-priority
>>> layer in a stable series is a potential source for weird problems.
>>
>> I would assume they'd be removed from meta-python at the same time 
>> they are added to meta.
> 
> "at the same time" is complicated since OE-core has releases,
> but meta-openembedded is just a branch.
> 

You bring an interesting point. while the oe-core release tags are
driven by Yocto release cadence which has point release schedules,
OE-Core in itself does not have its release mechanism besides creating
and maintaining the branch. because oe-core independently is just a repo
you will need bitbake repo atleast to build something and bitbake has
its own release cadence.

> I would also assume that not all users of stable series are updating
> meta-openembedded to the latest on the dunfell branch at the same
> time as OE-core, some users might end up updating one but never
> updating the other one.

Thats very likely, although not preferred, most of products just freeze
these layers and do tweaking in their own layers because they have
control over them, where as they dont have much control over the
upstream layers. Its common with all third party and OSS projects.

So, if someone is basing their distro on oe-core + bitbake + meta-oe and
more then I would think they should be aware of these kind of changes
but I also think this could create some unexpected behaviour. Ideally
these recipes should be updated in meta-python to latest and same moved
to oe-core, then the move does not have to be atomic.

> 
>> //Peter
> 
> cu
> Adrian
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 2843 bytes --]



On 5/17/20 6:22 AM, Adrian Bunk wrote:
> On Sat, May 16, 2020 at 10:24:04PM +0200, Andrey Zhizhikin wrote:
>> On Sat, May 16, 2020 at 10:10 PM Adrian Bunk <bunk@stusta.de> wrote:
>>>
>>> On Sat, May 16, 2020 at 07:54:32PM +0000, Peter Kjellerstedt wrote:
>>>>>
>>>>> meta-openembedded/meta-python has a higher layer priority than OE-core.
>>>>>
>>>>> Adding higher upstream versions of these recipes to a lower-priority
>>>>> layer in a stable series is a potential source for weird problems.
>>>>
>>>> I would assume they'd be removed from meta-python at the same time
>>>> they are added to meta.
>>>
>>> "at the same time" is complicated since OE-core has releases,
>>> but meta-openembedded is just a branch.
>>>
>>> I would also assume that not all users of stable series are updating
>>> meta-openembedded to the latest on the dunfell branch at the same
>>> time as OE-core, some users might end up updating one but never
>>> updating the other one.
>>
>> That I believe would be a terrible mistake when people opt-in to take
>> a layer, but never care about updating it... :(
> 
> The typical Yocto user starts with whatever prehistoric Yocto release 
> came with the BSP distribution for the reference hardware, and then 
> develops a new product on top of that.
> 
> I would not be surprised if someone will have the initial Yocto 2.7
> release, but uses the latest from the corresponding  meta-openembedded 
> branch on top

To be honest, we should not consider supporting such usecases IMO,
somewhere we have to draw a line, mixing layer releases is not something
upstream can even test those combos.


> 
>> On the contrary, a quick run of "bitbake-layers show-overlayed"
>> exhibits the following on the [master] of both OE-Core and
> 
> Thanks a lot for this.
> 
>> meta-openembedded:
>> =================
>> python3-cython:
>>   meta-python          0.29.14
>>   meta                 0.29.16
>> python3-dbusmock:
>>   meta-python          0.16.7
>>   meta                 0.19
>> python3-docutils:
>>   meta-python          0.15.2
>>   meta                 0.16
>> python3-pyparsing:
>>   meta-python          2.4.6
>>   meta                 2.4.7
>> =================
> 
> I'll take care of getting these removed from meta-openembedded.
> 
>> Judging be versions, it looks like OE-Core recipes are maintained, but
>> in meta-openembedded they are left on the side...
> 
> python3-docutils is ouch, since this problem is also in dunfell.
> 
> On a more positive note 0.16 is already in OE-core in the initial
> Yocto 3.1 release, so removing it from meta-openembedded will only
> be like a 0.15.2 -> 0.16 upgrade for some users but cannot result
> in losing the recipe in weird layer combinations.
> 
>> Regards,
>> Andrey.
> 
> cu
> Adrian
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 1270 bytes --]



On 5/17/20 8:45 AM, Adrian Bunk wrote:
> On Sun, May 17, 2020 at 03:56:20PM +0200, Martin Jansa wrote:
>> ...
>> so upgrading
>> from e.g. initial release revision of dunfell to latest in oe-core together
>> with all other layers,
>> ...
> 
> Your "together" is an assumption that is not always true in my experience.
> 
>> Point releases are nice, but people should really use whatever is latest
>> revisions in the branches they use and that's IMHO the recommendation from
>> Yocto Project itself as well (IIRC we discussed it in OE TSC when talking
>> about tags for point releases).
> 
> What often strikes me about OE/Yocto is how far apart what developer 
> think how things should be done is from what users are actually doing.

This intrigues me and I think we should pay attention to usecases and
try to address them as much as we can, so it would help in perhaps
creating a wiki document or some such on how end users are instrumenting
their distro policies and challenges. How good is layered structure
helping them and how far can it be taken etc.

> 
> Yocto lacks novice level documentation how users should maintain
> a distribution they got from somewhere for building a product.
> 
>> Regards,
> 
> cu
> Adrian
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 647 bytes --]



On 5/17/20 9:14 AM, Adrian Bunk wrote:
> On Sun, May 17, 2020 at 06:00:58PM +0200, Martin Jansa wrote:
>> ...
>> But IMHO carrying duplicate recipes in oe-core and meta-oe, just because
>> some users aren't updating meta-oe, only adds more confusion.
>> ...
> 
> IMHO moving recipes from meta-openembedded (or other layers) to oe-core 
> in a released stable series is not a good idea.

I think in current form in this situation it certainly seems a bit
complex sadly, however, we also need to explore if there is some "good"
way of doing such changes without breaking distro builders.

> 
>> Cheers,
> 
> cu
> Adrian
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Alejandro Hernandez]

[-- Attachment #1: Type: text/plain, Size: 3862 bytes --]

On Sun, May 17, 2020, 5:41 PM Khem Raj <raj.khem@gmail.com> wrote:

>
>
> On 5/17/20 6:22 AM, Adrian Bunk wrote:
> > On Sat, May 16, 2020 at 10:24:04PM +0200, Andrey Zhizhikin wrote:
> >> On Sat, May 16, 2020 at 10:10 PM Adrian Bunk <bunk@stusta.de> wrote:
> >>>
> >>> On Sat, May 16, 2020 at 07:54:32PM +0000, Peter Kjellerstedt wrote:
> >>>>>
> >>>>> meta-openembedded/meta-python has a higher layer priority than
> OE-core.
> >>>>>
> >>>>> Adding higher upstream versions of these recipes to a lower-priority
> >>>>> layer in a stable series is a potential source for weird problems.
> >>>>
> >>>> I would assume they'd be removed from meta-python at the same time
> >>>> they are added to meta.
> >>>
> >>> "at the same time" is complicated since OE-core has releases,
> >>> but meta-openembedded is just a branch.
> >>>
> >>> I would also assume that not all users of stable series are updating
> >>> meta-openembedded to the latest on the dunfell branch at the same
> >>> time as OE-core, some users might end up updating one but never
> >>> updating the other one.
> >>
> >> That I believe would be a terrible mistake when people opt-in to take
> >> a layer, but never care about updating it... :(
> >
> > The typical Yocto user starts with whatever prehistoric Yocto release
> > came with the BSP distribution for the reference hardware, and then
> > develops a new product on top of that.
> >
> > I would not be surprised if someone will have the initial Yocto 2.7
> > release, but uses the latest from the corresponding  meta-openembedded
> > branch on top
>
> To be honest, we should not consider supporting such usecases IMO,
> somewhere we have to draw a line, mixing layer releases is not something
> upstream can even test those combos.
>

I think it should be pretty clear that the product owner has the
responsibility
to update whatever BSP distribution they're using, and make that available
to
their users.

At least IMO it was clear that users should maintain their distribution to
the
latest update of their stable branch, we can certainly put that on the docs
or
a wiki but I thought it was sort of given, since it simply makes sense, but
perhaps making that assumption was a mistake.

We can only do as much as bring updates after a certain release, wether its
a case like this or its a CVE. We cannot control what some products do, yet
we shouldn't let it stop us from doing changes that require updates in
multiple
layers if its a change that makes sense and most users will benefit from.


Alejandro


>
> >
> >> On the contrary, a quick run of "bitbake-layers show-overlayed"
> >> exhibits the following on the [master] of both OE-Core and
> >
> > Thanks a lot for this.
> >
> >> meta-openembedded:
> >> =================
> >> python3-cython:
> >>   meta-python          0.29.14
> >>   meta                 0.29.16
> >> python3-dbusmock:
> >>   meta-python          0.16.7
> >>   meta                 0.19
> >> python3-docutils:
> >>   meta-python          0.15.2
> >>   meta                 0.16
> >> python3-pyparsing:
> >>   meta-python          2.4.6
> >>   meta                 2.4.7
> >> =================
> >
> > I'll take care of getting these removed from meta-openembedded.
> >
> >> Judging be versions, it looks like OE-Core recipes are maintained, but
> >> in meta-openembedded they are left on the side...
> >
> > python3-docutils is ouch, since this problem is also in dunfell.
> >
> > On a more positive note 0.16 is already in OE-core in the initial
> > Yocto 3.1 release, so removing it from meta-openembedded will only
> > be like a 0.15.2 -> 0.16 upgrade for some users but cannot result
> > in losing the recipe in weird layer combinations.
> >
> >> Regards,
> >> Andrey.
> >
> > cu
> > Adrian
> >
>
> 
>

[-- Attachment #2: Type: text/html, Size: 5704 bytes --]

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Steve Sakoman]

On Sun, May 17, 2020 at 1:47 PM Khem Raj <raj.khem@gmail.com> wrote:

> On 5/17/20 9:14 AM, Adrian Bunk wrote:

> > IMHO moving recipes from meta-openembedded (or other layers) to oe-core
> > in a released stable series is not a good idea.
>
> I think in current form in this situation it certainly seems a bit
> complex sadly, however, we also need to explore if there is some "good"
> way of doing such changes without breaking distro builders.

It seems we don't have a clear consensus that backporting these
patches to dunfell is a good idea.

So for now I'm going to take the conservative approach and refrain
from taking these patches in LTS.

We can always revisit this later if the situation changes.

Steve

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1249 bytes --]

We have discussed this in OE TSC meeting yesterday:
https://lists.openembedded.org/g/tsc/topic/openembedded_tsc_meeting/74313532

tsc suggested to use LAYERVERSION bump in oe-core with
LAYERDEPENDS_meta-python update in meta-python, so that people will need to
update both layers (or neither if they really cannot).

Joshua volunteered to send patches if everybody agrees on this proposal.

On Tue, May 19, 2020 at 8:21 PM Steve Sakoman <sakoman@gmail.com> wrote:

> On Sun, May 17, 2020 at 1:47 PM Khem Raj <raj.khem@gmail.com> wrote:
>
> > On 5/17/20 9:14 AM, Adrian Bunk wrote:
>
> > > IMHO moving recipes from meta-openembedded (or other layers) to oe-core
> > > in a released stable series is not a good idea.
> >
> > I think in current form in this situation it certainly seems a bit
> > complex sadly, however, we also need to explore if there is some "good"
> > way of doing such changes without breaking distro builders.
>
> It seems we don't have a clear consensus that backporting these
> patches to dunfell is a good idea.
>
> So for now I'm going to take the conservative approach and refrain
> from taking these patches in LTS.
>
> We can always revisit this later if the situation changes.
>
> Steve
>

[-- Attachment #2: Type: text/html, Size: 1823 bytes --]

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Andre McCurdy]

On Tue, May 19, 2020 at 11:25 AM Martin Jansa <Martin.Jansa@gmail.com> wrote:
>
> We have discussed this in OE TSC meeting yesterday:
> https://lists.openembedded.org/g/tsc/topic/openembedded_tsc_meeting/74313532
>
> tsc suggested to use LAYERVERSION bump in oe-core with LAYERDEPENDS_meta-python update in meta-python, so that people will need to update both layers (or neither if they really cannot).

OE living up to it's nickname... Over Engineered :-)

> Joshua volunteered to send patches if everybody agrees on this proposal.
>
> On Tue, May 19, 2020 at 8:21 PM Steve Sakoman <sakoman@gmail.com> wrote:
>>
>> On Sun, May 17, 2020 at 1:47 PM Khem Raj <raj.khem@gmail.com> wrote:
>>
>> > On 5/17/20 9:14 AM, Adrian Bunk wrote:
>>
>> > > IMHO moving recipes from meta-openembedded (or other layers) to oe-core
>> > > in a released stable series is not a good idea.
>> >
>> > I think in current form in this situation it certainly seems a bit
>> > complex sadly, however, we also need to explore if there is some "good"
>> > way of doing such changes without breaking distro builders.
>>
>> It seems we don't have a clear consensus that backporting these
>> patches to dunfell is a good idea.
>>
>> So for now I'm going to take the conservative approach and refrain
>> from taking these patches in LTS.
>>
>> We can always revisit this later if the situation changes.
>>
>> Steve
>
> 

Re: [OE-core][PATCH 0/4] Import recipes from meta-python

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 661 bytes --]

On Tue, May 19, 2020 at 9:57 PM Andre McCurdy <armccurdy@gmail.com> wrote:

> On Tue, May 19, 2020 at 11:25 AM Martin Jansa <Martin.Jansa@gmail.com>
> wrote:
> >
> > We have discussed this in OE TSC meeting yesterday:
> >
> https://lists.openembedded.org/g/tsc/topic/openembedded_tsc_meeting/74313532
> >
> > tsc suggested to use LAYERVERSION bump in oe-core with
> LAYERDEPENDS_meta-python update in meta-python, so that people will need to
> update both layers (or neither if they really cannot).
>
> OE living up to it's nickname... Over Engineered :-)
>

Compared with some really Over Engineered stuff I've seen, this feels like
KISS :).

[-- Attachment #2: Type: text/html, Size: 1153 bytes --]