[RFC PATCH 1/2] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock

[RFC PATCH 1/2] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock

[Alexander Kanavin]

For better or worse, more and more rust components are appearing that do
not include their dependencies in tarballs (or git trees), and rely on cargo
to fetch them. On the other hand, bitbake does not use cargo (and quite possible
won't ever be able to), and relies on having each item explicitly listed in SRC_URI
with a crate:// prefix. This however creates a problem of both making such lists in
the first place and updating them when a recipe is updated to a newer version.

So this class can be used to perform such updates by implementing a task that does it;
the next commit shows the outcome for python3-bcrypt (which has been tested to work
and produce a successful build).

Note: the python script relies on tomllib library, which appears in Python 3.11 and
does not exist in earlier versions - I've tested this by first updating python to 3.11-rc2
in oe-core.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 .../cargo-update-recipe-crates.bbclass        | 41 +++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 meta/classes-recipe/cargo-update-recipe-crates.bbclass

diff --git a/meta/classes-recipe/cargo-update-recipe-crates.bbclass b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
new file mode 100644
index 0000000000..f90938c734
--- /dev/null
+++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
@@ -0,0 +1,41 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+##
+## Purpose:
+## This class is used to update the list of crates in SRC_URI
+## by reading Cargo.lock in the source tree.
+##
+## See meta/recipes-devtools/python/python3-bcrypt_*.bb for an example
+##
+## To perform the update: bitbake -c update_crates recipe-name
+
+addtask do_update_crates after do_patch
+do_update_crates[depends] = "python3-native:do_populate_sysroot"
+
+do_update_crates() {
+    nativepython3 - <<EOF
+
+def get_crates(f):
+    import tomllib
+    c_list = 'SRC_URI += " \\ \n'
+    crates = tomllib.load(open(f, 'rb'))
+    for c in crates['package']:
+        if 'source' in c and 'crates.io' in c['source']:
+            c_list += "        crate://crates.io/{}/{} \\ \n".format(c['name'], c['version'])
+    c_list += '"\n'
+    return c_list
+
+import os
+crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
+for root, dirs, files in os.walk('${S}'):
+    for file in files:
+        if file == 'Cargo.lock':
+            crates += get_crates(os.path.join(root, file))
+open(os.path.join('${THISDIR}', '${PN}'+"-crates.inc"), 'w').write(crates)
+
+EOF
+}
-- 
2.30.2

[RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0

[Alexander Kanavin]

The component has been reimplemented in rust, and comes
with a large list of dependencies in Cargo.toml/Cargo.lock.

Rather than list them by hand, use a file generated with
cargo-update-recipe-crates class.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 .../python/python3-bcrypt-crates.inc          | 53 +++++++++++++++++++
 ...crypt_3.2.2.bb => python3-bcrypt_4.0.0.bb} |  6 ++-
 2 files changed, 57 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-devtools/python/python3-bcrypt-crates.inc
 rename meta/recipes-devtools/python/{python3-bcrypt_3.2.2.bb => python3-bcrypt_4.0.0.bb} (76%)

diff --git a/meta/recipes-devtools/python/python3-bcrypt-crates.inc b/meta/recipes-devtools/python/python3-bcrypt-crates.inc
new file mode 100644
index 0000000000..78c5d5aa22
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-bcrypt-crates.inc
@@ -0,0 +1,53 @@
+# Autogenerated with 'bitbake -c update_crates python3-bcrypt'
+
+SRC_URI += " \ 
+        crate://crates.io/autocfg/1.1.0 \ 
+        crate://crates.io/base64/0.13.0 \ 
+        crate://crates.io/bcrypt/0.13.0 \ 
+        crate://crates.io/bcrypt-pbkdf/0.8.1 \ 
+        crate://crates.io/bitflags/1.3.2 \ 
+        crate://crates.io/block-buffer/0.10.2 \ 
+        crate://crates.io/blowfish/0.9.1 \ 
+        crate://crates.io/byteorder/1.4.3 \ 
+        crate://crates.io/cfg-if/1.0.0 \ 
+        crate://crates.io/cipher/0.4.3 \ 
+        crate://crates.io/cpufeatures/0.2.4 \ 
+        crate://crates.io/crypto-common/0.1.6 \ 
+        crate://crates.io/digest/0.10.3 \ 
+        crate://crates.io/generic-array/0.14.6 \ 
+        crate://crates.io/getrandom/0.2.7 \ 
+        crate://crates.io/indoc/0.3.6 \ 
+        crate://crates.io/indoc-impl/0.3.6 \ 
+        crate://crates.io/inout/0.1.3 \ 
+        crate://crates.io/instant/0.1.12 \ 
+        crate://crates.io/libc/0.2.132 \ 
+        crate://crates.io/lock_api/0.4.7 \ 
+        crate://crates.io/once_cell/1.13.1 \ 
+        crate://crates.io/parking_lot/0.11.2 \ 
+        crate://crates.io/parking_lot_core/0.8.5 \ 
+        crate://crates.io/paste/0.1.18 \ 
+        crate://crates.io/paste-impl/0.1.18 \ 
+        crate://crates.io/pbkdf2/0.10.1 \ 
+        crate://crates.io/proc-macro-hack/0.5.19 \ 
+        crate://crates.io/proc-macro2/1.0.43 \ 
+        crate://crates.io/pyo3/0.15.2 \ 
+        crate://crates.io/pyo3-build-config/0.15.2 \ 
+        crate://crates.io/pyo3-macros/0.15.2 \ 
+        crate://crates.io/pyo3-macros-backend/0.15.2 \ 
+        crate://crates.io/quote/1.0.21 \ 
+        crate://crates.io/redox_syscall/0.2.16 \ 
+        crate://crates.io/scopeguard/1.1.0 \ 
+        crate://crates.io/sha2/0.10.2 \ 
+        crate://crates.io/smallvec/1.9.0 \ 
+        crate://crates.io/subtle/2.4.1 \ 
+        crate://crates.io/syn/1.0.99 \ 
+        crate://crates.io/typenum/1.15.0 \ 
+        crate://crates.io/unicode-ident/1.0.3 \ 
+        crate://crates.io/unindent/0.1.10 \ 
+        crate://crates.io/version_check/0.9.4 \ 
+        crate://crates.io/wasi/0.11.0+wasi-snapshot-preview1 \ 
+        crate://crates.io/winapi/0.3.9 \ 
+        crate://crates.io/winapi-i686-pc-windows-gnu/0.4.0 \ 
+        crate://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0 \ 
+        crate://crates.io/zeroize/1.5.7 \ 
+"
diff --git a/meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb b/meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb
similarity index 76%
rename from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb
rename to meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb
index 54070e4b9c..857b38df2c 100644
--- a/meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb
+++ b/meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb
@@ -5,14 +5,16 @@ HOMEPAGE = "https://pypi.org/project/bcrypt/"
 
 DEPENDS += "${PYTHON_PN}-cffi-native"
 
-SRC_URI[sha256sum] = "433c410c2177057705da2a9f2cd01dd157493b2a7ac14c8593a16b3dab6b6bfb"
+SRC_URI[sha256sum] = "c59c170fc9225faad04dde1ba61d85b413946e8ce2e5f5f5ff30dfd67283f319"
 
-inherit pypi python_setuptools_build_meta ptest
+inherit pypi python_setuptools3_rust ptest cargo-update-recipe-crates
 
 SRC_URI += " \
 	file://run-ptest \
 "
 
+require ${BPN}-crates.inc
+
 RDEPENDS:${PN}-ptest += " \
 	${PYTHON_PN}-pytest \
 "
-- 
2.30.2

RE: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Alexander Kanavin
> Sent: den 30 september 2022 19:55
> To: openembedded-core@lists.openembedded.org
> Cc: Alexander Kanavin <alex@linutronix.de>
> Subject: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0
> 
> The component has been reimplemented in rust, and comes
> with a large list of dependencies in Cargo.toml/Cargo.lock.
> 
> Rather than list them by hand, use a file generated with
> cargo-update-recipe-crates class.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  .../python/python3-bcrypt-crates.inc          | 53 +++++++++++++++++++
>  ...crypt_3.2.2.bb => python3-bcrypt_4.0.0.bb} |  6 ++-
>  2 files changed, 57 insertions(+), 2 deletions(-)
>  create mode 100644 meta/recipes-devtools/python/python3-bcrypt-crates.inc
>  rename meta/recipes-devtools/python/{python3-bcrypt_3.2.2.bb => python3-bcrypt_4.0.0.bb} (76%)
> 
> diff --git a/meta/recipes-devtools/python/python3-bcrypt-crates.inc b/meta/recipes-devtools/python/python3-bcrypt-crates.inc
> new file mode 100644
> index 0000000000..78c5d5aa22
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python3-bcrypt-crates.inc
> @@ -0,0 +1,53 @@
> +# Autogenerated with 'bitbake -c update_crates python3-bcrypt'
> +
> +SRC_URI += " \
> +        crate://crates.io/autocfg/1.1.0 \
> +        crate://crates.io/base64/0.13.0 \
> +        crate://crates.io/bcrypt/0.13.0 \
> +        crate://crates.io/bcrypt-pbkdf/0.8.1 \
> +        crate://crates.io/bitflags/1.3.2 \
> +        crate://crates.io/block-buffer/0.10.2 \
> +        crate://crates.io/blowfish/0.9.1 \
> +        crate://crates.io/byteorder/1.4.3 \
> +        crate://crates.io/cfg-if/1.0.0 \
> +        crate://crates.io/cipher/0.4.3 \
> +        crate://crates.io/cpufeatures/0.2.4 \
> +        crate://crates.io/crypto-common/0.1.6 \
> +        crate://crates.io/digest/0.10.3 \
> +        crate://crates.io/generic-array/0.14.6 \
> +        crate://crates.io/getrandom/0.2.7 \
> +        crate://crates.io/indoc/0.3.6 \
> +        crate://crates.io/indoc-impl/0.3.6 \
> +        crate://crates.io/inout/0.1.3 \
> +        crate://crates.io/instant/0.1.12 \
> +        crate://crates.io/libc/0.2.132 \
> +        crate://crates.io/lock_api/0.4.7 \
> +        crate://crates.io/once_cell/1.13.1 \
> +        crate://crates.io/parking_lot/0.11.2 \
> +        crate://crates.io/parking_lot_core/0.8.5 \
> +        crate://crates.io/paste/0.1.18 \
> +        crate://crates.io/paste-impl/0.1.18 \
> +        crate://crates.io/pbkdf2/0.10.1 \
> +        crate://crates.io/proc-macro-hack/0.5.19 \
> +        crate://crates.io/proc-macro2/1.0.43 \
> +        crate://crates.io/pyo3/0.15.2 \
> +        crate://crates.io/pyo3-build-config/0.15.2 \
> +        crate://crates.io/pyo3-macros/0.15.2 \
> +        crate://crates.io/pyo3-macros-backend/0.15.2 \
> +        crate://crates.io/quote/1.0.21 \
> +        crate://crates.io/redox_syscall/0.2.16 \
> +        crate://crates.io/scopeguard/1.1.0 \
> +        crate://crates.io/sha2/0.10.2 \
> +        crate://crates.io/smallvec/1.9.0 \
> +        crate://crates.io/subtle/2.4.1 \
> +        crate://crates.io/syn/1.0.99 \
> +        crate://crates.io/typenum/1.15.0 \
> +        crate://crates.io/unicode-ident/1.0.3 \
> +        crate://crates.io/unindent/0.1.10 \
> +        crate://crates.io/version_check/0.9.4 \
> +        crate://crates.io/wasi/0.11.0+wasi-snapshot-preview1 \
> +        crate://crates.io/winapi/0.3.9 \
> +        crate://crates.io/winapi-i686-pc-windows-gnu/0.4.0 \
> +        crate://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0 \
> +        crate://crates.io/zeroize/1.5.7 \
> +"
> diff --git a/meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb b/meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb
> similarity index 76%
> rename from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb
> rename to meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb
> index 54070e4b9c..857b38df2c 100644
> --- a/meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb
> +++ b/meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb
> @@ -5,14 +5,16 @@ HOMEPAGE = "https://pypi.org/project/bcrypt/"
> 
>  DEPENDS += "${PYTHON_PN}-cffi-native"
> 
> -SRC_URI[sha256sum] = "433c410c2177057705da2a9f2cd01dd157493b2a7ac14c8593a16b3dab6b6bfb"
> +SRC_URI[sha256sum] = "c59c170fc9225faad04dde1ba61d85b413946e8ce2e5f5f5ff30dfd67283f319"
> 
> -inherit pypi python_setuptools_build_meta ptest
> +inherit pypi python_setuptools3_rust ptest cargo-update-recipe-crates
> 
>  SRC_URI += " \
>  	file://run-ptest \
>  "
> 
> +require ${BPN}-crates.inc

Would it make sense to instead use:

require ${BP}-crates.inc

i.e., make the crates.inc file versioned? That would help to make sure one 
does not forget to update the versions of the crates when the version of 
the recipe is updated. (My assumption is that with the huge lists of crates 
for a typical rust application, there will most likely always be updates to 
them when the application is updated.)

> +
>  RDEPENDS:${PN}-ptest += " \
>  	${PYTHON_PN}-pytest \
>  "
> --
> 2.30.2

//Peter

Re: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 5770 bytes --]

Any breakage would quickly be exposed in do_compile as cargo will fail
against mismatching lock file. I usually do the opposite and remove version
numbers everywhere except .bb, as it tends to cause trouble with automated
updates.

Alex

On Sat 1. Oct 2022 at 14.05, Peter Kjellerstedt <peter.kjellerstedt@axis.com>
wrote:

> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <
> openembedded-core@lists.openembedded.org> On Behalf Of Alexander Kanavin
> > Sent: den 30 september 2022 19:55
> > To: openembedded-core@lists.openembedded.org
> > Cc: Alexander Kanavin <alex@linutronix.de>
> > Subject: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0
> >
> > The component has been reimplemented in rust, and comes
> > with a large list of dependencies in Cargo.toml/Cargo.lock.
> >
> > Rather than list them by hand, use a file generated with
> > cargo-update-recipe-crates class.
> >
> > Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> > ---
> >  .../python/python3-bcrypt-crates.inc          | 53 +++++++++++++++++++
> >  ...crypt_3.2.2.bb => python3-bcrypt_4.0.0.bb} |  6 ++-
> >  2 files changed, 57 insertions(+), 2 deletions(-)
> >  create mode 100644
> meta/recipes-devtools/python/python3-bcrypt-crates.inc
> >  rename meta/recipes-devtools/python/{python3-bcrypt_3.2.2.bb =>
> python3-bcrypt_4.0.0.bb} (76%)
> >
> > diff --git a/meta/recipes-devtools/python/python3-bcrypt-crates.inc
> b/meta/recipes-devtools/python/python3-bcrypt-crates.inc
> > new file mode 100644
> > index 0000000000..78c5d5aa22
> > --- /dev/null
> > +++ b/meta/recipes-devtools/python/python3-bcrypt-crates.inc
> > @@ -0,0 +1,53 @@
> > +# Autogenerated with 'bitbake -c update_crates python3-bcrypt'
> > +
> > +SRC_URI += " \
> > +        crate://crates.io/autocfg/1.1.0 \
> > +        crate://crates.io/base64/0.13.0 \
> > +        crate://crates.io/bcrypt/0.13.0 \
> > +        crate://crates.io/bcrypt-pbkdf/0.8.1 \
> > +        crate://crates.io/bitflags/1.3.2 \
> > +        crate://crates.io/block-buffer/0.10.2 \
> > +        crate://crates.io/blowfish/0.9.1 \
> > +        crate://crates.io/byteorder/1.4.3 \
> > +        crate://crates.io/cfg-if/1.0.0 \
> > +        crate://crates.io/cipher/0.4.3 \
> > +        crate://crates.io/cpufeatures/0.2.4 \
> > +        crate://crates.io/crypto-common/0.1.6 \
> > +        crate://crates.io/digest/0.10.3 \
> > +        crate://crates.io/generic-array/0.14.6 \
> > +        crate://crates.io/getrandom/0.2.7 \
> > +        crate://crates.io/indoc/0.3.6 \
> > +        crate://crates.io/indoc-impl/0.3.6 \
> > +        crate://crates.io/inout/0.1.3 \
> > +        crate://crates.io/instant/0.1.12 \
> > +        crate://crates.io/libc/0.2.132 \
> > +        crate://crates.io/lock_api/0.4.7 \
> > +        crate://crates.io/once_cell/1.13.1 \
> > +        crate://crates.io/parking_lot/0.11.2 \
> > +        crate://crates.io/parking_lot_core/0.8.5 \
> > +        crate://crates.io/paste/0.1.18 \
> > +        crate://crates.io/paste-impl/0.1.18 \
> > +        crate://crates.io/pbkdf2/0.10.1 \
> > +        crate://crates.io/proc-macro-hack/0.5.19 \
> > +        crate://crates.io/proc-macro2/1.0.43 \
> > +        crate://crates.io/pyo3/0.15.2 \
> > +        crate://crates.io/pyo3-build-config/0.15.2 \
> > +        crate://crates.io/pyo3-macros/0.15.2 \
> > +        crate://crates.io/pyo3-macros-backend/0.15.2 \
> > +        crate://crates.io/quote/1.0.21 \
> > +        crate://crates.io/redox_syscall/0.2.16 \
> > +        crate://crates.io/scopeguard/1.1.0 \
> > +        crate://crates.io/sha2/0.10.2 \
> > +        crate://crates.io/smallvec/1.9.0 \
> > +        crate://crates.io/subtle/2.4.1 \
> > +        crate://crates.io/syn/1.0.99 \
> > +        crate://crates.io/typenum/1.15.0 \
> > +        crate://crates.io/unicode-ident/1.0.3 \
> > +        crate://crates.io/unindent/0.1.10 \
> > +        crate://crates.io/version_check/0.9.4 \
> > +        crate://crates.io/wasi/0.11.0+wasi-snapshot-preview1 \
> > +        crate://crates.io/winapi/0.3.9 \
> > +        crate://crates.io/winapi-i686-pc-windows-gnu/0.4.0 \
> > +        crate://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0 \
> > +        crate://crates.io/zeroize/1.5.7 \
> > +"
> > diff --git a/meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb
> b/meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb
> > similarity index 76%
> > rename from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb
> > rename to meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb
> > index 54070e4b9c..857b38df2c 100644
> > --- a/meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb
> > +++ b/meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb
> > @@ -5,14 +5,16 @@ HOMEPAGE = "https://pypi.org/project/bcrypt/"
> >
> >  DEPENDS += "${PYTHON_PN}-cffi-native"
> >
> > -SRC_URI[sha256sum] =
> "433c410c2177057705da2a9f2cd01dd157493b2a7ac14c8593a16b3dab6b6bfb"
> > +SRC_URI[sha256sum] =
> "c59c170fc9225faad04dde1ba61d85b413946e8ce2e5f5f5ff30dfd67283f319"
> >
> > -inherit pypi python_setuptools_build_meta ptest
> > +inherit pypi python_setuptools3_rust ptest cargo-update-recipe-crates
> >
> >  SRC_URI += " \
> >       file://run-ptest \
> >  "
> >
> > +require ${BPN}-crates.inc
>
> Would it make sense to instead use:
>
> require ${BP}-crates.inc
>
> i.e., make the crates.inc file versioned? That would help to make sure one
> does not forget to update the versions of the crates when the version of
> the recipe is updated. (My assumption is that with the huge lists of
> crates
> for a typical rust application, there will most likely always be updates
> to
> them when the application is updated.)
>
> > +
> >  RDEPENDS:${PN}-ptest += " \
> >       ${PYTHON_PN}-pytest \
> >  "
> > --
> > 2.30.2
>
> //Peter
>
>

[-- Attachment #2: Type: text/html, Size: 12363 bytes --]

RE: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0

[Peter Kjellerstedt]

[-- Attachment #1: Type: text/plain, Size: 8335 bytes --]

Ok, as long as any problems are automatically detected, your proposed solution seems fine then.

//Peter

From: Alexander Kanavin <alex.kanavin@gmail.com>
Sent: den 1 oktober 2022 14:42
To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Cc: Alexander Kanavin <alex@linutronix.de>; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0

Any breakage would quickly be exposed in do_compile as cargo will fail against mismatching lock file. I usually do the opposite and remove version numbers everywhere except .bb, as it tends to cause trouble with automated updates.

Alex

On Sat 1. Oct 2022 at 14.05, Peter Kjellerstedt <peter.kjellerstedt@axis.com<mailto:peter.kjellerstedt@axis.com>> wrote:
> -----Original Message-----
> From: openembedded-core@lists.openembedded.org<mailto:openembedded-core@lists.openembedded.org> <openembedded-core@lists.openembedded.org<mailto:openembedded-core@lists.openembedded.org>> On Behalf Of Alexander Kanavin
> Sent: den 30 september 2022 19:55
> To: openembedded-core@lists.openembedded.org<mailto:openembedded-core@lists.openembedded.org>
> Cc: Alexander Kanavin <alex@linutronix.de<mailto:alex@linutronix.de>>
> Subject: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0
>
> The component has been reimplemented in rust, and comes
> with a large list of dependencies in Cargo.toml/Cargo.lock.
>
> Rather than list them by hand, use a file generated with
> cargo-update-recipe-crates class.
>
> Signed-off-by: Alexander Kanavin <alex@linutronix.de<mailto:alex@linutronix.de>>
> ---
>  .../python/python3-bcrypt-crates.inc          | 53 +++++++++++++++++++
>  ...crypt_3.2.2.bb<http://crypt_3.2.2.bb> => python3-bcrypt_4.0.0.bb<http://python3-bcrypt_4.0.0.bb>} |  6 ++-
>  2 files changed, 57 insertions(+), 2 deletions(-)
>  create mode 100644 meta/recipes-devtools/python/python3-bcrypt-crates.inc
>  rename meta/recipes-devtools/python/{python3-bcrypt_3.2.2.bb<http://python3-bcrypt_3.2.2.bb> => python3-bcrypt_4.0.0.bb<http://python3-bcrypt_4.0.0.bb>} (76%)
>
> diff --git a/meta/recipes-devtools/python/python3-bcrypt-crates.inc b/meta/recipes-devtools/python/python3-bcrypt-crates.inc
> new file mode 100644
> index 0000000000..78c5d5aa22
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python3-bcrypt-crates.inc
> @@ -0,0 +1,53 @@
> +# Autogenerated with 'bitbake -c update_crates python3-bcrypt'
> +
> +SRC_URI += " \
> +        crate://crates.io/autocfg/1.1.0<http://crates.io/autocfg/1.1.0> \
> +        crate://crates.io/base64/0.13.0<http://crates.io/base64/0.13.0> \
> +        crate://crates.io/bcrypt/0.13.0<http://crates.io/bcrypt/0.13.0> \
> +        crate://crates.io/bcrypt-pbkdf/0.8.1<http://crates.io/bcrypt-pbkdf/0.8.1> \
> +        crate://crates.io/bitflags/1.3.2<http://crates.io/bitflags/1.3.2> \
> +        crate://crates.io/block-buffer/0.10.2<http://crates.io/block-buffer/0.10.2> \
> +        crate://crates.io/blowfish/0.9.1<http://crates.io/blowfish/0.9.1> \
> +        crate://crates.io/byteorder/1.4.3<http://crates.io/byteorder/1.4.3> \
> +        crate://crates.io/cfg-if/1.0.0<http://crates.io/cfg-if/1.0.0> \
> +        crate://crates.io/cipher/0.4.3<http://crates.io/cipher/0.4.3> \
> +        crate://crates.io/cpufeatures/0.2.4<http://crates.io/cpufeatures/0.2.4> \
> +        crate://crates.io/crypto-common/0.1.6<http://crates.io/crypto-common/0.1.6> \
> +        crate://crates.io/digest/0.10.3<http://crates.io/digest/0.10.3> \
> +        crate://crates.io/generic-array/0.14.6<http://crates.io/generic-array/0.14.6> \
> +        crate://crates.io/getrandom/0.2.7<http://crates.io/getrandom/0.2.7> \
> +        crate://crates.io/indoc/0.3.6<http://crates.io/indoc/0.3.6> \
> +        crate://crates.io/indoc-impl/0.3.6<http://crates.io/indoc-impl/0.3.6> \
> +        crate://crates.io/inout/0.1.3<http://crates.io/inout/0.1.3> \
> +        crate://crates.io/instant/0.1.12<http://crates.io/instant/0.1.12> \
> +        crate://crates.io/libc/0.2.132<http://crates.io/libc/0.2.132> \
> +        crate://crates.io/lock_api/0.4.7<http://crates.io/lock_api/0.4.7> \
> +        crate://crates.io/once_cell/1.13.1<http://crates.io/once_cell/1.13.1> \
> +        crate://crates.io/parking_lot/0.11.2<http://crates.io/parking_lot/0.11.2> \
> +        crate://crates.io/parking_lot_core/0.8.5<http://crates.io/parking_lot_core/0.8.5> \
> +        crate://crates.io/paste/0.1.18<http://crates.io/paste/0.1.18> \
> +        crate://crates.io/paste-impl/0.1.18<http://crates.io/paste-impl/0.1.18> \
> +        crate://crates.io/pbkdf2/0.10.1<http://crates.io/pbkdf2/0.10.1> \
> +        crate://crates.io/proc-macro-hack/0.5.19<http://crates.io/proc-macro-hack/0.5.19> \
> +        crate://crates.io/proc-macro2/1.0.43<http://crates.io/proc-macro2/1.0.43> \
> +        crate://crates.io/pyo3/0.15.2<http://crates.io/pyo3/0.15.2> \
> +        crate://crates.io/pyo3-build-config/0.15.2<http://crates.io/pyo3-build-config/0.15.2> \
> +        crate://crates.io/pyo3-macros/0.15.2<http://crates.io/pyo3-macros/0.15.2> \
> +        crate://crates.io/pyo3-macros-backend/0.15.2<http://crates.io/pyo3-macros-backend/0.15.2> \
> +        crate://crates.io/quote/1.0.21<http://crates.io/quote/1.0.21> \
> +        crate://crates.io/redox_syscall/0.2.16<http://crates.io/redox_syscall/0.2.16> \
> +        crate://crates.io/scopeguard/1.1.0<http://crates.io/scopeguard/1.1.0> \
> +        crate://crates.io/sha2/0.10.2<http://crates.io/sha2/0.10.2> \
> +        crate://crates.io/smallvec/1.9.0<http://crates.io/smallvec/1.9.0> \
> +        crate://crates.io/subtle/2.4.1<http://crates.io/subtle/2.4.1> \
> +        crate://crates.io/syn/1.0.99<http://crates.io/syn/1.0.99> \
> +        crate://crates.io/typenum/1.15.0<http://crates.io/typenum/1.15.0> \
> +        crate://crates.io/unicode-ident/1.0.3<http://crates.io/unicode-ident/1.0.3> \
> +        crate://crates.io/unindent/0.1.10<http://crates.io/unindent/0.1.10> \
> +        crate://crates.io/version_check/0.9.4<http://crates.io/version_check/0.9.4> \
> +        crate://crates.io/wasi/0.11.0+wasi-snapshot-preview1<http://crates.io/wasi/0.11.0+wasi-snapshot-preview1> \
> +        crate://crates.io/winapi/0.3.9<http://crates.io/winapi/0.3.9> \
> +        crate://crates.io/winapi-i686-pc-windows-gnu/0.4.0<http://crates.io/winapi-i686-pc-windows-gnu/0.4.0> \
> +        crate://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0<http://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0> \
> +        crate://crates.io/zeroize/1.5.7<http://crates.io/zeroize/1.5.7> \
> +"
> diff --git a/meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb<http://python3-bcrypt_3.2.2.bb> b/meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb<http://python3-bcrypt_4.0.0.bb>
> similarity index 76%
> rename from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb<http://python3-bcrypt_3.2.2.bb>
> rename to meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb<http://python3-bcrypt_4.0.0.bb>
> index 54070e4b9c..857b38df2c 100644
> --- a/meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb<http://python3-bcrypt_3.2.2.bb>
> +++ b/meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb<http://python3-bcrypt_4.0.0.bb>
> @@ -5,14 +5,16 @@ HOMEPAGE = "https://pypi.org/project/bcrypt/"
>
>  DEPENDS += "${PYTHON_PN}-cffi-native"
>
> -SRC_URI[sha256sum] = "433c410c2177057705da2a9f2cd01dd157493b2a7ac14c8593a16b3dab6b6bfb"
> +SRC_URI[sha256sum] = "c59c170fc9225faad04dde1ba61d85b413946e8ce2e5f5f5ff30dfd67283f319"
>
> -inherit pypi python_setuptools_build_meta ptest
> +inherit pypi python_setuptools3_rust ptest cargo-update-recipe-crates
>
>  SRC_URI += " \
>       file://run-ptest \
>  "
>
> +require ${BPN}-crates.inc

Would it make sense to instead use:

require ${BP}-crates.inc

i.e., make the crates.inc file versioned? That would help to make sure one
does not forget to update the versions of the crates when the version of
the recipe is updated. (My assumption is that with the huge lists of crates
for a typical rust application, there will most likely always be updates to
them when the application is updated.)

> +
>  RDEPENDS:${PN}-ptest += " \
>       ${PYTHON_PN}-pytest \
>  "
> --
> 2.30.2

//Peter

[-- Attachment #2: Type: text/html, Size: 15620 bytes --]

Re: [OE-core] [RFC PATCH 1/2] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock

[Quentin Schulz]

Hi Alex,

On 9/30/22 19:55, Alexander Kanavin wrote:
> For better or worse, more and more rust components are appearing that do
> not include their dependencies in tarballs (or git trees), and rely on cargo
> to fetch them. On the other hand, bitbake does not use cargo (and quite possible
> won't ever be able to), and relies on having each item explicitly listed in SRC_URI
> with a crate:// prefix. This however creates a problem of both making such lists in
> the first place and updating them when a recipe is updated to a newer version.
> 
> So this class can be used to perform such updates by implementing a task that does it;
> the next commit shows the outcome for python3-bcrypt (which has been tested to work
> and produce a successful build).
> 
> Note: the python script relies on tomllib library, which appears in Python 3.11 and
> does not exist in earlier versions - I've tested this by first updating python to 3.11-rc2
> in oe-core.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>   .../cargo-update-recipe-crates.bbclass        | 41 +++++++++++++++++++
>   1 file changed, 41 insertions(+)
>   create mode 100644 meta/classes-recipe/cargo-update-recipe-crates.bbclass
> 
> diff --git a/meta/classes-recipe/cargo-update-recipe-crates.bbclass b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
> new file mode 100644
> index 0000000000..f90938c734
> --- /dev/null
> +++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
> @@ -0,0 +1,41 @@
> +#
> +# Copyright OpenEmbedded Contributors
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +##
> +## Purpose:
> +## This class is used to update the list of crates in SRC_URI
> +## by reading Cargo.lock in the source tree.
> +##
> +## See meta/recipes-devtools/python/python3-bcrypt_*.bb for an example
> +##
> +## To perform the update: bitbake -c update_crates recipe-name
> +
> +addtask do_update_crates after do_patch
> +do_update_crates[depends] = "python3-native:do_populate_sysroot"
> +
> +do_update_crates() {
> +    nativepython3 - <<EOF
> +
> +def get_crates(f):
> +    import tomllib
> +    c_list = 'SRC_URI += " \\ \n'
> +    crates = tomllib.load(open(f, 'rb'))

Aren't we supposed to use context for this or close the file descriptor 
after opening it manually?

e.g.
with open(f, 'rb') as file:
     crates = tommlib.load(file)

?

> +    for c in crates['package']:
> +        if 'source' in c and 'crates.io' in c['source']:

if 'crates.io' in c.get('source', '')

> +            c_list += "        crate://crates.io/{}/{} \\ \n".format(c['name'], c['version'])
> +    c_list += '"\n'
> +    return c_list
> +
> +import os
> +crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
> +for root, dirs, files in os.walk('${S}'):
> +    for file in files:
> +        if file == 'Cargo.lock':
> +            crates += get_crates(os.path.join(root, file))

from pathlib import Path

for file in Path('${S}').glob('**/Cargo.lock'):
     crates += get_crates(str(file))


> +open(os.path.join('${THISDIR}', '${PN}'+"-crates.inc"), 'w').write(crates)
> +

with open... here again?

Cheers,
Quentin

Re: [OE-core] [RFC PATCH 1/2] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock

[Alexander Kanavin]

On Mon, 3 Oct 2022 at 10:09, Quentin Schulz
<quentin.schulz@theobroma-systems.com> wrote:
> > +def get_crates(f):
> > +    import tomllib
> > +    c_list = 'SRC_URI += " \\ \n'
> > +    crates = tomllib.load(open(f, 'rb'))
>
> Aren't we supposed to use context for this or close the file descriptor
> after opening it manually?
>
> e.g.
> with open(f, 'rb') as file:
>      crates = tommlib.load(file)
>
> ?

python3.11 no longer prints warnings about this, so I assume it is now ok.

For the other two suggested changes please explain why.


Alex

Re: [OE-core] [RFC PATCH 1/2] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock

[Chuck Wolber]

[-- Attachment #1: Type: text/plain, Size: 650 bytes --]

On Fri, Sep 30, 2022 at 10:55 AM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

%> Snip %<


> On the other hand, bitbake does not use cargo (and quite possible
> won't ever be able to),
>

Can you elaborate on this?

I only ask because we are using this[1] and this[2] to build this[3] recipe
for hardknott.

1. https://github.com/meta-rust/meta-rust/blob/master/lib/crate.py
2. https://crates.io/crates/cargo-bitbake
3.
https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3-cryptography-vectors_36.0.2.bb?h=kirkstone

..Ch:W..

-- 
*"Perfection must be reached by degrees; she requires the slow hand of
time." - Voltaire*

[-- Attachment #2: Type: text/html, Size: 1505 bytes --]

Re: [OE-core] [RFC PATCH 1/2] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock

[Alexander Kanavin]

On Wed, 5 Oct 2022 at 07:32, Chuck Wolber <chuckwolber@gmail.com> wrote:
> Can you elaborate on this?
>
> I only ask because we are using this[1] and this[2] to build this[3] recipe for hardknott.
>
> 1. https://github.com/meta-rust/meta-rust/blob/master/lib/crate.py
> 2. https://crates.io/crates/cargo-bitbake
> 3. https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3-cryptography-vectors_36.0.2.bb?h=kirkstone

As you can see in [1], crate:// fetcher is not using cargo, it's using
wget - so the question of resolving and listing dependencies from
Cargo.lock falls on the tooling that writes and updates the recipe
itself. cargo-bitbake failed at the installation step for me (both
with --locked and without it), and in general I do not think we should
rely on external tools with unknown maintenance status. The lists of
open issues and pull requests are not at all reassuring:
https://github.com/meta-rust/cargo-bitbake

Alex

Re: [OE-core] [RFC PATCH 1/2] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 1993 bytes --]

On Wed, Oct 5, 2022 at 1:56 AM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> On Wed, 5 Oct 2022 at 07:32, Chuck Wolber <chuckwolber@gmail.com> wrote:
> > Can you elaborate on this?
> >
> > I only ask because we are using this[1] and this[2] to build this[3]
> recipe for hardknott.
> >
> > 1. https://github.com/meta-rust/meta-rust/blob/master/lib/crate.py
> > 2. https://crates.io/crates/cargo-bitbake
> > 3.
> https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3-cryptography-vectors_36.0.2.bb?h=kirkstone
>
> As you can see in [1], crate:// fetcher is not using cargo, it's using
> wget - so the question of resolving and listing dependencies from
> Cargo.lock falls on the tooling that writes and updates the recipe
> itself. cargo-bitbake failed at the installation step for me (both
> with --locked and without it), and in general I do not think we should
> rely on external tools with unknown maintenance status. The lists of
> open issues and pull requests are not at all reassuring:
> https://github.com/meta-rust/cargo-bitbake
>
>
I have cargo bitbake running locally, but in recent python3-cryptography
and python3-bcrypt upgrades
it produced crate://* SRC_URI that actually failed to build. I had to
manually loop through builds until
I sussed out all the versions that would work. This proves Alex's point.

Having looked at the implementation, I am very much in favor of this
proposal. In fact, I wonder if
a similar approach could help with golang and maybe even nodejs/npm. But
that is for another day.

Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#171453):
> https://lists.openembedded.org/g/openembedded-core/message/171453
> Mute This Topic: https://lists.openembedded.org/mt/94022674/924729
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 3609 bytes --]

[PATCH 3/3] cargo-update-recipe-crates: small improvements

[Martin Jansa]

* use 4 spaces
* avoid trailing space
* add CARGO_LOCK_SRC_DIR to allow searching outside ${S}
* use BPN in output filename
* First I've used CARGO_LOCK_SRC_DIR as relative to ${S}, because that's what CARGO_SRC_DIR
  in cargo.bbclass is using:
  meta/classes-recipe/cargo.bbclass:CARGO_SRC_DIR ??= ""
  meta/classes-recipe/cargo.bbclass:MANIFEST_PATH ??= "${S}/${CARGO_SRC_DIR}/Cargo.toml"

  but change to absolute path (defaulting to ${S}) as requested by Alex:
  11:44 < kanavin> JaMa, would prefer CARGO_LOCK_SRC_DIR ??= "${S}"
  11:46 < kanavin> otherwise looks good

* I've resolved my usecase for CARGO_LOCK_SRC_DIR by changing
  S back to ${WORKDIR}/git and using CARGO_SRC_DIR to select
  the right subdirectory to be built, because the Cargo.toml
  in this subdirectory was also referencing other subdirectories
  with relative path:
  https://github.com/solana-labs/solana-program-library/blob/88b147506d5b9515f3a4762421a0b8c309188dc9/token/cli/Cargo.toml#L30
  so including all Cargo.lock files in whole ${WORKDIR}/git
  seems like reasonable approach

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 .../cargo-update-recipe-crates.bbclass             | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/meta/classes-recipe/cargo-update-recipe-crates.bbclass b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
index f90938c734..3a12ba247d 100644
--- a/meta/classes-recipe/cargo-update-recipe-crates.bbclass
+++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
@@ -16,26 +16,30 @@
 addtask do_update_crates after do_patch
 do_update_crates[depends] = "python3-native:do_populate_sysroot"
 
+# The directory where to search for Cargo.lock files
+CARGO_LOCK_SRC_DIR ??= "${S}"
+
 do_update_crates() {
     nativepython3 - <<EOF
 
 def get_crates(f):
     import tomllib
-    c_list = 'SRC_URI += " \\ \n'
+    c_list = '# from %s' % os.path.relpath(f, '${CARGO_LOCK_SRC_DIR}')
+    c_list += '\nSRC_URI += " \\\'
     crates = tomllib.load(open(f, 'rb'))
     for c in crates['package']:
         if 'source' in c and 'crates.io' in c['source']:
-            c_list += "        crate://crates.io/{}/{} \\ \n".format(c['name'], c['version'])
-    c_list += '"\n'
+            c_list += '\n    crate://crates.io/%s/%s \\\' % (c['name'], c['version'])
+    c_list += '\n"\n'
     return c_list
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-for root, dirs, files in os.walk('${S}'):
+for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
     for file in files:
         if file == 'Cargo.lock':
             crates += get_crates(os.path.join(root, file))
-open(os.path.join('${THISDIR}', '${PN}'+"-crates.inc"), 'w').write(crates)
+open(os.path.join('${THISDIR}', '${BPN}'+"-crates.inc"), 'w').write(crates)
 
 EOF
 }
-- 
2.38.1