By the way, about this "chickens and eggs" problem, isn't it the same for a regular recipe you just upgraded?
I don't know, I haven't upgraded a recipe with crates yet :)
It's just this time I was aware that the .inc file will need to be refreshed and it wasn't obvious at first that easiest way to avoid do_fetch issue blocking .inc refresh is to remove all crates first by emptying .inc file.
Personnaly, when I update a recipe, I let bitbake tell me what is the new checksum expected and put it the recipe (as the error message says).
For regular recipes with a few checksum I do the same. But it doesn't scale here as you mention bellow.
as it reported only the first wrong checksum from MANY.
But I'm aware this is not exactly the same, since a cargo based recipe could contain a ton of crate:// uri and if you apply this method, you have to copy the new checksum one by one, bitbake error after another...
Don't know how to make this better and I plead guilty for not mentioning that in a dedicated commit message.
Better late than never, you can include it in v6 of this patch (I have read the git log of the .bbclass before going to refresh the .inc files, so others might notice it there as well).
Of course, if someone come up with a smoothier way of doing this, I'll make a new patch.
I think I found what the other issue with arrayvec create was:
meta-webos/recipes-upstreamable/solana/solana-crates.inc: crate://
crates.io/arrayvec/0.7.2 \
meta-webos/recipes-upstreamable/solana/solana-crates.inc:SRC_URI[arrayvec.sha256sum] = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"
meta-webos/recipes-upstreamable/solana/solana-crates.inc: crate://
crates.io/arrayvec/0.7.1 \
meta-webos/recipes-upstreamable/solana/solana-crates.inc:SRC_URI[arrayvec.sha256sum] = "be4dc07131ffa69b8072d35f5007352af944213cde02545e2103680baed38fcd"
meta-webos/recipes-upstreamable/sugar/sugar-crates.inc: crate://
crates.io/arrayvec/0.7.2 \
meta-webos/recipes-upstreamable/sugar/sugar-crates.inc:SRC_URI[arrayvec.sha256sum] = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"
So different Cargo.lock files included in the same ${S} have different version, but in this case it didn't set different name parameter, probably because different Cargo.lock files are processed separately?
Thanks
Well I see what you mean, I'll take a look at your example to try to find out if multiple Cargo.lock could be expected.
And for your second remark, yes, there is a chicken and eggs issue for updating crates checksum from scratch.
You didn't miss anything, I came across this when updating crates checksum that was "pristine"
and I only manage to execute update_crates by locally patching bitbake/lib/bb/fetch2/crate.py for
recommends_checksum method to return False .
I'm aware that is not ideal but I don't know how to make this better (maybe make do_update_crates
after do_fetch instead of after do_patch ? )
I don't remember the exact details now, but when I was working on updating solana recipes to use this
the S was pointing to just some subdirectory and multiple Cargo.locks files were parsed in other directories as well, that's why I was adding
in the end I've changed to use the root directory in S and just set CARGO_SRC_DIR to right subdirectory to build.
I don't have much experience with crates other than these solana recipes, but isn't there some valid use-case for multiple Cargo.locks?
I assume Alex in original implementation didn't use the os.walk just to make it more complicated :).
And FWIW when trying to regenerate these .inc files with current master and with:
$ bitbake -c update_crates solana-keygen
it fails with:
ERROR: solana-keygen-1.14.5-r0 do_fetch: No checksum specified for '/OE/lge/build/webos/mickledore/downloads/Inflector-0.11.4.crate', please add at least one to the recipe:
SRC_URI[Inflector.sha256sum] = "fe438c63458706e03479442743baae6c88256498e6431708f6dfc520a26515d3"
ERROR: solana-keygen-1.14.5-r0 do_fetch: Bitbake Fetcher Error: NoChecksumError('Missing SRC_URI checksum', '
https://crates.io/api/v1/crates/Inflector/0.11.4/download')
Isn't it an chicken&egg issue now when do_fetch enforces checksums to be used? Or am I just missing some of the pending changes or just using it incorrectly?
On Thu, Mar 30, 2023 at 4:45 PM <frederic.martinsons@gmail.com> wrote:
>
> From: Frederic Martinsons <frederic.martinsons@gmail.com>
>
> There is no need to do such things, Cargo.lock file
> has to be at the root of CARGO_LOCK_SRC_DIR.
> This avoid finding other possible Cargo.lock that
> would be in subdir (for example if a patch is applied
> on the recipe, we can have .pc subdir in S and a Cargo.lock
> can be there)
>
> Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
> ---
> .../cargo-update-recipe-crates.bbclass | 12 ++++++++----
> 1 file changed, 8 insertions(+), 4 deletions(-)
>
> diff --git a/meta/classes-recipe/cargo-update-recipe-crates.bbclass b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
> index daa363b0dd..549cfe627e 100644
> --- a/meta/classes-recipe/cargo-update-recipe-crates.bbclass
> +++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
> @@ -68,10 +68,14 @@ def get_crates(f):
> import os
> crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
> found = False
> -for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
> - for file in files:
> - if file == 'Cargo.lock':
> - crates += get_crates(os.path.join(root, file))
> +for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):
> + if file == 'Cargo.lock':
> + try:
> + cargo_lock_path = os.path.join('${CARGO_LOCK_SRC_DIR}', file)
> + crates += get_crates(cargo_lock_path)
> + except Exception as e:
> + raise ValueError("Cannot parse '%s'" % cargo_lock_path) from e
> + else:
> found = True
> if not found:
> raise ValueError("Unable to find Cargo.lock in ${CARGO_LOCK_SRC_DIR}")
Isn't this just a long-winded version of something like this
(completely untested):
diff --git a/meta/classes-recipe/cargo-update-recipe-crates.bbclass
b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
index daa363b0dd65..22ddcfa5c1e3 100644
--- a/meta/classes-recipe/cargo-update-recipe-crates.bbclass
+++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
@@ -67,14 +67,7 @@ def get_crates(f):
import os
crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
- for file in files:
- if file == 'Cargo.lock':
- crates += get_crates(os.path.join(root, file))
- found = True
-if not found:
- raise ValueError("Unable to find Cargo.lock in ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}", "Cargo.lock"))
open("${TARGET_FILE}", 'w').write(crates)
EOF
--
Alex Kiernan