On 2021-08-20 10:48 p.m., Randy MacLeod wrote: > On 2021-08-20 6:06 a.m., Randy MacLeod wrote: >> Added Trevor who may be working on upgrading python-cryptography. >> Added Anatol since he was interested last time we worked on merging. >> >> Again, let me know if you want to be dropped by replying privately. >> No one from the meta-rust community responded last time so I'm >> assuming that y'all are just on vacation! ;-) >> >> On 2021-08-17 10:52 a.m., Randy MacLeod wrote: >>> On 2021-08-13 11:22 a.m., Vinay Kumar wrote: >>>> Hi Randy, >>>> >>>>>> Vinay or I will look into glibc/qemuppc64 >>>>>> if no one else is working on that. >>>> >>>> The failure is due to the below error in libstd-rs package build. >>>> "error: unrecognized arch "powerpc64le" in target specification" >>>> >>>> The reason for this is "arch": "powerpc64le" from the below target >>>> specific file. >>>> >> >> ,,, >> >>> >>> Thanks Vinay, applied. >>> >>> I'll push a new branch to poky-contrib later today. >> >> >> I fixed and worked-around a few issues and pushed a new branch: >> >> >> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rmacleod/rust-redo-aug-17-2021-a >> >> >> 1. There are only a few new commits including Vinay's: >> >>    757da4cbf1e rust: remove Rust version 1.51.0 toolchain >> >>    a4808f787ca cargo_common: remove http_proxy >> >>    b6941135c00 rust-common.inc: Fix build failure with qemuppc64. >> >> >> If anyone wants to help by doing test builds, trying to generate and use >> >> and SDK with rust in it or anything else, that'd be great. >> >> >> >> Alex also ran a test build on the Autobuilder: >> >> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2439 >> >> >> >> but I have yet to look at the failed stages in the run. >> >> >> Testing logs and debugging notes below for those interested. >> >> ../Randy >> >> >> >> The test: >>    oe-selftest -r distrodata.Distrodata.test_maintainers >> >> >> is fixed by creating maintainer records for all rusty packages >> and by removing the older version of the rust toolchain: >>    757da4cbf1e rust: remove Rust version 1.51.0 toolchain >> >> >> >> I suppose that if the meta-rust users want to keep an older TC >> it can live in meta-rust. >> >> Then, these tests: >> >>    oe-selftest -r sstatetests.SStateTests.test_sstate_noop_samesigs >> >>    oe-selftest -r sstatetests.SStateTests.test_sstate_sametune_samesigs >> >> >> were failing. One issue was fixed by the work-around of: >> >>    a4808f787ca cargo_common: remove http_proxy >> >> >> Debugging the next step was a challenge since I hadn't worked >> with diffsigs before. Then to simplify, I removed all rusty .bb package >>    ( by mv foo.bb >> >> foo.bb.hide) >> until I was down to the first recipe in the dependency tree: >> >>    rust-llvm-native. >> >> and then running: >>   oe-selftest -r sstatetests.SStateTests.test_sstate_noop_samesigs \ >> sstatetests.SStateTests.test_sstate_sametune_samesigs \ >> >> >> >> 2021-08-19 17:54:12,878 - oe-selftest - INFO >> - Ran 2 tests in 142.010s >> >> - OK >> >> - RESULTS: >> >> - RESULTS - sstatetests.SStateTests.test_sstate_noop_samesigs: >>       PASSED (41.45s) >> >> - RESULTS - sstatetests.SStateTests.test_sstate_sametune_samesigs: >>       PASSED (98.89s) >> >> - SUMMARY: >> >> - oe-selftest () - Ran 2 tests in 142.012s >> >> - oe-selftest - OK - All required tests passed >>     (successes=2, skipped=0, failures=0, errors=0) >> >> >> adding the next self consistent group of package: >>    cargo/cargo-cross-canadian_1.54.0.bb >> >> >> >>    cargo/cargo_1.54.0.bb >> >> >> >>    rust/libstd-rs_1.54.0.bb >> >> >> >>    rust/rust-cross-canadian_1.54.0.bb >> >> >> >>    rust/rust-cross_1.54.0.bb >> >> >> >>    rust/rust-tools-cross-canadian_1.54.0.bb >> >> >> >>    rust/rust_1.54.0.bb >> >> >> >> >> >> I see another http_proxy failure that seems to require a feature >> drop in cargo: >> >> https://github.com/rust-lang/cargo/blob/master/src/cargo/ops/registry.rs#L524 >> >> >> Patching out a feature doesn't seem like a good idea but >> maybe someone else has a better one? >> >> To see if there were other problems lurking menacingly behind the proxy >> error, I disabled the http_proxy test in: >>     meta/lib/oeqa/selftest/cases/sstatetests.py >> >> >> >> >>    -http_proxy = "http://example.com/" >> >>    +http_proxy = "" >> >> >> and ran again to find: >> >> - RESULTS - sstatetests.SStateTests.test_sstate_noop_samesigs: >>       PASSED (50.63s) >> - RESULTS - sstatetests.SStateTests.test_sstate_sametune_samesigs: >>       FAILED (160.05s) >> >> so that's somewhat encouraging but there is still the error: >> >>      File ".../meta/lib/oeqa/selftest/cases/sstatetests.py", line 449, >>    in test_sstate_sametune_samesigs >> >>      self.assertCountEqual(files1, files2) >> >> AssertionError: Element counts were not equal: >> >> First has 1, Second has 0: >> '/ala-lpggp31/rmacleod/src/distro/yocto/b/aug-19-a-st/\ >> tmp-sstatesamehash/stamps/x86_64-linux/\ >> lib32-rust-cross-i686/1.54.0-r0.do_rust_gen_targets.sigdata.\ >> fc54579cc139c777172242c3da3e0' >> >> >> First has 1, Second has 0: >> '/ala-lpggp31/rmacleod/src/distro/yocto/b/aug-19-a-st/\ >> tmp-sstatesamehash/stamps/x86_64-linux/\ >> lib32-rust-cross-i686/1.54.0-r0.do_compile.sigdata.\ >> 86cb5a1346b0615cb21fab3c0110cc964ecced' >> >> >> ... lots of similar lines of diffsig info ... >> >> >> Hacking the test to not clean up and >> running bitbake-dumpsigs on the two sig files for the first error, >> I get the two files attached. I also attached the >>    tmp/log/oe-selftest-results.log file. >> >> That's it for now. > > > I think I've gotten to the bottom of a chain of sigs that differ in > part due to: > "Hash for dependent task" differing and found that on leaf node is: > > stamps/x86_64-linux/rust-cross-i686/1.54.0-r0.do_rust_gen_target > > See below for my notes and my assertion/question about the next step > needing > to be deterministically ordering the elements used in rust_gen_target. > > ../Randy > With a patch from Richard, and the http_proxy check removed, $ git diff | grep "^[+-]" --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py -http_proxy = "http://example.com/" +http_proxy = "" we have fixed the oe-selftest that was failing. Any volunteers to stub out the http[s]_proxy code in cargo ? I've written up a commit log that could use some review and pushed an update to: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rmacleod/rust-redo-aug-17-2021-a I starting to work on the YP autobuilder issues. I can build:    $ bitbake cargo-cross-canadian-x86-64 Adding this: require conf/multilib.conf MULTILIBS = 'multilib:lib32' DEFAULTTUNE:virtclass-multilib-lib32 = 'x86' to my local.conf, I see that: $ bitbake lib32-rust fails with the error: --- failed to execute command: "/ala-lpggp31/rmacleod/src/distro/yocto/b/aug-19-a/tmp/work/x86-pokymllib32-linux/lib32-rust/1.54.0-r0/lib32-recipe-sysroot/usr/lib/llvm-rust/bin/llvm-config" "--bindir" | error: No such file or directory (os error 2) --- Khem, a MUSL build fails: $ TCLIBC=musl  bitbake rust-hello-world as shown below (1) but I'll see if that's reproducible with a clean build. I've also rebased to poky master:   3fd2dff4eb3 (origin/master, origin/HEAD, master) ref-manual: Fix reference to bbappend section of dev-manual and the two key oe-selftests still pass with the http_proxy work-around as does core-image-minimal+rust-hello-world (2) so I've pushed a new branch: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rmacleod/rust-redo-aug-21-2021-a Richard, Alex,   Is it worth another AB run? ../Randy 1) $ TCLIBC=musl  bitbake rust-hello-world ERROR: rust-cross-x86_64-1.54.0-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function:      0001:  *** 0002:extend_recipe_sysroot(d)      0003: File: '/ala-lpggp31/rmacleod/src/distro/yocto/poky-contrib.git/meta/classes/staging.bbclass', lineno: 582, function: extend_recipe_sysroot      0578:                    if "/bin/" in l or "/sbin/" in l:      0579:                        # defer /*bin/* files until last in case they need libs      0580:                        binfiles[l] = (targetdir, dest)      0581:                    else:  *** 0582:                        staging_copyfile(l, targetdir, dest, postinsts, seendirs)      0583:      0584:    # Handle deferred binfiles      0585:    for l in binfiles:      0586:        (targetdir, dest) = binfiles[l] File: '/ala-lpggp31/rmacleod/src/distro/yocto/poky-contrib.git/meta/classes/staging.bbclass', lineno: 149, function: staging_copyfile      0145:    if os.path.islink(c):      0146:        linkto = os.readlink(c)      0147:        if os.path.lexists(dest):      0148:            if not os.path.islink(dest):  *** 0149:                raise OSError(errno.EEXIST, "Link %s already exists as a file" % dest, dest)      0150:            if os.readlink(dest) == linkto:      0151:                return dest      0152:            raise OSError(errno.EEXIST, "Link %s already exists to a different location? (%s vs %s)" % (dest, os.readlink(dest), linkto), dest)      0153:        os.symlink(linkto, dest) Exception: FileExistsError: [Errno 17] Link /ala-lpggp31/rmacleod/src/distro/yocto/b/aug-19-a/tmp/work/x86_64-linux/rust-cross-x86_64/1.54.0-r0/recipe-sysroot/usr/include/c++/11.2.0/x86_64-poky-linux already ex' ERROR: Logfile of failure stored in: /ala-lpggp31/rmacleod/src/distro/yocto/b/aug-19-a/tmp/work/x86_64-linux/rust-cross-x86_64/1.54.0-r0/temp/log.do_prepare_recipe_sysroot.192760 ERROR: Task (/ala-lpggp31/rmacleod/src/distro/yocto/poky-contrib.git/meta/recipes-devtools/rust/rust-cross_1.54.0.bb:do_prepare_recipe_sysroot) failed with exit code '1' 2) There is a warning for rust-native: grep WARNING tmp/work/x86_64-linux/rust-native/1.54.0-r0/temp/log.do_populate_sysroot | fmt WARNING: File '/ala-lpggp31/rmacleod/src/distro/yocto/b/aug-21-a/tmp/work/x86_64-linux/rust-native/1.54.0-r0/recipe-sysroot-native/usr/lib/rustlib/x86_64-linux/bin/rust-llvm-dwp' from rust-native was already stripped, this will prevent future debugging! > > I did learn a bit about this test code that is failing: > > https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/selftest/cases/sstatetests.py#n404 > > as the comment explains: > > | > The sstate checksums of two identical machines (using the same tune) > should be the > same, apart from changes within the machine specific stamps directory. > We use the > qemux86copy machine to test this. Also include multilibs in the test.| > > so the good news is that there are: 23185files being checked and > only 43 are different and if you look at what packages and arches > differ it's: > > x86_64-linux/lib32-rust-cross-i686 > x86_64-linux/rust-cross-i686 > core2-32-poky-linux/rust > core2-32-poky-linux/libstd-rs > x86-pokymllib32-linux/lib32-libstd-rs > x86-pokymllib32-linux/lib32-rust > > Note that these are all 32bit multilib builds. > Maybe that's not so important for the initial merge to oe-core... Richard? > > > Looking at core2-32-poky-linux/rust the stages that have different > signatures are: > > do_configure > do_populate_sysroot > do_package_qa > do_install > do_package > do_packagedata > do_compile > do_prepare_recipe_sysroot > do_package_write_rpm > > 9 of the 19 stages. Ok so maybe it's not great news after all. ;-) > > > I'll kept digging down through the tree of dependant diffsigs > since when looking at do_prepare_recipe_sysroot, it was clear > that a key difference in it's signatures was due to: >   "Hash for dependent task ..." > differing. Finally I found a set of signatures that did not have a > diff in the dependent task: > > stamps/x86_64-linux/rust-cross-i686/1.54.0-r0.do_rust_gen_target* > > The simplified diff is below and the files are attached. > > There are a variety of variables that differ only in order, such as: > > -List of dependencies for variable TUNE_FEATURES is > {'TUNE_FEATURES:tune-core2-32', 'DEFAULTTUNE'} > +List of dependencies for variable TUNE_FEATURES is {'DEFAULTTUNE', > 'TUNE_FEATURES:tune-core2-32'} > > I assume the next step is to find out where those variables are produced: > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/tree/meta/recipes-devtools/rust/rust-common.inc?h=rmacleod/rust-redo-aug-10-2021-a#n292 > > and to introduce deterministic order when building them. > > ../Randy > > > diff -Naur x86_64-rust-cross-i686-rust-gen-targets/* | grep "^[+-]" | > cut -c -166 > --- > x86_64-rust-cross-i686-rust-gen-targets/1.rust-cross-i686.do_rust_gen_target > 2021-08-20 22:19:19.883320107 -0400 > +++ > x86_64-rust-cross-i686-rust-gen-targets/2.rust-cross-i686.do_rust_gen_target > 2021-08-20 22:19:19.651319015 -0400 > -basewhitelist: {'CCACHE_TOP_DIR', 'SOURCE_DATE_EPOCH', > 'SDKPKGSUFFIX', 'BB_WORKERCONTEXT', 'FILESEXTRAPATHS', 'HOME', > 'OMP_NUM_THREADS', 'PRSERV_DUMPDIR', 'PRSERV_HO > +basewhitelist: {'PWD', 'SSTATE_HASHEQUIV_OWNER', 'STAMPS_DIR', > 'CCACHE', 'BB_WORKERCONTEXT', 'COREBASE', 'BBPATH', 'HOME', 'TMPDIR', > 'ERROR_QA', 'SSTATE_HASHEQUIV_ME > -basehash: > bd66629631e757dcccfc103c13e7de6ac3568677df3ef5ad47dd99e62427caab > +basehash: > d697cf2cbb5807e83f8b55de8012d5bb9ab8674f4bbf95587fc3ca822a47269b > -List of dependencies for variable TUNE_ARCH is {'X86ARCH32', > 'X86ARCH64'} > +List of dependencies for variable TUNE_ARCH is {'X86ARCH64', > 'X86ARCH32'} > -List of dependencies for variable TUNE_FEATURES is > {'TUNE_FEATURES:tune-core2-32', 'DEFAULTTUNE'} > +List of dependencies for variable TUNE_FEATURES is {'DEFAULTTUNE', > 'TUNE_FEATURES:tune-core2-32'} > -List of dependencies for variable llvm_cpu is {'PACKAGE_ARCH', > 'TUNE_FEATURES', 'TRANSLATED_TARGET_ARCH'} > -List of dependencies for variable llvm_features is > {'llvm_features_from_cc_arch', 'llvm_features_from_tune', > 'llvm_features_from_target_fpu'} > +List of dependencies for variable llvm_cpu is {'PACKAGE_ARCH', > 'TRANSLATED_TARGET_ARCH', 'TUNE_FEATURES'} > +List of dependencies for variable llvm_features is > {'llvm_features_from_target_fpu', 'llvm_features_from_cc_arch', > 'llvm_features_from_tune'} > -List of dependencies for variable llvm_features_from_tune is > {'target_is_armv7', 'TUNE_FEATURES', 'ARM_THUMB_OPT', 'MACHINEOVERRIDES'} > +List of dependencies for variable llvm_features_from_tune is > {'ARM_THUMB_OPT', 'TUNE_FEATURES', 'target_is_armv7', 'MACHINEOVERRIDES'} > -List of dependencies for variable rust_gen_target is {'sys_for', > 'arch_to_rust_target_arch', 'RUST_PANIC_STRATEGY', 'prefix_for'} > +List of dependencies for variable rust_gen_target is {'sys_for', > 'prefix_for', 'arch_to_rust_target_arch', 'RUST_PANIC_STRATEGY'} > -List of dependencies for variable target_is_armv7 is {'TARGET_ARCH', > 'TUNE_FEATURES', 'MACHINEOVERRIDES'} > +List of dependencies for variable target_is_armv7 is > {'TUNE_FEATURES', 'TARGET_ARCH', 'MACHINEOVERRIDES'} > -Variable MACHINE value is qemux86 > -Variable MACHINEOVERRIDES value is > ${@bb.utils.contains('TUNE_FEATURES', 'mx32', 'x86-x32:', '', > d)}${@bb.utils.contains('TUNE_FEATURES', 'm32', 'x86:', '', d)}qemua > +Variable MACHINE value is qemux86copy > +Variable MACHINEOVERRIDES value is > ${@bb.utils.contains('TUNE_FEATURES', 'mx32', 'x86-x32:', '', > d)}${@bb.utils.contains('TUNE_FEATURES', 'm32', 'x86:', '', d)}qemua > -Computed base hash is > bd66629631e757dcccfc103c13e7de6ac3568677df3ef5ad47dd99e62427caab and > from file > bd66629631e757dcccfc103c13e7de6ac3568677df3ef5ad47dd99e62427caab > -Computed task hash is > 448a8113230a16ac5d606de18989f98b3700edeb3e7956c6f02207aa76a5d4f3 > +Computed base hash is > d697cf2cbb5807e83f8b55de8012d5bb9ab8674f4bbf95587fc3ca822a47269b and > from file > d697cf2cbb5807e83f8b55de8012d5bb9ab8674f4bbf95587fc3ca822a47269b > +Computed task hash is > 83d1a26ebe7e2df3f496024a297ba782e145e6982457e09102d5e7c4fbf1b3b7 > > >> >> >> ../Randy >> >> >>> >>> ../Randy >>>> >>>> On Fri, Aug 13, 2021 at 8:50 PM Vinay Kumar >>>> wrote: >>>>> >>>>> rust-common.inc: Fix build failure with qemuppc64. >>>>> >>>>> The glibc build of "rust-hello-world" throws error in libstd-rs >>>>> package. >>>>> error: unrecognized arch "powerpc64le" in target specification >>>>> >>>>> The same got fixed by changing the arch to "powerpc64". >>>>> >>>>> Signed-off-by: Vinay Kumar >>>>> --- >>>>>   meta/recipes-devtools/rust/rust-common.inc | 2 ++ >>>>>   1 file changed, 2 insertions(+) >>>>> >>>>> diff --git a/meta/recipes-devtools/rust/rust-common.inc >>>>> b/meta/recipes-devtools/rust/rust-common.inc >>>>> index 350517c723..f574152815 100644 >>>>> --- a/meta/recipes-devtools/rust/rust-common.inc >>>>> +++ b/meta/recipes-devtools/rust/rust-common.inc >>>>> @@ -248,6 +248,8 @@ def arch_to_rust_target_arch(arch): >>>>>           return "mips64" >>>>>       elif arch == "armv7": >>>>>           return "arm" >>>>> +    elif arch == "powerpc64le": >>>>> +        return "powerpc64" >>>>>       else: >>>>>           return arch >>>>> >>>>> -- >>>>> 2.31.1 >>>>> >>> >>> >>> >>> >>> >>> >> >> > > -- > # Randy MacLeod > # Wind River Linux > > > -- # Randy MacLeod # Wind River Linux