[OE-Core][PATCH 1/2] cargo_common.bbclass: Fix typos

[OE-Core][PATCH 1/2] cargo_common.bbclass: Fix typos

[Alex Kiernan]

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
---
 meta/classes-recipe/cargo_common.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass
index dea0fbe2f653..f503a001dd8e 100644
--- a/meta/classes-recipe/cargo_common.bbclass
+++ b/meta/classes-recipe/cargo_common.bbclass
@@ -56,7 +56,7 @@ cargo_common_do_configure () {
 
 		[source.crates-io]
 		replace-with = "bitbake"
-		local-registry = "/nonexistant"
+		local-registry = "/nonexistent"
 		EOF
 	fi
 
@@ -103,7 +103,7 @@ cargo_common_do_configure () {
 		cat <<- EOF >> ${CARGO_HOME}/config
 
 		[build]
-		# Use out of tree build destination to avoid poluting the source tree
+		# Use out of tree build destination to avoid polluting the source tree
 		target-dir = "${B}/target"
 		EOF
 	fi
-- 
2.35.1

[OE-Core][PATCH 2/2] cargo_common.bbclass: Support local github repos

[Alex Kiernan]

Since disable network was added cargo configurations which reference git
repos fail as they attempt to fetch across the network as part of
do_compile, even if EXTRA_OECARGO_PATHS to add them as part of `paths`
is used, as this is documented as only working for packages which exist
in crates.io.

Add parsing of the SRC_URIs for git repos and include `[patch]` sections
to redirect to the checked out source repos which the bitbake fetcher
has already populated.

There are still cases which don't work - if you have multiple copies of
the same repo with different revisions, there's currently no way to
represent that and anything using a repo which has a virtual manifest
will fail to build (see https://github.com/rust-lang/cargo/issues/4934).

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
---
 meta/classes-recipe/cargo_common.bbclass | 30 ++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass
index f503a001dd8e..63b13829084b 100644
--- a/meta/classes-recipe/cargo_common.bbclass
+++ b/meta/classes-recipe/cargo_common.bbclass
@@ -116,6 +116,36 @@ cargo_common_do_configure () {
 	EOF
 }
 
+python cargo_common_do_patch_paths() {
+    cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config")
+    if not os.path.exists(cargo_config):
+        return
+
+    src_uri = (d.getVar('SRC_URI') or "").split()
+    if len(src_uri) == 0:
+        return
+
+    patches = dict()
+    workdir = d.getVar('WORKDIR')
+    fetcher = bb.fetch2.Fetch(src_uri, d)
+    for url in fetcher.urls:
+        ud = fetcher.ud[url]
+        if ud.type == 'git':
+            name = ud.parm.get('name')
+            destsuffix = ud.parm.get('destsuffix')
+            if name is not None and destsuffix is not None:
+                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
+                path = '%s = { path = "%s" }' % (name, os.path.join(workdir, destsuffix))
+                patches.setdefault(repo, []).append(path)
+
+    with open(cargo_config, "a+") as config:
+        for k, v in patches.items():
+            print('\n[patch."%s"]' % k, file=config)
+            for name in v:
+                print(name, file=config)
+}
+do_configure[postfuncs] += "cargo_common_do_patch_paths"
+
 oe_cargo_fix_env () {
 	export CC="${RUST_TARGET_CC}"
 	export CXX="${RUST_TARGET_CXX}"
-- 
2.35.1

Re: [OE-Core][PATCH 2/2] cargo_common.bbclass: Support local github repos

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 3258 bytes --]

There doesn’t seem to be a test case or an example for this. How can we
ensure the code is correct?

Alex

On Sun 30. Oct 2022 at 18.38, Alex Kiernan <alex.kiernan@gmail.com> wrote:

> Since disable network was added cargo configurations which reference git
> repos fail as they attempt to fetch across the network as part of
> do_compile, even if EXTRA_OECARGO_PATHS to add them as part of `paths`
> is used, as this is documented as only working for packages which exist
> in crates.io.
>
> Add parsing of the SRC_URIs for git repos and include `[patch]` sections
> to redirect to the checked out source repos which the bitbake fetcher
> has already populated.
>
> There are still cases which don't work - if you have multiple copies of
> the same repo with different revisions, there's currently no way to
> represent that and anything using a repo which has a virtual manifest
> will fail to build (see https://github.com/rust-lang/cargo/issues/4934).
>
> Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
> ---
>  meta/classes-recipe/cargo_common.bbclass | 30 ++++++++++++++++++++++++
>  1 file changed, 30 insertions(+)
>
> diff --git a/meta/classes-recipe/cargo_common.bbclass
> b/meta/classes-recipe/cargo_common.bbclass
> index f503a001dd8e..63b13829084b 100644
> --- a/meta/classes-recipe/cargo_common.bbclass
> +++ b/meta/classes-recipe/cargo_common.bbclass
> @@ -116,6 +116,36 @@ cargo_common_do_configure () {
>         EOF
>  }
>
> +python cargo_common_do_patch_paths() {
> +    cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config")
> +    if not os.path.exists(cargo_config):
> +        return
> +
> +    src_uri = (d.getVar('SRC_URI') or "").split()
> +    if len(src_uri) == 0:
> +        return
> +
> +    patches = dict()
> +    workdir = d.getVar('WORKDIR')
> +    fetcher = bb.fetch2.Fetch(src_uri, d)
> +    for url in fetcher.urls:
> +        ud = fetcher.ud[url]
> +        if ud.type == 'git':
> +            name = ud.parm.get('name')
> +            destsuffix = ud.parm.get('destsuffix')
> +            if name is not None and destsuffix is not None:
> +                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
> +                path = '%s = { path = "%s" }' % (name,
> os.path.join(workdir, destsuffix))
> +                patches.setdefault(repo, []).append(path)
> +
> +    with open(cargo_config, "a+") as config:
> +        for k, v in patches.items():
> +            print('\n[patch."%s"]' % k, file=config)
> +            for name in v:
> +                print(name, file=config)
> +}
> +do_configure[postfuncs] += "cargo_common_do_patch_paths"
> +
>  oe_cargo_fix_env () {
>         export CC="${RUST_TARGET_CC}"
>         export CXX="${RUST_TARGET_CXX}"
> --
> 2.35.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#172284):
> https://lists.openembedded.org/g/openembedded-core/message/172284
> Mute This Topic: https://lists.openembedded.org/mt/94668827/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 4737 bytes --]

Re: [OE-Core][PATCH 2/2] cargo_common.bbclass: Support local github repos

[Alex Kiernan]

I was wondering about how to do that - my test case was
https://github.com/akiernan/uuid-test, but that's clearly not
adequate; the recipe there comes from `cargo bitbake` as `bitbake -c
update_crates` doesn't deal with git dependencies, or dig down into
transitive deps, which also needs solving.

I'm not clear if something like this could be added as part of the
stuff that's in master-next for rust tests? Or if we need to build out
something new.

On Sun, Oct 30, 2022 at 5:58 PM Alexander Kanavin
<alex.kanavin@gmail.com> wrote:
>
> There doesn’t seem to be a test case or an example for this. How can we ensure the code is correct?
>
> Alex
>
> On Sun 30. Oct 2022 at 18.38, Alex Kiernan <alex.kiernan@gmail.com> wrote:
>>
>> Since disable network was added cargo configurations which reference git
>> repos fail as they attempt to fetch across the network as part of
>> do_compile, even if EXTRA_OECARGO_PATHS to add them as part of `paths`
>> is used, as this is documented as only working for packages which exist
>> in crates.io.
>>
>> Add parsing of the SRC_URIs for git repos and include `[patch]` sections
>> to redirect to the checked out source repos which the bitbake fetcher
>> has already populated.
>>
>> There are still cases which don't work - if you have multiple copies of
>> the same repo with different revisions, there's currently no way to
>> represent that and anything using a repo which has a virtual manifest
>> will fail to build (see https://github.com/rust-lang/cargo/issues/4934).
>>
>> Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
>> ---
>>  meta/classes-recipe/cargo_common.bbclass | 30 ++++++++++++++++++++++++
>>  1 file changed, 30 insertions(+)
>>
>> diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass
>> index f503a001dd8e..63b13829084b 100644
>> --- a/meta/classes-recipe/cargo_common.bbclass
>> +++ b/meta/classes-recipe/cargo_common.bbclass
>> @@ -116,6 +116,36 @@ cargo_common_do_configure () {
>>         EOF
>>  }
>>
>> +python cargo_common_do_patch_paths() {
>> +    cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config")
>> +    if not os.path.exists(cargo_config):
>> +        return
>> +
>> +    src_uri = (d.getVar('SRC_URI') or "").split()
>> +    if len(src_uri) == 0:
>> +        return
>> +
>> +    patches = dict()
>> +    workdir = d.getVar('WORKDIR')
>> +    fetcher = bb.fetch2.Fetch(src_uri, d)
>> +    for url in fetcher.urls:
>> +        ud = fetcher.ud[url]
>> +        if ud.type == 'git':
>> +            name = ud.parm.get('name')
>> +            destsuffix = ud.parm.get('destsuffix')
>> +            if name is not None and destsuffix is not None:
>> +                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
>> +                path = '%s = { path = "%s" }' % (name, os.path.join(workdir, destsuffix))
>> +                patches.setdefault(repo, []).append(path)
>> +
>> +    with open(cargo_config, "a+") as config:
>> +        for k, v in patches.items():
>> +            print('\n[patch."%s"]' % k, file=config)
>> +            for name in v:
>> +                print(name, file=config)
>> +}
>> +do_configure[postfuncs] += "cargo_common_do_patch_paths"
>> +
>>  oe_cargo_fix_env () {
>>         export CC="${RUST_TARGET_CC}"
>>         export CXX="${RUST_TARGET_CXX}"
>> --
>> 2.35.1
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#172284): https://lists.openembedded.org/g/openembedded-core/message/172284
>> Mute This Topic: https://lists.openembedded.org/mt/94668827/1686489
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>


-- 
Alex Kiernan

Re: [OE-Core][PATCH 2/2] cargo_common.bbclass: Support local github repos

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 4517 bytes --]

It would also help if there’s an actual recipe and component somewhere
where the problem exists now. The uuid-test does look like a made up
example.

Let’s merge the crate updater to core first, then we can think of handling
other ways to specify dependencies. ‘cargo bitbake’ is pretty broken and
dead.

Alex

On Sun 30. Oct 2022 at 19.12, Alex Kiernan <alex.kiernan@gmail.com> wrote:

> I was wondering about how to do that - my test case was
> https://github.com/akiernan/uuid-test, but that's clearly not
> adequate; the recipe there comes from `cargo bitbake` as `bitbake -c
> update_crates` doesn't deal with git dependencies, or dig down into
> transitive deps, which also needs solving.
>
> I'm not clear if something like this could be added as part of the
> stuff that's in master-next for rust tests? Or if we need to build out
> something new.
>
> On Sun, Oct 30, 2022 at 5:58 PM Alexander Kanavin
> <alex.kanavin@gmail.com> wrote:
> >
> > There doesn’t seem to be a test case or an example for this. How can we
> ensure the code is correct?
> >
> > Alex
> >
> > On Sun 30. Oct 2022 at 18.38, Alex Kiernan <alex.kiernan@gmail.com>
> wrote:
> >>
> >> Since disable network was added cargo configurations which reference git
> >> repos fail as they attempt to fetch across the network as part of
> >> do_compile, even if EXTRA_OECARGO_PATHS to add them as part of `paths`
> >> is used, as this is documented as only working for packages which exist
> >> in crates.io.
> >>
> >> Add parsing of the SRC_URIs for git repos and include `[patch]` sections
> >> to redirect to the checked out source repos which the bitbake fetcher
> >> has already populated.
> >>
> >> There are still cases which don't work - if you have multiple copies of
> >> the same repo with different revisions, there's currently no way to
> >> represent that and anything using a repo which has a virtual manifest
> >> will fail to build (see https://github.com/rust-lang/cargo/issues/4934
> ).
> >>
> >> Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
> >> ---
> >>  meta/classes-recipe/cargo_common.bbclass | 30 ++++++++++++++++++++++++
> >>  1 file changed, 30 insertions(+)
> >>
> >> diff --git a/meta/classes-recipe/cargo_common.bbclass
> b/meta/classes-recipe/cargo_common.bbclass
> >> index f503a001dd8e..63b13829084b 100644
> >> --- a/meta/classes-recipe/cargo_common.bbclass
> >> +++ b/meta/classes-recipe/cargo_common.bbclass
> >> @@ -116,6 +116,36 @@ cargo_common_do_configure () {
> >>         EOF
> >>  }
> >>
> >> +python cargo_common_do_patch_paths() {
> >> +    cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config")
> >> +    if not os.path.exists(cargo_config):
> >> +        return
> >> +
> >> +    src_uri = (d.getVar('SRC_URI') or "").split()
> >> +    if len(src_uri) == 0:
> >> +        return
> >> +
> >> +    patches = dict()
> >> +    workdir = d.getVar('WORKDIR')
> >> +    fetcher = bb.fetch2.Fetch(src_uri, d)
> >> +    for url in fetcher.urls:
> >> +        ud = fetcher.ud[url]
> >> +        if ud.type == 'git':
> >> +            name = ud.parm.get('name')
> >> +            destsuffix = ud.parm.get('destsuffix')
> >> +            if name is not None and destsuffix is not None:
> >> +                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
> >> +                path = '%s = { path = "%s" }' % (name,
> os.path.join(workdir, destsuffix))
> >> +                patches.setdefault(repo, []).append(path)
> >> +
> >> +    with open(cargo_config, "a+") as config:
> >> +        for k, v in patches.items():
> >> +            print('\n[patch."%s"]' % k, file=config)
> >> +            for name in v:
> >> +                print(name, file=config)
> >> +}
> >> +do_configure[postfuncs] += "cargo_common_do_patch_paths"
> >> +
> >>  oe_cargo_fix_env () {
> >>         export CC="${RUST_TARGET_CC}"
> >>         export CXX="${RUST_TARGET_CXX}"
> >> --
> >> 2.35.1
> >>
> >>
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >> Links: You receive all messages sent to this group.
> >> View/Reply Online (#172284):
> https://lists.openembedded.org/g/openembedded-core/message/172284
> >> Mute This Topic: https://lists.openembedded.org/mt/94668827/1686489
> >> Group Owner: openembedded-core+owner@lists.openembedded.org
> >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >>
>
>
> --
> Alex Kiernan
>

[-- Attachment #2: Type: text/html, Size: 6805 bytes --]

Re: [OE-Core][PATCH 2/2] cargo_common.bbclass: Support local github repos

[Alex Kiernan]

On Sun, Oct 30, 2022 at 6:35 PM Alexander Kanavin
<alex.kanavin@gmail.com> wrote:
>
> It would also help if there’s an actual recipe and component somewhere where the problem exists now. The uuid-test does look like a made up example.
>

That's fair... basically it was demonstrating the problem I have with
a bunch of private repos, which I expect will crop up a lot in the
embedded rust world - if you're doing public dev, everything exists as
crates, in the private world you're consuming crates + internal stuff.
Not sure where to try and find a real world example as I'd expect
pretty much everyone will push to crates.io if their code is open -
searching around for a bit, I've failed to find any (though a couple
which have some dev only dependencies which come through git).

> Let’s merge the crate updater to core first, then we can think of handling other ways to specify dependencies. ‘cargo bitbake’ is pretty broken and dead.
>

The crate updater feels like it's going to be directionally better -
testing it locally, it works for things which only consume things from
crates.io, but that's only one out of thirty or so components I have
to deal with.


> Alex
>
> On Sun 30. Oct 2022 at 19.12, Alex Kiernan <alex.kiernan@gmail.com> wrote:
>>
>> I was wondering about how to do that - my test case was
>> https://github.com/akiernan/uuid-test, but that's clearly not
>> adequate; the recipe there comes from `cargo bitbake` as `bitbake -c
>> update_crates` doesn't deal with git dependencies, or dig down into
>> transitive deps, which also needs solving.
>>
>> I'm not clear if something like this could be added as part of the
>> stuff that's in master-next for rust tests? Or if we need to build out
>> something new.
>>
>> On Sun, Oct 30, 2022 at 5:58 PM Alexander Kanavin
>> <alex.kanavin@gmail.com> wrote:
>> >
>> > There doesn’t seem to be a test case or an example for this. How can we ensure the code is correct?
>> >
>> > Alex
>> >
>> > On Sun 30. Oct 2022 at 18.38, Alex Kiernan <alex.kiernan@gmail.com> wrote:
>> >>
>> >> Since disable network was added cargo configurations which reference git
>> >> repos fail as they attempt to fetch across the network as part of
>> >> do_compile, even if EXTRA_OECARGO_PATHS to add them as part of `paths`
>> >> is used, as this is documented as only working for packages which exist
>> >> in crates.io.
>> >>
>> >> Add parsing of the SRC_URIs for git repos and include `[patch]` sections
>> >> to redirect to the checked out source repos which the bitbake fetcher
>> >> has already populated.
>> >>
>> >> There are still cases which don't work - if you have multiple copies of
>> >> the same repo with different revisions, there's currently no way to
>> >> represent that and anything using a repo which has a virtual manifest
>> >> will fail to build (see https://github.com/rust-lang/cargo/issues/4934).
>> >>
>> >> Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
>> >> ---
>> >>  meta/classes-recipe/cargo_common.bbclass | 30 ++++++++++++++++++++++++
>> >>  1 file changed, 30 insertions(+)
>> >>
>> >> diff --git a/meta/classes-recipe/cargo_common.bbclass b/meta/classes-recipe/cargo_common.bbclass
>> >> index f503a001dd8e..63b13829084b 100644
>> >> --- a/meta/classes-recipe/cargo_common.bbclass
>> >> +++ b/meta/classes-recipe/cargo_common.bbclass
>> >> @@ -116,6 +116,36 @@ cargo_common_do_configure () {
>> >>         EOF
>> >>  }
>> >>
>> >> +python cargo_common_do_patch_paths() {
>> >> +    cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config")
>> >> +    if not os.path.exists(cargo_config):
>> >> +        return
>> >> +
>> >> +    src_uri = (d.getVar('SRC_URI') or "").split()
>> >> +    if len(src_uri) == 0:
>> >> +        return
>> >> +
>> >> +    patches = dict()
>> >> +    workdir = d.getVar('WORKDIR')
>> >> +    fetcher = bb.fetch2.Fetch(src_uri, d)
>> >> +    for url in fetcher.urls:
>> >> +        ud = fetcher.ud[url]
>> >> +        if ud.type == 'git':
>> >> +            name = ud.parm.get('name')
>> >> +            destsuffix = ud.parm.get('destsuffix')
>> >> +            if name is not None and destsuffix is not None:
>> >> +                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
>> >> +                path = '%s = { path = "%s" }' % (name, os.path.join(workdir, destsuffix))
>> >> +                patches.setdefault(repo, []).append(path)
>> >> +
>> >> +    with open(cargo_config, "a+") as config:
>> >> +        for k, v in patches.items():
>> >> +            print('\n[patch."%s"]' % k, file=config)
>> >> +            for name in v:
>> >> +                print(name, file=config)
>> >> +}
>> >> +do_configure[postfuncs] += "cargo_common_do_patch_paths"
>> >> +
>> >>  oe_cargo_fix_env () {
>> >>         export CC="${RUST_TARGET_CC}"
>> >>         export CXX="${RUST_TARGET_CXX}"
>> >> --
>> >> 2.35.1
>> >>
>> >>
>> >> -=-=-=-=-=-=-=-=-=-=-=-
>> >> Links: You receive all messages sent to this group.
>> >> View/Reply Online (#172284): https://lists.openembedded.org/g/openembedded-core/message/172284
>> >> Mute This Topic: https://lists.openembedded.org/mt/94668827/1686489
>> >> Group Owner: openembedded-core+owner@lists.openembedded.org
>> >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
>> >> -=-=-=-=-=-=-=-=-=-=-=-
>> >>
>>
>>
>> --
>> Alex Kiernan



--
Alex Kiernan