[OE-core][PATCH 0/4] Move crate fetcher to OE_IMPORTS

[OE-core][PATCH 0/4] Move crate fetcher to OE_IMPORTS

[Matt Madison]

This patch series switches to using the OE_IMPORTS mechanism
for installing the crate fetcher. This simplifies the addition
and clears up issues with interactions between SRCPV and
the shared-state fetching logic.

Matt Madison (4):
  lib/crate.py: install crate fetcher on import
  base.bbclass: add crate to OE_IMPORTS
  cargo_common.bbclass: remove inherit of crate-fetch
  classes: remove crate-fetch.bbclass

 meta/classes/base.bbclass         |  2 +-
 meta/classes/cargo_common.bbclass |  2 --
 meta/classes/crate-fetch.bbclass  | 28 ----------------------------
 meta/lib/crate.py                 |  2 ++
 4 files changed, 3 insertions(+), 31 deletions(-)
 delete mode 100644 meta/classes/crate-fetch.bbclass

-- 
2.32.0

[OE-core][PATCH 1/4] lib/crate.py: install crate fetcher on import

[Matt Madison]

to enable the move to using OE_IMPORTS for adding
this fetcher.

Signed-off-by: Matt Madison <matt@madison.systems>
---
 meta/lib/crate.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/lib/crate.py b/meta/lib/crate.py
index d10f441875..f9db7c28cd 100644
--- a/meta/lib/crate.py
+++ b/meta/lib/crate.py
@@ -147,3 +147,5 @@ class Crate(Wget):
             mdpath = os.path.join(bbpath, cratepath, mdfile)
             with open(mdpath, "w") as f:
                 json.dump(metadata, f)
+
+bb.fetch2.methods.append(Crate())
-- 
2.32.0

[OE-core][PATCH 2/4] base.bbclass: add crate to OE_IMPORTS

[Matt Madison]

to install the crate:// fetch handler.

Signed-off-by: Matt Madison <matt@madison.systems>
---
 meta/classes/base.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index b709777f24..ee9950f207 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -12,7 +12,7 @@ inherit logging
 
 OE_EXTRA_IMPORTS ?= ""
 
-OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license oe.qa oe.reproducible ${OE_EXTRA_IMPORTS}"
+OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license oe.qa oe.reproducible crate ${OE_EXTRA_IMPORTS}"
 OE_IMPORTS[type] = "list"
 
 PACKAGECONFIG_CONFARGS ??= ""
-- 
2.32.0

[OE-core][PATCH 3/4] cargo_common.bbclass: remove inherit of crate-fetch

[Matt Madison]

No longer needed now that the crate fetcher is
installed via OE_IMPORTS.

Signed-off-by: Matt Madison <matt@madison.systems>
---
 meta/classes/cargo_common.bbclass | 2 --
 1 file changed, 2 deletions(-)

diff --git a/meta/classes/cargo_common.bbclass b/meta/classes/cargo_common.bbclass
index 23d82aa6ab..c403591434 100644
--- a/meta/classes/cargo_common.bbclass
+++ b/meta/classes/cargo_common.bbclass
@@ -8,8 +8,6 @@
 ## is used by cargo.bbclass and Rust
 ##
 
-# add crate fetch support
-inherit crate-fetch
 inherit rust-common
 
 # Where we download our registry and dependencies to
-- 
2.32.0

[OE-core][PATCH 4/4] classes: remove crate-fetch.bbclass

[Matt Madison]

No longer required now that the crate fetcher
gets installed via OE_IMPORTS.

Signed-off-by: Matt Madison <matt@madison.systems>
---
 meta/classes/crate-fetch.bbclass | 28 ----------------------------
 1 file changed, 28 deletions(-)
 delete mode 100644 meta/classes/crate-fetch.bbclass

diff --git a/meta/classes/crate-fetch.bbclass b/meta/classes/crate-fetch.bbclass
deleted file mode 100644
index a7fa22b2a0..0000000000
--- a/meta/classes/crate-fetch.bbclass
+++ /dev/null
@@ -1,28 +0,0 @@
-#
-# crate-fetch class
-#
-# Registers 'crate' method for Bitbake fetch2.
-#
-# Adds support for following format in recipe SRC_URI:
-# crate://<packagename>/<version>
-#
-
-def import_crate(d):
-    import crate
-    if not getattr(crate, 'imported', False):
-        bb.fetch2.methods.append(crate.Crate())
-        crate.imported = True
-
-python crate_import_handler() {
-    import_crate(d)
-}
-
-addhandler crate_import_handler
-crate_import_handler[eventmask] = "bb.event.RecipePreFinalise"
-
-def crate_get_srcrev(d):
-    import_crate(d)
-    return bb.fetch2.get_srcrev(d)
-
-# Override SRCPV to make sure it imports the fetcher first
-SRCPV = "${@crate_get_srcrev(d)}"
-- 
2.32.0

Re: [OE-core][PATCH 2/4] base.bbclass: add crate to OE_IMPORTS

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1468 bytes --]

Should this be named oe.crate for consistency and avoiding potential
clashes?

Alex

On Fri, 17 Dec 2021 at 18:37, Matt Madison <matt@madison.systems> wrote:

> to install the crate:// fetch handler.
>
> Signed-off-by: Matt Madison <matt@madison.systems>
> ---
>  meta/classes/base.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index b709777f24..ee9950f207 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -12,7 +12,7 @@ inherit logging
>
>  OE_EXTRA_IMPORTS ?= ""
>
> -OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package
> oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license oe.qa
> oe.reproducible ${OE_EXTRA_IMPORTS}"
> +OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package
> oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license oe.qa
> oe.reproducible crate ${OE_EXTRA_IMPORTS}"
>  OE_IMPORTS[type] = "list"
>
>  PACKAGECONFIG_CONFARGS ??= ""
> --
> 2.32.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#159830):
> https://lists.openembedded.org/g/openembedded-core/message/159830
> Mute This Topic: https://lists.openembedded.org/mt/87791269/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 2462 bytes --]

Re: [OE-core][PATCH 2/4] base.bbclass: add crate to OE_IMPORTS

[Matt Madison]

On Fri, Dec 17, 2021 at 8:37 AM Alexander Kanavin
<alex.kanavin@gmail.com> wrote:
>
> Should this be named oe.crate for consistency and avoiding potential clashes?

I retained the existing name for this change set.  The same thought
had occurred to me, though.

-Matt

>
> Alex
>
> On Fri, 17 Dec 2021 at 18:37, Matt Madison <matt@madison.systems> wrote:
>>
>> to install the crate:// fetch handler.
>>
>> Signed-off-by: Matt Madison <matt@madison.systems>
>> ---
>>  meta/classes/base.bbclass | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>> index b709777f24..ee9950f207 100644
>> --- a/meta/classes/base.bbclass
>> +++ b/meta/classes/base.bbclass
>> @@ -12,7 +12,7 @@ inherit logging
>>
>>  OE_EXTRA_IMPORTS ?= ""
>>
>> -OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license oe.qa oe.reproducible ${OE_EXTRA_IMPORTS}"
>> +OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license oe.qa oe.reproducible crate ${OE_EXTRA_IMPORTS}"
>>  OE_IMPORTS[type] = "list"
>>
>>  PACKAGECONFIG_CONFARGS ??= ""
>> --
>> 2.32.0
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#159830): https://lists.openembedded.org/g/openembedded-core/message/159830
>> Mute This Topic: https://lists.openembedded.org/mt/87791269/1686489
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>

Re: [OE-core][PATCH 0/4] Move crate fetcher to OE_IMPORTS

[Paul Barker]

[-- Attachment #1.1.1: Type: text/plain, Size: 1039 bytes --]

On 17/12/2021 15:36, Matt Madison wrote:
> This patch series switches to using the OE_IMPORTS mechanism
> for installing the crate fetcher. This simplifies the addition
> and clears up issues with interactions between SRCPV and
> the shared-state fetching logic.
> 
> Matt Madison (4):
>    lib/crate.py: install crate fetcher on import
>    base.bbclass: add crate to OE_IMPORTS
>    cargo_common.bbclass: remove inherit of crate-fetch
>    classes: remove crate-fetch.bbclass
> 
>   meta/classes/base.bbclass         |  2 +-
>   meta/classes/cargo_common.bbclass |  2 --
>   meta/classes/crate-fetch.bbclass  | 28 ----------------------------
>   meta/lib/crate.py                 |  2 ++
>   4 files changed, 3 insertions(+), 31 deletions(-)
>   delete mode 100644 meta/classes/crate-fetch.bbclass
> 

Would a better solution here be to move the crate fetcher into
lib/bb/fetch2/ in bitbake?

-- 
Paul Barker
Principal Software Engineer
SanCloud Ltd

e: paul.barker@sancloud.com
w: https://sancloud.co.uk/

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7643 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [OE-core][PATCH 0/4] Move crate fetcher to OE_IMPORTS

[Richard Purdie]

On Mon, 2021-12-20 at 17:01 +0000, Paul Barker wrote:
> On 17/12/2021 15:36, Matt Madison wrote:
> > This patch series switches to using the OE_IMPORTS mechanism
> > for installing the crate fetcher. This simplifies the addition
> > and clears up issues with interactions between SRCPV and
> > the shared-state fetching logic.
> > 
> > Matt Madison (4):
> >    lib/crate.py: install crate fetcher on import
> >    base.bbclass: add crate to OE_IMPORTS
> >    cargo_common.bbclass: remove inherit of crate-fetch
> >    classes: remove crate-fetch.bbclass
> > 
> >   meta/classes/base.bbclass         |  2 +-
> >   meta/classes/cargo_common.bbclass |  2 --
> >   meta/classes/crate-fetch.bbclass  | 28 ----------------------------
> >   meta/lib/crate.py                 |  2 ++
> >   4 files changed, 3 insertions(+), 31 deletions(-)
> >   delete mode 100644 meta/classes/crate-fetch.bbclass
> > 
> 
> Would a better solution here be to move the crate fetcher into
> lib/bb/fetch2/ in bitbake?

I've been wondering about this too.

Does anyone remember how the fetcher lines up against the general fetcher
requirements we have and is there a list of anything that remained to be done
before it could be added to bitbake?

Cheers,

Richard

Re: [OE-core][PATCH 0/4] Move crate fetcher to OE_IMPORTS

[Paul Barker]

On 20/12/2021 22:34, Richard Purdie wrote:
> On Mon, 2021-12-20 at 17:01 +0000, Paul Barker wrote:
>> On 17/12/2021 15:36, Matt Madison wrote:
>>> This patch series switches to using the OE_IMPORTS mechanism
>>> for installing the crate fetcher. This simplifies the addition
>>> and clears up issues with interactions between SRCPV and
>>> the shared-state fetching logic.
>>>
>>> Matt Madison (4):
>>>     lib/crate.py: install crate fetcher on import
>>>     base.bbclass: add crate to OE_IMPORTS
>>>     cargo_common.bbclass: remove inherit of crate-fetch
>>>     classes: remove crate-fetch.bbclass
>>>
>>>    meta/classes/base.bbclass         |  2 +-
>>>    meta/classes/cargo_common.bbclass |  2 --
>>>    meta/classes/crate-fetch.bbclass  | 28 ----------------------------
>>>    meta/lib/crate.py                 |  2 ++
>>>    4 files changed, 3 insertions(+), 31 deletions(-)
>>>    delete mode 100644 meta/classes/crate-fetch.bbclass
>>>
>>
>> Would a better solution here be to move the crate fetcher into
>> lib/bb/fetch2/ in bitbake?
> 
> I've been wondering about this too.
> 
> Does anyone remember how the fetcher lines up against the general fetcher
> requirements we have and is there a list of anything that remained to be done
> before it could be added to bitbake?

If I remember correctly, it just needed some test cases writing. The 
crate fetcher is a subclass of the wget fetcher with changes to how 
urldata is set up and how unpack is handled, so it should integrate well 
into bitbake.

Thanks,

-- 
Paul Barker
Principal Software Engineer
SanCloud Ltd

e: paul.barker@sancloud.com
w: https://sancloud.co.uk/

Re: [OE-core][PATCH 0/4] Move crate fetcher to OE_IMPORTS

[Matt Madison]

On Tue, Dec 21, 2021 at 6:07 AM Paul Barker <paul.barker@sancloud.com> wrote:
>
> On 20/12/2021 22:34, Richard Purdie wrote:
> > On Mon, 2021-12-20 at 17:01 +0000, Paul Barker wrote:
> >> On 17/12/2021 15:36, Matt Madison wrote:
> >>> This patch series switches to using the OE_IMPORTS mechanism
> >>> for installing the crate fetcher. This simplifies the addition
> >>> and clears up issues with interactions between SRCPV and
> >>> the shared-state fetching logic.
> >>>
> >>> Matt Madison (4):
> >>>     lib/crate.py: install crate fetcher on import
> >>>     base.bbclass: add crate to OE_IMPORTS
> >>>     cargo_common.bbclass: remove inherit of crate-fetch
> >>>     classes: remove crate-fetch.bbclass
> >>>
> >>>    meta/classes/base.bbclass         |  2 +-
> >>>    meta/classes/cargo_common.bbclass |  2 --
> >>>    meta/classes/crate-fetch.bbclass  | 28 ----------------------------
> >>>    meta/lib/crate.py                 |  2 ++
> >>>    4 files changed, 3 insertions(+), 31 deletions(-)
> >>>    delete mode 100644 meta/classes/crate-fetch.bbclass
> >>>
> >>
> >> Would a better solution here be to move the crate fetcher into
> >> lib/bb/fetch2/ in bitbake?
> >
> > I've been wondering about this too.
> >
> > Does anyone remember how the fetcher lines up against the general fetcher
> > requirements we have and is there a list of anything that remained to be done
> > before it could be added to bitbake?
>
> If I remember correctly, it just needed some test cases writing. The
> crate fetcher is a subclass of the wget fetcher with changes to how
> urldata is set up and how unpack is handled, so it should integrate well
> into bitbake.

One thing I see that's a bit worrisome is that it disables
recommends_checksum for the downloaded files,
and populates metadata for use (I assume) by cargo with the calculated
SHA256 on the download.

-Matt

>
> Thanks,
>
> --
> Paul Barker
> Principal Software Engineer
> SanCloud Ltd
>
> e: paul.barker@sancloud.com
> w: https://sancloud.co.uk/

Re: [OE-core][PATCH 0/4] Move crate fetcher to OE_IMPORTS

[Matt Madison]

On Tue, Dec 21, 2021 at 10:15 AM Matt Madison via
lists.openembedded.org <matt=madison.systems@lists.openembedded.org>
wrote:
>
> On Tue, Dec 21, 2021 at 6:07 AM Paul Barker <paul.barker@sancloud.com> wrote:
> >
> > On 20/12/2021 22:34, Richard Purdie wrote:
> > > On Mon, 2021-12-20 at 17:01 +0000, Paul Barker wrote:
> > >> On 17/12/2021 15:36, Matt Madison wrote:
> > >>> This patch series switches to using the OE_IMPORTS mechanism
> > >>> for installing the crate fetcher. This simplifies the addition
> > >>> and clears up issues with interactions between SRCPV and
> > >>> the shared-state fetching logic.
> > >>>
> > >>> Matt Madison (4):
> > >>>     lib/crate.py: install crate fetcher on import
> > >>>     base.bbclass: add crate to OE_IMPORTS
> > >>>     cargo_common.bbclass: remove inherit of crate-fetch
> > >>>     classes: remove crate-fetch.bbclass
> > >>>
> > >>>    meta/classes/base.bbclass         |  2 +-
> > >>>    meta/classes/cargo_common.bbclass |  2 --
> > >>>    meta/classes/crate-fetch.bbclass  | 28 ----------------------------
> > >>>    meta/lib/crate.py                 |  2 ++
> > >>>    4 files changed, 3 insertions(+), 31 deletions(-)
> > >>>    delete mode 100644 meta/classes/crate-fetch.bbclass
> > >>>
> > >>
> > >> Would a better solution here be to move the crate fetcher into
> > >> lib/bb/fetch2/ in bitbake?
> > >
> > > I've been wondering about this too.
> > >
> > > Does anyone remember how the fetcher lines up against the general fetcher
> > > requirements we have and is there a list of anything that remained to be done
> > > before it could be added to bitbake?
> >
> > If I remember correctly, it just needed some test cases writing. The
> > crate fetcher is a subclass of the wget fetcher with changes to how
> > urldata is set up and how unpack is handled, so it should integrate well
> > into bitbake.
>
> One thing I see that's a bit worrisome is that it disables
> recommends_checksum for the downloaded files,
> and populates metadata for use (I assume) by cargo with the calculated
> SHA256 on the download.

Any other thoughts on this?  I've modified my distro to work around
the problem that led me to propose these patches, but something needs
to be fixed here. crate-fetch.bbclass should not be setting SRCPV.

Thanks,
-Matt

>
> -Matt
>
> >
> > Thanks,
> >
> > --
> > Paul Barker
> > Principal Software Engineer
> > SanCloud Ltd
> >
> > e: paul.barker@sancloud.com
> > w: https://sancloud.co.uk/
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#159928): https://lists.openembedded.org/g/openembedded-core/message/159928
> Mute This Topic: https://lists.openembedded.org/mt/87791267/3618418
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [matt@madison.systems]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core][PATCH 0/4] Move crate fetcher to OE_IMPORTS

[Richard Purdie]

On Fri, 2022-01-14 at 02:31 -0800, Matt Madison wrote:
> On Tue, Dec 21, 2021 at 10:15 AM Matt Madison via
> lists.openembedded.org <matt=madison.systems@lists.openembedded.org>
> wrote:
> > 
> > On Tue, Dec 21, 2021 at 6:07 AM Paul Barker <paul.barker@sancloud.com> wrote:
> > > 
> > > On 20/12/2021 22:34, Richard Purdie wrote:
> > > > On Mon, 2021-12-20 at 17:01 +0000, Paul Barker wrote:
> > > > > On 17/12/2021 15:36, Matt Madison wrote:
> > > > > > This patch series switches to using the OE_IMPORTS mechanism
> > > > > > for installing the crate fetcher. This simplifies the addition
> > > > > > and clears up issues with interactions between SRCPV and
> > > > > > the shared-state fetching logic.
> > > > > > 
> > > > > > Matt Madison (4):
> > > > > >     lib/crate.py: install crate fetcher on import
> > > > > >     base.bbclass: add crate to OE_IMPORTS
> > > > > >     cargo_common.bbclass: remove inherit of crate-fetch
> > > > > >     classes: remove crate-fetch.bbclass
> > > > > > 
> > > > > >    meta/classes/base.bbclass         |  2 +-
> > > > > >    meta/classes/cargo_common.bbclass |  2 --
> > > > > >    meta/classes/crate-fetch.bbclass  | 28 ----------------------------
> > > > > >    meta/lib/crate.py                 |  2 ++
> > > > > >    4 files changed, 3 insertions(+), 31 deletions(-)
> > > > > >    delete mode 100644 meta/classes/crate-fetch.bbclass
> > > > > > 
> > > > > 
> > > > > Would a better solution here be to move the crate fetcher into
> > > > > lib/bb/fetch2/ in bitbake?
> > > > 
> > > > I've been wondering about this too.
> > > > 
> > > > Does anyone remember how the fetcher lines up against the general fetcher
> > > > requirements we have and is there a list of anything that remained to be done
> > > > before it could be added to bitbake?
> > > 
> > > If I remember correctly, it just needed some test cases writing. The
> > > crate fetcher is a subclass of the wget fetcher with changes to how
> > > urldata is set up and how unpack is handled, so it should integrate well
> > > into bitbake.
> > 
> > One thing I see that's a bit worrisome is that it disables
> > recommends_checksum for the downloaded files,
> > and populates metadata for use (I assume) by cargo with the calculated
> > SHA256 on the download.
> 
> Any other thoughts on this?  I've modified my distro to work around
> the problem that led me to propose these patches, but something needs
> to be fixed here. crate-fetch.bbclass should not be setting SRCPV.

My thoughts are that we should look to getting this merged into bitbake. To do
that we need some tests and then we can probably make it happen.

I am worried about your comments on the checksum and that may be something we
need to fix before it can be merged.

Cheers,

Richard