Support for git+https://foo.example.com like URIs

Support for git+https://foo.example.com like URIs

[Enrico Scholz]

Hello,

for some years we are using a mechanism which allows to encode the
protocol ("https", "file", ...) within the base uri.  So, we can write


| PROJECT_BASE_URI = "git+https://foo.example.com/project"
| SRC_URI = "${PROJECT_BASE_URI}/some-repo"

instead of

| PROJECT_BASE_URI = "git://foo.example.com/project"
| PROJECT_BASE_PROTOCOL = "https"
| SRC_URI = "${PROJECT_BASE_URI}/some-repo;protocol=${PROJECT_BASE_PROTOCOL}"


Four years ago I wrote a bug report [1] with a patch against bitbake
itself which was more or less rejected.

In the meantime, I moved this out in an OE class [2] which overrides
'bb.fetch.decodeurl'.


Is there some generic interest in this functionality so that I can
create and send a patch for bitbake?  Or is this too special and people
are happy with altering ${MIRROR}?



Enrico

Footnotes: 
[1]  https://bugzilla.yoctoproject.org/show_bug.cgi?id=9738

[2]  https://github.com/sigma-embedded/meta-de.sigma-chemnitz/blob/zeus/classes/elito-uridecode.bbclass
     https://github.com/sigma-embedded/meta-de.sigma-chemnitz/blob/zeus/lib/elito/__init__.py

Re: [bitbake-devel] Support for git+https://foo.example.com like URIs

[Paul Barker]

On Thu, 25 Jun 2020 at 14:42, Enrico Scholz via lists.openembedded.org
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> wrote:
>
> Hello,
>
> for some years we are using a mechanism which allows to encode the
> protocol ("https", "file", ...) within the base uri.  So, we can write
>
>
> | PROJECT_BASE_URI = "git+https://foo.example.com/project"
> | SRC_URI = "${PROJECT_BASE_URI}/some-repo"
>
> instead of
>
> | PROJECT_BASE_URI = "git://foo.example.com/project"
> | PROJECT_BASE_PROTOCOL = "https"
> | SRC_URI = "${PROJECT_BASE_URI}/some-repo;protocol=${PROJECT_BASE_PROTOCOL}"
>
>
> Four years ago I wrote a bug report [1] with a patch against bitbake
> itself which was more or less rejected.
>
> In the meantime, I moved this out in an OE class [2] which overrides
> 'bb.fetch.decodeurl'.
>
>
> Is there some generic interest in this functionality so that I can
> create and send a patch for bitbake?  Or is this too special and people
> are happy with altering ${MIRROR}?
>
>
>
> Enrico
>
> Footnotes:
> [1]  https://bugzilla.yoctoproject.org/show_bug.cgi?id=9738
>
> [2]  https://github.com/sigma-embedded/meta-de.sigma-chemnitz/blob/zeus/classes/elito-uridecode.bbclass
>      https://github.com/sigma-embedded/meta-de.sigma-chemnitz/blob/zeus/lib/elito/__init__.py

I'd implement this in an inverse way, add a new class for your
projects, eg. my-project.bbclass:

    MY_REPO_NAME ?= "${BPN}"
    SRC_URI = "git://foo.example.com/project/${MY_REPO_NAME};protocol=https"

Then in your recipe, inherit the bbclass and override the repo name if
it doesn't match the recipe name:

    MY_REPO_NAME = "some-repo"
    inherit my-project

That can be done with no modification or monkey patching of the fetcher.

Thanks,

--
Paul Barker
Konsulko Group

Re: [bitbake-devel] Support for git+https://foo.example.com like URIs

[Enrico Scholz]

Paul Barker <pbarker@konsulko.com> writes:

> I'd implement this in an inverse way, add a new class for your
> projects, eg. my-project.bbclass:
>
>     MY_REPO_NAME ?= "${BPN}"
>     SRC_URI =
>     "git://foo.example.com/project/${MY_REPO_NAME};protocol=https"

yes; we did such a thing a other projects too and it is difficulty to
handle:

- beside name, you have to add ';branch=...' information somewhere too
  -> either add yet another variable, or add this to MY_REPO_NAME

- it breaks common SRC_URI usage; now you have to do the inherit thing
  and then 'SRC_URI += "..."'

- it is not universal; for every repository source you have to write
  such a class and/or add yet another variable.

A 'git+ssh://' like syntax is common and allows a usual workflow (just
add it to $SRC_URI) and does require introduction of dozens of extra
variables and/or classes.


Enrico

Re: [bitbake-devel] Support for git+https://foo.example.com like URIs

[Richard Purdie]

On Thu, 2020-06-25 at 15:42 +0200, Enrico Scholz via
lists.openembedded.org wrote:
> Hello,
> 
> for some years we are using a mechanism which allows to encode the
> protocol ("https", "file", ...) within the base uri.  So, we can
> write
> 
> 
> > PROJECT_BASE_URI = "git+https://foo.example.com/project"
> > SRC_URI = "${PROJECT_BASE_URI}/some-repo"
> 
> instead of
> 
> > PROJECT_BASE_URI = "git://foo.example.com/project"
> > PROJECT_BASE_PROTOCOL = "https"
> > SRC_URI = "${PROJECT_BASE_URI}/some-
> > repo;protocol=${PROJECT_BASE_PROTOCOL}"
> 
> Four years ago I wrote a bug report [1] with a patch against bitbake
> itself which was more or less rejected.
> 
> In the meantime, I moved this out in an OE class [2] which overrides
> 'bb.fetch.decodeurl'.
> 
> 
> Is there some generic interest in this functionality so that I can
> create and send a patch for bitbake?  Or is this too special and
> people
> are happy with altering ${MIRROR}?

I didn't (and don't) like writing special cases into general code. The
fetchers once had a lot of that, it was hard to clean up and I'd prefer
not to go back there.

There are multiple different issues with fetcher urls in this context
and multiple open bugs from what I remember about related issues. Your
patch solves your specific use case and doesn't help those other open
bugs.

My intent has always been (and still is) to see if a more generic
solution is possible which not only addresses your use case but the
ones others have as well. That will likely mean using the mirrors
variable to remap things which will be slightly more complex but will
also enable others.

Cheers,

Richard

Re: [bitbake-devel] Support for git+https://foo.example.com like URIs

[Enrico Scholz]

"Richard Purdie" <richard.purdie@linuxfoundation.org> writes:

>> > PROJECT_BASE_URI = "git+https://foo.example.com/project"
>>
>> Is there some generic interest in this functionality so that I can
>> create and send a patch for bitbake?  Or is this too special and
>> people are happy with altering ${MIRROR}?
>
> I didn't (and don't) like writing special cases into general code.

mmmh... In my original report you complained that code is not generic
enough and works for git only.  Now, that is has been generalized, you
say it is too generic...


> My intent has always been (and still is) to see if a more generic
> solution is possible which not only addresses your use case but the
> ones others have as well.
>
> That will likely mean using the mirrors variable to remap things which
> will be slightly more complex but will also enable others.

I do not think, that bothering around with mirrors (or better PREMIRRORS)
is any helpful for this cases.  In 99% of cases, people want to set http://
or perhaps ssh:// as a transport protocol.

People want to completely rewrite an uri; they do not want to try another
one first and then another one.  This is nothing which can be solved by
mirrors; especially when you start to use AUTOREV.


Enrico

Re: [bitbake-devel] Support for git+https://foo.example.com like URIs

[Richard Purdie]

On Sun, 2020-06-28 at 11:48 +0200, Enrico Scholz wrote:
> "Richard Purdie" <richard.purdie@linuxfoundation.org> writes:
> 
> > > > PROJECT_BASE_URI = "git+https://foo.example.com/project"
> > > 
> > > Is there some generic interest in this functionality so that I
> > > can
> > > create and send a patch for bitbake?  Or is this too special and
> > > people are happy with altering ${MIRROR}?
> > 
> > I didn't (and don't) like writing special cases into general code.
> 
> mmmh... In my original report you complained that code is not generic
> enough and works for git only.  Now, that is has been generalized,
> you say it is too generic...

I'm making comment on what I said four years ago, I'm not making
comment on the code you have now. The code you have now is much neater
and more general. Its simple enough its worth considering.

> > My intent has always been (and still is) to see if a more generic
> > solution is possible which not only addresses your use case but the
> > ones others have as well.
> > 
> > That will likely mean using the mirrors variable to remap things
> > which
> > will be slightly more complex but will also enable others.
> 
> I do not think, that bothering around with mirrors (or better
> PREMIRRORS)

I was meaning mirrors in general, the mirrors/premirrors code all uses
the same code so yes, PREMIRRORS would be most useful in this context.

> is any helpful for this cases.  In 99% of cases, people want to set
> http:// or perhaps ssh:// as a transport protocol.
> 
> People want to completely rewrite an uri; they do not want to try
> another one first and then another one.  This is nothing which can be
> solved by mirrors; especially when you start to use AUTOREV.

The single rewritten url use case is handled by only allowing
premirrors which there is an option for. Adding more ways to do things,
each with limitations isn't usually a good idea. One way or another we
are going to have to extend the mirrors code to support the protocol
remapping.

Cheers,

Richard