Re: [bitbake-devel] [PATCH 1/2] fetch/git: Handle github dropping git:// support

Re: [bitbake-devel] [PATCH 1/2] fetch/git: Handle github dropping git:// support

[Richard Purdie]

On Tue, 2021-11-02 at 12:44 +0000, Richard Purdie via lists.openembedded.org
wrote:
> github is dropping support for git procotol iun git urls. Add code to remap
> this to https in a way that could be used in older bitbake versions.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  lib/bb/fetch2/git.py | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> index e974445fbe..f5be24d6ab 100644
> --- a/lib/bb/fetch2/git.py
> +++ b/lib/bb/fetch2/git.py
> @@ -142,6 +142,10 @@ class Git(FetchMethod):
>              ud.proto = 'file'
>          else:
>              ud.proto = "git"
> +        if ud.host == "github.com" and ud.proto == "git":
> +            # github stopped supporting git protocol
> +            # https://github.blog/2021-09-01-improving-git-protocol-security-github/#no-more-unauthenticated-git
> +            ud.proto = "https"
>  
>          if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
>              raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)

To make things a bit easier for everyone, I did backport this to a number of the
older bitbake branches back to 1.40.

Cheers,

Richard

Re: [PATCH 1/2] fetch/git: Handle github dropping git:// support

[gabriel.valcazar]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

Hi Richard,

Are there any plans to backport this fix to older versions of bitbake/poky? My company has a line of legacy embedded products that are only supported in older versions of our Yocto distribution (1.3 in the most extreme case), and although we can manually change the protocol from git to https in the recipes in our layer, we can't do the same for other third party layers our distribution depends on (meta-qt5, meta-freescale, poky...). Since backporting the fix to older bitbake versions is trivial, do you intend to incorporate it in older bibake/poky versions in the near future?

Otherwise, are there any alternative approaches we can take to ensure that users that depend on our legacy products can continue to build our Yocto distribution? Upgrading our entire legacy line to newer Yocto versions is not a viable solution for us.

Thanks and best regards,
Gabriel

[-- Attachment #2: Type: text/html, Size: 919 bytes --]

Re: [bitbake-devel] [PATCH 1/2] fetch/git: Handle github dropping git:// support

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1674 bytes --]

If the versions are very old (anything older than dunfell really), I think
the only viable path for you is to make private copies of respective yocto
repositories, and patch them as needed.

Alex

On Tue, 9 Nov 2021 at 16:40, gabriel.valcazar via lists.openembedded.org
<gabriel.valcazar=digi.com@lists.openembedded.org> wrote:

> Hi Richard,
>
> Are there any plans to backport this fix to older versions of
> bitbake/poky? My company has a line of legacy embedded products that are
> only supported in older versions of our Yocto distribution (1.3 in the most
> extreme case), and although we can manually change the protocol from git to
> https in the recipes in our layer, we can't do the same for other third
> party layers our distribution depends on (meta-qt5, meta-freescale,
> poky...). Since backporting the fix to older bitbake versions is trivial,
> do you intend to incorporate it in older bibake/poky versions in the near
> future?
>
> Otherwise, are there any alternative approaches we can take to ensure that
> users that depend on our legacy products can continue to build our Yocto
> distribution? Upgrading our entire legacy line to newer Yocto versions is
> not a viable solution for us.
>
> Thanks and best regards,
> Gabriel
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#12982):
> https://lists.openembedded.org/g/bitbake-devel/message/12982
> Mute This Topic: https://lists.openembedded.org/mt/86764631/1686489
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 2511 bytes --]

Re: [bitbake-devel] [PATCH 1/2] fetch/git: Handle github dropping git:// support

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1932 bytes --]

FWIW: It was already backported all the way to 1.40 version used in Yocto
2.6 Thud:
https://git.openembedded.org/bitbake/log/?h=1.40

All the way till 1.16 version used in Yocto 1.3 Danny, is still quite long.
Maybe you can backport the bitbake patch on your end? I would be surprised
if you don't have bunch of various other changes backported just to keep
these ancient builds going _and_ reliable PREMIRROR to never access
upstream repos from 2012.

On Tue, Nov 9, 2021 at 4:40 PM gabriel.valcazar via lists.openembedded.org
<gabriel.valcazar=digi.com@lists.openembedded.org> wrote:

> Hi Richard,
>
> Are there any plans to backport this fix to older versions of
> bitbake/poky? My company has a line of legacy embedded products that are
> only supported in older versions of our Yocto distribution (1.3 in the most
> extreme case), and although we can manually change the protocol from git to
> https in the recipes in our layer, we can't do the same for other third
> party layers our distribution depends on (meta-qt5, meta-freescale,
> poky...). Since backporting the fix to older bitbake versions is trivial,
> do you intend to incorporate it in older bibake/poky versions in the near
> future?
>
> Otherwise, are there any alternative approaches we can take to ensure that
> users that depend on our legacy products can continue to build our Yocto
> distribution? Upgrading our entire legacy line to newer Yocto versions is
> not a viable solution for us.
>
> Thanks and best regards,
> Gabriel
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#12982):
> https://lists.openembedded.org/g/bitbake-devel/message/12982
> Mute This Topic: https://lists.openembedded.org/mt/86764631/3617156
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> Martin.Jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 2831 bytes --]

Re: [PATCH 1/2] fetch/git: Handle github dropping git:// support

[gabriel.valcazar]

[-- Attachment #1: Type: text/plain, Size: 508 bytes --]

Thanks all for the suggestions. Indeed, creating a fork of poky and applying the fix ourselves seems like a reasonable solution if there is no official fix for old versions.

I've just tested applying the patch to *bitbake 1.18* , and I'm able to build my Yocto 1.4 distribution with no problem. Is it not possible to officially backport the patch all the way back to bitbake 1.18? I could provide the patches, but they're literally just clean cherry-picks of the bitbake 1.40 patch.

Regards,
Gabriel

[-- Attachment #2: Type: text/html, Size: 542 bytes --]

Re: [bitbake-devel] [PATCH 1/2] fetch/git: Handle github dropping git:// support

[Richard Purdie]

On Wed, 2021-11-10 at 07:15 -0800, gabriel.valcazar via lists.openembedded.org
wrote:
> Thanks all for the suggestions. Indeed, creating a fork of poky and applying
> the
> fix ourselves seems like a reasonable solution if there is no official fix for
> old versions.
> 
> I've just tested applying the patch to bitbake 1.18, and I'm able to build my
> Yocto 1.4 distribution with no problem. Is it not possible to officially
> backport the patch all the way back to bitbake 1.18? I could provide the
> patches, but they're literally just clean cherry-picks of the bitbake 1.40
> patch.

The trouble is there are a *lot* of fixes missing from 1.18. If we start
backporting this one, which other ones should we backport? If I don't say "no"
to this one, which other ones would I then need to take?

I appreciate this one is relatively safe and easy but it is also very easy to
backport something which applies cleanly but causes all kinds of issues.

We can't really start giving the impression we support things as old as 1.18,
much as I'd like to help. I can't imagine this is the only fix you need with
1.18?

Cheers,

Richard

Re: [PATCH 1/2] fetch/git: Handle github dropping git:// support

[gabriel.valcazar]

[-- Attachment #1: Type: text/plain, Size: 1920 bytes --]

Hi Richard,

Thanks for the explanation. I fully understand the reasoning behind your decision, but I really think this issue requires specific treatment due to its nature.

Ideally, customers using our legacy products (and thus, really old versions of our Yocto distribution) would migrate to newer products running newer software, and these situations would be avoided altogether. However, we're talking about products that are already in production, with several devices in the field that require periodic updates. Maintaining outdated software, while not recommendable from several standpoints, is oftentimes a more efficient approach than migrating to something newer, especially in these cases.

Older Yocto builds are still possible for us and our customers via docker containers or virtual machines, which provide "old" environments where the builds still work, despite all of the missing fixes in bitbake/poky. Basically, rather than patching the Yocto stack to fit our environment, we adapt our environment to fit the Yocto stack - again, because several legacy customers have made the decision to depend on old Yocto versions.

Having said this, I believe GitHub's deprecation of the git protocol is a special case because, regardless of the environment you're using or how many patches you've backported so far, builds will simply stop working altogether after the plug is pulled. No matter which environment we use, this is an external factor that's going to force changes in the Yocto stack, in areas that we have little to no control over. We're going to have to fix it either way, but it would be a much smoother transition if the fix were included by the community, and it's a unique enough case to warrant it being patched in older bitbake versions (in my opinion).

Forgive me for the insistence, but I truly believe this situation requires exceptional treatment.

Best regards,
Gabriel

[-- Attachment #2: Type: text/html, Size: 1981 bytes --]

Re: [bitbake-devel] [PATCH 1/2] fetch/git: Handle github dropping git:// support

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2807 bytes --]

Are your customers paying anyone in the ‘community’ for this work or any
other work we do? Seriously, if you want anyone to pay attention to ancient
yocto versions, you have to support that financially.

Alex

On Thu 11. Nov 2021 at 13.18, gabriel.valcazar via lists.openembedded.org
<gabriel.valcazar=digi.com@lists.openembedded.org> wrote:

> Hi Richard,
>
> Thanks for the explanation. I fully understand the reasoning behind your
> decision, but I really think this issue requires specific treatment due to
> its nature.
>
> Ideally, customers using our legacy products (and thus, really old
> versions of our Yocto distribution) would migrate to newer products running
> newer software, and these situations would be avoided altogether. However,
> we're talking about products that are already in production, with several
> devices in the field that require periodic updates. Maintaining outdated
> software, while not recommendable from several standpoints, is oftentimes a
> more efficient approach than migrating to something newer, especially in
> these cases.
>
> Older Yocto builds are still possible for us and our customers via docker
> containers or virtual machines, which provide "old" environments where the
> builds still work, despite all of the missing fixes in bitbake/poky.
> Basically, rather than patching the Yocto stack to fit our environment, we
> adapt our environment to fit the Yocto stack - again, because several
> legacy customers have made the decision to depend on old Yocto versions.
>
> Having said this, I believe GitHub's deprecation of the *git *protocol is
> a special case because, regardless of the environment you're using or how
> many patches you've backported so far, builds will simply stop working
> altogether after the plug is pulled. No matter which environment we use,
> this is an external factor that's going to force changes in the Yocto
> stack, in areas that we have little to no control over. We're going to have
> to fix it either way, but it would be a much smoother transition if the fix
> were included by the community, and it's a unique enough case to warrant it
> being patched in older bitbake versions (in my opinion).
>
> Forgive me for the insistence, but I truly believe this situation requires
> exceptional treatment.
>
> Best regards,
> Gabriel
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#13015):
> https://lists.openembedded.org/g/bitbake-devel/message/13015
> Mute This Topic: https://lists.openembedded.org/mt/86764631/1686489
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 3674 bytes --]

Re: [PATCH 1/2] fetch/git: Handle github dropping git:// support

[gabriel.valcazar]

[-- Attachment #1: Type: text/plain, Size: 557 bytes --]

Hi Richard,

Thanks for your sincerity, you have every right to be frustrated with my request when you put it that way. It's not fair of us to ask you to introduce a change that helps us with our very specific situation; and have you deal with all of the consequences. I was hoping my request would be harmless but I wasn't fully aware of its implications, so I appreciate you shedding some light so I could get the full picture.

We'll get this fixed on our end and stop clogging up this thread. Apologies for the annoyance.

Best regards,
Gabriel

[-- Attachment #2: Type: text/html, Size: 585 bytes --]

[PATCH 1/2] fetch/git: Handle github dropping git:// support

[Richard Purdie]

github is dropping support for git procotol iun git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/fetch2/git.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index e974445fbe..f5be24d6ab 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -142,6 +142,10 @@ class Git(FetchMethod):
             ud.proto = 'file'
         else:
             ud.proto = "git"
+        if ud.host == "github.com" and ud.proto == "git":
+            # github stopped supporting git protocol
+            # https://github.blog/2021-09-01-improving-git-protocol-security-github/#no-more-unauthenticated-git
+            ud.proto = "https"
 
         if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
             raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
-- 
2.32.0