bitbake-devel.lists.openembedded.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 0/1] fetch/git: Fix local clone url to make it work with repo
@ 2023-03-24  7:09 Robert Yang
  2023-03-24  7:09 ` [PATCH 1/1] " Robert Yang
  0 siblings, 1 reply; 6+ messages in thread
From: Robert Yang @ 2023-03-24  7:09 UTC (permalink / raw)
  To: bitbake-devel; +Cc: sam, troth

The following changes since commit 92993b4f77ec49b7f04458b722ede0f8a218e0e8:

  bitbake: fetch2/sftp: Fix fetching URIs with spaces (2023-03-23 22:41:32 +0000)

are available in the Git repository at:

  https://github.com/robertlinux/yocto rbt/git
  https://github.com/robertlinux/yocto/tree/rbt/git

Robert Yang (1):
  fetch/git: Fix local clone url to make it work with repo

 bitbake/lib/bb/fetch2/git.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

-- 
2.33.1



^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 1/1] fetch/git: Fix local clone url to make it work with repo
  2023-03-24  7:09 [PATCH 0/1] fetch/git: Fix local clone url to make it work with repo Robert Yang
@ 2023-03-24  7:09 ` Robert Yang
  2023-03-24 19:50   ` Theodore A. Roth
  0 siblings, 1 reply; 6+ messages in thread
From: Robert Yang @ 2023-03-24  7:09 UTC (permalink / raw)
  To: bitbake-devel; +Cc: sam, troth

The "git clone /path/to/git/objects_symlink" couldn't work after the following
change:

https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56

But repo command manages the git repo as symlinks, so check whether the objects
is an symlink to fix the problem:

* Nothing is changed if git/objects is not a symlink
* Use "git clone file://" when git/objects is a symlink

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/fetch2/git.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index d0d68538e2..2a3c06fe4e 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -367,9 +367,13 @@ class Git(FetchMethod):
 
         # If the repo still doesn't exist, fallback to cloning it
         if not os.path.exists(ud.clonedir):
-            # We do this since git will use a "-l" option automatically for local urls where possible
+            # We do this since git will use a "-l" option automatically for local urls where possible,
+            # but it doesn't work when git/objects is a symlink, only works when it is a directory.
             if repourl.startswith("file://"):
-                repourl = repourl[7:]
+                repourl_path = repourl[7:]
+                objects = os.path.join(repourl_path, 'objects')
+                if os.path.isdir(objects) and not os.path.islink(objects):
+                    repourl = repourl_path
             clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, shlex.quote(repourl), ud.clonedir)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, clone_cmd, ud.url)
-- 
2.33.1



^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH 1/1] fetch/git: Fix local clone url to make it work with repo
  2023-03-24  7:09 ` [PATCH 1/1] " Robert Yang
@ 2023-03-24 19:50   ` Theodore A. Roth
  2023-03-26 14:18     ` Sam Liddicott
  0 siblings, 1 reply; 6+ messages in thread
From: Theodore A. Roth @ 2023-03-24 19:50 UTC (permalink / raw)
  To: Robert Yang; +Cc: bitbake-devel, sam

On Fri, Mar 24, 2023 at 1:09 AM Robert Yang <liezhi.yang@windriver.com> wrote:
>
> The "git clone /path/to/git/objects_symlink" couldn't work after the following
> change:
>
> https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56
>
> But repo command manages the git repo as symlinks, so check whether the objects
> is an symlink to fix the problem:
>
> * Nothing is changed if git/objects is not a symlink
> * Use "git clone file://" when git/objects is a symlink
>
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>

Tested-by: Theodore A. Roth <troth@openavr.org>


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH 1/1] fetch/git: Fix local clone url to make it work with repo
  2023-03-24 19:50   ` Theodore A. Roth
@ 2023-03-26 14:18     ` Sam Liddicott
  2023-03-27  5:06       ` Robert Yang
  2023-03-27  9:30       ` [bitbake-devel] " Richard Purdie
  0 siblings, 2 replies; 6+ messages in thread
From: Sam Liddicott @ 2023-03-26 14:18 UTC (permalink / raw)
  To: Theodore A. Roth; +Cc: Robert Yang, bitbake-devel

[-- Attachment #1: Type: text/plain, Size: 1161 bytes --]

Thanks for responding to this.

Does this method allow symlinking for local clones?

I have a large git repository checked out using the repo --mirror option,
which is then cloned for each target workspace.

I'd like to avoid doing large local clones for each MACHINE workspace
(there's a good dozen) but instead just symlink git objects as has been
working previously and according to the patch I submitted.

Sam

On Fri, 24 Mar 2023, 19:51 Theodore A. Roth, <troth@openavr.org> wrote:

> On Fri, Mar 24, 2023 at 1:09 AM Robert Yang <liezhi.yang@windriver.com>
> wrote:
> >
> > The "git clone /path/to/git/objects_symlink" couldn't work after the
> following
> > change:
> >
> >
> https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56
> >
> > But repo command manages the git repo as symlinks, so check whether the
> objects
> > is an symlink to fix the problem:
> >
> > * Nothing is changed if git/objects is not a symlink
> > * Use "git clone file://" when git/objects is a symlink
> >
> > Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>
> Tested-by: Theodore A. Roth <troth@openavr.org>
>

[-- Attachment #2: Type: text/html, Size: 2061 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH 1/1] fetch/git: Fix local clone url to make it work with repo
  2023-03-26 14:18     ` Sam Liddicott
@ 2023-03-27  5:06       ` Robert Yang
  2023-03-27  9:30       ` [bitbake-devel] " Richard Purdie
  1 sibling, 0 replies; 6+ messages in thread
From: Robert Yang @ 2023-03-27  5:06 UTC (permalink / raw)
  To: Sam Liddicott, Theodore A. Roth, Richard Purdie; +Cc: bitbake-devel

Hi Sam,

On 3/26/23 10:18 PM, Sam Liddicott wrote:
> Thanks for responding to this.
> 
> Does this method allow symlinking for local clones?

Did you mean "git clone -s/--shared"? No, it doesn't support that as the code 
shows. Use "git clone --shared" would save time and disk spaces, but
it would cause problems, for example, it doesn't work with 
BB_GENERATE_MIRROR_TARBALL since the .git/objects/info/alternates will be
invalid on other hosts.

I'm not sure whether we need fix other parts to make "--shared" work, let's see
whether RP has any suggestions.

> 
> I have a large git repository checked out using the repo --mirror option, which 
> is then cloned for each target workspace.

Maybe consider using a shared DL_DIR among the builds, e.g.:

DL_DIR = "/path/to/shared/downloads"

// Robert

> 
> I'd like to avoid doing large local clones for each MACHINE workspace (there's a 
> good dozen) but instead just symlink git objects as has been working previously 
> and according to the patch I submitted.
> 
> Sam
> 
> On Fri, 24 Mar 2023, 19:51 Theodore A. Roth, <troth@openavr.org 
> <mailto:troth@openavr.org>> wrote:
> 
>     On Fri, Mar 24, 2023 at 1:09 AM Robert Yang <liezhi.yang@windriver.com
>     <mailto:liezhi.yang@windriver.com>> wrote:
>      >
>      > The "git clone /path/to/git/objects_symlink" couldn't work after the
>     following
>      > change:
>      >
>      >
>     https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56
>     <https://urldefense.com/v3/__https://github.com/git/git/commit/6f054f9fb3a501c35b55c65e547a244f14c38d56__;!!AjveYdw8EvQ!bAhzaY_csImWvjvDB4Fa1yvAmmEJ32SC4duzLyDarpR7o36SD7vwI2lra1HmH7jt_x22MPCDWbmT2knQRA$>
>      >
>      > But repo command manages the git repo as symlinks, so check whether the
>     objects
>      > is an symlink to fix the problem:
>      >
>      > * Nothing is changed if git/objects is not a symlink
>      > * Use "git clone file://" when git/objects is a symlink
>      >
>      > Signed-off-by: Robert Yang <liezhi.yang@windriver.com
>     <mailto:liezhi.yang@windriver.com>>
> 
>     Tested-by: Theodore A. Roth <troth@openavr.org <mailto:troth@openavr.org>>
> 


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [bitbake-devel] [PATCH 1/1] fetch/git: Fix local clone url to make it work with repo
  2023-03-26 14:18     ` Sam Liddicott
  2023-03-27  5:06       ` Robert Yang
@ 2023-03-27  9:30       ` Richard Purdie
  1 sibling, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2023-03-27  9:30 UTC (permalink / raw)
  To: Sam Liddicott, Theodore A. Roth; +Cc: Robert Yang, bitbake-devel

On Sun, 2023-03-26 at 15:18 +0100, Sam Liddicott wrote:
> Thanks for responding to this.
> 
> Does this method allow symlinking for local clones?
> 
> I have a large git repository checked out using the repo --mirror
> option, which is then cloned for each target workspace.
> 
> I'd like to avoid doing large local clones for each MACHINE workspace
> (there's a good dozen) but instead just symlink git objects as has
> been working previously and according to the patch I submitted.

I don't think we can make everything work for everyone.

As Robert mentioned, using the alternates mechanism will break the
mirror code. I think some users will also have issues if DL_DIR changes
break the checkout from under them.

We are going to have to do something to address the changes made in
upstream git and Robert's patch looks like a reasonable step to me to
fix that.

I'm open to other proposals but we can't regress other workflows. I'm
also reluctant to complicate the code more than we need to, there are
already far too many different codepaths we have to worry about :/.

Cheers,

Richard


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2023-03-27  9:30 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-24  7:09 [PATCH 0/1] fetch/git: Fix local clone url to make it work with repo Robert Yang
2023-03-24  7:09 ` [PATCH 1/1] " Robert Yang
2023-03-24 19:50   ` Theodore A. Roth
2023-03-26 14:18     ` Sam Liddicott
2023-03-27  5:06       ` Robert Yang
2023-03-27  9:30       ` [bitbake-devel] " Richard Purdie

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).