[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Vivien Didelot]

When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to
specify the SHA of a reference different than a branch or tag.

For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and
Github stores the pull requests under refs/pull/xxx/head.

When cloning a repository with --bare, you don't fetch these references.
This patch uses --mirror for a full clone, in order to give the user
access to all references of the Git repository.

Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
---
 support/download/git | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/download/git b/support/download/git
index 0e6103b..357a558 100755
--- a/support/download/git
+++ b/support/download/git
@@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
 fi
 if [ ${git_done} -eq 0 ]; then
     printf "Doing full clone\n"
-    ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
+    ${GIT} clone ${verbose} --mirror "${repo}" "${basename}"
 fi
 
 GIT_DIR="${basename}" \
-- 
2.6.2

[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Maxime Hadjinlian]

Hi Vivien, all

On Sat, Nov 7, 2015 at 5:39 AM, Vivien Didelot <
vivien.didelot@savoirfairelinux.com> wrote:

> When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to
> specify the SHA of a reference different than a branch or tag.
>
> For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and
> Github stores the pull requests under refs/pull/xxx/head.
>
> When cloning a repository with --bare, you don't fetch these references.
> This patch uses --mirror for a full clone, in order to give the user
> access to all references of the Git repository.
>
> Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
> ---
>  support/download/git | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/support/download/git b/support/download/git
> index 0e6103b..357a558 100755
> --- a/support/download/git
> +++ b/support/download/git
> @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)"
> ]; then
>  fi
>  if [ ${git_done} -eq 0 ]; then
>      printf "Doing full clone\n"
> -    ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
> +    ${GIT} clone ${verbose} --mirror "${repo}" "${basename}"
>  fi
>
>  GIT_DIR="${basename}" \
> --
> 2.6.2
>
> I've read the different between bare and mirror and from what I
understand, mirrors implies bare and also maps the remotes branch to the
local branches.

Given that understanding, I am afraid I don't really get what you are
trying to do. From what I understand of Git, everything has a SHA ID, be
it, branches, tags, merge commit, even when you push -f you still have
access to previous state of your HEAD.

I cloned a repository from Github that had pull requests ongoing, I could
not find anything non-standard under .git/refs/, is there a specific
example that you could point to ? I don't know Gerrit nor how it works, so
maybe this was the issues you had ?

Finally, with the current mechanisms, you can specify a SHA, so if you are
trying to clone a PR, since it should be made from either a branch from the
local repository, or a whole other repository, you should be able to
specify it.

Again, I'm not sure I got what you were trying to do, maybe there's a
special use cases with Gerrit that I'm not aware of.

> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20151112/4abcf84a/attachment.html>

[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Vivien Didelot]

Hi Maxime,

On Nov. Thursday 12 (46) 12:02 PM, Maxime Hadjinlian wrote:
> Hi Vivien, all
> 
> On Sat, Nov 7, 2015 at 5:39 AM, Vivien Didelot <
> vivien.didelot at savoirfairelinux.com> wrote:
> 
> > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to
> > specify the SHA of a reference different than a branch or tag.
> >
> > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and
> > Github stores the pull requests under refs/pull/xxx/head.
> >
> > When cloning a repository with --bare, you don't fetch these references.
> > This patch uses --mirror for a full clone, in order to give the user
> > access to all references of the Git repository.
> >
> > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
> > ---
> >  support/download/git | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/support/download/git b/support/download/git
> > index 0e6103b..357a558 100755
> > --- a/support/download/git
> > +++ b/support/download/git
> > @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)"
> > ]; then
> >  fi
> >  if [ ${git_done} -eq 0 ]; then
> >      printf "Doing full clone\n"
> > -    ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
> > +    ${GIT} clone ${verbose} --mirror "${repo}" "${basename}"
> >  fi
> >
> >  GIT_DIR="${basename}" \
> > --
> > 2.6.2
> >
> I've read the different between bare and mirror and from what I
> understand, mirrors implies bare and also maps the remotes branch to the
> local branches.

Both maps source and target branches. But --mirror also maps *all* refs,
i.e. any (standard) ref that is not a tag or a branch.

> Given that understanding, I am afraid I don't really get what you are
> trying to do. From what I understand of Git, everything has a SHA ID, be
> it, branches, tags, merge commit, even when you push -f you still have
> access to previous state of your HEAD.
> 
> I cloned a repository from Github that had pull requests ongoing, I could
> not find anything non-standard under .git/refs/, is there a specific
> example that you could point to ? I don't know Gerrit nor how it works, so
> maybe this was the issues you had ?
> 
> Finally, with the current mechanisms, you can specify a SHA, so if you are
> trying to clone a PR, since it should be made from either a branch from the
> local repository, or a whole other repository, you should be able to
> specify it.
> 
> Again, I'm not sure I got what you were trying to do, maybe there's a
> special use cases with Gerrit that I'm not aware of.

Here's an example with Github:

    $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --bare /tmp/github.bare
    $ git -C /tmp/github.bare show refs/pull/24/head # => unknown revision
    $ git -C /tmp/github.bare show bd029a9ace8810259f73933894b665f5b0a5bb36 # => bad object

    $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --mirror /tmp/github.mirror
    $ git -C /tmp/github.mirror show refs/pull/24/head
    $ git -C /tmp/github.mirror show bd029a9ace8810259f73933894b665f5b0a5bb36

Now an example with Gerrit:

    $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --bare /tmp/gerrit.bare
    $ git -C /tmp/gerrit.bare show refs/changes/62/2462/8 # => unknown revision
    $ git -C /tmp/gerrit.bare show 6f5cf35b6644b0de2a3e005805b25e6a91a429cb # => bad object

    $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --mirror /tmp/gerrit.mirror
    $ git -C /tmp/gerrit.mirror show refs/changes/62/2462/8
    $ git -C /tmp/gerrit.mirror show 6f5cf35b6644b0de2a3e005805b25e6a91a429cb

With a mirror clone you can specify the target version as either the SHA
or the ref path.

Off-topic: cgit nicely displays these refs in its Web interface too.

Thanks,
-v

[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Maxime Hadjinlian]

Hi Vivien, all

On Thu, Nov 12, 2015 at 5:12 PM, Vivien Didelot <
vivien.didelot@savoirfairelinux.com> wrote:

> Hi Maxime,
>
> On Nov. Thursday 12 (46) 12:02 PM, Maxime Hadjinlian wrote:
> > Hi Vivien, all
> >
> > On Sat, Nov 7, 2015 at 5:39 AM, Vivien Didelot <
> > vivien.didelot at savoirfairelinux.com> wrote:
> >
> > > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want
> to
> > > specify the SHA of a reference different than a branch or tag.
> > >
> > > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx,
> and
> > > Github stores the pull requests under refs/pull/xxx/head.
> > >
> > > When cloning a repository with --bare, you don't fetch these
> references.
> > > This patch uses --mirror for a full clone, in order to give the user
> > > access to all references of the Git repository.
> > >
> > > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
> > > ---
> > >  support/download/git | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/support/download/git b/support/download/git
> > > index 0e6103b..357a558 100755
> > > --- a/support/download/git
> > > +++ b/support/download/git
> > > @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}"
> 2>&1)"
> > > ]; then
> > >  fi
> > >  if [ ${git_done} -eq 0 ]; then
> > >      printf "Doing full clone\n"
> > > -    ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
> > > +    ${GIT} clone ${verbose} --mirror "${repo}" "${basename}"
> > >  fi
> > >
> > >  GIT_DIR="${basename}" \
> > > --
> > > 2.6.2
> > >
> > I've read the different between bare and mirror and from what I
> > understand, mirrors implies bare and also maps the remotes branch to the
> > local branches.
>
> Both maps source and target branches. But --mirror also maps *all* refs,
> i.e. any (standard) ref that is not a tag or a branch.
>
> > Given that understanding, I am afraid I don't really get what you are
> > trying to do. From what I understand of Git, everything has a SHA ID, be
> > it, branches, tags, merge commit, even when you push -f you still have
> > access to previous state of your HEAD.
> >
> > I cloned a repository from Github that had pull requests ongoing, I could
> > not find anything non-standard under .git/refs/, is there a specific
> > example that you could point to ? I don't know Gerrit nor how it works,
> so
> > maybe this was the issues you had ?
> >
> > Finally, with the current mechanisms, you can specify a SHA, so if you
> are
> > trying to clone a PR, since it should be made from either a branch from
> the
> > local repository, or a whole other repository, you should be able to
> > specify it.
> >
> > Again, I'm not sure I got what you were trying to do, maybe there's a
> > special use cases with Gerrit that I'm not aware of.
>
> Here's an example with Github:
>
>     $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --bare
> /tmp/github.bare
>     $ git -C /tmp/github.bare show refs/pull/24/head # => unknown revision
>     $ git -C /tmp/github.bare show
> bd029a9ace8810259f73933894b665f5b0a5bb36 # => bad object
>
>     $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --mirror
> /tmp/github.mirror
>     $ git -C /tmp/github.mirror show refs/pull/24/head
>     $ git -C /tmp/github.mirror show
> bd029a9ace8810259f73933894b665f5b0a5bb36
>
> Now an example with Gerrit:
>
>     $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --bare
> /tmp/gerrit.bare
>     $ git -C /tmp/gerrit.bare show refs/changes/62/2462/8 # => unknown
> revision
>     $ git -C /tmp/gerrit.bare show
> 6f5cf35b6644b0de2a3e005805b25e6a91a429cb # => bad object
>
>     $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc
> --mirror /tmp/gerrit.mirror
>     $ git -C /tmp/gerrit.mirror show refs/changes/62/2462/8
>     $ git -C /tmp/gerrit.mirror show
> 6f5cf35b6644b0de2a3e005805b25e6a91a429cb
>
> With a mirror clone you can specify the target version as either the SHA
> or the ref path.
>
> Off-topic: cgit nicely displays these refs in its Web interface too.
>
> Thanks a lot for your detailed explanation, I did not knew that. With that
understanding, your changes seems perfectly fine to me.

Reviewed-by: "Maxime Hadjinlian" <maxime.hadjinlian@gmail.com>

Let's wait to see what other thinks but I don't see how it could break
anything since bare is a subset of mirror.

Thanks,
> -v
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20151112/784030e4/attachment.html>

[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Vivien Didelot]

On Nov. Thursday 12 (46) 05:26 PM, Maxime Hadjinlian wrote:
> Hi Vivien, all
> 
> On Thu, Nov 12, 2015 at 5:12 PM, Vivien Didelot <
> vivien.didelot at savoirfairelinux.com> wrote:
> 
> > Hi Maxime,
> >
> > On Nov. Thursday 12 (46) 12:02 PM, Maxime Hadjinlian wrote:
> > > Hi Vivien, all
> > >
> > > On Sat, Nov 7, 2015 at 5:39 AM, Vivien Didelot <
> > > vivien.didelot at savoirfairelinux.com> wrote:
> > >
> > > > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want
> > to
> > > > specify the SHA of a reference different than a branch or tag.
> > > >
> > > > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx,
> > and
> > > > Github stores the pull requests under refs/pull/xxx/head.
> > > >
> > > > When cloning a repository with --bare, you don't fetch these
> > references.
> > > > This patch uses --mirror for a full clone, in order to give the user
> > > > access to all references of the Git repository.
> > > >
> > > > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
> > > > ---
> > > >  support/download/git | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/support/download/git b/support/download/git
> > > > index 0e6103b..357a558 100755
> > > > --- a/support/download/git
> > > > +++ b/support/download/git
> > > > @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}"
> > 2>&1)"
> > > > ]; then
> > > >  fi
> > > >  if [ ${git_done} -eq 0 ]; then
> > > >      printf "Doing full clone\n"
> > > > -    ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
> > > > +    ${GIT} clone ${verbose} --mirror "${repo}" "${basename}"
> > > >  fi
> > > >
> > > >  GIT_DIR="${basename}" \
> > > > --
> > > > 2.6.2
> > > >
> > > I've read the different between bare and mirror and from what I
> > > understand, mirrors implies bare and also maps the remotes branch to the
> > > local branches.
> >
> > Both maps source and target branches. But --mirror also maps *all* refs,
> > i.e. any (standard) ref that is not a tag or a branch.
> >
> > > Given that understanding, I am afraid I don't really get what you are
> > > trying to do. From what I understand of Git, everything has a SHA ID, be
> > > it, branches, tags, merge commit, even when you push -f you still have
> > > access to previous state of your HEAD.
> > >
> > > I cloned a repository from Github that had pull requests ongoing, I could
> > > not find anything non-standard under .git/refs/, is there a specific
> > > example that you could point to ? I don't know Gerrit nor how it works,
> > so
> > > maybe this was the issues you had ?
> > >
> > > Finally, with the current mechanisms, you can specify a SHA, so if you
> > are
> > > trying to clone a PR, since it should be made from either a branch from
> > the
> > > local repository, or a whole other repository, you should be able to
> > > specify it.
> > >
> > > Again, I'm not sure I got what you were trying to do, maybe there's a
> > > special use cases with Gerrit that I'm not aware of.
> >
> > Here's an example with Github:
> >
> >     $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --bare
> > /tmp/github.bare
> >     $ git -C /tmp/github.bare show refs/pull/24/head # => unknown revision
> >     $ git -C /tmp/github.bare show
> > bd029a9ace8810259f73933894b665f5b0a5bb36 # => bad object
> >
> >     $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --mirror
> > /tmp/github.mirror
> >     $ git -C /tmp/github.mirror show refs/pull/24/head
> >     $ git -C /tmp/github.mirror show
> > bd029a9ace8810259f73933894b665f5b0a5bb36
> >
> > Now an example with Gerrit:
> >
> >     $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --bare
> > /tmp/gerrit.bare
> >     $ git -C /tmp/gerrit.bare show refs/changes/62/2462/8 # => unknown
> > revision
> >     $ git -C /tmp/gerrit.bare show
> > 6f5cf35b6644b0de2a3e005805b25e6a91a429cb # => bad object
> >
> >     $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc
> > --mirror /tmp/gerrit.mirror
> >     $ git -C /tmp/gerrit.mirror show refs/changes/62/2462/8
> >     $ git -C /tmp/gerrit.mirror show
> > 6f5cf35b6644b0de2a3e005805b25e6a91a429cb
> >
> > With a mirror clone you can specify the target version as either the SHA
> > or the ref path.
> >
> > Off-topic: cgit nicely displays these refs in its Web interface too.
> >
> > Thanks a lot for your detailed explanation, I did not knew that. With that
> understanding, your changes seems perfectly fine to me.
> 
> Reviewed-by: "Maxime Hadjinlian" <maxime.hadjinlian@gmail.com>
> 
> Let's wait to see what other thinks but I don't see how it could break
> anything since bare is a subset of mirror.

The change seemed OK to Thomas and Yann on IRC, I add them in the loop.

This is orthogonal, but I'm wondering if it'd make sense to stamp a git
remote update, similary to the local kernel setup + rsync. That would
ease the build process of a team, when a parallel kernel development is
on going.

Thanks for the review,
-v

[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Arnout Vandecappelle]

On 12-11-15 18:07, Vivien Didelot wrote:
[snip]
> This is orthogonal, but I'm wondering if it'd make sense to stamp a git
> remote update, similary to the local kernel setup + rsync. That would
> ease the build process of a team, when a parallel kernel development is
> on going.

 I'm not sure what you mean with stamping a git remote update. What I think
would make sense is that for distributed VCS downloads (i.e. git and hg) we keep
a full (shallow) clone in DL_DIR in addition to the tarball. That makes handling
updates a lot more efficient. We probably would want to expose that to PRIMARY
and SECONDARY download sites as well. Especially the PRIMARY would be useful,
since it would allow you to start from a local clone of e.g. linux.git (which
may not have the ref you need), and then just fetch the missing refs from upstream.

 This will not be easy to implement, though, since we'd need to handle e.g.
multiple remotes that end up in the same DL_DIR/foo.git repository.


 Was this what you meant, Vivien?

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Vivien Didelot]

Hi Arnout,

On Nov. Friday 13 (46) 07:49 AM, Arnout Vandecappelle wrote:
> On 12-11-15 18:07, Vivien Didelot wrote:
> [snip]
> > This is orthogonal, but I'm wondering if it'd make sense to stamp a git
> > remote update, similary to the local kernel setup + rsync. That would
> > ease the build process of a team, when a parallel kernel development is
> > on going.
> 
>  I'm not sure what you mean with stamping a git remote update. What I think
> would make sense is that for distributed VCS downloads (i.e. git and hg) we keep
> a full (shallow) clone in DL_DIR in addition to the tarball. That makes handling
> updates a lot more efficient. We probably would want to expose that to PRIMARY
> and SECONDARY download sites as well. Especially the PRIMARY would be useful,
> since it would allow you to start from a local clone of e.g. linux.git (which
> may not have the ref you need), and then just fetch the missing refs from upstream.
> 
>  This will not be easy to implement, though, since we'd need to handle e.g.
> multiple remotes that end up in the same DL_DIR/foo.git repository.
> 
> 
>  Was this what you meant, Vivien?

What I meant is, when you use BR2_LINUX_KERNEL_CUSTOM_LOCAL, Buildroot will 
stamp a .stamp_rsynced file and re-sync on later `make linux-rebuild`. This is 
really handy.

I don't think pointing to a Git repository is much different from this 
(especially during the development phase). Similary to rsync, `git remote 
update` can be used to sync a mirror clone (says the git-clone man page).

I'm thinking that we may either rework a generic $(1)-sync target which handles 
both rsync and `git remote update`, or introduce a $(1)-update target?

Caching full mirror clones is indeed the way to go. Several defconfig use the 
same Git repository. (e.g. 7 use linux-at91 and 7 use linux-2.6-imx). We need 
a clone to have a non-git identifier. I'd suggest to hash 
BR2_LINUX_KERNEL_CUSTOM_REPO_URL. sha1sum would give us the following paths:

"git://git.freescale.com/imx/linux-2.6-imx.git" # => linux-36a3f7006936f66cbf83b1de5bd179b2e72ceb4b.git
"https://github.com/linux4sam/linux-at91.git" # => linux-75076bea97e59045a5cfc8d84078976952d751bf.git

We might not want to store the clones in the same directory as the 
ready-to-be-extracted sources, so maybe a DL_VCS_DIR can be introduced.

Thanks,
-v

[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Yann E. MORIN]

Vivien, All,

On 2015-11-12 11:12 -0500, Vivien Didelot spake thusly:
> On Nov. Thursday 12 (46) 12:02 PM, Maxime Hadjinlian wrote:
> > On Sat, Nov 7, 2015 at 5:39 AM, Vivien Didelot <
> > vivien.didelot at savoirfairelinux.com> wrote:
> > 
> > > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to
> > > specify the SHA of a reference different than a branch or tag.
> > >
> > > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and
> > > Github stores the pull requests under refs/pull/xxx/head.
> > >
> > > When cloning a repository with --bare, you don't fetch these references.
> > > This patch uses --mirror for a full clone, in order to give the user
> > > access to all references of the Git repository.
> > >
> > > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
> > > ---
> > >  support/download/git | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/support/download/git b/support/download/git
> > > index 0e6103b..357a558 100755
> > > --- a/support/download/git
> > > +++ b/support/download/git
> > > @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)"
> > > ]; then
> > >  fi
> > >  if [ ${git_done} -eq 0 ]; then
> > >      printf "Doing full clone\n"
> > > -    ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
> > > +    ${GIT} clone ${verbose} --mirror "${repo}" "${basename}"
> > >  fi
> > >
> > >  GIT_DIR="${basename}" \
> > > --
> > > 2.6.2
> > >
> > I've read the different between bare and mirror and from what I
> > understand, mirrors implies bare and also maps the remotes branch to the
> > local branches.
> 
> Both maps source and target branches. But --mirror also maps *all* refs,
> i.e. any (standard) ref that is not a tag or a branch.
> 
> > Given that understanding, I am afraid I don't really get what you are
> > trying to do. From what I understand of Git, everything has a SHA ID, be
> > it, branches, tags, merge commit, even when you push -f you still have
> > access to previous state of your HEAD.
> > 
> > I cloned a repository from Github that had pull requests ongoing, I could
> > not find anything non-standard under .git/refs/, is there a specific
> > example that you could point to ? I don't know Gerrit nor how it works, so
> > maybe this was the issues you had ?
> > 
> > Finally, with the current mechanisms, you can specify a SHA, so if you are
> > trying to clone a PR, since it should be made from either a branch from the
> > local repository, or a whole other repository, you should be able to
> > specify it.
> > 
> > Again, I'm not sure I got what you were trying to do, maybe there's a
> > special use cases with Gerrit that I'm not aware of.
> 
> Here's an example with Github:
> 
>     $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --bare /tmp/github.bare
>     $ git -C /tmp/github.bare show refs/pull/24/head # => unknown revision
>     $ git -C /tmp/github.bare show bd029a9ace8810259f73933894b665f5b0a5bb36 # => bad object
> 
>     $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --mirror /tmp/github.mirror
>     $ git -C /tmp/github.mirror show refs/pull/24/head
>     $ git -C /tmp/github.mirror show bd029a9ace8810259f73933894b665f5b0a5bb36
> 
> Now an example with Gerrit:
> 
>     $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --bare /tmp/gerrit.bare
>     $ git -C /tmp/gerrit.bare show refs/changes/62/2462/8 # => unknown revision
>     $ git -C /tmp/gerrit.bare show 6f5cf35b6644b0de2a3e005805b25e6a91a429cb # => bad object
> 
>     $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --mirror /tmp/gerrit.mirror
>     $ git -C /tmp/gerrit.mirror show refs/changes/62/2462/8
>     $ git -C /tmp/gerrit.mirror show 6f5cf35b6644b0de2a3e005805b25e6a91a429cb
> 
> With a mirror clone you can specify the target version as either the SHA
> or the ref path.

Yes, but for PR, I'm afraid this is not very stable: one may re-push a
PR, or one may delete it (e.g. the PR initiator may remove his tree and
the PR is no longe ravailable. Or is it?).

Yet, those do not provide stable references, like sha1 and tags do
provide.

I understand it can be interesting during development, though, to
quickly test a PR without having to manually handle fetching it...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Vivien Didelot]

Hi Yann,

On Nov. Friday 13 (46) 11:48 PM, Yann E. MORIN wrote:
> Vivien, All,
> 
> On 2015-11-12 11:12 -0500, Vivien Didelot spake thusly:
> > On Nov. Thursday 12 (46) 12:02 PM, Maxime Hadjinlian wrote:
> > > On Sat, Nov 7, 2015 at 5:39 AM, Vivien Didelot <
> > > vivien.didelot at savoirfairelinux.com> wrote:
> > > 
> > > > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to
> > > > specify the SHA of a reference different than a branch or tag.
> > > >
> > > > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and
> > > > Github stores the pull requests under refs/pull/xxx/head.
> > > >
> > > > When cloning a repository with --bare, you don't fetch these references.
> > > > This patch uses --mirror for a full clone, in order to give the user
> > > > access to all references of the Git repository.
> > > >
> > > > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
> > > > ---
> > > >  support/download/git | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/support/download/git b/support/download/git
> > > > index 0e6103b..357a558 100755
> > > > --- a/support/download/git
> > > > +++ b/support/download/git
> > > > @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)"
> > > > ]; then
> > > >  fi
> > > >  if [ ${git_done} -eq 0 ]; then
> > > >      printf "Doing full clone\n"
> > > > -    ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
> > > > +    ${GIT} clone ${verbose} --mirror "${repo}" "${basename}"
> > > >  fi
> > > >
> > > >  GIT_DIR="${basename}" \
> > > > --
> > > > 2.6.2
> > > >
> > > I've read the different between bare and mirror and from what I
> > > understand, mirrors implies bare and also maps the remotes branch to the
> > > local branches.
> > 
> > Both maps source and target branches. But --mirror also maps *all* refs,
> > i.e. any (standard) ref that is not a tag or a branch.
> > 
> > > Given that understanding, I am afraid I don't really get what you are
> > > trying to do. From what I understand of Git, everything has a SHA ID, be
> > > it, branches, tags, merge commit, even when you push -f you still have
> > > access to previous state of your HEAD.
> > > 
> > > I cloned a repository from Github that had pull requests ongoing, I could
> > > not find anything non-standard under .git/refs/, is there a specific
> > > example that you could point to ? I don't know Gerrit nor how it works, so
> > > maybe this was the issues you had ?
> > > 
> > > Finally, with the current mechanisms, you can specify a SHA, so if you are
> > > trying to clone a PR, since it should be made from either a branch from the
> > > local repository, or a whole other repository, you should be able to
> > > specify it.
> > > 
> > > Again, I'm not sure I got what you were trying to do, maybe there's a
> > > special use cases with Gerrit that I'm not aware of.
> > 
> > Here's an example with Github:
> > 
> >     $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --bare /tmp/github.bare
> >     $ git -C /tmp/github.bare show refs/pull/24/head # => unknown revision
> >     $ git -C /tmp/github.bare show bd029a9ace8810259f73933894b665f5b0a5bb36 # => bad object
> > 
> >     $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --mirror /tmp/github.mirror
> >     $ git -C /tmp/github.mirror show refs/pull/24/head
> >     $ git -C /tmp/github.mirror show bd029a9ace8810259f73933894b665f5b0a5bb36
> > 
> > Now an example with Gerrit:
> > 
> >     $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --bare /tmp/gerrit.bare
> >     $ git -C /tmp/gerrit.bare show refs/changes/62/2462/8 # => unknown revision
> >     $ git -C /tmp/gerrit.bare show 6f5cf35b6644b0de2a3e005805b25e6a91a429cb # => bad object
> > 
> >     $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --mirror /tmp/gerrit.mirror
> >     $ git -C /tmp/gerrit.mirror show refs/changes/62/2462/8
> >     $ git -C /tmp/gerrit.mirror show 6f5cf35b6644b0de2a3e005805b25e6a91a429cb
> > 
> > With a mirror clone you can specify the target version as either the SHA
> > or the ref path.
> 
> Yes, but for PR, I'm afraid this is not very stable: one may re-push a
> PR, or one may delete it (e.g. the PR initiator may remove his tree and
> the PR is no longe ravailable. Or is it?).
> 
> Yet, those do not provide stable references, like sha1 and tags do
> provide.
> 
> I understand it can be interesting during development, though, to
> quickly test a PR without having to manually handle fetching it...

Github PR related blob goes into the target repository. I don't know if
Github removes them if the PR is removed from the source repository.

But note that Github was just an example. However this fixes my build
for an under-review Linux we have behind a Gerrit.

Thanks,
-v

[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Jérôme Pouiller]

Hello Arnout,

On Friday 13 November 2015 07:49:48 Arnout Vandecappelle wrote:
[...]
> What I
> think would make sense is that for distributed VCS downloads (i.e.
> git and hg) we keep a full (shallow) clone in DL_DIR in addition to
> the tarball. That makes handling updates a lot more efficient. We
> probably would want to expose that to PRIMARY and SECONDARY download
> sites as well. Especially the PRIMARY would be useful, since it would
> allow you to start from a local clone of e.g. linux.git (which may
> not have the ref you need), and then just fetch the missing refs from
> upstream.
> 
>  This will not be easy to implement, though, since we'd need to handle
> e.g. multiple remotes that end up in the same DL_DIR/foo.git
> repository.
For your information, I have in my drafts a patch which implement a 
similar idea. I think I would be able to send a first version in 1 or 2 
weeks.

BR,

-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Vivien Didelot]

On Nov. Friday 06 (45) 11:39 PM, Vivien Didelot wrote:
> When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to
> specify the SHA of a reference different than a branch or tag.
> 
> For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and
> Github stores the pull requests under refs/pull/xxx/head.
> 
> When cloning a repository with --bare, you don't fetch these references.
> This patch uses --mirror for a full clone, in order to give the user
> access to all references of the Git repository.
> 
> Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com>

Reviewed-by: "Maxime Hadjinlian" <maxime.hadjinlian@gmail.com>

(I pasted Maxime's review here for clarity)

Guys, could you merge this patch please? Also having it applied as a fix
for 2015.11 would be a plus.

Best,
-v

[Buildroot] [PATCH] support/download: fetch all refs on full git clone

[Thomas Petazzoni]

Dear Vivien Didelot,

On Fri,  6 Nov 2015 23:39:24 -0500, Vivien Didelot wrote:
> When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to
> specify the SHA of a reference different than a branch or tag.
> 
> For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and
> Github stores the pull requests under refs/pull/xxx/head.
> 
> When cloning a repository with --bare, you don't fetch these references.
> This patch uses --mirror for a full clone, in order to give the user
> access to all references of the Git repository.
> 
> Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
> ---
>  support/download/git | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Applied to next, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com