* [Buildroot] [PATCH] support/download: fetch all refs on full git clone @ 2015-11-07 4:39 Vivien Didelot 2015-11-12 11:02 ` Maxime Hadjinlian ` (2 more replies) 0 siblings, 3 replies; 12+ messages in thread From: Vivien Didelot @ 2015-11-07 4:39 UTC (permalink / raw) To: buildroot When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to specify the SHA of a reference different than a branch or tag. For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and Github stores the pull requests under refs/pull/xxx/head. When cloning a repository with --bare, you don't fetch these references. This patch uses --mirror for a full clone, in order to give the user access to all references of the Git repository. Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com> --- support/download/git | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/support/download/git b/support/download/git index 0e6103b..357a558 100755 --- a/support/download/git +++ b/support/download/git @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then fi if [ ${git_done} -eq 0 ]; then printf "Doing full clone\n" - ${GIT} clone ${verbose} --bare "${repo}" "${basename}" + ${GIT} clone ${verbose} --mirror "${repo}" "${basename}" fi GIT_DIR="${basename}" \ -- 2.6.2 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH] support/download: fetch all refs on full git clone 2015-11-07 4:39 [Buildroot] [PATCH] support/download: fetch all refs on full git clone Vivien Didelot @ 2015-11-12 11:02 ` Maxime Hadjinlian 2015-11-12 16:12 ` Vivien Didelot 2015-11-19 17:38 ` Vivien Didelot 2015-11-29 17:53 ` Thomas Petazzoni 2 siblings, 1 reply; 12+ messages in thread From: Maxime Hadjinlian @ 2015-11-12 11:02 UTC (permalink / raw) To: buildroot Hi Vivien, all On Sat, Nov 7, 2015 at 5:39 AM, Vivien Didelot < vivien.didelot@savoirfairelinux.com> wrote: > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to > specify the SHA of a reference different than a branch or tag. > > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and > Github stores the pull requests under refs/pull/xxx/head. > > When cloning a repository with --bare, you don't fetch these references. > This patch uses --mirror for a full clone, in order to give the user > access to all references of the Git repository. > > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com> > --- > support/download/git | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/support/download/git b/support/download/git > index 0e6103b..357a558 100755 > --- a/support/download/git > +++ b/support/download/git > @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" > ]; then > fi > if [ ${git_done} -eq 0 ]; then > printf "Doing full clone\n" > - ${GIT} clone ${verbose} --bare "${repo}" "${basename}" > + ${GIT} clone ${verbose} --mirror "${repo}" "${basename}" > fi > > GIT_DIR="${basename}" \ > -- > 2.6.2 > > I've read the different between bare and mirror and from what I understand, mirrors implies bare and also maps the remotes branch to the local branches. Given that understanding, I am afraid I don't really get what you are trying to do. From what I understand of Git, everything has a SHA ID, be it, branches, tags, merge commit, even when you push -f you still have access to previous state of your HEAD. I cloned a repository from Github that had pull requests ongoing, I could not find anything non-standard under .git/refs/, is there a specific example that you could point to ? I don't know Gerrit nor how it works, so maybe this was the issues you had ? Finally, with the current mechanisms, you can specify a SHA, so if you are trying to clone a PR, since it should be made from either a branch from the local repository, or a whole other repository, you should be able to specify it. Again, I'm not sure I got what you were trying to do, maybe there's a special use cases with Gerrit that I'm not aware of. > _______________________________________________ > buildroot mailing list > buildroot at busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20151112/4abcf84a/attachment.html> ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH] support/download: fetch all refs on full git clone 2015-11-12 11:02 ` Maxime Hadjinlian @ 2015-11-12 16:12 ` Vivien Didelot 2015-11-12 16:26 ` Maxime Hadjinlian 2015-11-13 22:48 ` Yann E. MORIN 0 siblings, 2 replies; 12+ messages in thread From: Vivien Didelot @ 2015-11-12 16:12 UTC (permalink / raw) To: buildroot Hi Maxime, On Nov. Thursday 12 (46) 12:02 PM, Maxime Hadjinlian wrote: > Hi Vivien, all > > On Sat, Nov 7, 2015 at 5:39 AM, Vivien Didelot < > vivien.didelot at savoirfairelinux.com> wrote: > > > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to > > specify the SHA of a reference different than a branch or tag. > > > > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and > > Github stores the pull requests under refs/pull/xxx/head. > > > > When cloning a repository with --bare, you don't fetch these references. > > This patch uses --mirror for a full clone, in order to give the user > > access to all references of the Git repository. > > > > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com> > > --- > > support/download/git | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/support/download/git b/support/download/git > > index 0e6103b..357a558 100755 > > --- a/support/download/git > > +++ b/support/download/git > > @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" > > ]; then > > fi > > if [ ${git_done} -eq 0 ]; then > > printf "Doing full clone\n" > > - ${GIT} clone ${verbose} --bare "${repo}" "${basename}" > > + ${GIT} clone ${verbose} --mirror "${repo}" "${basename}" > > fi > > > > GIT_DIR="${basename}" \ > > -- > > 2.6.2 > > > I've read the different between bare and mirror and from what I > understand, mirrors implies bare and also maps the remotes branch to the > local branches. Both maps source and target branches. But --mirror also maps *all* refs, i.e. any (standard) ref that is not a tag or a branch. > Given that understanding, I am afraid I don't really get what you are > trying to do. From what I understand of Git, everything has a SHA ID, be > it, branches, tags, merge commit, even when you push -f you still have > access to previous state of your HEAD. > > I cloned a repository from Github that had pull requests ongoing, I could > not find anything non-standard under .git/refs/, is there a specific > example that you could point to ? I don't know Gerrit nor how it works, so > maybe this was the issues you had ? > > Finally, with the current mechanisms, you can specify a SHA, so if you are > trying to clone a PR, since it should be made from either a branch from the > local repository, or a whole other repository, you should be able to > specify it. > > Again, I'm not sure I got what you were trying to do, maybe there's a > special use cases with Gerrit that I'm not aware of. Here's an example with Github: $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --bare /tmp/github.bare $ git -C /tmp/github.bare show refs/pull/24/head # => unknown revision $ git -C /tmp/github.bare show bd029a9ace8810259f73933894b665f5b0a5bb36 # => bad object $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --mirror /tmp/github.mirror $ git -C /tmp/github.mirror show refs/pull/24/head $ git -C /tmp/github.mirror show bd029a9ace8810259f73933894b665f5b0a5bb36 Now an example with Gerrit: $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --bare /tmp/gerrit.bare $ git -C /tmp/gerrit.bare show refs/changes/62/2462/8 # => unknown revision $ git -C /tmp/gerrit.bare show 6f5cf35b6644b0de2a3e005805b25e6a91a429cb # => bad object $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --mirror /tmp/gerrit.mirror $ git -C /tmp/gerrit.mirror show refs/changes/62/2462/8 $ git -C /tmp/gerrit.mirror show 6f5cf35b6644b0de2a3e005805b25e6a91a429cb With a mirror clone you can specify the target version as either the SHA or the ref path. Off-topic: cgit nicely displays these refs in its Web interface too. Thanks, -v ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH] support/download: fetch all refs on full git clone 2015-11-12 16:12 ` Vivien Didelot @ 2015-11-12 16:26 ` Maxime Hadjinlian 2015-11-12 17:07 ` Vivien Didelot 2015-11-13 22:48 ` Yann E. MORIN 1 sibling, 1 reply; 12+ messages in thread From: Maxime Hadjinlian @ 2015-11-12 16:26 UTC (permalink / raw) To: buildroot Hi Vivien, all On Thu, Nov 12, 2015 at 5:12 PM, Vivien Didelot < vivien.didelot@savoirfairelinux.com> wrote: > Hi Maxime, > > On Nov. Thursday 12 (46) 12:02 PM, Maxime Hadjinlian wrote: > > Hi Vivien, all > > > > On Sat, Nov 7, 2015 at 5:39 AM, Vivien Didelot < > > vivien.didelot at savoirfairelinux.com> wrote: > > > > > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want > to > > > specify the SHA of a reference different than a branch or tag. > > > > > > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, > and > > > Github stores the pull requests under refs/pull/xxx/head. > > > > > > When cloning a repository with --bare, you don't fetch these > references. > > > This patch uses --mirror for a full clone, in order to give the user > > > access to all references of the Git repository. > > > > > > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com> > > > --- > > > support/download/git | 2 +- > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > diff --git a/support/download/git b/support/download/git > > > index 0e6103b..357a558 100755 > > > --- a/support/download/git > > > +++ b/support/download/git > > > @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" > 2>&1)" > > > ]; then > > > fi > > > if [ ${git_done} -eq 0 ]; then > > > printf "Doing full clone\n" > > > - ${GIT} clone ${verbose} --bare "${repo}" "${basename}" > > > + ${GIT} clone ${verbose} --mirror "${repo}" "${basename}" > > > fi > > > > > > GIT_DIR="${basename}" \ > > > -- > > > 2.6.2 > > > > > I've read the different between bare and mirror and from what I > > understand, mirrors implies bare and also maps the remotes branch to the > > local branches. > > Both maps source and target branches. But --mirror also maps *all* refs, > i.e. any (standard) ref that is not a tag or a branch. > > > Given that understanding, I am afraid I don't really get what you are > > trying to do. From what I understand of Git, everything has a SHA ID, be > > it, branches, tags, merge commit, even when you push -f you still have > > access to previous state of your HEAD. > > > > I cloned a repository from Github that had pull requests ongoing, I could > > not find anything non-standard under .git/refs/, is there a specific > > example that you could point to ? I don't know Gerrit nor how it works, > so > > maybe this was the issues you had ? > > > > Finally, with the current mechanisms, you can specify a SHA, so if you > are > > trying to clone a PR, since it should be made from either a branch from > the > > local repository, or a whole other repository, you should be able to > > specify it. > > > > Again, I'm not sure I got what you were trying to do, maybe there's a > > special use cases with Gerrit that I'm not aware of. > > Here's an example with Github: > > $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --bare > /tmp/github.bare > $ git -C /tmp/github.bare show refs/pull/24/head # => unknown revision > $ git -C /tmp/github.bare show > bd029a9ace8810259f73933894b665f5b0a5bb36 # => bad object > > $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --mirror > /tmp/github.mirror > $ git -C /tmp/github.mirror show refs/pull/24/head > $ git -C /tmp/github.mirror show > bd029a9ace8810259f73933894b665f5b0a5bb36 > > Now an example with Gerrit: > > $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --bare > /tmp/gerrit.bare > $ git -C /tmp/gerrit.bare show refs/changes/62/2462/8 # => unknown > revision > $ git -C /tmp/gerrit.bare show > 6f5cf35b6644b0de2a3e005805b25e6a91a429cb # => bad object > > $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc > --mirror /tmp/gerrit.mirror > $ git -C /tmp/gerrit.mirror show refs/changes/62/2462/8 > $ git -C /tmp/gerrit.mirror show > 6f5cf35b6644b0de2a3e005805b25e6a91a429cb > > With a mirror clone you can specify the target version as either the SHA > or the ref path. > > Off-topic: cgit nicely displays these refs in its Web interface too. > > Thanks a lot for your detailed explanation, I did not knew that. With that understanding, your changes seems perfectly fine to me. Reviewed-by: "Maxime Hadjinlian" <maxime.hadjinlian@gmail.com> Let's wait to see what other thinks but I don't see how it could break anything since bare is a subset of mirror. Thanks, > -v > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20151112/784030e4/attachment.html> ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH] support/download: fetch all refs on full git clone 2015-11-12 16:26 ` Maxime Hadjinlian @ 2015-11-12 17:07 ` Vivien Didelot 2015-11-13 6:49 ` Arnout Vandecappelle 0 siblings, 1 reply; 12+ messages in thread From: Vivien Didelot @ 2015-11-12 17:07 UTC (permalink / raw) To: buildroot On Nov. Thursday 12 (46) 05:26 PM, Maxime Hadjinlian wrote: > Hi Vivien, all > > On Thu, Nov 12, 2015 at 5:12 PM, Vivien Didelot < > vivien.didelot at savoirfairelinux.com> wrote: > > > Hi Maxime, > > > > On Nov. Thursday 12 (46) 12:02 PM, Maxime Hadjinlian wrote: > > > Hi Vivien, all > > > > > > On Sat, Nov 7, 2015 at 5:39 AM, Vivien Didelot < > > > vivien.didelot at savoirfairelinux.com> wrote: > > > > > > > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want > > to > > > > specify the SHA of a reference different than a branch or tag. > > > > > > > > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, > > and > > > > Github stores the pull requests under refs/pull/xxx/head. > > > > > > > > When cloning a repository with --bare, you don't fetch these > > references. > > > > This patch uses --mirror for a full clone, in order to give the user > > > > access to all references of the Git repository. > > > > > > > > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com> > > > > --- > > > > support/download/git | 2 +- > > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > > > diff --git a/support/download/git b/support/download/git > > > > index 0e6103b..357a558 100755 > > > > --- a/support/download/git > > > > +++ b/support/download/git > > > > @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" > > 2>&1)" > > > > ]; then > > > > fi > > > > if [ ${git_done} -eq 0 ]; then > > > > printf "Doing full clone\n" > > > > - ${GIT} clone ${verbose} --bare "${repo}" "${basename}" > > > > + ${GIT} clone ${verbose} --mirror "${repo}" "${basename}" > > > > fi > > > > > > > > GIT_DIR="${basename}" \ > > > > -- > > > > 2.6.2 > > > > > > > I've read the different between bare and mirror and from what I > > > understand, mirrors implies bare and also maps the remotes branch to the > > > local branches. > > > > Both maps source and target branches. But --mirror also maps *all* refs, > > i.e. any (standard) ref that is not a tag or a branch. > > > > > Given that understanding, I am afraid I don't really get what you are > > > trying to do. From what I understand of Git, everything has a SHA ID, be > > > it, branches, tags, merge commit, even when you push -f you still have > > > access to previous state of your HEAD. > > > > > > I cloned a repository from Github that had pull requests ongoing, I could > > > not find anything non-standard under .git/refs/, is there a specific > > > example that you could point to ? I don't know Gerrit nor how it works, > > so > > > maybe this was the issues you had ? > > > > > > Finally, with the current mechanisms, you can specify a SHA, so if you > > are > > > trying to clone a PR, since it should be made from either a branch from > > the > > > local repository, or a whole other repository, you should be able to > > > specify it. > > > > > > Again, I'm not sure I got what you were trying to do, maybe there's a > > > special use cases with Gerrit that I'm not aware of. > > > > Here's an example with Github: > > > > $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --bare > > /tmp/github.bare > > $ git -C /tmp/github.bare show refs/pull/24/head # => unknown revision > > $ git -C /tmp/github.bare show > > bd029a9ace8810259f73933894b665f5b0a5bb36 # => bad object > > > > $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --mirror > > /tmp/github.mirror > > $ git -C /tmp/github.mirror show refs/pull/24/head > > $ git -C /tmp/github.mirror show > > bd029a9ace8810259f73933894b665f5b0a5bb36 > > > > Now an example with Gerrit: > > > > $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --bare > > /tmp/gerrit.bare > > $ git -C /tmp/gerrit.bare show refs/changes/62/2462/8 # => unknown > > revision > > $ git -C /tmp/gerrit.bare show > > 6f5cf35b6644b0de2a3e005805b25e6a91a429cb # => bad object > > > > $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc > > --mirror /tmp/gerrit.mirror > > $ git -C /tmp/gerrit.mirror show refs/changes/62/2462/8 > > $ git -C /tmp/gerrit.mirror show > > 6f5cf35b6644b0de2a3e005805b25e6a91a429cb > > > > With a mirror clone you can specify the target version as either the SHA > > or the ref path. > > > > Off-topic: cgit nicely displays these refs in its Web interface too. > > > > Thanks a lot for your detailed explanation, I did not knew that. With that > understanding, your changes seems perfectly fine to me. > > Reviewed-by: "Maxime Hadjinlian" <maxime.hadjinlian@gmail.com> > > Let's wait to see what other thinks but I don't see how it could break > anything since bare is a subset of mirror. The change seemed OK to Thomas and Yann on IRC, I add them in the loop. This is orthogonal, but I'm wondering if it'd make sense to stamp a git remote update, similary to the local kernel setup + rsync. That would ease the build process of a team, when a parallel kernel development is on going. Thanks for the review, -v ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH] support/download: fetch all refs on full git clone 2015-11-12 17:07 ` Vivien Didelot @ 2015-11-13 6:49 ` Arnout Vandecappelle 2015-11-13 15:35 ` Vivien Didelot 2015-11-16 8:43 ` Jérôme Pouiller 0 siblings, 2 replies; 12+ messages in thread From: Arnout Vandecappelle @ 2015-11-13 6:49 UTC (permalink / raw) To: buildroot On 12-11-15 18:07, Vivien Didelot wrote: [snip] > This is orthogonal, but I'm wondering if it'd make sense to stamp a git > remote update, similary to the local kernel setup + rsync. That would > ease the build process of a team, when a parallel kernel development is > on going. I'm not sure what you mean with stamping a git remote update. What I think would make sense is that for distributed VCS downloads (i.e. git and hg) we keep a full (shallow) clone in DL_DIR in addition to the tarball. That makes handling updates a lot more efficient. We probably would want to expose that to PRIMARY and SECONDARY download sites as well. Especially the PRIMARY would be useful, since it would allow you to start from a local clone of e.g. linux.git (which may not have the ref you need), and then just fetch the missing refs from upstream. This will not be easy to implement, though, since we'd need to handle e.g. multiple remotes that end up in the same DL_DIR/foo.git repository. Was this what you meant, Vivien? Regards, Arnout -- Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH] support/download: fetch all refs on full git clone 2015-11-13 6:49 ` Arnout Vandecappelle @ 2015-11-13 15:35 ` Vivien Didelot 2015-11-16 8:43 ` Jérôme Pouiller 1 sibling, 0 replies; 12+ messages in thread From: Vivien Didelot @ 2015-11-13 15:35 UTC (permalink / raw) To: buildroot Hi Arnout, On Nov. Friday 13 (46) 07:49 AM, Arnout Vandecappelle wrote: > On 12-11-15 18:07, Vivien Didelot wrote: > [snip] > > This is orthogonal, but I'm wondering if it'd make sense to stamp a git > > remote update, similary to the local kernel setup + rsync. That would > > ease the build process of a team, when a parallel kernel development is > > on going. > > I'm not sure what you mean with stamping a git remote update. What I think > would make sense is that for distributed VCS downloads (i.e. git and hg) we keep > a full (shallow) clone in DL_DIR in addition to the tarball. That makes handling > updates a lot more efficient. We probably would want to expose that to PRIMARY > and SECONDARY download sites as well. Especially the PRIMARY would be useful, > since it would allow you to start from a local clone of e.g. linux.git (which > may not have the ref you need), and then just fetch the missing refs from upstream. > > This will not be easy to implement, though, since we'd need to handle e.g. > multiple remotes that end up in the same DL_DIR/foo.git repository. > > > Was this what you meant, Vivien? What I meant is, when you use BR2_LINUX_KERNEL_CUSTOM_LOCAL, Buildroot will stamp a .stamp_rsynced file and re-sync on later `make linux-rebuild`. This is really handy. I don't think pointing to a Git repository is much different from this (especially during the development phase). Similary to rsync, `git remote update` can be used to sync a mirror clone (says the git-clone man page). I'm thinking that we may either rework a generic $(1)-sync target which handles both rsync and `git remote update`, or introduce a $(1)-update target? Caching full mirror clones is indeed the way to go. Several defconfig use the same Git repository. (e.g. 7 use linux-at91 and 7 use linux-2.6-imx). We need a clone to have a non-git identifier. I'd suggest to hash BR2_LINUX_KERNEL_CUSTOM_REPO_URL. sha1sum would give us the following paths: "git://git.freescale.com/imx/linux-2.6-imx.git" # => linux-36a3f7006936f66cbf83b1de5bd179b2e72ceb4b.git "https://github.com/linux4sam/linux-at91.git" # => linux-75076bea97e59045a5cfc8d84078976952d751bf.git We might not want to store the clones in the same directory as the ready-to-be-extracted sources, so maybe a DL_VCS_DIR can be introduced. Thanks, -v ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH] support/download: fetch all refs on full git clone 2015-11-13 6:49 ` Arnout Vandecappelle 2015-11-13 15:35 ` Vivien Didelot @ 2015-11-16 8:43 ` Jérôme Pouiller 1 sibling, 0 replies; 12+ messages in thread From: Jérôme Pouiller @ 2015-11-16 8:43 UTC (permalink / raw) To: buildroot Hello Arnout, On Friday 13 November 2015 07:49:48 Arnout Vandecappelle wrote: [...] > What I > think would make sense is that for distributed VCS downloads (i.e. > git and hg) we keep a full (shallow) clone in DL_DIR in addition to > the tarball. That makes handling updates a lot more efficient. We > probably would want to expose that to PRIMARY and SECONDARY download > sites as well. Especially the PRIMARY would be useful, since it would > allow you to start from a local clone of e.g. linux.git (which may > not have the ref you need), and then just fetch the missing refs from > upstream. > > This will not be easy to implement, though, since we'd need to handle > e.g. multiple remotes that end up in the same DL_DIR/foo.git > repository. For your information, I have in my drafts a patch which implement a similar idea. I think I would be able to send a first version in 1 or 2 weeks. BR, -- J?r?me Pouiller, Sysmic Embedded Linux specialist http://www.sysmic.fr ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH] support/download: fetch all refs on full git clone 2015-11-12 16:12 ` Vivien Didelot 2015-11-12 16:26 ` Maxime Hadjinlian @ 2015-11-13 22:48 ` Yann E. MORIN 2015-11-13 23:04 ` Vivien Didelot 1 sibling, 1 reply; 12+ messages in thread From: Yann E. MORIN @ 2015-11-13 22:48 UTC (permalink / raw) To: buildroot Vivien, All, On 2015-11-12 11:12 -0500, Vivien Didelot spake thusly: > On Nov. Thursday 12 (46) 12:02 PM, Maxime Hadjinlian wrote: > > On Sat, Nov 7, 2015 at 5:39 AM, Vivien Didelot < > > vivien.didelot at savoirfairelinux.com> wrote: > > > > > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to > > > specify the SHA of a reference different than a branch or tag. > > > > > > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and > > > Github stores the pull requests under refs/pull/xxx/head. > > > > > > When cloning a repository with --bare, you don't fetch these references. > > > This patch uses --mirror for a full clone, in order to give the user > > > access to all references of the Git repository. > > > > > > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com> > > > --- > > > support/download/git | 2 +- > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > diff --git a/support/download/git b/support/download/git > > > index 0e6103b..357a558 100755 > > > --- a/support/download/git > > > +++ b/support/download/git > > > @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" > > > ]; then > > > fi > > > if [ ${git_done} -eq 0 ]; then > > > printf "Doing full clone\n" > > > - ${GIT} clone ${verbose} --bare "${repo}" "${basename}" > > > + ${GIT} clone ${verbose} --mirror "${repo}" "${basename}" > > > fi > > > > > > GIT_DIR="${basename}" \ > > > -- > > > 2.6.2 > > > > > I've read the different between bare and mirror and from what I > > understand, mirrors implies bare and also maps the remotes branch to the > > local branches. > > Both maps source and target branches. But --mirror also maps *all* refs, > i.e. any (standard) ref that is not a tag or a branch. > > > Given that understanding, I am afraid I don't really get what you are > > trying to do. From what I understand of Git, everything has a SHA ID, be > > it, branches, tags, merge commit, even when you push -f you still have > > access to previous state of your HEAD. > > > > I cloned a repository from Github that had pull requests ongoing, I could > > not find anything non-standard under .git/refs/, is there a specific > > example that you could point to ? I don't know Gerrit nor how it works, so > > maybe this was the issues you had ? > > > > Finally, with the current mechanisms, you can specify a SHA, so if you are > > trying to clone a PR, since it should be made from either a branch from the > > local repository, or a whole other repository, you should be able to > > specify it. > > > > Again, I'm not sure I got what you were trying to do, maybe there's a > > special use cases with Gerrit that I'm not aware of. > > Here's an example with Github: > > $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --bare /tmp/github.bare > $ git -C /tmp/github.bare show refs/pull/24/head # => unknown revision > $ git -C /tmp/github.bare show bd029a9ace8810259f73933894b665f5b0a5bb36 # => bad object > > $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --mirror /tmp/github.mirror > $ git -C /tmp/github.mirror show refs/pull/24/head > $ git -C /tmp/github.mirror show bd029a9ace8810259f73933894b665f5b0a5bb36 > > Now an example with Gerrit: > > $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --bare /tmp/gerrit.bare > $ git -C /tmp/gerrit.bare show refs/changes/62/2462/8 # => unknown revision > $ git -C /tmp/gerrit.bare show 6f5cf35b6644b0de2a3e005805b25e6a91a429cb # => bad object > > $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --mirror /tmp/gerrit.mirror > $ git -C /tmp/gerrit.mirror show refs/changes/62/2462/8 > $ git -C /tmp/gerrit.mirror show 6f5cf35b6644b0de2a3e005805b25e6a91a429cb > > With a mirror clone you can specify the target version as either the SHA > or the ref path. Yes, but for PR, I'm afraid this is not very stable: one may re-push a PR, or one may delete it (e.g. the PR initiator may remove his tree and the PR is no longe ravailable. Or is it?). Yet, those do not provide stable references, like sha1 and tags do provide. I understand it can be interesting during development, though, to quickly test a PR without having to manually handle fetching it... Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH] support/download: fetch all refs on full git clone 2015-11-13 22:48 ` Yann E. MORIN @ 2015-11-13 23:04 ` Vivien Didelot 0 siblings, 0 replies; 12+ messages in thread From: Vivien Didelot @ 2015-11-13 23:04 UTC (permalink / raw) To: buildroot Hi Yann, On Nov. Friday 13 (46) 11:48 PM, Yann E. MORIN wrote: > Vivien, All, > > On 2015-11-12 11:12 -0500, Vivien Didelot spake thusly: > > On Nov. Thursday 12 (46) 12:02 PM, Maxime Hadjinlian wrote: > > > On Sat, Nov 7, 2015 at 5:39 AM, Vivien Didelot < > > > vivien.didelot at savoirfairelinux.com> wrote: > > > > > > > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to > > > > specify the SHA of a reference different than a branch or tag. > > > > > > > > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and > > > > Github stores the pull requests under refs/pull/xxx/head. > > > > > > > > When cloning a repository with --bare, you don't fetch these references. > > > > This patch uses --mirror for a full clone, in order to give the user > > > > access to all references of the Git repository. > > > > > > > > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com> > > > > --- > > > > support/download/git | 2 +- > > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > > > diff --git a/support/download/git b/support/download/git > > > > index 0e6103b..357a558 100755 > > > > --- a/support/download/git > > > > +++ b/support/download/git > > > > @@ -43,7 +43,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" > > > > ]; then > > > > fi > > > > if [ ${git_done} -eq 0 ]; then > > > > printf "Doing full clone\n" > > > > - ${GIT} clone ${verbose} --bare "${repo}" "${basename}" > > > > + ${GIT} clone ${verbose} --mirror "${repo}" "${basename}" > > > > fi > > > > > > > > GIT_DIR="${basename}" \ > > > > -- > > > > 2.6.2 > > > > > > > I've read the different between bare and mirror and from what I > > > understand, mirrors implies bare and also maps the remotes branch to the > > > local branches. > > > > Both maps source and target branches. But --mirror also maps *all* refs, > > i.e. any (standard) ref that is not a tag or a branch. > > > > > Given that understanding, I am afraid I don't really get what you are > > > trying to do. From what I understand of Git, everything has a SHA ID, be > > > it, branches, tags, merge commit, even when you push -f you still have > > > access to previous state of your HEAD. > > > > > > I cloned a repository from Github that had pull requests ongoing, I could > > > not find anything non-standard under .git/refs/, is there a specific > > > example that you could point to ? I don't know Gerrit nor how it works, so > > > maybe this was the issues you had ? > > > > > > Finally, with the current mechanisms, you can specify a SHA, so if you are > > > trying to clone a PR, since it should be made from either a branch from the > > > local repository, or a whole other repository, you should be able to > > > specify it. > > > > > > Again, I'm not sure I got what you were trying to do, maybe there's a > > > special use cases with Gerrit that I'm not aware of. > > > > Here's an example with Github: > > > > $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --bare /tmp/github.bare > > $ git -C /tmp/github.bare show refs/pull/24/head # => unknown revision > > $ git -C /tmp/github.bare show bd029a9ace8810259f73933894b665f5b0a5bb36 # => bad object > > > > $ git clone git at github.com:maximeh/leaflet.bouncemarker.git --mirror /tmp/github.mirror > > $ git -C /tmp/github.mirror show refs/pull/24/head > > $ git -C /tmp/github.mirror show bd029a9ace8810259f73933894b665f5b0a5bb36 > > > > Now an example with Gerrit: > > > > $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --bare /tmp/gerrit.bare > > $ git -C /tmp/gerrit.bare show refs/changes/62/2462/8 # => unknown revision > > $ git -C /tmp/gerrit.bare show 6f5cf35b6644b0de2a3e005805b25e6a91a429cb # => bad object > > > > $ git clone https://gerrit-ring.savoirfairelinux.com/ring-lrc --mirror /tmp/gerrit.mirror > > $ git -C /tmp/gerrit.mirror show refs/changes/62/2462/8 > > $ git -C /tmp/gerrit.mirror show 6f5cf35b6644b0de2a3e005805b25e6a91a429cb > > > > With a mirror clone you can specify the target version as either the SHA > > or the ref path. > > Yes, but for PR, I'm afraid this is not very stable: one may re-push a > PR, or one may delete it (e.g. the PR initiator may remove his tree and > the PR is no longe ravailable. Or is it?). > > Yet, those do not provide stable references, like sha1 and tags do > provide. > > I understand it can be interesting during development, though, to > quickly test a PR without having to manually handle fetching it... Github PR related blob goes into the target repository. I don't know if Github removes them if the PR is removed from the source repository. But note that Github was just an example. However this fixes my build for an under-review Linux we have behind a Gerrit. Thanks, -v ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH] support/download: fetch all refs on full git clone 2015-11-07 4:39 [Buildroot] [PATCH] support/download: fetch all refs on full git clone Vivien Didelot 2015-11-12 11:02 ` Maxime Hadjinlian @ 2015-11-19 17:38 ` Vivien Didelot 2015-11-29 17:53 ` Thomas Petazzoni 2 siblings, 0 replies; 12+ messages in thread From: Vivien Didelot @ 2015-11-19 17:38 UTC (permalink / raw) To: buildroot On Nov. Friday 06 (45) 11:39 PM, Vivien Didelot wrote: > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to > specify the SHA of a reference different than a branch or tag. > > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and > Github stores the pull requests under refs/pull/xxx/head. > > When cloning a repository with --bare, you don't fetch these references. > This patch uses --mirror for a full clone, in order to give the user > access to all references of the Git repository. > > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com> Reviewed-by: "Maxime Hadjinlian" <maxime.hadjinlian@gmail.com> (I pasted Maxime's review here for clarity) Guys, could you merge this patch please? Also having it applied as a fix for 2015.11 would be a plus. Best, -v ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH] support/download: fetch all refs on full git clone 2015-11-07 4:39 [Buildroot] [PATCH] support/download: fetch all refs on full git clone Vivien Didelot 2015-11-12 11:02 ` Maxime Hadjinlian 2015-11-19 17:38 ` Vivien Didelot @ 2015-11-29 17:53 ` Thomas Petazzoni 2 siblings, 0 replies; 12+ messages in thread From: Thomas Petazzoni @ 2015-11-29 17:53 UTC (permalink / raw) To: buildroot Dear Vivien Didelot, On Fri, 6 Nov 2015 23:39:24 -0500, Vivien Didelot wrote: > When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to > specify the SHA of a reference different than a branch or tag. > > For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and > Github stores the pull requests under refs/pull/xxx/head. > > When cloning a repository with --bare, you don't fetch these references. > This patch uses --mirror for a full clone, in order to give the user > access to all references of the Git repository. > > Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com> > --- > support/download/git | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) Applied to next, thanks. Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2015-11-29 17:53 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-11-07 4:39 [Buildroot] [PATCH] support/download: fetch all refs on full git clone Vivien Didelot 2015-11-12 11:02 ` Maxime Hadjinlian 2015-11-12 16:12 ` Vivien Didelot 2015-11-12 16:26 ` Maxime Hadjinlian 2015-11-12 17:07 ` Vivien Didelot 2015-11-13 6:49 ` Arnout Vandecappelle 2015-11-13 15:35 ` Vivien Didelot 2015-11-16 8:43 ` Jérôme Pouiller 2015-11-13 22:48 ` Yann E. MORIN 2015-11-13 23:04 ` Vivien Didelot 2015-11-19 17:38 ` Vivien Didelot 2015-11-29 17:53 ` Thomas Petazzoni
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.