[BUG] Git clone of a bundle fails, but works (somewhat) when run with strace

[BUG] Git clone of a bundle fails, but works (somewhat) when run with strace

[Alain Kalker]

tl;dr:

- `git bundle create` without <git-rev-list-args> gives git rev-list 
help, then dies.
   Should point out missing <git-rev-list-args> instead.
- `git clone <bundle> <dir> gives "ERROR: Repository not found."
- `strace ... git clone <bundle> <dir>` (magically) appears to work but
   cannot checkout files b/c of nonexistent ref.
- Heisenbug? Race condition?
- Zaphod Beeblebrox has left the building, sulking.

Full description:

When I try to clone from a bundle created from a local repository, `git 
clone <bundle> <dir>` fails with: "ERROR: Repository not found. fatal: 
Could not read from remote repository." unless I run it with strace.

OS: Arch Linux (rolling release)
Git versions: 1.8.1.3 and git://github.com/git.git master@02339dd

Steps to reproduce:

$ # Clone the Linux kernel repository
$ git clone git://github.com/torvalds/linux.git
Cloning into 'linux'...
remote: Counting objects: 2841147, done.
remote: Compressing objects: 100% (670736/670736), done.
remote: Total 2841147 (delta 2308339), reused 2657487 (delta 2143012)
Receiving objects: 100% (2841147/2841147), 797.62 MiB | 2.59 MiB/s, done.
Resolving deltas: 100% (2308339/2308339), done.
Checking out files: 100% (41521/41521), done.
$ cd linux
$ git branch -av
* master                323a72d Merge 
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
   remotes/origin/HEAD   -> origin/master
   remotes/origin/master 323a72d Merge 
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net

$ # Try to create a bundle
$ git bundle create ../linux.bundle
usage: git rev-list [OPTION] <commit-id>... [ -- paths... ]
   limiting output:
     --max-count=<n>
     --max-age=<epoch>
     --min-age=<epoch>
     --sparse
     --no-merges
     --min-parents=<n>
     --no-min-parents
     --max-parents=<n>
     --no-max-parents
     --remove-empty
     --all
     --branches
     --tags
     --remotes
     --stdin
     --quiet
   ordering output:
     --topo-order
     --date-order
     --reverse
   formatting output:
     --parents
     --children
     --objects | --objects-edge
     --unpacked
     --header | --pretty
     --abbrev=<n> | --no-abbrev
     --abbrev-commit
     --left-right
   special purpose:
     --bisect
     --bisect-vars
     --bisect-all
error: rev-list died
$ # IMHO the error should refer to the usage of `git bundle` with a 
proper basis, not `git rev-list`.
$ # Also nothing should die loudly because of a missing parameter.
$ git bundle create ../linux.bundle master
Counting objects: 2836191, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (505627/505627), done.
Writing objects: 100% (2836191/2836191), 796.59 MiB | 16.23 MiB/s, done.
Total 2836191 (delta 2304454), reused 2834391 (delta 2303193)

$ # Try to clone a new repository from the bundle
$ cd ..
$ git clone linux.bundle linuxfrombundle
Cloning into 'linuxfrombundle'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
$ git clone linux.bundle -b master linuxfrombundle
Cloning into 'linuxfrombundle'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

# Try again using strace
$ # (Replace /dev/null with a filename if you really want to try and 
debug this, or if you just want to torture your hard drive ;) )
$ strace -o /dev/null git clone linux.bundle linuxfrombundle
Cloning into 'linuxfrombundle'...
Receiving objects: 100% (2836191/2836191), 796.59 MiB | 24.64 MiB/s, done.
Resolving deltas: 100% (2304454/2304454), done.
warning: remote HEAD refers to nonexistent ref, unable to checkout.

$ # Let's have a look at what we cloned
$ cd linuxfrombundle
$ ls
$ git branch -av
   remotes/origin/master 323a72d Merge 
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
$ git checkout master
Checking out files: 100% (41521/41521), done.
Branch master set up to track remote branch master from origin.
Already on 'master'
$ git branch -av
* master                323a72d Merge 
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
   remotes/origin/master 323a72d Merge 
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
$ # Where's my HEAD?

Kind regards,

Alain Kalker

Re: [BUG] Git clone of a bundle fails, but works (somewhat) when run with strace

[Alain Kalker]

On Fri, 15 Feb 2013 20:33:24 +0100, Alain Kalker wrote:

> tl;dr:
> 
> - `git bundle create` without <git-rev-list-args> gives git rev-list
> help, then dies.
>    Should point out missing <git-rev-list-args> instead.
> - `git clone <bundle> <dir> gives "ERROR: Repository not found."
> - `strace ... git clone <bundle> <dir>` (magically) appears to work but
>    cannot checkout files b/c of nonexistent ref.
> - Heisenbug? Race condition?
> - Zaphod Beeblebrox has left the building, sulking.
> 
> Full description:
> 
> When I try to clone from a bundle created from a local repository, `git
> clone <bundle> <dir>` fails with: "ERROR: Repository not found. fatal:
> Could not read from remote repository." unless I run it with strace.
> 
> OS: Arch Linux (rolling release)
> Git versions: 1.8.1.3 and git://github.com/git.git master@02339dd
> 
> Steps to reproduce:

For those who like to "save the trees" (source code or otherwise), here 
is a much simplified test case:

$ # Create test repository with a single commit in it
$ mkdir testrepo
$ cd testrepo
$ git init
$ echo Test > test.txt
$ git add test.txt
$ git commit -m "Add test.txt"

$ # Create bundle
$ git bundle create ../testrepo.bundle master

$ # Try to clone from bundle
$ cd ..
$ git clone testrepo.bundle testrepofrombundle

$ # Clone from bundle, wrapped with strace
$ strace -f -o /dev/null git clone testrepo.bundle testrepofrombundle

$ # Examine cloned repository
$ cd testrepofrombundle
$ ls
$ git branch -av
$ git checkout master
$ git branch -av
$ # Where's my HEAD?

> Kind regards,
> 
> Alain Kalker

Re: [BUG] Git clone of a bundle fails, but works (somewhat) when run with strace

[Alain Kalker]

On Fri, 15 Feb 2013 22:25:47 +0000, Alain Kalker wrote:

> On Fri, 15 Feb 2013 20:33:24 +0100, Alain Kalker wrote:
> 
>> tl;dr:
>> 
>> - `git bundle create` without <git-rev-list-args> gives git rev-list
>> help, then dies.
>>    Should point out missing <git-rev-list-args> instead.
>> - `git clone <bundle> <dir> gives "ERROR: Repository not found."
>> - `strace ... git clone <bundle> <dir>` (magically) appears to work but
>>    cannot checkout files b/c of nonexistent ref.
>> - Heisenbug? Race condition?
>> - Zaphod Beeblebrox has left the building, sulking.
>> 
>> Full description:
>> 
>> When I try to clone from a bundle created from a local repository, `git
>> clone <bundle> <dir>` fails with: "ERROR: Repository not found. fatal:
>> Could not read from remote repository." unless I run it with strace.

After trying to bisect this using `bisect start; bisect good v1.5.1; git 
bisect bad HEAD; git bisect run ..test.sh`:

---test.sh---
#!/bin/sh

make clean
make || return 125
GIT=$(pwd)/git

cd /tmp
rm -rf testrepo
mkdir testrepo
cd testrepo
$GIT init
echo test > test.txt
$GIT add test.txt
$GIT commit -m "Add test.txt"
$GIT bundle create ../testrepo.bundle master || return 125
cd ..

rm -rf testrepofrombundle
$GIT clone testrepo.bundle testrepofrombundle || return 1
---
I was unable to find a bad revision.
After a lot more searching I found that I had `git` aliased to `hub`, a 
tool used to make Github actions easier.
Eliminating `hub` from the equation resolved most problems.
The only ones remaining are the confusing error message from `git bundle 
create` and the "missing HEAD" (you can interpret that in different 
ways) ;-)

P.S. I hereby promise to _never_ _ever_ alias `git` to something else and 
then post a Git bug about that "something else" on this ML.

Sorry to have wasted your time,

Alain

Re: [BUG] Git clone of a bundle fails, but works (somewhat) when run with strace

[Junio C Hamano]

Alain Kalker <a.c.kalker@gmail.com> writes:

> P.S. I hereby promise to _never_ _ever_ alias `git` to something else and 
> then post a Git bug about that "something else" on this ML.
>
> Sorry to have wasted your time,

Thanks.

People around here tend to be quiet until they sufficiently have dug
the issue themselves; unless the initial report grossly lack
necessary level of details, you may not hear "does not reproduce for
me" for some time, so some may still have been scratching their
head, and your honestly following-up on your message will save time
for them.  Thanks again, and happy Gitting ;-)

Re: [BUG] Git clone of a bundle fails, but works (somewhat) when run with strace

[Jeff King]

On Sat, Feb 16, 2013 at 12:03:58AM +0000, Alain Kalker wrote:

> ---test.sh---
> #!/bin/sh
> 
> make clean
> make || return 125
> GIT=$(pwd)/git
> 
> cd /tmp
> rm -rf testrepo
> mkdir testrepo
> cd testrepo
> $GIT init
> echo test > test.txt
> $GIT add test.txt
> $GIT commit -m "Add test.txt"
> $GIT bundle create ../testrepo.bundle master || return 125
> cd ..
> 
> rm -rf testrepofrombundle
> $GIT clone testrepo.bundle testrepofrombundle || return 1
> ---
> I was unable to find a bad revision.
> After a lot more searching I found that I had `git` aliased to `hub`, a 
> tool used to make Github actions easier.
> Eliminating `hub` from the equation resolved most problems.

Great.

> The only ones remaining are the confusing error message from `git bundle 
> create` and the "missing HEAD" (you can interpret that in different 
> ways) ;-)

I do not see any odd message from "bundle create" in the recipe above.
Mine says:

$ git bundle create ../repo.bundle master
Counting objects: 3, done.
Writing objects: 100% (3/3), 209 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)

What you _might_ be seeing is the fact that the invocation above is
likely to be running two different versions of git under the hood. "git
bundle" will invoke "git rev-list", and it will use the first git in
your PATH, even if it is not $GIT. The proper way to test an un-installed
version of git is to use $YOUR_GIT_BUILD/bin-wrappers/git, which will
set up environment variables sufficient to make sure all sub-gits are
from the same version. Sometimes mixing versions can have weird results
(e.g., the new "git bundle" expects "rev-list" to have a particular
option, but the older version does not have it).

Secondly, I do get the same warning about HEAD:

  $ git clone repo.bundle repofrombundle
  Cloning into 'repofrombundle'...
  Receiving objects: 100% (3/3), done.
  warning: remote HEAD refers to nonexistent ref, unable to checkout.

but that warning makes sense. You did not create a bundle that contains
HEAD, therefore when we clone it, we do not know what to point HEAD to.
You probably wanted "git bundle create ../repo.bundle --all" which
includes both "master" and "HEAD".

It would be slightly more accurate to say "the remote HEAD does not
exist", rather than "refers to nonexistent ref".  It would perhaps be
nicer still for "git clone" to make a guess about the correct HEAD when
one is not present (especially in the single-branch case, it is easy to
make the right guess).

Patches welcome. In the meantime, you can clone with "-b master" to tell
it explicitly, or you can "git checkout master" inside the newly-cloned
repository.

-Peff

Re: [BUG] Git clone of a bundle fails, but works (somewhat) when run with strace

[Alain Kalker]

On Fri, 15 Feb 2013 23:01:09 -0500, Jeff King wrote:

> On Sat, Feb 16, 2013 at 12:03:58AM +0000, Alain Kalker wrote:
> 
>> ---test.sh---
>> #!/bin/sh
>> 
>> make clean make || return 125 GIT=$(pwd)/git
>> 
>> cd /tmp rm -rf testrepo mkdir testrepo cd testrepo $GIT init echo test
>> > test.txt $GIT add test.txt $GIT commit -m "Add test.txt"
>> $GIT bundle create ../testrepo.bundle master || return 125 cd ..
>> 
>> rm -rf testrepofrombundle $GIT clone testrepo.bundle testrepofrombundle
>> || return 1 ---
>> I was unable to find a bad revision.
>> After a lot more searching I found that I had `git` aliased to `hub`, a
>> tool used to make Github actions easier.
>> Eliminating `hub` from the equation resolved most problems.
> 
> Great.
> 
>> The only ones remaining are the confusing error message from `git
>> bundle create` and the "missing HEAD" (you can interpret that in
>> different ways) ;-)
> 
> I do not see any odd message from "bundle create" in the recipe above.
> Mine says:
> 
> $ git bundle create ../repo.bundle master Counting objects: 3, done.
> Writing objects: 100% (3/3), 209 bytes, done.
> Total 3 (delta 0), reused 0 (delta 0)
> 
> What you _might_ be seeing is the fact that the invocation above is
> likely to be running two different versions of git under the hood. "git
> bundle" will invoke "git rev-list", and it will use the first git in
> your PATH, even if it is not $GIT. The proper way to test an
> un-installed version of git is to use $YOUR_GIT_BUILD/bin-wrappers/git,
> which will set up environment variables sufficient to make sure all
> sub-gits are from the same version. Sometimes mixing versions can have
> weird results (e.g., the new "git bundle" expects "rev-list" to have a
> particular option, but the older version does not have it).

Thanks for the very useful tip! I will try to remember to use this for 
any future testing.

> Secondly, I do get the same warning about HEAD:
> 
>   $ git clone repo.bundle repofrombundle Cloning into
>   'repofrombundle'...
>   Receiving objects: 100% (3/3), done.
>   warning: remote HEAD refers to nonexistent ref, unable to checkout.
> 
> but that warning makes sense. You did not create a bundle that contains
> HEAD, therefore when we clone it, we do not know what to point HEAD to.
> You probably wanted "git bundle create ../repo.bundle --all" which
> includes both "master" and "HEAD".

That explains it, thanks! Maybe this could be added as an example to the 
documentation for `git bundle`? People looking at `man git-bundle` hoping 
to use it as one possible way to make a backup of a Git repository might 
not realize right away that --all is the way to specify all refs, like 
with `git log`.

> 
> It would be slightly more accurate to say "the remote HEAD does not
> exist", rather than "refers to nonexistent ref".  It would perhaps be
> nicer still for "git clone" to make a guess about the correct HEAD when
> one is not present (especially in the single-branch case, it is easy to
> make the right guess).
> 
> Patches welcome. In the meantime, you can clone with "-b master" to tell
> it explicitly, or you can "git checkout master" inside the newly-cloned
> repository.

Thanks again for your help, very welcome :-)

Alain

Re: [BUG] Git clone of a bundle fails, but works (somewhat) when run with strace

[Philip Oakley]

From: "Alain Kalker" <a.c.kalker@gmail.com>
Sent: Sunday, February 17, 2013 1:01 PM
> On Fri, 15 Feb 2013 23:01:09 -0500, Jeff King wrote:
>
>> On Sat, Feb 16, 2013 at 12:03:58AM +0000, Alain Kalker wrote:
>>
>>> ---test.sh---
>>> #!/bin/sh
[...]
>>
>> but that warning makes sense. You did not create a bundle that 
>> contains
>> HEAD, therefore when we clone it, we do not know what to point HEAD 
>> to.
>> You probably wanted "git bundle create ../repo.bundle --all" which
>> includes both "master" and "HEAD".
>
> That explains it, thanks! Maybe this could be added as an example to 
> the
> documentation for `git bundle`? People looking at `man git-bundle` 
> hoping
> to use it as one possible way to make a backup of a Git repository 
> might
> not realize right away that --all is the way to specify all refs, like
> with `git log`.

I had posted a documentation patch back in Setemeber last year,
http://thread.gmane.org/gmane.comp.version-control.git/205887/focus=205897,
however Junio highlighted some additional concerns that I wasn't able to 
respond to at the time.

It may be worth resurrecting once the concerns have been addressed.

>
>>
>> It would be slightly more accurate to say "the remote HEAD does not
>> exist", rather than "refers to nonexistent ref".  It would perhaps be
>> nicer still for "git clone" to make a guess about the correct HEAD 
>> when
>> one is not present (especially in the single-branch case, it is easy 
>> to
>> make the right guess).
>>
>> Patches welcome. In the meantime, you can clone with "-b master" to 
>> tell
>> it explicitly, or you can "git checkout master" inside the 
>> newly-cloned
>> repository.
>
> Thanks again for your help, very welcome :-)
>
> Alain
>
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2013.0.2899 / Virus Database: 2639/6108 - Release Date: 
> 02/16/13
> 

Re: [BUG] Git clone of a bundle fails, but works (somewhat) when run with strace

[Junio C Hamano]

"Philip Oakley" <philipoakley@iee.org> writes:

>>> You probably wanted "git bundle create ../repo.bundle --all" which
>>> includes both "master" and "HEAD".
>>
>> That explains it, thanks! Maybe this could be added as an example to
>> the
>> documentation for `git bundle`? People looking at `man git-bundle`
>> hoping
>> to use it as one possible way to make a backup of a Git repository
>> might
>> not realize right away that --all is the way to specify all refs, like
>> with `git log`.
>
> I had posted a documentation patch back in Setemeber last year,
> http://thread.gmane.org/gmane.comp.version-control.git/205887/focus=205897,
> however Junio highlighted some additional concerns that I wasn't able
> to respond to at the time.
>
> It may be worth resurrecting once the concerns have been addressed.

A saner thing to do, instead of explaining away the lack of HEAD as
"the creator of the bundle did not bother to name it", might be to
automatically add an artificial HEAD to the resulting bundle when
the arguments given to specify the "range" do not have any negative
ones (because by definition such a bundle is unsuitable for use with
"git clone" [*1*]), and HEAD is not among the refs.

The heuristics to pick what to record as the artificial HEAD could
vary, though. Without thinking things through...

 * When only one positive ref is given, use it (sort of obvious);

 * When two or more positive refs are given, and the current branch
   is one of them, use that;

 * Otherwise, pick the "first" positive ref given from the command
   line.

perhaps?


[Footnote]

*1* Strictly speaking, this condition could be loosened, as long as
cloning side uses an appropriate --depth, but I do not know such an
"advanced use case" needs a hand-holding change to add a HEAD that
was not asked by the user.

Re: [BUG] Git clone of a bundle fails, but works (somewhat) when run with strace

[Bram Geron]

Jeff King <peff <at> peff.net> writes:
> [..]
> Secondly, I do get the same warning about HEAD:
> 
>   $ git clone repo.bundle repofrombundle
>   Cloning into 'repofrombundle'...
>   Receiving objects: 100% (3/3), done.
>   warning: remote HEAD refers to nonexistent ref, unable to checkout.
> 
> but that warning makes sense. You did not create a bundle that contains
> HEAD, therefore when we clone it, we do not know what to point HEAD to.
> You probably wanted "git bundle create ../repo.bundle --all" which
> includes both "master" and "HEAD".

I'd like to revive this discussion and submit a patch, as I just spent
significant time wondering why git clone failed. It's been a while, so I'll
summarize: when you make a git bundle without including HEAD explicitly,
then clone from that bundle, Git throws a warning and leaves you with a
broken HEAD.

I do not agree that the warning makes sense. It implies that HEAD exists but
is invalid. In reality, no ref is referred to by HEAD in the first place.
Furthermore, .git/HEAD in the clone is "autocorrected" to be
refs/heads/master, so the error message is even more misleading.

It's like saying "Our CEO's guitar is actually an air guitar", then
explaining where he stores his guitar, when I don't work in a company in the
first place.

> It would be slightly more accurate to say "the remote HEAD does not
> exist", rather than "refers to nonexistent ref".  It would perhaps be
> nicer still for "git clone" to make a guess about the correct HEAD when
> one is not present (especially in the single-branch case, it is easy to
> make the right guess).

Seems sensible at first sight, though it seems orthogonal to the warning
message.

> Patches welcome. In the meantime, you can clone with "-b master" to tell
> it explicitly, or you can "git checkout master" inside the newly-cloned
> repository.

Alright :) See below.

Cheers, Bram


>From bc799b12b659d7ab20df7fe420d5f1f1c90450aa Mon Sep 17 00:00:00 2001
From: Bram Geron <bgeron@bgeron.nl>
Date: Wed, 28 May 2014 15:54:37 +0100
Subject: [PATCH] Clearer error message when cloning a bundle without a HEAD
 ref.

---
 builtin/clone.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index 9b3c04d..e3c1447 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -623,7 +623,7 @@ static int checkout(void)
 
 	head = resolve_refdup("HEAD", sha1, 1, NULL);
 	if (!head) {
-		warning(_("remote HEAD refers to nonexistent ref, "
+		warning(_("no HEAD in remote or HEAD refers to nonexistent ref, "
 			  "unable to checkout.\n"));
 		return 0;
 	}
-- 
1.9.1