Malformed branch name in fast-export when specifying non-HEAD/branch revision

Malformed branch name in fast-export when specifying non-HEAD/branch revision

[Owen Stephens]

Hi,

fast-export gives an invalid branch name for commits, if the revision specified
is not HEAD or a branch name.

$ git --version
git version 1.7.6

$ # Create
$ git init export_test && cd export_test
$ touch a && git add a && git commit -m 'Add a'
$ touch b && git add b && git commit -m 'Add b'
$ git branch branch1

$ # ok
$ git fast-export HEAD

$ # also ok
$ git fast-export branch1

$ # uses HEAD~1 instead of refs/heads/master
$ git fast-export HEAD~1

blob
mark :1
data 0

reset HEAD~1
commit HEAD~1
mark :2
author Owen Stephens <git@owenstephens.co.uk> 1313597436 +0100
committer Owen Stephens <git@owenstephens.co.uk> 1313597436 +0100
data 6
Add a
M 100644 :1 a

$ # Uses the commit hash rather than refs/heads/master
$ git fast-export $(git rev-parse HEAD~1)

blob
mark :1
data 0

reset 5a8a8abb1ab44890501c64f2d51f671ab2108d60
commit 5a8a8abb1ab44890501c64f2d51f671ab2108d60
mark :2
author Owen Stephens <git@owenstephens.co.uk> 1313597436 +0100
committer Owen Stephens <git@owenstephens.co.uk> 1313597436 +0100
data 6
Add a
M 100644 :1 a

$ # fast-import cannot handle this stream:
$ git init subgit
$ git fast-export HEAD~1 | (cd subgit/ && git fast-import)
fatal: Branch name doesn't conform to GIT standards: HEAD~1
fast-import: dumping crash report to .git/fast_import_crash_26863

Regards,
Owen.

Re: Malformed branch name in fast-export when specifying non-HEAD/branch revision

[Elijah Newren]

Hi,

On Wed, Aug 17, 2011 at 10:21 AM, Owen Stephens <git@owenstephens.co.uk> wrote:
> fast-export gives an invalid branch name for commits, if the revision specified
> is not HEAD or a branch name.
>
> $ git --version
> git version 1.7.6
>
> $ # Create
> $ git init export_test && cd export_test
> $ touch a && git add a && git commit -m 'Add a'
> $ touch b && git add b && git commit -m 'Add b'
> $ git branch branch1
>
> $ # ok
> $ git fast-export HEAD
>
> $ # also ok
> $ git fast-export branch1
>
> $ # uses HEAD~1 instead of refs/heads/master
> $ git fast-export HEAD~1
>
> blob
> mark :1
> data 0
>
> reset HEAD~1
> commit HEAD~1

Thanks for the report.  It turns out this bug has been reported and is
in the testsuite as t9350.19 -- currently marked as expected to fail.
I looked at the problem a couple years ago for a little bit but never
finished that particular patch and never got back around to it.

Re: Malformed branch name in fast-export when specifying non-HEAD/branch revision

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

>> $ # uses HEAD~1 instead of refs/heads/master
>> $ git fast-export HEAD~1
>>
>> blob
>> mark :1
>> data 0
>>
>> reset HEAD~1
>> commit HEAD~1
>
> Thanks for the report.  It turns out this bug has been reported and is
> in the testsuite as t9350.19 -- currently marked as expected to fail.
> I looked at the problem a couple years ago for a little bit but never
> finished that particular patch and never got back around to it.

What _should_ be the right behaviour to begin with, I have to wonder.

Even though it is very clear that the set of objects that are exported are
defined by the "rev-list arguments" given to the command, I do not think
fast-export's semantics is not clearly defined as to what "refs" are to be
updated.

The easiest fix for this issue would be to forbid "git fast-export HEAD~1"
(or any range whose positive endpoints are _not_ refs), and I think that
would be in line with the original motivation of the command to export the
whole repository in a format fast-import would understand.  The original
f2dc849 (Add 'git fast-export', the sister of 'git fast-import',
2007-12-02) says "This program dumps (parts of) a git repository...",
implying that partial export is within the scope of the command, but I do
not think it was designed carefully enough to deal with ranges more
complex than just "a set of branches".

I however have a feeling that people would want to say:

 - I want to export up to that commit, and have that commit on this branch
   on the importing side; or even better

 - I want to export up to that commit, but what refs points at the commits
   contained in the output stream will be decided when the output is
   imported.

I do not think the latter meshes well with how "fast-import" works,
though.  But fast-export should be fixable to allow the former without
breaking the semantics of fast-import.

You can think of "fast-export" an off-line "push" command [*1*]; instead
of giving a random commit object, e.g. "git fast-export HEAD~1", that can
not be used as a ref, you can use the refspec notation to tell where the
result should go, e.g. "git fast-export HEAD~1:refs/heads/a-bit-older",
from the command line of fast-export.

I suspect that also may clarify what Sverre was trying to do in his recent
series. The root cause of both this and the issue Sverre wanted to fix is
the design mistake of fast-export that tries to reuse the notation of
object range specification for a different purpose of telling which "ref"
to update, I think.

[Footnote]

*1* In a similar sense, unpacking "git bundle" output is an off-line
"fetch"; the bundle creator gave anchor points for tip objects, and allows
the unpacker to map them into its own namespace.

Re: Malformed branch name in fast-export when specifying non-HEAD/branch revision

[Jeff King]

On Wed, Aug 17, 2011 at 03:30:14PM -0700, Junio C Hamano wrote:

> You can think of "fast-export" an off-line "push" command [*1*]; instead
> of giving a random commit object, e.g. "git fast-export HEAD~1", that can
> not be used as a ref, you can use the refspec notation to tell where the
> result should go, e.g. "git fast-export HEAD~1:refs/heads/a-bit-older",
> from the command line of fast-export.
> 
> I suspect that also may clarify what Sverre was trying to do in his recent
> series. The root cause of both this and the issue Sverre wanted to fix is
> the design mistake of fast-export that tries to reuse the notation of
> object range specification for a different purpose of telling which "ref"
> to update, I think.

Yes, this was the conclusion I came to when I looked at this a month or
so ago. You really need to give fast-export a mapping of objects to
refnames, and it should output ref names _only_ for the mapping. That
would handle this "not a ref" case, but would also let you push
"refs/heads/foo" when it is equivalent to "refs/heads/master", without
fast-export mentioning "refs/heads/master" at all.

-Peff

Re: Malformed branch name in fast-export when specifying non-HEAD/branch revision

[Sverre Rabbelier]

Heya,

On Wed, Aug 17, 2011 at 16:19, Jeff King <peff@peff.net> wrote:
> On Wed, Aug 17, 2011 at 03:30:14PM -0700, Junio C Hamano wrote:
>
>> You can think of "fast-export" an off-line "push" command [*1*]; instead
>> of giving a random commit object, e.g. "git fast-export HEAD~1", that can
>> not be used as a ref, you can use the refspec notation to tell where the
>> result should go, e.g. "git fast-export HEAD~1:refs/heads/a-bit-older",
>> from the command line of fast-export.
>>
>> I suspect that also may clarify what Sverre was trying to do in his recent
>> series. The root cause of both this and the issue Sverre wanted to fix is
>> the design mistake of fast-export that tries to reuse the notation of
>> object range specification for a different purpose of telling which "ref"
>> to update, I think.
>
> Yes, this was the conclusion I came to when I looked at this a month or
> so ago. You really need to give fast-export a mapping of objects to
> refnames, and it should output ref names _only_ for the mapping. That
> would handle this "not a ref" case, but would also let you push
> "refs/heads/foo" when it is equivalent to "refs/heads/master", without
> fast-export mentioning "refs/heads/master" at all.

Does this bring any new insights into how the problem I was pointing
out (trying to push next if master points at the same commit does
nothing) could/should be solved?

-- 
Cheers,

Sverre Rabbelier

Re: Malformed branch name in fast-export when specifying non-HEAD/branch revision

[Jeff King]

On Sun, Aug 21, 2011 at 03:29:38PM -0700, Sverre Rabbelier wrote:

> > Yes, this was the conclusion I came to when I looked at this a month or
> > so ago. You really need to give fast-export a mapping of objects to
> > refnames, and it should output ref names _only_ for the mapping. That
> > would handle this "not a ref" case, but would also let you push
> > "refs/heads/foo" when it is equivalent to "refs/heads/master", without
> > fast-export mentioning "refs/heads/master" at all.
> 
> Does this bring any new insights into how the problem I was pointing
> out (trying to push next if master points at the same commit does
> nothing) could/should be solved?

Hmm. Maybe I am misremembering the problem, but I thought that worked
already. If you say:

  git fast-export refs/heads/foo

you should get only reset/commit lines in the output for refs/heads/foo,
no?

Now I can't seem to replicate the case where refs/heads/master is
mentioned, but you didn't want it to be. I may have to go back and
re-read the thread from a month or two ago when we discussed these
issues.

-Peff

Re: Malformed branch name in fast-export when specifying non-HEAD/branch revision

[Sverre Rabbelier]

Heya,

On Mon, Aug 22, 2011 at 09:19, Jeff King <peff@peff.net> wrote:
> Hmm. Maybe I am misremembering the problem, but I thought that worked
> already. If you say:
>
>  git fast-export refs/heads/foo
>
> you should get only reset/commit lines in the output for refs/heads/foo,
> no?
>
> Now I can't seem to replicate the case where refs/heads/master is
> mentioned, but you didn't want it to be. I may have to go back and
> re-read the thread from a month or two ago when we discussed these
> issues.

Do you agree that this is expected behavior?

$ git init test
Initialized empty Git repository in /home/sverre/code/test/.git/
$ cd test/
$ echo content >> foo
sverre@laptop-sverre:~/code/test
$ git add foo
$ git commit -m first
[master (root-commit) 821176f] first
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 foo
$ echo content >> foo
$ git commit -am second
[master 1934282] second
 1 files changed, 1 insertions(+), 0 deletions(-)
$ git branch other
$ git fast-export ^master other
reset refs/heads/other
from 1934282469e3a83a5ef827fd31e074cfb4f3eadf

Because in current git.git, this doesn't work (the above is generated
using a git that has the patch series Dscho and I sent out). Current
git will instead do the following:

$ git fast-export ^master other
reset refs/heads/other
from :0

The 'from :0' here is obviously a bug (which is fixed by our patch series).

You might wonder, 'why would anyone do that', well, for example, they
might be using marks:

$ git fast-export --export-marks=marksfile  master > /dev/null
$ git fast-export --import-marks=marksfile  other
reset refs/heads/other
from :4

Again, the above is generated with my patched git, current git.git
simply outputs nothing.

$ git fast-export --import-marks=marksfile other

-- 
Cheers,

Sverre Rabbelier

Re: Malformed branch name in fast-export when specifying non-HEAD/branch revision

[Jeff King]

On Mon, Aug 22, 2011 at 09:54:47AM -0700, Sverre Rabbelier wrote:

> Do you agree that this is expected behavior?
> 
> $ git init test
> Initialized empty Git repository in /home/sverre/code/test/.git/
> $ cd test/
> $ echo content >> foo
> sverre@laptop-sverre:~/code/test
> $ git add foo
> $ git commit -m first
> [master (root-commit) 821176f] first
>  1 files changed, 1 insertions(+), 0 deletions(-)
>  create mode 100644 foo
> $ echo content >> foo
> $ git commit -am second
> [master 1934282] second
>  1 files changed, 1 insertions(+), 0 deletions(-)
> $ git branch other
> $ git fast-export ^master other
> reset refs/heads/other
> from 1934282469e3a83a5ef827fd31e074cfb4f3eadf

Yeah, that seems reasonable to me.

> Because in current git.git, this doesn't work (the above is generated
> using a git that has the patch series Dscho and I sent out). Current
> git will instead do the following:
> 
> $ git fast-export ^master other
> reset refs/heads/other
> from :0
> 
> The 'from :0' here is obviously a bug (which is fixed by our patch series).

Yep, the current behavior is definitely wrong. But I thought your
question was about accidentally mentioning refs/heads/master, which this
doesn't do (nor should it).

I just read through the remote-helper threads from early June, and the
only mention of triggering that is when you actually have a rename
(i.e., your "refs/heads/foo" becomes remote's "refs/heads/bar", but we
mention "refs/heads/foo" in the export stream). I was thinking there was
another case, but I couldn't find mention of it.

> You might wonder, 'why would anyone do that', well, for example, they
> might be using marks:
> 
> $ git fast-export --export-marks=marksfile  master > /dev/null
> $ git fast-export --import-marks=marksfile  other
> reset refs/heads/other
> from :4
> 
> Again, the above is generated with my patched git, current git.git
> simply outputs nothing.
> 
> $ git fast-export --import-marks=marksfile other

Yeah, the behavior of your patch looks fine to me. I thought the point
in contention was that having export understand refspecs would fix a lot
of _other_ cases, too.

-Peff

Re: Malformed branch name in fast-export when specifying non-HEAD/branch revision

[Sverre Rabbelier]

Heya,

On Mon, Aug 22, 2011 at 10:57, Jeff King <peff@peff.net> wrote:
> I just read through the remote-helper threads from early June, and the
> only mention of triggering that is when you actually have a rename
> (i.e., your "refs/heads/foo" becomes remote's "refs/heads/bar", but we
> mention "refs/heads/foo" in the export stream). I was thinking there was
> another case, but I couldn't find mention of it.

The patches Dscho and I sent are in response to the RFC patches (that
are currently "stalled" in whats-cooking) I added on top of the series
that rerolled yours.

> Yeah, the behavior of your patch looks fine to me. I thought the point
> in contention was that having export understand refspecs would fix a lot
> of _other_ cases, too.

Right. Sadly it doesn't look like I'll have time to try and fix 'git
bundle' anytime soon, so this'll remain broken.

-- 
Cheers,

Sverre Rabbelier

Re: Malformed branch name in fast-export when specifying non-HEAD/branch revision

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Yeah, the behavior of your patch looks fine to me. I thought the point
> in contention was that having export understand refspecs would fix a lot
> of _other_ cases, too.

Perhaps if we added refspecs we could cover other cases, too, but I do not
think we need to require the patch to cover additional cases. The more
problematic was that the particular implementation in the patch smelled
bad.

Re: Malformed branch name in fast-export when specifying non-HEAD/branch revision

[Jeff King]

On Mon, Aug 22, 2011 at 02:27:00PM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > Yeah, the behavior of your patch looks fine to me. I thought the point
> > in contention was that having export understand refspecs would fix a lot
> > of _other_ cases, too.
> 
> Perhaps if we added refspecs we could cover other cases, too, but I do not
> think we need to require the patch to cover additional cases. The more
> problematic was that the particular implementation in the patch smelled
> bad.

OK. I didn't look at that at all. I just wanted to express my support
for "refspecs would solve other problems, too". :)

-Peff