git fast-export | git fast-import doesn't work

git fast-export | git fast-import doesn't work

[Ondrej Certik]

Hi,

I would like to export our whole git repository to patches, and then
reconstruct it again from scratch. Following the man page of "git
fast-export":

$ git clone git://git.sympy.org/sympy-full-history-20081023.git
$ cd sympy-full-history-20081023
$ git fast-export --all --export-marks=marks > patches
$ cd ..
$ mkdir sympy-new
$ cd sympy-new
$ git init
$ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches
git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects:      25000
Total objects:        21355 (       144 duplicates                  )
      blobs  :         8009 (         0 duplicates       4529 deltas)
      trees  :        10627 (       144 duplicates       9189 deltas)
      commits:         2719 (         0 duplicates          0 deltas)
      tags   :            0 (         0 duplicates          0 deltas)
Total branches:          21 (        26 loads     )
      marks:        1048576 (     10728 unique    )
      atoms:            726
Memory total:          2880 KiB
       pools:          2098 KiB
     objects:           781 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize =   33554432
pack_report: core.packedGitLimit      =  268435456
pack_report: pack_used_ctr            =      40706
pack_report: pack_mmap_calls          =       2791
pack_report: pack_open_windows        =          1 /          2
pack_report: pack_mapped              =   26177739 /   35513414
---------------------------------------------------------------------



However, the repository is very different to the original one. It
contains only 191 patches:

$ git log --pretty=oneline | wc -l
191

and it only contains couple files. Compare this with the original repository:

$ git log --pretty=oneline | wc -l
2719

What am I doing wrong? Is there some other way to do it? I also tried
"git format-patch" and "git am" and that almost works, only it changes
hashes. Is there some way to tell "git am" to preserve the hash?

Thanks,
Ondrej

Re: git fast-export | git fast-import doesn't work

[Michael J Gruber]

Ondrej Certik venit, vidit, dixit 25.11.2008 17:44:
> Hi,
> 
> I would like to export our whole git repository to patches, and then
> reconstruct it again from scratch. Following the man page of "git
> fast-export":
> 
> $ git clone git://git.sympy.org/sympy-full-history-20081023.git
> $ cd sympy-full-history-20081023
> $ git fast-export --all --export-marks=marks > patches
> $ cd ..
> $ mkdir sympy-new
> $ cd sympy-new
> $ git init
> $ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches
> git-fast-import statistics:
> ---------------------------------------------------------------------
> Alloc'd objects:      25000
> Total objects:        21355 (       144 duplicates                  )
>       blobs  :         8009 (         0 duplicates       4529 deltas)
>       trees  :        10627 (       144 duplicates       9189 deltas)
>       commits:         2719 (         0 duplicates          0 deltas)
>       tags   :            0 (         0 duplicates          0 deltas)
> Total branches:          21 (        26 loads     )
>       marks:        1048576 (     10728 unique    )
>       atoms:            726
> Memory total:          2880 KiB
>        pools:          2098 KiB
>      objects:           781 KiB
> ---------------------------------------------------------------------
> pack_report: getpagesize()            =       4096
> pack_report: core.packedGitWindowSize =   33554432
> pack_report: core.packedGitLimit      =  268435456
> pack_report: pack_used_ctr            =      40706
> pack_report: pack_mmap_calls          =       2791
> pack_report: pack_open_windows        =          1 /          2
> pack_report: pack_mapped              =   26177739 /   35513414
> ---------------------------------------------------------------------
> 
> 
> 
> However, the repository is very different to the original one. It
> contains only 191 patches:
> 
> $ git log --pretty=oneline | wc -l
> 191
> 
> and it only contains couple files. Compare this with the original repository:
> 
> $ git log --pretty=oneline | wc -l
> 2719

I get the same stats (with the dups) but a perfect rev count, when I use
git log --all. The reason is that the history in the imported repo is
disconnected at various places (at tagging commits)! Your command counts
only the revs backwards to the first "disconnection".

So, the real issue is: Why has the result these cuts in the history?
I don't know, I just noticed that turning on rename and copy detection
makes git-fast-import crash, which shouldn't happen either. Something's
not right here. CC'ing the authors of im- and export.

BTW: Maybe you can accomplish what you want with different means? Why
export|import directly to git?

Michael
--
git 1.6.0.4.608.ga9645

Re: git fast-export | git fast-import doesn't work

[Peter Baumann]

On Tue, Nov 25, 2008 at 05:44:28PM +0100, Ondrej Certik wrote:
> Hi,
> 
> I would like to export our whole git repository to patches, and then
> reconstruct it again from scratch. Following the man page of "git
> fast-export":
> 

Perhabs you are looking for git filter-branch, because it seems you want
to change the history in some way (e.g. remove a wrongly committed file)?

Nevertheless, I expect your shown commands below to procude the same
repo again, so you might be on something ...

-Peter


> $ git clone git://git.sympy.org/sympy-full-history-20081023.git
> $ cd sympy-full-history-20081023
> $ git fast-export --all --export-marks=marks > patches
> $ cd ..
> $ mkdir sympy-new
> $ cd sympy-new
> $ git init
> $ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches
> git-fast-import statistics:
> ---------------------------------------------------------------------
> Alloc'd objects:      25000
> Total objects:        21355 (       144 duplicates                  )
>       blobs  :         8009 (         0 duplicates       4529 deltas)
>       trees  :        10627 (       144 duplicates       9189 deltas)
>       commits:         2719 (         0 duplicates          0 deltas)
>       tags   :            0 (         0 duplicates          0 deltas)
> Total branches:          21 (        26 loads     )
>       marks:        1048576 (     10728 unique    )
>       atoms:            726
> Memory total:          2880 KiB
>        pools:          2098 KiB
>      objects:           781 KiB
> ---------------------------------------------------------------------
> pack_report: getpagesize()            =       4096
> pack_report: core.packedGitWindowSize =   33554432
> pack_report: core.packedGitLimit      =  268435456
> pack_report: pack_used_ctr            =      40706
> pack_report: pack_mmap_calls          =       2791
> pack_report: pack_open_windows        =          1 /          2
> pack_report: pack_mapped              =   26177739 /   35513414
> ---------------------------------------------------------------------
> 
> 
> 
> However, the repository is very different to the original one. It
> contains only 191 patches:
> 
> $ git log --pretty=oneline | wc -l
> 191
> 
> and it only contains couple files. Compare this with the original repository:
> 
> $ git log --pretty=oneline | wc -l
> 2719
> 
> What am I doing wrong? Is there some other way to do it? I also tried
> "git format-patch" and "git am" and that almost works, only it changes
> hashes. Is there some way to tell "git am" to preserve the hash?
> 
> Thanks,
> Ondrej

Re: git fast-export | git fast-import doesn't work

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

On Tue, Nov 25, 2008 at 06:31:41PM +0100, Michael J Gruber <git@drmicha.warpmail.net> wrote:
> I don't know, I just noticed that turning on rename and copy detection
> makes git-fast-import crash, which shouldn't happen either. Something's
> not right here. CC'ing the authors of im- and export.

Could you please write a testcase that reproduces your problem?

> Why export|import directly to git?

I guess he did not know about filter-branch. :)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: git fast-export | git fast-import doesn't work

[Ondrej Certik]

On Tue, Nov 25, 2008 at 9:41 PM, Miklos Vajna <vmiklos@frugalware.org> wrote:
> On Tue, Nov 25, 2008 at 06:31:41PM +0100, Michael J Gruber <git@drmicha.warpmail.net> wrote:
>> I don't know, I just noticed that turning on rename and copy detection
>> makes git-fast-import crash, which shouldn't happen either. Something's
>> not right here. CC'ing the authors of im- and export.
>
> Could you please write a testcase that reproduces your problem?
>
>> Why export|import directly to git?
>
> I guess he did not know about filter-branch. :)

I know about filter-branch (but I am not sure it can do what I want).
I made a mistake of not explaining what I want, instead I suggested (a
possibly wrong) solution. I want to export the whole git repository as
a set of human readable patches, that can be assembled back into a git
repository (with the same hashes as the original one) if needed. The
reason I want that is that if we later decide to switch to another
VCS, we have all the information to reproduce the repository. Another
reason is to be sure that we know all the sources that are needed to
construct the repository, e.g. that there are no binary blobs
(possibly containing malicious code). Another reason I want that is to
be able to rewrite the history, in particular, we have one Mercurial
repository with some old history and another Mercurial history with a
newer history and I just want to concatenate them together into one
git repository.

In each case I know several workarounds, but if there is a way to just
convert the whole git repository into a set of patches and (and be
able to convert everything back including the same hashes), then it'd
be awesome.

See also this thread why people want this (and I assumed git can do
this from this thread):

http://groups.google.com/group/sage-devel/browse_thread/thread/7b116d902ee20d9c/

Thanks,
Ondrej

Re: git fast-export | git fast-import doesn't work

[Ondrej Certik]

On Tue, Nov 25, 2008 at 6:31 PM, Michael J Gruber
<git@drmicha.warpmail.net> wrote:
> Ondrej Certik venit, vidit, dixit 25.11.2008 17:44:
>> Hi,
>>
>> I would like to export our whole git repository to patches, and then
>> reconstruct it again from scratch. Following the man page of "git
>> fast-export":
>>
>> $ git clone git://git.sympy.org/sympy-full-history-20081023.git
>> $ cd sympy-full-history-20081023
>> $ git fast-export --all --export-marks=marks > patches
>> $ cd ..
>> $ mkdir sympy-new
>> $ cd sympy-new
>> $ git init
>> $ git fast-import --export-marks=marks < ../sympy-full-history-20081023/patches
>> git-fast-import statistics:
>> ---------------------------------------------------------------------
>> Alloc'd objects:      25000
>> Total objects:        21355 (       144 duplicates                  )
>>       blobs  :         8009 (         0 duplicates       4529 deltas)
>>       trees  :        10627 (       144 duplicates       9189 deltas)
>>       commits:         2719 (         0 duplicates          0 deltas)
>>       tags   :            0 (         0 duplicates          0 deltas)
>> Total branches:          21 (        26 loads     )
>>       marks:        1048576 (     10728 unique    )
>>       atoms:            726
>> Memory total:          2880 KiB
>>        pools:          2098 KiB
>>      objects:           781 KiB
>> ---------------------------------------------------------------------
>> pack_report: getpagesize()            =       4096
>> pack_report: core.packedGitWindowSize =   33554432
>> pack_report: core.packedGitLimit      =  268435456
>> pack_report: pack_used_ctr            =      40706
>> pack_report: pack_mmap_calls          =       2791
>> pack_report: pack_open_windows        =          1 /          2
>> pack_report: pack_mapped              =   26177739 /   35513414
>> ---------------------------------------------------------------------
>>
>>
>>
>> However, the repository is very different to the original one. It
>> contains only 191 patches:
>>
>> $ git log --pretty=oneline | wc -l
>> 191
>>
>> and it only contains couple files. Compare this with the original repository:
>>
>> $ git log --pretty=oneline | wc -l
>> 2719
>
> I get the same stats (with the dups) but a perfect rev count, when I use
> git log --all. The reason is that the history in the imported repo is
> disconnected at various places (at tagging commits)! Your command counts
> only the revs backwards to the first "disconnection".

You are right! I didn't know about "git log --all".

>
> So, the real issue is: Why has the result these cuts in the history?

Yes, I would like to know this too. E.g. if it is a problem with our
repository, or a problem in git, or whether it is just not supposed to
work.

> I don't know, I just noticed that turning on rename and copy detection
> makes git-fast-import crash, which shouldn't happen either. Something's
> not right here. CC'ing the authors of im- and export.
>
> BTW: Maybe you can accomplish what you want with different means? Why
> export|import directly to git?

I just answered this in my other email. Basically there are
workarounds, but I would feel safe if I can (correctly) reconstruct
the whole git repository from a human readable patches.

Ondrej

Re: git fast-export | git fast-import doesn't work

[david]

On Wed, 26 Nov 2008, Ondrej Certik wrote:

> On Tue, Nov 25, 2008 at 9:41 PM, Miklos Vajna <vmiklos@frugalware.org> wrote:
>> On Tue, Nov 25, 2008 at 06:31:41PM +0100, Michael J Gruber <git@drmicha.warpmail.net> wrote:
>>> I don't know, I just noticed that turning on rename and copy detection
>>> makes git-fast-import crash, which shouldn't happen either. Something's
>>> not right here. CC'ing the authors of im- and export.
>>
>> Could you please write a testcase that reproduces your problem?
>>
>>> Why export|import directly to git?
>>
>> I guess he did not know about filter-branch. :)
>
> I know about filter-branch (but I am not sure it can do what I want).
> I made a mistake of not explaining what I want, instead I suggested (a
> possibly wrong) solution. I want to export the whole git repository as
> a set of human readable patches, that can be assembled back into a git
> repository (with the same hashes as the original one)

the same hashes on the file is easy, the same hashes on the commits is 
extremely hard. which is it that you are looking for.

that being said, the test of being able to do a export|import is a good 
one to test that the export format and import format actually match.

David Lang

> if needed. The
> reason I want that is that if we later decide to switch to another
> VCS, we have all the information to reproduce the repository. Another
> reason is to be sure that we know all the sources that are needed to
> construct the repository, e.g. that there are no binary blobs
> (possibly containing malicious code). Another reason I want that is to
> be able to rewrite the history, in particular, we have one Mercurial
> repository with some old history and another Mercurial history with a
> newer history and I just want to concatenate them together into one
> git repository.
>
> In each case I know several workarounds, but if there is a way to just
> convert the whole git repository into a set of patches and (and be
> able to convert everything back including the same hashes), then it'd
> be awesome.
>
> See also this thread why people want this (and I assumed git can do
> this from this thread):
>
> http://groups.google.com/group/sage-devel/browse_thread/thread/7b116d902ee20d9c/
>
> Thanks,
> Ondrej
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: git fast-export | git fast-import doesn't work

[Johannes Schindelin]

Hi,

On Tue, 25 Nov 2008, Ondrej Certik wrote:

> I would like to export our whole git repository to patches, and then 
> reconstruct it again from scratch. Following the man page of "git 
> fast-export":
>
> [...] 
> 
> However, the repository is very different to the original one. It 
> contains only 191 patches:

Can you try again with a Git version that contains the commit 
2075ffb5(fast-export: use an unsorted string list for extra_refs)?

Ciao,
Dscho

Re: git fast-export | git fast-import doesn't work

[Ondrej Certik]

On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Tue, 25 Nov 2008, Ondrej Certik wrote:
>
>> I would like to export our whole git repository to patches, and then
>> reconstruct it again from scratch. Following the man page of "git
>> fast-export":
>>
>> [...]
>>
>> However, the repository is very different to the original one. It
>> contains only 191 patches:
>
> Can you try again with a Git version that contains the commit
> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?

I tried the next branch:

$ git --version
git version 1.6.0.4.1060.g9433b

that contains the 2075ffb5 patch. I haven't observed any change ---
the "git log" still only shows 191 commits (git log --all shows
everything).

Ondrej

Re: git fast-export | git fast-import doesn't work

[Ondrej Certik]

On Wed, Nov 26, 2008 at 10:35 AM, Ondrej Certik <ondrej@certik.cz> wrote:
> On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>> Hi,
>>
>> On Tue, 25 Nov 2008, Ondrej Certik wrote:
>>
>>> I would like to export our whole git repository to patches, and then
>>> reconstruct it again from scratch. Following the man page of "git
>>> fast-export":
>>>
>>> [...]
>>>
>>> However, the repository is very different to the original one. It
>>> contains only 191 patches:
>>
>> Can you try again with a Git version that contains the commit
>> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?
>
> I tried the next branch:
>
> $ git --version
> git version 1.6.0.4.1060.g9433b
>
> that contains the 2075ffb5 patch. I haven't observed any change ---
> the "git log" still only shows 191 commits (git log --all shows
> everything).

I deleted all tags and then fast-exported and imported, now all the
commits show in "git log", however, the patches are wrongly connected.
Basically, both repositories are identical (including hashes) up to
this commit:

d717177d4  (fixed downloads instructions in the README and a typo)

However, the original repo (sympy-full-history-20081023) contains 3
children at this commit:

Parent: fecac34251934e98a05631440d3ce151585f2391 (David added to credits)
Child:  03ccb60798d62f94ac9d2ec9472dc7333f67b420 (Allow to specify
line width in 2D plotting.)
Child:  203124d834488781db5429d941eeb60e396990c8 (credits improvements)
Child:  77146885f1b7aa49184f27c2297488c3d1201106 (Speed "import sympy"
up as in the last release.)

but the newly created repository only 2:

Parent: fecac34251934e98a05631440d3ce151585f2391 (David added to credits)
Child:  203124d834488781db5429d941eeb60e396990c8 (credits improvements)
Child:  77146885f1b7aa49184f27c2297488c3d1201106 (Speed "import sympy"
up as in the last release.)


And from that point on, the hashes mishmatch and sometimes the commits
are just wrongly connected (e.g. for example d2dc6b3's parent is
0adafe3, but 0adafe3 was committed half a year later after
d2dc6b3...), so it's a mess. Also the checkouted files are not
complete.


Now, if you look at the "patches" file, to which I saved the results
of "git fast-export", you can find that the commit d717177d4  (fixed
downloads instructions in the README and a typo) has the mark :6540,
and if you search for this mark in the patches file, you can only find
2 children:

commit refs/heads/master
mark :6542
author Ondrej Certik <ondrej@certik.cz> 1198803347 +0100
committer Ondrej Certik <ondrej@certik.cz> 1198803347 +0100
data 21
credits improvements
from :6540
M 100644 :6541 README


and:

commit refs/heads/master
mark :6551
author Ondrej Certik <ondrej@certik.cz> 1198951670 +0100
committer Ondrej Certik <ondrej@certik.cz> 1198951670 +0100
data 48
Speed "import sympy" up as in the last release.
from :6540
M 100644 :6550 sympy/printing/preview.py


however, the third child doesn't contain the "from :6540":

commit refs/heads/master
mark :24
author Ondrej Certik <ondrej@certik.cz> 1198798384 +0100
committer Ondrej Certik <ondrej@certik.cz> 1198798384 +0100
data 101
Allow to specify line width in 2D plotting.


So imho that's a bug in git fast-export. What do you think?

Ondrej

Re: git fast-export | git fast-import doesn't work

[Michael J Gruber]

Johannes Schindelin venit, vidit, dixit 26.11.2008 01:14:
> Hi,
> 
> On Tue, 25 Nov 2008, Ondrej Certik wrote:
> 
>> I would like to export our whole git repository to patches, and then 
>> reconstruct it again from scratch. Following the man page of "git 
>> fast-export":
>>
>> [...] 
>>
>> However, the repository is very different to the original one. It 
>> contains only 191 patches:
> 
> Can you try again with a Git version that contains the commit 
> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?

With that commit cherry-picked onto today's master I get the same effect:

A) git fast-import crashes with dump generated by git fast-export -M -C
B) git fast-import produces disconnected DAG with dump generated by git
fast-export

With git 1.5.4, A) is good but B) is still bad.
Bisecting gives me (after remembering to put make in the run script,
uhm...)...

...

...tadah:


ae7c5dcef92d46cfc8987fde2c264614fe475bd1 is first bad commit
commit ae7c5dcef92d46cfc8987fde2c264614fe475bd1
Author: Alexander Gavrilov <angavrilov@gmail.com>
Date:   Sun Jul 27 00:52:54 2008 +0400

    Support copy and rename detection in fast-export.

Oh well :(

Michael

Re: git fast-export | git fast-import doesn't work

[Johannes Schindelin]

Hi Ondrej & Michael,

On Wed, 26 Nov 2008, Michael J Gruber wrote:

> Johannes Schindelin venit, vidit, dixit 26.11.2008 01:14:
> 
> > Can you try again with a Git version that contains the commit 
> > 2075ffb5(fast-export: use an unsorted string list for extra_refs)?

Okay, so both of your use cases seem to be real bugs in fast-export.  May 
I respectfully submit a request for a test script (as patch to 
t/t9301-fast-export.sh)  which is as short as possible and shows the 
respective bugs?

My Git time budget these days is almost negative, so I will not be able to 
work on these issues without having a small and concise example.

Thanks so much,
Dscho

Re: git fast-export | git fast-import doesn't work

[Ondrej Certik]

On Wed, Nov 26, 2008 at 1:46 PM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> Hi Ondrej & Michael,
>
> On Wed, 26 Nov 2008, Michael J Gruber wrote:
>
>> Johannes Schindelin venit, vidit, dixit 26.11.2008 01:14:
>>
>> > Can you try again with a Git version that contains the commit
>> > 2075ffb5(fast-export: use an unsorted string list for extra_refs)?
>
> Okay, so both of your use cases seem to be real bugs in fast-export.  May
> I respectfully submit a request for a test script (as patch to
> t/t9301-fast-export.sh)  which is as short as possible and shows the
> respective bugs?
>
> My Git time budget these days is almost negative, so I will not be able to
> work on these issues without having a small and concise example.

I'll do my best and try to create a simple example. I'll report back
if I have something.

Ondrej

Re: git fast-export | git fast-import doesn't work

[Michael J Gruber]

Ondrej Certik venit, vidit, dixit 26.11.2008 11:18:
> On Wed, Nov 26, 2008 at 10:35 AM, Ondrej Certik <ondrej@certik.cz> wrote:
>> On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin
>> <Johannes.Schindelin@gmx.de> wrote:
>>> Hi,
>>>
>>> On Tue, 25 Nov 2008, Ondrej Certik wrote:
>>>
>>>> I would like to export our whole git repository to patches, and then
>>>> reconstruct it again from scratch. Following the man page of "git
>>>> fast-export":
>>>>
>>>> [...]
>>>>
>>>> However, the repository is very different to the original one. It
>>>> contains only 191 patches:
>>> Can you try again with a Git version that contains the commit
>>> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?
>> I tried the next branch:
>>
>> $ git --version
>> git version 1.6.0.4.1060.g9433b
>>
>> that contains the 2075ffb5 patch. I haven't observed any change ---
>> the "git log" still only shows 191 commits (git log --all shows
>> everything).
> 
> I deleted all tags and then fast-exported and imported, now all the
> commits show in "git log", however, the patches are wrongly connected.
> Basically, both repositories are identical (including hashes) up to
> this commit:
> 
> d717177d4  (fixed downloads instructions in the README and a typo)
> 
> However, the original repo (sympy-full-history-20081023) contains 3
> children at this commit:

There's some nice 3 way branching and double 2 way merging going on. I
cut out the interesting part of the graph, making d717177d4 and
6e869485f parentless. The resulting mini DAG is reproduced correctly by
export|import, even with -M -C.

Michael

Re: git fast-export | git fast-import doesn't work

[Ondrej Certik]

On Wed, Nov 26, 2008 at 4:35 PM, Michael J Gruber
<git@drmicha.warpmail.net> wrote:
> Ondrej Certik venit, vidit, dixit 26.11.2008 11:18:
>> On Wed, Nov 26, 2008 at 10:35 AM, Ondrej Certik <ondrej@certik.cz> wrote:
>>> On Wed, Nov 26, 2008 at 1:14 AM, Johannes Schindelin
>>> <Johannes.Schindelin@gmx.de> wrote:
>>>> Hi,
>>>>
>>>> On Tue, 25 Nov 2008, Ondrej Certik wrote:
>>>>
>>>>> I would like to export our whole git repository to patches, and then
>>>>> reconstruct it again from scratch. Following the man page of "git
>>>>> fast-export":
>>>>>
>>>>> [...]
>>>>>
>>>>> However, the repository is very different to the original one. It
>>>>> contains only 191 patches:
>>>> Can you try again with a Git version that contains the commit
>>>> 2075ffb5(fast-export: use an unsorted string list for extra_refs)?
>>> I tried the next branch:
>>>
>>> $ git --version
>>> git version 1.6.0.4.1060.g9433b
>>>
>>> that contains the 2075ffb5 patch. I haven't observed any change ---
>>> the "git log" still only shows 191 commits (git log --all shows
>>> everything).
>>
>> I deleted all tags and then fast-exported and imported, now all the
>> commits show in "git log", however, the patches are wrongly connected.
>> Basically, both repositories are identical (including hashes) up to
>> this commit:
>>
>> d717177d4  (fixed downloads instructions in the README and a typo)
>>
>> However, the original repo (sympy-full-history-20081023) contains 3
>> children at this commit:
>
> There's some nice 3 way branching and double 2 way merging going on. I
> cut out the interesting part of the graph, making d717177d4 and
> 6e869485f parentless. The resulting mini DAG is reproduced correctly by
> export|import, even with -M -C.

I am also trying to make the example simpler. I tried to squash the
first uninteresting ~1500 commits into one, but "git rebase -i"
uterrly fails after squashing about 600 commits. Still investigating.

Ondrej

Re: git fast-export | git fast-import doesn't work

[Johannes Sixt]

Ondrej Certik schrieb:
> I am also trying to make the example simpler. I tried to squash the
> first uninteresting ~1500 commits into one, but "git rebase -i"
> uterrly fails after squashing about 600 commits. Still investigating.

Don't use rebase. Set a graft and rewrite the history:

   $ echo $(git rev-parse HEAD) $(git rev-parse HEAD~1500) >> \
		.git/info/grafts

Assuming "first 1500" means the "most recent 1500" commits. But you get
the idea. You can truncate history as well by omitting the second SHA1.
It's very convenient to keep gitk open and File->Reload after each graft
that you set.

When you're done with setting grafts:

   $ git filter-branch -f --tag-name-filter cat -- --all

(You are doing this on a copy of your repository, don't you?)

-- Hannes

Re: git fast-export | git fast-import doesn't work

[Johannes Schindelin]

Hi,

On Wed, 26 Nov 2008, Ondrej Certik wrote:

> I am also trying to make the example simpler. I tried to squash the 
> first uninteresting ~1500 commits into one, but "git rebase -i" uterrly 
> fails after squashing about 600 commits. Still investigating.

1500... wow.

The best idea would probably be to just "edit" the first, delete the rest 
of the 1500, and then 'git read-tree -u -m <last-of-the-1500-commits>"' on 
the command line (when git rebase stops after the "edit" command).

rebase -i was _never_ meant for such _massive_ interactions; that's just 
too much for a shell script.

Ciao,
Dscho

Re: git fast-export | git fast-import doesn't work

[Michael J Gruber]

Johannes Schindelin venit, vidit, dixit 26.11.2008 17:40:
> Hi,
> 
> On Wed, 26 Nov 2008, Ondrej Certik wrote:
> 
>> I am also trying to make the example simpler. I tried to squash the 
>> first uninteresting ~1500 commits into one, but "git rebase -i" uterrly 
>> fails after squashing about 600 commits. Still investigating.
> 
> 1500... wow.
> 
> The best idea would probably be to just "edit" the first, delete the rest 
> of the 1500, and then 'git read-tree -u -m <last-of-the-1500-commits>"' on 
> the command line (when git rebase stops after the "edit" command).
> 
> rebase -i was _never_ meant for such _massive_ interactions; that's just 
> too much for a shell script.

Or chop the DAG with grafts.

Removing the tags one by one I noticed that for several of them, removal
of the tag increases the number of commits on the connected DAG
component containing master (in the ex/imported repo), and that one
reaches the correct number with still a few tags left in there. Yet, the
topology is wrong in several places; I think all of them can be
attributed to missing parent info (which even creates new roots in some
places).

Looking at the source I suspect that fast-export fails to denote
parenthood in the case of yet unmarked parents (last for-loop of
handle_commit() in builtin_fast_export.c). But I don't really know that
code at all.

Michael

P.S.: That git repo itself is a product of
hg-fast-export|git-fast-import, right?

Re: git fast-export | git fast-import doesn't work

[Johannes Schindelin]

Hi,

On Wed, 26 Nov 2008, Michael J Gruber wrote:

> Looking at the source I suspect that fast-export fails to denote 
> parenthood in the case of yet unmarked parents (last for-loop of 
> handle_commit() in builtin_fast_export.c). But I don't really know that 
> code at all.

I strongly doubt so.  Noticed the use of has_unshown_parent(commit) in 
both cases before calling handle_commit()?

In any case, here is a script that I wrote _long_ time ago, to be able to 
reconstruct history from the output of "git rev-list --all --parents".  
Maybe this helps you in reconstructing something that is handled 
incorrectly by fast-export | fast-import, but is lighter than a full-blown 
repository.

-- snip --
#!/bin/sh

# Given the output of git-rev-list, this reconstructs the DAG of the 
history

i=0
tac | while read rev parents; do
        let i=$i+1
        echo $i > a1
        git add a1
        tree=$(git write-tree)
        parents="$(for parent in $parents
                do
                        echo -n "-p $(git rev-parse sp-$parent) "
                done)"
        commit=$(echo "$rev $i" | git commit-tree $tree $parents)
        git tag sp-$rev $commit
done
-- snap --

Ciao,
Dscho

--

          |

Ceci n'est pas une pipe

Re: git fast-export | git fast-import doesn't work

[Ondrej Certik]

On Wed, Nov 26, 2008 at 5:40 PM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Wed, 26 Nov 2008, Ondrej Certik wrote:
>
>> I am also trying to make the example simpler. I tried to squash the
>> first uninteresting ~1500 commits into one, but "git rebase -i" uterrly
>> fails after squashing about 600 commits. Still investigating.
>
> 1500... wow.
>
> The best idea would probably be to just "edit" the first, delete the rest
> of the 1500, and then 'git read-tree -u -m <last-of-the-1500-commits>"' on
> the command line (when git rebase stops after the "edit" command).

That worked, thanks! My original repo:

A -- B -- ... --- D --- E --- ...

where E and the rest of the commits (there are branches and merges in
there) are the ones that I need to preserve, but all the commits
between B and D can be squashed (~1500 of them). So I created a
branch:

A -- B -- ... --- D

then squashed the commits using the technique you described above, so
now I have:

A -- BD --

and now I would like to append "E -- ..." to it -- is there any way to
do that? I tried rebase, but that destroys all the branches and merges
and those are necessary to reproduce the fast-export bug.

>
> rebase -i was _never_ meant for such _massive_ interactions; that's just
> too much for a shell script.

In fact, I think it would work, but there is probably another bug,
that I am hitting, maybe due to whitespace problems --- in the
original repository, the patches are linear, but when I create a
branch before the failing patch and then cherry-pick it (that should
work), it fails and creates conflicts. The same behavior is with git
rebase. So I'll investigate more and report it in a separate thread,
as it is not related to fast-export.

Ondrej

Re: git fast-export | git fast-import doesn't work

[Ondrej Certik]

Hi Johannes!

On Wed, Nov 26, 2008 at 5:35 PM, Johannes Sixt <j.sixt@viscovery.net> wrote:
> Ondrej Certik schrieb:
>> I am also trying to make the example simpler. I tried to squash the
>> first uninteresting ~1500 commits into one, but "git rebase -i"
>> uterrly fails after squashing about 600 commits. Still investigating.
>
> Don't use rebase. Set a graft and rewrite the history:
>
>   $ echo $(git rev-parse HEAD) $(git rev-parse HEAD~1500) >> \
>                .git/info/grafts
>
> Assuming "first 1500" means the "most recent 1500" commits. But you get
> the idea. You can truncate history as well by omitting the second SHA1.
> It's very convenient to keep gitk open and File->Reload after each graft
> that you set.
>
> When you're done with setting grafts:
>
>   $ git filter-branch -f --tag-name-filter cat -- --all

Indeed, this seems to be working robustly. Thanks!

>
> (You are doing this on a copy of your repository, don't you?)

Yes.


I spent the whole today trying to isolate the bug, but so far I
haven't succeeded. Unfortunately, I need to work on other things now,
so I am postponing this to some later time. The repository that
reproduces it will stay online, so anyone feel free to produce a nice
and simple (failing) test for the bug.

Thanks,
Ondrej

Re: git fast-export | git fast-import doesn't work

[Andreas Ericsson]

Ondrej Certik wrote:
> On Wed, Nov 26, 2008 at 5:40 PM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>> Hi,
>>
>> On Wed, 26 Nov 2008, Ondrej Certik wrote:
>>
>>> I am also trying to make the example simpler. I tried to squash the
>>> first uninteresting ~1500 commits into one, but "git rebase -i" uterrly
>>> fails after squashing about 600 commits. Still investigating.
>> 1500... wow.
>>
>> The best idea would probably be to just "edit" the first, delete the rest
>> of the 1500, and then 'git read-tree -u -m <last-of-the-1500-commits>"' on
>> the command line (when git rebase stops after the "edit" command).
> 
> That worked, thanks! My original repo:
> 
> A -- B -- ... --- D --- E --- ...
> 
> where E and the rest of the commits (there are branches and merges in
> there) are the ones that I need to preserve, but all the commits
> between B and D can be squashed (~1500 of them). So I created a
> branch:
> 
> A -- B -- ... --- D
> 
> then squashed the commits using the technique you described above, so
> now I have:
> 
> A -- BD --
> 
> and now I would like to append "E -- ..." to it -- is there any way to
> do that? I tried rebase, but that destroys all the branches and merges
> and those are necessary to reproduce the fast-export bug.
> 

  git rebase -p

If your git is old, you'll need

  git rebase -i -p

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: git fast-export | git fast-import doesn't work

[Alexander Gavrilov]

[-- Attachment #1: Type: text/plain, Size: 3026 bytes --]

On Wednesday 26 November 2008 20:08:54 Johannes Schindelin wrote:
> On Wed, 26 Nov 2008, Michael J Gruber wrote:
> > Looking at the source I suspect that fast-export fails to denote 
> > parenthood in the case of yet unmarked parents (last for-loop of 
> > handle_commit() in builtin_fast_export.c). But I don't really know that 
> > code at all.
> 
> I strongly doubt so.  Noticed the use of has_unshown_parent(commit) in 
> both cases before calling handle_commit()?
> 
> In any case, here is a script that I wrote _long_ time ago, to be able to 
> reconstruct history from the output of "git rev-list --all --parents".  
> Maybe this helps you in reconstructing something that is handled 
> incorrectly by fast-export | fast-import, but is lighter than a full-blown 
> repository.

Today I had time to investigate this problem, and found:

1) The root of the problem is that fast-export really wants to walk
    revisions in topological order, but actually receives them in date
    order. While it is usually a good guess at topology, this repository
    contains some children that are older than their parent commits,
    e.g. see dd22c7d51a4debf18a3b2e35c61a1fec0175e4e0

2) It tries to fix minor deviations by checking the SHOWN flag.
   However, it still breaks in two ways:

  a) SHOWN is apparently set by simply walking the commits, so
      if the parent was earlier encountered on a different branch of
      the DAG, it will be handled as already shown. Basically, this
      check is only good to determine if we reached the end of the
      chain, and should start to backtrack.

  b) If I modify the code to use a completely separate flag, it still
      doesn't work, because the commits are placed on the stack in
      the wrong order, so handle_tail gets stuck, and fails to unwind
      it completely.

So, apparently, the only way to fix it is to require topological order:


diff --git a/builtin-fast-export.c b/builtin-fast-export.c
index 7d5d57a..d9261fa 100644
--- a/builtin-fast-export.c
+++ b/builtin-fast-export.c
@@ -490,6 +490,7 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)
        git_config(git_default_config, NULL);

        init_revisions(&revs, prefix);
+       revs.topo_order = 1; /* force topological ordering */
        argc = setup_revisions(argc, argv, &revs, NULL);
        argc = parse_options(argc, argv, options, fast_export_usage, 0);
        if (argc > 1)


(It is also possible to specify --topo-order on the command line)

As for the failure to import the output of fast-export with copy detection,
it is a natural consequence of a messed up order of commits, because
copy and move commands depend on the original file being there.

The attachment contains a (proper) export of a simplified sample of the
structure around dd22c7d51a4d, which clearly reproduces the problem
because all of the timestamps were made identical as a side effect of
simplification. By crafting timestamps it is probably possible to minimize
it even further.

Alexander

[-- Attachment #2: patchesx3b --]
[-- Type: text/plain, Size: 6707 bytes --]

blob
mark :1
data 5
2696

reset refs/heads/test
commit refs/heads/test
mark :2
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
e4644a29ab6cb1993fc0dab55c320b7f2076e17b 2696
M 100644 :1 foo/a1

blob
mark :3
data 5
2697

commit refs/heads/test
mark :4
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
d558c3feba34d69e8accc990267d60c3a11644c1 2697
from :2
M 100644 :3 foo/a1

blob
mark :5
data 5
2698

commit refs/heads/test
mark :6
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
2e24b35039e9253a9089162f40113e28069cce56 2698
from :4
M 100644 :5 foo/a1

blob
mark :7
data 5
2699

commit refs/heads/test
mark :8
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
0f78bea330e3a3b32f9204dfbdb9ca466ef78962 2699
from :6
M 100644 :7 foo/a1

blob
mark :9
data 5
2700

commit refs/heads/test
mark :10
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
681ea67a40527f192bf279ea6346004120c64702 2700
from :8
M 100644 :9 foo/a1

blob
mark :11
data 5
2701

commit refs/heads/test
mark :12
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
7faf0e5fb3c060385859caead8692d37c9700e55 2701
from :10
M 100644 :11 foo/a1

blob
mark :13
data 5
2702

commit refs/heads/test
mark :14
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
d6ea85387203d59a9cddda6746bd3266598dc91d 2702
from :12
M 100644 :13 foo/a1

blob
mark :15
data 5
2703

commit refs/heads/test
mark :16
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
fa22fab1d6d4cc192e288696fcb8947ebfd9e4a1 2703
from :14
M 100644 :15 foo/a1

blob
mark :17
data 5
2704

commit refs/heads/test
mark :18
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
dd22c7d51a4debf18a3b2e35c61a1fec0175e4e0 2704
from :16
M 100644 :17 foo/a1

blob
mark :19
data 5
2705

commit refs/heads/test
mark :20
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
84e2addc90ccf17d1cfcf657c241cf6cda5d46c1 2705
from :18
M 100644 :19 foo/a1

blob
mark :21
data 5
2706

commit refs/heads/test
mark :22
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
6584b0f9e720d3035739a33e5a24d3b4e3129127 2706
from :20
M 100644 :21 foo/a1

blob
mark :23
data 5
2707

commit refs/heads/test
mark :24
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
75e46ad7079a9841332731e4701b911b73b5be23 2707
from :22
M 100644 :23 foo/a1

blob
mark :25
data 5
2708

commit refs/heads/test
mark :26
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
52d087c1e3aba5c1d158e6fb35fea9c40602ac11 2708
from :24
M 100644 :25 foo/a1

blob
mark :27
data 5
2709

commit refs/heads/test
mark :28
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
384679f01372baa92ce6e43475b44e2eb607ce67 2709
from :26
M 100644 :27 foo/a1

blob
mark :29
data 5
2710

commit refs/heads/test
mark :30
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
98def8453c487d7f54ff01d92268a0aa2c68f872 2710
from :28
M 100644 :29 foo/a1

blob
mark :31
data 5
2711

commit refs/heads/test
mark :32
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
529c626be804ea6201d53856f9d4a30d62b75248 2711
from :30
M 100644 :31 foo/a1

blob
mark :33
data 5
2712

commit refs/heads/test
mark :34
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
c75ea627851e93d6fcbc3d45db1aff00832a403e 2712
from :32
merge :10
M 100644 :33 foo/a1

blob
mark :35
data 5
2713

commit refs/heads/test
mark :36
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
8c436c18ab0922c71fa2a111632d2fc9e1783342 2713
from :34
M 100644 :35 foo/a1

blob
mark :37
data 5
2714

commit refs/heads/test
mark :38
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
c71f0c84d5b812445f1857e23c8737eeb886ae3e 2714
from :36
merge :12
M 100644 :37 foo/a1

blob
mark :39
data 5
2715

commit refs/heads/test
mark :40
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
1086e78f66e10ca873e75d245442f8001f7893cf 2715
from :38
merge :14
M 100644 :39 foo/a1

blob
mark :41
data 5
2716

commit refs/heads/test
mark :42
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
8c3e2e17eb6892069c32696784795c737fb703dd 2716
from :40
merge :16
M 100644 :41 foo/a1

blob
mark :43
data 5
2717

commit refs/heads/test
mark :44
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
c9d47e2b0166eb0d3ff8af2cb65e315b96d02af8 2717
from :42
M 100644 :43 foo/a1

blob
mark :45
data 5
2718

commit refs/heads/test
mark :46
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
537dfcc8c4d15b6a1774e8b37e2ac808972c0cb5 2718
from :44
M 100644 :45 foo/a1

blob
mark :47
data 5
2719

commit refs/heads/test
mark :48
author Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
committer Alexander Gavrilov <angavrilov@gmail.com> 1228639325 +0300
data 46
594b654469e74362bf7e77872e5bc802a03fbf01 2719
from :46
M 100644 :47 foo/a1

Re: git fast-export | git fast-import doesn't work

[Johannes Schindelin]

Hi,

On Sun, 7 Dec 2008, Alexander Gavrilov wrote:

> On Wednesday 26 November 2008 20:08:54 Johannes Schindelin wrote:
> > On Wed, 26 Nov 2008, Michael J Gruber wrote:
> > > Looking at the source I suspect that fast-export fails to denote 
> > > parenthood in the case of yet unmarked parents (last for-loop of 
> > > handle_commit() in builtin_fast_export.c). But I don't really know 
> > > that code at all.
> > 
> > I strongly doubt so.  Noticed the use of has_unshown_parent(commit) in 
> > both cases before calling handle_commit()?
> > 
> > In any case, here is a script that I wrote _long_ time ago, to be able 
> > to reconstruct history from the output of "git rev-list --all 
> > --parents".  Maybe this helps you in reconstructing something that is 
> > handled incorrectly by fast-export | fast-import, but is lighter than 
> > a full-blown repository.
> 
> Today I had time to investigate this problem, and found:
> 
> 1) The root of the problem is that fast-export really wants to walk
>     revisions in topological order, but actually receives them in date
>     order.

Indeed.  Can you submit this patch with a proper commit message, adding a 
test for the issue by setting a bogus GIT_COMMITTER_DATE explicitly?

Thanks,
Dscho

Re: git fast-export | git fast-import doesn't work

[Ondrej Certik]

On Wed, Nov 26, 2008 at 9:35 AM, Johannes Sixt <j.sixt@viscovery.net> wrote:
> Ondrej Certik schrieb:
>> I am also trying to make the example simpler. I tried to squash the
>> first uninteresting ~1500 commits into one, but "git rebase -i"
>> uterrly fails after squashing about 600 commits. Still investigating.
>
> Don't use rebase. Set a graft and rewrite the history:
>
>   $ echo $(git rev-parse HEAD) $(git rev-parse HEAD~1500) >> \
>                .git/info/grafts
>
> Assuming "first 1500" means the "most recent 1500" commits. But you get
> the idea. You can truncate history as well by omitting the second SHA1.
> It's very convenient to keep gitk open and File->Reload after each graft
> that you set.
>
> When you're done with setting grafts:
>
>   $ git filter-branch -f --tag-name-filter cat -- --all
>
> (You are doing this on a copy of your repository, don't you?)

Thanks for the tip with grafts. I tried that on some other repository
when I needed to truncate the history and it works like a charm.

Ondrej