How to deal with historic tar-balls

How to deal with historic tar-balls

[nn6eumtr]

I have a number of older projects that I want to bring into a git 
repository. They predate a lot of the popular scm systems, so they are 
primarily a collection of tarballs today.

I'm fairly new to git so I have a couple questions related to this:

- What is the best approach for bringing them in? Do I just create a 
repository, then unpack the files, commit them, clean out the directory 
unpack the next tarball, and repeat until everything is loaded?

- Do I need to pay special attention to files that are renamed/removed 
from version to version?

- If the timestamps change on a file but the actual content does not, 
will git treat it as a non-change once it realizes the content hasn't 
changed?

- Last, if after loading the repository I find another version of the 
files that predates those I've loaded, or are intermediate between two 
commits I've already loaded, is there a way to go say that commit B is 
actually the ancestor of commit C? (i.e. a->c becomes a->b->c if you 
were to visualize the commit timeline or do diffs) Or do I just reload 
the tarballs in order to achieve this?

All replies appreciated!

Re: How to deal with historic tar-balls

[Tomas Carnecky]

On 12/31/11 8:04 PM, nn6eumtr wrote:
> I have a number of older projects that I want to bring into a git 
> repository. They predate a lot of the popular scm systems, so they are 
> primarily a collection of tarballs today.
>
> I'm fairly new to git so I have a couple questions related to this:
>
> - What is the best approach for bringing them in? Do I just create a 
> repository, then unpack the files, commit them, clean out the 
> directory unpack the next tarball, and repeat until everything is loaded?
>
> - Do I need to pay special attention to files that are renamed/removed 
> from version to version?
>
> - If the timestamps change on a file but the actual content does not, 
> will git treat it as a non-change once it realizes the content hasn't 
> changed?
>
> - Last, if after loading the repository I find another version of the 
> files that predates those I've loaded, or are intermediate between two 
> commits I've already loaded, is there a way to go say that commit B is 
> actually the ancestor of commit C? (i.e. a->c becomes a->b->c if you 
> were to visualize the commit timeline or do diffs) Or do I just reload 
> the tarballs in order to achieve this?

There is a script which will import sources from multiple tarballs, 
creating a commit with the contents of each tarball. It's in the git 
repository under contrib/fast-import/import-tars.perl.

tom

Re: How to deal with historic tar-balls

[Philip Oakley]

From: "Tomas Carnecky" <tom@dbservice.com> Sent: Sunday, January 01, 2012
12:27 AM
>On 12/31/11 8:04 PM, nn6eumtr wrote:
>> I have a number of older projects that I want to bring into a git
>> repository. They predate a lot of the popular scm systems, so they are
>> primarily a collection of tarballs today.
I'm doing a similar thing with a set of zip files. I grouped mine into
batches for easier checking and putting on to separate branches. Planning
your branch requirements is probably the biggest task, and will depend on
how you hope to use the new repo.

>> I'm fairly new to git so I have a couple questions related to this:
>>
>> - What is the best approach for bringing them in? Do I just create a
>> repository, then unpack the files, commit them, clean out the
>> directory unpack the next tarball, and repeat until everything is loaded?
Essentially yes; Obviously if you have an organisation in mind then you can
introduce maintenance branches etc as you develop the import.

If it is simply to create a nice history that isn't really looked at, then a
simple linear model is OK. If you need to keep old mintenance versions and
obtain diffs with newer versions then look at the various branching models
and populate appropriately.

>>
>> - Do I need to pay special attention to files that are renamed/removed
>> from version to version?
No; It is only if you use those file dates to determine the implied date for
the commit.

>>
>> - If the timestamps change on a file but the actual content does not,
>> will git treat it as a non-change once it realizes the content hasn't
>> changed?
Correct. In fact git doesn't record the time stamps anyway. It simply
records the content, and structure, of the snapshot.

>>
>> - Last, if after loading the repository I find another version of the
>> files that predates those I've loaded, or are intermediate between two
>> commits I've already loaded, is there a way to go say that commit B is
>> actually the ancestor of commit C? (i.e. a->>c becomes a->>b->>c if you
>> were to visualize the commit timeline or do diffs) Or do I just reload
>> the tarballs in order to achieve this?
You can use 'grafts' as a mechanism to re-arrange the commit order, and/or
join partial repos, and then use git filter-branch to re-write the lot as a
single cohesive repo. But this 'hack' does re-write all the commit SHA1
values, so you should minimise the number of times that happens...

It is worth capturing your import sequence as a script so that you can wash 
/ rinse / repeat as often as needed to get a result you like.

> There is a script which will import sources from multiple tarballs,
> creating a commit with the contents of each tarball. It's in the git
> repository under contrib/fast-import/import-tars.perl.
I wasn't aware of those scripts. I'll be having a look at the zip import
script for my needs.

My extra problem is that almost all my zips have an extra top level
directory that changes its name for every zip (but some don't..). The TLD
changes confuses the git rename detection if I don't remove them before
committing. Fortunately it's an internal development project with no formal
releases so creating the history is a bit of a personal project which
doesn't affect ongoing development (which is the crunch question for
fidelity of the repo you create).

> tom
Philip

Re: How to deal with historic tar-balls

[Dirk Süsserott]

Am 01.01.2012 01:27 schrieb Tomas Carnecky:
> On 12/31/11 8:04 PM, nn6eumtr wrote:
>> I have a number of older projects that I want to bring into a git
>> repository. They predate a lot of the popular scm systems, so they are
>> primarily a collection of tarballs today.
>>
>> I'm fairly new to git so I have a couple questions related to this:
>>
>> - What is the best approach for bringing them in? Do I just create a
>> repository, then unpack the files, commit them, clean out the
>> directory unpack the next tarball, and repeat until everything is loaded?
>>
>> - Do I need to pay special attention to files that are renamed/removed
>> from version to version?
>>
>> - If the timestamps change on a file but the actual content does not,
>> will git treat it as a non-change once it realizes the content hasn't
>> changed?
>>
>> - Last, if after loading the repository I find another version of the
>> files that predates those I've loaded, or are intermediate between two
>> commits I've already loaded, is there a way to go say that commit B is
>> actually the ancestor of commit C? (i.e. a->c becomes a->b->c if you
>> were to visualize the commit timeline or do diffs) Or do I just reload
>> the tarballs in order to achieve this?
> 
> There is a script which will import sources from multiple tarballs,
> creating a commit with the contents of each tarball. It's in the git
> repository under contrib/fast-import/import-tars.perl.
> 
> tom

@tom: True. I didn't know about that script, but it should work.

@nn6eumtr: Basically your workflow is perfect. But let me give you some
explanation:

git init
foreach archive in *.tar; do
    tar xf $archive
    git add --all .
    git commit -m "Added $archive"
    # now remove everything except for the .git directory
    # with regular shell commands (rm -rf *). Also remove
    # any dot-files (and the tarball itself, if it's in the
    # current directory).
done

Notice the '--all' switch to 'git add': Normally, 'git add .' adds all
files that match the given pattern '.', i.e. all files in the current
directory (and below, it's recursive). The '--all' switch together with
the pattern '.' adds or updates all files already known to git *AND*
adds the files not yet known *AND* removes the files that are no longer
in the working tree. That's exactly what you want.

Consider archive1.tar with files A, B, C:

  git add --all . # will add A, B, and C

Now remove A, B, C, and unpack archive2.tar. Assume it has files B, C,
D. A was deleted, B was changed, C is unchanged, D is new.

  git add --all . # will remove A, add B, leave C, add D.

git will notice that C hasn't changed its content (timestamp doesn't
matter).

Without the '--all' switch, git would simply add B and D.

There is no problem re-arranging the history after your import (see "git
rebase --help", especially the --interactive section), but then you
probably will have conflicts and have to resolve them. I'd suggest to
re-start the import instead.

Please note that "for archive in *.tar" will pick the tarballs in
lexicographical order. That might not be your intention.

HTH,
    Dirk

Re: How to deal with historic tar-balls

[Philip Oakley]

From: "Philip Oakley" <philipoakley@iee.org> Sent: Sunday, January 01, 2012 
6:30 PM
> From: "Tomas Carnecky" <tom@dbservice.com> Sent: Sunday, January 01, 2012
> 12:27 AM
>>On 12/31/11 8:04 PM, nn6eumtr wrote:
>>> I have a number of older projects that I want to bring into a git
>>> repository. They predate a lot of the popular scm systems, so they are
>>> primarily a collection of tarballs today.
> I'm doing a similar thing with a set of zip files. I grouped mine into
> batches for easier checking and putting on to separate branches. Planning
> your branch requirements is probably the biggest task, and will depend on
> how you hope to use the new repo.
>
<snip>
>> There is a script which will import sources from multiple tarballs,
>> creating a commit with the contents of each tarball. It's in the git
>> repository under contrib/fast-import/import-tars.perl.
> I wasn't aware of those scripts. I'll be having a look at the zip import
> script for my needs.

Is there a mechanism for either having fast-import respect a .gitignore,
or determining if a given file/path should be ignored?
My zips contain a lot of compile by-products that should be excluded from 
the repo.

> My extra problem is that almost all my zips have an extra top level
> directory that changes its name for every zip (but some don't..). The TLD
> changes confuses the git rename detection if I don't remove them before
> committing. Fortunately it's an internal development project with no 
> formal
> releases so creating the history is a bit of a personal project which
> doesn't affect ongoing development (which is the crunch question for
> fidelity of the repo you create).
>
>> tom
> Philip

Re: How to deal with historic tar-balls

[Philip Oakley]

From: "Philip Oakley" <philipoakley@iee.org>: Sunday, January 01, 2012 6:30 
PM
> From: "Tomas Carnecky" <tom@dbservice.com> : Sunday, January 01, 2012 
> 12:27 AM
>>On 12/31/11 8:04 PM, nn6eumtr wrote:
>>> I have a number of older projects that I want to bring into a git
>>> repository. They predate a lot of the popular scm systems, so they are
>>> primarily a collection of tarballs today.
>> There is a script which will import sources from multiple tarballs,
>> creating a commit with the contents of each tarball. It's in the git
>> repository under contrib/fast-import/import-tars.perl.
> I wasn't aware of those scripts. I'll be having a look at the zip import
> script for my needs.
>
>> tom
> Philip
>
I had a look at the script but Python isn't part of the Msysgit install, so 
the example wouldn't run.

Also I couldn't see how the "fast_import.write(" method was being created - 
my ignorance of Python? Otherwise I could look at scripting it.

Philip 

Re: How to deal with historic tar-balls

[Dirk Süsserott]

Am 02.01.2012 11:07 schrieb Philip Oakley:
> From: "Philip Oakley" <philipoakley@iee.org>: Sunday, January 01, 2012
> 6:30 PM
>> From: "Tomas Carnecky" <tom@dbservice.com> : Sunday, January 01, 2012
>> 12:27 AM
>>> On 12/31/11 8:04 PM, nn6eumtr wrote:
>>>> I have a number of older projects that I want to bring into a git
>>>> repository. They predate a lot of the popular scm systems, so they are
>>>> primarily a collection of tarballs today.
>>> There is a script which will import sources from multiple tarballs,
>>> creating a commit with the contents of each tarball. It's in the git
>>> repository under contrib/fast-import/import-tars.perl.
>> I wasn't aware of those scripts. I'll be having a look at the zip import
>> script for my needs.
>>
>>> tom
>> Philip
>>
> I had a look at the script but Python isn't part of the Msysgit install,
> so the example wouldn't run.
> 
> Also I couldn't see how the "fast_import.write(" method was being
> created - my ignorance of Python? Otherwise I could look at scripting it.
> 
> Philip

Philip,

I'm not a Python guy, but I think fast_import.write() writes sth. to
whatever the popen() call in line 24 returned:

  fast_import = popen('git fast-import --quiet', 'w')

I guess it returns a filehandle and 'git fast-import' reads its data
from stdin. My guess is, that -- instead of writing to that pipe -- you
could as well write everything to a temporary file and finally call

  git fast-import < $tempfile

But that's only a guess.

Dirk

Re: How to deal with historic tar-balls

[Philip Oakley]

Many thanks - That explanation works for me. I just hadn't seen the 
association.
Philip

From: "Dirk Süsserott" <newsletter@dirk.my1.cc>: Monday, January 02, 2012 
6:26 PM
> Am 02.01.2012 11:07 schrieb Philip Oakley:
>>> From: "Tomas Carnecky" <tom@dbservice.com> : Sunday, January 01, 2012
>>> 12:27 AM
>>>> On 12/31/11 8:04 PM, nn6eumtr wrote:
>>>>> I have a number of older projects that I want to bring into a git
>>>>> repository. They predate a lot of the popular scm systems, so they are
>>>>> primarily a collection of tarballs today.
>>>> There is a script which will import sources from multiple tarballs,
>>>> creating a commit with the contents of each tarball. It's in the git
>>>> repository under contrib/fast-import/import-tars.perl.
>>> I wasn't aware of those scripts. I'll be having a look at the zip import
>>> script for my needs.
>>>
>>>> tom
>>> Philip
>>>
>> I had a look at the script but Python isn't part of the Msysgit install,
>> so the example wouldn't run.
>>
>> Also I couldn't see how the "fast_import.write(" method was being
>> created - my ignorance of Python? Otherwise I could look at scripting it.
>>
>> Philip
>
> Philip,
>
> I'm not a Python guy, but I think fast_import.write() writes sth. to
> whatever the popen() call in line 24 returned:
>
>  fast_import = popen('git fast-import --quiet', 'w')
>
> I guess it returns a filehandle and 'git fast-import' reads its data
> from stdin. My guess is, that -- instead of writing to that pipe -- you
> could as well write everything to a temporary file and finally call
>
>  git fast-import < $tempfile
>
> But that's only a guess.
>
> Dirk
>> 

Re: How to deal with historic tar-balls

[Neal Kreitzinger]

On 12/31/2011 1:04 PM, nn6eumtr wrote:
> I have a number of older projects that I want to bring into a git
> repository. They predate a lot of the popular scm systems, so they
> are primarily a collection of tarballs today.
>
> I'm fairly new to git so I have a couple questions related to this:
>
> - What is the best approach for bringing them in? Do I just create a
>  repository, then unpack the files, commit them, clean out the
> directory unpack the next tarball, and repeat until everything is
> loaded?
>
> - Do I need to pay special attention to files that are
> renamed/removed from version to version?
>
> - If the timestamps change on a file but the actual content does not,
>  will git treat it as a non-change once it realizes the content
> hasn't changed?
>
> - Last, if after loading the repository I find another version of the
>  files that predates those I've loaded, or are intermediate between
> two commits I've already loaded, is there a way to go say that commit
> B is actually the ancestor of commit C? (i.e. a->c becomes a->b->c if
> you were to visualize the commit timeline or do diffs) Or do I just
> reload the tarballs in order to achieve this?
>
The git-rm manpage contains instructions under the "vendor code drop"
section on how to do this.  I imagine you will want to do each one
manually instead of queueing them up in a script because you are likely 
going to want to do appropriate clean up of the working tree in each 
iteration before committing.  This is where you would review 
renames/removes with git-status before you git-add and git-commit. 
Also, if you are tracking permissions in git (the executable bit) then 
you will want to filter out any noise generated by frivolous permissions 
changes between the tarball contents.

In regard to inserting tarballs into the history that depends on when 
you think you plan on doing that.  You are only going to be able to do 
that before the history is published (made "public" for other repos to 
pull down).  Otherwise you will be rewriting published history which is 
a big no-no (see git-rebase manpage).  I suggest you do your homework 
and order them properly before you start because that will be less work. 
  If you still find that you missed something then you can use 
interactive git-rebase to insert.  I'm assuming a single "master" branch 
with linear history is your desired end result.  If you want to create 
maintenance branches showing release history then you will definitely 
need to do your homework first (see gitworkflow manpage).

If you venture into rebase territory by rewriting history (inserting 
missed tarballs in between older commits) you will need to be sure to 
review your automatic merge resolutions.  Git only generates 
merge-conflicts on same-file-same-line conflicts.  It will auto-merge 
same-file-different-line changes.

You also need to ask yourself if you really need a history of all those 
versions.  To exaggerate, if all you really need is the current state 
then you need to ask yourself if it's worth the effort to record the 
previous states.  Maybe what you want is something in-between (a happy 
medium).

In regard to the 'start-over' method of inserting missed tarballs you 
would just git-reset --hard to the commit you want to insert on-top-of, 
add the tarball, and then re-apply the subsequent tarballs.  If you are 
doing cleanup between commits then the rebase or cherry-pick of the 
already cleaned-up subsequent commits from the "old-branch" (previous 
attempt) onto the 'do-over' branch will likely be easier.  (You can just 
do 'git branch old-branch' on your branch before the git-reset --hard 
(do-over) and that will give you a "backup copy" of the "previous 
attempt" called "old-branch" that you can salvage already-done-work from 
by using rebase or cherry-pick.)

Hope this helps.

v/r,
neal

Re: How to deal with historic tar-balls

[nn6eumtr]

Thanks for the response, there is lots of good information there.

One clarification - can you track renames in git? I tried using git mv 
but from the status output it looks like it deleted the old file  and 
added the new file. I was expecting it to record some sort of indicator 
of the name change, instead it looks like a short-cut for delete & add, 
the docs aren't clear if that is the case.

On 1/5/2012 10:25 AM, Neal Kreitzinger wrote:
...
> going to want to do appropriate clean up of the working tree in each
> iteration before committing. This is where you would review
> renames/removes with git-status before you git-add and git-commit. Also,
> if you are tracking permissions in git (the executable bit) then you
> will want to filter out any noise generated by frivolous permissions
> changes between the tarball contents.
...

Re: How to deal with historic tar-balls

[Thomas Rast]

nn6eumtr <nn6eumtr@gmail.com> writes:

> Thanks for the response, there is lots of good information there.
>
> One clarification - can you track renames in git? I tried using git mv
> but from the status output it looks like it deleted the old file  and
> added the new file. I was expecting it to record some sort of
> indicator of the name change, instead it looks like a short-cut for
> delete & add, the docs aren't clear if that is the case.

Git only stores snapshots; so for an ordinary (non-merge, non-root)
commit, you have the "before" (parent) and "after" (commit's) snapshot.
Everything is generated on the fly from that, including diffs, heuristic
rename detection, pickaxe, ...

To apply rename detection when diffing (e.g. in diff, log, show,
format-patch), use the -M flag.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: How to deal with historic tar-balls

[Neal Kreitzinger]

On 1/6/2012 7:10 PM, nn6eumtr wrote:
> Thanks for the response, there is lots of good information there.
>
> One clarification - can you track renames in git? I tried using git mv 
> but from the status output it looks like it deleted the old file  and 
> added the new file. I was expecting it to record some sort of 
> indicator of the name change, instead it looks like a short-cut for 
> delete & add, the docs aren't clear if that is the case.
>
(note: top-posting is not advised.)
You are exactly right in your observation:  git-mv is only a shortcut 
for 'remove old then add new'.  Git does not explicitly track 
"renames".  It can detect renames easily in the cases where you really 
just renamed the file and left the contents the same.  Git tracks 
content (and trees) as opposed to files (and file names).  Git stores 
the 'blob', aka 'contents' of files in the object store.  So if you have 
30 files with different names and the exact same contents in your work 
tree they are stored as a single blob in the .git/objects "object 
store".  If some of your "renames" are really "I renamed it and then I 
modified it" then git will have a harder time detecting the "rename" 
depending on how much you modified it.  In such cases what you really 
did is arguably not a "rename" anyway.  You can record your "renames" 
manually in your commit message if appropriate.  If you have 5 minutes 
you can watch this video from the 15:00 min to 20:59 min marks to get an 
explanation of git-mv and rename-detection: 
http://www.youtube.com/watch?v=j45cs5_nY2k (youtube searchstring: 'git 
google tech talks', result: 'contributing with git'.)

v/r,
neal