SVN -> Git *but* with special changes

SVN -> Git *but* with special changes

[Abscissa]

I have a couple big projects in SVN that I'd like to convert to Git. Being in
SVN, they've operated under a couple assumptions that are not true under
Git. These assumptions are:

1. Directories can exist even if there's nothing in them (just like any
filesystem).

2. Keeping binary files in version control isn't a big deal because the
whole repo doesn't get copied to everyone's system or use up people's GitHub
storage space.

The SVN repos have been relying on those, but both are false under Git, so I
need to do a "modified" conversion, rather than just a straight one.

So, how can I convert an SVN repo to Git, and have the conversion add dummy
files to empty directories and exclude specific files? (Also, there are tags
and branches to be converted too, in the SVN-standard "tags" and "branches"
directories.)

--
View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p6840904.html
Sent from the git mailing list archive at Nabble.com.

Re: SVN -> Git *but* with special changes

[Stephen Bash]

----- Original Message -----
> From: "Abscissa" <bus_nabble_git@semitwist.com>
> To: git@vger.kernel.org
> Sent: Wednesday, September 28, 2011 12:37:29 PM
> Subject: SVN -> Git *but* with special changes
>
> I have a couple big projects in SVN that I'd like to convert to Git.
> Being in SVN, they've operated under a couple assumptions that are 
> not true under Git. These assumptions are:
> 
> 1. Directories can exist even if there's nothing in them (just like
> any filesystem).

What requires the empty directories to exist?  The build system?  Can you just let them go away in Git and fix it downstream in the user's working copy?
 
> 2. Keeping binary files in version control isn't a big deal because
> the whole repo doesn't get copied to everyone's system or use up 
> people's GitHub storage space.

I'd eliminate these on the SVN side before converting to Git.  If you have svnadmin access to the SVN repo svnadmin dump, svndumpfilter, svnadmin load is a pretty easy process.

> The SVN repos have been relying on those, but both are false under
> Git, so I need to do a "modified" conversion, rather than just a 
> straight one.
> 
> So, how can I convert an SVN repo to Git, and have the conversion add
> dummy files to empty directories and exclude specific files? (Also, 
> there are tags and branches to be converted too, in the SVN-standard 
> "tags" and "branches" directories.)

To do the actual conversion, svn-fe and git fast-import are by far the quickest way to get the data into Git.  Filtering into tags and branches is a bit of a trick though [1].  git-svn has (IMO) a good branching UI, but can be very slow for large repositories.

[1] http://thread.gmane.org/gmane.comp.version-control.git/158940/focus=159151 : note this thread is almost a year out of date now, and even I know much better ways to go about this now -- but the scripts are not written.

HTH,
Stephen

Re: SVN -> Git *but* with special changes

[Matthieu Moy]

Stephen Bash <bash@genarts.com> writes:

>> 2. Keeping binary files in version control isn't a big deal because
>> the whole repo doesn't get copied to everyone's system or use up 
>> people's GitHub storage space.
>
> I'd eliminate these on the SVN side before converting to Git. If you
> have svnadmin access to the SVN repo svnadmin dump, svndumpfilter,
> svnadmin load is a pretty easy process.

Otherwise, you can do it on the Git side with:

  git filter-branch --tree-filter 'rm -f some-large-blob'

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: SVN -> Git *but* with special changes

[Abscissa]

>> 1. Directories can exist even if there's nothing in them (just like
>> any filesystem).
>
>What requires the empty directories to exist?  The build system?

Yes.

>Can you just let them go away in Git and fix it downstream in the user's
working copy?

That won't work retroactively for already-existing revisions. I'd rather not
break the build system for older revisions just for such a goofy reason as
empty dirs being deemed unnecessary.


--
View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p6841234.html
Sent from the git mailing list archive at Nabble.com.

Re: SVN -> Git *but* with special changes

[Jeff King]

On Wed, Sep 28, 2011 at 09:37:29AM -0700, Abscissa wrote:

> I have a couple big projects in SVN that I'd like to convert to Git. Being in
> SVN, they've operated under a couple assumptions that are not true under
> Git. These assumptions are:
> 
> 1. Directories can exist even if there's nothing in them (just like any
> filesystem).

The usual technique is to put an empty .gitignore file into the empty
directory, which will make sure it always exists.  If you import with
git-svn, it supports "--preserve-empty-dirs", which will do this for you
automatically.

> 2. Keeping binary files in version control isn't a big deal because the
> whole repo doesn't get copied to everyone's system or use up people's GitHub
> storage space.

git-svn --ignore-paths will handle this for you.

Side note on the GitHub thing: all of the forks will share objects, so
it's not much as space as you might think. And all of the paid plans
count repos, not bytes. Of course, the painful part is probably cloning
the big objects to everybody's workstation. :)

> (Also, there are tags and branches to be converted too, in the
> SVN-standard "tags" and "branches" directories.)

git svn --stdlayout ?

-Peff

Re: SVN -> Git *but* with special changes

[Jeff King]

On Wed, Sep 28, 2011 at 07:44:24PM +0200, Matthieu Moy wrote:

> > I'd eliminate these on the SVN side before converting to Git. If you
> > have svnadmin access to the SVN repo svnadmin dump, svndumpfilter,
> > svnadmin load is a pretty easy process.
> 
> Otherwise, you can do it on the Git side with:
> 
>   git filter-branch --tree-filter 'rm -f some-large-blob'

It's much more efficient to do:

    git filter-branch --index-filter \
      'git rm --cached --ignore-unmatch some-large-blob'

which avoids checking out the large files over and over[1], just to
delete them. Of course, it's even more efficient not to import them from
svn in the first place. :)

-Peff

[1] This is straight from the filter-branch manpage, btw.

Re: SVN -> Git *but* with special changes

[Abscissa]

I got busy with other things and haven't had a chance to get back to this
until now. I very much appreciate the help but I'm still unclear on this
part:

>git-svn has (IMO) a good branching UI, but can be very slow for large
repositories.
>
>[1]
http://thread.gmane.org/gmane.comp.version-control.git/158940/focus=159151 :
note this thread is almost a year out of date now, and even I know much
better ways to go about this now -- but the scripts are not written. 

I don't really understand that. So...how do I convert from svn to git while
keeping branches and tags intact? If it's automated, I don't care about
speed: I can just start it and move on to other things while it's working.


--
View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7162793.html
Sent from the git mailing list archive at Nabble.com.

Re: SVN -> Git *but* with special changes

[Abscissa]

>If you import with
>git-svn, it supports "--preserve-empty-dirs", which will do this for you
>automatically. 

I came across this, and it seems to mostly work:
http://john.albin.net/git/git-svn-migrate

Except it fails to do the "--preserve-empty-dirs". It does say that it
passes any options it doesn't recognize directly to git-svn, so I gave it
that, but just got:

- Cloning repository...
Unknown option: preserve-empty-dirs

Now, I realize this isn't a forum for that particular "git-svn-migrate"
tool, and I don't expect anyone to go digging though it on my account, so
I'm willing to try to add it into the script if I can just find out the
proper way to use "--preserve-empty-dirs".

However, I suspect that script may not be my problem at all: If I do this:

git svn help | grep preserve

I get absolutely nothing. There doesn't seem to be a
"--preserve-empty-dirs".


--
View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7163706.html
Sent from the git mailing list archive at Nabble.com.

Re: SVN -> Git *but* with special changes

[Jeff King]

On Sat, Jan 07, 2012 at 09:03:51PM -0800, Abscissa wrote:

> However, I suspect that script may not be my problem at all: If I do this:
> 
> git svn help | grep preserve
> 
> I get absolutely nothing. There doesn't seem to be a
> "--preserve-empty-dirs".

What version of git are you using? That option was added in git 1.7.7.

-Peff

Re: SVN -> Git *but* with special changes

[Abscissa]

Ok, I see. It's reporting 1.7.0, so that would explain it. One "sudo apt-get
upgrade git" and...erm...well, it seems to be upgrading my whole damn
computer, but I'll see how it all works out. Thanks!


--
View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7163737.html
Sent from the git mailing list archive at Nabble.com.

Re: SVN -> Git *but* with special changes

[Abscissa]

Well that's strange, it finished "upgrading", but now git is still just
reporting 1.7.0.4, which is *exactly* the same version it said before. The
git-svn package should already be up-to-date because I just installed it 
today. So I don't know what's up with that.


--
View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7163752.html
Sent from the git mailing list archive at Nabble.com.

Re: SVN -> Git *but* with special changes

[Carlos Martín Nieto]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

On Sat, Jan 07, 2012 at 09:25:27PM -0800, Abscissa wrote:
> Well that's strange, it finished "upgrading", but now git is still just
> reporting 1.7.0.4, which is *exactly* the same version it said before. The
> git-svn package should already be up-to-date because I just installed it 
> today. So I don't know what's up with that.

Nothing odd about that. apt-get upgrade means "upgrade my system". If
you want to get a newer version of package X, you do apt-get install X
and it will install the latest version of that package.

Your OS seems ancient, you might be better off installing from source.

   cmn

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: SVN -> Git *but* with special changes

[Andreas Schwab]

Carlos Martín Nieto <cmn@elego.de> writes:

> On Sat, Jan 07, 2012 at 09:25:27PM -0800, Abscissa wrote:
>> Well that's strange, it finished "upgrading", but now git is still just
>> reporting 1.7.0.4, which is *exactly* the same version it said before. The
>> git-svn package should already be up-to-date because I just installed it 
>> today. So I don't know what's up with that.
>
> Nothing odd about that. apt-get upgrade means "upgrade my system". If
> you want to get a newer version of package X, you do apt-get install X
> and it will install the latest version of that package.

If apt-get upgrade doesn't get you a newer version then apt-get install
won't help you either.  Both use the same installation source.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: SVN -> Git *but* with special changes

[Thomas Hochstein]

Abscissa schrieb:

> Ok, I see. It's reporting 1.7.0, so that would explain it. One "sudo apt-get
> upgrade git" and...erm...well, it seems to be upgrading my whole damn
> computer, but I'll see how it all works out. Thanks!

"apt-get upgrade" will upgrade _all_ packages and doesn't take a
parameter:
| upgrade
|    upgrade is used to install the newest versions of all packages
|    currently installed on the system from the sources enumerated in
|    /etc/apt/sources.list. 

-thh

Re: SVN -> Git *but* with special changes

[Thomas Hochstein]

Abscissa wrote:

> Well that's strange, it finished "upgrading", but now git is still just
> reporting 1.7.0.4, which is *exactly* the same version it said before. 

What kind of distribution do you use?

"apt-get" sounds like Debian or Ubuntu, but those all have at least
git 1.7.1:

| Debian versions:
| stable      1:1.7.2.5-3 
| stable-bpo  1:1.7.7.3-1~bpo60+2 
| testing     1:1.7.7.3-1 
| unstable    1:1.7.8.2-1 
| exp         1:1.7.8~rc3-1 

| Ubuntu versions:
| Precise Pangolin  1:1.7.7.3-1
| Oneiric Ocelot    1:1.7.5.4-1
| Natty Narwhal     1:1.7.4.1-3
| Maverick Meerkat  1:1.7.1-1.1ubuntu0.1

-thh

Re: SVN -> Git *but* with special changes

[Adam Borowski]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

On Sun, Jan 08, 2012 at 11:47:52AM +0100, Andreas Schwab wrote:
> Carlos Martín Nieto <cmn@elego.de> writes:
> > On Sat, Jan 07, 2012 at 09:25:27PM -0800, Abscissa wrote:
> >> Well that's strange, it finished "upgrading", but now git is still just
> >> reporting 1.7.0.4, which is *exactly* the same version it said before. The
> >> git-svn package should already be up-to-date because I just installed it 
> >> today. So I don't know what's up with that.
> >
> > Nothing odd about that. apt-get upgrade means "upgrade my system". If
> > you want to get a newer version of package X, you do apt-get install X
> > and it will install the latest version of that package.
> 
> If apt-get upgrade doesn't get you a newer version then apt-get install
> won't help you either.

No, this is true only if none of packages involved uses a new library
(including new sonames).  "apt-get upgrade" is forbidden to add or remove
packages, and thus will skip upgrades that need a new dependency.
The message at the end will mention "## not upgraded" though.

"apt-get install X" or "apt-get dist-upgrade" have no such restrictions.

-- 
1KB		// Yo momma uses IPv4!

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: SVN -> Git *but* with special changes

[Abscissa]

I see. That's strange, I've been using "apt-get upgrade xxx" for at least a
couple years, I can't believe I never knew that! Thanks for the detailed
info, everyone (and for the patience!)

In answer to someone's question, yea, it's Ubuntu (Kubuntu 10.04, and yea, I
know that's old, but it's not my primary system and I haven't had a chance
yet to upgrade it and get everything set back up again). I also tried it on
a Debian 6 Live/Persistent system and got the same results...apparently for
the same reason.

I did manage to get 1.7.8 installed (on both systems) by downloading and
installing the source (it was kind of a pain figuring out the names of some
of the dependent packages, but I managed to get it.)

So hopefully this should all work now...


--
View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7165979.html
Sent from the git mailing list archive at Nabble.com.

Re: SVN -> Git *but* with special changes

[Abscissa]

Damn. That 'git-svn-migrate' tool is just giving me errors now. It just gets
through little over 10% of the revisions, and gives:

-----------------------------
Failed to strip path 'bin/lang/.gitignore' ((?-xism:^trunk(/|$)))

- Converting svn:ignore properties into a .gitignore file...
fatal: ambiguous argument 'HEAD': unknown revision or path not in the
working tree.
Use '--' to separate paths from revisions
log --no-color --no-decorate --first-parent --pretty=medium HEAD: command
returned error: 128
-----------------------------

And then it keeps going and gives:

-----------------------------
To /var/git/Goldie.git
 ! [rejected]        master -> trunk (non-fast-forward)
error: failed to push some refs to '/var/git/Goldie.git'
To prevent you from losing history, non-fast-forward updates were rejected
-----------------------------

I guess I'll have to try some other approach. :/


--
View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7166084.html
Sent from the git mailing list archive at Nabble.com.

Re: SVN -> Git *but* with special changes

[Michael Haggerty]

On 01/08/2012 11:28 PM, Abscissa wrote:
> In answer to someone's question, yea, it's Ubuntu (Kubuntu 10.04, and yea, I
> know that's old, but it's not my primary system and I haven't had a chance
> yet to upgrade it and get everything set back up again). I also tried it on
> a Debian 6 Live/Persistent system and got the same results...apparently for
> the same reason.

For Ubuntu users: there is a git PPA [1] which usually has quite
up-to-date git packages for recent Ubuntu releases.  If you configure
your system to use this PPA (instructions are on the page) then you can
stay up-to-date with minimal effort.  Currently they have version
1.7.8.2 available for hardy, lucid, maverick, natty, and oneiric and
older versions for some other Ubuntu releases.

Caveat: I am not affiliated with the PPA and cannot vouch for its integrity.

Michael

[1] https://launchpad.net/~git-core/+archive/ppa

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

Re: SVN -> Git *but* with special changes

[Abscissa]

I'm trying to do it manually with git-svn, but I'm getting the same problem.
I have two different SVN repos I want to do (turns out neither have
branches, I had thought they did, but one of them does have tags). Doing
this:

git svn clone {the url to the SVN repo} --prefix=svn/ --preserve-empty-dirs
--authors-file={already prepared authors file} --trunk=trunk --tags=tags

It gets partway through, and then gives me:

Failed to strip path '{some path}/.gitignore' ((?-xism:^trunk(/|$)))

Where {some path} is an empty dir in the trunk of one repo, and a completely
non-existent path in the other. (In both cases I'm looking at the revision
git-svn had gotten to when bailing, plus the revision before and the
revision after).

I don't have a clue what that error message is trying to tell me or what is
going wrong. :(

If it helps, these are the repos I'm trying to convert:

http://svn.dsource.org/projects/semitwist   (git svn clone fails at r46 (out
of 242) on 'src/nonFatalAssertTest/.gitignore')
http://svn.dsource.org/projects/goldie   (git svn clone fails at r85 (out of
557) on 'bin/lang/.gitignore')

(Yes, they do have a non-standard top-level "downloads" directory, but
that's just how that host does file downloads, and I *don't* need to
preserve it)


--
View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7181897.html
Sent from the git mailing list archive at Nabble.com.

Re: SVN -> Git *but* with special changes

[Abscissa]

Someone on a different message board noticed that it seemed like it was
trying to strip and already-stripped path in git-svn.perl, so I went in
there, changed the "die" to "print", and after a "make && sudo make install"
it seems to work fine now. Probably not the proper way to fix it, but it
seems to work for me.

It does now fail to delete directories once they actually *are* deleted in
the SVN repo, which is kinda sloppy, but it doesn't hurt anything, so I can
live with that.


--
View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7186699.html
Sent from the git mailing list archive at Nabble.com.