Why does git-svn redownload revisions?

Why does git-svn redownload revisions?

[Daniel Trebbien]

For my first time using git-svn, I decided that I wanted to convert the GNU Nano
Subversion repository to a git repo. I finally settled on the following series
of commands after some initial trial-and-error (e.g. having the wrong
`i18n.commitencoding` and not specifying a Subversion authors file):
git svn init -s svn://svn.sv.gnu.org/nano
git config svn.authorsfile ~/projects/nano/svn.authorsfile
git config i18n.commitencoding 'ISO-8859-1'
git svn fetch

The issue is that the fetch runs up to revision 4250, but then mysteriously
begins redownloading revisions 1 up to four thousand something in importing
the `nano_2_1_1` tag:
...
r4248 = 7c444dd667b9629ce92a53e8d35ad2d178e5735f (refs/remotes/trunk)
	M	nano/ChangeLog
	M	nano/configure.ac
	M	nano/po/cs.po
	M	nano/po/pt_BR.po
	M	nano/po/es.po
	M	nano/po/eu.po
	M	nano/po/hu.po
	M	nano/po/vi.po
	M	nano/po/nano.pot
	M	nano/po/ms.po
	M	nano/po/uk.po
	M	nano/po/ro.po
	M	nano/po/ru.po
	M	nano/po/rw.po
	M	nano/po/id.po
	M	nano/po/nb.po
	M	nano/po/gl.po
	M	nano/po/fr.po
	M	nano/po/nl.po
	M	nano/po/nn.po
	M	nano/po/pl.po
	M	nano/po/it.po
	M	nano/po/ca.po
	M	nano/po/da.po
	M	nano/po/sr.po
	M	nano/po/tr.po
	M	nano/po/ga.po
	M	nano/po/bg.po
	M	nano/po/sv.po
	M	nano/po/de.po
	M	nano/po/zh_TW.po
	M	nano/po/fi.po
	M	nano/po/zh_CN.po
r4249 = 6fbad8a00fa067d1e3de913f77db08c6117843c7 (refs/remotes/trunk)
	M	nano/NEWS
r4250 = 704700855e5112d75654e3e7461e896f49e10fd8 (refs/remotes/trunk)
Found possible branch point: svn://svn.sv.gnu.org/nano/trunk/nano =>
svn://svn.sv.gnu.org/nano/tags/nano_2_1_1, 4248
Initializing parent: refs/remotes/tags/nano_2_1_1@4248
	A	mkinstalldirs
	A	utils.c
	A	nano.h
	A	global.c
	A	configure
	A	Makefile.in
	A	AUTHORS
	A	configure.in
	A	ChangeLog
	A	proto.h
	A	nano.1
	A	nano.1.html
	A	README
	A	acconfig.h
	A	BUGS
	A	config.h.in
	A	ABOUT-NLS
	A	TODO
	A	INSTALL
	A	intl/po2tbl.sed.in
	A	intl/loadinfo.h
	A	intl/Makefile.in
	A	intl/explodename.c
	A	intl/VERSION
	A	intl/xopen-msg.sed
	A	intl/ChangeLog
	A	intl/finddomain.c
	A	intl/localealias.c
	A	intl/gettextP.h
	A	intl/textdomain.c
	A	intl/linux-msg.sed
	A	intl/l10nflist.c
	A	intl/loadmsgcat.c
	A	intl/libgettext.h
	A	intl/bindtextdom.c
	A	intl/gettext.c
	A	intl/intl-compat.c
	A	intl/dgettext.c
	A	intl/cat-compat.c
	A	intl/gettext.h
	A	intl/dcgettext.c
	A	intl/hash-string.h
	A	winio.c
	A	COPYING
	A	Makefile.am
	A	missing
	A	NEWS
	A	cut.c
	A	nano.c
	A	aclocal.m4
	A	install-sh
	A	po/cat-id-tbl.c
	A	po/stamp-cat-id
	A	po/es.po
	A	po/fr.po
	A	po/de.po
	A	po/ChangeLog
	A	po/Makefile.in.in
	A	po/es.gmo
	A	po/fr.gmo
	A	po/de.gmo
	A	po/it.po
	A	po/POTFILES.in
	A	po/nano.pot
	A	po/it.gmo
	A	stamp-h.in
r2 = 842a208235bd2a1181250766996f7797e74a8608
(refs/remotes/tags/nano_2_1_1@4248)
	M	winio.c
r6 = 280cfdcb8533ffcb424b3a86b7392ccfbc054e60
(refs/remotes/tags/nano_2_1_1@4248)
	M	ChangeLog
r7 = 6951e65102dd1e005761dec6eb082f0a2d9327a2
(refs/remotes/tags/nano_2_1_1@4248)
	M	AUTHORS
...


Why is git-svn redownloading revision history? Also, why hasn't git-svn
redownloaded revisions to import other tags such as `nano_1_3_9`, `nano_2_0_0`,
and `nano_0_9_14`? I have yet to complete the fetch, but git-svn finished
redownloading revisions for `nano_2_1_1`, seemed to skip redownloading for
`nano_2_1_2`, and is now redownloading for `nano_2_1_3`.

Re: Why does git-svn redownload revisions?

[Brian Gernhardt]

This happens when you have a tag or branch that is a subdirectory of trunk.  It looks like tags/nano_2_1_1 is a branch off trunk/nano.  Git-svn doesn't recognize that trunk/nano@4248 is a subdirectory of trunk@4248 so it starts downloading the complete history of trunk/nano again.

On Sep 8, 2010, at 12:13 PM, Daniel Trebbien wrote:

> The issue is that the fetch runs up to revision 4250, but then mysteriously
> begins redownloading revisions 1 up to four thousand something in importing
> the `nano_2_1_1` tag:
> ...
> r4248 = 7c444dd667b9629ce92a53e8d35ad2d178e5735f (refs/remotes/trunk)
> 	M	nano/ChangeLog
> 	M	nano/configure.ac
> 	M	nano/po/cs.po

> 	M	nano/po/fi.po
> 	M	nano/po/zh_CN.po
> r4249 = 6fbad8a00fa067d1e3de913f77db08c6117843c7 (refs/remotes/trunk)
> 	M	nano/NEWS
> r4250 = 704700855e5112d75654e3e7461e896f49e10fd8 (refs/remotes/trunk)
> Found possible branch point: svn://svn.sv.gnu.org/nano/trunk/nano =>
> svn://svn.sv.gnu.org/nano/tags/nano_2_1_1, 4248
> Initializing parent: refs/remotes/tags/nano_2_1_1@4248
> 	A	mkinstalldirs
> 	A	utils.c
> 	A	nano.h
> 	A	global.c
> 	A	configure
> 	A	Makefile.in

I've encountered the same problem when importing other SVN repos.  If nano is the only subdirectory in trunk, you can fix this by changing your .git/config like this:

-	fetch = trunk:refs/remotes/trunk
+	fetch = trunk/nano:refs/remotes/trunk

This will cause issues if nano is not the only directory in trunk (it won't fetch the others) or if you have branches or tags that branch off the full trunk (you get the same problem).

Or you can downloading both trunk and nano as separate branches.  I think it fetches each revision twice, but it will prevent it from downloading the entire history for each subdirectory branch.

 	fetch = trunk:refs/remotes/trunk
+	fetch = trunk/nano:refs/remotes/nano

This will make your history look a little odd since trunk will have every commit that nano does but be a completely separate branch.  (It also seems to confuse `git-svn find-rev`.)

You can also speed imports like this by creating a local mirror with svnsync and using `git svn clone --use-svnsync-props file:///local/mirror`.  This probably isn't a good long term solution, but can be very handy when you're experimenting with getting it right.  Once you have it set up as you like delete svn-remote.svn.useSvnsyncProps, change svn-remote.svn.url to the original URL, and delete the local mirror.

I've been investigating altering the algorithm that finds branch points to understand branching off subdirectories, but haven't had the time.  It's good to know that the Minix 3 repo isn't the only one that does it.

~~ Brian

Re: Why does git-svn redownload revisions?

[Daniel Trebbien]

On Wed, Sep 8, 2010 at 10:19 AM, Brian Gernhardt
<benji@silverinsanity.com> wrote:
> This happens when you have a tag or branch that is a subdirectory of trunk.  It looks like tags/nano_2_1_1 is a branch off trunk/nano.  Git-svn doesn't recognize that trunk/nano@4248 is a subdirectory of trunk@4248 so it starts downloading the complete history of trunk/nano again.
>
> I've encountered the same problem when importing other SVN repos.  If nano is the only subdirectory in trunk, you can fix this by changing your .git/config like this:
>
> -       fetch = trunk:refs/remotes/trunk
> +       fetch = trunk/nano:refs/remotes/trunk
>
> This will cause issues if nano is not the only directory in trunk (it won't fetch the others) or if you have branches or tags that branch off the full trunk (you get the same problem).
>
> Or you can downloading both trunk and nano as separate branches.  I think it fetches each revision twice, but it will prevent it from downloading the entire history for each subdirectory branch.
>
>        fetch = trunk:refs/remotes/trunk
> +       fetch = trunk/nano:refs/remotes/nano
>
> This will make your history look a little odd since trunk will have every commit that nano does but be a completely separate branch.  (It also seems to confuse `git-svn find-rev`.)
>
> You can also speed imports like this by creating a local mirror with svnsync and using `git svn clone --use-svnsync-props file:///local/mirror`.  This probably isn't a good long term solution, but can be very handy when you're experimenting with getting it right.  Once you have it set up as you like delete svn-remote.svn.useSvnsyncProps, change svn-remote.svn.url to the original URL, and delete the local mirror.
>
> I've been investigating altering the algorithm that finds branch points to understand branching off subdirectories, but haven't had the time.  It's good to know that the Minix 3 repo isn't the only one that does it.
>
> ~~ Brian

Looking through the Nano repository, it appears that early tags were
copies of the entire trunk, but later tags (`nano_2_0_8` and onward)
were copies of just `trunk/nano`.

Is there a way that I can inform git-svn that tags `nano_2_0_8`,
`nano_2_0_9`, ... should all be based off of `remotes/trunk/nano`?

Re: Why does git-svn redownload revisions?

[Brian Gernhardt]

On Sep 8, 2010, at 6:36 PM, Daniel Trebbien wrote:

>> Or you can downloading both trunk and nano as separate branches.  I think it fetches each revision twice, but it will prevent it from downloading the entire history for each subdirectory branch.
>> 
>>        fetch = trunk:refs/remotes/trunk
>> +       fetch = trunk/nano:refs/remotes/nano
>> 
>> This will make your history look a little odd since trunk will have every commit that nano does but be a completely separate branch.  (It also seems to confuse `git-svn find-rev`.)

> Looking through the Nano repository, it appears that early tags were
> copies of the entire trunk, but later tags (`nano_2_0_8` and onward)
> were copies of just `trunk/nano`.
> 
> Is there a way that I can inform git-svn that tags `nano_2_0_8`,
> `nano_2_0_9`, ... should all be based off of `remotes/trunk/nano`?

git-svn will automatically determine that it's branched off of trunk/nano, but will only save the commits for trunk/nano if you create a branch for it.  That's I think adding a fetch line for trunk/nano as the lesser of two evils.

~~ Brian