Tracking and committing back to Subversion?

Tracking and committing back to Subversion?

[Sam Vilain]

Hi all,

Has anyone done any work on bidirectional access to SVN repositories?
ie, tracking and committing.

That would be porcelain that behaves like SVK (http://svk.elixus.org)

Ideally it would probably need to link against the Subversion RA (remote
access) library, neon.

I can see forsee two potential issues;

  1. file properties - such as mime type, ignores and custom properties.
     Linus, when I asked you about this in Dunedin, you mentioned that
     there is a place at the end of the directory entry where this could
     fit without breaking backwards compatibility.  Perhaps this could
     be an optional pointer to another directory node;

     /
      trunk/       # after all, we're tracking an SVN repo ;)
        MyModule/
          Makefile.PL
          MyModule.pm
          <prop-dir>/
            svn:ignore

     <prop-dir> is a directory object, with one file in this example;
     called svn:ignore.  The contents of this file would be something
     like the .gitignore;

          Makefile
          Makefile.old
          blib
          ...etc...

     This configures the directory "/trunk/MyModule" to have a property
     "svn:ignore", which makes SVN ignore those files.

  2. "forensic" file movement history - as opposed to the uncached,
     (and unversioned), automatic "analytical" file movement history.

     It would be easy for a tool to provide 100% interface compatibility
     with SVN client/SVK using properties, but properties that hang off
     the head rather than the file itself (so that they don't stuff up
     the ability to merge identical trees reached via independent
     paths).  SVN calls these "revision properties".  If a good
     convention is adopted for this, it could be used as a nice way to
     supplement git's automatic analysis of the revision history.

     So, then, git could support these three methods of finding
     revision history;

        1. automatic/analytical, as at present.  Cachable information,
           and such a cache would not form part of the revision history.

        2. forensic, if using porcelain that supports it; this would
           form part of the revision history, but be out of band to the
           content.

        3. (potentially) retrospective, for instance, somebody studying
           an old code base and tracking history for some reason - or
           even 'correcting' forensic revision history that is causing
           their advanced patch calculus algorithm to trip up (should
           someone be using git as a storage back-end for darcs, for
           instance)

     Ideally, it would be possible to distinguish between 2 and 3, of
     course this would require two head identifiers; the content head,
     and the revision history head.

This might seem like functionality not needed for merging kernel
trees ;) but I think it would add value to be able to make git truly
join the trump suit in the VCS deck (along with SVK, which can, for
instance, track Perforce and even git repositories).

Sam.

Re: Tracking and committing back to Subversion?

[Eric Wong]

Sam Vilain <sam@vilain.net> wrote:
> Hi all,
> 
> Has anyone done any work on bidirectional access to SVN repositories?
> ie, tracking and committing.

AFAIK, Not yet.  But it's something that's been on the back of my mind
for a while.  I've attempted similar things with Arch <-> SVN in the
past but didn't get anything extremely robust going from Arch -> SVN
although I'm pretty satisfied with my SVN -> Arch product
(svn-arch-mirror).

> That would be porcelain that behaves like SVK (http://svk.elixus.org)
> 
> Ideally it would probably need to link against the Subversion RA (remote
> access) library, neon.

I prefer to use git-svnimport for pulling from svn since it's pretty
good at what it does.  That already depends on SVN::Core and SVN::Ra.
SVK is really nice tool, just a bit slow after I've been using git for a
while.

> I can see forsee two potential issues;
> 
>   1. file properties - such as mime type, ignores and custom properties.
>      Linus, when I asked you about this in Dunedin, you mentioned that
>      there is a place at the end of the directory entry where this could
>      fit without breaking backwards compatibility.  Perhaps this could
>      be an optional pointer to another directory node;

Mostly ignore-able, imho.  svn:executable is the one that makes the most
sense (and is easiest) to support.

svn:ignore <-> .gitignore mapping are pretty close (identical, save for
recursive properties that svn gets very wrong.  I don't think svn:ignore
is affected by line order, whereas .gitignore is a plain file and line
order does matter.

svn:external is almost definitely better off ignored entirely when
interfacing with other RCSes

svn:keywords: I don't think there's a way to disable this like there
is with CVS, is there?  keywords are evil imho.

I don't use or know much about the other properties...

>   2. "forensic" file movement history - as opposed to the uncached,
>      (and unversioned), automatic "analytical" file movement history.
> 
>      It would be easy for a tool to provide 100% interface compatibility
>      with SVN client/SVK using properties, but properties that hang off
>      the head rather than the file itself (so that they don't stuff up
>      the ability to merge identical trees reached via independent
>      paths).  SVN calls these "revision properties".  If a good
>      convention is adopted for this, it could be used as a nice way to
>      supplement git's automatic analysis of the revision history.

Just parsing the output of diff-tree -C and marking them in SVN as
copies/renames should be sufficient for letting SVN do its thing.

Doing this kind of file movement history on the git side sounds like
overkill to me.  I was a _huge_ fan of logical file-identities in GNU
Arch in the past, but the complexity destroyed it from both a UI and
performance perspective.

-- 
Eric Wong

Re: Tracking and committing back to Subversion?

[Brian Smith]

[-- Attachment #1: Type: text/plain, Size: 1130 bytes --]

Well..

On Wednesday 01 February 2006 14:51, Sam Vilain wrote:
> Hi all,
>
> Has anyone done any work on bidirectional access to SVN repositories?
> ie, tracking and committing.
>

I'm working on something that does just that, I haven't gotten to the 
bidirectional part just yet. (The only reason I didn't use git-svnimport is 
that it has a /terrible/ time importing from the KDE svn repo. )

>      <prop-dir> is a directory object, with one file in this example;
>      called svn:ignore.  The contents of this file would be something
>      like the .gitignore;
>
>           Makefile
>           Makefile.old
>           blib
>           ...etc...

My take on the properties, though I like your thoughts on how to handle at 
least svn:ignore, is to basically just let svn deal with them. I will make a 
couple tools that will modify them, for interface reasons, but they'll just 
be wrappers around the svn commands themselves.

At the moment, the tools are written as bash shell scripts with an interface 
I'm trying to keep as intuitive as cogito (even though I use the core-git 
tools now.)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Tracking and committing back to Subversion?

[Seth Falcon]

On  3 Feb 2006, normalperson@yhbt.net wrote:
>> Sam Vilain <sam@vilain.net> wrote:
>> Has anyone done any work on bidirectional access to SVN
>> repositories?  ie, tracking and committing.

I think tailor claims to be able to handle this:
http://www.darcs.net/DarcsWiki/Tailor

The README in the devel version seems to indicate that two-way sync
for svn <--> git is implemented (despite the table on the main page).  

--
 + seth

Re: Tracking and committing back to Subversion?

[Sam Vilain]

Eric Wong wrote:
>>  1. file properties - such as mime type, ignores and custom properties.
>>     Linus, when I asked you about this in Dunedin, you mentioned that
>>     there is a place at the end of the directory entry where this could
>>     fit without breaking backwards compatibility.  Perhaps this could
>>     be an optional pointer to another directory node;
> Mostly ignore-able, imho.  svn:executable is the one that makes the most
> sense (and is easiest) to support.
  [...]
> svn:keywords: I don't think there's a way to disable this like there
> is with CVS, is there?  keywords are evil imho.
> I don't use or know much about the other properties...

Right, but it is also possible to hang directory or file properties,
with any name, off any file or directory.  It is often important to
at least preserve or be able to update these when dealing with some SVN
repositories, where people are using them for their own purpose.

Perhaps a "standard" mapping to dotfiles would be the best solution
then.  A .svnprops file or something like that.

>>  2. "forensic" file movement history - as opposed to the uncached,
>>     (and unversioned), automatic "analytical" file movement history.
>>
>>     It would be easy for a tool to provide 100% interface compatibility
>>     with SVN client/SVK using properties, but properties that hang off
>>     the head rather than the file itself (so that they don't stuff up
>>     the ability to merge identical trees reached via independent
>>     paths).  SVN calls these "revision properties".  If a good
>>     convention is adopted for this, it could be used as a nice way to
>>     supplement git's automatic analysis of the revision history.

> Just parsing the output of diff-tree -C and marking them in SVN as
> copies/renames should be sufficient for letting SVN do its thing.
> 
> Doing this kind of file movement history on the git side sounds like
> overkill to me.  I was a _huge_ fan of logical file-identities in GNU
> Arch in the past, but the complexity destroyed it from both a UI and
> performance perspective.

Right now, tagging gets an exemption from this requirement, as does the
extra state required by tools like StGIT.  I think this could be
generalised to support this kind of application; however we will see
whether this has any real effect or just operational side effect (eg, to
make sure renames are tracked correctly then commit before changing the
copied version, etc).

Sam.

Re: Tracking and committing back to Subversion?

[Sam Vilain]

Brian Smith wrote:
>>Has anyone done any work on bidirectional access to SVN repositories?
>>ie, tracking and committing.
> I'm working on something that does just that, I haven't gotten to the 
> bidirectional part just yet. (The only reason I didn't use git-svnimport is 
> that it has a /terrible/ time importing from the KDE svn repo. )

Indeed, mirroring SVN repositories via SVN::Mirror (which is used by
SVK) can take hours for large projects with thousands of commits.

> My take on the properties, though I like your thoughts on how to handle at 
> least svn:ignore, is to basically just let svn deal with them. I will make a 
> couple tools that will modify them, for interface reasons, but they'll just 
> be wrappers around the svn commands themselves.
 > At the moment, the tools are written as bash shell scripts with an
 > interface I'm trying to keep as intuitive as cogito (even though I use
 > the core-git tools now.)

Right; I was looking for an approach that did not require working copies
of the remote subversion repository to be kept locally.  Still, perhaps
that approach has merit, though I would probably start in Perl and use
SVK::Simple (see CPAN) to give a richer SVN mirroring API.

Are you planning on publishing these scripts?

Sam.

Re: Tracking and committing back to Subversion?

[Brian Smith]

On Monday 06 February 2006 15:23, Sam Vilain wrote:
> >>Has anyone done any work on bidirectional access to SVN repositories?
> >>ie, tracking and committing.
> >
>> [snip]
>
> Indeed, mirroring SVN repositories via SVN::Mirror (which is used by
> SVK) can take hours for large projects with thousands of commits.
>

Actually, I wasn't so much talking about the time required as the fact that
git-svnimport expects a very specific directory format, and KDE, since it was
migrated from CVS doesn't follow it, not to mention, I'm writing my tools
for the purpose of tracking single projects (i.e. Kopete which is part of KDE 
project).

>> [snip]
>
> Right; I was looking for an approach that did not require working copies
> of the remote subversion repository to be kept locally.  Still, perhaps
> that approach has merit, though I would probably start in Perl and use
> SVK::Simple (see CPAN) to give a richer SVN mirroring API.
>

While that is an admirable goal, unless you can point me to something that
will allow you to actually commit back to SVN without a working copy, it 
defeats the purpose of my tools which is basically to use to git for the 
purpose of holding intermediate development before sending it into SVN as a 
final commit. That, and being able to use git tools which speak to me on a 
level far greater than SVN tools. ;)

> Are you planning on publishing these scripts?
>

Sure, you can actually pick up the current development straight from
git://linuxfood.net/pub/git/kosek.git

Brian

Re: Tracking and committing back to Subversion?

[Junio C Hamano]

Brian Smith <linuxfood@linuxfood.net> writes:

> While that is an admirable goal, unless you can point me to something that
> will allow you to actually commit back to SVN without a working copy, it 
> defeats the purpose of my tools which is basically to use to git for the 
> purpose of holding intermediate development before sending it into SVN as a 
> final commit. 

Wouldn't svk (or svl or whatever it is called these days) be a
better match for that kind of "keep my work while disconnected
from master svn repository" purpose?

Re: Tracking and committing back to Subversion?

[Brian Smith]

On Thursday 09 February 2006 16:54, Junio C Hamano wrote:
> Brian Smith <linuxfood@linuxfood.net> writes:
> > While that is an admirable goal, unless you can point me to something
> > that will allow you to actually commit back to SVN without a working
> > copy, it defeats the purpose of my tools which is basically to use to git
> > for the purpose of holding intermediate development before sending it
> > into SVN as a final commit.
>
> Wouldn't svk (or svl or whatever it is called these days) be a
> better match for that kind of "keep my work while disconnected
> from master svn repository" purpose?

It would, if it wasn't god awful slow. And besides, AFAIK, it doesn't allow 
you to do anything other than keep offline work in a separate SVN repo hosted 
on your machine, which means I can't use the git tools.

Re: Tracking and committing back to Subversion?

[Sam Vilain]

Brian Smith wrote:
>>Right; I was looking for an approach that did not require working copies
>>of the remote subversion repository to be kept locally.  Still, perhaps
>>that approach has merit, though I would probably start in Perl and use
>>SVK::Simple (see CPAN) to give a richer SVN mirroring API.
> While that is an admirable goal, unless you can point me to something that
> will allow you to actually commit back to SVN without a working copy,

Such as SVK

   http://svk.elixus.org/
   (svn url: http://svn.openfoundry.org/svk/)

> it 
> defeats the purpose of my tools which is basically to use to git for the 
> purpose of holding intermediate development before sending it into SVN as a 
> final commit. That, and being able to use git tools which speak to me on a 
> level far greater than SVN tools. ;)

Your solution may ultimately be the most pragmatic approach.  Issues 
surrounding the C (SWIG) bindings between Perl and the Subversion RA
(remote access) API were "the hard part" in making SVK work, according
to the lead author.

That being said, I am unaware of any drastic outstanding issues, so
perhaps there are useful components there.

> Sure, you can actually pick up the current development straight from
> git://linuxfood.net/pub/git/kosek.git

I will take a look.

Sam.

Re: Tracking and committing back to Subversion?

[Brian Smith]

On Thursday 09 February 2006 20:27, Sam Vilain wrote:
>>> [snip]
> >
> > While that is an admirable goal, unless you can point me to something
> > that will allow you to actually commit back to SVN without a working
> > copy,
>
> Such as SVK
>
>    http://svk.elixus.org/
>    (svn url: http://svn.openfoundry.org/svk/)

Well, after reading through a number of pages I discovered something:

"The repository created is actually a Subversion repository. In other words, 
you could use any subversion commands with that repository." [*1*]

So, while it is a technicality, there is an SVN Repo, which is what I'm trying 
to avoid (the point being to have a git repo after all, for the purpose of 
using git commands), perhaps it would be worth looking into to see if I could 
somehow utilize the svk svn repo (since there are space savings) for the SVN 
working copy. [*2*]

>
> > [snip]
>
> Your solution may ultimately be the most pragmatic approach.  Issues
> surrounding the C (SWIG) bindings between Perl and the Subversion RA
> (remote access) API were "the hard part" in making SVK work, according
> to the lead author.
>
> That being said, I am unaware of any drastic outstanding issues, so
> perhaps there are useful components there.

There are useful things in there. I have considered rewriting in perl so that 
I don't have to use my kludge of a rewrite of Getopt::Long (yes, I did. No, 
not everything is supported.) but have decided against it mostly because it 
would take weeks, and Kosek (my program) isn't even complete yet.

(Incidentally, the allure of the SVN perl bindings had me for awhile, but 
after trying to decipher the horrid PODs for it decided that Bash was just 
easier. )

>
> > Sure, you can actually pick up the current development straight from
> > git://linuxfood.net/pub/git/kosek.git
>
> I will take a look.
>
Awesome let me know what you think. Please try to ignore the massive lack of 
documentation, up until someone in the kopete irc channel showed interest I 
had been writing it for my own purposes. Though, since I posted the link here 
I did go ahead and fixup the readme with a little howto. If you like it, feel 
free to email me with questions and suggestions.

[*1*] http://svk.elixus.org/?SingleUserSVK (under Mapping Repositories)

[*2*] 
While writing this, I had an absolutely insane idea, which I may or may not 
persue.  The idea being that it may be possible to remove 99% of the actual 
SVN working copy (except for perhaps the base directory at the root of the 
project), and just use symlinks for the rest of the hierarchy. This would 
require careful management though, since every .svn directory contains 
specific and distinct information.

Re: Tracking and committing back to Subversion?

[Sam Vilain]

On Thu, 2006-02-09 at 23:01 -0800, Brian Smith wrote:
> > > While that is an admirable goal, unless you can point me to something
> > > that will allow you to actually commit back to SVN without a working
> > > copy,
> > Such as SVK
> >    http://svk.elixus.org/
> >    (svn url: http://svn.openfoundry.org/svk/)
> Well, after reading through a number of pages I discovered something:
> "The repository created is actually a Subversion repository. In other words, 
> you could use any subversion commands with that repository." [*1*]
> So, while it is a technicality, there is an SVN Repo, which is what
> I'm trying to avoid[...

Yes, currently there is - but let me show you what's under the hood
briefly... as you notice later;

> (Incidentally, the allure of the SVN perl bindings had me for awhile, but 
> after trying to decipher the horrid PODs for it decided that Bash was just 
> easier. )

Yes, there are SWIG bindings for various languages.  In debian/ubuntu the
Perl bindings will be built to the package libsvn-core-perl.

Take a look at this module:

  http://search.cpan.org/dist/SVN-Simple/lib/SVN/Simple/Edit.pm

That uses the above module and allows you to construct a Subversion
commit and try to apply it.

The documentation's a bit weak, but AFAIK the basic functionality is
there and a lot better an abstraction than the raw bindings themselves.

SVN::Mirror:

  http://search.cpan.org/dist/SVN-Mirror/lib/SVN/Mirror.pm

can go the other way.

> (the point being to have a git repo after all, for
> the purpose of using git commands), perhaps it would be worth looking
> into to see if I could somehow utilize the svk svn repo (since there
> are space savings) for the SVN working copy. [*2*]

That's right - in principle, SVK could be plumbed to have a git
repository in its ~/.svk/local - then you could have one git repo, and a
head for each mirrored project.

But don't concern yourself with my plans - it seems our tastes in
porcelain differ ;)

Sam.