Google Summer of Code 2011

Google Summer of Code 2011

[Shawn Pearce]

Anyone want to mentor this year?

-- 
Shawn.

Re: Google Summer of Code 2011

[Jeff King]

On Thu, Mar 03, 2011 at 10:08:25AM -0800, Shawn O. Pearce wrote:

> Anyone want to mentor this year?

I'd be happy to. I can also act as org admin this year if you want. I
bowed out last year due to impending baby, but have no such excuse this
year. :)

Should we also start a call for project suggestions? I haven't been
paying attention to the GSoC timeline.

-Peff

Re: Google Summer of Code 2011

[Shawn Pearce]

On Thu, Mar 3, 2011 at 10:59, Jeff King <peff@peff.net> wrote:
> On Thu, Mar 03, 2011 at 10:08:25AM -0800, Shawn O. Pearce wrote:
>
>> Anyone want to mentor this year?
>
> I'd be happy to. I can also act as org admin this year if you want. I
> bowed out last year due to impending baby, but have no such excuse this
> year. :)

I would appreciate that, I'm too busy this year. :-(

> Should we also start a call for project suggestions? I haven't been
> paying attention to the GSoC timeline.

Org application deadline is March 11: 23:00 UTC

-- 
Shawn.

Re: Google Summer of Code 2011

[Jeff King]

On Thu, Mar 03, 2011 at 11:04:51AM -0800, Shawn O. Pearce wrote:

> > I'd be happy to. I can also act as org admin this year if you want. I
> > bowed out last year due to impending baby, but have no such excuse this
> > year. :)
> 
> I would appreciate that, I'm too busy this year. :-(

OK, there is now:

  https://git.wiki.kernel.org/index.php/SoC2011Application

which is mostly just an adapted version of last year's application.

I'll give people a week or so to make changes, and then do the
application with Google probably next Wednesday or Thursday.

> > Should we also start a call for project suggestions? I haven't been
> > paying attention to the GSoC timeline.
> 
> Org application deadline is March 11: 23:00 UTC

Looks like accepted projects are announced on the 18th, and then we
should be talking to students about it. So we should probably have a
relatively complete list of ideas by the 18th. I set up a bare-bones
idea page here if people want to start adding ideas:

  https://git.wiki.kernel.org/index.php/SoC2011Ideas

-Peff

Re: Google Summer of Code 2011

[Ramkumar Ramachandra]

Hi,

Shawn Pearce writes:
> Anyone want to mentor this year?

Although I'm probably not experienced enough to independently mentor a
project, I'd be more than happy to co-mentor, or step in if a mentor
is suddenly unavailable. I should have a lot of time to give in
summer.

-- Ram

Re: Google Summer of Code 2011

[Jakub Narebski]

Jeff King <peff@peff.net> writes:

> On Thu, Mar 03, 2011 at 11:04:51AM -0800, Shawn O. Pearce wrote:
> 
> > > I'd be happy to. I can also act as org admin this year if you want. I
> > > bowed out last year due to impending baby, but have no such excuse this
> > > year. :)
> > 
> > I would appreciate that, I'm too busy this year. :-(
> 
> OK, there is now:
> 
>   https://git.wiki.kernel.org/index.php/SoC2011Application

Quote:

 "In 2010, we had three projects: native svn support, libgit2, and a
  line level history browser.

  All three projects were successful. [...]"

What about 'integrated web client for git', aka. "Splitting gitweb and
developing write functionalities" project wit Pavan Kumar Sankara
(pkumar), which failed midterm evaluations?

See https://git.wiki.kernel.org/index.php/SoC2010Projects#Splitting_gitweb_and_developing_write_functionalities_.28Integrated_web_client_for_git.29

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Google Summer of Code 2011

[Jonathan Nieder]

Hi,

Ramkumar Ramachandra wrote:

> Although I'm probably not experienced enough to independently mentor a
> project, I'd be more than happy to co-mentor, or step in if a mentor
> is suddenly unavailable. I should have a lot of time to give in
> summer.

I'd also be willing to co-mentor.  Time might be a constraint, but I can
find time.

Jonathan

Re: Google Summer of Code 2011

[Jens Lehmann]

Am 03.03.2011 19:08, schrieb Shawn Pearce:
> Anyone want to mentor this year?

I can offer to either be a co-mentor (being a fill in for a mentor
who is not available the whole time, like I did last year for Thomas)
or to mentor a student full time when (s)he is working on submodules.

Re: Google Summer of Code 2011

[Christian Couder]

Hi,

On Thursday 03 March 2011 19:08:25 Shawn Pearce wrote:
> Anyone want to mentor this year?

I can co-mentor this year too.

Thanks,
Christian.

Re: Google Summer of Code 2011

[Sam Vilain]

Hi Shawn,

On 04/03/11 07:08, Shawn Pearce wrote:
> Anyone want to mentor this year?

This year I'm likely to be far too busy to mentor anyone or review
applications; however I expect to find time to deliver on my earlier
promise to revisit GitTorrent with a series of articles and how-to's on
the comp.git section of my blog.  I've made at least administrative
progress towards this goal; by setting up ikiwiki and arranging a
discrete comp.git section, so that readers can follow without getting my
other articles.

The introductory article is at
http://vilain.net/comp/git/gittorrent/past_synthesis.html and I hope to
be able to keep producing one article a week.  I would love following,
input, criticism etc from anyone who has been involved in the GitTorrent
project or its many spin-offs over the years.

Cheers and hope GSoC 2011 works out well for git!
Sam

Re: Google Summer of Code 2011

[Sverre Rabbelier]

Heya,


On Thu, Mar 3, 2011 at 23:08, Jonathan Nieder <jrnieder@gmail.com> wrote:
> On Thu, Mar 3, 2011 at 22:04, Ramkumar Ramachandra <artagnon@gmail.com> wrote:
>> Although I'm probably not experienced enough to independently mentor a
>> project, I'd be more than happy to co-mentor, or step in if a mentor
>> is suddenly unavailable. I should have a lot of time to give in
>> summer.
>
> I'd also be willing to co-mentor.  Time might be a constraint, but I can
> find time.

Perhaps we can mentor a student to work on git-remote-svn with the
three of us? :)

-- 
Cheers,

Sverre Rabbelier

Re: Google Summer of Code 2011

[Heiko Voigt]

Hi,

On Thu, Mar 03, 2011 at 10:08:25AM -0800, Shawn Pearce wrote:
> Anyone want to mentor this year?

I can offer to mentor a project either about submodule improvements or
on gui stuff (git-gui or gitk). Since I am still not a native speaker of
the full git code base co-mentoring would be an option for other
projects.

Cheers Heiko

Re: Google Summer of Code 2011

[Fredrik Gustafsson]

On Mon, Mar 07, 2011 at 08:40:47PM +0100, Heiko Voigt wrote:
> I can offer to mentor a project either about submodule improvements or
> on gui stuff (git-gui or gitk). Since I am still not a native speaker of
> the full git code base co-mentoring would be an option for other
> projects.

That's sounds very interesting.

I'm using git with submodules a lot (daily in 5-10 different smaller
projects with 1-5 developers in each) and suffers from lacking submodule
support, specially in the gui-tools. 

Being a student this is a part of git I would be very interested in
helping to improve.
-- 
Med vänliga hälsningar
Fredrik Gustafsson

tel: +46 (0)733-608274
e-post: iveqy@iveqy.com

Re: Google Summer of Code 2011

[Ramkumar Ramachandra]

Hi,

Sverre Rabbelier writes:
> On Thu, Mar 3, 2011 at 23:08, Jonathan Nieder <jrnieder@gmail.com> wrote:
> > On Thu, Mar 3, 2011 at 22:04, Ramkumar Ramachandra <artagnon@gmail.com> wrote:
> >> Although I'm probably not experienced enough to independently mentor a
> >> project, I'd be more than happy to co-mentor, or step in if a mentor
> >> is suddenly unavailable. I should have a lot of time to give in
> >> summer.
> >
> > I'd also be willing to co-mentor.  Time might be a constraint, but I can
> > find time.
> 
> Perhaps we can mentor a student to work on git-remote-svn with the
> three of us? :)

Sure, but I was hoping for an exciting new project idea- what do you
think?

-- Ram

Re: Google Summer of Code 2011

[Sverre Rabbelier]

Heya,

On Tue, Mar 8, 2011 at 13:33, Ramkumar Ramachandra <artagnon@gmail.com> wrote:
> Sure, but I was hoping for an exciting new project idea- what do you
> think?

I don't know, having a working bidi git-remote-svn is pretty exciting
in itself :)

-- 
Cheers,

Sverre Rabbelier

Re: Google Summer of Code 2011

[Thomas Rast]

Hi Shawn

Shawn Pearce wrote:
> Anyone want to mentor this year?

I can mentor (or co-mentor) again.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: Google Summer of Code 2011

[Jeff King]

On Thu, Mar 03, 2011 at 01:25:16PM -0800, Jakub Narebski wrote:

> > OK, there is now:
> > 
> >   https://git.wiki.kernel.org/index.php/SoC2011Application
> 
> Quote:
> 
>  "In 2010, we had three projects: native svn support, libgit2, and a
>   line level history browser.
> 
>   All three projects were successful. [...]"
> 
> What about 'integrated web client for git', aka. "Splitting gitweb and
> developing write functionalities" project wit Pavan Kumar Sankara
> (pkumar), which failed midterm evaluations?
> 
> See https://git.wiki.kernel.org/index.php/SoC2010Projects#Splitting_gitweb_and_developing_write_functionalities_.28Integrated_web_client_for_git.29

Thanks, I had totally forgotten about that project and didn't find it
mentioned on the GSoC site (I guess because it failed). I couldn't find
any discussion of its failure, though. Is there more information
somewhere?

-Peff

Re: Google Summer of Code 2011

[Jeff King]

On Thu, Mar 03, 2011 at 03:33:23PM -0500, Jeff King wrote:

> > I would appreciate that, I'm too busy this year. :-(
> 
> OK, there is now:
> 
>   https://git.wiki.kernel.org/index.php/SoC2011Application
> 
> which is mostly just an adapted version of last year's application.

We need to list a backup admin. Can I volunteer you for that (or are
there are volunteers)?

-Peff

Re: Google Summer of Code 2011

[Shawn Pearce]

On Wed, Mar 9, 2011 at 08:39, Jeff King <peff@peff.net> wrote:
> On Thu, Mar 03, 2011 at 03:33:23PM -0500, Jeff King wrote:
>
>> > I would appreciate that, I'm too busy this year. :-(
>>
>> OK, there is now:
>>
>>   https://git.wiki.kernel.org/index.php/SoC2011Application
>>
>> which is mostly just an adapted version of last year's application.
>
> We need to list a backup admin. Can I volunteer you for that (or are
> there are volunteers)?

Sure, I can be the backup admin.

-- 
Shawn.

Re: Google Summer of Code 2011

[Jeff King]

On Thu, Mar 03, 2011 at 03:33:23PM -0500, Jeff King wrote:

> OK, there is now:
> 
>   https://git.wiki.kernel.org/index.php/SoC2011Application
> 
> which is mostly just an adapted version of last year's application.
> 
> I'll give people a week or so to make changes, and then do the
> application with Google probably next Wednesday or Thursday.

Our application is officially in. However, we can still edit it until
the deadline on Friday. If you have changes, feel free to make them on
the wiki page but make sure to let me know, as I have to migrate them to
the official application. I'll also probably do a once-over on Friday to
check for any updates. If only the wiki was kept in git. :)

Also, the application links to our ideas page. Please add ideas! It will
give the GSoC people a sense of what we are thinking of for projects,
and students will probably start looking at them after the list of
accepted organizations is published (which is next Friday, the 18th).

-Peff

Re: Google Summer of Code 2011

[Shawn Pearce]

On Wed, Mar 9, 2011 at 09:49, Jeff King <peff@peff.net> wrote:
> On Thu, Mar 03, 2011 at 03:33:23PM -0500, Jeff King wrote:
>
>> OK, there is now:
>>
>>   https://git.wiki.kernel.org/index.php/SoC2011Application
>>
>> which is mostly just an adapted version of last year's application.
>>
>> I'll give people a week or so to make changes, and then do the
>> application with Google probably next Wednesday or Thursday.
>
> Our application is officially in. However, we can still edit it until
> the deadline on Friday. If you have changes, feel free to make them on
> the wiki page but make sure to let me know, as I have to migrate them to
> the official application. I'll also probably do a once-over on Friday to
> check for any updates. If only the wiki was kept in git. :)
>
> Also, the application links to our ideas page. Please add ideas! It will
> give the GSoC people a sense of what we are thinking of for projects,
> and students will probably start looking at them after the list of
> accepted organizations is published (which is next Friday, the 18th).

The ideas page is very important. Git needs a good set of ideas on its
ideas page in order to be accepted. Google has stated this many times
in the past, its a key part of the decision making process (not the
only part, but an important part nonetheless).

We should have a good ideas page by the application deadline, so that
when Google goes to review applications, they can at least make a fair
assessment of our ideas list.

-- 
Shawn.

Re: Re: Google Summer of Code 2011

[Heiko Voigt]

Hi,

On Mon, Mar 07, 2011 at 09:50:23PM +0100, Fredrik Gustafsson wrote:
> On Mon, Mar 07, 2011 at 08:40:47PM +0100, Heiko Voigt wrote:
> > I can offer to mentor a project either about submodule improvements or
> > on gui stuff (git-gui or gitk). Since I am still not a native speaker of
> > the full git code base co-mentoring would be an option for other
> > projects.
> 
> That's sounds very interesting.
> 
> I'm using git with submodules a lot (daily in 5-10 different smaller
> projects with 1-5 developers in each) and suffers from lacking submodule
> support, specially in the gui-tools. 
> 
> Being a student this is a part of git I would be very interested in
> helping to improve.

Sounds promising. When I was talking about gui or submodules I did not
think about gui *and* submodules. Do you have some concrete areas you
would like to improve in mind?

Cheers Heiko

Summer of Code project ideas due this Friday

[Jeff King]

On Wed, Mar 09, 2011 at 09:52:17AM -0800, Shawn O. Pearce wrote:

> The ideas page is very important. Git needs a good set of ideas on its
> ideas page in order to be accepted. Google has stated this many times
> in the past, its a key part of the decision making process (not the
> only part, but an important part nonetheless).
> 
> We should have a good ideas page by the application deadline, so that
> when Google goes to review applications, they can at least make a fair
> assessment of our ideas list.

Then let's repeat it with a more eye-catching subject, and cc all of the
people who have said they would mentor so far.

For those just joining us:

The Google Summer of Code application deadline is this Friday (March 11)
at 23:00 UTC. They will be looking at our ideas page at:

    https://git.wiki.kernel.org/index.php/SoC2011Ideas

If you have any ideas, please add them to the page! Whether you are
available to mentor the project, or simply think it would make a good
project and want to inspire others to mentor it, it's appropriate to go
there.

-Peff

Re: Re: Google Summer of Code 2011

[Fredrik Gustafsson]

On Wed, Mar 09, 2011 at 10:52:56PM +0100, Heiko Voigt wrote:
> Hi,
> 
> On Mon, Mar 07, 2011 at 09:50:23PM +0100, Fredrik Gustafsson wrote:
> > On Mon, Mar 07, 2011 at 08:40:47PM +0100, Heiko Voigt wrote:
> > > I can offer to mentor a project either about submodule improvements or
> > > on gui stuff (git-gui or gitk). Since I am still not a native speaker of
> > > the full git code base co-mentoring would be an option for other
> > > projects.
> > 
> > That's sounds very interesting.
> > 
> > I'm using git with submodules a lot (daily in 5-10 different smaller
> > projects with 1-5 developers in each) and suffers from lacking submodule
> > support, specially in the gui-tools. 
> > 
> > Being a student this is a part of git I would be very interested in
> > helping to improve.
> 
> Sounds promising. When I was talking about gui or submodules I did not
> think about gui *and* submodules. Do you have some concrete areas you
> would like to improve in mind?

Many of the ideas I had was already mentioned in 
https://github.com/jlehmann/git-submod-enhancements/wiki/

The ones I feel is most important for me is:
* gitk: Add popup menu for submodules to see the detailed history of
changes
* Check before a push in the superproject that all submodules HEADs are
pushed
* Showing that a submodule has a HEAD not on any branch in “git status”
* Move the submodules git directories into the superproject’s .git so that
submodules can be created and deleted

With this said, I'm not familiar with the git codebase yet and does not
have any concrete solutions. For example, I don't understand why a
submodule doesn't get a default branch.

-- 
Med vänliga hälsningar
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

Re: Summer of Code project ideas due this Friday

[Jonathan Nieder]

Hi,

Jeff King wrote:

> If you have any ideas, please add them to the page!

Thanks for a pointer.  Some ideas still at the "throw them against the
wall and see if they stick" stage: please feel free to add to the page
if you think you can find subsets with the right scope.  Later today I
can look more carefully.

1. Cross-compilable msysgit: ideally, allowing someone to run
   "make msysgit-installer" on Linux from an msysgit-source.git repo
   to get an installer.  Nice subsets to start with might be msys.dll
   (I think there has been some work on that already) and the basic
   msys utilities.

2. Merge libgit2 into git.git: polish the existing code in libgit2 so
   that standard git can use it.  Presumably this would happen by
   porting one function at a time, not by making git link to libgit2
   immediately.  People familiar with libgit2 might be able to pick
   out particularly interesting subsets to start with (diff
   generation?).

3. Remote helpers: bidi git remote-svn (or one-way remote-hg, or
   remote-cvs, or ...).

4. filter-branch killer: using fast-import's new features to implement
   common filter-branch operations (--subdirectory-filter,
   --prune-empty, obliterating certain files) faster.

5. rev-list: Coping with timeskew.

6. shallow clone: Push support.

7. fixing workdirs (and alternates), along the lines explained in the
   thread <http://thread.gmane.org/gmane.comp.version-control.git/150559>.
   I would be very much interested in this.

7. packfilev4.

Re: Summer of Code project ideas due this Friday

[Nguyen Thai Ngoc Duy]

On Thu, Mar 10, 2011 at 4:58 AM, Jeff King <peff@peff.net> wrote:
> They will be looking at our ideas page at:
>
>    https://git.wiki.kernel.org/index.php/SoC2011Ideas
>
> If you have any ideas, please add them to the page!

Jeff, how is the "support multiple transport (torrent, bundle, http..)
cloning" going? The one that you were working on from the "initial
clone from bundle" thread? Is it ok to make it a SoC project
(presumably with at least bundle clone support)?
-- 
Duy

Re: Summer of Code project ideas due this Friday

[Jeff King]

On Wed, Mar 09, 2011 at 06:10:17PM -0600, Jonathan Nieder wrote:

> 2. Merge libgit2 into git.git: polish the existing code in libgit2 so
>    that standard git can use it.  Presumably this would happen by
>    porting one function at a time, not by making git link to libgit2
>    immediately.  People familiar with libgit2 might be able to pick
>    out particularly interesting subsets to start with (diff
>    generation?).

That's definitely bigger than a SoC project at this point. However, one
smaller project might be to start implementing a subset of git commands
(probably plumbing) on top of libgit2, and then see how well the result
passes git.git tests.

> 4. filter-branch killer: using fast-import's new features to implement
>    common filter-branch operations (--subdirectory-filter,
>    --prune-empty, obliterating certain files) faster.

Yeah, I like that. Or maybe somebody can pick up git-sequencer, which in
theory could be used to filter-branch, too (or maybe sequencer can go on
top of fast-import? :) ).

> 5. rev-list: Coping with timeskew.

I don't think this is big enough for a SoC project. It's on my todo list
for the near-term, though.

> 7. packfilev4.

I suspect that is too complex for a SoC project. But you never know.

-Peff

Re: Summer of Code project ideas due this Friday

[Jeff King]

On Thu, Mar 10, 2011 at 07:19:36AM +0700, Nguyen Thai Ngoc Duy wrote:

> On Thu, Mar 10, 2011 at 4:58 AM, Jeff King <peff@peff.net> wrote:
> > They will be looking at our ideas page at:
> >
> >    https://git.wiki.kernel.org/index.php/SoC2011Ideas
> >
> > If you have any ideas, please add them to the page!
> 
> Jeff, how is the "support multiple transport (torrent, bundle, http..)
> cloning" going? The one that you were working on from the "initial
> clone from bundle" thread? Is it ok to make it a SoC project
> (presumably with at least bundle clone support)?

I pushed it back in favor of other topics, but I will get to it
eventually. If you want to make a SoC proposal, go for it. Though I
think it might not be big enough by itself, and might need to be folded
into a larger project.

-Peff

Re: Summer of Code project ideas due this Friday

[Thomas Rast]

Jonathan Nieder wrote:
> Thanks for a pointer.  Some ideas still at the "throw them against the
> wall and see if they stick" stage: please feel free to add to the page
> if you think you can find subsets with the right scope.

Some stuff off the top of my head, please apply a similar filter:

* Word-based merge helper

  The existing merge algorithms are all tailored to line-based formats
  such as code.  Writing, e.g., LaTeX or even asciidoc requires
  sticking to a strict word-wrapped format.  Worse even, re-wrapping
  leads to headaches if people work on the same areas a lot, much like
  the effects of code reindents.

  Something similar was already proposed in 2010, IIRC by Dscho:

    https://git.wiki.kernel.org/index.php/SoC2010Ideas#Merge_helper_for_LaTeX_files

  One possible angle of attack given --word-diff=porcelain would be:

  1. Fix --word-diff to properly represent both sides of the diff at
     least optionally.  (It was observed in a list post some months
     ago that it doesn't even represent *one* side faithfully.)

  2. Use --word-diff=porcelain as input to some to-be-written merge
     algorithm.



* Clean up add -p

  git-add--interactive.perl became a bit of a mess.  Partly due to my
  own efforts with {checkout,stash,...} it has bolted-on interfaces to
  other commands.  There are some UI issues that simply fall out of
  its design, e.g., you cannot go back from one file to another,
  Ctrl-C stops applying to the current file but does not discard
  earlier files, etc.  And that's not saying anything about 'add -i'
  which I don't really know.

  This would probably not be a very fun project, but it could add a
  little edge of usability to the tool and it's probably one of the
  few pure-Perl ideas we can offer.



> 4. filter-branch killer: using fast-import's new features to implement
>    common filter-branch operations (--subdirectory-filter,
>    --prune-empty, obliterating certain files) faster.

I would phrase this as follows:

* fast-import-filter

  Implement a utility that can execute certain transformations to
  fast-import streams, such as:

  - delete or rename entire files/directories
  - split out subtrees
  - ...

  Ideally, this should be a thin command line interface around a set
  of primitives (such as a Perl package) that allow a competent
  scripter to go beyond the CLI.  (There have been threads on this.)

  Later on, make an interface that automatically does

    git fast-export | fast-import-filter | git fast-import

  with suitable options.  Nice because it should be largely orthogonal
  to everything except the fast-import stream format.

In addition, Jeff wrote:
> Yeah, I like that. Or maybe somebody can pick up git-sequencer, which in
> theory could be used to filter-branch, too (or maybe sequencer can go on
> top of fast-import? :) ).

I'd be interested to hear how that goes, because I think the tools are
fundamentally different.  The rebase and thus sequencer family is
delta-based, and the fast-import and filter-branch families are
tree-based.  Feel free to prove me wrong of course.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: Summer of Code project ideas due this Friday

[Shawn Pearce]

On Thu, Mar 10, 2011 at 08:30, Jeff King <peff@peff.net> wrote:
>
>> 7. packfilev4.
>
> I suspect that is too complex for a SoC project. But you never know.

It is. Nico is also probably working on it.

My big concern with pack v4 right now is the on-disk format of
commits/trees is different from the current wire protocol. If a pack
v2 only client connected to a server using pack v4, the server would
have to spend a lot of time to inflate/deflate about 60% of the
objects in the repository (commits and trees). That is a lot of CPU
time. So even if a SoC-er finished the work, it probably wouldn't be
merged because of this penalty.

-- 
Shawn.

Re: Summer of Code project ideas due this Friday

[Jakub Narebski]

Jonathan Nieder <jrnieder@gmail.com> writes:

> 4. filter-branch killer: using fast-import's new features to implement
>    common filter-branch operations (--subdirectory-filter,
>    --prune-empty, obliterating certain files) faster.

How it would be different from existing reposurgeon tool by ESR
(cross-VC thanks to using fast-import format), or git_fast_filter by
Elijah Newren (more of a library than a ready tool)?

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Summer of Code project ideas due this Friday

[Santi Béjar]

On Thu, Mar 10, 2011 at 6:15 PM, Thomas Rast <trast@student.ethz.ch> wrote:
> Jonathan Nieder wrote:
>> Thanks for a pointer.  Some ideas still at the "throw them against the
>> wall and see if they stick" stage: please feel free to add to the page
>> if you think you can find subsets with the right scope.
>
> Some stuff off the top of my head, please apply a similar filter:
>
> * Word-based merge helper
>
>  The existing merge algorithms are all tailored to line-based formats
>  such as code.  Writing, e.g., LaTeX or even asciidoc requires
>  sticking to a strict word-wrapped format.  Worse even, re-wrapping
>  leads to headaches if people work on the same areas a lot, much like
>  the effects of code reindents.

Not a complete/right solution (and not even extensively tested), but I
did a "trivial" merge program for LaTeX files that:

1.- Rewraps all files as the first one, using dwdiff
2.- Perform a normal line based merge

[wlmerge]
#!/bin/bash
tmp1=$(mktemp -t wlmerge.XXXXXXX)
tmp2=$(mktemp -t wlmerge.XXXXXXX)
trap "rm -f $tmp1 $tmp2" 0 1 2 3 15
[ ! -e "$1" ] && exit 1
file1=$1 && shift
[ ! -e "$1" ] && exit 1
orig=$1 && shift
[ ! -e "$1" ] && exit 1
file2=$1 && shift
dwdiff -2 -w "" -x "" "$orig" "$file1" > $tmp1
dwdiff -2 -w "" -x "" "$file2" "$file1" > $tmp2
merge "$file1" $tmp1 $tmp2

I even have a "wldiff", but nowadays I use --color-words exclusively!

HTH,
Santi

Re: Summer of Code project ideas due this Friday

[Jeff King]

On Thu, Mar 10, 2011 at 06:15:23PM +0100, Thomas Rast wrote:

> * Clean up add -p
> 
>   git-add--interactive.perl became a bit of a mess.  Partly due to my
>   own efforts with {checkout,stash,...} it has bolted-on interfaces to
>   other commands.  There are some UI issues that simply fall out of
>   its design, e.g., you cannot go back from one file to another,
>   Ctrl-C stops applying to the current file but does not discard
>   earlier files, etc.  And that's not saying anything about 'add -i'
>   which I don't really know.
> 
>   This would probably not be a very fun project, but it could add a
>   little edge of usability to the tool and it's probably one of the
>   few pure-Perl ideas we can offer.

One more wishlist item for this. I use "add -p" for almost all of my
adds these days, because I like the final review check. So after a
conflicted merge, I find myself doing "add -p" to stage my resolution.
The current behavior is that it shows the --cc diff and exits. It would
be cool to handle staging the resolution, which would involve converting
the combined diff into something that can be applied.

> I'd be interested to hear how that goes, because I think the tools are
> fundamentally different.  The rebase and thus sequencer family is
> delta-based, and the fast-import and filter-branch families are
> tree-based.  Feel free to prove me wrong of course.

Hmm, yeah, that is a fundamental difference. It might be possible to
overcome it, but I admit I haven't really given it any thought yet at
this point.

-Peff

Re: Summer of Code project ideas due this Friday

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> One more wishlist item for this. I use "add -p" for almost all of my
> adds these days, because I like the final review check. So after a
> conflicted merge, I find myself doing "add -p" to stage my resolution.
> The current behavior is that it shows the --cc diff and exits. It would
> be cool to handle staging the resolution, which would involve converting
> the combined diff into something that can be applied.

I think the end-result would be a nice feature. I suspect that it would
not involve conversion from --cc, but more like using the difference
between the HEAD and the working tree, generated as if there is no
multi-stage index.

Re: Summer of Code project ideas due this Friday

[Jeff King]

On Thu, Mar 10, 2011 at 11:21:47AM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > One more wishlist item for this. I use "add -p" for almost all of my
> > adds these days, because I like the final review check. So after a
> > conflicted merge, I find myself doing "add -p" to stage my resolution.
> > The current behavior is that it shows the --cc diff and exits. It would
> > be cool to handle staging the resolution, which would involve converting
> > the combined diff into something that can be applied.
> 
> I think the end-result would be a nice feature. I suspect that it would
> not involve conversion from --cc, but more like using the difference
> between the HEAD and the working tree, generated as if there is no
> multi-stage index.

The trouble is that I would like to see the combined diff, then say "OK"
and have it apply the result to the index. But because we work on a
per-hunk basis, you need to match the combined diff hunks to the regular
diff hunks, taking into account that hunks could be split.  Which maybe
is straightforward, but I haven't convinced myself yet that there are no
corner cases where they don't line up.

I guess one could argue that I should be using mergetool, and making a
mergetool helper that is more like "add -p" (show hunks, say "choose
left, choose right, choose none"). But I generally fix up the conflict
using an editor and just want "add -p" as a final check.

I'll probably look into it more eventually, but it's not a high priority
at this point.

-Peff

Re: Summer of Code project ideas due this Friday

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

>> I think the end-result would be a nice feature. I suspect that it would
>> not involve conversion from --cc, but more like using the difference
>> between the HEAD and the working tree, generated as if there is no
>> multi-stage index.
>
> The trouble is that I would like to see the combined diff, then say "OK"
> and have it apply the result to the index. But because we work on a
> per-hunk basis, you need to match the combined diff hunks to the regular
> diff hunks, taking into account that hunks could be split.  Which maybe
> is straightforward, but I haven't convinced myself yet that there are no
> corner cases where they don't line up.

Forgetting for now the implementation, I _think_ what you would want is
for "git add -p" to notice that you are resolving conflicts, and do not
bother you about cleanly merged parts (I take it is a given that you would
always want to add them to the index without even inspecting when running
"add -p"), and make the per-hunk selection loop ask only about the parts
that originally had conflicts.

But it is rather hard to arrange.  Neither "--cc" nor "-c" during the
merge is about showing conflicts but is about showing the result with
respect to the two originals.  If you resolved conflicts in your editor
already, there are no lines with "<<</>>>" markers that are different from
either original to cause the "conflicted parts" to appear in their output.
If your conflict resolution ended up in taking what one side did, "--cc"
will hide it as a non-event.  So at least you would be using "-c" to
implement this.

Also, you have to remember that "add -p" is about adding the state you
deem Ok incrementally to the index, that once you add a path to the index
the higher stages for the path will collapse to stage #0, and that "-c"
and "--cc" make their comparison based on what you have in higher stages.

I would imagine that a workable implementation might look like this:

 0. The solution introduces a new index extension, "PRSF" (partial
    resolution so far)".  This is a mapping from pathname to a blob object
    name.

 1. "git add -p" notices that you are in the middle of conflict
    resolution, by noticing that you have higher stages to the path.

 2. Look up the path from the PRSF extension.  If there is no entry for
    the path, recreate the content-level 3-way merge using the content of
    three stages (i.e. the state immediately after the mergy operation
    that caused the conflicts before you touched the corresponding file in
    your working tree), and register this image (with the full glory of
    "<<</>>>" conflict markers) to the extension.  If there already is an
    entry in the extension for the path, skip this step.

 3. The patch the user will see in the interactive hunk selection is the
    difference between the working tree and the PRSF image, not your
    regular index (nor the HEAD version). This allows you to see how you
    resolved the conflict incrementally so far.

 4. Choosing a hunk to "apply" would not affect the index entry for the
    path, as it would collapse its higher stages.  It instead updates the
    blob registered in the PRSF extension using the hunk you are applying.

If you are running "git add -p" incrementally (i.e. edit a little, review
with diff, then "add -p" the part you are sure about, and repeat the whole
thing), the next invocation of "git add -p" would notice that you still
have unresolved conflicts in PRSF.  It will skip the step 2 above and the
step 3 will let you review the remaining difference between the PRSF image
(which you updated in step 4 during the last round) and what you have in
the working tree.  The step 4 will update the PRSF image for the next
invocation of "git add -p".  And you continue until you are done.

At the end (we would probably need a good way to detect the user might
want to declare "end" automatically for a good user experience), use the
PRSF image to collapse the higher stages for the path down to stage 0.

Re: Summer of Code project ideas due this Friday

[Alexander Miseler]

Comments on the "Better big-file support" section:


"While git can handle arbitrary-sized binary content [...]"

This is very much not true. Git tries at many places to load the complete file into memory and usually fails with "out of memory" if it can't. With the 32bit msysGit client this places the upper file size limit, from purely empirical observation, at 600-700 MByte. When a file is to large git fails late, adding and committing works (as long as there are no filters or other complications), but you can forget about pushing, rebasing or otherwise manipulating that commit. Even worse yet, commits consisting of smaller files but with a combined size over the limit will also cause out-of-memories.

Thus a main focus should be the memory problem, e.g. by using stream-like file handling everywhere, since not working at all is orders of magnitude worse than working slowly :)



"In some cases, this may be as simple as having a "large file" codepath that avoids pulling whole files into memory (e.g., during "git add")."

Ironically git add is one of the few things that work with large files, as mentioned above. Presumably the stream-oriented zlib enforced/encouraged a steam-like handling here :)
Slow as hell though and of course it is usually not sensible to compress a 1.5 GByte file.




I'm very willing to work on this topic. Though I'm not a student and as a git code newbie I also don't have the skills for mentoring yet.

Re: Summer of Code project ideas due this Friday

[Jeff King]

On Thu, Mar 10, 2011 at 12:54:29PM -0800, Junio C Hamano wrote:

> Forgetting for now the implementation, I _think_ what you would want is
> for "git add -p" to notice that you are resolving conflicts, and do not
> bother you about cleanly merged parts (I take it is a given that you would
> always want to add them to the index without even inspecting when running
> "add -p"), and make the per-hunk selection loop ask only about the parts
> that originally had conflicts.

Yes, I don't want to see cleanly merged parts. And "--cc" already does
what I want by not showing them. But of course they still need to be
added to the index.

So my thinking was more along the lines of:

  1. Get "git diff HEAD file" and store its hunks.

  2. Get "git diff --cc file" and stores its hunks.

  3. For each hunk in (1), if it does not have an analagous hunk in (2),
     mark it for staging without asking the user.

  4. For the remaining hunks in (1), show the user the analagous --cc
     hunk from (2), and mark the hunk from (1) for staging if requested.

  5. Create the final patch from the marked hunks, apply it to
     HEAD:file, and put that in the index.

There are two issues (and I think you know this, but it took me
reasoning out why this wouldn't work to quite understand what you were
saying in your email, so I'll elaborate here for the benefit of other
readers):

  a. Step (3) glosses over the definition of "analagous hunk". In simple
     cases the hunk headers match up (i.e., they start at the same
     offsets). But the --cc diff is actually a diff against the merge
     base, and the diff in step (1) is against HEAD. So actually we
     would want step (1) and (5) to deal not with the file in HEAD, but
     the file in the merge-base.

     Or we can use "-c" in the first place, which gives us interesting
     and uninteresting parts, and then suppress the uninteresting ones.

  b. You can't stage part of a resolution. This is the "adding a path
     collapses stages to #0" that you mentioned. So either you need an
     index extension, or you need to make it all-or-nothing.

> [discussion of how this could be done right]

That description makes sense to me, but is way overkill for my workflow.
I think at its simplest, what I would like is to be shown the entire
--cc diff, have it say "do you want this and all of the cleanly merged
bits added?" and then either stage the whole thing or not.

Which really I could do with:

  for i in `git diff-files --name-only --diff-filter=U`; do
    git diff --cc $i
    echo 'OK?'
    read r
    test "$r" = y && git add $i
  done

It would just be a little nicer to have it integrated into the "git add
-p" loop.

Even though it is obviously a less featureful solution than what you
proposed, I am tempted to implement it anyway because the current
behavior is so bad (it shows the diff but doesn't consider it a
selectable hunk at all, so it just skips it, exiting immediately if
there are no other files).

And then if somebody wants to spend time allowing partial resolution
selection, it would be a nice improvement on top of that.

-Peff

Re: Summer of Code project ideas due this Friday

[Alexander Miseler]

On 10.03.2011 18:31, Shawn Pearce wrote:
> On Thu, Mar 10, 2011 at 08:30, Jeff King <peff@peff.net> wrote:
>>
>>> 7. packfilev4.
>>
>> I suspect that is too complex for a SoC project. But you never know.
> 
> It is. Nico is also probably working on it.
> 
> My big concern with pack v4 right now is the on-disk format of
> commits/trees is different from the current wire protocol. If a pack
> v2 only client connected to a server using pack v4, the server would
> have to spend a lot of time to inflate/deflate about 60% of the
> objects in the repository (commits and trees). That is a lot of CPU
> time. So even if a SoC-er finished the work, it probably wouldn't be
> merged because of this penalty.

I'm very interested in this. Can anyone please point me towards further information on the packfile format and proposed/in-progress changes?

Re: Summer of Code project ideas due this Friday

[Jeff King]

On Thu, Mar 10, 2011 at 10:40:01PM +0100, Alexander Miseler wrote:

> "While git can handle arbitrary-sized binary content [...]"
> 
> This is very much not true. Git tries at many places to load the
> complete file into memory and usually fails with "out of memory" if it
> can't. With the 32bit msysGit client this places the upper file size
> limit, from purely empirical observation, at 600-700 MByte.

I think we are picking nits here. What I meant was two things:

  1. The fundamental design of git does not prevent storing
     arbitrary-sized binary data.

  2. How big a piece of data the current implementation can handle
     sanely is up to how much hardware you throw at it. On my 64-bit
     machine with 8G of RAM running Linux, I can easily work with 2
     gigabyte files. Some operations are slow, of course, but it works.

     I'm willing to accept that 32-bit msysgit has more trouble with
     a case like that.

But I think we are probably in agreement with what needs to be done to
make things better. Specifically, I am thinking of:

  1. Streaming blobs wherever possible (e.g., add, filters, textconv).

  2. Converting the diff code not to have in-memory files is probably
     going to be quite difficult. But most of these files don't have
     interesting diffs _anyway_. They're usually binary, and we don't
     generate binary diffs by default. So what we need to focus on is
     avoiding loading them when we can. Things like:

       a. Using caching textconv, and when we do run the textconv
          filter, streaming the blob to the filter.

       b. Avoid loading the whole file to check whether it is binary. We
          can already avoid this by marking it binary with
          gitattributes, but there is no reason git can't just load the
          first 4K or so to check for binary-ness, and get this
          optimization automatically. We can also consider caching
          binary-ness for large files so we don't have to look at them
          at all after the first time.

       c. Handle rename detection better. It may be a matter of saying
          "this file is too big for detection". But we may also be able
          to stream it through the spantree-hashing, and then possibly
          also cache the resultant data.

  3. The above deal with memory problems. There is also a storage
     problem. If I have a 100G repo, right now I use at least 100G in
     the .git directory and 100G in the working tree. That's a problem
     for repos of that size. If I have a storage server on the LAN and
     want to accept the latency hit, it would be nice to keep the
     commits local and the giant blobs on the server. Especially coupled
     with the optimizations in (2), we can possibly avoid even having to
     touch those blobs at all in many cases, so the latency wouldn't be
     a big deal.

  4. A related storage problem is that we put big files in packs, and
     the I/O on rewriting the pack becomes a problem. They would do
     better loose or in their own packs.

And I'm sure there are more variations on those things. Part of the
project would be identifying the problem areas.

> Even worse yet, commits consisting of smaller files but with a
> combined size over the limit will also cause out-of-memories.

That generally should work OK. The diff and packing code tries to keep
memory usage reasonable, which generally equates to two times the
largest file. If you have a test case that shows problems, there may
very well be a bug.

> Thus a main focus should be the memory problem, e.g. by using
> stream-like file handling everywhere, since not working at all is
> orders of magnitude worse than working slowly :)

Agreed. I think they are sort of the same problem. Whether it works
slowly or not at all is simply a matter of how much memory you have. ;)

> Ironically git add is one of the few things that work with large
> files, as mentioned above. Presumably the stream-oriented zlib
> enforced/encouraged a steam-like handling here :) Slow as hell though
> and of course it is usually not sensible to compress a 1.5 GByte file.

I just tried "git add" on a 2G file of random bytes. It took about a
minute or so to calculate the sha1 and compress it, but the memory usage
did jump to 2G. So we could obviously do better on the memory, and there
is almost certainly no point in zlib compressing something that big. In
my case, it was obviously just random junk. But most files of that size
are already going to have some kind of lossy compression, so we are just
wasting CPU. You can always set core.compression, but I really just want
it off for certain files.

> I'm very willing to work on this topic. Though I'm not a student and
> as a git code newbie I also don't have the skills for mentoring yet.

It's on my agenda, too. We'll see if a student steps up for the GSoC
project. But don't let that stop you if you want to take a look at it;
I'm sure there is plenty of work to go around. :)

-Peff

Re: Re: Re: Google Summer of Code 2011

[Heiko Voigt]

Hi,

On Thu, Mar 10, 2011 at 12:16:04AM +0100, Fredrik Gustafsson wrote:
> On Wed, Mar 09, 2011 at 10:52:56PM +0100, Heiko Voigt wrote:
> > Sounds promising. When I was talking about gui or submodules I did not
> > think about gui *and* submodules. Do you have some concrete areas you
> > would like to improve in mind?
> 
> Many of the ideas I had was already mentioned in 
> https://github.com/jlehmann/git-submod-enhancements/wiki/
> 
> The ones I feel is most important for me is:
> * gitk: Add popup menu for submodules to see the detailed history of
> changes
> * Check before a push in the superproject that all submodules HEADs are
> pushed
> * Showing that a submodule has a HEAD not on any branch in “git status”
> * Move the submodules git directories into the superproject’s .git so that
> submodules can be created and deleted

Sounds like a good roadmap. From my viewpoint this includes enough work
for a summer. There are also enough small steps so even if only part of
this would be finished it would contain enough improvement for git.
Since this is the first time I am offering to mentor I am not that
familar how much work can be done during summer. What do others think?

(added Jens to the CC)

> With this said, I'm not familiar with the git codebase yet and does not
> have any concrete solutions. For example, I don't understand why a
> submodule doesn't get a default branch.

The familiarity with the code should not be a problem since the idea of
the Summer of Code is to allow more people to get involved into a
project.

Regarding the default branch: The short answer is that git does not
record any branchname but a specific revision using its SHA1. This is on
purpose to make sure that you get exactly the code you had when you
created the commit. But a default branch has been requested many times
to make working easier. AFAIR it the consensus was that creating a
branch recursively by default would not be a good idea. But doing that on
explicit user request could be implemented as an option to ease users
workflow. So something like this could be discussed.

Cheers Heiko

Re: Summer of Code project ideas due this Friday

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Yes, I don't want to see cleanly merged parts. And "--cc" already does
> what I want by not showing them.

Are you sure about that?

What "--cc" would show largely depends on what you have in your working
tree. If your two branches fixed the same bug with very different
approaches, you may resolve the conflict favouring what one side did while
discarding everything the other side did. The file in your work tree might
be the same as "git checkout --ours $that_path" after a mergy operation.
"diff --cc" won't show anything to you in such a case.

> So my thinking was more along the lines of:
>
>   1. Get "git diff HEAD file" and store its hunks.
>
>   2. Get "git diff --cc file" and stores its hunks.
>
>   3. For each hunk in (1), if it does not have an analagous hunk in (2),
>      mark it for staging without asking the user.
>
>   4. For the remaining hunks in (1), show the user the analagous --cc
>      hunk from (2), and mark the hunk from (1) for staging if requested.
>
>   5. Create the final patch from the marked hunks, apply it to
>      HEAD:file, and put that in the index.

Actually, the largest problem I see in the above approach is _when_ you
perform the first two steps, not the details of the later steps.

The user has complete freedom outside our control between the time the
mergy operation leaves conflicts in the index and the time "add -p" is
invoked (except obviously the user cannot do any "add" to register the
resolution for the whole file to the index, as there won't be any per-hunk
resolution recording left to do by "add -p").

And as I repeatedly said, grabbing "--cc file" must be done before the
user starts mucking with the file in the working tree for the approach to
be any useful.

>> [discussion of how this could be done right]
>
> That description makes sense to me, but is way overkill for my workflow.

Obviously I don't think so.  I'd rather consider what it does an absolute
minimum (not the part that it stores in an index extension, but the part
that the reference it uses to compare against the working tree file is a
original, mechanical merge result).  By the way, I just relized that an
index extension similar to PRSF might be a way to properly give index log
similar to the reflog you wanted to have the other day---every time you
"git add", you record the previous state of the index entry there, so that
it can be replayed in reverse.

> Which really I could do with:
>
>   for i in `git diff-files --name-only --diff-filter=U`; do
>     git diff --cc $i
>     echo 'OK?'

As you already know, I disagree the usefulness of this approach (see the
"Are you sure about that?" discussion), hence I doubt the usefulness of
"have it integrated into the "git add -p" loop".

Re: Summer of Code project ideas due this Friday

[Jeff King]

On Thu, Mar 10, 2011 at 02:58:07PM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > Yes, I don't want to see cleanly merged parts. And "--cc" already does
> > what I want by not showing them.
> 
> Are you sure about that?

Well, no. But my experience has been that its output is useful.

> What "--cc" would show largely depends on what you have in your working
> tree. If your two branches fixed the same bug with very different
> approaches, you may resolve the conflict favouring what one side did while
> discarding everything the other side did. The file in your work tree might
> be the same as "git checkout --ours $that_path" after a mergy operation.
> "diff --cc" won't show anything to you in such a case.

Actually, I think diff --cc is doing what I want there. If I take one
side's content completely, then it is not interesting to me anymore.
What I am really concerned about is doing a tricky content-level merge
in my editor and then screwing up the result. Or _trying_ to take one
side of the merge, and then screwing it up. So the "diff --cc" output
after I have mucked is useful for both those cases.

> And as I repeatedly said, grabbing "--cc file" must be done before the
> user starts mucking with the file in the working tree for the approach to
> be any useful.

I'm not sure I agree. The output after I have mucked is useful to me,
and that is what this use-case is based on. So I think we are talking
about related but slightly different use cases.

> > Which really I could do with:
> >
> >   for i in `git diff-files --name-only --diff-filter=U`; do
> >     git diff --cc $i
> >     echo 'OK?'
> 
> As you already know, I disagree the usefulness of this approach (see the
> "Are you sure about that?" discussion), hence I doubt the usefulness of
> "have it integrated into the "git add -p" loop".

OK, let me put it this way. I am not volunteering to work on the
approach you outlined. If you are, great. But if not, then what should
be done? The current behavior to show the diff and then exit is quite
confusing. At the very least, we should say something to the user about
what happened (or even suppress the diff and just say "these paths are
unmerged, and we can't handle them in add -p").

-Peff

Re: Summer of Code project ideas due this Friday

[Alexander Miseler]

On 09.03.2011 22:58, Jeff King wrote:
> If you have any ideas, please add them to the page! Whether you are
> available to mentor the project, or simply think it would make a good
> project and want to inspire others to mentor it, it's appropriate to go
> there.

Added two new ideas:
Port Git to Android
Resumable clone

I'm new to the Git code and therefore don't necessarily have an idea what I'm talking about. Please feel free to change, 
correct, mutilate and massacre this ideas. This are just two things that I would like to see, but don't want to work on 
myself since I have BIGGER fishes to fry ^_^

Re: Summer of Code project ideas due this Friday

[Ævar Arnfjörð Bjarmason]

On Wed, Mar 9, 2011 at 22:58, Jeff King <peff@peff.net> wrote:
> If you have any ideas, please add them to the page!

Here's a simple one:

Currently we have a global UI color setting, but pretty much only
git-status/log/etc use it.

Change git-bisect, git-am etc. to also support color output. It would
be a easy but big UI win.

Re: Summer of Code project ideas due this Friday

[Ilari Liusvaara]

On Fri, Mar 11, 2011 at 01:18:45PM +0100, Alexander Miseler wrote:
> On 09.03.2011 22:58, Jeff King wrote:
> >If you have any ideas, please add them to the page! Whether you are
> >available to mentor the project, or simply think it would make a good
> >project and want to inspire others to mentor it, it's appropriate to go
> >there.
> 

> Resumable clone

This is very very hard. Not so much to implement but to design the way it
can be done without assuming things (like object sort orders) that aren't
stable.

-Ilari

Re: Summer of Code project ideas due this Friday

[Thomas Rast]

Jakub Narebski wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:
> 
> > 4. filter-branch killer: using fast-import's new features to implement
> >    common filter-branch operations (--subdirectory-filter,
> >    --prune-empty, obliterating certain files) faster.
> 
> How it would be different from existing reposurgeon tool by ESR
> (cross-VC thanks to using fast-import format), or git_fast_filter by
> Elijah Newren (more of a library than a ready tool)?

Git integration?  I haven't looked at these tools for long enough to
say how much they fit the ticket and how fast they are, so for me it's
hard to say how much work it would be.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: Summer of Code project ideas due this Friday

[Thomas Rast]

Jeff King wrote:
> On Thu, Mar 10, 2011 at 06:15:23PM +0100, Thomas Rast wrote:
> 
> > * Clean up add -p
> 
> One more wishlist item for this. I use "add -p" for almost all of my
> adds these days, because I like the final review check. So after a
> conflicted merge, I find myself doing "add -p" to stage my resolution.
> The current behavior is that it shows the --cc diff and exits. It would
> be cool to handle staging the resolution, which would involve converting
> the combined diff into something that can be applied.

Good idea, thanks.

I put this (with your idea) and the word-merge thing on the wiki.
I'll let Jonathan decide what to do about the fast-import stream
filtering.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: Summer of Code project ideas due this Friday

[Nguyen Thai Ngoc Duy]

On Fri, Mar 11, 2011 at 7:52 PM, Ilari Liusvaara
<ilari.liusvaara@elisanet.fi> wrote:
> On Fri, Mar 11, 2011 at 01:18:45PM +0100, Alexander Miseler wrote:
>> On 09.03.2011 22:58, Jeff King wrote:
>> >If you have any ideas, please add them to the page! Whether you are
>> >available to mentor the project, or simply think it would make a good
>> >project and want to inspire others to mentor it, it's appropriate to go
>> >there.
>>
>
>> Resumable clone
>
> This is very very hard. Not so much to implement but to design the way it
> can be done without assuming things (like object sort orders) that aren't
> stable.

Yes it's hard. I have some experimental thing that nearly works [1],
although whether it is an acceptable approach is to be seen. If
anyone's interested, I'll post it some time

A simpler way to restartable clone is to facilitate bundles (Nicolas'
idea). Some glue is needed to teach git-fetch/git-daemon to use the
bundles, and git-push to automatically create bundles periodically (or
a new command that can be run from cron). I think this way fit in GSoC
scope better.

[1] The idea of my work above was mentioned elsewhere, history is cut
down by path. Each file/dir's history a very long chain of deltas. We
can stream deltas (in parallel if needed) over the wire, resuming
where the chain stops last time.

There are many problems. One is that a deep chain can make git run out
of stack, so chains have to be broken down before storing (not done).
Another one is that not many deltas can be reused, so it will consume
more power than normal clone. But once you clone this way, the cloned
repo have lots of delta suitable for another clone (but probably not
for anything else).
-- 
Duy

Re: Summer of Code project ideas due this Friday

[Alexander Miseler]

On 11.03.2011 14:48, Nguyen Thai Ngoc Duy wrote:
> On Fri, Mar 11, 2011 at 7:52 PM, Ilari Liusvaara
> <ilari.liusvaara@elisanet.fi>  wrote:
>> On Fri, Mar 11, 2011 at 01:18:45PM +0100, Alexander Miseler wrote:
>>> Resumable clone
>>
>> This is very very hard. Not so much to implement but to design the way it
>> can be done without assuming things (like object sort orders) that aren't
>> stable.
>
> Yes it's hard. I have some experimental thing that nearly works [1],
> although whether it is an acceptable approach is to be seen. If
> anyone's interested, I'll post it some time
>
> A simpler way to restartable clone is to facilitate bundles (Nicolas'
> idea). Some glue is needed to teach git-fetch/git-daemon to use the
> bundles, and git-push to automatically create bundles periodically (or
> a new command that can be run from cron). I think this way fit in GSoC
> scope better.
>
> [1] The idea of my work above was mentioned elsewhere, history is cut
> down by path. Each file/dir's history a very long chain of deltas. We
> can stream deltas (in parallel if needed) over the wire, resuming
> where the chain stops last time.
>
> There are many problems. One is that a deep chain can make git run out
> of stack, so chains have to be broken down before storing (not done).
> Another one is that not many deltas can be reused, so it will consume
> more power than normal clone. But once you clone this way, the cloned
> repo have lots of delta suitable for another clone (but probably not
> for anything else).


This may all be aiming to short. IMHO the best solution would be some generic way for the client to specify exactly what 
it wants to get and to get just that. This would lay the groundwork for:
- lazy clones
- sparse clones
- resumable cloning
- resumable fetching
and probably quite a few other nifty tricks.

I guess that would be far beyond the scope of a SoC project though.

Re: Summer of Code project ideas due this Friday

[Alexander Miseler]

On 10.03.2011 23:18, Jeff King wrote:
>    1. The fundamental design of git does not prevent storing
>       arbitrary-sized binary data.

agreed


> But I think we are probably in agreement with what needs to be done to
> make things better. Specifically, I am thinking of:

agreed


>> Even worse yet, commits consisting of smaller files but with a
>> combined size over the limit will also cause out-of-memories.
>
> That generally should work OK. The diff and packing code tries to keep
> memory usage reasonable, which generally equates to two times the
> largest file. If you have a test case that shows problems, there may
> very well be a bug.

I will look into it.


> But don't let that stop you if you want to take a look at it;
> I'm sure there is plenty of work to go around. :)

Certainly ^_^

Re: Summer of Code project ideas due this Friday

[code.sculptor]

Great idea!
Sent from my BlackBerry device on the Rogers Wireless Network

-----Original Message-----
From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Sender: git-owner@vger.kernel.org
Date: Fri, 11 Mar 2011 13:43:39 
To: Jeff King<peff@peff.net>
Cc: Shawn Pearce<spearce@spearce.org>; Ramkumar Ramachandra<artagnon@gmail.com>; Jonathan Nieder<jrnieder@gmail.com>; Jens Lehmann<Jens.Lehmann@web.de>; Christian Couder<chriscool@tuxfamily.org>; Thomas Rast<trast@student.ethz.ch>; git<git@vger.kernel.org>
Subject: Re: Summer of Code project ideas due this Friday

On Wed, Mar 9, 2011 at 22:58, Jeff King <peff@peff.net> wrote:
> If you have any ideas, please add them to the page!

Here's a simple one:

Currently we have a global UI color setting, but pretty much only
git-status/log/etc use it.

Change git-bisect, git-am etc. to also support color output. It would
be a easy but big UI win.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Summer of Code project ideas due this Friday

[Nguyen Thai Ngoc Duy]

On Fri, Mar 11, 2011 at 9:10 PM, Alexander Miseler <alexander@miseler.de> wrote:
> This may all be aiming to short. IMHO the best solution would be some
> generic way for the client to specify exactly what it wants to get and to
> get just that.

But how do you define "what it wants to get"? Pros and cons are
different and depend on that defintion.

Git's way of saying now is "I have these commits, give me these
branches (cutting at certain depth)". A near future extension would be
"I have these commits _restricted to these paths only_, give me...".
If it's broken half way, start again.

The bundle way is basically Git's way except that the remote side says
"I've got this bundle, fetch it here, then you can ask us again for
the rest in a normal way".

mirrorsync (aka gittorrent) says "I have this commit, give me objects
so that I can have this commit", which leaves the (probably) huge
initial commit out of view.

My way is "give me a chain of deltas starting from this SHA-1". A big
blob can be considered as a history of smaller pieces.

The last way of saying is just "give me this object", which would be a
big waste of bandwidth.

Your move :)
-- 
Duy

Re: Summer of Code project ideas due this Friday

[Sam Vilain]

On 12/03/11 03:27, Nguyen Thai Ngoc Duy wrote:
> On Fri, Mar 11, 2011 at 9:10 PM, Alexander Miseler <alexander@miseler.de> wrote:
>> This may all be aiming to short. IMHO the best solution would be some
>> generic way for the client to specify exactly what it wants to get and to
>> get just that.
> But how do you define "what it wants to get"? Pros and cons are
> different and depend on that defintion.
>
> Git's way of saying now is "I have these commits, give me these
> branches (cutting at certain depth)". A near future extension would be
> "I have these commits _restricted to these paths only_, give me...".
> If it's broken half way, start again.
>
> The bundle way is basically Git's way except that the remote side says
> "I've got this bundle, fetch it here, then you can ask us again for
> the rest in a normal way".
>
> mirrorsync (aka gittorrent) says "I have this commit, give me objects
> so that I can have this commit", which leaves the (probably) huge
> initial commit out of view.
>
> My way is "give me a chain of deltas starting from this SHA-1". A big
> blob can be considered as a history of smaller pieces.
>
> The last way of saying is just "give me this object", which would be a
> big waste of bandwidth.
>
> Your move :)

Just FWIW, I recently produced some diagrams and demonstrated the
resumable clone algorithm; speed is currently very limited by the
implementation at 15 minutes to slice up git.git; but I'm playing with
libgit2 to see if I can make a more optimal version.

See http://vilain.net/comp/git/gittorrent/commit_reel.html

Sam

History surgery with fast-import (Re: Summer of Code project ideas due this Friday)

[Jonathan Nieder]

Jakub Narebski wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:

>> 4. filter-branch killer: using fast-import's new features to implement
>>    common filter-branch operations (--subdirectory-filter,
>>    --prune-empty, obliterating certain files) faster.
>
> How it would be different from existing reposurgeon tool by ESR
> (cross-VC thanks to using fast-import format), or git_fast_filter by
> Elijah Newren (more of a library than a ready tool)?

Good question.

. reposurgeon loads the whole history into a convenient format,
  manipulates it, then dumps it as a separate step.  So it is even more
  flexible than filter-branch, but with a large history and especially
  on a small machine it can be slower.

. git_fast_filter is what Thomas mentioned --- a filter to go between
  fast-export and fast-import.

Both involve unpacking all trees.  Which is not a huge expense, mind
you, but there is room to go faster.

What I was suggesting is instead a tool that relies on fast-import to
do the heavy lifting.  So, for example, when you ask to delete
path/to/remove.txt from all commits, it would write:

 commit refs/heads/master
 mark :172
 author A U Thor <author@example.com> ...
 committer ...
 data ...
 from ...
 merge ...
 M 040000 [old tree name for that commit] ""
 D path/to/remove.txt

 commit refs/heads/mater
 mark :173
 ...

The idea is inspired by
http://thread.gmane.org/gmane.comp.version-control.git/158375
I am interested in it because I imagine something like this could be
useful for splitting out branches from a naive import of an svn repo
(and similar surgeries).

Re: Summer of Code project ideas due this Friday

[Alexander Miseler]

>>> Even worse yet, commits consisting of smaller files but with a
>>> combined size over the limit will also cause out-of-memories.
>>
>> That generally should work OK. The diff and packing code tries to keep
>> memory usage reasonable, which generally equates to two times the
>> largest file. If you have a test case that shows problems, there may
>> very well be a bug.
> 
> I will look into it.

I failed to create an artificial test case for this (thus probably misunderstanding the exact circumstances leading to this). I will investigate it when I get a real life test case again.

Re: Summer of Code project ideas due this Friday

[Alexander Miseler]

> Your move :)

Heh, I'm still pretty hampered by having no clue ^_^


> My way is "give me a chain of deltas starting from this SHA-1". A big
> blob can be considered as a history of smaller pieces.

I originally missed the fact that it is completely irrelevant where in the chain the reference point is and that you can go easily from any arbitrary point in the chain to any other arbitrary point in the same chain. This makes it a lot more flexible than I thought. But to support more than just resumable cloning You would still need a more powerful way to negotiate "I have this; I want that" between client and server.

My point was mostly that lazy clone, sparse clone and resumable clone are closely related and should ideally be solved in one go. You currently seem to aim at only one of them, which is a shame.
For lazy clone the client would start with all blobs referenced by HEAD (and possibly other branches) and then it would walk backwards through the delta chains if/as far/whenever it wants. Though it would also need to detect when it needs to start reading new delta chains as it walks backwards through history. Sparse clone would use the same mechanism and just restrict it to certain paths.

Re: Summer of Code project ideas due this Friday

[Ramkumar Ramachandra]

Hi Jonathan,

Jonathan Nieder writes:
> 3. Remote helpers: bidi git remote-svn (or one-way remote-hg, or
>    remote-cvs, or ...).

I wrote a small introduction and created an entry for git-remote-svn
in the wiki [1]. Two minor concerns:
1. The student will have to work very closely with us, and pick up a
   lot of scattered WIP patches. I hope he doesn't get confused
   between what's merged and what's in-progress.
2. The project isn't as de-coupled as we'd ideally like. The student
   has to learn about the new fast-import features, and the
   information contained in an SVN replay dumpstream before diving in.

Other than these, I think most of the pending work is in the mapper.

> 4. filter-branch killer: using fast-import's new features to implement
>    common filter-branch operations (--subdirectory-filter,
>    --prune-empty, obliterating certain files) faster.

This is an interesting project that I'd also thought about: it might
get tangled up along with the Sequencer project though. Should we put
up this project on the wiki nevertheless?

p.s- Sorry about the late reply; I wasn't in station.

[1] SoC2011Ideas#Welcome and SoC2011Ideas#Remote_helper_for_Subversion

-- Ram

Re: Summer of Code project ideas

[Jakub Narebski]

Jeff King <peff@peff.net> writes:

> https://git.wiki.kernel.org/index.php/SoC2011Ideas
> 
> If you have any ideas, please add them to the page!

A few project ideas I am not sure if they are feasible for GSoC:

* merging-in support for caching in gitweb, and benchmarking/profiling
  gitweb in high load situation

  J.H., you would probably want gitweb [output] caching to be merged-in
  sooner that the end of Google Summer of Code 2011, isn't it?

* embedding graphical diff and graphical merge tool in git-gui, e.g. as
  "git gui diff".  I think that we can use xxdiff; the license is 
  compatibile.

  Pat and Shawn, is it something worth doing?  Does it look like a good
  project for GSoC2011, or is it too small of a project for this?  Would
  we be able to find mentor for this idea?

* splitting gitk, common library (Tcl/Tk bindings) for gitk and git-gui

  Pat and Paul, do you think it is right scope, or is it too large project
  to put as an GSoC idea?

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Re: Summer of Code project ideas

[Heiko Voigt]

Hi,

On Thu, Mar 17, 2011 at 04:40:44PM -0700, Jakub Narebski wrote:
> * embedding graphical diff and graphical merge tool in git-gui, e.g. as
>   "git gui diff".  I think that we can use xxdiff; the license is 
>   compatibile.
> 
>   Pat and Shawn, is it something worth doing?  Does it look like a good
>   project for GSoC2011, or is it too small of a project for this?  Would
>   we be able to find mentor for this idea?
> 
> * splitting gitk, common library (Tcl/Tk bindings) for gitk and git-gui
> 
>   Pat and Paul, do you think it is right scope, or is it too large project
>   to put as an GSoC idea?

I would be interested in mentoring those projects. I do not know whether
we can reach consensus how to split up things. The last time we
discussed a possible approach[1] to share code between gitk and git gui
we did not reach one.

Cheers Heiko

[1] http://thread.gmane.org/gmane.comp.version-control.git/166663/focus=166695

Re: Summer of Code project ideas

[J.H.]

On 03/17/2011 04:40 PM, Jakub Narebski wrote:
> Jeff King <peff@peff.net> writes:
> 
>> https://git.wiki.kernel.org/index.php/SoC2011Ideas
>>
>> If you have any ideas, please add them to the page!
> 
> A few project ideas I am not sure if they are feasible for GSoC:
> 
> * merging-in support for caching in gitweb, and benchmarking/profiling
>   gitweb in high load situation
> 
>   J.H., you would probably want gitweb [output] caching to be merged-in
>   sooner that the end of Google Summer of Code 2011, isn't it?

I think the work needed to get gitweb-caching actually merged upstream
is going to be a slight bit messier than I'm willing to put a student
through, and couple that with the fact I'm going to be herding 2 or 3
students for kernel.org as already, I won't have enough time to try and
mentor a student through that quagmire.

Should a student be interested in the graphing stuff for gitweb, I'm
happy to make commentary and review things as they come, but that's at
best a co-mentor from me.

Re: Summer of Code project ideas

[Pat Thoyts]

Jakub Narebski <jnareb@gmail.com> writes:

>Jeff King <peff@peff.net> writes:
>
>> https://git.wiki.kernel.org/index.php/SoC2011Ideas
>> 
>> If you have any ideas, please add them to the page!
>
>A few project ideas I am not sure if they are feasible for GSoC:
>
>* merging-in support for caching in gitweb, and benchmarking/profiling
>  gitweb in high load situation
>
>  J.H., you would probably want gitweb [output] caching to be merged-in
>  sooner that the end of Google Summer of Code 2011, isn't it?
>
>* embedding graphical diff and graphical merge tool in git-gui, e.g. as
>  "git gui diff".  I think that we can use xxdiff; the license is 
>  compatibile.
>
>  Pat and Shawn, is it something worth doing?  Does it look like a good
>  project for GSoC2011, or is it too small of a project for this?  Would
>  we be able to find mentor for this idea?

There is also tkdiff for stealing from. I'm not sure about the worth -
there are lots of free merge tools around. But if someone wants to do
that then fine.

>* splitting gitk, common library (Tcl/Tk bindings) for gitk and git-gui
>
>  Pat and Paul, do you think it is right scope, or is it too large project
>  to put as an GSoC idea?

It shouldn't be too large. You are likely looking at doing Git.pm as a
Tcl package (without looking in much detail). Testing that gitk and
git-gui didn't get broken will probably be tedious. So adding some test
suite to each could help. The Tcl test package is quite capable of being
used to test Tk apps provided some tests get written.

-- 
Pat Thoyts                            http://www.patthoyts.tk/
PGP fingerprint 2C 6E 98 07 2C 59 C8 97  10 CE 11 E6 04 E0 B9 DD

Re: Summer of Code project ideas

[Jakub Narebski]

On Fri, 25 Mar 2011, Pat Thoyts wrote:
> Jakub Narebski <jnareb@gmail.com> writes:

> > A few project ideas I am not sure if they are feasible for GSoC:

> > * embedding graphical diff and graphical merge tool in git-gui, e.g. as
> >   "git gui diff".  I think that we can use xxdiff; the license is 
> >   compatibile.
> >
> >   Pat and Shawn, is it something worth doing?  Does it look like a good
> >   project for GSoC2011, or is it too small of a project for this?  Would
> >   we be able to find mentor for this idea?
> 
> There is also tkdiff for stealing from. I'm not sure about the worth -
> there are lots of free merge tools around. But if someone wants to do
> that then fine.

I meant TkDiff, not xxdiff here; I'm sorry for the mistake.

TkDiff has the advantage that is already written in Tcl/Tk, so we can
borrow code (like e.g. "refining" diff), not only algorithms.

> > * splitting gitk, common library (Tcl/Tk bindings) for gitk and git-gui
> >
> >   Pat and Paul, do you think it is right scope, or is it too large project
> >   to put as an GSoC idea?
> 
> It shouldn't be too large. You are likely looking at doing Git.pm as a
> Tcl package (without looking in much detail). Testing that gitk and
> git-gui didn't get broken will probably be tedious. So adding some test
> suite to each could help. The Tcl test package is quite capable of being
> used to test Tk apps provided some tests get written.

I thought that would be mainly re-using library part of git-gui in gitk;
tests are decidely a good idea, though I don't know if there are ready
tools to do tests for programs written in Tcl/Tk , and for testing UI.

-- 
Jakub Narebski
Poland

Re: Google Summer of Code 2011

[Luiz Capitulino]

On Sun, 30 Jan 2011 16:06:20 +0100
Alexander Graf <agraf@suse.de> wrote:

> 
> On 28.01.2011, at 21:10, Luiz Capitulino wrote:
> 
> > Hi there,
> > 
> > GSoC 2011 has been announced[1]. As we were pretty successful last year,
> > I think we should participate again. I've already created a wiki page:
> > 
> > http://wiki.qemu.org/Google_Summer_of_Code_2011
> > 
> > We should now populate it with projects and people willing to be mentors
> > should say so (or just add a project)[2].
> > 
> > Also, I'd like to do something different this year, I'd like to invite
> > libvirt people to join. There are two ways of doing this:
> > 
> > 1. They join in the program as a regular mentoring organization, or
> > 
> > 2. They join with QEMU
> > 
> > The second option means that libvirt can suggest and run its own projects
> > (preferably with QEMU relevance), but from a GSoC perspective, the project
> > will be part of the QEMU org.
> 
> Keep in mind that every full org gets a free trip to the west coast for 2 people ;). So splitting up means we could almost do a mini-summit at the google campus on google's expenses ;).

Actually, they have a limited budget and if you live too far (say, in Brazil),
the trip might not be 100% free :)

> Please coordinate that with Carol. Apparently traction for GSOC is declining (according to last year's summit). So there might be plenty of available slots this year. So I'd say sign up separately for now and if you don't get accepted, just join forces with us!

Yes, that's a good plan and I fully agree that we get more benefits if we
apply separately. It's a call to libvirt's people.

Re: Google Summer of Code 2011

[Alexander Graf]

On 28.01.2011, at 21:10, Luiz Capitulino wrote:

> Hi there,
> 
> GSoC 2011 has been announced[1]. As we were pretty successful last year,
> I think we should participate again. I've already created a wiki page:
> 
> http://wiki.qemu.org/Google_Summer_of_Code_2011
> 
> We should now populate it with projects and people willing to be mentors
> should say so (or just add a project)[2].
> 
> Also, I'd like to do something different this year, I'd like to invite
> libvirt people to join. There are two ways of doing this:
> 
> 1. They join in the program as a regular mentoring organization, or
> 
> 2. They join with QEMU
> 
> The second option means that libvirt can suggest and run its own projects
> (preferably with QEMU relevance), but from a GSoC perspective, the project
> will be part of the QEMU org.

Keep in mind that every full org gets a free trip to the west coast for 2 people ;). So splitting up means we could almost do a mini-summit at the google campus on google's expenses ;).

Please coordinate that with Carol. Apparently traction for GSOC is declining (according to last year's summit). So there might be plenty of available slots this year. So I'd say sign up separately for now and if you don't get accepted, just join forces with us!


Alex

Google Summer of Code 2011

[Luiz Capitulino]

Hi there,

GSoC 2011 has been announced[1]. As we were pretty successful last year,
I think we should participate again. I've already created a wiki page:

 http://wiki.qemu.org/Google_Summer_of_Code_2011

We should now populate it with projects and people willing to be mentors
should say so (or just add a project)[2].

Also, I'd like to do something different this year, I'd like to invite
libvirt people to join. There are two ways of doing this:

 1. They join in the program as a regular mentoring organization, or

 2. They join with QEMU

The second option means that libvirt can suggest and run its own projects
(preferably with QEMU relevance), but from a GSoC perspective, the project
will be part of the QEMU org.

Thanks!

PS: Hope you don't mind the cross posting :)

 [1] http://google-opensource.blogspot.com/2011/01/google-summer-of-code-announced-at-lca.html

 [2] Please, note that being a mentor means having time to dedicate to
     your student