GSoC Project | Convert interactive rebase to C

GSoC Project | Convert interactive rebase to C

[Ivan Tham]

Hi everyone,

I am Ivan Tham. Currently studying in Computer Science in APIIT Malaysia. I am
interested particapate in Google Summer of Code 2017 under git organization. I
would like to attempt "Add more builtin patterns for userdiff" particularly for
shell for my microproject.

I am interested to work on "Convert interactive rebase to C" aiming to port
most builtins stuff to C in which we can reduce the size of git. Additionally,
I would also like to convert scripts to builtins as an additional milestone.

What do you think of these projects? Would it collide with
[Valery Tolstov's Shell to Builtins proposal][0]?

[0]: https://public-inbox.org/git/1489145258.10535.0@smtp.yandex.ru/

Re: GSoC Project | Convert interactive rebase to C

[Stefan Beller]

On Mon, Mar 20, 2017 at 9:41 AM, Ivan Tham <pickfire@riseup.net> wrote:
> Hi everyone,
>
> I am Ivan Tham. Currently studying in Computer Science in APIIT Malaysia. I am
> interested particapate in Google Summer of Code 2017 under git organization. I
> would like to attempt "Add more builtin patterns for userdiff" particularly for
> shell for my microproject.

I'd love to see proper shell support!
Although there is already some support for shell (by looking at diffs
on our test
suite) ? So I am not sure what there is left to do? Can you clarify what you're
trying there?

> I am interested to work on "Convert interactive rebase to C"

+cc Johannes, who recently worked on rebase and the sequencer.

>  aiming to port
> most builtins stuff to C in which we can reduce the size of git. Additionally,
> I would also like to convert scripts to builtins as an additional milestone.
>
> What do you think of these projects? Would it collide with
> [Valery Tolstov's Shell to Builtins proposal][0]?

Curious why all people ask about colliding with Valerys proposal here?
I do not think it would collide, as submodules and rebase are very different
areas of the code base.

Thanks,
Stefan

>
> [0]: https://public-inbox.org/git/1489145258.10535.0@smtp.yandex.ru/
>

Re: Re: GSoC Project | Convert interactive rebase to C

[Ivan Tham]

Stefan Beller <sbeller@google.com> wrote:
> On Mon, Mar 20, 2017 at 9:41 AM, Ivan Tham <pickfire@riseup.net> wrote:
> > I am Ivan Tham. Currently studying in Computer Science in APIIT Malaysia. I am
> > interested particapate in Google Summer of Code 2017 under git organization. I
> > would like to attempt "Add more builtin patterns for userdiff" particularly for
> > shell for my microproject.
>
> I'd love to see proper shell support!
> Although there is already some support for shell (by looking at diffs
> on our test
> suite) ? So I am not sure what there is left to do? Can you clarify what you're
> trying there?
>
> > I am interested to work on "Convert interactive rebase to C"

Are you sure about that? From what I had looked into userdiff.c, there is no
support for shell. There just a recent patch for [go patterns][0]. Or perhaps
I should have rename it as "userdiff.c: patterns for "shell" language"?

> +cc Johannes, who recently worked on rebase and the sequencer.
>
> >  aiming to port
> > most builtins stuff to C in which we can reduce the size of git. Additionally,
> > I would also like to convert scripts to builtins as an additional milestone.
> >
> > What do you think of these projects? Would it collide with
> > Valery Tolstov's Shell to Builtins proposal?
>
> Curious why all people ask about colliding with Valerys proposal here?
> I do not think it would collide, as submodules and rebase are very different
> areas of the code base.

[0]: https://public-inbox.org/git/xmqqk27jvg28.fsf@gitster.mtv.corp.google.com/

Re: Re: GSoC Project | Convert interactive rebase to C

[Johannes Schindelin]

Hi Ivan,

On Tue, 21 Mar 2017, Ivan Tham wrote:

> Stefan Beller <sbeller@google.com> wrote:
> > On Mon, Mar 20, 2017 at 9:41 AM, Ivan Tham <pickfire@riseup.net> wrote:
> > > I am Ivan Tham. Currently studying in Computer Science in APIIT
> > > Malaysia. I am interested particapate in Google Summer of Code 2017
> > > under git organization. I would like to attempt "Add more builtin
> > > patterns for userdiff" particularly for shell for my microproject.
> >
> > I'd love to see proper shell support!  Although there is already some
> > support for shell (by looking at diffs on our test suite) ? So I am
> > not sure what there is left to do? Can you clarify what you're trying
> > there?
> 
> Are you sure about that? From what I had looked into userdiff.c, there
> is no support for shell. There just a recent patch for [go patterns][0].
> Or perhaps I should have rename it as "userdiff.c: patterns for "shell"
> language"?

I also could not find any shell patterns in the userdiff code...

> > > I am interested to work on "Convert interactive rebase to C"
> >
> > +cc Johannes, who recently worked on rebase and the sequencer.

Glad you are interested! Please note that large parts of the interactive
rebase are already in C now, but there is enough work left in that corner.

> > > aiming to port most builtins stuff to C in which we can reduce the
> > > size of git. Additionally, I would also like to convert scripts to
> > > builtins as an additional milestone.

Careful. It is a ton of work to get the rebase -i conversion done, and
then a ton of work to get it integrated. That will fill 3 months, very
easily.

> > > What do you think of these projects? Would it collide with Valery
> > > Tolstov's Shell to Builtins proposal?

I missed that proposal, and could only find submodule-related mails on the
public-inbox server. Care to provide a pointer?

> > Curious why all people ask about colliding with Valerys proposal here?
> > I do not think it would collide, as submodules and rebase are very
> > different areas of the code base.

Indeed ;-)

Ciao,
Johannes

Re: Re: GSoC Project | Convert interactive rebase to C

[Stefan Beller]

On Thu, Mar 23, 2017 at 10:30 AM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>> Are you sure about that? From what I had looked into userdiff.c, there
>> is no support for shell. There just a recent patch for [go patterns][0].
>> Or perhaps I should have rename it as "userdiff.c: patterns for "shell"
>> language"?
>
> I also could not find any shell patterns in the userdiff code...

Sorry for the confusion on my part. I am unfamiliar with that part of the
diffing lib. I based my assumption on the output of "$ git log -p t/"
which contains hunk headers, that make sense most of the time for shell
code.

>
> Careful. It is a ton of work to get the rebase -i conversion done, and
> then a ton of work to get it integrated. That will fill 3 months, very
> easily.
>
>> > > What do you think of these projects? Would it collide with Valery
>> > > Tolstov's Shell to Builtins proposal?
>
> I missed that proposal, and could only find submodule-related mails on the
> public-inbox server. Care to provide a pointer?

Well I don't think it is a full grown proposal, but a confident attempt to
find a good starting point :)
e.g.
https://public-inbox.org/git/1489145258.10535.0@smtp.yandex.ru/

>> > Curious why all people ask about colliding with Valerys proposal here?
>> > I do not think it would collide, as submodules and rebase are very
>> > different areas of the code base.
>
> Indeed ;-)

Although I had some discussion on what the correct behavior of
"rebase --recurse-submodules" in a superproject should be. But this
sounds like future work after the GSoC is over. ;)

Thanks,
Stefan

Re: Re: Re: GSoC Project | Convert interactive rebase to C

[Inaw Tham]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> On Tue, 21 Mar 2017, Ivan Tham wrote:
> > Stefan Beller <sbeller@google.com> wrote:
> > > On Mon, Mar 20, 2017 at 9:41 AM, Ivan Tham <pickfire@riseup.net> wrote:
> > > > I am Ivan Tham. Currently studying in Computer Science in APIIT
> > > > Malaysia. I am interested particapate in Google Summer of Code 2017
> > > > under git organization. I would like to attempt "Add more builtin
> > > > patterns for userdiff" particularly for shell for my microproject.
> > >
> > > I'd love to see proper shell support!  Although there is already some
> > > support for shell (by looking at diffs on our test suite) ? So I am
> > > not sure what there is left to do? Can you clarify what you're trying
> > > there?
> > 
> > Are you sure about that? From what I had looked into userdiff.c, there
> > is no support for shell. There just a recent patch for [go patterns][0].
> > Or perhaps I should have rename it as "userdiff.c: patterns for "shell"
> > language"?
> 
> I also could not find any shell patterns in the userdiff code...

Thanks a lot for replying me, I thought no one was intereted. :D

> > > > I am interested to work on "Convert interactive rebase to C"
> > >
> > > +cc Johannes, who recently worked on rebase and the sequencer.
> 
> Glad you are interested! Please note that large parts of the interactive
> rebase are already in C now, but there is enough work left in that corner.

Glad to hear that, I would really like to see interactive rebase in C.

> > > > aiming to port most builtins stuff to C in which we can reduce the
> > > > size of git. Additionally, I would also like to convert scripts to
> > > > builtins as an additional milestone.
> 
> Careful. It is a ton of work to get the rebase -i conversion done, and
> then a ton of work to get it integrated. That will fill 3 months, very
> easily.

My main aim is to reduce the extra dependency of perl, but planning to start
with rebase, can I make that an optional task where I can help out after I
had completed my main task during gsoc?

> > > > What do you think of these projects? Would it collide with Valery
> > > > Tolstov's Shell to Builtins proposal?
> 
> I missed that proposal, and could only find submodule-related mails on the
> public-inbox server. Care to provide a pointer?
> 
> > > Curious why all people ask about colliding with Valerys proposal here?
> > > I do not think it would collide, as submodules and rebase are very
> > > different areas of the code base.
> 
> Indeed ;-)

Cheers,
Ivan Tham

Re: Re: Re: GSoC Project | Convert interactive rebase to C

[Johannes Schindelin]

Hi Ivan,

On Sat, 25 Mar 2017, Inaw Tham wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > On Tue, 21 Mar 2017, Ivan Tham wrote:
> > > Stefan Beller <sbeller@google.com> wrote:
> > > > On Mon, Mar 20, 2017 at 9:41 AM, Ivan Tham <pickfire@riseup.net> wrote:
> > > >
> > > > > I am interested to work on "Convert interactive rebase to C"
> > > >
> > > > +cc Johannes, who recently worked on rebase and the sequencer.
> > 
> > Glad you are interested! Please note that large parts of the
> > interactive rebase are already in C now, but there is enough work left
> > in that corner.
> 
> Glad to hear that, I would really like to see interactive rebase in C.

Please note that a notable part already made it into C in v2.12.1. There
are still a few loose ends to tie, of course; it still makes for a great
head start on your project, methinks.

> > > > > aiming to port most builtins stuff to C in which we can reduce
> > > > > the size of git. Additionally, I would also like to convert
> > > > > scripts to builtins as an additional milestone.
> > 
> > Careful. It is a ton of work to get the rebase -i conversion done, and
> > then a ton of work to get it integrated. That will fill 3 months, very
> > easily.
> 
> My main aim is to reduce the extra dependency of perl, but planning to
> start with rebase, can I make that an optional task where I can help out
> after I had completed my main task during gsoc?

Sure, you can make it an optional task, and I would be very happy if you
followed up on it even after GSoC!

As far as the Perl dependency is concerned, I actually think there is only
one serious one left: git add -i.

Granted, there is send-email, but it really does not matter all that much
these days *except* if you want to use Git to contribute to projects that
still use a mailing list-based patch submission process (the ones that
come to mind are: Git, Linux and Cygwin). Most Git users actually do not
submit any patches to mailing lists, therefore I tend to ignore this one.

The rest of the Perl scripts interacts with foreign SCMs (archimport,
cvsexportcommit, cvsimport, cvsserver, and svn). I *guess* that it would
be nice to follow up on the remote-svn work (which has not really gone
anywhere so far, AFAICT the main driving contributor pursues different
projects these days), but IMHO none of these are really needed to run Git.

Ciao,
Johannes

Re: Re: Re: Re: GSoC Project | Convert interactive rebase to C

[Pickfire]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

> On Sat, 25 Mar 2017, Ivan Tham wrote:
>
> > Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > > On Tue, 21 Mar 2017, Ivan Tham wrote:
> > > > Stefan Beller <sbeller@google.com> wrote:
> > > > > On Mon, Mar 20, 2017 at 9:41 AM, Ivan Tham <pickfire@riseup.net> wrote:
> > > > >
> > > > > > I am interested to work on "Convert interactive rebase to C"
> > > > >
> > > > > +cc Johannes, who recently worked on rebase and the sequencer.
> > >
> > > Glad you are interested! Please note that large parts of the
> > > interactive rebase are already in C now, but there is enough work left
> > > in that corner.
> >
> > Glad to hear that, I would really like to see interactive rebase in C.
>
> Please note that a notable part already made it into C in v2.12.1. There
> are still a few loose ends to tie, of course; it still makes for a great
> head start on your project, methinks.

Ah, that's great.

And while I was working on the microproject (shell patterns in user diff),
I can't produce the output of t/t4034-diff-words.sh manually with:

    mkdir cpp/ && cd cpp/ && git init

    cat > pre <<EOF
    Foo():x(0&&1){}
    cout<<"Hello World!\n"<<endl;
    1 -1e10 0xabcdef 'x'
    [a] a->b a.b
    !a ~a a++ a-- a*b a&b
    a*b a/b a%b
    a+b a-b
    a<<b a>>b
    a<b a<=b a>b a>=b
    a==b a!=b
    a&b
    a^b
    a|b
    a&&b
    a||b
    a?b:z
    a=b a+=b a-=b a*=b a/=b a%=b a<<=b a>>=b a&=b a^=b a|=b
    a,y
    a::b
    EOF

    cat > post <<EOF
    Foo() : x(0&42) { bar(x); }
    cout<<"Hello World?\n"<<endl;
    (1) (-1e10) (0xabcdef) 'y'
    [x] x->y x.y
    !x ~x x++ x-- x*y x&y
    x*y x/y x%y
    x+y x-y
    x<<y x>>y
    x<y x<=y x>y x>=y
    x==y x!=y
    x&y
    x^y
    x|y
    x&&y
    x||y
    x?y:z
    x=y x+=y x-=y x*=y x/=y x%=y x<<=y x>>=y x&=y x^=y x|=y
    x,y
    x::y
    EOF

    echo '* diff="cpp"' > .gitmodules
    git diff --no-index --color-words pre post > output

Surprisingly, it shows (which is very different from the expected output):

^[[1mdiff --git a/pre b/post^[[m
^[[1mindex 23d5c8a..7e8c026 100644^[[m
^[[1m--- a/pre^[[m
^[[1m+++ b/post^[[m
^[[36m@@ -1,19 +1,19 @@^[[m
^[[31mFoo():x(0&&1){}^[[m^[[32mFoo() : x(0&42) { bar(x); }^[[m
cout<<"Hello ^[[31mWorld!\n"<<endl;^[[m
^[[31m1 -1e10 0xabcdef 'x'^[[m
^[[31m[a] a->b a.b^[[m
^[[31m!a ~a a++ a-- a*b a&b^[[m
^[[31ma*b a/b a%b^[[m
^[[31ma+b a-b^[[m
^[[31ma<<b a>>b^[[m
^[[31ma<b a<=b a>b a>=b^[[m
^[[31ma==b a!=b^[[m
^[[31ma&b^[[m
^[[31ma^b^[[m
^[[31ma|b^[[m
^[[31ma&&b^[[m
^[[31ma||b^[[m
^[[31ma?b:z^[[m
^[[31ma=b a+=b a-=b a*=b a/=b a%=b a<<=b a>>=b a&=b a^=b a|=b^[[m
^[[31ma,y^[[m
^[[31ma::b^[[m^[[32mWorld?\n"<<endl;^[[m
^[[32m(1) (-1e10) (0xabcdef) 'y'^[[m
^[[32m[x] x->y x.y^[[m
^[[32m!x ~x x++ x-- x*y x&y^[[m
^[[32mx*y x/y x%y^[[m
^[[32mx+y x-y^[[m
^[[32mx<<y x>>y^[[m
^[[32mx<y x<=y x>y x>=y^[[m
^[[32mx==y x!=y^[[m
^[[32mx&y^[[m
^[[32mx^y^[[m
^[[32mx|y^[[m
^[[32mx&&y^[[m
^[[32mx||y^[[m
^[[32mx?y:z^[[m
^[[32mx=y x+=y x-=y x*=y x/=y x%=y x<<=y x>>=y x&=y x^=y x|=y^[[m
^[[32mx,y^[[m
^[[32mx::y^[[m

Instead of:

<BOLD>diff --git a/pre b/post<RESET>
<BOLD>index 23d5c8a..7e8c026 100644<RESET>
<BOLD>--- a/pre<RESET>
<BOLD>+++ b/post<RESET>
<CYAN>@@ -1,19 +1,19 @@<RESET>
Foo() : x(0<RED>&&1<RESET><GREEN>&42<RESET>) { <GREEN>bar(x);<RESET> }
cout<<"Hello World<RED>!<RESET><GREEN>?<RESET>\n"<<endl;
<GREEN>(<RESET>1<GREEN>) (<RESET>-1e10<GREEN>) (<RESET>0xabcdef<GREEN>)<RESET> '<RED>x<RESET><GREEN>y<RESET>'
[<RED>a<RESET><GREEN>x<RESET>] <RED>a<RESET><GREEN>x<RESET>-><RED>b a<RESET><GREEN>y x<RESET>.<RED>b<RESET><GREEN>y<RESET>
!<RED>a<RESET><GREEN>x<RESET> ~<RED>a a<RESET><GREEN>x x<RESET>++ <RED>a<RESET><GREEN>x<RESET>-- <RED>a<RESET><GREEN>x<RESET>*<RED>b a<RESET><GREEN>y x<RESET>&<RED>b<RESET>
<RED>a<RESET><GREEN>y<RESET>
<GREEN>x<RESET>*<RED>b a<RESET><GREEN>y x<RESET>/<RED>b a<RESET><GREEN>y x<RESET>%<RED>b<RESET>
<RED>a<RESET><GREEN>y<RESET>
<GREEN>x<RESET>+<RED>b a<RESET><GREEN>y x<RESET>-<RED>b<RESET>
<RED>a<RESET><GREEN>y<RESET>
<GREEN>x<RESET><<<RED>b a<RESET><GREEN>y x<RESET>>><RED>b<RESET>
<RED>a<RESET><GREEN>y<RESET>
<GREEN>x<RESET><<RED>b a<RESET><GREEN>y x<RESET><=<RED>b a<RESET><GREEN>y x<RESET>><RED>b a<RESET><GREEN>y x<RESET>>=<RED>b<RESET>
<RED>a<RESET><GREEN>y<RESET>
<GREEN>x<RESET>==<RED>b a<RESET><GREEN>y x<RESET>!=<RED>b<RESET>
<RED>a<RESET><GREEN>y<RESET>
<GREEN>x<RESET>&<RED>b<RESET>
<RED>a<RESET><GREEN>y<RESET>
<GREEN>x<RESET>^<RED>b<RESET>
<RED>a<RESET><GREEN>y<RESET>
<GREEN>x<RESET>|<RED>b<RESET>
<RED>a<RESET><GREEN>y<RESET>
<GREEN>x<RESET>&&<RED>b<RESET>
<RED>a<RESET><GREEN>y<RESET>
<GREEN>x<RESET>||<RED>b<RESET>
<RED>a<RESET><GREEN>y<RESET>
<GREEN>x<RESET>?<RED>b<RESET><GREEN>y<RESET>:z
<RED>a<RESET><GREEN>x<RESET>=<RED>b a<RESET><GREEN>y x<RESET>+=<RED>b a<RESET><GREEN>y x<RESET>-=<RED>b a<RESET><GREEN>y x<RESET>*=<RED>b a<RESET><GREEN>y x<RESET>/=<RED>b a<RESET><GREEN>y x<RESET>%=<RED>b a<RESET><GREEN>y x<RESET><<=<RED>b a<RESET><GREEN>y x<RESET>>>=<RED>b a<RESET><GREEN>y x<RESET>&=<RED>b a<RESET><GREEN>y x<RESET>^=<RED>b a<RESET><GREEN>y x<RESET>|=<RED>b<RESET>
<RED>a<RESET><GREEN>y<RESET>
<GREEN>x<RESET>,y
<RED>a<RESET><GREEN>x<RESET>::<RED>b<RESET><GREEN>y<RESET>

That's does not just happens to cpp builtins, it happens to bibtex as well.
Is it that I had missed some configuration since I have tested this on a
few machines?

> > > > > > aiming to port most builtins stuff to C in which we can reduce
> > > > > > the size of git. Additionally, I would also like to convert
> > > > > > scripts to builtins as an additional milestone.
> > >
> > > Careful. It is a ton of work to get the rebase -i conversion done, and
> > > then a ton of work to get it integrated. That will fill 3 months, very
> > > easily.
> >
> > My main aim is to reduce the extra dependency of perl, but planning to
> > start with rebase, can I make that an optional task where I can help out
> > after I had completed my main task during gsoc?
>
> Sure, you can make it an optional task, and I would be very happy if you
> followed up on it even after GSoC!

Yes, I can do that as well. I will be happy to have git be smaller. ^^

> As far as the Perl dependency is concerned, I actually think there is only
> one serious one left: git add -i.

Yes, that as well. But basically the core parts first.

> Granted, there is send-email, but it really does not matter all that much
> these days *except* if you want to use Git to contribute to projects that
> still use a mailing list-based patch submission process (the ones that
> come to mind are: Git, Linux and Cygwin). Most Git users actually do not
> submit any patches to mailing lists, therefore I tend to ignore this one.

I won't ignore that but I will do the others first since some package
manager pack it with git but instead let it be a subpackage.

> The rest of the Perl scripts interacts with foreign SCMs (archimport,
> cvsexportcommit, cvsimport, cvsserver, and svn). I *guess* that it would
> be nice to follow up on the remote-svn work (which has not really gone
> anywhere so far, AFAICT the main driving contributor pursues different
> projects these days), but IMHO none of these are really needed to run Git.

As far as I have concern, the conversion stuff are rarely runned, I
don't think it is worth converting to C.

Good luck and have a nice day!  - Ivan

Re: Re: Re: Re: GSoC Project | Convert interactive rebase to C

[Christian Couder]

On Mon, Mar 27, 2017 at 6:31 PM, Pickfire <pickfire@riseup.net> wrote:
> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>
>> On Sat, 25 Mar 2017, Ivan Tham wrote:
>>
>> > Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>> > > On Tue, 21 Mar 2017, Ivan Tham wrote:
>> > > > Stefan Beller <sbeller@google.com> wrote:
>> > > > > On Mon, Mar 20, 2017 at 9:41 AM, Ivan Tham <pickfire@riseup.net> wrote:
>> > > > >
>> > > > > > I am interested to work on "Convert interactive rebase to C"
>> > > > >
>> > > > > +cc Johannes, who recently worked on rebase and the sequencer.
>> > >
>> > > Glad you are interested! Please note that large parts of the
>> > > interactive rebase are already in C now, but there is enough work left
>> > > in that corner.
>> >
>> > Glad to hear that, I would really like to see interactive rebase in C.
>>
>> Please note that a notable part already made it into C in v2.12.1. There
>> are still a few loose ends to tie, of course; it still makes for a great
>> head start on your project, methinks.
>
> Ah, that's great.
>
> And while I was working on the microproject (shell patterns in user diff),
> I can't produce the output of t/t4034-diff-words.sh manually with:

I don't think it's a good idea to discuss a microproject in the same
thread where a project is discussed.
I would suggest to move it in another thread where you describe in
more details what you want to do and why, what you expect and what
happened, and so on.

[...]

> That's does not just happens to cpp builtins, it happens to bibtex as well.
> Is it that I had missed some configuration since I have tested this on a
> few machines?

From a very quick look the problem seems related to how
test_decode_color() is used or not.