[PATCH] t*: avoid using pipes

[PATCH] t*: avoid using pipes

[Prathamesh Chavan]

Hi,
I'm Prathamesh Chavan. As a part of my micropraoject I have been working on
"Avoid pipes for git related commands in test suites". I tried sending the
patch, but it got blocked since the mail contained more than 100 000
characters.
Hence I'll like to attach the link to my branch 'micro-proj', where I did the
required changes.

https://github.com/pratham-pc/git/tree/micro-proj

Thanks.

Re: [PATCH] t*: avoid using pipes

[Stefan Beller]

On Tue, Mar 7, 2017 at 8:10 AM, Prathamesh Chavan <pc44800@gmail.com> wrote:
> Hi,
> I'm Prathamesh Chavan. As a part of my micropraoject I have been working on
> "Avoid pipes for git related commands in test suites".

Thanks for working on that microproject!

> I tried sending the
> patch, but it got blocked since the mail contained more than 100 000
> characters.

Yeah, even the github UI seems to have trouble with that commit.
(A bit slow, not showing the full content, but rather I needed to click
on "load diff" for tests files 7000+)

This is a lot of change (in terms of lines) for a micro project. :)
I'd have two competing advices:
* keep it micro first, e.g. just convert one file,
  send to list, wait for reviewers feedback and incorporate that
  (optional step after having done the full development cycle:
  convert all the other files; each as its own patch)
* split up this one patch into multiple patches, e.g. one
  file per patch, then send a patch series.

The outcome will be the same, but in the first you get feedback
quicker, such that hopefully you only need to touch the rest of
files after the first file just once.


> Hence I'll like to attach the link to my branch 'micro-proj', where I did the
> required changes.
>
> https://github.com/pratham-pc/git/tree/micro-proj

While I did look at that, not everyone here in the git community
does so. (Also for getting your change in, Junio seems to strongly prefer
patches on list instead of e.g. fetching and cherry-picking from your
github)

When looking at the content, the conversion seems a bit mechanical
(which is fine for a micro project), such as:
...
- test "$(git show --pretty=format:%s | head -n 1)" = "one"
+ test "$(git show --pretty=format:%s >out && head -n 1 <out)" = "one"
...

specifically for the "head" command I don't think it makes a
difference in correctness whether you pipe the file into the tool
or give the filename, i.e.  "head -n 1 out" would work just as fine.

There is a difference in readability, though. For consistency I'd
suggest to drop the "<", as the numbers might support:

$ cd t
$ git grep head |wc -l
# This also counts other occurrences of the string,
# not just the invocation of the head tool
2871
$ git grep head |grep "<" |wc -l
# same here
58

Another aspect might be performance at scale as the "<" will
let the shell open the file and pipe the content via stdin to the
head tool, whereas when giving a filename the head tool needs
to open the file. Both times the file doesn't need to be read completely,
but "head -n 1" can close the file handle early in the game.
I dunno.

Thanks,
Stefan

>
> Thanks.

Re: [PATCH] t*: avoid using pipes

[Johannes Sixt]

Am 07.03.2017 um 18:21 schrieb Stefan Beller:
> On Tue, Mar 7, 2017 at 8:10 AM, Prathamesh Chavan <pc44800@gmail.com> wrote:
>> I'm Prathamesh Chavan. As a part of my micropraoject I have been working on
>> "Avoid pipes for git related commands in test suites".

Welcome to the Git community!

> * keep it micro first, e.g. just convert one file,
>   send to list, wait for reviewers feedback and incorporate that
>   (optional step after having done the full development cycle:
>   convert all the other files; each as its own patch)
> * split up this one patch into multiple patches, e.g. one
>   file per patch, then send a patch series.

Actually, being a *micro* project, it should stay so. Not doing all of 
the changes would leave some tasks for other apprentices to get warm 
with our review process.

>> https://github.com/pratham-pc/git/tree/micro-proj
>
> While I did look at that, not everyone here in the git community
> does so. (Also for getting your change in, Junio seems to strongly prefer
> patches on list instead of e.g. fetching and cherry-picking from your
> github)

Thank you, Stefan, for digging out one particularly interesting case.

> When looking at the content, the conversion seems a bit mechanical
> (which is fine for a micro project), such as:
> ...
> - test "$(git show --pretty=format:%s | head -n 1)" = "one"
> + test "$(git show --pretty=format:%s >out && head -n 1 <out)" = "one"
> ...
>
> specifically for the "head" command I don't think it makes a
> difference in correctness whether you pipe the file into the tool
> or give the filename, i.e.  "head -n 1 out" would work just as fine.

True, but!

The intent of removing git invocations from the upstream of a pipe is 
that a failure exit code is able to stop a && chain.

The example above does not achieve this even after removal of the pipe. 
The reason is that exit code of process expansions $(...) is usually 
ignored. To meet the intent, further changes are necessary, for example to:

	git show --pretty=format:%s >out &&
	test "$(head -n 1 out)" = "one"

Side note: There is one exception where the exit code of $(...) is not 
ignored: when it occurs in the last variable assignment of a command 
that consists only of variable assignments.

-- Hannes

Re: [PATCH] t*: avoid using pipes

[Stefan Beller]

On Tue, Mar 7, 2017 at 12:39 PM, Johannes Sixt <j6t@kdbg.org> wrote:

> Welcome to the Git community!

>
> Actually, being a *micro* project, it should stay so. Not doing all of the
> changes would leave some tasks for other apprentices to get warm with our
> review process.

right, so just pick one file.


> Thank you, Stefan, for digging out one particularly interesting case.
>
>> When looking at the content, the conversion seems a bit mechanical
>> (which is fine for a micro project), such as:
>> ...
>> - test "$(git show --pretty=format:%s | head -n 1)" = "one"
>> + test "$(git show --pretty=format:%s >out && head -n 1 <out)" = "one"
>> ...
>>
>> specifically for the "head" command I don't think it makes a
>> difference in correctness whether you pipe the file into the tool
>> or give the filename, i.e.  "head -n 1 out" would work just as fine.
>
>
> True, but!
>
> The intent of removing git invocations from the upstream of a pipe is that a
> failure exit code is able to stop a && chain.

Doh! I was so fixated over discussing whether to use "<" or not,
to miss looking for the actual goal.

Thanks for spotting!
Stefan

Re: [PATCH] t*: avoid using pipes

[Jeff King]

On Tue, Mar 07, 2017 at 12:52:49PM -0800, Stefan Beller wrote:

> On Tue, Mar 7, 2017 at 12:39 PM, Johannes Sixt <j6t@kdbg.org> wrote:
> 
> > Welcome to the Git community!
> 
> >
> > Actually, being a *micro* project, it should stay so. Not doing all of the
> > changes would leave some tasks for other apprentices to get warm with our
> > review process.
> 
> right, so just pick one file.

I also wonder if we really want all invocations of git to be marked up
in this way. If the primary goal of the test is checking that a certain
git command runs successfully and generates the expected output, then I
think it is a good candidate for conversion.

So in a hunk like this:

   test_expect_success 'git commit-tree records the correct tree in a commit' '
   	commit0=$(echo NO | git commit-tree $P) &&
  -	tree=$(git show --pretty=raw $commit0 |
  -		 sed -n -e "s/^tree //p" -e "/^author /q") &&
  +	tree=$(git show --pretty=raw $commit0 >out &&
  +	sed -n -e "s/^tree //p" -e "/^author /q" <out) &&
   	test "z$tree" = "z$P"

we are interested in testing commit-tree, not "git show". Is it worth
avoiding pipes there? I admit the cost to using the intermediate file is
not huge there, but it feels more awkward and un-shell-like to me as a
reader.

-Peff

Re: [PATCH] t*: avoid using pipes

[Prathamesh Chavan]

On Wed, Mar 8, 2017 at 11:33 AM, Jeff King <peff@peff.net> wrote:
> On Tue, Mar 07, 2017 at 12:52:49PM -0800, Stefan Beller wrote:
>
>> On Tue, Mar 7, 2017 at 12:39 PM, Johannes Sixt <j6t@kdbg.org> wrote:
>>
>> > Welcome to the Git community!
>>
>> >
>> > Actually, being a *micro* project, it should stay so. Not doing all of the
>> > changes would leave some tasks for other apprentices to get warm with our
>> > review process.
>>
>> right, so just pick one file.
>
> I also wonder if we really want all invocations of git to be marked up
> in this way. If the primary goal of the test is checking that a certain
> git command runs successfully and generates the expected output, then I
> think it is a good candidate for conversion.
>
> So in a hunk like this:
>
>    test_expect_success 'git commit-tree records the correct tree in a commit' '
>         commit0=$(echo NO | git commit-tree $P) &&
>   -     tree=$(git show --pretty=raw $commit0 |
>   -              sed -n -e "s/^tree //p" -e "/^author /q") &&
>   +     tree=$(git show --pretty=raw $commit0 >out &&
>   +     sed -n -e "s/^tree //p" -e "/^author /q" <out) &&
>         test "z$tree" = "z$P"
>
> we are interested in testing commit-tree, not "git show". Is it worth
> avoiding pipes there? I admit the cost to using the intermediate file is
> not huge there, but it feels more awkward and un-shell-like to me as a
> reader.
>
> -Peff

Thank you everyone, for reviewing my changes. And as said in the
reviews, I'll send a single patch file as my microproject, leaving the other
files as low hanging fruit for the others to look at. Also, I try to include as
many suggested improvements as possible and will also remember them for
my future patches.

Re: [PATCH] t*: avoid using pipes

[Prathamesh Chavan]

I have created the required changes and submitted a single file patch.
Also I tried my best to include each of the suggestions in that patch.

https://public-inbox.org/git/0102015aae7b8536-00c57d0a-1d48-4153-a202-87c4ea9e0e19-000000@eu-west-1.amazonses.com/

On Wed, Mar 8, 2017 at 7:02 PM, Prathamesh Chavan <pc44800@gmail.com> wrote:
> On Wed, Mar 8, 2017 at 11:33 AM, Jeff King <peff@peff.net> wrote:
>> On Tue, Mar 07, 2017 at 12:52:49PM -0800, Stefan Beller wrote:
>>
>>> On Tue, Mar 7, 2017 at 12:39 PM, Johannes Sixt <j6t@kdbg.org> wrote:
>>>
>>> > Welcome to the Git community!
>>>
>>> >
>>> > Actually, being a *micro* project, it should stay so. Not doing all of the
>>> > changes would leave some tasks for other apprentices to get warm with our
>>> > review process.
>>>
>>> right, so just pick one file.
>>
>> I also wonder if we really want all invocations of git to be marked up
>> in this way. If the primary goal of the test is checking that a certain
>> git command runs successfully and generates the expected output, then I
>> think it is a good candidate for conversion.
>>
>> So in a hunk like this:
>>
>>    test_expect_success 'git commit-tree records the correct tree in a commit' '
>>         commit0=$(echo NO | git commit-tree $P) &&
>>   -     tree=$(git show --pretty=raw $commit0 |
>>   -              sed -n -e "s/^tree //p" -e "/^author /q") &&
>>   +     tree=$(git show --pretty=raw $commit0 >out &&
>>   +     sed -n -e "s/^tree //p" -e "/^author /q" <out) &&
>>         test "z$tree" = "z$P"
>>
>> we are interested in testing commit-tree, not "git show". Is it worth
>> avoiding pipes there? I admit the cost to using the intermediate file is
>> not huge there, but it feels more awkward and un-shell-like to me as a
>> reader.
>>
>> -Peff
>
> Thank you everyone, for reviewing my changes. And as said in the
> reviews, I'll send a single patch file as my microproject, leaving the other
> files as low hanging fruit for the others to look at. Also, I try to include as
> many suggested improvements as possible and will also remember them for
> my future patches.

[PATCH] t*: avoid using pipes

[Prathamesh Chavan]

Hi,
I'm Prathamesh Chavan. As a part of my micropraoject I have been working on
"Avoid pipes for git related commands in test suites". I tried sending the
patch, but it got blocked since the mail contained more than 100 000
characters.
Hence, I have made the required changes in branch "micro-proj" and you may
find it in 'git' repository on my github account. My user name is: pratham-pc.
Please review the changes.

Thanks.