[PATCH] Sleep 1 millisecond in poll() to avoid busy wait

[PATCH] Sleep 1 millisecond in poll() to avoid busy wait

[Stepan Kasal]

From: theoleblond <theodore.leblond@gmail.com>
Date: Wed, 16 May 2012 06:52:49 -0700

I played around with this quite a bit. After trying some more complex
schemes, I found that what worked best is to just sleep 1 millisecond
between iterations. Though it's a very short time, it still completely
eliminates the busy wait condition, without hurting perf.

There code uses SleepEx(1, TRUE) to sleep. See this page for a good
discussion of why that is better than calling SwitchToThread, which
is what was used previously:
http://stackoverflow.com/questions/1383943/switchtothread-vs-sleep1

Note that calling SleepEx(0, TRUE) does *not* solve the busy wait.

The most striking case was when testing on a UNC share with a large repo,
on a single CPU machine. Without the fix, it took 4 minutes 15 seconds,
and with the fix it took just 1:08! I think it's because git-upload-pack's
busy wait was eating the CPU away from the git process that's doing the
real work. With multi-proc, the timing is not much different, but tons of
CPU time is still wasted, which can be a killer on a server that needs to
do bunch of other things.

I also tested the very fast local case, and didn't see any measurable
difference. On a big repo with 4500 files, the upload-pack took about 2
seconds with and without the fix.
---

This is one of the patches that lives in msysGit, could it be
accepted upstream?
It modifies the Windows compat function only.

Have a nice day,
	Stepan

 compat/poll/poll.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/compat/poll/poll.c b/compat/poll/poll.c
index 31163f2..c5a9a93 100644
--- a/compat/poll/poll.c
+++ b/compat/poll/poll.c
@@ -605,7 +605,9 @@ restart:
 
   if (!rc && timeout == INFTIM)
     {
-      SwitchToThread();
+      /* Sleep 1 millisecond to avoid busy wait */
+      SleepEx(1, TRUE);
+
       goto restart;
     }
 
-- 
1.9.2.msysgit.0.158.g6070cee

Re: [PATCH] Sleep 1 millisecond in poll() to avoid busy wait

[Erik Faye-Lund]

On Mon, Apr 28, 2014 at 10:39 AM, Stepan Kasal <kasal@ucw.cz> wrote:
> From: theoleblond <theodore.leblond@gmail.com>
> Date: Wed, 16 May 2012 06:52:49 -0700
>
> I played around with this quite a bit. After trying some more complex
> schemes, I found that what worked best is to just sleep 1 millisecond
> between iterations. Though it's a very short time, it still completely
> eliminates the busy wait condition, without hurting perf.
>
> There code uses SleepEx(1, TRUE) to sleep. See this page for a good
> discussion of why that is better than calling SwitchToThread, which
> is what was used previously:
> http://stackoverflow.com/questions/1383943/switchtothread-vs-sleep1
>
> Note that calling SleepEx(0, TRUE) does *not* solve the busy wait.
>
> The most striking case was when testing on a UNC share with a large repo,
> on a single CPU machine. Without the fix, it took 4 minutes 15 seconds,
> and with the fix it took just 1:08! I think it's because git-upload-pack's
> busy wait was eating the CPU away from the git process that's doing the
> real work. With multi-proc, the timing is not much different, but tons of
> CPU time is still wasted, which can be a killer on a server that needs to
> do bunch of other things.
>
> I also tested the very fast local case, and didn't see any measurable
> difference. On a big repo with 4500 files, the upload-pack took about 2
> seconds with and without the fix.
> ---
>
> This is one of the patches that lives in msysGit, could it be
> accepted upstream?
> It modifies the Windows compat function only.

compat/poll/poll.c comes from Gnulib, so it would be better to submit
the patch there and update.

Re: [PATCH] Sleep 1 millisecond in poll() to avoid busy wait

[Stepan Kasal]

Hello,

On Mon, Apr 28, 2014 at 11:07:24AM +0200, Erik Faye-Lund wrote:
> compat/poll/poll.c comes from Gnulib, so it would be better to submit
> the patch there and update.

well, the change is in gnulib since 2012-05-21.
But the two versions has diverged a lot.
Could you please just accept a backport of this one patch?
I would be glad if msysgit could drop this patch.

Updated patch follows.

Stepan

[PATCH] poll/select: prevent busy-waiting

[Stepan Kasal]

From: Paolo Bonzini <bonzini@gnu.org>
Date: Mon, 21 May 2012 09:52:42 +0200

Backported from Gnulib.

2012-05-21  Paolo Bonzini  <bonzini@gnu.org>

	poll/select: prevent busy-waiting.  SwitchToThread() only gives away
	the rest of the current time slice to another thread in the current
	process. So if the thread that feeds the file decscriptor we're
	polling is not in the current process, we get busy-waiting.
	* lib/poll.c: Use SleepEx(1, TRUE) instead of SwitchToThread().
	Patch from Theodore Leblond.
	* lib/select.c: Split polling out of the loop that sets the output
	fd_sets.  Check for zero result and loop if the wait timeout is
	infinite.

Signed-off-by: Stepan Kasal <kasal@ucw.cz>
---
 compat/poll/poll.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/compat/poll/poll.c b/compat/poll/poll.c
index 31163f2..a9b41d8 100644
--- a/compat/poll/poll.c
+++ b/compat/poll/poll.c
@@ -605,7 +605,7 @@ restart:
 
   if (!rc && timeout == INFTIM)
     {
-      SwitchToThread();
+      SleepEx (1, TRUE);
       goto restart;
     }
 
-- 
1.9.2.msysgit.0.158.g6070cee

Re: [PATCH] poll/select: prevent busy-waiting

[Erik Faye-Lund]

On Mon, Apr 28, 2014 at 1:42 PM, Stepan Kasal <kasal@ucw.cz> wrote:
> From: Paolo Bonzini <bonzini@gnu.org>
> Date: Mon, 21 May 2012 09:52:42 +0200
>
> Backported from Gnulib.
>
> 2012-05-21  Paolo Bonzini  <bonzini@gnu.org>
>
>         poll/select: prevent busy-waiting.  SwitchToThread() only gives away
>         the rest of the current time slice to another thread in the current
>         process. So if the thread that feeds the file decscriptor we're
>         polling is not in the current process, we get busy-waiting.
>         * lib/poll.c: Use SleepEx(1, TRUE) instead of SwitchToThread().
>         Patch from Theodore Leblond.
>         * lib/select.c: Split polling out of the loop that sets the output
>         fd_sets.  Check for zero result and loop if the wait timeout is
>         infinite.
>
> Signed-off-by: Stepan Kasal <kasal@ucw.cz>
> ---
>  compat/poll/poll.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/compat/poll/poll.c b/compat/poll/poll.c
> index 31163f2..a9b41d8 100644
> --- a/compat/poll/poll.c
> +++ b/compat/poll/poll.c
> @@ -605,7 +605,7 @@ restart:
>
>    if (!rc && timeout == INFTIM)
>      {
> -      SwitchToThread();
> +      SleepEx (1, TRUE);
>        goto restart;
>      }
>
> --
> 1.9.2.msysgit.0.158.g6070cee
>

Thanks for taking the effort!

Acked-by: Erik Faye-Lund <kusmabite@gmail.com>

[PATCH] Windows: Always normalize paths to Windows-style

[Stepan Kasal]

From: Heiko Voigt <hvoigt@hvoigt.net>
Date: Thu, 22 Mar 2012 19:17:03 +0100

It appears that `pwd` returns the POSIX-style or the DOS-style path
depending which style the previous `cd` used. To normalize, enforce `pwd
-W` in scripts.

>From the original e-mail exchange:

On Thu, Mar 22, 2012 at 11:13:37AM +0100, Sebastian Schuberth wrote:
> On Wed, Mar 21, 2012 at 22:21, Johannes Sixt <j6t@kdbg.org> wrote:
>
> > I build git and run its tests outside the msysgit environment. Does that
> > explain the difference? (And I use CMD.)
>
> It does not make a difference for me. I started cmd.exe at
> c:\msysgit\git\t, added c:\msysgit\bin temporarily to PATH, and ran
> "sh t5526-fetch-submodules.sh -i -v", and the test still fails.

Yes it probably does. Johannes said that he runs the tests outside of
the msysgit folder. That way there is only one path the submodule script
gets reported and not two like '/c/msysgit/git' and '/git'.

That would explain to me why it is passing.

I am afraid that the only solution is to patch msys itself to report the
long absolute path when passing window style paths to cd. Currently when
I do

	cd c:/msysgit/git

I will end up in '/git' instead of the long path.

I found that there is a -W option to pwd in msys bash which makes it
always return the real windows path. A normalization in that direction
is unique and thus might be more robust. Have a look at the attached
patch. With this at least t5526 passes. I was not able to run the whole
testsuite properly at the moment. I can have a look at that tomorrow.

What do you think?

Cheers Heiko

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---

Hello,
this is another patch that lives in msysGit for a long time.
Originally, it had two parts:
(Cf https://github.com/msysgit/git/commit/64a8a03 )

1) adding alias pwd='pwd -W' to git-sh-setup.sh
   This one went upstream, though as a shell function.

2) revert of commit 4dce7d9b by Johannes Sixt <j6t@kdbg.org>
This mingw-specific commit was created less than 3 weeks before
it was reverted.  And it stayed reverted for two years.

Could you please either accept this patch, or revert 4dce7d9b ?
(Both alternatives are exactly the same.)

Have a nice day,
	Stepan Kasal

 git-submodule.sh | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 4a30087..247273e 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -285,9 +285,6 @@ module_clone()
 	# resolve any symlinks that might be present in $PWD
 	a=$(cd_to_toplevel && cd "$gitdir" && pwd)/
 	b=$(cd_to_toplevel && cd "$sm_path" && pwd)/
-	# normalize Windows-style absolute paths to POSIX-style absolute paths
-	case $a in [a-zA-Z]:/*) a=/${a%%:*}${a#*:} ;; esac
-	case $b in [a-zA-Z]:/*) b=/${b%%:*}${b#*:} ;; esac
 	# Remove all common leading directories after a sanity check
 	if test "${a#$b}" != "$a" || test "${b#$a}" != "$b"; then
 		die "$(eval_gettext "Gitdir '\$a' is part of the submodule path '\$b' or vice versa")"
-- 
1.9.2.msysgit.0.158.g6070cee

Re: [PATCH] Windows: Always normalize paths to Windows-style

[Stepan Kasal]

The parent of this mail should have started a new thread.
Sorry for the typo.
Stepan

Re: [PATCH] Windows: Always normalize paths to Windows-style

[Heiko Voigt]

Hi,

On Mon, Apr 28, 2014 at 04:29:31PM +0200, Stepan Kasal wrote:
> this is another patch that lives in msysGit for a long time.
> Originally, it had two parts:
> (Cf https://github.com/msysgit/git/commit/64a8a03 )
> 
> 1) adding alias pwd='pwd -W' to git-sh-setup.sh
>    This one went upstream, though as a shell function.
> 
> 2) revert of commit 4dce7d9b by Johannes Sixt <j6t@kdbg.org>
> This mingw-specific commit was created less than 3 weeks before
> it was reverted.  And it stayed reverted for two years.
> 
> Could you please either accept this patch, or revert 4dce7d9b ?
> (Both alternatives are exactly the same.)

Sorry for the late reply.  To me reverting (or omitting at the next
rebasing merge) this patch sound fine, as it seems to be superseeded by
the upstream change.

As I can see thats already done on master, so it seems to be all good.

Cheers Heiko

Re: [PATCH] Windows: Always normalize paths to Windows-style

[Junio C Hamano]

Heiko Voigt <hvoigt@hvoigt.net> writes:

> On Mon, Apr 28, 2014 at 04:29:31PM +0200, Stepan Kasal wrote:
>> this is another patch that lives in msysGit for a long time.
>> Originally, it had two parts:
>> (Cf https://github.com/msysgit/git/commit/64a8a03 )
>> 
>> 1) adding alias pwd='pwd -W' to git-sh-setup.sh
>>    This one went upstream, though as a shell function.
>> 
>> 2) revert of commit 4dce7d9b by Johannes Sixt <j6t@kdbg.org>
>> This mingw-specific commit was created less than 3 weeks before
>> it was reverted.  And it stayed reverted for two years.
>> 
>> Could you please either accept this patch, or revert 4dce7d9b ?
>> (Both alternatives are exactly the same.)
>
> Sorry for the late reply.  To me reverting (or omitting at the next
> rebasing merge) this patch sound fine, as it seems to be superseeded by
> the upstream change.
>
> As I can see thats already done on master, so it seems to be all good.

Are you guys talking about be39048a (git-sh-setup.sh: Add an pwd()
function for MinGW, 2012-04-17) which has been in since v1.7.11?

The change introduced by 4dce7d9b (submodules: fix ambiguous
absolute paths under Windows, 2012-03-04) still exists, but your
"reverting this patch sound fine" confuses me.

Re: [PATCH] Windows: Always normalize paths to Windows-style

[Stepan Kasal]

Hello,

On Wed, May 07, 2014 at 01:40:05PM -0700, Junio C Hamano wrote:
> Heiko Voigt <hvoigt@hvoigt.net> writes:
> > On Mon, Apr 28, 2014 at 04:29:31PM +0200, Stepan Kasal wrote:
> >> this is another patch that lives in msysGit for a long time.
> >> Originally, it had two parts:
> >> (Cf https://github.com/msysgit/git/commit/64a8a03 )
> >> 
> >> 1) adding alias pwd='pwd -W' to git-sh-setup.sh
> >>    This one went upstream, though as a shell function.
> >> 
> >> 2) revert of commit 4dce7d9b by Johannes Sixt <j6t@kdbg.org>
> >> This mingw-specific commit was created less than 3 weeks before
> >> it was reverted.  And it stayed reverted for two years.
> >> 
> >> Could you please either accept this patch, or revert 4dce7d9b ?
> >> (Both alternatives are exactly the same.)
> >
> > Sorry for the late reply.  To me reverting (or omitting at the next
> > rebasing merge) this patch sound fine, as it seems to be superseeded by
> > the upstream change.
> >
> > As I can see thats already done on master, so it seems to be all good.

Thank you, Junio, for asking.
(I'm afraid my previous mail was unclear.)

> Are you guys talking about be39048a (git-sh-setup.sh: Add an pwd()
> function for MinGW, 2012-04-17) which has been in since v1.7.11?

This one is ok, keep it.  (msysGit has been using an alias instead of
function for several years, but msysGit/master was synced recently.)

> The change introduced by 4dce7d9b (submodules: fix ambiguous
> absolute paths under Windows, 2012-03-04) still exists, but your
> "reverting this patch sound fine" confuses me.

This one was accepted to git, but was reverted in msysGit almost
immediately by 64a8a03.  Things stayed that way for 2 years.
So it seems no one has ever actually used this code.
Consequently, I propose to revert 4dce7d9b from git.

Stepan

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] Windows: Always normalize paths to Windows-style

[Junio C Hamano]

Stepan Kasal <kasal@ucw.cz> writes:

> This one was accepted to git, but was reverted in msysGit almost
> immediately by 64a8a03.  Things stayed that way for 2 years.
> So it seems no one has ever actually used this code.
> Consequently, I propose to revert 4dce7d9b from git.

OK, let's do this.  I'd like you to take the authorship of it (and
correct any mistake I may have made in the description) with your
sign-off.

Thanks.

-- >8 --
From: Stepan Kasal <kasal@ucw.cz>
Subject: [PATCH] Revert "submodules: fix ambiguous absolute paths under Windows"

This reverts commit 4dce7d9b408b2935b85721b54a2010eda7ec1be9,
which was originally done to help Windows but was almost
immediately reverted in msysGit, and the codebase kept this
unnecessary divergence for almost two years.

Not-Signed-off-yet-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 git-submodule.sh | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 66f5f75..821e6d4 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -285,9 +285,6 @@ module_clone()
 	# resolve any symlinks that might be present in $PWD
 	a=$(cd_to_toplevel && cd "$gitdir" && pwd)/
 	b=$(cd_to_toplevel && cd "$sm_path" && pwd)/
-	# normalize Windows-style absolute paths to POSIX-style absolute paths
-	case $a in [a-zA-Z]:/*) a=/${a%%:*}${a#*:} ;; esac
-	case $b in [a-zA-Z]:/*) b=/${b%%:*}${b#*:} ;; esac
 	# Remove all common leading directories after a sanity check
 	if test "${a#$b}" != "$a" || test "${b#$a}" != "$b"; then
 		die "$(eval_gettext "Gitdir '\$a' is part of the submodule path '\$b' or vice versa")"
-- 
2.0.0-rc2-397-ga0fd1fc

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH] Revert "submodules: fix ambiguous absolute paths under Windows"

[Stepan Kasal]

This reverts commit 4dce7d9b408b2935b85721b54a2010eda7ec1be9,
which was originally done to help Windows but was almost
immediately reverted in msysGit, and the codebase kept this
unnecessary divergence for almost two years.

Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
On Thu, May 08, 2014 at 01:13:25PM -0700, Junio C Hamano wrote:
> OK, let's do this.  I'd like you to take the authorship of it (and

ok; thank you for the nice description, no need for any correction.

 git-submodule.sh | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 66f5f75..821e6d4 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -285,9 +285,6 @@ module_clone()
 	# resolve any symlinks that might be present in $PWD
 	a=$(cd_to_toplevel && cd "$gitdir" && pwd)/
 	b=$(cd_to_toplevel && cd "$sm_path" && pwd)/
-	# normalize Windows-style absolute paths to POSIX-style absolute paths
-	case $a in [a-zA-Z]:/*) a=/${a%%:*}${a#*:} ;; esac
-	case $b in [a-zA-Z]:/*) b=/${b%%:*}${b#*:} ;; esac
 	# Remove all common leading directories after a sanity check
 	if test "${a#$b}" != "$a" || test "${b#$a}" != "$b"; then
 		die "$(eval_gettext "Gitdir '\$a' is part of the submodule path '\$b' or vice versa")"
-- 
2.0.0-rc2-397-ga0fd1fc

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] poll/select: prevent busy-waiting

[Johannes Sixt]

Am 4/28/2014 13:44, schrieb Erik Faye-Lund:
> On Mon, Apr 28, 2014 at 1:42 PM, Stepan Kasal <kasal@ucw.cz> wrote:
>> From: Paolo Bonzini <bonzini@gnu.org>
>> Date: Mon, 21 May 2012 09:52:42 +0200
>>
>> Backported from Gnulib.
>>
>> 2012-05-21  Paolo Bonzini  <bonzini@gnu.org>
>>
>>         poll/select: prevent busy-waiting.  SwitchToThread() only gives away
>>         the rest of the current time slice to another thread in the current
>>         process. So if the thread that feeds the file decscriptor we're
>>         polling is not in the current process, we get busy-waiting.
>>         * lib/poll.c: Use SleepEx(1, TRUE) instead of SwitchToThread().
>>         Patch from Theodore Leblond.
>>         * lib/select.c: Split polling out of the loop that sets the output
>>         fd_sets.  Check for zero result and loop if the wait timeout is
>>         infinite.
>>
>> Signed-off-by: Stepan Kasal <kasal@ucw.cz>
>> ---
>>  compat/poll/poll.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/compat/poll/poll.c b/compat/poll/poll.c
>> index 31163f2..a9b41d8 100644
>> --- a/compat/poll/poll.c
>> +++ b/compat/poll/poll.c
>> @@ -605,7 +605,7 @@ restart:
>>
>>    if (!rc && timeout == INFTIM)
>>      {
>> -      SwitchToThread();
>> +      SleepEx (1, TRUE);
>>        goto restart;
>>      }
>>
>> --
>> 1.9.2.msysgit.0.158.g6070cee
>>
> 
> Thanks for taking the effort!
> 
> Acked-by: Erik Faye-Lund <kusmabite@gmail.com>

The patch text has my ACK, too (I've been using it since 2 years now), but
I very much prefer the commit message of the earlier post. GNU standards
w.r.t. commit messages are simply sub-par. ;-)

-- Hannes

[PATCH v2] Sleep 1 millisecond in poll() to avoid busy wait

[Stepan Kasal]

From: theoleblond <theodore.leblond@gmail.com>
Date: Wed, 16 May 2012 06:52:49 -0700

SwitchToThread() only gives away the rest of the current time slice
to another thread in the current process. So if the thread that feeds
the file decscriptor we're polling is not in the current process, we
get busy-waiting.

I played around with this quite a bit. After trying some more complex
schemes, I found that what worked best is to just sleep 1 millisecond
between iterations. Though it's a very short time, it still completely
eliminates the busy wait condition, without hurting perf.

There code uses SleepEx(1, TRUE) to sleep. See this page for a good
discussion of why that is better than calling SwitchToThread, which
is what was used previously:
http://stackoverflow.com/questions/1383943/switchtothread-vs-sleep1

Note that calling SleepEx(0, TRUE) does *not* solve the busy wait.

The most striking case was when testing on a UNC share with a large repo,
on a single CPU machine. Without the fix, it took 4 minutes 15 seconds,
and with the fix it took just 1:08! I think it's because git-upload-pack's
busy wait was eating the CPU away from the git process that's doing the
real work. With multi-proc, the timing is not much different, but tons of
CPU time is still wasted, which can be a killer on a server that needs to
do bunch of other things.

I also tested the very fast local case, and didn't see any measurable
difference. On a big repo with 4500 files, the upload-pack took about 2
seconds with and without the fix.
---

On Mon, Apr 28, 2014 at 05:05:47PM +0200, Johannes Sixt wrote:
> [...] but I very much prefer the commit message of the earlier post.

... but Paolo had a nice short description of the issue there;
I inserted that to the top of the earlier commit message.

The latter diff (without the comment), gets us closer to gnulib's poll.c.

Have a nice day,
	Stepan

 compat/poll/poll.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/compat/poll/poll.c b/compat/poll/poll.c
index 31163f2..a9b41d8 100644
--- a/compat/poll/poll.c
+++ b/compat/poll/poll.c
@@ -605,7 +605,7 @@ restart:
 
   if (!rc && timeout == INFTIM)
     {
-      SwitchToThread();
+      SleepEx (1, TRUE);
       goto restart;
     }
 
-- 
1.9.2.msysgit.0.158.g6070cee

Re: [PATCH v2] Sleep 1 millisecond in poll() to avoid busy wait

[Erik Faye-Lund]

On Mon, Apr 28, 2014 at 5:35 PM, Stepan Kasal <kasal@ucw.cz> wrote:
> From: theoleblond <theodore.leblond@gmail.com>
> Date: Wed, 16 May 2012 06:52:49 -0700
>
> SwitchToThread() only gives away the rest of the current time slice
> to another thread in the current process. So if the thread that feeds
> the file decscriptor we're polling is not in the current process, we
> get busy-waiting.
>
> I played around with this quite a bit. After trying some more complex
> schemes, I found that what worked best is to just sleep 1 millisecond
> between iterations. Though it's a very short time, it still completely
> eliminates the busy wait condition, without hurting perf.
>
> There code uses SleepEx(1, TRUE) to sleep. See this page for a good
> discussion of why that is better than calling SwitchToThread, which
> is what was used previously:
> http://stackoverflow.com/questions/1383943/switchtothread-vs-sleep1
>
> Note that calling SleepEx(0, TRUE) does *not* solve the busy wait.
>
> The most striking case was when testing on a UNC share with a large repo,
> on a single CPU machine. Without the fix, it took 4 minutes 15 seconds,
> and with the fix it took just 1:08! I think it's because git-upload-pack's
> busy wait was eating the CPU away from the git process that's doing the
> real work. With multi-proc, the timing is not much different, but tons of
> CPU time is still wasted, which can be a killer on a server that needs to
> do bunch of other things.
>
> I also tested the very fast local case, and didn't see any measurable
> difference. On a big repo with 4500 files, the upload-pack took about 2
> seconds with and without the fix.
> ---
>
> On Mon, Apr 28, 2014 at 05:05:47PM +0200, Johannes Sixt wrote:
>> [...] but I very much prefer the commit message of the earlier post.
>
> ... but Paolo had a nice short description of the issue there;
> I inserted that to the top of the earlier commit message.
>
> The latter diff (without the comment), gets us closer to gnulib's poll.c.
>

Good stuff!

Re: [PATCH v2] Sleep 1 millisecond in poll() to avoid busy wait

[Junio C Hamano]

Erik Faye-Lund <kusmabite@gmail.com> writes:

> On Mon, Apr 28, 2014 at 5:35 PM, Stepan Kasal <kasal@ucw.cz> wrote:
>> From: theoleblond <theodore.leblond@gmail.com>
>> Date: Wed, 16 May 2012 06:52:49 -0700
>>
>> ...
>> I also tested the very fast local case, and didn't see any measurable
>> difference. On a big repo with 4500 files, the upload-pack took about 2
>> seconds with and without the fix.
>> ---
>>
> Good stuff!

Do I expect to be forwarded this patch with proper provenance
sign-off chain thru you?

I do not mind adding Ack from you and J6t I saw in this thread
myself, perhaps like the following:

    Subject: compat/poll: sleep 1 millisecond to avoid busy wait
    From: Theodore Leblond <theodore.leblond@gmail.com>

    SwitchToTrhead() only gives...
    ...
    seconds with and without the fix.

    Signed-off-by: Theodore Leblond <theodore.leblond@gmail.com>
    Signed-off-by: Stepan Kasal <kasal@ucw.cz>
    Acked-by: Johannes Sixt <j6t@kdbg.org>
    Acked-by: Erik Faye-Lund <kusmabite@gmail.com>

I am just having this feeling that I lack the background on the
evolution of this patch to say the above properly records the
provenance.

Oh, also in the last paragraph, who is "I" who tested?  Theodore, or
Stepan?

Thanks.
    

Re: [PATCH v2] Sleep 1 millisecond in poll() to avoid busy wait

[Stepan Kasal]

Hello Junio,

thank you for pointing out the problems.

Let me explain the background:

After some discussion a one line fix to win32/poll.c was accepted to msysgit/git
at 2012-05-16 https://github.com/msysgit/git/pull/7

The description of the commit looked like this:
> I played around with this [...]
> [...]
> I also tested the very fast local case, and didn't see any measurable
> difference. On a big repo with 4500 files, the upload-pack took about 2
> seconds with and without the fix.
... but there was no sign-off by Theodore.

Because poll.c comes from Gnulib, it was reported there as well.
Paolo Bonzini accepted the fix for poll.c and added a fix for select.
The combined commit got this changelog entry:

> 2012-05-21  Paolo Bonzini  <bonzini@gnu.org>
> 
>         poll/select: prevent busy-waiting.  SwitchToThread() only gives away
>         the rest of the current time slice to another thread in the current
>         process. So if the thread that feeds the file decscriptor we're
>         polling is not in the current process, we get busy-waiting.
>         * lib/poll.c: Use SleepEx(1, TRUE) instead of SwitchToThread().
>         Patch from Theodore Leblond.
>         * lib/select.c: Split polling out of the loop that sets the output
>         fd_sets.  Check for zero result and loop if the wait timeout is
>         infinite.

The patch that I (Stepan) submitted as "v2" combines things like this:
- subject by Theodore
- first paragraph by Paolo, concise problem description
- rest from Theodore's original commit ("I" = Theodore, I suppose)
- diff exctly as in gnulib commit

On Mon, Apr 28, 2014 at 11:58:50AM -0700, Junio C Hamano wrote:
>     Subject: compat/poll: sleep 1 millisecond to avoid busy wait

Thanks for improving it.

>     Signed-off-by: Theodore Leblond <theodore.leblond@gmail.com>
>     Signed-off-by: Stepan Kasal <kasal@ucw.cz>
>     Acked-by: Johannes Sixt <j6t@kdbg.org>
>     Acked-by: Erik Faye-Lund <kusmabite@gmail.com>

Sorry that I forgot to add my sign-off (Stepan).
But I'm afraid I cannot add Theodore's, it was not in the original
commit.  But it is one line change; the real work was the analysis.

Stepan

Re: [PATCH v2] Sleep 1 millisecond in poll() to avoid busy wait

[Junio C Hamano]

Stepan Kasal <kasal@ucw.cz> writes:

> Hello Junio,
>
> thank you for pointing out the problems.
>
> Let me explain the background:
>
> After some discussion a one line fix to win32/poll.c was accepted to msysgit/git
> at 2012-05-16 https://github.com/msysgit/git/pull/7
>
> The description of the commit looked like this:
>> I played around with this [...]
>> [...]
>> I also tested the very fast local case, and didn't see any measurable
>> difference. On a big repo with 4500 files, the upload-pack took about 2
>> seconds with and without the fix.
> ... but there was no sign-off by Theodore.
>
> Because poll.c comes from Gnulib, it was reported there as well.
> Paolo Bonzini accepted the fix for poll.c and added a fix for select.
> The combined commit got this changelog entry:
>
>> 2012-05-21  Paolo Bonzini  <bonzini@gnu.org>
>> 
>>         poll/select: prevent busy-waiting.  SwitchToThread() only gives away
>>         the rest of the current time slice to another thread in the current
>>         process. So if the thread that feeds the file decscriptor we're
>>         polling is not in the current process, we get busy-waiting.
>>         * lib/poll.c: Use SleepEx(1, TRUE) instead of SwitchToThread().
>>         Patch from Theodore Leblond.
>>         * lib/select.c: Split polling out of the loop that sets the output
>>         fd_sets.  Check for zero result and loop if the wait timeout is
>>         infinite.
>
> The patch that I (Stepan) submitted as "v2" combines things like this:
> - subject by Theodore
> - first paragraph by Paolo, concise problem description
> - rest from Theodore's original commit ("I" = Theodore, I suppose)
> - diff exctly as in gnulib commit
>
> On Mon, Apr 28, 2014 at 11:58:50AM -0700, Junio C Hamano wrote:
>>     Subject: compat/poll: sleep 1 millisecond to avoid busy wait
>
> Thanks for improving it.
>
>>     Signed-off-by: Theodore Leblond <theodore.leblond@gmail.com>
>>     Signed-off-by: Stepan Kasal <kasal@ucw.cz>
>>     Acked-by: Johannes Sixt <j6t@kdbg.org>
>>     Acked-by: Erik Faye-Lund <kusmabite@gmail.com>
>
> Sorry that I forgot to add my sign-off (Stepan).
> But I'm afraid I cannot add Theodore's, it was not in the original
> commit.  But it is one line change; the real work was the analysis.

Well, the original git/pull/7 in msysgit repository says that is a
patch by Theodore, so the kosher thing to do is to ask him (and I
see he is on the Cc list), and also ask msysgit folks (and I see
they are on the Cc list as well) to be a bit more careful when
responding to pull requests, especially given that it is our mutual
benefit to make sure we keep the changes between our two trees to
the minimum by upstreaming.

I'll queue without the forged sign-off by Theodore, as I think DCO
(1.1) (c) read loosely would let me do so ;-)

Thanks.