t5551 hangs ?

t5551 hangs ?

[Torsten Bögershausen]

Hej,
I found 2 threads about hanging t5551, one in 2016, and one question
from Junio somewhen in 2017.
I have it reproducable here:
- Debian 8, 64 bit
- SSD
- Half-modern processor ;-)

The last thing I can see is:
ok 29 - test allowanysha1inwant with unreachable
-----------
A simplified ps -ef shows:
/usr/sbin/apache2 -d /home/blalbla/t/trash directory.t5551-http-fetch-smart/httpd -f /home/blalbla/t/lib-httpd/apache.conf -c Listen 127.0.0.1:5551 -k start
/usr/sbin/apache2 -d /home/blalbla/t/trash directory.t5551-http-fetch-smart/httpd -f /home/blalbla/t/lib-httpd/apache.conf -c Listen 127.0.0.1:5551 -k start
/usr/sbin/apache2 -d /home/blalbla/t/trash directory.t5551-http-fetch-smart/httpd -f /home/blalbla/t/lib-httpd/apache.conf -c Listen 127.0.0.1:5551 -k start
/usr/sbin/apache2 -d /home/blalbla/t/trash directory.t5551-http-fetch-smart/httpd -f /home/blalbla/t/lib-httpd/apache.conf -c Listen 127.0.0.1:5551 -k start
/usr/sbin/apache2 -d /home/blalbla/t/trash directory.t5551-http-fetch-smart/httpd -f /home/blalbla/t/lib-httpd/apache.conf -c Listen 127.0.0.1:5551 -k start
/usr/sbin/apache2 -d /home/blalbla/t/trash directory.t5551-http-fetch-smart/httpd -f /home/blalbla/t/lib-httpd/apache.conf -c Listen 127.0.0.1:5551 -k start
/usr/sbin/apache2 -d /home/blalbla/t/trash directory.t5551-http-fetch-smart/httpd -f /home/blalbla/t/lib-httpd/apache.conf -c Listen 127.0.0.1:5551 -k start
/home/blalbla/git -C too-many-refs fetch -q --tags
/home/blalbla/git-remote-http origin http://127.0.0.1:5551/smart/repo.git
-------------
The tests passes on another box (real old laptop, spinning disk)

Anything that can be done to debug it ?

Re: t5551 hangs ?

[Santiago Torres]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

Hello, Torsten. 

On Tue, Aug 22, 2017 at 07:10:59AM +0200, Torsten Bögershausen wrote:
> Hej,
> I found 2 threads about hanging t5551, one in 2016, and one question
> from Junio somewhen in 2017.
> I have it reproducable here:
> - Debian 8, 64 bit
> - SSD
> - Half-modern processor ;-)

I don't think the SSD/regular disk have anything to do with it. 

- Are you running tests concurrently? (e.g., with -j x)
- What happens if you run the test individually (i.e., cd t and
  ./t5551...)
- You probably want to see the version of apache this is running/etc.
- What happens if you kill the apache processes? 

I can't reproduce on my side, but let me see if I can dig a little into
it.

Cheers!
-Santiago.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: t5551 hangs ?

[Torsten Bögershausen]

On Tue, Aug 22, 2017 at 01:26:25AM -0400, Santiago Torres wrote:
> Hello, Torsten. 
> 
> On Tue, Aug 22, 2017 at 07:10:59AM +0200, Torsten Bögershausen wrote:
> > Hej,
> > I found 2 threads about hanging t5551, one in 2016, and one question
> > from Junio somewhen in 2017.
> > I have it reproducable here:
> > - Debian 8, 64 bit
> > - SSD
> > - Half-modern processor ;-)
> 
> I don't think the SSD/regular disk have anything to do with it. 
> 
> - Are you running tests concurrently? (e.g., with -j x)
> - What happens if you run the test individually (i.e., cd t and
>   ./t5551...)

No concurrency.
That's what I do: ./t5551-http-fetch-smart.sh

> - You probably want to see the version of apache this is running/etc.

The one that comes with Debian -
I am not an expert here, what is it that is interesting ?

> - What happens if you kill the apache processes? 

I am left with these processes:
/home/blabla/pu/git -C too-many-refs fetch -q --tags
/home/blabla/pu/git-remote-http origin http://127.0.0.1:5551/smart/repo.git

> 
> I can't reproduce on my side, but let me see if I can dig a little into
> it.

Thanks, more tips or things I can do are welcome.
t5551 seems to be flaky - from time to time.
It seems that I have it reproducable unstable, so if someone has more
ideas, please.

> 
> Cheers!
> -Santiago.

Re: t5551 hangs ?

[Santiago Torres]

[-- Attachment #1: Type: text/plain, Size: 1457 bytes --]

> No concurrency.
> That's what I do: ./t5551-http-fetch-smart.sh

Oh ok! I was just trying to weed out what could out of place. Just to
make sure, your old compute is also running debian 8 latest? 
> 
> > - You probably want to see the version of apache this is running/etc.
> 
> The one that comes with Debian -
> I am not an expert here, what is it that is interesting ?

I'm not sure, but sometimes it's helpful to look at changelogs between
versions of packages to get a poiner as to what could be wrong. I wonder
if it's apache the one that's failing on this case. Probably a paste of
dpkg-query --show apache would help me try to reproduce..

> 
> > - What happens if you kill the apache processes? 
> 
> I am left with these processes:
> /home/blabla/pu/git -C too-many-refs fetch -q --tags
> /home/blabla/pu/git-remote-http origin http://127.0.0.1:5551/smart/repo.git
> 

So it's probably not an issue with apache, as the socket should be
hopefully closed gracefully.

> > 
> > I can't reproduce on my side, but let me see if I can dig a little into
> > it.
> 
> Thanks, more tips or things I can do are welcome.
> t5551 seems to be flaky - from time to time.
> It seems that I have it reproducable unstable, so if someone has more
> ideas, please.

I'm still unable to reproduce. Do you think you can enable GIT_TRACE,
GIT_TRACE_PACK and GIT_TRACE_CURL and pastebin/paste what you see?

Cheers!
-Santiago.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v7 0/3] Add support for sending additional HTTP headers (part 2)

[Johannes Schindelin]

My use case is an army of build agents that need only limited and
selective access to otherwise private repositories.

The first part already made it into `master`, this is the remainder.

This iteration still has the specific patch to make `git -c
http.extraHeader=... submodule update` work; I plan to keep only the
test (and adjust the commit message) as soon as Peff's patch is applied
that skips -c ... sanitizing for submodules.


Johannes Schindelin (3):
  tests: Adjust the configuration for Apache 2.2
  t5551: make the test for extra HTTP headers more robust
  submodule: pass on http.extraheader config settings

 builtin/submodule--helper.c |  3 ++-
 t/lib-httpd/apache.conf     | 12 ++++++++----
 t/t5551-http-fetch-smart.sh | 14 ++++++++++++--
 3 files changed, 22 insertions(+), 7 deletions(-)

Published-As: https://github.com/dscho/git/releases/tag/extra-http-headers-v7
Interdiff vs v6:

 diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
 index b8ed96f..29b34bb 100644
 --- a/t/lib-httpd/apache.conf
 +++ b/t/lib-httpd/apache.conf
 @@ -103,10 +103,6 @@ Alias /auth/dumb/ www/auth/dumb/
  	Header set Set-Cookie name=value
  </LocationMatch>
  <LocationMatch /smart_headers/>
 -	<RequireAll>
 -		Require expr %{HTTP:x-magic-one} == 'abra'
 -		Require expr %{HTTP:x-magic-two} == 'cadabra'
 -	</RequireAll>
  	SetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}
  	SetEnv GIT_HTTP_EXPORT_ALL
  </LocationMatch>
 @@ -136,6 +132,14 @@ RewriteRule ^/ftp-redir/(.*)$ ftp://localhost:1000/$1 [R=302]
  RewriteRule ^/loop-redir/x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-(.*) /$1 [R=302]
  RewriteRule ^/loop-redir/(.*)$ /loop-redir/x-$1 [R=302]
  
 +# Apache 2.2 does not understand <RequireAll>, so we use RewriteCond.
 +# And as RewriteCond unfortunately lacks "not equal" matching, we use this
 +# ugly trick to fail *unless* the two headers are present.
 +RewriteCond %{HTTP:x-magic-one} =abra
 +RewriteCond %{HTTP:x-magic-two} =cadabra
 +RewriteRule ^/smart_headers/.* - [L]
 +RewriteRule ^/smart_headers/.* - [F]
 +
  <IfDefine SSL>
  LoadModule ssl_module modules/mod_ssl.so
  
 diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
 index 1794168..2f375eb 100755
 --- a/t/t5551-http-fetch-smart.sh
 +++ b/t/t5551-http-fetch-smart.sh
 @@ -283,7 +283,8 @@ test_expect_success EXPENSIVE 'http can handle enormous ref negotiation' '
  '
  
  test_expect_success 'custom http headers' '
 -	test_must_fail git fetch "$HTTPD_URL/smart_headers/repo.git" &&
 +	test_must_fail git -c http.extraheader="x-magic-two: cadabra" \
 +		fetch "$HTTPD_URL/smart_headers/repo.git" &&
  	git -c http.extraheader="x-magic-one: abra" \
  	    -c http.extraheader="x-magic-two: cadabra" \
  	    fetch "$HTTPD_URL/smart_headers/repo.git" &&

-- 
2.8.2.463.g99156ee

[PATCH v8 0/3] Add support for sending additional HTTP headers (part 2)

[Johannes Schindelin]

My use case is an army of build agents that need only limited and
selective access to otherwise private repositories.

The first part already made it into `master`, this is the remainder.

This iteration is based on 'jk/submodule-c-credential' and therefore
converted the original config-sanitizing patch into a test-only patch.
This iteration also replaces the "ugly" comment with the explanation
preferred by Junio.


Johannes Schindelin (3):
  tests: adjust the configuration for Apache 2.2
  t5551: make the test for extra HTTP headers more robust
  submodule: ensure that -c http.extraheader is heeded

 t/lib-httpd/apache.conf     | 16 ++++++++++++----
 t/t5551-http-fetch-smart.sh | 14 ++++++++++++--
 2 files changed, 24 insertions(+), 6 deletions(-)

Published-As: https://github.com/dscho/git/releases/tag/extra-http-headers-v8
-- 
Interdiff vs v7:

 diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
 [no longer applies; skipped]
 diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
 index 29b34bb..018a83a 100644
 --- a/t/lib-httpd/apache.conf
 +++ b/t/lib-httpd/apache.conf
 @@ -133,8 +133,12 @@ RewriteRule ^/loop-redir/x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-(.*) /$1 [R=302
  RewriteRule ^/loop-redir/(.*)$ /loop-redir/x-$1 [R=302]
  
  # Apache 2.2 does not understand <RequireAll>, so we use RewriteCond.
 -# And as RewriteCond unfortunately lacks "not equal" matching, we use this
 -# ugly trick to fail *unless* the two headers are present.
 +# And as RewriteCond does not allow testing for non-matches, we match
 +# the desired case first (one has abra, two has cadabra), and let it
 +# pass by marking the RewriteRule as [L], "last rule, do not process
 +# any other matching RewriteRules after this"), and then have another
 +# RewriteRule that matches all other cases and lets them fail via '[F]',
 +# "fail the request".
  RewriteCond %{HTTP:x-magic-one} =abra
  RewriteCond %{HTTP:x-magic-two} =cadabra
  RewriteRule ^/smart_headers/.* - [L]

2.8.2.463.g99156ee

[PATCH v8 2/3] t5551: make the test for extra HTTP headers more robust

[Johannes Schindelin]

To test that extra HTTP headers are passed correctly, t5551 verifies that
a fetch succeeds when two required headers are passed, and that the fetch
does not succeed when those headers are not passed.

However, this test would also succeed if the configuration required only
one header. As Apache's configuration is notoriously tricky (this
developer frequently requires StackOverflow's help to understand Apache's
documentation), especially when still supporting the 2.2 line, let's just
really make sure that the test verifies what we want it to verify.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 t/t5551-http-fetch-smart.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index e44fe72..43b257e 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -283,7 +283,8 @@ test_expect_success EXPENSIVE 'http can handle enormous ref negotiation' '
 '
 
 test_expect_success 'custom http headers' '
-	test_must_fail git fetch "$HTTPD_URL/smart_headers/repo.git" &&
+	test_must_fail git -c http.extraheader="x-magic-two: cadabra" \
+		fetch "$HTTPD_URL/smart_headers/repo.git" &&
 	git -c http.extraheader="x-magic-one: abra" \
 	    -c http.extraheader="x-magic-two: cadabra" \
 	    fetch "$HTTPD_URL/smart_headers/repo.git"
-- 
2.8.2.463.g99156ee

t5551 hangs ?

[Torsten Bögershausen]

On 10.05.16 09:08, Johannes Schindelin wrote:
- I'm not sure, if this is the right thread to report on -

It seems as if t5551 is hanging ?
This is the last line from the log:
ok 25 - large fetch-pack requests can be split across POSTs

I have 7 such processes running:
/trash directory.t5551-http-fetch-smart/httpd -f
/Users/tb/projects/git/git.pu/t/lib-httpd/apache.conf -DDarwin -c Listen
127.0.0.1:5551 -k start

This happens both under Mac OS X and Debian.

Does anybody have the same hanging ?

Re: t5551 hangs ?

[Jeff King]

On Wed, May 11, 2016 at 07:13:56PM +0200, Torsten Bögershausen wrote:

> On 10.05.16 09:08, Johannes Schindelin wrote:
> - I'm not sure, if this is the right thread to report on -
> 
> It seems as if t5551 is hanging ?
> This is the last line from the log:
> ok 25 - large fetch-pack requests can be split across POSTs

Are you running the tests with "--long" or GIT_TEST_LONG in the
environment? The next line should show it skipping test 26 unless one of
those is set.

If you are, can you confirm that it's actually hanging, and not just
slow? On my system, test 26 takes about a minute to run (which is why we
don't do it by default).

> I have 7 such processes running:
> /trash directory.t5551-http-fetch-smart/httpd -f
> /Users/tb/projects/git/git.pu/t/lib-httpd/apache.conf -DDarwin -c Listen
> 127.0.0.1:5551 -k start

That's normal while the test is running; apache pre-forks a bunch of
worker threads.

-Peff

Re: t5551 hangs ?

[Torsten Bögershausen]

On 11.05.16 19:31, Jeff King wrote:
> On Wed, May 11, 2016 at 07:13:56PM +0200, Torsten Bögershausen wrote:
> 
>> On 10.05.16 09:08, Johannes Schindelin wrote:
>> - I'm not sure, if this is the right thread to report on -
>>
>> It seems as if t5551 is hanging ?
>> This is the last line from the log:
>> ok 25 - large fetch-pack requests can be split across POSTs
> 
> Are you running the tests with "--long" or GIT_TEST_LONG in the
> environment? The next line should show it skipping test 26 unless one of
> those is set.
Yes
> 
> If you are, can you confirm that it's actually hanging, and not just
> slow? On my system, test 26 takes about a minute to run (which is why we
> don't do it by default).
Nearly sure. After 10 minutes, the test was still running.

Yesterday another machine was running even longer.

Any tips, how to debug, are welcome.

Re: t5551 hangs ?

[Jeff King]

On Wed, May 11, 2016 at 10:03:45PM +0200, Torsten Bögershausen wrote:

> > If you are, can you confirm that it's actually hanging, and not just
> > slow? On my system, test 26 takes about a minute to run (which is why we
> > don't do it by default).
> Nearly sure. After 10 minutes, the test was still running.
> 
> Yesterday another machine was running even longer.
> 
> Any tips, how to debug, are welcome.

Try running with "-x" to see what the test is doing. It will probably be
in:

   + git -C too-many-refs fetch -q --tags

after a while. Check "ps" to see if you have a fetch-pack sub-process
running. It should be writing "have" lines and reading lots of ACK
lines, which you can check via strace.

If it's blocked on read() or write(), then it's probably some kind of
I/O deadlock.

-Peff

Re: t5551 hangs ?

[Torsten Bögershausen]

On 12.05.16 05:16, Jeff King wrote:
> On Wed, May 11, 2016 at 10:03:45PM +0200, Torsten Bögershausen wrote:
> 
>>> If you are, can you confirm that it's actually hanging, and not just
>>> slow? On my system, test 26 takes about a minute to run (which is why we
>>> don't do it by default).
>> Nearly sure. After 10 minutes, the test was still running.
>>
>> Yesterday another machine was running even longer.
>>
>> Any tips, how to debug, are welcome.
> 
> Try running with "-x" to see what the test is doing. It will probably be
> in:
> 
>    + git -C too-many-refs fetch -q --tags
> 
> after a while. Check "ps" to see if you have a fetch-pack sub-process
> running. It should be writing "have" lines and reading lots of ACK
> lines, which you can check via strace.
> 
> If it's blocked on read() or write(), then it's probably some kind of
> I/O deadlock.
> 
> -Peff

This is the last log that I see:
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit      = 8589934592
pack_report: pack_used_ctr            =      96001
pack_report: pack_mmap_calls          =      48002
pack_report: pack_open_windows        =          2 /          2
pack_report: pack_mapped              =    6605494 /    6605494
---------------------------------------------------------------------

+++ perl -e 'print "bla" x 30'
+++ command /usr/bin/perl -e 'print "bla" x 30'
+++ /usr/bin/perl -e 'print "bla" x 30'
++ tag=blablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
++ sed -e 's|^:\([^ ]*\) \(.*\)$|\2 refs/tags/blablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla-\1|'
++ git -C too-many-refs fetch -q --tags

And this may be the processes :
(Not sure, probaly need to reboot & clean ?)
/bin/sh ./t5551-http-fetch-smart.sh -x
73459 ttys010    0:21.45 /Users/tb/projects/git/git.pu/git -C too-many-refs fetch -q --tags
73460 ttys010    0:00.40 git-remote-http origin http://127.0.0.1:5551/smart/repo.git

 ps | grep fetch
73540 ttys006    0:00.00 grep fetch
73025 ttys010    0:00.14 /bin/sh ./t5551-http-fetch-smart.sh -x
73459 ttys010    3:40.70 /Users/tb/projects/git/git.pu/git -C too-many-refs fetch -q --tags


Beside that, reverting the last 2 commits on 5551 doesn't seem to help:
 Revert "t5551: make the test for extra HTTP headers more robust"
 Revert "submodule: ensure that -c http.extraheader is heeded"

Re: t5551 hangs ?

[Jeff King]

On Thu, May 12, 2016 at 08:21:10AM +0200, Torsten Bögershausen wrote:

> This is the last log that I see:
> [...]
> ++ git -C too-many-refs fetch -q --tags

Not surprising.

> And this may be the processes :
> (Not sure, probaly need to reboot & clean ?)

If you're killing the hung test with "^C", you shouldn't need to; that
tries to clean up any processes and shut down apache.

> /bin/sh ./t5551-http-fetch-smart.sh -x
> 73459 ttys010    0:21.45 /Users/tb/projects/git/git.pu/git -C too-many-refs fetch -q --tags
> 73460 ttys010    0:00.40 git-remote-http origin http://127.0.0.1:5551/smart/repo.git
> 
>  ps | grep fetch
> 73540 ttys006    0:00.00 grep fetch
> 73025 ttys010    0:00.14 /bin/sh ./t5551-http-fetch-smart.sh -x
> 73459 ttys010    3:40.70 /Users/tb/projects/git/git.pu/git -C too-many-refs fetch -q --tags

I'm surprised not to see fetch-pack in that list. And to see so much CPU
going to fetch itself. But perhaps you are simply at a different stage
in the test. 3:40 of CPU time is a lot (the whole thing runs in under a
minute on my machine).

Hmm. Switching to "pu" seems to make things slow on my machine, too, and
the time all goes to fetch. So perhaps there is some recent regression
there. It should be bisectable.

-Peff

Re: t5551 hangs ?

[Jeff King]

On Thu, May 12, 2016 at 02:40:39AM -0400, Jeff King wrote:

> Hmm. Switching to "pu" seems to make things slow on my machine, too, and
> the time all goes to fetch. So perhaps there is some recent regression
> there. It should be bisectable.

It's 66d33af21bd1e398973414435af43d06f2e2099c. I don't think it's
hanging, but it is _really_ slow. I'll reply to that patch separately
with a report.

-Peff