Git 2.3.7 hangs on fetch but not clone

Git 2.3.7 hangs on fetch but not clone

[Randall S. Becker]

I have some strange behaviour I am trying to diagnose on the NonStop port of
git 2.3.7. The situation is we have a *LARGE* cloned repository with some
local modifications of openssl, which we are trying to clone again for a
Jenkins build. The command:
	git clone /local/openssl openssl
works fine and rapidly (well under 30 seconds), but with a newly created
empty repo, simulating what the Jenkins Git plugin does:
	mkdir /local/test/openssl
	cd /local/test/openssl
	git init /local/test/openssl
	git -c core.askpass=true fetch --verbose --progress
/local/git/openssl +refs/heads/*:refs/remotes/origin/*
does the usual:
	remote: Counting objects: 113436, done.
	remote: Compressing objects: 100% (23462/23462), done.
then hangs forever without creating any files in the working directory.
There are also no files or directories modified since the init operation.
Processes left around, and without consuming resources, are:
1493172290 2030043151 - 15:58:29       00:15 git pack-objects --revs --thin
--stdout --progress --delta-base-offset --include-tag
452984908  402653262 - 15:58:29       00:00 git -c core.askpass=true fetch
--verbose --progress /local/git/openssl +refs/heads/*:refs/remotes/origin/*
402653262 1694498903 - 15:58:28       00:00 git -c core.askpass=true fetch
--verbose --progress /local/git/openssl +refs/heads/*:refs/remotes/origin/*
2030043151  402653262 - 15:58:29       00:00 git-upload-pack
/local/git/openssl

This does not happen for our smaller repositories. Any pointers on where to
look would be appreciated.

Kindly,
Randall

-- Brief whoami: NonStop&UNIX developer since approximately
UNIX(421664400)/NonStop(211288444200000000)
-- In my real life, I talk too much.

Re: Git 2.3.7 hangs on fetch but not clone

[Jeff King]

On Sun, Dec 06, 2015 at 05:12:12PM -0500, Randall S. Becker wrote:

> I have some strange behaviour I am trying to diagnose on the NonStop port of
> git 2.3.7. The situation is we have a *LARGE* cloned repository with some
> local modifications of openssl, which we are trying to clone again for a
> Jenkins build. The command:
> 	git clone /local/openssl openssl

Because the two repositories are on the same local filesystem, git will
skip the usual pack transfer and just copy or hardlink the object files.

You can add "--no-local" to make it more like the fetch that is failing
(I suspect it will then fail, too, but if it doesn't, that would be an
interesting data point).

> 	remote: Counting objects: 113436, done.
> 	remote: Compressing objects: 100% (23462/23462), done.
> then hangs forever without creating any files in the working directory.
> There are also no files or directories modified since the init operation.
> Processes left around, and without consuming resources, are:
> 1493172290 2030043151 - 15:58:29       00:15 git pack-objects --revs --thin
> --stdout --progress --delta-base-offset --include-tag
> 452984908  402653262 - 15:58:29       00:00 git -c core.askpass=true fetch
> --verbose --progress /local/git/openssl +refs/heads/*:refs/remotes/origin/*
> 402653262 1694498903 - 15:58:28       00:00 git -c core.askpass=true fetch
> --verbose --progress /local/git/openssl +refs/heads/*:refs/remotes/origin/*
> 2030043151  402653262 - 15:58:29       00:00 git-upload-pack
> /local/git/openssl

What are the processes doing?

Upload-pack on the "server" side spawns pack-objects, writes the set of
wanted objects to its stdin, and then waits for it to produce pack data
to stdout (which it then multiplexes back to the client). You clearly
got past the write (since pack-objects finished "Counting objects...").
I'd expect it to be waiting for input in poll() for input from the
stdout or stderr of pack-objects. It may also be blocking on write()
back to the client.

Pack-objects clearly completed delta compression, and so should be
writing out the final pack.  It doesn't look like it ever wrote the
"writing objects..." progress line (or perhaps upload-pack got stuck and
failed to relay it). It's probably blocking on write() to upload-pack.

You have two fetch processes. Presumably you build with NO_PTHREADS and
one of them is a fork()ed copy to demux the data from upload-pack. It
clearly runs at least a little while (it relays the "Counting" message).
It should either be blocking on read() from upload-pack, or perhaps
blocking on write() back to main fetch.

The main fetch hasn't yet spawned index-pack or unpack-objects, which
means it hasn't gotten enough of the pack header to do so. So it's
probably blocking on read() from the sideband demuxer process.

Do you have some equivalent of strace on your system? Or better yet, a
debugger?

> This does not happen for our smaller repositories. Any pointers on where to
> look would be appreciated.

My wild guess would be some pipe deadlock between the processes, but I
don't think we've had any of those lately. And v2.3.7 is not _too_ old
(there are some fixes for http deadlocks in v2.3.7..v2.6.3, but that
should not affect you here).

-Peff