From: Bryan Turner <bturner@atlassian.com>
To: Git Users <git@vger.kernel.org>
Subject: git push over HTTP; long delay with no progress, then hang?
Date: Fri, 15 May 2020 21:09:27 -0700 [thread overview]
Message-ID: <CAGyf7-GQSPcdheKKiZPBpfGxAj_xu4oWdwRq_esSbuqLu5P08g@mail.gmail.com> (raw)
When running a huge "git push" via protocol v0/v1 over HTTP
(repository is ~10GB, with ~104,000 refs), I observe that:
* Git makes an initial connection for a ref advertisement. This
completes almost instantly because the repository is empty
* "git push" then sits in absolute silence for ~10 minutes
The process chain looks like:
git push <URL>
git-remote-http <URL> <URL>
git send-pack --stateless-rpc --helper-status --thin
--progress <URL> --stdin
The "git send-pack" process runs at 100% usage for a single CPU core
for this entire duration. Does anyone have any insight into what Git
might be doing during this long delay? Whatever it is, is it perhaps
something Git should actually print some sort of status for? (I've
reproduced this long silence with both Git 2.20.1 and the new Git
2.27.0-rc0.)
After the long delay, I see another HTTP request to the server and
then the "git push" process finally goes into "Enumerating objects",
"Counting objects", "Compressing objects" and finally "Writing
objects".
Another thing I've noticed is that, during this latter part, it
_appears_ Git opens a connection to the remote server and _then_
starts enumerating/counting/compressing, without having actually said
anything to the server. For this huge push, that actually results in
the server aborting the connection due to a read timeout. Why would
"git push" open a server connection and _then_ do all the work
necessary to create a pack to actually send? (Perhaps this is really
an HTTP keepalive issue, where the connection had been used for a
previous request?) As with the long silence, I reproduced this with
both Git 2.20.1 and 2.27.0-rc0.
Lastly, after the timeout, I observed that my "git push" hung forever
with this output displayed:
bturner$ /opt/git/2.27.0-rc0/bin/git push <URL> --all
Enumerating objects: 13135246, done.
Counting objects: 100% (13135246/13135246), done.
Delta compression using up to 20 threads
Compressing objects: 100% (3867748/3867748), done.
Writing objects: 0% (1/13135246)
An lsof on the "git-remote-http" process showed:
bturner$ lsof -p 64855
COMMAND PID USER FD TYPE DEVICE SIZE/OFF
NODE NAME
git-remot 64855 bturner cwd DIR 1,4 352
168415036 <CWD>
git-remot 64855 bturner txt REG 1,4 2305984
170771507 /opt/git/2.27.0-rc0/libexec/git-core/git-remote-http
git-remot 64855 bturner txt REG 1,4 59156
165457100 /usr/local/Cellar/gettext/0.20.2_1/lib/libintl.8.dylib
git-remot 64855 bturner txt REG 1,4 28420
167195923 /Library/Preferences/Logging/.plist-cache.hR8QH5S4
git-remot 64855 bturner txt REG 1,4 1568368
1152921500312496125 /usr/lib/dyld
git-remot 64855 bturner 0 PIPE 0x9a358149ffa29591 65536
->0xa5ef9caffef8462c
git-remot 64855 bturner 1 PIPE 0x2dce7cc3b04ce8b8 16384
->0x3f62dce60d4a355
git-remot 64855 bturner 2u CHR 16,8 0t1903241
2841 /dev/ttys008
git-remot 64855 bturner 3u systm 0xb078b23901649d47 0t0
[ctl com.apple.netsrc id 7 unit 48]
git-remot 64855 bturner 4u unix 0xb078b238edae81ff 0t0
->0xb078b238edae8777
git-remot 64855 bturner 5u IPv6 0xb078b238f873ac47 0t0
TCP localhost:58980-><host>:<port> (CLOSED)
git-remot 64855 bturner 7 PIPE 0x8e4a8565c7dd32eb 65536
->0xaec041bd7407b713
git-remot 64855 bturner 8 PIPE 0x6ed7631819b64580 65536
->0x52e7cf331dfd1de7
So at some level it was known that the remote host had closed the
socket, but "git-remote-http" was still sitting there.
I can readily reproduce all of this, but unfortunately can't readily
share the repository. I'm happy to do anything I can to contribute to
debugging, if anyone has any thoughts to share!
Best regards,
Bryan Turner
next reply other threads:[~2020-05-16 4:09 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-16 4:09 Bryan Turner [this message]
2020-05-16 6:37 ` git push over HTTP; long delay with no progress, then hang? SZEDER Gábor
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAGyf7-GQSPcdheKKiZPBpfGxAj_xu4oWdwRq_esSbuqLu5P08g@mail.gmail.com \
--to=bturner@atlassian.com \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).