possible race condition (fetch-pack: unexpected disconnect while reading sideband packet)

possible race condition (fetch-pack: unexpected disconnect while reading sideband packet)

[Simon, Martin (Digiteq Automotive)]

Hello,

I'd like to report a bug, if it's not already known, but I wasn't able to find it on https://lore.kernel.org/git/.

Setup:
We use Git for Windows in version 2.31.0 on MS Windows Server 2019, to access repository on Gitlab 13.10. Both machines are virtual under different physical machines, but still in the same server room connected with high speed network to each other.

Reproducibility:
random (like 50:50)

Reproducer:
Try to clone (initial clone, or initial init && fetch, doesn't matter) a repo from the gitlab to the machine

Expected results:
Repository cloned/fetched successfully


Actual result (when failed):
Cloning into 'local-repo-name'...
remote: Enumerating objects: 237, done.
remote: Counting objects: 100% (237/237), done.
remote: Compressing objects: 100% (129/129), done.
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

Notes:
Some repositories are almost always successful (doesn't matter if they are bigger or smaller), clone and init+fetch seem to behave similarly (from the perspective of the issue). From any machine on the same network, but e.g. through several network routers or VPN, the issue was never observed.
Our current git config:
C:\Users\username>git config --list
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=schannel
core.autocrlf=true
core.fscache=true
core.symlinks=false
core.editor=notepad
pull.rebase=false
credential.helper=manager-core
credential.https://dev.azure.com.usehttppath=true
init.defaultbranch=master
http.postbuffer=512000000*
core.compression=9*
core.packedgitlimit=512m*
core.packedgitwindowsize=512m*
pack.deltacachesize=2047m*
pack.packsizelimit=2047m*
pack.windowmemory=2047m*
sendpack.sideband=false*

These marked with asterisk(*) were added to localize this issue, without any observable effect.

I'm willing to provide more information or even contribute on the fix, just let me know what should I do/try. Thank you.

Best Regards,
Martin Simon

Re: possible race condition (fetch-pack: unexpected disconnect while reading sideband packet)

[Jeff King]

On Wed, Apr 14, 2021 at 10:12:44AM +0000, Simon, Martin (Digiteq Automotive) wrote:

> Actual result (when failed):
> Cloning into 'local-repo-name'...
> remote: Enumerating objects: 237, done.
> remote: Counting objects: 100% (237/237), done.
> remote: Compressing objects: 100% (129/129), done.
> fetch-pack: unexpected disconnect while reading sideband packet
> fatal: early EOF
> fatal: fetch-pack: invalid index-pack output

This could be many things. A bug or race on the client side is possible,
but it could also be that the network session really _is_ being
disconnected. This could be something in the network stack itself
terminating the connection, or it could be a bug or other error on the
server.

Some things to consider/try:

  - you can run with GIT_TRACE_PACKET=1 set in the environment, which
    will dump the protocol messages that Git sees (this may not be
    enlightening since the problem case seems to be a hangup, but
    perhaps the moment it happens will be relevant)

  - you didn't say which protocol you're using. If it's http, try
    setting GIT_TRACE_CURL, too, and it will dump debugging data about
    the http session

  - if possible, try a different protocol (like ssh) and see if the
    problem continues to show up

  - look at server-side logs to see how the server perceives the
    sessions. Does it report any errors that might cause it to hang up?
    Does it see a client-side hangup (which might be indicative of
    something flaky in the network)?

-Peff

Re: possible race condition (fetch-pack: unexpected disconnect while reading sideband packet)

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1620 bytes --]

On 2021-04-15 at 08:29:34, Jeff King wrote:
> On Wed, Apr 14, 2021 at 10:12:44AM +0000, Simon, Martin (Digiteq Automotive) wrote:
> 
> > Actual result (when failed):
> > Cloning into 'local-repo-name'...
> > remote: Enumerating objects: 237, done.
> > remote: Counting objects: 100% (237/237), done.
> > remote: Compressing objects: 100% (129/129), done.
> > fetch-pack: unexpected disconnect while reading sideband packet
> > fatal: early EOF
> > fatal: fetch-pack: invalid index-pack output
> 
> This could be many things. A bug or race on the client side is possible,
> but it could also be that the network session really _is_ being
> disconnected. This could be something in the network stack itself
> terminating the connection, or it could be a bug or other error on the
> server.

I'll point out that this kind of occurrence also happens if folks are
using proxies or TLS middleboxes, as well as some antivirus and firewall
programs on Windows (which often end up being well-hidden proxies due to
their configuration, unfortunately).  That usually only applies to
HTTPS, so oftentimes switching to SSH, as Peff suggested, may be a good
choice.

You can also try running the operation under WSL 2, which generally is
not subject to most of those programs and mostly handles its own
networking, to see if it is one of those programs or if there's a more
general network- or protocol-specific problem.  That will at least give
you more information about the nature of what's going on and help you
troubleshoot further.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]