git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Bug with Git shallow clones and submodules
@ 2021-08-11 14:59 Kevin Phillips
  2021-08-15 14:31 ` Philippe Blain
  0 siblings, 1 reply; 5+ messages in thread
From: Kevin Phillips @ 2021-08-11 14:59 UTC (permalink / raw)
  To: git

We recently came across a bug in one of our Git projects which make
use of submodules. The basic problem is we have a Git repo that has 1
submodule within it which is set up to check out a specific branch,
and when we attempt to perform a shallow clone on the parent repo the
checkout fails with a "Server does not allow request for unadvertised
object" error. After doing a lot of testing and reading online I found
this StackOverflow thread (https://stackoverflow.com/a/61492323) that
explains our problem in great detail, which indicates to me that this
is likely not a problem specific to us, so I thought I'd report it to
see if it could be fixed.

In our specific use case, we found the problem specifically when
running a build of our project using Gitlab CI. This tool
automatically performs a shallow checkout of projects being built by
using a "--depth=50" parameter. This caused odd Git checkout errors to
appear on our CI builds which weren't directly reproducible by our
developers local builds. Upon closer examination we were able to
reproduce outside of the CI environment by performing a checkout as
follows:

    git clone --recurse-submodules --depth=50  git@<path_to_repo>.git .

Based on my initial research, which was helped greatly by the
StackOverflow thread I linked to above, I was able to deduce the
following contributing factors:

The submodule that was causing us grief was the dpdk library located
here: http://git.dpdk.org/dpdk-stable/

This repository has a default branch named "default" which has only 2
commits on it: one for initializing the repo and one additional commit
All releases of this library are created from branches named by the
release version, and they all seem disjoint from one another, and they
are all disjoint from the default branch (ie: changes from the release
branches are never merged into the default branch)

our parent repo, which is an internal / non-public repository that can
not be shared, has a single submodule defined in it with the following
.gitmodule definition:


[submodule "dpdk-stable"]
path = dpdk-stable
url = https://git@dpdk.org/git/dpdk-stable
branch = 19.11

the specific revision we are checking out has the OID
78bc27846101e481438a98c68cac47e4476085c0 which corresponds to the
19.11.8 tag
the following git clone operation succeeds:

git clone --recurse-submodules git@<path_to_repo>.git .

the following git clone operation fails:

git clone --recurse-submodules --depth=50  git@<path_to_repo>.git .

we can work around the problem by doing a shallow clone with the
no-single-branch parameter, as in:

git clone --recurse-submodules --depth=50 --no-single-branch
git@<path_to_repo>.git .

Based on my current level of understanding, Git is automatically
setting the --single-branch flag when cloning submodules but only when
doing a shallow clone of the parent. This seems wrong to me. In
addition, even if Git was going to enforce the single branch option on
submodules it stands to reason that it should be respecting the branch
defined in the .gitmodules file (ie: 19.11 in our case) instead of
using the default branch (ie: "default" in this case). My guess is, if
it were respecting the branch name then we wouldn't be getting the
"unadvertised object" error because the referenced OID would exist in
the submodule checkout because it would have the correct branch
checked out.

For reference, this problem appears to be reproducible on several
different platforms (ie: several different Linux distros and Mac OSX
at least) and several different Git versions up to v2.32.0 which is
the latest version at the moment. Hopefully this problem can be fixed
in a future release of the command line tools. In the meantime we've
had to hack around the problem in our Gitlab CI builds by disabling
the shallow checkout features, which is less than ideal, so again I am
hopeful that this can be fixed directly in Git itself.

Kevin Phillips
Lead Developer, Configuration Management Team, IBM

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-09-04 14:51 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-11 14:59 Bug with Git shallow clones and submodules Kevin Phillips
2021-08-15 14:31 ` Philippe Blain
2021-09-02 13:49   ` Weird interaction between Git protocol versions and git:// and https:// URLs (was: Re: Bug with Git shallow clones and submodules) Philippe Blain
2021-09-04 12:57     ` Jeff King
2021-09-04 14:51       ` Konstantin Ryabitsev

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).