Re: nested submodules detection w/o .gitmodules file

From: Emily Shaffer <nasamuffin@google.com>
To: Andry <andry@inbox.ru>
Cc: git@vger.kernel.org
Subject: Re: nested submodules detection w/o .gitmodules file
Date: Mon, 23 Jan 2023 12:12:32 -0800	[thread overview]
Message-ID: <CAJoAoZkjfOQwkeQqzQY5qDo7Md5QWSz1pOTBQKHL5KwNu2VoDg@mail.gmail.com> (raw)
In-Reply-To: <1716310675.20230122233403@inbox.ru>

On Sun, Jan 22, 2023 at 12:45 PM Andry <andry@inbox.ru> wrote:
>
> Hello Git,
>
> I have a pretty long investigation has been started from usage 3dparty projects related directly or indirectly to the git submodules:
>
> `svn externals replacement` : https://github.com/chronoxor/gil/issues/6
> `svn complete replacement for externals` : https://github.com/dirk-thomas/vcstool/issues/243
>
> And stumbled on this discussion:
>
> `nested submodules detection w/o .gitmodules file` : https://github.com/gitextensions/gitextensions/discussions/10644 (https://github.com/gitextensions/gitextensions/issues/10642)
>
> The main question here is that, could the git have has submodules without `.submodules` file?

There is a little nuance here. Git can have nested repositories in a
few different ways; submodules are just one of them. A submodule is
the combination of a gitlink object in the object graph *and* a
corresponding entry in the .gitmodules file. It's certainly possible
to embed a nested repository in other ways, such as by ignoring the
.gitmodules file, but then your nested repository is no longer a
submodule, and operations which recurse over submodules will not
consider that nested repository. The reason is that we need to
understand where to clone the submodule from - that information isn't
contained in the superproject's repository URL, and it can't be
contained in the gitlink directly (which is in essence just a commit
object, but one that exists in the nested repository, not in the
superproject repository). If we didn't have a way of writing down the
submodule's remote URL and version controlling it, we wouldn't have a
way to populate the submodules when a user is cloning.

>
> If no, then all side projects which utilizes it's own input file for the externals may subsequentially fail:
>
> https://github.com/chronoxor/gil
> https://github.com/dirk-thomas/vcstool
> https://github.com/ingydotnet/git-subrepo
>
> If yes, then other projects which does rely on the `.submodules` would have not actual or even invalid state:
>
> https://github.com/gitextensions/gitextensions
>
> Or even the github itself:
>
> `Zip archive to include submodule` : https://github.com/dear-github/dear-github/issues/214
>
> (`[PATCH] archive: add –recurse-submodules to git-archive command` : https://git.github.io/rev_news/2022/11/30/edition-93/, https://lore.kernel.org/git/pull.1359.git.git.1665597148042.gitgitgadget@gmail.com/)
>
> Mine point here is that:
>
> Git database is a primary storage. The `.gitmodules` file is not a primary storage, so can be in not an actual or desync state with the database.
> And any application or a 3dparty project must read the database directly.

The .gitmodules exists to help at clone time; it's possible, as I
think you're pointing out, to have some intermediate state locally.
But this file is what needs to be the source of truth for putting
together the repository on a new machine for the first time.

>
> But another problem here is that the git still does not have a stable API for that.
> For example, a submodule can be declared directly from the `.git/config` file in a working copy:
>
> https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-init--ltpathgt82308203
> https://git-scm.com/docs/gitsubmodules#_active_submodules
>
> So, who is right and what is wrong here?
>