Git Mailing List Archive on lore.kernel.org
 help / color / Atom feed
* Nested submodule checkout
@ 2020-02-14 22:42 Damien Robert
  2020-02-17  4:51 ` Philippe Blain
  0 siblings, 1 reply; 4+ messages in thread
From: Damien Robert @ 2020-02-14 22:42 UTC (permalink / raw)
  To: git

Dear git developers,

I stumbled into this situation from which it was a bit painful to recover:

-> test script:

    mkdir ploum
    cd ploum
    git init
    echo 'foo' > foo
    git add foo
    git commit -m foo
    git branch nosubmodules

    mkdir plam
    cd plam
    git init
    echo 'bar' > bar
    git add bar
    git commit -m bar

    mkdir plim
    cd plim
    git init
    echo 'baz' > baz
    git add baz
    git commit -m baz

    cd ..
    git submodule add ./plim
    git commit -am 'Add submodule plim'

    cd ..
    git submodule add ./plam
    git commit -am 'Add submodule plam'

    git checkout nosubmodules
    git checkout --recurse-submodules master

-> The result is as follow:

Initialized empty Git repository in /data/dams/var/tmp/ploum/.git/
[master (root-commit) ec7c09a] foo
 1 file changed, 1 insertion(+)
 create mode 100644 foo
Branch 'nosubmodules' set up to track local branch 'master'.
Initialized empty Git repository in /data/dams/var/tmp/ploum/plam/.git/
[master (root-commit) 35e6696] bar
 1 file changed, 1 insertion(+)
 create mode 100644 bar
Initialized empty Git repository in /data/dams/var/tmp/ploum/plam/plim/.git/
[master (root-commit) b4712c1] baz
 1 file changed, 1 insertion(+)
 create mode 100644 baz
Adding existing repo at 'plim' to the index
[master 989c11d] Add submodule plim
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 plim
Adding existing repo at 'plam' to the index
[master 5b34041] Add submodule plam
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 plam
Migrating git directory of 'plam' from
'/data/dams/var/tmp/ploum/plam/.git' to
'/data/dams/var/tmp/ploum/.git/modules/plam'
Migrating git directory of 'plam/plim' from
'/data/dams/var/tmp/ploum/plam/plim/.git' to
'/data/dams/var/tmp/ploum/.git/modules/plam/modules/plim'
Switched to branch 'nosubmodules'
Your branch is behind 'master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
fatal: exec '--super-prefix=plam/plim/': cd to 'plim' failed: No such file or directory
error: Submodule 'plim' could not be updated.
error: Submodule 'plam/plim' cannot checkout new HEAD.
error: Submodule 'plam' could not be updated.
M	plam
Switched to branch 'master'

As you can see, the nested plim submodules could not be recreated since the
folder does not exists yet in the 'nosubmodules' branch. This makes the
'plam' submodule update fails, and in the following state

Unstaged changes after reset:
D	.gitmodules
D	bar
D	plim

-> To recover

In the folder plam, do a `git reset` followed by a `git reset --hard`
(`git reset --hard` directly does not work:
fatal: exec '--super-prefix=plim/': cd to 'plim' failed: No such file or directory)

Indeed the first reset, which puts .gitmodules back in the index, is what
allows to do the `git submodule update` implied by `git reset --hard`.
Since (from what I understand) the path is only read from .gitmodules and
not from the .git/config (where there are only the submodules name which
are distinct from the path), this explain the failures observed.

Note that I wasn't able to reproduce in this small examples, but when
trying to repair I also add some strange errors of the form
'.git is not a git directory' (where .git was a pseudo symlink
gitdir: ../.git/modules/plam).

-> Question

My usage is probably non standard (I have quite a lot of nested
submodules), so I had a hard time to recover from this checkout. Is there a
better way? Would it be possible to make nested submodules checkout of this
form work out of the box?

Thanks!
Damien Robert

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Nested submodule checkout
  2020-02-14 22:42 Nested submodule checkout Damien Robert
@ 2020-02-17  4:51 ` Philippe Blain
  2020-02-18 17:08   ` Damien Robert
  0 siblings, 1 reply; 4+ messages in thread
From: Philippe Blain @ 2020-02-17  4:51 UTC (permalink / raw)
  To: Damien Robert; +Cc: git

Hi Damien,

I reported the same bug to the list back in September [1], and I’m glad to say I just finished (today!) a patch series [2] that fixes this bug.

> Le 14 févr. 2020 à 17:42, Damien Robert <damien.olivier.robert@gmail.com> a écrit :
> 
> Dear git developers,
> 
> I stumbled into this situation from which it was a bit painful to recover:
> 
> -> test script:
> 
>    mkdir ploum
>    cd ploum
>    git init
>    echo 'foo' > foo
>    git add foo
>    git commit -m foo
>    git branch nosubmodules
> 
>    mkdir plam
>    cd plam
>    git init
>    echo 'bar' > bar
>    git add bar
>    git commit -m bar
> 
>    mkdir plim
>    cd plim
>    git init
>    echo 'baz' > baz
>    git add baz
>    git commit -m baz
> 
>    cd ..
>    git submodule add ./plim
>    git commit -am 'Add submodule plim'
> 
>    cd ..
>    git submodule add ./plam
>    git commit -am 'Add submodule plam'
> 
>    git checkout nosubmodules
>    git checkout --recurse-submodules master
> 
> -> The result is as follow:
> 
> Initialized empty Git repository in /data/dams/var/tmp/ploum/.git/
> [master (root-commit) ec7c09a] foo
> 1 file changed, 1 insertion(+)
> create mode 100644 foo
> Branch 'nosubmodules' set up to track local branch 'master'.
> Initialized empty Git repository in /data/dams/var/tmp/ploum/plam/.git/
> [master (root-commit) 35e6696] bar
> 1 file changed, 1 insertion(+)
> create mode 100644 bar
> Initialized empty Git repository in /data/dams/var/tmp/ploum/plam/plim/.git/
> [master (root-commit) b4712c1] baz
> 1 file changed, 1 insertion(+)
> create mode 100644 baz
> Adding existing repo at 'plim' to the index
> [master 989c11d] Add submodule plim
> 2 files changed, 4 insertions(+)
> create mode 100644 .gitmodules
> create mode 160000 plim
> Adding existing repo at 'plam' to the index
> [master 5b34041] Add submodule plam
> 2 files changed, 4 insertions(+)
> create mode 100644 .gitmodules
> create mode 160000 plam

Here you just did ` git commit -am 'Add submodule plam’` so the next command according to your reproducer above would be `git checkout nosubmodules`

> Migrating git directory of 'plam' from
> '/data/dams/var/tmp/ploum/plam/.git' to
> '/data/dams/var/tmp/ploum/.git/modules/plam'
> Migrating git directory of 'plam/plim' from
> '/data/dams/var/tmp/ploum/plam/plim/.git' to
> '/data/dams/var/tmp/ploum/.git/modules/plam/modules/plim'
> Switched to branch 'nosubmodules'
> Your branch is behind 'master' by 1 commit, and can be fast-forwarded.
>  (use "git pull" to update your local branch)

Here, git is migrating the git directories of both submodules to the git directory of the superproject (ploum). This tells me you probably have the `submodule.recurse` config set somewhere, as this is the behaviour I get I if I do `git checkout --recurse-submodules nosubmodules`.
If I just do `git checkout nosubmodules`, I get 
    
    $ git checkout nosubmodules 
    warning: unable to rmdir 'plam': Directory not empty
    Switched to branch 'nosubmodules'
    Your branch is behind 'master' by 1 commit, and can be fast-forwarded.
      (use "git pull" to update your local branch)

and then doing `git checkout --recurse-submodules master` actually works. 

> fatal: exec '--super-prefix=plam/plim/': cd to 'plim' failed: No such file or directory
> error: Submodule 'plim' could not be updated.
> error: Submodule 'plam/plim' cannot checkout new HEAD.
> error: Submodule 'plam' could not be updated.
> M	plam
> Switched to branch 'master'
> 
> As you can see, the nested plim submodules could not be recreated since the
> folder does not exists yet in the 'nosubmodules' branch.

That’s the cause of the bug: in fact git tries to change directory into ‘plim’ before the ‘plim’ directory is created in the filesystem.

> This makes the
> 'plam' submodule update fails, and in the following state
> 
> Unstaged changes after reset:
> D	.gitmodules
> D	bar
> D	plim

At this point, if you go into ‘plam’ and do `git ls-files -s` (to list the content of the index), you will see that the index is empty (the checkout died before the index could be populated.) 

> -> To recover
> 
> In the folder plam, do a `git reset` followed by a `git reset --hard`
> (`git reset --hard` directly does not work:
> fatal: exec '--super-prefix=plim/': cd to 'plim' failed: No such file or directory)

That’s another hint that you have `submodule.recurse` set. I don’t get this error doing `git reset --hard`, but I get it doing `git reset --hard --recurse-submodules` (or `git reset --hard --r`, which works and is quicker to type!). `git reset` populates the index, so now `git ls-files -s` would now show the correct content of ‘plam’.

> Indeed the first reset, which puts .gitmodules back in the index, is what
> allows to do the `git submodule update` implied by `git reset --hard`.

In fact, `git reset --hard` does not spawn `git submodule update`, it calls functions in unpack-trees.c (that actually spawn `git read-tree —recurse-submodules`) to update the submodules recursively. 

> Note that I wasn't able to reproduce in this small examples, but when
> trying to repair I also add some strange errors of the form
> '.git is not a git directory' (where .git was a pseudo symlink
> gitdir: ../.git/modules/plam).
> 
> -> Question
> 
> My usage is probably non standard (I have quite a lot of nested
> submodules), so I had a hard time to recover from this checkout. Is there a
> better way? Would it be possible to make nested submodules checkout of this
> form work out of the box?

It is supposed to work out of the box when using `--recurse-submodules` or the `submodule.recurse` config. Although, it’s always possible to run into some bugs. 
One thing you can do to get out of tricky situations is to temporarily deactivate the config (`git -c submodule.recurse=0 <command>`). For example, after the failed `git checkout --recurse-submodules master` above, issuing

    git -c submodule.recurse=0 submodule update --recursive --force

would have correctly checked out the submodules. I have a git alias ‘no-rs’ (for no recurse-submodules) that I use in these situations:

    git config --global alias.no-rs ‘-c submodule.recurse=0’

Then the `submodule update` call above could be shortened to 

    git no-rs submodule update --recursive --force

Note that using the `submodule.recurse` config also applies to internal calls to git commands (issued by other git commands), so using adding `--no-recurse-submodules` to the command line might not be enough to completely turn off the effect of that config, hence this handy alias.

> 
> Thanks!
> Damien Robert
> 

Cheers,
Philippe.

[1] https://lore.kernel.org/git/7437BB59-4605-48EC-B05E-E2BDB2D9DABC@gmail.com/
[2] https://github.com/gitgitgadget/git/pull/555

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Nested submodule checkout
  2020-02-17  4:51 ` Philippe Blain
@ 2020-02-18 17:08   ` Damien Robert
  2020-02-19  4:19     ` Philippe Blain
  0 siblings, 1 reply; 4+ messages in thread
From: Damien Robert @ 2020-02-18 17:08 UTC (permalink / raw)
  To: Philippe Blain; +Cc: git

Hi Philippe,

From Philippe Blain, Sun 16 Feb 2020 at 23:51:43 (-0500) :
> I reported the same bug to the list back in September [1]

Meta question: is there an easy way I could have found your bug report?

> and I’m glad to say I just finished (today!) a patch series [2] that fixes this bug.

Great! Thanks for the patches!

> Here you just did ` git commit -am 'Add submodule plam’` so the next command according to your reproducer above would be `git checkout nosubmodules`
> 
> > Migrating git directory of 'plam' from
> > '/data/dams/var/tmp/ploum/plam/.git' to
> > '/data/dams/var/tmp/ploum/.git/modules/plam'
> > Migrating git directory of 'plam/plim' from
> > '/data/dams/var/tmp/ploum/plam/plim/.git' to
> > '/data/dams/var/tmp/ploum/.git/modules/plam/modules/plim'
> > Switched to branch 'nosubmodules'
> > Your branch is behind 'master' by 1 commit, and can be fast-forwarded.
> >  (use "git pull" to update your local branch)
> 
> Here, git is migrating the git directories of both submodules to the git directory of the superproject (ploum). This tells me you probably have the `submodule.recurse` config set somewhere, as this is the behaviour I get I if I do `git checkout --recurse-submodules nosubmodules`.

Yes, you are of course right. I should have specified it in my bug report,
I tried to specify the setting explicitely in the last checkout but as you
noticed I forgot to specify it in all other commands, sorry about the
noise.

> That’s another hint that you have `submodule.recurse` set. I don’t get this error doing `git reset --hard`, but I get it doing `git reset --hard --recurse-submodules` (or `git reset --hard --r`, which works and is quicker to type!). `git reset` populates the index, so now `git ls-files -s` would now show the correct content of ‘plam’.

Oh, I did not know git expand unambiguous long options!

By the way the fact that `git reset` also support `--recurse-submodules` is not
specified in the man page. (It is in the help text thought).

And it would be nice if the documentation of submodule.recurse in
git-config specify the list of all affected commands, rather than just "all
commands that have a --recurse-submodules options".

(I could send a patch for this if there is interest)

> > Note that I wasn't able to reproduce in this small examples, but when
> > trying to repair I also add some strange errors of the form
> > '.git is not a git directory' (where .git was a pseudo symlink
> > gitdir: ../.git/modules/plam).

-> This is explained by your Patch 5/6

> would have correctly checked out the submodules. I have a git alias ‘no-rs’ (for no recurse-submodules) that I use in these situations:
>     git config --global alias.no-rs ‘-c submodule.recurse=0’

Can you use alias to define option settings (rather than commands followed
by options) without using the 'f() {}' trick?

Using your alias, I get
fatal: empty alias for no-rs

> Then the `submodule update` call above could be shortened to 
>     git no-rs submodule update --recursive --force

> Note that using the `submodule.recurse` config also applies to internal calls to git commands (issued by other git commands), so using adding `--no-recurse-submodules` to the command line might not be enough to completely turn off the effect of that config, hence this handy alias.

Ohh, good to know!

-- 
Damien Robert
http://www.normalesup.org/~robert/pro

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Nested submodule checkout
  2020-02-18 17:08   ` Damien Robert
@ 2020-02-19  4:19     ` Philippe Blain
  0 siblings, 0 replies; 4+ messages in thread
From: Philippe Blain @ 2020-02-19  4:19 UTC (permalink / raw)
  To: Damien Robert; +Cc: git

Hi Damien,

> Le 18 févr. 2020 à 12:08, Damien Robert <damien.olivier.robert@gmail.com> a écrit :
> 
> Hi Philippe,
> 
> From Philippe Blain, Sun 16 Feb 2020 at 23:51:43 (-0500) :
>> I reported the same bug to the list back in September [1]
> 
> Meta question: is there an easy way I could have found your bug report?

That’s a good question. The search capabilities of public-inbox (the software running at lore.kernel.org) are explained at 
https://lore.kernel.org/git/_/text/help/

In this case, my bug report comes out in the following searches:
b:(nested submodule checkout)
s:(bug* submodule* checkout)

the first one can be refined with
b:(nested submodule checkout) NOT s:(cooking) NOT s:(ANNOUNCE) NOT s:(PATCH*)

> 
>> That’s another hint that you have `submodule.recurse` set. I don’t get this error doing `git reset --hard`, but I get it doing `git reset --hard --recurse-submodules` (or `git reset --hard --r`, which works and is quicker to type!). `git reset` populates the index, so now `git ls-files -s` would now show the correct content of ‘plam’.
> 
> Oh, I did not know git expand unambiguous long options!

Yes, it’s not widely known but can be useful! It’s documented in gitcli(7) (`man gitcli`) or online at
https://git-scm.com/docs/gitcli#_abbreviating_long_options

> By the way the fact that `git reset` also support `--recurse-submodules` is not
> specified in the man page. (It is in the help text thought).

Yes, this is on my radar, I’d like to get some time to clean this up sometimes soon. It’s also missing for `git restore`.

> And it would be nice if the documentation of submodule.recurse in
> git-config specify the list of all affected commands, rather than just "all
> commands that have a --recurse-submodules options".
> 
> (I could send a patch for this if there is interest)

That’s a very good idea in my opinion, especially since the search functionality on git-scm.com is presently limited to 10 hits from the manpages so it fails to find all occurrences of `--recurse-submodules` [1].

[1] https://github.com/git/git-scm.com/issues/1374

>> would have correctly checked out the submodules. I have a git alias ‘no-rs’ (for no recurse-submodules) that I use in these situations:
>>    git config --global alias.no-rs ‘-c submodule.recurse=0’
> 
> Can you use alias to define option settings (rather than commands followed
> by options) without using the 'f() {}' trick?
> 
> Using your alias, I get
> fatal: empty alias for no-rs

The alias specifies an option directly to the git "main" executable instead of a specific Git command (in this case setting a particular configuration variable for the duration of the command that will follow). This alias works for me, at least on Git 2.25. The git-config man page mentions aliases can start with an option to git [2]. If it does not work in an older version of Git then you can use 

    git config --global alias.no-rs ‘!git -c submodule.recurse=0'

which should do the trick.

[2] https://git-scm.com/docs/git-config#Documentation/git-config.txt-alias

>> Note that using the `submodule.recurse` config also applies to internal calls to git commands (issued by other git commands), so using adding `--no-recurse-submodules` to the command line might not be enough to completely turn off the effect of that config, hence this handy alias.
> 
> Ohh, good to know!

If you do go ahead with a patch listing the commands affected by submodule.recurse in git-config, then I think this should also be mentioned. 

Cheers,
Philippe.

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, back to index

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-14 22:42 Nested submodule checkout Damien Robert
2020-02-17  4:51 ` Philippe Blain
2020-02-18 17:08   ` Damien Robert
2020-02-19  4:19     ` Philippe Blain

Git Mailing List Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/git/0 git/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 git git/ https://lore.kernel.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.git


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git