gitconfig get out of sync with submodule entries on branch switch

gitconfig get out of sync with submodule entries on branch switch

[Benjamin Schindler]

Hi

Consider the following usecase: I have the master branch where I have a 
submodule A. I create a branch where I rename the submodule to be in the 
directory B. After doing all of this, everything looks good.
Now, I switch back to master. The first oddity is, that it fails to 
remove the folder B because there are still files in there:

bschindler@metis ~/Projects/submodule_test (testbranch) $ git checkout 
master
warning: unable to rmdir other_submodule: Directory not empty
Switched to branch 'master'

Git submodule deinit on B fails because the submodule is not known to 
git anymore (after all, the folder B exists only in the other branch). I 
can easily just remove the folder B from disk and initialize the 
submodule A again, so all seems good.

However, what is not good is that the submodule b is still known in 
.git/config. This is in particular a problem for us, because I know a 
number of tools which use git config to retrieve the submodule list. Is 
it therefore a bug that upon branch switch, the submodule gets 
deregistered, but its entry in .git/config remains?

thanks a lot
Benjamin Schindler

P.s. I did not subscribe to the mailing list, please add me at least do 
CC. Thanks

Re: gitconfig get out of sync with submodule entries on branch switch

[Brandon Williams]

On 01/30, Benjamin Schindler wrote:
> Hi
> 
> Consider the following usecase: I have the master branch where I
> have a submodule A. I create a branch where I rename the submodule
> to be in the directory B. After doing all of this, everything looks
> good.
> Now, I switch back to master. The first oddity is, that it fails to
> remove the folder B because there are still files in there:
> 
> bschindler@metis ~/Projects/submodule_test (testbranch) $ git
> checkout master
> warning: unable to rmdir other_submodule: Directory not empty
> Switched to branch 'master'
> 
> Git submodule deinit on B fails because the submodule is not known
> to git anymore (after all, the folder B exists only in the other
> branch). I can easily just remove the folder B from disk and
> initialize the submodule A again, so all seems good.
> 
> However, what is not good is that the submodule b is still known in
> .git/config. This is in particular a problem for us, because I know
> a number of tools which use git config to retrieve the submodule
> list. Is it therefore a bug that upon branch switch, the submodule
> gets deregistered, but its entry in .git/config remains?
> 
> thanks a lot
> Benjamin Schindler
> 
> P.s. I did not subscribe to the mailing list, please add me at least
> do CC. Thanks

submodules and checkout don't really play nicely with each other at the
moment.  Stefan (cc'd) is currently working on a patch series to improve
the behavior of checkout with submodules.  Currently, if you want to
ensure you have a good working state after a checkout you should run
`git submodule update` to update all of the submoules.  As far as your
submodule still being listed in the config, that should be expected
given the scenario you described.

If I'm understanding you correctly, A and B are both the same submodule
just renamed on a different branch.  The moment you add a submoule to a
repository it is given a name which is fixed.  Typically this is the
path from the root of the repository.  The thing is, since you are able
to freely move a submodule, its path can change.  To account for this
there is the .gitmodules file which allows you to do a lookup from
submodule name to the path at which it exists (or vice versa).  The
submodules that are stored in .git/config are those which are
'initialize' or rather the submodules in which you are interested in and
will be updated by `git submodule update`.  So given your scenario you
should only have a single submodule in .git/config and the .gitmodules
file should have a single entry with a differing path for each branch.

Hopefully this gives you a bit more information to work with.  Since
Stefan has been working with this more recently than me he may have some
more input.

-- 
Brandon Williams

Re: gitconfig get out of sync with submodule entries on branch switch

[Benjamin Schindler]

Hi Brandon

I did try your suggestion, so basically:

git checkout branch
git submodule init
git submodule update

Unfortunately, I still have two entries in my git config this way. It 
seems that git submodule update only considers submodules listed in 
.gitmodules.

The background of my question is this - we have a jenkins farm which 
needs to switch branches continuously in an automated fashion and this 
needs to work even in when submodules are around. I did however, just 
now, find a reliable way to switch a branch, keeping the gitconfig in sync:
The basic workflow for switching a branch is:
git submodule deinit .
git checkout branch
git submodule init
git submodule update

Because the .git folder of the submodules are not within the submodule 
directories, this is, while still quite heavy-handed, reasonably fast 
and robust. At least it is better than deleting the entire repository 
every time a branch switch is issued.

Regards

Benjamin Schindler

On 30.01.2017 18:51, Brandon Williams wrote:
> On 01/30, Benjamin Schindler wrote:
>> Hi
>>
>> Consider the following usecase: I have the master branch where I
>> have a submodule A. I create a branch where I rename the submodule
>> to be in the directory B. After doing all of this, everything looks
>> good.
>> Now, I switch back to master. The first oddity is, that it fails to
>> remove the folder B because there are still files in there:
>>
>> bschindler@metis ~/Projects/submodule_test (testbranch) $ git
>> checkout master
>> warning: unable to rmdir other_submodule: Directory not empty
>> Switched to branch 'master'
>>
>> Git submodule deinit on B fails because the submodule is not known
>> to git anymore (after all, the folder B exists only in the other
>> branch). I can easily just remove the folder B from disk and
>> initialize the submodule A again, so all seems good.
>>
>> However, what is not good is that the submodule b is still known in
>> .git/config. This is in particular a problem for us, because I know
>> a number of tools which use git config to retrieve the submodule
>> list. Is it therefore a bug that upon branch switch, the submodule
>> gets deregistered, but its entry in .git/config remains?
>>
>> thanks a lot
>> Benjamin Schindler
>>
>> P.s. I did not subscribe to the mailing list, please add me at least
>> do CC. Thanks
> submodules and checkout don't really play nicely with each other at the
> moment.  Stefan (cc'd) is currently working on a patch series to improve
> the behavior of checkout with submodules.  Currently, if you want to
> ensure you have a good working state after a checkout you should run
> `git submodule update` to update all of the submoules.  As far as your
> submodule still being listed in the config, that should be expected
> given the scenario you described.
>
> If I'm understanding you correctly, A and B are both the same submodule
> just renamed on a different branch.  The moment you add a submoule to a
> repository it is given a name which is fixed.  Typically this is the
> path from the root of the repository.  The thing is, since you are able
> to freely move a submodule, its path can change.  To account for this
> there is the .gitmodules file which allows you to do a lookup from
> submodule name to the path at which it exists (or vice versa).  The
> submodules that are stored in .git/config are those which are
> 'initialize' or rather the submodules in which you are interested in and
> will be updated by `git submodule update`.  So given your scenario you
> should only have a single submodule in .git/config and the .gitmodules
> file should have a single entry with a differing path for each branch.
>
> Hopefully this gives you a bit more information to work with.  Since
> Stefan has been working with this more recently than me he may have some
> more input.
>

Re: gitconfig get out of sync with submodule entries on branch switch

[Stefan Beller]

On Mon, Jan 30, 2017 at 11:46 PM, Benjamin Schindler
<beschindler@gmail.com> wrote:
> Hi Brandon
>
> I did try your suggestion, so basically:
>
> git checkout branch
> git submodule init
> git submodule update

Eventually this becomes

    git submudule update --init
    git checkout --recurse-submodules $branch


>
> Unfortunately, I still have two entries in my git config this way. It seems
> that git submodule update only considers submodules listed in .gitmodules.

Did you rename the name in the gitmodules file or rename the path on the FS?

>
> The background of my question is this - we have a jenkins farm which needs
> to switch branches continuously in an automated fashion and this needs to
> work even in when submodules are around. I did however, just now, find a
> reliable way to switch a branch, keeping the gitconfig in sync:
> The basic workflow for switching a branch is:
> git submodule deinit .

This will become 'git submodule deinit --all' eventually

> git checkout branch
> git submodule init
> git submodule update

This ought to update all the submodules, sounds fine to me.

>
> Because the .git folder of the submodules are not within the submodule
> directories, this is, while still quite heavy-handed, reasonably fast and
> robust. At least it is better than deleting the entire repository every time
> a branch switch is issued.

In the next version there will be 'git submodule absorbgitdirs'
which moves the git dirs of submodules inside the superproject; would
a reversion of this also be useful?

>
> Regards
>
> Benjamin Schindler
>
>
> On 30.01.2017 18:51, Brandon Williams wrote:
>>
>> On 01/30, Benjamin Schindler wrote:
>>>
>>> Hi
>>>
>>> Consider the following usecase: I have the master branch where I
>>> have a submodule A. I create a branch where I rename the submodule
>>> to be in the directory B. After doing all of this, everything looks
>>> good.
>>> Now, I switch back to master. The first oddity is, that it fails to
>>> remove the folder B because there are still files in there:
>>>
>>> bschindler@metis ~/Projects/submodule_test (testbranch) $ git
>>> checkout master
>>> warning: unable to rmdir other_submodule: Directory not empty
>>> Switched to branch 'master'
>>>
>>> Git submodule deinit on B fails because the submodule is not known
>>> to git anymore (after all, the folder B exists only in the other
>>> branch). I can easily just remove the folder B from disk and
>>> initialize the submodule A again, so all seems good.
>>>
>>> However, what is not good is that the submodule b is still known in
>>> .git/config. This is in particular a problem for us, because I know
>>> a number of tools which use git config to retrieve the submodule
>>> list. Is it therefore a bug that upon branch switch, the submodule
>>> gets deregistered, but its entry in .git/config remains?
>>>
>>> thanks a lot
>>> Benjamin Schindler
>>>
>>> P.s. I did not subscribe to the mailing list, please add me at least
>>> do CC. Thanks
>>
>> submodules and checkout don't really play nicely with each other at the
>> moment.  Stefan (cc'd) is currently working on a patch series to improve
>> the behavior of checkout with submodules.  Currently, if you want to
>> ensure you have a good working state after a checkout you should run
>> `git submodule update` to update all of the submoules.  As far as your
>> submodule still being listed in the config, that should be expected
>> given the scenario you described.
>>
>> If I'm understanding you correctly, A and B are both the same submodule
>> just renamed on a different branch.  The moment you add a submoule to a
>> repository it is given a name which is fixed.  Typically this is the
>> path from the root of the repository.  The thing is, since you are able
>> to freely move a submodule, its path can change.  To account for this
>> there is the .gitmodules file which allows you to do a lookup from
>> submodule name to the path at which it exists (or vice versa).  The
>> submodules that are stored in .git/config are those which are
>> 'initialize' or rather the submodules in which you are interested in and
>> will be updated by `git submodule update`.  So given your scenario you
>> should only have a single submodule in .git/config and the .gitmodules
>> file should have a single entry with a differing path for each branch.
>>
>> Hopefully this gives you a bit more information to work with.  Since
>> Stefan has been working with this more recently than me he may have some
>> more input.
>>
>

Re: gitconfig get out of sync with submodule entries on branch switch

[Stefan Beller]

Answering the original email, as I feel we're going down the wrong rabbit
hole in the existing thread.

On Mon, Jan 30, 2017 at 8:21 AM, Benjamin Schindler
<beschindler@gmail.com> wrote:
> Hi
>
> Consider the following usecase: I have the master branch where I have a
> submodule A. I create a branch where I rename the submodule to be in the
> directory B. After doing all of this, everything looks good.
> Now, I switch back to master. The first oddity is, that it fails to remove
> the folder B because there are still files in there:
>
> bschindler@metis ~/Projects/submodule_test (testbranch) $ git checkout
> master
> warning: unable to rmdir other_submodule: Directory not empty
> Switched to branch 'master'

checkout currently doesn't support submodules, so it should neither
try to delete B nor try to repopulate A when switching back to master.
checkout ought to not even touch the existing submodule B.

>
> Git submodule deinit on B fails because the submodule is not known to git
> anymore (after all, the folder B exists only in the other branch). I can
> easily just remove the folder B from disk and initialize the submodule A
> again, so all seems good.

by initializing you mean populating(?), i.e.

    git submodule update

would work without the --init flag or preceding "git submodule init A".
That ought to not redownload A, but just put files back in the working tree
from the submodule git directory inside the superprojects git dir.

>
> However, what is not good is that the submodule b is still known in
> .git/config.

Oh, I see. You did not just rename the path, but also the name
in the .gitmodules?

> This is in particular a problem for us, because I know a number
> of tools which use git config to retrieve the submodule list. Is it
> therefore a bug that upon branch switch, the submodule gets deregistered,
> but its entry in .git/config remains?

The config remains as it indicates that you express(ed) interest in
submodule A, such that when switching branches

  master->renamedToB->master

then we'd still care about A. As for the tools, I'd rather see them use

    git submodule status/summary

instead of directly looking at the config, because the config may
change in the future.

>
> thanks a lot
> Benjamin Schindler
>
> P.s. I did not subscribe to the mailing list, please add me at least do CC.
> Thanks

Re: gitconfig get out of sync with submodule entries on branch switch

[Benjamin Schindler]

On 06.02.2017 11:35, Stefan Beller wrote:
> Answering the original email, as I feel we're going down the wrong rabbit
> hole in the existing thread.
>
> On Mon, Jan 30, 2017 at 8:21 AM, Benjamin Schindler
> <beschindler@gmail.com> wrote:
>> Hi
>>
>> Consider the following usecase: I have the master branch where I have a
>> submodule A. I create a branch where I rename the submodule to be in the
>> directory B. After doing all of this, everything looks good.
>> Now, I switch back to master. The first oddity is, that it fails to remove
>> the folder B because there are still files in there:
>>
>> bschindler@metis ~/Projects/submodule_test (testbranch) $ git checkout
>> master
>> warning: unable to rmdir other_submodule: Directory not empty
>> Switched to branch 'master'
>
> checkout currently doesn't support submodules, so it should neither
> try to delete B nor try to repopulate A when switching back to master.
> checkout ought to not even touch the existing submodule B.

Well, it tried to remove the folder (the rmdir warning) but it failed so 
in some sense you are right. Is there a technical reason for this 
default though? Here, I frequently have to point out to people that they 
need to initialize/update the submodule on e.g. clone.

>
>>
>> Git submodule deinit on B fails because the submodule is not known to git
>> anymore (after all, the folder B exists only in the other branch). I can
>> easily just remove the folder B from disk and initialize the submodule A
>> again, so all seems good.
>
> by initializing you mean populating(?), i.e.
>
>     git submodule update
>
> would work without the --init flag or preceding "git submodule init A".
> That ought to not redownload A, but just put files back in the working tree
> from the submodule git directory inside the superprojects git dir.
>
>>
>> However, what is not good is that the submodule b is still known in
>> .git/config.
>
> Oh, I see. You did not just rename the path, but also the name
> in the .gitmodules?

I wasn't even aware that the submodule name was something different from 
the path because the name is by default set to be the path to it. So 
yes, I didn't just relocate it, it had a different name.

>
>> This is in particular a problem for us, because I know a number
>> of tools which use git config to retrieve the submodule list. Is it
>> therefore a bug that upon branch switch, the submodule gets deregistered,
>> but its entry in .git/config remains?
>
> The config remains as it indicates that you express(ed) interest in
> submodule A, such that when switching branches
>
>   master->renamedToB->master
>
> then we'd still care about A. As for the tools, I'd rather see them use
>
>     git submodule status/summary
>
> instead of directly looking at the config, because the config may
> change in the future.

That was my feeling but its good to know to have more solid reasons why 
that would be.

Cheers
Benjamin

>
>>
>> thanks a lot
>> Benjamin Schindler
>>
>> P.s. I did not subscribe to the mailing list, please add me at least do CC.
>> Thanks

Re: gitconfig get out of sync with submodule entries on branch switch

[Stefan Beller]

On Mon, Feb 6, 2017 at 4:17 AM, Benjamin Schindler
<beschindler@gmail.com> wrote:
>
>
> On 06.02.2017 11:35, Stefan Beller wrote:
>>
>> Answering the original email, as I feel we're going down the wrong rabbit
>> hole in the existing thread.
>>
>> On Mon, Jan 30, 2017 at 8:21 AM, Benjamin Schindler
>> <beschindler@gmail.com> wrote:
>>>
>>> Hi
>>>
>>> Consider the following usecase: I have the master branch where I have a
>>> submodule A. I create a branch where I rename the submodule to be in the
>>> directory B. After doing all of this, everything looks good.
>>> Now, I switch back to master. The first oddity is, that it fails to
>>> remove
>>> the folder B because there are still files in there:
>>>
>>> bschindler@metis ~/Projects/submodule_test (testbranch) $ git checkout
>>> master
>>> warning: unable to rmdir other_submodule: Directory not empty
>>> Switched to branch 'master'
>>
>>
>> checkout currently doesn't support submodules, so it should neither
>> try to delete B nor try to repopulate A when switching back to master.
>> checkout ought to not even touch the existing submodule B.
>
>
> Well, it tried to remove the folder (the rmdir warning) but it failed so in
> some sense you are right. Is there a technical reason for this default
> though? Here, I frequently have to point out to people that they need to
> initialize/update the submodule on e.g. clone.

The reasoning is more based on historical momentum.
Back then when gitlinks/submodules were invented (git repositories
inside other git repositories! How frickin' cool is that?) nobody quite knew
how they were going to be used eventually.  So the safe play at the time was
to not touch them at all. (Also easy to implement, but that was not the point
as I learned).

And now we have a lot of people expecting just that. So we cannot change
the behavior overnight. So we'd first implement the alternative (e.g. the
--recurse-submodules flag for checkout) and then once we do a major
release we may want to flip the default.

>
>>
>>>
>>> Git submodule deinit on B fails because the submodule is not known to git
>>> anymore (after all, the folder B exists only in the other branch). I can
>>> easily just remove the folder B from disk and initialize the submodule A
>>> again, so all seems good.
>>
>>
>> by initializing you mean populating(?), i.e.
>>
>>     git submodule update
>>
>> would work without the --init flag or preceding "git submodule init A".
>> That ought to not redownload A, but just put files back in the working
>> tree
>> from the submodule git directory inside the superprojects git dir.
>>
>>>
>>> However, what is not good is that the submodule b is still known in
>>> .git/config.
>>
>>
>> Oh, I see. You did not just rename the path, but also the name
>> in the .gitmodules?
>
>
> I wasn't even aware that the submodule name was something different from the
> path because the name is by default set to be the path to it. So yes, I
> didn't just relocate it, it had a different name.

Yeah the path/name is tricky and usually only differs when you move the
submodule inside the working tree. (As the name stays constant we know
where the git directory is expected: .git/modules<name> so we can check there
instead of re-cloning to the "new" submodule-path.)

>
>>
>>> This is in particular a problem for us, because I know a number
>>> of tools which use git config to retrieve the submodule list. Is it
>>> therefore a bug that upon branch switch, the submodule gets deregistered,
>>> but its entry in .git/config remains?
>>
>>
>> The config remains as it indicates that you express(ed) interest in
>> submodule A, such that when switching branches
>>
>>   master->renamedToB->master
>>
>> then we'd still care about A. As for the tools, I'd rather see them use
>>
>>     git submodule status/summary
>>
>> instead of directly looking at the config, because the config may
>> change in the future.
>
>
> That was my feeling but its good to know to have more solid reasons why that
> would be.

The reasons here are backward/forward compatibility.
When trying to change the behavior of submodule related things, there is no
clear distinction of plumbing/porcelain as we have in the rest of Git.
So even in gits own test suite we sometimes rely on things that may change
later on, and that makes it very hard to move forward, which is why I try to
have an opinion on how to do things properly.

Thanks,
Stefan