Nesting a submodule inside of another...

Nesting a submodule inside of another...

[John Szakmeister]

Hi all,

I've got a project where we have several frameworks involved, and
external modules we want to pull into the framework tree.  We'd like
to make use of submodules and have something like this:
    top-level/            <-- .gitmodules lives here
        src/
        framework1/   <-- a submodule
            module/     <-- another submodule
        framework2/   <-- a submodule
            module2/    <-- another submodule

Currently, git fails trying to do this.  It's not happy about
.gitmodules living at the top-level and nesting a submodule inside of
another[1].  Is there a technical reason that this is not allowed?
Limiting the traversal up the tree, etc.?  I've worked around the lack
of support to do this for now, but it would be really nice if we could
do such a thing.

Thanks!

-John

[1]  I may generate a patch for the error message, as the current
implementation obscures the real error message.  When I try this, git
says:
    The following path is ignored by one of your .gitignore files:
    framework1/module
    Use -f if you really want to add it.

    After some careful debugging, I found out the real error message
was about this sort of nesting being disallowed.

Re: Nesting a submodule inside of another...

[Jens Lehmann]

Am 12.07.2011 14:01, schrieb John Szakmeister:
> I've got a project where we have several frameworks involved, and
> external modules we want to pull into the framework tree.  We'd like
> to make use of submodules and have something like this:
>     top-level/            <-- .gitmodules lives here
>         src/
>         framework1/   <-- a submodule
>             module/     <-- another submodule
>         framework2/   <-- a submodule
>             module2/    <-- another submodule
> 
> Currently, git fails trying to do this.  It's not happy about
> .gitmodules living at the top-level and nesting a submodule inside of
> another[1].  Is there a technical reason that this is not allowed?

Submodules are repositories of their own, so their work tree is outside
the superproject's work tree. You're attempting to have a submodule of
the superproject inside another submodule of the superproject, but this
won't work because it does not live in the superproject's work tree (as
that doesn't include submodules and their subdirectories).

To have nested submodules you must add "module" as a submodule of
"framework1" and "module2" as a submodule of "framework2". Then git
will handle them just fine (assuming you use the --recursive option for
"git submodule update" and friends). Of course that means that if you
want to record a new commit of a sub-submodule in the superpoject, you'll
have to commit that in it's parent submodule first and then can record
that commit in the superproject. (But if "framework1" wouldn't depend on
a specific version of "module", there would be no reason to put it inside
it in the first place, right?)

> Limiting the traversal up the tree, etc.?  I've worked around the lack
> of support to do this for now, but it would be really nice if we could
> do such a thing.

I can't think of a sane way to make that work.

> [1]  I may generate a patch for the error message, as the current
> implementation obscures the real error message.  When I try this, git
> says:
>     The following path is ignored by one of your .gitignore files:
>     framework1/module
>     Use -f if you really want to add it.
> 
>     After some careful debugging, I found out the real error message
> was about this sort of nesting being disallowed.

But at least since 1.6.2 git should issue a meaningful error message.
With current master I get:

$ git add sub/file
fatal: Path 'sub/file' is in submodule 'sub'

What version are you using?

Re: Nesting a submodule inside of another...

[John Szakmeister]

On Tue, Jul 12, 2011 at 3:23 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 12.07.2011 14:01, schrieb John Szakmeister:
>> I've got a project where we have several frameworks involved, and
>> external modules we want to pull into the framework tree.  We'd like
>> to make use of submodules and have something like this:
>>     top-level/            <-- .gitmodules lives here
>>         src/
>>         framework1/   <-- a submodule
>>             module/     <-- another submodule
>>         framework2/   <-- a submodule
>>             module2/    <-- another submodule
>>
>> Currently, git fails trying to do this.  It's not happy about
>> .gitmodules living at the top-level and nesting a submodule inside of
>> another[1].  Is there a technical reason that this is not allowed?
>
> Submodules are repositories of their own, so their work tree is outside
> the superproject's work tree. You're attempting to have a submodule of
> the superproject inside another submodule of the superproject, but this
> won't work because it does not live in the superproject's work tree (as
> that doesn't include submodules and their subdirectories).

I guess I still don't see how that's a problem.  It seems to me that
for the superproject the ignore rule needs to be:
  ignore framework1, except framework1/module

And then it could check its status.  Of course, framework1 would
somehow need to know to ignore it... and I'm not sure how that can be
communicated.

> To have nested submodules you must add "module" as a submodule of
> "framework1" and "module2" as a submodule of "framework2". Then git
> will handle them just fine (assuming you use the --recursive option for
> "git submodule update" and friends). Of course that means that if you
> want to record a new commit of a sub-submodule in the superpoject, you'll
> have to commit that in it's parent submodule first and then can record
> that commit in the superproject. (But if "framework1" wouldn't depend on
> a specific version of "module", there would be no reason to put it inside
> it in the first place, right?)

We can't do that.  The base framework should not include the module in
its repository.  We drop in different modules that get shared across
some projects, but not others.  They get mixed and matched.  This was
pretty straight-forward to do with Subversion.  Admittedly, I don't
like the layout of it... but unfortunately, I'm not really in control
of that either.

>> Limiting the traversal up the tree, etc.?  I've worked around the lack
>> of support to do this for now, but it would be really nice if we could
>> do such a thing.
>
> I can't think of a sane way to make that work.

:-(

[snip]
> But at least since 1.6.2 git should issue a meaningful error message.
> With current master I get:
>
> $ git add sub/file
> fatal: Path 'sub/file' is in submodule 'sub'

Sorry, I wasn't trying to do a 'git add'.  I was trying to do a 'git
submodule add'.

> What version are you using?

I'm using 1.7.6rc3.  I updated to the latest trunk and still get the same error:

:: git submodule add $PWD/../c b/c
The following path is ignored by one of your .gitignore files:
b/c
Use -f if you really want to add it.

HTH.

-John

Re: Nesting a submodule inside of another...

[Andreas Ericsson]

On 07/12/2011 02:01 PM, John Szakmeister wrote:
> Hi all,
> 
> I've got a project where we have several frameworks involved, and
> external modules we want to pull into the framework tree.  We'd like
> to make use of submodules and have something like this:
>      top-level/<-- .gitmodules lives here
>          src/
>          framework1/<-- a submodule
>              module/<-- another submodule
>          framework2/<-- a submodule
>              module2/<-- another submodule
> 
> Currently, git fails trying to do this.  It's not happy about
> .gitmodules living at the top-level and nesting a submodule inside of
> another[1].  Is there a technical reason that this is not allowed?

Yes. Everything inside a submodule is owned by that submodule, so the
master repo can't know anything about it. You can have a submodule
which in turn has submodules though. If you couldn't, it wouldn't be
possible to checkout only the framework1 repository and get all its
dependencies.

I have no idea what problems you run into with more than 2 tiers of
submodules though, but I guess that's for you to find out and report
about.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Nesting a submodule inside of another...

[Jens Lehmann]

Am 13.07.2011 03:21, schrieb John Szakmeister:
> On Tue, Jul 12, 2011 at 3:23 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> Submodules are repositories of their own, so their work tree is outside
>> the superproject's work tree. You're attempting to have a submodule of
>> the superproject inside another submodule of the superproject, but this
>> won't work because it does not live in the superproject's work tree (as
>> that doesn't include submodules and their subdirectories).
> 
> I guess I still don't see how that's a problem.  It seems to me that
> for the superproject the ignore rule needs to be:
>   ignore framework1, except framework1/module

But that would mean the superproject would have to scan framework1's
work tree for changes too, which is conflicting with the idea that each
submodule is a repo of it's own, so that won't fly.

> And then it could check its status.  Of course, framework1 would
> somehow need to know to ignore it... and I'm not sure how that can be
> communicated.

It can't be communicated, as submodules are unaware of their superproject.
And adding module manually to framework1's .gitignore is just a problem
waiting to happen. Having a file or submodule in the work tree of one
git repository being referenced by another is just asking for trouble ...

>> But at least since 1.6.2 git should issue a meaningful error message.
>> With current master I get:
>>
>> $ git add sub/file
>> fatal: Path 'sub/file' is in submodule 'sub'
> 
> Sorry, I wasn't trying to do a 'git add'.  I was trying to do a 'git
> submodule add'.

Ok.

>> What version are you using?
> 
> I'm using 1.7.6rc3.  I updated to the latest trunk and still get the same error:
> 
> :: git submodule add $PWD/../c b/c
> The following path is ignored by one of your .gitignore files:
> b/c
> Use -f if you really want to add it.

You're right, the error message of "git submodule add" should be better
here.

Re: Nesting a submodule inside of another...

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> It can't be communicated, as submodules are unaware of their superproject.

It's more like s/can't/shouldn't/ and s/are/shouldn't be/, no?

Re: Nesting a submodule inside of another...

[Jens Lehmann]

Am 13.07.2011 23:27, schrieb Junio C Hamano:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> It can't be communicated, as submodules are unaware of their superproject.
> 
> It's more like s/can't/shouldn't/ and s/are/shouldn't be/, no?

Right. Submodules /could/ be taught to check if they are part of a
superproject, but unless I miss something obvious they are currently
totally unaware of that (at least when it concerns their work tree, the
gitfile effort is trying to introduce some superproject awareness). And as
far as I can see that is a sound design decision as far as it concerns the
work tree. Imagine the files and/or directories a submodule has to ignore
would depend on the settings of their superproject too (which is what this
thread is about). That would lead to very interesting problems when the same
submodule would be used by different superprojects which ignore different
files and/or directories inside the submodule, which - to make things even
more interesting - might be recorded in the submodule itself ... not good.

So what about: s/can't/isn't and shouldn't/ and s/are/are and shouldn't/?

Re: Nesting a submodule inside of another...

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> ... (at least when it concerns their work tree, the
> gitfile effort is trying to introduce some superproject awareness).

I don't think so.

The "gitfile effort" just makes it more convenient and potentially safer
for superproject to remove submodule working tree by running an equivalent
of "rm -fr submodule". The submodule shouldn't even care where its .git
metadirectory is by looking at the "gitfile: " line, nor should it assume
that location its .git metadirectory moved out of its working tree has
some fixed relationship with its supermodule's repository.  E.g. the
superproject could choose to store it under $HOME/cache or /var/state as
long as it knows where to point it back to after it runs "rm -rf submodule"
and then it has to re-instantiate the submodule.

IOW, the superproject must be aware of that location, but submodule
shouldn't depend on what convention the superproject chose to use.
After all, there may not even be a superproject when gitfile is in use,
so running "test -f .git" and assuming that it is used as a submodule of
somebody else is already a bogus test.

Re: Nesting a submodule inside of another...

[John Szakmeister]

On Wed, Jul 13, 2011 at 5:20 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 13.07.2011 03:21, schrieb John Szakmeister:
[snip]
>> I guess I still don't see how that's a problem.  It seems to me that
>> for the superproject the ignore rule needs to be:
>>   ignore framework1, except framework1/module
>
> But that would mean the superproject would have to scan framework1's
> work tree for changes too, which is conflicting with the idea that each
> submodule is a repo of it's own, so that won't fly.

I guess I just don't see that.  Even though the path is inside of a
submodules, it would be part of the superproject's work tree.  But, I
can see how it's a violation of a submodule being a repo of it's own.
And I certainly understand the desire to maintain that quality.

>> And then it could check its status.  Of course, framework1 would
>> somehow need to know to ignore it... and I'm not sure how that can be
>> communicated.
>
> It can't be communicated, as submodules are unaware of their superproject.
> And adding module manually to framework1's .gitignore is just a problem
> waiting to happen. Having a file or submodule in the work tree of one
> git repository being referenced by another is just asking for trouble ...

Again, I don't see how the submodule needs to be aware of the
superproject.  In this case, it'd be the responsibility of the
superproject to setup whatever is necessary at 'git submodule
init/add'.  I don't see how the submodule *must* know about the
superproject for it to succeed.  I see the opposite, the superproject
needs to communicate some information down to the submodule, but I
don't see the reverse.

What I'm hearing is that while it may be possible, the idea of
violating the concept that the "subrepo is standalone" is
unacceptable.  Which means, unfortunately, git isn't a solution for us
in this case.

[snip]
>> I'm using 1.7.6rc3.  I updated to the latest trunk and still get the same error:
>>
>> :: git submodule add $PWD/../c b/c
>> The following path is ignored by one of your .gitignore files:
>> b/c
>> Use -f if you really want to add it.
>
> You're right, the error message of "git submodule add" should be better
> here.

I'll write a patch for this.  Thanks.

And thanks for the input.  I was just looking to see if there were any
technical/morale reasons that submodules weren't allowed in this way.
I appreciate you all taking the time to answer.

-John

Re: Nesting a submodule inside of another...

[Seth Robertson]

In message <CAEBDL5VUPE9YCX1C4pqkjb+EODkAWo9h774B=Jv5eUNbocMuZQ@mail.gmail.com>, John Szakmeister writes:

    > I've got a project where we have several frameworks involved, and
    > external modules we want to pull into the framework tree.  We'd like
    > to make use of submodules and have something like this:
    >      top-level/<-- .gitmodules lives here
    >          src/
    >          framework1/<-- a submodule
    >              module/<-- another submodule
    >          framework2/<-- a submodule
    >              module2/<-- another submodule

    Again, I don't see how the submodule needs to be aware of the
    superproject.  In this case, it'd be the responsibility of the
    superproject to setup whatever is necessary at 'git submodule
    init/add'.  I don't see how the submodule *must* know about the
    superproject for it to succeed.  I see the opposite, the superproject
    needs to communicate some information down to the submodule, but I
    don't see the reverse.

    What I'm hearing is that while it may be possible, the idea of
    violating the concept that the "subrepo is standalone" is
    unacceptable.  Which means, unfortunately, git isn't a solution for us
    in this case.

You might find that gitslave (http://gitslave.sf.net) might be a
better solution for you than git-submodules in this case.  It works
better for many workflows (and worse for others) but is much simpler
to understand since with gitslave you have JBOR (just a bunch of
repositories) with a program which can be thought of as running the
requested git command over each repository in turn.  Gitslave thus has
a loose binding between the repositories, and you can only guarantee
the relationship between repositories at tagged locations, though in
practice this isn't a major concern.

gitslave supports nested repositories (and recursive gitslave
repositories, but those are different).  With gitslave nested
repositories it is also true that you would have to have a
supplemental gitignore entry in framework1 (which gitslave will
create).

If you have any questions, please let me know.

					-Seth Robertson

Re: Nesting a submodule inside of another...

[John Szakmeister]

On Thu, Jul 14, 2011 at 10:33 AM, Seth Robertson <in-gitvger@baka.org> wrote:
[snip]
>    Again, I don't see how the submodule needs to be aware of the
>    superproject.  In this case, it'd be the responsibility of the
>    superproject to setup whatever is necessary at 'git submodule
>    init/add'.  I don't see how the submodule *must* know about the
>    superproject for it to succeed.  I see the opposite, the superproject
>    needs to communicate some information down to the submodule, but I
>    don't see the reverse.
>
>    What I'm hearing is that while it may be possible, the idea of
>    violating the concept that the "subrepo is standalone" is
>    unacceptable.  Which means, unfortunately, git isn't a solution for us
>    in this case.
>
> You might find that gitslave (http://gitslave.sf.net) might be a
> better solution for you than git-submodules in this case.  It works
> better for many workflows (and worse for others) but is much simpler
> to understand since with gitslave you have JBOR (just a bunch of
> repositories) with a program which can be thought of as running the
> requested git command over each repository in turn.  Gitslave thus has
> a loose binding between the repositories, and you can only guarantee
> the relationship between repositories at tagged locations, though in
> practice this isn't a major concern.

Interesting.  I'll take a look!

> gitslave supports nested repositories (and recursive gitslave
> repositories, but those are different).  With gitslave nested
> repositories it is also true that you would have to have a
> supplemental gitignore entry in framework1 (which gitslave will
> create).
>
> If you have any questions, please let me know.

I appreciate it.  Thanks Seth!

-John