[wishlist?] make submodule commands robust to having non-submodule Subprojects

[wishlist?] make submodule commands robust to having non-submodule Subprojects

[Yaroslav Halchenko]

NB echos some questions of mine a few days back on IRC about Subprojects
and submodules 

If e.g. you just 'git add' a subdirectory which is a git repository, git
adds it as a subproject but doesn't initiate any entry in .gitmodules
since it is the job done by submodule and git core itself is
agnostic of those beasts.

But having then this "Subproject"s which aren't registered as submodules
(and I haven't found any other use for them besides being a submodule)
brakes "git submodule" commands, e.g.

$> git submodule
 cc6a09ac06c13cf06b4f4c8b54cda9a535e4e385 ds000001 (2.0.0+4)
 0a9f3b66e06a2137311a537b7377c336f1fb30ad ds000002 (1.0.0-3-g0a9f3b6)
 9da7e4f4221699915645ac2003298c6aba2db109 ds000003 (1.1.0+4)
 fe16cacb5cb9b4d53c50e498298fab182500e147 ds000005 (2.0.0+3)
 6898d99ff3ba26880183ed3672a458a7fcde1737 ds000006 (2.0.0+2)
 bbd10f634fe87e9d5853df3a891edbdb18cda7f9 ds000007 (2.0.0+3)
 138e6730193c0585a69b8baf5b9d7a4439e83ecc ds000008 (2.0.0+2)
 ddf3a4cf7ce51a01a664e6faff4b8334b8414b1f ds000009 (2.0.1+1)
 7fa73b4df8166dba950c7dc07c3f8cdd50fca313 ds000011 (1.0.0-5-g7fa73b4)
fatal: no submodule mapping found in .gitmodules for path 'ds000017

which then stops without even looking at other submodules.

I think it would be more logical to make it a 'warning:' not a 'fatal:' and
proceed.

Thank you for consideration
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Stefan Beller]

On Thu, Sep 15, 2016 at 6:02 AM, Yaroslav Halchenko <yoh@onerussian.com> wrote:
> NB echos some questions of mine a few days back on IRC about Subprojects
> and submodules
>
> If e.g. you just 'git add' a subdirectory which is a git repository, git
> adds it as a subproject but doesn't initiate any entry in .gitmodules
> since it is the job done by submodule and git core itself is
> agnostic of those beasts.
>
> But having then this "Subproject"s which aren't registered as submodules
> (and I haven't found any other use for them besides being a submodule)
> brakes "git submodule" commands, e.g.
>
> $> git submodule
>  cc6a09ac06c13cf06b4f4c8b54cda9a535e4e385 ds000001 (2.0.0+4)
>  0a9f3b66e06a2137311a537b7377c336f1fb30ad ds000002 (1.0.0-3-g0a9f3b6)
>  9da7e4f4221699915645ac2003298c6aba2db109 ds000003 (1.1.0+4)
>  fe16cacb5cb9b4d53c50e498298fab182500e147 ds000005 (2.0.0+3)
>  6898d99ff3ba26880183ed3672a458a7fcde1737 ds000006 (2.0.0+2)
>  bbd10f634fe87e9d5853df3a891edbdb18cda7f9 ds000007 (2.0.0+3)
>  138e6730193c0585a69b8baf5b9d7a4439e83ecc ds000008 (2.0.0+2)
>  ddf3a4cf7ce51a01a664e6faff4b8334b8414b1f ds000009 (2.0.1+1)
>  7fa73b4df8166dba950c7dc07c3f8cdd50fca313 ds000011 (1.0.0-5-g7fa73b4)
> fatal: no submodule mapping found in .gitmodules for path 'ds000017
>
> which then stops without even looking at other submodules.
>
> I think it would be more logical to make it a 'warning:' not a 'fatal:' and
> proceed.

So maybe we would want to introduce a switch
  `--existing-but-unconfigure-gitlinks=(warn|ignore)`
as well as
`git config submodule.existing-but-unconfigured (warn|ignore)`
for a more permanent solution?



>
> Thank you for consideration
> --
> Yaroslav O. Halchenko
> Center for Open Neuroscience     http://centerforopenneuroscience.org
> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
> WWW:   http://www.linkedin.com/in/yarik

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Yaroslav Halchenko]

On Thu, 15 Sep 2016, Stefan Beller wrote:

> > I think it would be more logical to make it a 'warning:' not a 'fatal:' and
> > proceed.

> So maybe we would want to introduce a switch
>   `--existing-but-unconfigure-gitlinks=(warn|ignore)`
> as well as
> `git config submodule.existing-but-unconfigured (warn|ignore)`
> for a more permanent solution?

possibly ignorant question:  is  gitlink === Subproject    or a
Subproject is a kinda of a gitlink and there are other gitlinks which
aren't Subprojects? ;)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Stefan Beller]

On Thu, Sep 15, 2016 at 9:40 AM, Yaroslav Halchenko <yoh@onerussian.com> wrote:
>
> On Thu, 15 Sep 2016, Stefan Beller wrote:
>
>> > I think it would be more logical to make it a 'warning:' not a 'fatal:' and
>> > proceed.
>
>> So maybe we would want to introduce a switch
>>   `--existing-but-unconfigure-gitlinks=(warn|ignore)`
>> as well as
>> `git config submodule.existing-but-unconfigured (warn|ignore)`
>> for a more permanent solution?
>
> possibly ignorant question:  is  gitlink === Subproject    or a
> Subproject is a kinda of a gitlink and there are other gitlinks which
> aren't Subprojects? ;)
>

gitlink is the internal name (just like a file is called blob, or a directory is
called tree; there is no file system equivalent for e.g. commits, or
gitlinks)

gitlinks as a basic building block only points at a sha1 to be part of
the repository.

Submodules use gitlinks to point at (usually) different projects, e.g.
a library.

Subprojects in Git is a loose term, it could mean submodule or part of your
repo merged in via the git-subtree command. Or a sub project can
be just a loose repository inside your repository.

gitlinks do not necessarily need to track other projects; it's "just a pointer"
to a specific version of a repository.

So to come back to your question: yeah In that response I meant
submodule==gitlink, though naming the internal data structure to be
more explicit. (What if we invent a new way to do submodules not
using gitlinks? Then that config option may be less confusing and
backwards compatible. That is unlikely though. So I guess we'll settle
with a different name)

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> On Thu, Sep 15, 2016 at 6:02 AM, Yaroslav Halchenko <yoh@onerussian.com> wrote:
>>
>> If e.g. you just 'git add' a subdirectory which is a git repository, git
>> adds it as a subproject but doesn't initiate any entry in .gitmodules
>> since it is the job done by submodule and git core itself is
>> agnostic of those beasts.
>> ...
>> $> git submodule
>>  cc6a09ac06c13cf06b4f4c8b54cda9a535e4e385 ds000001 (2.0.0+4)
>>  0a9f3b66e06a2137311a537b7377c336f1fb30ad ds000002 (1.0.0-3-g0a9f3b6)
>>  9da7e4f4221699915645ac2003298c6aba2db109 ds000003 (1.1.0+4)
>>  fe16cacb5cb9b4d53c50e498298fab182500e147 ds000005 (2.0.0+3)
>>  6898d99ff3ba26880183ed3672a458a7fcde1737 ds000006 (2.0.0+2)
>>  bbd10f634fe87e9d5853df3a891edbdb18cda7f9 ds000007 (2.0.0+3)
>>  138e6730193c0585a69b8baf5b9d7a4439e83ecc ds000008 (2.0.0+2)
>>  ddf3a4cf7ce51a01a664e6faff4b8334b8414b1f ds000009 (2.0.1+1)
>>  7fa73b4df8166dba950c7dc07c3f8cdd50fca313 ds000011 (1.0.0-5-g7fa73b4)
>> fatal: no submodule mapping found in .gitmodules for path 'ds000017
>>
>> which then stops without even looking at other submodules.
>>
>> I think it would be more logical to make it a 'warning:' not a 'fatal:' and
>> proceed.

Making "git submodule" listing to continue from that point may be
one thing, but do we have a sensible way in "git submodule add" to
allow the user to recover from this condition?  That is, "git add"
is a right way to tell the core level that there is a gitlink, but
as Yaroslav correctly observed in the early part of his message,
having that gitlink alone is not good enough for the world view of 
"git submodule" that sits at higher-layer.  And the usual way to
tell the submodule layer that there is a submodule is with "git
submodule add", but

	$ git init top
        $ cd top
        $ git commit --allow-empty -m 'initial in top'
        $ git init sub
        $ git -C sub commit --allow-empty -m 'initial in sub'

        $ git add sub
	$ git submodule
        fatal: no submodule mapping found in .gitmodules for path 'sub'

	$ git submodule add ./sub sub
        'sub' already exists in the index
	$ git submodule add -f ./sub sub
        'sub' already exists in the index

I highly suspect that the user will then get stuck at this point,
after trying to "submodule add" and then even attempting to force
it.

I think that is a more pressing thing to address.  Once we make it
easier for the user to bring a half-initialized submodule properly
into the world view of the submodule subsystem, we would have to
worry about the reported failure case even less and you do not need 
to pile on workaround options to let things continue in a state that
is half-broken (that is, in a state that is perfectly sane to the
core layer, but is not liked by the submodule layer).

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Yaroslav Halchenko]

On Thu, 15 Sep 2016, Junio C Hamano wrote:

> >> which then stops without even looking at other submodules.

> >> I think it would be more logical to make it a 'warning:' not a 'fatal:' and
> >> proceed.

> Making "git submodule" listing to continue from that point may be
> one thing, but do we have a sensible way in "git submodule add" to
> allow the user to recover from this condition?  That is, "git add"
> is a right way to tell the core level that there is a gitlink, but
> as Yaroslav correctly observed in the early part of his message,
> having that gitlink alone is not good enough for the world view of 
> "git submodule" that sits at higher-layer.  And the usual way to
> tell the submodule layer that there is a submodule is with "git
> submodule add", but

> 	$ git init top
>         $ cd top
>         $ git commit --allow-empty -m 'initial in top'
>         $ git init sub
>         $ git -C sub commit --allow-empty -m 'initial in sub'

>         $ git add sub
> 	$ git submodule
>         fatal: no submodule mapping found in .gitmodules for path 'sub'

> 	$ git submodule add ./sub sub
>         'sub' already exists in the index
> 	$ git submodule add -f ./sub sub
>         'sub' already exists in the index

FWIW
I could have sworn that I have tried to 'submodule add' it and it
worked... but pragmatically I just did edit .gitmodules, added the
record for it, committed it, and then iirc git submodule update --init
which seemed to make  git happy... FTR:

$> git submodule add ./sub ./sub
'sub' already exists in the index

$> git submodule add ./sub/ ./sub/
'sub' already exists in the index

$> vim .gitmodules

$> git add .gitmodules
cached/staged changes:                                                                                                                                                                                                                                                                                            
 .gitmodules | 4 ++++
 sub         | 1 +

$> git submodule update --init
Submodule 'sub' (/tmp/111/top/sub) registered for path 'sub'
cached/staged changes:                                                                                                                                                                                                                                                                                            
 .gitmodules | 4 ++++
 sub         | 1 +

$> git commit -m 'added finally'
[master aa6d912] added finally
 2 files changed, 5 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 sub

$> git submodule
 6f574b298ef51aebd36daafad450a3e38802ca03 sub (heads/master)


> I highly suspect that the user will then get stuck at this point,
> after trying to "submodule add" and then even attempting to force
> it.

> I think that is a more pressing thing to address.  Once we make it
> easier for the user to bring a half-initialized submodule properly
> into the world view of the submodule subsystem, we would have to
> worry about the reported failure case even less and you do not need 
> to pile on workaround options to let things continue in a state that
> is half-broken (that is, in a state that is perfectly sane to the
> core layer, but is not liked by the submodule layer).

do you foresee any unpleasant side-effects from above manual editing
.gitmodules/submodule update --init ?

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Stefan Beller]

On Thu, Sep 15, 2016 at 11:12 AM, Yaroslav Halchenko <yoh@onerussian.com> wrote:

>
> do you foresee any unpleasant side-effects from above manual editing
> .gitmodules/submodule update --init ?

I think that is fine, but un(der)documented. So you have to figure it out
from experience what to do exactly.

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Junio C Hamano]

Yaroslav Halchenko <yoh@onerussian.com> writes:

> do you foresee any unpleasant side-effects from above manual editing
> .gitmodules/submodule update --init ?

I do not think so; you essentially did what a canned command we
should have had should have done by hand because of a lack of such a
feature.

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Stefan Beller]

On Thu, Sep 15, 2016 at 11:02 AM, Junio C Hamano <gitster@pobox.com> wrote:

> I think that is a more pressing thing to address.  Once we make it
> easier for the user to bring a half-initialized submodule properly
> into the world view of the submodule subsystem, we would have to
> worry about the reported failure case even less and you do not need
> to pile on workaround options to let things continue in a state that
> is half-broken (that is, in a state that is perfectly sane to the
> core layer, but is not liked by the submodule layer).

Heh, I see.

    When searching around the net, some people use half
    initialized submodules intentionally, e.g. I'll store some private keys in
    sub and I publish the superproject asking for collaborators to my new shiny
    webbased thing. The submodule containing the private keys never leaves
    my hard drive, hence no .gitmodules entry is necessary.

Not sure I agree with such a setup, but people use it.

So how about this fictional work flow:

         $ git init top
         $ cd top
         $ git commit --allow-empty -m 'initial in top'
         $ git init sub
         $ git -C sub commit --allow-empty -m 'initial in sub'
         $ git add sub
        You added a gitlink, but no corresponding entry in
        .gitmodules is found. This is fine for gits core functionality, but
        the submodule command gets confused by this unless you add 'sub'
        to your .gitmodules via `git submodule add --already-in-tree \
        --reuse-submodules-origin-as-URL sub`. Alternatively you can make this
        message disappear by configuring advice.gitlinkPitfalls.

         $ git submodule
         ... similar advice goes here...
         fatal: no submodule mapping found in .gitmodules for path 'sub'

         $ git submodule add --fixup-modules-file ./sub sub
         Adding .gitmodule entry only for `sub` to use `git -C remote
show origin` as URL.

         # user is happy now.

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

>     When searching around the net, some people use half
>     initialized submodules intentionally,...
>
> Not sure I agree with such a setup, but people use it.

In such a top-level project, people would not use "git submodule"
command, would they?  I do not think anybody in this thread was
pushing to forbid such a use, and it may be perfectly fine if "git
submodule" does not work for such a gitlink; after all such a
subdirectory is not even meant to be a submodule.

> So how about this fictional work flow:
>
>          $ git init top
>          $ cd top
>          $ git commit --allow-empty -m 'initial in top'
>          $ git init sub
>          $ git -C sub commit --allow-empty -m 'initial in sub'
>          $ git add sub
>         You added a gitlink, but no corresponding entry in
>         .gitmodules is found. This is fine for gits core functionality, but
>         the submodule command gets confused by this unless you add 'sub'
>         to your .gitmodules via `git submodule add --already-in-tree \
>         --reuse-submodules-origin-as-URL sub`. Alternatively you can make this
>         message disappear by configuring advice.gitlinkPitfalls.

I am not sure if I agree with that direction.

If the trend in Git community collectively these days is to make
usage of submodules easier and smoother, I'd imagine that you would
want to teach "git add" that was given a submodule to "git submodule
add" instead by default, with an option "git add --no-gitmodules
sub" to disable it, or something like that.

>          $ git submodule add --fixup-modules-file ./sub sub
>          Adding .gitmodule entry only for `sub` to use `git -C remote
> show origin` as URL.

I agree that a feature like this is needed regardless of what
happens at "git add" time.

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Heiko Voigt]

On Thu, Sep 15, 2016 at 11:27:54AM -0700, Junio C Hamano wrote:
> Stefan Beller <sbeller@google.com> writes:
> > So how about this fictional work flow:
> >
> >          $ git init top
> >          $ cd top
> >          $ git commit --allow-empty -m 'initial in top'
> >          $ git init sub
> >          $ git -C sub commit --allow-empty -m 'initial in sub'
> >          $ git add sub
> >         You added a gitlink, but no corresponding entry in
> >         .gitmodules is found. This is fine for gits core functionality, but
> >         the submodule command gets confused by this unless you add 'sub'
> >         to your .gitmodules via `git submodule add --already-in-tree \
> >         --reuse-submodules-origin-as-URL sub`. Alternatively you can make this
> >         message disappear by configuring advice.gitlinkPitfalls.
> 
> I am not sure if I agree with that direction.
> 
> If the trend in Git community collectively these days is to make
> usage of submodules easier and smoother, I'd imagine that you would
> want to teach "git add" that was given a submodule to "git submodule
> add" instead by default, with an option "git add --no-gitmodules
> sub" to disable it, or something like that.
> 
> >          $ git submodule add --fixup-modules-file ./sub sub
> >          Adding .gitmodule entry only for `sub` to use `git -C remote
> > show origin` as URL.
> 
> I agree that a feature like this is needed regardless of what
> happens at "git add" time.

How about just

   git submodule add <submodulepath>

? I remember back in the days when I started with submodules thats the
way I imagined submodules would work:

1. clone the submodule into a directory
2. git submodule add it
3. git commit everything

Because that how you basically work with files.  So instead of adding
another option I would rather like to autodetect that:

 * its a relative path inside this repo that is passed to
   'git submodule add'
 * there is no .gitmodules entry
 * and no .git/config
==> create those from a remote in the submodule

Corner cases:

 * If there is more than one remote we could tell the user to use an
   option to specify which one to use.
 * Barf in case there is no remote (not adding the submodule except -f
   is used).
 * If the gitlink is already there but no .gitmodules entry, 'git
   submodule add' will just add the entry as if it was initially added.

Instead of giving an error message that the submodule is already added
we could actually be nicer to the user and try to fix things for him
instead.

Cheers Heiko

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Jacob Keller]

On Fri, Sep 16, 2016 at 7:11 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> How about just
>
>    git submodule add <submodulepath>
>
> ? I remember back in the days when I started with submodules thats the
> way I imagined submodules would work:
>
> 1. clone the submodule into a directory
> 2. git submodule add it
> 3. git commit everything
>
> Because that how you basically work with files.  So instead of adding
> another option I would rather like to autodetect that:
>
>  * its a relative path inside this repo that is passed to
>    'git submodule add'
>  * there is no .gitmodules entry
>  * and no .git/config
> ==> create those from a remote in the submodule
>
> Corner cases:
>
>  * If there is more than one remote we could tell the user to use an
>    option to specify which one to use.
>  * Barf in case there is no remote (not adding the submodule except -f
>    is used).
>  * If the gitlink is already there but no .gitmodules entry, 'git
>    submodule add' will just add the entry as if it was initially added.
>
> Instead of giving an error message that the submodule is already added
> we could actually be nicer to the user and try to fix things for him
> instead.
>

This makes sense to me. Possibly we could warn in this case, so that
the user knows that something was "off" but I don't think we should be
failing here...

Regards,
Jake

> Cheers Heiko

Re: [wishlist?] make submodule commands robust to having non-submodule Subprojects

[Junio C Hamano]

Heiko Voigt <hvoigt@hvoigt.net> writes:

> On Thu, Sep 15, 2016 at 11:27:54AM -0700, Junio C Hamano wrote:
>
>> If the trend in Git community collectively these days is to make
>> usage of submodules easier and smoother, I'd imagine that you would
>> want to teach "git add" that was given a submodule to "git submodule
>> add" instead by default, with an option "git add --no-gitmodules
>> sub" to disable it, or something like that.
>> 
>> >          $ git submodule add --fixup-modules-file ./sub sub
>> >          Adding .gitmodule entry only for `sub` to use `git -C remote
>> > show origin` as URL.
>> 
>> I agree that a feature like this is needed regardless of what
>> happens at "git add" time.
>
> How about just
>
>    git submodule add <submodulepath>

When I said "a feature like this is needed", I didn't care about
exact syntax.  I am not sure how often people need the "fixup", what
kind of causes there are that they need the "fixup", and what the
distribution of vaious causes would be like.  If the _ONLY_ kind of
fixup necessary is "I meant to say 'git submodule add ./path path'
but I said 'git add path' instead", then I think it makes sense to
teach "submodule add" that the form "git submodule add <path>" is a
short-pand for "git submodule add ./<path> <path>".  I am not sure
if we want to _ignore_ a gitlink that is already in the index
unconditionally, i.e. if it is a good idea to let the second one
override the first one

	git submodule add $URL sub &&
        git submodule add sub

in this sequence, though.

> ? I remember back in the days when I started with submodules thats the
> way I imagined submodules would work:
>
> 1. clone the submodule into a directory
> 2. git submodule add it
> 3. git commit everything
>
> Because that how you basically work with files.  So instead of adding
> another option I would rather like to autodetect that:
>
>  * its a relative path inside this repo that is passed to
>    'git submodule add'
>  * there is no .gitmodules entry
>  * and no .git/config
> ==> create those from a remote in the submodule

In other words, I agree with the general direction but I'd add
another condition to the above three, i.e.

   * and there is no gitlink for that path in the index yet.