tracking submodules out of main directory.

tracking submodules out of main directory.

[henri GEIST]

Hello,

Here is my topic.

Is it possible to track a submodule through a relative path outside of a
submodule ?

The purpose of this thing is to maintain dependency through different
library or projects.

Let says this generic dependency tree.


project 1 --+--> library 1
            |
            +--> library 2


project 2 --+--> library 1
            |
            +--> library 3


project 3 --+--> library 2
            |
            +--> library 4
            |
            +--> project 1 --+--> library 1
                             |
                             +--> library 2

project 4 --+--> library 4
            |
            +--> project 1 --+--> library 1
            |                |
            |                +--> library 2
            |
            +--> project 2 --+--> library 1
            |                |
            |                +--> library 3
            |
            +--> project 3 --+--> library 2
                             |
                             +--> library 4
                             |
                             +--> project 1 --+--> library 1
                                              |
                                              +--> library 2

CAUTION : this is a tree of dependency not directory.

We can obviously solve this by doing trees of submodules just reflecting
the trees of dependency but it create somme problems.

  1. In project 4 I have 2 times project 1 and 3 times libraries 1 and 2
     And 2 times library 4.
  2. It is a wast of space.
  3. Different version of the same libraries or projects could be used.
  4. when linking object, multiples objects will export the same symbols

Point 2 could be acceptable in little project.

Point 4 could be addressed by a careful compilation schema.

Point 3 could be no issue and discard point 2 meaning if it is what you
        want in your work flow.
        But in my work flow it is a bug every thing NEED to be
        synchronized.

what I would like to have is every git tree in the same directory where
I put my code (not necessary a super project tree).

like this :

Anything/library_1
Anything/library_2
Anything/library_3
Anything/library_4
Anything/project_1 with a git submodule add ../library_1 ../library_2
Anything/project_2 with a git submodule add ../library_1 ../library_3
Anything/project_3 with a git submodule add ../library_2 ../library_4
                                            ../project_1
Anything/project_4 with a git submodule add ../library_4 ../project_1
                                            ../project_2 ../project_3

But when I do this I receive a
"fatal: '../library_1' is outside repository"

This is due to the fact that 'git add submodule' use the tests of
'git add' which prevent creating files outside of the working tree for
obvious security reasons.

But in my point of view the meaning of a submodule should be more like a
symlink on a repository than a sub tree.
And symlink does not seams to be prohibited even if they link to
something outside of the repository.

I have tried a work around by doing :
"git submodule add library_1 library_2" in project 1
and then replacing library_1 and library_2 by symlinks to
../library_1 and ../library_2
but git immediately track the symlink itself and not what is pointed to.
And even if it had worked, this type of work around is not clean.

I have think about removing the test in setup.c in case of adding a
submodule. But I am not deeply involved in git source code then I am not
sure of the possibles side effects of this modification.

As well may be there is an other solution that I still haven not think
about.

	Henri GEIST

Re: tracking submodules out of main directory.

[Junio C Hamano]

henri GEIST <henri.geist@flying-robots.com> writes:

> We can obviously solve this by doing trees of submodules just reflecting
> the trees of dependency but it create somme problems.
>
>   1. In project 4 I have 2 times project 1 and 3 times libraries 1 and 2
>      And 2 times library 4.
>   2. It is a wast of space.
>   3. Different version of the same libraries or projects could be used.
>   4. when linking object, multiples objects will export the same symbols

All of the above are something your build procedure needs to solve
regardless, even if you are not using git submodules (or even if you are
not using any SCM, for that matter).  If you want to (and you do want to)
avoid duplication, version drift and associated issues of use of the
common library 1 across project 1 and project 2, you would organize your
source tree so that both of your (sub)projects to use the library code
from a common location.

One possible working tree organization may look like this:

	-+- lib1
         +- project1/Makefile -- refers to ../lib1
         +- project2/Makefile -- refers to ../lib1

The top-level superproject (what you called "Anything") binds project1,
project2 and lib1 as its submodules, and it can say where you want your
users to fetch these submodules in its .gitmodules, and at what version
lib1 (and projects) to use in its tree objects as gitlinks.

People who are _only_ interested in project1 can still clone that
top-level superproject, and "submodule init" only lib1 and project1,
without running "submodule init" on project2.

An interesting point your situation raises is that there is no direct way
to express module dependencies in .gitmodules file right now, I think.
Ideally you would want "submodule init project1" to infer automatically
that project1 needs lib1 and run "submodule init lib1" for you. My gut
feeling is that it belongs to .gitmodules of the superproject.

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 27.06.2011 18:51, schrieb Junio C Hamano:
> One possible working tree organization may look like this:
> 
> 	-+- lib1
>          +- project1/Makefile -- refers to ../lib1
>          +- project2/Makefile -- refers to ../lib1

This is what we do at work and it works really well for us. The
possible downside (that you can't tie project1 and project2 to a
specific version of lib1 in their own repo) is not a real problem
in our experience, as the superproject ties the correct combination
together.

> An interesting point your situation raises is that there is no direct way
> to express module dependencies in .gitmodules file right now, I think.
> Ideally you would want "submodule init project1" to infer automatically
> that project1 needs lib1 and run "submodule init lib1" for you. My gut
> feeling is that it belongs to .gitmodules of the superproject

That is where this is handled now, but having a submodule refer to a
submodule outside of it as a dependency is an interesting thought. But
as that only matters at the moment you add project1 (and it won't compile
because ../lib1 is missing, which can easily handled by: "oh, then I have
to add lib1 as a submodule to the superproject too"), I'm not sure that
would be a huge improvement over just having a sane convention of where
to get lib1 from.

Re: tracking submodules out of main directory.

[henri GEIST]

First of all thank you for your answer.

Le lundi 27 juin 2011 à 09:51 -0700, Junio C Hamano a écrit :
> henri GEIST <henri.geist@flying-robots.com> writes:
> 
>> We can obviously solve this by doing trees of submodules just reflecting
>> the trees of dependency but it create somme problems.
>>
>>   1. In project 4 I have 2 times project 1 and 3 times libraries 1 and 2
>>      And 2 times library 4.
>>   2. It is a wast of space.
>>   3. Different version of the same libraries or projects could be used.
>>   4. when linking object, multiples objects will export the same symbols
> 
> All of the above are something your build procedure needs to solve
> regardless, even if you are not using git submodules (or even if you are
> not using any SCM, for that matter).  If you want to (and you do want to)
> avoid duplication, version drift and associated issues of use of the
> common library 1 across project 1 and project 2, you would organize your
> source tree so that both of your (sub)projects to use the library code
> from a common location.
> 
> One possible working tree organization may look like this:
> 
> 	-+- lib1
>          +- project1/Makefile -- refers to ../lib1
>          +- project2/Makefile -- refers to ../lib1
> 

I agree on this point.
And that is just what I do and mean by :

>> Anything/library_1
>> Anything/library_2
>> Anything/library_3
>> Anything/library_4
>> Anything/project_1 with a git submodule add ../library_1 ../library_2
>> Anything/project_2 with a git submodule add ../library_1 ../library_3
>> Anything/project_3 with a git submodule add ../library_2 ../library_4
>>                                             ../project_1
>> Anything/project_4 with a git submodule add ../library_4 ../project_1
>>                                             ../project_2 ../project_3

I would like git to do just the same thing that do my Makefile.

> The top-level superproject (what you called "Anything") binds project1,
> project2 and lib1 as its submodules, and it can say where you want your
> users to fetch these submodules in its .gitmodules, and at what version
> lib1 (and projects) to use in its tree objects as gitlinks.
> 

Until now I did not thought about making "Anything" a git repository.
It was just a simple directory containing all my projects.
And project_4 was not a submodule of anything.

May be I need to think more about it.

But actually it is just pushing the problem one rank more by making
"Anything" a kind of project_5 which contain submodules in sub
directories and will rase the same problem the day project_6 will need
project_5.
And it does not solve the problem of making project_1 knowing is need to
be linked to library_1 in version "abcd1234..."

> People who are _only_ interested in project1 can still clone that
> top-level superproject, and "submodule init" only lib1 and project1,
> without running "submodule init" on project2.
> 

I agree on this point.

> An interesting point your situation raises is that there is no direct way
> to express module dependencies in .gitmodules file right now, I think.

And that is the real problem.
It could be done for submodule inside of the project main directory.
But not for modules outside of it.

> Ideally you would want "submodule init project1" to infer automatically
> that project1 needs lib1 and run "submodule init lib1" for you. My gut
> feeling is that it belongs to .gitmodules of the superproject.
> 

I do not really care about project_1 doing the submodule init.
I can easily do it by myself.
And in fact I use to write the libraries before the project using them.
then there git repositories already exist

what I really need is to do :

cd project_1
git add ../library_1

then in case of modification in library_1

A git status in project_1 directory will say me :

modified:   ../library_1 (modified content)
or
modified:   ../library_1 (new commits)
or even
modified:   ../library_1 (new commits, modified content)

Just like it do for submodules with out "../" in the path.

And I really think the metadata to do so should be stored in project_1
since it is him which depend on library_1 version "abcd1234..." and this
information need to be self contained.
May be in project_1/.git or project_1/.gitmodules

I do not see the point of having a third party project "Anything" Just
to say to project_1 hey you need library_1.
If it need it, it should already know it.


	Henri GEIST

Re: tracking submodules out of main directory.

[henri GEIST]

Le lundi 27 juin 2011 à 20:14 +0200, Jens Lehmann a écrit :
> Am 27.06.2011 18:51, schrieb Junio C Hamano:
> > One possible working tree organization may look like this:
> > 
> > 	-+- lib1
> >          +- project1/Makefile -- refers to ../lib1
> >          +- project2/Makefile -- refers to ../lib1
> 
> This is what we do at work and it works really well for us. The
> possible downside (that you can't tie project1 and project2 to a
> specific version of lib1 in their own repo) is not a real problem
> in our experience, as the superproject ties the correct combination
> together.
> 

That is a good starting point.
I have used this method for some little projects and it work great.

But I had never find a mean to handle the case of project1 depending on
both lib1 and lib2.

	Henri GEIST

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 27.06.2011 20:52, schrieb henri GEIST:
> Le lundi 27 juin 2011 à 20:14 +0200, Jens Lehmann a écrit :
>> Am 27.06.2011 18:51, schrieb Junio C Hamano:
>>> One possible working tree organization may look like this:
>>>
>>> 	-+- lib1
>>>          +- project1/Makefile -- refers to ../lib1
>>>          +- project2/Makefile -- refers to ../lib1
>>
>> This is what we do at work and it works really well for us. The
>> possible downside (that you can't tie project1 and project2 to a
>> specific version of lib1 in their own repo) is not a real problem
>> in our experience, as the superproject ties the correct combination
>> together.
>>
> 
> That is a good starting point.
> I have used this method for some little projects and it work great.
> 
> But I had never find a mean to handle the case of project1 depending on
> both lib1 and lib2.

Hmm, but isn't that covered by having lib1, lib2 & project1 right next
to each other in your superproject?

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 27.06.2011 20:40, schrieb henri GEIST:
> what I really need is to do :
> 
> cd project_1
> git add ../library_1
> 
> then in case of modification in library_1
> 
> A git status in project_1 directory will say me :
> 
> modified:   ../library_1 (modified content)
> or
> modified:   ../library_1 (new commits)
> or even
> modified:   ../library_1 (new commits, modified content)
> 
> Just like it do for submodules with out "../" in the path.
> 
> And I really think the metadata to do so should be stored in project_1
> since it is him which depend on library_1 version "abcd1234..." and this
> information need to be self contained.
> May be in project_1/.git or project_1/.gitmodules
> 
> I do not see the point of having a third party project "Anything" Just
> to say to project_1 hey you need library_1.
> If it need it, it should already know it.

The point is it avoids ambiguity and cross-dependencies. If you have
project_1 requesting the version "123456" of library_1 and project_2
wants "abcdef", you'll always have one project disagreeing (= being
dirty). But "abcdef" might be a direct descendant of "123456" and is
just fixing a bug that affects only project_2. So it would be ok for
both to have "abcdef", but you'll have to add an extra commit to
project_1 saying "updating library_1 because of bug in project_2" to
make it clean again. That would make the projects not so modular and
independent anymore and won't scale well.

If the superproject records these dependencies you just update
library_1, run all tests (including those in project_1 & project_2)
and commit that in the superproject.

Re: tracking submodules out of main directory.

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> Am 27.06.2011 18:51, schrieb Junio C Hamano:
>> One possible working tree organization may look like this:
>> 
>> 	-+- lib1
>>          +- project1/Makefile -- refers to ../lib1
>>          +- project2/Makefile -- refers to ../lib1
> ...
>> An interesting point your situation raises is that there is no direct way
>> to express module dependencies in .gitmodules file right now, I think.
>> Ideally you would want "submodule init project1" to infer automatically
>> that project1 needs lib1 and run "submodule init lib1" for you. My gut
>> feeling is that it belongs to .gitmodules of the superproject
>
> That is where this is handled now, but having a submodule refer to a
> submodule outside of it as a dependency is an interesting thought. But
> as that only matters at the moment you add project1 (and it won't compile
> because ../lib1 is missing, which can easily handled by: "oh, then I have
> to add lib1 as a submodule to the superproject too"), ...

That is what I called "there is no direct way". Wouldn't it be nicer if
the .gitmodules file in the superproject said something like

	[module "project one"]
		path = project1
        	url = ...
                depends = lib1
	[module "lib1"]
        	path = lib1
                url = ...

and then "git submodule init project1" run by the end user implied running
also "git submodule init lib1"?

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 27.06.2011 21:05, schrieb Junio C Hamano:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> Am 27.06.2011 18:51, schrieb Junio C Hamano:
>>> One possible working tree organization may look like this:
>>>
>>> 	-+- lib1
>>>          +- project1/Makefile -- refers to ../lib1
>>>          +- project2/Makefile -- refers to ../lib1
>> ...
>>> An interesting point your situation raises is that there is no direct way
>>> to express module dependencies in .gitmodules file right now, I think.
>>> Ideally you would want "submodule init project1" to infer automatically
>>> that project1 needs lib1 and run "submodule init lib1" for you. My gut
>>> feeling is that it belongs to .gitmodules of the superproject
>>
>> That is where this is handled now, but having a submodule refer to a
>> submodule outside of it as a dependency is an interesting thought. But
>> as that only matters at the moment you add project1 (and it won't compile
>> because ../lib1 is missing, which can easily handled by: "oh, then I have
>> to add lib1 as a submodule to the superproject too"), ...
> 
> That is what I called "there is no direct way". Wouldn't it be nicer if
> the .gitmodules file in the superproject said something like
> 
> 	[module "project one"]
> 		path = project1
>         	url = ...
>                 depends = lib1
> 	[module "lib1"]
>         	path = lib1
>                 url = ...
> 
> and then "git submodule init project1" run by the end user implied running
> also "git submodule init lib1"?

And if lib1 happens to have another dependency, that will be initialized
too? That would make life much easier for users who only want certain
submodules populated to work on, as they won't have to chase compile
errors anymore until they found all necessary submodules ... very nice.

Re: tracking submodules out of main directory.

[henri GEIST]

Le lundi 27 juin 2011 à 20:56 +0200, Jens Lehmann a écrit :
> Am 27.06.2011 20:52, schrieb henri GEIST:
> > Le lundi 27 juin 2011 à 20:14 +0200, Jens Lehmann a écrit :
> >> Am 27.06.2011 18:51, schrieb Junio C Hamano:
> >>> One possible working tree organization may look like this:
> >>>
> >>> 	-+- lib1
> >>>          +- project1/Makefile -- refers to ../lib1
> >>>          +- project2/Makefile -- refers to ../lib1
> >>
> >> This is what we do at work and it works really well for us. The
> >> possible downside (that you can't tie project1 and project2 to a
> >> specific version of lib1 in their own repo) is not a real problem
> >> in our experience, as the superproject ties the correct combination
> >> together.
> >>
> > 
> > That is a good starting point.
> > I have used this method for some little projects and it work great.
> > 
> > But I had never find a mean to handle the case of project1 depending on
> > both lib1 and lib2.
> 
> Hmm, but isn't that covered by having lib1, lib2 & project1 right next
> to each other in your superproject?
> 

Sorry I havent read well your schema. I have thought that you reversed
dependency and put project1 and project2 inside libs1.

	Henri GEIST

Re: tracking submodules out of main directory.

[henri GEIST]

Le lundi 27 juin 2011 à 21:02 +0200, Jens Lehmann a écrit :
> Am 27.06.2011 20:40, schrieb henri GEIST:
> > what I really need is to do :
> > 
> > cd project_1
> > git add ../library_1
> > 
> > then in case of modification in library_1
> > 
> > A git status in project_1 directory will say me :
> > 
> > modified:   ../library_1 (modified content)
> > or
> > modified:   ../library_1 (new commits)
> > or even
> > modified:   ../library_1 (new commits, modified content)
> > 
> > Just like it do for submodules with out "../" in the path.
> > 
> > And I really think the metadata to do so should be stored in project_1
> > since it is him which depend on library_1 version "abcd1234..." and this
> > information need to be self contained.
> > May be in project_1/.git or project_1/.gitmodules
> > 
> > I do not see the point of having a third party project "Anything" Just
> > to say to project_1 hey you need library_1.
> > If it need it, it should already know it.
> 
> The point is it avoids ambiguity and cross-dependencies. If you have
> project_1 requesting the version "123456" of library_1 and project_2
> wants "abcdef", you'll always have one project disagreeing (= being
> dirty).

Yes that is exactly what I want.

> But "abcdef" might be a direct descendant of "123456" and is
> just fixing a bug that affects only project_2.

Yes but you can not make a rule of this the difference can have any
reason.
And mare often the bug has a chance to affect both project.
In any case both will be improved with a version more correct.

> So it would be ok for
> both to have "abcdef", but you'll have to add an extra commit to
> project_1 saying "updating library_1 because of bug in project_2" to
> make it clean again.

Yes that is also exactly what I want.

> That would make the projects not so modular and
> independent anymore and won't scale well.
> 

It is sufficiently independent for me.
  - If some one pull only project1 from me he will have a consistent
    library_1 at "123456"

  - If some one pull only project2 from me he will have a consistent
    library_1 at "abcdef"

  - The conflicting version dependency is only in my working directory
    to be solved. And in my work flow updating project_1 to reflect the
    correction in library_1 is the thing to do.

  - If some one decide tu pull from me both project_1 and project_2 in
    the same working tree he will effectively inherit my conflict.
    But that also mean he is working on both project and decide to take
    account of the relation between them like me.
    Then the first one who upgrade project1 will be pulled by the other
    one.

  - In any case if you have no time to update a project an need to
    immediately work on it you always can :

      - Go in the library one directory and checkout the right version
        for the project.

      - Or clone project alone with its libraries in a completely
        separate directory.

> If the superproject records these dependencies you just update
> library_1, run all tests (including those in project_1 & project_2)
> and commit that in the superproject.
> 
In this case you could not commit anything until every thing has been
done without breaking something.

When committing library_1 to "abcdef" nothing wrong happen in both case
all project still point to "123456"

when committing project_2 it will need to point to "abcdef" but in your
work flow the super project still say him it point to "123456" and
project_2 is broken.

If you immediately commit the superproject to repair project_2 you will
brake project_1 until it is also upgraded and superproject is reupgraded
to reflect it.



In my work flow when committing project it will immediately point to
"abcdef" and not been broken.

project_1 still point to "123456" and is still not broken

superproject which is not needed still point on all old versions and is
not broken.

On commit of project1 it will immediately point to "abcdef" an is not
broken

superproject will till point on all old versions and is not broken.

On commit of super project if we have one every thing is up to date.

	Henri GEIST

Re: tracking submodules out of main directory.

[henri GEIST]

Le lundi 27 juin 2011 à 12:05 -0700, Junio C Hamano a écrit :
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
> > Am 27.06.2011 18:51, schrieb Junio C Hamano:
> >> One possible working tree organization may look like this:
> >> 
> >> 	-+- lib1
> >>          +- project1/Makefile -- refers to ../lib1
> >>          +- project2/Makefile -- refers to ../lib1
> > ...
> >> An interesting point your situation raises is that there is no direct way
> >> to express module dependencies in .gitmodules file right now, I think.
> >> Ideally you would want "submodule init project1" to infer automatically
> >> that project1 needs lib1 and run "submodule init lib1" for you. My gut
> >> feeling is that it belongs to .gitmodules of the superproject
> >
> > That is where this is handled now, but having a submodule refer to a
> > submodule outside of it as a dependency is an interesting thought. But
> > as that only matters at the moment you add project1 (and it won't compile
> > because ../lib1 is missing, which can easily handled by: "oh, then I have
> > to add lib1 as a submodule to the superproject too"), ...
> 
> That is what I called "there is no direct way". Wouldn't it be nicer if
> the .gitmodules file in the superproject said something like
> 
> 	[module "project one"]
> 		path = project1
>         	url = ...
>                 depends = lib1
> 	[module "lib1"]
>         	path = lib1
>                 url = ...
> 
> and then "git submodule init project1" run by the end user implied running
> also "git submodule init lib1"?
> 
> 

This could be a way if .gitmodules can contain something like

	[module "project one"]
		path = project1
        	url = ...
                depends = lib1	"123456"
	[module "project two"]
		path = project2
        	url = ...
                depends = lib1	"abcdef"
	[module "lib1"]
        	path = lib1
                url = ...



Henri GEIST

Re: tracking submodules out of main directory.

[henri GEIST]

Le lundi 27 juin 2011 à 21:40 +0200, Jens Lehmann a écrit :
> Am 27.06.2011 21:05, schrieb Junio C Hamano:
> > Jens Lehmann <Jens.Lehmann@web.de> writes:
> > 
> >> Am 27.06.2011 18:51, schrieb Junio C Hamano:
> >>> One possible working tree organization may look like this:
> >>>
> >>> 	-+- lib1
> >>>          +- project1/Makefile -- refers to ../lib1
> >>>          +- project2/Makefile -- refers to ../lib1
> >> ...
> >>> An interesting point your situation raises is that there is no direct way
> >>> to express module dependencies in .gitmodules file right now, I think.
> >>> Ideally you would want "submodule init project1" to infer automatically
> >>> that project1 needs lib1 and run "submodule init lib1" for you. My gut
> >>> feeling is that it belongs to .gitmodules of the superproject
> >>
> >> That is where this is handled now, but having a submodule refer to a
> >> submodule outside of it as a dependency is an interesting thought. But
> >> as that only matters at the moment you add project1 (and it won't compile
> >> because ../lib1 is missing, which can easily handled by: "oh, then I have
> >> to add lib1 as a submodule to the superproject too"), ...
> > 
> > That is what I called "there is no direct way". Wouldn't it be nicer if
> > the .gitmodules file in the superproject said something like
> > 
> > 	[module "project one"]
> > 		path = project1
> >         	url = ...
> >                 depends = lib1
> > 	[module "lib1"]
> >         	path = lib1
> >                 url = ...
> > 
> > and then "git submodule init project1" run by the end user implied running
> > also "git submodule init lib1"?
> 
> And if lib1 happens to have another dependency, that will be initialized
> too? That would make life much easier for users who only want certain
> submodules populated to work on, as they won't have to chase compile
> errors anymore until they found all necessary submodules ... very nice.
> 

That is right.
But could also be done with the .gitmodules in project1 containing

	[module "lib1"]
        	path = ../lib1
                url = ...

and making implicite if the module is describe in .gitmodules that
mean ./ depend on it.

	Henri GEIST

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 27.06.2011 23:51, schrieb henri GEIST:
> Le lundi 27 juin 2011 à 12:05 -0700, Junio C Hamano a écrit :
>> Jens Lehmann <Jens.Lehmann@web.de> writes:
>>
>>> Am 27.06.2011 18:51, schrieb Junio C Hamano:
>>>> One possible working tree organization may look like this:
>>>>
>>>> 	-+- lib1
>>>>          +- project1/Makefile -- refers to ../lib1
>>>>          +- project2/Makefile -- refers to ../lib1
>>> ...
>>>> An interesting point your situation raises is that there is no direct way
>>>> to express module dependencies in .gitmodules file right now, I think.
>>>> Ideally you would want "submodule init project1" to infer automatically
>>>> that project1 needs lib1 and run "submodule init lib1" for you. My gut
>>>> feeling is that it belongs to .gitmodules of the superproject
>>>
>>> That is where this is handled now, but having a submodule refer to a
>>> submodule outside of it as a dependency is an interesting thought. But
>>> as that only matters at the moment you add project1 (and it won't compile
>>> because ../lib1 is missing, which can easily handled by: "oh, then I have
>>> to add lib1 as a submodule to the superproject too"), ...
>>
>> That is what I called "there is no direct way". Wouldn't it be nicer if
>> the .gitmodules file in the superproject said something like
>>
>> 	[module "project one"]
>> 		path = project1
>>         	url = ...
>>                 depends = lib1
>> 	[module "lib1"]
>>         	path = lib1
>>                 url = ...
>>
>> and then "git submodule init project1" run by the end user implied running
>> also "git submodule init lib1"?
>>
>>
> 
> This could be a way if .gitmodules can contain something like
> 
> 	[module "project one"]
> 		path = project1
>         	url = ...
>                 depends = lib1	"123456"
> 	[module "project two"]
> 		path = project2
>         	url = ...
>                 depends = lib1	"abcdef"
> 	[module "lib1"]
>         	path = lib1
>                 url = ...

But there is no need for that as "123456" and "abcdef" are already present
as the sha1 values for the gitlinks "lib1" and "lib2".

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 27.06.2011 23:57, schrieb henri GEIST:
> Le lundi 27 juin 2011 à 21:40 +0200, Jens Lehmann a écrit :
>> Am 27.06.2011 21:05, schrieb Junio C Hamano:
>>> Jens Lehmann <Jens.Lehmann@web.de> writes:
>>>
>>>> Am 27.06.2011 18:51, schrieb Junio C Hamano:
>>>>> One possible working tree organization may look like this:
>>>>>
>>>>> 	-+- lib1
>>>>>          +- project1/Makefile -- refers to ../lib1
>>>>>          +- project2/Makefile -- refers to ../lib1
>>>> ...
>>>>> An interesting point your situation raises is that there is no direct way
>>>>> to express module dependencies in .gitmodules file right now, I think.
>>>>> Ideally you would want "submodule init project1" to infer automatically
>>>>> that project1 needs lib1 and run "submodule init lib1" for you. My gut
>>>>> feeling is that it belongs to .gitmodules of the superproject
>>>>
>>>> That is where this is handled now, but having a submodule refer to a
>>>> submodule outside of it as a dependency is an interesting thought. But
>>>> as that only matters at the moment you add project1 (and it won't compile
>>>> because ../lib1 is missing, which can easily handled by: "oh, then I have
>>>> to add lib1 as a submodule to the superproject too"), ...
>>>
>>> That is what I called "there is no direct way". Wouldn't it be nicer if
>>> the .gitmodules file in the superproject said something like
>>>
>>> 	[module "project one"]
>>> 		path = project1
>>>         	url = ...
>>>                 depends = lib1
>>> 	[module "lib1"]
>>>         	path = lib1
>>>                 url = ...
>>>
>>> and then "git submodule init project1" run by the end user implied running
>>> also "git submodule init lib1"?
>>
>> And if lib1 happens to have another dependency, that will be initialized
>> too? That would make life much easier for users who only want certain
>> submodules populated to work on, as they won't have to chase compile
>> errors anymore until they found all necessary submodules ... very nice.
>>
> 
> That is right.
> But could also be done with the .gitmodules in project1 containing
> 
> 	[module "lib1"]
>         	path = ../lib1
>                 url = ...
> 
> and making implicite if the module is describe in .gitmodules that
> mean ./ depend on it.

But that would be much more complicated than just recording that in the
superproject's .gitmodules. In a Git repo everything lives in a directory
tree, if you want to reference something outside of that tree things will
get very complicated ... and I'm not sure the benefit is worth the hassle.

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 28.06.2011 09:20, schrieb Jens Lehmann:
>> This could be a way if .gitmodules can contain something like
>>
>> 	[module "project one"]
>> 		path = project1
>>         	url = ...
>>                 depends = lib1	"123456"
>> 	[module "project two"]
>> 		path = project2
>>         	url = ...
>>                 depends = lib1	"abcdef"
>> 	[module "lib1"]
>>         	path = lib1
>>                 url = ...
> 
> But there is no need for that as "123456" and "abcdef" are already present
> as the sha1 values for the gitlinks "lib1" and "lib2".

... but of course you would have to decide which sha1 is the right one for
lib1, while your proposal offers two contradicting sha1s (note to self: must
drink coffee *before* answering emails in the morning ;-)

Re: tracking submodules out of main directory.

[Alexei Sholik]

On 27 June 2011 22:05, Junio C Hamano <gitster@pobox.com> wrote:
> That is what I called "there is no direct way". Wouldn't it be nicer if
> the .gitmodules file in the superproject said something like
>
>        [module "project one"]
>                path = project1
>                url = ...
>                depends = lib1
>        [module "lib1"]
>                path = lib1
>                url = ...
>
> and then "git submodule init project1" run by the end user implied running
> also "git submodule init lib1"?

This is a very nice idea. In my workflow, I find it that I more often
need to clone a repo _including_ its submodules, because the top-level
project won't compile without them. If we had a way to specify
dependencies on submodules, `git pull` could automatically init and
update them.

If a user really wants to clone only the top-level repo without
submodules, git could provide him with an option for `git pull` (like
`git pull --shallow`) to do just that. I think this second scenario is
less common, so it is more reasonable to have a '--shallow' option for
it, instead of '--recursive' counterpart.

-- 
Best regards,
Alexei Sholik

Re: tracking submodules out of main directory.

[henri GEIST]

Le mardi 28 juin 2011 à 09:20 +0200, Jens Lehmann a écrit :
> Am 27.06.2011 23:51, schrieb henri GEIST:
> > Le lundi 27 juin 2011 à 12:05 -0700, Junio C Hamano a écrit :
> >> Jens Lehmann <Jens.Lehmann@web.de> writes:
> >>
> >>> Am 27.06.2011 18:51, schrieb Junio C Hamano:
> >>>> One possible working tree organization may look like this:
> >>>>
> >>>> 	-+- lib1
> >>>>          +- project1/Makefile -- refers to ../lib1
> >>>>          +- project2/Makefile -- refers to ../lib1
> >>> ...
> >>>> An interesting point your situation raises is that there is no direct way
> >>>> to express module dependencies in .gitmodules file right now, I think.
> >>>> Ideally you would want "submodule init project1" to infer automatically
> >>>> that project1 needs lib1 and run "submodule init lib1" for you. My gut
> >>>> feeling is that it belongs to .gitmodules of the superproject
> >>>
> >>> That is where this is handled now, but having a submodule refer to a
> >>> submodule outside of it as a dependency is an interesting thought. But
> >>> as that only matters at the moment you add project1 (and it won't compile
> >>> because ../lib1 is missing, which can easily handled by: "oh, then I have
> >>> to add lib1 as a submodule to the superproject too"), ...
> >>
> >> That is what I called "there is no direct way". Wouldn't it be nicer if
> >> the .gitmodules file in the superproject said something like
> >>
> >> 	[module "project one"]
> >> 		path = project1
> >>         	url = ...
> >>                 depends = lib1
> >> 	[module "lib1"]
> >>         	path = lib1
> >>                 url = ...
> >>
> >> and then "git submodule init project1" run by the end user implied running
> >> also "git submodule init lib1"?
> >>
> >>
> > 
> > This could be a way if .gitmodules can contain something like
> > 
> > 	[module "project one"]
> > 		path = project1
> >         	url = ...
> >                 depends = lib1	"123456"
> > 	[module "project two"]
> > 		path = project2
> >         	url = ...
> >                 depends = lib1	"abcdef"
> > 	[module "lib1"]
> >         	path = lib1
> >                 url = ...
> 
> But there is no need for that as "123456" and "abcdef" are already present
> as the sha1 values for the gitlinks "lib1" and "lib2".
> 

For what I understand of gitlinks reading the git source code it
effectively seem to be the functionality I am looking for.
It seem to be the pointer to the version of the module stored in the
index. Am I right ?

But I dont see a way to put 2 gitlinks pointing on 2 version of the same
submodule in the index of the superproject. And then specifying
gitlink_1 represent lib1 "123456" the dependency of project_1 and
gitlink_2 represent lib1 "abcdef" the dependency of project_2.

	Henri GEIST

Re: tracking submodules out of main directory.

[henri GEIST]

> >>>
> >>> That is what I called "there is no direct way". Wouldn't it be nicer if
> >>> the .gitmodules file in the superproject said something like
> >>>
> >>> 	[module "project one"]
> >>> 		path = project1
> >>>         	url = ...
> >>>                 depends = lib1
> >>> 	[module "lib1"]
> >>>         	path = lib1
> >>>                 url = ...
> >>>
> >>> and then "git submodule init project1" run by the end user implied running
> >>> also "git submodule init lib1"?
> >>
> >> And if lib1 happens to have another dependency, that will be initialized
> >> too? That would make life much easier for users who only want certain
> >> submodules populated to work on, as they won't have to chase compile
> >> errors anymore until they found all necessary submodules ... very nice.
> >>
> > 
> > That is right.
> > But could also be done with the .gitmodules in project1 containing
> > 
> > 	[module "lib1"]
> >         	path = ../lib1
> >                 url = ...
> > 
> > and making implicite if the module is describe in .gitmodules that
> > mean ./ depend on it.
> 
> But that would be much more complicated than just recording that in the
> superproject's .gitmodules. In a Git repo everything lives in a directory
> tree, if you want to reference something outside of that tree things will
> get very complicated ... and I'm not sure the benefit is worth the hassle.
> 

I am actually working on the git source code to give it a try and see if
I can managed to make a patch without breaking anything else.

	Henri GEIST

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 28.06.2011 12:05, schrieb Alexei Sholik:
> On 27 June 2011 22:05, Junio C Hamano <gitster@pobox.com> wrote:
>> That is what I called "there is no direct way". Wouldn't it be nicer if
>> the .gitmodules file in the superproject said something like
>>
>>        [module "project one"]
>>                path = project1
>>                url = ...
>>                depends = lib1
>>        [module "lib1"]
>>                path = lib1
>>                url = ...
>>
>> and then "git submodule init project1" run by the end user implied running
>> also "git submodule init lib1"?
> 
> This is a very nice idea. In my workflow, I find it that I more often
> need to clone a repo _including_ its submodules, because the top-level
> project won't compile without them. If we had a way to specify
> dependencies on submodules, `git pull` could automatically init and
> update them.

While Junio's proposition was about initializing submodules that live
next to each other you seem to be interested in recursive initialization.
My plan is to add a flag to .gitmodules, say "autoinit", which will, when
set, lead to a initialized and checked out submodule on clone.

> If a user really wants to clone only the top-level repo without
> submodules, git could provide him with an option for `git pull` (like
> `git pull --shallow`) to do just that. I think this second scenario is
> less common, so it is more reasonable to have a '--shallow' option for
> it, instead of '--recursive' counterpart.

Apart from the name, where I think "--recurse-submodules=none" is more
consistent with the other options we have, this is the plan when clone
learns to check out submodules  too.

Re: tracking submodules out of main directory.

[henri GEIST]

	Hello,

sorry for the delay but here is a patch to include the feature.
I work with it and it full fill my needs.

In hope that some one else will find it useful.

	Henri GEIST


>From 09b7cda863c4443d11808f5b3f1a46ce05aa1e0d Mon Sep 17 00:00:00 2001
From: Henri GEIST <henri@flying-robots.com>
Date: Wed, 27 Jul 2011 20:16:13 +0200
Subject: [PATCH] Enabeling (sub)modules linking out of repository.

Depending on there workflow somme people needs to link there repository
to depend on external modules.
Just like library could be shared by different programme and other
libraries.
It's a mean to track dependency between source code projects.

In the current code it was not possible to add a gitlink to a repository
outside of the main repository.

This pach :
  - Enable adding an external git directory.
  - Still forbid to add anything else.
  - Take care of prohibitting git to overright any data outside of the
    current directory.
  - Incrase some tests to validate the new feature.

This way you can have :
  - Project depending of multiple subprojects themselves depending on
    one third rank common suproject without clash at compilation
    linking.
  - Confidance that all suproject use the same version of the third rank
    subroject (git status will tell you.)
  - All subproject and subsubproject could be esealy worked on there one
    and synchronized after in the big project.

Signed-off-by: Henri GEIST <henri@flying-robots.com>
---
 builtin/clean.c       |    6 ++++--
 path.c                |   20 +++++++++++++++++---
 read-cache.c          |   14 ++++++++++++++
 setup.c               |   10 ++++++++--
 t/t0060-path-utils.sh |   16 ++++++++--------
 test-path-utils.c     |    6 ++++--
 6 files changed, 55 insertions(+), 17 deletions(-)

diff --git a/builtin/clean.c b/builtin/clean.c
index 75697f7..e234e5d 100644
--- a/builtin/clean.c
+++ b/builtin/clean.c
@@ -149,8 +149,10 @@ int cmd_clean(int argc, const char **argv, const char *prefix)
 		if (S_ISDIR(st.st_mode)) {
 			strbuf_addstr(&directory, ent->name);
 			qname = quote_path_relative(directory.buf, directory.len, &buf, prefix);
-			if (show_only && (remove_directories ||
-			    (matches == MATCHED_EXACTLY))) {
+			if (!strncmp ("../", qname, 3)) {
+				die("'%s' is outside repository", qname);
+			} else if (show_only && (remove_directories ||
+				   (matches == MATCHED_EXACTLY))) {
 				printf(_("Would remove %s\n"), qname);
 			} else if (remove_directories ||
 				   (matches == MATCHED_EXACTLY)) {
diff --git a/path.c b/path.c
index 4d73cc9..daf1573 100644
--- a/path.c
+++ b/path.c
@@ -444,7 +444,8 @@ const char *relative_path(const char *abs, const char *base)
  * - Removes "." components.
  * - Removes ".." components, and the components the precede them.
  * Returns failure (non-zero) if a ".." component appears as first path
- * component anytime during the normalization. Otherwise, returns success (0).
+ * component but finish the normalization before.
+ * Otherwise, returns success (0).
  *
  * Note that this function is purely textual.  It does not follow symlinks,
  * verify the existence of the path, or make any system calls.
@@ -519,13 +520,26 @@ int normalize_path_copy(char *dst, const char *src)
 		 * go up one level.
 		 */
 		dst--;	/* go to trailing '/' */
-		if (dst <= dst0)
-			return -1;
+		if (dst <= dst0
+		    || (dst0 + 2 <= dst
+			&& dst[-1] == '.' && dst[-2] == '.'
+			&& (dst0 + 2 == dst || dst[-3] == '/'))) {
+			dst++;
+			*dst++ = '.';
+			*dst++ = '.';
+			*dst++ = '/';
+			continue;
+		}
 		/* Windows: dst[-1] cannot be backslash anymore */
 		while (dst0 < dst && dst[-1] != '/')
 			dst--;
 	}
 	*dst = '\0';
+	if (*dst0 == '/')
+		dst0++;
+	if (2 <= strlen (dst0)
+	    && dst0[0] == '.' && dst0[1] == '.' && dst0[2] == '/')
+		return -1;
 	return 0;
 }
 
diff --git a/read-cache.c b/read-cache.c
index 46a9e60..7fb695a 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -753,6 +753,20 @@ static int verify_dotfile(const char *rest)
 int verify_path(const char *path)
 {
 	char c;
+	struct stat buf;
+
+	lstat (path, &buf);
+	if (buf.st_mode & S_IFDIR) {
+		for (;;) {
+			if (path[0] != '.')
+				break;
+			if (path[1] != '.')
+				break;
+			if (path[2] != '/')
+				break;
+			path += 3;
+		}
+	}
 
 	if (has_dos_drive_prefix(path))
 		return 0;
diff --git a/setup.c b/setup.c
index 5ea5502..ce7993e 100644
--- a/setup.c
+++ b/setup.c
@@ -8,6 +8,8 @@ char *prefix_path(const char *prefix, int len, const char *path)
 {
 	const char *orig = path;
 	char *sanitized;
+	struct stat buf;
+
 	if (is_absolute_path(orig)) {
 		const char *temp = real_path(path);
 		sanitized = xmalloc(len + strlen(temp) + 1);
@@ -18,8 +20,12 @@ char *prefix_path(const char *prefix, int len, const char *path)
 			memcpy(sanitized, prefix, len);
 		strcpy(sanitized + len, path);
 	}
-	if (normalize_path_copy(sanitized, sanitized))
-		goto error_out;
+	if (normalize_path_copy(sanitized, sanitized)) {
+		if (0 != lstat(sanitized, &buf))
+			goto error_out;
+		if (!(buf.st_mode & S_IFDIR))
+			goto error_out;
+	}
 	if (is_absolute_path(orig)) {
 		size_t root_len, len, total;
 		const char *work_tree = get_git_work_tree();
diff --git a/t/t0060-path-utils.sh b/t/t0060-path-utils.sh
index 53cf1f8..b4b9b1f 100755
--- a/t/t0060-path-utils.sh
+++ b/t/t0060-path-utils.sh
@@ -48,12 +48,12 @@ norm_path "" ""
 norm_path . ""
 norm_path ./ ""
 norm_path ./. ""
-norm_path ./.. ++failed++
-norm_path ../. ++failed++
-norm_path ./../.// ++failed++
+norm_path ./.. "../ ++failed++"
+norm_path ../. "../ ++failed++"
+norm_path ./../.// "../ ++failed++"
 norm_path dir/.. ""
 norm_path dir/sub/../.. ""
-norm_path dir/sub/../../.. ++failed++
+norm_path dir/sub/../../.. "../ ++failed++"
 norm_path dir dir
 norm_path dir// dir/
 norm_path ./dir dir
@@ -73,12 +73,12 @@ norm_path // / POSIX
 norm_path /// / POSIX
 norm_path /. / POSIX
 norm_path /./ / POSIX
-norm_path /./.. ++failed++ POSIX
-norm_path /../. ++failed++ POSIX
-norm_path /./../.// ++failed++ POSIX
+norm_path /./.. "/../ ++failed++" POSIX
+norm_path /../. "/../ ++failed++" POSIX
+norm_path /./../.// "/../ ++failed++" POSIX
 norm_path /dir/.. / POSIX
 norm_path /dir/sub/../.. / POSIX
-norm_path /dir/sub/../../.. ++failed++ POSIX
+norm_path /dir/sub/../../.. "/../ ++failed++" POSIX
 norm_path /dir /dir POSIX
 norm_path /dir// /dir/ POSIX
 norm_path /./dir /dir POSIX
diff --git a/test-path-utils.c b/test-path-utils.c
index e767159..ba6c8ac 100644
--- a/test-path-utils.c
+++ b/test-path-utils.c
@@ -5,8 +5,10 @@ int main(int argc, char **argv)
 	if (argc == 3 && !strcmp(argv[1], "normalize_path_copy")) {
 		char *buf = xmalloc(PATH_MAX + 1);
 		int rv = normalize_path_copy(buf, argv[2]);
-		if (rv)
-			buf = "++failed++";
+		if (rv) {
+			fputs(buf, stdout);
+			buf = " ++failed++";
+		}
 		puts(buf);
 		return 0;
 	}
-- 
1.7.2.5

Re: tracking submodules out of main directory.

[henri GEIST]

	Hello,

The same with some orthographique corrections by
Sverre Rabbelier <srabbelier@gmail.com> (unofficial)


>From 9a4e10fbb884583638c1651cc6cb98e0ce59cb9b Mon Sep 17 00:00:00 2001
From: Henri GEIST <henri@flying-robots.com>
Date: Thu, 28 Jul 2011 10:15:55 +0200
Subject: [PATCH] Enabeling (sub)modules linking out of repository.

Depending on there workflow some people needs to link there repository
to depend on external modules.
Just like library could be shared by different program and other libraries.
It's a means to track dependency between source code projects.

In the current code it was not possible to add a gitlink to a repository
outside of the main repository.

This pach :
  - Enables adding an external git directory.
  - Still forbids to add anything else.
  - Takes care of prohibitting git to overwrite any data outside of the
    current directory.
  - Increase some tests to validate the new feature.

This way you can have :
  - Project depending of multiple subprojects themselves depending on
    one third rank common suproject without clashing at compilation
    linking.
  - Confidence that all suproject use the same version of the third rank
    subproject (git status will tell you.)
  - All subproject and subsubproject could be easily worked on there one
    and synchronized after in the big project.

Signed-off-by: Henri GEIST <henri@flying-robots.com>
---
 builtin/clean.c       |    6 ++++--
 path.c                |   20 +++++++++++++++++---
 read-cache.c          |   14 ++++++++++++++
 setup.c               |   10 ++++++++--
 t/t0060-path-utils.sh |   16 ++++++++--------
 test-path-utils.c     |    6 ++++--
 6 files changed, 55 insertions(+), 17 deletions(-)

diff --git a/builtin/clean.c b/builtin/clean.c
index 75697f7..e234e5d 100644
--- a/builtin/clean.c
+++ b/builtin/clean.c
@@ -149,8 +149,10 @@ int cmd_clean(int argc, const char **argv, const char *prefix)
 		if (S_ISDIR(st.st_mode)) {
 			strbuf_addstr(&directory, ent->name);
 			qname = quote_path_relative(directory.buf, directory.len, &buf, prefix);
-			if (show_only && (remove_directories ||
-			    (matches == MATCHED_EXACTLY))) {
+			if (!strncmp ("../", qname, 3)) {
+				die("'%s' is outside repository", qname);
+			} else if (show_only && (remove_directories ||
+				   (matches == MATCHED_EXACTLY))) {
 				printf(_("Would remove %s\n"), qname);
 			} else if (remove_directories ||
 				   (matches == MATCHED_EXACTLY)) {
diff --git a/path.c b/path.c
index 4d73cc9..daf1573 100644
--- a/path.c
+++ b/path.c
@@ -444,7 +444,8 @@ const char *relative_path(const char *abs, const char *base)
  * - Removes "." components.
  * - Removes ".." components, and the components the precede them.
  * Returns failure (non-zero) if a ".." component appears as first path
- * component anytime during the normalization. Otherwise, returns success (0).
+ * component but finish the normalization before.
+ * Otherwise, returns success (0).
  *
  * Note that this function is purely textual.  It does not follow symlinks,
  * verify the existence of the path, or make any system calls.
@@ -519,13 +520,26 @@ int normalize_path_copy(char *dst, const char *src)
 		 * go up one level.
 		 */
 		dst--;	/* go to trailing '/' */
-		if (dst <= dst0)
-			return -1;
+		if (dst <= dst0
+		    || (dst0 + 2 <= dst
+			&& dst[-1] == '.' && dst[-2] == '.'
+			&& (dst0 + 2 == dst || dst[-3] == '/'))) {
+			dst++;
+			*dst++ = '.';
+			*dst++ = '.';
+			*dst++ = '/';
+			continue;
+		}
 		/* Windows: dst[-1] cannot be backslash anymore */
 		while (dst0 < dst && dst[-1] != '/')
 			dst--;
 	}
 	*dst = '\0';
+	if (*dst0 == '/')
+		dst0++;
+	if (2 <= strlen (dst0)
+	    && dst0[0] == '.' && dst0[1] == '.' && dst0[2] == '/')
+		return -1;
 	return 0;
 }
 
diff --git a/read-cache.c b/read-cache.c
index 46a9e60..7fb695a 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -753,6 +753,20 @@ static int verify_dotfile(const char *rest)
 int verify_path(const char *path)
 {
 	char c;
+	struct stat buf;
+
+	lstat (path, &buf);
+	if (buf.st_mode & S_IFDIR) {
+		for (;;) {
+			if (path[0] != '.')
+				break;
+			if (path[1] != '.')
+				break;
+			if (path[2] != '/')
+				break;
+			path += 3;
+		}
+	}
 
 	if (has_dos_drive_prefix(path))
 		return 0;
diff --git a/setup.c b/setup.c
index 5ea5502..ce7993e 100644
--- a/setup.c
+++ b/setup.c
@@ -8,6 +8,8 @@ char *prefix_path(const char *prefix, int len, const char *path)
 {
 	const char *orig = path;
 	char *sanitized;
+	struct stat buf;
+
 	if (is_absolute_path(orig)) {
 		const char *temp = real_path(path);
 		sanitized = xmalloc(len + strlen(temp) + 1);
@@ -18,8 +20,12 @@ char *prefix_path(const char *prefix, int len, const char *path)
 			memcpy(sanitized, prefix, len);
 		strcpy(sanitized + len, path);
 	}
-	if (normalize_path_copy(sanitized, sanitized))
-		goto error_out;
+	if (normalize_path_copy(sanitized, sanitized)) {
+		if (0 != lstat(sanitized, &buf))
+			goto error_out;
+		if (!(buf.st_mode & S_IFDIR))
+			goto error_out;
+	}
 	if (is_absolute_path(orig)) {
 		size_t root_len, len, total;
 		const char *work_tree = get_git_work_tree();
diff --git a/t/t0060-path-utils.sh b/t/t0060-path-utils.sh
index 53cf1f8..b4b9b1f 100755
--- a/t/t0060-path-utils.sh
+++ b/t/t0060-path-utils.sh
@@ -48,12 +48,12 @@ norm_path "" ""
 norm_path . ""
 norm_path ./ ""
 norm_path ./. ""
-norm_path ./.. ++failed++
-norm_path ../. ++failed++
-norm_path ./../.// ++failed++
+norm_path ./.. "../ ++failed++"
+norm_path ../. "../ ++failed++"
+norm_path ./../.// "../ ++failed++"
 norm_path dir/.. ""
 norm_path dir/sub/../.. ""
-norm_path dir/sub/../../.. ++failed++
+norm_path dir/sub/../../.. "../ ++failed++"
 norm_path dir dir
 norm_path dir// dir/
 norm_path ./dir dir
@@ -73,12 +73,12 @@ norm_path // / POSIX
 norm_path /// / POSIX
 norm_path /. / POSIX
 norm_path /./ / POSIX
-norm_path /./.. ++failed++ POSIX
-norm_path /../. ++failed++ POSIX
-norm_path /./../.// ++failed++ POSIX
+norm_path /./.. "/../ ++failed++" POSIX
+norm_path /../. "/../ ++failed++" POSIX
+norm_path /./../.// "/../ ++failed++" POSIX
 norm_path /dir/.. / POSIX
 norm_path /dir/sub/../.. / POSIX
-norm_path /dir/sub/../../.. ++failed++ POSIX
+norm_path /dir/sub/../../.. "/../ ++failed++" POSIX
 norm_path /dir /dir POSIX
 norm_path /dir// /dir/ POSIX
 norm_path /./dir /dir POSIX
diff --git a/test-path-utils.c b/test-path-utils.c
index e767159..ba6c8ac 100644
--- a/test-path-utils.c
+++ b/test-path-utils.c
@@ -5,8 +5,10 @@ int main(int argc, char **argv)
 	if (argc == 3 && !strcmp(argv[1], "normalize_path_copy")) {
 		char *buf = xmalloc(PATH_MAX + 1);
 		int rv = normalize_path_copy(buf, argv[2]);
-		if (rv)
-			buf = "++failed++";
+		if (rv) {
+			fputs(buf, stdout);
+			buf = " ++failed++";
+		}
 		puts(buf);
 		return 0;
 	}
-- 
1.7.2.5

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 28.07.2011 10:57, schrieb henri GEIST:
> In the current code it was not possible to add a gitlink to a repository
> outside of the main repository.

Me thinks this is a *feature* this patch removes (as I understand it it was
a major design decision that everything /inside/ a directory is controlled
by git).

> This pach :
<snip>
>   - Still forbids to add anything else.

Why? If you let submodules live outside the tree I don't see any reason why
regular files shouldn't live there too (Disclaimer: I d not think that would
be a good idea either ;-).

What you want looks like this:

-+- lib1    #registered as submodule of project1 *and* project2 but not here
 +- project1            # submodule of the superproject
 |  +- ../lib1
 +- project2            # submodule of the superproject
    +- ../lib1

You are opening a can of worms by having two different repos point to the same
submodule living in a third repo (which also happens to be their superproject
and must somehow ignore it). You'll have two SHA1s for a single submodule;
"git submodule foreach --recursive" will have interesting results too; and so
on. Not good.

What about solving that with a "ln -s ../lib1" in "project1" and "project2"
(you seem to need that for your build environment) and adding the submodule
"lib1" to the superproject just like "project1" and "project2"?

Re: tracking submodules out of main directory.

[henri GEIST]

Le jeudi 28 juillet 2011 à 18:48 +0200, Jens Lehmann a écrit :
> Am 28.07.2011 10:57, schrieb henri GEIST:
> > In the current code it was not possible to add a gitlink to a repository
> > outside of the main repository.
> 

Of course that is why I have done the patch.

> Me thinks this is a *feature* this patch removes (as I understand it it was
> a major design decision that everything /inside/ a directory is controlled
> by git).
> 

That is still the case.
It is not a matter of disabling any control of git in its own
repository.
It is just a matter of adding inside the git repository a reference
(dependency) to an other git repository.

> > This pach :
> <snip>
> >   - Still forbids to add anything else.
> 
> Why? If you let submodules live outside the tree I don't see any reason why
> regular files shouldn't live there too (Disclaimer: I d not think that would
> be a good idea either ;-).
> 

there is regular file in the submodule repository but they are controled
by the submodule itself. Then the main repository do not need to handle
them.

Just like you I think that regular file should not be touche outside of
the repository by git.
Because in this case it is not just a reference that is managed but the
file itself. And this way there is a risk to overwrite some data not
under revision control outside of the repository.

> What you want looks like this:
> 
> -+- lib1    #registered as submodule of project1 *and* project2 but not here
>  +- project1            # submodule of the superproject
>  |  +- ../lib1
>  +- project2            # submodule of the superproject
>     +- ../lib1
> 

In fact no.
What I want is to have this:

-+- lib1
 +- project1
 +- project2

With :
 - lib1 not knowing anything about projects 1 & 2
 - project1 not knowing anything about project2 and vice et versa
 - project1 knowing that it need lib1 in version N.
 - project2 knowing that it need lib1 in version M.

I know that means a conflict in the required lib version but it is
required by the independence of projects 1 & 2 which do not have to know
what happen in the other one.

And in fact it is just what I want, it enable me if I decide to work on
an optional "BigProject" depending on both project 1 & 2.

Then If lib1 is in version M:
 - a git status in project2 will say nothing
 - a git status in project1 will say
   "modified:   ../lib1 (modified content)
 - a git status in BigProject will say
   "modified:   ../project1 (modified content)

Then I know that I need to update project1 to work with the last version
M of lib1.

> You are opening a can of worms by having two different repos point to the same
> submodule living in a third repo (which also happens to be their superproject
> and must somehow ignore it). You'll have two SHA1s for a single submodule;
> "git submodule foreach --recursive" will have interesting results too; and so
> on. Not good.
> 

As I just said before it is my purpose to do it like that.

If you want to avoid this just put (project1 lib1) and (project2 lib2)
in different places on your hard disk

Let say a concret exemple

3 different teams work on libtiff, libpng, and libjpeg they are totally
unrelated.

One more team is working on the "gimp". And they need those 3 libs in
specific versions not necessarily there heads.

One other unrelated team is working on "gqview" and need the same libs
in other specifics versions (Why should they know what te gimp team
does)

Neither "gimp" and "gqview" project will contain directory with those
libs inside. They just depend on them.

And the last team work on the gnome project which need the "gimp" and
"gqview". It will be this team witch have to care about having both
"gimp" and "gqview" sharing the same libs version>
And has well the gnome project will not contain "gqview" and "gimp" in
its own tree.
It will also depend on them.

It is just the same with aptitude on debian.
Each package know there dependency by themselves, does not contain there
dependencies, and do not need a bigger superpackage to tell them what
are there own dependencies.

> What about solving that with a "ln -s ../lib1" in "project1" and "project2"
> (you seem to need that for your build environment) and adding the submodule
> "lib1" to the superproject just like "project1" and "project2"?
> 

For my build environment I do not use simlinks.
I use tu put :

#include "../lib1/lib1.h"
in project1/project1.c

And even if I do not face this problem myself, simlinks do not work so
well on Windows.

And Still I realy want to have every project knowing there own
dependency by themselves and not needing an external superproject to
tell them what they need.

        Henri GEIST

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 29.07.2011 11:39, schrieb henri GEIST:
> Le jeudi 28 juillet 2011 à 18:48 +0200, Jens Lehmann a écrit :
>> Am 28.07.2011 10:57, schrieb henri GEIST:
>>> In the current code it was not possible to add a gitlink to a repository
>>> outside of the main repository.
> 
> Of course that is why I have done the patch.
> 
>> Me thinks this is a *feature* this patch removes (as I understand it it was
>> a major design decision that everything /inside/ a directory is controlled
>> by git).
> 
> That is still the case.

No, as "everything /inside/ a directory" also means "nothing /outside/ that
directory" in this context.

> It is not a matter of disabling any control of git in its own
> repository.
> It is just a matter of adding inside the git repository a reference
> (dependency) to an other git repository.

... which you want to have *outside* of the containing repository! That will
then be registered in other git repositories too in your model, which gets
rid of the "one file/submodule, one repo" assumption we now have and will
introduce ambiguities which are *really* hard to handle.

>>> This pach :
>> <snip>
>>>   - Still forbids to add anything else.
>>
>> Why? If you let submodules live outside the tree I don't see any reason why
>> regular files shouldn't live there too (Disclaimer: I d not think that would
>> be a good idea either ;-).
> 
> there is regular file in the submodule repository but they are controled
> by the submodule itself. Then the main repository do not need to handle
> them.
> 
> Just like you I think that regular file should not be touche outside of
> the repository by git.

No, I think we should /either/ allow gitlinks *and* files to live outside of
the repository /or/ *none* of them, as they have a lot in common (and I'm
working on making them to behave as similar as possible). And in the use case
you describe here it is totally irrelevant if you dependency consists of a
directory tree in a submodule (= a bunch of files) or just a single file (say
a header containing project wide definitions).

> Because in this case it is not just a reference that is managed but the
> file itself. And this way there is a risk to overwrite some data not
> under revision control outside of the repository.

You have the same risk when a gitlink points outside, as a submodule is a
way of controlling a bunch of files through that reference. And the file
would be under version control in the repository where it is registered, no?

>> What you want looks like this:
>>
>> -+- lib1    #registered as submodule of project1 *and* project2 but not here
>>  +- project1            # submodule of the superproject
>>  |  +- ../lib1
>>  +- project2            # submodule of the superproject
>>     +- ../lib1
> 
> In fact no.
> What I want is to have this:
> 
> -+- lib1
>  +- project1
>  +- project2
> 
> With :
>  - lib1 not knowing anything about projects 1 & 2
>  - project1 not knowing anything about project2 and vice et versa
>  - project1 knowing that it need lib1 in version N.
>  - project2 knowing that it need lib1 in version M.

Right, I should have explained the "../lib1" entries better.

> I know that means a conflict in the required lib version but it is
> required by the independence of projects 1 & 2 which do not have to know
> what happen in the other one.
> 
> And in fact it is just what I want, it enable me if I decide to work on
> an optional "BigProject" depending on both project 1 & 2.
> 
> Then If lib1 is in version M:
>  - a git status in project2 will say nothing
>  - a git status in project1 will say
>    "modified:   ../lib1 (modified content)
>  - a git status in BigProject will say
>    "modified:   ../project1 (modified content)
> 
> Then I know that I need to update project1 to work with the last version
> M of lib1.

Maybe no update for project1 is needed, because M only contains a bugfix
which doesn't even need a recompilation of project1. But now you need to
add a commit to project1 nonetheless with a message like "Updated lib1
with a bugfix which is needed by project2" which makes your idea of
independent submodules break down.

>> You are opening a can of worms by having two different repos point to the same
>> submodule living in a third repo (which also happens to be their superproject
>> and must somehow ignore it). You'll have two SHA1s for a single submodule;
>> "git submodule foreach --recursive" will have interesting results too; and so
>> on. Not good.
> 
> As I just said before it is my purpose to do it like that.

I understood that, but what are you proposing to do to solve all the
problems your approach introduces? You can't just hand wave them away.

> Let say a concret exemple
> 
> 3 different teams work on libtiff, libpng, and libjpeg they are totally
> unrelated.
> 
> One more team is working on the "gimp". And they need those 3 libs in
> specific versions not necessarily there heads.
> 
> One other unrelated team is working on "gqview" and need the same libs
> in other specifics versions (Why should they know what te gimp team
> does)
> 
> Neither "gimp" and "gqview" project will contain directory with those
> libs inside. They just depend on them.
> 
> And the last team work on the gnome project which need the "gimp" and
> "gqview". It will be this team witch have to care about having both
> "gimp" and "gqview" sharing the same libs version>
> And has well the gnome project will not contain "gqview" and "gimp" in
> its own tree.
> It will also depend on them.

Cool, that is a real life example resembling what we have a my dayjob. But
a "gimp" and "gqview" project will only have dependencies like "use libpng
of version 1.2.3 or newer (because we need a feature/bugfix introduced
there)" and won't be tied to a special version of that library. This means
they need a dependency like "SHA1 or newer" instead of "exactly this SHA1".

> It is just the same with aptitude on debian.
> Each package know there dependency by themselves, does not contain there
> dependencies, and do not need a bigger superpackage to tell them what
> are there own dependencies.

And this is a very good point for the "version x.yy-z *or newer*" argument,
they are /never/ tied to the /exact/ x.yy-z version, as that would make the
dependencies pretty much unusable. They use a "newer than x.yy-z" scheme.

>> What about solving that with a "ln -s ../lib1" in "project1" and "project2"
>> (you seem to need that for your build environment) and adding the submodule
>> "lib1" to the superproject just like "project1" and "project2"?
>>
> 
> For my build environment I do not use simlinks.
> I use tu put :
> 
> #include "../lib1/lib1.h"
> in project1/project1.c
> 
> And even if I do not face this problem myself, simlinks do not work so
> well on Windows.

Agreed.

> And Still I realy want to have every project knowing there own
> dependency by themselves and not needing an external superproject to
> tell them what they need.

I want to have that too! I'm just convinced using a gitlink to achieve that
is wrong in so many ways. I'd rather prefer to express such dependencies in
something like a config file, and I believe they should not be as strict as
"I need exactly that version" but rather like "this version or newer (and by
the way: we of course only tested that specific version ;-)". These
dependencies could then be checked and displayed by git status.

Re: tracking submodules out of main directory.

[henri GEIST]

Le samedi 30 juillet 2011 à 16:16 +0200, Jens Lehmann a écrit :
> Am 29.07.2011 11:39, schrieb henri GEIST:
> > Le jeudi 28 juillet 2011 à 18:48 +0200, Jens Lehmann a écrit :
> >> Am 28.07.2011 10:57, schrieb henri GEIST:
> > 
> > It is not a matter of disabling any control of git in its own
> > repository.
> > It is just a matter of adding inside the git repository a reference
> > (dependency) to an other git repository.
> 
> ... which you want to have *outside* of the containing repository!

yes

> That will then be registered in other git repositories too in your model,
> which gets rid of the "one file/submodule, one repo" assumption we now have
> and will introduce ambiguities which are *really* hard to handle.
> 

I am sorry, I am not a native English speaker. This sentence is to
complex for me. And google translator is of no help in this case.

> >>> This pach :
> >> <snip>
> >>>   - Still forbids to add anything else.
> >>
> >> Why? If you let submodules live outside the tree I don't see any reason why
> >> regular files shouldn't live there too (Disclaimer: I d not think that would
> >> be a good idea either ;-).
> > 
> > there is regular file in the submodule repository but they are controled
> > by the submodule itself. Then the main repository do not need to handle
> > them.
> > 
> > Just like you I think that regular file should not be touche outside of
> > the repository by git.
> 
> No, I think we should /either/ allow gitlinks *and* files to live outside of
> the repository /or/ *none* of them, as they have a lot in common (and I'm
> working on making them to behave as similar as possible). And in the use case
> you describe here it is totally irrelevant if you dependency consists of a
> directory tree in a submodule (= a bunch of files) or just a single file (say
> a header containing project wide definitions).
> 

Yes they have a lot in common but they still have a different nature.
git links pointer (not files on a hard-drive) to something which has a
special meaning for git, (a git repository).
Just like the .git directory which also have a special handling.

But I agree the step is really weak before enabling to put any regular
file outside of the directory.
I do not see any reasonable workflow (to my eyes) for it but' maybe some
day someone will came with a justifiable workflow which need it. we will
never know.

But in this case we need solve some questions :
  - Will we extend git status signaling untracked files out of the
    repository ?
  - What will do git-clean ? it is already dangerous inside the
    repository. and it will be worst if it can access outside of it.

May be some other points that I do not see for the moment.

> > Because in this case it is not just a reference that is managed but the
> > file itself. And this way there is a risk to overwrite some data not
> > under revision control outside of the repository.
> 
> You have the same risk when a gitlink points outside, as a submodule is a
> way of controlling a bunch of files through that reference. And the file
> would be under version control in the repository where it is registered, no?
> 

I agree on this point.

But they are still confined in an another git repository not
disseminated all over the file system.
And it never corrupt this pointed repository. just ask it to do by it's
own regular git commands.

In fact you can argue that it can disseminate some complete git
repository anywhere in the file system.
And you will be right. (nothing is perfect.)

I can do a second patch to prevent git submodule command to make clones
outside of the repository.
It will requires the user to do those clones manually.
In fact this is already what I do.
My only use of this is to track dependencies.
 
> > And in fact it is just what I want, it enable me if I decide to work on
> > an optional "BigProject" depending on both project 1 & 2.
> > 
> > Then If lib1 is in version M:
> >  - a git status in project2 will say nothing
> >  - a git status in project1 will say
> >    "modified:   ../lib1 (modified content)
> >  - a git status in BigProject will say
> >    "modified:   ../project1 (modified content)
> > 
> > Then I know that I need to update project1 to work with the last version
> > M of lib1.
> 
> Maybe no update for project1 is needed, because M only contains a bugfix
> which doesn't even need a recompilation of project1. But now you need to
> add a commit to project1 nonetheless with a message like "Updated lib1
> with a bugfix which is needed by project2" which makes your idea of
> independent submodules break down.
> 

In fact I work ni the world of "high integrity programming" then It is
just what I need.
If there is a bugfix in any library, used by the program it is no more
the same program.
I need the "SHA1" to correspond to the exact and complete source code
involved in my executable.

And this way the "SHA1" of the project sign the "SHA1" of the
libraries. 

> >> You are opening a can of worms by having two different repos point to the same
> >> submodule living in a third repo (which also happens to be their superproject
> >> and must somehow ignore it). You'll have two SHA1s for a single submodule;
> >> "git submodule foreach --recursive" will have interesting results too; and so
> >> on. Not good.
> > 
> > As I just said before it is my purpose to do it like that.
> 
> I understood that, but what are you proposing to do to solve all the
> problems your approach introduces? You can't just hand wave them away.
> 

There is some solutions :

  - First it is one more **feature** if it does not correspond to your
    work flow it does not prevent you to work exactly the way you did
    until now.

  - Second if you want to use the feature but not want to have the
    conflict **feature** (for me it is one), just put the independent
    project with there libs in different directory

      -+- foo -+- lib1     (in version N)
       |       +- project1
       |
       +- bar -+- lib1     (in version M)
               +- project2

  - Third if you really need to have project 1 & 2 in the same
    directory foo, that means they are needed by a third BigProject in
    the same directory foo depending on project 1 & 2.
    And then you really need git to declare a conflict.

> > Let say a concret exemple
> > 
> > 3 different teams work on libtiff, libpng, and libjpeg they are totally
> > unrelated.
> > 
> > One more team is working on the "gimp". And they need those 3 libs in
> > specific versions not necessarily there heads.
> > 
> > One other unrelated team is working on "gqview" and need the same libs
> > in other specifics versions (Why should they know what te gimp team
> > does)
> > 
> > Neither "gimp" and "gqview" project will contain directory with those
> > libs inside. They just depend on them.
> > 
> > And the last team work on the gnome project which need the "gimp" and
> > "gqview". It will be this team witch have to care about having both
> > "gimp" and "gqview" sharing the same libs version>
> > And has well the gnome project will not contain "gqview" and "gimp" in
> > its own tree.
> > It will also depend on them.
> 
> Cool, that is a real life example resembling what we have a my dayjob. But
> a "gimp" and "gqview" project will only have dependencies like "use libpng
> of version 1.2.3 or newer (because we need a feature/bugfix introduced
> there)" and won't be tied to a special version of that library. This means
> they need a dependency like "SHA1 or newer" instead of "exactly this SHA1".
> 

It is useful and simpler to work like this but could introduce some
bugs.

The "gimp" team has tested it with libpng 1.2.3 and maybe know that it
did not work with previous versions but if they do not have any crystal
ball they never know if newer versions will not break something.
In fact I doubt that the first version of gimp will work with the last
version of libpng.

> > It is just the same with aptitude on debian.
> > Each package know there dependency by themselves, does not contain there
> > dependencies, and do not need a bigger superpackage to tell them what
> > are there own dependencies.
> 
> And this is a very good point for the "version x.yy-z *or newer*" argument,
> they are /never/ tied to the /exact/ x.yy-z version, as that would make the
> dependencies pretty much unusable. They use a "newer than x.yy-z" scheme.
> 

It is an other feature that the one I need.
But it is a good idea.

Nothing prevent us to make a patch to add a new test in git status to
see if the current SHA1 in the libpng repository has the SHA1 of the
gitlink in the gimp in its ancestor.


> 
> > And Still I realy want to have every project knowing there own
> > dependency by themselves and not needing an external superproject to
> > tell them what they need.
> 
> I want to have that too! I'm just convinced using a gitlink to achieve that
> is wrong in so many ways. I'd rather prefer to express such dependencies in
> something like a config file, and I believe they should not be as strict as
> "I need exactly that version" but rather like "this version or newer (and by
> the way: we of course only tested that specific version ;-)". These
> dependencies could then be checked and displayed by git status.
> 

It effectively could be in a config file it seem good to me as well.

But if git handle this config file.
Update it on a "git add ../libpng && git commit"
And control the matching between the project and libraries on
"git status">

I can not see the difference with a gitlink.

	Henri GEIST

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 30.07.2011 23:55, schrieb henri GEIST:
> Le samedi 30 juillet 2011 à 16:16 +0200, Jens Lehmann a écrit :
>> Am 29.07.2011 11:39, schrieb henri GEIST:
>>> Le jeudi 28 juillet 2011 à 18:48 +0200, Jens Lehmann a écrit :
>>>> Am 28.07.2011 10:57, schrieb henri GEIST:
>>>
>>> It is not a matter of disabling any control of git in its own
>>> repository.
>>> It is just a matter of adding inside the git repository a reference
>>> (dependency) to an other git repository.
>>
>> ... which you want to have *outside* of the containing repository!
> 
> yes
> 
>> That will then be registered in other git repositories too in your model,
>> which gets rid of the "one file/submodule, one repo" assumption we now have
>> and will introduce ambiguities which are *really* hard to handle.
> 
> I am sorry, I am not a native English speaker. This sentence is to
> complex for me. And google translator is of no help in this case.

Your proposal of letting multiple gitlinks in different repos point to the
same submodule will break the assumption that each file is only handled by
a single git repo. For example when you have a conflict and do a "git
submodule update --recursive" in the superproject, the SHA1 used for "lib"
will depend on the alphabetical order of "project1" and "project2". And
normally after running "git submodule update --recursive" you expect all
submodules of the superproject to be clean. But your change breaks this
expectation, it will still contain unclean submodule entries even though
you just told git it should clean them. What will a "git submodule sync
--recursive" do when "project1" and "project2" use different urls in their
.gitmodules? And so on.

Commands won't always behave like you expect them to and sometimes will
give different results just because different names are used. That's what
I meant with ambiguities and that's why I don't think gitlinks are the
right method here.

> But I agree the step is really weak before enabling to put any regular
> file outside of the directory.
> I do not see any reasonable workflow (to my eyes) for it but' maybe some
> day someone will came with a justifiable workflow which need it. we will
> never know.
> 
> But in this case we need solve some questions :
>   - Will we extend git status signaling untracked files out of the
>     repository ?

I don't think that would work well.

>   - What will do git-clean ? it is already dangerous inside the
>     repository. and it will be worst if it can access outside of it.

Hopefully git clean will learn the --recurse-submodules option in the not
too distant future, then you will have just the same danger for the files
inside a submodule.

>>> Because in this case it is not just a reference that is managed but the
>>> file itself. And this way there is a risk to overwrite some data not
>>> under revision control outside of the repository.
>>
>> You have the same risk when a gitlink points outside, as a submodule is a
>> way of controlling a bunch of files through that reference. And the file
>> would be under version control in the repository where it is registered, no?
> 
> I agree on this point.
> 
> But they are still confined in an another git repository not
> disseminated all over the file system.
> And it never corrupt this pointed repository. just ask it to do by it's
> own regular git commands.

The only difference here is that a submodule can contain more than one file,
but you can corrupt those files just as easily as a single file using git
commands.

> In fact you can argue that it can disseminate some complete git
> repository anywhere in the file system.
> And you will be right. (nothing is perfect.)

I'm not concerned about not being perfect (nothing is perfect), but it is
dangerous.

> I can do a second patch to prevent git submodule command to make clones
> outside of the repository.
> It will requires the user to do those clones manually.
> In fact this is already what I do.
> My only use of this is to track dependencies.

But gitlinks are more than simple dependencies, they are followed! "git
submodule", status, diff and fetch already follow them. push is learning
that right now. checkout, reset, merge and friends are being taught that
too (see the enhance_submodule branch in my github repo for the current
state). So a gitlink is more than just a simple reference, it is followed
by a lot of commands and the submodule it points to is manipulated by
those commands. We had a patch for "git archive --recurse-submodules" on
the list, what will that do when used in "project1"?

>>> And in fact it is just what I want, it enable me if I decide to work on
>>> an optional "BigProject" depending on both project 1 & 2.
>>>
>>> Then If lib1 is in version M:
>>>  - a git status in project2 will say nothing
>>>  - a git status in project1 will say
>>>    "modified:   ../lib1 (modified content)
>>>  - a git status in BigProject will say
>>>    "modified:   ../project1 (modified content)
>>>
>>> Then I know that I need to update project1 to work with the last version
>>> M of lib1.
>>
>> Maybe no update for project1 is needed, because M only contains a bugfix
>> which doesn't even need a recompilation of project1. But now you need to
>> add a commit to project1 nonetheless with a message like "Updated lib1
>> with a bugfix which is needed by project2" which makes your idea of
>> independent submodules break down.
> 
> In fact I work ni the world of "high integrity programming" then It is
> just what I need.
> If there is a bugfix in any library, used by the program it is no more
> the same program.
> I need the "SHA1" to correspond to the exact and complete source code
> involved in my executable.
> 
> And this way the "SHA1" of the project sign the "SHA1" of the
> libraries. 

I cannot believe you want single commits in your "Gimp" repo for every
combination of distributions and library versions where someone said
"this works". This is insane and won't scale at all.

What you do is that each distribution tests their combination of programs
and libraries and says "that works". And that is why the only sane way to
record this "high integrity programming" test result is in the superproject
(= distribution) and not in each of the program repositories.

I also see that it would be cool when a program could record "I do work with
that library version, if you use another you are on your own". But it will
never say "I only work with *this* specific library version", which is what
your proposal is trying to do.

>>>> You are opening a can of worms by having two different repos point to the same
>>>> submodule living in a third repo (which also happens to be their superproject
>>>> and must somehow ignore it). You'll have two SHA1s for a single submodule;
>>>> "git submodule foreach --recursive" will have interesting results too; and so
>>>> on. Not good.
>>>
>>> As I just said before it is my purpose to do it like that.
>>
>> I understood that, but what are you proposing to do to solve all the
>> problems your approach introduces? You can't just hand wave them away.
> 
> There is some solutions :
> 
>   - First it is one more **feature** if it does not correspond to your
>     work flow it does not prevent you to work exactly the way you did
>     until now.
> 
>   - Second if you want to use the feature but not want to have the
>     conflict **feature** (for me it is one), just put the independent
>     project with there libs in different directory
> 
>       -+- foo -+- lib1     (in version N)
>        |       +- project1
>        |
>        +- bar -+- lib1     (in version M)
>                +- project2
> 
>   - Third if you really need to have project 1 & 2 in the same
>     directory foo, that means they are needed by a third BigProject in
>     the same directory foo depending on project 1 & 2.
>     And then you really need git to declare a conflict.

No you don't. You just need to git to tell you: this is not the version I
was tested against, repeat the tests to be sure.

>>> Let say a concret exemple
>>>
>>> 3 different teams work on libtiff, libpng, and libjpeg they are totally
>>> unrelated.
>>>
>>> One more team is working on the "gimp". And they need those 3 libs in
>>> specific versions not necessarily there heads.
>>>
>>> One other unrelated team is working on "gqview" and need the same libs
>>> in other specifics versions (Why should they know what te gimp team
>>> does)
>>>
>>> Neither "gimp" and "gqview" project will contain directory with those
>>> libs inside. They just depend on them.
>>>
>>> And the last team work on the gnome project which need the "gimp" and
>>> "gqview". It will be this team witch have to care about having both
>>> "gimp" and "gqview" sharing the same libs version>
>>> And has well the gnome project will not contain "gqview" and "gimp" in
>>> its own tree.
>>> It will also depend on them.
>>
>> Cool, that is a real life example resembling what we have a my dayjob. But
>> a "gimp" and "gqview" project will only have dependencies like "use libpng
>> of version 1.2.3 or newer (because we need a feature/bugfix introduced
>> there)" and won't be tied to a special version of that library. This means
>> they need a dependency like "SHA1 or newer" instead of "exactly this SHA1".
> 
> It is useful and simpler to work like this but could introduce some
> bugs.

But that model is awfully successful and is used by all distributions I know,
so I suspect it is not that dangerous (especially when you do your own QA).

> The "gimp" team has tested it with libpng 1.2.3 and maybe know that it
> did not work with previous versions but if they do not have any crystal
> ball they never know if newer versions will not break something.
> In fact I doubt that the first version of gimp will work with the last
> version of libpng.

But in the real world it is exactly like that: gimp will work with all libpng
1.2.3 and newer, only when libpng is updated to 2.0.0 you have to check that
again. Of course there will be bugs in some combinations. But the advantage of
being able to then only fix libpng and have the bug fixed in Gimp without
having to change it is far greater than the possible problem you are describing
here.

>>> It is just the same with aptitude on debian.
>>> Each package know there dependency by themselves, does not contain there
>>> dependencies, and do not need a bigger superpackage to tell them what
>>> are there own dependencies.
>>
>> And this is a very good point for the "version x.yy-z *or newer*" argument,
>> they are /never/ tied to the /exact/ x.yy-z version, as that would make the
>> dependencies pretty much unusable. They use a "newer than x.yy-z" scheme.
> 
> It is an other feature that the one I need.
> But it is a good idea.
> 
> Nothing prevent us to make a patch to add a new test in git status to
> see if the current SHA1 in the libpng repository has the SHA1 of the
> gitlink in the gimp in its ancestor.

To make that feature useful for others (e.g. at my dayjob) this would be
necessary. And we would never want the exact SHA1 match, even though that
information might be what others (like you) want.

>>> And Still I realy want to have every project knowing there own
>>> dependency by themselves and not needing an external superproject to
>>> tell them what they need.
>>
>> I want to have that too! I'm just convinced using a gitlink to achieve that
>> is wrong in so many ways. I'd rather prefer to express such dependencies in
>> something like a config file, and I believe they should not be as strict as
>> "I need exactly that version" but rather like "this version or newer (and by
>> the way: we of course only tested that specific version ;-)". These
>> dependencies could then be checked and displayed by git status.
> 
> It effectively could be in a config file it seem good to me as well.

Ok.

> But if git handle this config file.
> Update it on a "git add ../libpng && git commit"

I'm not sure an automatic update at "git commit" would be the right thing to
do, as I think that should only happen after all tests have run successful,
not at the time you commit it. But anyways, that could be done with a post
commit hook. Or the test script can do it when it succeeded.

> And control the matching between the project and libraries on
> "git status">

An extension to "git status" to display the dependencies that aren't met is
a valid goal. What about starting with a script ("git depends"?) and then see
what can go into status?

> I can not see the difference with a gitlink.

Then you can just use a config file for that, no? ;-)

Re: tracking submodules out of main directory.

[Heiko Voigt]

Hi,

On Fri, Jul 29, 2011 at 11:39:37AM +0200, henri GEIST wrote:
> Let say a concret exemple
> 
> 3 different teams work on libtiff, libpng, and libjpeg they are totally
> unrelated.
> 
> One more team is working on the "gimp". And they need those 3 libs in
> specific versions not necessarily there heads.
> 
> One other unrelated team is working on "gqview" and need the same libs
> in other specifics versions (Why should they know what te gimp team
> does)
> 
> Neither "gimp" and "gqview" project will contain directory with those
> libs inside. They just depend on them.
> 
> And the last team work on the gnome project which need the "gimp" and
> "gqview". It will be this team witch have to care about having both
> "gimp" and "gqview" sharing the same libs version>
> And has well the gnome project will not contain "gqview" and "gimp" in
> its own tree.
> It will also depend on them.
> 
> It is just the same with aptitude on debian.
> Each package know there dependency by themselves, does not contain there
> dependencies, and do not need a bigger superpackage to tell them what
> are there own dependencies.

As Jens mentioned already in this example you have a

        somemodule A needs a version of lib C higher than X
	somemodule B needs a version of lib C higher than Y

relation. Which in the case of submodules is A points to X and B points
to Y. Lets assume X is contained in Y. Since only the superproject knows
about both A and B its the only instance that can resolve this conflict
of dependence on C and can choose Y. In your example aptitude would be
the superproject containing everything.

This is actually (simplified) the way submodule merge is implemented. So
you see if you want both A and B to use the same version of C you need a
superproject recording this knowledge.

Adding the ability to point to git repositories outside of the worktree
does not solve anything but rather creates more problems. Resolving such
dependencies can not be achieved if only A knows that it needs version X
and only B knows that it needs version Y.

Cheers Heiko

Re: tracking submodules out of main directory.

[henri GEIST]

Le lundi 01 août 2011 à 21:39 +0200, Jens Lehmann a écrit :
> Am 30.07.2011 23:55, schrieb henri GEIST:
> > Le samedi 30 juillet 2011 à 16:16 +0200, Jens Lehmann a écrit :
> >> Am 29.07.2011 11:39, schrieb henri GEIST:
> >>> Le jeudi 28 juillet 2011 à 18:48 +0200, Jens Lehmann a écrit :
> >>>> Am 28.07.2011 10:57, schrieb henri GEIST:
> > 
> >> That will then be registered in other git repositories too in your model,
> >> which gets rid of the "one file/submodule, one repo" assumption we now have
> >> and will introduce ambiguities which are *really* hard to handle.
> > 
> > I am sorry, I am not a native English speaker. This sentence is to
> > complex for me. And google translator is of no help in this case.
> 
> Your proposal of letting multiple gitlinks in different repos point to the
> same submodule will break the assumption that each file is only handled by
> a single git repo.

I Agree

This is not an issue for me but I can understand that it could be for
some one else. Even if I do not see why. As it only do it for other git
repository that you already now they could have there own autonomous
live.

> For example when you have a conflict and do a "git
> submodule update --recursive" in the superproject, the SHA1 used for "lib"
> will depend on the alphabetical order of "project1" and "project2".

Exact.

> And normally after running "git submodule update --recursive" you expect all
> submodules of the superproject to be clean.

Not me. and the next "git status" will immediately tell me.
I do not know about you but "git status" is for me a reflex after any
important command.

> But your change breaks this expectation, it will still contain unclean
> submodule entries even though you just told git it should clean them.
> What will a "git submodule sync --recursive" do when "project1" and
> "project2" use different urls in their gitmodules? And so on.

I suspect exactly the same conflict as with "git submodule update
--recursive" or any recursive submodule management.
But again if I deliberately chose this model I know what I am doing.

> Commands won't always behave like you expect them to and sometimes will
> give different results just because different names are used. That's what
> I meant with ambiguities and that's why I don't think gitlinks are the
> right method here.

I am sorry. but I do not understand the relation with names.

> > But I agree the step is really weak before enabling to put any regular
> > file outside of the directory.
> > I do not see any reasonable workflow (to my eyes) for it but' maybe some
> > day someone will came with a justifiable workflow which need it. we will
> > never know.
> > 
> > But in this case we need solve some questions :
> >   - Will we extend git status signaling untracked files out of the
> >     repository ?
> 
> I don't think that would work well.
Me to but it is not me to decide.
> 
> >   - What will do git-clean ? it is already dangerous inside the
> >     repository. and it will be worst if it can access outside of it.
> 
> Hopefully git clean will learn the --recurse-submodules option in the not
> too distant future, then you will have just the same danger for the files
> inside a submodule.

Then this command will be once again more dangerous.
But it was before and pepole using it already knows then I suspect it is
not worst than before.

> > But they are still confined in an another git repository not
> > disseminated all over the file system.
> > And it never corrupt this pointed repository. just ask it to do by it's
> > own regular git commands.
> 
> The only difference here is that a submodule can contain more than one file,
> but you can corrupt those files just as easily as a single file using git
> commands.

Your right.

> > I can do a second patch to prevent git submodule command to make clones
> > outside of the repository.
> > It will requires the user to do those clones manually.
> > In fact this is already what I do.
> > My only use of this is to track dependencies.
> 
> But gitlinks are more than simple dependencies, they are followed! "git
> submodule", status, diff and fetch already follow them. push is learning
> that right now. checkout, reset, merge and friends are being taught that
> too (see the enhance_submodule branch in my github repo for the current
> state). So a gitlink is more than just a simple reference, it is followed
> by a lot of commands and the submodule it points to is manipulated by
> those commands. We had a patch for "git archive --recurse-submodules" on
> the list, what will that do when used in "project1"?

I think your right. I will make a parallel way not using gitlinks but
something similar which is not followed!

> > In fact I work in the world of "high integrity programming" then It is
> > just what I need.
> > If there is a bugfix in any library, used by the program it is no more
> > the same program.
> > I need the "SHA1" to correspond to the exact and complete source code
> > involved in my executable.
> > 
> > And this way the "SHA1" of the project sign the "SHA1" of the
> > libraries. 
> 
> I cannot believe you want single commits in your "Gimp" repo for every
> combination of distributions and library versions where someone said
> "this works". This is insane and won't scale at all.
> 

That is exactly what I do. but my team is only 9 people.
But we do not make a commit for each possible solution.
It. is only when I work on gimp to improve it that I made a commit which
is update for the ne libpng. but between to commit on the gimp there may
bu 10 update of libpng an I do not do a commit for each of them only the
last stable one.

It is only on the gnome project that I also need tu update my old
version of gqview to work with the libpng used by gimp but as long I do
not work on gnome I do not need to synchronize them.

But I write aircraft autopilot and if I am not at least as strict as
that. The certification department of the FAA and EASA will reject my
code and my boss will fire me.

> What you do is that each distribution tests their combination of programs
> and libraries and says "that works". And that is why the only sane way to
> record this "high integrity programming" test result is in the superproject
> (= distribution) and not in each of the program repositories.
> 
> I also see that it would be cool when a program could record "I do work with
> that library version, if you use another you are on your own". But it will
> never say "I only work with *this* specific library version", which is what
> your proposal is trying to do.

No git status only said "modified:   ../libpng (new commits)"
For me modified just mean "you are on your own" It do not prevent to
compile, execute or do anything else.

> >> I understood that, but what are you proposing to do to solve all the
> >> problems your approach introduces? You can't just hand wave them away.
> > 
> > There is some solutions :
> > 
> >   - First it is one more **feature** if it does not correspond to your
> >     work flow it does not prevent you to work exactly the way you did
> >     until now.
> > 
> >   - Second if you want to use the feature but not want to have the
> >     conflict **feature** (for me it is one), just put the independent
> >     project with there libs in different directory
> > 
> >       -+- foo -+- lib1     (in version N)
> >        |       +- project1
> >        |
> >        +- bar -+- lib1     (in version M)
> >                +- project2
> > 
> >   - Third if you really need to have project 1 & 2 in the same
> >     directory foo, that means they are needed by a third BigProject in
> >     the same directory foo depending on project 1 & 2.
> >     And then you really need git to declare a conflict.
> 
> No you don't. You just need to git to tell you: this is not the version I
> was tested against, repeat the tests to be sure.
> 

That is just what it will do.

> >> Cool, that is a real life example resembling what we have a my dayjob. But
> >> a "gimp" and "gqview" project will only have dependencies like "use libpng
> >> of version 1.2.3 or newer (because we need a feature/bugfix introduced
> >> there)" and won't be tied to a special version of that library. This means
> >> they need a dependency like "SHA1 or newer" instead of "exactly this SHA1".
> > 
> > It is useful and simpler to work like this but could introduce some
> > bugs.
> 
> But that model is awfully successful and is used by all distributions I know,
> so I suspect it is not that dangerous (especially when you do your own QA).
> 

Depending of how critical is a bug.
For an image manipulation programme the ratio effort/needed safety is in
the favor of your approche.

But will you be confident if the nuclear power-plant 50 km from you use
this kind of devloppement to contrôle the uranium bars ?
 
In any case in my workflow it is not about testing and not seeing
obvious bugs. It is about having a development process that permit to
prove there is obviously no bugs. And that model obviously break the
proof process.

> > The "gimp" team has tested it with libpng 1.2.3 and maybe know that it
> > did not work with previous versions but if they do not have any crystal
> > ball they never know if newer versions will not break something.
> > In fact I doubt that the first version of gimp will work with the last
> > version of libpng.
> 
> But in the real world it is exactly like that: gimp will work with all libpng
> 1.2.3 and newer, only when libpng is updated to 2.0.0 you have to check that
> again. Of course there will be bugs in some combinations. But the advantage of
> being able to then only fix libpng and have the bug fixed in Gimp without
> having to change it is far greater than the possible problem you are describing
> here.
> 

In this case the commit is not about a change in gimp code, but is a
stamp to validate the use of gimp with this new version of libpng is
certified.
But just like in your current use of submodules.

> >>> It is just the same with aptitude on debian.
> >>> Each package know there dependency by themselves, does not contain there
> >>> dependencies, and do not need a bigger superpackage to tell them what
> >>> are there own dependencies.
> >>
> >> And this is a very good point for the "version x.yy-z *or newer*" argument,
> >> they are /never/ tied to the /exact/ x.yy-z version, as that would make the
> >> dependencies pretty much unusable. They use a "newer than x.yy-z" scheme.
> > 
> > It is an other feature that the one I need.
> > But it is a good idea.
> > 
> > Nothing prevent us to make a patch to add a new test in git status to
> > see if the current SHA1 in the libpng repository has the SHA1 of the
> > gitlink in the gimp in its ancestor.
> 
> To make that feature useful for others (e.g. at my dayjob) this would be
> necessary. And we would never want the exact SHA1 match, even though that
> information might be what others (like you) want.
> 

Ok I will do it.


> > But if git handle this config file.
> > Update it on a "git add ../libpng && git commit"
> 
> I'm not sure an automatic update at "git commit" would be the right thing to
> do, as I think that should only happen after all tests have run successful,
> not at the time you commit it. But anyways, that could be done with a post
> commit hook. Or the test script can do it when it succeeded.
> 

I am sorry "make && make test && git add ../libpng && git commit"

> > And control the matching between the project and libraries on
> > "git status".
> 
> An extension to "git status" to display the dependencies that aren't met is
> a valid goal. What about starting with a script ("git depends"?) and then see
> what can go into status?
> 
> > I can not see the difference with a gitlink.
> 
> Then you can just use a config file for that, no? ;-)
> 

Off corse, I immediately start to work on it.

	Henri GEIST

Re: tracking submodules out of main directory.

[henri GEIST]

Le mardi 02 août 2011 à 00:12 +0200, Heiko Voigt a écrit :
> Hi,
> 
> On Fri, Jul 29, 2011 at 11:39:37AM +0200, henri GEIST wrote:
> > Let say a concret exemple
> > 
> > 3 different teams work on libtiff, libpng, and libjpeg they are totally
> > unrelated.
> > 
> > One more team is working on the "gimp". And they need those 3 libs in
> > specific versions not necessarily there heads.
> > 
> > One other unrelated team is working on "gqview" and need the same libs
> > in other specifics versions (Why should they know what te gimp team
> > does)
> > 
> > Neither "gimp" and "gqview" project will contain directory with those
> > libs inside. They just depend on them.
> > 
> > And the last team work on the gnome project which need the "gimp" and
> > "gqview". It will be this team witch have to care about having both
> > "gimp" and "gqview" sharing the same libs version>
> > And has well the gnome project will not contain "gqview" and "gimp" in
> > its own tree.
> > It will also depend on them.
> > 
> > It is just the same with aptitude on debian.
> > Each package know there dependency by themselves, does not contain there
> > dependencies, and do not need a bigger superpackage to tell them what
> > are there own dependencies.
> 
> As Jens mentioned already in this example you have a
> 
>         somemodule A needs a version of lib C higher than X
> 	somemodule B needs a version of lib C higher than Y
> 
> relation. Which in the case of submodules is A points to X and B points
> to Y. Lets assume X is contained in Y. Since only the superproject knows
> about both A and B its the only instance that can resolve this conflict
> of dependence on C and can choose Y. In your example aptitude would be
> the superproject containing everything.
> 

I do not want to have a superproject. just as with aptitude. Each
package store its own dependencies itself.
I do not want to need a super package who now every dependencies of
every possible packages.
First because it is impractical to maintain an exhaustive list of all
possible packages (including unofficial ones.)
Secondly because I have no need for this and it will require somme more
works.
Third for different people witch use and share there own subset off
unofficial package they needs to cook a specific super package for each
unique case.

> This is actually (simplified) the way submodule merge is implemented. So
> you see if you want both A and B to use the same version of C you need a
> superproject recording this knowledge.

And tha is my problem.

> Adding the ability to point to git repositories outside of the worktree
> does not solve anything but rather creates more problems. Resolving such
> dependencies can not be achieved if only A knows that it needs version X
> and only B knows that it needs version Y.
> 

Why not it work perfect for me and for debian as well.
Yes I now for speed purpose they scans all the package header and store
their dependency requirement in a database. but it is only for speed and
it is automatic generated by the info "In the packages them selves". I
do not think they ever edit it by hand to define the dependency in the
DB.

In fact I suppose this by what I seen by using it I never looked in the
apt source code.

	Henri GEIST

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 02.08.2011 14:19, schrieb henri GEIST:
> Le lundi 01 août 2011 à 21:39 +0200, Jens Lehmann a écrit :
>> Am 30.07.2011 23:55, schrieb henri GEIST:
>>> I can not see the difference with a gitlink.
>>
>> Then you can just use a config file for that, no? ;-)
> 
> Off corse, I immediately start to work on it.

I'm looking forward to that!

Re: tracking submodules out of main directory.

[Heiko Voigt]

Hi Henri,

On Tue, Aug 02, 2011 at 08:42:24PM +0200, Jens Lehmann wrote:
> Am 02.08.2011 14:19, schrieb henri GEIST:
> > Le lundi 01 ao??t 2011 ?? 21:39 +0200, Jens Lehmann a ??crit :
> >> Am 30.07.2011 23:55, schrieb henri GEIST:
> >>> I can not see the difference with a gitlink.
> >>
> >> Then you can just use a config file for that, no? ;-)
> > 
> > Off corse, I immediately start to work on it.
> 
> I'm looking forward to that!

Before hacking away please share some design information about this.

Another thing:

It sounds like the workflow you want to achieve is similar to the one
the android developers are using. They have a lot of submodules which
need automatic updating.

The last time I checked they used repo (similar tool to submodules)
together with gerrit.

AFAIR they wanted to switch to submodules and have gerrit automatically
make the superproject commits for single submodule changes. Additionally
if a change involves multiple submodules the developer should be able to
tie them together using a superproject commit.

Have a look over there whether that workflow might help you?

Cheers Heiko

Re: tracking submodules out of main directory.

[henri GEIST]

Le mardi 02 août 2011 à 12:19 -0700, PJ Weisberg a écrit :
> On Tuesday, August 2, 2011, henri GEIST
> <henri.geist@flying-robots.com> wrote:
> > Le mardi 02 août 2011 à 00:12 +0200, Heiko Voigt a écrit :
> >> Adding the ability to point to git repositories outside of the
> worktree
> >> does not solve anything but rather creates more problems. Resolving
> such
> >> dependencies can not be achieved if only A knows that it needs
> version X
> >> and only B knows that it needs version Y.
> >>
> >
> > Why not it work perfect for me and for debian as well.
> 
> In the Debian analogy: A knows that it needs version X, B knows that
> it needs version Y, but A does not know that if it installs version X,
> B will break.  That's the job of the superproject (Aptitude).
> 

Your right.

Re: tracking submodules out of main directory.

[henri GEIST]

Le mercredi 03 août 2011 à 08:25 +0200, Heiko Voigt a écrit :
> Hi Henri,
> 
> On Tue, Aug 02, 2011 at 08:42:24PM +0200, Jens Lehmann wrote:
> > Am 02.08.2011 14:19, schrieb henri GEIST:
> > > Le lundi 01 ao??t 2011 ?? 21:39 +0200, Jens Lehmann a ??crit :
> > >> Am 30.07.2011 23:55, schrieb henri GEIST:
> > >>> I can not see the difference with a gitlink.
> > >>
> > >> Then you can just use a config file for that, no? ;-)
> > > 
> > > Off corse, I immediately start to work on it.
> > 
> > I'm looking forward to that!
> 
> Before hacking away please share some design information about this.
> 

Ok this is what I intend to do.

I plan to use a config file containing lines like

"path_to_poited_repo   SHA1_of_intended_commit   URL_of_origin"

the URL part will not be required.

this file will be a list of pointer to other project.

I have plane to name the file ".gitdependencies" and put it at the root
of the current repository.


Then I will :

1) Adding to git-status the ability to scan this file and report :
   - if the repository pointed by the path does not exist
   - if the SHA1 of the current checkout of this repository mismatch.
   - if this repository content differ from its checkout.

2) Adding to git status the ability to say if the current checkout has
   the expected one in its ancestors.

3) Adding to git-add the ability to:
   - create/update this file when trying to add an external repository
   - giving up if it is not an external repository.
   - automatically add the dependency file as well.

4) Adding to git-rm the ability to:
   - remove lines in this file.
   - add the result to index.
   - maybe removing the complete file if it is empty.

5) Adding to git-reset the ability to:
   - reset those lines one by one.
   - adding the the file to the index.

I think I will do those steps as separate commits because they can works
independently. it just need you to do the unimplemented step manually
until it is done.

That is a first proposal, all your comment will be appreciate.

> Another thing:
> 
> It sounds like the workflow you want to achieve is similar to the one
> the android developers are using. They have a lot of submodules which
> need automatic updating.
> 
> The last time I checked they used repo (similar tool to submodules)
> together with gerrit.
> 
> AFAIR they wanted to switch to submodules and have gerrit automatically
> make the superproject commits for single submodule changes. Additionally
> if a change involves multiple submodules the developer should be able to
> tie them together using a superproject commit.
> 
> Have a look over there whether that workflow might help you?

I just add a quick look on the android web site and I did not understand
what exactly repo do.

All there documentation only say how to install it and checkout android
but it dose not explain anything about repo itself.

I will search more info about it.

	Henri GEIST

Re: tracking submodules out of main directory.

[Junio C Hamano]

henri GEIST <henri.geist@flying-robots.com> writes:

> I plan to use a config file containing lines like
>
> "path_to_poited_repo   SHA1_of_intended_commit   URL_of_origin"
>
> the URL part will not be required.
>
> this file will be a list of pointer to other project.

I wasn't paying attention to this thread, but I have to ask "why" here.

The first two are what gitlink was designed to do in the superproject that
ties multiple submodules together, and the last one is also supplied by
the .gitmodules in that superproject. This seems to be adding the same
information in a redundant way by saying "this version A0 of submodule A
wants version B0 of submodule B and version C0 of submodule C" when the
supermodule can say "the consistent view I record is to have version A0,
B0 and C0 of submodules A, B and C, respectively".

I also suspect that allowing each submodule to know and demand specific
versions of other submodules will lead to inconsistencies. Which version
of submodule C would you demand to have when submodule A wants version C0
and submodule B wants version C1 of it?

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 03.08.2011 19:11, schrieb Junio C Hamano:
> henri GEIST <henri.geist@flying-robots.com> writes:
> 
>> I plan to use a config file containing lines like
>>
>> "path_to_poited_repo   SHA1_of_intended_commit   URL_of_origin"
>>
>> the URL part will not be required.
>>
>> this file will be a list of pointer to other project.
> 
> I wasn't paying attention to this thread, but I have to ask "why" here.
> 
> The first two are what gitlink was designed to do in the superproject that
> ties multiple submodules together, and the last one is also supplied by
> the .gitmodules in that superproject. This seems to be adding the same
> information in a redundant way by saying "this version A0 of submodule A
> wants version B0 of submodule B and version C0 of submodule C" when the
> supermodule can say "the consistent view I record is to have version A0,
> B0 and C0 of submodules A, B and C, respectively".

During the discussion this evolved from a simple "I need that submodule
with exactly this version" to something I believe is more generic and
very useful for others. As I see it now a submodule should be able to say:

1) To use me, you need another submodule "foo"

   This is very helpful when you want to add the Gimp submodule and it
   can tell you you'll need the libpng submodule too in your superproject
   (but I'd vote to use the submodule name here, not the path as that
   should be the superproject's decision).

In addition to that, it can (but mustn't) specify any of the following:

a) Of this submodule "foo" I need at least that version because I won't
   compile/work with older versions of that. (this can be tightened to
   "exactly that version" to give henri the behavior he wants, but that
   should be policy, not mandatory)

   Gimp could say it needs at least libpng 012345 because in that version
   the function foobar() was added it now depends on. Normally this won't
   be updated very often, but if people like henri use that to say "I'll
   only promise to work well with that exact version, as that went through
   extensive QA" they might change that on virtually every commit.

b) And if you don't know where to get it, use this url

   That can give the superproject a hint where it can clone that
   repository from. That could be helpful for distributions to sort out
   the dependencies of the packages they pull in.

That is all stuff the submodule knows better than the superproject. And
that information can be used to *inform* the user about the submodule's
needs, maybe using "git status --submodule-dependencies" will print:

# submodule "Gimp" requests a libpng 567890 or newer
# submodule "foo" has missing dependency "bar"

But the user can choose to ignore that (because he knows he has the png
support disabled and he doesn't need the fancy help files from bar).

And maybe "git submodule add" learns an option to automatically add all
the other submodules the new one depends on too (for that we would need
the url).

But the superproject is still the place to say: I know these versions of
all submodules work together, so I commit their gitlinks here. But this
scheme enables submodules to give hints to help the superproject's user.

> I also suspect that allowing each submodule to know and demand specific
> versions of other submodules will lead to inconsistencies. Which version
> of submodule C would you demand to have when submodule A wants version C0
> and submodule B wants version C1 of it?

Right, in the discussion so far it seemed like henri seems to be the only
user who is wanting an exact match, and he says he needs to see these
inconsistencies. But I think he can modify the "version xxx or newer" to
his needs without imposing these inconsistencies on users (like me) who
don't want to see them.

Re: tracking submodules out of main directory.

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> 1) To use me, you need another submodule "foo"
>
>    This is very helpful when you want to add the Gimp submodule and it
>    can tell you you'll need the libpng submodule too in your superproject
>    (but I'd vote to use the submodule name here, not the path as that
>    should be the superproject's decision).

That is something you can add to .gitmodules in the superproject, no?
E.g.

	[submodule "Gimp"]
        	url = http://some/where/
		path = gimp/
        	depends = version 1.2.3 of "Glib"

	[submodule "Glib"]
        	url = http://some/where/else/
		path = libs/glib

> In addition to that, it can (but mustn't) specify any of the following:

I am guessing you meant "does not have to", instead of mustn't, here...

> a) Of this submodule "foo" I need at least that version because I won't
>    compile/work with older versions of that. (this can be tightened to
>    "exactly that version" to give henri the behavior he wants, but that
>    should be policy, not mandatory)

The "loose collection of projects" approach like that has its uses, and it
is called "repo". Why re-invent it? The behaviour Henri wants to specify
the exact version is how git submodules work already, so I do not see
there is anything to be done here.

> b) And if you don't know where to get it, use this url

Again that is the job of .gitmodules in the superproject.

> That is all stuff the submodule knows better than the superproject.

Not necessarily. The version A0 of submodule A may depend on submodule B
and may also know it must have at least version B0 of that submodule, but
the superproject would know other constraints, e.g. the superproject
itself also calls into submodule B and wants a newer version B1 of it.

Re: tracking submodules out of main directory.

[henri GEIST]

Le mercredi 03 août 2011 à 10:11 -0700, Junio C Hamano a écrit :
> henri GEIST <henri.geist@flying-robots.com> writes:
> 
> > I plan to use a config file containing lines like
> >
> > "path_to_poited_repo   SHA1_of_intended_commit   URL_of_origin"
> >
> > the URL part will not be required.
> >
> > this file will be a list of pointer to other project.
> 
> I wasn't paying attention to this thread, but I have to ask "why" here.
> 
> The first two are what gitlink was designed to do in the superproject that
> ties multiple submodules together, and the last one is also supplied by
> the .gitmodules in that superproject.

Yes my only problem is that it is forbidden to put a submodule outside
of the repository.
That is way I had done a patch witch enable me to do so and I use it
flawlessly every day. with total satisfaction.

> This seems to be adding the same information in a redundant way by saying
> "this version A0 of submodule A wants version B0 of submodule B and
> version C0 of submodule C" when the supermodule can say "the consistent
> view I record is to have version A0, B0 and C0 of submodules A, B and C,
> respectively".
> 

Exact but my goal is to get ride of the superproject.
This is how I will remove the redondency.
Cause creating a projet just to said to other project what they need is
a wast. Each project has to now by itself what it need.

And each user will have his own list of project to work on.
Then they we will create one different superproject for each user.

One user can work on a superproject containing :
gimp, gqview, libpng

One other will work on a superproject containing :
gimp, gphoto, libpng, libusb

The next one will work on a superproject containing :
xsane, libusb

They can absolutely not share there superproject only the normal
projects.
And if the dependency are defined in superproject, and not in the
projects themselves. Users can not share their dependency constructs.

I have no intend to have "sub"modules but to have generic modules with
no hierarchies only dependence relations. Which could even be crossed.
(Module A require module B and module B require module A.)
Even if in this particular case I do not see the point to make two
distinct modules.


> I also suspect that allowing each submodule to know and demand specific
> versions of other submodules will lead to inconsistencies. Which version
> of submodule C would you demand to have when submodule A wants version C0
> and submodule B wants version C1 of it?
> 

That is already the case with normal submodules.
And of corse if you have a project which recursively depend of to
version of the same library, you need to update it, and it has nothing
to do with git. Git will just make it obvious.


The reason we decide to make a parallel file ".gitdependencies",
containing something really similar to the content of ".gitmodules"
is that in .gitdependencies we will put references outside of repository
and in ".gitmodules" the references inside git repository as actual.

This separation will be done because Jens Lehmann show us
that .gitmodules are lot more that what I expected from those reference
and do not think that the others abilities of submodules should apply to
external modules.

	Henri

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 03.08.2011 21:41, schrieb Junio C Hamano:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> 1) To use me, you need another submodule "foo"
>>
>>    This is very helpful when you want to add the Gimp submodule and it
>>    can tell you you'll need the libpng submodule too in your superproject
>>    (but I'd vote to use the submodule name here, not the path as that
>>    should be the superproject's decision).
> 
> That is something you can add to .gitmodules in the superproject, no?
> E.g.
> 
> 	[submodule "Gimp"]
>         	url = http://some/where/
> 		path = gimp/
>         	depends = version 1.2.3 of "Glib"
> 
> 	[submodule "Glib"]
>         	url = http://some/where/else/
> 		path = libs/glib

The "depends" information is not very useful inside the superproject,
because when you already know that there you can simply commit version
1.2.3 of Glib together with Gimp instead of adding that information.
That's what gitlinks are for, no?

But when you fetch a new version of Gimp into your submodule, it would be
really nice if the superproject could be notified that the Gimp developers
updated to 1.2.4 of Glib and inform you that an update of Glib might be
appropriate. That could avoid having you to dig through compiler errors to
find out that the new foobar() function from Glib 1.2.4 is needed (and if
you need to pull in a bugfix in Glib, you might notice that *a lot* later
when you forget to do that).

>> In addition to that, it can (but mustn't) specify any of the following:
> 
> I am guessing you meant "does not have to", instead of mustn't, here...

Sure, thanks for deciphering that.

>> a) Of this submodule "foo" I need at least that version because I won't
>>    compile/work with older versions of that. (this can be tightened to
>>    "exactly that version" to give henri the behavior he wants, but that
>>    should be policy, not mandatory)
> 
> The "loose collection of projects" approach like that has its uses, and it
> is called "repo". Why re-invent it? The behaviour Henri wants to specify
> the exact version is how git submodules work already, so I do not see
> there is anything to be done here.

Let me make this clear: this is not about changing how submodules are
committed in a superproject. It is not about having a loose collection of
projects, they stay tied together in a defined state by the superproject.

Henri wanted it a bit upside down: any submodule could request a certain
version of another submodule somewhere else in the repo. And he wanted to
use gitlinks from one submodule to another for that, which I - hopefully -
convinced him was no good idea.

But I understand his need to have some kind of "version hint" from one
submodule to another. Just that he wants to take the hint very serious,
while I see it as means to communicate from the submodule maintainer to
the superproject developers that another submodule might have to be
updated too when they do that to his.

>> b) And if you don't know where to get it, use this url
> 
> Again that is the job of .gitmodules in the superproject.

Yes. But this idea is about how the url could get into the .gitmodules of
the superproject in the first place. That can make it easier for the
superproject's developer to import a submodule into his repo and much more
important: it makes it possible to pull in submodule dependencies
automatically e.g. when running "git submodule add --resolve-dependencies
Gimp".

>> That is all stuff the submodule knows better than the superproject.
> 
> Not necessarily. The version A0 of submodule A may depend on submodule B
> and may also know it must have at least version B0 of that submodule, but
> the superproject would know other constraints, e.g. the superproject
> itself also calls into submodule B and wants a newer version B1 of it.

Right. That's what I tried to explain to Henri, the superproject ties it all
together. But I also like his idea to add a way to communicate information
from the submodule to the superproject, and give the superproject a choice
if it wants to use it.

Re: tracking submodules out of main directory.

[Heiko Voigt]

Hi,

On Wed, Aug 03, 2011 at 09:07:14PM +0200, Jens Lehmann wrote:
> Am 03.08.2011 19:11, schrieb Junio C Hamano:
> But the superproject is still the place to say: I know these versions of
> all submodules work together, so I commit their gitlinks here. But this
> scheme enables submodules to give hints to help the superproject's user.
> 
> > I also suspect that allowing each submodule to know and demand specific
> > versions of other submodules will lead to inconsistencies. Which version
> > of submodule C would you demand to have when submodule A wants version C0
> > and submodule B wants version C1 of it?
> 
> Right, in the discussion so far it seemed like henri seems to be the only
> user who is wanting an exact match, and he says he needs to see these
> inconsistencies. But I think he can modify the "version xxx or newer" to
> his needs without imposing these inconsistencies on users (like me) who
> don't want to see them.

And I imagine if a submodule has such hints we could add a command say

	git submodule resolve-dependencies

which could resolve such "I need a version newer than" hints given by a
submodule to help the user to update a submodule in the superproject.

Disclaimer: I think we need to think about all the implications such a
scheme introduces very carefully. The picture is still a bit blurry for
me.

Cheers Heiko

Re: tracking submodules out of main directory.

[henri GEIST]

Le mercredi 03 août 2011 à 21:07 +0200, Jens Lehmann a écrit :
> Am 03.08.2011 19:11, schrieb Junio C Hamano:
> > henri GEIST <henri.geist@flying-robots.com> writes:
> > 
> >> I plan to use a config file containing lines like
> >>
> >> "path_to_poited_repo   SHA1_of_intended_commit   URL_of_origin"
> >>
> >> the URL part will not be required.
> >>
> >> this file will be a list of pointer to other project.
> > 
> > I wasn't paying attention to this thread, but I have to ask "why" here.
> > 
> > The first two are what gitlink was designed to do in the superproject that
> > ties multiple submodules together, and the last one is also supplied by
> > the .gitmodules in that superproject. This seems to be adding the same
> > information in a redundant way by saying "this version A0 of submodule A
> > wants version B0 of submodule B and version C0 of submodule C" when the
> > supermodule can say "the consistent view I record is to have version A0,
> > B0 and C0 of submodules A, B and C, respectively".
> 
> During the discussion this evolved from a simple "I need that submodule
> with exactly this version" to something I believe is more generic and
> very useful for others. As I see it now a submodule should be able to say:
> 
> 1) To use me, you need another submodule "foo"
> 
>    This is very helpful when you want to add the Gimp submodule and it
>    can tell you you'll need the libpng submodule too in your superproject
>    (but I'd vote to use the submodule name here, not the path as that
>    should be the superproject's decision).
> 
> In addition to that, it can (but mustn't) specify any of the following:
> 
> a) Of this submodule "foo" I need at least that version because I won't
>    compile/work with older versions of that. (this can be tightened to
>    "exactly that version" to give henri the behavior he wants, but that
>    should be policy, not mandatory)
> 

Of corse and git will not enforce this policy.
Git status will only say the version is unrelated, match or is higher.
And it is left to the human reading the status to decide if it is OK
with his policy.

>    Gimp could say it needs at least libpng 012345 because in that version
>    the function foobar() was added it now depends on. Normally this won't
>    be updated very often, but if people like henri use that to say "I'll
>    only promise to work well with that exact version, as that went through
>    extensive QA" they might change that on virtually every commit.
> 

In fact I do so very rarely I do not update my project to track the last
version of the library until the project need it. If the library get new
features and improvement that I do not need I keep the old version.
I will not rewrite the tones of certification papers to certify the use
of the new library version without needs. It will destroy all the
forests.

> b) And if you don't know where to get it, use this url
> 
>    That can give the superproject a hint where it can clone that
>    repository from. That could be helpful for distributions to sort out
>    the dependencies of the packages they pull in.
> 
> That is all stuff the submodule knows better than the superproject. And
> that information can be used to *inform* the user about the submodule's
> needs, maybe using "git status --submodule-dependencies" will print:
> 
> # submodule "Gimp" requests a libpng 567890 or newer
> # submodule "foo" has missing dependency "bar"

> But the user can choose to ignore that (because he knows he has the png
> support disabled and he doesn't need the fancy help files from bar).
> 
> And maybe "git submodule add" learns an option to automatically add all
> the other submodules the new one depends on too (for that we would need
> the url).
> 

Provided that you use a superproject. But my goal is to eliminate it.

> But the superproject is still the place to say: I know these versions of
> all submodules work together, so I commit their gitlinks here. But this
> scheme enables submodules to give hints to help the superproject's user.
> 

or they can do it by them selves has now they have all the needed infos.

> > I also suspect that allowing each submodule to know and demand specific
> > versions of other submodules will lead to inconsistencies. Which version
> > of submodule C would you demand to have when submodule A wants version C0
> > and submodule B wants version C1 of it?
> 
> Right, in the discussion so far it seemed like henri seems to be the only
> user who is wanting an exact match, and he says he needs to see these
> inconsistencies.

I suspect I am just the only one you now about.
Because that is just what actual submodules does, and nobody complain.

If you
  - cd into one of your submodules
  - make a commit
  - go back into the main repository
  - make git status
It will tell you :
"modified:   the_submodule_name/ (new commits)"

Note it actually will say "new commits" even if it is older or totally
unrelated. It just signal a mismatch.
Then what I need on this point is just what it actually do.

> But I think he can modify the "version xxx or newer" to
> his needs without imposing these inconsistencies on users (like me) who
> don't want to see them.
> 

Of corse git status has never imposed anything.

But I will enabling it to make the distinction between unmatched and
newer version.

	Henri

Re: tracking submodules out of main directory.

[henri GEIST]

Le mercredi 03 août 2011 à 23:30 +0200, Jens Lehmann a écrit :
> Am 03.08.2011 21:41, schrieb Junio C Hamano:
> > Jens Lehmann <Jens.Lehmann@web.de> writes:
> > 
> >> 1) To use me, you need another submodule "foo"
> >>
> >>    This is very helpful when you want to add the Gimp submodule and it
> >>    can tell you you'll need the libpng submodule too in your superproject
> >>    (but I'd vote to use the submodule name here, not the path as that
> >>    should be the superproject's decision).
> > 
> > That is something you can add to .gitmodules in the superproject, no?
> > E.g.
> > 
> > 	[submodule "Gimp"]
> >         	url = http://some/where/
> > 		path = gimp/
> >         	depends = version 1.2.3 of "Glib"
> > 
> > 	[submodule "Glib"]
> >         	url = http://some/where/else/
> > 		path = libs/glib
> 
> The "depends" information is not very useful inside the superproject,
> because when you already know that there you can simply commit version
> 1.2.3 of Glib together with Gimp instead of adding that information.
> That's what gitlinks are for, no?
> 
> But when you fetch a new version of Gimp into your submodule, it would be
> really nice if the superproject could be notified that the Gimp developers
> updated to 1.2.4 of Glib and inform you that an update of Glib might be
> appropriate. That could avoid having you to dig through compiler errors to
> find out that the new foobar() function from Glib 1.2.4 is needed (and if
> you need to pull in a bugfix in Glib, you might notice that *a lot* later
> when you forget to do that).
> 

Exact, I am really happy to read this.
And better do not bother to have the suproject.
cd to gimp directory, type git status it can tell you every thing and
when your satisfied you just have to type make.
At this point the superproject have not any use. 

> >> In addition to that, it can (but mustn't) specify any of the following:
> > 
> > I am guessing you meant "does not have to", instead of mustn't, here...
> 
> Sure, thanks for deciphering that.
> 
> >> a) Of this submodule "foo" I need at least that version because I won't
> >>    compile/work with older versions of that. (this can be tightened to
> >>    "exactly that version" to give henri the behavior he wants, but that
> >>    should be policy, not mandatory)
> > 
> > The "loose collection of projects" approach like that has its uses, and it
> > is called "repo". Why re-invent it? The behaviour Henri wants to specify
> > the exact version is how git submodules work already, so I do not see
> > there is anything to be done here.
> 
> Let me make this clear: this is not about changing how submodules are
> committed in a superproject. It is not about having a loose collection of
> projects, they stay tied together in a defined state by the superproject.
> 

Yes but for me, from when I started this this topic, it was all about
having a loose collection of project with dependency references between
them. And get rid of the superproject.
It is my first and only goal.

> Henri wanted it a bit upside down: any submodule could request a certain
> version of another submodule somewhere else in the repo. And he wanted to
> use gitlinks from one submodule to another for that, which I - hopefully -
> convinced him was no good idea.
> 

You just convince me that submodules are more than I need and to make a
lighter independent version of submodules which will never been followed
by git commands.

> But I understand his need to have some kind of "version hint" from one
> submodule to another. Just that he wants to take the hint very serious,
> while I see it as means to communicate from the submodule maintainer to
> the superproject developers that another submodule might have to be
> updated too when they do that to his.
> 

Yes

> >> b) And if you don't know where to get it, use this url
> > 
> > Again that is the job of .gitmodules in the superproject.
> 
> Yes. But this idea is about how the url could get into the .gitmodules of
> the superproject in the first place. That can make it easier for the
> superproject's developer to import a submodule into his repo and much more
> important: it makes it possible to pull in submodule dependencies
> automatically e.g. when running "git submodule add --resolve-dependencies
> Gimp".

Only if you have a superproject.
If not do the same thing from the gimp repository, now it contain all
necessary infos to do the job.

> >> That is all stuff the submodule knows better than the superproject.
> > 
> > Not necessarily. The version A0 of submodule A may depend on submodule B
> > and may also know it must have at least version B0 of that submodule, but
> > the superproject would know other constraints, e.g. the superproject
> > itself also calls into submodule B and wants a newer version B1 of it.
> 
> Right. That's what I tried to explain to Henri, the superproject ties it all
> together. But I also like his idea to add a way to communicate information
> from the submodule to the superproject, and give the superproject a choice
> if it wants to use it.
> 

yes but the superproject contain no code in your design.
Then it will never need anything by itself.
It is only a container which you will inform with data already known by
the submodules I do not see any value to it.

	Henri

Re: tracking submodules out of main directory.

[henri GEIST]

Le mercredi 03 août 2011 à 23:45 +0200, Heiko Voigt a écrit :
> Hi,
> 
> On Wed, Aug 03, 2011 at 09:07:14PM +0200, Jens Lehmann wrote:
> > Am 03.08.2011 19:11, schrieb Junio C Hamano:
> > But the superproject is still the place to say: I know these versions of
> > all submodules work together, so I commit their gitlinks here. But this
> > scheme enables submodules to give hints to help the superproject's user.
> > 
> > > I also suspect that allowing each submodule to know and demand specific
> > > versions of other submodules will lead to inconsistencies. Which version
> > > of submodule C would you demand to have when submodule A wants version C0
> > > and submodule B wants version C1 of it?
> > 
> > Right, in the discussion so far it seemed like henri seems to be the only
> > user who is wanting an exact match, and he says he needs to see these
> > inconsistencies. But I think he can modify the "version xxx or newer" to
> > his needs without imposing these inconsistencies on users (like me) who
> > don't want to see them.
> 
> And I imagine if a submodule has such hints we could add a command say
> 
> 	git submodule resolve-dependencies
> 
> which could resolve such "I need a version newer than" hints given by a
> submodule to help the user to update a submodule in the superproject.
> 
> Disclaimer: I think we need to think about all the implications such a
> scheme introduces very carefully. The picture is still a bit blurry for
> me.
> 

As I see it if you want a command like
"git submodule resolve-dependencies"

  - the process need to decide if it will chose te lower version above
    all requierments or directly the head.
  - if one or more say I want exactly this version and it is not
    satisfied we need at least a warning.
  - We need to issue an error if all the required version can not be
    found in the same branch.

	Henri

Re: tracking submodules out of main directory.

[Jens Lehmann]

Am 04.08.2011 00:29, schrieb henri GEIST:
> Le mercredi 03 août 2011 à 23:30 +0200, Jens Lehmann a écrit :
>> Am 03.08.2011 21:41, schrieb Junio C Hamano:
>>> Jens Lehmann <Jens.Lehmann@web.de> writes:
>> But when you fetch a new version of Gimp into your submodule, it would be
>> really nice if the superproject could be notified that the Gimp developers
>> updated to 1.2.4 of Glib and inform you that an update of Glib might be
>> appropriate. That could avoid having you to dig through compiler errors to
>> find out that the new foobar() function from Glib 1.2.4 is needed (and if
>> you need to pull in a bugfix in Glib, you might notice that *a lot* later
>> when you forget to do that).
>>
> 
> Exact, I am really happy to read this.
> And better do not bother to have the suproject.

I don't get this, you can't get rid of the superproject.

> cd to gimp directory, type git status it can tell you every thing and
> when your satisfied you just have to type make.
> At this point the superproject have not any use. 

"git status" inside the submodule won't tell you anything about the
dependencies, but a "git status" in the superproject should. The submodule
shouldn't know where exactly the submodules it depends on lives, that is
the decision of the superproject, not the submodule.

>>>> In addition to that, it can (but mustn't) specify any of the following:
>>>
>>> I am guessing you meant "does not have to", instead of mustn't, here...
>>
>> Sure, thanks for deciphering that.
>>
>>>> a) Of this submodule "foo" I need at least that version because I won't
>>>>    compile/work with older versions of that. (this can be tightened to
>>>>    "exactly that version" to give henri the behavior he wants, but that
>>>>    should be policy, not mandatory)
>>>
>>> The "loose collection of projects" approach like that has its uses, and it
>>> is called "repo". Why re-invent it? The behaviour Henri wants to specify
>>> the exact version is how git submodules work already, so I do not see
>>> there is anything to be done here.
>>
>> Let me make this clear: this is not about changing how submodules are
>> committed in a superproject. It is not about having a loose collection of
>> projects, they stay tied together in a defined state by the superproject.
>>
> 
> Yes but for me, from when I started this this topic, it was all about
> having a loose collection of project with dependency references between
> them. And get rid of the superproject.
> It is my first and only goal.

But I fail to see how that would improve anything.

>> Henri wanted it a bit upside down: any submodule could request a certain
>> version of another submodule somewhere else in the repo. And he wanted to
>> use gitlinks from one submodule to another for that, which I - hopefully -
>> convinced him was no good idea.
>>
> 
> You just convince me that submodules are more than I need and to make a
> lighter independent version of submodules which will never been followed
> by git commands.

Submodules are what you need, but it's no use to implement dependencies by
using gitlinks that point outside of their repositories.

>>>> b) And if you don't know where to get it, use this url
>>>
>>> Again that is the job of .gitmodules in the superproject.
>>
>> Yes. But this idea is about how the url could get into the .gitmodules of
>> the superproject in the first place. That can make it easier for the
>> superproject's developer to import a submodule into his repo and much more
>> important: it makes it possible to pull in submodule dependencies
>> automatically e.g. when running "git submodule add --resolve-dependencies
>> Gimp".
> 
> Only if you have a superproject.
> If not do the same thing from the gimp repository, now it contain all
> necessary infos to do the job.

No, it doesn't. Apart from the Gimp project telling you what version it
wants, you need to have a place to record the version that the user really
used. And that won't work without a superproject.

>>>> That is all stuff the submodule knows better than the superproject.
>>>
>>> Not necessarily. The version A0 of submodule A may depend on submodule B
>>> and may also know it must have at least version B0 of that submodule, but
>>> the superproject would know other constraints, e.g. the superproject
>>> itself also calls into submodule B and wants a newer version B1 of it.
>>
>> Right. That's what I tried to explain to Henri, the superproject ties it all
>> together. But I also like his idea to add a way to communicate information
>> from the submodule to the superproject, and give the superproject a choice
>> if it wants to use it.
>>
> 
> yes but the superproject contain no code in your design.
> Then it will never need anything by itself.
> It is only a container which you will inform with data already known by
> the submodules I do not see any value to it.

But being the container that ties it all together is more than enough value.

Re: tracking submodules out of main directory.

[Heiko Voigt]

Hi,

On Thu, Aug 04, 2011 at 12:29:22AM +0200, henri GEIST wrote:
> Le mercredi 03 ao??t 2011 ?? 23:30 +0200, Jens Lehmann a ??crit :
> > Let me make this clear: this is not about changing how submodules are
> > committed in a superproject. It is not about having a loose collection of
> > projects, they stay tied together in a defined state by the superproject.
> > 
> 
> Yes but for me, from when I started this this topic, it was all about
> having a loose collection of project with dependency references between
> them. And get rid of the superproject.
> It is my first and only goal.

In that case maybe what you want is your own completely independent
implementation of a tool which manages a collection of submodules. I
doubt that tracking of submodules outside of the worktree will ever be
accepted inside core git. Some dependency scheme which makes use of the
current submodule implementation could be a feasible way but you can not
get rid of the superproject.

Your approach introduces many problems which you were not able to
present solutions for. So if you really want to work on this I suggest
you try to implement your solution outside of core git first. If you can
prove that you can solve all the immanent problems we can discuss
integrating it into git again.

> > Henri wanted it a bit upside down: any submodule could request a certain
> > version of another submodule somewhere else in the repo. And he wanted to
> > use gitlinks from one submodule to another for that, which I - hopefully -
> > convinced him was no good idea.
> > 
> 
> You just convince me that submodules are more than I need and to make a
> lighter independent version of submodules which will never been followed
> by git commands.

As described above this is probably the best way for you. Maybe you can
prove that such a tool works but git's submodules implementation can
currently not really assist you with your approach.

Cheers Heiko

Re: tracking submodules out of main directory.

[henri GEIST]

Le jeudi 04 août 2011 à 19:45 +0200, Jens Lehmann a écrit :
> Am 04.08.2011 00:29, schrieb henri GEIST:
> > Le mercredi 03 août 2011 à 23:30 +0200, Jens Lehmann a écrit :
> >> Am 03.08.2011 21:41, schrieb Junio C Hamano:
> >>> Jens Lehmann <Jens.Lehmann@web.de> writes:
> >> But when you fetch a new version of Gimp into your submodule, it would be
> >> really nice if the superproject could be notified that the Gimp developers
> >> updated to 1.2.4 of Glib and inform you that an update of Glib might be
> >> appropriate. That could avoid having you to dig through compiler errors to
> >> find out that the new foobar() function from Glib 1.2.4 is needed (and if
> >> you need to pull in a bugfix in Glib, you might notice that *a lot* later
> >> when you forget to do that).
> >>
> > 
> > Exact, I am really happy to read this.
> > And better do not bother to have the suproject.
> 
> I don't get this, you can't get rid of the superproject.

Of corse I can.
I have done It the patch I have submit has been done for an do it
successfully.
I am doing it every day.

I have in the same "directory" not repository.
Two repository "gimp" and "libpng".
I cd in the gimp directory.
And do "git add ../libpng" and it is done.
The gimp repository track the "../libpng" repository by its path and
sha1.

I do not need anything else.

> 
> > cd to gimp directory, type git status it can tell you every thing and
> > when your satisfied you just have to type make.
> > At this point the superproject have not any use. 
> 
> "git status" inside the submodule won't tell you anything about the
> dependencies,

Of corse It does if the repository add been add
It will just like any other submodules inside of the repository
Git status is even unable to see the difference.
Essential because there is no difference.

> but a "git status" in the superproject should.

Yes but I have no and want not to have a superproject.

> The submodule shouldn't know where exactly the submodules it depends on lives, that is
> the decision of the superproject, not the submodule.

Why not in my case it is needed has in "gimp.c" I have

#include "../libpng.h"

If the code need to know where is libpng why the repository should not
know it as well ?

> 
> >>>> In addition to that, it can (but mustn't) specify any of the following:
> >>>
> >>> I am guessing you meant "does not have to", instead of mustn't, here...
> >>
> >> Sure, thanks for deciphering that.
> >>
> >>>> a) Of this submodule "foo" I need at least that version because I won't
> >>>>    compile/work with older versions of that. (this can be tightened to
> >>>>    "exactly that version" to give henri the behavior he wants, but that
> >>>>    should be policy, not mandatory)
> >>>
> >>> The "loose collection of projects" approach like that has its uses, and it
> >>> is called "repo". Why re-invent it? The behaviour Henri wants to specify
> >>> the exact version is how git submodules work already, so I do not see
> >>> there is anything to be done here.
> >>
> >> Let me make this clear: this is not about changing how submodules are
> >> committed in a superproject. It is not about having a loose collection of
> >> projects, they stay tied together in a defined state by the superproject.
> >>
> > 
> > Yes but for me, from when I started this this topic, it was all about
> > having a loose collection of project with dependency references between
> > them. And get rid of the superproject.
> > It is my first and only goal.
> 
> But I fail to see how that would improve anything.
> 

It improve my life and the life of my team has developers everyday.
All my team agree on this point. In fact they asked me to implement it.
Because we used to have a superproject and it does not scale well. 


The "gimp" team and the "gqview" team shares the libpng but at there
point of view they are not related they even not need to know that the
other one exist. Then they will never share the same superproject. It
has no sens for them.
Then in your model you will need a superproject by project.
I think we should put all that you put in the superproject in the
project itself. Then there will be only one project by project.

And if the dependency are not part of the superproject, this remove the
burden of each team to redefine all the dependencies recursively in any
dependency they have. has the original "libpng" team have done it in the
"libpng" project and not in there own superprojec witch the can not
share.

> >> Henri wanted it a bit upside down: any submodule could request a certain
> >> version of another submodule somewhere else in the repo. And he wanted to
> >> use gitlinks from one submodule to another for that, which I - hopefully -
> >> convinced him was no good idea.
> >>
> > 
> > You just convince me that submodules are more than I need and to make a
> > lighter independent version of submodules which will never been followed
> > by git commands.
> 
> Submodules are what you need, but it's no use to implement dependencies by
> using gitlinks that point outside of their repositories.
> 

Submodules is not what I need.
What I need is pointers to external repository tracked by git.
Submodule was to my eyes the logical tool to do it.
And I had made a patch to enable the tracking of external repository
just has they do for internal ones.

You convince me that you think prohibiting me to do so is a feature to
your eyes. Because gitlinks and submodules can do more than this and you
do not want those other things to work outside of the repository.

Then I propose to cook a kind of "my_light_gitlinks" which do only that.

> >>>> b) And if you don't know where to get it, use this url
> >>>
> >>> Again that is the job of .gitmodules in the superproject.
> >>
> >> Yes. But this idea is about how the url could get into the .gitmodules of
> >> the superproject in the first place. That can make it easier for the
> >> superproject's developer to import a submodule into his repo and much more
> >> important: it makes it possible to pull in submodule dependencies
> >> automatically e.g. when running "git submodule add --resolve-dependencies
> >> Gimp".
> > 
> > Only if you have a superproject.
> > If not do the same thing from the gimp repository, now it contain all
> > necessary infos to do the job.
> 
> No, it doesn't. Apart from the Gimp project telling you what version it
> wants, you need to have a place to record the version that the user really
> used. And that won't work without a superproject.
> 

the version of "libpng" that the user really use is stored in his
"libpng" repository !
And tracked by every project depending on it.

> >>>> That is all stuff the submodule knows better than the superproject.
> >>>
> >>> Not necessarily. The version A0 of submodule A may depend on submodule B
> >>> and may also know it must have at least version B0 of that submodule, but
> >>> the superproject would know other constraints, e.g. the superproject
> >>> itself also calls into submodule B and wants a newer version B1 of it.
> >>
> >> Right. That's what I tried to explain to Henri, the superproject ties it all
> >> together. But I also like his idea to add a way to communicate information
> >> from the submodule to the superproject, and give the superproject a choice
> >> if it wants to use it.
> >>
> > 
> > yes but the superproject contain no code in your design.
> > Then it will never need anything by itself.
> > It is only a container which you will inform with data already known by
> > the submodules I do not see any value to it.
> 
> But being the container that ties it all together is more than enough value.
> 

It has absolutely no value to have a git repository as a container.
A simple directory will do it very well.

	Henri

Re: tracking submodules out of main directory.

[henri GEIST]

Le jeudi 04 août 2011 à 22:05 +0200, Heiko Voigt a écrit :
> Hi,
> 
> On Thu, Aug 04, 2011 at 12:29:22AM +0200, henri GEIST wrote:
> > Le mercredi 03 ao??t 2011 ?? 23:30 +0200, Jens Lehmann a ??crit :
> > > Let me make this clear: this is not about changing how submodules are
> > > committed in a superproject. It is not about having a loose collection of
> > > projects, they stay tied together in a defined state by the superproject.
> > > 
> > 
> > Yes but for me, from when I started this this topic, it was all about
> > having a loose collection of project with dependency references between
> > them. And get rid of the superproject.
> > It is my first and only goal.
> 
> In that case maybe what you want is your own completely independent
> implementation of a tool which manages a collection of submodules.

If I make a tool to managed a collection of submodules, it will became
the superproject I want to get rid of.
I do not want to have any concept of a collection. And that is why I do
not want a superproject.

I have no exhaustive list of project/modules/submodules.
Only some independent projects witch knows if they need some others.

Then I realy need the tool to be embedded in the repository.

> I doubt that tracking of submodules outside of the worktree will ever be
> accepted inside core git. Some dependency scheme which makes use of the
> current submodule implementation could be a feasible way but you can not
> get rid of the superproject.

I can not but have already done it.

> 
> Your approach introduces many problems which you were not able to
> present solutions for.

But it does not break anything.
And generate no problem for me.
There is only when you try to get one step more and try to use even more
advanced things than I do on top of it.
And those problem also apply to normal submodules.

> So if you really want to work on this I suggest
> you try to implement your solution outside of core git first. If you can
> prove that you can solve all the immanent problems we can discuss
> integrating it into git again.
> 

That is what I had proposed to do by making a file .gitdependencies
similar to .gitmodules and create shell script wrappers for :
  - git add
  - git status
  - git reset
  - git rm

the only thing is that with the previous patch I only had to touch "git
add" every thing else was already ready to handle it.

> > > Henri wanted it a bit upside down: any submodule could request a certain
> > > version of another submodule somewhere else in the repo. And he wanted to
> > > use gitlinks from one submodule to another for that, which I - hopefully -
> > > convinced him was no good idea.
> > > 
> > 
> > You just convince me that submodules are more than I need and to make a
> > lighter independent version of submodules which will never been followed
> > by git commands.
> 
> As described above this is probably the best way for you. Maybe you can
> prove that such a tool works but git's submodules implementation can
> currently not really assist you with your approach.

We will see.

	Henri