[RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

This configuration variable comes into effect when 'git clone' is
invoked inside an existing git repository's worktree.  When set,
instead of cloning the given repository as-is, it relocates the gitdir
of the repository to the path specified by this variable.  This
setting is especially useful when working with submodules.

Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
 Okay, so this is part of my evil plan to make 'git add' DTRT wrt
 submodules, and deprecate 'git submodule add' (I have some code
 written down, but this is a prerequisite: I don't like the
 .git/modules nonsense).

 Unfortunately, this patch is in pathetic shape and is an RFC for
 three reasons:

 1. I've used setup_git_directory_gently() at the start of
    builtin/clone.c to check if I'm inside a git directory.  This
    breaks a lot of existing tests (I'm yet to understand these
    failures fully).

 2. setup_git_directory_gently() has the side-effect of changing the
    current directory and calling set_git_work_tree(), both of which
    must be done away with if we want the rest of clone.c to work.
    I've hacked around the issue in a very dirty manner.  What is the
    solution to this?

  3. I don't know how to test the case "clone.submoduleGitDir has no
     effect outside a git repository", because our entire test
     environment is a git repository.  Even if I remove the .git
     directory, we're still inside the soure tree's git repository.
     What do I do about this?  Even if we decide that this patch is
     fundamentally unworkable, we should try to fix this issue so that
     we can verify that a plain 'git clone' works outside a git
     repository.

  Thanks for reading.  And I'm truly sorry for making you read through
  such ugly code.

 Documentation/config.txt | 11 +++++++++++
 builtin/clone.c          | 33 ++++++++++++++++++++++++++++++++-
 environment.c            | 11 -----------
 t/t5702-clone-options.sh | 41 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 84 insertions(+), 12 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 3d750e0..aac26c3 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -798,6 +798,17 @@ clean.requireForce::
 	A boolean to make git-clean do nothing unless given -f
 	or -n.   Defaults to true.
 
+clone.submoduleGitDir::
+	An absolute path on the filesystem where gitdirs of submodules
+	should be stored away safely.  When not set, a 'git clone'
+	executed inside a git repository will do exactly what it does
+	outside a git repository.  When set, a 'git clone' executed
+	inside a git repository will create the worktree in place of
+	the full repository, and put the object store in a
+	subdirectory of clone.submoduleGitDir, choosing the name to be
+	the "humanish" part of the source repository (`repo.git` for
+	`/path/to/repo.git` and `foo.git` for `host.xz:foo/.git`).
+
 color.branch::
 	A boolean to enable/disable color in the output of
 	linkgit:git-branch[1]. May be set to `always`,
diff --git a/builtin/clone.c b/builtin/clone.c
index f9c380e..4a845a4 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -44,6 +44,7 @@ static char *option_template, *option_depth;
 static char *option_origin = NULL;
 static char *option_branch = NULL;
 static const char *real_git_dir;
+static const char *submodule_gitdir;
 static char *option_upload_pack = "git-upload-pack";
 static int option_verbosity;
 static int option_progress = -1;
@@ -707,12 +708,22 @@ static void write_refspec_config(const char* src_ref_prefix,
 	strbuf_release(&value);
 }
 
+static int git_clone_config(const char *var, const char *value, void *cb)
+{
+	if (!strcmp(var, "clone.submodulegitdir")) {
+		git_config_string(&submodule_gitdir, var, value);
+		return 0;
+	}
+	return git_default_config(var, value, cb);
+}
+
 int cmd_clone(int argc, const char **argv, const char *prefix)
 {
 	int is_bundle = 0, is_local;
 	struct stat buf;
 	const char *repo_name, *repo, *work_tree, *git_dir;
-	char *path, *dir;
+	char *path, *dir, *dest_git_dir;
+	char cwd[PATH_MAX];
 	int dest_exists;
 	const struct ref *refs, *remote_head;
 	const struct ref *remote_head_points_at;
@@ -725,6 +736,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	const char *src_ref_prefix = "refs/heads/";
 	struct remote *remote;
 	int err = 0, complete_refs_before_fetch = 1;
+	int nongit = 1;
 
 	struct refspec *refspec;
 	const char *fetch_pattern;
@@ -732,6 +744,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	junk_pid = getpid();
 
 	packet_trace_identity("clone");
+
+	/* setup_git_directory_gently without changing directories */
+	getcwd(cwd, sizeof(cwd) - 1);
+	setup_git_directory_gently(&nongit);
+	chdir(cwd);
+
+	git_config(git_clone_config, NULL);
+
 	argc = parse_options(argc, argv, prefix, builtin_clone_options,
 			     builtin_clone_usage, 0);
 
@@ -785,6 +805,17 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		die(_("destination path '%s' already exists and is not "
 			"an empty directory."), dir);
 
+	if (!nongit && submodule_gitdir) {
+		char  *user_path = expand_user_path(submodule_gitdir);
+		if (!user_path)
+			die(_("Unable to expand path in clone.submoduleGitDir: %s"), submodule_gitdir);
+		dest_git_dir = mkpathdup("%s/%s.git", user_path, dir);
+		if (!stat(dest_git_dir, &buf) && !is_empty_dir(dest_git_dir))
+			die(_("destination path '%s' already exists and is not "
+					"an empty directory."), dest_git_dir);
+		real_git_dir = dest_git_dir;
+	}
+
 	strbuf_addf(&reflog_msg, "clone: from %s", repo);
 
 	if (option_bare)
diff --git a/environment.c b/environment.c
index e2e75c1..9dce4c7 100644
--- a/environment.c
+++ b/environment.c
@@ -182,8 +182,6 @@ const char *strip_namespace(const char *namespaced_ref)
 	return namespaced_ref + namespace_len;
 }
 
-static int git_work_tree_initialized;
-
 /*
  * Note.  This works only before you used a work tree.  This was added
  * primarily to support git-clone to work in a new repository it just
@@ -191,15 +189,6 @@ static int git_work_tree_initialized;
  */
 void set_git_work_tree(const char *new_work_tree)
 {
-	if (git_work_tree_initialized) {
-		new_work_tree = real_path(new_work_tree);
-		if (strcmp(new_work_tree, work_tree))
-			die("internal error: work tree has already been set\n"
-			    "Current worktree: %s\nNew worktree: %s",
-			    work_tree, new_work_tree);
-		return;
-	}
-	git_work_tree_initialized = 1;
 	work_tree = xstrdup(real_path(new_work_tree));
 }
 
diff --git a/t/t5702-clone-options.sh b/t/t5702-clone-options.sh
index 02cb024..9b845d8 100755
--- a/t/t5702-clone-options.sh
+++ b/t/t5702-clone-options.sh
@@ -33,4 +33,45 @@ test_expect_success 'redirected clone -v' '
 
 '
 
+test_expect_success 'clone.submoduleGitDir takes effect in a git repository' '
+	cd ~ &&
+	rm -rf bare newrepo superproject &&
+	mkdir bare &&
+	bare_path="$(pwd)/bare" &&
+	git init newrepo &&
+	(
+		cd newrepo &&
+		echo quux >foo &&
+		git add foo &&
+		git commit -m "Add foo"
+	) &&
+	git init superproject &&
+	cd superproject &&
+	test_config clone.submoduleGitDir "$bare_path" &&
+	git clone ../newrepo &&
+	test_path_is_file newrepo/.git &&
+	cd ../bare/newrepo.git &&
+	git rev-parse --is-bare-repository
+'
+
+test_expect_success 'clone.submoduleGitDir path get tilde-expansion' '
+	cd ~ &&
+	rm -rf bare newrepo superproject &&
+	mkdir bare &&
+	git init newrepo &&
+	(
+		cd newrepo &&
+		echo quux >foo &&
+		git add foo &&
+		git commit -m "Add foo"
+	) &&
+	git init superproject &&
+	cd superproject &&
+	test_config clone.submoduleGitDir ~/bare &&
+	git clone ../newrepo &&
+	test_path_is_file newrepo/.git &&
+	cd ../bare/newrepo.git &&
+	git rev-parse --is-bare-repository
+'
+
 test_done
-- 
1.8.2.1.389.gcaa7d79.dirty

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Junio C Hamano]

Ramkumar Ramachandra <artagnon@gmail.com> writes:

> This configuration variable comes into effect when 'git clone' is
> invoked inside an existing git repository's worktree.  When set,
> instead of cloning the given repository as-is, it relocates the gitdir
> of the repository to the path specified by this variable.

Relocate to where in the superproject's gitdir?  Presumably you can
do this more than once in a given superproject, so there needs to be
a key per such a clone, no?  I am guessing that you would follow the
usual "when adding a submodule without name, use its path as the
initial name" convention, but then I would suggest it to be spelled
out (and if you are doing it differently, that choice needs to be
spelled out and defended).

>  Okay, so this is part of my evil plan to make 'git add' DTRT wrt
>  submodules,...

If the envisioned use of this is to use it as a building block of
something else that is user-facing (e.g. the user says "git add",
and before the command finishes, somewhere we internally run "git
clone"), then would it be possible that you are better off running
that clone with --separate-git-dir and let it make the gitfile for
you?

Any new configuration variable brings its own problem by forcing
existing users to countermand it explicitly from the command line.
If the --separate-git-dir would not work for your application, you
need a new feature and you can achieve the same by adding a new
command line option (say, --submodule-git-dir), that would be more
preferrable.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> If the envisioned use of this is to use it as a building block of
> something else that is user-facing (e.g. the user says "git add",
> and before the command finishes, somewhere we internally run "git
> clone"), then would it be possible that you are better off running
> that clone with --separate-git-dir and let it make the gitfile for
> you?

As you may have already guessed, in principle I am all for teaching
"git add" not just to add a submodule itself (which we already do)
but also to record information about the submodule, without having
to delegate it to "git submodule".  "git submodule add" was meant as
an interim measure until we figure out what kind of metainformation
is necessary, and doing things in "git add" has always been a longer
term goal.

There are two ways to "add" a submodule to a superproject.  You may
bring an existing project with "git clone" inside the working tree
of a superproject (which I am guessing is the use case that inspired
this patch), but it will leave the git dir of the submodule embedded
in its working tree.  

You could continue "git clone" and then teach "git add" (or "git
submodule add") to relocate the embedded git directory from the
submodule working tree, you could "git clone" with separate-git-dir
from the beginning, or you could extend "git add", perhaps

    git add --url=git://up.stre.am/repository [--name=name] sub/mod/ule

and do that "git clone --separate-git-dir" internally (which will
mean that the end user will not run "git clone").

Another way ti "add" a submodule is to run "git init" to originate a
new project inside the working tree of a superproject. The resulting
submodule working tree will have the embedded git dir, and again
"git add" (or "git submodule add") could notice and relocate it, but
if the extended "git add" wants to help that use case as well, I
think it is the matter of running "git init --separate-git-dir",
just like "add by cloning from elsewhere" can do the same with the
flag to "git clone".

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Junio C Hamano wrote:
> Relocate to where in the superproject's gitdir?  Presumably you can
> do this more than once in a given superproject, so there needs to be
> a key per such a clone, no?  I am guessing that you would follow the
> usual "when adding a submodule without name, use its path as the
> initial name" convention, but then I would suggest it to be spelled
> out (and if you are doing it differently, that choice needs to be
> spelled out and defended).

I probably wasn't clear enough in the commit message, but this is what
happens when I set clone.submoduleGitDir to ~/bare: a git clone
gh:artagnon/clayoven inside the superproject's worktree will make
~/bare/clayoven.git and ./clayoven corresponding to the GITDIR and the
worktree of the newly cloned repository.  If there are conflicts, it
will complain as usual saying that the destination path %s already
exists, in which case the user has to choose a name for the GITDIR
(not yet implemented) and/or the worktree path (as the final
command-line argument to git clone).

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

> You could continue "git clone" and then teach "git add" (or "git
> submodule add") to relocate the embedded git directory from the
> submodule working tree, you could "git clone" with separate-git-dir
> from the beginning, or you could extend "git add", perhaps
>
>     git add --url=git://up.stre.am/repository [--name=name] sub/mod/ule
>
> and do that "git clone --separate-git-dir" internally (which will
> mean that the end user will not run "git clone").

I specifically did not go down this route, because I think it is
gross.  Where does moving a GITDIR fit into what git add's normal job
(index manipulation) is?  Tools should do one specific thing, and do
it well: not a mixed bag of unrelated things.  git clone, on the other
hand, was always intended to have a way to point to a location for
GITDIR and the worktree: isn't this feature very close to
--separate-git-dir already?  It is, therefore, git clone's job to
relocate the GITDIR.  My future plan is to deny git add'ing anything
but a worktree-with-a-gitfile.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Junio C Hamano wrote:
> Any new configuration variable brings its own problem by forcing
> existing users to countermand it explicitly from the command line.
> If the --separate-git-dir would not work for your application, you
> need a new feature and you can achieve the same by adding a new
> command line option (say, --submodule-git-dir), that would be more
> preferrable.

I'm getting a little tired of your first instinct to oppose every new
addition to git. (Ofcourse I understand your attitude as the
maintainer, but still)

It doesn't make sense as a command-line option, because it is "magic"
that kicks in only when git clone is executed inside an existing git
worktree.  The point is that the user doesn't have to remember
anything special: a normal git clone already does the right thing
outside a git worktree; my proposal is to make it do the right thing
inside a git worktree as well.  Although I'm not against allowing a
user to create a "full clone" inside a git repository by overriding
clone.submoduleGitDir via a command-line option, I really cannot see
why this would be anything but rare.  Why would a user *want* a full
clone inside a git worktree?

Also, naming it --submodule-git-dir can cause a lot of confusion:
--separate-git-dir names a specific directory to put the GITDIR in,
while --submodule-git-dir names a directory inside which to create
other named directories to put GITDIRs in.  Ofcourse
clone.submoduleGitDir is a bad name too: any suggestions?

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Duy Nguyen]

On Mon, Apr 15, 2013 at 6:19 PM, Ramkumar Ramachandra
<artagnon@gmail.com> wrote:
> It doesn't make sense as a command-line option, because it is "magic"
> that kicks in only when git clone is executed inside an existing git
> worktree.  The point is that the user doesn't have to remember
> anything special: a normal git clone already does the right thing
> outside a git worktree; my proposal is to make it do the right thing
> inside a git worktree as well.  Although I'm not against allowing a
> user to create a "full clone" inside a git repository by overriding
> clone.submoduleGitDir via a command-line option, I really cannot see
> why this would be anything but rare.  Why would a user *want* a full
> clone inside a git worktree?

If a user is inside .git, I believe setup_git_directory() will also
find correct gitdir. In that case, we do not want magic (i.e. only do
your magic when you are inside worktree). Still I'd rather see no
magic (i.e. command line option) first. Let people try it out for a
while. If people like it and find it inconvenient, magic can come
later. I suspect you might want more magic in other places. Maybe if
you hold it back  until you see full picture, you'll only need a few
new config keys (instead of one per separate magic).

>  Unfortunately, this patch is in pathetic shape and is an RFC for
>  three reasons:
>
>  1. I've used setup_git_directory_gently() at the start of
>     builtin/clone.c to check if I'm inside a git directory.  This
>     breaks a lot of existing tests (I'm yet to understand these
>     failures fully).
>
>  2. setup_git_directory_gently() has the side-effect of changing the
>     current directory and calling set_git_work_tree(), both of which
>     must be done away with if we want the rest of clone.c to work.
>     I've hacked around the issue in a very dirty manner.  What is the
>     solution to this?

Just do what scripts do: spawn a process to run rev-parse so that it
does not mess up the main process. You might be able to introduce
"dry-run" mode for setup_git_directory(), but that won't be easy.
--
Duy

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Junio C Hamano]

Ramkumar Ramachandra <artagnon@gmail.com> writes:

> Junio C Hamano wrote:
>> Any new configuration variable brings its own problem by forcing
>> existing users to countermand it explicitly from the command line.
>> If the --separate-git-dir would not work for your application, you
>> need a new feature and you can achieve the same by adding a new
>> command line option (say, --submodule-git-dir), that would be more
>> preferrable.
>
> I'm getting a little tired of your first instinct to oppose every new
> addition to git. (Ofcourse I understand your attitude as the
> maintainer, but still)

It was purly about "do not add anything that makes no sense." and 
not about "oppose all new addition."

When you add a submodule with the current system, bypassing "git
submodule add", you can either

    (1) "git clone $URL here" and then "git add here"; or
    (2) "git init here" and then "git add here".

Because you didn't say what you are aiming for in the grander
picture, I thought you were "making the UI simpler" by making it
unnecessary for the users to say "git clone" himself as a separate
step before doing "git add". In such a world, "add" would internally
run "clone". If that were the case (I now know it is not), then the
configuration _is_ unnecessary, and it is perfectly valid to
question why you thought it is needed.

If your plan is instead to keep "git clone" followed by "git add" as
the pattern for use case (1), teaching "clone" to automatically use
the --separate-git-dir mechanism to point at the right place inside
the $GIT_DIR of the superproject does make sense to help the use
case.

But if that is the direction you are aiming for, would it be
possible that the same configuration variable can and should cover
the use case (2) as well?  After all, between "git init here" and
"git add here", the user may say (cd here && git pull $URL) and the
expected end result would be the same as (1), no?

I do not recall the details of the codepaths involved offhand, but
when you "git clone $URL [here]", after running "mkdir here", it
would create a $GIT_DIR for the "here" repository in "here/.git"
(and with --separate-git-dir, it would create it elsewhere and drop
gitfile at "here/.git").  When you "git init here", after running
"mkdir here", the same thing happens.

How common are these two implementations?

If "clone" just calls init_db(), I would imagine that it might be
trivial to cover both cases by telling init_db() to pay attention to
the configuration, without doing much in the "clone" itself.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Duy Nguyen wrote:
> If a user is inside .git, I believe setup_git_directory() will also
> find correct gitdir. In that case, we do not want magic (i.e. only do
> your magic when you are inside worktree). Still I'd rather see no
> magic (i.e. command line option) first. Let people try it out for a
> while. If people like it and find it inconvenient, magic can come
> later. I suspect you might want more magic in other places. Maybe if
> you hold it back  until you see full picture, you'll only need a few
> new config keys (instead of one per separate magic).

Good suggestion.  I'll make it a command-line option for now.

> Just do what scripts do: spawn a process to run rev-parse so that it
> does not mess up the main process. You might be able to introduce
> "dry-run" mode for setup_git_directory(), but that won't be easy.

Okay, thanks.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Junio C Hamano]

Ramkumar Ramachandra <artagnon@gmail.com> writes:

> I specifically did not go down this route, because I think it is
> gross.  Where does moving a GITDIR fit into what git add's normal job
> (index manipulation) is?  Tools should do one specific thing, and do
> it well: not a mixed bag of unrelated things.

I see you are trying to repeat the UNIX mantra, but I do not think
it is working.

When we discuss "git add", the "one unit of work" is at much higher
level than that of "git update-index".  "git add dir/" has to do a
lot more than "git add file", and "git add symlink" has to do quite
a different thing from "git add file", but to the end user, all of
them are about doing everything necessary to add what the user named
to the index. "git add submodule/" that does whatever necessary to
add the submodule to the index is still doing one thing well inside
the same framework, and that may include moving the $GIT_DIR and
turning it into a gitfile.

Not that I am saying I prefer "add --url=xxx". Quite the opposite.
I very much prefer the "clone and then add, but clone drops the
repository at the right place from the beginning" approach than "add
that knows about URL only for submodules", which is an ugly kludge.

If the user creates here/.git without gitlink with whatever means,
it is "git add here"'s job, if it wants to make it a submodule and
if it wants to make it possible to later check out another branch
that does not have the submodule, to stash away the repository and
turn it into gitfile, if it is part of what is needed to add a
submodule.

Of course, we could start from teaching "submodule add" to do so,
and then internally redirect "git add subm" to "git submodule add",
but that is a minor implementation detail that does not affect the
end user experience.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Junio C Hamano wrote:
> When we discuss "git add", the "one unit of work" is at much higher
> level than that of "git update-index".  "git add dir/" has to do a
> lot more than "git add file", and "git add symlink" has to do quite
> a different thing from "git add file", but to the end user, all of
> them are about doing everything necessary to add what the user named
> to the index. "git add submodule/" that does whatever necessary to
> add the submodule to the index is still doing one thing well inside
> the same framework, and that may include moving the $GIT_DIR and
> turning it into a gitfile.

You're looking at it from an end-user point of view, while I'm looking
at it from the implementation point of view.  Here's a coarse
simplification of what git add does:

1. Lock the index file, and grab a FILE handle to read/ write the file.

2. Update active_cache.  Depending on the pathspec, we might be adding
one entry or multiple entries, with different modes to the index.
Nowhere did I say that it should add exactly one entry to active_cache
with a predefined mode.  Sure, tools that operate a lower layer of
abstraction like update-index can be more picky about this.

3. Write tree and blob objects to the database corresponding to the
worktree entries.  Files and symbolic links get blob objects, while
directories get tree objects.

4. Write active_cache to FILE handler we grabbed in step 1, and
release the lock.

What it does not do:

1. Move random files/ directory around in the worktree.

2. Mangle existing files in the worktree. (Although I know that the
.gitmodules-mangling is coming soon, I'm not exactly elated with it
[1])

3. Write commit or tag objects to the database.

4. Update random refs.

5. Make coffee for the user to applaud him on the successful add.

In my opinion, with some minor exceptions, all git tools follow these
principles.  Briefly, branch is a refs/heads/* helper, checkout is a
HEAD + worktree helper, fetch is a receive-pack + refs/remotes/*
helper, and reset is a bit of a swiss army knife that operates on HEAD
+ index + worktree.

In general, I like git because commands don't create unnatural or
heavy abstractions on top of these concepts.  With some minor
exceptions, all the commands are easy to understand and consistent.

[1]: This is what led to my OBJ_LINK proposal.

> Not that I am saying I prefer "add --url=xxx". Quite the opposite.
> I very much prefer the "clone and then add, but clone drops the
> repository at the right place from the beginning" approach than "add
> that knows about URL only for submodules", which is an ugly kludge.

I don't know why you brought up the alternative in the first place.
We both agree that it is git clone's job, although your reason is more
superficial and mine's tied to the implementation.

> If the user creates here/.git without gitlink with whatever means,
> it is "git add here"'s job, if it wants to make it a submodule and
> if it wants to make it possible to later check out another branch
> that does not have the submodule, to stash away the repository and
> turn it into gitfile, if it is part of what is needed to add a
> submodule.

I disagree.  I think we should get a first-class tool to attach/
detach worktrees from a GITDIR.  It can incoporate the logic from
contrib/workdir/git-new-workdir to optionally create a worktree with
an independent index, HEAD, and logs/HEAD.

> Of course, we could start from teaching "submodule add" to do so,
> and then internally redirect "git add subm" to "git submodule add",
> but that is a minor implementation detail that does not affect the
> end user experience.

Yuck.  Don't you care about the implementation, as long as it fixes
the end-user's problem?

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Junio C Hamano wrote:
> When you add a submodule with the current system, bypassing "git
> submodule add", you can either
>
>     (1) "git clone $URL here" and then "git add here"; or
>     (2) "git init here" and then "git add here".
>
> Because you didn't say what you are aiming for in the grander
> picture, I thought you were "making the UI simpler"

In the original email, I wrote:
>  Okay, so this is part of my evil plan to make 'git add' DTRT wrt
>  submodules, and deprecate 'git submodule add' (I have some code
>  written down, but this is a prerequisite: I don't like the
>  .git/modules nonsense).

I'm not sure how you inferred "making the UI simpler" from that, or the tests.

> by making it
> unnecessary for the users to say "git clone" himself as a separate
> step before doing "git add". In such a world, "add" would internally
> run "clone". If that were the case (I now know it is not), then the
> configuration _is_ unnecessary, and it is perfectly valid to
> question why you thought it is needed.

No, I would _never_ propose something that ugly.  Neither my code,
tests, nor my commit message indicates that I was going in that
direction, so I don't know where you got the idea from.

> But if that is the direction you are aiming for, would it be
> possible that the same configuration variable can and should cover
> the use case (2) as well?  After all, between "git init here" and
> "git add here", the user may say (cd here && git pull $URL) and the
> expected end result would be the same as (1), no?

Good point.  Yes, I would definitely want that.

> If "clone" just calls init_db(), I would imagine that it might be
> trivial to cover both cases by telling init_db() to pay attention to
> the configuration, without doing much in the "clone" itself.

Right.  I'll start hacking.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Marc Branchaud]

In general I think it is a mistake to overload "git clone" with the notion of
adding a submodule.  If I want to *add* something to a repository, I'll use
some kind of "add" command.  To me "git clone" is not the kind of verb I
would expect to add something to some distant-parent .git directory.

Instead of mucking around with"git clone" I would much rather see "git add"
autodetect URLs and do the submodule thing:
	git add ssh://host/blammo.git
would clone blammo.git into ./blammo/ and set it up as a submodule inside
$PWD's git repo.  (This may benefit from "git clone" learning some kind of
--separate-git-dir option, but that's irrelevant to me.)

On 13-04-15 04:19 AM, Ramkumar Ramachandra wrote:
>
> Why would a user *want* a full clone inside a git worktree?

Please try to be careful with your assumptions.

I could have
	~/.git/
to maintain revisions of various personal files, config .dotfiles, scripts in
~/bin/ and so on.

I could also have various projects' repos under ~/Code, where I do my "real"
work:
	~/Code/git/.git/
	~/Code/DayJob/.git/
	~/Code/project-foo/.git/

Now, are these Code/* repos inside ~/.git/'s worktree or not?  I'd really
prefer them not to be.  I would be especially upset to have some "magic" that
automatically adds new clones inside ~/Code/ to ~/.git/.

		M.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Junio C Hamano]

Marc Branchaud <mbranchaud@xiplink.com> writes:

> In general I think it is a mistake to overload "git clone" with the notion of
> adding a submodule.

I agree with that principle, but my understanding is that this
effort is not about teaching "git clone" to create a submodule.

Both "git clone" and "git init" already know how to use a directory
that is outside the working tree of the newly created repository to
store its $GIT_DIR and point at it with .git in the working tree
using the gitfile mechanism (their --separate-git-dir option).  My
understanding is that this "config" is about making that option
easier to use when you _know_ any new repository you create with
"git clone" or "git init" inside your (toplevel super)project's
working tree will become its submodule, as it is more convenient to
have their $GIT_DIR inside the .git/modules/$name of the
superproject.

After that "clone" or "init" creates a repository, you still have to
"add" if you want to make it a submodule to the toplevel.

> If I want to *add* something to a repository, I'll use
> some kind of "add" command.  To me "git clone" is not the kind of verb I
> would expect to add something to some distant-parent .git directory.
>
> Instead of mucking around with"git clone" I would much rather see "git add"
> autodetect URLs and do the submodule thing:
>
> 	git add ssh://host/blammo.git
>
> would clone blammo.git into ./blammo/ and set it up as a submodule inside
> $PWD's git repo.

I do not think the addition Ram is envisioning in the patch will
prevent you from teaching "add" to do that.  An implemention of such
an addition indeed would most likely use the same --separate-git-dir
mechanism anyway.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Marc Branchaud wrote:
>         git add ssh://host/blammo.git

Heh.  And I want git add *coffee* to make me coffee.
What's your gripe with git submodule add?

> I could have
>         ~/.git/
> to maintain revisions of various personal files, config .dotfiles, scripts in
> ~/bin/ and so on.
> [...]
> Now, are these Code/* repos inside ~/.git/'s worktree or not?

Please don't version your entire ~, effectively shooting yourself in the face?

Use a dotfiles repo and write a simple Makefile to symlink ~/.etc to
~/dotfiles/.etc.  If you're looking for a good example, see
https://github.com/artagnon/dotfiles.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Junio C Hamano wrote:
> My
> understanding is that this "config" is about making that option
> easier to use when you _know_ any new repository you create with
> "git clone" or "git init" inside your (toplevel super)project's
> working tree will become its submodule, as it is more convenient to
> have their $GIT_DIR inside the .git/modules/$name of the
> superproject.

Right.  But I'm still worried about .git/modules/$name.  Can you
explain why it's a better idea than having a dedicated ~/bare?  In the
case when I have it in ~/bare, I can do many more interesting things:
for instance, if I cloned a repository that is actually another
project's submodule for instance, I don't have to re-clone it when I
clone that superproject.  What's more?  I can remove submodules and
attach a worktree to my ~/bare/repo.git and use it as a separate
repository easily.  I can move submodules between projects.  In
comparison, .git/modules/$name just seems like a mess.

> I do not think the addition Ram is envisioning in the patch will
> prevent you from teaching "add" to do that.  An implemention of such
> an addition indeed would most likely use the same --separate-git-dir
> mechanism anyway.

Well, I'm against the change in principle because add operates on
worktree paths, not URLs.  I don't want to change that arbitrarily.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Marc Branchaud]

On 13-04-15 01:50 PM, Junio C Hamano wrote:
> Marc Branchaud <mbranchaud@xiplink.com> writes:
> 
>> In general I think it is a mistake to overload "git clone" with the notion of
>> adding a submodule.
> 
> I agree with that principle, but my understanding is that this
> effort is not about teaching "git clone" to create a submodule.
> 
> Both "git clone" and "git init" already know how to use a directory
> that is outside the working tree of the newly created repository to
> store its $GIT_DIR and point at it with .git in the working tree
> using the gitfile mechanism (their --separate-git-dir option).  My
> understanding is that this "config" is about making that option
> easier to use when you _know_ any new repository you create with
> "git clone" or "git init" inside your (toplevel super)project's
> working tree will become its submodule, as it is more convenient to
> have their $GIT_DIR inside the .git/modules/$name of the
> superproject.
> 
> After that "clone" or "init" creates a repository, you still have to
> "add" if you want to make it a submodule to the toplevel.

To me it makes more sense to move the .git directory when the user invokes
"git submodule add" instead of creating it in an unusual place when the
sub-repo is cloned.  After all, git can't *know* that it'll be a submodule
until it's submodule-added to the super-repo.  Sure, the user might have set
clone.submoduleGitDir somewhere, but users make mistakes, and this setting
makes it harder to clean up a mistake:
	git clone foo.git
	# Doh!  I mean to clone foof.git!
	rm -rf foo
	# Gah, now there's cruft in my clone.submoduleGitDir...

All that said, the basic idea of being able to configure where "git clone"
stores .git directories might be reasonable.  Something like
clone.gitDirHome.  It seems like something only a git hacker would ever care
about, but that's no reason not to have such a config option.  OTOH, I still
don't see a reason for it, because I don't buy the submodule-at-clone-time
argument.

		M.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Jeff King]

On Mon, Apr 15, 2013 at 11:30:40PM +0530, Ramkumar Ramachandra wrote:

> Junio C Hamano wrote:
> > My
> > understanding is that this "config" is about making that option
> > easier to use when you _know_ any new repository you create with
> > "git clone" or "git init" inside your (toplevel super)project's
> > working tree will become its submodule, as it is more convenient to
> > have their $GIT_DIR inside the .git/modules/$name of the
> > superproject.
> 
> Right.  But I'm still worried about .git/modules/$name.  Can you
> explain why it's a better idea than having a dedicated ~/bare?

I do not have too much deep knowledge of submodules, nor have I been
following this thread very closely, but I have not seen how ~/bare would
handle per-submodule information?

That is, let us imagine I do:

  git clone $PROJECT one && cd one && git submodule update foo
  git clone $PROJECT two && cd two && git submodule update foo

The current scheme would put the cloned modules into
one/.git/modules/foo and two/.git/modules/foo, respectively. Let us
imagine instead that the first one writes to ~/modules/$URL (assuming
some sane mapping of the URL into the filesystem), and the second one
says "A-ha, I already have ~/modules/$URL, so I can skip cloning it".

But that is not the end of the story. If I do:

  cd one/foo &&
  hack hack hack &&
  git commit -m foo &&
  cd .. &&
  git commit -m 'updated submodule'

you would not want to see a dirty, updated submodule in project "two".
You did not touch "two/foo" nor advance its HEAD at all.

So there is some information that is per-clone (the objects, the remote
tips), but there is some information that is per-submodule (where our
local branches are, the index, the worktree). I can see why it is
advantageous to share the per-clone information between similar clones
(because it avoids disk space and network transfer). But I do not think
you can escape having some form of per-submodule repo, even if it is a
thin git-new-workdir-ish repo that points back to a parent repo for the
clone.

Is there some part of your proposal that I am missing? It seems like you
would still need one/.git/modules/foo for this "thin" repo.

And once we separate out those concerns, I also do not see why sharing
per-clone information needs to be related to submodules at all. If I do:

  git clone $URL one &&
  git clone $URL two

those can potentially be shared in the same way as two submodule repos
that happen to point to the same $URL. It would make sense to me to
improve such a shared-object setup independently, and then build the
shared-submodule storage on top of that.

And by the way, I am actually not sure that such a shared-object setup
is a good idea, but only that _if_ you are going to do it with
submodules, you might as well do it for all repos. In theory, it is not
that hard to have a big per-user object-only repository (either for all
repos, or for related ones). But we can do that already with "git clone
-s", and people do not generally bother, because the maintenance is very
tricky (especially dealing with reachability and pruning).

I am open to the argument that solving it in a specific case
(submodules) lets us make assumptions that simplify the problem from the
general case, but I do not offhand see how it would be any easier in
this case.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Junio C Hamano]

Marc Branchaud <marcnarc@xiplink.com> writes:

>> After that "clone" or "init" creates a repository, you still have to
>> "add" if you want to make it a submodule to the toplevel.
>
> To me it makes more sense to move the .git directory when the user invokes
> "git submodule add" instead of creating it in an unusual place when the
> sub-repo is cloned.  After all, git can't *know* that it'll be a submodule
> until it's submodule-added to the super-repo.

It does not relieve "git add" (or "git submodulea add") from the
responsibility of moving .git directory.  It only reduces the need
to do so.

When the user says "add" and the repository has .git directory in
it, "add" (or "submodule add") is still responsible for relocating
it.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Marc Branchaud]

On 13-04-15 02:00 PM, Ramkumar Ramachandra wrote:
> Junio C Hamano wrote:
>> 
>> I do not think the addition Ram is envisioning in the patch will
>> prevent you from teaching "add" to do that.  An implemention of such
>> an addition indeed would most likely use the same --separate-git-dir
>> mechanism anyway.
> 
> Well, I'm against the change in principle because add operates on
> worktree paths, not URLs.  I don't want to change that arbitrarily.

I don't understand that statement.

If "git add" is all about specifying what lives under paths in the worktree,
what's wrong with letting "git add" go beyond specifying just files?

Syntax aside for the moment, I think a command like
	git add git-repo-reference foo
is perfectly natural:  It specifies what is inside worktree path foo.

		M.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Marc Branchaud]

On 13-04-15 02:50 PM, Junio C Hamano wrote:
> Marc Branchaud <marcnarc@xiplink.com> writes:
> 
>>> After that "clone" or "init" creates a repository, you still have to
>>> "add" if you want to make it a submodule to the toplevel.
>>
>> To me it makes more sense to move the .git directory when the user invokes
>> "git submodule add" instead of creating it in an unusual place when the
>> sub-repo is cloned.  After all, git can't *know* that it'll be a submodule
>> until it's submodule-added to the super-repo.
> 
> It does not relieve "git add" (or "git submodulea add") from the
> responsibility of moving .git directory.  It only reduces the need
> to do so.
> 
> When the user says "add" and the repository has .git directory in
> it, "add" (or "submodule add") is still responsible for relocating
> it.

So it looks like the proposed change to git-clone provides no benefit to the
submodule-adding machinery, which still needs to know when and how to
relocate .git directories.

Ram, assuming Junio's explanations match your intentions, if the whole
motivation for this change is "to make 'git add' DTRT wrt submodules, and
deprecate 'git submodule add'" then I don't think it's bringing you any
closer to that goal.

		M.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> And by the way, I am actually not sure that such a shared-object setup
> is a good idea, but only that _if_ you are going to do it with
> submodules, you might as well do it for all repos. In theory, it is not
> that hard to have a big per-user object-only repository (either for all
> repos, or for related ones). But we can do that already with "git clone
> -s", and people do not generally bother, because the maintenance is very
> tricky (especially dealing with reachability and pruning).
>
> I am open to the argument that solving it in a specific case
> (submodules) lets us make assumptions that simplify the problem from the
> general case, but I do not offhand see how it would be any easier in
> this case.

Nicely put.

Making it easier to manage such a shared object store by limiting
use cases is somewhat an intriguing idea, but those I can think of
offhand all have to involve a use case without any rewound history,
so being a submodules repository would not help.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Junio C Hamano]

Marc Branchaud <marcnarc@xiplink.com> writes:

> So it looks like the proposed change to git-clone provides no benefit to the
> submodule-adding machinery, which still needs to know when and how to
> relocate .git directories.
>
> Ram, assuming Junio's explanations match your intentions, if the whole

That is a huge assumption, given that I have a proven track record
of guessing Ram's intention wrong ;-).

> motivation for this change is "to make 'git add' DTRT wrt submodules, and
> deprecate 'git submodule add'" then I don't think it's bringing you any
> closer to that goal.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Jonathan Nieder]

Ramkumar Ramachandra wrote:

>                                                        When set,
> instead of cloning the given repository as-is, it relocates the gitdir
> of the repository to the path specified by this variable.

Interesting.  As the discussion downthread from this illustrated, I am
not convinced this is better than a subcommand of "git submodule" for
that particular purpose, yet.

Is the goal to be able to, under some certain configuration, make
"git clone" + "git add" behave like "git submodule add"?

[...]
>                                            I don't like the
>  .git/modules nonsense).

As Jeff mentioned, a given repository can be a subproject of multiple
different containing projects, that use different versions of it.
It doesn't make sense for different directories on the filesystem to
share an index anyway.

Do you want the subprojects to be symlinks to the One True Version
of each project?  (I can see that working ok in some workflows.)  Or
do you want subprojects to be lightweight workdirs like
git-new-workdir creates, with .git/objects pointing to the project's
One True Object Store?

That is the part of this design that seems least well fleshed out to
me at the moment.

I quite like .git/modules/<subproject name> (for some reasons that
I've mentioned in other threads) and don't consider it nonsense, which
makes me assume I don't understand the goal of this patch, either.
Please don't take that personally.

Hope that helps,
Jonathan

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Jeff King wrote:
> So there is some information that is per-clone (the objects, the remote
> tips), but there is some information that is per-submodule (where our
> local branches are, the index, the worktree). I can see why it is
> advantageous to share the per-clone information between similar clones
> (because it avoids disk space and network transfer). But I do not think
> you can escape having some form of per-submodule repo, even if it is a
> thin git-new-workdir-ish repo that points back to a parent repo for the
> clone.

I want the flexibility to do the following:

1. Do a "simple clone", where the clone contains the GITDIR embedded
in the worktree.  This is the most common case, and there is no reason
to complicate it.  I can optionally attach additional workdirs to this
clone.  I can also optionally relocate the GITDIR at a later date, if
I feel the need to do so.

2. Attach a worktree to any object store without having to write a
gitfile and set core.worktree by hand.  The limitation is that you
can't have two submodules from two different superprojects sharing the
same object store (since both of them are worktrees).  However, for
the purpose of working on the submodule repository as an independent
repository (this is a very common case for me), I can attach a new
"workdir" to the GITDIR very easily.

3. Attach multiple submodules to the same object store.  This will
require maintaining a separate index, HEAD and logs/HEAD (aka.
workdir) for each additional submodule (the first one doesn't need it)
in .git/modules of the superproject.

> Is there some part of your proposal that I am missing? It seems like you
> would still need one/.git/modules/foo for this "thin" repo.

You're talking about #3, while I'm still working on #2.  And why do
you want to use a hammer again if I don't want to share the same
object store with multiple submodules?  This .git/modules/<name> is
completely optional, and is only required for the _second_ submodule
onwards that I'm attaching to the same object store.

> And by the way, I am actually not sure that such a shared-object setup
> is a good idea, but only that _if_ you are going to do it with
> submodules, you might as well do it for all repos. In theory, it is not
> that hard to have a big per-user object-only repository (either for all
> repos, or for related ones). But we can do that already with "git clone
> -s", and people do not generally bother, because the maintenance is very
> tricky (especially dealing with reachability and pruning).

No, no. I'm against dumping objects  from all repositories into one
giant object store.  That's a sledgehammer solution, while I'm looking
for control and flexibility.  Moreover, it has lots of downsides, as
you already pointed out.

> I am open to the argument that solving it in a specific case
> (submodules) lets us make assumptions that simplify the problem from the
> general case, but I do not offhand see how it would be any easier in
> this case.

So my proposal is to build a new first-class tool to make
manipulations in #1, #2 and #3 easily possible.  The first step is to
formalize the names "bare worktree" (which refers to a worktree with a
gitfile), "worktree" (which refers to a worktree with a GITDIR
embedded in it), and "workdir" (which refers to a worktree with a
"thin" GITDIR).

The reason I want to build it for submodules first is because the
non-submodule case (#2) is simply a reduced case of the submodule case
(#3):

- When I attempt to attach a new worktree to an existing GITDIR with a
worktree attached, I will create a workdir instead.  This simply
involves creating a thin .git directory in the worktree in the
non-submodule case.  In the submodule case, it is more complicated: I
have to locate the superproject's .git directory, and put it there.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Marc Branchaud wrote:
> If "git add" is all about specifying what lives under paths in the worktree,
> what's wrong with letting "git add" go beyond specifying just files?
>
> Syntax aside for the moment, I think a command like
>         git add git-repo-reference foo
> is perfectly natural:  It specifies what is inside worktree path foo.

I never said "just files".  Files, directories, symlinks and
submodules are all "things in the worktree", and all fine.  Remote
URLs, on the other hand, have nothing to do with the worktree.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Junio C Hamano wrote:
> It does not relieve "git add" (or "git submodulea add") from the
> responsibility of moving .git directory.  It only reduces the need
> to do so.
>
> When the user says "add" and the repository has .git directory in
> it, "add" (or "submodule add") is still responsible for relocating
> it.

Since you're so stubborn about it, I suppose 'git add' could call a
function in my "new first-class program to attach detach
worktrees/workdirs and relocate GITDIRs" as a last resort (if the user
somehow managed to put a GITDIR in the submodule worktree despite our
well-designed tools).  But last resort is not what we should be
discussing now: we're discussing what the design should ideally be.
And ideally, I think we both agree that it's best if init/clone did
the relocation.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Jonathan Nieder wrote:
> I quite like .git/modules/<subproject name> (for some reasons that
> I've mentioned in other threads) and don't consider it nonsense, which
> makes me assume I don't understand the goal of this patch, either.
> Please don't take that personally.

There's nothing to take personally, Jonathan.  We're designing
software, and the rationale for choosing a design is never "Jonathan
personally likes this particular design, so therefore we'll go with
it", but rather "Ram's design is objectively superior, and therefore
we'll go with it".  I'll proceed with bashing .git/modules, while your
job is to defend it:

1. The path to the object store of a submodule depends upon how deeply
it is nested in other submodules, and hence how many /modules/
components to add to the path to the project's name.  Presumably, this
is to avoid conflicts: but it's an overkill for such a simple job.  In
the 98% case, I never have two submodules with the same name in my
superproject; for the 2% case, I can live with the inconvenience of
naming a directory by hand, rather than putting up with this ugliness.

2. This ugliness complicates implementation of add/ rm/ mv, because
each of them will have to know about this contrived path solution.

3. The paths in the gitfiles in various submodules is horribly ugly
with tons of ../ components.  This is especially the case in deeply
nested submodules.  We can't use an absolute path, because the
superproject directory can be moved anywhere in the filesystem.

4. To relocate the object store and reuse it elsewhere is almost
impossible.  What if I want to remove the submodule, but work on it
independently from the superproject?  Re-clone?

My solution fixes all these problems, and we need
.git/modules/<name>.git (no path-to-submodule nonsense) only as a last
resort: #3 (ref: my email to Peff).

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Marc Branchaud]

On 13-04-16 04:13 AM, Ramkumar Ramachandra wrote:
> Jeff King wrote:
>> So there is some information that is per-clone (the objects, the remote
>> tips), but there is some information that is per-submodule (where our
>> local branches are, the index, the worktree). I can see why it is
>> advantageous to share the per-clone information between similar clones
>> (because it avoids disk space and network transfer). But I do not think
>> you can escape having some form of per-submodule repo, even if it is a
>> thin git-new-workdir-ish repo that points back to a parent repo for the
>> clone.
> 
> I want the flexibility to do the following:
> 
> 1. Do a "simple clone", where the clone contains the GITDIR embedded
> in the worktree.  This is the most common case, and there is no reason
> to complicate it.  I can optionally attach additional workdirs to this
> clone.  I can also optionally relocate the GITDIR at a later date, if
> I feel the need to do so.
> 
> 2. Attach a worktree to any object store without having to write a
> gitfile and set core.worktree by hand.  The limitation is that you
> can't have two submodules from two different superprojects sharing the
> same object store (since both of them are worktrees).  However, for
> the purpose of working on the submodule repository as an independent
> repository (this is a very common case for me), I can attach a new
> "workdir" to the GITDIR very easily.

Doesn't contrib/workdir/git-new-workdir do this?

		M.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Marc Branchaud]

On 13-04-16 04:17 AM, Ramkumar Ramachandra wrote:
> Marc Branchaud wrote:
>> If "git add" is all about specifying what lives under paths in the worktree,
>> what's wrong with letting "git add" go beyond specifying just files?
>>
>> Syntax aside for the moment, I think a command like
>>         git add git-repo-reference foo
>> is perfectly natural:  It specifies what is inside worktree path foo.
> 
> I never said "just files".  Files, directories, symlinks and
> submodules are all "things in the worktree", and all fine.  Remote
> URLs, on the other hand, have nothing to do with the worktree.

But they have everything to do with submodules.  You need a URL to identify a
submodule.  If you want a submodule in your worktree, at some point you have
to specify the submodule's URL.

I really feel like I'm missing something here.  You seem to be saying that
it's wrong to let "git add" interpret a URL as a submodule.  Instead you seem
to want to have some other mechanism create the files, directories and
symlinks that make up a submodule, so that "git add" can then operate with
the purity you desire.  That's what I don't understand.

As a submodule user, I want to "git add" a submodule.  I don't see why it's
necessary to have more than one command to do that.  But if you're saying
that it's fine for "git add" to work this way, then I don't see the point of
the proposed change to "git clone".

		M.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Marc Branchaud]

On 13-04-16 04:21 AM, Ramkumar Ramachandra wrote:
> Junio C Hamano wrote:
>> It does not relieve "git add" (or "git submodulea add") from the
>> responsibility of moving .git directory.  It only reduces the need
>> to do so.
>>
>> When the user says "add" and the repository has .git directory in
>> it, "add" (or "submodule add") is still responsible for relocating
>> it.
> 
> Since you're so stubborn about it, I suppose 'git add' could call a
> function in my "new first-class program to attach detach
> worktrees/workdirs and relocate GITDIRs" as a last resort (if the user
> somehow managed to put a GITDIR in the submodule worktree despite our
> well-designed tools).  But last resort is not what we should be
> discussing now: we're discussing what the design should ideally be.
> And ideally, I think we both agree that it's best if init/clone did
> the relocation.

If that's the question, then put me on the "disagree" side.  I just don't see
why that approach is "best", especially if the intention is "to make 'git
add' DTRT wrt submodules, and deprecate 'git submodule add'".

		M.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Junio C Hamano]

Ramkumar Ramachandra <artagnon@gmail.com> writes:

> My solution fixes all these problems, and we need
> .git/modules/<name>.git (no path-to-submodule nonsense) only as a last
> resort: #3 (ref: my email to Peff).

Have you noticed that there are distinction between submodule path
and submodule name already in the current system, and name is
derived from path if you do not give it when adding a submodule
merely as a convenience?

If some existing code uses .git/modules/<path>.git in "git submodule",
that is a bug that needs to be fixed.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Duy Nguyen]

On Sun, Apr 14, 2013 at 5:23 AM, Ramkumar Ramachandra
<artagnon@gmail.com> wrote:
> This configuration variable comes into effect when 'git clone' is
> invoked inside an existing git repository's worktree.  When set,
> instead of cloning the given repository as-is, it relocates the gitdir
> of the repository to the path specified by this variable.  This
> setting is especially useful when working with submodules.

What if I clone a repo then realize it was a mistake and remove it?
With current clone, a "rm -rf" would do. With this, I'll need to
figure out which subdir in the top .git contains the repo I want to
remove. I'm not sure how "git submodule" handles this case though
(i.e. total submodule ignorant speaking..)
--
Duy

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Duy Nguyen wrote:
> What if I clone a repo then realize it was a mistake and remove it?
> With current clone, a "rm -rf" would do. With this, I'll need to
> figure out which subdir in the top .git contains the repo I want to
> remove. I'm not sure how "git submodule" handles this case though
> (i.e. total submodule ignorant speaking..)

Currently, submodules relocate the GITDIR of submodules to
.git/modules.  So, my proposed patch doesn't make the situation any
worse.  In fact, it improves the situation because you're guaranteed
that all your GITDIRs will be in ~/bare (or whatever your
core.submoduleGitDir is), as opposed to a complex path in .git/modules
of your containing superproject.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Duy Nguyen]

On Wed, Apr 17, 2013 at 8:53 PM, Ramkumar Ramachandra
<artagnon@gmail.com> wrote:
> Duy Nguyen wrote:
>> What if I clone a repo then realize it was a mistake and remove it?
>> With current clone, a "rm -rf" would do. With this, I'll need to
>> figure out which subdir in the top .git contains the repo I want to
>> remove. I'm not sure how "git submodule" handles this case though
>> (i.e. total submodule ignorant speaking..)
>
> Currently, submodules relocate the GITDIR of submodules to
> .git/modules.  So, my proposed patch doesn't make the situation any
> worse.  In fact, it improves the situation because you're guaranteed
> that all your GITDIRs will be in ~/bare (or whatever your
> core.submoduleGitDir is), as opposed to a complex path in .git/modules
> of your containing superproject.

No, submodule code does not change "git clone". If I'm not mistaken,
submodule will not kick in until you type "git submodule something".
If I turn clone.submoduleGitDir on, how can I undo my mistake in a
user friendly way?
--
Duy

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Duy Nguyen wrote:
> No, submodule code does not change "git clone". If I'm not mistaken,
> submodule will not kick in until you type "git submodule something".
> If I turn clone.submoduleGitDir on, how can I undo my mistake in a
> user friendly way?

So, if you currently want to add a submodule, you have to 'git
submodule add', which runs clone internally apart from other things.
How do you undo this mistake?

What I'm essentially proposing is to give the job of cloning back to
clone, and the job of adding back to add, instead of creating an
unnatural abstraction over them using 'git submodule add'.  The point
being: why would you ever _want_ to clone inside a worktree unless you
intend to add a submodule?  In other words, you intent for running a
'git clone' inside a worktree is exactly the same as your intent for
running a 'git submodule add' inside a worktree.  Ofcourse, if you
have a fringe case where that was _not_ your intent, we'll provide a
command-line switch to turn off the relocation for that clone.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Duy Nguyen]

On Wed, Apr 17, 2013 at 9:13 PM, Ramkumar Ramachandra
<artagnon@gmail.com> wrote:
> Duy Nguyen wrote:
>> No, submodule code does not change "git clone". If I'm not mistaken,
>> submodule will not kick in until you type "git submodule something".
>> If I turn clone.submoduleGitDir on, how can I undo my mistake in a
>> user friendly way?
>
> So, if you currently want to add a submodule, you have to 'git
> submodule add', which runs clone internally apart from other things.
> How do you undo this mistake?

Well, it has "submodule" in the command line. My first reaction would
be looking for "git submodule rm" or something.

> What I'm essentially proposing is to give the job of cloning back to
> clone, and the job of adding back to add, instead of creating an
> unnatural abstraction over them using 'git submodule add'.  The point
> being: why would you ever _want_ to clone inside a worktree unless you
> intend to add a submodule?  In other words, you intent for running a
> 'git clone' inside a worktree is exactly the same as your intent for
> running a 'git submodule add' inside a worktree.  Ofcourse, if you
> have a fringe case where that was _not_ your intent, we'll provide a
> command-line switch to turn off the relocation for that clone.

No, the point is people make mistakes. What do we do in that case? Or
will you introduce yet another "gc" command for clean up ~/bare?
--
Duy

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Ramkumar Ramachandra]

Duy Nguyen wrote:
> Well, it has "submodule" in the command line. My first reaction would
> be looking for "git submodule rm" or something.

No, 'git submodule rm' cannot remove the corresponding GITDIR.  What
if there are other branches that refer to the submodule?  What if you
want to remove it from this branch and add it to another branch?

> No, the point is people make mistakes. What do we do in that case? Or
> will you introduce yet another "gc" command for clean up ~/bare?

So, people don't make mistakes when they use 'git submodule add', but
do make mistakes when using 'git clone'?  How has the problem
_changed_ with my patch?  It's reasonable to point it out as an
existing problem, and ask for it to be fixed independent of this
discussion, but that is not what you are doing.

git cannot read your mind to determine if you made a mistake, if
that's what you're asking.  No, a gc equivalent won't work either (and
there's nothing in the current submodule world), because it is
impossible to determine if a workdir is attached to that GITDIR
somewhere on your filesystem.

You'll have to do _something_ to say that you don't want that GITDIR
anymore.  It's reasonable to request tooling to help with this task,
but your request is entirely different.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Jonathan Nieder]

Ramkumar Ramachandra wrote:

> 2. This ugliness complicates implementation of add/ rm/ mv, because
> each of them will have to know about this contrived path solution.

Why is that?  Can't they look at the gitfile or call some helper
(that happens to be part of the same binary)?

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

> No, the point is people make mistakes. What do we do in that case? Or
> will you introduce yet another "gc" command for clean up ~/bare?

I do not know if it will be a "gc", but we would need a way for the
user to say "I no longer need the repository for this submodule kept
locally here (I may have to re-clone when I check out a version that
needs the submodule)" to free up the .git/modules/<name> directories
in the superproject.  We might want to allow "submodule deinit" to
also ask for it, but "deinit" will not be the only occasion the user
might want it.

It is already a problem that needs to be addressed in the current setup.

Re: [RFC/PATCH] clone: introduce clone.submoduleGitDir to relocate $GITDIR

[Duy Nguyen]

On Thu, Apr 18, 2013 at 1:02 AM, Ramkumar Ramachandra
<artagnon@gmail.com> wrote:
>> No, the point is people make mistakes. What do we do in that case? Or
>> will you introduce yet another "gc" command for clean up ~/bare?
>
> So, people don't make mistakes when they use 'git submodule add', but
> do make mistakes when using 'git clone'?  How has the problem
> _changed_ with my patch?  It's reasonable to point it out as an
> existing problem, and ask for it to be fixed independent of this
> discussion, but that is not what you are doing.

It's the magic in git-clone that changes its behavior that I want to
address. I know you agree to go with a command line option. But I
think in the end there will be a switch hidden somewhere in config to
make things smooth, unless you make this mode the default (*). With
normal mode, "rm -rf repo" is enough, with the new submodule mode, it
leaves some garbage behind that the user may not be aware about. Maybe
this is something that should be addressed anyway even for .gitmodules
mode like Junio said. But I wonder if there are any other traps that
come with the config switch.

(*) I don't think you can make the new mode the default though. There
are repos in repos in the field that are not managed by "git
submodule". Switching the default will disrupt those setups. Some
deprecation cycles might help, I don't know.
--
Duy