Add repos but not as normal files, not submodule

Add repos but not as normal files, not submodule

[Antony Stubbs]

Hi there. 

I have a special situation (maven git scm provider) where I have sub directories with git repps in them which I use as data to run tests on. 

Is it possible to track these "internal" repos without using submodules? 

The only alternative I see is to compress all the test repos in an archive and track that - but that's not nice.

Upon trying to "Add" the internal repos to track, I get error: Invalid path
for example:
antony-stubbss-macbook-pro-4:buildnumber-maven-plugin antonystubbs$ git add src/test/test-repositories/git/branch-tests-japanese/.git/config
error: Invalid path 'src/test/test-repositories/git/branch-tests-japanese/.git/config'
error: unable to add src/test/test-repositories/git/branch-tests-japanese/.git/config to index
fatal: adding files failed

The reason I don't want to do submodules is that I don't want to have the submodules as serperate projects as such, because they're not. they're "test data" so to speak.

Cheers,
Antony.

Re: Add repos but not as normal files, not submodule

[Jonathan Nieder]

Hi,

Antony Stubbs wrote:

> I have a special situation (maven git scm provider) where I have sub
> directories with git repps in them which I use as data to run tests
> on. 
> 
> Is it possible to track these "internal" repos without using submodules? 

First I should say that whatever I say below is about git of the
present, not git of the future.  In other words, if you have a patch
that would change this without breaking things, go for it.

Okay, so why does git not allow adding files or directories named
.git?

The first hint is in dir.c:

| /*
|  * Read a directory tree. We currently ignore anything but
|  * directories, regular files and symlinks. That's because git
|  * doesn't handle them at all yet. Maybe that will change some
|  * day.
|  *
|  * Also, we ignore the name ".git" (even if it is not a directory).
|  * That likely will not change.
|  */
| static int read_directory_recursive(struct dir_struct *dir,

Odd, yes?

| $ git log -S'Also, we ignore the name ".git" (even if' --follow -- dir.c
| commit 453ec4bdf403c2e89892266a0a660c21680d3f9d
| Author: Linus Torvalds <torvalds@osdl.org>
| Date:   Tue May 16 19:02:14 2006 -0700
| 
|     libify git-ls-files directory traversal
| 
|     This moves the core directory traversal and filename exclusion logic into
|     the general git library, making it available for other users directly.
[...]
| $ git name-rev --tags 453ec4bdf403c2e89892266a0a660c21680d3f9d
| 453ec4bdf403c2e89892266a0a660c21680d3f9d tags/v1.4.0-rc1~79^2~8
| $ git diff v1.4.0-rc1~80..v1.4.0-rc1~79
[...]
| --- a/update-index.c
| +++ b/update-index.c
| @@ -120,70 +120,6 @@ static int add_file_to_cache(const char *path)
|  	return 0;
|  }
|  
| -/*
| - * We fundamentally don't like some paths: we don't want
| - * dot or dot-dot anywhere, and for obvious reasons don't
| - * want to recurse into ".git" either.
| - *

So much for origins.  Starting out, it was easier to ignore all files
named .git than just the .git directory.

Of course, a lot has changed since then.  For example, consider the
following:

 ; git init foo
 ; cd foo
 ; mkdir bar
 ; >bar/.git
 ; git add .
 fatal: Invalid gitfile format: bar/.git

Adding an abort() to die_routine() and grabbing a back trace from the
core dump reveals that this comes from

| $ git log -S'if (resolve_gitlink_ref(dirname, "HEAD", sha1) == 0)' -- dir.c
| commit 095952585c2a955f45deac69df17a702d7584c80
| Author: Linus Torvalds <torvalds@linux-foundation.org>
| Date:   Wed Apr 11 14:49:44 2007 -0700
|
|     Teach directory traversal about subprojects
|
|     This is the promised cleaned-up version of teaching directory traversal
|     (ie the "read_directory()" logic) about subprojects. That makes "git add" 
|     understand to add/update subprojects.
[...]
| $ git name-rev --tags 095952585c2a955f45deac69df17a702d7584c80
| 095952585c2a955f45deac69df17a702d7584c80 tags/v1.5.2-rc0~19^2~9

In other words, if bar is itself a git repository with a valid HEAD,
then ‘git add bar’ will register it in the index as a submodule.

I should also mention the problem of finding the right git directory:
if foo and foo/bar are git trees and you try the following from the
foo directory, what should happen?

 ; git add bar/something

How about the following?

 ; (cd bar && git add something)

The answers are less ambiguous with submodules.

> The only alternative I see is to compress all the test repos in an
> archive and track that - but that's not nice.

If the concern is making it pack nicely, you can use an uncompressed
tar file for each test (or shar, or whatever your favorite
uncompressed format is).  Still ugly, but it should work okay.

If you can rely on build systems to have git installed, I’d suggest
writing a simple script to build the test repository from scratch.

You can also try using repositories not named .git if you are careful.
I don’t really encourage this.

> The reason I don't want to do submodules is that I don't want to
> have the submodules as serperate projects as such, because they're
> not. they're "test data" so to speak.

Yes, makes sense to me.

Hope that helps,
Jonathan

Re: Add repos but not as normal files, not submodule

[Antony Stubbs]

> In other words, if bar is itself a git repository with a valid HEAD,
> then ‘git add bar’ will register it in the index as a submodule.

That was the behavior I was seeing...

> If you can rely on build systems to have git installed, I’d suggest
> writing a simple script to build the test repository from scratch.

Yup - will probably go down this route.

> You can also try using repositories not named .git if you are careful.
> I don’t really encourage this.

Didn't realize this was possible. Any pointers? Links?

>> The reason I don't want to do submodules is that I don't want to
>> have the submodules as serperate projects as such, because they're
>> not. they're "test data" so to speak.
> 
> Yes, makes sense to me.
> 
> Hope that helps,
> Jonathan


Cheers for the deep dig! I started looking around the C code, but was following the wrong trail (verify_path) - I like your demo - very nice.

I suppose this problem only stems from them wanting git add to "nicely" automatically recognize .git sub dirs as submodules, thinking no one would ever in their right mind actually want to *track* a sub .git dir ;)

Cheers!

Re: Add repos but not as normal files, not submodule

[Jonathan Nieder]

Antony Stubbs wrote:

>> I’d suggest
>> writing a simple script to build the test repository from scratch.
>
> Yup - will probably go down this route.

Ah, good.

>> You can also try using repositories not named .git if you are careful.
>> I don’t really encourage this.
>
> Didn't realize this was possible.

 ; git init foo
 ; git init --bare foo/bar.git
 ; cd foo
 ; git add bar.git

or

 ; git init foo
 ; git init foo/bar
 ; mv foo/bar/.git foo/bar.git
 ; git add bar bar.git
 ; cd bar
 ; GIT_DIR=$(pwd)/../bar.git git status
 ; (ln -s ../bar.git .git && git status && rm .git)

Please, don’t do this.  For all I know, some git command will end
up chdir()ing into bar.git and getting confused.

> Any pointers? Links?

I’m not aware of the relevant rules being documented anywhere.  How
much of the current behavior do we want to commit to continuing to
support?

Here’s a (very rough) start.  Thoughts?

-- 8< --
Subject: Documentation/update-index: indicate what the index can contain

Based on cache.h.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 Documentation/git-update-index.txt |   37 ++++++++++++++++++++++++++++++++++++
 1 files changed, 37 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt
index 68dc187..3f74938 100644
--- a/Documentation/git-update-index.txt
+++ b/Documentation/git-update-index.txt
@@ -29,6 +29,12 @@ Modifies the index or directory cache. Each file mentioned is updated
 into the index and any 'unmerged' or 'needs updating' state is
 cleared.
 
+Paths that do not represent directories, regular files, or symlinks
+are ignored (though other file types may be supported in future
+versions of git).  Files and directories named .git are ignored except
+for the sake of detecting subrepositories.  See the "Content model"
+section below for more about what the index can track.
+
 See also linkgit:git-add[1] for a more user-friendly way to do some of
 the most common operations on the index.
 
@@ -159,6 +165,37 @@ you will need to handle the situation manually.
 	cleaner names.
 	The same applies to directories ending '/' and paths with '//'
 
+Content model
+-------------
+Roughly speaking, the index represents the state of the working tree
+at some moment.  Each entry indicates the name of a file, its mode,
+the name of a blob or tree object representing its content (0 for a
+removed file or directory), a stage number, flags, and some lstat(2)
+information that can be used to check if the entry is up-to-date.
+
+Filenames are relative to the top level of the working tree and
+contain no empty path components or path components named ., ..,
+or .git, or symbolic link path components.
+
+Modes are based on the octal values as returned by lstat(2):
+
+. `100644` (regular file)
+. `100755` (executable)
+. `120000` (symbolic link)
+. `040000` (directory)
+. `160000` (git submodule)
+
+All other file types are either reduced to one of these five or
+ignored.
+
+See linkgit:git-read-tree(1) for a description of the stages in
+the index.  When a merge is not in progress, all entries have stage
+number 0.
+
+There are currently three boolean flags that can be set.  See
+the descriptions of the `--asumed-unchanged` and `--skip-worktree`
+options above and the `--intent-to-add` option to linkgit:git-add[1].
+
 Using --refresh
 ---------------
 '--refresh' does not calculate a new sha1 file or bring the index
-- 
1.7.1.rc1