[RFC PATCH v4 00/19] Sparse checkout

[RFC PATCH v4 00/19] Sparse checkout

[Nguyễn Thái Ngọc Duy]

Welcome to the fourth round of sparse checkout this year, dubbed "The
mummy^W^W^W^W^Wno-checkout returns", although the bit now comes under
a new name "skip-worktree" [1]. This series has two parts: sparse
worktree and sparse checkout. Details will be given below.

For brave users of this series: I have pushed it to my sparse-checkout
branch [2] so you can just clone it and test away. Visible changes:

 - the spec is now .git/info/sparse-checkout
 - the spec is positive matching, that is it matches what _is_ in
   worktree, not what is out of worktree like the last series
 - you need to set core.sparsecheckout no additional command argument
   is needed.

Except from running tests, I have only used it a little bit, so be
careful, it may burn your trees.


Nguyễn Thái Ngọc Duy (8):
  update-index: refactor mark_valid() in preparation for new options
  Add test-index-version
  Introduce "skip-worktree" bit in index, teach Git to get/set this bit
  Teach Git to respect skip-worktree bit (reading part)
  Teach Git to respect skip-worktree bit (writing part)
  Avoid writing to buffer in add_excludes_from_file_1()
  Read .gitignore from index if it is skip-worktree
  unpack-trees(): carry skip-worktree bit over in merged_entry()

The three main patches in this part are 3, 4 and 5, which define how
"skip-worktree" should work (and I'd love to have feedback whether I
get it right) and implement it.

.gitignore patches are from last series and do not change much.


Nguyễn Thái Ngọc Duy (11):
  excluded_1(): support exclude files in index
  dir.c: export excluded_1() and add_excludes_from_file_1()
  Introduce "sparse checkout"
  unpack-trees(): add CE_WT_REMOVE to remove on worktree alone
  unpack-trees.c: generalize verify_* functions
  unpack-trees(): "enable" sparse checkout and load
    $GIT_DIR/info/sparse-checkout
  unpack_trees(): apply $GIT_DIR/info/sparse-checkout to the final
    index
  unpack-trees(): ignore worktree check outside checkout area
  read-tree: add --no-sparse-checkout to disable sparse checkout
    support
  Add tests for sparse checkout
  sparse checkout: inhibit empty worktree

The recent assume-unchanged "breakage" that lets Git overwrite
assume-unchanged files worried me. I sat back, checked and wrote tests
to catch similar situations. As a result, core sparse checkout code
becomes more complicated and is splitted into smaller parts for easier
read.

Interesting patches are 11/19 (OK not that interesting, it's just
boring text), 14/19-16/19. Patch 18/19 also covers "foo" vs "foo/"
matching bug in tests.


[1] There are a few reasons behind this rename:
 - there is nothing about "checkout" in the first part, it's about
   "sparse worktree"
 - the double negation issue with "no-" part of "no-checkout"
 - new name makes cache.h diff align beautifully

[2] http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/sparse-checkout

 .gitignore                                        |    1 +
 Documentation/config.txt                          |    4 +
 Documentation/git-ls-files.txt                    |    1 +
 Documentation/git-read-tree.txt                   |   52 ++++++-
 Documentation/git-update-index.txt                |   29 ++++
 Documentation/technical/api-directory-listing.txt |    3 +
 Makefile                                          |    1 +
 builtin-clean.c                                   |    4 +-
 builtin-commit.c                                  |    5 +
 builtin-grep.c                                    |    2 +-
 builtin-ls-files.c                                |   11 +-
 builtin-read-tree.c                               |    4 +-
 builtin-update-index.c                            |   78 ++++++----
 cache.h                                           |    8 +-
 config.c                                          |    5 +
 diff-lib.c                                        |    5 +-
 diff.c                                            |    2 +-
 dir.c                                             |  100 ++++++++----
 dir.h                                             |    4 +
 environment.c                                     |    1 +
 read-cache.c                                      |    8 +-
 t/t1009-read-tree-sparse-checkout.sh              |  150 ++++++++++++++++++
 t/t2104-update-index-skip-worktree.sh             |   57 +++++++
 t/t3001-ls-files-others-exclude.sh                |   22 +++
 t/t7011-skip-worktree-reading.sh                  |  163 +++++++++++++++++++
 t/t7012-skip-worktree-writing.sh                  |  146 +++++++++++++++++
 t/t7300-clean.sh                                  |   19 +++
 test-index-version.c                              |   14 ++
 unpack-trees.c                                    |  175 +++++++++++++++++++--
 unpack-trees.h                                    |    6 +
 30 files changed, 983 insertions(+), 97 deletions(-)
 create mode 100755 t/t1009-read-tree-sparse-checkout.sh
 create mode 100755 t/t2104-update-index-skip-worktree.sh
 create mode 100755 t/t7011-skip-worktree-reading.sh
 create mode 100755 t/t7012-skip-worktree-writing.sh
 create mode 100644 test-index-version.c

[RFC PATCH v4 01/19] update-index: refactor mark_valid() in preparation for new options

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin-update-index.c |   24 ++++++++++--------------
 1 files changed, 10 insertions(+), 14 deletions(-)

diff --git a/builtin-update-index.c b/builtin-update-index.c
index 92beaaf..f1b6c8e 100644
--- a/builtin-update-index.c
+++ b/builtin-update-index.c
@@ -24,8 +24,8 @@ static int info_only;
 static int force_remove;
 static int verbose;
 static int mark_valid_only;
-#define MARK_VALID 1
-#define UNMARK_VALID 2
+#define MARK_FLAG 1
+#define UNMARK_FLAG 2
 
 static void report(const char *fmt, ...)
 {
@@ -40,19 +40,15 @@ static void report(const char *fmt, ...)
 	va_end(vp);
 }
 
-static int mark_valid(const char *path)
+static int mark_ce_flags(const char *path, int flag, int mark)
 {
 	int namelen = strlen(path);
 	int pos = cache_name_pos(path, namelen);
 	if (0 <= pos) {
-		switch (mark_valid_only) {
-		case MARK_VALID:
-			active_cache[pos]->ce_flags |= CE_VALID;
-			break;
-		case UNMARK_VALID:
-			active_cache[pos]->ce_flags &= ~CE_VALID;
-			break;
-		}
+		if (mark)
+			active_cache[pos]->ce_flags |= flag;
+		else
+			active_cache[pos]->ce_flags &= ~flag;
 		cache_tree_invalidate_path(active_cache_tree, path);
 		active_cache_changed = 1;
 		return 0;
@@ -276,7 +272,7 @@ static void update_one(const char *path, const char *prefix, int prefix_length)
 		goto free_return;
 	}
 	if (mark_valid_only) {
-		if (mark_valid(p))
+		if (mark_ce_flags(p, CE_VALID, mark_valid_only == MARK_FLAG))
 			die("Unable to mark file %s", path);
 		goto free_return;
 	}
@@ -647,11 +643,11 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 				continue;
 			}
 			if (!strcmp(path, "--assume-unchanged")) {
-				mark_valid_only = MARK_VALID;
+				mark_valid_only = MARK_FLAG;
 				continue;
 			}
 			if (!strcmp(path, "--no-assume-unchanged")) {
-				mark_valid_only = UNMARK_VALID;
+				mark_valid_only = UNMARK_FLAG;
 				continue;
 			}
 			if (!strcmp(path, "--info-only")) {
-- 
1.6.3.GIT

[RFC PATCH v4 02/19] Add test-index-version

[Nguyễn Thái Ngọc Duy]

Commit 06aaaa0bf70fe37d198893f4e25fa73b6516f8a9 may step index format
version up and down, depends on whether extended flags present in the
index. This adds a test to check for index format version.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 .gitignore           |    1 +
 Makefile             |    1 +
 test-index-version.c |   14 ++++++++++++++
 3 files changed, 16 insertions(+), 0 deletions(-)
 create mode 100644 test-index-version.c

diff --git a/.gitignore b/.gitignore
index c446290..01753f8 100644
--- a/.gitignore
+++ b/.gitignore
@@ -157,6 +157,7 @@ test-date
 test-delta
 test-dump-cache-tree
 test-genrandom
+test-index-version
 test-match-trees
 test-parse-options
 test-path-utils
diff --git a/Makefile b/Makefile
index 4190a5d..f49435a 100644
--- a/Makefile
+++ b/Makefile
@@ -1586,6 +1586,7 @@ TEST_PROGRAMS += test-parse-options$X
 TEST_PROGRAMS += test-path-utils$X
 TEST_PROGRAMS += test-sha1$X
 TEST_PROGRAMS += test-sigchain$X
+TEST_PROGRAMS += test-index-version$X
 
 all:: $(TEST_PROGRAMS)
 
diff --git a/test-index-version.c b/test-index-version.c
new file mode 100644
index 0000000..bfaad9e
--- /dev/null
+++ b/test-index-version.c
@@ -0,0 +1,14 @@
+#include "cache.h"
+
+int main(int argc, const char **argv)
+{
+	struct cache_header hdr;
+	int version;
+
+	memset(&hdr,0,sizeof(hdr));
+	if (read(0, &hdr, sizeof(hdr)) != sizeof(hdr))
+		return 0;
+	version = ntohl(hdr.hdr_version);
+	printf("%d\n", version);
+	return 0;
+}
-- 
1.6.3.GIT

[RFC PATCH v4 03/19] Introduce "skip-worktree" bit in index, teach Git to get/set this bit

[Nguyễn Thái Ngọc Duy]

Detail about this bit is in Documentation/git-update-index.txt.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-ls-files.txt        |    1 +
 Documentation/git-update-index.txt    |   29 +++++++++++++++++
 builtin-ls-files.c                    |    5 ++-
 builtin-update-index.c                |   16 +++++++++-
 cache.h                               |    4 ++-
 t/t2104-update-index-skip-worktree.sh |   57 +++++++++++++++++++++++++++++++++
 6 files changed, 109 insertions(+), 3 deletions(-)
 create mode 100755 t/t2104-update-index-skip-worktree.sh

diff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt
index 021066e..6f9d880 100644
--- a/Documentation/git-ls-files.txt
+++ b/Documentation/git-ls-files.txt
@@ -107,6 +107,7 @@ OPTIONS
 	Identify the file status with the following tags (followed by
 	a space) at the start of each line:
 	H::	cached
+	S::	skip-worktree
 	M::	unmerged
 	R::	removed/deleted
 	C::	modified/changed
diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt
index 25e0bbe..a10f355 100644
--- a/Documentation/git-update-index.txt
+++ b/Documentation/git-update-index.txt
@@ -15,6 +15,7 @@ SYNOPSIS
 	     [--cacheinfo <mode> <object> <file>]\*
 	     [--chmod=(+|-)x]
 	     [--assume-unchanged | --no-assume-unchanged]
+	     [--skip-worktree | --no-skip-worktree]
 	     [--ignore-submodules]
 	     [--really-refresh] [--unresolve] [--again | -g]
 	     [--info-only] [--index-info]
@@ -99,6 +100,13 @@ in the index e.g. when merging in a commit;
 thus, in case the assumed-untracked file is changed upstream,
 you will need to handle the situation manually.
 
+--skip-worktree::
+--no-skip-worktree::
+	When one of these flags is specified, the object name recorded
+	for the paths are not updated. Instead, these options
+	set and unset the "skip-worktree" bit for the paths. See
+	section "Skip-worktree bit" below for more information.
+
 -g::
 --again::
 	Runs 'git-update-index' itself on the paths whose index
@@ -304,6 +312,27 @@ M foo.c
 <9> now it checks with lstat(2) and finds it has been changed.
 
 
+Skip-worktree bit
+-----------------
+
+Skip-worktree bit can be defined in one (long) sentence: When reading
+an entry, if it is marked as skip-worktree, then Git pretends its
+working directory version is up to date and read the index version
+instead.
+
+To elaborate, "reading" means checking for file existence, reading
+file attributes or file content. The working directory version may be
+present or absent. If present, its content may match against the index
+version or not. Writing is not affected by this bit, content safety
+is still first priority. Note that Git _can_ update working directory
+file, that is marked skip-worktree, if it is safe to do so (i.e.
+working directory version matches index version)
+
+Although this bit looks similar to assume-unchanged bit, its goal is
+different from assume-unchanged bit's. Skip-worktree also takes
+precedence over assume-unchanged bit when both are set.
+
+
 Configuration
 -------------
 
diff --git a/builtin-ls-files.c b/builtin-ls-files.c
index f473220..c1afbad 100644
--- a/builtin-ls-files.c
+++ b/builtin-ls-files.c
@@ -37,6 +37,7 @@ static const char *tag_removed = "";
 static const char *tag_other = "";
 static const char *tag_killed = "";
 static const char *tag_modified = "";
+static const char *tag_skip_worktree = "";
 
 static void show_dir_entry(const char *tag, struct dir_entry *ent)
 {
@@ -178,7 +179,8 @@ static void show_files(struct dir_struct *dir, const char *prefix)
 				continue;
 			if (ce->ce_flags & CE_UPDATE)
 				continue;
-			show_ce_entry(ce_stage(ce) ? tag_unmerged : tag_cached, ce);
+			show_ce_entry(ce_stage(ce) ? tag_unmerged :
+				(ce_skip_worktree(ce) ? tag_skip_worktree : tag_cached), ce);
 		}
 	}
 	if (show_deleted | show_modified) {
@@ -490,6 +492,7 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 		tag_modified = "C ";
 		tag_other = "? ";
 		tag_killed = "K ";
+		tag_skip_worktree = "S ";
 	}
 	if (show_modified || show_others || show_deleted || (dir.flags & DIR_SHOW_IGNORED) || show_killed)
 		require_work_tree = 1;
diff --git a/builtin-update-index.c b/builtin-update-index.c
index f1b6c8e..5e97d09 100644
--- a/builtin-update-index.c
+++ b/builtin-update-index.c
@@ -24,6 +24,7 @@ static int info_only;
 static int force_remove;
 static int verbose;
 static int mark_valid_only;
+static int mark_skip_worktree_only;
 #define MARK_FLAG 1
 #define UNMARK_FLAG 2
 
@@ -276,6 +277,11 @@ static void update_one(const char *path, const char *prefix, int prefix_length)
 			die("Unable to mark file %s", path);
 		goto free_return;
 	}
+	if (mark_skip_worktree_only) {
+		if (mark_ce_flags(p, CE_SKIP_WORKTREE, mark_skip_worktree_only == MARK_FLAG))
+			die("Unable to mark file %s", path);
+		goto free_return;
+	}
 
 	if (force_remove) {
 		if (remove_file_from_cache(p))
@@ -384,7 +390,7 @@ static void read_index_info(int line_termination)
 }
 
 static const char update_index_usage[] =
-"git update-index [-q] [--add] [--replace] [--remove] [--unmerged] [--refresh] [--really-refresh] [--cacheinfo] [--chmod=(+|-)x] [--assume-unchanged] [--info-only] [--force-remove] [--stdin] [--index-info] [--unresolve] [--again | -g] [--ignore-missing] [-z] [--verbose] [--] <file>...";
+"git update-index [-q] [--add] [--replace] [--remove] [--unmerged] [--refresh] [--really-refresh] [--cacheinfo] [--chmod=(+|-)x] [--assume-unchanged] [--skip-worktree|--no-skip-worktree] [--info-only] [--force-remove] [--stdin] [--index-info] [--unresolve] [--again | -g] [--ignore-missing] [-z] [--verbose] [--] <file>...";
 
 static unsigned char head_sha1[20];
 static unsigned char merge_head_sha1[20];
@@ -650,6 +656,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 				mark_valid_only = UNMARK_FLAG;
 				continue;
 			}
+			if (!strcmp(path, "--no-skip-worktree")) {
+				mark_skip_worktree_only = UNMARK_FLAG;
+				continue;
+			}
+			if (!strcmp(path, "--skip-worktree")) {
+				mark_skip_worktree_only = MARK_FLAG;
+				continue;
+			}
 			if (!strcmp(path, "--info-only")) {
 				info_only = 1;
 				continue;
diff --git a/cache.h b/cache.h
index f793789..3b0e5fc 100644
--- a/cache.h
+++ b/cache.h
@@ -181,10 +181,11 @@ struct cache_entry {
  * Extended on-disk flags
  */
 #define CE_INTENT_TO_ADD 0x20000000
+#define CE_SKIP_WORKTREE 0x40000000
 /* CE_EXTENDED2 is for future extension */
 #define CE_EXTENDED2 0x80000000
 
-#define CE_EXTENDED_FLAGS (CE_INTENT_TO_ADD)
+#define CE_EXTENDED_FLAGS (CE_INTENT_TO_ADD | CE_SKIP_WORKTREE)
 
 /*
  * Safeguard to avoid saving wrong flags:
@@ -233,6 +234,7 @@ static inline size_t ce_namelen(const struct cache_entry *ce)
 			    ondisk_cache_entry_size(ce_namelen(ce)))
 #define ce_stage(ce) ((CE_STAGEMASK & (ce)->ce_flags) >> CE_STAGESHIFT)
 #define ce_uptodate(ce) ((ce)->ce_flags & CE_UPTODATE)
+#define ce_skip_worktree(ce) ((ce)->ce_flags & CE_SKIP_WORKTREE)
 #define ce_mark_uptodate(ce) ((ce)->ce_flags |= CE_UPTODATE)
 
 #define ce_permissions(mode) (((mode) & 0100) ? 0755 : 0644)
diff --git a/t/t2104-update-index-skip-worktree.sh b/t/t2104-update-index-skip-worktree.sh
new file mode 100755
index 0000000..1d0879b
--- /dev/null
+++ b/t/t2104-update-index-skip-worktree.sh
@@ -0,0 +1,57 @@
+#!/bin/sh
+#
+# Copyright (c) 2008 Nguyễn Thái Ngọc Duy
+#
+
+test_description='skip-worktree bit test'
+
+. ./test-lib.sh
+
+cat >expect.full <<EOF
+H 1
+H 2
+H sub/1
+H sub/2
+EOF
+
+cat >expect.skip <<EOF
+S 1
+H 2
+S sub/1
+H sub/2
+EOF
+
+test_expect_success 'setup' '
+	mkdir sub &&
+	touch ./1 ./2 sub/1 sub/2 &&
+	git add 1 2 sub/1 sub/2 &&
+	git ls-files -t | test_cmp expect.full -
+'
+
+test_expect_success 'index is at version 2' '
+	test "$(test-index-version < .git/index)" = 2
+'
+
+test_expect_success 'update-index --skip-worktree' '
+	git update-index --skip-worktree 1 sub/1 &&
+	git ls-files -t | test_cmp expect.skip -
+'
+
+test_expect_success 'index is at version 3 after having some skip-worktree entries' '
+	test "$(test-index-version < .git/index)" = 3
+'
+
+test_expect_success 'ls-files -t' '
+	git ls-files -t | test_cmp expect.skip -
+'
+
+test_expect_success 'update-index --no-skip-worktree' '
+	git update-index --no-skip-worktree 1 sub/1 &&
+	git ls-files -t | test_cmp expect.full -
+'
+
+test_expect_success 'index version is back to 2 when there is no skip-worktree entry' '
+	test "$(test-index-version < .git/index)" = 2
+'
+
+test_done
-- 
1.6.3.GIT

[RFC PATCH v4 04/19] Teach Git to respect skip-worktree bit (reading part)

[Nguyễn Thái Ngọc Duy]

grep: turn on --cached for files that is marked skip-worktree
ls-files: do not check for deleted file that is marked skip-worktree
update-index: ignore update request if it's skip-worktree, while still allows removing
diff*: skip worktree version

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin-commit.c                 |    5 +
 builtin-grep.c                   |    2 +-
 builtin-ls-files.c               |    2 +
 builtin-update-index.c           |   38 +++++----
 diff-lib.c                       |    5 +-
 diff.c                           |    2 +-
 read-cache.c                     |    8 +--
 t/t7011-skip-worktree-reading.sh |  163 ++++++++++++++++++++++++++++++++++++++
 8 files changed, 199 insertions(+), 26 deletions(-)
 create mode 100755 t/t7011-skip-worktree-reading.sh

diff --git a/builtin-commit.c b/builtin-commit.c
index 200ffda..cb64cde 100644
--- a/builtin-commit.c
+++ b/builtin-commit.c
@@ -181,6 +181,11 @@ static void add_remove_files(struct string_list *list)
 	for (i = 0; i < list->nr; i++) {
 		struct stat st;
 		struct string_list_item *p = &(list->items[i]);
+		int pos = index_name_pos(&the_index, p->string, strlen(p->string));
+		struct cache_entry *ce = pos < 0 ? NULL : active_cache[pos];
+
+		if (ce && ce_skip_worktree(ce))
+			continue;
 
 		if (!lstat(p->string, &st)) {
 			if (add_to_cache(p->string, &st, 0))
diff --git a/builtin-grep.c b/builtin-grep.c
index ad0e0a5..813fe97 100644
--- a/builtin-grep.c
+++ b/builtin-grep.c
@@ -517,7 +517,7 @@ static int grep_cache(struct grep_opt *opt, const char **paths, int cached,
 		 * are identical, even if worktree file has been modified, so use
 		 * cache version instead
 		 */
-		if (cached || (ce->ce_flags & CE_VALID)) {
+		if (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {
 			if (ce_stage(ce))
 				continue;
 			hit |= grep_sha1(opt, ce->sha1, ce->name, 0);
diff --git a/builtin-ls-files.c b/builtin-ls-files.c
index c1afbad..ad7e447 100644
--- a/builtin-ls-files.c
+++ b/builtin-ls-files.c
@@ -194,6 +194,8 @@ static void show_files(struct dir_struct *dir, const char *prefix)
 				continue;
 			if (ce->ce_flags & CE_UPDATE)
 				continue;
+			if (ce_skip_worktree(ce))
+				continue;
 			err = lstat(ce->name, &st);
 			if (show_deleted && err)
 				show_ce_entry(tag_removed, ce);
diff --git a/builtin-update-index.c b/builtin-update-index.c
index 5e97d09..97b9ea6 100644
--- a/builtin-update-index.c
+++ b/builtin-update-index.c
@@ -172,29 +172,29 @@ static int process_directory(const char *path, int len, struct stat *st)
 	return error("%s: is a directory - add files inside instead", path);
 }
 
-/*
- * Process a regular file
- */
-static int process_file(const char *path, int len, struct stat *st)
-{
-	int pos = cache_name_pos(path, len);
-	struct cache_entry *ce = pos < 0 ? NULL : active_cache[pos];
-
-	if (ce && S_ISGITLINK(ce->ce_mode))
-		return error("%s is already a gitlink, not replacing", path);
-
-	return add_one_path(ce, path, len, st);
-}
-
 static int process_path(const char *path)
 {
-	int len;
+	int pos, len;
 	struct stat st;
+	struct cache_entry *ce;
 
 	len = strlen(path);
 	if (has_symlink_leading_path(path, len))
 		return error("'%s' is beyond a symbolic link", path);
 
+	pos = cache_name_pos(path, len);
+	ce = pos < 0 ? NULL : active_cache[pos];
+	if (ce && ce_skip_worktree(ce)) {
+		/*
+		 * working directory version is assumed "good"
+		 * so updating it does not make sense.
+		 * On the other hand, removing it from index should work
+		 */
+		if (allow_remove && remove_file_from_cache(path))
+			return error("%s: cannot remove from the index", path);
+		return 0;
+	}
+
 	/*
 	 * First things first: get the stat information, to decide
 	 * what to do about the pathname!
@@ -205,7 +205,13 @@ static int process_path(const char *path)
 	if (S_ISDIR(st.st_mode))
 		return process_directory(path, len, &st);
 
-	return process_file(path, len, &st);
+	/*
+	 * Process a regular file
+	 */
+	if (ce && S_ISGITLINK(ce->ce_mode))
+		return error("%s is already a gitlink, not replacing", path);
+
+	return add_one_path(ce, path, len, &st);
 }
 
 static int add_cacheinfo(unsigned int mode, const unsigned char *sha1,
diff --git a/diff-lib.c b/diff-lib.c
index e7e8e88..fc98678 100644
--- a/diff-lib.c
+++ b/diff-lib.c
@@ -159,7 +159,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)
 				continue;
 		}
 
-		if (ce_uptodate(ce))
+		if (ce_uptodate(ce) || ce_skip_worktree(ce))
 			continue;
 
 		changed = check_removed(ce, &st);
@@ -321,6 +321,8 @@ static void do_oneway_diff(struct unpack_trees_options *o,
 	struct rev_info *revs = o->unpack_data;
 	int match_missing, cached;
 
+	/* if the entry is not checked out, don't examine work tree */
+	cached = o->index_only || (idx && ce_skip_worktree(idx));
 	/*
 	 * Backward compatibility wart - "diff-index -m" does
 	 * not mean "do not ignore merges", but "match_missing".
@@ -328,7 +330,6 @@ static void do_oneway_diff(struct unpack_trees_options *o,
 	 * But with the revision flag parsing, that's found in
 	 * "!revs->ignore_merges".
 	 */
-	cached = o->index_only;
 	match_missing = !revs->ignore_merges;
 
 	if (cached && idx && ce_stage(idx)) {
diff --git a/diff.c b/diff.c
index 91d6ea2..215e6b3 100644
--- a/diff.c
+++ b/diff.c
@@ -1805,7 +1805,7 @@ static int reuse_worktree_file(const char *name, const unsigned char *sha1, int
 	 * If ce is marked as "assume unchanged", there is no
 	 * guarantee that work tree matches what we are looking for.
 	 */
-	if (ce->ce_flags & CE_VALID)
+	if ((ce->ce_flags & CE_VALID) || ce_skip_worktree(ce))
 		return 0;
 
 	/*
diff --git a/read-cache.c b/read-cache.c
index 4e3e272..5ee7d9d 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -265,7 +265,7 @@ int ie_match_stat(const struct index_state *istate,
 	 * If it's marked as always valid in the index, it's
 	 * valid whatever the checked-out copy says.
 	 */
-	if (!ignore_valid && (ce->ce_flags & CE_VALID))
+	if (!ignore_valid && ((ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)))
 		return 0;
 
 	/*
@@ -1004,11 +1004,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,
 	if (ce_uptodate(ce))
 		return ce;
 
-	/*
-	 * CE_VALID means the user promised us that the change to
-	 * the work tree does not matter and told us not to worry.
-	 */
-	if (!ignore_valid && (ce->ce_flags & CE_VALID)) {
+	if (!ignore_valid && ((ce->ce_flags & CE_VALID) || ce_skip_worktree(ce))) {
 		ce_mark_uptodate(ce);
 		return ce;
 	}
diff --git a/t/t7011-skip-worktree-reading.sh b/t/t7011-skip-worktree-reading.sh
new file mode 100755
index 0000000..e996928
--- /dev/null
+++ b/t/t7011-skip-worktree-reading.sh
@@ -0,0 +1,163 @@
+#!/bin/sh
+#
+# Copyright (c) 2008 Nguyễn Thái Ngọc Duy
+#
+
+test_description='skip-worktree bit test'
+
+. ./test-lib.sh
+
+cat >expect.full <<EOF
+H 1
+H 2
+H init.t
+H sub/1
+H sub/2
+EOF
+
+cat >expect.skip <<EOF
+S 1
+H 2
+H init.t
+S sub/1
+H sub/2
+EOF
+
+NULL_SHA1=e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
+ZERO_SHA0=0000000000000000000000000000000000000000
+setup_absent() {
+	test -f 1 && rm 1
+	git update-index --remove 1 &&
+	git update-index --add --cacheinfo 100644 $NULL_SHA1 1 &&
+	git update-index --skip-worktree 1
+}
+
+test_absent() {
+	echo "100644 $NULL_SHA1 0	1" > expected &&
+	git ls-files --stage 1 > result &&
+	test_cmp expected result &&
+	test ! -f 1
+}
+
+setup_dirty() {
+	git update-index --force-remove 1 &&
+	echo dirty > 1 &&
+	git update-index --add --cacheinfo 100644 $NULL_SHA1 1 &&
+	git update-index --skip-worktree 1
+}
+
+test_dirty() {
+	echo "100644 $NULL_SHA1 0	1" > expected &&
+	git ls-files --stage 1 > result &&
+	test_cmp expected result &&
+	echo dirty > expected
+	test_cmp expected 1
+}
+
+test_expect_success 'setup' '
+	test_commit init &&
+	mkdir sub &&
+	touch ./1 ./2 sub/1 sub/2 &&
+	git add 1 2 sub/1 sub/2 &&
+	git update-index --skip-worktree 1 sub/1 &&
+	git ls-files -t > result &&
+	test_cmp expect.skip result
+'
+
+test_expect_success 'update-index' '
+	setup_absent &&
+	git update-index 1 &&
+	test_absent
+'
+
+test_expect_success 'update-index' '
+	setup_dirty &&
+	git update-index 1 &&
+	test_dirty
+'
+
+test_expect_success 'update-index --remove' '
+	setup_absent &&
+	git update-index --remove 1 &&
+	test -z "$(git ls-files 1)" &&
+	test ! -f 1
+'
+
+test_expect_success 'update-index --remove' '
+	setup_dirty &&
+	git update-index --remove 1 &&
+	test -z "$(git ls-files 1)" &&
+	echo dirty > expected &&
+	test_cmp expected 1
+'
+
+test_expect_success 'ls-files --delete' '
+	setup_absent &&
+	test -z "$(git ls-files -d)"
+'
+
+test_expect_success 'ls-files --delete' '
+	setup_dirty &&
+	test -z "$(git ls-files -d)"
+'
+
+test_expect_success 'ls-files --modified' '
+	setup_absent &&
+	test -z "$(git ls-files -m)"
+'
+
+test_expect_success 'ls-files --modified' '
+	setup_dirty &&
+	test -z "$(git ls-files -m)"
+'
+
+test_expect_success 'grep with skip-worktree file' '
+	git update-index --no-skip-worktree 1 &&
+	echo test > 1 &&
+	git update-index 1 &&
+	git update-index --skip-worktree 1 &&
+	rm 1 &&
+	test "$(git grep --no-ext-grep test)" = "1:test"
+'
+
+echo ":000000 100644 $ZERO_SHA0 $NULL_SHA1 A	1" > expected
+test_expect_success 'diff-index does not examine skip-worktree absent entries' '
+	setup_absent &&
+	git diff-index HEAD -- 1 > result &&
+	test_cmp expected result
+'
+
+test_expect_success 'diff-index does not examine skip-worktree dirty entries' '
+	setup_dirty &&
+	git diff-index HEAD -- 1 > result &&
+	test_cmp expected result
+'
+
+test_expect_success 'diff-files does not examine skip-worktree absent entries' '
+	setup_absent &&
+	test -z "$(git diff-files -- one)"
+'
+
+test_expect_success 'diff-files does not examine skip-worktree dirty entries' '
+	setup_dirty &&
+	test -z "$(git diff-files -- one)"
+'
+
+test_expect_success 'git-rm succeeds on skip-worktree absent entries' '
+	setup_absent &&
+	git rm 1
+'
+
+test_expect_failure 'commit on skip-worktree absent entries' '
+	git reset &&
+	setup_absent &&
+	test_must_fail git commit -m null 1
+'
+
+test_expect_failure 'commit on skip-worktree dirty entries' '
+	git reset &&
+	setup_dirty &&
+	test_must_fail git commit -m null 1
+'
+
+test_done
-- 
1.6.3.GIT

[RFC PATCH v4 05/19] Teach Git to respect skip-worktree bit (writing part)

[Nguyễn Thái Ngọc Duy]

This part is mainly to remove CE_VALID shortcuts (and as a
consequence, ce_uptodate() shortcuts as it may be turned on by
CE_VALID) in writing code path if skip-worktree is used. Various tests
are added to avoid future breakages.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 t/t7012-skip-worktree-writing.sh |  146 ++++++++++++++++++++++++++++++++++++++
 unpack-trees.c                   |    4 +-
 2 files changed, 148 insertions(+), 2 deletions(-)
 create mode 100755 t/t7012-skip-worktree-writing.sh

diff --git a/t/t7012-skip-worktree-writing.sh b/t/t7012-skip-worktree-writing.sh
new file mode 100755
index 0000000..8d8b1c0
--- /dev/null
+++ b/t/t7012-skip-worktree-writing.sh
@@ -0,0 +1,146 @@
+#!/bin/sh
+#
+# Copyright (c) 2008 Nguyễn Thái Ngọc Duy
+#
+
+test_description='test worktree writing operations when skip-worktree is used'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	test_commit init &&
+	echo modified >> init.t &&
+	touch added &&
+	git add init.t added &&
+	git commit -m "modified and added" &&
+	git tag top
+'
+
+test_expect_success 'read-tree updates worktree, absent case' '
+	git checkout -f top &&
+	git update-index --skip-worktree init.t &&
+	rm init.t &&
+	git read-tree -m -u HEAD^ &&
+	echo init > expected &&
+	test_cmp expected init.t
+'
+
+test_expect_success 'read-tree updates worktree, dirty case' '
+	git checkout -f top &&
+	git update-index --skip-worktree init.t &&
+	echo dirty >> init.t &&
+	test_must_fail git read-tree -m -u HEAD^ &&
+	grep -q dirty init.t &&
+	test "$(git ls-files -t init.t)" = "S init.t" &&
+	git update-index --no-skip-worktree init.t
+'
+
+test_expect_success 'read-tree removes worktree, absent case' '
+	git checkout -f top &&
+	git update-index --skip-worktree added &&
+	rm added &&
+	git read-tree -m -u HEAD^ &&
+	test ! -f added
+'
+
+test_expect_success 'read-tree removes worktree, dirty case' '
+	git checkout -f top &&
+	git update-index --skip-worktree added &&
+	echo dirty >> added &&
+	test_must_fail git read-tree -m -u HEAD^ &&
+	grep -q dirty added &&
+	test "$(git ls-files -t added)" = "S added" &&
+	git update-index --no-skip-worktree added
+'
+
+NULL_SHA1=e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
+ZERO_SHA0=0000000000000000000000000000000000000000
+setup_absent() {
+	test -f 1 && rm 1
+	git update-index --remove 1 &&
+	git update-index --add --cacheinfo 100644 $NULL_SHA1 1 &&
+	git update-index --skip-worktree 1
+}
+
+test_absent() {
+	echo "100644 $NULL_SHA1 0	1" > expected &&
+	git ls-files --stage 1 > result &&
+	test_cmp expected result &&
+	test ! -f 1
+}
+
+setup_dirty() {
+	git update-index --force-remove 1 &&
+	echo dirty > 1 &&
+	git update-index --add --cacheinfo 100644 $NULL_SHA1 1 &&
+	git update-index --skip-worktree 1
+}
+
+test_dirty() {
+	echo "100644 $NULL_SHA1 0	1" > expected &&
+	git ls-files --stage 1 > result &&
+	test_cmp expected result &&
+	echo dirty > expected
+	test_cmp expected 1
+}
+
+cat >expected <<EOF
+S 1
+H 2
+H init.t
+S sub/1
+H sub/2
+EOF
+
+test_expect_success 'index setup' '
+	git checkout -f init &&
+	mkdir sub &&
+	touch ./1 ./2 sub/1 sub/2 &&
+	git add 1 2 sub/1 sub/2 &&
+	git update-index --skip-worktree 1 sub/1 &&
+	git ls-files -t > result &&
+	test_cmp expected result
+'
+
+test_expect_success 'git-add ignores worktree content' '
+	setup_absent &&
+	git add 1 &&
+	test_absent
+'
+
+test_expect_success 'git-add ignores worktree content' '
+	setup_dirty &&
+	git add 1 &&
+	test_dirty
+'
+
+test_expect_success 'git-rm fails if worktree is dirty' '
+	setup_dirty &&
+	test_must_fail git rm 1 &&
+	test_dirty
+'
+
+cat >expected <<EOF
+Would remove expected
+Would remove result
+EOF
+test_expect_success 'git-clean, absent case' '
+	setup_absent &&
+	git clean -n > result &&
+	test_cmp expected result
+'
+
+test_expect_success 'git-clean, dirty case' '
+	setup_dirty &&
+	git clean -n > result &&
+	test_cmp expected result
+'
+
+test_expect_failure 'git-apply adds file' false
+test_expect_failure 'git-apply updates file' false
+test_expect_failure 'git-apply removes file' false
+test_expect_failure 'git-mv to skip-worktree' false
+test_expect_failure 'git-mv from skip-worktree' false
+test_expect_failure 'git-checkout' false
+
+test_done
diff --git a/unpack-trees.c b/unpack-trees.c
index 720f7a1..3eda263 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -450,7 +450,7 @@ static int verify_uptodate(struct cache_entry *ce,
 {
 	struct stat st;
 
-	if (o->index_only || o->reset || ce_uptodate(ce))
+	if (o->index_only || (!ce_skip_worktree(ce) && (o->reset || ce_uptodate(ce))))
 		return 0;
 
 	if (!lstat(ce->name, &st)) {
@@ -1004,7 +1004,7 @@ int oneway_merge(struct cache_entry **src, struct unpack_trees_options *o)
 
 	if (old && same(old, a)) {
 		int update = 0;
-		if (o->reset && !ce_uptodate(old)) {
+		if (o->reset && !ce_uptodate(old) && !ce_skip_worktree(old)) {
 			struct stat st;
 			if (lstat(old->name, &st) ||
 			    ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID))
-- 
1.6.3.GIT

[RFC PATCH v4 06/19] Avoid writing to buffer in add_excludes_from_file_1()

[Nguyễn Thái Ngọc Duy]

In the next patch, the buffer that is being used within
add_excludes_from_file_1() comes from another function and does not
have extra space to put \n at the end.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 dir.c |    5 ++---
 1 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/dir.c b/dir.c
index d0999ba..e9254d6 100644
--- a/dir.c
+++ b/dir.c
@@ -229,10 +229,9 @@ static int add_excludes_from_file_1(const char *fname,
 
 	if (buf_p)
 		*buf_p = buf;
-	buf[size++] = '\n';
 	entry = buf;
-	for (i = 0; i < size; i++) {
-		if (buf[i] == '\n') {
+	for (i = 0; i <= size; i++) {
+		if (i == size || buf[i] == '\n') {
 			if (entry != buf + i && entry[0] != '#') {
 				buf[i - (i && buf[i-1] == '\r')] = 0;
 				add_exclude(entry, base, baselen, which);
-- 
1.6.3.GIT

[RFC PATCH v4 07/19] Read .gitignore from index if it is skip-worktree

[Nguyễn Thái Ngọc Duy]

This adds index as a prerequisite for directory listing (with
exclude).  At the moment directory listing is used by "git clean",
"git add", "git ls-files" and "git status"/"git commit" and
unpack_trees()-related commands.  These commands have been
checked/modified to populate index before doing directory listing.

add_excludes_from_file() does not enable this feature, because it
is used to read .git/info/exclude and some explicit files specified
by "git ls-files".

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/technical/api-directory-listing.txt |    3 +
 builtin-clean.c                                   |    4 +-
 builtin-ls-files.c                                |    4 +-
 dir.c                                             |   65 ++++++++++++++------
 t/t3001-ls-files-others-exclude.sh                |   22 +++++++
 t/t7300-clean.sh                                  |   19 ++++++
 6 files changed, 95 insertions(+), 22 deletions(-)

diff --git a/Documentation/technical/api-directory-listing.txt b/Documentation/technical/api-directory-listing.txt
index 5bbd18f..add6f43 100644
--- a/Documentation/technical/api-directory-listing.txt
+++ b/Documentation/technical/api-directory-listing.txt
@@ -58,6 +58,9 @@ The result of the enumeration is left in these fields::
 Calling sequence
 ----------------
 
+Note: index may be looked at for .gitignore files that are CE_SKIP_WORKTREE
+marked. If you to exclude files, make sure you have loaded index first.
+
 * Prepare `struct dir_struct dir` and clear it with `memset(&dir, 0,
   sizeof(dir))`.
 
diff --git a/builtin-clean.c b/builtin-clean.c
index 05c763c..96ada75 100644
--- a/builtin-clean.c
+++ b/builtin-clean.c
@@ -75,11 +75,13 @@ int cmd_clean(int argc, const char **argv, const char *prefix)
 
 	dir.flags |= DIR_SHOW_OTHER_DIRECTORIES;
 
+	if (read_cache() < 0)
+		die("index file corrupt");
+
 	if (!ignored)
 		setup_standard_excludes(&dir);
 
 	pathspec = get_pathspec(prefix, argv);
-	read_cache();
 
 	fill_directory(&dir, pathspec);
 
diff --git a/builtin-ls-files.c b/builtin-ls-files.c
index ad7e447..2e47242 100644
--- a/builtin-ls-files.c
+++ b/builtin-ls-files.c
@@ -485,6 +485,9 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 		prefix_offset = strlen(prefix);
 	git_config(git_default_config, NULL);
 
+	if (read_cache() < 0)
+		die("index file corrupt");
+
 	argc = parse_options(argc, argv, prefix, builtin_ls_files_options,
 			ls_files_usage, 0);
 	if (show_tag || show_valid_bit) {
@@ -513,7 +516,6 @@ int cmd_ls_files(int argc, const char **argv, const char *prefix)
 	pathspec = get_pathspec(prefix, argv);
 
 	/* be nice with submodule paths ending in a slash */
-	read_cache();
 	if (pathspec)
 		strip_trailing_slash_from_submodules();
 
diff --git a/dir.c b/dir.c
index e9254d6..cda2556 100644
--- a/dir.c
+++ b/dir.c
@@ -200,11 +200,35 @@ void add_exclude(const char *string, const char *base,
 	which->excludes[which->nr++] = x;
 }
 
+static void *read_skip_worktree_file_from_index(const char *path, size_t *size)
+{
+	int pos, len;
+	unsigned long sz;
+	enum object_type type;
+	void *data;
+	struct index_state *istate = &the_index;
+
+	len = strlen(path);
+	pos = index_name_pos(istate, path, len);
+	if (pos < 0)
+		return NULL;
+	if (!ce_skip_worktree(istate->cache[pos]))
+		return NULL;
+	data = read_sha1_file(istate->cache[pos]->sha1, &type, &sz);
+	if (!data || type != OBJ_BLOB) {
+		free(data);
+		return NULL;
+	}
+	*size = xsize_t(sz);
+	return data;
+}
+
 static int add_excludes_from_file_1(const char *fname,
 				    const char *base,
 				    int baselen,
 				    char **buf_p,
-				    struct exclude_list *which)
+				    struct exclude_list *which,
+				    int check_index)
 {
 	struct stat st;
 	int fd, i;
@@ -212,20 +236,26 @@ static int add_excludes_from_file_1(const char *fname,
 	char *buf, *entry;
 
 	fd = open(fname, O_RDONLY);
-	if (fd < 0 || fstat(fd, &st) < 0)
-		goto err;
-	size = xsize_t(st.st_size);
-	if (size == 0) {
-		close(fd);
-		return 0;
+	if (fd < 0 || fstat(fd, &st) < 0) {
+		if (0 <= fd)
+			close(fd);
+		if (!check_index ||
+		    (buf = read_skip_worktree_file_from_index(fname, &size)) == NULL)
+			return -1;
 	}
-	buf = xmalloc(size+1);
-	if (read_in_full(fd, buf, size) != size)
-	{
-		free(buf);
-		goto err;
+	else {
+		size = xsize_t(st.st_size);
+		if (size == 0) {
+			close(fd);
+			return 0;
+		}
+		buf = xmalloc(size);
+		if (read_in_full(fd, buf, size) != size) {
+			close(fd);
+			return -1;
+		}
+		close(fd);
 	}
-	close(fd);
 
 	if (buf_p)
 		*buf_p = buf;
@@ -240,17 +270,12 @@ static int add_excludes_from_file_1(const char *fname,
 		}
 	}
 	return 0;
-
- err:
-	if (0 <= fd)
-		close(fd);
-	return -1;
 }
 
 void add_excludes_from_file(struct dir_struct *dir, const char *fname)
 {
 	if (add_excludes_from_file_1(fname, "", 0, NULL,
-				     &dir->exclude_list[EXC_FILE]) < 0)
+				     &dir->exclude_list[EXC_FILE], 0) < 0)
 		die("cannot use %s as an exclude file", fname);
 }
 
@@ -301,7 +326,7 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)
 		strcpy(dir->basebuf + stk->baselen, dir->exclude_per_dir);
 		add_excludes_from_file_1(dir->basebuf,
 					 dir->basebuf, stk->baselen,
-					 &stk->filebuf, el);
+					 &stk->filebuf, el, 1);
 		dir->exclude_stack = stk;
 		current = stk->baselen;
 	}
diff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh
index c65bca8..132c476 100755
--- a/t/t3001-ls-files-others-exclude.sh
+++ b/t/t3001-ls-files-others-exclude.sh
@@ -64,6 +64,8 @@ two/*.4
 echo '!*.2
 !*.8' >one/two/.gitignore
 
+allignores='.gitignore one/.gitignore one/two/.gitignore'
+
 test_expect_success \
     'git ls-files --others with various exclude options.' \
     'git ls-files --others \
@@ -85,6 +87,26 @@ test_expect_success \
        >output &&
      test_cmp expect output'
 
+test_expect_success 'setup skip-worktree gitignore' '
+	git add $allignores &&
+	git update-index --skip-worktree $allignores &&
+	rm $allignores
+'
+
+test_expect_success \
+    'git ls-files --others with various exclude options.' \
+    'git ls-files --others \
+       --exclude=\*.6 \
+       --exclude-per-directory=.gitignore \
+       --exclude-from=.git/ignore \
+       >output &&
+     test_cmp expect output'
+
+test_expect_success 'restore gitignore' '
+	git checkout $allignores &&
+	rm .git/index
+'
+
 cat > excludes-file <<\EOF
 *.[1-8]
 e*
diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh
index 118c6eb..7d8ed68 100755
--- a/t/t7300-clean.sh
+++ b/t/t7300-clean.sh
@@ -22,6 +22,25 @@ test_expect_success 'setup' '
 
 '
 
+test_expect_success 'git clean with skip-worktree .gitignore' '
+	git update-index --skip-worktree .gitignore &&
+	rm .gitignore &&
+	mkdir -p build docs &&
+	touch a.out src/part3.c docs/manual.txt obj.o build/lib.so &&
+	git clean &&
+	test -f Makefile &&
+	test -f README &&
+	test -f src/part1.c &&
+	test -f src/part2.c &&
+	test ! -f a.out &&
+	test ! -f src/part3.c &&
+	test -f docs/manual.txt &&
+	test -f obj.o &&
+	test -f build/lib.so &&
+	git update-index --no-skip-worktree .gitignore &&
+	git checkout .gitignore
+'
+
 test_expect_success 'git clean' '
 
 	mkdir -p build docs &&
-- 
1.6.3.GIT

[RFC PATCH v4 08/19] unpack-trees(): carry skip-worktree bit over in merged_entry()

[Nguyễn Thái Ngọc Duy]

In this code path, we would remove "old" and replace it with "merge".
"old" may have skip-worktree bit, so re-add it to "merge".

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 unpack-trees.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index 3eda263..dc6d74a 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -680,6 +680,8 @@ static int merged_entry(struct cache_entry *merge, struct cache_entry *old,
 		} else {
 			if (verify_uptodate(old, o))
 				return -1;
+			if (ce_skip_worktree(old))
+				update |= CE_SKIP_WORKTREE;
 			invalidate_ce_path(old, o);
 		}
 	}
-- 
1.6.3.GIT

[RFC PATCH v4 09/19] excluded_1(): support exclude files in index

[Nguyễn Thái Ngọc Duy]

Index does not really have "directories", attempts to match "foo/"
against index will fail unless someone tries to reconstruct directories
from a list of file.

Observing that dtype in this function can never be NULL (otherwise
it would segfault), dtype NULL will be used to say "hey.. you are
matching against index" and behave properly.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 dir.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/dir.c b/dir.c
index cda2556..a4a91d5 100644
--- a/dir.c
+++ b/dir.c
@@ -349,6 +349,12 @@ static int excluded_1(const char *pathname,
 			int to_exclude = x->to_exclude;
 
 			if (x->flags & EXC_FLAG_MUSTBEDIR) {
+				if (!dtype) {
+					if (!prefixcmp(pathname, exclude))
+						return to_exclude;
+					else
+						continue;
+				}
 				if (*dtype == DT_UNKNOWN)
 					*dtype = get_dtype(NULL, pathname, pathlen);
 				if (*dtype != DT_DIR)
-- 
1.6.3.GIT

[RFC PATCH v4 10/19] dir.c: export excluded_1() and add_excludes_from_file_1()

[Nguyễn Thái Ngọc Duy]

These functions are used to handle .gitignore. They are now exported
so that sparse checkout can reuse.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 dir.c |   32 ++++++++++++++++----------------
 dir.h |    4 ++++
 2 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/dir.c b/dir.c
index a4a91d5..3a8d3e6 100644
--- a/dir.c
+++ b/dir.c
@@ -223,12 +223,12 @@ static void *read_skip_worktree_file_from_index(const char *path, size_t *size)
 	return data;
 }
 
-static int add_excludes_from_file_1(const char *fname,
-				    const char *base,
-				    int baselen,
-				    char **buf_p,
-				    struct exclude_list *which,
-				    int check_index)
+int add_excludes_from_file_to_list(const char *fname,
+				   const char *base,
+				   int baselen,
+				   char **buf_p,
+				   struct exclude_list *which,
+				   int check_index)
 {
 	struct stat st;
 	int fd, i;
@@ -274,8 +274,8 @@ static int add_excludes_from_file_1(const char *fname,
 
 void add_excludes_from_file(struct dir_struct *dir, const char *fname)
 {
-	if (add_excludes_from_file_1(fname, "", 0, NULL,
-				     &dir->exclude_list[EXC_FILE], 0) < 0)
+	if (add_excludes_from_file_to_list(fname, "", 0, NULL,
+					   &dir->exclude_list[EXC_FILE], 0) < 0)
 		die("cannot use %s as an exclude file", fname);
 }
 
@@ -324,9 +324,9 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)
 		memcpy(dir->basebuf + current, base + current,
 		       stk->baselen - current);
 		strcpy(dir->basebuf + stk->baselen, dir->exclude_per_dir);
-		add_excludes_from_file_1(dir->basebuf,
-					 dir->basebuf, stk->baselen,
-					 &stk->filebuf, el, 1);
+		add_excludes_from_file_to_list(dir->basebuf,
+					       dir->basebuf, stk->baselen,
+					       &stk->filebuf, el, 1);
 		dir->exclude_stack = stk;
 		current = stk->baselen;
 	}
@@ -336,9 +336,9 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)
 /* Scan the list and let the last match determine the fate.
  * Return 1 for exclude, 0 for include and -1 for undecided.
  */
-static int excluded_1(const char *pathname,
-		      int pathlen, const char *basename, int *dtype,
-		      struct exclude_list *el)
+int excluded_from_list(const char *pathname,
+		       int pathlen, const char *basename, int *dtype,
+		       struct exclude_list *el)
 {
 	int i;
 
@@ -412,8 +412,8 @@ int excluded(struct dir_struct *dir, const char *pathname, int *dtype_p)
 
 	prep_exclude(dir, pathname, basename-pathname);
 	for (st = EXC_CMDL; st <= EXC_FILE; st++) {
-		switch (excluded_1(pathname, pathlen, basename,
-				   dtype_p, &dir->exclude_list[st])) {
+		switch (excluded_from_list(pathname, pathlen, basename,
+					   dtype_p, &dir->exclude_list[st])) {
 		case 0:
 			return 0;
 		case 1:
diff --git a/dir.h b/dir.h
index 320b6a2..3bead5f 100644
--- a/dir.h
+++ b/dir.h
@@ -69,7 +69,11 @@ extern int match_pathspec(const char **pathspec, const char *name, int namelen,
 extern int fill_directory(struct dir_struct *dir, const char **pathspec);
 extern int read_directory(struct dir_struct *, const char *path, int len, const char **pathspec);
 
+extern int excluded_from_list(const char *pathname, int pathlen, const char *basename,
+			      int *dtype, struct exclude_list *el);
 extern int excluded(struct dir_struct *, const char *, int *);
+extern int add_excludes_from_file_to_list(const char *fname, const char *base, int baselen,
+					  char **buf_p, struct exclude_list *which, int check_index);
 extern void add_excludes_from_file(struct dir_struct *, const char *fname);
 extern void add_exclude(const char *string, const char *base,
 			int baselen, struct exclude_list *which);
-- 
1.6.3.GIT

[RFC PATCH v4 11/19] Introduce "sparse checkout"

[Nguyễn Thái Ngọc Duy]

With skip-worktree bit, you can manually set it to unwanted files,
then remove them: you would have the so-called sparse checkout. The
disadvantages are:

 - Porcelain tools are not aware of this. Everytime you do an
   operation that may update working directory, skip-worktree may be
   cleared out. You have to set them again.

 - You still have to remove skip-worktree'd files manually, which is
   boring and ineffective.

These will be addressed in the following patches. This patch gives an
idea what is "sparse checkout" in Documentation/git-read-tree.txt.
This file is chosen instead of git-checkout.txt because it is quite
technical and user-unfriendly. I'd expect git-checkout.txt to have
something when Porcelain support is done.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-read-tree.txt |   44 +++++++++++++++++++++++++++++++++++++++
 1 files changed, 44 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt
index 4a932b0..8b39716 100644
--- a/Documentation/git-read-tree.txt
+++ b/Documentation/git-read-tree.txt
@@ -360,6 +360,50 @@ middle of doing, and when your working tree is ready (i.e. you
 have finished your work-in-progress), attempt the merge again.
 
 
+Sparse checkout
+---------------
+
+"Sparse checkout" allows to sparsely populate working directory.
+It uses skip-worktree bit (see linkgit:git-update-index[1]) to tell
+Git whether a file on working directory is worth looking at.
+
+"git read-tree" and other merge-based commands ("git merge", "git
+checkout"...) can help maintaining skip-worktree bitmap and working
+directory update. `$GIT_DIR/info/sparse-checkout` is used to
+define the skip-worktree reference bitmap. When "git read-tree" needs
+to update working directory, it will reset skip-worktree bit in index
+based on this file, which uses the same syntax as .gitignore files.
+If an entry matches a pattern in this file, skip-worktree will be
+set on that entry. Otherwise, skip-worktree will be unset.
+
+Then it compares the new skip-worktree value with the previous one. If
+skip-worktree turns from unset to set, it will add the corresponding
+file back. If it turns from set to unset, that file will be removed.
+
+While `$GIT_DIR/info/sparse-checkout` is usually used to specify what
+files are in. You can also specify what files are _not_ in, using
+negate patterns. For example, to remove file "unwanted":
+
+----------------
+*
+!unwanted
+----------------
+
+Another tricky thing is fully repopulating working directory when you
+no longer want sparse checkout. You cannot just disable "sparse
+checkout" because skip-worktree are still in the index and you working
+directory is still sparsely populated. You should re-populate working
+directory with the `$GIT_DIR/info/sparse-checkout` file content as
+follows:
+
+----------------
+*
+----------------
+
+Then you can disable sparse checkout. Sparse checkout support in "git
+read-tree" and similar commands is disabled by default.
+
+
 SEE ALSO
 --------
 linkgit:git-write-tree[1]; linkgit:git-ls-files[1];
-- 
1.6.3.GIT

[RFC PATCH v4 12/19] unpack-trees(): add CE_WT_REMOVE to remove on worktree alone

[Nguyễn Thái Ngọc Duy]

CE_REMOVE now removes both worktree and index versions. Sparse
checkout must be able to remove worktree version while keep the
index intact when checkout area is narrowed.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 cache.h        |    3 +++
 unpack-trees.c |    9 ++++++++-
 2 files changed, 11 insertions(+), 1 deletions(-)

diff --git a/cache.h b/cache.h
index 3b0e5fc..0e912a9 100644
--- a/cache.h
+++ b/cache.h
@@ -177,6 +177,9 @@ struct cache_entry {
 #define CE_HASHED    (0x100000)
 #define CE_UNHASHED  (0x200000)
 
+/* Only remove in work directory, not index */
+#define CE_WT_REMOVE (0x400000)
+
 /*
  * Extended on-disk flags
  */
diff --git a/unpack-trees.c b/unpack-trees.c
index dc6d74a..6a51a69 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -78,7 +78,7 @@ static int check_updates(struct unpack_trees_options *o)
 	if (o->update && o->verbose_update) {
 		for (total = cnt = 0; cnt < index->cache_nr; cnt++) {
 			struct cache_entry *ce = index->cache[cnt];
-			if (ce->ce_flags & (CE_UPDATE | CE_REMOVE))
+			if (ce->ce_flags & (CE_UPDATE | CE_REMOVE | CE_WT_REMOVE))
 				total++;
 		}
 
@@ -92,6 +92,13 @@ static int check_updates(struct unpack_trees_options *o)
 	for (i = 0; i < index->cache_nr; i++) {
 		struct cache_entry *ce = index->cache[i];
 
+		if (ce->ce_flags & CE_WT_REMOVE) {
+			display_progress(progress, ++cnt);
+			if (o->update)
+				unlink_entry(ce);
+			continue;
+		}
+
 		if (ce->ce_flags & CE_REMOVE) {
 			display_progress(progress, ++cnt);
 			if (o->update)
-- 
1.6.3.GIT

[RFC PATCH v4 13/19] unpack-trees.c: generalize verify_* functions

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 unpack-trees.c |   23 ++++++++++++++++++-----
 1 files changed, 18 insertions(+), 5 deletions(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index 6a51a69..8eb4b70 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -452,8 +452,9 @@ static int same(struct cache_entry *a, struct cache_entry *b)
  * When a CE gets turned into an unmerged entry, we
  * want it to be up-to-date
  */
-static int verify_uptodate(struct cache_entry *ce,
-		struct unpack_trees_options *o)
+static int verify_uptodate_1(struct cache_entry *ce,
+				   struct unpack_trees_options *o,
+				   const char *error_msg)
 {
 	struct stat st;
 
@@ -478,7 +479,13 @@ static int verify_uptodate(struct cache_entry *ce,
 	if (errno == ENOENT)
 		return 0;
 	return o->gently ? -1 :
-		error(ERRORMSG(o, not_uptodate_file), ce->name);
+		error(error_msg, ce->name);
+}
+
+static int verify_uptodate(struct cache_entry *ce,
+			   struct unpack_trees_options *o)
+{
+	return verify_uptodate_1(ce, o, ERRORMSG(o, not_uptodate_file));
 }
 
 static void invalidate_ce_path(struct cache_entry *ce, struct unpack_trees_options *o)
@@ -586,8 +593,9 @@ static int icase_exists(struct unpack_trees_options *o, struct cache_entry *dst,
  * We do not want to remove or overwrite a working tree file that
  * is not tracked, unless it is ignored.
  */
-static int verify_absent(struct cache_entry *ce, const char *action,
-			 struct unpack_trees_options *o)
+static int verify_absent_1(struct cache_entry *ce, const char *action,
+				 struct unpack_trees_options *o,
+				 const char *error_msg)
 {
 	struct stat st;
 
@@ -667,6 +675,11 @@ static int verify_absent(struct cache_entry *ce, const char *action,
 	}
 	return 0;
 }
+static int verify_absent(struct cache_entry *ce, const char *action,
+			 struct unpack_trees_options *o)
+{
+	return verify_absent_1(ce, action, o, ERRORMSG(o, would_lose_untracked));
+}
 
 static int merged_entry(struct cache_entry *merge, struct cache_entry *old,
 		struct unpack_trees_options *o)
-- 
1.6.3.GIT

[RFC PATCH v4 14/19] unpack-trees(): "enable" sparse checkout and load $GIT_DIR/info/sparse-checkout

[Nguyễn Thái Ngọc Duy]

This patch introduces core.sparseCheckout, which will control whether
sparse checkout support is enabled in unpack_trees()

It also loads sparse-checkout file that will be used in the next patch.
I split it out so the next patch will be shorter, easier to read.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/config.txt        |    4 ++++
 Documentation/git-read-tree.txt |    4 +++-
 cache.h                         |    1 +
 config.c                        |    5 +++++
 environment.c                   |    1 +
 unpack-trees.c                  |   36 ++++++++++++++++++++++++++++++------
 unpack-trees.h                  |    4 ++++
 7 files changed, 48 insertions(+), 7 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 5256c7f..bfaa78d 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -439,6 +439,10 @@ On some file system/operating system combinations, this is unreliable.
 Set this config setting to 'rename' there; However, This will remove the
 check that makes sure that existing object files will not get overwritten.
 
+core.sparseCheckout::
+	Enable "sparse checkout" feature. See section "Sparse checkout" in
+	linkgit:git-read-tree[1] for more information.
+
 add.ignore-errors::
 	Tells 'git-add' to continue adding files when some files cannot be
 	added due to indexing errors. Equivalent to the '--ignore-errors'
diff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt
index 8b39716..fc3f08b 100644
--- a/Documentation/git-read-tree.txt
+++ b/Documentation/git-read-tree.txt
@@ -401,7 +401,9 @@ follows:
 ----------------
 
 Then you can disable sparse checkout. Sparse checkout support in "git
-read-tree" and similar commands is disabled by default.
+read-tree" and similar commands is disabled by default. You need to
+turn `core.sparseCheckout` on in order to have sparse checkout
+support.
 
 
 SEE ALSO
diff --git a/cache.h b/cache.h
index 0e912a9..a401daf 100644
--- a/cache.h
+++ b/cache.h
@@ -528,6 +528,7 @@ extern int auto_crlf;
 extern int read_replace_refs;
 extern int fsync_object_files;
 extern int core_preload_index;
+extern int core_apply_sparse_checkout;
 
 enum safe_crlf {
 	SAFE_CRLF_FALSE = 0,
diff --git a/config.c b/config.c
index e87edea..abd762e 100644
--- a/config.c
+++ b/config.c
@@ -503,6 +503,11 @@ static int git_default_core_config(const char *var, const char *value)
 		return 0;
 	}
 
+	if (!strcmp(var, "core.sparsecheckout")) {
+		core_apply_sparse_checkout = git_config_bool(var, value);
+		return 0;
+	}
+
 	/* Add other config variables here and to Documentation/config.txt. */
 	return 0;
 }
diff --git a/environment.c b/environment.c
index 5de6837..f9ea207 100644
--- a/environment.c
+++ b/environment.c
@@ -50,6 +50,7 @@ enum push_default_type push_default = PUSH_DEFAULT_MATCHING;
 #endif
 enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;
 int grafts_replace_parents = 1;
+int core_apply_sparse_checkout;
 
 /* Parallel index stat data preload? */
 int core_preload_index = 0;
diff --git a/unpack-trees.c b/unpack-trees.c
index 8eb4b70..44f8fdf 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -378,6 +378,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 {
 	int ret;
 	static struct cache_entry *dfc;
+	struct exclude_list el;
 
 	if (len > MAX_UNPACK_TREES)
 		die("unpack_trees takes at most %d trees", MAX_UNPACK_TREES);
@@ -387,6 +388,16 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 	state.quiet = 1;
 	state.refresh_cache = 1;
 
+	memset(&el, 0, sizeof(el));
+	if (!core_apply_sparse_checkout || !o->update)
+		o->skip_sparse_checkout = 1;
+	if (!o->skip_sparse_checkout) {
+		if (add_excludes_from_file_to_list(git_path("info/sparse-checkout"), "", 0, NULL, &el, 0) < 0)
+			o->skip_sparse_checkout = 1;
+		else
+			o->el = &el;
+	}
+
 	memset(&o->result, 0, sizeof(o->result));
 	o->result.initialized = 1;
 	if (o->src_index) {
@@ -407,26 +418,39 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 		info.fn = unpack_callback;
 		info.data = o;
 
-		if (traverse_trees(len, t, &info) < 0)
-			return unpack_failed(o, NULL);
+		if (traverse_trees(len, t, &info) < 0) {
+			ret = unpack_failed(o, NULL);
+			goto done;
+		}
 	}
 
 	/* Any left-over entries in the index? */
 	if (o->merge) {
 		while (o->pos < o->src_index->cache_nr) {
 			struct cache_entry *ce = o->src_index->cache[o->pos];
-			if (unpack_index_entry(ce, o) < 0)
-				return unpack_failed(o, NULL);
+			if (unpack_index_entry(ce, o) < 0) {
+				ret = unpack_failed(o, NULL);
+				goto done;
+			}
 		}
 	}
 
-	if (o->trivial_merges_only && o->nontrivial_merge)
-		return unpack_failed(o, "Merge requires file-level merging");
+	if (o->trivial_merges_only && o->nontrivial_merge) {
+		ret = unpack_failed(o, "Merge requires file-level merging");
+		goto done;
+	}
 
 	o->src_index = NULL;
 	ret = check_updates(o) ? (-2) : 0;
 	if (o->dst_index)
 		*o->dst_index = o->result;
+
+done:
+	for (i = 0;i < el.nr;i++)
+		free(el.excludes[i]);
+	if (el.excludes)
+		free(el.excludes);
+
 	return ret;
 }
 
diff --git a/unpack-trees.h b/unpack-trees.h
index d19df44..5c9e98a 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -4,6 +4,7 @@
 #define MAX_UNPACK_TREES 8
 
 struct unpack_trees_options;
+struct exclude_list;
 
 typedef int (*merge_fn_t)(struct cache_entry **src,
 		struct unpack_trees_options *options);
@@ -28,6 +29,7 @@ struct unpack_trees_options {
 		     skip_unmerged,
 		     initial_checkout,
 		     diff_index_cached,
+		     skip_sparse_checkout,
 		     gently;
 	const char *prefix;
 	int pos;
@@ -44,6 +46,8 @@ struct unpack_trees_options {
 	struct index_state *dst_index;
 	struct index_state *src_index;
 	struct index_state result;
+
+	struct exclude_list *el; /* for internal use */
 };
 
 extern int unpack_trees(unsigned n, struct tree_desc *t,
-- 
1.6.3.GIT

[RFC PATCH v4 15/19] unpack_trees(): apply $GIT_DIR/info/sparse-checkout to the final index

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 unpack-trees.c |   82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 unpack-trees.h |    2 +
 2 files changed, 83 insertions(+), 1 deletions(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index 44f8fdf..2d8ecb7 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -32,6 +32,12 @@ static struct unpack_trees_error_msgs unpack_plumbing_errors = {
 
 	/* bind_overlap */
 	"Entry '%s' overlaps with '%s'.  Cannot bind.",
+
+	/* sparse_not_uptodate_file */
+	"Entry '%s' not uptodate. Cannot update sparse checkout.",
+
+	/* would_lose_orphaned */
+	"Working tree file '%s' would be %s by sparse checkout update.",
 };
 
 #define ERRORMSG(o,fld) \
@@ -125,6 +131,57 @@ static int check_updates(struct unpack_trees_options *o)
 	return errs != 0;
 }
 
+static int verify_uptodate_sparse(struct cache_entry *ce, struct unpack_trees_options *o);
+static int verify_absent_sparse(struct cache_entry *ce, const char *action, struct unpack_trees_options *o);
+
+static int will_have_skip_worktree(const struct cache_entry *ce, struct unpack_trees_options *o)
+{
+	const char *basename;
+
+	if (ce_stage(ce))
+		return 0;
+
+	basename = strrchr(ce->name, '/');
+	basename = basename ? basename+1 : ce->name;
+	return excluded_from_list(ce->name, ce_namelen(ce), basename, NULL, o->el) <= 0;
+}
+
+static int apply_sparse_checkout(struct cache_entry *ce, struct unpack_trees_options *o)
+{
+	int was_skip_worktree = ce_skip_worktree(ce);
+
+	if (will_have_skip_worktree(ce, o))
+		ce->ce_flags |= CE_SKIP_WORKTREE;
+	else
+		ce->ce_flags &= ~CE_SKIP_WORKTREE;
+
+	/*
+	 * We only care about files getting into the checkout area
+	 * If merge strategies want to remove some, go ahead, this
+	 * flag will be removed eventually in unpack_trees() if it's
+	 * outside checkout area.
+	 */
+	if (ce->ce_flags & CE_REMOVE)
+		return 0;
+
+	if (!was_skip_worktree && ce_skip_worktree(ce)) {
+		/*
+		 * If CE_UPDATE is set, verify_uptodate() must be called already
+		 * also stat info may have lost after merged_entry() so calling
+		 * verify_uptodate() again may fail
+		 */
+		if (!(ce->ce_flags & CE_UPDATE) && verify_uptodate_sparse(ce, o))
+			return -1;
+		ce->ce_flags |= CE_WT_REMOVE;
+	}
+	if (was_skip_worktree && !ce_skip_worktree(ce)) {
+		if (verify_absent_sparse(ce, "overwritten", o))
+			return -1;
+		ce->ce_flags |= CE_UPDATE;
+	}
+	return 0;
+}
+
 static inline int call_unpack_fn(struct cache_entry **src, struct unpack_trees_options *o)
 {
 	int ret = o->fn(src, o);
@@ -376,7 +433,7 @@ static int unpack_failed(struct unpack_trees_options *o, const char *message)
  */
 int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o)
 {
-	int ret;
+	int i, ret;
 	static struct cache_entry *dfc;
 	struct exclude_list el;
 
@@ -440,6 +497,17 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 		goto done;
 	}
 
+	if (!o->skip_sparse_checkout) {
+		for (i = 0;i < o->result.cache_nr;i++) {
+			struct cache_entry *ce = o->result.cache[i];
+
+			if (apply_sparse_checkout(ce, o)) {
+				ret = -1;
+				goto done;
+			}
+		}
+	}
+
 	o->src_index = NULL;
 	ret = check_updates(o) ? (-2) : 0;
 	if (o->dst_index)
@@ -512,6 +580,12 @@ static int verify_uptodate(struct cache_entry *ce,
 	return verify_uptodate_1(ce, o, ERRORMSG(o, not_uptodate_file));
 }
 
+static int verify_uptodate_sparse(struct cache_entry *ce,
+				  struct unpack_trees_options *o)
+{
+	return verify_uptodate_1(ce, o, ERRORMSG(o, sparse_not_uptodate_file));
+}
+
 static void invalidate_ce_path(struct cache_entry *ce, struct unpack_trees_options *o)
 {
 	if (ce)
@@ -705,6 +779,12 @@ static int verify_absent(struct cache_entry *ce, const char *action,
 	return verify_absent_1(ce, action, o, ERRORMSG(o, would_lose_untracked));
 }
 
+static int verify_absent_sparse(struct cache_entry *ce, const char *action,
+			 struct unpack_trees_options *o)
+{
+	return verify_absent_1(ce, action, o, ERRORMSG(o, would_lose_orphaned));
+}
+
 static int merged_entry(struct cache_entry *merge, struct cache_entry *old,
 		struct unpack_trees_options *o)
 {
diff --git a/unpack-trees.h b/unpack-trees.h
index 5c9e98a..95ff36c 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -15,6 +15,8 @@ struct unpack_trees_error_msgs {
 	const char *not_uptodate_dir;
 	const char *would_lose_untracked;
 	const char *bind_overlap;
+	const char *sparse_not_uptodate_file;
+	const char *would_lose_orphaned;
 };
 
 struct unpack_trees_options {
-- 
1.6.3.GIT

[RFC PATCH v4 16/19] unpack-trees(): ignore worktree check outside checkout area

[Nguyễn Thái Ngọc Duy]

verify_absent() and verify_uptodate() are used to ensure worktree
is safe to be updated, then CE_REMOVE or CE_UPDATE will be set.
Finally check_updates() bases on CE_REMOVE, CE_UPDATE and the
recently added CE_WT_REMOVE to update working directory accordingly.

The entries that are checked may eventually be left out of checkout
area (done later in apply_sparse_checkout()). We don't want to update
outside checkout area. This patch teaches Git to assume "good",
skip these checks when it's sure those entries will be outside checkout
area, and clear CE_REMOVE|CE_UPDATE that could be set due to this
assumption.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 unpack-trees.c |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/unpack-trees.c b/unpack-trees.c
index 2d8ecb7..72743b3 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -505,6 +505,14 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 				ret = -1;
 				goto done;
 			}
+			/*
+			 * Merge strategies may set CE_UPDATE|CE_REMOVE outside checkout
+			 * area as a result of ce_skip_worktree() shortcuts in
+			 * verify_absent() and verify_uptodate(). Clear them.
+			 */
+			if (ce_skip_worktree(ce))
+				ce->ce_flags &= ~(CE_UPDATE | CE_REMOVE);
+
 		}
 	}
 
@@ -577,6 +585,8 @@ static int verify_uptodate_1(struct cache_entry *ce,
 static int verify_uptodate(struct cache_entry *ce,
 			   struct unpack_trees_options *o)
 {
+	if (!o->skip_sparse_checkout && will_have_skip_worktree(ce, o))
+		return 0;
 	return verify_uptodate_1(ce, o, ERRORMSG(o, not_uptodate_file));
 }
 
@@ -776,6 +786,8 @@ static int verify_absent_1(struct cache_entry *ce, const char *action,
 static int verify_absent(struct cache_entry *ce, const char *action,
 			 struct unpack_trees_options *o)
 {
+	if (!o->skip_sparse_checkout && will_have_skip_worktree(ce, o))
+		return 0;
 	return verify_absent_1(ce, action, o, ERRORMSG(o, would_lose_untracked));
 }
 
-- 
1.6.3.GIT

[RFC PATCH v4 17/19] read-tree: add --no-sparse-checkout to disable sparse checkout support

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-read-tree.txt |    6 +++++-
 builtin-read-tree.c             |    4 +++-
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt
index fc3f08b..ea7b0b2 100644
--- a/Documentation/git-read-tree.txt
+++ b/Documentation/git-read-tree.txt
@@ -10,7 +10,7 @@ SYNOPSIS
 --------
 'git read-tree' [[-m [--trivial] [--aggressive] | --reset | --prefix=<prefix>]
 		[-u [--exclude-per-directory=<gitignore>] | -i]]
-		[--index-output=<file>]
+		[--index-output=<file>] [--no-sparse-checkout]
 		<tree-ish1> [<tree-ish2> [<tree-ish3>]]
 
 
@@ -110,6 +110,10 @@ OPTIONS
 	directories the index file and index output file are
 	located in.
 
+--no-sparse-checkout::
+	Disable sparse checkout support even if `core.sparseCheckout`
+	is true.
+
 <tree-ish#>::
 	The id of the tree object(s) to be read/merged.
 
diff --git a/builtin-read-tree.c b/builtin-read-tree.c
index 9c2d634..f5acb1a 100644
--- a/builtin-read-tree.c
+++ b/builtin-read-tree.c
@@ -31,7 +31,7 @@ static int list_tree(unsigned char *sha1)
 }
 
 static const char * const read_tree_usage[] = {
-	"git read-tree [[-m [--trivial] [--aggressive] | --reset | --prefix=<prefix>] [-u [--exclude-per-directory=<gitignore>] | -i]]  [--index-output=<file>] <tree-ish1> [<tree-ish2> [<tree-ish3>]]",
+	"git read-tree [[-m [--trivial] [--aggressive] | --reset | --prefix=<prefix>] [-u [--exclude-per-directory=<gitignore>] | -i]] [--no-sparse-checkout] [--index-output=<file>] <tree-ish1> [<tree-ish2> [<tree-ish3>]]",
 	NULL
 };
 
@@ -98,6 +98,8 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)
 		  PARSE_OPT_NONEG, exclude_per_directory_cb },
 		OPT_SET_INT('i', NULL, &opts.index_only,
 			    "don't check the working tree after merging", 1),
+		OPT_SET_INT(0, "no-sparse-checkout", &opts.skip_sparse_checkout,
+			    "skip applying sparse checkout filter", 1),
 		OPT_END()
 	};
 
-- 
1.6.3.GIT

[RFC PATCH v4 18/19] Add tests for sparse checkout

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 t/t1009-read-tree-sparse-checkout.sh |  154 ++++++++++++++++++++++++++++++++++
 1 files changed, 154 insertions(+), 0 deletions(-)
 create mode 100755 t/t1009-read-tree-sparse-checkout.sh

diff --git a/t/t1009-read-tree-sparse-checkout.sh b/t/t1009-read-tree-sparse-checkout.sh
new file mode 100755
index 0000000..2192f5a
--- /dev/null
+++ b/t/t1009-read-tree-sparse-checkout.sh
@@ -0,0 +1,154 @@
+#!/bin/sh
+
+test_description='sparse checkout tests'
+
+. ./test-lib.sh
+
+cat >expected <<EOF
+100644 77f0ba1734ed79d12881f81b36ee134de6a3327b 0	init.t
+100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0	sub/added
+EOF
+test_expect_success 'setup' '
+	test_commit init &&
+	echo modified >> init.t &&
+	mkdir sub &&
+	touch sub/added &&
+	git add init.t sub/added &&
+	git commit -m "modified and added" &&
+	git tag top &&
+	git rm sub/added &&
+	git commit -m removed &&
+	git tag removed &&
+	git checkout top &&
+	git ls-files --stage > result &&
+	test_cmp expected result
+'
+
+cat >expected.swt <<EOF
+H init.t
+H sub/added
+EOF
+test_expect_success 'read-tree without .git/info/sparse-checkout' '
+	git read-tree -m -u HEAD &&
+	git ls-files --stage > result &&
+	test_cmp expected result &&
+	git ls-files -t > result &&
+	test_cmp expected.swt result
+'
+
+test_expect_success 'read-tree with .git/info/sparse-checkout but disabled' '
+	echo > .git/info/sparse-checkout
+	git read-tree -m -u HEAD &&
+	git ls-files -t > result &&
+	test_cmp expected.swt result &&
+	test -f init.t &&
+	test -f sub/added
+'
+
+test_expect_success 'read-tree --no-sparse-checkout with empty .git/info/sparse-checkout and enabled' '
+	git config core.sparsecheckout true &&
+	echo > .git/info/sparse-checkout &&
+	git read-tree --no-sparse-checkout -m -u HEAD &&
+	git ls-files -t > result &&
+	test_cmp expected.swt result &&
+	test -f init.t &&
+	test -f sub/added
+'
+
+cat >expected.swt <<EOF
+S init.t
+S sub/added
+EOF
+test_expect_success 'read-tree with empty .git/info/sparse-checkout' '
+	git config core.sparsecheckout true &&
+	echo > .git/info/sparse-checkout &&
+	git read-tree -m -u HEAD &&
+	git ls-files --stage > result &&
+	test_cmp expected result &&
+	git ls-files -t > result &&
+	test_cmp expected.swt result &&
+	test ! -f init.t &&
+	test ! -f sub/added
+'
+
+cat >expected.swt <<EOF
+S init.t
+H sub/added
+EOF
+test_expect_success 'match directories with trailing slash' '
+	echo sub/ > .git/info/sparse-checkout &&
+	git read-tree -m -u HEAD &&
+	git ls-files -t > result &&
+	test_cmp expected.swt result &&
+	test ! -f init.t &&
+	test -f sub/added
+'
+
+cat >expected.swt <<EOF
+H init.t
+H sub/added
+EOF
+test_expect_failure 'match directories without trailing slash' '
+	echo init.t > .git/info/sparse-checkout &&
+	echo sub >> .git/info/sparse-checkout &&
+	git read-tree -m -u HEAD &&
+	git ls-files -t > result &&
+	test_cmp expected.swt result &&
+	test ! -f init.t &&
+	test -f sub/added
+'
+
+cat >expected.swt <<EOF
+H init.t
+S sub/added
+EOF
+test_expect_success 'checkout area changes' '
+	echo init.t > .git/info/sparse-checkout &&
+	git read-tree -m -u HEAD &&
+	git ls-files -t > result &&
+	test_cmp expected.swt result &&
+	test -f init.t &&
+	test ! -f sub/added
+'
+
+test_expect_success 'read-tree updates worktree, absent case' '
+	echo sub/added > .git/info/sparse-checkout &&
+	git checkout -f top &&
+	git read-tree -m -u HEAD^ &&
+	test ! -f init.t
+'
+
+test_expect_success 'read-tree updates worktree, dirty case' '
+	echo sub/added > .git/info/sparse-checkout &&
+	git checkout -f top &&
+	echo dirty > init.t &&
+	git read-tree -m -u HEAD^ &&
+	grep -q dirty init.t &&
+	rm init.t
+'
+
+test_expect_success 'read-tree removes worktree, dirty case' '
+	echo init.t > .git/info/sparse-checkout &&
+	git checkout -f top &&
+	echo dirty > added &&
+	git read-tree -m -u HEAD^ &&
+	grep -q dirty added
+'
+
+test_expect_success 'read-tree adds to worktree, absent case' '
+	echo init.t > .git/info/sparse-checkout &&
+	git checkout -f removed &&
+	git read-tree -u -m HEAD^ &&
+	test ! -f sub/added
+'
+
+test_expect_success 'read-tree adds to worktree, dirty case' '
+	echo init.t > .git/info/sparse-checkout &&
+	git checkout -f removed &&
+	mkdir sub &&
+	echo dirty > sub/added &&
+	git read-tree -u -m HEAD^ &&
+	grep -q dirty sub/added
+'
+
+test_done
-- 
1.6.3.GIT

[RFC PATCH v4 19/19] sparse checkout: inhibit empty worktree

[Nguyễn Thái Ngọc Duy]

The way sparse checkout works, users may empty their worktree
completely, because of non-matching sparse-checkout spec, or empty
spec. I believe this is not desired. This patch makes Git refuse to
produce such worktree.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 t/t1009-read-tree-sparse-checkout.sh |   10 +++-------
 unpack-trees.c                       |    7 +++++++
 2 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/t/t1009-read-tree-sparse-checkout.sh b/t/t1009-read-tree-sparse-checkout.sh
index 2192f5a..62246db 100755
--- a/t/t1009-read-tree-sparse-checkout.sh
+++ b/t/t1009-read-tree-sparse-checkout.sh
@@ -55,20 +55,16 @@ test_expect_success 'read-tree --no-sparse-checkout with empty .git/info/sparse-
 	test -f sub/added
 '
 
-cat >expected.swt <<EOF
-S init.t
-S sub/added
-EOF
 test_expect_success 'read-tree with empty .git/info/sparse-checkout' '
 	git config core.sparsecheckout true &&
 	echo > .git/info/sparse-checkout &&
-	git read-tree -m -u HEAD &&
+	test_must_fail git read-tree -m -u HEAD &&
 	git ls-files --stage > result &&
 	test_cmp expected result &&
 	git ls-files -t > result &&
 	test_cmp expected.swt result &&
-	test ! -f init.t &&
-	test ! -f sub/added
+	test -f init.t &&
+	test -f sub/added
 '
 
 cat >expected.swt <<EOF
diff --git a/unpack-trees.c b/unpack-trees.c
index 72743b3..80ae2a0 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -498,6 +498,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 	}
 
 	if (!o->skip_sparse_checkout) {
+		int empty_worktree = 1;
 		for (i = 0;i < o->result.cache_nr;i++) {
 			struct cache_entry *ce = o->result.cache[i];
 
@@ -512,8 +513,14 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 			 */
 			if (ce_skip_worktree(ce))
 				ce->ce_flags &= ~(CE_UPDATE | CE_REMOVE);
+			else
+				empty_worktree = 0;
 
 		}
+		if (o->result.cache_nr && empty_worktree) {
+			ret = unpack_failed(o, "Sparse checkout leaves no entry on working directory");
+			goto done;
+		}
 	}
 
 	o->src_index = NULL;
-- 
1.6.3.GIT

Re: [RFC PATCH v4 00/19] Sparse checkout

[Jakub Narebski]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-16be, Size: 1805 bytes --]

Nguyþÿ\x1eÅn Thþÿ
> Welcome to the fourth round of sparse checkout this year, dubbed "The
> mummy^W^W^W^W^Wno-checkout returns", although the bit now comes under
> a new name "skip-worktree" [1]. This series has two parts: sparse
> worktree and sparse checkout. Details will be given below.
> 
> For brave users of this series: I have pushed it to my sparse-checkout
> branch [2] so you can just clone it and test away. Visible changes:
> 
>  - the spec is now .git/info/sparse-checkout
>  - the spec is positive matching, that is it matches what _is_ in
>    worktree, not what is out of worktree like the last series
>  - you need to set core.sparsecheckout no additional command argument
>    is needed.
[...]

> [1] There are a few reasons behind this rename:
>  - there is nothing about "checkout" in the first part, it's about
>    "sparse worktree"
>  - the double negation issue with "no-" part of "no-checkout"
>  - new name makes cache.h diff align beautifully

Well, on one hand side it is more clear what is checked out, 
and perhaps more common case of checking out only selected files
is simpler.

On the other hand if it was .git/info/no-checkout or checkout-exclude,
or worktree-exclude, empty file or lack of such file would mean full
checkout.  Then core.sparsecheckout (or core.sparsefile) would be
needed only for performance reasons (no need to examine
checkout-exclude file) even if we decide to ship example file with
comments describing its syntax, in templates/info--checkout-exclude

But that is just bikeshedding, don't let it distract you from the real
issue of this series...

-- 
Jakub Narþÿ\x01\x19bski

In related news: there is really no question that bike sheds should be
painted red. Really.  
(Johannes Schindelin, on git mailing list)

Re: [RFC PATCH v4 00/19] Sparse checkout

[Matthieu Moy]

Jakub Narebski <jnareb@gmail.com> writes:

> Content-Type: text/plain; charset=utf-16be
                                    ^^^^^^^^

I don't know what it should have been, but not UTF-16 ;-)

(readable message below)

> Nguyen Thai
>> Welcome to the fourth round of sparse checkout this year, dubbed "The
>> mummy^W^W^W^W^Wno-checkout returns", although the bit now comes under
>> a new name "skip-worktree" [1]. This series has two parts: sparse
>> worktree and sparse checkout. Details will be given below.
>> 
>> For brave users of this series: I have pushed it to my sparse-checkout
>> branch [2] so you can just clone it and test away. Visible changes:
>> 
>>  - the spec is now .git/info/sparse-checkout
>>  - the spec is positive matching, that is it matches what _is_ in
>>    worktree, not what is out of worktree like the last series
>>  - you need to set core.sparsecheckout no additional command argument
>>    is needed.
> [...]
>
>> [1] There are a few reasons behind this rename:
>>  - there is nothing about "checkout" in the first part, it's about
>>    "sparse worktree"
>>  - the double negation issue with "no-" part of "no-checkout"
>>  - new name makes cache.h diff align beautifully
>
> Well, on one hand side it is more clear what is checked out, 
> and perhaps more common case of checking out only selected files
> is simpler.
>
> On the other hand if it was .git/info/no-checkout or checkout-exclude,
> or worktree-exclude, empty file or lack of such file would mean full
> checkout.  Then core.sparsecheckout (or core.sparsefile) would be
> needed only for performance reasons (no need to examine
> checkout-exclude file) even if we decide to ship example file with
> comments describing its syntax, in templates/info--checkout-exclude
>
> But that is just bikeshedding, don't let it distract you from the real
> issue of this series...
>
> -- 
> Jakub Narebski
>
> In related news: there is really no question that bike sheds should be
> painted red. Really.  
> (Johannes Schindelin, on git mailing list)

-- 
Matthieu

Re: [RFC PATCH v4 00/19] Sparse checkout

[Junio C Hamano]

Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:

> The recent assume-unchanged "breakage" that lets Git overwrite
> assume-unchanged files worried me. I sat back, checked and wrote tests

Yeah, it worries me, too.  Does the fix to make sure the sparse stuff
won't be broken apply equally to assume-unchanged?  Does the series have
such fixes to assume-unchanged bit as well?

By the way, I think the first patch in the earlier series, 540e694
(Prevent diff machinery from examining assume-unchanged entries on
worktree, 2009-08-11), is a good change regardless of the sparse
implementation, and I'm inclined to say that we should merge that part
(and I suspect there will be similar fixes to really ignore differences to
CE_VALID entries) first and then queue this new series on top.

Re: [RFC PATCH v4 00/19] Sparse checkout

[Nguyen Thai Ngoc Duy]

2009/8/21 Junio C Hamano <gitster@pobox.com>:
> Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:
>
>> The recent assume-unchanged "breakage" that lets Git overwrite
>> assume-unchanged files worried me. I sat back, checked and wrote tests
>
> Yeah, it worries me, too.  Does the fix to make sure the sparse stuff
> won't be broken apply equally to assume-unchanged?  Does the series have
> such fixes to assume-unchanged bit as well?

This series does not fix assume-unchanged bit. I'd like to focus on
skip-worktree bit now. I still need to write a few more tests for
git-apply, git-checkout... but I think they are safe. It's up to you
to see if changes apply to assume-unchanged bit, in patches 4/19 and
5/19. I don't know if I understand assume-unchanged semantics
correctly anymore :-)

Anyway I think we could put a big fat warning above ce_uptodate()
macro definition, saying that this bit/macro could be faked by
assume-unchanged or skip-worktree bit, so don't rely on that macro
when it comes to writing (at least for skip-worktree part).

Hmm.. _or_ we could make it clear whether it is truly uptodate, or
faked uptodate. Some code path will be updated to only trust "truly
uptodate" bit, which is clearer and easier to grasp than messy logics
like "if (ce_uptodate(ce) && !ce_skip_worktree(ce))". Something like
this as a starting point (for demonstration only, I don't think it
compiles)

diff --git a/cache.h b/cache.h
index a401daf..05fb746 100644
--- a/cache.h
+++ b/cache.h
@@ -179,6 +179,7 @@ struct cache_entry {

 /* Only remove in work directory, not index */
 #define CE_WT_REMOVE (0x400000)
+#define CE_ASSUME_UPTODATE  (0x800000)

 /*
  * Extended on-disk flags
@@ -236,9 +237,11 @@ static inline size_t ce_namelen(const struct
cache_entry *ce)
 			    ondisk_cache_entry_extended_size(ce_namelen(ce)) : \
 			    ondisk_cache_entry_size(ce_namelen(ce)))
 #define ce_stage(ce) ((CE_STAGEMASK & (ce)->ce_flags) >> CE_STAGESHIFT)
-#define ce_uptodate(ce) ((ce)->ce_flags & CE_UPTODATE)
+#define ce_truely_uptodate(ce) ((ce)->ce_flags & CE_UPTODATE)
+#define ce_uptodate(ce) ((ce)->ce_flags & (CE_UPTODATE | CE_ASSUME_UPTODATE))
 #define ce_skip_worktree(ce) ((ce)->ce_flags & CE_SKIP_WORKTREE)
 #define ce_mark_uptodate(ce) ((ce)->ce_flags |= CE_UPTODATE)
+#define ce_mark_assume_uptodate(ce) ((ce)->ce_flags |= CE_ASSUME_UPTODATE)

 #define ce_permissions(mode) (((mode) & 0100) ? 0755 : 0644)
 static inline unsigned int create_ce_mode(unsigned int mode)
diff --git a/read-cache.c b/read-cache.c
index 5ee7d9d..c022955 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1005,7 +1005,7 @@ static struct cache_entry
*refresh_cache_ent(struct index_state *istate,
 		return ce;

 	if (!ignore_valid && ((ce->ce_flags & CE_VALID) || ce_skip_worktree(ce))) {
-		ce_mark_uptodate(ce);
+		ce_mark_assume_uptodate(ce);
 		return ce;
 	}

Still thinking of it. Seems like a good change...

> By the way, I think the first patch in the earlier series, 540e694
> (Prevent diff machinery from examining assume-unchanged entries on
> worktree, 2009-08-11), is a good change regardless of the sparse
> implementation, and I'm inclined to say that we should merge that part
> (and I suspect there will be similar fixes to really ignore differences to
> CE_VALID entries) first and then queue this new series on top.

I have no problem with that.
-- 
Duy

Re: [RFC PATCH v4 04/19] Teach Git to respect skip-worktree bit (reading part)

[Nguyen Thai Ngoc Duy]

2009/8/20 Nguyễn Thái Ngọc Duy <pclouds@gmail.com>:
> grep: turn on --cached for files that is marked skip-worktree
> ls-files: do not check for deleted file that is marked skip-worktree
> update-index: ignore update request if it's skip-worktree, while still allows removing
> diff*: skip worktree version

I kept reminding myself and still managed to miss it. skip-worktree
bit is not completely independent from assume-unchanged. It shares
CE_MATCH_IGNORE_VALID flag in ie_match_stat(). Should have mentioned
that in commit log.
-- 
Duy

Re: [RFC PATCH v4 04/19] Teach Git to respect skip-worktree bit (reading part)

[Junio C Hamano]

Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:

> 2009/8/20 Nguyễn Thái Ngọc Duy <pclouds@gmail.com>:
>> grep: turn on --cached for files that is marked skip-worktree
>> ls-files: do not check for deleted file that is marked skip-worktree
>> update-index: ignore update request if it's skip-worktree, while still allows removing
>> diff*: skip worktree version
>
> I kept reminding myself and still managed to miss it. skip-worktree
> bit is not completely independent from assume-unchanged. It shares
> CE_MATCH_IGNORE_VALID flag in ie_match_stat(). Should have mentioned
> that in commit log.

Yeah, that is why I feel uneasy reading your response to my other message:

  This series does not fix assume-unchanged bit. I'd like to focus on
  skip-worktree bit now. ... I don't know if I understand assume-unchanged
  semantics correctly anymore :-)

Re: [RFC PATCH v4 04/19] Teach Git to respect skip-worktree bit (reading part)

[Nguyen Thai Ngoc Duy]

On Sat, Aug 22, 2009 at 12:32 AM, Junio C Hamano<gitster@pobox.com> wrote:
> Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:
>
>> 2009/8/20 Nguyễn Thái Ngọc Duy <pclouds@gmail.com>:
>>> grep: turn on --cached for files that is marked skip-worktree
>>> ls-files: do not check for deleted file that is marked skip-worktree
>>> update-index: ignore update request if it's skip-worktree, while still allows removing
>>> diff*: skip worktree version
>>
>> I kept reminding myself and still managed to miss it. skip-worktree
>> bit is not completely independent from assume-unchanged. It shares
>> CE_MATCH_IGNORE_VALID flag in ie_match_stat(). Should have mentioned
>> that in commit log.

I should have just renamed it to CE_MATCH_PARANOID. That would fit
both assume-unchanged and skip-worktree bits.

> Yeah, that is why I feel uneasy reading your response to my other message:
>
>  This series does not fix assume-unchanged bit. I'd like to focus on
>  skip-worktree bit now. ... I don't know if I understand assume-unchanged
>  semantics correctly anymore :-)

The thing is there are grey areas where assume-unchanged is used. If
worktree version is there and really uptodate, nothing to worry about.
We seemed to care of local changes in assume-unchanged files, but
that's not written in stone. So I started off with a new bit, clearer
semantics. Perhaps new changes could be applied back to
assume-unchanged bit, but I'd stay away from that for now.
-- 
Duy