Git Mailing List Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH 00/10] Sparse-index: integrate with status and add
@ 2021-04-13 14:01 Derrick Stolee via GitGitGadget
  2021-04-13 14:01 ` [PATCH 01/10] t1092: add tests for status/add and sparse files Derrick Stolee via GitGitGadget
                   ` (11 more replies)
  0 siblings, 12 replies; 127+ messages in thread
From: Derrick Stolee via GitGitGadget @ 2021-04-13 14:01 UTC (permalink / raw)
  To: git; +Cc: gitster, newren, Derrick Stolee

This is the first "payoff" series in the sparse-index work. It makes 'git
status' and 'git add' very fast when a sparse-index is enabled on a
repository with cone-mode sparse-checkout (and a small populated set).

This is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.
The latter branch is needed because it changes the behavior of 'git add'
around sparse entries, which changes the expectations of a test added in
patch 1.

The approach here is to audit the places where ensure_full_index() pops up
while doing normal commands with pathspecs within the sparse-checkout
definition. Each of these are checked and tested. In the end, the
sparse-index is integrated with these features:

 * git status
 * git add -A
 * git add . (and other pathspecs)
 * FS Monitor index extension.

The performance tests in p2000-sparse-operations.sh improve by 95% or more,
even when compared with the full-index cases, not just the sparse-index
cases that previously had extra overhead.

Hopefully this is the first example of how ds/sparse-index-protections has
done the basic work to do these conversions safely, making them look easier
than they seemed when starting this adventure.

Thanks, -Stolee

Derrick Stolee (10):
  t1092: add tests for status/add and sparse files
  unpack-trees: make sparse aware
  dir.c: accept a directory as part of cone-mode patterns
  status: skip sparse-checkout percentage with sparse-index
  status: use sparse-index throughout
  dir: use expand_to_path() for sparse directories
  add: allow operating on a sparse-only index
  pathspec: stop calling ensure_full_index
  t7519: add sparse directories to FS monitor tests
  fsmonitor: test with sparse index

 builtin/add.c                            |  3 +
 builtin/commit.c                         |  3 +
 dir.c                                    |  5 ++
 dir.h                                    |  2 +-
 pathspec.c                               |  2 -
 preload-index.c                          |  2 +
 read-cache.c                             |  5 +-
 t/t1092-sparse-checkout-compatibility.sh | 73 +++++++++++++++++++++++-
 t/t7519-status-fsmonitor.sh              | 65 +++++++++++++++++++++
 unpack-trees.c                           | 24 +++++++-
 wt-status.c                              | 14 ++++-
 wt-status.h                              |  1 +
 12 files changed, 186 insertions(+), 13 deletions(-)


base-commit: f723f370c89ad61f4f40aabfd3540b1ce19c00e5
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-932%2Fderrickstolee%2Fsparse-index%2Fstatus-and-add-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-932/derrickstolee/sparse-index/status-and-add-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/932
-- 
gitgitgadget

^ permalink raw reply	[flat|nested] 127+ messages in thread

end of thread, back to index

Thread overview: 127+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-13 14:01 [PATCH 00/10] Sparse-index: integrate with status and add Derrick Stolee via GitGitGadget
2021-04-13 14:01 ` [PATCH 01/10] t1092: add tests for status/add and sparse files Derrick Stolee via GitGitGadget
2021-04-20 21:52   ` Elijah Newren
2021-04-21 13:21     ` Derrick Stolee
2021-04-21 15:14   ` Matheus Tavares Bernardino
2021-04-23 20:12     ` Derrick Stolee
2021-04-13 14:01 ` [PATCH 02/10] unpack-trees: make sparse aware Derrick Stolee via GitGitGadget
2021-04-20 23:00   ` Elijah Newren
2021-04-21 13:41     ` Derrick Stolee
2021-04-21 16:11       ` Elijah Newren
2021-04-22  2:24         ` Matheus Tavares Bernardino
2021-04-21 17:27     ` Derrick Stolee
2021-04-21 18:55       ` Matheus Tavares Bernardino
2021-04-21 19:10         ` Elijah Newren
2021-04-21 19:51           ` Matheus Tavares Bernardino
2021-04-21 18:56       ` Elijah Newren
2021-04-23 20:16         ` Derrick Stolee
2021-04-13 14:01 ` [PATCH 03/10] dir.c: accept a directory as part of cone-mode patterns Derrick Stolee via GitGitGadget
2021-04-20 23:21   ` Elijah Newren
2021-04-21 13:47     ` Derrick Stolee
2021-04-13 14:01 ` [PATCH 04/10] status: skip sparse-checkout percentage with sparse-index Derrick Stolee via GitGitGadget
2021-04-20 23:26   ` Elijah Newren
2021-04-21 13:51     ` Derrick Stolee
2021-04-13 14:01 ` [PATCH 05/10] status: use sparse-index throughout Derrick Stolee via GitGitGadget
2021-04-21  0:44   ` Elijah Newren
2021-04-21 13:55     ` Derrick Stolee
2021-04-13 14:01 ` [PATCH 06/10] dir: use expand_to_path() for sparse directories Derrick Stolee via GitGitGadget
2021-04-21  0:52   ` Elijah Newren
2021-04-21  0:53     ` Elijah Newren
2021-04-21 14:03       ` Derrick Stolee
2021-04-13 14:01 ` [PATCH 07/10] add: allow operating on a sparse-only index Derrick Stolee via GitGitGadget
2021-04-13 14:01 ` [PATCH 08/10] pathspec: stop calling ensure_full_index Derrick Stolee via GitGitGadget
2021-04-21  0:57   ` Elijah Newren
2021-04-13 14:01 ` [PATCH 09/10] t7519: add sparse directories to FS monitor tests Derrick Stolee via GitGitGadget
2021-04-13 14:01 ` [PATCH 10/10] fsmonitor: test with sparse index Derrick Stolee via GitGitGadget
2021-04-21  7:00   ` Elijah Newren
2021-04-13 20:45 ` [PATCH 00/10] Sparse-index: integrate with status and add Matheus Tavares Bernardino
2021-04-14 16:31   ` Derrick Stolee
2021-04-23 21:34 ` [PATCH v2 0/8] Sparse-index: integrate with status Derrick Stolee via GitGitGadget
2021-04-23 21:34   ` [PATCH v2 1/8] t1092: add tests for status/add and sparse files Derrick Stolee via GitGitGadget
2021-05-13 12:40     ` Matheus Tavares Bernardino
2021-05-14 12:27       ` Derrick Stolee
2021-04-23 21:34   ` [PATCH v2 2/8] unpack-trees: preserve cache_bottom Derrick Stolee via GitGitGadget
2021-04-23 21:34   ` [PATCH v2 3/8] unpack-trees: compare sparse directories correctly Derrick Stolee via GitGitGadget
2021-05-13  3:26     ` Elijah Newren
2021-04-23 21:34   ` [PATCH v2 4/8] unpack-trees: stop recursing into sparse directories Derrick Stolee via GitGitGadget
2021-05-13  3:31     ` Elijah Newren
2021-04-23 21:34   ` [PATCH v2 5/8] dir.c: accept a directory as part of cone-mode patterns Derrick Stolee via GitGitGadget
2021-04-23 21:34   ` [PATCH v2 6/8] status: skip sparse-checkout percentage with sparse-index Derrick Stolee via GitGitGadget
2021-04-23 21:34   ` [PATCH v2 7/8] status: use sparse-index throughout Derrick Stolee via GitGitGadget
2021-04-23 21:34   ` [PATCH v2 8/8] fsmonitor: test with sparse index Derrick Stolee via GitGitGadget
2021-05-13  4:12   ` [PATCH v2 0/8] Sparse-index: integrate with status Elijah Newren
2021-05-14 18:28     ` Derrick Stolee
2021-05-14 18:30   ` [PATCH v3 00/12] " Derrick Stolee via GitGitGadget
2021-05-14 18:31     ` [PATCH v3 01/12] sparse-index: skip indexes with unmerged entries Derrick Stolee via GitGitGadget
2021-05-14 18:31     ` [PATCH v3 02/12] sparse-index: include EXTENDED flag when expanding Derrick Stolee via GitGitGadget
2021-05-18  1:33       ` Elijah Newren
2021-05-18 14:57         ` Derrick Stolee
2021-05-18 17:48           ` Elijah Newren
2021-05-18 18:16             ` Derrick Stolee
2021-05-14 18:31     ` [PATCH v3 03/12] t1092: expand repository data shape Derrick Stolee via GitGitGadget
2021-05-18  1:49       ` Elijah Newren
2021-05-18 14:59         ` Derrick Stolee
2021-05-14 18:31     ` [PATCH v3 04/12] t1092: add tests for status/add and sparse files Derrick Stolee via GitGitGadget
2021-05-14 18:31     ` [PATCH v3 05/12] unpack-trees: preserve cache_bottom Derrick Stolee via GitGitGadget
2021-05-14 18:31     ` [PATCH v3 06/12] unpack-trees: compare sparse directories correctly Derrick Stolee via GitGitGadget
2021-05-14 18:31     ` [PATCH v3 07/12] unpack-trees: stop recursing into sparse directories Derrick Stolee via GitGitGadget
2021-05-18  2:03       ` Elijah Newren
2021-05-18  2:06         ` Elijah Newren
2021-05-18 19:20           ` Derrick Stolee
2021-05-14 18:31     ` [PATCH v3 08/12] dir.c: accept a directory as part of cone-mode patterns Derrick Stolee via GitGitGadget
2021-05-14 18:31     ` [PATCH v3 09/12] status: skip sparse-checkout percentage with sparse-index Derrick Stolee via GitGitGadget
2021-05-14 18:31     ` [PATCH v3 10/12] status: use sparse-index throughout Derrick Stolee via GitGitGadget
2021-05-14 18:31     ` [PATCH v3 11/12] wt-status: expand added sparse directory entries Derrick Stolee via GitGitGadget
2021-05-18  2:27       ` Elijah Newren
2021-05-18 18:26         ` Derrick Stolee
2021-05-18 19:04           ` Derrick Stolee
2021-05-19  8:38             ` Elijah Newren
2021-05-14 18:31     ` [PATCH v3 12/12] fsmonitor: integrate with sparse index Derrick Stolee via GitGitGadget
2021-05-21 11:59     ` [PATCH v4 00/12] Sparse-index: integrate with status Derrick Stolee via GitGitGadget
2021-05-21 11:59       ` [PATCH v4 01/12] sparse-index: skip indexes with unmerged entries Derrick Stolee via GitGitGadget
2021-05-21 11:59       ` [PATCH v4 02/12] sparse-index: include EXTENDED flag when expanding Derrick Stolee via GitGitGadget
2021-05-21 11:59       ` [PATCH v4 03/12] t1092: expand repository data shape Derrick Stolee via GitGitGadget
2021-05-21 11:59       ` [PATCH v4 04/12] t1092: add tests for status/add and sparse files Derrick Stolee via GitGitGadget
2021-05-21 11:59       ` [PATCH v4 05/12] unpack-trees: preserve cache_bottom Derrick Stolee via GitGitGadget
2021-05-21 11:59       ` [PATCH v4 06/12] unpack-trees: compare sparse directories correctly Derrick Stolee via GitGitGadget
2021-05-21 11:59       ` [PATCH v4 07/12] unpack-trees: be careful around sparse directory entries Derrick Stolee via GitGitGadget
2021-05-28 11:36         ` Derrick Stolee
2021-05-21 11:59       ` [PATCH v4 08/12] dir.c: accept a directory as part of cone-mode patterns Derrick Stolee via GitGitGadget
2021-05-21 11:59       ` [PATCH v4 09/12] status: skip sparse-checkout percentage with sparse-index Derrick Stolee via GitGitGadget
2021-05-21 11:59       ` [PATCH v4 10/12] status: use sparse-index throughout Derrick Stolee via GitGitGadget
2021-05-21 11:59       ` [PATCH v4 11/12] wt-status: expand added sparse directory entries Derrick Stolee via GitGitGadget
2021-05-21 11:59       ` [PATCH v4 12/12] fsmonitor: integrate with sparse index Derrick Stolee via GitGitGadget
2021-06-07 12:33       ` [PATCH v5 00/14] Sparse-index: integrate with status Derrick Stolee via GitGitGadget
2021-06-07 12:33         ` [PATCH v5 01/14] sparse-index: skip indexes with unmerged entries Derrick Stolee via GitGitGadget
2021-06-07 12:34         ` [PATCH v5 02/14] sparse-index: include EXTENDED flag when expanding Derrick Stolee via GitGitGadget
2021-06-08 18:56           ` Elijah Newren
2021-06-09 17:39             ` Derrick Stolee
2021-06-09 18:11               ` Elijah Newren
2021-06-07 12:34         ` [PATCH v5 03/14] t1092: replace incorrect 'echo' with 'cat' Derrick Stolee via GitGitGadget
2021-06-08 19:18           ` Elijah Newren
2021-06-07 12:34         ` [PATCH v5 04/14] t1092: expand repository data shape Derrick Stolee via GitGitGadget
2021-06-07 12:34         ` [PATCH v5 05/14] t1092: add tests for status/add and sparse files Derrick Stolee via GitGitGadget
2021-06-07 12:34         ` [PATCH v5 06/14] unpack-trees: preserve cache_bottom Derrick Stolee via GitGitGadget
2021-06-07 12:34         ` [PATCH v5 07/14] unpack-trees: compare sparse directories correctly Derrick Stolee via GitGitGadget
2021-06-07 12:34         ` [PATCH v5 08/14] unpack-trees: unpack sparse directory entries Derrick Stolee via GitGitGadget
2021-06-09  3:48           ` Elijah Newren
2021-06-09 20:21             ` Derrick Stolee
2021-06-07 12:34         ` [PATCH v5 09/14] dir.c: accept a directory as part of cone-mode patterns Derrick Stolee via GitGitGadget
2021-06-07 12:34         ` [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs Derrick Stolee via GitGitGadget
2021-06-07 15:26           ` Derrick Stolee
2021-06-08  1:05             ` Junio C Hamano
2021-06-08 13:00               ` Derrick Stolee
2021-06-09  5:47           ` Elijah Newren
2021-06-09  6:32             ` Junio C Hamano
2021-06-09  8:11               ` Elijah Newren
2021-06-09 20:33                 ` Derrick Stolee
2021-06-10 17:45                   ` Derrick Stolee
2021-06-10 21:31                     ` Elijah Newren
2021-06-11 12:57                       ` Derrick Stolee
2021-06-11 17:27                         ` Derrick Stolee
2021-06-07 12:34         ` [PATCH v5 11/14] status: skip sparse-checkout percentage with sparse-index Derrick Stolee via GitGitGadget
2021-06-07 12:34         ` [PATCH v5 12/14] status: use sparse-index throughout Derrick Stolee via GitGitGadget
2021-06-07 12:34         ` [PATCH v5 13/14] wt-status: expand added sparse directory entries Derrick Stolee via GitGitGadget
2021-06-09  5:27           ` Elijah Newren
2021-06-09 20:49             ` Derrick Stolee
2021-06-07 12:34         ` [PATCH v5 14/14] fsmonitor: integrate with sparse index Derrick Stolee via GitGitGadget

Git Mailing List Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/git/0 git/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 git git/ https://lore.kernel.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.git


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git