From: Victoria Dye <vdye@github.com>
To: Elijah Newren via GitGitGadget <gitgitgadget@gmail.com>,
git@vger.kernel.org
Cc: Derrick Stolee <stolee@gmail.com>,
Lessley Dennington <lessleydennington@gmail.com>,
Elijah Newren <newren@gmail.com>
Subject: Re: [PATCH v2 0/5] Remove the present-despite-SKIP_WORKTREE class of bugs (for sparse-checkouts)
Date: Fri, 14 Jan 2022 20:51:03 -0500 [thread overview]
Message-ID: <14b6dbeb-5a12-3ac0-9a0e-9cf2c3a381fe@github.com> (raw)
In-Reply-To: <pull.1114.v2.git.1642175983.gitgitgadget@gmail.com>
Elijah Newren via GitGitGadget wrote:
> (Maintainer note: This series builds on (v2 of) vd/sparse-clean-etc, because
> it tweaks one of the testcases added there.)
>
> (Note 2: There was a previous RFC round of this series at
> https://lore.kernel.org/git/20220109045732.2497526-1-newren@gmail.com/.)
>
> Files in the present-despite-SKIP_WORKTREE state have caused no ends of
> discussions and bugs[1,2,3,4,5,6,...and lots of others]. Trying to address
> the big issue of discovering & recovering from this state has befuddled me
> for over a year because I was worried we'd need additional code at every
> skip_worktree-checking path in the code (and they are all over the place),
> and that we'd make the code significantly slower unless we plumbed a bunch
> of additional information all over the place to allow some reasonable
> optimizations.
>
> This series tries to solve the problem a bit differently by automatic early
> discovery and recovery; as a result, it greatly simplifies the landscape,
> reduces our testing matrix burden, and fixes a large swath of bugs. And I
> figured out how to get the perf cost down to essentially negligible.
>
> Changes since v1 (or v2 if you count RFC as v1):
>
> * now includes some fixes for testcases from
> ds/fetch-pull-with-sparse-index (which topic marked its own new tests as
> potentially suboptimal; with my series, the sparse behavior now matches
> the full tree behavior on that test. Wahoo!). Note that Junio's version
> of vd/sparse-clean-etc already includes ds/fetch-pull-with-sparse-index,
> so no need to merge in anything extra.
>
> Changes since RFC version:
>
> * updated the commit messages as per suggestions from Victoria, including
> adding performance measurements
> * renamed the new function to use a clearer name
> * replaced the final patch with a different optimization, which is both
> simpler and performs quite a bit better (the cost for my previous patch 5
> was already decent in many cases, but had a few cases where the cost was
> significant).
>
> Quick overview:
>
> * Patches 1 & 2 add a test to demonstrate accidental deletion of
> possibly-modified files, and then fix the bug.
> * Patch 3 is the crux of this series; a small amount of code with a huge
> commit message
> * Patch 4 updates the documentation
> * Patch 5 adds an optimization to reduce the performance impact of patch 3
>
Betwee the RFC and this version, you've made the changes I was looking for
and answered any remaining questions I had. This series (including patches
1, 2, and 4 - I didn't have anything substantial to say/ask/add to those)
looks good to me.
Thank you for working on this!
> [1]
> https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
> [2]
> https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/
> [3]
> https://lore.kernel.org/git/pull.809.git.git.1592356884310.gitgitgadget@gmail.com/
> [4] commit 66b209b ("merge-ort: implement CE_SKIP_WORKTREE handling with
> conflicted entries", 2021-03-20) [5] commit ba359fd ("stash: fix stash
> application in sparse-checkouts", 2020-12-01) [6]
> https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
>
> Elijah Newren (5):
> t1011: add testcase demonstrating accidental loss of user
> modifications
> unpack-trees: fix accidental loss of user changes
> repo_read_index: clear SKIP_WORKTREE bit from files present in
> worktree
> Update documentation related to sparsity and the skip-worktree bit
> Accelerate clear_skip_worktree_from_present_files() by caching
>
> Documentation/git-read-tree.txt | 12 +++-
> Documentation/git-sparse-checkout.txt | 76 ++++++++++++++----------
> Documentation/git-update-index.txt | 57 +++++++++++++-----
> repository.c | 7 +++
> sparse-index.c | 73 +++++++++++++++++++++++
> sparse-index.h | 1 +
> t/t1011-read-tree-sparse-checkout.sh | 23 ++++++-
> t/t1092-sparse-checkout-compatibility.sh | 41 ++++++-------
> t/t3705-add-sparse-checkout.sh | 2 +
> t/t6428-merge-conflicts-sparse.sh | 23 ++-----
> t/t7012-skip-worktree-writing.sh | 44 +++-----------
> t/t7817-grep-sparse-checkout.sh | 11 +++-
> unpack-trees.c | 4 +-
> 13 files changed, 246 insertions(+), 128 deletions(-)
>
>
> base-commit: 48609de3bf32befb69c40c1a2595a98dac0448b4
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1114%2Fnewren%2Ffix-present-despite-skip-worktree-v2
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1114/newren/fix-present-despite-skip-worktree-v2
> Pull-Request: https://github.com/gitgitgadget/git/pull/1114
>
> Range-diff vs v1:
>
> 1: c553d558c2f = 1: d50d804af4e t1011: add testcase demonstrating accidental loss of user modifications
> 2: 1e3958576e2 = 2: 206c638fa90 unpack-trees: fix accidental loss of user changes
> 3: b263cc75b7d ! 3: 11d46a399d2 repo_read_index: clear SKIP_WORKTREE bit from files present in worktree
> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'update-index modi
>
> # When skip-worktree is disabled (even on files outside sparse cone), file
> # is updated in the index
> +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'ls-files' '
> + test_cmp dense sparse &&
> +
> + # Set up a strange condition of having a file edit
> +- # outside of the sparse-checkout cone. This is just
> +- # to verify that sparse-checkout and sparse-index
> +- # behave the same in this case.
> ++ # outside of the sparse-checkout cone. We want to verify
> ++ # that all modes handle this the same, and detect the
> ++ # modification.
> + write_script edit-content <<-\EOF &&
> +- mkdir folder1 &&
> ++ mkdir -p folder1 &&
> + echo content >>folder1/a
> + EOF
> +- run_on_sparse ../edit-content &&
> ++ run_on_all ../edit-content &&
> +
> +- # ls-files does not currently notice modified files whose
> +- # cache entries are marked SKIP_WORKTREE. This may change
> +- # in the future, but here we test that sparse index does
> +- # not accidentally create a change of behavior.
> +- test_sparse_match git ls-files --modified &&
> +- test_must_be_empty sparse-checkout-out &&
> +- test_must_be_empty sparse-index-out &&
> ++ test_all_match git ls-files --modified &&
> +
> + git -C sparse-index ls-files --sparse --modified >sparse-index-out &&
> +- test_must_be_empty sparse-index-out &&
> ++ cat >expect <<-\EOF &&
> ++ folder1/a
> ++ EOF
> ++ test_cmp expect sparse-index-out &&
> +
> + # Add folder1 to the sparse-checkout cone and
> + # check that ls-files shows the expanded files.
> + test_sparse_match git sparse-checkout add folder1 &&
> +- test_sparse_match git ls-files --modified &&
> ++ test_all_match git ls-files --modified &&
> +
> + test_all_match git ls-files &&
> + git -C sparse-index ls-files --sparse >actual &&
>
> ## t/t3705-add-sparse-checkout.sh ##
> @@ t/t3705-add-sparse-checkout.sh: setup_sparse_entry () {
> 4: c74ad19616e = 4: 0af00779128 Update documentation related to sparsity and the skip-worktree bit
> 5: e68028ebe0a = 5: 05ac964e630 Accelerate clear_skip_worktree_from_present_files() by caching
>
prev parent reply other threads:[~2022-01-15 1:51 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-13 16:43 [PATCH 0/5] Remove the present-despite-SKIP_WORKTREE class of bugs (for sparse-checkouts) Elijah Newren via GitGitGadget
2022-01-13 16:43 ` [PATCH 1/5] t1011: add testcase demonstrating accidental loss of user modifications Elijah Newren via GitGitGadget
2022-01-13 16:43 ` [PATCH 2/5] unpack-trees: fix accidental loss of user changes Elijah Newren via GitGitGadget
2022-01-13 16:43 ` [PATCH 3/5] repo_read_index: clear SKIP_WORKTREE bit from files present in worktree Elijah Newren via GitGitGadget
2022-01-13 16:43 ` [PATCH 4/5] Update documentation related to sparsity and the skip-worktree bit Elijah Newren via GitGitGadget
2022-01-13 16:43 ` [PATCH 5/5] Accelerate clear_skip_worktree_from_present_files() by caching Elijah Newren via GitGitGadget
2022-01-13 23:35 ` Elijah Newren
2022-01-14 15:59 ` [PATCH v2 0/5] Remove the present-despite-SKIP_WORKTREE class of bugs (for sparse-checkouts) Elijah Newren via GitGitGadget
2022-01-14 15:59 ` [PATCH v2 1/5] t1011: add testcase demonstrating accidental loss of user modifications Elijah Newren via GitGitGadget
2022-02-16 8:51 ` Ævar Arnfjörð Bjarmason
2022-02-16 16:02 ` Elijah Newren
2022-01-14 15:59 ` [PATCH v2 2/5] unpack-trees: fix accidental loss of user changes Elijah Newren via GitGitGadget
2022-01-14 15:59 ` [PATCH v2 3/5] repo_read_index: clear SKIP_WORKTREE bit from files present in worktree Elijah Newren via GitGitGadget
2022-02-16 8:57 ` Ævar Arnfjörð Bjarmason
2022-02-16 16:08 ` Elijah Newren
2022-02-19 1:06 ` Jonathan Nieder
2022-02-19 16:42 ` Elijah Newren
2022-02-19 18:14 ` Jonathan Nieder
2022-02-20 5:28 ` Elijah Newren
2022-02-20 16:56 ` Derrick Stolee
2022-02-22 23:17 ` Jonathan Nieder
2022-01-14 15:59 ` [PATCH v2 4/5] Update documentation related to sparsity and the skip-worktree bit Elijah Newren via GitGitGadget
2022-02-16 9:15 ` Ævar Arnfjörð Bjarmason
2022-02-16 16:21 ` Elijah Newren
2022-01-14 15:59 ` [PATCH v2 5/5] Accelerate clear_skip_worktree_from_present_files() by caching Elijah Newren via GitGitGadget
2022-01-15 1:39 ` Victoria Dye
2022-02-16 9:32 ` Ævar Arnfjörð Bjarmason
2022-02-16 16:30 ` Elijah Newren
2022-02-17 4:40 ` Elijah Newren
2022-01-15 1:51 ` Victoria Dye [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=14b6dbeb-5a12-3ac0-9a0e-9cf2c3a381fe@github.com \
--to=vdye@github.com \
--cc=git@vger.kernel.org \
--cc=gitgitgadget@gmail.com \
--cc=lessleydennington@gmail.com \
--cc=newren@gmail.com \
--cc=stolee@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).