Re: [PATCH] Provide config option to expect files outside sparse patterns

From: Elijah Newren <newren@gmail.com>
To: Derrick Stolee <derrickstolee@github.com>
Cc: Elijah Newren via GitGitGadget <gitgitgadget@gmail.com>,
	Git Mailing List <git@vger.kernel.org>,
	Jonathan Nieder <jrnieder@gmail.com>,
	Jonathan Tan <jonathantanmy@google.com>,
	Jose Lopes <jabolopes@google.com>,
	Jeff Hostetler <jeffhostetler@github.com>
Subject: Re: [PATCH] Provide config option to expect files outside sparse patterns
Date: Mon, 21 Feb 2022 18:17:21 -0800	[thread overview]
Message-ID: <CABPp-BGz0Y+Gk0uLy7K-pqBY955=rN-E2SJFofDaje60b-XZuw@mail.gmail.com> (raw)
In-Reply-To: <54a0aa74-57c2-ee65-ae07-cb1b0daf947f@github.com>

On Sun, Feb 20, 2022 at 11:41 AM Derrick Stolee
<derrickstolee@github.com> wrote:
>
> On 2/20/2022 12:05 AM, Elijah Newren via GitGitGadget wrote:
> > From: Elijah Newren <newren@gmail.com>
> >
> > Typically with sparse checkouts, we expect files outside the sparsity
> > patterns to be marked as SKIP_WORKTREE and be missing from the working
> > tree.  VFS for Git can be used to turn this expectation on its head:
> > all files are considered present in the working copy, though they are
> > not vivified until first access access.  With VFS for Git, most of the
> > files do not match the sparsity patterns at first, and the VFS layer
> > automatically updates the sparsity patterns to add more files whenever
> > files are written.
> >
> > With this background, this special usecase does not play well with the
> > safety check we added in commit 11d46a399d ("repo_read_index: clear
> > SKIP_WORKTREE bit from files present in worktree", 2022-01-06).
> > Checking SKIP_WORKTREE files to see if they are present in the working
> > tree causes them all to be immediately vivified.  Further, the special
> > VFS layer, by virtue of automatically updating the sparsity patterns and
> > catching all accesses, isn't in need of that safety check either.
> > Provide a configuration option, core.expectFilesOutsideSparsePatterns
> > so that those with this special usecase can turn off the safety check.
>
> This patch looks like a good solution to the concerns brought up by
> Jonathan N. around vfsd. VFS for Git uses the microsoft/git fork with
> its own custom config to protect things like this. I imagine that we
> will start setting your core_expect_files_outside_sparse_patterns
> variable when reading the virtual filesystem info. We might even modify
> some of our custom checks to use this variable instead. That would make
> them appropriate to send upstream.
>
> Should we update Documentation/config/core.txt describing this config
> key? Or is this intended to be an internal detail only for something
> like vfsd?

I think it's probably only for vfsd, but vfsd users probably deserve
to be able to look it up in the documentation, so it's probably worth
documenting.   Once we figure out it's name and section, that is (as
per Dscho's email -- comments from you on that would be appreciated).

> The only concern here really is if we want to be picky about the "VFS
> for Git" references instead of "vfsd" references in the commit message.

I'm not sure I'm understanding the distinction.  Was "VFS for Git"
renamed to "vfsd"?  Is "vfsd" Google's modified version?  Something
else?

Also, vfsd doesn't seem to be Google-able whereas "VFS for Git" is,
and since it's not an internal git project, it might be nice to use a
name that users can find.

But anyway, I have no idea what the "correct" text here is (as
probably evidenced by my questions), so if anyone wants to provide
suggestions or corrections to the commit message, I'm happy to take
them.