Re: [PATCH 19/19] hack: watchman/untracked cache mashup

From: Duy Nguyen <pclouds@gmail.com>
To: David Turner <dturner@twopensource.com>
Cc: Git Mailing List <git@vger.kernel.org>
Subject: Re: [PATCH 19/19] hack: watchman/untracked cache mashup
Date: Thu, 17 Mar 2016 20:06:14 +0700	[thread overview]
Message-ID: <CACsJy8Cx7LnPC1+17qov_pJK9OMpYtdOzS_uTBh5Fnr=A8z_OQ@mail.gmail.com> (raw)
In-Reply-To: <1458176165.9385.9.camel@twopensource.com>

On Thu, Mar 17, 2016 at 7:56 AM, David Turner <dturner@twopensource.com> wrote:
>> So if we detect an updated file that's not in the index, we are
>> prepared to invalidate that path, correct? We may invalidate more
>> than
>> necessary if that's true. Imagine a.o is already ignored. If it's
>> rebuilt, we should not need to update untracked cache.
>
> Yes, that's true.  But it would be true with the mtime system too. This
> is no worse, even if it's no better.  In-tree builds are a hard case to
> support, and I'm totally OK with a system that encourages out-of-tree
> builds.
>
> We could check if it's ignored, but then if someone changes their
> gitignore, we could be wrong.
>
> Or we could suggest that people make their watchmanignore match their
> gitignore.

So your purpose is to reduce stat() on those "quiet" directories?
Putting it that way, yes it's reasonable to do this way. I want to "do
better" but we can do in steps and the first step "no worse, no
better" is already good.

>> What I had in mind (and argued with watchman devs a bit [1]) was
>> maintain the file list of each clock and compare the file list of
>> different clocks to figure out what files are added or deleted. Then
>> we can generate invalidation list more accurately. We need to keep at
>> least one file list corresponds to  the clock saved in the index.
>> When
>> we get the refresh request, we get a new file list (with new clock),
>> do a diff then save the invalidation list as an extension. Once we
>> notice that new clock is written back in index, we can discard older
>> file lists. In theory we should not need to keep too many file lists,
>> so even if one list is big, it should not be a big problem.
>>
>> I have a note with me about race conditions with this approach, but I
>> haven't remembered exactly why or how yet.. My recent thoughts about
>> it though, are race conditions when you do "git status" is probably
>> tolerable. After all if you're doing some I/O when you do git-status,
>> you're guaranteed to miss some updates.
>>
>> [1] https://github.com/facebook/watchman/issues/65
>
> I think it would be possible to just check the UNTR extension and only
> add a dir to it if that dir doesn't already contain (untracked) the
> file that's being modified.  Or is that also racy?

Sorry can't comment on this racy thing yet. Still haven't sat down,
redrawn the timeline and retraced what I thought months ago :)
-- 
Duy