* [PATCH v3 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman
2017-10-27 23:26 ` [PATCH v3 1/4] fsmonitor: Set the PWD to the top of the working tree Alex Vandiver
@ 2017-10-27 23:26 ` Alex Vandiver
2017-10-27 23:26 ` [PATCH v3 3/4] fsmonitor: Document GIT_TRACE_FSMONITOR Alex Vandiver
2017-10-27 23:26 ` [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged Alex Vandiver
2 siblings, 0 replies; 10+ messages in thread
From: Alex Vandiver @ 2017-10-27 23:26 UTC (permalink / raw)
To: git; +Cc: Johannes Schindelin, Ben Peart
This provides modest performance savings. Benchmarking with the
following program, with and without `--no-pretty`, we find savings of
23% (0.316s -> 0.242s) in the git repository, and savings of 8% (5.24s
-> 4.86s) on a large repository with 580k files in the working copy.
#!/usr/bin/perl
use strict;
use warnings;
use IPC::Open2;
use JSON::XS;
my $pid = open2(\*CHLD_OUT, \*CHLD_IN, "watchman -j @ARGV")
or die "open2() failed: $!\n" .
"Falling back to scanning...\n";
my $query = qq|["query", "$ENV{PWD}", {}]|;
print CHLD_IN $query;
close CHLD_IN;
my $response = do {local $/; <CHLD_OUT>};
JSON::XS->new->utf8->decode($response);
Signed-off-by: Alex Vandiver <alexmv@dropbox.com>
---
templates/hooks--fsmonitor-watchman.sample | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/templates/hooks--fsmonitor-watchman.sample b/templates/hooks--fsmonitor-watchman.sample
index 9eba8a740..9a082f278 100755
--- a/templates/hooks--fsmonitor-watchman.sample
+++ b/templates/hooks--fsmonitor-watchman.sample
@@ -49,7 +49,7 @@ launch_watchman();
sub launch_watchman {
- my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j')
+ my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty')
or die "open2() failed: $!\n" .
"Falling back to scanning...\n";
--
2.15.0.rc1.413.g76aedb451
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v3 3/4] fsmonitor: Document GIT_TRACE_FSMONITOR
2017-10-27 23:26 ` [PATCH v3 1/4] fsmonitor: Set the PWD to the top of the working tree Alex Vandiver
2017-10-27 23:26 ` [PATCH v3 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman Alex Vandiver
@ 2017-10-27 23:26 ` Alex Vandiver
2017-10-27 23:26 ` [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged Alex Vandiver
2 siblings, 0 replies; 10+ messages in thread
From: Alex Vandiver @ 2017-10-27 23:26 UTC (permalink / raw)
To: git; +Cc: Johannes Schindelin, Ben Peart
Signed-off-by: Alex Vandiver <alexmv@dropbox.com>
---
Documentation/git.txt | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/Documentation/git.txt b/Documentation/git.txt
index 1fca63634..720db196e 100644
--- a/Documentation/git.txt
+++ b/Documentation/git.txt
@@ -594,6 +594,10 @@ into it.
Unsetting the variable, or setting it to empty, "0" or
"false" (case insensitive) disables trace messages.
+`GIT_TRACE_FSMONITOR`::
+ Enables trace messages for the filesystem monitor extension.
+ See `GIT_TRACE` for available trace output options.
+
`GIT_TRACE_PACK_ACCESS`::
Enables trace messages for all accesses to any packs. For each
access, the pack file name and an offset in the pack is
--
2.15.0.rc1.413.g76aedb451
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged
2017-10-27 23:26 ` [PATCH v3 1/4] fsmonitor: Set the PWD to the top of the working tree Alex Vandiver
2017-10-27 23:26 ` [PATCH v3 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman Alex Vandiver
2017-10-27 23:26 ` [PATCH v3 3/4] fsmonitor: Document GIT_TRACE_FSMONITOR Alex Vandiver
@ 2017-10-27 23:26 ` Alex Vandiver
2017-10-31 5:24 ` Junio C Hamano
2 siblings, 1 reply; 10+ messages in thread
From: Alex Vandiver @ 2017-10-27 23:26 UTC (permalink / raw)
To: git; +Cc: Johannes Schindelin, Ben Peart
If the fsmonitor extension is used in conjunction with the split index
extension, the set of entries in the index when it is first loaded is
only a subset of the real index. This leads to only the non-"base"
index being marked as CE_FSMONITOR_VALID.
Delay the expansion of the ewah bitmap until after tweak_split_index
has been called to merge in the base index as well.
The new fsmonitor_dirty is kept from being leaked by dint of being
cleaned up in post_read_index_from, which is guaranteed to be called
after do_read_index in read_index_from.
Signed-off-by: Alex Vandiver <alexmv@dropbox.com>
---
cache.h | 1 +
fsmonitor.c | 39 ++++++++++++++++++++++++---------------
2 files changed, 25 insertions(+), 15 deletions(-)
diff --git a/cache.h b/cache.h
index 25adcf681..0a4f43ec2 100644
--- a/cache.h
+++ b/cache.h
@@ -348,6 +348,7 @@ struct index_state {
unsigned char sha1[20];
struct untracked_cache *untracked;
uint64_t fsmonitor_last_update;
+ struct ewah_bitmap *fsmonitor_dirty;
};
extern struct index_state the_index;
diff --git a/fsmonitor.c b/fsmonitor.c
index 4ea44dcc6..417759224 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -49,20 +49,7 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,
ewah_free(fsmonitor_dirty);
return error("failed to parse ewah bitmap reading fsmonitor index extension");
}
-
- if (git_config_get_fsmonitor()) {
- /* Mark all entries valid */
- for (i = 0; i < istate->cache_nr; i++)
- istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;
-
- /* Mark all previously saved entries as dirty */
- ewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);
-
- /* Now mark the untracked cache for fsmonitor usage */
- if (istate->untracked)
- istate->untracked->use_fsmonitor = 1;
- }
- ewah_free(fsmonitor_dirty);
+ istate->fsmonitor_dirty = fsmonitor_dirty;
trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful");
return 0;
@@ -239,7 +226,29 @@ void remove_fsmonitor(struct index_state *istate)
void tweak_fsmonitor(struct index_state *istate)
{
- switch (git_config_get_fsmonitor()) {
+ int i;
+ int fsmonitor_enabled = git_config_get_fsmonitor();
+
+ if (istate->fsmonitor_dirty) {
+ if (fsmonitor_enabled) {
+ /* Mark all entries valid */
+ for (i = 0; i < istate->cache_nr; i++) {
+ istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;
+ }
+
+ /* Mark all previously saved entries as dirty */
+ ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);
+
+ /* Now mark the untracked cache for fsmonitor usage */
+ if (istate->untracked)
+ istate->untracked->use_fsmonitor = 1;
+ }
+
+ ewah_free(istate->fsmonitor_dirty);
+ istate->fsmonitor_dirty = NULL;
+ }
+
+ switch (fsmonitor_enabled) {
case -1: /* keep: do nothing */
break;
case 0: /* false */
--
2.15.0.rc1.413.g76aedb451
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged
2017-10-27 23:26 ` [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged Alex Vandiver
@ 2017-10-31 5:24 ` Junio C Hamano
2017-10-31 17:31 ` Johannes Schindelin
0 siblings, 1 reply; 10+ messages in thread
From: Junio C Hamano @ 2017-10-31 5:24 UTC (permalink / raw)
To: Alex Vandiver; +Cc: git, Johannes Schindelin, Ben Peart
Alex Vandiver <alexmv@dropbox.com> writes:
> diff --git a/fsmonitor.c b/fsmonitor.c
> index 4ea44dcc6..417759224 100644
> --- a/fsmonitor.c
> +++ b/fsmonitor.c
> @@ -49,20 +49,7 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,
> ewah_free(fsmonitor_dirty);
> return error("failed to parse ewah bitmap reading fsmonitor index extension");
> }
> -
> - if (git_config_get_fsmonitor()) {
> - /* Mark all entries valid */
> - for (i = 0; i < istate->cache_nr; i++)
> - istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;
> -
> - /* Mark all previously saved entries as dirty */
> - ewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);
> -
> - /* Now mark the untracked cache for fsmonitor usage */
> - if (istate->untracked)
> - istate->untracked->use_fsmonitor = 1;
> - }
> - ewah_free(fsmonitor_dirty);
> + istate->fsmonitor_dirty = fsmonitor_dirty;
This makes local variable "int i;" in this function unused and gets
compiler warning.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged
2017-10-31 5:24 ` Junio C Hamano
@ 2017-10-31 17:31 ` Johannes Schindelin
2017-10-31 18:43 ` Alex Vandiver
0 siblings, 1 reply; 10+ messages in thread
From: Johannes Schindelin @ 2017-10-31 17:31 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Alex Vandiver, git, Ben Peart
Hi,
On Tue, 31 Oct 2017, Junio C Hamano wrote:
> Alex Vandiver <alexmv@dropbox.com> writes:
>
> > diff --git a/fsmonitor.c b/fsmonitor.c
> > index 4ea44dcc6..417759224 100644
> > --- a/fsmonitor.c
> > +++ b/fsmonitor.c
> > @@ -49,20 +49,7 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,
> > ewah_free(fsmonitor_dirty);
> > return error("failed to parse ewah bitmap reading fsmonitor index extension");
> > }
> > -
> > - if (git_config_get_fsmonitor()) {
> > - /* Mark all entries valid */
> > - for (i = 0; i < istate->cache_nr; i++)
> > - istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;
> > -
> > - /* Mark all previously saved entries as dirty */
> > - ewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);
> > -
> > - /* Now mark the untracked cache for fsmonitor usage */
> > - if (istate->untracked)
> > - istate->untracked->use_fsmonitor = 1;
> > - }
> > - ewah_free(fsmonitor_dirty);
> > + istate->fsmonitor_dirty = fsmonitor_dirty;
>
> This makes local variable "int i;" in this function unused and gets
> compiler warning.
... to which end we introduced the DEVELOPER flag to catch these: if you
call
make DEVELOPER=1
and compile with GCC or Clang, it will elevate such warnings to errors,
and we highly encourage contributors to build their patched source code
with said flag.
Thanks,
Johannes
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged
2017-10-31 17:31 ` Johannes Schindelin
@ 2017-10-31 18:43 ` Alex Vandiver
0 siblings, 0 replies; 10+ messages in thread
From: Alex Vandiver @ 2017-10-31 18:43 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: Junio C Hamano, git, Ben Peart
On Tue, 31 Oct 2017, Junio C Hamano wrote:
> This makes local variable "int i;" in this function unused and gets
> compiler warning.
Apologies for leaving that detritus -- I saw you added a 'SQUASH??' commit
to deal with it, which LGTM.
On Tue, 31 Oct 2017, Johannes Schindelin wrote:
> ... to which end we introduced the DEVELOPER flag to catch these: if you
> call
>
> make DEVELOPER=1
Aha! Thanks for the tip; I'll be sure to use that from now on.
- Alex
^ permalink raw reply [flat|nested] 10+ messages in thread