Git Mailing List Archive on lore.kernel.org
 help / color / Atom feed
From: David Turner <dturner@twopensource.com>
To: git@vger.kernel.org, pclouds@gmail.com
Cc: David Turner <dturner@twopensource.com>
Subject: [PATCH 00/19] index-helper, watchman
Date: Wed,  9 Mar 2016 13:36:03 -0500
Message-ID: <1457548582-28302-1-git-send-email-dturner@twopensource.com> (raw)

This is a rebase and extension of Duy's work on git index-helper and
watchman support.

Patches 1-14 and are more-or-less just Duy's patches, rebased. I fixed
up a race condition with signal handling, but otherwise didn't change
much.

Patch 15 preserves the untracked cache and watchman extensions across
checkouts.

Patches 16-18 are new index-helper patches that I wrote to improve the
general usability of index-helper.

And patch 19 is for discussion only -- if the overall concept is
correct, I'll rewrite it into the places where it should go.  I didn't
do it yet in part because I am not sure folks will agree with it and
in part because I didn't want to mess too much with Duy's code before
re-sending it.

I haven't used this code much.  I just wrote patches 16-18 today, in
fact. It's got fewer tests than I would like, in part because it is
somewhat difficult to test since it involves two separate persistent
daemons (index-helper, watchman), and the interaction between them.

At Twitter, we're still using something based on my earlier watchman
patches[1].  We would like to switch to this (especially if this is
the version that mainstream git is going towards).  But we have other
local patches[2], and I haven't fully finished migrating them to 2.8.
My informal testing shows that with the untracked cache and watchman
index extensions, and git index-helper[3], performance is slightly
better than my earlier code.  That's not a surprise, since the
index-helper eliminates index verification time.

[1] https://github.com/dturner-tw/git/tree/dturner/watchman

[2] e.g. something like this:
https://github.com/dturner-tw/git/tree/dturner/journal

[3] In both cases, I'm using a version which has the SSE ref-parsing
and hashmap code; these were rejected due to complexity but they provide
a benefit for us, so we still use them.

David Turner (5):
  unpack-trees: preserve index extensions
  index-helper: rewrite pidfile after daemonizing
  index-helper: process management
  index-helper: autorun
  hack: watchman/untracked cache mashup

Nguyễn Thái Ngọc Duy (14):
  trace.c: add GIT_TRACE_PACK_STATS for pack usage statistics
  read-cache.c: fix constness of verify_hdr()
  read-cache: allow to keep mmap'd memory after reading
  index-helper: new daemon for caching index and related stuff
  trace.c: add GIT_TRACE_INDEX_STATS for index statistics
  index-helper: add --strict
  daemonize(): set a flag before exiting the main process
  index-helper: add --detach
  index-helper: add Windows support
  read-cache: add watchman 'WAMA' extension
  Add watchman support to reduce index refresh cost
  read-cache: allow index-helper to prepare shm before git reads it
  index-helper: use watchman to avoid refreshing index with lstat()
  update-index: enable/disable watchman support

 .gitignore                         |   1 +
 Documentation/git-index-helper.txt |  56 +++++
 Documentation/git.txt              |   4 +
 Makefile                           |  21 ++
 builtin/gc.c                       |   2 +-
 builtin/update-index.c             |  11 +
 cache.h                            |  20 +-
 config.c                           |   5 +
 config.mak.uname                   |   3 +
 configure.ac                       |   8 +
 daemon.c                           |   2 +-
 dir.c                              |  23 +-
 dir.h                              |   6 +
 environment.c                      |   3 +
 git-compat-util.h                  |   1 +
 git.c                              |  37 +++-
 index-helper.c                     | 437 ++++++++++++++++++++++++++++++++++++
 read-cache.c                       | 441 +++++++++++++++++++++++++++++++++++--
 setup.c                            |   4 +-
 sha1_file.c                        |  24 ++
 shm.c                              | 163 ++++++++++++++
 shm.h                              |  23 ++
 t/t7063-status-untracked-cache.sh  |  23 ++
 t/t7900-index-helper.sh            |  23 ++
 t/test-lib-functions.sh            |   4 +
 trace.c                            |  16 ++
 trace.h                            |   1 +
 unpack-trees.c                     |   1 +
 watchman-support.c                 | 134 +++++++++++
 watchman-support.h                 |   7 +
 30 files changed, 1481 insertions(+), 23 deletions(-)
 create mode 100644 Documentation/git-index-helper.txt
 create mode 100644 index-helper.c
 create mode 100644 shm.c
 create mode 100644 shm.h
 create mode 100755 t/t7900-index-helper.sh
 create mode 100644 watchman-support.c
 create mode 100644 watchman-support.h

-- 
2.4.2.767.g62658d5-twtrsrc

             reply index

Thread overview: 60+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-03-09 18:36 David Turner [this message]
2016-03-09 18:36 ` [PATCH 01/19] trace.c: add GIT_TRACE_PACK_STATS for pack usage statistics David Turner
2016-03-09 22:58   ` Junio C Hamano
2016-03-10  0:05     ` David Turner
2016-03-10 10:59       ` Duy Nguyen
2016-03-09 18:36 ` [PATCH 02/19] read-cache.c: fix constness of verify_hdr() David Turner
2016-03-09 18:36 ` [PATCH 03/19] read-cache: allow to keep mmap'd memory after reading David Turner
2016-03-09 23:02   ` Junio C Hamano
2016-03-10  0:09     ` David Turner
2016-03-09 18:36 ` [PATCH 04/19] index-helper: new daemon for caching index and related stuff David Turner
2016-03-09 23:09   ` Junio C Hamano
2016-03-09 23:21     ` Junio C Hamano
2016-03-10  0:01       ` David Turner
2016-03-10 11:17       ` Duy Nguyen
2016-03-10 20:22         ` David Turner
2016-03-11  1:11           ` Duy Nguyen
2016-03-10  0:18     ` David Turner
2016-03-15 11:56     ` Duy Nguyen
2016-03-15 15:56       ` Junio C Hamano
2016-03-15 11:52   ` Duy Nguyen
2016-03-09 18:36 ` [PATCH 05/19] trace.c: add GIT_TRACE_INDEX_STATS for index statistics David Turner
2016-03-09 18:36 ` [PATCH 06/19] index-helper: add --strict David Turner
2016-03-09 18:36 ` [PATCH 07/19] daemonize(): set a flag before exiting the main process David Turner
2016-03-09 18:36 ` [PATCH 08/19] index-helper: add --detach David Turner
2016-03-09 18:36 ` [PATCH 09/19] index-helper: add Windows support David Turner
2016-03-16 11:42   ` Duy Nguyen
2016-03-17 12:18     ` Johannes Schindelin
2016-03-17 12:59       ` Duy Nguyen
2016-03-09 18:36 ` [PATCH 10/19] read-cache: add watchman 'WAMA' extension David Turner
2016-03-09 18:36 ` [PATCH 11/19] Add watchman support to reduce index refresh cost David Turner
2016-03-09 18:36 ` [PATCH 12/19] read-cache: allow index-helper to prepare shm before git reads it David Turner
2016-03-09 18:36 ` [PATCH 13/19] index-helper: use watchman to avoid refreshing index with lstat() David Turner
2016-03-09 18:36 ` [PATCH 14/19] update-index: enable/disable watchman support David Turner
2016-03-09 18:36 ` [PATCH 15/19] unpack-trees: preserve index extensions David Turner
2016-03-09 18:36 ` [PATCH 16/19] index-helper: rewrite pidfile after daemonizing David Turner
2016-03-09 18:36 ` [PATCH 17/19] index-helper: process management David Turner
2016-03-09 18:36 ` [PATCH 18/19] index-helper: autorun David Turner
2016-03-15 12:12   ` Duy Nguyen
2016-03-15 14:26     ` Johannes Schindelin
2016-03-16 11:37       ` Duy Nguyen
2016-03-16 18:11       ` David Turner
2016-03-16 18:27         ` Johannes Schindelin
2016-03-17 13:02           ` Duy Nguyen
2016-03-17 14:43             ` Johannes Schindelin
2016-03-17 18:31               ` David Turner
2016-03-18  0:50               ` Duy Nguyen
2016-03-18  7:14                 ` Johannes Schindelin
2016-03-18  7:44                   ` Duy Nguyen
2016-03-18 17:22                     ` David Turner
2016-03-18 23:09                       ` Duy Nguyen
2016-03-18  7:17                 ` Johannes Schindelin
2016-03-18  7:34                   ` Duy Nguyen
2016-03-18 15:57                     ` Johannes Schindelin
2016-03-09 18:36 ` [PATCH 19/19] hack: watchman/untracked cache mashup David Turner
2016-03-15 12:31   ` Duy Nguyen
2016-03-17  0:56     ` David Turner
2016-03-17 13:06       ` Duy Nguyen
2016-03-17 18:08         ` David Turner
2016-03-29 17:09 ` [PATCH 00/19] index-helper, watchman Torsten Bögershausen
2016-03-29 21:51   ` David Turner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1457548582-28302-1-git-send-email-dturner@twopensource.com \
    --to=dturner@twopensource.com \
    --cc=git@vger.kernel.org \
    --cc=pclouds@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Git Mailing List Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/git/0 git/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 git git/ https://lore.kernel.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.git


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git