From: David Turner <novalis@novalis.org>
To: Duy Nguyen <pclouds@gmail.com>
Cc: Git Mailing List <git@vger.kernel.org>,
kmaggg@gmail.com,
Johannes Schindelin <Johannes.Schindelin@gmx.de>
Subject: Re: [PATCH v14 00/21] index-helper/watchman
Date: Wed, 13 Jul 2016 17:59:42 -0400 [thread overview]
Message-ID: <5786B9CE.6010703@novalis.org> (raw)
In-Reply-To: <CACsJy8Br-rSTKjpt2ykn8YyFruy8CZoXWm287BtTRcAYY2DjVw@mail.gmail.com>
On 07/12/2016 02:24 PM, Duy Nguyen wrote:
> Just thinking out loud. I've been thinking about this more about this.
> After the move from signal-based to unix socket for communication, we
> probably are better off with a simpler design than the shm-alike one
> we have now.
>
> What if we send everything over a socket or a pipe? Sending 500MB over
> a unix socket takes 253ms, that's insignificant when operations on an
> index that size usually take seconds. If we send everything over
> socket/pipe, we can trust data integrity and don't have to verify,
> even the trailing SHA-1 in shm file.
I think it would be good to make index operations not take seconds.
In general, we should not need to verify the trailing SHA-1 for shm
data. So the index-helper verifies it when it loads it, but the git
(e.g.) status should not need to verify.
Also, if we have two git commands running at the same time, the
index-helper can only serve one at a time; with shm, both can run at
full speed.
> So, what I have in mind is this, at read index time, instead of open a
> socket, we run a separate program and communicate via pipes. We can
> exchange capabilities if needed, then the program sends the entire
> current index, the list of updated files back (and/or the list of dirs
> to invalidate). The design looks very much like a smudge/clean filter.
This seems very complicated. Now git status talks to the separate
program, which talks to the index-helper, which talks to watchman. That
is a lot of steps!
I think we should wait until we heard from the Windows folks what the
problems with the current solution are, and see what design they come up
with.
> For people who don't want extra daemon, they can write a short script
> that saves indexes somewhere in tmpfs, and talk to watchman or
> something else. I haven't written this script, but I don't think it
> takes long to write one. Windows folks have total freedom to implement
> a daemon, a service or whatever and use this program as front end. How
> the service talks to this program is totally up to them. For people
> who want to centralize everything, they can have just one daemon and
> have the script to talk to this daemon.
>
> I can see that getting rid of file-based stuff simplifies some
> patches. We can still provide a daemon to do more advanced stuff (or
> to make it work out of the box). But it's not a hard requirement and
> we probably don't need to include one right now. And I think it makes
> it easier to test as well because we can just go with some fake file
> monitor service instead of real watchman.
I think the daemon also has the advantage that it can reload the index
as soon as it changes. This is not quite implemented, but it would be
pretty easy to do. That would save a lot of time in the typical workflow.
next prev parent reply other threads:[~2016-07-13 21:59 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-07-03 7:57 [PATCH v14 00/21] index-helper/watchman David Turner
2016-07-03 7:57 ` [PATCH v14 01/21] read-cache: allow to keep mmap'd memory after reading David Turner
2016-07-03 7:57 ` [PATCH v14 02/21] pkt-line: add gentle version of packet_write David Turner
2016-07-03 7:57 ` [PATCH v14 03/21] unix-socket.c: add stub implementation when unix sockets are not supported David Turner
2016-07-03 7:57 ` [PATCH v14 04/21] index-helper: new daemon for caching index and related stuff David Turner
2016-07-03 7:57 ` [PATCH v14 05/21] index-helper: add --strict David Turner
2016-07-03 7:57 ` [PATCH v14 06/21] daemonize(): set a flag before exiting the main process David Turner
2016-07-03 7:57 ` [PATCH v14 07/21] index-helper: add --detach David Turner
2016-07-03 7:58 ` [PATCH v14 08/21] index-helper: log warnings David Turner
2016-07-03 7:58 ` [PATCH v14 09/21] read-cache: add watchman 'WAMA' extension David Turner
2016-07-03 7:58 ` [PATCH v14 10/21] watchman: support watchman to reduce index refresh cost David Turner
2016-07-03 7:58 ` [PATCH v14 11/21] index-helper: use watchman to avoid refreshing index with lstat() David Turner
2016-07-03 7:58 ` [PATCH v14 12/21] update-index: enable/disable watchman support David Turner
2016-07-03 7:58 ` [PATCH v14 13/21] unpack-trees: preserve index extensions David Turner
2016-07-03 7:58 ` [PATCH v14 14/21] watchman: add a config option to enable the extension David Turner
2016-07-03 7:58 ` [PATCH v14 15/21] index-helper: kill mode David Turner
2016-07-06 8:20 ` Johannes Schindelin
2016-07-06 15:33 ` Duy Nguyen
2016-07-03 7:58 ` [PATCH v14 16/21] index-helper: don't run if already running David Turner
2016-07-03 7:58 ` [PATCH v14 17/21] index-helper: autorun mode David Turner
2016-07-03 7:58 ` [PATCH v14 18/21] index-helper: optionally automatically run David Turner
2016-07-03 7:58 ` [PATCH v14 19/21] trace: measure where the time is spent in the index-heavy operations David Turner
2016-07-03 11:51 ` [PATCH v14 00/21] index-helper/watchman Johannes Schindelin
2016-07-04 6:40 ` Johannes Schindelin
2016-07-06 18:11 ` Junio C Hamano
2016-07-12 18:24 ` Duy Nguyen
2016-07-13 21:59 ` David Turner [this message]
2016-07-14 15:56 ` Duy Nguyen
2016-07-14 15:58 ` Duy Nguyen
2016-07-15 1:20 ` Ben Peart
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5786B9CE.6010703@novalis.org \
--to=novalis@novalis.org \
--cc=Johannes.Schindelin@gmx.de \
--cc=git@vger.kernel.org \
--cc=kmaggg@gmail.com \
--cc=pclouds@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).