From: "Nipunn Koorapati via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: Derrick Stolee <stolee@gmail.com>, Utsav Shah <utsav@dropbox.com>,
Nipunn Koorapati <nipunn1313@gmail.com>
Subject: [PATCH 0/4] use fsmonitor data in git diff eliminating O(num_files) calls to lstat
Date: Sat, 17 Oct 2020 21:04:32 +0000 [thread overview]
Message-ID: <pull.756.git.1602968677.gitgitgadget@gmail.com> (raw)
Credit to alexmv who made this commit back in Dec, 2017 when he was at dbx.
I've rebased it and am submitting it now.
With fsmonitor enabled, git diff currently lstats every file in the repo
This makes use of the fsmonitor extension to skip lstat() calls on files
that fsmonitor judged as unmodified.
I was able to do some testing with/without this change in a large in-house
repo (~ 400k files)
-----------------------------------------
(1) With fsmonitor enabled - on master of git (2.29.0)
-----------------------------------------
../git/bin-wrappers/git checkout HEAD~200
strace -c ../git/bin-wrappers/git diff
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.64 4.358994 10 446257 3 lstat
0.12 0.005353 7 764 360 open
(A subsequent call)
strace -c ../git/bin-wrappers/git diff
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.84 4.380955 10 444904 3 lstat
0.06 0.002564 135 19 munmap
...
-----------------------------------------
(2) With fsmonitor enabled - with my patch
-----------------------------------------
../git/bin-wrappers/git checkout HEAD~200
strace -c ../git/bin-wrappers/git diff
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
50.72 0.003090 163 19 munmap
19.63 0.001196 598 2 futex
...
0.00 0.000000 0 4 3 lstat
-----------------------------------------
(3) With fsmonitor disabled entirely
-----------------------------------------
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
98.52 0.277085 92362 3 futex
0.27 0.000752 4 191 63 open
...
0.14 0.000397 3 158 3 lstat
I encoded this into a perf test with results as follow:
On master (2.29)
Test this tree
--------------------------------------------------------------------------------
7519.2: status (fsmonitor=.git/hooks/fsmonitor-watchman) 2.52(1.59+1.56)
7519.3: status -uno (fsmonitor=.git/hooks/fsmonitor-watchman) 0.18(0.12+0.06)
7519.4: status -uall (fsmonitor=.git/hooks/fsmonitor-watchman) 1.36(0.73+0.62)
7519.5: diff (fsmonitor=.git/hooks/fsmonitor-watchman) 0.85(0.30+0.54)
7519.7: status (fsmonitor=) 0.69(0.52+0.90)
7519.8: status -uno (fsmonitor=) 0.37(0.28+0.81)
7519.9: status -uall (fsmonitor=) 1.53(0.93+1.32)
7519.10: diff (fsmonitor=) 0.34(0.26+0.81)
With this patch
Test this tree
--------------------------------------------------------------------------------
7519.2: status (fsmonitor=.git/hooks/fsmonitor-watchman) 2.84(1.70+1.76)
7519.3: status -uno (fsmonitor=.git/hooks/fsmonitor-watchman) 0.18(0.13+0.05)
7519.4: status -uall (fsmonitor=.git/hooks/fsmonitor-watchman) 1.35(0.81+0.53)
7519.5: diff (fsmonitor=.git/hooks/fsmonitor-watchman) 0.15(0.11+0.05)
7519.7: status (fsmonitor=) 0.71(0.54+0.90)
7519.8: status -uno (fsmonitor=) 0.38(0.30+0.81)
7519.9: status -uall (fsmonitor=) 1.55(0.93+1.34)
7519.10: diff (fsmonitor=) 0.35(0.32+0.76)
Alex Vandiver (1):
fsmonitor: use fsmonitor data in `git diff`
Nipunn Koorapati (3):
t/perf/README: elaborate on output format
t/perf/p7519-fsmonitor.sh: warm cache on first git status
t/perf: add fsmonitor perf test for git diff
diff-lib.c | 17 +++++++++++++++--
t/perf/README | 2 ++
t/perf/p7519-fsmonitor.sh | 19 ++++++++++++++++++-
3 files changed, 35 insertions(+), 3 deletions(-)
base-commit: d4a392452e292ff924e79ec8458611c0f679d6d4
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-756%2Fnipunn1313%2Fdiff_fsmon-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-756/nipunn1313/diff_fsmon-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/756
--
gitgitgadget
next reply other threads:[~2020-10-17 21:04 UTC|newest]
Thread overview: 52+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-17 21:04 Nipunn Koorapati via GitGitGadget [this message]
2020-10-17 21:04 ` [PATCH 1/4] fsmonitor: use fsmonitor data in `git diff` Alex Vandiver via GitGitGadget
2020-10-17 22:25 ` Junio C Hamano
2020-10-18 0:54 ` Nipunn Koorapati
2020-10-18 4:17 ` Taylor Blau
2020-10-18 5:02 ` Junio C Hamano
2020-10-18 23:43 ` Taylor Blau
2020-10-19 17:23 ` Junio C Hamano
2020-10-19 17:37 ` Taylor Blau
2020-10-19 18:07 ` Nipunn Koorapati
2020-10-17 21:04 ` [PATCH 2/4] t/perf/README: elaborate on output format Nipunn Koorapati via GitGitGadget
2020-10-17 21:04 ` [PATCH 3/4] t/perf/p7519-fsmonitor.sh: warm cache on first git status Nipunn Koorapati via GitGitGadget
2020-10-18 4:22 ` Taylor Blau
2020-10-17 21:04 ` [PATCH 4/4] t/perf: add fsmonitor perf test for git diff Nipunn Koorapati via GitGitGadget
2020-10-17 22:28 ` Junio C Hamano
2020-10-19 21:35 ` [PATCH v2 0/4] use fsmonitor data in git diff eliminating O(num_files) calls to lstat Nipunn Koorapati via GitGitGadget
2020-10-19 21:35 ` [PATCH v2 1/4] fsmonitor: use fsmonitor data in `git diff` Alex Vandiver via GitGitGadget
2020-10-19 21:35 ` [PATCH v2 2/4] t/perf/README: elaborate on output format Nipunn Koorapati via GitGitGadget
2020-10-19 21:35 ` [PATCH v2 3/4] t/perf/p7519-fsmonitor.sh: warm cache on first git status Nipunn Koorapati via GitGitGadget
2020-10-19 21:35 ` [PATCH v2 4/4] t/perf: add fsmonitor perf test for git diff Nipunn Koorapati via GitGitGadget
2020-10-19 21:43 ` Taylor Blau
2020-10-19 21:54 ` Taylor Blau
2020-10-19 22:00 ` Nipunn Koorapati
2020-10-19 22:02 ` Taylor Blau
2020-10-19 22:25 ` Nipunn Koorapati
2020-10-19 22:47 ` [PATCH v3 0/7] use fsmonitor data in git diff eliminating O(num_files) calls to lstat Nipunn Koorapati via GitGitGadget
2020-10-19 22:47 ` [PATCH v3 1/7] fsmonitor: use fsmonitor data in `git diff` Alex Vandiver via GitGitGadget
2020-10-19 22:47 ` [PATCH v3 2/7] t/perf/README: elaborate on output format Nipunn Koorapati via GitGitGadget
2020-10-19 22:47 ` [PATCH v3 3/7] t/perf/p7519-fsmonitor.sh: warm cache on first git status Nipunn Koorapati via GitGitGadget
2020-10-19 22:47 ` [PATCH v3 4/7] t/perf: add fsmonitor perf test for git diff Nipunn Koorapati via GitGitGadget
2020-10-19 22:47 ` [PATCH v3 5/7] perf lint: check test-lint-shell-syntax in perf tests Nipunn Koorapati via GitGitGadget
2020-10-20 2:38 ` Taylor Blau
2020-10-20 3:10 ` Junio C Hamano
2020-10-20 3:15 ` Taylor Blau
2020-10-20 10:16 ` Nipunn Koorapati
2020-10-20 10:09 ` Nipunn Koorapati
2020-10-19 22:47 ` [PATCH v3 6/7] p7519-fsmonitor: refactor to avoid code duplication Nipunn Koorapati via GitGitGadget
2020-10-20 2:43 ` Taylor Blau
2020-10-19 22:47 ` [PATCH v3 7/7] p7519-fsmonitor: add a git add benchmark Nipunn Koorapati via GitGitGadget
2020-10-19 23:02 ` Nipunn Koorapati
2020-10-20 2:40 ` Taylor Blau
2020-10-20 13:40 ` [PATCH v4 0/7] use fsmonitor data in git diff eliminating O(num_files) calls to lstat Nipunn Koorapati via GitGitGadget
2020-10-20 13:40 ` [PATCH v4 1/7] fsmonitor: use fsmonitor data in `git diff` Alex Vandiver via GitGitGadget
2020-10-20 13:40 ` [PATCH v4 2/7] t/perf/README: elaborate on output format Nipunn Koorapati via GitGitGadget
2020-10-20 13:41 ` [PATCH v4 3/7] t/perf/p7519-fsmonitor.sh: warm cache on first git status Nipunn Koorapati via GitGitGadget
2020-10-20 13:41 ` [PATCH v4 4/7] t/perf: add fsmonitor perf test for git diff Nipunn Koorapati via GitGitGadget
2020-10-20 13:41 ` [PATCH v4 5/7] perf lint: add make test-lint to perf tests Nipunn Koorapati via GitGitGadget
2020-10-20 22:06 ` Taylor Blau
2020-10-20 22:17 ` Nipunn Koorapati
2020-10-20 22:19 ` Taylor Blau
2020-10-20 13:41 ` [PATCH v4 6/7] p7519-fsmonitor: refactor to avoid code duplication Nipunn Koorapati via GitGitGadget
2020-10-20 13:41 ` [PATCH v4 7/7] p7519-fsmonitor: add a git add benchmark Nipunn Koorapati via GitGitGadget
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=pull.756.git.1602968677.gitgitgadget@gmail.com \
--to=gitgitgadget@gmail.com \
--cc=git@vger.kernel.org \
--cc=nipunn1313@gmail.com \
--cc=stolee@gmail.com \
--cc=utsav@dropbox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).