From: Ben Peart <peartben@gmail.com>
To: Duy Nguyen <pclouds@gmail.com>, Ben Peart <benpeart@microsoft.com>
Cc: Git Mailing List <git@vger.kernel.org>,
Junio C Hamano <gitster@pobox.com>,
Ben Peart <Ben.Peart@microsoft.com>
Subject: Re: [PATCH v5 3/5] read-cache: load cache entries on worker threads
Date: Mon, 17 Sep 2018 14:52:05 -0400 [thread overview]
Message-ID: <f8f021ce-5a3d-a5f3-ef47-e9cc94460b24@gmail.com> (raw)
In-Reply-To: <20180915110951.GA17817@duynguyen.home>
On 9/15/2018 7:09 AM, Duy Nguyen wrote:
> On Sat, Sep 15, 2018 at 01:07:46PM +0200, Duy Nguyen wrote:
>> 12:50:00.084237 read-cache.c:1721 start loading index
>> 12:50:00.119941 read-cache.c:1943 performance: 0.034778758 s: loaded all extensions (1667075 bytes)
>> 12:50:00.185352 read-cache.c:2029 performance: 0.100152079 s: loaded 367110 entries
>> 12:50:00.189683 read-cache.c:2126 performance: 0.104566615 s: finished scanning all entries
>> 12:50:00.217900 read-cache.c:2029 performance: 0.082309193 s: loaded 367110 entries
>> 12:50:00.259969 read-cache.c:2029 performance: 0.070257130 s: loaded 367108 entries
>> 12:50:00.263662 read-cache.c:2278 performance: 0.179344458 s: read cache .git/index
>
> The previous mail wraps these lines and make it a bit hard to read. Corrected now.
>
> --
> Duy
>
Interesting! Clearly the data shape makes a big difference here as I
had run a similar test but in my case, the extensions thread actually
finished last (and it's cost is what drove me to move that onto a
separate thread that starts first).
Purpose First Last Duration
load_index_extensions_thread 719.40 968.50 249.10
load_cache_entries_thread 718.89 738.65 19.76
load_cache_entries_thread 730.39 753.83 23.43
load_cache_entries_thread 741.23 751.23 10.00
load_cache_entries_thread 751.93 780.88 28.95
load_cache_entries_thread 763.60 791.31 27.72
load_cache_entries_thread 773.46 783.46 10.00
load_cache_entries_thread 783.96 794.28 10.32
load_cache_entries_thread 795.61 805.52 9.91
load_cache_entries_thread 805.99 827.21 21.22
load_cache_entries_thread 816.85 826.85 10.00
load_cache_entries_thread 827.03 837.96 10.93
In my tests, the scanning thread clearly delayed the later ce threads
but given the extension was so slow, it didn't impact the overall time
nearly as much as your case.
I completely agree that the optimal solution would be to go back to my
original patch/design. It eliminates the overhead of the scanning
thread entirely and allows all threads to start at the same time. This
would ensure the best performance whether the extensions were the
longest thread or the cache entry threads took the longest.
I ran out of time and energy last year so dropped it to work on other
tasks. I appreciate your offer of help. Perhaps between the two of us
we could successfully get it through the mailing list this time. :-)
Let me go back and see what it would take to combine the current EOIE
patch with the older IEOT patch.
I'm also intrigued with your observation that over committing the cpu
actually results in time savings. I hadn't tested that. It looks like
that could have a positive impact on the overall time and warrant a
change to the default nr_threads logic.
next prev parent reply other threads:[~2018-09-17 18:52 UTC|newest]
Thread overview: 199+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-23 15:41 [PATCH v1] read-cache: speed up index load through parallelization Ben Peart
2018-08-23 17:31 ` Stefan Beller
2018-08-23 19:44 ` Ben Peart
2018-08-24 18:40 ` Duy Nguyen
2018-08-28 14:53 ` Ben Peart
2018-08-23 18:06 ` Junio C Hamano
2018-08-23 20:33 ` Ben Peart
2018-08-24 15:37 ` Duy Nguyen
2018-08-24 15:57 ` Duy Nguyen
2018-08-24 17:28 ` Ben Peart
2018-08-25 6:44 ` [PATCH] read-cache.c: optimize reading index format v4 Nguyễn Thái Ngọc Duy
2018-08-27 19:36 ` Junio C Hamano
2018-08-28 19:25 ` Duy Nguyen
2018-08-28 23:54 ` Ben Peart
2018-08-29 17:14 ` Junio C Hamano
2018-09-04 16:08 ` Duy Nguyen
2018-09-02 13:19 ` [PATCH v2 0/1] " Nguyễn Thái Ngọc Duy
2018-09-02 13:19 ` [PATCH v2 1/1] read-cache.c: " Nguyễn Thái Ngọc Duy
2018-09-04 18:58 ` Junio C Hamano
2018-09-04 19:31 ` Junio C Hamano
2018-08-24 18:20 ` [PATCH v1] read-cache: speed up index load through parallelization Duy Nguyen
2018-08-24 18:40 ` Ben Peart
2018-08-24 19:00 ` Duy Nguyen
2018-08-24 19:57 ` Ben Peart
2018-08-29 15:25 ` [PATCH v2 0/3] " Ben Peart
2018-08-29 15:25 ` [PATCH v2 1/3] " Ben Peart
2018-08-29 17:14 ` Junio C Hamano
2018-08-29 21:35 ` Ben Peart
2018-09-03 19:16 ` Duy Nguyen
2018-08-29 15:25 ` [PATCH v2 2/3] read-cache: load cache extensions on worker thread Ben Peart
2018-08-29 17:12 ` Junio C Hamano
2018-08-29 21:42 ` Ben Peart
2018-08-29 22:19 ` Junio C Hamano
2018-09-03 19:21 ` Duy Nguyen
2018-09-03 19:27 ` Duy Nguyen
2018-08-29 15:25 ` [PATCH v2 3/3] read-cache: micro-optimize expand_name_field() to speed up V4 index parsing Ben Peart
2018-09-06 21:03 ` [PATCH v3 0/4] read-cache: speed up index load through parallelization Ben Peart
2018-09-06 21:03 ` [PATCH v3 1/4] read-cache: optimize expand_name_field() to speed up V4 index parsing Ben Peart
2018-09-06 21:03 ` [PATCH v3 2/4] eoie: add End of Index Entry (EOIE) extension Ben Peart
2018-09-07 17:55 ` Junio C Hamano
2018-09-07 20:23 ` Ben Peart
2018-09-08 6:29 ` Martin Ågren
2018-09-08 14:03 ` Ben Peart
2018-09-08 17:08 ` Martin Ågren
2018-09-06 21:03 ` [PATCH v3 3/4] read-cache: load cache extensions on a worker thread Ben Peart
2018-09-07 21:10 ` Junio C Hamano
2018-09-08 14:56 ` Ben Peart
2018-09-06 21:03 ` [PATCH v3 4/4] read-cache: speed up index load through parallelization Ben Peart
2018-09-07 4:16 ` Torsten Bögershausen
2018-09-07 13:43 ` Ben Peart
2018-09-07 17:21 ` [PATCH v3 0/4] " Junio C Hamano
2018-09-07 18:31 ` Ben Peart
2018-09-08 13:18 ` Duy Nguyen
2018-09-11 23:26 ` [PATCH v4 0/5] " Ben Peart
2018-09-11 23:26 ` [PATCH v4 1/5] eoie: add End of Index Entry (EOIE) extension Ben Peart
2018-09-11 23:26 ` [PATCH v4 2/5] read-cache: load cache extensions on a worker thread Ben Peart
2018-09-11 23:26 ` [PATCH v4 3/5] read-cache: speed up index load through parallelization Ben Peart
2018-09-11 23:26 ` [PATCH v4 4/5] read-cache.c: optimize reading index format v4 Ben Peart
2018-09-11 23:26 ` [PATCH v4 5/5] read-cache: clean up casting and byte decoding Ben Peart
2018-09-12 14:34 ` [PATCH v4 0/5] read-cache: speed up index load through parallelization Ben Peart
2018-09-12 16:18 ` [PATCH v5 " Ben Peart
2018-09-12 16:18 ` [PATCH v5 1/5] eoie: add End of Index Entry (EOIE) extension Ben Peart
2018-09-13 22:44 ` Junio C Hamano
2018-09-15 10:02 ` Duy Nguyen
2018-09-17 14:54 ` Ben Peart
2018-09-17 16:05 ` Duy Nguyen
2018-09-17 17:31 ` Junio C Hamano
2018-09-17 17:38 ` Duy Nguyen
2018-09-17 19:08 ` Junio C Hamano
2018-09-12 16:18 ` [PATCH v5 2/5] read-cache: load cache extensions on a worker thread Ben Peart
2018-09-15 10:22 ` Duy Nguyen
2018-09-15 10:24 ` Duy Nguyen
2018-09-17 16:38 ` Ben Peart
2018-09-15 16:23 ` Duy Nguyen
2018-09-17 17:19 ` Junio C Hamano
2018-09-17 16:26 ` Ben Peart
2018-09-17 16:45 ` Duy Nguyen
2018-09-17 21:32 ` Junio C Hamano
2018-09-12 16:18 ` [PATCH v5 3/5] read-cache: load cache entries on worker threads Ben Peart
2018-09-15 10:31 ` Duy Nguyen
2018-09-17 17:25 ` Ben Peart
2018-09-15 11:07 ` Duy Nguyen
2018-09-15 11:09 ` Duy Nguyen
2018-09-17 18:52 ` Ben Peart [this message]
2018-09-15 11:29 ` Duy Nguyen
2018-09-12 16:18 ` [PATCH v5 4/5] read-cache.c: optimize reading index format v4 Ben Peart
2018-09-12 16:18 ` [PATCH v5 5/5] read-cache: clean up casting and byte decoding Ben Peart
2018-09-26 19:54 ` [PATCH v6 0/7] speed up index load through parallelization Ben Peart
2018-09-26 19:54 ` [PATCH v6 1/7] read-cache.c: optimize reading index format v4 Ben Peart
2018-09-26 19:54 ` [PATCH v6 2/7] read-cache: clean up casting and byte decoding Ben Peart
2018-09-26 19:54 ` [PATCH v6 3/7] eoie: add End of Index Entry (EOIE) extension Ben Peart
2018-09-28 0:19 ` SZEDER Gábor
2018-09-28 18:38 ` Ben Peart
2018-09-29 0:51 ` SZEDER Gábor
2018-09-29 5:45 ` Duy Nguyen
2018-09-29 18:24 ` Junio C Hamano
2018-09-26 19:54 ` [PATCH v6 4/7] config: add new index.threads config setting Ben Peart
2018-09-28 0:26 ` SZEDER Gábor
2018-09-28 13:39 ` Ben Peart
2018-09-28 17:07 ` Junio C Hamano
2018-09-28 19:41 ` Ben Peart
2018-09-28 20:30 ` Ramsay Jones
2018-09-28 22:15 ` Junio C Hamano
2018-10-01 13:17 ` Ben Peart
2018-10-01 15:06 ` SZEDER Gábor
2018-09-26 19:54 ` [PATCH v6 5/7] read-cache: load cache extensions on a worker thread Ben Peart
2018-09-26 19:54 ` [PATCH v6 6/7] ieot: add Index Entry Offset Table (IEOT) extension Ben Peart
2018-09-26 19:54 ` [PATCH v6 7/7] read-cache: load cache entries on worker threads Ben Peart
2018-09-26 22:06 ` [PATCH v6 0/7] speed up index load through parallelization Junio C Hamano
2018-09-27 17:13 ` Duy Nguyen
2018-10-01 13:45 ` [PATCH v7 " Ben Peart
2018-10-01 13:45 ` [PATCH v7 1/7] read-cache.c: optimize reading index format v4 Ben Peart
2018-10-01 13:45 ` [PATCH v7 2/7] read-cache: clean up casting and byte decoding Ben Peart
2018-10-01 15:10 ` Duy Nguyen
2018-10-01 13:45 ` [PATCH v7 3/7] eoie: add End of Index Entry (EOIE) extension Ben Peart
2018-10-01 15:17 ` SZEDER Gábor
2018-10-02 14:34 ` Ben Peart
2018-10-01 15:30 ` Duy Nguyen
2018-10-02 15:13 ` Ben Peart
2018-10-01 13:45 ` [PATCH v7 4/7] config: add new index.threads config setting Ben Peart
2018-10-01 13:45 ` [PATCH v7 5/7] read-cache: load cache extensions on a worker thread Ben Peart
2018-10-01 15:50 ` Duy Nguyen
2018-10-02 15:00 ` Ben Peart
2018-10-01 13:45 ` [PATCH v7 6/7] ieot: add Index Entry Offset Table (IEOT) extension Ben Peart
2018-10-01 16:27 ` Duy Nguyen
2018-10-02 16:34 ` Ben Peart
2018-10-02 17:02 ` Duy Nguyen
2018-10-01 13:45 ` [PATCH v7 7/7] read-cache: load cache entries on worker threads Ben Peart
2018-10-01 17:09 ` Duy Nguyen
2018-10-02 19:09 ` Ben Peart
2018-10-10 15:59 ` [PATCH v8 0/7] speed up index load through parallelization Ben Peart
2018-10-10 15:59 ` [PATCH v8 1/7] read-cache.c: optimize reading index format v4 Ben Peart
2018-10-10 15:59 ` [PATCH v8 2/7] read-cache: clean up casting and byte decoding Ben Peart
2018-10-10 15:59 ` [PATCH v8 3/7] eoie: add End of Index Entry (EOIE) extension Ben Peart
2018-10-10 15:59 ` [PATCH v8 4/7] config: add new index.threads config setting Ben Peart
2018-10-10 15:59 ` [PATCH v8 5/7] read-cache: load cache extensions on a worker thread Ben Peart
2018-10-10 15:59 ` [PATCH v8 6/7] ieot: add Index Entry Offset Table (IEOT) extension Ben Peart
2018-10-10 15:59 ` [PATCH v8 7/7] read-cache: load cache entries on worker threads Ben Peart
2018-10-19 16:11 ` Jeff King
2018-10-22 2:14 ` Junio C Hamano
2018-10-22 14:40 ` Ben Peart
2018-10-12 3:18 ` [PATCH v8 0/7] speed up index load through parallelization Junio C Hamano
2018-10-14 12:28 ` Duy Nguyen
2018-10-15 17:33 ` Ben Peart
2018-11-13 0:38 ` [PATCH 0/3] Avoid confusing messages from new index extensions (Re: [PATCH v8 0/7] speed up index load through parallelization) Jonathan Nieder
2018-11-13 0:39 ` [PATCH 1/3] eoie: default to not writing EOIE section Jonathan Nieder
2018-11-13 1:05 ` Junio C Hamano
2018-11-13 15:14 ` Ben Peart
2018-11-13 18:25 ` Jonathan Nieder
2018-11-14 1:36 ` Junio C Hamano
2018-11-15 0:19 ` Jonathan Nieder
2018-11-13 0:39 ` [PATCH 2/3] ieot: default to not writing IEOT section Jonathan Nieder
2018-11-13 0:58 ` Jonathan Tan
2018-11-13 1:09 ` Junio C Hamano
2018-11-13 1:12 ` Jonathan Nieder
2018-11-13 15:37 ` Duy Nguyen
2018-11-13 18:09 ` Jonathan Nieder
2018-11-13 15:22 ` Ben Peart
2018-11-13 18:18 ` Jonathan Nieder
2018-11-13 19:15 ` Ben Peart
2018-11-13 21:08 ` Jonathan Nieder
2018-11-14 18:09 ` Ben Peart
2018-11-15 0:05 ` Jonathan Nieder
2018-11-14 3:05 ` Junio C Hamano
2018-11-20 6:09 ` [PATCH v2 0/5] Avoid confusing messages from new index extensions Jonathan Nieder
2018-11-20 6:11 ` [PATCH 1/5] eoie: default to not writing EOIE section Jonathan Nieder
2018-11-20 13:06 ` Ben Peart
2018-11-20 13:21 ` SZEDER Gábor
2018-11-21 16:46 ` Jeff King
2018-11-22 0:47 ` Junio C Hamano
2018-11-20 15:01 ` Ben Peart
2018-11-20 6:12 ` [PATCH 2/5] ieot: default to not writing IEOT section Jonathan Nieder
2018-11-20 13:07 ` Ben Peart
2018-11-26 19:59 ` Stefan Beller
2018-11-26 21:47 ` Ben Peart
2018-11-26 22:02 ` Stefan Beller
2018-11-27 0:50 ` Junio C Hamano
2018-11-20 6:12 ` [PATCH 3/5] index: do not warn about unrecognized extensions Jonathan Nieder
2018-11-20 6:14 ` [PATCH 4/5] index: make index.threads=true enable ieot and eoie Jonathan Nieder
2018-11-20 13:24 ` Ben Peart
2018-11-20 6:15 ` [PATCH 5/5] index: offer advice for unknown index extensions Jonathan Nieder
2018-11-20 9:26 ` Ævar Arnfjörð Bjarmason
2018-11-20 13:30 ` Ben Peart
2018-11-21 0:22 ` Junio C Hamano
2018-11-21 0:39 ` Jonathan Nieder
2018-11-21 0:44 ` Jonathan Nieder
2018-11-21 5:01 ` Junio C Hamano
2018-11-21 5:04 ` Jonathan Nieder
2018-11-21 5:15 ` Junio C Hamano
2018-11-21 5:31 ` Junio C Hamano
2018-11-21 1:03 ` Jonathan Nieder
2018-11-21 4:23 ` Junio C Hamano
2018-11-21 4:57 ` Jonathan Nieder
2018-11-21 9:30 ` Ævar Arnfjörð Bjarmason
2018-11-13 0:40 ` [PATCH 3/3] index: do not warn about unrecognized extensions Jonathan Nieder
2018-11-13 1:10 ` Junio C Hamano
2018-11-13 15:25 ` Ben Peart
2018-11-14 3:24 ` Junio C Hamano
2018-11-14 18:19 ` Ben Peart
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f8f021ce-5a3d-a5f3-ef47-e9cc94460b24@gmail.com \
--to=peartben@gmail.com \
--cc=Ben.Peart@microsoft.com \
--cc=benpeart@microsoft.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=pclouds@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).