From: "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: gitster@pobox.com, vdye@github.com, avarab@gmail.com,
newren@gmail.com, Jacob Keller <jacob.keller@gmail.com>,
Derrick Stolee <derrickstolee@github.com>,
Derrick Stolee <derrickstolee@github.com>
Subject: [PATCH v4 1/4] hashfile: allow skipping the hash function
Date: Fri, 16 Dec 2022 15:31:15 +0000 [thread overview]
Message-ID: <c99470d46763cdfbde01629888c379c448b3579d.1671204678.git.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.1439.v4.git.1671204678.gitgitgadget@gmail.com>
From: Derrick Stolee <derrickstolee@github.com>
The hashfile API is useful for generating files that include a trailing
hash of the file's contents up to that point. Using such a hash is
helpful for verifying the file for corruption-at-rest, such as a faulty
drive causing flipped bits.
Git's index file includes this trailing hash, so it uses a 'struct
hashfile' to handle the I/O to the file. This was very convenient to
allow using the hashfile methods during these operations.
However, hashing the file contents during write comes at a performance
penalty. It's slower to hash the bytes on their way to the disk than
without that step. This problem is made worse by the replacement of
hardware-accelerated SHA1 computations with the software-based sha1dc
computation.
This write cost is significant, and the checksum capability is likely
not worth that cost for such a short-lived file. The index is rewritten
frequently and the only time the checksum is checked is during 'git
fsck'. Thus, it would be helpful to allow a user to opt-out of the hash
computation.
We first need to allow Git to opt-out of the hash computation in the
hashfile API. The buffered writes of the API are still helpful, so it
makes sense to make the change here.
Introduce a new 'skip_hash' option to 'struct hashfile'. When set, the
update_fn and final_fn members of the_hash_algo are skipped. When
finalizing the hashfile, the trailing hash is replaced with the null
hash.
This use of a trailing null hash would be desireable in either case,
since we do not want to special case a file format to have a different
length depending on whether it was hashed or not. When the final bytes
of a file are all zero, we can infer that it was written without
hashing, and thus that verification is not available as a check for file
consistency. This also means that we could easily toggle hashing for any
file format we desire.
A version of this patch has existed in the microsoft/git fork since
2017 [1] (the linked commit was rebased in 2018, but the original dates
back to January 2017). Here, the change to make the index use this fast
path is delayed until a later change.
[1] https://github.com/microsoft/git/commit/21fed2d91410f45d85279467f21d717a2db45201
Co-authored-by: Kevin Willford <kewillf@microsoft.com>
Signed-off-by: Kevin Willford <kewillf@microsoft.com>
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
csum-file.c | 14 +++++++++++---
csum-file.h | 7 +++++++
2 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/csum-file.c b/csum-file.c
index 59ef3398ca2..cce13c0f047 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -45,7 +45,8 @@ void hashflush(struct hashfile *f)
unsigned offset = f->offset;
if (offset) {
- the_hash_algo->update_fn(&f->ctx, f->buffer, offset);
+ if (!f->skip_hash)
+ the_hash_algo->update_fn(&f->ctx, f->buffer, offset);
flush(f, f->buffer, offset);
f->offset = 0;
}
@@ -64,7 +65,12 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,
int fd;
hashflush(f);
- the_hash_algo->final_fn(f->buffer, &f->ctx);
+
+ if (f->skip_hash)
+ hashclr(f->buffer);
+ else
+ the_hash_algo->final_fn(f->buffer, &f->ctx);
+
if (result)
hashcpy(result, f->buffer);
if (flags & CSUM_HASH_IN_STREAM)
@@ -108,7 +114,8 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)
* the hashfile's buffer. In this block,
* f->offset is necessarily zero.
*/
- the_hash_algo->update_fn(&f->ctx, buf, nr);
+ if (!f->skip_hash)
+ the_hash_algo->update_fn(&f->ctx, buf, nr);
flush(f, buf, nr);
} else {
/*
@@ -153,6 +160,7 @@ static struct hashfile *hashfd_internal(int fd, const char *name,
f->tp = tp;
f->name = name;
f->do_crc = 0;
+ f->skip_hash = 0;
the_hash_algo->init_fn(&f->ctx);
f->buffer_len = buffer_len;
diff --git a/csum-file.h b/csum-file.h
index 0d29f528fbc..793a59da12b 100644
--- a/csum-file.h
+++ b/csum-file.h
@@ -20,6 +20,13 @@ struct hashfile {
size_t buffer_len;
unsigned char *buffer;
unsigned char *check_buffer;
+
+ /**
+ * If non-zero, skip_hash indicates that we should
+ * not actually compute the hash for this hashfile and
+ * instead only use it as a buffered write.
+ */
+ int skip_hash;
};
/* Checkpoint */
--
gitgitgadget
next prev parent reply other threads:[~2022-12-16 15:31 UTC|newest]
Thread overview: 59+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-12-07 17:25 [PATCH 0/4] Optionally skip hashing index on write Derrick Stolee via GitGitGadget
2022-12-07 17:25 ` [PATCH 1/4] hashfile: allow skipping the hash function Derrick Stolee via GitGitGadget
2022-12-07 22:13 ` Ævar Arnfjörð Bjarmason
2022-12-08 7:32 ` Jeff King
2022-12-07 17:25 ` [PATCH 2/4] read-cache: add index.skipHash config option Derrick Stolee via GitGitGadget
2022-12-07 18:59 ` Eric Sunshine
2022-12-12 13:59 ` Derrick Stolee
2022-12-12 18:55 ` Eric Sunshine
2022-12-07 22:25 ` Ævar Arnfjörð Bjarmason
2022-12-07 23:06 ` Ævar Arnfjörð Bjarmason
2022-12-08 0:05 ` Junio C Hamano
2022-12-12 14:05 ` Derrick Stolee
2022-12-12 18:01 ` Ævar Arnfjörð Bjarmason
2022-12-07 17:25 ` [PATCH 3/4] test-lib-functions: add helper for trailing hash Derrick Stolee via GitGitGadget
2022-12-07 22:27 ` Ævar Arnfjörð Bjarmason
2022-12-12 14:10 ` Derrick Stolee
2022-12-07 17:25 ` [PATCH 4/4] features: feature.manyFiles implies fast index writes Derrick Stolee via GitGitGadget
2022-12-07 22:30 ` Ævar Arnfjörð Bjarmason
2022-12-12 14:18 ` Derrick Stolee
2022-12-12 18:27 ` Ævar Arnfjörð Bjarmason
2022-12-07 23:27 ` [PATCH 0/4] Optionally skip hashing index on write Junio C Hamano
2022-12-07 23:42 ` Ævar Arnfjörð Bjarmason
2022-12-08 16:38 ` Derrick Stolee
2022-12-12 22:22 ` Jacob Keller
2022-12-12 16:31 ` [PATCH v2 " Derrick Stolee via GitGitGadget
2022-12-12 16:31 ` [PATCH v2 1/4] hashfile: allow skipping the hash function Derrick Stolee via GitGitGadget
2022-12-12 16:31 ` [PATCH v2 2/4] read-cache: add index.skipHash config option Derrick Stolee via GitGitGadget
2022-12-12 16:31 ` [PATCH v2 3/4] test-lib-functions: add helper for trailing hash Derrick Stolee via GitGitGadget
2022-12-12 18:14 ` SZEDER Gábor
2022-12-13 0:55 ` Junio C Hamano
2022-12-17 17:37 ` SZEDER Gábor
2022-12-12 16:31 ` [PATCH v2 4/4] features: feature.manyFiles implies fast index writes Derrick Stolee via GitGitGadget
2022-12-15 15:06 ` [PATCH v3 0/4] Optionally skip hashing index on write Derrick Stolee via GitGitGadget
2022-12-15 15:06 ` [PATCH v3 1/4] hashfile: allow skipping the hash function Derrick Stolee via GitGitGadget
2022-12-15 15:06 ` [PATCH v3 2/4] read-cache: add index.skipHash config option Derrick Stolee via GitGitGadget
2022-12-15 16:12 ` Ævar Arnfjörð Bjarmason
2022-12-15 15:06 ` [PATCH v3 3/4] test-lib-functions: add helper for trailing hash Derrick Stolee via GitGitGadget
2022-12-15 15:07 ` [PATCH v3 4/4] features: feature.manyFiles implies fast index writes Derrick Stolee via GitGitGadget
2022-12-15 15:56 ` [PATCH v3 0/4] Optionally skip hashing index on write Ævar Arnfjörð Bjarmason
2022-12-16 13:41 ` Derrick Stolee
2022-12-16 15:31 ` [PATCH v4 " Derrick Stolee via GitGitGadget
2022-12-16 15:31 ` Derrick Stolee via GitGitGadget [this message]
2022-12-16 15:31 ` [PATCH v4 2/4] read-cache: add index.skipHash config option Derrick Stolee via GitGitGadget
2022-12-16 15:31 ` [PATCH v4 3/4] test-lib-functions: add helper for trailing hash Derrick Stolee via GitGitGadget
2022-12-16 15:31 ` [PATCH v4 4/4] features: feature.manyFiles implies fast index writes Derrick Stolee via GitGitGadget
2022-12-16 15:43 ` [PATCH v4 0/4] Optionally skip hashing index on write Ævar Arnfjörð Bjarmason
2023-01-06 15:33 ` Derrick Stolee
2023-01-06 22:45 ` Junio C Hamano
2023-01-06 23:40 ` Derrick Stolee
2023-01-09 17:15 ` Ævar Arnfjörð Bjarmason
2023-01-09 18:00 ` Derrick Stolee
2023-01-09 19:22 ` Ævar Arnfjörð Bjarmason
2023-01-06 16:31 ` [PATCH v5 " Derrick Stolee via GitGitGadget
2023-01-06 16:31 ` [PATCH v5 1/4] hashfile: allow skipping the hash function Derrick Stolee via GitGitGadget
2023-01-06 16:31 ` [PATCH v5 2/4] read-cache: add index.skipHash config option Derrick Stolee via GitGitGadget
2023-01-06 16:31 ` [PATCH v5 3/4] test-lib-functions: add helper for trailing hash Derrick Stolee via GitGitGadget
2023-01-06 16:31 ` [PATCH v5 4/4] features: feature.manyFiles implies fast index writes Derrick Stolee via GitGitGadget
2023-01-15 9:31 ` [PATCH v5 0/4] Optionally skip hashing index on write Junio C Hamano
2023-01-17 14:49 ` Derrick Stolee
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c99470d46763cdfbde01629888c379c448b3579d.1671204678.git.gitgitgadget@gmail.com \
--to=gitgitgadget@gmail.com \
--cc=avarab@gmail.com \
--cc=derrickstolee@github.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=jacob.keller@gmail.com \
--cc=newren@gmail.com \
--cc=vdye@github.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).