From: Junio C Hamano <gitster@pobox.com> To: Jonathan Nieder <jrnieder@gmail.com> Cc: Shawn Pearce <spearce@spearce.org>, Linus Torvalds <torvalds@linux-foundation.org>, Git Mailing List <git@vger.kernel.org>, Stefan Beller <sbeller@google.com>, bmwill@google.com, Jonathan Tan <jonathantanmy@google.com>, Jeff King <peff@peff.net>, David Lang <david@lang.hm>, "brian m. carlson" <sandals@crustytoothpaste.net>, Masaya Suzuki <masayasuzuki@google.com>, demerphq@gmail.com, The Keccak Team <keccak@noekeon.org>, Johannes Schindelin <Johannes.Schindelin@gmx.de> Subject: Re: [PATCH v4] technical doc: add a design doc for hash function transition Date: Fri, 29 Sep 2017 17:09:04 +0900 Message-ID: <xmqqk20ivsdb.fsf@gitster.mtv.corp.google.com> (raw) In-Reply-To: <xmqqo9puvy1w.fsf@gitster.mtv.corp.google.com> (Junio C. Hamano's message of "Fri, 29 Sep 2017 15:06:19 +0900") Junio C Hamano <gitster@pobox.com> writes: > Or perhaps we could. There is nothing that says a signed tag > created in the SHA-1 world must have the PGP/SHA-1 signature in the > NewHash payload---it could be split off of the object data and > stored in a local metadata cache, to be used only when we need to > convert it back to the SHA-1 world. > ... >> +The format allows round-trip conversion between newhash-content and >> +sha1-content. > > If it is a goal to eventually be able to lose SHA-1 compatibility > metadata from the objects, then we might want to remove SHA-1 based > signature bits (e.g. PGP trailer in signed tag, gpgsig header in the > commit object) from NewHash contents, and instead have them stored > in a side "metadata" table, only to be used while converting back. > I dunno if that is desirable. Let's keep it simple by ignoring all of the above. Even though leaving the sha1-gpgsig and other crufts would etch these compatibility metadata in objects forever, these remain only in objects that originate from SHA-1 world, or in objects created in the NewHash world only while the project participants still care about SHA-1 compatibility. Strictly speaking, it would be super nice if we can do without contaminating these newly created objects with SHA-1 compatibility headers, just like we wish to be able to drop the SHA-1 vs NewHash mapping table after projects participants stop careing about SHA-1 compatiblity, it may not be worth it. Of course, if we decide to spend a bit more brain cycle to design how we push these out of the object proper, the same solution would automatically allow us to omit SHA-1 compatibility headers from the objects that were converted from SHA-1 world. > >> + - A table of 4-byte CRC32 values of the packed object data, in the >> + order that the objects appear in the pack file. This is to allow >> + compressed data to be copied directly from pack to pack during >> + repacking without undetected data corruption. > > An obvious alternative would be to have the CRC32 checksum near > (e.g. immediately before) the object data in the packfile (as > opposed to the .idx file like this document specifies). I am not > sure what the pros and cons are between the two, though, and that is > why I mention the possiblity here. > > Hmm, as the corresponding packfile stores object data only in > NewHash content format, it is somewhat curious that this table that > stores CRC32 of the data appears in the "Tables for each object > format" section, as they would be identical, no? Unless I am > grossly misleading the spec, the checksum should either go outside > the "Tables for each object format" section but still in .idx, or > should be eliminated and become part of the packdata stream instead, > perhaps? Thinking about this a bit more, I think a single table per .idx file would be the right way to go, not a checksum immediately after or before the object data that is embedded in the pack stream. In the NewHash world (after this initial migration), we would want to be able to stream NewHash packstream that comes from the network straight to disk, which would mean these in-line CRC32 data would need to be sent over the wire (i.e. 4-byte per object sent); that is an unneeded overhead, as the packstream has its trailing checksum to protect the whole thing anyway.
next prev parent reply index Thread overview: 110+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-03-04 1:12 RFC: Another proposed hash function transition plan Jonathan Nieder 2017-03-05 2:35 ` Linus Torvalds 2017-03-06 0:26 ` brian m. carlson 2017-03-06 18:24 ` Brandon Williams 2017-06-15 10:30 ` Which hash function to use, was " Johannes Schindelin 2017-06-15 11:05 ` Mike Hommey 2017-06-15 13:01 ` Jeff King 2017-06-15 16:30 ` Ævar Arnfjörð Bjarmason 2017-06-15 19:34 ` Johannes Schindelin 2017-06-15 21:59 ` Adam Langley 2017-06-15 22:41 ` brian m. carlson 2017-06-15 23:36 ` Ævar Arnfjörð Bjarmason 2017-06-16 0:17 ` brian m. carlson 2017-06-16 6:25 ` Ævar Arnfjörð Bjarmason 2017-06-16 13:24 ` Johannes Schindelin 2017-06-16 17:38 ` Adam Langley 2017-06-16 20:52 ` Junio C Hamano 2017-06-16 21:12 ` Junio C Hamano 2017-06-16 21:24 ` Jonathan Nieder 2017-06-16 21:39 ` Ævar Arnfjörð Bjarmason 2017-06-16 20:42 ` Jeff King 2017-06-19 9:26 ` Johannes Schindelin 2017-06-15 21:10 ` Mike Hommey 2017-06-16 4:30 ` Jeff King 2017-06-15 17:36 ` Brandon Williams 2017-06-15 19:20 ` Junio C Hamano 2017-06-15 19:13 ` Jonathan Nieder 2017-03-07 0:17 ` RFC v3: " Jonathan Nieder 2017-03-09 19:14 ` Shawn Pearce 2017-03-09 20:24 ` Jonathan Nieder 2017-03-10 19:38 ` Jeff King 2017-03-10 19:55 ` Jonathan Nieder 2017-09-28 4:43 ` [PATCH v4] technical doc: add a design doc for hash function transition Jonathan Nieder 2017-09-29 6:06 ` Junio C Hamano 2017-09-29 8:09 ` Junio C Hamano [this message] 2017-09-29 17:34 ` Jonathan Nieder 2017-10-02 8:25 ` Junio C Hamano 2017-10-02 19:41 ` Jason Cooper 2017-10-02 9:02 ` Junio C Hamano 2017-10-02 19:23 ` Jason Cooper 2017-10-03 5:40 ` Junio C Hamano 2017-10-03 13:08 ` Jason Cooper 2017-10-04 1:44 ` Junio C Hamano 2017-09-06 6:28 ` RFC v3: Another proposed hash function transition plan Junio C Hamano 2017-09-08 2:40 ` Junio C Hamano 2017-09-08 3:34 ` Jeff King 2017-09-11 18:59 ` Brandon Williams 2017-09-13 12:05 ` Johannes Schindelin 2017-09-13 13:43 ` demerphq 2017-09-13 22:51 ` Jonathan Nieder 2017-09-14 18:26 ` Johannes Schindelin 2017-09-14 18:40 ` Jonathan Nieder 2017-09-14 22:09 ` Johannes Schindelin 2017-09-13 23:30 ` Linus Torvalds 2017-09-14 18:45 ` Johannes Schindelin 2017-09-18 12:17 ` Gilles Van Assche 2017-09-18 22:16 ` Johannes Schindelin 2017-09-19 16:45 ` Gilles Van Assche 2017-09-29 13:17 ` Johannes Schindelin 2017-09-29 14:54 ` Joan Daemen 2017-09-29 22:33 ` Johannes Schindelin 2017-09-30 22:02 ` Joan Daemen 2017-10-02 14:26 ` Johannes Schindelin 2017-09-18 22:25 ` Jonathan Nieder 2017-09-26 17:05 ` Jason Cooper 2017-09-26 22:11 ` Johannes Schindelin 2017-09-26 22:25 ` [PATCH] technical doc: add a design doc for hash function transition Stefan Beller 2017-09-26 23:38 ` Jonathan Nieder 2017-09-26 23:51 ` RFC v3: Another proposed hash function transition plan Jonathan Nieder 2017-10-02 14:54 ` Jason Cooper 2017-10-02 16:50 ` Brandon Williams 2017-10-02 14:00 ` Jason Cooper 2017-10-02 17:18 ` Linus Torvalds 2017-10-02 19:37 ` Jeff King 2017-09-13 16:30 ` Jonathan Nieder 2017-09-13 21:52 ` Junio C Hamano 2017-09-13 22:07 ` Stefan Beller 2017-09-13 22:18 ` Jonathan Nieder 2017-09-14 2:13 ` Junio C Hamano 2017-09-14 15:23 ` Johannes Schindelin 2017-09-14 15:45 ` demerphq 2017-09-14 22:06 ` Johannes Schindelin 2017-09-13 22:15 ` Junio C Hamano 2017-09-13 22:27 ` Jonathan Nieder 2017-09-14 2:10 ` Junio C Hamano 2017-09-14 12:39 ` Johannes Schindelin 2017-09-14 16:36 ` Brandon Williams 2017-09-14 18:49 ` Jonathan Nieder 2017-09-15 20:42 ` Philip Oakley 2017-03-05 11:02 ` RFC: " David Lang [not found] ` <CA+dhYEXHbQfJ6KUB1tWS9u1MLEOJL81fTYkbxu4XO-i+379LPw@mail.gmail.com> 2017-03-06 9:43 ` Jeff King 2017-03-06 23:40 ` Jonathan Nieder 2017-03-07 0:03 ` Mike Hommey 2017-03-06 8:43 ` Jeff King 2017-03-06 18:39 ` Jonathan Tan 2017-03-06 19:22 ` Linus Torvalds 2017-03-06 19:59 ` Brandon Williams 2017-03-06 21:53 ` Junio C Hamano 2017-03-07 8:59 ` Jeff King 2017-03-06 18:43 ` Junio C Hamano 2017-03-07 18:57 ` Ian Jackson 2017-03-07 19:15 ` Linus Torvalds 2017-03-08 11:20 ` Ian Jackson 2017-03-08 15:37 ` Johannes Schindelin 2017-03-13 9:24 ` The Keccak Team 2017-03-13 17:48 ` Jonathan Nieder 2017-03-13 18:34 ` ankostis 2017-03-17 11:07 ` Johannes Schindelin 2017-03-08 15:40 Johannes Schindelin 2017-03-20 5:21 ` Use base32? Jason Hennessey 2017-03-20 5:58 ` Michael Steuer 2017-03-20 8:05 ` Jacob Keller 2017-03-21 3:07 ` Michael Steuer
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=xmqqk20ivsdb.fsf@gitster.mtv.corp.google.com \ --to=gitster@pobox.com \ --cc=Johannes.Schindelin@gmx.de \ --cc=bmwill@google.com \ --cc=david@lang.hm \ --cc=demerphq@gmail.com \ --cc=git@vger.kernel.org \ --cc=jonathantanmy@google.com \ --cc=jrnieder@gmail.com \ --cc=keccak@noekeon.org \ --cc=masayasuzuki@google.com \ --cc=peff@peff.net \ --cc=sandals@crustytoothpaste.net \ --cc=sbeller@google.com \ --cc=spearce@spearce.org \ --cc=torvalds@linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Git Mailing List Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/git/0 git/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 git git/ https://lore.kernel.org/git \ git@vger.kernel.org public-inbox-index git Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.git AGPL code for this site: git clone https://public-inbox.org/public-inbox.git