From: Jeff King <peff@peff.net>
To: Shawn Pearce <spearce@spearce.org>
Cc: Stefan Beller <sbeller@google.com>, git <git@vger.kernel.org>
Subject: Re: reftable: new ref storage format
Date: Fri, 14 Jul 2017 16:10:41 -0400 [thread overview]
Message-ID: <20170714201040.hwrr5gwrc23lp3jt@sigill.intra.peff.net> (raw)
In-Reply-To: <CAJo=hJv7kaT3m6k1nz1-tGuVAMmgnrS0dcfycGfE3PyXjG3xRA@mail.gmail.com>
On Thu, Jul 13, 2017 at 05:27:44PM -0700, Shawn Pearce wrote:
> > We _could_ consider gzipping individual blocks of
> > a reftable (or any structure that allows you to search to a
> > constant-sized block and do a linear search from there). But given that
> > they're in the same ballpark, I'm happy with whatever ends up the
> > simplest to code and debug. ;)
>
> This does help to shrink the file, e.g. it drops from 28M to 23M.
>
> It makes it more CPU costly to access a block, as we have to inflate
> that to walk through the records. It also messes with alignment. When
> you touch a block, that may be straddling two virtual memory pages in
> your kernel/filesystem.
>
> I'm not sure those penalties are worth the additional 16% reduction in size.
Yeah, I don't really care about a 16% reduction in size. I care much
more about simplicity of implementation and debugging. Using zlib is
kind-of simple to implement. But if you've ever had to debug it (or
figure out what is going on with maybe-corrupted output), it's pretty
nasty.
So I don't mind a more readable custom compression if it's not too
complicated. And especially if it buys us extra performance by being
able to jump around non-sequentially in the block.
-Peff
next prev parent reply other threads:[~2017-07-14 20:10 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-07-13 0:17 reftable: new ref storage format Shawn Pearce
2017-07-13 19:32 ` Jeff King
2017-07-13 19:56 ` Stefan Beller
2017-07-13 20:35 ` Jeff King
2017-07-13 21:51 ` Eric Wong
2017-07-14 0:27 ` Shawn Pearce
2017-07-14 20:10 ` Jeff King [this message]
2017-07-14 0:11 ` Shawn Pearce
2017-07-14 14:27 ` Dave Borowitz
2017-07-14 15:31 ` Shawn Pearce
2017-07-14 20:08 ` Jeff King
2017-07-16 6:01 ` Shawn Pearce
2017-07-16 10:01 ` Jeff King
2017-07-16 8:07 ` Johannes Sixt
2017-07-16 10:03 ` Jeff King
2017-07-16 10:10 ` Johannes Sixt
2017-07-16 17:33 ` Michael Haggerty
2017-07-16 19:43 ` Shawn Pearce
2017-07-16 21:12 ` Shawn Pearce
2017-07-16 21:13 ` Dave Borowitz
2017-07-16 21:31 ` Shawn Pearce
2017-07-18 1:43 ` Michael Haggerty
2017-07-18 18:53 ` Junio C Hamano
2017-07-23 22:56 ` Shawn Pearce
2017-07-23 23:03 ` Shawn Pearce
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170714201040.hwrr5gwrc23lp3jt@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=git@vger.kernel.org \
--cc=sbeller@google.com \
--cc=spearce@spearce.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).