From: Eric Wong <e@80x24.org>
To: git@vger.kernel.org
Subject: Re: Strategy to deal with slow cloners
Date: Mon, 19 Apr 2021 18:08:03 +0000 [thread overview]
Message-ID: <20210419180803.GA10171@dcvr> (raw)
In-Reply-To: <20210419124623.wwps2s35x2mrrhi6@nitro.local>
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
>
> I try to keep repositories routinely repacked and optimized for clones, in
> hopes that most operations needing lots of objects would be sending packs
> straight from disk. However, every now and again a client from a slow
> connection requests a large clone and then takes half a day downloading it,
> resulting in gigabytes of RAM being occupied by a temporary pack.
Yeah, I'm familiar with the problem.
> Are there any strategies to reduce RAM usage in such cases, other than
> vm.swappiness (which I'm not sure would work, since it's not a sleeping
> process)? Is there a way to write large temporary packs somewhere to disk
> before sendfile'ing them?
public-inbox-httpd actually switched buffering strategies in
2019 to favor hitting ENOSPC instead of ENOMEM :)
https://public-inbox.org/meta/20190629195951.32160-11-e@80x24.org/
It doesn't support sendfile, currently (I didn't want separate
HTTPS vs HTTP code paths), but that's probably not too big of a
deal, especially with slow clients.
It's capable of serving non-public-inbox coderepos (and running
cgit). Instead of configuring every [coderepo "..."] manually,
publicinbox.cgitrc can be set in ~/.public-inbox/config to
mass-configure [coderepo] sections. It's only lightly-tested
for my setup atm, though.
Mapping publicinbox.<name>.coderepo to [coderepo "..."]
entries for solver (blob reconstruction) isn't required;
it's a bit of a pain at a large scale and I haven't figured
out how to make it easier.
next prev parent reply other threads:[~2021-04-19 18:08 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-19 12:46 Strategy to deal with slow cloners Konstantin Ryabitsev
2021-04-19 18:08 ` Eric Wong [this message]
2021-04-21 20:08 ` Eric Wong
2021-04-20 14:52 ` Thomas Braun
2021-04-22 9:16 ` Ævar Arnfjörð Bjarmason
2021-04-23 10:02 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210419180803.GA10171@dcvr \
--to=e@80x24.org \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).