Why 10k remotes? Was: Re: [RFC/PATCH 0/5] remote: eliminate remote->{fetch,push}_refspec and lazy parsing of refspecs

Why 10k remotes? Was: Re: [RFC/PATCH 0/5] remote: eliminate remote->{fetch,push}_refspec and lazy parsing of refspecs

[SZEDER Gábor]

On Sat, Jun 17, 2017 at 2:33 PM, Jeff King <peff@peff.net> wrote:
> On Sat, Jun 17, 2017 at 02:25:39PM +0200, SZEDER Gábor wrote:
>> Indeed.  Even in my repos with close to 10k remotes the amount of
>> memory wasted by the duplicated refspecs is not an problem, there are
>> more pressing issues there.
>
> This is sort of a side note, but I'm curious why you need 10k remotes.

For fun! :o)

I have a couple of repos containing all branches from all forks (and
forks of forks, recursively) of a few interesting repos on GitHub,
where each fork is a remote.

  (Why on earth would I want to do that?
  Sometimes it's good to see what others are up to.  Apparently a lot
  of people fork projects, push their changes, but then never upstream
  them.
  Such an all-forks-in-one repo allows me to run e.g. 'git log --all
  -p master.. relevant.c' and then search its output for changes in
  interesting functions (thankfully function names are included in
  hunk headers; alas line log doesn't work with --all).  Occasionally
  this unearths some treasures: other people's commits and branches
  scratching the same itch that I was about to scratch, or at least
  solving part of my problem and I can build on top of them.)

> We used to do this at GitHub as part of our fork storage (the shared
> repository had each fork as a remote). We fixed the quadratic issues
> like d0da003d5 (use a hashmap to make remotes faster, 2014-07-29). But
> even the linear work to read the config is noticeable (and hits you on
> every command, even ones that don't care about remotes).

Indeed, that's one of those "more pressing issues".
I worked it around by storing the remotes in a separate config file
which is not included from .git/config, because they're not needed for
local operations.  And when I occasionally do want to fetch, then I
run 'git -c include.path=..../.git/config.forks fetch ...'.

> Now instead we
> pass the refspecs directly to fetch whenever move objects between the
> storage repos. They were the same for every remote anyway (and I'd guess
> that is true, too, of your 10k remotes).

I do have different fetch refspecs for every remote, i.e. the repo
'github.com/user/repo' has '+refs/heads/*:refs/forks/user/repo/*'.

Incidentally, this was what triggered
sg/clone-refspec-from-command-line-config back then, because 'git
clone' didn't grab 'refs/forks/*', and the means advertised in the
docs to do so didn't work.


Gábor

Re: Why 10k remotes? Was: Re: [RFC/PATCH 0/5] remote: eliminate remote->{fetch,push}_refspec and lazy parsing of refspecs

[Jeff King]

On Wed, Jun 21, 2017 at 02:21:19AM +0200, SZEDER Gábor wrote:

>   Such an all-forks-in-one repo allows me to run e.g. 'git log --all
>   -p master.. relevant.c' and then search its output for changes in
>   interesting functions (thankfully function names are included in
>   hunk headers; alas line log doesn't work with --all).  Occasionally
>   this unearths some treasures: other people's commits and branches
>   scratching the same itch that I was about to scratch, or at least
>   solving part of my problem and I can build on top of them.)

OK, that's indeed an interesting use case.

> > Now instead we
> > pass the refspecs directly to fetch whenever move objects between the
> > storage repos. They were the same for every remote anyway (and I'd guess
> > that is true, too, of your 10k remotes).
> 
> I do have different fetch refspecs for every remote, i.e. the repo
> 'github.com/user/repo' has '+refs/heads/*:refs/forks/user/repo/*'.

Right, ours have different refspecs, too. But they can be generated
programatically. So we just do:

  git fetch ../$id +refs/*:refs/forks/$id/*

without bothering to have a configured remote.

-Peff