Fetching a lot of repos

From: "Soni \"They/Them\" L." <fakedme@gmail.com>
To: git@vger.kernel.org
Subject: Fetching a lot of repos
Date: Sun, 7 Jun 2020 17:04:03 -0300	[thread overview]
Message-ID: <ef6f2275-536b-f4dc-9012-0467b1a4134a@gmail.com> (raw)

For... reasons, I need to fetch a lot of repos, and fetching them one by 
one is *extremely* slow, often taking upwards of 30 minutes.

So I decided to try something different. My first attempt was a complete 
failure:

-----

[soniex2@soniex-pc multigit]$ git fetch 
https://soniex2.autistic.space/git-repos/ganarchy.git +HEAD:repo_a & git 
fetch https://github.com/ganarchy/GAnarchy +HEAD:repo_b & git fetch 
https://cybre.tech/SoniEx2/ganarchy +HEAD:repo_c & git fetch 
https://soniex2.autistic.space/git-repos/abdl.git +HEAD:repo_d & git 
fetch https://cybre.tech/SoniEx2/rust.hexchat.hexchat-plugin +HEAD:repo_e
[1] 2236
[2] 2237
[3] 2238
[4] 2239
remote: Enumerating objects: 87, done.
remote: Total 87 (delta 0), reused 0 (delta 0), pack-reused 87
Unpacking objects: 100% (87/87), 36.06 KiB | 225.00 KiB/s, done.
 From https://github.com/ganarchy/GAnarchy
  * [new ref]                    -> repo_b
 From https://cybre.tech/SoniEx2/ganarchy
  * [new ref]                    -> repo_c
warning: no common commits
remote: Counting objects: 113, done.
remote: Compressing objects: 100% (74/74), done.
remote: Total 113 (delta 48), reused 88 (delta 38)
Receiving objects: 100% (113/113), 30.07 KiB | 138.00 KiB/s, done.
Resolving deltas: 100% (48/48), done.
 From https://cybre.tech/SoniEx2/rust.hexchat.hexchat-plugin
  * [new ref]                    -> repo_e
[2]   Done                    git fetch 
https://github.com/ganarchy/GAnarchy +HEAD:repo_b
[3]-  Done                    git fetch 
https://cybre.tech/SoniEx2/ganarchy +HEAD:repo_c
[soniex2@soniex-pc multigit]$ error: unable to write file 
.git/objects/d2/5baa9c0a78b0007a34a569b774d983b905f0b5: No such file or 
directory
error: unable to write file 
.git/objects/ba/f9414a35a2f48ed1b22644fd4522272fb4bc66: No such file or 
directory
error: unable to write sha1 filename 
.git/objects/ba/f9414a35a2f48ed1b22644fd4522272fb4bc66
error: Unable to find baf9414a35a2f48ed1b22644fd4522272fb4bc66 under 
https://soniex2.autistic.space/git-repos/abdl.git
Fetching objects: 12, done.
Cannot obtain needed blob baf9414a35a2f48ed1b22644fd4522272fb4bc66
while processing commit 3f9f66712aaa071bd3bb32c46e1e4dc1fed13378.
error: fetch failed.
Fetching objects: 78, done.
 From https://soniex2.autistic.space/git-repos/ganarchy
  * [new ref]                    -> repo_a

-----

So I figured, "okay this is a git gc issue", and started over (rm -rf 
.git, git init) and turned off the GC (git config --local gc.auto 0, and 
that long command to run 5 git fetch at the same time). At first, it 
seemed to work fine, but then...

-----

$ git gc --aggressive
Enumerating objects: 365, done.
error: object file 
.git/objects/ba/f9414a35a2f48ed1b22644fd4522272fb4bc66 is empty
Counting objects: 100% (365/365), done.
Delta compression using up to 4 threads
Compressing objects: 100% (357/357), done.
error: object file 
.git/objects/ba/f9414a35a2f48ed1b22644fd4522272fb4bc66 is empty
fatal: loose object baf9414a35a2f48ed1b22644fd4522272fb4bc66 (stored in 
.git/objects/ba/f9414a35a2f48ed1b22644fd4522272fb4bc66) is corrupt
fatal: failed to run repack

-----

Well that didn't work did it... I'm not sure what to do about this, but 
I kinda need it to work, and it's currently not working. How can I make 
it work?