Linux maintainer tooling and workflows
 help / color / Atom feed
* grokmirror-2.0 is available
@ 2020-09-21 17:06 Konstantin Ryabitsev
  0 siblings, 0 replies; only message in thread
From: Konstantin Ryabitsev @ 2020-09-21 17:06 UTC (permalink / raw)
  To: git; +Cc: tools

[-- Attachment #1: Type: text/plain, Size: 4071 bytes --]


I am pleased to announce version 2.0 of's git mirroring 
software, grokmirror. This is a major rewrite that intentionally breaks 
the upgrade path from grokmirror-1.x due to significant backend changes 
requiring replica administrator's thoughtful consideration -- please see 
the UPGRADING.rst document provided with this release.

## New in grokmirror-2.0

- Drop support for python < 3.6
- Introduce "object storage" repositories that benefit from git-pack
  delta islands and improve overall disk storage footprint (results will 
  directly depend on the number of forks).
- Drop dependency on GitPython: use git calls directly for all operations
- Remove progress bars to slim down dependencies (drops enlighten)
- Make grok-pull operate in daemon mode (with -o) (see contrib for
  systemd unit files). This is more efficient than the cron mode when
  run very frequently.
- Provide a socket listener for pubsub push updates (see contrib for
- Merge fsck.conf and repos.conf into a single config file. This
  requires creating a new configuration file after the upgrade. See
  UPGRADING.rst for details.
- Record and propagate HEAD position using the manifest file.
- Add grok-bundle command to create clone.bundle files for CDN-offloaded
  cloning (mostly used by Android's repo command).
- Add SELinux policy for EL7 (see contrib).

## Object Storage Repositories

Grokmirror 2.0 introduces the concept of "object storage repositories", which
aims to optimize how repository forks are stored on disk and served to the
cloning clients.

When grok-fsck runs, it will automatically recognize related repositories by
analyzing their root commits. If it finds two or more related repositories, it
will set up a unified "object storage" repo and fetch all refs from each
related repository into it.

For example, you can have two forks of linux.git:

and its fork:


Grok-fsck will set up an object storage repository and fetch all refs from both


Then both torvalds/linux.git and maintainer/linux.git with be configured to use
objstore/[random-guid-name].git via objects/info/alternates and repacked to
just contain metadata and no objects.

The alternates repository will be repacked with "delta islands" enabled,
which should help optimize clone operations for each "sibling"

Please see the example grokmirror.conf for more details about configuring
objstore repositories.

## Space savings using object storage repositories

Any disk space savings will depend on how many repositories are forks of 
each other. For, which already aggressively used 
alternates for all linux.git forks, we saw reduction from 60GB to 20GB 
for the entirety of content. On some of the systems, especially those containing a lot of pre-release 
forks of entire AOSP repo collections, we saw space usage go from 3TB to 
under 1TB.

## Stability

This release has proven pretty stable and has been operating on and a subset of systems for over the past 
month. However, since the trickiest part is initial repository 
conversion towards the use of object storage repos, we urge proceeding 
with caution. Please study the UPGRADING.rst document before making any 
changes to your infrastructure.

With all support questions, please email

Best regards,

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 235 bytes --]

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, back to index

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-21 17:06 grokmirror-2.0 is available Konstantin Ryabitsev

Linux maintainer tooling and workflows

Archives are clonable:
	git clone --mirror tools/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 tools tools/ \
	public-inbox-index tools

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone