Git Mailing List Archive on lore.kernel.org
 help / color / Atom feed
From: Ben Peart <peartben@gmail.com>
To: git@vger.kernel.org
Cc: gitster@pobox.com, benpeart@microsoft.com, pclouds@gmail.com,
	christian.couder@gmail.com, git@jeffhostetler.com,
	jonathantanmy@google.com
Subject: [RFC/PATCH v2 0/1] Add support for downloading blobs on demand
Date: Fri, 14 Jul 2017 09:26:50 -0400
Message-ID: <20170714132651.170708-1-benpeart@microsoft.com> (raw)

[-- Warning: decoded text below may be mangled --]
[-- Attachment #0: Type: text/plain; charset=UTF-8, Size: 3131 bytes --]

This patch series enables git to request missing objects when they are
not found in the object store. This is a fault-in model where the
"read-object" sub-process will fetch the missing object and store it in
the object store as a loose, alternate, or pack file. On success, git
will retry the operation and find the requested object.

It utilizes the recent sub-process refactoring to spawn a "read-object"
hook as a sub-process on the first request and then all subsequent
requests are made to that existing sub-process. This significantly
reduces the cost of making multiple request within a single git command.

An earlier version [1] of this patch series is pulled into [2] which
enables registering multiple ODB handlers which can be a simple command
or a sub-process. My hope is that this patch series can be completed to
meet the needs of the various efforts where faulting in missing objects
is required.

In the meantime, now that the sub-process refactoring has made it to
master, I’ve refactored this patch series to be as small and focused as
possible so it can be easily incorporated in other patch series where it
makes sense.

[3] has a need for faulting in missing objects and will be reworked to
take advantage of this patch series for its next iteration.

[4] has a similar function that uses a registered command instead of a
sub-process. Spawning a separate process for every missing object only
works if there are very few missing objects.  It does not scale well
when there are many missing objects (especially small objects like
commits and trees).

Patch is available here:
https://github.com/benpeart/git-for-windows/commits/read-object-process


[RFC] Add support for downloading blobs on demand 
[1] https://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/

[RFC/PATCH v4 00/49] Add initial experimental external ODB support 
[2] https://public-inbox.org/git/20170620075523.26961-1-chriscool@tuxfamily.org/

[PATCH v2 00/19] WIP object filtering for partial clone
[3] https://public-inbox.org/git/20170713173459.3559-1-git@jeffhostetler.com/

[RFC PATCH 0/3] Partial clone: promised blobs (formerly "missing blobs")
[4] https://public-inbox.org/git/cover.1499800530.git.jonathantanmy@google.com/

Ben Peart (1):
  sha1_file: Add support for downloading blobs on demand

 Documentation/technical/read-object-protocol.txt | 102 ++++++++++++
 cache.h                                          |   1 +
 config.c                                         |   5 +
 contrib/long-running-read-object/example.pl      | 114 +++++++++++++
 environment.c                                    |   1 +
 sha1_file.c                                      | 193 ++++++++++++++++++++++-
 t/t0410-read-object.sh                           |  27 ++++
 t/t0410/read-object                              | 114 +++++++++++++
 8 files changed, 550 insertions(+), 7 deletions(-)
 create mode 100644 Documentation/technical/read-object-protocol.txt
 create mode 100755 contrib/long-running-read-object/example.pl
 create mode 100755 t/t0410-read-object.sh
 create mode 100755 t/t0410/read-object

-- 
2.13.2.windows.1


             reply index

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-07-14 13:26 Ben Peart [this message]
2017-07-14 13:26 ` [PATCH v2 1/1] sha1_file: " Ben Peart
2017-07-14 15:18   ` Christian Couder
2017-07-17 18:06   ` Jonathan Tan
2017-07-17 20:09     ` Ben Peart
2017-07-17 23:24       ` Jonathan Tan

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170714132651.170708-1-benpeart@microsoft.com \
    --to=peartben@gmail.com \
    --cc=benpeart@microsoft.com \
    --cc=christian.couder@gmail.com \
    --cc=git@jeffhostetler.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=jonathantanmy@google.com \
    --cc=pclouds@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Git Mailing List Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/git/0 git/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 git git/ https://lore.kernel.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.git


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git