All of lore.kernel.org
 help / color / mirror / Atom feed
From: Felipe Contreras <felipe.contreras@gmail.com>
To: git@vger.kernel.org
Cc: Junio C Hamano <gitster@pobox.com>,
	Antoine Pelisse <apelisse@gmail.com>,
	Johannes Schindelin <johannes.schindelin@gmx.de>,
	Felipe Contreras <felipe.contreras@gmail.com>
Subject: [PATCH 2/4] fast-export: improve speed by skipping blobs
Date: Thu,  2 May 2013 23:31:40 -0500	[thread overview]
Message-ID: <1367555502-4706-3-git-send-email-felipe.contreras@gmail.com> (raw)
In-Reply-To: <1367555502-4706-1-git-send-email-felipe.contreras@gmail.com>

We don't care about blobs, or any object other than commits, but in
order to find the type of object, we are parsing the whole thing, which
is slow, specially in big repositories with lots of big files.

There's no need for that, we can query the object information with
sha1_object_info();

Before this, loading the objects of a fresh emacs import, with 260598
blobs took 14 minutes, after this patch, it takes 3 seconds.

This is the way fast-import does it. Also die if the object is not
found (like fast-import).

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 builtin/fast-export.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/builtin/fast-export.c b/builtin/fast-export.c
index a4dee14..a5b8da8 100644
--- a/builtin/fast-export.c
+++ b/builtin/fast-export.c
@@ -613,6 +613,7 @@ static void import_marks(char *input_file)
 		char *line_end, *mark_end;
 		unsigned char sha1[20];
 		struct object *object;
+		enum object_type type;
 
 		line_end = strchr(line, '\n');
 		if (line[0] != ':' || !line_end)
@@ -627,17 +628,19 @@ static void import_marks(char *input_file)
 		if (last_idnum < mark)
 			last_idnum = mark;
 
-		object = parse_object(sha1);
-		if (!object)
+		type = sha1_object_info(sha1, NULL);
+		if (type < 0)
+			die("object not found: %s", sha1_to_hex(sha1));
+
+		if (type != OBJ_COMMIT)
+			/* only commits */
 			continue;
 
+		object = parse_object(sha1);
+
 		if (object->flags & SHOWN)
 			error("Object %s already has a mark", sha1_to_hex(sha1));
 
-		if (object->type != OBJ_COMMIT)
-			/* only commits */
-			continue;
-
 		mark_object(object, mark);
 
 		object->flags |= SHOWN;
-- 
1.8.3.rc0.401.g45bba44

  parent reply	other threads:[~2013-05-03  4:33 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-05-03  4:31 [PATCH 0/4] fast-export: speed improvements Felipe Contreras
2013-05-03  4:31 ` [PATCH 1/4] fast-{import,export}: use get_sha1_hex() directly Felipe Contreras
2013-05-03 21:50   ` Junio C Hamano
2013-05-03  4:31 ` Felipe Contreras [this message]
2013-05-03 21:51   ` [PATCH 2/4] fast-export: improve speed by skipping blobs Junio C Hamano
2013-05-03  4:31 ` [PATCH 3/4] fast-export: don't parse all the commits Felipe Contreras
2013-05-03 21:54   ` Junio C Hamano
2013-05-04  0:06     ` Felipe Contreras
2013-05-04 19:22       ` Junio C Hamano
2013-05-03  4:31 ` [PATCH 4/4] fast-import: only store commit objects Felipe Contreras
2013-05-03 17:56   ` Thomas Rast
2013-05-03 18:23     ` Felipe Contreras
2013-05-06 10:28       ` Michael Haggerty
2013-05-06 10:32         ` Thomas Rast
2013-05-06 10:45           ` Michael Haggerty
2013-05-06 15:18             ` Junio C Hamano
2013-05-06 21:19               ` Felipe Contreras
2013-05-06 21:36                 ` Felipe Contreras
2013-05-07  3:14                   ` Michael Haggerty
2013-05-07  4:32                     ` Johannes Schindelin
2013-05-07  4:36                       ` Felipe Contreras
2013-05-07  2:58                 ` Michael Haggerty
2013-05-07  4:37                   ` Felipe Contreras
2013-05-06 21:04             ` Felipe Contreras
2013-05-07  3:27               ` Michael Haggerty
2013-05-07  4:39                 ` Johannes Schindelin
2013-05-07  4:49                   ` Felipe Contreras
2013-05-07  4:47                 ` Felipe Contreras
2013-05-07  6:47                   ` Michael Haggerty
2013-05-07  7:07                     ` Felipe Contreras
2013-05-07  7:12                 ` Junio C Hamano
2013-05-07  7:34                   ` Michael Haggerty
2013-05-06 12:20           ` Johannes Schindelin
2013-05-06 21:06             ` Felipe Contreras
2013-05-03 22:08     ` Junio C Hamano
2013-05-03 22:19       ` Felipe Contreras
2013-05-03 23:45         ` Junio C Hamano
2013-05-04  0:01           ` Felipe Contreras

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1367555502-4706-3-git-send-email-felipe.contreras@gmail.com \
    --to=felipe.contreras@gmail.com \
    --cc=apelisse@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=johannes.schindelin@gmx.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.