[regression] Newer gits cannot clone any remote repos

[regression] Newer gits cannot clone any remote repos

[Douglas Mencken]

*Any* git clone fails with:

fatal: premature end of pack file, 106 bytes missing
fatal: index-pack failed

At first, I tried 1.8.0, and it failed. Then I tried to build 1.7.10.5
then, and it worked. Then I tried 1.7.12.2, but it fails the same way
as 1.8.0.
So I decided to git bisect.

b8a2486f1524947f232f657e9f2ebf44e3e7a243 is the first bad commit
``index-pack: support multithreaded delta resolving''

--------------------------------------------------------------------------------

b8a2486f1524947f232f657e9f2ebf44e3e7a243 is the first bad commit
commit b8a2486f1524947f232f657e9f2ebf44e3e7a243
Author: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Date:   Sun May 6 19:31:55 2012 +0700

    index-pack: support multithreaded delta resolving

    This puts delta resolving on each base on a separate thread, one base
    cache per thread. Per-thread data is grouped in struct thread_local.
    When running with nr_threads == 1, no pthreads calls are made. The
    system essentially runs in non-thread mode.

    An experiment on a Xeon 24 core machine with git.git shows that
    performance does not increase proportional to the number of cores. So
    by default, we use maximum 3 cores. Some numbers with --threads from 1
    to 16:

    1..4
    real    0m8.003s  0m5.307s  0m4.321s  0m3.830s
    user    0m7.720s  0m8.009s  0m8.133s  0m8.305s
    sys     0m0.224s  0m0.372s  0m0.360s  0m0.360s

    5..8
    real    0m3.727s  0m3.604s  0m3.332s  0m3.369s
    user    0m9.361s  0m9.817s  0m9.525s  0m9.769s
    sys     0m0.584s  0m0.624s  0m0.540s  0m0.560s

    9..12
    real    0m3.036s  0m3.139s  0m3.177s  0m2.961s
    user    0m8.977s  0m10.205s 0m9.737s  0m10.073s
    sys     0m0.596s  0m0.680s  0m0.684s  0m0.680s

    13..16
    real    0m2.985s  0m2.894s  0m2.975s  0m2.971s
    user    0m9.825s  0m10.573s 0m10.833s 0m11.361s
    sys     0m0.788s  0m0.732s  0m0.904s  0m1.016s

    On an Intel dual core and linux-2.6.git

    1..4
    real    2m37.789s 2m7.963s  2m0.920s  1m58.213s
    user    2m28.415s 2m52.325s 2m50.176s 2m41.187s
    sys     0m7.808s  0m11.181s 0m11.224s 0m10.731s

    Thanks Ramsay Jones for troubleshooting and support on MinGW platform.

    Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
    Signed-off-by: Junio C Hamano <gitster@pobox.com>

:040000 040000 e5597fe041cd7985648030c8cf1cc35ea57786bb
6e64f3e76b350c04c9d0c435e0a53b89e0821339 M      Documentation
:100644 100644 cf2c40b44f8383d002235400660d76ef7f6de33c
e41955ff955693ddc78722c14fff2a6e98663a46 M      Makefile
:040000 040000 1e6b5ac5f11bf7ba3eff7b5be4410bad379feee6
85bc79414f00b7fc3698c069161ea10039641082 M      builtin
:040000 040000 04916545d959c8af772cd3a2de5e6b470529bea1
325c8c748dd41d469135db84c7621d8225cc1cdd M      t

$ git bisect log
    git bisect start
    # bad: [d8cf053dacb4f78920c112d10c7be21e4f5a5817] Git 1.7.12.2
    git bisect bad d8cf053dacb4f78920c112d10c7be21e4f5a5817
    # good: [785ee4960c3d334cbc2b17ab74d2cebdf1b4db64] Git 1.7.10.5
    git bisect good 785ee4960c3d334cbc2b17ab74d2cebdf1b4db64
    # bad: [d692d34653f74be6b16add3e993e957f33fe049b] Merge branch
'rs/git-blame-mapcar-mapc'
    git bisect bad d692d34653f74be6b16add3e993e957f33fe049b
    # good: [73ff8cf784b6ee447072fad6c06fd0eef0e9c9f6] Merge branch
'lp/diffstat-with-graph'
    git bisect good 73ff8cf784b6ee447072fad6c06fd0eef0e9c9f6
    # bad: [7903e66a3e10e98f29fc9524c13274376bae5303] Merge branch
'mh/test-keep-prove-cache'
    git bisect bad 7903e66a3e10e98f29fc9524c13274376bae5303
    # good: [499e7b31509cfbb59dcb2a046f8e2fd1a3e73d6f] Merge branch
'jc/install-no-hardlinks'
    git bisect good 499e7b31509cfbb59dcb2a046f8e2fd1a3e73d6f
    # bad: [2e464dd5b220d4d2d8a16d5c43efe7af3c9adda9] Merge branch
'rs/xdiff-lose-emit-func'
    git bisect bad 2e464dd5b220d4d2d8a16d5c43efe7af3c9adda9
    # good: [8fbe0db4fce68ed8239742b14e9d77d45201870b] Merge branch
'master' of git://github.com/git-l10n/git-po
    git bisect good 8fbe0db4fce68ed8239742b14e9d77d45201870b
    # bad: [2b26b65f9abc77c4af87626452005a73edda0c8f] git-svn: clarify
the referent of dcommit's optional argument
    git bisect bad 2b26b65f9abc77c4af87626452005a73edda0c8f
    # bad: [cc13431a49800a6a1d2b7ff0b94f67da0fecdbab] Merge branch
'nd/threaded-index-pack'
    git bisect bad cc13431a49800a6a1d2b7ff0b94f67da0fecdbab
    # bad: [b8a2486f1524947f232f657e9f2ebf44e3e7a243] index-pack:
support multithreaded delta resolving
    git bisect bad b8a2486f1524947f232f657e9f2ebf44e3e7a243
    # good: [5272f75587abb4cb396059ecbf1a6130bb2e69d3] index-pack:
restructure pack processing into three main functions
    git bisect good 5272f75587abb4cb396059ecbf1a6130bb2e69d3

$ git bisect reset
Previous HEAD position was 5272f75... index-pack: restructure pack
processing into three main functions
HEAD is now at d8cf053... Git 1.7.12.2

Re: [regression] Newer gits cannot clone any remote repos

[Kevin]

Maybe handy to say that you're on a Powerpc platform.

On Mon, Nov 12, 2012 at 5:32 PM, Douglas Mencken <dougmencken@gmail.com> wrote:
> *Any* git clone fails with:
>
> fatal: premature end of pack file, 106 bytes missing
> fatal: index-pack failed
>
> At first, I tried 1.8.0, and it failed. Then I tried to build 1.7.10.5
> then, and it worked. Then I tried 1.7.12.2, but it fails the same way
> as 1.8.0.
> So I decided to git bisect.
>

Re: [regression] Newer gits cannot clone any remote repos

[Douglas Mencken]

> Maybe handy to say that you're on a Powerpc platform.

My host, build and target is: powerpc-linux-gnu

Re: [regression] Newer gits cannot clone any remote repos

[Douglas Mencken]

> On Mon, Nov 12, 2012 at 12:12 PM, Kevin <ikke@ikke.info> wrote:
> > Maybe handy to say that you're on a Powerpc platform.

Oh, and yes, I'm on 2 x 2-core ("4-core") machine.

Re: [regression] Newer gits cannot clone any remote repos

[Ramsay Jones]

Douglas Mencken wrote:
> *Any* git clone fails with:
> 
> fatal: premature end of pack file, 106 bytes missing
> fatal: index-pack failed
> 
> At first, I tried 1.8.0, and it failed. Then I tried to build 1.7.10.5
> then, and it worked. Then I tried 1.7.12.2, but it fails the same way
> as 1.8.0.
> So I decided to git bisect.
> 
> b8a2486f1524947f232f657e9f2ebf44e3e7a243 is the first bad commit
> ``index-pack: support multithreaded delta resolving''

This looks like the same problem I had on cygwin, which lead to
commit c0f86547c ("index-pack: Disable threading on cygwin", 26-06-2012).

I didn't notice which platform you are on, but maybe you also have a
thread-unsafe pread()? Could you try re-building git with the
NO_THREAD_SAFE_PREAD build variable set?

HTH.

ATB,
Ramsay Jones

Re: [regression] Newer gits cannot clone any remote repos

[Douglas Mencken]

>  Could you try re-building git with the
> NO_THREAD_SAFE_PREAD build variable set?

Yeah! It works!!!

--- evil/Makefile
+++ good/Makefile
@@ -957,6 +957,7 @@
 	HAVE_PATHS_H = YesPlease
 	LIBC_CONTAINS_LIBINTL = YesPlease
 	HAVE_DEV_TTY = YesPlease
+	NO_THREAD_SAFE_PREAD = YesPlease
 endif
 ifeq ($(uname_S),GNU/kFreeBSD)
 	NO_STRLCPY = YesPlease

With this, I do have correctly working git clone.

Re: [regression] Newer gits cannot clone any remote repos

[Douglas Mencken]

By the way, it is the job of autotools to check such stuff, isn't it?

Re: [regression] Newer gits cannot clone any remote repos

[Torsten Bögershausen]

On 13.11.12 19:55, Ramsay Jones wrote:
> Douglas Mencken wrote:
>> *Any* git clone fails with:
>>
>> fatal: premature end of pack file, 106 bytes missing
>> fatal: index-pack failed
>>
>> At first, I tried 1.8.0, and it failed. Then I tried to build 1.7.10.5
>> then, and it worked. Then I tried 1.7.12.2, but it fails the same way
>> as 1.8.0.
>> So I decided to git bisect.
>>
>> b8a2486f1524947f232f657e9f2ebf44e3e7a243 is the first bad commit
>> ``index-pack: support multithreaded delta resolving''
> 
> This looks like the same problem I had on cygwin, which lead to
> commit c0f86547c ("index-pack: Disable threading on cygwin", 26-06-2012).
> 
> I didn't notice which platform you are on, but maybe you also have a
> thread-unsafe pread()? Could you try re-building git with the
> NO_THREAD_SAFE_PREAD build variable set?
> 
> HTH.
> 
> ATB,
> Ramsay Jones

This is interesting.
I had the same problem on a PowerPC 
(Old PowerBook G4 running Linux).

Using NO_THREAD_SAFE_PREAD helped, thanks for the hint.
(After recompiling without NO_THREAD_SAFE_PREAD I could clone
from this machine again, so the problem is not really reproducable)

Are there more people running PowerPC (on the server side) ?
/Torsten

 

Re: [regression] Newer gits cannot clone any remote repos

[Andreas Schwab]

Torsten Bögershausen <tboegi@web.de> writes:

> Are there more people running PowerPC (on the server side) ?

I cannot reproduce the problem (on openSUSE 12.2).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [regression] Newer gits cannot clone any remote repos

[Douglas Mencken]

> I cannot reproduce the problem (on openSUSE 12.2).

You do need multiple CPU/multi-core machine, as I got it.

Re: [regression] Newer gits cannot clone any remote repos

[Andreas Schwab]

Douglas Mencken <dougmencken@gmail.com> writes:

>> I cannot reproduce the problem (on openSUSE 12.2).
>
> You do need multiple CPU/multi-core machine, as I got it.

Which is what I have.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [regression] Newer gits cannot clone any remote repos

[Douglas Mencken]

>>> I cannot reproduce the problem (on openSUSE 12.2).
>>
>> You do need multiple CPU/multi-core machine, as I got it.
>
> Which is what I have.

Then try to build *vanilla* git 1.8.0, not OpenSuSE's one (with a lot
of patches inside srcrpm).

Re: [regression] Newer gits cannot clone any remote repos

[Andreas Schwab]

Douglas Mencken <dougmencken@gmail.com> writes:

>>>> I cannot reproduce the problem (on openSUSE 12.2).
>>>
>>> You do need multiple CPU/multi-core machine, as I got it.
>>
>> Which is what I have.
>
> Then try to build *vanilla* git 1.8.0,

Which is what I did.

> not OpenSuSE's one (with a lot of patches inside srcrpm).

Which lot of patches?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [regression] Newer gits cannot clone any remote repos

[Nguyen Thai Ngoc Duy]

On Wed, Nov 14, 2012 at 2:55 AM, Douglas Mencken <dougmencken@gmail.com> wrote:
>>  Could you try re-building git with the
>> NO_THREAD_SAFE_PREAD build variable set?
>
> Yeah! It works!!!
>
> --- evil/Makefile
> +++ good/Makefile
> @@ -957,6 +957,7 @@
>         HAVE_PATHS_H = YesPlease
>         LIBC_CONTAINS_LIBINTL = YesPlease
>         HAVE_DEV_TTY = YesPlease
> +       NO_THREAD_SAFE_PREAD = YesPlease
>  endif
>  ifeq ($(uname_S),GNU/kFreeBSD)
>         NO_STRLCPY = YesPlease
>
> With this, I do have correctly working git clone.

Sorry you had to figure that out the hard way. Could you make it a
proper patch? I'm surprised that Linux pread does not behave the same
way across platforms though. Or maybe it only happens with certain
Linux versions. What version are you using?
-- 
Duy

Re: [regression] Newer gits cannot clone any remote repos

[Ramsay Jones]

Douglas Mencken wrote:
>>  Could you try re-building git with the
>> NO_THREAD_SAFE_PREAD build variable set?
> 
> Yeah! It works!!!
> 
> --- evil/Makefile
> +++ good/Makefile
> @@ -957,6 +957,7 @@
>  	HAVE_PATHS_H = YesPlease
>  	LIBC_CONTAINS_LIBINTL = YesPlease
>  	HAVE_DEV_TTY = YesPlease
> +	NO_THREAD_SAFE_PREAD = YesPlease
>  endif
>  ifeq ($(uname_S),GNU/kFreeBSD)
>  	NO_STRLCPY = YesPlease
> 
> With this, I do have correctly working git clone.

OK, good.

You didn't mention which platform you are on; from the above Makefile
hunk, however, I can deduce that you are on *some* version of Linux.

Hmm, it doesn't seem too likely that your pread() is thread-unsafe
(possible, just unlikely), so it could be a more fundamental problem
with the threaded index-pack code (ie commit b8a2486f1 et. seq.).

However, as a first step could you try running the test program
(given below) on your system to determine if your pread() is thread-safe
or not. (gcc -I. -o test-pread test-pread.c; ./test-pread)

Also, what is the output of "uname -a".

ATB,
Ramsay Jones

-- >8 --
#include "git-compat-util.h"
#include "thread-utils.h"

#define DATA_FILE "junk.data"
#define MAX_DATA 256 * 1024
#define NUM_THREADS 3
#define TRIALS 500000

struct thread_data {
	pthread_t t;
	int fd;
	int cnt;
	int fails;
	unsigned long n;
};

static struct thread_data t[NUM_THREADS+1];

int create_data_file(void)
{
	int i, fd = open(DATA_FILE, O_CREAT | O_TRUNC | O_WRONLY, 0600);
	if (fd < 0)
		return -1;
	for (i = 0; i < MAX_DATA; i++)
		if (write(fd, &i, sizeof(int)) < 0) {
			close(fd);
			unlink(DATA_FILE);
			return -1;
		}
	close(fd);
	return 0;
}

void *read_thread(void *data)
{
	struct thread_data *d = (struct thread_data *)data;
	int i, j, rd;
	for (i = 0; i < TRIALS; i += MAX_DATA) {
		for (j = 0; j < MAX_DATA; j++) {
			ssize_t sz = read(d->fd, &rd, sizeof(int));
			if (sz < 0 || rd != j)
				d->fails++;
			d->cnt++;
		}
		lseek(d->fd, 0, SEEK_SET);
	}
	return NULL;
}

void *pread_thread(void *data)
{
	struct thread_data *d = (struct thread_data *)data;
	int i, j, rd;
	for (i = 0; i < TRIALS; i++) {
		ssize_t sz;
		d->n = d->n * 1103515245 + 12345;
		j = d->n % MAX_DATA;
		sz = pread(d->fd, &rd, sizeof(int), j * sizeof(int));
		if (sz < 0 || rd != j)
			d->fails++;
		d->cnt++;
	}
	return NULL;
}

int main(int argc, char *argv[])
{
	int fd, i;

	if (create_data_file() < 0) {
		printf("can't create data file\n");
		return 1;
	}

	if ((fd = open(DATA_FILE, O_RDONLY)) < 0) {
		printf("can't open data file\n");
		unlink(DATA_FILE);
		return 1;
	}

	for (i = 0; i < NUM_THREADS+1; i++) {
		int ret;

		t[i].fd = fd;
		t[i].cnt = 0;
		t[i].fails = 0;
		t[i].n = i * 16381;
		ret = pthread_create(&t[i].t, NULL,
				(i == 0) ? read_thread : pread_thread,
				&t[i]);
		if (ret) {
			printf("can't create thread %d (%s)\n", i, strerror(ret));
			unlink(DATA_FILE);
			return 1;
		}
	}

	for (i = 0; i < NUM_THREADS+1; i++)
		pthread_join(t[i].t, NULL);
	close(fd);

	for (i = 0; i < NUM_THREADS+1; i++)
		printf("%2d: trials %d, failed %d\n", i, t[i].cnt, t[i].fails);

	unlink(DATA_FILE);
	return 0;
}

Re: [regression] Newer gits cannot clone any remote repos

[Ramsay Jones]

Torsten Bögershausen wrote:
> On 13.11.12 19:55, Ramsay Jones wrote:
>> Douglas Mencken wrote:
>>> *Any* git clone fails with:
>>>
>>> fatal: premature end of pack file, 106 bytes missing
>>> fatal: index-pack failed
>>>
>>> At first, I tried 1.8.0, and it failed. Then I tried to build 1.7.10.5
>>> then, and it worked. Then I tried 1.7.12.2, but it fails the same way
>>> as 1.8.0.
>>> So I decided to git bisect.
>>>
>>> b8a2486f1524947f232f657e9f2ebf44e3e7a243 is the first bad commit
>>> ``index-pack: support multithreaded delta resolving''
>>
>> This looks like the same problem I had on cygwin, which lead to
>> commit c0f86547c ("index-pack: Disable threading on cygwin", 26-06-2012).
>>
>> I didn't notice which platform you are on, but maybe you also have a
>> thread-unsafe pread()? Could you try re-building git with the
>> NO_THREAD_SAFE_PREAD build variable set?
>>
>> HTH.
>>
>> ATB,
>> Ramsay Jones
> 
> This is interesting.
> I had the same problem on a PowerPC 
> (Old PowerBook G4 running Linux).
> 
> Using NO_THREAD_SAFE_PREAD helped, thanks for the hint.
> (After recompiling without NO_THREAD_SAFE_PREAD I could clone
> from this machine again, so the problem is not really reproducable)

Yes, the failures would be intermittent (and often not easily
reproducible). The threaded index-pack code did not fail for
me on cygwin at all during development, including tests, but failed
immediately I installed v1.7.11. On real repositories, it failed
intermittently. On some repos it always failed, on some it never
failed and on some others it would sometimes fail, sometimes not.

ATB,
Ramsay Jones

Re: [regression] Newer gits cannot clone any remote repos

[Douglas Mencken]

> The threaded index-pack code did not fail for
> me on cygwin at all during development, including tests, but failed
> immediately I installed v1.7.11. On real repositories, it failed
> intermittently. On some repos it always failed, on some it never
> failed and on some others it would sometimes fail, sometimes not.

Then why did you commit it? If it has so high random failure rate.