Meaning of "fatal: protocol error: bad line length character"?

Meaning of "fatal: protocol error: bad line length character"?

[Bill Lear]

[Many thanks to Junio for explaining in generous detail how to think
properly about branches in git's distributed model.]

We are making progress in our conversion to git, but one of our
developers, on pushing into our company repo, has encountered this
error:

% git push
updating 'refs/heads/master'
  from 6b421066e842203e383e1dc466c1cdef10de56b1
  to   2a8e554ae0c99d44988690c9fce693b3f5f128fa
Generating pack...
Done counting 61 objects.
Result has 32 objects.
Deltifying 32 objects.
 100% (32/32) done
Writing 32 objects.
 100% (32/32) done
Total 32, written 32 (delta 18), reused 0 (delta 0)
Unpacking 32 objects
fatal: protocol error: bad line length character

The notion of fatality led him to think, quite plausibly, that
something very bad had happened with his push.  However, we can find
no evidence that anything bad really did happen.

I did read through the code, and pkt-line.c seems to be the origin of
this, and it does seem to short-circuit the read of the packet if this
error is encountered.  Of course, I have no idea what this really
means...

Could someone please explain what this error means, and how to deal
with it?

Thank you.


Bill

Re: Meaning of "fatal: protocol error: bad line length character"?

[Junio C Hamano]

Bill Lear <rael@zopyra.com> writes:

> % git push
> updating 'refs/heads/master'
>   from 6b421066e842203e383e1dc466c1cdef10de56b1
>   to   2a8e554ae0c99d44988690c9fce693b3f5f128fa
> Generating pack...
> Done counting 61 objects.
> Result has 32 objects.
> Deltifying 32 objects.
>  100% (32/32) done
> Writing 32 objects.
>  100% (32/32) done
> Total 32, written 32 (delta 18), reused 0 (delta 0)
> Unpacking 32 objects
> fatal: protocol error: bad line length character
>
> The notion of fatality led him to think, quite plausibly, that
> something very bad had happened with his push.  However, we can find
> no evidence that anything bad really did happen.

I've seen this "bad line length character" mentioned in #git and
on this list but nobody seems to have hunted down what this is.

	http://www.gelato.unsw.edu.au/archives/git/0603/17807.html

is another (Google finds some others for the error message on
xcb list but that is about fetch-pack which is totally different
codepath).

Your report and the above one both mention there was no harm,
which is somewhat of consolation but it definitely is not a good
sign.

I've tried to reproduce it, suspecting it could be some
interaction with hook scripts output, without success.

I've committed the following to 'master' so that when it
reproduces we could have a slightly better clue on what we are
getting instead of what we are expecting.

diff --git a/pkt-line.c b/pkt-line.c
index b4cb7e2..369eec9 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -81,13 +81,13 @@ int packet_read_line(int fd, char *buffer, unsigned size)
 {
 	int n;
 	unsigned len;
-	char linelen[4];
+	unsigned char linelen[4];
 
-	safe_read(fd, linelen, 4);
+	safe_read(fd, (char *)linelen, 4);
 
 	len = 0;
 	for (n = 0; n < 4; n++) {
-		unsigned char c = linelen[n];
+		char c = linelen[n];
 		len <<= 4;
 		if (c >= '0' && c <= '9') {
 			len += c - '0';
@@ -101,7 +101,9 @@ int packet_read_line(int fd, char *buffer, unsigned size)
 			len += c - 'A' + 10;
 			continue;
 		}
-		die("protocol error: bad line length character");
+		die("protocol error: bad line length character: "
+		    "%02x %02x %02x %02x",
+		    linelen[0], linelen[1], linelen[2], linelen[3]);
 	}
 	if (!len)
 		return 0;

Re: Meaning of "fatal: protocol error: bad line length character"?

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> Bill Lear <rael@zopyra.com> writes:
>
>> % git push
>> updating 'refs/heads/master'
>>   from 6b421066e842203e383e1dc466c1cdef10de56b1
>>   to   2a8e554ae0c99d44988690c9fce693b3f5f128fa
>> Generating pack...
>> Done counting 61 objects.
>> Result has 32 objects.
>> Deltifying 32 objects.
>>  100% (32/32) done
>> Writing 32 objects.
>>  100% (32/32) done
>> Total 32, written 32 (delta 18), reused 0 (delta 0)
>> Unpacking 32 objects
>> fatal: protocol error: bad line length character

By the way, I can see from the pack-objects output above that
you seem to be using git before commit 67c08ce1 (Nov 29, 2006);
most likely you are using the official 1.4.4.4?

Is it possible for you to try git built from the tip of 'master'
to see if it reproduces?

Re: Meaning of "fatal: protocol error: bad line length character"?

[Bill Lear]

We are using git 1.4.4.1, the latest I thought available, not 1.4.4.4
(perhaps you mis-typed?).

In any case, I personally would love to try the latest build of git,
but I don't think I can convince the rest of the company to do so.

I just got an email from one of the developers.  He seems to think
this is alleviated by using bash instead of tcsh.  He said he is
debugging his environment to see if he can isolate the problem.  I
have asked for more details and if I can find out more, I will
certainly communicate it here.

If I were to get people to agree to try the latest git build, would I
be able to just let the developer install and use it on his machine,
or would I need to also update our company repo?  If the former, I
would have a much easier time convincing them.

Thanks.


Bill

On Saturday, January 20, 2007 at 11:33:38 (-0800) Junio C Hamano writes:
>Junio C Hamano <junkio@cox.net> writes:
>
>> Bill Lear <rael@zopyra.com> writes:
>>
>>> % git push
>>> updating 'refs/heads/master'
>>>   from 6b421066e842203e383e1dc466c1cdef10de56b1
>>>   to   2a8e554ae0c99d44988690c9fce693b3f5f128fa
>>> Generating pack...
>>> Done counting 61 objects.
>>> Result has 32 objects.
>>> Deltifying 32 objects.
>>>  100% (32/32) done
>>> Writing 32 objects.
>>>  100% (32/32) done
>>> Total 32, written 32 (delta 18), reused 0 (delta 0)
>>> Unpacking 32 objects
>>> fatal: protocol error: bad line length character
>
>By the way, I can see from the pack-objects output above that
>you seem to be using git before commit 67c08ce1 (Nov 29, 2006);
>most likely you are using the official 1.4.4.4?
>
>Is it possible for you to try git built from the tip of 'master'
>to see if it reproduces?

Re: Meaning of "fatal: protocol error: bad line length character"?

[Jakub Narebski]

Bill Lear wrote:

> We are using git 1.4.4.1, the latest I thought available, not 1.4.4.4
> (perhaps you mis-typed?).

The latest is git 1.4.4.4. Available for example from
http://kernel.org/pub/software/scm/git/
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: Meaning of "fatal: protocol error: bad line length character"?

[Junio C Hamano]

Bill Lear <rael@zopyra.com> writes:

> We are using git 1.4.4.1, the latest I thought available, not 1.4.4.4
> (perhaps you mis-typed?).
>
> In any case, I personally would love to try the latest build of git,
> but I don't think I can convince the rest of the company to do so.
>
> I just got an email from one of the developers.  He seems to think
> this is alleviated by using bash instead of tcsh.

Heh, you'll never know what you would get until you ask.  I
temporarily run chsh on myself to use tcsh as my login shell,
and sure enough I am getting the error.

fatal: protocol error: bad line length character: 75 70 64 61

It reads "u p d a"; most likely the command name we are running,
which is "update".  I think it is spitting out an error message
or something silly like that saying "update hook is not
executable".  Let me dig a bit further and report later.

This is with my previous patch; you do not have to update to
1.5.0-rc1 -- it would not fix anything in this area.

Re: Meaning of "fatal: protocol error: bad line length character"?

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> Bill Lear <rael@zopyra.com> writes:
>
>> We are using git 1.4.4.1, the latest I thought available, not 1.4.4.4
>> (perhaps you mis-typed?).
>>
>> In any case, I personally would love to try the latest build of git,
>> but I don't think I can convince the rest of the company to do so.
>>
>> I just got an email from one of the developers.  He seems to think
>> this is alleviated by using bash instead of tcsh.
>
> Heh, you'll never know what you would get until you ask.  I
> temporarily run chsh on myself to use tcsh as my login shell,
> and sure enough I am getting the error.
>
> fatal: protocol error: bad line length character: 75 70 64 61
>
> It reads "u p d a"; most likely the command name we are running,
> which is "update".  I think it is spitting out an error message
> or something silly like that saying "update hook is not
> executable".  Let me dig a bit further and report later.

That was output to stdout made from the update hook in my case.
I do not know your setup, but if you make sure your update hook
does not spit out anything to its stdout (diag can go to stderr),
you should be able to work it around.

Funny thing is, at least in recent enough git, I think we set up
redirection to force output from hook scripts to stderr, but I
do not remember when it happened.  Let's see...

	...goes and looks...

That was supposed to have been fixed with commit cd83c74c (Dec
30, 2006), but this indicates the fix is not working at all when
your login shell is tcsh.

Hmmmmmm.....

Re: Meaning of "fatal: protocol error: bad line length character"?

[Shawn O. Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Funny thing is, at least in recent enough git, I think we set up
> redirection to force output from hook scripts to stderr, but I
> do not remember when it happened.  Let's see...
> 
> 	...goes and looks...
> 
> That was supposed to have been fixed with commit cd83c74c (Dec
> 30, 2006), but this indicates the fix is not working at all when
> your login shell is tcsh.

Don't tell us tcsh is doing something ugly like opening the tty for
stdout/stderr instead of using the ones it inherited from its parent.
'cause that's just useless!

-- 
Shawn.

Re: Meaning of "fatal: protocol error: bad line length character"?

[Bill Lear]

On Saturday, January 20, 2007 at 12:42:18 (-0800) Junio C Hamano writes:
>Junio C Hamano <junkio@cox.net> writes:
>...
>That was output to stdout made from the update hook in my case.
>I do not know your setup, but if you make sure your update hook
>does not spit out anything to its stdout (diag can go to stderr),
>you should be able to work it around.
>
>Funny thing is, at least in recent enough git, I think we set up
>redirection to force output from hook scripts to stderr, but I
>do not remember when it happened.  Let's see...
>
>	...goes and looks...
>
>That was supposed to have been fixed with commit cd83c74c (Dec
>30, 2006), but this indicates the fix is not working at all when
>your login shell is tcsh. ...

One of our developers who uses bash is seeing this error in his pushes
from his private repo to his public repo, through the file system.

Here is one example he conveyed:

% git push
error: remote 'refs/heads/master' is not a strict subset of local ref 'refs/heads/master'. maybe you are not up-to-date and need to pull first?
error: remote 'refs/heads/origin' is not a strict subset of local ref 'refs/heads/origin'. maybe you are not up-to-date and need to pull first?
updating 'refs/heads/fig_mt_1'
  from 85f1ff556f95b0177e69f99d1196a2db26213812
  to   0a5a7ac9b81a90500b9c9c44a22793890513cd5a
Generating pack...
Done counting 16 objects.
Result has 10 objects.
Deltifying 10 objects.
 100% (10/10) done
Writing 10 objects.
 100% (10/10) done
Total 10, written 10 (delta 6), reused 0 (delta 0)
Unpacking 10 objects
 100% (10/10) done
fatal: protocol error: bad line length character
xiphi:~/devel/fig_mt_1/fusion | fig_mt_1 % refs/heads/fig_mt_1: 85f1ff556f95b01
77e69f99d1196a2db26213812 -> 0a5a7ac9b81a90500b9c9c44a22793890513cd5a

In this case, he forgot to limit the push to his fig_mt_1 branch, but
last night he did a similar push (git push origin fig_mt_1), and got
the same bad line length character error report.  His pushes seem to
be propagating, but he's a bit nervous nonetheless.

He has an active update hook in his public repo, to send notices to
his collaborators when his work is ready for them to pull, but we
looked through it and don't see any printing to stdout.


Bill

Re: Meaning of "fatal: protocol error: bad line length character"?

[Junio C Hamano]

Bill Lear <rael@zopyra.com> writes:

> He has an active update hook in his public repo, to send notices to
> his collaborators when his work is ready for them to pull, but we
> looked through it and don't see any printing to stdout.

Just a hunch.

Does the bash start-up sequence for the user emit something to
stdout?

The attached patch on the local side (i.e. the one that pushes)
dumps "garbage" you are getting on the line from the remote
side, to help diagnosing the problem.


diff --git a/pkt-line.c b/pkt-line.c
index b4cb7e2..9bb1c90 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -101,6 +101,16 @@ int packet_read_line(int fd, char *buffer, unsigned size)
 			len += c - 'A' + 10;
 			continue;
 		}
+		error("protocol error: bad line length character");
+		fprintf(stderr, "<<<<<\n%.*s", 4, linelen);
+		while (1) {
+			char buf[1024];
+			int cnt = xread(fd, buf, sizeof(buf));
+			if (cnt <= 0)
+				break;
+			fprintf(stderr, "%.*s", cnt, buf);
+		}
+		fprintf(stderr, "\n>>>>>\n");
 		die("protocol error: bad line length character");
 	}
 	if (!len)

Re: Meaning of "fatal: protocol error: bad line length character"?

[Bill Lear]

On Monday, January 22, 2007 at 11:45:29 (-0800) Junio C Hamano writes:
>Bill Lear <rael@zopyra.com> writes:
>
>> He has an active update hook in his public repo, to send notices to
>> his collaborators when his work is ready for them to pull, but we
>> looked through it and don't see any printing to stdout.
>
>Just a hunch.
>
>Does the bash start-up sequence for the user emit something to
>stdout?

Not that I can see.  I logged into his machine, su root, su - <his id>,
and nothing but a prompt came up.

>The attached patch on the local side (i.e. the one that pushes)
>dumps "garbage" you are getting on the line from the remote
>side, to help diagnosing the problem. ...

I have asked the developer to install this and hopefully we'll
get some feedback soon.

Thank you.


Bill

Re: Meaning of "fatal: protocol error: bad line length character"?

[Junio C Hamano]

"Shawn O. Pearce" <spearce@spearce.org> writes:

> Junio C Hamano <junkio@cox.net> wrote:
>> Funny thing is, at least in recent enough git, I think we set up
>> redirection to force output from hook scripts to stderr, but I
>> do not remember when it happened.  Let's see...
>> 
>> 	...goes and looks...
>> 
>> That was supposed to have been fixed with commit cd83c74c (Dec
>> 30, 2006), but this indicates the fix is not working at all when
>> your login shell is tcsh.
>
> Don't tell us tcsh is doing something ugly like opening the tty for
> stdout/stderr instead of using the ones it inherited from its parent.
> 'cause that's just useless!

It turns out that this was a stupidity on the part of the tester
(i.e. me).

I let the distro to install a version of git in /usr/bin/, and
my .cshrc is not set up to add /home/junio/git-active/bin (I
have $HOME/git-{master,next,pu,maint}/bin and git-active is a
symlink to git-master right now -- usually it points at
git-next), for non-interactive session like invoking
receive-pack.  But my bash startup does.  The breakage I was
seeing was _NOT_ because of differences between tcsh and bash,
but because the fix is fairly recent.

I forced the master version of receive-pack to be used on the
remote side using --receive-pack command line parameter, and
things seem to work fine with either tcsh or bash.

So, no, there might be valid reasons to hate tcsh, but this is
not one of them.

Re: Meaning of "fatal: protocol error: bad line length character"?

[Bill Lear]

On Monday, January 22, 2007 at 13:55:25 (-0600) Bill Lear writes:
>...
>>The attached patch on the local side (i.e. the one that pushes)
>>dumps "garbage" you are getting on the line from the remote
>>side, to help diagnosing the problem. ...
>
>I have asked the developer to install this and hopefully we'll
>get some feedback soon.

Looks like it is our developer's fault after all.  

% git push
[...]
Unpacking 7 objects
 100% (7/7) done
error: protocol error: bad line length character
<<<<<
Unknown command: "#alias"
refs/heads/tau_mt_1: 0a5a7ac9b81a90500b9c9c44a22793890513cd5a -> 3d0c74c15953ad5bc6a71eefa623933163c4f05f
[...]

The problem is that his mailer is putting out: 

    Unknown command: "#alias"

when it gets to the mail command at the bottom of the update hook.  So,
he's got a bad line in his .mailrc file.

I do think that some augmentation of the error output is in order,
either as previously posted, or something fancier.  It would be even
better if this sort of thing did not trip up git, though perhaps
I don't understand enough of how the update hook and git interact.


Bill

Re: Meaning of "fatal: protocol error: bad line length character"?

[Junio C Hamano]

Bill Lear <rael@zopyra.com> writes:

> Looks like it is our developer's fault after all.  
> ...
> I do think that some augmentation of the error output is in order,
> either as previously posted, or something fancier.  It would be even
> better if this sort of thing did not trip up git, though perhaps
> I don't understand enough of how the update hook and git interact.

The updated receive-pack (was fixed with cd83c74c on Dec 30th
2006, has been in 'master', is in 1.5.0-rc2, and will be in
1.5.0) is supposed to make this a non-issue.  Is it possible for
your developer to use it?  You can have a private install of
receive-pack somewhere in /home/me/bin and tell git-push to use
it with --exec=/home/me/bin/git-receive-pack.

With the fixed receive-pack, whatever the mailer says should not
break the protocol exchange by leaking into it; instead, it
should come back and shown to stderr, just like any output to
stderr from the hook script.

The test patch I sent you should probably not be used as-is in
the production, as it dumps literally everything to your
terminal without limit, but I do agree that it helped us
diagnose the problem in this case.  Maybe limiting it to show
the first few hundred bytes would be sufficient for debugging.

Re: Meaning of "fatal: protocol error: bad line length character"?

[Han-Wen Nienhuys]

Junio C Hamano escreveu:
> I've seen this "bad line length character" mentioned in #git and
> on this list but nobody seems to have hunted down what this is.
> 
> 	http://www.gelato.unsw.edu.au/archives/git/0603/17807.html
> 
> is another (Google finds some others for the error message on
> xcb list but that is about fetch-pack which is totally different
> codepath).

if you need another example where this is happening, the anongit
repo for lilypond seems to have gotten in this state as well,

[hanwen@haring tmp]$ git clone git://git.sv.gnu.org/lilypond.git
Initialized empty Git repository in /tmp/lilypond/.git/
fatal: protocol error: bad line length character
fetch-pack from 'git://git.sv.gnu.org/lilypond.git' failed.


pushing/pulling over ssh+git and http still work.
  

(git 1.5.0rc2, not sure what git.sv.gnu.org is running)

-- 
 Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen

Re: Meaning of "fatal: protocol error: bad line length character"?

[Junio C Hamano]

Han-Wen Nienhuys <hanwen@xs4all.nl> writes:

> if you need another example where this is happening, the anongit
> repo for lilypond seems to have gotten in this state as well,
>
> [hanwen@haring tmp]$ git clone git://git.sv.gnu.org/lilypond.git
> Initialized empty Git repository in /tmp/lilypond/.git/
> fatal: protocol error: bad line length character
> fetch-pack from 'git://git.sv.gnu.org/lilypond.git' failed.

With my debugging patch applied to peek-remote:

        $ git-peek-remote git://git.sv.gnu.org/lilypond.git
        error: protocol error: bad line length character
        <<<<<
        env: 
        >>>>>
        fatal: protocol error: bad line length character

So somebody says "env:" immediately followed by an EOF to the
stdout when upload-pack is invoked via git-daemon.

The first thing that happens in the fetch/clone/peek-remote
protocol is reading from the server side, so unfortunately there
is no way to debug any further without having an access to their
server.

Re: Meaning of "fatal: protocol error: bad line length character"?

[Han-Wen Nienhuys]

Sylvain, can you help out here? 

The lilypond repo got into a state where it will no longer
allow reading over the git:// protocol; for troubleshooting,
we need some info on what is happening on the GNU side of things.

Han-Wen

Junio C Hamano escreveu:
> Han-Wen Nienhuys <hanwen@xs4all.nl> writes:
> 
>> if you need another example where this is happening, the anongit
>> repo for lilypond seems to have gotten in this state as well,
>>
>> [hanwen@haring tmp]$ git clone git://git.sv.gnu.org/lilypond.git
>> Initialized empty Git repository in /tmp/lilypond/.git/
>> fatal: protocol error: bad line length character
>> fetch-pack from 'git://git.sv.gnu.org/lilypond.git' failed.
> 
> With my debugging patch applied to peek-remote:
> 
>         $ git-peek-remote git://git.sv.gnu.org/lilypond.git
>         error: protocol error: bad line length character
>         <<<<<
>         env: 
>         >>>>>
>         fatal: protocol error: bad line length character
> 
> So somebody says "env:" immediately followed by an EOF to the
> stdout when upload-pack is invoked via git-daemon.
> 
> The first thing that happens in the fetch/clone/peek-remote
> protocol is reading from the server side, so unfortunately there
> is no way to debug any further without having an access to their
> server.
> 


-- 
 Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen