Git's Perl scripts can fail if user is configured for perlbrew

Git's Perl scripts can fail if user is configured for perlbrew

[Randy J. Ray]

I use git on MacOS via homebrew (http://brew.sh/), and a custom Perl 
installation built and managed via perlbrew (http://perlbrew.pl/). At 
some point, commands like "git add -i" broke. I say "at some point", 
because I'm not a git power-user and I only just noticed it this week.

I am running Git 2.2.1 with a perlbrew'd Perl 5.20.1. When I would run 
"git add -i" (or "git add -p"), it would immediately die with a signal 
11. Some poking around showed that those git commands that are 
implemented as Perl scripts run under /usr/bin/perl, and also prefix 
some directories to the module search-path. The problem stems from the 
fact that, when you are using perlbrew, you also have the PERL5LIB 
environment variable set. The contents of it lay between the 
git-provided paths and the default contents of @INC. When the Git module 
is loaded, it (eventually) triggers a load of List::Util, whose C-level 
code fails to load because of a version mismatch; you got List::Util 
from the paths in PERL5LIB, but it doesn't match the version of perl 
from /usr/bin/perl.

After poking around and trying a few different things, I have found that 
using the following line in place of "#!/usr/bin/perl" solves this problem:

	#!/usr/bin/env perl

This can be done by defaulting PERL_PATH to "/usr/bin/env perl" in Makefile.

I don't know enough about the overall git ecosystem to know if this 
would have an adverse effect on anything else (in particular, Windows 
compatibility, but then Windows probably isn't having this issue in the 
first place).

I could just create and mail in the one-line patch for this, but I 
thought it might be better to open it up for some discussion first?

Randy
-- 
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Randy J. Ray      Sunnyvale, CA      http://www.dereferenced.com
rjray@blackperl.com
twitter.com/rjray
Silicon Valley Scale Modelers: http://www.svsm.org

Re: Git's Perl scripts can fail if user is configured for perlbrew

[Ævar Arnfjörð Bjarmason]

On Sun, Dec 28, 2014 at 11:36 PM, Randy J. Ray <rjray@blackperl.com> wrote:
> I use git on MacOS via homebrew (http://brew.sh/), and a custom Perl
> installation built and managed via perlbrew (http://perlbrew.pl/). At some
> point, commands like "git add -i" broke. I say "at some point", because I'm
> not a git power-user and I only just noticed it this week.
>
> I am running Git 2.2.1 with a perlbrew'd Perl 5.20.1. When I would run "git
> add -i" (or "git add -p"), it would immediately die with a signal 11. Some
> poking around showed that those git commands that are implemented as Perl
> scripts run under /usr/bin/perl, and also prefix some directories to the
> module search-path. The problem stems from the fact that, when you are using
> perlbrew, you also have the PERL5LIB environment variable set. The contents
> of it lay between the git-provided paths and the default contents of @INC.
> When the Git module is loaded, it (eventually) triggers a load of
> List::Util, whose C-level code fails to load because of a version mismatch;
> you got List::Util from the paths in PERL5LIB, but it doesn't match the
> version of perl from /usr/bin/perl.
>
> After poking around and trying a few different things, I have found that
> using the following line in place of "#!/usr/bin/perl" solves this problem:
>
>         #!/usr/bin/env perl
>
> This can be done by defaulting PERL_PATH to "/usr/bin/env perl" in Makefile.
>
> I don't know enough about the overall git ecosystem to know if this would
> have an adverse effect on anything else (in particular, Windows
> compatibility, but then Windows probably isn't having this issue in the
> first place).
>
> I could just create and mail in the one-line patch for this, but I thought
> it might be better to open it up for some discussion first?

[CC'd the perlbrew author]

This is a bit of a tricky issue.

Using whatever perl is defined in the environment is just as likely to
break, in general the build process tries to pick these assets at
compile-time. Imagine you're experimenting with some custom perl
version and now Git inexplicably breaks.

It's better if Git detects a working perl when you compile it and
sticks with that, which is why we use /usr/bin/perl by default.

When you're setting PERL5LIB you're indicating to whatever perl
interpreter you're going to run that that's where they it should pick
up its modules. IMO they way perlbrew does this is broken, instead of
setting PATH + PERL5LIB globally for your login shell it should set
the PATH, and then the "perl" in that path should be a pointer to some
small shellscript that sets PERL5LIB for *its* perl.

I don't know what the right tradeoff here is, but I think it would be
just as sensible to unset PERL5LIB in our own perl scripts + modules,
it would make live monkeypatching when you wanted to harder, but we
could always add a GITPERL5LIB or something...

Re: Git's Perl scripts can fail if user is configured for perlbrew

[Torsten Bögershausen]

On 2014-12-28 23.36, Randy J. Ray wrote:
> I use git on MacOS via homebrew (http://brew.sh/), and a custom Perl installation built and managed via perlbrew (http://perlbrew.pl/). At some point, commands like "git add -i" broke. I say "at some point", because I'm not a git power-user and I only just noticed it this week.
> 
> I am running Git 2.2.1 with a perlbrew'd Perl 5.20.1. When I would run "git add -i" (or "git add -p"), it would immediately die with a signal 11. Some poking around showed that those git commands that are implemented as Perl scripts run under /usr/bin/perl, and also prefix some directories to the module search-path. The problem stems from the fact that, when you are using perlbrew, you also have the PERL5LIB environment variable set. The contents of it lay between the git-provided paths and the default contents of @INC. When the Git module is loaded, it (eventually) triggers a load of List::Util, whose C-level code fails to load because of a version mismatch; you got List::Util from the paths in PERL5LIB, but it doesn't match the version of perl from /usr/bin/perl.
> 
> After poking around and trying a few different things, I have found that using the following line in place of "#!/usr/bin/perl" solves this problem:
> 
>     #!/usr/bin/env perl
> 
> This can be done by defaulting PERL_PATH to "/usr/bin/env perl" in Makefile.
> 
> I don't know enough about the overall git ecosystem to know if this would have an adverse effect on anything else (in particular, Windows compatibility, but then Windows probably isn't having this issue in the first place).
> 
> I could just create and mail in the one-line patch for this, but I thought it might be better to open it up for some discussion first?
> 
> Randy

Having problems with different perl installations is not an unknown problem
in Git, I would say.

And Git itself is prepared to handle this situation:

In Makefile I can read:
# Define PERL_PATH to the path of your Perl binary (usually /usr/bin/perl).

(What Git can not decide is which perl it should use, the one pointed out by $PATH or /usr/bin/perl.)

What does  
"type perl" say ?

And what happens when you build and install Git like this:
PERL_PATH=/XX/YY/perl make install

-----------
Are you thinking about changing
ifndef PERL_PATH
	PERL_PATH = /usr/bin/perl
endif
-- into --
ifndef PERL_PATH
	PERL_PATH = $(shell which perl)
endif
---

At first glance that could make sense, at least to me.

Re: Git's Perl scripts can fail if user is configured for perlbrew

[Randy J. Ray]

On 12/29/14, 7:21 AM, Ævar Arnfjörð Bjarmason wrote:
> [CC'd the perlbrew author]
>
> This is a bit of a tricky issue.
>
> Using whatever perl is defined in the environment is just as likely to
> break, in general the build process tries to pick these assets at
> compile-time. Imagine you're experimenting with some custom perl
> version and now Git inexplicably breaks.
>
> It's better if Git detects a working perl when you compile it and
> sticks with that, which is why we use /usr/bin/perl by default.

These are good points. I'm not sure when this stopped working for me... 
I don't use the -i or -p options to "git add" very often. So I can't say 
at what point it stopped working with the current configuration, only 
that it "used to work".

> When you're setting PERL5LIB you're indicating to whatever perl
> interpreter you're going to run that that's where they it should pick
> up its modules. IMO they way perlbrew does this is broken, instead of
> setting PATH + PERL5LIB globally for your login shell it should set
> the PATH, and then the "perl" in that path should be a pointer to some
> small shellscript that sets PERL5LIB for *its* perl.

That would be for the perlbrew author to consider, of course.

> I don't know what the right tradeoff here is, but I think it would be
> just as sensible to unset PERL5LIB in our own perl scripts + modules,
> it would make live monkeypatching when you wanted to harder, but we
> could always add a GITPERL5LIB or something...

You would have to have a shell script that un-set PERL5LIB and then 
invoked the given git script, because by the time script execution has 
begun, the contents of PERL5LIB have already been added to the head of 
the list of search paths. One approach I tried was to set the 
environment variable GITPERLLIB (which you already use and recognize, so 
there is no need for GITPERL5LIB), but that did not help. The base 
problem still remained: The content of PERL5LIB (which pointed to 
5.20.1-compiled extensions) took priority over the default @INC contents 
(which were for a 5.16.2 perl).

I don't know the right trade-off, either. I started out reporting this 
as an issue against the homebrew project's recipe for git, because they 
actually add more explicit library paths to @INC than a vanilla 
build/install of git does. But the problem is really in the interaction 
between /usr/bin/perl and a PERL5LIB set for an alternate perl. So the 
solution, if there is one, will lay here in git somewhere...

Randy
-- 
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Randy J. Ray      Sunnyvale, CA      http://www.dereferenced.com
rjray@blackperl.com
twitter.com/rjray
Silicon Valley Scale Modelers: http://www.svsm.org

Re: Git's Perl scripts can fail if user is configured for perlbrew

[Randy J. Ray]

On 12/29/14, 7:40 AM, Torsten Bögershausen wrote:
> Having problems with different perl installations is not an unknown problem
> in Git, I would say.
>
> And Git itself is prepared to handle this situation:
>
> In Makefile I can read:
> # Define PERL_PATH to the path of your Perl binary (usually /usr/bin/perl).
>
> (What Git can not decide is which perl it should use, the one pointed out by $PATH or /usr/bin/perl.)
>
> What does
> "type perl" say ?
>
> And what happens when you build and install Git like this:
> PERL_PATH=/XX/YY/perl make install
>
> -----------
> Are you thinking about changing
> ifndef PERL_PATH
> 	PERL_PATH = /usr/bin/perl
> endif
> -- into --
> ifndef PERL_PATH
> 	PERL_PATH = $(shell which perl)
> endif
> ---
>
> At first glance that could make sense, at least to me.

The problem in this case is the Perl being used at run-time, not 
build-time. The building of git is done by the homebrew project in this 
case, so I don't have direct control over it.

Randy
-- 
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Randy J. Ray      Sunnyvale, CA      http://www.dereferenced.com
rjray@blackperl.com
twitter.com/rjray
Silicon Valley Scale Modelers: http://www.svsm.org

Re: Git's Perl scripts can fail if user is configured for perlbrew

[Kang-min Liu]

> On 12/29/14, 7:21 AM, Ævar Arnfjörð Bjarmason wrote:
>> [CC'd the perlbrew author]
>>
>> This is a bit of a tricky issue.
>>
>> Using whatever perl is defined in the environment is just as likely to
>> break, in general the build process tries to pick these assets at
>> compile-time. Imagine you're experimenting with some custom perl
>> version and now Git inexplicably breaks.
>>
>> It's better if Git detects a working perl when you compile it and
>> sticks with that, which is why we use /usr/bin/perl by default.

With "perl" being an external dependency, sticking with whatever at the
compile time basically means these should stick:

- `which perl`, and the same $Config options
- Every module contained in PERL5LIB
- Other external executable dependencies of some .pm files that lives
  somewhere in PATH

We could of course build an app bundle dir like a lightweight container.
to mitigate this... but that would'nt be usefull without tweaking the
shebang line of the scripts -- to point to the perl script (shim or
real) that should be compatible with the newly built git.

I'd argue that trying to compile git (or other stuff in general) against
a perlbrew-managed perl is something that "perlbrew" executable cannot
manage. Because it is both valid that the user is doing this
intentionally and want the outcome, or the user is doing this
unintentionally. And even if we have a shim "perl" script, it would'nt
help as long as it is the "whatever perl" in PATH -- which might just be
incompitble.

I wonder if disabling perlbrew per-building can deal with this:

    'perlbrew off'

If this end up changing the shebang line then maybe, if not then it'll
still intefere after perlbrew is re-activated.

The other option to minimize perlbrew interferince is of course to
completely deprecate the global env var approach and make them project
local. But that's another story.

-- 
Cheers,
Kang-min Liu

Re: Git's Perl scripts can fail if user is configured for perlbrew

[Ævar Arnfjörð Bjarmason]

On Mon, Dec 29, 2014 at 10:57 PM, Randy J. Ray <rjray@blackperl.com> wrote:
> On 12/29/14, 7:40 AM, Torsten Bögershausen wrote:
>>
>> Having problems with different perl installations is not an unknown
>> problem
>> in Git, I would say.
>>
>> And Git itself is prepared to handle this situation:
>>
>> In Makefile I can read:
>> # Define PERL_PATH to the path of your Perl binary (usually
>> /usr/bin/perl).
>>
>> (What Git can not decide is which perl it should use, the one pointed out
>> by $PATH or /usr/bin/perl.)
>>
>> What does
>> "type perl" say ?
>>
>> And what happens when you build and install Git like this:
>> PERL_PATH=/XX/YY/perl make install
>>
>> -----------
>> Are you thinking about changing
>> ifndef PERL_PATH
>>         PERL_PATH = /usr/bin/perl
>> endif
>> -- into --
>> ifndef PERL_PATH
>>         PERL_PATH = $(shell which perl)
>> endif
>> ---
>>
>> At first glance that could make sense, at least to me.
>
>
> The problem in this case is the Perl being used at run-time, not build-time.
> The building of git is done by the homebrew project in this case, so I don't
> have direct control over it.

Correct, but we don't change /usr/bin/perl at runtime, we hardcode
that at compile-time.

Similarly we could hardcode PERL5LIB at compile-time, but we don't, if
we did you wouldn't have this problem.

I.e. the problem is that we're using the system-provided perl with a
custom PERL5LIB set for the benefit of a non-system provided perl
installed after you built Git (or built in a different environment...)

Re: Git's Perl scripts can fail if user is configured for perlbrew

[Ben Aveling]

On 29/12/14 09:36, Randy J. Ray wrote:
> I don't know enough about the overall git ecosystem to know if this 
> would have an adverse effect on anything else (in particular, Windows 
> compatibility, but then Windows probably isn't having this issue in 
> the first place).

Perlbrew doesn't support Windows, so no, not an issue for Windows.

Regards, Ben