The imporantance of including http credential caching in 1.7.7

The imporantance of including http credential caching in 1.7.7

[Kyle Neath]

Earlier today, Scott Chacon alerted me to this email thread:
http://www.spinics.net/lists/git/msg164192.html (many apologies to the list, I
am not sure how to properly reference this email or reply to it since I have
not been subscribed until today).

> * jk/http-auth-keyring (2011-08-03) 13 commits
>   (merged to 'next' on 2011-08-03 at b06e80e)
>  + credentials: add "getpass" helper
>  + credentials: add "store" helper
>  + credentials: add "cache" helper
>  + docs: end-user documentation for the credential subsystem
>  + http: use hostname in credential description
>  + allow the user to configure credential helpers
>  + look for credentials in config before prompting
>  + http: use credential API to get passwords
>  + introduce credentials API
>  + http: retry authentication failures for all http requests
>  + remote-curl: don't retry auth failures with dumb protocol
>  + improve httpd auth tests
>  + url: decode buffers that are not NUL-terminated
>
> Looked mostly reasonable except for the limitation that it is not clear
> how to deal with a site at which a user needs to use different passwords
> for different repositories. Will keep it in "next" at least for one cycle,
> until we start hearing real-world success reports on the list.
>
> Not urgent; will not be in 1.7.7.

This is very disheartening to hear. Of all the minor changes, bugs, and
potential features, I believe that http credential caching is the absolute
most important thing that git core can do to promote adoption. I've believed
this for more than a year now and I'm incredibly disappointed that it's being
shelved for yet another release.

Over the past two years as a designer for GitHub, I've answered ~thousands of
support requests and talked face to face with ~thousands of developers of
varying git skill levels. Once a month our company does online git training
for newbies - https://github.com/training/online and I've had many discussions
about newcomer's struggles with Matthew McCullough and Scott Chacon.
Previously, I worked at ENTP where I helped polish the very popular "Git for
Designers" guide http://hoth.entp.com/output/git_for_designers.html based on
feedback. I was also lead designer for GitHub for Mac - an OSX GUI aimed at
people less familiar with the command line.

I bring all of this up because I'd like to think I have a good handle on
common problems that git newcomers or people resisting git adoption run into.
I've been deeply involved in spreading git adoption full time for nearly 3
years - it's something that's incredibly important to me professionally and
personally. And the number one problem that *always* comes up is SSH key
complexity.

A lot of these problems surface themselves in people saying it's hard to setup
git on Windows. When I push these people to explain further, it turns out that
the problem is always setting up SSH key authentication on Windows, not
necessarily git.

It's incredibly frustrating that git's biggest roadblock has nothing to do
with version control at all, but rather network authentication. Just as I
believe only *you* can own your availability, I believe git should do it's
best to own authentication.

HTTP credential caching combined with Smart HTTP make git one hell of an
amazing tool for newcomers and experts alike. I like to envision a world in
which git with both these features shipped with the latest OSX install.
Developers, designers, or anyone with an inkling for code would have exactly
two steps to get started with any given git host:

  1. Set your git config email / username
  2. git clone https://example.com/repository

Contrast this to our current help page for OSX:
http://help.github.com/mac-set-up-git/ or worse yet, our Windows setup page
with all of it's yelling about what kind of SSH keys to setup:
http://help.github.com/win-set-up-git/

Please note that I am not arguing against the merits of SSH keys - for those
developers who understand them, they're fantastic. But the reality is the
great majority of people who interact with version control do not understand
them at all. This results in passwordless SSH keys, confusion, and
frustration.

If another git version comes and goes without http credential caching, I fear
we as a community have purposefully ignored the largest potential for git
adoption currently available. This is important enough for me that I believe
it would be in git's best interest to delay the release of 1.7.7 until this
feature has been patched to the core team's standards.

I'll summarize with a graph of my opinion of where git's potential for
adoption lies:

------------------------------------------------------------------------------
            OPPORTUNITY FOR GIT ADOPTION ACCORDING TO KYLE NEATH

                      Caching of http credentials
                                  |
[=====================================================================||=====]
                                                                          |
                                               Everything else in the universe

------------------------------------------------------------------------------

With hopes and butterflies,

Kyle Neath
Director of Design at GitHub

Re: The imporantance of including http credential caching in 1.7.7

[Sverre Rabbelier]

Heya,

On Wed, Sep 7, 2011 at 07:33, Kyle Neath <kneath@gmail.com> wrote:
> I'll summarize with a graph of my opinion of where git's potential for
> adoption lies:
>
> ------------------------------------------------------------------------------
>            OPPORTUNITY FOR GIT ADOPTION ACCORDING TO KYLE NEATH
>
>                      Caching of http credentials
>                                  |
> [=====================================================================||=====]
>                                                                          |
>                                               Everything else in the universe
>
> ------------------------------------------------------------------------------

I, personally, find the graph very convincing :).

Junio mentioned in WC that he wants to see some feedback on it's
usage, perhaps that you can help provide this by providing a git
patched with this functionality to some of your users and see how they
respond?

-- 
Cheers,

Sverre Rabbelier

Re: The imporantance of including http credential caching in 1.7.7

[Kyle Neath]

On Wed, Sep 7, 2011 at 12:46 AM, Sverre Rabbelier <srabbelier@gmail.com> wrote:
> Junio mentioned in WC that he wants to see some feedback on it's
> usage, perhaps that you can help provide this by providing a git
> patched with this functionality to some of your users and see how they
> respond?

Perhaps this is where things get complicated. Due to the nature of the people
I'm discussing (people afraid of SSH keys), getting them to run a custom
version of git is pretty far out of the question. Peff alluded to this in his
reply.

I think the best evidence I can provide is in context of GitHub for Mac.
GitHub for Mac defaults to Smart HTTP, with the credentials cached in OSX's
Keychain. Effectively, it functions as a patched git with credential caching.

Two months after shipping, we have around 20,000 people that use the
application regularly. In that time, we've gotten a ton of positive feedback.
So there is a great number of people interested in a git client that uses
Smart HTTP. One of the bigger complaints we get is that people constantly have
to enter their username & password if they choose to drop to the command line.

Of course, this is all fuzzy data since it's not really comparable with git
core. But it's the only data I have.

Then again, perhaps the fact that we spent development time hacking around
git's limitation is a data point in itself. Git GUI developers are spending
development time to fill a hole in git core.

Kyle

Re: The imporantance of including http credential caching in 1.7.7

[Junio C Hamano]

Kyle Neath <kneath@gmail.com> writes:

>> * jk/http-auth-keyring (2011-08-03) 13 commits
>> ...
>>
>> Looked mostly reasonable except for the limitation that it is not clear
>> how to deal with a site at which a user needs to use different passwords
>> for different repositories. Will keep it in "next" at least for one cycle,
>> until we start hearing real-world success reports on the list.
>>
>> Not urgent; will not be in 1.7.7.
>
> This is very disheartening to hear. Of all the minor changes, bugs, and
> potential features, I believe that http credential caching is the absolute
> most important thing that git core can do to promote adoption. I've believed
> this for more than a year now and I'm incredibly disappointed that it's being
> shelved for yet another release.

You are misreading the message utterly wrong that it is not even funny.

If this were a new, insignificant, and obscure feature in a piece of
software with mere 20k users, it may be OK to release a new version with
the feature in an uncooked shape. The feature may turn out to work only in
a very limited scope but not general enough to accomodate other needs you
did not anticipate, and your next version may have to fix it in a backward
incompatible way, forcing existing users to unlearn what the initial
uncooked design and implementation gave them, but by definition such an
insignificant and obscure new feature in a system with a small userbase
wouldn't have been used by too many people by the time you have to fix it,
and the extent of the damage is limited.

Git is not that piece of software with only 20k users, and a lot more
importantly, credential caching API is not a feature in an insignificant
and obscure corner of the user experience. We have to have at least a warm
fuzzy feeling that the API is right from the start so the plug-ins people
write for our initial version and the data its users will accumulate with
it will not go to waste by hastily releasing a version with an uncooked
design and implementation.

"Not urgent" is not "Not important". Quite the contrary, it is important
enough that we cannot afford to be hasty.

See http://thread.gmane.org/gmane.comp.version-control.git/180245 for an
example of how a discussion can be done to help us get closer to the warm
fuzzy feeling of getting the API right in a more mature manner. Don't feel
too bad if you become ashamed of the rant you wrote contrasting with that
exchange. It is a sign that you are learning.

> Then again, perhaps the fact that we spent development time hacking around
> git's limitation is a data point in itself. Git GUI developers are spending
> development time to fill a hole in git core.

Welcome to open source. Do not expect others to scratch every itch of your
niche. Think of it as an opportunity granted to you to make a difference,
not as a roadblock somebody else threw at you because they hate you.

Be thankful, not whiny.

Be good.

Have fun.

Re: The imporantance of including http credential caching in 1.7.7

[Michael J Gruber]

Kyle Neath venit, vidit, dixit 07.09.2011 07:33:
> Earlier today, Scott Chacon alerted me to this email thread:
> http://www.spinics.net/lists/git/msg164192.html (many apologies to the list, I
> am not sure how to properly reference this email or reply to it since I have
> not been subscribed until today).
> 
>> * jk/http-auth-keyring (2011-08-03) 13 commits
>>   (merged to 'next' on 2011-08-03 at b06e80e)
>>  + credentials: add "getpass" helper
>>  + credentials: add "store" helper
>>  + credentials: add "cache" helper
>>  + docs: end-user documentation for the credential subsystem
>>  + http: use hostname in credential description
>>  + allow the user to configure credential helpers
>>  + look for credentials in config before prompting
>>  + http: use credential API to get passwords
>>  + introduce credentials API
>>  + http: retry authentication failures for all http requests
>>  + remote-curl: don't retry auth failures with dumb protocol
>>  + improve httpd auth tests
>>  + url: decode buffers that are not NUL-terminated
>>
>> Looked mostly reasonable except for the limitation that it is not clear
>> how to deal with a site at which a user needs to use different passwords
>> for different repositories. Will keep it in "next" at least for one cycle,
>> until we start hearing real-world success reports on the list.
>>
>> Not urgent; will not be in 1.7.7.
> 
> This is very disheartening to hear. Of all the minor changes, bugs, and
> potential features, I believe that http credential caching is the absolute
> most important thing that git core can do to promote adoption. I've believed
> this for more than a year now and I'm incredibly disappointed that it's being
> shelved for yet another release.
> 
> Over the past two years as a designer for GitHub, I've answered ~thousands of
> support requests and talked face to face with ~thousands of developers of
> varying git skill levels. Once a month our company does online git training
> for newbies - https://github.com/training/online and I've had many discussions
> about newcomer's struggles with Matthew McCullough and Scott Chacon.
> Previously, I worked at ENTP where I helped polish the very popular "Git for
> Designers" guide http://hoth.entp.com/output/git_for_designers.html based on
> feedback. I was also lead designer for GitHub for Mac - an OSX GUI aimed at
> people less familiar with the command line.

So, it's been a year or more that you've been aware of the importance of
this issue (from your/github's perspective), and we hear about it now,
at the end of the rc phase. I don't know whether jk/http-auth-keyring
has been done on github payroll or during spare time. But as you can
see, all that is missing is real-world success reports, along with the
single-user-multiple-passwords issue which is probably answered best
from the real-world perspective (how common is it, do we even need to
address it now). So, given the importance this has for you and the
company, you sure can help out with what is still left to do, can you?

But note that the credential api is only as good a solution as the
helpers. Do we really want all users to employ credential-store because
it is "much simpler than ssh"? Deploying what we have now and advocating
it as a solution for the ssh-challenged (which will happen whether we,
you or someone else advocates it) could easily mean telling people to
store their credentials unencrypted. The slash-dot story will be "git
security hole", "git stores passwords unencrypted" and so on. And I
don't care as much about the PR as about the users whom we'd be serving
badly.

So, for the ssh-challenged, we really should make sure that secure
helpers are in their hand when the credential api hits a released
version (or revert the insecure store helper). There's a KDE attempt on
the list. Gnome, Win, Mac helpers anyone? Win version of the cache helper?

Michael

Re: The imporantance of including http credential caching in 1.7.7

[Kyle Neath]

Junio C Hamano <gitster@pobox.com> wrote:
> If this were a new, insignificant, and obscure feature in a piece of
> software with mere 20k users, it may be OK to release a new version with
> the feature in an uncooked shape.

For the sake of my paycheck, I should certainly hope not! I'm not at all
suggesting we merge what we have in. However, I do think this feature is
important enough to delay the release. I trust in the judgement of the core
members to know when something is ready for inclusion in master.

Michael J Gruber <git@drmicha.warpmail.net> wrote:
> So, it's been a year or more that you've been aware of the importance of
> this issue (from your/github's perspective), and we hear about it now,
> at the end of the rc phase.

I apologize if it sounds like that. I've been discussing this situation with
many people (including Jeff King) for a very long time now, and it was my
understanding that the credential caching was done and simply waiting for a
new release. This is the first I've heard that it will not be included in
1.7.7, so I'm voicing my opinion now. Admittedly, late in the game - and I
apologize for that.

I'd be happy to help in any capacity I can. Unfortunately I'm no C hacker, and
I've accepted that as a character flaw (it's something I'm working on). I'm
afraid I can't be of much help with the actual code. What I can provide is an
alternate viewpoint to the core team. A viewpoint of someone who's spent 3
years trying to make git easier for newcomers.

I urge the core team to think about what kind of opportunity we have here.
Credential caching isn't some minor feature. It's the last piece of the puzzle
that promotes Smart HTTP to a viable alternative over the git & ssh protocols.
Smart HTTP solves an huge collection of problems people have with using git
day to day. One URL to push privately and pull anonymously. No firewall
restrictions at universities or workplaces. Username & password
authentication. I get kind of turned on just thinking about it.

Kyle

Re: The imporantance of including http credential caching in 1.7.7

[Junio C Hamano]

Kyle Neath <kneath@gmail.com> writes:

> ... However, I do think this feature is
> important enough to delay the release. I trust in the judgement of the core
> members to know when something is ready for inclusion in master.

We do not delay a release for any new feature, as we shoot for 8 to 10
week release cycle these days. For an important feature that we must avoid
painting ourselves into a corner that we cannot get out of if we screw up
the initial design due to possible backward compatibility issues, it is
even more true than other minor features.

This cycle is being delayed longer than anybody wishes for an external
reason, and unfortunately it does not mean we have longer period to
eyeball the topics in cooking due to that same external reason.

Re: The imporantance of including http credential caching in 1.7.7

[Philip Oakley]

From: "Kyle Neath" <kneath@gmail.com>
> Junio C Hamano <gitster@pobox.com> wrote:
>> If this were a new, insignificant, and obscure feature in a piece of
>> software with mere 20k users, it may be OK to release a new version with
>> the feature in an uncooked shape.
>
> Michael J Gruber <git@drmicha.warpmail.net> wrote:
>> So, it's been a year or more that you've been aware of the importance of
>> this issue (from your/github's perspective), and we hear about it now,
>> at the end of the rc phase.
>
> I apologize if it sounds like that. I've been discussing this situation 
> with
> many people (including Jeff King) for a very long time now, and it was my
> understanding that the credential caching was done and simply waiting for 
> a
> new release. This is the first I've heard that it will not be included in
> 1.7.7, so I'm voicing my opinion now. Admittedly, late in the game - and I
> apologize for that.
>
> I'd be happy to help in any capacity I can. Unfortunately I'm no C hacker, 
> and
> I've accepted that as a character flaw (it's something I'm working on). 
> I'm
> afraid I can't be of much help with the actual code. What I can provide is 
> an
> alternate viewpoint to the core team. A viewpoint of someone who's spent 3
> years trying to make git easier for newcomers.

Help in any capacity :
Would it not be possible for GitHub to provide for those key users such a 
trial version
that includes the patches identified to obtain the "real-world success 
reports" that
are needed, as mentioned in the "Re: What's cooking in git.git (Aug 2011, 
#07; Wed, 24)"

This should help satisfy the needs from both sides, even if you can only 
push it to a few clients.

>>What's cooking in git.git (Aug 2011, #07; Wed, 24)
>> Looked mostly reasonable except for the limitation that it is not clear
>> how to deal with a site at which a user needs to use different passwords
>> for different repositories. Will keep it in "next" at least for one 
>> cycle,
>> until we start hearing real-world success reports on the list.

Re: The imporantance of including http credential caching in 1.7.7

[Junio C Hamano]

"Philip Oakley" <philipoakley@iee.org> writes:

> Would it not be possible for GitHub to provide for those key users such
> a trial version that includes the patches identified to obtain the
> "real-world success reports" that are needed, as mentioned in the "Re:
> What's cooking in git.git (Aug 2011, #07; Wed, 24)"
>
> This should help satisfy the needs from both sides, even if you can
> only push it to a few clients.

That would not help very much, as (1) we know what Jeff included as sample
keystores are more or less cooked and good, but (2) nobody in your style
of trial will come up with different keystore to exercise the API to make
sure that will not paint us into a corner we cannot upgrade without major
pain in the backward compatibility area. The "real-world success reports"
you can generate would only for (1) but at this point we are not worried
about that. We are more (much more) worried about (2).

Re: The imporantance of including http credential caching in 1.7.7

[Michael J Gruber]

Kyle Neath venit, vidit, dixit 07.09.2011 22:14:
> Junio C Hamano <gitster@pobox.com> wrote:
>> If this were a new, insignificant, and obscure feature in a piece of
>> software with mere 20k users, it may be OK to release a new version with
>> the feature in an uncooked shape.
> 
> For the sake of my paycheck, I should certainly hope not! I'm not at all
> suggesting we merge what we have in. However, I do think this feature is
> important enough to delay the release. I trust in the judgement of the core
> members to know when something is ready for inclusion in master.
> 
> Michael J Gruber <git@drmicha.warpmail.net> wrote:
>> So, it's been a year or more that you've been aware of the importance of
>> this issue (from your/github's perspective), and we hear about it now,
>> at the end of the rc phase.
> 
> I apologize if it sounds like that. I've been discussing this situation with
> many people (including Jeff King) for a very long time now, and it was my
> understanding that the credential caching was done and simply waiting for a
> new release. This is the first I've heard that it will not be included in
> 1.7.7, so I'm voicing my opinion now. Admittedly, late in the game - and I
> apologize for that.

OK, I've calmed down :)

> I'd be happy to help in any capacity I can. Unfortunately I'm no C hacker, and
> I've accepted that as a character flaw (it's something I'm working on). I'm
> afraid I can't be of much help with the actual code. What I can provide is an
> alternate viewpoint to the core team. A viewpoint of someone who's spent 3
> years trying to make git easier for newcomers.

It would be interesting to know what we can rely on in the user group
you're thinking about (which I called ssh-challenged). Setting up ssh
keys is too complicated. Can we require a working gpg setup? They do
want to check sigs, don't they?

What I have in mind is a very simple, but secure version of Jeff's
credential-store, respectively his example, somewhat like:

---%<---
STORAGE=$HOME/.credentials

for i in "$@"; do
	case "$i" in
	--unique=*)
		unique=${i#--unique=} ;;
	esac
done

key=$(git config get credential.gpgkey) # or error out

if ! test -e "$STORAGE/$unique"; then
	mkdir -m 0700 "$STORAGE"
	git credential-getpass "$@" | gpg -ear $key >"$STORAGE/$unique"
fi

gpg <"$STORAGE/$unique"
---%<---

Or that in C, probably using Junio's gpg-lib. That would be secure and
useful *if* we can rely on people having a convenient gpg setup
(gpg-agent or such).

So: What credential store/password wallet/etc. can we rely on for this
group? Is gpg fair game?

Michael

Re: The imporantance of including http credential caching in 1.7.7

[John Szakmeister]

On Thu, Sep 8, 2011 at 9:17 AM, Michael J Gruber
<git@drmicha.warpmail.net> wrote:
[snip]
> It would be interesting to know what we can rely on in the user group
> you're thinking about (which I called ssh-challenged). Setting up ssh
> keys is too complicated. Can we require a working gpg setup? They do
> want to check sigs, don't they?

I don't think you can require a working gpg setup (at least for not
addressing the ssh-challenged group).

[snip]
> Or that in C, probably using Junio's gpg-lib. That would be secure and
> useful *if* we can rely on people having a convenient gpg setup
> (gpg-agent or such).
>
> So: What credential store/password wallet/etc. can we rely on for this
> group? Is gpg fair game?

I think there probably need to be providers for using Keychain under
the Mac, gnome-keyring and kwallet under Linux, and probably something
using the wincrypt API under Windows.  I don't think there's a
one-store-fits-all solution here, unfortunately. :-(

I'm actually tempted try and work on a couple of those myself.

-John

Re: The imporantance of including http credential caching in 1.7.7

[Jeff King]

On Wed, Sep 07, 2011 at 02:56:03PM +0200, Michael J Gruber wrote:

> So, it's been a year or more that you've been aware of the importance
> of this issue (from your/github's perspective), and we hear about it
> now, at the end of the rc phase. I don't know whether
> jk/http-auth-keyring has been done on github payroll or during spare
> time.

To be absolutely clear here, this feature was 100% paid for by GitHub
(which isn't to say that I don't think it's a good idea. On the
contrary, I think it's awesome; but GitHub money is what provided the
time for me to work on it).

When I started at GitHub in January, I was given a giant list of things
that GitHub felt would make core git better, but that they hadn't the
personnel to improve. And I was told to use my own judgement in adding
or removing items from the list based on what I thought git needed, and
to prioritize as I saw fit. The fact that it took six months for me to
come up with credential patches is because that's how long it took me to
figure out what I wanted to write, and to clear my backlog of other git
tasks.

So I think the wheels have been turning on this for quite a while from
GitHub's perspective.

At the same time, I agree very much with Junio; releasing something with
a bad API and then having to fix it later is much worse than delaying
the release of a feature by a little bit. And we have very little data
on whether the API is "right" at this point. Initially I was concerned
that there wasn't going to be enough interest while the patches were in
'next', and that we would have to make a release in order to get people
interested enough in writing helpers. But right after I said that, Lukas
Sandström showed up with a kdewallet helper. And Ted Zlatanov is working
on something for the freedesktop secrets API.

And already there's been some discussion that perhaps the current
interface isn't quite what we want and is going to need tweaking.
So we are moving forward, and I still hope that we can target the next
release of "master" in 8-10 weeks. But this time with more confidence
that what's being released is actually right.

In the meantime, the best thing we can do to push it forward is to write
helpers. I implemented some basic ones that should work anywhere, but
aren't as nice as integration with existing keychains. Some people are
working on Linux ones. The single best thing GitHub can do to push this
forward right now is to provide a well-written OS X Keychain helper, and
to provide feedback on whether git's end of the API is good enough.

-Peff

Re: The imporantance of including http credential caching in 1.7.7

[Jeff King]

On Thu, Sep 08, 2011 at 11:02:11AM -0400, John Szakmeister wrote:

> On Thu, Sep 8, 2011 at 9:17 AM, Michael J Gruber
> <git@drmicha.warpmail.net> wrote:
> [snip]
> > It would be interesting to know what we can rely on in the user group
> > you're thinking about (which I called ssh-challenged). Setting up ssh
> > keys is too complicated. Can we require a working gpg setup? They do
> > want to check sigs, don't they?
> 
> I don't think you can require a working gpg setup (at least for not
> addressing the ssh-challenged group).

Agreed. Anything harder than ssh keys is right out the window, because
they're always the alternative these people could be using (but can't or
don't want to).

We could make our own gpg-based password wallet system, but I think it's
a really bad idea, for two reasons:

  1. It's reinventing the wheel. Which is bad enough as it is, but is
     doubly bad with security-related code, because it's very easy to
     screw something up when you're writing a lot of new code.

  2. It's inconvenient for users. Nobody wants a separate wallet system
     with its own master password. They want to integrate with the
     wallet system they're already using. Which is generally going to be
     way nicer _anyway_, because it's going to be part of the OS and do
     helpful things like unlock the secret store using their login
     credentials.

> > So: What credential store/password wallet/etc. can we rely on for this
> > group? Is gpg fair game?
> 
> I think there probably need to be providers for using Keychain under
> the Mac, gnome-keyring and kwallet under Linux, and probably something
> using the wincrypt API under Windows.  I don't think there's a
> one-store-fits-all solution here, unfortunately. :-(

Exactly. That's why the helpers communicate via pipes. They don't have
to be included with core git at all; you should be able to just drop a
third-party git-credential-foo into your PATH.

> I'm actually tempted try and work on a couple of those myself.

Please do! I mentioned a few people working on helpers elsewhere in this
thread, so you may want to see what they've done and/or coordinate to
avoid duplicate effort. Let me know if you have trouble finding the
appropriate threads in the list archive.

-Peff

Re: The imporantance of including http credential caching in 1.7.7

[Michael J Gruber]

Jeff King venit, vidit, dixit 08.09.2011 21:18:
> On Thu, Sep 08, 2011 at 11:02:11AM -0400, John Szakmeister wrote:
> 
>> On Thu, Sep 8, 2011 at 9:17 AM, Michael J Gruber
>> <git@drmicha.warpmail.net> wrote:
>> [snip]
>>> It would be interesting to know what we can rely on in the user group
>>> you're thinking about (which I called ssh-challenged). Setting up ssh
>>> keys is too complicated. Can we require a working gpg setup? They do
>>> want to check sigs, don't they?
>>
>> I don't think you can require a working gpg setup (at least for not
>> addressing the ssh-challenged group).
> 
> Agreed. Anything harder than ssh keys is right out the window, because
> they're always the alternative these people could be using (but can't or
> don't want to).

Sue, the question was: What is easy enough? I hoped that people would be
using gpg to check signed tags, and that there might be a simple,
convenient gnupg installer for Win and Mac which ties into the
respective wallet systems or provides one they use already.

> We could make our own gpg-based password wallet system, but I think it's
> a really bad idea, for two reasons:
> 
>   1. It's reinventing the wheel. Which is bad enough as it is, but is
>      doubly bad with security-related code, because it's very easy to
>      screw something up when you're writing a lot of new code.

So please let's not deploy credential-store...

>   2. It's inconvenient for users. Nobody wants a separate wallet system
>      with its own master password. They want to integrate with the
>      wallet system they're already using. Which is generally going to be
>      way nicer _anyway_, because it's going to be part of the OS and do
>      helpful things like unlock the secret store using their login
>      credentials.

On 1.+2.: The idea/hope was to use an existing wallet system which
people use for gnupg already to store their passphrase. If that is not
used then my suggestion does not help much (the issue of widespread
deployment), though it still is a secure version of credential-store for
those who want a desktop-independent secure credential store.

>>> So: What credential store/password wallet/etc. can we rely on for this
>>> group? Is gpg fair game?
>>
>> I think there probably need to be providers for using Keychain under
>> the Mac, gnome-keyring and kwallet under Linux, and probably something
>> using the wincrypt API under Windows.  I don't think there's a
>> one-store-fits-all solution here, unfortunately. :-(
> 
> Exactly. That's why the helpers communicate via pipes. They don't have
> to be included with core git at all; you should be able to just drop a
> third-party git-credential-foo into your PATH.
> 
>> I'm actually tempted try and work on a couple of those myself.
> 
> Please do! I mentioned a few people working on helpers elsewhere in this
> thread, so you may want to see what they've done and/or coordinate to
> avoid duplicate effort. Let me know if you have trouble finding the
> appropriate threads in the list archive.

It seemed appropriate to leverage GitHub for this:

https://github.com/gitigit/git/wiki/Git-Credentials-Hub

Feel free to add!

Cheers,
Michael

Re: The imporantance of including http credential caching in 1.7.7

[Michael J Gruber]

Jeff King venit, vidit, dixit 08.09.2011 21:10:
> On Wed, Sep 07, 2011 at 02:56:03PM +0200, Michael J Gruber wrote:
> 
>> So, it's been a year or more that you've been aware of the importance
>> of this issue (from your/github's perspective), and we hear about it
>> now, at the end of the rc phase. I don't know whether
>> jk/http-auth-keyring has been done on github payroll or during spare
>> time.
> 
> To be absolutely clear here, this feature was 100% paid for by GitHub
> (which isn't to say that I don't think it's a good idea. On the
> contrary, I think it's awesome; but GitHub money is what provided the
> time for me to work on it).
> 
> When I started at GitHub in January, I was given a giant list of things
> that GitHub felt would make core git better, but that they hadn't the
> personnel to improve. And I was told to use my own judgement in adding
> or removing items from the list based on what I thought git needed, and
> to prioritize as I saw fit. The fact that it took six months for me to
> come up with credential patches is because that's how long it took me to
> figure out what I wanted to write, and to clear my backlog of other git
> tasks.
> 
> So I think the wheels have been turning on this for quite a while from
> GitHub's perspective.

Thanks for clarifying. While it should make no difference for the
acceptance of patches, it's great to see GitHub invest into scratching
their Git itches, and thus contribute back. That's how open source works
as a business model :)

...
> In the meantime, the best thing we can do to push it forward is to write
> helpers. I implemented some basic ones that should work anywhere, but
> aren't as nice as integration with existing keychains. Some people are
> working on Linux ones. The single best thing GitHub can do to push this
> forward right now is to provide a well-written OS X Keychain helper, and
> to provide feedback on whether git's end of the API is good enough.

... and one for Git on Windows? It seems we're lacking both Win and OS X
developers here.

... continuing in the other subthread...

Re: The imporantance of including http credential caching in 1.7.7

[Miles Bader]

Michael J Gruber <git@drmicha.warpmail.net> writes:
>> Agreed. Anything harder than ssh keys is right out the window,
>> because they're always the alternative these people could be using
>> (but can't or don't want to).
>
> Sue, the question was: What is easy enough? I hoped that people
> would be using gpg to check signed tags, and that there might be a
> simple, convenient gnupg installer for Win and Mac which ties into
> the respective wallet systems or provides one they use already.

I wouldn't be surprised if many people just don't check signed tags at
all -- if the repositories they're using even have them in the first
place -- particularly amongst the audience in question.

-miles

-- 
What the fuck do white people have to be blue about!?  Banana Republic ran
out of Khakis?  The Espresso Machine is jammed?  Hootie and The Blowfish
are breaking up??!  Shit, white people oughtta understand, their job is to
GIVE people the blues, not to get them!  -- George Carlin

Re: The imporantance of including http credential caching in 1.7.7

[Ted Zlatanov]

On Fri, 09 Sep 2011 10:06:08 +0200 Michael J Gruber <git@drmicha.warpmail.net> wrote: 

>> In the meantime, the best thing we can do to push it forward is to write
>> helpers. I implemented some basic ones that should work anywhere, but
>> aren't as nice as integration with existing keychains. Some people are
>> working on Linux ones. The single best thing GitHub can do to push this
>> forward right now is to provide a well-written OS X Keychain helper, and
>> to provide feedback on whether git's end of the API is good enough.

MJG> ... and one for Git on Windows? It seems we're lacking both Win and OS X
MJG> developers here.

Windows doesn't have a standard keychain service, does it?

The OS X Keychain helper should be pretty easy in terms of the system
calls (he says after a quick Google search), the hard part IMHO is
figuring out the right way to store credentials in it.  There are
several ways to structure the schema.

For modern Linux systems it's best to target the Secrets API, which is
DBUS and XML-based and works with both the KDE and GNOME keychains.  I
only know about it what I have learned from Michael Albinus' interface
in the Emacs source tree, but it certainly seems capable enough.  This
is what Jeff King was alluding to, I think, about what I'm working on.
I have not been able to work on it so far, not for lack of trying.

My #1 target is to implement a GPG-based credential helper using a
netrc-style file.  I believe that would be the most useful one, though
not the easiest one to set up for inexperienced users.

Ted

Re: The imporantance of including http credential caching in 1.7.7

[John Szakmeister]

[Added back some of the CC's]

Ted: we don't usually cull the CC list on the git mailing list.

2011/9/9 Ted Zlatanov <tzz@lifelogs.com>:
[snip]
> MJG> ... and one for Git on Windows? It seems we're lacking both Win and OS X
> MJG> developers here.
>
> Windows doesn't have a standard keychain service, does it?

No, it doesn't, but you can use the wincrypt API which allows you to
at least encrypt the password from the user's login credentials.  In
particular, CryptProtectData() and CryptUnprotectData().  That way you
can at least have the password stored encrypted on disk.

-John

Re: The imporantance of including http credential caching in 1.7.7

[Erik Faye-Lund]

On Fri, Sep 9, 2011 at 12:32 PM, John Szakmeister <john@szakmeister.net> wrote:
> [Added back some of the CC's]
>
> Ted: we don't usually cull the CC list on the git mailing list.
>
> 2011/9/9 Ted Zlatanov <tzz@lifelogs.com>:
> [snip]
>> MJG> ... and one for Git on Windows? It seems we're lacking both Win and OS X
>> MJG> developers here.
>>
>> Windows doesn't have a standard keychain service, does it?
>
> No, it doesn't, but you can use the wincrypt API which allows you to
> at least encrypt the password from the user's login credentials.  In
> particular, CryptProtectData() and CryptUnprotectData().  That way you
> can at least have the password stored encrypted on disk.

Actually, it seems recent Windows versions does have a credential
manager, including an API:

http://www.yanzzee.com/2009/09/windows-keychain.html
http://msdn.microsoft.com/en-us/library/aa374731(v=VS.85).aspx#credentials_management_functions

Re: The imporantance of including http credential caching in 1.7.7

[John Szakmeister]

On Fri, Sep 9, 2011 at 6:48 AM, Erik Faye-Lund <kusmabite@gmail.com> wrote:
[snip]
> Actually, it seems recent Windows versions does have a credential
> manager, including an API:
>
> http://www.yanzzee.com/2009/09/windows-keychain.html
> http://msdn.microsoft.com/en-us/library/aa374731(v=VS.85).aspx#credentials_management_functions

Yay!  It's about time they grew that feature. :-)

-John

Re: The imporantance of including http credential caching in 1.7.7

[Ted Zlatanov]

On Fri, 9 Sep 2011 06:32:25 -0400 John Szakmeister <john@szakmeister.net> wrote: 

JS> [Added back some of the CC's]
JS> Ted: we don't usually cull the CC list on the git mailing list.

Sorry, I followed up via GMane (a NNTP interface to the mailing list).
I'll use `r'eply instead of `f'ollowup.

I actually set

Mail-Copies-To: never
Gmane-Reply-To-List: yes

because I hate getting CC'd on discussions I already follow via GMane.
But that's just my preference :)

JS> 2011/9/9 Ted Zlatanov <tzz@lifelogs.com>:
JS> [snip]
MJG> ... and one for Git on Windows? It seems we're lacking both Win and OS X
MJG> developers here.
>> 
>> Windows doesn't have a standard keychain service, does it?

JS> No, it doesn't, but you can use the wincrypt API which allows you to
JS> at least encrypt the password from the user's login credentials.  In
JS> particular, CryptProtectData() and CryptUnprotectData().  That way you
JS> can at least have the password stored encrypted on disk.

I don't think that's sufficient but could be wrong.

Ted

Re: The imporantance of including http credential caching in 1.7.7

[Ted Zlatanov]

On Fri, 9 Sep 2011 06:54:15 -0400 John Szakmeister <john@szakmeister.net> wrote: 

JS> On Fri, Sep 9, 2011 at 6:48 AM, Erik Faye-Lund <kusmabite@gmail.com> wrote:
JS> [snip]
>> Actually, it seems recent Windows versions does have a credential
>> manager, including an API:
>> 
>> http://www.yanzzee.com/2009/09/windows-keychain.html
>> http://msdn.microsoft.com/en-us/library/aa374731(v=VS.85).aspx#credentials_management_functions

JS> Yay!  It's about time they grew that feature. :-)

That could work.  I hope a Windows developer can take a look.  That
leaves VMS credential support... Come on guys!

Ted

Re: The imporantance of including http credential caching in 1.7.7

[Jeff King]

On Fri, Sep 09, 2011 at 10:05:48AM +0200, Michael J Gruber wrote:

> > Agreed. Anything harder than ssh keys is right out the window, because
> > they're always the alternative these people could be using (but can't or
> > don't want to).
> 
> Sue, the question was: What is easy enough? I hoped that people would be
> using gpg to check signed tags, and that there might be a simple,
> convenient gnupg installer for Win and Mac which ties into the
> respective wallet systems or provides one they use already.

I suspect most people aren't checking signed tags. And even if they did
have gpg installed, most people aren't going to want a new password
wallet.  They're going to want integration with what they're already
using.

Which isn't to say that a gpg-based wallet is wrong, it's just that I
don't think it's filling the role that really needs filled. If you want
to make such a wallet helper, you're welcome to. But it doesn't
necessarily need to be a part of git core, and if it's not, then maybe
it's worth looking at the zillion other password wallet programs that
exist.

FWIW, I keep my passwords in a gpg-encrypted file and wrote a 10-line
shell script helper to do lookups for git. :)

> > We could make our own gpg-based password wallet system, but I think it's
> > a really bad idea, for two reasons:
> > 
> >   1. It's reinventing the wheel. Which is bad enough as it is, but is
> >      doubly bad with security-related code, because it's very easy to
> >      screw something up when you're writing a lot of new code.
> 
> So please let's not deploy credential-store...

I'm tempted to agree. But I also think it represents a nice lowest
common denominator. No hassle, no setup, but no security either. And
there are situations where that's appropriate (e.g., for unattended
cron operation, it's not much different than an unencrypted ssh key on
disk). My compromise was to put a big warning at the top of the
documentation. Maybe that's not enough, though.

And as far as reinventing the wheel with security code, I don't think
git-credential-store counts. It's not secure at all, so there's very
little to screw up. :)

> On 1.+2.: The idea/hope was to use an existing wallet system which
> people use for gnupg already to store their passphrase. If that is not
> used then my suggestion does not help much (the issue of widespread
> deployment), though it still is a secure version of credential-store for
> those who want a desktop-independent secure credential store.

Yeah, if there is an existing wallet system based around gpg, then
absolutely there should be a helper for it. But I don't know that there
is such a widely deployed system. And the helper for it doesn't need to
ship with git-core; anybody who uses their wallet system is free to
write and distribute the helper.

-Peff

Re: The imporantance of including http credential caching in 1.7.7

[Jeff King]

On Fri, Sep 09, 2011 at 10:06:08AM +0200, Michael J Gruber wrote:

> > So I think the wheels have been turning on this for quite a while from
> > GitHub's perspective.
> 
> Thanks for clarifying. While it should make no difference for the
> acceptance of patches, it's great to see GitHub invest into scratching
> their Git itches, and thus contribute back. That's how open source works
> as a business model :)

Yes. I don't often enough mention how awesome GitHub is for funding me
and giving me a free hand to improve git. They're doing everything
right. So let me mention it here one more time. :)

> > In the meantime, the best thing we can do to push it forward is to write
> > helpers. I implemented some basic ones that should work anywhere, but
> > aren't as nice as integration with existing keychains. Some people are
> > working on Linux ones. The single best thing GitHub can do to push this
> > forward right now is to provide a well-written OS X Keychain helper, and
> > to provide feedback on whether git's end of the API is good enough.
> 
> ... and one for Git on Windows? It seems we're lacking both Win and OS X
> developers here.

I mentioned OS X because of Kyle's mention of the GitHub for Mac client.
But yes, I do think in the long term we want something similar on
Windows. GitHub recently hired a developer with some Windows experience;
I'll try to see if he's interested in writing a credential helper.

-Peff