[PATCH 00/13] Support for arbitrary schemes in credentials

[PATCH 00/13] Support for arbitrary schemes in credentials

[brian m. carlson]

Right now, HTTP authentication in Git is mostly limited to approaches
that require a username and password or are Kerberos (GSSAPI).  In
addition, we effectively require that libcurl (or, for other software,
such as Git LFS, using the credential helper, that HTTP library) knows
how to implement the authentication scheme.

However, this poses two sets of problems.  First, some sites, such as
Azure DevOps, want to use Bearer authentication, which we don't support.
This is implemented using `http.extraHeader`, which is not a secure way
to store credentials, since our credential helper protocol does not
support this functionality.

In addition, other tools using the credential helper protocol do not
support the variety of authentication mechanisms that Git does.
Specifically, making NTLM function in a useful way on Windows is
nontrivial and requires extensive integration and testing with C code,
and because of this difficulty and the fact that NTLM uses cryptography
known to be insecure since 1995, there is often little interest in
implementing this support outside of libcurl. However, it would be
helpful if people who want to use it can still use it.

This series introduces new functionality to the credential helper
protocol that allows helpers to produce credentials for arbitrary HTTP
authentication schemes using the `authtype` and `credential`[0] fields.
This allows a suitable credential helper to send Bearer credentials or
any other standard or custom authentication scheme.  (It may be able to
be extended to other functionality in the future, such as
git-send-email, to implement custom SASL functionality, and due care has
been taken to make the protocol adequately flexible for that purpose.)

In addition, the protocol is also expanded to include per-helper state
and multi-legged authentication (the former is effectively required for
the latter).  The per-helper state can be useful to help credential
helpers identify where the credential is stored, or any other
information necessary.  Because NTLM and Negotiate (Kerberos/wrapped
NTLM) require two rounds of authentication, the multi-legged
authentication support along with per-helper state allows the helper to
support these authentication methods without Git or other clients having
to be aware of how they work.  (This would also be useful for SASL, as
mentioned above.)

This series introduces a capability mechanism to announce this
functionality, which allows a helper to provide a username and password
on older versions of Git while supporting more advanced functionality on
newer versions.  (This is especially important on Azure DevOps, where
NTLM uses a username and password but Basic or Bearer can use a personal
access token.)  It is also designed such that extremely simple
credential helpers, such as the shell one-liner in the Git FAQ that
reads from the environment, don't accidentally claim to support
functionality they don't offer.

In addition, there is documentation for the expanded protocol, although
none of the built-in helpers have been updated (that will be a future
series for those for which it's possible).

My personal interest here is getting credentials out of config files
with `http.extraHeader` (which a future series will produce a warning
for) and also allowing Git LFS to support Digest and NTLM with a
suitable credential helper.  Git LFS used to support NTLM using custom
code (because the Go standard library does not), but it was found to be
broken in lots of ways on Windows, and nobody with a Windows system
wanted to fix it or support it, so we removed it.  However, there are
still some people who do want to use it, so allowing them to use a
custom credential helper they maintain themselves seems like the best
way forward.  Despite the advantages of this series for Azure DevOps, I
have no personal or professional stake in their product; my only
interest is the general one in whether their users can securely store
credentials.  I believe the changes here are of general advantage to the
Git userbase in a variety of ways such that the goal of this series
should be uncontroversial.

Feedback on any portion of this series is of course welcome.

[0] A name different from `password` was explicitly chosen to avoid
confusion from less capable protocol helpers so that they don't
accidentally send invalid data.  This does have the downside that
credential helpers must learn a new field to not log, but that should be
generally easy to fix in most cases.

brian m. carlson (13):
  credential: add an authtype field
  remote-curl: reset headers on new request
  http: use new headers for each object request
  credential: add a field for pre-encoded credentials
  credential: gate new fields on capability
  docs: indicate new credential protocol fields
  http: add support for authtype and credential
  credential: add an argument to keep state
  credential: enable state capability
  docs: set a limit on credential line length
  t5563: refactor for multi-stage authentication
  strvec: implement swapping two strvecs
  credential: add support for multistage credential rounds

 Documentation/git-credential.txt   |  59 +++++-
 builtin/credential-cache--daemon.c |   2 +-
 builtin/credential-store.c         |   2 +-
 builtin/credential.c               |   7 +-
 credential.c                       | 114 ++++++++++-
 credential.h                       |  69 ++++++-
 http.c                             | 128 +++++++-----
 http.h                             |   5 +
 imap-send.c                        |   2 +-
 remote-curl.c                      |  14 +-
 strvec.c                           |   7 +
 strvec.h                           |   5 +
 t/lib-httpd/nph-custom-auth.sh     |  17 +-
 t/t0300-credentials.sh             | 136 ++++++++++++-
 t/t5563-simple-http-auth.sh        | 308 +++++++++++++++++++++++++----
 15 files changed, 760 insertions(+), 115 deletions(-)

[PATCH 01/13] credential: add an authtype field

[brian m. carlson]

When Git makes an HTTP request, it can negotiate the type of
authentication to use with the server provided the authentication scheme
is one of a few well-known types (Basic, Digest, NTLM, or Negotiate).
However, some servers wish to use other types of authentication, such as
the Bearer type from OAuth2.  Since libcurl doesn't natively support
this type, it isn't possible to use it, and the user is forced to
specify the Authorization header using the http.extraheader setting.

However, storing a plaintext token in the repository configuration is
not very secure, especially if a repository can be shared by multiple
parties.  We already have support for many types of secure credential
storage by using credential helpers, so let's teach credential helpers
how to produce credentials for an arbitrary scheme.

If the credential helper specifies an authtype field, then it specifies
an authentication scheme (e.g., Bearer) and the password field specifies
the raw authentication token, with any encoding already specified.  We
reuse the password field for this because some credential helpers store
the metadata without encryption even though the password is encrypted,
and we'd like to avoid insecure storage if an older version of the
credential helper gets ahold of the data.

The username is not used in this case, but it is still preserved for the
purpose of finding the right credential if the user has multiple
accounts.

If the authtype field is not specified, then the password behaves as
normal and it is passed along with the username to libcurl.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 credential.c | 5 +++++
 credential.h | 6 ++++++
 2 files changed, 11 insertions(+)

diff --git a/credential.c b/credential.c
index 18098bd35e..3dec433df5 100644
--- a/credential.c
+++ b/credential.c
@@ -26,6 +26,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	free(c->oauth_refresh_token);
+	free(c->authtype);
 	string_list_clear(&c->helpers, 0);
 	strvec_clear(&c->wwwauth_headers);
 
@@ -252,6 +253,9 @@ int credential_read(struct credential *c, FILE *fp)
 		} else if (!strcmp(key, "oauth_refresh_token")) {
 			free(c->oauth_refresh_token);
 			c->oauth_refresh_token = xstrdup(value);
+		} else if (!strcmp(key, "authtype")) {
+			free(c->authtype);
+			c->authtype = xstrdup(value);
 		} else if (!strcmp(key, "url")) {
 			credential_from_url(c, value);
 		} else if (!strcmp(key, "quit")) {
@@ -295,6 +299,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	}
 	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
 		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
+	credential_write_item(fp, "authtype", c->authtype, 0);
 }
 
 static int run_credential_helper(struct credential *c,
diff --git a/credential.h b/credential.h
index acc41adf54..dc96ca0318 100644
--- a/credential.h
+++ b/credential.h
@@ -143,6 +143,12 @@ struct credential {
 	char *path;
 	char *oauth_refresh_token;
 	timestamp_t password_expiry_utc;
+
+	/**
+	 * The authorization scheme to use.  If this is NULL, libcurl is free to
+	 * negotiate any scheme it likes.
+	 */
+	char *authtype;
 };
 
 #define CREDENTIAL_INIT { \

[PATCH 02/13] remote-curl: reset headers on new request

[brian m. carlson]

When we retry a post_rpc request, we currently reuse the same headers as
before.  In the future, we'd like to be able to modify them based on the
result we get back, so let's reset them on each retry so we can avoid
sending potentially duplicate headers if the values change.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 remote-curl.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/remote-curl.c b/remote-curl.c
index 1161dc7fed..e37eaa17b7 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -893,7 +893,7 @@ static curl_off_t xcurl_off_t(size_t len)
 static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_received)
 {
 	struct active_request_slot *slot;
-	struct curl_slist *headers = http_copy_default_headers();
+	struct curl_slist *headers = NULL;
 	int use_gzip = rpc->gzip_request;
 	char *gzip_body = NULL;
 	size_t gzip_size = 0;
@@ -935,6 +935,8 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 			needs_100_continue = 1;
 	}
 
+retry:
+	headers = http_copy_default_headers();
 	headers = curl_slist_append(headers, rpc->hdr_content_type);
 	headers = curl_slist_append(headers, rpc->hdr_accept);
 	headers = curl_slist_append(headers, needs_100_continue ?
@@ -948,7 +950,6 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 	if (rpc->protocol_header)
 		headers = curl_slist_append(headers, rpc->protocol_header);
 
-retry:
 	slot = get_active_slot();
 
 	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
@@ -1044,6 +1045,7 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 	err = run_slot(slot, NULL);
 	if (err == HTTP_REAUTH && !large_request) {
 		credential_fill(&http_auth);
+		curl_slist_free_all(headers);
 		goto retry;
 	}
 	if (err != HTTP_OK)

[PATCH 03/13] http: use new headers for each object request

[brian m. carlson]

Currently we create one set of headers for all object requests and reuse
it.  However, we'll need to adjust the headers for authentication
purposes in the future, so let's create a new set for each request so
that we can adjust them if the authentication changes.

Note that the cost of allocation here is tiny compared to the fact that
we're making a network call, not to mention probably a full TLS
connection, so this shouldn't have a significant impact on performance.
Moreover, nobody who cares about performance is using the dumb HTTP
protocol anyway, since it often makes huge numbers of requests compared
to the smart protocol.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 http.c | 19 +++++++++++--------
 http.h |  2 ++
 2 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/http.c b/http.c
index e73b136e58..1c2200da77 100644
--- a/http.c
+++ b/http.c
@@ -128,7 +128,6 @@ static unsigned long empty_auth_useless =
 	| CURLAUTH_DIGEST;
 
 static struct curl_slist *pragma_header;
-static struct curl_slist *no_pragma_header;
 static struct string_list extra_http_headers = STRING_LIST_INIT_DUP;
 
 static struct curl_slist *host_resolutions;
@@ -299,6 +298,11 @@ size_t fwrite_null(char *ptr UNUSED, size_t eltsize UNUSED, size_t nmemb,
 	return nmemb;
 }
 
+static struct curl_slist *object_request_headers(void)
+{
+	return curl_slist_append(http_copy_default_headers(), "Pragma:");
+}
+
 static void closedown_active_slot(struct active_request_slot *slot)
 {
 	active_requests--;
@@ -1275,8 +1279,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)
 
 	pragma_header = curl_slist_append(http_copy_default_headers(),
 		"Pragma: no-cache");
-	no_pragma_header = curl_slist_append(http_copy_default_headers(),
-		"Pragma:");
 
 	{
 		char *http_max_requests = getenv("GIT_HTTP_MAX_REQUESTS");
@@ -1360,8 +1362,6 @@ void http_cleanup(void)
 	curl_slist_free_all(pragma_header);
 	pragma_header = NULL;
 
-	curl_slist_free_all(no_pragma_header);
-	no_pragma_header = NULL;
 
 	curl_slist_free_all(host_resolutions);
 	host_resolutions = NULL;
@@ -2370,6 +2370,7 @@ void release_http_pack_request(struct http_pack_request *preq)
 	}
 	preq->slot = NULL;
 	strbuf_release(&preq->tmpfile);
+	curl_slist_free_all(preq->headers);
 	free(preq->url);
 	free(preq);
 }
@@ -2454,11 +2455,11 @@ struct http_pack_request *new_direct_http_pack_request(
 	}
 
 	preq->slot = get_active_slot();
+	preq->headers = object_request_headers();
 	curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEDATA, preq->packfile);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_URL, preq->url);
-	curl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER,
-		no_pragma_header);
+	curl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER, preq->headers);
 
 	/*
 	 * If there is data present from a previous transfer attempt,
@@ -2624,13 +2625,14 @@ struct http_object_request *new_http_object_request(const char *base_url,
 	}
 
 	freq->slot = get_active_slot();
+	freq->headers = object_request_headers();
 
 	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEDATA, freq);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_FAILONERROR, 0);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);
-	curl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
+	curl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, freq->headers);
 
 	/*
 	 * If we have successfully processed data from a previous fetch
@@ -2718,5 +2720,6 @@ void release_http_object_request(struct http_object_request *freq)
 		release_active_slot(freq->slot);
 		freq->slot = NULL;
 	}
+	curl_slist_free_all(freq->headers);
 	strbuf_release(&freq->tmpfile);
 }
diff --git a/http.h b/http.h
index 3af19a8bf5..c5f8cc4620 100644
--- a/http.h
+++ b/http.h
@@ -196,6 +196,7 @@ struct http_pack_request {
 	FILE *packfile;
 	struct strbuf tmpfile;
 	struct active_request_slot *slot;
+	struct curl_slist *headers;
 };
 
 struct http_pack_request *new_http_pack_request(
@@ -229,6 +230,7 @@ struct http_object_request {
 	int zret;
 	int rename;
 	struct active_request_slot *slot;
+	struct curl_slist *headers;
 };
 
 struct http_object_request *new_http_object_request(

[PATCH 04/13] credential: add a field for pre-encoded credentials

[brian m. carlson]

At the moment, our credential code wants to find a username and password
for access, which, for HTTP, it will pass to libcurl to encode and
process.  However, many users want to use authentication schemes that
libcurl doesn't support, such as Bearer authentication.  In these
schemes, the secret is not a username and password pair, but some sort
of token that meets the production for authentication data in the RFC.

In fact, in general, it's useful to allow our credential helper to have
knowledge about what specifically to put in the protocol header.  Thus,
add a field, credential, which contains data that's preencoded to be
suitable for the protocol in question.  If we have such data, we need
neither a username nor a password, so make that adjustment as well.

It is in theory possible to reuse the password field for this.  However,
if we do so, we must know whether the credential helper supports our new
scheme before sending it data, which necessitates some sort of
capability inquiry, because otherwise an uninformed credential helper
would store our preencoded data as a password, which would fail the next
time we attempted to connect to the remote server.  This design is
substantially simpler, and we can hint to the credential helper that we
support this approach with a simple new field instead of needing to
query it first.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 credential.c | 14 ++++++++++----
 credential.h |  1 +
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/credential.c b/credential.c
index 3dec433df5..c521822e5a 100644
--- a/credential.c
+++ b/credential.c
@@ -25,6 +25,7 @@ void credential_clear(struct credential *c)
 	free(c->path);
 	free(c->username);
 	free(c->password);
+	free(c->credential);
 	free(c->oauth_refresh_token);
 	free(c->authtype);
 	string_list_clear(&c->helpers, 0);
@@ -234,6 +235,9 @@ int credential_read(struct credential *c, FILE *fp)
 		} else if (!strcmp(key, "password")) {
 			free(c->password);
 			c->password = xstrdup(value);
+		} else if (!strcmp(key, "credential")) {
+			free(c->credential);
+			c->credential = xstrdup(value);
 		} else if (!strcmp(key, "protocol")) {
 			free(c->protocol);
 			c->protocol = xstrdup(value);
@@ -291,6 +295,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_item(fp, "credential", c->credential, 0);
 	credential_write_item(fp, "oauth_refresh_token", c->oauth_refresh_token, 0);
 	if (c->password_expiry_utc != TIME_MAX) {
 		char *s = xstrfmt("%"PRItime, c->password_expiry_utc);
@@ -366,7 +371,7 @@ void credential_fill(struct credential *c)
 {
 	int i;
 
-	if (c->username && c->password)
+	if ((c->username && c->password) || c->credential)
 		return;
 
 	credential_apply_config(c);
@@ -379,7 +384,7 @@ void credential_fill(struct credential *c)
 			/* Reset expiry to maintain consistency */
 			c->password_expiry_utc = TIME_MAX;
 		}
-		if (c->username && c->password)
+		if ((c->username && c->password) || c->credential)
 			return;
 		if (c->quit)
 			die("credential helper '%s' told us to quit",
@@ -387,7 +392,7 @@ void credential_fill(struct credential *c)
 	}
 
 	credential_getpass(c);
-	if (!c->username && !c->password)
+	if (!c->username && !c->password && !c->credential)
 		die("unable to get password from user");
 }
 
@@ -397,7 +402,7 @@ void credential_approve(struct credential *c)
 
 	if (c->approved)
 		return;
-	if (!c->username || !c->password || c->password_expiry_utc < time(NULL))
+	if (((!c->username || !c->password) && !c->credential) || c->password_expiry_utc < time(NULL))
 		return;
 
 	credential_apply_config(c);
@@ -418,6 +423,7 @@ void credential_reject(struct credential *c)
 
 	FREE_AND_NULL(c->username);
 	FREE_AND_NULL(c->password);
+	FREE_AND_NULL(c->credential);
 	FREE_AND_NULL(c->oauth_refresh_token);
 	c->password_expiry_utc = TIME_MAX;
 	c->approved = 0;
diff --git a/credential.h b/credential.h
index dc96ca0318..9db892cf4d 100644
--- a/credential.h
+++ b/credential.h
@@ -138,6 +138,7 @@ struct credential {
 
 	char *username;
 	char *password;
+	char *credential;
 	char *protocol;
 	char *host;
 	char *path;

[PATCH 05/13] credential: gate new fields on capability

[brian m. carlson]

We support the new credential and authtype fields, but we lack a way to
indicate to a credential helper that we'd like them to be used.  Without
some sort of indication, the credential helper doesn't know if it should
try to provide us a username and password, or a pre-encoded credential.
For example, the helper might prefer a more restricted Bearer token if
pre-encoded credentials are possible, but might have to fall back to
more general username and password if not.

Let's provide a simple way to indicate whether Git (or, for that matter,
the helper) is capable of understanding the authtype and credential
fields.  We send this capability when we generate a request, and the
other side may reply to indicate to us that it does, too.

For now, don't enable sending capabilities for the HTTP code.  In a
future commit, we'll introduce appropriate handling for that code,
which requires more in-depth work.

The logic for determining whether a capability is supported may seem
complex, but it is not.  At each stage, we emit the capability to the
following stage if all preceding stages have declared it.  Thus, if the
caller to git credential fill didn't declare it, then we won't send it
to the helper, and if fill's caller did send but the helper doesn't
understand it, then we won't send it on in the response.  If we're an
internal user, then we know about all capabilities and will request
them.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 builtin/credential-cache--daemon.c |   2 +-
 builtin/credential-store.c         |   2 +-
 builtin/credential.c               |   6 +-
 credential.c                       |  58 ++++++++++++++--
 credential.h                       |  30 +++++++-
 http.c                             |  10 +--
 imap-send.c                        |   2 +-
 remote-curl.c                      |   4 +-
 t/t0300-credentials.sh             | 107 ++++++++++++++++++++++++++++-
 9 files changed, 197 insertions(+), 24 deletions(-)

diff --git a/builtin/credential-cache--daemon.c b/builtin/credential-cache--daemon.c
index 3a6a750a8e..ccbcf99ac1 100644
--- a/builtin/credential-cache--daemon.c
+++ b/builtin/credential-cache--daemon.c
@@ -115,7 +115,7 @@ static int read_request(FILE *fh, struct credential *c,
 		return error("client sent bogus timeout line: %s", item.buf);
 	*timeout = atoi(p);
 
-	if (credential_read(c, fh) < 0)
+	if (credential_read(c, fh, CREDENTIAL_OP_HELPER) < 0)
 		return -1;
 	return 0;
 }
diff --git a/builtin/credential-store.c b/builtin/credential-store.c
index 4a492411bb..494c809332 100644
--- a/builtin/credential-store.c
+++ b/builtin/credential-store.c
@@ -205,7 +205,7 @@ int cmd_credential_store(int argc, const char **argv, const char *prefix)
 	if (!fns.nr)
 		die("unable to set up default path; use --file");
 
-	if (credential_read(&c, stdin) < 0)
+	if (credential_read(&c, stdin, CREDENTIAL_OP_HELPER) < 0)
 		die("unable to read credential");
 
 	if (!strcmp(op, "get"))
diff --git a/builtin/credential.c b/builtin/credential.c
index 7010752987..5123dabcf1 100644
--- a/builtin/credential.c
+++ b/builtin/credential.c
@@ -17,12 +17,12 @@ int cmd_credential(int argc, const char **argv, const char *prefix UNUSED)
 		usage(usage_msg);
 	op = argv[1];
 
-	if (credential_read(&c, stdin) < 0)
+	if (credential_read(&c, stdin, CREDENTIAL_OP_INITIAL) < 0)
 		die("unable to read credential from stdin");
 
 	if (!strcmp(op, "fill")) {
-		credential_fill(&c);
-		credential_write(&c, stdout);
+		credential_fill(&c, 0);
+		credential_write(&c, stdout, CREDENTIAL_OP_RESPONSE);
 	} else if (!strcmp(op, "approve")) {
 		credential_approve(&c);
 	} else if (!strcmp(op, "reject")) {
diff --git a/credential.c b/credential.c
index c521822e5a..f2a26b8672 100644
--- a/credential.c
+++ b/credential.c
@@ -34,6 +34,11 @@ void credential_clear(struct credential *c)
 	credential_init(c);
 }
 
+static void credential_set_all_capabilities(struct credential *c)
+{
+	c->capa_authtype.request_initial = 1;
+}
+
 int credential_match(const struct credential *want,
 		     const struct credential *have, int match_password)
 {
@@ -210,7 +215,39 @@ static void credential_getpass(struct credential *c)
 						 PROMPT_ASKPASS);
 }
 
-int credential_read(struct credential *c, FILE *fp)
+static void credential_set_capability(struct credential_capability *capa, int op_type)
+{
+	switch (op_type) {
+	case CREDENTIAL_OP_INITIAL:
+		capa->request_initial = 1;
+		break;
+	case CREDENTIAL_OP_HELPER:
+		capa->request_helper = 1;
+		break;
+	case CREDENTIAL_OP_RESPONSE:
+		capa->response = 1;
+		break;
+	}
+}
+
+static int credential_has_capability(const struct credential_capability *capa, int op_type)
+{
+	/*
+	 * We're checking here if each previous step indicated that we had the
+	 * capability.  If it did, then we want to pass it along; conversely, if
+	 * it did not, we don't want to report that to our caller.
+	 */
+	switch (op_type) {
+	case CREDENTIAL_OP_HELPER:
+		return capa->request_initial;
+	case CREDENTIAL_OP_RESPONSE:
+		return capa->request_initial && capa->request_helper;
+	default:
+		return 0;
+	}
+}
+
+int credential_read(struct credential *c, FILE *fp, int op_type)
 {
 	struct strbuf line = STRBUF_INIT;
 
@@ -249,6 +286,8 @@ int credential_read(struct credential *c, FILE *fp)
 			c->path = xstrdup(value);
 		} else if (!strcmp(key, "wwwauth[]")) {
 			strvec_push(&c->wwwauth_headers, value);
+		} else if (!strcmp(key, "capability[]") && !strcmp(value, "authtype")) {
+			credential_set_capability(&c->capa_authtype, op_type);
 		} else if (!strcmp(key, "password_expiry_utc")) {
 			errno = 0;
 			c->password_expiry_utc = parse_timestamp(value, NULL, 10);
@@ -288,14 +327,18 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
 	fprintf(fp, "%s=%s\n", key, value);
 }
 
-void credential_write(const struct credential *c, FILE *fp)
+void credential_write(const struct credential *c, FILE *fp, int op_type)
 {
+	if (credential_has_capability(&c->capa_authtype, op_type)) {
+		credential_write_item(fp, "capability[]", "authtype", 0);
+		credential_write_item(fp, "authtype", c->authtype, 0);
+		credential_write_item(fp, "credential", c->credential, 0);
+	}
 	credential_write_item(fp, "protocol", c->protocol, 1);
 	credential_write_item(fp, "host", c->host, 1);
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
-	credential_write_item(fp, "credential", c->credential, 0);
 	credential_write_item(fp, "oauth_refresh_token", c->oauth_refresh_token, 0);
 	if (c->password_expiry_utc != TIME_MAX) {
 		char *s = xstrfmt("%"PRItime, c->password_expiry_utc);
@@ -304,7 +347,6 @@ void credential_write(const struct credential *c, FILE *fp)
 	}
 	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
 		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
-	credential_write_item(fp, "authtype", c->authtype, 0);
 }
 
 static int run_credential_helper(struct credential *c,
@@ -327,14 +369,14 @@ static int run_credential_helper(struct credential *c,
 
 	fp = xfdopen(helper.in, "w");
 	sigchain_push(SIGPIPE, SIG_IGN);
-	credential_write(c, fp);
+	credential_write(c, fp, want_output ? CREDENTIAL_OP_HELPER : CREDENTIAL_OP_RESPONSE);
 	fclose(fp);
 	sigchain_pop(SIGPIPE);
 
 	if (want_output) {
 		int r;
 		fp = xfdopen(helper.out, "r");
-		r = credential_read(c, fp);
+		r = credential_read(c, fp, CREDENTIAL_OP_HELPER);
 		fclose(fp);
 		if (r < 0) {
 			finish_command(&helper);
@@ -367,7 +409,7 @@ static int credential_do(struct credential *c, const char *helper,
 	return r;
 }
 
-void credential_fill(struct credential *c)
+void credential_fill(struct credential *c, int all_capabilities)
 {
 	int i;
 
@@ -375,6 +417,8 @@ void credential_fill(struct credential *c)
 		return;
 
 	credential_apply_config(c);
+	if (all_capabilities)
+		credential_set_all_capabilities(c);
 
 	for (i = 0; i < c->helpers.nr; i++) {
 		credential_do(c, c->helpers.items[i].string, "get");
diff --git a/credential.h b/credential.h
index 9db892cf4d..2051d04c5a 100644
--- a/credential.h
+++ b/credential.h
@@ -93,6 +93,25 @@
  * -----------------------------------------------------------------------
  */
 
+/*
+ * These values define the kind of operation we're performing and the
+ * capabilities at each stage.  The first is either an external request (via git
+ * credential fill) or an internal request (e.g., via the HTTP) code.  The
+ * second is the call to the credential helper, and the third is the response
+ * we're providing.
+ *
+ * At each stage, we will emit the capability only if the previous stage
+ * supported it.
+ */
+#define CREDENTIAL_OP_INITIAL  1
+#define CREDENTIAL_OP_HELPER   2
+#define CREDENTIAL_OP_RESPONSE 3
+
+struct credential_capability {
+	unsigned request_initial:1,
+		 request_helper:1,
+		 response:1;
+};
 
 /**
  * This struct represents a single username/password combination
@@ -136,6 +155,8 @@ struct credential {
 		 use_http_path:1,
 		 username_from_proto:1;
 
+	struct credential_capability capa_authtype;
+
 	char *username;
 	char *password;
 	char *credential;
@@ -174,8 +195,11 @@ void credential_clear(struct credential *);
  * returns, the username and password fields of the credential are
  * guaranteed to be non-NULL. If an error occurs, the function will
  * die().
+ *
+ * If all_capabilities is set, this is an internal user that is prepared
+ * to deal with all known capabilities, and we should advertise that fact.
  */
-void credential_fill(struct credential *);
+void credential_fill(struct credential *, int all_capabilities);
 
 /**
  * Inform the credential subsystem that the provided credentials
@@ -198,8 +222,8 @@ void credential_approve(struct credential *);
  */
 void credential_reject(struct credential *);
 
-int credential_read(struct credential *, FILE *);
-void credential_write(const struct credential *, FILE *);
+int credential_read(struct credential *, FILE *, int);
+void credential_write(const struct credential *, FILE *, int);
 
 /*
  * Parse a url into a credential struct, replacing any existing contents.
diff --git a/http.c b/http.c
index 1c2200da77..4f5df6fb14 100644
--- a/http.c
+++ b/http.c
@@ -569,7 +569,7 @@ static void init_curl_http_auth(CURL *result)
 		return;
 	}
 
-	credential_fill(&http_auth);
+	credential_fill(&http_auth, 0);
 
 	curl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);
 	curl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);
@@ -596,7 +596,7 @@ static void init_curl_proxy_auth(CURL *result)
 {
 	if (proxy_auth.username) {
 		if (!proxy_auth.password)
-			credential_fill(&proxy_auth);
+			credential_fill(&proxy_auth, 0);
 		set_proxyauth_name_password(result);
 	}
 
@@ -630,7 +630,7 @@ static int has_cert_password(void)
 		cert_auth.host = xstrdup("");
 		cert_auth.username = xstrdup("");
 		cert_auth.path = xstrdup(ssl_cert);
-		credential_fill(&cert_auth);
+		credential_fill(&cert_auth, 0);
 	}
 	return 1;
 }
@@ -645,7 +645,7 @@ static int has_proxy_cert_password(void)
 		proxy_cert_auth.host = xstrdup("");
 		proxy_cert_auth.username = xstrdup("");
 		proxy_cert_auth.path = xstrdup(http_proxy_ssl_cert);
-		credential_fill(&proxy_cert_auth);
+		credential_fill(&proxy_cert_auth, 0);
 	}
 	return 1;
 }
@@ -2191,7 +2191,7 @@ static int http_request_reauth(const char *url,
 		BUG("Unknown http_request target");
 	}
 
-	credential_fill(&http_auth);
+	credential_fill(&http_auth, 0);
 
 	return http_request(url, result, target, options);
 }
diff --git a/imap-send.c b/imap-send.c
index f2e1947e63..8c89e866b6 100644
--- a/imap-send.c
+++ b/imap-send.c
@@ -944,7 +944,7 @@ static void server_fill_credential(struct imap_server_conf *srvc, struct credent
 	cred->username = xstrdup_or_null(srvc->user);
 	cred->password = xstrdup_or_null(srvc->pass);
 
-	credential_fill(cred);
+	credential_fill(cred, 1);
 
 	if (!srvc->user)
 		srvc->user = xstrdup(cred->username);
diff --git a/remote-curl.c b/remote-curl.c
index e37eaa17b7..f96bda2431 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -926,7 +926,7 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 		do {
 			err = probe_rpc(rpc, &results);
 			if (err == HTTP_REAUTH)
-				credential_fill(&http_auth);
+				credential_fill(&http_auth, 0);
 		} while (err == HTTP_REAUTH);
 		if (err != HTTP_OK)
 			return -1;
@@ -1044,7 +1044,7 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 	rpc->any_written = 0;
 	err = run_slot(slot, NULL);
 	if (err == HTTP_REAUTH && !large_request) {
-		credential_fill(&http_auth);
+		credential_fill(&http_auth, 0);
 		curl_slist_free_all(headers);
 		goto retry;
 	}
diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
index 400f6bdbca..8477108b28 100755
--- a/t/t0300-credentials.sh
+++ b/t/t0300-credentials.sh
@@ -12,7 +12,13 @@ test_expect_success 'setup helper scripts' '
 	IFS==
 	while read key value; do
 		echo >&2 "$whoami: $key=$value"
-		eval "$key=$value"
+		if test -z "${key%%*\[\]}"
+		then
+			key=${key%%\[\]}
+			eval "$key=\"\$$key $value\""
+		else
+			eval "$key=$value"
+		fi
 	done
 	IFS=$OIFS
 	EOF
@@ -35,6 +41,16 @@ test_expect_success 'setup helper scripts' '
 	test -z "$pass" || echo password=$pass
 	EOF
 
+	write_script git-credential-verbatim-cred <<-\EOF &&
+	authtype=$1; shift
+	credential=$1; shift
+	. ./dump
+	echo capability[]=authtype
+	test -z "${capability##*authtype*}" || return
+	test -z "$authtype" || echo authtype=$authtype
+	test -z "$credential" || echo credential=$credential
+	EOF
+
 	write_script git-credential-verbatim-with-expiry <<-\EOF &&
 	user=$1; shift
 	pass=$1; shift
@@ -64,6 +80,26 @@ test_expect_success 'credential_fill invokes helper' '
 	EOF
 '
 
+test_expect_success 'credential_fill invokes helper with credential' '
+	check fill "verbatim-cred Bearer token" <<-\EOF
+	capability[]=authtype
+	protocol=http
+	host=example.com
+	--
+	capability[]=authtype
+	authtype=Bearer
+	credential=token
+	protocol=http
+	host=example.com
+	--
+	verbatim-cred: get
+	verbatim-cred: capability[]=authtype
+	verbatim-cred: protocol=http
+	verbatim-cred: host=example.com
+	EOF
+'
+
+
 test_expect_success 'credential_fill invokes multiple helpers' '
 	check fill useless "verbatim foo bar" <<-\EOF
 	protocol=http
@@ -83,6 +119,42 @@ test_expect_success 'credential_fill invokes multiple helpers' '
 	EOF
 '
 
+test_expect_success 'credential_fill response does not get capabilities when helpers are incapable' '
+	check fill useless "verbatim foo bar" <<-\EOF
+	capability[]=authtype
+	protocol=http
+	host=example.com
+	--
+	protocol=http
+	host=example.com
+	username=foo
+	password=bar
+	--
+	useless: get
+	useless: capability[]=authtype
+	useless: protocol=http
+	useless: host=example.com
+	verbatim: get
+	verbatim: capability[]=authtype
+	verbatim: protocol=http
+	verbatim: host=example.com
+	EOF
+'
+
+test_expect_success 'credential_fill response does not get capabilities when caller is incapable' '
+	check fill "verbatim-cred Bearer token" <<-\EOF
+	protocol=http
+	host=example.com
+	--
+	protocol=http
+	host=example.com
+	--
+	verbatim-cred: get
+	verbatim-cred: protocol=http
+	verbatim-cred: host=example.com
+	EOF
+'
+
 test_expect_success 'credential_fill stops when we get a full response' '
 	check fill "verbatim one two" "verbatim three four" <<-\EOF
 	protocol=http
@@ -99,6 +171,25 @@ test_expect_success 'credential_fill stops when we get a full response' '
 	EOF
 '
 
+test_expect_success 'credential_fill thinks a credential is a full response' '
+	check fill "verbatim-cred Bearer token" "verbatim three four" <<-\EOF
+	capability[]=authtype
+	protocol=http
+	host=example.com
+	--
+	capability[]=authtype
+	authtype=Bearer
+	credential=token
+	protocol=http
+	host=example.com
+	--
+	verbatim-cred: get
+	verbatim-cred: capability[]=authtype
+	verbatim-cred: protocol=http
+	verbatim-cred: host=example.com
+	EOF
+'
+
 test_expect_success 'credential_fill continues through partial response' '
 	check fill "verbatim one \"\"" "verbatim two three" <<-\EOF
 	protocol=http
@@ -175,6 +266,20 @@ test_expect_success 'credential_fill passes along metadata' '
 	EOF
 '
 
+test_expect_success 'credential_fill produces no credential without capability' '
+	check fill "verbatim-cred Bearer token" <<-\EOF
+	protocol=http
+	host=example.com
+	--
+	protocol=http
+	host=example.com
+	--
+	verbatim-cred: get
+	verbatim-cred: protocol=http
+	verbatim-cred: host=example.com
+	EOF
+'
+
 test_expect_success 'credential_approve calls all helpers' '
 	check approve useless "verbatim one two" <<-\EOF
 	protocol=http

[PATCH 06/13] docs: indicate new credential protocol fields

[brian m. carlson]

Now that we have new fields (authtype and credential), let's document
them for users and credential helper implementers.

Indicate specifically what common values of authtype are and what values
are allowed.  Note that, while common, digest and NTLM authentication
are insecure because they require unsalted, uniterated password hashes
to be stored.

Tell users that they can continue to use a username and password even if
the new capability is supported.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/git-credential.txt | 34 +++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index 918a0aa42b..f3ed3a82fa 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -178,6 +178,24 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`authtype`::
+	This indicates that the authentication scheme in question should be used.
+	Common values for HTTP and HTTPS include `basic`, `digest`, and `ntlm`,
+	although the latter two are insecure and should not be used.  If `credential`
+	is used, this may be set to an arbitrary string suitable for the protocol in
+	question (usually HTTP).
++
+This value should not be sent unless the appropriate capability (see below) is
+provided on input.
+
+`credential`::
+	The pre-encoded credential, suitable for the protocol in question (usually
+	HTTP).  If this key is sent, `authtype` is mandatory, and `username` and
+	`password` are not used.
++
+This value should not be sent unless the appropriate capability (see below) is
+provided on input.
+
 `wwwauth[]`::
 
 	When an HTTP response is received by Git that includes one or more
@@ -189,7 +207,21 @@ attribute 'wwwauth[]', where the order of the attributes is the same as
 they appear in the HTTP response. This attribute is 'one-way' from Git
 to pass additional information to credential helpers.
 
-Unrecognised attributes are silently discarded.
+`capability[]`::
+	This signals that the caller supports the capability in question.
+	This can be used to provide better, more specific data as part of the
+	protocol.
++
+The only capability currently supported is `authtype`, which indicates that the
+`authtype` and `credential` values are understood.  It is not obligatory to use
+these values in such a case, but they should not be provided without this
+capability.
++
+Callers of `git credential` and credential helpers should emit the
+capabilities they support unconditionally, and Git will gracefully
+handle passing them on.
+
+Unrecognised attributes and capabilities are silently discarded.
 
 GIT
 ---

[PATCH 07/13] http: add support for authtype and credential

[brian m. carlson]

Now that we have the credential helper code set up to handle arbitrary
authentications schemes, let's add support for this in the HTTP code,
where we really want to use it.  If we're using this new functionality,
don't set a username and password, and instead set a header wherever
we'd normally do so, including for proxy authentication.

Since we can now handle this case, ask the credential helper to enable
the appropriate capabilities.

Finally, if we're using the authtype value, set "Expect: 100-continue".
Any type of authentication that requires multiple rounds (such as NTLM
or Kerberos) requires a 100 Continue (if we're larger than
http.postBuffer) because otherwise we send the pack data before we're
authenticated, the push gets a 401 response, and we can't rewind the
stream.  We don't know for certain what other custom schemes might
require this, the HTTP/1.1 standard has required handling this since
1999, the broken HTTP server for which we disabled this (Google's) is
now fixed and has been for some time, and libcurl has a 1-second
fallback in case the HTTP server is still broken.  In addition, it is
not unreasonable to require compliance with a 25-year old standard to
use new Git features.  For all of these reasons, do so here.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 http.c                      |  48 ++++++++++---
 http.h                      |   3 +
 remote-curl.c               |   4 +-
 t/t5563-simple-http-auth.sh | 133 ++++++++++++++++++++++++++++++++++++
 4 files changed, 176 insertions(+), 12 deletions(-)

diff --git a/http.c b/http.c
index 4f5df6fb14..f98c520924 100644
--- a/http.c
+++ b/http.c
@@ -561,18 +561,34 @@ static int curl_empty_auth_enabled(void)
 	return 0;
 }
 
+struct curl_slist *http_append_auth_header(const struct credential *c,
+					   struct curl_slist *headers)
+{
+	if (c->authtype && c->credential) {
+		struct strbuf auth = STRBUF_INIT;
+		strbuf_addf(&auth, "Authorization: %s %s",
+			    c->authtype, c->credential);
+		headers = curl_slist_append(headers, auth.buf);
+		strbuf_release(&auth);
+	}
+	return headers;
+}
+
 static void init_curl_http_auth(CURL *result)
 {
-	if (!http_auth.username || !*http_auth.username) {
+	if ((!http_auth.username || !*http_auth.username) &&
+	    (!http_auth.credential || !*http_auth.credential)) {
 		if (curl_empty_auth_enabled())
 			curl_easy_setopt(result, CURLOPT_USERPWD, ":");
 		return;
 	}
 
-	credential_fill(&http_auth, 0);
+	credential_fill(&http_auth, 1);
 
-	curl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);
-	curl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);
+	if (http_auth.password) {
+		curl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);
+		curl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);
+	}
 }
 
 /* *var must be free-able */
@@ -586,17 +602,22 @@ static void var_override(const char **var, char *value)
 
 static void set_proxyauth_name_password(CURL *result)
 {
+	if (proxy_auth.password) {
 		curl_easy_setopt(result, CURLOPT_PROXYUSERNAME,
 			proxy_auth.username);
 		curl_easy_setopt(result, CURLOPT_PROXYPASSWORD,
 			proxy_auth.password);
+	} else if (proxy_auth.authtype && proxy_auth.credential) {
+		curl_easy_setopt(result, CURLOPT_PROXYHEADER,
+				 http_append_auth_header(&proxy_auth, NULL));
+	}
 }
 
 static void init_curl_proxy_auth(CURL *result)
 {
 	if (proxy_auth.username) {
-		if (!proxy_auth.password)
-			credential_fill(&proxy_auth, 0);
+		if (!proxy_auth.password && !proxy_auth.credential)
+			credential_fill(&proxy_auth, 1);
 		set_proxyauth_name_password(result);
 	}
 
@@ -1469,7 +1490,7 @@ struct active_request_slot *get_active_slot(void)
 
 	curl_easy_setopt(slot->curl, CURLOPT_IPRESOLVE, git_curl_ipresolve);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);
-	if (http_auth.password || curl_empty_auth_enabled())
+	if (http_auth.password || http_auth.credential || curl_empty_auth_enabled())
 		init_curl_http_auth(slot->curl);
 
 	return slot;
@@ -1758,7 +1779,8 @@ static int handle_curl_result(struct slot_results *results)
 	} else if (missing_target(results))
 		return HTTP_MISSING_TARGET;
 	else if (results->http_code == 401) {
-		if (http_auth.username && http_auth.password) {
+		if ((http_auth.username && http_auth.password) ||\
+		    (http_auth.authtype && http_auth.credential)) {
 			credential_reject(&http_auth);
 			return HTTP_NOAUTH;
 		} else {
@@ -2066,11 +2088,15 @@ static int http_request(const char *url,
 	/* Add additional headers here */
 	if (options && options->extra_headers) {
 		const struct string_list_item *item;
-		for_each_string_list_item(item, options->extra_headers) {
-			headers = curl_slist_append(headers, item->string);
+		if (options && options->extra_headers) {
+			for_each_string_list_item(item, options->extra_headers) {
+				headers = curl_slist_append(headers, item->string);
+			}
 		}
 	}
 
+	headers = http_append_auth_header(&http_auth, headers);
+
 	curl_easy_setopt(slot->curl, CURLOPT_URL, url);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
 	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "");
@@ -2191,7 +2217,7 @@ static int http_request_reauth(const char *url,
 		BUG("Unknown http_request target");
 	}
 
-	credential_fill(&http_auth, 0);
+	credential_fill(&http_auth, 1);
 
 	return http_request(url, result, target, options);
 }
diff --git a/http.h b/http.h
index c5f8cc4620..a516ca4a9a 100644
--- a/http.h
+++ b/http.h
@@ -175,6 +175,9 @@ int http_get_file(const char *url, const char *filename,
 
 int http_fetch_ref(const char *base, struct ref *ref);
 
+struct curl_slist *http_append_auth_header(const struct credential *c,
+					   struct curl_slist *headers);
+
 /* Helpers for fetching packs */
 int http_get_info_packs(const char *base_url,
 			struct packed_git **packs_head);
diff --git a/remote-curl.c b/remote-curl.c
index f96bda2431..1c5416812a 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -931,7 +931,7 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 		if (err != HTTP_OK)
 			return -1;
 
-		if (results.auth_avail & CURLAUTH_GSSNEGOTIATE)
+		if (results.auth_avail & CURLAUTH_GSSNEGOTIATE || http_auth.authtype)
 			needs_100_continue = 1;
 	}
 
@@ -942,6 +942,8 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 	headers = curl_slist_append(headers, needs_100_continue ?
 		"Expect: 100-continue" : "Expect:");
 
+	headers = http_append_auth_header(&http_auth, headers);
+
 	/* Add Accept-Language header */
 	if (rpc->hdr_accept_language)
 		headers = curl_slist_append(headers, rpc->hdr_accept_language);
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index ab8a721ccc..b3ed0d9fc2 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -74,6 +74,7 @@ test_expect_success 'access using basic auth' '
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=Basic realm="example.com"
@@ -87,6 +88,43 @@ test_expect_success 'access using basic auth' '
 	EOF
 '
 
+test_expect_success 'access using basic auth via authtype' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	capability[]=authtype
+	authtype=Basic
+	credential=YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	GIT_CURL_VERBOSE=1 git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	capability[]=authtype
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	capability[]=authtype
+	authtype=Basic
+	credential=YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	protocol=http
+	host=$HTTPD_DEST
+	EOF
+'
+
 test_expect_success 'access using basic auth invalid credentials' '
 	test_when_finished "per_test_cleanup" &&
 
@@ -108,6 +146,7 @@ test_expect_success 'access using basic auth invalid credentials' '
 	test_must_fail git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=Basic realm="example.com"
@@ -145,6 +184,7 @@ test_expect_success 'access using basic auth with extra challenges' '
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -183,6 +223,7 @@ test_expect_success 'access using basic auth mixed-case wwwauth header name' '
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=foobar param1="value1" param2="value2"
@@ -226,6 +267,7 @@ test_expect_success 'access using basic auth with wwwauth header continuations'
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -271,6 +313,7 @@ test_expect_success 'access using basic auth with wwwauth header empty continuat
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -312,6 +355,7 @@ test_expect_success 'access using basic auth with wwwauth header mixed line-endi
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -326,4 +370,93 @@ test_expect_success 'access using basic auth with wwwauth header mixed line-endi
 	EOF
 '
 
+test_expect_success 'access using bearer auth' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	capability[]=authtype
+	authtype=Bearer
+	credential=YS1naXQtdG9rZW4=
+	EOF
+
+	# Basic base64(a-git-token)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Bearer YS1naXQtdG9rZW4=
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: FooBar param1="value1" param2="value2"
+	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	capability[]=authtype
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	capability[]=authtype
+	authtype=Bearer
+	credential=YS1naXQtdG9rZW4=
+	protocol=http
+	host=$HTTPD_DEST
+	EOF
+'
+
+test_expect_success 'access using bearer auth with invalid credentials' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	capability[]=authtype
+	authtype=Bearer
+	credential=incorrect-token
+	EOF
+
+	# Basic base64(a-git-token)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Bearer YS1naXQtdG9rZW4=
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: FooBar param1="value1" param2="value2"
+	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	test_must_fail git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	capability[]=authtype
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query erase <<-EOF
+	capability[]=authtype
+	authtype=Bearer
+	credential=incorrect-token
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+'
+
 test_done

[PATCH 08/13] credential: add an argument to keep state

[brian m. carlson]

Until now, our credential code has mostly deal with usernames and
passwords and we've let libcurl deal with the variant of authentication
to be used.  However, now that we have the credential value, the
credential helper can take control of the authentication, so the value
provided might be something that's generated, such as a Digest hash
value.

In such a case, it would be helpful for a credential helper that gets an
erase or store command to be able to keep track of an identifier for the
original secret that went into the computation.  Furthermore, some types
of authentication, such as NTLM and Kerberos, actually need two round
trips to authenticate, which will require that the credential helper
keep some state.

In order to allow for these use cases and others, allow storing state in
a field called "state[]".  This value is passed back to the credential
helper that created it, which avoids confusion caused by parsing values
from different helpers.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/git-credential.txt | 29 ++++++++++++++++++-----------
 credential.c                     | 20 +++++++++++++++++---
 credential.h                     |  7 +++++++
 t/t0300-credentials.sh           | 29 +++++++++++++++++++++++++++++
 4 files changed, 71 insertions(+), 14 deletions(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index f3ed3a82fa..ef30c89c00 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -196,6 +196,15 @@ provided on input.
 This value should not be sent unless the appropriate capability (see below) is
 provided on input.
 
+`state[]`::
+	This value provides an opaque state that will be passed back to this helper
+	if it is called again.  Each different credential helper may specify this
+	once.  The value should include a prefix unique to the credential helper and
+	should ignore values that don't match its prefix.
++
+This value should not be sent unless the appropriate capability (see below) is
+provided on input.
+
 `wwwauth[]`::
 
 	When an HTTP response is received by Git that includes one or more
@@ -208,18 +217,16 @@ they appear in the HTTP response. This attribute is 'one-way' from Git
 to pass additional information to credential helpers.
 
 `capability[]`::
-	This signals that the caller supports the capability in question.
-	This can be used to provide better, more specific data as part of the
-	protocol.
+  This signals that Git, or the helper, as appropriate, supports the
+	capability in question.  This can be used to provide better, more specific
+	data as part of the protocol.
 +
-The only capability currently supported is `authtype`, which indicates that the
-`authtype` and `credential` values are understood.  It is not obligatory to use
-these values in such a case, but they should not be provided without this
-capability.
-+
-Callers of `git credential` and credential helpers should emit the
-capabilities they support unconditionally, and Git will gracefully
-handle passing them on.
+There are two currently supported capabilities.  The first is `authtype`, which
+indicates that the `authtype` and `credential` values are understood.  The
+second is `state`, which indicates that the `state[]` value is understood.
+
+It is not obligatory to use the additional features just because the capability
+is supported, but they should not be provided without this capability.
 
 Unrecognised attributes and capabilities are silently discarded.
 
diff --git a/credential.c b/credential.c
index f2a26b8672..0cd7dd2a00 100644
--- a/credential.c
+++ b/credential.c
@@ -30,6 +30,7 @@ void credential_clear(struct credential *c)
 	free(c->authtype);
 	string_list_clear(&c->helpers, 0);
 	strvec_clear(&c->wwwauth_headers);
+	strvec_clear(&c->state_headers);
 
 	credential_init(c);
 }
@@ -286,8 +287,13 @@ int credential_read(struct credential *c, FILE *fp, int op_type)
 			c->path = xstrdup(value);
 		} else if (!strcmp(key, "wwwauth[]")) {
 			strvec_push(&c->wwwauth_headers, value);
-		} else if (!strcmp(key, "capability[]") && !strcmp(value, "authtype")) {
-			credential_set_capability(&c->capa_authtype, op_type);
+		} else if (!strcmp(key, "state[]")) {
+			strvec_push(&c->state_headers, value);
+		} else if (!strcmp(key, "capability[]")) {
+			if (!strcmp(value, "authtype"))
+				credential_set_capability(&c->capa_authtype, op_type);
+			else if (!strcmp(value, "state"))
+				credential_set_capability(&c->capa_state, op_type);
 		} else if (!strcmp(key, "password_expiry_utc")) {
 			errno = 0;
 			c->password_expiry_utc = parse_timestamp(value, NULL, 10);
@@ -329,8 +335,12 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
 
 void credential_write(const struct credential *c, FILE *fp, int op_type)
 {
-	if (credential_has_capability(&c->capa_authtype, op_type)) {
+	if (credential_has_capability(&c->capa_authtype, op_type))
 		credential_write_item(fp, "capability[]", "authtype", 0);
+	if (credential_has_capability(&c->capa_state, op_type))
+		credential_write_item(fp, "capability[]", "state", 0);
+
+	if (credential_has_capability(&c->capa_authtype, op_type)) {
 		credential_write_item(fp, "authtype", c->authtype, 0);
 		credential_write_item(fp, "credential", c->credential, 0);
 	}
@@ -347,6 +357,10 @@ void credential_write(const struct credential *c, FILE *fp, int op_type)
 	}
 	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
 		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
+	if (credential_has_capability(&c->capa_state, op_type)) {
+		for (size_t i = 0; i < c->state_headers.nr; i++)
+			credential_write_item(fp, "state[]", c->state_headers.v[i], 0);
+	}
 }
 
 static int run_credential_helper(struct credential *c,
diff --git a/credential.h b/credential.h
index 2051d04c5a..e2021455fe 100644
--- a/credential.h
+++ b/credential.h
@@ -142,6 +142,11 @@ struct credential {
 	 */
 	struct strvec wwwauth_headers;
 
+	/**
+	 * A `strvec` of state headers from credential helpers.
+	 */
+	struct strvec state_headers;
+
 	/**
 	 * Internal use only. Keeps track of if we previously matched against a
 	 * WWW-Authenticate header line in order to re-fold future continuation
@@ -156,6 +161,7 @@ struct credential {
 		 username_from_proto:1;
 
 	struct credential_capability capa_authtype;
+	struct credential_capability capa_state;
 
 	char *username;
 	char *password;
@@ -177,6 +183,7 @@ struct credential {
 	.helpers = STRING_LIST_INIT_DUP, \
 	.password_expiry_utc = TIME_MAX, \
 	.wwwauth_headers = STRVEC_INIT, \
+	.state_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
index 8477108b28..aa56e0dc84 100755
--- a/t/t0300-credentials.sh
+++ b/t/t0300-credentials.sh
@@ -46,9 +46,12 @@ test_expect_success 'setup helper scripts' '
 	credential=$1; shift
 	. ./dump
 	echo capability[]=authtype
+	echo capability[]=state
 	test -z "${capability##*authtype*}" || return
 	test -z "$authtype" || echo authtype=$authtype
 	test -z "$credential" || echo credential=$credential
+	test -z "${capability##*state*}" || return
+	echo state[]=verbatim-cred:foo
 	EOF
 
 	write_script git-credential-verbatim-with-expiry <<-\EOF &&
@@ -99,6 +102,29 @@ test_expect_success 'credential_fill invokes helper with credential' '
 	EOF
 '
 
+test_expect_success 'credential_fill invokes helper with credential and state' '
+	check fill "verbatim-cred Bearer token" <<-\EOF
+	capability[]=authtype
+	capability[]=state
+	protocol=http
+	host=example.com
+	--
+	capability[]=authtype
+	capability[]=state
+	authtype=Bearer
+	credential=token
+	protocol=http
+	host=example.com
+	state[]=verbatim-cred:foo
+	--
+	verbatim-cred: get
+	verbatim-cred: capability[]=authtype
+	verbatim-cred: capability[]=state
+	verbatim-cred: protocol=http
+	verbatim-cred: host=example.com
+	EOF
+'
+
 
 test_expect_success 'credential_fill invokes multiple helpers' '
 	check fill useless "verbatim foo bar" <<-\EOF
@@ -122,6 +148,7 @@ test_expect_success 'credential_fill invokes multiple helpers' '
 test_expect_success 'credential_fill response does not get capabilities when helpers are incapable' '
 	check fill useless "verbatim foo bar" <<-\EOF
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=example.com
 	--
@@ -132,10 +159,12 @@ test_expect_success 'credential_fill response does not get capabilities when hel
 	--
 	useless: get
 	useless: capability[]=authtype
+	useless: capability[]=state
 	useless: protocol=http
 	useless: host=example.com
 	verbatim: get
 	verbatim: capability[]=authtype
+	verbatim: capability[]=state
 	verbatim: protocol=http
 	verbatim: host=example.com
 	EOF

[PATCH 09/13] credential: enable state capability

[brian m. carlson]

Now that we've implemented the state capability, let's send it along by
default when filling credentials so we can make use of it.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 credential.c                |  1 +
 t/t5563-simple-http-auth.sh | 10 ++++++++++
 2 files changed, 11 insertions(+)

diff --git a/credential.c b/credential.c
index 0cd7dd2a00..0ca7c12895 100644
--- a/credential.c
+++ b/credential.c
@@ -38,6 +38,7 @@ void credential_clear(struct credential *c)
 static void credential_set_all_capabilities(struct credential *c)
 {
 	c->capa_authtype.request_initial = 1;
+	c->capa_state.request_initial = 1;
 }
 
 int credential_match(const struct credential *want,
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index b3ed0d9fc2..b098cd0fdf 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -75,6 +75,7 @@ test_expect_success 'access using basic auth' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=Basic realm="example.com"
@@ -111,6 +112,7 @@ test_expect_success 'access using basic auth via authtype' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=Basic realm="example.com"
@@ -147,6 +149,7 @@ test_expect_success 'access using basic auth invalid credentials' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=Basic realm="example.com"
@@ -185,6 +188,7 @@ test_expect_success 'access using basic auth with extra challenges' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -224,6 +228,7 @@ test_expect_success 'access using basic auth mixed-case wwwauth header name' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=foobar param1="value1" param2="value2"
@@ -268,6 +273,7 @@ test_expect_success 'access using basic auth with wwwauth header continuations'
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -314,6 +320,7 @@ test_expect_success 'access using basic auth with wwwauth header empty continuat
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -356,6 +363,7 @@ test_expect_success 'access using basic auth with wwwauth header mixed line-endi
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -397,6 +405,7 @@ test_expect_success 'access using bearer auth' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -440,6 +449,7 @@ test_expect_success 'access using bearer auth with invalid credentials' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"

[PATCH 10/13] docs: set a limit on credential line length

[brian m. carlson]

We recently introduced a way for credential helpers to add arbitrary
state as part of the protocol.  Set some limits on line length to avoid
helpers passing extremely large amounts of data.  While Git doesn't have
a fixed parsing length, there are other tools which support this
protocol and it's kind to allow them to use a reasonable fixed-size
buffer for parsing.  In addition, we would like to be moderate in our
memory usage and imposing reasonable limits is helpful for that purpose.

In the event a credential helper is incapable of storing its serialized
state in 64 KiB, it can feel free to serialize it on disk and store a
reference instead.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/git-credential.txt | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index ef30c89c00..6b7e017066 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -111,7 +111,9 @@ attribute per line. Each attribute is specified by a key-value pair,
 separated by an `=` (equals) sign, followed by a newline.
 
 The key may contain any bytes except `=`, newline, or NUL. The value may
-contain any bytes except newline or NUL.
+contain any bytes except newline or NUL.  A line, including the trailing
+newline, may not exceed 65535 bytes in order to allow implementations to
+parse efficiently.
 
 Attributes with keys that end with C-style array brackets `[]` can have
 multiple values. Each instance of a multi-valued attribute forms an

[PATCH 11/13] t5563: refactor for multi-stage authentication

[brian m. carlson]

Some HTTP authentication schemes, such as NTLM- and Kerberos-based
options, require more than one round trip to authenticate.  Currently,
these can only be supported in libcurl, since Git does not have support
for this in the credential helper protocol.

However, in a future commit, we'll add support for this functionality
into the credential helper protocol and Git itself. Because we don't
really want to implement either NTLM or Kerberos, both of which are
complex protocols, we'll want to test this using a fake credential
authentication scheme.  In order to do so, update t5563 and its backend
to allow us to accept multiple sets of credentials and respond with
different behavior in each case.

Since we can now provide any number of possible status codes, provide a
non-specific reason phrase so we don't have to generate a more specific
one based on the response.  The reason phrase is mandatory according to
the status-line production in RFC 7230, but clients SHOULD ignore it,
and curl does (except to print it).

Each entry in the authorization and challenge fields contains an ID,
which indicates a corresponding credential and response.  If the
response is a 200 status, then we continue to execute git-http-backend.
Otherwise, we print the corresponding status and response.  If no ID is
matched, we use the default response with a status of 401.

Note that there is an implicit order to the parameters.  The ID is
always first and the creds or response value is always last, and
therefore may contain spaces, equals signs, or other arbitrary data.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/lib-httpd/nph-custom-auth.sh | 17 ++++--
 t/t5563-simple-http-auth.sh    | 96 +++++++++++++++++++---------------
 2 files changed, 66 insertions(+), 47 deletions(-)

diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh
index f5345e775e..d408d2caad 100644
--- a/t/lib-httpd/nph-custom-auth.sh
+++ b/t/lib-httpd/nph-custom-auth.sh
@@ -19,21 +19,30 @@ CHALLENGE_FILE=custom-auth.challenge
 #
 
 if test -n "$HTTP_AUTHORIZATION" && \
-	grep -Fqsx "${HTTP_AUTHORIZATION}" "$VALID_CREDS_FILE"
+	grep -Fqs "creds=${HTTP_AUTHORIZATION}" "$VALID_CREDS_FILE"
 then
+	idno=$(grep -F "creds=${HTTP_AUTHORIZATION}" "$VALID_CREDS_FILE" | sed -e 's/^id=\([a-z0-9-][a-z0-9-]*\) .*$/\1/')
+	status=$(sed -ne "s/^id=$idno.*status=\\([0-9][0-9][0-9]\\).*\$/\\1/p" "$CHALLENGE_FILE" | head -n1)
 	# Note that although git-http-backend returns a status line, it
 	# does so using a CGI 'Status' header. Because this script is an
 	# No Parsed Headers (NPH) script, we must return a real HTTP
 	# status line.
 	# This is only a test script, so we don't bother to check for
 	# the actual status from git-http-backend and always return 200.
-	echo 'HTTP/1.1 200 OK'
-	exec "$GIT_EXEC_PATH"/git-http-backend
+	echo "HTTP/1.1 $status Nonspecific Reason Phrase"
+	if test "$status" -eq 200
+	then
+		exec "$GIT_EXEC_PATH"/git-http-backend
+	else
+		sed -ne "s/^id=$idno.*response=//p" "$CHALLENGE_FILE"
+		echo
+		exit
+	fi
 fi
 
 echo 'HTTP/1.1 401 Authorization Required'
 if test -f "$CHALLENGE_FILE"
 then
-	cat "$CHALLENGE_FILE"
+	sed -ne 's/^id=default.*response=//p' "$CHALLENGE_FILE"
 fi
 echo
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index b098cd0fdf..515185ae00 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -63,11 +63,12 @@ test_expect_success 'access using basic auth' '
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -100,11 +101,12 @@ test_expect_success 'access using basic auth via authtype' '
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -137,11 +139,12 @@ test_expect_success 'access using basic auth invalid credentials' '
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -174,13 +177,14 @@ test_expect_success 'access using basic auth with extra challenges' '
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: FooBar param1="value1" param2="value2"
-	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: FooBar param1="value1" param2="value2"
+	id=default response=WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -214,13 +218,14 @@ test_expect_success 'access using basic auth mixed-case wwwauth header name' '
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	www-authenticate: foobar param1="value1" param2="value2"
-	WWW-AUTHENTICATE: BEARER authorize_uri="id.example.com" p=1 q=0
-	WwW-aUtHeNtIcAtE: baSiC realm="example.com"
+	id=1 status=200
+	id=default response=www-authenticate: foobar param1="value1" param2="value2"
+	id=default response=WWW-AUTHENTICATE: BEARER authorize_uri="id.example.com" p=1 q=0
+	id=default response=WwW-aUtHeNtIcAtE: baSiC realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -254,18 +259,19 @@ test_expect_success 'access using basic auth with wwwauth header continuations'
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	# Note that leading and trailing whitespace is important to correctly
 	# simulate a continuation/folded header.
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: FooBar param1="value1"
-	 param2="value2"
-	WWW-Authenticate: Bearer authorize_uri="id.example.com"
-	 p=1
-	 q=0
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: FooBar param1="value1"
+	id=default response= param2="value2"
+	id=default response=WWW-Authenticate: Bearer authorize_uri="id.example.com"
+	id=default response= p=1
+	id=default response= q=0
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -299,21 +305,22 @@ test_expect_success 'access using basic auth with wwwauth header empty continuat
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
 
 	# Note that leading and trailing whitespace is important to correctly
 	# simulate a continuation/folded header.
-	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >"$CHALLENGE" &&
-	printf " \r\n" >>"$CHALLENGE" &&
-	printf " param2=\"value2\"\r\n" >>"$CHALLENGE" &&
-	printf "WWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\r\n" >>"$CHALLENGE" &&
-	printf " p=1\r\n" >>"$CHALLENGE" &&
-	printf " \r\n" >>"$CHALLENGE" &&
-	printf " q=0\r\n" >>"$CHALLENGE" &&
-	printf "WWW-Authenticate: Basic realm=\"example.com\"\r\n" >>"$CHALLENGE" &&
+	printf "id=1 status=200\n" >"$CHALLENGE" &&
+	printf "id=default response=WWW-Authenticate: FooBar param1=\"value1\"\r\n" >>"$CHALLENGE" &&
+	printf "id=default response= \r\n" >>"$CHALLENGE" &&
+	printf "id=default response= param2=\"value2\"\r\n" >>"$CHALLENGE" &&
+	printf "id=default response=WWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\r\n" >>"$CHALLENGE" &&
+	printf "id=default response= p=1\r\n" >>"$CHALLENGE" &&
+	printf "id=default response= \r\n" >>"$CHALLENGE" &&
+	printf "id=default response= q=0\r\n" >>"$CHALLENGE" &&
+	printf "id=default response=WWW-Authenticate: Basic realm=\"example.com\"\r\n" >>"$CHALLENGE" &&
 
 	test_config_global credential.helper test-helper &&
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
@@ -346,17 +353,18 @@ test_expect_success 'access using basic auth with wwwauth header mixed line-endi
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
 
 	# Note that leading and trailing whitespace is important to correctly
 	# simulate a continuation/folded header.
-	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >"$CHALLENGE" &&
-	printf " \r\n" >>"$CHALLENGE" &&
-	printf "\tparam2=\"value2\"\r\n" >>"$CHALLENGE" &&
-	printf "WWW-Authenticate: Basic realm=\"example.com\"" >>"$CHALLENGE" &&
+	printf "id=1 status=200\n" >"$CHALLENGE" &&
+	printf "id=default response=WWW-Authenticate: FooBar param1=\"value1\"\r\n" >>"$CHALLENGE" &&
+	printf "id=default response= \r\n" >>"$CHALLENGE" &&
+	printf "id=default response=\tparam2=\"value2\"\r\n" >>"$CHALLENGE" &&
+	printf "id=default response=WWW-Authenticate: Basic realm=\"example.com\"" >>"$CHALLENGE" &&
 
 	test_config_global credential.helper test-helper &&
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
@@ -389,15 +397,16 @@ test_expect_success 'access using bearer auth' '
 
 	# Basic base64(a-git-token)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Bearer YS1naXQtdG9rZW4=
+	id=1 creds=Bearer YS1naXQtdG9rZW4=
 	EOF
 
 	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: FooBar param1="value1" param2="value2"
-	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: FooBar param1="value1" param2="value2"
+	id=default response=WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -433,15 +442,16 @@ test_expect_success 'access using bearer auth with invalid credentials' '
 
 	# Basic base64(a-git-token)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Bearer YS1naXQtdG9rZW4=
+	id=1 creds=Bearer YS1naXQtdG9rZW4=
 	EOF
 
 	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: FooBar param1="value1" param2="value2"
-	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: FooBar param1="value1" param2="value2"
+	id=default response=WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&

[PATCH 12/13] strvec: implement swapping two strvecs

[brian m. carlson]

In a future commit, we'll want the ability to efficiently swap the
contents of two strvec instances without needing to copy any data.
Since a strvec is simply a pointer and two sizes, swapping them is as
simply as copying the two pointers and sizes, so let's do that.

We use a temporary here simply because C doesn't provide a standard
swapping function, unlike C++ and Rust, but a good optimizing compiler
will recognize this syntax and handle it appropriately using an
optimization pass.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 strvec.c | 7 +++++++
 strvec.h | 5 +++++
 2 files changed, 12 insertions(+)

diff --git a/strvec.c b/strvec.c
index 178f4f3748..93006f1e63 100644
--- a/strvec.c
+++ b/strvec.c
@@ -106,3 +106,10 @@ const char **strvec_detach(struct strvec *array)
 		return ret;
 	}
 }
+
+void strvec_swap(struct strvec *a, struct strvec *b)
+{
+	struct strvec t = *a;
+	*a = *b;
+	*b = t;
+}
diff --git a/strvec.h b/strvec.h
index 4715d3e51f..5b762532bb 100644
--- a/strvec.h
+++ b/strvec.h
@@ -88,4 +88,9 @@ void strvec_clear(struct strvec *);
  */
 const char **strvec_detach(struct strvec *);
 
+/**
+ * Swap the contents of two `strvec` structs without allocating.
+ */
+void strvec_swap(struct strvec *, struct strvec *);
+
 #endif /* STRVEC_H */

[PATCH 13/13] credential: add support for multistage credential rounds

[brian m. carlson]

Over HTTP, NTLM and Kerberos require two rounds of authentication on the
client side.  It's possible that there are custom authentication schemes
that also implement this same approach.  Since these are tricky schemes
to implement and the HTTP library in use may not always handle them
gracefully on all systems, it would be helpful to allow the credential
helper to implement them instead for increased portability and
robustness.

To allow this to happen, add a boolean flag, continue, that indicates
that instead of failing when we get a 401, we should retry another round
of authentication.  However, this necessitates some changes in our
current credential code so that we can make this work.

Keep the state[] headers between iterations, but only use them to send
to the helper and only consider the new ones we read from the credential
helper to be valid on subsequent iterations.  That avoids us passing
stale data when we finally approve or reject the credential.  Similarly,
clear the multistage and wwwauth[] values appropriately so that we
don't pass stale data or think we're trying a multiround response when
we're not.  Remove the credential values so that we can actually fill a
second time with new responses.

Limit the number of iterations of reauthentication we do to 3.  This
means that if there's a problem, we'll terminate with an error message
instead of retrying indefinitely and not informing the user (and
possibly conducting a DoS on the server).

In our tests, handle creating multiple response output files from our
helper so we can verify that each of the messages sent is correct.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/git-credential.txt | 16 +++++-
 builtin/credential.c             |  1 +
 credential.c                     | 32 ++++++++++--
 credential.h                     | 27 +++++++++-
 http.c                           | 59 +++++++++++----------
 t/t5563-simple-http-auth.sh      | 89 ++++++++++++++++++++++++++++++--
 6 files changed, 187 insertions(+), 37 deletions(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index 6b7e017066..160dee5c6a 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -207,6 +207,19 @@ provided on input.
 This value should not be sent unless the appropriate capability (see below) is
 provided on input.
 
+`continue`::
+	This is a boolean value, which, if enabled, indicates that this
+	authentication is a non-final part of a multistage authentication step. This
+	is common in protocols such as NTLM and Kerberos, where two rounds of client
+	authentication are required, and setting this flag allows the credential
+	helper to implement the multistage authentication step.  This flag should
+	only be sent if a further stage is required; that is, if another round of
+	authentication is expected.
++
+This value should not be sent unless the appropriate capability (see below) is
+provided on input.  This attribute is 'one-way' from a credential helper to
+pass information to Git (or other programs invoking `git credential`).
+
 `wwwauth[]`::
 
 	When an HTTP response is received by Git that includes one or more
@@ -225,7 +238,8 @@ to pass additional information to credential helpers.
 +
 There are two currently supported capabilities.  The first is `authtype`, which
 indicates that the `authtype` and `credential` values are understood.  The
-second is `state`, which indicates that the `state[]` value is understood.
+second is `state`, which indicates that the `state[]` and `continue` values are
+understood.
 
 It is not obligatory to use the additional features just because the capability
 is supported, but they should not be provided without this capability.
diff --git a/builtin/credential.c b/builtin/credential.c
index 5123dabcf1..f14d1b5ed6 100644
--- a/builtin/credential.c
+++ b/builtin/credential.c
@@ -22,6 +22,7 @@ int cmd_credential(int argc, const char **argv, const char *prefix UNUSED)
 
 	if (!strcmp(op, "fill")) {
 		credential_fill(&c, 0);
+		credential_next_state(&c);
 		credential_write(&c, stdout, CREDENTIAL_OP_RESPONSE);
 	} else if (!strcmp(op, "approve")) {
 		credential_approve(&c);
diff --git a/credential.c b/credential.c
index 0ca7c12895..9a08efe113 100644
--- a/credential.c
+++ b/credential.c
@@ -31,10 +31,23 @@ void credential_clear(struct credential *c)
 	string_list_clear(&c->helpers, 0);
 	strvec_clear(&c->wwwauth_headers);
 	strvec_clear(&c->state_headers);
+	strvec_clear(&c->state_headers_to_send);
 
 	credential_init(c);
 }
 
+void credential_next_state(struct credential *c)
+{
+	strvec_clear(&c->state_headers_to_send);
+	strvec_swap(&c->state_headers, &c->state_headers_to_send);
+}
+
+void credential_clear_secrets(struct credential *c)
+{
+	FREE_AND_NULL(c->password);
+	FREE_AND_NULL(c->credential);
+}
+
 static void credential_set_all_capabilities(struct credential *c)
 {
 	c->capa_authtype.request_initial = 1;
@@ -295,6 +308,8 @@ int credential_read(struct credential *c, FILE *fp, int op_type)
 				credential_set_capability(&c->capa_authtype, op_type);
 			else if (!strcmp(value, "state"))
 				credential_set_capability(&c->capa_state, op_type);
+		} else if (!strcmp(key, "continue")) {
+			c->multistage = !!git_config_bool("continue", value);
 		} else if (!strcmp(key, "password_expiry_utc")) {
 			errno = 0;
 			c->password_expiry_utc = parse_timestamp(value, NULL, 10);
@@ -359,8 +374,10 @@ void credential_write(const struct credential *c, FILE *fp, int op_type)
 	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
 		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
 	if (credential_has_capability(&c->capa_state, op_type)) {
-		for (size_t i = 0; i < c->state_headers.nr; i++)
-			credential_write_item(fp, "state[]", c->state_headers.v[i], 0);
+		if (c->multistage)
+			credential_write_item(fp, "continue", "1", 0);
+		for (size_t i = 0; i < c->state_headers_to_send.nr; i++)
+			credential_write_item(fp, "state[]", c->state_headers_to_send.v[i], 0);
 	}
 }
 
@@ -431,6 +448,9 @@ void credential_fill(struct credential *c, int all_capabilities)
 	if ((c->username && c->password) || c->credential)
 		return;
 
+	credential_next_state(c);
+	c->multistage = 0;
+
 	credential_apply_config(c);
 	if (all_capabilities)
 		credential_set_all_capabilities(c);
@@ -443,8 +463,10 @@ void credential_fill(struct credential *c, int all_capabilities)
 			/* Reset expiry to maintain consistency */
 			c->password_expiry_utc = TIME_MAX;
 		}
-		if ((c->username && c->password) || c->credential)
+		if ((c->username && c->password) || c->credential) {
+			strvec_clear(&c->wwwauth_headers);
 			return;
+		}
 		if (c->quit)
 			die("credential helper '%s' told us to quit",
 			    c->helpers.items[i].string);
@@ -464,6 +486,8 @@ void credential_approve(struct credential *c)
 	if (((!c->username || !c->password) && !c->credential) || c->password_expiry_utc < time(NULL))
 		return;
 
+	credential_next_state(c);
+
 	credential_apply_config(c);
 
 	for (i = 0; i < c->helpers.nr; i++)
@@ -475,6 +499,8 @@ void credential_reject(struct credential *c)
 {
 	int i;
 
+	credential_next_state(c);
+
 	credential_apply_config(c);
 
 	for (i = 0; i < c->helpers.nr; i++)
diff --git a/credential.h b/credential.h
index e2021455fe..adb1fc370a 100644
--- a/credential.h
+++ b/credential.h
@@ -143,10 +143,15 @@ struct credential {
 	struct strvec wwwauth_headers;
 
 	/**
-	 * A `strvec` of state headers from credential helpers.
+	 * A `strvec` of state headers received from credential helpers.
 	 */
 	struct strvec state_headers;
 
+	/**
+	 * A `strvec` of state headers to send to credential helpers.
+	 */
+	struct strvec state_headers_to_send;
+
 	/**
 	 * Internal use only. Keeps track of if we previously matched against a
 	 * WWW-Authenticate header line in order to re-fold future continuation
@@ -156,6 +161,7 @@ struct credential {
 
 	unsigned approved:1,
 		 configured:1,
+		 multistage: 1,
 		 quit:1,
 		 use_http_path:1,
 		 username_from_proto:1;
@@ -184,6 +190,7 @@ struct credential {
 	.password_expiry_utc = TIME_MAX, \
 	.wwwauth_headers = STRVEC_INIT, \
 	.state_headers = STRVEC_INIT, \
+	.state_headers_to_send = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
@@ -229,6 +236,24 @@ void credential_approve(struct credential *);
  */
 void credential_reject(struct credential *);
 
+/**
+ * Clear the secrets in this credential, but leave other data intact.
+ *
+ * This is useful for resetting credentials in preparation for a subsequent
+ * stage of filling.
+ */
+void credential_clear_secrets(struct credential *c);
+
+/**
+ * Prepares the credential for the next iteration of the helper protocol by
+ * updating the state headers to send with the ones read by the last iteration
+ * of the protocol.
+ *
+ * Except for internal callers, this should be called exactly once between
+ * reading credentials with `credential_fill` and writing them.
+ */
+void credential_next_state(struct credential *c);
+
 int credential_read(struct credential *, FILE *, int);
 void credential_write(const struct credential *, FILE *, int);
 
diff --git a/http.c b/http.c
index f98c520924..9d373c6460 100644
--- a/http.c
+++ b/http.c
@@ -1781,6 +1781,10 @@ static int handle_curl_result(struct slot_results *results)
 	else if (results->http_code == 401) {
 		if ((http_auth.username && http_auth.password) ||\
 		    (http_auth.authtype && http_auth.credential)) {
+			if (http_auth.multistage) {
+				credential_clear_secrets(&http_auth);
+				return HTTP_REAUTH;
+			}
 			credential_reject(&http_auth);
 			return HTTP_NOAUTH;
 		} else {
@@ -2178,6 +2182,7 @@ static int http_request_reauth(const char *url,
 			       void *result, int target,
 			       struct http_get_options *options)
 {
+	int i = 3;
 	int ret = http_request(url, result, target, options);
 
 	if (ret != HTTP_OK && ret != HTTP_REAUTH)
@@ -2191,35 +2196,35 @@ static int http_request_reauth(const char *url,
 		}
 	}
 
-	if (ret != HTTP_REAUTH)
-		return ret;
+	while (ret == HTTP_REAUTH && --i) {
+		/*
+		 * The previous request may have put cruft into our output stream; we
+		 * should clear it out before making our next request.
+		 */
+		switch (target) {
+		case HTTP_REQUEST_STRBUF:
+			strbuf_reset(result);
+			break;
+		case HTTP_REQUEST_FILE:
+			if (fflush(result)) {
+				error_errno("unable to flush a file");
+				return HTTP_START_FAILED;
+			}
+			rewind(result);
+			if (ftruncate(fileno(result), 0) < 0) {
+				error_errno("unable to truncate a file");
+				return HTTP_START_FAILED;
+			}
+			break;
+		default:
+			BUG("Unknown http_request target");
+		}
 
-	/*
-	 * The previous request may have put cruft into our output stream; we
-	 * should clear it out before making our next request.
-	 */
-	switch (target) {
-	case HTTP_REQUEST_STRBUF:
-		strbuf_reset(result);
-		break;
-	case HTTP_REQUEST_FILE:
-		if (fflush(result)) {
-			error_errno("unable to flush a file");
-			return HTTP_START_FAILED;
-		}
-		rewind(result);
-		if (ftruncate(fileno(result), 0) < 0) {
-			error_errno("unable to truncate a file");
-			return HTTP_START_FAILED;
-		}
-		break;
-	default:
-		BUG("Unknown http_request target");
+		credential_fill(&http_auth, 1);
+
+		ret = http_request(url, result, target, options);
 	}
-
-	credential_fill(&http_auth, 1);
-
-	return http_request(url, result, target, options);
+	return ret;
 }
 
 int http_get_strbuf(const char *url,
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index 515185ae00..5d5caa3f58 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -21,9 +21,17 @@ test_expect_success 'setup_credential_helper' '
 	CREDENTIAL_HELPER="$TRASH_DIRECTORY/bin/git-credential-test-helper" &&
 	write_script "$CREDENTIAL_HELPER" <<-\EOF
 	cmd=$1
-	teefile=$cmd-query.cred
+	teefile=$cmd-query-temp.cred
 	catfile=$cmd-reply.cred
 	sed -n -e "/^$/q" -e "p" >>$teefile
+	state=$(sed -ne "s/^state\[\]=helper://p" "$teefile")
+	if test -z "$state"
+	then
+		mv "$teefile" "$cmd-query.cred"
+	else
+		mv "$teefile" "$cmd-query-$state.cred"
+		catfile="$cmd-reply-$state.cred"
+	fi
 	if test "$cmd" = "get"
 	then
 		cat $catfile
@@ -32,13 +40,15 @@ test_expect_success 'setup_credential_helper' '
 '
 
 set_credential_reply () {
-	cat >"$TRASH_DIRECTORY/$1-reply.cred"
+	local suffix="$(test -n "$2" && echo "-$2")"
+	cat >"$TRASH_DIRECTORY/$1-reply$suffix.cred"
 }
 
 expect_credential_query () {
-	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
-	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
-		 "$TRASH_DIRECTORY/$1-query.cred"
+	local suffix="$(test -n "$2" && echo "-$2")"
+	cat >"$TRASH_DIRECTORY/$1-expect$suffix.cred" &&
+	test_cmp "$TRASH_DIRECTORY/$1-expect$suffix.cred" \
+		 "$TRASH_DIRECTORY/$1-query$suffix.cred"
 }
 
 per_test_cleanup () {
@@ -479,4 +489,73 @@ test_expect_success 'access using bearer auth with invalid credentials' '
 	EOF
 '
 
+test_expect_success 'access using three-legged auth' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	capability[]=authtype
+	capability[]=state
+	authtype=Multistage
+	credential=YS1naXQtdG9rZW4=
+	state[]=helper:foobar
+	continue=1
+	EOF
+
+	set_credential_reply get foobar <<-EOF &&
+	capability[]=authtype
+	capability[]=state
+	authtype=Multistage
+	credential=YW5vdGhlci10b2tlbg==
+	state[]=helper:bazquux
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	id=1 creds=Multistage YS1naXQtdG9rZW4=
+	id=2 creds=Multistage YW5vdGhlci10b2tlbg==
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	id=1 status=401 response=WWW-Authenticate: Multistage challenge="456"
+	id=1 status=401 response=WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	id=2 status=200
+	id=default response=WWW-Authenticate: Multistage challenge="123"
+	id=default response=WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	capability[]=authtype
+	capability[]=state
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=Multistage challenge="123"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	EOF
+
+	expect_credential_query get foobar <<-EOF &&
+	capability[]=authtype
+	capability[]=state
+	authtype=Multistage
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=Multistage challenge="456"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	state[]=helper:foobar
+	EOF
+
+	expect_credential_query store bazquux <<-EOF
+	capability[]=authtype
+	capability[]=state
+	authtype=Multistage
+	credential=YW5vdGhlci10b2tlbg==
+	protocol=http
+	host=$HTTPD_DEST
+	state[]=helper:bazquux
+	EOF
+'
+
 test_done

Re: [PATCH 00/13] Support for arbitrary schemes in credentials

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> ... and because of this difficulty and the fact that NTLM uses cryptography
> known to be insecure since 1995, there is often little interest in
> implementing this support outside of libcurl. However, it would be
> helpful if people who want to use it can still use it.

This position was a bit surprising to me to come from you, but
perhaps I am mixing up my recollection of your past work on this
project with somebody else's?  I somehow expected to hear something
more like "if a less secure thing is cumbersome to implement, let it
be, as that is better for the world".  But I am OK to add less secure
thing as long as it is an opt-in "easy way out".

Everything else I read in the cover letter made sense to me.  I just
wanted to say that the above part was a bit surprising.

Thanks.

Re: [PATCH 00/13] Support for arbitrary schemes in credentials

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2772 bytes --]

On 2024-03-24 at 02:24:46, Junio C Hamano wrote:
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
> > ... and because of this difficulty and the fact that NTLM uses cryptography
> > known to be insecure since 1995, there is often little interest in
> > implementing this support outside of libcurl. However, it would be
> > helpful if people who want to use it can still use it.
> 
> This position was a bit surprising to me to come from you, but
> perhaps I am mixing up my recollection of your past work on this
> project with somebody else's?  I somehow expected to hear something
> more like "if a less secure thing is cumbersome to implement, let it
> be, as that is better for the world".  But I am OK to add less secure
> thing as long as it is an opt-in "easy way out".
> 
> Everything else I read in the cover letter made sense to me.  I just
> wanted to say that the above part was a bit surprising.

I do firmly feel that NTLM should go gentle into that good night.

However, we've seen a lot of user questions on Git LFS from users who
want to use NTLM or Digest, which we don't support and probably will not
for security and maintainability reasons, but which their corporate
environment imprudently requires. Thus, my goal is to make it _possible_
for people to implement this, but it to make it their responsibility (or
that of a suitable open source project) to do so instead of asking me to
maintain it.  That, I think, is a fair and equitable tradeoff for this
situation.

Also, right now, Windows users sometimes choose to use the older v2.13.3
version of Git LFS which, due to a Go standard library bug, has an
arbitrary code execution vulnerability, but which did support NTLM.
Thus, it would also be better for security to have people on a suitably
patched version of Git LFS with an external NTLM helper.

This series would also, in an approach that's better for security, allow
people to use better Kerberos mechanisms than what Windows supports
natively, or to use an AWS HMAC-v4-style request signing (using
HMAC-SHA-256) if they want to do so, both of which would be a win for
security.

I could have put all of this into the cover letter, but I felt it was
pretty long and didn't want to sell this as an advantage only for Git
LFS when I think it provides general benefit for Git users.  I know
the policy of the project is not to prioritize any one external
project, and I try to be sensitive to that here.

I don't think this constitutes a marked change in my historical
"let's-remove-all-the-obsolete-cryptography-and-obsolete-operating-systems"
approach, but I see how it might look that way at first glance.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 00/13] Support for arbitrary schemes in credentials

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

>> be, as that is better for the world".  But I am OK to add less secure
>> thing as long as it is an opt-in "easy way out".
> 
> ... Thus, my goal is to make it _possible_
> for people to implement this, but it to make it their responsibility (or
> that of a suitable open source project) to do so instead of asking me to
> maintain it.  That, I think, is a fair and equitable tradeoff for this
> situation.

Yup, I think we are in agreement.  I have no problem with giving
them a long rope ;-)

Re: [PATCH 06/13] docs: indicate new credential protocol fields

[M Hickford]

> Tell users that they can continue to use a username and password even if
> the new capability is supported.
> 
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  Documentation/git-credential.txt | 34 +++++++++++++++++++++++++++++++-
>  1 file changed, 33 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
> index 918a0aa42b..f3ed3a82fa 100644
> --- a/Documentation/git-credential.txt
> +++ b/Documentation/git-credential.txt
> @@ -178,6 +178,24 @@ empty string.
>  Components which are missing from the URL (e.g., there is no
>  username in the example above) will be left unset.
>  
> +`authtype`::
> +	This indicates that the authentication scheme in question should be used.
> +	Common values for HTTP and HTTPS include `basic`, `digest`, and `ntlm`,
> +	although the latter two are insecure and should not be used.  If `credential`
> +	is used, this may be set to an arbitrary string suitable for the protocol in
> +	question (usually HTTP).

How about adding 'bearer' to this list? Popular hosts Bitbucket https://bitbucket.org and Gitea/Forgejo (such as https://codeberg.org) support Bearer auth with OAuth tokens.

> ++
> +This value should not be sent unless the appropriate capability (see below) is
> +provided on input.
> +
> +`credential`::
> +	The pre-encoded credential, suitable for the protocol in question (usually
> +	HTTP).  If this key is sent, `authtype` is mandatory, and `username` and
> +	`password` are not used.

A credential protocol attribute named 'credential' is confusing. How about 'authorization' since it determines the HTTP Authorization header? This detail is surely worth mentioning too.

> ++
> +This value should not be sent unless the appropriate capability (see below) is
> +provided on input.
> +
>  `wwwauth[]`::
>  
>  	When an HTTP response is received by Git that includes one or more
> @@ -189,7 +207,21 @@ attribute 'wwwauth[]', where the order of the attributes is the same as
>  they appear in the HTTP response. This attribute is 'one-way' from Git
>  to pass additional information to credential helpers.
>  
> -Unrecognised attributes are silently discarded.
> +`capability[]`::
> +	This signals that the caller supports the capability in question.
>
>
> 
> +	This can be used to provide better, more specific data as part of the
> +	protocol.
> ++
> +The only capability currently supported is `authtype`, which indicates that the
> +`authtype` and `credential` values are understood.  It is not obligatory to use
> +these values in such a case, but they should not be provided without this
> +capability.
>
> ++
> +Callers of `git credential` and credential helpers should emit the
> +capabilities they support unconditionally, and Git will gracefully
> +handle passing them on.
> +> +Unrecognised attributes and capabilities are silently discarded.

Re: [PATCH 06/13] docs: indicate new credential protocol fields

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1846 bytes --]

On 2024-03-25 at 23:16:09, M Hickford wrote:
> > +`authtype`::
> > +	This indicates that the authentication scheme in question should be used.
> > +	Common values for HTTP and HTTPS include `basic`, `digest`, and `ntlm`,
> > +	although the latter two are insecure and should not be used.  If `credential`
> > +	is used, this may be set to an arbitrary string suitable for the protocol in
> > +	question (usually HTTP).
> 
> How about adding 'bearer' to this list? Popular hosts Bitbucket
> https://bitbucket.org and Gitea/Forgejo (such as https://codeberg.org)
> support Bearer auth with OAuth tokens.

Sure, I can do that.

> > ++
> > +This value should not be sent unless the appropriate capability (see below) is
> > +provided on input.
> > +
> > +`credential`::
> > +	The pre-encoded credential, suitable for the protocol in question (usually
> > +	HTTP).  If this key is sent, `authtype` is mandatory, and `username` and
> > +	`password` are not used.
> 
> A credential protocol attribute named 'credential' is confusing. How
> about 'authorization' since it determines the HTTP Authorization
> header? This detail is surely worth mentioning too.

I don't want this to be very specific to HTTP, so I don't think that's a
great name.  As I mentioned in the cover letter, I might well extend
this to IMAP and SMTP for our mail handling in the future, and that name
wouldn't work well there.

I named it `credential` because, well, it's the credential that's used
in the protocol.  I feel like saying that the field represents "the
authorization" sounds unnatural.  It's not wrong, per se, but it sounds
confusing.

I'm open to other ideas if you or others have them, but between these
two, I think I'd prefer to stick with `credential`.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 03/13] http: use new headers for each object request

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 4997 bytes --]

On Sun, Mar 24, 2024 at 01:12:51AM +0000, brian m. carlson wrote:
> Currently we create one set of headers for all object requests and reuse
> it.  However, we'll need to adjust the headers for authentication
> purposes in the future, so let's create a new set for each request so
> that we can adjust them if the authentication changes.
> 
> Note that the cost of allocation here is tiny compared to the fact that
> we're making a network call, not to mention probably a full TLS
> connection, so this shouldn't have a significant impact on performance.
> Moreover, nobody who cares about performance is using the dumb HTTP
> protocol anyway, since it often makes huge numbers of requests compared
> to the smart protocol.
> 
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  http.c | 19 +++++++++++--------
>  http.h |  2 ++
>  2 files changed, 13 insertions(+), 8 deletions(-)
> 
> diff --git a/http.c b/http.c
> index e73b136e58..1c2200da77 100644
> --- a/http.c
> +++ b/http.c
> @@ -128,7 +128,6 @@ static unsigned long empty_auth_useless =
>  	| CURLAUTH_DIGEST;
>  
>  static struct curl_slist *pragma_header;
> -static struct curl_slist *no_pragma_header;
>  static struct string_list extra_http_headers = STRING_LIST_INIT_DUP;
>  
>  static struct curl_slist *host_resolutions;

Nice to see that this also allows us to get rid of one more global
variable.

> @@ -299,6 +298,11 @@ size_t fwrite_null(char *ptr UNUSED, size_t eltsize UNUSED, size_t nmemb,
>  	return nmemb;
>  }
>  
> +static struct curl_slist *object_request_headers(void)
> +{
> +	return curl_slist_append(http_copy_default_headers(), "Pragma:");
> +}
> +
>  static void closedown_active_slot(struct active_request_slot *slot)
>  {
>  	active_requests--;
> @@ -1275,8 +1279,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)
>  
>  	pragma_header = curl_slist_append(http_copy_default_headers(),
>  		"Pragma: no-cache");
> -	no_pragma_header = curl_slist_append(http_copy_default_headers(),
> -		"Pragma:");
>  
>  	{
>  		char *http_max_requests = getenv("GIT_HTTP_MAX_REQUESTS");
> @@ -1360,8 +1362,6 @@ void http_cleanup(void)
>  	curl_slist_free_all(pragma_header);
>  	pragma_header = NULL;
>  
> -	curl_slist_free_all(no_pragma_header);
> -	no_pragma_header = NULL;
>  

Nit: this leaves behind two consecutive empty lines.

Patrick

>  	curl_slist_free_all(host_resolutions);
>  	host_resolutions = NULL;
> @@ -2370,6 +2370,7 @@ void release_http_pack_request(struct http_pack_request *preq)
>  	}
>  	preq->slot = NULL;
>  	strbuf_release(&preq->tmpfile);
> +	curl_slist_free_all(preq->headers);
>  	free(preq->url);
>  	free(preq);
>  }
> @@ -2454,11 +2455,11 @@ struct http_pack_request *new_direct_http_pack_request(
>  	}
>  
>  	preq->slot = get_active_slot();
> +	preq->headers = object_request_headers();
>  	curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEDATA, preq->packfile);
>  	curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite);
>  	curl_easy_setopt(preq->slot->curl, CURLOPT_URL, preq->url);
> -	curl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER,
> -		no_pragma_header);
> +	curl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER, preq->headers);
>  
>  	/*
>  	 * If there is data present from a previous transfer attempt,
> @@ -2624,13 +2625,14 @@ struct http_object_request *new_http_object_request(const char *base_url,
>  	}
>  
>  	freq->slot = get_active_slot();
> +	freq->headers = object_request_headers();
>  
>  	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEDATA, freq);
>  	curl_easy_setopt(freq->slot->curl, CURLOPT_FAILONERROR, 0);
>  	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
>  	curl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);
>  	curl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);
> -	curl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
> +	curl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, freq->headers);
>  
>  	/*
>  	 * If we have successfully processed data from a previous fetch
> @@ -2718,5 +2720,6 @@ void release_http_object_request(struct http_object_request *freq)
>  		release_active_slot(freq->slot);
>  		freq->slot = NULL;
>  	}
> +	curl_slist_free_all(freq->headers);
>  	strbuf_release(&freq->tmpfile);
>  }
> diff --git a/http.h b/http.h
> index 3af19a8bf5..c5f8cc4620 100644
> --- a/http.h
> +++ b/http.h
> @@ -196,6 +196,7 @@ struct http_pack_request {
>  	FILE *packfile;
>  	struct strbuf tmpfile;
>  	struct active_request_slot *slot;
> +	struct curl_slist *headers;
>  };
>  
>  struct http_pack_request *new_http_pack_request(
> @@ -229,6 +230,7 @@ struct http_object_request {
>  	int zret;
>  	int rename;
>  	struct active_request_slot *slot;
> +	struct curl_slist *headers;
>  };
>  
>  struct http_object_request *new_http_object_request(
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 05/13] credential: gate new fields on capability

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 19755 bytes --]

On Sun, Mar 24, 2024 at 01:12:53AM +0000, brian m. carlson wrote:
> We support the new credential and authtype fields, but we lack a way to
> indicate to a credential helper that we'd like them to be used.  Without
> some sort of indication, the credential helper doesn't know if it should
> try to provide us a username and password, or a pre-encoded credential.
> For example, the helper might prefer a more restricted Bearer token if
> pre-encoded credentials are possible, but might have to fall back to
> more general username and password if not.
> 
> Let's provide a simple way to indicate whether Git (or, for that matter,
> the helper) is capable of understanding the authtype and credential
> fields.  We send this capability when we generate a request, and the
> other side may reply to indicate to us that it does, too.
> 
> For now, don't enable sending capabilities for the HTTP code.  In a
> future commit, we'll introduce appropriate handling for that code,
> which requires more in-depth work.
> 
> The logic for determining whether a capability is supported may seem
> complex, but it is not.  At each stage, we emit the capability to the
> following stage if all preceding stages have declared it.  Thus, if the
> caller to git credential fill didn't declare it, then we won't send it
> to the helper, and if fill's caller did send but the helper doesn't
> understand it, then we won't send it on in the response.  If we're an
> internal user, then we know about all capabilities and will request
> them.
> 
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  builtin/credential-cache--daemon.c |   2 +-
>  builtin/credential-store.c         |   2 +-
>  builtin/credential.c               |   6 +-
>  credential.c                       |  58 ++++++++++++++--
>  credential.h                       |  30 +++++++-
>  http.c                             |  10 +--
>  imap-send.c                        |   2 +-
>  remote-curl.c                      |   4 +-
>  t/t0300-credentials.sh             | 107 ++++++++++++++++++++++++++++-
>  9 files changed, 197 insertions(+), 24 deletions(-)
> 
> diff --git a/builtin/credential-cache--daemon.c b/builtin/credential-cache--daemon.c
> index 3a6a750a8e..ccbcf99ac1 100644
> --- a/builtin/credential-cache--daemon.c
> +++ b/builtin/credential-cache--daemon.c
> @@ -115,7 +115,7 @@ static int read_request(FILE *fh, struct credential *c,
>  		return error("client sent bogus timeout line: %s", item.buf);
>  	*timeout = atoi(p);
>  
> -	if (credential_read(c, fh) < 0)
> +	if (credential_read(c, fh, CREDENTIAL_OP_HELPER) < 0)
>  		return -1;
>  	return 0;
>  }
> diff --git a/builtin/credential-store.c b/builtin/credential-store.c
> index 4a492411bb..494c809332 100644
> --- a/builtin/credential-store.c
> +++ b/builtin/credential-store.c
> @@ -205,7 +205,7 @@ int cmd_credential_store(int argc, const char **argv, const char *prefix)
>  	if (!fns.nr)
>  		die("unable to set up default path; use --file");
>  
> -	if (credential_read(&c, stdin) < 0)
> +	if (credential_read(&c, stdin, CREDENTIAL_OP_HELPER) < 0)
>  		die("unable to read credential");
>  
>  	if (!strcmp(op, "get"))
> diff --git a/builtin/credential.c b/builtin/credential.c
> index 7010752987..5123dabcf1 100644
> --- a/builtin/credential.c
> +++ b/builtin/credential.c
> @@ -17,12 +17,12 @@ int cmd_credential(int argc, const char **argv, const char *prefix UNUSED)
>  		usage(usage_msg);
>  	op = argv[1];
>  
> -	if (credential_read(&c, stdin) < 0)
> +	if (credential_read(&c, stdin, CREDENTIAL_OP_INITIAL) < 0)
>  		die("unable to read credential from stdin");
>  
>  	if (!strcmp(op, "fill")) {
> -		credential_fill(&c);
> -		credential_write(&c, stdout);
> +		credential_fill(&c, 0);
> +		credential_write(&c, stdout, CREDENTIAL_OP_RESPONSE);
>  	} else if (!strcmp(op, "approve")) {
>  		credential_approve(&c);
>  	} else if (!strcmp(op, "reject")) {
> diff --git a/credential.c b/credential.c
> index c521822e5a..f2a26b8672 100644
> --- a/credential.c
> +++ b/credential.c
> @@ -34,6 +34,11 @@ void credential_clear(struct credential *c)
>  	credential_init(c);
>  }
>  
> +static void credential_set_all_capabilities(struct credential *c)
> +{
> +	c->capa_authtype.request_initial = 1;
> +}
> +
>  int credential_match(const struct credential *want,
>  		     const struct credential *have, int match_password)
>  {
> @@ -210,7 +215,39 @@ static void credential_getpass(struct credential *c)
>  						 PROMPT_ASKPASS);
>  }
>  
> -int credential_read(struct credential *c, FILE *fp)
> +static void credential_set_capability(struct credential_capability *capa, int op_type)
> +{
> +	switch (op_type) {
> +	case CREDENTIAL_OP_INITIAL:
> +		capa->request_initial = 1;
> +		break;
> +	case CREDENTIAL_OP_HELPER:
> +		capa->request_helper = 1;
> +		break;
> +	case CREDENTIAL_OP_RESPONSE:
> +		capa->response = 1;
> +		break;
> +	}
> +}
> +
> +static int credential_has_capability(const struct credential_capability *capa, int op_type)
> +{
> +	/*
> +	 * We're checking here if each previous step indicated that we had the
> +	 * capability.  If it did, then we want to pass it along; conversely, if
> +	 * it did not, we don't want to report that to our caller.
> +	 */
> +	switch (op_type) {
> +	case CREDENTIAL_OP_HELPER:
> +		return capa->request_initial;
> +	case CREDENTIAL_OP_RESPONSE:
> +		return capa->request_initial && capa->request_helper;
> +	default:
> +		return 0;
> +	}
> +}

I think I'm missing the bigger picture here, so please bear with me.

What you provide here is simply an `op_type` that indicates the phase we
are currently in and thus allows us to check whether all of the
preceding phases had the capability set. But to me it seems like a phase
and the actual capability should be different things. So why is it that
the capability seems to be a mere boolean value instead of something
like a bitfield indicating whether a specific capability is supported or
not? Or is all of this infra really only to support a single capability,
namely the credential capability?

I'm mostly coming from the angle of how capabilities work with remote
helpers. When asked, the helper will announce a set of capabilities that
it supports, e.g. "capabilities stateless-connect". So from thereon the
client of the helper knows that it can assume "stateless-connect" to be
understood by the helper.

I would have expected capabilities to work similarly for the credential
helper, where it announces "I know to handle pre-encoded credentials".
But given that I have basically no clue at all for how the credential
helper works there may very well be good reasons why things work so
differently here.

> +int credential_read(struct credential *c, FILE *fp, int op_type)
>  {
>  	struct strbuf line = STRBUF_INIT;
>  
> @@ -249,6 +286,8 @@ int credential_read(struct credential *c, FILE *fp)
>  			c->path = xstrdup(value);
>  		} else if (!strcmp(key, "wwwauth[]")) {
>  			strvec_push(&c->wwwauth_headers, value);
> +		} else if (!strcmp(key, "capability[]") && !strcmp(value, "authtype")) {
> +			credential_set_capability(&c->capa_authtype, op_type);
>  		} else if (!strcmp(key, "password_expiry_utc")) {
>  			errno = 0;
>  			c->password_expiry_utc = parse_timestamp(value, NULL, 10);
> @@ -288,14 +327,18 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
>  	fprintf(fp, "%s=%s\n", key, value);
>  }
>  
> -void credential_write(const struct credential *c, FILE *fp)
> +void credential_write(const struct credential *c, FILE *fp, int op_type)
>  {
> +	if (credential_has_capability(&c->capa_authtype, op_type)) {
> +		credential_write_item(fp, "capability[]", "authtype", 0);
> +		credential_write_item(fp, "authtype", c->authtype, 0);
> +		credential_write_item(fp, "credential", c->credential, 0);
> +	}
>  	credential_write_item(fp, "protocol", c->protocol, 1);
>  	credential_write_item(fp, "host", c->host, 1);
>  	credential_write_item(fp, "path", c->path, 0);
>  	credential_write_item(fp, "username", c->username, 0);
>  	credential_write_item(fp, "password", c->password, 0);
> -	credential_write_item(fp, "credential", c->credential, 0);
>  	credential_write_item(fp, "oauth_refresh_token", c->oauth_refresh_token, 0);
>  	if (c->password_expiry_utc != TIME_MAX) {
>  		char *s = xstrfmt("%"PRItime, c->password_expiry_utc);
> @@ -304,7 +347,6 @@ void credential_write(const struct credential *c, FILE *fp)
>  	}
>  	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
>  		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
> -	credential_write_item(fp, "authtype", c->authtype, 0);
>  }
>  
>  static int run_credential_helper(struct credential *c,
> @@ -327,14 +369,14 @@ static int run_credential_helper(struct credential *c,
>  
>  	fp = xfdopen(helper.in, "w");
>  	sigchain_push(SIGPIPE, SIG_IGN);
> -	credential_write(c, fp);
> +	credential_write(c, fp, want_output ? CREDENTIAL_OP_HELPER : CREDENTIAL_OP_RESPONSE);
>  	fclose(fp);
>  	sigchain_pop(SIGPIPE);
>  
>  	if (want_output) {
>  		int r;
>  		fp = xfdopen(helper.out, "r");
> -		r = credential_read(c, fp);
> +		r = credential_read(c, fp, CREDENTIAL_OP_HELPER);
>  		fclose(fp);
>  		if (r < 0) {
>  			finish_command(&helper);
> @@ -367,7 +409,7 @@ static int credential_do(struct credential *c, const char *helper,
>  	return r;
>  }
>  
> -void credential_fill(struct credential *c)
> +void credential_fill(struct credential *c, int all_capabilities)
>  {
>  	int i;
>  
> @@ -375,6 +417,8 @@ void credential_fill(struct credential *c)
>  		return;
>  
>  	credential_apply_config(c);
> +	if (all_capabilities)
> +		credential_set_all_capabilities(c);
>  
>  	for (i = 0; i < c->helpers.nr; i++) {
>  		credential_do(c, c->helpers.items[i].string, "get");
> diff --git a/credential.h b/credential.h
> index 9db892cf4d..2051d04c5a 100644
> --- a/credential.h
> +++ b/credential.h
> @@ -93,6 +93,25 @@
>   * -----------------------------------------------------------------------
>   */
>  
> +/*
> + * These values define the kind of operation we're performing and the
> + * capabilities at each stage.  The first is either an external request (via git
> + * credential fill) or an internal request (e.g., via the HTTP) code.  The
> + * second is the call to the credential helper, and the third is the response
> + * we're providing.
> + *
> + * At each stage, we will emit the capability only if the previous stage
> + * supported it.
> + */
> +#define CREDENTIAL_OP_INITIAL  1
> +#define CREDENTIAL_OP_HELPER   2
> +#define CREDENTIAL_OP_RESPONSE 3

Is there any specific reason why you're using defines instead of an enum
here? I think the latter would be more self-explanatory when you see
that functions take `enum credential_op` as input instead of an `int`.

Patrick

> +struct credential_capability {
> +	unsigned request_initial:1,
> +		 request_helper:1,
> +		 response:1;
> +};
>  
>  /**
>   * This struct represents a single username/password combination
> @@ -136,6 +155,8 @@ struct credential {
>  		 use_http_path:1,
>  		 username_from_proto:1;
>  
> +	struct credential_capability capa_authtype;
> +
>  	char *username;
>  	char *password;
>  	char *credential;
> @@ -174,8 +195,11 @@ void credential_clear(struct credential *);
>   * returns, the username and password fields of the credential are
>   * guaranteed to be non-NULL. If an error occurs, the function will
>   * die().
> + *
> + * If all_capabilities is set, this is an internal user that is prepared
> + * to deal with all known capabilities, and we should advertise that fact.
>   */
> -void credential_fill(struct credential *);
> +void credential_fill(struct credential *, int all_capabilities);
>  
>  /**
>   * Inform the credential subsystem that the provided credentials
> @@ -198,8 +222,8 @@ void credential_approve(struct credential *);
>   */
>  void credential_reject(struct credential *);
>  
> -int credential_read(struct credential *, FILE *);
> -void credential_write(const struct credential *, FILE *);
> +int credential_read(struct credential *, FILE *, int);
> +void credential_write(const struct credential *, FILE *, int);
>  
>  /*
>   * Parse a url into a credential struct, replacing any existing contents.
> diff --git a/http.c b/http.c
> index 1c2200da77..4f5df6fb14 100644
> --- a/http.c
> +++ b/http.c
> @@ -569,7 +569,7 @@ static void init_curl_http_auth(CURL *result)
>  		return;
>  	}
>  
> -	credential_fill(&http_auth);
> +	credential_fill(&http_auth, 0);
>  
>  	curl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);
>  	curl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);
> @@ -596,7 +596,7 @@ static void init_curl_proxy_auth(CURL *result)
>  {
>  	if (proxy_auth.username) {
>  		if (!proxy_auth.password)
> -			credential_fill(&proxy_auth);
> +			credential_fill(&proxy_auth, 0);
>  		set_proxyauth_name_password(result);
>  	}
>  
> @@ -630,7 +630,7 @@ static int has_cert_password(void)
>  		cert_auth.host = xstrdup("");
>  		cert_auth.username = xstrdup("");
>  		cert_auth.path = xstrdup(ssl_cert);
> -		credential_fill(&cert_auth);
> +		credential_fill(&cert_auth, 0);
>  	}
>  	return 1;
>  }
> @@ -645,7 +645,7 @@ static int has_proxy_cert_password(void)
>  		proxy_cert_auth.host = xstrdup("");
>  		proxy_cert_auth.username = xstrdup("");
>  		proxy_cert_auth.path = xstrdup(http_proxy_ssl_cert);
> -		credential_fill(&proxy_cert_auth);
> +		credential_fill(&proxy_cert_auth, 0);
>  	}
>  	return 1;
>  }
> @@ -2191,7 +2191,7 @@ static int http_request_reauth(const char *url,
>  		BUG("Unknown http_request target");
>  	}
>  
> -	credential_fill(&http_auth);
> +	credential_fill(&http_auth, 0);
>  
>  	return http_request(url, result, target, options);
>  }
> diff --git a/imap-send.c b/imap-send.c
> index f2e1947e63..8c89e866b6 100644
> --- a/imap-send.c
> +++ b/imap-send.c
> @@ -944,7 +944,7 @@ static void server_fill_credential(struct imap_server_conf *srvc, struct credent
>  	cred->username = xstrdup_or_null(srvc->user);
>  	cred->password = xstrdup_or_null(srvc->pass);
>  
> -	credential_fill(cred);
> +	credential_fill(cred, 1);
>  
>  	if (!srvc->user)
>  		srvc->user = xstrdup(cred->username);
> diff --git a/remote-curl.c b/remote-curl.c
> index e37eaa17b7..f96bda2431 100644
> --- a/remote-curl.c
> +++ b/remote-curl.c
> @@ -926,7 +926,7 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
>  		do {
>  			err = probe_rpc(rpc, &results);
>  			if (err == HTTP_REAUTH)
> -				credential_fill(&http_auth);
> +				credential_fill(&http_auth, 0);
>  		} while (err == HTTP_REAUTH);
>  		if (err != HTTP_OK)
>  			return -1;
> @@ -1044,7 +1044,7 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
>  	rpc->any_written = 0;
>  	err = run_slot(slot, NULL);
>  	if (err == HTTP_REAUTH && !large_request) {
> -		credential_fill(&http_auth);
> +		credential_fill(&http_auth, 0);
>  		curl_slist_free_all(headers);
>  		goto retry;
>  	}
> diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
> index 400f6bdbca..8477108b28 100755
> --- a/t/t0300-credentials.sh
> +++ b/t/t0300-credentials.sh
> @@ -12,7 +12,13 @@ test_expect_success 'setup helper scripts' '
>  	IFS==
>  	while read key value; do
>  		echo >&2 "$whoami: $key=$value"
> -		eval "$key=$value"
> +		if test -z "${key%%*\[\]}"
> +		then
> +			key=${key%%\[\]}
> +			eval "$key=\"\$$key $value\""
> +		else
> +			eval "$key=$value"
> +		fi
>  	done
>  	IFS=$OIFS
>  	EOF
> @@ -35,6 +41,16 @@ test_expect_success 'setup helper scripts' '
>  	test -z "$pass" || echo password=$pass
>  	EOF
>  
> +	write_script git-credential-verbatim-cred <<-\EOF &&
> +	authtype=$1; shift
> +	credential=$1; shift
> +	. ./dump
> +	echo capability[]=authtype
> +	test -z "${capability##*authtype*}" || return
> +	test -z "$authtype" || echo authtype=$authtype
> +	test -z "$credential" || echo credential=$credential
> +	EOF
> +
>  	write_script git-credential-verbatim-with-expiry <<-\EOF &&
>  	user=$1; shift
>  	pass=$1; shift
> @@ -64,6 +80,26 @@ test_expect_success 'credential_fill invokes helper' '
>  	EOF
>  '
>  
> +test_expect_success 'credential_fill invokes helper with credential' '
> +	check fill "verbatim-cred Bearer token" <<-\EOF
> +	capability[]=authtype
> +	protocol=http
> +	host=example.com
> +	--
> +	capability[]=authtype
> +	authtype=Bearer
> +	credential=token
> +	protocol=http
> +	host=example.com
> +	--
> +	verbatim-cred: get
> +	verbatim-cred: capability[]=authtype
> +	verbatim-cred: protocol=http
> +	verbatim-cred: host=example.com
> +	EOF
> +'
> +
> +
>  test_expect_success 'credential_fill invokes multiple helpers' '
>  	check fill useless "verbatim foo bar" <<-\EOF
>  	protocol=http
> @@ -83,6 +119,42 @@ test_expect_success 'credential_fill invokes multiple helpers' '
>  	EOF
>  '
>  
> +test_expect_success 'credential_fill response does not get capabilities when helpers are incapable' '
> +	check fill useless "verbatim foo bar" <<-\EOF
> +	capability[]=authtype
> +	protocol=http
> +	host=example.com
> +	--
> +	protocol=http
> +	host=example.com
> +	username=foo
> +	password=bar
> +	--
> +	useless: get
> +	useless: capability[]=authtype
> +	useless: protocol=http
> +	useless: host=example.com
> +	verbatim: get
> +	verbatim: capability[]=authtype
> +	verbatim: protocol=http
> +	verbatim: host=example.com
> +	EOF
> +'
> +
> +test_expect_success 'credential_fill response does not get capabilities when caller is incapable' '
> +	check fill "verbatim-cred Bearer token" <<-\EOF
> +	protocol=http
> +	host=example.com
> +	--
> +	protocol=http
> +	host=example.com
> +	--
> +	verbatim-cred: get
> +	verbatim-cred: protocol=http
> +	verbatim-cred: host=example.com
> +	EOF
> +'
> +
>  test_expect_success 'credential_fill stops when we get a full response' '
>  	check fill "verbatim one two" "verbatim three four" <<-\EOF
>  	protocol=http
> @@ -99,6 +171,25 @@ test_expect_success 'credential_fill stops when we get a full response' '
>  	EOF
>  '
>  
> +test_expect_success 'credential_fill thinks a credential is a full response' '
> +	check fill "verbatim-cred Bearer token" "verbatim three four" <<-\EOF
> +	capability[]=authtype
> +	protocol=http
> +	host=example.com
> +	--
> +	capability[]=authtype
> +	authtype=Bearer
> +	credential=token
> +	protocol=http
> +	host=example.com
> +	--
> +	verbatim-cred: get
> +	verbatim-cred: capability[]=authtype
> +	verbatim-cred: protocol=http
> +	verbatim-cred: host=example.com
> +	EOF
> +'
> +
>  test_expect_success 'credential_fill continues through partial response' '
>  	check fill "verbatim one \"\"" "verbatim two three" <<-\EOF
>  	protocol=http
> @@ -175,6 +266,20 @@ test_expect_success 'credential_fill passes along metadata' '
>  	EOF
>  '
>  
> +test_expect_success 'credential_fill produces no credential without capability' '
> +	check fill "verbatim-cred Bearer token" <<-\EOF
> +	protocol=http
> +	host=example.com
> +	--
> +	protocol=http
> +	host=example.com
> +	--
> +	verbatim-cred: get
> +	verbatim-cred: protocol=http
> +	verbatim-cred: host=example.com
> +	EOF
> +'
> +
>  test_expect_success 'credential_approve calls all helpers' '
>  	check approve useless "verbatim one two" <<-\EOF
>  	protocol=http
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 12/13] strvec: implement swapping two strvecs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

On Sun, Mar 24, 2024 at 01:13:00AM +0000, brian m. carlson wrote:
> In a future commit, we'll want the ability to efficiently swap the
> contents of two strvec instances without needing to copy any data.
> Since a strvec is simply a pointer and two sizes, swapping them is as
> simply as copying the two pointers and sizes, so let's do that.
> 
> We use a temporary here simply because C doesn't provide a standard
> swapping function, unlike C++ and Rust, but a good optimizing compiler
> will recognize this syntax and handle it appropriately using an
> optimization pass.
> 
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  strvec.c | 7 +++++++
>  strvec.h | 5 +++++
>  2 files changed, 12 insertions(+)
> 
> diff --git a/strvec.c b/strvec.c
> index 178f4f3748..93006f1e63 100644
> --- a/strvec.c
> +++ b/strvec.c
> @@ -106,3 +106,10 @@ const char **strvec_detach(struct strvec *array)
>  		return ret;
>  	}
>  }
> +
> +void strvec_swap(struct strvec *a, struct strvec *b)
> +{
> +	struct strvec t = *a;
> +	*a = *b;
> +	*b = t;
> +}

Isn't this equivalent to `SWAP(*a, *b)`?

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 12/13] strvec: implement swapping two strvecs

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

>> +
>> +void strvec_swap(struct strvec *a, struct strvec *b)
>> +{
>> +	struct strvec t = *a;
>> +	*a = *b;
>> +	*b = t;
>> +}
>
> Isn't this equivalent to `SWAP(*a, *b)`?

Yes.  "make coccicheck" does flag this one.

Let's drop this step, and tweak the 13/13 patch to make its sole
caller directly use SWAP() instead, perhaps like so:

 credential.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git c/credential.c w/credential.c
index 9a08efe113..5ea52ddd68 100644
--- c/credential.c
+++ w/credential.c
@@ -39,7 +39,7 @@ void credential_clear(struct credential *c)
 void credential_next_state(struct credential *c)
 {
 	strvec_clear(&c->state_headers_to_send);
-	strvec_swap(&c->state_headers, &c->state_headers_to_send);
+	SWAP(c->state_headers, c->state_headers_to_send);
 }
 
 void credential_clear_secrets(struct credential *c)

Re: [PATCH 05/13] credential: gate new fields on capability

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 5213 bytes --]

On 2024-03-27 at 08:02:39, Patrick Steinhardt wrote:
> On Sun, Mar 24, 2024 at 01:12:53AM +0000, brian m. carlson wrote:
> > +static int credential_has_capability(const struct credential_capability *capa, int op_type)
> > +{
> > +	/*
> > +	 * We're checking here if each previous step indicated that we had the
> > +	 * capability.  If it did, then we want to pass it along; conversely, if
> > +	 * it did not, we don't want to report that to our caller.
> > +	 */
> > +	switch (op_type) {
> > +	case CREDENTIAL_OP_HELPER:
> > +		return capa->request_initial;
> > +	case CREDENTIAL_OP_RESPONSE:
> > +		return capa->request_initial && capa->request_helper;
> > +	default:
> > +		return 0;
> > +	}
> > +}
> 
> I think I'm missing the bigger picture here, so please bear with me.
> 
> What you provide here is simply an `op_type` that indicates the phase we
> are currently in and thus allows us to check whether all of the
> preceding phases had the capability set. But to me it seems like a phase
> and the actual capability should be different things. So why is it that
> the capability seems to be a mere boolean value instead of something
> like a bitfield indicating whether a specific capability is supported or
> not? Or is all of this infra really only to support a single capability,
> namely the credential capability?
> 
> I'm mostly coming from the angle of how capabilities work with remote
> helpers. When asked, the helper will announce a set of capabilities that
> it supports, e.g. "capabilities stateless-connect". So from thereon the
> client of the helper knows that it can assume "stateless-connect" to be
> understood by the helper.
> 
> I would have expected capabilities to work similarly for the credential
> helper, where it announces "I know to handle pre-encoded credentials".
> But given that I have basically no clue at all for how the credential
> helper works there may very well be good reasons why things work so
> differently here.

Let me explain a little bit.  There are two possible flows that we can
have for a credential request:

  git-credential input -> credential.c -> helper -> credential.c -> git-credential output

  git-http-backend -> credential.c -> helper -> credential.c -> git-http-backend

Let's look at the first one (which might, say, come from Git LFS or
another external tool), but the second one works similarly.  When we get
a request from `git credential fill`, we need to know first whether the
requester supports the capability.  If we're using an external tool from
last decade, it's not going to do so.

If it _does_ support that, then we want to pass that along to the
helper, but if it doesn't, we don't.  That's because if the caller
doesn't support `credential` and `authtype`, the helper might
legitimately want to provide a username and password (or token) instead,
knowing that that's more likely to work.

Similarly, in the final response, we want to indicate to the external
caller whether the capability was in fact supported.  That's useful to
know in case we want to pass the response back to `git credential
store`, and it also discloses functionality about what the credential
helper in use supports.

We can't assume that the helper does or doesn't support the same
capabilities as Git because it might come from outside Git (e.g., Git
Credential Manager Core, or a site-specific credential helper) or it
just might not be capable of storing or handling that kind of
credential.  By not making the assumption that the helper is implicitly
capable, we allow users to continue to use very simple shell scripts as
credential helpers.

As to why this functionality exists, it exists to support the two new
capabilities in this series, `credential` and `state`.  A pie in the sky
goal for the future is to support additional request signing
functionality, so it might learn things like method, URI, and TLS
channel binding info, which would be an additional capability.  (I might
implement that, or I might not.)  All of those are boolean: they either
are supported, or not.  If folks in the future want to expose
non-boolean capabilities, I don't think that should be a problem.

> > +/*
> > + * These values define the kind of operation we're performing and the
> > + * capabilities at each stage.  The first is either an external request (via git
> > + * credential fill) or an internal request (e.g., via the HTTP) code.  The
> > + * second is the call to the credential helper, and the third is the response
> > + * we're providing.
> > + *
> > + * At each stage, we will emit the capability only if the previous stage
> > + * supported it.
> > + */
> > +#define CREDENTIAL_OP_INITIAL  1
> > +#define CREDENTIAL_OP_HELPER   2
> > +#define CREDENTIAL_OP_RESPONSE 3
> 
> Is there any specific reason why you're using defines instead of an enum
> here? I think the latter would be more self-explanatory when you see
> that functions take `enum credential_op` as input instead of an `int`.

I think an enum would be a nice improvement.  I'll include that in a
reroll.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 12/13] strvec: implement swapping two strvecs

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

On 2024-03-27 at 21:22:17, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> >> +
> >> +void strvec_swap(struct strvec *a, struct strvec *b)
> >> +{
> >> +	struct strvec t = *a;
> >> +	*a = *b;
> >> +	*b = t;
> >> +}
> >
> > Isn't this equivalent to `SWAP(*a, *b)`?
> 
> Yes.  "make coccicheck" does flag this one.
> 
> Let's drop this step, and tweak the 13/13 patch to make its sole
> caller directly use SWAP() instead, perhaps like so:

Great, I'll include that in a reroll, likely this weekend.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 13/13] credential: add support for multistage credential rounds

[M Hickford]

On Sun, 24 Mar 2024 at 01:13, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> Over HTTP, NTLM and Kerberos require two rounds of authentication on the
> client side.  It's possible that there are custom authentication schemes
> that also implement this same approach.  Since these are tricky schemes
> to implement and the HTTP library in use may not always handle them
> gracefully on all systems, it would be helpful to allow the credential
> helper to implement them instead for increased portability and
> robustness.

Is this design sufficiently flexible for OAuth DPoP (RFC 9449), or at
least to make it work in future?

OAuth 2.0 Demonstrating Proof of Possession describes "a mechanism for
sender-constraining OAuth 2.0 tokens via a proof-of-possession
mechanism on the application level. This mechanism allows for the
detection of replay attacks with access and refresh tokens."
https://www.rfc-editor.org/rfc/rfc9449.html

Popular hosts GitHub, GitLab, Bitbucket and Gitea already support
OAuth. OAuth DPoP "provides a general defense in depth against the
impact of unanticipated token leakage". Motivated by a 2022 GitHub
attack involving stolen tokens
(https://github.blog/2022-04-15-security-alert-stolen-oauth-user-tokens/),
some hosts are already experimenting with it.
https://lore.kernel.org/git/20230128142827.17397-1-mirth.hickford@gmail.com/

In particular, the http request has to include both Authorization and
DPoP headers https://www.rfc-editor.org/rfc/rfc9449.html#name-the-dpop-authentication-sch.
The latter depends on timestamp and a server-optional challenge in a
DPoP-Nonce header.
https://www.rfc-editor.org/rfc/rfc9449.html#name-resource-server-provided-no.


On Sun, 24 Mar 2024 at 01:13, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> Over HTTP, NTLM and Kerberos require two rounds of authentication on the
> client side.  It's possible that there are custom authentication schemes
> that also implement this same approach.  Since these are tricky schemes
> to implement and the HTTP library in use may not always handle them
> gracefully on all systems, it would be helpful to allow the credential
> helper to implement them instead for increased portability and
> robustness.
>
> To allow this to happen, add a boolean flag, continue, that indicates
> that instead of failing when we get a 401, we should retry another round
> of authentication.  However, this necessitates some changes in our
> current credential code so that we can make this work.
>
> Keep the state[] headers between iterations, but only use them to send
> to the helper and only consider the new ones we read from the credential
> helper to be valid on subsequent iterations.  That avoids us passing
> stale data when we finally approve or reject the credential.  Similarly,
> clear the multistage and wwwauth[] values appropriately so that we
> don't pass stale data or think we're trying a multiround response when
> we're not.  Remove the credential values so that we can actually fill a
> second time with new responses.
>
> Limit the number of iterations of reauthentication we do to 3.  This
> means that if there's a problem, we'll terminate with an error message
> instead of retrying indefinitely and not informing the user (and
> possibly conducting a DoS on the server).
>
> In our tests, handle creating multiple response output files from our
> helper so we can verify that each of the messages sent is correct.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  Documentation/git-credential.txt | 16 +++++-
>  builtin/credential.c             |  1 +
>  credential.c                     | 32 ++++++++++--
>  credential.h                     | 27 +++++++++-
>  http.c                           | 59 +++++++++++----------
>  t/t5563-simple-http-auth.sh      | 89 ++++++++++++++++++++++++++++++--
>  6 files changed, 187 insertions(+), 37 deletions(-)
>
> diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
> index 6b7e017066..160dee5c6a 100644
> --- a/Documentation/git-credential.txt
> +++ b/Documentation/git-credential.txt
> @@ -207,6 +207,19 @@ provided on input.
>  This value should not be sent unless the appropriate capability (see below) is
>  provided on input.
>
> +`continue`::
> +       This is a boolean value, which, if enabled, indicates that this
> +       authentication is a non-final part of a multistage authentication step. This
> +       is common in protocols such as NTLM and Kerberos, where two rounds of client
> +       authentication are required, and setting this flag allows the credential
> +       helper to implement the multistage authentication step.  This flag should
> +       only be sent if a further stage is required; that is, if another round of
> +       authentication is expected.
> ++
> +This value should not be sent unless the appropriate capability (see below) is
> +provided on input.  This attribute is 'one-way' from a credential helper to
> +pass information to Git (or other programs invoking `git credential`).
> +
>  `wwwauth[]`::
>
>         When an HTTP response is received by Git that includes one or more
> @@ -225,7 +238,8 @@ to pass additional information to credential helpers.
>  +
>  There are two currently supported capabilities.  The first is `authtype`, which
>  indicates that the `authtype` and `credential` values are understood.  The
> -second is `state`, which indicates that the `state[]` value is understood.
> +second is `state`, which indicates that the `state[]` and `continue` values are
> +understood.
>
>  It is not obligatory to use the additional features just because the capability
>  is supported, but they should not be provided without this capability.
> diff --git a/builtin/credential.c b/builtin/credential.c
> index 5123dabcf1..f14d1b5ed6 100644
> --- a/builtin/credential.c
> +++ b/builtin/credential.c
> @@ -22,6 +22,7 @@ int cmd_credential(int argc, const char **argv, const char *prefix UNUSED)
>
>         if (!strcmp(op, "fill")) {
>                 credential_fill(&c, 0);
> +               credential_next_state(&c);
>                 credential_write(&c, stdout, CREDENTIAL_OP_RESPONSE);
>         } else if (!strcmp(op, "approve")) {
>                 credential_approve(&c);
> diff --git a/credential.c b/credential.c
> index 0ca7c12895..9a08efe113 100644
> --- a/credential.c
> +++ b/credential.c
> @@ -31,10 +31,23 @@ void credential_clear(struct credential *c)
>         string_list_clear(&c->helpers, 0);
>         strvec_clear(&c->wwwauth_headers);
>         strvec_clear(&c->state_headers);
> +       strvec_clear(&c->state_headers_to_send);
>
>         credential_init(c);
>  }
>
> +void credential_next_state(struct credential *c)
> +{
> +       strvec_clear(&c->state_headers_to_send);
> +       strvec_swap(&c->state_headers, &c->state_headers_to_send);
> +}
> +
> +void credential_clear_secrets(struct credential *c)
> +{
> +       FREE_AND_NULL(c->password);
> +       FREE_AND_NULL(c->credential);
> +}
> +
>  static void credential_set_all_capabilities(struct credential *c)
>  {
>         c->capa_authtype.request_initial = 1;
> @@ -295,6 +308,8 @@ int credential_read(struct credential *c, FILE *fp, int op_type)
>                                 credential_set_capability(&c->capa_authtype, op_type);
>                         else if (!strcmp(value, "state"))
>                                 credential_set_capability(&c->capa_state, op_type);
> +               } else if (!strcmp(key, "continue")) {
> +                       c->multistage = !!git_config_bool("continue", value);
>                 } else if (!strcmp(key, "password_expiry_utc")) {
>                         errno = 0;
>                         c->password_expiry_utc = parse_timestamp(value, NULL, 10);
> @@ -359,8 +374,10 @@ void credential_write(const struct credential *c, FILE *fp, int op_type)
>         for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
>                 credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
>         if (credential_has_capability(&c->capa_state, op_type)) {
> -               for (size_t i = 0; i < c->state_headers.nr; i++)
> -                       credential_write_item(fp, "state[]", c->state_headers.v[i], 0);
> +               if (c->multistage)
> +                       credential_write_item(fp, "continue", "1", 0);
> +               for (size_t i = 0; i < c->state_headers_to_send.nr; i++)
> +                       credential_write_item(fp, "state[]", c->state_headers_to_send.v[i], 0);
>         }
>  }
>
> @@ -431,6 +448,9 @@ void credential_fill(struct credential *c, int all_capabilities)
>         if ((c->username && c->password) || c->credential)
>                 return;
>
> +       credential_next_state(c);
> +       c->multistage = 0;
> +
>         credential_apply_config(c);
>         if (all_capabilities)
>                 credential_set_all_capabilities(c);
> @@ -443,8 +463,10 @@ void credential_fill(struct credential *c, int all_capabilities)
>                         /* Reset expiry to maintain consistency */
>                         c->password_expiry_utc = TIME_MAX;
>                 }
> -               if ((c->username && c->password) || c->credential)
> +               if ((c->username && c->password) || c->credential) {
> +                       strvec_clear(&c->wwwauth_headers);
>                         return;
> +               }
>                 if (c->quit)
>                         die("credential helper '%s' told us to quit",
>                             c->helpers.items[i].string);
> @@ -464,6 +486,8 @@ void credential_approve(struct credential *c)
>         if (((!c->username || !c->password) && !c->credential) || c->password_expiry_utc < time(NULL))
>                 return;
>
> +       credential_next_state(c);
> +
>         credential_apply_config(c);
>
>         for (i = 0; i < c->helpers.nr; i++)
> @@ -475,6 +499,8 @@ void credential_reject(struct credential *c)
>  {
>         int i;
>
> +       credential_next_state(c);
> +
>         credential_apply_config(c);
>
>         for (i = 0; i < c->helpers.nr; i++)
> diff --git a/credential.h b/credential.h
> index e2021455fe..adb1fc370a 100644
> --- a/credential.h
> +++ b/credential.h
> @@ -143,10 +143,15 @@ struct credential {
>         struct strvec wwwauth_headers;
>
>         /**
> -        * A `strvec` of state headers from credential helpers.
> +        * A `strvec` of state headers received from credential helpers.
>          */
>         struct strvec state_headers;
>
> +       /**
> +        * A `strvec` of state headers to send to credential helpers.
> +        */
> +       struct strvec state_headers_to_send;
> +
>         /**
>          * Internal use only. Keeps track of if we previously matched against a
>          * WWW-Authenticate header line in order to re-fold future continuation
> @@ -156,6 +161,7 @@ struct credential {
>
>         unsigned approved:1,
>                  configured:1,
> +                multistage: 1,
>                  quit:1,
>                  use_http_path:1,
>                  username_from_proto:1;
> @@ -184,6 +190,7 @@ struct credential {
>         .password_expiry_utc = TIME_MAX, \
>         .wwwauth_headers = STRVEC_INIT, \
>         .state_headers = STRVEC_INIT, \
> +       .state_headers_to_send = STRVEC_INIT, \
>  }
>
>  /* Initialize a credential structure, setting all fields to empty. */
> @@ -229,6 +236,24 @@ void credential_approve(struct credential *);
>   */
>  void credential_reject(struct credential *);
>
> +/**
> + * Clear the secrets in this credential, but leave other data intact.
> + *
> + * This is useful for resetting credentials in preparation for a subsequent
> + * stage of filling.
> + */
> +void credential_clear_secrets(struct credential *c);
> +
> +/**
> + * Prepares the credential for the next iteration of the helper protocol by
> + * updating the state headers to send with the ones read by the last iteration
> + * of the protocol.
> + *
> + * Except for internal callers, this should be called exactly once between
> + * reading credentials with `credential_fill` and writing them.
> + */
> +void credential_next_state(struct credential *c);
> +
>  int credential_read(struct credential *, FILE *, int);
>  void credential_write(const struct credential *, FILE *, int);
>
> diff --git a/http.c b/http.c
> index f98c520924..9d373c6460 100644
> --- a/http.c
> +++ b/http.c
> @@ -1781,6 +1781,10 @@ static int handle_curl_result(struct slot_results *results)
>         else if (results->http_code == 401) {
>                 if ((http_auth.username && http_auth.password) ||\
>                     (http_auth.authtype && http_auth.credential)) {
> +                       if (http_auth.multistage) {
> +                               credential_clear_secrets(&http_auth);
> +                               return HTTP_REAUTH;
> +                       }
>                         credential_reject(&http_auth);
>                         return HTTP_NOAUTH;
>                 } else {
> @@ -2178,6 +2182,7 @@ static int http_request_reauth(const char *url,
>                                void *result, int target,
>                                struct http_get_options *options)
>  {
> +       int i = 3;
>         int ret = http_request(url, result, target, options);
>
>         if (ret != HTTP_OK && ret != HTTP_REAUTH)
> @@ -2191,35 +2196,35 @@ static int http_request_reauth(const char *url,
>                 }
>         }
>
> -       if (ret != HTTP_REAUTH)
> -               return ret;
> +       while (ret == HTTP_REAUTH && --i) {
> +               /*
> +                * The previous request may have put cruft into our output stream; we
> +                * should clear it out before making our next request.
> +                */
> +               switch (target) {
> +               case HTTP_REQUEST_STRBUF:
> +                       strbuf_reset(result);
> +                       break;
> +               case HTTP_REQUEST_FILE:
> +                       if (fflush(result)) {
> +                               error_errno("unable to flush a file");
> +                               return HTTP_START_FAILED;
> +                       }
> +                       rewind(result);
> +                       if (ftruncate(fileno(result), 0) < 0) {
> +                               error_errno("unable to truncate a file");
> +                               return HTTP_START_FAILED;
> +                       }
> +                       break;
> +               default:
> +                       BUG("Unknown http_request target");
> +               }
>
> -       /*
> -        * The previous request may have put cruft into our output stream; we
> -        * should clear it out before making our next request.
> -        */
> -       switch (target) {
> -       case HTTP_REQUEST_STRBUF:
> -               strbuf_reset(result);
> -               break;
> -       case HTTP_REQUEST_FILE:
> -               if (fflush(result)) {
> -                       error_errno("unable to flush a file");
> -                       return HTTP_START_FAILED;
> -               }
> -               rewind(result);
> -               if (ftruncate(fileno(result), 0) < 0) {
> -                       error_errno("unable to truncate a file");
> -                       return HTTP_START_FAILED;
> -               }
> -               break;
> -       default:
> -               BUG("Unknown http_request target");
> +               credential_fill(&http_auth, 1);
> +
> +               ret = http_request(url, result, target, options);
>         }
> -
> -       credential_fill(&http_auth, 1);
> -
> -       return http_request(url, result, target, options);
> +       return ret;
>  }
>
>  int http_get_strbuf(const char *url,
> diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
> index 515185ae00..5d5caa3f58 100755
> --- a/t/t5563-simple-http-auth.sh
> +++ b/t/t5563-simple-http-auth.sh
> @@ -21,9 +21,17 @@ test_expect_success 'setup_credential_helper' '
>         CREDENTIAL_HELPER="$TRASH_DIRECTORY/bin/git-credential-test-helper" &&
>         write_script "$CREDENTIAL_HELPER" <<-\EOF
>         cmd=$1
> -       teefile=$cmd-query.cred
> +       teefile=$cmd-query-temp.cred
>         catfile=$cmd-reply.cred
>         sed -n -e "/^$/q" -e "p" >>$teefile
> +       state=$(sed -ne "s/^state\[\]=helper://p" "$teefile")
> +       if test -z "$state"
> +       then
> +               mv "$teefile" "$cmd-query.cred"
> +       else
> +               mv "$teefile" "$cmd-query-$state.cred"
> +               catfile="$cmd-reply-$state.cred"
> +       fi
>         if test "$cmd" = "get"
>         then
>                 cat $catfile
> @@ -32,13 +40,15 @@ test_expect_success 'setup_credential_helper' '
>  '
>
>  set_credential_reply () {
> -       cat >"$TRASH_DIRECTORY/$1-reply.cred"
> +       local suffix="$(test -n "$2" && echo "-$2")"
> +       cat >"$TRASH_DIRECTORY/$1-reply$suffix.cred"
>  }
>
>  expect_credential_query () {
> -       cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
> -       test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
> -                "$TRASH_DIRECTORY/$1-query.cred"
> +       local suffix="$(test -n "$2" && echo "-$2")"
> +       cat >"$TRASH_DIRECTORY/$1-expect$suffix.cred" &&
> +       test_cmp "$TRASH_DIRECTORY/$1-expect$suffix.cred" \
> +                "$TRASH_DIRECTORY/$1-query$suffix.cred"
>  }
>
>  per_test_cleanup () {
> @@ -479,4 +489,73 @@ test_expect_success 'access using bearer auth with invalid credentials' '
>         EOF
>  '
>
> +test_expect_success 'access using three-legged auth' '
> +       test_when_finished "per_test_cleanup" &&
> +
> +       set_credential_reply get <<-EOF &&
> +       capability[]=authtype
> +       capability[]=state
> +       authtype=Multistage
> +       credential=YS1naXQtdG9rZW4=
> +       state[]=helper:foobar
> +       continue=1
> +       EOF
> +
> +       set_credential_reply get foobar <<-EOF &&
> +       capability[]=authtype
> +       capability[]=state
> +       authtype=Multistage
> +       credential=YW5vdGhlci10b2tlbg==
> +       state[]=helper:bazquux
> +       EOF
> +
> +       cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
> +       id=1 creds=Multistage YS1naXQtdG9rZW4=
> +       id=2 creds=Multistage YW5vdGhlci10b2tlbg==
> +       EOF
> +
> +       CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
> +
> +       cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
> +       id=1 status=401 response=WWW-Authenticate: Multistage challenge="456"
> +       id=1 status=401 response=WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
> +       id=2 status=200
> +       id=default response=WWW-Authenticate: Multistage challenge="123"
> +       id=default response=WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
> +       EOF
> +
> +       test_config_global credential.helper test-helper &&
> +       git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
> +
> +       expect_credential_query get <<-EOF &&
> +       capability[]=authtype
> +       capability[]=state
> +       protocol=http
> +       host=$HTTPD_DEST
> +       wwwauth[]=Multistage challenge="123"
> +       wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
> +       EOF
> +
> +       expect_credential_query get foobar <<-EOF &&
> +       capability[]=authtype
> +       capability[]=state
> +       authtype=Multistage
> +       protocol=http
> +       host=$HTTPD_DEST
> +       wwwauth[]=Multistage challenge="456"
> +       wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
> +       state[]=helper:foobar
> +       EOF
> +
> +       expect_credential_query store bazquux <<-EOF
> +       capability[]=authtype
> +       capability[]=state
> +       authtype=Multistage
> +       credential=YW5vdGhlci10b2tlbg==
> +       protocol=http
> +       host=$HTTPD_DEST
> +       state[]=helper:bazquux
> +       EOF
> +'
> +
>  test_done

Re: [PATCH 05/13] credential: gate new fields on capability

[Jeff King]

On Sun, Mar 24, 2024 at 01:12:53AM +0000, brian m. carlson wrote:

> @@ -35,6 +41,16 @@ test_expect_success 'setup helper scripts' '
>  	test -z "$pass" || echo password=$pass
>  	EOF
>  
> +	write_script git-credential-verbatim-cred <<-\EOF &&
> +	authtype=$1; shift
> +	credential=$1; shift
> +	. ./dump
> +	echo capability[]=authtype
> +	test -z "${capability##*authtype*}" || return
> +	test -z "$authtype" || echo authtype=$authtype
> +	test -z "$credential" || echo credential=$credential
> +	EOF

I think this "|| return" needs to be "|| exit 0" or similar. The Windows
CI jobs fail with:

  --- a/expect-stderr
  +++ b/stderr
  @@ -2,3 +2,4 @@ verbatim-cred: get
   verbatim-cred: capability[]=authtype
   verbatim-cred: protocol=http
   verbatim-cred: host=example.com
  +D:\a\git\git\t\trash directory.t0300-credentials\git-credential-verbatim-cred: line 10: return: can only `return' from a function or sourced script

(actually if you count the line numbers, I think this particular case is
the similar "|| return" added to the script later, but both should be
fixed).

It doesn't show up elsewhere because only bash complains, but not dash.
Even running the test script with bash isn't enough, because
write_script uses $SHELL_PATH under the hood. But building with "make
SHELL_PATH=/bin/bash test" shows the problem on other platforms.

-Peff

Re: [PATCH 05/13] credential: gate new fields on capability

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> It doesn't show up elsewhere because only bash complains, but not dash.
> Even running the test script with bash isn't enough, because
> write_script uses $SHELL_PATH under the hood. But building with "make
> SHELL_PATH=/bin/bash test" shows the problem on other platforms.

Can we sneak it in to the GitHub Actions CI, I wonder, so that we
can catch tests that only fail with bash.  Would this be sufficient,
or can we just export it without using $use_bash to place it on the
command line of make?

 ci/run-build-and-tests.sh | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git i/ci/run-build-and-tests.sh w/ci/run-build-and-tests.sh
index c192bd613c..8fb1114bc5 100755
--- i/ci/run-build-and-tests.sh
+++ w/ci/run-build-and-tests.sh
@@ -11,6 +11,7 @@ windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";;
 esac
 
 run_tests=t
+use_bash=
 
 case "$jobname" in
 linux-gcc)
@@ -30,6 +31,7 @@ linux-TEST-vars)
 	export GIT_TEST_NO_WRITE_REV_INDEX=1
 	export GIT_TEST_CHECKOUT_WORKERS=2
 	export GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL=1
+	use_bash=/bin/bash
 	;;
 linux-clang)
 	export GIT_TEST_DEFAULT_HASH=sha1
@@ -51,7 +53,8 @@ esac
 group Build make
 if test -n "$run_tests"
 then
-	group "Run tests" make test ||
+	group "Run tests" \
+		make ${use_bash:+SHELL_PATH="$use_bash"} test ||
 	handle_failed_tests
 	group "Run unit tests" \
 		make DEFAULT_UNIT_TEST_TARGET=unit-tests-prove unit-tests

Re: [PATCH 05/13] credential: gate new fields on capability

[Jeff King]

On Thu, Mar 28, 2024 at 09:13:38AM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > It doesn't show up elsewhere because only bash complains, but not dash.
> > Even running the test script with bash isn't enough, because
> > write_script uses $SHELL_PATH under the hood. But building with "make
> > SHELL_PATH=/bin/bash test" shows the problem on other platforms.
> 
> Can we sneak it in to the GitHub Actions CI, I wonder, so that we
> can catch tests that only fail with bash.  Would this be sufficient,
> or can we just export it without using $use_bash to place it on the
> command line of make?

I think the sneaking has already been done, because Windows CI uses bash
(which is after all how I noticed this). I'm not sure if using bash more
places would be helpful. On the one hand, there are enough _other_
differences in Windows that it is not always immediately obvious that
the shell is the culprit. On the other hand, I would probably forget
that linux-gcc is the special one with bash, and just end up reading the
test output anyway. So I dunno.

As to your question, yes, I think you could just export SHELL_PATH; our
Makefile uses "ifndef".

-Peff

Re: [PATCH 05/13] credential: gate new fields on capability

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I think the sneaking has already been done, because Windows CI uses bash
> (which is after all how I noticed this). I'm not sure if using bash more
> places would be helpful.

Yeah, you are absolutely right.  Let's not pile another difference
in the linux-TEST-vars that is already a kitchen sink.

Thanks for a quick dose of sanity.

Re: [PATCH 05/13] credential: gate new fields on capability

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1151 bytes --]

On 2024-03-28 at 10:20:53, Jeff King wrote:
> I think this "|| return" needs to be "|| exit 0" or similar. The Windows
> CI jobs fail with:
> 
>   --- a/expect-stderr
>   +++ b/stderr
>   @@ -2,3 +2,4 @@ verbatim-cred: get
>    verbatim-cred: capability[]=authtype
>    verbatim-cred: protocol=http
>    verbatim-cred: host=example.com
>   +D:\a\git\git\t\trash directory.t0300-credentials\git-credential-verbatim-cred: line 10: return: can only `return' from a function or sourced script
> 
> (actually if you count the line numbers, I think this particular case is
> the similar "|| return" added to the script later, but both should be
> fixed).
> 
> It doesn't show up elsewhere because only bash complains, but not dash.
> Even running the test script with bash isn't enough, because
> write_script uses $SHELL_PATH under the hood. But building with "make
> SHELL_PATH=/bin/bash test" shows the problem on other platforms.

I'll definitely make that change.  I run Debian, and I've left the
default dash as /bin/sh because it's faster, so I didn't notice.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 13/13] credential: add support for multistage credential rounds

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2102 bytes --]

On 2024-03-28 at 08:00:00, M Hickford wrote:
> Is this design sufficiently flexible for OAuth DPoP (RFC 9449), or at
> least to make it work in future?
> 
> OAuth 2.0 Demonstrating Proof of Possession describes "a mechanism for
> sender-constraining OAuth 2.0 tokens via a proof-of-possession
> mechanism on the application level. This mechanism allows for the
> detection of replay attacks with access and refresh tokens."
> https://www.rfc-editor.org/rfc/rfc9449.html
> 
> Popular hosts GitHub, GitLab, Bitbucket and Gitea already support
> OAuth. OAuth DPoP "provides a general defense in depth against the
> impact of unanticipated token leakage". Motivated by a 2022 GitHub
> attack involving stolen tokens
> (https://github.blog/2022-04-15-security-alert-stolen-oauth-user-tokens/),
> some hosts are already experimenting with it.
> https://lore.kernel.org/git/20230128142827.17397-1-mirth.hickford@gmail.com/
> 
> In particular, the http request has to include both Authorization and
> DPoP headers https://www.rfc-editor.org/rfc/rfc9449.html#name-the-dpop-authentication-sch.
> The latter depends on timestamp and a server-optional challenge in a
> DPoP-Nonce header.
> https://www.rfc-editor.org/rfc/rfc9449.html#name-resource-server-provided-no.

It will likely be sufficient with further extensions.  Right now, we
don't have a way to provide DPoP headers or send nonces to the client.
However, there's no reason we cannot provide that functionality in the
future via additional key/value pairs, in which case this design should
be fine.

This would have been sufficient if the OAuth working group had not added
extra additional headers that other authentication mechanisms would have
simply put (and, honestly, should have been put) in the WWW-Authenticate
and Authorization headers, but alas, we can't change it now.

Since I think this gets us at least part of the way where we need to be,
I think we should be able to keep it for now and implement the extra
support for DPoP later.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 00/13] Support for arbitrary schemes in credentials

[M Hickford]

On Sun, 24 Mar 2024 at 01:13, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> Right now, HTTP authentication in Git is mostly limited to approaches
> that require a username and password or are Kerberos (GSSAPI).  In
> addition, we effectively require that libcurl (or, for other software,
> such as Git LFS, using the credential helper, that HTTP library) knows
> how to implement the authentication scheme.
>
> However, this poses two sets of problems.  First, some sites, such as
> Azure DevOps, want to use Bearer authentication, which we don't support.

Git's lack of support for Bearer auth is also a frustration for OAuth
-- OAuth tokens are intended as Bearer tokens. As a workaround,
popular Git hosts support Basic auth with an OAuth token as password.
For certain hosts, this requires a magic username such as 'oauth2'
(GitLab) or 'x-token-auth' (Bitbucket). This is problematic if the
user has multiple accounts, or a different username stored in
credential.username or the remote URL.

> This is implemented using `http.extraHeader`, which is not a secure way
> to store credentials, since our credential helper protocol does not
> support this functionality.

Config key http.extraHeader is particularly unsuitable to hold
short-lived Bearer tokens. For example, GitLab's OAuth tokens expire
after 2 hours.

>
> In addition, other tools using the credential helper protocol do not
> support the variety of authentication mechanisms that Git does.
> Specifically, making NTLM function in a useful way on Windows is
> nontrivial and requires extensive integration and testing with C code,
> and because of this difficulty and the fact that NTLM uses cryptography
> known to be insecure since 1995, there is often little interest in
> implementing this support outside of libcurl. However, it would be
> helpful if people who want to use it can still use it.
>
> This series introduces new functionality to the credential helper
> protocol that allows helpers to produce credentials for arbitrary HTTP
> authentication schemes using the `authtype` and `credential`[0] fields.

Exciting! I drafted a patch to make git-credential-oauth generate
Bearer credentials
https://github.com/hickford/git-credential-oauth/pull/51 . Tested
together with your patch, it works for host bitbucket.org. Though of
course it isn't practical while storage helpers (such as
credential-cache, credential-wincred, credential-libsecret and
credential-osxkeychain) are unable to store the new attributes -- the
user has to complete the OAuth flow for every authentication.

> This allows a suitable credential helper to send Bearer credentials or
> any other standard or custom authentication scheme.  (It may be able to
> be extended to other functionality in the future, such as
> git-send-email, to implement custom SASL functionality, and due care has
> been taken to make the protocol adequately flexible for that purpose.)
>
> In addition, the protocol is also expanded to include per-helper state
> and multi-legged authentication (the former is effectively required for
> the latter).  The per-helper state can be useful to help credential
> helpers identify where the credential is stored, or any other
> information necessary.  Because NTLM and Negotiate (Kerberos/wrapped
> NTLM) require two rounds of authentication, the multi-legged
> authentication support along with per-helper state allows the helper to
> support these authentication methods without Git or other clients having
> to be aware of how they work.  (This would also be useful for SASL, as
> mentioned above.)
>
> This series introduces a capability mechanism to announce this
> functionality, which allows a helper to provide a username and password
> on older versions of Git while supporting more advanced functionality on
> newer versions.  (This is especially important on Azure DevOps, where
> NTLM uses a username and password but Basic or Bearer can use a personal
> access token.)  It is also designed such that extremely simple
> credential helpers, such as the shell one-liner in the Git FAQ that
> reads from the environment, don't accidentally claim to support
> functionality they don't offer.




>
> In addition, there is documentation for the expanded protocol, although
> none of the built-in helpers have been updated (that will be a future
> series for those for which it's possible).
>
> My personal interest here is getting credentials out of config files
> with `http.extraHeader` (which a future series will produce a warning
> for) and also allowing Git LFS to support Digest and NTLM with a
> suitable credential helper.  Git LFS used to support NTLM using custom
> code (because the Go standard library does not), but it was found to be
> broken in lots of ways on Windows, and nobody with a Windows system
> wanted to fix it or support it, so we removed it.  However, there are
> still some people who do want to use it, so allowing them to use a
> custom credential helper they maintain themselves seems like the best
> way forward.  Despite the advantages of this series for Azure DevOps, I
> have no personal or professional stake in their product; my only
> interest is the general one in whether their users can securely store
> credentials.  I believe the changes here are of general advantage to the
> Git userbase in a variety of ways such that the goal of this series
> should be uncontroversial.
>
> Feedback on any portion of this series is of course welcome.
>
> [0] A name different from `password` was explicitly chosen to avoid
> confusion from less capable protocol helpers so that they don't
> accidentally send invalid data.  This does have the downside that
> credential helpers must learn a new field to not log, but that should be
> generally easy to fix in most cases.
>
> brian m. carlson (13):
>   credential: add an authtype field
>   remote-curl: reset headers on new request
>   http: use new headers for each object request
>   credential: add a field for pre-encoded credentials
>   credential: gate new fields on capability
>   docs: indicate new credential protocol fields
>   http: add support for authtype and credential
>   credential: add an argument to keep state
>   credential: enable state capability
>   docs: set a limit on credential line length
>   t5563: refactor for multi-stage authentication
>   strvec: implement swapping two strvecs
>   credential: add support for multistage credential rounds
>
>  Documentation/git-credential.txt   |  59 +++++-
>  builtin/credential-cache--daemon.c |   2 +-
>  builtin/credential-store.c         |   2 +-
>  builtin/credential.c               |   7 +-
>  credential.c                       | 114 ++++++++++-
>  credential.h                       |  69 ++++++-
>  http.c                             | 128 +++++++-----
>  http.h                             |   5 +
>  imap-send.c                        |   2 +-
>  remote-curl.c                      |  14 +-
>  strvec.c                           |   7 +
>  strvec.h                           |   5 +
>  t/lib-httpd/nph-custom-auth.sh     |  17 +-
>  t/t0300-credentials.sh             | 136 ++++++++++++-
>  t/t5563-simple-http-auth.sh        | 308 +++++++++++++++++++++++++----
>  15 files changed, 760 insertions(+), 115 deletions(-)
>

Re: [PATCH 00/13] Support for arbitrary schemes in credentials

[M Hickford]

On Sun, 24 Mar 2024 at 01:13, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> Right now, HTTP authentication in Git is mostly limited to approaches
> that require a username and password or are Kerberos (GSSAPI).  In
> addition, we effectively require that libcurl (or, for other software,
> such as Git LFS, using the credential helper, that HTTP library) knows
> how to implement the authentication scheme.
>
> However, this poses two sets of problems.  First, some sites, such as
> Azure DevOps, want to use Bearer authentication, which we don't support.
> This is implemented using `http.extraHeader`, which is not a secure way
> to store credentials, since our credential helper protocol does not
> support this functionality.
>
> In addition, other tools using the credential helper protocol do not
> support the variety of authentication mechanisms that Git does.
> Specifically, making NTLM function in a useful way on Windows is
> nontrivial and requires extensive integration and testing with C code,
> and because of this difficulty and the fact that NTLM uses cryptography
> known to be insecure since 1995, there is often little interest in
> implementing this support outside of libcurl. However, it would be
> helpful if people who want to use it can still use it.
>
> This series introduces new functionality to the credential helper
> protocol that allows helpers to produce credentials for arbitrary HTTP
> authentication schemes using the `authtype` and `credential`[0] fields.
> This allows a suitable credential helper to send Bearer credentials or
> any other standard or custom authentication scheme.  (It may be able to
> be extended to other functionality in the future, such as
> git-send-email, to implement custom SASL functionality, and due care has
> been taken to make the protocol adequately flexible for that purpose.)
>
> In addition, the protocol is also expanded to include per-helper state
> and multi-legged authentication (the former is effectively required for
> the latter).  The per-helper state can be useful to help credential
> helpers identify where the credential is stored, or any other
> information necessary.  Because NTLM and Negotiate (Kerberos/wrapped
> NTLM) require two rounds of authentication, the multi-legged
> authentication support along with per-helper state allows the helper to
> support these authentication methods without Git or other clients having
> to be aware of how they work.  (This would also be useful for SASL, as
> mentioned above.)
>
> This series introduces a capability mechanism to announce this
> functionality, which allows a helper to provide a username and password
> on older versions of Git while supporting more advanced functionality on
> newer versions.  (This is especially important on Azure DevOps, where
> NTLM uses a username and password but Basic or Bearer can use a personal
> access token.)  It is also designed such that extremely simple
> credential helpers, such as the shell one-liner in the Git FAQ that
> reads from the environment, don't accidentally claim to support
> functionality they don't offer.
>
> In addition, there is documentation for the expanded protocol, although
> none of the built-in helpers have been updated (that will be a future
> series for those for which it's possible).

If you've time, I think it would be instructive to add (conditional)
tests to libcredential and support in at least one storage helper,
such as credential-cache.

This would clarify the behaviour in subtle cases involving expiry and
multiple helpers.

>
> My personal interest here is getting credentials out of config files
> with `http.extraHeader` (which a future series will produce a warning
> for) and also allowing Git LFS to support Digest and NTLM with a
> suitable credential helper.  Git LFS used to support NTLM using custom
> code (because the Go standard library does not), but it was found to be
> broken in lots of ways on Windows, and nobody with a Windows system
> wanted to fix it or support it, so we removed it.  However, there are
> still some people who do want to use it, so allowing them to use a
> custom credential helper they maintain themselves seems like the best
> way forward.  Despite the advantages of this series for Azure DevOps, I
> have no personal or professional stake in their product; my only
> interest is the general one in whether their users can securely store
> credentials.  I believe the changes here are of general advantage to the
> Git userbase in a variety of ways such that the goal of this series
> should be uncontroversial.
>
> Feedback on any portion of this series is of course welcome.
>
> [0] A name different from `password` was explicitly chosen to avoid
> confusion from less capable protocol helpers so that they don't
> accidentally send invalid data.  This does have the downside that
> credential helpers must learn a new field to not log, but that should be
> generally easy to fix in most cases.
>
> brian m. carlson (13):
>   credential: add an authtype field
>   remote-curl: reset headers on new request
>   http: use new headers for each object request
>   credential: add a field for pre-encoded credentials
>   credential: gate new fields on capability
>   docs: indicate new credential protocol fields
>   http: add support for authtype and credential
>   credential: add an argument to keep state
>   credential: enable state capability
>   docs: set a limit on credential line length
>   t5563: refactor for multi-stage authentication
>   strvec: implement swapping two strvecs
>   credential: add support for multistage credential rounds
>
>  Documentation/git-credential.txt   |  59 +++++-
>  builtin/credential-cache--daemon.c |   2 +-
>  builtin/credential-store.c         |   2 +-
>  builtin/credential.c               |   7 +-
>  credential.c                       | 114 ++++++++++-
>  credential.h                       |  69 ++++++-
>  http.c                             | 128 +++++++-----
>  http.h                             |   5 +
>  imap-send.c                        |   2 +-
>  remote-curl.c                      |  14 +-
>  strvec.c                           |   7 +
>  strvec.h                           |   5 +
>  t/lib-httpd/nph-custom-auth.sh     |  17 +-
>  t/t0300-credentials.sh             | 136 ++++++++++++-
>  t/t5563-simple-http-auth.sh        | 308 +++++++++++++++++++++++++----
>  15 files changed, 760 insertions(+), 115 deletions(-)
>

Re: [PATCH 06/13] docs: indicate new credential protocol fields

[M Hickford]

On Mon, 25 Mar 2024 at 23:37, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> On 2024-03-25 at 23:16:09, M Hickford wrote:
> > > +`authtype`::
> > > +   This indicates that the authentication scheme in question should be used.
> > > +   Common values for HTTP and HTTPS include `basic`, `digest`, and `ntlm`,
> > > +   although the latter two are insecure and should not be used.  If `credential`
> > > +   is used, this may be set to an arbitrary string suitable for the protocol in
> > > +   question (usually HTTP).
> >
> > How about adding 'bearer' to this list? Popular hosts Bitbucket
> > https://bitbucket.org and Gitea/Forgejo (such as https://codeberg.org)
> > support Bearer auth with OAuth tokens.
>
> Sure, I can do that.
>
> > > ++
> > > +This value should not be sent unless the appropriate capability (see below) is
> > > +provided on input.
> > > +
> > > +`credential`::
> > > +   The pre-encoded credential, suitable for the protocol in question (usually
> > > +   HTTP).  If this key is sent, `authtype` is mandatory, and `username` and
> > > +   `password` are not used.
> >
> > A credential protocol attribute named 'credential' is confusing. How
> > about 'authorization' since it determines the HTTP Authorization
> > header? This detail is surely worth mentioning too.

Would it be accurate to add "For HTTP, Git concatenates the authtype
and credential attributes to determine the Authorization header"?

>
> I don't want this to be very specific to HTTP, so I don't think that's a
> great name.  As I mentioned in the cover letter, I might well extend
> this to IMAP and SMTP for our mail handling in the future, and that name
> wouldn't work well there.

Good point, you've dissuaded me against 'authorization'.

>
> I named it `credential` because, well, it's the credential that's used
> in the protocol.  I feel like saying that the field represents "the
> authorization" sounds unnatural.  It's not wrong, per se, but it sounds
> confusing.

We already use 'credential' to describe the whole collection of
attributes, as in "The credential is split into a set of named
attributes".

>
> I'm open to other ideas if you or others have them, but between these
> two, I think I'd prefer to stick with `credential`.

Ideas anyone?


> --
> brian m. carlson (they/them or he/him)
> Toronto, Ontario, CA

Re: [PATCH 06/13] docs: indicate new credential protocol fields

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 294 bytes --]

On 2024-03-30 at 13:00:00, M Hickford wrote:
> Would it be accurate to add "For HTTP, Git concatenates the authtype
> and credential attributes to determine the Authorization header"?

Yes, I think that would be accurate.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 13/13] credential: add support for multistage credential rounds

[M Hickford]

On Thu, 28 Mar 2024 at 21:53, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> On 2024-03-28 at 08:00:00, M Hickford wrote:
> > Is this design sufficiently flexible for OAuth DPoP (RFC 9449), or at
> > least to make it work in future?
> >
> > OAuth 2.0 Demonstrating Proof of Possession describes "a mechanism for
> > sender-constraining OAuth 2.0 tokens via a proof-of-possession
> > mechanism on the application level. This mechanism allows for the
> > detection of replay attacks with access and refresh tokens."
> > https://www.rfc-editor.org/rfc/rfc9449.html
> >
> > Popular hosts GitHub, GitLab, Bitbucket and Gitea already support
> > OAuth. OAuth DPoP "provides a general defense in depth against the
> > impact of unanticipated token leakage". Motivated by a 2022 GitHub
> > attack involving stolen tokens
> > (https://github.blog/2022-04-15-security-alert-stolen-oauth-user-tokens/),
> > some hosts are already experimenting with it.
> > https://lore.kernel.org/git/20230128142827.17397-1-mirth.hickford@gmail.com/
> >
> > In particular, the http request has to include both Authorization and
> > DPoP headers https://www.rfc-editor.org/rfc/rfc9449.html#name-the-dpop-authentication-sch.
> > The latter depends on timestamp and a server-optional challenge in a
> > DPoP-Nonce header.
> > https://www.rfc-editor.org/rfc/rfc9449.html#name-resource-server-provided-no.
>
> It will likely be sufficient with further extensions.  Right now, we
> don't have a way to provide DPoP headers or send nonces to the client.
> However, there's no reason we cannot provide that functionality in the
> future via additional key/value pairs, in which case this design should
> be fine.
>
> This would have been sufficient if the OAuth working group had not added
> extra additional headers that other authentication mechanisms would have
> simply put (and, honestly, should have been put) in the WWW-Authenticate
> and Authorization headers, but alas, we can't change it now.
>
> Since I think this gets us at least part of the way where we need to be,
> I think we should be able to keep it for now and implement the extra
> support for DPoP later.

Fantastic, thanks for considering this.

I look forward to sharing a test Git remote with OAuth DPoP when I can
figure out how.

> --
> brian m. carlson (they/them or he/him)
> Toronto, Ontario, CA

Re: [PATCH 08/13] credential: add an argument to keep state

[mirth hickford]

On Sun, Mar 24, 2024 at 1:13 AM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> Until now, our credential code has mostly deal with usernames and
> passwords and we've let libcurl deal with the variant of authentication
> to be used.  However, now that we have the credential value, the
> credential helper can take control of the authentication, so the value
> provided might be something that's generated, such as a Digest hash
> value.
>
> In such a case, it would be helpful for a credential helper that gets an
> erase or store command to be able to keep track of an identifier for the
> original secret that went into the computation.  Furthermore, some types
> of authentication, such as NTLM and Kerberos, actually need two round
> trips to authenticate, which will require that the credential helper
> keep some state.
>
> In order to allow for these use cases and others, allow storing state in
> a field called "state[]".  This value is passed back to the credential
> helper that created it, which avoids confusion caused by parsing values
> from different helpers.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  Documentation/git-credential.txt | 29 ++++++++++++++++++-----------
>  credential.c                     | 20 +++++++++++++++++---
>  credential.h                     |  7 +++++++
>  t/t0300-credentials.sh           | 29 +++++++++++++++++++++++++++++
>  4 files changed, 71 insertions(+), 14 deletions(-)
>
> diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
> index f3ed3a82fa..ef30c89c00 100644
> --- a/Documentation/git-credential.txt
> +++ b/Documentation/git-credential.txt
> @@ -196,6 +196,15 @@ provided on input.
>  This value should not be sent unless the appropriate capability (see below) is
>  provided on input.
>
> +`state[]`::
> +       This value provides an opaque state that will be passed back to this helper
> +       if it is called again.  Each different credential helper may specify this
> +       once.  The value should include a prefix unique to the credential helper and
> +       should ignore values that don't match its prefix.

Does Git ever populate state[] in 'store' or 'erase' requests,  or
only 'get' requests? It might be worthwhile to spell this out.

This seems somewhat different to other multi-valued attributes,
particularly the "set at most one value" constraint. As an
alternative, how about a single-valued attribute stored independently
for each helper (vector length equal to the number of configured
helpers)? Then in repeat requests send the "nth state to the nth
helper". This would avoid the complexity of the prefix mechanism.

> ++
> +This value should not be sent unless the appropriate capability (see below) is
> +provided on input.
> +
>  `wwwauth[]`::
>
>         When an HTTP response is received by Git that includes one or more
> @@ -208,18 +217,16 @@ they appear in the HTTP response. This attribute is 'one-way' from Git
>  to pass additional information to credential helpers.
>
>  `capability[]`::
> -       This signals that the caller supports the capability in question.
> -       This can be used to provide better, more specific data as part of the
> -       protocol.
> +  This signals that Git, or the helper, as appropriate, supports the
> +       capability in question.  This can be used to provide better, more specific
> +       data as part of the protocol.
>  +
> -The only capability currently supported is `authtype`, which indicates that the
> -`authtype` and `credential` values are understood.  It is not obligatory to use
> -these values in such a case, but they should not be provided without this
> -capability.
> -+
> -Callers of `git credential` and credential helpers should emit the
> -capabilities they support unconditionally, and Git will gracefully
> -handle passing them on.
> +There are two currently supported capabilities.  The first is `authtype`, which
> +indicates that the `authtype` and `credential` values are understood.  The
> +second is `state`, which indicates that the `state[]` value is understood.
>
> +
> +It is not obligatory to use the additional features just because the capability
> +is supported, but they should not be provided without this capability.
>
>  Unrecognised attributes and capabilities are silently discarded.
>
> diff --git a/credential.c b/credential.c
> index f2a26b8672..0cd7dd2a00 100644
> --- a/credential.c
> +++ b/credential.c
> @@ -30,6 +30,7 @@ void credential_clear(struct credential *c)
>         free(c->authtype);
>         string_list_clear(&c->helpers, 0);
>         strvec_clear(&c->wwwauth_headers);
> +       strvec_clear(&c->state_headers);
>
>         credential_init(c);
>  }
> @@ -286,8 +287,13 @@ int credential_read(struct credential *c, FILE *fp, int op_type)
>                         c->path = xstrdup(value);
>                 } else if (!strcmp(key, "wwwauth[]")) {
>                         strvec_push(&c->wwwauth_headers, value);
> -               } else if (!strcmp(key, "capability[]") && !strcmp(value, "authtype")) {
> -                       credential_set_capability(&c->capa_authtype, op_type);
> +               } else if (!strcmp(key, "state[]")) {
> +                       strvec_push(&c->state_headers, value);
> +               } else if (!strcmp(key, "capability[]")) {
> +                       if (!strcmp(value, "authtype"))
> +                               credential_set_capability(&c->capa_authtype, op_type);
> +                       else if (!strcmp(value, "state"))
> +                               credential_set_capability(&c->capa_state, op_type);
>                 } else if (!strcmp(key, "password_expiry_utc")) {
>                         errno = 0;
>                         c->password_expiry_utc = parse_timestamp(value, NULL, 10);
> @@ -329,8 +335,12 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
>
>  void credential_write(const struct credential *c, FILE *fp, int op_type)
>  {
> -       if (credential_has_capability(&c->capa_authtype, op_type)) {
> +       if (credential_has_capability(&c->capa_authtype, op_type))
>                 credential_write_item(fp, "capability[]", "authtype", 0);
> +       if (credential_has_capability(&c->capa_state, op_type))
> +               credential_write_item(fp, "capability[]", "state", 0);
> +
> +       if (credential_has_capability(&c->capa_authtype, op_type)) {
>                 credential_write_item(fp, "authtype", c->authtype, 0);
>                 credential_write_item(fp, "credential", c->credential, 0);
>         }
> @@ -347,6 +357,10 @@ void credential_write(const struct credential *c, FILE *fp, int op_type)
>         }
>         for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
>                 credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
> +       if (credential_has_capability(&c->capa_state, op_type)) {
> +               for (size_t i = 0; i < c->state_headers.nr; i++)
> +                       credential_write_item(fp, "state[]", c->state_headers.v[i], 0);
> +       }
>  }
>
>  static int run_credential_helper(struct credential *c,
> diff --git a/credential.h b/credential.h
> index 2051d04c5a..e2021455fe 100644
> --- a/credential.h
> +++ b/credential.h
> @@ -142,6 +142,11 @@ struct credential {
>          */
>         struct strvec wwwauth_headers;
>
> +       /**
> +        * A `strvec` of state headers from credential helpers.
> +        */
> +       struct strvec state_headers;
> +
>
>         /**
>          * Internal use only. Keeps track of if we previously matched against a
>          * WWW-Authenticate header line in order to re-fold future continuation
> @@ -156,6 +161,7 @@ struct credential {
>                  username_from_proto:1;
>
>         struct credential_capability capa_authtype;
> +       struct credential_capability capa_state;
>
>         char *username;
>         char *password;
> @@ -177,6 +183,7 @@ struct credential {
>         .helpers = STRING_LIST_INIT_DUP, \
>         .password_expiry_utc = TIME_MAX, \
>         .wwwauth_headers = STRVEC_INIT, \
> +       .state_headers = STRVEC_INIT, \
>  }
>
>  /* Initialize a credential structure, setting all fields to empty. */
> diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
> index 8477108b28..aa56e0dc84 100755
> --- a/t/t0300-credentials.sh
> +++ b/t/t0300-credentials.sh
> @@ -46,9 +46,12 @@ test_expect_success 'setup helper scripts' '
>         credential=$1; shift
>         . ./dump
>         echo capability[]=authtype
> +       echo capability[]=state
>         test -z "${capability##*authtype*}" || return
>         test -z "$authtype" || echo authtype=$authtype
>         test -z "$credential" || echo credential=$credential
> +       test -z "${capability##*state*}" || return
> +       echo state[]=verbatim-cred:foo
>         EOF
>
>         write_script git-credential-verbatim-with-expiry <<-\EOF &&
> @@ -99,6 +102,29 @@ test_expect_success 'credential_fill invokes helper with credential' '
>         EOF
>  '
>
> +test_expect_success 'credential_fill invokes helper with credential and state' '
> +       check fill "verbatim-cred Bearer token" <<-\EOF
> +       capability[]=authtype
> +       capability[]=state
> +       protocol=http
> +       host=example.com
> +       --
> +       capability[]=authtype
> +       capability[]=state
> +       authtype=Bearer
> +       credential=token
> +       protocol=http
> +       host=example.com
> +       state[]=verbatim-cred:foo
> +       --
> +       verbatim-cred: get
> +       verbatim-cred: capability[]=authtype
> +       verbatim-cred: capability[]=state
> +       verbatim-cred: protocol=http
> +       verbatim-cred: host=example.com
> +       EOF
> +'
> +
>
>  test_expect_success 'credential_fill invokes multiple helpers' '
>         check fill useless "verbatim foo bar" <<-\EOF
> @@ -122,6 +148,7 @@ test_expect_success 'credential_fill invokes multiple helpers' '
>  test_expect_success 'credential_fill response does not get capabilities when helpers are incapable' '
>         check fill useless "verbatim foo bar" <<-\EOF
>         capability[]=authtype
> +       capability[]=state
>         protocol=http
>         host=example.com
>         --
> @@ -132,10 +159,12 @@ test_expect_success 'credential_fill response does not get capabilities when hel
>         --
>         useless: get
>         useless: capability[]=authtype
> +       useless: capability[]=state
>         useless: protocol=http
>         useless: host=example.com
>         verbatim: get
>         verbatim: capability[]=authtype
> +       verbatim: capability[]=state
>         verbatim: protocol=http
>         verbatim: host=example.com
>         EOF

Re: [PATCH 08/13] credential: add an argument to keep state

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3500 bytes --]

On 2024-04-01 at 21:05:28, mirth hickford wrote:
> On Sun, Mar 24, 2024 at 1:13 AM brian m. carlson
> <sandals@crustytoothpaste.net> wrote:
> >
> > Until now, our credential code has mostly deal with usernames and
> > passwords and we've let libcurl deal with the variant of authentication
> > to be used.  However, now that we have the credential value, the
> > credential helper can take control of the authentication, so the value
> > provided might be something that's generated, such as a Digest hash
> > value.
> >
> > In such a case, it would be helpful for a credential helper that gets an
> > erase or store command to be able to keep track of an identifier for the
> > original secret that went into the computation.  Furthermore, some types
> > of authentication, such as NTLM and Kerberos, actually need two round
> > trips to authenticate, which will require that the credential helper
> > keep some state.
> >
> > In order to allow for these use cases and others, allow storing state in
> > a field called "state[]".  This value is passed back to the credential
> > helper that created it, which avoids confusion caused by parsing values
> > from different helpers.
> >
> > Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> > ---
> >  Documentation/git-credential.txt | 29 ++++++++++++++++++-----------
> >  credential.c                     | 20 +++++++++++++++++---
> >  credential.h                     |  7 +++++++
> >  t/t0300-credentials.sh           | 29 +++++++++++++++++++++++++++++
> >  4 files changed, 71 insertions(+), 14 deletions(-)
> >
> > diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
> > index f3ed3a82fa..ef30c89c00 100644
> > --- a/Documentation/git-credential.txt
> > +++ b/Documentation/git-credential.txt
> > @@ -196,6 +196,15 @@ provided on input.
> >  This value should not be sent unless the appropriate capability (see below) is
> >  provided on input.
> >
> > +`state[]`::
> > +       This value provides an opaque state that will be passed back to this helper
> > +       if it is called again.  Each different credential helper may specify this
> > +       once.  The value should include a prefix unique to the credential helper and
> > +       should ignore values that don't match its prefix.
> 
> Does Git ever populate state[] in 'store' or 'erase' requests,  or
> only 'get' requests? It might be worthwhile to spell this out.

Yes, it's populated with whatever the last state value was from `get`.

> This seems somewhat different to other multi-valued attributes,
> particularly the "set at most one value" constraint. As an
> alternative, how about a single-valued attribute stored independently
> for each helper (vector length equal to the number of configured
> helpers)? Then in repeat requests send the "nth state to the nth
> helper". This would avoid the complexity of the prefix mechanism.

I originally tried that approach, but if you have external callers of
`git credential` (like Git LFS), that doesn't work, since you need to
make two separate calls: one (with `get`) to fetch the credentials that
returns multiple state values, and one (with `store` or `erase`) that
sends the data back to accept or reject the credentials.  Since there's
no internal state in Git between the two calls, it's not possible to
only send certain data to certain helpers.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 05/13] credential: gate new fields on capability

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 6843 bytes --]

On Wed, Mar 27, 2024 at 09:33:35PM +0000, brian m. carlson wrote:
> On 2024-03-27 at 08:02:39, Patrick Steinhardt wrote:
> > On Sun, Mar 24, 2024 at 01:12:53AM +0000, brian m. carlson wrote:
> > > +static int credential_has_capability(const struct credential_capability *capa, int op_type)
> > > +{
> > > +	/*
> > > +	 * We're checking here if each previous step indicated that we had the
> > > +	 * capability.  If it did, then we want to pass it along; conversely, if
> > > +	 * it did not, we don't want to report that to our caller.
> > > +	 */
> > > +	switch (op_type) {
> > > +	case CREDENTIAL_OP_HELPER:
> > > +		return capa->request_initial;
> > > +	case CREDENTIAL_OP_RESPONSE:
> > > +		return capa->request_initial && capa->request_helper;
> > > +	default:
> > > +		return 0;
> > > +	}
> > > +}
> > 
> > I think I'm missing the bigger picture here, so please bear with me.
> > 
> > What you provide here is simply an `op_type` that indicates the phase we
> > are currently in and thus allows us to check whether all of the
> > preceding phases had the capability set. But to me it seems like a phase
> > and the actual capability should be different things. So why is it that
> > the capability seems to be a mere boolean value instead of something
> > like a bitfield indicating whether a specific capability is supported or
> > not? Or is all of this infra really only to support a single capability,
> > namely the credential capability?
> > 
> > I'm mostly coming from the angle of how capabilities work with remote
> > helpers. When asked, the helper will announce a set of capabilities that
> > it supports, e.g. "capabilities stateless-connect". So from thereon the
> > client of the helper knows that it can assume "stateless-connect" to be
> > understood by the helper.
> > 
> > I would have expected capabilities to work similarly for the credential
> > helper, where it announces "I know to handle pre-encoded credentials".
> > But given that I have basically no clue at all for how the credential
> > helper works there may very well be good reasons why things work so
> > differently here.
> 
> Let me explain a little bit.  There are two possible flows that we can
> have for a credential request:
> 
>   git-credential input -> credential.c -> helper -> credential.c -> git-credential output
> 
>   git-http-backend -> credential.c -> helper -> credential.c -> git-http-backend
> 
> Let's look at the first one (which might, say, come from Git LFS or
> another external tool), but the second one works similarly.  When we get
> a request from `git credential fill`, we need to know first whether the
> requester supports the capability.  If we're using an external tool from
> last decade, it's not going to do so.
> 
> If it _does_ support that, then we want to pass that along to the
> helper, but if it doesn't, we don't.  That's because if the caller
> doesn't support `credential` and `authtype`, the helper might
> legitimately want to provide a username and password (or token) instead,
> knowing that that's more likely to work.
> 
> Similarly, in the final response, we want to indicate to the external
> caller whether the capability was in fact supported.  That's useful to
> know in case we want to pass the response back to `git credential
> store`, and it also discloses functionality about what the credential
> helper in use supports.
> 
> We can't assume that the helper does or doesn't support the same
> capabilities as Git because it might come from outside Git (e.g., Git
> Credential Manager Core, or a site-specific credential helper) or it
> just might not be capable of storing or handling that kind of
> credential.  By not making the assumption that the helper is implicitly
> capable, we allow users to continue to use very simple shell scripts as
> credential helpers.

The intent of this is quite clear to me, but thanks for re-explaining
the bigger picture :)

> As to why this functionality exists, it exists to support the two new
> capabilities in this series, `credential` and `state`.  A pie in the sky
> goal for the future is to support additional request signing
> functionality, so it might learn things like method, URI, and TLS
> channel binding info, which would be an additional capability.  (I might
> implement that, or I might not.)  All of those are boolean: they either
> are supported, or not.  If folks in the future want to expose
> non-boolean capabilities, I don't think that should be a problem.

I think you misunderstood my confusion. I didn't meant to say that there
should be non-boolean capabilities. I was rather missing the picture of
how exactly you can advertise multiple capabilities with the infra that
currently exists, and why the infra supports per-phase capabilities.

Basically, what I would have expected is a protocol where both Git and
the credential helper initially did a single "handshake" that also
announces capabilities. So something like:

    HELPER: capability foobar
    HELPER: capability barfoo
       GIT: capability foobar

Git would only acknowledge capabilities that it both understands and
that have been announced by the helper. So at the end of this both have
agreed on a single capability "foobar".

This is roughly how the remote helper capability subsystem works. What
this patch is introducing seems quite a bit more complicated than that
though because we have "staged" capabilities. I assume there is good
reason for this complexity, but I didn't yet manage to figure out the
reasoning behind it.

To ask more specifically: why would one side ever announce a capability
in phase 1, but not in phase 2? Is the reason that capabilities are in
fact tied to credentials?

Patrick

> > > +/*
> > > + * These values define the kind of operation we're performing and the
> > > + * capabilities at each stage.  The first is either an external request (via git
> > > + * credential fill) or an internal request (e.g., via the HTTP) code.  The
> > > + * second is the call to the credential helper, and the third is the response
> > > + * we're providing.
> > > + *
> > > + * At each stage, we will emit the capability only if the previous stage
> > > + * supported it.
> > > + */
> > > +#define CREDENTIAL_OP_INITIAL  1
> > > +#define CREDENTIAL_OP_HELPER   2
> > > +#define CREDENTIAL_OP_RESPONSE 3
> > 
> > Is there any specific reason why you're using defines instead of an enum
> > here? I think the latter would be more self-explanatory when you see
> > that functions take `enum credential_op` as input instead of an `int`.
> 
> I think an enum would be a nice improvement.  I'll include that in a
> reroll.
> -- 
> brian m. carlson (they/them or he/him)
> Toronto, Ontario, CA



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 00/13] Support for arbitrary schemes in credentials

[Calvin Wan]

Hi Brian,

While I personally do not know the specifics of how Git authentication
works at Google, I am passing this series along to the team that does
own Git authentication (adding Jackson to this reply).

"brian m. carlson" <sandals@crustytoothpaste.net> writes:
> Right now, HTTP authentication in Git is mostly limited to approaches
> that require a username and password or are Kerberos (GSSAPI).  In
> addition, we effectively require that libcurl (or, for other software,
> such as Git LFS, using the credential helper, that HTTP library) knows
> how to implement the authentication scheme.
> 
> However, this poses two sets of problems.  First, some sites, such as
> Azure DevOps, want to use Bearer authentication, which we don't support.
> This is implemented using `http.extraHeader`, which is not a secure way
> to store credentials, since our credential helper protocol does not
> support this functionality.

My first thought was if using `http.extraHeader` is insecure as you
claim and we use that internally, then how do we ensure that it is
secure? Or are you claiming that using `http.extraHeader` out of the box
without an additional security layer is insecure?

Thanks,
Calvin

Re: [PATCH 05/13] credential: gate new fields on capability

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3588 bytes --]

On 2024-04-02 at 10:04:31, Patrick Steinhardt wrote:
> The intent of this is quite clear to me, but thanks for re-explaining
> the bigger picture :)

Sorry I misunderstood what you were getting it.

> I think you misunderstood my confusion. I didn't meant to say that there
> should be non-boolean capabilities. I was rather missing the picture of
> how exactly you can advertise multiple capabilities with the infra that
> currently exists, and why the infra supports per-phase capabilities.
> 
> Basically, what I would have expected is a protocol where both Git and
> the credential helper initially did a single "handshake" that also
> announces capabilities. So something like:
> 
>     HELPER: capability foobar
>     HELPER: capability barfoo
>        GIT: capability foobar
> 
> Git would only acknowledge capabilities that it both understands and
> that have been announced by the helper. So at the end of this both have
> agreed on a single capability "foobar".
> 
> This is roughly how the remote helper capability subsystem works. What
> this patch is introducing seems quite a bit more complicated than that
> though because we have "staged" capabilities. I assume there is good
> reason for this complexity, but I didn't yet manage to figure out the
> reasoning behind it.
> 
> To ask more specifically: why would one side ever announce a capability
> in phase 1, but not in phase 2? Is the reason that capabilities are in
> fact tied to credentials?

More that they're tied to the credential helper.  For example, say we
have helpers A and B, in that order.  B is incapable, but both A and Git
understand the new authtype capability.  If we announce the capability
as in this series, we can get a new credential using the authtype
capability if A is willing to provide something to us, but we can't if A
has no credentials for us and B wants to provide them for us.

This would also be true if we used your proposal of negotiation, but
because we have external callers (e.g., Git LFS) who may invoke `git
credential fill`, which would be a separate process from `git credential
capability`, we'd still have to have some way to tell `git credential
fill` what capabilities the external caller supported.

The per-phase capabilities are such that we don't request functionality
that our callers can't use.  For example, if our external caller (phase
1) doesn't support the `authtype` credential, then we don't pass it to
the helper (phase 2), since the external caller might not be able to use
the result if we do.  If the external caller (phase 1) _does_ support
it, but the helper does not (phase 2), then we won't return the
capability as the result of `git credential fill` (phase 3), so our
external caller will know that this isn't supported.  As a practical
matter, that doesn't provide a great deal of useful information to the
caller at the moment, but it definitely could in the future (say, if we
had a capability for a certain form of data encoding).

All of this is also true for internal (e.g., git-http-backend) callers,
except that phase 1 has all the capabilities we know about automatically
set, and phase 3 stores the data in the internal structure we'll use for
the `store` and `erase` calls.

I do, however, think some way to query capabilities more generically
would be helpful, so I'll see if I can add such changes into a v2.  I
think we still need the current approach to make the use case I
mentioned work, though.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 00/13] Support for arbitrary schemes in credentials

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3275 bytes --]

On 2024-04-02 at 22:26:19, Calvin Wan wrote:
> Hi Brian,
> 
> While I personally do not know the specifics of how Git authentication
> works at Google, I am passing this series along to the team that does
> own Git authentication (adding Jackson to this reply).
> 
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> > Right now, HTTP authentication in Git is mostly limited to approaches
> > that require a username and password or are Kerberos (GSSAPI).  In
> > addition, we effectively require that libcurl (or, for other software,
> > such as Git LFS, using the credential helper, that HTTP library) knows
> > how to implement the authentication scheme.
> > 
> > However, this poses two sets of problems.  First, some sites, such as
> > Azure DevOps, want to use Bearer authentication, which we don't support.
> > This is implemented using `http.extraHeader`, which is not a secure way
> > to store credentials, since our credential helper protocol does not
> > support this functionality.
> 
> My first thought was if using `http.extraHeader` is insecure as you
> claim and we use that internally, then how do we ensure that it is
> secure? Or are you claiming that using `http.extraHeader` out of the box
> without an additional security layer is insecure?

Storing plaintext credentials on disk is just not a good idea, and it's
not a secure way to store them.  This is why `.netrc` is a less than
great idea, but for Git, it's also possible to have a shared repository
where information in `.git/config` can leak, and sometimes people also
just expose Git repositories accidentally over HTTP (say, on websites)
and leak all of their config.  Sometimes people put `http.extraheader`
in `~/.gitconfig` and then check it into their public dotfiles, and then
push it to GitHub, for example.

It's a little less of a problem if it's a personal laptop and nobody
else uses it, but it's still a lot easier to accidentally expose an
arbitrary file or for an attack to exfiltrate an existing file (just
through a bug in existing software) than it is to necessarily execute
the arbitrary code necessary to read the data out of the system
credential store.

`http.extraheader` for Authorization headers usually necessitates that
the data is either stored in the config file or passed on the command
line, and that's why it's insecure.  Certainly, you could configure it
to read only from the environment using `--config-env` or you could
configure your system to store the data in the config only with a
single, highly restricted service account and it might be okay.

The kind of usage of `http.extraheader` that's likely to be fine is just
passing an extra header that some broken proxy needs to be satisfied,
like setting a specific language or faking a header that the proxy needs
to think Git's a web browser (since, of course, if it's not Internet
Explorer, it's insecure).  As long as you're not storing credentials or
secrets in `http.extraheader`, I have no objections.

I don't know what you're using it for at Google, but of course if it is
Authorization headers, then I'm hoping this series will help you avoid
needing to do that.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 05/13] credential: gate new fields on capability

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 3779 bytes --]

On Thu, Apr 04, 2024 at 12:39:58AM +0000, brian m. carlson wrote:
> On 2024-04-02 at 10:04:31, Patrick Steinhardt wrote:
> > The intent of this is quite clear to me, but thanks for re-explaining
> > the bigger picture :)
> 
> Sorry I misunderstood what you were getting it.
> 
> > I think you misunderstood my confusion. I didn't meant to say that there
> > should be non-boolean capabilities. I was rather missing the picture of
> > how exactly you can advertise multiple capabilities with the infra that
> > currently exists, and why the infra supports per-phase capabilities.
> > 
> > Basically, what I would have expected is a protocol where both Git and
> > the credential helper initially did a single "handshake" that also
> > announces capabilities. So something like:
> > 
> >     HELPER: capability foobar
> >     HELPER: capability barfoo
> >        GIT: capability foobar
> > 
> > Git would only acknowledge capabilities that it both understands and
> > that have been announced by the helper. So at the end of this both have
> > agreed on a single capability "foobar".
> > 
> > This is roughly how the remote helper capability subsystem works. What
> > this patch is introducing seems quite a bit more complicated than that
> > though because we have "staged" capabilities. I assume there is good
> > reason for this complexity, but I didn't yet manage to figure out the
> > reasoning behind it.
> > 
> > To ask more specifically: why would one side ever announce a capability
> > in phase 1, but not in phase 2? Is the reason that capabilities are in
> > fact tied to credentials?
> 
> More that they're tied to the credential helper.  For example, say we
> have helpers A and B, in that order.  B is incapable, but both A and Git
> understand the new authtype capability.  If we announce the capability
> as in this series, we can get a new credential using the authtype
> capability if A is willing to provide something to us, but we can't if A
> has no credentials for us and B wants to provide them for us.
> 
> This would also be true if we used your proposal of negotiation, but
> because we have external callers (e.g., Git LFS) who may invoke `git
> credential fill`, which would be a separate process from `git credential
> capability`, we'd still have to have some way to tell `git credential
> fill` what capabilities the external caller supported.
> 
> The per-phase capabilities are such that we don't request functionality
> that our callers can't use.  For example, if our external caller (phase
> 1) doesn't support the `authtype` credential, then we don't pass it to
> the helper (phase 2), since the external caller might not be able to use
> the result if we do.  If the external caller (phase 1) _does_ support
> it, but the helper does not (phase 2), then we won't return the
> capability as the result of `git credential fill` (phase 3), so our
> external caller will know that this isn't supported.  As a practical
> matter, that doesn't provide a great deal of useful information to the
> caller at the moment, but it definitely could in the future (say, if we
> had a capability for a certain form of data encoding).
> 
> All of this is also true for internal (e.g., git-http-backend) callers,
> except that phase 1 has all the capabilities we know about automatically
> set, and phase 3 stores the data in the internal structure we'll use for
> the `store` and `erase` calls.
> 
> I do, however, think some way to query capabilities more generically
> would be helpful, so I'll see if I can add such changes into a v2.  I
> think we still need the current approach to make the use case I
> mentioned work, though.

Great. Thanks a lot for your explanations!

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 00/13] Support for arbitrary schemes in credentials

[Jackson Toeniskoetter]

Just to clarify, we at Google do not use extraheader to pass along
credentials. Instead we use the .gitcookies file[1], which iiuc gets
read by the git process directly. I'm not a security expert but I
imagine the risk surface of extraheader and .gitcookies is similar.

Reading your post on
https://lore.kernel.org/git/20240324011301.1553072-1-sandals@crustytoothpaste.net/,
it's unclear to me why the credential helper protocol needs to be
updated. If the goal is to support Bearer tokens, can that not just be
implemented using extraheader? It seems like the goal of getting
credentials out of the config file can be accomplished without
updating the credential helper protocol. So is the bigger goal to
support more robust and modern auth schemes which require multiple
steps? That would be useful to Google; multi-step auth would probably
be a more elegant way for us to stop using .gitcookies than other
solutions we were considering.

[1] This is actually only for external contributors. Google employees
have a more robust authentication mechanism. Concerns have been raised
internally about our usage of gitcookies, but it hasn't been made a
priority to address because a leaked credential would not allow an
attacker to commit bad code, only read it or initiate a code review.

Re: [PATCH 00/13] Support for arbitrary schemes in credentials

[M Hickford]

On Mon, 8 Apr 2024 at 19:43, Jackson Toeniskoetter <jackdt@google.com> wrote:
>
> Just to clarify, we at Google do not use extraheader to pass along
> credentials. Instead we use the .gitcookies file[1], which iiuc gets
> read by the git process directly. I'm not a security expert but I
> imagine the risk surface of extraheader and .gitcookies is similar.

Hi Jackson. To clarify, are you describing hosts *.googlesource.com
such as https://go.googlesource.com/? It always confused me that the
'generate password' feature gives you something different to a
password.

https://www.googlesource.com/new-password
https://gerrit-review.googlesource.com/Documentation/user-upload.html

Have you tried OAuth credential helper git-credential-oauth
https://github.com/hickford/git-credential-oauth? It can authenticate
to *.googlesource.com without setup. OAuth has security advantages
over .gitcookies because the OAuth access tokens expire after 2 hours
and the OAuth refresh tokens are single use. These credentials are
stored in the user's choice of secure storage such as
git-credential-cache, git-credential-libsecret, git-credential-wincred
or git-credential-osxkeychain.

I encourage you to try it out. You'll need a local web browser because
Google's OAuth configuration currently forbids device authorization
grant for the relevant scope
https://issues.gerritcodereview.com/issues/300279941

>
> Reading your post on
> https://lore.kernel.org/git/20240324011301.1553072-1-sandals@crustytoothpaste.net/,
> it's unclear to me why the credential helper protocol needs to be
> updated. If the goal is to support Bearer tokens, can that not just be
> implemented using extraheader? It seems like the goal of getting
> credentials out of the config file can be accomplished without
> updating the credential helper protocol. So is the bigger goal to
> support more robust and modern auth schemes which require multiple
> steps? That would be useful to Google; multi-step auth would probably
> be a more elegant way for us to stop using .gitcookies than other
> solutions we were considering.
>
> [1] This is actually only for external contributors. Google employees
> have a more robust authentication mechanism. Concerns have been raised
> internally about our usage of gitcookies, but it hasn't been made a
> priority to address because a leaked credential would not allow an
> attacker to commit bad code, only read it or initiate a code review.

Re: [PATCH 00/13] Support for arbitrary schemes in credentials

[M Hickford]

On Sun, 24 Mar 2024 at 01:13, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> Right now, HTTP authentication in Git is mostly limited to approaches
> that require a username and password or are Kerberos (GSSAPI).  In
> addition, we effectively require that libcurl (or, for other software,
> such as Git LFS, using the credential helper, that HTTP library) knows
> how to implement the authentication scheme.
>
> However, this poses two sets of problems.  First, some sites, such as
> Azure DevOps, want to use Bearer authentication, which we don't support.
> This is implemented using `http.extraHeader`, which is not a secure way
> to store credentials, since our credential helper protocol does not
> support this functionality.
>
> In addition, other tools using the credential helper protocol do not
> support the variety of authentication mechanisms that Git does.
> Specifically, making NTLM function in a useful way on Windows is
> nontrivial and requires extensive integration and testing with C code,
> and because of this difficulty and the fact that NTLM uses cryptography
> known to be insecure since 1995, there is often little interest in
> implementing this support outside of libcurl. However, it would be
> helpful if people who want to use it can still use it.
>
> This series introduces new functionality to the credential helper
> protocol that allows helpers to produce credentials for arbitrary HTTP
> authentication schemes using the `authtype` and `credential`[0] fields.
> This allows a suitable credential helper to send Bearer credentials or
> any other standard or custom authentication scheme.  (It may be able to
> be extended to other functionality in the future, such as
> git-send-email, to implement custom SASL functionality, and due care has
> been taken to make the protocol adequately flexible for that purpose.)
>
> In addition, the protocol is also expanded to include per-helper state
> and multi-legged authentication (the former is effectively required for
> the latter).  The per-helper state can be useful to help credential
> helpers identify where the credential is stored, or any other
> information necessary.  Because NTLM and Negotiate (Kerberos/wrapped
> NTLM) require two rounds of authentication, the multi-legged
> authentication support along with per-helper state allows the helper to
> support these authentication methods without Git or other clients having
> to be aware of how they work.  (This would also be useful for SASL, as
> mentioned above.)
>
> This series introduces a capability mechanism to announce this
> functionality, which allows a helper to provide a username and password
> on older versions of Git while supporting more advanced functionality on
> newer versions.  (This is especially important on Azure DevOps, where
> NTLM uses a username and password but Basic or Bearer can use a personal
> access token.)  It is also designed such that extremely simple
> credential helpers, such as the shell one-liner in the Git FAQ that
> reads from the environment, don't accidentally claim to support
> functionality they don't offer.
>
> In addition, there is documentation for the expanded protocol, although
> none of the built-in helpers have been updated (that will be a future
> series for those for which it's possible).
>
> My personal interest here is getting credentials out of config files
> with `http.extraHeader` (which a future series will produce a warning
> for) and also allowing Git LFS to support Digest and NTLM with a
> suitable credential helper.  Git LFS used to support NTLM using custom
> code (because the Go standard library does not), but it was found to be
> broken in lots of ways on Windows, and nobody with a Windows system
> wanted to fix it or support it, so we removed it.  However, there are
> still some people who do want to use it, so allowing them to use a
> custom credential helper they maintain themselves seems like the best
> way forward.  Despite the advantages of this series for Azure DevOps, I
> have no personal or professional stake in their product; my only
> interest is the general one in whether their users can securely store
> credentials.  I believe the changes here are of general advantage to the
> Git userbase in a variety of ways such that the goal of this series
> should be uncontroversial.

Do you happen to know any public Git remotes that support Digest or
NTLM authentication?



>
> Feedback on any portion of this series is of course welcome.
>
> [0] A name different from `password` was explicitly chosen to avoid
> confusion from less capable protocol helpers so that they don't
> accidentally send invalid data.  This does have the downside that
> credential helpers must learn a new field to not log, but that should be
> generally easy to fix in most cases.
>
> brian m. carlson (13):
>   credential: add an authtype field
>   remote-curl: reset headers on new request
>   http: use new headers for each object request
>   credential: add a field for pre-encoded credentials
>   credential: gate new fields on capability
>   docs: indicate new credential protocol fields
>   http: add support for authtype and credential
>   credential: add an argument to keep state
>   credential: enable state capability
>   docs: set a limit on credential line length
>   t5563: refactor for multi-stage authentication
>   strvec: implement swapping two strvecs
>   credential: add support for multistage credential rounds
>
>  Documentation/git-credential.txt   |  59 +++++-
>  builtin/credential-cache--daemon.c |   2 +-
>  builtin/credential-store.c         |   2 +-
>  builtin/credential.c               |   7 +-
>  credential.c                       | 114 ++++++++++-
>  credential.h                       |  69 ++++++-
>  http.c                             | 128 +++++++-----
>  http.h                             |   5 +
>  imap-send.c                        |   2 +-
>  remote-curl.c                      |  14 +-
>  strvec.c                           |   7 +
>  strvec.h                           |   5 +
>  t/lib-httpd/nph-custom-auth.sh     |  17 +-
>  t/t0300-credentials.sh             | 136 ++++++++++++-
>  t/t5563-simple-http-auth.sh        | 308 +++++++++++++++++++++++++----
>  15 files changed, 760 insertions(+), 115 deletions(-)
>

Re: [PATCH 00/13] Support for arbitrary schemes in credentials

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]

On 2024-04-08 at 18:42:56, Jackson Toeniskoetter wrote:
> Just to clarify, we at Google do not use extraheader to pass along
> credentials. Instead we use the .gitcookies file[1], which iiuc gets
> read by the git process directly. I'm not a security expert but I
> imagine the risk surface of extraheader and .gitcookies is similar.
> 
> Reading your post on
> https://lore.kernel.org/git/20240324011301.1553072-1-sandals@crustytoothpaste.net/,
> it's unclear to me why the credential helper protocol needs to be
> updated. If the goal is to support Bearer tokens, can that not just be
> implemented using extraheader? It seems like the goal of getting
> credentials out of the config file can be accomplished without
> updating the credential helper protocol. So is the bigger goal to
> support more robust and modern auth schemes which require multiple
> steps? That would be useful to Google; multi-step auth would probably
> be a more elegant way for us to stop using .gitcookies than other
> solutions we were considering.

Bearer authentication certainly can be implemented using
http.extraheader and the config file is also not a secure way to store
credentials, which is why the credential helper protocol is being
updated, since then people will be able to store the credentials in a
password manager or other secure store.  Other HTTP schemes will also be
supported as long as they don't require headers other than Authorization
and the credential helper can implement them on behalf of Git.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 00/13] Support for arbitrary schemes in credentials

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 630 bytes --]

On 2024-04-11 at 07:00:00, M Hickford wrote:
> Do you happen to know any public Git remotes that support Digest or
> NTLM authentication?

Azure DevOps supports NTLM in some cases.  I don't use it, so I can't
say in what situations you can use it, but I know from the Git LFS issue
tracker it's a common configuration.

Note that if you just want a test environment, you can use Apache with
Digest authentication.  However, it's limited to MD5, which should no
longer be used for any reason, so you'll probably want to try a
different implementation.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

[PATCH v2 00/16] Support for arbitrary schemes in credentials

[brian m. carlson]

This series introduces support for multiple schemes into the credential
helper protocol and Git (currently only for HTTP) so that we can support
other types of authentication, such as Bearer, that libcurl doesn't
support natively.  Special consideration has been given to making this
functionality work for external users of `git credential`.

For the truly gory details, please see v1's cover letter.

Changes from v1:
* Add support for indicating that a credential is ephemeral.
* Add support for the authtype capability to git credential-cache.
* Fix a bug where credential helpers might not be passed the capability
  correctly.
* Add a method for querying capabilities from `git credential` and
  helpers, giving due consideration to gracefully handling simple shell
  scripts as credential helpers.
* Remove `strvec_swap` in favour of `SWAP`.
* Add support for testing authtype support in credential helpers.
* Fix a bug in the test that caused errors with bash.
* Use an enum instead of define constants for credential stages.
* Documentation improvements.
* Possibly other cleanups and fixes mentioned in review which I have
  forgotten about fixing by now.

brian m. carlson (16):
  credential: add an authtype field
  remote-curl: reset headers on new request
  http: use new headers for each object request
  credential: add a field for pre-encoded credentials
  credential: gate new fields on capability
  credential: add a field called "ephemeral"
  docs: indicate new credential protocol fields
  http: add support for authtype and credential
  credential: add an argument to keep state
  credential: enable state capability
  docs: set a limit on credential line length
  t5563: refactor for multi-stage authentication
  credential: add support for multistage credential rounds
  t: add credential tests for authtype
  credential-cache: implement authtype capability
  credential: add method for querying capabilities

 Documentation/git-credential.txt   | 104 +++++++++-
 builtin/credential-cache--daemon.c |  22 ++-
 builtin/credential-cache.c         |  10 +
 builtin/credential-store.c         |   2 +-
 builtin/credential.c               |  15 +-
 credential.c                       | 138 ++++++++++++-
 credential.h                       |  92 ++++++++-
 http.c                             | 129 +++++++-----
 http.h                             |   5 +
 imap-send.c                        |   2 +-
 remote-curl.c                      |  14 +-
 t/lib-credential.sh                | 123 ++++++++++++
 t/lib-httpd/nph-custom-auth.sh     |  17 +-
 t/t0300-credentials.sh             | 165 +++++++++++++++-
 t/t0301-credential-cache.sh        |   1 +
 t/t5563-simple-http-auth.sh        | 308 +++++++++++++++++++++++++----
 16 files changed, 1026 insertions(+), 121 deletions(-)

Range-diff against v1:
 1:  ac2507fabb =  1:  ac2507fabb credential: add an authtype field
 2:  f1f3367826 =  2:  f1f3367826 remote-curl: reset headers on new request
 3:  1cccbc9d8b !  3:  cd3b1a8c3a http: use new headers for each object request
    @@ http.c: void http_cleanup(void)
      
     -	curl_slist_free_all(no_pragma_header);
     -	no_pragma_header = NULL;
    - 
    +-
      	curl_slist_free_all(host_resolutions);
      	host_resolutions = NULL;
    + 
     @@ http.c: void release_http_pack_request(struct http_pack_request *preq)
      	}
      	preq->slot = NULL;
 4:  a88e6b26c8 =  4:  7b88a32294 credential: add a field for pre-encoded credentials
 5:  e0df6734a3 !  5:  912d06b70b credential: gate new fields on capability
    @@ Commit message
         internal user, then we know about all capabilities and will request
         them.
     
    +    For "git credential approve" and "git credential reject", we set the
    +    helper capability before calling the helper, since we assume that the
    +    input we're getting from the external program comes from a previous call
    +    to "git credential fill", and thus we'll invoke send a capability to the
    +    helper if and only if we got one from the standard input, which is the
    +    correct behavior.
    +
         Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
     
      ## builtin/credential-cache--daemon.c ##
    @@ builtin/credential.c: int cmd_credential(int argc, const char **argv, const char
     +		credential_fill(&c, 0);
     +		credential_write(&c, stdout, CREDENTIAL_OP_RESPONSE);
      	} else if (!strcmp(op, "approve")) {
    ++		credential_set_all_capabilities(&c, CREDENTIAL_OP_HELPER);
      		credential_approve(&c);
      	} else if (!strcmp(op, "reject")) {
    ++		credential_set_all_capabilities(&c, CREDENTIAL_OP_HELPER);
    + 		credential_reject(&c);
    + 	} else {
    + 		usage(usage_msg);
     
      ## credential.c ##
     @@ credential.c: void credential_clear(struct credential *c)
      	credential_init(c);
      }
      
    -+static void credential_set_all_capabilities(struct credential *c)
    -+{
    -+	c->capa_authtype.request_initial = 1;
    -+}
    -+
    - int credential_match(const struct credential *want,
    - 		     const struct credential *have, int match_password)
    - {
    -@@ credential.c: static void credential_getpass(struct credential *c)
    - 						 PROMPT_ASKPASS);
    - }
    - 
    --int credential_read(struct credential *c, FILE *fp)
    -+static void credential_set_capability(struct credential_capability *capa, int op_type)
    ++static void credential_set_capability(struct credential_capability *capa,
    ++				      enum credential_op_type op_type)
     +{
     +	switch (op_type) {
     +	case CREDENTIAL_OP_INITIAL:
    @@ credential.c: static void credential_getpass(struct credential *c)
     +	}
     +}
     +
    -+static int credential_has_capability(const struct credential_capability *capa, int op_type)
    ++
    ++void credential_set_all_capabilities(struct credential *c,
    ++				     enum credential_op_type op_type)
    ++{
    ++	credential_set_capability(&c->capa_authtype, op_type);
    ++}
    ++
    + int credential_match(const struct credential *want,
    + 		     const struct credential *have, int match_password)
    + {
    +@@ credential.c: static void credential_getpass(struct credential *c)
    + 						 PROMPT_ASKPASS);
    + }
    + 
    +-int credential_read(struct credential *c, FILE *fp)
    ++static int credential_has_capability(const struct credential_capability *capa,
    ++				     enum credential_op_type op_type)
     +{
     +	/*
     +	 * We're checking here if each previous step indicated that we had the
    @@ credential.c: static void credential_getpass(struct credential *c)
     +	}
     +}
     +
    -+int credential_read(struct credential *c, FILE *fp, int op_type)
    ++int credential_read(struct credential *c, FILE *fp,
    ++		    enum credential_op_type op_type)
      {
      	struct strbuf line = STRBUF_INIT;
      
    @@ credential.c: static void credential_write_item(FILE *fp, const char *key, const
      }
      
     -void credential_write(const struct credential *c, FILE *fp)
    -+void credential_write(const struct credential *c, FILE *fp, int op_type)
    ++void credential_write(const struct credential *c, FILE *fp,
    ++		      enum credential_op_type op_type)
      {
     +	if (credential_has_capability(&c->capa_authtype, op_type)) {
     +		credential_write_item(fp, "capability[]", "authtype", 0);
    @@ credential.c: void credential_fill(struct credential *c)
      
      	credential_apply_config(c);
     +	if (all_capabilities)
    -+		credential_set_all_capabilities(c);
    ++		credential_set_all_capabilities(c, CREDENTIAL_OP_INITIAL);
      
      	for (i = 0; i < c->helpers.nr; i++) {
      		credential_do(c, c->helpers.items[i].string, "get");
    @@ credential.h
     + * At each stage, we will emit the capability only if the previous stage
     + * supported it.
     + */
    -+#define CREDENTIAL_OP_INITIAL  1
    -+#define CREDENTIAL_OP_HELPER   2
    -+#define CREDENTIAL_OP_RESPONSE 3
    ++enum credential_op_type {
    ++	CREDENTIAL_OP_INITIAL  = 1,
    ++	CREDENTIAL_OP_HELPER   = 2,
    ++	CREDENTIAL_OP_RESPONSE = 3,
    ++};
     +
     +struct credential_capability {
     +	unsigned request_initial:1,
    @@ credential.h: void credential_approve(struct credential *);
      
     -int credential_read(struct credential *, FILE *);
     -void credential_write(const struct credential *, FILE *);
    -+int credential_read(struct credential *, FILE *, int);
    -+void credential_write(const struct credential *, FILE *, int);
    ++/**
    ++ * Enable all of the supported credential flags in this credential.
    ++ */
    ++void credential_set_all_capabilities(struct credential *c,
    ++				     enum credential_op_type op_type);
    ++
    ++int credential_read(struct credential *, FILE *,
    ++		    enum credential_op_type);
    ++void credential_write(const struct credential *, FILE *,
    ++		      enum credential_op_type);
      
      /*
       * Parse a url into a credential struct, replacing any existing contents.
    @@ t/t0300-credentials.sh: test_expect_success 'setup helper scripts' '
     +	credential=$1; shift
     +	. ./dump
     +	echo capability[]=authtype
    -+	test -z "${capability##*authtype*}" || return
    ++	test -z "${capability##*authtype*}" || exit 0
     +	test -z "$authtype" || echo authtype=$authtype
     +	test -z "$credential" || echo credential=$credential
     +	EOF
 -:  ---------- >  6:  f96183e273 credential: add a field called "ephemeral"
 6:  78fc8cdccf !  7:  efaab8eba4 docs: indicate new credential protocol fields
    @@ Documentation/git-credential.txt: empty string.
      
     +`authtype`::
     +	This indicates that the authentication scheme in question should be used.
    -+	Common values for HTTP and HTTPS include `basic`, `digest`, and `ntlm`,
    -+	although the latter two are insecure and should not be used.  If `credential`
    ++	Common values for HTTP and HTTPS include `basic`, `bearer`, and `digest`,
    ++	although the latter is insecure and should not be used.  If `credential`
     +	is used, this may be set to an arbitrary string suitable for the protocol in
     +	question (usually HTTP).
     ++
    @@ Documentation/git-credential.txt: empty string.
     +`credential`::
     +	The pre-encoded credential, suitable for the protocol in question (usually
     +	HTTP).  If this key is sent, `authtype` is mandatory, and `username` and
    -+	`password` are not used.
    ++	`password` are not used.  For HTTP, Git concatenates the `authtype` value and
    ++	this value with a single space to determine the `Authorization` header.
    +++
    ++This value should not be sent unless the appropriate capability (see below) is
    ++provided on input.
    ++
    ++`ephemeral`::
    ++	This boolean value indicates, if true, that the value in the `credential`
    ++	field should not be saved by the credential helper because its usefulness is
    ++	limited in time.  For example, an HTTP Digest `credential` value is computed
    ++	using a nonce and reusing it will not result in successful authentication.
    ++	This may also be used for situations with short duration (e.g., 24-hour)
    ++	credentials.  The default value is false.
    +++
    ++The credential helper will still be invoked with `store` or `erase` so that it
    ++can determine whether the operation was successful.
     ++
     +This value should not be sent unless the appropriate capability (see below) is
     +provided on input.
    @@ Documentation/git-credential.txt: attribute 'wwwauth[]', where the order of the
     +	protocol.
     ++
     +The only capability currently supported is `authtype`, which indicates that the
    -+`authtype` and `credential` values are understood.  It is not obligatory to use
    -+these values in such a case, but they should not be provided without this
    -+capability.
    ++`authtype`, `credential`, and `ephemeral` values are understood.  It is not
    ++obligatory to use these values in such a case, but they should not be provided
    ++without this capability.
     ++
     +Callers of `git credential` and credential helpers should emit the
     +capabilities they support unconditionally, and Git will gracefully
 7:  82f7582268 =  8:  3b0a426fa1 http: add support for authtype and credential
 8:  c67540de6f !  9:  8897ecf5e0 credential: add an argument to keep state
    @@ Commit message
         Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
     
      ## Documentation/git-credential.txt ##
    -@@ Documentation/git-credential.txt: provided on input.
    +@@ Documentation/git-credential.txt: can determine whether the operation was successful.
      This value should not be sent unless the appropriate capability (see below) is
      provided on input.
      
    @@ Documentation/git-credential.txt: they appear in the HTTP response. This attribu
      `capability[]`::
     -	This signals that the caller supports the capability in question.
     -	This can be used to provide better, more specific data as part of the
    --	protocol.
    -+  This signals that Git, or the helper, as appropriate, supports the
    -+	capability in question.  This can be used to provide better, more specific
    -+	data as part of the protocol.
    ++	This signals that Git, or the helper, as appropriate, supports the capability
    ++	in question.  This can be used to provide better, more specific data as part
    ++	of the protocol.  A `capability[]` directive must precede any value depending
    ++	on it and these directives _should_ be the first item announced in the
    + 	protocol.
      +
     -The only capability currently supported is `authtype`, which indicates that the
    --`authtype` and `credential` values are understood.  It is not obligatory to use
    --these values in such a case, but they should not be provided without this
    --capability.
    --+
    +-`authtype`, `credential`, and `ephemeral` values are understood.  It is not
    +-obligatory to use these values in such a case, but they should not be provided
    +-without this capability.
    ++There are two currently supported capabilities.  The first is `authtype`, which
    ++indicates that the `authtype`, `credential`, and `ephemeral` values are
    ++understood.  The second is `state`, which indicates that the `state[]` and
    ++`continue` values are understood.
    + +
     -Callers of `git credential` and credential helpers should emit the
     -capabilities they support unconditionally, and Git will gracefully
     -handle passing them on.
    -+There are two currently supported capabilities.  The first is `authtype`, which
    -+indicates that the `authtype` and `credential` values are understood.  The
    -+second is `state`, which indicates that the `state[]` value is understood.
    -+
     +It is not obligatory to use the additional features just because the capability
    -+is supported, but they should not be provided without this capability.
    ++is supported, but they should not be provided without the capability.
      
      Unrecognised attributes and capabilities are silently discarded.
      
    @@ credential.c: void credential_clear(struct credential *c)
      
      	credential_init(c);
      }
    -@@ credential.c: int credential_read(struct credential *c, FILE *fp, int op_type)
    - 			c->path = xstrdup(value);
    +@@ credential.c: int credential_read(struct credential *c, FILE *fp,
    + 			c->ephemeral = !!git_config_bool("ephemeral", value);
      		} else if (!strcmp(key, "wwwauth[]")) {
      			strvec_push(&c->wwwauth_headers, value);
     -		} else if (!strcmp(key, "capability[]") && !strcmp(value, "authtype")) {
    @@ credential.c: int credential_read(struct credential *c, FILE *fp, int op_type)
      			errno = 0;
      			c->password_expiry_utc = parse_timestamp(value, NULL, 10);
     @@ credential.c: static void credential_write_item(FILE *fp, const char *key, const char *value,
    - 
    - void credential_write(const struct credential *c, FILE *fp, int op_type)
    + void credential_write(const struct credential *c, FILE *fp,
    + 		      enum credential_op_type op_type)
      {
     -	if (credential_has_capability(&c->capa_authtype, op_type)) {
     +	if (credential_has_capability(&c->capa_authtype, op_type))
    @@ credential.c: static void credential_write_item(FILE *fp, const char *key, const
     +	if (credential_has_capability(&c->capa_authtype, op_type)) {
      		credential_write_item(fp, "authtype", c->authtype, 0);
      		credential_write_item(fp, "credential", c->credential, 0);
    - 	}
    -@@ credential.c: void credential_write(const struct credential *c, FILE *fp, int op_type)
    + 		if (c->ephemeral)
    +@@ credential.c: void credential_write(const struct credential *c, FILE *fp,
      	}
      	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
      		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
    @@ t/t0300-credentials.sh: test_expect_success 'setup helper scripts' '
      	. ./dump
      	echo capability[]=authtype
     +	echo capability[]=state
    - 	test -z "${capability##*authtype*}" || return
    + 	test -z "${capability##*authtype*}" || exit 0
      	test -z "$authtype" || echo authtype=$authtype
      	test -z "$credential" || echo credential=$credential
    -+	test -z "${capability##*state*}" || return
    ++	test -z "${capability##*state*}" || exit 0
     +	echo state[]=verbatim-cred:foo
      	EOF
      
    - 	write_script git-credential-verbatim-with-expiry <<-\EOF &&
    -@@ t/t0300-credentials.sh: test_expect_success 'credential_fill invokes helper with credential' '
    + 	write_script git-credential-verbatim-ephemeral <<-\EOF &&
    +@@ t/t0300-credentials.sh: test_expect_success 'credential_fill invokes helper with ephemeral credential' '
    + 	verbatim-ephemeral: host=example.com
      	EOF
      '
    - 
     +test_expect_success 'credential_fill invokes helper with credential and state' '
     +	check fill "verbatim-cred Bearer token" <<-\EOF
     +	capability[]=authtype
    @@ t/t0300-credentials.sh: test_expect_success 'credential_fill invokes helper with
     +	verbatim-cred: host=example.com
     +	EOF
     +'
    -+
      
      test_expect_success 'credential_fill invokes multiple helpers' '
      	check fill useless "verbatim foo bar" <<-\EOF
 9:  6321e0d5e6 ! 10:  6e56145ace credential: enable state capability
    @@ Commit message
         Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
     
      ## credential.c ##
    -@@ credential.c: void credential_clear(struct credential *c)
    - static void credential_set_all_capabilities(struct credential *c)
    +@@ credential.c: void credential_set_all_capabilities(struct credential *c,
    + 				     enum credential_op_type op_type)
      {
    - 	c->capa_authtype.request_initial = 1;
    -+	c->capa_state.request_initial = 1;
    + 	credential_set_capability(&c->capa_authtype, op_type);
    ++	credential_set_capability(&c->capa_state, op_type);
      }
      
      int credential_match(const struct credential *want,
10:  aaac77a3ce = 11:  3f5d2aa5e0 docs: set a limit on credential line length
11:  f4b01b3ba5 = 12:  1dd81f7859 t5563: refactor for multi-stage authentication
12:  a2f82a4043 <  -:  ---------- strvec: implement swapping two strvecs
13:  ec925ae9fb ! 13:  4a0ed175a6 credential: add support for multistage credential rounds
    @@ Documentation/git-credential.txt: provided on input.
      `wwwauth[]`::
      
      	When an HTTP response is received by Git that includes one or more
    -@@ Documentation/git-credential.txt: to pass additional information to credential helpers.
    - +
    - There are two currently supported capabilities.  The first is `authtype`, which
    - indicates that the `authtype` and `credential` values are understood.  The
    --second is `state`, which indicates that the `state[]` value is understood.
    -+second is `state`, which indicates that the `state[]` and `continue` values are
    -+understood.
    - 
    - It is not obligatory to use the additional features just because the capability
    - is supported, but they should not be provided without this capability.
     
      ## builtin/credential.c ##
     @@ builtin/credential.c: int cmd_credential(int argc, const char **argv, const char *prefix UNUSED)
    @@ builtin/credential.c: int cmd_credential(int argc, const char **argv, const char
     +		credential_next_state(&c);
      		credential_write(&c, stdout, CREDENTIAL_OP_RESPONSE);
      	} else if (!strcmp(op, "approve")) {
    - 		credential_approve(&c);
    + 		credential_set_all_capabilities(&c, CREDENTIAL_OP_HELPER);
     
      ## credential.c ##
     @@ credential.c: void credential_clear(struct credential *c)
    @@ credential.c: void credential_clear(struct credential *c)
     +void credential_next_state(struct credential *c)
     +{
     +	strvec_clear(&c->state_headers_to_send);
    -+	strvec_swap(&c->state_headers, &c->state_headers_to_send);
    ++	SWAP(c->state_headers, c->state_headers_to_send);
     +}
     +
     +void credential_clear_secrets(struct credential *c)
    @@ credential.c: void credential_clear(struct credential *c)
     +	FREE_AND_NULL(c->credential);
     +}
     +
    - static void credential_set_all_capabilities(struct credential *c)
    + static void credential_set_capability(struct credential_capability *capa,
    + 				      enum credential_op_type op_type)
      {
    - 	c->capa_authtype.request_initial = 1;
    -@@ credential.c: int credential_read(struct credential *c, FILE *fp, int op_type)
    +@@ credential.c: int credential_read(struct credential *c, FILE *fp,
      				credential_set_capability(&c->capa_authtype, op_type);
      			else if (!strcmp(value, "state"))
      				credential_set_capability(&c->capa_state, op_type);
    @@ credential.c: int credential_read(struct credential *c, FILE *fp, int op_type)
      		} else if (!strcmp(key, "password_expiry_utc")) {
      			errno = 0;
      			c->password_expiry_utc = parse_timestamp(value, NULL, 10);
    -@@ credential.c: void credential_write(const struct credential *c, FILE *fp, int op_type)
    +@@ credential.c: void credential_write(const struct credential *c, FILE *fp,
      	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
      		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
      	if (credential_has_capability(&c->capa_state, op_type)) {
    @@ credential.c: void credential_fill(struct credential *c, int all_capabilities)
     +
      	credential_apply_config(c);
      	if (all_capabilities)
    - 		credential_set_all_capabilities(c);
    + 		credential_set_all_capabilities(c, CREDENTIAL_OP_INITIAL);
     @@ credential.c: void credential_fill(struct credential *c, int all_capabilities)
      			/* Reset expiry to maintain consistency */
      			c->password_expiry_utc = TIME_MAX;
    @@ credential.h: struct credential {
      	 * Internal use only. Keeps track of if we previously matched against a
      	 * WWW-Authenticate header line in order to re-fold future continuation
     @@ credential.h: struct credential {
    - 
      	unsigned approved:1,
    + 		 ephemeral:1,
      		 configured:1,
     +		 multistage: 1,
      		 quit:1,
    @@ credential.h: struct credential {
      }
      
      /* Initialize a credential structure, setting all fields to empty. */
    -@@ credential.h: void credential_approve(struct credential *);
    -  */
    - void credential_reject(struct credential *);
    +@@ credential.h: void credential_reject(struct credential *);
    + void credential_set_all_capabilities(struct credential *c,
    + 				     enum credential_op_type op_type);
      
     +/**
     + * Clear the secrets in this credential, but leave other data intact.
    @@ credential.h: void credential_approve(struct credential *);
     + */
     +void credential_next_state(struct credential *c);
     +
    - int credential_read(struct credential *, FILE *, int);
    - void credential_write(const struct credential *, FILE *, int);
    - 
    + int credential_read(struct credential *, FILE *,
    + 		    enum credential_op_type);
    + void credential_write(const struct credential *, FILE *,
     
      ## http.c ##
     @@ http.c: static int handle_curl_result(struct slot_results *results)
 -:  ---------- > 14:  c4c07c1539 t: add credential tests for authtype
 -:  ---------- > 15:  b82650e219 credential-cache: implement authtype capability
 -:  ---------- > 16:  2f6e160693 credential: add method for querying capabilities

[PATCH v2 01/16] credential: add an authtype field

[brian m. carlson]

When Git makes an HTTP request, it can negotiate the type of
authentication to use with the server provided the authentication scheme
is one of a few well-known types (Basic, Digest, NTLM, or Negotiate).
However, some servers wish to use other types of authentication, such as
the Bearer type from OAuth2.  Since libcurl doesn't natively support
this type, it isn't possible to use it, and the user is forced to
specify the Authorization header using the http.extraheader setting.

However, storing a plaintext token in the repository configuration is
not very secure, especially if a repository can be shared by multiple
parties.  We already have support for many types of secure credential
storage by using credential helpers, so let's teach credential helpers
how to produce credentials for an arbitrary scheme.

If the credential helper specifies an authtype field, then it specifies
an authentication scheme (e.g., Bearer) and the password field specifies
the raw authentication token, with any encoding already specified.  We
reuse the password field for this because some credential helpers store
the metadata without encryption even though the password is encrypted,
and we'd like to avoid insecure storage if an older version of the
credential helper gets ahold of the data.

The username is not used in this case, but it is still preserved for the
purpose of finding the right credential if the user has multiple
accounts.

If the authtype field is not specified, then the password behaves as
normal and it is passed along with the username to libcurl.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 credential.c | 5 +++++
 credential.h | 6 ++++++
 2 files changed, 11 insertions(+)

diff --git a/credential.c b/credential.c
index 18098bd35e..3dec433df5 100644
--- a/credential.c
+++ b/credential.c
@@ -26,6 +26,7 @@ void credential_clear(struct credential *c)
 	free(c->username);
 	free(c->password);
 	free(c->oauth_refresh_token);
+	free(c->authtype);
 	string_list_clear(&c->helpers, 0);
 	strvec_clear(&c->wwwauth_headers);
 
@@ -252,6 +253,9 @@ int credential_read(struct credential *c, FILE *fp)
 		} else if (!strcmp(key, "oauth_refresh_token")) {
 			free(c->oauth_refresh_token);
 			c->oauth_refresh_token = xstrdup(value);
+		} else if (!strcmp(key, "authtype")) {
+			free(c->authtype);
+			c->authtype = xstrdup(value);
 		} else if (!strcmp(key, "url")) {
 			credential_from_url(c, value);
 		} else if (!strcmp(key, "quit")) {
@@ -295,6 +299,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	}
 	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
 		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
+	credential_write_item(fp, "authtype", c->authtype, 0);
 }
 
 static int run_credential_helper(struct credential *c,
diff --git a/credential.h b/credential.h
index acc41adf54..dc96ca0318 100644
--- a/credential.h
+++ b/credential.h
@@ -143,6 +143,12 @@ struct credential {
 	char *path;
 	char *oauth_refresh_token;
 	timestamp_t password_expiry_utc;
+
+	/**
+	 * The authorization scheme to use.  If this is NULL, libcurl is free to
+	 * negotiate any scheme it likes.
+	 */
+	char *authtype;
 };
 
 #define CREDENTIAL_INIT { \

[PATCH v2 02/16] remote-curl: reset headers on new request

[brian m. carlson]

When we retry a post_rpc request, we currently reuse the same headers as
before.  In the future, we'd like to be able to modify them based on the
result we get back, so let's reset them on each retry so we can avoid
sending potentially duplicate headers if the values change.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 remote-curl.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/remote-curl.c b/remote-curl.c
index 1161dc7fed..e37eaa17b7 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -893,7 +893,7 @@ static curl_off_t xcurl_off_t(size_t len)
 static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_received)
 {
 	struct active_request_slot *slot;
-	struct curl_slist *headers = http_copy_default_headers();
+	struct curl_slist *headers = NULL;
 	int use_gzip = rpc->gzip_request;
 	char *gzip_body = NULL;
 	size_t gzip_size = 0;
@@ -935,6 +935,8 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 			needs_100_continue = 1;
 	}
 
+retry:
+	headers = http_copy_default_headers();
 	headers = curl_slist_append(headers, rpc->hdr_content_type);
 	headers = curl_slist_append(headers, rpc->hdr_accept);
 	headers = curl_slist_append(headers, needs_100_continue ?
@@ -948,7 +950,6 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 	if (rpc->protocol_header)
 		headers = curl_slist_append(headers, rpc->protocol_header);
 
-retry:
 	slot = get_active_slot();
 
 	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
@@ -1044,6 +1045,7 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 	err = run_slot(slot, NULL);
 	if (err == HTTP_REAUTH && !large_request) {
 		credential_fill(&http_auth);
+		curl_slist_free_all(headers);
 		goto retry;
 	}
 	if (err != HTTP_OK)

[PATCH v2 03/16] http: use new headers for each object request

[brian m. carlson]

Currently we create one set of headers for all object requests and reuse
it.  However, we'll need to adjust the headers for authentication
purposes in the future, so let's create a new set for each request so
that we can adjust them if the authentication changes.

Note that the cost of allocation here is tiny compared to the fact that
we're making a network call, not to mention probably a full TLS
connection, so this shouldn't have a significant impact on performance.
Moreover, nobody who cares about performance is using the dumb HTTP
protocol anyway, since it often makes huge numbers of requests compared
to the smart protocol.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 http.c | 20 +++++++++++---------
 http.h |  2 ++
 2 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/http.c b/http.c
index e73b136e58..22639b0974 100644
--- a/http.c
+++ b/http.c
@@ -128,7 +128,6 @@ static unsigned long empty_auth_useless =
 	| CURLAUTH_DIGEST;
 
 static struct curl_slist *pragma_header;
-static struct curl_slist *no_pragma_header;
 static struct string_list extra_http_headers = STRING_LIST_INIT_DUP;
 
 static struct curl_slist *host_resolutions;
@@ -299,6 +298,11 @@ size_t fwrite_null(char *ptr UNUSED, size_t eltsize UNUSED, size_t nmemb,
 	return nmemb;
 }
 
+static struct curl_slist *object_request_headers(void)
+{
+	return curl_slist_append(http_copy_default_headers(), "Pragma:");
+}
+
 static void closedown_active_slot(struct active_request_slot *slot)
 {
 	active_requests--;
@@ -1275,8 +1279,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)
 
 	pragma_header = curl_slist_append(http_copy_default_headers(),
 		"Pragma: no-cache");
-	no_pragma_header = curl_slist_append(http_copy_default_headers(),
-		"Pragma:");
 
 	{
 		char *http_max_requests = getenv("GIT_HTTP_MAX_REQUESTS");
@@ -1360,9 +1362,6 @@ void http_cleanup(void)
 	curl_slist_free_all(pragma_header);
 	pragma_header = NULL;
 
-	curl_slist_free_all(no_pragma_header);
-	no_pragma_header = NULL;
-
 	curl_slist_free_all(host_resolutions);
 	host_resolutions = NULL;
 
@@ -2370,6 +2369,7 @@ void release_http_pack_request(struct http_pack_request *preq)
 	}
 	preq->slot = NULL;
 	strbuf_release(&preq->tmpfile);
+	curl_slist_free_all(preq->headers);
 	free(preq->url);
 	free(preq);
 }
@@ -2454,11 +2454,11 @@ struct http_pack_request *new_direct_http_pack_request(
 	}
 
 	preq->slot = get_active_slot();
+	preq->headers = object_request_headers();
 	curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEDATA, preq->packfile);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite);
 	curl_easy_setopt(preq->slot->curl, CURLOPT_URL, preq->url);
-	curl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER,
-		no_pragma_header);
+	curl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER, preq->headers);
 
 	/*
 	 * If there is data present from a previous transfer attempt,
@@ -2624,13 +2624,14 @@ struct http_object_request *new_http_object_request(const char *base_url,
 	}
 
 	freq->slot = get_active_slot();
+	freq->headers = object_request_headers();
 
 	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEDATA, freq);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_FAILONERROR, 0);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);
 	curl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);
-	curl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
+	curl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, freq->headers);
 
 	/*
 	 * If we have successfully processed data from a previous fetch
@@ -2718,5 +2719,6 @@ void release_http_object_request(struct http_object_request *freq)
 		release_active_slot(freq->slot);
 		freq->slot = NULL;
 	}
+	curl_slist_free_all(freq->headers);
 	strbuf_release(&freq->tmpfile);
 }
diff --git a/http.h b/http.h
index 3af19a8bf5..c5f8cc4620 100644
--- a/http.h
+++ b/http.h
@@ -196,6 +196,7 @@ struct http_pack_request {
 	FILE *packfile;
 	struct strbuf tmpfile;
 	struct active_request_slot *slot;
+	struct curl_slist *headers;
 };
 
 struct http_pack_request *new_http_pack_request(
@@ -229,6 +230,7 @@ struct http_object_request {
 	int zret;
 	int rename;
 	struct active_request_slot *slot;
+	struct curl_slist *headers;
 };
 
 struct http_object_request *new_http_object_request(

[PATCH v2 04/16] credential: add a field for pre-encoded credentials

[brian m. carlson]

At the moment, our credential code wants to find a username and password
for access, which, for HTTP, it will pass to libcurl to encode and
process.  However, many users want to use authentication schemes that
libcurl doesn't support, such as Bearer authentication.  In these
schemes, the secret is not a username and password pair, but some sort
of token that meets the production for authentication data in the RFC.

In fact, in general, it's useful to allow our credential helper to have
knowledge about what specifically to put in the protocol header.  Thus,
add a field, credential, which contains data that's preencoded to be
suitable for the protocol in question.  If we have such data, we need
neither a username nor a password, so make that adjustment as well.

It is in theory possible to reuse the password field for this.  However,
if we do so, we must know whether the credential helper supports our new
scheme before sending it data, which necessitates some sort of
capability inquiry, because otherwise an uninformed credential helper
would store our preencoded data as a password, which would fail the next
time we attempted to connect to the remote server.  This design is
substantially simpler, and we can hint to the credential helper that we
support this approach with a simple new field instead of needing to
query it first.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 credential.c | 14 ++++++++++----
 credential.h |  1 +
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/credential.c b/credential.c
index 3dec433df5..c521822e5a 100644
--- a/credential.c
+++ b/credential.c
@@ -25,6 +25,7 @@ void credential_clear(struct credential *c)
 	free(c->path);
 	free(c->username);
 	free(c->password);
+	free(c->credential);
 	free(c->oauth_refresh_token);
 	free(c->authtype);
 	string_list_clear(&c->helpers, 0);
@@ -234,6 +235,9 @@ int credential_read(struct credential *c, FILE *fp)
 		} else if (!strcmp(key, "password")) {
 			free(c->password);
 			c->password = xstrdup(value);
+		} else if (!strcmp(key, "credential")) {
+			free(c->credential);
+			c->credential = xstrdup(value);
 		} else if (!strcmp(key, "protocol")) {
 			free(c->protocol);
 			c->protocol = xstrdup(value);
@@ -291,6 +295,7 @@ void credential_write(const struct credential *c, FILE *fp)
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
+	credential_write_item(fp, "credential", c->credential, 0);
 	credential_write_item(fp, "oauth_refresh_token", c->oauth_refresh_token, 0);
 	if (c->password_expiry_utc != TIME_MAX) {
 		char *s = xstrfmt("%"PRItime, c->password_expiry_utc);
@@ -366,7 +371,7 @@ void credential_fill(struct credential *c)
 {
 	int i;
 
-	if (c->username && c->password)
+	if ((c->username && c->password) || c->credential)
 		return;
 
 	credential_apply_config(c);
@@ -379,7 +384,7 @@ void credential_fill(struct credential *c)
 			/* Reset expiry to maintain consistency */
 			c->password_expiry_utc = TIME_MAX;
 		}
-		if (c->username && c->password)
+		if ((c->username && c->password) || c->credential)
 			return;
 		if (c->quit)
 			die("credential helper '%s' told us to quit",
@@ -387,7 +392,7 @@ void credential_fill(struct credential *c)
 	}
 
 	credential_getpass(c);
-	if (!c->username && !c->password)
+	if (!c->username && !c->password && !c->credential)
 		die("unable to get password from user");
 }
 
@@ -397,7 +402,7 @@ void credential_approve(struct credential *c)
 
 	if (c->approved)
 		return;
-	if (!c->username || !c->password || c->password_expiry_utc < time(NULL))
+	if (((!c->username || !c->password) && !c->credential) || c->password_expiry_utc < time(NULL))
 		return;
 
 	credential_apply_config(c);
@@ -418,6 +423,7 @@ void credential_reject(struct credential *c)
 
 	FREE_AND_NULL(c->username);
 	FREE_AND_NULL(c->password);
+	FREE_AND_NULL(c->credential);
 	FREE_AND_NULL(c->oauth_refresh_token);
 	c->password_expiry_utc = TIME_MAX;
 	c->approved = 0;
diff --git a/credential.h b/credential.h
index dc96ca0318..9db892cf4d 100644
--- a/credential.h
+++ b/credential.h
@@ -138,6 +138,7 @@ struct credential {
 
 	char *username;
 	char *password;
+	char *credential;
 	char *protocol;
 	char *host;
 	char *path;

[PATCH v2 05/16] credential: gate new fields on capability

[brian m. carlson]

We support the new credential and authtype fields, but we lack a way to
indicate to a credential helper that we'd like them to be used.  Without
some sort of indication, the credential helper doesn't know if it should
try to provide us a username and password, or a pre-encoded credential.
For example, the helper might prefer a more restricted Bearer token if
pre-encoded credentials are possible, but might have to fall back to
more general username and password if not.

Let's provide a simple way to indicate whether Git (or, for that matter,
the helper) is capable of understanding the authtype and credential
fields.  We send this capability when we generate a request, and the
other side may reply to indicate to us that it does, too.

For now, don't enable sending capabilities for the HTTP code.  In a
future commit, we'll introduce appropriate handling for that code,
which requires more in-depth work.

The logic for determining whether a capability is supported may seem
complex, but it is not.  At each stage, we emit the capability to the
following stage if all preceding stages have declared it.  Thus, if the
caller to git credential fill didn't declare it, then we won't send it
to the helper, and if fill's caller did send but the helper doesn't
understand it, then we won't send it on in the response.  If we're an
internal user, then we know about all capabilities and will request
them.

For "git credential approve" and "git credential reject", we set the
helper capability before calling the helper, since we assume that the
input we're getting from the external program comes from a previous call
to "git credential fill", and thus we'll invoke send a capability to the
helper if and only if we got one from the standard input, which is the
correct behavior.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 builtin/credential-cache--daemon.c |   2 +-
 builtin/credential-store.c         |   2 +-
 builtin/credential.c               |   8 ++-
 credential.c                       |  64 +++++++++++++++--
 credential.h                       |  40 ++++++++++-
 http.c                             |  10 +--
 imap-send.c                        |   2 +-
 remote-curl.c                      |   4 +-
 t/t0300-credentials.sh             | 107 ++++++++++++++++++++++++++++-
 9 files changed, 215 insertions(+), 24 deletions(-)

diff --git a/builtin/credential-cache--daemon.c b/builtin/credential-cache--daemon.c
index 3a6a750a8e..ccbcf99ac1 100644
--- a/builtin/credential-cache--daemon.c
+++ b/builtin/credential-cache--daemon.c
@@ -115,7 +115,7 @@ static int read_request(FILE *fh, struct credential *c,
 		return error("client sent bogus timeout line: %s", item.buf);
 	*timeout = atoi(p);
 
-	if (credential_read(c, fh) < 0)
+	if (credential_read(c, fh, CREDENTIAL_OP_HELPER) < 0)
 		return -1;
 	return 0;
 }
diff --git a/builtin/credential-store.c b/builtin/credential-store.c
index 4a492411bb..494c809332 100644
--- a/builtin/credential-store.c
+++ b/builtin/credential-store.c
@@ -205,7 +205,7 @@ int cmd_credential_store(int argc, const char **argv, const char *prefix)
 	if (!fns.nr)
 		die("unable to set up default path; use --file");
 
-	if (credential_read(&c, stdin) < 0)
+	if (credential_read(&c, stdin, CREDENTIAL_OP_HELPER) < 0)
 		die("unable to read credential");
 
 	if (!strcmp(op, "get"))
diff --git a/builtin/credential.c b/builtin/credential.c
index 7010752987..643bf0b5e4 100644
--- a/builtin/credential.c
+++ b/builtin/credential.c
@@ -17,15 +17,17 @@ int cmd_credential(int argc, const char **argv, const char *prefix UNUSED)
 		usage(usage_msg);
 	op = argv[1];
 
-	if (credential_read(&c, stdin) < 0)
+	if (credential_read(&c, stdin, CREDENTIAL_OP_INITIAL) < 0)
 		die("unable to read credential from stdin");
 
 	if (!strcmp(op, "fill")) {
-		credential_fill(&c);
-		credential_write(&c, stdout);
+		credential_fill(&c, 0);
+		credential_write(&c, stdout, CREDENTIAL_OP_RESPONSE);
 	} else if (!strcmp(op, "approve")) {
+		credential_set_all_capabilities(&c, CREDENTIAL_OP_HELPER);
 		credential_approve(&c);
 	} else if (!strcmp(op, "reject")) {
+		credential_set_all_capabilities(&c, CREDENTIAL_OP_HELPER);
 		credential_reject(&c);
 	} else {
 		usage(usage_msg);
diff --git a/credential.c b/credential.c
index c521822e5a..f5396629df 100644
--- a/credential.c
+++ b/credential.c
@@ -34,6 +34,29 @@ void credential_clear(struct credential *c)
 	credential_init(c);
 }
 
+static void credential_set_capability(struct credential_capability *capa,
+				      enum credential_op_type op_type)
+{
+	switch (op_type) {
+	case CREDENTIAL_OP_INITIAL:
+		capa->request_initial = 1;
+		break;
+	case CREDENTIAL_OP_HELPER:
+		capa->request_helper = 1;
+		break;
+	case CREDENTIAL_OP_RESPONSE:
+		capa->response = 1;
+		break;
+	}
+}
+
+
+void credential_set_all_capabilities(struct credential *c,
+				     enum credential_op_type op_type)
+{
+	credential_set_capability(&c->capa_authtype, op_type);
+}
+
 int credential_match(const struct credential *want,
 		     const struct credential *have, int match_password)
 {
@@ -210,7 +233,26 @@ static void credential_getpass(struct credential *c)
 						 PROMPT_ASKPASS);
 }
 
-int credential_read(struct credential *c, FILE *fp)
+static int credential_has_capability(const struct credential_capability *capa,
+				     enum credential_op_type op_type)
+{
+	/*
+	 * We're checking here if each previous step indicated that we had the
+	 * capability.  If it did, then we want to pass it along; conversely, if
+	 * it did not, we don't want to report that to our caller.
+	 */
+	switch (op_type) {
+	case CREDENTIAL_OP_HELPER:
+		return capa->request_initial;
+	case CREDENTIAL_OP_RESPONSE:
+		return capa->request_initial && capa->request_helper;
+	default:
+		return 0;
+	}
+}
+
+int credential_read(struct credential *c, FILE *fp,
+		    enum credential_op_type op_type)
 {
 	struct strbuf line = STRBUF_INIT;
 
@@ -249,6 +291,8 @@ int credential_read(struct credential *c, FILE *fp)
 			c->path = xstrdup(value);
 		} else if (!strcmp(key, "wwwauth[]")) {
 			strvec_push(&c->wwwauth_headers, value);
+		} else if (!strcmp(key, "capability[]") && !strcmp(value, "authtype")) {
+			credential_set_capability(&c->capa_authtype, op_type);
 		} else if (!strcmp(key, "password_expiry_utc")) {
 			errno = 0;
 			c->password_expiry_utc = parse_timestamp(value, NULL, 10);
@@ -288,14 +332,19 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
 	fprintf(fp, "%s=%s\n", key, value);
 }
 
-void credential_write(const struct credential *c, FILE *fp)
+void credential_write(const struct credential *c, FILE *fp,
+		      enum credential_op_type op_type)
 {
+	if (credential_has_capability(&c->capa_authtype, op_type)) {
+		credential_write_item(fp, "capability[]", "authtype", 0);
+		credential_write_item(fp, "authtype", c->authtype, 0);
+		credential_write_item(fp, "credential", c->credential, 0);
+	}
 	credential_write_item(fp, "protocol", c->protocol, 1);
 	credential_write_item(fp, "host", c->host, 1);
 	credential_write_item(fp, "path", c->path, 0);
 	credential_write_item(fp, "username", c->username, 0);
 	credential_write_item(fp, "password", c->password, 0);
-	credential_write_item(fp, "credential", c->credential, 0);
 	credential_write_item(fp, "oauth_refresh_token", c->oauth_refresh_token, 0);
 	if (c->password_expiry_utc != TIME_MAX) {
 		char *s = xstrfmt("%"PRItime, c->password_expiry_utc);
@@ -304,7 +353,6 @@ void credential_write(const struct credential *c, FILE *fp)
 	}
 	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
 		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
-	credential_write_item(fp, "authtype", c->authtype, 0);
 }
 
 static int run_credential_helper(struct credential *c,
@@ -327,14 +375,14 @@ static int run_credential_helper(struct credential *c,
 
 	fp = xfdopen(helper.in, "w");
 	sigchain_push(SIGPIPE, SIG_IGN);
-	credential_write(c, fp);
+	credential_write(c, fp, want_output ? CREDENTIAL_OP_HELPER : CREDENTIAL_OP_RESPONSE);
 	fclose(fp);
 	sigchain_pop(SIGPIPE);
 
 	if (want_output) {
 		int r;
 		fp = xfdopen(helper.out, "r");
-		r = credential_read(c, fp);
+		r = credential_read(c, fp, CREDENTIAL_OP_HELPER);
 		fclose(fp);
 		if (r < 0) {
 			finish_command(&helper);
@@ -367,7 +415,7 @@ static int credential_do(struct credential *c, const char *helper,
 	return r;
 }
 
-void credential_fill(struct credential *c)
+void credential_fill(struct credential *c, int all_capabilities)
 {
 	int i;
 
@@ -375,6 +423,8 @@ void credential_fill(struct credential *c)
 		return;
 
 	credential_apply_config(c);
+	if (all_capabilities)
+		credential_set_all_capabilities(c, CREDENTIAL_OP_INITIAL);
 
 	for (i = 0; i < c->helpers.nr; i++) {
 		credential_do(c, c->helpers.items[i].string, "get");
diff --git a/credential.h b/credential.h
index 9db892cf4d..b524fdba59 100644
--- a/credential.h
+++ b/credential.h
@@ -93,6 +93,27 @@
  * -----------------------------------------------------------------------
  */
 
+/*
+ * These values define the kind of operation we're performing and the
+ * capabilities at each stage.  The first is either an external request (via git
+ * credential fill) or an internal request (e.g., via the HTTP) code.  The
+ * second is the call to the credential helper, and the third is the response
+ * we're providing.
+ *
+ * At each stage, we will emit the capability only if the previous stage
+ * supported it.
+ */
+enum credential_op_type {
+	CREDENTIAL_OP_INITIAL  = 1,
+	CREDENTIAL_OP_HELPER   = 2,
+	CREDENTIAL_OP_RESPONSE = 3,
+};
+
+struct credential_capability {
+	unsigned request_initial:1,
+		 request_helper:1,
+		 response:1;
+};
 
 /**
  * This struct represents a single username/password combination
@@ -136,6 +157,8 @@ struct credential {
 		 use_http_path:1,
 		 username_from_proto:1;
 
+	struct credential_capability capa_authtype;
+
 	char *username;
 	char *password;
 	char *credential;
@@ -174,8 +197,11 @@ void credential_clear(struct credential *);
  * returns, the username and password fields of the credential are
  * guaranteed to be non-NULL. If an error occurs, the function will
  * die().
+ *
+ * If all_capabilities is set, this is an internal user that is prepared
+ * to deal with all known capabilities, and we should advertise that fact.
  */
-void credential_fill(struct credential *);
+void credential_fill(struct credential *, int all_capabilities);
 
 /**
  * Inform the credential subsystem that the provided credentials
@@ -198,8 +224,16 @@ void credential_approve(struct credential *);
  */
 void credential_reject(struct credential *);
 
-int credential_read(struct credential *, FILE *);
-void credential_write(const struct credential *, FILE *);
+/**
+ * Enable all of the supported credential flags in this credential.
+ */
+void credential_set_all_capabilities(struct credential *c,
+				     enum credential_op_type op_type);
+
+int credential_read(struct credential *, FILE *,
+		    enum credential_op_type);
+void credential_write(const struct credential *, FILE *,
+		      enum credential_op_type);
 
 /*
  * Parse a url into a credential struct, replacing any existing contents.
diff --git a/http.c b/http.c
index 22639b0974..54ddff03fb 100644
--- a/http.c
+++ b/http.c
@@ -569,7 +569,7 @@ static void init_curl_http_auth(CURL *result)
 		return;
 	}
 
-	credential_fill(&http_auth);
+	credential_fill(&http_auth, 0);
 
 	curl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);
 	curl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);
@@ -596,7 +596,7 @@ static void init_curl_proxy_auth(CURL *result)
 {
 	if (proxy_auth.username) {
 		if (!proxy_auth.password)
-			credential_fill(&proxy_auth);
+			credential_fill(&proxy_auth, 0);
 		set_proxyauth_name_password(result);
 	}
 
@@ -630,7 +630,7 @@ static int has_cert_password(void)
 		cert_auth.host = xstrdup("");
 		cert_auth.username = xstrdup("");
 		cert_auth.path = xstrdup(ssl_cert);
-		credential_fill(&cert_auth);
+		credential_fill(&cert_auth, 0);
 	}
 	return 1;
 }
@@ -645,7 +645,7 @@ static int has_proxy_cert_password(void)
 		proxy_cert_auth.host = xstrdup("");
 		proxy_cert_auth.username = xstrdup("");
 		proxy_cert_auth.path = xstrdup(http_proxy_ssl_cert);
-		credential_fill(&proxy_cert_auth);
+		credential_fill(&proxy_cert_auth, 0);
 	}
 	return 1;
 }
@@ -2190,7 +2190,7 @@ static int http_request_reauth(const char *url,
 		BUG("Unknown http_request target");
 	}
 
-	credential_fill(&http_auth);
+	credential_fill(&http_auth, 0);
 
 	return http_request(url, result, target, options);
 }
diff --git a/imap-send.c b/imap-send.c
index f2e1947e63..8c89e866b6 100644
--- a/imap-send.c
+++ b/imap-send.c
@@ -944,7 +944,7 @@ static void server_fill_credential(struct imap_server_conf *srvc, struct credent
 	cred->username = xstrdup_or_null(srvc->user);
 	cred->password = xstrdup_or_null(srvc->pass);
 
-	credential_fill(cred);
+	credential_fill(cred, 1);
 
 	if (!srvc->user)
 		srvc->user = xstrdup(cred->username);
diff --git a/remote-curl.c b/remote-curl.c
index e37eaa17b7..f96bda2431 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -926,7 +926,7 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 		do {
 			err = probe_rpc(rpc, &results);
 			if (err == HTTP_REAUTH)
-				credential_fill(&http_auth);
+				credential_fill(&http_auth, 0);
 		} while (err == HTTP_REAUTH);
 		if (err != HTTP_OK)
 			return -1;
@@ -1044,7 +1044,7 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 	rpc->any_written = 0;
 	err = run_slot(slot, NULL);
 	if (err == HTTP_REAUTH && !large_request) {
-		credential_fill(&http_auth);
+		credential_fill(&http_auth, 0);
 		curl_slist_free_all(headers);
 		goto retry;
 	}
diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
index 400f6bdbca..daf330ddd8 100755
--- a/t/t0300-credentials.sh
+++ b/t/t0300-credentials.sh
@@ -12,7 +12,13 @@ test_expect_success 'setup helper scripts' '
 	IFS==
 	while read key value; do
 		echo >&2 "$whoami: $key=$value"
-		eval "$key=$value"
+		if test -z "${key%%*\[\]}"
+		then
+			key=${key%%\[\]}
+			eval "$key=\"\$$key $value\""
+		else
+			eval "$key=$value"
+		fi
 	done
 	IFS=$OIFS
 	EOF
@@ -35,6 +41,16 @@ test_expect_success 'setup helper scripts' '
 	test -z "$pass" || echo password=$pass
 	EOF
 
+	write_script git-credential-verbatim-cred <<-\EOF &&
+	authtype=$1; shift
+	credential=$1; shift
+	. ./dump
+	echo capability[]=authtype
+	test -z "${capability##*authtype*}" || exit 0
+	test -z "$authtype" || echo authtype=$authtype
+	test -z "$credential" || echo credential=$credential
+	EOF
+
 	write_script git-credential-verbatim-with-expiry <<-\EOF &&
 	user=$1; shift
 	pass=$1; shift
@@ -64,6 +80,26 @@ test_expect_success 'credential_fill invokes helper' '
 	EOF
 '
 
+test_expect_success 'credential_fill invokes helper with credential' '
+	check fill "verbatim-cred Bearer token" <<-\EOF
+	capability[]=authtype
+	protocol=http
+	host=example.com
+	--
+	capability[]=authtype
+	authtype=Bearer
+	credential=token
+	protocol=http
+	host=example.com
+	--
+	verbatim-cred: get
+	verbatim-cred: capability[]=authtype
+	verbatim-cred: protocol=http
+	verbatim-cred: host=example.com
+	EOF
+'
+
+
 test_expect_success 'credential_fill invokes multiple helpers' '
 	check fill useless "verbatim foo bar" <<-\EOF
 	protocol=http
@@ -83,6 +119,42 @@ test_expect_success 'credential_fill invokes multiple helpers' '
 	EOF
 '
 
+test_expect_success 'credential_fill response does not get capabilities when helpers are incapable' '
+	check fill useless "verbatim foo bar" <<-\EOF
+	capability[]=authtype
+	protocol=http
+	host=example.com
+	--
+	protocol=http
+	host=example.com
+	username=foo
+	password=bar
+	--
+	useless: get
+	useless: capability[]=authtype
+	useless: protocol=http
+	useless: host=example.com
+	verbatim: get
+	verbatim: capability[]=authtype
+	verbatim: protocol=http
+	verbatim: host=example.com
+	EOF
+'
+
+test_expect_success 'credential_fill response does not get capabilities when caller is incapable' '
+	check fill "verbatim-cred Bearer token" <<-\EOF
+	protocol=http
+	host=example.com
+	--
+	protocol=http
+	host=example.com
+	--
+	verbatim-cred: get
+	verbatim-cred: protocol=http
+	verbatim-cred: host=example.com
+	EOF
+'
+
 test_expect_success 'credential_fill stops when we get a full response' '
 	check fill "verbatim one two" "verbatim three four" <<-\EOF
 	protocol=http
@@ -99,6 +171,25 @@ test_expect_success 'credential_fill stops when we get a full response' '
 	EOF
 '
 
+test_expect_success 'credential_fill thinks a credential is a full response' '
+	check fill "verbatim-cred Bearer token" "verbatim three four" <<-\EOF
+	capability[]=authtype
+	protocol=http
+	host=example.com
+	--
+	capability[]=authtype
+	authtype=Bearer
+	credential=token
+	protocol=http
+	host=example.com
+	--
+	verbatim-cred: get
+	verbatim-cred: capability[]=authtype
+	verbatim-cred: protocol=http
+	verbatim-cred: host=example.com
+	EOF
+'
+
 test_expect_success 'credential_fill continues through partial response' '
 	check fill "verbatim one \"\"" "verbatim two three" <<-\EOF
 	protocol=http
@@ -175,6 +266,20 @@ test_expect_success 'credential_fill passes along metadata' '
 	EOF
 '
 
+test_expect_success 'credential_fill produces no credential without capability' '
+	check fill "verbatim-cred Bearer token" <<-\EOF
+	protocol=http
+	host=example.com
+	--
+	protocol=http
+	host=example.com
+	--
+	verbatim-cred: get
+	verbatim-cred: protocol=http
+	verbatim-cred: host=example.com
+	EOF
+'
+
 test_expect_success 'credential_approve calls all helpers' '
 	check approve useless "verbatim one two" <<-\EOF
 	protocol=http

[PATCH v2 06/16] credential: add a field called "ephemeral"

[brian m. carlson]

Now that we have support for a wide variety of types of authentication,
it's important to indicate to other credential helpers whether they
should store credentials, since not every credential helper may
intuitively understand all possible values of the authtype field.  Do so
with a boolean field called "ephemeral", to indicate whether the
credential is expected to be temporary.

For example, in HTTP Digest authentication, the Authorization header
value is based off a nonce.  It isn't useful to store this value
for later use because reusing the credential long term will not result
in successful authentication due to the nonce necessarily differing.

An additional case is potentially short-lived credentials, which may
last only a few hours.  It similarly wouldn't be helper for other
credential helpers to attempt to provide these much later.

We do still pass the value to "git credential store" or "git credential
erase", since it may be helpful to the original helper to know whether
the operation was successful.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 credential.c           |  4 ++++
 credential.h           |  1 +
 t/t0300-credentials.sh | 30 ++++++++++++++++++++++++++++++
 3 files changed, 35 insertions(+)

diff --git a/credential.c b/credential.c
index f5396629df..3531d74346 100644
--- a/credential.c
+++ b/credential.c
@@ -289,6 +289,8 @@ int credential_read(struct credential *c, FILE *fp,
 		} else if (!strcmp(key, "path")) {
 			free(c->path);
 			c->path = xstrdup(value);
+		} else if (!strcmp(key, "ephemeral")) {
+			c->ephemeral = !!git_config_bool("ephemeral", value);
 		} else if (!strcmp(key, "wwwauth[]")) {
 			strvec_push(&c->wwwauth_headers, value);
 		} else if (!strcmp(key, "capability[]") && !strcmp(value, "authtype")) {
@@ -339,6 +341,8 @@ void credential_write(const struct credential *c, FILE *fp,
 		credential_write_item(fp, "capability[]", "authtype", 0);
 		credential_write_item(fp, "authtype", c->authtype, 0);
 		credential_write_item(fp, "credential", c->credential, 0);
+		if (c->ephemeral)
+			credential_write_item(fp, "ephemeral", "1", 0);
 	}
 	credential_write_item(fp, "protocol", c->protocol, 1);
 	credential_write_item(fp, "host", c->host, 1);
diff --git a/credential.h b/credential.h
index b524fdba59..da2a4802b7 100644
--- a/credential.h
+++ b/credential.h
@@ -152,6 +152,7 @@ struct credential {
 	unsigned header_is_last_match:1;
 
 	unsigned approved:1,
+		 ephemeral:1,
 		 configured:1,
 		 quit:1,
 		 use_http_path:1,
diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
index daf330ddd8..eceb6bbfbe 100755
--- a/t/t0300-credentials.sh
+++ b/t/t0300-credentials.sh
@@ -51,6 +51,17 @@ test_expect_success 'setup helper scripts' '
 	test -z "$credential" || echo credential=$credential
 	EOF
 
+	write_script git-credential-verbatim-ephemeral <<-\EOF &&
+	authtype=$1; shift
+	credential=$1; shift
+	. ./dump
+	echo capability[]=authtype
+	test -z "${capability##*authtype*}" || exit 0
+	test -z "$authtype" || echo authtype=$authtype
+	test -z "$credential" || echo credential=$credential
+	echo "ephemeral=1"
+	EOF
+
 	write_script git-credential-verbatim-with-expiry <<-\EOF &&
 	user=$1; shift
 	pass=$1; shift
@@ -99,6 +110,25 @@ test_expect_success 'credential_fill invokes helper with credential' '
 	EOF
 '
 
+test_expect_success 'credential_fill invokes helper with ephemeral credential' '
+	check fill "verbatim-ephemeral Bearer token" <<-\EOF
+	capability[]=authtype
+	protocol=http
+	host=example.com
+	--
+	capability[]=authtype
+	authtype=Bearer
+	credential=token
+	ephemeral=1
+	protocol=http
+	host=example.com
+	--
+	verbatim-ephemeral: get
+	verbatim-ephemeral: capability[]=authtype
+	verbatim-ephemeral: protocol=http
+	verbatim-ephemeral: host=example.com
+	EOF
+'
 
 test_expect_success 'credential_fill invokes multiple helpers' '
 	check fill useless "verbatim foo bar" <<-\EOF

[PATCH v2 07/16] docs: indicate new credential protocol fields

[brian m. carlson]

Now that we have new fields (authtype and credential), let's document
them for users and credential helper implementers.

Indicate specifically what common values of authtype are and what values
are allowed.  Note that, while common, digest and NTLM authentication
are insecure because they require unsalted, uniterated password hashes
to be stored.

Tell users that they can continue to use a username and password even if
the new capability is supported.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/git-credential.txt | 49 +++++++++++++++++++++++++++++++-
 1 file changed, 48 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index 918a0aa42b..230ac4c2c3 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -178,6 +178,39 @@ empty string.
 Components which are missing from the URL (e.g., there is no
 username in the example above) will be left unset.
 
+`authtype`::
+	This indicates that the authentication scheme in question should be used.
+	Common values for HTTP and HTTPS include `basic`, `bearer`, and `digest`,
+	although the latter is insecure and should not be used.  If `credential`
+	is used, this may be set to an arbitrary string suitable for the protocol in
+	question (usually HTTP).
++
+This value should not be sent unless the appropriate capability (see below) is
+provided on input.
+
+`credential`::
+	The pre-encoded credential, suitable for the protocol in question (usually
+	HTTP).  If this key is sent, `authtype` is mandatory, and `username` and
+	`password` are not used.  For HTTP, Git concatenates the `authtype` value and
+	this value with a single space to determine the `Authorization` header.
++
+This value should not be sent unless the appropriate capability (see below) is
+provided on input.
+
+`ephemeral`::
+	This boolean value indicates, if true, that the value in the `credential`
+	field should not be saved by the credential helper because its usefulness is
+	limited in time.  For example, an HTTP Digest `credential` value is computed
+	using a nonce and reusing it will not result in successful authentication.
+	This may also be used for situations with short duration (e.g., 24-hour)
+	credentials.  The default value is false.
++
+The credential helper will still be invoked with `store` or `erase` so that it
+can determine whether the operation was successful.
++
+This value should not be sent unless the appropriate capability (see below) is
+provided on input.
+
 `wwwauth[]`::
 
 	When an HTTP response is received by Git that includes one or more
@@ -189,7 +222,21 @@ attribute 'wwwauth[]', where the order of the attributes is the same as
 they appear in the HTTP response. This attribute is 'one-way' from Git
 to pass additional information to credential helpers.
 
-Unrecognised attributes are silently discarded.
+`capability[]`::
+	This signals that the caller supports the capability in question.
+	This can be used to provide better, more specific data as part of the
+	protocol.
++
+The only capability currently supported is `authtype`, which indicates that the
+`authtype`, `credential`, and `ephemeral` values are understood.  It is not
+obligatory to use these values in such a case, but they should not be provided
+without this capability.
++
+Callers of `git credential` and credential helpers should emit the
+capabilities they support unconditionally, and Git will gracefully
+handle passing them on.
+
+Unrecognised attributes and capabilities are silently discarded.
 
 GIT
 ---

[PATCH v2 08/16] http: add support for authtype and credential

[brian m. carlson]

Now that we have the credential helper code set up to handle arbitrary
authentications schemes, let's add support for this in the HTTP code,
where we really want to use it.  If we're using this new functionality,
don't set a username and password, and instead set a header wherever
we'd normally do so, including for proxy authentication.

Since we can now handle this case, ask the credential helper to enable
the appropriate capabilities.

Finally, if we're using the authtype value, set "Expect: 100-continue".
Any type of authentication that requires multiple rounds (such as NTLM
or Kerberos) requires a 100 Continue (if we're larger than
http.postBuffer) because otherwise we send the pack data before we're
authenticated, the push gets a 401 response, and we can't rewind the
stream.  We don't know for certain what other custom schemes might
require this, the HTTP/1.1 standard has required handling this since
1999, the broken HTTP server for which we disabled this (Google's) is
now fixed and has been for some time, and libcurl has a 1-second
fallback in case the HTTP server is still broken.  In addition, it is
not unreasonable to require compliance with a 25-year old standard to
use new Git features.  For all of these reasons, do so here.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 http.c                      |  48 ++++++++++---
 http.h                      |   3 +
 remote-curl.c               |   4 +-
 t/t5563-simple-http-auth.sh | 133 ++++++++++++++++++++++++++++++++++++
 4 files changed, 176 insertions(+), 12 deletions(-)

diff --git a/http.c b/http.c
index 54ddff03fb..906eb098c8 100644
--- a/http.c
+++ b/http.c
@@ -561,18 +561,34 @@ static int curl_empty_auth_enabled(void)
 	return 0;
 }
 
+struct curl_slist *http_append_auth_header(const struct credential *c,
+					   struct curl_slist *headers)
+{
+	if (c->authtype && c->credential) {
+		struct strbuf auth = STRBUF_INIT;
+		strbuf_addf(&auth, "Authorization: %s %s",
+			    c->authtype, c->credential);
+		headers = curl_slist_append(headers, auth.buf);
+		strbuf_release(&auth);
+	}
+	return headers;
+}
+
 static void init_curl_http_auth(CURL *result)
 {
-	if (!http_auth.username || !*http_auth.username) {
+	if ((!http_auth.username || !*http_auth.username) &&
+	    (!http_auth.credential || !*http_auth.credential)) {
 		if (curl_empty_auth_enabled())
 			curl_easy_setopt(result, CURLOPT_USERPWD, ":");
 		return;
 	}
 
-	credential_fill(&http_auth, 0);
+	credential_fill(&http_auth, 1);
 
-	curl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);
-	curl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);
+	if (http_auth.password) {
+		curl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);
+		curl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);
+	}
 }
 
 /* *var must be free-able */
@@ -586,17 +602,22 @@ static void var_override(const char **var, char *value)
 
 static void set_proxyauth_name_password(CURL *result)
 {
+	if (proxy_auth.password) {
 		curl_easy_setopt(result, CURLOPT_PROXYUSERNAME,
 			proxy_auth.username);
 		curl_easy_setopt(result, CURLOPT_PROXYPASSWORD,
 			proxy_auth.password);
+	} else if (proxy_auth.authtype && proxy_auth.credential) {
+		curl_easy_setopt(result, CURLOPT_PROXYHEADER,
+				 http_append_auth_header(&proxy_auth, NULL));
+	}
 }
 
 static void init_curl_proxy_auth(CURL *result)
 {
 	if (proxy_auth.username) {
-		if (!proxy_auth.password)
-			credential_fill(&proxy_auth, 0);
+		if (!proxy_auth.password && !proxy_auth.credential)
+			credential_fill(&proxy_auth, 1);
 		set_proxyauth_name_password(result);
 	}
 
@@ -1468,7 +1489,7 @@ struct active_request_slot *get_active_slot(void)
 
 	curl_easy_setopt(slot->curl, CURLOPT_IPRESOLVE, git_curl_ipresolve);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);
-	if (http_auth.password || curl_empty_auth_enabled())
+	if (http_auth.password || http_auth.credential || curl_empty_auth_enabled())
 		init_curl_http_auth(slot->curl);
 
 	return slot;
@@ -1757,7 +1778,8 @@ static int handle_curl_result(struct slot_results *results)
 	} else if (missing_target(results))
 		return HTTP_MISSING_TARGET;
 	else if (results->http_code == 401) {
-		if (http_auth.username && http_auth.password) {
+		if ((http_auth.username && http_auth.password) ||\
+		    (http_auth.authtype && http_auth.credential)) {
 			credential_reject(&http_auth);
 			return HTTP_NOAUTH;
 		} else {
@@ -2065,11 +2087,15 @@ static int http_request(const char *url,
 	/* Add additional headers here */
 	if (options && options->extra_headers) {
 		const struct string_list_item *item;
-		for_each_string_list_item(item, options->extra_headers) {
-			headers = curl_slist_append(headers, item->string);
+		if (options && options->extra_headers) {
+			for_each_string_list_item(item, options->extra_headers) {
+				headers = curl_slist_append(headers, item->string);
+			}
 		}
 	}
 
+	headers = http_append_auth_header(&http_auth, headers);
+
 	curl_easy_setopt(slot->curl, CURLOPT_URL, url);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
 	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "");
@@ -2190,7 +2216,7 @@ static int http_request_reauth(const char *url,
 		BUG("Unknown http_request target");
 	}
 
-	credential_fill(&http_auth, 0);
+	credential_fill(&http_auth, 1);
 
 	return http_request(url, result, target, options);
 }
diff --git a/http.h b/http.h
index c5f8cc4620..a516ca4a9a 100644
--- a/http.h
+++ b/http.h
@@ -175,6 +175,9 @@ int http_get_file(const char *url, const char *filename,
 
 int http_fetch_ref(const char *base, struct ref *ref);
 
+struct curl_slist *http_append_auth_header(const struct credential *c,
+					   struct curl_slist *headers);
+
 /* Helpers for fetching packs */
 int http_get_info_packs(const char *base_url,
 			struct packed_git **packs_head);
diff --git a/remote-curl.c b/remote-curl.c
index f96bda2431..1c5416812a 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -931,7 +931,7 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 		if (err != HTTP_OK)
 			return -1;
 
-		if (results.auth_avail & CURLAUTH_GSSNEGOTIATE)
+		if (results.auth_avail & CURLAUTH_GSSNEGOTIATE || http_auth.authtype)
 			needs_100_continue = 1;
 	}
 
@@ -942,6 +942,8 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece
 	headers = curl_slist_append(headers, needs_100_continue ?
 		"Expect: 100-continue" : "Expect:");
 
+	headers = http_append_auth_header(&http_auth, headers);
+
 	/* Add Accept-Language header */
 	if (rpc->hdr_accept_language)
 		headers = curl_slist_append(headers, rpc->hdr_accept_language);
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index ab8a721ccc..b3ed0d9fc2 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -74,6 +74,7 @@ test_expect_success 'access using basic auth' '
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=Basic realm="example.com"
@@ -87,6 +88,43 @@ test_expect_success 'access using basic auth' '
 	EOF
 '
 
+test_expect_success 'access using basic auth via authtype' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	capability[]=authtype
+	authtype=Basic
+	credential=YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	# Basic base64(alice:secret-passwd)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	GIT_CURL_VERBOSE=1 git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	capability[]=authtype
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	capability[]=authtype
+	authtype=Basic
+	credential=YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	protocol=http
+	host=$HTTPD_DEST
+	EOF
+'
+
 test_expect_success 'access using basic auth invalid credentials' '
 	test_when_finished "per_test_cleanup" &&
 
@@ -108,6 +146,7 @@ test_expect_success 'access using basic auth invalid credentials' '
 	test_must_fail git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=Basic realm="example.com"
@@ -145,6 +184,7 @@ test_expect_success 'access using basic auth with extra challenges' '
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -183,6 +223,7 @@ test_expect_success 'access using basic auth mixed-case wwwauth header name' '
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=foobar param1="value1" param2="value2"
@@ -226,6 +267,7 @@ test_expect_success 'access using basic auth with wwwauth header continuations'
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -271,6 +313,7 @@ test_expect_success 'access using basic auth with wwwauth header empty continuat
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -312,6 +355,7 @@ test_expect_success 'access using basic auth with wwwauth header mixed line-endi
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
 
 	expect_credential_query get <<-EOF &&
+	capability[]=authtype
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -326,4 +370,93 @@ test_expect_success 'access using basic auth with wwwauth header mixed line-endi
 	EOF
 '
 
+test_expect_success 'access using bearer auth' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	capability[]=authtype
+	authtype=Bearer
+	credential=YS1naXQtdG9rZW4=
+	EOF
+
+	# Basic base64(a-git-token)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Bearer YS1naXQtdG9rZW4=
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: FooBar param1="value1" param2="value2"
+	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	capability[]=authtype
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query store <<-EOF
+	capability[]=authtype
+	authtype=Bearer
+	credential=YS1naXQtdG9rZW4=
+	protocol=http
+	host=$HTTPD_DEST
+	EOF
+'
+
+test_expect_success 'access using bearer auth with invalid credentials' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	capability[]=authtype
+	authtype=Bearer
+	credential=incorrect-token
+	EOF
+
+	# Basic base64(a-git-token)
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	Bearer YS1naXQtdG9rZW4=
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	WWW-Authenticate: FooBar param1="value1" param2="value2"
+	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	WWW-Authenticate: Basic realm="example.com"
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	test_must_fail git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	capability[]=authtype
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+
+	expect_credential_query erase <<-EOF
+	capability[]=authtype
+	authtype=Bearer
+	credential=incorrect-token
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=FooBar param1="value1" param2="value2"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	wwwauth[]=Basic realm="example.com"
+	EOF
+'
+
 test_done

[PATCH v2 09/16] credential: add an argument to keep state

[brian m. carlson]

Until now, our credential code has mostly deal with usernames and
passwords and we've let libcurl deal with the variant of authentication
to be used.  However, now that we have the credential value, the
credential helper can take control of the authentication, so the value
provided might be something that's generated, such as a Digest hash
value.

In such a case, it would be helpful for a credential helper that gets an
erase or store command to be able to keep track of an identifier for the
original secret that went into the computation.  Furthermore, some types
of authentication, such as NTLM and Kerberos, actually need two round
trips to authenticate, which will require that the credential helper
keep some state.

In order to allow for these use cases and others, allow storing state in
a field called "state[]".  This value is passed back to the credential
helper that created it, which avoids confusion caused by parsing values
from different helpers.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/git-credential.txt | 28 +++++++++++++++++++---------
 credential.c                     | 20 +++++++++++++++++---
 credential.h                     |  7 +++++++
 t/t0300-credentials.sh           | 28 ++++++++++++++++++++++++++++
 4 files changed, 71 insertions(+), 12 deletions(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index 230ac4c2c3..f63a8e0458 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -211,6 +211,15 @@ can determine whether the operation was successful.
 This value should not be sent unless the appropriate capability (see below) is
 provided on input.
 
+`state[]`::
+	This value provides an opaque state that will be passed back to this helper
+	if it is called again.  Each different credential helper may specify this
+	once.  The value should include a prefix unique to the credential helper and
+	should ignore values that don't match its prefix.
++
+This value should not be sent unless the appropriate capability (see below) is
+provided on input.
+
 `wwwauth[]`::
 
 	When an HTTP response is received by Git that includes one or more
@@ -223,18 +232,19 @@ they appear in the HTTP response. This attribute is 'one-way' from Git
 to pass additional information to credential helpers.
 
 `capability[]`::
-	This signals that the caller supports the capability in question.
-	This can be used to provide better, more specific data as part of the
+	This signals that Git, or the helper, as appropriate, supports the capability
+	in question.  This can be used to provide better, more specific data as part
+	of the protocol.  A `capability[]` directive must precede any value depending
+	on it and these directives _should_ be the first item announced in the
 	protocol.
 +
-The only capability currently supported is `authtype`, which indicates that the
-`authtype`, `credential`, and `ephemeral` values are understood.  It is not
-obligatory to use these values in such a case, but they should not be provided
-without this capability.
+There are two currently supported capabilities.  The first is `authtype`, which
+indicates that the `authtype`, `credential`, and `ephemeral` values are
+understood.  The second is `state`, which indicates that the `state[]` and
+`continue` values are understood.
 +
-Callers of `git credential` and credential helpers should emit the
-capabilities they support unconditionally, and Git will gracefully
-handle passing them on.
+It is not obligatory to use the additional features just because the capability
+is supported, but they should not be provided without the capability.
 
 Unrecognised attributes and capabilities are silently discarded.
 
diff --git a/credential.c b/credential.c
index 3531d74346..48826fb5a2 100644
--- a/credential.c
+++ b/credential.c
@@ -30,6 +30,7 @@ void credential_clear(struct credential *c)
 	free(c->authtype);
 	string_list_clear(&c->helpers, 0);
 	strvec_clear(&c->wwwauth_headers);
+	strvec_clear(&c->state_headers);
 
 	credential_init(c);
 }
@@ -293,8 +294,13 @@ int credential_read(struct credential *c, FILE *fp,
 			c->ephemeral = !!git_config_bool("ephemeral", value);
 		} else if (!strcmp(key, "wwwauth[]")) {
 			strvec_push(&c->wwwauth_headers, value);
-		} else if (!strcmp(key, "capability[]") && !strcmp(value, "authtype")) {
-			credential_set_capability(&c->capa_authtype, op_type);
+		} else if (!strcmp(key, "state[]")) {
+			strvec_push(&c->state_headers, value);
+		} else if (!strcmp(key, "capability[]")) {
+			if (!strcmp(value, "authtype"))
+				credential_set_capability(&c->capa_authtype, op_type);
+			else if (!strcmp(value, "state"))
+				credential_set_capability(&c->capa_state, op_type);
 		} else if (!strcmp(key, "password_expiry_utc")) {
 			errno = 0;
 			c->password_expiry_utc = parse_timestamp(value, NULL, 10);
@@ -337,8 +343,12 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,
 void credential_write(const struct credential *c, FILE *fp,
 		      enum credential_op_type op_type)
 {
-	if (credential_has_capability(&c->capa_authtype, op_type)) {
+	if (credential_has_capability(&c->capa_authtype, op_type))
 		credential_write_item(fp, "capability[]", "authtype", 0);
+	if (credential_has_capability(&c->capa_state, op_type))
+		credential_write_item(fp, "capability[]", "state", 0);
+
+	if (credential_has_capability(&c->capa_authtype, op_type)) {
 		credential_write_item(fp, "authtype", c->authtype, 0);
 		credential_write_item(fp, "credential", c->credential, 0);
 		if (c->ephemeral)
@@ -357,6 +367,10 @@ void credential_write(const struct credential *c, FILE *fp,
 	}
 	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
 		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
+	if (credential_has_capability(&c->capa_state, op_type)) {
+		for (size_t i = 0; i < c->state_headers.nr; i++)
+			credential_write_item(fp, "state[]", c->state_headers.v[i], 0);
+	}
 }
 
 static int run_credential_helper(struct credential *c,
diff --git a/credential.h b/credential.h
index da2a4802b7..c307300d12 100644
--- a/credential.h
+++ b/credential.h
@@ -144,6 +144,11 @@ struct credential {
 	 */
 	struct strvec wwwauth_headers;
 
+	/**
+	 * A `strvec` of state headers from credential helpers.
+	 */
+	struct strvec state_headers;
+
 	/**
 	 * Internal use only. Keeps track of if we previously matched against a
 	 * WWW-Authenticate header line in order to re-fold future continuation
@@ -159,6 +164,7 @@ struct credential {
 		 username_from_proto:1;
 
 	struct credential_capability capa_authtype;
+	struct credential_capability capa_state;
 
 	char *username;
 	char *password;
@@ -180,6 +186,7 @@ struct credential {
 	.helpers = STRING_LIST_INIT_DUP, \
 	.password_expiry_utc = TIME_MAX, \
 	.wwwauth_headers = STRVEC_INIT, \
+	.state_headers = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
index eceb6bbfbe..432f029d48 100755
--- a/t/t0300-credentials.sh
+++ b/t/t0300-credentials.sh
@@ -46,9 +46,12 @@ test_expect_success 'setup helper scripts' '
 	credential=$1; shift
 	. ./dump
 	echo capability[]=authtype
+	echo capability[]=state
 	test -z "${capability##*authtype*}" || exit 0
 	test -z "$authtype" || echo authtype=$authtype
 	test -z "$credential" || echo credential=$credential
+	test -z "${capability##*state*}" || exit 0
+	echo state[]=verbatim-cred:foo
 	EOF
 
 	write_script git-credential-verbatim-ephemeral <<-\EOF &&
@@ -129,6 +132,28 @@ test_expect_success 'credential_fill invokes helper with ephemeral credential' '
 	verbatim-ephemeral: host=example.com
 	EOF
 '
+test_expect_success 'credential_fill invokes helper with credential and state' '
+	check fill "verbatim-cred Bearer token" <<-\EOF
+	capability[]=authtype
+	capability[]=state
+	protocol=http
+	host=example.com
+	--
+	capability[]=authtype
+	capability[]=state
+	authtype=Bearer
+	credential=token
+	protocol=http
+	host=example.com
+	state[]=verbatim-cred:foo
+	--
+	verbatim-cred: get
+	verbatim-cred: capability[]=authtype
+	verbatim-cred: capability[]=state
+	verbatim-cred: protocol=http
+	verbatim-cred: host=example.com
+	EOF
+'
 
 test_expect_success 'credential_fill invokes multiple helpers' '
 	check fill useless "verbatim foo bar" <<-\EOF
@@ -152,6 +177,7 @@ test_expect_success 'credential_fill invokes multiple helpers' '
 test_expect_success 'credential_fill response does not get capabilities when helpers are incapable' '
 	check fill useless "verbatim foo bar" <<-\EOF
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=example.com
 	--
@@ -162,10 +188,12 @@ test_expect_success 'credential_fill response does not get capabilities when hel
 	--
 	useless: get
 	useless: capability[]=authtype
+	useless: capability[]=state
 	useless: protocol=http
 	useless: host=example.com
 	verbatim: get
 	verbatim: capability[]=authtype
+	verbatim: capability[]=state
 	verbatim: protocol=http
 	verbatim: host=example.com
 	EOF

[PATCH v2 10/16] credential: enable state capability

[brian m. carlson]

Now that we've implemented the state capability, let's send it along by
default when filling credentials so we can make use of it.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 credential.c                |  1 +
 t/t5563-simple-http-auth.sh | 10 ++++++++++
 2 files changed, 11 insertions(+)

diff --git a/credential.c b/credential.c
index 48826fb5a2..c93de92f65 100644
--- a/credential.c
+++ b/credential.c
@@ -56,6 +56,7 @@ void credential_set_all_capabilities(struct credential *c,
 				     enum credential_op_type op_type)
 {
 	credential_set_capability(&c->capa_authtype, op_type);
+	credential_set_capability(&c->capa_state, op_type);
 }
 
 int credential_match(const struct credential *want,
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index b3ed0d9fc2..b098cd0fdf 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -75,6 +75,7 @@ test_expect_success 'access using basic auth' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=Basic realm="example.com"
@@ -111,6 +112,7 @@ test_expect_success 'access using basic auth via authtype' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=Basic realm="example.com"
@@ -147,6 +149,7 @@ test_expect_success 'access using basic auth invalid credentials' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=Basic realm="example.com"
@@ -185,6 +188,7 @@ test_expect_success 'access using basic auth with extra challenges' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -224,6 +228,7 @@ test_expect_success 'access using basic auth mixed-case wwwauth header name' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=foobar param1="value1" param2="value2"
@@ -268,6 +273,7 @@ test_expect_success 'access using basic auth with wwwauth header continuations'
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -314,6 +320,7 @@ test_expect_success 'access using basic auth with wwwauth header empty continuat
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -356,6 +363,7 @@ test_expect_success 'access using basic auth with wwwauth header mixed line-endi
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -397,6 +405,7 @@ test_expect_success 'access using bearer auth' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"
@@ -440,6 +449,7 @@ test_expect_success 'access using bearer auth with invalid credentials' '
 
 	expect_credential_query get <<-EOF &&
 	capability[]=authtype
+	capability[]=state
 	protocol=http
 	host=$HTTPD_DEST
 	wwwauth[]=FooBar param1="value1" param2="value2"

[PATCH v2 11/16] docs: set a limit on credential line length

[brian m. carlson]

We recently introduced a way for credential helpers to add arbitrary
state as part of the protocol.  Set some limits on line length to avoid
helpers passing extremely large amounts of data.  While Git doesn't have
a fixed parsing length, there are other tools which support this
protocol and it's kind to allow them to use a reasonable fixed-size
buffer for parsing.  In addition, we would like to be moderate in our
memory usage and imposing reasonable limits is helpful for that purpose.

In the event a credential helper is incapable of storing its serialized
state in 64 KiB, it can feel free to serialize it on disk and store a
reference instead.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/git-credential.txt | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index f63a8e0458..4bbf2db9ca 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -111,7 +111,9 @@ attribute per line. Each attribute is specified by a key-value pair,
 separated by an `=` (equals) sign, followed by a newline.
 
 The key may contain any bytes except `=`, newline, or NUL. The value may
-contain any bytes except newline or NUL.
+contain any bytes except newline or NUL.  A line, including the trailing
+newline, may not exceed 65535 bytes in order to allow implementations to
+parse efficiently.
 
 Attributes with keys that end with C-style array brackets `[]` can have
 multiple values. Each instance of a multi-valued attribute forms an

[PATCH v2 12/16] t5563: refactor for multi-stage authentication

[brian m. carlson]

Some HTTP authentication schemes, such as NTLM- and Kerberos-based
options, require more than one round trip to authenticate.  Currently,
these can only be supported in libcurl, since Git does not have support
for this in the credential helper protocol.

However, in a future commit, we'll add support for this functionality
into the credential helper protocol and Git itself. Because we don't
really want to implement either NTLM or Kerberos, both of which are
complex protocols, we'll want to test this using a fake credential
authentication scheme.  In order to do so, update t5563 and its backend
to allow us to accept multiple sets of credentials and respond with
different behavior in each case.

Since we can now provide any number of possible status codes, provide a
non-specific reason phrase so we don't have to generate a more specific
one based on the response.  The reason phrase is mandatory according to
the status-line production in RFC 7230, but clients SHOULD ignore it,
and curl does (except to print it).

Each entry in the authorization and challenge fields contains an ID,
which indicates a corresponding credential and response.  If the
response is a 200 status, then we continue to execute git-http-backend.
Otherwise, we print the corresponding status and response.  If no ID is
matched, we use the default response with a status of 401.

Note that there is an implicit order to the parameters.  The ID is
always first and the creds or response value is always last, and
therefore may contain spaces, equals signs, or other arbitrary data.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/lib-httpd/nph-custom-auth.sh | 17 ++++--
 t/t5563-simple-http-auth.sh    | 96 +++++++++++++++++++---------------
 2 files changed, 66 insertions(+), 47 deletions(-)

diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh
index f5345e775e..d408d2caad 100644
--- a/t/lib-httpd/nph-custom-auth.sh
+++ b/t/lib-httpd/nph-custom-auth.sh
@@ -19,21 +19,30 @@ CHALLENGE_FILE=custom-auth.challenge
 #
 
 if test -n "$HTTP_AUTHORIZATION" && \
-	grep -Fqsx "${HTTP_AUTHORIZATION}" "$VALID_CREDS_FILE"
+	grep -Fqs "creds=${HTTP_AUTHORIZATION}" "$VALID_CREDS_FILE"
 then
+	idno=$(grep -F "creds=${HTTP_AUTHORIZATION}" "$VALID_CREDS_FILE" | sed -e 's/^id=\([a-z0-9-][a-z0-9-]*\) .*$/\1/')
+	status=$(sed -ne "s/^id=$idno.*status=\\([0-9][0-9][0-9]\\).*\$/\\1/p" "$CHALLENGE_FILE" | head -n1)
 	# Note that although git-http-backend returns a status line, it
 	# does so using a CGI 'Status' header. Because this script is an
 	# No Parsed Headers (NPH) script, we must return a real HTTP
 	# status line.
 	# This is only a test script, so we don't bother to check for
 	# the actual status from git-http-backend and always return 200.
-	echo 'HTTP/1.1 200 OK'
-	exec "$GIT_EXEC_PATH"/git-http-backend
+	echo "HTTP/1.1 $status Nonspecific Reason Phrase"
+	if test "$status" -eq 200
+	then
+		exec "$GIT_EXEC_PATH"/git-http-backend
+	else
+		sed -ne "s/^id=$idno.*response=//p" "$CHALLENGE_FILE"
+		echo
+		exit
+	fi
 fi
 
 echo 'HTTP/1.1 401 Authorization Required'
 if test -f "$CHALLENGE_FILE"
 then
-	cat "$CHALLENGE_FILE"
+	sed -ne 's/^id=default.*response=//p' "$CHALLENGE_FILE"
 fi
 echo
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index b098cd0fdf..515185ae00 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -63,11 +63,12 @@ test_expect_success 'access using basic auth' '
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -100,11 +101,12 @@ test_expect_success 'access using basic auth via authtype' '
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -137,11 +139,12 @@ test_expect_success 'access using basic auth invalid credentials' '
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -174,13 +177,14 @@ test_expect_success 'access using basic auth with extra challenges' '
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: FooBar param1="value1" param2="value2"
-	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: FooBar param1="value1" param2="value2"
+	id=default response=WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -214,13 +218,14 @@ test_expect_success 'access using basic auth mixed-case wwwauth header name' '
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	www-authenticate: foobar param1="value1" param2="value2"
-	WWW-AUTHENTICATE: BEARER authorize_uri="id.example.com" p=1 q=0
-	WwW-aUtHeNtIcAtE: baSiC realm="example.com"
+	id=1 status=200
+	id=default response=www-authenticate: foobar param1="value1" param2="value2"
+	id=default response=WWW-AUTHENTICATE: BEARER authorize_uri="id.example.com" p=1 q=0
+	id=default response=WwW-aUtHeNtIcAtE: baSiC realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -254,18 +259,19 @@ test_expect_success 'access using basic auth with wwwauth header continuations'
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	# Note that leading and trailing whitespace is important to correctly
 	# simulate a continuation/folded header.
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: FooBar param1="value1"
-	 param2="value2"
-	WWW-Authenticate: Bearer authorize_uri="id.example.com"
-	 p=1
-	 q=0
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: FooBar param1="value1"
+	id=default response= param2="value2"
+	id=default response=WWW-Authenticate: Bearer authorize_uri="id.example.com"
+	id=default response= p=1
+	id=default response= q=0
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -299,21 +305,22 @@ test_expect_success 'access using basic auth with wwwauth header empty continuat
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
 
 	# Note that leading and trailing whitespace is important to correctly
 	# simulate a continuation/folded header.
-	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >"$CHALLENGE" &&
-	printf " \r\n" >>"$CHALLENGE" &&
-	printf " param2=\"value2\"\r\n" >>"$CHALLENGE" &&
-	printf "WWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\r\n" >>"$CHALLENGE" &&
-	printf " p=1\r\n" >>"$CHALLENGE" &&
-	printf " \r\n" >>"$CHALLENGE" &&
-	printf " q=0\r\n" >>"$CHALLENGE" &&
-	printf "WWW-Authenticate: Basic realm=\"example.com\"\r\n" >>"$CHALLENGE" &&
+	printf "id=1 status=200\n" >"$CHALLENGE" &&
+	printf "id=default response=WWW-Authenticate: FooBar param1=\"value1\"\r\n" >>"$CHALLENGE" &&
+	printf "id=default response= \r\n" >>"$CHALLENGE" &&
+	printf "id=default response= param2=\"value2\"\r\n" >>"$CHALLENGE" &&
+	printf "id=default response=WWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\r\n" >>"$CHALLENGE" &&
+	printf "id=default response= p=1\r\n" >>"$CHALLENGE" &&
+	printf "id=default response= \r\n" >>"$CHALLENGE" &&
+	printf "id=default response= q=0\r\n" >>"$CHALLENGE" &&
+	printf "id=default response=WWW-Authenticate: Basic realm=\"example.com\"\r\n" >>"$CHALLENGE" &&
 
 	test_config_global credential.helper test-helper &&
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
@@ -346,17 +353,18 @@ test_expect_success 'access using basic auth with wwwauth header mixed line-endi
 
 	# Basic base64(alice:secret-passwd)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
+	id=1 creds=Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==
 	EOF
 
 	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
 
 	# Note that leading and trailing whitespace is important to correctly
 	# simulate a continuation/folded header.
-	printf "WWW-Authenticate: FooBar param1=\"value1\"\r\n" >"$CHALLENGE" &&
-	printf " \r\n" >>"$CHALLENGE" &&
-	printf "\tparam2=\"value2\"\r\n" >>"$CHALLENGE" &&
-	printf "WWW-Authenticate: Basic realm=\"example.com\"" >>"$CHALLENGE" &&
+	printf "id=1 status=200\n" >"$CHALLENGE" &&
+	printf "id=default response=WWW-Authenticate: FooBar param1=\"value1\"\r\n" >>"$CHALLENGE" &&
+	printf "id=default response= \r\n" >>"$CHALLENGE" &&
+	printf "id=default response=\tparam2=\"value2\"\r\n" >>"$CHALLENGE" &&
+	printf "id=default response=WWW-Authenticate: Basic realm=\"example.com\"" >>"$CHALLENGE" &&
 
 	test_config_global credential.helper test-helper &&
 	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
@@ -389,15 +397,16 @@ test_expect_success 'access using bearer auth' '
 
 	# Basic base64(a-git-token)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Bearer YS1naXQtdG9rZW4=
+	id=1 creds=Bearer YS1naXQtdG9rZW4=
 	EOF
 
 	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: FooBar param1="value1" param2="value2"
-	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: FooBar param1="value1" param2="value2"
+	id=default response=WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&
@@ -433,15 +442,16 @@ test_expect_success 'access using bearer auth with invalid credentials' '
 
 	# Basic base64(a-git-token)
 	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
-	Bearer YS1naXQtdG9rZW4=
+	id=1 creds=Bearer YS1naXQtdG9rZW4=
 	EOF
 
 	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
 
 	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
-	WWW-Authenticate: FooBar param1="value1" param2="value2"
-	WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
-	WWW-Authenticate: Basic realm="example.com"
+	id=1 status=200
+	id=default response=WWW-Authenticate: FooBar param1="value1" param2="value2"
+	id=default response=WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	id=default response=WWW-Authenticate: Basic realm="example.com"
 	EOF
 
 	test_config_global credential.helper test-helper &&

[PATCH v2 13/16] credential: add support for multistage credential rounds

[brian m. carlson]

Over HTTP, NTLM and Kerberos require two rounds of authentication on the
client side.  It's possible that there are custom authentication schemes
that also implement this same approach.  Since these are tricky schemes
to implement and the HTTP library in use may not always handle them
gracefully on all systems, it would be helpful to allow the credential
helper to implement them instead for increased portability and
robustness.

To allow this to happen, add a boolean flag, continue, that indicates
that instead of failing when we get a 401, we should retry another round
of authentication.  However, this necessitates some changes in our
current credential code so that we can make this work.

Keep the state[] headers between iterations, but only use them to send
to the helper and only consider the new ones we read from the credential
helper to be valid on subsequent iterations.  That avoids us passing
stale data when we finally approve or reject the credential.  Similarly,
clear the multistage and wwwauth[] values appropriately so that we
don't pass stale data or think we're trying a multiround response when
we're not.  Remove the credential values so that we can actually fill a
second time with new responses.

Limit the number of iterations of reauthentication we do to 3.  This
means that if there's a problem, we'll terminate with an error message
instead of retrying indefinitely and not informing the user (and
possibly conducting a DoS on the server).

In our tests, handle creating multiple response output files from our
helper so we can verify that each of the messages sent is correct.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/git-credential.txt | 13 +++++
 builtin/credential.c             |  1 +
 credential.c                     | 32 ++++++++++--
 credential.h                     | 27 +++++++++-
 http.c                           | 59 +++++++++++----------
 t/t5563-simple-http-auth.sh      | 89 ++++++++++++++++++++++++++++++--
 6 files changed, 185 insertions(+), 36 deletions(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index 4bbf2db9ca..3d3accc273 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -222,6 +222,19 @@ provided on input.
 This value should not be sent unless the appropriate capability (see below) is
 provided on input.
 
+`continue`::
+	This is a boolean value, which, if enabled, indicates that this
+	authentication is a non-final part of a multistage authentication step. This
+	is common in protocols such as NTLM and Kerberos, where two rounds of client
+	authentication are required, and setting this flag allows the credential
+	helper to implement the multistage authentication step.  This flag should
+	only be sent if a further stage is required; that is, if another round of
+	authentication is expected.
++
+This value should not be sent unless the appropriate capability (see below) is
+provided on input.  This attribute is 'one-way' from a credential helper to
+pass information to Git (or other programs invoking `git credential`).
+
 `wwwauth[]`::
 
 	When an HTTP response is received by Git that includes one or more
diff --git a/builtin/credential.c b/builtin/credential.c
index 643bf0b5e4..3568e57025 100644
--- a/builtin/credential.c
+++ b/builtin/credential.c
@@ -22,6 +22,7 @@ int cmd_credential(int argc, const char **argv, const char *prefix UNUSED)
 
 	if (!strcmp(op, "fill")) {
 		credential_fill(&c, 0);
+		credential_next_state(&c);
 		credential_write(&c, stdout, CREDENTIAL_OP_RESPONSE);
 	} else if (!strcmp(op, "approve")) {
 		credential_set_all_capabilities(&c, CREDENTIAL_OP_HELPER);
diff --git a/credential.c b/credential.c
index c93de92f65..98b040cf11 100644
--- a/credential.c
+++ b/credential.c
@@ -31,10 +31,23 @@ void credential_clear(struct credential *c)
 	string_list_clear(&c->helpers, 0);
 	strvec_clear(&c->wwwauth_headers);
 	strvec_clear(&c->state_headers);
+	strvec_clear(&c->state_headers_to_send);
 
 	credential_init(c);
 }
 
+void credential_next_state(struct credential *c)
+{
+	strvec_clear(&c->state_headers_to_send);
+	SWAP(c->state_headers, c->state_headers_to_send);
+}
+
+void credential_clear_secrets(struct credential *c)
+{
+	FREE_AND_NULL(c->password);
+	FREE_AND_NULL(c->credential);
+}
+
 static void credential_set_capability(struct credential_capability *capa,
 				      enum credential_op_type op_type)
 {
@@ -302,6 +315,8 @@ int credential_read(struct credential *c, FILE *fp,
 				credential_set_capability(&c->capa_authtype, op_type);
 			else if (!strcmp(value, "state"))
 				credential_set_capability(&c->capa_state, op_type);
+		} else if (!strcmp(key, "continue")) {
+			c->multistage = !!git_config_bool("continue", value);
 		} else if (!strcmp(key, "password_expiry_utc")) {
 			errno = 0;
 			c->password_expiry_utc = parse_timestamp(value, NULL, 10);
@@ -369,8 +384,10 @@ void credential_write(const struct credential *c, FILE *fp,
 	for (size_t i = 0; i < c->wwwauth_headers.nr; i++)
 		credential_write_item(fp, "wwwauth[]", c->wwwauth_headers.v[i], 0);
 	if (credential_has_capability(&c->capa_state, op_type)) {
-		for (size_t i = 0; i < c->state_headers.nr; i++)
-			credential_write_item(fp, "state[]", c->state_headers.v[i], 0);
+		if (c->multistage)
+			credential_write_item(fp, "continue", "1", 0);
+		for (size_t i = 0; i < c->state_headers_to_send.nr; i++)
+			credential_write_item(fp, "state[]", c->state_headers_to_send.v[i], 0);
 	}
 }
 
@@ -441,6 +458,9 @@ void credential_fill(struct credential *c, int all_capabilities)
 	if ((c->username && c->password) || c->credential)
 		return;
 
+	credential_next_state(c);
+	c->multistage = 0;
+
 	credential_apply_config(c);
 	if (all_capabilities)
 		credential_set_all_capabilities(c, CREDENTIAL_OP_INITIAL);
@@ -453,8 +473,10 @@ void credential_fill(struct credential *c, int all_capabilities)
 			/* Reset expiry to maintain consistency */
 			c->password_expiry_utc = TIME_MAX;
 		}
-		if ((c->username && c->password) || c->credential)
+		if ((c->username && c->password) || c->credential) {
+			strvec_clear(&c->wwwauth_headers);
 			return;
+		}
 		if (c->quit)
 			die("credential helper '%s' told us to quit",
 			    c->helpers.items[i].string);
@@ -474,6 +496,8 @@ void credential_approve(struct credential *c)
 	if (((!c->username || !c->password) && !c->credential) || c->password_expiry_utc < time(NULL))
 		return;
 
+	credential_next_state(c);
+
 	credential_apply_config(c);
 
 	for (i = 0; i < c->helpers.nr; i++)
@@ -485,6 +509,8 @@ void credential_reject(struct credential *c)
 {
 	int i;
 
+	credential_next_state(c);
+
 	credential_apply_config(c);
 
 	for (i = 0; i < c->helpers.nr; i++)
diff --git a/credential.h b/credential.h
index c307300d12..19163fc6a0 100644
--- a/credential.h
+++ b/credential.h
@@ -145,10 +145,15 @@ struct credential {
 	struct strvec wwwauth_headers;
 
 	/**
-	 * A `strvec` of state headers from credential helpers.
+	 * A `strvec` of state headers received from credential helpers.
 	 */
 	struct strvec state_headers;
 
+	/**
+	 * A `strvec` of state headers to send to credential helpers.
+	 */
+	struct strvec state_headers_to_send;
+
 	/**
 	 * Internal use only. Keeps track of if we previously matched against a
 	 * WWW-Authenticate header line in order to re-fold future continuation
@@ -159,6 +164,7 @@ struct credential {
 	unsigned approved:1,
 		 ephemeral:1,
 		 configured:1,
+		 multistage: 1,
 		 quit:1,
 		 use_http_path:1,
 		 username_from_proto:1;
@@ -187,6 +193,7 @@ struct credential {
 	.password_expiry_utc = TIME_MAX, \
 	.wwwauth_headers = STRVEC_INIT, \
 	.state_headers = STRVEC_INIT, \
+	.state_headers_to_send = STRVEC_INIT, \
 }
 
 /* Initialize a credential structure, setting all fields to empty. */
@@ -238,6 +245,24 @@ void credential_reject(struct credential *);
 void credential_set_all_capabilities(struct credential *c,
 				     enum credential_op_type op_type);
 
+/**
+ * Clear the secrets in this credential, but leave other data intact.
+ *
+ * This is useful for resetting credentials in preparation for a subsequent
+ * stage of filling.
+ */
+void credential_clear_secrets(struct credential *c);
+
+/**
+ * Prepares the credential for the next iteration of the helper protocol by
+ * updating the state headers to send with the ones read by the last iteration
+ * of the protocol.
+ *
+ * Except for internal callers, this should be called exactly once between
+ * reading credentials with `credential_fill` and writing them.
+ */
+void credential_next_state(struct credential *c);
+
 int credential_read(struct credential *, FILE *,
 		    enum credential_op_type);
 void credential_write(const struct credential *, FILE *,
diff --git a/http.c b/http.c
index 906eb098c8..9a514404d0 100644
--- a/http.c
+++ b/http.c
@@ -1780,6 +1780,10 @@ static int handle_curl_result(struct slot_results *results)
 	else if (results->http_code == 401) {
 		if ((http_auth.username && http_auth.password) ||\
 		    (http_auth.authtype && http_auth.credential)) {
+			if (http_auth.multistage) {
+				credential_clear_secrets(&http_auth);
+				return HTTP_REAUTH;
+			}
 			credential_reject(&http_auth);
 			return HTTP_NOAUTH;
 		} else {
@@ -2177,6 +2181,7 @@ static int http_request_reauth(const char *url,
 			       void *result, int target,
 			       struct http_get_options *options)
 {
+	int i = 3;
 	int ret = http_request(url, result, target, options);
 
 	if (ret != HTTP_OK && ret != HTTP_REAUTH)
@@ -2190,35 +2195,35 @@ static int http_request_reauth(const char *url,
 		}
 	}
 
-	if (ret != HTTP_REAUTH)
-		return ret;
+	while (ret == HTTP_REAUTH && --i) {
+		/*
+		 * The previous request may have put cruft into our output stream; we
+		 * should clear it out before making our next request.
+		 */
+		switch (target) {
+		case HTTP_REQUEST_STRBUF:
+			strbuf_reset(result);
+			break;
+		case HTTP_REQUEST_FILE:
+			if (fflush(result)) {
+				error_errno("unable to flush a file");
+				return HTTP_START_FAILED;
+			}
+			rewind(result);
+			if (ftruncate(fileno(result), 0) < 0) {
+				error_errno("unable to truncate a file");
+				return HTTP_START_FAILED;
+			}
+			break;
+		default:
+			BUG("Unknown http_request target");
+		}
 
-	/*
-	 * The previous request may have put cruft into our output stream; we
-	 * should clear it out before making our next request.
-	 */
-	switch (target) {
-	case HTTP_REQUEST_STRBUF:
-		strbuf_reset(result);
-		break;
-	case HTTP_REQUEST_FILE:
-		if (fflush(result)) {
-			error_errno("unable to flush a file");
-			return HTTP_START_FAILED;
-		}
-		rewind(result);
-		if (ftruncate(fileno(result), 0) < 0) {
-			error_errno("unable to truncate a file");
-			return HTTP_START_FAILED;
-		}
-		break;
-	default:
-		BUG("Unknown http_request target");
+		credential_fill(&http_auth, 1);
+
+		ret = http_request(url, result, target, options);
 	}
-
-	credential_fill(&http_auth, 1);
-
-	return http_request(url, result, target, options);
+	return ret;
 }
 
 int http_get_strbuf(const char *url,
diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh
index 515185ae00..5d5caa3f58 100755
--- a/t/t5563-simple-http-auth.sh
+++ b/t/t5563-simple-http-auth.sh
@@ -21,9 +21,17 @@ test_expect_success 'setup_credential_helper' '
 	CREDENTIAL_HELPER="$TRASH_DIRECTORY/bin/git-credential-test-helper" &&
 	write_script "$CREDENTIAL_HELPER" <<-\EOF
 	cmd=$1
-	teefile=$cmd-query.cred
+	teefile=$cmd-query-temp.cred
 	catfile=$cmd-reply.cred
 	sed -n -e "/^$/q" -e "p" >>$teefile
+	state=$(sed -ne "s/^state\[\]=helper://p" "$teefile")
+	if test -z "$state"
+	then
+		mv "$teefile" "$cmd-query.cred"
+	else
+		mv "$teefile" "$cmd-query-$state.cred"
+		catfile="$cmd-reply-$state.cred"
+	fi
 	if test "$cmd" = "get"
 	then
 		cat $catfile
@@ -32,13 +40,15 @@ test_expect_success 'setup_credential_helper' '
 '
 
 set_credential_reply () {
-	cat >"$TRASH_DIRECTORY/$1-reply.cred"
+	local suffix="$(test -n "$2" && echo "-$2")"
+	cat >"$TRASH_DIRECTORY/$1-reply$suffix.cred"
 }
 
 expect_credential_query () {
-	cat >"$TRASH_DIRECTORY/$1-expect.cred" &&
-	test_cmp "$TRASH_DIRECTORY/$1-expect.cred" \
-		 "$TRASH_DIRECTORY/$1-query.cred"
+	local suffix="$(test -n "$2" && echo "-$2")"
+	cat >"$TRASH_DIRECTORY/$1-expect$suffix.cred" &&
+	test_cmp "$TRASH_DIRECTORY/$1-expect$suffix.cred" \
+		 "$TRASH_DIRECTORY/$1-query$suffix.cred"
 }
 
 per_test_cleanup () {
@@ -479,4 +489,73 @@ test_expect_success 'access using bearer auth with invalid credentials' '
 	EOF
 '
 
+test_expect_success 'access using three-legged auth' '
+	test_when_finished "per_test_cleanup" &&
+
+	set_credential_reply get <<-EOF &&
+	capability[]=authtype
+	capability[]=state
+	authtype=Multistage
+	credential=YS1naXQtdG9rZW4=
+	state[]=helper:foobar
+	continue=1
+	EOF
+
+	set_credential_reply get foobar <<-EOF &&
+	capability[]=authtype
+	capability[]=state
+	authtype=Multistage
+	credential=YW5vdGhlci10b2tlbg==
+	state[]=helper:bazquux
+	EOF
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.valid" <<-EOF &&
+	id=1 creds=Multistage YS1naXQtdG9rZW4=
+	id=2 creds=Multistage YW5vdGhlci10b2tlbg==
+	EOF
+
+	CHALLENGE="$HTTPD_ROOT_PATH/custom-auth.challenge" &&
+
+	cat >"$HTTPD_ROOT_PATH/custom-auth.challenge" <<-EOF &&
+	id=1 status=401 response=WWW-Authenticate: Multistage challenge="456"
+	id=1 status=401 response=WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	id=2 status=200
+	id=default response=WWW-Authenticate: Multistage challenge="123"
+	id=default response=WWW-Authenticate: Bearer authorize_uri="id.example.com" p=1 q=0
+	EOF
+
+	test_config_global credential.helper test-helper &&
+	git ls-remote "$HTTPD_URL/custom_auth/repo.git" &&
+
+	expect_credential_query get <<-EOF &&
+	capability[]=authtype
+	capability[]=state
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=Multistage challenge="123"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	EOF
+
+	expect_credential_query get foobar <<-EOF &&
+	capability[]=authtype
+	capability[]=state
+	authtype=Multistage
+	protocol=http
+	host=$HTTPD_DEST
+	wwwauth[]=Multistage challenge="456"
+	wwwauth[]=Bearer authorize_uri="id.example.com" p=1 q=0
+	state[]=helper:foobar
+	EOF
+
+	expect_credential_query store bazquux <<-EOF
+	capability[]=authtype
+	capability[]=state
+	authtype=Multistage
+	credential=YW5vdGhlci10b2tlbg==
+	protocol=http
+	host=$HTTPD_DEST
+	state[]=helper:bazquux
+	EOF
+'
+
 test_done

[PATCH v2 14/16] t: add credential tests for authtype

[brian m. carlson]

It's helpful to have some basic tests for credential helpers supporting
the authtype and credential fields.  Let's add some tests for this case
so that we can make sure newly supported helpers work correctly.

Note that we explicitly check that credential helpers can produce
different sets of authtype and credential values based on the username.
While the username is not used in the HTTP protocol with authtype and
credential, it can still be specified in the URL and thus may be part of
the protocol.  Additionally, because it is common for users to have
multiple accounts on one service (say, both personal and professional
accounts), it's very helpful to be able to store different credentials
for different accounts in the same helper, and that doesn't become less
useful if one is using, say, Bearer authentication instead of Basic.
Thus, credential helpers should be expected to support this
functionality as basic functionality, so verify here that they do so.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/lib-credential.sh | 123 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 123 insertions(+)

diff --git a/t/lib-credential.sh b/t/lib-credential.sh
index 44799c0d38..58b9c74060 100644
--- a/t/lib-credential.sh
+++ b/t/lib-credential.sh
@@ -538,6 +538,129 @@ helper_test_oauth_refresh_token() {
 	'
 }
 
+helper_test_authtype() {
+	HELPER=$1
+
+	test_expect_success "helper ($HELPER) stores authtype and credential" '
+		check approve $HELPER <<-\EOF
+		capability[]=authtype
+		authtype=Bearer
+		credential=random-token
+		protocol=https
+		host=git.example.com
+		EOF
+	'
+
+	test_expect_success "helper ($HELPER) gets authtype and credential" '
+		check fill $HELPER <<-\EOF
+		capability[]=authtype
+		protocol=https
+		host=git.example.com
+		--
+		capability[]=authtype
+		authtype=Bearer
+		credential=random-token
+		protocol=https
+		host=git.example.com
+		--
+		EOF
+	'
+
+	test_expect_success "helper ($HELPER) stores authtype and credential with username" '
+		check approve $HELPER <<-\EOF
+		capability[]=authtype
+		authtype=Bearer
+		credential=other-token
+		protocol=https
+		host=git.example.com
+		username=foobar
+		EOF
+	'
+
+	test_expect_success "helper ($HELPER) gets authtype and credential with username" '
+		check fill $HELPER <<-\EOF
+		capability[]=authtype
+		protocol=https
+		host=git.example.com
+		username=foobar
+		--
+		capability[]=authtype
+		authtype=Bearer
+		credential=other-token
+		protocol=https
+		host=git.example.com
+		username=foobar
+		--
+		EOF
+	'
+
+	test_expect_success "helper ($HELPER) does not get authtype and credential with different username" '
+		check fill $HELPER <<-\EOF
+		capability[]=authtype
+		protocol=https
+		host=git.example.com
+		username=barbaz
+		--
+		protocol=https
+		host=git.example.com
+		username=barbaz
+		password=askpass-password
+		--
+		askpass: Password for '\''https://barbaz@git.example.com'\'':
+		EOF
+	'
+
+	test_expect_success "helper ($HELPER) does not store ephemeral authtype and credential" '
+		check approve $HELPER <<-\EOF &&
+		capability[]=authtype
+		authtype=Bearer
+		credential=git2-token
+		protocol=https
+		host=git2.example.com
+		ephemeral=1
+		EOF
+
+		check fill $HELPER <<-\EOF
+		capability[]=authtype
+		protocol=https
+		host=git2.example.com
+		--
+		protocol=https
+		host=git2.example.com
+		username=askpass-username
+		password=askpass-password
+		--
+		askpass: Username for '\''https://git2.example.com'\'':
+		askpass: Password for '\''https://askpass-username@git2.example.com'\'':
+		EOF
+	'
+
+	test_expect_success "helper ($HELPER) does not store ephemeral username and password" '
+		check approve $HELPER <<-\EOF &&
+		capability[]=authtype
+		protocol=https
+		host=git2.example.com
+		user=barbaz
+		password=secret
+		ephemeral=1
+		EOF
+
+		check fill $HELPER <<-\EOF
+		capability[]=authtype
+		protocol=https
+		host=git2.example.com
+		--
+		protocol=https
+		host=git2.example.com
+		username=askpass-username
+		password=askpass-password
+		--
+		askpass: Username for '\''https://git2.example.com'\'':
+		askpass: Password for '\''https://askpass-username@git2.example.com'\'':
+		EOF
+	'
+}
+
 write_script askpass <<\EOF
 echo >&2 askpass: $*
 what=$(echo $1 | cut -d" " -f1 | tr A-Z a-z | tr -cd a-z)

[PATCH v2 15/16] credential-cache: implement authtype capability

[brian m. carlson]

Now that we have full support in Git for the authtype capability, let's
add support to the cache credential helper.

When parsing data, we always set the initial capabilities because we're
the helper, and we need both the initial and helper capabilities to be
set in order to have the helper capabilities take effect.

When emitting data, always emit the supported capability and make sure
we emit items only if we have them and they're supported by the caller.
Since we may no longer have a username or password, be sure to emit
those conditionally as well so we don't segfault on a NULL pointer.
Similarly, when comparing credentials, consider both the password and
credential fields when we're matching passwords.

Adjust the partial credential detection code so that we can store
credentials missing a username or password as long as they have an
authtype and credential.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 builtin/credential-cache--daemon.c | 20 +++++++++++++++++---
 credential.c                       |  7 ++++---
 credential.h                       |  6 ++++++
 t/t0301-credential-cache.sh        |  1 +
 4 files changed, 28 insertions(+), 6 deletions(-)

diff --git a/builtin/credential-cache--daemon.c b/builtin/credential-cache--daemon.c
index ccbcf99ac1..6ffedcd17c 100644
--- a/builtin/credential-cache--daemon.c
+++ b/builtin/credential-cache--daemon.c
@@ -115,6 +115,8 @@ static int read_request(FILE *fh, struct credential *c,
 		return error("client sent bogus timeout line: %s", item.buf);
 	*timeout = atoi(p);
 
+	credential_set_all_capabilities(c, CREDENTIAL_OP_INITIAL);
+
 	if (credential_read(c, fh, CREDENTIAL_OP_HELPER) < 0)
 		return -1;
 	return 0;
@@ -131,8 +133,18 @@ static void serve_one_client(FILE *in, FILE *out)
 	else if (!strcmp(action.buf, "get")) {
 		struct credential_cache_entry *e = lookup_credential(&c);
 		if (e) {
-			fprintf(out, "username=%s\n", e->item.username);
-			fprintf(out, "password=%s\n", e->item.password);
+			e->item.capa_authtype.request_initial = 1;
+			e->item.capa_authtype.request_helper = 1;
+
+			fprintf(out, "capability[]=authtype\n");
+			if (e->item.username)
+				fprintf(out, "username=%s\n", e->item.username);
+			if (e->item.password)
+				fprintf(out, "password=%s\n", e->item.password);
+			if (credential_has_capability(&c.capa_authtype, CREDENTIAL_OP_HELPER) && e->item.authtype)
+				fprintf(out, "authtype=%s\n", e->item.authtype);
+			if (credential_has_capability(&c.capa_authtype, CREDENTIAL_OP_HELPER) && e->item.credential)
+				fprintf(out, "credential=%s\n", e->item.credential);
 			if (e->item.password_expiry_utc != TIME_MAX)
 				fprintf(out, "password_expiry_utc=%"PRItime"\n",
 					e->item.password_expiry_utc);
@@ -157,8 +169,10 @@ static void serve_one_client(FILE *in, FILE *out)
 	else if (!strcmp(action.buf, "store")) {
 		if (timeout < 0)
 			warning("cache client didn't specify a timeout");
-		else if (!c.username || !c.password)
+		else if ((!c.username || !c.password) && (!c.authtype && !c.credential))
 			warning("cache client gave us a partial credential");
+		else if (c.ephemeral)
+			warning("not storing ephemeral credential");
 		else {
 			remove_credential(&c, 0);
 			cache_credential(&c, timeout);
diff --git a/credential.c b/credential.c
index 98b040cf11..ffaf31499e 100644
--- a/credential.c
+++ b/credential.c
@@ -80,7 +80,8 @@ int credential_match(const struct credential *want,
 	       CHECK(host) &&
 	       CHECK(path) &&
 	       CHECK(username) &&
-	       (!match_password || CHECK(password));
+	       (!match_password || CHECK(password)) &&
+	       (!match_password || CHECK(credential));
 #undef CHECK
 }
 
@@ -248,8 +249,8 @@ static void credential_getpass(struct credential *c)
 						 PROMPT_ASKPASS);
 }
 
-static int credential_has_capability(const struct credential_capability *capa,
-				     enum credential_op_type op_type)
+int credential_has_capability(const struct credential_capability *capa,
+			      enum credential_op_type op_type)
 {
 	/*
 	 * We're checking here if each previous step indicated that we had the
diff --git a/credential.h b/credential.h
index 19163fc6a0..f8df10937c 100644
--- a/credential.h
+++ b/credential.h
@@ -263,6 +263,12 @@ void credential_clear_secrets(struct credential *c);
  */
 void credential_next_state(struct credential *c);
 
+/**
+ * Return true if the capability is enabled for an operation of op_type.
+ */
+int credential_has_capability(const struct credential_capability *capa,
+			      enum credential_op_type op_type);
+
 int credential_read(struct credential *, FILE *,
 		    enum credential_op_type);
 void credential_write(const struct credential *, FILE *,
diff --git a/t/t0301-credential-cache.sh b/t/t0301-credential-cache.sh
index 8300faadea..3897353d55 100755
--- a/t/t0301-credential-cache.sh
+++ b/t/t0301-credential-cache.sh
@@ -31,6 +31,7 @@ test_atexit 'git credential-cache exit'
 helper_test cache
 helper_test_password_expiry_utc cache
 helper_test_oauth_refresh_token cache
+helper_test_authtype cache
 
 test_expect_success 'socket defaults to ~/.cache/git/credential/socket' '
 	test_when_finished "

[PATCH v2 16/16] credential: add method for querying capabilities

[brian m. carlson]

Right now, there's no specific way to determine whether a credential
helper or git credential itself supports a given set of capabilities.
It would be helpful to have such a way, so let's let credential helpers
and git credential take an argument, "capability", which has it list the
capabilities and a version number on standard output.

Specifically choose a format that is slightly different from regular
credential output and assume that no capabilities are supported if a
non-zero exit status occurs or the data deviates from the format.  It is
common for users to write small shell scripts as the argument to
credential.helper, which will almost never be designed to emit
capabilities.  We want callers to gracefully handle this case by
assuming that they are not capable of extended support because that is
almost certainly the case, and specifying the error behavior up front
does this and preserves backwards compatibility in a graceful way.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/git-credential.txt | 28 +++++++++++++++++++++++++++-
 builtin/credential-cache.c       | 10 ++++++++++
 builtin/credential.c             |  6 ++++++
 credential.c                     | 11 +++++++++++
 credential.h                     |  6 ++++++
 5 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt
index 3d3accc273..e41493292f 100644
--- a/Documentation/git-credential.txt
+++ b/Documentation/git-credential.txt
@@ -8,7 +8,7 @@ git-credential - Retrieve and store user credentials
 SYNOPSIS
 --------
 ------------------
-'git credential' (fill|approve|reject)
+'git credential' (fill|approve|reject|capability)
 ------------------
 
 DESCRIPTION
@@ -41,6 +41,9 @@ If the action is `reject`, git-credential will send the description to
 any configured credential helpers, which may erase any stored
 credentials matching the description.
 
+If the action is `capability`, git-credential will announce any capabilities
+it supports to standard output.
+
 If the action is `approve` or `reject`, no output should be emitted.
 
 TYPICAL USE OF GIT CREDENTIAL
@@ -263,6 +266,29 @@ is supported, but they should not be provided without the capability.
 
 Unrecognised attributes and capabilities are silently discarded.
 
+[[CAPA-IOFMT]]
+CAPABILITY INPUT/OUTPUT FORMAT
+------------------------------
+
+For `git credential capability`, the format is slightly different. First, a
+`version 0` announcement is made to indicate the current version of the
+protocol, and then each capability is announced with a line like `capability
+authtype`. Credential helpers may also implement this format, again with the
+`capability` argument. Additional lines may be added in the future; callers
+should ignore lines which they don't understand.
+
+Because this is a new part of the credential helper protocol, older versions of
+Git, as well as some credential helpers, may not support it.  If a non-zero
+exit status is received, or if the first line doesn't start with the word
+`version` and a space, callers should assume that no capabilities are supported.
+
+The intention of this format is to differentiate it from the credential output
+in an unambiguous way.  It is possible to use very simple credential helpers
+(e.g., inline shell scripts) which always produce identical output.  Using a
+distinct format allows users to continue to use this syntax without having to
+worry about correctly implementing capability advertisements or accidentally
+confusing callers querying for capabilities.
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/builtin/credential-cache.c b/builtin/credential-cache.c
index bba96d4ffd..f5c989e2b2 100644
--- a/builtin/credential-cache.c
+++ b/builtin/credential-cache.c
@@ -1,4 +1,5 @@
 #include "builtin.h"
+#include "credential.h"
 #include "gettext.h"
 #include "parse-options.h"
 #include "path.h"
@@ -127,6 +128,13 @@ static char *get_socket_path(void)
 	return socket;
 }
 
+static void announce_capabilities(void)
+{
+	struct credential c = CREDENTIAL_INIT;
+	c.capa_authtype.request_initial = 1;
+	credential_announce_capabilities(&c, stdout);
+}
+
 int cmd_credential_cache(int argc, const char **argv, const char *prefix)
 {
 	char *socket_path = NULL;
@@ -160,6 +168,8 @@ int cmd_credential_cache(int argc, const char **argv, const char *prefix)
 		do_cache(socket_path, op, timeout, FLAG_RELAY);
 	else if (!strcmp(op, "store"))
 		do_cache(socket_path, op, timeout, FLAG_RELAY|FLAG_SPAWN);
+	else if (!strcmp(op, "capability"))
+		announce_capabilities();
 	else
 		; /* ignore unknown operation */
 
diff --git a/builtin/credential.c b/builtin/credential.c
index 3568e57025..5100d441f2 100644
--- a/builtin/credential.c
+++ b/builtin/credential.c
@@ -17,6 +17,12 @@ int cmd_credential(int argc, const char **argv, const char *prefix UNUSED)
 		usage(usage_msg);
 	op = argv[1];
 
+	if (!strcmp(op, "capability")) {
+		credential_set_all_capabilities(&c, CREDENTIAL_OP_INITIAL);
+		credential_announce_capabilities(&c, stdout);
+		return 0;
+	}
+
 	if (credential_read(&c, stdin, CREDENTIAL_OP_INITIAL) < 0)
 		die("unable to read credential from stdin");
 
diff --git a/credential.c b/credential.c
index ffaf31499e..758528b291 100644
--- a/credential.c
+++ b/credential.c
@@ -72,6 +72,17 @@ void credential_set_all_capabilities(struct credential *c,
 	credential_set_capability(&c->capa_state, op_type);
 }
 
+static void announce_one(struct credential_capability *cc, const char *name, FILE *fp) {
+	if (cc->request_initial)
+		fprintf(fp, "capability %s\n", name);
+}
+
+void credential_announce_capabilities(struct credential *c, FILE *fp) {
+	fprintf(fp, "version 0\n");
+	announce_one(&c->capa_authtype, "authtype", fp);
+	announce_one(&c->capa_state, "state", fp);
+}
+
 int credential_match(const struct credential *want,
 		     const struct credential *have, int match_password)
 {
diff --git a/credential.h b/credential.h
index f8df10937c..af8c287ff2 100644
--- a/credential.h
+++ b/credential.h
@@ -253,6 +253,12 @@ void credential_set_all_capabilities(struct credential *c,
  */
 void credential_clear_secrets(struct credential *c);
 
+/**
+ * Print a list of supported capabilities and version numbers to standard
+ * output.
+ */
+void credential_announce_capabilities(struct credential *c, FILE *fp);
+
 /**
  * Prepares the credential for the next iteration of the helper protocol by
  * updating the state headers to send with the ones read by the last iteration