* [PATCH] transport: add core.allowProtocol config option @ 2016-11-02 22:20 Brandon Williams 2016-11-02 22:41 ` Stefan Beller ` (4 more replies) 0 siblings, 5 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-02 22:20 UTC (permalink / raw) To: git; +Cc: sbeller, Brandon Williams Add configuration option 'core.allowProtocol' to allow users to create a whitelist of allowed protocols for fetch/push/clone in their gitconfig. For git-submodule.sh, fallback to default whitelist only if the user hasn't explicitly set `GIT_ALLOW_PROTOCOL` or doesn't have a whitelist in their gitconfig. Signed-off-by: Brandon Williams <bmwill@google.com> --- Documentation/config.txt | 9 +++++++++ git-submodule.sh | 3 ++- transport.c | 2 +- 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index 27069ac..7f83e40 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -455,6 +455,15 @@ core.sshCommand:: the `GIT_SSH_COMMAND` environment variable and is overridden when the environment variable is set. +core.allowProtocol:: + Provide a colon-separated list of protocols which are allowed to be + used with fetch/push/clone. This is useful to restrict recursive + submodule initialization from an untrusted repository. Any protocol not + mentioned will be disallowed (i.e., this is a whitelist, not a + blacklist). If the variable is not set at all, all protocols are + enabled. If the `GIT_ALLOW_PROTOCOL` enviornment variable is set, it is + used as the protocol whitelist instead of this config option. + core.ignoreStat:: If true, Git will avoid using lstat() calls to detect if files have changed by setting the "assume-unchanged" bit for those tracked files diff --git a/git-submodule.sh b/git-submodule.sh index a024a13..ad94c75 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -27,7 +27,8 @@ cd_to_toplevel # # If the user has already specified a set of allowed protocols, # we assume they know what they're doing and use that instead. -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} +config_whitelist=$(git config core.allowProtocol) +: ${GIT_ALLOW_PROTOCOL=${config_whitelist:-file:git:http:https:ssh}} export GIT_ALLOW_PROTOCOL command= diff --git a/transport.c b/transport.c index d57e8de..b1098cd 100644 --- a/transport.c +++ b/transport.c @@ -652,7 +652,7 @@ static const struct string_list *protocol_whitelist(void) if (enabled < 0) { const char *v = getenv("GIT_ALLOW_PROTOCOL"); - if (v) { + if (v || !git_config_get_value("core.allowProtocol", &v)) { string_list_split(&allowed, v, ':', -1); string_list_sort(&allowed); enabled = 1; -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-02 22:20 [PATCH] transport: add core.allowProtocol config option Brandon Williams @ 2016-11-02 22:41 ` Stefan Beller 2016-11-02 22:47 ` Brandon Williams 2016-11-02 23:05 ` Jeff King ` (3 subsequent siblings) 4 siblings, 1 reply; 124+ messages in thread From: Stefan Beller @ 2016-11-02 22:41 UTC (permalink / raw) To: Brandon Williams; +Cc: git On Wed, Nov 2, 2016 at 3:20 PM, Brandon Williams <bmwill@google.com> wrote: > Add configuration option 'core.allowProtocol' to allow users to create a > whitelist of allowed protocols for fetch/push/clone in their gitconfig. > > For git-submodule.sh, fallback to default whitelist only if the user > hasn't explicitly set `GIT_ALLOW_PROTOCOL` or doesn't have a whitelist > in their gitconfig. > > Signed-off-by: Brandon Williams <bmwill@google.com> > --- > Documentation/config.txt | 9 +++++++++ > git-submodule.sh | 3 ++- > transport.c | 2 +- > 3 files changed, 12 insertions(+), 2 deletions(-) > > diff --git a/Documentation/config.txt b/Documentation/config.txt > index 27069ac..7f83e40 100644 > --- a/Documentation/config.txt > +++ b/Documentation/config.txt > @@ -455,6 +455,15 @@ core.sshCommand:: > the `GIT_SSH_COMMAND` environment variable and is overridden > when the environment variable is set. > > +core.allowProtocol:: > + Provide a colon-separated list of protocols which are allowed to be > + used with fetch/push/clone. ok. > This is useful to restrict recursive > + submodule initialization from an untrusted repository. ok. Though as a user submodules may not spring to mind immediately here. I think this is generally useful, too. e.g. an admin could put this in the system wide config to prevent certain protocols from being used. > Any protocol not > + mentioned will be disallowed For the regular fetch/clone/pull case. For the submodule case we still fall back to the hardcoded list of known good things? > (i.e., this is a whitelist, not a > + blacklist). That is very explicit, I'd drop it. However this inspires bike shedding on the name: What about core.protocolWhitelist instead? > If the variable is not set at all, all protocols are > + enabled. If the `GIT_ALLOW_PROTOCOL` enviornment variable is set, it is > + used as the protocol whitelist instead of this config option. So the env var is of higher priority than this config. > + > core.ignoreStat:: > If true, Git will avoid using lstat() calls to detect if files have > changed by setting the "assume-unchanged" bit for those tracked files > diff --git a/git-submodule.sh b/git-submodule.sh > index a024a13..ad94c75 100755 > --- a/git-submodule.sh > +++ b/git-submodule.sh > @@ -27,7 +27,8 @@ cd_to_toplevel > # > # If the user has already specified a set of allowed protocols, > # we assume they know what they're doing and use that instead. > -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} > +config_whitelist=$(git config core.allowProtocol) So first we lookup the configured protocols. > +: ${GIT_ALLOW_PROTOCOL=${config_whitelist:-file:git:http:https:ssh}} Then if they are not configured use the current hard coded white list. > export GIT_ALLOW_PROTOCOL > > command= > diff --git a/transport.c b/transport.c > index d57e8de..b1098cd 100644 > --- a/transport.c > +++ b/transport.c > @@ -652,7 +652,7 @@ static const struct string_list *protocol_whitelist(void) > > if (enabled < 0) { > const char *v = getenv("GIT_ALLOW_PROTOCOL"); > - if (v) { > + if (v || !git_config_get_value("core.allowProtocol", &v)) { This implementation matches what the config promised, I would think. Do we have any tests for this that could be extended? (Otherwise we'd maybe want to add a test for both the regular case as well as a forbidden submodule?) > string_list_split(&allowed, v, ':', -1); > string_list_sort(&allowed); > enabled = 1; > -- > 2.8.0.rc3.226.g39d4020 > ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-02 22:41 ` Stefan Beller @ 2016-11-02 22:47 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-02 22:47 UTC (permalink / raw) To: Stefan Beller; +Cc: git On 11/02, Stefan Beller wrote: > > This is useful to restrict recursive > > + submodule initialization from an untrusted repository. > > ok. Though as a user submodules may not spring to mind immediately here. > I think this is generally useful, too. e.g. an admin could put this in > the system wide > config to prevent certain protocols from being used. Oh I pretty much copied the description from what exists for `GIT_ALLOW_PROTOCOL` which included this bit about submodules. > > Any protocol not > > + mentioned will be disallowed > > For the regular fetch/clone/pull case. For the submodule case we still > fall back to > the hardcoded list of known good things? Yep! This is done by explicitly setting GIT_ALLOW_PROTOCOL to the hardcoded list if the user hasn't supplied a whitelist. > > > (i.e., this is a whitelist, not a > > + blacklist). > > That is very explicit, I'd drop it. However this inspires bike > shedding on the name: > What about core.protocolWhitelist instead? Simply to keep the name similar to the env variable that already exists for this functionality. > So the env var is of higher priority than this config. > > -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} > > +config_whitelist=$(git config core.allowProtocol) > > So first we lookup the configured protocols. > > > +: ${GIT_ALLOW_PROTOCOL=${config_whitelist:-file:git:http:https:ssh}} > > Then if they are not configured use the current hard coded white list. The lookup of the configured whitelist is done first but wont be used unless GIT_ALLOW_PROTOCOL is unset. If neither is set it will fallback to the hardcoded list. > Do we have any tests for this that could be extended? (Otherwise we'd > maybe want to add a test for both the regular case as well as a forbidden > submodule?) > I can write a couple tests for a v2 of the patch. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-02 22:20 [PATCH] transport: add core.allowProtocol config option Brandon Williams 2016-11-02 22:41 ` Stefan Beller @ 2016-11-02 23:05 ` Jeff King 2016-11-02 23:08 ` Jeff King 2016-11-02 23:33 ` Brandon Williams 2016-11-03 0:22 ` Jonathan Nieder ` (2 subsequent siblings) 4 siblings, 2 replies; 124+ messages in thread From: Jeff King @ 2016-11-02 23:05 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller On Wed, Nov 02, 2016 at 03:20:47PM -0700, Brandon Williams wrote: > Add configuration option 'core.allowProtocol' to allow users to create a > whitelist of allowed protocols for fetch/push/clone in their gitconfig. > > For git-submodule.sh, fallback to default whitelist only if the user > hasn't explicitly set `GIT_ALLOW_PROTOCOL` or doesn't have a whitelist > in their gitconfig. This says "what", but not "why". What's the use case? I can see somebody wanting to pare down the whitelist further (e.g., because they are carrying ssh credentials that they don't want to use on behalf of a malicious repo). But in general I'd expect this setting to be a function of the environment you're operating in, and not the on-disk config. Or is the intent to broaden it for cases where you have a clone that uses some non-standard protocol, and you want it to Just Work on subsequent recursive fetches? > +core.allowProtocol:: > + Provide a colon-separated list of protocols which are allowed to be > + used with fetch/push/clone. This is useful to restrict recursive > + submodule initialization from an untrusted repository. Any protocol not > + mentioned will be disallowed (i.e., this is a whitelist, not a > + blacklist). If the variable is not set at all, all protocols are > + enabled. If the `GIT_ALLOW_PROTOCOL` enviornment variable is set, it is > + used as the protocol whitelist instead of this config option. The "not set at all, all protocols are enabled" bit is not quite correct, is it? It is true for a top-level fetch, but not for submodule recursion (and especially since you are talking about submodule recursion immediately before, it is rather confusing). > --- a/git-submodule.sh > +++ b/git-submodule.sh > @@ -27,7 +27,8 @@ cd_to_toplevel > # > # If the user has already specified a set of allowed protocols, > # we assume they know what they're doing and use that instead. > -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} > +config_whitelist=$(git config core.allowProtocol) > +: ${GIT_ALLOW_PROTOCOL=${config_whitelist:-file:git:http:https:ssh}} The original uses "=" without a ":" so that an empty variable takes precedence over the stock list (i.e., allowing nothing). Would you want the same behavior for the config variable? I.e.: # this should probably allow nothing, right? git config core.allowProtocol "" I think you'd have to check the return code of "git config" to distinguish those cases. > diff --git a/transport.c b/transport.c > index d57e8de..b1098cd 100644 > --- a/transport.c > +++ b/transport.c > @@ -652,7 +652,7 @@ static const struct string_list *protocol_whitelist(void) > > if (enabled < 0) { > const char *v = getenv("GIT_ALLOW_PROTOCOL"); > - if (v) { > + if (v || !git_config_get_value("core.allowProtocol", &v)) { > string_list_split(&allowed, v, ':', -1); > string_list_sort(&allowed); > enabled = 1; I thought at first we'd have to deal with leaking "v", but "get_value" is the "raw" version that gives you the uninterpreted value. I think that means it may give you NULL, though if we see an implicit bool like: [core] allowProtocol That's nonsense, of course, but we would still segfault. I think the easiest way to test is: git -c core.allowProtocol fetch which seems to segfault for me with this patch. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-02 23:05 ` Jeff King @ 2016-11-02 23:08 ` Jeff King 2016-11-02 23:34 ` Brandon Williams 2016-11-02 23:33 ` Brandon Williams 1 sibling, 1 reply; 124+ messages in thread From: Jeff King @ 2016-11-02 23:08 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller On Wed, Nov 02, 2016 at 07:05:39PM -0400, Jeff King wrote: > > +core.allowProtocol:: > > + Provide a colon-separated list of protocols which are allowed to be > > + used with fetch/push/clone. This is useful to restrict recursive > > + submodule initialization from an untrusted repository. Any protocol not > > + mentioned will be disallowed (i.e., this is a whitelist, not a > > + blacklist). If the variable is not set at all, all protocols are > > + enabled. If the `GIT_ALLOW_PROTOCOL` enviornment variable is set, it is > > + used as the protocol whitelist instead of this config option. > > The "not set at all, all protocols are enabled" bit is not quite > correct, is it? It is true for a top-level fetch, but not for submodule > recursion (and especially since you are talking about submodule > recursion immediately before, it is rather confusing). Heh, just saw that you copied this straight from the discussion of GIT_ALLOW_PROTOCOL. What idiot wrote the original? :) It might be worth fixing both places (or possibly just fixing the original and phrasing this one as "If GIT_ALLOW_PROTOCOL is not set, use this as the default value; see git(1) for details"). -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-02 23:08 ` Jeff King @ 2016-11-02 23:34 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-02 23:34 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller On 11/02, Jeff King wrote: > On Wed, Nov 02, 2016 at 07:05:39PM -0400, Jeff King wrote: > > > > +core.allowProtocol:: > > > + Provide a colon-separated list of protocols which are allowed to be > > > + used with fetch/push/clone. This is useful to restrict recursive > > > + submodule initialization from an untrusted repository. Any protocol not > > > + mentioned will be disallowed (i.e., this is a whitelist, not a > > > + blacklist). If the variable is not set at all, all protocols are > > > + enabled. If the `GIT_ALLOW_PROTOCOL` enviornment variable is set, it is > > > + used as the protocol whitelist instead of this config option. > > > > The "not set at all, all protocols are enabled" bit is not quite > > correct, is it? It is true for a top-level fetch, but not for submodule > > recursion (and especially since you are talking about submodule > > recursion immediately before, it is rather confusing). > > Heh, just saw that you copied this straight from the discussion of > GIT_ALLOW_PROTOCOL. What idiot wrote the original? :) > > It might be worth fixing both places (or possibly just fixing the > original and phrasing this one as "If GIT_ALLOW_PROTOCOL is not set, use > this as the default value; see git(1) for details"). > > -Peff haha K I'll fix the original as well. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-02 23:05 ` Jeff King 2016-11-02 23:08 ` Jeff King @ 2016-11-02 23:33 ` Brandon Williams 2016-11-02 23:46 ` Brandon Williams 1 sibling, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-11-02 23:33 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller On 11/02, Jeff King wrote: > On Wed, Nov 02, 2016 at 03:20:47PM -0700, Brandon Williams wrote: > > > Add configuration option 'core.allowProtocol' to allow users to create a > > whitelist of allowed protocols for fetch/push/clone in their gitconfig. > > > > For git-submodule.sh, fallback to default whitelist only if the user > > hasn't explicitly set `GIT_ALLOW_PROTOCOL` or doesn't have a whitelist > > in their gitconfig. > > This says "what", but not "why". What's the use case? > > I can see somebody wanting to pare down the whitelist further (e.g., > because they are carrying ssh credentials that they don't want to use on > behalf of a malicious repo). But in general I'd expect this setting to > be a function of the environment you're operating in, and not the > on-disk config. > > Or is the intent to broaden it for cases where you have a clone that > uses some non-standard protocol, and you want it to Just Work on > subsequent recursive fetches? > > > +core.allowProtocol:: > > + Provide a colon-separated list of protocols which are allowed to be > > + used with fetch/push/clone. This is useful to restrict recursive > > + submodule initialization from an untrusted repository. Any protocol not > > + mentioned will be disallowed (i.e., this is a whitelist, not a > > + blacklist). If the variable is not set at all, all protocols are > > + enabled. If the `GIT_ALLOW_PROTOCOL` enviornment variable is set, it is > > + used as the protocol whitelist instead of this config option. > > The "not set at all, all protocols are enabled" bit is not quite > correct, is it? It is true for a top-level fetch, but not for submodule > recursion (and especially since you are talking about submodule > recursion immediately before, it is rather confusing). Yeah stefan mentioned this to me. I simply copied the documentaion from GIT_ALLOW_PROTOCOL, perhaps that should be updated as well? > > > --- a/git-submodule.sh > > +++ b/git-submodule.sh > > @@ -27,7 +27,8 @@ cd_to_toplevel > > # > > # If the user has already specified a set of allowed protocols, > > # we assume they know what they're doing and use that instead. > > -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} > > +config_whitelist=$(git config core.allowProtocol) > > +: ${GIT_ALLOW_PROTOCOL=${config_whitelist:-file:git:http:https:ssh}} > > The original uses "=" without a ":" so that an empty variable takes > precedence over the stock list (i.e., allowing nothing). Would you want > the same behavior for the config variable? I.e.: > > # this should probably allow nothing, right? > git config core.allowProtocol "" > > I think you'd have to check the return code of "git config" to > distinguish those cases. Oh, I didn't think of that case. That can be done easy enough, just makes the code a bit more verbose. > > > diff --git a/transport.c b/transport.c > > index d57e8de..b1098cd 100644 > > --- a/transport.c > > +++ b/transport.c > > @@ -652,7 +652,7 @@ static const struct string_list *protocol_whitelist(void) > > > > if (enabled < 0) { > > const char *v = getenv("GIT_ALLOW_PROTOCOL"); > > - if (v) { > > + if (v || !git_config_get_value("core.allowProtocol", &v)) { > > string_list_split(&allowed, v, ':', -1); > > string_list_sort(&allowed); > > enabled = 1; > > I thought at first we'd have to deal with leaking "v", but "get_value" > is the "raw" version that gives you the uninterpreted value. I think > that means it may give you NULL, though if we see an implicit bool like: > > [core] > allowProtocol > > That's nonsense, of course, but we would still segfault. I > think the easiest way to test is: > > git -c core.allowProtocol fetch > > which seems to segfault for me with this patch. what is the desired behavior when a user provides a config in a way that isn't intended? -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-02 23:33 ` Brandon Williams @ 2016-11-02 23:46 ` Brandon Williams 2016-11-03 0:08 ` Jeff King 0 siblings, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-11-02 23:46 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller > > > diff --git a/transport.c b/transport.c > > > index d57e8de..b1098cd 100644 > > > --- a/transport.c > > > +++ b/transport.c > > > @@ -652,7 +652,7 @@ static const struct string_list *protocol_whitelist(void) > > > > > > if (enabled < 0) { > > > const char *v = getenv("GIT_ALLOW_PROTOCOL"); > > > - if (v) { > > > + if (v || !git_config_get_value("core.allowProtocol", &v)) { > > > string_list_split(&allowed, v, ':', -1); > > > string_list_sort(&allowed); > > > enabled = 1; > > > > I thought at first we'd have to deal with leaking "v", but "get_value" > > is the "raw" version that gives you the uninterpreted value. I think > > that means it may give you NULL, though if we see an implicit bool like: > > > > [core] > > allowProtocol > > > > That's nonsense, of course, but we would still segfault. I > > think the easiest way to test is: > > > > git -c core.allowProtocol fetch > > > > which seems to segfault for me with this patch. > > what is the desired behavior when a user provides a config in a way that > isn't intended? oh...I can just drop in git_config_get_string_const() instead. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-02 23:46 ` Brandon Williams @ 2016-11-03 0:08 ` Jeff King 0 siblings, 0 replies; 124+ messages in thread From: Jeff King @ 2016-11-03 0:08 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller On Wed, Nov 02, 2016 at 04:46:13PM -0700, Brandon Williams wrote: > > > I thought at first we'd have to deal with leaking "v", but "get_value" > > > is the "raw" version that gives you the uninterpreted value. I think > > > that means it may give you NULL, though if we see an implicit bool like: > > > > > > [core] > > > allowProtocol > > > > > > That's nonsense, of course, but we would still segfault. I > > > think the easiest way to test is: > > > > > > git -c core.allowProtocol fetch > > > > > > which seems to segfault for me with this patch. > > > > what is the desired behavior when a user provides a config in a way that > > isn't intended? > > oh...I can just drop in git_config_get_string_const() instead. Yes, it will call git_config_string(), which will make sure there's an actual value and die otherwise. But note that it also duplicates the string, so you'd have to deal with freeing it. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-02 22:20 [PATCH] transport: add core.allowProtocol config option Brandon Williams 2016-11-02 22:41 ` Stefan Beller 2016-11-02 23:05 ` Jeff King @ 2016-11-03 0:22 ` Jonathan Nieder 2016-11-03 0:41 ` Blake Burkhart ` (2 more replies) 2016-11-03 0:50 ` [PATCH v2] " Brandon Williams 2016-11-04 20:55 ` [PATCH v3] transport: add protocol policy " Brandon Williams 4 siblings, 3 replies; 124+ messages in thread From: Jonathan Nieder @ 2016-11-03 0:22 UTC (permalink / raw) To: Brandon Williams; +Cc: git, Stefan Beller, Blake Burkhart, Jeff King (+peff and bburky, who introduced GIT_ALLOW_PROTOCOL) Brandon Williams wrote: > Add configuration option 'core.allowProtocol' to allow users to create a > whitelist of allowed protocols for fetch/push/clone in their gitconfig. Ooh. This would be especially useful at $DAYJOB, where there is a custom sso:// protocol that is often used by submodules. Using an envvar to whitelist it globally is painful because - it disables other protocols even when explicitly requested on a plain "git clone" command line by the user. By comparison, the built-in git-submodule.sh whitelist only applies to submodules. - platform-specific instructions to set an environment variable can be more difficult than "just set this git configuration" Another difficulty with setting GIT_ALLOW_PROTOCOL globally is that it requires copy/pasting the default value from upstream and then adding the values I want. There's no straightforward way to get the current value and add to it, in case I want to benefit from future upstream fixes to the default list. That is, would it be possible to use something like [protocol "sso"] allow = always instead of [core] allowProtocol = file:git:http:https:....:sso ? [...] > --- a/git-submodule.sh > +++ b/git-submodule.sh > @@ -27,7 +27,8 @@ cd_to_toplevel > # > # If the user has already specified a set of allowed protocols, > # we assume they know what they're doing and use that instead. > -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} > +config_whitelist=$(git config core.allowProtocol) > +: ${GIT_ALLOW_PROTOCOL=${config_whitelist:-file:git:http:https:ssh}} optional: To avoid config parsing when GIT_ALLOW_PROTOCOL is already set, could do something like if ! test "${GIT_ALLOW_PROTOCOL+set}" then GIT_ALLOW_PROTOCOL=$( git config --name-only --get-regexp 'protocol\..*\.allow' always | sed -e 's/^protocol.//' -e 's/.allow$//' | tr '\n' ':' ) GIT_ALLOW_PROTOCOL=${GIT_ALLOW_PROTOCOL%:} : ${GIT_ALLOW_PROTOCOL:=file:git:http:https:ssh} fi [...] > --- a/transport.c > +++ b/transport.c > @@ -652,7 +652,7 @@ static const struct string_list *protocol_whitelist(void) > > if (enabled < 0) { > const char *v = getenv("GIT_ALLOW_PROTOCOL"); > - if (v) { > + if (v || !git_config_get_value("core.allowProtocol", &v)) { > string_list_split(&allowed, v, ':', -1); This has the effect of always disabling other protocols when core.allowProtocol is set. Is that intended? Like the default list used by submodule, I'd be happiest if this only applied to repositories cloned implicitly instead of those passed directly to 'git clone'. That reminds me: external tools also set GIT_ALLOW_PROTOCOL when the user hasn't set it explicitly, like git-submodule.sh does. E.g. repo <https://gerrit.googlesource.com/git-repo/+/466b8c4e/git_command.py#171>, mercurial <https://www.mercurial-scm.org/repo/hg/file/b032a7b676c6/mercurial/subrepo.py#l1404>. Other external tools consume GIT_ALLOW_PROTOCOL, like 'go get' <https://go.googlesource.com/go/+/55620a0e/src/cmd/go/vcs.go#64>. Can we make it more convenient for them to support this configuration too? An example approach would be a GIT_ALLOW_PROTOCOL var returned by "git var". That way git-submodule.sh could do : ${GIT_ALLOW_PROTOCOL=$(git var GIT_ALLOW_PROTOCOL)} and it would just work. Other tools could do the same, with a fallback to the current default until new enough git is in widespread use. Thanks and hope that helps, Jonathan ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 0:22 ` Jonathan Nieder @ 2016-11-03 0:41 ` Blake Burkhart 2016-11-03 2:44 ` Junio C Hamano 2016-11-03 14:38 ` Jeff King 2 siblings, 0 replies; 124+ messages in thread From: Blake Burkhart @ 2016-11-03 0:41 UTC (permalink / raw) To: Jonathan Nieder; +Cc: Brandon Williams, git, Stefan Beller, Jeff King Thanks for CCing me. I haven't looked at this implementation in detail, but it would be good to move this configuration into the config system because I think we can more easily provide a default safe configuration. It would be nice to use this to introduce a default list of whitelisted protocols that even applies to `git clone`. I strongly think we need to find a way to have git-remote-ext disabled by default. This could be a way to do it. On Wed, Nov 2, 2016 at 7:22 PM, Jonathan Nieder <jrnieder@gmail.com> wrote: > That reminds me: external tools also set GIT_ALLOW_PROTOCOL when the > user hasn't set it explicitly, like git-submodule.sh does. E.g. > repo <https://gerrit.googlesource.com/git-repo/+/466b8c4e/git_command.py#171>, > mercurial <https://www.mercurial-scm.org/repo/hg/file/b032a7b676c6/mercurial/subrepo.py#l1404>. > Other external tools consume GIT_ALLOW_PROTOCOL, like 'go get' > <https://go.googlesource.com/go/+/55620a0e/src/cmd/go/vcs.go#64>. > Can we make it more convenient for them to support this configuration > too? Most of these are my fault too. I encouraged git-repo and mercurial to use GIT_ALLOW_PROTOCOL to avoid security issues from git-remote-ext. -- Blake Burkhart ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 0:22 ` Jonathan Nieder 2016-11-03 0:41 ` Blake Burkhart @ 2016-11-03 2:44 ` Junio C Hamano 2016-11-03 14:38 ` Jeff King 2 siblings, 0 replies; 124+ messages in thread From: Junio C Hamano @ 2016-11-03 2:44 UTC (permalink / raw) To: Jonathan Nieder Cc: Brandon Williams, git, Stefan Beller, Blake Burkhart, Jeff King Jonathan Nieder <jrnieder@gmail.com> writes: > That is, would it be possible to use something like > > [protocol "sso"] > allow = always > > instead of > > [core] > allowProtocol = file:git:http:https:....:sso > > ? That sounds like quite a large usability improvement to me. ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 0:22 ` Jonathan Nieder 2016-11-03 0:41 ` Blake Burkhart 2016-11-03 2:44 ` Junio C Hamano @ 2016-11-03 14:38 ` Jeff King 2016-11-03 17:25 ` Brandon Williams 2 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-11-03 14:38 UTC (permalink / raw) To: Jonathan Nieder; +Cc: Brandon Williams, git, Stefan Beller, Blake Burkhart On Wed, Nov 02, 2016 at 05:22:25PM -0700, Jonathan Nieder wrote: > Another difficulty with setting GIT_ALLOW_PROTOCOL globally is that it > requires copy/pasting the default value from upstream and then adding > the values I want. There's no straightforward way to get the current > value and add to it, in case I want to benefit from future upstream > fixes to the default list. I agree that this is a big drawback of the current scheme, and it would be nice to be able to say "also allow". > That is, would it be possible to use something like > > [protocol "sso"] > allow = always > > instead of > > [core] > allowProtocol = file:git:http:https:....:sso > > ? One complication is that the whitelist has multiple states: 1. if it's not used at all, anything goes 2. if it exists and has zero or more entries, only those entries are allowed And then submodules are an exception to (1), because it's not anything goes. It's "this default safe whitelist". So when does protocol.sso.allow kick in? We wouldn't want it to trigger case (2) for things like fetch (disabling other non-allowed protocols). Nor do I think we'd only want it for the submodule case, as I would assume that "protocol.sso.allow = false" should disable it. So I think this probably needs to be a separate parallel system where each protocol can be white- or black-listed in a context-specific way. Like: protocol.X.allow = always | user | never Where "user" specifies that the protocol is OK coming directly from the user, but not from other sources. And we default known-common-and-good ones like protocol.http.allow to "always", unknown ones (like "foo" which runs "remote-foo") to "user"), and possibly known-scary ones like "ext") to "never". Then we need some way of telling git "you are in a context where the URL parameter is not coming from the user". Probably via the environment in GIT_PROTOCOL_FROM_USER or similar. Which git-submodule would sent when recursing clones, along with things like "go get". In other words, stop asking git-submodule or "go get" to specify policy, and let them specify context that can be used to implement policy that the user specifies (and have git provide a sane default policy). I think this would all take a backseat to GIT_ALLOW_PROTOCOL, for backwards compatibility, and then GIT_ALLOW_PROTOCOL could slowly die off over time. > That reminds me: external tools also set GIT_ALLOW_PROTOCOL when the > user hasn't set it explicitly, like git-submodule.sh does. E.g. > repo <https://gerrit.googlesource.com/git-repo/+/466b8c4e/git_command.py#171>, > mercurial <https://www.mercurial-scm.org/repo/hg/file/b032a7b676c6/mercurial/subrepo.py#l1404>. > Other external tools consume GIT_ALLOW_PROTOCOL, like 'go get' > <https://go.googlesource.com/go/+/55620a0e/src/cmd/go/vcs.go#64>. > Can we make it more convenient for them to support this configuration > too? I think under my proposal above this ugliness just goes away, as all they have to do is say "trust my URLs less; they come from an automated source" without specifying policy themselves. But we may still want... > An example approach would be a GIT_ALLOW_PROTOCOL var returned by > "git var". > > That way git-submodule.sh could do > > : ${GIT_ALLOW_PROTOCOL=$(git var GIT_ALLOW_PROTOCOL)} > > and it would just work. Other tools could do the same, with a > fallback to the current default until new enough git is in widespread > use. ...some automated way to say "is this protocol supported"? I think it is not just "give me ALLOW_PROTOCOL" anymore, though, but "apply your rules to this protocol, and tell me if it is supported". I don't think things like "go get" would need it, but you would if you had a porcelain built around git that was accessing a URL _not_ via git-fetch, but wanted to apply git's rules. That could come as a step 2 later, though. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 14:38 ` Jeff King @ 2016-11-03 17:25 ` Brandon Williams 2016-11-03 17:39 ` Stefan Beller 2016-11-03 17:53 ` Jeff King 0 siblings, 2 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-03 17:25 UTC (permalink / raw) To: Jeff King; +Cc: Jonathan Nieder, git, Stefan Beller, Blake Burkhart On 11/03, Jeff King wrote: > On Wed, Nov 02, 2016 at 05:22:25PM -0700, Jonathan Nieder wrote: > > > Another difficulty with setting GIT_ALLOW_PROTOCOL globally is that it > > requires copy/pasting the default value from upstream and then adding > > the values I want. There's no straightforward way to get the current > > value and add to it, in case I want to benefit from future upstream > > fixes to the default list. > > I agree that this is a big drawback of the current scheme, and it would > be nice to be able to say "also allow". > > > That is, would it be possible to use something like > > > > [protocol "sso"] > > allow = always > > > > instead of > > > > [core] > > allowProtocol = file:git:http:https:....:sso > > > > ? > > One complication is that the whitelist has multiple states: > > 1. if it's not used at all, anything goes > > 2. if it exists and has zero or more entries, only those entries are > allowed > > And then submodules are an exception to (1), because it's not anything > goes. It's "this default safe whitelist". > > So when does protocol.sso.allow kick in? We wouldn't want it to trigger > case (2) for things like fetch (disabling other non-allowed protocols). > Nor do I think we'd only want it for the submodule case, as I would > assume that "protocol.sso.allow = false" should disable it. > > So I think this probably needs to be a separate parallel system where > each protocol can be white- or black-listed in a context-specific way. > Like: > > protocol.X.allow = always | user | never It sounds like there is interest for this sort of behavior, it would definitely require a larger change than what I initially proposed. One problem I see though is that with this we have support for both a blacklist and a whitelist. Which wins? Or do we simply generate a whitelist of allowed protocols which includes all protocols with allow set to 'always' and if it is set to 'never' then it just isn't included in the whitelist? I don't know if I'm sold on a 'user' state just yet, perhaps that's just because I view a whitelist or blacklist as well black and white and having this user state adds in a gray area. > > An example approach would be a GIT_ALLOW_PROTOCOL var returned by > > "git var". > > > > That way git-submodule.sh could do > > > > : ${GIT_ALLOW_PROTOCOL=$(git var GIT_ALLOW_PROTOCOL)} > > > > and it would just work. Other tools could do the same, with a > > fallback to the current default until new enough git is in widespread > > use. > > ...some automated way to say "is this protocol supported"? I think it is > not just "give me ALLOW_PROTOCOL" anymore, though, but "apply your rules > to this protocol, and tell me if it is supported". I agree, if we do add different states to a protocol then we couldn't simply ask for a whitelist/blacklist of protocols anymore since its more of a graylist :) (if such a thing exits). -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 17:25 ` Brandon Williams @ 2016-11-03 17:39 ` Stefan Beller 2016-11-03 17:51 ` Brandon Williams 2016-11-03 18:00 ` Jeff King 2016-11-03 17:53 ` Jeff King 1 sibling, 2 replies; 124+ messages in thread From: Stefan Beller @ 2016-11-03 17:39 UTC (permalink / raw) To: Brandon Williams; +Cc: Jeff King, Jonathan Nieder, git, Blake Burkhart >> protocol.X.allow = always | user | never > > It sounds like there is interest for this sort of behavior, it would > definitely require a larger change than what I initially proposed. One > problem I see though is that with this we have support for both a > blacklist and a whitelist. Which wins? For the submodule operations we'll use a whitelist, because we want to provide security and for the other case we can offer a blacklist as a bandaid. My opinion on blacklists is roughly aligned with e.g. : https://blog.codinghorror.com/blacklists-dont-work/ http://blog.deepinstinct.com/2016/02/04/when-blacklists-dont-really-work/ So IMHO we could drop the "never" and substitute it with a "warn" or "ask-user", such that this configuration becomes a white list for both cases: protocol.X.allow = always | user | warn > Or do we simply generate a > whitelist of allowed protocols which includes all protocols with allow > set to 'always' and if it is set to 'never' then it just isn't included > in the whitelist? So you're suggesting that setting it to "never" doesn't have any effect except for cluttering the config file? I don't think we should do that; each setting should have an impact. So maybe the "never" would be there to disallow protocols of the hardcoded white list (e.g. http) > > I don't know if I'm sold on a 'user' state just yet, perhaps that's just > because I view a whitelist or blacklist as well black and white and > having this user state adds in a gray area. Well the "user" state is to differentiate between the * "I consciously typed `git clone ...` (and e.g. I know what happens as I know the server admin and they are trustworthy.) * a repository contains a possible hostile .gitmodules file such that I am not aware of the network connection. ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 17:39 ` Stefan Beller @ 2016-11-03 17:51 ` Brandon Williams 2016-11-03 18:02 ` Jeff King 2016-11-03 18:00 ` Jeff King 1 sibling, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-11-03 17:51 UTC (permalink / raw) To: Stefan Beller; +Cc: Jeff King, Jonathan Nieder, git, Blake Burkhart On 11/03, Stefan Beller wrote: > >> protocol.X.allow = always | user | never > > > > It sounds like there is interest for this sort of behavior, it would > > definitely require a larger change than what I initially proposed. One > > problem I see though is that with this we have support for both a > > blacklist and a whitelist. Which wins? > > For the submodule operations we'll use a whitelist, because we want to > provide security and for the other case we can offer a blacklist as a bandaid. > > My opinion on blacklists is roughly aligned with e.g. : > https://blog.codinghorror.com/blacklists-dont-work/ > http://blog.deepinstinct.com/2016/02/04/when-blacklists-dont-really-work/ > > So IMHO we could drop the "never" and substitute it with a "warn" or > "ask-user", such that this configuration becomes a white list for both cases: > > protocol.X.allow = always | user | warn > > > Or do we simply generate a > > whitelist of allowed protocols which includes all protocols with allow > > set to 'always' and if it is set to 'never' then it just isn't included > > in the whitelist? > > So you're suggesting that setting it to "never" doesn't have any effect > except for cluttering the config file? > I don't think we should do that; each setting should have an impact. > So maybe the "never" would be there to disallow protocols of the hardcoded > white list (e.g. http) Thats what I meant, if a protocol is listed as 'never' then it just removes that protocol from the whitelist. That way we still have the benefit of using a whitelist vs a blacklist. Also, if we move in this direction should we setup a default whitelist of allowed protocols? > > > > I don't know if I'm sold on a 'user' state just yet, perhaps that's just > > because I view a whitelist or blacklist as well black and white and > > having this user state adds in a gray area. > > Well the "user" state is to differentiate between the > * "I consciously typed `git clone ...` (and e.g. I know what happens as > I know the server admin and they are trustworthy.) > * a repository contains a possible hostile .gitmodules file such > that I am not aware of the network connection. This is still a gray area to me. I think that if we have a whitelist of protocols then it should be a true whitelist and not have some means of going around it. It just seems like something that could be exploited. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 17:51 ` Brandon Williams @ 2016-11-03 18:02 ` Jeff King 2016-11-03 18:08 ` Brandon Williams 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-11-03 18:02 UTC (permalink / raw) To: Brandon Williams; +Cc: Stefan Beller, Jonathan Nieder, git, Blake Burkhart On Thu, Nov 03, 2016 at 10:51:31AM -0700, Brandon Williams wrote: > > > I don't know if I'm sold on a 'user' state just yet, perhaps that's just > > > because I view a whitelist or blacklist as well black and white and > > > having this user state adds in a gray area. > > > > Well the "user" state is to differentiate between the > > * "I consciously typed `git clone ...` (and e.g. I know what happens as > > I know the server admin and they are trustworthy.) > > * a repository contains a possible hostile .gitmodules file such > > that I am not aware of the network connection. > > This is still a gray area to me. I think that if we have a whitelist of > protocols then it should be a true whitelist and not have some means of > going around it. It just seems like something that could be exploited. How do you implement: git clone --recursive trusted:foo.git and use your ssh keys for the "trusted" server, but not for any servers mentioned in .gitmodules? You need some way of distinguishing between the two contexts (and setting policy for each). -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 18:02 ` Jeff King @ 2016-11-03 18:08 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-03 18:08 UTC (permalink / raw) To: Jeff King; +Cc: Stefan Beller, Jonathan Nieder, git, Blake Burkhart On 11/03, Jeff King wrote: > On Thu, Nov 03, 2016 at 10:51:31AM -0700, Brandon Williams wrote: > > > > > I don't know if I'm sold on a 'user' state just yet, perhaps that's just > > > > because I view a whitelist or blacklist as well black and white and > > > > having this user state adds in a gray area. > > > > > > Well the "user" state is to differentiate between the > > > * "I consciously typed `git clone ...` (and e.g. I know what happens as > > > I know the server admin and they are trustworthy.) > > > * a repository contains a possible hostile .gitmodules file such > > > that I am not aware of the network connection. > > > > This is still a gray area to me. I think that if we have a whitelist of > > protocols then it should be a true whitelist and not have some means of > > going around it. It just seems like something that could be exploited. > > How do you implement: > > git clone --recursive trusted:foo.git > > and use your ssh keys for the "trusted" server, but not for any servers > mentioned in .gitmodules? > > You need some way of distinguishing between the two contexts (and > setting policy for each). > > -Peff Interesting. Ok I can see how this would be a useful now. Thanks for the example :) -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 17:39 ` Stefan Beller 2016-11-03 17:51 ` Brandon Williams @ 2016-11-03 18:00 ` Jeff King 1 sibling, 0 replies; 124+ messages in thread From: Jeff King @ 2016-11-03 18:00 UTC (permalink / raw) To: Stefan Beller; +Cc: Brandon Williams, Jonathan Nieder, git, Blake Burkhart On Thu, Nov 03, 2016 at 10:39:35AM -0700, Stefan Beller wrote: > >> protocol.X.allow = always | user | never > > > > It sounds like there is interest for this sort of behavior, it would > > definitely require a larger change than what I initially proposed. One > > problem I see though is that with this we have support for both a > > blacklist and a whitelist. Which wins? > > For the submodule operations we'll use a whitelist, because we want to > provide security and for the other case we can offer a blacklist as a bandaid. > > My opinion on blacklists is roughly aligned with e.g. : > https://blog.codinghorror.com/blacklists-dont-work/ > http://blog.deepinstinct.com/2016/02/04/when-blacklists-dont-really-work/ > > So IMHO we could drop the "never" and substitute it with a "warn" or > "ask-user", such that this configuration becomes a white list for both cases: > > protocol.X.allow = always | user | warn I don't think blacklists work in the general case, because they grow out of date and fail-open. But you want to have _some_ blacklisting mechanism, in order override a decision of the whitelist. For instance, the default submodule whitelist would probably include https and ssh. But if I'm cloning potentially malicious repos and I don't ever want them to trigger ssh (because I don't want them to use my ssh keys, whereas I have explicitly set up my credentials such that http is safe to use), I would want to be able to do: git config protocol.ssh.allow never (or "git -c", or whatever). True, a whitelist is safer. If we add a new "foo" protocol that also looks at your ssh keys, you're screwed. And that's why I designed it as a pure-whitelist in the first place. But it comes at the price of convenience, because you have to manually add each new innocent protocol to the whitelist. > So you're suggesting that setting it to "never" doesn't have any effect > except for cluttering the config file? > I don't think we should do that; each setting should have an impact. > So maybe the "never" would be there to disallow protocols of the hardcoded > white list (e.g. http) Exactly. > > I don't know if I'm sold on a 'user' state just yet, perhaps that's just > > because I view a whitelist or blacklist as well black and white and > > having this user state adds in a gray area. > > Well the "user" state is to differentiate between the > * "I consciously typed `git clone ...` (and e.g. I know what happens as > I know the server admin and they are trustworthy.) > * a repository contains a possible hostile .gitmodules file such > that I am not aware of the network connection. Right. I had assumed that we would tell the difference between those automatically (by seeing if we got the URL on the command line), but things like "go get" show that the context is often before git is even called. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 17:25 ` Brandon Williams 2016-11-03 17:39 ` Stefan Beller @ 2016-11-03 17:53 ` Jeff King 2016-11-03 18:19 ` Brandon Williams 2016-11-03 18:24 ` Jeff King 1 sibling, 2 replies; 124+ messages in thread From: Jeff King @ 2016-11-03 17:53 UTC (permalink / raw) To: Brandon Williams; +Cc: Jonathan Nieder, git, Stefan Beller, Blake Burkhart On Thu, Nov 03, 2016 at 10:25:15AM -0700, Brandon Williams wrote: > > So I think this probably needs to be a separate parallel system where > > each protocol can be white- or black-listed in a context-specific way. > > Like: > > > > protocol.X.allow = always | user | never > > It sounds like there is interest for this sort of behavior, it would > definitely require a larger change than what I initially proposed. One > problem I see though is that with this we have support for both a > blacklist and a whitelist. Which wins? Or do we simply generate a > whitelist of allowed protocols which includes all protocols with allow > set to 'always' and if it is set to 'never' then it just isn't included > in the whitelist? I think trying to combine the two or generate the whitelist from the more flexible format is a recipe for madness. I'd design the new system from scratch, and have it kick in _only_ when GIT_ALLOW_PROTOCOL is not set. That lets existing callers continue to have the safe behavior until they are ready to move to the new format. Something like the patch below (which is just for illustration, and not tested beyond compilation). > I don't know if I'm sold on a 'user' state just yet, perhaps that's just > because I view a whitelist or blacklist as well black and white and > having this user state adds in a gray area. The lack of understanding the "user" context is what makes the current system so painful. The only way a caller can influence the system is to hand over the policy directly: this is allowed, this is not. But systems like "go get" should not be setting policy. They should be giving us a hint about the context, and letting git implement the policy. -- >8 -- diff --git a/transport.c b/transport.c index d57e8dec2..0d5b382db 100644 --- a/transport.c +++ b/transport.c @@ -664,10 +664,69 @@ static const struct string_list *protocol_whitelist(void) return enabled ? &allowed : NULL; } +enum protocol_allow_config { + PROTOCOL_ALLOW_NEVER, + PROTOCOL_ALLOW_USER_ONLY, + PROTOCOL_ALLOW_ALWAYS +}; + +static enum protocol_allow_config parse_protocol_config(const char *key, + const char *value) +{ + if (!strcasecmp(value, "always")) + return PROTOCOL_ALLOW_ALWAYS; + else if (!strcasecmp(value, "never")) + return PROTOCOL_ALLOW_NEVER; + else if (!strcasecmp(value, "user")) + return PROTOCOL_ALLOW_USER_ONLY; + + /* XXX maybe also interpret git_config_bool() here? */ + die("unknown value for config '%s': %s", key, value); +} + +static enum protocol_allow_config get_protocol_config(const char *type) +{ + char *key = xstrfmt("protocol.%s.allow", type); + char *value; + + if (!git_config_get_string(key, &value)) { + enum protocol_allow_config ret = + parse_protocol_config(key, value); + free(key); + free(value); + return ret; + } + free(key); + + /* known safe */ + if (!strcmp(type, "http") || + !strcmp(type, "https") || + !strcmp(type, "git") || + !strcmp(type, "ssh")) + return PROTOCOL_ALLOW_ALWAYS; + + /* known scary; err on the side of caution */ + if (!strcmp(type, "ext")) + return PROTOCOL_ALLOW_NEVER; + + /* unknown; let them be used only directly by the user */ + return PROTOCOL_ALLOW_USER_ONLY; +} + int is_transport_allowed(const char *type) { - const struct string_list *allowed = protocol_whitelist(); - return !allowed || string_list_has_string(allowed, type); + const struct string_list *whitelist = protocol_whitelist(); + if (whitelist) + return string_list_has_string(whitelist, type); + + switch (get_protocol_config(type)) { + case PROTOCOL_ALLOW_ALWAYS: + return 1; + case PROTOCOL_ALLOW_NEVER: + return 0; + case PROTOCOL_ALLOW_USER_ONLY: + return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + } } void transport_check_allowed(const char *type) ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 17:53 ` Jeff King @ 2016-11-03 18:19 ` Brandon Williams 2016-11-03 18:25 ` Jeff King 2016-11-03 18:24 ` Jeff King 1 sibling, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-11-03 18:19 UTC (permalink / raw) To: Jeff King; +Cc: Jonathan Nieder, git, Stefan Beller, Blake Burkhart On 11/03, Jeff King wrote: > + > + /* unknown; let them be used only directly by the user */ > + return PROTOCOL_ALLOW_USER_ONLY; > +} > + > int is_transport_allowed(const char *type) > { > - const struct string_list *allowed = protocol_whitelist(); > - return !allowed || string_list_has_string(allowed, type); > + const struct string_list *whitelist = protocol_whitelist(); > + if (whitelist) > + return string_list_has_string(whitelist, type); > + > + switch (get_protocol_config(type)) { > + case PROTOCOL_ALLOW_ALWAYS: > + return 1; > + case PROTOCOL_ALLOW_NEVER: > + return 0; > + case PROTOCOL_ALLOW_USER_ONLY: > + return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); > + } I know this is just a rough patch you wiped up but one question: With the 'user' state, how exactly do you envision this env variable working? Do we want the user to have to explicitly set GIT_PROTOCOL_FROM_USER in their environment and then have these other commands (like git-submodule) explicitly clear the env var or would we rather these subcommands set a variable indicating they aren't coming from the user and the deafult state (no var set) is a user run command? -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 18:19 ` Brandon Williams @ 2016-11-03 18:25 ` Jeff King 0 siblings, 0 replies; 124+ messages in thread From: Jeff King @ 2016-11-03 18:25 UTC (permalink / raw) To: Brandon Williams; +Cc: Jonathan Nieder, git, Stefan Beller, Blake Burkhart On Thu, Nov 03, 2016 at 11:19:54AM -0700, Brandon Williams wrote: > On 11/03, Jeff King wrote: > > + > > + /* unknown; let them be used only directly by the user */ > > + return PROTOCOL_ALLOW_USER_ONLY; > > +} > > + > > int is_transport_allowed(const char *type) > > { > > - const struct string_list *allowed = protocol_whitelist(); > > - return !allowed || string_list_has_string(allowed, type); > > + const struct string_list *whitelist = protocol_whitelist(); > > + if (whitelist) > > + return string_list_has_string(whitelist, type); > > + > > + switch (get_protocol_config(type)) { > > + case PROTOCOL_ALLOW_ALWAYS: > > + return 1; > > + case PROTOCOL_ALLOW_NEVER: > > + return 0; > > + case PROTOCOL_ALLOW_USER_ONLY: > > + return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); > > + } > > I know this is just a rough patch you wiped up but one question: > With the 'user' state, how exactly do you envision this env variable > working? Do we want the user to have to explicitly set > GIT_PROTOCOL_FROM_USER in their environment and then have these other > commands (like git-submodule) explicitly clear the env var or would we > rather these subcommands set a variable indicating they aren't coming > from the user and the deafult state (no var set) is a user run command? See the follow-up I just posted, but basically, the rules are: - if you don't say anything, then the URL is from the user - git-submodule would set it to "0" (i.e., tell us to be more careful) - tools like "go get" would similarly set it to "0" if they are passing untrusted URLs -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 17:53 ` Jeff King 2016-11-03 18:19 ` Brandon Williams @ 2016-11-03 18:24 ` Jeff King 2016-11-03 18:45 ` Brandon Williams 1 sibling, 1 reply; 124+ messages in thread From: Jeff King @ 2016-11-03 18:24 UTC (permalink / raw) To: Brandon Williams; +Cc: Jonathan Nieder, git, Stefan Beller, Blake Burkhart On Thu, Nov 03, 2016 at 01:53:27PM -0400, Jeff King wrote: > I'd design the new system from scratch, and have it kick in _only_ when > GIT_ALLOW_PROTOCOL is not set. That lets existing callers continue to > have the safe behavior until they are ready to move to the new format. > > Something like the patch below (which is just for illustration, and not > tested beyond compilation). Here's that same patch with a few tweaks: - it changes git-submodule to use the new, more flexible system (which also gets a it a lot more test coverage) - it tweaks two tests which use the "ext" helper to enable it (since it's blacklisted by default; I have mixed feelings on that, but I see why Blake wants it, as it would have protected things like "go get" out of the box). - it adds "file://" as a known-good protocol, even for submodules, which matches the current code. I am not sure if this is reasonable or not. A malicious repository probably can't do much by pointing you to cloning your own repo as a submodule unless you then _also_ run some arbitrary code to expose it, at which point it's generally game-over anyway. And I'd expect automated services (like GitHub Pages) to already have a cut-down whitelist via GIT_ALLOW_PROTOCOL (and I happen to know that it goes). So this seems like a reasonable direction to me. It obviously needs documentation and tests. Arguably there should be a fallback "allow" value when a protocol is not mentioned in the config so that you could convert the default from "user" to "never" if you wanted your config to specify a pure whitelist. Without that, I think we'd want to keep GIT_ALLOW_PROTOCOL for the truly paranoid (though we should keep it indefinitely either way for backwards compatibility). Do you have interest in picking this up and running with it? -Peff diff --git a/git-submodule.sh b/git-submodule.sh index a024a135d..0a477b4c9 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -21,14 +21,10 @@ require_work_tree wt_prefix=$(git rev-parse --show-prefix) cd_to_toplevel -# Restrict ourselves to a vanilla subset of protocols; the URLs -# we get are under control of a remote repository, and we do not -# want them kicking off arbitrary git-remote-* programs. -# -# If the user has already specified a set of allowed protocols, -# we assume they know what they're doing and use that instead. -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} -export GIT_ALLOW_PROTOCOL +# Tell the rest of git that any URLs we get don't come +# directly from the user, so it can apply policy as appropriate. +GIT_PROTOCOL_FROM_USER=0 +export GIT_PROTOCOL_FROM_USER command= branch= diff --git a/t/t5509-fetch-push-namespaces.sh b/t/t5509-fetch-push-namespaces.sh index bc44ac36d..75c570adc 100755 --- a/t/t5509-fetch-push-namespaces.sh +++ b/t/t5509-fetch-push-namespaces.sh @@ -4,6 +4,7 @@ test_description='fetch/push involving ref namespaces' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git init original && ( diff --git a/t/t5802-connect-helper.sh b/t/t5802-connect-helper.sh index b7a7f9d58..c6c266187 100755 --- a/t/t5802-connect-helper.sh +++ b/t/t5802-connect-helper.sh @@ -4,6 +4,7 @@ test_description='ext::cmd remote "connect" helper' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git commit --allow-empty -m initial && test_tick && diff --git a/transport.c b/transport.c index d57e8dec2..cd603fbf5 100644 --- a/transport.c +++ b/transport.c @@ -664,10 +664,70 @@ static const struct string_list *protocol_whitelist(void) return enabled ? &allowed : NULL; } +enum protocol_allow_config { + PROTOCOL_ALLOW_NEVER = 0, + PROTOCOL_ALLOW_USER_ONLY, + PROTOCOL_ALLOW_ALWAYS +}; + +static enum protocol_allow_config parse_protocol_config(const char *key, + const char *value) +{ + if (!strcasecmp(value, "always")) + return PROTOCOL_ALLOW_ALWAYS; + else if (!strcasecmp(value, "never")) + return PROTOCOL_ALLOW_NEVER; + else if (!strcasecmp(value, "user")) + return PROTOCOL_ALLOW_USER_ONLY; + + /* XXX maybe also interpret git_config_bool() here? */ + die("unknown value for config '%s': %s", key, value); +} + +static enum protocol_allow_config get_protocol_config(const char *type) +{ + char *key = xstrfmt("protocol.%s.allow", type); + char *value; + + if (!git_config_get_string(key, &value)) { + enum protocol_allow_config ret = + parse_protocol_config(key, value); + free(key); + free(value); + return ret; + } + free(key); + + /* known safe */ + if (!strcmp(type, "http") || + !strcmp(type, "https") || + !strcmp(type, "git") || + !strcmp(type, "ssh") || + !strcmp(type, "file")) + return PROTOCOL_ALLOW_ALWAYS; + + /* known scary; err on the side of caution */ + if (!strcmp(type, "ext")) + return PROTOCOL_ALLOW_NEVER; + + /* unknown; let them be used only directly by the user */ + return PROTOCOL_ALLOW_USER_ONLY; +} + int is_transport_allowed(const char *type) { - const struct string_list *allowed = protocol_whitelist(); - return !allowed || string_list_has_string(allowed, type); + const struct string_list *whitelist = protocol_whitelist(); + if (whitelist) + return string_list_has_string(whitelist, type); + + switch (get_protocol_config(type)) { + case PROTOCOL_ALLOW_ALWAYS: + return 1; + case PROTOCOL_ALLOW_NEVER: + return 0; + case PROTOCOL_ALLOW_USER_ONLY: + return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + } } void transport_check_allowed(const char *type) ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 18:24 ` Jeff King @ 2016-11-03 18:45 ` Brandon Williams 2016-11-03 18:50 ` Jeff King 0 siblings, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-11-03 18:45 UTC (permalink / raw) To: Jeff King; +Cc: Jonathan Nieder, git, Stefan Beller, Blake Burkhart On 11/03, Jeff King wrote: > > So this seems like a reasonable direction to me. It obviously needs > documentation and tests. Arguably there should be a fallback "allow" > value when a protocol is not mentioned in the config so that you could > convert the default from "user" to "never" if you wanted your config to > specify a pure whitelist. Yes I agree there should probably be a fallback value of 'never' maybe? What you currently have preserves the behavior of what git does now, if we did instead have a fallback of 'never' it would break current users who don't already use GIT_ALLOW_PROTOCOL (well only if they use crazy protocols). We could ease into it though and start with default to allow and then transition to a true whitelist sometime after this change has been made? > > Without that, I think we'd want to keep GIT_ALLOW_PROTOCOL for the truly > paranoid (though we should keep it indefinitely either way for backwards > compatibility). > > Do you have interest in picking this up and running with it? Yep! Thanks for the help in shaping this. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 18:45 ` Brandon Williams @ 2016-11-03 18:50 ` Jeff King 2016-11-03 18:56 ` Brandon Williams 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-11-03 18:50 UTC (permalink / raw) To: Brandon Williams; +Cc: Jonathan Nieder, git, Stefan Beller, Blake Burkhart On Thu, Nov 03, 2016 at 11:45:38AM -0700, Brandon Williams wrote: > On 11/03, Jeff King wrote: > > > > So this seems like a reasonable direction to me. It obviously needs > > documentation and tests. Arguably there should be a fallback "allow" > > value when a protocol is not mentioned in the config so that you could > > convert the default from "user" to "never" if you wanted your config to > > specify a pure whitelist. > > Yes I agree there should probably be a fallback value of 'never' maybe? > What you currently have preserves the behavior of what git does > now, if we did instead have a fallback of 'never' it would break current > users who don't already use GIT_ALLOW_PROTOCOL (well only if they use > crazy protocols). We could ease into it though and start with default > to allow and then transition to a true whitelist sometime after this > change has been made? I don't see the value in moving the out-of-the-box install to any default except "user". Right now the experience of using a third-party helper is something like: cp git-remote-hg /somewhere/in/your/PATH git clone hg::whatever We restrict its use in submodules by default, which is unlikely to bite many people. But if we started falling back to "never" all the time, then that second command would break until you officially "approve" remote-hg in your config. I was thinking of just something to let people decide to have that level of paranoia themselves (especially if they want to just set up a whole-system white list via the config without bothering with environment variables). Like: git config --system protocol.allow never git config --system protocol.https.allow always That behaves exactly like: export GIT_ALLOW_PROTOCOL=https except it just works everywhere, without having to tweak the environment of every process. > > Do you have interest in picking this up and running with it? > > Yep! Thanks for the help in shaping this. Great, thanks. I'm happy to review or discuss further as necessary. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH] transport: add core.allowProtocol config option 2016-11-03 18:50 ` Jeff King @ 2016-11-03 18:56 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-03 18:56 UTC (permalink / raw) To: Jeff King; +Cc: Jonathan Nieder, git, Stefan Beller, Blake Burkhart On 11/03, Jeff King wrote: > On Thu, Nov 03, 2016 at 11:45:38AM -0700, Brandon Williams wrote: > > > On 11/03, Jeff King wrote: > > > > > > So this seems like a reasonable direction to me. It obviously needs > > > documentation and tests. Arguably there should be a fallback "allow" > > > value when a protocol is not mentioned in the config so that you could > > > convert the default from "user" to "never" if you wanted your config to > > > specify a pure whitelist. > > > > Yes I agree there should probably be a fallback value of 'never' maybe? > > What you currently have preserves the behavior of what git does > > now, if we did instead have a fallback of 'never' it would break current > > users who don't already use GIT_ALLOW_PROTOCOL (well only if they use > > crazy protocols). We could ease into it though and start with default > > to allow and then transition to a true whitelist sometime after this > > change has been made? > > I don't see the value in moving the out-of-the-box install to any > default except "user". Right now the experience of using a third-party > helper is something like: > > cp git-remote-hg /somewhere/in/your/PATH > git clone hg::whatever > > We restrict its use in submodules by default, which is unlikely to bite > many people. But if we started falling back to "never" all the time, > then that second command would break until you officially "approve" > remote-hg in your config. > > I was thinking of just something to let people decide to have that level > of paranoia themselves (especially if they want to just set up a > whole-system white list via the config without bothering with > environment variables). Like: > > git config --system protocol.allow never > git config --system protocol.https.allow always > > That behaves exactly like: > > export GIT_ALLOW_PROTOCOL=https > > except it just works everywhere, without having to tweak the environment > of every process. > Ah ok, so essentially letting the user specify a default behaviour themselves. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v2] transport: add core.allowProtocol config option 2016-11-02 22:20 [PATCH] transport: add core.allowProtocol config option Brandon Williams ` (2 preceding siblings ...) 2016-11-03 0:22 ` Jonathan Nieder @ 2016-11-03 0:50 ` Brandon Williams 2016-11-04 20:55 ` [PATCH v3] transport: add protocol policy " Brandon Williams 4 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-03 0:50 UTC (permalink / raw) To: git; +Cc: Brandon Williams, sbeller, bburky, peff, jrnieder Add configuration option 'core.allowProtocol' to allow users to create a whitelist of allowed protocols for fetch/push/clone in their gitconfig. For git-submodule.sh, fallback to default whitelist only if the user hasn't explicitly set `GIT_ALLOW_PROTOCOL` or doesn't have a whitelist in their gitconfig. Signed-off-by: Brandon Williams <bmwill@google.com> --- Documentation/config.txt | 8 ++++++++ Documentation/git.txt | 6 ++++-- git-submodule.sh | 3 ++- t/lib-proto-disable.sh | 27 +++++++++++++++++++++++++++ t/t5815-submodule-protos.sh | 22 ++++++++++++++++++++++ transport.c | 6 ++++++ 6 files changed, 69 insertions(+), 3 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index 27069ac..78c97e3 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -455,6 +455,14 @@ core.sshCommand:: the `GIT_SSH_COMMAND` environment variable and is overridden when the environment variable is set. +core.allowProtocol:: + Provide a colon-separated list of protocols which are allowed to be + used with fetch/push/clone. Any protocol not mentioned will be + disallowed (i.e., this is a whitelist, not a blacklist). If the + `GIT_ALLOW_PROTOCOL` environment variable is set, it is used as the + protocol whitelist instead of this config option. If neither is set, + all protocols are enabled. See git(1) for more details. + core.ignoreStat:: If true, Git will avoid using lstat() calls to detect if files have changed by setting the "assume-unchanged" bit for those tracked files diff --git a/Documentation/git.txt b/Documentation/git.txt index ab7215e..a86ec16 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -1155,8 +1155,10 @@ of clones and fetches. restrict recursive submodule initialization from an untrusted repository. Any protocol not mentioned will be disallowed (i.e., this is a whitelist, not a blacklist). If the variable is not - set at all, all protocols are enabled. The protocol names - currently used by git are: + set at all, all protocols are enabled. The exception to this is when + running `git-submodule` which will use a default list of known-safe + protocols (file:git:http:https:ssh) in the event no whitelist is + provided. The protocol names currently used by git are: - `file`: any local file-based path (including `file://` URLs, or local paths) diff --git a/git-submodule.sh b/git-submodule.sh index a024a13..bfbfb8e 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -27,7 +27,8 @@ cd_to_toplevel # # If the user has already specified a set of allowed protocols, # we assume they know what they're doing and use that instead. -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} +whitelist=$(git config core.allowProtocol) || whitelist=file:git:http:https:ssh +: ${GIT_ALLOW_PROTOCOL=$whitelist} export GIT_ALLOW_PROTOCOL command= diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index b0917d9..e8820a6 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -62,6 +62,33 @@ test_proto () { test_must_fail git clone --bare "$url" tmp.git ) ' + + # Run tests again using the gitconfig method for setting a whitelist + test_expect_success "clone $1 (enabled)" ' + rm -rf tmp.git && + git -c core.allowProtocol="$proto" clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $1 (enabled)" ' + git -C tmp.git -c core.allowProtocol="$proto" fetch + ' + + test_expect_success "push $1 (enabled)" ' + git -C tmp.git -c core.allowProtocol="$proto" push origin HEAD:pushed + ' + + test_expect_success "push $1 (disabled)" ' + test_must_fail git -C tmp.git -c core.allowProtocol=none push origin HEAD:pushed + ' + + test_expect_success "fetch $1 (disabled)" ' + test_must_fail git -C tmp.git -c core.allowProtocol=none fetch + ' + + test_expect_success "clone $1 (disabled)" ' + rm -rf tmp.git && + test_must_fail git -C tmp.git -c core.allowProtocol=none clone --bare "$url" tmp.git + ' } # set up an ssh wrapper that will access $host/$repo in the diff --git a/t/t5815-submodule-protos.sh b/t/t5815-submodule-protos.sh index 06f55a1..f85e71c 100755 --- a/t/t5815-submodule-protos.sh +++ b/t/t5815-submodule-protos.sh @@ -40,4 +40,26 @@ test_expect_success 'user can override whitelist' ' GIT_ALLOW_PROTOCOL=ext git -C dst submodule update ext-module ' +test_expect_success 'reset dst repo for config tests' ' + rm -rf dst && + git clone . dst && + git -C dst submodule init +' + +test_expect_success 'update of ssh not allowed when not in config whitelist' ' + test_must_fail git -C dst -c core.allowProtocol=none submodule update ssh-module +' + +test_expect_success 'update of ssh allowed via config whitelist' ' + git -C dst -c core.allowProtocol="ssh:http:https" submodule update ssh-module +' + +test_expect_success 'update of ext not allowed' ' + test_must_fail git -C dst -c core.allowProtocol=ssh submodule update ext-module +' + +test_expect_success 'user can override whitelist' ' + git -C dst -c core.allowProtocol=ext submodule update ext-module +' + test_done diff --git a/transport.c b/transport.c index d57e8de..e3d8e88 100644 --- a/transport.c +++ b/transport.c @@ -652,10 +652,16 @@ static const struct string_list *protocol_whitelist(void) if (enabled < 0) { const char *v = getenv("GIT_ALLOW_PROTOCOL"); + char *w = NULL; if (v) { string_list_split(&allowed, v, ':', -1); string_list_sort(&allowed); enabled = 1; + } else if (!git_config_get_string("core.allowProtocol", &w)) { + string_list_split(&allowed, w, ':', -1); + string_list_sort(&allowed); + enabled = 1; + free(w); } else { enabled = 0; } -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v3] transport: add protocol policy config option 2016-11-02 22:20 [PATCH] transport: add core.allowProtocol config option Brandon Williams ` (3 preceding siblings ...) 2016-11-03 0:50 ` [PATCH v2] " Brandon Williams @ 2016-11-04 20:55 ` Brandon Williams 2016-11-04 20:58 ` Brandon Williams ` (3 more replies) 4 siblings, 4 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-04 20:55 UTC (permalink / raw) To: git; +Cc: Brandon Williams, sbeller, bburky, peff, jrnieder Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to specify a whitelist of protocols to be used in clone/fetch/pull commands. This patch introduces new configuration options for more fine-grained control for allowing/disallowing protocols. This also has the added benefit of allowing easier construction of a protocol whitelist on systems where setting an environment variable is non-trivial. Now users can specify a policy to be used for each type of protocol via the 'protocol.<name>.allow' config option. A default policy for all unknown protocols can be set with the 'protocol.allow' config option. If no user configured default is made git, by default, will allow known-safe protocols (http, https, git, ssh, file), disallow known-dangerous protocols (ext), and have a default poliy of `user` for all other protocols. The supported policies are `always`, `never`, and `user`. The `user` policy can be used to configure a protocol to be usable when explicitly used by a user, while disallowing it for commands which run clone/fetch/pull commands without direct user intervention (e.g. recursive initialization of submodules). Commands which can potentially clone/fetch/pull from untrusted repositories without user intervention can export `GIT_PROTOCOL_FROM_USER` with a value of '0' to prevent protocols configured to the `user` policy from being used. Signed-off-by: Brandon Williams <bmwill@google.com> --- Documentation/config.txt | 25 ++++++++ Documentation/git.txt | 19 +++--- git-submodule.sh | 12 ++-- t/lib-proto-disable.sh | 132 +++++++++++++++++++++++++++++++++++---- t/t5509-fetch-push-namespaces.sh | 1 + t/t5802-connect-helper.sh | 1 + transport.c | 73 +++++++++++++++++++++- 7 files changed, 235 insertions(+), 28 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index 27069ac..5d845c4 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -2308,6 +2308,31 @@ pretty.<name>:: Note that an alias with the same name as a built-in format will be silently ignored. +protocol.allow:: + If set, provide a user defined default policy for all protocols which + don't explicitly have a policy (protocol.<name>.allow). By default, + if unset, known-safe protocols (http, https, git, ssh, file) have a + default policy of `always`, known-dangerous protocols (ext) have a + default policy of `never`, and all other protocols have a default policy + of `user`. Supported policies: ++ +-- + +* `always` - protocol is always able to be used. + +* `never` - protocol is never able to be used. + +* `user` - protocol is only able to be used when `GIT_PROTOCOL_FROM_USER` is + either unset or has a value of 1. This policy should be used when you want a + protocol to be usable by the user but don't want it used by commands which + execute clone/fetch/pull commands without user input, e.g. recursive + submodule initialization. + +-- + +protocol.<name>.allow:: + Set a policy to be used by protocol <name> with clone/fetch/pull commands. + pull.ff:: By default, Git does not create an extra merge commit when merging a commit that is a descendant of the current commit. Instead, the diff --git a/Documentation/git.txt b/Documentation/git.txt index ab7215e..ab25580 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -1150,13 +1150,13 @@ of clones and fetches. cloning a repository to make a backup). `GIT_ALLOW_PROTOCOL`:: - If set, provide a colon-separated list of protocols which are - allowed to be used with fetch/push/clone. This is useful to - restrict recursive submodule initialization from an untrusted - repository. Any protocol not mentioned will be disallowed (i.e., - this is a whitelist, not a blacklist). If the variable is not - set at all, all protocols are enabled. The protocol names - currently used by git are: + The new way to configure allowed protocols is done through the config + interface, though this setting takes precedences. See + linkgit:git-config[1] for more details. If set, provide a + colon-separated list of protocols which are allowed to be used with + fetch/push/clone. Any protocol not mentioned will be disallowed (i.e., + this is a whitelist, not a blacklist). The protocol names currently + used by git are: - `file`: any local file-based path (including `file://` URLs, or local paths) @@ -1174,6 +1174,11 @@ of clones and fetches. - any external helpers are named by their protocol (e.g., use `hg` to allow the `git-remote-hg` helper) +`GIT_PROTOCOL_FROM_USER`:: + Set to 0 to prevent protocols used by fetch/push/clone which are + configured to the `user` state. This is useful to restrict recursive + submodule initialization from an untrusted repository. See + linkgit:git-config[1] for more details. Discussion[[Discussion]] ------------------------ diff --git a/git-submodule.sh b/git-submodule.sh index a024a13..0a477b4 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -21,14 +21,10 @@ require_work_tree wt_prefix=$(git rev-parse --show-prefix) cd_to_toplevel -# Restrict ourselves to a vanilla subset of protocols; the URLs -# we get are under control of a remote repository, and we do not -# want them kicking off arbitrary git-remote-* programs. -# -# If the user has already specified a set of allowed protocols, -# we assume they know what they're doing and use that instead. -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} -export GIT_ALLOW_PROTOCOL +# Tell the rest of git that any URLs we get don't come +# directly from the user, so it can apply policy as appropriate. +GIT_PROTOCOL_FROM_USER=0 +export GIT_PROTOCOL_FROM_USER command= branch= diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index b0917d9..5950fbf 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -1,15 +1,12 @@ # Test routines for checking protocol disabling. -# test cloning a particular protocol -# $1 - description of the protocol -# $2 - machine-readable name of the protocol -# $3 - the URL to try cloning -test_proto () { +# Test clone/fetch/push with GIT_ALLOW_PROTOCOL whitelist +test_whitelist () { desc=$1 proto=$2 url=$3 - test_expect_success "clone $1 (enabled)" ' + test_expect_success "clone $desc (enabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=$proto && @@ -18,7 +15,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (enabled)" ' + test_expect_success "fetch $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -27,7 +24,7 @@ test_proto () { ) ' - test_expect_success "push $1 (enabled)" ' + test_expect_success "push $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -36,7 +33,7 @@ test_proto () { ) ' - test_expect_success "push $1 (disabled)" ' + test_expect_success "push $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -45,7 +42,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (disabled)" ' + test_expect_success "fetch $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -54,7 +51,7 @@ test_proto () { ) ' - test_expect_success "clone $1 (disabled)" ' + test_expect_success "clone $desc (disabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=none && @@ -64,6 +61,119 @@ test_proto () { ' } +test_config () { + desc=$1 + proto=$2 + url=$3 + + # Test clone/fetch/push with protocol.<type>.allow config + test_expect_success "clone $desc (enabled with config)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git -c protocol.$proto.allow=never clone --bare "$url" tmp.git + ' + + # Test clone/fetch/push with protocol.user.allow and its env var + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user push origin HEAD:pushed + ) + ' + + test_expect_success "fetch $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user fetch + ) + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + ( + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ) + ' + + # Test clone/fetch/push with protocol.allow user defined default + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git config --global protocol.allow always && + git clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + git config --global protocol.allow never && + test_must_fail git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git clone --bare "$url" tmp.git + ' +} + +# test cloning a particular protocol +# $1 - description of the protocol +# $2 - machine-readable name of the protocol +# $3 - the URL to try cloning +test_proto () { + test_whitelist "$@" + + test_config "$@" +} + # set up an ssh wrapper that will access $host/$repo in the # trash directory, and enable it for subsequent tests. setup_ssh_wrapper () { diff --git a/t/t5509-fetch-push-namespaces.sh b/t/t5509-fetch-push-namespaces.sh index bc44ac3..75c570a 100755 --- a/t/t5509-fetch-push-namespaces.sh +++ b/t/t5509-fetch-push-namespaces.sh @@ -4,6 +4,7 @@ test_description='fetch/push involving ref namespaces' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git init original && ( diff --git a/t/t5802-connect-helper.sh b/t/t5802-connect-helper.sh index b7a7f9d..c6c2661 100755 --- a/t/t5802-connect-helper.sh +++ b/t/t5802-connect-helper.sh @@ -4,6 +4,7 @@ test_description='ext::cmd remote "connect" helper' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git commit --allow-empty -m initial && test_tick && diff --git a/transport.c b/transport.c index d57e8de..c2d0538 100644 --- a/transport.c +++ b/transport.c @@ -664,10 +664,79 @@ static const struct string_list *protocol_whitelist(void) return enabled ? &allowed : NULL; } +enum protocol_allow_config { + PROTOCOL_ALLOW_NEVER = 0, + PROTOCOL_ALLOW_USER_ONLY, + PROTOCOL_ALLOW_ALWAYS +}; + +static enum protocol_allow_config parse_protocol_config(const char *key, + const char *value) +{ + if (!strcasecmp(value, "always")) + return PROTOCOL_ALLOW_ALWAYS; + else if (!strcasecmp(value, "never")) + return PROTOCOL_ALLOW_NEVER; + else if (!strcasecmp(value, "user")) + return PROTOCOL_ALLOW_USER_ONLY; + + die("unknown value for config '%s': %s", key, value); +} + +static enum protocol_allow_config get_protocol_config(const char *type) +{ + char *key = xstrfmt("protocol.%s.allow", type); + char *value; + + if (!git_config_get_string(key, &value)) { + enum protocol_allow_config ret = + parse_protocol_config(key, value); + free(key); + free(value); + return ret; + } + free(key); + + /* if defined, use user default for unknown protocols */ + if (!git_config_get_string("protocol.allow", &value)) { + enum protocol_allow_config ret = + parse_protocol_config("protocol.allow", value); + free(value); + return ret; + } + + /* known safe */ + if (!strcmp(type, "http") || + !strcmp(type, "https") || + !strcmp(type, "git") || + !strcmp(type, "ssh") || + !strcmp(type, "file")) + return PROTOCOL_ALLOW_ALWAYS; + + /* known scary; err on the side of caution */ + if (!strcmp(type, "ext")) + return PROTOCOL_ALLOW_NEVER; + + /* unknown; by default let them be used only directly by the user */ + return PROTOCOL_ALLOW_USER_ONLY; +} + int is_transport_allowed(const char *type) { - const struct string_list *allowed = protocol_whitelist(); - return !allowed || string_list_has_string(allowed, type); + const struct string_list *whitelist = protocol_whitelist(); + if (whitelist) + return string_list_has_string(whitelist, type); + + switch (get_protocol_config(type)) { + case PROTOCOL_ALLOW_ALWAYS: + return 1; + case PROTOCOL_ALLOW_NEVER: + return 0; + case PROTOCOL_ALLOW_USER_ONLY: + return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + } + + die("BUG: invalid protocol_allow_config type"); } void transport_check_allowed(const char *type) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v3] transport: add protocol policy config option 2016-11-04 20:55 ` [PATCH v3] transport: add protocol policy " Brandon Williams @ 2016-11-04 20:58 ` Brandon Williams 2016-11-04 21:35 ` Stefan Beller 2016-11-04 22:38 ` Stefan Beller ` (2 subsequent siblings) 3 siblings, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-11-04 20:58 UTC (permalink / raw) To: git; +Cc: sbeller, bburky, peff, jrnieder On 11/04, Brandon Williams wrote: > Signed-off-by: Brandon Williams <bmwill@google.com> Is there an acceptable way to give credit to Jeff for helping with this patch? -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v3] transport: add protocol policy config option 2016-11-04 20:58 ` Brandon Williams @ 2016-11-04 21:35 ` Stefan Beller 2016-11-04 23:09 ` Jeff King 0 siblings, 1 reply; 124+ messages in thread From: Stefan Beller @ 2016-11-04 21:35 UTC (permalink / raw) To: Brandon Williams; +Cc: git, Blake Burkhart, Jeff King, Jonathan Nieder On Fri, Nov 4, 2016 at 1:58 PM, Brandon Williams <bmwill@google.com> wrote: > On 11/04, Brandon Williams wrote: >> Signed-off-by: Brandon Williams <bmwill@google.com> > > Is there an acceptable way to give credit to Jeff for helping with this patch? What about: Helped-by: Jeff King <peff@peff.net> > > -- > Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v3] transport: add protocol policy config option 2016-11-04 21:35 ` Stefan Beller @ 2016-11-04 23:09 ` Jeff King 2016-11-05 0:18 ` Brandon Williams 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-11-04 23:09 UTC (permalink / raw) To: Stefan Beller; +Cc: Brandon Williams, git, Blake Burkhart, Jonathan Nieder On Fri, Nov 04, 2016 at 02:35:57PM -0700, Stefan Beller wrote: > On Fri, Nov 4, 2016 at 1:58 PM, Brandon Williams <bmwill@google.com> wrote: > > On 11/04, Brandon Williams wrote: > >> Signed-off-by: Brandon Williams <bmwill@google.com> > > > > Is there an acceptable way to give credit to Jeff for helping with this patch? > > What about: > Helped-by: Jeff King <peff@peff.net> That, or often I would write: Based on a patch by Jeff King <peff@peff.net> in the commit message. Basically anything is OK _except_ forging signed-off-by, because it has a very specific meaning. So let me also say that I am happy to give my: Signed-off-by: Jeff King <peff@peff.net> to the original (which you should add in, to make clear that the copyright issues are OK). In some cases it makes sense to just roll somebody's patch into your series, and then build on top. I'm fine with it all going into a single patch here. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v3] transport: add protocol policy config option 2016-11-04 23:09 ` Jeff King @ 2016-11-05 0:18 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-05 0:18 UTC (permalink / raw) To: Jeff King; +Cc: Stefan Beller, git, Blake Burkhart, Jonathan Nieder On 11/04, Jeff King wrote: > On Fri, Nov 04, 2016 at 02:35:57PM -0700, Stefan Beller wrote: > > > On Fri, Nov 4, 2016 at 1:58 PM, Brandon Williams <bmwill@google.com> wrote: > > > On 11/04, Brandon Williams wrote: > > >> Signed-off-by: Brandon Williams <bmwill@google.com> > > > > > > Is there an acceptable way to give credit to Jeff for helping with this patch? > > > > What about: > > Helped-by: Jeff King <peff@peff.net> > > That, or often I would write: > > Based on a patch by Jeff King <peff@peff.net> > > in the commit message. Basically anything is OK _except_ forging > signed-off-by, because it has a very specific meaning. So let me also > say that I am happy to give my: > > Signed-off-by: Jeff King <peff@peff.net> > > to the original (which you should add in, to make clear that the > copyright issues are OK). > > In some cases it makes sense to just roll somebody's patch into your > series, and then build on top. I'm fine with it all going into a single > patch here. > > -Peff Oh if it would be more clear I can easily break it up into two patches. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v3] transport: add protocol policy config option 2016-11-04 20:55 ` [PATCH v3] transport: add protocol policy " Brandon Williams 2016-11-04 20:58 ` Brandon Williams @ 2016-11-04 22:38 ` Stefan Beller 2016-11-07 18:14 ` Brandon Williams 2016-11-04 23:06 ` Jeff King 2016-11-07 19:35 ` [PATCH v4 1/2] lib-proto-disable: variable name fix Brandon Williams 3 siblings, 1 reply; 124+ messages in thread From: Stefan Beller @ 2016-11-04 22:38 UTC (permalink / raw) To: Brandon Williams; +Cc: git, Blake Burkhart, Jeff King, Jonathan Nieder On Fri, Nov 4, 2016 at 1:55 PM, Brandon Williams <bmwill@google.com> wrote: > Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to > specify a whitelist of protocols to be used in clone/fetch/pull > commands. This patch introduces new configuration options for more > fine-grained control for allowing/disallowing protocols. This also has > the added benefit of allowing easier construction of a protocol > whitelist on systems where setting an environment variable is > non-trivial. > > Now users can specify a policy to be used for each type of protocol via > the 'protocol.<name>.allow' config option. A default policy for all > unknown protocols can be set with the 'protocol.allow' config option. > If no user configured default is made git, by default, will allow > known-safe protocols (http, https, git, ssh, file), disallow > known-dangerous protocols (ext), and have a default poliy of `user` for > all other protocols. > > The supported policies are `always`, `never`, and `user`. The `user` > policy can be used to configure a protocol to be usable when explicitly > used by a user, while disallowing it for commands which run > clone/fetch/pull commands without direct user intervention (e.g. > recursive initialization of submodules). Commands which can potentially > clone/fetch/pull from untrusted repositories without user intervention > can export `GIT_PROTOCOL_FROM_USER` with a value of '0' to prevent > protocols configured to the `user` policy from being used. > > Signed-off-by: Brandon Williams <bmwill@google.com> > --- > Documentation/config.txt | 25 ++++++++ > Documentation/git.txt | 19 +++--- > git-submodule.sh | 12 ++-- > t/lib-proto-disable.sh | 132 +++++++++++++++++++++++++++++++++++---- > t/t5509-fetch-push-namespaces.sh | 1 + > t/t5802-connect-helper.sh | 1 + > transport.c | 73 +++++++++++++++++++++- > 7 files changed, 235 insertions(+), 28 deletions(-) > > diff --git a/Documentation/config.txt b/Documentation/config.txt > index 27069ac..5d845c4 100644 > --- a/Documentation/config.txt > +++ b/Documentation/config.txt > @@ -2308,6 +2308,31 @@ pretty.<name>:: > Note that an alias with the same name as a built-in format > will be silently ignored. > > +protocol.allow:: > + If set, provide a user defined default policy for all protocols which > + don't explicitly have a policy (protocol.<name>.allow). By default, Use hyphens (`protocol.<name>.allow`) to highlight the config option. By default, if unset, ... have a default policy ... sounds strange. How about just dropping the first 4 words here: Known-safe protocols (http, https, git, ssh, file) have a default policy of `always`, known-dangerous protocols (ext) have a default policy of `never`, and all other protocols have a default policy of `user`. Supported policies: What happens if protocol.allow is set to true? > + if unset, known-safe protocols (http, https, git, ssh, file) have a > + default policy of `always`, known-dangerous protocols (ext) have a > + default policy of `never`, and all other protocols have a default policy > + of `user`. Supported policies: > ++ > +-- > + > +* `always` - protocol is always able to be used. > + > +* `never` - protocol is never able to be used. > + > +* `user` - protocol is only able to be used when `GIT_PROTOCOL_FROM_USER` is > + either unset or has a value of 1. This policy should be used when you want a > + protocol to be usable by the user but don't want it used by commands which > + execute clone/fetch/pull commands without user input, e.g. recursive > + submodule initialization. > + > +-- > + > +protocol.<name>.allow:: > + Set a policy to be used by protocol <name> with clone/fetch/pull commands. How does this interact with protocol.allow? When protocol.allow is set, this overrides the specific protocol. If protocol is not set, it overrides the specific protocol as well(?) > + > pull.ff:: > By default, Git does not create an extra merge commit when merging > a commit that is a descendant of the current commit. Instead, the > diff --git a/Documentation/git.txt b/Documentation/git.txt > index ab7215e..ab25580 100644 > --- a/Documentation/git.txt > +++ b/Documentation/git.txt > @@ -1150,13 +1150,13 @@ of clones and fetches. > cloning a repository to make a backup). > > `GIT_ALLOW_PROTOCOL`:: > - If set, provide a colon-separated list of protocols which are > - allowed to be used with fetch/push/clone. This is useful to > - restrict recursive submodule initialization from an untrusted > - repository. Any protocol not mentioned will be disallowed (i.e., > - this is a whitelist, not a blacklist). If the variable is not > - set at all, all protocols are enabled. The protocol names > - currently used by git are: > + The new way to configure allowed protocols is done through the config This is not the right place to mention what is newer. ;) However it is useful to know about the config interface, which is * (supposedly) easier to use * more fine grained * taking less priority than this env var. > + test_expect_success "clone $desc (disabled)" ' > + rm -rf tmp.git && > + test_must_fail git clone --bare "$url" tmp.git > + ' I could not spot a test for GIT_ALLOW_PROTOCOL overriding any protocol*allow policy. Is that also worth testing? (for backwards compatibility of tools that make use of GIT_ALLOW_PROTOCOL but the user already setup a policy. > > void transport_check_allowed(const char *type) > -- Thanks! Stefan ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v3] transport: add protocol policy config option 2016-11-04 22:38 ` Stefan Beller @ 2016-11-07 18:14 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-07 18:14 UTC (permalink / raw) To: Stefan Beller; +Cc: git, Blake Burkhart, Jeff King, Jonathan Nieder On 11/04, Stefan Beller wrote: > By default, if unset, ... have a default policy ... > sounds strange. How about just dropping the first 4 words here: > > Known-safe protocols (http, https, git, ssh, file) have a > default policy of `always`, known-dangerous protocols (ext) have a > default policy of `never`, and all other protocols have a default policy > of `user`. Supported policies: > > > What happens if protocol.allow is set to true? That wouldn't be allowed, its an unknown value as the only permitted values are always, never, and user. > > > > > + if unset, known-safe protocols (http, https, git, ssh, file) have a > > + default policy of `always`, known-dangerous protocols (ext) have a > > + default policy of `never`, and all other protocols have a default policy > > + of `user`. Supported policies: > > ++ > > +-- > > + > > +* `always` - protocol is always able to be used. > > + > > +* `never` - protocol is never able to be used. > > + > > +* `user` - protocol is only able to be used when `GIT_PROTOCOL_FROM_USER` is > > + either unset or has a value of 1. This policy should be used when you want a > > + protocol to be usable by the user but don't want it used by commands which > > + execute clone/fetch/pull commands without user input, e.g. recursive > > + submodule initialization. > > + > > +-- > > + > > +protocol.<name>.allow:: > > + Set a policy to be used by protocol <name> with clone/fetch/pull commands. > > How does this interact with protocol.allow? > > When protocol.allow is set, this overrides the specific protocol. > If protocol is not set, it overrides the specific protocol as well(?) protocol.allow is a default for protocols which don't have a specific protocol.<name>.allow entry > > + test_expect_success "clone $desc (disabled)" ' > > + rm -rf tmp.git && > > + test_must_fail git clone --bare "$url" tmp.git > > + ' > > > I could not spot a test for GIT_ALLOW_PROTOCOL overriding > any protocol*allow policy. Is that also worth testing? (for > backwards compatibility of tools that make use of GIT_ALLOW_PROTOCOL > but the user already setup a policy. I can add in one quick test for that. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v3] transport: add protocol policy config option 2016-11-04 20:55 ` [PATCH v3] transport: add protocol policy " Brandon Williams 2016-11-04 20:58 ` Brandon Williams 2016-11-04 22:38 ` Stefan Beller @ 2016-11-04 23:06 ` Jeff King 2016-11-07 19:17 ` Brandon Williams 2016-11-07 19:35 ` [PATCH v4 1/2] lib-proto-disable: variable name fix Brandon Williams 3 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-11-04 23:06 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, jrnieder On Fri, Nov 04, 2016 at 01:55:33PM -0700, Brandon Williams wrote: > Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to > specify a whitelist of protocols to be used in clone/fetch/pull > commands. This patch introduces new configuration options for more > fine-grained control for allowing/disallowing protocols. This also has > the added benefit of allowing easier construction of a protocol > whitelist on systems where setting an environment variable is > non-trivial. Good rationale. > Now users can specify a policy to be used for each type of protocol via > the 'protocol.<name>.allow' config option. A default policy for all > unknown protocols can be set with the 'protocol.allow' config option. I think "unconfigured" is a better word here than "unknown", as it would apply to known protocols like "https", etc. That made me wonder if "unknown" would be a better behavior, but I'm pretty sure it is not. It is harder to explain, and I think would be less convenient in practice. I.e., you really do want: git config protocol.allow never git config protocol.https.allow always to allow nothing but https. > If no user configured default is made git, by default, will allow > known-safe protocols (http, https, git, ssh, file), disallow > known-dangerous protocols (ext), and have a default poliy of `user` for > all other protocols. I think this is a good way of thinking about it. The order of enforcement becomes: - GIT_ALLOW_PROTOCOL; environment variables always take precedence over config, so this makes sense. And it also is nice to put the blunt hammer at the front for backwards-compatibility. - protocol-specific config - protocol-generic config - built-in defaults (known-safe, known-scary, unknown) which seems right. Also, s/poliy/policy/. > The supported policies are `always`, `never`, and `user`. The `user` > policy can be used to configure a protocol to be usable when explicitly > used by a user, while disallowing it for commands which run > clone/fetch/pull commands without direct user intervention (e.g. > recursive initialization of submodules). Commands which can potentially > clone/fetch/pull from untrusted repositories without user intervention > can export `GIT_PROTOCOL_FROM_USER` with a value of '0' to prevent > protocols configured to the `user` policy from being used. Makes sense. I know "user" came from me. I don't know if there is a better word to describe it. I originally called it "cmdline", but that seemed too obscure (especially when a tool external to git sets it). Something like "trusted" might make sense (we allow it only in a more-trusted setting), but it's kind of vague. And it also doesn't leave room for there to be more types of trust in the future. So "user" is probably reasonable (or perhaps "user-only" or similar). > diff --git a/Documentation/config.txt b/Documentation/config.txt > index 27069ac..5d845c4 100644 > --- a/Documentation/config.txt > +++ b/Documentation/config.txt > @@ -2308,6 +2308,31 @@ pretty.<name>:: > Note that an alias with the same name as a built-in format > will be silently ignored. > > +protocol.allow:: > + If set, provide a user defined default policy for all protocols which > + don't explicitly have a policy (protocol.<name>.allow). By default, > + if unset, known-safe protocols (http, https, git, ssh, file) have a > + default policy of `always`, known-dangerous protocols (ext) have a > + default policy of `never`, and all other protocols have a default policy > + of `user`. Supported policies: > ++ > +-- > + > +* `always` - protocol is always able to be used. > + > +* `never` - protocol is never able to be used. > + > +* `user` - protocol is only able to be used when `GIT_PROTOCOL_FROM_USER` is > + either unset or has a value of 1. This policy should be used when you want a > + protocol to be usable by the user but don't want it used by commands which > + execute clone/fetch/pull commands without user input, e.g. recursive > + submodule initialization. Makes sense. I wonder if it would be good to emphasize _directly_ usable here. I.e., "...when you want a protocol to be directly usable by the user but don't want...". Should clone/fetch/pull also include push? > +protocol.<name>.allow:: > + Set a policy to be used by protocol <name> with clone/fetch/pull commands. > + Nice that this matches protocol.allow, so we don't need to re-explain that. Should the list of protocols be here? I know they're covered under GIT_ALLOW_PROTOCOL already, but if this is the preferred system, we should probably explain them here, and then just have GIT_ALLOW_PROTOCOL refer the user. > diff --git a/Documentation/git.txt b/Documentation/git.txt > index ab7215e..ab25580 100644 > --- a/Documentation/git.txt > +++ b/Documentation/git.txt > @@ -1150,13 +1150,13 @@ of clones and fetches. > cloning a repository to make a backup). > > `GIT_ALLOW_PROTOCOL`:: > - If set, provide a colon-separated list of protocols which are > - allowed to be used with fetch/push/clone. This is useful to > - restrict recursive submodule initialization from an untrusted > - repository. Any protocol not mentioned will be disallowed (i.e., > - this is a whitelist, not a blacklist). If the variable is not > - set at all, all protocols are enabled. The protocol names > - currently used by git are: > + The new way to configure allowed protocols is done through the config > + interface, though this setting takes precedences. See > + linkgit:git-config[1] for more details. If set, provide a > + colon-separated list of protocols which are allowed to be used with > + fetch/push/clone. Any protocol not mentioned will be disallowed (i.e., > + this is a whitelist, not a blacklist). The protocol names currently > + used by git are: I wonder if we can explain this in terms of the config system. Something like: If set to a colon-separated list of zero or more protocols, behave as if `protocol.allow` is set to `never`, and each of the listed protocols has `protocol.$protocol.allow` set to `always`. > +`GIT_PROTOCOL_FROM_USER`:: > + Set to 0 to prevent protocols used by fetch/push/clone which are > + configured to the `user` state. This is useful to restrict recursive > + submodule initialization from an untrusted repository. See > + linkgit:git-config[1] for more details. Under "this is useful", it may make sense to make it clear that external programs can use this, too. Something like: It may also be useful for programs which feed potentially-untrusted URLs to git commands. > diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh > index b0917d9..5950fbf 100644 > --- a/t/lib-proto-disable.sh > +++ b/t/lib-proto-disable.sh > @@ -1,15 +1,12 @@ > # Test routines for checking protocol disabling. > > -# test cloning a particular protocol > -# $1 - description of the protocol > -# $2 - machine-readable name of the protocol > -# $3 - the URL to try cloning > -test_proto () { > +# Test clone/fetch/push with GIT_ALLOW_PROTOCOL whitelist > +test_whitelist () { > desc=$1 > proto=$2 > url=$3 > > - test_expect_success "clone $1 (enabled)" ' > + test_expect_success "clone $desc (enabled)" ' Yeah, this should have been $desc all along. It makes the diff really noisy, though. Should it be split out into a preparatory change? > +# test cloning a particular protocol > +# $1 - description of the protocol > +# $2 - machine-readable name of the protocol > +# $3 - the URL to try cloning > +test_proto () { > + test_whitelist "$@" > + > + test_config "$@" > +} This makes sense. It's probably more testing than we actually need. We could just check the config version per-protocol, and then confirm that GIT_ALLOW_PROTOCOL behaves as I described above for at least one protocol. The per-protocol code paths are really just making sure that the protocol is correctly named for each code path. That being said, simple and stupid test setup is nice as long as it does not take too long to run. > diff --git a/t/t5509-fetch-push-namespaces.sh b/t/t5509-fetch-push-namespaces.sh > index bc44ac3..75c570a 100755 > --- a/t/t5509-fetch-push-namespaces.sh > +++ b/t/t5509-fetch-push-namespaces.sh > @@ -4,6 +4,7 @@ test_description='fetch/push involving ref namespaces' > . ./test-lib.sh > > test_expect_success setup ' > + git config --global protocol.ext.allow user && > test_tick && > git init original && These remote-ext fixups might be worth a note in the commit message, or a comment here explaining what is going on. > +static enum protocol_allow_config get_protocol_config(const char *type) > +{ > + char *key = xstrfmt("protocol.%s.allow", type); > + char *value; > + > + if (!git_config_get_string(key, &value)) { > + enum protocol_allow_config ret = > + parse_protocol_config(key, value); > + free(key); > + free(value); > + return ret; > + } > + free(key); > + > + /* if defined, use user default for unknown protocols */ > + if (!git_config_get_string("protocol.allow", &value)) { > + enum protocol_allow_config ret = > + parse_protocol_config("protocol.allow", value); > + free(value); > + return ret; > + } > + > + /* known safe */ > [...] It's probably worth a comment at this point in the function to follow-up on your "if defined" comment above. So the end result reads something like: /* first check the per-protocol config */ ... /* now fallback to the generic config */ ... /* and then fallback to our built-in defaults */ -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v3] transport: add protocol policy config option 2016-11-04 23:06 ` Jeff King @ 2016-11-07 19:17 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-07 19:17 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, jrnieder On 11/04, Jeff King wrote: > > diff --git a/Documentation/config.txt b/Documentation/config.txt > > index 27069ac..5d845c4 100644 > > --- a/Documentation/config.txt > > +++ b/Documentation/config.txt > > @@ -2308,6 +2308,31 @@ pretty.<name>:: > > Note that an alias with the same name as a built-in format > > will be silently ignored. > > > > +protocol.allow:: > > + If set, provide a user defined default policy for all protocols which > > + don't explicitly have a policy (protocol.<name>.allow). By default, > > + if unset, known-safe protocols (http, https, git, ssh, file) have a > > + default policy of `always`, known-dangerous protocols (ext) have a > > + default policy of `never`, and all other protocols have a default policy > > + of `user`. Supported policies: > > ++ > > +-- > > + > > +* `always` - protocol is always able to be used. > > + > > +* `never` - protocol is never able to be used. > > + > > +* `user` - protocol is only able to be used when `GIT_PROTOCOL_FROM_USER` is > > + either unset or has a value of 1. This policy should be used when you want a > > + protocol to be usable by the user but don't want it used by commands which > > + execute clone/fetch/pull commands without user input, e.g. recursive > > + submodule initialization. > > Makes sense. I wonder if it would be good to emphasize _directly_ usable > here. I.e., "...when you want a protocol to be directly usable by the > user but don't want...". > > Should clone/fetch/pull also include push? You're right, that should really have been clone/fetch/push. > > > +protocol.<name>.allow:: > > + Set a policy to be used by protocol <name> with clone/fetch/pull commands. > > + > > Nice that this matches protocol.allow, so we don't need to re-explain > that. > > Should the list of protocols be here? I know they're covered under > GIT_ALLOW_PROTOCOL already, but if this is the preferred system, we > should probably explain them here, and then just have GIT_ALLOW_PROTOCOL > refer the user. Right now the list of protocols under GIT_ALLOW_PROTOCOL looks like it has a bit more documentation with how the colon list works. The protocols are also mentioned above with their default behaviour. > > > diff --git a/Documentation/git.txt b/Documentation/git.txt > > index ab7215e..ab25580 100644 > > --- a/Documentation/git.txt > > +++ b/Documentation/git.txt > > @@ -1150,13 +1150,13 @@ of clones and fetches. > > cloning a repository to make a backup). > > > > `GIT_ALLOW_PROTOCOL`:: > > - If set, provide a colon-separated list of protocols which are > > - allowed to be used with fetch/push/clone. This is useful to > > - restrict recursive submodule initialization from an untrusted > > - repository. Any protocol not mentioned will be disallowed (i.e., > > - this is a whitelist, not a blacklist). If the variable is not > > - set at all, all protocols are enabled. The protocol names > > - currently used by git are: > > + The new way to configure allowed protocols is done through the config > > + interface, though this setting takes precedences. See > > + linkgit:git-config[1] for more details. If set, provide a > > + colon-separated list of protocols which are allowed to be used with > > + fetch/push/clone. Any protocol not mentioned will be disallowed (i.e., > > + this is a whitelist, not a blacklist). The protocol names currently > > + used by git are: > > I wonder if we can explain this in terms of the config system. Something > like: > > If set to a colon-separated list of zero or more protocols, behave as > if `protocol.allow` is set to `never`, and each of the listed > protocols has `protocol.$protocol.allow` set to `always`. Yeah that makes sense. > > > +`GIT_PROTOCOL_FROM_USER`:: > > + Set to 0 to prevent protocols used by fetch/push/clone which are > > + configured to the `user` state. This is useful to restrict recursive > > + submodule initialization from an untrusted repository. See > > + linkgit:git-config[1] for more details. > > Under "this is useful", it may make sense to make it clear that external > programs can use this, too. Something like: > > It may also be useful for programs which feed potentially-untrusted > URLs to git commands. I'll put that in too. > > > diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh > > index b0917d9..5950fbf 100644 > > --- a/t/lib-proto-disable.sh > > +++ b/t/lib-proto-disable.sh > > @@ -1,15 +1,12 @@ > > # Test routines for checking protocol disabling. > > > > -# test cloning a particular protocol > > -# $1 - description of the protocol > > -# $2 - machine-readable name of the protocol > > -# $3 - the URL to try cloning > > -test_proto () { > > +# Test clone/fetch/push with GIT_ALLOW_PROTOCOL whitelist > > +test_whitelist () { > > desc=$1 > > proto=$2 > > url=$3 > > > > - test_expect_success "clone $1 (enabled)" ' > > + test_expect_success "clone $desc (enabled)" ' > > Yeah, this should have been $desc all along. It makes the diff really > noisy, though. Should it be split out into a preparatory change? I'll pull it out to make the patch a bit cleaner. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v4 1/2] lib-proto-disable: variable name fix 2016-11-04 20:55 ` [PATCH v3] transport: add protocol policy " Brandon Williams ` (2 preceding siblings ...) 2016-11-04 23:06 ` Jeff King @ 2016-11-07 19:35 ` Brandon Williams 2016-11-07 19:35 ` [PATCH v4 2/2] transport: add protocol policy config option Brandon Williams ` (2 more replies) 3 siblings, 3 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-07 19:35 UTC (permalink / raw) To: git; +Cc: Brandon Williams, sbeller, bburky, peff, jrnieder Small fix to use '$desc' instead of '$1' in lib-proto-disable.sh. Signed-off-by: Brandon Williams <bmwill@google.com> --- t/lib-proto-disable.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index b0917d9..be88e9a 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -9,7 +9,7 @@ test_proto () { proto=$2 url=$3 - test_expect_success "clone $1 (enabled)" ' + test_expect_success "clone $desc (enabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=$proto && @@ -18,7 +18,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (enabled)" ' + test_expect_success "fetch $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -27,7 +27,7 @@ test_proto () { ) ' - test_expect_success "push $1 (enabled)" ' + test_expect_success "push $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -36,7 +36,7 @@ test_proto () { ) ' - test_expect_success "push $1 (disabled)" ' + test_expect_success "push $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -45,7 +45,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (disabled)" ' + test_expect_success "fetch $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -54,7 +54,7 @@ test_proto () { ) ' - test_expect_success "clone $1 (disabled)" ' + test_expect_success "clone $desc (disabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=none && -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v4 2/2] transport: add protocol policy config option 2016-11-07 19:35 ` [PATCH v4 1/2] lib-proto-disable: variable name fix Brandon Williams @ 2016-11-07 19:35 ` Brandon Williams 2016-11-07 20:44 ` Jeff King 2016-11-07 20:26 ` [PATCH v4 1/2] lib-proto-disable: variable name fix Jeff King 2016-11-07 21:51 ` [PATCH v5 " Brandon Williams 2 siblings, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-11-07 19:35 UTC (permalink / raw) To: git; +Cc: Brandon Williams, sbeller, bburky, peff, jrnieder Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to specify a whitelist of protocols to be used in clone/fetch/push commands. This patch introduces new configuration options for more fine-grained control for allowing/disallowing protocols. This also has the added benefit of allowing easier construction of a protocol whitelist on systems where setting an environment variable is non-trivial. Now users can specify a policy to be used for each type of protocol via the 'protocol.<name>.allow' config option. A default policy for all unconfigured protocols can be set with the 'protocol.allow' config option. If no user configured default is made git, by default, will allow known-safe protocols (http, https, git, ssh, file), disallow known-dangerous protocols (ext), and have a default policy of `user` for all other protocols. The supported policies are `always`, `never`, and `user`. The `user` policy can be used to configure a protocol to be usable when explicitly used by a user, while disallowing it for commands which run clone/fetch/push commands without direct user intervention (e.g. recursive initialization of submodules). Commands which can potentially clone/fetch/push from untrusted repositories without user intervention can export `GIT_PROTOCOL_FROM_USER` with a value of '0' to prevent protocols configured to the `user` policy from being used. Fix remote-ext tests to use the new config to allow the ext protocol to be tested. Based on a patch by Jeff King <peff@peff.net> Signed-off-by: Brandon Williams <bmwill@google.com> --- Documentation/config.txt | 25 ++++++++ Documentation/git.txt | 21 ++++--- git-submodule.sh | 12 ++-- t/lib-proto-disable.sh | 129 +++++++++++++++++++++++++++++++++++++-- t/t5509-fetch-push-namespaces.sh | 1 + t/t5802-connect-helper.sh | 1 + transport.c | 75 ++++++++++++++++++++++- 7 files changed, 242 insertions(+), 22 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index 27069ac..0b1c186 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -2308,6 +2308,31 @@ pretty.<name>:: Note that an alias with the same name as a built-in format will be silently ignored. +protocol.allow:: + If set, provide a user defined default policy for all protocols which + don't explicitly have a policy (`protocol.<name>.allow`). By default, + if unset, known-safe protocols (http, https, git, ssh, file) have a + default policy of `always`, known-dangerous protocols (ext) have a + default policy of `never`, and all other protocols have a default + policy of `user`. Supported policies: ++ +-- + +* `always` - protocol is always able to be used. + +* `never` - protocol is never able to be used. + +* `user` - protocol is only able to be used when `GIT_PROTOCOL_FROM_USER` is + either unset or has a value of 1. This policy should be used when you want a + protocol to be directly usable by the user but don't want it used by commands which + execute clone/fetch/push commands without user input, e.g. recursive + submodule initialization. + +-- + +protocol.<name>.allow:: + Set a policy to be used by protocol <name> with clone/fetch/push commands. + pull.ff:: By default, Git does not create an extra merge commit when merging a commit that is a descendant of the current commit. Instead, the diff --git a/Documentation/git.txt b/Documentation/git.txt index ab7215e..c9823e3 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -1150,13 +1150,14 @@ of clones and fetches. cloning a repository to make a backup). `GIT_ALLOW_PROTOCOL`:: - If set, provide a colon-separated list of protocols which are - allowed to be used with fetch/push/clone. This is useful to - restrict recursive submodule initialization from an untrusted - repository. Any protocol not mentioned will be disallowed (i.e., - this is a whitelist, not a blacklist). If the variable is not - set at all, all protocols are enabled. The protocol names - currently used by git are: + The preferred way to configure allowed protocols is done through the + config interface, though this setting takes precedences. See + linkgit:git-config[1] for more details. If set to a colon-separated + list of protocols, behave as if `protocol.allow` is set to `never`, and + each of the listed protocols has `protocol.<name>.allow` set to + `always`. In other words, any protocol not mentioned will be + disallowed (i.e., this is a whitelist, not a blacklist). The protocol + names currently used by git are: - `file`: any local file-based path (including `file://` URLs, or local paths) @@ -1174,6 +1175,12 @@ of clones and fetches. - any external helpers are named by their protocol (e.g., use `hg` to allow the `git-remote-hg` helper) +`GIT_PROTOCOL_FROM_USER`:: + Set to 0 to prevent protocols used by fetch/push/clone which are + configured to the `user` state. This is useful to restrict recursive + submodule initialization from an untrusted repository or for programs + which feed potentially-untrusted URLS to git commands. See + linkgit:git-config[1] for more details. Discussion[[Discussion]] ------------------------ diff --git a/git-submodule.sh b/git-submodule.sh index a024a13..0a477b4 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -21,14 +21,10 @@ require_work_tree wt_prefix=$(git rev-parse --show-prefix) cd_to_toplevel -# Restrict ourselves to a vanilla subset of protocols; the URLs -# we get are under control of a remote repository, and we do not -# want them kicking off arbitrary git-remote-* programs. -# -# If the user has already specified a set of allowed protocols, -# we assume they know what they're doing and use that instead. -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} -export GIT_ALLOW_PROTOCOL +# Tell the rest of git that any URLs we get don't come +# directly from the user, so it can apply policy as appropriate. +GIT_PROTOCOL_FROM_USER=0 +export GIT_PROTOCOL_FROM_USER command= branch= diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index be88e9a..4b2db2f 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -1,10 +1,7 @@ # Test routines for checking protocol disabling. -# test cloning a particular protocol -# $1 - description of the protocol -# $2 - machine-readable name of the protocol -# $3 - the URL to try cloning -test_proto () { +# Test clone/fetch/push with GIT_ALLOW_PROTOCOL whitelist +test_whitelist () { desc=$1 proto=$2 url=$3 @@ -62,6 +59,128 @@ test_proto () { test_must_fail git clone --bare "$url" tmp.git ) ' + + test_expect_success "clone $desc (env var has precedence)" ' + rm -rf tmp.git && + ( + GIT_ALLOW_PROTOCOL=none && + export GIT_ALLOW_PROTOCOL && + test_must_fail git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ) + ' +} + +test_config () { + desc=$1 + proto=$2 + url=$3 + + # Test clone/fetch/push with protocol.<type>.allow config + test_expect_success "clone $desc (enabled with config)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git -c protocol.$proto.allow=never clone --bare "$url" tmp.git + ' + + # Test clone/fetch/push with protocol.user.allow and its env var + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user push origin HEAD:pushed + ) + ' + + test_expect_success "fetch $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user fetch + ) + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + ( + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ) + ' + + # Test clone/fetch/push with protocol.allow user defined default + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git config --global protocol.allow always && + git clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + git config --global protocol.allow never && + test_must_fail git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git clone --bare "$url" tmp.git + ' +} + +# test cloning a particular protocol +# $1 - description of the protocol +# $2 - machine-readable name of the protocol +# $3 - the URL to try cloning +test_proto () { + test_whitelist "$@" + + test_config "$@" } # set up an ssh wrapper that will access $host/$repo in the diff --git a/t/t5509-fetch-push-namespaces.sh b/t/t5509-fetch-push-namespaces.sh index bc44ac3..75c570a 100755 --- a/t/t5509-fetch-push-namespaces.sh +++ b/t/t5509-fetch-push-namespaces.sh @@ -4,6 +4,7 @@ test_description='fetch/push involving ref namespaces' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git init original && ( diff --git a/t/t5802-connect-helper.sh b/t/t5802-connect-helper.sh index b7a7f9d..c6c2661 100755 --- a/t/t5802-connect-helper.sh +++ b/t/t5802-connect-helper.sh @@ -4,6 +4,7 @@ test_description='ext::cmd remote "connect" helper' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git commit --allow-empty -m initial && test_tick && diff --git a/transport.c b/transport.c index d57e8de..2c0ec76 100644 --- a/transport.c +++ b/transport.c @@ -664,10 +664,81 @@ static const struct string_list *protocol_whitelist(void) return enabled ? &allowed : NULL; } +enum protocol_allow_config { + PROTOCOL_ALLOW_NEVER = 0, + PROTOCOL_ALLOW_USER_ONLY, + PROTOCOL_ALLOW_ALWAYS +}; + +static enum protocol_allow_config parse_protocol_config(const char *key, + const char *value) +{ + if (!strcasecmp(value, "always")) + return PROTOCOL_ALLOW_ALWAYS; + else if (!strcasecmp(value, "never")) + return PROTOCOL_ALLOW_NEVER; + else if (!strcasecmp(value, "user")) + return PROTOCOL_ALLOW_USER_ONLY; + + die("unknown value for config '%s': %s", key, value); +} + +static enum protocol_allow_config get_protocol_config(const char *type) +{ + char *key = xstrfmt("protocol.%s.allow", type); + char *value; + + /* first check the per-protocol config */ + if (!git_config_get_string(key, &value)) { + enum protocol_allow_config ret = + parse_protocol_config(key, value); + free(key); + free(value); + return ret; + } + free(key); + + /* if defined, fallback to user-defined default for unknown protocols */ + if (!git_config_get_string("protocol.allow", &value)) { + enum protocol_allow_config ret = + parse_protocol_config("protocol.allow", value); + free(value); + return ret; + } + + /* fallback to built-in defaults */ + /* known safe */ + if (!strcmp(type, "http") || + !strcmp(type, "https") || + !strcmp(type, "git") || + !strcmp(type, "ssh") || + !strcmp(type, "file")) + return PROTOCOL_ALLOW_ALWAYS; + + /* known scary; err on the side of caution */ + if (!strcmp(type, "ext")) + return PROTOCOL_ALLOW_NEVER; + + /* unknown; by default let them be used only directly by the user */ + return PROTOCOL_ALLOW_USER_ONLY; +} + int is_transport_allowed(const char *type) { - const struct string_list *allowed = protocol_whitelist(); - return !allowed || string_list_has_string(allowed, type); + const struct string_list *whitelist = protocol_whitelist(); + if (whitelist) + return string_list_has_string(whitelist, type); + + switch (get_protocol_config(type)) { + case PROTOCOL_ALLOW_ALWAYS: + return 1; + case PROTOCOL_ALLOW_NEVER: + return 0; + case PROTOCOL_ALLOW_USER_ONLY: + return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + } + + die("BUG: invalid protocol_allow_config type"); } void transport_check_allowed(const char *type) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v4 2/2] transport: add protocol policy config option 2016-11-07 19:35 ` [PATCH v4 2/2] transport: add protocol policy config option Brandon Williams @ 2016-11-07 20:44 ` Jeff King 2016-11-07 21:02 ` Brandon Williams 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-11-07 20:44 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, jrnieder On Mon, Nov 07, 2016 at 11:35:23AM -0800, Brandon Williams wrote: > Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to > specify a whitelist of protocols to be used in clone/fetch/push > commands. This patch introduces new configuration options for more > fine-grained control for allowing/disallowing protocols. This also has > the added benefit of allowing easier construction of a protocol > whitelist on systems where setting an environment variable is > non-trivial. > > Now users can specify a policy to be used for each type of protocol via > the 'protocol.<name>.allow' config option. A default policy for all > unconfigured protocols can be set with the 'protocol.allow' config > option. If no user configured default is made git, by default, will > allow known-safe protocols (http, https, git, ssh, file), disallow A minor nit, but in "If no user configured default is made git, by default, will..." the second "by default" is redundant. And possibly misleading. This _is_ the default case, there is no other way to change it. :) > +protocol.<name>.allow:: > + Set a policy to be used by protocol <name> with clone/fetch/push commands. `<name>` isn't defined here at all. I still think the list of protocols should go here, but at the very least, you need to point the user to the existing list in git(1). > `GIT_ALLOW_PROTOCOL`:: > - If set, provide a colon-separated list of protocols which are > - allowed to be used with fetch/push/clone. This is useful to > - restrict recursive submodule initialization from an untrusted > - repository. Any protocol not mentioned will be disallowed (i.e., > - this is a whitelist, not a blacklist). If the variable is not > - set at all, all protocols are enabled. The protocol names > - currently used by git are: > + The preferred way to configure allowed protocols is done through the > + config interface, though this setting takes precedences. See s/precedences/precedence/. I actually wonder if we should even drop "the preferred way" here. I had initially thought we would want it just for backwards-compatibility, but I actually think it is useful in its own right as a shorthand for more complicated config (and since we have to keep it around effectively forever anyway, there's no real cost to continuing to call it a feature versus a deprecated feature). I'm including a squashable patch at the end of this email with suggested wording (and which also moves the protocol list). > + test_expect_success "clone $desc (env var has precedence)" ' > + rm -rf tmp.git && > + ( > + GIT_ALLOW_PROTOCOL=none && > + export GIT_ALLOW_PROTOCOL && > + test_must_fail git -c protocol.$proto.allow=always clone --bare "$url" tmp.git > + ) > + ' This test is a good addition in this round. I suppose we could test also that GIT_ALLOW_PROTOCOL overrides protocol.allow, but I'm not sure if there is a point. If git were a black box, it's a thing I might check, but we know from the design that this is an unlikely bug (and that the implementation is unlikely to change in a way to cause it). So I could go either way. > [...] The rest of it looks good to me. Squashable documentation suggestions are below. -Peff --- diff --git a/Documentation/config.txt b/Documentation/config.txt index e89b33f9e..a9dc23f82 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -2331,7 +2331,28 @@ protocol.allow:: -- protocol.<name>.allow:: - Set a policy to be used by protocol <name> with clone/fetch/push commands. + Set a policy to be used by protocol `<name>` with clone/fetch/push + commands. See `protocol.allow` above for the available policies. ++ +The protocol names currently used by git are: ++ +-- + - `file`: any local file-based path (including `file://` URLs, + or local paths) + + - `git`: the anonymous git protocol over a direct TCP + connection (or proxy, if configured) + + - `ssh`: git over ssh (including `host:path` syntax, + `ssh://`, etc). + + - `http`: git over http, both "smart http" and "dumb http". + Note that this does _not_ include `https`; if you want to configure + both, you must do so individually. + + - any external helpers are named by their protocol (e.g., use + `hg` to allow the `git-remote-hg` helper) +-- pull.ff:: By default, Git does not create an extra merge commit when merging diff --git a/Documentation/git.txt b/Documentation/git.txt index c9823e34a..c52cec840 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -1150,30 +1150,13 @@ of clones and fetches. cloning a repository to make a backup). `GIT_ALLOW_PROTOCOL`:: - The preferred way to configure allowed protocols is done through the - config interface, though this setting takes precedences. See - linkgit:git-config[1] for more details. If set to a colon-separated - list of protocols, behave as if `protocol.allow` is set to `never`, and - each of the listed protocols has `protocol.<name>.allow` set to - `always`. In other words, any protocol not mentioned will be - disallowed (i.e., this is a whitelist, not a blacklist). The protocol - names currently used by git are: - - - `file`: any local file-based path (including `file://` URLs, - or local paths) - - - `git`: the anonymous git protocol over a direct TCP - connection (or proxy, if configured) - - - `ssh`: git over ssh (including `host:path` syntax, - `ssh://`, etc). - - - `http`: git over http, both "smart http" and "dumb http". - Note that this does _not_ include `https`; if you want both, - you should specify both as `http:https`. - - - any external helpers are named by their protocol (e.g., use - `hg` to allow the `git-remote-hg` helper) + If set to a colon-separated list of protocols, behave as if + `protocol.allow` is set to `never`, and each of the listed + protocols has `protocol.<name>.allow` set to `always` + (overriding any existing configuration). In other words, any + protocol not mentioned will be disallowed (i.e., this is a + whitelist, not a blacklist). See the description of + `protocol.allow` in linkgit:git-config[1] for more details. `GIT_PROTOCOL_FROM_USER`:: Set to 0 to prevent protocols used by fetch/push/clone which are ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v4 2/2] transport: add protocol policy config option 2016-11-07 20:44 ` Jeff King @ 2016-11-07 21:02 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-07 21:02 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, jrnieder On 11/07, Jeff King wrote: > > + test_expect_success "clone $desc (env var has precedence)" ' > > + rm -rf tmp.git && > > + ( > > + GIT_ALLOW_PROTOCOL=none && > > + export GIT_ALLOW_PROTOCOL && > > + test_must_fail git -c protocol.$proto.allow=always clone --bare "$url" tmp.git > > + ) > > + ' > > This test is a good addition in this round. > > I suppose we could test also that GIT_ALLOW_PROTOCOL overrides > protocol.allow, but I'm not sure if there is a point. If git were a > black box, it's a thing I might check, but we know from the design that > this is an unlikely bug (and that the implementation is unlikely to > change in a way to cause it). So I could go either way. I'll add in another test for that, no reason not to test it. > > Squashable documentation suggestions are below. > Sounds good -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v4 1/2] lib-proto-disable: variable name fix 2016-11-07 19:35 ` [PATCH v4 1/2] lib-proto-disable: variable name fix Brandon Williams 2016-11-07 19:35 ` [PATCH v4 2/2] transport: add protocol policy config option Brandon Williams @ 2016-11-07 20:26 ` Jeff King 2016-11-07 20:40 ` Brandon Williams 2016-11-07 21:51 ` [PATCH v5 " Brandon Williams 2 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-11-07 20:26 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, jrnieder On Mon, Nov 07, 2016 at 11:35:22AM -0800, Brandon Williams wrote: > Small fix to use '$desc' instead of '$1' in lib-proto-disable.sh. Even for a trivial fixup like this, I think it's good to say why. Because what seems trivial and obvious to you while working on the patch may not be so to a reviewer, or somebody reading it 6 months later. Just something simple like: The test_proto function assigns the positional parameters to named variables, but then still refers to "$desc" as "$1". Using $desc is more readable and less error-prone. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v4 1/2] lib-proto-disable: variable name fix 2016-11-07 20:26 ` [PATCH v4 1/2] lib-proto-disable: variable name fix Jeff King @ 2016-11-07 20:40 ` Brandon Williams 2016-11-07 20:48 ` Jeff King 0 siblings, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-11-07 20:40 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, jrnieder On 11/07, Jeff King wrote: > On Mon, Nov 07, 2016 at 11:35:22AM -0800, Brandon Williams wrote: > > > Small fix to use '$desc' instead of '$1' in lib-proto-disable.sh. > > Even for a trivial fixup like this, I think it's good to say why. > Because what seems trivial and obvious to you while working on the patch > may not be so to a reviewer, or somebody reading it 6 months later. > > Just something simple like: > > The test_proto function assigns the positional parameters to named > variables, but then still refers to "$desc" as "$1". Using $desc is > more readable and less error-prone. > > -Peff Alright will do. Commit messages don't seem to be an area of strength for me, but I'm working on it! :D -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v4 1/2] lib-proto-disable: variable name fix 2016-11-07 20:40 ` Brandon Williams @ 2016-11-07 20:48 ` Jeff King 2016-11-08 3:32 ` Jacob Keller 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-11-07 20:48 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, jrnieder On Mon, Nov 07, 2016 at 12:40:28PM -0800, Brandon Williams wrote: > On 11/07, Jeff King wrote: > > On Mon, Nov 07, 2016 at 11:35:22AM -0800, Brandon Williams wrote: > > > > > Small fix to use '$desc' instead of '$1' in lib-proto-disable.sh. > > > > Even for a trivial fixup like this, I think it's good to say why. > > Because what seems trivial and obvious to you while working on the patch > > may not be so to a reviewer, or somebody reading it 6 months later. > > > > Just something simple like: > > > > The test_proto function assigns the positional parameters to named > > variables, but then still refers to "$desc" as "$1". Using $desc is > > more readable and less error-prone. > > > > -Peff > > Alright will do. Commit messages don't seem to be an area of strength > for me, but I'm working on it! :D It's possible that I'm overly picky about my commit messages, but that does not stop me from trying to train an army of picky-commit-message clones. :) -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v4 1/2] lib-proto-disable: variable name fix 2016-11-07 20:48 ` Jeff King @ 2016-11-08 3:32 ` Jacob Keller 2016-11-08 20:52 ` Junio C Hamano 0 siblings, 1 reply; 124+ messages in thread From: Jacob Keller @ 2016-11-08 3:32 UTC (permalink / raw) To: Jeff King Cc: Brandon Williams, Git mailing list, Stefan Beller, bburky, Jonathan Nieder On Mon, Nov 7, 2016 at 12:48 PM, Jeff King <peff@peff.net> wrote: > It's possible that I'm overly picky about my commit messages, but that > does not stop me from trying to train an army of picky-commit-message > clones. :) > > -Peff You're not the only one ;) Regards, Jake ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v4 1/2] lib-proto-disable: variable name fix 2016-11-08 3:32 ` Jacob Keller @ 2016-11-08 20:52 ` Junio C Hamano 0 siblings, 0 replies; 124+ messages in thread From: Junio C Hamano @ 2016-11-08 20:52 UTC (permalink / raw) To: Jacob Keller Cc: Jeff King, Brandon Williams, Git mailing list, Stefan Beller, bburky, Jonathan Nieder Jacob Keller <jacob.keller@gmail.com> writes: > On Mon, Nov 7, 2016 at 12:48 PM, Jeff King <peff@peff.net> wrote: >> It's possible that I'm overly picky about my commit messages, but that >> does not stop me from trying to train an army of picky-commit-message >> clones. :) >> >> -Peff > > You're not the only one ;) Somebody seems to have trained y'all very well ;-) ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v5 1/2] lib-proto-disable: variable name fix 2016-11-07 19:35 ` [PATCH v4 1/2] lib-proto-disable: variable name fix Brandon Williams 2016-11-07 19:35 ` [PATCH v4 2/2] transport: add protocol policy config option Brandon Williams 2016-11-07 20:26 ` [PATCH v4 1/2] lib-proto-disable: variable name fix Jeff King @ 2016-11-07 21:51 ` Brandon Williams 2016-11-07 21:51 ` [PATCH v5 2/2] transport: add protocol policy config option Brandon Williams 2016-12-01 19:44 ` [PATCH v6 0/4] transport protocol policy configuration Brandon Williams 2 siblings, 2 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-07 21:51 UTC (permalink / raw) To: git; +Cc: Brandon Williams, sbeller, bburky, peff, jrnieder The test_proto function assigns the positional parameters to named variables, but then still refers to "$desc" as "$1". Using $desc is more readable and less error-prone. Signed-off-by: Brandon Williams <bmwill@google.com> --- t/lib-proto-disable.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index b0917d9..be88e9a 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -9,7 +9,7 @@ test_proto () { proto=$2 url=$3 - test_expect_success "clone $1 (enabled)" ' + test_expect_success "clone $desc (enabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=$proto && @@ -18,7 +18,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (enabled)" ' + test_expect_success "fetch $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -27,7 +27,7 @@ test_proto () { ) ' - test_expect_success "push $1 (enabled)" ' + test_expect_success "push $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -36,7 +36,7 @@ test_proto () { ) ' - test_expect_success "push $1 (disabled)" ' + test_expect_success "push $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -45,7 +45,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (disabled)" ' + test_expect_success "fetch $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -54,7 +54,7 @@ test_proto () { ) ' - test_expect_success "clone $1 (disabled)" ' + test_expect_success "clone $desc (disabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=none && -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v5 2/2] transport: add protocol policy config option 2016-11-07 21:51 ` [PATCH v5 " Brandon Williams @ 2016-11-07 21:51 ` Brandon Williams 2016-11-08 22:04 ` Jeff King 2016-12-01 19:44 ` [PATCH v6 0/4] transport protocol policy configuration Brandon Williams 1 sibling, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-11-07 21:51 UTC (permalink / raw) To: git; +Cc: Brandon Williams, sbeller, bburky, peff, jrnieder Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to specify a whitelist of protocols to be used in clone/fetch/push commands. This patch introduces new configuration options for more fine-grained control for allowing/disallowing protocols. This also has the added benefit of allowing easier construction of a protocol whitelist on systems where setting an environment variable is non-trivial. Now users can specify a policy to be used for each type of protocol via the 'protocol.<name>.allow' config option. A default policy for all unconfigured protocols can be set with the 'protocol.allow' config option. If no user configured default is made git will allow known-safe protocols (http, https, git, ssh, file), disallow known-dangerous protocols (ext), and have a default policy of `user` for all other protocols. The supported policies are `always`, `never`, and `user`. The `user` policy can be used to configure a protocol to be usable when explicitly used by a user, while disallowing it for commands which run clone/fetch/push commands without direct user intervention (e.g. recursive initialization of submodules). Commands which can potentially clone/fetch/push from untrusted repositories without user intervention can export `GIT_PROTOCOL_FROM_USER` with a value of '0' to prevent protocols configured to the `user` policy from being used. Fix remote-ext tests to use the new config to allow the ext protocol to be tested. Based on a patch by Jeff King <peff@peff.net> Signed-off-by: Brandon Williams <bmwill@google.com> --- Documentation/config.txt | 46 ++++++++++++++ Documentation/git.txt | 38 +++++------- git-submodule.sh | 12 ++-- t/lib-proto-disable.sh | 130 +++++++++++++++++++++++++++++++++++++-- t/t5509-fetch-push-namespaces.sh | 1 + t/t5802-connect-helper.sh | 1 + transport.c | 75 +++++++++++++++++++++- 7 files changed, 264 insertions(+), 39 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index 27069ac..5fe50bc 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -2308,6 +2308,52 @@ pretty.<name>:: Note that an alias with the same name as a built-in format will be silently ignored. +protocol.allow:: + If set, provide a user defined default policy for all protocols which + don't explicitly have a policy (`protocol.<name>.allow`). By default, + if unset, known-safe protocols (http, https, git, ssh, file) have a + default policy of `always`, known-dangerous protocols (ext) have a + default policy of `never`, and all other protocols have a default + policy of `user`. Supported policies: ++ +-- + +* `always` - protocol is always able to be used. + +* `never` - protocol is never able to be used. + +* `user` - protocol is only able to be used when `GIT_PROTOCOL_FROM_USER` is + either unset or has a value of 1. This policy should be used when you want a + protocol to be directly usable by the user but don't want it used by commands which + execute clone/fetch/push commands without user input, e.g. recursive + submodule initialization. + +-- + +protocol.<name>.allow:: + Set a policy to be used by protocol `<name>` with clone/fetch/push + commands. See `protocol.allow` above for the available policies. ++ +The protocol names currently used by git are: ++ +-- + - `file`: any local file-based path (including `file://` URLs, + or local paths) + + - `git`: the anonymous git protocol over a direct TCP + connection (or proxy, if configured) + + - `ssh`: git over ssh (including `host:path` syntax, + `ssh://`, etc). + + - `http`: git over http, both "smart http" and "dumb http". + Note that this does _not_ include `https`; if you want to configure + both, you must do so individually. + + - any external helpers are named by their protocol (e.g., use + `hg` to allow the `git-remote-hg` helper) +-- + pull.ff:: By default, Git does not create an extra merge commit when merging a commit that is a descendant of the current commit. Instead, the diff --git a/Documentation/git.txt b/Documentation/git.txt index ab7215e..c52cec8 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -1150,30 +1150,20 @@ of clones and fetches. cloning a repository to make a backup). `GIT_ALLOW_PROTOCOL`:: - If set, provide a colon-separated list of protocols which are - allowed to be used with fetch/push/clone. This is useful to - restrict recursive submodule initialization from an untrusted - repository. Any protocol not mentioned will be disallowed (i.e., - this is a whitelist, not a blacklist). If the variable is not - set at all, all protocols are enabled. The protocol names - currently used by git are: - - - `file`: any local file-based path (including `file://` URLs, - or local paths) - - - `git`: the anonymous git protocol over a direct TCP - connection (or proxy, if configured) - - - `ssh`: git over ssh (including `host:path` syntax, - `ssh://`, etc). - - - `http`: git over http, both "smart http" and "dumb http". - Note that this does _not_ include `https`; if you want both, - you should specify both as `http:https`. - - - any external helpers are named by their protocol (e.g., use - `hg` to allow the `git-remote-hg` helper) - + If set to a colon-separated list of protocols, behave as if + `protocol.allow` is set to `never`, and each of the listed + protocols has `protocol.<name>.allow` set to `always` + (overriding any existing configuration). In other words, any + protocol not mentioned will be disallowed (i.e., this is a + whitelist, not a blacklist). See the description of + `protocol.allow` in linkgit:git-config[1] for more details. + +`GIT_PROTOCOL_FROM_USER`:: + Set to 0 to prevent protocols used by fetch/push/clone which are + configured to the `user` state. This is useful to restrict recursive + submodule initialization from an untrusted repository or for programs + which feed potentially-untrusted URLS to git commands. See + linkgit:git-config[1] for more details. Discussion[[Discussion]] ------------------------ diff --git a/git-submodule.sh b/git-submodule.sh index a024a13..0a477b4 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -21,14 +21,10 @@ require_work_tree wt_prefix=$(git rev-parse --show-prefix) cd_to_toplevel -# Restrict ourselves to a vanilla subset of protocols; the URLs -# we get are under control of a remote repository, and we do not -# want them kicking off arbitrary git-remote-* programs. -# -# If the user has already specified a set of allowed protocols, -# we assume they know what they're doing and use that instead. -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} -export GIT_ALLOW_PROTOCOL +# Tell the rest of git that any URLs we get don't come +# directly from the user, so it can apply policy as appropriate. +GIT_PROTOCOL_FROM_USER=0 +export GIT_PROTOCOL_FROM_USER command= branch= diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index be88e9a..02f49cb 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -1,10 +1,7 @@ # Test routines for checking protocol disabling. -# test cloning a particular protocol -# $1 - description of the protocol -# $2 - machine-readable name of the protocol -# $3 - the URL to try cloning -test_proto () { +# Test clone/fetch/push with GIT_ALLOW_PROTOCOL whitelist +test_whitelist () { desc=$1 proto=$2 url=$3 @@ -62,6 +59,129 @@ test_proto () { test_must_fail git clone --bare "$url" tmp.git ) ' + + test_expect_success "clone $desc (env var has precedence)" ' + rm -rf tmp.git && + ( + GIT_ALLOW_PROTOCOL=none && + export GIT_ALLOW_PROTOCOL && + test_must_fail git -c protocol.allow=always clone --bare "$url" tmp.git && + test_must_fail git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ) + ' +} + +test_config () { + desc=$1 + proto=$2 + url=$3 + + # Test clone/fetch/push with protocol.<type>.allow config + test_expect_success "clone $desc (enabled with config)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git -c protocol.$proto.allow=never clone --bare "$url" tmp.git + ' + + # Test clone/fetch/push with protocol.user.allow and its env var + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user push origin HEAD:pushed + ) + ' + + test_expect_success "fetch $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user fetch + ) + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + ( + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ) + ' + + # Test clone/fetch/push with protocol.allow user defined default + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git config --global protocol.allow always && + git clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + git config --global protocol.allow never && + test_must_fail git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git clone --bare "$url" tmp.git + ' +} + +# test cloning a particular protocol +# $1 - description of the protocol +# $2 - machine-readable name of the protocol +# $3 - the URL to try cloning +test_proto () { + test_whitelist "$@" + + test_config "$@" } # set up an ssh wrapper that will access $host/$repo in the diff --git a/t/t5509-fetch-push-namespaces.sh b/t/t5509-fetch-push-namespaces.sh index bc44ac3..75c570a 100755 --- a/t/t5509-fetch-push-namespaces.sh +++ b/t/t5509-fetch-push-namespaces.sh @@ -4,6 +4,7 @@ test_description='fetch/push involving ref namespaces' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git init original && ( diff --git a/t/t5802-connect-helper.sh b/t/t5802-connect-helper.sh index b7a7f9d..c6c2661 100755 --- a/t/t5802-connect-helper.sh +++ b/t/t5802-connect-helper.sh @@ -4,6 +4,7 @@ test_description='ext::cmd remote "connect" helper' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git commit --allow-empty -m initial && test_tick && diff --git a/transport.c b/transport.c index d57e8de..2c0ec76 100644 --- a/transport.c +++ b/transport.c @@ -664,10 +664,81 @@ static const struct string_list *protocol_whitelist(void) return enabled ? &allowed : NULL; } +enum protocol_allow_config { + PROTOCOL_ALLOW_NEVER = 0, + PROTOCOL_ALLOW_USER_ONLY, + PROTOCOL_ALLOW_ALWAYS +}; + +static enum protocol_allow_config parse_protocol_config(const char *key, + const char *value) +{ + if (!strcasecmp(value, "always")) + return PROTOCOL_ALLOW_ALWAYS; + else if (!strcasecmp(value, "never")) + return PROTOCOL_ALLOW_NEVER; + else if (!strcasecmp(value, "user")) + return PROTOCOL_ALLOW_USER_ONLY; + + die("unknown value for config '%s': %s", key, value); +} + +static enum protocol_allow_config get_protocol_config(const char *type) +{ + char *key = xstrfmt("protocol.%s.allow", type); + char *value; + + /* first check the per-protocol config */ + if (!git_config_get_string(key, &value)) { + enum protocol_allow_config ret = + parse_protocol_config(key, value); + free(key); + free(value); + return ret; + } + free(key); + + /* if defined, fallback to user-defined default for unknown protocols */ + if (!git_config_get_string("protocol.allow", &value)) { + enum protocol_allow_config ret = + parse_protocol_config("protocol.allow", value); + free(value); + return ret; + } + + /* fallback to built-in defaults */ + /* known safe */ + if (!strcmp(type, "http") || + !strcmp(type, "https") || + !strcmp(type, "git") || + !strcmp(type, "ssh") || + !strcmp(type, "file")) + return PROTOCOL_ALLOW_ALWAYS; + + /* known scary; err on the side of caution */ + if (!strcmp(type, "ext")) + return PROTOCOL_ALLOW_NEVER; + + /* unknown; by default let them be used only directly by the user */ + return PROTOCOL_ALLOW_USER_ONLY; +} + int is_transport_allowed(const char *type) { - const struct string_list *allowed = protocol_whitelist(); - return !allowed || string_list_has_string(allowed, type); + const struct string_list *whitelist = protocol_whitelist(); + if (whitelist) + return string_list_has_string(whitelist, type); + + switch (get_protocol_config(type)) { + case PROTOCOL_ALLOW_ALWAYS: + return 1; + case PROTOCOL_ALLOW_NEVER: + return 0; + case PROTOCOL_ALLOW_USER_ONLY: + return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + } + + die("BUG: invalid protocol_allow_config type"); } void transport_check_allowed(const char *type) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v5 2/2] transport: add protocol policy config option 2016-11-07 21:51 ` [PATCH v5 2/2] transport: add protocol policy config option Brandon Williams @ 2016-11-08 22:04 ` Jeff King 2016-11-08 22:05 ` Brandon Williams 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-11-08 22:04 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, jrnieder On Mon, Nov 07, 2016 at 01:51:02PM -0800, Brandon Williams wrote: > Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to > specify a whitelist of protocols to be used in clone/fetch/push > commands. This patch introduces new configuration options for more > fine-grained control for allowing/disallowing protocols. This also has > the added benefit of allowing easier construction of a protocol > whitelist on systems where setting an environment variable is > non-trivial. This v5 looks good to me (both patches 1 and 2). -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v5 2/2] transport: add protocol policy config option 2016-11-08 22:04 ` Jeff King @ 2016-11-08 22:05 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-11-08 22:05 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, jrnieder On 11/08, Jeff King wrote: > On Mon, Nov 07, 2016 at 01:51:02PM -0800, Brandon Williams wrote: > > > Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to > > specify a whitelist of protocols to be used in clone/fetch/push > > commands. This patch introduces new configuration options for more > > fine-grained control for allowing/disallowing protocols. This also has > > the added benefit of allowing easier construction of a protocol > > whitelist on systems where setting an environment variable is > > non-trivial. > > This v5 looks good to me (both patches 1 and 2). Thanks again for the help with this series! -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v6 0/4] transport protocol policy configuration 2016-11-07 21:51 ` [PATCH v5 " Brandon Williams 2016-11-07 21:51 ` [PATCH v5 2/2] transport: add protocol policy config option Brandon Williams @ 2016-12-01 19:44 ` Brandon Williams 2016-12-01 19:44 ` [PATCH v6 1/4] lib-proto-disable: variable name fix Brandon Williams ` (4 more replies) 1 sibling, 5 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-01 19:44 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder v6 introduces 2 additional patches which address problems with protocols that libcurl is allowed to use for redirection. Brandon Williams (4): lib-proto-disable: variable name fix transport: add protocol policy config option http: always warn if libcurl version is too old transport: check if protocol can be used on a redirect Documentation/config.txt | 46 +++++++++++++ Documentation/git.txt | 38 ++++------- git-submodule.sh | 12 ++-- http.c | 13 ++-- t/lib-proto-disable.sh | 142 ++++++++++++++++++++++++++++++++++++--- t/t5509-fetch-push-namespaces.sh | 1 + t/t5802-connect-helper.sh | 1 + transport.c | 82 +++++++++++++++++++--- transport.h | 13 ++-- 9 files changed, 281 insertions(+), 67 deletions(-) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v6 1/4] lib-proto-disable: variable name fix 2016-12-01 19:44 ` [PATCH v6 0/4] transport protocol policy configuration Brandon Williams @ 2016-12-01 19:44 ` Brandon Williams 2016-12-01 19:44 ` [PATCH v6 2/4] transport: add protocol policy config option Brandon Williams ` (3 subsequent siblings) 4 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-01 19:44 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder The test_proto function assigns the positional parameters to named variables, but then still refers to "$desc" as "$1". Using $desc is more readable and less error-prone. Signed-off-by: Brandon Williams <bmwill@google.com> --- t/lib-proto-disable.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index b0917d9..be88e9a 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -9,7 +9,7 @@ test_proto () { proto=$2 url=$3 - test_expect_success "clone $1 (enabled)" ' + test_expect_success "clone $desc (enabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=$proto && @@ -18,7 +18,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (enabled)" ' + test_expect_success "fetch $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -27,7 +27,7 @@ test_proto () { ) ' - test_expect_success "push $1 (enabled)" ' + test_expect_success "push $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -36,7 +36,7 @@ test_proto () { ) ' - test_expect_success "push $1 (disabled)" ' + test_expect_success "push $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -45,7 +45,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (disabled)" ' + test_expect_success "fetch $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -54,7 +54,7 @@ test_proto () { ) ' - test_expect_success "clone $1 (disabled)" ' + test_expect_success "clone $desc (disabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=none && -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v6 2/4] transport: add protocol policy config option 2016-12-01 19:44 ` [PATCH v6 0/4] transport protocol policy configuration Brandon Williams 2016-12-01 19:44 ` [PATCH v6 1/4] lib-proto-disable: variable name fix Brandon Williams @ 2016-12-01 19:44 ` Brandon Williams 2016-12-01 19:44 ` [PATCH v6 3/4] http: always warn if libcurl version is too old Brandon Williams ` (2 subsequent siblings) 4 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-01 19:44 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to specify a whitelist of protocols to be used in clone/fetch/push commands. This patch introduces new configuration options for more fine-grained control for allowing/disallowing protocols. This also has the added benefit of allowing easier construction of a protocol whitelist on systems where setting an environment variable is non-trivial. Now users can specify a policy to be used for each type of protocol via the 'protocol.<name>.allow' config option. A default policy for all unconfigured protocols can be set with the 'protocol.allow' config option. If no user configured default is made git will allow known-safe protocols (http, https, git, ssh, file), disallow known-dangerous protocols (ext), and have a default policy of `user` for all other protocols. The supported policies are `always`, `never`, and `user`. The `user` policy can be used to configure a protocol to be usable when explicitly used by a user, while disallowing it for commands which run clone/fetch/push commands without direct user intervention (e.g. recursive initialization of submodules). Commands which can potentially clone/fetch/push from untrusted repositories without user intervention can export `GIT_PROTOCOL_FROM_USER` with a value of '0' to prevent protocols configured to the `user` policy from being used. Fix remote-ext tests to use the new config to allow the ext protocol to be tested. Based on a patch by Jeff King <peff@peff.net> Signed-off-by: Brandon Williams <bmwill@google.com> --- Documentation/config.txt | 46 ++++++++++++++ Documentation/git.txt | 38 +++++------- git-submodule.sh | 12 ++-- t/lib-proto-disable.sh | 130 +++++++++++++++++++++++++++++++++++++-- t/t5509-fetch-push-namespaces.sh | 1 + t/t5802-connect-helper.sh | 1 + transport.c | 75 +++++++++++++++++++++- 7 files changed, 264 insertions(+), 39 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index 27069ac..5fe50bc 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -2308,6 +2308,52 @@ pretty.<name>:: Note that an alias with the same name as a built-in format will be silently ignored. +protocol.allow:: + If set, provide a user defined default policy for all protocols which + don't explicitly have a policy (`protocol.<name>.allow`). By default, + if unset, known-safe protocols (http, https, git, ssh, file) have a + default policy of `always`, known-dangerous protocols (ext) have a + default policy of `never`, and all other protocols have a default + policy of `user`. Supported policies: ++ +-- + +* `always` - protocol is always able to be used. + +* `never` - protocol is never able to be used. + +* `user` - protocol is only able to be used when `GIT_PROTOCOL_FROM_USER` is + either unset or has a value of 1. This policy should be used when you want a + protocol to be directly usable by the user but don't want it used by commands which + execute clone/fetch/push commands without user input, e.g. recursive + submodule initialization. + +-- + +protocol.<name>.allow:: + Set a policy to be used by protocol `<name>` with clone/fetch/push + commands. See `protocol.allow` above for the available policies. ++ +The protocol names currently used by git are: ++ +-- + - `file`: any local file-based path (including `file://` URLs, + or local paths) + + - `git`: the anonymous git protocol over a direct TCP + connection (or proxy, if configured) + + - `ssh`: git over ssh (including `host:path` syntax, + `ssh://`, etc). + + - `http`: git over http, both "smart http" and "dumb http". + Note that this does _not_ include `https`; if you want to configure + both, you must do so individually. + + - any external helpers are named by their protocol (e.g., use + `hg` to allow the `git-remote-hg` helper) +-- + pull.ff:: By default, Git does not create an extra merge commit when merging a commit that is a descendant of the current commit. Instead, the diff --git a/Documentation/git.txt b/Documentation/git.txt index ab7215e..c52cec8 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -1150,30 +1150,20 @@ of clones and fetches. cloning a repository to make a backup). `GIT_ALLOW_PROTOCOL`:: - If set, provide a colon-separated list of protocols which are - allowed to be used with fetch/push/clone. This is useful to - restrict recursive submodule initialization from an untrusted - repository. Any protocol not mentioned will be disallowed (i.e., - this is a whitelist, not a blacklist). If the variable is not - set at all, all protocols are enabled. The protocol names - currently used by git are: - - - `file`: any local file-based path (including `file://` URLs, - or local paths) - - - `git`: the anonymous git protocol over a direct TCP - connection (or proxy, if configured) - - - `ssh`: git over ssh (including `host:path` syntax, - `ssh://`, etc). - - - `http`: git over http, both "smart http" and "dumb http". - Note that this does _not_ include `https`; if you want both, - you should specify both as `http:https`. - - - any external helpers are named by their protocol (e.g., use - `hg` to allow the `git-remote-hg` helper) - + If set to a colon-separated list of protocols, behave as if + `protocol.allow` is set to `never`, and each of the listed + protocols has `protocol.<name>.allow` set to `always` + (overriding any existing configuration). In other words, any + protocol not mentioned will be disallowed (i.e., this is a + whitelist, not a blacklist). See the description of + `protocol.allow` in linkgit:git-config[1] for more details. + +`GIT_PROTOCOL_FROM_USER`:: + Set to 0 to prevent protocols used by fetch/push/clone which are + configured to the `user` state. This is useful to restrict recursive + submodule initialization from an untrusted repository or for programs + which feed potentially-untrusted URLS to git commands. See + linkgit:git-config[1] for more details. Discussion[[Discussion]] ------------------------ diff --git a/git-submodule.sh b/git-submodule.sh index a024a13..0a477b4 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -21,14 +21,10 @@ require_work_tree wt_prefix=$(git rev-parse --show-prefix) cd_to_toplevel -# Restrict ourselves to a vanilla subset of protocols; the URLs -# we get are under control of a remote repository, and we do not -# want them kicking off arbitrary git-remote-* programs. -# -# If the user has already specified a set of allowed protocols, -# we assume they know what they're doing and use that instead. -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} -export GIT_ALLOW_PROTOCOL +# Tell the rest of git that any URLs we get don't come +# directly from the user, so it can apply policy as appropriate. +GIT_PROTOCOL_FROM_USER=0 +export GIT_PROTOCOL_FROM_USER command= branch= diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index be88e9a..02f49cb 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -1,10 +1,7 @@ # Test routines for checking protocol disabling. -# test cloning a particular protocol -# $1 - description of the protocol -# $2 - machine-readable name of the protocol -# $3 - the URL to try cloning -test_proto () { +# Test clone/fetch/push with GIT_ALLOW_PROTOCOL whitelist +test_whitelist () { desc=$1 proto=$2 url=$3 @@ -62,6 +59,129 @@ test_proto () { test_must_fail git clone --bare "$url" tmp.git ) ' + + test_expect_success "clone $desc (env var has precedence)" ' + rm -rf tmp.git && + ( + GIT_ALLOW_PROTOCOL=none && + export GIT_ALLOW_PROTOCOL && + test_must_fail git -c protocol.allow=always clone --bare "$url" tmp.git && + test_must_fail git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ) + ' +} + +test_config () { + desc=$1 + proto=$2 + url=$3 + + # Test clone/fetch/push with protocol.<type>.allow config + test_expect_success "clone $desc (enabled with config)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git -c protocol.$proto.allow=never clone --bare "$url" tmp.git + ' + + # Test clone/fetch/push with protocol.user.allow and its env var + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user push origin HEAD:pushed + ) + ' + + test_expect_success "fetch $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user fetch + ) + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + ( + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ) + ' + + # Test clone/fetch/push with protocol.allow user defined default + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git config --global protocol.allow always && + git clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + git config --global protocol.allow never && + test_must_fail git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git clone --bare "$url" tmp.git + ' +} + +# test cloning a particular protocol +# $1 - description of the protocol +# $2 - machine-readable name of the protocol +# $3 - the URL to try cloning +test_proto () { + test_whitelist "$@" + + test_config "$@" } # set up an ssh wrapper that will access $host/$repo in the diff --git a/t/t5509-fetch-push-namespaces.sh b/t/t5509-fetch-push-namespaces.sh index bc44ac3..75c570a 100755 --- a/t/t5509-fetch-push-namespaces.sh +++ b/t/t5509-fetch-push-namespaces.sh @@ -4,6 +4,7 @@ test_description='fetch/push involving ref namespaces' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git init original && ( diff --git a/t/t5802-connect-helper.sh b/t/t5802-connect-helper.sh index b7a7f9d..c6c2661 100755 --- a/t/t5802-connect-helper.sh +++ b/t/t5802-connect-helper.sh @@ -4,6 +4,7 @@ test_description='ext::cmd remote "connect" helper' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git commit --allow-empty -m initial && test_tick && diff --git a/transport.c b/transport.c index d57e8de..2c0ec76 100644 --- a/transport.c +++ b/transport.c @@ -664,10 +664,81 @@ static const struct string_list *protocol_whitelist(void) return enabled ? &allowed : NULL; } +enum protocol_allow_config { + PROTOCOL_ALLOW_NEVER = 0, + PROTOCOL_ALLOW_USER_ONLY, + PROTOCOL_ALLOW_ALWAYS +}; + +static enum protocol_allow_config parse_protocol_config(const char *key, + const char *value) +{ + if (!strcasecmp(value, "always")) + return PROTOCOL_ALLOW_ALWAYS; + else if (!strcasecmp(value, "never")) + return PROTOCOL_ALLOW_NEVER; + else if (!strcasecmp(value, "user")) + return PROTOCOL_ALLOW_USER_ONLY; + + die("unknown value for config '%s': %s", key, value); +} + +static enum protocol_allow_config get_protocol_config(const char *type) +{ + char *key = xstrfmt("protocol.%s.allow", type); + char *value; + + /* first check the per-protocol config */ + if (!git_config_get_string(key, &value)) { + enum protocol_allow_config ret = + parse_protocol_config(key, value); + free(key); + free(value); + return ret; + } + free(key); + + /* if defined, fallback to user-defined default for unknown protocols */ + if (!git_config_get_string("protocol.allow", &value)) { + enum protocol_allow_config ret = + parse_protocol_config("protocol.allow", value); + free(value); + return ret; + } + + /* fallback to built-in defaults */ + /* known safe */ + if (!strcmp(type, "http") || + !strcmp(type, "https") || + !strcmp(type, "git") || + !strcmp(type, "ssh") || + !strcmp(type, "file")) + return PROTOCOL_ALLOW_ALWAYS; + + /* known scary; err on the side of caution */ + if (!strcmp(type, "ext")) + return PROTOCOL_ALLOW_NEVER; + + /* unknown; by default let them be used only directly by the user */ + return PROTOCOL_ALLOW_USER_ONLY; +} + int is_transport_allowed(const char *type) { - const struct string_list *allowed = protocol_whitelist(); - return !allowed || string_list_has_string(allowed, type); + const struct string_list *whitelist = protocol_whitelist(); + if (whitelist) + return string_list_has_string(whitelist, type); + + switch (get_protocol_config(type)) { + case PROTOCOL_ALLOW_ALWAYS: + return 1; + case PROTOCOL_ALLOW_NEVER: + return 0; + case PROTOCOL_ALLOW_USER_ONLY: + return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + } + + die("BUG: invalid protocol_allow_config type"); } void transport_check_allowed(const char *type) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v6 3/4] http: always warn if libcurl version is too old 2016-12-01 19:44 ` [PATCH v6 0/4] transport protocol policy configuration Brandon Williams 2016-12-01 19:44 ` [PATCH v6 1/4] lib-proto-disable: variable name fix Brandon Williams 2016-12-01 19:44 ` [PATCH v6 2/4] transport: add protocol policy config option Brandon Williams @ 2016-12-01 19:44 ` Brandon Williams 2016-12-01 19:44 ` [PATCH v6 4/4] transport: check if protocol can be used on a redirect Brandon Williams 2016-12-01 20:25 ` [PATCH v7 0/4] transport protocol policy configuration Brandon Williams 4 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-01 19:44 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder Now that there are default "known-good" and "known-bad" protocols which are allowed/disallowed by 'is_transport_allowed' we should always warn the user that older versions of libcurl can't respect the allowed protocols for redirects. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 5 ++--- transport.c | 5 ----- transport.h | 6 ------ 3 files changed, 2 insertions(+), 14 deletions(-) diff --git a/http.c b/http.c index 4c4a812..fee128b 100644 --- a/http.c +++ b/http.c @@ -735,9 +735,8 @@ static CURL *get_curl_handle(void) allowed_protocols |= CURLPROTO_FTPS; curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); #else - if (transport_restrict_protocols()) - warning("protocol restrictions not applied to curl redirects because\n" - "your curl version is too old (>= 7.19.4)"); + warning("protocol restrictions not applied to curl redirects because\n" + "your curl version is too old (>= 7.19.4)"); #endif if (getenv("GIT_CURL_VERBOSE")) curl_easy_setopt(result, CURLOPT_VERBOSE, 1L); diff --git a/transport.c b/transport.c index 2c0ec76..186de9a 100644 --- a/transport.c +++ b/transport.c @@ -747,11 +747,6 @@ void transport_check_allowed(const char *type) die("transport '%s' not allowed", type); } -int transport_restrict_protocols(void) -{ - return !!protocol_whitelist(); -} - struct transport *transport_get(struct remote *remote, const char *url) { const char *helper; diff --git a/transport.h b/transport.h index b8e4ee8..f4998bc 100644 --- a/transport.h +++ b/transport.h @@ -164,12 +164,6 @@ int is_transport_allowed(const char *type); */ void transport_check_allowed(const char *type); -/* - * Returns true if the user has attempted to turn on protocol - * restrictions at all. - */ -int transport_restrict_protocols(void); - /* Transport options which apply to git:// and scp-style URLs */ /* The program to use on the remote side to send a pack */ -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v6 4/4] transport: check if protocol can be used on a redirect 2016-12-01 19:44 ` [PATCH v6 0/4] transport protocol policy configuration Brandon Williams ` (2 preceding siblings ...) 2016-12-01 19:44 ` [PATCH v6 3/4] http: always warn if libcurl version is too old Brandon Williams @ 2016-12-01 19:44 ` Brandon Williams 2016-12-01 19:48 ` Brandon Williams 2016-12-01 19:50 ` Jeff King 2016-12-01 20:25 ` [PATCH v7 0/4] transport protocol policy configuration Brandon Williams 4 siblings, 2 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-01 19:44 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder Add a the 'redirect' parameter to 'is_transport_allowed' which allows callers to query if a transport protocol can be used on a redirect. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 8 ++++---- transport.c | 6 +++--- transport.h | 7 ++++--- 3 files changed, 11 insertions(+), 10 deletions(-) diff --git a/http.c b/http.c index fee128b..d31ded8 100644 --- a/http.c +++ b/http.c @@ -725,13 +725,13 @@ static CURL *get_curl_handle(void) curl_easy_setopt(result, CURLOPT_POST301, 1); #endif #if LIBCURL_VERSION_NUM >= 0x071304 - if (is_transport_allowed("http")) + if (is_transport_allowed("http", 1)) allowed_protocols |= CURLPROTO_HTTP; - if (is_transport_allowed("https")) + if (is_transport_allowed("https", 1)) allowed_protocols |= CURLPROTO_HTTPS; - if (is_transport_allowed("ftp")) + if (is_transport_allowed("ftp", 1)) allowed_protocols |= CURLPROTO_FTP; - if (is_transport_allowed("ftps")) + if (is_transport_allowed("ftps", 1)) allowed_protocols |= CURLPROTO_FTPS; curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); #else diff --git a/transport.c b/transport.c index 186de9a..9fee241 100644 --- a/transport.c +++ b/transport.c @@ -723,7 +723,7 @@ static enum protocol_allow_config get_protocol_config(const char *type) return PROTOCOL_ALLOW_USER_ONLY; } -int is_transport_allowed(const char *type) +int is_transport_allowed(const char *type, int redirect) { const struct string_list *whitelist = protocol_whitelist(); if (whitelist) @@ -735,7 +735,7 @@ int is_transport_allowed(const char *type) case PROTOCOL_ALLOW_NEVER: return 0; case PROTOCOL_ALLOW_USER_ONLY: - return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + return git_env_bool("GIT_PROTOCOL_FROM_USER", !redirect); } die("BUG: invalid protocol_allow_config type"); @@ -743,7 +743,7 @@ int is_transport_allowed(const char *type) void transport_check_allowed(const char *type) { - if (!is_transport_allowed(type)) + if (!is_transport_allowed(type, 0)) die("transport '%s' not allowed", type); } diff --git a/transport.h b/transport.h index f4998bc..72971ad 100644 --- a/transport.h +++ b/transport.h @@ -153,10 +153,11 @@ extern int transport_summary_width(const struct ref *refs); struct transport *transport_get(struct remote *, const char *); /* - * Check whether a transport is allowed by the environment. Type should - * generally be the URL scheme, as described in Documentation/git.txt + * Check whether a transport is allowed by the environment. Setting 'redirect' + * can be used to querry if the transport can be used in a redirect. Type + * should generally be the URL scheme, as described in Documentation/git.txt */ -int is_transport_allowed(const char *type); +int is_transport_allowed(const char *type, int redirect); /* * Check whether a transport is allowed by the environment, -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v6 4/4] transport: check if protocol can be used on a redirect 2016-12-01 19:44 ` [PATCH v6 4/4] transport: check if protocol can be used on a redirect Brandon Williams @ 2016-12-01 19:48 ` Brandon Williams 2016-12-01 19:49 ` Brandon Williams 2016-12-01 19:50 ` Jeff King 1 sibling, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-12-01 19:48 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder Add a the 'redirect' parameter to 'is_transport_allowed' which allows callers to query if a transport protocol can be used on a redirect. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 8 ++++---- transport.c | 6 +++--- transport.h | 7 ++++--- 3 files changed, 11 insertions(+), 10 deletions(-) diff --git a/http.c b/http.c index fee128b..d31ded8 100644 --- a/http.c +++ b/http.c @@ -725,13 +725,13 @@ static CURL *get_curl_handle(void) curl_easy_setopt(result, CURLOPT_POST301, 1); #endif #if LIBCURL_VERSION_NUM >= 0x071304 - if (is_transport_allowed("http")) + if (is_transport_allowed("http", 1)) allowed_protocols |= CURLPROTO_HTTP; - if (is_transport_allowed("https")) + if (is_transport_allowed("https", 1)) allowed_protocols |= CURLPROTO_HTTPS; - if (is_transport_allowed("ftp")) + if (is_transport_allowed("ftp", 1)) allowed_protocols |= CURLPROTO_FTP; - if (is_transport_allowed("ftps")) + if (is_transport_allowed("ftps", 1)) allowed_protocols |= CURLPROTO_FTPS; curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); #else diff --git a/transport.c b/transport.c index 186de9a..7c4a757 100644 --- a/transport.c +++ b/transport.c @@ -723,7 +723,7 @@ static enum protocol_allow_config get_protocol_config(const char *type) return PROTOCOL_ALLOW_USER_ONLY; } -int is_transport_allowed(const char *type) +int is_transport_allowed(const char *type, int redirect) { const struct string_list *whitelist = protocol_whitelist(); if (whitelist) @@ -735,7 +735,7 @@ int is_transport_allowed(const char *type) case PROTOCOL_ALLOW_NEVER: return 0; case PROTOCOL_ALLOW_USER_ONLY: - return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + return redirect ? 0 : git_env_bool("GIT_PROTOCOL_FROM_USER", 1); } die("BUG: invalid protocol_allow_config type"); @@ -743,7 +743,7 @@ int is_transport_allowed(const char *type) void transport_check_allowed(const char *type) { - if (!is_transport_allowed(type)) + if (!is_transport_allowed(type, 0)) die("transport '%s' not allowed", type); } diff --git a/transport.h b/transport.h index f4998bc..4bcf5d3 100644 --- a/transport.h +++ b/transport.h @@ -153,10 +153,11 @@ extern int transport_summary_width(const struct ref *refs); struct transport *transport_get(struct remote *, const char *); /* - * Check whether a transport is allowed by the environment. Type should - * generally be the URL scheme, as described in Documentation/git.txt + * Check whether a transport is allowed by the environment. Setting 'redirect' + * can be used to query if the transport can be used in a redirect. Type + * should generally be the URL scheme, as described in Documentation/git.txt */ -int is_transport_allowed(const char *type); +int is_transport_allowed(const char *type, int redirect); /* * Check whether a transport is allowed by the environment, -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v6 4/4] transport: check if protocol can be used on a redirect 2016-12-01 19:48 ` Brandon Williams @ 2016-12-01 19:49 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-01 19:49 UTC (permalink / raw) To: git; +Cc: peff, sbeller, bburky, jrnieder On 12/01, Brandon Williams wrote: > Add a the 'redirect' parameter to 'is_transport_allowed' which allows > callers to query if a transport protocol can be used on a redirect. > > Signed-off-by: Brandon Williams <bmwill@google.com> > --- > http.c | 8 ++++---- > transport.c | 6 +++--- > transport.h | 7 ++++--- > 3 files changed, 11 insertions(+), 10 deletions(-) > Accidentally sent out an old version of just this patch. Here is the updated one. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v6 4/4] transport: check if protocol can be used on a redirect 2016-12-01 19:44 ` [PATCH v6 4/4] transport: check if protocol can be used on a redirect Brandon Williams 2016-12-01 19:48 ` Brandon Williams @ 2016-12-01 19:50 ` Jeff King 2016-12-01 19:54 ` Junio C Hamano 1 sibling, 1 reply; 124+ messages in thread From: Jeff King @ 2016-12-01 19:50 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, jrnieder On Thu, Dec 01, 2016 at 11:44:07AM -0800, Brandon Williams wrote: > Add a the 'redirect' parameter to 'is_transport_allowed' which allows > callers to query if a transport protocol can be used on a redirect. s/a the/a/ > -int is_transport_allowed(const char *type) > +int is_transport_allowed(const char *type, int redirect) > { > const struct string_list *whitelist = protocol_whitelist(); > if (whitelist) > @@ -735,7 +735,7 @@ int is_transport_allowed(const char *type) > case PROTOCOL_ALLOW_NEVER: > return 0; > case PROTOCOL_ALLOW_USER_ONLY: > - return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); > + return git_env_bool("GIT_PROTOCOL_FROM_USER", !redirect); > } This has the older logic still. I'm not sure if we should call this "redirect" here. That's how it's used by the curl code, but I think from the perspective of the transport whitelist, it is really "are you overriding the from_user environment". Calling it "from_user" may be confusing though, as the default value would become "1", even though it means only "as far as I know this is from the user, but maybe the environment says otherwise". So bizarrely, I think calling it "not_from_user" is the clearest value. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v6 4/4] transport: check if protocol can be used on a redirect 2016-12-01 19:50 ` Jeff King @ 2016-12-01 19:54 ` Junio C Hamano 2016-12-01 19:59 ` Jeff King 0 siblings, 1 reply; 124+ messages in thread From: Junio C Hamano @ 2016-12-01 19:54 UTC (permalink / raw) To: Jeff King; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder Jeff King <peff@peff.net> writes: > On Thu, Dec 01, 2016 at 11:44:07AM -0800, Brandon Williams wrote: > >> Add a the 'redirect' parameter to 'is_transport_allowed' which allows >> callers to query if a transport protocol can be used on a redirect. > > s/a the/a/ > >> -int is_transport_allowed(const char *type) >> +int is_transport_allowed(const char *type, int redirect) >> { >> const struct string_list *whitelist = protocol_whitelist(); >> if (whitelist) >> @@ -735,7 +735,7 @@ int is_transport_allowed(const char *type) >> case PROTOCOL_ALLOW_NEVER: >> return 0; >> case PROTOCOL_ALLOW_USER_ONLY: >> - return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); >> + return git_env_bool("GIT_PROTOCOL_FROM_USER", !redirect); >> } > > This has the older logic still. > > I'm not sure if we should call this "redirect" here. That's how it's > used by the curl code, but I think from the perspective of the transport > whitelist, it is really "are you overriding the from_user environment". > > Calling it "from_user" may be confusing though, as the default value > would become "1", even though it means only "as far as I know this is > from the user, but maybe the environment says otherwise". So bizarrely, > I think calling it "not_from_user" is the clearest value. Bikeshedding: perhaps call it "unsafe" (in the sense that it is "not known to be safe")? ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v6 4/4] transport: check if protocol can be used on a redirect 2016-12-01 19:54 ` Junio C Hamano @ 2016-12-01 19:59 ` Jeff King 2016-12-01 20:01 ` Brandon Williams 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-12-01 19:59 UTC (permalink / raw) To: Junio C Hamano; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder On Thu, Dec 01, 2016 at 11:54:09AM -0800, Junio C Hamano wrote: > > I'm not sure if we should call this "redirect" here. That's how it's > > used by the curl code, but I think from the perspective of the transport > > whitelist, it is really "are you overriding the from_user environment". > > > > Calling it "from_user" may be confusing though, as the default value > > would become "1", even though it means only "as far as I know this is > > from the user, but maybe the environment says otherwise". So bizarrely, > > I think calling it "not_from_user" is the clearest value. > > Bikeshedding: perhaps call it "unsafe" (in the sense that it is "not > known to be safe")? That is definitely what we are going for, but it is vague about how it is unsafe. :) I think I may have converted Brandon in the other thread to my way of thinking of it as a tristate[1]. That lets us call it "from_user", and just do: case PROTOCOL_ALLOW_FROM_USER: if (from_user < 0) from_user = git_env_bool("GIT_PROTOCOL_FROM_USER", 1); return from_user; which is pretty clear. Nobody would ever pass "1" as from_user to the function, but it does the sensible thing if they do. -Peff [1] The original I posted calling it "redirect" was totally bogus because the logic between the two names is inverted. ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v6 4/4] transport: check if protocol can be used on a redirect 2016-12-01 19:59 ` Jeff King @ 2016-12-01 20:01 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-01 20:01 UTC (permalink / raw) To: Jeff King; +Cc: Junio C Hamano, git, sbeller, bburky, jrnieder On 12/01, Jeff King wrote: > On Thu, Dec 01, 2016 at 11:54:09AM -0800, Junio C Hamano wrote: > > > > I'm not sure if we should call this "redirect" here. That's how it's > > > used by the curl code, but I think from the perspective of the transport > > > whitelist, it is really "are you overriding the from_user environment". > > > > > > Calling it "from_user" may be confusing though, as the default value > > > would become "1", even though it means only "as far as I know this is > > > from the user, but maybe the environment says otherwise". So bizarrely, > > > I think calling it "not_from_user" is the clearest value. > > > > Bikeshedding: perhaps call it "unsafe" (in the sense that it is "not > > known to be safe")? > > That is definitely what we are going for, but it is vague about how it > is unsafe. :) > > I think I may have converted Brandon in the other thread to my way of Yep, I've been converted :D If we agree on that then I can make the change and resend the patch. > thinking of it as a tristate[1]. That lets us call it "from_user", and > just do: > > case PROTOCOL_ALLOW_FROM_USER: > if (from_user < 0) > from_user = git_env_bool("GIT_PROTOCOL_FROM_USER", 1); > return from_user; > > which is pretty clear. Nobody would ever pass "1" as from_user to the > function, but it does the sensible thing if they do. > > -Peff > > [1] The original I posted calling it "redirect" was totally bogus > because the logic between the two names is inverted. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v7 0/4] transport protocol policy configuration 2016-12-01 19:44 ` [PATCH v6 0/4] transport protocol policy configuration Brandon Williams ` (3 preceding siblings ...) 2016-12-01 19:44 ` [PATCH v6 4/4] transport: check if protocol can be used on a redirect Brandon Williams @ 2016-12-01 20:25 ` Brandon Williams 2016-12-01 20:25 ` [PATCH v7 1/4] lib-proto-disable: variable name fix Brandon Williams ` (4 more replies) 4 siblings, 5 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-01 20:25 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder Changed the last patch in the series to use the parameter 'from_user' instead of 'redirect'. This allows us to use the same logic polarity and maintain use of the same vocabulary. Brandon Williams (4): lib-proto-disable: variable name fix transport: add protocol policy config option http: always warn if libcurl version is too old transport: add from_user parameter to is_transport_allowed Documentation/config.txt | 46 +++++++++++++ Documentation/git.txt | 38 ++++------- git-submodule.sh | 12 ++-- http.c | 13 ++-- t/lib-proto-disable.sh | 142 ++++++++++++++++++++++++++++++++++++--- t/t5509-fetch-push-namespaces.sh | 1 + t/t5802-connect-helper.sh | 1 + transport.c | 84 ++++++++++++++++++++--- transport.h | 19 +++--- 9 files changed, 289 insertions(+), 67 deletions(-) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v7 1/4] lib-proto-disable: variable name fix 2016-12-01 20:25 ` [PATCH v7 0/4] transport protocol policy configuration Brandon Williams @ 2016-12-01 20:25 ` Brandon Williams 2016-12-01 20:25 ` [PATCH v7 2/4] transport: add protocol policy config option Brandon Williams ` (3 subsequent siblings) 4 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-01 20:25 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder The test_proto function assigns the positional parameters to named variables, but then still refers to "$desc" as "$1". Using $desc is more readable and less error-prone. Signed-off-by: Brandon Williams <bmwill@google.com> --- t/lib-proto-disable.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index b0917d9..be88e9a 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -9,7 +9,7 @@ test_proto () { proto=$2 url=$3 - test_expect_success "clone $1 (enabled)" ' + test_expect_success "clone $desc (enabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=$proto && @@ -18,7 +18,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (enabled)" ' + test_expect_success "fetch $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -27,7 +27,7 @@ test_proto () { ) ' - test_expect_success "push $1 (enabled)" ' + test_expect_success "push $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -36,7 +36,7 @@ test_proto () { ) ' - test_expect_success "push $1 (disabled)" ' + test_expect_success "push $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -45,7 +45,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (disabled)" ' + test_expect_success "fetch $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -54,7 +54,7 @@ test_proto () { ) ' - test_expect_success "clone $1 (disabled)" ' + test_expect_success "clone $desc (disabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=none && -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v7 2/4] transport: add protocol policy config option 2016-12-01 20:25 ` [PATCH v7 0/4] transport protocol policy configuration Brandon Williams 2016-12-01 20:25 ` [PATCH v7 1/4] lib-proto-disable: variable name fix Brandon Williams @ 2016-12-01 20:25 ` Brandon Williams 2016-12-01 20:25 ` [PATCH v7 3/4] http: always warn if libcurl version is too old Brandon Williams ` (2 subsequent siblings) 4 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-01 20:25 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to specify a whitelist of protocols to be used in clone/fetch/push commands. This patch introduces new configuration options for more fine-grained control for allowing/disallowing protocols. This also has the added benefit of allowing easier construction of a protocol whitelist on systems where setting an environment variable is non-trivial. Now users can specify a policy to be used for each type of protocol via the 'protocol.<name>.allow' config option. A default policy for all unconfigured protocols can be set with the 'protocol.allow' config option. If no user configured default is made git will allow known-safe protocols (http, https, git, ssh, file), disallow known-dangerous protocols (ext), and have a default policy of `user` for all other protocols. The supported policies are `always`, `never`, and `user`. The `user` policy can be used to configure a protocol to be usable when explicitly used by a user, while disallowing it for commands which run clone/fetch/push commands without direct user intervention (e.g. recursive initialization of submodules). Commands which can potentially clone/fetch/push from untrusted repositories without user intervention can export `GIT_PROTOCOL_FROM_USER` with a value of '0' to prevent protocols configured to the `user` policy from being used. Fix remote-ext tests to use the new config to allow the ext protocol to be tested. Based on a patch by Jeff King <peff@peff.net> Signed-off-by: Brandon Williams <bmwill@google.com> --- Documentation/config.txt | 46 ++++++++++++++ Documentation/git.txt | 38 +++++------- git-submodule.sh | 12 ++-- t/lib-proto-disable.sh | 130 +++++++++++++++++++++++++++++++++++++-- t/t5509-fetch-push-namespaces.sh | 1 + t/t5802-connect-helper.sh | 1 + transport.c | 75 +++++++++++++++++++++- 7 files changed, 264 insertions(+), 39 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index 27069ac..5fe50bc 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -2308,6 +2308,52 @@ pretty.<name>:: Note that an alias with the same name as a built-in format will be silently ignored. +protocol.allow:: + If set, provide a user defined default policy for all protocols which + don't explicitly have a policy (`protocol.<name>.allow`). By default, + if unset, known-safe protocols (http, https, git, ssh, file) have a + default policy of `always`, known-dangerous protocols (ext) have a + default policy of `never`, and all other protocols have a default + policy of `user`. Supported policies: ++ +-- + +* `always` - protocol is always able to be used. + +* `never` - protocol is never able to be used. + +* `user` - protocol is only able to be used when `GIT_PROTOCOL_FROM_USER` is + either unset or has a value of 1. This policy should be used when you want a + protocol to be directly usable by the user but don't want it used by commands which + execute clone/fetch/push commands without user input, e.g. recursive + submodule initialization. + +-- + +protocol.<name>.allow:: + Set a policy to be used by protocol `<name>` with clone/fetch/push + commands. See `protocol.allow` above for the available policies. ++ +The protocol names currently used by git are: ++ +-- + - `file`: any local file-based path (including `file://` URLs, + or local paths) + + - `git`: the anonymous git protocol over a direct TCP + connection (or proxy, if configured) + + - `ssh`: git over ssh (including `host:path` syntax, + `ssh://`, etc). + + - `http`: git over http, both "smart http" and "dumb http". + Note that this does _not_ include `https`; if you want to configure + both, you must do so individually. + + - any external helpers are named by their protocol (e.g., use + `hg` to allow the `git-remote-hg` helper) +-- + pull.ff:: By default, Git does not create an extra merge commit when merging a commit that is a descendant of the current commit. Instead, the diff --git a/Documentation/git.txt b/Documentation/git.txt index ab7215e..c52cec8 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -1150,30 +1150,20 @@ of clones and fetches. cloning a repository to make a backup). `GIT_ALLOW_PROTOCOL`:: - If set, provide a colon-separated list of protocols which are - allowed to be used with fetch/push/clone. This is useful to - restrict recursive submodule initialization from an untrusted - repository. Any protocol not mentioned will be disallowed (i.e., - this is a whitelist, not a blacklist). If the variable is not - set at all, all protocols are enabled. The protocol names - currently used by git are: - - - `file`: any local file-based path (including `file://` URLs, - or local paths) - - - `git`: the anonymous git protocol over a direct TCP - connection (or proxy, if configured) - - - `ssh`: git over ssh (including `host:path` syntax, - `ssh://`, etc). - - - `http`: git over http, both "smart http" and "dumb http". - Note that this does _not_ include `https`; if you want both, - you should specify both as `http:https`. - - - any external helpers are named by their protocol (e.g., use - `hg` to allow the `git-remote-hg` helper) - + If set to a colon-separated list of protocols, behave as if + `protocol.allow` is set to `never`, and each of the listed + protocols has `protocol.<name>.allow` set to `always` + (overriding any existing configuration). In other words, any + protocol not mentioned will be disallowed (i.e., this is a + whitelist, not a blacklist). See the description of + `protocol.allow` in linkgit:git-config[1] for more details. + +`GIT_PROTOCOL_FROM_USER`:: + Set to 0 to prevent protocols used by fetch/push/clone which are + configured to the `user` state. This is useful to restrict recursive + submodule initialization from an untrusted repository or for programs + which feed potentially-untrusted URLS to git commands. See + linkgit:git-config[1] for more details. Discussion[[Discussion]] ------------------------ diff --git a/git-submodule.sh b/git-submodule.sh index a024a13..0a477b4 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -21,14 +21,10 @@ require_work_tree wt_prefix=$(git rev-parse --show-prefix) cd_to_toplevel -# Restrict ourselves to a vanilla subset of protocols; the URLs -# we get are under control of a remote repository, and we do not -# want them kicking off arbitrary git-remote-* programs. -# -# If the user has already specified a set of allowed protocols, -# we assume they know what they're doing and use that instead. -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} -export GIT_ALLOW_PROTOCOL +# Tell the rest of git that any URLs we get don't come +# directly from the user, so it can apply policy as appropriate. +GIT_PROTOCOL_FROM_USER=0 +export GIT_PROTOCOL_FROM_USER command= branch= diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index be88e9a..02f49cb 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -1,10 +1,7 @@ # Test routines for checking protocol disabling. -# test cloning a particular protocol -# $1 - description of the protocol -# $2 - machine-readable name of the protocol -# $3 - the URL to try cloning -test_proto () { +# Test clone/fetch/push with GIT_ALLOW_PROTOCOL whitelist +test_whitelist () { desc=$1 proto=$2 url=$3 @@ -62,6 +59,129 @@ test_proto () { test_must_fail git clone --bare "$url" tmp.git ) ' + + test_expect_success "clone $desc (env var has precedence)" ' + rm -rf tmp.git && + ( + GIT_ALLOW_PROTOCOL=none && + export GIT_ALLOW_PROTOCOL && + test_must_fail git -c protocol.allow=always clone --bare "$url" tmp.git && + test_must_fail git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ) + ' +} + +test_config () { + desc=$1 + proto=$2 + url=$3 + + # Test clone/fetch/push with protocol.<type>.allow config + test_expect_success "clone $desc (enabled with config)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git -c protocol.$proto.allow=never clone --bare "$url" tmp.git + ' + + # Test clone/fetch/push with protocol.user.allow and its env var + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user push origin HEAD:pushed + ) + ' + + test_expect_success "fetch $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user fetch + ) + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + ( + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ) + ' + + # Test clone/fetch/push with protocol.allow user defined default + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git config --global protocol.allow always && + git clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + git config --global protocol.allow never && + test_must_fail git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git clone --bare "$url" tmp.git + ' +} + +# test cloning a particular protocol +# $1 - description of the protocol +# $2 - machine-readable name of the protocol +# $3 - the URL to try cloning +test_proto () { + test_whitelist "$@" + + test_config "$@" } # set up an ssh wrapper that will access $host/$repo in the diff --git a/t/t5509-fetch-push-namespaces.sh b/t/t5509-fetch-push-namespaces.sh index bc44ac3..75c570a 100755 --- a/t/t5509-fetch-push-namespaces.sh +++ b/t/t5509-fetch-push-namespaces.sh @@ -4,6 +4,7 @@ test_description='fetch/push involving ref namespaces' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git init original && ( diff --git a/t/t5802-connect-helper.sh b/t/t5802-connect-helper.sh index b7a7f9d..c6c2661 100755 --- a/t/t5802-connect-helper.sh +++ b/t/t5802-connect-helper.sh @@ -4,6 +4,7 @@ test_description='ext::cmd remote "connect" helper' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git commit --allow-empty -m initial && test_tick && diff --git a/transport.c b/transport.c index d57e8de..2c0ec76 100644 --- a/transport.c +++ b/transport.c @@ -664,10 +664,81 @@ static const struct string_list *protocol_whitelist(void) return enabled ? &allowed : NULL; } +enum protocol_allow_config { + PROTOCOL_ALLOW_NEVER = 0, + PROTOCOL_ALLOW_USER_ONLY, + PROTOCOL_ALLOW_ALWAYS +}; + +static enum protocol_allow_config parse_protocol_config(const char *key, + const char *value) +{ + if (!strcasecmp(value, "always")) + return PROTOCOL_ALLOW_ALWAYS; + else if (!strcasecmp(value, "never")) + return PROTOCOL_ALLOW_NEVER; + else if (!strcasecmp(value, "user")) + return PROTOCOL_ALLOW_USER_ONLY; + + die("unknown value for config '%s': %s", key, value); +} + +static enum protocol_allow_config get_protocol_config(const char *type) +{ + char *key = xstrfmt("protocol.%s.allow", type); + char *value; + + /* first check the per-protocol config */ + if (!git_config_get_string(key, &value)) { + enum protocol_allow_config ret = + parse_protocol_config(key, value); + free(key); + free(value); + return ret; + } + free(key); + + /* if defined, fallback to user-defined default for unknown protocols */ + if (!git_config_get_string("protocol.allow", &value)) { + enum protocol_allow_config ret = + parse_protocol_config("protocol.allow", value); + free(value); + return ret; + } + + /* fallback to built-in defaults */ + /* known safe */ + if (!strcmp(type, "http") || + !strcmp(type, "https") || + !strcmp(type, "git") || + !strcmp(type, "ssh") || + !strcmp(type, "file")) + return PROTOCOL_ALLOW_ALWAYS; + + /* known scary; err on the side of caution */ + if (!strcmp(type, "ext")) + return PROTOCOL_ALLOW_NEVER; + + /* unknown; by default let them be used only directly by the user */ + return PROTOCOL_ALLOW_USER_ONLY; +} + int is_transport_allowed(const char *type) { - const struct string_list *allowed = protocol_whitelist(); - return !allowed || string_list_has_string(allowed, type); + const struct string_list *whitelist = protocol_whitelist(); + if (whitelist) + return string_list_has_string(whitelist, type); + + switch (get_protocol_config(type)) { + case PROTOCOL_ALLOW_ALWAYS: + return 1; + case PROTOCOL_ALLOW_NEVER: + return 0; + case PROTOCOL_ALLOW_USER_ONLY: + return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + } + + die("BUG: invalid protocol_allow_config type"); } void transport_check_allowed(const char *type) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v7 3/4] http: always warn if libcurl version is too old 2016-12-01 20:25 ` [PATCH v7 0/4] transport protocol policy configuration Brandon Williams 2016-12-01 20:25 ` [PATCH v7 1/4] lib-proto-disable: variable name fix Brandon Williams 2016-12-01 20:25 ` [PATCH v7 2/4] transport: add protocol policy config option Brandon Williams @ 2016-12-01 20:25 ` Brandon Williams 2016-12-01 20:25 ` [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed Brandon Williams 2016-12-02 0:00 ` [PATCH v8 0/5] transport protocol policy configuration Brandon Williams 4 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-01 20:25 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder Now that there are default "known-good" and "known-bad" protocols which are allowed/disallowed by 'is_transport_allowed' we should always warn the user that older versions of libcurl can't respect the allowed protocols for redirects. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 5 ++--- transport.c | 5 ----- transport.h | 6 ------ 3 files changed, 2 insertions(+), 14 deletions(-) diff --git a/http.c b/http.c index 4c4a812..fee128b 100644 --- a/http.c +++ b/http.c @@ -735,9 +735,8 @@ static CURL *get_curl_handle(void) allowed_protocols |= CURLPROTO_FTPS; curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); #else - if (transport_restrict_protocols()) - warning("protocol restrictions not applied to curl redirects because\n" - "your curl version is too old (>= 7.19.4)"); + warning("protocol restrictions not applied to curl redirects because\n" + "your curl version is too old (>= 7.19.4)"); #endif if (getenv("GIT_CURL_VERBOSE")) curl_easy_setopt(result, CURLOPT_VERBOSE, 1L); diff --git a/transport.c b/transport.c index 2c0ec76..186de9a 100644 --- a/transport.c +++ b/transport.c @@ -747,11 +747,6 @@ void transport_check_allowed(const char *type) die("transport '%s' not allowed", type); } -int transport_restrict_protocols(void) -{ - return !!protocol_whitelist(); -} - struct transport *transport_get(struct remote *remote, const char *url) { const char *helper; diff --git a/transport.h b/transport.h index b8e4ee8..f4998bc 100644 --- a/transport.h +++ b/transport.h @@ -164,12 +164,6 @@ int is_transport_allowed(const char *type); */ void transport_check_allowed(const char *type); -/* - * Returns true if the user has attempted to turn on protocol - * restrictions at all. - */ -int transport_restrict_protocols(void); - /* Transport options which apply to git:// and scp-style URLs */ /* The program to use on the remote side to send a pack */ -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-01 20:25 ` [PATCH v7 0/4] transport protocol policy configuration Brandon Williams ` (2 preceding siblings ...) 2016-12-01 20:25 ` [PATCH v7 3/4] http: always warn if libcurl version is too old Brandon Williams @ 2016-12-01 20:25 ` Brandon Williams 2016-12-01 21:40 ` Jeff King 2016-12-02 0:00 ` [PATCH v8 0/5] transport protocol policy configuration Brandon Williams 4 siblings, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-12-01 20:25 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder Add the from_user parameter to the 'is_transport_allowed' function. This allows callers to query if a transport protocol is allowed, given that the caller knows that the protocol is coming from the user (1) or not from the user (0), such as redirects in libcurl. If unknown, a -1 should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER` to determine if the protocol came from the user. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 8 ++++---- transport.c | 8 +++++--- transport.h | 13 ++++++++++--- 3 files changed, 19 insertions(+), 10 deletions(-) diff --git a/http.c b/http.c index fee128b..e74c0f0 100644 --- a/http.c +++ b/http.c @@ -725,13 +725,13 @@ static CURL *get_curl_handle(void) curl_easy_setopt(result, CURLOPT_POST301, 1); #endif #if LIBCURL_VERSION_NUM >= 0x071304 - if (is_transport_allowed("http")) + if (is_transport_allowed("http", 0)) allowed_protocols |= CURLPROTO_HTTP; - if (is_transport_allowed("https")) + if (is_transport_allowed("https", 0)) allowed_protocols |= CURLPROTO_HTTPS; - if (is_transport_allowed("ftp")) + if (is_transport_allowed("ftp", 0)) allowed_protocols |= CURLPROTO_FTP; - if (is_transport_allowed("ftps")) + if (is_transport_allowed("ftps", 0)) allowed_protocols |= CURLPROTO_FTPS; curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); #else diff --git a/transport.c b/transport.c index 186de9a..8a3597b 100644 --- a/transport.c +++ b/transport.c @@ -723,7 +723,7 @@ static enum protocol_allow_config get_protocol_config(const char *type) return PROTOCOL_ALLOW_USER_ONLY; } -int is_transport_allowed(const char *type) +int is_transport_allowed(const char *type, int from_user) { const struct string_list *whitelist = protocol_whitelist(); if (whitelist) @@ -735,7 +735,9 @@ int is_transport_allowed(const char *type) case PROTOCOL_ALLOW_NEVER: return 0; case PROTOCOL_ALLOW_USER_ONLY: - return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + if (from_user < 0) + from_user = git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + return from_user; } die("BUG: invalid protocol_allow_config type"); @@ -743,7 +745,7 @@ int is_transport_allowed(const char *type) void transport_check_allowed(const char *type) { - if (!is_transport_allowed(type)) + if (!is_transport_allowed(type, -1)) die("transport '%s' not allowed", type); } diff --git a/transport.h b/transport.h index f4998bc..9820f10 100644 --- a/transport.h +++ b/transport.h @@ -153,10 +153,17 @@ extern int transport_summary_width(const struct ref *refs); struct transport *transport_get(struct remote *, const char *); /* - * Check whether a transport is allowed by the environment. Type should - * generally be the URL scheme, as described in Documentation/git.txt + * Check whether a transport is allowed by the environment. + * + * Type should generally be the URL scheme, as described in + * Documentation/git.txt + * + * from_user specifies if the transport was given by the user. If unknown pass + * a -1 to read from the environment to determine if the transport was given by + * the user. + * */ -int is_transport_allowed(const char *type); +int is_transport_allowed(const char *type, int from_user); /* * Check whether a transport is allowed by the environment, -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-01 20:25 ` [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed Brandon Williams @ 2016-12-01 21:40 ` Jeff King 2016-12-01 22:25 ` Junio C Hamano 2016-12-01 23:07 ` Brandon Williams 0 siblings, 2 replies; 124+ messages in thread From: Jeff King @ 2016-12-01 21:40 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, jrnieder On Thu, Dec 01, 2016 at 12:25:59PM -0800, Brandon Williams wrote: > Add the from_user parameter to the 'is_transport_allowed' function. > This allows callers to query if a transport protocol is allowed, given > that the caller knows that the protocol is coming from the user (1) or > not from the user (0), such as redirects in libcurl. If unknown, a -1 > should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER` > to determine if the protocol came from the user. Patches 3 and 4 look good to me (1 and 2 are unchanged, right? They are already in 'next' anyway, though I guess we are due for a post-release reset of 'next'). > diff --git a/http.c b/http.c > index fee128b..e74c0f0 100644 > --- a/http.c > +++ b/http.c > @@ -725,13 +725,13 @@ static CURL *get_curl_handle(void) > curl_easy_setopt(result, CURLOPT_POST301, 1); > #endif > #if LIBCURL_VERSION_NUM >= 0x071304 > - if (is_transport_allowed("http")) > + if (is_transport_allowed("http", 0)) > allowed_protocols |= CURLPROTO_HTTP; > - if (is_transport_allowed("https")) > + if (is_transport_allowed("https", 0)) > allowed_protocols |= CURLPROTO_HTTPS; > - if (is_transport_allowed("ftp")) > + if (is_transport_allowed("ftp", 0)) > allowed_protocols |= CURLPROTO_FTP; > - if (is_transport_allowed("ftps")) > + if (is_transport_allowed("ftps", 0)) > allowed_protocols |= CURLPROTO_FTPS; > curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); > #else This is better, but I think we still need to deal with http-alternates on top. I think we'd need to move this allowed_protocols setup into a function like: int generate_allowed_protocols(int from_user) { int ret; if (is_transport_allowed("http", from_user)) ret |= CURLPROTO_HTTP; ... etc ... return ret; } and then create a protocol list for each situation: allowed_protocols = generate_allowed_protocols(-1); allowed_redir_protocols = generate_allowed_protocols(0); and then we know we can always set up the redir protocols: curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_redir_protocols); and which we feed for CURLOPT_PROTOCOLS depends on whether we are following an http-alternates redirect or not. But I suspect it will be a nasty change to plumb through the idea of "this request is on behalf of an http-alternates redirect". Given how few people probably care, I'm tempted to document it as a quirk and direct people to the upcoming http.followRedirects. The newly proposed default value of that disables http-alternates entirely anyway. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-01 21:40 ` Jeff King @ 2016-12-01 22:25 ` Junio C Hamano 2016-12-01 23:07 ` Brandon Williams 1 sibling, 0 replies; 124+ messages in thread From: Junio C Hamano @ 2016-12-01 22:25 UTC (permalink / raw) To: Jeff King; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder Jeff King <peff@peff.net> writes: > On Thu, Dec 01, 2016 at 12:25:59PM -0800, Brandon Williams wrote: > >> Add the from_user parameter to the 'is_transport_allowed' function. >> This allows callers to query if a transport protocol is allowed, given >> that the caller knows that the protocol is coming from the user (1) or >> not from the user (0), such as redirects in libcurl. If unknown, a -1 >> should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER` >> to determine if the protocol came from the user. > > Patches 3 and 4 look good to me (1 and 2 are unchanged, right? They are > already in 'next' anyway, though I guess we are due for a post-release > reset of 'next'). Yes. I am planning to take a day off tomorrow, and probably will rewind 'next' sometime this the weekend. I agree with the comments you made in the remainder of the message I am responding to, so I'll snip it. >> diff --git a/http.c b/http.c >> index fee128b..e74c0f0 100644 >> --- a/http.c >> +++ b/http.c >> ... > > This is better, but I think we still need to deal with http-alternates > on top. > ... ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-01 21:40 ` Jeff King 2016-12-01 22:25 ` Junio C Hamano @ 2016-12-01 23:07 ` Brandon Williams 2016-12-01 23:26 ` Brandon Williams 2016-12-01 23:34 ` Junio C Hamano 1 sibling, 2 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-01 23:07 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, jrnieder On 12/01, Jeff King wrote: > On Thu, Dec 01, 2016 at 12:25:59PM -0800, Brandon Williams wrote: > > > Add the from_user parameter to the 'is_transport_allowed' function. > > This allows callers to query if a transport protocol is allowed, given > > that the caller knows that the protocol is coming from the user (1) or > > not from the user (0), such as redirects in libcurl. If unknown, a -1 > > should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER` > > to determine if the protocol came from the user. > > Patches 3 and 4 look good to me (1 and 2 are unchanged, right? They are > already in 'next' anyway, though I guess we are due for a post-release > reset of 'next'). > > > diff --git a/http.c b/http.c > > index fee128b..e74c0f0 100644 > > --- a/http.c > > +++ b/http.c > > @@ -725,13 +725,13 @@ static CURL *get_curl_handle(void) > > curl_easy_setopt(result, CURLOPT_POST301, 1); > > #endif > > #if LIBCURL_VERSION_NUM >= 0x071304 > > - if (is_transport_allowed("http")) > > + if (is_transport_allowed("http", 0)) > > allowed_protocols |= CURLPROTO_HTTP; > > - if (is_transport_allowed("https")) > > + if (is_transport_allowed("https", 0)) > > allowed_protocols |= CURLPROTO_HTTPS; > > - if (is_transport_allowed("ftp")) > > + if (is_transport_allowed("ftp", 0)) > > allowed_protocols |= CURLPROTO_FTP; > > - if (is_transport_allowed("ftps")) > > + if (is_transport_allowed("ftps", 0)) > > allowed_protocols |= CURLPROTO_FTPS; > > curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); > > #else > > This is better, but I think we still need to deal with http-alternates > on top. > > I think we'd need to move this allowed_protocols setup into a function > like: > > int generate_allowed_protocols(int from_user) > { > int ret; > if (is_transport_allowed("http", from_user)) > ret |= CURLPROTO_HTTP; > ... etc ... > return ret; > } > > and then create a protocol list for each situation: > > allowed_protocols = generate_allowed_protocols(-1); > allowed_redir_protocols = generate_allowed_protocols(0); > > and then we know we can always set up the redir protocols: > > curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_redir_protocols); > > and which we feed for CURLOPT_PROTOCOLS depends on whether we are > following an http-alternates redirect or not. But I suspect it will be a > nasty change to plumb through the idea of "this request is on behalf of > an http-alternates redirect". > > Given how few people probably care, I'm tempted to document it as a > quirk and direct people to the upcoming http.followRedirects. The newly > proposed default value of that disables http-alternates entirely anyway. > > -Peff I started taking a look at your http redirect series (I really should have taking a look at it sooner) and I see exactly what you're talking about. We can easily move this logic into a function to make it easier to generate the two whitelists. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-01 23:07 ` Brandon Williams @ 2016-12-01 23:26 ` Brandon Williams 2016-12-02 0:13 ` Jeff King 2016-12-01 23:34 ` Junio C Hamano 1 sibling, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-12-01 23:26 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, jrnieder On 12/01, Brandon Williams wrote: > On 12/01, Jeff King wrote: > > On Thu, Dec 01, 2016 at 12:25:59PM -0800, Brandon Williams wrote: > > > > > Add the from_user parameter to the 'is_transport_allowed' function. > > > This allows callers to query if a transport protocol is allowed, given > > > that the caller knows that the protocol is coming from the user (1) or > > > not from the user (0), such as redirects in libcurl. If unknown, a -1 > > > should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER` > > > to determine if the protocol came from the user. > > > > Patches 3 and 4 look good to me (1 and 2 are unchanged, right? They are > > already in 'next' anyway, though I guess we are due for a post-release > > reset of 'next'). > > > > > diff --git a/http.c b/http.c > > > index fee128b..e74c0f0 100644 > > > --- a/http.c > > > +++ b/http.c > > > @@ -725,13 +725,13 @@ static CURL *get_curl_handle(void) > > > curl_easy_setopt(result, CURLOPT_POST301, 1); > > > #endif > > > #if LIBCURL_VERSION_NUM >= 0x071304 > > > - if (is_transport_allowed("http")) > > > + if (is_transport_allowed("http", 0)) > > > allowed_protocols |= CURLPROTO_HTTP; > > > - if (is_transport_allowed("https")) > > > + if (is_transport_allowed("https", 0)) > > > allowed_protocols |= CURLPROTO_HTTPS; > > > - if (is_transport_allowed("ftp")) > > > + if (is_transport_allowed("ftp", 0)) > > > allowed_protocols |= CURLPROTO_FTP; > > > - if (is_transport_allowed("ftps")) > > > + if (is_transport_allowed("ftps", 0)) > > > allowed_protocols |= CURLPROTO_FTPS; > > > curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); > > > #else > > > > This is better, but I think we still need to deal with http-alternates > > on top. > > > > I think we'd need to move this allowed_protocols setup into a function > > like: > > > > int generate_allowed_protocols(int from_user) > > { > > int ret; > > if (is_transport_allowed("http", from_user)) > > ret |= CURLPROTO_HTTP; > > ... etc ... > > return ret; > > } > > > > and then create a protocol list for each situation: > > > > allowed_protocols = generate_allowed_protocols(-1); > > allowed_redir_protocols = generate_allowed_protocols(0); > > > > and then we know we can always set up the redir protocols: > > > > curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_redir_protocols); > > > > and which we feed for CURLOPT_PROTOCOLS depends on whether we are > > following an http-alternates redirect or not. But I suspect it will be a > > nasty change to plumb through the idea of "this request is on behalf of > > an http-alternates redirect". > > > > Given how few people probably care, I'm tempted to document it as a > > quirk and direct people to the upcoming http.followRedirects. The newly > > proposed default value of that disables http-alternates entirely anyway. > > > > -Peff > > I started taking a look at your http redirect series (I really should > have taking a look at it sooner) and I see exactly what you're talking > about. We can easily move this logic into a function to make it easier > to generate the two whitelists. Thinking about this some more...I was told that having http redirect to file:// could be scary. The way the new protocol configuration is setup we have file:// as a default known-safe protocol. Do we need to worry about this or can we leave this be since this can be overridden by the user? -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-01 23:26 ` Brandon Williams @ 2016-12-02 0:13 ` Jeff King 2016-12-02 17:33 ` Brandon Williams 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-12-02 0:13 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, jrnieder On Thu, Dec 01, 2016 at 03:26:56PM -0800, Brandon Williams wrote: > > I started taking a look at your http redirect series (I really should > > have taking a look at it sooner) and I see exactly what you're talking > > about. We can easily move this logic into a function to make it easier > > to generate the two whitelists. > > Thinking about this some more...I was told that having http redirect to > file:// could be scary. The way the new protocol configuration is setup > we have file:// as a default known-safe protocol. Do we need to worry > about this or can we leave this be since this can be overridden by the > user? Hmm. I'm not sure if file:// should actually be USER_ONLY. The submodule code allows it, and it's certainly a convenience, but I guess you could do tricky things by probing somebody's filesystem with submodules URLs. On the other hand, if you are recursively cloning untrusted repos and have sensitive contents on disk, you really _should_ be setting up a protocol whitelist. For HTTP redirects within curl, I think it's a non-issue; curl automatically disallows file:// for redirects, even without us telling it so. For redirects via http-alternates, it's a bit more tricky, as we feed the URL to curl ourselves, so it can't tell the difference between trusted and untrusted input. The main protection provided by my series is "don't follow http-alternates at all". But assuming you did want to use them (by setting http.followRedirects to "true", at least for the server in question), we could then feed file:// directly to curl. But I think we are still OK, because the restricted CURLOPT_PROTOCOL setting would prevent that from working. I.e., git _never_ wants curl to handle file://, because it handles it without calling into remote-curl.c at all. So arguably file:// should be USER_ONLY, but I'm not sure how much it matters in practice. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-02 0:13 ` Jeff King @ 2016-12-02 17:33 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-02 17:33 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, jrnieder On 12/01, Jeff King wrote: > On Thu, Dec 01, 2016 at 03:26:56PM -0800, Brandon Williams wrote: > > > > I started taking a look at your http redirect series (I really should > > > have taking a look at it sooner) and I see exactly what you're talking > > > about. We can easily move this logic into a function to make it easier > > > to generate the two whitelists. > > > > Thinking about this some more...I was told that having http redirect to > > file:// could be scary. The way the new protocol configuration is setup > > we have file:// as a default known-safe protocol. Do we need to worry > > about this or can we leave this be since this can be overridden by the > > user? > > Hmm. I'm not sure if file:// should actually be USER_ONLY. The submodule > code allows it, and it's certainly a convenience, but I guess you could > do tricky things by probing somebody's filesystem with submodules URLs. > On the other hand, if you are recursively cloning untrusted repos and > have sensitive contents on disk, you really _should_ be setting up a > protocol whitelist. > > For HTTP redirects within curl, I think it's a non-issue; curl > automatically disallows file:// for redirects, even without us telling > it so. > > For redirects via http-alternates, it's a bit more tricky, as we feed > the URL to curl ourselves, so it can't tell the difference between > trusted and untrusted input. The main protection provided by my series > is "don't follow http-alternates at all". But assuming you did want to > use them (by setting http.followRedirects to "true", at least for the > server in question), we could then feed file:// directly to curl. But I > think we are still OK, because the restricted CURLOPT_PROTOCOL setting > would prevent that from working. I.e., git _never_ wants curl to handle > file://, because it handles it without calling into remote-curl.c at > all. > > So arguably file:// should be USER_ONLY, but I'm not sure how much it > matters in practice. Ah ok thanks for the good explanation. I was mostly interested in the http redirect case which, as you said, becomes a non-issue due to how we configure curl. Thanks! -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-01 23:07 ` Brandon Williams 2016-12-01 23:26 ` Brandon Williams @ 2016-12-01 23:34 ` Junio C Hamano 2016-12-01 23:58 ` Brandon Williams 1 sibling, 1 reply; 124+ messages in thread From: Junio C Hamano @ 2016-12-01 23:34 UTC (permalink / raw) To: Brandon Williams; +Cc: Jeff King, git, sbeller, bburky, jrnieder Brandon Williams <bmwill@google.com> writes: > I started taking a look at your http redirect series (I really should > have taking a look at it sooner) and I see exactly what you're talking > about. We can easily move this logic into a function to make it easier > to generate the two whitelists. OK. With both of them queued, t5812 seems to barf, just FYI; I'll expect that a future reroll would make things better. Thanks. ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-01 23:34 ` Junio C Hamano @ 2016-12-01 23:58 ` Brandon Williams 2016-12-05 20:04 ` Junio C Hamano 0 siblings, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-12-01 23:58 UTC (permalink / raw) To: Junio C Hamano; +Cc: Jeff King, git, sbeller, bburky, jrnieder On 12/01, Junio C Hamano wrote: > Brandon Williams <bmwill@google.com> writes: > > > I started taking a look at your http redirect series (I really should > > have taking a look at it sooner) and I see exactly what you're talking > > about. We can easily move this logic into a function to make it easier > > to generate the two whitelists. > > OK. With both of them queued, t5812 seems to barf, just FYI; I'll > expect that a future reroll would make things better. Yeah I expected we would see an issue since both these series collide in http.c I'm sending out another reroll of this series so that in Jeff's he can just call 'get_curl_allowed_protocols(-1)' for the non-redirection curl option, which should make this test stop barfing. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-01 23:58 ` Brandon Williams @ 2016-12-05 20:04 ` Junio C Hamano 2016-12-05 22:22 ` Brandon Williams 2016-12-06 13:51 ` Jeff King 0 siblings, 2 replies; 124+ messages in thread From: Junio C Hamano @ 2016-12-05 20:04 UTC (permalink / raw) To: Brandon Williams; +Cc: Jeff King, git, sbeller, bburky, jrnieder Brandon Williams <bmwill@google.com> writes: > On 12/01, Junio C Hamano wrote: >> Brandon Williams <bmwill@google.com> writes: >> >> > I started taking a look at your http redirect series (I really should >> > have taking a look at it sooner) and I see exactly what you're talking >> > about. We can easily move this logic into a function to make it easier >> > to generate the two whitelists. >> >> OK. With both of them queued, t5812 seems to barf, just FYI; I'll >> expect that a future reroll would make things better. > > Yeah I expected we would see an issue since both these series collide in > http.c > > I'm sending out another reroll of this series so that in Jeff's he can > just call 'get_curl_allowed_protocols(-1)' for the non-redirection curl > option, which should make this test stop barfing. I was hoping to eventually merge Peff's series to older maintenance tracks. How bad would it be if we rebased the v8 of this series together with Peff's series to say v2.9 (or even older if it does not look too bad)? ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-05 20:04 ` Junio C Hamano @ 2016-12-05 22:22 ` Brandon Williams 2016-12-05 23:19 ` Junio C Hamano 2016-12-06 13:51 ` Jeff King 1 sibling, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-12-05 22:22 UTC (permalink / raw) To: Junio C Hamano; +Cc: Jeff King, git, sbeller, bburky, jrnieder On 12/05, Junio C Hamano wrote: > Brandon Williams <bmwill@google.com> writes: > > > On 12/01, Junio C Hamano wrote: > >> Brandon Williams <bmwill@google.com> writes: > >> > >> > I started taking a look at your http redirect series (I really should > >> > have taking a look at it sooner) and I see exactly what you're talking > >> > about. We can easily move this logic into a function to make it easier > >> > to generate the two whitelists. > >> > >> OK. With both of them queued, t5812 seems to barf, just FYI; I'll > >> expect that a future reroll would make things better. > > > > Yeah I expected we would see an issue since both these series collide in > > http.c > > > > I'm sending out another reroll of this series so that in Jeff's he can > > just call 'get_curl_allowed_protocols(-1)' for the non-redirection curl > > option, which should make this test stop barfing. > > I was hoping to eventually merge Peff's series to older maintenance > tracks. How bad would it be if we rebased the v8 of this series > together with Peff's series to say v2.9 (or even older if it does > not look too bad)? I just took Jeff's series and applied it on top of mine (and fixed the small problem causing t5812 to fail) and then rebased it to v2.9.0. There were a few issues that needed to be resolved but nothing too hairy. So it would definitely be doable to backport it to the maintenance tracks. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-05 22:22 ` Brandon Williams @ 2016-12-05 23:19 ` Junio C Hamano 2016-12-05 23:22 ` Brandon Williams 0 siblings, 1 reply; 124+ messages in thread From: Junio C Hamano @ 2016-12-05 23:19 UTC (permalink / raw) To: Brandon Williams; +Cc: Jeff King, git, sbeller, bburky, jrnieder Brandon Williams <bmwill@google.com> writes: > I just took Jeff's series and applied it on top of mine (and fixed the > small problem causing t5812 to fail) and then rebased it to v2.9.0. > There were a few issues that needed to be resolved but nothing too > hairy. So it would definitely be doable to backport it to the > maintenance tracks. Thanks for trying. I'd definitely prefer a series that is based on an older codebase that is merged to pu->next->master->maint (in other words, avoid "backport" and rather have "forward merge"). ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-05 23:19 ` Junio C Hamano @ 2016-12-05 23:22 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-05 23:22 UTC (permalink / raw) To: Junio C Hamano; +Cc: Jeff King, git, sbeller, bburky, jrnieder On 12/05, Junio C Hamano wrote: > Brandon Williams <bmwill@google.com> writes: > > > I just took Jeff's series and applied it on top of mine (and fixed the > > small problem causing t5812 to fail) and then rebased it to v2.9.0. > > There were a few issues that needed to be resolved but nothing too > > hairy. So it would definitely be doable to backport it to the > > maintenance tracks. > > Thanks for trying. I'd definitely prefer a series that is based on > an older codebase that is merged to pu->next->master->maint (in > other words, avoid "backport" and rather have "forward merge"). Ah ok. Do you want me to send out the combined patch series based on 2.9.0 (or some other point in time) then? -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-05 20:04 ` Junio C Hamano 2016-12-05 22:22 ` Brandon Williams @ 2016-12-06 13:51 ` Jeff King 2016-12-06 17:53 ` Junio C Hamano 2016-12-06 22:24 ` [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed Brandon Williams 1 sibling, 2 replies; 124+ messages in thread From: Jeff King @ 2016-12-06 13:51 UTC (permalink / raw) To: Junio C Hamano; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder On Mon, Dec 05, 2016 at 12:04:52PM -0800, Junio C Hamano wrote: > > I'm sending out another reroll of this series so that in Jeff's he can > > just call 'get_curl_allowed_protocols(-1)' for the non-redirection curl > > option, which should make this test stop barfing. > > I was hoping to eventually merge Peff's series to older maintenance > tracks. How bad would it be if we rebased the v8 of this series > together with Peff's series to say v2.9 (or even older if it does > not look too bad)? My series actually fixes existing security problems, so I'd consider it a bug-fix. I _think_ Brandon's series is purely about allowing more expressiveness in the whitelist policy, and so could be considered more of a feature. So one option is to apply my series for older 'maint', and then just rebase Brandon's on top of that for 'master'. I don't know if that makes things any easier. I feel funny saying "no, no, mine preempts yours because it is more maint-worthy", but I think that order does make sense. I think it would be OK to put Brandon's on maint, too, though. It is a refactor of an existing security feature to make it more featureful, but the way it is implemented could not cause security regressions unless you use the new feature (IOW, we still respect the whitelist environment exactly as before). -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-06 13:51 ` Jeff King @ 2016-12-06 17:53 ` Junio C Hamano 2016-12-06 18:10 ` Jeff King 2016-12-06 22:24 ` [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed Brandon Williams 1 sibling, 1 reply; 124+ messages in thread From: Junio C Hamano @ 2016-12-06 17:53 UTC (permalink / raw) To: Jeff King; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder Jeff King <peff@peff.net> writes: > I don't know if that makes things any easier. I feel funny saying "no, > no, mine preempts yours because it is more maint-worthy", but I think > that order does make sense. > > I think it would be OK to put Brandon's on maint, too, though. It is a > refactor of an existing security feature to make it more featureful, but > the way it is implemented could not cause security regressions unless > you use the new feature (IOW, we still respect the whitelist environment > exactly as before). I think I merged yours and then Brandon's on jch/pu branches in that order, and the conflict resolution should look OK. I however forked yours on v2.11.0-rc1, which would need to be rebased to one of the earlier maintenance tracks, before we can merge it to 'next'. ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-06 17:53 ` Junio C Hamano @ 2016-12-06 18:10 ` Jeff King 2016-12-06 18:24 ` [PATCH v2] jk/http-walker-limit-redirect rebased to maint-2.9 Jeff King 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-12-06 18:10 UTC (permalink / raw) To: Junio C Hamano; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder On Tue, Dec 06, 2016 at 09:53:53AM -0800, Junio C Hamano wrote: > Jeff King <peff@peff.net> writes: > > > I don't know if that makes things any easier. I feel funny saying "no, > > no, mine preempts yours because it is more maint-worthy", but I think > > that order does make sense. > > > > I think it would be OK to put Brandon's on maint, too, though. It is a > > refactor of an existing security feature to make it more featureful, but > > the way it is implemented could not cause security regressions unless > > you use the new feature (IOW, we still respect the whitelist environment > > exactly as before). > > I think I merged yours and then Brandon's on jch/pu branches in that > order, and the conflict resolution should look OK. > > I however forked yours on v2.11.0-rc1, which would need to be > rebased to one of the earlier maintenance tracks, before we can > merge it to 'next'. Yeah, I built it on top of master. It does depend on some of the http-walker changes Eric made a few months ago. In particular, 17966c0a6 (http: avoid disconnecting on 404s for loose objects, 2016-07-11) added some checks against the HTTP status code, and my series modifies the checks (mostly so that ">= 400" becomes ">= 300"). Rebasing on maint-2.9 means omitting those changes. That preserves the security properties, but means that the error handling is worse when we see an illegal redirect. That may be OK, though. Since the resolution is to omit the changes entirely from my series, merging up to v2.11 wouldn't produce any conflicts. We'd need to have a separate set of patches adding those changes back in. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v2] jk/http-walker-limit-redirect rebased to maint-2.9 2016-12-06 18:10 ` Jeff King @ 2016-12-06 18:24 ` Jeff King 2016-12-06 18:24 ` [PATCH v2 1/6] http: simplify update_url_from_redirect Jeff King ` (5 more replies) 0 siblings, 6 replies; 124+ messages in thread From: Jeff King @ 2016-12-06 18:24 UTC (permalink / raw) To: Junio C Hamano; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder On Tue, Dec 06, 2016 at 01:10:08PM -0500, Jeff King wrote: > > I think I merged yours and then Brandon's on jch/pu branches in that > > order, and the conflict resolution should look OK. > > > > I however forked yours on v2.11.0-rc1, which would need to be > > rebased to one of the earlier maintenance tracks, before we can > > merge it to 'next'. > > Yeah, I built it on top of master. > > It does depend on some of the http-walker changes Eric made a few months > ago. In particular, 17966c0a6 (http: avoid disconnecting on 404s for > loose objects, 2016-07-11) added some checks against the HTTP status > code, and my series modifies the checks (mostly so that ">= 400" becomes > ">= 300"). > > Rebasing on maint-2.9 means omitting those changes. That preserves the > security properties, but means that the error handling is worse when we > see an illegal redirect. That may be OK, though. > > Since the resolution is to omit the changes entirely from my series, > merging up to v2.11 wouldn't produce any conflicts. We'd need to have a > separate set of patches adding those changes back in. This actually turned out to be pretty easy. The final patch from my original series is the one that tweaks the error-handling from 17966c0a6. The rest of them are _almost_ untouched, except that one of the error-handling tweaks gets bumped to the final patch. So here's a resend of the patches, rebased on your maint-2.9 branch. Patches 1-5 should be applied directly there. On maint-2.10, you can merge up maint-2.9, and then apply patch 6. Hopefully that makes sense, but I've pushed branches jk/maint-X-http-redirect to https://github.com/peff/git that show the final configuration (and a diff of jk/maint-2.10-http-redirect shows the outcome is identical to applying the original series on top of 2.10). Merging up to 2.11 should be trivial. [1/6]: http: simplify update_url_from_redirect [2/6]: http: always update the base URL for redirects [3/6]: remote-curl: rename shadowed options variable [4/6]: http: make redirects more obvious [5/6]: http: treat http-alternates like redirects [6/6]: http-walker: complain about non-404 loose object errors -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v2 1/6] http: simplify update_url_from_redirect 2016-12-06 18:24 ` [PATCH v2] jk/http-walker-limit-redirect rebased to maint-2.9 Jeff King @ 2016-12-06 18:24 ` Jeff King 2016-12-06 18:24 ` [PATCH v2 2/6] http: always update the base URL for redirects Jeff King ` (4 subsequent siblings) 5 siblings, 0 replies; 124+ messages in thread From: Jeff King @ 2016-12-06 18:24 UTC (permalink / raw) To: Junio C Hamano; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder This function looks for a common tail between what we asked for and where we were redirected to, but it open-codes the comparison. We can avoid some confusing subtractions by using strip_suffix_mem(). Signed-off-by: Jeff King <peff@peff.net> --- http.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/http.c b/http.c index d8e427b69..d4034a14b 100644 --- a/http.c +++ b/http.c @@ -1500,7 +1500,7 @@ static int update_url_from_redirect(struct strbuf *base, const struct strbuf *got) { const char *tail; - size_t tail_len; + size_t new_len; if (!strcmp(asked, got->buf)) return 0; @@ -1509,14 +1509,12 @@ static int update_url_from_redirect(struct strbuf *base, die("BUG: update_url_from_redirect: %s is not a superset of %s", asked, base->buf); - tail_len = strlen(tail); - - if (got->len < tail_len || - strcmp(tail, got->buf + got->len - tail_len)) + new_len = got->len; + if (!strip_suffix_mem(got->buf, &new_len, tail)) return 0; /* insane redirect scheme */ strbuf_reset(base); - strbuf_add(base, got->buf, got->len - tail_len); + strbuf_add(base, got->buf, new_len); return 1; } -- 2.11.0.191.gdb26c57 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v2 2/6] http: always update the base URL for redirects 2016-12-06 18:24 ` [PATCH v2] jk/http-walker-limit-redirect rebased to maint-2.9 Jeff King 2016-12-06 18:24 ` [PATCH v2 1/6] http: simplify update_url_from_redirect Jeff King @ 2016-12-06 18:24 ` Jeff King 2016-12-06 18:24 ` [PATCH v2 3/6] remote-curl: rename shadowed options variable Jeff King ` (3 subsequent siblings) 5 siblings, 0 replies; 124+ messages in thread From: Jeff King @ 2016-12-06 18:24 UTC (permalink / raw) To: Junio C Hamano; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder If a malicious server redirects the initial ref advertisement, it may be able to leak sha1s from other, unrelated servers that the client has access to. For example, imagine that Alice is a git user, she has access to a private repository on a server hosted by Bob, and Mallory runs a malicious server and wants to find out about Bob's private repository. Mallory asks Alice to clone an unrelated repository from her over HTTP. When Alice's client contacts Mallory's server for the initial ref advertisement, the server issues an HTTP redirect for Bob's server. Alice contacts Bob's server and gets the ref advertisement for the private repository. If there is anything to fetch, she then follows up by asking the server for one or more sha1 objects. But who is the server? If it is still Mallory's server, then Alice will leak the existence of those sha1s to her. Since commit c93c92f30 (http: update base URLs when we see redirects, 2013-09-28), the client usually rewrites the base URL such that all further requests will go to Bob's server. But this is done by textually matching the URL. If we were originally looking for "http://mallory/repo.git/info/refs", and we got pointed at "http://bob/other.git/info/refs", then we know that the right root is "http://bob/other.git". If the redirect appears to change more than just the root, we punt and continue to use the original server. E.g., imagine the redirect adds a URL component that Bob's server will ignore, like "http://bob/other.git/info/refs?dummy=1". We can solve this by aborting in this case rather than silently continuing to use Mallory's server. In addition to protecting from sha1 leakage, it's arguably safer and more sane to refuse a confusing redirect like that in general. For example, part of the motivation in c93c92f30 is avoiding accidentally sending credentials over clear http, just to get a response that says "try again over https". So even in a non-malicious case, we'd prefer to err on the side of caution. The downside is that it's possible this will break a legitimate but complicated server-side redirection scheme. The setup given in the newly added test does work, but it's convoluted enough that we don't need to care about it. A more plausible case would be a server which redirects a request for "info/refs?service=git-upload-pack" to just "info/refs" (because it does not do smart HTTP, and for some reason really dislikes query parameters). Right now we would transparently downgrade to dumb-http, but with this patch, we'd complain (and the user would have to set GIT_SMART_HTTP=0 to fetch). Reported-by: Jann Horn <jannh@google.com> Signed-off-by: Jeff King <peff@peff.net> --- http.c | 12 ++++++++---- t/lib-httpd/apache.conf | 8 ++++++++ t/t5551-http-fetch-smart.sh | 4 ++++ t/t5812-proto-disable-http.sh | 1 + 4 files changed, 21 insertions(+), 4 deletions(-) diff --git a/http.c b/http.c index d4034a14b..718d2109b 100644 --- a/http.c +++ b/http.c @@ -1491,9 +1491,9 @@ static int http_request(const char *url, * * Note that this assumes a sane redirect scheme. It's entirely possible * in the example above to end up at a URL that does not even end in - * "info/refs". In such a case we simply punt, as there is not much we can - * do (and such a scheme is unlikely to represent a real git repository, - * which means we are likely about to abort anyway). + * "info/refs". In such a case we die. There's not much we can do, such a + * scheme is unlikely to represent a real git repository, and failing to + * rewrite the base opens options for malicious redirects to do funny things. */ static int update_url_from_redirect(struct strbuf *base, const char *asked, @@ -1511,10 +1511,14 @@ static int update_url_from_redirect(struct strbuf *base, new_len = got->len; if (!strip_suffix_mem(got->buf, &new_len, tail)) - return 0; /* insane redirect scheme */ + die(_("unable to update url base from redirection:\n" + " asked for: %s\n" + " redirect: %s"), + asked, got->buf); strbuf_reset(base); strbuf_add(base, got->buf, new_len); + return 1; } diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf index 018a83a5a..9a355fb1c 100644 --- a/t/lib-httpd/apache.conf +++ b/t/lib-httpd/apache.conf @@ -132,6 +132,14 @@ RewriteRule ^/ftp-redir/(.*)$ ftp://localhost:1000/$1 [R=302] RewriteRule ^/loop-redir/x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-(.*) /$1 [R=302] RewriteRule ^/loop-redir/(.*)$ /loop-redir/x-$1 [R=302] +# The first rule issues a client-side redirect to something +# that _doesn't_ look like a git repo. The second rule is a +# server-side rewrite, so that it turns out the odd-looking +# thing _is_ a git repo. The "[PT]" tells Apache to match +# the usual ScriptAlias rules for /smart. +RewriteRule ^/insane-redir/(.*)$ /intern-redir/$1/foo [R=301] +RewriteRule ^/intern-redir/(.*)/foo$ /smart/$1 [PT] + # Apache 2.2 does not understand <RequireAll>, so we use RewriteCond. # And as RewriteCond does not allow testing for non-matches, we match # the desired case first (one has abra, two has cadabra), and let it diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh index 2f375eb94..d8826acde 100755 --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@ -110,6 +110,10 @@ test_expect_success 'redirects re-root further requests' ' git clone $HTTPD_URL/smart-redir-limited/repo.git repo-redir-limited ' +test_expect_success 're-rooting dies on insane schemes' ' + test_must_fail git clone $HTTPD_URL/insane-redir/repo.git insane +' + test_expect_success 'clone from password-protected repository' ' echo two >expect && set_askpass user@host pass@host && diff --git a/t/t5812-proto-disable-http.sh b/t/t5812-proto-disable-http.sh index 0d105d541..044cc152f 100755 --- a/t/t5812-proto-disable-http.sh +++ b/t/t5812-proto-disable-http.sh @@ -18,6 +18,7 @@ test_proto "smart http" http "$HTTPD_URL/smart/repo.git" test_expect_success 'curl redirects respect whitelist' ' test_must_fail env GIT_ALLOW_PROTOCOL=http:https \ + GIT_SMART_HTTP=0 \ git clone "$HTTPD_URL/ftp-redir/repo.git" 2>stderr && { test_i18ngrep "ftp.*disabled" stderr || -- 2.11.0.191.gdb26c57 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v2 3/6] remote-curl: rename shadowed options variable 2016-12-06 18:24 ` [PATCH v2] jk/http-walker-limit-redirect rebased to maint-2.9 Jeff King 2016-12-06 18:24 ` [PATCH v2 1/6] http: simplify update_url_from_redirect Jeff King 2016-12-06 18:24 ` [PATCH v2 2/6] http: always update the base URL for redirects Jeff King @ 2016-12-06 18:24 ` Jeff King 2016-12-06 18:24 ` [PATCH v2 4/6] http: make redirects more obvious Jeff King ` (2 subsequent siblings) 5 siblings, 0 replies; 124+ messages in thread From: Jeff King @ 2016-12-06 18:24 UTC (permalink / raw) To: Junio C Hamano; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder The discover_refs() function has a local "options" variable to hold the http_get_options we pass to http_get_strbuf(). But this shadows the global "struct options" that holds our program-level options, which cannot be accessed from this function. Let's give the local one a more descriptive name so we can tell the two apart. Signed-off-by: Jeff King <peff@peff.net> --- remote-curl.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/remote-curl.c b/remote-curl.c index 6b83b7783..e3803daa3 100644 --- a/remote-curl.c +++ b/remote-curl.c @@ -254,7 +254,7 @@ static struct discovery *discover_refs(const char *service, int for_push) struct strbuf effective_url = STRBUF_INIT; struct discovery *last = last_discovery; int http_ret, maybe_smart = 0; - struct http_get_options options; + struct http_get_options http_options; if (last && !strcmp(service, last->service)) return last; @@ -271,15 +271,15 @@ static struct discovery *discover_refs(const char *service, int for_push) strbuf_addf(&refs_url, "service=%s", service); } - memset(&options, 0, sizeof(options)); - options.content_type = &type; - options.charset = &charset; - options.effective_url = &effective_url; - options.base_url = &url; - options.no_cache = 1; - options.keep_error = 1; + memset(&http_options, 0, sizeof(http_options)); + http_options.content_type = &type; + http_options.charset = &charset; + http_options.effective_url = &effective_url; + http_options.base_url = &url; + http_options.no_cache = 1; + http_options.keep_error = 1; - http_ret = http_get_strbuf(refs_url.buf, &buffer, &options); + http_ret = http_get_strbuf(refs_url.buf, &buffer, &http_options); switch (http_ret) { case HTTP_OK: break; -- 2.11.0.191.gdb26c57 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v2 4/6] http: make redirects more obvious 2016-12-06 18:24 ` [PATCH v2] jk/http-walker-limit-redirect rebased to maint-2.9 Jeff King ` (2 preceding siblings ...) 2016-12-06 18:24 ` [PATCH v2 3/6] remote-curl: rename shadowed options variable Jeff King @ 2016-12-06 18:24 ` Jeff King 2016-12-06 18:24 ` [PATCH v2 5/6] http: treat http-alternates like redirects Jeff King 2016-12-06 18:25 ` [PATCH v2 6/6] http-walker: complain about non-404 loose object errors Jeff King 5 siblings, 0 replies; 124+ messages in thread From: Jeff King @ 2016-12-06 18:24 UTC (permalink / raw) To: Junio C Hamano; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder We instruct curl to always follow HTTP redirects. This is convenient, but it creates opportunities for malicious servers to create confusing situations. For instance, imagine Alice is a git user with access to a private repository on Bob's server. Mallory runs her own server and wants to access objects from Bob's repository. Mallory may try a few tricks that involve asking Alice to clone from her, build on top, and then push the result: 1. Mallory may simply redirect all fetch requests to Bob's server. Git will transparently follow those redirects and fetch Bob's history, which Alice may believe she got from Mallory. The subsequent push seems like it is just feeding Mallory back her own objects, but is actually leaking Bob's objects. There is nothing in git's output to indicate that Bob's repository was involved at all. The downside (for Mallory) of this attack is that Alice will have received Bob's entire repository, and is likely to notice that when building on top of it. 2. If Mallory happens to know the sha1 of some object X in Bob's repository, she can instead build her own history that references that object. She then runs a dumb http server, and Alice's client will fetch each object individually. When it asks for X, Mallory redirects her to Bob's server. The end result is that Alice obtains objects from Bob, but they may be buried deep in history. Alice is less likely to notice. Both of these attacks are fairly hard to pull off. There's a social component in getting Mallory to convince Alice to work with her. Alice may be prompted for credentials in accessing Bob's repository (but not always, if she is using a credential helper that caches). Attack (1) requires a certain amount of obliviousness on Alice's part while making a new commit. Attack (2) requires that Mallory knows a sha1 in Bob's repository, that Bob's server supports dumb http, and that the object in question is loose on Bob's server. But we can probably make things a bit more obvious without any loss of functionality. This patch does two things to that end. First, when we encounter a whole-repo redirect during the initial ref discovery, we now inform the user on stderr, making attack (1) much more obvious. Second, the decision to follow redirects is now configurable. The truly paranoid can set the new http.followRedirects to false to avoid any redirection entirely. But for a more practical default, we will disallow redirects only after the initial ref discovery. This is enough to thwart attacks similar to (2), while still allowing the common use of redirects at the repository level. Since c93c92f30 (http: update base URLs when we see redirects, 2013-09-28) we re-root all further requests from the redirect destination, which should generally mean that no further redirection is necessary. As an escape hatch, in case there really is a server that needs to redirect individual requests, the user can set http.followRedirects to "true" (and this can be done on a per-server basis via http.*.followRedirects config). Reported-by: Jann Horn <jannh@google.com> Signed-off-by: Jeff King <peff@peff.net> --- Documentation/config.txt | 10 ++++++++++ http.c | 31 +++++++++++++++++++++++++++++-- http.h | 10 +++++++++- remote-curl.c | 4 ++++ t/lib-httpd/apache.conf | 6 ++++++ t/t5550-http-fetch-dumb.sh | 23 +++++++++++++++++++++++ 6 files changed, 81 insertions(+), 3 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index f4721a048..815333643 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -1833,6 +1833,16 @@ http.userAgent:: of common USER_AGENT strings (but not including those like git/1.7.1). Can be overridden by the `GIT_HTTP_USER_AGENT` environment variable. +http.followRedirects:: + Whether git should follow HTTP redirects. If set to `true`, git + will transparently follow any redirect issued by a server it + encounters. If set to `false`, git will treat all redirects as + errors. If set to `initial`, git will follow redirects only for + the initial request to a remote, but not for subsequent + follow-up HTTP requests. Since git uses the redirected URL as + the base for the follow-up requests, this is generally + sufficient. The default is `initial`. + http.<url>.*:: Any of the http.* options above can be applied selectively to some URLs. For a config key to match a URL, each element of the config key is diff --git a/http.c b/http.c index 718d2109b..b99ade5fa 100644 --- a/http.c +++ b/http.c @@ -98,6 +98,8 @@ static int http_proactive_auth; static const char *user_agent; static int curl_empty_auth; +enum http_follow_config http_follow_config = HTTP_FOLLOW_INITIAL; + #if LIBCURL_VERSION_NUM >= 0x071700 /* Use CURLOPT_KEYPASSWD as is */ #elif LIBCURL_VERSION_NUM >= 0x070903 @@ -337,6 +339,16 @@ static int http_options(const char *var, const char *value, void *cb) return 0; } + if (!strcmp("http.followredirects", var)) { + if (value && !strcmp(value, "initial")) + http_follow_config = HTTP_FOLLOW_INITIAL; + else if (git_config_bool(var, value)) + http_follow_config = HTTP_FOLLOW_ALWAYS; + else + http_follow_config = HTTP_FOLLOW_NONE; + return 0; + } + /* Fall back on the default ones */ return git_default_config(var, value, cb); } @@ -553,7 +565,6 @@ static CURL *get_curl_handle(void) curl_low_speed_time); } - curl_easy_setopt(result, CURLOPT_FOLLOWLOCATION, 1); curl_easy_setopt(result, CURLOPT_MAXREDIRS, 20); #if LIBCURL_VERSION_NUM >= 0x071301 curl_easy_setopt(result, CURLOPT_POSTREDIR, CURL_REDIR_POST_ALL); @@ -882,6 +893,16 @@ struct active_request_slot *get_active_slot(void) curl_easy_setopt(slot->curl, CURLOPT_FAILONERROR, 1); curl_easy_setopt(slot->curl, CURLOPT_RANGE, NULL); + /* + * Default following to off unless "ALWAYS" is configured; this gives + * callers a sane starting point, and they can tweak for individual + * HTTP_FOLLOW_* cases themselves. + */ + if (http_follow_config == HTTP_FOLLOW_ALWAYS) + curl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 1); + else + curl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 0); + #if LIBCURL_VERSION_NUM >= 0x070a08 curl_easy_setopt(slot->curl, CURLOPT_IPRESOLVE, git_curl_ipresolve); #endif @@ -1122,9 +1143,12 @@ static int handle_curl_result(struct slot_results *results) * If we see a failing http code with CURLE_OK, we have turned off * FAILONERROR (to keep the server's custom error response), and should * translate the code into failure here. + * + * Likewise, if we see a redirect (30x code), that means we turned off + * redirect-following, and we should treat the result as an error. */ if (results->curl_result == CURLE_OK && - results->http_code >= 400) { + results->http_code >= 300) { results->curl_result = CURLE_HTTP_RETURNED_ERROR; /* * Normally curl will already have put the "reason phrase" @@ -1443,6 +1467,9 @@ static int http_request(const char *url, strbuf_addstr(&buf, " no-cache"); if (options && options->keep_error) curl_easy_setopt(slot->curl, CURLOPT_FAILONERROR, 0); + if (options && options->initial_request && + http_follow_config == HTTP_FOLLOW_INITIAL) + curl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 1); headers = curl_slist_append(headers, buf.buf); diff --git a/http.h b/http.h index 36f558bfb..31b4cc94b 100644 --- a/http.h +++ b/http.h @@ -116,6 +116,13 @@ extern struct credential http_auth; extern char curl_errorstr[CURL_ERROR_SIZE]; +enum http_follow_config { + HTTP_FOLLOW_NONE, + HTTP_FOLLOW_ALWAYS, + HTTP_FOLLOW_INITIAL +}; +extern enum http_follow_config http_follow_config; + static inline int missing__target(int code, int result) { return /* file:// URL -- do we ever use one??? */ @@ -139,7 +146,8 @@ extern char *get_remote_object_url(const char *url, const char *hex, /* Options for http_get_*() */ struct http_get_options { unsigned no_cache:1, - keep_error:1; + keep_error:1, + initial_request:1; /* If non-NULL, returns the content-type of the response. */ struct strbuf *content_type; diff --git a/remote-curl.c b/remote-curl.c index e3803daa3..05ae8dead 100644 --- a/remote-curl.c +++ b/remote-curl.c @@ -276,6 +276,7 @@ static struct discovery *discover_refs(const char *service, int for_push) http_options.charset = &charset; http_options.effective_url = &effective_url; http_options.base_url = &url; + http_options.initial_request = 1; http_options.no_cache = 1; http_options.keep_error = 1; @@ -294,6 +295,9 @@ static struct discovery *discover_refs(const char *service, int for_push) die("unable to access '%s': %s", url.buf, curl_errorstr); } + if (options.verbosity && !starts_with(refs_url.buf, url.buf)) + warning(_("redirecting to %s"), url.buf); + last= xcalloc(1, sizeof(*last_discovery)); last->service = service; last->buf_alloc = strbuf_detach(&buffer, &last->len); diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf index 9a355fb1c..5b408d649 100644 --- a/t/lib-httpd/apache.conf +++ b/t/lib-httpd/apache.conf @@ -123,6 +123,7 @@ ScriptAlias /error/ error.sh/ </Files> RewriteEngine on +RewriteRule ^/dumb-redir/(.*)$ /dumb/$1 [R=301] RewriteRule ^/smart-redir-perm/(.*)$ /smart/$1 [R=301] RewriteRule ^/smart-redir-temp/(.*)$ /smart/$1 [R=302] RewriteRule ^/smart-redir-auth/(.*)$ /auth/smart/$1 [R=301] @@ -140,6 +141,11 @@ RewriteRule ^/loop-redir/(.*)$ /loop-redir/x-$1 [R=302] RewriteRule ^/insane-redir/(.*)$ /intern-redir/$1/foo [R=301] RewriteRule ^/intern-redir/(.*)/foo$ /smart/$1 [PT] +# Serve info/refs internally without redirecting, but +# issue a redirect for any object requests. +RewriteRule ^/redir-objects/(.*/info/refs)$ /dumb/$1 [PT] +RewriteRule ^/redir-objects/(.*/objects/.*)$ /dumb/$1 [R=301] + # Apache 2.2 does not understand <RequireAll>, so we use RewriteCond. # And as RewriteCond does not allow testing for non-matches, we match # the desired case first (one has abra, two has cadabra), and let it diff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh index 3484b6f0f..ad94ed7b1 100755 --- a/t/t5550-http-fetch-dumb.sh +++ b/t/t5550-http-fetch-dumb.sh @@ -299,5 +299,28 @@ test_expect_success 'git client does not send an empty Accept-Language' ' ! grep "^Accept-Language:" stderr ' +test_expect_success 'redirects can be forbidden/allowed' ' + test_must_fail git -c http.followRedirects=false \ + clone $HTTPD_URL/dumb-redir/repo.git dumb-redir && + git -c http.followRedirects=true \ + clone $HTTPD_URL/dumb-redir/repo.git dumb-redir 2>stderr +' + +test_expect_success 'redirects are reported to stderr' ' + # just look for a snippet of the redirected-to URL + test_i18ngrep /dumb/ stderr +' + +test_expect_success 'non-initial redirects can be forbidden' ' + test_must_fail git -c http.followRedirects=initial \ + clone $HTTPD_URL/redir-objects/repo.git redir-objects && + git -c http.followRedirects=true \ + clone $HTTPD_URL/redir-objects/repo.git redir-objects +' + +test_expect_success 'http.followRedirects defaults to "initial"' ' + test_must_fail git clone $HTTPD_URL/redir-objects/repo.git default +' + stop_httpd test_done -- 2.11.0.191.gdb26c57 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v2 5/6] http: treat http-alternates like redirects 2016-12-06 18:24 ` [PATCH v2] jk/http-walker-limit-redirect rebased to maint-2.9 Jeff King ` (3 preceding siblings ...) 2016-12-06 18:24 ` [PATCH v2 4/6] http: make redirects more obvious Jeff King @ 2016-12-06 18:24 ` Jeff King 2016-12-06 18:25 ` [PATCH v2 6/6] http-walker: complain about non-404 loose object errors Jeff King 5 siblings, 0 replies; 124+ messages in thread From: Jeff King @ 2016-12-06 18:24 UTC (permalink / raw) To: Junio C Hamano; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder The previous commit made HTTP redirects more obvious and tightened up the default behavior. However, there's another way for a server to ask a git client to fetch arbitrary content: by having an http-alternates file (or a regular alternates file, which is used as a backup). Similar to the HTTP redirect case, a malicious server can claim to have refs pointing at object X, return a 404 when the client asks for X, but point to some other URL via http-alternates, which the client will transparently fetch. The end result is that it looks from the user's perspective like the objects came from the malicious server, as the other URL is not mentioned at all. Worse, because we feed the new URL to curl ourselves, the usual protocol restrictions do not kick in (neither curl's default of disallowing file://, nor the protocol whitelisting in f4113cac0 (http: limit redirection to protocol-whitelist, 2015-09-22). Let's apply the same rules here as we do for HTTP redirects. Namely: - unless http.followRedirects is set to "always", we will not follow remote redirects from http-alternates (or alternates) at all - set CURLOPT_PROTOCOLS alongside CURLOPT_REDIR_PROTOCOLS restrict ourselves to a known-safe set and respect any user-provided whitelist. - mention alternate object stores on stderr so that the user is aware another source of objects may be involved The first item may prove to be too restrictive. The most common use of alternates is to point to another path on the same server. While it's possible for a single-server redirect to be an attack, it takes a fairly obscure setup (victim and evil repository on the same host, host speaks dumb http, and evil repository has access to edit its own http-alternates file). So we could make the checks more specific, and only cover cross-server redirects. But that means parsing the URLs ourselves, rather than letting curl handle them. This patch goes for the simpler approach. Given that they are only used with dumb http, http-alternates are probably pretty rare. And there's an escape hatch: the user can allow redirects on a specific server by setting http.<url>.followRedirects to "always". Reported-by: Jann Horn <jannh@google.com> Signed-off-by: Jeff King <peff@peff.net> --- http-walker.c | 8 +++++--- http.c | 1 + t/t5550-http-fetch-dumb.sh | 38 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 44 insertions(+), 3 deletions(-) diff --git a/http-walker.c b/http-walker.c index 2c721f0c3..4bff31c44 100644 --- a/http-walker.c +++ b/http-walker.c @@ -290,9 +290,8 @@ static void process_alternates_response(void *callback_data) struct strbuf target = STRBUF_INIT; strbuf_add(&target, base, serverlen); strbuf_add(&target, data + i, posn - i - 7); - if (walker->get_verbosely) - fprintf(stderr, "Also look at %s\n", - target.buf); + warning("adding alternate object store: %s", + target.buf); newalt = xmalloc(sizeof(*newalt)); newalt->next = NULL; newalt->base = strbuf_detach(&target, NULL); @@ -318,6 +317,9 @@ static void fetch_alternates(struct walker *walker, const char *base) struct alternates_request alt_req; struct walker_data *cdata = walker->data; + if (http_follow_config != HTTP_FOLLOW_ALWAYS) + return; + /* * If another request has already started fetching alternates, * wait for them to arrive and return to processing this request's diff --git a/http.c b/http.c index b99ade5fa..a9778bfa4 100644 --- a/http.c +++ b/http.c @@ -581,6 +581,7 @@ static CURL *get_curl_handle(void) if (is_transport_allowed("ftps")) allowed_protocols |= CURLPROTO_FTPS; curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); + curl_easy_setopt(result, CURLOPT_PROTOCOLS, allowed_protocols); #else if (transport_restrict_protocols()) warning("protocol restrictions not applied to curl redirects because\n" diff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh index ad94ed7b1..22011f0b6 100755 --- a/t/t5550-http-fetch-dumb.sh +++ b/t/t5550-http-fetch-dumb.sh @@ -322,5 +322,43 @@ test_expect_success 'http.followRedirects defaults to "initial"' ' test_must_fail git clone $HTTPD_URL/redir-objects/repo.git default ' +# The goal is for a clone of the "evil" repository, which has no objects +# itself, to cause the client to fetch objects from the "victim" repository. +test_expect_success 'set up evil alternates scheme' ' + victim=$HTTPD_DOCUMENT_ROOT_PATH/victim.git && + git init --bare "$victim" && + git -C "$victim" --work-tree=. commit --allow-empty -m secret && + git -C "$victim" repack -ad && + git -C "$victim" update-server-info && + sha1=$(git -C "$victim" rev-parse HEAD) && + + evil=$HTTPD_DOCUMENT_ROOT_PATH/evil.git && + git init --bare "$evil" && + # do this by hand to avoid object existence check + printf "%s\\t%s\\n" $sha1 refs/heads/master >"$evil/info/refs" +' + +# Here we'll just redirect via HTTP. In a real-world attack these would be on +# different servers, but we should reject it either way. +test_expect_success 'http-alternates is a non-initial redirect' ' + echo "$HTTPD_URL/dumb/victim.git/objects" \ + >"$evil/objects/info/http-alternates" && + test_must_fail git -c http.followRedirects=initial \ + clone $HTTPD_URL/dumb/evil.git evil-initial && + git -c http.followRedirects=true \ + clone $HTTPD_URL/dumb/evil.git evil-initial +' + +# Curl supports a lot of protocols that we'd prefer not to allow +# http-alternates to use, but it's hard to test whether curl has +# accessed, say, the SMTP protocol, because we are not running an SMTP server. +# But we can check that it does not allow access to file://, which would +# otherwise allow this clone to complete. +test_expect_success 'http-alternates cannot point at funny protocols' ' + echo "file://$victim/objects" >"$evil/objects/info/http-alternates" && + test_must_fail git -c http.followRedirects=true \ + clone "$HTTPD_URL/dumb/evil.git" evil-file +' + stop_httpd test_done -- 2.11.0.191.gdb26c57 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v2 6/6] http-walker: complain about non-404 loose object errors 2016-12-06 18:24 ` [PATCH v2] jk/http-walker-limit-redirect rebased to maint-2.9 Jeff King ` (4 preceding siblings ...) 2016-12-06 18:24 ` [PATCH v2 5/6] http: treat http-alternates like redirects Jeff King @ 2016-12-06 18:25 ` Jeff King 5 siblings, 0 replies; 124+ messages in thread From: Jeff King @ 2016-12-06 18:25 UTC (permalink / raw) To: Junio C Hamano; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder Since commit 17966c0a6 (http: avoid disconnecting on 404s for loose objects, 2016-07-11), we turn off curl's FAILONERROR option and instead manually deal with failing HTTP codes. However, the logic to do so only recognizes HTTP 404 as a failure. This is probably the most common result, but if we were to get another code, the curl result remains CURLE_OK, and we treat it as success. We still end up detecting the failure when we try to zlib-inflate the object (which will fail), but instead of reporting the HTTP error, we just claim that the object is corrupt. Instead, let's catch anything in the 300's or above as an error (300's are redirects which are not an error at the HTTP level, but are an indication that we've explicitly disabled redirects, so we should treat them as such; we certainly don't have the resulting object content). Note that we also fill in req->errorstr, which we didn't do before. Without FAILONERROR, curl will not have filled this in, and it will remain a blank string. This never mattered for the 404 case, because in the logic below we hit the "missing_target()" branch and print nothing. But for other errors, we'd want to say _something_, if only to fill in the blank slot in the error message. Signed-off-by: Jeff King <peff@peff.net> --- The second hunk here is new in v2; earlier it appeared in patch 3. But arguably it goes better here anyway; I didn't even need to modify the commit message to explain it. http-walker.c | 7 +++++-- http.c | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/http-walker.c b/http-walker.c index 25a8b1ad4..c2f81cd6a 100644 --- a/http-walker.c +++ b/http-walker.c @@ -482,10 +482,13 @@ static int fetch_object(struct walker *walker, unsigned char *sha1) * we turned off CURLOPT_FAILONERROR to avoid losing a * persistent connection and got CURLE_OK. */ - if (req->http_code == 404 && req->curl_result == CURLE_OK && + if (req->http_code >= 300 && req->curl_result == CURLE_OK && (starts_with(req->url, "http://") || - starts_with(req->url, "https://"))) + starts_with(req->url, "https://"))) { req->curl_result = CURLE_HTTP_RETURNED_ERROR; + xsnprintf(req->errorstr, sizeof(req->errorstr), + "HTTP request failed"); + } if (obj_req->state == ABORTED) { ret = error("Request for %s aborted", hex); diff --git a/http.c b/http.c index ff4d88231..c25d1d540 100644 --- a/http.c +++ b/http.c @@ -2021,7 +2021,7 @@ static size_t fwrite_sha1_file(char *ptr, size_t eltsize, size_t nmemb, if (c != CURLE_OK) die("BUG: curl_easy_getinfo for HTTP code failed: %s", curl_easy_strerror(c)); - if (slot->http_code >= 400) + if (slot->http_code >= 300) return size; } -- 2.11.0.191.gdb26c57 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed 2016-12-06 13:51 ` Jeff King 2016-12-06 17:53 ` Junio C Hamano @ 2016-12-06 22:24 ` Brandon Williams 1 sibling, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-06 22:24 UTC (permalink / raw) To: Jeff King; +Cc: Junio C Hamano, git, sbeller, bburky, jrnieder On 12/06, Jeff King wrote: > On Mon, Dec 05, 2016 at 12:04:52PM -0800, Junio C Hamano wrote: > > > > I'm sending out another reroll of this series so that in Jeff's he can > > > just call 'get_curl_allowed_protocols(-1)' for the non-redirection curl > > > option, which should make this test stop barfing. > > > > I was hoping to eventually merge Peff's series to older maintenance > > tracks. How bad would it be if we rebased the v8 of this series > > together with Peff's series to say v2.9 (or even older if it does > > not look too bad)? > > My series actually fixes existing security problems, so I'd consider it > a bug-fix. I _think_ Brandon's series is purely about allowing more > expressiveness in the whitelist policy, and so could be considered more > of a feature. Yes this was really the main intent on my series. > So one option is to apply my series for older 'maint', and then just > rebase Brandon's on top of that for 'master'. > > I don't know if that makes things any easier. I feel funny saying "no, > no, mine preempts yours because it is more maint-worthy", but I think > that order does make sense. > > I think it would be OK to put Brandon's on maint, too, though. It is a > refactor of an existing security feature to make it more featureful, but > the way it is implemented could not cause security regressions unless > you use the new feature (IOW, we still respect the whitelist environment > exactly as before). Either way let me know if there is something I need to do. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v8 0/5] transport protocol policy configuration 2016-12-01 20:25 ` [PATCH v7 0/4] transport protocol policy configuration Brandon Williams ` (3 preceding siblings ...) 2016-12-01 20:25 ` [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed Brandon Williams @ 2016-12-02 0:00 ` Brandon Williams 2016-12-02 0:00 ` [PATCH v8 1/5] lib-proto-disable: variable name fix Brandon Williams ` (5 more replies) 4 siblings, 6 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-02 0:00 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder v8 of this series moves the creation of an allowed protocol whitelist for CURLOPT_REDIR_PROTOCOLS to a helper function. This is to help out another series which depends on the creation of a whitelist for CURLOPT_PROTOCOLS. Brandon Williams (5): lib-proto-disable: variable name fix transport: add protocol policy config option http: always warn if libcurl version is too old http: create function to get curl allowed protocols transport: add from_user parameter to is_transport_allowed Documentation/config.txt | 46 +++++++++++++ Documentation/git.txt | 38 ++++------- git-submodule.sh | 12 ++-- http.c | 32 +++++---- t/lib-proto-disable.sh | 142 ++++++++++++++++++++++++++++++++++++--- t/t5509-fetch-push-namespaces.sh | 1 + t/t5802-connect-helper.sh | 1 + transport.c | 84 ++++++++++++++++++++--- transport.h | 19 +++--- 9 files changed, 302 insertions(+), 73 deletions(-) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v8 1/5] lib-proto-disable: variable name fix 2016-12-02 0:00 ` [PATCH v8 0/5] transport protocol policy configuration Brandon Williams @ 2016-12-02 0:00 ` Brandon Williams 2016-12-02 0:00 ` [PATCH v8 2/5] transport: add protocol policy config option Brandon Williams ` (4 subsequent siblings) 5 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-02 0:00 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder The test_proto function assigns the positional parameters to named variables, but then still refers to "$desc" as "$1". Using $desc is more readable and less error-prone. Signed-off-by: Brandon Williams <bmwill@google.com> --- t/lib-proto-disable.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index b0917d9..be88e9a 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -9,7 +9,7 @@ test_proto () { proto=$2 url=$3 - test_expect_success "clone $1 (enabled)" ' + test_expect_success "clone $desc (enabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=$proto && @@ -18,7 +18,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (enabled)" ' + test_expect_success "fetch $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -27,7 +27,7 @@ test_proto () { ) ' - test_expect_success "push $1 (enabled)" ' + test_expect_success "push $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -36,7 +36,7 @@ test_proto () { ) ' - test_expect_success "push $1 (disabled)" ' + test_expect_success "push $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -45,7 +45,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (disabled)" ' + test_expect_success "fetch $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -54,7 +54,7 @@ test_proto () { ) ' - test_expect_success "clone $1 (disabled)" ' + test_expect_success "clone $desc (disabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=none && -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v8 2/5] transport: add protocol policy config option 2016-12-02 0:00 ` [PATCH v8 0/5] transport protocol policy configuration Brandon Williams 2016-12-02 0:00 ` [PATCH v8 1/5] lib-proto-disable: variable name fix Brandon Williams @ 2016-12-02 0:00 ` Brandon Williams 2016-12-02 0:01 ` [PATCH v8 3/5] http: always warn if libcurl version is too old Brandon Williams ` (3 subsequent siblings) 5 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-02 0:00 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to specify a whitelist of protocols to be used in clone/fetch/push commands. This patch introduces new configuration options for more fine-grained control for allowing/disallowing protocols. This also has the added benefit of allowing easier construction of a protocol whitelist on systems where setting an environment variable is non-trivial. Now users can specify a policy to be used for each type of protocol via the 'protocol.<name>.allow' config option. A default policy for all unconfigured protocols can be set with the 'protocol.allow' config option. If no user configured default is made git will allow known-safe protocols (http, https, git, ssh, file), disallow known-dangerous protocols (ext), and have a default policy of `user` for all other protocols. The supported policies are `always`, `never`, and `user`. The `user` policy can be used to configure a protocol to be usable when explicitly used by a user, while disallowing it for commands which run clone/fetch/push commands without direct user intervention (e.g. recursive initialization of submodules). Commands which can potentially clone/fetch/push from untrusted repositories without user intervention can export `GIT_PROTOCOL_FROM_USER` with a value of '0' to prevent protocols configured to the `user` policy from being used. Fix remote-ext tests to use the new config to allow the ext protocol to be tested. Based on a patch by Jeff King <peff@peff.net> Signed-off-by: Brandon Williams <bmwill@google.com> --- Documentation/config.txt | 46 ++++++++++++++ Documentation/git.txt | 38 +++++------- git-submodule.sh | 12 ++-- t/lib-proto-disable.sh | 130 +++++++++++++++++++++++++++++++++++++-- t/t5509-fetch-push-namespaces.sh | 1 + t/t5802-connect-helper.sh | 1 + transport.c | 75 +++++++++++++++++++++- 7 files changed, 264 insertions(+), 39 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index 27069ac..5fe50bc 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -2308,6 +2308,52 @@ pretty.<name>:: Note that an alias with the same name as a built-in format will be silently ignored. +protocol.allow:: + If set, provide a user defined default policy for all protocols which + don't explicitly have a policy (`protocol.<name>.allow`). By default, + if unset, known-safe protocols (http, https, git, ssh, file) have a + default policy of `always`, known-dangerous protocols (ext) have a + default policy of `never`, and all other protocols have a default + policy of `user`. Supported policies: ++ +-- + +* `always` - protocol is always able to be used. + +* `never` - protocol is never able to be used. + +* `user` - protocol is only able to be used when `GIT_PROTOCOL_FROM_USER` is + either unset or has a value of 1. This policy should be used when you want a + protocol to be directly usable by the user but don't want it used by commands which + execute clone/fetch/push commands without user input, e.g. recursive + submodule initialization. + +-- + +protocol.<name>.allow:: + Set a policy to be used by protocol `<name>` with clone/fetch/push + commands. See `protocol.allow` above for the available policies. ++ +The protocol names currently used by git are: ++ +-- + - `file`: any local file-based path (including `file://` URLs, + or local paths) + + - `git`: the anonymous git protocol over a direct TCP + connection (or proxy, if configured) + + - `ssh`: git over ssh (including `host:path` syntax, + `ssh://`, etc). + + - `http`: git over http, both "smart http" and "dumb http". + Note that this does _not_ include `https`; if you want to configure + both, you must do so individually. + + - any external helpers are named by their protocol (e.g., use + `hg` to allow the `git-remote-hg` helper) +-- + pull.ff:: By default, Git does not create an extra merge commit when merging a commit that is a descendant of the current commit. Instead, the diff --git a/Documentation/git.txt b/Documentation/git.txt index ab7215e..c52cec8 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -1150,30 +1150,20 @@ of clones and fetches. cloning a repository to make a backup). `GIT_ALLOW_PROTOCOL`:: - If set, provide a colon-separated list of protocols which are - allowed to be used with fetch/push/clone. This is useful to - restrict recursive submodule initialization from an untrusted - repository. Any protocol not mentioned will be disallowed (i.e., - this is a whitelist, not a blacklist). If the variable is not - set at all, all protocols are enabled. The protocol names - currently used by git are: - - - `file`: any local file-based path (including `file://` URLs, - or local paths) - - - `git`: the anonymous git protocol over a direct TCP - connection (or proxy, if configured) - - - `ssh`: git over ssh (including `host:path` syntax, - `ssh://`, etc). - - - `http`: git over http, both "smart http" and "dumb http". - Note that this does _not_ include `https`; if you want both, - you should specify both as `http:https`. - - - any external helpers are named by their protocol (e.g., use - `hg` to allow the `git-remote-hg` helper) - + If set to a colon-separated list of protocols, behave as if + `protocol.allow` is set to `never`, and each of the listed + protocols has `protocol.<name>.allow` set to `always` + (overriding any existing configuration). In other words, any + protocol not mentioned will be disallowed (i.e., this is a + whitelist, not a blacklist). See the description of + `protocol.allow` in linkgit:git-config[1] for more details. + +`GIT_PROTOCOL_FROM_USER`:: + Set to 0 to prevent protocols used by fetch/push/clone which are + configured to the `user` state. This is useful to restrict recursive + submodule initialization from an untrusted repository or for programs + which feed potentially-untrusted URLS to git commands. See + linkgit:git-config[1] for more details. Discussion[[Discussion]] ------------------------ diff --git a/git-submodule.sh b/git-submodule.sh index a024a13..0a477b4 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -21,14 +21,10 @@ require_work_tree wt_prefix=$(git rev-parse --show-prefix) cd_to_toplevel -# Restrict ourselves to a vanilla subset of protocols; the URLs -# we get are under control of a remote repository, and we do not -# want them kicking off arbitrary git-remote-* programs. -# -# If the user has already specified a set of allowed protocols, -# we assume they know what they're doing and use that instead. -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} -export GIT_ALLOW_PROTOCOL +# Tell the rest of git that any URLs we get don't come +# directly from the user, so it can apply policy as appropriate. +GIT_PROTOCOL_FROM_USER=0 +export GIT_PROTOCOL_FROM_USER command= branch= diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index be88e9a..02f49cb 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -1,10 +1,7 @@ # Test routines for checking protocol disabling. -# test cloning a particular protocol -# $1 - description of the protocol -# $2 - machine-readable name of the protocol -# $3 - the URL to try cloning -test_proto () { +# Test clone/fetch/push with GIT_ALLOW_PROTOCOL whitelist +test_whitelist () { desc=$1 proto=$2 url=$3 @@ -62,6 +59,129 @@ test_proto () { test_must_fail git clone --bare "$url" tmp.git ) ' + + test_expect_success "clone $desc (env var has precedence)" ' + rm -rf tmp.git && + ( + GIT_ALLOW_PROTOCOL=none && + export GIT_ALLOW_PROTOCOL && + test_must_fail git -c protocol.allow=always clone --bare "$url" tmp.git && + test_must_fail git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ) + ' +} + +test_config () { + desc=$1 + proto=$2 + url=$3 + + # Test clone/fetch/push with protocol.<type>.allow config + test_expect_success "clone $desc (enabled with config)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git -c protocol.$proto.allow=never clone --bare "$url" tmp.git + ' + + # Test clone/fetch/push with protocol.user.allow and its env var + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user push origin HEAD:pushed + ) + ' + + test_expect_success "fetch $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user fetch + ) + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + ( + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ) + ' + + # Test clone/fetch/push with protocol.allow user defined default + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git config --global protocol.allow always && + git clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + git config --global protocol.allow never && + test_must_fail git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git clone --bare "$url" tmp.git + ' +} + +# test cloning a particular protocol +# $1 - description of the protocol +# $2 - machine-readable name of the protocol +# $3 - the URL to try cloning +test_proto () { + test_whitelist "$@" + + test_config "$@" } # set up an ssh wrapper that will access $host/$repo in the diff --git a/t/t5509-fetch-push-namespaces.sh b/t/t5509-fetch-push-namespaces.sh index bc44ac3..75c570a 100755 --- a/t/t5509-fetch-push-namespaces.sh +++ b/t/t5509-fetch-push-namespaces.sh @@ -4,6 +4,7 @@ test_description='fetch/push involving ref namespaces' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git init original && ( diff --git a/t/t5802-connect-helper.sh b/t/t5802-connect-helper.sh index b7a7f9d..c6c2661 100755 --- a/t/t5802-connect-helper.sh +++ b/t/t5802-connect-helper.sh @@ -4,6 +4,7 @@ test_description='ext::cmd remote "connect" helper' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git commit --allow-empty -m initial && test_tick && diff --git a/transport.c b/transport.c index d57e8de..2c0ec76 100644 --- a/transport.c +++ b/transport.c @@ -664,10 +664,81 @@ static const struct string_list *protocol_whitelist(void) return enabled ? &allowed : NULL; } +enum protocol_allow_config { + PROTOCOL_ALLOW_NEVER = 0, + PROTOCOL_ALLOW_USER_ONLY, + PROTOCOL_ALLOW_ALWAYS +}; + +static enum protocol_allow_config parse_protocol_config(const char *key, + const char *value) +{ + if (!strcasecmp(value, "always")) + return PROTOCOL_ALLOW_ALWAYS; + else if (!strcasecmp(value, "never")) + return PROTOCOL_ALLOW_NEVER; + else if (!strcasecmp(value, "user")) + return PROTOCOL_ALLOW_USER_ONLY; + + die("unknown value for config '%s': %s", key, value); +} + +static enum protocol_allow_config get_protocol_config(const char *type) +{ + char *key = xstrfmt("protocol.%s.allow", type); + char *value; + + /* first check the per-protocol config */ + if (!git_config_get_string(key, &value)) { + enum protocol_allow_config ret = + parse_protocol_config(key, value); + free(key); + free(value); + return ret; + } + free(key); + + /* if defined, fallback to user-defined default for unknown protocols */ + if (!git_config_get_string("protocol.allow", &value)) { + enum protocol_allow_config ret = + parse_protocol_config("protocol.allow", value); + free(value); + return ret; + } + + /* fallback to built-in defaults */ + /* known safe */ + if (!strcmp(type, "http") || + !strcmp(type, "https") || + !strcmp(type, "git") || + !strcmp(type, "ssh") || + !strcmp(type, "file")) + return PROTOCOL_ALLOW_ALWAYS; + + /* known scary; err on the side of caution */ + if (!strcmp(type, "ext")) + return PROTOCOL_ALLOW_NEVER; + + /* unknown; by default let them be used only directly by the user */ + return PROTOCOL_ALLOW_USER_ONLY; +} + int is_transport_allowed(const char *type) { - const struct string_list *allowed = protocol_whitelist(); - return !allowed || string_list_has_string(allowed, type); + const struct string_list *whitelist = protocol_whitelist(); + if (whitelist) + return string_list_has_string(whitelist, type); + + switch (get_protocol_config(type)) { + case PROTOCOL_ALLOW_ALWAYS: + return 1; + case PROTOCOL_ALLOW_NEVER: + return 0; + case PROTOCOL_ALLOW_USER_ONLY: + return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + } + + die("BUG: invalid protocol_allow_config type"); } void transport_check_allowed(const char *type) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v8 3/5] http: always warn if libcurl version is too old 2016-12-02 0:00 ` [PATCH v8 0/5] transport protocol policy configuration Brandon Williams 2016-12-02 0:00 ` [PATCH v8 1/5] lib-proto-disable: variable name fix Brandon Williams 2016-12-02 0:00 ` [PATCH v8 2/5] transport: add protocol policy config option Brandon Williams @ 2016-12-02 0:01 ` Brandon Williams 2016-12-02 0:01 ` [PATCH v8 4/5] http: create function to get curl allowed protocols Brandon Williams ` (2 subsequent siblings) 5 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-02 0:01 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder Now that there are default "known-good" and "known-bad" protocols which are allowed/disallowed by 'is_transport_allowed' we should always warn the user that older versions of libcurl can't respect the allowed protocols for redirects. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 5 ++--- transport.c | 5 ----- transport.h | 6 ------ 3 files changed, 2 insertions(+), 14 deletions(-) diff --git a/http.c b/http.c index 4c4a812..fee128b 100644 --- a/http.c +++ b/http.c @@ -735,9 +735,8 @@ static CURL *get_curl_handle(void) allowed_protocols |= CURLPROTO_FTPS; curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); #else - if (transport_restrict_protocols()) - warning("protocol restrictions not applied to curl redirects because\n" - "your curl version is too old (>= 7.19.4)"); + warning("protocol restrictions not applied to curl redirects because\n" + "your curl version is too old (>= 7.19.4)"); #endif if (getenv("GIT_CURL_VERBOSE")) curl_easy_setopt(result, CURLOPT_VERBOSE, 1L); diff --git a/transport.c b/transport.c index 2c0ec76..186de9a 100644 --- a/transport.c +++ b/transport.c @@ -747,11 +747,6 @@ void transport_check_allowed(const char *type) die("transport '%s' not allowed", type); } -int transport_restrict_protocols(void) -{ - return !!protocol_whitelist(); -} - struct transport *transport_get(struct remote *remote, const char *url) { const char *helper; diff --git a/transport.h b/transport.h index b8e4ee8..f4998bc 100644 --- a/transport.h +++ b/transport.h @@ -164,12 +164,6 @@ int is_transport_allowed(const char *type); */ void transport_check_allowed(const char *type); -/* - * Returns true if the user has attempted to turn on protocol - * restrictions at all. - */ -int transport_restrict_protocols(void); - /* Transport options which apply to git:// and scp-style URLs */ /* The program to use on the remote side to send a pack */ -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v8 4/5] http: create function to get curl allowed protocols 2016-12-02 0:00 ` [PATCH v8 0/5] transport protocol policy configuration Brandon Williams ` (2 preceding siblings ...) 2016-12-02 0:01 ` [PATCH v8 3/5] http: always warn if libcurl version is too old Brandon Williams @ 2016-12-02 0:01 ` Brandon Williams 2016-12-02 0:01 ` [PATCH v8 5/5] transport: add from_user parameter to is_transport_allowed Brandon Williams 2016-12-14 1:40 ` [PATCH v9 0/5] transport protocol policy configuration Brandon Williams 5 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-02 0:01 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder Move the creation of an allowed protocols whitelist to a helper function. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/http.c b/http.c index fee128b..a1c3a0e 100644 --- a/http.c +++ b/http.c @@ -624,11 +624,25 @@ void setup_curl_trace(CURL *handle) curl_easy_setopt(handle, CURLOPT_DEBUGDATA, NULL); } +static long get_curl_allowed_protocols(void) +{ + long allowed_protocols = 0; + + if (is_transport_allowed("http")) + allowed_protocols |= CURLPROTO_HTTP; + if (is_transport_allowed("https")) + allowed_protocols |= CURLPROTO_HTTPS; + if (is_transport_allowed("ftp")) + allowed_protocols |= CURLPROTO_FTP; + if (is_transport_allowed("ftps")) + allowed_protocols |= CURLPROTO_FTPS; + + return allowed_protocols; +} static CURL *get_curl_handle(void) { CURL *result = curl_easy_init(); - long allowed_protocols = 0; if (!result) die("curl_easy_init failed"); @@ -725,15 +739,8 @@ static CURL *get_curl_handle(void) curl_easy_setopt(result, CURLOPT_POST301, 1); #endif #if LIBCURL_VERSION_NUM >= 0x071304 - if (is_transport_allowed("http")) - allowed_protocols |= CURLPROTO_HTTP; - if (is_transport_allowed("https")) - allowed_protocols |= CURLPROTO_HTTPS; - if (is_transport_allowed("ftp")) - allowed_protocols |= CURLPROTO_FTP; - if (is_transport_allowed("ftps")) - allowed_protocols |= CURLPROTO_FTPS; - curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); + curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, + get_curl_allowed_protocols()); #else warning("protocol restrictions not applied to curl redirects because\n" "your curl version is too old (>= 7.19.4)"); -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v8 5/5] transport: add from_user parameter to is_transport_allowed 2016-12-02 0:00 ` [PATCH v8 0/5] transport protocol policy configuration Brandon Williams ` (3 preceding siblings ...) 2016-12-02 0:01 ` [PATCH v8 4/5] http: create function to get curl allowed protocols Brandon Williams @ 2016-12-02 0:01 ` Brandon Williams 2016-12-14 1:40 ` [PATCH v9 0/5] transport protocol policy configuration Brandon Williams 5 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-02 0:01 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, jrnieder Add the from_user parameter to the 'is_transport_allowed' function. This allows callers to query if a transport protocol is allowed, given that the caller knows that the protocol is coming from the user (1) or not from the user (0) such as redirects in libcurl. If unknown a -1 should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER` to determine if the protocol came from the user. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 12 ++++++------ transport.c | 8 +++++--- transport.h | 13 ++++++++++--- 3 files changed, 21 insertions(+), 12 deletions(-) diff --git a/http.c b/http.c index a1c3a0e..2a02941 100644 --- a/http.c +++ b/http.c @@ -624,17 +624,17 @@ void setup_curl_trace(CURL *handle) curl_easy_setopt(handle, CURLOPT_DEBUGDATA, NULL); } -static long get_curl_allowed_protocols(void) +static long get_curl_allowed_protocols(int from_user) { long allowed_protocols = 0; - if (is_transport_allowed("http")) + if (is_transport_allowed("http", from_user)) allowed_protocols |= CURLPROTO_HTTP; - if (is_transport_allowed("https")) + if (is_transport_allowed("https", from_user)) allowed_protocols |= CURLPROTO_HTTPS; - if (is_transport_allowed("ftp")) + if (is_transport_allowed("ftp", from_user)) allowed_protocols |= CURLPROTO_FTP; - if (is_transport_allowed("ftps")) + if (is_transport_allowed("ftps", from_user)) allowed_protocols |= CURLPROTO_FTPS; return allowed_protocols; @@ -740,7 +740,7 @@ static CURL *get_curl_handle(void) #endif #if LIBCURL_VERSION_NUM >= 0x071304 curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, - get_curl_allowed_protocols()); + get_curl_allowed_protocols(0)); #else warning("protocol restrictions not applied to curl redirects because\n" "your curl version is too old (>= 7.19.4)"); diff --git a/transport.c b/transport.c index 186de9a..8a3597b 100644 --- a/transport.c +++ b/transport.c @@ -723,7 +723,7 @@ static enum protocol_allow_config get_protocol_config(const char *type) return PROTOCOL_ALLOW_USER_ONLY; } -int is_transport_allowed(const char *type) +int is_transport_allowed(const char *type, int from_user) { const struct string_list *whitelist = protocol_whitelist(); if (whitelist) @@ -735,7 +735,9 @@ int is_transport_allowed(const char *type) case PROTOCOL_ALLOW_NEVER: return 0; case PROTOCOL_ALLOW_USER_ONLY: - return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + if (from_user < 0) + from_user = git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + return from_user; } die("BUG: invalid protocol_allow_config type"); @@ -743,7 +745,7 @@ int is_transport_allowed(const char *type) void transport_check_allowed(const char *type) { - if (!is_transport_allowed(type)) + if (!is_transport_allowed(type, -1)) die("transport '%s' not allowed", type); } diff --git a/transport.h b/transport.h index f4998bc..9820f10 100644 --- a/transport.h +++ b/transport.h @@ -153,10 +153,17 @@ extern int transport_summary_width(const struct ref *refs); struct transport *transport_get(struct remote *, const char *); /* - * Check whether a transport is allowed by the environment. Type should - * generally be the URL scheme, as described in Documentation/git.txt + * Check whether a transport is allowed by the environment. + * + * Type should generally be the URL scheme, as described in + * Documentation/git.txt + * + * from_user specifies if the transport was given by the user. If unknown pass + * a -1 to read from the environment to determine if the transport was given by + * the user. + * */ -int is_transport_allowed(const char *type); +int is_transport_allowed(const char *type, int from_user); /* * Check whether a transport is allowed by the environment, -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v9 0/5] transport protocol policy configuration 2016-12-02 0:00 ` [PATCH v8 0/5] transport protocol policy configuration Brandon Williams ` (4 preceding siblings ...) 2016-12-02 0:01 ` [PATCH v8 5/5] transport: add from_user parameter to is_transport_allowed Brandon Williams @ 2016-12-14 1:40 ` Brandon Williams 2016-12-14 1:40 ` [PATCH v9 1/5] lib-proto-disable: variable name fix Brandon Williams ` (5 more replies) 5 siblings, 6 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 1:40 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, gitster, jrnieder Only difference between v8 and v9 is that v9 has been rebased ontop of Jeff's http-walker-limit-redirect series 'jk/http-walker-limit-redirect'. Brandon Williams (5): lib-proto-disable: variable name fix transport: add protocol policy config option http: always warn if libcurl version is too old http: create function to get curl allowed protocols transport: add from_user parameter to is_transport_allowed Documentation/config.txt | 46 +++++++++++++ Documentation/git.txt | 38 ++++------- git-submodule.sh | 12 ++-- http.c | 36 ++++++---- t/lib-proto-disable.sh | 142 ++++++++++++++++++++++++++++++++++++--- t/t5509-fetch-push-namespaces.sh | 1 + t/t5802-connect-helper.sh | 1 + transport.c | 84 ++++++++++++++++++++--- transport.h | 19 +++--- 9 files changed, 305 insertions(+), 74 deletions(-) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v9 1/5] lib-proto-disable: variable name fix 2016-12-14 1:40 ` [PATCH v9 0/5] transport protocol policy configuration Brandon Williams @ 2016-12-14 1:40 ` Brandon Williams 2016-12-14 1:40 ` [PATCH v9 2/5] transport: add protocol policy config option Brandon Williams ` (4 subsequent siblings) 5 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 1:40 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, gitster, jrnieder The test_proto function assigns the positional parameters to named variables, but then still refers to "$desc" as "$1". Using $desc is more readable and less error-prone. Signed-off-by: Brandon Williams <bmwill@google.com> --- t/lib-proto-disable.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index b0917d9..be88e9a 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -9,7 +9,7 @@ test_proto () { proto=$2 url=$3 - test_expect_success "clone $1 (enabled)" ' + test_expect_success "clone $desc (enabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=$proto && @@ -18,7 +18,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (enabled)" ' + test_expect_success "fetch $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -27,7 +27,7 @@ test_proto () { ) ' - test_expect_success "push $1 (enabled)" ' + test_expect_success "push $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -36,7 +36,7 @@ test_proto () { ) ' - test_expect_success "push $1 (disabled)" ' + test_expect_success "push $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -45,7 +45,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (disabled)" ' + test_expect_success "fetch $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -54,7 +54,7 @@ test_proto () { ) ' - test_expect_success "clone $1 (disabled)" ' + test_expect_success "clone $desc (disabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=none && -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v9 2/5] transport: add protocol policy config option 2016-12-14 1:40 ` [PATCH v9 0/5] transport protocol policy configuration Brandon Williams 2016-12-14 1:40 ` [PATCH v9 1/5] lib-proto-disable: variable name fix Brandon Williams @ 2016-12-14 1:40 ` Brandon Williams 2016-12-14 1:40 ` [PATCH v9 3/5] http: always warn if libcurl version is too old Brandon Williams ` (3 subsequent siblings) 5 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 1:40 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, gitster, jrnieder Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to specify a whitelist of protocols to be used in clone/fetch/push commands. This patch introduces new configuration options for more fine-grained control for allowing/disallowing protocols. This also has the added benefit of allowing easier construction of a protocol whitelist on systems where setting an environment variable is non-trivial. Now users can specify a policy to be used for each type of protocol via the 'protocol.<name>.allow' config option. A default policy for all unconfigured protocols can be set with the 'protocol.allow' config option. If no user configured default is made git will allow known-safe protocols (http, https, git, ssh, file), disallow known-dangerous protocols (ext), and have a default policy of `user` for all other protocols. The supported policies are `always`, `never`, and `user`. The `user` policy can be used to configure a protocol to be usable when explicitly used by a user, while disallowing it for commands which run clone/fetch/push commands without direct user intervention (e.g. recursive initialization of submodules). Commands which can potentially clone/fetch/push from untrusted repositories without user intervention can export `GIT_PROTOCOL_FROM_USER` with a value of '0' to prevent protocols configured to the `user` policy from being used. Fix remote-ext tests to use the new config to allow the ext protocol to be tested. Based on a patch by Jeff King <peff@peff.net> Signed-off-by: Brandon Williams <bmwill@google.com> --- Documentation/config.txt | 46 ++++++++++++++ Documentation/git.txt | 38 +++++------- git-submodule.sh | 12 ++-- t/lib-proto-disable.sh | 130 +++++++++++++++++++++++++++++++++++++-- t/t5509-fetch-push-namespaces.sh | 1 + t/t5802-connect-helper.sh | 1 + transport.c | 75 +++++++++++++++++++++- 7 files changed, 264 insertions(+), 39 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index 8153336..50d3d06 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -2260,6 +2260,52 @@ pretty.<name>:: Note that an alias with the same name as a built-in format will be silently ignored. +protocol.allow:: + If set, provide a user defined default policy for all protocols which + don't explicitly have a policy (`protocol.<name>.allow`). By default, + if unset, known-safe protocols (http, https, git, ssh, file) have a + default policy of `always`, known-dangerous protocols (ext) have a + default policy of `never`, and all other protocols have a default + policy of `user`. Supported policies: ++ +-- + +* `always` - protocol is always able to be used. + +* `never` - protocol is never able to be used. + +* `user` - protocol is only able to be used when `GIT_PROTOCOL_FROM_USER` is + either unset or has a value of 1. This policy should be used when you want a + protocol to be directly usable by the user but don't want it used by commands which + execute clone/fetch/push commands without user input, e.g. recursive + submodule initialization. + +-- + +protocol.<name>.allow:: + Set a policy to be used by protocol `<name>` with clone/fetch/push + commands. See `protocol.allow` above for the available policies. ++ +The protocol names currently used by git are: ++ +-- + - `file`: any local file-based path (including `file://` URLs, + or local paths) + + - `git`: the anonymous git protocol over a direct TCP + connection (or proxy, if configured) + + - `ssh`: git over ssh (including `host:path` syntax, + `ssh://`, etc). + + - `http`: git over http, both "smart http" and "dumb http". + Note that this does _not_ include `https`; if you want to configure + both, you must do so individually. + + - any external helpers are named by their protocol (e.g., use + `hg` to allow the `git-remote-hg` helper) +-- + pull.ff:: By default, Git does not create an extra merge commit when merging a commit that is a descendant of the current commit. Instead, the diff --git a/Documentation/git.txt b/Documentation/git.txt index 923aa49..d9fb937 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -1129,30 +1129,20 @@ of clones and fetches. cloning a repository to make a backup). `GIT_ALLOW_PROTOCOL`:: - If set, provide a colon-separated list of protocols which are - allowed to be used with fetch/push/clone. This is useful to - restrict recursive submodule initialization from an untrusted - repository. Any protocol not mentioned will be disallowed (i.e., - this is a whitelist, not a blacklist). If the variable is not - set at all, all protocols are enabled. The protocol names - currently used by git are: - - - `file`: any local file-based path (including `file://` URLs, - or local paths) - - - `git`: the anonymous git protocol over a direct TCP - connection (or proxy, if configured) - - - `ssh`: git over ssh (including `host:path` syntax, - `ssh://`, etc). - - - `http`: git over http, both "smart http" and "dumb http". - Note that this does _not_ include `https`; if you want both, - you should specify both as `http:https`. - - - any external helpers are named by their protocol (e.g., use - `hg` to allow the `git-remote-hg` helper) - + If set to a colon-separated list of protocols, behave as if + `protocol.allow` is set to `never`, and each of the listed + protocols has `protocol.<name>.allow` set to `always` + (overriding any existing configuration). In other words, any + protocol not mentioned will be disallowed (i.e., this is a + whitelist, not a blacklist). See the description of + `protocol.allow` in linkgit:git-config[1] for more details. + +`GIT_PROTOCOL_FROM_USER`:: + Set to 0 to prevent protocols used by fetch/push/clone which are + configured to the `user` state. This is useful to restrict recursive + submodule initialization from an untrusted repository or for programs + which feed potentially-untrusted URLS to git commands. See + linkgit:git-config[1] for more details. Discussion[[Discussion]] ------------------------ diff --git a/git-submodule.sh b/git-submodule.sh index 78fdac9..fc44076 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -22,14 +22,10 @@ require_work_tree wt_prefix=$(git rev-parse --show-prefix) cd_to_toplevel -# Restrict ourselves to a vanilla subset of protocols; the URLs -# we get are under control of a remote repository, and we do not -# want them kicking off arbitrary git-remote-* programs. -# -# If the user has already specified a set of allowed protocols, -# we assume they know what they're doing and use that instead. -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} -export GIT_ALLOW_PROTOCOL +# Tell the rest of git that any URLs we get don't come +# directly from the user, so it can apply policy as appropriate. +GIT_PROTOCOL_FROM_USER=0 +export GIT_PROTOCOL_FROM_USER command= branch= diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index be88e9a..02f49cb 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -1,10 +1,7 @@ # Test routines for checking protocol disabling. -# test cloning a particular protocol -# $1 - description of the protocol -# $2 - machine-readable name of the protocol -# $3 - the URL to try cloning -test_proto () { +# Test clone/fetch/push with GIT_ALLOW_PROTOCOL whitelist +test_whitelist () { desc=$1 proto=$2 url=$3 @@ -62,6 +59,129 @@ test_proto () { test_must_fail git clone --bare "$url" tmp.git ) ' + + test_expect_success "clone $desc (env var has precedence)" ' + rm -rf tmp.git && + ( + GIT_ALLOW_PROTOCOL=none && + export GIT_ALLOW_PROTOCOL && + test_must_fail git -c protocol.allow=always clone --bare "$url" tmp.git && + test_must_fail git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ) + ' +} + +test_config () { + desc=$1 + proto=$2 + url=$3 + + # Test clone/fetch/push with protocol.<type>.allow config + test_expect_success "clone $desc (enabled with config)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git -c protocol.$proto.allow=never clone --bare "$url" tmp.git + ' + + # Test clone/fetch/push with protocol.user.allow and its env var + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user push origin HEAD:pushed + ) + ' + + test_expect_success "fetch $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user fetch + ) + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + ( + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ) + ' + + # Test clone/fetch/push with protocol.allow user defined default + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git config --global protocol.allow always && + git clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + git config --global protocol.allow never && + test_must_fail git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git clone --bare "$url" tmp.git + ' +} + +# test cloning a particular protocol +# $1 - description of the protocol +# $2 - machine-readable name of the protocol +# $3 - the URL to try cloning +test_proto () { + test_whitelist "$@" + + test_config "$@" } # set up an ssh wrapper that will access $host/$repo in the diff --git a/t/t5509-fetch-push-namespaces.sh b/t/t5509-fetch-push-namespaces.sh index bc44ac3..75c570a 100755 --- a/t/t5509-fetch-push-namespaces.sh +++ b/t/t5509-fetch-push-namespaces.sh @@ -4,6 +4,7 @@ test_description='fetch/push involving ref namespaces' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git init original && ( diff --git a/t/t5802-connect-helper.sh b/t/t5802-connect-helper.sh index b7a7f9d..c6c2661 100755 --- a/t/t5802-connect-helper.sh +++ b/t/t5802-connect-helper.sh @@ -4,6 +4,7 @@ test_description='ext::cmd remote "connect" helper' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git commit --allow-empty -m initial && test_tick && diff --git a/transport.c b/transport.c index 41eb82c..e1ba78b 100644 --- a/transport.c +++ b/transport.c @@ -617,10 +617,81 @@ static const struct string_list *protocol_whitelist(void) return enabled ? &allowed : NULL; } +enum protocol_allow_config { + PROTOCOL_ALLOW_NEVER = 0, + PROTOCOL_ALLOW_USER_ONLY, + PROTOCOL_ALLOW_ALWAYS +}; + +static enum protocol_allow_config parse_protocol_config(const char *key, + const char *value) +{ + if (!strcasecmp(value, "always")) + return PROTOCOL_ALLOW_ALWAYS; + else if (!strcasecmp(value, "never")) + return PROTOCOL_ALLOW_NEVER; + else if (!strcasecmp(value, "user")) + return PROTOCOL_ALLOW_USER_ONLY; + + die("unknown value for config '%s': %s", key, value); +} + +static enum protocol_allow_config get_protocol_config(const char *type) +{ + char *key = xstrfmt("protocol.%s.allow", type); + char *value; + + /* first check the per-protocol config */ + if (!git_config_get_string(key, &value)) { + enum protocol_allow_config ret = + parse_protocol_config(key, value); + free(key); + free(value); + return ret; + } + free(key); + + /* if defined, fallback to user-defined default for unknown protocols */ + if (!git_config_get_string("protocol.allow", &value)) { + enum protocol_allow_config ret = + parse_protocol_config("protocol.allow", value); + free(value); + return ret; + } + + /* fallback to built-in defaults */ + /* known safe */ + if (!strcmp(type, "http") || + !strcmp(type, "https") || + !strcmp(type, "git") || + !strcmp(type, "ssh") || + !strcmp(type, "file")) + return PROTOCOL_ALLOW_ALWAYS; + + /* known scary; err on the side of caution */ + if (!strcmp(type, "ext")) + return PROTOCOL_ALLOW_NEVER; + + /* unknown; by default let them be used only directly by the user */ + return PROTOCOL_ALLOW_USER_ONLY; +} + int is_transport_allowed(const char *type) { - const struct string_list *allowed = protocol_whitelist(); - return !allowed || string_list_has_string(allowed, type); + const struct string_list *whitelist = protocol_whitelist(); + if (whitelist) + return string_list_has_string(whitelist, type); + + switch (get_protocol_config(type)) { + case PROTOCOL_ALLOW_ALWAYS: + return 1; + case PROTOCOL_ALLOW_NEVER: + return 0; + case PROTOCOL_ALLOW_USER_ONLY: + return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + } + + die("BUG: invalid protocol_allow_config type"); } void transport_check_allowed(const char *type) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v9 3/5] http: always warn if libcurl version is too old 2016-12-14 1:40 ` [PATCH v9 0/5] transport protocol policy configuration Brandon Williams 2016-12-14 1:40 ` [PATCH v9 1/5] lib-proto-disable: variable name fix Brandon Williams 2016-12-14 1:40 ` [PATCH v9 2/5] transport: add protocol policy config option Brandon Williams @ 2016-12-14 1:40 ` Brandon Williams 2016-12-14 16:01 ` Jeff King 2016-12-14 1:40 ` [PATCH v9 4/5] http: create function to get curl allowed protocols Brandon Williams ` (2 subsequent siblings) 5 siblings, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-12-14 1:40 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, gitster, jrnieder Now that there are default "known-good" and "known-bad" protocols which are allowed/disallowed by 'is_transport_allowed' we should always warn the user that older versions of libcurl can't respect the allowed protocols for redirects. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 5 ++--- transport.c | 5 ----- transport.h | 6 ------ 3 files changed, 2 insertions(+), 14 deletions(-) diff --git a/http.c b/http.c index 5cd3ffd..034426e 100644 --- a/http.c +++ b/http.c @@ -583,9 +583,8 @@ static CURL *get_curl_handle(void) curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); curl_easy_setopt(result, CURLOPT_PROTOCOLS, allowed_protocols); #else - if (transport_restrict_protocols()) - warning("protocol restrictions not applied to curl redirects because\n" - "your curl version is too old (>= 7.19.4)"); + warning("protocol restrictions not applied to curl redirects because\n" + "your curl version is too old (>= 7.19.4)"); #endif if (getenv("GIT_CURL_VERBOSE")) diff --git a/transport.c b/transport.c index e1ba78b..fbd799d 100644 --- a/transport.c +++ b/transport.c @@ -700,11 +700,6 @@ void transport_check_allowed(const char *type) die("transport '%s' not allowed", type); } -int transport_restrict_protocols(void) -{ - return !!protocol_whitelist(); -} - struct transport *transport_get(struct remote *remote, const char *url) { const char *helper; diff --git a/transport.h b/transport.h index c681408..3396e1d 100644 --- a/transport.h +++ b/transport.h @@ -153,12 +153,6 @@ int is_transport_allowed(const char *type); */ void transport_check_allowed(const char *type); -/* - * Returns true if the user has attempted to turn on protocol - * restrictions at all. - */ -int transport_restrict_protocols(void); - /* Transport options which apply to git:// and scp-style URLs */ /* The program to use on the remote side to send a pack */ -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v9 3/5] http: always warn if libcurl version is too old 2016-12-14 1:40 ` [PATCH v9 3/5] http: always warn if libcurl version is too old Brandon Williams @ 2016-12-14 16:01 ` Jeff King 2016-12-14 17:56 ` Brandon Williams 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-12-14 16:01 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, gitster, jrnieder On Tue, Dec 13, 2016 at 05:40:35PM -0800, Brandon Williams wrote: > diff --git a/transport.c b/transport.c > index e1ba78b..fbd799d 100644 > --- a/transport.c > +++ b/transport.c > @@ -700,11 +700,6 @@ void transport_check_allowed(const char *type) > die("transport '%s' not allowed", type); > } > > -int transport_restrict_protocols(void) > -{ > - return !!protocol_whitelist(); > -} > - This function was subtly broken as of patch 2 of the series. It's probably not a big deal in the long run, but should the series be re-ordered to put this one first? I think the commit message would need adjusted, but it probably should mention the reasons this is a good idea even _without_ the new config system. Namely that even when there's no protocol whitelist, newer versions of curl have all of the other non-http protocols disabled. I wonder if anybody is actually using a version of curl old enough to trigger this. If so, they're going to get the warning every time they fetch via http. We might need to stick it behind an "advice.*" config option, though I'm inclined to leave it as-is and see if anybody actually complains. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v9 3/5] http: always warn if libcurl version is too old 2016-12-14 16:01 ` Jeff King @ 2016-12-14 17:56 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 17:56 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, gitster, jrnieder On 12/14, Jeff King wrote: > On Tue, Dec 13, 2016 at 05:40:35PM -0800, Brandon Williams wrote: > > > diff --git a/transport.c b/transport.c > > index e1ba78b..fbd799d 100644 > > --- a/transport.c > > +++ b/transport.c > > @@ -700,11 +700,6 @@ void transport_check_allowed(const char *type) > > die("transport '%s' not allowed", type); > > } > > > > -int transport_restrict_protocols(void) > > -{ > > - return !!protocol_whitelist(); > > -} > > - > > This function was subtly broken as of patch 2 of the series. It's > probably not a big deal in the long run, but should the series be > re-ordered to put this one first? > > I think the commit message would need adjusted, but it probably should > mention the reasons this is a good idea even _without_ the new config > system. Namely that even when there's no protocol whitelist, newer > versions of curl have all of the other non-http protocols disabled. > > I wonder if anybody is actually using a version of curl old enough to > trigger this. If so, they're going to get the warning every time they > fetch via http. We might need to stick it behind an "advice.*" config > option, though I'm inclined to leave it as-is and see if anybody > actually complains. > > -Peff Yeah you're right, transport_restrict_protocols() is definitely broken after patch 2. Since I'm probably going to need to do a reroll based on some of your comments in the other patches in the series we might as well reorder patch 2 and 3 so this isn't broken between patches. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v9 4/5] http: create function to get curl allowed protocols 2016-12-14 1:40 ` [PATCH v9 0/5] transport protocol policy configuration Brandon Williams ` (2 preceding siblings ...) 2016-12-14 1:40 ` [PATCH v9 3/5] http: always warn if libcurl version is too old Brandon Williams @ 2016-12-14 1:40 ` Brandon Williams 2016-12-14 16:03 ` Jeff King 2016-12-14 1:40 ` [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed Brandon Williams 2016-12-14 22:39 ` [PATCH v10 0/6] transport protocol policy configuration Brandon Williams 5 siblings, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-12-14 1:40 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, gitster, jrnieder Move the creation of an allowed protocols whitelist to a helper function. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/http.c b/http.c index 034426e..f7c488a 100644 --- a/http.c +++ b/http.c @@ -489,10 +489,25 @@ static void set_curl_keepalive(CURL *c) } #endif +static long get_curl_allowed_protocols(void) +{ + long allowed_protocols = 0; + + if (is_transport_allowed("http")) + allowed_protocols |= CURLPROTO_HTTP; + if (is_transport_allowed("https")) + allowed_protocols |= CURLPROTO_HTTPS; + if (is_transport_allowed("ftp")) + allowed_protocols |= CURLPROTO_FTP; + if (is_transport_allowed("ftps")) + allowed_protocols |= CURLPROTO_FTPS; + + return allowed_protocols; +} + static CURL *get_curl_handle(void) { CURL *result = curl_easy_init(); - long allowed_protocols = 0; if (!result) die("curl_easy_init failed"); @@ -572,16 +587,10 @@ static CURL *get_curl_handle(void) curl_easy_setopt(result, CURLOPT_POST301, 1); #endif #if LIBCURL_VERSION_NUM >= 0x071304 - if (is_transport_allowed("http")) - allowed_protocols |= CURLPROTO_HTTP; - if (is_transport_allowed("https")) - allowed_protocols |= CURLPROTO_HTTPS; - if (is_transport_allowed("ftp")) - allowed_protocols |= CURLPROTO_FTP; - if (is_transport_allowed("ftps")) - allowed_protocols |= CURLPROTO_FTPS; - curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); - curl_easy_setopt(result, CURLOPT_PROTOCOLS, allowed_protocols); + curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, + get_curl_allowed_protocols()); + curl_easy_setopt(result, CURLOPT_PROTOCOLS, + get_curl_allowed_protocols()); #else warning("protocol restrictions not applied to curl redirects because\n" "your curl version is too old (>= 7.19.4)"); -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v9 4/5] http: create function to get curl allowed protocols 2016-12-14 1:40 ` [PATCH v9 4/5] http: create function to get curl allowed protocols Brandon Williams @ 2016-12-14 16:03 ` Jeff King 2016-12-14 18:00 ` Brandon Williams 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-12-14 16:03 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, gitster, jrnieder On Tue, Dec 13, 2016 at 05:40:36PM -0800, Brandon Williams wrote: > Move the creation of an allowed protocols whitelist to a helper > function. This is "what" but not "why". You can figure it out if you see the next patch, but it's often nice to make a brief mention, like: This will be useful when we need to compute the set of allowed protocols differently for normal and redirect cases. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v9 4/5] http: create function to get curl allowed protocols 2016-12-14 16:03 ` Jeff King @ 2016-12-14 18:00 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 18:00 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, gitster, jrnieder On 12/14, Jeff King wrote: > On Tue, Dec 13, 2016 at 05:40:36PM -0800, Brandon Williams wrote: > > > Move the creation of an allowed protocols whitelist to a helper > > function. > > This is "what" but not "why". You can figure it out if you see the next > patch, but it's often nice to make a brief mention, like: > > This will be useful when we need to compute the set of allowed > protocols differently for normal and redirect cases. Commit message writing is hard (at least for me :). I'll update the message to indicate the why. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed 2016-12-14 1:40 ` [PATCH v9 0/5] transport protocol policy configuration Brandon Williams ` (3 preceding siblings ...) 2016-12-14 1:40 ` [PATCH v9 4/5] http: create function to get curl allowed protocols Brandon Williams @ 2016-12-14 1:40 ` Brandon Williams 2016-12-14 16:40 ` Jeff King 2016-12-14 22:39 ` [PATCH v10 0/6] transport protocol policy configuration Brandon Williams 5 siblings, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-12-14 1:40 UTC (permalink / raw) To: git; +Cc: Brandon Williams, peff, sbeller, bburky, gitster, jrnieder Add the from_user parameter to the 'is_transport_allowed' function. This allows callers to query if a transport protocol is allowed, given that the caller knows that the protocol is coming from the user (1) or not from the user (0) such as redirects in libcurl. If unknown a -1 should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER` to determine if the protocol came from the user. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 14 +++++++------- transport.c | 8 +++++--- transport.h | 13 ++++++++++--- 3 files changed, 22 insertions(+), 13 deletions(-) diff --git a/http.c b/http.c index f7c488a..2208269 100644 --- a/http.c +++ b/http.c @@ -489,17 +489,17 @@ static void set_curl_keepalive(CURL *c) } #endif -static long get_curl_allowed_protocols(void) +static long get_curl_allowed_protocols(int from_user) { long allowed_protocols = 0; - if (is_transport_allowed("http")) + if (is_transport_allowed("http", from_user)) allowed_protocols |= CURLPROTO_HTTP; - if (is_transport_allowed("https")) + if (is_transport_allowed("https", from_user)) allowed_protocols |= CURLPROTO_HTTPS; - if (is_transport_allowed("ftp")) + if (is_transport_allowed("ftp", from_user)) allowed_protocols |= CURLPROTO_FTP; - if (is_transport_allowed("ftps")) + if (is_transport_allowed("ftps", from_user)) allowed_protocols |= CURLPROTO_FTPS; return allowed_protocols; @@ -588,9 +588,9 @@ static CURL *get_curl_handle(void) #endif #if LIBCURL_VERSION_NUM >= 0x071304 curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, - get_curl_allowed_protocols()); + get_curl_allowed_protocols(0)); curl_easy_setopt(result, CURLOPT_PROTOCOLS, - get_curl_allowed_protocols()); + get_curl_allowed_protocols(-1)); #else warning("protocol restrictions not applied to curl redirects because\n" "your curl version is too old (>= 7.19.4)"); diff --git a/transport.c b/transport.c index fbd799d..f50c31a 100644 --- a/transport.c +++ b/transport.c @@ -676,7 +676,7 @@ static enum protocol_allow_config get_protocol_config(const char *type) return PROTOCOL_ALLOW_USER_ONLY; } -int is_transport_allowed(const char *type) +int is_transport_allowed(const char *type, int from_user) { const struct string_list *whitelist = protocol_whitelist(); if (whitelist) @@ -688,7 +688,9 @@ int is_transport_allowed(const char *type) case PROTOCOL_ALLOW_NEVER: return 0; case PROTOCOL_ALLOW_USER_ONLY: - return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + if (from_user < 0) + from_user = git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + return from_user; } die("BUG: invalid protocol_allow_config type"); @@ -696,7 +698,7 @@ int is_transport_allowed(const char *type) void transport_check_allowed(const char *type) { - if (!is_transport_allowed(type)) + if (!is_transport_allowed(type, -1)) die("transport '%s' not allowed", type); } diff --git a/transport.h b/transport.h index 3396e1d..4f1c801 100644 --- a/transport.h +++ b/transport.h @@ -142,10 +142,17 @@ struct transport { struct transport *transport_get(struct remote *, const char *); /* - * Check whether a transport is allowed by the environment. Type should - * generally be the URL scheme, as described in Documentation/git.txt + * Check whether a transport is allowed by the environment. + * + * Type should generally be the URL scheme, as described in + * Documentation/git.txt + * + * from_user specifies if the transport was given by the user. If unknown pass + * a -1 to read from the environment to determine if the transport was given by + * the user. + * */ -int is_transport_allowed(const char *type); +int is_transport_allowed(const char *type, int from_user); /* * Check whether a transport is allowed by the environment, -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed 2016-12-14 1:40 ` [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed Brandon Williams @ 2016-12-14 16:40 ` Jeff King 2016-12-14 20:13 ` Brandon Williams 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-12-14 16:40 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, gitster, jrnieder On Tue, Dec 13, 2016 at 05:40:37PM -0800, Brandon Williams wrote: > Add the from_user parameter to the 'is_transport_allowed' function. > This allows callers to query if a transport protocol is allowed, given > that the caller knows that the protocol is coming from the user (1) or > not from the user (0) such as redirects in libcurl. If unknown a -1 > should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER` > to determine if the protocol came from the user. I think your commit message is upside-down with respect to the purpose of the patch. The end goal we want is for http to distinguish between protocol restrictions for redirects versus initial requests. The rest is an implementation detail. It's definitely still worth discussing that implementation detail (though I think your in-code comments may be sufficient), but I don't see the rationale discussed here at all. > Signed-off-by: Brandon Williams <bmwill@google.com> > --- > http.c | 14 +++++++------- > transport.c | 8 +++++--- > transport.h | 13 ++++++++++--- > 3 files changed, 22 insertions(+), 13 deletions(-) I'm trying to think of a way to test this. I guess the case we are covering here is when a server redirects, but the protocol is only allowed from the user. So: diff --git a/t/t5812-proto-disable-http.sh b/t/t5812-proto-disable-http.sh index 044cc152f..d911afd24 100755 --- a/t/t5812-proto-disable-http.sh +++ b/t/t5812-proto-disable-http.sh @@ -30,5 +30,12 @@ test_expect_success 'curl limits redirects' ' test_must_fail git clone "$HTTPD_URL/loop-redir/smart/repo.git" ' +test_expect_success 'http can be limited to from-user' ' + git -c protocol.http.allow=user \ + clone "$HTTPD_URL/smart/repo.git" plain.git && + test_must_fail git -c protocol.http.allow=user \ + clone "$HTTPD_URL/smart-redir-perm/repo.git" redir.git +' + stop_httpd test_done It's an oddball configuration, and you'd probably just set http.followRedirects=false in practice, but it does correctly check this case. > @@ -588,9 +588,9 @@ static CURL *get_curl_handle(void) > #endif > #if LIBCURL_VERSION_NUM >= 0x071304 > curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, > - get_curl_allowed_protocols()); > + get_curl_allowed_protocols(0)); > curl_easy_setopt(result, CURLOPT_PROTOCOLS, > - get_curl_allowed_protocols()); > + get_curl_allowed_protocols(-1)); This covers internal redirects done by libcurl, but not the dumb-walker http-alternates nonsense. We have to feed the URL from http-alternates back to curl ourselves, so it uses CURLOPT_PROTOCOLS even though it should count as "not from the user". To fix that, I think we'd need something like: - get_curl_handle() stops setting these options, as it is done only once when the curl handle is initialized. Instead, the protocol restrictions should go into get_active_slot(), which is called for each request. The values set would remain the same, and be the baseline. - the http-walker.c code would need to know when it's requesting from the base URL, and when it's an alternate. I think this would depend on the position of the "alt" in in the linked list it keeps. - when requesting from an alternate, http-walker would set CURLOPT_PROTOCOLS with get_curl_allowed_protocols(0) I have to admit that it sounds like a fair bit of work for a pretty obscure case. You'd have to: 1. Turn http.allowRedirects to "true", to allow redirects even for non-initial contact. 2. Turn one of protocol.{http,https,ftp,ftps}.allow to "user" to restrict it from being used in a redirect. I'm tempted to punt on it and just do: if (http_follow_config == HTTP_FOLLOW_ALWAYS && get_curl_allowed_protocols(0) != get_curl_allowed_protocols(-1)) die("user-only protocol restrictions not implemented for http-alternates"); which errs on the safe side. We could even shove that down into the case where we actually see some alternates, like: diff --git a/http-walker.c b/http-walker.c index c2f81cd6a..5bcc850b1 100644 --- a/http-walker.c +++ b/http-walker.c @@ -160,6 +160,12 @@ static void prefetch(struct walker *walker, unsigned char *sha1) #endif } +static void check_alternates_protocol_restrictions(void) +{ + if (get_curl_allowed_protocols(0) != get_curl_allowed_protocol(-1)) + die("user-only protocol restrictions not implemented for http alternates"); +} + static void process_alternates_response(void *callback_data) { struct alternates_request *alt_req = @@ -272,6 +278,7 @@ static void process_alternates_response(void *callback_data) /* skip "objects\n" at end */ if (okay) { struct strbuf target = STRBUF_INIT; + check_alternates_protocol_restrictions(); strbuf_add(&target, base, serverlen); strbuf_add(&target, data + i, posn - i - 7); warning("adding alternate object store: %s", I find it unlikely that anybody would ever care, but at least we'd do the safe thing. I dunno. Maybe I am just being lazy. -Peff ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed 2016-12-14 16:40 ` Jeff King @ 2016-12-14 20:13 ` Brandon Williams 2016-12-14 20:33 ` Jeff King 2016-12-14 20:37 ` Brandon Williams 0 siblings, 2 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 20:13 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, gitster, jrnieder On 12/14, Jeff King wrote: > On Tue, Dec 13, 2016 at 05:40:37PM -0800, Brandon Williams wrote: > > > Add the from_user parameter to the 'is_transport_allowed' function. > > This allows callers to query if a transport protocol is allowed, given > > that the caller knows that the protocol is coming from the user (1) or > > not from the user (0) such as redirects in libcurl. If unknown a -1 > > should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER` > > to determine if the protocol came from the user. > > I think your commit message is upside-down with respect to the purpose > of the patch. The end goal we want is for http to distinguish between > protocol restrictions for redirects versus initial requests. The rest is > an implementation detail. It's definitely still worth discussing that > implementation detail (though I think your in-code comments may be > sufficient), but I don't see the rationale discussed here at all. I'll fix the commit message to better discuss the reasoning behind the change. > > Signed-off-by: Brandon Williams <bmwill@google.com> > > --- > > http.c | 14 +++++++------- > > transport.c | 8 +++++--- > > transport.h | 13 ++++++++++--- > > 3 files changed, 22 insertions(+), 13 deletions(-) > > I'm trying to think of a way to test this. I guess the case we are > covering here is when a server redirects, but the protocol is only > allowed from the user. So: > > diff --git a/t/t5812-proto-disable-http.sh b/t/t5812-proto-disable-http.sh > index 044cc152f..d911afd24 100755 > --- a/t/t5812-proto-disable-http.sh > +++ b/t/t5812-proto-disable-http.sh > @@ -30,5 +30,12 @@ test_expect_success 'curl limits redirects' ' > test_must_fail git clone "$HTTPD_URL/loop-redir/smart/repo.git" > ' > > +test_expect_success 'http can be limited to from-user' ' > + git -c protocol.http.allow=user \ > + clone "$HTTPD_URL/smart/repo.git" plain.git && > + test_must_fail git -c protocol.http.allow=user \ > + clone "$HTTPD_URL/smart-redir-perm/repo.git" redir.git > +' > + > stop_httpd > test_done > > It's an oddball configuration, and you'd probably just set > http.followRedirects=false in practice, but it does correctly check this > case. K I'll add this in as a test. > > @@ -588,9 +588,9 @@ static CURL *get_curl_handle(void) > > #endif > > #if LIBCURL_VERSION_NUM >= 0x071304 > > curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, > > - get_curl_allowed_protocols()); > > + get_curl_allowed_protocols(0)); > > curl_easy_setopt(result, CURLOPT_PROTOCOLS, > > - get_curl_allowed_protocols()); > > + get_curl_allowed_protocols(-1)); > > This covers internal redirects done by libcurl, but not the dumb-walker > http-alternates nonsense. We have to feed the URL from http-alternates > back to curl ourselves, so it uses CURLOPT_PROTOCOLS even though it > should count as "not from the user". > > To fix that, I think we'd need something like: > > - get_curl_handle() stops setting these options, as it is done only > once when the curl handle is initialized. Instead, the protocol > restrictions should go into get_active_slot(), which is called for > each request. The values set would remain the same, and be the > baseline. > > - the http-walker.c code would need to know when it's requesting from > the base URL, and when it's an alternate. I think this would depend > on the position of the "alt" in in the linked list it keeps. > > - when requesting from an alternate, http-walker would set > CURLOPT_PROTOCOLS with get_curl_allowed_protocols(0) > > I have to admit that it sounds like a fair bit of work for a pretty > obscure case. You'd have to: > > 1. Turn http.allowRedirects to "true", to allow redirects even for > non-initial contact. > > 2. Turn one of protocol.{http,https,ftp,ftps}.allow to "user" to > restrict it from being used in a redirect. > > I'm tempted to punt on it and just do: > > if (http_follow_config == HTTP_FOLLOW_ALWAYS && > get_curl_allowed_protocols(0) != get_curl_allowed_protocols(-1)) > die("user-only protocol restrictions not implemented for http-alternates"); > > which errs on the safe side. We could even shove that down into the case > where we actually see some alternates, like: > > diff --git a/http-walker.c b/http-walker.c > index c2f81cd6a..5bcc850b1 100644 > --- a/http-walker.c > +++ b/http-walker.c > @@ -160,6 +160,12 @@ static void prefetch(struct walker *walker, unsigned char *sha1) > #endif > } > > +static void check_alternates_protocol_restrictions(void) > +{ > + if (get_curl_allowed_protocols(0) != get_curl_allowed_protocol(-1)) > + die("user-only protocol restrictions not implemented for http alternates"); > +} > + > static void process_alternates_response(void *callback_data) > { > struct alternates_request *alt_req = > @@ -272,6 +278,7 @@ static void process_alternates_response(void *callback_data) > /* skip "objects\n" at end */ > if (okay) { > struct strbuf target = STRBUF_INIT; > + check_alternates_protocol_restrictions(); > strbuf_add(&target, base, serverlen); > strbuf_add(&target, data + i, posn - i - 7); > warning("adding alternate object store: %s", > > I find it unlikely that anybody would ever care, but at least we'd do > the safe thing. I dunno. Maybe I am just being lazy. Well, that's unfortunate! It does sound like a more full-proof solution to these dumb http alternates could be involved. I don't think your simple "lazy" solution may be enough to not just die by default. By default ftp/ftps will have a policy of "user only" which means they will be set by the call to get_curl_allowed_protocol(-1) but not set by get_curl_allowed_protocol(0). This would result in the call to check_alternates_protocol_restrictions failing all the time unless the user explicitly sets ftp/ftps to "always" or "never". If that is the desired behavior then your proposed solution would be fine, otherwise we may have to do the more involved approach. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed 2016-12-14 20:13 ` Brandon Williams @ 2016-12-14 20:33 ` Jeff King 2016-12-14 21:12 ` Jeff King [not found] ` <CAP3OtXiOPbAkr5Mn+5tEmZZAZzJXQ4CvtpHCg=wt+k-bi6K2vA@mail.gmail.com> 2016-12-14 20:37 ` Brandon Williams 1 sibling, 2 replies; 124+ messages in thread From: Jeff King @ 2016-12-14 20:33 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, gitster, jrnieder On Wed, Dec 14, 2016 at 12:13:23PM -0800, Brandon Williams wrote: > > I find it unlikely that anybody would ever care, but at least we'd do > > the safe thing. I dunno. Maybe I am just being lazy. > > Well, that's unfortunate! It does sound like a more full-proof solution > to these dumb http alternates could be involved. I don't think your > simple "lazy" solution may be enough to not just die by default. > > By default ftp/ftps will have a policy of "user only" which means they > will be set by the call to get_curl_allowed_protocol(-1) but not set by > get_curl_allowed_protocol(0). This would result in the call to > check_alternates_protocol_restrictions failing all the time unless the > user explicitly sets ftp/ftps to "always" or "never". If that is the > desired behavior then your proposed solution would be fine, otherwise we > may have to do the more involved approach. Oh, hrm, you're right. I was definitely meaning for this to kick in only when you had explicitly configured a protocol to "user" (they'd still need to enable redirects, but that's much more likely). Arguably ftp should be on the "safe" list, since it's implemented via the exact same code as https. That list comes originally from 33cfccbbf (submodule: allow only certain protocols for submodule fetches, 2015-09-16), but that was just based on what I thought was reasonable at the time. I find it hard to believe anybody actually uses git-over-ftp these days, let alone as a protocol redirect via http-alternates. But AFAIK it does work. One other "simple" fix is at the moment we parse http-alternates, to parse the URL ourself and check the policy. Like: const char *protocols[] = { "http", "https", "ftp", "ftps" }; for (i = 0; i < ARRAY_SIZE(protocols); i++) { const char *end; if (skip_prefix(url, protocols[i], end) && starts_with(end, "://")) break; } if (i >= ARRAY_SIZE(protocols)) warning("ignoring alternate with unknown protocol: %s", url); else if (!is_transport_allowed(protocols[i], 0)) warning("ignoring alternate with restricted protocol: %s", url); else { /* actually set up the alt struct */ } That keeps the logic nicely contained. The downside is that if our interpretation of the URL is ever different than curl's, it could create a security problem. The bit above seems fairly foolproof, though, because it functions as a whitelist. So it doesn't matter if you can trigger an http request via curl with exotic syntax; it matters whether curl will trigger anything _besides_ an http syntax if the string starts with "http://". Which seems unlikely. I may have convinced myself this is a reasonable approach. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed 2016-12-14 20:33 ` Jeff King @ 2016-12-14 21:12 ` Jeff King 2016-12-14 21:58 ` Brandon Williams [not found] ` <CAP3OtXiOPbAkr5Mn+5tEmZZAZzJXQ4CvtpHCg=wt+k-bi6K2vA@mail.gmail.com> 1 sibling, 1 reply; 124+ messages in thread From: Jeff King @ 2016-12-14 21:12 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, gitster, jrnieder On Wed, Dec 14, 2016 at 03:33:49PM -0500, Jeff King wrote: > One other "simple" fix is at the moment we parse http-alternates, to > parse the URL ourself and check the policy. Like: > [...] > I may have convinced myself this is a reasonable approach. So here it is in patch form, with a test. I also took a look at how bad it would be to plumb through the "this is an alternate" flag. And it's actually a little nasty, because the http-walker isn't calling get_active_slot() itself. It's relying on http_get_file() and other wrappers. The recent http_get_options makes this slightly less terrible, but I'd still rather avoid infecting the general http code that is used for the smart-http code paths. So I think this patch on top of your series, plus the other minor fixes we've discussed, the topic should be ready for 'next'. -- >8 -- Subject: http: respect protocol.*.allow=user for http-alternates The http-walker may fetch the http-alternates (or alternates) file from a remote in order to find more objects. This should count as a "not from the user" use of the protocol. But because we implement the redirection ourselves and feed the new URL to curl, it will use the CURLOPT_PROTOCOLS rules, not the more restrictive CURLOPT_REDIR_PROTOCOLS. The ideal solution would be for each curl request we make to know whether or not is directly from the user or part of an alternates redirect, and then set CURLOPT_PROTOCOLS as appropriate. However, that would require plumbing that information through all of the various layers of the http code. Instead, let's check the protocol at the source: when we are parsing the remote http-alternates file. The only downside is that if there's any mismatch between what protocol we think it is versus what curl thinks it is, it could violate the policy. To address this, we'll make the parsing err on the picky side, and only allow protocols that it can parse definitively. So for example, you can't elude the "http" policy by asking for "HTTP://", even though curl might handle it; we would reject it as unknown. The only unsafe case would be if you have a URL that starts with "http://" but curl interprets as another protocol. That seems like an unlikely failure mode (and we are still protected by our base CURLOPT_PROTOCOL setting, so the worst you could do is trigger one of https, ftp, or ftps). Signed-off-by: Jeff King <peff@peff.net> --- I actually do reject "HTTP://" here, though I suspect curl does take it. I notice that you cannot ask to clone "HTTP://..." in the first place, as our remote-helper interface is case sensitive (or maybe it would even respect "HTTP://" on a case-insensitive file system!). Possibly we should be more consistent about down-casing protocols when comparing them, but I'm not sure if anybody actually cares in practice. http-walker.c | 52 ++++++++++++++++++++++++++++++++++++---------- t/t5550-http-fetch-dumb.sh | 10 +++++++++ 2 files changed, 51 insertions(+), 11 deletions(-) diff --git a/http-walker.c b/http-walker.c index c2f81cd6a..b34b6ace7 100644 --- a/http-walker.c +++ b/http-walker.c @@ -3,6 +3,7 @@ #include "walker.h" #include "http.h" #include "list.h" +#include "transport.h" struct alt_base { char *base; @@ -160,6 +161,32 @@ static void prefetch(struct walker *walker, unsigned char *sha1) #endif } +static int is_alternate_allowed(const char *url) +{ + const char *protocols[] = { + "http", "https", "ftp", "ftps" + }; + int i; + + for (i = 0; i < ARRAY_SIZE(protocols); i++) { + const char *end; + if (skip_prefix(url, protocols[i], &end) && + starts_with(end, "://")) + break; + } + + if (i >= ARRAY_SIZE(protocols)) { + warning("ignoring alternate with unknown protocol: %s", url); + return 0; + } + if (!is_transport_allowed(protocols[i], 0)) { + warning("ignoring alternate with restricted protocol: %s", url); + return 0; + } + + return 1; +} + static void process_alternates_response(void *callback_data) { struct alternates_request *alt_req = @@ -274,17 +301,20 @@ static void process_alternates_response(void *callback_data) struct strbuf target = STRBUF_INIT; strbuf_add(&target, base, serverlen); strbuf_add(&target, data + i, posn - i - 7); - warning("adding alternate object store: %s", - target.buf); - newalt = xmalloc(sizeof(*newalt)); - newalt->next = NULL; - newalt->base = strbuf_detach(&target, NULL); - newalt->got_indices = 0; - newalt->packs = NULL; - - while (tail->next != NULL) - tail = tail->next; - tail->next = newalt; + + if (is_alternate_allowed(target.buf)) { + warning("adding alternate object store: %s", + target.buf); + newalt = xmalloc(sizeof(*newalt)); + newalt->next = NULL; + newalt->base = strbuf_detach(&target, NULL); + newalt->got_indices = 0; + newalt->packs = NULL; + + while (tail->next != NULL) + tail = tail->next; + tail->next = newalt; + } } } i = posn + 1; diff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh index 22011f0b6..c0ee29c65 100755 --- a/t/t5550-http-fetch-dumb.sh +++ b/t/t5550-http-fetch-dumb.sh @@ -360,5 +360,15 @@ test_expect_success 'http-alternates cannot point at funny protocols' ' clone "$HTTPD_URL/dumb/evil.git" evil-file ' +test_expect_success 'http-alternates triggers not-from-user protocol check' ' + echo "$HTTPD_URL/dumb/victim.git/objects" \ + >"$evil/objects/info/http-alternates" && + test_config_global http.followRedirects true && + test_must_fail git -c protocol.http.allow=user \ + clone $HTTPD_URL/dumb/evil.git evil-user && + git -c protocol.http.allow=always \ + clone $HTTPD_URL/dumb/evil.git evil-user +' + stop_httpd test_done -- 2.11.0.341.g202cd3142 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed 2016-12-14 21:12 ` Jeff King @ 2016-12-14 21:58 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 21:58 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, gitster, jrnieder On 12/14, Jeff King wrote: > On Wed, Dec 14, 2016 at 03:33:49PM -0500, Jeff King wrote: > > > One other "simple" fix is at the moment we parse http-alternates, to > > parse the URL ourself and check the policy. Like: > > [...] > > I may have convinced myself this is a reasonable approach. > > So here it is in patch form, with a test. > > I also took a look at how bad it would be to plumb through the "this is > an alternate" flag. And it's actually a little nasty, because the > http-walker isn't calling get_active_slot() itself. It's relying on > http_get_file() and other wrappers. The recent http_get_options makes > this slightly less terrible, but I'd still rather avoid infecting the > general http code that is used for the smart-http code paths. > > So I think this patch on top of your series, plus the other minor fixes > we've discussed, the topic should be ready for 'next'. Sounds good, I'll make those few changes, place this patch ontop of the series and send out v10 of the series in just a bit. -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
[parent not found: <CAP3OtXiOPbAkr5Mn+5tEmZZAZzJXQ4CvtpHCg=wt+k-bi6K2vA@mail.gmail.com>]
[parent not found: <CAP3OtXhH++szRws20MaHt-ftLBMUJuYiTmfL50mOFP4FA4Mn6Q@mail.gmail.com>]
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed [not found] ` <CAP3OtXhH++szRws20MaHt-ftLBMUJuYiTmfL50mOFP4FA4Mn6Q@mail.gmail.com> @ 2016-12-14 22:52 ` Jeff King 0 siblings, 0 replies; 124+ messages in thread From: Jeff King @ 2016-12-14 22:52 UTC (permalink / raw) To: Blake Burkhart; +Cc: git, jrnieder, Brandon Williams, gitster, sbeller On Wed, Dec 14, 2016 at 04:29:52PM -0600, Blake Burkhart wrote: > You may want to set CURLOPT_DEFAULT_PROTOCOL if we don't already. Apparently > the default value of NULL causes it to make a guess based on the host if no > protocol is present. But you are discussing a situation where "http://" is > present, so that doesn't apply. Cute. I agree it doesn't matter here, where we're sure there's a protocol specifier at the beginning. It might matter if you instruct git to use a specific remote-helper, like: $ echo 127.0.0.1 ftp.example.com >>/etc/hosts $ git clone http::ftp.example.com which will try to connect via ftp. Of course that's no different than: $ git clone http::ftp://example.com which you can already do. git-clone sees "http::" and hands it off to git-remote-http, which then processes the rest of the arguments as it sees fit (in this case, handing it off to curl). Prior to these more recent patches I suspect you could do: $ git clone http::file://whatever but now we set CURLOPT_PROTOCOL to restrict it to just http/ftp (so you can be confusing by asking for http and ending up in curl to do ftp, but in any such case you could also have just asked for ftp in the first place). > Also, I thought we left out ftp because it was deprecated, but I don't > remember exactly. I couldn't find anything interesting in the archives (neither the public list nor git-security). Given how unlikely it is to be used, it does seem like a good idea to keep it in the "maybe" category, if only because it decreases the attack surface (and for those following along, we're just talking about not-from-user uses here, so people sticking a funny ftp URL in .gitmodules, or redirecting http to ftp, etc). -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed 2016-12-14 20:13 ` Brandon Williams 2016-12-14 20:33 ` Jeff King @ 2016-12-14 20:37 ` Brandon Williams 2016-12-14 20:41 ` Jeff King 1 sibling, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-12-14 20:37 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, gitster, jrnieder On 12/14, Brandon Williams wrote: > On 12/14, Jeff King wrote: > > On Tue, Dec 13, 2016 at 05:40:37PM -0800, Brandon Williams wrote: > > > > > Add the from_user parameter to the 'is_transport_allowed' function. > > > This allows callers to query if a transport protocol is allowed, given > > > that the caller knows that the protocol is coming from the user (1) or > > > not from the user (0) such as redirects in libcurl. If unknown a -1 > > > should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER` > > > to determine if the protocol came from the user. > > > > I think your commit message is upside-down with respect to the purpose > > of the patch. The end goal we want is for http to distinguish between > > protocol restrictions for redirects versus initial requests. The rest is > > an implementation detail. It's definitely still worth discussing that > > implementation detail (though I think your in-code comments may be > > sufficient), but I don't see the rationale discussed here at all. > > I'll fix the commit message to better discuss the reasoning behind the > change. > > > > Signed-off-by: Brandon Williams <bmwill@google.com> > > > --- > > > http.c | 14 +++++++------- > > > transport.c | 8 +++++--- > > > transport.h | 13 ++++++++++--- > > > 3 files changed, 22 insertions(+), 13 deletions(-) > > > > I'm trying to think of a way to test this. I guess the case we are > > covering here is when a server redirects, but the protocol is only > > allowed from the user. So: > > > > diff --git a/t/t5812-proto-disable-http.sh b/t/t5812-proto-disable-http.sh > > index 044cc152f..d911afd24 100755 > > --- a/t/t5812-proto-disable-http.sh > > +++ b/t/t5812-proto-disable-http.sh > > @@ -30,5 +30,12 @@ test_expect_success 'curl limits redirects' ' > > test_must_fail git clone "$HTTPD_URL/loop-redir/smart/repo.git" > > ' > > > > +test_expect_success 'http can be limited to from-user' ' > > + git -c protocol.http.allow=user \ > > + clone "$HTTPD_URL/smart/repo.git" plain.git && > > + test_must_fail git -c protocol.http.allow=user \ > > + clone "$HTTPD_URL/smart-redir-perm/repo.git" redir.git > > +' > > + > > stop_httpd > > test_done > > > > It's an oddball configuration, and you'd probably just set > > http.followRedirects=false in practice, but it does correctly check this > > case. > > K I'll add this in as a test. > > > > @@ -588,9 +588,9 @@ static CURL *get_curl_handle(void) > > > #endif > > > #if LIBCURL_VERSION_NUM >= 0x071304 > > > curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, > > > - get_curl_allowed_protocols()); > > > + get_curl_allowed_protocols(0)); > > > curl_easy_setopt(result, CURLOPT_PROTOCOLS, > > > - get_curl_allowed_protocols()); > > > + get_curl_allowed_protocols(-1)); > > > > This covers internal redirects done by libcurl, but not the dumb-walker > > http-alternates nonsense. We have to feed the URL from http-alternates > > back to curl ourselves, so it uses CURLOPT_PROTOCOLS even though it > > should count as "not from the user". > > > > To fix that, I think we'd need something like: > > > > - get_curl_handle() stops setting these options, as it is done only > > once when the curl handle is initialized. Instead, the protocol > > restrictions should go into get_active_slot(), which is called for > > each request. The values set would remain the same, and be the > > baseline. > > > > - the http-walker.c code would need to know when it's requesting from > > the base URL, and when it's an alternate. I think this would depend > > on the position of the "alt" in in the linked list it keeps. > > > > - when requesting from an alternate, http-walker would set > > CURLOPT_PROTOCOLS with get_curl_allowed_protocols(0) > > > > I have to admit that it sounds like a fair bit of work for a pretty > > obscure case. You'd have to: > > > > 1. Turn http.allowRedirects to "true", to allow redirects even for > > non-initial contact. > > > > 2. Turn one of protocol.{http,https,ftp,ftps}.allow to "user" to > > restrict it from being used in a redirect. > > > > I'm tempted to punt on it and just do: > > > > if (http_follow_config == HTTP_FOLLOW_ALWAYS && > > get_curl_allowed_protocols(0) != get_curl_allowed_protocols(-1)) > > die("user-only protocol restrictions not implemented for http-alternates"); > > > > which errs on the safe side. We could even shove that down into the case > > where we actually see some alternates, like: > > > > diff --git a/http-walker.c b/http-walker.c > > index c2f81cd6a..5bcc850b1 100644 > > --- a/http-walker.c > > +++ b/http-walker.c > > @@ -160,6 +160,12 @@ static void prefetch(struct walker *walker, unsigned char *sha1) > > #endif > > } > > > > +static void check_alternates_protocol_restrictions(void) > > +{ > > + if (get_curl_allowed_protocols(0) != get_curl_allowed_protocol(-1)) > > + die("user-only protocol restrictions not implemented for http alternates"); > > +} > > + > > static void process_alternates_response(void *callback_data) > > { > > struct alternates_request *alt_req = > > @@ -272,6 +278,7 @@ static void process_alternates_response(void *callback_data) > > /* skip "objects\n" at end */ > > if (okay) { > > struct strbuf target = STRBUF_INIT; > > + check_alternates_protocol_restrictions(); > > strbuf_add(&target, base, serverlen); > > strbuf_add(&target, data + i, posn - i - 7); > > warning("adding alternate object store: %s", > > > > I find it unlikely that anybody would ever care, but at least we'd do > > the safe thing. I dunno. Maybe I am just being lazy. > > Well, that's unfortunate! It does sound like a more full-proof solution > to these dumb http alternates could be involved. I don't think your > simple "lazy" solution may be enough to not just die by default. > > By default ftp/ftps will have a policy of "user only" which means they > will be set by the call to get_curl_allowed_protocol(-1) but not set by > get_curl_allowed_protocol(0). This would result in the call to > check_alternates_protocol_restrictions failing all the time unless the > user explicitly sets ftp/ftps to "always" or "never". If that is the > desired behavior then your proposed solution would be fine, otherwise we > may have to do the more involved approach. Naively looking at the code (and your longer suggestion), is there a reason why we couldn't simply have http-walker set CURLOPT_PROTOCOLS with get_curl_allowed_protocols(0) in the fetch_alternates() function? That way we just override the CURLOPT_PROTOCOLS value when alternates are involved. Like so: diff --git a/http-walker.c b/http-walker.c index c2f81cd..b284cec 100644 --- a/http-walker.c +++ b/http-walker.c @@ -339,6 +339,8 @@ static void fetch_alternates(struct walker *walker, const char *base) curl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer); curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer); curl_easy_setopt(slot->curl, CURLOPT_URL, url.buf); + curl_easy_setopt(slot->curl, CURLOPT_PROTOCOLS, + get_curl_allowed_protocols(0)); alt_req.base = base; alt_req.url = &url; -- Brandon Williams ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed 2016-12-14 20:37 ` Brandon Williams @ 2016-12-14 20:41 ` Jeff King 2016-12-14 20:50 ` Brandon Williams 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-12-14 20:41 UTC (permalink / raw) To: Brandon Williams; +Cc: git, sbeller, bburky, gitster, jrnieder On Wed, Dec 14, 2016 at 12:37:52PM -0800, Brandon Williams wrote: > Naively looking at the code (and your longer suggestion), is there a > reason why we couldn't simply have http-walker set CURLOPT_PROTOCOLS > with get_curl_allowed_protocols(0) in the fetch_alternates() function? > That way we just override the CURLOPT_PROTOCOLS value when alternates > are involved. No, because we may have many curl handles (especially for the http-walker, which wants to fetch several objects simultaneously), and they get recycled as needed for many requests. So setting a restriction there on slot->curl will only cover the one handle, and miss other ones which may be used later (and likewise, that one handle with the restriction may get recycled and used for a non-alternate fetch, and would be unnecessarily restrictive). That's why any curl-level settings have to happen when we call get_active_slot(), since that's when we know what we're actually using the handle for. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed 2016-12-14 20:41 ` Jeff King @ 2016-12-14 20:50 ` Brandon Williams 0 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 20:50 UTC (permalink / raw) To: Jeff King; +Cc: git, sbeller, bburky, gitster, jrnieder On 12/14, Jeff King wrote: > On Wed, Dec 14, 2016 at 12:37:52PM -0800, Brandon Williams wrote: > > > Naively looking at the code (and your longer suggestion), is there a > > reason why we couldn't simply have http-walker set CURLOPT_PROTOCOLS > > with get_curl_allowed_protocols(0) in the fetch_alternates() function? > > That way we just override the CURLOPT_PROTOCOLS value when alternates > > are involved. > > No, because we may have many curl handles (especially for the > http-walker, which wants to fetch several objects simultaneously), and > they get recycled as needed for many requests. > > So setting a restriction there on slot->curl will only cover the one > handle, and miss other ones which may be used later (and likewise, that > one handle with the restriction may get recycled and used for a > non-alternate fetch, and would be unnecessarily restrictive). > > That's why any curl-level settings have to happen when we call > get_active_slot(), since that's when we know what we're actually using > the handle for. Fair enough, I figured there may be some reuse happening with the curl handles but didn't do enough digging to discover that myself. Thanks :) -- Brandon Williams ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v10 0/6] transport protocol policy configuration 2016-12-14 1:40 ` [PATCH v9 0/5] transport protocol policy configuration Brandon Williams ` (4 preceding siblings ...) 2016-12-14 1:40 ` [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed Brandon Williams @ 2016-12-14 22:39 ` Brandon Williams 2016-12-14 22:39 ` [PATCH v10 1/6] lib-proto-disable: variable name fix Brandon Williams ` (7 more replies) 5 siblings, 8 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 22:39 UTC (permalink / raw) To: git; +Cc: Brandon Williams, gitster, peff, sbeller, bburky, jrnieder v10 of this series fixes the following: * A few updates to the commit messages in order to better convey the reasoning behind the a few of the patches. * Additional test to verify that curl redirects respect configured protocol policies. * Patch added by Jeff King to make http alternates respect configured protocol policies. Brandon Williams (5): lib-proto-disable: variable name fix http: always warn if libcurl version is too old transport: add protocol policy config option http: create function to get curl allowed protocols transport: add from_user parameter to is_transport_allowed Jeff King (1): http: respect protocol.*.allow=user for http-alternates Documentation/config.txt | 46 +++++++++++++ Documentation/git.txt | 38 ++++------- git-submodule.sh | 12 ++-- http-walker.c | 52 +++++++++++--- http.c | 36 ++++++---- t/lib-proto-disable.sh | 142 ++++++++++++++++++++++++++++++++++++--- t/t5509-fetch-push-namespaces.sh | 1 + t/t5550-http-fetch-dumb.sh | 10 +++ t/t5802-connect-helper.sh | 1 + t/t5812-proto-disable-http.sh | 7 ++ transport.c | 84 ++++++++++++++++++++--- transport.h | 19 +++--- 12 files changed, 363 insertions(+), 85 deletions(-) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v10 1/6] lib-proto-disable: variable name fix 2016-12-14 22:39 ` [PATCH v10 0/6] transport protocol policy configuration Brandon Williams @ 2016-12-14 22:39 ` Brandon Williams 2016-12-14 22:39 ` [PATCH v10 2/6] http: always warn if libcurl version is too old Brandon Williams ` (6 subsequent siblings) 7 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 22:39 UTC (permalink / raw) To: git; +Cc: Brandon Williams, gitster, peff, sbeller, bburky, jrnieder The test_proto function assigns the positional parameters to named variables, but then still refers to "$desc" as "$1". Using $desc is more readable and less error-prone. Signed-off-by: Brandon Williams <bmwill@google.com> --- t/lib-proto-disable.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index b0917d9..be88e9a 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -9,7 +9,7 @@ test_proto () { proto=$2 url=$3 - test_expect_success "clone $1 (enabled)" ' + test_expect_success "clone $desc (enabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=$proto && @@ -18,7 +18,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (enabled)" ' + test_expect_success "fetch $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -27,7 +27,7 @@ test_proto () { ) ' - test_expect_success "push $1 (enabled)" ' + test_expect_success "push $desc (enabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=$proto && @@ -36,7 +36,7 @@ test_proto () { ) ' - test_expect_success "push $1 (disabled)" ' + test_expect_success "push $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -45,7 +45,7 @@ test_proto () { ) ' - test_expect_success "fetch $1 (disabled)" ' + test_expect_success "fetch $desc (disabled)" ' ( cd tmp.git && GIT_ALLOW_PROTOCOL=none && @@ -54,7 +54,7 @@ test_proto () { ) ' - test_expect_success "clone $1 (disabled)" ' + test_expect_success "clone $desc (disabled)" ' rm -rf tmp.git && ( GIT_ALLOW_PROTOCOL=none && -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v10 2/6] http: always warn if libcurl version is too old 2016-12-14 22:39 ` [PATCH v10 0/6] transport protocol policy configuration Brandon Williams 2016-12-14 22:39 ` [PATCH v10 1/6] lib-proto-disable: variable name fix Brandon Williams @ 2016-12-14 22:39 ` Brandon Williams 2016-12-15 0:21 ` Jeff King 2016-12-14 22:39 ` [PATCH v10 3/6] transport: add protocol policy config option Brandon Williams ` (5 subsequent siblings) 7 siblings, 1 reply; 124+ messages in thread From: Brandon Williams @ 2016-12-14 22:39 UTC (permalink / raw) To: git; +Cc: Brandon Williams, gitster, peff, sbeller, bburky, jrnieder Always warn if libcurl version is too old because: 1. Even without a protocol whitelist, newer versions of curl have all non-http protocols disabled by default. 2. A future patch will introduce default "known-good" and "known-bad" protocols which are allowed/disallowed by 'is_transport_allowed' which older version of libcurl can't respect. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 5 ++--- transport.c | 5 ----- transport.h | 6 ------ 3 files changed, 2 insertions(+), 14 deletions(-) diff --git a/http.c b/http.c index 5cd3ffd..034426e 100644 --- a/http.c +++ b/http.c @@ -583,9 +583,8 @@ static CURL *get_curl_handle(void) curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); curl_easy_setopt(result, CURLOPT_PROTOCOLS, allowed_protocols); #else - if (transport_restrict_protocols()) - warning("protocol restrictions not applied to curl redirects because\n" - "your curl version is too old (>= 7.19.4)"); + warning("protocol restrictions not applied to curl redirects because\n" + "your curl version is too old (>= 7.19.4)"); #endif if (getenv("GIT_CURL_VERBOSE")) diff --git a/transport.c b/transport.c index 41eb82c..dff929e 100644 --- a/transport.c +++ b/transport.c @@ -629,11 +629,6 @@ void transport_check_allowed(const char *type) die("transport '%s' not allowed", type); } -int transport_restrict_protocols(void) -{ - return !!protocol_whitelist(); -} - struct transport *transport_get(struct remote *remote, const char *url) { const char *helper; diff --git a/transport.h b/transport.h index c681408..3396e1d 100644 --- a/transport.h +++ b/transport.h @@ -153,12 +153,6 @@ int is_transport_allowed(const char *type); */ void transport_check_allowed(const char *type); -/* - * Returns true if the user has attempted to turn on protocol - * restrictions at all. - */ -int transport_restrict_protocols(void); - /* Transport options which apply to git:// and scp-style URLs */ /* The program to use on the remote side to send a pack */ -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v10 2/6] http: always warn if libcurl version is too old 2016-12-14 22:39 ` [PATCH v10 2/6] http: always warn if libcurl version is too old Brandon Williams @ 2016-12-15 0:21 ` Jeff King 2016-12-15 17:29 ` Junio C Hamano 0 siblings, 1 reply; 124+ messages in thread From: Jeff King @ 2016-12-15 0:21 UTC (permalink / raw) To: Brandon Williams; +Cc: git, gitster, sbeller, bburky, jrnieder On Wed, Dec 14, 2016 at 02:39:51PM -0800, Brandon Williams wrote: > Always warn if libcurl version is too old because: > > 1. Even without a protocol whitelist, newer versions of curl have all > non-http protocols disabled by default. Technically, non-http and non-ftp. Maybe just "non-standard" would be more accurate. Not worth a re-roll, but if Junio hasn't applied yet, maybe worth fixing up while applying. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v10 2/6] http: always warn if libcurl version is too old 2016-12-15 0:21 ` Jeff King @ 2016-12-15 17:29 ` Junio C Hamano 0 siblings, 0 replies; 124+ messages in thread From: Junio C Hamano @ 2016-12-15 17:29 UTC (permalink / raw) To: Jeff King; +Cc: Brandon Williams, git, sbeller, bburky, jrnieder Jeff King <peff@peff.net> writes: > On Wed, Dec 14, 2016 at 02:39:51PM -0800, Brandon Williams wrote: > >> Always warn if libcurl version is too old because: >> >> 1. Even without a protocol whitelist, newer versions of curl have all >> non-http protocols disabled by default. > > Technically, non-http and non-ftp. Maybe just "non-standard" would be > more accurate. > > Not worth a re-roll, but if Junio hasn't applied yet, maybe worth fixing > up while applying. I just did a "rebase -i"; thanks always for your careful reading. ^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH v10 3/6] transport: add protocol policy config option 2016-12-14 22:39 ` [PATCH v10 0/6] transport protocol policy configuration Brandon Williams 2016-12-14 22:39 ` [PATCH v10 1/6] lib-proto-disable: variable name fix Brandon Williams 2016-12-14 22:39 ` [PATCH v10 2/6] http: always warn if libcurl version is too old Brandon Williams @ 2016-12-14 22:39 ` Brandon Williams 2016-12-14 22:39 ` [PATCH v10 4/6] http: create function to get curl allowed protocols Brandon Williams ` (4 subsequent siblings) 7 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 22:39 UTC (permalink / raw) To: git; +Cc: Brandon Williams, gitster, peff, sbeller, bburky, jrnieder Previously the `GIT_ALLOW_PROTOCOL` environment variable was used to specify a whitelist of protocols to be used in clone/fetch/push commands. This patch introduces new configuration options for more fine-grained control for allowing/disallowing protocols. This also has the added benefit of allowing easier construction of a protocol whitelist on systems where setting an environment variable is non-trivial. Now users can specify a policy to be used for each type of protocol via the 'protocol.<name>.allow' config option. A default policy for all unconfigured protocols can be set with the 'protocol.allow' config option. If no user configured default is made git will allow known-safe protocols (http, https, git, ssh, file), disallow known-dangerous protocols (ext), and have a default policy of `user` for all other protocols. The supported policies are `always`, `never`, and `user`. The `user` policy can be used to configure a protocol to be usable when explicitly used by a user, while disallowing it for commands which run clone/fetch/push commands without direct user intervention (e.g. recursive initialization of submodules). Commands which can potentially clone/fetch/push from untrusted repositories without user intervention can export `GIT_PROTOCOL_FROM_USER` with a value of '0' to prevent protocols configured to the `user` policy from being used. Fix remote-ext tests to use the new config to allow the ext protocol to be tested. Based on a patch by Jeff King <peff@peff.net> Signed-off-by: Brandon Williams <bmwill@google.com> --- Documentation/config.txt | 46 ++++++++++++++ Documentation/git.txt | 38 +++++------- git-submodule.sh | 12 ++-- t/lib-proto-disable.sh | 130 +++++++++++++++++++++++++++++++++++++-- t/t5509-fetch-push-namespaces.sh | 1 + t/t5802-connect-helper.sh | 1 + transport.c | 75 +++++++++++++++++++++- 7 files changed, 264 insertions(+), 39 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index 8153336..50d3d06 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -2260,6 +2260,52 @@ pretty.<name>:: Note that an alias with the same name as a built-in format will be silently ignored. +protocol.allow:: + If set, provide a user defined default policy for all protocols which + don't explicitly have a policy (`protocol.<name>.allow`). By default, + if unset, known-safe protocols (http, https, git, ssh, file) have a + default policy of `always`, known-dangerous protocols (ext) have a + default policy of `never`, and all other protocols have a default + policy of `user`. Supported policies: ++ +-- + +* `always` - protocol is always able to be used. + +* `never` - protocol is never able to be used. + +* `user` - protocol is only able to be used when `GIT_PROTOCOL_FROM_USER` is + either unset or has a value of 1. This policy should be used when you want a + protocol to be directly usable by the user but don't want it used by commands which + execute clone/fetch/push commands without user input, e.g. recursive + submodule initialization. + +-- + +protocol.<name>.allow:: + Set a policy to be used by protocol `<name>` with clone/fetch/push + commands. See `protocol.allow` above for the available policies. ++ +The protocol names currently used by git are: ++ +-- + - `file`: any local file-based path (including `file://` URLs, + or local paths) + + - `git`: the anonymous git protocol over a direct TCP + connection (or proxy, if configured) + + - `ssh`: git over ssh (including `host:path` syntax, + `ssh://`, etc). + + - `http`: git over http, both "smart http" and "dumb http". + Note that this does _not_ include `https`; if you want to configure + both, you must do so individually. + + - any external helpers are named by their protocol (e.g., use + `hg` to allow the `git-remote-hg` helper) +-- + pull.ff:: By default, Git does not create an extra merge commit when merging a commit that is a descendant of the current commit. Instead, the diff --git a/Documentation/git.txt b/Documentation/git.txt index 923aa49..d9fb937 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -1129,30 +1129,20 @@ of clones and fetches. cloning a repository to make a backup). `GIT_ALLOW_PROTOCOL`:: - If set, provide a colon-separated list of protocols which are - allowed to be used with fetch/push/clone. This is useful to - restrict recursive submodule initialization from an untrusted - repository. Any protocol not mentioned will be disallowed (i.e., - this is a whitelist, not a blacklist). If the variable is not - set at all, all protocols are enabled. The protocol names - currently used by git are: - - - `file`: any local file-based path (including `file://` URLs, - or local paths) - - - `git`: the anonymous git protocol over a direct TCP - connection (or proxy, if configured) - - - `ssh`: git over ssh (including `host:path` syntax, - `ssh://`, etc). - - - `http`: git over http, both "smart http" and "dumb http". - Note that this does _not_ include `https`; if you want both, - you should specify both as `http:https`. - - - any external helpers are named by their protocol (e.g., use - `hg` to allow the `git-remote-hg` helper) - + If set to a colon-separated list of protocols, behave as if + `protocol.allow` is set to `never`, and each of the listed + protocols has `protocol.<name>.allow` set to `always` + (overriding any existing configuration). In other words, any + protocol not mentioned will be disallowed (i.e., this is a + whitelist, not a blacklist). See the description of + `protocol.allow` in linkgit:git-config[1] for more details. + +`GIT_PROTOCOL_FROM_USER`:: + Set to 0 to prevent protocols used by fetch/push/clone which are + configured to the `user` state. This is useful to restrict recursive + submodule initialization from an untrusted repository or for programs + which feed potentially-untrusted URLS to git commands. See + linkgit:git-config[1] for more details. Discussion[[Discussion]] ------------------------ diff --git a/git-submodule.sh b/git-submodule.sh index 78fdac9..fc44076 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -22,14 +22,10 @@ require_work_tree wt_prefix=$(git rev-parse --show-prefix) cd_to_toplevel -# Restrict ourselves to a vanilla subset of protocols; the URLs -# we get are under control of a remote repository, and we do not -# want them kicking off arbitrary git-remote-* programs. -# -# If the user has already specified a set of allowed protocols, -# we assume they know what they're doing and use that instead. -: ${GIT_ALLOW_PROTOCOL=file:git:http:https:ssh} -export GIT_ALLOW_PROTOCOL +# Tell the rest of git that any URLs we get don't come +# directly from the user, so it can apply policy as appropriate. +GIT_PROTOCOL_FROM_USER=0 +export GIT_PROTOCOL_FROM_USER command= branch= diff --git a/t/lib-proto-disable.sh b/t/lib-proto-disable.sh index be88e9a..02f49cb 100644 --- a/t/lib-proto-disable.sh +++ b/t/lib-proto-disable.sh @@ -1,10 +1,7 @@ # Test routines for checking protocol disabling. -# test cloning a particular protocol -# $1 - description of the protocol -# $2 - machine-readable name of the protocol -# $3 - the URL to try cloning -test_proto () { +# Test clone/fetch/push with GIT_ALLOW_PROTOCOL whitelist +test_whitelist () { desc=$1 proto=$2 url=$3 @@ -62,6 +59,129 @@ test_proto () { test_must_fail git clone --bare "$url" tmp.git ) ' + + test_expect_success "clone $desc (env var has precedence)" ' + rm -rf tmp.git && + ( + GIT_ALLOW_PROTOCOL=none && + export GIT_ALLOW_PROTOCOL && + test_must_fail git -c protocol.allow=always clone --bare "$url" tmp.git && + test_must_fail git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ) + ' +} + +test_config () { + desc=$1 + proto=$2 + url=$3 + + # Test clone/fetch/push with protocol.<type>.allow config + test_expect_success "clone $desc (enabled with config)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=always clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=always push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git -c protocol.$proto.allow=never fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git -c protocol.$proto.allow=never clone --bare "$url" tmp.git + ' + + # Test clone/fetch/push with protocol.user.allow and its env var + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git -c protocol.$proto.allow=user push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user push origin HEAD:pushed + ) + ' + + test_expect_success "fetch $desc (disabled)" ' + ( + cd tmp.git && + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user fetch + ) + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + ( + GIT_PROTOCOL_FROM_USER=0 && + export GIT_PROTOCOL_FROM_USER && + test_must_fail git -c protocol.$proto.allow=user clone --bare "$url" tmp.git + ) + ' + + # Test clone/fetch/push with protocol.allow user defined default + test_expect_success "clone $desc (enabled)" ' + rm -rf tmp.git && + git config --global protocol.allow always && + git clone --bare "$url" tmp.git + ' + + test_expect_success "fetch $desc (enabled)" ' + git -C tmp.git fetch + ' + + test_expect_success "push $desc (enabled)" ' + git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "push $desc (disabled)" ' + git config --global protocol.allow never && + test_must_fail git -C tmp.git push origin HEAD:pushed + ' + + test_expect_success "fetch $desc (disabled)" ' + test_must_fail git -C tmp.git fetch + ' + + test_expect_success "clone $desc (disabled)" ' + rm -rf tmp.git && + test_must_fail git clone --bare "$url" tmp.git + ' +} + +# test cloning a particular protocol +# $1 - description of the protocol +# $2 - machine-readable name of the protocol +# $3 - the URL to try cloning +test_proto () { + test_whitelist "$@" + + test_config "$@" } # set up an ssh wrapper that will access $host/$repo in the diff --git a/t/t5509-fetch-push-namespaces.sh b/t/t5509-fetch-push-namespaces.sh index bc44ac3..75c570a 100755 --- a/t/t5509-fetch-push-namespaces.sh +++ b/t/t5509-fetch-push-namespaces.sh @@ -4,6 +4,7 @@ test_description='fetch/push involving ref namespaces' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git init original && ( diff --git a/t/t5802-connect-helper.sh b/t/t5802-connect-helper.sh index b7a7f9d..c6c2661 100755 --- a/t/t5802-connect-helper.sh +++ b/t/t5802-connect-helper.sh @@ -4,6 +4,7 @@ test_description='ext::cmd remote "connect" helper' . ./test-lib.sh test_expect_success setup ' + git config --global protocol.ext.allow user && test_tick && git commit --allow-empty -m initial && test_tick && diff --git a/transport.c b/transport.c index dff929e..fbd799d 100644 --- a/transport.c +++ b/transport.c @@ -617,10 +617,81 @@ static const struct string_list *protocol_whitelist(void) return enabled ? &allowed : NULL; } +enum protocol_allow_config { + PROTOCOL_ALLOW_NEVER = 0, + PROTOCOL_ALLOW_USER_ONLY, + PROTOCOL_ALLOW_ALWAYS +}; + +static enum protocol_allow_config parse_protocol_config(const char *key, + const char *value) +{ + if (!strcasecmp(value, "always")) + return PROTOCOL_ALLOW_ALWAYS; + else if (!strcasecmp(value, "never")) + return PROTOCOL_ALLOW_NEVER; + else if (!strcasecmp(value, "user")) + return PROTOCOL_ALLOW_USER_ONLY; + + die("unknown value for config '%s': %s", key, value); +} + +static enum protocol_allow_config get_protocol_config(const char *type) +{ + char *key = xstrfmt("protocol.%s.allow", type); + char *value; + + /* first check the per-protocol config */ + if (!git_config_get_string(key, &value)) { + enum protocol_allow_config ret = + parse_protocol_config(key, value); + free(key); + free(value); + return ret; + } + free(key); + + /* if defined, fallback to user-defined default for unknown protocols */ + if (!git_config_get_string("protocol.allow", &value)) { + enum protocol_allow_config ret = + parse_protocol_config("protocol.allow", value); + free(value); + return ret; + } + + /* fallback to built-in defaults */ + /* known safe */ + if (!strcmp(type, "http") || + !strcmp(type, "https") || + !strcmp(type, "git") || + !strcmp(type, "ssh") || + !strcmp(type, "file")) + return PROTOCOL_ALLOW_ALWAYS; + + /* known scary; err on the side of caution */ + if (!strcmp(type, "ext")) + return PROTOCOL_ALLOW_NEVER; + + /* unknown; by default let them be used only directly by the user */ + return PROTOCOL_ALLOW_USER_ONLY; +} + int is_transport_allowed(const char *type) { - const struct string_list *allowed = protocol_whitelist(); - return !allowed || string_list_has_string(allowed, type); + const struct string_list *whitelist = protocol_whitelist(); + if (whitelist) + return string_list_has_string(whitelist, type); + + switch (get_protocol_config(type)) { + case PROTOCOL_ALLOW_ALWAYS: + return 1; + case PROTOCOL_ALLOW_NEVER: + return 0; + case PROTOCOL_ALLOW_USER_ONLY: + return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + } + + die("BUG: invalid protocol_allow_config type"); } void transport_check_allowed(const char *type) -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v10 4/6] http: create function to get curl allowed protocols 2016-12-14 22:39 ` [PATCH v10 0/6] transport protocol policy configuration Brandon Williams ` (2 preceding siblings ...) 2016-12-14 22:39 ` [PATCH v10 3/6] transport: add protocol policy config option Brandon Williams @ 2016-12-14 22:39 ` Brandon Williams 2016-12-14 22:39 ` [PATCH v10 5/6] transport: add from_user parameter to is_transport_allowed Brandon Williams ` (3 subsequent siblings) 7 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 22:39 UTC (permalink / raw) To: git; +Cc: Brandon Williams, gitster, peff, sbeller, bburky, jrnieder Move the creation of an allowed protocols whitelist to a helper function. This will be useful when we need to compute the set of allowed protocols differently for normal and redirect cases. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/http.c b/http.c index 034426e..f7c488a 100644 --- a/http.c +++ b/http.c @@ -489,10 +489,25 @@ static void set_curl_keepalive(CURL *c) } #endif +static long get_curl_allowed_protocols(void) +{ + long allowed_protocols = 0; + + if (is_transport_allowed("http")) + allowed_protocols |= CURLPROTO_HTTP; + if (is_transport_allowed("https")) + allowed_protocols |= CURLPROTO_HTTPS; + if (is_transport_allowed("ftp")) + allowed_protocols |= CURLPROTO_FTP; + if (is_transport_allowed("ftps")) + allowed_protocols |= CURLPROTO_FTPS; + + return allowed_protocols; +} + static CURL *get_curl_handle(void) { CURL *result = curl_easy_init(); - long allowed_protocols = 0; if (!result) die("curl_easy_init failed"); @@ -572,16 +587,10 @@ static CURL *get_curl_handle(void) curl_easy_setopt(result, CURLOPT_POST301, 1); #endif #if LIBCURL_VERSION_NUM >= 0x071304 - if (is_transport_allowed("http")) - allowed_protocols |= CURLPROTO_HTTP; - if (is_transport_allowed("https")) - allowed_protocols |= CURLPROTO_HTTPS; - if (is_transport_allowed("ftp")) - allowed_protocols |= CURLPROTO_FTP; - if (is_transport_allowed("ftps")) - allowed_protocols |= CURLPROTO_FTPS; - curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); - curl_easy_setopt(result, CURLOPT_PROTOCOLS, allowed_protocols); + curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, + get_curl_allowed_protocols()); + curl_easy_setopt(result, CURLOPT_PROTOCOLS, + get_curl_allowed_protocols()); #else warning("protocol restrictions not applied to curl redirects because\n" "your curl version is too old (>= 7.19.4)"); -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v10 5/6] transport: add from_user parameter to is_transport_allowed 2016-12-14 22:39 ` [PATCH v10 0/6] transport protocol policy configuration Brandon Williams ` (3 preceding siblings ...) 2016-12-14 22:39 ` [PATCH v10 4/6] http: create function to get curl allowed protocols Brandon Williams @ 2016-12-14 22:39 ` Brandon Williams 2016-12-14 22:39 ` [PATCH v10 6/6] http: respect protocol.*.allow=user for http-alternates Brandon Williams ` (2 subsequent siblings) 7 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 22:39 UTC (permalink / raw) To: git; +Cc: Brandon Williams, gitster, peff, sbeller, bburky, jrnieder Add a from_user parameter to is_transport_allowed() to allow http to be able to distinguish between protocol restrictions for redirects versus initial requests. CURLOPT_REDIR_PROTOCOLS can now be set differently from CURLOPT_PROTOCOLS to disallow use of protocols with the "user" policy in redirects. This change allows callers to query if a transport protocol is allowed, given that the caller knows that the protocol is coming from the user (1) or not from the user (0) such as redirects in libcurl. If unknown a -1 should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER` to determine if the protocol came from the user. Signed-off-by: Brandon Williams <bmwill@google.com> --- http.c | 14 +++++++------- t/t5812-proto-disable-http.sh | 7 +++++++ transport.c | 8 +++++--- transport.h | 13 ++++++++++--- 4 files changed, 29 insertions(+), 13 deletions(-) diff --git a/http.c b/http.c index f7c488a..2208269 100644 --- a/http.c +++ b/http.c @@ -489,17 +489,17 @@ static void set_curl_keepalive(CURL *c) } #endif -static long get_curl_allowed_protocols(void) +static long get_curl_allowed_protocols(int from_user) { long allowed_protocols = 0; - if (is_transport_allowed("http")) + if (is_transport_allowed("http", from_user)) allowed_protocols |= CURLPROTO_HTTP; - if (is_transport_allowed("https")) + if (is_transport_allowed("https", from_user)) allowed_protocols |= CURLPROTO_HTTPS; - if (is_transport_allowed("ftp")) + if (is_transport_allowed("ftp", from_user)) allowed_protocols |= CURLPROTO_FTP; - if (is_transport_allowed("ftps")) + if (is_transport_allowed("ftps", from_user)) allowed_protocols |= CURLPROTO_FTPS; return allowed_protocols; @@ -588,9 +588,9 @@ static CURL *get_curl_handle(void) #endif #if LIBCURL_VERSION_NUM >= 0x071304 curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, - get_curl_allowed_protocols()); + get_curl_allowed_protocols(0)); curl_easy_setopt(result, CURLOPT_PROTOCOLS, - get_curl_allowed_protocols()); + get_curl_allowed_protocols(-1)); #else warning("protocol restrictions not applied to curl redirects because\n" "your curl version is too old (>= 7.19.4)"); diff --git a/t/t5812-proto-disable-http.sh b/t/t5812-proto-disable-http.sh index 044cc15..d911afd 100755 --- a/t/t5812-proto-disable-http.sh +++ b/t/t5812-proto-disable-http.sh @@ -30,5 +30,12 @@ test_expect_success 'curl limits redirects' ' test_must_fail git clone "$HTTPD_URL/loop-redir/smart/repo.git" ' +test_expect_success 'http can be limited to from-user' ' + git -c protocol.http.allow=user \ + clone "$HTTPD_URL/smart/repo.git" plain.git && + test_must_fail git -c protocol.http.allow=user \ + clone "$HTTPD_URL/smart-redir-perm/repo.git" redir.git +' + stop_httpd test_done diff --git a/transport.c b/transport.c index fbd799d..f50c31a 100644 --- a/transport.c +++ b/transport.c @@ -676,7 +676,7 @@ static enum protocol_allow_config get_protocol_config(const char *type) return PROTOCOL_ALLOW_USER_ONLY; } -int is_transport_allowed(const char *type) +int is_transport_allowed(const char *type, int from_user) { const struct string_list *whitelist = protocol_whitelist(); if (whitelist) @@ -688,7 +688,9 @@ int is_transport_allowed(const char *type) case PROTOCOL_ALLOW_NEVER: return 0; case PROTOCOL_ALLOW_USER_ONLY: - return git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + if (from_user < 0) + from_user = git_env_bool("GIT_PROTOCOL_FROM_USER", 1); + return from_user; } die("BUG: invalid protocol_allow_config type"); @@ -696,7 +698,7 @@ int is_transport_allowed(const char *type) void transport_check_allowed(const char *type) { - if (!is_transport_allowed(type)) + if (!is_transport_allowed(type, -1)) die("transport '%s' not allowed", type); } diff --git a/transport.h b/transport.h index 3396e1d..4f1c801 100644 --- a/transport.h +++ b/transport.h @@ -142,10 +142,17 @@ struct transport { struct transport *transport_get(struct remote *, const char *); /* - * Check whether a transport is allowed by the environment. Type should - * generally be the URL scheme, as described in Documentation/git.txt + * Check whether a transport is allowed by the environment. + * + * Type should generally be the URL scheme, as described in + * Documentation/git.txt + * + * from_user specifies if the transport was given by the user. If unknown pass + * a -1 to read from the environment to determine if the transport was given by + * the user. + * */ -int is_transport_allowed(const char *type); +int is_transport_allowed(const char *type, int from_user); /* * Check whether a transport is allowed by the environment, -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH v10 6/6] http: respect protocol.*.allow=user for http-alternates 2016-12-14 22:39 ` [PATCH v10 0/6] transport protocol policy configuration Brandon Williams ` (4 preceding siblings ...) 2016-12-14 22:39 ` [PATCH v10 5/6] transport: add from_user parameter to is_transport_allowed Brandon Williams @ 2016-12-14 22:39 ` Brandon Williams 2016-12-14 23:25 ` [PATCH v10 0/6] transport protocol policy configuration Junio C Hamano 2016-12-15 0:22 ` Jeff King 7 siblings, 0 replies; 124+ messages in thread From: Brandon Williams @ 2016-12-14 22:39 UTC (permalink / raw) To: git; +Cc: Jeff King, gitster, sbeller, bburky, jrnieder From: Jeff King <peff@peff.net> The http-walker may fetch the http-alternates (or alternates) file from a remote in order to find more objects. This should count as a "not from the user" use of the protocol. But because we implement the redirection ourselves and feed the new URL to curl, it will use the CURLOPT_PROTOCOLS rules, not the more restrictive CURLOPT_REDIR_PROTOCOLS. The ideal solution would be for each curl request we make to know whether or not is directly from the user or part of an alternates redirect, and then set CURLOPT_PROTOCOLS as appropriate. However, that would require plumbing that information through all of the various layers of the http code. Instead, let's check the protocol at the source: when we are parsing the remote http-alternates file. The only downside is that if there's any mismatch between what protocol we think it is versus what curl thinks it is, it could violate the policy. To address this, we'll make the parsing err on the picky side, and only allow protocols that it can parse definitively. So for example, you can't elude the "http" policy by asking for "HTTP://", even though curl might handle it; we would reject it as unknown. The only unsafe case would be if you have a URL that starts with "http://" but curl interprets as another protocol. That seems like an unlikely failure mode (and we are still protected by our base CURLOPT_PROTOCOL setting, so the worst you could do is trigger one of https, ftp, or ftps). Signed-off-by: Jeff King <peff@peff.net> --- http-walker.c | 52 ++++++++++++++++++++++++++++++++++++---------- t/t5550-http-fetch-dumb.sh | 10 +++++++++ 2 files changed, 51 insertions(+), 11 deletions(-) diff --git a/http-walker.c b/http-walker.c index c2f81cd..b34b6ac 100644 --- a/http-walker.c +++ b/http-walker.c @@ -3,6 +3,7 @@ #include "walker.h" #include "http.h" #include "list.h" +#include "transport.h" struct alt_base { char *base; @@ -160,6 +161,32 @@ static void prefetch(struct walker *walker, unsigned char *sha1) #endif } +static int is_alternate_allowed(const char *url) +{ + const char *protocols[] = { + "http", "https", "ftp", "ftps" + }; + int i; + + for (i = 0; i < ARRAY_SIZE(protocols); i++) { + const char *end; + if (skip_prefix(url, protocols[i], &end) && + starts_with(end, "://")) + break; + } + + if (i >= ARRAY_SIZE(protocols)) { + warning("ignoring alternate with unknown protocol: %s", url); + return 0; + } + if (!is_transport_allowed(protocols[i], 0)) { + warning("ignoring alternate with restricted protocol: %s", url); + return 0; + } + + return 1; +} + static void process_alternates_response(void *callback_data) { struct alternates_request *alt_req = @@ -274,17 +301,20 @@ static void process_alternates_response(void *callback_data) struct strbuf target = STRBUF_INIT; strbuf_add(&target, base, serverlen); strbuf_add(&target, data + i, posn - i - 7); - warning("adding alternate object store: %s", - target.buf); - newalt = xmalloc(sizeof(*newalt)); - newalt->next = NULL; - newalt->base = strbuf_detach(&target, NULL); - newalt->got_indices = 0; - newalt->packs = NULL; - - while (tail->next != NULL) - tail = tail->next; - tail->next = newalt; + + if (is_alternate_allowed(target.buf)) { + warning("adding alternate object store: %s", + target.buf); + newalt = xmalloc(sizeof(*newalt)); + newalt->next = NULL; + newalt->base = strbuf_detach(&target, NULL); + newalt->got_indices = 0; + newalt->packs = NULL; + + while (tail->next != NULL) + tail = tail->next; + tail->next = newalt; + } } } i = posn + 1; diff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh index 22011f0..c0ee29c 100755 --- a/t/t5550-http-fetch-dumb.sh +++ b/t/t5550-http-fetch-dumb.sh @@ -360,5 +360,15 @@ test_expect_success 'http-alternates cannot point at funny protocols' ' clone "$HTTPD_URL/dumb/evil.git" evil-file ' +test_expect_success 'http-alternates triggers not-from-user protocol check' ' + echo "$HTTPD_URL/dumb/victim.git/objects" \ + >"$evil/objects/info/http-alternates" && + test_config_global http.followRedirects true && + test_must_fail git -c protocol.http.allow=user \ + clone $HTTPD_URL/dumb/evil.git evil-user && + git -c protocol.http.allow=always \ + clone $HTTPD_URL/dumb/evil.git evil-user +' + stop_httpd test_done -- 2.8.0.rc3.226.g39d4020 ^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH v10 0/6] transport protocol policy configuration 2016-12-14 22:39 ` [PATCH v10 0/6] transport protocol policy configuration Brandon Williams ` (5 preceding siblings ...) 2016-12-14 22:39 ` [PATCH v10 6/6] http: respect protocol.*.allow=user for http-alternates Brandon Williams @ 2016-12-14 23:25 ` Junio C Hamano 2016-12-15 0:22 ` Jeff King 7 siblings, 0 replies; 124+ messages in thread From: Junio C Hamano @ 2016-12-14 23:25 UTC (permalink / raw) To: Brandon Williams; +Cc: git, peff, sbeller, bburky, jrnieder Brandon Williams <bmwill@google.com> writes: > v10 of this series fixes the following: > * A few updates to the commit messages in order to better convey the reasoning > behind the a few of the patches. > * Additional test to verify that curl redirects respect configured protocol > policies. > * Patch added by Jeff King to make http alternates respect configured > protocol policies. Thanks. Will replace the previous one. ^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH v10 0/6] transport protocol policy configuration 2016-12-14 22:39 ` [PATCH v10 0/6] transport protocol policy configuration Brandon Williams ` (6 preceding siblings ...) 2016-12-14 23:25 ` [PATCH v10 0/6] transport protocol policy configuration Junio C Hamano @ 2016-12-15 0:22 ` Jeff King 7 siblings, 0 replies; 124+ messages in thread From: Jeff King @ 2016-12-15 0:22 UTC (permalink / raw) To: Brandon Williams; +Cc: git, gitster, sbeller, bburky, jrnieder On Wed, Dec 14, 2016 at 02:39:49PM -0800, Brandon Williams wrote: > v10 of this series fixes the following: > * A few updates to the commit messages in order to better convey the reasoning > behind the a few of the patches. > * Additional test to verify that curl redirects respect configured protocol > policies. > * Patch added by Jeff King to make http alternates respect configured > protocol policies. Thanks, this one looks fine to me. -Peff ^ permalink raw reply [flat|nested] 124+ messages in thread
end of thread, other threads:[~2016-12-15 17:29 UTC | newest] Thread overview: 124+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-11-02 22:20 [PATCH] transport: add core.allowProtocol config option Brandon Williams 2016-11-02 22:41 ` Stefan Beller 2016-11-02 22:47 ` Brandon Williams 2016-11-02 23:05 ` Jeff King 2016-11-02 23:08 ` Jeff King 2016-11-02 23:34 ` Brandon Williams 2016-11-02 23:33 ` Brandon Williams 2016-11-02 23:46 ` Brandon Williams 2016-11-03 0:08 ` Jeff King 2016-11-03 0:22 ` Jonathan Nieder 2016-11-03 0:41 ` Blake Burkhart 2016-11-03 2:44 ` Junio C Hamano 2016-11-03 14:38 ` Jeff King 2016-11-03 17:25 ` Brandon Williams 2016-11-03 17:39 ` Stefan Beller 2016-11-03 17:51 ` Brandon Williams 2016-11-03 18:02 ` Jeff King 2016-11-03 18:08 ` Brandon Williams 2016-11-03 18:00 ` Jeff King 2016-11-03 17:53 ` Jeff King 2016-11-03 18:19 ` Brandon Williams 2016-11-03 18:25 ` Jeff King 2016-11-03 18:24 ` Jeff King 2016-11-03 18:45 ` Brandon Williams 2016-11-03 18:50 ` Jeff King 2016-11-03 18:56 ` Brandon Williams 2016-11-03 0:50 ` [PATCH v2] " Brandon Williams 2016-11-04 20:55 ` [PATCH v3] transport: add protocol policy " Brandon Williams 2016-11-04 20:58 ` Brandon Williams 2016-11-04 21:35 ` Stefan Beller 2016-11-04 23:09 ` Jeff King 2016-11-05 0:18 ` Brandon Williams 2016-11-04 22:38 ` Stefan Beller 2016-11-07 18:14 ` Brandon Williams 2016-11-04 23:06 ` Jeff King 2016-11-07 19:17 ` Brandon Williams 2016-11-07 19:35 ` [PATCH v4 1/2] lib-proto-disable: variable name fix Brandon Williams 2016-11-07 19:35 ` [PATCH v4 2/2] transport: add protocol policy config option Brandon Williams 2016-11-07 20:44 ` Jeff King 2016-11-07 21:02 ` Brandon Williams 2016-11-07 20:26 ` [PATCH v4 1/2] lib-proto-disable: variable name fix Jeff King 2016-11-07 20:40 ` Brandon Williams 2016-11-07 20:48 ` Jeff King 2016-11-08 3:32 ` Jacob Keller 2016-11-08 20:52 ` Junio C Hamano 2016-11-07 21:51 ` [PATCH v5 " Brandon Williams 2016-11-07 21:51 ` [PATCH v5 2/2] transport: add protocol policy config option Brandon Williams 2016-11-08 22:04 ` Jeff King 2016-11-08 22:05 ` Brandon Williams 2016-12-01 19:44 ` [PATCH v6 0/4] transport protocol policy configuration Brandon Williams 2016-12-01 19:44 ` [PATCH v6 1/4] lib-proto-disable: variable name fix Brandon Williams 2016-12-01 19:44 ` [PATCH v6 2/4] transport: add protocol policy config option Brandon Williams 2016-12-01 19:44 ` [PATCH v6 3/4] http: always warn if libcurl version is too old Brandon Williams 2016-12-01 19:44 ` [PATCH v6 4/4] transport: check if protocol can be used on a redirect Brandon Williams 2016-12-01 19:48 ` Brandon Williams 2016-12-01 19:49 ` Brandon Williams 2016-12-01 19:50 ` Jeff King 2016-12-01 19:54 ` Junio C Hamano 2016-12-01 19:59 ` Jeff King 2016-12-01 20:01 ` Brandon Williams 2016-12-01 20:25 ` [PATCH v7 0/4] transport protocol policy configuration Brandon Williams 2016-12-01 20:25 ` [PATCH v7 1/4] lib-proto-disable: variable name fix Brandon Williams 2016-12-01 20:25 ` [PATCH v7 2/4] transport: add protocol policy config option Brandon Williams 2016-12-01 20:25 ` [PATCH v7 3/4] http: always warn if libcurl version is too old Brandon Williams 2016-12-01 20:25 ` [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed Brandon Williams 2016-12-01 21:40 ` Jeff King 2016-12-01 22:25 ` Junio C Hamano 2016-12-01 23:07 ` Brandon Williams 2016-12-01 23:26 ` Brandon Williams 2016-12-02 0:13 ` Jeff King 2016-12-02 17:33 ` Brandon Williams 2016-12-01 23:34 ` Junio C Hamano 2016-12-01 23:58 ` Brandon Williams 2016-12-05 20:04 ` Junio C Hamano 2016-12-05 22:22 ` Brandon Williams 2016-12-05 23:19 ` Junio C Hamano 2016-12-05 23:22 ` Brandon Williams 2016-12-06 13:51 ` Jeff King 2016-12-06 17:53 ` Junio C Hamano 2016-12-06 18:10 ` Jeff King 2016-12-06 18:24 ` [PATCH v2] jk/http-walker-limit-redirect rebased to maint-2.9 Jeff King 2016-12-06 18:24 ` [PATCH v2 1/6] http: simplify update_url_from_redirect Jeff King 2016-12-06 18:24 ` [PATCH v2 2/6] http: always update the base URL for redirects Jeff King 2016-12-06 18:24 ` [PATCH v2 3/6] remote-curl: rename shadowed options variable Jeff King 2016-12-06 18:24 ` [PATCH v2 4/6] http: make redirects more obvious Jeff King 2016-12-06 18:24 ` [PATCH v2 5/6] http: treat http-alternates like redirects Jeff King 2016-12-06 18:25 ` [PATCH v2 6/6] http-walker: complain about non-404 loose object errors Jeff King 2016-12-06 22:24 ` [PATCH v7 4/4] transport: add from_user parameter to is_transport_allowed Brandon Williams 2016-12-02 0:00 ` [PATCH v8 0/5] transport protocol policy configuration Brandon Williams 2016-12-02 0:00 ` [PATCH v8 1/5] lib-proto-disable: variable name fix Brandon Williams 2016-12-02 0:00 ` [PATCH v8 2/5] transport: add protocol policy config option Brandon Williams 2016-12-02 0:01 ` [PATCH v8 3/5] http: always warn if libcurl version is too old Brandon Williams 2016-12-02 0:01 ` [PATCH v8 4/5] http: create function to get curl allowed protocols Brandon Williams 2016-12-02 0:01 ` [PATCH v8 5/5] transport: add from_user parameter to is_transport_allowed Brandon Williams 2016-12-14 1:40 ` [PATCH v9 0/5] transport protocol policy configuration Brandon Williams 2016-12-14 1:40 ` [PATCH v9 1/5] lib-proto-disable: variable name fix Brandon Williams 2016-12-14 1:40 ` [PATCH v9 2/5] transport: add protocol policy config option Brandon Williams 2016-12-14 1:40 ` [PATCH v9 3/5] http: always warn if libcurl version is too old Brandon Williams 2016-12-14 16:01 ` Jeff King 2016-12-14 17:56 ` Brandon Williams 2016-12-14 1:40 ` [PATCH v9 4/5] http: create function to get curl allowed protocols Brandon Williams 2016-12-14 16:03 ` Jeff King 2016-12-14 18:00 ` Brandon Williams 2016-12-14 1:40 ` [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed Brandon Williams 2016-12-14 16:40 ` Jeff King 2016-12-14 20:13 ` Brandon Williams 2016-12-14 20:33 ` Jeff King 2016-12-14 21:12 ` Jeff King 2016-12-14 21:58 ` Brandon Williams [not found] ` <CAP3OtXiOPbAkr5Mn+5tEmZZAZzJXQ4CvtpHCg=wt+k-bi6K2vA@mail.gmail.com> [not found] ` <CAP3OtXhH++szRws20MaHt-ftLBMUJuYiTmfL50mOFP4FA4Mn6Q@mail.gmail.com> 2016-12-14 22:52 ` Jeff King 2016-12-14 20:37 ` Brandon Williams 2016-12-14 20:41 ` Jeff King 2016-12-14 20:50 ` Brandon Williams 2016-12-14 22:39 ` [PATCH v10 0/6] transport protocol policy configuration Brandon Williams 2016-12-14 22:39 ` [PATCH v10 1/6] lib-proto-disable: variable name fix Brandon Williams 2016-12-14 22:39 ` [PATCH v10 2/6] http: always warn if libcurl version is too old Brandon Williams 2016-12-15 0:21 ` Jeff King 2016-12-15 17:29 ` Junio C Hamano 2016-12-14 22:39 ` [PATCH v10 3/6] transport: add protocol policy config option Brandon Williams 2016-12-14 22:39 ` [PATCH v10 4/6] http: create function to get curl allowed protocols Brandon Williams 2016-12-14 22:39 ` [PATCH v10 5/6] transport: add from_user parameter to is_transport_allowed Brandon Williams 2016-12-14 22:39 ` [PATCH v10 6/6] http: respect protocol.*.allow=user for http-alternates Brandon Williams 2016-12-14 23:25 ` [PATCH v10 0/6] transport protocol policy configuration Junio C Hamano 2016-12-15 0:22 ` Jeff King
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.