[PATCH 1/6] oe-git-proxy: allow setting SOCAT from outside

[PATCH 1/6] oe-git-proxy: allow setting SOCAT from outside

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This allows to write selftests where we can mock the real socat.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/oe-git-proxy | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
index bb2ed2a46e..8499a99a71 100755
--- a/scripts/oe-git-proxy
+++ b/scripts/oe-git-proxy
@@ -41,10 +41,12 @@ if [ $# -lt 2 -o "$1" = '--help' -o "$1" = '-h' ] ; then
 fi
 
 # Locate the netcat binary
-SOCAT=$(which socat 2>/dev/null)
-if [ $? -ne 0 ]; then
-	echo "ERROR: socat binary not in PATH" 1>&2
-	exit 1
+if [ -z "$SOCAT" ]; then
+	SOCAT=$(which socat 2>/dev/null)
+	if [ $? -ne 0 ]; then
+		echo "ERROR: socat binary not in PATH" 1>&2
+		exit 1
+	fi
 fi
 METHOD=""
 
-- 
2.21.0

[PATCH 2/6] oeqa: add case for oe-git-proxy

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

The escaping, splitting and matching of NO_PROXY in oe-git-proxy
deserves its own testcase, add it.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/lib/oeqa/selftest/cases/oescripts.py | 58 +++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/oescripts.py b/meta/lib/oeqa/selftest/cases/oescripts.py
index 7770b66a26..c169885cf3 100644
--- a/meta/lib/oeqa/selftest/cases/oescripts.py
+++ b/meta/lib/oeqa/selftest/cases/oescripts.py
@@ -3,10 +3,12 @@
 #
 
 import os
+import shutil
 import unittest
 from oeqa.selftest.case import OESelftestTestCase
 from oeqa.selftest.cases.buildhistory import BuildhistoryBase
 from oeqa.utils.commands import Command, runCmd, bitbake, get_bb_var, get_test_layer
+from oeqa.utils import CommandError
 
 class BuildhistoryDiffTests(BuildhistoryBase):
 
@@ -63,3 +65,59 @@ class OEPybootchartguyTests(OEScriptTests):
         runCmd('%s/pybootchartgui/pybootchartgui.py  %s -o %s/charts -f pdf' % (self.scripts_dir, self.buildstats, self.tmpdir))
         self.assertTrue(os.path.exists(self.tmpdir + "/charts.pdf"))
 
+class OEGitproxyTests(OESelftestTestCase):
+
+    scripts_dir = os.path.join(get_bb_var('COREBASE'), 'scripts')
+
+    def test_oegitproxy_help(self):
+        try:
+            res = runCmd('%s/oe-git-proxy  --help' % self.scripts_dir, assert_error=False)
+            self.assertTrue(False)
+        except CommandError as e:
+            self.assertEqual(2, e.retcode)
+
+    def run_oegitproxy(self, custom_shell=None):
+        os.environ['SOCAT'] = shutil.which("echo")
+        os.environ['ALL_PROXY'] = "https://proxy.example.com:3128"
+        os.environ['NO_PROXY'] = "*.example.com,.no-proxy.org,192.168.42.0/24,127.*.*.*"
+
+        if custom_shell is None:
+            prefix = ''
+        else:
+            prefix = custom_shell + ' '
+
+        # outside, use the proxy
+        res = runCmd('%s%s/oe-git-proxy host.outside-example.com 9418' %
+                     (prefix,self.scripts_dir))
+        self.assertIn('PROXY:', res.output)
+        # match with wildcard suffix
+        res = runCmd('%s%s/oe-git-proxy host.example.com 9418' %
+                     (prefix, self.scripts_dir))
+        self.assertIn('TCP:', res.output)
+        # match just suffix
+        res = runCmd('%s%s/oe-git-proxy host.no-proxy.org 9418' %
+                     (prefix, self.scripts_dir))
+        self.assertIn('TCP:', res.output)
+        # match IP subnet
+        res = runCmd('%s%s/oe-git-proxy 192.168.42.42 9418' %
+                     (prefix, self.scripts_dir))
+        self.assertIn('TCP:', res.output)
+        # match IP wildcard
+        res = runCmd('%s%s/oe-git-proxy 127.1.2.3 9418' %
+                     (prefix, self.scripts_dir))
+        self.assertIn('TCP:', res.output)
+        
+        # test that * globbering is off
+        os.environ['NO_PROXY'] = "*"
+        res = runCmd('%s%s/oe-git-proxy host.example.com 9418' %
+                     (prefix, self.scripts_dir))
+        self.assertIn('TCP:', res.output)
+
+    def test_oegitproxy_proxy(self):
+        self.run_oegitproxy()
+
+    def test_oegitproxy_proxy_dash(self):
+        dash = shutil.which("dash")
+        if dash is None:
+            self.skipTest("No \"dash\" found on test system.")
+        self.run_oegitproxy(custom_shell=dash)
-- 
2.21.0

[PATCH 3/6] Revert "oe-git-proxy: Avoid resolving NO_PROXY against local files"

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This reverts commit cbc148d5d93d5f3531434fee7b234a16196b3088.

The quoting causes H to be one string with spaces, so looping over
multiple entries does not work anymore.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/oe-git-proxy | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
index 8499a99a71..10e6560da4 100755
--- a/scripts/oe-git-proxy
+++ b/scripts/oe-git-proxy
@@ -134,8 +134,8 @@ if [ -z "$ALL_PROXY" ]; then
 fi
 
 # Connect directly to hosts in NO_PROXY
-for H in "${NO_PROXY//,/ }"; do
-	if match_host $1 "$H"; then
+for H in ${NO_PROXY//,/ }; do
+	if match_host $1 $H; then
 		exec $SOCAT STDIO $METHOD
 	fi
 done
-- 
2.21.0

[PATCH 4/6] oe-git-proxy: disable shell pathname expansion for the whole script

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This truly fixes the issue that cbc148d5d93d5f3531434fee7b234a16196b3088
wanted to solve, without breaking the iteration over multiple entries.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/oe-git-proxy | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
index 10e6560da4..2f65f0c482 100755
--- a/scripts/oe-git-proxy
+++ b/scripts/oe-git-proxy
@@ -133,6 +133,9 @@ if [ -z "$ALL_PROXY" ]; then
 	exec $SOCAT STDIO $METHOD
 fi
 
+# disable pathname expansion, NO_PROXY fielnd could start with "*" or be it
+set -f
+
 # Connect directly to hosts in NO_PROXY
 for H in ${NO_PROXY//,/ }; do
 	if match_host $1 $H; then
-- 
2.21.0

[PATCH 5/6] oe-git-proxy: NO_PROXY suffix matching without wildcard for match_host

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

NO_PROXY can also contain just suffixes that do not start with a "*". We
failed to match those so far. Just add an extra "*" to also match those
suffixes. If one was there we get "**" which does not hurt.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/oe-git-proxy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
index 2f65f0c482..8f7b26c15e 100755
--- a/scripts/oe-git-proxy
+++ b/scripts/oe-git-proxy
@@ -104,7 +104,7 @@ match_host() {
 	HOST=$1
 	GLOB=$2
 
-	if [ -z "${HOST%%$GLOB}" ]; then
+	if [ -z "${HOST%%*$GLOB}" ]; then
 		return 0
 	fi
 
-- 
2.21.0

[PATCH 6/6] oe-git-proxy: fix dash "Bad substitution"

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

The script claims it works with dash, make sure that is actually the
case.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/oe-git-proxy | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
index 8f7b26c15e..2792020467 100755
--- a/scripts/oe-git-proxy
+++ b/scripts/oe-git-proxy
@@ -61,7 +61,7 @@ ipv4_val() {
 	IP="$1"
 	SHIFT=24
 	VAL=0
-	for B in ${IP//./ }; do
+	for B in $( echo "$IP" | tr '.' ' ' ); do
 		VAL=$(($VAL+$(($B<<$SHIFT))))
 		SHIFT=$(($SHIFT-8))
 	done
@@ -136,8 +136,9 @@ fi
 # disable pathname expansion, NO_PROXY fielnd could start with "*" or be it
 set -f
 
+
 # Connect directly to hosts in NO_PROXY
-for H in ${NO_PROXY//,/ }; do
+for H in $( echo "$NO_PROXY" | tr ',' ' ' ); do
 	if match_host $1 $H; then
 		exec $SOCAT STDIO $METHOD
 	fi
-- 
2.21.0

Re: [PATCH 1/6] oe-git-proxy: allow setting SOCAT from outside

[Henning Schild]

This series fixes several issues in oe-git-proxy, mostly related to
matching NO_PROXY entries.

Patch 2 introduces a test that will actually fail until all other
patches are applied. If that is a problem and every commit should be
testable, Patch2 needs to be moved to then end.

Let me know what you think

Henning

Am Tue, 3 Sep 2019 14:22:19 +0200
schrieb Henning Schild <henning.schild@siemens.com>:

> From: Henning Schild <henning.schild@siemens.com>
> 
> This allows to write selftests where we can mock the real socat.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  scripts/oe-git-proxy | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
> index bb2ed2a46e..8499a99a71 100755
> --- a/scripts/oe-git-proxy
> +++ b/scripts/oe-git-proxy
> @@ -41,10 +41,12 @@ if [ $# -lt 2 -o "$1" = '--help' -o "$1" =
> '-h' ] ; then fi
>  
>  # Locate the netcat binary
> -SOCAT=$(which socat 2>/dev/null)
> -if [ $? -ne 0 ]; then
> -	echo "ERROR: socat binary not in PATH" 1>&2
> -	exit 1
> +if [ -z "$SOCAT" ]; then
> +	SOCAT=$(which socat 2>/dev/null)
> +	if [ $? -ne 0 ]; then
> +		echo "ERROR: socat binary not in PATH" 1>&2
> +		exit 1
> +	fi
>  fi
>  METHOD=""
>  

Re: [PATCH 4/6] oe-git-proxy: disable shell pathname expansion for the whole script

[Jan Kiszka]

On 03.09.19 14:22, Henning Schild wrote:
> From: Henning Schild <henning.schild@siemens.com>
> 
> This truly fixes the issue that cbc148d5d93d5f3531434fee7b234a16196b3088
> wanted to solve, without breaking the iteration over multiple entries.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   scripts/oe-git-proxy | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
> index 10e6560da4..2f65f0c482 100755
> --- a/scripts/oe-git-proxy
> +++ b/scripts/oe-git-proxy
> @@ -133,6 +133,9 @@ if [ -z "$ALL_PROXY" ]; then
>   	exec $SOCAT STDIO $METHOD
>   fi
>   
> +# disable pathname expansion, NO_PROXY fielnd could start with "*" or be it

field

> +set -f

Ah, that's much better than any (mis-)quoting.

Jan

> +
>   # Connect directly to hosts in NO_PROXY
>   for H in ${NO_PROXY//,/ }; do
>   	if match_host $1 $H; then
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux