[PATCHv2 0/6] solve several issues in oe-git-proxy

[PATCHv2 0/6] solve several issues in oe-git-proxy

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

Change to v1:
 - fix comment in p4
 - move "set -f" all the way up in p4, that was the intention in the
   first place

Henning Schild (6):
  oe-git-proxy: allow setting SOCAT from outside
  oeqa: add case for oe-git-proxy
  Revert "oe-git-proxy: Avoid resolving NO_PROXY against local files"
  oe-git-proxy: disable shell pathname expansion for the whole script
  oe-git-proxy: NO_PROXY suffix matching without wildcard for match_host
  oe-git-proxy: fix dash "Bad substitution"

 meta/lib/oeqa/selftest/cases/oescripts.py | 58 +++++++++++++++++++++++
 scripts/oe-git-proxy                      | 21 ++++----
 2 files changed, 71 insertions(+), 8 deletions(-)

-- 
2.21.0

[PATCHv2 1/6] oe-git-proxy: allow setting SOCAT from outside

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This allows to write selftests where we can mock the real socat.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/oe-git-proxy | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
index bb2ed2a46e..8499a99a71 100755
--- a/scripts/oe-git-proxy
+++ b/scripts/oe-git-proxy
@@ -41,10 +41,12 @@ if [ $# -lt 2 -o "$1" = '--help' -o "$1" = '-h' ] ; then
 fi
 
 # Locate the netcat binary
-SOCAT=$(which socat 2>/dev/null)
-if [ $? -ne 0 ]; then
-	echo "ERROR: socat binary not in PATH" 1>&2
-	exit 1
+if [ -z "$SOCAT" ]; then
+	SOCAT=$(which socat 2>/dev/null)
+	if [ $? -ne 0 ]; then
+		echo "ERROR: socat binary not in PATH" 1>&2
+		exit 1
+	fi
 fi
 METHOD=""
 
-- 
2.21.0

[PATCHv2 2/6] oeqa: add case for oe-git-proxy

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

The escaping, splitting and matching of NO_PROXY in oe-git-proxy
deserves its own testcase, add it.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/lib/oeqa/selftest/cases/oescripts.py | 58 +++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/oescripts.py b/meta/lib/oeqa/selftest/cases/oescripts.py
index 7770b66a26..c169885cf3 100644
--- a/meta/lib/oeqa/selftest/cases/oescripts.py
+++ b/meta/lib/oeqa/selftest/cases/oescripts.py
@@ -3,10 +3,12 @@
 #
 
 import os
+import shutil
 import unittest
 from oeqa.selftest.case import OESelftestTestCase
 from oeqa.selftest.cases.buildhistory import BuildhistoryBase
 from oeqa.utils.commands import Command, runCmd, bitbake, get_bb_var, get_test_layer
+from oeqa.utils import CommandError
 
 class BuildhistoryDiffTests(BuildhistoryBase):
 
@@ -63,3 +65,59 @@ class OEPybootchartguyTests(OEScriptTests):
         runCmd('%s/pybootchartgui/pybootchartgui.py  %s -o %s/charts -f pdf' % (self.scripts_dir, self.buildstats, self.tmpdir))
         self.assertTrue(os.path.exists(self.tmpdir + "/charts.pdf"))
 
+class OEGitproxyTests(OESelftestTestCase):
+
+    scripts_dir = os.path.join(get_bb_var('COREBASE'), 'scripts')
+
+    def test_oegitproxy_help(self):
+        try:
+            res = runCmd('%s/oe-git-proxy  --help' % self.scripts_dir, assert_error=False)
+            self.assertTrue(False)
+        except CommandError as e:
+            self.assertEqual(2, e.retcode)
+
+    def run_oegitproxy(self, custom_shell=None):
+        os.environ['SOCAT'] = shutil.which("echo")
+        os.environ['ALL_PROXY'] = "https://proxy.example.com:3128"
+        os.environ['NO_PROXY'] = "*.example.com,.no-proxy.org,192.168.42.0/24,127.*.*.*"
+
+        if custom_shell is None:
+            prefix = ''
+        else:
+            prefix = custom_shell + ' '
+
+        # outside, use the proxy
+        res = runCmd('%s%s/oe-git-proxy host.outside-example.com 9418' %
+                     (prefix,self.scripts_dir))
+        self.assertIn('PROXY:', res.output)
+        # match with wildcard suffix
+        res = runCmd('%s%s/oe-git-proxy host.example.com 9418' %
+                     (prefix, self.scripts_dir))
+        self.assertIn('TCP:', res.output)
+        # match just suffix
+        res = runCmd('%s%s/oe-git-proxy host.no-proxy.org 9418' %
+                     (prefix, self.scripts_dir))
+        self.assertIn('TCP:', res.output)
+        # match IP subnet
+        res = runCmd('%s%s/oe-git-proxy 192.168.42.42 9418' %
+                     (prefix, self.scripts_dir))
+        self.assertIn('TCP:', res.output)
+        # match IP wildcard
+        res = runCmd('%s%s/oe-git-proxy 127.1.2.3 9418' %
+                     (prefix, self.scripts_dir))
+        self.assertIn('TCP:', res.output)
+        
+        # test that * globbering is off
+        os.environ['NO_PROXY'] = "*"
+        res = runCmd('%s%s/oe-git-proxy host.example.com 9418' %
+                     (prefix, self.scripts_dir))
+        self.assertIn('TCP:', res.output)
+
+    def test_oegitproxy_proxy(self):
+        self.run_oegitproxy()
+
+    def test_oegitproxy_proxy_dash(self):
+        dash = shutil.which("dash")
+        if dash is None:
+            self.skipTest("No \"dash\" found on test system.")
+        self.run_oegitproxy(custom_shell=dash)
-- 
2.21.0

[PATCHv2 3/6] Revert "oe-git-proxy: Avoid resolving NO_PROXY against local files"

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This reverts commit cbc148d5d93d5f3531434fee7b234a16196b3088.

The quoting causes H to be one string with spaces, so looping over
multiple entries does not work anymore.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/oe-git-proxy | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
index 8499a99a71..10e6560da4 100755
--- a/scripts/oe-git-proxy
+++ b/scripts/oe-git-proxy
@@ -134,8 +134,8 @@ if [ -z "$ALL_PROXY" ]; then
 fi
 
 # Connect directly to hosts in NO_PROXY
-for H in "${NO_PROXY//,/ }"; do
-	if match_host $1 "$H"; then
+for H in ${NO_PROXY//,/ }; do
+	if match_host $1 $H; then
 		exec $SOCAT STDIO $METHOD
 	fi
 done
-- 
2.21.0

[PATCHv2 4/6] oe-git-proxy: disable shell pathname expansion for the whole script

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This truly fixes the issue that cbc148d5d93d5f3531434fee7b234a16196b3088
wanted to solve, without breaking the iteration over multiple entries.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/oe-git-proxy | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
index 10e6560da4..9eeef45dcf 100755
--- a/scripts/oe-git-proxy
+++ b/scripts/oe-git-proxy
@@ -19,6 +19,9 @@
 # AUTHORS
 # Darren Hart <dvhart@linux.intel.com>
 
+# disable pathname expansion, NO_PROXY fields could start with "*" or be it
+set -f
+
 if [ $# -lt 2 -o "$1" = '--help' -o "$1" = '-h' ] ; then
     echo 'oe-git-proxy: error: the following arguments are required: host port'
     echo 'Usage: oe-git-proxy host port'
-- 
2.21.0

[PATCHv2 5/6] oe-git-proxy: NO_PROXY suffix matching without wildcard for match_host

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

NO_PROXY can also contain just suffixes that do not start with a "*". We
failed to match those so far. Just add an extra "*" to also match those
suffixes. If one was there we get "**" which does not hurt.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/oe-git-proxy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
index 9eeef45dcf..2cb995f43c 100755
--- a/scripts/oe-git-proxy
+++ b/scripts/oe-git-proxy
@@ -107,7 +107,7 @@ match_host() {
 	HOST=$1
 	GLOB=$2
 
-	if [ -z "${HOST%%$GLOB}" ]; then
+	if [ -z "${HOST%%*$GLOB}" ]; then
 		return 0
 	fi
 
-- 
2.21.0

[PATCHv2 6/6] oe-git-proxy: fix dash "Bad substitution"

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

The script claims it works with dash, make sure that is actually the
case.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/oe-git-proxy | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
index 2cb995f43c..aa9b9dc9a9 100755
--- a/scripts/oe-git-proxy
+++ b/scripts/oe-git-proxy
@@ -64,7 +64,7 @@ ipv4_val() {
 	IP="$1"
 	SHIFT=24
 	VAL=0
-	for B in ${IP//./ }; do
+	for B in $( echo "$IP" | tr '.' ' ' ); do
 		VAL=$(($VAL+$(($B<<$SHIFT))))
 		SHIFT=$(($SHIFT-8))
 	done
@@ -137,7 +137,7 @@ if [ -z "$ALL_PROXY" ]; then
 fi
 
 # Connect directly to hosts in NO_PROXY
-for H in ${NO_PROXY//,/ }; do
+for H in $( echo "$NO_PROXY" | tr ',' ' ' ); do
 	if match_host $1 $H; then
 		exec $SOCAT STDIO $METHOD
 	fi
-- 
2.21.0

Re: [PATCHv2 0/6] solve several issues in oe-git-proxy

[Henning Schild]

Seems that was merged without further notice ... again. But hey it is
in.

poky
49bb6cefb507c116cb2548ae842eb0653053fee4...4b0bf01c14a90313ab294ead2ca1e5536bafd632

Henning

Am Tue, 3 Sep 2019 15:43:44 +0200
schrieb Henning Schild <henning.schild@siemens.com>:

> From: Henning Schild <henning.schild@siemens.com>
> 
> Change to v1:
>  - fix comment in p4
>  - move "set -f" all the way up in p4, that was the intention in the
>    first place
> 
> Henning Schild (6):
>   oe-git-proxy: allow setting SOCAT from outside
>   oeqa: add case for oe-git-proxy
>   Revert "oe-git-proxy: Avoid resolving NO_PROXY against local files"
>   oe-git-proxy: disable shell pathname expansion for the whole script
>   oe-git-proxy: NO_PROXY suffix matching without wildcard for
> match_host oe-git-proxy: fix dash "Bad substitution"
> 
>  meta/lib/oeqa/selftest/cases/oescripts.py | 58
> +++++++++++++++++++++++ scripts/oe-git-proxy                      |
> 21 ++++---- 2 files changed, 71 insertions(+), 8 deletions(-)
> 

Re: [PATCHv2 0/6] solve several issues in oe-git-proxy

[Ross Burton]

On 17/09/2019 09:44, Henning Schild wrote:
> Seems that was merged without further notice ... again. But hey it is
> in.

Yes, we still don't have 'your patch was integrated' notification mails.

Ross