* [PATCH 0/3] Bitbake 1.28-next
@ 2016-09-16 23:36 Armin Kuster
2016-09-16 23:36 ` [PATCH 1/3] bb/tests/fetch: Update cups url Armin Kuster
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Armin Kuster @ 2016-09-16 23:36 UTC (permalink / raw)
To: bitbake-devel, akuster808
Please consider these changes for 1.28
to fix bitbake-selftest failures.
Richard Purdie (1):
bb/tests/fetch: Update cups url
Ross Burton (2):
lib/bb/tests/fetch: remove URL that doesn't exist anymore
fetch2/wget: fallback to GET if HEAD is rejected in checkstatus()
lib/bb/fetch2/wget.py | 72 +++++++++++++++++++++++++++++++++++----------------
lib/bb/tests/fetch.py | 7 ++---
2 files changed, 53 insertions(+), 26 deletions(-)
--
2.7.4
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/3] bb/tests/fetch: Update cups url
2016-09-16 23:36 [PATCH 0/3] Bitbake 1.28-next Armin Kuster
@ 2016-09-16 23:36 ` Armin Kuster
2016-09-16 23:36 ` [PATCH 2/3] lib/bb/tests/fetch: remove URL that doesn't exist anymore Armin Kuster
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Armin Kuster @ 2016-09-16 23:36 UTC (permalink / raw)
To: bitbake-devel, akuster808
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the upstream url used for testing cups versions after upstream website
changes.
minor fixup to apply
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[Bitbake upstream: 5f06041d4936fc22297945bbbad7020bfa9083c6 ]
Signed-off-by: Armin Kuster <akuster@mvista.com>
---
lib/bb/tests/fetch.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 94173c1..73eba8c 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -692,8 +692,8 @@ class FetchLatestVersionTest(FetcherTest):
: "5.0",
("xserver-xorg", "http://xorg.freedesktop.org/releases/individual/xserver/xorg-server-1.15.1.tar.bz2", "", "")
: "1.15.1",
- # packages with valid REGEX_URI and REGEX
- ("cups", "http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2", "http://www.cups.org/software.php", "(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz")
+ # packages with valid UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX
+ ("cups", "http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2", "https://github.com/apple/cups/releases", "(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz")
: "2.0.0",
("db", "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz", "http://www.oracle.com/technetwork/products/berkeleydb/downloads/index-082944.html", "http://download.oracle.com/otn/berkeley-db/(?P<name>db-)(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz")
: "6.1.19",
--
2.7.4
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/3] lib/bb/tests/fetch: remove URL that doesn't exist anymore
2016-09-16 23:36 [PATCH 0/3] Bitbake 1.28-next Armin Kuster
2016-09-16 23:36 ` [PATCH 1/3] bb/tests/fetch: Update cups url Armin Kuster
@ 2016-09-16 23:36 ` Armin Kuster
2016-09-16 23:36 ` [PATCH 3/3] fetch2/wget: fallback to GET if HEAD is rejected in checkstatus() Armin Kuster
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Armin Kuster @ 2016-09-16 23:36 UTC (permalink / raw)
To: bitbake-devel, akuster808
From: Ross Burton <ross.burton@intel.com>
The CUPS ipptool URL we were checking now redirects to github where the tarball
isn't present, so remove it from the test suite.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4b50895fb3462b21e3874a2e99c363c8d05e89e6)
Signed-off-by: Armin Kuster <akuster@mvista.com>
---
lib/bb/tests/fetch.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 73eba8c..7a08be0 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -726,7 +726,6 @@ class FetchLatestVersionTest(FetcherTest):
class FetchCheckStatusTest(FetcherTest):
test_wget_uris = ["http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2",
- "http://www.cups.org/software/ipptool/ipptool-20130731-linux-ubuntu-i686.tar.gz",
"http://www.cups.org/",
"http://downloads.yoctoproject.org/releases/sato/sato-engine-0.1.tar.gz",
"http://downloads.yoctoproject.org/releases/sato/sato-engine-0.2.tar.gz",
--
2.7.4
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 3/3] fetch2/wget: fallback to GET if HEAD is rejected in checkstatus()
2016-09-16 23:36 [PATCH 0/3] Bitbake 1.28-next Armin Kuster
2016-09-16 23:36 ` [PATCH 1/3] bb/tests/fetch: Update cups url Armin Kuster
2016-09-16 23:36 ` [PATCH 2/3] lib/bb/tests/fetch: remove URL that doesn't exist anymore Armin Kuster
@ 2016-09-16 23:36 ` Armin Kuster
2016-09-17 5:54 ` [PATCH 0/3] Bitbake 1.28-next akuster
2016-09-23 17:19 ` akuster808
4 siblings, 0 replies; 6+ messages in thread
From: Armin Kuster @ 2016-09-16 23:36 UTC (permalink / raw)
To: bitbake-devel, akuster808
From: Ross Burton <ross.burton@intel.com>
The core change here is to fall back to GET requests if HEAD is rejected in the
checkstatus() method, as you can't do a HEAD on Amazon S3 (used by Github
archives). This meant removing the monkey patch that the default method was GET
and adding a fixed redirect handler that doesn't reset to GET.
Also, change the way the opener is constructed from an if/elif cluster to a
conditionally constructed list.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6ec70d5d2e330b41b932b0a655b838a5f37df01e)
Signed-off-by: Armin Kuster <akuster@mvista.com>
---
lib/bb/fetch2/wget.py | 72 +++++++++++++++++++++++++++++++++++----------------
lib/bb/tests/fetch.py | 2 ++
2 files changed, 51 insertions(+), 23 deletions(-)
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index bd2a897..0ff5b2b 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -234,38 +234,64 @@ class Wget(FetchMethod):
return exported
- def head_method(self):
- return "HEAD"
-
+ class HTTPMethodFallback(urllib2.BaseHandler):
+ """
+ Fallback to GET if HEAD is not allowed (405 HTTP error)
+ """
+ def http_error_405(self, req, fp, code, msg, headers):
+ fp.read()
+ fp.close()
+
+ newheaders = dict((k,v) for k,v in req.headers.items()
+ if k.lower() not in ("content-length", "content-type"))
+ return self.parent.open(urllib2.Request(req.get_full_url(),
+ headers=newheaders,
+ origin_req_host=req.get_origin_req_host(),
+ unverifiable=True))
+
+ """
+ Some servers (e.g. GitHub archives, hosted on Amazon S3) return 403
+ Forbidden when they actually mean 405 Method Not Allowed.
+ """
+ http_error_403 = http_error_405
+
+ """
+ Some servers (e.g. FusionForge) returns 406 Not Acceptable when they
+ actually mean 405 Method Not Allowed.
+ """
+ http_error_406 = http_error_405
+
+ class FixedHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
+ """
+ urllib2.HTTPRedirectHandler resets the method to GET on redirect,
+ when we want to follow redirects using the original method.
+ """
+ def redirect_request(self, req, fp, code, msg, headers, newurl):
+ newreq = urllib2.HTTPRedirectHandler.redirect_request(self, req, fp, code, msg, headers, newurl)
+ newreq.get_method = lambda: req.get_method()
+ return newreq
exported_proxies = export_proxies(d)
+ handlers = [FixedHTTPRedirectHandler, HTTPMethodFallback]
+ if export_proxies:
+ handlers.append(urllib2.ProxyHandler())
+ handlers.append(CacheHTTPHandler())
# XXX: Since Python 2.7.9 ssl cert validation is enabled by default
# see PEP-0476, this causes verification errors on some https servers
# so disable by default.
import ssl
- ssl_context = None
if hasattr(ssl, '_create_unverified_context'):
- ssl_context = ssl._create_unverified_context()
-
- if exported_proxies == True and ssl_context is not None:
- opener = urllib2.build_opener(urllib2.ProxyHandler, CacheHTTPHandler,
- urllib2.HTTPSHandler(context=ssl_context))
- elif exported_proxies == False and ssl_context is not None:
- opener = urllib2.build_opener(CacheHTTPHandler,
- urllib2.HTTPSHandler(context=ssl_context))
- elif exported_proxies == True and ssl_context is None:
- opener = urllib2.build_opener(urllib2.ProxyHandler, CacheHTTPHandler)
- else:
- opener = urllib2.build_opener(CacheHTTPHandler)
-
- urllib2.Request.get_method = head_method
- urllib2.install_opener(opener)
-
- uri = ud.url.split(";")[0]
+ handlers.append(urllib2.HTTPSHandler(context=ssl._create_unverified_context()))
+ opener = urllib2.build_opener(*handlers)
try:
- urllib2.urlopen(uri)
- except:
+ uri = ud.url.split(";")[0]
+ r = urllib2.Request(uri)
+ r.get_method = lambda: "HEAD"
+ opener.open(r)
+ except urllib2.URLError as e:
+ # debug for now to avoid spamming the logs in e.g. remote sstate searches
+ logger.debug(2, "checkstatus() urlopen failed: %s" % e)
return False
return True
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 7a08be0..3b74de0 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -737,6 +737,8 @@ class FetchCheckStatusTest(FetcherTest):
"ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.60.tar.gz",
"ftp://ftp.gnu.org/gnu/chess/gnuchess-5.08.tar.gz",
"ftp://ftp.gnu.org/gnu/gmp/gmp-4.0.tar.gz",
+ # GitHub releases are hosted on Amazon S3, which doesn't support HEAD
+ "https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz"
]
if os.environ.get("BB_SKIP_NETTESTS") == "yes":
--
2.7.4
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 0/3] Bitbake 1.28-next
2016-09-16 23:36 [PATCH 0/3] Bitbake 1.28-next Armin Kuster
` (2 preceding siblings ...)
2016-09-16 23:36 ` [PATCH 3/3] fetch2/wget: fallback to GET if HEAD is rejected in checkstatus() Armin Kuster
@ 2016-09-17 5:54 ` akuster
2016-09-23 17:19 ` akuster808
4 siblings, 0 replies; 6+ messages in thread
From: akuster @ 2016-09-17 5:54 UTC (permalink / raw)
To: bitbake-devel
sheesh. I thought I was working on bitbake used for jethro, looks like I
was off. patches are still applicable.
On 09/16/2016 04:36 PM, Armin Kuster wrote:
> Please consider these changes for 1.28
> to fix bitbake-selftest failures.
>
>
> Richard Purdie (1):
> bb/tests/fetch: Update cups url
>
> Ross Burton (2):
> lib/bb/tests/fetch: remove URL that doesn't exist anymore
> fetch2/wget: fallback to GET if HEAD is rejected in checkstatus()
>
> lib/bb/fetch2/wget.py | 72 +++++++++++++++++++++++++++++++++++----------------
> lib/bb/tests/fetch.py | 7 ++---
> 2 files changed, 53 insertions(+), 26 deletions(-)
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 0/3] Bitbake 1.28-next
2016-09-16 23:36 [PATCH 0/3] Bitbake 1.28-next Armin Kuster
` (3 preceding siblings ...)
2016-09-17 5:54 ` [PATCH 0/3] Bitbake 1.28-next akuster
@ 2016-09-23 17:19 ` akuster808
4 siblings, 0 replies; 6+ messages in thread
From: akuster808 @ 2016-09-23 17:19 UTC (permalink / raw)
To: bitbake-devel, Richard Purdie
[-- Attachment #1: Type: text/plain, Size: 726 bytes --]
Ping?
-------- Forwarded Message --------
Subject: [PATCH 0/3] Bitbake 1.28-next
Date: Fri, 16 Sep 2016 16:36:41 -0700
From: Armin Kuster <akuster808@gmail.com>
To: bitbake-devel@lists.openembedded.org, akuster808@gmail.com
Please consider these changes for 1.28
to fix bitbake-selftest failures.
Richard Purdie (1):
bb/tests/fetch: Update cups url
Ross Burton (2):
lib/bb/tests/fetch: remove URL that doesn't exist anymore
fetch2/wget: fallback to GET if HEAD is rejected in checkstatus()
lib/bb/fetch2/wget.py | 72 +++++++++++++++++++++++++++++++++++----------------
lib/bb/tests/fetch.py | 7 ++---
2 files changed, 53 insertions(+), 26 deletions(-)
--
2.7.4
[-- Attachment #2: Type: text/html, Size: 1950 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-09-23 17:19 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-09-16 23:36 [PATCH 0/3] Bitbake 1.28-next Armin Kuster
2016-09-16 23:36 ` [PATCH 1/3] bb/tests/fetch: Update cups url Armin Kuster
2016-09-16 23:36 ` [PATCH 2/3] lib/bb/tests/fetch: remove URL that doesn't exist anymore Armin Kuster
2016-09-16 23:36 ` [PATCH 3/3] fetch2/wget: fallback to GET if HEAD is rejected in checkstatus() Armin Kuster
2016-09-17 5:54 ` [PATCH 0/3] Bitbake 1.28-next akuster
2016-09-23 17:19 ` akuster808
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.