(PRE)MIRRORS doesn't work with https?$:// anymore Was: [PATCH] fetch2: Add new mirror syntax to simplify mirror specifications

[Martin Jansa <martin.jansa@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 18172 bytes --]

On Thu, Jun 21, 2012 at 03:34:57PM +0100, Richard Purdie wrote:
> When writing mirror specifications, the current regexp syntax can be awkward
> and hard to get it to do what you want. For example, extracting the 'basename'
> of a repository:
> 
> PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"
> 
> can now become:
> 
> PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"
> 
> which is much clearer. A MIRRORNAME substitution is also added which contains
> an encoded form of both host and path. One of the problems with the existing
> regexp syntax is you couldn't access HOST information from PATH and vice-versa
> which is an issue this patch also addresses.

One of this fetch2 patches break https?:// in MIRRORS

It's used e.g in openembedded-core/meta/classes/mirrors.bbclass:
https?$://.*/.* http://downloads.yoctoproject.org/mirror/sources/
https?$://.*/.* http://sources.openembedded.org/

I've tested it with older bitbake in PREMIRRORS (to make log.do_fetch
log shorter) on libcap (which doesn't exist on upstream URL).

PREMIRRORS_append () {
cvs://.*/.* http://build.shr-project.org/sources/
svn://.*/.* http://build.shr-project.org/sources/
git://.*/.* http://build.shr-project.org/sources/
hg://.*/.* http://build.shr-project.org/sources/
bzr://.*/.* http://build.shr-project.org/sources/
https?$://.*/.* http://build.shr-project.org/sources/
}

bitbake d316f28ed725ff40daa8771c1aa224ac46d5b224:
DEBUG: Executing python function do_fetch
DEBUG: Executing python function base_do_fetch
DEBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['cvs', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['svn', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['git', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['hg', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['bzr', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https?$', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url http://kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2 returning http://build.shr-project.org/sources/libcap-2.22.tar.bz2
NOTE: fetch http://build.shr-project.org/sources/libcap-2.22.tar.bz2
DEBUG: executing /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Running export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished

bitbake e6ff1d4bab43fdcd8af1230f1d54615f53c1978e
DEBUG: Executing python function do_fetch
DEBUG: Executing python function base_do_fetch
DEBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['cvs', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['svn', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['git', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['hg', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['bzr', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https?$', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: Trying Upstream
NOTE: fetch http://kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2
...and after trying Upstream and all MIRRORS (none will match) it fails to fetch it..

Removing '$' doesn't work too:
PREMIRRORS_append () {
https?://.*/.* http://build.shr-project.org/sources/
}
DEBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https?', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: Trying Upstream
NOTE: fetch http://kernel.org/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2

Using http$ doesn't work too
EBUG: Trying PREMIRRORS
DEBUG: For url ['http', 'kernel.org', '/pub/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['http$', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: Trying Upstream

And without any regexp with different PREMIRRORS for http and https it also does weird things (looks like trying to resolve returned PREMIRROR again)
PREMIRRORS_append () {
http://.*/.* http://build.shr-project.org/sources/
https://.*/.* http://build.shr-project.org/https-sources/
}

changed libcap SRC_URI to https:
-SRC_URI = "${KERNELORG_MIRROR}/linux/libs/security/linux-privs/libcap2/${BPN}-${PV}.tar.bz2"
+SRC_URI = "https://foo.org/linux/libs/security/linux-privs/libcap2/${BPN}-${PV}.tar.bz2"


DEBUG: Trying PREMIRRORS
DEBUG: For url ['https', 'foo.org', '/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['https', 'foo.org', '/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2', '', '', {}] comparing ['https', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/https-sources/', '', '', {}]
DEBUG: For url https://foo.org/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2 returning http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2
DEBUG: For url ['http', 'build.shr-project.org', '/https-sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2 returning http://build.shr-project.org/sources/libcap-2.22.tar.bz2
DEBUG: For url ['http', 'build.shr-project.org', '/sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
DEBUG: For url ['http', 'build.shr-project.org', '/sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['https', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/https-sources/', '', '', {}]
DEBUG: For url ['http', 'build.shr-project.org', '/https-sources/libcap-2.22.tar.bz2', '', '', {}] comparing ['https', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/https-sources/', '', '', {}]
NOTE: fetch http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2
DEBUG: executing /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2'
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2'
DEBUG: Running export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2'
DEBUG: Mirror fetch failure for url http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2 (original url: https://foo.org/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.bz2)
DEBUG: Fetcher failure: Fetch command export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2' failed with exit code 8, output:
STDOUT:
STDERR: http://build.shr-project.org/https-sources/libcap-2.22.tar.bz2:
2012-06-26 19:23:37 ERROR 404: Not Found.

NOTE: fetch http://build.shr-project.org/sources/libcap-2.22.tar.bz2
DEBUG: executing /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Running export HOME="/OE"; export GIT_CONFIG="/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/etc/gitconfig"; export PATH="/OE/shr-core/openembedded-core/scripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/armv4t-oe-linux-gnueabi:/OE/shr-core/tmp-eglibc/sysroots/om-gta02/usr/bin/crossscripts:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux/sbin:/OE/shr-core/tmp-eglibc/sysroots/x86_64-linux//bin:/OE/shr-core/openembedded-core/scripts:/OE/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3"; /usr/bin/env wget  -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /OE/shr-core/downloads 'http://build.shr-project.org/sources/libcap-2.22.tar.bz2'
DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished


Cheers,
> 
> Tests for the new syntax are also added.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index 1b99a0f..e6f55f8 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -182,7 +182,7 @@ def encodeurl(decoded):
>  
>      return url
>  
> -def uri_replace(ud, uri_find, uri_replace, d):
> +def uri_replace(ud, uri_find, uri_replace, replacements, d):
>      if not ud.url or not uri_find or not uri_replace:
>          logger.error("uri_replace: passed an undefined value, not replacing")
>          return None
> @@ -213,6 +213,8 @@ def uri_replace(ud, uri_find, uri_replace, d):
>              if not uri_replace_decoded[loc]:
>                  result_decoded[loc] = ""    
>              else:
> +                for k in replacements:
> +                    uri_replace_decoded[loc] = uri_replace_decoded[loc].replace(k, replacements[k])
>                  #bb.note("%s %s %s" % (i, uri_replace_decoded[loc], uri_decoded[loc]))
>                  result_decoded[loc] = re.sub(i, uri_replace_decoded[loc], uri_decoded[loc])
>              if loc == 2:
> @@ -485,13 +487,20 @@ def build_mirroruris(origud, mirrors, ld):
>      uris = []
>      uds = []
>  
> +    replacements = {}
> +    replacements["TYPE"] = origud.type
> +    replacements["HOST"] = origud.host
> +    replacements["PATH"] = origud.path
> +    replacements["BASENAME"] = origud.path.split("/")[-1]
> +    replacements["MIRRORNAME"] = origud.host.replace(':','.') + origud.path.replace('/', '.')
> +
>      def adduri(uri, ud, uris, uds):
>          for line in mirrors:
>              try:
>                  (find, replace) = line
>              except ValueError:
>                  continue
> -            newuri = uri_replace(ud, find, replace, ld)
> +            newuri = uri_replace(ud, find, replace, replacements, ld)
>              if not newuri or newuri in uris or newuri == origud.url:
>                  continue
>              uris.append(newuri)
> diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
> index 9961343..1477fab 100644
> --- a/bitbake/lib/bb/tests/fetch.py
> +++ b/bitbake/lib/bb/tests/fetch.py
> @@ -48,7 +48,14 @@ class FetcherTest(unittest.TestCase):
>          ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://www.apache.org/dist", "http://archive.apache.org/dist")
>              : "http://archive.apache.org/dist/subversion/subversion-1.7.1.tar.bz2",
>          ("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://.*/.*", "file:///somepath/downloads/")
> -            : "file:///somepath/downloads/subversion-1.7.1.tar.bz2"
> +            : "file:///somepath/downloads/subversion-1.7.1.tar.bz2",
> +        ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") 
> +            : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
> +        ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http") 
> +            : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
> +        ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http") 
> +            : "git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http", 
> +
>          #Renaming files doesn't work
>          #("http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz") : "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz"
>          #("file://sstate-xyz.tgz", "file://.*/.*", "file:///somewhere/1234/sstate-cache") : "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz",
> @@ -143,8 +150,9 @@ class FetcherTest(unittest.TestCase):
>          for k, v in self.replaceuris.items():
>              ud = bb.fetch.FetchData(k[0], self.d)
>              ud.setup_localpath(self.d)
> -            newuris = bb.fetch2.uri_replace(ud, k[1], k[2], self.d)
> -            self.assertEqual(newuris, v)
> +            mirrors = bb.fetch2.mirror_from_string("%s %s" % (k[1], k[2]))
> +            newuris, uds = bb.fetch2.build_mirroruris(ud, mirrors, self.d)
> +            self.assertEqual(newuris, [v])
>  
>      def test_urilist1(self):
>          fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
> 
> 
> 
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]