* [PATCH] fetch2/wget: include paramters in URL
@ 2020-12-29 15:13 Patrick Williams
2020-12-29 21:05 ` Patrick Williams
` (3 more replies)
0 siblings, 4 replies; 7+ messages in thread
From: Patrick Williams @ 2020-12-29 15:13 UTC (permalink / raw)
To: bitbake-devel; +Cc: Patrick Williams
The wget fetcher currently silently drops all parameters and has
been that way since it was originally introduced in fetch via 080eb8764.
This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
additional parameters.
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
---
lib/bb/fetch2/wget.py | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index e6d9f528..d41d7408 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -99,7 +99,13 @@ class Wget(FetchMethod):
if ud.user and ud.pswd:
fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
- uri = ud.url.split(";")[0]
+ # Join the URL components together.
+ urls = ud.url.split(";")
+ if len(urls) == 1:
+ uri = urls[0]
+ else:
+ uri = urls[0] + "?" + "&".join(urls[1:])
+
if os.path.exists(ud.localpath):
# file exists, but we didnt complete it.. trying again..
fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
--
2.26.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] fetch2/wget: include paramters in URL
2020-12-29 15:13 [PATCH] fetch2/wget: include paramters in URL Patrick Williams
@ 2020-12-29 21:05 ` Patrick Williams
2020-12-29 21:16 ` [PATCH v2] fetch2/wget: include parameters " Patrick Williams
` (2 subsequent siblings)
3 siblings, 0 replies; 7+ messages in thread
From: Patrick Williams @ 2020-12-29 21:05 UTC (permalink / raw)
To: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 1506 bytes --]
On Tue, Dec 29, 2020 at 09:13:16AM -0600, Patrick Williams wrote:
> The wget fetcher currently silently drops all parameters and has
> been that way since it was originally introduced in fetch via 080eb8764.
> This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
> additional parameters.
>
> Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
> ---
> lib/bb/fetch2/wget.py | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
> index e6d9f528..d41d7408 100644
> --- a/lib/bb/fetch2/wget.py
> +++ b/lib/bb/fetch2/wget.py
> @@ -99,7 +99,13 @@ class Wget(FetchMethod):
> if ud.user and ud.pswd:
> fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
>
> - uri = ud.url.split(";")[0]
> + # Join the URL components together.
> + urls = ud.url.split(";")
> + if len(urls) == 1:
> + uri = urls[0]
> + else:
> + uri = urls[0] + "?" + "&".join(urls[1:])
> +
> if os.path.exists(ud.localpath):
> # file exists, but we didnt complete it.. trying again..
> fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
> --
> 2.26.2
>
This causes some problems that I need to work out more. In my downloads
directory I ended up with files like:
'libxml2-2.9.10.tar.gz?name=libtar'
'libxml2-2.9.10.tar.gz?name=libtar.1'
--
Patrick Williams
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v2] fetch2/wget: include parameters in URL
2020-12-29 15:13 [PATCH] fetch2/wget: include paramters in URL Patrick Williams
2020-12-29 21:05 ` Patrick Williams
@ 2020-12-29 21:16 ` Patrick Williams
2020-12-29 21:57 ` [PATCH v3] " Patrick Williams
2020-12-30 22:03 ` [PATCH v4] " Patrick Williams
3 siblings, 0 replies; 7+ messages in thread
From: Patrick Williams @ 2020-12-29 21:16 UTC (permalink / raw)
To: bitbake-devel; +Cc: Patrick Williams
The wget fetcher currently silently drops all parameters and has
been that way since it was originally introduced in fetch via 080eb8764.
This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
additional parameters.
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
---
lib/bb/fetch2/wget.py | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index e6d9f528..d9432c5b 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -99,7 +99,14 @@ class Wget(FetchMethod):
if ud.user and ud.pswd:
fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
- uri = ud.url.split(";")[0]
+ # Join the URL components together.
+ urls = ud.url.split(";")
+ if len(urls) == 1:
+ uri = urls[0]
+ else:
+ uri = urls[0] + "?" + "&".join(urls[1:])
+ fetchcmd += " -O '%s'" % urls[0].split("/")[-1]
+
if os.path.exists(ud.localpath):
# file exists, but we didnt complete it.. trying again..
fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
--
2.26.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH v3] fetch2/wget: include parameters in URL
2020-12-29 15:13 [PATCH] fetch2/wget: include paramters in URL Patrick Williams
2020-12-29 21:05 ` Patrick Williams
2020-12-29 21:16 ` [PATCH v2] fetch2/wget: include parameters " Patrick Williams
@ 2020-12-29 21:57 ` Patrick Williams
2020-12-29 22:02 ` Patrick Williams
2020-12-30 22:03 ` [PATCH v4] " Patrick Williams
3 siblings, 1 reply; 7+ messages in thread
From: Patrick Williams @ 2020-12-29 21:57 UTC (permalink / raw)
To: bitbake-devel; +Cc: Patrick Williams
The wget fetcher currently silently drops all parameters and has
been that way since it was originally introduced in fetch via 080eb8764.
This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
additional parameters.
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
---
lib/bb/fetch2/wget.py | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index e6d9f528..80baf04a 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -99,7 +99,14 @@ class Wget(FetchMethod):
if ud.user and ud.pswd:
fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
- uri = ud.url.split(";")[0]
+ # Join the URL components together.
+ urls = ud.url.split(";")
+ if len(urls) == 1:
+ uri = urls[0]
+ else:
+ uri = urls[0] + "?" + "&".join(urls[1:])
+ fetchcmd += " -O '%s/%s'" % (d.getVar("DL_DIR"), urls[0].split("/")[-1])
+
if os.path.exists(ud.localpath):
# file exists, but we didnt complete it.. trying again..
fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
--
2.26.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v3] fetch2/wget: include parameters in URL
2020-12-29 21:57 ` [PATCH v3] " Patrick Williams
@ 2020-12-29 22:02 ` Patrick Williams
0 siblings, 0 replies; 7+ messages in thread
From: Patrick Williams @ 2020-12-29 22:02 UTC (permalink / raw)
To: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 1814 bytes --]
On Tue, Dec 29, 2020 at 03:57:46PM -0600, Patrick Williams wrote:
> The wget fetcher currently silently drops all parameters and has
> been that way since it was originally introduced in fetch via 080eb8764.
> This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
> additional parameters.
>
> Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
> ---
> lib/bb/fetch2/wget.py | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
> index e6d9f528..80baf04a 100644
> --- a/lib/bb/fetch2/wget.py
> +++ b/lib/bb/fetch2/wget.py
> @@ -99,7 +99,14 @@ class Wget(FetchMethod):
> if ud.user and ud.pswd:
> fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
>
> - uri = ud.url.split(";")[0]
> + # Join the URL components together.
> + urls = ud.url.split(";")
> + if len(urls) == 1:
> + uri = urls[0]
> + else:
> + uri = urls[0] + "?" + "&".join(urls[1:])
> + fetchcmd += " -O '%s/%s'" % (d.getVar("DL_DIR"), urls[0].split("/")[-1])
> +
I've now tested this with a `bitbake <image> --runall=fetch` for one of
my machine images and was successful. The `-O` above feels a little
fragile but it is required for the uninative blob because that ends up
in a `uninative/<checksum>` subdirectory.
In uninative.bbclass there is:
localdata.setVar('DL_DIR', tarballdir)
The `-P ${DL_DIR}` below seems to not take effect if `-O` is given.
> if os.path.exists(ud.localpath):
> # file exists, but we didnt complete it.. trying again..
> fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
> --
> 2.26.2
>
--
Patrick Williams
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v4] fetch2/wget: include parameters in URL
2020-12-29 15:13 [PATCH] fetch2/wget: include paramters in URL Patrick Williams
` (2 preceding siblings ...)
2020-12-29 21:57 ` [PATCH v3] " Patrick Williams
@ 2020-12-30 22:03 ` Patrick Williams
2020-12-30 22:06 ` Patrick Williams
3 siblings, 1 reply; 7+ messages in thread
From: Patrick Williams @ 2020-12-30 22:03 UTC (permalink / raw)
To: bitbake-devel; +Cc: Patrick Williams
The wget fetcher currently silently drops all parameters and has
been that way since it was originally introduced in fetch via 080eb8764.
This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
additional parameters.
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
---
lib/bb/fetch2/wget.py | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index e6d9f528..0019f10c 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -91,7 +91,14 @@ class Wget(FetchMethod):
fetchcmd = self.basecmd
- if 'downloadfilename' in ud.parm:
+ # Join the URL components together.
+ url_parts = ud.url.split(";")
+ if len(url_parts) == 1:
+ uri = url_parts[0]
+ else:
+ uri = url_parts[0] + "?" + "&".join(url_parts[1:])
+
+ if 'downloadfilename' in ud.parm or len(url_parts) != 1:
localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile)
bb.utils.mkdirhier(os.path.dirname(localpath))
fetchcmd += " -O %s" % shlex.quote(localpath)
@@ -99,7 +106,6 @@ class Wget(FetchMethod):
if ud.user and ud.pswd:
fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
- uri = ud.url.split(";")[0]
if os.path.exists(ud.localpath):
# file exists, but we didnt complete it.. trying again..
fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
@@ -441,7 +447,7 @@ class Wget(FetchMethod):
valid = 1
elif self._vercmp(version, newver) < 0:
version = newver
-
+
pupver = re.sub('_', '.', version[1])
bb.debug(3, "*** %s -> UpstreamVersion = %s (CurrentVersion = %s)" %
--
2.26.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v4] fetch2/wget: include parameters in URL
2020-12-30 22:03 ` [PATCH v4] " Patrick Williams
@ 2020-12-30 22:06 ` Patrick Williams
0 siblings, 0 replies; 7+ messages in thread
From: Patrick Williams @ 2020-12-30 22:06 UTC (permalink / raw)
To: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 2407 bytes --]
On Wed, Dec 30, 2020 at 04:03:23PM -0600, Patrick Williams wrote:
> The wget fetcher currently silently drops all parameters and has
> been that way since it was originally introduced in fetch via 080eb8764.
> This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
> additional parameters.
>
> Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
> ---
> lib/bb/fetch2/wget.py | 12 +++++++++---
> 1 file changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
> index e6d9f528..0019f10c 100644
> --- a/lib/bb/fetch2/wget.py
> +++ b/lib/bb/fetch2/wget.py
> @@ -91,7 +91,14 @@ class Wget(FetchMethod):
>
> fetchcmd = self.basecmd
>
> - if 'downloadfilename' in ud.parm:
> + # Join the URL components together.
> + url_parts = ud.url.split(";")
> + if len(url_parts) == 1:
> + uri = url_parts[0]
> + else:
> + uri = url_parts[0] + "?" + "&".join(url_parts[1:])
> +
> + if 'downloadfilename' in ud.parm or len(url_parts) != 1:
> localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile)
> bb.utils.mkdirhier(os.path.dirname(localpath))
> fetchcmd += " -O %s" % shlex.quote(localpath)
> @@ -99,7 +106,6 @@ class Wget(FetchMethod):
> if ud.user and ud.pswd:
> fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
>
> - uri = ud.url.split(";")[0]
I ran into one more edge case trying to re-download a file from my
internal mirror service: netbase_6.1~bpo10+1.tar.xz . The URL
contains a URL-quoted version of the filename. Switched the code
to reuse the 'downloadfilename' logic earlier in the function, which
is safer anyhow because it uses shlex, etc.
> if os.path.exists(ud.localpath):
> # file exists, but we didnt complete it.. trying again..
> fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
> @@ -441,7 +447,7 @@ class Wget(FetchMethod):
> valid = 1
> elif self._vercmp(version, newver) < 0:
> version = newver
> -
> +
> pupver = re.sub('_', '.', version[1])
>
> bb.debug(3, "*** %s -> UpstreamVersion = %s (CurrentVersion = %s)" %
> --
> 2.26.2
>
--
Patrick Williams
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-12-30 22:06 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-29 15:13 [PATCH] fetch2/wget: include paramters in URL Patrick Williams
2020-12-29 21:05 ` Patrick Williams
2020-12-29 21:16 ` [PATCH v2] fetch2/wget: include parameters " Patrick Williams
2020-12-29 21:57 ` [PATCH v3] " Patrick Williams
2020-12-29 22:02 ` Patrick Williams
2020-12-30 22:03 ` [PATCH v4] " Patrick Williams
2020-12-30 22:06 ` Patrick Williams
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.