* [PATCH 0/1] fetcher2: retry mirror if upstream checksum mismatch
@ 2011-07-05 13:11 Yu Ke
2011-07-05 13:11 ` [PATCH 1/1] " Yu Ke
0 siblings, 1 reply; 6+ messages in thread
From: Yu Ke @ 2011-07-05 13:11 UTC (permalink / raw)
To: bitbake-devel
fetcher2: retry mirror if upstream checksum mismatch
This patch is for [YOCTO #1085] fix.
The following changes since commit ad2363278f0ea86fcf3464f8f6073d3a3d06be63:
Khem Raj (1):
uclibc: Fix compilation in thumb mode
are available in the git repository at:
git://git.pokylinux.org/poky-contrib kyu3/bug1085
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/bug1085
Yu Ke (1):
fetcher2: retry mirror if upstream checksum mismatch
bitbake/lib/bb/fetch2/__init__.py | 27 ++++++++++++++++-----------
1 files changed, 16 insertions(+), 11 deletions(-)
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch
2011-07-05 13:11 [PATCH 0/1] fetcher2: retry mirror if upstream checksum mismatch Yu Ke
@ 2011-07-05 13:11 ` Yu Ke
2011-07-08 2:41 ` Yu Ke
2011-07-12 9:41 ` Richard Purdie
0 siblings, 2 replies; 6+ messages in thread
From: Yu Ke @ 2011-07-05 13:11 UTC (permalink / raw)
To: bitbake-devel
This patch is for [YOCTO #1085] fix.
If the upstream fails a checksum, retry from the MIRROR before giving up.
This will add more robust fetching if an upstream serves a bad file or webpage.
fetching of distcc prior to the move from samba -> googlecode is a good example
of this.
Signed-off-by: Yu Ke <ke.yu@intel.com>
---
bitbake/lib/bb/fetch2/__init__.py | 27 ++++++++++++++++-----------
1 files changed, 16 insertions(+), 11 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index e9a64c5..bb9dbbe 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -278,6 +278,15 @@ def verify_checksum(u, ud, d):
if BB_STRICT_CHECKSUM = "1" then return false as unmatched, otherwise return true as
matched
"""
+ # Only check the checksums if we've not seen this item before
+ if os.path.exists(ud.donestamp):
+ # Touch the done stamp file to show active use of the download
+ try:
+ os.utime(ud.donestamp, None)
+ except:
+ # Errors aren't fatal here
+ pass
+ return
if not ud.type in ["http", "https", "ftp", "ftps"]:
return
@@ -300,6 +309,9 @@ def verify_checksum(u, ud, d):
if ud.sha256_expected != sha256data:
raise SHA256SumError(ud.localpath, ud.sha256_expected, sha256data, u)
+ # Create the stamp to avoid duplicate checking
+ open(ud.donestamp, 'w').close()
+
def subprocess_setup():
import signal
# Python installs a SIGPIPE handler by default. This is usually not what
@@ -932,6 +944,9 @@ class Fetch(object):
if hasattr(m, "build_mirror_data"):
m.build_mirror_data(u, ud, self.d)
localpath = ud.localpath
+ # early checksum verify, so that if checksum mismatched,
+ # fetcher still have chance to fetch from mirror
+ verify_checksum(u, ud, self.d)
except bb.fetch2.NetworkAccess:
raise
@@ -948,17 +963,7 @@ class Fetch(object):
if not localpath or ((not os.path.exists(localpath)) and localpath.find("*") == -1):
raise FetchError("Unable to fetch URL %s from any source." % u, u)
- if os.path.exists(ud.donestamp):
- # Touch the done stamp file to show active use of the download
- try:
- os.utime(ud.donestamp, None)
- except:
- # Errors aren't fatal here
- pass
- else:
- # Only check the checksums if we've not seen this item before, then create the stamp
- verify_checksum(u, ud, self.d)
- open(ud.donestamp, 'w').close()
+ verify_checksum(u, ud, self.d)
finally:
bb.utils.unlockfile(lf)
--
1.7.0.4
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch
2011-07-05 13:11 ` [PATCH 1/1] " Yu Ke
@ 2011-07-08 2:41 ` Yu Ke
2011-07-12 9:41 ` Richard Purdie
1 sibling, 0 replies; 6+ messages in thread
From: Yu Ke @ 2011-07-08 2:41 UTC (permalink / raw)
To: Yu Ke; +Cc: bitbake-devel
Ping. any comment on this one?
on 2011-7-5 21:11, Yu Ke wrote:
> This patch is for [YOCTO #1085] fix.
>
> If the upstream fails a checksum, retry from the MIRROR before giving up.
> This will add more robust fetching if an upstream serves a bad file or webpage.
>
> fetching of distcc prior to the move from samba -> googlecode is a good example
> of this.
>
> Signed-off-by: Yu Ke<ke.yu@intel.com>
> ---
> bitbake/lib/bb/fetch2/__init__.py | 27 ++++++++++++++++-----------
> 1 files changed, 16 insertions(+), 11 deletions(-)
>
> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index e9a64c5..bb9dbbe 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -278,6 +278,15 @@ def verify_checksum(u, ud, d):
> if BB_STRICT_CHECKSUM = "1" then return false as unmatched, otherwise return true as
> matched
> """
> + # Only check the checksums if we've not seen this item before
> + if os.path.exists(ud.donestamp):
> + # Touch the done stamp file to show active use of the download
> + try:
> + os.utime(ud.donestamp, None)
> + except:
> + # Errors aren't fatal here
> + pass
> + return
>
> if not ud.type in ["http", "https", "ftp", "ftps"]:
> return
> @@ -300,6 +309,9 @@ def verify_checksum(u, ud, d):
> if ud.sha256_expected != sha256data:
> raise SHA256SumError(ud.localpath, ud.sha256_expected, sha256data, u)
>
> + # Create the stamp to avoid duplicate checking
> + open(ud.donestamp, 'w').close()
> +
> def subprocess_setup():
> import signal
> # Python installs a SIGPIPE handler by default. This is usually not what
> @@ -932,6 +944,9 @@ class Fetch(object):
> if hasattr(m, "build_mirror_data"):
> m.build_mirror_data(u, ud, self.d)
> localpath = ud.localpath
> + # early checksum verify, so that if checksum mismatched,
> + # fetcher still have chance to fetch from mirror
> + verify_checksum(u, ud, self.d)
>
> except bb.fetch2.NetworkAccess:
> raise
> @@ -948,17 +963,7 @@ class Fetch(object):
> if not localpath or ((not os.path.exists(localpath)) and localpath.find("*") == -1):
> raise FetchError("Unable to fetch URL %s from any source." % u, u)
>
> - if os.path.exists(ud.donestamp):
> - # Touch the done stamp file to show active use of the download
> - try:
> - os.utime(ud.donestamp, None)
> - except:
> - # Errors aren't fatal here
> - pass
> - else:
> - # Only check the checksums if we've not seen this item before, then create the stamp
> - verify_checksum(u, ud, self.d)
> - open(ud.donestamp, 'w').close()
> + verify_checksum(u, ud, self.d)
>
> finally:
> bb.utils.unlockfile(lf)
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch
2011-07-05 13:11 ` [PATCH 1/1] " Yu Ke
2011-07-08 2:41 ` Yu Ke
@ 2011-07-12 9:41 ` Richard Purdie
1 sibling, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2011-07-12 9:41 UTC (permalink / raw)
To: Yu Ke; +Cc: bitbake-devel
On Tue, 2011-07-05 at 21:11 +0800, Yu Ke wrote:
> This patch is for [YOCTO #1085] fix.
>
> If the upstream fails a checksum, retry from the MIRROR before giving up.
> This will add more robust fetching if an upstream serves a bad file or webpage.
>
> fetching of distcc prior to the move from samba -> googlecode is a good example
> of this.
>
> Signed-off-by: Yu Ke <ke.yu@intel.com>
> ---
> bitbake/lib/bb/fetch2/__init__.py | 27 ++++++++++++++++-----------
> 1 files changed, 16 insertions(+), 11 deletions(-)
>
> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index e9a64c5..bb9dbbe 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -278,6 +278,15 @@ def verify_checksum(u, ud, d):
> if BB_STRICT_CHECKSUM = "1" then return false as unmatched, otherwise return true as
> matched
> """
> + # Only check the checksums if we've not seen this item before
> + if os.path.exists(ud.donestamp):
> + # Touch the done stamp file to show active use of the download
> + try:
> + os.utime(ud.donestamp, None)
> + except:
> + # Errors aren't fatal here
> + pass
> + return
>
> if not ud.type in ["http", "https", "ftp", "ftps"]:
> return
> @@ -300,6 +309,9 @@ def verify_checksum(u, ud, d):
> if ud.sha256_expected != sha256data:
> raise SHA256SumError(ud.localpath, ud.sha256_expected, sha256data, u)
>
> + # Create the stamp to avoid duplicate checking
> + open(ud.donestamp, 'w').close()
> +
> def subprocess_setup():
> import signal
> # Python installs a SIGPIPE handler by default. This is usually not what
> @@ -932,6 +944,9 @@ class Fetch(object):
> if hasattr(m, "build_mirror_data"):
> m.build_mirror_data(u, ud, self.d)
> localpath = ud.localpath
> + # early checksum verify, so that if checksum mismatched,
> + # fetcher still have chance to fetch from mirror
> + verify_checksum(u, ud, self.d)
>
> except bb.fetch2.NetworkAccess:
> raise
> @@ -948,17 +963,7 @@ class Fetch(object):
> if not localpath or ((not os.path.exists(localpath)) and localpath.find("*") == -1):
> raise FetchError("Unable to fetch URL %s from any source." % u, u)
>
> - if os.path.exists(ud.donestamp):
> - # Touch the done stamp file to show active use of the download
> - try:
> - os.utime(ud.donestamp, None)
> - except:
> - # Errors aren't fatal here
> - pass
> - else:
> - # Only check the checksums if we've not seen this item before, then create the stamp
> - verify_checksum(u, ud, self.d)
> - open(ud.donestamp, 'w').close()
> + verify_checksum(u, ud, self.d)
>
> finally:
> bb.utils.unlockfile(lf)
Hi Ke,
I find this patch a little confusing since you're altering what the
verify_checksum() function does to including something else which is
partly unrelated.
I like the idea of moving that logic to a separate function but perhaps
we need a new one which then calls verify_checksum() rather than having
the two functionalities merged together?
Perhaps an update_stamps() function would be a good name for this?
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch
2011-07-13 9:08 ` [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch Yu Ke
@ 2011-07-13 11:13 ` Richard Purdie
0 siblings, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2011-07-13 11:13 UTC (permalink / raw)
To: Yu Ke; +Cc: bitbake-devel
On Wed, 2011-07-13 at 17:08 +0800, Yu Ke wrote:
> This patch is for [YOCTO #1085] fix.
>
> If the upstream fails a checksum, retry from the MIRROR before giving up.
> This will add more robust fetching if an upstream serves a bad file or webpage.
>
> fetching of distcc prior to the move from samba -> googlecode is a good example
> of this.
>
> Signed-off-by: Yu Ke <ke.yu@intel.com>
> ---
> bitbake/lib/bb/fetch2/__init__.py | 31 ++++++++++++++++++++-----------
> 1 files changed, 20 insertions(+), 11 deletions(-)
Merged to master, thanks.
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch
2011-07-13 9:08 [PATCH 0/1] fetcher2: retry mirror if upstream checksum mismatch V2 Yu Ke
@ 2011-07-13 9:08 ` Yu Ke
2011-07-13 11:13 ` Richard Purdie
0 siblings, 1 reply; 6+ messages in thread
From: Yu Ke @ 2011-07-13 9:08 UTC (permalink / raw)
To: bitbake-devel, richard.purdie
This patch is for [YOCTO #1085] fix.
If the upstream fails a checksum, retry from the MIRROR before giving up.
This will add more robust fetching if an upstream serves a bad file or webpage.
fetching of distcc prior to the move from samba -> googlecode is a good example
of this.
Signed-off-by: Yu Ke <ke.yu@intel.com>
---
bitbake/lib/bb/fetch2/__init__.py | 31 ++++++++++++++++++++-----------
1 files changed, 20 insertions(+), 11 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index e9a64c5..d39f094 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -300,6 +300,22 @@ def verify_checksum(u, ud, d):
if ud.sha256_expected != sha256data:
raise SHA256SumError(ud.localpath, ud.sha256_expected, sha256data, u)
+def update_stamp(u, ud, d):
+ """
+ donestamp is file stamp indicating the whole fetching is done
+ this function update the stamp after verifying the checksum
+ """
+ if os.path.exists(ud.donestamp):
+ # Touch the done stamp file to show active use of the download
+ try:
+ os.utime(ud.donestamp, None)
+ except:
+ # Errors aren't fatal here
+ pass
+ else:
+ verify_checksum(u, ud, d)
+ open(ud.donestamp, 'w').close()
+
def subprocess_setup():
import signal
# Python installs a SIGPIPE handler by default. This is usually not what
@@ -932,6 +948,9 @@ class Fetch(object):
if hasattr(m, "build_mirror_data"):
m.build_mirror_data(u, ud, self.d)
localpath = ud.localpath
+ # early checksum verify, so that if checksum mismatched,
+ # fetcher still have chance to fetch from mirror
+ update_stamp(u, ud, self.d)
except bb.fetch2.NetworkAccess:
raise
@@ -948,17 +967,7 @@ class Fetch(object):
if not localpath or ((not os.path.exists(localpath)) and localpath.find("*") == -1):
raise FetchError("Unable to fetch URL %s from any source." % u, u)
- if os.path.exists(ud.donestamp):
- # Touch the done stamp file to show active use of the download
- try:
- os.utime(ud.donestamp, None)
- except:
- # Errors aren't fatal here
- pass
- else:
- # Only check the checksums if we've not seen this item before, then create the stamp
- verify_checksum(u, ud, self.d)
- open(ud.donestamp, 'w').close()
+ update_stamp(u, ud, self.d)
finally:
bb.utils.unlockfile(lf)
--
1.7.0.4
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-07-13 11:17 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-07-05 13:11 [PATCH 0/1] fetcher2: retry mirror if upstream checksum mismatch Yu Ke
2011-07-05 13:11 ` [PATCH 1/1] " Yu Ke
2011-07-08 2:41 ` Yu Ke
2011-07-12 9:41 ` Richard Purdie
2011-07-13 9:08 [PATCH 0/1] fetcher2: retry mirror if upstream checksum mismatch V2 Yu Ke
2011-07-13 9:08 ` [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch Yu Ke
2011-07-13 11:13 ` Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.