* [PATCH 0/1] fetcher2: retry mirror if upstream checksum mismatch V2
@ 2011-07-13 9:08 Yu Ke
2011-07-13 9:08 ` [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch Yu Ke
0 siblings, 1 reply; 6+ messages in thread
From: Yu Ke @ 2011-07-13 9:08 UTC (permalink / raw)
To: bitbake-devel, richard.purdie
This is the V2 patch. the main change is introduce new function update_stamp
per Richard's suggestion.
CC: Richard Purdie <richard.purdie@linuxfoundation.org>
The following changes since commit 7354fc9213f27aa1b643dbe88070437f1ee4c063:
Paul Eggleton (1):
insane.bbclass: skip rdepends QA checks for kernel / modules
are available in the git repository at:
git://git.pokylinux.org/poky-contrib kyu3/verify-checksum
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/verify-checksum
Yu Ke (1):
fetcher2: retry mirror if upstream checksum mismatch
bitbake/lib/bb/fetch2/__init__.py | 31 ++++++++++++++++++++-----------
1 files changed, 20 insertions(+), 11 deletions(-)
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch 2011-07-13 9:08 [PATCH 0/1] fetcher2: retry mirror if upstream checksum mismatch V2 Yu Ke @ 2011-07-13 9:08 ` Yu Ke 2011-07-13 11:13 ` Richard Purdie 0 siblings, 1 reply; 6+ messages in thread From: Yu Ke @ 2011-07-13 9:08 UTC (permalink / raw) To: bitbake-devel, richard.purdie This patch is for [YOCTO #1085] fix. If the upstream fails a checksum, retry from the MIRROR before giving up. This will add more robust fetching if an upstream serves a bad file or webpage. fetching of distcc prior to the move from samba -> googlecode is a good example of this. Signed-off-by: Yu Ke <ke.yu@intel.com> --- bitbake/lib/bb/fetch2/__init__.py | 31 ++++++++++++++++++++----------- 1 files changed, 20 insertions(+), 11 deletions(-) diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py index e9a64c5..d39f094 100644 --- a/bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/__init__.py @@ -300,6 +300,22 @@ def verify_checksum(u, ud, d): if ud.sha256_expected != sha256data: raise SHA256SumError(ud.localpath, ud.sha256_expected, sha256data, u) +def update_stamp(u, ud, d): + """ + donestamp is file stamp indicating the whole fetching is done + this function update the stamp after verifying the checksum + """ + if os.path.exists(ud.donestamp): + # Touch the done stamp file to show active use of the download + try: + os.utime(ud.donestamp, None) + except: + # Errors aren't fatal here + pass + else: + verify_checksum(u, ud, d) + open(ud.donestamp, 'w').close() + def subprocess_setup(): import signal # Python installs a SIGPIPE handler by default. This is usually not what @@ -932,6 +948,9 @@ class Fetch(object): if hasattr(m, "build_mirror_data"): m.build_mirror_data(u, ud, self.d) localpath = ud.localpath + # early checksum verify, so that if checksum mismatched, + # fetcher still have chance to fetch from mirror + update_stamp(u, ud, self.d) except bb.fetch2.NetworkAccess: raise @@ -948,17 +967,7 @@ class Fetch(object): if not localpath or ((not os.path.exists(localpath)) and localpath.find("*") == -1): raise FetchError("Unable to fetch URL %s from any source." % u, u) - if os.path.exists(ud.donestamp): - # Touch the done stamp file to show active use of the download - try: - os.utime(ud.donestamp, None) - except: - # Errors aren't fatal here - pass - else: - # Only check the checksums if we've not seen this item before, then create the stamp - verify_checksum(u, ud, self.d) - open(ud.donestamp, 'w').close() + update_stamp(u, ud, self.d) finally: bb.utils.unlockfile(lf) -- 1.7.0.4 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch 2011-07-13 9:08 ` [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch Yu Ke @ 2011-07-13 11:13 ` Richard Purdie 0 siblings, 0 replies; 6+ messages in thread From: Richard Purdie @ 2011-07-13 11:13 UTC (permalink / raw) To: Yu Ke; +Cc: bitbake-devel On Wed, 2011-07-13 at 17:08 +0800, Yu Ke wrote: > This patch is for [YOCTO #1085] fix. > > If the upstream fails a checksum, retry from the MIRROR before giving up. > This will add more robust fetching if an upstream serves a bad file or webpage. > > fetching of distcc prior to the move from samba -> googlecode is a good example > of this. > > Signed-off-by: Yu Ke <ke.yu@intel.com> > --- > bitbake/lib/bb/fetch2/__init__.py | 31 ++++++++++++++++++++----------- > 1 files changed, 20 insertions(+), 11 deletions(-) Merged to master, thanks. Richard ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 0/1] fetcher2: retry mirror if upstream checksum mismatch
@ 2011-07-05 13:11 Yu Ke
2011-07-05 13:11 ` [PATCH 1/1] " Yu Ke
0 siblings, 1 reply; 6+ messages in thread
From: Yu Ke @ 2011-07-05 13:11 UTC (permalink / raw)
To: bitbake-devel
fetcher2: retry mirror if upstream checksum mismatch
This patch is for [YOCTO #1085] fix.
The following changes since commit ad2363278f0ea86fcf3464f8f6073d3a3d06be63:
Khem Raj (1):
uclibc: Fix compilation in thumb mode
are available in the git repository at:
git://git.pokylinux.org/poky-contrib kyu3/bug1085
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/bug1085
Yu Ke (1):
fetcher2: retry mirror if upstream checksum mismatch
bitbake/lib/bb/fetch2/__init__.py | 27 ++++++++++++++++-----------
1 files changed, 16 insertions(+), 11 deletions(-)
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch 2011-07-05 13:11 [PATCH 0/1] " Yu Ke @ 2011-07-05 13:11 ` Yu Ke 2011-07-08 2:41 ` Yu Ke 2011-07-12 9:41 ` Richard Purdie 0 siblings, 2 replies; 6+ messages in thread From: Yu Ke @ 2011-07-05 13:11 UTC (permalink / raw) To: bitbake-devel This patch is for [YOCTO #1085] fix. If the upstream fails a checksum, retry from the MIRROR before giving up. This will add more robust fetching if an upstream serves a bad file or webpage. fetching of distcc prior to the move from samba -> googlecode is a good example of this. Signed-off-by: Yu Ke <ke.yu@intel.com> --- bitbake/lib/bb/fetch2/__init__.py | 27 ++++++++++++++++----------- 1 files changed, 16 insertions(+), 11 deletions(-) diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py index e9a64c5..bb9dbbe 100644 --- a/bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/__init__.py @@ -278,6 +278,15 @@ def verify_checksum(u, ud, d): if BB_STRICT_CHECKSUM = "1" then return false as unmatched, otherwise return true as matched """ + # Only check the checksums if we've not seen this item before + if os.path.exists(ud.donestamp): + # Touch the done stamp file to show active use of the download + try: + os.utime(ud.donestamp, None) + except: + # Errors aren't fatal here + pass + return if not ud.type in ["http", "https", "ftp", "ftps"]: return @@ -300,6 +309,9 @@ def verify_checksum(u, ud, d): if ud.sha256_expected != sha256data: raise SHA256SumError(ud.localpath, ud.sha256_expected, sha256data, u) + # Create the stamp to avoid duplicate checking + open(ud.donestamp, 'w').close() + def subprocess_setup(): import signal # Python installs a SIGPIPE handler by default. This is usually not what @@ -932,6 +944,9 @@ class Fetch(object): if hasattr(m, "build_mirror_data"): m.build_mirror_data(u, ud, self.d) localpath = ud.localpath + # early checksum verify, so that if checksum mismatched, + # fetcher still have chance to fetch from mirror + verify_checksum(u, ud, self.d) except bb.fetch2.NetworkAccess: raise @@ -948,17 +963,7 @@ class Fetch(object): if not localpath or ((not os.path.exists(localpath)) and localpath.find("*") == -1): raise FetchError("Unable to fetch URL %s from any source." % u, u) - if os.path.exists(ud.donestamp): - # Touch the done stamp file to show active use of the download - try: - os.utime(ud.donestamp, None) - except: - # Errors aren't fatal here - pass - else: - # Only check the checksums if we've not seen this item before, then create the stamp - verify_checksum(u, ud, self.d) - open(ud.donestamp, 'w').close() + verify_checksum(u, ud, self.d) finally: bb.utils.unlockfile(lf) -- 1.7.0.4 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch 2011-07-05 13:11 ` [PATCH 1/1] " Yu Ke @ 2011-07-08 2:41 ` Yu Ke 2011-07-12 9:41 ` Richard Purdie 1 sibling, 0 replies; 6+ messages in thread From: Yu Ke @ 2011-07-08 2:41 UTC (permalink / raw) To: Yu Ke; +Cc: bitbake-devel Ping. any comment on this one? on 2011-7-5 21:11, Yu Ke wrote: > This patch is for [YOCTO #1085] fix. > > If the upstream fails a checksum, retry from the MIRROR before giving up. > This will add more robust fetching if an upstream serves a bad file or webpage. > > fetching of distcc prior to the move from samba -> googlecode is a good example > of this. > > Signed-off-by: Yu Ke<ke.yu@intel.com> > --- > bitbake/lib/bb/fetch2/__init__.py | 27 ++++++++++++++++----------- > 1 files changed, 16 insertions(+), 11 deletions(-) > > diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py > index e9a64c5..bb9dbbe 100644 > --- a/bitbake/lib/bb/fetch2/__init__.py > +++ b/bitbake/lib/bb/fetch2/__init__.py > @@ -278,6 +278,15 @@ def verify_checksum(u, ud, d): > if BB_STRICT_CHECKSUM = "1" then return false as unmatched, otherwise return true as > matched > """ > + # Only check the checksums if we've not seen this item before > + if os.path.exists(ud.donestamp): > + # Touch the done stamp file to show active use of the download > + try: > + os.utime(ud.donestamp, None) > + except: > + # Errors aren't fatal here > + pass > + return > > if not ud.type in ["http", "https", "ftp", "ftps"]: > return > @@ -300,6 +309,9 @@ def verify_checksum(u, ud, d): > if ud.sha256_expected != sha256data: > raise SHA256SumError(ud.localpath, ud.sha256_expected, sha256data, u) > > + # Create the stamp to avoid duplicate checking > + open(ud.donestamp, 'w').close() > + > def subprocess_setup(): > import signal > # Python installs a SIGPIPE handler by default. This is usually not what > @@ -932,6 +944,9 @@ class Fetch(object): > if hasattr(m, "build_mirror_data"): > m.build_mirror_data(u, ud, self.d) > localpath = ud.localpath > + # early checksum verify, so that if checksum mismatched, > + # fetcher still have chance to fetch from mirror > + verify_checksum(u, ud, self.d) > > except bb.fetch2.NetworkAccess: > raise > @@ -948,17 +963,7 @@ class Fetch(object): > if not localpath or ((not os.path.exists(localpath)) and localpath.find("*") == -1): > raise FetchError("Unable to fetch URL %s from any source." % u, u) > > - if os.path.exists(ud.donestamp): > - # Touch the done stamp file to show active use of the download > - try: > - os.utime(ud.donestamp, None) > - except: > - # Errors aren't fatal here > - pass > - else: > - # Only check the checksums if we've not seen this item before, then create the stamp > - verify_checksum(u, ud, self.d) > - open(ud.donestamp, 'w').close() > + verify_checksum(u, ud, self.d) > > finally: > bb.utils.unlockfile(lf) ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch 2011-07-05 13:11 ` [PATCH 1/1] " Yu Ke 2011-07-08 2:41 ` Yu Ke @ 2011-07-12 9:41 ` Richard Purdie 1 sibling, 0 replies; 6+ messages in thread From: Richard Purdie @ 2011-07-12 9:41 UTC (permalink / raw) To: Yu Ke; +Cc: bitbake-devel On Tue, 2011-07-05 at 21:11 +0800, Yu Ke wrote: > This patch is for [YOCTO #1085] fix. > > If the upstream fails a checksum, retry from the MIRROR before giving up. > This will add more robust fetching if an upstream serves a bad file or webpage. > > fetching of distcc prior to the move from samba -> googlecode is a good example > of this. > > Signed-off-by: Yu Ke <ke.yu@intel.com> > --- > bitbake/lib/bb/fetch2/__init__.py | 27 ++++++++++++++++----------- > 1 files changed, 16 insertions(+), 11 deletions(-) > > diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py > index e9a64c5..bb9dbbe 100644 > --- a/bitbake/lib/bb/fetch2/__init__.py > +++ b/bitbake/lib/bb/fetch2/__init__.py > @@ -278,6 +278,15 @@ def verify_checksum(u, ud, d): > if BB_STRICT_CHECKSUM = "1" then return false as unmatched, otherwise return true as > matched > """ > + # Only check the checksums if we've not seen this item before > + if os.path.exists(ud.donestamp): > + # Touch the done stamp file to show active use of the download > + try: > + os.utime(ud.donestamp, None) > + except: > + # Errors aren't fatal here > + pass > + return > > if not ud.type in ["http", "https", "ftp", "ftps"]: > return > @@ -300,6 +309,9 @@ def verify_checksum(u, ud, d): > if ud.sha256_expected != sha256data: > raise SHA256SumError(ud.localpath, ud.sha256_expected, sha256data, u) > > + # Create the stamp to avoid duplicate checking > + open(ud.donestamp, 'w').close() > + > def subprocess_setup(): > import signal > # Python installs a SIGPIPE handler by default. This is usually not what > @@ -932,6 +944,9 @@ class Fetch(object): > if hasattr(m, "build_mirror_data"): > m.build_mirror_data(u, ud, self.d) > localpath = ud.localpath > + # early checksum verify, so that if checksum mismatched, > + # fetcher still have chance to fetch from mirror > + verify_checksum(u, ud, self.d) > > except bb.fetch2.NetworkAccess: > raise > @@ -948,17 +963,7 @@ class Fetch(object): > if not localpath or ((not os.path.exists(localpath)) and localpath.find("*") == -1): > raise FetchError("Unable to fetch URL %s from any source." % u, u) > > - if os.path.exists(ud.donestamp): > - # Touch the done stamp file to show active use of the download > - try: > - os.utime(ud.donestamp, None) > - except: > - # Errors aren't fatal here > - pass > - else: > - # Only check the checksums if we've not seen this item before, then create the stamp > - verify_checksum(u, ud, self.d) > - open(ud.donestamp, 'w').close() > + verify_checksum(u, ud, self.d) > > finally: > bb.utils.unlockfile(lf) Hi Ke, I find this patch a little confusing since you're altering what the verify_checksum() function does to including something else which is partly unrelated. I like the idea of moving that logic to a separate function but perhaps we need a new one which then calls verify_checksum() rather than having the two functionalities merged together? Perhaps an update_stamps() function would be a good name for this? Cheers, Richard ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-07-13 11:17 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-07-13 9:08 [PATCH 0/1] fetcher2: retry mirror if upstream checksum mismatch V2 Yu Ke 2011-07-13 9:08 ` [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch Yu Ke 2011-07-13 11:13 ` Richard Purdie -- strict thread matches above, loose matches on Subject: below -- 2011-07-05 13:11 [PATCH 0/1] " Yu Ke 2011-07-05 13:11 ` [PATCH 1/1] " Yu Ke 2011-07-08 2:41 ` Yu Ke 2011-07-12 9:41 ` Richard Purdie
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.