SRC checksum checking broken

SRC checksum checking broken

[Gary Thomas]

[-- Attachment #1: Type: text/plain, Size: 1105 bytes --]

I'm not sure how, but this revision broke the checking of checksums:

commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Fri Mar 23 14:52:23 2012 +0000

     bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them

     [YOCTO #2154]

     (Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)

     Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

The attached log shows this failure - I've modified the busybox_1.19.4.bb
recipe to have incorrect checksums.  With revision 67c83a8, the failure
is correctly identified.  With revision 3d69d94, it is not.

The error happens with direct downloads (using DL_DIR) and also
with a mirror setup (using own-mirrors).

I've reviewed the change and I don't see how/why this happens.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

[-- Attachment #2: bb.fail --]
[-- Type: text/plain, Size: 10695 bytes --]

[gthomas@titan poky_test]$ bitbake busybox -c patch
Loading cache: 100% |####################################################################################| ETA:  00:00:00
Loaded 1108 entries from dependency cache.

OE Build Configuration:
BB_VERSION        = "1.15.1"
TARGET_ARCH       = "arm"
TARGET_OS         = "linux-gnueabi"
MACHINE           = "beagleboard"
DISTRO            = "poky"
DISTRO_VERSION    = "1.1+snapshot-20120410"
TUNE_FEATURES     = "armv7a vfp neon cortexa8"
TARGET_FPU        = "vfp-neon"
meta              
meta-yocto        = "(nobranch):67c83a8c0838c28cfbbb792daa83aa1f3c331302"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Running task 8 of 10 (ID: 2, /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch)
NOTE: package busybox-1.19.4-r0: task do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Checksum mismatch!
File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has md5 checksum 9c0cae5a0379228e7b55e5b29528df8e when 9c0cae5a0379228e7b55e5b29528df8f was expected
File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has sha256 checksum 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ee when 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ef was expected
ERROR: Logfile of failure stored in: /home/local/poky_test/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/busybox-1.19.4-r0/temp/log.do_fetch.3868
Log data follows:
| DEBUG: Trying PREMIRRORS
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['bzr', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['cvs', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['git', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['hg', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['osc', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['p4', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['svk', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['svn', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: Trying Upstream
| NOTE: fetch http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2
| DEBUG: executing /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2'
| DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2'
| DEBUG: Running export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts"; /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2'
| WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
| DEBUG: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Checksum mismatch!
| File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has md5 checksum 9c0cae5a0379228e7b55e5b29528df8e when 9c0cae5a0379228e7b55e5b29528df8f was expected
| File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has sha256 checksum 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ee when 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ef was expected
| DEBUG: Trying MIRRORS
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['ftp', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball returning http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2;name=tarball
| NOTE: fetch http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2
| DEBUG: executing /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2'
| DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2'
| DEBUG: Running export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts"; /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2'
| ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Checksum mismatch!
| File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has md5 checksum 9c0cae5a0379228e7b55e5b29528df8e when 9c0cae5a0379228e7b55e5b29528df8f was expected
| File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has sha256 checksum 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ee when 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ef was expected
NOTE: package busybox-1.19.4-r0: task do_fetch: Failed
ERROR: Task 2 (/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 8 tasks of which 7 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch
Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.


[gthomas@titan poky_test]$ bitbake busybox -c patch
Loading cache: 100% |####################################################################################| ETA:  00:00:00
Loaded 1108 entries from dependency cache.

OE Build Configuration:
BB_VERSION        = "1.15.1"
TARGET_ARCH       = "arm"
TARGET_OS         = "linux-gnueabi"
MACHINE           = "beagleboard"
DISTRO            = "poky"
DISTRO_VERSION    = "1.1+snapshot-20120410"
TUNE_FEATURES     = "armv7a vfp neon cortexa8"
TARGET_FPU        = "vfp-neon"
meta              
meta-yocto        = "(nobranch):3d69d9462d550ce4e00e14768cc616bc9ad7e8a5"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Running task 8 of 10 (ID: 2, /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch)
NOTE: package busybox-1.19.4-r0: task do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
NOTE: package busybox-1.19.4-r0: task do_fetch: Succeeded
NOTE: Running task 9 of 10 (ID: 0, /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_unpack)
NOTE: package busybox-1.19.4-r0: task do_unpack: Started
NOTE: package busybox-1.19.4-r0: task do_unpack: Succeeded
NOTE: Running task 10 of 10 (ID: 1, /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_patch)
NOTE: package busybox-1.19.4-r0: task do_patch: Started
NOTE: package busybox-1.19.4-r0: task do_patch: Succeeded
NOTE: Tasks Summary: Attempted 10 tasks of which 7 didn't need to be rerun and all succeeded.

Summary: There was 1 WARNING message shown.

Re: SRC checksum checking broken

[Richard Purdie]

On Tue, 2012-04-10 at 14:10 -0600, Gary Thomas wrote:
> I'm not sure how, but this revision broke the checking of checksums:
> 
> commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> Date:   Fri Mar 23 14:52:23 2012 +0000
> 
>      bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
> 
>      [YOCTO #2154]
> 
>      (Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)
> 
>      Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> The attached log shows this failure - I've modified the busybox_1.19.4.bb
> recipe to have incorrect checksums.  With revision 67c83a8, the failure
> is correctly identified.  With revision 3d69d94, it is not.
> 
> The error happens with direct downloads (using DL_DIR) and also
> with a mirror setup (using own-mirrors).
> 
> I've reviewed the change and I don't see how/why this happens.

There was a patch just merged for something that sounds very similar. I
was going to give you a link and realised something has gone wrong with
the commit:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6

as its been squashed with a subsequent commit I had to resolve a
conflict with. 

Regardless, could you see whether that change solves your problem
please?

Cheers,

Richard

Re: SRC checksum checking broken

[Gary Thomas]

On 2012-04-10 17:27, Richard Purdie wrote:
> On Tue, 2012-04-10 at 14:10 -0600, Gary Thomas wrote:
>> I'm not sure how, but this revision broke the checking of checksums:
>>
>> commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
>> Author: Richard Purdie<richard.purdie@linuxfoundation.org>
>> Date:   Fri Mar 23 14:52:23 2012 +0000
>>
>>       bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
>>
>>       [YOCTO #2154]
>>
>>       (Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)
>>
>>       Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
>>
>> The attached log shows this failure - I've modified the busybox_1.19.4.bb
>> recipe to have incorrect checksums.  With revision 67c83a8, the failure
>> is correctly identified.  With revision 3d69d94, it is not.
>>
>> The error happens with direct downloads (using DL_DIR) and also
>> with a mirror setup (using own-mirrors).
>>
>> I've reviewed the change and I don't see how/why this happens.
>
> There was a patch just merged for something that sounds very similar. I
> was going to give you a link and realised something has gone wrong with
> the commit:
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
>
> as its been squashed with a subsequent commit I had to resolve a
> conflict with.
>
> Regardless, could you see whether that change solves your problem
> please?

No change, sorry.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: SRC checksum checking broken

[Gary Thomas]

On 2012-04-10 18:14, Gary Thomas wrote:
> On 2012-04-10 17:27, Richard Purdie wrote:
>> On Tue, 2012-04-10 at 14:10 -0600, Gary Thomas wrote:
>>> I'm not sure how, but this revision broke the checking of checksums:
>>>
>>> commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
>>> Author: Richard Purdie<richard.purdie@linuxfoundation.org>
>>> Date: Fri Mar 23 14:52:23 2012 +0000
>>>
>>> bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
>>>
>>> [YOCTO #2154]
>>>
>>> (Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)
>>>
>>> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
>>>
>>> The attached log shows this failure - I've modified the busybox_1.19.4.bb
>>> recipe to have incorrect checksums. With revision 67c83a8, the failure
>>> is correctly identified. With revision 3d69d94, it is not.
>>>
>>> The error happens with direct downloads (using DL_DIR) and also
>>> with a mirror setup (using own-mirrors).
>>>
>>> I've reviewed the change and I don't see how/why this happens.
>>
>> There was a patch just merged for something that sounds very similar. I
>> was going to give you a link and realised something has gone wrong with
>> the commit:
>>
>> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
>>
>> as its been squashed with a subsequent commit I had to resolve a
>> conflict with.
>>
>> Regardless, could you see whether that change solves your problem
>> please?
>
> No change, sorry.
>

I think the problem is this part of the change:
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 7524fa9..e5c4b34 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):

              if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
                  ud.method.download(newuri, ud, ld)
+                open(ud.donestamp, 'w').close()
                  if hasattr(ud.method,"build_mirror_data"):
                      ud.method.build_mirror_data(newuri, ud, ld)

I think the call to open() that creates the .done stamp file should really
be a call to the update_stamp() function.  This is the only place checksums
are being checked (as far as I can tell) and the fact that the .done file
has already been created is getting in the way. I'd suggest a patch, but I don't
understand enough of the context to figure out the parameters to call update_stamp() with
[parameters named u, ud & d don't help me much]

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: SRC checksum checking broken

[Gary Thomas]

On 2012-04-10 19:36, Gary Thomas wrote:
> On 2012-04-10 18:14, Gary Thomas wrote:
>> On 2012-04-10 17:27, Richard Purdie wrote:
>>> On Tue, 2012-04-10 at 14:10 -0600, Gary Thomas wrote:
>>>> I'm not sure how, but this revision broke the checking of checksums:
>>>>
>>>> commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
>>>> Author: Richard Purdie<richard.purdie@linuxfoundation.org>
>>>> Date: Fri Mar 23 14:52:23 2012 +0000
>>>>
>>>> bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
>>>>
>>>> [YOCTO #2154]
>>>>
>>>> (Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)
>>>>
>>>> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
>>>>
>>>> The attached log shows this failure - I've modified the busybox_1.19.4.bb
>>>> recipe to have incorrect checksums. With revision 67c83a8, the failure
>>>> is correctly identified. With revision 3d69d94, it is not.
>>>>
>>>> The error happens with direct downloads (using DL_DIR) and also
>>>> with a mirror setup (using own-mirrors).
>>>>
>>>> I've reviewed the change and I don't see how/why this happens.
>>>
>>> There was a patch just merged for something that sounds very similar. I
>>> was going to give you a link and realised something has gone wrong with
>>> the commit:
>>>
>>> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
>>>
>>> as its been squashed with a subsequent commit I had to resolve a
>>> conflict with.
>>>
>>> Regardless, could you see whether that change solves your problem
>>> please?
>>
>> No change, sorry.
>>
>
> I think the problem is this part of the change:
> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index 7524fa9..e5c4b34 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):
>
> if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
> ud.method.download(newuri, ud, ld)
> + open(ud.donestamp, 'w').close()
> if hasattr(ud.method,"build_mirror_data"):
> ud.method.build_mirror_data(newuri, ud, ld)
>
> I think the call to open() that creates the .done stamp file should really
> be a call to the update_stamp() function. This is the only place checksums
> are being checked (as far as I can tell) and the fact that the .done file
> has already been created is getting in the way. I'd suggest a patch, but I don't
> understand enough of the context to figure out the parameters to call update_stamp() with
> [parameters named u, ud & d don't help me much]

This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6)
Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: SRC checksum checking broken

[Richard Purdie]

On Tue, 2012-04-17 at 06:17 -0600, Gary Thomas wrote:
> On 2012-04-10 19:36, Gary Thomas wrote:
> > On 2012-04-10 18:14, Gary Thomas wrote:
> >> On 2012-04-10 17:27, Richard Purdie wrote:
> >>> On Tue, 2012-04-10 at 14:10 -0600, Gary Thomas wrote:
> >>>> I'm not sure how, but this revision broke the checking of checksums:
> >>>>
> >>>> commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
> >>>> Author: Richard Purdie<richard.purdie@linuxfoundation.org>
> >>>> Date: Fri Mar 23 14:52:23 2012 +0000
> >>>>
> >>>> bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
> >>>>
> >>>> [YOCTO #2154]
> >>>>
> >>>> (Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)
> >>>>
> >>>> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
> >>>>
> >>>> The attached log shows this failure - I've modified the busybox_1.19.4.bb
> >>>> recipe to have incorrect checksums. With revision 67c83a8, the failure
> >>>> is correctly identified. With revision 3d69d94, it is not.
> >>>>
> >>>> The error happens with direct downloads (using DL_DIR) and also
> >>>> with a mirror setup (using own-mirrors).
> >>>>
> >>>> I've reviewed the change and I don't see how/why this happens.
> >>>
> >>> There was a patch just merged for something that sounds very similar. I
> >>> was going to give you a link and realised something has gone wrong with
> >>> the commit:
> >>>
> >>> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
> >>>
> >>> as its been squashed with a subsequent commit I had to resolve a
> >>> conflict with.
> >>>
> >>> Regardless, could you see whether that change solves your problem
> >>> please?
> >>
> >> No change, sorry.
> >>
> >
> > I think the problem is this part of the change:
> > diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> > index 7524fa9..e5c4b34 100644
> > --- a/bitbake/lib/bb/fetch2/__init__.py
> > +++ b/bitbake/lib/bb/fetch2/__init__.py
> > @@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):
> >
> > if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
> > ud.method.download(newuri, ud, ld)
> > + open(ud.donestamp, 'w').close()
> > if hasattr(ud.method,"build_mirror_data"):
> > ud.method.build_mirror_data(newuri, ud, ld)
> >
> > I think the call to open() that creates the .done stamp file should really
> > be a call to the update_stamp() function. This is the only place checksums
> > are being checked (as far as I can tell) and the fact that the .done file
> > has already been created is getting in the way. I'd suggest a patch, but I don't
> > understand enough of the context to figure out the parameters to call update_stamp() with
> > [parameters named u, ud & d don't help me much]
> 
> This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6)
> Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311

Does this help?:

bitbake/fetch2: Ensure directly mirrored files have their checksum validated

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 414cc2b..e36d3ec 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False):
             if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
                 ud.method.download(newuri, ud, ld)
                 if os.path.exists(ud.localpath):
-                    open(ud.donestamp, 'w').close()
+                    if ud.localpath == origud.localpath:
+                        update_stamp(newuri, origud, ld)
+                    else:
+                        open(ud.donestamp, 'w').close()
                     if hasattr(ud.method,"build_mirror_data"):
                         ud.method.build_mirror_data(newuri, ud, ld)
 

Cheers,

Richard

Re: SRC checksum checking broken

[Gary Thomas]

On 2012-04-17 09:01, Richard Purdie wrote:
> On Tue, 2012-04-17 at 06:17 -0600, Gary Thomas wrote:
>> On 2012-04-10 19:36, Gary Thomas wrote:
>>> On 2012-04-10 18:14, Gary Thomas wrote:
>>>> On 2012-04-10 17:27, Richard Purdie wrote:
>>>>> On Tue, 2012-04-10 at 14:10 -0600, Gary Thomas wrote:
>>>>>> I'm not sure how, but this revision broke the checking of checksums:
>>>>>>
>>>>>> commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
>>>>>> Author: Richard Purdie<richard.purdie@linuxfoundation.org>
>>>>>> Date: Fri Mar 23 14:52:23 2012 +0000
>>>>>>
>>>>>> bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
>>>>>>
>>>>>> [YOCTO #2154]
>>>>>>
>>>>>> (Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)
>>>>>>
>>>>>> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
>>>>>>
>>>>>> The attached log shows this failure - I've modified the busybox_1.19.4.bb
>>>>>> recipe to have incorrect checksums. With revision 67c83a8, the failure
>>>>>> is correctly identified. With revision 3d69d94, it is not.
>>>>>>
>>>>>> The error happens with direct downloads (using DL_DIR) and also
>>>>>> with a mirror setup (using own-mirrors).
>>>>>>
>>>>>> I've reviewed the change and I don't see how/why this happens.
>>>>>
>>>>> There was a patch just merged for something that sounds very similar. I
>>>>> was going to give you a link and realised something has gone wrong with
>>>>> the commit:
>>>>>
>>>>> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
>>>>>
>>>>> as its been squashed with a subsequent commit I had to resolve a
>>>>> conflict with.
>>>>>
>>>>> Regardless, could you see whether that change solves your problem
>>>>> please?
>>>>
>>>> No change, sorry.
>>>>
>>>
>>> I think the problem is this part of the change:
>>> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
>>> index 7524fa9..e5c4b34 100644
>>> --- a/bitbake/lib/bb/fetch2/__init__.py
>>> +++ b/bitbake/lib/bb/fetch2/__init__.py
>>> @@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):
>>>
>>> if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
>>> ud.method.download(newuri, ud, ld)
>>> + open(ud.donestamp, 'w').close()
>>> if hasattr(ud.method,"build_mirror_data"):
>>> ud.method.build_mirror_data(newuri, ud, ld)
>>>
>>> I think the call to open() that creates the .done stamp file should really
>>> be a call to the update_stamp() function. This is the only place checksums
>>> are being checked (as far as I can tell) and the fact that the .done file
>>> has already been created is getting in the way. I'd suggest a patch, but I don't
>>> understand enough of the context to figure out the parameters to call update_stamp() with
>>> [parameters named u, ud&  d don't help me much]
>>
>> This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6)
>> Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311
>
> Does this help?:
>
> bitbake/fetch2: Ensure directly mirrored files have their checksum validated
>
> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
> ---
> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index 414cc2b..e36d3ec 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False):
>               if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
>                   ud.method.download(newuri, ud, ld)
>                   if os.path.exists(ud.localpath):
> -                    open(ud.donestamp, 'w').close()
> +                    if ud.localpath == origud.localpath:
> +                        update_stamp(newuri, origud, ld)
> +                    else:
> +                        open(ud.donestamp, 'w').close()
>                       if hasattr(ud.method,"build_mirror_data"):
>                           ud.method.build_mirror_data(newuri, ud, ld)

It does fix the case when using DL_DIR, but own-mirrors is still broken.
Note: both used to work correctly.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: SRC checksum checking broken

[Richard Purdie]

On Tue, 2012-04-17 at 09:21 -0600, Gary Thomas wrote:
> On 2012-04-17 09:01, Richard Purdie wrote:
> > On Tue, 2012-04-17 at 06:17 -0600, Gary Thomas wrote:
> >> On 2012-04-10 19:36, Gary Thomas wrote:
> >>> On 2012-04-10 18:14, Gary Thomas wrote:
> >>>> On 2012-04-10 17:27, Richard Purdie wrote:
> >>>>> On Tue, 2012-04-10 at 14:10 -0600, Gary Thomas wrote:
> >>>>>> I'm not sure how, but this revision broke the checking of checksums:
> >>>>>>
> >>>>>> commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
> >>>>>> Author: Richard Purdie<richard.purdie@linuxfoundation.org>
> >>>>>> Date: Fri Mar 23 14:52:23 2012 +0000
> >>>>>>
> >>>>>> bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
> >>>>>>
> >>>>>> [YOCTO #2154]
> >>>>>>
> >>>>>> (Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)
> >>>>>>
> >>>>>> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
> >>>>>>
> >>>>>> The attached log shows this failure - I've modified the busybox_1.19.4.bb
> >>>>>> recipe to have incorrect checksums. With revision 67c83a8, the failure
> >>>>>> is correctly identified. With revision 3d69d94, it is not.
> >>>>>>
> >>>>>> The error happens with direct downloads (using DL_DIR) and also
> >>>>>> with a mirror setup (using own-mirrors).
> >>>>>>
> >>>>>> I've reviewed the change and I don't see how/why this happens.
> >>>>>
> >>>>> There was a patch just merged for something that sounds very similar. I
> >>>>> was going to give you a link and realised something has gone wrong with
> >>>>> the commit:
> >>>>>
> >>>>> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
> >>>>>
> >>>>> as its been squashed with a subsequent commit I had to resolve a
> >>>>> conflict with.
> >>>>>
> >>>>> Regardless, could you see whether that change solves your problem
> >>>>> please?
> >>>>
> >>>> No change, sorry.
> >>>>
> >>>
> >>> I think the problem is this part of the change:
> >>> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> >>> index 7524fa9..e5c4b34 100644
> >>> --- a/bitbake/lib/bb/fetch2/__init__.py
> >>> +++ b/bitbake/lib/bb/fetch2/__init__.py
> >>> @@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):
> >>>
> >>> if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
> >>> ud.method.download(newuri, ud, ld)
> >>> + open(ud.donestamp, 'w').close()
> >>> if hasattr(ud.method,"build_mirror_data"):
> >>> ud.method.build_mirror_data(newuri, ud, ld)
> >>>
> >>> I think the call to open() that creates the .done stamp file should really
> >>> be a call to the update_stamp() function. This is the only place checksums
> >>> are being checked (as far as I can tell) and the fact that the .done file
> >>> has already been created is getting in the way. I'd suggest a patch, but I don't
> >>> understand enough of the context to figure out the parameters to call update_stamp() with
> >>> [parameters named u, ud&  d don't help me much]
> >>
> >> This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6)
> >> Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311
> >
> > Does this help?:
> >
> > bitbake/fetch2: Ensure directly mirrored files have their checksum validated
> >
> > Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
> > ---
> > diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> > index 414cc2b..e36d3ec 100644
> > --- a/bitbake/lib/bb/fetch2/__init__.py
> > +++ b/bitbake/lib/bb/fetch2/__init__.py
> > @@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False):
> >               if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
> >                   ud.method.download(newuri, ud, ld)
> >                   if os.path.exists(ud.localpath):
> > -                    open(ud.donestamp, 'w').close()
> > +                    if ud.localpath == origud.localpath:
> > +                        update_stamp(newuri, origud, ld)
> > +                    else:
> > +                        open(ud.donestamp, 'w').close()
> >                       if hasattr(ud.method,"build_mirror_data"):
> >                           ud.method.build_mirror_data(newuri, ud, ld)
> 
> It does fix the case when using DL_DIR, but own-mirrors is still broken.
> Note: both used to work correctly.

Can you give more details of the configuration please? Is this using
file:// urls?

For example in the bug you said you did "-c cleansstate" and I'd not
expect that to recheck checksums, you'd need to use a "cleanall" for
that.

Cheers,

Richard

Re: SRC checksum checking broken

[Gary Thomas]

On 2012-04-17 09:44, Richard Purdie wrote:
> On Tue, 2012-04-17 at 09:21 -0600, Gary Thomas wrote:
>> On 2012-04-17 09:01, Richard Purdie wrote:
>>> On Tue, 2012-04-17 at 06:17 -0600, Gary Thomas wrote:
>>>> On 2012-04-10 19:36, Gary Thomas wrote:
>>>>> On 2012-04-10 18:14, Gary Thomas wrote:
>>>>>> On 2012-04-10 17:27, Richard Purdie wrote:
>>>>>>> On Tue, 2012-04-10 at 14:10 -0600, Gary Thomas wrote:
>>>>>>>> I'm not sure how, but this revision broke the checking of checksums:
>>>>>>>>
>>>>>>>> commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
>>>>>>>> Author: Richard Purdie<richard.purdie@linuxfoundation.org>
>>>>>>>> Date: Fri Mar 23 14:52:23 2012 +0000
>>>>>>>>
>>>>>>>> bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
>>>>>>>>
>>>>>>>> [YOCTO #2154]
>>>>>>>>
>>>>>>>> (Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)
>>>>>>>>
>>>>>>>> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
>>>>>>>>
>>>>>>>> The attached log shows this failure - I've modified the busybox_1.19.4.bb
>>>>>>>> recipe to have incorrect checksums. With revision 67c83a8, the failure
>>>>>>>> is correctly identified. With revision 3d69d94, it is not.
>>>>>>>>
>>>>>>>> The error happens with direct downloads (using DL_DIR) and also
>>>>>>>> with a mirror setup (using own-mirrors).
>>>>>>>>
>>>>>>>> I've reviewed the change and I don't see how/why this happens.
>>>>>>>
>>>>>>> There was a patch just merged for something that sounds very similar. I
>>>>>>> was going to give you a link and realised something has gone wrong with
>>>>>>> the commit:
>>>>>>>
>>>>>>> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
>>>>>>>
>>>>>>> as its been squashed with a subsequent commit I had to resolve a
>>>>>>> conflict with.
>>>>>>>
>>>>>>> Regardless, could you see whether that change solves your problem
>>>>>>> please?
>>>>>>
>>>>>> No change, sorry.
>>>>>>
>>>>>
>>>>> I think the problem is this part of the change:
>>>>> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
>>>>> index 7524fa9..e5c4b34 100644
>>>>> --- a/bitbake/lib/bb/fetch2/__init__.py
>>>>> +++ b/bitbake/lib/bb/fetch2/__init__.py
>>>>> @@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):
>>>>>
>>>>> if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
>>>>> ud.method.download(newuri, ud, ld)
>>>>> + open(ud.donestamp, 'w').close()
>>>>> if hasattr(ud.method,"build_mirror_data"):
>>>>> ud.method.build_mirror_data(newuri, ud, ld)
>>>>>
>>>>> I think the call to open() that creates the .done stamp file should really
>>>>> be a call to the update_stamp() function. This is the only place checksums
>>>>> are being checked (as far as I can tell) and the fact that the .done file
>>>>> has already been created is getting in the way. I'd suggest a patch, but I don't
>>>>> understand enough of the context to figure out the parameters to call update_stamp() with
>>>>> [parameters named u, ud&   d don't help me much]
>>>>
>>>> This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6)
>>>> Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311
>>>
>>> Does this help?:
>>>
>>> bitbake/fetch2: Ensure directly mirrored files have their checksum validated
>>>
>>> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
>>> ---
>>> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
>>> index 414cc2b..e36d3ec 100644
>>> --- a/bitbake/lib/bb/fetch2/__init__.py
>>> +++ b/bitbake/lib/bb/fetch2/__init__.py
>>> @@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False):
>>>                if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
>>>                    ud.method.download(newuri, ud, ld)
>>>                    if os.path.exists(ud.localpath):
>>> -                    open(ud.donestamp, 'w').close()
>>> +                    if ud.localpath == origud.localpath:
>>> +                        update_stamp(newuri, origud, ld)
>>> +                    else:
>>> +                        open(ud.donestamp, 'w').close()
>>>                        if hasattr(ud.method,"build_mirror_data"):
>>>                            ud.method.build_mirror_data(newuri, ud, ld)
>>
>> It does fix the case when using DL_DIR, but own-mirrors is still broken.
>> Note: both used to work correctly.
>
> Can you give more details of the configuration please? Is this using
> file:// urls?

Here's what I use.  I've not tried any other transport method:
   SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/"
   INHERIT += "own-mirrors"

>
> For example in the bug you said you did "-c cleansstate" and I'd not
> expect that to recheck checksums, you'd need to use a "cleanall" for
> that.

I manually removed the ${BUILD}/downloads/busybox* - tarballs and stamps
to force it.  I didn't think to use cleanall.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: SRC checksum checking broken

[Richard Purdie]

On Tue, 2012-04-17 at 09:59 -0600, Gary Thomas wrote:
> On 2012-04-17 09:44, Richard Purdie wrote:
> > On Tue, 2012-04-17 at 09:21 -0600, Gary Thomas wrote:
> >> On 2012-04-17 09:01, Richard Purdie wrote:
> >>> On Tue, 2012-04-17 at 06:17 -0600, Gary Thomas wrote:
> >>>> On 2012-04-10 19:36, Gary Thomas wrote:
> >>>> This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6)
> >>>> Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311
> >>>
> >>> Does this help?:
> >>>
> >>> bitbake/fetch2: Ensure directly mirrored files have their checksum validated
> >>>
> >>> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
> >>> ---
> >>> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> >>> index 414cc2b..e36d3ec 100644
> >>> --- a/bitbake/lib/bb/fetch2/__init__.py
> >>> +++ b/bitbake/lib/bb/fetch2/__init__.py
> >>> @@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False):
> >>>                if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
> >>>                    ud.method.download(newuri, ud, ld)
> >>>                    if os.path.exists(ud.localpath):
> >>> -                    open(ud.donestamp, 'w').close()
> >>> +                    if ud.localpath == origud.localpath:
> >>> +                        update_stamp(newuri, origud, ld)
> >>> +                    else:
> >>> +                        open(ud.donestamp, 'w').close()
> >>>                        if hasattr(ud.method,"build_mirror_data"):
> >>>                            ud.method.build_mirror_data(newuri, ud, ld)
> >>
> >> It does fix the case when using DL_DIR, but own-mirrors is still broken.
> >> Note: both used to work correctly.
> >
> > Can you give more details of the configuration please? Is this using
> > file:// urls?
> 
> Here's what I use.  I've not tried any other transport method:
>    SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/"
>    INHERIT += "own-mirrors"

Part of me wants to argue that its not expected to check checksums for
file:// urls, as indicated for example by the code:

def verify_checksum(u, ud, d):
    [...]
    if not ud.type in ["http", "https", "ftp", "ftps"]:
        return

but I can see a case for wanting to check the checksums I guess.

Perhaps this will work better?

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 414cc2b..329b5bc 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -477,7 +477,6 @@ def try_mirrors(d, origud, mirrors, check = False):
             if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
                 ud.method.download(newuri, ud, ld)
                 if os.path.exists(ud.localpath):
-                    open(ud.donestamp, 'w').close()
                     if hasattr(ud.method,"build_mirror_data"):
                         ud.method.build_mirror_data(newuri, ud, ld)
 
@@ -492,12 +491,14 @@ def try_mirrors(d, origud, mirrors, check = False):
             dldir = ld.getVar("DL_DIR", True)
             if os.path.basename(ud.localpath) != os.path.basename(origud.localpath):
                 dest = os.path.join(dldir, os.path.basename(ud.localpath))
+                open(ud.donestamp, 'w').close()
                 if not os.path.exists(dest):
                     os.symlink(ud.localpath, dest)
                 return None
             # Otherwise the result is a local file:// and we symlink to it
             if not os.path.exists(origud.localpath):
                  os.symlink(ud.localpath, origud.localpath)
+            update_stamp(newuri, origud, ld)
             return ud.localpath
 
         except bb.fetch2.NetworkAccess:


Cheers,

Richard

Re: SRC checksum checking broken

[Gary Thomas]

On 2012-04-17 10:19, Richard Purdie wrote:
> On Tue, 2012-04-17 at 09:59 -0600, Gary Thomas wrote:
>> On 2012-04-17 09:44, Richard Purdie wrote:
>>> On Tue, 2012-04-17 at 09:21 -0600, Gary Thomas wrote:
>>>> On 2012-04-17 09:01, Richard Purdie wrote:
>>>>> On Tue, 2012-04-17 at 06:17 -0600, Gary Thomas wrote:
>>>>>> On 2012-04-10 19:36, Gary Thomas wrote:
>>>>>> This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6)
>>>>>> Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311
>>>>>
>>>>> Does this help?:
>>>>>
>>>>> bitbake/fetch2: Ensure directly mirrored files have their checksum validated
>>>>>
>>>>> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
>>>>> ---
>>>>> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
>>>>> index 414cc2b..e36d3ec 100644
>>>>> --- a/bitbake/lib/bb/fetch2/__init__.py
>>>>> +++ b/bitbake/lib/bb/fetch2/__init__.py
>>>>> @@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False):
>>>>>                 if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
>>>>>                     ud.method.download(newuri, ud, ld)
>>>>>                     if os.path.exists(ud.localpath):
>>>>> -                    open(ud.donestamp, 'w').close()
>>>>> +                    if ud.localpath == origud.localpath:
>>>>> +                        update_stamp(newuri, origud, ld)
>>>>> +                    else:
>>>>> +                        open(ud.donestamp, 'w').close()
>>>>>                         if hasattr(ud.method,"build_mirror_data"):
>>>>>                             ud.method.build_mirror_data(newuri, ud, ld)
>>>>
>>>> It does fix the case when using DL_DIR, but own-mirrors is still broken.
>>>> Note: both used to work correctly.
>>>
>>> Can you give more details of the configuration please? Is this using
>>> file:// urls?
>>
>> Here's what I use.  I've not tried any other transport method:
>>     SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/"
>>     INHERIT += "own-mirrors"
>
> Part of me wants to argue that its not expected to check checksums for
> file:// urls, as indicated for example by the code:
>
> def verify_checksum(u, ud, d):
>      [...]
>      if not ud.type in ["http", "https", "ftp", "ftps"]:
>          return
>
> but I can see a case for wanting to check the checksums I guess.

Well, it seems that the checksums are encouraged, if not required, these days.
I was just making sure that they actually work & in my opinion, it shouldn't
matter how you got the file, the checksums should still be checked and match.

>
> Perhaps this will work better?
>
> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index 414cc2b..329b5bc 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -477,7 +477,6 @@ def try_mirrors(d, origud, mirrors, check = False):
>               if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
>                   ud.method.download(newuri, ud, ld)
>                   if os.path.exists(ud.localpath):
> -                    open(ud.donestamp, 'w').close()
>                       if hasattr(ud.method,"build_mirror_data"):
>                           ud.method.build_mirror_data(newuri, ud, ld)
>
> @@ -492,12 +491,14 @@ def try_mirrors(d, origud, mirrors, check = False):
>               dldir = ld.getVar("DL_DIR", True)
>               if os.path.basename(ud.localpath) != os.path.basename(origud.localpath):
>                   dest = os.path.join(dldir, os.path.basename(ud.localpath))
> +                open(ud.donestamp, 'w').close()
>                   if not os.path.exists(dest):
>                       os.symlink(ud.localpath, dest)
>                   return None
>               # Otherwise the result is a local file:// and we symlink to it
>               if not os.path.exists(origud.localpath):
>                    os.symlink(ud.localpath, origud.localpath)
> +            update_stamp(newuri, origud, ld)
>               return ud.localpath
>
>           except bb.fetch2.NetworkAccess:

Sadly, this is much worse.  It even [somehow] reached "through" my
mirror setup and deleted the file on the backing store (i.e. the
file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted
when the checksum fails even though it's part of the own-mirror)

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: SRC checksum checking broken

[Richard Purdie]

On Tue, 2012-04-17 at 10:59 -0600, Gary Thomas wrote:
> On 2012-04-17 10:19, Richard Purdie wrote:
> Well, it seems that the checksums are encouraged, if not required, these days.
> I was just making sure that they actually work & in my opinion, it shouldn't
> matter how you got the file, the checksums should still be checked and match.

You can argue this both ways. I'm leaning that direction although it has
its risks.

> Sadly, this is much worse.  It even [somehow] reached "through" my
> mirror setup and deleted the file on the backing store (i.e. the
> file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted
> when the checksum fails even though it's part of the own-mirror)

Was that the only issue? I can see how it managed to do that, I agree it
shouldn't :/.

Cheers,

Richard

Re: SRC checksum checking broken

[Gary Thomas]

On 2012-04-17 15:20, Richard Purdie wrote:
> On Tue, 2012-04-17 at 10:59 -0600, Gary Thomas wrote:
>> On 2012-04-17 10:19, Richard Purdie wrote:
>> Well, it seems that the checksums are encouraged, if not required, these days.
>> I was just making sure that they actually work&  in my opinion, it shouldn't
>> matter how you got the file, the checksums should still be checked and match.
>
> You can argue this both ways. I'm leaning that direction although it has
> its risks.
>
>> Sadly, this is much worse.  It even [somehow] reached "through" my
>> mirror setup and deleted the file on the backing store (i.e. the
>> file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted
>> when the checksum fails even though it's part of the own-mirror)
>
> Was that the only issue? I can see how it managed to do that, I agree it
> shouldn't :/.

It sort of worked - at least it didn't pass silently - but the messages
don't say why it failed.  Here's what I get:

NOTE: package busybox-1.19.4-r2: task do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
ERROR: Fetcher failure: Fetch command export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export 
PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts"; 
/usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2' could not be run:
None
ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /home/local/poky_test/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/busybox-1.19.4-r2/temp/log.do_fetch.7907
NOTE: package busybox-1.19.4-r2: task do_fetch: Failed
ERROR: Task 2 (/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 8 tasks of which 7 didn't need to be rerun and 1 failed.

No indication of checksum failures anywhere.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: SRC checksum checking broken

[Richard Purdie]

On Tue, 2012-04-17 at 16:14 -0600, Gary Thomas wrote:
> On 2012-04-17 15:20, Richard Purdie wrote:
> > On Tue, 2012-04-17 at 10:59 -0600, Gary Thomas wrote:
> >> On 2012-04-17 10:19, Richard Purdie wrote:
> >> Well, it seems that the checksums are encouraged, if not required, these days.
> >> I was just making sure that they actually work&  in my opinion, it shouldn't
> >> matter how you got the file, the checksums should still be checked and match.
> >
> > You can argue this both ways. I'm leaning that direction although it has
> > its risks.
> >
> >> Sadly, this is much worse.  It even [somehow] reached "through" my
> >> mirror setup and deleted the file on the backing store (i.e. the
> >> file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted
> >> when the checksum fails even though it's part of the own-mirror)
> >
> > Was that the only issue? I can see how it managed to do that, I agree it
> > shouldn't :/.
> 
> It sort of worked - at least it didn't pass silently - but the messages
> don't say why it failed.  Here's what I get:
> 
> NOTE: package busybox-1.19.4-r2: task do_fetch: Started
> WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
> ERROR: Fetcher failure: Fetch command export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export 
> PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts"; 
> /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2' could not be run:
> None
> ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Unable to fetch URL from any source.
> ERROR: Logfile of failure stored in: /home/local/poky_test/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/busybox-1.19.4-r2/temp/log.do_fetch.7907
> NOTE: package busybox-1.19.4-r2: task do_fetch: Failed
> ERROR: Task 2 (/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch) failed with exit code '1'
> NOTE: Tasks Summary: Attempted 8 tasks of which 7 didn't need to be rerun and 1 failed.
> 
> No indication of checksum failures anywhere.

Having gone over this several times, I think the original patch is
correct but it highlights some other issues. The nastiest of those is
the deletion of files. I think I've found a way of fixing that by
correctly using fetcher clean functions (which are null operations for
the local fetcher).

The remaining issue is then one of better output from the fetcher as
there wasn't enough info in the above logs but that is really a separate
problem.

I've sent out several fetcher patches to the bitbake list which will
hopefully address some of these issues. I'm travelling at the moment and
these were written without internet access so my testing is limited to
the local file:// urls but I'm confident they move things in the right
direction.

Cheers,

Richard

Re: SRC checksum checking broken

[Gary Thomas]

On 2012-05-13 06:39, Richard Purdie wrote:
> On Tue, 2012-04-17 at 16:14 -0600, Gary Thomas wrote:
>> On 2012-04-17 15:20, Richard Purdie wrote:
>>> On Tue, 2012-04-17 at 10:59 -0600, Gary Thomas wrote:
>>>> On 2012-04-17 10:19, Richard Purdie wrote:
>>>> Well, it seems that the checksums are encouraged, if not required, these days.
>>>> I was just making sure that they actually work&   in my opinion, it shouldn't
>>>> matter how you got the file, the checksums should still be checked and match.
>>>
>>> You can argue this both ways. I'm leaning that direction although it has
>>> its risks.
>>>
>>>> Sadly, this is much worse.  It even [somehow] reached "through" my
>>>> mirror setup and deleted the file on the backing store (i.e. the
>>>> file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted
>>>> when the checksum fails even though it's part of the own-mirror)
>>>
>>> Was that the only issue? I can see how it managed to do that, I agree it
>>> shouldn't :/.
>>
>> It sort of worked - at least it didn't pass silently - but the messages
>> don't say why it failed.  Here's what I get:
>>
>> NOTE: package busybox-1.19.4-r2: task do_fetch: Started
>> WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
>> ERROR: Fetcher failure: Fetch command export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export
>> PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts";
>> /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2' could not be run:
>> None
>> ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Unable to fetch URL from any source.
>> ERROR: Logfile of failure stored in: /home/local/poky_test/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/busybox-1.19.4-r2/temp/log.do_fetch.7907
>> NOTE: package busybox-1.19.4-r2: task do_fetch: Failed
>> ERROR: Task 2 (/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch) failed with exit code '1'
>> NOTE: Tasks Summary: Attempted 8 tasks of which 7 didn't need to be rerun and 1 failed.
>>
>> No indication of checksum failures anywhere.
>
> Having gone over this several times, I think the original patch is
> correct but it highlights some other issues. The nastiest of those is
> the deletion of files. I think I've found a way of fixing that by
> correctly using fetcher clean functions (which are null operations for
> the local fetcher).
>
> The remaining issue is then one of better output from the fetcher as
> there wasn't enough info in the above logs but that is really a separate
> problem.
>
> I've sent out several fetcher patches to the bitbake list which will
> hopefully address some of these issues. I'm travelling at the moment and
> these were written without internet access so my testing is limited to
> the local file:// urls but I'm confident they move things in the right
> direction.

Is there something I should check wrt this problem?  A bitbake URL/version?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: SRC checksum checking broken

[Richard Purdie]

On Sun, 2012-05-13 at 16:24 -0600, Gary Thomas wrote:
> On 2012-05-13 06:39, Richard Purdie wrote:
> > Having gone over this several times, I think the original patch is
> > correct but it highlights some other issues. The nastiest of those is
> > the deletion of files. I think I've found a way of fixing that by
> > correctly using fetcher clean functions (which are null operations for
> > the local fetcher).
> >
> > The remaining issue is then one of better output from the fetcher as
> > there wasn't enough info in the above logs but that is really a separate
> > problem.
> >
> > I've sent out several fetcher patches to the bitbake list which will
> > hopefully address some of these issues. I'm travelling at the moment and
> > these were written without internet access so my testing is limited to
> > the local file:// urls but I'm confident they move things in the right
> > direction.
> 
> Is there something I should check wrt this problem?  A bitbake URL/version?

I've just checked in a number of patches to master (bitbake and poky)
which should fix the various fetcher issues people have been seeing.

I'd appreciate it if you could test with that and see if this issue is
now resolved.

Cheers,

Richard

Re: SRC checksum checking broken

[Gary Thomas]

On 2012-05-20 02:29, Richard Purdie wrote:
> On Sun, 2012-05-13 at 16:24 -0600, Gary Thomas wrote:
>> On 2012-05-13 06:39, Richard Purdie wrote:
>>> Having gone over this several times, I think the original patch is
>>> correct but it highlights some other issues. The nastiest of those is
>>> the deletion of files. I think I've found a way of fixing that by
>>> correctly using fetcher clean functions (which are null operations for
>>> the local fetcher).
>>>
>>> The remaining issue is then one of better output from the fetcher as
>>> there wasn't enough info in the above logs but that is really a separate
>>> problem.
>>>
>>> I've sent out several fetcher patches to the bitbake list which will
>>> hopefully address some of these issues. I'm travelling at the moment and
>>> these were written without internet access so my testing is limited to
>>> the local file:// urls but I'm confident they move things in the right
>>> direction.
>>
>> Is there something I should check wrt this problem?  A bitbake URL/version?
>
> I've just checked in a number of patches to master (bitbake and poky)
> which should fix the various fetcher issues people have been seeing.
>
> I'd appreciate it if you could test with that and see if this issue is
> now resolved.

Yes, SRC_URI checksum checking seems to be working again.  Tested with
"own-mirrors" and normal fetching.

Thanks

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------