[Buildroot] [Bug 10511] New: Packages get downloaded uncompressed with wget

[Buildroot] [Bug 10511] New: Packages get downloaded uncompressed with wget

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10511

            Bug ID: 10511
           Summary: Packages get downloaded uncompressed with wget
           Product: buildroot
           Version: 2017.08
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: Other
          Assignee: unassigned at buildroot.uclibc.org
          Reporter: jean.parpaillon at free.fr
                CC: buildroot at uclibc.org
  Target Milestone: ---

Using buildroot with wget version 1.19.2 on Debian, through nerves toolchain.
(https://github.com/nerves-project/nerves_system_br)

When downloading otp (http://erlang.org/download/otp_src_20.1.tar.gz), wget
(through pkg-download.mk recipes) uncompress the data, resulting with an
gunzipe'd .tar.gz, hence md5 mismatch.

Downloading the package with 'wget --compress=none' fixed the problem, but may
depend on wget versions.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10511] Packages get downloaded uncompressed with wget

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10511

--- Comment #1 from Thomas Petazzoni <thomas.petazzoni@free-electrons.com> ---
Could you indicate how to reproduce the problem you're seeing with the official
Buildroot, and provide a minimal Buildroot .config / scenario that exhibits the
issue ? Thanks!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10511] Packages get downloaded uncompressed with wget

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10511

--- Comment #2 from Frank Hunleth <fhunleth@troodon-software.com> ---
(In reply to Thomas Petazzoni from comment #1)

This is due to a change in behavior with wget 1.19.2. This discussion describes
the issue: http://lists.gnu.org/archive/html/bug-wget/2017-11/msg00000.html. To
reproduce, I installed wget 1.19.2 on my system and added the following line to
qemu_arm_versatile_defconfig:

BR2_PACKAGE_ERLANG=y

Here's the result of running "make source". You can see the MD5 mismatch from
the erlang.org copy. That's because it was automatically decompressed by wget.

umask 0022 && make -C /home/fhunleth/buildroot.upstream
O=/home/fhunleth/buildroot.upstream/o/wgettest/. source
>>> erlang 20.0 Downloading
--2017-12-18 08:46:07--  http://www.erlang.org/download/otp_src_20.0.tar.gz
Resolving www.erlang.org... 31.172.186.60
Connecting to www.erlang.org|31.172.186.60|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://erlang.org/download/otp_src_20.0.tar.gz [following]
--2017-12-18 08:46:08--  http://erlang.org/download/otp_src_20.0.tar.gz
Resolving erlang.org... 192.121.151.106
Connecting to erlang.org|192.121.151.106|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 87346046 (83M) [application/x-tar]
Saving to:
?/home/fhunleth/buildroot.upstream/o/wgettest/build/.otp_src_20.0.tar.gz.5GQ6qy/output?

/home/fhunleth/buildroot.upstream/o/
100%[======================================================================>] 
83.30M   582KB/s    in 2m 27s  

2017-12-18 08:48:35 (582 KB/s) -
?/home/fhunleth/buildroot.upstream/o/wgettest/build/.otp_src_20.0.tar.gz.5GQ6qy/output?
saved [237383680]

ERROR: otp_src_20.0.tar.gz has wrong md5 hash:
ERROR: expected: 2faed2c3519353e6bc2501ed4d8e6ae7
ERROR: got     : 6d1210c2a3a9b178a751809016ef51d2
ERROR: Incomplete download, or man-in-the-middle (MITM) attack
--2017-12-18 08:48:35--  http://sources.buildroot.net/otp_src_20.0.tar.gz
Resolving sources.buildroot.net... 176.9.16.109
Connecting to sources.buildroot.net|176.9.16.109|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 87346046 (83M) [application/x-gzip]
Saving to:
?/home/fhunleth/buildroot.upstream/o/wgettest/build/.otp_src_20.0.tar.gz.hY7Oae/output?

/home/fhunleth/buildroot.upstream/o/
100%[======================================================================>] 
83.30M  3.56MB/s    in 23s     

2017-12-18 08:48:59 (3.59 MB/s) -
?/home/fhunleth/buildroot.upstream/o/wgettest/build/.otp_src_20.0.tar.gz.hY7Oae/output?
saved [87346046/87346046]

otp_src_20.0.tar.gz: OK (md5: 2faed2c3519353e6bc2501ed4d8e6ae7)
otp_src_20.0.tar.gz: OK (sha256:
fe80e1e14a2772901be717694bb30ac4e9a07eee0cc7a28988724cbd21476811)
otp_src_20.0.tar.gz: OK (md5: 2faed2c3519353e6bc2501ed4d8e6ae7)
otp_src_20.0.tar.gz: OK (sha256:
fe80e1e14a2772901be717694bb30ac4e9a07eee0cc7a28988724cbd21476811)

Verify that the invalidate MD5 was done on the uncompressed file:

$ gunzip otp_src_20.0.tar.gz 
$ md5sum otp_src_20.0.tar 
6d1210c2a3a9b178a751809016ef51d2  otp_src_20.0.tar

It seems like a configuration change on erlang.org could fix the problem as
well, but I'm not sure that's the right answer for Buildroot.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10511] Packages get downloaded uncompressed with wget

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10511

--- Comment #3 from Thomas Petazzoni <thomas.petazzoni@free-electrons.com> ---
Thanks for the investigation. This problem has been fixed/worked-around
upstream by commit
http://git.savannah.gnu.org/cgit/wget.git/commit/?id=047746eb765d212cbf05afac16f4bd73257d8186
which ensures that .gz and .tgz files don't get uncompressed.

However wget 1.19.2 remains affected.

Options:

 - Ignore the problem, and leave users of 1.19.2 affected by the problem.

 - Bail out if wget 1.19.2 is used, and leave it to the user to use a suitable
wget version

 - Detect if wget >= 1.19.2, and pass --compression=none

Thoughts?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10511] Packages get downloaded uncompressed with wget

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10511

Yann E. MORIN <yann.morin.1998@free.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |yann.morin.1998 at free.fr

--- Comment #4 from Yann E. MORIN <yann.morin.1998@free.fr> ---
Thomas, Jean, All,

> - Detect if wget >= 1.19.2, and pass --compression=none

That looks like the best solution IMHO. Unfortunately, wget versions < 1.19.2
do not accept the --compression option, so we really must make it conditional.

My 2ct.

Regards,
Yann E. MORIN.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10511] Packages get downloaded uncompressed with wget

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10511

Thomas Petazzoni <thomas.petazzoni@free-electrons.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |WONTFIX
             Status|NEW                         |RESOLVED

--- Comment #5 from Thomas Petazzoni <thomas.petazzoni@free-electrons.com> ---
This problem only affects wget 1.19.2. In 1.19.3, a patch was merged that
disables auto-decompression for files ending in .gz and .tgz, and then in
1.19.4, the auto-decompression was disabled entirely, as it was causing too
many problems.

Since the problem has been fixed in dot-releases of wget, most reasonable
distributions that were using 1.19.2 have probably upgraded to 1.19.3 or later.

In addition, for those stuck with wget 1.19.2, the fallback to
sources.buildroot.net works (even if admittedly the bogus hashes can be scary).
But considering all of this, we don't want to introduce extra complexity in
Buildroot just to workaround a problem affecting only wget 1.19.2 when
downloading files only from HTTP servers having a specific configuration.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10511] Packages get downloaded uncompressed with wget

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10511

--- Comment #6 from Jean Parpaillon <jean.parpaillon@free.fr> ---
Thanks for your time !

-- 
You are receiving this mail because:
You are on the CC list for the bug.