[PATCH] scripts: make extract-vmlinux support armel/armhf

[PATCH] scripts: make extract-vmlinux support armel/armhf

[Roger Shimizu]

vmlinux/zImage on armel/armhf seems not an ELF, so update the script
scripts/extract-vmlinux to support such case.

This fix is tested on Debian amd64, armel, and armhf platform, with
Debian kernels.

Fixes: 09d481270d44 ("scripts: add extract-vmlinux")
Cc: Corentin Chary <corentincj@iksaif.net>
Cc: Michal Marek <mmarek@suse.com>
Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kbuild@vger.kernel.org
Signed-off-by: Roger Shimizu <rogershimizu@gmail.com>
---

Dear Michal, Masahiro-san, and Russell,

I posted this patch before [0] but didn't get positive response.
Recently when I trace a Debian kernel issue, I still find this patch
useful to locate the root cause [1].

Besides, it's a well known bug confuses other developers [2][3].
So I decide to submit the patch again.

[0]: https://patchwork.kernel.org/patch/8120831/
[1]: https://bugs.debian.org/870185#50
[2]: https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
[3]: https://bugs.linaro.org/show_bug.cgi?id=461

Please kindly help to review. Thank you!

Cheers,
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

 scripts/extract-vmlinux | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/scripts/extract-vmlinux b/scripts/extract-vmlinux
index 5061abcc2540..0c72ecd24969 100755
--- a/scripts/extract-vmlinux
+++ b/scripts/extract-vmlinux
@@ -6,6 +6,7 @@
 # (c) 2009,2010 Dick Streefland <dick@streefland.net>
 #
 # (c) 2011      Corentin Chary <corentin.chary@gmail.com>
+# (c) 2016      Roger Shimizu <rogershimizu@gmail.com>
 #
 # Licensed under the GNU General Public License, version 2 (GPLv2).
 # ----------------------------------------------------------------------
@@ -15,7 +16,14 @@ check_vmlinux()
 	# Use readelf to check if it's a valid ELF
 	# TODO: find a better to way to check that it's really vmlinux
 	#       and not just an elf
-	readelf -h $1 > /dev/null 2>&1 || return 1
+	case "$2" in
+	0|"")
+		readelf -h $1 > /dev/null 2>&1 || return 1
+		;;
+	1)
+	# For ARCH like armel/armhf, vmlinux is not ELF, so we skip the check
+		;;
+	esac
 
 	cat $1
 	exit 0
@@ -31,7 +39,7 @@ try_decompress()
 	do
 		pos=${pos%%:*}
 		tail -c+$pos "$img" | $3 > $tmp 2> /dev/null
-		check_vmlinux $tmp
+		test $? -eq 0 && check_vmlinux $tmp 1
 	done
 }
 
@@ -53,7 +61,7 @@ check_vmlinux $img
 
 # That didn't work, so retry after decompression.
 try_decompress '\037\213\010' xy    gunzip
-try_decompress '\3757zXZ\000' abcde unxz
+try_decompress '\3757zXZ\000' abcde "unxz --single-stream"
 try_decompress 'BZh'          xy    bunzip2
 try_decompress '\135\0\0\0'   xxx   unlzma
 try_decompress '\211\114\132' xy    'lzop -d'
-- 
2.11.0

[PATCH] scripts: make extract-vmlinux support armel/armhf

[Roger Shimizu]

vmlinux/zImage on armel/armhf seems not an ELF, so update the script
scripts/extract-vmlinux to support such case.

This fix is tested on Debian amd64, armel, and armhf platform, with
Debian kernels.

Fixes: 09d481270d44 ("scripts: add extract-vmlinux")
Cc: Corentin Chary <corentincj@iksaif.net>
Cc: Michal Marek <mmarek@suse.com>
Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: linux-arm-kernel at lists.infradead.org
Cc: linux-kbuild at vger.kernel.org
Signed-off-by: Roger Shimizu <rogershimizu@gmail.com>
---

Dear Michal, Masahiro-san, and Russell,

I posted this patch before [0] but didn't get positive response.
Recently when I trace a Debian kernel issue, I still find this patch
useful to locate the root cause [1].

Besides, it's a well known bug confuses other developers [2][3].
So I decide to submit the patch again.

[0]: https://patchwork.kernel.org/patch/8120831/
[1]: https://bugs.debian.org/870185#50
[2]: https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
[3]: https://bugs.linaro.org/show_bug.cgi?id=461

Please kindly help to review. Thank you!

Cheers,
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

 scripts/extract-vmlinux | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/scripts/extract-vmlinux b/scripts/extract-vmlinux
index 5061abcc2540..0c72ecd24969 100755
--- a/scripts/extract-vmlinux
+++ b/scripts/extract-vmlinux
@@ -6,6 +6,7 @@
 # (c) 2009,2010 Dick Streefland <dick@streefland.net>
 #
 # (c) 2011      Corentin Chary <corentin.chary@gmail.com>
+# (c) 2016      Roger Shimizu <rogershimizu@gmail.com>
 #
 # Licensed under the GNU General Public License, version 2 (GPLv2).
 # ----------------------------------------------------------------------
@@ -15,7 +16,14 @@ check_vmlinux()
 	# Use readelf to check if it's a valid ELF
 	# TODO: find a better to way to check that it's really vmlinux
 	#       and not just an elf
-	readelf -h $1 > /dev/null 2>&1 || return 1
+	case "$2" in
+	0|"")
+		readelf -h $1 > /dev/null 2>&1 || return 1
+		;;
+	1)
+	# For ARCH like armel/armhf, vmlinux is not ELF, so we skip the check
+		;;
+	esac
 
 	cat $1
 	exit 0
@@ -31,7 +39,7 @@ try_decompress()
 	do
 		pos=${pos%%:*}
 		tail -c+$pos "$img" | $3 > $tmp 2> /dev/null
-		check_vmlinux $tmp
+		test $? -eq 0 && check_vmlinux $tmp 1
 	done
 }
 
@@ -53,7 +61,7 @@ check_vmlinux $img
 
 # That didn't work, so retry after decompression.
 try_decompress '\037\213\010' xy    gunzip
-try_decompress '\3757zXZ\000' abcde unxz
+try_decompress '\3757zXZ\000' abcde "unxz --single-stream"
 try_decompress 'BZh'          xy    bunzip2
 try_decompress '\135\0\0\0'   xxx   unlzma
 try_decompress '\211\114\132' xy    'lzop -d'
-- 
2.11.0

Re: [PATCH] scripts: make extract-vmlinux support armel/armhf

[Russell King - ARM Linux]

On Fri, Sep 01, 2017 at 12:36:31AM +0900, Roger Shimizu wrote:
> vmlinux/zImage on armel/armhf seems not an ELF, so update the script
> scripts/extract-vmlinux to support such case.
> 
> This fix is tested on Debian amd64, armel, and armhf platform, with
> Debian kernels.
> 
> Fixes: 09d481270d44 ("scripts: add extract-vmlinux")
> Cc: Corentin Chary <corentincj@iksaif.net>
> Cc: Michal Marek <mmarek@suse.com>
> Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
> Cc: Russell King <linux@arm.linux.org.uk>
> Cc: linux-arm-kernel@lists.infradead.org
> Cc: linux-kbuild@vger.kernel.org
> Signed-off-by: Roger Shimizu <rogershimizu@gmail.com>
> ---
> 
> Dear Michal, Masahiro-san, and Russell,
> 
> I posted this patch before [0] but didn't get positive response.
> Recently when I trace a Debian kernel issue, I still find this patch
> useful to locate the root cause [1].
> 
> Besides, it's a well known bug confuses other developers [2][3].
> So I decide to submit the patch again.
> 
> [0]: https://patchwork.kernel.org/patch/8120831/
> [1]: https://bugs.debian.org/870185#50
> [2]: https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
> [3]: https://bugs.linaro.org/show_bug.cgi?id=461
> 
> Please kindly help to review. Thank you!

I don't think this makes sense.  This script claims to extract a "vmlinux"
from a compressed kernel image.  "vmlinux" is normally the term of the
ELF object, and indeed as the script currently stands, it always guarantees
to output an ELF file.

However, ARM zImage does not store a compressed ELF object, it stores a
compressed binary image of the executable in memory, so what you get out
of this script is not an ELF file, but the binary image (iow, what
arch/arm/boot/Image is.)

What's the use case - what are you using the output of this script with?

> diff --git a/scripts/extract-vmlinux b/scripts/extract-vmlinux
> index 5061abcc2540..0c72ecd24969 100755
> --- a/scripts/extract-vmlinux
> +++ b/scripts/extract-vmlinux
> @@ -6,6 +6,7 @@
>  # (c) 2009,2010 Dick Streefland <dick@streefland.net>
>  #
>  # (c) 2011      Corentin Chary <corentin.chary@gmail.com>
> +# (c) 2016      Roger Shimizu <rogershimizu@gmail.com>
>  #
>  # Licensed under the GNU General Public License, version 2 (GPLv2).
>  # ----------------------------------------------------------------------
> @@ -15,7 +16,14 @@ check_vmlinux()
>  	# Use readelf to check if it's a valid ELF
>  	# TODO: find a better to way to check that it's really vmlinux
>  	#       and not just an elf
> -	readelf -h $1 > /dev/null 2>&1 || return 1
> +	case "$2" in
> +	0|"")
> +		readelf -h $1 > /dev/null 2>&1 || return 1
> +		;;
> +	1)
> +	# For ARCH like armel/armhf, vmlinux is not ELF, so we skip the check
> +		;;
> +	esac

Right, so passing the second argument as 1 bypasses the "is it ELF" check.

> @@ -31,7 +39,7 @@ try_decompress()
>  	do
>  		pos=${pos%%:*}
>  		tail -c+$pos "$img" | $3 > $tmp 2> /dev/null
> -		check_vmlinux $tmp
> +		test $? -eq 0 && check_vmlinux $tmp 1

and here you always pass '1', thereby bypassing the ELF check for every
architecture.  What if some architecture stores another compressed
object in the same compressed image?  I suspect the check is there to
ensure that it works on architectures where the compressed kernel image
contains other compressed objects.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up
According to speedtest.net: 8.21Mbps down 510kbps up

[PATCH] scripts: make extract-vmlinux support armel/armhf

[Russell King - ARM Linux]

On Fri, Sep 01, 2017 at 12:36:31AM +0900, Roger Shimizu wrote:
> vmlinux/zImage on armel/armhf seems not an ELF, so update the script
> scripts/extract-vmlinux to support such case.
> 
> This fix is tested on Debian amd64, armel, and armhf platform, with
> Debian kernels.
> 
> Fixes: 09d481270d44 ("scripts: add extract-vmlinux")
> Cc: Corentin Chary <corentincj@iksaif.net>
> Cc: Michal Marek <mmarek@suse.com>
> Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
> Cc: Russell King <linux@arm.linux.org.uk>
> Cc: linux-arm-kernel at lists.infradead.org
> Cc: linux-kbuild at vger.kernel.org
> Signed-off-by: Roger Shimizu <rogershimizu@gmail.com>
> ---
> 
> Dear Michal, Masahiro-san, and Russell,
> 
> I posted this patch before [0] but didn't get positive response.
> Recently when I trace a Debian kernel issue, I still find this patch
> useful to locate the root cause [1].
> 
> Besides, it's a well known bug confuses other developers [2][3].
> So I decide to submit the patch again.
> 
> [0]: https://patchwork.kernel.org/patch/8120831/
> [1]: https://bugs.debian.org/870185#50
> [2]: https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
> [3]: https://bugs.linaro.org/show_bug.cgi?id=461
> 
> Please kindly help to review. Thank you!

I don't think this makes sense.  This script claims to extract a "vmlinux"
from a compressed kernel image.  "vmlinux" is normally the term of the
ELF object, and indeed as the script currently stands, it always guarantees
to output an ELF file.

However, ARM zImage does not store a compressed ELF object, it stores a
compressed binary image of the executable in memory, so what you get out
of this script is not an ELF file, but the binary image (iow, what
arch/arm/boot/Image is.)

What's the use case - what are you using the output of this script with?

> diff --git a/scripts/extract-vmlinux b/scripts/extract-vmlinux
> index 5061abcc2540..0c72ecd24969 100755
> --- a/scripts/extract-vmlinux
> +++ b/scripts/extract-vmlinux
> @@ -6,6 +6,7 @@
>  # (c) 2009,2010 Dick Streefland <dick@streefland.net>
>  #
>  # (c) 2011      Corentin Chary <corentin.chary@gmail.com>
> +# (c) 2016      Roger Shimizu <rogershimizu@gmail.com>
>  #
>  # Licensed under the GNU General Public License, version 2 (GPLv2).
>  # ----------------------------------------------------------------------
> @@ -15,7 +16,14 @@ check_vmlinux()
>  	# Use readelf to check if it's a valid ELF
>  	# TODO: find a better to way to check that it's really vmlinux
>  	#       and not just an elf
> -	readelf -h $1 > /dev/null 2>&1 || return 1
> +	case "$2" in
> +	0|"")
> +		readelf -h $1 > /dev/null 2>&1 || return 1
> +		;;
> +	1)
> +	# For ARCH like armel/armhf, vmlinux is not ELF, so we skip the check
> +		;;
> +	esac

Right, so passing the second argument as 1 bypasses the "is it ELF" check.

> @@ -31,7 +39,7 @@ try_decompress()
>  	do
>  		pos=${pos%%:*}
>  		tail -c+$pos "$img" | $3 > $tmp 2> /dev/null
> -		check_vmlinux $tmp
> +		test $? -eq 0 && check_vmlinux $tmp 1

and here you always pass '1', thereby bypassing the ELF check for every
architecture.  What if some architecture stores another compressed
object in the same compressed image?  I suspect the check is there to
ensure that it works on architectures where the compressed kernel image
contains other compressed objects.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up
According to speedtest.net: 8.21Mbps down 510kbps up

Re: [PATCH] scripts: make extract-vmlinux support armel/armhf

[Roger Shimizu]

Dear Russell,

Thanks for your review!

On Fri, Sep 1, 2017 at 12:49 AM, Russell King - ARM Linux
<linux@armlinux.org.uk> wrote:
> On Fri, Sep 01, 2017 at 12:36:31AM +0900, Roger Shimizu wrote:
>> vmlinux/zImage on armel/armhf seems not an ELF, so update the script
>> scripts/extract-vmlinux to support such case.
>>
>> This fix is tested on Debian amd64, armel, and armhf platform, with
>> Debian kernels.
>>
>> Fixes: 09d481270d44 ("scripts: add extract-vmlinux")
>> Cc: Corentin Chary <corentincj@iksaif.net>
>> Cc: Michal Marek <mmarek@suse.com>
>> Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
>> Cc: Russell King <linux@arm.linux.org.uk>
>> Cc: linux-arm-kernel@lists.infradead.org
>> Cc: linux-kbuild@vger.kernel.org
>> Signed-off-by: Roger Shimizu <rogershimizu@gmail.com>
>> ---
>>
>> Dear Michal, Masahiro-san, and Russell,
>>
>> I posted this patch before [0] but didn't get positive response.
>> Recently when I trace a Debian kernel issue, I still find this patch
>> useful to locate the root cause [1].
>>
>> Besides, it's a well known bug confuses other developers [2][3].
>> So I decide to submit the patch again.
>>
>> [0]: https://patchwork.kernel.org/patch/8120831/
>> [1]: https://bugs.debian.org/870185#50
>> [2]: https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
>> [3]: https://bugs.linaro.org/show_bug.cgi?id=461
>>
>> Please kindly help to review. Thank you!
>
> I don't think this makes sense.  This script claims to extract a "vmlinux"
> from a compressed kernel image.  "vmlinux" is normally the term of the
> ELF object, and indeed as the script currently stands, it always guarantees
> to output an ELF file.

If you think creating a new "scripts/extract-zImage" file is more
proper way, we can do it.

> However, ARM zImage does not store a compressed ELF object, it stores a
> compressed binary image of the executable in memory, so what you get out
> of this script is not an ELF file, but the binary image (iow, what
> arch/arm/boot/Image is.)
>
> What's the use case - what are you using the output of this script with?

I already provided a use case, to trace a Debian kernel issue caused
by kernel size.
- https://bugs.debian.org/870185#50

I also see other people complaining this issue:
 - https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
 - https://bugs.linaro.org/show_bug.cgi?id=461

By a random web search, I find another blog post on how to dissamble the kernel.
 - https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/


>
>> diff --git a/scripts/extract-vmlinux b/scripts/extract-vmlinux
>> index 5061abcc2540..0c72ecd24969 100755
>> --- a/scripts/extract-vmlinux
>> +++ b/scripts/extract-vmlinux
>> @@ -6,6 +6,7 @@
>>  # (c) 2009,2010 Dick Streefland <dick@streefland.net>
>>  #
>>  # (c) 2011      Corentin Chary <corentin.chary@gmail.com>
>> +# (c) 2016      Roger Shimizu <rogershimizu@gmail.com>
>>  #
>>  # Licensed under the GNU General Public License, version 2 (GPLv2).
>>  # ----------------------------------------------------------------------
>> @@ -15,7 +16,14 @@ check_vmlinux()
>>       # Use readelf to check if it's a valid ELF
>>       # TODO: find a better to way to check that it's really vmlinux
>>       #       and not just an elf
>> -     readelf -h $1 > /dev/null 2>&1 || return 1
>> +     case "$2" in
>> +     0|"")
>> +             readelf -h $1 > /dev/null 2>&1 || return 1
>> +             ;;
>> +     1)
>> +     # For ARCH like armel/armhf, vmlinux is not ELF, so we skip the check
>> +             ;;
>> +     esac
>
> Right, so passing the second argument as 1 bypasses the "is it ELF" check.
>
>> @@ -31,7 +39,7 @@ try_decompress()
>>       do
>>               pos=${pos%%:*}
>>               tail -c+$pos "$img" | $3 > $tmp 2> /dev/null
>> -             check_vmlinux $tmp
>> +             test $? -eq 0 && check_vmlinux $tmp 1
>
> and here you always pass '1', thereby bypassing the ELF check for every

There's another call without parameter "1", which handles the ELF's case.

> architecture.  What if some architecture stores another compressed
> object in the same compressed image?  I suspect the check is there to
> ensure that it works on architectures where the compressed kernel image
> contains other compressed objects.

The output of this script is output the first compressed object.
If there's need to process the 2nd, it should be handled separately.
I don't think it's a blocker to this patch.

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

[PATCH] scripts: make extract-vmlinux support armel/armhf

[Roger Shimizu]

Dear Russell,

Thanks for your review!

On Fri, Sep 1, 2017 at 12:49 AM, Russell King - ARM Linux
<linux@armlinux.org.uk> wrote:
> On Fri, Sep 01, 2017 at 12:36:31AM +0900, Roger Shimizu wrote:
>> vmlinux/zImage on armel/armhf seems not an ELF, so update the script
>> scripts/extract-vmlinux to support such case.
>>
>> This fix is tested on Debian amd64, armel, and armhf platform, with
>> Debian kernels.
>>
>> Fixes: 09d481270d44 ("scripts: add extract-vmlinux")
>> Cc: Corentin Chary <corentincj@iksaif.net>
>> Cc: Michal Marek <mmarek@suse.com>
>> Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
>> Cc: Russell King <linux@arm.linux.org.uk>
>> Cc: linux-arm-kernel at lists.infradead.org
>> Cc: linux-kbuild at vger.kernel.org
>> Signed-off-by: Roger Shimizu <rogershimizu@gmail.com>
>> ---
>>
>> Dear Michal, Masahiro-san, and Russell,
>>
>> I posted this patch before [0] but didn't get positive response.
>> Recently when I trace a Debian kernel issue, I still find this patch
>> useful to locate the root cause [1].
>>
>> Besides, it's a well known bug confuses other developers [2][3].
>> So I decide to submit the patch again.
>>
>> [0]: https://patchwork.kernel.org/patch/8120831/
>> [1]: https://bugs.debian.org/870185#50
>> [2]: https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
>> [3]: https://bugs.linaro.org/show_bug.cgi?id=461
>>
>> Please kindly help to review. Thank you!
>
> I don't think this makes sense.  This script claims to extract a "vmlinux"
> from a compressed kernel image.  "vmlinux" is normally the term of the
> ELF object, and indeed as the script currently stands, it always guarantees
> to output an ELF file.

If you think creating a new "scripts/extract-zImage" file is more
proper way, we can do it.

> However, ARM zImage does not store a compressed ELF object, it stores a
> compressed binary image of the executable in memory, so what you get out
> of this script is not an ELF file, but the binary image (iow, what
> arch/arm/boot/Image is.)
>
> What's the use case - what are you using the output of this script with?

I already provided a use case, to trace a Debian kernel issue caused
by kernel size.
- https://bugs.debian.org/870185#50

I also see other people complaining this issue:
 - https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
 - https://bugs.linaro.org/show_bug.cgi?id=461

By a random web search, I find another blog post on how to dissamble the kernel.
 - https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/


>
>> diff --git a/scripts/extract-vmlinux b/scripts/extract-vmlinux
>> index 5061abcc2540..0c72ecd24969 100755
>> --- a/scripts/extract-vmlinux
>> +++ b/scripts/extract-vmlinux
>> @@ -6,6 +6,7 @@
>>  # (c) 2009,2010 Dick Streefland <dick@streefland.net>
>>  #
>>  # (c) 2011      Corentin Chary <corentin.chary@gmail.com>
>> +# (c) 2016      Roger Shimizu <rogershimizu@gmail.com>
>>  #
>>  # Licensed under the GNU General Public License, version 2 (GPLv2).
>>  # ----------------------------------------------------------------------
>> @@ -15,7 +16,14 @@ check_vmlinux()
>>       # Use readelf to check if it's a valid ELF
>>       # TODO: find a better to way to check that it's really vmlinux
>>       #       and not just an elf
>> -     readelf -h $1 > /dev/null 2>&1 || return 1
>> +     case "$2" in
>> +     0|"")
>> +             readelf -h $1 > /dev/null 2>&1 || return 1
>> +             ;;
>> +     1)
>> +     # For ARCH like armel/armhf, vmlinux is not ELF, so we skip the check
>> +             ;;
>> +     esac
>
> Right, so passing the second argument as 1 bypasses the "is it ELF" check.
>
>> @@ -31,7 +39,7 @@ try_decompress()
>>       do
>>               pos=${pos%%:*}
>>               tail -c+$pos "$img" | $3 > $tmp 2> /dev/null
>> -             check_vmlinux $tmp
>> +             test $? -eq 0 && check_vmlinux $tmp 1
>
> and here you always pass '1', thereby bypassing the ELF check for every

There's another call without parameter "1", which handles the ELF's case.

> architecture.  What if some architecture stores another compressed
> object in the same compressed image?  I suspect the check is there to
> ensure that it works on architectures where the compressed kernel image
> contains other compressed objects.

The output of this script is output the first compressed object.
If there's need to process the 2nd, it should be handled separately.
I don't think it's a blocker to this patch.

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

Re: [PATCH] scripts: make extract-vmlinux support armel/armhf

[Russell King - ARM Linux]

On Fri, Sep 01, 2017 at 01:05:16AM +0900, Roger Shimizu wrote:
> Dear Russell,
> 
> Thanks for your review!
> 
> On Fri, Sep 1, 2017 at 12:49 AM, Russell King - ARM Linux
> <linux@armlinux.org.uk> wrote:
> > On Fri, Sep 01, 2017 at 12:36:31AM +0900, Roger Shimizu wrote:
> >> vmlinux/zImage on armel/armhf seems not an ELF, so update the script
> >> scripts/extract-vmlinux to support such case.
> >>
> >> This fix is tested on Debian amd64, armel, and armhf platform, with
> >> Debian kernels.
> >>
> >> Fixes: 09d481270d44 ("scripts: add extract-vmlinux")
> >> Cc: Corentin Chary <corentincj@iksaif.net>
> >> Cc: Michal Marek <mmarek@suse.com>
> >> Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
> >> Cc: Russell King <linux@arm.linux.org.uk>
> >> Cc: linux-arm-kernel@lists.infradead.org
> >> Cc: linux-kbuild@vger.kernel.org
> >> Signed-off-by: Roger Shimizu <rogershimizu@gmail.com>
> >> ---
> >>
> >> Dear Michal, Masahiro-san, and Russell,
> >>
> >> I posted this patch before [0] but didn't get positive response.
> >> Recently when I trace a Debian kernel issue, I still find this patch
> >> useful to locate the root cause [1].
> >>
> >> Besides, it's a well known bug confuses other developers [2][3].
> >> So I decide to submit the patch again.
> >>
> >> [0]: https://patchwork.kernel.org/patch/8120831/
> >> [1]: https://bugs.debian.org/870185#50
> >> [2]: https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
> >> [3]: https://bugs.linaro.org/show_bug.cgi?id=461
> >>
> >> Please kindly help to review. Thank you!
> >
> > I don't think this makes sense.  This script claims to extract a "vmlinux"
> > from a compressed kernel image.  "vmlinux" is normally the term of the
> > ELF object, and indeed as the script currently stands, it always guarantees
> > to output an ELF file.
> 
> If you think creating a new "scripts/extract-zImage" file is more
> proper way, we can do it.
> 
> > However, ARM zImage does not store a compressed ELF object, it stores a
> > compressed binary image of the executable in memory, so what you get out
> > of this script is not an ELF file, but the binary image (iow, what
> > arch/arm/boot/Image is.)
> >
> > What's the use case - what are you using the output of this script with?
> 
> I already provided a use case, to trace a Debian kernel issue caused
> by kernel size.
> - https://bugs.debian.org/870185#50

Sorry, I haven't time to look - it needs me to use firefox on a
different machine.  Maybe you could help by providing some details
by email, otherwise I've got extra work to do at some point in the
future (probably days away) if I remember (your mail will get buried).
Much of my time is in front of a _textual_ _only_ interface, and elinks
does not work with several modern SSL-only sites.

> I also see other people complaining this issue:
>  - https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
>  - https://bugs.linaro.org/show_bug.cgi?id=461
> 
> By a random web search, I find another blog post on how to dissamble the kernel.
>  - https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/

Yea, all SSL sites, I'm not going to try elinks with them, I don't
have the time to mess around.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up
According to speedtest.net: 8.21Mbps down 510kbps up

[PATCH] scripts: make extract-vmlinux support armel/armhf

[Russell King - ARM Linux]

On Fri, Sep 01, 2017 at 01:05:16AM +0900, Roger Shimizu wrote:
> Dear Russell,
> 
> Thanks for your review!
> 
> On Fri, Sep 1, 2017 at 12:49 AM, Russell King - ARM Linux
> <linux@armlinux.org.uk> wrote:
> > On Fri, Sep 01, 2017 at 12:36:31AM +0900, Roger Shimizu wrote:
> >> vmlinux/zImage on armel/armhf seems not an ELF, so update the script
> >> scripts/extract-vmlinux to support such case.
> >>
> >> This fix is tested on Debian amd64, armel, and armhf platform, with
> >> Debian kernels.
> >>
> >> Fixes: 09d481270d44 ("scripts: add extract-vmlinux")
> >> Cc: Corentin Chary <corentincj@iksaif.net>
> >> Cc: Michal Marek <mmarek@suse.com>
> >> Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
> >> Cc: Russell King <linux@arm.linux.org.uk>
> >> Cc: linux-arm-kernel at lists.infradead.org
> >> Cc: linux-kbuild at vger.kernel.org
> >> Signed-off-by: Roger Shimizu <rogershimizu@gmail.com>
> >> ---
> >>
> >> Dear Michal, Masahiro-san, and Russell,
> >>
> >> I posted this patch before [0] but didn't get positive response.
> >> Recently when I trace a Debian kernel issue, I still find this patch
> >> useful to locate the root cause [1].
> >>
> >> Besides, it's a well known bug confuses other developers [2][3].
> >> So I decide to submit the patch again.
> >>
> >> [0]: https://patchwork.kernel.org/patch/8120831/
> >> [1]: https://bugs.debian.org/870185#50
> >> [2]: https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
> >> [3]: https://bugs.linaro.org/show_bug.cgi?id=461
> >>
> >> Please kindly help to review. Thank you!
> >
> > I don't think this makes sense.  This script claims to extract a "vmlinux"
> > from a compressed kernel image.  "vmlinux" is normally the term of the
> > ELF object, and indeed as the script currently stands, it always guarantees
> > to output an ELF file.
> 
> If you think creating a new "scripts/extract-zImage" file is more
> proper way, we can do it.
> 
> > However, ARM zImage does not store a compressed ELF object, it stores a
> > compressed binary image of the executable in memory, so what you get out
> > of this script is not an ELF file, but the binary image (iow, what
> > arch/arm/boot/Image is.)
> >
> > What's the use case - what are you using the output of this script with?
> 
> I already provided a use case, to trace a Debian kernel issue caused
> by kernel size.
> - https://bugs.debian.org/870185#50

Sorry, I haven't time to look - it needs me to use firefox on a
different machine.  Maybe you could help by providing some details
by email, otherwise I've got extra work to do at some point in the
future (probably days away) if I remember (your mail will get buried).
Much of my time is in front of a _textual_ _only_ interface, and elinks
does not work with several modern SSL-only sites.

> I also see other people complaining this issue:
>  - https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
>  - https://bugs.linaro.org/show_bug.cgi?id=461
> 
> By a random web search, I find another blog post on how to dissamble the kernel.
>  - https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/

Yea, all SSL sites, I'm not going to try elinks with them, I don't
have the time to mess around.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up
According to speedtest.net: 8.21Mbps down 510kbps up

Re: [PATCH] scripts: make extract-vmlinux support armel/armhf

[Tony Lindgren]

* Russell King - ARM Linux <linux@armlinux.org.uk> [170831 09:21]:
> On Fri, Sep 01, 2017 at 01:05:16AM +0900, Roger Shimizu wrote:
> > On Fri, Sep 1, 2017 at 12:49 AM, Russell King - ARM Linux
> > <linux@armlinux.org.uk> wrote:
> > > What's the use case - what are you using the output of this script with?
> > 
> > I already provided a use case, to trace a Debian kernel issue caused
> > by kernel size.
> > - https://bugs.debian.org/870185#50
> 
> Sorry, I haven't time to look - it needs me to use firefox on a
> different machine.  Maybe you could help by providing some details
> by email, otherwise I've got extra work to do at some point in the
> future (probably days away) if I remember (your mail will get buried).
> Much of my time is in front of a _textual_ _only_ interface, and elinks
> does not work with several modern SSL-only sites.
> 
> > I also see other people complaining this issue:
> >  - https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
> >  - https://bugs.linaro.org/show_bug.cgi?id=461
> > 
> > By a random web search, I find another blog post on how to dissamble the kernel.
> >  - https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/
> 
> Yea, all SSL sites, I'm not going to try elinks with them, I don't
> have the time to mess around.

I think the use case is to get the booted kernel size from zImage
to avoid overwriting dts or initramfs. Don't we already have that
at the end of zImage somewhere for kexec?

Regards,

Tony

[PATCH] scripts: make extract-vmlinux support armel/armhf

[Tony Lindgren]

* Russell King - ARM Linux <linux@armlinux.org.uk> [170831 09:21]:
> On Fri, Sep 01, 2017 at 01:05:16AM +0900, Roger Shimizu wrote:
> > On Fri, Sep 1, 2017 at 12:49 AM, Russell King - ARM Linux
> > <linux@armlinux.org.uk> wrote:
> > > What's the use case - what are you using the output of this script with?
> > 
> > I already provided a use case, to trace a Debian kernel issue caused
> > by kernel size.
> > - https://bugs.debian.org/870185#50
> 
> Sorry, I haven't time to look - it needs me to use firefox on a
> different machine.  Maybe you could help by providing some details
> by email, otherwise I've got extra work to do at some point in the
> future (probably days away) if I remember (your mail will get buried).
> Much of my time is in front of a _textual_ _only_ interface, and elinks
> does not work with several modern SSL-only sites.
> 
> > I also see other people complaining this issue:
> >  - https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
> >  - https://bugs.linaro.org/show_bug.cgi?id=461
> > 
> > By a random web search, I find another blog post on how to dissamble the kernel.
> >  - https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/
> 
> Yea, all SSL sites, I'm not going to try elinks with them, I don't
> have the time to mess around.

I think the use case is to get the booted kernel size from zImage
to avoid overwriting dts or initramfs. Don't we already have that
at the end of zImage somewhere for kexec?

Regards,

Tony

Re: [PATCH] scripts: make extract-vmlinux support armel/armhf

[Russell King - ARM Linux]

On Thu, Aug 31, 2017 at 10:43:19AM -0700, Tony Lindgren wrote:
> * Russell King - ARM Linux <linux@armlinux.org.uk> [170831 09:21]:
> > On Fri, Sep 01, 2017 at 01:05:16AM +0900, Roger Shimizu wrote:
> > > On Fri, Sep 1, 2017 at 12:49 AM, Russell King - ARM Linux
> > > <linux@armlinux.org.uk> wrote:
> > > > What's the use case - what are you using the output of this script with?
> > > 
> > > I already provided a use case, to trace a Debian kernel issue caused
> > > by kernel size.
> > > - https://bugs.debian.org/870185#50
> > 
> > Sorry, I haven't time to look - it needs me to use firefox on a
> > different machine.  Maybe you could help by providing some details
> > by email, otherwise I've got extra work to do at some point in the
> > future (probably days away) if I remember (your mail will get buried).
> > Much of my time is in front of a _textual_ _only_ interface, and elinks
> > does not work with several modern SSL-only sites.
> > 
> > > I also see other people complaining this issue:
> > >  - https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
> > >  - https://bugs.linaro.org/show_bug.cgi?id=461
> > > 
> > > By a random web search, I find another blog post on how to dissamble the kernel.
> > >  - https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/
> > 
> > Yea, all SSL sites, I'm not going to try elinks with them, I don't
> > have the time to mess around.
> 
> I think the use case is to get the booted kernel size from zImage
> to avoid overwriting dts or initramfs. Don't we already have that
> at the end of zImage somewhere for kexec?

We do, but finding where that is is difficult if a DTB has been appended
as it's right at the end of the compressed data.  kexec doesn't use it,
it just assumes there's a 5x expansion of the kernel compressed image.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up
According to speedtest.net: 8.21Mbps down 510kbps up

[PATCH] scripts: make extract-vmlinux support armel/armhf

[Russell King - ARM Linux]

On Thu, Aug 31, 2017 at 10:43:19AM -0700, Tony Lindgren wrote:
> * Russell King - ARM Linux <linux@armlinux.org.uk> [170831 09:21]:
> > On Fri, Sep 01, 2017 at 01:05:16AM +0900, Roger Shimizu wrote:
> > > On Fri, Sep 1, 2017 at 12:49 AM, Russell King - ARM Linux
> > > <linux@armlinux.org.uk> wrote:
> > > > What's the use case - what are you using the output of this script with?
> > > 
> > > I already provided a use case, to trace a Debian kernel issue caused
> > > by kernel size.
> > > - https://bugs.debian.org/870185#50
> > 
> > Sorry, I haven't time to look - it needs me to use firefox on a
> > different machine.  Maybe you could help by providing some details
> > by email, otherwise I've got extra work to do at some point in the
> > future (probably days away) if I remember (your mail will get buried).
> > Much of my time is in front of a _textual_ _only_ interface, and elinks
> > does not work with several modern SSL-only sites.
> > 
> > > I also see other people complaining this issue:
> > >  - https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
> > >  - https://bugs.linaro.org/show_bug.cgi?id=461
> > > 
> > > By a random web search, I find another blog post on how to dissamble the kernel.
> > >  - https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/
> > 
> > Yea, all SSL sites, I'm not going to try elinks with them, I don't
> > have the time to mess around.
> 
> I think the use case is to get the booted kernel size from zImage
> to avoid overwriting dts or initramfs. Don't we already have that
> at the end of zImage somewhere for kexec?

We do, but finding where that is is difficult if a DTB has been appended
as it's right at the end of the compressed data.  kexec doesn't use it,
it just assumes there's a 5x expansion of the kernel compressed image.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up
According to speedtest.net: 8.21Mbps down 510kbps up

Re: [PATCH] scripts: make extract-vmlinux support armel/armhf

[Roger Shimizu]

On Sat, Sep 2, 2017 at 8:16 AM, Russell King - ARM Linux
<linux@armlinux.org.uk> wrote:
> On Thu, Aug 31, 2017 at 10:43:19AM -0700, Tony Lindgren wrote:
>> * Russell King - ARM Linux <linux@armlinux.org.uk> [170831 09:21]:
>> > On Fri, Sep 01, 2017 at 01:05:16AM +0900, Roger Shimizu wrote:
>> > > On Fri, Sep 1, 2017 at 12:49 AM, Russell King - ARM Linux
>> > > <linux@armlinux.org.uk> wrote:
>> > > > What's the use case - what are you using the output of this script with?
>> > >
>> > > I already provided a use case, to trace a Debian kernel issue caused
>> > > by kernel size.
>> > > - https://bugs.debian.org/870185#50
>> >
>> > Sorry, I haven't time to look - it needs me to use firefox on a
>> > different machine.  Maybe you could help by providing some details
>> > by email, otherwise I've got extra work to do at some point in the
>> > future (probably days away) if I remember (your mail will get buried).
>> > Much of my time is in front of a _textual_ _only_ interface, and elinks
>> > does not work with several modern SSL-only sites.
>> >
>> > > I also see other people complaining this issue:
>> > >  - https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
>> > >  - https://bugs.linaro.org/show_bug.cgi?id=461
>> > >
>> > > By a random web search, I find another blog post on how to dissamble the kernel.
>> > >  - https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/
>> >
>> > Yea, all SSL sites, I'm not going to try elinks with them, I don't
>> > have the time to mess around.
>>
>> I think the use case is to get the booted kernel size from zImage
>> to avoid overwriting dts or initramfs. Don't we already have that
>> at the end of zImage somewhere for kexec?
>
> We do, but finding where that is is difficult if a DTB has been appended
> as it's right at the end of the compressed data.  kexec doesn't use it,
> it just assumes there's a 5x expansion of the kernel compressed image.

My patch already take the appended DTB blob part into consideration.
It uses "unxz --single-stream" instead of "unxz" to extract, so unxz
just ignore the DTB.

I confirmed it works well on my armel box, which need to append a
device specific DTB.

For the use case, I already provided several bug reports and blog posts.
I think current script is broken for armel/armhf, which can be fixed
by my patch.
So please kindly consider this patch. Thank you!

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

[PATCH] scripts: make extract-vmlinux support armel/armhf

[Roger Shimizu]

On Sat, Sep 2, 2017 at 8:16 AM, Russell King - ARM Linux
<linux@armlinux.org.uk> wrote:
> On Thu, Aug 31, 2017 at 10:43:19AM -0700, Tony Lindgren wrote:
>> * Russell King - ARM Linux <linux@armlinux.org.uk> [170831 09:21]:
>> > On Fri, Sep 01, 2017 at 01:05:16AM +0900, Roger Shimizu wrote:
>> > > On Fri, Sep 1, 2017 at 12:49 AM, Russell King - ARM Linux
>> > > <linux@armlinux.org.uk> wrote:
>> > > > What's the use case - what are you using the output of this script with?
>> > >
>> > > I already provided a use case, to trace a Debian kernel issue caused
>> > > by kernel size.
>> > > - https://bugs.debian.org/870185#50
>> >
>> > Sorry, I haven't time to look - it needs me to use firefox on a
>> > different machine.  Maybe you could help by providing some details
>> > by email, otherwise I've got extra work to do at some point in the
>> > future (probably days away) if I remember (your mail will get buried).
>> > Much of my time is in front of a _textual_ _only_ interface, and elinks
>> > does not work with several modern SSL-only sites.
>> >
>> > > I also see other people complaining this issue:
>> > >  - https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
>> > >  - https://bugs.linaro.org/show_bug.cgi?id=461
>> > >
>> > > By a random web search, I find another blog post on how to dissamble the kernel.
>> > >  - https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/
>> >
>> > Yea, all SSL sites, I'm not going to try elinks with them, I don't
>> > have the time to mess around.
>>
>> I think the use case is to get the booted kernel size from zImage
>> to avoid overwriting dts or initramfs. Don't we already have that
>> at the end of zImage somewhere for kexec?
>
> We do, but finding where that is is difficult if a DTB has been appended
> as it's right at the end of the compressed data.  kexec doesn't use it,
> it just assumes there's a 5x expansion of the kernel compressed image.

My patch already take the appended DTB blob part into consideration.
It uses "unxz --single-stream" instead of "unxz" to extract, so unxz
just ignore the DTB.

I confirmed it works well on my armel box, which need to append a
device specific DTB.

For the use case, I already provided several bug reports and blog posts.
I think current script is broken for armel/armhf, which can be fixed
by my patch.
So please kindly consider this patch. Thank you!

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

Re: [PATCH] scripts: make extract-vmlinux support armel/armhf

[Russell King - ARM Linux]

On Sat, Sep 09, 2017 at 04:33:55PM +0900, Roger Shimizu wrote:
> On Sat, Sep 2, 2017 at 8:16 AM, Russell King - ARM Linux
> <linux@armlinux.org.uk> wrote:
> > On Thu, Aug 31, 2017 at 10:43:19AM -0700, Tony Lindgren wrote:
> >> I think the use case is to get the booted kernel size from zImage
> >> to avoid overwriting dts or initramfs. Don't we already have that
> >> at the end of zImage somewhere for kexec?
> >
> > We do, but finding where that is is difficult if a DTB has been appended
> > as it's right at the end of the compressed data.  kexec doesn't use it,
> > it just assumes there's a 5x expansion of the kernel compressed image.
> 
> My patch already take the appended DTB blob part into consideration.
> It uses "unxz --single-stream" instead of "unxz" to extract, so unxz
> just ignore the DTB.

Sorry, I wasn't talking about the patch here.

The whole point of _this_ method of replying, where the relevant context
of the message being replied to is left and the reply is placed directly
beneath the paragraph to which the reply is relevant is to make clear
_what_ is being replied to.

My comments above are directly beneath a comment from Tony Lindgren asking
about the size field, which is appended after the compressed kernel image
data.  My reply was to Tony on that point alone, and why we aren't using
that for kexec-tools.

kexec-tools used to assume too small a size for the expansion of the
kernel image, and that has been changed.

I'm afraid that I still haven't got around to looking at _any_ of the
URLs you've provided so I'm still in the dark about (a) why this is
needed and (b) why it's acceptable for this script to output a binary
blob for ARM rather than an ELF file that it guarantees today.  It's
annoying that elinks' SSL implementation is buggy and causes connections
to certain SSL sites to fail, but alas that's the inconvenience that
buggy software causes.

Until I'm able to see some reasoning, I can't begin to consider this
patch.

Moreover, I'm not convinced by any of your arguments about why it's
acceptable to defeat the check for the ELF object _for all users_ of
this script.  What if some other architecture has multiple compressed
objects in their image and the "vmlinux" is not the first.  Your
argument seems to be "they should use a different script", yet you
are the one changing the script from outputting the first compressed
ELF object to merely producing the first compressed image.  "vmlinux"
is the name of the ELF kernel image, and the script is called
"extract-vmlinux" so this is obviously the right script which should
be extracting the correct compressed ELF object.

Hence, I don't buy your reasoning that in such a situation, another
architecture's use of this script should use some other script because
your changes fix it for ARM but cause a regression elsewhere.

The way we work on the kernel, if a change causes a user visible
regression, then the change was wrong and an alternative solution
needs to be sought.

I percieve the risk of this patch causing a regression on other
architectures to be high because it gratuitously changes functionality
of the script which didn't have to be there if all we cared about was
extracting the first image.  Hence, it must have been added because
there is a need for it.

So far, nothing has been said that lowers that risk.

It's not up to me to search around to try and evaluate risk - it's
my responsibility to ask the question (I already did) and it's the
submitter's responsibility to show that it's likely safe.

Thus far, the reasoning seems to be purely "several people are having
a problem with this script on ARM, here's a patch that fixes it for
ARM but might break existing users, but we don't care about existing
users."  That makes me even more wary.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up
According to speedtest.net: 8.21Mbps down 510kbps up

[PATCH] scripts: make extract-vmlinux support armel/armhf

[Russell King - ARM Linux]

On Sat, Sep 09, 2017 at 04:33:55PM +0900, Roger Shimizu wrote:
> On Sat, Sep 2, 2017 at 8:16 AM, Russell King - ARM Linux
> <linux@armlinux.org.uk> wrote:
> > On Thu, Aug 31, 2017 at 10:43:19AM -0700, Tony Lindgren wrote:
> >> I think the use case is to get the booted kernel size from zImage
> >> to avoid overwriting dts or initramfs. Don't we already have that
> >> at the end of zImage somewhere for kexec?
> >
> > We do, but finding where that is is difficult if a DTB has been appended
> > as it's right at the end of the compressed data.  kexec doesn't use it,
> > it just assumes there's a 5x expansion of the kernel compressed image.
> 
> My patch already take the appended DTB blob part into consideration.
> It uses "unxz --single-stream" instead of "unxz" to extract, so unxz
> just ignore the DTB.

Sorry, I wasn't talking about the patch here.

The whole point of _this_ method of replying, where the relevant context
of the message being replied to is left and the reply is placed directly
beneath the paragraph to which the reply is relevant is to make clear
_what_ is being replied to.

My comments above are directly beneath a comment from Tony Lindgren asking
about the size field, which is appended after the compressed kernel image
data.  My reply was to Tony on that point alone, and why we aren't using
that for kexec-tools.

kexec-tools used to assume too small a size for the expansion of the
kernel image, and that has been changed.

I'm afraid that I still haven't got around to looking at _any_ of the
URLs you've provided so I'm still in the dark about (a) why this is
needed and (b) why it's acceptable for this script to output a binary
blob for ARM rather than an ELF file that it guarantees today.  It's
annoying that elinks' SSL implementation is buggy and causes connections
to certain SSL sites to fail, but alas that's the inconvenience that
buggy software causes.

Until I'm able to see some reasoning, I can't begin to consider this
patch.

Moreover, I'm not convinced by any of your arguments about why it's
acceptable to defeat the check for the ELF object _for all users_ of
this script.  What if some other architecture has multiple compressed
objects in their image and the "vmlinux" is not the first.  Your
argument seems to be "they should use a different script", yet you
are the one changing the script from outputting the first compressed
ELF object to merely producing the first compressed image.  "vmlinux"
is the name of the ELF kernel image, and the script is called
"extract-vmlinux" so this is obviously the right script which should
be extracting the correct compressed ELF object.

Hence, I don't buy your reasoning that in such a situation, another
architecture's use of this script should use some other script because
your changes fix it for ARM but cause a regression elsewhere.

The way we work on the kernel, if a change causes a user visible
regression, then the change was wrong and an alternative solution
needs to be sought.

I percieve the risk of this patch causing a regression on other
architectures to be high because it gratuitously changes functionality
of the script which didn't have to be there if all we cared about was
extracting the first image.  Hence, it must have been added because
there is a need for it.

So far, nothing has been said that lowers that risk.

It's not up to me to search around to try and evaluate risk - it's
my responsibility to ask the question (I already did) and it's the
submitter's responsibility to show that it's likely safe.

Thus far, the reasoning seems to be purely "several people are having
a problem with this script on ARM, here's a patch that fixes it for
ARM but might break existing users, but we don't care about existing
users."  That makes me even more wary.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up
According to speedtest.net: 8.21Mbps down 510kbps up

Re: [PATCH] scripts: make extract-vmlinux support armel/armhf

[Russell King - ARM Linux]

On Fri, Sep 01, 2017 at 01:05:16AM +0900, Roger Shimizu wrote:
> On Fri, Sep 1, 2017 at 12:49 AM, Russell King - ARM Linux
> <linux@armlinux.org.uk> wrote:
> > However, ARM zImage does not store a compressed ELF object, it stores a
> > compressed binary image of the executable in memory, so what you get out
> > of this script is not an ELF file, but the binary image (iow, what
> > arch/arm/boot/Image is.)
> >
> > What's the use case - what are you using the output of this script with?
> 
> I already provided a use case, to trace a Debian kernel issue caused
> by kernel size.
> - https://bugs.debian.org/870185#50

Knowing the decompressed kernel image size alone is not sufficient
to guarantee that a kernel can boot, since both the decompressed and
compressed kernel images must be in memory together.

The temptation (as can be seen from kexec tooling) is to take the
size of the compressed image and allow for a 4x expansion of that
image, forgetting that the compressed image needs to remain available
during the decompression stage.  Hence, kexec was fixed to assume
that 5x the compressed kernel image size would be needed for the
kernel to successfully boot.

As far as I'm aware, that fixed the problem and no one has since
shown that it fails.  This simple method should also work without
needing to resort to extracting the compressed image.

Now, we do store the decompressed image size within the compressed
image (it's at the end of the compressed image) but it's very fiddly
to get to - there is other data after the end of the compressed image
which is of variable length, and although we have a header on the
compressed kernel which tells us the end of its own data, we have no
way to know how far back from the end of the image that size
information is stored.

This is a tad annoying, but it is the result of "evolution" rather
than explicit design - and this is why kexec tooling has had to
resort to the 5x heuristic.

> I also see other people complaining this issue:
>  - https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
>  - https://bugs.linaro.org/show_bug.cgi?id=461

These URLs are identical to those which you listed in your first
email.  All of them do not say why people want to do this, only
that they're trying to do it and it doesn't work.

In my first reply to you, I explicitly asked for use cases.  That
means providing the reason why the feature is necessary, and what
it is being used for, not providing evidence that folk want to do
it.

Having a "want" is not the same as explaining "why" something is
required.

> By a random web search, I find another blog post on how to dissamble the kernel.
>  - https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/

This is a great example which backs up _my_ assertion below:

> > and here you always pass '1', thereby bypassing the ELF check for every
> > architecture.  What if some architecture stores another compressed
> > object in the same compressed image?  I suspect the check is there to
> > ensure that it works on architectures where the compressed kernel image
> > contains other compressed objects.

The "blog.packagecloud.io" blog entry you've pointed at gives an
example of how to extract the *ELF object* from the compressed
kernel image and then run objdump on it.

As I've already explained, "vmlinux" is the name of the ELF kernel
image, and the script is called "extract-vmlinux".  Ergo, the
script is used to extract the ELF kernel image from a compressed
kernel image.  That is its purpose.  The resulting image should be
acceptable to objdump to disassemble.

So, even if the extract-vmlinux script were to be modified in the
way you want, the rest of the blog post is irrelevant, because the
resulting output from the modified script would produce a non-ELF
object and the following steps given there would fail.

So I repeat my questions:

1. what is the use case?

2. what use is the output of this script put to?

   (for the above two questions, I want to know what user*s* (plural)
    are wanting this script to do and what purpose they're putting it
    to.)

3. what impact does this modification have to other architectures
   which may be using this script?

4. is it safe to modify the script to provide the first compressed
   object whatever it may be rather than the first compressed ELF
   object?

And some new questions:

5. what about other scripts using extract-vmlinux which then go on
   to expect the output of the script to be an ELF file?

6. should the script _on ARM only_ turn the zImage back into an ELF
   file suitable for other tooling, so the script provides an
   identical experience for all architectures?

7. have these modifications been tested with other architecture's
   kernel images to confirm that it doesn't cause a regression for
   existing users?

Here's a much wider question: with the advent of KASLR for ARM, should
we store a partially linked kernel ELF image within the zImage instead
of a fully linked image, and then have a minimal linker as part of
the zImage decompressor to link the ELF image for the randomized
address?  If we did that, then the existing script would just work for
ARM.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up
According to speedtest.net: 8.21Mbps down 510kbps up

[PATCH] scripts: make extract-vmlinux support armel/armhf

[Russell King - ARM Linux]

On Fri, Sep 01, 2017 at 01:05:16AM +0900, Roger Shimizu wrote:
> On Fri, Sep 1, 2017 at 12:49 AM, Russell King - ARM Linux
> <linux@armlinux.org.uk> wrote:
> > However, ARM zImage does not store a compressed ELF object, it stores a
> > compressed binary image of the executable in memory, so what you get out
> > of this script is not an ELF file, but the binary image (iow, what
> > arch/arm/boot/Image is.)
> >
> > What's the use case - what are you using the output of this script with?
> 
> I already provided a use case, to trace a Debian kernel issue caused
> by kernel size.
> - https://bugs.debian.org/870185#50

Knowing the decompressed kernel image size alone is not sufficient
to guarantee that a kernel can boot, since both the decompressed and
compressed kernel images must be in memory together.

The temptation (as can be seen from kexec tooling) is to take the
size of the compressed image and allow for a 4x expansion of that
image, forgetting that the compressed image needs to remain available
during the decompression stage.  Hence, kexec was fixed to assume
that 5x the compressed kernel image size would be needed for the
kernel to successfully boot.

As far as I'm aware, that fixed the problem and no one has since
shown that it fails.  This simple method should also work without
needing to resort to extracting the compressed image.

Now, we do store the decompressed image size within the compressed
image (it's at the end of the compressed image) but it's very fiddly
to get to - there is other data after the end of the compressed image
which is of variable length, and although we have a header on the
compressed kernel which tells us the end of its own data, we have no
way to know how far back from the end of the image that size
information is stored.

This is a tad annoying, but it is the result of "evolution" rather
than explicit design - and this is why kexec tooling has had to
resort to the 5x heuristic.

> I also see other people complaining this issue:
>  - https://bugs.launchpad.net/linaro-ubuntu/+bug/1050453
>  - https://bugs.linaro.org/show_bug.cgi?id=461

These URLs are identical to those which you listed in your first
email.  All of them do not say why people want to do this, only
that they're trying to do it and it doesn't work.

In my first reply to you, I explicitly asked for use cases.  That
means providing the reason why the feature is necessary, and what
it is being used for, not providing evidence that folk want to do
it.

Having a "want" is not the same as explaining "why" something is
required.

> By a random web search, I find another blog post on how to dissamble the kernel.
>  - https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/

This is a great example which backs up _my_ assertion below:

> > and here you always pass '1', thereby bypassing the ELF check for every
> > architecture.  What if some architecture stores another compressed
> > object in the same compressed image?  I suspect the check is there to
> > ensure that it works on architectures where the compressed kernel image
> > contains other compressed objects.

The "blog.packagecloud.io" blog entry you've pointed at gives an
example of how to extract the *ELF object* from the compressed
kernel image and then run objdump on it.

As I've already explained, "vmlinux" is the name of the ELF kernel
image, and the script is called "extract-vmlinux".  Ergo, the
script is used to extract the ELF kernel image from a compressed
kernel image.  That is its purpose.  The resulting image should be
acceptable to objdump to disassemble.

So, even if the extract-vmlinux script were to be modified in the
way you want, the rest of the blog post is irrelevant, because the
resulting output from the modified script would produce a non-ELF
object and the following steps given there would fail.

So I repeat my questions:

1. what is the use case?

2. what use is the output of this script put to?

   (for the above two questions, I want to know what user*s* (plural)
    are wanting this script to do and what purpose they're putting it
    to.)

3. what impact does this modification have to other architectures
   which may be using this script?

4. is it safe to modify the script to provide the first compressed
   object whatever it may be rather than the first compressed ELF
   object?

And some new questions:

5. what about other scripts using extract-vmlinux which then go on
   to expect the output of the script to be an ELF file?

6. should the script _on ARM only_ turn the zImage back into an ELF
   file suitable for other tooling, so the script provides an
   identical experience for all architectures?

7. have these modifications been tested with other architecture's
   kernel images to confirm that it doesn't cause a regression for
   existing users?

Here's a much wider question: with the advent of KASLR for ARM, should
we store a partially linked kernel ELF image within the zImage instead
of a fully linked image, and then have a minimal linker as part of
the zImage decompressor to link the ELF image for the randomized
address?  If we did that, then the existing script would just work for
ARM.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up
According to speedtest.net: 8.21Mbps down 510kbps up