[PATCH 0/2] selftests: firmware: fw_filesystem fix for busybox

[PATCH 0/2] selftests: firmware: fw_filesystem fix for busybox

[Dan Rue]

Fix fw_filesystem.sh to run in an automated environment under busybox.

After this change, fw_run_tests.sh still fails at some point in fw_fallback.sh,
with error "usermode helper disabled so ignoring test". This is coming from
fw_lib.sh:verify_reqs() because $HAS_FW_LOADER_USER_HELPER is set to no.

Dan Rue (2):
  selftests: firmware: remove use of non-standard diff -Z option
  selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to
    config

 tools/testing/selftests/firmware/config           | 1 +
 tools/testing/selftests/firmware/fw_filesystem.sh | 9 +++------
 2 files changed, 4 insertions(+), 6 deletions(-)

-- 
2.19.1

[PATCH 0/2] selftests: firmware: fw_filesystem fix for busybox

[dan.rue]

Fix fw_filesystem.sh to run in an automated environment under busybox.

After this change, fw_run_tests.sh still fails at some point in fw_fallback.sh,
with error "usermode helper disabled so ignoring test". This is coming from
fw_lib.sh:verify_reqs() because $HAS_FW_LOADER_USER_HELPER is set to no.

Dan Rue (2):
  selftests: firmware: remove use of non-standard diff -Z option
  selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to
    config

 tools/testing/selftests/firmware/config           | 1 +
 tools/testing/selftests/firmware/fw_filesystem.sh | 9 +++------
 2 files changed, 4 insertions(+), 6 deletions(-)

-- 
2.19.1

[PATCH 0/2] selftests: firmware: fw_filesystem fix for busybox

[Dan Rue]

Fix fw_filesystem.sh to run in an automated environment under busybox.

After this change, fw_run_tests.sh still fails at some point in fw_fallback.sh,
with error "usermode helper disabled so ignoring test". This is coming from
fw_lib.sh:verify_reqs() because $HAS_FW_LOADER_USER_HELPER is set to no.

Dan Rue (2):
  selftests: firmware: remove use of non-standard diff -Z option
  selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to
    config

 tools/testing/selftests/firmware/config           | 1 +
 tools/testing/selftests/firmware/fw_filesystem.sh | 9 +++------
 2 files changed, 4 insertions(+), 6 deletions(-)

-- 
2.19.1

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[Dan Rue]

diff -Z is used to trim the trailing whitespace when comparing the
loaded firmware file with the source firmware file. However, per the
comment in the source code, -Z should not be necessary. In testing, the
input and output files are identical.

Additionally, -Z is not a standard option and is not available in
environments such as busybox. When -Z is not supported, diff fails with
a usage error, which is suppressed, but then causes read_firmwares() to
exit with a false failure message.

Signed-off-by: Dan Rue <dan.rue@linaro.org>
---
 tools/testing/selftests/firmware/fw_filesystem.sh | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/tools/testing/selftests/firmware/fw_filesystem.sh b/tools/testing/selftests/firmware/fw_filesystem.sh
index a4320c4b44dc..466cf2f91ba0 100755
--- a/tools/testing/selftests/firmware/fw_filesystem.sh
+++ b/tools/testing/selftests/firmware/fw_filesystem.sh
@@ -155,11 +155,8 @@ read_firmwares()
 {
 	for i in $(seq 0 3); do
 		config_set_read_fw_idx $i
-		# Verify the contents are what we expect.
-		# -Z required for now -- check for yourself, md5sum
-		# on $FW and DIR/read_firmware will yield the same. Even
-		# cmp agrees, so something is off.
-		if ! diff -q -Z "$FW" $DIR/read_firmware 2>/dev/null ; then
+		# Verify the contents match
+		if ! diff -q "$FW" $DIR/read_firmware 2>/dev/null ; then
 			echo "request #$i: firmware was not loaded" >&2
 			exit 1
 		fi
@@ -171,7 +168,7 @@ read_firmwares_expect_nofile()
 	for i in $(seq 0 3); do
 		config_set_read_fw_idx $i
 		# Ensures contents differ
-		if diff -q -Z "$FW" $DIR/read_firmware 2>/dev/null ; then
+		if diff -q "$FW" $DIR/read_firmware 2>/dev/null ; then
 			echo "request $i: file was not expected to match" >&2
 			exit 1
 		fi
-- 
2.19.1

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[dan.rue]

diff -Z is used to trim the trailing whitespace when comparing the
loaded firmware file with the source firmware file. However, per the
comment in the source code, -Z should not be necessary. In testing, the
input and output files are identical.

Additionally, -Z is not a standard option and is not available in
environments such as busybox. When -Z is not supported, diff fails with
a usage error, which is suppressed, but then causes read_firmwares() to
exit with a false failure message.

Signed-off-by: Dan Rue <dan.rue at linaro.org>
---
 tools/testing/selftests/firmware/fw_filesystem.sh | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/tools/testing/selftests/firmware/fw_filesystem.sh b/tools/testing/selftests/firmware/fw_filesystem.sh
index a4320c4b44dc..466cf2f91ba0 100755
--- a/tools/testing/selftests/firmware/fw_filesystem.sh
+++ b/tools/testing/selftests/firmware/fw_filesystem.sh
@@ -155,11 +155,8 @@ read_firmwares()
 {
 	for i in $(seq 0 3); do
 		config_set_read_fw_idx $i
-		# Verify the contents are what we expect.
-		# -Z required for now -- check for yourself, md5sum
-		# on $FW and DIR/read_firmware will yield the same. Even
-		# cmp agrees, so something is off.
-		if ! diff -q -Z "$FW" $DIR/read_firmware 2>/dev/null ; then
+		# Verify the contents match
+		if ! diff -q "$FW" $DIR/read_firmware 2>/dev/null ; then
 			echo "request #$i: firmware was not loaded" >&2
 			exit 1
 		fi
@@ -171,7 +168,7 @@ read_firmwares_expect_nofile()
 	for i in $(seq 0 3); do
 		config_set_read_fw_idx $i
 		# Ensures contents differ
-		if diff -q -Z "$FW" $DIR/read_firmware 2>/dev/null ; then
+		if diff -q "$FW" $DIR/read_firmware 2>/dev/null ; then
 			echo "request $i: file was not expected to match" >&2
 			exit 1
 		fi
-- 
2.19.1

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[Dan Rue]

diff -Z is used to trim the trailing whitespace when comparing the
loaded firmware file with the source firmware file. However, per the
comment in the source code, -Z should not be necessary. In testing, the
input and output files are identical.

Additionally, -Z is not a standard option and is not available in
environments such as busybox. When -Z is not supported, diff fails with
a usage error, which is suppressed, but then causes read_firmwares() to
exit with a false failure message.

Signed-off-by: Dan Rue <dan.rue at linaro.org>
---
 tools/testing/selftests/firmware/fw_filesystem.sh | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/tools/testing/selftests/firmware/fw_filesystem.sh b/tools/testing/selftests/firmware/fw_filesystem.sh
index a4320c4b44dc..466cf2f91ba0 100755
--- a/tools/testing/selftests/firmware/fw_filesystem.sh
+++ b/tools/testing/selftests/firmware/fw_filesystem.sh
@@ -155,11 +155,8 @@ read_firmwares()
 {
 	for i in $(seq 0 3); do
 		config_set_read_fw_idx $i
-		# Verify the contents are what we expect.
-		# -Z required for now -- check for yourself, md5sum
-		# on $FW and DIR/read_firmware will yield the same. Even
-		# cmp agrees, so something is off.
-		if ! diff -q -Z "$FW" $DIR/read_firmware 2>/dev/null ; then
+		# Verify the contents match
+		if ! diff -q "$FW" $DIR/read_firmware 2>/dev/null ; then
 			echo "request #$i: firmware was not loaded" >&2
 			exit 1
 		fi
@@ -171,7 +168,7 @@ read_firmwares_expect_nofile()
 	for i in $(seq 0 3); do
 		config_set_read_fw_idx $i
 		# Ensures contents differ
-		if diff -q -Z "$FW" $DIR/read_firmware 2>/dev/null ; then
+		if diff -q "$FW" $DIR/read_firmware 2>/dev/null ; then
 			echo "request $i: file was not expected to match" >&2
 			exit 1
 		fi
-- 
2.19.1

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[Dan Rue]

CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
Without it, fw_fallback.sh fails with 'usermode helper disabled so
ignoring test'. Enable the config in selftest so that it gets built by
default.

Signed-off-by: Dan Rue <dan.rue@linaro.org>
---
 tools/testing/selftests/firmware/config | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
index bf634dda0720..913a25a4a32b 100644
--- a/tools/testing/selftests/firmware/config
+++ b/tools/testing/selftests/firmware/config
@@ -1,5 +1,6 @@
 CONFIG_TEST_FIRMWARE=y
 CONFIG_FW_LOADER=y
 CONFIG_FW_LOADER_USER_HELPER=y
+CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
 CONFIG_IKCONFIG=y
 CONFIG_IKCONFIG_PROC=y
-- 
2.19.1

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[dan.rue]

CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
Without it, fw_fallback.sh fails with 'usermode helper disabled so
ignoring test'. Enable the config in selftest so that it gets built by
default.

Signed-off-by: Dan Rue <dan.rue at linaro.org>
---
 tools/testing/selftests/firmware/config | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
index bf634dda0720..913a25a4a32b 100644
--- a/tools/testing/selftests/firmware/config
+++ b/tools/testing/selftests/firmware/config
@@ -1,5 +1,6 @@
 CONFIG_TEST_FIRMWARE=y
 CONFIG_FW_LOADER=y
 CONFIG_FW_LOADER_USER_HELPER=y
+CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
 CONFIG_IKCONFIG=y
 CONFIG_IKCONFIG_PROC=y
-- 
2.19.1

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[Dan Rue]

CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
Without it, fw_fallback.sh fails with 'usermode helper disabled so
ignoring test'. Enable the config in selftest so that it gets built by
default.

Signed-off-by: Dan Rue <dan.rue at linaro.org>
---
 tools/testing/selftests/firmware/config | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
index bf634dda0720..913a25a4a32b 100644
--- a/tools/testing/selftests/firmware/config
+++ b/tools/testing/selftests/firmware/config
@@ -1,5 +1,6 @@
 CONFIG_TEST_FIRMWARE=y
 CONFIG_FW_LOADER=y
 CONFIG_FW_LOADER_USER_HELPER=y
+CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
 CONFIG_IKCONFIG=y
 CONFIG_IKCONFIG_PROC=y
-- 
2.19.1

Re: [PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[Kees Cook]

On Mon, Nov 26, 2018 at 7:12 PM, Dan Rue <dan.rue@linaro.org> wrote:
> diff -Z is used to trim the trailing whitespace when comparing the
> loaded firmware file with the source firmware file. However, per the
> comment in the source code, -Z should not be necessary. In testing, the
> input and output files are identical.
>
> Additionally, -Z is not a standard option and is not available in
> environments such as busybox. When -Z is not supported, diff fails with
> a usage error, which is suppressed, but then causes read_firmwares() to
> exit with a false failure message.
>
> Signed-off-by: Dan Rue <dan.rue@linaro.org>

Acked-by: Kees Cook <keescook@chromium.org>

-Kees

> ---
>  tools/testing/selftests/firmware/fw_filesystem.sh | 9 +++------
>  1 file changed, 3 insertions(+), 6 deletions(-)
>
> diff --git a/tools/testing/selftests/firmware/fw_filesystem.sh b/tools/testing/selftests/firmware/fw_filesystem.sh
> index a4320c4b44dc..466cf2f91ba0 100755
> --- a/tools/testing/selftests/firmware/fw_filesystem.sh
> +++ b/tools/testing/selftests/firmware/fw_filesystem.sh
> @@ -155,11 +155,8 @@ read_firmwares()
>  {
>         for i in $(seq 0 3); do
>                 config_set_read_fw_idx $i
> -               # Verify the contents are what we expect.
> -               # -Z required for now -- check for yourself, md5sum
> -               # on $FW and DIR/read_firmware will yield the same. Even
> -               # cmp agrees, so something is off.
> -               if ! diff -q -Z "$FW" $DIR/read_firmware 2>/dev/null ; then
> +               # Verify the contents match
> +               if ! diff -q "$FW" $DIR/read_firmware 2>/dev/null ; then
>                         echo "request #$i: firmware was not loaded" >&2
>                         exit 1
>                 fi
> @@ -171,7 +168,7 @@ read_firmwares_expect_nofile()
>         for i in $(seq 0 3); do
>                 config_set_read_fw_idx $i
>                 # Ensures contents differ
> -               if diff -q -Z "$FW" $DIR/read_firmware 2>/dev/null ; then
> +               if diff -q "$FW" $DIR/read_firmware 2>/dev/null ; then
>                         echo "request $i: file was not expected to match" >&2
>                         exit 1
>                 fi
> --
> 2.19.1
>



-- 
Kees Cook

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[keescook]

On Mon, Nov 26, 2018 at 7:12 PM, Dan Rue <dan.rue at linaro.org> wrote:
> diff -Z is used to trim the trailing whitespace when comparing the
> loaded firmware file with the source firmware file. However, per the
> comment in the source code, -Z should not be necessary. In testing, the
> input and output files are identical.
>
> Additionally, -Z is not a standard option and is not available in
> environments such as busybox. When -Z is not supported, diff fails with
> a usage error, which is suppressed, but then causes read_firmwares() to
> exit with a false failure message.
>
> Signed-off-by: Dan Rue <dan.rue at linaro.org>

Acked-by: Kees Cook <keescook at chromium.org>

-Kees

> ---
>  tools/testing/selftests/firmware/fw_filesystem.sh | 9 +++------
>  1 file changed, 3 insertions(+), 6 deletions(-)
>
> diff --git a/tools/testing/selftests/firmware/fw_filesystem.sh b/tools/testing/selftests/firmware/fw_filesystem.sh
> index a4320c4b44dc..466cf2f91ba0 100755
> --- a/tools/testing/selftests/firmware/fw_filesystem.sh
> +++ b/tools/testing/selftests/firmware/fw_filesystem.sh
> @@ -155,11 +155,8 @@ read_firmwares()
>  {
>         for i in $(seq 0 3); do
>                 config_set_read_fw_idx $i
> -               # Verify the contents are what we expect.
> -               # -Z required for now -- check for yourself, md5sum
> -               # on $FW and DIR/read_firmware will yield the same. Even
> -               # cmp agrees, so something is off.
> -               if ! diff -q -Z "$FW" $DIR/read_firmware 2>/dev/null ; then
> +               # Verify the contents match
> +               if ! diff -q "$FW" $DIR/read_firmware 2>/dev/null ; then
>                         echo "request #$i: firmware was not loaded" >&2
>                         exit 1
>                 fi
> @@ -171,7 +168,7 @@ read_firmwares_expect_nofile()
>         for i in $(seq 0 3); do
>                 config_set_read_fw_idx $i
>                 # Ensures contents differ
> -               if diff -q -Z "$FW" $DIR/read_firmware 2>/dev/null ; then
> +               if diff -q "$FW" $DIR/read_firmware 2>/dev/null ; then
>                         echo "request $i: file was not expected to match" >&2
>                         exit 1
>                 fi
> --
> 2.19.1
>



-- 
Kees Cook

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[Kees Cook]

On Mon, Nov 26, 2018@7:12 PM, Dan Rue <dan.rue@linaro.org> wrote:
> diff -Z is used to trim the trailing whitespace when comparing the
> loaded firmware file with the source firmware file. However, per the
> comment in the source code, -Z should not be necessary. In testing, the
> input and output files are identical.
>
> Additionally, -Z is not a standard option and is not available in
> environments such as busybox. When -Z is not supported, diff fails with
> a usage error, which is suppressed, but then causes read_firmwares() to
> exit with a false failure message.
>
> Signed-off-by: Dan Rue <dan.rue at linaro.org>

Acked-by: Kees Cook <keescook at chromium.org>

-Kees

> ---
>  tools/testing/selftests/firmware/fw_filesystem.sh | 9 +++------
>  1 file changed, 3 insertions(+), 6 deletions(-)
>
> diff --git a/tools/testing/selftests/firmware/fw_filesystem.sh b/tools/testing/selftests/firmware/fw_filesystem.sh
> index a4320c4b44dc..466cf2f91ba0 100755
> --- a/tools/testing/selftests/firmware/fw_filesystem.sh
> +++ b/tools/testing/selftests/firmware/fw_filesystem.sh
> @@ -155,11 +155,8 @@ read_firmwares()
>  {
>         for i in $(seq 0 3); do
>                 config_set_read_fw_idx $i
> -               # Verify the contents are what we expect.
> -               # -Z required for now -- check for yourself, md5sum
> -               # on $FW and DIR/read_firmware will yield the same. Even
> -               # cmp agrees, so something is off.
> -               if ! diff -q -Z "$FW" $DIR/read_firmware 2>/dev/null ; then
> +               # Verify the contents match
> +               if ! diff -q "$FW" $DIR/read_firmware 2>/dev/null ; then
>                         echo "request #$i: firmware was not loaded" >&2
>                         exit 1
>                 fi
> @@ -171,7 +168,7 @@ read_firmwares_expect_nofile()
>         for i in $(seq 0 3); do
>                 config_set_read_fw_idx $i
>                 # Ensures contents differ
> -               if diff -q -Z "$FW" $DIR/read_firmware 2>/dev/null ; then
> +               if diff -q "$FW" $DIR/read_firmware 2>/dev/null ; then
>                         echo "request $i: file was not expected to match" >&2
>                         exit 1
>                 fi
> --
> 2.19.1
>



-- 
Kees Cook

Re: [PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[Kees Cook]

On Mon, Nov 26, 2018 at 7:12 PM, Dan Rue <dan.rue@linaro.org> wrote:
> CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> Without it, fw_fallback.sh fails with 'usermode helper disabled so
> ignoring test'. Enable the config in selftest so that it gets built by
> default.
>
> Signed-off-by: Dan Rue <dan.rue@linaro.org>

Acked-by: Kees Cook <keescook@chromium.org>

-Kees

> ---
>  tools/testing/selftests/firmware/config | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> index bf634dda0720..913a25a4a32b 100644
> --- a/tools/testing/selftests/firmware/config
> +++ b/tools/testing/selftests/firmware/config
> @@ -1,5 +1,6 @@
>  CONFIG_TEST_FIRMWARE=y
>  CONFIG_FW_LOADER=y
>  CONFIG_FW_LOADER_USER_HELPER=y
> +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
>  CONFIG_IKCONFIG=y
>  CONFIG_IKCONFIG_PROC=y
> --
> 2.19.1
>



-- 
Kees Cook

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[keescook]

On Mon, Nov 26, 2018 at 7:12 PM, Dan Rue <dan.rue at linaro.org> wrote:
> CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> Without it, fw_fallback.sh fails with 'usermode helper disabled so
> ignoring test'. Enable the config in selftest so that it gets built by
> default.
>
> Signed-off-by: Dan Rue <dan.rue at linaro.org>

Acked-by: Kees Cook <keescook at chromium.org>

-Kees

> ---
>  tools/testing/selftests/firmware/config | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> index bf634dda0720..913a25a4a32b 100644
> --- a/tools/testing/selftests/firmware/config
> +++ b/tools/testing/selftests/firmware/config
> @@ -1,5 +1,6 @@
>  CONFIG_TEST_FIRMWARE=y
>  CONFIG_FW_LOADER=y
>  CONFIG_FW_LOADER_USER_HELPER=y
> +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
>  CONFIG_IKCONFIG=y
>  CONFIG_IKCONFIG_PROC=y
> --
> 2.19.1
>



-- 
Kees Cook

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[Kees Cook]

On Mon, Nov 26, 2018@7:12 PM, Dan Rue <dan.rue@linaro.org> wrote:
> CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> Without it, fw_fallback.sh fails with 'usermode helper disabled so
> ignoring test'. Enable the config in selftest so that it gets built by
> default.
>
> Signed-off-by: Dan Rue <dan.rue at linaro.org>

Acked-by: Kees Cook <keescook at chromium.org>

-Kees

> ---
>  tools/testing/selftests/firmware/config | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> index bf634dda0720..913a25a4a32b 100644
> --- a/tools/testing/selftests/firmware/config
> +++ b/tools/testing/selftests/firmware/config
> @@ -1,5 +1,6 @@
>  CONFIG_TEST_FIRMWARE=y
>  CONFIG_FW_LOADER=y
>  CONFIG_FW_LOADER_USER_HELPER=y
> +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
>  CONFIG_IKCONFIG=y
>  CONFIG_IKCONFIG_PROC=y
> --
> 2.19.1
>



-- 
Kees Cook

Re: [PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[Luis Chamberlain]

On Mon, Nov 26, 2018 at 09:12:16PM -0600, Dan Rue wrote:
> CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> Without it, fw_fallback.sh fails with 'usermode helper disabled so
> ignoring test'. Enable the config in selftest so that it gets built by
> default.
> 
> Signed-off-by: Dan Rue <dan.rue@linaro.org>
> ---
>  tools/testing/selftests/firmware/config | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> index bf634dda0720..913a25a4a32b 100644
> --- a/tools/testing/selftests/firmware/config
> +++ b/tools/testing/selftests/firmware/config
> @@ -1,5 +1,6 @@
>  CONFIG_TEST_FIRMWARE=y
>  CONFIG_FW_LOADER=y
>  CONFIG_FW_LOADER_USER_HELPER=y
> +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
>  CONFIG_IKCONFIG=y
>  CONFIG_IKCONFIG_PROC=y

NACK -- the point of the changes was to *allow* us to mimic such
configuration through a proc sysctl knob.

You aren forcing CONFIG_FW_LOADER_USER_HELPER_FALLBACK but just having
CONFIG_FW_LOADER_USER_HELPER suffices to emulate the_FALLBACK
functionality.

The issue here seems to be that *all* tests fail once a configuration is
found which is not suitable a tests. With the shiny new proc sysctls we
can test all 3 kernel configurations in one shot. Since we test 3
different kernel configurations naturally some of these won't have the
features needed, so that failure should be treated as non-fatal to allow
the chain of other tests to continue.

This issue was a regression due to commit a6a9be9270c87 ("selftests:
firmware: return Kselftest Skip code for skipped tests") by Shuah for
the verify_reqs(). We need to treat this as a non-fatal / don't skip
return value.

The following would fix this chaining issue:

diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh
index 6c5f1b2ffb74..1cbb12e284a6 100755
--- a/tools/testing/selftests/firmware/fw_lib.sh
+++ b/tools/testing/selftests/firmware/fw_lib.sh
@@ -91,7 +91,7 @@ verify_reqs()
 	if [ "$TEST_REQS_FW_SYSFS_FALLBACK" = "yes" ]; then
 		if [ ! "$HAS_FW_LOADER_USER_HELPER" = "yes" ]; then
 			echo "usermode helper disabled so ignoring test"
-			exit $ksft_skip
+			exit 0
 		fi
 	fi
 }

However its not clear to me if instead we want some new special return
value for selftests so that the framework can detect an that an error
is non-fatal, and can continue. This is a tricky situation given the
script, existing upstream kernel module, are aware of such emulation
hacks via sysctl, but knowledge of this is not obvious to selftests.

Shuah, how do you suggest we handle this corner case? If you are OK
with the above hunk for now I can send a fix for it. In either case
this commit was added on v4.18, so the fix would be a stable fix.

  Luis

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[mcgrof]

On Mon, Nov 26, 2018 at 09:12:16PM -0600, Dan Rue wrote:
> CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> Without it, fw_fallback.sh fails with 'usermode helper disabled so
> ignoring test'. Enable the config in selftest so that it gets built by
> default.
> 
> Signed-off-by: Dan Rue <dan.rue at linaro.org>
> ---
>  tools/testing/selftests/firmware/config | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> index bf634dda0720..913a25a4a32b 100644
> --- a/tools/testing/selftests/firmware/config
> +++ b/tools/testing/selftests/firmware/config
> @@ -1,5 +1,6 @@
>  CONFIG_TEST_FIRMWARE=y
>  CONFIG_FW_LOADER=y
>  CONFIG_FW_LOADER_USER_HELPER=y
> +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
>  CONFIG_IKCONFIG=y
>  CONFIG_IKCONFIG_PROC=y

NACK -- the point of the changes was to *allow* us to mimic such
configuration through a proc sysctl knob.

You aren forcing CONFIG_FW_LOADER_USER_HELPER_FALLBACK but just having
CONFIG_FW_LOADER_USER_HELPER suffices to emulate the_FALLBACK
functionality.

The issue here seems to be that *all* tests fail once a configuration is
found which is not suitable a tests. With the shiny new proc sysctls we
can test all 3 kernel configurations in one shot. Since we test 3
different kernel configurations naturally some of these won't have the
features needed, so that failure should be treated as non-fatal to allow
the chain of other tests to continue.

This issue was a regression due to commit a6a9be9270c87 ("selftests:
firmware: return Kselftest Skip code for skipped tests") by Shuah for
the verify_reqs(). We need to treat this as a non-fatal / don't skip
return value.

The following would fix this chaining issue:

diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh
index 6c5f1b2ffb74..1cbb12e284a6 100755
--- a/tools/testing/selftests/firmware/fw_lib.sh
+++ b/tools/testing/selftests/firmware/fw_lib.sh
@@ -91,7 +91,7 @@ verify_reqs()
 	if [ "$TEST_REQS_FW_SYSFS_FALLBACK" = "yes" ]; then
 		if [ ! "$HAS_FW_LOADER_USER_HELPER" = "yes" ]; then
 			echo "usermode helper disabled so ignoring test"
-			exit $ksft_skip
+			exit 0
 		fi
 	fi
 }

However its not clear to me if instead we want some new special return
value for selftests so that the framework can detect an that an error
is non-fatal, and can continue. This is a tricky situation given the
script, existing upstream kernel module, are aware of such emulation
hacks via sysctl, but knowledge of this is not obvious to selftests.

Shuah, how do you suggest we handle this corner case? If you are OK
with the above hunk for now I can send a fix for it. In either case
this commit was added on v4.18, so the fix would be a stable fix.

  Luis

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[Luis Chamberlain]

On Mon, Nov 26, 2018@09:12:16PM -0600, Dan Rue wrote:
> CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> Without it, fw_fallback.sh fails with 'usermode helper disabled so
> ignoring test'. Enable the config in selftest so that it gets built by
> default.
> 
> Signed-off-by: Dan Rue <dan.rue at linaro.org>
> ---
>  tools/testing/selftests/firmware/config | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> index bf634dda0720..913a25a4a32b 100644
> --- a/tools/testing/selftests/firmware/config
> +++ b/tools/testing/selftests/firmware/config
> @@ -1,5 +1,6 @@
>  CONFIG_TEST_FIRMWARE=y
>  CONFIG_FW_LOADER=y
>  CONFIG_FW_LOADER_USER_HELPER=y
> +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
>  CONFIG_IKCONFIG=y
>  CONFIG_IKCONFIG_PROC=y

NACK -- the point of the changes was to *allow* us to mimic such
configuration through a proc sysctl knob.

You aren forcing CONFIG_FW_LOADER_USER_HELPER_FALLBACK but just having
CONFIG_FW_LOADER_USER_HELPER suffices to emulate the_FALLBACK
functionality.

The issue here seems to be that *all* tests fail once a configuration is
found which is not suitable a tests. With the shiny new proc sysctls we
can test all 3 kernel configurations in one shot. Since we test 3
different kernel configurations naturally some of these won't have the
features needed, so that failure should be treated as non-fatal to allow
the chain of other tests to continue.

This issue was a regression due to commit a6a9be9270c87 ("selftests:
firmware: return Kselftest Skip code for skipped tests") by Shuah for
the verify_reqs(). We need to treat this as a non-fatal / don't skip
return value.

The following would fix this chaining issue:

diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh
index 6c5f1b2ffb74..1cbb12e284a6 100755
--- a/tools/testing/selftests/firmware/fw_lib.sh
+++ b/tools/testing/selftests/firmware/fw_lib.sh
@@ -91,7 +91,7 @@ verify_reqs()
 	if [ "$TEST_REQS_FW_SYSFS_FALLBACK" = "yes" ]; then
 		if [ ! "$HAS_FW_LOADER_USER_HELPER" = "yes" ]; then
 			echo "usermode helper disabled so ignoring test"
-			exit $ksft_skip
+			exit 0
 		fi
 	fi
 }

However its not clear to me if instead we want some new special return
value for selftests so that the framework can detect an that an error
is non-fatal, and can continue. This is a tricky situation given the
script, existing upstream kernel module, are aware of such emulation
hacks via sysctl, but knowledge of this is not obvious to selftests.

Shuah, how do you suggest we handle this corner case? If you are OK
with the above hunk for now I can send a fix for it. In either case
this commit was added on v4.18, so the fix would be a stable fix.

  Luis

Re: [PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[Luis Chamberlain]

On Mon, Nov 26, 2018 at 09:12:15PM -0600, Dan Rue wrote:
> diff -Z is used to trim the trailing whitespace when comparing the
> loaded firmware file with the source firmware file. However, per the
> comment in the source code, -Z should not be necessary. In testing, the
> input and output files are identical.
> 
> Additionally, -Z is not a standard option and is not available in
> environments such as busybox. When -Z is not supported, diff fails with
> a usage error, which is suppressed, but then causes read_firmwares() to
> exit with a false failure message.

NACK -- this breaks testing on debian:

Testing with the file present...
Batched request_firmware() try #1: Files
/tmp/tmp.8GWkoSo5jZ/test-firmware.bin and
/sys/devices/virtual/misc/test_firmware/read_firmware differ
request #0: firmware was not loaded

Please add a quirks check, enable it by default, and remove it for
Busybox.

   Luis

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[mcgrof]

On Mon, Nov 26, 2018 at 09:12:15PM -0600, Dan Rue wrote:
> diff -Z is used to trim the trailing whitespace when comparing the
> loaded firmware file with the source firmware file. However, per the
> comment in the source code, -Z should not be necessary. In testing, the
> input and output files are identical.
> 
> Additionally, -Z is not a standard option and is not available in
> environments such as busybox. When -Z is not supported, diff fails with
> a usage error, which is suppressed, but then causes read_firmwares() to
> exit with a false failure message.

NACK -- this breaks testing on debian:

Testing with the file present...
Batched request_firmware() try #1: Files
/tmp/tmp.8GWkoSo5jZ/test-firmware.bin and
/sys/devices/virtual/misc/test_firmware/read_firmware differ
request #0: firmware was not loaded

Please add a quirks check, enable it by default, and remove it for
Busybox.

   Luis

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[Luis Chamberlain]

On Mon, Nov 26, 2018@09:12:15PM -0600, Dan Rue wrote:
> diff -Z is used to trim the trailing whitespace when comparing the
> loaded firmware file with the source firmware file. However, per the
> comment in the source code, -Z should not be necessary. In testing, the
> input and output files are identical.
> 
> Additionally, -Z is not a standard option and is not available in
> environments such as busybox. When -Z is not supported, diff fails with
> a usage error, which is suppressed, but then causes read_firmwares() to
> exit with a false failure message.

NACK -- this breaks testing on debian:

Testing with the file present...
Batched request_firmware() try #1: Files
/tmp/tmp.8GWkoSo5jZ/test-firmware.bin and
/sys/devices/virtual/misc/test_firmware/read_firmware differ
request #0: firmware was not loaded

Please add a quirks check, enable it by default, and remove it for
Busybox.

   Luis

Re: [PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[Dan Rue]

On Thu, Nov 29, 2018 at 06:37:32PM -0800, Luis Chamberlain wrote:
> On Mon, Nov 26, 2018 at 09:12:15PM -0600, Dan Rue wrote:
> > diff -Z is used to trim the trailing whitespace when comparing the
> > loaded firmware file with the source firmware file. However, per the
> > comment in the source code, -Z should not be necessary. In testing, the
> > input and output files are identical.
> > 
> > Additionally, -Z is not a standard option and is not available in
> > environments such as busybox. When -Z is not supported, diff fails with
> > a usage error, which is suppressed, but then causes read_firmwares() to
> > exit with a false failure message.
> 
> NACK -- this breaks testing on debian:
> 
> Testing with the file present...
> Batched request_firmware() try #1: Files
> /tmp/tmp.8GWkoSo5jZ/test-firmware.bin and
> /sys/devices/virtual/misc/test_firmware/read_firmware differ
> request #0: firmware was not loaded
> 
> Please add a quirks check, enable it by default, and remove it for
> Busybox.

Thanks for the review. Shuah, can you please drop this one?

Dan

-- 
Linaro - Kernel Validation

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[dan.rue]

On Thu, Nov 29, 2018 at 06:37:32PM -0800, Luis Chamberlain wrote:
> On Mon, Nov 26, 2018 at 09:12:15PM -0600, Dan Rue wrote:
> > diff -Z is used to trim the trailing whitespace when comparing the
> > loaded firmware file with the source firmware file. However, per the
> > comment in the source code, -Z should not be necessary. In testing, the
> > input and output files are identical.
> > 
> > Additionally, -Z is not a standard option and is not available in
> > environments such as busybox. When -Z is not supported, diff fails with
> > a usage error, which is suppressed, but then causes read_firmwares() to
> > exit with a false failure message.
> 
> NACK -- this breaks testing on debian:
> 
> Testing with the file present...
> Batched request_firmware() try #1: Files
> /tmp/tmp.8GWkoSo5jZ/test-firmware.bin and
> /sys/devices/virtual/misc/test_firmware/read_firmware differ
> request #0: firmware was not loaded
> 
> Please add a quirks check, enable it by default, and remove it for
> Busybox.

Thanks for the review. Shuah, can you please drop this one?

Dan

-- 
Linaro - Kernel Validation

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[Dan Rue]

On Thu, Nov 29, 2018@06:37:32PM -0800, Luis Chamberlain wrote:
> On Mon, Nov 26, 2018@09:12:15PM -0600, Dan Rue wrote:
> > diff -Z is used to trim the trailing whitespace when comparing the
> > loaded firmware file with the source firmware file. However, per the
> > comment in the source code, -Z should not be necessary. In testing, the
> > input and output files are identical.
> > 
> > Additionally, -Z is not a standard option and is not available in
> > environments such as busybox. When -Z is not supported, diff fails with
> > a usage error, which is suppressed, but then causes read_firmwares() to
> > exit with a false failure message.
> 
> NACK -- this breaks testing on debian:
> 
> Testing with the file present...
> Batched request_firmware() try #1: Files
> /tmp/tmp.8GWkoSo5jZ/test-firmware.bin and
> /sys/devices/virtual/misc/test_firmware/read_firmware differ
> request #0: firmware was not loaded
> 
> Please add a quirks check, enable it by default, and remove it for
> Busybox.

Thanks for the review. Shuah, can you please drop this one?

Dan

-- 
Linaro - Kernel Validation

Re: [PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[Luis Chamberlain]

On Thu, Nov 29, 2018 at 8:31 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> On Mon, Nov 26, 2018 at 09:12:16PM -0600, Dan Rue wrote:
> > CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> > Without it, fw_fallback.sh fails with 'usermode helper disabled so
> > ignoring test'. Enable the config in selftest so that it gets built by
> > default.
> >
> > Signed-off-by: Dan Rue <dan.rue@linaro.org>
> > ---
> >  tools/testing/selftests/firmware/config | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> > index bf634dda0720..913a25a4a32b 100644
> > --- a/tools/testing/selftests/firmware/config
> > +++ b/tools/testing/selftests/firmware/config
> > @@ -1,5 +1,6 @@
> >  CONFIG_TEST_FIRMWARE=y
> >  CONFIG_FW_LOADER=y
> >  CONFIG_FW_LOADER_USER_HELPER=y
> > +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
> >  CONFIG_IKCONFIG=y
> >  CONFIG_IKCONFIG_PROC=y
>
> NACK -- the point of the changes was to *allow* us to mimic such
> configuration through a proc sysctl knob.
>
> You aren forcing CONFIG_FW_LOADER_USER_HELPER_FALLBACK but just having
> CONFIG_FW_LOADER_USER_HELPER suffices to emulate the_FALLBACK
> functionality.

Dan, again, you broke the whole point to the amount of work that went
into emulating testing. As such anyone testing their changes would
yield incorrect results.

> The issue here seems to be that *all* tests fail once a configuration is
> found which is not suitable a tests. With the shiny new proc sysctls we
> can test all 3 kernel configurations in one shot. Since we test 3
> different kernel configurations naturally some of these won't have the
> features needed, so that failure should be treated as non-fatal to allow
> the chain of other tests to continue.
>
> This issue was a regression due to commit a6a9be9270c87 ("selftests:
> firmware: return Kselftest Skip code for skipped tests") by Shuah for
> the verify_reqs(). We need to treat this as a non-fatal / don't skip
> return value.
>
> The following would fix this chaining issue:
>
> diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh
> index 6c5f1b2ffb74..1cbb12e284a6 100755
> --- a/tools/testing/selftests/firmware/fw_lib.sh
> +++ b/tools/testing/selftests/firmware/fw_lib.sh
> @@ -91,7 +91,7 @@ verify_reqs()
>         if [ "$TEST_REQS_FW_SYSFS_FALLBACK" = "yes" ]; then
>                 if [ ! "$HAS_FW_LOADER_USER_HELPER" = "yes" ]; then
>                         echo "usermode helper disabled so ignoring test"
> -                       exit $ksft_skip
> +                       exit 0
>                 fi
>         fi
>  }
>
> However its not clear to me if instead we want some new special return
> value for selftests so that the framework can detect an that an error
> is non-fatal, and can continue. This is a tricky situation given the
> script, existing upstream kernel module, are aware of such emulation
> hacks via sysctl, but knowledge of this is not obvious to selftests.
>
> Shuah, how do you suggest we handle this corner case? If you are OK
> with the above hunk for now I can send a fix for it. In either case
> this commit was added on v4.18, so the fix would be a stable fix.

In lieu of any suggestion I'm going to request we revert this commit
and send the above fix.

 Luis

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[mcgrof]

On Thu, Nov 29, 2018 at 8:31 PM Luis Chamberlain <mcgrof at kernel.org> wrote:
>
> On Mon, Nov 26, 2018 at 09:12:16PM -0600, Dan Rue wrote:
> > CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> > Without it, fw_fallback.sh fails with 'usermode helper disabled so
> > ignoring test'. Enable the config in selftest so that it gets built by
> > default.
> >
> > Signed-off-by: Dan Rue <dan.rue at linaro.org>
> > ---
> >  tools/testing/selftests/firmware/config | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> > index bf634dda0720..913a25a4a32b 100644
> > --- a/tools/testing/selftests/firmware/config
> > +++ b/tools/testing/selftests/firmware/config
> > @@ -1,5 +1,6 @@
> >  CONFIG_TEST_FIRMWARE=y
> >  CONFIG_FW_LOADER=y
> >  CONFIG_FW_LOADER_USER_HELPER=y
> > +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
> >  CONFIG_IKCONFIG=y
> >  CONFIG_IKCONFIG_PROC=y
>
> NACK -- the point of the changes was to *allow* us to mimic such
> configuration through a proc sysctl knob.
>
> You aren forcing CONFIG_FW_LOADER_USER_HELPER_FALLBACK but just having
> CONFIG_FW_LOADER_USER_HELPER suffices to emulate the_FALLBACK
> functionality.

Dan, again, you broke the whole point to the amount of work that went
into emulating testing. As such anyone testing their changes would
yield incorrect results.

> The issue here seems to be that *all* tests fail once a configuration is
> found which is not suitable a tests. With the shiny new proc sysctls we
> can test all 3 kernel configurations in one shot. Since we test 3
> different kernel configurations naturally some of these won't have the
> features needed, so that failure should be treated as non-fatal to allow
> the chain of other tests to continue.
>
> This issue was a regression due to commit a6a9be9270c87 ("selftests:
> firmware: return Kselftest Skip code for skipped tests") by Shuah for
> the verify_reqs(). We need to treat this as a non-fatal / don't skip
> return value.
>
> The following would fix this chaining issue:
>
> diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh
> index 6c5f1b2ffb74..1cbb12e284a6 100755
> --- a/tools/testing/selftests/firmware/fw_lib.sh
> +++ b/tools/testing/selftests/firmware/fw_lib.sh
> @@ -91,7 +91,7 @@ verify_reqs()
>         if [ "$TEST_REQS_FW_SYSFS_FALLBACK" = "yes" ]; then
>                 if [ ! "$HAS_FW_LOADER_USER_HELPER" = "yes" ]; then
>                         echo "usermode helper disabled so ignoring test"
> -                       exit $ksft_skip
> +                       exit 0
>                 fi
>         fi
>  }
>
> However its not clear to me if instead we want some new special return
> value for selftests so that the framework can detect an that an error
> is non-fatal, and can continue. This is a tricky situation given the
> script, existing upstream kernel module, are aware of such emulation
> hacks via sysctl, but knowledge of this is not obvious to selftests.
>
> Shuah, how do you suggest we handle this corner case? If you are OK
> with the above hunk for now I can send a fix for it. In either case
> this commit was added on v4.18, so the fix would be a stable fix.

In lieu of any suggestion I'm going to request we revert this commit
and send the above fix.

 Luis

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[Luis Chamberlain]

On Thu, Nov 29, 2018@8:31 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> On Mon, Nov 26, 2018@09:12:16PM -0600, Dan Rue wrote:
> > CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> > Without it, fw_fallback.sh fails with 'usermode helper disabled so
> > ignoring test'. Enable the config in selftest so that it gets built by
> > default.
> >
> > Signed-off-by: Dan Rue <dan.rue at linaro.org>
> > ---
> >  tools/testing/selftests/firmware/config | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> > index bf634dda0720..913a25a4a32b 100644
> > --- a/tools/testing/selftests/firmware/config
> > +++ b/tools/testing/selftests/firmware/config
> > @@ -1,5 +1,6 @@
> >  CONFIG_TEST_FIRMWARE=y
> >  CONFIG_FW_LOADER=y
> >  CONFIG_FW_LOADER_USER_HELPER=y
> > +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
> >  CONFIG_IKCONFIG=y
> >  CONFIG_IKCONFIG_PROC=y
>
> NACK -- the point of the changes was to *allow* us to mimic such
> configuration through a proc sysctl knob.
>
> You aren forcing CONFIG_FW_LOADER_USER_HELPER_FALLBACK but just having
> CONFIG_FW_LOADER_USER_HELPER suffices to emulate the_FALLBACK
> functionality.

Dan, again, you broke the whole point to the amount of work that went
into emulating testing. As such anyone testing their changes would
yield incorrect results.

> The issue here seems to be that *all* tests fail once a configuration is
> found which is not suitable a tests. With the shiny new proc sysctls we
> can test all 3 kernel configurations in one shot. Since we test 3
> different kernel configurations naturally some of these won't have the
> features needed, so that failure should be treated as non-fatal to allow
> the chain of other tests to continue.
>
> This issue was a regression due to commit a6a9be9270c87 ("selftests:
> firmware: return Kselftest Skip code for skipped tests") by Shuah for
> the verify_reqs(). We need to treat this as a non-fatal / don't skip
> return value.
>
> The following would fix this chaining issue:
>
> diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh
> index 6c5f1b2ffb74..1cbb12e284a6 100755
> --- a/tools/testing/selftests/firmware/fw_lib.sh
> +++ b/tools/testing/selftests/firmware/fw_lib.sh
> @@ -91,7 +91,7 @@ verify_reqs()
>         if [ "$TEST_REQS_FW_SYSFS_FALLBACK" = "yes" ]; then
>                 if [ ! "$HAS_FW_LOADER_USER_HELPER" = "yes" ]; then
>                         echo "usermode helper disabled so ignoring test"
> -                       exit $ksft_skip
> +                       exit 0
>                 fi
>         fi
>  }
>
> However its not clear to me if instead we want some new special return
> value for selftests so that the framework can detect an that an error
> is non-fatal, and can continue. This is a tricky situation given the
> script, existing upstream kernel module, are aware of such emulation
> hacks via sysctl, but knowledge of this is not obvious to selftests.
>
> Shuah, how do you suggest we handle this corner case? If you are OK
> with the above hunk for now I can send a fix for it. In either case
> this commit was added on v4.18, so the fix would be a stable fix.

In lieu of any suggestion I'm going to request we revert this commit
and send the above fix.

 Luis

Re: [PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[Dan Rue]

On Mon, Feb 04, 2019 at 05:39:57PM -0600, Luis Chamberlain wrote:
> On Thu, Nov 29, 2018 at 8:31 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > On Mon, Nov 26, 2018 at 09:12:16PM -0600, Dan Rue wrote:
> > > CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> > > Without it, fw_fallback.sh fails with 'usermode helper disabled so
> > > ignoring test'. Enable the config in selftest so that it gets built by
> > > default.
> > >
> > > Signed-off-by: Dan Rue <dan.rue@linaro.org>
> > > ---
> > >  tools/testing/selftests/firmware/config | 1 +
> > >  1 file changed, 1 insertion(+)
> > >
> > > diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> > > index bf634dda0720..913a25a4a32b 100644
> > > --- a/tools/testing/selftests/firmware/config
> > > +++ b/tools/testing/selftests/firmware/config
> > > @@ -1,5 +1,6 @@
> > >  CONFIG_TEST_FIRMWARE=y
> > >  CONFIG_FW_LOADER=y
> > >  CONFIG_FW_LOADER_USER_HELPER=y
> > > +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
> > >  CONFIG_IKCONFIG=y
> > >  CONFIG_IKCONFIG_PROC=y
> >
> > NACK -- the point of the changes was to *allow* us to mimic such
> > configuration through a proc sysctl knob.
> >
> > You aren forcing CONFIG_FW_LOADER_USER_HELPER_FALLBACK but just having
> > CONFIG_FW_LOADER_USER_HELPER suffices to emulate the_FALLBACK
> > functionality.
> 
> Dan, again, you broke the whole point to the amount of work that went
> into emulating testing. As such anyone testing their changes would
> yield incorrect results.
> 
> > The issue here seems to be that *all* tests fail once a configuration is
> > found which is not suitable a tests. With the shiny new proc sysctls we
> > can test all 3 kernel configurations in one shot. Since we test 3
> > different kernel configurations naturally some of these won't have the
> > features needed, so that failure should be treated as non-fatal to allow
> > the chain of other tests to continue.
> >
> > This issue was a regression due to commit a6a9be9270c87 ("selftests:
> > firmware: return Kselftest Skip code for skipped tests") by Shuah for
> > the verify_reqs(). We need to treat this as a non-fatal / don't skip
> > return value.
> >
> > The following would fix this chaining issue:
> >
> > diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh
> > index 6c5f1b2ffb74..1cbb12e284a6 100755
> > --- a/tools/testing/selftests/firmware/fw_lib.sh
> > +++ b/tools/testing/selftests/firmware/fw_lib.sh
> > @@ -91,7 +91,7 @@ verify_reqs()
> >         if [ "$TEST_REQS_FW_SYSFS_FALLBACK" = "yes" ]; then
> >                 if [ ! "$HAS_FW_LOADER_USER_HELPER" = "yes" ]; then
> >                         echo "usermode helper disabled so ignoring test"
> > -                       exit $ksft_skip
> > +                       exit 0
> >                 fi
> >         fi
> >  }
> >
> > However its not clear to me if instead we want some new special return
> > value for selftests so that the framework can detect an that an error
> > is non-fatal, and can continue. This is a tricky situation given the
> > script, existing upstream kernel module, are aware of such emulation
> > hacks via sysctl, but knowledge of this is not obvious to selftests.
> >
> > Shuah, how do you suggest we handle this corner case? If you are OK
> > with the above hunk for now I can send a fix for it. In either case
> > this commit was added on v4.18, so the fix would be a stable fix.
> 
> In lieu of any suggestion I'm going to request we revert this commit
> and send the above fix.

Sorry, I didn't realize this was waiting on me. I agree with all of your
feedback. Please revert 7492902e8d22 ("selftests: firmware: add
CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config") and add my Acked-by to
the proposed fix above.

Shuah, do I need to send a patch for that revert?

It would be nice if there were a way (maybe there is?) to let each of
the individual tests be exposed and run by run_kselftest.sh so that each
test gets its own proper pass/skip/fail. It could be done in this case
by making fw_run_tests.sh look more like run_kselftest.sh (running each
test in a subshell and capturing its exit code), but that starts to get
a bit fragile and ugly, too.

Dan

> 
>  Luis

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[dan.rue]

On Mon, Feb 04, 2019 at 05:39:57PM -0600, Luis Chamberlain wrote:
> On Thu, Nov 29, 2018 at 8:31 PM Luis Chamberlain <mcgrof at kernel.org> wrote:
> >
> > On Mon, Nov 26, 2018 at 09:12:16PM -0600, Dan Rue wrote:
> > > CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> > > Without it, fw_fallback.sh fails with 'usermode helper disabled so
> > > ignoring test'. Enable the config in selftest so that it gets built by
> > > default.
> > >
> > > Signed-off-by: Dan Rue <dan.rue at linaro.org>
> > > ---
> > >  tools/testing/selftests/firmware/config | 1 +
> > >  1 file changed, 1 insertion(+)
> > >
> > > diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> > > index bf634dda0720..913a25a4a32b 100644
> > > --- a/tools/testing/selftests/firmware/config
> > > +++ b/tools/testing/selftests/firmware/config
> > > @@ -1,5 +1,6 @@
> > >  CONFIG_TEST_FIRMWARE=y
> > >  CONFIG_FW_LOADER=y
> > >  CONFIG_FW_LOADER_USER_HELPER=y
> > > +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
> > >  CONFIG_IKCONFIG=y
> > >  CONFIG_IKCONFIG_PROC=y
> >
> > NACK -- the point of the changes was to *allow* us to mimic such
> > configuration through a proc sysctl knob.
> >
> > You aren forcing CONFIG_FW_LOADER_USER_HELPER_FALLBACK but just having
> > CONFIG_FW_LOADER_USER_HELPER suffices to emulate the_FALLBACK
> > functionality.
> 
> Dan, again, you broke the whole point to the amount of work that went
> into emulating testing. As such anyone testing their changes would
> yield incorrect results.
> 
> > The issue here seems to be that *all* tests fail once a configuration is
> > found which is not suitable a tests. With the shiny new proc sysctls we
> > can test all 3 kernel configurations in one shot. Since we test 3
> > different kernel configurations naturally some of these won't have the
> > features needed, so that failure should be treated as non-fatal to allow
> > the chain of other tests to continue.
> >
> > This issue was a regression due to commit a6a9be9270c87 ("selftests:
> > firmware: return Kselftest Skip code for skipped tests") by Shuah for
> > the verify_reqs(). We need to treat this as a non-fatal / don't skip
> > return value.
> >
> > The following would fix this chaining issue:
> >
> > diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh
> > index 6c5f1b2ffb74..1cbb12e284a6 100755
> > --- a/tools/testing/selftests/firmware/fw_lib.sh
> > +++ b/tools/testing/selftests/firmware/fw_lib.sh
> > @@ -91,7 +91,7 @@ verify_reqs()
> >         if [ "$TEST_REQS_FW_SYSFS_FALLBACK" = "yes" ]; then
> >                 if [ ! "$HAS_FW_LOADER_USER_HELPER" = "yes" ]; then
> >                         echo "usermode helper disabled so ignoring test"
> > -                       exit $ksft_skip
> > +                       exit 0
> >                 fi
> >         fi
> >  }
> >
> > However its not clear to me if instead we want some new special return
> > value for selftests so that the framework can detect an that an error
> > is non-fatal, and can continue. This is a tricky situation given the
> > script, existing upstream kernel module, are aware of such emulation
> > hacks via sysctl, but knowledge of this is not obvious to selftests.
> >
> > Shuah, how do you suggest we handle this corner case? If you are OK
> > with the above hunk for now I can send a fix for it. In either case
> > this commit was added on v4.18, so the fix would be a stable fix.
> 
> In lieu of any suggestion I'm going to request we revert this commit
> and send the above fix.

Sorry, I didn't realize this was waiting on me. I agree with all of your
feedback. Please revert 7492902e8d22 ("selftests: firmware: add
CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config") and add my Acked-by to
the proposed fix above.

Shuah, do I need to send a patch for that revert?

It would be nice if there were a way (maybe there is?) to let each of
the individual tests be exposed and run by run_kselftest.sh so that each
test gets its own proper pass/skip/fail. It could be done in this case
by making fw_run_tests.sh look more like run_kselftest.sh (running each
test in a subshell and capturing its exit code), but that starts to get
a bit fragile and ugly, too.

Dan

> 
>  Luis

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[Dan Rue]

On Mon, Feb 04, 2019@05:39:57PM -0600, Luis Chamberlain wrote:
> On Thu, Nov 29, 2018@8:31 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > On Mon, Nov 26, 2018@09:12:16PM -0600, Dan Rue wrote:
> > > CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> > > Without it, fw_fallback.sh fails with 'usermode helper disabled so
> > > ignoring test'. Enable the config in selftest so that it gets built by
> > > default.
> > >
> > > Signed-off-by: Dan Rue <dan.rue at linaro.org>
> > > ---
> > >  tools/testing/selftests/firmware/config | 1 +
> > >  1 file changed, 1 insertion(+)
> > >
> > > diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> > > index bf634dda0720..913a25a4a32b 100644
> > > --- a/tools/testing/selftests/firmware/config
> > > +++ b/tools/testing/selftests/firmware/config
> > > @@ -1,5 +1,6 @@
> > >  CONFIG_TEST_FIRMWARE=y
> > >  CONFIG_FW_LOADER=y
> > >  CONFIG_FW_LOADER_USER_HELPER=y
> > > +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
> > >  CONFIG_IKCONFIG=y
> > >  CONFIG_IKCONFIG_PROC=y
> >
> > NACK -- the point of the changes was to *allow* us to mimic such
> > configuration through a proc sysctl knob.
> >
> > You aren forcing CONFIG_FW_LOADER_USER_HELPER_FALLBACK but just having
> > CONFIG_FW_LOADER_USER_HELPER suffices to emulate the_FALLBACK
> > functionality.
> 
> Dan, again, you broke the whole point to the amount of work that went
> into emulating testing. As such anyone testing their changes would
> yield incorrect results.
> 
> > The issue here seems to be that *all* tests fail once a configuration is
> > found which is not suitable a tests. With the shiny new proc sysctls we
> > can test all 3 kernel configurations in one shot. Since we test 3
> > different kernel configurations naturally some of these won't have the
> > features needed, so that failure should be treated as non-fatal to allow
> > the chain of other tests to continue.
> >
> > This issue was a regression due to commit a6a9be9270c87 ("selftests:
> > firmware: return Kselftest Skip code for skipped tests") by Shuah for
> > the verify_reqs(). We need to treat this as a non-fatal / don't skip
> > return value.
> >
> > The following would fix this chaining issue:
> >
> > diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh
> > index 6c5f1b2ffb74..1cbb12e284a6 100755
> > --- a/tools/testing/selftests/firmware/fw_lib.sh
> > +++ b/tools/testing/selftests/firmware/fw_lib.sh
> > @@ -91,7 +91,7 @@ verify_reqs()
> >         if [ "$TEST_REQS_FW_SYSFS_FALLBACK" = "yes" ]; then
> >                 if [ ! "$HAS_FW_LOADER_USER_HELPER" = "yes" ]; then
> >                         echo "usermode helper disabled so ignoring test"
> > -                       exit $ksft_skip
> > +                       exit 0
> >                 fi
> >         fi
> >  }
> >
> > However its not clear to me if instead we want some new special return
> > value for selftests so that the framework can detect an that an error
> > is non-fatal, and can continue. This is a tricky situation given the
> > script, existing upstream kernel module, are aware of such emulation
> > hacks via sysctl, but knowledge of this is not obvious to selftests.
> >
> > Shuah, how do you suggest we handle this corner case? If you are OK
> > with the above hunk for now I can send a fix for it. In either case
> > this commit was added on v4.18, so the fix would be a stable fix.
> 
> In lieu of any suggestion I'm going to request we revert this commit
> and send the above fix.

Sorry, I didn't realize this was waiting on me. I agree with all of your
feedback. Please revert 7492902e8d22 ("selftests: firmware: add
CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config") and add my Acked-by to
the proposed fix above.

Shuah, do I need to send a patch for that revert?

It would be nice if there were a way (maybe there is?) to let each of
the individual tests be exposed and run by run_kselftest.sh so that each
test gets its own proper pass/skip/fail. It could be done in this case
by making fw_run_tests.sh look more like run_kselftest.sh (running each
test in a subshell and capturing its exit code), but that starts to get
a bit fragile and ugly, too.

Dan

> 
>  Luis

Re: [PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[Luis Chamberlain]

On Mon, Feb 04, 2019 at 08:41:50PM -0600, Dan Rue wrote:
> On Mon, Feb 04, 2019 at 05:39:57PM -0600, Luis Chamberlain wrote:
> > On Thu, Nov 29, 2018 at 8:31 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > >
> > > On Mon, Nov 26, 2018 at 09:12:16PM -0600, Dan Rue wrote:
> > > > CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> > > > Without it, fw_fallback.sh fails with 'usermode helper disabled so
> > > > ignoring test'. Enable the config in selftest so that it gets built by
> > > > default.
> > > >
> > > > Signed-off-by: Dan Rue <dan.rue@linaro.org>
> > > > ---
> > > >  tools/testing/selftests/firmware/config | 1 +
> > > >  1 file changed, 1 insertion(+)
> > > >
> > > > diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> > > > index bf634dda0720..913a25a4a32b 100644
> > > > --- a/tools/testing/selftests/firmware/config
> > > > +++ b/tools/testing/selftests/firmware/config
> > > > @@ -1,5 +1,6 @@
> > > >  CONFIG_TEST_FIRMWARE=y
> > > >  CONFIG_FW_LOADER=y
> > > >  CONFIG_FW_LOADER_USER_HELPER=y
> > > > +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
> > > >  CONFIG_IKCONFIG=y
> > > >  CONFIG_IKCONFIG_PROC=y
> > >
> > > NACK -- the point of the changes was to *allow* us to mimic such
> > > configuration through a proc sysctl knob.
> > >
> > > You aren forcing CONFIG_FW_LOADER_USER_HELPER_FALLBACK but just having
> > > CONFIG_FW_LOADER_USER_HELPER suffices to emulate the_FALLBACK
> > > functionality.
> > 
> > Dan, again, you broke the whole point to the amount of work that went
> > into emulating testing. As such anyone testing their changes would
> > yield incorrect results.
> > 
> > > The issue here seems to be that *all* tests fail once a configuration is
> > > found which is not suitable a tests. With the shiny new proc sysctls we
> > > can test all 3 kernel configurations in one shot. Since we test 3
> > > different kernel configurations naturally some of these won't have the
> > > features needed, so that failure should be treated as non-fatal to allow
> > > the chain of other tests to continue.
> > >
> > > This issue was a regression due to commit a6a9be9270c87 ("selftests:
> > > firmware: return Kselftest Skip code for skipped tests") by Shuah for
> > > the verify_reqs(). We need to treat this as a non-fatal / don't skip
> > > return value.
> > >
> > > The following would fix this chaining issue:
> > >
> > > diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh
> > > index 6c5f1b2ffb74..1cbb12e284a6 100755
> > > --- a/tools/testing/selftests/firmware/fw_lib.sh
> > > +++ b/tools/testing/selftests/firmware/fw_lib.sh
> > > @@ -91,7 +91,7 @@ verify_reqs()
> > >         if [ "$TEST_REQS_FW_SYSFS_FALLBACK" = "yes" ]; then
> > >                 if [ ! "$HAS_FW_LOADER_USER_HELPER" = "yes" ]; then
> > >                         echo "usermode helper disabled so ignoring test"
> > > -                       exit $ksft_skip
> > > +                       exit 0
> > >                 fi
> > >         fi
> > >  }
> > >
> > > However its not clear to me if instead we want some new special return
> > > value for selftests so that the framework can detect an that an error
> > > is non-fatal, and can continue. This is a tricky situation given the
> > > script, existing upstream kernel module, are aware of such emulation
> > > hacks via sysctl, but knowledge of this is not obvious to selftests.
> > >
> > > Shuah, how do you suggest we handle this corner case? If you are OK
> > > with the above hunk for now I can send a fix for it. In either case
> > > this commit was added on v4.18, so the fix would be a stable fix.
> > 
> > In lieu of any suggestion I'm going to request we revert this commit
> > and send the above fix.
> 
> Sorry, I didn't realize this was waiting on me. I agree with all of your
> feedback. Please revert 7492902e8d22 ("selftests: firmware: add
> CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config") and add my Acked-by to
> the proposed fix above.
> 
> Shuah, do I need to send a patch for that revert?

I can send the revert and the subsequent fix.

  Luis

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[mcgrof]

On Mon, Feb 04, 2019 at 08:41:50PM -0600, Dan Rue wrote:
> On Mon, Feb 04, 2019 at 05:39:57PM -0600, Luis Chamberlain wrote:
> > On Thu, Nov 29, 2018 at 8:31 PM Luis Chamberlain <mcgrof at kernel.org> wrote:
> > >
> > > On Mon, Nov 26, 2018 at 09:12:16PM -0600, Dan Rue wrote:
> > > > CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> > > > Without it, fw_fallback.sh fails with 'usermode helper disabled so
> > > > ignoring test'. Enable the config in selftest so that it gets built by
> > > > default.
> > > >
> > > > Signed-off-by: Dan Rue <dan.rue at linaro.org>
> > > > ---
> > > >  tools/testing/selftests/firmware/config | 1 +
> > > >  1 file changed, 1 insertion(+)
> > > >
> > > > diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> > > > index bf634dda0720..913a25a4a32b 100644
> > > > --- a/tools/testing/selftests/firmware/config
> > > > +++ b/tools/testing/selftests/firmware/config
> > > > @@ -1,5 +1,6 @@
> > > >  CONFIG_TEST_FIRMWARE=y
> > > >  CONFIG_FW_LOADER=y
> > > >  CONFIG_FW_LOADER_USER_HELPER=y
> > > > +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
> > > >  CONFIG_IKCONFIG=y
> > > >  CONFIG_IKCONFIG_PROC=y
> > >
> > > NACK -- the point of the changes was to *allow* us to mimic such
> > > configuration through a proc sysctl knob.
> > >
> > > You aren forcing CONFIG_FW_LOADER_USER_HELPER_FALLBACK but just having
> > > CONFIG_FW_LOADER_USER_HELPER suffices to emulate the_FALLBACK
> > > functionality.
> > 
> > Dan, again, you broke the whole point to the amount of work that went
> > into emulating testing. As such anyone testing their changes would
> > yield incorrect results.
> > 
> > > The issue here seems to be that *all* tests fail once a configuration is
> > > found which is not suitable a tests. With the shiny new proc sysctls we
> > > can test all 3 kernel configurations in one shot. Since we test 3
> > > different kernel configurations naturally some of these won't have the
> > > features needed, so that failure should be treated as non-fatal to allow
> > > the chain of other tests to continue.
> > >
> > > This issue was a regression due to commit a6a9be9270c87 ("selftests:
> > > firmware: return Kselftest Skip code for skipped tests") by Shuah for
> > > the verify_reqs(). We need to treat this as a non-fatal / don't skip
> > > return value.
> > >
> > > The following would fix this chaining issue:
> > >
> > > diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh
> > > index 6c5f1b2ffb74..1cbb12e284a6 100755
> > > --- a/tools/testing/selftests/firmware/fw_lib.sh
> > > +++ b/tools/testing/selftests/firmware/fw_lib.sh
> > > @@ -91,7 +91,7 @@ verify_reqs()
> > >         if [ "$TEST_REQS_FW_SYSFS_FALLBACK" = "yes" ]; then
> > >                 if [ ! "$HAS_FW_LOADER_USER_HELPER" = "yes" ]; then
> > >                         echo "usermode helper disabled so ignoring test"
> > > -                       exit $ksft_skip
> > > +                       exit 0
> > >                 fi
> > >         fi
> > >  }
> > >
> > > However its not clear to me if instead we want some new special return
> > > value for selftests so that the framework can detect an that an error
> > > is non-fatal, and can continue. This is a tricky situation given the
> > > script, existing upstream kernel module, are aware of such emulation
> > > hacks via sysctl, but knowledge of this is not obvious to selftests.
> > >
> > > Shuah, how do you suggest we handle this corner case? If you are OK
> > > with the above hunk for now I can send a fix for it. In either case
> > > this commit was added on v4.18, so the fix would be a stable fix.
> > 
> > In lieu of any suggestion I'm going to request we revert this commit
> > and send the above fix.
> 
> Sorry, I didn't realize this was waiting on me. I agree with all of your
> feedback. Please revert 7492902e8d22 ("selftests: firmware: add
> CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config") and add my Acked-by to
> the proposed fix above.
> 
> Shuah, do I need to send a patch for that revert?

I can send the revert and the subsequent fix.

  Luis

[PATCH 2/2] selftests: firmware: add CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config

[Luis Chamberlain]

On Mon, Feb 04, 2019@08:41:50PM -0600, Dan Rue wrote:
> On Mon, Feb 04, 2019@05:39:57PM -0600, Luis Chamberlain wrote:
> > On Thu, Nov 29, 2018@8:31 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> > >
> > > On Mon, Nov 26, 2018@09:12:16PM -0600, Dan Rue wrote:
> > > > CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y is required for fw_fallback.sh.
> > > > Without it, fw_fallback.sh fails with 'usermode helper disabled so
> > > > ignoring test'. Enable the config in selftest so that it gets built by
> > > > default.
> > > >
> > > > Signed-off-by: Dan Rue <dan.rue at linaro.org>
> > > > ---
> > > >  tools/testing/selftests/firmware/config | 1 +
> > > >  1 file changed, 1 insertion(+)
> > > >
> > > > diff --git a/tools/testing/selftests/firmware/config b/tools/testing/selftests/firmware/config
> > > > index bf634dda0720..913a25a4a32b 100644
> > > > --- a/tools/testing/selftests/firmware/config
> > > > +++ b/tools/testing/selftests/firmware/config
> > > > @@ -1,5 +1,6 @@
> > > >  CONFIG_TEST_FIRMWARE=y
> > > >  CONFIG_FW_LOADER=y
> > > >  CONFIG_FW_LOADER_USER_HELPER=y
> > > > +CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
> > > >  CONFIG_IKCONFIG=y
> > > >  CONFIG_IKCONFIG_PROC=y
> > >
> > > NACK -- the point of the changes was to *allow* us to mimic such
> > > configuration through a proc sysctl knob.
> > >
> > > You aren forcing CONFIG_FW_LOADER_USER_HELPER_FALLBACK but just having
> > > CONFIG_FW_LOADER_USER_HELPER suffices to emulate the_FALLBACK
> > > functionality.
> > 
> > Dan, again, you broke the whole point to the amount of work that went
> > into emulating testing. As such anyone testing their changes would
> > yield incorrect results.
> > 
> > > The issue here seems to be that *all* tests fail once a configuration is
> > > found which is not suitable a tests. With the shiny new proc sysctls we
> > > can test all 3 kernel configurations in one shot. Since we test 3
> > > different kernel configurations naturally some of these won't have the
> > > features needed, so that failure should be treated as non-fatal to allow
> > > the chain of other tests to continue.
> > >
> > > This issue was a regression due to commit a6a9be9270c87 ("selftests:
> > > firmware: return Kselftest Skip code for skipped tests") by Shuah for
> > > the verify_reqs(). We need to treat this as a non-fatal / don't skip
> > > return value.
> > >
> > > The following would fix this chaining issue:
> > >
> > > diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh
> > > index 6c5f1b2ffb74..1cbb12e284a6 100755
> > > --- a/tools/testing/selftests/firmware/fw_lib.sh
> > > +++ b/tools/testing/selftests/firmware/fw_lib.sh
> > > @@ -91,7 +91,7 @@ verify_reqs()
> > >         if [ "$TEST_REQS_FW_SYSFS_FALLBACK" = "yes" ]; then
> > >                 if [ ! "$HAS_FW_LOADER_USER_HELPER" = "yes" ]; then
> > >                         echo "usermode helper disabled so ignoring test"
> > > -                       exit $ksft_skip
> > > +                       exit 0
> > >                 fi
> > >         fi
> > >  }
> > >
> > > However its not clear to me if instead we want some new special return
> > > value for selftests so that the framework can detect an that an error
> > > is non-fatal, and can continue. This is a tricky situation given the
> > > script, existing upstream kernel module, are aware of such emulation
> > > hacks via sysctl, but knowledge of this is not obvious to selftests.
> > >
> > > Shuah, how do you suggest we handle this corner case? If you are OK
> > > with the above hunk for now I can send a fix for it. In either case
> > > this commit was added on v4.18, so the fix would be a stable fix.
> > 
> > In lieu of any suggestion I'm going to request we revert this commit
> > and send the above fix.
> 
> Sorry, I didn't realize this was waiting on me. I agree with all of your
> feedback. Please revert 7492902e8d22 ("selftests: firmware: add
> CONFIG_FW_LOADER_USER_HELPER_FALLBACK to config") and add my Acked-by to
> the proposed fix above.
> 
> Shuah, do I need to send a patch for that revert?

I can send the revert and the subsequent fix.

  Luis

Re: [PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[Luis Chamberlain]

On Wed, Dec 5, 2018 at 2:43 PM Dan Rue <dan.rue@linaro.org> wrote:
>
> On Thu, Nov 29, 2018 at 06:37:32PM -0800, Luis Chamberlain wrote:
> > On Mon, Nov 26, 2018 at 09:12:15PM -0600, Dan Rue wrote:
> > > diff -Z is used to trim the trailing whitespace when comparing the
> > > loaded firmware file with the source firmware file. However, per the
> > > comment in the source code, -Z should not be necessary. In testing, the
> > > input and output files are identical.
> > >
> > > Additionally, -Z is not a standard option and is not available in
> > > environments such as busybox. When -Z is not supported, diff fails with
> > > a usage error, which is suppressed, but then causes read_firmwares() to
> > > exit with a false failure message.
> >
> > NACK -- this breaks testing on debian:
> >
> > Testing with the file present...
> > Batched request_firmware() try #1: Files
> > /tmp/tmp.8GWkoSo5jZ/test-firmware.bin and
> > /sys/devices/virtual/misc/test_firmware/read_firmware differ
> > request #0: firmware was not loaded
> >
> > Please add a quirks check, enable it by default, and remove it for
> > Busybox.
>
> Thanks for the review. Shuah, can you please drop this one?

So much for review. This patch was still merged. I'll have to request
this to be reverted now too.

  Luis

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[mcgrof]

On Wed, Dec 5, 2018 at 2:43 PM Dan Rue <dan.rue at linaro.org> wrote:
>
> On Thu, Nov 29, 2018 at 06:37:32PM -0800, Luis Chamberlain wrote:
> > On Mon, Nov 26, 2018 at 09:12:15PM -0600, Dan Rue wrote:
> > > diff -Z is used to trim the trailing whitespace when comparing the
> > > loaded firmware file with the source firmware file. However, per the
> > > comment in the source code, -Z should not be necessary. In testing, the
> > > input and output files are identical.
> > >
> > > Additionally, -Z is not a standard option and is not available in
> > > environments such as busybox. When -Z is not supported, diff fails with
> > > a usage error, which is suppressed, but then causes read_firmwares() to
> > > exit with a false failure message.
> >
> > NACK -- this breaks testing on debian:
> >
> > Testing with the file present...
> > Batched request_firmware() try #1: Files
> > /tmp/tmp.8GWkoSo5jZ/test-firmware.bin and
> > /sys/devices/virtual/misc/test_firmware/read_firmware differ
> > request #0: firmware was not loaded
> >
> > Please add a quirks check, enable it by default, and remove it for
> > Busybox.
>
> Thanks for the review. Shuah, can you please drop this one?

So much for review. This patch was still merged. I'll have to request
this to be reverted now too.

  Luis

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[Luis Chamberlain]

On Wed, Dec 5, 2018@2:43 PM Dan Rue <dan.rue@linaro.org> wrote:
>
> On Thu, Nov 29, 2018@06:37:32PM -0800, Luis Chamberlain wrote:
> > On Mon, Nov 26, 2018@09:12:15PM -0600, Dan Rue wrote:
> > > diff -Z is used to trim the trailing whitespace when comparing the
> > > loaded firmware file with the source firmware file. However, per the
> > > comment in the source code, -Z should not be necessary. In testing, the
> > > input and output files are identical.
> > >
> > > Additionally, -Z is not a standard option and is not available in
> > > environments such as busybox. When -Z is not supported, diff fails with
> > > a usage error, which is suppressed, but then causes read_firmwares() to
> > > exit with a false failure message.
> >
> > NACK -- this breaks testing on debian:
> >
> > Testing with the file present...
> > Batched request_firmware() try #1: Files
> > /tmp/tmp.8GWkoSo5jZ/test-firmware.bin and
> > /sys/devices/virtual/misc/test_firmware/read_firmware differ
> > request #0: firmware was not loaded
> >
> > Please add a quirks check, enable it by default, and remove it for
> > Busybox.
>
> Thanks for the review. Shuah, can you please drop this one?

So much for review. This patch was still merged. I'll have to request
this to be reverted now too.

  Luis

Re: [PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[shuah]

On 2/7/19 11:20 AM, Luis Chamberlain wrote:
> On Wed, Dec 5, 2018 at 2:43 PM Dan Rue <dan.rue@linaro.org> wrote:
>>
>> On Thu, Nov 29, 2018 at 06:37:32PM -0800, Luis Chamberlain wrote:
>>> On Mon, Nov 26, 2018 at 09:12:15PM -0600, Dan Rue wrote:
>>>> diff -Z is used to trim the trailing whitespace when comparing the
>>>> loaded firmware file with the source firmware file. However, per the
>>>> comment in the source code, -Z should not be necessary. In testing, the
>>>> input and output files are identical.
>>>>
>>>> Additionally, -Z is not a standard option and is not available in
>>>> environments such as busybox. When -Z is not supported, diff fails with
>>>> a usage error, which is suppressed, but then causes read_firmwares() to
>>>> exit with a false failure message.
>>>
>>> NACK -- this breaks testing on debian:
>>>
>>> Testing with the file present...
>>> Batched request_firmware() try #1: Files
>>> /tmp/tmp.8GWkoSo5jZ/test-firmware.bin and
>>> /sys/devices/virtual/misc/test_firmware/read_firmware differ
>>> request #0: firmware was not loaded
>>>
>>> Please add a quirks check, enable it by default, and remove it for
>>> Busybox.
>>
>> Thanks for the review. Shuah, can you please drop this one?
> 
> So much for review. This patch was still merged. I'll have to request
> this to be reverted now too.
> 
>    Luis
> 

Sorry about this. I will pull the reverts in.

thanks,
-- Shuah

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[shuah]

On 2/7/19 11:20 AM, Luis Chamberlain wrote:
> On Wed, Dec 5, 2018 at 2:43 PM Dan Rue <dan.rue at linaro.org> wrote:
>>
>> On Thu, Nov 29, 2018 at 06:37:32PM -0800, Luis Chamberlain wrote:
>>> On Mon, Nov 26, 2018 at 09:12:15PM -0600, Dan Rue wrote:
>>>> diff -Z is used to trim the trailing whitespace when comparing the
>>>> loaded firmware file with the source firmware file. However, per the
>>>> comment in the source code, -Z should not be necessary. In testing, the
>>>> input and output files are identical.
>>>>
>>>> Additionally, -Z is not a standard option and is not available in
>>>> environments such as busybox. When -Z is not supported, diff fails with
>>>> a usage error, which is suppressed, but then causes read_firmwares() to
>>>> exit with a false failure message.
>>>
>>> NACK -- this breaks testing on debian:
>>>
>>> Testing with the file present...
>>> Batched request_firmware() try #1: Files
>>> /tmp/tmp.8GWkoSo5jZ/test-firmware.bin and
>>> /sys/devices/virtual/misc/test_firmware/read_firmware differ
>>> request #0: firmware was not loaded
>>>
>>> Please add a quirks check, enable it by default, and remove it for
>>> Busybox.
>>
>> Thanks for the review. Shuah, can you please drop this one?
> 
> So much for review. This patch was still merged. I'll have to request
> this to be reverted now too.
> 
>    Luis
> 

Sorry about this. I will pull the reverts in.

thanks,
-- Shuah

[PATCH 1/2] selftests: firmware: remove use of non-standard diff -Z option

[shuah]

On 2/7/19 11:20 AM, Luis Chamberlain wrote:
> On Wed, Dec 5, 2018@2:43 PM Dan Rue <dan.rue@linaro.org> wrote:
>>
>> On Thu, Nov 29, 2018@06:37:32PM -0800, Luis Chamberlain wrote:
>>> On Mon, Nov 26, 2018@09:12:15PM -0600, Dan Rue wrote:
>>>> diff -Z is used to trim the trailing whitespace when comparing the
>>>> loaded firmware file with the source firmware file. However, per the
>>>> comment in the source code, -Z should not be necessary. In testing, the
>>>> input and output files are identical.
>>>>
>>>> Additionally, -Z is not a standard option and is not available in
>>>> environments such as busybox. When -Z is not supported, diff fails with
>>>> a usage error, which is suppressed, but then causes read_firmwares() to
>>>> exit with a false failure message.
>>>
>>> NACK -- this breaks testing on debian:
>>>
>>> Testing with the file present...
>>> Batched request_firmware() try #1: Files
>>> /tmp/tmp.8GWkoSo5jZ/test-firmware.bin and
>>> /sys/devices/virtual/misc/test_firmware/read_firmware differ
>>> request #0: firmware was not loaded
>>>
>>> Please add a quirks check, enable it by default, and remove it for
>>> Busybox.
>>
>> Thanks for the review. Shuah, can you please drop this one?
> 
> So much for review. This patch was still merged. I'll have to request
> this to be reverted now too.
> 
>    Luis
> 

Sorry about this. I will pull the reverts in.

thanks,
-- Shuah