[PATCH 1/2] glibc: Exclude common code to build tests to glibc-tests.inc

[PATCH 1/2] glibc: Exclude common code to build tests to glibc-tests.inc

[?ukasz Majewski]

The common code to build tests has been excluded from glibc-testsuite
recipe to glibc-tests.inc

This code will be reused in the recipe necessary for providing glibc
tests executed with ptest framework.

Signed-off-by: Lukasz Majewski <lukma@denx.de>
---
 meta/recipes-core/glibc/glibc-tests.inc       | 32 ++++++++++++++++
 .../glibc/glibc-testsuite_2.34.bb             | 37 ++-----------------
 2 files changed, 36 insertions(+), 33 deletions(-)
 create mode 100644 meta/recipes-core/glibc/glibc-tests.inc

diff --git a/meta/recipes-core/glibc/glibc-tests.inc b/meta/recipes-core/glibc/glibc-tests.inc
new file mode 100644
index 0000000000..a8fe160a54
--- /dev/null
+++ b/meta/recipes-core/glibc/glibc-tests.inc
@@ -0,0 +1,32 @@
+EXCLUDE_FROM_WORLD = "1"
+
+# handle PN differences
+FILESEXTRAPATHS:prepend := "${THISDIR}/glibc:"
+
+# setup depends
+INHIBIT_DEFAULT_DEPS = ""
+
+python () {
+    libc = d.getVar("PREFERRED_PROVIDER_virtual/libc")
+    libclocale = d.getVar("PREFERRED_PROVIDER_virtual/libc-locale")
+    if libc != "glibc" or libclocale != "glibc-locale":
+        raise bb.parse.SkipRecipe("glibc-testsuite requires that virtual/libc is glibc")
+}
+
+DEPENDS += "glibc-locale libgcc gcc-runtime"
+
+# remove the initial depends
+DEPENDS:remove = "libgcc-initial"
+
+do_check[dirs] += "${B}"
+do_check () {
+    # clean out previous test results
+    oe_runmake tests-clean
+    # makefiles don't clean entirely (and also sometimes fails due to too many args)
+    find ${B} -type f -name "*.out" -delete
+    find ${B} -type f -name "*.test-result" -delete
+    find ${B}/catgets -name "*.cat" -delete
+    find ${B}/conform -name "symlist-*" -delete
+    [ ! -e ${B}/timezone/testdata ] || rm -rf ${B}/timezone/testdata
+}
+addtask do_check after do_compile
diff --git a/meta/recipes-core/glibc/glibc-testsuite_2.34.bb b/meta/recipes-core/glibc/glibc-testsuite_2.34.bb
index 72aa332171..e8ad2a938b 100644
--- a/meta/recipes-core/glibc/glibc-testsuite_2.34.bb
+++ b/meta/recipes-core/glibc/glibc-testsuite_2.34.bb
@@ -1,30 +1,12 @@
 require glibc_${PV}.bb
+require glibc-tests.inc
 
-EXCLUDE_FROM_WORLD = "1"
+inherit qemu
 
-# handle PN differences
-FILESEXTRAPATHS:prepend := "${THISDIR}/glibc:"
+SRC_URI += "file://check-test-wrapper"
 
 # strip provides
 PROVIDES = ""
-# setup depends
-INHIBIT_DEFAULT_DEPS = ""
-
-python () {
-    libc = d.getVar("PREFERRED_PROVIDER_virtual/libc")
-    libclocale = d.getVar("PREFERRED_PROVIDER_virtual/libc-locale")
-    if libc != "glibc" or libclocale != "glibc-locale":
-        raise bb.parse.SkipRecipe("glibc-testsuite requires that virtual/libc is glibc")
-}
-
-DEPENDS += "glibc-locale libgcc gcc-runtime"
-
-# remove the initial depends
-DEPENDS:remove = "libgcc-initial"
-
-inherit qemu
-
-SRC_URI += "file://check-test-wrapper"
 
 DEPENDS += "${@'qemu-native' if d.getVar('TOOLCHAIN_TEST_TARGET') == 'user' else ''}"
 
@@ -33,20 +15,10 @@ TOOLCHAIN_TEST_HOST ??= "localhost"
 TOOLCHAIN_TEST_HOST_USER ??= "root"
 TOOLCHAIN_TEST_HOST_PORT ??= "2222"
 
-do_check[dirs] += "${B}"
 do_check[nostamp] = "1"
-do_check () {
+do_check:append () {
     chmod 0755 ${WORKDIR}/check-test-wrapper
 
-    # clean out previous test results
-    oe_runmake tests-clean
-    # makefiles don't clean entirely (and also sometimes fails due to too many args)
-    find ${B} -type f -name "*.out" -delete
-    find ${B} -type f -name "*.test-result" -delete
-    find ${B}/catgets -name "*.cat" -delete
-    find ${B}/conform -name "symlist-*" -delete
-    [ ! -e ${B}/timezone/testdata ] || rm -rf ${B}/timezone/testdata
-
     oe_runmake -i \
         QEMU_SYSROOT="${RECIPE_SYSROOT}" \
         QEMU_OPTIONS="${@qemu_target_binary(d)} ${QEMU_OPTIONS}" \
@@ -56,7 +28,6 @@ do_check () {
         test-wrapper="${WORKDIR}/check-test-wrapper ${TOOLCHAIN_TEST_TARGET}" \
         check
 }
-addtask do_check after do_compile
 
 inherit nopackages
 deltask do_stash_locale
-- 
2.20.1

[PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[?ukasz Majewski]

This patch introduces new recipe - namely 'glibc-tests', which
builds and installs glibc test suite to OE/Yocto built image.

It reuses code from already available 'glibc-testsuite' recipe,
which is run with 'bitbake glibc-testsuite -c check' and uses qemu
to execute remotely (via SSH) tests on some emulated machine.

This recipe installs eligible tests on some rootfs image.
Afterwards, either all of them or only time related subset, those
tests can be executed on the real hardware, to facilitate validation
of this platform with for example Y2038 problem compliance.

By default all tests are executed, with 'ptest-runner glibc-tests'.
To test only time related subset - one needs to call:
cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests

To facilitate debugging, source files are provided by default with
the unstripped debugging symbols. Such approach would reduce the
already complex recipe (as it inherits base glibc one), so there
is no need to also install *-dbg and *-src packages.

Signed-off-by: Lukasz Majewski <lukma@denx.de>
---
 .../distro/include/ptest-packagelists.inc     |   1 +
 meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101 ++++++++++++++++++
 meta/recipes-core/glibc/glibc/run-ptest-all   |  25 +++++
 meta/recipes-core/glibc/glibc/run-ptest-time  |  30 ++++++
 4 files changed, 157 insertions(+)
 create mode 100644 meta/recipes-core/glibc/glibc-tests_2.34.bb
 create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all
 create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-time

diff --git a/meta/conf/distro/include/ptest-packagelists.inc b/meta/conf/distro/include/ptest-packagelists.inc
index 3872bdc942..e460ad1c8b 100644
--- a/meta/conf/distro/include/ptest-packagelists.inc
+++ b/meta/conf/distro/include/ptest-packagelists.inc
@@ -59,6 +59,7 @@ PTESTS_FAST = "\
     slang-ptest \
     wayland-ptest \
     zlib-ptest \
+    glibc-tests-ptest \
 "
 PTESTS_FAST:remove:mips64 = "qemu-ptest"
 PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
diff --git a/meta/recipes-core/glibc/glibc-tests_2.34.bb b/meta/recipes-core/glibc/glibc-tests_2.34.bb
new file mode 100644
index 0000000000..896809f700
--- /dev/null
+++ b/meta/recipes-core/glibc/glibc-tests_2.34.bb
@@ -0,0 +1,101 @@
+require glibc_${PV}.bb
+require glibc-tests.inc
+
+inherit ptest
+
+SRC_URI:append = " \
+	file://run-ptest-all file://run-ptest-time \
+"
+
+SUMMARY = "glibc tests to be run with ptest"
+
+# Erase some variables already set by glibc_${PV}
+python __anonymous() {
+       # Remove packages provided by glibc build, we only need a subset of them
+       d.setVar("PACKAGES", "${PN}")
+
+       d.setVar("PROVIDES", "${PN}")
+       d.setVar("RPROVIDES", "${PN} glibc-ptest")
+
+       d.setVar("RRECOMMENDS", "")
+}
+
+# Remove any lefovers from original glibc recipe
+RPROVIDES:${PN} = "${PN}"
+RRECOMMENDS:${PN} = ""
+RDEPENDS:${PN} = "sed"
+
+# Just build tests for target - do not run them
+do_check:append () {
+	oe_runmake -i check run-built-tests=no
+}
+addtask do_check after do_compile before do_install_ptest_base
+
+glibc_strip_build_directory () {
+	# Delete all non executable files from build directory
+	find ${B} ! -executable -type f -delete
+
+	# Remove build dynamic libraries and links to them as
+	# those are already installed in the target device
+	find ${B} -type f -name "*.so" -delete
+	find ${B} -type l -name "*.so*" -delete
+
+	# Remove headers (installed with glibc)
+	find ${B} -type f -name "*.h" -delete
+
+	find ${B} -type f -name "isomac" -delete
+	find ${B} -type f -name "annexc" -delete
+}
+
+do_install_ptest_base () {
+	glibc_strip_build_directory
+
+	# Install build test programs to the image
+	install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
+	cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
+
+	install -d ${D}${PTEST_PATH}
+	cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
+
+	# By default run all glibc tests
+	cd ${D}${PTEST_PATH}/
+	ln -s run-ptest-all run-ptest
+}
+
+# The datadir directory is required to allow core (and reused)
+# glibc cleanup function to finish correctly, as this directory
+# is not created for ptests
+stash_locale_package_cleanup:prepend () {
+	mkdir -p ${PKGD}${datadir}
+}
+
+stash_locale_sysroot_cleanup:prepend () {
+	mkdir -p ${SYSROOT_DESTDIR}${datadir}
+}
+
+# Prevent the do_package() task to set 'libc6' prefix
+# for glibc tests related packages
+python populate_packages:prepend () {
+    if d.getVar('DEBIAN_NAMES'):
+        d.setVar('DEBIAN_NAMES', '')
+}
+
+FILES:${PN} = "${PTEST_PATH}/* /usr/src/debug/glibc-tests/*"
+
+EXCLUDE_FROM_SHLIBS = "1"
+
+# Install debug data in .debug and sources in /usr/src/debug
+# It is more handy to have _all_ the sources and symbols in one
+# place (package) as this recipe will be used for validation and
+# debugging.
+PACKAGE_DEBUG_SPLIT_STYLE = "debug"
+
+# glibc test cases violate by default some Yocto/OE checks (staticdev,
+# textrel)
+# 'debug-files' - add everything (including debug) into one package
+#                 (no need to install/build *-src package)
+INSANE_SKIP:${PN} += "staticdev textrel debug-files rpaths"
+
+deltask do_stash_locale
+do_install[noexec] = "1"
+do_populate_sysroot[noexec] = "1"
diff --git a/meta/recipes-core/glibc/glibc/run-ptest-all b/meta/recipes-core/glibc/glibc/run-ptest-all
new file mode 100755
index 0000000000..47bfe5238c
--- /dev/null
+++ b/meta/recipes-core/glibc/glibc/run-ptest-all
@@ -0,0 +1,25 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
+
+for i in ${tst}
+do
+	$i >/dev/null 2>&1
+	output
+done
diff --git a/meta/recipes-core/glibc/glibc/run-ptest-time b/meta/recipes-core/glibc/glibc/run-ptest-time
new file mode 100755
index 0000000000..24dedc6e32
--- /dev/null
+++ b/meta/recipes-core/glibc/glibc/run-ptest-time
@@ -0,0 +1,30 @@
+#!/bin/sh
+# ptest script for glibc - to run time related tests to
+# facilitate Y2038 validation
+# Run with 'ptest-runner glibc-tests'
+
+output() {
+  retcode=$?
+  if [ $retcode -eq 0 ]
+    then echo "PASS: $i"
+  elif [ $retcode -eq 77 ]
+    then echo "SKIP: $i"
+  else echo "FAIL: $i"
+  fi
+}
+
+# Allow altering time on the target
+export GLIBC_TEST_ALLOW_TIME_SETTING="1"
+
+tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
+
+# Remove '-time64' suffix - those tests are also time
+# related
+tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
+tst_time="${tst_time_tmp} ${tst_time64}"
+
+for i in ${tst_time}
+do
+	$i >/dev/null 2>&1
+	output
+done
-- 
2.20.1

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[Khem Raj]

On 8/23/21 8:08 AM, ?ukasz Majewski wrote:
> This patch introduces new recipe - namely 'glibc-tests', which
> builds and installs glibc test suite to OE/Yocto built image.
> 
> It reuses code from already available 'glibc-testsuite' recipe,
> which is run with 'bitbake glibc-testsuite -c check' and uses qemu
> to execute remotely (via SSH) tests on some emulated machine.
> 
> This recipe installs eligible tests on some rootfs image.
> Afterwards, either all of them or only time related subset, those
> tests can be executed on the real hardware, to facilitate validation
> of this platform with for example Y2038 problem compliance.
> 
> By default all tests are executed, with 'ptest-runner glibc-tests'.
> To test only time related subset - one needs to call:
> cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
> ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests
> 
> To facilitate debugging, source files are provided by default with
> the unstripped debugging symbols. Such approach would reduce the
> already complex recipe (as it inherits base glibc one), so there
> is no need to also install *-dbg and *-src packages.


does it have to be a separate recipe I wonder if we can have it built as 
part of glibc itself controlled via ptest knob

> 
> Signed-off-by: Lukasz Majewski <lukma@denx.de>
> ---
>   .../distro/include/ptest-packagelists.inc     |   1 +
>   meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101 ++++++++++++++++++
>   meta/recipes-core/glibc/glibc/run-ptest-all   |  25 +++++
>   meta/recipes-core/glibc/glibc/run-ptest-time  |  30 ++++++
>   4 files changed, 157 insertions(+)
>   create mode 100644 meta/recipes-core/glibc/glibc-tests_2.34.bb
>   create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all
>   create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-time
> 
> diff --git a/meta/conf/distro/include/ptest-packagelists.inc b/meta/conf/distro/include/ptest-packagelists.inc
> index 3872bdc942..e460ad1c8b 100644
> --- a/meta/conf/distro/include/ptest-packagelists.inc
> +++ b/meta/conf/distro/include/ptest-packagelists.inc
> @@ -59,6 +59,7 @@ PTESTS_FAST = "\
>       slang-ptest \
>       wayland-ptest \
>       zlib-ptest \
> +    glibc-tests-ptest \
>   "
>   PTESTS_FAST:remove:mips64 = "qemu-ptest"
>   PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
> diff --git a/meta/recipes-core/glibc/glibc-tests_2.34.bb b/meta/recipes-core/glibc/glibc-tests_2.34.bb
> new file mode 100644
> index 0000000000..896809f700
> --- /dev/null
> +++ b/meta/recipes-core/glibc/glibc-tests_2.34.bb
> @@ -0,0 +1,101 @@
> +require glibc_${PV}.bb
> +require glibc-tests.inc
> +
> +inherit ptest
> +
> +SRC_URI:append = " \
> +	file://run-ptest-all file://run-ptest-time \
> +"
> +
> +SUMMARY = "glibc tests to be run with ptest"
> +
> +# Erase some variables already set by glibc_${PV}
> +python __anonymous() {
> +       # Remove packages provided by glibc build, we only need a subset of them
> +       d.setVar("PACKAGES", "${PN}")
> +
> +       d.setVar("PROVIDES", "${PN}")
> +       d.setVar("RPROVIDES", "${PN} glibc-ptest")
> +
> +       d.setVar("RRECOMMENDS", "")
> +}
> +
> +# Remove any lefovers from original glibc recipe
> +RPROVIDES:${PN} = "${PN}"
> +RRECOMMENDS:${PN} = ""
> +RDEPENDS:${PN} = "sed"
> +
> +# Just build tests for target - do not run them
> +do_check:append () {
> +	oe_runmake -i check run-built-tests=no
> +}
> +addtask do_check after do_compile before do_install_ptest_base
> +
> +glibc_strip_build_directory () {
> +	# Delete all non executable files from build directory
> +	find ${B} ! -executable -type f -delete
> +
> +	# Remove build dynamic libraries and links to them as
> +	# those are already installed in the target device
> +	find ${B} -type f -name "*.so" -delete
> +	find ${B} -type l -name "*.so*" -delete
> +
> +	# Remove headers (installed with glibc)
> +	find ${B} -type f -name "*.h" -delete
> +
> +	find ${B} -type f -name "isomac" -delete
> +	find ${B} -type f -name "annexc" -delete
> +}
> +
> +do_install_ptest_base () {
> +	glibc_strip_build_directory
> +
> +	# Install build test programs to the image
> +	install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
> +	cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
> +
> +	install -d ${D}${PTEST_PATH}
> +	cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
> +
> +	# By default run all glibc tests
> +	cd ${D}${PTEST_PATH}/
> +	ln -s run-ptest-all run-ptest
> +}
> +
> +# The datadir directory is required to allow core (and reused)
> +# glibc cleanup function to finish correctly, as this directory
> +# is not created for ptests
> +stash_locale_package_cleanup:prepend () {
> +	mkdir -p ${PKGD}${datadir}
> +}
> +
> +stash_locale_sysroot_cleanup:prepend () {
> +	mkdir -p ${SYSROOT_DESTDIR}${datadir}
> +}
> +
> +# Prevent the do_package() task to set 'libc6' prefix
> +# for glibc tests related packages
> +python populate_packages:prepend () {
> +    if d.getVar('DEBIAN_NAMES'):
> +        d.setVar('DEBIAN_NAMES', '')
> +}
> +
> +FILES:${PN} = "${PTEST_PATH}/* /usr/src/debug/glibc-tests/*"
> +
> +EXCLUDE_FROM_SHLIBS = "1"
> +
> +# Install debug data in .debug and sources in /usr/src/debug
> +# It is more handy to have _all_ the sources and symbols in one
> +# place (package) as this recipe will be used for validation and
> +# debugging.
> +PACKAGE_DEBUG_SPLIT_STYLE = "debug"
> +
> +# glibc test cases violate by default some Yocto/OE checks (staticdev,
> +# textrel)
> +# 'debug-files' - add everything (including debug) into one package
> +#                 (no need to install/build *-src package)
> +INSANE_SKIP:${PN} += "staticdev textrel debug-files rpaths"
> +
> +deltask do_stash_locale
> +do_install[noexec] = "1"
> +do_populate_sysroot[noexec] = "1"
> diff --git a/meta/recipes-core/glibc/glibc/run-ptest-all b/meta/recipes-core/glibc/glibc/run-ptest-all
> new file mode 100755
> index 0000000000..47bfe5238c
> --- /dev/null
> +++ b/meta/recipes-core/glibc/glibc/run-ptest-all
> @@ -0,0 +1,25 @@
> +#!/bin/sh
> +# ptest script for glibc - to run time related tests to
> +# facilitate Y2038 validation
> +# Run with 'ptest-runner glibc-tests'
> +
> +output() {
> +  retcode=$?
> +  if [ $retcode -eq 0 ]
> +    then echo "PASS: $i"
> +  elif [ $retcode -eq 77 ]
> +    then echo "SKIP: $i"
> +  else echo "FAIL: $i"
> +  fi
> +}
> +
> +# Allow altering time on the target
> +export GLIBC_TEST_ALLOW_TIME_SETTING="1"
> +
> +tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
> +
> +for i in ${tst}
> +do
> +	$i >/dev/null 2>&1
> +	output
> +done
> diff --git a/meta/recipes-core/glibc/glibc/run-ptest-time b/meta/recipes-core/glibc/glibc/run-ptest-time
> new file mode 100755
> index 0000000000..24dedc6e32
> --- /dev/null
> +++ b/meta/recipes-core/glibc/glibc/run-ptest-time
> @@ -0,0 +1,30 @@
> +#!/bin/sh
> +# ptest script for glibc - to run time related tests to
> +# facilitate Y2038 validation
> +# Run with 'ptest-runner glibc-tests'
> +
> +output() {
> +  retcode=$?
> +  if [ $retcode -eq 0 ]
> +    then echo "PASS: $i"
> +  elif [ $retcode -eq 77 ]
> +    then echo "SKIP: $i"
> +  else echo "FAIL: $i"
> +  fi
> +}
> +
> +# Allow altering time on the target
> +export GLIBC_TEST_ALLOW_TIME_SETTING="1"
> +
> +tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
> +
> +# Remove '-time64' suffix - those tests are also time
> +# related
> +tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
> +tst_time="${tst_time_tmp} ${tst_time64}"
> +
> +for i in ${tst_time}
> +do
> +	$i >/dev/null 2>&1
> +	output
> +done
> 
> 
> 
> 
> 

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[?ukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 9002 bytes --]

Hi Khem,

> On 8/23/21 8:08 AM, ?ukasz Majewski wrote:
> > This patch introduces new recipe - namely 'glibc-tests', which
> > builds and installs glibc test suite to OE/Yocto built image.
> > 
> > It reuses code from already available 'glibc-testsuite' recipe,
> > which is run with 'bitbake glibc-testsuite -c check' and uses qemu
> > to execute remotely (via SSH) tests on some emulated machine.
> > 
> > This recipe installs eligible tests on some rootfs image.
> > Afterwards, either all of them or only time related subset, those
> > tests can be executed on the real hardware, to facilitate validation
> > of this platform with for example Y2038 problem compliance.
> > 
> > By default all tests are executed, with 'ptest-runner glibc-tests'.
> > To test only time related subset - one needs to call:
> > cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
> > ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests
> > 
> > To facilitate debugging, source files are provided by default with
> > the unstripped debugging symbols. Such approach would reduce the
> > already complex recipe (as it inherits base glibc one), so there
> > is no need to also install *-dbg and *-src packages.  
> 
> 
> does it have to be a separate recipe I wonder if we can have it built
> as part of glibc itself controlled via ptest knob

I've followed the glibc-testsuite recipe to provide tests for ptest.
Test creation is similar to it, but doesn't require QEMU run (tests are
executed on the HW).

Another rationale was to have some kind of "features" separation in
different file (liked glibc-mtrace), which is easier to maintain and
review.

Last but not least - separate recipe (and built binaries) allow some
kind of magic with selection of used test programs (this may be useful
if one would like to have different sets of tests in different packages)

> 
> > 
> > Signed-off-by: Lukasz Majewski <lukma@denx.de>
> > ---
> >   .../distro/include/ptest-packagelists.inc     |   1 +
> >   meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101
> > ++++++++++++++++++ meta/recipes-core/glibc/glibc/run-ptest-all   |
> > 25 +++++ meta/recipes-core/glibc/glibc/run-ptest-time  |  30 ++++++
> >   4 files changed, 157 insertions(+)
> >   create mode 100644 meta/recipes-core/glibc/glibc-tests_2.34.bb
> >   create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all
> >   create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-time
> > 
> > diff --git a/meta/conf/distro/include/ptest-packagelists.inc
> > b/meta/conf/distro/include/ptest-packagelists.inc index
> > 3872bdc942..e460ad1c8b 100644 ---
> > a/meta/conf/distro/include/ptest-packagelists.inc +++
> > b/meta/conf/distro/include/ptest-packagelists.inc @@ -59,6 +59,7 @@
> > PTESTS_FAST = "\ slang-ptest \
> >       wayland-ptest \
> >       zlib-ptest \
> > +    glibc-tests-ptest \
> >   "
> >   PTESTS_FAST:remove:mips64 = "qemu-ptest"
> >   PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
> > diff --git a/meta/recipes-core/glibc/glibc-tests_2.34.bb
> > b/meta/recipes-core/glibc/glibc-tests_2.34.bb new file mode 100644
> > index 0000000000..896809f700
> > --- /dev/null
> > +++ b/meta/recipes-core/glibc/glibc-tests_2.34.bb
> > @@ -0,0 +1,101 @@
> > +require glibc_${PV}.bb
> > +require glibc-tests.inc
> > +
> > +inherit ptest
> > +
> > +SRC_URI:append = " \
> > +	file://run-ptest-all file://run-ptest-time \
> > +"
> > +
> > +SUMMARY = "glibc tests to be run with ptest"
> > +
> > +# Erase some variables already set by glibc_${PV}
> > +python __anonymous() {
> > +       # Remove packages provided by glibc build, we only need a
> > subset of them
> > +       d.setVar("PACKAGES", "${PN}")
> > +
> > +       d.setVar("PROVIDES", "${PN}")
> > +       d.setVar("RPROVIDES", "${PN} glibc-ptest")
> > +
> > +       d.setVar("RRECOMMENDS", "")
> > +}
> > +
> > +# Remove any lefovers from original glibc recipe
> > +RPROVIDES:${PN} = "${PN}"
> > +RRECOMMENDS:${PN} = ""
> > +RDEPENDS:${PN} = "sed"
> > +
> > +# Just build tests for target - do not run them
> > +do_check:append () {
> > +	oe_runmake -i check run-built-tests=no
> > +}
> > +addtask do_check after do_compile before do_install_ptest_base
> > +
> > +glibc_strip_build_directory () {
> > +	# Delete all non executable files from build directory
> > +	find ${B} ! -executable -type f -delete
> > +
> > +	# Remove build dynamic libraries and links to them as
> > +	# those are already installed in the target device
> > +	find ${B} -type f -name "*.so" -delete
> > +	find ${B} -type l -name "*.so*" -delete
> > +
> > +	# Remove headers (installed with glibc)
> > +	find ${B} -type f -name "*.h" -delete
> > +
> > +	find ${B} -type f -name "isomac" -delete
> > +	find ${B} -type f -name "annexc" -delete
> > +}
> > +
> > +do_install_ptest_base () {
> > +	glibc_strip_build_directory
> > +
> > +	# Install build test programs to the image
> > +	install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
> > +	cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
> > +
> > +	install -d ${D}${PTEST_PATH}
> > +	cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
> > +
> > +	# By default run all glibc tests
> > +	cd ${D}${PTEST_PATH}/
> > +	ln -s run-ptest-all run-ptest
> > +}
> > +
> > +# The datadir directory is required to allow core (and reused)
> > +# glibc cleanup function to finish correctly, as this directory
> > +# is not created for ptests
> > +stash_locale_package_cleanup:prepend () {
> > +	mkdir -p ${PKGD}${datadir}
> > +}
> > +
> > +stash_locale_sysroot_cleanup:prepend () {
> > +	mkdir -p ${SYSROOT_DESTDIR}${datadir}
> > +}
> > +
> > +# Prevent the do_package() task to set 'libc6' prefix
> > +# for glibc tests related packages
> > +python populate_packages:prepend () {
> > +    if d.getVar('DEBIAN_NAMES'):
> > +        d.setVar('DEBIAN_NAMES', '')
> > +}
> > +
> > +FILES:${PN} = "${PTEST_PATH}/* /usr/src/debug/glibc-tests/*"
> > +
> > +EXCLUDE_FROM_SHLIBS = "1"
> > +
> > +# Install debug data in .debug and sources in /usr/src/debug
> > +# It is more handy to have _all_ the sources and symbols in one
> > +# place (package) as this recipe will be used for validation and
> > +# debugging.
> > +PACKAGE_DEBUG_SPLIT_STYLE = "debug"
> > +
> > +# glibc test cases violate by default some Yocto/OE checks
> > (staticdev, +# textrel)
> > +# 'debug-files' - add everything (including debug) into one package
> > +#                 (no need to install/build *-src package)
> > +INSANE_SKIP:${PN} += "staticdev textrel debug-files rpaths"
> > +
> > +deltask do_stash_locale
> > +do_install[noexec] = "1"
> > +do_populate_sysroot[noexec] = "1"
> > diff --git a/meta/recipes-core/glibc/glibc/run-ptest-all
> > b/meta/recipes-core/glibc/glibc/run-ptest-all new file mode 100755
> > index 0000000000..47bfe5238c
> > --- /dev/null
> > +++ b/meta/recipes-core/glibc/glibc/run-ptest-all
> > @@ -0,0 +1,25 @@
> > +#!/bin/sh
> > +# ptest script for glibc - to run time related tests to
> > +# facilitate Y2038 validation
> > +# Run with 'ptest-runner glibc-tests'
> > +
> > +output() {
> > +  retcode=$?
> > +  if [ $retcode -eq 0 ]
> > +    then echo "PASS: $i"
> > +  elif [ $retcode -eq 77 ]
> > +    then echo "SKIP: $i"
> > +  else echo "FAIL: $i"
> > +  fi
> > +}
> > +
> > +# Allow altering time on the target
> > +export GLIBC_TEST_ALLOW_TIME_SETTING="1"
> > +
> > +tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
> > +
> > +for i in ${tst}
> > +do
> > +	$i >/dev/null 2>&1
> > +	output
> > +done
> > diff --git a/meta/recipes-core/glibc/glibc/run-ptest-time
> > b/meta/recipes-core/glibc/glibc/run-ptest-time new file mode 100755
> > index 0000000000..24dedc6e32
> > --- /dev/null
> > +++ b/meta/recipes-core/glibc/glibc/run-ptest-time
> > @@ -0,0 +1,30 @@
> > +#!/bin/sh
> > +# ptest script for glibc - to run time related tests to
> > +# facilitate Y2038 validation
> > +# Run with 'ptest-runner glibc-tests'
> > +
> > +output() {
> > +  retcode=$?
> > +  if [ $retcode -eq 0 ]
> > +    then echo "PASS: $i"
> > +  elif [ $retcode -eq 77 ]
> > +    then echo "SKIP: $i"
> > +  else echo "FAIL: $i"
> > +  fi
> > +}
> > +
> > +# Allow altering time on the target
> > +export GLIBC_TEST_ALLOW_TIME_SETTING="1"
> > +
> > +tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
> > +
> > +# Remove '-time64' suffix - those tests are also time
> > +# related
> > +tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
> > +tst_time="${tst_time_tmp} ${tst_time64}"
> > +
> > +for i in ${tst_time}
> > +do
> > +	$i >/dev/null 2>&1
> > +	output
> > +done
> > 
> > 
> > 
> > 
> >   




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 9920 bytes --]

On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski <lukma@denx.de> wrote:

> Hi Khem,
>
> > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:
> > > This patch introduces new recipe - namely 'glibc-tests', which
> > > builds and installs glibc test suite to OE/Yocto built image.
> > >
> > > It reuses code from already available 'glibc-testsuite' recipe,
> > > which is run with 'bitbake glibc-testsuite -c check' and uses qemu
> > > to execute remotely (via SSH) tests on some emulated machine.
> > >
> > > This recipe installs eligible tests on some rootfs image.
> > > Afterwards, either all of them or only time related subset, those
> > > tests can be executed on the real hardware, to facilitate validation
> > > of this platform with for example Y2038 problem compliance.
> > >
> > > By default all tests are executed, with 'ptest-runner glibc-tests'.
> > > To test only time related subset - one needs to call:
> > > cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
> > > ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests
> > >
> > > To facilitate debugging, source files are provided by default with
> > > the unstripped debugging symbols. Such approach would reduce the
> > > already complex recipe (as it inherits base glibc one), so there
> > > is no need to also install *-dbg and *-src packages.
> >
> >
> > does it have to be a separate recipe I wonder if we can have it built
> > as part of glibc itself controlled via ptest knob
>
> I've followed the glibc-testsuite recipe to provide tests for ptest.
> Test creation is similar to it, but doesn't require QEMU run (tests are
> executed on the HW).
>
> Another rationale was to have some kind of "features" separation in
> different file (liked glibc-mtrace), which is easier to maintain and
> review.
>
> Last but not least - separate recipe (and built binaries) allow some
> kind of magic with selection of used test programs (this may be useful
> if one would like to have different sets of tests in different packages)
>

That’s seems ok I think the names are quite confusing now not that it was
not so much better before but now we have glibc-tests and glibc-testsuites
which do same things but in very different way maybe glibc-testsuite should
be renamed to something like glibc-tests-crosd or some such

>
> >
> > >
> > > Signed-off-by: Lukasz Majewski <lukma@denx.de>
> > > ---
> > >   .../distro/include/ptest-packagelists.inc     |   1 +
> > >   meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101
> > > ++++++++++++++++++ meta/recipes-core/glibc/glibc/run-ptest-all   |
> > > 25 +++++ meta/recipes-core/glibc/glibc/run-ptest-time  |  30 ++++++
> > >   4 files changed, 157 insertions(+)
> > >   create mode 100644 meta/recipes-core/glibc/glibc-tests_2.34.bb
> > >   create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all
> > >   create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-time
> > >
> > > diff --git a/meta/conf/distro/include/ptest-packagelists.inc
> > > b/meta/conf/distro/include/ptest-packagelists.inc index
> > > 3872bdc942..e460ad1c8b 100644 ---
> > > a/meta/conf/distro/include/ptest-packagelists.inc +++
> > > b/meta/conf/distro/include/ptest-packagelists.inc @@ -59,6 +59,7 @@
> > > PTESTS_FAST = "\ slang-ptest \
> > >       wayland-ptest \
> > >       zlib-ptest \
> > > +    glibc-tests-ptest \
> > >   "
> > >   PTESTS_FAST:remove:mips64 = "qemu-ptest"
> > >   PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
> > > diff --git a/meta/recipes-core/glibc/glibc-tests_2.34.bb
> > > b/meta/recipes-core/glibc/glibc-tests_2.34.bb new file mode 100644
> > > index 0000000000..896809f700
> > > --- /dev/null
> > > +++ b/meta/recipes-core/glibc/glibc-tests_2.34.bb
> > > @@ -0,0 +1,101 @@
> > > +require glibc_${PV}.bb
> > > +require glibc-tests.inc
> > > +
> > > +inherit ptest
> > > +
> > > +SRC_URI:append = " \
> > > +   file://run-ptest-all file://run-ptest-time \
> > > +"
> > > +
> > > +SUMMARY = "glibc tests to be run with ptest"
> > > +
> > > +# Erase some variables already set by glibc_${PV}
> > > +python __anonymous() {
> > > +       # Remove packages provided by glibc build, we only need a
> > > subset of them
> > > +       d.setVar("PACKAGES", "${PN}")
> > > +
> > > +       d.setVar("PROVIDES", "${PN}")
> > > +       d.setVar("RPROVIDES", "${PN} glibc-ptest")
> > > +
> > > +       d.setVar("RRECOMMENDS", "")
> > > +}
> > > +
> > > +# Remove any lefovers from original glibc recipe
> > > +RPROVIDES:${PN} = "${PN}"
> > > +RRECOMMENDS:${PN} = ""
> > > +RDEPENDS:${PN} = "sed"
> > > +
> > > +# Just build tests for target - do not run them
> > > +do_check:append () {
> > > +   oe_runmake -i check run-built-tests=no
> > > +}
> > > +addtask do_check after do_compile before do_install_ptest_base
> > > +
> > > +glibc_strip_build_directory () {
> > > +   # Delete all non executable files from build directory
> > > +   find ${B} ! -executable -type f -delete
> > > +
> > > +   # Remove build dynamic libraries and links to them as
> > > +   # those are already installed in the target device
> > > +   find ${B} -type f -name "*.so" -delete
> > > +   find ${B} -type l -name "*.so*" -delete
> > > +
> > > +   # Remove headers (installed with glibc)
> > > +   find ${B} -type f -name "*.h" -delete
> > > +
> > > +   find ${B} -type f -name "isomac" -delete
> > > +   find ${B} -type f -name "annexc" -delete
> > > +}
> > > +
> > > +do_install_ptest_base () {
> > > +   glibc_strip_build_directory
> > > +
> > > +   # Install build test programs to the image
> > > +   install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
> > > +   cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
> > > +
> > > +   install -d ${D}${PTEST_PATH}
> > > +   cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
> > > +
> > > +   # By default run all glibc tests
> > > +   cd ${D}${PTEST_PATH}/
> > > +   ln -s run-ptest-all run-ptest
> > > +}
> > > +
> > > +# The datadir directory is required to allow core (and reused)
> > > +# glibc cleanup function to finish correctly, as this directory
> > > +# is not created for ptests
> > > +stash_locale_package_cleanup:prepend () {
> > > +   mkdir -p ${PKGD}${datadir}
> > > +}
> > > +
> > > +stash_locale_sysroot_cleanup:prepend () {
> > > +   mkdir -p ${SYSROOT_DESTDIR}${datadir}
> > > +}
> > > +
> > > +# Prevent the do_package() task to set 'libc6' prefix
> > > +# for glibc tests related packages
> > > +python populate_packages:prepend () {
> > > +    if d.getVar('DEBIAN_NAMES'):
> > > +        d.setVar('DEBIAN_NAMES', '')
> > > +}
> > > +
> > > +FILES:${PN} = "${PTEST_PATH}/* /usr/src/debug/glibc-tests/*"
> > > +
> > > +EXCLUDE_FROM_SHLIBS = "1"
> > > +
> > > +# Install debug data in .debug and sources in /usr/src/debug
> > > +# It is more handy to have _all_ the sources and symbols in one
> > > +# place (package) as this recipe will be used for validation and
> > > +# debugging.
> > > +PACKAGE_DEBUG_SPLIT_STYLE = "debug"
> > > +
> > > +# glibc test cases violate by default some Yocto/OE checks
> > > (staticdev, +# textrel)
> > > +# 'debug-files' - add everything (including debug) into one package
> > > +#                 (no need to install/build *-src package)
> > > +INSANE_SKIP:${PN} += "staticdev textrel debug-files rpaths"
> > > +
> > > +deltask do_stash_locale
> > > +do_install[noexec] = "1"
> > > +do_populate_sysroot[noexec] = "1"
> > > diff --git a/meta/recipes-core/glibc/glibc/run-ptest-all
> > > b/meta/recipes-core/glibc/glibc/run-ptest-all new file mode 100755
> > > index 0000000000..47bfe5238c
> > > --- /dev/null
> > > +++ b/meta/recipes-core/glibc/glibc/run-ptest-all
> > > @@ -0,0 +1,25 @@
> > > +#!/bin/sh
> > > +# ptest script for glibc - to run time related tests to
> > > +# facilitate Y2038 validation
> > > +# Run with 'ptest-runner glibc-tests'
> > > +
> > > +output() {
> > > +  retcode=$?
> > > +  if [ $retcode -eq 0 ]
> > > +    then echo "PASS: $i"
> > > +  elif [ $retcode -eq 77 ]
> > > +    then echo "SKIP: $i"
> > > +  else echo "FAIL: $i"
> > > +  fi
> > > +}
> > > +
> > > +# Allow altering time on the target
> > > +export GLIBC_TEST_ALLOW_TIME_SETTING="1"
> > > +
> > > +tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
> > > +
> > > +for i in ${tst}
> > > +do
> > > +   $i >/dev/null 2>&1
> > > +   output
> > > +done
> > > diff --git a/meta/recipes-core/glibc/glibc/run-ptest-time
> > > b/meta/recipes-core/glibc/glibc/run-ptest-time new file mode 100755
> > > index 0000000000..24dedc6e32
> > > --- /dev/null
> > > +++ b/meta/recipes-core/glibc/glibc/run-ptest-time
> > > @@ -0,0 +1,30 @@
> > > +#!/bin/sh
> > > +# ptest script for glibc - to run time related tests to
> > > +# facilitate Y2038 validation
> > > +# Run with 'ptest-runner glibc-tests'
> > > +
> > > +output() {
> > > +  retcode=$?
> > > +  if [ $retcode -eq 0 ]
> > > +    then echo "PASS: $i"
> > > +  elif [ $retcode -eq 77 ]
> > > +    then echo "SKIP: $i"
> > > +  else echo "FAIL: $i"
> > > +  fi
> > > +}
> > > +
> > > +# Allow altering time on the target
> > > +export GLIBC_TEST_ALLOW_TIME_SETTING="1"
> > > +
> > > +tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
> > > +
> > > +# Remove '-time64' suffix - those tests are also time
> > > +# related
> > > +tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
> > > +tst_time="${tst_time_tmp} ${tst_time64}"
> > > +
> > > +for i in ${tst_time}
> > > +do
> > > +   $i >/dev/null 2>&1
> > > +   output
> > > +done
> > >
> > >
> > >
> > >
> > >
>
>
>
>
> Best regards,
>
> Lukasz Majewski
>
> --
>
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de
>

[-- Attachment #2: Type: text/html, Size: 13510 bytes --]

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[?ukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 11327 bytes --]

On Mon, 23 Aug 2021 12:52:44 -0700
Khem Raj <raj.khem@gmail.com> wrote:

> On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski <lukma@denx.de>
> wrote:
> 
> > Hi Khem,
> >  
> > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:  
> > > > This patch introduces new recipe - namely 'glibc-tests', which
> > > > builds and installs glibc test suite to OE/Yocto built image.
> > > >
> > > > It reuses code from already available 'glibc-testsuite' recipe,
> > > > which is run with 'bitbake glibc-testsuite -c check' and uses
> > > > qemu to execute remotely (via SSH) tests on some emulated
> > > > machine.
> > > >
> > > > This recipe installs eligible tests on some rootfs image.
> > > > Afterwards, either all of them or only time related subset,
> > > > those tests can be executed on the real hardware, to facilitate
> > > > validation of this platform with for example Y2038 problem
> > > > compliance.
> > > >
> > > > By default all tests are executed, with 'ptest-runner
> > > > glibc-tests'. To test only time related subset - one needs to
> > > > call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
> > > > ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests
> > > >
> > > > To facilitate debugging, source files are provided by default
> > > > with the unstripped debugging symbols. Such approach would
> > > > reduce the already complex recipe (as it inherits base glibc
> > > > one), so there is no need to also install *-dbg and *-src
> > > > packages.  
> > >
> > >
> > > does it have to be a separate recipe I wonder if we can have it
> > > built as part of glibc itself controlled via ptest knob  
> >
> > I've followed the glibc-testsuite recipe to provide tests for ptest.
> > Test creation is similar to it, but doesn't require QEMU run (tests
> > are executed on the HW).
> >
> > Another rationale was to have some kind of "features" separation in
> > different file (liked glibc-mtrace), which is easier to maintain and
> > review.
> >
> > Last but not least - separate recipe (and built binaries) allow some
> > kind of magic with selection of used test programs (this may be
> > useful if one would like to have different sets of tests in
> > different packages) 
> 
> That’s seems ok I think the names are quite confusing now not that it
> was not so much better before but now we have glibc-tests and
> glibc-testsuites which do same things but in very different way maybe
> glibc-testsuite should be renamed to something like glibc-tests-crosd
> or some such

I think that glibc-testsuite_2.34.bb shall be renamed to
glibc-tests-qemu_2.34.bb as it is more descriptive.

Then, glibc-tests_2.34.bb could be left as it is now, as inheriting
ptest requires it to be added as 'glibc-tests-ptest' in
meta/conf/distro/include/ptest-packagelists.inc

(I've tried to rename it to glibc-tests-ptest_2.34.bb, but then it was
required to add glibc-tests-ptest-ptest to the ptest-packagelist.inc
file, which is a bit clumsy IMHO).

> 
> >  
> > >  
> > > >
> > > > Signed-off-by: Lukasz Majewski <lukma@denx.de>
> > > > ---
> > > >   .../distro/include/ptest-packagelists.inc     |   1 +
> > > >   meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101
> > > > ++++++++++++++++++ meta/recipes-core/glibc/glibc/run-ptest-all
> > > >  | 25 +++++ meta/recipes-core/glibc/glibc/run-ptest-time  |  30
> > > > ++++++ 4 files changed, 157 insertions(+)
> > > >   create mode 100644 meta/recipes-core/glibc/glibc-tests_2.34.bb
> > > >   create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all
> > > >   create mode 100755
> > > > meta/recipes-core/glibc/glibc/run-ptest-time
> > > >
> > > > diff --git a/meta/conf/distro/include/ptest-packagelists.inc
> > > > b/meta/conf/distro/include/ptest-packagelists.inc index
> > > > 3872bdc942..e460ad1c8b 100644 ---
> > > > a/meta/conf/distro/include/ptest-packagelists.inc +++
> > > > b/meta/conf/distro/include/ptest-packagelists.inc @@ -59,6
> > > > +59,7 @@ PTESTS_FAST = "\ slang-ptest \
> > > >       wayland-ptest \
> > > >       zlib-ptest \
> > > > +    glibc-tests-ptest \
> > > >   "
> > > >   PTESTS_FAST:remove:mips64 = "qemu-ptest"
> > > >   PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
> > > > diff --git a/meta/recipes-core/glibc/glibc-tests_2.34.bb
> > > > b/meta/recipes-core/glibc/glibc-tests_2.34.bb new file mode
> > > > 100644 index 0000000000..896809f700
> > > > --- /dev/null
> > > > +++ b/meta/recipes-core/glibc/glibc-tests_2.34.bb
> > > > @@ -0,0 +1,101 @@
> > > > +require glibc_${PV}.bb
> > > > +require glibc-tests.inc
> > > > +
> > > > +inherit ptest
> > > > +
> > > > +SRC_URI:append = " \
> > > > +   file://run-ptest-all file://run-ptest-time \
> > > > +"
> > > > +
> > > > +SUMMARY = "glibc tests to be run with ptest"
> > > > +
> > > > +# Erase some variables already set by glibc_${PV}
> > > > +python __anonymous() {
> > > > +       # Remove packages provided by glibc build, we only need
> > > > a subset of them
> > > > +       d.setVar("PACKAGES", "${PN}")
> > > > +
> > > > +       d.setVar("PROVIDES", "${PN}")
> > > > +       d.setVar("RPROVIDES", "${PN} glibc-ptest")
> > > > +
> > > > +       d.setVar("RRECOMMENDS", "")
> > > > +}
> > > > +
> > > > +# Remove any lefovers from original glibc recipe
> > > > +RPROVIDES:${PN} = "${PN}"
> > > > +RRECOMMENDS:${PN} = ""
> > > > +RDEPENDS:${PN} = "sed"
> > > > +
> > > > +# Just build tests for target - do not run them
> > > > +do_check:append () {
> > > > +   oe_runmake -i check run-built-tests=no
> > > > +}
> > > > +addtask do_check after do_compile before do_install_ptest_base
> > > > +
> > > > +glibc_strip_build_directory () {
> > > > +   # Delete all non executable files from build directory
> > > > +   find ${B} ! -executable -type f -delete
> > > > +
> > > > +   # Remove build dynamic libraries and links to them as
> > > > +   # those are already installed in the target device
> > > > +   find ${B} -type f -name "*.so" -delete
> > > > +   find ${B} -type l -name "*.so*" -delete
> > > > +
> > > > +   # Remove headers (installed with glibc)
> > > > +   find ${B} -type f -name "*.h" -delete
> > > > +
> > > > +   find ${B} -type f -name "isomac" -delete
> > > > +   find ${B} -type f -name "annexc" -delete
> > > > +}
> > > > +
> > > > +do_install_ptest_base () {
> > > > +   glibc_strip_build_directory
> > > > +
> > > > +   # Install build test programs to the image
> > > > +   install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
> > > > +   cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
> > > > +
> > > > +   install -d ${D}${PTEST_PATH}
> > > > +   cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
> > > > +
> > > > +   # By default run all glibc tests
> > > > +   cd ${D}${PTEST_PATH}/
> > > > +   ln -s run-ptest-all run-ptest
> > > > +}
> > > > +
> > > > +# The datadir directory is required to allow core (and reused)
> > > > +# glibc cleanup function to finish correctly, as this directory
> > > > +# is not created for ptests
> > > > +stash_locale_package_cleanup:prepend () {
> > > > +   mkdir -p ${PKGD}${datadir}
> > > > +}
> > > > +
> > > > +stash_locale_sysroot_cleanup:prepend () {
> > > > +   mkdir -p ${SYSROOT_DESTDIR}${datadir}
> > > > +}
> > > > +
> > > > +# Prevent the do_package() task to set 'libc6' prefix
> > > > +# for glibc tests related packages
> > > > +python populate_packages:prepend () {
> > > > +    if d.getVar('DEBIAN_NAMES'):
> > > > +        d.setVar('DEBIAN_NAMES', '')
> > > > +}
> > > > +
> > > > +FILES:${PN} = "${PTEST_PATH}/* /usr/src/debug/glibc-tests/*"
> > > > +
> > > > +EXCLUDE_FROM_SHLIBS = "1"
> > > > +
> > > > +# Install debug data in .debug and sources in /usr/src/debug
> > > > +# It is more handy to have _all_ the sources and symbols in one
> > > > +# place (package) as this recipe will be used for validation
> > > > and +# debugging.
> > > > +PACKAGE_DEBUG_SPLIT_STYLE = "debug"
> > > > +
> > > > +# glibc test cases violate by default some Yocto/OE checks
> > > > (staticdev, +# textrel)
> > > > +# 'debug-files' - add everything (including debug) into one
> > > > package +#                 (no need to install/build *-src
> > > > package) +INSANE_SKIP:${PN} += "staticdev textrel debug-files
> > > > rpaths" +
> > > > +deltask do_stash_locale
> > > > +do_install[noexec] = "1"
> > > > +do_populate_sysroot[noexec] = "1"
> > > > diff --git a/meta/recipes-core/glibc/glibc/run-ptest-all
> > > > b/meta/recipes-core/glibc/glibc/run-ptest-all new file mode
> > > > 100755 index 0000000000..47bfe5238c
> > > > --- /dev/null
> > > > +++ b/meta/recipes-core/glibc/glibc/run-ptest-all
> > > > @@ -0,0 +1,25 @@
> > > > +#!/bin/sh
> > > > +# ptest script for glibc - to run time related tests to
> > > > +# facilitate Y2038 validation
> > > > +# Run with 'ptest-runner glibc-tests'
> > > > +
> > > > +output() {
> > > > +  retcode=$?
> > > > +  if [ $retcode -eq 0 ]
> > > > +    then echo "PASS: $i"
> > > > +  elif [ $retcode -eq 77 ]
> > > > +    then echo "SKIP: $i"
> > > > +  else echo "FAIL: $i"
> > > > +  fi
> > > > +}
> > > > +
> > > > +# Allow altering time on the target
> > > > +export GLIBC_TEST_ALLOW_TIME_SETTING="1"
> > > > +
> > > > +tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
> > > > +
> > > > +for i in ${tst}
> > > > +do
> > > > +   $i >/dev/null 2>&1
> > > > +   output
> > > > +done
> > > > diff --git a/meta/recipes-core/glibc/glibc/run-ptest-time
> > > > b/meta/recipes-core/glibc/glibc/run-ptest-time new file mode
> > > > 100755 index 0000000000..24dedc6e32
> > > > --- /dev/null
> > > > +++ b/meta/recipes-core/glibc/glibc/run-ptest-time
> > > > @@ -0,0 +1,30 @@
> > > > +#!/bin/sh
> > > > +# ptest script for glibc - to run time related tests to
> > > > +# facilitate Y2038 validation
> > > > +# Run with 'ptest-runner glibc-tests'
> > > > +
> > > > +output() {
> > > > +  retcode=$?
> > > > +  if [ $retcode -eq 0 ]
> > > > +    then echo "PASS: $i"
> > > > +  elif [ $retcode -eq 77 ]
> > > > +    then echo "SKIP: $i"
> > > > +  else echo "FAIL: $i"
> > > > +  fi
> > > > +}
> > > > +
> > > > +# Allow altering time on the target
> > > > +export GLIBC_TEST_ALLOW_TIME_SETTING="1"
> > > > +
> > > > +tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
> > > > +
> > > > +# Remove '-time64' suffix - those tests are also time
> > > > +# related
> > > > +tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
> > > > +tst_time="${tst_time_tmp} ${tst_time64}"
> > > > +
> > > > +for i in ${tst_time}
> > > > +do
> > > > +   $i >/dev/null 2>&1
> > > > +   output
> > > > +done
> > > >
> > > >
> > > >
> > > >
> > > >  
> >
> >
> >
> >
> > Best regards,
> >
> > Lukasz Majewski
> >
> > --
> >
> > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > Germany Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
> > lukma@denx.de 




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[Khem Raj]

On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski <lukma@denx.de> wrote:
>
> On Mon, 23 Aug 2021 12:52:44 -0700
> Khem Raj <raj.khem@gmail.com> wrote:
>
> > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski <lukma@denx.de>
> > wrote:
> >
> > > Hi Khem,
> > >
> > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:
> > > > > This patch introduces new recipe - namely 'glibc-tests', which
> > > > > builds and installs glibc test suite to OE/Yocto built image.
> > > > >
> > > > > It reuses code from already available 'glibc-testsuite' recipe,
> > > > > which is run with 'bitbake glibc-testsuite -c check' and uses
> > > > > qemu to execute remotely (via SSH) tests on some emulated
> > > > > machine.
> > > > >
> > > > > This recipe installs eligible tests on some rootfs image.
> > > > > Afterwards, either all of them or only time related subset,
> > > > > those tests can be executed on the real hardware, to facilitate
> > > > > validation of this platform with for example Y2038 problem
> > > > > compliance.
> > > > >
> > > > > By default all tests are executed, with 'ptest-runner
> > > > > glibc-tests'. To test only time related subset - one needs to
> > > > > call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
> > > > > ln -s run-ptest-time run-ptest; cd -; ptest-runner glibc-tests
> > > > >
> > > > > To facilitate debugging, source files are provided by default
> > > > > with the unstripped debugging symbols. Such approach would
> > > > > reduce the already complex recipe (as it inherits base glibc
> > > > > one), so there is no need to also install *-dbg and *-src
> > > > > packages.
> > > >
> > > >
> > > > does it have to be a separate recipe I wonder if we can have it
> > > > built as part of glibc itself controlled via ptest knob
> > >
> > > I've followed the glibc-testsuite recipe to provide tests for ptest.
> > > Test creation is similar to it, but doesn't require QEMU run (tests
> > > are executed on the HW).
> > >
> > > Another rationale was to have some kind of "features" separation in
> > > different file (liked glibc-mtrace), which is easier to maintain and
> > > review.
> > >
> > > Last but not least - separate recipe (and built binaries) allow some
> > > kind of magic with selection of used test programs (this may be
> > > useful if one would like to have different sets of tests in
> > > different packages)
> >
> > That’s seems ok I think the names are quite confusing now not that it
> > was not so much better before but now we have glibc-tests and
> > glibc-testsuites which do same things but in very different way maybe
> > glibc-testsuite should be renamed to something like glibc-tests-crosd
> > or some such
>
> I think that glibc-testsuite_2.34.bb shall be renamed to
> glibc-tests-qemu_2.34.bb as it is more descriptive.

is it only runnable in qemu ? I thought we could use a read board as
well with IP address

>
> Then, glibc-tests_2.34.bb could be left as it is now, as inheriting
> ptest requires it to be added as 'glibc-tests-ptest' in
> meta/conf/distro/include/ptest-packagelists.inc
>
> (I've tried to rename it to glibc-tests-ptest_2.34.bb, but then it was
> required to add glibc-tests-ptest-ptest to the ptest-packagelist.inc
> file, which is a bit clumsy IMHO).
>
> >
> > >
> > > >
> > > > >
> > > > > Signed-off-by: Lukasz Majewski <lukma@denx.de>
> > > > > ---
> > > > >   .../distro/include/ptest-packagelists.inc     |   1 +
> > > > >   meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101
> > > > > ++++++++++++++++++ meta/recipes-core/glibc/glibc/run-ptest-all
> > > > >  | 25 +++++ meta/recipes-core/glibc/glibc/run-ptest-time  |  30
> > > > > ++++++ 4 files changed, 157 insertions(+)
> > > > >   create mode 100644 meta/recipes-core/glibc/glibc-tests_2.34.bb
> > > > >   create mode 100755 meta/recipes-core/glibc/glibc/run-ptest-all
> > > > >   create mode 100755
> > > > > meta/recipes-core/glibc/glibc/run-ptest-time
> > > > >
> > > > > diff --git a/meta/conf/distro/include/ptest-packagelists.inc
> > > > > b/meta/conf/distro/include/ptest-packagelists.inc index
> > > > > 3872bdc942..e460ad1c8b 100644 ---
> > > > > a/meta/conf/distro/include/ptest-packagelists.inc +++
> > > > > b/meta/conf/distro/include/ptest-packagelists.inc @@ -59,6
> > > > > +59,7 @@ PTESTS_FAST = "\ slang-ptest \
> > > > >       wayland-ptest \
> > > > >       zlib-ptest \
> > > > > +    glibc-tests-ptest \
> > > > >   "
> > > > >   PTESTS_FAST:remove:mips64 = "qemu-ptest"
> > > > >   PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
> > > > > diff --git a/meta/recipes-core/glibc/glibc-tests_2.34.bb
> > > > > b/meta/recipes-core/glibc/glibc-tests_2.34.bb new file mode
> > > > > 100644 index 0000000000..896809f700
> > > > > --- /dev/null
> > > > > +++ b/meta/recipes-core/glibc/glibc-tests_2.34.bb
> > > > > @@ -0,0 +1,101 @@
> > > > > +require glibc_${PV}.bb
> > > > > +require glibc-tests.inc
> > > > > +
> > > > > +inherit ptest
> > > > > +
> > > > > +SRC_URI:append = " \
> > > > > +   file://run-ptest-all file://run-ptest-time \
> > > > > +"
> > > > > +
> > > > > +SUMMARY = "glibc tests to be run with ptest"
> > > > > +
> > > > > +# Erase some variables already set by glibc_${PV}
> > > > > +python __anonymous() {
> > > > > +       # Remove packages provided by glibc build, we only need
> > > > > a subset of them
> > > > > +       d.setVar("PACKAGES", "${PN}")
> > > > > +
> > > > > +       d.setVar("PROVIDES", "${PN}")
> > > > > +       d.setVar("RPROVIDES", "${PN} glibc-ptest")
> > > > > +
> > > > > +       d.setVar("RRECOMMENDS", "")
> > > > > +}
> > > > > +
> > > > > +# Remove any lefovers from original glibc recipe
> > > > > +RPROVIDES:${PN} = "${PN}"
> > > > > +RRECOMMENDS:${PN} = ""
> > > > > +RDEPENDS:${PN} = "sed"
> > > > > +
> > > > > +# Just build tests for target - do not run them
> > > > > +do_check:append () {
> > > > > +   oe_runmake -i check run-built-tests=no
> > > > > +}
> > > > > +addtask do_check after do_compile before do_install_ptest_base
> > > > > +
> > > > > +glibc_strip_build_directory () {
> > > > > +   # Delete all non executable files from build directory
> > > > > +   find ${B} ! -executable -type f -delete
> > > > > +
> > > > > +   # Remove build dynamic libraries and links to them as
> > > > > +   # those are already installed in the target device
> > > > > +   find ${B} -type f -name "*.so" -delete
> > > > > +   find ${B} -type l -name "*.so*" -delete
> > > > > +
> > > > > +   # Remove headers (installed with glibc)
> > > > > +   find ${B} -type f -name "*.h" -delete
> > > > > +
> > > > > +   find ${B} -type f -name "isomac" -delete
> > > > > +   find ${B} -type f -name "annexc" -delete
> > > > > +}
> > > > > +
> > > > > +do_install_ptest_base () {
> > > > > +   glibc_strip_build_directory
> > > > > +
> > > > > +   # Install build test programs to the image
> > > > > +   install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
> > > > > +   cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
> > > > > +
> > > > > +   install -d ${D}${PTEST_PATH}
> > > > > +   cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
> > > > > +
> > > > > +   # By default run all glibc tests
> > > > > +   cd ${D}${PTEST_PATH}/
> > > > > +   ln -s run-ptest-all run-ptest
> > > > > +}
> > > > > +
> > > > > +# The datadir directory is required to allow core (and reused)
> > > > > +# glibc cleanup function to finish correctly, as this directory
> > > > > +# is not created for ptests
> > > > > +stash_locale_package_cleanup:prepend () {
> > > > > +   mkdir -p ${PKGD}${datadir}
> > > > > +}
> > > > > +
> > > > > +stash_locale_sysroot_cleanup:prepend () {
> > > > > +   mkdir -p ${SYSROOT_DESTDIR}${datadir}
> > > > > +}
> > > > > +
> > > > > +# Prevent the do_package() task to set 'libc6' prefix
> > > > > +# for glibc tests related packages
> > > > > +python populate_packages:prepend () {
> > > > > +    if d.getVar('DEBIAN_NAMES'):
> > > > > +        d.setVar('DEBIAN_NAMES', '')
> > > > > +}
> > > > > +
> > > > > +FILES:${PN} = "${PTEST_PATH}/* /usr/src/debug/glibc-tests/*"
> > > > > +
> > > > > +EXCLUDE_FROM_SHLIBS = "1"
> > > > > +
> > > > > +# Install debug data in .debug and sources in /usr/src/debug
> > > > > +# It is more handy to have _all_ the sources and symbols in one
> > > > > +# place (package) as this recipe will be used for validation
> > > > > and +# debugging.
> > > > > +PACKAGE_DEBUG_SPLIT_STYLE = "debug"
> > > > > +
> > > > > +# glibc test cases violate by default some Yocto/OE checks
> > > > > (staticdev, +# textrel)
> > > > > +# 'debug-files' - add everything (including debug) into one
> > > > > package +#                 (no need to install/build *-src
> > > > > package) +INSANE_SKIP:${PN} += "staticdev textrel debug-files
> > > > > rpaths" +
> > > > > +deltask do_stash_locale
> > > > > +do_install[noexec] = "1"
> > > > > +do_populate_sysroot[noexec] = "1"
> > > > > diff --git a/meta/recipes-core/glibc/glibc/run-ptest-all
> > > > > b/meta/recipes-core/glibc/glibc/run-ptest-all new file mode
> > > > > 100755 index 0000000000..47bfe5238c
> > > > > --- /dev/null
> > > > > +++ b/meta/recipes-core/glibc/glibc/run-ptest-all
> > > > > @@ -0,0 +1,25 @@
> > > > > +#!/bin/sh
> > > > > +# ptest script for glibc - to run time related tests to
> > > > > +# facilitate Y2038 validation
> > > > > +# Run with 'ptest-runner glibc-tests'
> > > > > +
> > > > > +output() {
> > > > > +  retcode=$?
> > > > > +  if [ $retcode -eq 0 ]
> > > > > +    then echo "PASS: $i"
> > > > > +  elif [ $retcode -eq 77 ]
> > > > > +    then echo "SKIP: $i"
> > > > > +  else echo "FAIL: $i"
> > > > > +  fi
> > > > > +}
> > > > > +
> > > > > +# Allow altering time on the target
> > > > > +export GLIBC_TEST_ALLOW_TIME_SETTING="1"
> > > > > +
> > > > > +tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
> > > > > +
> > > > > +for i in ${tst}
> > > > > +do
> > > > > +   $i >/dev/null 2>&1
> > > > > +   output
> > > > > +done
> > > > > diff --git a/meta/recipes-core/glibc/glibc/run-ptest-time
> > > > > b/meta/recipes-core/glibc/glibc/run-ptest-time new file mode
> > > > > 100755 index 0000000000..24dedc6e32
> > > > > --- /dev/null
> > > > > +++ b/meta/recipes-core/glibc/glibc/run-ptest-time
> > > > > @@ -0,0 +1,30 @@
> > > > > +#!/bin/sh
> > > > > +# ptest script for glibc - to run time related tests to
> > > > > +# facilitate Y2038 validation
> > > > > +# Run with 'ptest-runner glibc-tests'
> > > > > +
> > > > > +output() {
> > > > > +  retcode=$?
> > > > > +  if [ $retcode -eq 0 ]
> > > > > +    then echo "PASS: $i"
> > > > > +  elif [ $retcode -eq 77 ]
> > > > > +    then echo "SKIP: $i"
> > > > > +  else echo "FAIL: $i"
> > > > > +  fi
> > > > > +}
> > > > > +
> > > > > +# Allow altering time on the target
> > > > > +export GLIBC_TEST_ALLOW_TIME_SETTING="1"
> > > > > +
> > > > > +tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
> > > > > +
> > > > > +# Remove '-time64' suffix - those tests are also time
> > > > > +# related
> > > > > +tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
> > > > > +tst_time="${tst_time_tmp} ${tst_time64}"
> > > > > +
> > > > > +for i in ${tst_time}
> > > > > +do
> > > > > +   $i >/dev/null 2>&1
> > > > > +   output
> > > > > +done
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
> > > Best regards,
> > >
> > > Lukasz Majewski
> > >
> > > --
> > >
> > > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > > Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > > Germany Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
> > > lukma@denx.de
>
>
>
>
> Best regards,
>
> Lukasz Majewski
>
> --
>
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[?ukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 13399 bytes --]

Hi Khem,

> On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski <lukma@denx.de> wrote:
> >
> > On Mon, 23 Aug 2021 12:52:44 -0700
> > Khem Raj <raj.khem@gmail.com> wrote:
> >  
> > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski <lukma@denx.de>
> > > wrote:
> > >  
> > > > Hi Khem,
> > > >  
> > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:  
> > > > > > This patch introduces new recipe - namely 'glibc-tests',
> > > > > > which builds and installs glibc test suite to OE/Yocto
> > > > > > built image.
> > > > > >
> > > > > > It reuses code from already available 'glibc-testsuite'
> > > > > > recipe, which is run with 'bitbake glibc-testsuite -c
> > > > > > check' and uses qemu to execute remotely (via SSH) tests on
> > > > > > some emulated machine.
> > > > > >
> > > > > > This recipe installs eligible tests on some rootfs image.
> > > > > > Afterwards, either all of them or only time related subset,
> > > > > > those tests can be executed on the real hardware, to
> > > > > > facilitate validation of this platform with for example
> > > > > > Y2038 problem compliance.
> > > > > >
> > > > > > By default all tests are executed, with 'ptest-runner
> > > > > > glibc-tests'. To test only time related subset - one needs
> > > > > > to call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
> > > > > > ln -s run-ptest-time run-ptest; cd -; ptest-runner
> > > > > > glibc-tests
> > > > > >
> > > > > > To facilitate debugging, source files are provided by
> > > > > > default with the unstripped debugging symbols. Such
> > > > > > approach would reduce the already complex recipe (as it
> > > > > > inherits base glibc one), so there is no need to also
> > > > > > install *-dbg and *-src packages.  
> > > > >
> > > > >
> > > > > does it have to be a separate recipe I wonder if we can have
> > > > > it built as part of glibc itself controlled via ptest knob  
> > > >
> > > > I've followed the glibc-testsuite recipe to provide tests for
> > > > ptest. Test creation is similar to it, but doesn't require QEMU
> > > > run (tests are executed on the HW).
> > > >
> > > > Another rationale was to have some kind of "features"
> > > > separation in different file (liked glibc-mtrace), which is
> > > > easier to maintain and review.
> > > >
> > > > Last but not least - separate recipe (and built binaries) allow
> > > > some kind of magic with selection of used test programs (this
> > > > may be useful if one would like to have different sets of tests
> > > > in different packages)  
> > >
> > > That’s seems ok I think the names are quite confusing now not
> > > that it was not so much better before but now we have glibc-tests
> > > and glibc-testsuites which do same things but in very different
> > > way maybe glibc-testsuite should be renamed to something like
> > > glibc-tests-crosd or some such  
> >
> > I think that glibc-testsuite_2.34.bb shall be renamed to
> > glibc-tests-qemu_2.34.bb as it is more descriptive.  
> 
> is it only runnable in qemu ? I thought we could use a read board as
> well with IP address

It looks like by default the QEMU is used in this recipe. 

This recipe provides custom check-test-wrapper, which can log
into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").

This looks a bit awkward for me, as the goal would be to run built
tests on target (exact) HW without the need of SSH.

It is also easier to debug the code with the real HW.

> 
> >
> > Then, glibc-tests_2.34.bb could be left as it is now, as inheriting
> > ptest requires it to be added as 'glibc-tests-ptest' in
> > meta/conf/distro/include/ptest-packagelists.inc
> >
> > (I've tried to rename it to glibc-tests-ptest_2.34.bb, but then it
> > was required to add glibc-tests-ptest-ptest to the
> > ptest-packagelist.inc file, which is a bit clumsy IMHO).
> >  
> > >  
> > > >  
> > > > >  
> > > > > >
> > > > > > Signed-off-by: Lukasz Majewski <lukma@denx.de>
> > > > > > ---
> > > > > >   .../distro/include/ptest-packagelists.inc     |   1 +
> > > > > >   meta/recipes-core/glibc/glibc-tests_2.34.bb   | 101
> > > > > > ++++++++++++++++++
> > > > > > meta/recipes-core/glibc/glibc/run-ptest-all | 25 +++++
> > > > > > meta/recipes-core/glibc/glibc/run-ptest-time  |  30 ++++++
> > > > > > 4 files changed, 157 insertions(+) create mode 100644
> > > > > > meta/recipes-core/glibc/glibc-tests_2.34.bb create mode
> > > > > > 100755 meta/recipes-core/glibc/glibc/run-ptest-all create
> > > > > > mode 100755 meta/recipes-core/glibc/glibc/run-ptest-time
> > > > > >
> > > > > > diff --git a/meta/conf/distro/include/ptest-packagelists.inc
> > > > > > b/meta/conf/distro/include/ptest-packagelists.inc index
> > > > > > 3872bdc942..e460ad1c8b 100644 ---
> > > > > > a/meta/conf/distro/include/ptest-packagelists.inc +++
> > > > > > b/meta/conf/distro/include/ptest-packagelists.inc @@ -59,6
> > > > > > +59,7 @@ PTESTS_FAST = "\ slang-ptest \
> > > > > >       wayland-ptest \
> > > > > >       zlib-ptest \
> > > > > > +    glibc-tests-ptest \
> > > > > >   "
> > > > > >   PTESTS_FAST:remove:mips64 = "qemu-ptest"
> > > > > >   PTESTS_PROBLEMS:append:mips64 = "qemu-ptest"
> > > > > > diff --git a/meta/recipes-core/glibc/glibc-tests_2.34.bb
> > > > > > b/meta/recipes-core/glibc/glibc-tests_2.34.bb new file mode
> > > > > > 100644 index 0000000000..896809f700
> > > > > > --- /dev/null
> > > > > > +++ b/meta/recipes-core/glibc/glibc-tests_2.34.bb
> > > > > > @@ -0,0 +1,101 @@
> > > > > > +require glibc_${PV}.bb
> > > > > > +require glibc-tests.inc
> > > > > > +
> > > > > > +inherit ptest
> > > > > > +
> > > > > > +SRC_URI:append = " \
> > > > > > +   file://run-ptest-all file://run-ptest-time \
> > > > > > +"
> > > > > > +
> > > > > > +SUMMARY = "glibc tests to be run with ptest"
> > > > > > +
> > > > > > +# Erase some variables already set by glibc_${PV}
> > > > > > +python __anonymous() {
> > > > > > +       # Remove packages provided by glibc build, we only
> > > > > > need a subset of them
> > > > > > +       d.setVar("PACKAGES", "${PN}")
> > > > > > +
> > > > > > +       d.setVar("PROVIDES", "${PN}")
> > > > > > +       d.setVar("RPROVIDES", "${PN} glibc-ptest")
> > > > > > +
> > > > > > +       d.setVar("RRECOMMENDS", "")
> > > > > > +}
> > > > > > +
> > > > > > +# Remove any lefovers from original glibc recipe
> > > > > > +RPROVIDES:${PN} = "${PN}"
> > > > > > +RRECOMMENDS:${PN} = ""
> > > > > > +RDEPENDS:${PN} = "sed"
> > > > > > +
> > > > > > +# Just build tests for target - do not run them
> > > > > > +do_check:append () {
> > > > > > +   oe_runmake -i check run-built-tests=no
> > > > > > +}
> > > > > > +addtask do_check after do_compile before
> > > > > > do_install_ptest_base +
> > > > > > +glibc_strip_build_directory () {
> > > > > > +   # Delete all non executable files from build directory
> > > > > > +   find ${B} ! -executable -type f -delete
> > > > > > +
> > > > > > +   # Remove build dynamic libraries and links to them as
> > > > > > +   # those are already installed in the target device
> > > > > > +   find ${B} -type f -name "*.so" -delete
> > > > > > +   find ${B} -type l -name "*.so*" -delete
> > > > > > +
> > > > > > +   # Remove headers (installed with glibc)
> > > > > > +   find ${B} -type f -name "*.h" -delete
> > > > > > +
> > > > > > +   find ${B} -type f -name "isomac" -delete
> > > > > > +   find ${B} -type f -name "annexc" -delete
> > > > > > +}
> > > > > > +
> > > > > > +do_install_ptest_base () {
> > > > > > +   glibc_strip_build_directory
> > > > > > +
> > > > > > +   # Install build test programs to the image
> > > > > > +   install -d ${D}${PTEST_PATH}/tests/glibc-ptest/
> > > > > > +   cp -r ${B}/* ${D}${PTEST_PATH}/tests/glibc-ptest/
> > > > > > +
> > > > > > +   install -d ${D}${PTEST_PATH}
> > > > > > +   cp ${WORKDIR}/run-ptest-* ${D}${PTEST_PATH}/
> > > > > > +
> > > > > > +   # By default run all glibc tests
> > > > > > +   cd ${D}${PTEST_PATH}/
> > > > > > +   ln -s run-ptest-all run-ptest
> > > > > > +}
> > > > > > +
> > > > > > +# The datadir directory is required to allow core (and
> > > > > > reused) +# glibc cleanup function to finish correctly, as
> > > > > > this directory +# is not created for ptests
> > > > > > +stash_locale_package_cleanup:prepend () {
> > > > > > +   mkdir -p ${PKGD}${datadir}
> > > > > > +}
> > > > > > +
> > > > > > +stash_locale_sysroot_cleanup:prepend () {
> > > > > > +   mkdir -p ${SYSROOT_DESTDIR}${datadir}
> > > > > > +}
> > > > > > +
> > > > > > +# Prevent the do_package() task to set 'libc6' prefix
> > > > > > +# for glibc tests related packages
> > > > > > +python populate_packages:prepend () {
> > > > > > +    if d.getVar('DEBIAN_NAMES'):
> > > > > > +        d.setVar('DEBIAN_NAMES', '')
> > > > > > +}
> > > > > > +
> > > > > > +FILES:${PN} = "${PTEST_PATH}/*
> > > > > > /usr/src/debug/glibc-tests/*" +
> > > > > > +EXCLUDE_FROM_SHLIBS = "1"
> > > > > > +
> > > > > > +# Install debug data in .debug and sources in
> > > > > > /usr/src/debug +# It is more handy to have _all_ the
> > > > > > sources and symbols in one +# place (package) as this
> > > > > > recipe will be used for validation and +# debugging.
> > > > > > +PACKAGE_DEBUG_SPLIT_STYLE = "debug"
> > > > > > +
> > > > > > +# glibc test cases violate by default some Yocto/OE checks
> > > > > > (staticdev, +# textrel)
> > > > > > +# 'debug-files' - add everything (including debug) into one
> > > > > > package +#                 (no need to install/build *-src
> > > > > > package) +INSANE_SKIP:${PN} += "staticdev textrel
> > > > > > debug-files rpaths" +
> > > > > > +deltask do_stash_locale
> > > > > > +do_install[noexec] = "1"
> > > > > > +do_populate_sysroot[noexec] = "1"
> > > > > > diff --git a/meta/recipes-core/glibc/glibc/run-ptest-all
> > > > > > b/meta/recipes-core/glibc/glibc/run-ptest-all new file mode
> > > > > > 100755 index 0000000000..47bfe5238c
> > > > > > --- /dev/null
> > > > > > +++ b/meta/recipes-core/glibc/glibc/run-ptest-all
> > > > > > @@ -0,0 +1,25 @@
> > > > > > +#!/bin/sh
> > > > > > +# ptest script for glibc - to run time related tests to
> > > > > > +# facilitate Y2038 validation
> > > > > > +# Run with 'ptest-runner glibc-tests'
> > > > > > +
> > > > > > +output() {
> > > > > > +  retcode=$?
> > > > > > +  if [ $retcode -eq 0 ]
> > > > > > +    then echo "PASS: $i"
> > > > > > +  elif [ $retcode -eq 77 ]
> > > > > > +    then echo "SKIP: $i"
> > > > > > +  else echo "FAIL: $i"
> > > > > > +  fi
> > > > > > +}
> > > > > > +
> > > > > > +# Allow altering time on the target
> > > > > > +export GLIBC_TEST_ALLOW_TIME_SETTING="1"
> > > > > > +
> > > > > > +tst=$(ls -r ${PWD}/tests/glibc-ptest/*/*)
> > > > > > +
> > > > > > +for i in ${tst}
> > > > > > +do
> > > > > > +   $i >/dev/null 2>&1
> > > > > > +   output
> > > > > > +done
> > > > > > diff --git a/meta/recipes-core/glibc/glibc/run-ptest-time
> > > > > > b/meta/recipes-core/glibc/glibc/run-ptest-time new file mode
> > > > > > 100755 index 0000000000..24dedc6e32
> > > > > > --- /dev/null
> > > > > > +++ b/meta/recipes-core/glibc/glibc/run-ptest-time
> > > > > > @@ -0,0 +1,30 @@
> > > > > > +#!/bin/sh
> > > > > > +# ptest script for glibc - to run time related tests to
> > > > > > +# facilitate Y2038 validation
> > > > > > +# Run with 'ptest-runner glibc-tests'
> > > > > > +
> > > > > > +output() {
> > > > > > +  retcode=$?
> > > > > > +  if [ $retcode -eq 0 ]
> > > > > > +    then echo "PASS: $i"
> > > > > > +  elif [ $retcode -eq 77 ]
> > > > > > +    then echo "SKIP: $i"
> > > > > > +  else echo "FAIL: $i"
> > > > > > +  fi
> > > > > > +}
> > > > > > +
> > > > > > +# Allow altering time on the target
> > > > > > +export GLIBC_TEST_ALLOW_TIME_SETTING="1"
> > > > > > +
> > > > > > +tst_time64=$(ls -r ${PWD}/tests/glibc-ptest/*/*-time64)
> > > > > > +
> > > > > > +# Remove '-time64' suffix - those tests are also time
> > > > > > +# related
> > > > > > +tst_time_tmp=$(sed -e "s/-time64$//" <<< ${tst_time64})
> > > > > > +tst_time="${tst_time_tmp} ${tst_time64}"
> > > > > > +
> > > > > > +for i in ${tst_time}
> > > > > > +do
> > > > > > +   $i >/dev/null 2>&1
> > > > > > +   output
> > > > > > +done
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >  
> > > >
> > > >
> > > >
> > > >
> > > > Best regards,
> > > >
> > > > Lukasz Majewski
> > > >
> > > > --
> > > >
> > > > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > > > Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194
> > > > Groebenzell, Germany Phone: (+49)-8142-66989-59 Fax:
> > > > (+49)-8142-66989-80 Email: lukma@denx.de  
> >
> >
> >
> >
> > Best regards,
> >
> > Lukasz Majewski
> >
> > --
> >
> > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > Germany Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
> > lukma@denx.de  




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[Richard Purdie]

On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote:
> Hi Khem,
> 
> > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski <lukma@denx.de> wrote:
> > > 
> > > On Mon, 23 Aug 2021 12:52:44 -0700
> > > Khem Raj <raj.khem@gmail.com> wrote:
> > >  
> > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski <lukma@denx.de>
> > > > wrote:
> > > >  
> > > > > Hi Khem,
> > > > >  
> > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:  
> > > > > > > This patch introduces new recipe - namely 'glibc-tests',
> > > > > > > which builds and installs glibc test suite to OE/Yocto
> > > > > > > built image.
> > > > > > > 
> > > > > > > It reuses code from already available 'glibc-testsuite'
> > > > > > > recipe, which is run with 'bitbake glibc-testsuite -c
> > > > > > > check' and uses qemu to execute remotely (via SSH) tests on
> > > > > > > some emulated machine.
> > > > > > > 
> > > > > > > This recipe installs eligible tests on some rootfs image.
> > > > > > > Afterwards, either all of them or only time related subset,
> > > > > > > those tests can be executed on the real hardware, to
> > > > > > > facilitate validation of this platform with for example
> > > > > > > Y2038 problem compliance.
> > > > > > > 
> > > > > > > By default all tests are executed, with 'ptest-runner
> > > > > > > glibc-tests'. To test only time related subset - one needs
> > > > > > > to call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
> > > > > > > ln -s run-ptest-time run-ptest; cd -; ptest-runner
> > > > > > > glibc-tests
> > > > > > > 
> > > > > > > To facilitate debugging, source files are provided by
> > > > > > > default with the unstripped debugging symbols. Such
> > > > > > > approach would reduce the already complex recipe (as it
> > > > > > > inherits base glibc one), so there is no need to also
> > > > > > > install *-dbg and *-src packages.  
> > > > > > 
> > > > > > 
> > > > > > does it have to be a separate recipe I wonder if we can have
> > > > > > it built as part of glibc itself controlled via ptest knob  
> > > > > 
> > > > > I've followed the glibc-testsuite recipe to provide tests for
> > > > > ptest. Test creation is similar to it, but doesn't require QEMU
> > > > > run (tests are executed on the HW).
> > > > > 
> > > > > Another rationale was to have some kind of "features"
> > > > > separation in different file (liked glibc-mtrace), which is
> > > > > easier to maintain and review.
> > > > > 
> > > > > Last but not least - separate recipe (and built binaries) allow
> > > > > some kind of magic with selection of used test programs (this
> > > > > may be useful if one would like to have different sets of tests
> > > > > in different packages)  
> > > > 
> > > > That’s seems ok I think the names are quite confusing now not
> > > > that it was not so much better before but now we have glibc-tests
> > > > and glibc-testsuites which do same things but in very different
> > > > way maybe glibc-testsuite should be renamed to something like
> > > > glibc-tests-crosd or some such  
> > > 
> > > I think that glibc-testsuite_2.34.bb shall be renamed to
> > > glibc-tests-qemu_2.34.bb as it is more descriptive.  
> > 
> > is it only runnable in qemu ? I thought we could use a read board as
> > well with IP address
> 
> It looks like by default the QEMU is used in this recipe. 
> 
> This recipe provides custom check-test-wrapper, which can log
> into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").
> 
> This looks a bit awkward for me, as the goal would be to run built
> tests on target (exact) HW without the need of SSH.
> 
> It is also easier to debug the code with the real HW.

I'm a bit worried we're going around in circles with this. We did have a
different form of tests a while back but I was told those didn't work well and
they were replaced with the current setup which worked with the driver in the
glibc source code. This was done around the time we added support for the other
toolchain test suites (binutils+gcc). We ended up with options to use full
system qemu or user mode qemu emulation depending on the speed of the target
(e.g. with kvm or not).

Has this patchset been run with all the tests in glibc or just a subset?

I'm always very nervous about having two ways to do similar things.

Cheers,

Richard

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[?ukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 7344 bytes --]

Hi Richard,

> On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote:
> > Hi Khem,
> >   
> > > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski <lukma@denx.de>
> > > wrote:  
> > > > 
> > > > On Mon, 23 Aug 2021 12:52:44 -0700
> > > > Khem Raj <raj.khem@gmail.com> wrote:
> > > >    
> > > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski
> > > > > <lukma@denx.de> wrote:
> > > > >    
> > > > > > Hi Khem,
> > > > > >    
> > > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:    
> > > > > > > > This patch introduces new recipe - namely 'glibc-tests',
> > > > > > > > which builds and installs glibc test suite to OE/Yocto
> > > > > > > > built image.
> > > > > > > > 
> > > > > > > > It reuses code from already available 'glibc-testsuite'
> > > > > > > > recipe, which is run with 'bitbake glibc-testsuite -c
> > > > > > > > check' and uses qemu to execute remotely (via SSH)
> > > > > > > > tests on some emulated machine.
> > > > > > > > 
> > > > > > > > This recipe installs eligible tests on some rootfs
> > > > > > > > image. Afterwards, either all of them or only time
> > > > > > > > related subset, those tests can be executed on the real
> > > > > > > > hardware, to facilitate validation of this platform
> > > > > > > > with for example Y2038 problem compliance.
> > > > > > > > 
> > > > > > > > By default all tests are executed, with 'ptest-runner
> > > > > > > > glibc-tests'. To test only time related subset - one
> > > > > > > > needs to call: cd /usr/lib/glibc-tests/ptest/; rm
> > > > > > > > run-ptest; \ ln -s run-ptest-time run-ptest; cd -;
> > > > > > > > ptest-runner glibc-tests
> > > > > > > > 
> > > > > > > > To facilitate debugging, source files are provided by
> > > > > > > > default with the unstripped debugging symbols. Such
> > > > > > > > approach would reduce the already complex recipe (as it
> > > > > > > > inherits base glibc one), so there is no need to also
> > > > > > > > install *-dbg and *-src packages.    
> > > > > > > 
> > > > > > > 
> > > > > > > does it have to be a separate recipe I wonder if we can
> > > > > > > have it built as part of glibc itself controlled via
> > > > > > > ptest knob    
> > > > > > 
> > > > > > I've followed the glibc-testsuite recipe to provide tests
> > > > > > for ptest. Test creation is similar to it, but doesn't
> > > > > > require QEMU run (tests are executed on the HW).
> > > > > > 
> > > > > > Another rationale was to have some kind of "features"
> > > > > > separation in different file (liked glibc-mtrace), which is
> > > > > > easier to maintain and review.
> > > > > > 
> > > > > > Last but not least - separate recipe (and built binaries)
> > > > > > allow some kind of magic with selection of used test
> > > > > > programs (this may be useful if one would like to have
> > > > > > different sets of tests in different packages)    
> > > > > 
> > > > > That’s seems ok I think the names are quite confusing now not
> > > > > that it was not so much better before but now we have
> > > > > glibc-tests and glibc-testsuites which do same things but in
> > > > > very different way maybe glibc-testsuite should be renamed to
> > > > > something like glibc-tests-crosd or some such    
> > > > 
> > > > I think that glibc-testsuite_2.34.bb shall be renamed to
> > > > glibc-tests-qemu_2.34.bb as it is more descriptive.    
> > > 
> > > is it only runnable in qemu ? I thought we could use a read board
> > > as well with IP address  
> > 
> > It looks like by default the QEMU is used in this recipe. 
> > 
> > This recipe provides custom check-test-wrapper, which can log
> > into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").
> > 
> > This looks a bit awkward for me, as the goal would be to run built
> > tests on target (exact) HW without the need of SSH.
> > 
> > It is also easier to debug the code with the real HW.  
> 
> I'm a bit worried we're going around in circles with this. We did
> have a different form of tests a while back but I was told those
> didn't work well and they were replaced with the current setup which
> worked with the driver in the glibc source code.

I've looked into the glibc-testing.inc from SHA1:
0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b

There glibc tests were built and copied on the target via NFS.

> This was done around
> the time we added support for the other toolchain test suites
> (binutils+gcc).

I'm not familiar with the "binutils+gcc" issue, which was solved in the
past. Could you share any reference?

> We ended up with options to use full system qemu or
> user mode qemu emulation depending on the speed of the target (e.g.
> with kvm or not).

From what I see in the recipe - you either run user mode QEMU by
default (at least when I do run it):

NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i
QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot
QEMU_OPTIONS=qemu-arm -r 5.1.0  SSH_HOST=localhost SSH_HOS T_USER=root
SSH_HOST_PORT=2222
test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper
user check

Or you log to the board via SSH (the 'ssh' mode).

> 
> Has this patchset been run with all the tests in glibc or just a
> subset?

This patch set allows running all tests [*], or (preferably in my use
case) only a time related ones. The latter is to validate Y2038 code,
which testing has two issues:

1. It cannot be tested with QEMU user mode [1]. Instead "linux
timespaces" were suggested, but it lacks support for CLOCK_REALTIME
manipulation for those tests [2].

2. Testing with SSH is possible, but is not as robust as expected (i.e.
imposes SSHD on the target board, and ETH connection).

> 
> I'm always very nervous about having two ways to do similar things.

Technically, I just reuse the glibc-testsuite_2.34.bb to the point
where tests are _only_ built. Then I pack those binaries and install on
the target's rootfs to be run with ptest framework.

Such approach has several advantages:

- No need for SSH (which can hang, as you need sshfs in user space for
  the setup)

- Run on the real HW (and environment) with the same version of glibc
  on the real target.

- Sources and debugging symbols available on the target, so easier GDB
  debugging.

- Using ptest framework to execute tests

- Time related tests are "self contained", so could be executed without
  built data.

At least for Y2038 validation such approach seems more appealing.

> 
> Cheers,
> 
> Richard
> 

Links:
[1] - https://www.mail-archive.com/qemu-discuss@nongnu.org/msg05898.html
[2] -
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2361284.html

Note:
[*] - some glibc tests rely on the built directory to provide correct
data/environment for execution. Time related tests are (and should be)
self contained, so those could be just executed as binaries.

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[?ukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 8101 bytes --]

Hi Richard, Khem,

> Hi Richard,
> 
> > On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote:  
> > > Hi Khem,
> > >     
> > > > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski <lukma@denx.de>
> > > > wrote:    
> > > > > 
> > > > > On Mon, 23 Aug 2021 12:52:44 -0700
> > > > > Khem Raj <raj.khem@gmail.com> wrote:
> > > > >      
> > > > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski
> > > > > > <lukma@denx.de> wrote:
> > > > > >      
> > > > > > > Hi Khem,
> > > > > > >      
> > > > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:      
> > > > > > > > > This patch introduces new recipe - namely
> > > > > > > > > 'glibc-tests', which builds and installs glibc test
> > > > > > > > > suite to OE/Yocto built image.
> > > > > > > > > 
> > > > > > > > > It reuses code from already available
> > > > > > > > > 'glibc-testsuite' recipe, which is run with 'bitbake
> > > > > > > > > glibc-testsuite -c check' and uses qemu to execute
> > > > > > > > > remotely (via SSH) tests on some emulated machine.
> > > > > > > > > 
> > > > > > > > > This recipe installs eligible tests on some rootfs
> > > > > > > > > image. Afterwards, either all of them or only time
> > > > > > > > > related subset, those tests can be executed on the
> > > > > > > > > real hardware, to facilitate validation of this
> > > > > > > > > platform with for example Y2038 problem compliance.
> > > > > > > > > 
> > > > > > > > > By default all tests are executed, with 'ptest-runner
> > > > > > > > > glibc-tests'. To test only time related subset - one
> > > > > > > > > needs to call: cd /usr/lib/glibc-tests/ptest/; rm
> > > > > > > > > run-ptest; \ ln -s run-ptest-time run-ptest; cd -;
> > > > > > > > > ptest-runner glibc-tests
> > > > > > > > > 
> > > > > > > > > To facilitate debugging, source files are provided by
> > > > > > > > > default with the unstripped debugging symbols. Such
> > > > > > > > > approach would reduce the already complex recipe (as
> > > > > > > > > it inherits base glibc one), so there is no need to
> > > > > > > > > also install *-dbg and *-src packages.      
> > > > > > > > 
> > > > > > > > 
> > > > > > > > does it have to be a separate recipe I wonder if we can
> > > > > > > > have it built as part of glibc itself controlled via
> > > > > > > > ptest knob      
> > > > > > > 
> > > > > > > I've followed the glibc-testsuite recipe to provide tests
> > > > > > > for ptest. Test creation is similar to it, but doesn't
> > > > > > > require QEMU run (tests are executed on the HW).
> > > > > > > 
> > > > > > > Another rationale was to have some kind of "features"
> > > > > > > separation in different file (liked glibc-mtrace), which
> > > > > > > is easier to maintain and review.
> > > > > > > 
> > > > > > > Last but not least - separate recipe (and built binaries)
> > > > > > > allow some kind of magic with selection of used test
> > > > > > > programs (this may be useful if one would like to have
> > > > > > > different sets of tests in different packages)      
> > > > > > 
> > > > > > That’s seems ok I think the names are quite confusing now
> > > > > > not that it was not so much better before but now we have
> > > > > > glibc-tests and glibc-testsuites which do same things but in
> > > > > > very different way maybe glibc-testsuite should be renamed
> > > > > > to something like glibc-tests-crosd or some such      
> > > > > 
> > > > > I think that glibc-testsuite_2.34.bb shall be renamed to
> > > > > glibc-tests-qemu_2.34.bb as it is more descriptive.      
> > > > 
> > > > is it only runnable in qemu ? I thought we could use a read
> > > > board as well with IP address    
> > > 
> > > It looks like by default the QEMU is used in this recipe. 
> > > 
> > > This recipe provides custom check-test-wrapper, which can log
> > > into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").
> > > 
> > > This looks a bit awkward for me, as the goal would be to run built
> > > tests on target (exact) HW without the need of SSH.
> > > 
> > > It is also easier to debug the code with the real HW.    
> > 
> > I'm a bit worried we're going around in circles with this. We did
> > have a different form of tests a while back but I was told those
> > didn't work well and they were replaced with the current setup which
> > worked with the driver in the glibc source code.  
> 
> I've looked into the glibc-testing.inc from SHA1:
> 0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b
> 
> There glibc tests were built and copied on the target via NFS.
> 
> > This was done around
> > the time we added support for the other toolchain test suites
> > (binutils+gcc).  
> 
> I'm not familiar with the "binutils+gcc" issue, which was solved in
> the past. Could you share any reference?
> 
> > We ended up with options to use full system qemu or
> > user mode qemu emulation depending on the speed of the target (e.g.
> > with kvm or not).  
> 
> From what I see in the recipe - you either run user mode QEMU by
> default (at least when I do run it):
> 
> NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i
> QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot
> QEMU_OPTIONS=qemu-arm -r 5.1.0  SSH_HOST=localhost SSH_HOS T_USER=root
> SSH_HOST_PORT=2222
> test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper
> user check
> 
> Or you log to the board via SSH (the 'ssh' mode).
> 
> > 
> > Has this patchset been run with all the tests in glibc or just a
> > subset?  
> 
> This patch set allows running all tests [*], or (preferably in my use
> case) only a time related ones. The latter is to validate Y2038 code,
> which testing has two issues:
> 
> 1. It cannot be tested with QEMU user mode [1]. Instead "linux
> timespaces" were suggested, but it lacks support for CLOCK_REALTIME
> manipulation for those tests [2].
> 
> 2. Testing with SSH is possible, but is not as robust as expected
> (i.e. imposes SSHD on the target board, and ETH connection).
> 
> > 
> > I'm always very nervous about having two ways to do similar things.
> >  
> 
> Technically, I just reuse the glibc-testsuite_2.34.bb to the point
> where tests are _only_ built. Then I pack those binaries and install
> on the target's rootfs to be run with ptest framework.
> 
> Such approach has several advantages:
> 
> - No need for SSH (which can hang, as you need sshfs in user space for
>   the setup)
> 
> - Run on the real HW (and environment) with the same version of glibc
>   on the real target.
> 
> - Sources and debugging symbols available on the target, so easier GDB
>   debugging.
> 
> - Using ptest framework to execute tests
> 
> - Time related tests are "self contained", so could be executed
> without built data.
> 
> At least for Y2038 validation such approach seems more appealing.
> 

Do you have any more feedback regarding this patch?

> > 
> > Cheers,
> > 
> > Richard
> >   
> 
> Links:
> [1] -
> https://www.mail-archive.com/qemu-discuss@nongnu.org/msg05898.html
> [2] -
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2361284.html
> 
> Note:
> [*] - some glibc tests rely on the built directory to provide correct
> data/environment for execution. Time related tests are (and should be)
> self contained, so those could be just executed as binaries.
> 
> Best regards,
> 
> Lukasz Majewski
> 
> --
> 
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
> lukma@denx.de




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[Richard Purdie]

On Thu, 2021-08-26 at 14:27 +0200, Lukasz Majewski wrote:
> Hi Richard, Khem,
> 
> > Hi Richard,
> > 
> > > On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote:  
> > > > Hi Khem,
> > > >     
> > > > > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski <lukma@denx.de>
> > > > > wrote:    
> > > > > > 
> > > > > > On Mon, 23 Aug 2021 12:52:44 -0700
> > > > > > Khem Raj <raj.khem@gmail.com> wrote:
> > > > > >      
> > > > > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski
> > > > > > > <lukma@denx.de> wrote:
> > > > > > >      
> > > > > > > > Hi Khem,
> > > > > > > >      
> > > > > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:      
> > > > > > > > > > This patch introduces new recipe - namely
> > > > > > > > > > 'glibc-tests', which builds and installs glibc test
> > > > > > > > > > suite to OE/Yocto built image.
> > > > > > > > > > 
> > > > > > > > > > It reuses code from already available
> > > > > > > > > > 'glibc-testsuite' recipe, which is run with 'bitbake
> > > > > > > > > > glibc-testsuite -c check' and uses qemu to execute
> > > > > > > > > > remotely (via SSH) tests on some emulated machine.
> > > > > > > > > > 
> > > > > > > > > > This recipe installs eligible tests on some rootfs
> > > > > > > > > > image. Afterwards, either all of them or only time
> > > > > > > > > > related subset, those tests can be executed on the
> > > > > > > > > > real hardware, to facilitate validation of this
> > > > > > > > > > platform with for example Y2038 problem compliance.
> > > > > > > > > > 
> > > > > > > > > > By default all tests are executed, with 'ptest-runner
> > > > > > > > > > glibc-tests'. To test only time related subset - one
> > > > > > > > > > needs to call: cd /usr/lib/glibc-tests/ptest/; rm
> > > > > > > > > > run-ptest; \ ln -s run-ptest-time run-ptest; cd -;
> > > > > > > > > > ptest-runner glibc-tests
> > > > > > > > > > 
> > > > > > > > > > To facilitate debugging, source files are provided by
> > > > > > > > > > default with the unstripped debugging symbols. Such
> > > > > > > > > > approach would reduce the already complex recipe (as
> > > > > > > > > > it inherits base glibc one), so there is no need to
> > > > > > > > > > also install *-dbg and *-src packages.      
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > does it have to be a separate recipe I wonder if we can
> > > > > > > > > have it built as part of glibc itself controlled via
> > > > > > > > > ptest knob      
> > > > > > > > 
> > > > > > > > I've followed the glibc-testsuite recipe to provide tests
> > > > > > > > for ptest. Test creation is similar to it, but doesn't
> > > > > > > > require QEMU run (tests are executed on the HW).
> > > > > > > > 
> > > > > > > > Another rationale was to have some kind of "features"
> > > > > > > > separation in different file (liked glibc-mtrace), which
> > > > > > > > is easier to maintain and review.
> > > > > > > > 
> > > > > > > > Last but not least - separate recipe (and built binaries)
> > > > > > > > allow some kind of magic with selection of used test
> > > > > > > > programs (this may be useful if one would like to have
> > > > > > > > different sets of tests in different packages)      
> > > > > > > 
> > > > > > > That’s seems ok I think the names are quite confusing now
> > > > > > > not that it was not so much better before but now we have
> > > > > > > glibc-tests and glibc-testsuites which do same things but in
> > > > > > > very different way maybe glibc-testsuite should be renamed
> > > > > > > to something like glibc-tests-crosd or some such      
> > > > > > 
> > > > > > I think that glibc-testsuite_2.34.bb shall be renamed to
> > > > > > glibc-tests-qemu_2.34.bb as it is more descriptive.      
> > > > > 
> > > > > is it only runnable in qemu ? I thought we could use a read
> > > > > board as well with IP address    
> > > > 
> > > > It looks like by default the QEMU is used in this recipe. 
> > > > 
> > > > This recipe provides custom check-test-wrapper, which can log
> > > > into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").
> > > > 
> > > > This looks a bit awkward for me, as the goal would be to run built
> > > > tests on target (exact) HW without the need of SSH.
> > > > 
> > > > It is also easier to debug the code with the real HW.    
> > > 
> > > I'm a bit worried we're going around in circles with this. We did
> > > have a different form of tests a while back but I was told those
> > > didn't work well and they were replaced with the current setup which
> > > worked with the driver in the glibc source code.  
> > 
> > I've looked into the glibc-testing.inc from SHA1:
> > 0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b
> > 
> > There glibc tests were built and copied on the target via NFS.
> > 
> > > This was done around
> > > the time we added support for the other toolchain test suites
> > > (binutils+gcc).  
> > 
> > I'm not familiar with the "binutils+gcc" issue, which was solved in
> > the past. Could you share any reference?
> > 
> > > We ended up with options to use full system qemu or
> > > user mode qemu emulation depending on the speed of the target (e.g.
> > > with kvm or not).  
> > 
> > From what I see in the recipe - you either run user mode QEMU by
> > default (at least when I do run it):
> > 
> > NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i
> > QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot
> > QEMU_OPTIONS=qemu-arm -r 5.1.0  SSH_HOST=localhost SSH_HOS T_USER=root
> > SSH_HOST_PORT=2222
> > test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper
> > user check
> > 
> > Or you log to the board via SSH (the 'ssh' mode).
> > 
> > > 
> > > Has this patchset been run with all the tests in glibc or just a
> > > subset?  
> > 
> > This patch set allows running all tests [*], or (preferably in my use
> > case) only a time related ones. The latter is to validate Y2038 code,
> > which testing has two issues:
> > 
> > 1. It cannot be tested with QEMU user mode [1]. Instead "linux
> > timespaces" were suggested, but it lacks support for CLOCK_REALTIME
> > manipulation for those tests [2].
> > 
> > 2. Testing with SSH is possible, but is not as robust as expected
> > (i.e. imposes SSHD on the target board, and ETH connection).
> > 
> > > 
> > > I'm always very nervous about having two ways to do similar things.
> > >  
> > 
> > Technically, I just reuse the glibc-testsuite_2.34.bb to the point
> > where tests are _only_ built. Then I pack those binaries and install
> > on the target's rootfs to be run with ptest framework.
> > 
> > Such approach has several advantages:
> > 
> > - No need for SSH (which can hang, as you need sshfs in user space for
> >   the setup)
> > 
> > - Run on the real HW (and environment) with the same version of glibc
> >   on the real target.
> > 
> > - Sources and debugging symbols available on the target, so easier GDB
> >   debugging.
> > 
> > - Using ptest framework to execute tests
> > 
> > - Time related tests are "self contained", so could be executed
> > without built data.
> > 
> > At least for Y2038 validation such approach seems more appealing.
> > 
> 
> Do you have any more feedback regarding this patch?

We're at feature freeze and this wasn't a planned change so it is too late for
this release cycle, I'm struggling with the things that were planned.

I need to spend some time looking at what the other code is doing and how we're
using it on the autobuilder and whether we want to change to use the ptest
approach or not.

I added Nathan Rossi to cc since he worked on the other code for this and may
have more insight.

Cheers,

Richard

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[?ukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 9287 bytes --]

Hi Richard,

> On Thu, 2021-08-26 at 14:27 +0200, Lukasz Majewski wrote:
> > Hi Richard, Khem,
> >   
> > > Hi Richard,
> > >   
> > > > On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote:    
> > > > > Hi Khem,
> > > > >       
> > > > > > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski
> > > > > > <lukma@denx.de> wrote:      
> > > > > > > 
> > > > > > > On Mon, 23 Aug 2021 12:52:44 -0700
> > > > > > > Khem Raj <raj.khem@gmail.com> wrote:
> > > > > > >        
> > > > > > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski
> > > > > > > > <lukma@denx.de> wrote:
> > > > > > > >        
> > > > > > > > > Hi Khem,
> > > > > > > > >        
> > > > > > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:        
> > > > > > > > > > > This patch introduces new recipe - namely
> > > > > > > > > > > 'glibc-tests', which builds and installs glibc
> > > > > > > > > > > test suite to OE/Yocto built image.
> > > > > > > > > > > 
> > > > > > > > > > > It reuses code from already available
> > > > > > > > > > > 'glibc-testsuite' recipe, which is run with
> > > > > > > > > > > 'bitbake glibc-testsuite -c check' and uses qemu
> > > > > > > > > > > to execute remotely (via SSH) tests on some
> > > > > > > > > > > emulated machine.
> > > > > > > > > > > 
> > > > > > > > > > > This recipe installs eligible tests on some rootfs
> > > > > > > > > > > image. Afterwards, either all of them or only time
> > > > > > > > > > > related subset, those tests can be executed on the
> > > > > > > > > > > real hardware, to facilitate validation of this
> > > > > > > > > > > platform with for example Y2038 problem
> > > > > > > > > > > compliance.
> > > > > > > > > > > 
> > > > > > > > > > > By default all tests are executed, with
> > > > > > > > > > > 'ptest-runner glibc-tests'. To test only time
> > > > > > > > > > > related subset - one needs to call: cd
> > > > > > > > > > > /usr/lib/glibc-tests/ptest/; rm run-ptest; \ ln
> > > > > > > > > > > -s run-ptest-time run-ptest; cd -; ptest-runner
> > > > > > > > > > > glibc-tests
> > > > > > > > > > > 
> > > > > > > > > > > To facilitate debugging, source files are
> > > > > > > > > > > provided by default with the unstripped debugging
> > > > > > > > > > > symbols. Such approach would reduce the already
> > > > > > > > > > > complex recipe (as it inherits base glibc one),
> > > > > > > > > > > so there is no need to also install *-dbg and
> > > > > > > > > > > *-src packages.        
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > does it have to be a separate recipe I wonder if we
> > > > > > > > > > can have it built as part of glibc itself
> > > > > > > > > > controlled via ptest knob        
> > > > > > > > > 
> > > > > > > > > I've followed the glibc-testsuite recipe to provide
> > > > > > > > > tests for ptest. Test creation is similar to it, but
> > > > > > > > > doesn't require QEMU run (tests are executed on the
> > > > > > > > > HW).
> > > > > > > > > 
> > > > > > > > > Another rationale was to have some kind of "features"
> > > > > > > > > separation in different file (liked glibc-mtrace),
> > > > > > > > > which is easier to maintain and review.
> > > > > > > > > 
> > > > > > > > > Last but not least - separate recipe (and built
> > > > > > > > > binaries) allow some kind of magic with selection of
> > > > > > > > > used test programs (this may be useful if one would
> > > > > > > > > like to have different sets of tests in different
> > > > > > > > > packages)        
> > > > > > > > 
> > > > > > > > That’s seems ok I think the names are quite confusing
> > > > > > > > now not that it was not so much better before but now
> > > > > > > > we have glibc-tests and glibc-testsuites which do same
> > > > > > > > things but in very different way maybe glibc-testsuite
> > > > > > > > should be renamed to something like glibc-tests-crosd
> > > > > > > > or some such        
> > > > > > > 
> > > > > > > I think that glibc-testsuite_2.34.bb shall be renamed to
> > > > > > > glibc-tests-qemu_2.34.bb as it is more descriptive.
> > > > > > >  
> > > > > > 
> > > > > > is it only runnable in qemu ? I thought we could use a read
> > > > > > board as well with IP address      
> > > > > 
> > > > > It looks like by default the QEMU is used in this recipe. 
> > > > > 
> > > > > This recipe provides custom check-test-wrapper, which can log
> > > > > into remote board via ssh (when TOOLCHAIN_TEST_TARGET =
> > > > > "ssh").
> > > > > 
> > > > > This looks a bit awkward for me, as the goal would be to run
> > > > > built tests on target (exact) HW without the need of SSH.
> > > > > 
> > > > > It is also easier to debug the code with the real HW.      
> > > > 
> > > > I'm a bit worried we're going around in circles with this. We
> > > > did have a different form of tests a while back but I was told
> > > > those didn't work well and they were replaced with the current
> > > > setup which worked with the driver in the glibc source code.    
> > > 
> > > I've looked into the glibc-testing.inc from SHA1:
> > > 0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b
> > > 
> > > There glibc tests were built and copied on the target via NFS.
> > >   
> > > > This was done around
> > > > the time we added support for the other toolchain test suites
> > > > (binutils+gcc).    
> > > 
> > > I'm not familiar with the "binutils+gcc" issue, which was solved
> > > in the past. Could you share any reference?
> > >   
> > > > We ended up with options to use full system qemu or
> > > > user mode qemu emulation depending on the speed of the target
> > > > (e.g. with kvm or not).    
> > > 
> > > From what I see in the recipe - you either run user mode QEMU by
> > > default (at least when I do run it):
> > > 
> > > NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i
> > > QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot
> > > QEMU_OPTIONS=qemu-arm -r 5.1.0  SSH_HOST=localhost SSH_HOS
> > > T_USER=root SSH_HOST_PORT=2222
> > > test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper
> > > user check
> > > 
> > > Or you log to the board via SSH (the 'ssh' mode).
> > >   
> > > > 
> > > > Has this patchset been run with all the tests in glibc or just a
> > > > subset?    
> > > 
> > > This patch set allows running all tests [*], or (preferably in my
> > > use case) only a time related ones. The latter is to validate
> > > Y2038 code, which testing has two issues:
> > > 
> > > 1. It cannot be tested with QEMU user mode [1]. Instead "linux
> > > timespaces" were suggested, but it lacks support for
> > > CLOCK_REALTIME manipulation for those tests [2].
> > > 
> > > 2. Testing with SSH is possible, but is not as robust as expected
> > > (i.e. imposes SSHD on the target board, and ETH connection).
> > >   
> > > > 
> > > > I'm always very nervous about having two ways to do similar
> > > > things. 
> > > 
> > > Technically, I just reuse the glibc-testsuite_2.34.bb to the point
> > > where tests are _only_ built. Then I pack those binaries and
> > > install on the target's rootfs to be run with ptest framework.
> > > 
> > > Such approach has several advantages:
> > > 
> > > - No need for SSH (which can hang, as you need sshfs in user
> > > space for the setup)
> > > 
> > > - Run on the real HW (and environment) with the same version of
> > > glibc on the real target.
> > > 
> > > - Sources and debugging symbols available on the target, so
> > > easier GDB debugging.
> > > 
> > > - Using ptest framework to execute tests
> > > 
> > > - Time related tests are "self contained", so could be executed
> > > without built data.
> > > 
> > > At least for Y2038 validation such approach seems more appealing.
> > >   
> > 
> > Do you have any more feedback regarding this patch?  
> 
> We're at feature freeze and this wasn't a planned change so it is too
> late for this release cycle,

I see. Thanks for the update.

BTW: When one can expect the Yocto/poky would accept new patches again?

Honister is going to be released on October 2021 [1]. After this date
this patch is going to be reconsidered?

> I'm struggling with the things that were
> planned.

I can imagine that after the "override" change a lot of attention is
necessary.

> 
> I need to spend some time looking at what the other code is doing and
> how we're using it on the autobuilder and whether we want to change
> to use the ptest approach or not.

I see.

> 
> I added Nathan Rossi to cc since he worked on the other code for this
> and may have more insight.

I do hope that Nathan will provide some valuable input as well, as
wrote the glibc-testsuite_%.bb recipe.

> 
> Cheers,
> 
> Richard
> 

Links:

[1] - https://wiki.yoctoproject.org/wiki/Releases


Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[Richard Purdie]

On Thu, 2021-08-26 at 16:12 +0200, Lukasz Majewski wrote:
> > On Thu, 2021-08-26 at 14:27 +0200, Lukasz Majewski wrote:
> > > 
> > > Do you have any more feedback regarding this patch?  
> > 
> > We're at feature freeze and this wasn't a planned change so it is too
> > late for this release cycle,
> 
> I see. Thanks for the update.
> 
> BTW: When one can expect the Yocto/poky would accept new patches again?
>
> 
> Honister is going to be released on October 2021 [1]. After this date
> this patch is going to be reconsidered?

I'd expect new development to start being looked at around the time of the
release, yes.

> > I'm struggling with the things that were
> > planned.
> 
> I can imagine that after the "override" change a lot of attention is
> necessary.
> 
> > 
> > I need to spend some time looking at what the other code is doing and
> > how we're using it on the autobuilder and whether we want to change
> > to use the ptest approach or not.
> 
> I see.
> 
> > 
> > I added Nathan Rossi to cc since he worked on the other code for this
> > and may have more insight.
> 
> I do hope that Nathan will provide some valuable input as well, as
> wrote the glibc-testsuite_%.bb recipe.

If you search the commits, e.g.:

http://git.yoctoproject.org/cgit.cgi/poky/log/?qt=grep&q=Rossi

you'll see he also worked on the gcc/binutils tests I mentioned.

Also, I'm not sure if you found them but these are run on our infrastructure
here:

https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/3877/steps/19/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/3892/steps/20/logs/stdio

(one of the above uses user mode qemu iirc, the other uses system mode).

We then get reports like:

https://autobuilder.yocto.io/pub/non-release/20210825-18/testresults/testresult-report.txt

Cheers,

Richard

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[Nathan Rossi]

On Thu, 26 Aug 2021 at 23:38, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Thu, 2021-08-26 at 14:27 +0200, Lukasz Majewski wrote:
> > Hi Richard, Khem,
> >
> > > Hi Richard,
> > >
> > > > On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote:
> > > > > Hi Khem,
> > > > >
> > > > > > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski <lukma@denx.de>
> > > > > > wrote:
> > > > > > >
> > > > > > > On Mon, 23 Aug 2021 12:52:44 -0700
> > > > > > > Khem Raj <raj.khem@gmail.com> wrote:
> > > > > > >
> > > > > > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski
> > > > > > > > <lukma@denx.de> wrote:
> > > > > > > >
> > > > > > > > > Hi Khem,
> > > > > > > > >
> > > > > > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:
> > > > > > > > > > > This patch introduces new recipe - namely
> > > > > > > > > > > 'glibc-tests', which builds and installs glibc test
> > > > > > > > > > > suite to OE/Yocto built image.
> > > > > > > > > > >
> > > > > > > > > > > It reuses code from already available
> > > > > > > > > > > 'glibc-testsuite' recipe, which is run with 'bitbake
> > > > > > > > > > > glibc-testsuite -c check' and uses qemu to execute
> > > > > > > > > > > remotely (via SSH) tests on some emulated machine.
> > > > > > > > > > >
> > > > > > > > > > > This recipe installs eligible tests on some rootfs
> > > > > > > > > > > image. Afterwards, either all of them or only time
> > > > > > > > > > > related subset, those tests can be executed on the
> > > > > > > > > > > real hardware, to facilitate validation of this
> > > > > > > > > > > platform with for example Y2038 problem compliance.
> > > > > > > > > > >
> > > > > > > > > > > By default all tests are executed, with 'ptest-runner
> > > > > > > > > > > glibc-tests'. To test only time related subset - one
> > > > > > > > > > > needs to call: cd /usr/lib/glibc-tests/ptest/; rm
> > > > > > > > > > > run-ptest; \ ln -s run-ptest-time run-ptest; cd -;
> > > > > > > > > > > ptest-runner glibc-tests
> > > > > > > > > > >
> > > > > > > > > > > To facilitate debugging, source files are provided by
> > > > > > > > > > > default with the unstripped debugging symbols. Such
> > > > > > > > > > > approach would reduce the already complex recipe (as
> > > > > > > > > > > it inherits base glibc one), so there is no need to
> > > > > > > > > > > also install *-dbg and *-src packages.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > does it have to be a separate recipe I wonder if we can
> > > > > > > > > > have it built as part of glibc itself controlled via
> > > > > > > > > > ptest knob
> > > > > > > > >
> > > > > > > > > I've followed the glibc-testsuite recipe to provide tests
> > > > > > > > > for ptest. Test creation is similar to it, but doesn't
> > > > > > > > > require QEMU run (tests are executed on the HW).
> > > > > > > > >
> > > > > > > > > Another rationale was to have some kind of "features"
> > > > > > > > > separation in different file (liked glibc-mtrace), which
> > > > > > > > > is easier to maintain and review.
> > > > > > > > >
> > > > > > > > > Last but not least - separate recipe (and built binaries)
> > > > > > > > > allow some kind of magic with selection of used test
> > > > > > > > > programs (this may be useful if one would like to have
> > > > > > > > > different sets of tests in different packages)
> > > > > > > >
> > > > > > > > That’s seems ok I think the names are quite confusing now
> > > > > > > > not that it was not so much better before but now we have
> > > > > > > > glibc-tests and glibc-testsuites which do same things but in
> > > > > > > > very different way maybe glibc-testsuite should be renamed
> > > > > > > > to something like glibc-tests-crosd or some such
> > > > > > >
> > > > > > > I think that glibc-testsuite_2.34.bb shall be renamed to
> > > > > > > glibc-tests-qemu_2.34.bb as it is more descriptive.
> > > > > >
> > > > > > is it only runnable in qemu ? I thought we could use a read
> > > > > > board as well with IP address
> > > > >
> > > > > It looks like by default the QEMU is used in this recipe.
> > > > >
> > > > > This recipe provides custom check-test-wrapper, which can log
> > > > > into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").
> > > > >
> > > > > This looks a bit awkward for me, as the goal would be to run built
> > > > > tests on target (exact) HW without the need of SSH.
> > > > >
> > > > > It is also easier to debug the code with the real HW.
> > > >
> > > > I'm a bit worried we're going around in circles with this. We did
> > > > have a different form of tests a while back but I was told those
> > > > didn't work well and they were replaced with the current setup which
> > > > worked with the driver in the glibc source code.
> > >
> > > I've looked into the glibc-testing.inc from SHA1:
> > > 0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b
> > >
> > > There glibc tests were built and copied on the target via NFS.
> > >
> > > > This was done around
> > > > the time we added support for the other toolchain test suites
> > > > (binutils+gcc).
> > >
> > > I'm not familiar with the "binutils+gcc" issue, which was solved in
> > > the past. Could you share any reference?
> > >
> > > > We ended up with options to use full system qemu or
> > > > user mode qemu emulation depending on the speed of the target (e.g.
> > > > with kvm or not).
> > >
> > > From what I see in the recipe - you either run user mode QEMU by
> > > default (at least when I do run it):
> > >
> > > NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i
> > > QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot
> > > QEMU_OPTIONS=qemu-arm -r 5.1.0  SSH_HOST=localhost SSH_HOS T_USER=root
> > > SSH_HOST_PORT=2222
> > > test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper
> > > user check
> > >
> > > Or you log to the board via SSH (the 'ssh' mode).
> > >
> > > >
> > > > Has this patchset been run with all the tests in glibc or just a
> > > > subset?
> > >
> > > This patch set allows running all tests [*], or (preferably in my use
> > > case) only a time related ones. The latter is to validate Y2038 code,
> > > which testing has two issues:
> > >
> > > 1. It cannot be tested with QEMU user mode [1]. Instead "linux
> > > timespaces" were suggested, but it lacks support for CLOCK_REALTIME
> > > manipulation for those tests [2].

So just a note, the qemu user mode testing is not a complete solution.
There are a number of problems with it, but in general it is still
quite useful for validating general functionally, especially when
implementing a new platform. A big one is due to the ability to run
tests in parallel with user mode, it easily reduces test run times by
an order of magnitude (compared to qemu system, which for some
platforms is faster than real hardware).

The reason why qemu user mode is the default behaviour is because
running "bitbake glibc-testsuite -c check" will work without any
external dependencies (the remote machine, NFS/sshfs, etc.).

> > >
> > > 2. Testing with SSH is possible, but is not as robust as expected
> > > (i.e. imposes SSHD on the target board, and ETH connection).
> > >
> > > >
> > > > I'm always very nervous about having two ways to do similar things.
> > > >
> > >
> > > Technically, I just reuse the glibc-testsuite_2.34.bb to the point
> > > where tests are _only_ built. Then I pack those binaries and install
> > > on the target's rootfs to be run with ptest framework.

So it appears the make variable "run-built-tests" excludes the
building of some tests. So I am not sure if the complete test suite
can be built in this way. Though I can see some value in having the
glibc-testsuite recipe being able to only build the tests for
development purposes (e.g. TOOLCHAIN_TEST_TARGET = "build"?).

When I had implemented the testsuite recipe the glibc testsuite relied
quite heavily on the environment it was built and run in, in
particular around configuration. That is probably still an issue.

It would be interesting to know how much of the test suite can be
run/passes and how that compares to the results of existing
system/user emulation with cross execution using the glibc makefile.

> > >
> > > Such approach has several advantages:
> > >
> > > - No need for SSH (which can hang, as you need sshfs in user space for
> > >   the setup)

I had not tested with sshfs that likely has quite low performance, the
OEQA tests rely on NFS for QEMU system emulation and that is what I
had been using for hardware testing. However it would be interesting
to see if virtiofs could improve the performance for testing with QEMU
system emulation.

> > >
> > > - Run on the real HW (and environment) with the same version of glibc
> > >   on the real target.
> > >
> > > - Sources and debugging symbols available on the target, so easier GDB
> > >   debugging.
> > >
> > > - Using ptest framework to execute tests
> > >
> > > - Time related tests are "self contained", so could be executed
> > > without built data.
> > >
> > > At least for Y2038 validation such approach seems more appealing.
> > >
> >
> > Do you have any more feedback regarding this patch?
>
> We're at feature freeze and this wasn't a planned change so it is too late for
> this release cycle, I'm struggling with the things that were planned.
>
> I need to spend some time looking at what the other code is doing and how we're
> using it on the autobuilder and whether we want to change to use the ptest
> approach or not.
>
> I added Nathan Rossi to cc since he worked on the other code for this and may
> have more insight.

Thanks for cc'ing, I somehow managed to miss the thread.

Regards,
Nathan

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[?ukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 12044 bytes --]

Hi Nathan,

> On Thu, 26 Aug 2021 at 23:38, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > On Thu, 2021-08-26 at 14:27 +0200, Lukasz Majewski wrote:  
> > > Hi Richard, Khem,
> > >  
> > > > Hi Richard,
> > > >  
> > > > > On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote:  
> > > > > > Hi Khem,
> > > > > >  
> > > > > > > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski
> > > > > > > <lukma@denx.de> wrote:  
> > > > > > > >
> > > > > > > > On Mon, 23 Aug 2021 12:52:44 -0700
> > > > > > > > Khem Raj <raj.khem@gmail.com> wrote:
> > > > > > > >  
> > > > > > > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski
> > > > > > > > > <lukma@denx.de> wrote:
> > > > > > > > >  
> > > > > > > > > > Hi Khem,
> > > > > > > > > >  
> > > > > > > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:  
> > > > > > > > > > > > This patch introduces new recipe - namely
> > > > > > > > > > > > 'glibc-tests', which builds and installs glibc
> > > > > > > > > > > > test suite to OE/Yocto built image.
> > > > > > > > > > > >
> > > > > > > > > > > > It reuses code from already available
> > > > > > > > > > > > 'glibc-testsuite' recipe, which is run with
> > > > > > > > > > > > 'bitbake glibc-testsuite -c check' and uses
> > > > > > > > > > > > qemu to execute remotely (via SSH) tests on
> > > > > > > > > > > > some emulated machine.
> > > > > > > > > > > >
> > > > > > > > > > > > This recipe installs eligible tests on some
> > > > > > > > > > > > rootfs image. Afterwards, either all of them or
> > > > > > > > > > > > only time related subset, those tests can be
> > > > > > > > > > > > executed on the real hardware, to facilitate
> > > > > > > > > > > > validation of this platform with for example
> > > > > > > > > > > > Y2038 problem compliance.
> > > > > > > > > > > >
> > > > > > > > > > > > By default all tests are executed, with
> > > > > > > > > > > > 'ptest-runner glibc-tests'. To test only time
> > > > > > > > > > > > related subset - one needs to call: cd
> > > > > > > > > > > > /usr/lib/glibc-tests/ptest/; rm run-ptest; \ ln
> > > > > > > > > > > > -s run-ptest-time run-ptest; cd -; ptest-runner
> > > > > > > > > > > > glibc-tests
> > > > > > > > > > > >
> > > > > > > > > > > > To facilitate debugging, source files are
> > > > > > > > > > > > provided by default with the unstripped
> > > > > > > > > > > > debugging symbols. Such approach would reduce
> > > > > > > > > > > > the already complex recipe (as it inherits base
> > > > > > > > > > > > glibc one), so there is no need to also install
> > > > > > > > > > > > *-dbg and *-src packages.  
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > does it have to be a separate recipe I wonder if
> > > > > > > > > > > we can have it built as part of glibc itself
> > > > > > > > > > > controlled via ptest knob  
> > > > > > > > > >
> > > > > > > > > > I've followed the glibc-testsuite recipe to provide
> > > > > > > > > > tests for ptest. Test creation is similar to it,
> > > > > > > > > > but doesn't require QEMU run (tests are executed on
> > > > > > > > > > the HW).
> > > > > > > > > >
> > > > > > > > > > Another rationale was to have some kind of
> > > > > > > > > > "features" separation in different file (liked
> > > > > > > > > > glibc-mtrace), which is easier to maintain and
> > > > > > > > > > review.
> > > > > > > > > >
> > > > > > > > > > Last but not least - separate recipe (and built
> > > > > > > > > > binaries) allow some kind of magic with selection
> > > > > > > > > > of used test programs (this may be useful if one
> > > > > > > > > > would like to have different sets of tests in
> > > > > > > > > > different packages)  
> > > > > > > > >
> > > > > > > > > That’s seems ok I think the names are quite confusing
> > > > > > > > > now not that it was not so much better before but now
> > > > > > > > > we have glibc-tests and glibc-testsuites which do
> > > > > > > > > same things but in very different way maybe
> > > > > > > > > glibc-testsuite should be renamed to something like
> > > > > > > > > glibc-tests-crosd or some such  
> > > > > > > >
> > > > > > > > I think that glibc-testsuite_2.34.bb shall be renamed to
> > > > > > > > glibc-tests-qemu_2.34.bb as it is more descriptive.  
> > > > > > >
> > > > > > > is it only runnable in qemu ? I thought we could use a
> > > > > > > read board as well with IP address  
> > > > > >
> > > > > > It looks like by default the QEMU is used in this recipe.
> > > > > >
> > > > > > This recipe provides custom check-test-wrapper, which can
> > > > > > log into remote board via ssh (when TOOLCHAIN_TEST_TARGET =
> > > > > > "ssh").
> > > > > >
> > > > > > This looks a bit awkward for me, as the goal would be to
> > > > > > run built tests on target (exact) HW without the need of
> > > > > > SSH.
> > > > > >
> > > > > > It is also easier to debug the code with the real HW.  
> > > > >
> > > > > I'm a bit worried we're going around in circles with this. We
> > > > > did have a different form of tests a while back but I was
> > > > > told those didn't work well and they were replaced with the
> > > > > current setup which worked with the driver in the glibc
> > > > > source code.  
> > > >
> > > > I've looked into the glibc-testing.inc from SHA1:
> > > > 0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b
> > > >
> > > > There glibc tests were built and copied on the target via NFS.
> > > >  
> > > > > This was done around
> > > > > the time we added support for the other toolchain test suites
> > > > > (binutils+gcc).  
> > > >
> > > > I'm not familiar with the "binutils+gcc" issue, which was
> > > > solved in the past. Could you share any reference?
> > > >  
> > > > > We ended up with options to use full system qemu or
> > > > > user mode qemu emulation depending on the speed of the target
> > > > > (e.g. with kvm or not).  
> > > >
> > > > From what I see in the recipe - you either run user mode QEMU by
> > > > default (at least when I do run it):
> > > >
> > > > NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i
> > > > QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot
> > > > QEMU_OPTIONS=qemu-arm -r 5.1.0  SSH_HOST=localhost SSH_HOS
> > > > T_USER=root SSH_HOST_PORT=2222
> > > > test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper
> > > > user check
> > > >
> > > > Or you log to the board via SSH (the 'ssh' mode).
> > > >  
> > > > >
> > > > > Has this patchset been run with all the tests in glibc or
> > > > > just a subset?  
> > > >
> > > > This patch set allows running all tests [*], or (preferably in
> > > > my use case) only a time related ones. The latter is to
> > > > validate Y2038 code, which testing has two issues:
> > > >
> > > > 1. It cannot be tested with QEMU user mode [1]. Instead "linux
> > > > timespaces" were suggested, but it lacks support for
> > > > CLOCK_REALTIME manipulation for those tests [2].  
> 
> So just a note, the qemu user mode testing is not a complete solution.
> There are a number of problems with it, but in general it is still
> quite useful for validating general functionally, especially when
> implementing a new platform. 

+1

> A big one is due to the ability to run
> tests in parallel with user mode, it easily reduces test run times by
> an order of magnitude (compared to qemu system, which for some
> platforms is faster than real hardware).

+1

> 
> The reason why qemu user mode is the default behaviour is because
> running "bitbake glibc-testsuite -c check" will work without any
> external dependencies (the remote machine, NFS/sshfs, etc.).

+1

> 
> > > >
> > > > 2. Testing with SSH is possible, but is not as robust as
> > > > expected (i.e. imposes SSHD on the target board, and ETH
> > > > connection). 
> > > > >
> > > > > I'm always very nervous about having two ways to do similar
> > > > > things. 
> > > >
> > > > Technically, I just reuse the glibc-testsuite_2.34.bb to the
> > > > point where tests are _only_ built. Then I pack those binaries
> > > > and install on the target's rootfs to be run with ptest
> > > > framework.  
> 
> So it appears the make variable "run-built-tests" excludes the
> building of some tests.

As fair as I remember, the 'run-built-tests=no' switch tells the glibc
build system to not execute built tests.

> So I am not sure if the complete test suite
> can be built in this way.

I'm pretty sure that all tests required for 'check' are build.

> Though I can see some value in having the
> glibc-testsuite recipe being able to only build the tests for
> development purposes (e.g. TOOLCHAIN_TEST_TARGET = "build"?).
> 
> When I had implemented the testsuite recipe the glibc testsuite relied
> quite heavily on the environment it was built and run in, in
> particular around configuration. That is probably still an issue.

Indeed, some (a few?) tests rely on the build environment in which they
were build. I do personally, regard this as a some kind of a bug. I do
believe, that when you build test for a particular version of glibc
(2.34 in this case), one shall expect that it can be run only by
linking the proper library.

This is the case for "time" related tests. On my test system with this
patch applied, all of them are executed with no errors. Those are self
contained.

> 
> It would be interesting to know how much of the test suite can be
> run/passes and how that compares to the results of existing
> system/user emulation with cross execution using the glibc makefile.

I can provide exact numbers when I finish porting more Y2038 related
patches to newest glibc -master.

> 
> > > >
> > > > Such approach has several advantages:
> > > >
> > > > - No need for SSH (which can hang, as you need sshfs in user
> > > > space for the setup)  
> 
> I had not tested with sshfs that likely has quite low performance, the
> OEQA tests rely on NFS for QEMU system emulation and that is what I
> had been using for hardware testing.

I had some issues with NFS - either I couldn't place some OE/Yocto
build directory in it or glibc's build directory was not working with
it correctly.

That was why I had to use the sshfs instead, which is _really_ slow and
often breaks.

> However it would be interesting
> to see if virtiofs could improve the performance for testing with QEMU
> system emulation.
> 
> > > >
> > > > - Run on the real HW (and environment) with the same version of
> > > > glibc on the real target.
> > > >
> > > > - Sources and debugging symbols available on the target, so
> > > > easier GDB debugging.
> > > >
> > > > - Using ptest framework to execute tests
> > > >
> > > > - Time related tests are "self contained", so could be executed
> > > > without built data.
> > > >
> > > > At least for Y2038 validation such approach seems more
> > > > appealing. 
> > >
> > > Do you have any more feedback regarding this patch?  
> >
> > We're at feature freeze and this wasn't a planned change so it is
> > too late for this release cycle, I'm struggling with the things
> > that were planned.
> >
> > I need to spend some time looking at what the other code is doing
> > and how we're using it on the autobuilder and whether we want to
> > change to use the ptest approach or not.
> >
> > I added Nathan Rossi to cc since he worked on the other code for
> > this and may have more insight.  
> 
> Thanks for cc'ing, I somehow managed to miss the thread.
> 
> Regards,
> Nathan


Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[Nathan Rossi]

On Fri, 27 Aug 2021 at 19:20, Lukasz Majewski <lukma@denx.de> wrote:
>
> Hi Nathan,
>
> > On Thu, 26 Aug 2021 at 23:38, Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > >
> > > On Thu, 2021-08-26 at 14:27 +0200, Lukasz Majewski wrote:
> > > > Hi Richard, Khem,
> > > >
> > > > > Hi Richard,
> > > > >
> > > > > > On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote:
> > > > > > > Hi Khem,
> > > > > > >
> > > > > > > > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski
> > > > > > > > <lukma@denx.de> wrote:
> > > > > > > > >
> > > > > > > > > On Mon, 23 Aug 2021 12:52:44 -0700
> > > > > > > > > Khem Raj <raj.khem@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski
> > > > > > > > > > <lukma@denx.de> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Khem,
> > > > > > > > > > >
> > > > > > > > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:
> > > > > > > > > > > > > This patch introduces new recipe - namely
> > > > > > > > > > > > > 'glibc-tests', which builds and installs glibc
> > > > > > > > > > > > > test suite to OE/Yocto built image.
> > > > > > > > > > > > >
> > > > > > > > > > > > > It reuses code from already available
> > > > > > > > > > > > > 'glibc-testsuite' recipe, which is run with
> > > > > > > > > > > > > 'bitbake glibc-testsuite -c check' and uses
> > > > > > > > > > > > > qemu to execute remotely (via SSH) tests on
> > > > > > > > > > > > > some emulated machine.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This recipe installs eligible tests on some
> > > > > > > > > > > > > rootfs image. Afterwards, either all of them or
> > > > > > > > > > > > > only time related subset, those tests can be
> > > > > > > > > > > > > executed on the real hardware, to facilitate
> > > > > > > > > > > > > validation of this platform with for example
> > > > > > > > > > > > > Y2038 problem compliance.
> > > > > > > > > > > > >
> > > > > > > > > > > > > By default all tests are executed, with
> > > > > > > > > > > > > 'ptest-runner glibc-tests'. To test only time
> > > > > > > > > > > > > related subset - one needs to call: cd
> > > > > > > > > > > > > /usr/lib/glibc-tests/ptest/; rm run-ptest; \ ln
> > > > > > > > > > > > > -s run-ptest-time run-ptest; cd -; ptest-runner
> > > > > > > > > > > > > glibc-tests
> > > > > > > > > > > > >
> > > > > > > > > > > > > To facilitate debugging, source files are
> > > > > > > > > > > > > provided by default with the unstripped
> > > > > > > > > > > > > debugging symbols. Such approach would reduce
> > > > > > > > > > > > > the already complex recipe (as it inherits base
> > > > > > > > > > > > > glibc one), so there is no need to also install
> > > > > > > > > > > > > *-dbg and *-src packages.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > does it have to be a separate recipe I wonder if
> > > > > > > > > > > > we can have it built as part of glibc itself
> > > > > > > > > > > > controlled via ptest knob
> > > > > > > > > > >
> > > > > > > > > > > I've followed the glibc-testsuite recipe to provide
> > > > > > > > > > > tests for ptest. Test creation is similar to it,
> > > > > > > > > > > but doesn't require QEMU run (tests are executed on
> > > > > > > > > > > the HW).
> > > > > > > > > > >
> > > > > > > > > > > Another rationale was to have some kind of
> > > > > > > > > > > "features" separation in different file (liked
> > > > > > > > > > > glibc-mtrace), which is easier to maintain and
> > > > > > > > > > > review.
> > > > > > > > > > >
> > > > > > > > > > > Last but not least - separate recipe (and built
> > > > > > > > > > > binaries) allow some kind of magic with selection
> > > > > > > > > > > of used test programs (this may be useful if one
> > > > > > > > > > > would like to have different sets of tests in
> > > > > > > > > > > different packages)
> > > > > > > > > >
> > > > > > > > > > That’s seems ok I think the names are quite confusing
> > > > > > > > > > now not that it was not so much better before but now
> > > > > > > > > > we have glibc-tests and glibc-testsuites which do
> > > > > > > > > > same things but in very different way maybe
> > > > > > > > > > glibc-testsuite should be renamed to something like
> > > > > > > > > > glibc-tests-crosd or some such
> > > > > > > > >
> > > > > > > > > I think that glibc-testsuite_2.34.bb shall be renamed to
> > > > > > > > > glibc-tests-qemu_2.34.bb as it is more descriptive.
> > > > > > > >
> > > > > > > > is it only runnable in qemu ? I thought we could use a
> > > > > > > > read board as well with IP address
> > > > > > >
> > > > > > > It looks like by default the QEMU is used in this recipe.
> > > > > > >
> > > > > > > This recipe provides custom check-test-wrapper, which can
> > > > > > > log into remote board via ssh (when TOOLCHAIN_TEST_TARGET =
> > > > > > > "ssh").
> > > > > > >
> > > > > > > This looks a bit awkward for me, as the goal would be to
> > > > > > > run built tests on target (exact) HW without the need of
> > > > > > > SSH.
> > > > > > >
> > > > > > > It is also easier to debug the code with the real HW.
> > > > > >
> > > > > > I'm a bit worried we're going around in circles with this. We
> > > > > > did have a different form of tests a while back but I was
> > > > > > told those didn't work well and they were replaced with the
> > > > > > current setup which worked with the driver in the glibc
> > > > > > source code.
> > > > >
> > > > > I've looked into the glibc-testing.inc from SHA1:
> > > > > 0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b
> > > > >
> > > > > There glibc tests were built and copied on the target via NFS.
> > > > >
> > > > > > This was done around
> > > > > > the time we added support for the other toolchain test suites
> > > > > > (binutils+gcc).
> > > > >
> > > > > I'm not familiar with the "binutils+gcc" issue, which was
> > > > > solved in the past. Could you share any reference?
> > > > >
> > > > > > We ended up with options to use full system qemu or
> > > > > > user mode qemu emulation depending on the speed of the target
> > > > > > (e.g. with kvm or not).
> > > > >
> > > > > From what I see in the recipe - you either run user mode QEMU by
> > > > > default (at least when I do run it):
> > > > >
> > > > > NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i
> > > > > QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot
> > > > > QEMU_OPTIONS=qemu-arm -r 5.1.0  SSH_HOST=localhost SSH_HOS
> > > > > T_USER=root SSH_HOST_PORT=2222
> > > > > test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper
> > > > > user check
> > > > >
> > > > > Or you log to the board via SSH (the 'ssh' mode).
> > > > >
> > > > > >
> > > > > > Has this patchset been run with all the tests in glibc or
> > > > > > just a subset?
> > > > >
> > > > > This patch set allows running all tests [*], or (preferably in
> > > > > my use case) only a time related ones. The latter is to
> > > > > validate Y2038 code, which testing has two issues:
> > > > >
> > > > > 1. It cannot be tested with QEMU user mode [1]. Instead "linux
> > > > > timespaces" were suggested, but it lacks support for
> > > > > CLOCK_REALTIME manipulation for those tests [2].
> >
> > So just a note, the qemu user mode testing is not a complete solution.
> > There are a number of problems with it, but in general it is still
> > quite useful for validating general functionally, especially when
> > implementing a new platform.
>
> +1
>
> > A big one is due to the ability to run
> > tests in parallel with user mode, it easily reduces test run times by
> > an order of magnitude (compared to qemu system, which for some
> > platforms is faster than real hardware).
>
> +1
>
> >
> > The reason why qemu user mode is the default behaviour is because
> > running "bitbake glibc-testsuite -c check" will work without any
> > external dependencies (the remote machine, NFS/sshfs, etc.).
>
> +1
>
> >
> > > > >
> > > > > 2. Testing with SSH is possible, but is not as robust as
> > > > > expected (i.e. imposes SSHD on the target board, and ETH
> > > > > connection).
> > > > > >
> > > > > > I'm always very nervous about having two ways to do similar
> > > > > > things.
> > > > >
> > > > > Technically, I just reuse the glibc-testsuite_2.34.bb to the
> > > > > point where tests are _only_ built. Then I pack those binaries
> > > > > and install on the target's rootfs to be run with ptest
> > > > > framework.
> >
> > So it appears the make variable "run-built-tests" excludes the
> > building of some tests.
>
> As fair as I remember, the 'run-built-tests=no' switch tells the glibc
> build system to not execute built tests.
>
> > So I am not sure if the complete test suite
> > can be built in this way.
>
> I'm pretty sure that all tests required for 'check' are build.

If you look at some of the Makefiles in glibc, it conditionally
includes/excludes some tests from being built based on the value of
run-built-tests. For example:

https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/Makefile;h=f585e0dd4115a99493e843aeed464090da19665f;hb=HEAD#l141

>
> > Though I can see some value in having the
> > glibc-testsuite recipe being able to only build the tests for
> > development purposes (e.g. TOOLCHAIN_TEST_TARGET = "build"?).
> >
> > When I had implemented the testsuite recipe the glibc testsuite relied
> > quite heavily on the environment it was built and run in, in
> > particular around configuration. That is probably still an issue.
>
> Indeed, some (a few?) tests rely on the build environment in which they
> were build. I do personally, regard this as a some kind of a bug. I do
> believe, that when you build test for a particular version of glibc
> (2.34 in this case), one shall expect that it can be run only by
> linking the proper library.
>
> This is the case for "time" related tests. On my test system with this
> patch applied, all of them are executed with no errors. Those are self
> contained.
>
> >
> > It would be interesting to know how much of the test suite can be
> > run/passes and how that compares to the results of existing
> > system/user emulation with cross execution using the glibc makefile.
>
> I can provide exact numbers when I finish porting more Y2038 related
> patches to newest glibc -master.

Also something to keep in mind is that sometimes the pass/fail of the
test binary is not what is being validated by the test. For example
this test uses mtrace, and compares the results of the output mtrace
file, not the test itself.

https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/Makefile;h=f585e0dd4115a99493e843aeed464090da19665f;hb=HEAD#l462

Regards,
Nathan

>
> >
> > > > >
> > > > > Such approach has several advantages:
> > > > >
> > > > > - No need for SSH (which can hang, as you need sshfs in user
> > > > > space for the setup)
> >
> > I had not tested with sshfs that likely has quite low performance, the
> > OEQA tests rely on NFS for QEMU system emulation and that is what I
> > had been using for hardware testing.
>
> I had some issues with NFS - either I couldn't place some OE/Yocto
> build directory in it or glibc's build directory was not working with
> it correctly.
>
> That was why I had to use the sshfs instead, which is _really_ slow and
> often breaks.
>
> > However it would be interesting
> > to see if virtiofs could improve the performance for testing with QEMU
> > system emulation.
> >
> > > > >
> > > > > - Run on the real HW (and environment) with the same version of
> > > > > glibc on the real target.
> > > > >
> > > > > - Sources and debugging symbols available on the target, so
> > > > > easier GDB debugging.
> > > > >
> > > > > - Using ptest framework to execute tests
> > > > >
> > > > > - Time related tests are "self contained", so could be executed
> > > > > without built data.
> > > > >
> > > > > At least for Y2038 validation such approach seems more
> > > > > appealing.
> > > >
> > > > Do you have any more feedback regarding this patch?
> > >
> > > We're at feature freeze and this wasn't a planned change so it is
> > > too late for this release cycle, I'm struggling with the things
> > > that were planned.
> > >
> > > I need to spend some time looking at what the other code is doing
> > > and how we're using it on the autobuilder and whether we want to
> > > change to use the ptest approach or not.
> > >
> > > I added Nathan Rossi to cc since he worked on the other code for
> > > this and may have more insight.
> >
> > Thanks for cc'ing, I somehow managed to miss the thread.
> >
> > Regards,
> > Nathan
>
>
> Best regards,
>
> Lukasz Majewski
>
> --
>
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

[?ukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 14631 bytes --]

Hi Nathan,

> On Fri, 27 Aug 2021 at 19:20, Lukasz Majewski <lukma@denx.de> wrote:
> >
> > Hi Nathan,
> >  
> > > On Thu, 26 Aug 2021 at 23:38, Richard Purdie
> > > <richard.purdie@linuxfoundation.org> wrote:  
> > > >
> > > > On Thu, 2021-08-26 at 14:27 +0200, Lukasz Majewski wrote:  
> > > > > Hi Richard, Khem,
> > > > >  
> > > > > > Hi Richard,
> > > > > >  
> > > > > > > On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote:
> > > > > > >  
> > > > > > > > Hi Khem,
> > > > > > > >  
> > > > > > > > > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski
> > > > > > > > > <lukma@denx.de> wrote:  
> > > > > > > > > >
> > > > > > > > > > On Mon, 23 Aug 2021 12:52:44 -0700
> > > > > > > > > > Khem Raj <raj.khem@gmail.com> wrote:
> > > > > > > > > >  
> > > > > > > > > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski
> > > > > > > > > > > <lukma@denx.de> wrote:
> > > > > > > > > > >  
> > > > > > > > > > > > Hi Khem,
> > > > > > > > > > > >  
> > > > > > > > > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:  
> > > > > > > > > > > > > > This patch introduces new recipe - namely
> > > > > > > > > > > > > > 'glibc-tests', which builds and installs
> > > > > > > > > > > > > > glibc test suite to OE/Yocto built image.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It reuses code from already available
> > > > > > > > > > > > > > 'glibc-testsuite' recipe, which is run with
> > > > > > > > > > > > > > 'bitbake glibc-testsuite -c check' and uses
> > > > > > > > > > > > > > qemu to execute remotely (via SSH) tests on
> > > > > > > > > > > > > > some emulated machine.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This recipe installs eligible tests on some
> > > > > > > > > > > > > > rootfs image. Afterwards, either all of
> > > > > > > > > > > > > > them or only time related subset, those
> > > > > > > > > > > > > > tests can be executed on the real hardware,
> > > > > > > > > > > > > > to facilitate validation of this platform
> > > > > > > > > > > > > > with for example Y2038 problem compliance.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > By default all tests are executed, with
> > > > > > > > > > > > > > 'ptest-runner glibc-tests'. To test only
> > > > > > > > > > > > > > time related subset - one needs to call: cd
> > > > > > > > > > > > > > /usr/lib/glibc-tests/ptest/; rm run-ptest;
> > > > > > > > > > > > > > \ ln -s run-ptest-time run-ptest; cd -;
> > > > > > > > > > > > > > ptest-runner glibc-tests
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > To facilitate debugging, source files are
> > > > > > > > > > > > > > provided by default with the unstripped
> > > > > > > > > > > > > > debugging symbols. Such approach would
> > > > > > > > > > > > > > reduce the already complex recipe (as it
> > > > > > > > > > > > > > inherits base glibc one), so there is no
> > > > > > > > > > > > > > need to also install *-dbg and *-src
> > > > > > > > > > > > > > packages.  
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > does it have to be a separate recipe I wonder
> > > > > > > > > > > > > if we can have it built as part of glibc
> > > > > > > > > > > > > itself controlled via ptest knob  
> > > > > > > > > > > >
> > > > > > > > > > > > I've followed the glibc-testsuite recipe to
> > > > > > > > > > > > provide tests for ptest. Test creation is
> > > > > > > > > > > > similar to it, but doesn't require QEMU run
> > > > > > > > > > > > (tests are executed on the HW).
> > > > > > > > > > > >
> > > > > > > > > > > > Another rationale was to have some kind of
> > > > > > > > > > > > "features" separation in different file (liked
> > > > > > > > > > > > glibc-mtrace), which is easier to maintain and
> > > > > > > > > > > > review.
> > > > > > > > > > > >
> > > > > > > > > > > > Last but not least - separate recipe (and built
> > > > > > > > > > > > binaries) allow some kind of magic with
> > > > > > > > > > > > selection of used test programs (this may be
> > > > > > > > > > > > useful if one would like to have different sets
> > > > > > > > > > > > of tests in different packages)  
> > > > > > > > > > >
> > > > > > > > > > > That’s seems ok I think the names are quite
> > > > > > > > > > > confusing now not that it was not so much better
> > > > > > > > > > > before but now we have glibc-tests and
> > > > > > > > > > > glibc-testsuites which do same things but in very
> > > > > > > > > > > different way maybe glibc-testsuite should be
> > > > > > > > > > > renamed to something like glibc-tests-crosd or
> > > > > > > > > > > some such  
> > > > > > > > > >
> > > > > > > > > > I think that glibc-testsuite_2.34.bb shall be
> > > > > > > > > > renamed to glibc-tests-qemu_2.34.bb as it is more
> > > > > > > > > > descriptive.  
> > > > > > > > >
> > > > > > > > > is it only runnable in qemu ? I thought we could use a
> > > > > > > > > read board as well with IP address  
> > > > > > > >
> > > > > > > > It looks like by default the QEMU is used in this
> > > > > > > > recipe.
> > > > > > > >
> > > > > > > > This recipe provides custom check-test-wrapper, which
> > > > > > > > can log into remote board via ssh (when
> > > > > > > > TOOLCHAIN_TEST_TARGET = "ssh").
> > > > > > > >
> > > > > > > > This looks a bit awkward for me, as the goal would be to
> > > > > > > > run built tests on target (exact) HW without the need of
> > > > > > > > SSH.
> > > > > > > >
> > > > > > > > It is also easier to debug the code with the real HW.  
> > > > > > >
> > > > > > > I'm a bit worried we're going around in circles with
> > > > > > > this. We did have a different form of tests a while back
> > > > > > > but I was told those didn't work well and they were
> > > > > > > replaced with the current setup which worked with the
> > > > > > > driver in the glibc source code.  
> > > > > >
> > > > > > I've looked into the glibc-testing.inc from SHA1:
> > > > > > 0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b
> > > > > >
> > > > > > There glibc tests were built and copied on the target via
> > > > > > NFS. 
> > > > > > > This was done around
> > > > > > > the time we added support for the other toolchain test
> > > > > > > suites (binutils+gcc).  
> > > > > >
> > > > > > I'm not familiar with the "binutils+gcc" issue, which was
> > > > > > solved in the past. Could you share any reference?
> > > > > >  
> > > > > > > We ended up with options to use full system qemu or
> > > > > > > user mode qemu emulation depending on the speed of the
> > > > > > > target (e.g. with kvm or not).  
> > > > > >
> > > > > > From what I see in the recipe - you either run user mode
> > > > > > QEMU by default (at least when I do run it):
> > > > > >
> > > > > > NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i
> > > > > > QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot
> > > > > > QEMU_OPTIONS=qemu-arm -r 5.1.0  SSH_HOST=localhost SSH_HOS
> > > > > > T_USER=root SSH_HOST_PORT=2222
> > > > > > test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper
> > > > > > user check
> > > > > >
> > > > > > Or you log to the board via SSH (the 'ssh' mode).
> > > > > >  
> > > > > > >
> > > > > > > Has this patchset been run with all the tests in glibc or
> > > > > > > just a subset?  
> > > > > >
> > > > > > This patch set allows running all tests [*], or (preferably
> > > > > > in my use case) only a time related ones. The latter is to
> > > > > > validate Y2038 code, which testing has two issues:
> > > > > >
> > > > > > 1. It cannot be tested with QEMU user mode [1]. Instead
> > > > > > "linux timespaces" were suggested, but it lacks support for
> > > > > > CLOCK_REALTIME manipulation for those tests [2].  
> > >
> > > So just a note, the qemu user mode testing is not a complete
> > > solution. There are a number of problems with it, but in general
> > > it is still quite useful for validating general functionally,
> > > especially when implementing a new platform.  
> >
> > +1
> >  
> > > A big one is due to the ability to run
> > > tests in parallel with user mode, it easily reduces test run
> > > times by an order of magnitude (compared to qemu system, which
> > > for some platforms is faster than real hardware).  
> >
> > +1
> >  
> > >
> > > The reason why qemu user mode is the default behaviour is because
> > > running "bitbake glibc-testsuite -c check" will work without any
> > > external dependencies (the remote machine, NFS/sshfs, etc.).  
> >
> > +1
> >  
> > >  
> > > > > >
> > > > > > 2. Testing with SSH is possible, but is not as robust as
> > > > > > expected (i.e. imposes SSHD on the target board, and ETH
> > > > > > connection).  
> > > > > > >
> > > > > > > I'm always very nervous about having two ways to do
> > > > > > > similar things.  
> > > > > >
> > > > > > Technically, I just reuse the glibc-testsuite_2.34.bb to the
> > > > > > point where tests are _only_ built. Then I pack those
> > > > > > binaries and install on the target's rootfs to be run with
> > > > > > ptest framework.  
> > >
> > > So it appears the make variable "run-built-tests" excludes the
> > > building of some tests.  
> >
> > As fair as I remember, the 'run-built-tests=no' switch tells the
> > glibc build system to not execute built tests.
> >  
> > > So I am not sure if the complete test suite
> > > can be built in this way.  
> >
> > I'm pretty sure that all tests required for 'check' are build.  
> 
> If you look at some of the Makefiles in glibc, it conditionally
> includes/excludes some tests from being built based on the value of
> run-built-tests. For example:
> 
> https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/Makefile;h=f585e0dd4115a99493e843aeed464090da19665f;hb=HEAD#l141
> 

I see. Thanks for pointing this out.

> >  
> > > Though I can see some value in having the
> > > glibc-testsuite recipe being able to only build the tests for
> > > development purposes (e.g. TOOLCHAIN_TEST_TARGET = "build"?).
> > >
> > > When I had implemented the testsuite recipe the glibc testsuite
> > > relied quite heavily on the environment it was built and run in,
> > > in particular around configuration. That is probably still an
> > > issue.  
> >
> > Indeed, some (a few?) tests rely on the build environment in which
> > they were build. I do personally, regard this as a some kind of a
> > bug. I do believe, that when you build test for a particular
> > version of glibc (2.34 in this case), one shall expect that it can
> > be run only by linking the proper library.
> >
> > This is the case for "time" related tests. On my test system with
> > this patch applied, all of them are executed with no errors. Those
> > are self contained.
> >  
> > >
> > > It would be interesting to know how much of the test suite can be
> > > run/passes and how that compares to the results of existing
> > > system/user emulation with cross execution using the glibc
> > > makefile.  
> >
> > I can provide exact numbers when I finish porting more Y2038 related
> > patches to newest glibc -master.  
> 
> Also something to keep in mind is that sometimes the pass/fail of the
> test binary is not what is being validated by the test. For example
> this test uses mtrace, and compares the results of the output mtrace
> file, not the test itself.
> 
> https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/Makefile;h=f585e0dd4115a99493e843aeed464090da19665f;hb=HEAD#l462
> 

Ok. I see.

My main objective is to use time related programs, which rely on the
interfaces provided by the kernel. 

From the above POV, those tests are a "standalone" ones.

> Regards,
> Nathan
> 
> >  
> > >  
> > > > > >
> > > > > > Such approach has several advantages:
> > > > > >
> > > > > > - No need for SSH (which can hang, as you need sshfs in user
> > > > > > space for the setup)  
> > >
> > > I had not tested with sshfs that likely has quite low
> > > performance, the OEQA tests rely on NFS for QEMU system emulation
> > > and that is what I had been using for hardware testing.  
> >
> > I had some issues with NFS - either I couldn't place some OE/Yocto
> > build directory in it or glibc's build directory was not working
> > with it correctly.
> >
> > That was why I had to use the sshfs instead, which is _really_ slow
> > and often breaks.
> >  
> > > However it would be interesting
> > > to see if virtiofs could improve the performance for testing with
> > > QEMU system emulation.
> > >  
> > > > > >
> > > > > > - Run on the real HW (and environment) with the same
> > > > > > version of glibc on the real target.
> > > > > >
> > > > > > - Sources and debugging symbols available on the target, so
> > > > > > easier GDB debugging.
> > > > > >
> > > > > > - Using ptest framework to execute tests
> > > > > >
> > > > > > - Time related tests are "self contained", so could be
> > > > > > executed without built data.
> > > > > >
> > > > > > At least for Y2038 validation such approach seems more
> > > > > > appealing.  
> > > > >
> > > > > Do you have any more feedback regarding this patch?  
> > > >
> > > > We're at feature freeze and this wasn't a planned change so it
> > > > is too late for this release cycle, I'm struggling with the
> > > > things that were planned.
> > > >
> > > > I need to spend some time looking at what the other code is
> > > > doing and how we're using it on the autobuilder and whether we
> > > > want to change to use the ptest approach or not.
> > > >
> > > > I added Nathan Rossi to cc since he worked on the other code for
> > > > this and may have more insight.  
> > >
> > > Thanks for cc'ing, I somehow managed to miss the thread.
> > >
> > > Regards,
> > > Nathan  
> >
> >
> > Best regards,
> >
> > Lukasz Majewski
> >
> > --
> >
> > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > Germany Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
> > lukma@denx.de  




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]