* [PATCHSET v3 00/15] fstests: improve junit xml reporting
@ 2023-03-15 0:52 Darrick J. Wong
2023-03-15 0:52 ` [PATCH 01/15] check: generate section reports between tests Darrick J. Wong
` (14 more replies)
0 siblings, 15 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:52 UTC (permalink / raw)
To: zlang, djwong
Cc: Qu Wenruo, Leah Rumancik, linux-xfs, fstests, guan,
leah.rumancik, quwenruo.btrfs, tytso
Hi all,
This series improves the fstests reporting code in several ways. First,
change the ./check code to generate the report after every test, so that
a cluster-based fstest scheduler can reschedule tests after a VM crash.
Personally, I was using it to get live status on my tests dashboard.
The bulk of the patches in here improve the junit xml reporting so that
we (a) actually declare which xml schema we're trying to emit and (b)
capture a lot more information about what was being tested.
v2: shorten indenting in the schema file, record .dmesg files as a
separate kernel-log tag, clarify what the timestamp attribute means,
record the test suite start time and report generation time as separate
attributes, make it possible to pass in a list of report variables,
encode cdata correctly
v3: Reviewed-and-tested-by: Leah Rumancik <leah.rumancik@gmail.com>
If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy!
Comments and questions are, as always, welcome.
--D
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=xunit-reporting-improvements
---
README | 3 +
check | 11 +++
common/ext4 | 5 +
common/report | 130 +++++++++++++++++++++++++-----
common/xfs | 11 +++
doc/xunit.xsd | 246 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6 files changed, 386 insertions(+), 20 deletions(-)
create mode 100644 doc/xunit.xsd
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH 01/15] check: generate section reports between tests
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
@ 2023-03-15 0:52 ` Darrick J. Wong
2023-03-15 0:52 ` [PATCH 02/15] report: derive an xml schema for the xunit report Darrick J. Wong
` (13 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:52 UTC (permalink / raw)
To: zlang, djwong
Cc: Leah Rumancik, Leah Rumancik, linux-xfs, fstests, guan,
leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
Generate the section report between tests so that the summary report
always reflects the outcome of the most recent test. Two usecases are
envisioned here -- if a cluster-based test runner anticipates that the
testrun could crash the VM, they can set REPORT_DIR to (say) an NFS
mount to preserve the intermediate results. If the VM does indeed
crash, the scheduler can examine the state of the crashed VM and move
the tests to another VM. The second usecase is a reporting agent that
runs in the VM to upload live results to a test dashboard.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Leah Rumancik <leah.rumancik@gmail.com>
Tested-by: Leah Rumancik <leah.rumancik@gmail.com>
---
check | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/check b/check
index 0bf5b22e06..14b398fd73 100755
--- a/check
+++ b/check
@@ -844,6 +844,15 @@ function run_section()
fi
seqres="$REPORT_DIR/$seqnum"
+ # Generate the entire section report with whatever test results
+ # we have so far. Leave the $sect_time parameter empty so that
+ # it's a little more obvious that this test run is incomplete.
+ if $do_report; then
+ _make_section_report "$section" "${#try[*]}" \
+ "${#bad[*]}" "${#notrun[*]}" \
+ "" &> /dev/null
+ fi
+
mkdir -p $RESULT_DIR
rm -f ${RESULT_DIR}/require_scratch*
rm -f ${RESULT_DIR}/require_test*
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 02/15] report: derive an xml schema for the xunit report
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
2023-03-15 0:52 ` [PATCH 01/15] check: generate section reports between tests Darrick J. Wong
@ 2023-03-15 0:52 ` Darrick J. Wong
2023-03-15 0:52 ` [PATCH 03/15] report: capture the time zone in the test report timestamp Darrick J. Wong
` (12 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:52 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
The "xunit" report format emits an XML document that more or less
follows the junit xml schema. However, there are two major exceptions:
1. fstests does not emit an @errors attribute on the testsuite element
because we don't have the concept of unanticipated errors such as
"unchecked throwables".
2. The system-out/system-err elements sound like they belong under the
testcase element, though the schema itself imprecisely says "while the
test was executed". The schema puts them under the top-level testsuite
element, but we put them under the testcase element.
Define an xml schema for the xunit report format, and update the xml
headers to link to the schema file. This enables consumers of the
reports to check mechanically that the incoming document follows the
format.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 15 +++-
doc/xunit.xsd | 226 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 238 insertions(+), 3 deletions(-)
create mode 100644 doc/xunit.xsd
diff --git a/common/report b/common/report
index 64f9c86648..1d84650270 100644
--- a/common/report
+++ b/common/report
@@ -48,9 +48,18 @@ _xunit_make_section_report()
if [ -z "$date_time" ]; then
date_time=$(date +"%F %T")
fi
- local stats="failures=\"$bad_count\" skipped=\"$notrun_count\" tests=\"$tests_count\" time=\"$sect_time\""
- local hw_info="hostname=\"$HOST\" timestamp=\"${date_time/ /T}\" "
- echo "<testsuite name=\"xfstests\" $stats $hw_info >" >> $REPORT_DIR/result.xml
+
+ local fstests_ns="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"
+ cat >> $REPORT_DIR/result.xml << ENDL
+<testsuite
+ xmlns="$fstests_ns"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="$fstests_ns $fstests_ns/tree/doc/xunit.xsd"
+
+ name="xfstests"
+ failures="$bad_count" skipped="$notrun_count" tests="$tests_count" time="$sect_time"
+ hostname="$HOST" timestamp="${date_time/ /T}">
+ENDL
# Properties
echo -e "\t<properties>" >> $REPORT_DIR/result.xml
diff --git a/doc/xunit.xsd b/doc/xunit.xsd
new file mode 100644
index 0000000000..ba97ccd67d
--- /dev/null
+++ b/doc/xunit.xsd
@@ -0,0 +1,226 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<xs:schema
+ xmlns="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+
+ targetNamespace="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"
+ elementFormDefault="qualified"
+ attributeFormDefault="unqualified">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">fstests xunit test result schema, derived from https://github.com/windyroad/JUnit-Schema</xs:documentation>
+ </xs:annotation>
+ <xs:element name="testsuite" type="testsuite"/>
+ <xs:simpleType name="ISO8601_DATETIME_PATTERN">
+ <xs:restriction base="xs:dateTime">
+ <xs:pattern value="[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}"/>
+ </xs:restriction>
+ </xs:simpleType>
+ <xs:element name="testsuites">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Contains an aggregation of testsuite results</xs:documentation>
+ </xs:annotation>
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="testsuite" minOccurs="0" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:complexContent>
+ <xs:extension base="testsuite">
+ <xs:attribute name="package" type="xs:token" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Derived from testsuite/@name in the non-aggregated documents</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="id" type="xs:int" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Starts at '0' for the first testsuite and is incremented by 1 for each following testsuite</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ </xs:extension>
+ </xs:complexContent>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ <xs:complexType name="testsuite">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Contains the results of executing a testsuite</xs:documentation>
+ </xs:annotation>
+ <xs:sequence>
+ <xs:element name="properties">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Properties (e.g., environment settings) set during test execution</xs:documentation>
+ </xs:annotation>
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="property" minOccurs="0" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:attribute name="name" use="required">
+ <xs:simpleType>
+ <xs:restriction base="xs:token">
+ <xs:minLength value="1"/>
+ </xs:restriction>
+ </xs:simpleType>
+ </xs:attribute>
+ <xs:attribute name="value" type="xs:string" use="required"/>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ <xs:element name="testcase" minOccurs="0" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:choice minOccurs="0" maxOccurs="1">
+ <xs:element name="skipped" minOccurs="0" maxOccurs="1">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Indicates that the test was skipped.</xs:documentation>
+ </xs:annotation>
+ <xs:complexType>
+ <xs:simpleContent>
+ <xs:extension base="pre-string">
+ <xs:attribute name="message" type="xs:string">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The message specifying why the test case was skipped</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ </xs:extension>
+ </xs:simpleContent>
+ </xs:complexType>
+ </xs:element>
+ <xs:element name="error" minOccurs="0" maxOccurs="1">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Indicates that the test errored. An errored test is one that had an unanticipated problem. e.g., an unchecked throwable; or a problem with the implementation of the test. Contains as a text node relevant data for the error, e.g., a stack trace</xs:documentation>
+ </xs:annotation>
+ <xs:complexType>
+ <xs:simpleContent>
+ <xs:extension base="pre-string">
+ <xs:attribute name="message" type="xs:string">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The error message. e.g., if a java exception is thrown, the return value of getMessage()</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="type" type="xs:string" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The type of error that occured. e.g., if a java execption is thrown the full class name of the exception.</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ </xs:extension>
+ </xs:simpleContent>
+ </xs:complexType>
+ </xs:element>
+ <xs:element name="failure">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Indicates that the test failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals. Contains as a text node relevant data for the failure, e.g., a stack trace</xs:documentation>
+ </xs:annotation>
+ <xs:complexType>
+ <xs:simpleContent>
+ <xs:extension base="pre-string">
+ <xs:attribute name="message" type="xs:string">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The message specified in the assert</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="type" type="xs:string" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The type of the assert.</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ </xs:extension>
+ </xs:simpleContent>
+ </xs:complexType>
+ </xs:element>
+ </xs:choice>
+ <xs:choice minOccurs="0" maxOccurs="2">
+ <xs:element name="system-out" minOccurs="0" maxOccurs="1">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Data that was written to the .full log file while the test was executed.</xs:documentation>
+ </xs:annotation>
+ <xs:simpleType>
+ <xs:restriction base="pre-string">
+ <xs:whiteSpace value="preserve"/>
+ </xs:restriction>
+ </xs:simpleType>
+ </xs:element>
+ <xs:element name="system-err" minOccurs="0" maxOccurs="1">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Kernel log or data that was compared to the golden output file after the test was executed.</xs:documentation>
+ </xs:annotation>
+ <xs:simpleType>
+ <xs:restriction base="pre-string">
+ <xs:whiteSpace value="preserve"/>
+ </xs:restriction>
+ </xs:simpleType>
+ </xs:element>
+ </xs:choice>
+ </xs:sequence>
+ <xs:attribute name="name" type="xs:token" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Name of the test method</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="classname" type="xs:token" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Full class name for the class the test method is in.</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="time" type="xs:decimal" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Time taken (in seconds) to execute the test</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ <xs:attribute name="name" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Full class name of the test for non-aggregated testsuite documents. Class name without the package for aggregated testsuites documents</xs:documentation>
+ </xs:annotation>
+ <xs:simpleType>
+ <xs:restriction base="xs:token">
+ <xs:minLength value="1"/>
+ </xs:restriction>
+ </xs:simpleType>
+ </xs:attribute>
+ <xs:attribute name="timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">when the test was executed. Timezone may not be specified.</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="hostname" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Host on which the tests were executed. 'localhost' should be used if the hostname cannot be determined.</xs:documentation>
+ </xs:annotation>
+ <xs:simpleType>
+ <xs:restriction base="xs:token">
+ <xs:minLength value="1"/>
+ </xs:restriction>
+ </xs:simpleType>
+ </xs:attribute>
+ <xs:attribute name="tests" type="xs:int" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The total number of tests in the suite</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="failures" type="xs:int" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The total number of tests in the suite that failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="skipped" type="xs:int" use="optional">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The total number of ignored or skipped tests in the suite.</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="time" type="xs:decimal" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Time taken (in seconds) to execute the tests in the suite</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ </xs:complexType>
+ <xs:simpleType name="pre-string">
+ <xs:restriction base="xs:string">
+ <xs:whiteSpace value="preserve"/>
+ </xs:restriction>
+ </xs:simpleType>
+</xs:schema>
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 03/15] report: capture the time zone in the test report timestamp
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
2023-03-15 0:52 ` [PATCH 01/15] check: generate section reports between tests Darrick J. Wong
2023-03-15 0:52 ` [PATCH 02/15] report: derive an xml schema for the xunit report Darrick J. Wong
@ 2023-03-15 0:52 ` Darrick J. Wong
2023-03-15 0:52 ` [PATCH 04/15] report: clarify the meaning of the timestamp attribute Darrick J. Wong
` (11 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:52 UTC (permalink / raw)
To: zlang, djwong
Cc: Qu Wenruo, linux-xfs, fstests, guan, leah.rumancik,
quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
Make sure we put the time zone of the system running the test in the
timestamp that is recorded in the xunit report. `date "+%F %T"' reports
the local time zone, not UTC.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Qu Wenruo <wqu@suse.com>
---
common/report | 9 ++++++---
doc/xunit.xsd | 4 ++--
2 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/common/report b/common/report
index 1d84650270..1817132d51 100644
--- a/common/report
+++ b/common/report
@@ -38,6 +38,7 @@ _xunit_make_section_report()
local bad_count="$3"
local notrun_count="$4"
local sect_time="$5"
+ local timestamp
if [ $sect_name == '-no-sections-' ]; then
sect_name='global'
@@ -45,8 +46,10 @@ _xunit_make_section_report()
local report=$tmp.report.xunit.$sect_name.xml
# Header
echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" > $REPORT_DIR/result.xml
- if [ -z "$date_time" ]; then
- date_time=$(date +"%F %T")
+ if [ -n "$date_time" ]; then
+ timestamp="$(date -Iseconds --date="$date_time")"
+ else
+ timestamp="$(date -Iseconds)"
fi
local fstests_ns="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"
@@ -58,7 +61,7 @@ _xunit_make_section_report()
name="xfstests"
failures="$bad_count" skipped="$notrun_count" tests="$tests_count" time="$sect_time"
- hostname="$HOST" timestamp="${date_time/ /T}">
+ hostname="$HOST" timestamp="$timestamp">
ENDL
# Properties
diff --git a/doc/xunit.xsd b/doc/xunit.xsd
index ba97ccd67d..9295c5dc82 100644
--- a/doc/xunit.xsd
+++ b/doc/xunit.xsd
@@ -12,7 +12,7 @@
<xs:element name="testsuite" type="testsuite"/>
<xs:simpleType name="ISO8601_DATETIME_PATTERN">
<xs:restriction base="xs:dateTime">
- <xs:pattern value="[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}"/>
+ <xs:pattern value="[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}[+-][0-9]{2}:[0-9]{2}"/>
</xs:restriction>
</xs:simpleType>
<xs:element name="testsuites">
@@ -184,7 +184,7 @@
</xs:attribute>
<xs:attribute name="timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
<xs:annotation>
- <xs:documentation xml:lang="en">when the test was executed. Timezone may not be specified.</xs:documentation>
+ <xs:documentation xml:lang="en">when the test was executed. Timezone must be specified as an offset from UTC.</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="hostname" use="required">
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 04/15] report: clarify the meaning of the timestamp attribute
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
` (2 preceding siblings ...)
2023-03-15 0:52 ` [PATCH 03/15] report: capture the time zone in the test report timestamp Darrick J. Wong
@ 2023-03-15 0:52 ` Darrick J. Wong
2023-03-15 0:52 ` [PATCH 05/15] report: record fstests start and report generation timestamps Darrick J. Wong
` (10 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:52 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
We've never specified what the timestamp attribute of the testsuite
element actually means, and it history is rather murky.
Prior to the introduction of the xml report format in commit f9fde7db2f,
the "date_time" variable was used only to scrape dmesg via the /dev/kmsg
device after each test. If /dev/kmsg was not a writable path, the
variable was not set at all. In this case, the report timestamp would
be blank.
In commit ffdecf7498a1, Ted changed the xunit report code to handle
empty date_time values by setting date_time to the time of report
generation. This change was done to handle the case where no tests are
run at all. However, it did not change the behavior that date_time is
not set if /dev/kmsg is not writable.
Clear up all this confusion by defining the timestamp attribute to
reflect the start time of the most recent test, regardless of the state
of /dev/kmsg. If no tests are run, then define the attribute to be the
time of report generation.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
check | 1 +
common/report | 8 +++++---
doc/xunit.xsd | 2 +-
3 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/check b/check
index 14b398fd73..f2be3d7d7d 100755
--- a/check
+++ b/check
@@ -917,6 +917,7 @@ function run_section()
# to be reported for each test
(echo 1 > $DEBUGFS_MNT/clear_warn_once) > /dev/null 2>&1
+ test_start_time="$(date +"%F %T")"
if [ "$DUMP_OUTPUT" = true ]; then
_run_seq 2>&1 | tee $tmp.out
# Because $? would get tee's return code
diff --git a/common/report b/common/report
index 1817132d51..8e19e9f557 100644
--- a/common/report
+++ b/common/report
@@ -46,8 +46,8 @@ _xunit_make_section_report()
local report=$tmp.report.xunit.$sect_name.xml
# Header
echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" > $REPORT_DIR/result.xml
- if [ -n "$date_time" ]; then
- timestamp="$(date -Iseconds --date="$date_time")"
+ if [ -n "$test_start_time" ]; then
+ timestamp="$(date -Iseconds --date="$test_start_time")"
else
timestamp="$(date -Iseconds)"
fi
@@ -61,7 +61,9 @@ _xunit_make_section_report()
name="xfstests"
failures="$bad_count" skipped="$notrun_count" tests="$tests_count" time="$sect_time"
- hostname="$HOST" timestamp="$timestamp">
+ hostname="$HOST"
+ timestamp="$timestamp"
+>
ENDL
# Properties
diff --git a/doc/xunit.xsd b/doc/xunit.xsd
index 9295c5dc82..653f486871 100644
--- a/doc/xunit.xsd
+++ b/doc/xunit.xsd
@@ -184,7 +184,7 @@
</xs:attribute>
<xs:attribute name="timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
<xs:annotation>
- <xs:documentation xml:lang="en">when the test was executed. Timezone must be specified as an offset from UTC.</xs:documentation>
+ <xs:documentation xml:lang="en">Time that the last testcase was started. If no tests are started, this is the time the report was generated. Timezone must be specified as an offset from UTC.</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="hostname" use="required">
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 05/15] report: record fstests start and report generation timestamps
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
` (3 preceding siblings ...)
2023-03-15 0:52 ` [PATCH 04/15] report: clarify the meaning of the timestamp attribute Darrick J. Wong
@ 2023-03-15 0:52 ` Darrick J. Wong
2023-03-15 0:53 ` [PATCH 06/15] report: encode cdata sections correctly Darrick J. Wong
` (9 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:52 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
Report two new timestamps in the xml report: the time that ./check was
started, and the time that the report was generated. We introduce new
timestamps to minimize breakage with parsing scripts.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
check | 1 +
common/report | 4 +++-
doc/xunit.xsd | 10 ++++++++++
3 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/check b/check
index f2be3d7d7d..1a58a2b269 100755
--- a/check
+++ b/check
@@ -668,6 +668,7 @@ _run_seq() {
_detect_kmemleak
_prepare_test_list
+fstests_start_time="$(date +"%F %T")"
if $OPTIONS_HAVE_SECTIONS; then
trap "_summary; exit \$status" 0 1 2 3 15
diff --git a/common/report b/common/report
index 8e19e9f557..be991b55f5 100644
--- a/common/report
+++ b/common/report
@@ -62,7 +62,9 @@ _xunit_make_section_report()
name="xfstests"
failures="$bad_count" skipped="$notrun_count" tests="$tests_count" time="$sect_time"
hostname="$HOST"
- timestamp="$timestamp"
+ start_timestamp="$(date -Iseconds --date="$fstests_start_time")"
+ timestamp="$timestamp"
+ report_timestamp="$(date -Iseconds)"
>
ENDL
diff --git a/doc/xunit.xsd b/doc/xunit.xsd
index 653f486871..3ed72f2f86 100644
--- a/doc/xunit.xsd
+++ b/doc/xunit.xsd
@@ -187,6 +187,16 @@
<xs:documentation xml:lang="en">Time that the last testcase was started. If no tests are started, this is the time the report was generated. Timezone must be specified as an offset from UTC.</xs:documentation>
</xs:annotation>
</xs:attribute>
+ <xs:attribute name="report_timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Time that the report was generated.</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="start_timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Time that fstests was started.</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
<xs:attribute name="hostname" use="required">
<xs:annotation>
<xs:documentation xml:lang="en">Host on which the tests were executed. 'localhost' should be used if the hostname cannot be determined.</xs:documentation>
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 06/15] report: encode cdata sections correctly
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
` (4 preceding siblings ...)
2023-03-15 0:52 ` [PATCH 05/15] report: record fstests start and report generation timestamps Darrick J. Wong
@ 2023-03-15 0:53 ` Darrick J. Wong
2023-03-15 0:53 ` [PATCH 07/15] report: encode the kernel log as a separate xml element Darrick J. Wong
` (8 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:53 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
The XML report format captures the contents of .full and .out.bad files
in CDATA sections. CDATA sections are supposed to be a stream of
verbatim data, terminated with a "]]>". Hence XML entities such as
quotation marks and angle brackes should not be escaped, and an actual
bracket-bracket-gt sequence in those files /does/ need escaping.
Create a separate filtering function so that these files are encoded
properly.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
diff --git a/common/report b/common/report
index be991b55f5..eb169175bc 100644
--- a/common/report
+++ b/common/report
@@ -19,6 +19,11 @@ encode_xml()
-e 's/"/\"/g'
}
+encode_cdata()
+{
+ cat -v | sed -e 's/]]>/]]]]><![CDATA[>/g'
+}
+
#
# Xunit format report functions
_xunit_add_property()
@@ -128,7 +133,7 @@ _xunit_make_testcase_report()
if [ -z "$quiet" -a -s "$full_file" ]; then
echo -e "\t\t<system-out>" >> $report
printf '<![CDATA[\n' >>$report
- cat "$full_file" | tr -dc '[:print:][:space:]' | encode_xml >>$report
+ cat "$full_file" | tr -dc '[:print:][:space:]' | encode_cdata >>$report
printf ']]>\n' >>$report
echo -e "\t\t</system-out>" >> $report
fi
@@ -137,13 +142,13 @@ _xunit_make_testcase_report()
elif [ -f "$dmesg_file" ]; then
echo -e "\t\t<system-err>" >> $report
printf '<![CDATA[\n' >>$report
- cat "$dmesg_file" | tr -dc '[:print:][:space:]' | encode_xml >>$report
+ cat "$dmesg_file" | tr -dc '[:print:][:space:]' | encode_cdata >>$report
printf ']]>\n' >>$report
echo -e "\t\t</system-err>" >> $report
elif [ -s "$outbad_file" ]; then
echo -e "\t\t<system-err>" >> $report
printf '<![CDATA[\n' >>$report
- $diff "$out_src" "$outbad_file" | encode_xml >>$report
+ $diff "$out_src" "$outbad_file" | encode_cdata >>$report
printf ']]>\n' >>$report
echo -e "\t\t</system-err>" >> $report
fi
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 07/15] report: encode the kernel log as a separate xml element
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
` (5 preceding siblings ...)
2023-03-15 0:53 ` [PATCH 06/15] report: encode cdata sections correctly Darrick J. Wong
@ 2023-03-15 0:53 ` Darrick J. Wong
2023-03-15 0:53 ` [PATCH 08/15] report: sort properties by name Darrick J. Wong
` (7 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:53 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
Record the .dmesg file in a new <kernel-log> element instead of
multiplexing it with <system-err>. This means that the xml report can
now capture kernel log and bad golden output.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 11 +++++------
doc/xunit.xsd | 14 ++++++++++++--
2 files changed, 17 insertions(+), 8 deletions(-)
diff --git a/common/report b/common/report
index eb169175bc..2e5959312a 100644
--- a/common/report
+++ b/common/report
@@ -137,15 +137,14 @@ _xunit_make_testcase_report()
printf ']]>\n' >>$report
echo -e "\t\t</system-out>" >> $report
fi
- if [ -n "$quiet" ]; then
- :
- elif [ -f "$dmesg_file" ]; then
- echo -e "\t\t<system-err>" >> $report
+ if [ -z "$quiet" -a -f "$dmesg_file" ]; then
+ echo -e "\t\t<kernel-log>" >> $report
printf '<![CDATA[\n' >>$report
cat "$dmesg_file" | tr -dc '[:print:][:space:]' | encode_cdata >>$report
printf ']]>\n' >>$report
- echo -e "\t\t</system-err>" >> $report
- elif [ -s "$outbad_file" ]; then
+ echo -e "\t\t</kernel-log>" >> $report
+ fi
+ if [ -z "$quiet" -a -s "$outbad_file" ]; then
echo -e "\t\t<system-err>" >> $report
printf '<![CDATA[\n' >>$report
$diff "$out_src" "$outbad_file" | encode_cdata >>$report
diff --git a/doc/xunit.xsd b/doc/xunit.xsd
index 3ed72f2f86..d287eaf5a2 100644
--- a/doc/xunit.xsd
+++ b/doc/xunit.xsd
@@ -131,7 +131,7 @@
</xs:complexType>
</xs:element>
</xs:choice>
- <xs:choice minOccurs="0" maxOccurs="2">
+ <xs:choice minOccurs="0" maxOccurs="3">
<xs:element name="system-out" minOccurs="0" maxOccurs="1">
<xs:annotation>
<xs:documentation xml:lang="en">Data that was written to the .full log file while the test was executed.</xs:documentation>
@@ -144,7 +144,17 @@
</xs:element>
<xs:element name="system-err" minOccurs="0" maxOccurs="1">
<xs:annotation>
- <xs:documentation xml:lang="en">Kernel log or data that was compared to the golden output file after the test was executed.</xs:documentation>
+ <xs:documentation xml:lang="en">Data that was compared to the golden output file after the test was executed.</xs:documentation>
+ </xs:annotation>
+ <xs:simpleType>
+ <xs:restriction base="pre-string">
+ <xs:whiteSpace value="preserve"/>
+ </xs:restriction>
+ </xs:simpleType>
+ </xs:element>
+ <xs:element name="kernel-log" minOccurs="0" maxOccurs="1">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Kernel log recorded while the test was executed.</xs:documentation>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="pre-string">
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 08/15] report: sort properties by name
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
` (6 preceding siblings ...)
2023-03-15 0:53 ` [PATCH 07/15] report: encode the kernel log as a separate xml element Darrick J. Wong
@ 2023-03-15 0:53 ` Darrick J. Wong
2023-03-15 0:53 ` [PATCH 09/15] report: pass property value to _xunit_add_property Darrick J. Wong
` (6 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:53 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
When we're generating a junit xml report, always sort the properties by
name. This makes it easier for humans to find a particular property.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/common/report b/common/report
index 2e5959312a..3ec2d88178 100644
--- a/common/report
+++ b/common/report
@@ -32,7 +32,7 @@ _xunit_add_property()
local value="${!name}"
if [ ! -z "$value" ]; then
- echo -e "\t\t<property name=\"$name\" value=\"$value\"/>" >> $REPORT_DIR/result.xml
+ echo -e "\t\t<property name=\"$name\" value=\"$value\"/>"
fi
}
_xunit_make_section_report()
@@ -77,7 +77,7 @@ ENDL
echo -e "\t<properties>" >> $REPORT_DIR/result.xml
for p in "${REPORT_ENV_LIST[@]}"; do
_xunit_add_property "$p"
- done
+ done | sort >> $REPORT_DIR/result.xml
echo -e "\t</properties>" >> $REPORT_DIR/result.xml
if [ -f $report ]; then
cat $report >> $REPORT_DIR/result.xml
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 09/15] report: pass property value to _xunit_add_property
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
` (7 preceding siblings ...)
2023-03-15 0:53 ` [PATCH 08/15] report: sort properties by name Darrick J. Wong
@ 2023-03-15 0:53 ` Darrick J. Wong
2023-03-15 0:53 ` [PATCH 10/15] report: encode xml entities in property values Darrick J. Wong
` (5 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:53 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
Change this helper to require the caller to pass the value as the second
parameter. This prepares us to start reporting a lot more information
about a test run, not all of which are encoded as bash variables.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/common/report b/common/report
index 3ec2d88178..2ab83928db 100644
--- a/common/report
+++ b/common/report
@@ -29,12 +29,13 @@ encode_cdata()
_xunit_add_property()
{
local name="$1"
- local value="${!name}"
+ local value="$2"
- if [ ! -z "$value" ]; then
- echo -e "\t\t<property name=\"$name\" value=\"$value\"/>"
- fi
+ test -z "$value" && return
+
+ echo -e "\t\t<property name=\"$name\" value=\"$value\"/>"
}
+
_xunit_make_section_report()
{
# xfstest:section ==> xunit:testsuite
@@ -76,7 +77,7 @@ ENDL
# Properties
echo -e "\t<properties>" >> $REPORT_DIR/result.xml
for p in "${REPORT_ENV_LIST[@]}"; do
- _xunit_add_property "$p"
+ _xunit_add_property "$p" "${!p}"
done | sort >> $REPORT_DIR/result.xml
echo -e "\t</properties>" >> $REPORT_DIR/result.xml
if [ -f $report ]; then
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 10/15] report: encode xml entities in property values
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
` (8 preceding siblings ...)
2023-03-15 0:53 ` [PATCH 09/15] report: pass property value to _xunit_add_property Darrick J. Wong
@ 2023-03-15 0:53 ` Darrick J. Wong
2023-03-15 0:53 ` [PATCH 11/15] report: collect basic information about a test run Darrick J. Wong
` (4 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:53 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
Avoid trouble with the properties reported in the xml reports by
translating xml-tricky characters in the property values into their xml
entity equivalents.
IOWs, if someone sets a property "NAME" to the value 'BOBBY"; DROP TABLES;',
the xml will be formatted:
<property name="NAME" value="BOBBY"; DROP TABLES;"/>
Thus avoiding XML problems.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/common/report b/common/report
index 2ab83928db..946ee4887c 100644
--- a/common/report
+++ b/common/report
@@ -33,7 +33,10 @@ _xunit_add_property()
test -z "$value" && return
- echo -e "\t\t<property name=\"$name\" value=\"$value\"/>"
+ local xname="$(echo "$name" | encode_xml)"
+ local xvalue="$(echo "$value" | encode_xml)"
+
+ echo -e "\t\t<property name=\"$xname\" value=\"$xvalue\"/>"
}
_xunit_make_section_report()
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 11/15] report: collect basic information about a test run
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
` (9 preceding siblings ...)
2023-03-15 0:53 ` [PATCH 10/15] report: encode xml entities in property values Darrick J. Wong
@ 2023-03-15 0:53 ` Darrick J. Wong
2023-03-15 0:53 ` [PATCH 12/15] report: record optional environment variables Darrick J. Wong
` (3 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:53 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
Record various generic information about an fstests run when generating
a junit xml report. This includes the cpu architecture, the kernel
revision, the CPU, memory, and numa node counts, and some information
about the block devices passed in.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 44 +++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 43 insertions(+), 1 deletion(-)
diff --git a/common/report b/common/report
index 946ee4887c..90d4f980d1 100644
--- a/common/report
+++ b/common/report
@@ -24,6 +24,42 @@ encode_cdata()
cat -v | sed -e 's/]]>/]]]]><![CDATA[>/g'
}
+# Fill out REPORT_VARS with information about the block device referred to by
+# the passed-in bash variable.
+__generate_blockdev_report_vars() {
+ local bdev_var="$1"
+ local bdev="${!bdev_var}"
+
+ test -z "$bdev" && return
+ test -b "$bdev" || return
+ local sysfs_bdev="$(_sysfs_dev "$bdev")"
+
+ REPORT_VARS["${bdev_var}_SIZE_KB"]="$(( "$(blockdev --getsz "$bdev")" / 2 ))"
+ REPORT_VARS["${bdev_var}_ROTATIONAL"]="$(cat "$sysfs_bdev/queue/rotational" 2>/dev/null)"
+ REPORT_VARS["${bdev_var}_DAX"]="$(cat "$sysfs_bdev/queue/dax" 2>/dev/null)"
+ REPORT_VARS["${bdev_var}_DISCARD"]="$(sed -e 's/[1-9][0-9]*/1/g' "$sysfs_bdev/queue/discard_max_bytes" 2>/dev/null)"
+ REPORT_VARS["${bdev_var}_WRITE_ZEROES"]="$(sed -e 's/[1-9][0-9]*/1/g' "$sysfs_bdev/queue/write_zeroes_max_bytes" 2>/dev/null)"
+ REPORT_VARS["${bdev_var}_PHYS_BLOCK_BYTES"]="$(cat "$sysfs_bdev/queue/physical_block_size" 2>/dev/null)"
+ REPORT_VARS["${bdev_var}_LBA_BYTES"]="$(cat "$sysfs_bdev/queue/logical_block_size" 2>/dev/null)"
+ REPORT_VARS["${bdev_var}_ZONES"]="$(cat "$sysfs_bdev/queue/nr_zones" 2>/dev/null)"
+}
+
+# Fill out REPORT_VARS with tidbits about our test runner configuration.
+# Caller is required to declare REPORT_VARS to be an associative array.
+__generate_report_vars() {
+ REPORT_VARS["ARCH"]="$(uname -m)"
+ REPORT_VARS["KERNEL"]="$(uname -r)"
+ REPORT_VARS["CPUS"]="$(getconf _NPROCESSORS_ONLN 2>/dev/null)"
+ REPORT_VARS["MEM_KB"]="$(grep MemTotal: /proc/meminfo | awk '{print $2}')"
+ REPORT_VARS["SWAP_KB"]="$(grep SwapTotal: /proc/meminfo | awk '{print $2}')"
+
+ test -e /sys/devices/system/node/possible && \
+ REPORT_VARS["NUMA_NODES"]="$(cat /sys/devices/system/node/possible 2>/dev/null)"
+
+ __generate_blockdev_report_vars "TEST_DEV"
+ __generate_blockdev_report_vars "SCRATCH_DEV"
+}
+
#
# Xunit format report functions
_xunit_add_property()
@@ -77,11 +113,17 @@ _xunit_make_section_report()
>
ENDL
+ declare -A REPORT_VARS
+ __generate_report_vars
+
# Properties
echo -e "\t<properties>" >> $REPORT_DIR/result.xml
+ (for key in "${!REPORT_VARS[@]}"; do
+ _xunit_add_property "$key" "${REPORT_VARS["$key"]}"
+ done;
for p in "${REPORT_ENV_LIST[@]}"; do
_xunit_add_property "$p" "${!p}"
- done | sort >> $REPORT_DIR/result.xml
+ done) | sort >> $REPORT_DIR/result.xml
echo -e "\t</properties>" >> $REPORT_DIR/result.xml
if [ -f $report ]; then
cat $report >> $REPORT_DIR/result.xml
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 12/15] report: record optional environment variables
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
` (10 preceding siblings ...)
2023-03-15 0:53 ` [PATCH 11/15] report: collect basic information about a test run Darrick J. Wong
@ 2023-03-15 0:53 ` Darrick J. Wong
2023-03-15 0:53 ` [PATCH 13/15] report: record xfs-specific information about a test run Darrick J. Wong
` (2 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:53 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
These environment variables are documented as being significant, but
optional. If they're set to a non-empty string, record them in the
reports.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/common/report b/common/report
index 90d4f980d1..af3c04db56 100644
--- a/common/report
+++ b/common/report
@@ -9,6 +9,11 @@ REPORT_ENV_LIST=("SECTION" "FSTYP" "PLATFORM" "MKFS_OPTIONS" "MOUNT_OPTIONS" \
"TIME_FACTOR" "LOAD_FACTOR" "TEST_DIR" "TEST_DEV" \
"SCRATCH_DEV" "SCRATCH_MNT" "OVL_UPPER" "OVL_LOWER" "OVL_WORK")
+# Variables that are captured in the report /if/ they are set.
+REPORT_ENV_LIST_OPT=("TAPE_DEV" "RMT_TAPE_DEV" "FSSTRES_AVOID" "FSX_AVOID"
+ "KCONFIG_PATH" "PERF_CONFIGNAME" "MIN_FSSIZE"
+ "IDMAPPED_MOUNTS")
+
encode_xml()
{
cat -v | \
@@ -58,6 +63,11 @@ __generate_report_vars() {
__generate_blockdev_report_vars "TEST_DEV"
__generate_blockdev_report_vars "SCRATCH_DEV"
+
+ # Optional environmental variables
+ for varname in "${REPORT_ENV_LIST_OPT[@]}"; do
+ test -n "${!varname}" && REPORT_VARS["${varname}"]="${!varname}"
+ done
}
#
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 13/15] report: record xfs-specific information about a test run
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
` (11 preceding siblings ...)
2023-03-15 0:53 ` [PATCH 12/15] report: record optional environment variables Darrick J. Wong
@ 2023-03-15 0:53 ` Darrick J. Wong
2023-03-15 0:53 ` [PATCH 14/15] report: record ext*-specific " Darrick J. Wong
2023-03-15 0:53 ` [PATCH 15/15] report: allow test runners to inject arbitrary values Darrick J. Wong
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:53 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
Report various XFS-specific information about a test run.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 3 +++
common/xfs | 11 +++++++++++
2 files changed, 14 insertions(+)
diff --git a/common/report b/common/report
index af3c04db56..86274af887 100644
--- a/common/report
+++ b/common/report
@@ -64,6 +64,9 @@ __generate_report_vars() {
__generate_blockdev_report_vars "TEST_DEV"
__generate_blockdev_report_vars "SCRATCH_DEV"
+ # Add per-filesystem variables to the report variable list
+ test "$FSTYP" = "xfs" && __generate_xfs_report_vars
+
# Optional environmental variables
for varname in "${REPORT_ENV_LIST_OPT[@]}"; do
test -n "${!varname}" && REPORT_VARS["${varname}"]="${!varname}"
diff --git a/common/xfs b/common/xfs
index e679af824f..e8e4832cea 100644
--- a/common/xfs
+++ b/common/xfs
@@ -2,6 +2,17 @@
# XFS specific common functions.
#
+__generate_xfs_report_vars() {
+ __generate_blockdev_report_vars TEST_RTDEV
+ __generate_blockdev_report_vars TEST_LOGDEV
+ __generate_blockdev_report_vars SCRATCH_RTDEV
+ __generate_blockdev_report_vars SCRATCH_LOGDEV
+
+ REPORT_VARS["XFS_ALWAYS_COW"]="$(cat /sys/fs/xfs/debug/always_cow 2>/dev/null)"
+ REPORT_VARS["XFS_LARP"]="$(cat /sys/fs/xfs/debug/larp 2>/dev/null)"
+ REPORT_ENV_LIST_OPT+=("TEST_XFS_REPAIR_REBUILD" "TEST_XFS_SCRUB_REBUILD")
+}
+
_setup_large_xfs_fs()
{
fs_size=$1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 14/15] report: record ext*-specific information about a test run
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
` (12 preceding siblings ...)
2023-03-15 0:53 ` [PATCH 13/15] report: record xfs-specific information about a test run Darrick J. Wong
@ 2023-03-15 0:53 ` Darrick J. Wong
2023-03-15 0:53 ` [PATCH 15/15] report: allow test runners to inject arbitrary values Darrick J. Wong
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:53 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
Report various ext* specific information about a test run.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/ext4 | 5 +++++
common/report | 1 +
2 files changed, 6 insertions(+)
diff --git a/common/ext4 b/common/ext4
index 8fd6dbc682..3dcbfe17c9 100644
--- a/common/ext4
+++ b/common/ext4
@@ -2,6 +2,11 @@
# ext4 specific common functions
#
+__generate_ext4_report_vars() {
+ __generate_blockdev_report_vars TEST_LOGDEV
+ __generate_blockdev_report_vars SCRATCH_LOGDEV
+}
+
_setup_large_ext4_fs()
{
local fs_size=$1
diff --git a/common/report b/common/report
index 86274af887..db15aec54f 100644
--- a/common/report
+++ b/common/report
@@ -66,6 +66,7 @@ __generate_report_vars() {
# Add per-filesystem variables to the report variable list
test "$FSTYP" = "xfs" && __generate_xfs_report_vars
+ [[ "$FSTYP" == ext[0-9]* ]] && __generate_ext4_report_vars
# Optional environmental variables
for varname in "${REPORT_ENV_LIST_OPT[@]}"; do
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 15/15] report: allow test runners to inject arbitrary values
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
` (13 preceding siblings ...)
2023-03-15 0:53 ` [PATCH 14/15] report: record ext*-specific " Darrick J. Wong
@ 2023-03-15 0:53 ` Darrick J. Wong
14 siblings, 0 replies; 16+ messages in thread
From: Darrick J. Wong @ 2023-03-15 0:53 UTC (permalink / raw)
To: zlang, djwong
Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs, tytso
From: Darrick J. Wong <djwong@kernel.org>
Per Ted's request, add to the test section reporting code the ability
for test runners to point to a file containing colon-separated key value
pairs. These key value pairs will be recorded in the report file as
extra properties.
Requested-by: Theodore Ts'o <tytso@mit.edu>
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
README | 3 +++
common/report | 10 ++++++++++
2 files changed, 13 insertions(+)
diff --git a/README b/README
index 1ca506492b..4ee877a962 100644
--- a/README
+++ b/README
@@ -268,6 +268,9 @@ Misc:
this option is supported for all filesystems currently only -overlay is
expected to run without issues. For other filesystems additional patches
and fixes to the test suite might be needed.
+ - Set REPORT_VARS_FILE to a file containing colon-separated name-value pairs
+ that will be recorded in the test section report. Names must be unique.
+ Whitespace surrounding the colon will be removed.
______________________
USING THE FSQA SUITE
diff --git a/common/report b/common/report
index db15aec54f..23ddbb096d 100644
--- a/common/report
+++ b/common/report
@@ -49,9 +49,19 @@ __generate_blockdev_report_vars() {
REPORT_VARS["${bdev_var}_ZONES"]="$(cat "$sysfs_bdev/queue/nr_zones" 2>/dev/null)"
}
+__import_report_vars() {
+ local fname="$1"
+
+ while IFS=':' read key value; do
+ REPORT_VARS["${key%% }"]="${value## }"
+ done < "$1"
+}
+
# Fill out REPORT_VARS with tidbits about our test runner configuration.
# Caller is required to declare REPORT_VARS to be an associative array.
__generate_report_vars() {
+ test "$REPORT_VARS_FILE" && __import_report_vars "$REPORT_VARS_FILE"
+
REPORT_VARS["ARCH"]="$(uname -m)"
REPORT_VARS["KERNEL"]="$(uname -r)"
REPORT_VARS["CPUS"]="$(getconf _NPROCESSORS_ONLN 2>/dev/null)"
^ permalink raw reply related [flat|nested] 16+ messages in thread
end of thread, other threads:[~2023-03-15 0:53 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-15 0:52 [PATCHSET v3 00/15] fstests: improve junit xml reporting Darrick J. Wong
2023-03-15 0:52 ` [PATCH 01/15] check: generate section reports between tests Darrick J. Wong
2023-03-15 0:52 ` [PATCH 02/15] report: derive an xml schema for the xunit report Darrick J. Wong
2023-03-15 0:52 ` [PATCH 03/15] report: capture the time zone in the test report timestamp Darrick J. Wong
2023-03-15 0:52 ` [PATCH 04/15] report: clarify the meaning of the timestamp attribute Darrick J. Wong
2023-03-15 0:52 ` [PATCH 05/15] report: record fstests start and report generation timestamps Darrick J. Wong
2023-03-15 0:53 ` [PATCH 06/15] report: encode cdata sections correctly Darrick J. Wong
2023-03-15 0:53 ` [PATCH 07/15] report: encode the kernel log as a separate xml element Darrick J. Wong
2023-03-15 0:53 ` [PATCH 08/15] report: sort properties by name Darrick J. Wong
2023-03-15 0:53 ` [PATCH 09/15] report: pass property value to _xunit_add_property Darrick J. Wong
2023-03-15 0:53 ` [PATCH 10/15] report: encode xml entities in property values Darrick J. Wong
2023-03-15 0:53 ` [PATCH 11/15] report: collect basic information about a test run Darrick J. Wong
2023-03-15 0:53 ` [PATCH 12/15] report: record optional environment variables Darrick J. Wong
2023-03-15 0:53 ` [PATCH 13/15] report: record xfs-specific information about a test run Darrick J. Wong
2023-03-15 0:53 ` [PATCH 14/15] report: record ext*-specific " Darrick J. Wong
2023-03-15 0:53 ` [PATCH 15/15] report: allow test runners to inject arbitrary values Darrick J. Wong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).