From: osstest service owner <osstest-admin@xenproject.org>
To: xen-devel@lists.xenproject.org
Subject: [xen-unstable-smoke bisection] complete build-amd64
Date: Tue, 28 Nov 2023 17:02:17 +0000 [thread overview]
Message-ID: <E1r81TV-0002fV-Uv@osstest.test-lab.xenproject.org> (raw)
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 15042 bytes --]
branch xen-unstable-smoke
xenbranch xen-unstable-smoke
job build-amd64
testid xen-build/dist-test
Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git
Tree: qemuu git://xenbits.xen.org/qemu-xen.git
Tree: xen git://xenbits.xen.org/xen.git
*** Found and reproduced problem changeset ***
Bug is in tree: xen git://xenbits.xen.org/xen.git
Bug introduced: 82182ad7b46e0f7a3856bb12c7a9bf2e2a4570bc
Bug not present: 46f2e2c3bcd5b17dae0fd1e45ed8619d6c047b55
Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/183905/
commit 82182ad7b46e0f7a3856bb12c7a9bf2e2a4570bc
Author: Roger Pau Monné <roger.pau@citrix.com>
Date: Mon Nov 27 15:16:01 2023 +0100
livepatch: do not use .livepatch.funcs section to store internal state
Currently the livepatch logic inside of Xen will use fields of struct
livepatch_func in order to cache internal state of patched functions. Note
this is a field that is part of the payload, and is loaded as an ELF section
(.livepatch.funcs), taking into account the SHF_* flags in the section
header.
The flags for the .livepatch.funcs section, as set by livepatch-build-tools,
are SHF_ALLOC, which leads to its contents (the array of livepatch_func
structures) being placed in read-only memory:
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[...]
[ 4] .livepatch.funcs PROGBITS 0000000000000000 00000080
0000000000000068 0000000000000000 A 0 0 8
This previously went unnoticed, as all writes to the fields of livepatch_func
happen in the critical region that had WP disabled in CR0. After 8676092a0f16
however WP is no longer toggled in CR0 for patch application, and only the
hypervisor .text mappings are made write-accessible. That leads to the
following page fault when attempting to apply a livepatch:
----[ Xen-4.19-unstable x86_64 debug=y Tainted: C ]----
CPU: 4
RIP: e008:[<ffff82d040221e81>] common/livepatch.c#apply_payload+0x45/0x1e1
[...]
Xen call trace:
[<ffff82d040221e81>] R common/livepatch.c#apply_payload+0x45/0x1e1
[<ffff82d0402235b2>] F check_for_livepatch_work+0x385/0xaa5
[<ffff82d04032508f>] F arch/x86/domain.c#idle_loop+0x92/0xee
Pagetable walk from ffff82d040625079:
L4[0x105] = 000000008c6c9063 ffffffffffffffff
L3[0x141] = 000000008c6c6063 ffffffffffffffff
L2[0x003] = 000000086a1e7063 ffffffffffffffff
L1[0x025] = 800000086ca5d121 ffffffffffffffff
****************************************
Panic on CPU 4:
FATAL PAGE FAULT
[error_code=0003]
Faulting linear address: ffff82d040625079
****************************************
Fix this by moving the internal Xen function patching state out of
livepatch_func into an area not allocated as part of the ELF payload. While
there also constify the array of livepatch_func structures in order to prevent
further surprises.
Note there's still one field (old_addr) that gets set during livepatch load. I
consider this fine since the field is read-only after load, and at the point
the field gets set the underlying mapping hasn't been made read-only yet.
Fixes: 8676092a0f16 ('x86/livepatch: Fix livepatch application when CET is active')
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Reviewed-by: Ross Lagerwall <ross.lagerwall@citrix.com>
For bisection revision-tuple graph see:
http://logs.test-lab.xenproject.org/osstest/results/bisect/xen-unstable-smoke/build-amd64.xen-build--dist-test.html
Revision IDs in each graph node refer, respectively, to the Trees above.
----------------------------------------
Running cs-bisection-step --graph-out=/home/logs/results/bisect/xen-unstable-smoke/build-amd64.xen-build--dist-test --summary-out=tmp/183905.bisection-summary --basis-template=183851 --blessings=real,real-bisect,real-retry xen-unstable-smoke build-amd64 xen-build/dist-test
Searching for failure / basis pass:
183891 fail [host=godello1] / 183851 [host=himrod2] 183846 [host=himrod2] 183844 [host=himrod2] 183840 [host=albana0] 183826 [host=himrod0] 183817 [host=nobling1] 183814 [host=godello0] 183809 [host=nobling1] 183802 [host=godello0] 183798 [host=godello0] 183786 [host=himrod2] 183784 [host=himrod2] 183774 [host=himrod2] 183770 [host=nobling1] 183767 [host=himrod2] 183762 ok.
Failure / basis pass flights: 183891 / 183762
(tree with no url: minios)
(tree with no url: ovmf)
(tree with no url: seabios)
Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git
Tree: qemuu git://xenbits.xen.org/qemu-xen.git
Tree: xen git://xenbits.xen.org/xen.git
Latest 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 72d51813d631fe27d37736b7a55eeec08f246983
Basis pass 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 7ad0c774e474f6d95dfda868d876af507d399657
Generating revisions with ./adhoc-revtuple-generator git://xenbits.xen.org/qemu-xen-traditional.git#3d273dd05e51e5a1ffba3d98c7437ee84e8f8764-3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 git://xenbits.xen.org/qemu-xen.git#0df9387c8983e1b1e72d8c574356f572342c03e6-0df9387c8983e1b1e72d8c574356f572342c03e6 git://xenbits.xen.org/xen.git#7ad0c774e474f6d95dfda868d876af507d399657-72d51813d631fe27d37736b7a55eeec08f246983
Loaded 5001 nodes in revision graph
Searching for test results:
183762 pass 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 7ad0c774e474f6d95dfda868d876af507d399657
183767 [host=himrod2]
183770 [host=nobling1]
183774 [host=himrod2]
183784 [host=himrod2]
183786 [host=himrod2]
183798 [host=godello0]
183802 [host=godello0]
183809 [host=nobling1]
183814 [host=godello0]
183817 [host=nobling1]
183826 [host=himrod0]
183840 [host=albana0]
183844 [host=himrod2]
183846 [host=himrod2]
183851 [host=himrod2]
183878 [host=albana0]
183881 [host=albana0]
183883 [host=albana0]
183885 [host=albana0]
183886 [host=albana0]
183887 [host=albana0]
183882 fail 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 72d51813d631fe27d37736b7a55eeec08f246983
183888 [host=albana0]
183889 pass 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 7ad0c774e474f6d95dfda868d876af507d399657
183892 fail 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 72d51813d631fe27d37736b7a55eeec08f246983
183893 pass 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 625f2cc66f53c7f1bf56562bf96c06510087d484
183894 pass 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 0dd323133022933dfb03de984c50eadd697cdd71
183891 fail 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 72d51813d631fe27d37736b7a55eeec08f246983
183896 pass 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 f02829592efe4f55f6d95bb9e2359717109e8ebc
183897 pass 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 fbcec32d6d3ea0ac329301925b317478316209ed
183898 fail 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 09c2fe438da1dfc83f70d921b52240bea576615f
183900 pass 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 46f2e2c3bcd5b17dae0fd1e45ed8619d6c047b55
183901 fail 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 82182ad7b46e0f7a3856bb12c7a9bf2e2a4570bc
183902 pass 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 46f2e2c3bcd5b17dae0fd1e45ed8619d6c047b55
183903 fail 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 82182ad7b46e0f7a3856bb12c7a9bf2e2a4570bc
183904 pass 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 46f2e2c3bcd5b17dae0fd1e45ed8619d6c047b55
183905 fail 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 82182ad7b46e0f7a3856bb12c7a9bf2e2a4570bc
Searching for interesting versions
Result found: flight 183762 (pass), for basis pass
For basis failure, parent search stopping at 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 46f2e2c3bcd5b17dae0fd1e45ed8619d6c047b55, results HASH(0x557f0ca6be10) HASH(0x557f0c579850) HASH(0x557f0c581f50) For basis failure, parent search stopping at 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 fbcec32d6d3ea0ac329301925b317478316209ed, results HASH(0x557f0cae6e98) For basis failure, parent search stopping at 3d273dd05e51e5a1\
ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 f02829592efe4f55f6d95bb9e2359717109e8ebc, results HASH(0x557f0cae4bb0) For basis failure, parent search stopping at 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 0dd323133022933dfb03de984c50eadd697cdd71, results HASH(0x557f0cadfb78) For basis failure, parent search stopping at 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 625f2cc66f53c7f1bf56562bf96c06510087d4\
84, results HASH(0x557f0cade140) For basis failure, parent search stopping at 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 7ad0c774e474f6d95dfda868d876af507d399657, results HASH(0x557f0ca75680) HASH(0x557f0ca7ef50) Result found: flight 183882 (fail), for basis failure (at ancestor ~2433)
Repro found: flight 183889 (pass), for basis pass
Repro found: flight 183891 (fail), for basis failure
0 revisions at 3d273dd05e51e5a1ffba3d98c7437ee84e8f8764 0df9387c8983e1b1e72d8c574356f572342c03e6 46f2e2c3bcd5b17dae0fd1e45ed8619d6c047b55
No revisions left to test, checking graph state.
Result found: flight 183900 (pass), for last pass
Result found: flight 183901 (fail), for first failure
Repro found: flight 183902 (pass), for last pass
Repro found: flight 183903 (fail), for first failure
Repro found: flight 183904 (pass), for last pass
Repro found: flight 183905 (fail), for first failure
*** Found and reproduced problem changeset ***
Bug is in tree: xen git://xenbits.xen.org/xen.git
Bug introduced: 82182ad7b46e0f7a3856bb12c7a9bf2e2a4570bc
Bug not present: 46f2e2c3bcd5b17dae0fd1e45ed8619d6c047b55
Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/183905/
commit 82182ad7b46e0f7a3856bb12c7a9bf2e2a4570bc
Author: Roger Pau Monné <roger.pau@citrix.com>
Date: Mon Nov 27 15:16:01 2023 +0100
livepatch: do not use .livepatch.funcs section to store internal state
Currently the livepatch logic inside of Xen will use fields of struct
livepatch_func in order to cache internal state of patched functions. Note
this is a field that is part of the payload, and is loaded as an ELF section
(.livepatch.funcs), taking into account the SHF_* flags in the section
header.
The flags for the .livepatch.funcs section, as set by livepatch-build-tools,
are SHF_ALLOC, which leads to its contents (the array of livepatch_func
structures) being placed in read-only memory:
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[...]
[ 4] .livepatch.funcs PROGBITS 0000000000000000 00000080
0000000000000068 0000000000000000 A 0 0 8
This previously went unnoticed, as all writes to the fields of livepatch_func
happen in the critical region that had WP disabled in CR0. After 8676092a0f16
however WP is no longer toggled in CR0 for patch application, and only the
hypervisor .text mappings are made write-accessible. That leads to the
following page fault when attempting to apply a livepatch:
----[ Xen-4.19-unstable x86_64 debug=y Tainted: C ]----
CPU: 4
RIP: e008:[<ffff82d040221e81>] common/livepatch.c#apply_payload+0x45/0x1e1
[...]
Xen call trace:
[<ffff82d040221e81>] R common/livepatch.c#apply_payload+0x45/0x1e1
[<ffff82d0402235b2>] F check_for_livepatch_work+0x385/0xaa5
[<ffff82d04032508f>] F arch/x86/domain.c#idle_loop+0x92/0xee
Pagetable walk from ffff82d040625079:
L4[0x105] = 000000008c6c9063 ffffffffffffffff
L3[0x141] = 000000008c6c6063 ffffffffffffffff
L2[0x003] = 000000086a1e7063 ffffffffffffffff
L1[0x025] = 800000086ca5d121 ffffffffffffffff
****************************************
Panic on CPU 4:
FATAL PAGE FAULT
[error_code=0003]
Faulting linear address: ffff82d040625079
****************************************
Fix this by moving the internal Xen function patching state out of
livepatch_func into an area not allocated as part of the ELF payload. While
there also constify the array of livepatch_func structures in order to prevent
further surprises.
Note there's still one field (old_addr) that gets set during livepatch load. I
consider this fine since the field is read-only after load, and at the point
the field gets set the underlying mapping hasn't been made read-only yet.
Fixes: 8676092a0f16 ('x86/livepatch: Fix livepatch application when CET is active')
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Reviewed-by: Ross Lagerwall <ross.lagerwall@citrix.com>
pnmtopng: 243 colors found
Revision graph left in /home/logs/results/bisect/xen-unstable-smoke/build-amd64.xen-build--dist-test.{dot,ps,png,html,svg}.
----------------------------------------
183905: tolerable all pass
flight 183905 xen-unstable-smoke real-bisect [real]
http://logs.test-lab.xenproject.org/osstest/logs/183905/
Failures :-/ but no regressions.
Tests which did not succeed,
including tests which could not be run:
build-amd64 7 xen-build/dist-test fail baseline untested
jobs:
build-amd64 pass
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
next reply other threads:[~2023-11-28 17:02 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-28 17:02 osstest service owner [this message]
-- strict thread matches above, loose matches on Subject: below --
2023-09-06 16:46 [xen-unstable-smoke bisection] complete build-amd64 osstest service owner
2023-06-07 16:24 osstest service owner
2023-03-24 22:28 osstest service owner
2022-10-16 8:24 osstest service owner
2022-03-15 2:25 osstest service owner
2022-01-21 3:38 osstest service owner
2022-01-21 7:54 ` Jan Beulich
2021-08-16 23:23 osstest service owner
2021-07-10 0:19 osstest service owner
2021-07-06 17:29 osstest service owner
2021-02-24 16:57 osstest service owner
2021-02-10 19:10 osstest service owner
2021-01-28 18:38 osstest service owner
2020-12-21 20:07 osstest service owner
2020-11-10 23:25 osstest service owner
2020-10-23 23:30 osstest service owner
2020-10-07 18:40 osstest service owner
2020-04-25 8:57 osstest service owner
2019-05-15 8:13 osstest service owner
2019-04-13 5:09 osstest service owner
2019-04-13 15:51 ` Julien Grall
2019-04-13 15:53 ` Andrew Cooper
2019-01-02 21:14 osstest service owner
2018-11-12 23:38 osstest service owner
2018-09-25 21:39 osstest service owner
2018-04-07 6:19 osstest service owner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=E1r81TV-0002fV-Uv@osstest.test-lab.xenproject.org \
--to=osstest-admin@xenproject.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).