All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: [Qemu-devel] [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
  2017-08-18 14:23 ` Dongjiu Geng
  (?)
@ 2017-08-18 14:21   ` no-reply
  -1 siblings, 0 replies; 129+ messages in thread
From: no-reply @ 2017-08-18 14:21 UTC (permalink / raw)
  To: gengdongjiu
  Cc: famz, mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, zhengqiang10,
	wuquanming, huangshaoyu

Hi,

This series failed build test on s390x host. Please find the details below.

Type: series
Message-id: 1503066227-18251-1-git-send-email-gengdongjiu@huawei.com
Subject: [Qemu-devel] [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e
echo "=== ENV ==="
env
echo "=== PACKAGES ==="
rpm -qa
echo "=== TEST BEGIN ==="
CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
echo -n "Using CC: "
realpath $CC
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1
make install
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]         patchew/1503066227-18251-1-git-send-email-gengdongjiu@huawei.com -> patchew/1503066227-18251-1-git-send-email-gengdongjiu@huawei.com
Switched to a new branch 'test'
06d4dbd target-arm: kvm64: Handle SError interrupt for the guest OS
8e5e362 target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
5d9b3fa target-arm: kvm64: detect guest RAS EXTENSION feature
e79f876 ACPI: build and enable APEI GHES in the Makefile and configuration
5e11799 ACPI: Add APEI GHES Table Generation support
d691b66 ACPI: add APEI/HEST/CPER structures and macros

=== OUTPUT BEGIN ===
=== ENV ===
XDG_SESSION_ID=32305
SHELL=/bin/sh
USER=fam
PATCHEW=/home/fam/patchew/patchew-cli -s http://patchew.org --nodebug
PATH=/usr/bin:/bin
PWD=/var/tmp/patchew-tester-tmp-hm2hdbnw/src
LANG=en_US.UTF-8
HOME=/home/fam
SHLVL=2
LOGNAME=fam
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1012/bus
XDG_RUNTIME_DIR=/run/user/1012
_=/usr/bin/env
=== PACKAGES ===
gpg-pubkey-873529b8-54e386ff
xz-libs-5.2.2-2.fc24.s390x
libxshmfence-1.2-3.fc24.s390x
giflib-4.1.6-15.fc24.s390x
trousers-lib-0.3.13-6.fc24.s390x
ncurses-base-6.0-6.20160709.fc25.noarch
gmp-6.1.1-1.fc25.s390x
libidn-1.33-1.fc25.s390x
slang-2.3.0-7.fc25.s390x
pkgconfig-0.29.1-1.fc25.s390x
alsa-lib-1.1.1-2.fc25.s390x
yum-metadata-parser-1.1.4-17.fc25.s390x
python3-slip-dbus-0.6.4-4.fc25.noarch
python2-cssselect-0.9.2-1.fc25.noarch
createrepo_c-libs-0.10.0-6.fc25.s390x
initscripts-9.69-1.fc25.s390x
parted-3.2-21.fc25.s390x
flex-2.6.0-3.fc25.s390x
colord-libs-1.3.4-1.fc25.s390x
python-osbs-client-0.33-3.fc25.noarch
perl-Pod-Simple-3.35-1.fc25.noarch
python2-simplejson-3.10.0-1.fc25.s390x
brltty-5.4-2.fc25.s390x
librados2-10.2.4-2.fc25.s390x
tcp_wrappers-7.6-83.fc25.s390x
libcephfs_jni1-10.2.4-2.fc25.s390x
nettle-devel-3.3-1.fc25.s390x
bzip2-devel-1.0.6-21.fc25.s390x
libuuid-2.28.2-2.fc25.s390x
python3-dnf-1.1.10-6.fc25.noarch
texlive-kpathsea-doc-svn41139-33.fc25.1.noarch
openssh-7.4p1-4.fc25.s390x
texlive-kpathsea-bin-svn40473-33.20160520.fc25.1.s390x
texlive-graphics-svn41015-33.fc25.1.noarch
texlive-dvipdfmx-def-svn40328-33.fc25.1.noarch
texlive-mfware-svn40768-33.fc25.1.noarch
texlive-texlive-scripts-svn41433-33.fc25.1.noarch
texlive-euro-svn22191.1.1-33.fc25.1.noarch
texlive-etex-svn37057.0-33.fc25.1.noarch
texlive-iftex-svn29654.0.2-33.fc25.1.noarch
texlive-palatino-svn31835.0-33.fc25.1.noarch
texlive-texlive-docindex-svn41430-33.fc25.1.noarch
texlive-xunicode-svn30466.0.981-33.fc25.1.noarch
texlive-koma-script-svn41508-33.fc25.1.noarch
texlive-pst-grad-svn15878.1.06-33.fc25.1.noarch
texlive-pst-blur-svn15878.2.0-33.fc25.1.noarch
texlive-jknapltx-svn19440.0-33.fc25.1.noarch
texinfo-6.1-4.fc25.s390x
openssl-devel-1.0.2k-1.fc25.s390x
gdk-pixbuf2-2.36.6-1.fc25.s390x
nspr-4.14.0-2.fc25.s390x
nss-softokn-freebl-3.30.2-1.0.fc25.s390x
jansson-2.10-2.fc25.s390x
fedora-repos-25-4.noarch
python3-libs-3.5.3-6.fc25.s390x
perl-Errno-1.25-387.fc25.s390x
acl-2.2.52-13.fc25.s390x
pcre2-utf16-10.23-8.fc25.s390x
pango-1.40.5-1.fc25.s390x
systemd-pam-231-17.fc25.s390x
python2-gluster-3.10.4-1.fc25.s390x
NetworkManager-libnm-1.4.4-5.fc25.s390x
selinux-policy-3.13.1-225.18.fc25.noarch
poppler-0.45.0-5.fc25.s390x
ccache-3.3.4-1.fc25.s390x
valgrind-3.12.0-9.fc25.s390x
perl-open-1.10-387.fc25.noarch
libaio-0.3.110-6.fc24.s390x
libfontenc-1.1.3-3.fc24.s390x
lzo-2.08-8.fc24.s390x
isl-0.14-5.fc24.s390x
libXau-1.0.8-6.fc24.s390x
linux-atm-libs-2.5.1-14.fc24.s390x
libXext-1.3.3-4.fc24.s390x
libXxf86vm-1.1.4-3.fc24.s390x
bison-3.0.4-4.fc24.s390x
perl-srpm-macros-1-20.fc25.noarch
gawk-4.1.3-8.fc25.s390x
libwayland-client-1.12.0-1.fc25.s390x
perl-Exporter-5.72-366.fc25.noarch
perl-version-0.99.17-1.fc25.s390x
fftw-libs-double-3.3.5-3.fc25.s390x
libssh2-1.8.0-1.fc25.s390x
ModemManager-glib-1.6.4-1.fc25.s390x
newt-python3-0.52.19-2.fc25.s390x
python-munch-2.0.4-3.fc25.noarch
python-bugzilla-1.2.2-4.fc25.noarch
libedit-3.1-16.20160618cvs.fc25.s390x
createrepo_c-0.10.0-6.fc25.s390x
device-mapper-multipath-libs-0.4.9-83.fc25.s390x
yum-3.4.3-510.fc25.noarch
dracut-config-rescue-044-78.fc25.s390x
mozjs17-17.0.0-16.fc25.s390x
libselinux-2.5-13.fc25.s390x
libgo-devel-6.3.1-1.fc25.s390x
python2-pyparsing-2.1.10-1.fc25.noarch
cairo-gobject-1.14.8-1.fc25.s390x
ethtool-4.8-1.fc25.s390x
xorg-x11-proto-devel-7.7-20.fc25.noarch
brlapi-0.6.5-2.fc25.s390x
librados-devel-10.2.4-2.fc25.s390x
libXinerama-devel-1.1.3-6.fc24.s390x
lua-posix-33.3.1-3.fc25.s390x
usbredir-devel-0.7.1-2.fc24.s390x
libepoxy-1.4.1-1.fc25.s390x
python3-dnf-plugins-core-0.1.21-5.fc25.noarch
texlive-pdftex-doc-svn41149-33.fc25.1.noarch
openssh-clients-7.4p1-4.fc25.s390x
iptables-1.6.0-3.fc25.s390x
texlive-texlive.infra-svn41280-33.fc25.1.noarch
texlive-graphics-cfg-svn40269-33.fc25.1.noarch
texlive-bibtex-svn40768-33.fc25.1.noarch
texlive-mfware-bin-svn40473-33.20160520.fc25.1.s390x
texlive-texlive-scripts-bin-svn29741.0-33.20160520.fc25.1.noarch
texlive-sauerj-svn15878.0-33.fc25.1.noarch
texlive-enctex-svn34957.0-33.fc25.1.noarch
texlive-ifetex-svn24853.1.2-33.fc25.1.noarch
texlive-ntgclass-svn15878.2.1a-33.fc25.1.noarch
texlive-tex-gyre-math-svn41264-33.fc25.1.noarch
texlive-bera-svn20031.0-33.fc25.1.noarch
texlive-ms-svn29849.0-33.fc25.1.noarch
texlive-pst-fill-svn15878.1.01-33.fc25.1.noarch
texlive-ctable-svn38672-33.fc25.1.noarch
texlive-extsizes-svn17263.1.4a-33.fc25.1.noarch
texlive-collection-latexrecommended-svn35765.0-33.20160520.fc25.1.noarch
perl-Filter-1.57-1.fc25.s390x
krb5-workstation-1.14.4-7.fc25.s390x
python2-rpm-macros-3-12.fc25.noarch
libglvnd-egl-0.2.999-14.20170308git8e6e102.fc25.s390x
libglvnd-opengl-0.2.999-14.20170308git8e6e102.fc25.s390x
gdbm-1.13-1.fc25.s390x
nss-util-3.30.2-1.0.fc25.s390x
libcrypt-nss-2.24-9.fc25.s390x
libtasn1-4.12-1.fc25.s390x
fedora-release-25-2.noarch
gdb-headless-7.12.1-48.fc25.s390x
perl-macros-5.24.2-387.fc25.s390x
sqlite-devel-3.14.2-2.fc25.s390x
pcre-devel-8.41-1.fc25.s390x
libX11-1.6.5-1.fc25.s390x
coreutils-8.25-17.fc25.s390x
python2-openidc-client-0-3.20170523git77cb3ee.fc25.noarch
nss-sysinit-3.30.2-1.1.fc25.s390x
systemtap-client-3.1-5.fc25.s390x
nss-devel-3.30.2-1.1.fc25.s390x
firewalld-0.4.4.5-1.fc25.noarch
rpmlint-1.9-9.fc25.noarch
system-python-3.5.3-6.fc25.s390x
gpg-pubkey-efe550f5-5220ba41
gpg-pubkey-81b46521-55b3ca9a
filesystem-3.2-37.fc24.s390x
libffi-3.1-9.fc24.s390x
keyutils-libs-1.5.9-8.fc24.s390x
libnfnetlink-1.0.1-8.fc24.s390x
libtheora-1.1.1-14.fc24.s390x
xml-common-0.6.3-44.fc24.noarch
autoconf-2.69-22.fc24.noarch
libXt-1.1.5-3.fc24.s390x
kbd-legacy-2.0.3-3.fc24.noarch
ghostscript-fonts-5.50-35.fc24.noarch
libXevie-1.0.3-11.fc24.s390x
libcap-2.25-2.fc25.s390x
mpfr-3.1.5-1.fc25.s390x
perl-Carp-1.40-365.fc25.noarch
libmnl-1.0.4-1.fc25.s390x
perl-Unicode-EastAsianWidth-1.33-8.fc25.noarch
libwayland-cursor-1.12.0-1.fc25.s390x
python-krbV-1.0.90-12.fc25.s390x
python2-urllib3-1.15.1-3.fc25.noarch
fipscheck-1.4.1-11.fc25.s390x
libndp-1.6-1.fc25.s390x
gnupg2-2.1.13-2.fc25.s390x
libXfixes-5.0.3-1.fc25.s390x
adwaita-icon-theme-3.22.0-1.fc25.noarch
dconf-0.26.0-1.fc25.s390x
ncurses-devel-6.0-6.20160709.fc25.s390x
dejagnu-1.6-1.fc25.noarch
libstdc++-devel-6.3.1-1.fc25.s390x
device-mapper-1.02.136-3.fc25.s390x
subversion-1.9.5-1.fc25.s390x
libtool-ltdl-2.4.6-13.fc25.s390x
libevent-2.0.22-1.fc25.s390x
atk-devel-2.22.0-1.fc25.s390x
libev-4.24-1.fc25.s390x
xorg-x11-fonts-Type1-7.5-16.fc24.noarch
brlapi-devel-0.6.5-2.fc25.s390x
pulseaudio-libs-10.0-2.fc25.s390x
glib2-2.50.3-1.fc25.s390x
python2-rpm-4.13.0.1-1.fc25.s390x
dnf-1.1.10-6.fc25.noarch
texlive-metafont-bin-svn40987-33.20160520.fc25.1.s390x
texlive-xkeyval-svn35741.2.7a-33.fc25.1.noarch
texlive-euler-svn17261.2.5-33.fc25.1.noarch
texlive-mptopdf-svn41282-33.fc25.1.noarch
texlive-wasy-svn35831.0-33.fc25.1.noarch
texlive-avantgar-svn31835.0-33.fc25.1.noarch
texlive-eurosym-svn17265.1.4_subrfix-33.fc25.1.noarch
texlive-knuth-lib-svn35820.0-33.fc25.1.noarch
texlive-parallel-svn15878.0-33.fc25.1.noarch
texlive-texlive-msg-translations-svn41431-33.fc25.1.noarch
texlive-latex-svn40218-33.fc25.1.noarch
texlive-lualatex-math-svn40621-33.fc25.1.noarch
texlive-auto-pst-pdf-svn23723.0.6-33.fc25.1.noarch
texlive-powerdot-svn38984-33.fc25.1.noarch
texlive-wasysym-svn15878.2.0-33.fc25.1.noarch
ImageMagick-libs-6.9.3.0-6.fc25.s390x
geoclue2-2.4.5-1.fc25.s390x
perl-IO-Socket-IP-0.39-1.fc25.noarch
python2-pyasn1-0.2.3-1.fc25.noarch
libglvnd-0.2.999-14.20170308git8e6e102.fc25.s390x
libglvnd-gles-0.2.999-14.20170308git8e6e102.fc25.s390x
gdk-pixbuf2-devel-2.36.6-1.fc25.s390x
at-spi2-core-devel-2.22.1-1.fc25.s390x
libacl-2.2.52-13.fc25.s390x
perl-libs-5.24.2-387.fc25.s390x
mesa-libglapi-17.0.5-3.fc25.s390x
kernel-headers-4.11.10-200.fc25.s390x
p11-kit-devel-0.23.2-4.fc25.s390x
python3-rpmconf-1.0.19-1.fc25.noarch
sqlite-3.14.2-2.fc25.s390x
pcre-utf32-8.41-1.fc25.s390x
libX11-common-1.6.5-1.fc25.noarch
coreutils-common-8.25-17.fc25.s390x
mesa-libEGL-17.0.5-3.fc25.s390x
nss-3.30.2-1.1.fc25.s390x
systemtap-runtime-3.1-5.fc25.s390x
NetworkManager-glib-1.4.4-5.fc25.s390x
audit-2.7.7-1.fc25.s390x
perl-Time-HiRes-1.9742-1.fc25.s390x
libsolv-0.6.28-1.fc25.s390x
gpg-pubkey-34ec9cba-54e38751
gpg-pubkey-030d5aed-55b577f0
basesystem-11-2.fc24.noarch
libmpc-1.0.2-5.fc24.s390x
libunistring-0.9.4-3.fc24.s390x
libmodman-2.0.1-12.fc24.s390x
lsscsi-0.28-3.fc24.s390x
kbd-misc-2.0.3-3.fc24.noarch
kmod-23-1.fc25.s390x
newt-0.52.19-2.fc25.s390x
perl-Text-Unidecode-1.27-3.fc25.noarch
plymouth-core-libs-0.9.3-0.6.20160620git0e65b86c.fc25.s390x
which-2.21-1.fc25.s390x
python3-slip-0.6.4-4.fc25.noarch
python3-systemd-232-1.fc25.s390x
python-lockfile-0.11.0-4.fc25.noarch
python2-requests-2.10.0-4.fc25.noarch
libnghttp2-1.13.0-2.fc25.s390x
python-urlgrabber-3.10.1-9.fc25.noarch
iputils-20161105-1.fc25.s390x
rest-0.8.0-1.fc25.s390x
adwaita-cursor-theme-3.22.0-1.fc25.noarch
authconfig-6.2.10-14.fc25.s390x
automake-1.15-7.fc25.noarch
shared-mime-info-1.8-1.fc25.s390x
pigz-2.3.4-1.fc25.s390x
device-mapper-libs-1.02.136-3.fc25.s390x
dnsmasq-2.76-2.fc25.s390x
fedora-packager-0.6.0.1-1.fc25.noarch
gcc-c++-6.3.1-1.fc25.s390x
libwebp-0.5.2-1.fc25.s390x
boost-system-1.60.0-10.fc25.s390x
libasyncns-0.8-10.fc24.s390x
libXau-devel-1.0.8-6.fc24.s390x
libverto-libev-0.2.6-6.fc24.s390x
python3-html5lib-0.999-9.fc25.noarch
ttmkfdir-3.0.9-48.fc24.s390x
pulseaudio-libs-glib2-10.0-2.fc25.s390x
wpa_supplicant-2.6-1.fc25.s390x
texlive-lib-2016-33.20160520.fc25.s390x
libXi-devel-1.7.9-1.fc25.s390x
python3-distro-1.0.3-1.fc25.noarch
rpm-plugin-systemd-inhibit-4.13.0.1-1.fc25.s390x
texlive-texlive-common-doc-svn40682-33.fc25.1.noarch
packagedb-cli-2.14.1-1.fc25.noarch
texlive-metafont-svn40793-33.fc25.1.noarch
texlive-tools-svn40934-33.fc25.1.noarch
texlive-enumitem-svn24146.3.5.2-33.fc25.1.noarch
texlive-mptopdf-bin-svn18674.0-33.20160520.fc25.1.noarch
texlive-underscore-svn18261.0-33.fc25.1.noarch
texlive-anysize-svn15878.0-33.fc25.1.noarch
texlive-euenc-svn19795.0.1h-33.fc25.1.noarch
texlive-kastrup-svn15878.0-33.fc25.1.noarch
texlive-paralist-svn39247-33.fc25.1.noarch
texlive-texlive-en-svn41185-33.fc25.1.noarch
texlive-tipa-svn29349.1.3-33.fc25.1.noarch
texlive-currfile-svn40725-33.fc25.1.noarch
texlive-pst-node-svn40743-33.fc25.1.noarch
texlive-pst-slpe-svn24391.1.31-33.fc25.1.noarch
texlive-typehtml-svn17134.0-33.fc25.1.noarch
SDL2-devel-2.0.5-3.fc25.s390x
libcroco-0.6.11-3.fc25.s390x
bluez-libs-devel-5.44-1.fc25.s390x
kernel-4.10.8-200.fc25.s390x
expat-2.2.1-1.fc25.s390x
system-python-libs-3.5.3-6.fc25.s390x
pcre2-10.23-8.fc25.s390x
firewalld-filesystem-0.4.4.5-1.fc25.noarch
pcre-cpp-8.41-1.fc25.s390x
python3-firewall-0.4.4.5-1.fc25.noarch
freetype-devel-2.6.5-9.fc25.s390x
pcre-utf16-8.41-1.fc25.s390x
linux-firmware-20170605-74.git37857004.fc25.noarch
distribution-gpg-keys-1.12-1.fc25.noarch
kernel-modules-4.11.10-200.fc25.s390x
gnutls-devel-3.5.14-1.fc25.s390x
systemtap-devel-3.1-5.fc25.s390x
java-1.8.0-openjdk-1.8.0.131-1.b12.fc25.s390x
polkit-0.113-8.fc25.s390x
perl-SelfLoader-1.23-387.fc25.noarch
libdb-utils-5.3.28-24.fc25.s390x
fontpackages-filesystem-1.44-17.fc24.noarch
groff-base-1.22.3-8.fc24.s390x
ilmbase-2.2.0-5.fc24.s390x
OpenEXR-libs-2.2.0-5.fc24.s390x
hesiod-3.2.1-6.fc24.s390x
sysfsutils-2.1.0-19.fc24.s390x
ocaml-srpm-macros-2-4.fc24.noarch
mailx-12.5-19.fc24.s390x
ncurses-libs-6.0-6.20160709.fc25.s390x
ipset-libs-6.29-1.fc25.s390x
gmp-devel-6.1.1-1.fc25.s390x
python-pip-8.1.2-2.fc25.noarch
harfbuzz-1.3.2-1.fc25.s390x
python2-iniparse-0.4-20.fc25.noarch
python3-iniparse-0.4-20.fc25.noarch
python3-kickstart-2.32-1.fc25.noarch
perl-Net-SSLeay-1.78-1.fc25.s390x
drpm-0.3.0-3.fc25.s390x
glib-networking-2.50.0-1.fc25.s390x
webkitgtk3-2.4.11-3.fc25.s390x
libXaw-1.0.13-4.fc25.s390x
xorg-x11-font-utils-7.5-32.fc25.s390x
hardlink-1.1-1.fc25.s390x
libcom_err-1.43.3-1.fc25.s390x
python2-dateutil-2.6.0-1.fc25.noarch
libXpm-3.5.12-1.fc25.s390x
python2-smmap-2.0.1-1.fc25.noarch
poppler-data-0.4.7-6.fc25.noarch
librbd1-10.2.4-2.fc25.s390x
perl-Digest-MD5-2.55-2.fc25.s390x
wayland-protocols-devel-1.7-1.fc25.noarch
texi2html-5.0-4.fc24.noarch
libxkbcommon-0.7.1-1.fc25.s390x
libuuid-devel-2.28.2-2.fc25.s390x
libcacard-2.5.3-1.fc25.s390x
libwmf-lite-0.2.8.4-50.fc25.s390x
texlive-tetex-svn41059-33.fc25.1.noarch
texlive-thumbpdf-svn34621.3.16-33.fc25.1.noarch
texlive-carlisle-svn18258.0-33.fc25.1.noarch
texlive-makeindex-bin-svn40473-33.20160520.fc25.1.s390x
texlive-pdftex-svn41149-33.fc25.1.noarch
texlive-csquotes-svn39538-33.fc25.1.noarch
texlive-courier-svn35058.0-33.fc25.1.noarch
texlive-helvetic-svn31835.0-33.fc25.1.noarch
texlive-mfnfss-svn19410.0-33.fc25.1.noarch
texlive-sepnum-svn20186.2.0-33.fc25.1.noarch
texlive-utopia-svn15878.0-33.fc25.1.noarch
texlive-luatexbase-svn38550-33.fc25.1.noarch
texlive-pst-3d-svn17257.1.10-33.fc25.1.noarch
texlive-latex-bin-bin-svn14050.0-33.20160520.fc25.1.noarch
texlive-l3experimental-svn41163-33.fc25.1.noarch
net-tools-2.0-0.40.20160329git.fc25.s390x
perl-Pod-Perldoc-3.28-1.fc25.noarch
openssl-1.0.2k-1.fc25.s390x
man-pages-4.06-4.fc25.noarch
python3-magic-5.29-4.fc25.noarch
libxml2-2.9.4-2.fc25.s390x
nss-softokn-3.30.2-1.0.fc25.s390x
p11-kit-trust-0.23.2-4.fc25.s390x
emacs-filesystem-25.2-2.fc25.noarch
python3-dateutil-2.6.0-1.fc25.noarch
perl-threads-shared-1.57-1.fc25.s390x
libnotify-0.7.7-1.fc25.s390x
unzip-6.0-32.fc25.s390x
pango-devel-1.40.5-1.fc25.s390x
libdrm-2.4.81-1.fc25.s390x
python-beautifulsoup4-4.6.0-1.fc25.noarch
libcurl-7.51.0-7.fc25.s390x
dhcp-client-4.3.5-3.fc25.s390x
python2-fedora-0.9.0-6.fc25.noarch
emacs-25.2-2.fc25.s390x
gdb-7.12.1-48.fc25.s390x
expat-devel-2.2.1-1.fc25.s390x
gpg-pubkey-95a43f54-5284415a
dejavu-fonts-common-2.35-3.fc24.noarch
libSM-1.2.2-4.fc24.s390x
diffutils-3.3-13.fc24.s390x
libogg-1.3.2-5.fc24.s390x
hunspell-en-US-0.20140811.1-5.fc24.noarch
libdaemon-0.14-10.fc24.s390x
patch-2.7.5-3.fc24.s390x
libsysfs-2.1.0-19.fc24.s390x
procmail-3.22-39.fc24.s390x
libXdamage-1.1.4-8.fc24.s390x
libotf-0.9.13-7.fc24.s390x
urw-fonts-2.4-22.fc24.noarch
crontabs-1.11-12.20150630git.fc24.noarch
ppp-2.4.7-9.fc24.s390x
cyrus-sasl-2.1.26-26.2.fc24.s390x
zlib-devel-1.2.8-10.fc24.s390x
time-1.7-49.fc24.s390x
gpg-pubkey-fdb19c98-56fd6333
libcap-ng-0.7.8-1.fc25.s390x
binutils-2.26.1-1.fc25.s390x
lcms2-2.8-2.fc25.s390x
libcomps-0.1.7-5.fc25.s390x
apr-1.5.2-4.fc25.s390x
perl-constant-1.33-367.fc25.noarch
perl-Data-Dumper-2.161-1.fc25.s390x
ipcalc-0.1.8-1.fc25.s390x
gmp-c++-6.1.1-1.fc25.s390x
fontconfig-2.12.1-1.fc25.s390x
enchant-1.6.0-14.fc25.s390x
pyliblzma-0.5.3-16.fc25.s390x
libsepol-devel-2.5-10.fc25.s390x
python3-ordered-set-2.0.0-4.fc25.noarch
python-ipaddress-1.0.16-3.fc25.noarch
python2-kerberos-1.2.5-1.fc25.s390x
python2-pysocks-1.5.6-5.fc25.noarch
fipscheck-lib-1.4.1-11.fc25.s390x
libatomic_ops-7.4.4-1.fc25.s390x
net-snmp-agent-libs-5.7.3-13.fc25.s390x
dracut-044-78.fc25.s390x
python2-pygpgme-0.3-18.fc25.s390x
orc-0.4.26-1.fc25.s390x
yum-utils-1.1.31-511.fc25.noarch
libXrender-0.9.10-1.fc25.s390x
libXrandr-1.5.1-1.fc25.s390x
go-srpm-macros-2-7.fc25.noarch
gnupg2-smime-2.1.13-2.fc25.s390x
guile-devel-2.0.13-1.fc25.s390x
uboot-tools-2016.09.01-2.fc25.s390x
pykickstart-2.32-1.fc25.noarch
python-bunch-1.0.1-9.fc25.noarch
perl-generators-1.10-1.fc25.noarch
perl-Mozilla-CA-20160104-3.fc25.noarch
bzip2-libs-1.0.6-21.fc25.s390x
libpng-1.6.27-1.fc25.s390x
desktop-file-utils-0.23-2.fc25.s390x
python2-cccolutils-1.4-1.fc25.s390x
python2-lxml-3.7.2-1.fc25.s390x
redhat-rpm-config-45-1.fc25.noarch
device-mapper-event-libs-1.02.136-3.fc25.s390x
lvm2-libs-2.02.167-3.fc25.s390x
python2-gitdb-2.0.0-1.fc25.noarch
gcc-gfortran-6.3.1-1.fc25.s390x
libselinux-python-2.5-13.fc25.s390x
openjpeg2-2.1.2-3.fc25.s390x
boost-thread-1.60.0-10.fc25.s390x
librbd-devel-10.2.4-2.fc25.s390x
libXcursor-devel-1.1.14-6.fc24.s390x
latex2html-2012-7.fc24.noarch
lksctp-tools-1.0.16-5.fc24.s390x
libfdt-1.4.2-1.fc25.s390x
libXft-devel-2.3.2-4.fc24.s390x
libattr-devel-2.4.47-16.fc24.s390x
libiscsi-devel-1.15.0-2.fc24.s390x
gettext-0.19.8.1-3.fc25.s390x
libjpeg-turbo-devel-1.5.1-0.fc25.s390x
pulseaudio-libs-devel-10.0-2.fc25.s390x
libepoxy-devel-1.4.1-1.fc25.s390x
krb5-libs-1.14.4-7.fc25.s390x
libmount-2.28.2-2.fc25.s390x
python3-decorator-4.0.11-1.fc25.noarch
rpm-plugin-selinux-4.13.0.1-1.fc25.s390x
tzdata-java-2017b-1.fc25.noarch
python-srpm-macros-3-12.fc25.noarch
libsmartcols-2.28.2-2.fc25.s390x
kernel-core-4.10.5-200.fc25.s390x
kernel-modules-4.10.5-200.fc25.s390x
texlive-kpathsea-svn41139-33.fc25.1.noarch
texlive-amsmath-svn41561-33.fc25.1.noarch
texlive-thumbpdf-bin-svn6898.0-33.20160520.fc25.1.noarch
texlive-psnfss-svn33946.9.2a-33.fc25.1.noarch
texlive-subfig-svn15878.1.3-33.fc25.1.noarch
texlive-fancybox-svn18304.1.4-33.fc25.1.noarch
texlive-lua-alt-getopt-svn29349.0.7.0-33.fc25.1.noarch
texlive-natbib-svn20668.8.31b-33.fc25.1.noarch
texlive-pdftex-bin-svn40987-33.20160520.fc25.1.s390x
texlive-xdvi-svn40768-33.fc25.1.noarch
texlive-crop-svn15878.1.5-33.fc25.1.noarch
texlive-babel-english-svn30264.3.3p-33.fc25.1.noarch
texlive-cmextra-svn32831.0-33.fc25.1.noarch
texlive-fancyhdr-svn15878.3.1-33.fc25.1.noarch
texlive-luatex-svn40963-33.fc25.1.noarch
texlive-knuth-local-svn38627-33.fc25.1.noarch
texlive-mflogo-font-svn36898.1.002-33.fc25.1.noarch
texlive-parskip-svn19963.2.0-33.fc25.1.noarch
texlive-section-svn20180.0-33.fc25.1.noarch
texlive-textcase-svn15878.0-33.fc25.1.noarch
texlive-updmap-map-svn41159-33.fc25.1.noarch
texlive-attachfile-svn38830-33.fc25.1.noarch
libglvnd-glx-0.2.999-14.20170308git8e6e102.fc25.s390x
libglvnd-core-devel-0.2.999-14.20170308git8e6e102.fc25.s390x
python-magic-5.29-4.fc25.noarch
glibc-common-2.24-9.fc25.s390x
sqlite-libs-3.14.2-2.fc25.s390x
libtiff-4.0.8-1.fc25.s390x
libdb-5.3.28-24.fc25.s390x
glusterfs-client-xlators-3.10.4-1.fc25.s390x
nss-util-devel-3.30.2-1.0.fc25.s390x
gnutls-3.5.14-1.fc25.s390x
bind-license-9.10.5-2.P2.fc25.noarch
mesa-libGLES-17.0.5-3.fc25.s390x
python3-requests-kerberos-0.10.0-2.fc25.noarch
python3-pyOpenSSL-16.2.0-1.fc25.noarch
perl-threads-2.16-1.fc25.s390x
cryptsetup-libs-1.7.5-1.fc25.s390x
vim-minimal-8.0.705-1.fc25.s390x
netpbm-10.79.00-1.fc25.s390x
qrencode-libs-3.4.4-1.fc25.s390x
mariadb-config-10.1.24-3.fc25.s390x
gstreamer1-plugins-base-1.10.5-1.fc25.s390x
elfutils-default-yama-scope-0.169-1.fc25.noarch
glusterfs-3.10.4-1.fc25.s390x
systemd-udev-231-17.fc25.s390x
python2-koji-1.13.0-2.fc25.noarch
unbound-libs-1.6.3-1.fc25.s390x
openldap-2.4.44-11.fc25.s390x
koji-1.13.0-2.fc25.noarch
bind99-libs-9.9.10-2.P3.fc25.s390x
libcurl-devel-7.51.0-7.fc25.s390x
mesa-libGL-devel-17.0.5-3.fc25.s390x
python2-sssdconfig-1.15.2-5.fc25.noarch
webkitgtk4-plugin-process-gtk2-2.16.5-1.fc25.s390x
graphite2-devel-1.3.10-1.fc25.s390x
systemtap-sdt-devel-3.1-5.fc25.s390x
iproute-tc-4.11.0-1.fc25.s390x
libarchive-3.2.2-2.fc25.s390x
publicsuffix-list-dafsa-20170424-1.fc25.noarch
texlive-luaotfload-svn40902-33.fc25.1.noarch
texlive-unicode-math-svn38462-33.fc25.1.noarch
texlive-fancyvrb-svn18492.2.8-33.fc25.1.noarch
texlive-pst-pdf-bin-svn7838.0-33.20160520.fc25.1.noarch
texlive-amscls-svn36804.0-33.fc25.1.noarch
texlive-ltxmisc-svn21927.0-33.fc25.1.noarch
texlive-breqn-svn38099.0.98d-33.fc25.1.noarch
texlive-xetex-def-svn40327-33.fc25.1.noarch
openssh-server-7.4p1-4.fc25.s390x
sendmail-8.15.2-8.fc25.s390x
tzdata-2017b-1.fc25.noarch
hunspell-1.4.1-2.fc25.s390x
gpg-pubkey-8e1431d5-53bcbac7
zlib-1.2.8-10.fc24.s390x
sed-4.2.2-15.fc24.s390x
psmisc-22.21-8.fc24.s390x
gpm-libs-1.20.7-9.fc24.s390x
zip-3.0-16.fc24.s390x
libyubikey-1.13-2.fc24.s390x
sg3_utils-libs-1.41-3.fc24.s390x
polkit-pkla-compat-0.1-7.fc24.s390x
passwd-0.79-8.fc24.s390x
trousers-0.3.13-6.fc24.s390x
grubby-8.40-3.fc24.s390x
rootfiles-8.1-19.fc24.noarch
nettle-3.3-1.fc25.s390x
libksba-1.3.5-1.fc25.s390x
perl-Text-ParseWords-3.30-365.fc25.noarch
perl-PathTools-3.63-366.fc25.s390x
perl-File-Temp-0.23.04-365.fc25.noarch
fuse-libs-2.9.7-1.fc25.s390x
perl-Pod-Escapes-1.07-365.fc25.noarch
perl-Term-ANSIColor-4.05-2.fc25.noarch
perl-URI-1.71-5.fc25.noarch
libXfont-1.5.2-1.fc25.s390x
python-six-1.10.0-3.fc25.noarch
dbus-glib-0.108-1.fc25.s390x
gobject-introspection-1.50.0-1.fc25.s390x
libpwquality-1.3.0-6.fc25.s390x
python-gobject-base-3.22.0-1.fc25.s390x
python-html5lib-0.999-9.fc25.noarch
python3-dbus-1.2.4-2.fc25.s390x
python3-chardet-2.3.0-1.fc25.noarch
python3-urllib3-1.15.1-3.fc25.noarch
python-offtrac-0.1.0-7.fc25.noarch
python2-cryptography-1.5.3-3.fc25.s390x
python2-requests-kerberos-0.10.0-2.fc25.noarch
libserf-1.3.9-1.fc25.s390x
libdatrie-0.2.9-3.fc25.s390x
s390utils-base-1.36.0-1.fc25.s390x
kpartx-0.4.9-83.fc25.s390x
s390utils-cpuplugd-1.36.0-1.fc25.s390x
s390utils-osasnmpd-1.36.0-1.fc25.s390x
python-dnf-plugins-extras-common-0.0.12-4.fc25.noarch
fpc-srpm-macros-1.0-1.fc25.noarch
libuser-0.62-4.fc25.s390x
man-db-2.7.5-3.fc25.s390x
python-systemd-doc-232-1.fc25.s390x
lz4-1.7.5-1.fc25.s390x
tar-1.29-3.fc25.s390x
bodhi-client-0.9.12.2-6.fc25.noarch
cairo-1.14.8-1.fc25.s390x
gcc-go-6.3.1-1.fc25.s390x
cracklib-dicts-2.9.6-4.fc25.s390x
libselinux-python3-2.5-13.fc25.s390x
python2-enchant-1.6.8-1.fc25.noarch
boost-iostreams-1.60.0-10.fc25.s390x
userspace-rcu-0.9.2-2.fc25.s390x
libXext-devel-1.3.3-4.fc24.s390x
libXrandr-devel-1.5.1-1.fc25.s390x
perl-XML-XPath-1.39-1.fc25.noarch
python3-lxml-3.7.2-1.fc25.s390x
libiscsi-1.15.0-2.fc24.s390x
fontconfig-devel-2.12.1-1.fc25.s390x
libfdt-devel-1.4.2-1.fc25.s390x
ceph-devel-compat-10.2.4-2.fc25.s390x
zlib-static-1.2.8-10.fc24.s390x
chrpath-0.16-3.fc24.s390x
info-6.1-4.fc25.s390x
iptables-libs-1.6.0-3.fc25.s390x
libfdisk-2.28.2-2.fc25.s390x
dnf-plugins-core-0.1.21-5.fc25.noarch
perl-Storable-2.56-368.fc25.s390x
python2-decorator-4.0.11-1.fc25.noarch
libnetfilter_conntrack-1.0.6-2.fc25.s390x
texlive-texlive.infra-bin-svn40312-33.20160520.fc25.1.s390x
texlive-ifluatex-svn41346-33.fc25.1.noarch
texlive-fp-svn15878.0-33.fc25.1.noarch
texlive-latex-fonts-svn28888.0-33.fc25.1.noarch
texlive-bibtex-bin-svn40473-33.20160520.fc25.1.s390x
texlive-glyphlist-svn28576.0-33.fc25.1.noarch
texlive-marvosym-svn29349.2.2a-33.fc25.1.noarch
texlive-tex-bin-svn40987-33.20160520.fc25.1.s390x
texlive-texconfig-svn40768-33.fc25.1.noarch
texlive-wasy2-ps-svn35830.0-33.fc25.1.noarch
texlive-psfrag-svn15878.3.04-33.fc25.1.noarch
texlive-charter-svn15878.0-33.fc25.1.noarch
texlive-ec-svn25033.1.0-33.fc25.1.noarch
texlive-lineno-svn21442.4.41-33.fc25.1.noarch
texlive-hyphen-base-svn41138-33.fc25.1.noarch
texlive-manfnt-font-svn35799.0-33.fc25.1.noarch
texlive-ncntrsbk-svn31835.0-33.fc25.1.noarch
texlive-pst-math-svn34786.0.63-33.fc25.1.noarch
texlive-symbol-svn31835.0-33.fc25.1.noarch
texlive-environ-svn33821.0.3-33.fc25.1.noarch
texlive-algorithms-svn38085.0.1-33.fc25.1.noarch
kernel-core-4.10.8-200.fc25.s390x
libglvnd-devel-0.2.999-14.20170308git8e6e102.fc25.s390x
python3-hawkey-0.6.4-3.fc25.s390x
glibc-all-langpacks-2.24-9.fc25.s390x
freetype-2.6.5-9.fc25.s390x
mesa-libwayland-egl-17.0.5-3.fc25.s390x
libicu-57.1-5.fc25.s390x
nspr-devel-4.14.0-2.fc25.s390x
libnl3-cli-3.2.29-3.fc25.s390x
cups-libs-2.2.0-9.fc25.s390x
bind-libs-lite-9.10.5-2.P2.fc25.s390x
ca-certificates-2017.2.14-1.0.fc25.noarch
python3-kerberos-1.2.5-1.fc25.s390x
python3-cryptography-1.5.3-3.fc25.s390x
perl-IO-1.36-387.fc25.s390x
dhcp-libs-4.3.5-3.fc25.s390x
rsync-3.1.2-4.fc25.s390x
make-4.1-6.fc25.s390x
pcre2-devel-10.23-8.fc25.s390x
quota-4.03-8.fc25.s390x
libX11-devel-1.6.5-1.fc25.s390x
ghostscript-9.20-9.fc25.s390x
dbus-1.11.14-1.fc25.s390x
rpcbind-0.2.4-6.rc2.fc25.s390x
pyOpenSSL-16.2.0-1.fc25.noarch
glusterfs-devel-3.10.4-1.fc25.s390x
nss-tools-3.30.2-1.1.fc25.s390x
python3-pycurl-7.43.0-6.fc25.s390x
bind99-license-9.9.10-2.P3.fc25.noarch
curl-7.51.0-7.fc25.s390x
glusterfs-api-devel-3.10.4-1.fc25.s390x
python-firewall-0.4.4.5-1.fc25.noarch
netpbm-progs-10.79.00-1.fc25.s390x
libsndfile-1.0.28-3.fc25.s390x
python3-sssdconfig-1.15.2-5.fc25.noarch
wget-1.18-3.fc25.s390x
libsemanage-2.5-9.fc25.s390x
telnet-0.17-68.fc25.s390x
texlive-ifplatform-svn21156.0.4-33.fc25.1.noarch
texlive-eso-pic-svn37925.2.0g-33.fc25.1.noarch
texlive-xcolor-svn41044-33.fc25.1.noarch
texlive-pst-eps-svn15878.1.0-33.fc25.1.noarch
texlive-pst-text-svn15878.1.00-33.fc25.1.noarch
texlive-rotating-svn16832.2.16b-33.fc25.1.noarch
texlive-pdfpages-svn40638-33.fc25.1.noarch
texlive-cm-super-svn15878.0-33.fc25.1.noarch
texlive-xetex-svn41438-33.fc25.1.noarch
dnf-yum-1.1.10-6.fc25.noarch
libseccomp-devel-2.3.2-1.fc25.s390x
gpgme-1.8.0-10.fc25.s390x
apr-util-1.5.4-3.fc24.s390x
jbigkit-libs-2.1-5.fc24.s390x
pixman-0.34.0-2.fc24.s390x
dwz-0.12-2.fc24.s390x
expect-5.45-22.fc24.s390x
libsigsegv-2.10-10.fc24.s390x
fakeroot-libs-1.20.2-4.fc24.s390x
m17n-lib-1.7.0-5.fc24.s390x
libverto-0.2.6-6.fc24.s390x
libXmu-1.1.2-4.fc24.s390x
libXcursor-1.1.14-6.fc24.s390x
python-kitchen-1.2.4-2.fc24.noarch
fakeroot-1.20.2-4.fc24.s390x
blktrace-1.1.0-3.fc24.s390x
usermode-1.111-8.fc24.s390x
kbd-2.0.3-3.fc24.s390x
libaio-devel-0.3.110-6.fc24.s390x
web-assets-filesystem-5-4.fc24.noarch
libgpg-error-1.24-1.fc25.s390x
findutils-4.6.0-8.fc25.s390x
libassuan-2.4.3-1.fc25.s390x
libusbx-1.0.21-1.fc25.s390x
libxslt-1.1.28-13.fc25.s390x
libmetalink-0.1.3-1.fc25.s390x
perl-MIME-Base64-3.15-365.fc25.s390x
ncurses-6.0-6.20160709.fc25.s390x
libwayland-server-1.12.0-1.fc25.s390x
perl-Fedora-VSP-0.001-4.fc25.noarch
perl-libintl-perl-1.26-1.fc25.s390x
shadow-utils-4.2.1-11.fc25.s390x
atk-2.22.0-1.fc25.s390x
pam-1.3.0-1.fc25.s390x
harfbuzz-icu-1.3.2-1.fc25.s390x
libsecret-0.18.5-2.fc25.s390x
s390utils-iucvterm-1.36.0-1.fc25.s390x
python3-requests-2.10.0-4.fc25.noarch
pyusb-1.0.0-2.fc25.noarch
python-enum34-1.0.4-6.fc25.noarch
pyxattr-0.5.3-8.fc25.s390x
libbabeltrace-1.4.0-3.fc25.s390x
libthai-0.1.25-1.fc25.s390x
deltarpm-3.6-17.fc25.s390x
s390utils-mon_statd-1.36.0-1.fc25.s390x
device-mapper-multipath-0.4.9-83.fc25.s390x
python3-pygpgme-0.3-18.fc25.s390x
libreport-filesystem-2.8.0-1.fc25.s390x
ghc-srpm-macros-1.4.2-4.fc25.noarch
rpmdevtools-8.9-1.fc25.noarch
python-dnf-plugins-extras-migrate-0.0.12-4.fc25.noarch
perl-IO-Socket-SSL-2.038-1.fc25.noarch
perl-File-ShareDir-1.102-7.fc25.noarch
tcl-8.6.6-1.fc25.s390x
bzip2-1.0.6-21.fc25.s390x
libss-1.43.3-1.fc25.s390x
libselinux-utils-2.5-13.fc25.s390x
python3-enchant-1.6.8-1.fc25.noarch
python2-dockerfile-parse-0.0.5-7.fc25.noarch
systemd-bootchart-231-2.fc25.s390x
gcc-objc-6.3.1-1.fc25.s390x
e2fsprogs-1.43.3-1.fc25.s390x
libstdc++-static-6.3.1-1.fc25.s390x
libpng-devel-1.6.27-1.fc25.s390x
perl-XML-Parser-2.44-5.fc25.s390x
lttng-ust-2.8.1-2.fc25.s390x
libXfixes-devel-5.0.3-1.fc25.s390x
libXcomposite-devel-0.4.4-8.fc24.s390x
python3-javapackages-4.7.0-6.1.fc25.noarch
libcephfs_jni-devel-10.2.4-2.fc25.s390x
keyutils-libs-devel-1.5.9-8.fc24.s390x
harfbuzz-devel-1.3.2-1.fc25.s390x
libidn-devel-1.33-1.fc25.s390x
libnfs-1.9.8-2.fc24.s390x
libssh2-devel-1.8.0-1.fc25.s390x
qemu-sanity-check-nodeps-1.1.5-5.fc24.s390x
alsa-lib-devel-1.1.1-2.fc25.s390x
libpsl-0.17.0-1.fc25.s390x
libseccomp-2.3.2-1.fc25.s390x
copy-jdk-configs-2.2-2.fc25.noarch
json-glib-1.2.6-1.fc25.s390x
python2-dnf-1.1.10-6.fc25.noarch
python2-GitPython-2.1.3-1.fc25.noarch
texlive-tetex-bin-svn36770.0-33.20160520.fc25.1.noarch
texlive-amsfonts-svn29208.3.04-33.fc25.1.noarch
texlive-babel-svn40706-33.fc25.1.noarch
texlive-colortbl-svn29803.v1.0a-33.fc25.1.noarch
texlive-babelbib-svn25245.1.31-33.fc25.1.noarch
texlive-footmisc-svn23330.5.5b-33.fc25.1.noarch
texlive-makeindex-svn40768-33.fc25.1.noarch
texlive-plain-svn40274-33.fc25.1.noarch
texlive-texconfig-bin-svn29741.0-33.20160520.fc25.1.noarch
texlive-zapfding-svn31835.0-33.fc25.1.noarch
texlive-microtype-svn41127-33.fc25.1.noarch
texlive-bookman-svn31835.0-33.fc25.1.noarch
texlive-dvisvgm-def-svn41011-33.fc25.1.noarch
texlive-finstrut-svn21719.0.5-33.fc25.1.noarch
texlive-hyph-utf8-svn41189-33.fc25.1.noarch
texlive-lualibs-svn40370-33.fc25.1.noarch
kernel-modules-4.10.8-200.fc25.s390x
python2-hawkey-0.6.4-3.fc25.s390x
glibc-2.24-9.fc25.s390x
elfutils-libelf-0.169-1.fc25.s390x
libsoup-2.56.0-3.fc25.s390x
libnl3-3.2.29-3.fc25.s390x
gstreamer1-1.10.5-1.fc25.s390x
polkit-libs-0.113-8.fc25.s390x
libtirpc-1.0.2-0.fc25.s390x
emacs-common-25.2-2.fc25.s390x
libteam-1.27-1.fc25.s390x
python3-3.5.3-6.fc25.s390x
python3-pyasn1-0.2.3-1.fc25.noarch
perl-File-Path-2.12-366.fc25.noarch
mesa-libwayland-egl-devel-17.0.5-3.fc25.s390x
libacl-devel-2.2.52-13.fc25.s390x
lua-libs-5.3.4-3.fc25.s390x
pcre2-utf32-10.23-8.fc25.s390x
quota-nls-4.03-8.fc25.noarch
gtk3-3.22.16-1.fc25.s390x
ghostscript-x11-9.20-9.fc25.s390x
systemd-231-17.fc25.s390x
glusterfs-api-3.10.4-1.fc25.s390x
glusterfs-extra-xlators-3.10.4-1.fc25.s390x
glusterfs-server-3.10.4-1.fc25.s390x
java-1.8.0-openjdk-headless-1.8.0.131-1.b12.fc25.s390x
git-2.9.4-1.fc25.s390x
dhcp-common-4.3.5-3.fc25.noarch
python2-rpkg-1.49-5.fc25.noarch
vte291-devel-0.46.2-1.fc25.s390x
python-devel-2.7.13-2.fc25.s390x
elfutils-0.169-1.fc25.s390x
lua-5.3.4-3.fc25.s390x
python3-beautifulsoup4-4.6.0-1.fc25.noarch
libmicrohttpd-0.9.55-1.fc25.s390x
screen-4.6.1-1.fc25.s390x
strace-4.18-1.fc25.s390x
texlive-mparhack-svn15878.1.4-33.fc25.1.noarch
texlive-pspicture-svn15878.0-33.fc25.1.noarch
texlive-soul-svn15878.2.4-33.fc25.1.noarch
texlive-trimspaces-svn15878.1.1-33.fc25.1.noarch
texlive-varwidth-svn24104.0.92-33.fc25.1.noarch
texlive-geometry-svn19716.5.6-33.fc25.1.noarch
texlive-memoir-svn41203-33.fc25.1.noarch
texlive-pgf-svn40966-33.fc25.1.noarch
texlive-pst-coil-svn37377.1.07-33.fc25.1.noarch
texlive-pst-plot-svn41242-33.fc25.1.noarch
texlive-latex-bin-svn41438-33.fc25.1.noarch
texlive-ucs-svn35853.2.2-33.fc25.1.noarch
texlive-ae-svn15878.1.4-33.fc25.1.noarch
texlive-xetex-bin-svn41091-33.20160520.fc25.1.s390x
fedora-upgrade-26.1-1.fc25.noarch
fedpkg-1.28-1.fc25.noarch
perl-Thread-Queue-3.12-1.fc25.noarch
cdparanoia-libs-10.2-21.fc24.s390x
ustr-1.0.4-21.fc24.s390x
libusb-0.1.5-7.fc24.s390x
readline-devel-6.3-8.fc24.s390x
chkconfig-1.8-1.fc25.s390x
avahi-libs-0.6.32-4.fc25.s390x
perl-Unicode-Normalize-1.25-365.fc25.s390x
perl-libnet-3.10-1.fc25.noarch
perl-podlators-4.09-1.fc25.noarch
dbus-python-1.2.4-2.fc25.s390x
libgnome-keyring-3.12.0-7.fc25.s390x
python-backports-1.0-8.fc25.s390x
python-pycparser-2.14-7.fc25.noarch
plymouth-scripts-0.9.3-0.6.20160620git0e65b86c.fc25.s390x
cronie-1.5.1-2.fc25.s390x
python2-librepo-1.7.18-3.fc25.s390x
libXv-1.0.11-1.fc25.s390x
python2-ndg_httpsclient-0.4.0-4.fc25.noarch
btrfs-progs-4.6.1-1.fc25.s390x
libgcc-6.3.1-1.fc25.s390x
libgomp-6.3.1-1.fc25.s390x
perl-Encode-2.88-5.fc25.s390x
cracklib-2.9.6-4.fc25.s390x
libobjc-6.3.1-1.fc25.s390x
gcc-6.3.1-1.fc25.s390x
python3-dnf-plugin-system-upgrade-0.7.1-4.fc25.noarch
boost-random-1.60.0-10.fc25.s390x
libref_array-0.1.5-29.fc25.s390x
libXrender-devel-0.9.10-1.fc25.s390x
javapackages-tools-4.7.0-6.1.fc25.noarch
keyutils-1.5.9-8.fc24.s390x
libcom_err-devel-1.43.3-1.fc25.s390x
lzo-minilzo-2.08-8.fc24.s390x
libusbx-devel-1.0.21-1.fc25.s390x
virglrenderer-devel-0.5.0-1.20160411git61846f92f.fc25.s390x
acpica-tools-20160831-1.fc25.s390x
grep-2.27-2.fc25.s390x
dnf-conf-1.1.10-6.fc25.noarch
crypto-policies-20160921-4.gitf3018dd.fc25.noarch
rpm-build-libs-4.13.0.1-1.fc25.s390x
libnfsidmap-0.27-1.fc25.s390x
SDL2-2.0.5-3.fc25.s390x
texlive-etex-pkg-svn39355-33.fc25.1.noarch
texlive-multido-svn18302.1.42-33.fc25.1.noarch
texlive-gsftopk-svn40768-33.fc25.1.noarch
texlive-pst-ovl-svn40873-33.fc25.1.noarch
texlive-ltabptch-svn17533.1.74d-33.fc25.1.noarch
texlive-cite-svn36428.5.5-33.fc25.1.noarch
texlive-fpl-svn15878.1.002-33.fc25.1.noarch
texlive-mathpazo-svn15878.1.003-33.fc25.1.noarch
texlive-rcs-svn15878.0-33.fc25.1.noarch
texlive-type1cm-svn21820.0-33.fc25.1.noarch
texlive-l3kernel-svn41246-33.fc25.1.noarch
texlive-hyperref-svn41396-33.fc25.1.noarch
texlive-pst-tree-svn24142.1.12-33.fc25.1.noarch
texlive-sansmathaccent-svn30187.0-33.fc25.1.noarch
texlive-dvipdfmx-bin-svn40273-33.20160520.fc25.1.s390x
texlive-zapfchan-svn31835.0-33.fc25.1.noarch
glib2-static-2.50.3-1.fc25.s390x
bash-completion-2.5-1.fc25.noarch
glusterfs-libs-3.10.4-1.fc25.s390x
p11-kit-0.23.2-4.fc25.s390x
hyphen-2.8.8-4.fc24.s390x
gnutls-c++-3.5.14-1.fc25.s390x
python3-idna-2.5-1.fc25.noarch
nss-softokn-devel-3.30.2-1.0.fc25.s390x
less-481-7.fc25.s390x
rpmconf-base-1.0.19-1.fc25.noarch
gtk2-2.24.31-2.fc25.s390x
mesa-libgbm-17.0.5-3.fc25.s390x
nfs-utils-2.1.1-5.rc4.fc25.s390x
perl-Git-2.9.4-1.fc25.noarch
mock-1.4.2-1.fc25.noarch
mc-4.8.19-5.fc25.s390x
pcre-static-8.41-1.fc25.s390x
bind-libs-9.10.5-2.P2.fc25.s390x
libproxy-0.4.15-2.fc25.s390x
gpg-pubkey-a29cb19c-53bcbba6
m4-1.4.17-9.fc24.s390x
liblockfile-1.09-4.fc24.s390x
sg3_utils-1.41-3.fc24.s390x
libXinerama-1.1.3-6.fc24.s390x
libXft-2.3.2-4.fc24.s390x
tcp_wrappers-libs-7.6-83.fc25.s390x
perl-Text-Tabs+Wrap-2013.0523-365.fc25.noarch
perl-Error-0.17024-7.fc25.noarch
perl-Term-Cap-1.17-365.fc25.noarch
perl-Pod-Usage-1.69-1.fc25.noarch
device-mapper-persistent-data-0.6.3-1.fc25.s390x
net-snmp-libs-5.7.3-13.fc25.s390x
python3-six-1.10.0-3.fc25.noarch
python3-pysocks-1.5.6-5.fc25.noarch
python-chardet-2.3.0-1.fc25.noarch
python2-cffi-1.7.0-2.fc25.s390x
gc-devel-7.4.4-1.fc25.s390x
plymouth-0.9.3-0.6.20160620git0e65b86c.fc25.s390x
ebtables-2.0.10-21.fc25.s390x
python3-librepo-1.7.18-3.fc25.s390x
net-snmp-5.7.3-13.fc25.s390x
at-spi2-atk-2.22.0-1.fc25.s390x
avahi-autoipd-0.6.32-4.fc25.s390x
libgo-6.3.1-1.fc25.s390x
cpp-6.3.1-1.fc25.s390x
pyparsing-2.1.10-1.fc25.noarch
python3-pyparsing-2.1.10-1.fc25.noarch
libcollection-0.7.0-29.fc25.s390x
libcephfs-devel-10.2.4-2.fc25.s390x
libXdamage-devel-1.1.4-8.fc24.s390x
libverto-devel-0.2.6-6.fc24.s390x
snappy-1.1.3-2.fc24.s390x
cairo-gobject-devel-1.14.8-1.fc25.s390x
cyrus-sasl-devel-2.1.26-26.2.fc24.s390x
libXi-1.7.9-1.fc25.s390x
texlive-base-2016-33.20160520.fc25.noarch
python3-rpm-4.13.0.1-1.fc25.s390x
texlive-booktabs-svn40846-33.fc25.1.noarch
texlive-lm-svn28119.2.004-33.fc25.1.noarch
texlive-gsftopk-bin-svn40473-33.20160520.fc25.1.s390x
texlive-tex-svn40793-33.fc25.1.noarch
texlive-fancyref-svn15878.0.9c-33.fc25.1.noarch
texlive-chngcntr-svn17157.1.0a-33.fc25.1.noarch
texlive-fix2col-svn38770-33.fc25.1.noarch
texlive-marginnote-svn41382-33.fc25.1.noarch
texlive-pxfonts-svn15878.0-33.fc25.1.noarch
texlive-txfonts-svn15878.0-33.fc25.1.noarch
texlive-l3packages-svn41246-33.fc25.1.noarch
texlive-oberdiek-svn41346-33.fc25.1.noarch
texlive-pst-tools-svn34067.0.05-33.fc25.1.noarch
texlive-tex-gyre-svn18651.2.004-33.fc25.1.noarch
texlive-dvipdfmx-svn41149-33.fc25.1.noarch
texlive-collection-fontsrecommended-svn35830.0-33.20160520.fc25.1.noarch
libcacard-devel-2.5.3-1.fc25.s390x
ykpers-1.18.0-2.fc25.s390x
python2-idna-2.5-1.fc25.noarch
file-libs-5.29-4.fc25.s390x
policycoreutils-2.5-20.fc25.s390x
libgcrypt-1.7.8-1.fc25.s390x
pcre-8.41-1.fc25.s390x
GeoIP-1.6.11-1.fc25.s390x
ghostscript-core-9.20-9.fc25.s390x
python3-cffi-1.7.0-2.fc25.s390x
nss-softokn-freebl-devel-3.30.2-1.0.fc25.s390x
json-c-0.12.1-2.fc25.s390x
vim-common-8.0.705-1.fc25.s390x
vte291-0.46.2-1.fc25.s390x
libdrm-devel-2.4.81-1.fc25.s390x
gssproxy-0.7.0-9.fc25.s390x
git-core-doc-2.9.4-1.fc25.s390x
systemtap-3.1-5.fc25.s390x
mesa-libgbm-devel-17.0.5-3.fc25.s390x
vim-enhanced-8.0.705-1.fc25.s390x
glibc-static-2.24-9.fc25.s390x
libgusb-0.2.10-1.fc25.s390x
python-async-0.6.1-9.fc22.s390x
dejavu-sans-mono-fonts-2.35-3.fc24.noarch
popt-1.16-7.fc24.s390x
cyrus-sasl-lib-2.1.26-26.2.fc24.s390x
xz-5.2.2-2.fc24.s390x
libpipeline-1.4.1-2.fc24.s390x
pinentry-0.9.7-2.fc24.s390x
pth-2.0.7-27.fc24.s390x
libsepol-2.5-10.fc25.s390x
libxcb-1.12-1.fc25.s390x
perl-Getopt-Long-2.49.1-1.fc25.noarch
avahi-glib-0.6.32-4.fc25.s390x
python3-pip-8.1.2-2.fc25.noarch
python3-libcomps-0.1.7-5.fc25.s390x
python-slip-0.6.4-4.fc25.noarch
python2-libcomps-0.1.7-5.fc25.s390x
gc-7.4.4-1.fc25.s390x
s390utils-cmsfs-1.36.0-1.fc25.s390x
newt-python-0.52.19-2.fc25.s390x
qt5-srpm-macros-5.7.1-1.fc25.noarch
device-mapper-event-1.02.136-3.fc25.s390x
perl-Class-Inspector-1.31-2.fc25.noarch
libbasicobjects-0.1.1-29.fc25.s390x
libradosstriper1-10.2.4-2.fc25.s390x
libXxf86vm-devel-1.1.4-3.fc24.s390x
zziplib-0.13.62-7.fc24.s390x
libpaper-1.1.24-12.fc24.s390x
libini_config-1.3.0-29.fc25.s390x
snappy-devel-1.1.3-2.fc24.s390x
libcap-ng-devel-0.7.8-1.fc25.s390x
libxkbcommon-devel-0.7.1-1.fc25.s390x
openssl-libs-1.0.2k-1.fc25.s390x
libkadm5-1.14.4-7.fc25.s390x
rpm-libs-4.13.0.1-1.fc25.s390x
util-linux-2.28.2-2.fc25.s390x
texlive-etoolbox-svn38031.2.2a-33.fc25.1.noarch
texlive-dvips-svn41149-33.fc25.1.noarch
texlive-latexconfig-svn40274-33.fc25.1.noarch
texlive-tex-ini-files-svn40533-33.fc25.1.noarch
texlive-qstest-svn15878.0-33.fc25.1.noarch
texlive-cmap-svn41168-33.fc25.1.noarch
texlive-luatex-bin-svn41091-33.20160520.fc25.1.s390x
texlive-mflogo-svn38628-33.fc25.1.noarch
texlive-sansmath-svn17997.1.1-33.fc25.1.noarch
texlive-unicode-data-svn39808-33.fc25.1.noarch
texlive-luaotfload-bin-svn34647.0-33.20160520.fc25.1.noarch
texlive-listings-svn37534.1.6-33.fc25.1.noarch
texlive-pstricks-svn41321-33.fc25.1.noarch
texlive-metalogo-svn18611.0.12-33.fc25.1.noarch
texlive-collection-latex-svn41011-33.20160520.fc25.1.noarch
kernel-4.10.5-200.fc25.s390x
python2-dnf-plugins-core-0.1.21-5.fc25.noarch
xkeyboard-config-2.20-2.fc25.noarch
file-5.29-4.fc25.s390x
perl-Test-Harness-3.39-1.fc25.noarch
systemd-libs-231-17.fc25.s390x
webkitgtk4-jsc-2.16.5-1.fc25.s390x
gtk-update-icon-cache-3.22.16-1.fc25.s390x
glibc-devel-2.24-9.fc25.s390x
python3-pycparser-2.14-7.fc25.noarch
kernel-devel-4.11.10-200.fc25.s390x
gsm-1.0.17-1.fc25.s390x
vim-filesystem-8.0.705-1.fc25.s390x
webkitgtk4-2.16.5-1.fc25.s390x
python-2.7.13-2.fc25.s390x
glusterfs-fuse-3.10.4-1.fc25.s390x
git-core-2.9.4-1.fc25.s390x
selinux-policy-targeted-3.13.1-225.18.fc25.noarch
kernel-4.11.10-200.fc25.s390x
rpmconf-1.0.19-1.fc25.noarch
teamd-1.27-1.fc25.s390x
jasper-libs-1.900.13-4.fc25.s390x
libattr-2.4.47-16.fc24.s390x
libvisual-0.4.0-20.fc24.s390x
libpcap-1.7.4-2.fc24.s390x
libutempter-1.1.6-8.fc24.s390x
libgudev-230-3.fc24.s390x
popt-devel-1.16-7.fc24.s390x
hicolor-icon-theme-0.15-3.fc24.noarch
setup-2.10.4-1.fc25.noarch
bash-4.3.43-4.fc25.s390x
libjpeg-turbo-1.5.1-0.fc25.s390x
perl-Socket-2.024-1.fc25.s390x
perl-HTTP-Tiny-0.070-1.fc25.noarch
ipset-6.29-1.fc25.s390x
python2-setuptools-25.1.1-1.fc25.noarch
gsettings-desktop-schemas-3.22.0-1.fc25.s390x
python3-setuptools-25.1.1-1.fc25.noarch
python-slip-dbus-0.6.4-4.fc25.noarch
python2-ply-3.8-2.fc25.noarch
dtc-1.4.2-1.fc25.s390x
guile-2.0.13-1.fc25.s390x
cronie-anacron-1.5.1-2.fc25.s390x
libXtst-1.2.3-1.fc25.s390x
iso-codes-3.70-1.fc25.noarch
s390utils-1.36.0-1.fc25.s390x
python-backports-ssl_match_hostname-3.5.0.1-3.fc25.noarch
fedora-cert-0.6.0.1-1.fc25.noarch
libstdc++-6.3.1-1.fc25.s390x
subversion-libs-1.9.5-1.fc25.s390x
libgfortran-6.3.1-1.fc25.s390x
dnf-plugin-system-upgrade-0.7.1-4.fc25.noarch
lvm2-2.02.167-3.fc25.s390x
libselinux-devel-2.5-13.fc25.s390x
perl-Time-Local-1.250-1.fc25.noarch
libradosstriper-devel-10.2.4-2.fc25.s390x
flac-libs-1.3.2-1.fc25.s390x
perl-Digest-1.17-366.fc25.noarch
teckit-2.5.1-15.fc24.s390x
libpath_utils-0.2.1-29.fc25.s390x
attr-2.4.47-16.fc24.s390x
usbredir-0.7.1-2.fc24.s390x
cairo-devel-1.14.8-1.fc25.s390x
lzo-devel-2.08-8.fc24.s390x
libcap-devel-2.25-2.fc25.s390x
krb5-devel-1.14.4-7.fc25.s390x
rpm-4.13.0.1-1.fc25.s390x
kernel-devel-4.10.5-200.fc25.s390x
libbsd-0.8.3-1.fc25.s390x
texlive-url-svn32528.3.4-33.fc25.1.noarch
texlive-dvips-bin-svn40987-33.20160520.fc25.1.s390x
texlive-index-svn24099.4.1beta-33.fc25.1.noarch
texlive-setspace-svn24881.6.7a-33.fc25.1.noarch
texlive-mathtools-svn38833-33.fc25.1.noarch
texlive-cm-svn32865.0-33.fc25.1.noarch
texlive-graphics-def-svn41879-33.fc25.1.noarch
texlive-mdwtools-svn15878.1.05.4-33.fc25.1.noarch
texlive-rsfs-svn15878.0-33.fc25.1.noarch
texlive-ucharcat-svn38907-33.fc25.1.noarch
texlive-fontspec-svn41262-33.fc25.1.noarch
texlive-showexpl-svn32737.v0.3l-33.fc25.1.noarch
texlive-pstricks-add-svn40744-33.fc25.1.noarch
texlive-beamer-svn36461.3.36-33.fc25.1.noarch
texlive-collection-basic-svn41149-33.20160520.fc25.1.noarch
rpm-build-4.13.0.1-1.fc25.s390x
xemacs-filesystem-21.5.34-20.20170124hgf412e9f093d4.fc25.noarch
hawkey-0.6.4-3.fc25.s390x
gdk-pixbuf2-modules-2.36.6-1.fc25.s390x
bluez-libs-5.44-1.fc25.s390x
audit-libs-2.7.7-1.fc25.s390x
iproute-4.11.0-1.fc25.s390x
libICE-1.0.9-9.fc25.s390x
glibc-headers-2.24-9.fc25.s390x
python3-ply-3.8-2.fc25.noarch
perl-5.24.2-387.fc25.s390x
graphite2-1.3.10-1.fc25.s390x
vte-profile-0.46.2-1.fc25.s390x
gtk3-devel-3.22.16-1.fc25.s390x
python-libs-2.7.13-2.fc25.s390x
mesa-libGL-17.0.5-3.fc25.s390x
python2-pycurl-7.43.0-6.fc25.s390x
NetworkManager-1.4.4-5.fc25.s390x
mesa-libEGL-devel-17.0.5-3.fc25.s390x
mariadb-libs-10.1.24-3.fc25.s390x
mesa-libGLES-devel-17.0.5-3.fc25.s390x
hostname-3.15-8.fc25.s390x
gpg-pubkey-a0a7badb-52844296
readline-6.3-8.fc24.s390x
cpio-2.12-3.fc24.s390x
libXcomposite-0.4.4-8.fc24.s390x
procps-ng-3.3.10-11.fc24.s390x
GConf2-3.2.6-16.fc24.s390x
xz-devel-5.2.2-2.fc24.s390x
fedora-logos-22.0.0-3.fc24.s390x
gpg-pubkey-e372e838-56fd7943
kmod-libs-23-1.fc25.s390x
perl-parent-0.236-1.fc25.noarch
perl-TermReadKey-2.37-1.fc25.s390x
ncurses-c++-libs-6.0-6.20160709.fc25.s390x
gzip-1.8-1.fc25.s390x
python3-gobject-base-3.22.0-1.fc25.s390x
python2-yubico-1.3.2-3.fc25.noarch
s390utils-ziomon-1.36.0-1.fc25.s390x
librepo-1.7.18-3.fc25.s390x
librsvg2-2.40.16-2.fc25.s390x
gnat-srpm-macros-4-1.fc25.noarch
python-decoratortools-1.8-12.fc25.noarch
m17n-db-1.7.0-7.fc25.noarch
e2fsprogs-libs-1.43.3-1.fc25.s390x
libvorbis-1.3.5-1.fc25.s390x
gcc-gdb-plugin-6.3.1-1.fc25.s390x
npth-1.3-1.fc25.s390x
libcephfs1-10.2.4-2.fc25.s390x
wayland-devel-1.12.0-1.fc25.s390x
libxcb-devel-1.12-1.fc25.s390x
perl-encoding-2.19-5.fc25.s390x
python3-cssselect-0.9.2-1.fc25.noarch
gettext-libs-0.19.8.1-3.fc25.s390x
at-spi2-atk-devel-2.22.0-1.fc25.s390x
virglrenderer-0.5.0-1.20160411git61846f92f.fc25.s390x
pixman-devel-0.34.0-2.fc24.s390x
libnfs-devel-1.9.8-2.fc24.s390x
libblkid-2.28.2-2.fc25.s390x
glib2-devel-2.50.3-1.fc25.s390x
texlive-ifxetex-svn19685.0.5-33.fc25.1.noarch
texlive-caption-svn41409-33.fc25.1.noarch
texlive-float-svn15878.1.3d-33.fc25.1.noarch
texlive-pdftex-def-svn22653.0.06d-33.fc25.1.noarch
texlive-xdvi-bin-svn40750-33.20160520.fc25.1.s390x
texlive-beton-svn15878.0-33.fc25.1.noarch
texlive-filecontents-svn24250.1.3-33.fc25.1.noarch
texlive-lm-math-svn36915.1.959-33.fc25.1.noarch
texlive-pslatex-svn16416.0-33.fc25.1.noarch
texlive-times-svn35058.0-33.fc25.1.noarch
texlive-breakurl-svn29901.1.40-33.fc25.1.noarch
texlive-filehook-svn24280.0.5d-33.fc25.1.noarch
texlive-pst-pdf-svn31660.1.1v-33.fc25.1.noarch
texlive-seminar-svn34011.1.62-33.fc25.1.noarch
texlive-xetexconfig-svn41133-33.fc25.1.noarch
python-rpm-macros-3-12.fc25.noarch
rpm-devel-4.13.0.1-1.fc25.s390x
nss-pem-1.0.3-3.fc25.s390x
at-spi2-core-2.22.1-1.fc25.s390x
GeoIP-GeoLite-data-2017.04-1.fc25.noarch
kernel-devel-4.10.8-200.fc25.s390x
dbus-libs-1.11.14-1.fc25.s390x
perl-Scalar-List-Utils-1.48-1.fc25.s390x
libidn2-2.0.2-1.fc25.s390x
libtasn1-devel-4.12-1.fc25.s390x
python3-koji-1.13.0-2.fc25.noarch
glusterfs-cli-3.10.4-1.fc25.s390x
opus-1.1.5-1.fc25.s390x
mariadb-common-10.1.24-3.fc25.s390x
elfutils-libs-0.169-1.fc25.s390x
kernel-core-4.11.10-200.fc25.s390x
gnutls-dane-3.5.14-1.fc25.s390x
systemd-container-231-17.fc25.s390x
sudo-1.8.20p2-1.fc25.s390x
dbus-devel-1.11.14-1.fc25.s390x
perl-Module-CoreList-5.20170621-1.fc25.noarch
libicu-devel-57.1-5.fc25.s390x
js-jquery-2.2.4-3.fc25.noarch
=== TEST BEGIN ===
Using CC: /home/fam/bin/cc
Install prefix    /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install
BIOS directory    /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/share/qemu
binary directory  /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/bin
library directory /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/lib
module directory  /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/lib/qemu
libexec directory /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/libexec
include directory /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/include
config directory  /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/etc
local state directory   /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/var
Manual directory  /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path       /var/tmp/patchew-tester-tmp-hm2hdbnw/src
C compiler        /home/fam/bin/cc
Host C compiler   cc
C++ compiler      c++
Objective-C compiler /home/fam/bin/cc
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/include/pixman-1  -Werror -DHAS_LIBSSH2_SFTP_FSYNC -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DNCURSES_WIDECHAR -D_GNU_SOURCE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1    -I/usr/include/libpng16 -I/usr/include/libdrm  -I/usr/include/cacard -I/usr/include/nss3 -I/usr/include/nspr4 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/libusb-1.0 
LDFLAGS           -Wl,--warn-common -m64 -g 
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          s390x
host big endian   yes
target list       aarch64-softmmu alpha-softmmu arm-softmmu cris-softmmu i386-softmmu lm32-softmmu m68k-softmmu microblazeel-softmmu microblaze-softmmu mips64el-softmmu mips64-softmmu mipsel-softmmu mips-softmmu moxie-softmmu nios2-softmmu or1k-softmmu ppc64-softmmu ppcemb-softmmu ppc-softmmu s390x-softmmu sh4eb-softmmu sh4-softmmu sparc64-softmmu sparc-softmmu tricore-softmmu unicore32-softmmu x86_64-softmmu xtensaeb-softmmu xtensa-softmmu aarch64-linux-user alpha-linux-user armeb-linux-user arm-linux-user cris-linux-user hppa-linux-user i386-linux-user m68k-linux-user microblazeel-linux-user microblaze-linux-user mips64el-linux-user mips64-linux-user mipsel-linux-user mips-linux-user mipsn32el-linux-user mipsn32-linux-user nios2-linux-user or1k-linux-user ppc64abi32-linux-user ppc64le-linux-user ppc64-linux-user ppc-linux-user s390x-linux-user sh4eb-linux-user sh4-linux-user sparc32plus-linux-user sparc64-linux-user sparc-linux-user tilegx-linux-user x86_64-linux-user
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
pixman            system
SDL support       yes (2.0.5)
GTK support       yes (3.22.16)
GTK GL support    yes
VTE support       yes (0.46.2)
TLS priority      NORMAL
GNUTLS support    yes
GNUTLS rnd        yes
libgcrypt         no
libgcrypt kdf     no
nettle            yes (3.3)
nettle kdf        yes
libtasn1          yes
curses support    yes
virgl support     yes
curl support      yes
mingw32 support   no
Audio drivers     oss
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    yes
VNC support       yes
VNC SASL support  yes
VNC JPEG support  yes
VNC PNG support   yes
xen support       no
brlapi support    yes
bluez  support    yes
Documentation     yes
PIE               no
vde support       no
netmap support    no
Linux AIO support yes
ATTR/XATTR support yes
Install blobs     yes
KVM support       yes
HAX support       no
TCG support       yes
TCG debug enabled no
TCG interpreter   no
RDMA support      no
fdt support       yes
preadv support    yes
fdatasync         yes
madvise           yes
posix_madvise     yes
libcap-ng support yes
vhost-net support yes
vhost-scsi support yes
vhost-vsock support yes
vhost-user support yes
Trace backends    log
spice support     no 
rbd support       yes
xfsctl support    no
smartcard support yes
libusb            yes
usb net redir     yes
OpenGL support    yes
OpenGL dmabufs    yes
libiscsi support  yes
libnfs support    yes
build guest agent yes
QGA VSS support   no
QGA w32 disk info no
QGA MSI support   no
seccomp support   no
coroutine backend ucontext
coroutine pool    yes
debug stack usage no
crypto afalg      no
GlusterFS support yes
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   yes
TPM passthrough   no
QOM debugging     yes
Live block migration yes
lzo support       yes
snappy support    yes
bzip2 support     yes
NUMA host support no
tcmalloc support  no
jemalloc support  no
avx2 optimization no
replication support yes
VxHS block device no
  GEN     aarch64-softmmu/config-devices.mak.tmp
  GEN     cris-softmmu/config-devices.mak.tmp
  GEN     alpha-softmmu/config-devices.mak.tmp
  GEN     arm-softmmu/config-devices.mak.tmp
  GEN     cris-softmmu/config-devices.mak
  GEN     i386-softmmu/config-devices.mak.tmp
  GEN     alpha-softmmu/config-devices.mak
  GEN     arm-softmmu/config-devices.mak
  GEN     lm32-softmmu/config-devices.mak.tmp
  GEN     m68k-softmmu/config-devices.mak.tmp
  GEN     aarch64-softmmu/config-devices.mak
  GEN     microblazeel-softmmu/config-devices.mak.tmp
  GEN     lm32-softmmu/config-devices.mak
  GEN     i386-softmmu/config-devices.mak
  GEN     m68k-softmmu/config-devices.mak
  GEN     microblaze-softmmu/config-devices.mak.tmp
  GEN     mips64el-softmmu/config-devices.mak.tmp
  GEN     mips64-softmmu/config-devices.mak.tmp
  GEN     microblazeel-softmmu/config-devices.mak
  GEN     microblaze-softmmu/config-devices.mak
  GEN     mipsel-softmmu/config-devices.mak.tmp
  GEN     mips-softmmu/config-devices.mak.tmp
  GEN     mips64-softmmu/config-devices.mak
  GEN     mips64el-softmmu/config-devices.mak
  GEN     nios2-softmmu/config-devices.mak.tmp
  GEN     moxie-softmmu/config-devices.mak.tmp
  GEN     mipsel-softmmu/config-devices.mak
  GEN     moxie-softmmu/config-devices.mak
  GEN     nios2-softmmu/config-devices.mak
  GEN     mips-softmmu/config-devices.mak
  GEN     ppc64-softmmu/config-devices.mak.tmp
  GEN     ppcemb-softmmu/config-devices.mak.tmp
  GEN     or1k-softmmu/config-devices.mak.tmp
  GEN     ppc-softmmu/config-devices.mak.tmp
  GEN     or1k-softmmu/config-devices.mak
  GEN     s390x-softmmu/config-devices.mak.tmp
  GEN     ppc64-softmmu/config-devices.mak
  GEN     ppcemb-softmmu/config-devices.mak
  GEN     sh4eb-softmmu/config-devices.mak.tmp
  GEN     ppc-softmmu/config-devices.mak
  GEN     sh4-softmmu/config-devices.mak.tmp
  GEN     sparc64-softmmu/config-devices.mak.tmp
  GEN     s390x-softmmu/config-devices.mak
  GEN     sparc-softmmu/config-devices.mak.tmp
  GEN     sh4eb-softmmu/config-devices.mak
  GEN     sh4-softmmu/config-devices.mak
  GEN     tricore-softmmu/config-devices.mak.tmp
  GEN     sparc64-softmmu/config-devices.mak
  GEN     unicore32-softmmu/config-devices.mak.tmp
  GEN     x86_64-softmmu/config-devices.mak.tmp
  GEN     sparc-softmmu/config-devices.mak
  GEN     xtensaeb-softmmu/config-devices.mak.tmp
  GEN     unicore32-softmmu/config-devices.mak
  GEN     tricore-softmmu/config-devices.mak
  GEN     xtensa-softmmu/config-devices.mak.tmp
  GEN     aarch64-linux-user/config-devices.mak.tmp
  GEN     xtensaeb-softmmu/config-devices.mak
  GEN     alpha-linux-user/config-devices.mak.tmp
  GEN     xtensa-softmmu/config-devices.mak
  GEN     x86_64-softmmu/config-devices.mak
  GEN     aarch64-linux-user/config-devices.mak
  GEN     armeb-linux-user/config-devices.mak.tmp
  GEN     alpha-linux-user/config-devices.mak
  GEN     arm-linux-user/config-devices.mak.tmp
  GEN     cris-linux-user/config-devices.mak.tmp
  GEN     hppa-linux-user/config-devices.mak.tmp
  GEN     armeb-linux-user/config-devices.mak
  GEN     arm-linux-user/config-devices.mak
  GEN     cris-linux-user/config-devices.mak
  GEN     i386-linux-user/config-devices.mak.tmp
  GEN     m68k-linux-user/config-devices.mak.tmp
  GEN     microblazeel-linux-user/config-devices.mak.tmp
  GEN     hppa-linux-user/config-devices.mak
  GEN     microblaze-linux-user/config-devices.mak.tmp
  GEN     i386-linux-user/config-devices.mak
  GEN     m68k-linux-user/config-devices.mak
  GEN     microblazeel-linux-user/config-devices.mak
  GEN     mips64el-linux-user/config-devices.mak.tmp
  GEN     mips64-linux-user/config-devices.mak.tmp
  GEN     mipsel-linux-user/config-devices.mak.tmp
  GEN     microblaze-linux-user/config-devices.mak
  GEN     mips64el-linux-user/config-devices.mak
  GEN     mips-linux-user/config-devices.mak.tmp
  GEN     mipsel-linux-user/config-devices.mak
  GEN     mipsn32el-linux-user/config-devices.mak.tmp
  GEN     mips64-linux-user/config-devices.mak
  GEN     mipsn32-linux-user/config-devices.mak.tmp
  GEN     mips-linux-user/config-devices.mak
  GEN     or1k-linux-user/config-devices.mak.tmp
  GEN     nios2-linux-user/config-devices.mak.tmp
  GEN     mipsn32el-linux-user/config-devices.mak
  GEN     mipsn32-linux-user/config-devices.mak
  GEN     ppc64le-linux-user/config-devices.mak.tmp
  GEN     ppc64abi32-linux-user/config-devices.mak.tmp
  GEN     nios2-linux-user/config-devices.mak
  GEN     or1k-linux-user/config-devices.mak
  GEN     ppc64-linux-user/config-devices.mak.tmp
  GEN     ppc-linux-user/config-devices.mak.tmp
  GEN     ppc64abi32-linux-user/config-devices.mak
  GEN     ppc64le-linux-user/config-devices.mak
  GEN     sh4eb-linux-user/config-devices.mak.tmp
  GEN     s390x-linux-user/config-devices.mak.tmp
  GEN     ppc64-linux-user/config-devices.mak
  GEN     ppc-linux-user/config-devices.mak
  GEN     sh4-linux-user/config-devices.mak.tmp
  GEN     sh4eb-linux-user/config-devices.mak
  GEN     sparc32plus-linux-user/config-devices.mak.tmp
  GEN     sparc64-linux-user/config-devices.mak.tmp
  GEN     s390x-linux-user/config-devices.mak
  GEN     sh4-linux-user/config-devices.mak
  GEN     sparc32plus-linux-user/config-devices.mak
  GEN     sparc-linux-user/config-devices.mak.tmp
  GEN     tilegx-linux-user/config-devices.mak.tmp
  GEN     x86_64-linux-user/config-devices.mak.tmp
  GEN     sparc64-linux-user/config-devices.mak
  GEN     sparc-linux-user/config-devices.mak
  GEN     config-host.h
  GEN     tilegx-linux-user/config-devices.mak
  GEN     qemu-options.def
  GEN     x86_64-linux-user/config-devices.mak
  GEN     qmp-commands.h
  GEN     qapi-types.h
  GEN     qapi-visit.h
  GEN     qapi-event.h
  GEN     qmp-marshal.c
  GEN     qapi-types.c
  GEN     qapi-visit.c
  GEN     qapi-event.c
  GEN     qmp-introspect.h
  GEN     qmp-introspect.c
  GEN     trace/generated-tcg-tracers.h
  GEN     trace/generated-helpers-wrappers.h
  GEN     trace/generated-helpers.c
  GEN     trace/generated-helpers.h
  GEN     module_block.h
  GEN     tests/test-qapi-types.h
  GEN     tests/test-qapi-visit.h
  GEN     tests/test-qmp-commands.h
  GEN     tests/test-qmp-introspect.h
  GEN     tests/test-qapi-event.h
  GEN     trace-root.h
  GEN     crypto/trace.h
  GEN     io/trace.h
  GEN     util/trace.h
  GEN     migration/trace.h
  GEN     block/trace.h
  GEN     chardev/trace.h
  GEN     hw/block/trace.h
  GEN     hw/block/dataplane/trace.h
  GEN     hw/char/trace.h
  GEN     hw/intc/trace.h
  GEN     hw/net/trace.h
  GEN     hw/virtio/trace.h
  GEN     hw/audio/trace.h
  GEN     hw/misc/trace.h
  GEN     hw/usb/trace.h
  GEN     hw/nvram/trace.h
  GEN     hw/scsi/trace.h
  GEN     hw/display/trace.h
  GEN     hw/input/trace.h
  GEN     hw/timer/trace.h
  GEN     hw/dma/trace.h
  GEN     hw/sparc/trace.h
  GEN     hw/isa/trace.h
  GEN     hw/sd/trace.h
  GEN     hw/mem/trace.h
  GEN     hw/i386/trace.h
  GEN     hw/i386/xen/trace.h
  GEN     hw/ppc/trace.h
  GEN     hw/pci/trace.h
  GEN     hw/9pfs/trace.h
  GEN     hw/s390x/trace.h
  GEN     hw/vfio/trace.h
  GEN     hw/acpi/trace.h
  GEN     hw/arm/trace.h
  GEN     hw/alpha/trace.h
  GEN     hw/xen/trace.h
  GEN     ui/trace.h
  GEN     audio/trace.h
  GEN     net/trace.h
  GEN     target/arm/trace.h
  GEN     target/i386/trace.h
  GEN     target/mips/trace.h
  GEN     target/s390x/trace.h
  GEN     target/ppc/trace.h
  GEN     target/sparc/trace.h
  GEN     qom/trace.h
  GEN     linux-user/trace.h
  GEN     qapi/trace.h
  GEN     accel/tcg/trace.h
  GEN     accel/kvm/trace.h
  GEN     trace-root.c
  GEN     nbd/trace.h
  GEN     util/trace.c
  GEN     crypto/trace.c
  GEN     io/trace.c
  GEN     migration/trace.c
  GEN     chardev/trace.c
  GEN     block/trace.c
  GEN     hw/block/trace.c
  GEN     hw/block/dataplane/trace.c
  GEN     hw/char/trace.c
  GEN     hw/intc/trace.c
  GEN     hw/net/trace.c
  GEN     hw/virtio/trace.c
  GEN     hw/audio/trace.c
  GEN     hw/misc/trace.c
  GEN     hw/usb/trace.c
  GEN     hw/scsi/trace.c
  GEN     hw/nvram/trace.c
  GEN     hw/display/trace.c
  GEN     hw/input/trace.c
  GEN     hw/dma/trace.c
  GEN     hw/timer/trace.c
  GEN     hw/sparc/trace.c
  GEN     hw/isa/trace.c
  GEN     hw/sd/trace.c
  GEN     hw/mem/trace.c
  GEN     hw/i386/trace.c
  GEN     hw/i386/xen/trace.c
  GEN     hw/9pfs/trace.c
  GEN     hw/ppc/trace.c
  GEN     hw/pci/trace.c
  GEN     hw/s390x/trace.c
  GEN     hw/vfio/trace.c
  GEN     hw/acpi/trace.c
  GEN     hw/arm/trace.c
  GEN     hw/alpha/trace.c
  GEN     ui/trace.c
  GEN     hw/xen/trace.c
  GEN     audio/trace.c
  GEN     net/trace.c
  GEN     target/arm/trace.c
  GEN     target/i386/trace.c
  GEN     target/mips/trace.c
  GEN     target/sparc/trace.c
  GEN     target/s390x/trace.c
  GEN     target/ppc/trace.c
  GEN     qom/trace.c
  GEN     linux-user/trace.c
  GEN     qapi/trace.c
  GEN     accel/tcg/trace.c
  GEN     accel/kvm/trace.c
  GEN     config-all-devices.mak
  GEN     nbd/trace.c
  GEN     docs/version.texi
  GEN     qemu-monitor.texi
  GEN     qemu-options.texi
  CC      tests/qemu-iotests/socket_scm_helper.o
  GEN     qemu-img-cmds.texi
  GEN     qemu-monitor-info.texi
  GEN     qemu-img.1
  GEN     qemu-nbd.8
  GEN     qemu-ga.8
  GEN     docs/interop/qemu-qmp-qapi.texi
  GEN     docs/interop/qemu-ga-qapi.texi
  GEN     fsdev/virtfs-proxy-helper.1
  GEN     qga/qapi-generated/qga-qapi-types.h
  GEN     qga/qapi-generated/qga-qapi-visit.h
  GEN     qga/qapi-generated/qga-qapi-types.c
  GEN     qga/qapi-generated/qga-qapi-visit.c
  GEN     qga/qapi-generated/qga-qmp-commands.h
  CC      qmp-introspect.o
  GEN     qga/qapi-generated/qga-qmp-marshal.c
  CC      qapi-types.o
  CC      qapi-event.o
  CC      qapi-visit.o
  CC      qapi/qapi-visit-core.o
  CC      qapi/qapi-dealloc-visitor.o
  CC      qapi/qobject-input-visitor.o
  CC      qapi/qobject-output-visitor.o
  CC      qapi/qmp-registry.o
  CC      qapi/qmp-dispatch.o
  CC      qapi/string-input-visitor.o
  CC      qapi/string-output-visitor.o
  CC      qapi/opts-visitor.o
  CC      qapi/qapi-clone-visitor.o
  CC      qapi/qmp-event.o
  CC      qapi/qapi-util.o
  CC      qobject/qnull.o
  CC      qobject/qnum.o
  CC      qobject/qstring.o
  CC      qobject/qdict.o
  CC      qobject/qlist.o
  CC      qobject/qbool.o
  CC      qobject/qjson.o
  CC      qobject/qobject.o
  CC      qobject/json-lexer.o
  CC      qobject/json-streamer.o
  CC      qobject/json-parser.o
  CC      trace/control.o
  CC      trace/qmp.o
  CC      util/osdep.o
  CC      util/cutils.o
  CC      util/unicode.o
  CC      util/qemu-timer-common.o
  CC      util/bufferiszero.o
  CC      util/lockcnt.o
  CC      util/aiocb.o
  CC      util/thread-pool.o
  CC      util/async.o
  CC      util/qemu-timer.o
  CC      util/iohandler.o
  CC      util/main-loop.o
  CC      util/aio-posix.o
  CC      util/compatfd.o
  CC      util/event_notifier-posix.o
  CC      util/mmap-alloc.o
  CC      util/oslib-posix.o
  CC      util/qemu-openpty.o
  CC      util/qemu-thread-posix.o
  CC      util/memfd.o
  CC      util/envlist.o
  CC      util/path.o
  CC      util/module.o
  CC      util/host-utils.o
  CC      util/bitops.o
  CC      util/bitmap.o
  CC      util/hbitmap.o
  CC      util/fifo8.o
  CC      util/acl.o
  CC      util/cacheinfo.o
  CC      util/error.o
  CC      util/qemu-error.o
  CC      util/id.o
  CC      util/iov.o
  CC      util/qemu-config.o
  CC      util/qemu-sockets.o
  CC      util/notify.o
  CC      util/uri.o
  CC      util/qemu-option.o
  CC      util/qemu-progress.o
  CC      util/keyval.o
  CC      util/hexdump.o
  CC      util/crc32c.o
  CC      util/uuid.o
  CC      util/throttle.o
  CC      util/getauxval.o
  CC      util/readline.o
  CC      util/rcu.o
  CC      util/qemu-coroutine.o
  CC      util/qemu-coroutine-lock.o
  CC      util/qemu-coroutine-io.o
  CC      util/coroutine-ucontext.o
  CC      util/qemu-coroutine-sleep.o
  CC      util/buffer.o
  CC      util/timed-average.o
  CC      util/base64.o
  CC      util/qdist.o
  CC      util/log.o
  CC      util/qht.o
  CC      util/range.o
  CC      util/stats64.o
  CC      util/systemd.o
  CC      trace-root.o
  CC      util/trace.o
  CC      crypto/trace.o
  CC      io/trace.o
  CC      migration/trace.o
  CC      block/trace.o
  CC      chardev/trace.o
  CC      hw/block/trace.o
  CC      hw/block/dataplane/trace.o
  CC      hw/char/trace.o
  CC      hw/intc/trace.o
  CC      hw/net/trace.o
  CC      hw/virtio/trace.o
  CC      hw/audio/trace.o
  CC      hw/misc/trace.o
  CC      hw/usb/trace.o
  CC      hw/scsi/trace.o
  CC      hw/nvram/trace.o
  CC      hw/display/trace.o
  CC      hw/input/trace.o
  CC      hw/timer/trace.o
  CC      hw/dma/trace.o
  CC      hw/sparc/trace.o
  CC      hw/sd/trace.o
  CC      hw/isa/trace.o
  CC      hw/mem/trace.o
  CC      hw/i386/trace.o
  CC      hw/i386/xen/trace.o
  CC      hw/9pfs/trace.o
  CC      hw/ppc/trace.o
  CC      hw/pci/trace.o
  CC      hw/s390x/trace.o
  CC      hw/vfio/trace.o
  CC      hw/acpi/trace.o
  CC      hw/arm/trace.o
  CC      hw/alpha/trace.o
  CC      hw/xen/trace.o
  CC      ui/trace.o
  CC      audio/trace.o
  CC      net/trace.o
  CC      target/i386/trace.o
  CC      target/arm/trace.o
  CC      target/mips/trace.o
  CC      target/s390x/trace.o
  CC      target/sparc/trace.o
  CC      target/ppc/trace.o
  CC      qom/trace.o
  CC      qapi/trace.o
  CC      linux-user/trace.o
  CC      accel/tcg/trace.o
  CC      accel/kvm/trace.o
  CC      nbd/trace.o
  CC      crypto/pbkdf-stub.o
  CC      stubs/arch-query-cpu-def.o
  CC      stubs/arch-query-cpu-model-expansion.o
  CC      stubs/arch-query-cpu-model-comparison.o
  CC      stubs/arch-query-cpu-model-baseline.o
  CC      stubs/bdrv-next-monitor-owned.o
  CC      stubs/blk-commit-all.o
  CC      stubs/clock-warp.o
  CC      stubs/blockdev-close-all-bdrv-states.o
  CC      stubs/cpu-get-clock.o
  CC      stubs/cpu-get-icount.o
  CC      stubs/dump.o
  CC      stubs/error-printf.o
  CC      stubs/fdset.o
  CC      stubs/gdbstub.o
  CC      stubs/iothread.o
  CC      stubs/get-vm-name.o
  CC      stubs/iothread-lock.o
  CC      stubs/is-daemonized.o
  CC      stubs/linux-aio.o
  CC      stubs/machine-init-done.o
  CC      stubs/migr-blocker.o
  CC      stubs/change-state-handler.o
  CC      stubs/monitor.o
  CC      stubs/notify-event.o
  CC      stubs/qtest.o
  CC      stubs/runstate-check.o
  CC      stubs/replay.o
  CC      stubs/set-fd-handler.o
  CC      stubs/slirp.o
  CC      stubs/sysbus.o
  CC      stubs/trace-control.o
  CC      stubs/uuid.o
  CC      stubs/vm-stop.o
  CC      stubs/vmstate.o
  CC      stubs/qmp_pc_dimm_device_list.o
  CC      stubs/target-monitor-defs.o
  CC      stubs/target-get-monitor-def.o
  CC      stubs/pc_madt_cpu_entry.o
  CC      stubs/vmgenid.o
  CC      stubs/xen-common.o
  CC      stubs/xen-hvm.o
  CC      contrib/ivshmem-client/ivshmem-client.o
  CC      contrib/ivshmem-client/main.o
  CC      contrib/ivshmem-server/ivshmem-server.o
  CC      contrib/ivshmem-server/main.o
  CC      qemu-nbd.o
  CC      block.o
  CC      blockjob.o
  CC      qemu-io-cmds.o
  CC      replication.o
  CC      block/raw-format.o
  CC      block/qcow.o
  CC      block/vdi.o
  CC      block/vmdk.o
  CC      block/cloop.o
  CC      block/bochs.o
  CC      block/vpc.o
  CC      block/vvfat.o
  CC      block/dmg.o
  CC      block/qcow2.o
  CC      block/qcow2-refcount.o
  CC      block/qcow2-cluster.o
  CC      block/qcow2-snapshot.o
  CC      block/qcow2-cache.o
  CC      block/qcow2-bitmap.o
  CC      block/qed.o
  CC      block/qed-l2-cache.o
  CC      block/qed-table.o
  CC      block/qed-cluster.o
  CC      block/qed-check.o
  CC      block/vhdx.o
  CC      block/vhdx-endian.o
  CC      block/vhdx-log.o
  CC      block/quorum.o
  CC      block/parallels.o
  CC      block/blkverify.o
  CC      block/blkdebug.o
  CC      block/blkreplay.o
  CC      block/block-backend.o
  CC      block/snapshot.o
  CC      block/qapi.o
  CC      block/file-posix.o
  CC      block/linux-aio.o
  CC      block/null.o
  CC      block/mirror.o
  CC      block/commit.o
  CC      block/io.o
  CC      block/throttle-groups.o
  CC      block/nbd.o
  CC      block/nbd-client.o
  CC      block/sheepdog.o
  CC      block/iscsi-opts.o
  CC      block/accounting.o
  CC      block/write-threshold.o
  CC      block/dirty-bitmap.o
  CC      block/backup.o
  CC      block/replication.o
  CC      block/crypto.o
  CC      nbd/server.o
  CC      nbd/client.o
  CC      nbd/common.o
  CC      block/iscsi.o
  CC      block/nfs.o
  CC      block/curl.o
  CC      block/rbd.o
  CC      block/gluster.o
  CC      block/ssh.o
  CC      block/dmg-bz2.o
  CC      crypto/init.o
  CC      crypto/hash.o
  CC      crypto/hash-nettle.o
  CC      crypto/hmac.o
  CC      crypto/aes.o
  CC      crypto/hmac-nettle.o
  CC      crypto/desrfb.o
  CC      crypto/cipher.o
  CC      crypto/tlscreds.o
  CC      crypto/tlscredsanon.o
  CC      crypto/tlscredsx509.o
  CC      crypto/tlssession.o
  CC      crypto/secret.o
  CC      crypto/random-gnutls.o
  CC      crypto/pbkdf.o
  CC      crypto/pbkdf-nettle.o
  CC      crypto/ivgen.o
  CC      crypto/ivgen-essiv.o
  CC      crypto/ivgen-plain64.o
  CC      crypto/ivgen-plain.o
  CC      crypto/afsplit.o
  CC      crypto/xts.o
  CC      crypto/block.o
  CC      crypto/block-qcow.o
  CC      io/channel.o
  CC      crypto/block-luks.o
  CC      io/channel-buffer.o
  CC      io/channel-command.o
  CC      io/channel-file.o
  CC      io/channel-socket.o
  CC      io/channel-tls.o
  CC      io/channel-websock.o
  CC      io/channel-watch.o
  CC      io/channel-util.o
  CC      io/dns-resolver.o
  CC      io/task.o
  CC      qom/object.o
  CC      qom/container.o
  CC      qom/qom-qobject.o
  CC      qom/object_interfaces.o
  GEN     qemu-img-cmds.h
  CC      qemu-io.o
  CC      fsdev/virtfs-proxy-helper.o
  CC      fsdev/9p-marshal.o
  CC      fsdev/9p-iov-marshal.o
  CC      qemu-bridge-helper.o
  CC      blockdev.o
  CC      blockdev-nbd.o
  CC      bootdevice.o
  CC      iothread.o
  CC      qdev-monitor.o
  CC      device-hotplug.o
  CC      os-posix.o
  CC      bt-host.o
  CC      bt-vhci.o
  CC      dma-helpers.o
  CC      vl.o
  CC      tpm.o
  CC      device_tree.o
  CC      qmp-marshal.o
  CC      qmp.o
  CC      hmp.o
  CC      cpus-common.o
  CC      audio/audio.o
  CC      audio/noaudio.o
  CC      audio/wavaudio.o
  CC      audio/mixeng.o
  CC      audio/sdlaudio.o
  CC      audio/ossaudio.o
  CC      audio/wavcapture.o
  CC      backends/rng-egd.o
  CC      backends/rng.o
  CC      backends/rng-random.o
  CC      backends/tpm.o
  CC      backends/hostmem.o
  CC      backends/hostmem-ram.o
  CC      backends/hostmem-file.o
  CC      backends/cryptodev.o
  CC      backends/cryptodev-builtin.o
  CC      block/stream.o
  CC      chardev/msmouse.o
  CC      chardev/wctablet.o
  CC      chardev/testdev.o
  CC      chardev/baum.o
  CC      disas/alpha.o
  CC      disas/arm.o
  CXX     disas/arm-a64.o
  CC      disas/cris.o
  CC      disas/hppa.o
  CC      disas/i386.o
  CC      disas/m68k.o
  CC      disas/microblaze.o
  CC      disas/mips.o
  CC      disas/nios2.o
  CC      disas/moxie.o
  CC      disas/ppc.o
  CC      disas/s390.o
  CC      disas/sh4.o
  CC      disas/sparc.o
  CC      disas/lm32.o
  CXX     disas/libvixl/vixl/utils.o
  CXX     disas/libvixl/vixl/compiler-intrinsics.o
  CXX     disas/libvixl/vixl/a64/decoder-a64.o
  CXX     disas/libvixl/vixl/a64/instructions-a64.o
  CXX     disas/libvixl/vixl/a64/disasm-a64.o
  CC      fsdev/qemu-fsdev.o
  CC      fsdev/qemu-fsdev-opts.o
  CC      fsdev/qemu-fsdev-throttle.o
  CC      fsdev/qemu-fsdev-dummy.o
  CC      hw/9pfs/9p.o
  CC      hw/9pfs/9p-util.o
  CC      hw/9pfs/9p-local.o
  CC      hw/9pfs/9p-xattr.o
  CC      hw/9pfs/9p-xattr-user.o
  CC      hw/9pfs/9p-posix-acl.o
  CC      hw/9pfs/cofs.o
  CC      hw/9pfs/coth.o
  CC      hw/9pfs/codir.o
  CC      hw/9pfs/cofile.o
  CC      hw/9pfs/9p-synth.o
  CC      hw/9pfs/coxattr.o
  CC      hw/9pfs/9p-proxy.o
  CC      hw/9pfs/9p-handle.o
  CC      hw/acpi/core.o
  CC      hw/acpi/piix4.o
  CC      hw/acpi/pcihp.o
  CC      hw/acpi/ich9.o
  CC      hw/acpi/tco.o
  CC      hw/acpi/cpu_hotplug.o
  CC      hw/acpi/memory_hotplug.o
  CC      hw/acpi/cpu.o
  CC      hw/acpi/nvdimm.o
  CC      hw/acpi/vmgenid.o
  CC      hw/acpi/hest_ghes.o
  CC      hw/acpi/acpi_interface.o
  CC      hw/acpi/bios-linker-loader.o
  CC      hw/acpi/aml-build.o
  CC      hw/acpi/ipmi.o
  CC      hw/acpi/acpi-stub.o
  CC      hw/acpi/ipmi-stub.o
  CC      hw/audio/sb16.o
  CC      hw/audio/es1370.o
  CC      hw/audio/ac97.o
  CC      hw/audio/fmopl.o
  CC      hw/audio/adlib.o
  CC      hw/audio/gus.o
  CC      hw/audio/gusemu_hal.o
  CC      hw/audio/gusemu_mixer.o
  CC      hw/audio/cs4231a.o
  CC      hw/audio/intel-hda.o
  CC      hw/audio/hda-codec.o
  CC      hw/audio/wm8750.o
  CC      hw/audio/pcspk.o
  CC      hw/audio/pl041.o
  CC      hw/audio/lm4549.o
  CC      hw/audio/cs4231.o
  CC      hw/audio/marvell_88w8618.o
  CC      hw/audio/milkymist-ac97.o
  CC      hw/audio/soundhw.o
  CC      hw/block/block.o
  CC      hw/block/cdrom.o
  CC      hw/block/hd-geometry.o
  CC      hw/block/fdc.o
  CC      hw/block/m25p80.o
  CC      hw/block/nand.o
  CC      hw/block/pflash_cfi01.o
  CC      hw/block/pflash_cfi02.o
  CC      hw/block/ecc.o
  CC      hw/block/onenand.o
  CC      hw/block/nvme.o
  CC      hw/bt/l2cap.o
  CC      hw/bt/core.o
  CC      hw/bt/sdp.o
  CC      hw/bt/hci.o
  CC      hw/bt/hid.o
  CC      hw/bt/hci-csr.o
  CC      hw/char/ipoctal232.o
  CC      hw/char/escc.o
  CC      hw/char/parallel.o
  CC      hw/char/pl011.o
  CC      hw/char/serial.o
  CC      hw/char/serial-isa.o
  CC      hw/char/serial-pci.o
  CC      hw/char/virtio-console.o
  CC      hw/char/xilinx_uartlite.o
  CC      hw/char/cadence_uart.o
  CC      hw/char/cmsdk-apb-uart.o
  CC      hw/char/etraxfs_ser.o
  CC      hw/char/debugcon.o
  CC      hw/char/grlib_apbuart.o
  CC      hw/char/lm32_juart.o
  CC      hw/char/imx_serial.o
  CC      hw/char/lm32_uart.o
  CC      hw/char/milkymist-uart.o
  CC      hw/char/sclpconsole.o
  CC      hw/char/sclpconsole-lm.o
  CC      hw/core/qdev.o
  CC      hw/core/qdev-properties.o
  CC      hw/core/bus.o
  CC      hw/core/reset.o
  CC      hw/core/fw-path-provider.o
  CC      hw/core/irq.o
  CC      hw/core/hotplug.o
  CC      hw/core/nmi.o
  CC      hw/core/empty_slot.o
  CC      hw/core/ptimer.o
  CC      hw/core/stream.o
  CC      hw/core/sysbus.o
  CC      hw/core/machine.o
  CC      hw/core/loader-fit.o
  CC      hw/core/loader.o
  CC      hw/core/qdev-properties-system.o
  CC      hw/core/register.o
  CC      hw/core/or-irq.o
  CC      hw/core/platform-bus.o
  CC      hw/cpu/core.o
  CC      hw/display/ads7846.o
  CC      hw/display/g364fb.o
  CC      hw/display/cirrus_vga.o
  CC      hw/display/jazz_led.o
  CC      hw/display/pl110.o
  CC      hw/display/ssd0303.o
  CC      hw/display/ssd0323.o
  CC      hw/display/vga-pci.o
  CC      hw/display/vga-isa.o
  CC      hw/display/vga-isa-mm.o
  CC      hw/display/vmware_vga.o
  CC      hw/display/blizzard.o
  CC      hw/display/exynos4210_fimd.o
  CC      hw/display/framebuffer.o
  CC      hw/display/milkymist-vgafb.o
  CC      hw/display/tc6393xb.o
  CC      hw/display/milkymist-tmu2.o
  CC      hw/dma/puv3_dma.o
  CC      hw/dma/rc4030.o
  CC      hw/dma/pl080.o
  CC      hw/dma/pl330.o
  CC      hw/dma/i82374.o
  CC      hw/dma/i8257.o
  CC      hw/dma/xilinx_axidma.o
  CC      hw/dma/xlnx-zynq-devcfg.o
  CC      hw/dma/etraxfs_dma.o
  CC      hw/dma/sparc32_dma.o
  CC      hw/dma/sun4m_iommu.o
  CC      hw/gpio/max7310.o
  CC      hw/gpio/pl061.o
  CC      hw/gpio/puv3_gpio.o
  CC      hw/gpio/zaurus.o
  CC      hw/gpio/gpio_key.o
  CC      hw/gpio/mpc8xxx.o
  CC      hw/i2c/core.o
  CC      hw/i2c/smbus.o
  CC      hw/i2c/smbus_eeprom.o
  CC      hw/i2c/i2c-ddc.o
  CC      hw/i2c/versatile_i2c.o
  CC      hw/i2c/smbus_ich9.o
  CC      hw/i2c/pm_smbus.o
  CC      hw/i2c/bitbang_i2c.o
  CC      hw/i2c/exynos4210_i2c.o
  CC      hw/i2c/imx_i2c.o
  CC      hw/i2c/aspeed_i2c.o
  CC      hw/ide/core.o
  CC      hw/ide/atapi.o
  CC      hw/ide/qdev.o
  CC      hw/ide/pci.o
  CC      hw/ide/isa.o
  CC      hw/ide/piix.o
  CC      hw/ide/cmd646.o
  CC      hw/ide/macio.o
  CC      hw/ide/via.o
  CC      hw/ide/mmio.o
  CC      hw/ide/microdrive.o
  CC      hw/ide/ahci.o
  CC      hw/input/adb.o
  CC      hw/input/hid.o
  CC      hw/ide/ich.o
  CC      hw/input/lm832x.o
  CC      hw/input/pckbd.o
  CC      hw/input/pl050.o
  CC      hw/input/ps2.o
  CC      hw/input/stellaris_input.o
  CC      hw/input/tsc2005.o
  CC      hw/input/vmmouse.o
  CC      hw/input/virtio-input.o
  CC      hw/input/virtio-input-hid.o
  CC      hw/input/virtio-input-host.o
  CC      hw/intc/heathrow_pic.o
  CC      hw/intc/i8259_common.o
  CC      hw/intc/i8259.o
  CC      hw/intc/pl190.o
  CC      hw/intc/puv3_intc.o
  CC      hw/intc/xilinx_intc.o
  CC      hw/intc/imx_avic.o
  CC      hw/intc/etraxfs_pic.o
  CC      hw/intc/realview_gic.o
  CC      hw/intc/lm32_pic.o
  CC      hw/intc/slavio_intctl.o
  CC      hw/intc/ioapic_common.o
  CC      hw/intc/arm_gic_common.o
  CC      hw/intc/arm_gic.o
  CC      hw/intc/arm_gicv2m.o
  CC      hw/intc/arm_gicv3_common.o
  CC      hw/intc/arm_gicv3.o
  CC      hw/intc/arm_gicv3_dist.o
  CC      hw/intc/arm_gicv3_redist.o
  CC      hw/intc/arm_gicv3_its_common.o
  CC      hw/intc/openpic.o
  CC      hw/intc/intc.o
  CC      hw/ipack/ipack.o
  CC      hw/ipack/tpci200.o
  CC      hw/ipmi/ipmi.o
  CC      hw/ipmi/ipmi_bmc_sim.o
  CC      hw/ipmi/ipmi_bmc_extern.o
  CC      hw/ipmi/isa_ipmi_kcs.o
  CC      hw/ipmi/isa_ipmi_bt.o
  CC      hw/isa/isa-bus.o
  CC      hw/isa/apm.o
  CC      hw/isa/i82378.o
  CC      hw/isa/pc87312.o
  CC      hw/isa/piix4.o
  CC      hw/isa/vt82c686.o
  CC      hw/mem/pc-dimm.o
  CC      hw/mem/nvdimm.o
  CC      hw/misc/applesmc.o
  CC      hw/misc/max111x.o
  CC      hw/misc/tmp105.o
  CC      hw/misc/tmp421.o
  CC      hw/misc/debugexit.o
  CC      hw/misc/sga.o
  CC      hw/misc/pc-testdev.o
  CC      hw/misc/pci-testdev.o
  CC      hw/misc/edu.o
  CC      hw/misc/arm_l2x0.o
  CC      hw/misc/unimp.o
  CC      hw/misc/arm_integrator_debug.o
  CC      hw/misc/a9scu.o
  CC      hw/misc/arm11scu.o
  CC      hw/misc/puv3_pm.o
  CC      hw/misc/macio/macio.o
  CC      hw/misc/macio/cuda.o
  CC      hw/misc/macio/mac_dbdma.o
  CC      hw/net/dp8393x.o
  CC      hw/net/ne2000.o
  CC      hw/net/eepro100.o
  CC      hw/net/pcnet-pci.o
  CC      hw/net/pcnet.o
  CC      hw/net/e1000.o
  CC      hw/net/e1000x_common.o
  CC      hw/net/net_tx_pkt.o
  CC      hw/net/net_rx_pkt.o
  CC      hw/net/e1000e.o
  CC      hw/net/e1000e_core.o
  CC      hw/net/rtl8139.o
  CC      hw/net/vmxnet3.o
  CC      hw/net/smc91c111.o
  CC      hw/net/lan9118.o
  CC      hw/net/ne2000-isa.o
  CC      hw/net/opencores_eth.o
  CC      hw/net/xgmac.o
  CC      hw/net/mipsnet.o
  CC      hw/net/xilinx_axienet.o
  CC      hw/net/allwinner_emac.o
  CC      hw/net/imx_fec.o
  CC      hw/net/cadence_gem.o
  CC      hw/net/stellaris_enet.o
  CC      hw/net/lance.o
  CC      hw/net/ftgmac100.o
  CC      hw/net/rocker/rocker.o
  CC      hw/net/rocker/rocker_fp.o
  CC      hw/net/rocker/rocker_desc.o
  CC      hw/net/rocker/rocker_world.o
  CC      hw/net/rocker/rocker_of_dpa.o
  CC      hw/nvram/ds1225y.o
  CC      hw/nvram/eeprom93xx.o
  CC      hw/nvram/fw_cfg.o
  CC      hw/nvram/chrp_nvram.o
  CC      hw/nvram/mac_nvram.o
  CC      hw/pci-bridge/pci_bridge_dev.o
  CC      hw/pci-bridge/pcie_root_port.o
  CC      hw/pci-bridge/gen_pcie_root_port.o
  CC      hw/pci-bridge/pci_expander_bridge.o
  CC      hw/pci-bridge/xio3130_upstream.o
  CC      hw/pci-bridge/xio3130_downstream.o
  CC      hw/pci-bridge/ioh3420.o
  CC      hw/pci-bridge/i82801b11.o
  CC      hw/pci-bridge/dec.o
  CC      hw/pci-host/pam.o
  CC      hw/pci-host/prep.o
  CC      hw/pci-host/grackle.o
  CC      hw/pci-host/uninorth.o
  CC      hw/pci-host/ppce500.o
  CC      hw/pci-host/apb.o
  CC      hw/pci-host/versatile.o
  CC      hw/pci-host/bonito.o
  CC      hw/pci-host/piix.o
  CC      hw/pci-host/q35.o
  CC      hw/pci-host/gpex.o
  CC      hw/pci-host/xilinx-pcie.o
  CC      hw/pci/pci.o
  CC      hw/pci/pci_bridge.o
  CC      hw/pci/msix.o
  CC      hw/pci/msi.o
  CC      hw/pci/shpc.o
  CC      hw/pci/slotid_cap.o
  CC      hw/pci/pci_host.o
  CC      hw/pci/pcie.o
  CC      hw/pci/pcie_host.o
  CC      hw/pci/pcie_aer.o
  CC      hw/pci/pcie_port.o
  CC      hw/pci/pci-stub.o
  CC      hw/pcmcia/pcmcia.o
  CC      hw/scsi/scsi-disk.o
  CC      hw/scsi/scsi-generic.o
  CC      hw/scsi/scsi-bus.o
  CC      hw/scsi/lsi53c895a.o
  CC      hw/scsi/mptsas.o
  CC      hw/scsi/mptconfig.o
  CC      hw/scsi/mptendian.o
  CC      hw/scsi/vmw_pvscsi.o
  CC      hw/scsi/megasas.o
  CC      hw/scsi/esp.o
  CC      hw/scsi/esp-pci.o
  CC      hw/sd/ssi-sd.o
  CC      hw/sd/pl181.o
  CC      hw/sd/sd.o
  CC      hw/sd/core.o
  CC      hw/sd/sdhci.o
  CC      hw/smbios/smbios.o
  CC      hw/smbios/smbios_type_38.o
  CC      hw/smbios/smbios-stub.o
  CC      hw/smbios/smbios_type_38-stub.o
  CC      hw/ssi/pl022.o
  CC      hw/ssi/ssi.o
  CC      hw/ssi/xilinx_spips.o
  CC      hw/ssi/xilinx_spi.o
  CC      hw/ssi/aspeed_smc.o
  CC      hw/ssi/stm32f2xx_spi.o
  CC      hw/timer/arm_timer.o
  CC      hw/timer/arm_mptimer.o
  CC      hw/timer/armv7m_systick.o
  CC      hw/timer/a9gtimer.o
  CC      hw/timer/cadence_ttc.o
  CC      hw/timer/ds1338.o
  CC      hw/timer/hpet.o
  CC      hw/timer/i8254.o
  CC      hw/timer/m48t59.o
  CC      hw/timer/i8254_common.o
  CC      hw/timer/m48t59-isa.o
  CC      hw/timer/pl031.o
  CC      hw/timer/puv3_ost.o
  CC      hw/timer/twl92230.o
  CC      hw/timer/xilinx_timer.o
  CC      hw/timer/etraxfs_timer.o
  CC      hw/timer/slavio_timer.o
  CC      hw/timer/grlib_gptimer.o
  CC      hw/timer/imx_epit.o
  CC      hw/timer/imx_gpt.o
  CC      hw/timer/lm32_timer.o
  CC      hw/timer/milkymist-sysctl.o
  CC      hw/timer/stm32f2xx_timer.o
  CC      hw/timer/aspeed_timer.o
  CC      hw/timer/cmsdk-apb-timer.o
  CC      hw/timer/sun4v-rtc.o
  CC      hw/tpm/tpm_tis.o
  CC      hw/usb/core.o
  CC      hw/usb/combined-packet.o
  CC      hw/usb/bus.o
  CC      hw/usb/libhw.o
  CC      hw/usb/desc.o
  CC      hw/usb/desc-msos.o
  CC      hw/usb/hcd-uhci.o
  CC      hw/usb/hcd-ohci.o
  CC      hw/usb/hcd-ehci.o
  CC      hw/usb/hcd-ehci-pci.o
  CC      hw/usb/hcd-ehci-sysbus.o
  CC      hw/usb/hcd-xhci.o
  CC      hw/usb/hcd-xhci-nec.o
  CC      hw/usb/hcd-musb.o
  CC      hw/usb/dev-hub.o
  CC      hw/usb/dev-hid.o
  CC      hw/usb/dev-wacom.o
  CC      hw/usb/dev-storage.o
  CC      hw/usb/dev-uas.o
  CC      hw/usb/dev-audio.o
  CC      hw/usb/dev-serial.o
  CC      hw/usb/dev-network.o
  CC      hw/usb/dev-bluetooth.o
  CC      hw/usb/dev-smartcard-reader.o
  CC      hw/usb/ccid-card-passthru.o
  CC      hw/usb/ccid-card-emulated.o
  CC      hw/usb/dev-mtp.o
  CC      hw/usb/redirect.o
  CC      hw/usb/quirks.o
  CC      hw/usb/host-libusb.o
  CC      hw/usb/host-legacy.o
  CC      hw/virtio/virtio-rng.o
  CC      hw/virtio/virtio-pci.o
  CC      hw/virtio/virtio-bus.o
  CC      hw/virtio/virtio-mmio.o
  CC      hw/virtio/vhost-stub.o
  CC      hw/watchdog/watchdog.o
  CC      hw/watchdog/wdt_i6300esb.o
  CC      hw/watchdog/wdt_ib700.o
  CC      hw/watchdog/wdt_diag288.o
  CC      hw/watchdog/wdt_aspeed.o
  CC      migration/migration.o
  CC      migration/fd.o
  CC      migration/socket.o
  CC      migration/exec.o
  CC      migration/tls.o
  CC      migration/channel.o
  CC      migration/savevm.o
  CC      migration/colo-comm.o
  CC      migration/colo.o
  CC      migration/colo-failover.o
  CC      migration/vmstate-types.o
  CC      migration/vmstate.o
  CC      migration/page_cache.o
  CC      migration/qemu-file.o
  CC      migration/global_state.o
  CC      migration/qemu-file-channel.o
  CC      migration/xbzrle.o
  CC      migration/postcopy-ram.o
  CC      migration/qjson.o
  CC      migration/block.o
  CC      net/net.o
  CC      net/queue.o
  CC      net/checksum.o
  CC      net/util.o
  CC      net/hub.o
  CC      net/socket.o
  CC      net/dump.o
  CC      net/eth.o
  CC      net/l2tpv3.o
  CC      net/vhost-user.o
  CC      net/slirp.o
  CC      net/filter.o
  CC      net/filter-buffer.o
  CC      net/colo-compare.o
  CC      net/filter-mirror.o
  CC      net/colo.o
  CC      net/filter-rewriter.o
  CC      net/filter-replay.o
  CC      net/tap.o
  CC      net/tap-linux.o
  CC      qom/cpu.o
  CC      replay/replay.o
  CC      replay/replay-internal.o
  CC      replay/replay-events.o
  CC      replay/replay-time.o
  CC      replay/replay-input.o
  CC      replay/replay-snapshot.o
  CC      replay/replay-char.o
  CC      replay/replay-net.o
  CC      replay/replay-audio.o
  CC      slirp/cksum.o
  CC      slirp/ip_icmp.o
  CC      slirp/if.o
  CC      slirp/ip6_icmp.o
  CC      slirp/ip6_input.o
  CC      slirp/ip_input.o
  CC      slirp/ip6_output.o
  CC      slirp/ip_output.o
  CC      slirp/dnssearch.o
  CC      slirp/dhcpv6.o
  CC      slirp/slirp.o
  CC      slirp/mbuf.o
  CC      slirp/sbuf.o
  CC      slirp/misc.o
  CC      slirp/socket.o
  CC      slirp/tcp_input.o
  CC      slirp/tcp_output.o
  CC      slirp/tcp_subr.o
  CC      slirp/tcp_timer.o
  CC      slirp/udp.o
  CC      slirp/udp6.o
  CC      slirp/bootp.o
  CC      slirp/tftp.o
  CC      slirp/arp_table.o
  CC      slirp/ndp_table.o
  CC      slirp/ncsi.o
  CC      ui/keymaps.o
  CC      ui/console.o
  CC      ui/cursor.o
  CC      ui/qemu-pixman.o
  CC      ui/input.o
  CC      ui/input-keymap.o
  CC      ui/input-legacy.o
  CC      ui/input-linux.o
  CC      ui/sdl2.o
  CC      ui/sdl2-input.o
  CC      ui/sdl2-2d.o
  CC      ui/sdl2-gl.o
  CC      ui/x_keymap.o
  CC      ui/curses.o
  CC      ui/vnc-enc-zlib.o
  CC      ui/vnc.o
  CC      ui/vnc-enc-hextile.o
  CC      ui/vnc-palette.o
  CC      ui/vnc-enc-tight.o
  CC      ui/vnc-enc-zrle.o
  CC      ui/vnc-auth-vencrypt.o
  CC      ui/vnc-auth-sasl.o
  CC      ui/vnc-ws.o
  CC      ui/vnc-jobs.o
  CC      ui/gtk.o
  CC      ui/shader.o
  VERT    ui/shader/texture-blit-vert.h
  FRAG    ui/shader/texture-blit-frag.h
  CC      ui/egl-helpers.o
  CC      ui/egl-context.o
  CC      ui/egl-headless.o
  CC      ui/gtk-gl-area.o
  CC      chardev/char.o
  CC      chardev/char-fd.o
  CC      chardev/char-fe.o
  CC      chardev/char-file.o
  CC      chardev/char-io.o
  CC      chardev/char-mux.o
  CC      chardev/char-null.o
  CC      chardev/char-parallel.o
  CC      chardev/char-pipe.o
  CC      chardev/char-pty.o
  CC      chardev/char-ringbuf.o
  CC      chardev/char-serial.o
  CC      chardev/char-socket.o
  CC      chardev/char-stdio.o
  CC      chardev/char-udp.o
  LINK    tests/qemu-iotests/socket_scm_helper
  CCAS    s390-ccw/start.o
  GEN     qemu-doc.html
  CC      s390-ccw/main.o
  CC      s390-ccw/bootmap.o
  CC      s390-ccw/sclp.o
  CC      s390-ccw/virtio.o
  CC      s390-ccw/virtio-scsi.o
  GEN     qemu-doc.txt
  CC      s390-ccw/virtio-blkdev.o
s390-netboot.img not built since roms/SLOF/ is not available.
  GEN     qemu.1
  BUILD   s390-ccw/s390-ccw.elf
  STRIP   s390-ccw/s390-ccw.img
  GEN     docs/interop/qemu-qmp-ref.html
  GEN     docs/interop/qemu-qmp-ref.txt
  GEN     docs/interop/qemu-qmp-ref.7
  GEN     docs/interop/qemu-ga-ref.html
  GEN     docs/interop/qemu-ga-ref.txt
  GEN     docs/interop/qemu-ga-ref.7
  CC      qga/commands.o
  CC      qga/guest-agent-command-state.o
  CC      qga/main.o
  CC      qga/channel-posix.o
  CC      qga/commands-posix.o
  CC      qga/qapi-generated/qga-qapi-types.o
  CC      qga/qapi-generated/qga-qapi-visit.o
  AR      libqemuutil.a
  CC      qga/qapi-generated/qga-qmp-marshal.o
  AR      libqemustub.a
  CC      qemu-img.o
  LINK    qemu-io
  LINK    fsdev/virtfs-proxy-helper
  LINK    qemu-bridge-helper
  CC      ui/console-gl.o
  LINK    qemu-ga
  LINK    ivshmem-client
  LINK    ivshmem-server
  LINK    qemu-nbd
  GEN     alpha-softmmu/hmp-commands.h
  GEN     alpha-softmmu/hmp-commands-info.h
  GEN     alpha-softmmu/config-target.h
  GEN     cris-softmmu/hmp-commands.h
  GEN     cris-softmmu/hmp-commands-info.h
  GEN     cris-softmmu/config-target.h
  CC      alpha-softmmu/exec.o
  CC      cris-softmmu/exec.o
  GEN     aarch64-softmmu/hmp-commands.h
  GEN     arm-softmmu/hmp-commands.h
  GEN     aarch64-softmmu/hmp-commands-info.h
  GEN     arm-softmmu/hmp-commands-info.h
  GEN     arm-softmmu/config-target.h
  GEN     aarch64-softmmu/config-target.h
  CC      arm-softmmu/exec.o
  CC      aarch64-softmmu/exec.o
  CC      arm-softmmu/tcg/tcg.o
  CC      alpha-softmmu/tcg/tcg.o
  CC      cris-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/tcg/tcg.o
  CC      cris-softmmu/tcg/tcg-op.o
  CC      arm-softmmu/tcg/tcg-op.o
  CC      alpha-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/tcg/tcg-op.o
  CC      cris-softmmu/tcg/optimize.o
  CC      arm-softmmu/tcg/optimize.o
  CC      cris-softmmu/tcg/tcg-common.o
  CC      alpha-softmmu/tcg/optimize.o
  CC      cris-softmmu/tcg/tcg-runtime.o
  CC      arm-softmmu/tcg/tcg-common.o
  CC      alpha-softmmu/tcg/tcg-common.o
  CC      cris-softmmu/fpu/softfloat.o
  CC      arm-softmmu/tcg/tcg-runtime.o
  CC      alpha-softmmu/tcg/tcg-runtime.o
  CC      aarch64-softmmu/tcg/optimize.o
  CC      arm-softmmu/fpu/softfloat.o
  CC      alpha-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/tcg/tcg-runtime.o
  CC      cris-softmmu/disas.o
  CC      aarch64-softmmu/fpu/softfloat.o
  CC      cris-softmmu/hax-stub.o
  CC      arm-softmmu/disas.o
  CC      cris-softmmu/arch_init.o
  CC      cris-softmmu/cpus.o
  CC      alpha-softmmu/disas.o
  GEN     arm-softmmu/gdbstub-xml.c
  CC      alpha-softmmu/hax-stub.o
  CC      cris-softmmu/monitor.o
  CC      alpha-softmmu/arch_init.o
  CC      arm-softmmu/hax-stub.o
  CC      arm-softmmu/arch_init.o
  CC      alpha-softmmu/cpus.o
  CC      arm-softmmu/cpus.o
  CC      alpha-softmmu/monitor.o
  CC      aarch64-softmmu/disas.o
  CC      cris-softmmu/gdbstub.o
  GEN     aarch64-softmmu/gdbstub-xml.c
  CC      arm-softmmu/monitor.o
  CC      aarch64-softmmu/hax-stub.o
  CC      cris-softmmu/balloon.o
  CC      aarch64-softmmu/arch_init.o
  CC      cris-softmmu/ioport.o
  CC      alpha-softmmu/gdbstub.o
  CC      aarch64-softmmu/cpus.o
  CC      cris-softmmu/numa.o
  CC      arm-softmmu/gdbstub.o
  CC      aarch64-softmmu/monitor.o
  CC      arm-softmmu/balloon.o
  CC      cris-softmmu/qtest.o
  CC      alpha-softmmu/balloon.o
  CC      arm-softmmu/ioport.o
  CC      cris-softmmu/memory.o
  CC      alpha-softmmu/ioport.o
  CC      arm-softmmu/numa.o
  CC      alpha-softmmu/numa.o
  CC      arm-softmmu/qtest.o
  CC      alpha-softmmu/qtest.o
  CC      aarch64-softmmu/gdbstub.o
  CC      cris-softmmu/memory_mapping.o
  CC      arm-softmmu/memory.o
  CC      alpha-softmmu/memory.o
  CC      aarch64-softmmu/balloon.o
  CC      cris-softmmu/dump.o
  CC      aarch64-softmmu/ioport.o
  CC      cris-softmmu/migration/ram.o
  CC      aarch64-softmmu/numa.o
  CC      arm-softmmu/memory_mapping.o
  CC      alpha-softmmu/memory_mapping.o
  CC      alpha-softmmu/dump.o
  CC      cris-softmmu/accel/accel.o
  CC      aarch64-softmmu/qtest.o
  CC      arm-softmmu/dump.o
  CC      cris-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/memory.o
  CC      alpha-softmmu/migration/ram.o
  CC      cris-softmmu/accel/tcg/tcg-all.o
  CC      arm-softmmu/migration/ram.o
  CC      cris-softmmu/accel/tcg/cputlb.o
  CC      alpha-softmmu/accel/accel.o
  CC      aarch64-softmmu/memory_mapping.o
  CC      arm-softmmu/accel/accel.o
  CC      alpha-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/dump.o
  CC      alpha-softmmu/accel/tcg/tcg-all.o
  CC      arm-softmmu/accel/stubs/kvm-stub.o
  CC      alpha-softmmu/accel/tcg/cputlb.o
  CC      cris-softmmu/accel/tcg/cpu-exec.o
  CC      arm-softmmu/accel/tcg/tcg-all.o
  CC      aarch64-softmmu/migration/ram.o
  CC      arm-softmmu/accel/tcg/cputlb.o
  CC      cris-softmmu/accel/tcg/cpu-exec-common.o
  CC      cris-softmmu/accel/tcg/translate-all.o
  CC      aarch64-softmmu/accel/accel.o
  CC      alpha-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/accel/stubs/kvm-stub.o
  CC      cris-softmmu/hw/core/generic-loader.o
  CC      alpha-softmmu/accel/tcg/cpu-exec-common.o
  CC      aarch64-softmmu/accel/tcg/tcg-all.o
  CC      cris-softmmu/hw/core/null-machine.o
  CC      alpha-softmmu/accel/tcg/translate-all.o
  CC      arm-softmmu/accel/tcg/cpu-exec.o
  CC      cris-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/accel/tcg/cputlb.o
  CC      cris-softmmu/hw/net/etraxfs_eth.o
  CC      alpha-softmmu/hw/9pfs/virtio-9p-device.o
  CC      cris-softmmu/hw/net/vhost_net.o
  CC      arm-softmmu/accel/tcg/cpu-exec-common.o
  CC      alpha-softmmu/hw/block/virtio-blk.o
  CC      cris-softmmu/hw/net/rocker/qmp-norocker.o
  CC      arm-softmmu/accel/tcg/translate-all.o
  CC      cris-softmmu/hw/vfio/common.o
  CC      alpha-softmmu/hw/block/dataplane/virtio-blk.o
  CC      arm-softmmu/hw/9pfs/virtio-9p-device.o
  CC      alpha-softmmu/hw/char/virtio-serial-bus.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec.o
  CC      cris-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/adc/stm32f2xx_adc.o
  CC      cris-softmmu/hw/vfio/spapr.o
  CC      alpha-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/block/virtio-blk.o
  CC      alpha-softmmu/hw/core/null-machine.o
  CC      cris-softmmu/hw/cris/boot.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec-common.o
  CC      cris-softmmu/hw/cris/axis_dev88.o
  CC      arm-softmmu/hw/block/dataplane/virtio-blk.o
  CC      alpha-softmmu/hw/display/vga.o
  CC      aarch64-softmmu/accel/tcg/translate-all.o
  CC      arm-softmmu/hw/char/exynos4210_uart.o
  CC      cris-softmmu/target/cris/translate.o
  CC      arm-softmmu/hw/char/omap_uart.o
  CC      aarch64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      arm-softmmu/hw/char/digic-uart.o
  CC      alpha-softmmu/hw/display/virtio-gpu.o
  CC      arm-softmmu/hw/char/stm32f2xx_usart.o
  CC      aarch64-softmmu/hw/adc/stm32f2xx_adc.o
  CC      arm-softmmu/hw/char/bcm2835_aux.o
  CC      aarch64-softmmu/hw/block/virtio-blk.o
  CC      arm-softmmu/hw/char/virtio-serial-bus.o
  CC      alpha-softmmu/hw/display/virtio-gpu-3d.o
  CC      cris-softmmu/target/cris/op_helper.o
  CC      arm-softmmu/hw/core/generic-loader.o
  CC      aarch64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      cris-softmmu/target/cris/helper.o
  CC      arm-softmmu/hw/core/null-machine.o
  CC      alpha-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-softmmu/hw/char/exynos4210_uart.o
  CC      cris-softmmu/target/cris/cpu.o
  CC      arm-softmmu/hw/cpu/arm11mpcore.o
  CC      aarch64-softmmu/hw/char/omap_uart.o
  CC      cris-softmmu/target/cris/gdbstub.o
  CC      alpha-softmmu/hw/misc/ivshmem.o
  CC      aarch64-softmmu/hw/char/digic-uart.o
  CC      arm-softmmu/hw/cpu/realview_mpcore.o
  CC      cris-softmmu/target/cris/mmu.o
  CC      alpha-softmmu/hw/misc/mmio_interface.o
  CC      arm-softmmu/hw/cpu/a9mpcore.o
  CC      aarch64-softmmu/hw/char/stm32f2xx_usart.o
  CC      alpha-softmmu/hw/net/virtio-net.o
  CC      cris-softmmu/target/cris/machine.o
  CC      arm-softmmu/hw/cpu/a15mpcore.o
  CC      aarch64-softmmu/hw/char/bcm2835_aux.o
  GEN     trace/generated-helpers.c
  CC      cris-softmmu/trace/control-target.o
  CC      arm-softmmu/hw/display/omap_dss.o
  CC      aarch64-softmmu/hw/char/virtio-serial-bus.o
  CC      alpha-softmmu/hw/net/vhost_net.o
  CC      cris-softmmu/trace/generated-helpers.o
  CC      arm-softmmu/hw/display/omap_lcdc.o
  CC      alpha-softmmu/hw/scsi/virtio-scsi.o
  LINK    cris-softmmu/qemu-system-cris
  CC      aarch64-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/display/pxa2xx_lcd.o
  CC      alpha-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/hw/cpu/arm11mpcore.o
  CC      alpha-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/cpu/realview_mpcore.o
  CC      alpha-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-softmmu/hw/cpu/a9mpcore.o
  CC      arm-softmmu/hw/display/bcm2835_fb.o
  CC      alpha-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/cpu/a15mpcore.o
  CC      alpha-softmmu/hw/timer/mc146818rtc.o
  GEN     i386-softmmu/hmp-commands.h
  CC      aarch64-softmmu/hw/display/omap_dss.o
  GEN     i386-softmmu/hmp-commands-info.h
  GEN     i386-softmmu/config-target.h
  CC      arm-softmmu/hw/display/vga.o
  CC      i386-softmmu/exec.o
  CC      alpha-softmmu/hw/vfio/common.o
  CC      aarch64-softmmu/hw/display/omap_lcdc.o
  CC      arm-softmmu/hw/display/virtio-gpu.o
  CC      aarch64-softmmu/hw/display/pxa2xx_lcd.o
  CC      alpha-softmmu/hw/vfio/pci.o
  CC      i386-softmmu/tcg/tcg.o
  CC      arm-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/display/bcm2835_fb.o
  CC      alpha-softmmu/hw/vfio/pci-quirks.o
  CC      aarch64-softmmu/hw/display/vga.o
  CC      arm-softmmu/hw/display/virtio-gpu-pci.o
  CC      alpha-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/dma/omap_dma.o
  CC      alpha-softmmu/hw/vfio/spapr.o
  CC      i386-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/hw/display/virtio-gpu.o
  CC      alpha-softmmu/hw/virtio/virtio.o
  CC      arm-softmmu/hw/dma/soc_dma.o
  CC      arm-softmmu/hw/dma/pxa2xx_dma.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-3d.o
  CC      alpha-softmmu/hw/virtio/virtio-balloon.o
  CC      arm-softmmu/hw/dma/bcm2835_dma.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-pci.o
  CC      alpha-softmmu/hw/virtio/vhost.o
  CC      arm-softmmu/hw/gpio/omap_gpio.o
  CC      arm-softmmu/hw/gpio/imx_gpio.o
  CC      aarch64-softmmu/hw/display/dpcd.o
  CC      i386-softmmu/tcg/optimize.o
  CC      arm-softmmu/hw/gpio/bcm2835_gpio.o
  CC      aarch64-softmmu/hw/display/xlnx_dp.o
  CC      alpha-softmmu/hw/virtio/vhost-backend.o
  CC      arm-softmmu/hw/i2c/omap_i2c.o
  CC      alpha-softmmu/hw/virtio/vhost-user.o
  CC      aarch64-softmmu/hw/dma/xlnx_dpdma.o
  CC      arm-softmmu/hw/input/pxa2xx_keypad.o
  CC      alpha-softmmu/hw/virtio/vhost-vsock.o
  CC      i386-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/hw/dma/omap_dma.o
  CC      arm-softmmu/hw/input/tsc210x.o
  CC      alpha-softmmu/hw/virtio/virtio-crypto.o
  CC      i386-softmmu/tcg/tcg-runtime.o
  CC      alpha-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      aarch64-softmmu/hw/dma/soc_dma.o
  CC      i386-softmmu/fpu/softfloat.o
  CC      arm-softmmu/hw/intc/armv7m_nvic.o
  CC      aarch64-softmmu/hw/dma/pxa2xx_dma.o
  CC      alpha-softmmu/hw/alpha/dp264.o
  CC      arm-softmmu/hw/intc/exynos4210_gic.o
  CC      aarch64-softmmu/hw/dma/bcm2835_dma.o
  CC      alpha-softmmu/hw/alpha/pci.o
  CC      aarch64-softmmu/hw/gpio/omap_gpio.o
  CC      arm-softmmu/hw/intc/exynos4210_combiner.o
  CC      alpha-softmmu/hw/alpha/typhoon.o
  CC      aarch64-softmmu/hw/gpio/imx_gpio.o
  CC      arm-softmmu/hw/intc/omap_intc.o
  CC      aarch64-softmmu/hw/gpio/bcm2835_gpio.o
  CC      aarch64-softmmu/hw/i2c/omap_i2c.o
  CC      alpha-softmmu/target/alpha/machine.o
  CC      arm-softmmu/hw/intc/bcm2835_ic.o
  CC      aarch64-softmmu/hw/input/pxa2xx_keypad.o
  CC      arm-softmmu/hw/intc/bcm2836_control.o
  CC      alpha-softmmu/target/alpha/translate.o
  CC      aarch64-softmmu/hw/input/tsc210x.o
  CC      arm-softmmu/hw/intc/allwinner-a10-pic.o
  CC      i386-softmmu/disas.o
  CC      arm-softmmu/hw/intc/aspeed_vic.o
  GEN     i386-softmmu/gdbstub-xml.c
  CC      aarch64-softmmu/hw/intc/armv7m_nvic.o
  CC      arm-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      alpha-softmmu/target/alpha/helper.o
  CC      i386-softmmu/hax-stub.o
  CC      aarch64-softmmu/hw/intc/exynos4210_gic.o
  CC      alpha-softmmu/target/alpha/cpu.o
  CC      i386-softmmu/arch_init.o
  CC      aarch64-softmmu/hw/intc/exynos4210_combiner.o
  CC      alpha-softmmu/target/alpha/int_helper.o
  CC      i386-softmmu/cpus.o
  CC      arm-softmmu/hw/misc/ivshmem.o
  CC      aarch64-softmmu/hw/intc/omap_intc.o
  CC      alpha-softmmu/target/alpha/fpu_helper.o
  CC      i386-softmmu/monitor.o
  CC      alpha-softmmu/target/alpha/vax_helper.o
  CC      arm-softmmu/hw/misc/arm_sysctl.o
  CC      aarch64-softmmu/hw/intc/bcm2835_ic.o
  CC      alpha-softmmu/target/alpha/sys_helper.o
  CC      aarch64-softmmu/hw/intc/bcm2836_control.o
  CC      arm-softmmu/hw/misc/cbus.o
  CC      alpha-softmmu/target/alpha/mem_helper.o
  CC      aarch64-softmmu/hw/intc/allwinner-a10-pic.o
  CC      arm-softmmu/hw/misc/exynos4210_pmu.o
  CC      alpha-softmmu/target/alpha/gdbstub.o
  CC      arm-softmmu/hw/misc/exynos4210_clk.o
  GEN     trace/generated-helpers.c
  CC      alpha-softmmu/trace/control-target.o
  CC      aarch64-softmmu/hw/intc/aspeed_vic.o
  CC      arm-softmmu/hw/misc/exynos4210_rng.o
  CC      i386-softmmu/gdbstub.o
  CC      aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      arm-softmmu/hw/misc/imx_ccm.o
  CC      alpha-softmmu/trace/generated-helpers.o
  CC      arm-softmmu/hw/misc/imx31_ccm.o
  LINK    alpha-softmmu/qemu-system-alpha
  CC      i386-softmmu/balloon.o
  CC      arm-softmmu/hw/misc/imx25_ccm.o
  CC      i386-softmmu/ioport.o
  CC      arm-softmmu/hw/misc/imx6_ccm.o
  CC      aarch64-softmmu/hw/misc/ivshmem.o
  CC      i386-softmmu/numa.o
  CC      arm-softmmu/hw/misc/imx6_src.o
  CC      aarch64-softmmu/hw/misc/arm_sysctl.o
  CC      arm-softmmu/hw/misc/mst_fpga.o
  CC      aarch64-softmmu/hw/misc/cbus.o
  CC      i386-softmmu/qtest.o
  CC      arm-softmmu/hw/misc/omap_clk.o
  CC      aarch64-softmmu/hw/misc/exynos4210_pmu.o
  CC      aarch64-softmmu/hw/misc/exynos4210_clk.o
  CC      i386-softmmu/memory.o
  CC      arm-softmmu/hw/misc/omap_gpmc.o
  GEN     lm32-softmmu/hmp-commands.h
  GEN     lm32-softmmu/hmp-commands-info.h
  GEN     lm32-softmmu/config-target.h
  CC      lm32-softmmu/exec.o
  CC      aarch64-softmmu/hw/misc/exynos4210_rng.o
  CC      arm-softmmu/hw/misc/omap_l4.o
  CC      arm-softmmu/hw/misc/omap_sdrc.o
  CC      aarch64-softmmu/hw/misc/imx_ccm.o
  CC      arm-softmmu/hw/misc/omap_tap.o
  CC      aarch64-softmmu/hw/misc/imx31_ccm.o
  CC      arm-softmmu/hw/misc/bcm2835_mbox.o
  CC      i386-softmmu/memory_mapping.o
  CC      aarch64-softmmu/hw/misc/imx25_ccm.o
  CC      arm-softmmu/hw/misc/bcm2835_property.o
  CC      i386-softmmu/dump.o
  CC      aarch64-softmmu/hw/misc/imx6_ccm.o
  CC      arm-softmmu/hw/misc/bcm2835_rng.o
  CC      lm32-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/hw/misc/imx6_src.o
  CC      arm-softmmu/hw/misc/zynq_slcr.o
  CC      i386-softmmu/migration/ram.o
  CC      aarch64-softmmu/hw/misc/mst_fpga.o
  CC      arm-softmmu/hw/misc/zynq-xadc.o
  CC      aarch64-softmmu/hw/misc/omap_clk.o
  CC      arm-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      i386-softmmu/accel/accel.o
  CC      aarch64-softmmu/hw/misc/omap_gpmc.o
  CC      i386-softmmu/accel/stubs/kvm-stub.o
  CC      arm-softmmu/hw/misc/mps2-scc.o
  CC      lm32-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/hw/misc/omap_l4.o
  CC      arm-softmmu/hw/misc/aspeed_scu.o
  CC      i386-softmmu/accel/tcg/tcg-all.o
  CC      aarch64-softmmu/hw/misc/omap_sdrc.o
  CC      i386-softmmu/accel/tcg/cputlb.o
  CC      arm-softmmu/hw/misc/aspeed_sdmc.o
  CC      aarch64-softmmu/hw/misc/omap_tap.o
  CC      arm-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/misc/bcm2835_mbox.o
  CC      aarch64-softmmu/hw/misc/bcm2835_property.o
  CC      arm-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/net/vhost_net.o
  CC      lm32-softmmu/tcg/optimize.o
  CC      aarch64-softmmu/hw/misc/bcm2835_rng.o
  CC      arm-softmmu/hw/pcmcia/pxa2xx.o
  CC      aarch64-softmmu/hw/misc/zynq_slcr.o
  CC      i386-softmmu/accel/tcg/cpu-exec.o
  CC      arm-softmmu/hw/scsi/virtio-scsi.o
  CC      aarch64-softmmu/hw/misc/zynq-xadc.o
  CC      i386-softmmu/accel/tcg/cpu-exec-common.o
  CC      aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      lm32-softmmu/tcg/tcg-common.o
  CC      arm-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/misc/mps2-scc.o
  CC      i386-softmmu/accel/tcg/translate-all.o
  CC      lm32-softmmu/tcg/tcg-runtime.o
  CC      aarch64-softmmu/hw/misc/auxbus.o
  CC      arm-softmmu/hw/scsi/vhost-scsi-common.o
  CC      lm32-softmmu/fpu/softfloat.o
  CC      i386-softmmu/hw/9pfs/virtio-9p-device.o
  CC      aarch64-softmmu/hw/misc/aspeed_scu.o
  CC      arm-softmmu/hw/scsi/vhost-scsi.o
  CC      i386-softmmu/hw/block/virtio-blk.o
  CC      aarch64-softmmu/hw/misc/aspeed_sdmc.o
  CC      arm-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/misc/mmio_interface.o
  CC      i386-softmmu/hw/block/dataplane/virtio-blk.o
  CC      aarch64-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/sd/omap_mmc.o
  CC      i386-softmmu/hw/char/virtio-serial-bus.o
  CC      arm-softmmu/hw/sd/pxa2xx_mmci.o
  CC      i386-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/sd/bcm2835_sdhost.o
  CC      aarch64-softmmu/hw/net/vhost_net.o
  CC      i386-softmmu/hw/core/null-machine.o
  CC      arm-softmmu/hw/ssi/omap_spi.o
  CC      aarch64-softmmu/hw/pcmcia/pxa2xx.o
  CC      i386-softmmu/hw/display/vga.o
  CC      lm32-softmmu/disas.o
  CC      arm-softmmu/hw/ssi/imx_spi.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi.o
  CC      lm32-softmmu/hax-stub.o
  CC      arm-softmmu/hw/timer/exynos4210_mct.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      lm32-softmmu/arch_init.o
  CC      arm-softmmu/hw/timer/exynos4210_pwm.o
  CC      lm32-softmmu/cpus.o
  CC      i386-softmmu/hw/display/virtio-gpu.o
  CC      arm-softmmu/hw/timer/exynos4210_rtc.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi.o
  CC      arm-softmmu/hw/timer/omap_gptimer.o
  CC      lm32-softmmu/monitor.o
  CC      i386-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      arm-softmmu/hw/timer/omap_synctimer.o
  CC      aarch64-softmmu/hw/sd/omap_mmc.o
  CC      arm-softmmu/hw/timer/pxa2xx_timer.o
  CC      aarch64-softmmu/hw/sd/pxa2xx_mmci.o
  CC      i386-softmmu/hw/display/virtio-gpu-pci.o
  CC      arm-softmmu/hw/timer/digic-timer.o
  CC      aarch64-softmmu/hw/sd/bcm2835_sdhost.o
  CC      lm32-softmmu/gdbstub.o
  CC      i386-softmmu/hw/display/virtio-vga.o
  CC      arm-softmmu/hw/timer/allwinner-a10-pit.o
  CC      aarch64-softmmu/hw/ssi/omap_spi.o
  CC      arm-softmmu/hw/usb/tusb6010.o
  CC      aarch64-softmmu/hw/ssi/imx_spi.o
  CC      i386-softmmu/hw/intc/apic.o
  CC      lm32-softmmu/balloon.o
  CC      lm32-softmmu/ioport.o
  CC      aarch64-softmmu/hw/timer/exynos4210_mct.o
  CC      arm-softmmu/hw/vfio/common.o
  CC      lm32-softmmu/numa.o
  CC      i386-softmmu/hw/intc/apic_common.o
  CC      aarch64-softmmu/hw/timer/exynos4210_pwm.o
  CC      arm-softmmu/hw/vfio/pci.o
  CC      lm32-softmmu/qtest.o
  CC      i386-softmmu/hw/intc/ioapic.o
  CC      aarch64-softmmu/hw/timer/exynos4210_rtc.o
  CC      aarch64-softmmu/hw/timer/omap_gptimer.o
  CC      lm32-softmmu/memory.o
  CC      i386-softmmu/hw/isa/lpc_ich9.o
  CC      aarch64-softmmu/hw/timer/omap_synctimer.o
  CC      arm-softmmu/hw/vfio/pci-quirks.o
  CC      i386-softmmu/hw/misc/vmport.o
  CC      aarch64-softmmu/hw/timer/pxa2xx_timer.o
  CC      i386-softmmu/hw/misc/ivshmem.o
  CC      arm-softmmu/hw/vfio/platform.o
  CC      lm32-softmmu/memory_mapping.o
  CC      aarch64-softmmu/hw/timer/digic-timer.o
  CC      i386-softmmu/hw/misc/pvpanic.o
  CC      aarch64-softmmu/hw/timer/allwinner-a10-pit.o
  CC      arm-softmmu/hw/vfio/calxeda-xgmac.o
  CC      lm32-softmmu/dump.o
  CC      i386-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/usb/tusb6010.o
  CC      arm-softmmu/hw/vfio/amd-xgbe.o
  CC      i386-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/vfio/spapr.o
  CC      aarch64-softmmu/hw/vfio/common.o
  CC      lm32-softmmu/migration/ram.o
  CC      arm-softmmu/hw/virtio/virtio.o
  CC      i386-softmmu/hw/net/vhost_net.o
  CC      aarch64-softmmu/hw/vfio/pci.o
  CC      lm32-softmmu/accel/accel.o
  CC      i386-softmmu/hw/scsi/virtio-scsi.o
  CC      lm32-softmmu/accel/stubs/kvm-stub.o
  CC      arm-softmmu/hw/virtio/virtio-balloon.o
  CC      i386-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      i386-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/vfio/pci-quirks.o
  CC      lm32-softmmu/accel/tcg/tcg-all.o
  CC      arm-softmmu/hw/virtio/vhost.o
  CC      i386-softmmu/hw/scsi/vhost-scsi.o
  CC      lm32-softmmu/accel/tcg/cputlb.o
  CC      aarch64-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/virtio/vhost-backend.o
  CC      i386-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/vfio/calxeda-xgmac.o
  CC      arm-softmmu/hw/virtio/vhost-user.o
  CC      i386-softmmu/hw/timer/mc146818rtc.o
  CC      aarch64-softmmu/hw/vfio/amd-xgbe.o
  CC      arm-softmmu/hw/virtio/vhost-vsock.o
  CC      lm32-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/hw/vfio/spapr.o
  CC      i386-softmmu/hw/vfio/common.o
  CC      arm-softmmu/hw/virtio/virtio-crypto.o
  CC      aarch64-softmmu/hw/virtio/virtio.o
  CC      lm32-softmmu/accel/tcg/cpu-exec-common.o
  CC      arm-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      i386-softmmu/hw/vfio/pci.o
  CC      lm32-softmmu/accel/tcg/translate-all.o
  CC      arm-softmmu/hw/arm/boot.o
  CC      aarch64-softmmu/hw/virtio/virtio-balloon.o
  CC      lm32-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/arm/collie.o
  CC      i386-softmmu/hw/vfio/pci-quirks.o
  CC      lm32-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/hw/virtio/vhost.o
  CC      arm-softmmu/hw/arm/exynos4_boards.o
  CC      lm32-softmmu/hw/input/milkymist-softusb.o
  CC      arm-softmmu/hw/arm/gumstix.o
  CC      aarch64-softmmu/hw/virtio/vhost-backend.o
  CC      i386-softmmu/hw/vfio/platform.o
  CC      lm32-softmmu/hw/misc/milkymist-hpdmc.o
  CC      arm-softmmu/hw/arm/highbank.o
  CC      lm32-softmmu/hw/misc/milkymist-pfpu.o
  CC      aarch64-softmmu/hw/virtio/vhost-user.o
  CC      i386-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/hw/arm/digic_boards.o
  CC      lm32-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/virtio/vhost-vsock.o
  CC      i386-softmmu/hw/virtio/virtio.o
  CC      arm-softmmu/hw/arm/integratorcp.o
  CC      lm32-softmmu/hw/net/milkymist-minimac2.o
  CC      arm-softmmu/hw/arm/mainstone.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto.o
  CC      lm32-softmmu/hw/net/vhost_net.o
  CC      lm32-softmmu/hw/net/rocker/qmp-norocker.o
  CC      arm-softmmu/hw/arm/musicpal.o
  CC      i386-softmmu/hw/virtio/virtio-balloon.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      lm32-softmmu/hw/sd/milkymist-memcard.o
  CC      i386-softmmu/hw/virtio/vhost.o
  CC      lm32-softmmu/hw/vfio/common.o
  CC      aarch64-softmmu/hw/arm/boot.o
  CC      arm-softmmu/hw/arm/nseries.o
  CC      i386-softmmu/hw/virtio/vhost-backend.o
  CC      aarch64-softmmu/hw/arm/collie.o
  CC      lm32-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/arm/omap_sx1.o
  CC      i386-softmmu/hw/virtio/vhost-user.o
  CC      aarch64-softmmu/hw/arm/exynos4_boards.o
  CC      lm32-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/hw/arm/palm.o
  CC      aarch64-softmmu/hw/arm/gumstix.o
  CC      i386-softmmu/hw/virtio/vhost-vsock.o
  CC      aarch64-softmmu/hw/arm/highbank.o
  CC      lm32-softmmu/hw/lm32/lm32_boards.o
  CC      arm-softmmu/hw/arm/realview.o
  CC      i386-softmmu/hw/virtio/virtio-crypto.o
  CC      lm32-softmmu/hw/lm32/milkymist.o
  CC      aarch64-softmmu/hw/arm/digic_boards.o
  CC      arm-softmmu/hw/arm/spitz.o
  CC      aarch64-softmmu/hw/arm/integratorcp.o
  CC      i386-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      lm32-softmmu/target/lm32/translate.o
  CC      aarch64-softmmu/hw/arm/mainstone.o
  CC      arm-softmmu/hw/arm/stellaris.o
  CC      aarch64-softmmu/hw/arm/musicpal.o
  CC      i386-softmmu/hw/i386/multiboot.o
  CC      lm32-softmmu/target/lm32/op_helper.o
  CC      lm32-softmmu/target/lm32/helper.o
  CC      i386-softmmu/hw/i386/pc.o
  CC      aarch64-softmmu/hw/arm/nseries.o
  CC      arm-softmmu/hw/arm/tosa.o
  CC      lm32-softmmu/target/lm32/cpu.o
  CC      arm-softmmu/hw/arm/versatilepb.o
  CC      lm32-softmmu/target/lm32/gdbstub.o
  CC      aarch64-softmmu/hw/arm/omap_sx1.o
  CC      arm-softmmu/hw/arm/vexpress.o
  CC      lm32-softmmu/target/lm32/lm32-semi.o
  CC      i386-softmmu/hw/i386/pc_piix.o
  CC      lm32-softmmu/target/lm32/machine.o
  CC      arm-softmmu/hw/arm/virt.o
  CC      aarch64-softmmu/hw/arm/palm.o
  GEN     trace/generated-helpers.c
  CC      lm32-softmmu/trace/control-target.o
  CC      i386-softmmu/hw/i386/pc_q35.o
  CC      aarch64-softmmu/hw/arm/realview.o
  CC      lm32-softmmu/trace/generated-helpers.o
  CC      arm-softmmu/hw/arm/xilinx_zynq.o
  CC      i386-softmmu/hw/i386/pc_sysfw.o
  CC      aarch64-softmmu/hw/arm/spitz.o
  LINK    lm32-softmmu/qemu-system-lm32
  CC      arm-softmmu/hw/arm/z2.o
  CC      i386-softmmu/hw/i386/x86-iommu.o
  CC      i386-softmmu/hw/i386/intel_iommu.o
  CC      arm-softmmu/hw/arm/virt-acpi-build.o
  CC      aarch64-softmmu/hw/arm/stellaris.o
  CC      arm-softmmu/hw/arm/netduino2.o
  CC      arm-softmmu/hw/arm/sysbus-fdt.o
  CC      aarch64-softmmu/hw/arm/tosa.o
  CC      arm-softmmu/hw/arm/armv7m.o
  CC      aarch64-softmmu/hw/arm/versatilepb.o
  CC      i386-softmmu/hw/i386/amd_iommu.o
  GEN     m68k-softmmu/hmp-commands.h
  CC      arm-softmmu/hw/arm/exynos4210.o
  GEN     m68k-softmmu/hmp-commands-info.h
  GEN     m68k-softmmu/config-target.h
  CC      aarch64-softmmu/hw/arm/vexpress.o
  CC      m68k-softmmu/exec.o
  CC      arm-softmmu/hw/arm/pxa2xx.o
  CC      i386-softmmu/hw/i386/kvmvapic.o
  CC      aarch64-softmmu/hw/arm/virt.o
  CC      arm-softmmu/hw/arm/pxa2xx_gpio.o
  CC      aarch64-softmmu/hw/arm/xilinx_zynq.o
  CC      i386-softmmu/hw/i386/acpi-build.o
  CC      arm-softmmu/hw/arm/pxa2xx_pic.o
  CC      aarch64-softmmu/hw/arm/z2.o
  CC      arm-softmmu/hw/arm/digic.o
  CC      i386-softmmu/hw/i386/pci-assign-load-rom.o
  CC      arm-softmmu/hw/arm/omap1.o
  CC      m68k-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/hw/arm/virt-acpi-build.o
  CC      i386-softmmu/target/i386/helper.o
  CC      aarch64-softmmu/hw/arm/netduino2.o
  CC      aarch64-softmmu/hw/arm/sysbus-fdt.o
  CC      i386-softmmu/target/i386/cpu.o
  CC      arm-softmmu/hw/arm/omap2.o
  CC      aarch64-softmmu/hw/arm/armv7m.o
  CC      m68k-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/hw/arm/exynos4210.o
  CC      i386-softmmu/target/i386/gdbstub.o
  CC      arm-softmmu/hw/arm/strongarm.o
  CC      aarch64-softmmu/hw/arm/pxa2xx.o
  CC      i386-softmmu/target/i386/xsave_helper.o
  CC      arm-softmmu/hw/arm/allwinner-a10.o
  CC      i386-softmmu/target/i386/translate.o
  CC      arm-softmmu/hw/arm/cubieboard.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_gpio.o
  CC      arm-softmmu/hw/arm/bcm2835_peripherals.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_pic.o
  CC      m68k-softmmu/tcg/optimize.o
  CC      arm-softmmu/hw/arm/bcm2836.o
  CC      aarch64-softmmu/hw/arm/digic.o
  CC      aarch64-softmmu/hw/arm/omap1.o
  CC      arm-softmmu/hw/arm/raspi.o
  CC      m68k-softmmu/tcg/tcg-common.o
  CC      m68k-softmmu/tcg/tcg-runtime.o
  CC      arm-softmmu/hw/arm/stm32f205_soc.o
  CC      m68k-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/hw/arm/omap2.o
  CC      arm-softmmu/hw/arm/fsl-imx25.o
  CC      arm-softmmu/hw/arm/imx25_pdk.o
  CC      aarch64-softmmu/hw/arm/strongarm.o
  CC      arm-softmmu/hw/arm/fsl-imx31.o
  CC      arm-softmmu/hw/arm/kzm.o
  CC      aarch64-softmmu/hw/arm/allwinner-a10.o
  CC      arm-softmmu/hw/arm/fsl-imx6.o
  CC      arm-softmmu/hw/arm/sabrelite.o
  CC      aarch64-softmmu/hw/arm/cubieboard.o
  CC      arm-softmmu/hw/arm/aspeed_soc.o
  CC      m68k-softmmu/disas.o
  CC      aarch64-softmmu/hw/arm/bcm2835_peripherals.o
  CC      i386-softmmu/target/i386/bpt_helper.o
  CC      arm-softmmu/hw/arm/aspeed.o
  GEN     m68k-softmmu/gdbstub-xml.c
  CC      i386-softmmu/target/i386/cc_helper.o
  CC      m68k-softmmu/hax-stub.o
  CC      arm-softmmu/hw/arm/mps2.o
  CC      aarch64-softmmu/hw/arm/bcm2836.o
  CC      i386-softmmu/target/i386/excp_helper.o
  CC      arm-softmmu/target/arm/arm-semi.o
  CC      m68k-softmmu/arch_init.o
  CC      aarch64-softmmu/hw/arm/raspi.o
  CC      i386-softmmu/target/i386/fpu_helper.o
  CC      m68k-softmmu/cpus.o
  CC      arm-softmmu/target/arm/machine.o
  CC      aarch64-softmmu/hw/arm/stm32f205_soc.o
  CC      arm-softmmu/target/arm/psci.o
  CC      m68k-softmmu/monitor.o
  CC      aarch64-softmmu/hw/arm/xlnx-zynqmp.o
  CC      arm-softmmu/target/arm/arch_dump.o
  CC      arm-softmmu/target/arm/monitor.o
  CC      aarch64-softmmu/hw/arm/xlnx-ep108.o
  CC      arm-softmmu/target/arm/kvm-stub.o
  CC      aarch64-softmmu/hw/arm/fsl-imx25.o
  CC      arm-softmmu/target/arm/translate.o
  CC      m68k-softmmu/gdbstub.o
  CC      aarch64-softmmu/hw/arm/imx25_pdk.o
  CC      i386-softmmu/target/i386/int_helper.o
  CC      aarch64-softmmu/hw/arm/fsl-imx31.o
  CC      i386-softmmu/target/i386/mem_helper.o
  CC      m68k-softmmu/balloon.o
  CC      aarch64-softmmu/hw/arm/kzm.o
  CC      i386-softmmu/target/i386/misc_helper.o
  CC      m68k-softmmu/ioport.o
  CC      aarch64-softmmu/hw/arm/fsl-imx6.o
  CC      m68k-softmmu/numa.o
  CC      i386-softmmu/target/i386/mpx_helper.o
  CC      aarch64-softmmu/hw/arm/sabrelite.o
  CC      m68k-softmmu/qtest.o
  CC      aarch64-softmmu/hw/arm/aspeed_soc.o
  CC      i386-softmmu/target/i386/seg_helper.o
  CC      m68k-softmmu/memory.o
  CC      aarch64-softmmu/hw/arm/aspeed.o
  CC      aarch64-softmmu/hw/arm/mps2.o
  CC      m68k-softmmu/memory_mapping.o
  CC      aarch64-softmmu/target/arm/arm-semi.o
  CC      m68k-softmmu/dump.o
  CC      aarch64-softmmu/target/arm/machine.o
  CC      i386-softmmu/target/i386/smm_helper.o
  CC      aarch64-softmmu/target/arm/psci.o
  CC      m68k-softmmu/migration/ram.o
  CC      i386-softmmu/target/i386/svm_helper.o
  CC      aarch64-softmmu/target/arm/arch_dump.o
  CC      i386-softmmu/target/i386/machine.o
  CC      arm-softmmu/target/arm/op_helper.o
  CC      aarch64-softmmu/target/arm/monitor.o
  CC      i386-softmmu/target/i386/arch_memory_mapping.o
  CC      m68k-softmmu/accel/accel.o
  CC      aarch64-softmmu/target/arm/kvm-stub.o
  CC      i386-softmmu/target/i386/arch_dump.o
  CC      m68k-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/target/arm/translate.o
  CC      arm-softmmu/target/arm/helper.o
  CC      i386-softmmu/target/i386/monitor.o
  CC      m68k-softmmu/accel/tcg/tcg-all.o
  CC      i386-softmmu/target/i386/kvm-stub.o
  CC      m68k-softmmu/accel/tcg/cputlb.o
  GEN     trace/generated-helpers.c
  CC      i386-softmmu/trace/control-target.o
  CC      i386-softmmu/gdbstub-xml.o
  CC      i386-softmmu/trace/generated-helpers.o
  CC      m68k-softmmu/accel/tcg/cpu-exec.o
  LINK    i386-softmmu/qemu-system-i386
  CC      m68k-softmmu/accel/tcg/cpu-exec-common.o
  CC      arm-softmmu/target/arm/cpu.o
  CC      m68k-softmmu/accel/tcg/translate-all.o
  CC      m68k-softmmu/hw/char/mcf_uart.o
  CC      m68k-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/target/arm/neon_helper.o
  CC      m68k-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/target/arm/op_helper.o
  CC      m68k-softmmu/hw/misc/mmio_interface.o
  CC      m68k-softmmu/hw/net/mcf_fec.o
  CC      arm-softmmu/target/arm/iwmmxt_helper.o
  CC      m68k-softmmu/hw/net/vhost_net.o
  CC      m68k-softmmu/hw/net/rocker/qmp-norocker.o
  CC      aarch64-softmmu/target/arm/helper.o
  CC      m68k-softmmu/hw/vfio/common.o
  CC      m68k-softmmu/hw/vfio/platform.o
  CC      m68k-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/target/arm/gdbstub.o
  CC      aarch64-softmmu/target/arm/cpu.o
  CC      m68k-softmmu/hw/m68k/an5206.o
  CC      arm-softmmu/target/arm/crypto_helper.o
  CC      m68k-softmmu/hw/m68k/mcf5208.o
  CC      arm-softmmu/target/arm/arm-powerctl.o
  GEN     trace/generated-helpers.c
  CC      arm-softmmu/trace/control-target.o
  CC      m68k-softmmu/hw/m68k/mcf5206.o
  CC      arm-softmmu/gdbstub-xml.o
  CC      aarch64-softmmu/target/arm/neon_helper.o
  CC      arm-softmmu/trace/generated-helpers.o
  CC      m68k-softmmu/hw/m68k/mcf_intc.o
  LINK    arm-softmmu/qemu-system-arm
  CC      m68k-softmmu/target/m68k/m68k-semi.o
  CC      aarch64-softmmu/target/arm/iwmmxt_helper.o
  CC      m68k-softmmu/target/m68k/translate.o
  CC      aarch64-softmmu/target/arm/gdbstub.o
  CC      m68k-softmmu/target/m68k/op_helper.o
  CC      aarch64-softmmu/target/arm/cpu64.o
  CC      aarch64-softmmu/target/arm/translate-a64.o
  CC      aarch64-softmmu/target/arm/helper-a64.o
  CC      aarch64-softmmu/target/arm/gdbstub64.o
  CC      m68k-softmmu/target/m68k/helper.o
  CC      m68k-softmmu/target/m68k/cpu.o
  CC      m68k-softmmu/target/m68k/fpu_helper.o
  CC      m68k-softmmu/target/m68k/gdbstub.o
  CC      aarch64-softmmu/target/arm/crypto_helper.o
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/target/arm/arm-powerctl.o
  CC      m68k-softmmu/trace/control-target.o
  CC      m68k-softmmu/gdbstub-xml.o
  CC      m68k-softmmu/trace/generated-helpers.o
  GEN     microblazeel-softmmu/hmp-commands.h
  GEN     microblaze-softmmu/hmp-commands.h
  GEN     microblazeel-softmmu/hmp-commands-info.h
  GEN     microblazeel-softmmu/config-target.h
  GEN     microblaze-softmmu/hmp-commands-info.h
  GEN     microblaze-softmmu/config-target.h
  CC      microblaze-softmmu/exec.o
  CC      microblazeel-softmmu/exec.o
  LINK    m68k-softmmu/qemu-system-m68k
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/trace/control-target.o
  CC      microblazeel-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/gdbstub-xml.o
  CC      aarch64-softmmu/trace/generated-helpers.o
  CC      microblazeel-softmmu/tcg/tcg-op.o
  CC      microblaze-softmmu/tcg/tcg.o
  LINK    aarch64-softmmu/qemu-system-aarch64
  CC      microblazeel-softmmu/tcg/optimize.o
  CC      microblazeel-softmmu/tcg/tcg-common.o
  CC      microblaze-softmmu/tcg/tcg-op.o
  CC      microblazeel-softmmu/tcg/tcg-runtime.o
  GEN     mips64el-softmmu/hmp-commands.h
  GEN     mips64el-softmmu/hmp-commands-info.h
  GEN     mips64el-softmmu/config-target.h
  CC      mips64el-softmmu/exec.o
  CC      microblazeel-softmmu/fpu/softfloat.o
  GEN     mips64-softmmu/hmp-commands.h
  GEN     mips64-softmmu/hmp-commands-info.h
  GEN     mips64-softmmu/config-target.h
  CC      mips64-softmmu/exec.o
  CC      microblaze-softmmu/tcg/optimize.o
  CC      mips64el-softmmu/tcg/tcg.o
  CC      microblazeel-softmmu/disas.o
  CC      microblaze-softmmu/tcg/tcg-common.o
  CC      microblazeel-softmmu/hax-stub.o
  CC      microblaze-softmmu/tcg/tcg-runtime.o
  CC      microblazeel-softmmu/arch_init.o
  CC      microblazeel-softmmu/cpus.o
  CC      microblaze-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/tcg/tcg.o
  CC      microblazeel-softmmu/monitor.o
  CC      mips64el-softmmu/tcg/tcg-op.o
  CC      mips64-softmmu/tcg/tcg-op.o
  CC      microblaze-softmmu/disas.o
  CC      microblazeel-softmmu/gdbstub.o
  CC      microblaze-softmmu/hax-stub.o
  CC      microblaze-softmmu/arch_init.o
  CC      microblaze-softmmu/cpus.o
  CC      microblazeel-softmmu/balloon.o
  CC      mips64el-softmmu/tcg/optimize.o
  CC      microblazeel-softmmu/ioport.o
  CC      microblaze-softmmu/monitor.o
  CC      microblazeel-softmmu/numa.o
  CC      microblazeel-softmmu/qtest.o
  CC      mips64el-softmmu/tcg/tcg-common.o
  CC      mips64-softmmu/tcg/optimize.o
  CC      mips64el-softmmu/tcg/tcg-runtime.o
  CC      microblazeel-softmmu/memory.o
  CC      microblaze-softmmu/gdbstub.o
  CC      mips64el-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/tcg/tcg-common.o
  CC      microblaze-softmmu/balloon.o
  CC      mips64-softmmu/tcg/tcg-runtime.o
  CC      microblazeel-softmmu/memory_mapping.o
  CC      microblaze-softmmu/ioport.o
  CC      microblazeel-softmmu/dump.o
  CC      microblaze-softmmu/numa.o
  CC      mips64-softmmu/fpu/softfloat.o
  CC      microblaze-softmmu/qtest.o
  CC      microblazeel-softmmu/migration/ram.o
  CC      microblaze-softmmu/memory.o
  CC      mips64el-softmmu/disas.o
  CC      microblazeel-softmmu/accel/accel.o
  CC      mips64el-softmmu/hax-stub.o
  CC      microblazeel-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/accel/tcg/tcg-all.o
  CC      mips64el-softmmu/arch_init.o
  CC      microblaze-softmmu/memory_mapping.o
  CC      mips64el-softmmu/cpus.o
  CC      microblazeel-softmmu/accel/tcg/cputlb.o
  CC      mips64-softmmu/disas.o
  CC      microblaze-softmmu/dump.o
  CC      mips64-softmmu/hax-stub.o
  CC      mips64el-softmmu/monitor.o
  CC      mips64-softmmu/arch_init.o
  CC      microblaze-softmmu/migration/ram.o
  CC      mips64-softmmu/cpus.o
  CC      microblazeel-softmmu/accel/tcg/cpu-exec.o
  CC      mips64-softmmu/monitor.o
  CC      microblaze-softmmu/accel/accel.o
  CC      mips64el-softmmu/gdbstub.o
  CC      microblazeel-softmmu/accel/tcg/cpu-exec-common.o
  CC      microblaze-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/accel/tcg/translate-all.o
  CC      microblaze-softmmu/accel/tcg/tcg-all.o
  CC      mips64el-softmmu/balloon.o
  CC      mips64el-softmmu/ioport.o
  CC      microblaze-softmmu/accel/tcg/cputlb.o
  CC      microblazeel-softmmu/hw/core/generic-loader.o
  CC      mips64el-softmmu/numa.o
  CC      microblazeel-softmmu/hw/core/null-machine.o
  CC      mips64-softmmu/gdbstub.o
  CC      mips64el-softmmu/qtest.o
  CC      microblazeel-softmmu/hw/misc/mmio_interface.o
  CC      microblazeel-softmmu/hw/net/xilinx_ethlite.o
  CC      mips64-softmmu/balloon.o
  CC      mips64el-softmmu/memory.o
  CC      microblazeel-softmmu/hw/net/vhost_net.o
  CC      mips64-softmmu/ioport.o
  CC      microblaze-softmmu/accel/tcg/cpu-exec.o
  CC      microblazeel-softmmu/hw/net/rocker/qmp-norocker.o
  CC      mips64-softmmu/numa.o
  CC      microblaze-softmmu/accel/tcg/cpu-exec-common.o
  CC      microblazeel-softmmu/hw/vfio/common.o
  CC      microblaze-softmmu/accel/tcg/translate-all.o
  CC      mips64-softmmu/qtest.o
  CC      mips64el-softmmu/memory_mapping.o
  CC      microblazeel-softmmu/hw/vfio/platform.o
  CC      mips64el-softmmu/dump.o
  CC      mips64-softmmu/memory.o
  CC      microblaze-softmmu/hw/core/generic-loader.o
  CC      microblaze-softmmu/hw/core/null-machine.o
  CC      microblazeel-softmmu/hw/vfio/spapr.o
  CC      mips64el-softmmu/migration/ram.o
  CC      microblaze-softmmu/hw/misc/mmio_interface.o
  CC      microblazeel-softmmu/hw/microblaze/petalogix_s3adsp1800_mmu.o
  CC      microblaze-softmmu/hw/net/xilinx_ethlite.o
  CC      microblazeel-softmmu/hw/microblaze/petalogix_ml605_mmu.o
  CC      mips64-softmmu/memory_mapping.o
  CC      microblaze-softmmu/hw/net/vhost_net.o
  CC      mips64el-softmmu/accel/accel.o
  CC      mips64-softmmu/dump.o
  CC      microblazeel-softmmu/hw/microblaze/boot.o
  CC      microblaze-softmmu/hw/net/rocker/qmp-norocker.o
  CC      microblaze-softmmu/hw/vfio/common.o
  CC      mips64el-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/target/microblaze/translate.o
  CC      mips64el-softmmu/accel/tcg/tcg-all.o
  CC      mips64-softmmu/migration/ram.o
  CC      mips64el-softmmu/accel/tcg/cputlb.o
  CC      microblaze-softmmu/hw/vfio/platform.o
  CC      microblazeel-softmmu/target/microblaze/op_helper.o
  CC      microblaze-softmmu/hw/vfio/spapr.o
  CC      mips64-softmmu/accel/accel.o
  CC      mips64-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/target/microblaze/helper.o
  CC      microblaze-softmmu/hw/microblaze/petalogix_s3adsp1800_mmu.o
  CC      mips64el-softmmu/accel/tcg/cpu-exec.o
  CC      mips64-softmmu/accel/tcg/tcg-all.o
  CC      microblaze-softmmu/hw/microblaze/petalogix_ml605_mmu.o
  CC      microblazeel-softmmu/target/microblaze/cpu.o
  CC      mips64-softmmu/accel/tcg/cputlb.o
  CC      mips64el-softmmu/accel/tcg/cpu-exec-common.o
  CC      microblaze-softmmu/hw/microblaze/boot.o
  CC      microblazeel-softmmu/target/microblaze/gdbstub.o
  CC      mips64el-softmmu/accel/tcg/translate-all.o
  CC      microblaze-softmmu/target/microblaze/translate.o
  CC      microblazeel-softmmu/target/microblaze/mmu.o
  GEN     trace/generated-helpers.c
  CC      microblazeel-softmmu/trace/control-target.o
  CC      mips64el-softmmu/hw/9pfs/virtio-9p-device.o
  CC      microblazeel-softmmu/trace/generated-helpers.o
  CC      mips64el-softmmu/hw/block/virtio-blk.o
  CC      mips64-softmmu/accel/tcg/cpu-exec.o
  LINK    microblazeel-softmmu/qemu-system-microblazeel
  CC      microblaze-softmmu/target/microblaze/op_helper.o
  CC      mips64-softmmu/accel/tcg/cpu-exec-common.o
  CC      mips64el-softmmu/hw/block/dataplane/virtio-blk.o
  CC      microblaze-softmmu/target/microblaze/helper.o
  CC      microblaze-softmmu/target/microblaze/cpu.o
  CC      mips64-softmmu/accel/tcg/translate-all.o
  CC      mips64el-softmmu/hw/char/virtio-serial-bus.o
  CC      microblaze-softmmu/target/microblaze/gdbstub.o
  CC      microblaze-softmmu/target/microblaze/mmu.o
  CC      mips64el-softmmu/hw/core/generic-loader.o
  CC      mips64-softmmu/hw/9pfs/virtio-9p-device.o
  GEN     trace/generated-helpers.c
  CC      microblaze-softmmu/trace/control-target.o
  CC      mips64-softmmu/hw/block/virtio-blk.o
  CC      microblaze-softmmu/trace/generated-helpers.o
  CC      mips64el-softmmu/hw/core/null-machine.o
  CC      mips64el-softmmu/hw/display/vga.o
  LINK    microblaze-softmmu/qemu-system-microblaze
  CC      mips64el-softmmu/hw/display/virtio-gpu.o
  CC      mips64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mips64-softmmu/hw/char/virtio-serial-bus.o
  CC      mips64el-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64-softmmu/hw/core/generic-loader.o
  GEN     mipsel-softmmu/hmp-commands.h
  CC      mips64-softmmu/hw/core/null-machine.o
  GEN     mipsel-softmmu/hmp-commands-info.h
  GEN     mipsel-softmmu/config-target.h
  CC      mips64el-softmmu/hw/intc/mips_gic.o
  CC      mipsel-softmmu/exec.o
  CC      mips64-softmmu/hw/display/vga.o
  CC      mips64el-softmmu/hw/misc/ivshmem.o
  CC      mips64el-softmmu/hw/misc/mips_cmgcr.o
  CC      mips64el-softmmu/hw/misc/mips_cpc.o
  CC      mips64-softmmu/hw/display/virtio-gpu.o
  GEN     mips-softmmu/hmp-commands.h
  GEN     mips-softmmu/hmp-commands-info.h
  GEN     mips-softmmu/config-target.h
  CC      mips64el-softmmu/hw/misc/mips_itu.o
  CC      mips-softmmu/exec.o
  CC      mips64el-softmmu/hw/misc/mmio_interface.o
  CC      mips64-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/hw/net/virtio-net.o
  CC      mipsel-softmmu/tcg/tcg.o
  CC      mips64el-softmmu/hw/net/vhost_net.o
  CC      mips64-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64el-softmmu/hw/scsi/virtio-scsi.o
  CC      mips64-softmmu/hw/intc/mips_gic.o
  CC      mips-softmmu/tcg/tcg.o
  CC      mips64el-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips64-softmmu/hw/misc/ivshmem.o
  CC      mips64el-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mips64-softmmu/hw/misc/mips_cmgcr.o
  CC      mipsel-softmmu/tcg/tcg-op.o
  CC      mips64el-softmmu/hw/scsi/vhost-scsi.o
  CC      mips64-softmmu/hw/misc/mips_cpc.o
  CC      mips64el-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mips64-softmmu/hw/misc/mips_itu.o
  CC      mips64el-softmmu/hw/timer/mips_gictimer.o
  CC      mips-softmmu/tcg/tcg-op.o
  CC      mips64-softmmu/hw/misc/mmio_interface.o
  CC      mips64el-softmmu/hw/timer/mc146818rtc.o
  CC      mips64-softmmu/hw/net/virtio-net.o
  CC      mips64el-softmmu/hw/vfio/common.o
  CC      mips64-softmmu/hw/net/vhost_net.o
  CC      mips64-softmmu/hw/scsi/virtio-scsi.o
  CC      mipsel-softmmu/tcg/optimize.o
  CC      mips64el-softmmu/hw/vfio/pci.o
  CC      mips64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips-softmmu/tcg/optimize.o
  CC      mips64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mipsel-softmmu/tcg/tcg-common.o
  CC      mips64el-softmmu/hw/vfio/pci-quirks.o
  CC      mips64-softmmu/hw/scsi/vhost-scsi.o
  CC      mipsel-softmmu/tcg/tcg-runtime.o
  CC      mips64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mips-softmmu/tcg/tcg-common.o
  CC      mips64el-softmmu/hw/vfio/platform.o
  CC      mipsel-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/hw/timer/mips_gictimer.o
  CC      mips64-softmmu/hw/timer/mc146818rtc.o
  CC      mips-softmmu/tcg/tcg-runtime.o
  CC      mips64el-softmmu/hw/vfio/spapr.o
  CC      mips64el-softmmu/hw/virtio/virtio.o
  CC      mips-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/hw/vfio/common.o
  CC      mips64-softmmu/hw/vfio/pci.o
  CC      mips64el-softmmu/hw/virtio/virtio-balloon.o
  CC      mips64el-softmmu/hw/virtio/vhost.o
  CC      mips64-softmmu/hw/vfio/pci-quirks.o
  CC      mipsel-softmmu/disas.o
  CC      mips64el-softmmu/hw/virtio/vhost-backend.o
  CC      mipsel-softmmu/hax-stub.o
  CC      mips64-softmmu/hw/vfio/platform.o
  CC      mips-softmmu/disas.o
  CC      mipsel-softmmu/arch_init.o
  CC      mips64el-softmmu/hw/virtio/vhost-user.o
  CC      mips64-softmmu/hw/vfio/spapr.o
  CC      mips-softmmu/hax-stub.o
  CC      mipsel-softmmu/cpus.o
  CC      mips64el-softmmu/hw/virtio/vhost-vsock.o
  CC      mips-softmmu/arch_init.o
  CC      mips64-softmmu/hw/virtio/virtio.o
  CC      mips-softmmu/cpus.o
  CC      mips64el-softmmu/hw/virtio/virtio-crypto.o
  CC      mipsel-softmmu/monitor.o
  CC      mips64el-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      mips-softmmu/monitor.o
  CC      mips64-softmmu/hw/virtio/virtio-balloon.o
  CC      mips64el-softmmu/hw/mips/mips_r4k.o
  CC      mips64el-softmmu/hw/mips/mips_malta.o
  CC      mipsel-softmmu/gdbstub.o
  CC      mips64-softmmu/hw/virtio/vhost.o
  CC      mips64el-softmmu/hw/mips/mips_mipssim.o
  CC      mipsel-softmmu/balloon.o
  CC      mips-softmmu/gdbstub.o
  CC      mips64-softmmu/hw/virtio/vhost-backend.o
  CC      mips64el-softmmu/hw/mips/addr.o
  CC      mipsel-softmmu/ioport.o
  CC      mips64-softmmu/hw/virtio/vhost-user.o
  CC      mips64el-softmmu/hw/mips/cputimer.o
  CC      mips-softmmu/balloon.o
  CC      mipsel-softmmu/numa.o
  CC      mips-softmmu/ioport.o
  CC      mips64-softmmu/hw/virtio/vhost-vsock.o
  CC      mips64el-softmmu/hw/mips/mips_int.o
  CC      mips-softmmu/numa.o
  CC      mipsel-softmmu/qtest.o
  CC      mips64-softmmu/hw/virtio/virtio-crypto.o
  CC      mips64el-softmmu/hw/mips/mips_jazz.o
  CC      mips64el-softmmu/hw/mips/mips_fulong2e.o
  CC      mips-softmmu/qtest.o
  CC      mips64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      mipsel-softmmu/memory.o
  CC      mips64el-softmmu/hw/mips/gt64xxx_pci.o
  CC      mips-softmmu/memory.o
  CC      mips64-softmmu/hw/mips/mips_r4k.o
  CC      mips64el-softmmu/hw/mips/cps.o
  CC      mips64-softmmu/hw/mips/mips_malta.o
  CC      mips64-softmmu/hw/mips/mips_mipssim.o
  CC      mips64el-softmmu/hw/mips/boston.o
  CC      mipsel-softmmu/memory_mapping.o
  CC      mips-softmmu/memory_mapping.o
  CC      mips64el-softmmu/target/mips/translate.o
  CC      mips64-softmmu/hw/mips/addr.o
  CC      mipsel-softmmu/dump.o
  CC      mips64-softmmu/hw/mips/cputimer.o
  CC      mips-softmmu/dump.o
  CC      mips64-softmmu/hw/mips/mips_int.o
  CC      mipsel-softmmu/migration/ram.o
  CC      mips-softmmu/migration/ram.o
  CC      mips64-softmmu/hw/mips/mips_jazz.o
  CC      mips64-softmmu/hw/mips/gt64xxx_pci.o
  CC      mipsel-softmmu/accel/accel.o
  CC      mips64-softmmu/hw/mips/cps.o
  CC      mips-softmmu/accel/accel.o
  CC      mipsel-softmmu/accel/stubs/kvm-stub.o
  CC      mips64-softmmu/target/mips/translate.o
  CC      mips-softmmu/accel/stubs/kvm-stub.o
  CC      mipsel-softmmu/accel/tcg/tcg-all.o
  CC      mipsel-softmmu/accel/tcg/cputlb.o
  CC      mips-softmmu/accel/tcg/tcg-all.o
  CC      mips-softmmu/accel/tcg/cputlb.o
  CC      mips-softmmu/accel/tcg/cpu-exec.o
  CC      mipsel-softmmu/accel/tcg/cpu-exec.o
  CC      mipsel-softmmu/accel/tcg/cpu-exec-common.o
  CC      mips-softmmu/accel/tcg/cpu-exec-common.o
  CC      mipsel-softmmu/accel/tcg/translate-all.o
  CC      mips-softmmu/accel/tcg/translate-all.o
  CC      mipsel-softmmu/hw/9pfs/virtio-9p-device.o
  CC      mipsel-softmmu/hw/block/virtio-blk.o
  CC      mips-softmmu/hw/9pfs/virtio-9p-device.o
  CC      mips-softmmu/hw/block/virtio-blk.o
  CC      mipsel-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mips64el-softmmu/target/mips/dsp_helper.o
  CC      mipsel-softmmu/hw/char/virtio-serial-bus.o
  CC      mips-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mipsel-softmmu/hw/core/generic-loader.o
  CC      mipsel-softmmu/hw/core/null-machine.o
  CC      mips-softmmu/hw/char/virtio-serial-bus.o
  CC      mipsel-softmmu/hw/display/vga.o
  CC      mips-softmmu/hw/core/generic-loader.o
  CC      mips64el-softmmu/target/mips/op_helper.o
  CC      mips64-softmmu/target/mips/dsp_helper.o
  CC      mips-softmmu/hw/core/null-machine.o
  CC      mips-softmmu/hw/display/vga.o
  CC      mipsel-softmmu/hw/display/virtio-gpu.o
  CC      mips-softmmu/hw/display/virtio-gpu.o
  CC      mips64-softmmu/target/mips/op_helper.o
  CC      mipsel-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/target/mips/lmi_helper.o
  CC      mips-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/target/mips/helper.o
  CC      mipsel-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64el-softmmu/target/mips/cpu.o
  CC      mipsel-softmmu/hw/intc/mips_gic.o
  CC      mips-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64el-softmmu/target/mips/gdbstub.o
  CC      mipsel-softmmu/hw/misc/ivshmem.o
  CC      mips64el-softmmu/target/mips/msa_helper.o
  CC      mips-softmmu/hw/intc/mips_gic.o
  CC      mips-softmmu/hw/misc/ivshmem.o
  CC      mipsel-softmmu/hw/misc/mips_cmgcr.o
  CC      mips64-softmmu/target/mips/lmi_helper.o
  CC      mipsel-softmmu/hw/misc/mips_cpc.o
  CC      mips-softmmu/hw/misc/mips_cmgcr.o
  CC      mipsel-softmmu/hw/misc/mips_itu.o
  CC      mips64-softmmu/target/mips/helper.o
  CC      mips-softmmu/hw/misc/mips_cpc.o
  CC      mipsel-softmmu/hw/misc/mmio_interface.o
  CC      mipsel-softmmu/hw/net/virtio-net.o
  CC      mips64-softmmu/target/mips/cpu.o
  CC      mips-softmmu/hw/misc/mips_itu.o
  CC      mips64-softmmu/target/mips/gdbstub.o
  CC      mips-softmmu/hw/misc/mmio_interface.o
  CC      mips64-softmmu/target/mips/msa_helper.o
  CC      mipsel-softmmu/hw/net/vhost_net.o
  CC      mips-softmmu/hw/net/virtio-net.o
  CC      mipsel-softmmu/hw/scsi/virtio-scsi.o
  CC      mips-softmmu/hw/net/vhost_net.o
  CC      mips-softmmu/hw/scsi/virtio-scsi.o
  CC      mipsel-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips64el-softmmu/target/mips/mips-semi.o
  CC      mipsel-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mips-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips64el-softmmu/target/mips/machine.o
  CC      mipsel-softmmu/hw/scsi/vhost-scsi.o
  GEN     trace/generated-helpers.c
  CC      mips-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mips64el-softmmu/trace/control-target.o
  CC      mipsel-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mips-softmmu/hw/scsi/vhost-scsi.o
  CC      mips64el-softmmu/trace/generated-helpers.o
  CC      mips-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mipsel-softmmu/hw/timer/mips_gictimer.o
  CC      mips-softmmu/hw/timer/mips_gictimer.o
  LINK    mips64el-softmmu/qemu-system-mips64el
  CC      mipsel-softmmu/hw/timer/mc146818rtc.o
  CC      mips-softmmu/hw/timer/mc146818rtc.o
  CC      mips64-softmmu/target/mips/mips-semi.o
  CC      mips-softmmu/hw/vfio/common.o
  CC      mipsel-softmmu/hw/vfio/common.o
  CC      mips64-softmmu/target/mips/machine.o
  CC      mips-softmmu/hw/vfio/pci.o
  CC      mipsel-softmmu/hw/vfio/pci.o
  GEN     trace/generated-helpers.c
  CC      mips64-softmmu/trace/control-target.o
  CC      mips64-softmmu/trace/generated-helpers.o
  CC      mips-softmmu/hw/vfio/pci-quirks.o
  GEN     moxie-softmmu/hmp-commands.h
  GEN     moxie-softmmu/hmp-commands-info.h
  LINK    mips64-softmmu/qemu-system-mips64
  GEN     moxie-softmmu/config-target.h
  CC      moxie-softmmu/exec.o
  CC      mips-softmmu/hw/vfio/platform.o
  CC      mipsel-softmmu/hw/vfio/pci-quirks.o
  CC      mips-softmmu/hw/vfio/spapr.o
  CC      mipsel-softmmu/hw/vfio/platform.o
  CC      mipsel-softmmu/hw/vfio/spapr.o
  CC      mips-softmmu/hw/virtio/virtio.o
  CC      mipsel-softmmu/hw/virtio/virtio.o
  CC      moxie-softmmu/tcg/tcg.o
  CC      mipsel-softmmu/hw/virtio/virtio-balloon.o
  CC      mips-softmmu/hw/virtio/virtio-balloon.o
  CC      moxie-softmmu/tcg/tcg-op.o
  CC      mips-softmmu/hw/virtio/vhost.o
  CC      mipsel-softmmu/hw/virtio/vhost.o
  CC      mipsel-softmmu/hw/virtio/vhost-backend.o
  CC      mipsel-softmmu/hw/virtio/vhost-user.o
  CC      mips-softmmu/hw/virtio/vhost-backend.o
  CC      mipsel-softmmu/hw/virtio/vhost-vsock.o
  CC      mips-softmmu/hw/virtio/vhost-user.o
  CC      mipsel-softmmu/hw/virtio/virtio-crypto.o
  CC      mipsel-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      mips-softmmu/hw/virtio/vhost-vsock.o
  CC      mipsel-softmmu/hw/mips/mips_r4k.o
  GEN     nios2-softmmu/hmp-commands.h
  GEN     nios2-softmmu/hmp-commands-info.h
  GEN     nios2-softmmu/config-target.h
  CC      mips-softmmu/hw/virtio/virtio-crypto.o
  CC      nios2-softmmu/exec.o
  CC      mipsel-softmmu/hw/mips/mips_malta.o
  CC      mips-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      moxie-softmmu/tcg/optimize.o
  CC      mips-softmmu/hw/mips/mips_r4k.o
  CC      mipsel-softmmu/hw/mips/mips_mipssim.o
  CC      mipsel-softmmu/hw/mips/addr.o
  CC      mips-softmmu/hw/mips/mips_malta.o
  CC      mipsel-softmmu/hw/mips/cputimer.o
  CC      moxie-softmmu/tcg/tcg-common.o
  CC      nios2-softmmu/tcg/tcg.o
  CC      moxie-softmmu/tcg/tcg-runtime.o
  CC      mipsel-softmmu/hw/mips/mips_int.o
  CC      mips-softmmu/hw/mips/mips_mipssim.o
  CC      mipsel-softmmu/hw/mips/gt64xxx_pci.o
  CC      moxie-softmmu/fpu/softfloat.o
  CC      mips-softmmu/hw/mips/addr.o
  CC      mips-softmmu/hw/mips/cputimer.o
  CC      mips-softmmu/hw/mips/mips_int.o
  CC      mipsel-softmmu/hw/mips/cps.o
  CC      mips-softmmu/hw/mips/gt64xxx_pci.o
  CC      mipsel-softmmu/target/mips/translate.o
  CC      nios2-softmmu/tcg/tcg-op.o
  CC      mips-softmmu/hw/mips/cps.o
  CC      mips-softmmu/target/mips/translate.o
  CC      moxie-softmmu/disas.o
  CC      moxie-softmmu/hax-stub.o
  CC      moxie-softmmu/arch_init.o
  CC      nios2-softmmu/tcg/optimize.o
  CC      moxie-softmmu/cpus.o
  CC      nios2-softmmu/tcg/tcg-common.o
  CC      moxie-softmmu/monitor.o
  CC      nios2-softmmu/tcg/tcg-runtime.o
  CC      nios2-softmmu/fpu/softfloat.o
  CC      moxie-softmmu/gdbstub.o
  CC      mipsel-softmmu/target/mips/dsp_helper.o
  CC      mipsel-softmmu/target/mips/op_helper.o
  CC      moxie-softmmu/balloon.o
  CC      moxie-softmmu/ioport.o
  CC      mips-softmmu/target/mips/dsp_helper.o
  CC      moxie-softmmu/numa.o
  CC      nios2-softmmu/disas.o
  CC      moxie-softmmu/qtest.o
  CC      nios2-softmmu/hax-stub.o
  CC      mips-softmmu/target/mips/op_helper.o
  CC      nios2-softmmu/arch_init.o
  CC      moxie-softmmu/memory.o
  CC      nios2-softmmu/cpus.o
  CC      mipsel-softmmu/target/mips/lmi_helper.o
  CC      nios2-softmmu/monitor.o
  CC      moxie-softmmu/memory_mapping.o
  CC      mipsel-softmmu/target/mips/helper.o
  CC      moxie-softmmu/dump.o
  CC      mipsel-softmmu/target/mips/cpu.o
  CC      mipsel-softmmu/target/mips/gdbstub.o
  CC      mips-softmmu/target/mips/lmi_helper.o
  CC      moxie-softmmu/migration/ram.o
  CC      nios2-softmmu/gdbstub.o
  CC      mipsel-softmmu/target/mips/msa_helper.o
  CC      mips-softmmu/target/mips/helper.o
  CC      moxie-softmmu/accel/accel.o
  CC      nios2-softmmu/balloon.o
  CC      mips-softmmu/target/mips/cpu.o
  CC      nios2-softmmu/ioport.o
  CC      mips-softmmu/target/mips/gdbstub.o
  CC      moxie-softmmu/accel/stubs/kvm-stub.o
  CC      nios2-softmmu/numa.o
  CC      mips-softmmu/target/mips/msa_helper.o
  CC      moxie-softmmu/accel/tcg/tcg-all.o
  CC      nios2-softmmu/qtest.o
  CC      moxie-softmmu/accel/tcg/cputlb.o
  CC      nios2-softmmu/memory.o
  CC      moxie-softmmu/accel/tcg/cpu-exec.o
  CC      moxie-softmmu/accel/tcg/cpu-exec-common.o
  CC      mipsel-softmmu/target/mips/mips-semi.o
  CC      moxie-softmmu/accel/tcg/translate-all.o
  CC      nios2-softmmu/memory_mapping.o
  CC      mipsel-softmmu/target/mips/machine.o
  CC      moxie-softmmu/hw/core/generic-loader.o
  CC      nios2-softmmu/dump.o
  CC      mips-softmmu/target/mips/mips-semi.o
  GEN     trace/generated-helpers.c
  CC      mipsel-softmmu/trace/control-target.o
  CC      moxie-softmmu/hw/core/null-machine.o
  CC      nios2-softmmu/migration/ram.o
  CC      moxie-softmmu/hw/display/vga.o
  CC      mips-softmmu/target/mips/machine.o
  CC      mipsel-softmmu/trace/generated-helpers.o
  GEN     trace/generated-helpers.c
  CC      mips-softmmu/trace/control-target.o
  LINK    mipsel-softmmu/qemu-system-mipsel
  CC      mips-softmmu/trace/generated-helpers.o
  CC      nios2-softmmu/accel/accel.o
  CC      nios2-softmmu/accel/stubs/kvm-stub.o
  CC      moxie-softmmu/hw/misc/mmio_interface.o
  LINK    mips-softmmu/qemu-system-mips
  CC      nios2-softmmu/accel/tcg/tcg-all.o
  CC      moxie-softmmu/hw/net/vhost_net.o
  CC      nios2-softmmu/accel/tcg/cputlb.o
  CC      moxie-softmmu/hw/net/rocker/qmp-norocker.o
  CC      moxie-softmmu/hw/timer/mc146818rtc.o
  CC      moxie-softmmu/hw/vfio/common.o
  CC      moxie-softmmu/hw/vfio/platform.o
  CC      moxie-softmmu/hw/vfio/spapr.o
  CC      moxie-softmmu/hw/moxie/moxiesim.o
  CC      moxie-softmmu/target/moxie/translate.o
  CC      nios2-softmmu/accel/tcg/cpu-exec.o
  GEN     or1k-softmmu/hmp-commands.h
  GEN     or1k-softmmu/hmp-commands-info.h
  GEN     or1k-softmmu/config-target.h
  CC      nios2-softmmu/accel/tcg/cpu-exec-common.o
  CC      or1k-softmmu/exec.o
  CC      moxie-softmmu/target/moxie/helper.o
  GEN     ppc64-softmmu/hmp-commands.h
  CC      nios2-softmmu/accel/tcg/translate-all.o
  GEN     ppc64-softmmu/hmp-commands-info.h
  CC      moxie-softmmu/target/moxie/machine.o
  GEN     ppc64-softmmu/config-target.h
  CC      ppc64-softmmu/exec.o
  CC      moxie-softmmu/target/moxie/cpu.o
  CC      nios2-softmmu/hw/core/generic-loader.o
  CC      moxie-softmmu/target/moxie/mmu.o
  CC      nios2-softmmu/hw/core/null-machine.o
  GEN     trace/generated-helpers.c
  CC      moxie-softmmu/trace/control-target.o
  CC      nios2-softmmu/hw/intc/nios2_iic.o
  CC      nios2-softmmu/hw/misc/mmio_interface.o
  CC      moxie-softmmu/trace/generated-helpers.o
  CC      or1k-softmmu/tcg/tcg.o
  CC      nios2-softmmu/hw/net/vhost_net.o
  LINK    moxie-softmmu/qemu-system-moxie
  CC      nios2-softmmu/hw/net/rocker/qmp-norocker.o
  CC      nios2-softmmu/hw/timer/altera_timer.o
  CC      nios2-softmmu/hw/vfio/common.o
  CC      ppc64-softmmu/tcg/tcg.o
  CC      or1k-softmmu/tcg/tcg-op.o
  CC      nios2-softmmu/hw/vfio/platform.o
  CC      nios2-softmmu/hw/vfio/spapr.o
  CC      nios2-softmmu/hw/nios2/boot.o
  GEN     ppcemb-softmmu/hmp-commands.h
  CC      nios2-softmmu/hw/nios2/cpu_pic.o
  GEN     ppcemb-softmmu/hmp-commands-info.h
  GEN     ppcemb-softmmu/config-target.h
  CC      nios2-softmmu/hw/nios2/10m50_devboard.o
  CC      ppcemb-softmmu/exec.o
  CC      nios2-softmmu/target/nios2/translate.o
  CC      ppc64-softmmu/tcg/tcg-op.o
  CC      or1k-softmmu/tcg/optimize.o
  CC      nios2-softmmu/target/nios2/op_helper.o
  CC      nios2-softmmu/target/nios2/helper.o
  CC      or1k-softmmu/tcg/tcg-common.o
  CC      or1k-softmmu/tcg/tcg-runtime.o
  CC      nios2-softmmu/target/nios2/cpu.o
  CC      ppcemb-softmmu/tcg/tcg.o
  CC      nios2-softmmu/target/nios2/mmu.o
  CC      or1k-softmmu/fpu/softfloat.o
  CC      nios2-softmmu/target/nios2/monitor.o
  GEN     trace/generated-helpers.c
  CC      ppc64-softmmu/tcg/optimize.o
  CC      nios2-softmmu/trace/control-target.o
  CC      nios2-softmmu/trace/generated-helpers.o
  CC      ppcemb-softmmu/tcg/tcg-op.o
  CC      ppc64-softmmu/tcg/tcg-common.o
  LINK    nios2-softmmu/qemu-system-nios2
  CC      ppc64-softmmu/tcg/tcg-runtime.o
  CC      ppc64-softmmu/fpu/softfloat.o
  CC      or1k-softmmu/disas.o
  CC      ppcemb-softmmu/tcg/optimize.o
  CC      or1k-softmmu/hax-stub.o
  CC      or1k-softmmu/arch_init.o
  CC      ppc64-softmmu/disas.o
  CC      or1k-softmmu/cpus.o
  CC      ppcemb-softmmu/tcg/tcg-common.o
  CC      ppcemb-softmmu/tcg/tcg-runtime.o
  CC      or1k-softmmu/monitor.o
  CC      ppcemb-softmmu/fpu/softfloat.o
  GEN     ppc64-softmmu/gdbstub-xml.c
  CC      ppcemb-softmmu/disas.o
  CC      or1k-softmmu/gdbstub.o
  CC      ppc64-softmmu/hax-stub.o
  CC      or1k-softmmu/balloon.o
  CC      or1k-softmmu/ioport.o
  GEN     ppcemb-softmmu/gdbstub-xml.c
  CC      ppc64-softmmu/libdecnumber/decContext.o
  CC      or1k-softmmu/numa.o
  CC      ppc64-softmmu/libdecnumber/decNumber.o
  CC      ppcemb-softmmu/hax-stub.o
  CC      or1k-softmmu/memory.o
  CC      or1k-softmmu/qtest.o
  CC      ppcemb-softmmu/libdecnumber/decContext.o
  CC      or1k-softmmu/memory_mapping.o
  CC      ppcemb-softmmu/libdecnumber/decNumber.o
  CC      or1k-softmmu/dump.o
  CC      ppc64-softmmu/libdecnumber/dpd/decimal32.o
  CC      ppc64-softmmu/libdecnumber/dpd/decimal64.o
  CC      or1k-softmmu/migration/ram.o
  GEN     ppc-softmmu/hmp-commands.h
  CC      ppc64-softmmu/libdecnumber/dpd/decimal128.o
  GEN     ppc-softmmu/hmp-commands-info.h
  GEN     ppc-softmmu/config-target.h
  CC      ppc-softmmu/exec.o
  CC      ppc64-softmmu/arch_init.o
  CC      ppcemb-softmmu/libdecnumber/dpd/decimal32.o
  CC      ppc64-softmmu/cpus.o
  CC      or1k-softmmu/accel/accel.o
  CC      ppcemb-softmmu/libdecnumber/dpd/decimal64.o
  CC      ppcemb-softmmu/libdecnumber/dpd/decimal128.o
  CC      ppc64-softmmu/monitor.o
  CC      or1k-softmmu/accel/stubs/kvm-stub.o
  CC      ppcemb-softmmu/arch_init.o
  CC      or1k-softmmu/accel/tcg/tcg-all.o
  CC      ppcemb-softmmu/cpus.o
  CC      or1k-softmmu/accel/tcg/cputlb.o
  CC      ppc-softmmu/tcg/tcg.o
  CC      ppcemb-softmmu/monitor.o
  CC      ppc64-softmmu/gdbstub.o
  CC      or1k-softmmu/accel/tcg/cpu-exec.o
  CC      ppc64-softmmu/balloon.o
  CC      ppcemb-softmmu/gdbstub.o
  CC      ppc-softmmu/tcg/tcg-op.o
  CC      ppc64-softmmu/ioport.o
  CC      or1k-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppc64-softmmu/numa.o
  CC      ppcemb-softmmu/balloon.o
  CC      or1k-softmmu/accel/tcg/translate-all.o
  CC      ppcemb-softmmu/ioport.o
  CC      ppc64-softmmu/qtest.o
  CC      or1k-softmmu/hw/core/generic-loader.o
  CC      ppcemb-softmmu/numa.o
  CC      or1k-softmmu/hw/core/null-machine.o
  CC      ppc64-softmmu/memory.o
  CC      or1k-softmmu/hw/misc/mmio_interface.o
  CC      ppcemb-softmmu/qtest.o
  CC      or1k-softmmu/hw/net/vhost_net.o
  CC      ppcemb-softmmu/memory.o
  CC      ppc-softmmu/tcg/optimize.o
  CC      or1k-softmmu/hw/net/rocker/qmp-norocker.o
  CC      ppc64-softmmu/memory_mapping.o
  CC      or1k-softmmu/hw/vfio/common.o
  CC      ppc64-softmmu/dump.o
  CC      ppc64-softmmu/migration/ram.o
  CC      ppcemb-softmmu/memory_mapping.o
  CC      or1k-softmmu/hw/vfio/platform.o
  CC      ppc-softmmu/tcg/tcg-common.o
  CC      ppcemb-softmmu/dump.o
  CC      ppc-softmmu/tcg/tcg-runtime.o
  CC      or1k-softmmu/hw/vfio/spapr.o
  CC      ppc-softmmu/fpu/softfloat.o
  CC      ppc64-softmmu/accel/accel.o
  CC      or1k-softmmu/hw/openrisc/pic_cpu.o
  CC      ppcemb-softmmu/migration/ram.o
  CC      ppc64-softmmu/accel/stubs/kvm-stub.o
  CC      or1k-softmmu/hw/openrisc/cputimer.o
  CC      ppc64-softmmu/accel/tcg/tcg-all.o
  CC      or1k-softmmu/hw/openrisc/openrisc_sim.o
  CC      ppc64-softmmu/accel/tcg/cputlb.o
  CC      ppcemb-softmmu/accel/accel.o
  CC      or1k-softmmu/target/openrisc/machine.o
  CC      ppcemb-softmmu/accel/stubs/kvm-stub.o
  CC      or1k-softmmu/target/openrisc/cpu.o
  CC      or1k-softmmu/target/openrisc/exception.o
  CC      ppcemb-softmmu/accel/tcg/tcg-all.o
  CC      or1k-softmmu/target/openrisc/interrupt.o
  CC      ppcemb-softmmu/accel/tcg/cputlb.o
  CC      or1k-softmmu/target/openrisc/mmu.o
  CC      ppc64-softmmu/accel/tcg/cpu-exec.o
  CC      ppc-softmmu/disas.o
  CC      or1k-softmmu/target/openrisc/translate.o
  CC      ppc64-softmmu/accel/tcg/cpu-exec-common.o
  GEN     ppc-softmmu/gdbstub-xml.c
  CC      ppc64-softmmu/accel/tcg/translate-all.o
  CC      ppcemb-softmmu/accel/tcg/cpu-exec.o
  CC      or1k-softmmu/target/openrisc/exception_helper.o
  CC      or1k-softmmu/target/openrisc/fpu_helper.o
  CC      ppc-softmmu/hax-stub.o
  CC      ppcemb-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppc64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppcemb-softmmu/accel/tcg/translate-all.o
  CC      ppc-softmmu/libdecnumber/decContext.o
  CC      ppc64-softmmu/hw/block/virtio-blk.o
  CC      or1k-softmmu/target/openrisc/interrupt_helper.o
  CC      ppc-softmmu/libdecnumber/decNumber.o
  CC      or1k-softmmu/target/openrisc/mmu_helper.o
  CC      ppcemb-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppc64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      or1k-softmmu/target/openrisc/sys_helper.o
  CC      ppcemb-softmmu/hw/block/virtio-blk.o
  CC      or1k-softmmu/target/openrisc/gdbstub.o
  CC      ppc64-softmmu/hw/char/spapr_vty.o
  GEN     trace/generated-helpers.c
  CC      or1k-softmmu/trace/control-target.o
  CC      ppcemb-softmmu/hw/block/dataplane/virtio-blk.o
  CC      ppc64-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc-softmmu/libdecnumber/dpd/decimal32.o
  CC      or1k-softmmu/trace/generated-helpers.o
  CC      ppcemb-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc64-softmmu/hw/core/generic-loader.o
  CC      ppc-softmmu/libdecnumber/dpd/decimal64.o
  LINK    or1k-softmmu/qemu-system-or1k
  CC      ppc64-softmmu/hw/core/null-machine.o
  CC      ppc-softmmu/libdecnumber/dpd/decimal128.o
  CC      ppcemb-softmmu/hw/core/generic-loader.o
  CC      ppc64-softmmu/hw/display/sm501.o
  CC      ppc-softmmu/arch_init.o
  CC      ppcemb-softmmu/hw/core/null-machine.o
  CC      ppcemb-softmmu/hw/display/sm501.o
  CC      ppc-softmmu/cpus.o
  CC      ppc64-softmmu/hw/display/vga.o
  CC      ppc-softmmu/monitor.o
  CC      ppcemb-softmmu/hw/display/vga.o
  CC      ppc64-softmmu/hw/display/virtio-gpu.o
  CC      ppc-softmmu/gdbstub.o
  GEN     s390x-softmmu/hmp-commands.h
  GEN     s390x-softmmu/hmp-commands-info.h
  CC      s390x-softmmu/gen-features
  GEN     s390x-softmmu/config-target.h
  GEN     s390x-softmmu/gen-features.h
  CC      ppcemb-softmmu/hw/display/virtio-gpu.o
  CC      s390x-softmmu/exec.o
  CC      ppc64-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppc-softmmu/balloon.o
  CC      ppc64-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppc-softmmu/ioport.o
  CC      ppcemb-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppc64-softmmu/hw/display/virtio-vga.o
  CC      ppc-softmmu/numa.o
  CC      ppc64-softmmu/hw/intc/xics.o
  CC      ppcemb-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppc-softmmu/qtest.o
  CC      ppcemb-softmmu/hw/misc/ivshmem.o
  CC      ppc64-softmmu/hw/intc/xics_spapr.o
  CC      s390x-softmmu/tcg/tcg.o
  CC      ppc-softmmu/memory.o
  CC      ppcemb-softmmu/hw/misc/mmio_interface.o
  CC      ppc64-softmmu/hw/intc/xics_pnv.o
  CC      ppcemb-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc64-softmmu/hw/misc/ivshmem.o
  CC      ppcemb-softmmu/hw/net/virtio-net.o
  CC      ppc-softmmu/memory_mapping.o
  CC      ppc64-softmmu/hw/misc/mmio_interface.o
  CC      s390x-softmmu/tcg/tcg-op.o
  CC      ppc-softmmu/dump.o
  CC      ppc64-softmmu/hw/net/spapr_llan.o
  CC      ppcemb-softmmu/hw/net/vhost_net.o
  CC      ppc64-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc-softmmu/migration/ram.o
  CC      ppcemb-softmmu/hw/scsi/virtio-scsi.o
  CC      ppc64-softmmu/hw/net/virtio-net.o
  CC      ppcemb-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      ppc64-softmmu/hw/net/vhost_net.o
  CC      ppcemb-softmmu/hw/scsi/vhost-scsi-common.o
  CC      ppc-softmmu/accel/accel.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/etsec.o
  CC      ppcemb-softmmu/hw/scsi/vhost-scsi.o
  CC      ppc-softmmu/accel/stubs/kvm-stub.o
  CC      s390x-softmmu/tcg/optimize.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/registers.o
  CC      ppcemb-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc-softmmu/accel/tcg/tcg-all.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/rings.o
  CC      ppcemb-softmmu/hw/vfio/common.o
  CC      ppc-softmmu/accel/tcg/cputlb.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/miim.o
  CC      ppcemb-softmmu/hw/vfio/pci.o
  CC      s390x-softmmu/tcg/tcg-common.o
  CC      ppc64-softmmu/hw/nvram/spapr_nvram.o
  CC      s390x-softmmu/tcg/tcg-runtime.o
  CC      ppc64-softmmu/hw/scsi/spapr_vscsi.o
  CC      s390x-softmmu/fpu/softfloat.o
  CC      ppcemb-softmmu/hw/vfio/pci-quirks.o
  CC      ppc-softmmu/accel/tcg/cpu-exec.o
  CC      ppc64-softmmu/hw/scsi/virtio-scsi.o
  CC      ppc-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppc64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      ppcemb-softmmu/hw/vfio/platform.o
  CC      ppc-softmmu/accel/tcg/translate-all.o
  CC      ppc64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      ppcemb-softmmu/hw/vfio/spapr.o
  CC      ppc64-softmmu/hw/scsi/vhost-scsi.o
  CC      ppc-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppcemb-softmmu/hw/virtio/virtio.o
  CC      ppc64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc-softmmu/hw/block/virtio-blk.o
  CC      ppc64-softmmu/hw/timer/mc146818rtc.o
  CC      s390x-softmmu/disas.o
  CC      ppcemb-softmmu/hw/virtio/virtio-balloon.o
  CC      ppc-softmmu/hw/block/dataplane/virtio-blk.o
  CC      ppc64-softmmu/hw/vfio/common.o
  GEN     s390x-softmmu/gdbstub-xml.c
  CC      ppcemb-softmmu/hw/virtio/vhost.o
  CC      ppc-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc64-softmmu/hw/vfio/pci.o
  CC      s390x-softmmu/hax-stub.o
  CC      ppcemb-softmmu/hw/virtio/vhost-backend.o
  CC      ppc-softmmu/hw/core/generic-loader.o
  CC      s390x-softmmu/arch_init.o
  CC      ppc-softmmu/hw/core/null-machine.o
  CC      ppcemb-softmmu/hw/virtio/vhost-user.o
  CC      s390x-softmmu/cpus.o
  CC      ppc-softmmu/hw/display/sm501.o
  CC      ppcemb-softmmu/hw/virtio/vhost-vsock.o
  CC      ppc64-softmmu/hw/vfio/pci-quirks.o
  CC      ppcemb-softmmu/hw/virtio/virtio-crypto.o
  CC      ppc-softmmu/hw/display/vga.o
  CC      s390x-softmmu/monitor.o
  CC      ppc64-softmmu/hw/vfio/platform.o
  CC      ppcemb-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      ppc64-softmmu/hw/vfio/spapr.o
  CC      ppcemb-softmmu/hw/ppc/ppc.o
  CC      ppc64-softmmu/hw/virtio/virtio.o
  CC      ppc-softmmu/hw/display/virtio-gpu.o
  CC      ppcemb-softmmu/hw/ppc/ppc_booke.o
  CC      s390x-softmmu/gdbstub.o
  CC      ppcemb-softmmu/hw/ppc/fdt.o
  CC      ppc-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppcemb-softmmu/hw/ppc/ppc405_boards.o
  CC      ppc64-softmmu/hw/virtio/virtio-balloon.o
  CC      s390x-softmmu/balloon.o
  CC      ppcemb-softmmu/hw/ppc/ppc4xx_devs.o
  CC      s390x-softmmu/ioport.o
  CC      ppc64-softmmu/hw/virtio/vhost.o
  CC      ppc-softmmu/hw/display/virtio-gpu-pci.o
  CC      s390x-softmmu/numa.o
  CC      ppcemb-softmmu/hw/ppc/ppc405_uc.o
  CC      ppc-softmmu/hw/misc/ivshmem.o
  CC      ppc64-softmmu/hw/virtio/vhost-backend.o
  CC      s390x-softmmu/qtest.o
  CC      ppcemb-softmmu/hw/ppc/ppc440_bamboo.o
  CC      ppc64-softmmu/hw/virtio/vhost-user.o
  CC      ppc-softmmu/hw/misc/mmio_interface.o
  CC      s390x-softmmu/memory.o
  CC      ppcemb-softmmu/hw/ppc/ppc4xx_pci.o
  CC      ppc-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc64-softmmu/hw/virtio/vhost-vsock.o
  CC      ppcemb-softmmu/hw/ppc/virtex_ml507.o
  CC      ppc-softmmu/hw/net/virtio-net.o
  CC      ppc64-softmmu/hw/virtio/virtio-crypto.o
  CC      ppcemb-softmmu/target/ppc/cpu-models.o
  CC      ppc64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      s390x-softmmu/memory_mapping.o
  CC      s390x-softmmu/dump.o
  CC      ppc-softmmu/hw/net/vhost_net.o
  CC      ppc64-softmmu/hw/ppc/ppc.o
  CC      ppcemb-softmmu/target/ppc/cpu.o
  CC      ppc-softmmu/hw/net/fsl_etsec/etsec.o
  CC      ppcemb-softmmu/target/ppc/translate.o
  CC      s390x-softmmu/migration/ram.o
  CC      ppc64-softmmu/hw/ppc/ppc_booke.o
  CC      ppc-softmmu/hw/net/fsl_etsec/registers.o
  CC      ppc-softmmu/hw/net/fsl_etsec/rings.o
  CC      ppc64-softmmu/hw/ppc/fdt.o
  CC      ppc-softmmu/hw/net/fsl_etsec/miim.o
  CC      ppc64-softmmu/hw/ppc/spapr.o
  CC      s390x-softmmu/accel/accel.o
  CC      ppc-softmmu/hw/scsi/virtio-scsi.o
  CC      s390x-softmmu/accel/kvm/kvm-all.o
  CC      ppc64-softmmu/hw/ppc/spapr_vio.o
  CC      ppc-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      s390x-softmmu/accel/tcg/tcg-all.o
  CC      ppc64-softmmu/hw/ppc/spapr_events.o
  CC      ppc-softmmu/hw/scsi/vhost-scsi-common.o
  CC      s390x-softmmu/accel/tcg/cputlb.o
  CC      ppc-softmmu/hw/scsi/vhost-scsi.o
  CC      ppc64-softmmu/hw/ppc/spapr_hcall.o
  CC      ppc-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc64-softmmu/hw/ppc/spapr_iommu.o
  CC      ppc-softmmu/hw/timer/mc146818rtc.o
  CC      ppc64-softmmu/hw/ppc/spapr_rtas.o
  CC      s390x-softmmu/accel/tcg/cpu-exec.o
  CC      ppc-softmmu/hw/vfio/common.o
  CC      ppc64-softmmu/hw/ppc/spapr_pci.o
  CC      s390x-softmmu/accel/tcg/cpu-exec-common.o
  CC      s390x-softmmu/accel/tcg/translate-all.o
  CC      ppc-softmmu/hw/vfio/pci.o
  CC      ppc64-softmmu/hw/ppc/spapr_rtc.o
  CC      s390x-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppc64-softmmu/hw/ppc/spapr_drc.o
  CC      ppc-softmmu/hw/vfio/pci-quirks.o
  CC      s390x-softmmu/hw/block/virtio-blk.o
  CC      ppc-softmmu/hw/vfio/platform.o
  CC      ppc64-softmmu/hw/ppc/spapr_rng.o
  CC      s390x-softmmu/hw/block/dataplane/virtio-blk.o
  CC      ppc64-softmmu/hw/ppc/spapr_cpu_core.o
  CC      s390x-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc-softmmu/hw/vfio/spapr.o
  CC      ppc-softmmu/hw/virtio/virtio.o
  CC      ppc64-softmmu/hw/ppc/spapr_ovec.o
  CC      s390x-softmmu/hw/char/terminal3270.o
  CC      ppc64-softmmu/hw/ppc/pnv.o
  CC      s390x-softmmu/hw/core/generic-loader.o
  CC      ppc-softmmu/hw/virtio/virtio-balloon.o
  CC      s390x-softmmu/hw/core/null-machine.o
  CC      ppc64-softmmu/hw/ppc/pnv_xscom.o
  CC      s390x-softmmu/hw/display/virtio-gpu.o
  CC      ppc64-softmmu/hw/ppc/pnv_core.o
  CC      ppc-softmmu/hw/virtio/vhost.o
  CC      ppc64-softmmu/hw/ppc/pnv_lpc.o
  CC      s390x-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppc64-softmmu/hw/ppc/pnv_psi.o
  CC      ppc-softmmu/hw/virtio/vhost-backend.o
  CC      ppc64-softmmu/hw/ppc/pnv_occ.o
  CC      s390x-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppc-softmmu/hw/virtio/vhost-user.o
  CC      ppc64-softmmu/hw/ppc/pnv_bmc.o
  CC      ppc64-softmmu/hw/ppc/spapr_pci_vfio.o
  CC      ppc-softmmu/hw/virtio/vhost-vsock.o
  CC      s390x-softmmu/hw/intc/s390_flic.o
  CC      ppc64-softmmu/hw/ppc/spapr_rtas_ddw.o
  CC      ppc-softmmu/hw/virtio/virtio-crypto.o
  CC      s390x-softmmu/hw/intc/s390_flic_kvm.o
  CC      ppc64-softmmu/hw/ppc/ppc405_boards.o
  CC      ppcemb-softmmu/target/ppc/machine.o
  CC      ppc-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      s390x-softmmu/hw/misc/mmio_interface.o
  CC      ppc64-softmmu/hw/ppc/ppc4xx_devs.o
  CC      ppcemb-softmmu/target/ppc/mmu_helper.o
  CC      s390x-softmmu/hw/net/virtio-net.o
  CC      ppc-softmmu/hw/ppc/ppc.o
  CC      ppc64-softmmu/hw/ppc/ppc405_uc.o
  CC      ppc-softmmu/hw/ppc/ppc_booke.o
  CC      s390x-softmmu/hw/net/vhost_net.o
  CC      ppc-softmmu/hw/ppc/fdt.o
  CC      ppcemb-softmmu/target/ppc/mmu-hash32.o
  CC      ppc64-softmmu/hw/ppc/ppc440_bamboo.o
  CC      s390x-softmmu/hw/net/rocker/qmp-norocker.o
  CC      ppc-softmmu/hw/ppc/ppc405_boards.o
  CC      ppcemb-softmmu/target/ppc/monitor.o
  CC      s390x-softmmu/hw/scsi/virtio-scsi.o
  CC      ppc-softmmu/hw/ppc/ppc4xx_devs.o
  CC      ppc64-softmmu/hw/ppc/ppc4xx_pci.o
  CC      ppcemb-softmmu/target/ppc/arch_dump.o
  CC      ppc64-softmmu/hw/ppc/prep.o
  CC      s390x-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      ppcemb-softmmu/target/ppc/kvm-stub.o
  CC      ppc-softmmu/hw/ppc/ppc405_uc.o
  CC      ppcemb-softmmu/target/ppc/dfp_helper.o
  CC      s390x-softmmu/hw/scsi/vhost-scsi-common.o
  CC      ppc64-softmmu/hw/ppc/prep_systemio.o
  CC      s390x-softmmu/hw/scsi/vhost-scsi.o
  CC      ppc-softmmu/hw/ppc/ppc440_bamboo.o
  CC      s390x-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc64-softmmu/hw/ppc/rs6000_mc.o
  CC      ppcemb-softmmu/target/ppc/excp_helper.o
  CC      s390x-softmmu/hw/vfio/common.o
  CC      ppc-softmmu/hw/ppc/ppc4xx_pci.o
  CC      ppc64-softmmu/hw/ppc/mac_oldworld.o
  CC      ppc64-softmmu/hw/ppc/mac_newworld.o
  CC      ppc-softmmu/hw/ppc/prep.o
  CC      ppcemb-softmmu/target/ppc/fpu_helper.o
  CC      s390x-softmmu/hw/vfio/pci.o
  CC      ppc64-softmmu/hw/ppc/e500.o
  CC      ppc-softmmu/hw/ppc/prep_systemio.o
  CC      ppc64-softmmu/hw/ppc/mpc8544ds.o
  CC      ppc-softmmu/hw/ppc/rs6000_mc.o
  CC      ppc64-softmmu/hw/ppc/e500plat.o
  CC      s390x-softmmu/hw/vfio/pci-quirks.o
  CC      ppc-softmmu/hw/ppc/mac_oldworld.o
  CC      ppc64-softmmu/hw/ppc/mpc8544_guts.o
  CC      ppc64-softmmu/hw/ppc/ppce500_spin.o
  CC      ppc-softmmu/hw/ppc/mac_newworld.o
  CC      s390x-softmmu/hw/vfio/ccw.o
  CC      ppc64-softmmu/hw/ppc/virtex_ml507.o
  CC      ppc-softmmu/hw/ppc/e500.o
  CC      s390x-softmmu/hw/vfio/platform.o
  CC      ppc64-softmmu/target/ppc/cpu-models.o
  CC      ppc-softmmu/hw/ppc/mpc8544ds.o
  CC      s390x-softmmu/hw/vfio/spapr.o
  CC      ppc-softmmu/hw/ppc/e500plat.o
  CC      s390x-softmmu/hw/virtio/virtio.o
  CC      ppc-softmmu/hw/ppc/mpc8544_guts.o
  CC      ppcemb-softmmu/target/ppc/int_helper.o
  CC      ppc-softmmu/hw/ppc/ppce500_spin.o
  CC      ppc64-softmmu/target/ppc/cpu.o
  CC      ppc-softmmu/hw/ppc/virtex_ml507.o
  CC      ppc-softmmu/target/ppc/cpu-models.o
  CC      s390x-softmmu/hw/virtio/virtio-balloon.o
  CC      ppc64-softmmu/target/ppc/translate.o
  CC      s390x-softmmu/hw/virtio/vhost.o
  CC      ppcemb-softmmu/target/ppc/timebase_helper.o
  CC      s390x-softmmu/hw/virtio/vhost-backend.o
  CC      ppc-softmmu/target/ppc/cpu.o
  CC      ppcemb-softmmu/target/ppc/misc_helper.o
  CC      s390x-softmmu/hw/virtio/vhost-user.o
  CC      ppc-softmmu/target/ppc/translate.o
  CC      s390x-softmmu/hw/virtio/vhost-vsock.o
  CC      ppcemb-softmmu/target/ppc/mem_helper.o
  CC      s390x-softmmu/hw/virtio/virtio-crypto.o
  CC      ppcemb-softmmu/target/ppc/gdbstub.o
  CC      s390x-softmmu/hw/virtio/virtio-crypto-pci.o
  GEN     trace/generated-helpers.c
  CC      ppcemb-softmmu/trace/control-target.o
  CC      s390x-softmmu/hw/s390x/s390-virtio.o
  CC      ppcemb-softmmu/gdbstub-xml.o
  CC      s390x-softmmu/hw/s390x/s390-virtio-hcall.o
  CC      ppcemb-softmmu/trace/generated-helpers.o
  CC      s390x-softmmu/hw/s390x/sclp.o
  LINK    ppcemb-softmmu/qemu-system-ppcemb
  CC      s390x-softmmu/hw/s390x/event-facility.o
  CC      s390x-softmmu/hw/s390x/sclpquiesce.o
  CC      s390x-softmmu/hw/s390x/sclpcpu.o
  CC      ppc-softmmu/target/ppc/machine.o
  CC      s390x-softmmu/hw/s390x/ipl.o
  CC      ppc64-softmmu/target/ppc/machine.o
  CC      s390x-softmmu/hw/s390x/css.o
  CC      ppc64-softmmu/target/ppc/mmu_helper.o
  CC      s390x-softmmu/hw/s390x/s390-virtio-ccw.o
  CC      s390x-softmmu/hw/s390x/3270-ccw.o
  GEN     sh4eb-softmmu/hmp-commands.h
  GEN     sh4eb-softmmu/hmp-commands-info.h
  GEN     sh4eb-softmmu/config-target.h
  CC      sh4eb-softmmu/exec.o
  CC      s390x-softmmu/hw/s390x/virtio-ccw.o
  CC      s390x-softmmu/hw/s390x/css-bridge.o
  CC      s390x-softmmu/hw/s390x/ccw-device.o
  CC      sh4eb-softmmu/tcg/tcg.o
  CC      s390x-softmmu/hw/s390x/s390-pci-bus.o
  CC      s390x-softmmu/hw/s390x/s390-pci-inst.o
  CC      sh4eb-softmmu/tcg/tcg-op.o
  CC      s390x-softmmu/hw/s390x/s390-skeys.o
  CC      ppc-softmmu/target/ppc/mmu_helper.o
  CC      s390x-softmmu/hw/s390x/s390-stattrib.o
  CC      s390x-softmmu/hw/s390x/s390-skeys-kvm.o
  CC      ppc64-softmmu/target/ppc/mmu-hash32.o
  CC      s390x-softmmu/hw/s390x/s390-stattrib-kvm.o
  CC      ppc-softmmu/target/ppc/mmu-hash32.o
  CC      ppc64-softmmu/target/ppc/monitor.o
  CC      s390x-softmmu/hw/s390x/s390-ccw.o
  CC      ppc-softmmu/target/ppc/monitor.o
  CC      ppc64-softmmu/target/ppc/arch_dump.o
  CC      ppc-softmmu/target/ppc/arch_dump.o
  CC      s390x-softmmu/target/s390x/cpu.o
  CC      ppc64-softmmu/target/ppc/mmu-hash64.o
  CC      sh4eb-softmmu/tcg/optimize.o
  CC      ppc-softmmu/target/ppc/kvm-stub.o
  CC      s390x-softmmu/target/s390x/cpu_models.o
  CC      ppc64-softmmu/target/ppc/mmu-book3s-v3.o
  CC      ppc-softmmu/target/ppc/dfp_helper.o
  CC      ppc64-softmmu/target/ppc/compat.o
  CC      s390x-softmmu/target/s390x/cpu_features.o
  CC      ppc64-softmmu/target/ppc/mmu-radix64.o
  CC      sh4eb-softmmu/tcg/tcg-common.o
  CC      s390x-softmmu/target/s390x/gdbstub.o
  CC      sh4eb-softmmu/tcg/tcg-runtime.o
  CC      ppc-softmmu/target/ppc/excp_helper.o
  CC      ppc64-softmmu/target/ppc/kvm-stub.o
  CC      sh4eb-softmmu/fpu/softfloat.o
  CC      s390x-softmmu/target/s390x/interrupt.o
  CC      ppc64-softmmu/target/ppc/dfp_helper.o
  CC      ppc-softmmu/target/ppc/fpu_helper.o
  CC      s390x-softmmu/target/s390x/helper.o
  CC      ppc64-softmmu/target/ppc/excp_helper.o
  CC      s390x-softmmu/target/s390x/translate.o
  CC      ppc64-softmmu/target/ppc/fpu_helper.o
  CC      sh4eb-softmmu/disas.o
  CC      sh4eb-softmmu/hax-stub.o
  CC      sh4eb-softmmu/arch_init.o
  CC      s390x-softmmu/target/s390x/cc_helper.o
  CC      ppc-softmmu/target/ppc/int_helper.o
  CC      sh4eb-softmmu/cpus.o
  CC      s390x-softmmu/target/s390x/excp_helper.o
  CC      s390x-softmmu/target/s390x/fpu_helper.o
  CC      sh4eb-softmmu/monitor.o
  CC      s390x-softmmu/target/s390x/int_helper.o
  CC      ppc64-softmmu/target/ppc/int_helper.o
  CC      s390x-softmmu/target/s390x/mem_helper.o
  CC      ppc-softmmu/target/ppc/timebase_helper.o
  CC      ppc-softmmu/target/ppc/misc_helper.o
  CC      sh4eb-softmmu/gdbstub.o
  CC      ppc-softmmu/target/ppc/mem_helper.o
  CC      sh4eb-softmmu/balloon.o
  CC      ppc64-softmmu/target/ppc/timebase_helper.o
  CC      ppc-softmmu/target/ppc/gdbstub.o
  CC      sh4eb-softmmu/ioport.o
  CC      ppc64-softmmu/target/ppc/misc_helper.o
  GEN     trace/generated-helpers.c
  CC      ppc-softmmu/trace/control-target.o
  CC      sh4eb-softmmu/numa.o
  CC      ppc64-softmmu/target/ppc/mem_helper.o
  CC      s390x-softmmu/target/s390x/misc_helper.o
  CC      ppc-softmmu/gdbstub-xml.o
  CC      sh4eb-softmmu/qtest.o
  CC      s390x-softmmu/target/s390x/machine.o
  CC      ppc-softmmu/trace/generated-helpers.o
  CC      ppc64-softmmu/target/ppc/gdbstub.o
  CC      sh4eb-softmmu/memory.o
  CC      s390x-softmmu/target/s390x/ioinst.o
  GEN     trace/generated-helpers.c
  LINK    ppc-softmmu/qemu-system-ppc
  CC      ppc64-softmmu/trace/control-target.o
  CC      s390x-softmmu/target/s390x/arch_dump.o
  CC      ppc64-softmmu/gdbstub-xml.o
  CC      ppc64-softmmu/trace/generated-helpers.o
  CC      s390x-softmmu/target/s390x/mmu_helper.o
  CC      sh4eb-softmmu/memory_mapping.o
  LINK    ppc64-softmmu/qemu-system-ppc64
  CC      sh4eb-softmmu/dump.o
  CC      s390x-softmmu/target/s390x/diag.o
  CC      s390x-softmmu/target/s390x/kvm.o
  CC      sh4eb-softmmu/migration/ram.o
  GEN     trace/generated-helpers.c
  CC      s390x-softmmu/trace/control-target.o
  CC      s390x-softmmu/gdbstub-xml.o
  CC      sh4eb-softmmu/accel/accel.o
  CC      s390x-softmmu/trace/generated-helpers.o
  CC      sh4eb-softmmu/accel/stubs/kvm-stub.o
  CC      sh4eb-softmmu/accel/tcg/tcg-all.o
  CC      sh4eb-softmmu/accel/tcg/cputlb.o
  CC      sh4eb-softmmu/accel/tcg/cpu-exec.o
  LINK    s390x-softmmu/qemu-system-s390x
  CC      sh4eb-softmmu/accel/tcg/cpu-exec-common.o
  CC      sh4eb-softmmu/accel/tcg/translate-all.o
  CC      sh4eb-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sh4eb-softmmu/hw/block/tc58128.o
  CC      sh4eb-softmmu/hw/block/virtio-blk.o
  CC      sh4eb-softmmu/hw/block/dataplane/virtio-blk.o
  GEN     sh4-softmmu/hmp-commands.h
  GEN     sh4-softmmu/hmp-commands-info.h
  GEN     sh4-softmmu/config-target.h
  CC      sh4eb-softmmu/hw/char/sh_serial.o
  GEN     sparc64-softmmu/hmp-commands.h
  GEN     sparc64-softmmu/hmp-commands-info.h
  GEN     sparc64-softmmu/config-target.h
  CC      sparc64-softmmu/exec.o
  CC      sh4-softmmu/exec.o
  CC      sparc64-softmmu/tcg/tcg.o
  CC      sh4eb-softmmu/hw/char/virtio-serial-bus.o
  CC      sh4eb-softmmu/hw/core/generic-loader.o
  CC      sh4eb-softmmu/hw/core/null-machine.o
  CC      sh4eb-softmmu/hw/display/sm501.o
  CC      sh4-softmmu/tcg/tcg.o
  GEN     sparc-softmmu/hmp-commands.h
  GEN     sparc-softmmu/hmp-commands-info.h
  GEN     sparc-softmmu/config-target.h
  CC      sparc-softmmu/exec.o
  CC      sparc64-softmmu/tcg/tcg-op.o
  CC      sh4eb-softmmu/hw/display/vga.o
  CC      sh4-softmmu/tcg/tcg-op.o
  CC      sh4eb-softmmu/hw/display/virtio-gpu.o
  CC      sparc-softmmu/tcg/tcg.o
  CC      sh4eb-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc64-softmmu/tcg/optimize.o
  CC      sparc-softmmu/tcg/tcg-op.o
  CC      sh4eb-softmmu/hw/display/virtio-gpu-pci.o
  CC      sparc64-softmmu/tcg/tcg-common.o
  CC      sh4-softmmu/tcg/optimize.o
  CC      sh4eb-softmmu/hw/intc/sh_intc.o
  CC      sparc64-softmmu/tcg/tcg-runtime.o
  CC      sh4eb-softmmu/hw/misc/ivshmem.o
  CC      sh4-softmmu/tcg/tcg-common.o
  CC      sparc64-softmmu/fpu/softfloat.o
  CC      sh4eb-softmmu/hw/misc/mmio_interface.o
  CC      sh4-softmmu/tcg/tcg-runtime.o
  CC      sh4eb-softmmu/hw/net/virtio-net.o
  CC      sh4-softmmu/fpu/softfloat.o
  CC      sparc-softmmu/tcg/optimize.o
  CC      sh4eb-softmmu/hw/net/vhost_net.o
  CC      sparc-softmmu/tcg/tcg-common.o
  CC      sh4eb-softmmu/hw/scsi/virtio-scsi.o
  CC      sparc64-softmmu/disas.o
  CC      sparc-softmmu/tcg/tcg-runtime.o
  CC      sh4eb-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      sparc64-softmmu/hax-stub.o
  CC      sparc-softmmu/fpu/softfloat.o
  CC      sparc64-softmmu/arch_init.o
  CC      sh4eb-softmmu/hw/scsi/vhost-scsi-common.o
  CC      sh4-softmmu/disas.o
  CC      sparc64-softmmu/cpus.o
  CC      sh4eb-softmmu/hw/scsi/vhost-scsi.o
  CC      sh4-softmmu/hax-stub.o
  CC      sparc64-softmmu/monitor.o
  CC      sh4-softmmu/arch_init.o
  CC      sh4eb-softmmu/hw/scsi/vhost-user-scsi.o
  CC      sh4-softmmu/cpus.o
  CC      sh4eb-softmmu/hw/timer/sh_timer.o
  CC      sh4-softmmu/monitor.o
  CC      sh4eb-softmmu/hw/timer/mc146818rtc.o
  CC      sparc64-softmmu/gdbstub.o
  CC      sparc-softmmu/disas.o
  CC      sh4eb-softmmu/hw/vfio/common.o
  CC      sparc-softmmu/hax-stub.o
  CC      sparc64-softmmu/balloon.o
  CC      sh4-softmmu/gdbstub.o
  CC      sparc-softmmu/arch_init.o
  CC      sh4eb-softmmu/hw/vfio/pci.o
  CC      sparc64-softmmu/ioport.o
  CC      sparc-softmmu/cpus.o
  CC      sh4-softmmu/balloon.o
  CC      sparc64-softmmu/numa.o
  CC      sparc-softmmu/monitor.o
  CC      sh4-softmmu/ioport.o
  CC      sh4eb-softmmu/hw/vfio/pci-quirks.o
  CC      sparc64-softmmu/qtest.o
  CC      sh4-softmmu/numa.o
  CC      sh4eb-softmmu/hw/vfio/platform.o
  CC      sparc64-softmmu/memory.o
  CC      sh4-softmmu/qtest.o
  CC      sh4eb-softmmu/hw/vfio/spapr.o
  CC      sh4eb-softmmu/hw/virtio/virtio.o
  CC      sparc-softmmu/gdbstub.o
  CC      sh4-softmmu/memory.o
  CC      sparc64-softmmu/memory_mapping.o
  CC      sh4eb-softmmu/hw/virtio/virtio-balloon.o
  CC      sparc-softmmu/balloon.o
  CC      sparc64-softmmu/dump.o
  CC      sparc-softmmu/ioport.o
  CC      sh4eb-softmmu/hw/virtio/vhost.o
  CC      sparc-softmmu/numa.o
  CC      sh4-softmmu/memory_mapping.o
  CC      sparc64-softmmu/migration/ram.o
  CC      sparc-softmmu/qtest.o
  CC      sh4-softmmu/dump.o
  CC      sh4eb-softmmu/hw/virtio/vhost-backend.o
  CC      sparc-softmmu/memory.o
  CC      sparc64-softmmu/accel/accel.o
  CC      sh4eb-softmmu/hw/virtio/vhost-user.o
  CC      sh4-softmmu/migration/ram.o
  CC      sparc64-softmmu/accel/stubs/kvm-stub.o
  CC      sh4eb-softmmu/hw/virtio/vhost-vsock.o
  CC      sh4eb-softmmu/hw/virtio/virtio-crypto.o
  CC      sparc64-softmmu/accel/tcg/tcg-all.o
  CC      sh4-softmmu/accel/accel.o
  CC      sparc-softmmu/memory_mapping.o
  CC      sparc64-softmmu/accel/tcg/cputlb.o
  CC      sh4eb-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      sh4-softmmu/accel/stubs/kvm-stub.o
  CC      sparc-softmmu/dump.o
  CC      sh4-softmmu/accel/tcg/tcg-all.o
  CC      sh4eb-softmmu/hw/sh4/shix.o
  CC      sh4-softmmu/accel/tcg/cputlb.o
  CC      sh4eb-softmmu/hw/sh4/r2d.o
  CC      sparc-softmmu/migration/ram.o
  CC      sh4eb-softmmu/hw/sh4/sh7750.o
  CC      sh4eb-softmmu/hw/sh4/sh7750_regnames.o
  CC      sparc64-softmmu/accel/tcg/cpu-exec.o
  CC      sh4eb-softmmu/hw/sh4/sh_pci.o
  CC      sh4eb-softmmu/target/sh4/translate.o
  CC      sparc-softmmu/accel/accel.o
  CC      sparc64-softmmu/accel/tcg/cpu-exec-common.o
  CC      sh4-softmmu/accel/tcg/cpu-exec.o
  CC      sparc-softmmu/accel/stubs/kvm-stub.o
  CC      sparc64-softmmu/accel/tcg/translate-all.o
  CC      sh4-softmmu/accel/tcg/cpu-exec-common.o
  CC      sparc-softmmu/accel/tcg/tcg-all.o
  CC      sparc64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sparc-softmmu/accel/tcg/cputlb.o
  CC      sh4-softmmu/accel/tcg/translate-all.o
  CC      sparc64-softmmu/hw/block/virtio-blk.o
  CC      sh4eb-softmmu/target/sh4/op_helper.o
  CC      sh4-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sh4eb-softmmu/target/sh4/helper.o
  CC      sparc64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      sh4-softmmu/hw/block/tc58128.o
  CC      sparc64-softmmu/hw/char/virtio-serial-bus.o
  CC      sh4eb-softmmu/target/sh4/cpu.o
  CC      sparc-softmmu/accel/tcg/cpu-exec.o
  CC      sh4-softmmu/hw/block/virtio-blk.o
  CC      sh4eb-softmmu/target/sh4/monitor.o
  CC      sparc-softmmu/accel/tcg/cpu-exec-common.o
  CC      sparc64-softmmu/hw/core/generic-loader.o
  CC      sh4eb-softmmu/target/sh4/gdbstub.o
  CC      sparc-softmmu/accel/tcg/translate-all.o
  CC      sh4-softmmu/hw/block/dataplane/virtio-blk.o
  GEN     trace/generated-helpers.c
  CC      sparc64-softmmu/hw/core/null-machine.o
  CC      sh4eb-softmmu/trace/control-target.o
  CC      sh4-softmmu/hw/char/sh_serial.o
  CC      sparc64-softmmu/hw/display/vga.o
  CC      sparc-softmmu/hw/core/generic-loader.o
  CC      sh4eb-softmmu/trace/generated-helpers.o
  CC      sh4-softmmu/hw/char/virtio-serial-bus.o
  CC      sparc-softmmu/hw/core/null-machine.o
  LINK    sh4eb-softmmu/qemu-system-sh4eb
  CC      sparc-softmmu/hw/display/tcx.o
  CC      sh4-softmmu/hw/core/generic-loader.o
  CC      sparc64-softmmu/hw/display/virtio-gpu.o
  CC      sh4-softmmu/hw/core/null-machine.o
  CC      sparc-softmmu/hw/display/cg3.o
  CC      sh4-softmmu/hw/display/sm501.o
  CC      sparc-softmmu/hw/intc/grlib_irqmp.o
  CC      sparc64-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc-softmmu/hw/misc/eccmemctl.o
  CC      sh4-softmmu/hw/display/vga.o
  CC      sparc-softmmu/hw/misc/slavio_misc.o
  CC      sparc64-softmmu/hw/display/virtio-gpu-pci.o
  CC      sparc-softmmu/hw/misc/mmio_interface.o
  GEN     tricore-softmmu/hmp-commands.h
  GEN     tricore-softmmu/hmp-commands-info.h
  GEN     tricore-softmmu/config-target.h
  CC      sparc64-softmmu/hw/misc/ivshmem.o
  CC      sparc-softmmu/hw/net/vhost_net.o
  CC      tricore-softmmu/exec.o
  CC      sh4-softmmu/hw/display/virtio-gpu.o
  CC      sparc-softmmu/hw/net/rocker/qmp-norocker.o
  CC      sparc64-softmmu/hw/misc/mmio_interface.o
  CC      sparc-softmmu/hw/vfio/common.o
  CC      sparc64-softmmu/hw/net/virtio-net.o
  CC      sh4-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc-softmmu/hw/vfio/platform.o
  CC      sparc-softmmu/hw/vfio/spapr.o
  CC      sparc64-softmmu/hw/net/vhost_net.o
  CC      sparc64-softmmu/hw/scsi/virtio-scsi.o
  CC      sh4-softmmu/hw/display/virtio-gpu-pci.o
  CC      sparc-softmmu/hw/sparc/sun4m.o
  CC      tricore-softmmu/tcg/tcg.o
  CC      sh4-softmmu/hw/intc/sh_intc.o
  CC      sparc64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      sparc64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      sparc-softmmu/hw/sparc/leon3.o
  CC      sh4-softmmu/hw/misc/ivshmem.o
  CC      sparc-softmmu/target/sparc/machine.o
  CC      sparc64-softmmu/hw/scsi/vhost-scsi.o
  CC      sparc-softmmu/target/sparc/monitor.o
  CC      sh4-softmmu/hw/misc/mmio_interface.o
  CC      sparc64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      sparc-softmmu/target/sparc/translate.o
  CC      sh4-softmmu/hw/net/virtio-net.o
  CC      sparc64-softmmu/hw/timer/mc146818rtc.o
  CC      tricore-softmmu/tcg/tcg-op.o
  CC      sparc64-softmmu/hw/vfio/common.o
  CC      sh4-softmmu/hw/net/vhost_net.o
  CC      sparc-softmmu/target/sparc/helper.o
  CC      sh4-softmmu/hw/scsi/virtio-scsi.o
  CC      sparc64-softmmu/hw/vfio/pci.o
  CC      sparc-softmmu/target/sparc/cpu.o
  CC      sh4-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      sparc-softmmu/target/sparc/fop_helper.o
  CC      sh4-softmmu/hw/scsi/vhost-scsi-common.o
  CC      sh4-softmmu/hw/scsi/vhost-scsi.o
  CC      sparc-softmmu/target/sparc/cc_helper.o
  CC      sparc64-softmmu/hw/vfio/pci-quirks.o
  CC      tricore-softmmu/tcg/optimize.o
  CC      sh4-softmmu/hw/scsi/vhost-user-scsi.o
  CC      sparc-softmmu/target/sparc/win_helper.o
  CC      sh4-softmmu/hw/timer/sh_timer.o
  CC      sparc-softmmu/target/sparc/mmu_helper.o
  CC      sparc64-softmmu/hw/vfio/platform.o
  CC      sh4-softmmu/hw/timer/mc146818rtc.o
  CC      sparc-softmmu/target/sparc/ldst_helper.o
  CC      tricore-softmmu/tcg/tcg-common.o
  CC      sparc64-softmmu/hw/vfio/spapr.o
  CC      tricore-softmmu/tcg/tcg-runtime.o
  CC      sparc-softmmu/target/sparc/int32_helper.o
  CC      sh4-softmmu/hw/vfio/common.o
  CC      tricore-softmmu/fpu/softfloat.o
  CC      sparc64-softmmu/hw/virtio/virtio.o
  CC      sparc-softmmu/target/sparc/gdbstub.o
  GEN     trace/generated-helpers.c
  CC      sh4-softmmu/hw/vfio/pci.o
  CC      sparc-softmmu/trace/control-target.o
  CC      sparc64-softmmu/hw/virtio/virtio-balloon.o
  CC      sparc-softmmu/trace/generated-helpers.o
  CC      sparc64-softmmu/hw/virtio/vhost.o
  LINK    sparc-softmmu/qemu-system-sparc
  CC      sh4-softmmu/hw/vfio/pci-quirks.o
  CC      sparc64-softmmu/hw/virtio/vhost-backend.o
  CC      tricore-softmmu/disas.o
  CC      sh4-softmmu/hw/vfio/platform.o
  CC      sparc64-softmmu/hw/virtio/vhost-user.o
  CC      tricore-softmmu/hax-stub.o
  CC      tricore-softmmu/arch_init.o
  CC      sh4-softmmu/hw/vfio/spapr.o
  CC      tricore-softmmu/cpus.o
  CC      sh4-softmmu/hw/virtio/virtio.o
  CC      sparc64-softmmu/hw/virtio/vhost-vsock.o
  CC      sparc64-softmmu/hw/virtio/virtio-crypto.o
  CC      sh4-softmmu/hw/virtio/virtio-balloon.o
  CC      tricore-softmmu/monitor.o
  CC      sparc64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      sh4-softmmu/hw/virtio/vhost.o
  CC      sparc64-softmmu/hw/sparc64/sparc64.o
  CC      sh4-softmmu/hw/virtio/vhost-backend.o
  CC      sparc64-softmmu/hw/sparc64/sun4u.o
  CC      sparc64-softmmu/hw/sparc64/niagara.o
  CC      sh4-softmmu/hw/virtio/vhost-user.o
  CC      sh4-softmmu/hw/virtio/vhost-vsock.o
  CC      tricore-softmmu/gdbstub.o
  CC      sparc64-softmmu/target/sparc/machine.o
  CC      sparc64-softmmu/target/sparc/monitor.o
  CC      sh4-softmmu/hw/virtio/virtio-crypto.o
  CC      tricore-softmmu/balloon.o
  CC      sparc64-softmmu/target/sparc/translate.o
  CC      sh4-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      tricore-softmmu/ioport.o
  GEN     unicore32-softmmu/hmp-commands.h
  GEN     unicore32-softmmu/hmp-commands-info.h
  GEN     unicore32-softmmu/config-target.h
  CC      sh4-softmmu/hw/sh4/shix.o
  CC      unicore32-softmmu/exec.o
  CC      tricore-softmmu/numa.o
  CC      sh4-softmmu/hw/sh4/r2d.o
  CC      tricore-softmmu/qtest.o
  CC      sh4-softmmu/hw/sh4/sh7750.o
  CC      tricore-softmmu/memory.o
  CC      sh4-softmmu/hw/sh4/sh7750_regnames.o
  CC      unicore32-softmmu/tcg/tcg.o
  CC      sparc64-softmmu/target/sparc/helper.o
  CC      sh4-softmmu/hw/sh4/sh_pci.o
  CC      sparc64-softmmu/target/sparc/cpu.o
  CC      tricore-softmmu/memory_mapping.o
  CC      sh4-softmmu/target/sh4/translate.o
  CC      tricore-softmmu/dump.o
  CC      sparc64-softmmu/target/sparc/fop_helper.o
  CC      sparc64-softmmu/target/sparc/cc_helper.o
  CC      unicore32-softmmu/tcg/tcg-op.o
  CC      tricore-softmmu/migration/ram.o
  CC      sparc64-softmmu/target/sparc/win_helper.o
  CC      sh4-softmmu/target/sh4/op_helper.o
  CC      sparc64-softmmu/target/sparc/mmu_helper.o
  CC      tricore-softmmu/accel/accel.o
  CC      sh4-softmmu/target/sh4/helper.o
  CC      sparc64-softmmu/target/sparc/ldst_helper.o
  CC      tricore-softmmu/accel/stubs/kvm-stub.o
  CC      sparc64-softmmu/target/sparc/int64_helper.o
  CC      tricore-softmmu/accel/tcg/tcg-all.o
  CC      sh4-softmmu/target/sh4/cpu.o
  CC      tricore-softmmu/accel/tcg/cputlb.o
  CC      sparc64-softmmu/target/sparc/vis_helper.o
  CC      sh4-softmmu/target/sh4/monitor.o
  CC      unicore32-softmmu/tcg/optimize.o
  CC      sparc64-softmmu/target/sparc/gdbstub.o
  CC      sh4-softmmu/target/sh4/gdbstub.o
  GEN     trace/generated-helpers.c
  CC      sparc64-softmmu/trace/control-target.o
  GEN     trace/generated-helpers.c
  CC      sh4-softmmu/trace/control-target.o
  CC      sparc64-softmmu/trace/generated-helpers.o
  CC      unicore32-softmmu/tcg/tcg-common.o
  CC      sh4-softmmu/trace/generated-helpers.o
  CC      tricore-softmmu/accel/tcg/cpu-exec.o
  CC      unicore32-softmmu/tcg/tcg-runtime.o
  LINK    sparc64-softmmu/qemu-system-sparc64
  CC      tricore-softmmu/accel/tcg/cpu-exec-common.o
  LINK    sh4-softmmu/qemu-system-sh4
  CC      tricore-softmmu/accel/tcg/translate-all.o
  CC      unicore32-softmmu/fpu/softfloat.o
  CC      tricore-softmmu/hw/core/generic-loader.o
  CC      tricore-softmmu/hw/core/null-machine.o
  CC      unicore32-softmmu/disas.o
  CC      tricore-softmmu/hw/misc/mmio_interface.o
  CC      tricore-softmmu/hw/net/vhost_net.o
  GEN     x86_64-softmmu/hmp-commands.h
  GEN     xtensaeb-softmmu/hmp-commands.h
  GEN     x86_64-softmmu/hmp-commands-info.h
  GEN     x86_64-softmmu/config-target.h
  CC      tricore-softmmu/hw/net/rocker/qmp-norocker.o
  GEN     xtensaeb-softmmu/hmp-commands-info.h
  GEN     xtensaeb-softmmu/config-target.h
  CC      x86_64-softmmu/exec.o
  CC      xtensaeb-softmmu/exec.o
  CC      tricore-softmmu/hw/vfio/common.o
  CC      unicore32-softmmu/hax-stub.o
  CC      tricore-softmmu/hw/vfio/platform.o
  CC      unicore32-softmmu/arch_init.o
  CC      tricore-softmmu/hw/vfio/spapr.o
  CC      unicore32-softmmu/cpus.o
  CC      tricore-softmmu/hw/tricore/tricore_testboard.o
  CC      x86_64-softmmu/tcg/tcg.o
  CC      xtensaeb-softmmu/tcg/tcg.o
  CC      tricore-softmmu/target/tricore/translate.o
  CC      unicore32-softmmu/monitor.o
  CC      xtensaeb-softmmu/tcg/tcg-op.o
  CC      unicore32-softmmu/gdbstub.o
  CC      x86_64-softmmu/tcg/tcg-op.o
  CC      unicore32-softmmu/balloon.o
  CC      unicore32-softmmu/ioport.o
  CC      unicore32-softmmu/numa.o
  CC      tricore-softmmu/target/tricore/helper.o
  CC      xtensaeb-softmmu/tcg/optimize.o
  CC      tricore-softmmu/target/tricore/cpu.o
  CC      unicore32-softmmu/qtest.o
  CC      x86_64-softmmu/tcg/optimize.o
  CC      tricore-softmmu/target/tricore/op_helper.o
  CC      unicore32-softmmu/memory.o
  CC      xtensaeb-softmmu/tcg/tcg-common.o
  CC      xtensaeb-softmmu/tcg/tcg-runtime.o
  CC      x86_64-softmmu/tcg/tcg-common.o
  CC      x86_64-softmmu/tcg/tcg-runtime.o
  CC      unicore32-softmmu/memory_mapping.o
  CC      xtensaeb-softmmu/fpu/softfloat.o
  CC      tricore-softmmu/target/tricore/fpu_helper.o
  CC      x86_64-softmmu/fpu/softfloat.o
  CC      unicore32-softmmu/dump.o
  GEN     trace/generated-helpers.c
  CC      tricore-softmmu/trace/control-target.o
  CC      unicore32-softmmu/migration/ram.o
  CC      tricore-softmmu/trace/generated-helpers.o
  CC      unicore32-softmmu/accel/accel.o
  LINK    tricore-softmmu/qemu-system-tricore
  CC      unicore32-softmmu/accel/stubs/kvm-stub.o
  CC      xtensaeb-softmmu/disas.o
  CC      unicore32-softmmu/accel/tcg/tcg-all.o
  CC      x86_64-softmmu/disas.o
  GEN     x86_64-softmmu/gdbstub-xml.c
  CC      xtensaeb-softmmu/hax-stub.o
  CC      unicore32-softmmu/accel/tcg/cputlb.o
  CC      x86_64-softmmu/hax-stub.o
  CC      xtensaeb-softmmu/arch_init.o
  GEN     xtensa-softmmu/hmp-commands.h
  GEN     xtensa-softmmu/hmp-commands-info.h
  GEN     xtensa-softmmu/config-target.h
  CC      x86_64-softmmu/arch_init.o
  CC      xtensa-softmmu/exec.o
  CC      xtensaeb-softmmu/cpus.o
  CC      x86_64-softmmu/cpus.o
  CC      xtensaeb-softmmu/monitor.o
  CC      unicore32-softmmu/accel/tcg/cpu-exec.o
  CC      x86_64-softmmu/monitor.o
  CC      unicore32-softmmu/accel/tcg/cpu-exec-common.o
  CC      unicore32-softmmu/accel/tcg/translate-all.o
  CC      xtensa-softmmu/tcg/tcg.o
  CC      xtensaeb-softmmu/gdbstub.o
  CC      unicore32-softmmu/hw/core/generic-loader.o
  CC      unicore32-softmmu/hw/core/null-machine.o
  CC      xtensaeb-softmmu/balloon.o
  CC      x86_64-softmmu/gdbstub.o
  CC      unicore32-softmmu/hw/misc/mmio_interface.o
  CC      xtensaeb-softmmu/ioport.o
  CC      unicore32-softmmu/hw/net/vhost_net.o
  CC      xtensa-softmmu/tcg/tcg-op.o
  CC      xtensaeb-softmmu/numa.o
  CC      x86_64-softmmu/balloon.o
  CC      unicore32-softmmu/hw/net/rocker/qmp-norocker.o
  CC      x86_64-softmmu/ioport.o
  CC      xtensaeb-softmmu/qtest.o
  CC      unicore32-softmmu/hw/vfio/common.o
  CC      x86_64-softmmu/numa.o
  CC      xtensaeb-softmmu/memory.o
  CC      unicore32-softmmu/hw/vfio/platform.o
  CC      x86_64-softmmu/qtest.o
  CC      unicore32-softmmu/hw/vfio/spapr.o
  CC      xtensaeb-softmmu/memory_mapping.o
  CC      x86_64-softmmu/memory.o
  CC      unicore32-softmmu/hw/unicore32/puv3.o
  CC      xtensaeb-softmmu/dump.o
  CC      xtensa-softmmu/tcg/optimize.o
  CC      unicore32-softmmu/target/unicore32/translate.o
  CC      xtensaeb-softmmu/migration/ram.o
  CC      x86_64-softmmu/memory_mapping.o
  CC      xtensa-softmmu/tcg/tcg-common.o
  CC      x86_64-softmmu/dump.o
  CC      xtensa-softmmu/tcg/tcg-runtime.o
  CC      unicore32-softmmu/target/unicore32/op_helper.o
  CC      xtensaeb-softmmu/accel/accel.o
  CC      xtensa-softmmu/fpu/softfloat.o
  CC      unicore32-softmmu/target/unicore32/helper.o
  CC      xtensaeb-softmmu/accel/stubs/kvm-stub.o
  CC      x86_64-softmmu/migration/ram.o
  CC      unicore32-softmmu/target/unicore32/cpu.o
  CC      xtensaeb-softmmu/accel/tcg/tcg-all.o
  CC      unicore32-softmmu/target/unicore32/ucf64_helper.o
  CC      xtensaeb-softmmu/accel/tcg/cputlb.o
  CC      unicore32-softmmu/target/unicore32/softmmu.o
  CC      x86_64-softmmu/accel/accel.o
  CC      x86_64-softmmu/accel/stubs/kvm-stub.o
  GEN     trace/generated-helpers.c
  CC      unicore32-softmmu/trace/control-target.o
  CC      x86_64-softmmu/accel/tcg/tcg-all.o
  CC      unicore32-softmmu/trace/generated-helpers.o
  CC      x86_64-softmmu/accel/tcg/cputlb.o
  CC      xtensaeb-softmmu/accel/tcg/cpu-exec.o
  CC      xtensa-softmmu/disas.o
  LINK    unicore32-softmmu/qemu-system-unicore32
  CC      xtensa-softmmu/hax-stub.o
  CC      xtensaeb-softmmu/accel/tcg/cpu-exec-common.o
  CC      xtensa-softmmu/arch_init.o
  CC      xtensaeb-softmmu/accel/tcg/translate-all.o
  CC      xtensa-softmmu/cpus.o
  CC      x86_64-softmmu/accel/tcg/cpu-exec.o
  CC      xtensaeb-softmmu/hw/core/generic-loader.o
  CC      xtensaeb-softmmu/hw/core/null-machine.o
  GEN     aarch64-linux-user/config-target.h
  CC      aarch64-linux-user/exec.o
  CC      xtensa-softmmu/monitor.o
  CC      x86_64-softmmu/accel/tcg/cpu-exec-common.o
  CC      xtensaeb-softmmu/hw/misc/mmio_interface.o
  CC      x86_64-softmmu/accel/tcg/translate-all.o
  CC      aarch64-linux-user/tcg/tcg.o
  CC      xtensaeb-softmmu/hw/net/vhost_net.o
  CC      x86_64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      xtensaeb-softmmu/hw/net/rocker/qmp-norocker.o
  CC      xtensaeb-softmmu/hw/vfio/common.o
  CC      x86_64-softmmu/hw/block/virtio-blk.o
  CC      xtensa-softmmu/gdbstub.o
  CC      xtensa-softmmu/balloon.o
  CC      xtensaeb-softmmu/hw/vfio/platform.o
  CC      x86_64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      xtensa-softmmu/ioport.o
  CC      aarch64-linux-user/tcg/tcg-op.o
  CC      x86_64-softmmu/hw/char/virtio-serial-bus.o
  CC      xtensaeb-softmmu/hw/vfio/spapr.o
  CC      xtensa-softmmu/numa.o
  CC      xtensaeb-softmmu/hw/xtensa/pic_cpu.o
  CC      x86_64-softmmu/hw/core/generic-loader.o
  CC      xtensa-softmmu/qtest.o
  CC      xtensaeb-softmmu/hw/xtensa/sim.o
  CC      x86_64-softmmu/hw/core/null-machine.o
  CC      xtensaeb-softmmu/hw/xtensa/xtfpga.o
  CC      xtensa-softmmu/memory.o
  CC      x86_64-softmmu/hw/display/vga.o
  CC      xtensaeb-softmmu/target/xtensa/xtensa-semi.o
  CC      xtensaeb-softmmu/target/xtensa/core-dc232b.o
  CC      x86_64-softmmu/hw/display/virtio-gpu.o
  CC      xtensa-softmmu/memory_mapping.o
  CC      aarch64-linux-user/tcg/optimize.o
  CC      xtensaeb-softmmu/target/xtensa/core-dc233c.o
  CC      xtensa-softmmu/dump.o
  CC      xtensaeb-softmmu/target/xtensa/core-fsf.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-3d.o
  CC      xtensaeb-softmmu/target/xtensa/monitor.o
  CC      xtensa-softmmu/migration/ram.o
  CC      aarch64-linux-user/tcg/tcg-common.o
  CC      xtensaeb-softmmu/target/xtensa/translate.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-linux-user/tcg/tcg-runtime.o
  CC      xtensa-softmmu/accel/accel.o
  CC      x86_64-softmmu/hw/display/virtio-vga.o
  CC      xtensa-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-linux-user/fpu/softfloat.o
  CC      x86_64-softmmu/hw/intc/apic.o
  CC      xtensa-softmmu/accel/tcg/tcg-all.o
  CC      xtensa-softmmu/accel/tcg/cputlb.o
  CC      xtensaeb-softmmu/target/xtensa/op_helper.o
  CC      x86_64-softmmu/hw/intc/apic_common.o
  CC      xtensaeb-softmmu/target/xtensa/helper.o
  CC      x86_64-softmmu/hw/intc/ioapic.o
  CC      xtensaeb-softmmu/target/xtensa/cpu.o
  CC      x86_64-softmmu/hw/isa/lpc_ich9.o
  CC      xtensa-softmmu/accel/tcg/cpu-exec.o
  CC      xtensaeb-softmmu/target/xtensa/gdbstub.o
  CC      x86_64-softmmu/hw/misc/vmport.o
  GEN     trace/generated-helpers.c
  CC      xtensaeb-softmmu/trace/control-target.o
  CC      xtensa-softmmu/accel/tcg/cpu-exec-common.o
  CC      x86_64-softmmu/hw/misc/ivshmem.o
  CC      aarch64-linux-user/disas.o
  CC      xtensaeb-softmmu/trace/generated-helpers.o
  CC      xtensa-softmmu/accel/tcg/translate-all.o
  CC      x86_64-softmmu/hw/misc/pvpanic.o
  GEN     aarch64-linux-user/gdbstub-xml.c
  LINK    xtensaeb-softmmu/qemu-system-xtensaeb
  CC      aarch64-linux-user/hax-stub.o
  CC      xtensa-softmmu/hw/core/generic-loader.o
  CC      x86_64-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-linux-user/gdbstub.o
  CC      x86_64-softmmu/hw/net/virtio-net.o
  CC      xtensa-softmmu/hw/core/null-machine.o
  CC      xtensa-softmmu/hw/misc/mmio_interface.o
  CC      x86_64-softmmu/hw/net/vhost_net.o
  CC      aarch64-linux-user/thunk.o
  CC      xtensa-softmmu/hw/net/vhost_net.o
  CC      xtensa-softmmu/hw/net/rocker/qmp-norocker.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi.o
  CC      xtensa-softmmu/hw/vfio/common.o
  CC      xtensa-softmmu/hw/vfio/platform.o
  CC      aarch64-linux-user/user-exec.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-linux-user/user-exec-stub.o
  CC      x86_64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      xtensa-softmmu/hw/vfio/spapr.o
  CC      x86_64-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-linux-user/accel/stubs/kvm-stub.o
  CC      xtensa-softmmu/hw/xtensa/pic_cpu.o
  CC      x86_64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-linux-user/accel/tcg/cpu-exec.o
  CC      x86_64-softmmu/hw/timer/mc146818rtc.o
  CC      xtensa-softmmu/hw/xtensa/sim.o
  CC      xtensa-softmmu/hw/xtensa/xtfpga.o
  CC      x86_64-softmmu/hw/vfio/common.o
  CC      xtensa-softmmu/target/xtensa/xtensa-semi.o
  CC      aarch64-linux-user/accel/tcg/cpu-exec-common.o
  CC      x86_64-softmmu/hw/vfio/pci.o
  CC      aarch64-linux-user/accel/tcg/translate-all.o
  CC      xtensa-softmmu/target/xtensa/core-dc232b.o
  CC      xtensa-softmmu/target/xtensa/core-dc233c.o
  CC      aarch64-linux-user/linux-user/main.o
  CC      xtensa-softmmu/target/xtensa/core-fsf.o
  CC      x86_64-softmmu/hw/vfio/pci-quirks.o
  CC      xtensa-softmmu/target/xtensa/monitor.o
  GEN     alpha-linux-user/config-target.h
  CC      alpha-linux-user/exec.o
  CC      aarch64-linux-user/linux-user/syscall.o
  CC      xtensa-softmmu/target/xtensa/translate.o
  CC      x86_64-softmmu/hw/vfio/platform.o
  CC      alpha-linux-user/tcg/tcg.o
  CC      x86_64-softmmu/hw/vfio/spapr.o
  CC      x86_64-softmmu/hw/virtio/virtio.o
  CC      alpha-linux-user/tcg/tcg-op.o
  CC      xtensa-softmmu/target/xtensa/op_helper.o
  CC      x86_64-softmmu/hw/virtio/virtio-balloon.o
  CC      xtensa-softmmu/target/xtensa/helper.o
  CC      aarch64-linux-user/linux-user/strace.o
  CC      x86_64-softmmu/hw/virtio/vhost.o
  CC      xtensa-softmmu/target/xtensa/cpu.o
  CC      aarch64-linux-user/linux-user/mmap.o
  CC      xtensa-softmmu/target/xtensa/gdbstub.o
  CC      alpha-linux-user/tcg/optimize.o
  CC      aarch64-linux-user/linux-user/signal.o
  CC      x86_64-softmmu/hw/virtio/vhost-backend.o
  GEN     trace/generated-helpers.c
  CC      x86_64-softmmu/hw/virtio/vhost-user.o
  CC      xtensa-softmmu/trace/control-target.o
  CC      aarch64-linux-user/linux-user/elfload.o
  CC      xtensa-softmmu/trace/generated-helpers.o
  CC      alpha-linux-user/tcg/tcg-common.o
  CC      x86_64-softmmu/hw/virtio/vhost-vsock.o
  CC      alpha-linux-user/tcg/tcg-runtime.o
  LINK    xtensa-softmmu/qemu-system-xtensa
  CC      aarch64-linux-user/linux-user/linuxload.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto.o
  CC      alpha-linux-user/fpu/softfloat.o
  CC      aarch64-linux-user/linux-user/uaccess.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      alpha-linux-user/disas.o
  CC      aarch64-linux-user/linux-user/uname.o
  CCAS    aarch64-linux-user/linux-user/safe-syscall.o
  GEN     armeb-linux-user/config-target.h
  CC      x86_64-softmmu/hw/i386/multiboot.o
  CC      armeb-linux-user/exec.o
  CC      aarch64-linux-user/linux-user/flatload.o
  CC      x86_64-softmmu/hw/i386/pc.o
  CC      armeb-linux-user/tcg/tcg.o
  CC      aarch64-linux-user/target/arm/arm-semi.o
  CC      aarch64-linux-user/target/arm/kvm-stub.o
  CC      x86_64-softmmu/hw/i386/pc_piix.o
In file included from /var/tmp/patchew-tester-tmp-hm2hdbnw/src/target/arm/kvm-stub.c:15:0:
/var/tmp/patchew-tester-tmp-hm2hdbnw/src/target/arm/kvm_arm.h:291:28: error: unknown type name ‘ram_addr_t’
 void kvm_hwpoison_page_add(ram_addr_t ram_addr);
                            ^~~~~~~~~~
/var/tmp/patchew-tester-tmp-hm2hdbnw/src/rules.mak:66: recipe for target 'target/arm/kvm-stub.o' failed
make[1]: *** [target/arm/kvm-stub.o] Error 1
Makefile:326: recipe for target 'subdir-aarch64-linux-user' failed
make: *** [subdir-aarch64-linux-user] Error 2
make: *** Waiting for unfinished jobs....
  CC      alpha-linux-user/hax-stub.o
  CC      x86_64-softmmu/hw/i386/pc_q35.o
  CC      alpha-linux-user/gdbstub.o
  CC      armeb-linux-user/tcg/tcg-op.o
  CC      x86_64-softmmu/hw/i386/pc_sysfw.o
  CC      x86_64-softmmu/hw/i386/x86-iommu.o
  CC      alpha-linux-user/thunk.o
  CC      armeb-linux-user/tcg/optimize.o
  CC      alpha-linux-user/user-exec.o
  CC      x86_64-softmmu/hw/i386/intel_iommu.o
  CC      alpha-linux-user/user-exec-stub.o
  CC      x86_64-softmmu/hw/i386/amd_iommu.o
  CC      alpha-linux-user/accel/stubs/kvm-stub.o
  CC      alpha-linux-user/accel/tcg/cpu-exec.o
  CC      x86_64-softmmu/hw/i386/kvmvapic.o
  CC      x86_64-softmmu/hw/i386/acpi-build.o
  CC      armeb-linux-user/tcg/tcg-common.o
  CC      alpha-linux-user/accel/tcg/cpu-exec-common.o
  CC      alpha-linux-user/accel/tcg/translate-all.o
  CC      armeb-linux-user/tcg/tcg-runtime.o
  CC      x86_64-softmmu/hw/i386/pci-assign-load-rom.o
  CC      alpha-linux-user/linux-user/main.o
  CC      armeb-linux-user/fpu/softfloat.o
  CC      alpha-linux-user/linux-user/syscall.o
  CC      x86_64-softmmu/target/i386/helper.o
  CC      alpha-linux-user/linux-user/strace.o
  CC      x86_64-softmmu/target/i386/cpu.o
  CC      x86_64-softmmu/target/i386/gdbstub.o
  CC      armeb-linux-user/disas.o
  CC      x86_64-softmmu/target/i386/xsave_helper.o
  CC      x86_64-softmmu/target/i386/translate.o
  GEN     armeb-linux-user/gdbstub-xml.c
  CC      alpha-linux-user/linux-user/mmap.o
  CC      x86_64-softmmu/target/i386/bpt_helper.o
  CC      armeb-linux-user/hax-stub.o
  CC      alpha-linux-user/linux-user/signal.o
  CC      alpha-linux-user/linux-user/elfload.o
  CC      armeb-linux-user/gdbstub.o
  CC      alpha-linux-user/linux-user/linuxload.o
  CC      x86_64-softmmu/target/i386/cc_helper.o
  CC      armeb-linux-user/thunk.o
  CC      alpha-linux-user/linux-user/uaccess.o
  CC      alpha-linux-user/linux-user/uname.o
  CC      armeb-linux-user/user-exec.o
  CCAS    alpha-linux-user/linux-user/safe-syscall.o
  CC      alpha-linux-user/target/alpha/translate.o
  CC      alpha-linux-user/target/alpha/helper.o
  CC      armeb-linux-user/user-exec-stub.o
  CC      x86_64-softmmu/target/i386/excp_helper.o
  CC      armeb-linux-user/accel/stubs/kvm-stub.o
  CC      alpha-linux-user/target/alpha/cpu.o
  CC      armeb-linux-user/accel/tcg/cpu-exec.o
  CC      alpha-linux-user/target/alpha/int_helper.o
  CC      x86_64-softmmu/target/i386/fpu_helper.o
  CC      armeb-linux-user/accel/tcg/cpu-exec-common.o
  CC      alpha-linux-user/target/alpha/fpu_helper.o
  CC      alpha-linux-user/target/alpha/vax_helper.o
  CC      armeb-linux-user/accel/tcg/translate-all.o
  CC      alpha-linux-user/target/alpha/sys_helper.o
  CC      alpha-linux-user/target/alpha/mem_helper.o
  CC      armeb-linux-user/linux-user/main.o
  CC      armeb-linux-user/linux-user/syscall.o
  CC      alpha-linux-user/target/alpha/gdbstub.o
  CC      armeb-linux-user/linux-user/strace.o
  GEN     trace/generated-helpers.c
  CC      alpha-linux-user/trace/control-target.o
  CC      alpha-linux-user/trace/generated-helpers.o
  CC      x86_64-softmmu/target/i386/int_helper.o
  CC      x86_64-softmmu/target/i386/mem_helper.o
  LINK    alpha-linux-user/qemu-alpha
  CC      armeb-linux-user/linux-user/mmap.o
  CC      x86_64-softmmu/target/i386/misc_helper.o
  CC      armeb-linux-user/linux-user/signal.o
  CC      x86_64-softmmu/target/i386/mpx_helper.o
  CC      x86_64-softmmu/target/i386/seg_helper.o
  CC      x86_64-softmmu/target/i386/smm_helper.o
  CC      x86_64-softmmu/target/i386/svm_helper.o
  CC      x86_64-softmmu/target/i386/machine.o
  CC      x86_64-softmmu/target/i386/arch_memory_mapping.o
  CC      x86_64-softmmu/target/i386/arch_dump.o
  CC      x86_64-softmmu/target/i386/monitor.o
  CC      x86_64-softmmu/target/i386/kvm-stub.o
  GEN     trace/generated-helpers.c
  CC      x86_64-softmmu/trace/control-target.o
  CC      x86_64-softmmu/gdbstub-xml.o
  CC      armeb-linux-user/linux-user/elfload.o
  CC      x86_64-softmmu/trace/generated-helpers.o
  CC      armeb-linux-user/linux-user/linuxload.o
  CC      armeb-linux-user/linux-user/uaccess.o
  CC      armeb-linux-user/linux-user/uname.o
  LINK    x86_64-softmmu/qemu-system-x86_64
  CCAS    armeb-linux-user/linux-user/safe-syscall.o
  CC      armeb-linux-user/linux-user/flatload.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11_cpdo.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11_cpdt.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11_cprt.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpopcode.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/double_cpdo.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/single_cpdo.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/extended_cpdo.o
  CC      armeb-linux-user/target/arm/arm-semi.o
  CC      armeb-linux-user/target/arm/kvm-stub.o
  CC      armeb-linux-user/target/arm/translate.o
In file included from /var/tmp/patchew-tester-tmp-hm2hdbnw/src/target/arm/kvm-stub.c:15:0:
/var/tmp/patchew-tester-tmp-hm2hdbnw/src/target/arm/kvm_arm.h:291:28: error: unknown type name ‘ram_addr_t’
 void kvm_hwpoison_page_add(ram_addr_t ram_addr);
                            ^~~~~~~~~~
/var/tmp/patchew-tester-tmp-hm2hdbnw/src/rules.mak:66: recipe for target 'target/arm/kvm-stub.o' failed
make[1]: *** [target/arm/kvm-stub.o] Error 1
make[1]: *** Waiting for unfinished jobs....
Makefile:326: recipe for target 'subdir-armeb-linux-user' failed
make: *** [subdir-armeb-linux-user] Error 2
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
@ 2017-08-18 14:21   ` no-reply
  0 siblings, 0 replies; 129+ messages in thread
From: no-reply @ 2017-08-18 14:21 UTC (permalink / raw)
  To: gengdongjiu
  Cc: famz, mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, zhengqiang10,
	wuquanming, huangshaoyu, linuxarm, gengdongjiu

Hi,

This series failed build test on s390x host. Please find the details below.

Type: series
Message-id: 1503066227-18251-1-git-send-email-gengdongjiu@huawei.com
Subject: [Qemu-devel] [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e
echo "=== ENV ==="
env
echo "=== PACKAGES ==="
rpm -qa
echo "=== TEST BEGIN ==="
CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
echo -n "Using CC: "
realpath $CC
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1
make install
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
>From https://github.com/patchew-project/qemu
 * [new tag]         patchew/1503066227-18251-1-git-send-email-gengdongjiu@huawei.com -> patchew/1503066227-18251-1-git-send-email-gengdongjiu@huawei.com
Switched to a new branch 'test'
06d4dbd target-arm: kvm64: Handle SError interrupt for the guest OS
8e5e362 target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
5d9b3fa target-arm: kvm64: detect guest RAS EXTENSION feature
e79f876 ACPI: build and enable APEI GHES in the Makefile and configuration
5e11799 ACPI: Add APEI GHES Table Generation support
d691b66 ACPI: add APEI/HEST/CPER structures and macros

=== OUTPUT BEGIN ===
=== ENV ===
XDG_SESSION_ID=32305
SHELL=/bin/sh
USER=fam
PATCHEW=/home/fam/patchew/patchew-cli -s http://patchew.org --nodebug
PATH=/usr/bin:/bin
PWD=/var/tmp/patchew-tester-tmp-hm2hdbnw/src
LANG=en_US.UTF-8
HOME=/home/fam
SHLVL=2
LOGNAME=fam
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1012/bus
XDG_RUNTIME_DIR=/run/user/1012
_=/usr/bin/env
=== PACKAGES ===
gpg-pubkey-873529b8-54e386ff
xz-libs-5.2.2-2.fc24.s390x
libxshmfence-1.2-3.fc24.s390x
giflib-4.1.6-15.fc24.s390x
trousers-lib-0.3.13-6.fc24.s390x
ncurses-base-6.0-6.20160709.fc25.noarch
gmp-6.1.1-1.fc25.s390x
libidn-1.33-1.fc25.s390x
slang-2.3.0-7.fc25.s390x
pkgconfig-0.29.1-1.fc25.s390x
alsa-lib-1.1.1-2.fc25.s390x
yum-metadata-parser-1.1.4-17.fc25.s390x
python3-slip-dbus-0.6.4-4.fc25.noarch
python2-cssselect-0.9.2-1.fc25.noarch
createrepo_c-libs-0.10.0-6.fc25.s390x
initscripts-9.69-1.fc25.s390x
parted-3.2-21.fc25.s390x
flex-2.6.0-3.fc25.s390x
colord-libs-1.3.4-1.fc25.s390x
python-osbs-client-0.33-3.fc25.noarch
perl-Pod-Simple-3.35-1.fc25.noarch
python2-simplejson-3.10.0-1.fc25.s390x
brltty-5.4-2.fc25.s390x
librados2-10.2.4-2.fc25.s390x
tcp_wrappers-7.6-83.fc25.s390x
libcephfs_jni1-10.2.4-2.fc25.s390x
nettle-devel-3.3-1.fc25.s390x
bzip2-devel-1.0.6-21.fc25.s390x
libuuid-2.28.2-2.fc25.s390x
python3-dnf-1.1.10-6.fc25.noarch
texlive-kpathsea-doc-svn41139-33.fc25.1.noarch
openssh-7.4p1-4.fc25.s390x
texlive-kpathsea-bin-svn40473-33.20160520.fc25.1.s390x
texlive-graphics-svn41015-33.fc25.1.noarch
texlive-dvipdfmx-def-svn40328-33.fc25.1.noarch
texlive-mfware-svn40768-33.fc25.1.noarch
texlive-texlive-scripts-svn41433-33.fc25.1.noarch
texlive-euro-svn22191.1.1-33.fc25.1.noarch
texlive-etex-svn37057.0-33.fc25.1.noarch
texlive-iftex-svn29654.0.2-33.fc25.1.noarch
texlive-palatino-svn31835.0-33.fc25.1.noarch
texlive-texlive-docindex-svn41430-33.fc25.1.noarch
texlive-xunicode-svn30466.0.981-33.fc25.1.noarch
texlive-koma-script-svn41508-33.fc25.1.noarch
texlive-pst-grad-svn15878.1.06-33.fc25.1.noarch
texlive-pst-blur-svn15878.2.0-33.fc25.1.noarch
texlive-jknapltx-svn19440.0-33.fc25.1.noarch
texinfo-6.1-4.fc25.s390x
openssl-devel-1.0.2k-1.fc25.s390x
gdk-pixbuf2-2.36.6-1.fc25.s390x
nspr-4.14.0-2.fc25.s390x
nss-softokn-freebl-3.30.2-1.0.fc25.s390x
jansson-2.10-2.fc25.s390x
fedora-repos-25-4.noarch
python3-libs-3.5.3-6.fc25.s390x
perl-Errno-1.25-387.fc25.s390x
acl-2.2.52-13.fc25.s390x
pcre2-utf16-10.23-8.fc25.s390x
pango-1.40.5-1.fc25.s390x
systemd-pam-231-17.fc25.s390x
python2-gluster-3.10.4-1.fc25.s390x
NetworkManager-libnm-1.4.4-5.fc25.s390x
selinux-policy-3.13.1-225.18.fc25.noarch
poppler-0.45.0-5.fc25.s390x
ccache-3.3.4-1.fc25.s390x
valgrind-3.12.0-9.fc25.s390x
perl-open-1.10-387.fc25.noarch
libaio-0.3.110-6.fc24.s390x
libfontenc-1.1.3-3.fc24.s390x
lzo-2.08-8.fc24.s390x
isl-0.14-5.fc24.s390x
libXau-1.0.8-6.fc24.s390x
linux-atm-libs-2.5.1-14.fc24.s390x
libXext-1.3.3-4.fc24.s390x
libXxf86vm-1.1.4-3.fc24.s390x
bison-3.0.4-4.fc24.s390x
perl-srpm-macros-1-20.fc25.noarch
gawk-4.1.3-8.fc25.s390x
libwayland-client-1.12.0-1.fc25.s390x
perl-Exporter-5.72-366.fc25.noarch
perl-version-0.99.17-1.fc25.s390x
fftw-libs-double-3.3.5-3.fc25.s390x
libssh2-1.8.0-1.fc25.s390x
ModemManager-glib-1.6.4-1.fc25.s390x
newt-python3-0.52.19-2.fc25.s390x
python-munch-2.0.4-3.fc25.noarch
python-bugzilla-1.2.2-4.fc25.noarch
libedit-3.1-16.20160618cvs.fc25.s390x
createrepo_c-0.10.0-6.fc25.s390x
device-mapper-multipath-libs-0.4.9-83.fc25.s390x
yum-3.4.3-510.fc25.noarch
dracut-config-rescue-044-78.fc25.s390x
mozjs17-17.0.0-16.fc25.s390x
libselinux-2.5-13.fc25.s390x
libgo-devel-6.3.1-1.fc25.s390x
python2-pyparsing-2.1.10-1.fc25.noarch
cairo-gobject-1.14.8-1.fc25.s390x
ethtool-4.8-1.fc25.s390x
xorg-x11-proto-devel-7.7-20.fc25.noarch
brlapi-0.6.5-2.fc25.s390x
librados-devel-10.2.4-2.fc25.s390x
libXinerama-devel-1.1.3-6.fc24.s390x
lua-posix-33.3.1-3.fc25.s390x
usbredir-devel-0.7.1-2.fc24.s390x
libepoxy-1.4.1-1.fc25.s390x
python3-dnf-plugins-core-0.1.21-5.fc25.noarch
texlive-pdftex-doc-svn41149-33.fc25.1.noarch
openssh-clients-7.4p1-4.fc25.s390x
iptables-1.6.0-3.fc25.s390x
texlive-texlive.infra-svn41280-33.fc25.1.noarch
texlive-graphics-cfg-svn40269-33.fc25.1.noarch
texlive-bibtex-svn40768-33.fc25.1.noarch
texlive-mfware-bin-svn40473-33.20160520.fc25.1.s390x
texlive-texlive-scripts-bin-svn29741.0-33.20160520.fc25.1.noarch
texlive-sauerj-svn15878.0-33.fc25.1.noarch
texlive-enctex-svn34957.0-33.fc25.1.noarch
texlive-ifetex-svn24853.1.2-33.fc25.1.noarch
texlive-ntgclass-svn15878.2.1a-33.fc25.1.noarch
texlive-tex-gyre-math-svn41264-33.fc25.1.noarch
texlive-bera-svn20031.0-33.fc25.1.noarch
texlive-ms-svn29849.0-33.fc25.1.noarch
texlive-pst-fill-svn15878.1.01-33.fc25.1.noarch
texlive-ctable-svn38672-33.fc25.1.noarch
texlive-extsizes-svn17263.1.4a-33.fc25.1.noarch
texlive-collection-latexrecommended-svn35765.0-33.20160520.fc25.1.noarch
perl-Filter-1.57-1.fc25.s390x
krb5-workstation-1.14.4-7.fc25.s390x
python2-rpm-macros-3-12.fc25.noarch
libglvnd-egl-0.2.999-14.20170308git8e6e102.fc25.s390x
libglvnd-opengl-0.2.999-14.20170308git8e6e102.fc25.s390x
gdbm-1.13-1.fc25.s390x
nss-util-3.30.2-1.0.fc25.s390x
libcrypt-nss-2.24-9.fc25.s390x
libtasn1-4.12-1.fc25.s390x
fedora-release-25-2.noarch
gdb-headless-7.12.1-48.fc25.s390x
perl-macros-5.24.2-387.fc25.s390x
sqlite-devel-3.14.2-2.fc25.s390x
pcre-devel-8.41-1.fc25.s390x
libX11-1.6.5-1.fc25.s390x
coreutils-8.25-17.fc25.s390x
python2-openidc-client-0-3.20170523git77cb3ee.fc25.noarch
nss-sysinit-3.30.2-1.1.fc25.s390x
systemtap-client-3.1-5.fc25.s390x
nss-devel-3.30.2-1.1.fc25.s390x
firewalld-0.4.4.5-1.fc25.noarch
rpmlint-1.9-9.fc25.noarch
system-python-3.5.3-6.fc25.s390x
gpg-pubkey-efe550f5-5220ba41
gpg-pubkey-81b46521-55b3ca9a
filesystem-3.2-37.fc24.s390x
libffi-3.1-9.fc24.s390x
keyutils-libs-1.5.9-8.fc24.s390x
libnfnetlink-1.0.1-8.fc24.s390x
libtheora-1.1.1-14.fc24.s390x
xml-common-0.6.3-44.fc24.noarch
autoconf-2.69-22.fc24.noarch
libXt-1.1.5-3.fc24.s390x
kbd-legacy-2.0.3-3.fc24.noarch
ghostscript-fonts-5.50-35.fc24.noarch
libXevie-1.0.3-11.fc24.s390x
libcap-2.25-2.fc25.s390x
mpfr-3.1.5-1.fc25.s390x
perl-Carp-1.40-365.fc25.noarch
libmnl-1.0.4-1.fc25.s390x
perl-Unicode-EastAsianWidth-1.33-8.fc25.noarch
libwayland-cursor-1.12.0-1.fc25.s390x
python-krbV-1.0.90-12.fc25.s390x
python2-urllib3-1.15.1-3.fc25.noarch
fipscheck-1.4.1-11.fc25.s390x
libndp-1.6-1.fc25.s390x
gnupg2-2.1.13-2.fc25.s390x
libXfixes-5.0.3-1.fc25.s390x
adwaita-icon-theme-3.22.0-1.fc25.noarch
dconf-0.26.0-1.fc25.s390x
ncurses-devel-6.0-6.20160709.fc25.s390x
dejagnu-1.6-1.fc25.noarch
libstdc++-devel-6.3.1-1.fc25.s390x
device-mapper-1.02.136-3.fc25.s390x
subversion-1.9.5-1.fc25.s390x
libtool-ltdl-2.4.6-13.fc25.s390x
libevent-2.0.22-1.fc25.s390x
atk-devel-2.22.0-1.fc25.s390x
libev-4.24-1.fc25.s390x
xorg-x11-fonts-Type1-7.5-16.fc24.noarch
brlapi-devel-0.6.5-2.fc25.s390x
pulseaudio-libs-10.0-2.fc25.s390x
glib2-2.50.3-1.fc25.s390x
python2-rpm-4.13.0.1-1.fc25.s390x
dnf-1.1.10-6.fc25.noarch
texlive-metafont-bin-svn40987-33.20160520.fc25.1.s390x
texlive-xkeyval-svn35741.2.7a-33.fc25.1.noarch
texlive-euler-svn17261.2.5-33.fc25.1.noarch
texlive-mptopdf-svn41282-33.fc25.1.noarch
texlive-wasy-svn35831.0-33.fc25.1.noarch
texlive-avantgar-svn31835.0-33.fc25.1.noarch
texlive-eurosym-svn17265.1.4_subrfix-33.fc25.1.noarch
texlive-knuth-lib-svn35820.0-33.fc25.1.noarch
texlive-parallel-svn15878.0-33.fc25.1.noarch
texlive-texlive-msg-translations-svn41431-33.fc25.1.noarch
texlive-latex-svn40218-33.fc25.1.noarch
texlive-lualatex-math-svn40621-33.fc25.1.noarch
texlive-auto-pst-pdf-svn23723.0.6-33.fc25.1.noarch
texlive-powerdot-svn38984-33.fc25.1.noarch
texlive-wasysym-svn15878.2.0-33.fc25.1.noarch
ImageMagick-libs-6.9.3.0-6.fc25.s390x
geoclue2-2.4.5-1.fc25.s390x
perl-IO-Socket-IP-0.39-1.fc25.noarch
python2-pyasn1-0.2.3-1.fc25.noarch
libglvnd-0.2.999-14.20170308git8e6e102.fc25.s390x
libglvnd-gles-0.2.999-14.20170308git8e6e102.fc25.s390x
gdk-pixbuf2-devel-2.36.6-1.fc25.s390x
at-spi2-core-devel-2.22.1-1.fc25.s390x
libacl-2.2.52-13.fc25.s390x
perl-libs-5.24.2-387.fc25.s390x
mesa-libglapi-17.0.5-3.fc25.s390x
kernel-headers-4.11.10-200.fc25.s390x
p11-kit-devel-0.23.2-4.fc25.s390x
python3-rpmconf-1.0.19-1.fc25.noarch
sqlite-3.14.2-2.fc25.s390x
pcre-utf32-8.41-1.fc25.s390x
libX11-common-1.6.5-1.fc25.noarch
coreutils-common-8.25-17.fc25.s390x
mesa-libEGL-17.0.5-3.fc25.s390x
nss-3.30.2-1.1.fc25.s390x
systemtap-runtime-3.1-5.fc25.s390x
NetworkManager-glib-1.4.4-5.fc25.s390x
audit-2.7.7-1.fc25.s390x
perl-Time-HiRes-1.9742-1.fc25.s390x
libsolv-0.6.28-1.fc25.s390x
gpg-pubkey-34ec9cba-54e38751
gpg-pubkey-030d5aed-55b577f0
basesystem-11-2.fc24.noarch
libmpc-1.0.2-5.fc24.s390x
libunistring-0.9.4-3.fc24.s390x
libmodman-2.0.1-12.fc24.s390x
lsscsi-0.28-3.fc24.s390x
kbd-misc-2.0.3-3.fc24.noarch
kmod-23-1.fc25.s390x
newt-0.52.19-2.fc25.s390x
perl-Text-Unidecode-1.27-3.fc25.noarch
plymouth-core-libs-0.9.3-0.6.20160620git0e65b86c.fc25.s390x
which-2.21-1.fc25.s390x
python3-slip-0.6.4-4.fc25.noarch
python3-systemd-232-1.fc25.s390x
python-lockfile-0.11.0-4.fc25.noarch
python2-requests-2.10.0-4.fc25.noarch
libnghttp2-1.13.0-2.fc25.s390x
python-urlgrabber-3.10.1-9.fc25.noarch
iputils-20161105-1.fc25.s390x
rest-0.8.0-1.fc25.s390x
adwaita-cursor-theme-3.22.0-1.fc25.noarch
authconfig-6.2.10-14.fc25.s390x
automake-1.15-7.fc25.noarch
shared-mime-info-1.8-1.fc25.s390x
pigz-2.3.4-1.fc25.s390x
device-mapper-libs-1.02.136-3.fc25.s390x
dnsmasq-2.76-2.fc25.s390x
fedora-packager-0.6.0.1-1.fc25.noarch
gcc-c++-6.3.1-1.fc25.s390x
libwebp-0.5.2-1.fc25.s390x
boost-system-1.60.0-10.fc25.s390x
libasyncns-0.8-10.fc24.s390x
libXau-devel-1.0.8-6.fc24.s390x
libverto-libev-0.2.6-6.fc24.s390x
python3-html5lib-0.999-9.fc25.noarch
ttmkfdir-3.0.9-48.fc24.s390x
pulseaudio-libs-glib2-10.0-2.fc25.s390x
wpa_supplicant-2.6-1.fc25.s390x
texlive-lib-2016-33.20160520.fc25.s390x
libXi-devel-1.7.9-1.fc25.s390x
python3-distro-1.0.3-1.fc25.noarch
rpm-plugin-systemd-inhibit-4.13.0.1-1.fc25.s390x
texlive-texlive-common-doc-svn40682-33.fc25.1.noarch
packagedb-cli-2.14.1-1.fc25.noarch
texlive-metafont-svn40793-33.fc25.1.noarch
texlive-tools-svn40934-33.fc25.1.noarch
texlive-enumitem-svn24146.3.5.2-33.fc25.1.noarch
texlive-mptopdf-bin-svn18674.0-33.20160520.fc25.1.noarch
texlive-underscore-svn18261.0-33.fc25.1.noarch
texlive-anysize-svn15878.0-33.fc25.1.noarch
texlive-euenc-svn19795.0.1h-33.fc25.1.noarch
texlive-kastrup-svn15878.0-33.fc25.1.noarch
texlive-paralist-svn39247-33.fc25.1.noarch
texlive-texlive-en-svn41185-33.fc25.1.noarch
texlive-tipa-svn29349.1.3-33.fc25.1.noarch
texlive-currfile-svn40725-33.fc25.1.noarch
texlive-pst-node-svn40743-33.fc25.1.noarch
texlive-pst-slpe-svn24391.1.31-33.fc25.1.noarch
texlive-typehtml-svn17134.0-33.fc25.1.noarch
SDL2-devel-2.0.5-3.fc25.s390x
libcroco-0.6.11-3.fc25.s390x
bluez-libs-devel-5.44-1.fc25.s390x
kernel-4.10.8-200.fc25.s390x
expat-2.2.1-1.fc25.s390x
system-python-libs-3.5.3-6.fc25.s390x
pcre2-10.23-8.fc25.s390x
firewalld-filesystem-0.4.4.5-1.fc25.noarch
pcre-cpp-8.41-1.fc25.s390x
python3-firewall-0.4.4.5-1.fc25.noarch
freetype-devel-2.6.5-9.fc25.s390x
pcre-utf16-8.41-1.fc25.s390x
linux-firmware-20170605-74.git37857004.fc25.noarch
distribution-gpg-keys-1.12-1.fc25.noarch
kernel-modules-4.11.10-200.fc25.s390x
gnutls-devel-3.5.14-1.fc25.s390x
systemtap-devel-3.1-5.fc25.s390x
java-1.8.0-openjdk-1.8.0.131-1.b12.fc25.s390x
polkit-0.113-8.fc25.s390x
perl-SelfLoader-1.23-387.fc25.noarch
libdb-utils-5.3.28-24.fc25.s390x
fontpackages-filesystem-1.44-17.fc24.noarch
groff-base-1.22.3-8.fc24.s390x
ilmbase-2.2.0-5.fc24.s390x
OpenEXR-libs-2.2.0-5.fc24.s390x
hesiod-3.2.1-6.fc24.s390x
sysfsutils-2.1.0-19.fc24.s390x
ocaml-srpm-macros-2-4.fc24.noarch
mailx-12.5-19.fc24.s390x
ncurses-libs-6.0-6.20160709.fc25.s390x
ipset-libs-6.29-1.fc25.s390x
gmp-devel-6.1.1-1.fc25.s390x
python-pip-8.1.2-2.fc25.noarch
harfbuzz-1.3.2-1.fc25.s390x
python2-iniparse-0.4-20.fc25.noarch
python3-iniparse-0.4-20.fc25.noarch
python3-kickstart-2.32-1.fc25.noarch
perl-Net-SSLeay-1.78-1.fc25.s390x
drpm-0.3.0-3.fc25.s390x
glib-networking-2.50.0-1.fc25.s390x
webkitgtk3-2.4.11-3.fc25.s390x
libXaw-1.0.13-4.fc25.s390x
xorg-x11-font-utils-7.5-32.fc25.s390x
hardlink-1.1-1.fc25.s390x
libcom_err-1.43.3-1.fc25.s390x
python2-dateutil-2.6.0-1.fc25.noarch
libXpm-3.5.12-1.fc25.s390x
python2-smmap-2.0.1-1.fc25.noarch
poppler-data-0.4.7-6.fc25.noarch
librbd1-10.2.4-2.fc25.s390x
perl-Digest-MD5-2.55-2.fc25.s390x
wayland-protocols-devel-1.7-1.fc25.noarch
texi2html-5.0-4.fc24.noarch
libxkbcommon-0.7.1-1.fc25.s390x
libuuid-devel-2.28.2-2.fc25.s390x
libcacard-2.5.3-1.fc25.s390x
libwmf-lite-0.2.8.4-50.fc25.s390x
texlive-tetex-svn41059-33.fc25.1.noarch
texlive-thumbpdf-svn34621.3.16-33.fc25.1.noarch
texlive-carlisle-svn18258.0-33.fc25.1.noarch
texlive-makeindex-bin-svn40473-33.20160520.fc25.1.s390x
texlive-pdftex-svn41149-33.fc25.1.noarch
texlive-csquotes-svn39538-33.fc25.1.noarch
texlive-courier-svn35058.0-33.fc25.1.noarch
texlive-helvetic-svn31835.0-33.fc25.1.noarch
texlive-mfnfss-svn19410.0-33.fc25.1.noarch
texlive-sepnum-svn20186.2.0-33.fc25.1.noarch
texlive-utopia-svn15878.0-33.fc25.1.noarch
texlive-luatexbase-svn38550-33.fc25.1.noarch
texlive-pst-3d-svn17257.1.10-33.fc25.1.noarch
texlive-latex-bin-bin-svn14050.0-33.20160520.fc25.1.noarch
texlive-l3experimental-svn41163-33.fc25.1.noarch
net-tools-2.0-0.40.20160329git.fc25.s390x
perl-Pod-Perldoc-3.28-1.fc25.noarch
openssl-1.0.2k-1.fc25.s390x
man-pages-4.06-4.fc25.noarch
python3-magic-5.29-4.fc25.noarch
libxml2-2.9.4-2.fc25.s390x
nss-softokn-3.30.2-1.0.fc25.s390x
p11-kit-trust-0.23.2-4.fc25.s390x
emacs-filesystem-25.2-2.fc25.noarch
python3-dateutil-2.6.0-1.fc25.noarch
perl-threads-shared-1.57-1.fc25.s390x
libnotify-0.7.7-1.fc25.s390x
unzip-6.0-32.fc25.s390x
pango-devel-1.40.5-1.fc25.s390x
libdrm-2.4.81-1.fc25.s390x
python-beautifulsoup4-4.6.0-1.fc25.noarch
libcurl-7.51.0-7.fc25.s390x
dhcp-client-4.3.5-3.fc25.s390x
python2-fedora-0.9.0-6.fc25.noarch
emacs-25.2-2.fc25.s390x
gdb-7.12.1-48.fc25.s390x
expat-devel-2.2.1-1.fc25.s390x
gpg-pubkey-95a43f54-5284415a
dejavu-fonts-common-2.35-3.fc24.noarch
libSM-1.2.2-4.fc24.s390x
diffutils-3.3-13.fc24.s390x
libogg-1.3.2-5.fc24.s390x
hunspell-en-US-0.20140811.1-5.fc24.noarch
libdaemon-0.14-10.fc24.s390x
patch-2.7.5-3.fc24.s390x
libsysfs-2.1.0-19.fc24.s390x
procmail-3.22-39.fc24.s390x
libXdamage-1.1.4-8.fc24.s390x
libotf-0.9.13-7.fc24.s390x
urw-fonts-2.4-22.fc24.noarch
crontabs-1.11-12.20150630git.fc24.noarch
ppp-2.4.7-9.fc24.s390x
cyrus-sasl-2.1.26-26.2.fc24.s390x
zlib-devel-1.2.8-10.fc24.s390x
time-1.7-49.fc24.s390x
gpg-pubkey-fdb19c98-56fd6333
libcap-ng-0.7.8-1.fc25.s390x
binutils-2.26.1-1.fc25.s390x
lcms2-2.8-2.fc25.s390x
libcomps-0.1.7-5.fc25.s390x
apr-1.5.2-4.fc25.s390x
perl-constant-1.33-367.fc25.noarch
perl-Data-Dumper-2.161-1.fc25.s390x
ipcalc-0.1.8-1.fc25.s390x
gmp-c++-6.1.1-1.fc25.s390x
fontconfig-2.12.1-1.fc25.s390x
enchant-1.6.0-14.fc25.s390x
pyliblzma-0.5.3-16.fc25.s390x
libsepol-devel-2.5-10.fc25.s390x
python3-ordered-set-2.0.0-4.fc25.noarch
python-ipaddress-1.0.16-3.fc25.noarch
python2-kerberos-1.2.5-1.fc25.s390x
python2-pysocks-1.5.6-5.fc25.noarch
fipscheck-lib-1.4.1-11.fc25.s390x
libatomic_ops-7.4.4-1.fc25.s390x
net-snmp-agent-libs-5.7.3-13.fc25.s390x
dracut-044-78.fc25.s390x
python2-pygpgme-0.3-18.fc25.s390x
orc-0.4.26-1.fc25.s390x
yum-utils-1.1.31-511.fc25.noarch
libXrender-0.9.10-1.fc25.s390x
libXrandr-1.5.1-1.fc25.s390x
go-srpm-macros-2-7.fc25.noarch
gnupg2-smime-2.1.13-2.fc25.s390x
guile-devel-2.0.13-1.fc25.s390x
uboot-tools-2016.09.01-2.fc25.s390x
pykickstart-2.32-1.fc25.noarch
python-bunch-1.0.1-9.fc25.noarch
perl-generators-1.10-1.fc25.noarch
perl-Mozilla-CA-20160104-3.fc25.noarch
bzip2-libs-1.0.6-21.fc25.s390x
libpng-1.6.27-1.fc25.s390x
desktop-file-utils-0.23-2.fc25.s390x
python2-cccolutils-1.4-1.fc25.s390x
python2-lxml-3.7.2-1.fc25.s390x
redhat-rpm-config-45-1.fc25.noarch
device-mapper-event-libs-1.02.136-3.fc25.s390x
lvm2-libs-2.02.167-3.fc25.s390x
python2-gitdb-2.0.0-1.fc25.noarch
gcc-gfortran-6.3.1-1.fc25.s390x
libselinux-python-2.5-13.fc25.s390x
openjpeg2-2.1.2-3.fc25.s390x
boost-thread-1.60.0-10.fc25.s390x
librbd-devel-10.2.4-2.fc25.s390x
libXcursor-devel-1.1.14-6.fc24.s390x
latex2html-2012-7.fc24.noarch
lksctp-tools-1.0.16-5.fc24.s390x
libfdt-1.4.2-1.fc25.s390x
libXft-devel-2.3.2-4.fc24.s390x
libattr-devel-2.4.47-16.fc24.s390x
libiscsi-devel-1.15.0-2.fc24.s390x
gettext-0.19.8.1-3.fc25.s390x
libjpeg-turbo-devel-1.5.1-0.fc25.s390x
pulseaudio-libs-devel-10.0-2.fc25.s390x
libepoxy-devel-1.4.1-1.fc25.s390x
krb5-libs-1.14.4-7.fc25.s390x
libmount-2.28.2-2.fc25.s390x
python3-decorator-4.0.11-1.fc25.noarch
rpm-plugin-selinux-4.13.0.1-1.fc25.s390x
tzdata-java-2017b-1.fc25.noarch
python-srpm-macros-3-12.fc25.noarch
libsmartcols-2.28.2-2.fc25.s390x
kernel-core-4.10.5-200.fc25.s390x
kernel-modules-4.10.5-200.fc25.s390x
texlive-kpathsea-svn41139-33.fc25.1.noarch
texlive-amsmath-svn41561-33.fc25.1.noarch
texlive-thumbpdf-bin-svn6898.0-33.20160520.fc25.1.noarch
texlive-psnfss-svn33946.9.2a-33.fc25.1.noarch
texlive-subfig-svn15878.1.3-33.fc25.1.noarch
texlive-fancybox-svn18304.1.4-33.fc25.1.noarch
texlive-lua-alt-getopt-svn29349.0.7.0-33.fc25.1.noarch
texlive-natbib-svn20668.8.31b-33.fc25.1.noarch
texlive-pdftex-bin-svn40987-33.20160520.fc25.1.s390x
texlive-xdvi-svn40768-33.fc25.1.noarch
texlive-crop-svn15878.1.5-33.fc25.1.noarch
texlive-babel-english-svn30264.3.3p-33.fc25.1.noarch
texlive-cmextra-svn32831.0-33.fc25.1.noarch
texlive-fancyhdr-svn15878.3.1-33.fc25.1.noarch
texlive-luatex-svn40963-33.fc25.1.noarch
texlive-knuth-local-svn38627-33.fc25.1.noarch
texlive-mflogo-font-svn36898.1.002-33.fc25.1.noarch
texlive-parskip-svn19963.2.0-33.fc25.1.noarch
texlive-section-svn20180.0-33.fc25.1.noarch
texlive-textcase-svn15878.0-33.fc25.1.noarch
texlive-updmap-map-svn41159-33.fc25.1.noarch
texlive-attachfile-svn38830-33.fc25.1.noarch
libglvnd-glx-0.2.999-14.20170308git8e6e102.fc25.s390x
libglvnd-core-devel-0.2.999-14.20170308git8e6e102.fc25.s390x
python-magic-5.29-4.fc25.noarch
glibc-common-2.24-9.fc25.s390x
sqlite-libs-3.14.2-2.fc25.s390x
libtiff-4.0.8-1.fc25.s390x
libdb-5.3.28-24.fc25.s390x
glusterfs-client-xlators-3.10.4-1.fc25.s390x
nss-util-devel-3.30.2-1.0.fc25.s390x
gnutls-3.5.14-1.fc25.s390x
bind-license-9.10.5-2.P2.fc25.noarch
mesa-libGLES-17.0.5-3.fc25.s390x
python3-requests-kerberos-0.10.0-2.fc25.noarch
python3-pyOpenSSL-16.2.0-1.fc25.noarch
perl-threads-2.16-1.fc25.s390x
cryptsetup-libs-1.7.5-1.fc25.s390x
vim-minimal-8.0.705-1.fc25.s390x
netpbm-10.79.00-1.fc25.s390x
qrencode-libs-3.4.4-1.fc25.s390x
mariadb-config-10.1.24-3.fc25.s390x
gstreamer1-plugins-base-1.10.5-1.fc25.s390x
elfutils-default-yama-scope-0.169-1.fc25.noarch
glusterfs-3.10.4-1.fc25.s390x
systemd-udev-231-17.fc25.s390x
python2-koji-1.13.0-2.fc25.noarch
unbound-libs-1.6.3-1.fc25.s390x
openldap-2.4.44-11.fc25.s390x
koji-1.13.0-2.fc25.noarch
bind99-libs-9.9.10-2.P3.fc25.s390x
libcurl-devel-7.51.0-7.fc25.s390x
mesa-libGL-devel-17.0.5-3.fc25.s390x
python2-sssdconfig-1.15.2-5.fc25.noarch
webkitgtk4-plugin-process-gtk2-2.16.5-1.fc25.s390x
graphite2-devel-1.3.10-1.fc25.s390x
systemtap-sdt-devel-3.1-5.fc25.s390x
iproute-tc-4.11.0-1.fc25.s390x
libarchive-3.2.2-2.fc25.s390x
publicsuffix-list-dafsa-20170424-1.fc25.noarch
texlive-luaotfload-svn40902-33.fc25.1.noarch
texlive-unicode-math-svn38462-33.fc25.1.noarch
texlive-fancyvrb-svn18492.2.8-33.fc25.1.noarch
texlive-pst-pdf-bin-svn7838.0-33.20160520.fc25.1.noarch
texlive-amscls-svn36804.0-33.fc25.1.noarch
texlive-ltxmisc-svn21927.0-33.fc25.1.noarch
texlive-breqn-svn38099.0.98d-33.fc25.1.noarch
texlive-xetex-def-svn40327-33.fc25.1.noarch
openssh-server-7.4p1-4.fc25.s390x
sendmail-8.15.2-8.fc25.s390x
tzdata-2017b-1.fc25.noarch
hunspell-1.4.1-2.fc25.s390x
gpg-pubkey-8e1431d5-53bcbac7
zlib-1.2.8-10.fc24.s390x
sed-4.2.2-15.fc24.s390x
psmisc-22.21-8.fc24.s390x
gpm-libs-1.20.7-9.fc24.s390x
zip-3.0-16.fc24.s390x
libyubikey-1.13-2.fc24.s390x
sg3_utils-libs-1.41-3.fc24.s390x
polkit-pkla-compat-0.1-7.fc24.s390x
passwd-0.79-8.fc24.s390x
trousers-0.3.13-6.fc24.s390x
grubby-8.40-3.fc24.s390x
rootfiles-8.1-19.fc24.noarch
nettle-3.3-1.fc25.s390x
libksba-1.3.5-1.fc25.s390x
perl-Text-ParseWords-3.30-365.fc25.noarch
perl-PathTools-3.63-366.fc25.s390x
perl-File-Temp-0.23.04-365.fc25.noarch
fuse-libs-2.9.7-1.fc25.s390x
perl-Pod-Escapes-1.07-365.fc25.noarch
perl-Term-ANSIColor-4.05-2.fc25.noarch
perl-URI-1.71-5.fc25.noarch
libXfont-1.5.2-1.fc25.s390x
python-six-1.10.0-3.fc25.noarch
dbus-glib-0.108-1.fc25.s390x
gobject-introspection-1.50.0-1.fc25.s390x
libpwquality-1.3.0-6.fc25.s390x
python-gobject-base-3.22.0-1.fc25.s390x
python-html5lib-0.999-9.fc25.noarch
python3-dbus-1.2.4-2.fc25.s390x
python3-chardet-2.3.0-1.fc25.noarch
python3-urllib3-1.15.1-3.fc25.noarch
python-offtrac-0.1.0-7.fc25.noarch
python2-cryptography-1.5.3-3.fc25.s390x
python2-requests-kerberos-0.10.0-2.fc25.noarch
libserf-1.3.9-1.fc25.s390x
libdatrie-0.2.9-3.fc25.s390x
s390utils-base-1.36.0-1.fc25.s390x
kpartx-0.4.9-83.fc25.s390x
s390utils-cpuplugd-1.36.0-1.fc25.s390x
s390utils-osasnmpd-1.36.0-1.fc25.s390x
python-dnf-plugins-extras-common-0.0.12-4.fc25.noarch
fpc-srpm-macros-1.0-1.fc25.noarch
libuser-0.62-4.fc25.s390x
man-db-2.7.5-3.fc25.s390x
python-systemd-doc-232-1.fc25.s390x
lz4-1.7.5-1.fc25.s390x
tar-1.29-3.fc25.s390x
bodhi-client-0.9.12.2-6.fc25.noarch
cairo-1.14.8-1.fc25.s390x
gcc-go-6.3.1-1.fc25.s390x
cracklib-dicts-2.9.6-4.fc25.s390x
libselinux-python3-2.5-13.fc25.s390x
python2-enchant-1.6.8-1.fc25.noarch
boost-iostreams-1.60.0-10.fc25.s390x
userspace-rcu-0.9.2-2.fc25.s390x
libXext-devel-1.3.3-4.fc24.s390x
libXrandr-devel-1.5.1-1.fc25.s390x
perl-XML-XPath-1.39-1.fc25.noarch
python3-lxml-3.7.2-1.fc25.s390x
libiscsi-1.15.0-2.fc24.s390x
fontconfig-devel-2.12.1-1.fc25.s390x
libfdt-devel-1.4.2-1.fc25.s390x
ceph-devel-compat-10.2.4-2.fc25.s390x
zlib-static-1.2.8-10.fc24.s390x
chrpath-0.16-3.fc24.s390x
info-6.1-4.fc25.s390x
iptables-libs-1.6.0-3.fc25.s390x
libfdisk-2.28.2-2.fc25.s390x
dnf-plugins-core-0.1.21-5.fc25.noarch
perl-Storable-2.56-368.fc25.s390x
python2-decorator-4.0.11-1.fc25.noarch
libnetfilter_conntrack-1.0.6-2.fc25.s390x
texlive-texlive.infra-bin-svn40312-33.20160520.fc25.1.s390x
texlive-ifluatex-svn41346-33.fc25.1.noarch
texlive-fp-svn15878.0-33.fc25.1.noarch
texlive-latex-fonts-svn28888.0-33.fc25.1.noarch
texlive-bibtex-bin-svn40473-33.20160520.fc25.1.s390x
texlive-glyphlist-svn28576.0-33.fc25.1.noarch
texlive-marvosym-svn29349.2.2a-33.fc25.1.noarch
texlive-tex-bin-svn40987-33.20160520.fc25.1.s390x
texlive-texconfig-svn40768-33.fc25.1.noarch
texlive-wasy2-ps-svn35830.0-33.fc25.1.noarch
texlive-psfrag-svn15878.3.04-33.fc25.1.noarch
texlive-charter-svn15878.0-33.fc25.1.noarch
texlive-ec-svn25033.1.0-33.fc25.1.noarch
texlive-lineno-svn21442.4.41-33.fc25.1.noarch
texlive-hyphen-base-svn41138-33.fc25.1.noarch
texlive-manfnt-font-svn35799.0-33.fc25.1.noarch
texlive-ncntrsbk-svn31835.0-33.fc25.1.noarch
texlive-pst-math-svn34786.0.63-33.fc25.1.noarch
texlive-symbol-svn31835.0-33.fc25.1.noarch
texlive-environ-svn33821.0.3-33.fc25.1.noarch
texlive-algorithms-svn38085.0.1-33.fc25.1.noarch
kernel-core-4.10.8-200.fc25.s390x
libglvnd-devel-0.2.999-14.20170308git8e6e102.fc25.s390x
python3-hawkey-0.6.4-3.fc25.s390x
glibc-all-langpacks-2.24-9.fc25.s390x
freetype-2.6.5-9.fc25.s390x
mesa-libwayland-egl-17.0.5-3.fc25.s390x
libicu-57.1-5.fc25.s390x
nspr-devel-4.14.0-2.fc25.s390x
libnl3-cli-3.2.29-3.fc25.s390x
cups-libs-2.2.0-9.fc25.s390x
bind-libs-lite-9.10.5-2.P2.fc25.s390x
ca-certificates-2017.2.14-1.0.fc25.noarch
python3-kerberos-1.2.5-1.fc25.s390x
python3-cryptography-1.5.3-3.fc25.s390x
perl-IO-1.36-387.fc25.s390x
dhcp-libs-4.3.5-3.fc25.s390x
rsync-3.1.2-4.fc25.s390x
make-4.1-6.fc25.s390x
pcre2-devel-10.23-8.fc25.s390x
quota-4.03-8.fc25.s390x
libX11-devel-1.6.5-1.fc25.s390x
ghostscript-9.20-9.fc25.s390x
dbus-1.11.14-1.fc25.s390x
rpcbind-0.2.4-6.rc2.fc25.s390x
pyOpenSSL-16.2.0-1.fc25.noarch
glusterfs-devel-3.10.4-1.fc25.s390x
nss-tools-3.30.2-1.1.fc25.s390x
python3-pycurl-7.43.0-6.fc25.s390x
bind99-license-9.9.10-2.P3.fc25.noarch
curl-7.51.0-7.fc25.s390x
glusterfs-api-devel-3.10.4-1.fc25.s390x
python-firewall-0.4.4.5-1.fc25.noarch
netpbm-progs-10.79.00-1.fc25.s390x
libsndfile-1.0.28-3.fc25.s390x
python3-sssdconfig-1.15.2-5.fc25.noarch
wget-1.18-3.fc25.s390x
libsemanage-2.5-9.fc25.s390x
telnet-0.17-68.fc25.s390x
texlive-ifplatform-svn21156.0.4-33.fc25.1.noarch
texlive-eso-pic-svn37925.2.0g-33.fc25.1.noarch
texlive-xcolor-svn41044-33.fc25.1.noarch
texlive-pst-eps-svn15878.1.0-33.fc25.1.noarch
texlive-pst-text-svn15878.1.00-33.fc25.1.noarch
texlive-rotating-svn16832.2.16b-33.fc25.1.noarch
texlive-pdfpages-svn40638-33.fc25.1.noarch
texlive-cm-super-svn15878.0-33.fc25.1.noarch
texlive-xetex-svn41438-33.fc25.1.noarch
dnf-yum-1.1.10-6.fc25.noarch
libseccomp-devel-2.3.2-1.fc25.s390x
gpgme-1.8.0-10.fc25.s390x
apr-util-1.5.4-3.fc24.s390x
jbigkit-libs-2.1-5.fc24.s390x
pixman-0.34.0-2.fc24.s390x
dwz-0.12-2.fc24.s390x
expect-5.45-22.fc24.s390x
libsigsegv-2.10-10.fc24.s390x
fakeroot-libs-1.20.2-4.fc24.s390x
m17n-lib-1.7.0-5.fc24.s390x
libverto-0.2.6-6.fc24.s390x
libXmu-1.1.2-4.fc24.s390x
libXcursor-1.1.14-6.fc24.s390x
python-kitchen-1.2.4-2.fc24.noarch
fakeroot-1.20.2-4.fc24.s390x
blktrace-1.1.0-3.fc24.s390x
usermode-1.111-8.fc24.s390x
kbd-2.0.3-3.fc24.s390x
libaio-devel-0.3.110-6.fc24.s390x
web-assets-filesystem-5-4.fc24.noarch
libgpg-error-1.24-1.fc25.s390x
findutils-4.6.0-8.fc25.s390x
libassuan-2.4.3-1.fc25.s390x
libusbx-1.0.21-1.fc25.s390x
libxslt-1.1.28-13.fc25.s390x
libmetalink-0.1.3-1.fc25.s390x
perl-MIME-Base64-3.15-365.fc25.s390x
ncurses-6.0-6.20160709.fc25.s390x
libwayland-server-1.12.0-1.fc25.s390x
perl-Fedora-VSP-0.001-4.fc25.noarch
perl-libintl-perl-1.26-1.fc25.s390x
shadow-utils-4.2.1-11.fc25.s390x
atk-2.22.0-1.fc25.s390x
pam-1.3.0-1.fc25.s390x
harfbuzz-icu-1.3.2-1.fc25.s390x
libsecret-0.18.5-2.fc25.s390x
s390utils-iucvterm-1.36.0-1.fc25.s390x
python3-requests-2.10.0-4.fc25.noarch
pyusb-1.0.0-2.fc25.noarch
python-enum34-1.0.4-6.fc25.noarch
pyxattr-0.5.3-8.fc25.s390x
libbabeltrace-1.4.0-3.fc25.s390x
libthai-0.1.25-1.fc25.s390x
deltarpm-3.6-17.fc25.s390x
s390utils-mon_statd-1.36.0-1.fc25.s390x
device-mapper-multipath-0.4.9-83.fc25.s390x
python3-pygpgme-0.3-18.fc25.s390x
libreport-filesystem-2.8.0-1.fc25.s390x
ghc-srpm-macros-1.4.2-4.fc25.noarch
rpmdevtools-8.9-1.fc25.noarch
python-dnf-plugins-extras-migrate-0.0.12-4.fc25.noarch
perl-IO-Socket-SSL-2.038-1.fc25.noarch
perl-File-ShareDir-1.102-7.fc25.noarch
tcl-8.6.6-1.fc25.s390x
bzip2-1.0.6-21.fc25.s390x
libss-1.43.3-1.fc25.s390x
libselinux-utils-2.5-13.fc25.s390x
python3-enchant-1.6.8-1.fc25.noarch
python2-dockerfile-parse-0.0.5-7.fc25.noarch
systemd-bootchart-231-2.fc25.s390x
gcc-objc-6.3.1-1.fc25.s390x
e2fsprogs-1.43.3-1.fc25.s390x
libstdc++-static-6.3.1-1.fc25.s390x
libpng-devel-1.6.27-1.fc25.s390x
perl-XML-Parser-2.44-5.fc25.s390x
lttng-ust-2.8.1-2.fc25.s390x
libXfixes-devel-5.0.3-1.fc25.s390x
libXcomposite-devel-0.4.4-8.fc24.s390x
python3-javapackages-4.7.0-6.1.fc25.noarch
libcephfs_jni-devel-10.2.4-2.fc25.s390x
keyutils-libs-devel-1.5.9-8.fc24.s390x
harfbuzz-devel-1.3.2-1.fc25.s390x
libidn-devel-1.33-1.fc25.s390x
libnfs-1.9.8-2.fc24.s390x
libssh2-devel-1.8.0-1.fc25.s390x
qemu-sanity-check-nodeps-1.1.5-5.fc24.s390x
alsa-lib-devel-1.1.1-2.fc25.s390x
libpsl-0.17.0-1.fc25.s390x
libseccomp-2.3.2-1.fc25.s390x
copy-jdk-configs-2.2-2.fc25.noarch
json-glib-1.2.6-1.fc25.s390x
python2-dnf-1.1.10-6.fc25.noarch
python2-GitPython-2.1.3-1.fc25.noarch
texlive-tetex-bin-svn36770.0-33.20160520.fc25.1.noarch
texlive-amsfonts-svn29208.3.04-33.fc25.1.noarch
texlive-babel-svn40706-33.fc25.1.noarch
texlive-colortbl-svn29803.v1.0a-33.fc25.1.noarch
texlive-babelbib-svn25245.1.31-33.fc25.1.noarch
texlive-footmisc-svn23330.5.5b-33.fc25.1.noarch
texlive-makeindex-svn40768-33.fc25.1.noarch
texlive-plain-svn40274-33.fc25.1.noarch
texlive-texconfig-bin-svn29741.0-33.20160520.fc25.1.noarch
texlive-zapfding-svn31835.0-33.fc25.1.noarch
texlive-microtype-svn41127-33.fc25.1.noarch
texlive-bookman-svn31835.0-33.fc25.1.noarch
texlive-dvisvgm-def-svn41011-33.fc25.1.noarch
texlive-finstrut-svn21719.0.5-33.fc25.1.noarch
texlive-hyph-utf8-svn41189-33.fc25.1.noarch
texlive-lualibs-svn40370-33.fc25.1.noarch
kernel-modules-4.10.8-200.fc25.s390x
python2-hawkey-0.6.4-3.fc25.s390x
glibc-2.24-9.fc25.s390x
elfutils-libelf-0.169-1.fc25.s390x
libsoup-2.56.0-3.fc25.s390x
libnl3-3.2.29-3.fc25.s390x
gstreamer1-1.10.5-1.fc25.s390x
polkit-libs-0.113-8.fc25.s390x
libtirpc-1.0.2-0.fc25.s390x
emacs-common-25.2-2.fc25.s390x
libteam-1.27-1.fc25.s390x
python3-3.5.3-6.fc25.s390x
python3-pyasn1-0.2.3-1.fc25.noarch
perl-File-Path-2.12-366.fc25.noarch
mesa-libwayland-egl-devel-17.0.5-3.fc25.s390x
libacl-devel-2.2.52-13.fc25.s390x
lua-libs-5.3.4-3.fc25.s390x
pcre2-utf32-10.23-8.fc25.s390x
quota-nls-4.03-8.fc25.noarch
gtk3-3.22.16-1.fc25.s390x
ghostscript-x11-9.20-9.fc25.s390x
systemd-231-17.fc25.s390x
glusterfs-api-3.10.4-1.fc25.s390x
glusterfs-extra-xlators-3.10.4-1.fc25.s390x
glusterfs-server-3.10.4-1.fc25.s390x
java-1.8.0-openjdk-headless-1.8.0.131-1.b12.fc25.s390x
git-2.9.4-1.fc25.s390x
dhcp-common-4.3.5-3.fc25.noarch
python2-rpkg-1.49-5.fc25.noarch
vte291-devel-0.46.2-1.fc25.s390x
python-devel-2.7.13-2.fc25.s390x
elfutils-0.169-1.fc25.s390x
lua-5.3.4-3.fc25.s390x
python3-beautifulsoup4-4.6.0-1.fc25.noarch
libmicrohttpd-0.9.55-1.fc25.s390x
screen-4.6.1-1.fc25.s390x
strace-4.18-1.fc25.s390x
texlive-mparhack-svn15878.1.4-33.fc25.1.noarch
texlive-pspicture-svn15878.0-33.fc25.1.noarch
texlive-soul-svn15878.2.4-33.fc25.1.noarch
texlive-trimspaces-svn15878.1.1-33.fc25.1.noarch
texlive-varwidth-svn24104.0.92-33.fc25.1.noarch
texlive-geometry-svn19716.5.6-33.fc25.1.noarch
texlive-memoir-svn41203-33.fc25.1.noarch
texlive-pgf-svn40966-33.fc25.1.noarch
texlive-pst-coil-svn37377.1.07-33.fc25.1.noarch
texlive-pst-plot-svn41242-33.fc25.1.noarch
texlive-latex-bin-svn41438-33.fc25.1.noarch
texlive-ucs-svn35853.2.2-33.fc25.1.noarch
texlive-ae-svn15878.1.4-33.fc25.1.noarch
texlive-xetex-bin-svn41091-33.20160520.fc25.1.s390x
fedora-upgrade-26.1-1.fc25.noarch
fedpkg-1.28-1.fc25.noarch
perl-Thread-Queue-3.12-1.fc25.noarch
cdparanoia-libs-10.2-21.fc24.s390x
ustr-1.0.4-21.fc24.s390x
libusb-0.1.5-7.fc24.s390x
readline-devel-6.3-8.fc24.s390x
chkconfig-1.8-1.fc25.s390x
avahi-libs-0.6.32-4.fc25.s390x
perl-Unicode-Normalize-1.25-365.fc25.s390x
perl-libnet-3.10-1.fc25.noarch
perl-podlators-4.09-1.fc25.noarch
dbus-python-1.2.4-2.fc25.s390x
libgnome-keyring-3.12.0-7.fc25.s390x
python-backports-1.0-8.fc25.s390x
python-pycparser-2.14-7.fc25.noarch
plymouth-scripts-0.9.3-0.6.20160620git0e65b86c.fc25.s390x
cronie-1.5.1-2.fc25.s390x
python2-librepo-1.7.18-3.fc25.s390x
libXv-1.0.11-1.fc25.s390x
python2-ndg_httpsclient-0.4.0-4.fc25.noarch
btrfs-progs-4.6.1-1.fc25.s390x
libgcc-6.3.1-1.fc25.s390x
libgomp-6.3.1-1.fc25.s390x
perl-Encode-2.88-5.fc25.s390x
cracklib-2.9.6-4.fc25.s390x
libobjc-6.3.1-1.fc25.s390x
gcc-6.3.1-1.fc25.s390x
python3-dnf-plugin-system-upgrade-0.7.1-4.fc25.noarch
boost-random-1.60.0-10.fc25.s390x
libref_array-0.1.5-29.fc25.s390x
libXrender-devel-0.9.10-1.fc25.s390x
javapackages-tools-4.7.0-6.1.fc25.noarch
keyutils-1.5.9-8.fc24.s390x
libcom_err-devel-1.43.3-1.fc25.s390x
lzo-minilzo-2.08-8.fc24.s390x
libusbx-devel-1.0.21-1.fc25.s390x
virglrenderer-devel-0.5.0-1.20160411git61846f92f.fc25.s390x
acpica-tools-20160831-1.fc25.s390x
grep-2.27-2.fc25.s390x
dnf-conf-1.1.10-6.fc25.noarch
crypto-policies-20160921-4.gitf3018dd.fc25.noarch
rpm-build-libs-4.13.0.1-1.fc25.s390x
libnfsidmap-0.27-1.fc25.s390x
SDL2-2.0.5-3.fc25.s390x
texlive-etex-pkg-svn39355-33.fc25.1.noarch
texlive-multido-svn18302.1.42-33.fc25.1.noarch
texlive-gsftopk-svn40768-33.fc25.1.noarch
texlive-pst-ovl-svn40873-33.fc25.1.noarch
texlive-ltabptch-svn17533.1.74d-33.fc25.1.noarch
texlive-cite-svn36428.5.5-33.fc25.1.noarch
texlive-fpl-svn15878.1.002-33.fc25.1.noarch
texlive-mathpazo-svn15878.1.003-33.fc25.1.noarch
texlive-rcs-svn15878.0-33.fc25.1.noarch
texlive-type1cm-svn21820.0-33.fc25.1.noarch
texlive-l3kernel-svn41246-33.fc25.1.noarch
texlive-hyperref-svn41396-33.fc25.1.noarch
texlive-pst-tree-svn24142.1.12-33.fc25.1.noarch
texlive-sansmathaccent-svn30187.0-33.fc25.1.noarch
texlive-dvipdfmx-bin-svn40273-33.20160520.fc25.1.s390x
texlive-zapfchan-svn31835.0-33.fc25.1.noarch
glib2-static-2.50.3-1.fc25.s390x
bash-completion-2.5-1.fc25.noarch
glusterfs-libs-3.10.4-1.fc25.s390x
p11-kit-0.23.2-4.fc25.s390x
hyphen-2.8.8-4.fc24.s390x
gnutls-c++-3.5.14-1.fc25.s390x
python3-idna-2.5-1.fc25.noarch
nss-softokn-devel-3.30.2-1.0.fc25.s390x
less-481-7.fc25.s390x
rpmconf-base-1.0.19-1.fc25.noarch
gtk2-2.24.31-2.fc25.s390x
mesa-libgbm-17.0.5-3.fc25.s390x
nfs-utils-2.1.1-5.rc4.fc25.s390x
perl-Git-2.9.4-1.fc25.noarch
mock-1.4.2-1.fc25.noarch
mc-4.8.19-5.fc25.s390x
pcre-static-8.41-1.fc25.s390x
bind-libs-9.10.5-2.P2.fc25.s390x
libproxy-0.4.15-2.fc25.s390x
gpg-pubkey-a29cb19c-53bcbba6
m4-1.4.17-9.fc24.s390x
liblockfile-1.09-4.fc24.s390x
sg3_utils-1.41-3.fc24.s390x
libXinerama-1.1.3-6.fc24.s390x
libXft-2.3.2-4.fc24.s390x
tcp_wrappers-libs-7.6-83.fc25.s390x
perl-Text-Tabs+Wrap-2013.0523-365.fc25.noarch
perl-Error-0.17024-7.fc25.noarch
perl-Term-Cap-1.17-365.fc25.noarch
perl-Pod-Usage-1.69-1.fc25.noarch
device-mapper-persistent-data-0.6.3-1.fc25.s390x
net-snmp-libs-5.7.3-13.fc25.s390x
python3-six-1.10.0-3.fc25.noarch
python3-pysocks-1.5.6-5.fc25.noarch
python-chardet-2.3.0-1.fc25.noarch
python2-cffi-1.7.0-2.fc25.s390x
gc-devel-7.4.4-1.fc25.s390x
plymouth-0.9.3-0.6.20160620git0e65b86c.fc25.s390x
ebtables-2.0.10-21.fc25.s390x
python3-librepo-1.7.18-3.fc25.s390x
net-snmp-5.7.3-13.fc25.s390x
at-spi2-atk-2.22.0-1.fc25.s390x
avahi-autoipd-0.6.32-4.fc25.s390x
libgo-6.3.1-1.fc25.s390x
cpp-6.3.1-1.fc25.s390x
pyparsing-2.1.10-1.fc25.noarch
python3-pyparsing-2.1.10-1.fc25.noarch
libcollection-0.7.0-29.fc25.s390x
libcephfs-devel-10.2.4-2.fc25.s390x
libXdamage-devel-1.1.4-8.fc24.s390x
libverto-devel-0.2.6-6.fc24.s390x
snappy-1.1.3-2.fc24.s390x
cairo-gobject-devel-1.14.8-1.fc25.s390x
cyrus-sasl-devel-2.1.26-26.2.fc24.s390x
libXi-1.7.9-1.fc25.s390x
texlive-base-2016-33.20160520.fc25.noarch
python3-rpm-4.13.0.1-1.fc25.s390x
texlive-booktabs-svn40846-33.fc25.1.noarch
texlive-lm-svn28119.2.004-33.fc25.1.noarch
texlive-gsftopk-bin-svn40473-33.20160520.fc25.1.s390x
texlive-tex-svn40793-33.fc25.1.noarch
texlive-fancyref-svn15878.0.9c-33.fc25.1.noarch
texlive-chngcntr-svn17157.1.0a-33.fc25.1.noarch
texlive-fix2col-svn38770-33.fc25.1.noarch
texlive-marginnote-svn41382-33.fc25.1.noarch
texlive-pxfonts-svn15878.0-33.fc25.1.noarch
texlive-txfonts-svn15878.0-33.fc25.1.noarch
texlive-l3packages-svn41246-33.fc25.1.noarch
texlive-oberdiek-svn41346-33.fc25.1.noarch
texlive-pst-tools-svn34067.0.05-33.fc25.1.noarch
texlive-tex-gyre-svn18651.2.004-33.fc25.1.noarch
texlive-dvipdfmx-svn41149-33.fc25.1.noarch
texlive-collection-fontsrecommended-svn35830.0-33.20160520.fc25.1.noarch
libcacard-devel-2.5.3-1.fc25.s390x
ykpers-1.18.0-2.fc25.s390x
python2-idna-2.5-1.fc25.noarch
file-libs-5.29-4.fc25.s390x
policycoreutils-2.5-20.fc25.s390x
libgcrypt-1.7.8-1.fc25.s390x
pcre-8.41-1.fc25.s390x
GeoIP-1.6.11-1.fc25.s390x
ghostscript-core-9.20-9.fc25.s390x
python3-cffi-1.7.0-2.fc25.s390x
nss-softokn-freebl-devel-3.30.2-1.0.fc25.s390x
json-c-0.12.1-2.fc25.s390x
vim-common-8.0.705-1.fc25.s390x
vte291-0.46.2-1.fc25.s390x
libdrm-devel-2.4.81-1.fc25.s390x
gssproxy-0.7.0-9.fc25.s390x
git-core-doc-2.9.4-1.fc25.s390x
systemtap-3.1-5.fc25.s390x
mesa-libgbm-devel-17.0.5-3.fc25.s390x
vim-enhanced-8.0.705-1.fc25.s390x
glibc-static-2.24-9.fc25.s390x
libgusb-0.2.10-1.fc25.s390x
python-async-0.6.1-9.fc22.s390x
dejavu-sans-mono-fonts-2.35-3.fc24.noarch
popt-1.16-7.fc24.s390x
cyrus-sasl-lib-2.1.26-26.2.fc24.s390x
xz-5.2.2-2.fc24.s390x
libpipeline-1.4.1-2.fc24.s390x
pinentry-0.9.7-2.fc24.s390x
pth-2.0.7-27.fc24.s390x
libsepol-2.5-10.fc25.s390x
libxcb-1.12-1.fc25.s390x
perl-Getopt-Long-2.49.1-1.fc25.noarch
avahi-glib-0.6.32-4.fc25.s390x
python3-pip-8.1.2-2.fc25.noarch
python3-libcomps-0.1.7-5.fc25.s390x
python-slip-0.6.4-4.fc25.noarch
python2-libcomps-0.1.7-5.fc25.s390x
gc-7.4.4-1.fc25.s390x
s390utils-cmsfs-1.36.0-1.fc25.s390x
newt-python-0.52.19-2.fc25.s390x
qt5-srpm-macros-5.7.1-1.fc25.noarch
device-mapper-event-1.02.136-3.fc25.s390x
perl-Class-Inspector-1.31-2.fc25.noarch
libbasicobjects-0.1.1-29.fc25.s390x
libradosstriper1-10.2.4-2.fc25.s390x
libXxf86vm-devel-1.1.4-3.fc24.s390x
zziplib-0.13.62-7.fc24.s390x
libpaper-1.1.24-12.fc24.s390x
libini_config-1.3.0-29.fc25.s390x
snappy-devel-1.1.3-2.fc24.s390x
libcap-ng-devel-0.7.8-1.fc25.s390x
libxkbcommon-devel-0.7.1-1.fc25.s390x
openssl-libs-1.0.2k-1.fc25.s390x
libkadm5-1.14.4-7.fc25.s390x
rpm-libs-4.13.0.1-1.fc25.s390x
util-linux-2.28.2-2.fc25.s390x
texlive-etoolbox-svn38031.2.2a-33.fc25.1.noarch
texlive-dvips-svn41149-33.fc25.1.noarch
texlive-latexconfig-svn40274-33.fc25.1.noarch
texlive-tex-ini-files-svn40533-33.fc25.1.noarch
texlive-qstest-svn15878.0-33.fc25.1.noarch
texlive-cmap-svn41168-33.fc25.1.noarch
texlive-luatex-bin-svn41091-33.20160520.fc25.1.s390x
texlive-mflogo-svn38628-33.fc25.1.noarch
texlive-sansmath-svn17997.1.1-33.fc25.1.noarch
texlive-unicode-data-svn39808-33.fc25.1.noarch
texlive-luaotfload-bin-svn34647.0-33.20160520.fc25.1.noarch
texlive-listings-svn37534.1.6-33.fc25.1.noarch
texlive-pstricks-svn41321-33.fc25.1.noarch
texlive-metalogo-svn18611.0.12-33.fc25.1.noarch
texlive-collection-latex-svn41011-33.20160520.fc25.1.noarch
kernel-4.10.5-200.fc25.s390x
python2-dnf-plugins-core-0.1.21-5.fc25.noarch
xkeyboard-config-2.20-2.fc25.noarch
file-5.29-4.fc25.s390x
perl-Test-Harness-3.39-1.fc25.noarch
systemd-libs-231-17.fc25.s390x
webkitgtk4-jsc-2.16.5-1.fc25.s390x
gtk-update-icon-cache-3.22.16-1.fc25.s390x
glibc-devel-2.24-9.fc25.s390x
python3-pycparser-2.14-7.fc25.noarch
kernel-devel-4.11.10-200.fc25.s390x
gsm-1.0.17-1.fc25.s390x
vim-filesystem-8.0.705-1.fc25.s390x
webkitgtk4-2.16.5-1.fc25.s390x
python-2.7.13-2.fc25.s390x
glusterfs-fuse-3.10.4-1.fc25.s390x
git-core-2.9.4-1.fc25.s390x
selinux-policy-targeted-3.13.1-225.18.fc25.noarch
kernel-4.11.10-200.fc25.s390x
rpmconf-1.0.19-1.fc25.noarch
teamd-1.27-1.fc25.s390x
jasper-libs-1.900.13-4.fc25.s390x
libattr-2.4.47-16.fc24.s390x
libvisual-0.4.0-20.fc24.s390x
libpcap-1.7.4-2.fc24.s390x
libutempter-1.1.6-8.fc24.s390x
libgudev-230-3.fc24.s390x
popt-devel-1.16-7.fc24.s390x
hicolor-icon-theme-0.15-3.fc24.noarch
setup-2.10.4-1.fc25.noarch
bash-4.3.43-4.fc25.s390x
libjpeg-turbo-1.5.1-0.fc25.s390x
perl-Socket-2.024-1.fc25.s390x
perl-HTTP-Tiny-0.070-1.fc25.noarch
ipset-6.29-1.fc25.s390x
python2-setuptools-25.1.1-1.fc25.noarch
gsettings-desktop-schemas-3.22.0-1.fc25.s390x
python3-setuptools-25.1.1-1.fc25.noarch
python-slip-dbus-0.6.4-4.fc25.noarch
python2-ply-3.8-2.fc25.noarch
dtc-1.4.2-1.fc25.s390x
guile-2.0.13-1.fc25.s390x
cronie-anacron-1.5.1-2.fc25.s390x
libXtst-1.2.3-1.fc25.s390x
iso-codes-3.70-1.fc25.noarch
s390utils-1.36.0-1.fc25.s390x
python-backports-ssl_match_hostname-3.5.0.1-3.fc25.noarch
fedora-cert-0.6.0.1-1.fc25.noarch
libstdc++-6.3.1-1.fc25.s390x
subversion-libs-1.9.5-1.fc25.s390x
libgfortran-6.3.1-1.fc25.s390x
dnf-plugin-system-upgrade-0.7.1-4.fc25.noarch
lvm2-2.02.167-3.fc25.s390x
libselinux-devel-2.5-13.fc25.s390x
perl-Time-Local-1.250-1.fc25.noarch
libradosstriper-devel-10.2.4-2.fc25.s390x
flac-libs-1.3.2-1.fc25.s390x
perl-Digest-1.17-366.fc25.noarch
teckit-2.5.1-15.fc24.s390x
libpath_utils-0.2.1-29.fc25.s390x
attr-2.4.47-16.fc24.s390x
usbredir-0.7.1-2.fc24.s390x
cairo-devel-1.14.8-1.fc25.s390x
lzo-devel-2.08-8.fc24.s390x
libcap-devel-2.25-2.fc25.s390x
krb5-devel-1.14.4-7.fc25.s390x
rpm-4.13.0.1-1.fc25.s390x
kernel-devel-4.10.5-200.fc25.s390x
libbsd-0.8.3-1.fc25.s390x
texlive-url-svn32528.3.4-33.fc25.1.noarch
texlive-dvips-bin-svn40987-33.20160520.fc25.1.s390x
texlive-index-svn24099.4.1beta-33.fc25.1.noarch
texlive-setspace-svn24881.6.7a-33.fc25.1.noarch
texlive-mathtools-svn38833-33.fc25.1.noarch
texlive-cm-svn32865.0-33.fc25.1.noarch
texlive-graphics-def-svn41879-33.fc25.1.noarch
texlive-mdwtools-svn15878.1.05.4-33.fc25.1.noarch
texlive-rsfs-svn15878.0-33.fc25.1.noarch
texlive-ucharcat-svn38907-33.fc25.1.noarch
texlive-fontspec-svn41262-33.fc25.1.noarch
texlive-showexpl-svn32737.v0.3l-33.fc25.1.noarch
texlive-pstricks-add-svn40744-33.fc25.1.noarch
texlive-beamer-svn36461.3.36-33.fc25.1.noarch
texlive-collection-basic-svn41149-33.20160520.fc25.1.noarch
rpm-build-4.13.0.1-1.fc25.s390x
xemacs-filesystem-21.5.34-20.20170124hgf412e9f093d4.fc25.noarch
hawkey-0.6.4-3.fc25.s390x
gdk-pixbuf2-modules-2.36.6-1.fc25.s390x
bluez-libs-5.44-1.fc25.s390x
audit-libs-2.7.7-1.fc25.s390x
iproute-4.11.0-1.fc25.s390x
libICE-1.0.9-9.fc25.s390x
glibc-headers-2.24-9.fc25.s390x
python3-ply-3.8-2.fc25.noarch
perl-5.24.2-387.fc25.s390x
graphite2-1.3.10-1.fc25.s390x
vte-profile-0.46.2-1.fc25.s390x
gtk3-devel-3.22.16-1.fc25.s390x
python-libs-2.7.13-2.fc25.s390x
mesa-libGL-17.0.5-3.fc25.s390x
python2-pycurl-7.43.0-6.fc25.s390x
NetworkManager-1.4.4-5.fc25.s390x
mesa-libEGL-devel-17.0.5-3.fc25.s390x
mariadb-libs-10.1.24-3.fc25.s390x
mesa-libGLES-devel-17.0.5-3.fc25.s390x
hostname-3.15-8.fc25.s390x
gpg-pubkey-a0a7badb-52844296
readline-6.3-8.fc24.s390x
cpio-2.12-3.fc24.s390x
libXcomposite-0.4.4-8.fc24.s390x
procps-ng-3.3.10-11.fc24.s390x
GConf2-3.2.6-16.fc24.s390x
xz-devel-5.2.2-2.fc24.s390x
fedora-logos-22.0.0-3.fc24.s390x
gpg-pubkey-e372e838-56fd7943
kmod-libs-23-1.fc25.s390x
perl-parent-0.236-1.fc25.noarch
perl-TermReadKey-2.37-1.fc25.s390x
ncurses-c++-libs-6.0-6.20160709.fc25.s390x
gzip-1.8-1.fc25.s390x
python3-gobject-base-3.22.0-1.fc25.s390x
python2-yubico-1.3.2-3.fc25.noarch
s390utils-ziomon-1.36.0-1.fc25.s390x
librepo-1.7.18-3.fc25.s390x
librsvg2-2.40.16-2.fc25.s390x
gnat-srpm-macros-4-1.fc25.noarch
python-decoratortools-1.8-12.fc25.noarch
m17n-db-1.7.0-7.fc25.noarch
e2fsprogs-libs-1.43.3-1.fc25.s390x
libvorbis-1.3.5-1.fc25.s390x
gcc-gdb-plugin-6.3.1-1.fc25.s390x
npth-1.3-1.fc25.s390x
libcephfs1-10.2.4-2.fc25.s390x
wayland-devel-1.12.0-1.fc25.s390x
libxcb-devel-1.12-1.fc25.s390x
perl-encoding-2.19-5.fc25.s390x
python3-cssselect-0.9.2-1.fc25.noarch
gettext-libs-0.19.8.1-3.fc25.s390x
at-spi2-atk-devel-2.22.0-1.fc25.s390x
virglrenderer-0.5.0-1.20160411git61846f92f.fc25.s390x
pixman-devel-0.34.0-2.fc24.s390x
libnfs-devel-1.9.8-2.fc24.s390x
libblkid-2.28.2-2.fc25.s390x
glib2-devel-2.50.3-1.fc25.s390x
texlive-ifxetex-svn19685.0.5-33.fc25.1.noarch
texlive-caption-svn41409-33.fc25.1.noarch
texlive-float-svn15878.1.3d-33.fc25.1.noarch
texlive-pdftex-def-svn22653.0.06d-33.fc25.1.noarch
texlive-xdvi-bin-svn40750-33.20160520.fc25.1.s390x
texlive-beton-svn15878.0-33.fc25.1.noarch
texlive-filecontents-svn24250.1.3-33.fc25.1.noarch
texlive-lm-math-svn36915.1.959-33.fc25.1.noarch
texlive-pslatex-svn16416.0-33.fc25.1.noarch
texlive-times-svn35058.0-33.fc25.1.noarch
texlive-breakurl-svn29901.1.40-33.fc25.1.noarch
texlive-filehook-svn24280.0.5d-33.fc25.1.noarch
texlive-pst-pdf-svn31660.1.1v-33.fc25.1.noarch
texlive-seminar-svn34011.1.62-33.fc25.1.noarch
texlive-xetexconfig-svn41133-33.fc25.1.noarch
python-rpm-macros-3-12.fc25.noarch
rpm-devel-4.13.0.1-1.fc25.s390x
nss-pem-1.0.3-3.fc25.s390x
at-spi2-core-2.22.1-1.fc25.s390x
GeoIP-GeoLite-data-2017.04-1.fc25.noarch
kernel-devel-4.10.8-200.fc25.s390x
dbus-libs-1.11.14-1.fc25.s390x
perl-Scalar-List-Utils-1.48-1.fc25.s390x
libidn2-2.0.2-1.fc25.s390x
libtasn1-devel-4.12-1.fc25.s390x
python3-koji-1.13.0-2.fc25.noarch
glusterfs-cli-3.10.4-1.fc25.s390x
opus-1.1.5-1.fc25.s390x
mariadb-common-10.1.24-3.fc25.s390x
elfutils-libs-0.169-1.fc25.s390x
kernel-core-4.11.10-200.fc25.s390x
gnutls-dane-3.5.14-1.fc25.s390x
systemd-container-231-17.fc25.s390x
sudo-1.8.20p2-1.fc25.s390x
dbus-devel-1.11.14-1.fc25.s390x
perl-Module-CoreList-5.20170621-1.fc25.noarch
libicu-devel-57.1-5.fc25.s390x
js-jquery-2.2.4-3.fc25.noarch
=== TEST BEGIN ===
Using CC: /home/fam/bin/cc
Install prefix    /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install
BIOS directory    /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/share/qemu
binary directory  /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/bin
library directory /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/lib
module directory  /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/lib/qemu
libexec directory /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/libexec
include directory /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/include
config directory  /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/etc
local state directory   /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/var
Manual directory  /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path       /var/tmp/patchew-tester-tmp-hm2hdbnw/src
C compiler        /home/fam/bin/cc
Host C compiler   cc
C++ compiler      c++
Objective-C compiler /home/fam/bin/cc
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/include/pixman-1  -Werror -DHAS_LIBSSH2_SFTP_FSYNC -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DNCURSES_WIDECHAR -D_GNU_SOURCE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1    -I/usr/include/libpng16 -I/usr/include/libdrm  -I/usr/include/cacard -I/usr/include/nss3 -I/usr/include/nspr4 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/libusb-1.0 
LDFLAGS           -Wl,--warn-common -m64 -g 
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          s390x
host big endian   yes
target list       aarch64-softmmu alpha-softmmu arm-softmmu cris-softmmu i386-softmmu lm32-softmmu m68k-softmmu microblazeel-softmmu microblaze-softmmu mips64el-softmmu mips64-softmmu mipsel-softmmu mips-softmmu moxie-softmmu nios2-softmmu or1k-softmmu ppc64-softmmu ppcemb-softmmu ppc-softmmu s390x-softmmu sh4eb-softmmu sh4-softmmu sparc64-softmmu sparc-softmmu tricore-softmmu unicore32-softmmu x86_64-softmmu xtensaeb-softmmu xtensa-softmmu aarch64-linux-user alpha-linux-user armeb-linux-user arm-linux-user cris-linux-user hppa-linux-user i386-linux-user m68k-linux-user microblazeel-linux-user microblaze-linux-user mips64el-linux-user mips64-linux-user mipsel-linux-user mips-linux-user mipsn32el-linux-user mipsn32-linux-user nios2-linux-user or1k-linux-user ppc64abi32-linux-user ppc64le-linux-user ppc64-linux-user ppc-linux-user s390x-linux-user sh4eb-linux-user sh4-linux-user sparc32plus-linux-user sparc64-linux-user sparc-linux-user tilegx-linux-user x86_64-linux-user
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
pixman            system
SDL support       yes (2.0.5)
GTK support       yes (3.22.16)
GTK GL support    yes
VTE support       yes (0.46.2)
TLS priority      NORMAL
GNUTLS support    yes
GNUTLS rnd        yes
libgcrypt         no
libgcrypt kdf     no
nettle            yes (3.3)
nettle kdf        yes
libtasn1          yes
curses support    yes
virgl support     yes
curl support      yes
mingw32 support   no
Audio drivers     oss
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    yes
VNC support       yes
VNC SASL support  yes
VNC JPEG support  yes
VNC PNG support   yes
xen support       no
brlapi support    yes
bluez  support    yes
Documentation     yes
PIE               no
vde support       no
netmap support    no
Linux AIO support yes
ATTR/XATTR support yes
Install blobs     yes
KVM support       yes
HAX support       no
TCG support       yes
TCG debug enabled no
TCG interpreter   no
RDMA support      no
fdt support       yes
preadv support    yes
fdatasync         yes
madvise           yes
posix_madvise     yes
libcap-ng support yes
vhost-net support yes
vhost-scsi support yes
vhost-vsock support yes
vhost-user support yes
Trace backends    log
spice support     no 
rbd support       yes
xfsctl support    no
smartcard support yes
libusb            yes
usb net redir     yes
OpenGL support    yes
OpenGL dmabufs    yes
libiscsi support  yes
libnfs support    yes
build guest agent yes
QGA VSS support   no
QGA w32 disk info no
QGA MSI support   no
seccomp support   no
coroutine backend ucontext
coroutine pool    yes
debug stack usage no
crypto afalg      no
GlusterFS support yes
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   yes
TPM passthrough   no
QOM debugging     yes
Live block migration yes
lzo support       yes
snappy support    yes
bzip2 support     yes
NUMA host support no
tcmalloc support  no
jemalloc support  no
avx2 optimization no
replication support yes
VxHS block device no
  GEN     aarch64-softmmu/config-devices.mak.tmp
  GEN     cris-softmmu/config-devices.mak.tmp
  GEN     alpha-softmmu/config-devices.mak.tmp
  GEN     arm-softmmu/config-devices.mak.tmp
  GEN     cris-softmmu/config-devices.mak
  GEN     i386-softmmu/config-devices.mak.tmp
  GEN     alpha-softmmu/config-devices.mak
  GEN     arm-softmmu/config-devices.mak
  GEN     lm32-softmmu/config-devices.mak.tmp
  GEN     m68k-softmmu/config-devices.mak.tmp
  GEN     aarch64-softmmu/config-devices.mak
  GEN     microblazeel-softmmu/config-devices.mak.tmp
  GEN     lm32-softmmu/config-devices.mak
  GEN     i386-softmmu/config-devices.mak
  GEN     m68k-softmmu/config-devices.mak
  GEN     microblaze-softmmu/config-devices.mak.tmp
  GEN     mips64el-softmmu/config-devices.mak.tmp
  GEN     mips64-softmmu/config-devices.mak.tmp
  GEN     microblazeel-softmmu/config-devices.mak
  GEN     microblaze-softmmu/config-devices.mak
  GEN     mipsel-softmmu/config-devices.mak.tmp
  GEN     mips-softmmu/config-devices.mak.tmp
  GEN     mips64-softmmu/config-devices.mak
  GEN     mips64el-softmmu/config-devices.mak
  GEN     nios2-softmmu/config-devices.mak.tmp
  GEN     moxie-softmmu/config-devices.mak.tmp
  GEN     mipsel-softmmu/config-devices.mak
  GEN     moxie-softmmu/config-devices.mak
  GEN     nios2-softmmu/config-devices.mak
  GEN     mips-softmmu/config-devices.mak
  GEN     ppc64-softmmu/config-devices.mak.tmp
  GEN     ppcemb-softmmu/config-devices.mak.tmp
  GEN     or1k-softmmu/config-devices.mak.tmp
  GEN     ppc-softmmu/config-devices.mak.tmp
  GEN     or1k-softmmu/config-devices.mak
  GEN     s390x-softmmu/config-devices.mak.tmp
  GEN     ppc64-softmmu/config-devices.mak
  GEN     ppcemb-softmmu/config-devices.mak
  GEN     sh4eb-softmmu/config-devices.mak.tmp
  GEN     ppc-softmmu/config-devices.mak
  GEN     sh4-softmmu/config-devices.mak.tmp
  GEN     sparc64-softmmu/config-devices.mak.tmp
  GEN     s390x-softmmu/config-devices.mak
  GEN     sparc-softmmu/config-devices.mak.tmp
  GEN     sh4eb-softmmu/config-devices.mak
  GEN     sh4-softmmu/config-devices.mak
  GEN     tricore-softmmu/config-devices.mak.tmp
  GEN     sparc64-softmmu/config-devices.mak
  GEN     unicore32-softmmu/config-devices.mak.tmp
  GEN     x86_64-softmmu/config-devices.mak.tmp
  GEN     sparc-softmmu/config-devices.mak
  GEN     xtensaeb-softmmu/config-devices.mak.tmp
  GEN     unicore32-softmmu/config-devices.mak
  GEN     tricore-softmmu/config-devices.mak
  GEN     xtensa-softmmu/config-devices.mak.tmp
  GEN     aarch64-linux-user/config-devices.mak.tmp
  GEN     xtensaeb-softmmu/config-devices.mak
  GEN     alpha-linux-user/config-devices.mak.tmp
  GEN     xtensa-softmmu/config-devices.mak
  GEN     x86_64-softmmu/config-devices.mak
  GEN     aarch64-linux-user/config-devices.mak
  GEN     armeb-linux-user/config-devices.mak.tmp
  GEN     alpha-linux-user/config-devices.mak
  GEN     arm-linux-user/config-devices.mak.tmp
  GEN     cris-linux-user/config-devices.mak.tmp
  GEN     hppa-linux-user/config-devices.mak.tmp
  GEN     armeb-linux-user/config-devices.mak
  GEN     arm-linux-user/config-devices.mak
  GEN     cris-linux-user/config-devices.mak
  GEN     i386-linux-user/config-devices.mak.tmp
  GEN     m68k-linux-user/config-devices.mak.tmp
  GEN     microblazeel-linux-user/config-devices.mak.tmp
  GEN     hppa-linux-user/config-devices.mak
  GEN     microblaze-linux-user/config-devices.mak.tmp
  GEN     i386-linux-user/config-devices.mak
  GEN     m68k-linux-user/config-devices.mak
  GEN     microblazeel-linux-user/config-devices.mak
  GEN     mips64el-linux-user/config-devices.mak.tmp
  GEN     mips64-linux-user/config-devices.mak.tmp
  GEN     mipsel-linux-user/config-devices.mak.tmp
  GEN     microblaze-linux-user/config-devices.mak
  GEN     mips64el-linux-user/config-devices.mak
  GEN     mips-linux-user/config-devices.mak.tmp
  GEN     mipsel-linux-user/config-devices.mak
  GEN     mipsn32el-linux-user/config-devices.mak.tmp
  GEN     mips64-linux-user/config-devices.mak
  GEN     mipsn32-linux-user/config-devices.mak.tmp
  GEN     mips-linux-user/config-devices.mak
  GEN     or1k-linux-user/config-devices.mak.tmp
  GEN     nios2-linux-user/config-devices.mak.tmp
  GEN     mipsn32el-linux-user/config-devices.mak
  GEN     mipsn32-linux-user/config-devices.mak
  GEN     ppc64le-linux-user/config-devices.mak.tmp
  GEN     ppc64abi32-linux-user/config-devices.mak.tmp
  GEN     nios2-linux-user/config-devices.mak
  GEN     or1k-linux-user/config-devices.mak
  GEN     ppc64-linux-user/config-devices.mak.tmp
  GEN     ppc-linux-user/config-devices.mak.tmp
  GEN     ppc64abi32-linux-user/config-devices.mak
  GEN     ppc64le-linux-user/config-devices.mak
  GEN     sh4eb-linux-user/config-devices.mak.tmp
  GEN     s390x-linux-user/config-devices.mak.tmp
  GEN     ppc64-linux-user/config-devices.mak
  GEN     ppc-linux-user/config-devices.mak
  GEN     sh4-linux-user/config-devices.mak.tmp
  GEN     sh4eb-linux-user/config-devices.mak
  GEN     sparc32plus-linux-user/config-devices.mak.tmp
  GEN     sparc64-linux-user/config-devices.mak.tmp
  GEN     s390x-linux-user/config-devices.mak
  GEN     sh4-linux-user/config-devices.mak
  GEN     sparc32plus-linux-user/config-devices.mak
  GEN     sparc-linux-user/config-devices.mak.tmp
  GEN     tilegx-linux-user/config-devices.mak.tmp
  GEN     x86_64-linux-user/config-devices.mak.tmp
  GEN     sparc64-linux-user/config-devices.mak
  GEN     sparc-linux-user/config-devices.mak
  GEN     config-host.h
  GEN     tilegx-linux-user/config-devices.mak
  GEN     qemu-options.def
  GEN     x86_64-linux-user/config-devices.mak
  GEN     qmp-commands.h
  GEN     qapi-types.h
  GEN     qapi-visit.h
  GEN     qapi-event.h
  GEN     qmp-marshal.c
  GEN     qapi-types.c
  GEN     qapi-visit.c
  GEN     qapi-event.c
  GEN     qmp-introspect.h
  GEN     qmp-introspect.c
  GEN     trace/generated-tcg-tracers.h
  GEN     trace/generated-helpers-wrappers.h
  GEN     trace/generated-helpers.c
  GEN     trace/generated-helpers.h
  GEN     module_block.h
  GEN     tests/test-qapi-types.h
  GEN     tests/test-qapi-visit.h
  GEN     tests/test-qmp-commands.h
  GEN     tests/test-qmp-introspect.h
  GEN     tests/test-qapi-event.h
  GEN     trace-root.h
  GEN     crypto/trace.h
  GEN     io/trace.h
  GEN     util/trace.h
  GEN     migration/trace.h
  GEN     block/trace.h
  GEN     chardev/trace.h
  GEN     hw/block/trace.h
  GEN     hw/block/dataplane/trace.h
  GEN     hw/char/trace.h
  GEN     hw/intc/trace.h
  GEN     hw/net/trace.h
  GEN     hw/virtio/trace.h
  GEN     hw/audio/trace.h
  GEN     hw/misc/trace.h
  GEN     hw/usb/trace.h
  GEN     hw/nvram/trace.h
  GEN     hw/scsi/trace.h
  GEN     hw/display/trace.h
  GEN     hw/input/trace.h
  GEN     hw/timer/trace.h
  GEN     hw/dma/trace.h
  GEN     hw/sparc/trace.h
  GEN     hw/isa/trace.h
  GEN     hw/sd/trace.h
  GEN     hw/mem/trace.h
  GEN     hw/i386/trace.h
  GEN     hw/i386/xen/trace.h
  GEN     hw/ppc/trace.h
  GEN     hw/pci/trace.h
  GEN     hw/9pfs/trace.h
  GEN     hw/s390x/trace.h
  GEN     hw/vfio/trace.h
  GEN     hw/acpi/trace.h
  GEN     hw/arm/trace.h
  GEN     hw/alpha/trace.h
  GEN     hw/xen/trace.h
  GEN     ui/trace.h
  GEN     audio/trace.h
  GEN     net/trace.h
  GEN     target/arm/trace.h
  GEN     target/i386/trace.h
  GEN     target/mips/trace.h
  GEN     target/s390x/trace.h
  GEN     target/ppc/trace.h
  GEN     target/sparc/trace.h
  GEN     qom/trace.h
  GEN     linux-user/trace.h
  GEN     qapi/trace.h
  GEN     accel/tcg/trace.h
  GEN     accel/kvm/trace.h
  GEN     trace-root.c
  GEN     nbd/trace.h
  GEN     util/trace.c
  GEN     crypto/trace.c
  GEN     io/trace.c
  GEN     migration/trace.c
  GEN     chardev/trace.c
  GEN     block/trace.c
  GEN     hw/block/trace.c
  GEN     hw/block/dataplane/trace.c
  GEN     hw/char/trace.c
  GEN     hw/intc/trace.c
  GEN     hw/net/trace.c
  GEN     hw/virtio/trace.c
  GEN     hw/audio/trace.c
  GEN     hw/misc/trace.c
  GEN     hw/usb/trace.c
  GEN     hw/scsi/trace.c
  GEN     hw/nvram/trace.c
  GEN     hw/display/trace.c
  GEN     hw/input/trace.c
  GEN     hw/dma/trace.c
  GEN     hw/timer/trace.c
  GEN     hw/sparc/trace.c
  GEN     hw/isa/trace.c
  GEN     hw/sd/trace.c
  GEN     hw/mem/trace.c
  GEN     hw/i386/trace.c
  GEN     hw/i386/xen/trace.c
  GEN     hw/9pfs/trace.c
  GEN     hw/ppc/trace.c
  GEN     hw/pci/trace.c
  GEN     hw/s390x/trace.c
  GEN     hw/vfio/trace.c
  GEN     hw/acpi/trace.c
  GEN     hw/arm/trace.c
  GEN     hw/alpha/trace.c
  GEN     ui/trace.c
  GEN     hw/xen/trace.c
  GEN     audio/trace.c
  GEN     net/trace.c
  GEN     target/arm/trace.c
  GEN     target/i386/trace.c
  GEN     target/mips/trace.c
  GEN     target/sparc/trace.c
  GEN     target/s390x/trace.c
  GEN     target/ppc/trace.c
  GEN     qom/trace.c
  GEN     linux-user/trace.c
  GEN     qapi/trace.c
  GEN     accel/tcg/trace.c
  GEN     accel/kvm/trace.c
  GEN     config-all-devices.mak
  GEN     nbd/trace.c
  GEN     docs/version.texi
  GEN     qemu-monitor.texi
  GEN     qemu-options.texi
  CC      tests/qemu-iotests/socket_scm_helper.o
  GEN     qemu-img-cmds.texi
  GEN     qemu-monitor-info.texi
  GEN     qemu-img.1
  GEN     qemu-nbd.8
  GEN     qemu-ga.8
  GEN     docs/interop/qemu-qmp-qapi.texi
  GEN     docs/interop/qemu-ga-qapi.texi
  GEN     fsdev/virtfs-proxy-helper.1
  GEN     qga/qapi-generated/qga-qapi-types.h
  GEN     qga/qapi-generated/qga-qapi-visit.h
  GEN     qga/qapi-generated/qga-qapi-types.c
  GEN     qga/qapi-generated/qga-qapi-visit.c
  GEN     qga/qapi-generated/qga-qmp-commands.h
  CC      qmp-introspect.o
  GEN     qga/qapi-generated/qga-qmp-marshal.c
  CC      qapi-types.o
  CC      qapi-event.o
  CC      qapi-visit.o
  CC      qapi/qapi-visit-core.o
  CC      qapi/qapi-dealloc-visitor.o
  CC      qapi/qobject-input-visitor.o
  CC      qapi/qobject-output-visitor.o
  CC      qapi/qmp-registry.o
  CC      qapi/qmp-dispatch.o
  CC      qapi/string-input-visitor.o
  CC      qapi/string-output-visitor.o
  CC      qapi/opts-visitor.o
  CC      qapi/qapi-clone-visitor.o
  CC      qapi/qmp-event.o
  CC      qapi/qapi-util.o
  CC      qobject/qnull.o
  CC      qobject/qnum.o
  CC      qobject/qstring.o
  CC      qobject/qdict.o
  CC      qobject/qlist.o
  CC      qobject/qbool.o
  CC      qobject/qjson.o
  CC      qobject/qobject.o
  CC      qobject/json-lexer.o
  CC      qobject/json-streamer.o
  CC      qobject/json-parser.o
  CC      trace/control.o
  CC      trace/qmp.o
  CC      util/osdep.o
  CC      util/cutils.o
  CC      util/unicode.o
  CC      util/qemu-timer-common.o
  CC      util/bufferiszero.o
  CC      util/lockcnt.o
  CC      util/aiocb.o
  CC      util/thread-pool.o
  CC      util/async.o
  CC      util/qemu-timer.o
  CC      util/iohandler.o
  CC      util/main-loop.o
  CC      util/aio-posix.o
  CC      util/compatfd.o
  CC      util/event_notifier-posix.o
  CC      util/mmap-alloc.o
  CC      util/oslib-posix.o
  CC      util/qemu-openpty.o
  CC      util/qemu-thread-posix.o
  CC      util/memfd.o
  CC      util/envlist.o
  CC      util/path.o
  CC      util/module.o
  CC      util/host-utils.o
  CC      util/bitops.o
  CC      util/bitmap.o
  CC      util/hbitmap.o
  CC      util/fifo8.o
  CC      util/acl.o
  CC      util/cacheinfo.o
  CC      util/error.o
  CC      util/qemu-error.o
  CC      util/id.o
  CC      util/iov.o
  CC      util/qemu-config.o
  CC      util/qemu-sockets.o
  CC      util/notify.o
  CC      util/uri.o
  CC      util/qemu-option.o
  CC      util/qemu-progress.o
  CC      util/keyval.o
  CC      util/hexdump.o
  CC      util/crc32c.o
  CC      util/uuid.o
  CC      util/throttle.o
  CC      util/getauxval.o
  CC      util/readline.o
  CC      util/rcu.o
  CC      util/qemu-coroutine.o
  CC      util/qemu-coroutine-lock.o
  CC      util/qemu-coroutine-io.o
  CC      util/coroutine-ucontext.o
  CC      util/qemu-coroutine-sleep.o
  CC      util/buffer.o
  CC      util/timed-average.o
  CC      util/base64.o
  CC      util/qdist.o
  CC      util/log.o
  CC      util/qht.o
  CC      util/range.o
  CC      util/stats64.o
  CC      util/systemd.o
  CC      trace-root.o
  CC      util/trace.o
  CC      crypto/trace.o
  CC      io/trace.o
  CC      migration/trace.o
  CC      block/trace.o
  CC      chardev/trace.o
  CC      hw/block/trace.o
  CC      hw/block/dataplane/trace.o
  CC      hw/char/trace.o
  CC      hw/intc/trace.o
  CC      hw/net/trace.o
  CC      hw/virtio/trace.o
  CC      hw/audio/trace.o
  CC      hw/misc/trace.o
  CC      hw/usb/trace.o
  CC      hw/scsi/trace.o
  CC      hw/nvram/trace.o
  CC      hw/display/trace.o
  CC      hw/input/trace.o
  CC      hw/timer/trace.o
  CC      hw/dma/trace.o
  CC      hw/sparc/trace.o
  CC      hw/sd/trace.o
  CC      hw/isa/trace.o
  CC      hw/mem/trace.o
  CC      hw/i386/trace.o
  CC      hw/i386/xen/trace.o
  CC      hw/9pfs/trace.o
  CC      hw/ppc/trace.o
  CC      hw/pci/trace.o
  CC      hw/s390x/trace.o
  CC      hw/vfio/trace.o
  CC      hw/acpi/trace.o
  CC      hw/arm/trace.o
  CC      hw/alpha/trace.o
  CC      hw/xen/trace.o
  CC      ui/trace.o
  CC      audio/trace.o
  CC      net/trace.o
  CC      target/i386/trace.o
  CC      target/arm/trace.o
  CC      target/mips/trace.o
  CC      target/s390x/trace.o
  CC      target/sparc/trace.o
  CC      target/ppc/trace.o
  CC      qom/trace.o
  CC      qapi/trace.o
  CC      linux-user/trace.o
  CC      accel/tcg/trace.o
  CC      accel/kvm/trace.o
  CC      nbd/trace.o
  CC      crypto/pbkdf-stub.o
  CC      stubs/arch-query-cpu-def.o
  CC      stubs/arch-query-cpu-model-expansion.o
  CC      stubs/arch-query-cpu-model-comparison.o
  CC      stubs/arch-query-cpu-model-baseline.o
  CC      stubs/bdrv-next-monitor-owned.o
  CC      stubs/blk-commit-all.o
  CC      stubs/clock-warp.o
  CC      stubs/blockdev-close-all-bdrv-states.o
  CC      stubs/cpu-get-clock.o
  CC      stubs/cpu-get-icount.o
  CC      stubs/dump.o
  CC      stubs/error-printf.o
  CC      stubs/fdset.o
  CC      stubs/gdbstub.o
  CC      stubs/iothread.o
  CC      stubs/get-vm-name.o
  CC      stubs/iothread-lock.o
  CC      stubs/is-daemonized.o
  CC      stubs/linux-aio.o
  CC      stubs/machine-init-done.o
  CC      stubs/migr-blocker.o
  CC      stubs/change-state-handler.o
  CC      stubs/monitor.o
  CC      stubs/notify-event.o
  CC      stubs/qtest.o
  CC      stubs/runstate-check.o
  CC      stubs/replay.o
  CC      stubs/set-fd-handler.o
  CC      stubs/slirp.o
  CC      stubs/sysbus.o
  CC      stubs/trace-control.o
  CC      stubs/uuid.o
  CC      stubs/vm-stop.o
  CC      stubs/vmstate.o
  CC      stubs/qmp_pc_dimm_device_list.o
  CC      stubs/target-monitor-defs.o
  CC      stubs/target-get-monitor-def.o
  CC      stubs/pc_madt_cpu_entry.o
  CC      stubs/vmgenid.o
  CC      stubs/xen-common.o
  CC      stubs/xen-hvm.o
  CC      contrib/ivshmem-client/ivshmem-client.o
  CC      contrib/ivshmem-client/main.o
  CC      contrib/ivshmem-server/ivshmem-server.o
  CC      contrib/ivshmem-server/main.o
  CC      qemu-nbd.o
  CC      block.o
  CC      blockjob.o
  CC      qemu-io-cmds.o
  CC      replication.o
  CC      block/raw-format.o
  CC      block/qcow.o
  CC      block/vdi.o
  CC      block/vmdk.o
  CC      block/cloop.o
  CC      block/bochs.o
  CC      block/vpc.o
  CC      block/vvfat.o
  CC      block/dmg.o
  CC      block/qcow2.o
  CC      block/qcow2-refcount.o
  CC      block/qcow2-cluster.o
  CC      block/qcow2-snapshot.o
  CC      block/qcow2-cache.o
  CC      block/qcow2-bitmap.o
  CC      block/qed.o
  CC      block/qed-l2-cache.o
  CC      block/qed-table.o
  CC      block/qed-cluster.o
  CC      block/qed-check.o
  CC      block/vhdx.o
  CC      block/vhdx-endian.o
  CC      block/vhdx-log.o
  CC      block/quorum.o
  CC      block/parallels.o
  CC      block/blkverify.o
  CC      block/blkdebug.o
  CC      block/blkreplay.o
  CC      block/block-backend.o
  CC      block/snapshot.o
  CC      block/qapi.o
  CC      block/file-posix.o
  CC      block/linux-aio.o
  CC      block/null.o
  CC      block/mirror.o
  CC      block/commit.o
  CC      block/io.o
  CC      block/throttle-groups.o
  CC      block/nbd.o
  CC      block/nbd-client.o
  CC      block/sheepdog.o
  CC      block/iscsi-opts.o
  CC      block/accounting.o
  CC      block/write-threshold.o
  CC      block/dirty-bitmap.o
  CC      block/backup.o
  CC      block/replication.o
  CC      block/crypto.o
  CC      nbd/server.o
  CC      nbd/client.o
  CC      nbd/common.o
  CC      block/iscsi.o
  CC      block/nfs.o
  CC      block/curl.o
  CC      block/rbd.o
  CC      block/gluster.o
  CC      block/ssh.o
  CC      block/dmg-bz2.o
  CC      crypto/init.o
  CC      crypto/hash.o
  CC      crypto/hash-nettle.o
  CC      crypto/hmac.o
  CC      crypto/aes.o
  CC      crypto/hmac-nettle.o
  CC      crypto/desrfb.o
  CC      crypto/cipher.o
  CC      crypto/tlscreds.o
  CC      crypto/tlscredsanon.o
  CC      crypto/tlscredsx509.o
  CC      crypto/tlssession.o
  CC      crypto/secret.o
  CC      crypto/random-gnutls.o
  CC      crypto/pbkdf.o
  CC      crypto/pbkdf-nettle.o
  CC      crypto/ivgen.o
  CC      crypto/ivgen-essiv.o
  CC      crypto/ivgen-plain64.o
  CC      crypto/ivgen-plain.o
  CC      crypto/afsplit.o
  CC      crypto/xts.o
  CC      crypto/block.o
  CC      crypto/block-qcow.o
  CC      io/channel.o
  CC      crypto/block-luks.o
  CC      io/channel-buffer.o
  CC      io/channel-command.o
  CC      io/channel-file.o
  CC      io/channel-socket.o
  CC      io/channel-tls.o
  CC      io/channel-websock.o
  CC      io/channel-watch.o
  CC      io/channel-util.o
  CC      io/dns-resolver.o
  CC      io/task.o
  CC      qom/object.o
  CC      qom/container.o
  CC      qom/qom-qobject.o
  CC      qom/object_interfaces.o
  GEN     qemu-img-cmds.h
  CC      qemu-io.o
  CC      fsdev/virtfs-proxy-helper.o
  CC      fsdev/9p-marshal.o
  CC      fsdev/9p-iov-marshal.o
  CC      qemu-bridge-helper.o
  CC      blockdev.o
  CC      blockdev-nbd.o
  CC      bootdevice.o
  CC      iothread.o
  CC      qdev-monitor.o
  CC      device-hotplug.o
  CC      os-posix.o
  CC      bt-host.o
  CC      bt-vhci.o
  CC      dma-helpers.o
  CC      vl.o
  CC      tpm.o
  CC      device_tree.o
  CC      qmp-marshal.o
  CC      qmp.o
  CC      hmp.o
  CC      cpus-common.o
  CC      audio/audio.o
  CC      audio/noaudio.o
  CC      audio/wavaudio.o
  CC      audio/mixeng.o
  CC      audio/sdlaudio.o
  CC      audio/ossaudio.o
  CC      audio/wavcapture.o
  CC      backends/rng-egd.o
  CC      backends/rng.o
  CC      backends/rng-random.o
  CC      backends/tpm.o
  CC      backends/hostmem.o
  CC      backends/hostmem-ram.o
  CC      backends/hostmem-file.o
  CC      backends/cryptodev.o
  CC      backends/cryptodev-builtin.o
  CC      block/stream.o
  CC      chardev/msmouse.o
  CC      chardev/wctablet.o
  CC      chardev/testdev.o
  CC      chardev/baum.o
  CC      disas/alpha.o
  CC      disas/arm.o
  CXX     disas/arm-a64.o
  CC      disas/cris.o
  CC      disas/hppa.o
  CC      disas/i386.o
  CC      disas/m68k.o
  CC      disas/microblaze.o
  CC      disas/mips.o
  CC      disas/nios2.o
  CC      disas/moxie.o
  CC      disas/ppc.o
  CC      disas/s390.o
  CC      disas/sh4.o
  CC      disas/sparc.o
  CC      disas/lm32.o
  CXX     disas/libvixl/vixl/utils.o
  CXX     disas/libvixl/vixl/compiler-intrinsics.o
  CXX     disas/libvixl/vixl/a64/decoder-a64.o
  CXX     disas/libvixl/vixl/a64/instructions-a64.o
  CXX     disas/libvixl/vixl/a64/disasm-a64.o
  CC      fsdev/qemu-fsdev.o
  CC      fsdev/qemu-fsdev-opts.o
  CC      fsdev/qemu-fsdev-throttle.o
  CC      fsdev/qemu-fsdev-dummy.o
  CC      hw/9pfs/9p.o
  CC      hw/9pfs/9p-util.o
  CC      hw/9pfs/9p-local.o
  CC      hw/9pfs/9p-xattr.o
  CC      hw/9pfs/9p-xattr-user.o
  CC      hw/9pfs/9p-posix-acl.o
  CC      hw/9pfs/cofs.o
  CC      hw/9pfs/coth.o
  CC      hw/9pfs/codir.o
  CC      hw/9pfs/cofile.o
  CC      hw/9pfs/9p-synth.o
  CC      hw/9pfs/coxattr.o
  CC      hw/9pfs/9p-proxy.o
  CC      hw/9pfs/9p-handle.o
  CC      hw/acpi/core.o
  CC      hw/acpi/piix4.o
  CC      hw/acpi/pcihp.o
  CC      hw/acpi/ich9.o
  CC      hw/acpi/tco.o
  CC      hw/acpi/cpu_hotplug.o
  CC      hw/acpi/memory_hotplug.o
  CC      hw/acpi/cpu.o
  CC      hw/acpi/nvdimm.o
  CC      hw/acpi/vmgenid.o
  CC      hw/acpi/hest_ghes.o
  CC      hw/acpi/acpi_interface.o
  CC      hw/acpi/bios-linker-loader.o
  CC      hw/acpi/aml-build.o
  CC      hw/acpi/ipmi.o
  CC      hw/acpi/acpi-stub.o
  CC      hw/acpi/ipmi-stub.o
  CC      hw/audio/sb16.o
  CC      hw/audio/es1370.o
  CC      hw/audio/ac97.o
  CC      hw/audio/fmopl.o
  CC      hw/audio/adlib.o
  CC      hw/audio/gus.o
  CC      hw/audio/gusemu_hal.o
  CC      hw/audio/gusemu_mixer.o
  CC      hw/audio/cs4231a.o
  CC      hw/audio/intel-hda.o
  CC      hw/audio/hda-codec.o
  CC      hw/audio/wm8750.o
  CC      hw/audio/pcspk.o
  CC      hw/audio/pl041.o
  CC      hw/audio/lm4549.o
  CC      hw/audio/cs4231.o
  CC      hw/audio/marvell_88w8618.o
  CC      hw/audio/milkymist-ac97.o
  CC      hw/audio/soundhw.o
  CC      hw/block/block.o
  CC      hw/block/cdrom.o
  CC      hw/block/hd-geometry.o
  CC      hw/block/fdc.o
  CC      hw/block/m25p80.o
  CC      hw/block/nand.o
  CC      hw/block/pflash_cfi01.o
  CC      hw/block/pflash_cfi02.o
  CC      hw/block/ecc.o
  CC      hw/block/onenand.o
  CC      hw/block/nvme.o
  CC      hw/bt/l2cap.o
  CC      hw/bt/core.o
  CC      hw/bt/sdp.o
  CC      hw/bt/hci.o
  CC      hw/bt/hid.o
  CC      hw/bt/hci-csr.o
  CC      hw/char/ipoctal232.o
  CC      hw/char/escc.o
  CC      hw/char/parallel.o
  CC      hw/char/pl011.o
  CC      hw/char/serial.o
  CC      hw/char/serial-isa.o
  CC      hw/char/serial-pci.o
  CC      hw/char/virtio-console.o
  CC      hw/char/xilinx_uartlite.o
  CC      hw/char/cadence_uart.o
  CC      hw/char/cmsdk-apb-uart.o
  CC      hw/char/etraxfs_ser.o
  CC      hw/char/debugcon.o
  CC      hw/char/grlib_apbuart.o
  CC      hw/char/lm32_juart.o
  CC      hw/char/imx_serial.o
  CC      hw/char/lm32_uart.o
  CC      hw/char/milkymist-uart.o
  CC      hw/char/sclpconsole.o
  CC      hw/char/sclpconsole-lm.o
  CC      hw/core/qdev.o
  CC      hw/core/qdev-properties.o
  CC      hw/core/bus.o
  CC      hw/core/reset.o
  CC      hw/core/fw-path-provider.o
  CC      hw/core/irq.o
  CC      hw/core/hotplug.o
  CC      hw/core/nmi.o
  CC      hw/core/empty_slot.o
  CC      hw/core/ptimer.o
  CC      hw/core/stream.o
  CC      hw/core/sysbus.o
  CC      hw/core/machine.o
  CC      hw/core/loader-fit.o
  CC      hw/core/loader.o
  CC      hw/core/qdev-properties-system.o
  CC      hw/core/register.o
  CC      hw/core/or-irq.o
  CC      hw/core/platform-bus.o
  CC      hw/cpu/core.o
  CC      hw/display/ads7846.o
  CC      hw/display/g364fb.o
  CC      hw/display/cirrus_vga.o
  CC      hw/display/jazz_led.o
  CC      hw/display/pl110.o
  CC      hw/display/ssd0303.o
  CC      hw/display/ssd0323.o
  CC      hw/display/vga-pci.o
  CC      hw/display/vga-isa.o
  CC      hw/display/vga-isa-mm.o
  CC      hw/display/vmware_vga.o
  CC      hw/display/blizzard.o
  CC      hw/display/exynos4210_fimd.o
  CC      hw/display/framebuffer.o
  CC      hw/display/milkymist-vgafb.o
  CC      hw/display/tc6393xb.o
  CC      hw/display/milkymist-tmu2.o
  CC      hw/dma/puv3_dma.o
  CC      hw/dma/rc4030.o
  CC      hw/dma/pl080.o
  CC      hw/dma/pl330.o
  CC      hw/dma/i82374.o
  CC      hw/dma/i8257.o
  CC      hw/dma/xilinx_axidma.o
  CC      hw/dma/xlnx-zynq-devcfg.o
  CC      hw/dma/etraxfs_dma.o
  CC      hw/dma/sparc32_dma.o
  CC      hw/dma/sun4m_iommu.o
  CC      hw/gpio/max7310.o
  CC      hw/gpio/pl061.o
  CC      hw/gpio/puv3_gpio.o
  CC      hw/gpio/zaurus.o
  CC      hw/gpio/gpio_key.o
  CC      hw/gpio/mpc8xxx.o
  CC      hw/i2c/core.o
  CC      hw/i2c/smbus.o
  CC      hw/i2c/smbus_eeprom.o
  CC      hw/i2c/i2c-ddc.o
  CC      hw/i2c/versatile_i2c.o
  CC      hw/i2c/smbus_ich9.o
  CC      hw/i2c/pm_smbus.o
  CC      hw/i2c/bitbang_i2c.o
  CC      hw/i2c/exynos4210_i2c.o
  CC      hw/i2c/imx_i2c.o
  CC      hw/i2c/aspeed_i2c.o
  CC      hw/ide/core.o
  CC      hw/ide/atapi.o
  CC      hw/ide/qdev.o
  CC      hw/ide/pci.o
  CC      hw/ide/isa.o
  CC      hw/ide/piix.o
  CC      hw/ide/cmd646.o
  CC      hw/ide/macio.o
  CC      hw/ide/via.o
  CC      hw/ide/mmio.o
  CC      hw/ide/microdrive.o
  CC      hw/ide/ahci.o
  CC      hw/input/adb.o
  CC      hw/input/hid.o
  CC      hw/ide/ich.o
  CC      hw/input/lm832x.o
  CC      hw/input/pckbd.o
  CC      hw/input/pl050.o
  CC      hw/input/ps2.o
  CC      hw/input/stellaris_input.o
  CC      hw/input/tsc2005.o
  CC      hw/input/vmmouse.o
  CC      hw/input/virtio-input.o
  CC      hw/input/virtio-input-hid.o
  CC      hw/input/virtio-input-host.o
  CC      hw/intc/heathrow_pic.o
  CC      hw/intc/i8259_common.o
  CC      hw/intc/i8259.o
  CC      hw/intc/pl190.o
  CC      hw/intc/puv3_intc.o
  CC      hw/intc/xilinx_intc.o
  CC      hw/intc/imx_avic.o
  CC      hw/intc/etraxfs_pic.o
  CC      hw/intc/realview_gic.o
  CC      hw/intc/lm32_pic.o
  CC      hw/intc/slavio_intctl.o
  CC      hw/intc/ioapic_common.o
  CC      hw/intc/arm_gic_common.o
  CC      hw/intc/arm_gic.o
  CC      hw/intc/arm_gicv2m.o
  CC      hw/intc/arm_gicv3_common.o
  CC      hw/intc/arm_gicv3.o
  CC      hw/intc/arm_gicv3_dist.o
  CC      hw/intc/arm_gicv3_redist.o
  CC      hw/intc/arm_gicv3_its_common.o
  CC      hw/intc/openpic.o
  CC      hw/intc/intc.o
  CC      hw/ipack/ipack.o
  CC      hw/ipack/tpci200.o
  CC      hw/ipmi/ipmi.o
  CC      hw/ipmi/ipmi_bmc_sim.o
  CC      hw/ipmi/ipmi_bmc_extern.o
  CC      hw/ipmi/isa_ipmi_kcs.o
  CC      hw/ipmi/isa_ipmi_bt.o
  CC      hw/isa/isa-bus.o
  CC      hw/isa/apm.o
  CC      hw/isa/i82378.o
  CC      hw/isa/pc87312.o
  CC      hw/isa/piix4.o
  CC      hw/isa/vt82c686.o
  CC      hw/mem/pc-dimm.o
  CC      hw/mem/nvdimm.o
  CC      hw/misc/applesmc.o
  CC      hw/misc/max111x.o
  CC      hw/misc/tmp105.o
  CC      hw/misc/tmp421.o
  CC      hw/misc/debugexit.o
  CC      hw/misc/sga.o
  CC      hw/misc/pc-testdev.o
  CC      hw/misc/pci-testdev.o
  CC      hw/misc/edu.o
  CC      hw/misc/arm_l2x0.o
  CC      hw/misc/unimp.o
  CC      hw/misc/arm_integrator_debug.o
  CC      hw/misc/a9scu.o
  CC      hw/misc/arm11scu.o
  CC      hw/misc/puv3_pm.o
  CC      hw/misc/macio/macio.o
  CC      hw/misc/macio/cuda.o
  CC      hw/misc/macio/mac_dbdma.o
  CC      hw/net/dp8393x.o
  CC      hw/net/ne2000.o
  CC      hw/net/eepro100.o
  CC      hw/net/pcnet-pci.o
  CC      hw/net/pcnet.o
  CC      hw/net/e1000.o
  CC      hw/net/e1000x_common.o
  CC      hw/net/net_tx_pkt.o
  CC      hw/net/net_rx_pkt.o
  CC      hw/net/e1000e.o
  CC      hw/net/e1000e_core.o
  CC      hw/net/rtl8139.o
  CC      hw/net/vmxnet3.o
  CC      hw/net/smc91c111.o
  CC      hw/net/lan9118.o
  CC      hw/net/ne2000-isa.o
  CC      hw/net/opencores_eth.o
  CC      hw/net/xgmac.o
  CC      hw/net/mipsnet.o
  CC      hw/net/xilinx_axienet.o
  CC      hw/net/allwinner_emac.o
  CC      hw/net/imx_fec.o
  CC      hw/net/cadence_gem.o
  CC      hw/net/stellaris_enet.o
  CC      hw/net/lance.o
  CC      hw/net/ftgmac100.o
  CC      hw/net/rocker/rocker.o
  CC      hw/net/rocker/rocker_fp.o
  CC      hw/net/rocker/rocker_desc.o
  CC      hw/net/rocker/rocker_world.o
  CC      hw/net/rocker/rocker_of_dpa.o
  CC      hw/nvram/ds1225y.o
  CC      hw/nvram/eeprom93xx.o
  CC      hw/nvram/fw_cfg.o
  CC      hw/nvram/chrp_nvram.o
  CC      hw/nvram/mac_nvram.o
  CC      hw/pci-bridge/pci_bridge_dev.o
  CC      hw/pci-bridge/pcie_root_port.o
  CC      hw/pci-bridge/gen_pcie_root_port.o
  CC      hw/pci-bridge/pci_expander_bridge.o
  CC      hw/pci-bridge/xio3130_upstream.o
  CC      hw/pci-bridge/xio3130_downstream.o
  CC      hw/pci-bridge/ioh3420.o
  CC      hw/pci-bridge/i82801b11.o
  CC      hw/pci-bridge/dec.o
  CC      hw/pci-host/pam.o
  CC      hw/pci-host/prep.o
  CC      hw/pci-host/grackle.o
  CC      hw/pci-host/uninorth.o
  CC      hw/pci-host/ppce500.o
  CC      hw/pci-host/apb.o
  CC      hw/pci-host/versatile.o
  CC      hw/pci-host/bonito.o
  CC      hw/pci-host/piix.o
  CC      hw/pci-host/q35.o
  CC      hw/pci-host/gpex.o
  CC      hw/pci-host/xilinx-pcie.o
  CC      hw/pci/pci.o
  CC      hw/pci/pci_bridge.o
  CC      hw/pci/msix.o
  CC      hw/pci/msi.o
  CC      hw/pci/shpc.o
  CC      hw/pci/slotid_cap.o
  CC      hw/pci/pci_host.o
  CC      hw/pci/pcie.o
  CC      hw/pci/pcie_host.o
  CC      hw/pci/pcie_aer.o
  CC      hw/pci/pcie_port.o
  CC      hw/pci/pci-stub.o
  CC      hw/pcmcia/pcmcia.o
  CC      hw/scsi/scsi-disk.o
  CC      hw/scsi/scsi-generic.o
  CC      hw/scsi/scsi-bus.o
  CC      hw/scsi/lsi53c895a.o
  CC      hw/scsi/mptsas.o
  CC      hw/scsi/mptconfig.o
  CC      hw/scsi/mptendian.o
  CC      hw/scsi/vmw_pvscsi.o
  CC      hw/scsi/megasas.o
  CC      hw/scsi/esp.o
  CC      hw/scsi/esp-pci.o
  CC      hw/sd/ssi-sd.o
  CC      hw/sd/pl181.o
  CC      hw/sd/sd.o
  CC      hw/sd/core.o
  CC      hw/sd/sdhci.o
  CC      hw/smbios/smbios.o
  CC      hw/smbios/smbios_type_38.o
  CC      hw/smbios/smbios-stub.o
  CC      hw/smbios/smbios_type_38-stub.o
  CC      hw/ssi/pl022.o
  CC      hw/ssi/ssi.o
  CC      hw/ssi/xilinx_spips.o
  CC      hw/ssi/xilinx_spi.o
  CC      hw/ssi/aspeed_smc.o
  CC      hw/ssi/stm32f2xx_spi.o
  CC      hw/timer/arm_timer.o
  CC      hw/timer/arm_mptimer.o
  CC      hw/timer/armv7m_systick.o
  CC      hw/timer/a9gtimer.o
  CC      hw/timer/cadence_ttc.o
  CC      hw/timer/ds1338.o
  CC      hw/timer/hpet.o
  CC      hw/timer/i8254.o
  CC      hw/timer/m48t59.o
  CC      hw/timer/i8254_common.o
  CC      hw/timer/m48t59-isa.o
  CC      hw/timer/pl031.o
  CC      hw/timer/puv3_ost.o
  CC      hw/timer/twl92230.o
  CC      hw/timer/xilinx_timer.o
  CC      hw/timer/etraxfs_timer.o
  CC      hw/timer/slavio_timer.o
  CC      hw/timer/grlib_gptimer.o
  CC      hw/timer/imx_epit.o
  CC      hw/timer/imx_gpt.o
  CC      hw/timer/lm32_timer.o
  CC      hw/timer/milkymist-sysctl.o
  CC      hw/timer/stm32f2xx_timer.o
  CC      hw/timer/aspeed_timer.o
  CC      hw/timer/cmsdk-apb-timer.o
  CC      hw/timer/sun4v-rtc.o
  CC      hw/tpm/tpm_tis.o
  CC      hw/usb/core.o
  CC      hw/usb/combined-packet.o
  CC      hw/usb/bus.o
  CC      hw/usb/libhw.o
  CC      hw/usb/desc.o
  CC      hw/usb/desc-msos.o
  CC      hw/usb/hcd-uhci.o
  CC      hw/usb/hcd-ohci.o
  CC      hw/usb/hcd-ehci.o
  CC      hw/usb/hcd-ehci-pci.o
  CC      hw/usb/hcd-ehci-sysbus.o
  CC      hw/usb/hcd-xhci.o
  CC      hw/usb/hcd-xhci-nec.o
  CC      hw/usb/hcd-musb.o
  CC      hw/usb/dev-hub.o
  CC      hw/usb/dev-hid.o
  CC      hw/usb/dev-wacom.o
  CC      hw/usb/dev-storage.o
  CC      hw/usb/dev-uas.o
  CC      hw/usb/dev-audio.o
  CC      hw/usb/dev-serial.o
  CC      hw/usb/dev-network.o
  CC      hw/usb/dev-bluetooth.o
  CC      hw/usb/dev-smartcard-reader.o
  CC      hw/usb/ccid-card-passthru.o
  CC      hw/usb/ccid-card-emulated.o
  CC      hw/usb/dev-mtp.o
  CC      hw/usb/redirect.o
  CC      hw/usb/quirks.o
  CC      hw/usb/host-libusb.o
  CC      hw/usb/host-legacy.o
  CC      hw/virtio/virtio-rng.o
  CC      hw/virtio/virtio-pci.o
  CC      hw/virtio/virtio-bus.o
  CC      hw/virtio/virtio-mmio.o
  CC      hw/virtio/vhost-stub.o
  CC      hw/watchdog/watchdog.o
  CC      hw/watchdog/wdt_i6300esb.o
  CC      hw/watchdog/wdt_ib700.o
  CC      hw/watchdog/wdt_diag288.o
  CC      hw/watchdog/wdt_aspeed.o
  CC      migration/migration.o
  CC      migration/fd.o
  CC      migration/socket.o
  CC      migration/exec.o
  CC      migration/tls.o
  CC      migration/channel.o
  CC      migration/savevm.o
  CC      migration/colo-comm.o
  CC      migration/colo.o
  CC      migration/colo-failover.o
  CC      migration/vmstate-types.o
  CC      migration/vmstate.o
  CC      migration/page_cache.o
  CC      migration/qemu-file.o
  CC      migration/global_state.o
  CC      migration/qemu-file-channel.o
  CC      migration/xbzrle.o
  CC      migration/postcopy-ram.o
  CC      migration/qjson.o
  CC      migration/block.o
  CC      net/net.o
  CC      net/queue.o
  CC      net/checksum.o
  CC      net/util.o
  CC      net/hub.o
  CC      net/socket.o
  CC      net/dump.o
  CC      net/eth.o
  CC      net/l2tpv3.o
  CC      net/vhost-user.o
  CC      net/slirp.o
  CC      net/filter.o
  CC      net/filter-buffer.o
  CC      net/colo-compare.o
  CC      net/filter-mirror.o
  CC      net/colo.o
  CC      net/filter-rewriter.o
  CC      net/filter-replay.o
  CC      net/tap.o
  CC      net/tap-linux.o
  CC      qom/cpu.o
  CC      replay/replay.o
  CC      replay/replay-internal.o
  CC      replay/replay-events.o
  CC      replay/replay-time.o
  CC      replay/replay-input.o
  CC      replay/replay-snapshot.o
  CC      replay/replay-char.o
  CC      replay/replay-net.o
  CC      replay/replay-audio.o
  CC      slirp/cksum.o
  CC      slirp/ip_icmp.o
  CC      slirp/if.o
  CC      slirp/ip6_icmp.o
  CC      slirp/ip6_input.o
  CC      slirp/ip_input.o
  CC      slirp/ip6_output.o
  CC      slirp/ip_output.o
  CC      slirp/dnssearch.o
  CC      slirp/dhcpv6.o
  CC      slirp/slirp.o
  CC      slirp/mbuf.o
  CC      slirp/sbuf.o
  CC      slirp/misc.o
  CC      slirp/socket.o
  CC      slirp/tcp_input.o
  CC      slirp/tcp_output.o
  CC      slirp/tcp_subr.o
  CC      slirp/tcp_timer.o
  CC      slirp/udp.o
  CC      slirp/udp6.o
  CC      slirp/bootp.o
  CC      slirp/tftp.o
  CC      slirp/arp_table.o
  CC      slirp/ndp_table.o
  CC      slirp/ncsi.o
  CC      ui/keymaps.o
  CC      ui/console.o
  CC      ui/cursor.o
  CC      ui/qemu-pixman.o
  CC      ui/input.o
  CC      ui/input-keymap.o
  CC      ui/input-legacy.o
  CC      ui/input-linux.o
  CC      ui/sdl2.o
  CC      ui/sdl2-input.o
  CC      ui/sdl2-2d.o
  CC      ui/sdl2-gl.o
  CC      ui/x_keymap.o
  CC      ui/curses.o
  CC      ui/vnc-enc-zlib.o
  CC      ui/vnc.o
  CC      ui/vnc-enc-hextile.o
  CC      ui/vnc-palette.o
  CC      ui/vnc-enc-tight.o
  CC      ui/vnc-enc-zrle.o
  CC      ui/vnc-auth-vencrypt.o
  CC      ui/vnc-auth-sasl.o
  CC      ui/vnc-ws.o
  CC      ui/vnc-jobs.o
  CC      ui/gtk.o
  CC      ui/shader.o
  VERT    ui/shader/texture-blit-vert.h
  FRAG    ui/shader/texture-blit-frag.h
  CC      ui/egl-helpers.o
  CC      ui/egl-context.o
  CC      ui/egl-headless.o
  CC      ui/gtk-gl-area.o
  CC      chardev/char.o
  CC      chardev/char-fd.o
  CC      chardev/char-fe.o
  CC      chardev/char-file.o
  CC      chardev/char-io.o
  CC      chardev/char-mux.o
  CC      chardev/char-null.o
  CC      chardev/char-parallel.o
  CC      chardev/char-pipe.o
  CC      chardev/char-pty.o
  CC      chardev/char-ringbuf.o
  CC      chardev/char-serial.o
  CC      chardev/char-socket.o
  CC      chardev/char-stdio.o
  CC      chardev/char-udp.o
  LINK    tests/qemu-iotests/socket_scm_helper
  CCAS    s390-ccw/start.o
  GEN     qemu-doc.html
  CC      s390-ccw/main.o
  CC      s390-ccw/bootmap.o
  CC      s390-ccw/sclp.o
  CC      s390-ccw/virtio.o
  CC      s390-ccw/virtio-scsi.o
  GEN     qemu-doc.txt
  CC      s390-ccw/virtio-blkdev.o
s390-netboot.img not built since roms/SLOF/ is not available.
  GEN     qemu.1
  BUILD   s390-ccw/s390-ccw.elf
  STRIP   s390-ccw/s390-ccw.img
  GEN     docs/interop/qemu-qmp-ref.html
  GEN     docs/interop/qemu-qmp-ref.txt
  GEN     docs/interop/qemu-qmp-ref.7
  GEN     docs/interop/qemu-ga-ref.html
  GEN     docs/interop/qemu-ga-ref.txt
  GEN     docs/interop/qemu-ga-ref.7
  CC      qga/commands.o
  CC      qga/guest-agent-command-state.o
  CC      qga/main.o
  CC      qga/channel-posix.o
  CC      qga/commands-posix.o
  CC      qga/qapi-generated/qga-qapi-types.o
  CC      qga/qapi-generated/qga-qapi-visit.o
  AR      libqemuutil.a
  CC      qga/qapi-generated/qga-qmp-marshal.o
  AR      libqemustub.a
  CC      qemu-img.o
  LINK    qemu-io
  LINK    fsdev/virtfs-proxy-helper
  LINK    qemu-bridge-helper
  CC      ui/console-gl.o
  LINK    qemu-ga
  LINK    ivshmem-client
  LINK    ivshmem-server
  LINK    qemu-nbd
  GEN     alpha-softmmu/hmp-commands.h
  GEN     alpha-softmmu/hmp-commands-info.h
  GEN     alpha-softmmu/config-target.h
  GEN     cris-softmmu/hmp-commands.h
  GEN     cris-softmmu/hmp-commands-info.h
  GEN     cris-softmmu/config-target.h
  CC      alpha-softmmu/exec.o
  CC      cris-softmmu/exec.o
  GEN     aarch64-softmmu/hmp-commands.h
  GEN     arm-softmmu/hmp-commands.h
  GEN     aarch64-softmmu/hmp-commands-info.h
  GEN     arm-softmmu/hmp-commands-info.h
  GEN     arm-softmmu/config-target.h
  GEN     aarch64-softmmu/config-target.h
  CC      arm-softmmu/exec.o
  CC      aarch64-softmmu/exec.o
  CC      arm-softmmu/tcg/tcg.o
  CC      alpha-softmmu/tcg/tcg.o
  CC      cris-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/tcg/tcg.o
  CC      cris-softmmu/tcg/tcg-op.o
  CC      arm-softmmu/tcg/tcg-op.o
  CC      alpha-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/tcg/tcg-op.o
  CC      cris-softmmu/tcg/optimize.o
  CC      arm-softmmu/tcg/optimize.o
  CC      cris-softmmu/tcg/tcg-common.o
  CC      alpha-softmmu/tcg/optimize.o
  CC      cris-softmmu/tcg/tcg-runtime.o
  CC      arm-softmmu/tcg/tcg-common.o
  CC      alpha-softmmu/tcg/tcg-common.o
  CC      cris-softmmu/fpu/softfloat.o
  CC      arm-softmmu/tcg/tcg-runtime.o
  CC      alpha-softmmu/tcg/tcg-runtime.o
  CC      aarch64-softmmu/tcg/optimize.o
  CC      arm-softmmu/fpu/softfloat.o
  CC      alpha-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/tcg/tcg-runtime.o
  CC      cris-softmmu/disas.o
  CC      aarch64-softmmu/fpu/softfloat.o
  CC      cris-softmmu/hax-stub.o
  CC      arm-softmmu/disas.o
  CC      cris-softmmu/arch_init.o
  CC      cris-softmmu/cpus.o
  CC      alpha-softmmu/disas.o
  GEN     arm-softmmu/gdbstub-xml.c
  CC      alpha-softmmu/hax-stub.o
  CC      cris-softmmu/monitor.o
  CC      alpha-softmmu/arch_init.o
  CC      arm-softmmu/hax-stub.o
  CC      arm-softmmu/arch_init.o
  CC      alpha-softmmu/cpus.o
  CC      arm-softmmu/cpus.o
  CC      alpha-softmmu/monitor.o
  CC      aarch64-softmmu/disas.o
  CC      cris-softmmu/gdbstub.o
  GEN     aarch64-softmmu/gdbstub-xml.c
  CC      arm-softmmu/monitor.o
  CC      aarch64-softmmu/hax-stub.o
  CC      cris-softmmu/balloon.o
  CC      aarch64-softmmu/arch_init.o
  CC      cris-softmmu/ioport.o
  CC      alpha-softmmu/gdbstub.o
  CC      aarch64-softmmu/cpus.o
  CC      cris-softmmu/numa.o
  CC      arm-softmmu/gdbstub.o
  CC      aarch64-softmmu/monitor.o
  CC      arm-softmmu/balloon.o
  CC      cris-softmmu/qtest.o
  CC      alpha-softmmu/balloon.o
  CC      arm-softmmu/ioport.o
  CC      cris-softmmu/memory.o
  CC      alpha-softmmu/ioport.o
  CC      arm-softmmu/numa.o
  CC      alpha-softmmu/numa.o
  CC      arm-softmmu/qtest.o
  CC      alpha-softmmu/qtest.o
  CC      aarch64-softmmu/gdbstub.o
  CC      cris-softmmu/memory_mapping.o
  CC      arm-softmmu/memory.o
  CC      alpha-softmmu/memory.o
  CC      aarch64-softmmu/balloon.o
  CC      cris-softmmu/dump.o
  CC      aarch64-softmmu/ioport.o
  CC      cris-softmmu/migration/ram.o
  CC      aarch64-softmmu/numa.o
  CC      arm-softmmu/memory_mapping.o
  CC      alpha-softmmu/memory_mapping.o
  CC      alpha-softmmu/dump.o
  CC      cris-softmmu/accel/accel.o
  CC      aarch64-softmmu/qtest.o
  CC      arm-softmmu/dump.o
  CC      cris-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/memory.o
  CC      alpha-softmmu/migration/ram.o
  CC      cris-softmmu/accel/tcg/tcg-all.o
  CC      arm-softmmu/migration/ram.o
  CC      cris-softmmu/accel/tcg/cputlb.o
  CC      alpha-softmmu/accel/accel.o
  CC      aarch64-softmmu/memory_mapping.o
  CC      arm-softmmu/accel/accel.o
  CC      alpha-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/dump.o
  CC      alpha-softmmu/accel/tcg/tcg-all.o
  CC      arm-softmmu/accel/stubs/kvm-stub.o
  CC      alpha-softmmu/accel/tcg/cputlb.o
  CC      cris-softmmu/accel/tcg/cpu-exec.o
  CC      arm-softmmu/accel/tcg/tcg-all.o
  CC      aarch64-softmmu/migration/ram.o
  CC      arm-softmmu/accel/tcg/cputlb.o
  CC      cris-softmmu/accel/tcg/cpu-exec-common.o
  CC      cris-softmmu/accel/tcg/translate-all.o
  CC      aarch64-softmmu/accel/accel.o
  CC      alpha-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/accel/stubs/kvm-stub.o
  CC      cris-softmmu/hw/core/generic-loader.o
  CC      alpha-softmmu/accel/tcg/cpu-exec-common.o
  CC      aarch64-softmmu/accel/tcg/tcg-all.o
  CC      cris-softmmu/hw/core/null-machine.o
  CC      alpha-softmmu/accel/tcg/translate-all.o
  CC      arm-softmmu/accel/tcg/cpu-exec.o
  CC      cris-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/accel/tcg/cputlb.o
  CC      cris-softmmu/hw/net/etraxfs_eth.o
  CC      alpha-softmmu/hw/9pfs/virtio-9p-device.o
  CC      cris-softmmu/hw/net/vhost_net.o
  CC      arm-softmmu/accel/tcg/cpu-exec-common.o
  CC      alpha-softmmu/hw/block/virtio-blk.o
  CC      cris-softmmu/hw/net/rocker/qmp-norocker.o
  CC      arm-softmmu/accel/tcg/translate-all.o
  CC      cris-softmmu/hw/vfio/common.o
  CC      alpha-softmmu/hw/block/dataplane/virtio-blk.o
  CC      arm-softmmu/hw/9pfs/virtio-9p-device.o
  CC      alpha-softmmu/hw/char/virtio-serial-bus.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec.o
  CC      cris-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/adc/stm32f2xx_adc.o
  CC      cris-softmmu/hw/vfio/spapr.o
  CC      alpha-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/block/virtio-blk.o
  CC      alpha-softmmu/hw/core/null-machine.o
  CC      cris-softmmu/hw/cris/boot.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec-common.o
  CC      cris-softmmu/hw/cris/axis_dev88.o
  CC      arm-softmmu/hw/block/dataplane/virtio-blk.o
  CC      alpha-softmmu/hw/display/vga.o
  CC      aarch64-softmmu/accel/tcg/translate-all.o
  CC      arm-softmmu/hw/char/exynos4210_uart.o
  CC      cris-softmmu/target/cris/translate.o
  CC      arm-softmmu/hw/char/omap_uart.o
  CC      aarch64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      arm-softmmu/hw/char/digic-uart.o
  CC      alpha-softmmu/hw/display/virtio-gpu.o
  CC      arm-softmmu/hw/char/stm32f2xx_usart.o
  CC      aarch64-softmmu/hw/adc/stm32f2xx_adc.o
  CC      arm-softmmu/hw/char/bcm2835_aux.o
  CC      aarch64-softmmu/hw/block/virtio-blk.o
  CC      arm-softmmu/hw/char/virtio-serial-bus.o
  CC      alpha-softmmu/hw/display/virtio-gpu-3d.o
  CC      cris-softmmu/target/cris/op_helper.o
  CC      arm-softmmu/hw/core/generic-loader.o
  CC      aarch64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      cris-softmmu/target/cris/helper.o
  CC      arm-softmmu/hw/core/null-machine.o
  CC      alpha-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-softmmu/hw/char/exynos4210_uart.o
  CC      cris-softmmu/target/cris/cpu.o
  CC      arm-softmmu/hw/cpu/arm11mpcore.o
  CC      aarch64-softmmu/hw/char/omap_uart.o
  CC      cris-softmmu/target/cris/gdbstub.o
  CC      alpha-softmmu/hw/misc/ivshmem.o
  CC      aarch64-softmmu/hw/char/digic-uart.o
  CC      arm-softmmu/hw/cpu/realview_mpcore.o
  CC      cris-softmmu/target/cris/mmu.o
  CC      alpha-softmmu/hw/misc/mmio_interface.o
  CC      arm-softmmu/hw/cpu/a9mpcore.o
  CC      aarch64-softmmu/hw/char/stm32f2xx_usart.o
  CC      alpha-softmmu/hw/net/virtio-net.o
  CC      cris-softmmu/target/cris/machine.o
  CC      arm-softmmu/hw/cpu/a15mpcore.o
  CC      aarch64-softmmu/hw/char/bcm2835_aux.o
  GEN     trace/generated-helpers.c
  CC      cris-softmmu/trace/control-target.o
  CC      arm-softmmu/hw/display/omap_dss.o
  CC      aarch64-softmmu/hw/char/virtio-serial-bus.o
  CC      alpha-softmmu/hw/net/vhost_net.o
  CC      cris-softmmu/trace/generated-helpers.o
  CC      arm-softmmu/hw/display/omap_lcdc.o
  CC      alpha-softmmu/hw/scsi/virtio-scsi.o
  LINK    cris-softmmu/qemu-system-cris
  CC      aarch64-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/display/pxa2xx_lcd.o
  CC      alpha-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/hw/cpu/arm11mpcore.o
  CC      alpha-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/cpu/realview_mpcore.o
  CC      alpha-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-softmmu/hw/cpu/a9mpcore.o
  CC      arm-softmmu/hw/display/bcm2835_fb.o
  CC      alpha-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/cpu/a15mpcore.o
  CC      alpha-softmmu/hw/timer/mc146818rtc.o
  GEN     i386-softmmu/hmp-commands.h
  CC      aarch64-softmmu/hw/display/omap_dss.o
  GEN     i386-softmmu/hmp-commands-info.h
  GEN     i386-softmmu/config-target.h
  CC      arm-softmmu/hw/display/vga.o
  CC      i386-softmmu/exec.o
  CC      alpha-softmmu/hw/vfio/common.o
  CC      aarch64-softmmu/hw/display/omap_lcdc.o
  CC      arm-softmmu/hw/display/virtio-gpu.o
  CC      aarch64-softmmu/hw/display/pxa2xx_lcd.o
  CC      alpha-softmmu/hw/vfio/pci.o
  CC      i386-softmmu/tcg/tcg.o
  CC      arm-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/display/bcm2835_fb.o
  CC      alpha-softmmu/hw/vfio/pci-quirks.o
  CC      aarch64-softmmu/hw/display/vga.o
  CC      arm-softmmu/hw/display/virtio-gpu-pci.o
  CC      alpha-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/dma/omap_dma.o
  CC      alpha-softmmu/hw/vfio/spapr.o
  CC      i386-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/hw/display/virtio-gpu.o
  CC      alpha-softmmu/hw/virtio/virtio.o
  CC      arm-softmmu/hw/dma/soc_dma.o
  CC      arm-softmmu/hw/dma/pxa2xx_dma.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-3d.o
  CC      alpha-softmmu/hw/virtio/virtio-balloon.o
  CC      arm-softmmu/hw/dma/bcm2835_dma.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-pci.o
  CC      alpha-softmmu/hw/virtio/vhost.o
  CC      arm-softmmu/hw/gpio/omap_gpio.o
  CC      arm-softmmu/hw/gpio/imx_gpio.o
  CC      aarch64-softmmu/hw/display/dpcd.o
  CC      i386-softmmu/tcg/optimize.o
  CC      arm-softmmu/hw/gpio/bcm2835_gpio.o
  CC      aarch64-softmmu/hw/display/xlnx_dp.o
  CC      alpha-softmmu/hw/virtio/vhost-backend.o
  CC      arm-softmmu/hw/i2c/omap_i2c.o
  CC      alpha-softmmu/hw/virtio/vhost-user.o
  CC      aarch64-softmmu/hw/dma/xlnx_dpdma.o
  CC      arm-softmmu/hw/input/pxa2xx_keypad.o
  CC      alpha-softmmu/hw/virtio/vhost-vsock.o
  CC      i386-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/hw/dma/omap_dma.o
  CC      arm-softmmu/hw/input/tsc210x.o
  CC      alpha-softmmu/hw/virtio/virtio-crypto.o
  CC      i386-softmmu/tcg/tcg-runtime.o
  CC      alpha-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      aarch64-softmmu/hw/dma/soc_dma.o
  CC      i386-softmmu/fpu/softfloat.o
  CC      arm-softmmu/hw/intc/armv7m_nvic.o
  CC      aarch64-softmmu/hw/dma/pxa2xx_dma.o
  CC      alpha-softmmu/hw/alpha/dp264.o
  CC      arm-softmmu/hw/intc/exynos4210_gic.o
  CC      aarch64-softmmu/hw/dma/bcm2835_dma.o
  CC      alpha-softmmu/hw/alpha/pci.o
  CC      aarch64-softmmu/hw/gpio/omap_gpio.o
  CC      arm-softmmu/hw/intc/exynos4210_combiner.o
  CC      alpha-softmmu/hw/alpha/typhoon.o
  CC      aarch64-softmmu/hw/gpio/imx_gpio.o
  CC      arm-softmmu/hw/intc/omap_intc.o
  CC      aarch64-softmmu/hw/gpio/bcm2835_gpio.o
  CC      aarch64-softmmu/hw/i2c/omap_i2c.o
  CC      alpha-softmmu/target/alpha/machine.o
  CC      arm-softmmu/hw/intc/bcm2835_ic.o
  CC      aarch64-softmmu/hw/input/pxa2xx_keypad.o
  CC      arm-softmmu/hw/intc/bcm2836_control.o
  CC      alpha-softmmu/target/alpha/translate.o
  CC      aarch64-softmmu/hw/input/tsc210x.o
  CC      arm-softmmu/hw/intc/allwinner-a10-pic.o
  CC      i386-softmmu/disas.o
  CC      arm-softmmu/hw/intc/aspeed_vic.o
  GEN     i386-softmmu/gdbstub-xml.c
  CC      aarch64-softmmu/hw/intc/armv7m_nvic.o
  CC      arm-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      alpha-softmmu/target/alpha/helper.o
  CC      i386-softmmu/hax-stub.o
  CC      aarch64-softmmu/hw/intc/exynos4210_gic.o
  CC      alpha-softmmu/target/alpha/cpu.o
  CC      i386-softmmu/arch_init.o
  CC      aarch64-softmmu/hw/intc/exynos4210_combiner.o
  CC      alpha-softmmu/target/alpha/int_helper.o
  CC      i386-softmmu/cpus.o
  CC      arm-softmmu/hw/misc/ivshmem.o
  CC      aarch64-softmmu/hw/intc/omap_intc.o
  CC      alpha-softmmu/target/alpha/fpu_helper.o
  CC      i386-softmmu/monitor.o
  CC      alpha-softmmu/target/alpha/vax_helper.o
  CC      arm-softmmu/hw/misc/arm_sysctl.o
  CC      aarch64-softmmu/hw/intc/bcm2835_ic.o
  CC      alpha-softmmu/target/alpha/sys_helper.o
  CC      aarch64-softmmu/hw/intc/bcm2836_control.o
  CC      arm-softmmu/hw/misc/cbus.o
  CC      alpha-softmmu/target/alpha/mem_helper.o
  CC      aarch64-softmmu/hw/intc/allwinner-a10-pic.o
  CC      arm-softmmu/hw/misc/exynos4210_pmu.o
  CC      alpha-softmmu/target/alpha/gdbstub.o
  CC      arm-softmmu/hw/misc/exynos4210_clk.o
  GEN     trace/generated-helpers.c
  CC      alpha-softmmu/trace/control-target.o
  CC      aarch64-softmmu/hw/intc/aspeed_vic.o
  CC      arm-softmmu/hw/misc/exynos4210_rng.o
  CC      i386-softmmu/gdbstub.o
  CC      aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      arm-softmmu/hw/misc/imx_ccm.o
  CC      alpha-softmmu/trace/generated-helpers.o
  CC      arm-softmmu/hw/misc/imx31_ccm.o
  LINK    alpha-softmmu/qemu-system-alpha
  CC      i386-softmmu/balloon.o
  CC      arm-softmmu/hw/misc/imx25_ccm.o
  CC      i386-softmmu/ioport.o
  CC      arm-softmmu/hw/misc/imx6_ccm.o
  CC      aarch64-softmmu/hw/misc/ivshmem.o
  CC      i386-softmmu/numa.o
  CC      arm-softmmu/hw/misc/imx6_src.o
  CC      aarch64-softmmu/hw/misc/arm_sysctl.o
  CC      arm-softmmu/hw/misc/mst_fpga.o
  CC      aarch64-softmmu/hw/misc/cbus.o
  CC      i386-softmmu/qtest.o
  CC      arm-softmmu/hw/misc/omap_clk.o
  CC      aarch64-softmmu/hw/misc/exynos4210_pmu.o
  CC      aarch64-softmmu/hw/misc/exynos4210_clk.o
  CC      i386-softmmu/memory.o
  CC      arm-softmmu/hw/misc/omap_gpmc.o
  GEN     lm32-softmmu/hmp-commands.h
  GEN     lm32-softmmu/hmp-commands-info.h
  GEN     lm32-softmmu/config-target.h
  CC      lm32-softmmu/exec.o
  CC      aarch64-softmmu/hw/misc/exynos4210_rng.o
  CC      arm-softmmu/hw/misc/omap_l4.o
  CC      arm-softmmu/hw/misc/omap_sdrc.o
  CC      aarch64-softmmu/hw/misc/imx_ccm.o
  CC      arm-softmmu/hw/misc/omap_tap.o
  CC      aarch64-softmmu/hw/misc/imx31_ccm.o
  CC      arm-softmmu/hw/misc/bcm2835_mbox.o
  CC      i386-softmmu/memory_mapping.o
  CC      aarch64-softmmu/hw/misc/imx25_ccm.o
  CC      arm-softmmu/hw/misc/bcm2835_property.o
  CC      i386-softmmu/dump.o
  CC      aarch64-softmmu/hw/misc/imx6_ccm.o
  CC      arm-softmmu/hw/misc/bcm2835_rng.o
  CC      lm32-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/hw/misc/imx6_src.o
  CC      arm-softmmu/hw/misc/zynq_slcr.o
  CC      i386-softmmu/migration/ram.o
  CC      aarch64-softmmu/hw/misc/mst_fpga.o
  CC      arm-softmmu/hw/misc/zynq-xadc.o
  CC      aarch64-softmmu/hw/misc/omap_clk.o
  CC      arm-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      i386-softmmu/accel/accel.o
  CC      aarch64-softmmu/hw/misc/omap_gpmc.o
  CC      i386-softmmu/accel/stubs/kvm-stub.o
  CC      arm-softmmu/hw/misc/mps2-scc.o
  CC      lm32-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/hw/misc/omap_l4.o
  CC      arm-softmmu/hw/misc/aspeed_scu.o
  CC      i386-softmmu/accel/tcg/tcg-all.o
  CC      aarch64-softmmu/hw/misc/omap_sdrc.o
  CC      i386-softmmu/accel/tcg/cputlb.o
  CC      arm-softmmu/hw/misc/aspeed_sdmc.o
  CC      aarch64-softmmu/hw/misc/omap_tap.o
  CC      arm-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/misc/bcm2835_mbox.o
  CC      aarch64-softmmu/hw/misc/bcm2835_property.o
  CC      arm-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/net/vhost_net.o
  CC      lm32-softmmu/tcg/optimize.o
  CC      aarch64-softmmu/hw/misc/bcm2835_rng.o
  CC      arm-softmmu/hw/pcmcia/pxa2xx.o
  CC      aarch64-softmmu/hw/misc/zynq_slcr.o
  CC      i386-softmmu/accel/tcg/cpu-exec.o
  CC      arm-softmmu/hw/scsi/virtio-scsi.o
  CC      aarch64-softmmu/hw/misc/zynq-xadc.o
  CC      i386-softmmu/accel/tcg/cpu-exec-common.o
  CC      aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      lm32-softmmu/tcg/tcg-common.o
  CC      arm-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/misc/mps2-scc.o
  CC      i386-softmmu/accel/tcg/translate-all.o
  CC      lm32-softmmu/tcg/tcg-runtime.o
  CC      aarch64-softmmu/hw/misc/auxbus.o
  CC      arm-softmmu/hw/scsi/vhost-scsi-common.o
  CC      lm32-softmmu/fpu/softfloat.o
  CC      i386-softmmu/hw/9pfs/virtio-9p-device.o
  CC      aarch64-softmmu/hw/misc/aspeed_scu.o
  CC      arm-softmmu/hw/scsi/vhost-scsi.o
  CC      i386-softmmu/hw/block/virtio-blk.o
  CC      aarch64-softmmu/hw/misc/aspeed_sdmc.o
  CC      arm-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/misc/mmio_interface.o
  CC      i386-softmmu/hw/block/dataplane/virtio-blk.o
  CC      aarch64-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/sd/omap_mmc.o
  CC      i386-softmmu/hw/char/virtio-serial-bus.o
  CC      arm-softmmu/hw/sd/pxa2xx_mmci.o
  CC      i386-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/sd/bcm2835_sdhost.o
  CC      aarch64-softmmu/hw/net/vhost_net.o
  CC      i386-softmmu/hw/core/null-machine.o
  CC      arm-softmmu/hw/ssi/omap_spi.o
  CC      aarch64-softmmu/hw/pcmcia/pxa2xx.o
  CC      i386-softmmu/hw/display/vga.o
  CC      lm32-softmmu/disas.o
  CC      arm-softmmu/hw/ssi/imx_spi.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi.o
  CC      lm32-softmmu/hax-stub.o
  CC      arm-softmmu/hw/timer/exynos4210_mct.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      lm32-softmmu/arch_init.o
  CC      arm-softmmu/hw/timer/exynos4210_pwm.o
  CC      lm32-softmmu/cpus.o
  CC      i386-softmmu/hw/display/virtio-gpu.o
  CC      arm-softmmu/hw/timer/exynos4210_rtc.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi.o
  CC      arm-softmmu/hw/timer/omap_gptimer.o
  CC      lm32-softmmu/monitor.o
  CC      i386-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      arm-softmmu/hw/timer/omap_synctimer.o
  CC      aarch64-softmmu/hw/sd/omap_mmc.o
  CC      arm-softmmu/hw/timer/pxa2xx_timer.o
  CC      aarch64-softmmu/hw/sd/pxa2xx_mmci.o
  CC      i386-softmmu/hw/display/virtio-gpu-pci.o
  CC      arm-softmmu/hw/timer/digic-timer.o
  CC      aarch64-softmmu/hw/sd/bcm2835_sdhost.o
  CC      lm32-softmmu/gdbstub.o
  CC      i386-softmmu/hw/display/virtio-vga.o
  CC      arm-softmmu/hw/timer/allwinner-a10-pit.o
  CC      aarch64-softmmu/hw/ssi/omap_spi.o
  CC      arm-softmmu/hw/usb/tusb6010.o
  CC      aarch64-softmmu/hw/ssi/imx_spi.o
  CC      i386-softmmu/hw/intc/apic.o
  CC      lm32-softmmu/balloon.o
  CC      lm32-softmmu/ioport.o
  CC      aarch64-softmmu/hw/timer/exynos4210_mct.o
  CC      arm-softmmu/hw/vfio/common.o
  CC      lm32-softmmu/numa.o
  CC      i386-softmmu/hw/intc/apic_common.o
  CC      aarch64-softmmu/hw/timer/exynos4210_pwm.o
  CC      arm-softmmu/hw/vfio/pci.o
  CC      lm32-softmmu/qtest.o
  CC      i386-softmmu/hw/intc/ioapic.o
  CC      aarch64-softmmu/hw/timer/exynos4210_rtc.o
  CC      aarch64-softmmu/hw/timer/omap_gptimer.o
  CC      lm32-softmmu/memory.o
  CC      i386-softmmu/hw/isa/lpc_ich9.o
  CC      aarch64-softmmu/hw/timer/omap_synctimer.o
  CC      arm-softmmu/hw/vfio/pci-quirks.o
  CC      i386-softmmu/hw/misc/vmport.o
  CC      aarch64-softmmu/hw/timer/pxa2xx_timer.o
  CC      i386-softmmu/hw/misc/ivshmem.o
  CC      arm-softmmu/hw/vfio/platform.o
  CC      lm32-softmmu/memory_mapping.o
  CC      aarch64-softmmu/hw/timer/digic-timer.o
  CC      i386-softmmu/hw/misc/pvpanic.o
  CC      aarch64-softmmu/hw/timer/allwinner-a10-pit.o
  CC      arm-softmmu/hw/vfio/calxeda-xgmac.o
  CC      lm32-softmmu/dump.o
  CC      i386-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/usb/tusb6010.o
  CC      arm-softmmu/hw/vfio/amd-xgbe.o
  CC      i386-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/vfio/spapr.o
  CC      aarch64-softmmu/hw/vfio/common.o
  CC      lm32-softmmu/migration/ram.o
  CC      arm-softmmu/hw/virtio/virtio.o
  CC      i386-softmmu/hw/net/vhost_net.o
  CC      aarch64-softmmu/hw/vfio/pci.o
  CC      lm32-softmmu/accel/accel.o
  CC      i386-softmmu/hw/scsi/virtio-scsi.o
  CC      lm32-softmmu/accel/stubs/kvm-stub.o
  CC      arm-softmmu/hw/virtio/virtio-balloon.o
  CC      i386-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      i386-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/vfio/pci-quirks.o
  CC      lm32-softmmu/accel/tcg/tcg-all.o
  CC      arm-softmmu/hw/virtio/vhost.o
  CC      i386-softmmu/hw/scsi/vhost-scsi.o
  CC      lm32-softmmu/accel/tcg/cputlb.o
  CC      aarch64-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/virtio/vhost-backend.o
  CC      i386-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/vfio/calxeda-xgmac.o
  CC      arm-softmmu/hw/virtio/vhost-user.o
  CC      i386-softmmu/hw/timer/mc146818rtc.o
  CC      aarch64-softmmu/hw/vfio/amd-xgbe.o
  CC      arm-softmmu/hw/virtio/vhost-vsock.o
  CC      lm32-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/hw/vfio/spapr.o
  CC      i386-softmmu/hw/vfio/common.o
  CC      arm-softmmu/hw/virtio/virtio-crypto.o
  CC      aarch64-softmmu/hw/virtio/virtio.o
  CC      lm32-softmmu/accel/tcg/cpu-exec-common.o
  CC      arm-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      i386-softmmu/hw/vfio/pci.o
  CC      lm32-softmmu/accel/tcg/translate-all.o
  CC      arm-softmmu/hw/arm/boot.o
  CC      aarch64-softmmu/hw/virtio/virtio-balloon.o
  CC      lm32-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/arm/collie.o
  CC      i386-softmmu/hw/vfio/pci-quirks.o
  CC      lm32-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/hw/virtio/vhost.o
  CC      arm-softmmu/hw/arm/exynos4_boards.o
  CC      lm32-softmmu/hw/input/milkymist-softusb.o
  CC      arm-softmmu/hw/arm/gumstix.o
  CC      aarch64-softmmu/hw/virtio/vhost-backend.o
  CC      i386-softmmu/hw/vfio/platform.o
  CC      lm32-softmmu/hw/misc/milkymist-hpdmc.o
  CC      arm-softmmu/hw/arm/highbank.o
  CC      lm32-softmmu/hw/misc/milkymist-pfpu.o
  CC      aarch64-softmmu/hw/virtio/vhost-user.o
  CC      i386-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/hw/arm/digic_boards.o
  CC      lm32-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/virtio/vhost-vsock.o
  CC      i386-softmmu/hw/virtio/virtio.o
  CC      arm-softmmu/hw/arm/integratorcp.o
  CC      lm32-softmmu/hw/net/milkymist-minimac2.o
  CC      arm-softmmu/hw/arm/mainstone.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto.o
  CC      lm32-softmmu/hw/net/vhost_net.o
  CC      lm32-softmmu/hw/net/rocker/qmp-norocker.o
  CC      arm-softmmu/hw/arm/musicpal.o
  CC      i386-softmmu/hw/virtio/virtio-balloon.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      lm32-softmmu/hw/sd/milkymist-memcard.o
  CC      i386-softmmu/hw/virtio/vhost.o
  CC      lm32-softmmu/hw/vfio/common.o
  CC      aarch64-softmmu/hw/arm/boot.o
  CC      arm-softmmu/hw/arm/nseries.o
  CC      i386-softmmu/hw/virtio/vhost-backend.o
  CC      aarch64-softmmu/hw/arm/collie.o
  CC      lm32-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/arm/omap_sx1.o
  CC      i386-softmmu/hw/virtio/vhost-user.o
  CC      aarch64-softmmu/hw/arm/exynos4_boards.o
  CC      lm32-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/hw/arm/palm.o
  CC      aarch64-softmmu/hw/arm/gumstix.o
  CC      i386-softmmu/hw/virtio/vhost-vsock.o
  CC      aarch64-softmmu/hw/arm/highbank.o
  CC      lm32-softmmu/hw/lm32/lm32_boards.o
  CC      arm-softmmu/hw/arm/realview.o
  CC      i386-softmmu/hw/virtio/virtio-crypto.o
  CC      lm32-softmmu/hw/lm32/milkymist.o
  CC      aarch64-softmmu/hw/arm/digic_boards.o
  CC      arm-softmmu/hw/arm/spitz.o
  CC      aarch64-softmmu/hw/arm/integratorcp.o
  CC      i386-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      lm32-softmmu/target/lm32/translate.o
  CC      aarch64-softmmu/hw/arm/mainstone.o
  CC      arm-softmmu/hw/arm/stellaris.o
  CC      aarch64-softmmu/hw/arm/musicpal.o
  CC      i386-softmmu/hw/i386/multiboot.o
  CC      lm32-softmmu/target/lm32/op_helper.o
  CC      lm32-softmmu/target/lm32/helper.o
  CC      i386-softmmu/hw/i386/pc.o
  CC      aarch64-softmmu/hw/arm/nseries.o
  CC      arm-softmmu/hw/arm/tosa.o
  CC      lm32-softmmu/target/lm32/cpu.o
  CC      arm-softmmu/hw/arm/versatilepb.o
  CC      lm32-softmmu/target/lm32/gdbstub.o
  CC      aarch64-softmmu/hw/arm/omap_sx1.o
  CC      arm-softmmu/hw/arm/vexpress.o
  CC      lm32-softmmu/target/lm32/lm32-semi.o
  CC      i386-softmmu/hw/i386/pc_piix.o
  CC      lm32-softmmu/target/lm32/machine.o
  CC      arm-softmmu/hw/arm/virt.o
  CC      aarch64-softmmu/hw/arm/palm.o
  GEN     trace/generated-helpers.c
  CC      lm32-softmmu/trace/control-target.o
  CC      i386-softmmu/hw/i386/pc_q35.o
  CC      aarch64-softmmu/hw/arm/realview.o
  CC      lm32-softmmu/trace/generated-helpers.o
  CC      arm-softmmu/hw/arm/xilinx_zynq.o
  CC      i386-softmmu/hw/i386/pc_sysfw.o
  CC      aarch64-softmmu/hw/arm/spitz.o
  LINK    lm32-softmmu/qemu-system-lm32
  CC      arm-softmmu/hw/arm/z2.o
  CC      i386-softmmu/hw/i386/x86-iommu.o
  CC      i386-softmmu/hw/i386/intel_iommu.o
  CC      arm-softmmu/hw/arm/virt-acpi-build.o
  CC      aarch64-softmmu/hw/arm/stellaris.o
  CC      arm-softmmu/hw/arm/netduino2.o
  CC      arm-softmmu/hw/arm/sysbus-fdt.o
  CC      aarch64-softmmu/hw/arm/tosa.o
  CC      arm-softmmu/hw/arm/armv7m.o
  CC      aarch64-softmmu/hw/arm/versatilepb.o
  CC      i386-softmmu/hw/i386/amd_iommu.o
  GEN     m68k-softmmu/hmp-commands.h
  CC      arm-softmmu/hw/arm/exynos4210.o
  GEN     m68k-softmmu/hmp-commands-info.h
  GEN     m68k-softmmu/config-target.h
  CC      aarch64-softmmu/hw/arm/vexpress.o
  CC      m68k-softmmu/exec.o
  CC      arm-softmmu/hw/arm/pxa2xx.o
  CC      i386-softmmu/hw/i386/kvmvapic.o
  CC      aarch64-softmmu/hw/arm/virt.o
  CC      arm-softmmu/hw/arm/pxa2xx_gpio.o
  CC      aarch64-softmmu/hw/arm/xilinx_zynq.o
  CC      i386-softmmu/hw/i386/acpi-build.o
  CC      arm-softmmu/hw/arm/pxa2xx_pic.o
  CC      aarch64-softmmu/hw/arm/z2.o
  CC      arm-softmmu/hw/arm/digic.o
  CC      i386-softmmu/hw/i386/pci-assign-load-rom.o
  CC      arm-softmmu/hw/arm/omap1.o
  CC      m68k-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/hw/arm/virt-acpi-build.o
  CC      i386-softmmu/target/i386/helper.o
  CC      aarch64-softmmu/hw/arm/netduino2.o
  CC      aarch64-softmmu/hw/arm/sysbus-fdt.o
  CC      i386-softmmu/target/i386/cpu.o
  CC      arm-softmmu/hw/arm/omap2.o
  CC      aarch64-softmmu/hw/arm/armv7m.o
  CC      m68k-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/hw/arm/exynos4210.o
  CC      i386-softmmu/target/i386/gdbstub.o
  CC      arm-softmmu/hw/arm/strongarm.o
  CC      aarch64-softmmu/hw/arm/pxa2xx.o
  CC      i386-softmmu/target/i386/xsave_helper.o
  CC      arm-softmmu/hw/arm/allwinner-a10.o
  CC      i386-softmmu/target/i386/translate.o
  CC      arm-softmmu/hw/arm/cubieboard.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_gpio.o
  CC      arm-softmmu/hw/arm/bcm2835_peripherals.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_pic.o
  CC      m68k-softmmu/tcg/optimize.o
  CC      arm-softmmu/hw/arm/bcm2836.o
  CC      aarch64-softmmu/hw/arm/digic.o
  CC      aarch64-softmmu/hw/arm/omap1.o
  CC      arm-softmmu/hw/arm/raspi.o
  CC      m68k-softmmu/tcg/tcg-common.o
  CC      m68k-softmmu/tcg/tcg-runtime.o
  CC      arm-softmmu/hw/arm/stm32f205_soc.o
  CC      m68k-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/hw/arm/omap2.o
  CC      arm-softmmu/hw/arm/fsl-imx25.o
  CC      arm-softmmu/hw/arm/imx25_pdk.o
  CC      aarch64-softmmu/hw/arm/strongarm.o
  CC      arm-softmmu/hw/arm/fsl-imx31.o
  CC      arm-softmmu/hw/arm/kzm.o
  CC      aarch64-softmmu/hw/arm/allwinner-a10.o
  CC      arm-softmmu/hw/arm/fsl-imx6.o
  CC      arm-softmmu/hw/arm/sabrelite.o
  CC      aarch64-softmmu/hw/arm/cubieboard.o
  CC      arm-softmmu/hw/arm/aspeed_soc.o
  CC      m68k-softmmu/disas.o
  CC      aarch64-softmmu/hw/arm/bcm2835_peripherals.o
  CC      i386-softmmu/target/i386/bpt_helper.o
  CC      arm-softmmu/hw/arm/aspeed.o
  GEN     m68k-softmmu/gdbstub-xml.c
  CC      i386-softmmu/target/i386/cc_helper.o
  CC      m68k-softmmu/hax-stub.o
  CC      arm-softmmu/hw/arm/mps2.o
  CC      aarch64-softmmu/hw/arm/bcm2836.o
  CC      i386-softmmu/target/i386/excp_helper.o
  CC      arm-softmmu/target/arm/arm-semi.o
  CC      m68k-softmmu/arch_init.o
  CC      aarch64-softmmu/hw/arm/raspi.o
  CC      i386-softmmu/target/i386/fpu_helper.o
  CC      m68k-softmmu/cpus.o
  CC      arm-softmmu/target/arm/machine.o
  CC      aarch64-softmmu/hw/arm/stm32f205_soc.o
  CC      arm-softmmu/target/arm/psci.o
  CC      m68k-softmmu/monitor.o
  CC      aarch64-softmmu/hw/arm/xlnx-zynqmp.o
  CC      arm-softmmu/target/arm/arch_dump.o
  CC      arm-softmmu/target/arm/monitor.o
  CC      aarch64-softmmu/hw/arm/xlnx-ep108.o
  CC      arm-softmmu/target/arm/kvm-stub.o
  CC      aarch64-softmmu/hw/arm/fsl-imx25.o
  CC      arm-softmmu/target/arm/translate.o
  CC      m68k-softmmu/gdbstub.o
  CC      aarch64-softmmu/hw/arm/imx25_pdk.o
  CC      i386-softmmu/target/i386/int_helper.o
  CC      aarch64-softmmu/hw/arm/fsl-imx31.o
  CC      i386-softmmu/target/i386/mem_helper.o
  CC      m68k-softmmu/balloon.o
  CC      aarch64-softmmu/hw/arm/kzm.o
  CC      i386-softmmu/target/i386/misc_helper.o
  CC      m68k-softmmu/ioport.o
  CC      aarch64-softmmu/hw/arm/fsl-imx6.o
  CC      m68k-softmmu/numa.o
  CC      i386-softmmu/target/i386/mpx_helper.o
  CC      aarch64-softmmu/hw/arm/sabrelite.o
  CC      m68k-softmmu/qtest.o
  CC      aarch64-softmmu/hw/arm/aspeed_soc.o
  CC      i386-softmmu/target/i386/seg_helper.o
  CC      m68k-softmmu/memory.o
  CC      aarch64-softmmu/hw/arm/aspeed.o
  CC      aarch64-softmmu/hw/arm/mps2.o
  CC      m68k-softmmu/memory_mapping.o
  CC      aarch64-softmmu/target/arm/arm-semi.o
  CC      m68k-softmmu/dump.o
  CC      aarch64-softmmu/target/arm/machine.o
  CC      i386-softmmu/target/i386/smm_helper.o
  CC      aarch64-softmmu/target/arm/psci.o
  CC      m68k-softmmu/migration/ram.o
  CC      i386-softmmu/target/i386/svm_helper.o
  CC      aarch64-softmmu/target/arm/arch_dump.o
  CC      i386-softmmu/target/i386/machine.o
  CC      arm-softmmu/target/arm/op_helper.o
  CC      aarch64-softmmu/target/arm/monitor.o
  CC      i386-softmmu/target/i386/arch_memory_mapping.o
  CC      m68k-softmmu/accel/accel.o
  CC      aarch64-softmmu/target/arm/kvm-stub.o
  CC      i386-softmmu/target/i386/arch_dump.o
  CC      m68k-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/target/arm/translate.o
  CC      arm-softmmu/target/arm/helper.o
  CC      i386-softmmu/target/i386/monitor.o
  CC      m68k-softmmu/accel/tcg/tcg-all.o
  CC      i386-softmmu/target/i386/kvm-stub.o
  CC      m68k-softmmu/accel/tcg/cputlb.o
  GEN     trace/generated-helpers.c
  CC      i386-softmmu/trace/control-target.o
  CC      i386-softmmu/gdbstub-xml.o
  CC      i386-softmmu/trace/generated-helpers.o
  CC      m68k-softmmu/accel/tcg/cpu-exec.o
  LINK    i386-softmmu/qemu-system-i386
  CC      m68k-softmmu/accel/tcg/cpu-exec-common.o
  CC      arm-softmmu/target/arm/cpu.o
  CC      m68k-softmmu/accel/tcg/translate-all.o
  CC      m68k-softmmu/hw/char/mcf_uart.o
  CC      m68k-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/target/arm/neon_helper.o
  CC      m68k-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/target/arm/op_helper.o
  CC      m68k-softmmu/hw/misc/mmio_interface.o
  CC      m68k-softmmu/hw/net/mcf_fec.o
  CC      arm-softmmu/target/arm/iwmmxt_helper.o
  CC      m68k-softmmu/hw/net/vhost_net.o
  CC      m68k-softmmu/hw/net/rocker/qmp-norocker.o
  CC      aarch64-softmmu/target/arm/helper.o
  CC      m68k-softmmu/hw/vfio/common.o
  CC      m68k-softmmu/hw/vfio/platform.o
  CC      m68k-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/target/arm/gdbstub.o
  CC      aarch64-softmmu/target/arm/cpu.o
  CC      m68k-softmmu/hw/m68k/an5206.o
  CC      arm-softmmu/target/arm/crypto_helper.o
  CC      m68k-softmmu/hw/m68k/mcf5208.o
  CC      arm-softmmu/target/arm/arm-powerctl.o
  GEN     trace/generated-helpers.c
  CC      arm-softmmu/trace/control-target.o
  CC      m68k-softmmu/hw/m68k/mcf5206.o
  CC      arm-softmmu/gdbstub-xml.o
  CC      aarch64-softmmu/target/arm/neon_helper.o
  CC      arm-softmmu/trace/generated-helpers.o
  CC      m68k-softmmu/hw/m68k/mcf_intc.o
  LINK    arm-softmmu/qemu-system-arm
  CC      m68k-softmmu/target/m68k/m68k-semi.o
  CC      aarch64-softmmu/target/arm/iwmmxt_helper.o
  CC      m68k-softmmu/target/m68k/translate.o
  CC      aarch64-softmmu/target/arm/gdbstub.o
  CC      m68k-softmmu/target/m68k/op_helper.o
  CC      aarch64-softmmu/target/arm/cpu64.o
  CC      aarch64-softmmu/target/arm/translate-a64.o
  CC      aarch64-softmmu/target/arm/helper-a64.o
  CC      aarch64-softmmu/target/arm/gdbstub64.o
  CC      m68k-softmmu/target/m68k/helper.o
  CC      m68k-softmmu/target/m68k/cpu.o
  CC      m68k-softmmu/target/m68k/fpu_helper.o
  CC      m68k-softmmu/target/m68k/gdbstub.o
  CC      aarch64-softmmu/target/arm/crypto_helper.o
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/target/arm/arm-powerctl.o
  CC      m68k-softmmu/trace/control-target.o
  CC      m68k-softmmu/gdbstub-xml.o
  CC      m68k-softmmu/trace/generated-helpers.o
  GEN     microblazeel-softmmu/hmp-commands.h
  GEN     microblaze-softmmu/hmp-commands.h
  GEN     microblazeel-softmmu/hmp-commands-info.h
  GEN     microblazeel-softmmu/config-target.h
  GEN     microblaze-softmmu/hmp-commands-info.h
  GEN     microblaze-softmmu/config-target.h
  CC      microblaze-softmmu/exec.o
  CC      microblazeel-softmmu/exec.o
  LINK    m68k-softmmu/qemu-system-m68k
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/trace/control-target.o
  CC      microblazeel-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/gdbstub-xml.o
  CC      aarch64-softmmu/trace/generated-helpers.o
  CC      microblazeel-softmmu/tcg/tcg-op.o
  CC      microblaze-softmmu/tcg/tcg.o
  LINK    aarch64-softmmu/qemu-system-aarch64
  CC      microblazeel-softmmu/tcg/optimize.o
  CC      microblazeel-softmmu/tcg/tcg-common.o
  CC      microblaze-softmmu/tcg/tcg-op.o
  CC      microblazeel-softmmu/tcg/tcg-runtime.o
  GEN     mips64el-softmmu/hmp-commands.h
  GEN     mips64el-softmmu/hmp-commands-info.h
  GEN     mips64el-softmmu/config-target.h
  CC      mips64el-softmmu/exec.o
  CC      microblazeel-softmmu/fpu/softfloat.o
  GEN     mips64-softmmu/hmp-commands.h
  GEN     mips64-softmmu/hmp-commands-info.h
  GEN     mips64-softmmu/config-target.h
  CC      mips64-softmmu/exec.o
  CC      microblaze-softmmu/tcg/optimize.o
  CC      mips64el-softmmu/tcg/tcg.o
  CC      microblazeel-softmmu/disas.o
  CC      microblaze-softmmu/tcg/tcg-common.o
  CC      microblazeel-softmmu/hax-stub.o
  CC      microblaze-softmmu/tcg/tcg-runtime.o
  CC      microblazeel-softmmu/arch_init.o
  CC      microblazeel-softmmu/cpus.o
  CC      microblaze-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/tcg/tcg.o
  CC      microblazeel-softmmu/monitor.o
  CC      mips64el-softmmu/tcg/tcg-op.o
  CC      mips64-softmmu/tcg/tcg-op.o
  CC      microblaze-softmmu/disas.o
  CC      microblazeel-softmmu/gdbstub.o
  CC      microblaze-softmmu/hax-stub.o
  CC      microblaze-softmmu/arch_init.o
  CC      microblaze-softmmu/cpus.o
  CC      microblazeel-softmmu/balloon.o
  CC      mips64el-softmmu/tcg/optimize.o
  CC      microblazeel-softmmu/ioport.o
  CC      microblaze-softmmu/monitor.o
  CC      microblazeel-softmmu/numa.o
  CC      microblazeel-softmmu/qtest.o
  CC      mips64el-softmmu/tcg/tcg-common.o
  CC      mips64-softmmu/tcg/optimize.o
  CC      mips64el-softmmu/tcg/tcg-runtime.o
  CC      microblazeel-softmmu/memory.o
  CC      microblaze-softmmu/gdbstub.o
  CC      mips64el-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/tcg/tcg-common.o
  CC      microblaze-softmmu/balloon.o
  CC      mips64-softmmu/tcg/tcg-runtime.o
  CC      microblazeel-softmmu/memory_mapping.o
  CC      microblaze-softmmu/ioport.o
  CC      microblazeel-softmmu/dump.o
  CC      microblaze-softmmu/numa.o
  CC      mips64-softmmu/fpu/softfloat.o
  CC      microblaze-softmmu/qtest.o
  CC      microblazeel-softmmu/migration/ram.o
  CC      microblaze-softmmu/memory.o
  CC      mips64el-softmmu/disas.o
  CC      microblazeel-softmmu/accel/accel.o
  CC      mips64el-softmmu/hax-stub.o
  CC      microblazeel-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/accel/tcg/tcg-all.o
  CC      mips64el-softmmu/arch_init.o
  CC      microblaze-softmmu/memory_mapping.o
  CC      mips64el-softmmu/cpus.o
  CC      microblazeel-softmmu/accel/tcg/cputlb.o
  CC      mips64-softmmu/disas.o
  CC      microblaze-softmmu/dump.o
  CC      mips64-softmmu/hax-stub.o
  CC      mips64el-softmmu/monitor.o
  CC      mips64-softmmu/arch_init.o
  CC      microblaze-softmmu/migration/ram.o
  CC      mips64-softmmu/cpus.o
  CC      microblazeel-softmmu/accel/tcg/cpu-exec.o
  CC      mips64-softmmu/monitor.o
  CC      microblaze-softmmu/accel/accel.o
  CC      mips64el-softmmu/gdbstub.o
  CC      microblazeel-softmmu/accel/tcg/cpu-exec-common.o
  CC      microblaze-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/accel/tcg/translate-all.o
  CC      microblaze-softmmu/accel/tcg/tcg-all.o
  CC      mips64el-softmmu/balloon.o
  CC      mips64el-softmmu/ioport.o
  CC      microblaze-softmmu/accel/tcg/cputlb.o
  CC      microblazeel-softmmu/hw/core/generic-loader.o
  CC      mips64el-softmmu/numa.o
  CC      microblazeel-softmmu/hw/core/null-machine.o
  CC      mips64-softmmu/gdbstub.o
  CC      mips64el-softmmu/qtest.o
  CC      microblazeel-softmmu/hw/misc/mmio_interface.o
  CC      microblazeel-softmmu/hw/net/xilinx_ethlite.o
  CC      mips64-softmmu/balloon.o
  CC      mips64el-softmmu/memory.o
  CC      microblazeel-softmmu/hw/net/vhost_net.o
  CC      mips64-softmmu/ioport.o
  CC      microblaze-softmmu/accel/tcg/cpu-exec.o
  CC      microblazeel-softmmu/hw/net/rocker/qmp-norocker.o
  CC      mips64-softmmu/numa.o
  CC      microblaze-softmmu/accel/tcg/cpu-exec-common.o
  CC      microblazeel-softmmu/hw/vfio/common.o
  CC      microblaze-softmmu/accel/tcg/translate-all.o
  CC      mips64-softmmu/qtest.o
  CC      mips64el-softmmu/memory_mapping.o
  CC      microblazeel-softmmu/hw/vfio/platform.o
  CC      mips64el-softmmu/dump.o
  CC      mips64-softmmu/memory.o
  CC      microblaze-softmmu/hw/core/generic-loader.o
  CC      microblaze-softmmu/hw/core/null-machine.o
  CC      microblazeel-softmmu/hw/vfio/spapr.o
  CC      mips64el-softmmu/migration/ram.o
  CC      microblaze-softmmu/hw/misc/mmio_interface.o
  CC      microblazeel-softmmu/hw/microblaze/petalogix_s3adsp1800_mmu.o
  CC      microblaze-softmmu/hw/net/xilinx_ethlite.o
  CC      microblazeel-softmmu/hw/microblaze/petalogix_ml605_mmu.o
  CC      mips64-softmmu/memory_mapping.o
  CC      microblaze-softmmu/hw/net/vhost_net.o
  CC      mips64el-softmmu/accel/accel.o
  CC      mips64-softmmu/dump.o
  CC      microblazeel-softmmu/hw/microblaze/boot.o
  CC      microblaze-softmmu/hw/net/rocker/qmp-norocker.o
  CC      microblaze-softmmu/hw/vfio/common.o
  CC      mips64el-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/target/microblaze/translate.o
  CC      mips64el-softmmu/accel/tcg/tcg-all.o
  CC      mips64-softmmu/migration/ram.o
  CC      mips64el-softmmu/accel/tcg/cputlb.o
  CC      microblaze-softmmu/hw/vfio/platform.o
  CC      microblazeel-softmmu/target/microblaze/op_helper.o
  CC      microblaze-softmmu/hw/vfio/spapr.o
  CC      mips64-softmmu/accel/accel.o
  CC      mips64-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/target/microblaze/helper.o
  CC      microblaze-softmmu/hw/microblaze/petalogix_s3adsp1800_mmu.o
  CC      mips64el-softmmu/accel/tcg/cpu-exec.o
  CC      mips64-softmmu/accel/tcg/tcg-all.o
  CC      microblaze-softmmu/hw/microblaze/petalogix_ml605_mmu.o
  CC      microblazeel-softmmu/target/microblaze/cpu.o
  CC      mips64-softmmu/accel/tcg/cputlb.o
  CC      mips64el-softmmu/accel/tcg/cpu-exec-common.o
  CC      microblaze-softmmu/hw/microblaze/boot.o
  CC      microblazeel-softmmu/target/microblaze/gdbstub.o
  CC      mips64el-softmmu/accel/tcg/translate-all.o
  CC      microblaze-softmmu/target/microblaze/translate.o
  CC      microblazeel-softmmu/target/microblaze/mmu.o
  GEN     trace/generated-helpers.c
  CC      microblazeel-softmmu/trace/control-target.o
  CC      mips64el-softmmu/hw/9pfs/virtio-9p-device.o
  CC      microblazeel-softmmu/trace/generated-helpers.o
  CC      mips64el-softmmu/hw/block/virtio-blk.o
  CC      mips64-softmmu/accel/tcg/cpu-exec.o
  LINK    microblazeel-softmmu/qemu-system-microblazeel
  CC      microblaze-softmmu/target/microblaze/op_helper.o
  CC      mips64-softmmu/accel/tcg/cpu-exec-common.o
  CC      mips64el-softmmu/hw/block/dataplane/virtio-blk.o
  CC      microblaze-softmmu/target/microblaze/helper.o
  CC      microblaze-softmmu/target/microblaze/cpu.o
  CC      mips64-softmmu/accel/tcg/translate-all.o
  CC      mips64el-softmmu/hw/char/virtio-serial-bus.o
  CC      microblaze-softmmu/target/microblaze/gdbstub.o
  CC      microblaze-softmmu/target/microblaze/mmu.o
  CC      mips64el-softmmu/hw/core/generic-loader.o
  CC      mips64-softmmu/hw/9pfs/virtio-9p-device.o
  GEN     trace/generated-helpers.c
  CC      microblaze-softmmu/trace/control-target.o
  CC      mips64-softmmu/hw/block/virtio-blk.o
  CC      microblaze-softmmu/trace/generated-helpers.o
  CC      mips64el-softmmu/hw/core/null-machine.o
  CC      mips64el-softmmu/hw/display/vga.o
  LINK    microblaze-softmmu/qemu-system-microblaze
  CC      mips64el-softmmu/hw/display/virtio-gpu.o
  CC      mips64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mips64-softmmu/hw/char/virtio-serial-bus.o
  CC      mips64el-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64-softmmu/hw/core/generic-loader.o
  GEN     mipsel-softmmu/hmp-commands.h
  CC      mips64-softmmu/hw/core/null-machine.o
  GEN     mipsel-softmmu/hmp-commands-info.h
  GEN     mipsel-softmmu/config-target.h
  CC      mips64el-softmmu/hw/intc/mips_gic.o
  CC      mipsel-softmmu/exec.o
  CC      mips64-softmmu/hw/display/vga.o
  CC      mips64el-softmmu/hw/misc/ivshmem.o
  CC      mips64el-softmmu/hw/misc/mips_cmgcr.o
  CC      mips64el-softmmu/hw/misc/mips_cpc.o
  CC      mips64-softmmu/hw/display/virtio-gpu.o
  GEN     mips-softmmu/hmp-commands.h
  GEN     mips-softmmu/hmp-commands-info.h
  GEN     mips-softmmu/config-target.h
  CC      mips64el-softmmu/hw/misc/mips_itu.o
  CC      mips-softmmu/exec.o
  CC      mips64el-softmmu/hw/misc/mmio_interface.o
  CC      mips64-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/hw/net/virtio-net.o
  CC      mipsel-softmmu/tcg/tcg.o
  CC      mips64el-softmmu/hw/net/vhost_net.o
  CC      mips64-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64el-softmmu/hw/scsi/virtio-scsi.o
  CC      mips64-softmmu/hw/intc/mips_gic.o
  CC      mips-softmmu/tcg/tcg.o
  CC      mips64el-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips64-softmmu/hw/misc/ivshmem.o
  CC      mips64el-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mips64-softmmu/hw/misc/mips_cmgcr.o
  CC      mipsel-softmmu/tcg/tcg-op.o
  CC      mips64el-softmmu/hw/scsi/vhost-scsi.o
  CC      mips64-softmmu/hw/misc/mips_cpc.o
  CC      mips64el-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mips64-softmmu/hw/misc/mips_itu.o
  CC      mips64el-softmmu/hw/timer/mips_gictimer.o
  CC      mips-softmmu/tcg/tcg-op.o
  CC      mips64-softmmu/hw/misc/mmio_interface.o
  CC      mips64el-softmmu/hw/timer/mc146818rtc.o
  CC      mips64-softmmu/hw/net/virtio-net.o
  CC      mips64el-softmmu/hw/vfio/common.o
  CC      mips64-softmmu/hw/net/vhost_net.o
  CC      mips64-softmmu/hw/scsi/virtio-scsi.o
  CC      mipsel-softmmu/tcg/optimize.o
  CC      mips64el-softmmu/hw/vfio/pci.o
  CC      mips64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips-softmmu/tcg/optimize.o
  CC      mips64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mipsel-softmmu/tcg/tcg-common.o
  CC      mips64el-softmmu/hw/vfio/pci-quirks.o
  CC      mips64-softmmu/hw/scsi/vhost-scsi.o
  CC      mipsel-softmmu/tcg/tcg-runtime.o
  CC      mips64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mips-softmmu/tcg/tcg-common.o
  CC      mips64el-softmmu/hw/vfio/platform.o
  CC      mipsel-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/hw/timer/mips_gictimer.o
  CC      mips64-softmmu/hw/timer/mc146818rtc.o
  CC      mips-softmmu/tcg/tcg-runtime.o
  CC      mips64el-softmmu/hw/vfio/spapr.o
  CC      mips64el-softmmu/hw/virtio/virtio.o
  CC      mips-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/hw/vfio/common.o
  CC      mips64-softmmu/hw/vfio/pci.o
  CC      mips64el-softmmu/hw/virtio/virtio-balloon.o
  CC      mips64el-softmmu/hw/virtio/vhost.o
  CC      mips64-softmmu/hw/vfio/pci-quirks.o
  CC      mipsel-softmmu/disas.o
  CC      mips64el-softmmu/hw/virtio/vhost-backend.o
  CC      mipsel-softmmu/hax-stub.o
  CC      mips64-softmmu/hw/vfio/platform.o
  CC      mips-softmmu/disas.o
  CC      mipsel-softmmu/arch_init.o
  CC      mips64el-softmmu/hw/virtio/vhost-user.o
  CC      mips64-softmmu/hw/vfio/spapr.o
  CC      mips-softmmu/hax-stub.o
  CC      mipsel-softmmu/cpus.o
  CC      mips64el-softmmu/hw/virtio/vhost-vsock.o
  CC      mips-softmmu/arch_init.o
  CC      mips64-softmmu/hw/virtio/virtio.o
  CC      mips-softmmu/cpus.o
  CC      mips64el-softmmu/hw/virtio/virtio-crypto.o
  CC      mipsel-softmmu/monitor.o
  CC      mips64el-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      mips-softmmu/monitor.o
  CC      mips64-softmmu/hw/virtio/virtio-balloon.o
  CC      mips64el-softmmu/hw/mips/mips_r4k.o
  CC      mips64el-softmmu/hw/mips/mips_malta.o
  CC      mipsel-softmmu/gdbstub.o
  CC      mips64-softmmu/hw/virtio/vhost.o
  CC      mips64el-softmmu/hw/mips/mips_mipssim.o
  CC      mipsel-softmmu/balloon.o
  CC      mips-softmmu/gdbstub.o
  CC      mips64-softmmu/hw/virtio/vhost-backend.o
  CC      mips64el-softmmu/hw/mips/addr.o
  CC      mipsel-softmmu/ioport.o
  CC      mips64-softmmu/hw/virtio/vhost-user.o
  CC      mips64el-softmmu/hw/mips/cputimer.o
  CC      mips-softmmu/balloon.o
  CC      mipsel-softmmu/numa.o
  CC      mips-softmmu/ioport.o
  CC      mips64-softmmu/hw/virtio/vhost-vsock.o
  CC      mips64el-softmmu/hw/mips/mips_int.o
  CC      mips-softmmu/numa.o
  CC      mipsel-softmmu/qtest.o
  CC      mips64-softmmu/hw/virtio/virtio-crypto.o
  CC      mips64el-softmmu/hw/mips/mips_jazz.o
  CC      mips64el-softmmu/hw/mips/mips_fulong2e.o
  CC      mips-softmmu/qtest.o
  CC      mips64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      mipsel-softmmu/memory.o
  CC      mips64el-softmmu/hw/mips/gt64xxx_pci.o
  CC      mips-softmmu/memory.o
  CC      mips64-softmmu/hw/mips/mips_r4k.o
  CC      mips64el-softmmu/hw/mips/cps.o
  CC      mips64-softmmu/hw/mips/mips_malta.o
  CC      mips64-softmmu/hw/mips/mips_mipssim.o
  CC      mips64el-softmmu/hw/mips/boston.o
  CC      mipsel-softmmu/memory_mapping.o
  CC      mips-softmmu/memory_mapping.o
  CC      mips64el-softmmu/target/mips/translate.o
  CC      mips64-softmmu/hw/mips/addr.o
  CC      mipsel-softmmu/dump.o
  CC      mips64-softmmu/hw/mips/cputimer.o
  CC      mips-softmmu/dump.o
  CC      mips64-softmmu/hw/mips/mips_int.o
  CC      mipsel-softmmu/migration/ram.o
  CC      mips-softmmu/migration/ram.o
  CC      mips64-softmmu/hw/mips/mips_jazz.o
  CC      mips64-softmmu/hw/mips/gt64xxx_pci.o
  CC      mipsel-softmmu/accel/accel.o
  CC      mips64-softmmu/hw/mips/cps.o
  CC      mips-softmmu/accel/accel.o
  CC      mipsel-softmmu/accel/stubs/kvm-stub.o
  CC      mips64-softmmu/target/mips/translate.o
  CC      mips-softmmu/accel/stubs/kvm-stub.o
  CC      mipsel-softmmu/accel/tcg/tcg-all.o
  CC      mipsel-softmmu/accel/tcg/cputlb.o
  CC      mips-softmmu/accel/tcg/tcg-all.o
  CC      mips-softmmu/accel/tcg/cputlb.o
  CC      mips-softmmu/accel/tcg/cpu-exec.o
  CC      mipsel-softmmu/accel/tcg/cpu-exec.o
  CC      mipsel-softmmu/accel/tcg/cpu-exec-common.o
  CC      mips-softmmu/accel/tcg/cpu-exec-common.o
  CC      mipsel-softmmu/accel/tcg/translate-all.o
  CC      mips-softmmu/accel/tcg/translate-all.o
  CC      mipsel-softmmu/hw/9pfs/virtio-9p-device.o
  CC      mipsel-softmmu/hw/block/virtio-blk.o
  CC      mips-softmmu/hw/9pfs/virtio-9p-device.o
  CC      mips-softmmu/hw/block/virtio-blk.o
  CC      mipsel-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mips64el-softmmu/target/mips/dsp_helper.o
  CC      mipsel-softmmu/hw/char/virtio-serial-bus.o
  CC      mips-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mipsel-softmmu/hw/core/generic-loader.o
  CC      mipsel-softmmu/hw/core/null-machine.o
  CC      mips-softmmu/hw/char/virtio-serial-bus.o
  CC      mipsel-softmmu/hw/display/vga.o
  CC      mips-softmmu/hw/core/generic-loader.o
  CC      mips64el-softmmu/target/mips/op_helper.o
  CC      mips64-softmmu/target/mips/dsp_helper.o
  CC      mips-softmmu/hw/core/null-machine.o
  CC      mips-softmmu/hw/display/vga.o
  CC      mipsel-softmmu/hw/display/virtio-gpu.o
  CC      mips-softmmu/hw/display/virtio-gpu.o
  CC      mips64-softmmu/target/mips/op_helper.o
  CC      mipsel-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/target/mips/lmi_helper.o
  CC      mips-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/target/mips/helper.o
  CC      mipsel-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64el-softmmu/target/mips/cpu.o
  CC      mipsel-softmmu/hw/intc/mips_gic.o
  CC      mips-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64el-softmmu/target/mips/gdbstub.o
  CC      mipsel-softmmu/hw/misc/ivshmem.o
  CC      mips64el-softmmu/target/mips/msa_helper.o
  CC      mips-softmmu/hw/intc/mips_gic.o
  CC      mips-softmmu/hw/misc/ivshmem.o
  CC      mipsel-softmmu/hw/misc/mips_cmgcr.o
  CC      mips64-softmmu/target/mips/lmi_helper.o
  CC      mipsel-softmmu/hw/misc/mips_cpc.o
  CC      mips-softmmu/hw/misc/mips_cmgcr.o
  CC      mipsel-softmmu/hw/misc/mips_itu.o
  CC      mips64-softmmu/target/mips/helper.o
  CC      mips-softmmu/hw/misc/mips_cpc.o
  CC      mipsel-softmmu/hw/misc/mmio_interface.o
  CC      mipsel-softmmu/hw/net/virtio-net.o
  CC      mips64-softmmu/target/mips/cpu.o
  CC      mips-softmmu/hw/misc/mips_itu.o
  CC      mips64-softmmu/target/mips/gdbstub.o
  CC      mips-softmmu/hw/misc/mmio_interface.o
  CC      mips64-softmmu/target/mips/msa_helper.o
  CC      mipsel-softmmu/hw/net/vhost_net.o
  CC      mips-softmmu/hw/net/virtio-net.o
  CC      mipsel-softmmu/hw/scsi/virtio-scsi.o
  CC      mips-softmmu/hw/net/vhost_net.o
  CC      mips-softmmu/hw/scsi/virtio-scsi.o
  CC      mipsel-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips64el-softmmu/target/mips/mips-semi.o
  CC      mipsel-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mips-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips64el-softmmu/target/mips/machine.o
  CC      mipsel-softmmu/hw/scsi/vhost-scsi.o
  GEN     trace/generated-helpers.c
  CC      mips-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mips64el-softmmu/trace/control-target.o
  CC      mipsel-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mips-softmmu/hw/scsi/vhost-scsi.o
  CC      mips64el-softmmu/trace/generated-helpers.o
  CC      mips-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mipsel-softmmu/hw/timer/mips_gictimer.o
  CC      mips-softmmu/hw/timer/mips_gictimer.o
  LINK    mips64el-softmmu/qemu-system-mips64el
  CC      mipsel-softmmu/hw/timer/mc146818rtc.o
  CC      mips-softmmu/hw/timer/mc146818rtc.o
  CC      mips64-softmmu/target/mips/mips-semi.o
  CC      mips-softmmu/hw/vfio/common.o
  CC      mipsel-softmmu/hw/vfio/common.o
  CC      mips64-softmmu/target/mips/machine.o
  CC      mips-softmmu/hw/vfio/pci.o
  CC      mipsel-softmmu/hw/vfio/pci.o
  GEN     trace/generated-helpers.c
  CC      mips64-softmmu/trace/control-target.o
  CC      mips64-softmmu/trace/generated-helpers.o
  CC      mips-softmmu/hw/vfio/pci-quirks.o
  GEN     moxie-softmmu/hmp-commands.h
  GEN     moxie-softmmu/hmp-commands-info.h
  LINK    mips64-softmmu/qemu-system-mips64
  GEN     moxie-softmmu/config-target.h
  CC      moxie-softmmu/exec.o
  CC      mips-softmmu/hw/vfio/platform.o
  CC      mipsel-softmmu/hw/vfio/pci-quirks.o
  CC      mips-softmmu/hw/vfio/spapr.o
  CC      mipsel-softmmu/hw/vfio/platform.o
  CC      mipsel-softmmu/hw/vfio/spapr.o
  CC      mips-softmmu/hw/virtio/virtio.o
  CC      mipsel-softmmu/hw/virtio/virtio.o
  CC      moxie-softmmu/tcg/tcg.o
  CC      mipsel-softmmu/hw/virtio/virtio-balloon.o
  CC      mips-softmmu/hw/virtio/virtio-balloon.o
  CC      moxie-softmmu/tcg/tcg-op.o
  CC      mips-softmmu/hw/virtio/vhost.o
  CC      mipsel-softmmu/hw/virtio/vhost.o
  CC      mipsel-softmmu/hw/virtio/vhost-backend.o
  CC      mipsel-softmmu/hw/virtio/vhost-user.o
  CC      mips-softmmu/hw/virtio/vhost-backend.o
  CC      mipsel-softmmu/hw/virtio/vhost-vsock.o
  CC      mips-softmmu/hw/virtio/vhost-user.o
  CC      mipsel-softmmu/hw/virtio/virtio-crypto.o
  CC      mipsel-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      mips-softmmu/hw/virtio/vhost-vsock.o
  CC      mipsel-softmmu/hw/mips/mips_r4k.o
  GEN     nios2-softmmu/hmp-commands.h
  GEN     nios2-softmmu/hmp-commands-info.h
  GEN     nios2-softmmu/config-target.h
  CC      mips-softmmu/hw/virtio/virtio-crypto.o
  CC      nios2-softmmu/exec.o
  CC      mipsel-softmmu/hw/mips/mips_malta.o
  CC      mips-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      moxie-softmmu/tcg/optimize.o
  CC      mips-softmmu/hw/mips/mips_r4k.o
  CC      mipsel-softmmu/hw/mips/mips_mipssim.o
  CC      mipsel-softmmu/hw/mips/addr.o
  CC      mips-softmmu/hw/mips/mips_malta.o
  CC      mipsel-softmmu/hw/mips/cputimer.o
  CC      moxie-softmmu/tcg/tcg-common.o
  CC      nios2-softmmu/tcg/tcg.o
  CC      moxie-softmmu/tcg/tcg-runtime.o
  CC      mipsel-softmmu/hw/mips/mips_int.o
  CC      mips-softmmu/hw/mips/mips_mipssim.o
  CC      mipsel-softmmu/hw/mips/gt64xxx_pci.o
  CC      moxie-softmmu/fpu/softfloat.o
  CC      mips-softmmu/hw/mips/addr.o
  CC      mips-softmmu/hw/mips/cputimer.o
  CC      mips-softmmu/hw/mips/mips_int.o
  CC      mipsel-softmmu/hw/mips/cps.o
  CC      mips-softmmu/hw/mips/gt64xxx_pci.o
  CC      mipsel-softmmu/target/mips/translate.o
  CC      nios2-softmmu/tcg/tcg-op.o
  CC      mips-softmmu/hw/mips/cps.o
  CC      mips-softmmu/target/mips/translate.o
  CC      moxie-softmmu/disas.o
  CC      moxie-softmmu/hax-stub.o
  CC      moxie-softmmu/arch_init.o
  CC      nios2-softmmu/tcg/optimize.o
  CC      moxie-softmmu/cpus.o
  CC      nios2-softmmu/tcg/tcg-common.o
  CC      moxie-softmmu/monitor.o
  CC      nios2-softmmu/tcg/tcg-runtime.o
  CC      nios2-softmmu/fpu/softfloat.o
  CC      moxie-softmmu/gdbstub.o
  CC      mipsel-softmmu/target/mips/dsp_helper.o
  CC      mipsel-softmmu/target/mips/op_helper.o
  CC      moxie-softmmu/balloon.o
  CC      moxie-softmmu/ioport.o
  CC      mips-softmmu/target/mips/dsp_helper.o
  CC      moxie-softmmu/numa.o
  CC      nios2-softmmu/disas.o
  CC      moxie-softmmu/qtest.o
  CC      nios2-softmmu/hax-stub.o
  CC      mips-softmmu/target/mips/op_helper.o
  CC      nios2-softmmu/arch_init.o
  CC      moxie-softmmu/memory.o
  CC      nios2-softmmu/cpus.o
  CC      mipsel-softmmu/target/mips/lmi_helper.o
  CC      nios2-softmmu/monitor.o
  CC      moxie-softmmu/memory_mapping.o
  CC      mipsel-softmmu/target/mips/helper.o
  CC      moxie-softmmu/dump.o
  CC      mipsel-softmmu/target/mips/cpu.o
  CC      mipsel-softmmu/target/mips/gdbstub.o
  CC      mips-softmmu/target/mips/lmi_helper.o
  CC      moxie-softmmu/migration/ram.o
  CC      nios2-softmmu/gdbstub.o
  CC      mipsel-softmmu/target/mips/msa_helper.o
  CC      mips-softmmu/target/mips/helper.o
  CC      moxie-softmmu/accel/accel.o
  CC      nios2-softmmu/balloon.o
  CC      mips-softmmu/target/mips/cpu.o
  CC      nios2-softmmu/ioport.o
  CC      mips-softmmu/target/mips/gdbstub.o
  CC      moxie-softmmu/accel/stubs/kvm-stub.o
  CC      nios2-softmmu/numa.o
  CC      mips-softmmu/target/mips/msa_helper.o
  CC      moxie-softmmu/accel/tcg/tcg-all.o
  CC      nios2-softmmu/qtest.o
  CC      moxie-softmmu/accel/tcg/cputlb.o
  CC      nios2-softmmu/memory.o
  CC      moxie-softmmu/accel/tcg/cpu-exec.o
  CC      moxie-softmmu/accel/tcg/cpu-exec-common.o
  CC      mipsel-softmmu/target/mips/mips-semi.o
  CC      moxie-softmmu/accel/tcg/translate-all.o
  CC      nios2-softmmu/memory_mapping.o
  CC      mipsel-softmmu/target/mips/machine.o
  CC      moxie-softmmu/hw/core/generic-loader.o
  CC      nios2-softmmu/dump.o
  CC      mips-softmmu/target/mips/mips-semi.o
  GEN     trace/generated-helpers.c
  CC      mipsel-softmmu/trace/control-target.o
  CC      moxie-softmmu/hw/core/null-machine.o
  CC      nios2-softmmu/migration/ram.o
  CC      moxie-softmmu/hw/display/vga.o
  CC      mips-softmmu/target/mips/machine.o
  CC      mipsel-softmmu/trace/generated-helpers.o
  GEN     trace/generated-helpers.c
  CC      mips-softmmu/trace/control-target.o
  LINK    mipsel-softmmu/qemu-system-mipsel
  CC      mips-softmmu/trace/generated-helpers.o
  CC      nios2-softmmu/accel/accel.o
  CC      nios2-softmmu/accel/stubs/kvm-stub.o
  CC      moxie-softmmu/hw/misc/mmio_interface.o
  LINK    mips-softmmu/qemu-system-mips
  CC      nios2-softmmu/accel/tcg/tcg-all.o
  CC      moxie-softmmu/hw/net/vhost_net.o
  CC      nios2-softmmu/accel/tcg/cputlb.o
  CC      moxie-softmmu/hw/net/rocker/qmp-norocker.o
  CC      moxie-softmmu/hw/timer/mc146818rtc.o
  CC      moxie-softmmu/hw/vfio/common.o
  CC      moxie-softmmu/hw/vfio/platform.o
  CC      moxie-softmmu/hw/vfio/spapr.o
  CC      moxie-softmmu/hw/moxie/moxiesim.o
  CC      moxie-softmmu/target/moxie/translate.o
  CC      nios2-softmmu/accel/tcg/cpu-exec.o
  GEN     or1k-softmmu/hmp-commands.h
  GEN     or1k-softmmu/hmp-commands-info.h
  GEN     or1k-softmmu/config-target.h
  CC      nios2-softmmu/accel/tcg/cpu-exec-common.o
  CC      or1k-softmmu/exec.o
  CC      moxie-softmmu/target/moxie/helper.o
  GEN     ppc64-softmmu/hmp-commands.h
  CC      nios2-softmmu/accel/tcg/translate-all.o
  GEN     ppc64-softmmu/hmp-commands-info.h
  CC      moxie-softmmu/target/moxie/machine.o
  GEN     ppc64-softmmu/config-target.h
  CC      ppc64-softmmu/exec.o
  CC      moxie-softmmu/target/moxie/cpu.o
  CC      nios2-softmmu/hw/core/generic-loader.o
  CC      moxie-softmmu/target/moxie/mmu.o
  CC      nios2-softmmu/hw/core/null-machine.o
  GEN     trace/generated-helpers.c
  CC      moxie-softmmu/trace/control-target.o
  CC      nios2-softmmu/hw/intc/nios2_iic.o
  CC      nios2-softmmu/hw/misc/mmio_interface.o
  CC      moxie-softmmu/trace/generated-helpers.o
  CC      or1k-softmmu/tcg/tcg.o
  CC      nios2-softmmu/hw/net/vhost_net.o
  LINK    moxie-softmmu/qemu-system-moxie
  CC      nios2-softmmu/hw/net/rocker/qmp-norocker.o
  CC      nios2-softmmu/hw/timer/altera_timer.o
  CC      nios2-softmmu/hw/vfio/common.o
  CC      ppc64-softmmu/tcg/tcg.o
  CC      or1k-softmmu/tcg/tcg-op.o
  CC      nios2-softmmu/hw/vfio/platform.o
  CC      nios2-softmmu/hw/vfio/spapr.o
  CC      nios2-softmmu/hw/nios2/boot.o
  GEN     ppcemb-softmmu/hmp-commands.h
  CC      nios2-softmmu/hw/nios2/cpu_pic.o
  GEN     ppcemb-softmmu/hmp-commands-info.h
  GEN     ppcemb-softmmu/config-target.h
  CC      nios2-softmmu/hw/nios2/10m50_devboard.o
  CC      ppcemb-softmmu/exec.o
  CC      nios2-softmmu/target/nios2/translate.o
  CC      ppc64-softmmu/tcg/tcg-op.o
  CC      or1k-softmmu/tcg/optimize.o
  CC      nios2-softmmu/target/nios2/op_helper.o
  CC      nios2-softmmu/target/nios2/helper.o
  CC      or1k-softmmu/tcg/tcg-common.o
  CC      or1k-softmmu/tcg/tcg-runtime.o
  CC      nios2-softmmu/target/nios2/cpu.o
  CC      ppcemb-softmmu/tcg/tcg.o
  CC      nios2-softmmu/target/nios2/mmu.o
  CC      or1k-softmmu/fpu/softfloat.o
  CC      nios2-softmmu/target/nios2/monitor.o
  GEN     trace/generated-helpers.c
  CC      ppc64-softmmu/tcg/optimize.o
  CC      nios2-softmmu/trace/control-target.o
  CC      nios2-softmmu/trace/generated-helpers.o
  CC      ppcemb-softmmu/tcg/tcg-op.o
  CC      ppc64-softmmu/tcg/tcg-common.o
  LINK    nios2-softmmu/qemu-system-nios2
  CC      ppc64-softmmu/tcg/tcg-runtime.o
  CC      ppc64-softmmu/fpu/softfloat.o
  CC      or1k-softmmu/disas.o
  CC      ppcemb-softmmu/tcg/optimize.o
  CC      or1k-softmmu/hax-stub.o
  CC      or1k-softmmu/arch_init.o
  CC      ppc64-softmmu/disas.o
  CC      or1k-softmmu/cpus.o
  CC      ppcemb-softmmu/tcg/tcg-common.o
  CC      ppcemb-softmmu/tcg/tcg-runtime.o
  CC      or1k-softmmu/monitor.o
  CC      ppcemb-softmmu/fpu/softfloat.o
  GEN     ppc64-softmmu/gdbstub-xml.c
  CC      ppcemb-softmmu/disas.o
  CC      or1k-softmmu/gdbstub.o
  CC      ppc64-softmmu/hax-stub.o
  CC      or1k-softmmu/balloon.o
  CC      or1k-softmmu/ioport.o
  GEN     ppcemb-softmmu/gdbstub-xml.c
  CC      ppc64-softmmu/libdecnumber/decContext.o
  CC      or1k-softmmu/numa.o
  CC      ppc64-softmmu/libdecnumber/decNumber.o
  CC      ppcemb-softmmu/hax-stub.o
  CC      or1k-softmmu/memory.o
  CC      or1k-softmmu/qtest.o
  CC      ppcemb-softmmu/libdecnumber/decContext.o
  CC      or1k-softmmu/memory_mapping.o
  CC      ppcemb-softmmu/libdecnumber/decNumber.o
  CC      or1k-softmmu/dump.o
  CC      ppc64-softmmu/libdecnumber/dpd/decimal32.o
  CC      ppc64-softmmu/libdecnumber/dpd/decimal64.o
  CC      or1k-softmmu/migration/ram.o
  GEN     ppc-softmmu/hmp-commands.h
  CC      ppc64-softmmu/libdecnumber/dpd/decimal128.o
  GEN     ppc-softmmu/hmp-commands-info.h
  GEN     ppc-softmmu/config-target.h
  CC      ppc-softmmu/exec.o
  CC      ppc64-softmmu/arch_init.o
  CC      ppcemb-softmmu/libdecnumber/dpd/decimal32.o
  CC      ppc64-softmmu/cpus.o
  CC      or1k-softmmu/accel/accel.o
  CC      ppcemb-softmmu/libdecnumber/dpd/decimal64.o
  CC      ppcemb-softmmu/libdecnumber/dpd/decimal128.o
  CC      ppc64-softmmu/monitor.o
  CC      or1k-softmmu/accel/stubs/kvm-stub.o
  CC      ppcemb-softmmu/arch_init.o
  CC      or1k-softmmu/accel/tcg/tcg-all.o
  CC      ppcemb-softmmu/cpus.o
  CC      or1k-softmmu/accel/tcg/cputlb.o
  CC      ppc-softmmu/tcg/tcg.o
  CC      ppcemb-softmmu/monitor.o
  CC      ppc64-softmmu/gdbstub.o
  CC      or1k-softmmu/accel/tcg/cpu-exec.o
  CC      ppc64-softmmu/balloon.o
  CC      ppcemb-softmmu/gdbstub.o
  CC      ppc-softmmu/tcg/tcg-op.o
  CC      ppc64-softmmu/ioport.o
  CC      or1k-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppc64-softmmu/numa.o
  CC      ppcemb-softmmu/balloon.o
  CC      or1k-softmmu/accel/tcg/translate-all.o
  CC      ppcemb-softmmu/ioport.o
  CC      ppc64-softmmu/qtest.o
  CC      or1k-softmmu/hw/core/generic-loader.o
  CC      ppcemb-softmmu/numa.o
  CC      or1k-softmmu/hw/core/null-machine.o
  CC      ppc64-softmmu/memory.o
  CC      or1k-softmmu/hw/misc/mmio_interface.o
  CC      ppcemb-softmmu/qtest.o
  CC      or1k-softmmu/hw/net/vhost_net.o
  CC      ppcemb-softmmu/memory.o
  CC      ppc-softmmu/tcg/optimize.o
  CC      or1k-softmmu/hw/net/rocker/qmp-norocker.o
  CC      ppc64-softmmu/memory_mapping.o
  CC      or1k-softmmu/hw/vfio/common.o
  CC      ppc64-softmmu/dump.o
  CC      ppc64-softmmu/migration/ram.o
  CC      ppcemb-softmmu/memory_mapping.o
  CC      or1k-softmmu/hw/vfio/platform.o
  CC      ppc-softmmu/tcg/tcg-common.o
  CC      ppcemb-softmmu/dump.o
  CC      ppc-softmmu/tcg/tcg-runtime.o
  CC      or1k-softmmu/hw/vfio/spapr.o
  CC      ppc-softmmu/fpu/softfloat.o
  CC      ppc64-softmmu/accel/accel.o
  CC      or1k-softmmu/hw/openrisc/pic_cpu.o
  CC      ppcemb-softmmu/migration/ram.o
  CC      ppc64-softmmu/accel/stubs/kvm-stub.o
  CC      or1k-softmmu/hw/openrisc/cputimer.o
  CC      ppc64-softmmu/accel/tcg/tcg-all.o
  CC      or1k-softmmu/hw/openrisc/openrisc_sim.o
  CC      ppc64-softmmu/accel/tcg/cputlb.o
  CC      ppcemb-softmmu/accel/accel.o
  CC      or1k-softmmu/target/openrisc/machine.o
  CC      ppcemb-softmmu/accel/stubs/kvm-stub.o
  CC      or1k-softmmu/target/openrisc/cpu.o
  CC      or1k-softmmu/target/openrisc/exception.o
  CC      ppcemb-softmmu/accel/tcg/tcg-all.o
  CC      or1k-softmmu/target/openrisc/interrupt.o
  CC      ppcemb-softmmu/accel/tcg/cputlb.o
  CC      or1k-softmmu/target/openrisc/mmu.o
  CC      ppc64-softmmu/accel/tcg/cpu-exec.o
  CC      ppc-softmmu/disas.o
  CC      or1k-softmmu/target/openrisc/translate.o
  CC      ppc64-softmmu/accel/tcg/cpu-exec-common.o
  GEN     ppc-softmmu/gdbstub-xml.c
  CC      ppc64-softmmu/accel/tcg/translate-all.o
  CC      ppcemb-softmmu/accel/tcg/cpu-exec.o
  CC      or1k-softmmu/target/openrisc/exception_helper.o
  CC      or1k-softmmu/target/openrisc/fpu_helper.o
  CC      ppc-softmmu/hax-stub.o
  CC      ppcemb-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppc64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppcemb-softmmu/accel/tcg/translate-all.o
  CC      ppc-softmmu/libdecnumber/decContext.o
  CC      ppc64-softmmu/hw/block/virtio-blk.o
  CC      or1k-softmmu/target/openrisc/interrupt_helper.o
  CC      ppc-softmmu/libdecnumber/decNumber.o
  CC      or1k-softmmu/target/openrisc/mmu_helper.o
  CC      ppcemb-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppc64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      or1k-softmmu/target/openrisc/sys_helper.o
  CC      ppcemb-softmmu/hw/block/virtio-blk.o
  CC      or1k-softmmu/target/openrisc/gdbstub.o
  CC      ppc64-softmmu/hw/char/spapr_vty.o
  GEN     trace/generated-helpers.c
  CC      or1k-softmmu/trace/control-target.o
  CC      ppcemb-softmmu/hw/block/dataplane/virtio-blk.o
  CC      ppc64-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc-softmmu/libdecnumber/dpd/decimal32.o
  CC      or1k-softmmu/trace/generated-helpers.o
  CC      ppcemb-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc64-softmmu/hw/core/generic-loader.o
  CC      ppc-softmmu/libdecnumber/dpd/decimal64.o
  LINK    or1k-softmmu/qemu-system-or1k
  CC      ppc64-softmmu/hw/core/null-machine.o
  CC      ppc-softmmu/libdecnumber/dpd/decimal128.o
  CC      ppcemb-softmmu/hw/core/generic-loader.o
  CC      ppc64-softmmu/hw/display/sm501.o
  CC      ppc-softmmu/arch_init.o
  CC      ppcemb-softmmu/hw/core/null-machine.o
  CC      ppcemb-softmmu/hw/display/sm501.o
  CC      ppc-softmmu/cpus.o
  CC      ppc64-softmmu/hw/display/vga.o
  CC      ppc-softmmu/monitor.o
  CC      ppcemb-softmmu/hw/display/vga.o
  CC      ppc64-softmmu/hw/display/virtio-gpu.o
  CC      ppc-softmmu/gdbstub.o
  GEN     s390x-softmmu/hmp-commands.h
  GEN     s390x-softmmu/hmp-commands-info.h
  CC      s390x-softmmu/gen-features
  GEN     s390x-softmmu/config-target.h
  GEN     s390x-softmmu/gen-features.h
  CC      ppcemb-softmmu/hw/display/virtio-gpu.o
  CC      s390x-softmmu/exec.o
  CC      ppc64-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppc-softmmu/balloon.o
  CC      ppc64-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppc-softmmu/ioport.o
  CC      ppcemb-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppc64-softmmu/hw/display/virtio-vga.o
  CC      ppc-softmmu/numa.o
  CC      ppc64-softmmu/hw/intc/xics.o
  CC      ppcemb-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppc-softmmu/qtest.o
  CC      ppcemb-softmmu/hw/misc/ivshmem.o
  CC      ppc64-softmmu/hw/intc/xics_spapr.o
  CC      s390x-softmmu/tcg/tcg.o
  CC      ppc-softmmu/memory.o
  CC      ppcemb-softmmu/hw/misc/mmio_interface.o
  CC      ppc64-softmmu/hw/intc/xics_pnv.o
  CC      ppcemb-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc64-softmmu/hw/misc/ivshmem.o
  CC      ppcemb-softmmu/hw/net/virtio-net.o
  CC      ppc-softmmu/memory_mapping.o
  CC      ppc64-softmmu/hw/misc/mmio_interface.o
  CC      s390x-softmmu/tcg/tcg-op.o
  CC      ppc-softmmu/dump.o
  CC      ppc64-softmmu/hw/net/spapr_llan.o
  CC      ppcemb-softmmu/hw/net/vhost_net.o
  CC      ppc64-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc-softmmu/migration/ram.o
  CC      ppcemb-softmmu/hw/scsi/virtio-scsi.o
  CC      ppc64-softmmu/hw/net/virtio-net.o
  CC      ppcemb-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      ppc64-softmmu/hw/net/vhost_net.o
  CC      ppcemb-softmmu/hw/scsi/vhost-scsi-common.o
  CC      ppc-softmmu/accel/accel.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/etsec.o
  CC      ppcemb-softmmu/hw/scsi/vhost-scsi.o
  CC      ppc-softmmu/accel/stubs/kvm-stub.o
  CC      s390x-softmmu/tcg/optimize.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/registers.o
  CC      ppcemb-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc-softmmu/accel/tcg/tcg-all.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/rings.o
  CC      ppcemb-softmmu/hw/vfio/common.o
  CC      ppc-softmmu/accel/tcg/cputlb.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/miim.o
  CC      ppcemb-softmmu/hw/vfio/pci.o
  CC      s390x-softmmu/tcg/tcg-common.o
  CC      ppc64-softmmu/hw/nvram/spapr_nvram.o
  CC      s390x-softmmu/tcg/tcg-runtime.o
  CC      ppc64-softmmu/hw/scsi/spapr_vscsi.o
  CC      s390x-softmmu/fpu/softfloat.o
  CC      ppcemb-softmmu/hw/vfio/pci-quirks.o
  CC      ppc-softmmu/accel/tcg/cpu-exec.o
  CC      ppc64-softmmu/hw/scsi/virtio-scsi.o
  CC      ppc-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppc64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      ppcemb-softmmu/hw/vfio/platform.o
  CC      ppc-softmmu/accel/tcg/translate-all.o
  CC      ppc64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      ppcemb-softmmu/hw/vfio/spapr.o
  CC      ppc64-softmmu/hw/scsi/vhost-scsi.o
  CC      ppc-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppcemb-softmmu/hw/virtio/virtio.o
  CC      ppc64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc-softmmu/hw/block/virtio-blk.o
  CC      ppc64-softmmu/hw/timer/mc146818rtc.o
  CC      s390x-softmmu/disas.o
  CC      ppcemb-softmmu/hw/virtio/virtio-balloon.o
  CC      ppc-softmmu/hw/block/dataplane/virtio-blk.o
  CC      ppc64-softmmu/hw/vfio/common.o
  GEN     s390x-softmmu/gdbstub-xml.c
  CC      ppcemb-softmmu/hw/virtio/vhost.o
  CC      ppc-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc64-softmmu/hw/vfio/pci.o
  CC      s390x-softmmu/hax-stub.o
  CC      ppcemb-softmmu/hw/virtio/vhost-backend.o
  CC      ppc-softmmu/hw/core/generic-loader.o
  CC      s390x-softmmu/arch_init.o
  CC      ppc-softmmu/hw/core/null-machine.o
  CC      ppcemb-softmmu/hw/virtio/vhost-user.o
  CC      s390x-softmmu/cpus.o
  CC      ppc-softmmu/hw/display/sm501.o
  CC      ppcemb-softmmu/hw/virtio/vhost-vsock.o
  CC      ppc64-softmmu/hw/vfio/pci-quirks.o
  CC      ppcemb-softmmu/hw/virtio/virtio-crypto.o
  CC      ppc-softmmu/hw/display/vga.o
  CC      s390x-softmmu/monitor.o
  CC      ppc64-softmmu/hw/vfio/platform.o
  CC      ppcemb-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      ppc64-softmmu/hw/vfio/spapr.o
  CC      ppcemb-softmmu/hw/ppc/ppc.o
  CC      ppc64-softmmu/hw/virtio/virtio.o
  CC      ppc-softmmu/hw/display/virtio-gpu.o
  CC      ppcemb-softmmu/hw/ppc/ppc_booke.o
  CC      s390x-softmmu/gdbstub.o
  CC      ppcemb-softmmu/hw/ppc/fdt.o
  CC      ppc-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppcemb-softmmu/hw/ppc/ppc405_boards.o
  CC      ppc64-softmmu/hw/virtio/virtio-balloon.o
  CC      s390x-softmmu/balloon.o
  CC      ppcemb-softmmu/hw/ppc/ppc4xx_devs.o
  CC      s390x-softmmu/ioport.o
  CC      ppc64-softmmu/hw/virtio/vhost.o
  CC      ppc-softmmu/hw/display/virtio-gpu-pci.o
  CC      s390x-softmmu/numa.o
  CC      ppcemb-softmmu/hw/ppc/ppc405_uc.o
  CC      ppc-softmmu/hw/misc/ivshmem.o
  CC      ppc64-softmmu/hw/virtio/vhost-backend.o
  CC      s390x-softmmu/qtest.o
  CC      ppcemb-softmmu/hw/ppc/ppc440_bamboo.o
  CC      ppc64-softmmu/hw/virtio/vhost-user.o
  CC      ppc-softmmu/hw/misc/mmio_interface.o
  CC      s390x-softmmu/memory.o
  CC      ppcemb-softmmu/hw/ppc/ppc4xx_pci.o
  CC      ppc-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc64-softmmu/hw/virtio/vhost-vsock.o
  CC      ppcemb-softmmu/hw/ppc/virtex_ml507.o
  CC      ppc-softmmu/hw/net/virtio-net.o
  CC      ppc64-softmmu/hw/virtio/virtio-crypto.o
  CC      ppcemb-softmmu/target/ppc/cpu-models.o
  CC      ppc64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      s390x-softmmu/memory_mapping.o
  CC      s390x-softmmu/dump.o
  CC      ppc-softmmu/hw/net/vhost_net.o
  CC      ppc64-softmmu/hw/ppc/ppc.o
  CC      ppcemb-softmmu/target/ppc/cpu.o
  CC      ppc-softmmu/hw/net/fsl_etsec/etsec.o
  CC      ppcemb-softmmu/target/ppc/translate.o
  CC      s390x-softmmu/migration/ram.o
  CC      ppc64-softmmu/hw/ppc/ppc_booke.o
  CC      ppc-softmmu/hw/net/fsl_etsec/registers.o
  CC      ppc-softmmu/hw/net/fsl_etsec/rings.o
  CC      ppc64-softmmu/hw/ppc/fdt.o
  CC      ppc-softmmu/hw/net/fsl_etsec/miim.o
  CC      ppc64-softmmu/hw/ppc/spapr.o
  CC      s390x-softmmu/accel/accel.o
  CC      ppc-softmmu/hw/scsi/virtio-scsi.o
  CC      s390x-softmmu/accel/kvm/kvm-all.o
  CC      ppc64-softmmu/hw/ppc/spapr_vio.o
  CC      ppc-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      s390x-softmmu/accel/tcg/tcg-all.o
  CC      ppc64-softmmu/hw/ppc/spapr_events.o
  CC      ppc-softmmu/hw/scsi/vhost-scsi-common.o
  CC      s390x-softmmu/accel/tcg/cputlb.o
  CC      ppc-softmmu/hw/scsi/vhost-scsi.o
  CC      ppc64-softmmu/hw/ppc/spapr_hcall.o
  CC      ppc-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc64-softmmu/hw/ppc/spapr_iommu.o
  CC      ppc-softmmu/hw/timer/mc146818rtc.o
  CC      ppc64-softmmu/hw/ppc/spapr_rtas.o
  CC      s390x-softmmu/accel/tcg/cpu-exec.o
  CC      ppc-softmmu/hw/vfio/common.o
  CC      ppc64-softmmu/hw/ppc/spapr_pci.o
  CC      s390x-softmmu/accel/tcg/cpu-exec-common.o
  CC      s390x-softmmu/accel/tcg/translate-all.o
  CC      ppc-softmmu/hw/vfio/pci.o
  CC      ppc64-softmmu/hw/ppc/spapr_rtc.o
  CC      s390x-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppc64-softmmu/hw/ppc/spapr_drc.o
  CC      ppc-softmmu/hw/vfio/pci-quirks.o
  CC      s390x-softmmu/hw/block/virtio-blk.o
  CC      ppc-softmmu/hw/vfio/platform.o
  CC      ppc64-softmmu/hw/ppc/spapr_rng.o
  CC      s390x-softmmu/hw/block/dataplane/virtio-blk.o
  CC      ppc64-softmmu/hw/ppc/spapr_cpu_core.o
  CC      s390x-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc-softmmu/hw/vfio/spapr.o
  CC      ppc-softmmu/hw/virtio/virtio.o
  CC      ppc64-softmmu/hw/ppc/spapr_ovec.o
  CC      s390x-softmmu/hw/char/terminal3270.o
  CC      ppc64-softmmu/hw/ppc/pnv.o
  CC      s390x-softmmu/hw/core/generic-loader.o
  CC      ppc-softmmu/hw/virtio/virtio-balloon.o
  CC      s390x-softmmu/hw/core/null-machine.o
  CC      ppc64-softmmu/hw/ppc/pnv_xscom.o
  CC      s390x-softmmu/hw/display/virtio-gpu.o
  CC      ppc64-softmmu/hw/ppc/pnv_core.o
  CC      ppc-softmmu/hw/virtio/vhost.o
  CC      ppc64-softmmu/hw/ppc/pnv_lpc.o
  CC      s390x-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppc64-softmmu/hw/ppc/pnv_psi.o
  CC      ppc-softmmu/hw/virtio/vhost-backend.o
  CC      ppc64-softmmu/hw/ppc/pnv_occ.o
  CC      s390x-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppc-softmmu/hw/virtio/vhost-user.o
  CC      ppc64-softmmu/hw/ppc/pnv_bmc.o
  CC      ppc64-softmmu/hw/ppc/spapr_pci_vfio.o
  CC      ppc-softmmu/hw/virtio/vhost-vsock.o
  CC      s390x-softmmu/hw/intc/s390_flic.o
  CC      ppc64-softmmu/hw/ppc/spapr_rtas_ddw.o
  CC      ppc-softmmu/hw/virtio/virtio-crypto.o
  CC      s390x-softmmu/hw/intc/s390_flic_kvm.o
  CC      ppc64-softmmu/hw/ppc/ppc405_boards.o
  CC      ppcemb-softmmu/target/ppc/machine.o
  CC      ppc-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      s390x-softmmu/hw/misc/mmio_interface.o
  CC      ppc64-softmmu/hw/ppc/ppc4xx_devs.o
  CC      ppcemb-softmmu/target/ppc/mmu_helper.o
  CC      s390x-softmmu/hw/net/virtio-net.o
  CC      ppc-softmmu/hw/ppc/ppc.o
  CC      ppc64-softmmu/hw/ppc/ppc405_uc.o
  CC      ppc-softmmu/hw/ppc/ppc_booke.o
  CC      s390x-softmmu/hw/net/vhost_net.o
  CC      ppc-softmmu/hw/ppc/fdt.o
  CC      ppcemb-softmmu/target/ppc/mmu-hash32.o
  CC      ppc64-softmmu/hw/ppc/ppc440_bamboo.o
  CC      s390x-softmmu/hw/net/rocker/qmp-norocker.o
  CC      ppc-softmmu/hw/ppc/ppc405_boards.o
  CC      ppcemb-softmmu/target/ppc/monitor.o
  CC      s390x-softmmu/hw/scsi/virtio-scsi.o
  CC      ppc-softmmu/hw/ppc/ppc4xx_devs.o
  CC      ppc64-softmmu/hw/ppc/ppc4xx_pci.o
  CC      ppcemb-softmmu/target/ppc/arch_dump.o
  CC      ppc64-softmmu/hw/ppc/prep.o
  CC      s390x-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      ppcemb-softmmu/target/ppc/kvm-stub.o
  CC      ppc-softmmu/hw/ppc/ppc405_uc.o
  CC      ppcemb-softmmu/target/ppc/dfp_helper.o
  CC      s390x-softmmu/hw/scsi/vhost-scsi-common.o
  CC      ppc64-softmmu/hw/ppc/prep_systemio.o
  CC      s390x-softmmu/hw/scsi/vhost-scsi.o
  CC      ppc-softmmu/hw/ppc/ppc440_bamboo.o
  CC      s390x-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc64-softmmu/hw/ppc/rs6000_mc.o
  CC      ppcemb-softmmu/target/ppc/excp_helper.o
  CC      s390x-softmmu/hw/vfio/common.o
  CC      ppc-softmmu/hw/ppc/ppc4xx_pci.o
  CC      ppc64-softmmu/hw/ppc/mac_oldworld.o
  CC      ppc64-softmmu/hw/ppc/mac_newworld.o
  CC      ppc-softmmu/hw/ppc/prep.o
  CC      ppcemb-softmmu/target/ppc/fpu_helper.o
  CC      s390x-softmmu/hw/vfio/pci.o
  CC      ppc64-softmmu/hw/ppc/e500.o
  CC      ppc-softmmu/hw/ppc/prep_systemio.o
  CC      ppc64-softmmu/hw/ppc/mpc8544ds.o
  CC      ppc-softmmu/hw/ppc/rs6000_mc.o
  CC      ppc64-softmmu/hw/ppc/e500plat.o
  CC      s390x-softmmu/hw/vfio/pci-quirks.o
  CC      ppc-softmmu/hw/ppc/mac_oldworld.o
  CC      ppc64-softmmu/hw/ppc/mpc8544_guts.o
  CC      ppc64-softmmu/hw/ppc/ppce500_spin.o
  CC      ppc-softmmu/hw/ppc/mac_newworld.o
  CC      s390x-softmmu/hw/vfio/ccw.o
  CC      ppc64-softmmu/hw/ppc/virtex_ml507.o
  CC      ppc-softmmu/hw/ppc/e500.o
  CC      s390x-softmmu/hw/vfio/platform.o
  CC      ppc64-softmmu/target/ppc/cpu-models.o
  CC      ppc-softmmu/hw/ppc/mpc8544ds.o
  CC      s390x-softmmu/hw/vfio/spapr.o
  CC      ppc-softmmu/hw/ppc/e500plat.o
  CC      s390x-softmmu/hw/virtio/virtio.o
  CC      ppc-softmmu/hw/ppc/mpc8544_guts.o
  CC      ppcemb-softmmu/target/ppc/int_helper.o
  CC      ppc-softmmu/hw/ppc/ppce500_spin.o
  CC      ppc64-softmmu/target/ppc/cpu.o
  CC      ppc-softmmu/hw/ppc/virtex_ml507.o
  CC      ppc-softmmu/target/ppc/cpu-models.o
  CC      s390x-softmmu/hw/virtio/virtio-balloon.o
  CC      ppc64-softmmu/target/ppc/translate.o
  CC      s390x-softmmu/hw/virtio/vhost.o
  CC      ppcemb-softmmu/target/ppc/timebase_helper.o
  CC      s390x-softmmu/hw/virtio/vhost-backend.o
  CC      ppc-softmmu/target/ppc/cpu.o
  CC      ppcemb-softmmu/target/ppc/misc_helper.o
  CC      s390x-softmmu/hw/virtio/vhost-user.o
  CC      ppc-softmmu/target/ppc/translate.o
  CC      s390x-softmmu/hw/virtio/vhost-vsock.o
  CC      ppcemb-softmmu/target/ppc/mem_helper.o
  CC      s390x-softmmu/hw/virtio/virtio-crypto.o
  CC      ppcemb-softmmu/target/ppc/gdbstub.o
  CC      s390x-softmmu/hw/virtio/virtio-crypto-pci.o
  GEN     trace/generated-helpers.c
  CC      ppcemb-softmmu/trace/control-target.o
  CC      s390x-softmmu/hw/s390x/s390-virtio.o
  CC      ppcemb-softmmu/gdbstub-xml.o
  CC      s390x-softmmu/hw/s390x/s390-virtio-hcall.o
  CC      ppcemb-softmmu/trace/generated-helpers.o
  CC      s390x-softmmu/hw/s390x/sclp.o
  LINK    ppcemb-softmmu/qemu-system-ppcemb
  CC      s390x-softmmu/hw/s390x/event-facility.o
  CC      s390x-softmmu/hw/s390x/sclpquiesce.o
  CC      s390x-softmmu/hw/s390x/sclpcpu.o
  CC      ppc-softmmu/target/ppc/machine.o
  CC      s390x-softmmu/hw/s390x/ipl.o
  CC      ppc64-softmmu/target/ppc/machine.o
  CC      s390x-softmmu/hw/s390x/css.o
  CC      ppc64-softmmu/target/ppc/mmu_helper.o
  CC      s390x-softmmu/hw/s390x/s390-virtio-ccw.o
  CC      s390x-softmmu/hw/s390x/3270-ccw.o
  GEN     sh4eb-softmmu/hmp-commands.h
  GEN     sh4eb-softmmu/hmp-commands-info.h
  GEN     sh4eb-softmmu/config-target.h
  CC      sh4eb-softmmu/exec.o
  CC      s390x-softmmu/hw/s390x/virtio-ccw.o
  CC      s390x-softmmu/hw/s390x/css-bridge.o
  CC      s390x-softmmu/hw/s390x/ccw-device.o
  CC      sh4eb-softmmu/tcg/tcg.o
  CC      s390x-softmmu/hw/s390x/s390-pci-bus.o
  CC      s390x-softmmu/hw/s390x/s390-pci-inst.o
  CC      sh4eb-softmmu/tcg/tcg-op.o
  CC      s390x-softmmu/hw/s390x/s390-skeys.o
  CC      ppc-softmmu/target/ppc/mmu_helper.o
  CC      s390x-softmmu/hw/s390x/s390-stattrib.o
  CC      s390x-softmmu/hw/s390x/s390-skeys-kvm.o
  CC      ppc64-softmmu/target/ppc/mmu-hash32.o
  CC      s390x-softmmu/hw/s390x/s390-stattrib-kvm.o
  CC      ppc-softmmu/target/ppc/mmu-hash32.o
  CC      ppc64-softmmu/target/ppc/monitor.o
  CC      s390x-softmmu/hw/s390x/s390-ccw.o
  CC      ppc-softmmu/target/ppc/monitor.o
  CC      ppc64-softmmu/target/ppc/arch_dump.o
  CC      ppc-softmmu/target/ppc/arch_dump.o
  CC      s390x-softmmu/target/s390x/cpu.o
  CC      ppc64-softmmu/target/ppc/mmu-hash64.o
  CC      sh4eb-softmmu/tcg/optimize.o
  CC      ppc-softmmu/target/ppc/kvm-stub.o
  CC      s390x-softmmu/target/s390x/cpu_models.o
  CC      ppc64-softmmu/target/ppc/mmu-book3s-v3.o
  CC      ppc-softmmu/target/ppc/dfp_helper.o
  CC      ppc64-softmmu/target/ppc/compat.o
  CC      s390x-softmmu/target/s390x/cpu_features.o
  CC      ppc64-softmmu/target/ppc/mmu-radix64.o
  CC      sh4eb-softmmu/tcg/tcg-common.o
  CC      s390x-softmmu/target/s390x/gdbstub.o
  CC      sh4eb-softmmu/tcg/tcg-runtime.o
  CC      ppc-softmmu/target/ppc/excp_helper.o
  CC      ppc64-softmmu/target/ppc/kvm-stub.o
  CC      sh4eb-softmmu/fpu/softfloat.o
  CC      s390x-softmmu/target/s390x/interrupt.o
  CC      ppc64-softmmu/target/ppc/dfp_helper.o
  CC      ppc-softmmu/target/ppc/fpu_helper.o
  CC      s390x-softmmu/target/s390x/helper.o
  CC      ppc64-softmmu/target/ppc/excp_helper.o
  CC      s390x-softmmu/target/s390x/translate.o
  CC      ppc64-softmmu/target/ppc/fpu_helper.o
  CC      sh4eb-softmmu/disas.o
  CC      sh4eb-softmmu/hax-stub.o
  CC      sh4eb-softmmu/arch_init.o
  CC      s390x-softmmu/target/s390x/cc_helper.o
  CC      ppc-softmmu/target/ppc/int_helper.o
  CC      sh4eb-softmmu/cpus.o
  CC      s390x-softmmu/target/s390x/excp_helper.o
  CC      s390x-softmmu/target/s390x/fpu_helper.o
  CC      sh4eb-softmmu/monitor.o
  CC      s390x-softmmu/target/s390x/int_helper.o
  CC      ppc64-softmmu/target/ppc/int_helper.o
  CC      s390x-softmmu/target/s390x/mem_helper.o
  CC      ppc-softmmu/target/ppc/timebase_helper.o
  CC      ppc-softmmu/target/ppc/misc_helper.o
  CC      sh4eb-softmmu/gdbstub.o
  CC      ppc-softmmu/target/ppc/mem_helper.o
  CC      sh4eb-softmmu/balloon.o
  CC      ppc64-softmmu/target/ppc/timebase_helper.o
  CC      ppc-softmmu/target/ppc/gdbstub.o
  CC      sh4eb-softmmu/ioport.o
  CC      ppc64-softmmu/target/ppc/misc_helper.o
  GEN     trace/generated-helpers.c
  CC      ppc-softmmu/trace/control-target.o
  CC      sh4eb-softmmu/numa.o
  CC      ppc64-softmmu/target/ppc/mem_helper.o
  CC      s390x-softmmu/target/s390x/misc_helper.o
  CC      ppc-softmmu/gdbstub-xml.o
  CC      sh4eb-softmmu/qtest.o
  CC      s390x-softmmu/target/s390x/machine.o
  CC      ppc-softmmu/trace/generated-helpers.o
  CC      ppc64-softmmu/target/ppc/gdbstub.o
  CC      sh4eb-softmmu/memory.o
  CC      s390x-softmmu/target/s390x/ioinst.o
  GEN     trace/generated-helpers.c
  LINK    ppc-softmmu/qemu-system-ppc
  CC      ppc64-softmmu/trace/control-target.o
  CC      s390x-softmmu/target/s390x/arch_dump.o
  CC      ppc64-softmmu/gdbstub-xml.o
  CC      ppc64-softmmu/trace/generated-helpers.o
  CC      s390x-softmmu/target/s390x/mmu_helper.o
  CC      sh4eb-softmmu/memory_mapping.o
  LINK    ppc64-softmmu/qemu-system-ppc64
  CC      sh4eb-softmmu/dump.o
  CC      s390x-softmmu/target/s390x/diag.o
  CC      s390x-softmmu/target/s390x/kvm.o
  CC      sh4eb-softmmu/migration/ram.o
  GEN     trace/generated-helpers.c
  CC      s390x-softmmu/trace/control-target.o
  CC      s390x-softmmu/gdbstub-xml.o
  CC      sh4eb-softmmu/accel/accel.o
  CC      s390x-softmmu/trace/generated-helpers.o
  CC      sh4eb-softmmu/accel/stubs/kvm-stub.o
  CC      sh4eb-softmmu/accel/tcg/tcg-all.o
  CC      sh4eb-softmmu/accel/tcg/cputlb.o
  CC      sh4eb-softmmu/accel/tcg/cpu-exec.o
  LINK    s390x-softmmu/qemu-system-s390x
  CC      sh4eb-softmmu/accel/tcg/cpu-exec-common.o
  CC      sh4eb-softmmu/accel/tcg/translate-all.o
  CC      sh4eb-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sh4eb-softmmu/hw/block/tc58128.o
  CC      sh4eb-softmmu/hw/block/virtio-blk.o
  CC      sh4eb-softmmu/hw/block/dataplane/virtio-blk.o
  GEN     sh4-softmmu/hmp-commands.h
  GEN     sh4-softmmu/hmp-commands-info.h
  GEN     sh4-softmmu/config-target.h
  CC      sh4eb-softmmu/hw/char/sh_serial.o
  GEN     sparc64-softmmu/hmp-commands.h
  GEN     sparc64-softmmu/hmp-commands-info.h
  GEN     sparc64-softmmu/config-target.h
  CC      sparc64-softmmu/exec.o
  CC      sh4-softmmu/exec.o
  CC      sparc64-softmmu/tcg/tcg.o
  CC      sh4eb-softmmu/hw/char/virtio-serial-bus.o
  CC      sh4eb-softmmu/hw/core/generic-loader.o
  CC      sh4eb-softmmu/hw/core/null-machine.o
  CC      sh4eb-softmmu/hw/display/sm501.o
  CC      sh4-softmmu/tcg/tcg.o
  GEN     sparc-softmmu/hmp-commands.h
  GEN     sparc-softmmu/hmp-commands-info.h
  GEN     sparc-softmmu/config-target.h
  CC      sparc-softmmu/exec.o
  CC      sparc64-softmmu/tcg/tcg-op.o
  CC      sh4eb-softmmu/hw/display/vga.o
  CC      sh4-softmmu/tcg/tcg-op.o
  CC      sh4eb-softmmu/hw/display/virtio-gpu.o
  CC      sparc-softmmu/tcg/tcg.o
  CC      sh4eb-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc64-softmmu/tcg/optimize.o
  CC      sparc-softmmu/tcg/tcg-op.o
  CC      sh4eb-softmmu/hw/display/virtio-gpu-pci.o
  CC      sparc64-softmmu/tcg/tcg-common.o
  CC      sh4-softmmu/tcg/optimize.o
  CC      sh4eb-softmmu/hw/intc/sh_intc.o
  CC      sparc64-softmmu/tcg/tcg-runtime.o
  CC      sh4eb-softmmu/hw/misc/ivshmem.o
  CC      sh4-softmmu/tcg/tcg-common.o
  CC      sparc64-softmmu/fpu/softfloat.o
  CC      sh4eb-softmmu/hw/misc/mmio_interface.o
  CC      sh4-softmmu/tcg/tcg-runtime.o
  CC      sh4eb-softmmu/hw/net/virtio-net.o
  CC      sh4-softmmu/fpu/softfloat.o
  CC      sparc-softmmu/tcg/optimize.o
  CC      sh4eb-softmmu/hw/net/vhost_net.o
  CC      sparc-softmmu/tcg/tcg-common.o
  CC      sh4eb-softmmu/hw/scsi/virtio-scsi.o
  CC      sparc64-softmmu/disas.o
  CC      sparc-softmmu/tcg/tcg-runtime.o
  CC      sh4eb-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      sparc64-softmmu/hax-stub.o
  CC      sparc-softmmu/fpu/softfloat.o
  CC      sparc64-softmmu/arch_init.o
  CC      sh4eb-softmmu/hw/scsi/vhost-scsi-common.o
  CC      sh4-softmmu/disas.o
  CC      sparc64-softmmu/cpus.o
  CC      sh4eb-softmmu/hw/scsi/vhost-scsi.o
  CC      sh4-softmmu/hax-stub.o
  CC      sparc64-softmmu/monitor.o
  CC      sh4-softmmu/arch_init.o
  CC      sh4eb-softmmu/hw/scsi/vhost-user-scsi.o
  CC      sh4-softmmu/cpus.o
  CC      sh4eb-softmmu/hw/timer/sh_timer.o
  CC      sh4-softmmu/monitor.o
  CC      sh4eb-softmmu/hw/timer/mc146818rtc.o
  CC      sparc64-softmmu/gdbstub.o
  CC      sparc-softmmu/disas.o
  CC      sh4eb-softmmu/hw/vfio/common.o
  CC      sparc-softmmu/hax-stub.o
  CC      sparc64-softmmu/balloon.o
  CC      sh4-softmmu/gdbstub.o
  CC      sparc-softmmu/arch_init.o
  CC      sh4eb-softmmu/hw/vfio/pci.o
  CC      sparc64-softmmu/ioport.o
  CC      sparc-softmmu/cpus.o
  CC      sh4-softmmu/balloon.o
  CC      sparc64-softmmu/numa.o
  CC      sparc-softmmu/monitor.o
  CC      sh4-softmmu/ioport.o
  CC      sh4eb-softmmu/hw/vfio/pci-quirks.o
  CC      sparc64-softmmu/qtest.o
  CC      sh4-softmmu/numa.o
  CC      sh4eb-softmmu/hw/vfio/platform.o
  CC      sparc64-softmmu/memory.o
  CC      sh4-softmmu/qtest.o
  CC      sh4eb-softmmu/hw/vfio/spapr.o
  CC      sh4eb-softmmu/hw/virtio/virtio.o
  CC      sparc-softmmu/gdbstub.o
  CC      sh4-softmmu/memory.o
  CC      sparc64-softmmu/memory_mapping.o
  CC      sh4eb-softmmu/hw/virtio/virtio-balloon.o
  CC      sparc-softmmu/balloon.o
  CC      sparc64-softmmu/dump.o
  CC      sparc-softmmu/ioport.o
  CC      sh4eb-softmmu/hw/virtio/vhost.o
  CC      sparc-softmmu/numa.o
  CC      sh4-softmmu/memory_mapping.o
  CC      sparc64-softmmu/migration/ram.o
  CC      sparc-softmmu/qtest.o
  CC      sh4-softmmu/dump.o
  CC      sh4eb-softmmu/hw/virtio/vhost-backend.o
  CC      sparc-softmmu/memory.o
  CC      sparc64-softmmu/accel/accel.o
  CC      sh4eb-softmmu/hw/virtio/vhost-user.o
  CC      sh4-softmmu/migration/ram.o
  CC      sparc64-softmmu/accel/stubs/kvm-stub.o
  CC      sh4eb-softmmu/hw/virtio/vhost-vsock.o
  CC      sh4eb-softmmu/hw/virtio/virtio-crypto.o
  CC      sparc64-softmmu/accel/tcg/tcg-all.o
  CC      sh4-softmmu/accel/accel.o
  CC      sparc-softmmu/memory_mapping.o
  CC      sparc64-softmmu/accel/tcg/cputlb.o
  CC      sh4eb-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      sh4-softmmu/accel/stubs/kvm-stub.o
  CC      sparc-softmmu/dump.o
  CC      sh4-softmmu/accel/tcg/tcg-all.o
  CC      sh4eb-softmmu/hw/sh4/shix.o
  CC      sh4-softmmu/accel/tcg/cputlb.o
  CC      sh4eb-softmmu/hw/sh4/r2d.o
  CC      sparc-softmmu/migration/ram.o
  CC      sh4eb-softmmu/hw/sh4/sh7750.o
  CC      sh4eb-softmmu/hw/sh4/sh7750_regnames.o
  CC      sparc64-softmmu/accel/tcg/cpu-exec.o
  CC      sh4eb-softmmu/hw/sh4/sh_pci.o
  CC      sh4eb-softmmu/target/sh4/translate.o
  CC      sparc-softmmu/accel/accel.o
  CC      sparc64-softmmu/accel/tcg/cpu-exec-common.o
  CC      sh4-softmmu/accel/tcg/cpu-exec.o
  CC      sparc-softmmu/accel/stubs/kvm-stub.o
  CC      sparc64-softmmu/accel/tcg/translate-all.o
  CC      sh4-softmmu/accel/tcg/cpu-exec-common.o
  CC      sparc-softmmu/accel/tcg/tcg-all.o
  CC      sparc64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sparc-softmmu/accel/tcg/cputlb.o
  CC      sh4-softmmu/accel/tcg/translate-all.o
  CC      sparc64-softmmu/hw/block/virtio-blk.o
  CC      sh4eb-softmmu/target/sh4/op_helper.o
  CC      sh4-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sh4eb-softmmu/target/sh4/helper.o
  CC      sparc64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      sh4-softmmu/hw/block/tc58128.o
  CC      sparc64-softmmu/hw/char/virtio-serial-bus.o
  CC      sh4eb-softmmu/target/sh4/cpu.o
  CC      sparc-softmmu/accel/tcg/cpu-exec.o
  CC      sh4-softmmu/hw/block/virtio-blk.o
  CC      sh4eb-softmmu/target/sh4/monitor.o
  CC      sparc-softmmu/accel/tcg/cpu-exec-common.o
  CC      sparc64-softmmu/hw/core/generic-loader.o
  CC      sh4eb-softmmu/target/sh4/gdbstub.o
  CC      sparc-softmmu/accel/tcg/translate-all.o
  CC      sh4-softmmu/hw/block/dataplane/virtio-blk.o
  GEN     trace/generated-helpers.c
  CC      sparc64-softmmu/hw/core/null-machine.o
  CC      sh4eb-softmmu/trace/control-target.o
  CC      sh4-softmmu/hw/char/sh_serial.o
  CC      sparc64-softmmu/hw/display/vga.o
  CC      sparc-softmmu/hw/core/generic-loader.o
  CC      sh4eb-softmmu/trace/generated-helpers.o
  CC      sh4-softmmu/hw/char/virtio-serial-bus.o
  CC      sparc-softmmu/hw/core/null-machine.o
  LINK    sh4eb-softmmu/qemu-system-sh4eb
  CC      sparc-softmmu/hw/display/tcx.o
  CC      sh4-softmmu/hw/core/generic-loader.o
  CC      sparc64-softmmu/hw/display/virtio-gpu.o
  CC      sh4-softmmu/hw/core/null-machine.o
  CC      sparc-softmmu/hw/display/cg3.o
  CC      sh4-softmmu/hw/display/sm501.o
  CC      sparc-softmmu/hw/intc/grlib_irqmp.o
  CC      sparc64-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc-softmmu/hw/misc/eccmemctl.o
  CC      sh4-softmmu/hw/display/vga.o
  CC      sparc-softmmu/hw/misc/slavio_misc.o
  CC      sparc64-softmmu/hw/display/virtio-gpu-pci.o
  CC      sparc-softmmu/hw/misc/mmio_interface.o
  GEN     tricore-softmmu/hmp-commands.h
  GEN     tricore-softmmu/hmp-commands-info.h
  GEN     tricore-softmmu/config-target.h
  CC      sparc64-softmmu/hw/misc/ivshmem.o
  CC      sparc-softmmu/hw/net/vhost_net.o
  CC      tricore-softmmu/exec.o
  CC      sh4-softmmu/hw/display/virtio-gpu.o
  CC      sparc-softmmu/hw/net/rocker/qmp-norocker.o
  CC      sparc64-softmmu/hw/misc/mmio_interface.o
  CC      sparc-softmmu/hw/vfio/common.o
  CC      sparc64-softmmu/hw/net/virtio-net.o
  CC      sh4-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc-softmmu/hw/vfio/platform.o
  CC      sparc-softmmu/hw/vfio/spapr.o
  CC      sparc64-softmmu/hw/net/vhost_net.o
  CC      sparc64-softmmu/hw/scsi/virtio-scsi.o
  CC      sh4-softmmu/hw/display/virtio-gpu-pci.o
  CC      sparc-softmmu/hw/sparc/sun4m.o
  CC      tricore-softmmu/tcg/tcg.o
  CC      sh4-softmmu/hw/intc/sh_intc.o
  CC      sparc64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      sparc64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      sparc-softmmu/hw/sparc/leon3.o
  CC      sh4-softmmu/hw/misc/ivshmem.o
  CC      sparc-softmmu/target/sparc/machine.o
  CC      sparc64-softmmu/hw/scsi/vhost-scsi.o
  CC      sparc-softmmu/target/sparc/monitor.o
  CC      sh4-softmmu/hw/misc/mmio_interface.o
  CC      sparc64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      sparc-softmmu/target/sparc/translate.o
  CC      sh4-softmmu/hw/net/virtio-net.o
  CC      sparc64-softmmu/hw/timer/mc146818rtc.o
  CC      tricore-softmmu/tcg/tcg-op.o
  CC      sparc64-softmmu/hw/vfio/common.o
  CC      sh4-softmmu/hw/net/vhost_net.o
  CC      sparc-softmmu/target/sparc/helper.o
  CC      sh4-softmmu/hw/scsi/virtio-scsi.o
  CC      sparc64-softmmu/hw/vfio/pci.o
  CC      sparc-softmmu/target/sparc/cpu.o
  CC      sh4-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      sparc-softmmu/target/sparc/fop_helper.o
  CC      sh4-softmmu/hw/scsi/vhost-scsi-common.o
  CC      sh4-softmmu/hw/scsi/vhost-scsi.o
  CC      sparc-softmmu/target/sparc/cc_helper.o
  CC      sparc64-softmmu/hw/vfio/pci-quirks.o
  CC      tricore-softmmu/tcg/optimize.o
  CC      sh4-softmmu/hw/scsi/vhost-user-scsi.o
  CC      sparc-softmmu/target/sparc/win_helper.o
  CC      sh4-softmmu/hw/timer/sh_timer.o
  CC      sparc-softmmu/target/sparc/mmu_helper.o
  CC      sparc64-softmmu/hw/vfio/platform.o
  CC      sh4-softmmu/hw/timer/mc146818rtc.o
  CC      sparc-softmmu/target/sparc/ldst_helper.o
  CC      tricore-softmmu/tcg/tcg-common.o
  CC      sparc64-softmmu/hw/vfio/spapr.o
  CC      tricore-softmmu/tcg/tcg-runtime.o
  CC      sparc-softmmu/target/sparc/int32_helper.o
  CC      sh4-softmmu/hw/vfio/common.o
  CC      tricore-softmmu/fpu/softfloat.o
  CC      sparc64-softmmu/hw/virtio/virtio.o
  CC      sparc-softmmu/target/sparc/gdbstub.o
  GEN     trace/generated-helpers.c
  CC      sh4-softmmu/hw/vfio/pci.o
  CC      sparc-softmmu/trace/control-target.o
  CC      sparc64-softmmu/hw/virtio/virtio-balloon.o
  CC      sparc-softmmu/trace/generated-helpers.o
  CC      sparc64-softmmu/hw/virtio/vhost.o
  LINK    sparc-softmmu/qemu-system-sparc
  CC      sh4-softmmu/hw/vfio/pci-quirks.o
  CC      sparc64-softmmu/hw/virtio/vhost-backend.o
  CC      tricore-softmmu/disas.o
  CC      sh4-softmmu/hw/vfio/platform.o
  CC      sparc64-softmmu/hw/virtio/vhost-user.o
  CC      tricore-softmmu/hax-stub.o
  CC      tricore-softmmu/arch_init.o
  CC      sh4-softmmu/hw/vfio/spapr.o
  CC      tricore-softmmu/cpus.o
  CC      sh4-softmmu/hw/virtio/virtio.o
  CC      sparc64-softmmu/hw/virtio/vhost-vsock.o
  CC      sparc64-softmmu/hw/virtio/virtio-crypto.o
  CC      sh4-softmmu/hw/virtio/virtio-balloon.o
  CC      tricore-softmmu/monitor.o
  CC      sparc64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      sh4-softmmu/hw/virtio/vhost.o
  CC      sparc64-softmmu/hw/sparc64/sparc64.o
  CC      sh4-softmmu/hw/virtio/vhost-backend.o
  CC      sparc64-softmmu/hw/sparc64/sun4u.o
  CC      sparc64-softmmu/hw/sparc64/niagara.o
  CC      sh4-softmmu/hw/virtio/vhost-user.o
  CC      sh4-softmmu/hw/virtio/vhost-vsock.o
  CC      tricore-softmmu/gdbstub.o
  CC      sparc64-softmmu/target/sparc/machine.o
  CC      sparc64-softmmu/target/sparc/monitor.o
  CC      sh4-softmmu/hw/virtio/virtio-crypto.o
  CC      tricore-softmmu/balloon.o
  CC      sparc64-softmmu/target/sparc/translate.o
  CC      sh4-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      tricore-softmmu/ioport.o
  GEN     unicore32-softmmu/hmp-commands.h
  GEN     unicore32-softmmu/hmp-commands-info.h
  GEN     unicore32-softmmu/config-target.h
  CC      sh4-softmmu/hw/sh4/shix.o
  CC      unicore32-softmmu/exec.o
  CC      tricore-softmmu/numa.o
  CC      sh4-softmmu/hw/sh4/r2d.o
  CC      tricore-softmmu/qtest.o
  CC      sh4-softmmu/hw/sh4/sh7750.o
  CC      tricore-softmmu/memory.o
  CC      sh4-softmmu/hw/sh4/sh7750_regnames.o
  CC      unicore32-softmmu/tcg/tcg.o
  CC      sparc64-softmmu/target/sparc/helper.o
  CC      sh4-softmmu/hw/sh4/sh_pci.o
  CC      sparc64-softmmu/target/sparc/cpu.o
  CC      tricore-softmmu/memory_mapping.o
  CC      sh4-softmmu/target/sh4/translate.o
  CC      tricore-softmmu/dump.o
  CC      sparc64-softmmu/target/sparc/fop_helper.o
  CC      sparc64-softmmu/target/sparc/cc_helper.o
  CC      unicore32-softmmu/tcg/tcg-op.o
  CC      tricore-softmmu/migration/ram.o
  CC      sparc64-softmmu/target/sparc/win_helper.o
  CC      sh4-softmmu/target/sh4/op_helper.o
  CC      sparc64-softmmu/target/sparc/mmu_helper.o
  CC      tricore-softmmu/accel/accel.o
  CC      sh4-softmmu/target/sh4/helper.o
  CC      sparc64-softmmu/target/sparc/ldst_helper.o
  CC      tricore-softmmu/accel/stubs/kvm-stub.o
  CC      sparc64-softmmu/target/sparc/int64_helper.o
  CC      tricore-softmmu/accel/tcg/tcg-all.o
  CC      sh4-softmmu/target/sh4/cpu.o
  CC      tricore-softmmu/accel/tcg/cputlb.o
  CC      sparc64-softmmu/target/sparc/vis_helper.o
  CC      sh4-softmmu/target/sh4/monitor.o
  CC      unicore32-softmmu/tcg/optimize.o
  CC      sparc64-softmmu/target/sparc/gdbstub.o
  CC      sh4-softmmu/target/sh4/gdbstub.o
  GEN     trace/generated-helpers.c
  CC      sparc64-softmmu/trace/control-target.o
  GEN     trace/generated-helpers.c
  CC      sh4-softmmu/trace/control-target.o
  CC      sparc64-softmmu/trace/generated-helpers.o
  CC      unicore32-softmmu/tcg/tcg-common.o
  CC      sh4-softmmu/trace/generated-helpers.o
  CC      tricore-softmmu/accel/tcg/cpu-exec.o
  CC      unicore32-softmmu/tcg/tcg-runtime.o
  LINK    sparc64-softmmu/qemu-system-sparc64
  CC      tricore-softmmu/accel/tcg/cpu-exec-common.o
  LINK    sh4-softmmu/qemu-system-sh4
  CC      tricore-softmmu/accel/tcg/translate-all.o
  CC      unicore32-softmmu/fpu/softfloat.o
  CC      tricore-softmmu/hw/core/generic-loader.o
  CC      tricore-softmmu/hw/core/null-machine.o
  CC      unicore32-softmmu/disas.o
  CC      tricore-softmmu/hw/misc/mmio_interface.o
  CC      tricore-softmmu/hw/net/vhost_net.o
  GEN     x86_64-softmmu/hmp-commands.h
  GEN     xtensaeb-softmmu/hmp-commands.h
  GEN     x86_64-softmmu/hmp-commands-info.h
  GEN     x86_64-softmmu/config-target.h
  CC      tricore-softmmu/hw/net/rocker/qmp-norocker.o
  GEN     xtensaeb-softmmu/hmp-commands-info.h
  GEN     xtensaeb-softmmu/config-target.h
  CC      x86_64-softmmu/exec.o
  CC      xtensaeb-softmmu/exec.o
  CC      tricore-softmmu/hw/vfio/common.o
  CC      unicore32-softmmu/hax-stub.o
  CC      tricore-softmmu/hw/vfio/platform.o
  CC      unicore32-softmmu/arch_init.o
  CC      tricore-softmmu/hw/vfio/spapr.o
  CC      unicore32-softmmu/cpus.o
  CC      tricore-softmmu/hw/tricore/tricore_testboard.o
  CC      x86_64-softmmu/tcg/tcg.o
  CC      xtensaeb-softmmu/tcg/tcg.o
  CC      tricore-softmmu/target/tricore/translate.o
  CC      unicore32-softmmu/monitor.o
  CC      xtensaeb-softmmu/tcg/tcg-op.o
  CC      unicore32-softmmu/gdbstub.o
  CC      x86_64-softmmu/tcg/tcg-op.o
  CC      unicore32-softmmu/balloon.o
  CC      unicore32-softmmu/ioport.o
  CC      unicore32-softmmu/numa.o
  CC      tricore-softmmu/target/tricore/helper.o
  CC      xtensaeb-softmmu/tcg/optimize.o
  CC      tricore-softmmu/target/tricore/cpu.o
  CC      unicore32-softmmu/qtest.o
  CC      x86_64-softmmu/tcg/optimize.o
  CC      tricore-softmmu/target/tricore/op_helper.o
  CC      unicore32-softmmu/memory.o
  CC      xtensaeb-softmmu/tcg/tcg-common.o
  CC      xtensaeb-softmmu/tcg/tcg-runtime.o
  CC      x86_64-softmmu/tcg/tcg-common.o
  CC      x86_64-softmmu/tcg/tcg-runtime.o
  CC      unicore32-softmmu/memory_mapping.o
  CC      xtensaeb-softmmu/fpu/softfloat.o
  CC      tricore-softmmu/target/tricore/fpu_helper.o
  CC      x86_64-softmmu/fpu/softfloat.o
  CC      unicore32-softmmu/dump.o
  GEN     trace/generated-helpers.c
  CC      tricore-softmmu/trace/control-target.o
  CC      unicore32-softmmu/migration/ram.o
  CC      tricore-softmmu/trace/generated-helpers.o
  CC      unicore32-softmmu/accel/accel.o
  LINK    tricore-softmmu/qemu-system-tricore
  CC      unicore32-softmmu/accel/stubs/kvm-stub.o
  CC      xtensaeb-softmmu/disas.o
  CC      unicore32-softmmu/accel/tcg/tcg-all.o
  CC      x86_64-softmmu/disas.o
  GEN     x86_64-softmmu/gdbstub-xml.c
  CC      xtensaeb-softmmu/hax-stub.o
  CC      unicore32-softmmu/accel/tcg/cputlb.o
  CC      x86_64-softmmu/hax-stub.o
  CC      xtensaeb-softmmu/arch_init.o
  GEN     xtensa-softmmu/hmp-commands.h
  GEN     xtensa-softmmu/hmp-commands-info.h
  GEN     xtensa-softmmu/config-target.h
  CC      x86_64-softmmu/arch_init.o
  CC      xtensa-softmmu/exec.o
  CC      xtensaeb-softmmu/cpus.o
  CC      x86_64-softmmu/cpus.o
  CC      xtensaeb-softmmu/monitor.o
  CC      unicore32-softmmu/accel/tcg/cpu-exec.o
  CC      x86_64-softmmu/monitor.o
  CC      unicore32-softmmu/accel/tcg/cpu-exec-common.o
  CC      unicore32-softmmu/accel/tcg/translate-all.o
  CC      xtensa-softmmu/tcg/tcg.o
  CC      xtensaeb-softmmu/gdbstub.o
  CC      unicore32-softmmu/hw/core/generic-loader.o
  CC      unicore32-softmmu/hw/core/null-machine.o
  CC      xtensaeb-softmmu/balloon.o
  CC      x86_64-softmmu/gdbstub.o
  CC      unicore32-softmmu/hw/misc/mmio_interface.o
  CC      xtensaeb-softmmu/ioport.o
  CC      unicore32-softmmu/hw/net/vhost_net.o
  CC      xtensa-softmmu/tcg/tcg-op.o
  CC      xtensaeb-softmmu/numa.o
  CC      x86_64-softmmu/balloon.o
  CC      unicore32-softmmu/hw/net/rocker/qmp-norocker.o
  CC      x86_64-softmmu/ioport.o
  CC      xtensaeb-softmmu/qtest.o
  CC      unicore32-softmmu/hw/vfio/common.o
  CC      x86_64-softmmu/numa.o
  CC      xtensaeb-softmmu/memory.o
  CC      unicore32-softmmu/hw/vfio/platform.o
  CC      x86_64-softmmu/qtest.o
  CC      unicore32-softmmu/hw/vfio/spapr.o
  CC      xtensaeb-softmmu/memory_mapping.o
  CC      x86_64-softmmu/memory.o
  CC      unicore32-softmmu/hw/unicore32/puv3.o
  CC      xtensaeb-softmmu/dump.o
  CC      xtensa-softmmu/tcg/optimize.o
  CC      unicore32-softmmu/target/unicore32/translate.o
  CC      xtensaeb-softmmu/migration/ram.o
  CC      x86_64-softmmu/memory_mapping.o
  CC      xtensa-softmmu/tcg/tcg-common.o
  CC      x86_64-softmmu/dump.o
  CC      xtensa-softmmu/tcg/tcg-runtime.o
  CC      unicore32-softmmu/target/unicore32/op_helper.o
  CC      xtensaeb-softmmu/accel/accel.o
  CC      xtensa-softmmu/fpu/softfloat.o
  CC      unicore32-softmmu/target/unicore32/helper.o
  CC      xtensaeb-softmmu/accel/stubs/kvm-stub.o
  CC      x86_64-softmmu/migration/ram.o
  CC      unicore32-softmmu/target/unicore32/cpu.o
  CC      xtensaeb-softmmu/accel/tcg/tcg-all.o
  CC      unicore32-softmmu/target/unicore32/ucf64_helper.o
  CC      xtensaeb-softmmu/accel/tcg/cputlb.o
  CC      unicore32-softmmu/target/unicore32/softmmu.o
  CC      x86_64-softmmu/accel/accel.o
  CC      x86_64-softmmu/accel/stubs/kvm-stub.o
  GEN     trace/generated-helpers.c
  CC      unicore32-softmmu/trace/control-target.o
  CC      x86_64-softmmu/accel/tcg/tcg-all.o
  CC      unicore32-softmmu/trace/generated-helpers.o
  CC      x86_64-softmmu/accel/tcg/cputlb.o
  CC      xtensaeb-softmmu/accel/tcg/cpu-exec.o
  CC      xtensa-softmmu/disas.o
  LINK    unicore32-softmmu/qemu-system-unicore32
  CC      xtensa-softmmu/hax-stub.o
  CC      xtensaeb-softmmu/accel/tcg/cpu-exec-common.o
  CC      xtensa-softmmu/arch_init.o
  CC      xtensaeb-softmmu/accel/tcg/translate-all.o
  CC      xtensa-softmmu/cpus.o
  CC      x86_64-softmmu/accel/tcg/cpu-exec.o
  CC      xtensaeb-softmmu/hw/core/generic-loader.o
  CC      xtensaeb-softmmu/hw/core/null-machine.o
  GEN     aarch64-linux-user/config-target.h
  CC      aarch64-linux-user/exec.o
  CC      xtensa-softmmu/monitor.o
  CC      x86_64-softmmu/accel/tcg/cpu-exec-common.o
  CC      xtensaeb-softmmu/hw/misc/mmio_interface.o
  CC      x86_64-softmmu/accel/tcg/translate-all.o
  CC      aarch64-linux-user/tcg/tcg.o
  CC      xtensaeb-softmmu/hw/net/vhost_net.o
  CC      x86_64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      xtensaeb-softmmu/hw/net/rocker/qmp-norocker.o
  CC      xtensaeb-softmmu/hw/vfio/common.o
  CC      x86_64-softmmu/hw/block/virtio-blk.o
  CC      xtensa-softmmu/gdbstub.o
  CC      xtensa-softmmu/balloon.o
  CC      xtensaeb-softmmu/hw/vfio/platform.o
  CC      x86_64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      xtensa-softmmu/ioport.o
  CC      aarch64-linux-user/tcg/tcg-op.o
  CC      x86_64-softmmu/hw/char/virtio-serial-bus.o
  CC      xtensaeb-softmmu/hw/vfio/spapr.o
  CC      xtensa-softmmu/numa.o
  CC      xtensaeb-softmmu/hw/xtensa/pic_cpu.o
  CC      x86_64-softmmu/hw/core/generic-loader.o
  CC      xtensa-softmmu/qtest.o
  CC      xtensaeb-softmmu/hw/xtensa/sim.o
  CC      x86_64-softmmu/hw/core/null-machine.o
  CC      xtensaeb-softmmu/hw/xtensa/xtfpga.o
  CC      xtensa-softmmu/memory.o
  CC      x86_64-softmmu/hw/display/vga.o
  CC      xtensaeb-softmmu/target/xtensa/xtensa-semi.o
  CC      xtensaeb-softmmu/target/xtensa/core-dc232b.o
  CC      x86_64-softmmu/hw/display/virtio-gpu.o
  CC      xtensa-softmmu/memory_mapping.o
  CC      aarch64-linux-user/tcg/optimize.o
  CC      xtensaeb-softmmu/target/xtensa/core-dc233c.o
  CC      xtensa-softmmu/dump.o
  CC      xtensaeb-softmmu/target/xtensa/core-fsf.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-3d.o
  CC      xtensaeb-softmmu/target/xtensa/monitor.o
  CC      xtensa-softmmu/migration/ram.o
  CC      aarch64-linux-user/tcg/tcg-common.o
  CC      xtensaeb-softmmu/target/xtensa/translate.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-linux-user/tcg/tcg-runtime.o
  CC      xtensa-softmmu/accel/accel.o
  CC      x86_64-softmmu/hw/display/virtio-vga.o
  CC      xtensa-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-linux-user/fpu/softfloat.o
  CC      x86_64-softmmu/hw/intc/apic.o
  CC      xtensa-softmmu/accel/tcg/tcg-all.o
  CC      xtensa-softmmu/accel/tcg/cputlb.o
  CC      xtensaeb-softmmu/target/xtensa/op_helper.o
  CC      x86_64-softmmu/hw/intc/apic_common.o
  CC      xtensaeb-softmmu/target/xtensa/helper.o
  CC      x86_64-softmmu/hw/intc/ioapic.o
  CC      xtensaeb-softmmu/target/xtensa/cpu.o
  CC      x86_64-softmmu/hw/isa/lpc_ich9.o
  CC      xtensa-softmmu/accel/tcg/cpu-exec.o
  CC      xtensaeb-softmmu/target/xtensa/gdbstub.o
  CC      x86_64-softmmu/hw/misc/vmport.o
  GEN     trace/generated-helpers.c
  CC      xtensaeb-softmmu/trace/control-target.o
  CC      xtensa-softmmu/accel/tcg/cpu-exec-common.o
  CC      x86_64-softmmu/hw/misc/ivshmem.o
  CC      aarch64-linux-user/disas.o
  CC      xtensaeb-softmmu/trace/generated-helpers.o
  CC      xtensa-softmmu/accel/tcg/translate-all.o
  CC      x86_64-softmmu/hw/misc/pvpanic.o
  GEN     aarch64-linux-user/gdbstub-xml.c
  LINK    xtensaeb-softmmu/qemu-system-xtensaeb
  CC      aarch64-linux-user/hax-stub.o
  CC      xtensa-softmmu/hw/core/generic-loader.o
  CC      x86_64-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-linux-user/gdbstub.o
  CC      x86_64-softmmu/hw/net/virtio-net.o
  CC      xtensa-softmmu/hw/core/null-machine.o
  CC      xtensa-softmmu/hw/misc/mmio_interface.o
  CC      x86_64-softmmu/hw/net/vhost_net.o
  CC      aarch64-linux-user/thunk.o
  CC      xtensa-softmmu/hw/net/vhost_net.o
  CC      xtensa-softmmu/hw/net/rocker/qmp-norocker.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi.o
  CC      xtensa-softmmu/hw/vfio/common.o
  CC      xtensa-softmmu/hw/vfio/platform.o
  CC      aarch64-linux-user/user-exec.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-linux-user/user-exec-stub.o
  CC      x86_64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      xtensa-softmmu/hw/vfio/spapr.o
  CC      x86_64-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-linux-user/accel/stubs/kvm-stub.o
  CC      xtensa-softmmu/hw/xtensa/pic_cpu.o
  CC      x86_64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-linux-user/accel/tcg/cpu-exec.o
  CC      x86_64-softmmu/hw/timer/mc146818rtc.o
  CC      xtensa-softmmu/hw/xtensa/sim.o
  CC      xtensa-softmmu/hw/xtensa/xtfpga.o
  CC      x86_64-softmmu/hw/vfio/common.o
  CC      xtensa-softmmu/target/xtensa/xtensa-semi.o
  CC      aarch64-linux-user/accel/tcg/cpu-exec-common.o
  CC      x86_64-softmmu/hw/vfio/pci.o
  CC      aarch64-linux-user/accel/tcg/translate-all.o
  CC      xtensa-softmmu/target/xtensa/core-dc232b.o
  CC      xtensa-softmmu/target/xtensa/core-dc233c.o
  CC      aarch64-linux-user/linux-user/main.o
  CC      xtensa-softmmu/target/xtensa/core-fsf.o
  CC      x86_64-softmmu/hw/vfio/pci-quirks.o
  CC      xtensa-softmmu/target/xtensa/monitor.o
  GEN     alpha-linux-user/config-target.h
  CC      alpha-linux-user/exec.o
  CC      aarch64-linux-user/linux-user/syscall.o
  CC      xtensa-softmmu/target/xtensa/translate.o
  CC      x86_64-softmmu/hw/vfio/platform.o
  CC      alpha-linux-user/tcg/tcg.o
  CC      x86_64-softmmu/hw/vfio/spapr.o
  CC      x86_64-softmmu/hw/virtio/virtio.o
  CC      alpha-linux-user/tcg/tcg-op.o
  CC      xtensa-softmmu/target/xtensa/op_helper.o
  CC      x86_64-softmmu/hw/virtio/virtio-balloon.o
  CC      xtensa-softmmu/target/xtensa/helper.o
  CC      aarch64-linux-user/linux-user/strace.o
  CC      x86_64-softmmu/hw/virtio/vhost.o
  CC      xtensa-softmmu/target/xtensa/cpu.o
  CC      aarch64-linux-user/linux-user/mmap.o
  CC      xtensa-softmmu/target/xtensa/gdbstub.o
  CC      alpha-linux-user/tcg/optimize.o
  CC      aarch64-linux-user/linux-user/signal.o
  CC      x86_64-softmmu/hw/virtio/vhost-backend.o
  GEN     trace/generated-helpers.c
  CC      x86_64-softmmu/hw/virtio/vhost-user.o
  CC      xtensa-softmmu/trace/control-target.o
  CC      aarch64-linux-user/linux-user/elfload.o
  CC      xtensa-softmmu/trace/generated-helpers.o
  CC      alpha-linux-user/tcg/tcg-common.o
  CC      x86_64-softmmu/hw/virtio/vhost-vsock.o
  CC      alpha-linux-user/tcg/tcg-runtime.o
  LINK    xtensa-softmmu/qemu-system-xtensa
  CC      aarch64-linux-user/linux-user/linuxload.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto.o
  CC      alpha-linux-user/fpu/softfloat.o
  CC      aarch64-linux-user/linux-user/uaccess.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      alpha-linux-user/disas.o
  CC      aarch64-linux-user/linux-user/uname.o
  CCAS    aarch64-linux-user/linux-user/safe-syscall.o
  GEN     armeb-linux-user/config-target.h
  CC      x86_64-softmmu/hw/i386/multiboot.o
  CC      armeb-linux-user/exec.o
  CC      aarch64-linux-user/linux-user/flatload.o
  CC      x86_64-softmmu/hw/i386/pc.o
  CC      armeb-linux-user/tcg/tcg.o
  CC      aarch64-linux-user/target/arm/arm-semi.o
  CC      aarch64-linux-user/target/arm/kvm-stub.o
  CC      x86_64-softmmu/hw/i386/pc_piix.o
In file included from /var/tmp/patchew-tester-tmp-hm2hdbnw/src/target/arm/kvm-stub.c:15:0:
/var/tmp/patchew-tester-tmp-hm2hdbnw/src/target/arm/kvm_arm.h:291:28: error: unknown type name ‘ram_addr_t’
 void kvm_hwpoison_page_add(ram_addr_t ram_addr);
                            ^~~~~~~~~~
/var/tmp/patchew-tester-tmp-hm2hdbnw/src/rules.mak:66: recipe for target 'target/arm/kvm-stub.o' failed
make[1]: *** [target/arm/kvm-stub.o] Error 1
Makefile:326: recipe for target 'subdir-aarch64-linux-user' failed
make: *** [subdir-aarch64-linux-user] Error 2
make: *** Waiting for unfinished jobs....
  CC      alpha-linux-user/hax-stub.o
  CC      x86_64-softmmu/hw/i386/pc_q35.o
  CC      alpha-linux-user/gdbstub.o
  CC      armeb-linux-user/tcg/tcg-op.o
  CC      x86_64-softmmu/hw/i386/pc_sysfw.o
  CC      x86_64-softmmu/hw/i386/x86-iommu.o
  CC      alpha-linux-user/thunk.o
  CC      armeb-linux-user/tcg/optimize.o
  CC      alpha-linux-user/user-exec.o
  CC      x86_64-softmmu/hw/i386/intel_iommu.o
  CC      alpha-linux-user/user-exec-stub.o
  CC      x86_64-softmmu/hw/i386/amd_iommu.o
  CC      alpha-linux-user/accel/stubs/kvm-stub.o
  CC      alpha-linux-user/accel/tcg/cpu-exec.o
  CC      x86_64-softmmu/hw/i386/kvmvapic.o
  CC      x86_64-softmmu/hw/i386/acpi-build.o
  CC      armeb-linux-user/tcg/tcg-common.o
  CC      alpha-linux-user/accel/tcg/cpu-exec-common.o
  CC      alpha-linux-user/accel/tcg/translate-all.o
  CC      armeb-linux-user/tcg/tcg-runtime.o
  CC      x86_64-softmmu/hw/i386/pci-assign-load-rom.o
  CC      alpha-linux-user/linux-user/main.o
  CC      armeb-linux-user/fpu/softfloat.o
  CC      alpha-linux-user/linux-user/syscall.o
  CC      x86_64-softmmu/target/i386/helper.o
  CC      alpha-linux-user/linux-user/strace.o
  CC      x86_64-softmmu/target/i386/cpu.o
  CC      x86_64-softmmu/target/i386/gdbstub.o
  CC      armeb-linux-user/disas.o
  CC      x86_64-softmmu/target/i386/xsave_helper.o
  CC      x86_64-softmmu/target/i386/translate.o
  GEN     armeb-linux-user/gdbstub-xml.c
  CC      alpha-linux-user/linux-user/mmap.o
  CC      x86_64-softmmu/target/i386/bpt_helper.o
  CC      armeb-linux-user/hax-stub.o
  CC      alpha-linux-user/linux-user/signal.o
  CC      alpha-linux-user/linux-user/elfload.o
  CC      armeb-linux-user/gdbstub.o
  CC      alpha-linux-user/linux-user/linuxload.o
  CC      x86_64-softmmu/target/i386/cc_helper.o
  CC      armeb-linux-user/thunk.o
  CC      alpha-linux-user/linux-user/uaccess.o
  CC      alpha-linux-user/linux-user/uname.o
  CC      armeb-linux-user/user-exec.o
  CCAS    alpha-linux-user/linux-user/safe-syscall.o
  CC      alpha-linux-user/target/alpha/translate.o
  CC      alpha-linux-user/target/alpha/helper.o
  CC      armeb-linux-user/user-exec-stub.o
  CC      x86_64-softmmu/target/i386/excp_helper.o
  CC      armeb-linux-user/accel/stubs/kvm-stub.o
  CC      alpha-linux-user/target/alpha/cpu.o
  CC      armeb-linux-user/accel/tcg/cpu-exec.o
  CC      alpha-linux-user/target/alpha/int_helper.o
  CC      x86_64-softmmu/target/i386/fpu_helper.o
  CC      armeb-linux-user/accel/tcg/cpu-exec-common.o
  CC      alpha-linux-user/target/alpha/fpu_helper.o
  CC      alpha-linux-user/target/alpha/vax_helper.o
  CC      armeb-linux-user/accel/tcg/translate-all.o
  CC      alpha-linux-user/target/alpha/sys_helper.o
  CC      alpha-linux-user/target/alpha/mem_helper.o
  CC      armeb-linux-user/linux-user/main.o
  CC      armeb-linux-user/linux-user/syscall.o
  CC      alpha-linux-user/target/alpha/gdbstub.o
  CC      armeb-linux-user/linux-user/strace.o
  GEN     trace/generated-helpers.c
  CC      alpha-linux-user/trace/control-target.o
  CC      alpha-linux-user/trace/generated-helpers.o
  CC      x86_64-softmmu/target/i386/int_helper.o
  CC      x86_64-softmmu/target/i386/mem_helper.o
  LINK    alpha-linux-user/qemu-alpha
  CC      armeb-linux-user/linux-user/mmap.o
  CC      x86_64-softmmu/target/i386/misc_helper.o
  CC      armeb-linux-user/linux-user/signal.o
  CC      x86_64-softmmu/target/i386/mpx_helper.o
  CC      x86_64-softmmu/target/i386/seg_helper.o
  CC      x86_64-softmmu/target/i386/smm_helper.o
  CC      x86_64-softmmu/target/i386/svm_helper.o
  CC      x86_64-softmmu/target/i386/machine.o
  CC      x86_64-softmmu/target/i386/arch_memory_mapping.o
  CC      x86_64-softmmu/target/i386/arch_dump.o
  CC      x86_64-softmmu/target/i386/monitor.o
  CC      x86_64-softmmu/target/i386/kvm-stub.o
  GEN     trace/generated-helpers.c
  CC      x86_64-softmmu/trace/control-target.o
  CC      x86_64-softmmu/gdbstub-xml.o
  CC      armeb-linux-user/linux-user/elfload.o
  CC      x86_64-softmmu/trace/generated-helpers.o
  CC      armeb-linux-user/linux-user/linuxload.o
  CC      armeb-linux-user/linux-user/uaccess.o
  CC      armeb-linux-user/linux-user/uname.o
  LINK    x86_64-softmmu/qemu-system-x86_64
  CCAS    armeb-linux-user/linux-user/safe-syscall.o
  CC      armeb-linux-user/linux-user/flatload.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11_cpdo.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11_cpdt.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11_cprt.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpopcode.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/double_cpdo.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/single_cpdo.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/extended_cpdo.o
  CC      armeb-linux-user/target/arm/arm-semi.o
  CC      armeb-linux-user/target/arm/kvm-stub.o
  CC      armeb-linux-user/target/arm/translate.o
In file included from /var/tmp/patchew-tester-tmp-hm2hdbnw/src/target/arm/kvm-stub.c:15:0:
/var/tmp/patchew-tester-tmp-hm2hdbnw/src/target/arm/kvm_arm.h:291:28: error: unknown type name ‘ram_addr_t’
 void kvm_hwpoison_page_add(ram_addr_t ram_addr);
                            ^~~~~~~~~~
/var/tmp/patchew-tester-tmp-hm2hdbnw/src/rules.mak:66: recipe for target 'target/arm/kvm-stub.o' failed
make[1]: *** [target/arm/kvm-stub.o] Error 1
make[1]: *** Waiting for unfinished jobs....
Makefile:326: recipe for target 'subdir-armeb-linux-user' failed
make: *** [subdir-armeb-linux-user] Error 2
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
@ 2017-08-18 14:21   ` no-reply
  0 siblings, 0 replies; 129+ messages in thread
From: no-reply @ 2017-08-18 14:21 UTC (permalink / raw)
  To: gengdongjiu
  Cc: famz, mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, zhengqiang10,
	wuquanming, huangshaoyu, linuxarm

Hi,

This series failed build test on s390x host. Please find the details below.

Type: series
Message-id: 1503066227-18251-1-git-send-email-gengdongjiu@huawei.com
Subject: [Qemu-devel] [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e
echo "=== ENV ==="
env
echo "=== PACKAGES ==="
rpm -qa
echo "=== TEST BEGIN ==="
CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
echo -n "Using CC: "
realpath $CC
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1
make install
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]         patchew/1503066227-18251-1-git-send-email-gengdongjiu@huawei.com -> patchew/1503066227-18251-1-git-send-email-gengdongjiu@huawei.com
Switched to a new branch 'test'
06d4dbd target-arm: kvm64: Handle SError interrupt for the guest OS
8e5e362 target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
5d9b3fa target-arm: kvm64: detect guest RAS EXTENSION feature
e79f876 ACPI: build and enable APEI GHES in the Makefile and configuration
5e11799 ACPI: Add APEI GHES Table Generation support
d691b66 ACPI: add APEI/HEST/CPER structures and macros

=== OUTPUT BEGIN ===
=== ENV ===
XDG_SESSION_ID=32305
SHELL=/bin/sh
USER=fam
PATCHEW=/home/fam/patchew/patchew-cli -s http://patchew.org --nodebug
PATH=/usr/bin:/bin
PWD=/var/tmp/patchew-tester-tmp-hm2hdbnw/src
LANG=en_US.UTF-8
HOME=/home/fam
SHLVL=2
LOGNAME=fam
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1012/bus
XDG_RUNTIME_DIR=/run/user/1012
_=/usr/bin/env
=== PACKAGES ===
gpg-pubkey-873529b8-54e386ff
xz-libs-5.2.2-2.fc24.s390x
libxshmfence-1.2-3.fc24.s390x
giflib-4.1.6-15.fc24.s390x
trousers-lib-0.3.13-6.fc24.s390x
ncurses-base-6.0-6.20160709.fc25.noarch
gmp-6.1.1-1.fc25.s390x
libidn-1.33-1.fc25.s390x
slang-2.3.0-7.fc25.s390x
pkgconfig-0.29.1-1.fc25.s390x
alsa-lib-1.1.1-2.fc25.s390x
yum-metadata-parser-1.1.4-17.fc25.s390x
python3-slip-dbus-0.6.4-4.fc25.noarch
python2-cssselect-0.9.2-1.fc25.noarch
createrepo_c-libs-0.10.0-6.fc25.s390x
initscripts-9.69-1.fc25.s390x
parted-3.2-21.fc25.s390x
flex-2.6.0-3.fc25.s390x
colord-libs-1.3.4-1.fc25.s390x
python-osbs-client-0.33-3.fc25.noarch
perl-Pod-Simple-3.35-1.fc25.noarch
python2-simplejson-3.10.0-1.fc25.s390x
brltty-5.4-2.fc25.s390x
librados2-10.2.4-2.fc25.s390x
tcp_wrappers-7.6-83.fc25.s390x
libcephfs_jni1-10.2.4-2.fc25.s390x
nettle-devel-3.3-1.fc25.s390x
bzip2-devel-1.0.6-21.fc25.s390x
libuuid-2.28.2-2.fc25.s390x
python3-dnf-1.1.10-6.fc25.noarch
texlive-kpathsea-doc-svn41139-33.fc25.1.noarch
openssh-7.4p1-4.fc25.s390x
texlive-kpathsea-bin-svn40473-33.20160520.fc25.1.s390x
texlive-graphics-svn41015-33.fc25.1.noarch
texlive-dvipdfmx-def-svn40328-33.fc25.1.noarch
texlive-mfware-svn40768-33.fc25.1.noarch
texlive-texlive-scripts-svn41433-33.fc25.1.noarch
texlive-euro-svn22191.1.1-33.fc25.1.noarch
texlive-etex-svn37057.0-33.fc25.1.noarch
texlive-iftex-svn29654.0.2-33.fc25.1.noarch
texlive-palatino-svn31835.0-33.fc25.1.noarch
texlive-texlive-docindex-svn41430-33.fc25.1.noarch
texlive-xunicode-svn30466.0.981-33.fc25.1.noarch
texlive-koma-script-svn41508-33.fc25.1.noarch
texlive-pst-grad-svn15878.1.06-33.fc25.1.noarch
texlive-pst-blur-svn15878.2.0-33.fc25.1.noarch
texlive-jknapltx-svn19440.0-33.fc25.1.noarch
texinfo-6.1-4.fc25.s390x
openssl-devel-1.0.2k-1.fc25.s390x
gdk-pixbuf2-2.36.6-1.fc25.s390x
nspr-4.14.0-2.fc25.s390x
nss-softokn-freebl-3.30.2-1.0.fc25.s390x
jansson-2.10-2.fc25.s390x
fedora-repos-25-4.noarch
python3-libs-3.5.3-6.fc25.s390x
perl-Errno-1.25-387.fc25.s390x
acl-2.2.52-13.fc25.s390x
pcre2-utf16-10.23-8.fc25.s390x
pango-1.40.5-1.fc25.s390x
systemd-pam-231-17.fc25.s390x
python2-gluster-3.10.4-1.fc25.s390x
NetworkManager-libnm-1.4.4-5.fc25.s390x
selinux-policy-3.13.1-225.18.fc25.noarch
poppler-0.45.0-5.fc25.s390x
ccache-3.3.4-1.fc25.s390x
valgrind-3.12.0-9.fc25.s390x
perl-open-1.10-387.fc25.noarch
libaio-0.3.110-6.fc24.s390x
libfontenc-1.1.3-3.fc24.s390x
lzo-2.08-8.fc24.s390x
isl-0.14-5.fc24.s390x
libXau-1.0.8-6.fc24.s390x
linux-atm-libs-2.5.1-14.fc24.s390x
libXext-1.3.3-4.fc24.s390x
libXxf86vm-1.1.4-3.fc24.s390x
bison-3.0.4-4.fc24.s390x
perl-srpm-macros-1-20.fc25.noarch
gawk-4.1.3-8.fc25.s390x
libwayland-client-1.12.0-1.fc25.s390x
perl-Exporter-5.72-366.fc25.noarch
perl-version-0.99.17-1.fc25.s390x
fftw-libs-double-3.3.5-3.fc25.s390x
libssh2-1.8.0-1.fc25.s390x
ModemManager-glib-1.6.4-1.fc25.s390x
newt-python3-0.52.19-2.fc25.s390x
python-munch-2.0.4-3.fc25.noarch
python-bugzilla-1.2.2-4.fc25.noarch
libedit-3.1-16.20160618cvs.fc25.s390x
createrepo_c-0.10.0-6.fc25.s390x
device-mapper-multipath-libs-0.4.9-83.fc25.s390x
yum-3.4.3-510.fc25.noarch
dracut-config-rescue-044-78.fc25.s390x
mozjs17-17.0.0-16.fc25.s390x
libselinux-2.5-13.fc25.s390x
libgo-devel-6.3.1-1.fc25.s390x
python2-pyparsing-2.1.10-1.fc25.noarch
cairo-gobject-1.14.8-1.fc25.s390x
ethtool-4.8-1.fc25.s390x
xorg-x11-proto-devel-7.7-20.fc25.noarch
brlapi-0.6.5-2.fc25.s390x
librados-devel-10.2.4-2.fc25.s390x
libXinerama-devel-1.1.3-6.fc24.s390x
lua-posix-33.3.1-3.fc25.s390x
usbredir-devel-0.7.1-2.fc24.s390x
libepoxy-1.4.1-1.fc25.s390x
python3-dnf-plugins-core-0.1.21-5.fc25.noarch
texlive-pdftex-doc-svn41149-33.fc25.1.noarch
openssh-clients-7.4p1-4.fc25.s390x
iptables-1.6.0-3.fc25.s390x
texlive-texlive.infra-svn41280-33.fc25.1.noarch
texlive-graphics-cfg-svn40269-33.fc25.1.noarch
texlive-bibtex-svn40768-33.fc25.1.noarch
texlive-mfware-bin-svn40473-33.20160520.fc25.1.s390x
texlive-texlive-scripts-bin-svn29741.0-33.20160520.fc25.1.noarch
texlive-sauerj-svn15878.0-33.fc25.1.noarch
texlive-enctex-svn34957.0-33.fc25.1.noarch
texlive-ifetex-svn24853.1.2-33.fc25.1.noarch
texlive-ntgclass-svn15878.2.1a-33.fc25.1.noarch
texlive-tex-gyre-math-svn41264-33.fc25.1.noarch
texlive-bera-svn20031.0-33.fc25.1.noarch
texlive-ms-svn29849.0-33.fc25.1.noarch
texlive-pst-fill-svn15878.1.01-33.fc25.1.noarch
texlive-ctable-svn38672-33.fc25.1.noarch
texlive-extsizes-svn17263.1.4a-33.fc25.1.noarch
texlive-collection-latexrecommended-svn35765.0-33.20160520.fc25.1.noarch
perl-Filter-1.57-1.fc25.s390x
krb5-workstation-1.14.4-7.fc25.s390x
python2-rpm-macros-3-12.fc25.noarch
libglvnd-egl-0.2.999-14.20170308git8e6e102.fc25.s390x
libglvnd-opengl-0.2.999-14.20170308git8e6e102.fc25.s390x
gdbm-1.13-1.fc25.s390x
nss-util-3.30.2-1.0.fc25.s390x
libcrypt-nss-2.24-9.fc25.s390x
libtasn1-4.12-1.fc25.s390x
fedora-release-25-2.noarch
gdb-headless-7.12.1-48.fc25.s390x
perl-macros-5.24.2-387.fc25.s390x
sqlite-devel-3.14.2-2.fc25.s390x
pcre-devel-8.41-1.fc25.s390x
libX11-1.6.5-1.fc25.s390x
coreutils-8.25-17.fc25.s390x
python2-openidc-client-0-3.20170523git77cb3ee.fc25.noarch
nss-sysinit-3.30.2-1.1.fc25.s390x
systemtap-client-3.1-5.fc25.s390x
nss-devel-3.30.2-1.1.fc25.s390x
firewalld-0.4.4.5-1.fc25.noarch
rpmlint-1.9-9.fc25.noarch
system-python-3.5.3-6.fc25.s390x
gpg-pubkey-efe550f5-5220ba41
gpg-pubkey-81b46521-55b3ca9a
filesystem-3.2-37.fc24.s390x
libffi-3.1-9.fc24.s390x
keyutils-libs-1.5.9-8.fc24.s390x
libnfnetlink-1.0.1-8.fc24.s390x
libtheora-1.1.1-14.fc24.s390x
xml-common-0.6.3-44.fc24.noarch
autoconf-2.69-22.fc24.noarch
libXt-1.1.5-3.fc24.s390x
kbd-legacy-2.0.3-3.fc24.noarch
ghostscript-fonts-5.50-35.fc24.noarch
libXevie-1.0.3-11.fc24.s390x
libcap-2.25-2.fc25.s390x
mpfr-3.1.5-1.fc25.s390x
perl-Carp-1.40-365.fc25.noarch
libmnl-1.0.4-1.fc25.s390x
perl-Unicode-EastAsianWidth-1.33-8.fc25.noarch
libwayland-cursor-1.12.0-1.fc25.s390x
python-krbV-1.0.90-12.fc25.s390x
python2-urllib3-1.15.1-3.fc25.noarch
fipscheck-1.4.1-11.fc25.s390x
libndp-1.6-1.fc25.s390x
gnupg2-2.1.13-2.fc25.s390x
libXfixes-5.0.3-1.fc25.s390x
adwaita-icon-theme-3.22.0-1.fc25.noarch
dconf-0.26.0-1.fc25.s390x
ncurses-devel-6.0-6.20160709.fc25.s390x
dejagnu-1.6-1.fc25.noarch
libstdc++-devel-6.3.1-1.fc25.s390x
device-mapper-1.02.136-3.fc25.s390x
subversion-1.9.5-1.fc25.s390x
libtool-ltdl-2.4.6-13.fc25.s390x
libevent-2.0.22-1.fc25.s390x
atk-devel-2.22.0-1.fc25.s390x
libev-4.24-1.fc25.s390x
xorg-x11-fonts-Type1-7.5-16.fc24.noarch
brlapi-devel-0.6.5-2.fc25.s390x
pulseaudio-libs-10.0-2.fc25.s390x
glib2-2.50.3-1.fc25.s390x
python2-rpm-4.13.0.1-1.fc25.s390x
dnf-1.1.10-6.fc25.noarch
texlive-metafont-bin-svn40987-33.20160520.fc25.1.s390x
texlive-xkeyval-svn35741.2.7a-33.fc25.1.noarch
texlive-euler-svn17261.2.5-33.fc25.1.noarch
texlive-mptopdf-svn41282-33.fc25.1.noarch
texlive-wasy-svn35831.0-33.fc25.1.noarch
texlive-avantgar-svn31835.0-33.fc25.1.noarch
texlive-eurosym-svn17265.1.4_subrfix-33.fc25.1.noarch
texlive-knuth-lib-svn35820.0-33.fc25.1.noarch
texlive-parallel-svn15878.0-33.fc25.1.noarch
texlive-texlive-msg-translations-svn41431-33.fc25.1.noarch
texlive-latex-svn40218-33.fc25.1.noarch
texlive-lualatex-math-svn40621-33.fc25.1.noarch
texlive-auto-pst-pdf-svn23723.0.6-33.fc25.1.noarch
texlive-powerdot-svn38984-33.fc25.1.noarch
texlive-wasysym-svn15878.2.0-33.fc25.1.noarch
ImageMagick-libs-6.9.3.0-6.fc25.s390x
geoclue2-2.4.5-1.fc25.s390x
perl-IO-Socket-IP-0.39-1.fc25.noarch
python2-pyasn1-0.2.3-1.fc25.noarch
libglvnd-0.2.999-14.20170308git8e6e102.fc25.s390x
libglvnd-gles-0.2.999-14.20170308git8e6e102.fc25.s390x
gdk-pixbuf2-devel-2.36.6-1.fc25.s390x
at-spi2-core-devel-2.22.1-1.fc25.s390x
libacl-2.2.52-13.fc25.s390x
perl-libs-5.24.2-387.fc25.s390x
mesa-libglapi-17.0.5-3.fc25.s390x
kernel-headers-4.11.10-200.fc25.s390x
p11-kit-devel-0.23.2-4.fc25.s390x
python3-rpmconf-1.0.19-1.fc25.noarch
sqlite-3.14.2-2.fc25.s390x
pcre-utf32-8.41-1.fc25.s390x
libX11-common-1.6.5-1.fc25.noarch
coreutils-common-8.25-17.fc25.s390x
mesa-libEGL-17.0.5-3.fc25.s390x
nss-3.30.2-1.1.fc25.s390x
systemtap-runtime-3.1-5.fc25.s390x
NetworkManager-glib-1.4.4-5.fc25.s390x
audit-2.7.7-1.fc25.s390x
perl-Time-HiRes-1.9742-1.fc25.s390x
libsolv-0.6.28-1.fc25.s390x
gpg-pubkey-34ec9cba-54e38751
gpg-pubkey-030d5aed-55b577f0
basesystem-11-2.fc24.noarch
libmpc-1.0.2-5.fc24.s390x
libunistring-0.9.4-3.fc24.s390x
libmodman-2.0.1-12.fc24.s390x
lsscsi-0.28-3.fc24.s390x
kbd-misc-2.0.3-3.fc24.noarch
kmod-23-1.fc25.s390x
newt-0.52.19-2.fc25.s390x
perl-Text-Unidecode-1.27-3.fc25.noarch
plymouth-core-libs-0.9.3-0.6.20160620git0e65b86c.fc25.s390x
which-2.21-1.fc25.s390x
python3-slip-0.6.4-4.fc25.noarch
python3-systemd-232-1.fc25.s390x
python-lockfile-0.11.0-4.fc25.noarch
python2-requests-2.10.0-4.fc25.noarch
libnghttp2-1.13.0-2.fc25.s390x
python-urlgrabber-3.10.1-9.fc25.noarch
iputils-20161105-1.fc25.s390x
rest-0.8.0-1.fc25.s390x
adwaita-cursor-theme-3.22.0-1.fc25.noarch
authconfig-6.2.10-14.fc25.s390x
automake-1.15-7.fc25.noarch
shared-mime-info-1.8-1.fc25.s390x
pigz-2.3.4-1.fc25.s390x
device-mapper-libs-1.02.136-3.fc25.s390x
dnsmasq-2.76-2.fc25.s390x
fedora-packager-0.6.0.1-1.fc25.noarch
gcc-c++-6.3.1-1.fc25.s390x
libwebp-0.5.2-1.fc25.s390x
boost-system-1.60.0-10.fc25.s390x
libasyncns-0.8-10.fc24.s390x
libXau-devel-1.0.8-6.fc24.s390x
libverto-libev-0.2.6-6.fc24.s390x
python3-html5lib-0.999-9.fc25.noarch
ttmkfdir-3.0.9-48.fc24.s390x
pulseaudio-libs-glib2-10.0-2.fc25.s390x
wpa_supplicant-2.6-1.fc25.s390x
texlive-lib-2016-33.20160520.fc25.s390x
libXi-devel-1.7.9-1.fc25.s390x
python3-distro-1.0.3-1.fc25.noarch
rpm-plugin-systemd-inhibit-4.13.0.1-1.fc25.s390x
texlive-texlive-common-doc-svn40682-33.fc25.1.noarch
packagedb-cli-2.14.1-1.fc25.noarch
texlive-metafont-svn40793-33.fc25.1.noarch
texlive-tools-svn40934-33.fc25.1.noarch
texlive-enumitem-svn24146.3.5.2-33.fc25.1.noarch
texlive-mptopdf-bin-svn18674.0-33.20160520.fc25.1.noarch
texlive-underscore-svn18261.0-33.fc25.1.noarch
texlive-anysize-svn15878.0-33.fc25.1.noarch
texlive-euenc-svn19795.0.1h-33.fc25.1.noarch
texlive-kastrup-svn15878.0-33.fc25.1.noarch
texlive-paralist-svn39247-33.fc25.1.noarch
texlive-texlive-en-svn41185-33.fc25.1.noarch
texlive-tipa-svn29349.1.3-33.fc25.1.noarch
texlive-currfile-svn40725-33.fc25.1.noarch
texlive-pst-node-svn40743-33.fc25.1.noarch
texlive-pst-slpe-svn24391.1.31-33.fc25.1.noarch
texlive-typehtml-svn17134.0-33.fc25.1.noarch
SDL2-devel-2.0.5-3.fc25.s390x
libcroco-0.6.11-3.fc25.s390x
bluez-libs-devel-5.44-1.fc25.s390x
kernel-4.10.8-200.fc25.s390x
expat-2.2.1-1.fc25.s390x
system-python-libs-3.5.3-6.fc25.s390x
pcre2-10.23-8.fc25.s390x
firewalld-filesystem-0.4.4.5-1.fc25.noarch
pcre-cpp-8.41-1.fc25.s390x
python3-firewall-0.4.4.5-1.fc25.noarch
freetype-devel-2.6.5-9.fc25.s390x
pcre-utf16-8.41-1.fc25.s390x
linux-firmware-20170605-74.git37857004.fc25.noarch
distribution-gpg-keys-1.12-1.fc25.noarch
kernel-modules-4.11.10-200.fc25.s390x
gnutls-devel-3.5.14-1.fc25.s390x
systemtap-devel-3.1-5.fc25.s390x
java-1.8.0-openjdk-1.8.0.131-1.b12.fc25.s390x
polkit-0.113-8.fc25.s390x
perl-SelfLoader-1.23-387.fc25.noarch
libdb-utils-5.3.28-24.fc25.s390x
fontpackages-filesystem-1.44-17.fc24.noarch
groff-base-1.22.3-8.fc24.s390x
ilmbase-2.2.0-5.fc24.s390x
OpenEXR-libs-2.2.0-5.fc24.s390x
hesiod-3.2.1-6.fc24.s390x
sysfsutils-2.1.0-19.fc24.s390x
ocaml-srpm-macros-2-4.fc24.noarch
mailx-12.5-19.fc24.s390x
ncurses-libs-6.0-6.20160709.fc25.s390x
ipset-libs-6.29-1.fc25.s390x
gmp-devel-6.1.1-1.fc25.s390x
python-pip-8.1.2-2.fc25.noarch
harfbuzz-1.3.2-1.fc25.s390x
python2-iniparse-0.4-20.fc25.noarch
python3-iniparse-0.4-20.fc25.noarch
python3-kickstart-2.32-1.fc25.noarch
perl-Net-SSLeay-1.78-1.fc25.s390x
drpm-0.3.0-3.fc25.s390x
glib-networking-2.50.0-1.fc25.s390x
webkitgtk3-2.4.11-3.fc25.s390x
libXaw-1.0.13-4.fc25.s390x
xorg-x11-font-utils-7.5-32.fc25.s390x
hardlink-1.1-1.fc25.s390x
libcom_err-1.43.3-1.fc25.s390x
python2-dateutil-2.6.0-1.fc25.noarch
libXpm-3.5.12-1.fc25.s390x
python2-smmap-2.0.1-1.fc25.noarch
poppler-data-0.4.7-6.fc25.noarch
librbd1-10.2.4-2.fc25.s390x
perl-Digest-MD5-2.55-2.fc25.s390x
wayland-protocols-devel-1.7-1.fc25.noarch
texi2html-5.0-4.fc24.noarch
libxkbcommon-0.7.1-1.fc25.s390x
libuuid-devel-2.28.2-2.fc25.s390x
libcacard-2.5.3-1.fc25.s390x
libwmf-lite-0.2.8.4-50.fc25.s390x
texlive-tetex-svn41059-33.fc25.1.noarch
texlive-thumbpdf-svn34621.3.16-33.fc25.1.noarch
texlive-carlisle-svn18258.0-33.fc25.1.noarch
texlive-makeindex-bin-svn40473-33.20160520.fc25.1.s390x
texlive-pdftex-svn41149-33.fc25.1.noarch
texlive-csquotes-svn39538-33.fc25.1.noarch
texlive-courier-svn35058.0-33.fc25.1.noarch
texlive-helvetic-svn31835.0-33.fc25.1.noarch
texlive-mfnfss-svn19410.0-33.fc25.1.noarch
texlive-sepnum-svn20186.2.0-33.fc25.1.noarch
texlive-utopia-svn15878.0-33.fc25.1.noarch
texlive-luatexbase-svn38550-33.fc25.1.noarch
texlive-pst-3d-svn17257.1.10-33.fc25.1.noarch
texlive-latex-bin-bin-svn14050.0-33.20160520.fc25.1.noarch
texlive-l3experimental-svn41163-33.fc25.1.noarch
net-tools-2.0-0.40.20160329git.fc25.s390x
perl-Pod-Perldoc-3.28-1.fc25.noarch
openssl-1.0.2k-1.fc25.s390x
man-pages-4.06-4.fc25.noarch
python3-magic-5.29-4.fc25.noarch
libxml2-2.9.4-2.fc25.s390x
nss-softokn-3.30.2-1.0.fc25.s390x
p11-kit-trust-0.23.2-4.fc25.s390x
emacs-filesystem-25.2-2.fc25.noarch
python3-dateutil-2.6.0-1.fc25.noarch
perl-threads-shared-1.57-1.fc25.s390x
libnotify-0.7.7-1.fc25.s390x
unzip-6.0-32.fc25.s390x
pango-devel-1.40.5-1.fc25.s390x
libdrm-2.4.81-1.fc25.s390x
python-beautifulsoup4-4.6.0-1.fc25.noarch
libcurl-7.51.0-7.fc25.s390x
dhcp-client-4.3.5-3.fc25.s390x
python2-fedora-0.9.0-6.fc25.noarch
emacs-25.2-2.fc25.s390x
gdb-7.12.1-48.fc25.s390x
expat-devel-2.2.1-1.fc25.s390x
gpg-pubkey-95a43f54-5284415a
dejavu-fonts-common-2.35-3.fc24.noarch
libSM-1.2.2-4.fc24.s390x
diffutils-3.3-13.fc24.s390x
libogg-1.3.2-5.fc24.s390x
hunspell-en-US-0.20140811.1-5.fc24.noarch
libdaemon-0.14-10.fc24.s390x
patch-2.7.5-3.fc24.s390x
libsysfs-2.1.0-19.fc24.s390x
procmail-3.22-39.fc24.s390x
libXdamage-1.1.4-8.fc24.s390x
libotf-0.9.13-7.fc24.s390x
urw-fonts-2.4-22.fc24.noarch
crontabs-1.11-12.20150630git.fc24.noarch
ppp-2.4.7-9.fc24.s390x
cyrus-sasl-2.1.26-26.2.fc24.s390x
zlib-devel-1.2.8-10.fc24.s390x
time-1.7-49.fc24.s390x
gpg-pubkey-fdb19c98-56fd6333
libcap-ng-0.7.8-1.fc25.s390x
binutils-2.26.1-1.fc25.s390x
lcms2-2.8-2.fc25.s390x
libcomps-0.1.7-5.fc25.s390x
apr-1.5.2-4.fc25.s390x
perl-constant-1.33-367.fc25.noarch
perl-Data-Dumper-2.161-1.fc25.s390x
ipcalc-0.1.8-1.fc25.s390x
gmp-c++-6.1.1-1.fc25.s390x
fontconfig-2.12.1-1.fc25.s390x
enchant-1.6.0-14.fc25.s390x
pyliblzma-0.5.3-16.fc25.s390x
libsepol-devel-2.5-10.fc25.s390x
python3-ordered-set-2.0.0-4.fc25.noarch
python-ipaddress-1.0.16-3.fc25.noarch
python2-kerberos-1.2.5-1.fc25.s390x
python2-pysocks-1.5.6-5.fc25.noarch
fipscheck-lib-1.4.1-11.fc25.s390x
libatomic_ops-7.4.4-1.fc25.s390x
net-snmp-agent-libs-5.7.3-13.fc25.s390x
dracut-044-78.fc25.s390x
python2-pygpgme-0.3-18.fc25.s390x
orc-0.4.26-1.fc25.s390x
yum-utils-1.1.31-511.fc25.noarch
libXrender-0.9.10-1.fc25.s390x
libXrandr-1.5.1-1.fc25.s390x
go-srpm-macros-2-7.fc25.noarch
gnupg2-smime-2.1.13-2.fc25.s390x
guile-devel-2.0.13-1.fc25.s390x
uboot-tools-2016.09.01-2.fc25.s390x
pykickstart-2.32-1.fc25.noarch
python-bunch-1.0.1-9.fc25.noarch
perl-generators-1.10-1.fc25.noarch
perl-Mozilla-CA-20160104-3.fc25.noarch
bzip2-libs-1.0.6-21.fc25.s390x
libpng-1.6.27-1.fc25.s390x
desktop-file-utils-0.23-2.fc25.s390x
python2-cccolutils-1.4-1.fc25.s390x
python2-lxml-3.7.2-1.fc25.s390x
redhat-rpm-config-45-1.fc25.noarch
device-mapper-event-libs-1.02.136-3.fc25.s390x
lvm2-libs-2.02.167-3.fc25.s390x
python2-gitdb-2.0.0-1.fc25.noarch
gcc-gfortran-6.3.1-1.fc25.s390x
libselinux-python-2.5-13.fc25.s390x
openjpeg2-2.1.2-3.fc25.s390x
boost-thread-1.60.0-10.fc25.s390x
librbd-devel-10.2.4-2.fc25.s390x
libXcursor-devel-1.1.14-6.fc24.s390x
latex2html-2012-7.fc24.noarch
lksctp-tools-1.0.16-5.fc24.s390x
libfdt-1.4.2-1.fc25.s390x
libXft-devel-2.3.2-4.fc24.s390x
libattr-devel-2.4.47-16.fc24.s390x
libiscsi-devel-1.15.0-2.fc24.s390x
gettext-0.19.8.1-3.fc25.s390x
libjpeg-turbo-devel-1.5.1-0.fc25.s390x
pulseaudio-libs-devel-10.0-2.fc25.s390x
libepoxy-devel-1.4.1-1.fc25.s390x
krb5-libs-1.14.4-7.fc25.s390x
libmount-2.28.2-2.fc25.s390x
python3-decorator-4.0.11-1.fc25.noarch
rpm-plugin-selinux-4.13.0.1-1.fc25.s390x
tzdata-java-2017b-1.fc25.noarch
python-srpm-macros-3-12.fc25.noarch
libsmartcols-2.28.2-2.fc25.s390x
kernel-core-4.10.5-200.fc25.s390x
kernel-modules-4.10.5-200.fc25.s390x
texlive-kpathsea-svn41139-33.fc25.1.noarch
texlive-amsmath-svn41561-33.fc25.1.noarch
texlive-thumbpdf-bin-svn6898.0-33.20160520.fc25.1.noarch
texlive-psnfss-svn33946.9.2a-33.fc25.1.noarch
texlive-subfig-svn15878.1.3-33.fc25.1.noarch
texlive-fancybox-svn18304.1.4-33.fc25.1.noarch
texlive-lua-alt-getopt-svn29349.0.7.0-33.fc25.1.noarch
texlive-natbib-svn20668.8.31b-33.fc25.1.noarch
texlive-pdftex-bin-svn40987-33.20160520.fc25.1.s390x
texlive-xdvi-svn40768-33.fc25.1.noarch
texlive-crop-svn15878.1.5-33.fc25.1.noarch
texlive-babel-english-svn30264.3.3p-33.fc25.1.noarch
texlive-cmextra-svn32831.0-33.fc25.1.noarch
texlive-fancyhdr-svn15878.3.1-33.fc25.1.noarch
texlive-luatex-svn40963-33.fc25.1.noarch
texlive-knuth-local-svn38627-33.fc25.1.noarch
texlive-mflogo-font-svn36898.1.002-33.fc25.1.noarch
texlive-parskip-svn19963.2.0-33.fc25.1.noarch
texlive-section-svn20180.0-33.fc25.1.noarch
texlive-textcase-svn15878.0-33.fc25.1.noarch
texlive-updmap-map-svn41159-33.fc25.1.noarch
texlive-attachfile-svn38830-33.fc25.1.noarch
libglvnd-glx-0.2.999-14.20170308git8e6e102.fc25.s390x
libglvnd-core-devel-0.2.999-14.20170308git8e6e102.fc25.s390x
python-magic-5.29-4.fc25.noarch
glibc-common-2.24-9.fc25.s390x
sqlite-libs-3.14.2-2.fc25.s390x
libtiff-4.0.8-1.fc25.s390x
libdb-5.3.28-24.fc25.s390x
glusterfs-client-xlators-3.10.4-1.fc25.s390x
nss-util-devel-3.30.2-1.0.fc25.s390x
gnutls-3.5.14-1.fc25.s390x
bind-license-9.10.5-2.P2.fc25.noarch
mesa-libGLES-17.0.5-3.fc25.s390x
python3-requests-kerberos-0.10.0-2.fc25.noarch
python3-pyOpenSSL-16.2.0-1.fc25.noarch
perl-threads-2.16-1.fc25.s390x
cryptsetup-libs-1.7.5-1.fc25.s390x
vim-minimal-8.0.705-1.fc25.s390x
netpbm-10.79.00-1.fc25.s390x
qrencode-libs-3.4.4-1.fc25.s390x
mariadb-config-10.1.24-3.fc25.s390x
gstreamer1-plugins-base-1.10.5-1.fc25.s390x
elfutils-default-yama-scope-0.169-1.fc25.noarch
glusterfs-3.10.4-1.fc25.s390x
systemd-udev-231-17.fc25.s390x
python2-koji-1.13.0-2.fc25.noarch
unbound-libs-1.6.3-1.fc25.s390x
openldap-2.4.44-11.fc25.s390x
koji-1.13.0-2.fc25.noarch
bind99-libs-9.9.10-2.P3.fc25.s390x
libcurl-devel-7.51.0-7.fc25.s390x
mesa-libGL-devel-17.0.5-3.fc25.s390x
python2-sssdconfig-1.15.2-5.fc25.noarch
webkitgtk4-plugin-process-gtk2-2.16.5-1.fc25.s390x
graphite2-devel-1.3.10-1.fc25.s390x
systemtap-sdt-devel-3.1-5.fc25.s390x
iproute-tc-4.11.0-1.fc25.s390x
libarchive-3.2.2-2.fc25.s390x
publicsuffix-list-dafsa-20170424-1.fc25.noarch
texlive-luaotfload-svn40902-33.fc25.1.noarch
texlive-unicode-math-svn38462-33.fc25.1.noarch
texlive-fancyvrb-svn18492.2.8-33.fc25.1.noarch
texlive-pst-pdf-bin-svn7838.0-33.20160520.fc25.1.noarch
texlive-amscls-svn36804.0-33.fc25.1.noarch
texlive-ltxmisc-svn21927.0-33.fc25.1.noarch
texlive-breqn-svn38099.0.98d-33.fc25.1.noarch
texlive-xetex-def-svn40327-33.fc25.1.noarch
openssh-server-7.4p1-4.fc25.s390x
sendmail-8.15.2-8.fc25.s390x
tzdata-2017b-1.fc25.noarch
hunspell-1.4.1-2.fc25.s390x
gpg-pubkey-8e1431d5-53bcbac7
zlib-1.2.8-10.fc24.s390x
sed-4.2.2-15.fc24.s390x
psmisc-22.21-8.fc24.s390x
gpm-libs-1.20.7-9.fc24.s390x
zip-3.0-16.fc24.s390x
libyubikey-1.13-2.fc24.s390x
sg3_utils-libs-1.41-3.fc24.s390x
polkit-pkla-compat-0.1-7.fc24.s390x
passwd-0.79-8.fc24.s390x
trousers-0.3.13-6.fc24.s390x
grubby-8.40-3.fc24.s390x
rootfiles-8.1-19.fc24.noarch
nettle-3.3-1.fc25.s390x
libksba-1.3.5-1.fc25.s390x
perl-Text-ParseWords-3.30-365.fc25.noarch
perl-PathTools-3.63-366.fc25.s390x
perl-File-Temp-0.23.04-365.fc25.noarch
fuse-libs-2.9.7-1.fc25.s390x
perl-Pod-Escapes-1.07-365.fc25.noarch
perl-Term-ANSIColor-4.05-2.fc25.noarch
perl-URI-1.71-5.fc25.noarch
libXfont-1.5.2-1.fc25.s390x
python-six-1.10.0-3.fc25.noarch
dbus-glib-0.108-1.fc25.s390x
gobject-introspection-1.50.0-1.fc25.s390x
libpwquality-1.3.0-6.fc25.s390x
python-gobject-base-3.22.0-1.fc25.s390x
python-html5lib-0.999-9.fc25.noarch
python3-dbus-1.2.4-2.fc25.s390x
python3-chardet-2.3.0-1.fc25.noarch
python3-urllib3-1.15.1-3.fc25.noarch
python-offtrac-0.1.0-7.fc25.noarch
python2-cryptography-1.5.3-3.fc25.s390x
python2-requests-kerberos-0.10.0-2.fc25.noarch
libserf-1.3.9-1.fc25.s390x
libdatrie-0.2.9-3.fc25.s390x
s390utils-base-1.36.0-1.fc25.s390x
kpartx-0.4.9-83.fc25.s390x
s390utils-cpuplugd-1.36.0-1.fc25.s390x
s390utils-osasnmpd-1.36.0-1.fc25.s390x
python-dnf-plugins-extras-common-0.0.12-4.fc25.noarch
fpc-srpm-macros-1.0-1.fc25.noarch
libuser-0.62-4.fc25.s390x
man-db-2.7.5-3.fc25.s390x
python-systemd-doc-232-1.fc25.s390x
lz4-1.7.5-1.fc25.s390x
tar-1.29-3.fc25.s390x
bodhi-client-0.9.12.2-6.fc25.noarch
cairo-1.14.8-1.fc25.s390x
gcc-go-6.3.1-1.fc25.s390x
cracklib-dicts-2.9.6-4.fc25.s390x
libselinux-python3-2.5-13.fc25.s390x
python2-enchant-1.6.8-1.fc25.noarch
boost-iostreams-1.60.0-10.fc25.s390x
userspace-rcu-0.9.2-2.fc25.s390x
libXext-devel-1.3.3-4.fc24.s390x
libXrandr-devel-1.5.1-1.fc25.s390x
perl-XML-XPath-1.39-1.fc25.noarch
python3-lxml-3.7.2-1.fc25.s390x
libiscsi-1.15.0-2.fc24.s390x
fontconfig-devel-2.12.1-1.fc25.s390x
libfdt-devel-1.4.2-1.fc25.s390x
ceph-devel-compat-10.2.4-2.fc25.s390x
zlib-static-1.2.8-10.fc24.s390x
chrpath-0.16-3.fc24.s390x
info-6.1-4.fc25.s390x
iptables-libs-1.6.0-3.fc25.s390x
libfdisk-2.28.2-2.fc25.s390x
dnf-plugins-core-0.1.21-5.fc25.noarch
perl-Storable-2.56-368.fc25.s390x
python2-decorator-4.0.11-1.fc25.noarch
libnetfilter_conntrack-1.0.6-2.fc25.s390x
texlive-texlive.infra-bin-svn40312-33.20160520.fc25.1.s390x
texlive-ifluatex-svn41346-33.fc25.1.noarch
texlive-fp-svn15878.0-33.fc25.1.noarch
texlive-latex-fonts-svn28888.0-33.fc25.1.noarch
texlive-bibtex-bin-svn40473-33.20160520.fc25.1.s390x
texlive-glyphlist-svn28576.0-33.fc25.1.noarch
texlive-marvosym-svn29349.2.2a-33.fc25.1.noarch
texlive-tex-bin-svn40987-33.20160520.fc25.1.s390x
texlive-texconfig-svn40768-33.fc25.1.noarch
texlive-wasy2-ps-svn35830.0-33.fc25.1.noarch
texlive-psfrag-svn15878.3.04-33.fc25.1.noarch
texlive-charter-svn15878.0-33.fc25.1.noarch
texlive-ec-svn25033.1.0-33.fc25.1.noarch
texlive-lineno-svn21442.4.41-33.fc25.1.noarch
texlive-hyphen-base-svn41138-33.fc25.1.noarch
texlive-manfnt-font-svn35799.0-33.fc25.1.noarch
texlive-ncntrsbk-svn31835.0-33.fc25.1.noarch
texlive-pst-math-svn34786.0.63-33.fc25.1.noarch
texlive-symbol-svn31835.0-33.fc25.1.noarch
texlive-environ-svn33821.0.3-33.fc25.1.noarch
texlive-algorithms-svn38085.0.1-33.fc25.1.noarch
kernel-core-4.10.8-200.fc25.s390x
libglvnd-devel-0.2.999-14.20170308git8e6e102.fc25.s390x
python3-hawkey-0.6.4-3.fc25.s390x
glibc-all-langpacks-2.24-9.fc25.s390x
freetype-2.6.5-9.fc25.s390x
mesa-libwayland-egl-17.0.5-3.fc25.s390x
libicu-57.1-5.fc25.s390x
nspr-devel-4.14.0-2.fc25.s390x
libnl3-cli-3.2.29-3.fc25.s390x
cups-libs-2.2.0-9.fc25.s390x
bind-libs-lite-9.10.5-2.P2.fc25.s390x
ca-certificates-2017.2.14-1.0.fc25.noarch
python3-kerberos-1.2.5-1.fc25.s390x
python3-cryptography-1.5.3-3.fc25.s390x
perl-IO-1.36-387.fc25.s390x
dhcp-libs-4.3.5-3.fc25.s390x
rsync-3.1.2-4.fc25.s390x
make-4.1-6.fc25.s390x
pcre2-devel-10.23-8.fc25.s390x
quota-4.03-8.fc25.s390x
libX11-devel-1.6.5-1.fc25.s390x
ghostscript-9.20-9.fc25.s390x
dbus-1.11.14-1.fc25.s390x
rpcbind-0.2.4-6.rc2.fc25.s390x
pyOpenSSL-16.2.0-1.fc25.noarch
glusterfs-devel-3.10.4-1.fc25.s390x
nss-tools-3.30.2-1.1.fc25.s390x
python3-pycurl-7.43.0-6.fc25.s390x
bind99-license-9.9.10-2.P3.fc25.noarch
curl-7.51.0-7.fc25.s390x
glusterfs-api-devel-3.10.4-1.fc25.s390x
python-firewall-0.4.4.5-1.fc25.noarch
netpbm-progs-10.79.00-1.fc25.s390x
libsndfile-1.0.28-3.fc25.s390x
python3-sssdconfig-1.15.2-5.fc25.noarch
wget-1.18-3.fc25.s390x
libsemanage-2.5-9.fc25.s390x
telnet-0.17-68.fc25.s390x
texlive-ifplatform-svn21156.0.4-33.fc25.1.noarch
texlive-eso-pic-svn37925.2.0g-33.fc25.1.noarch
texlive-xcolor-svn41044-33.fc25.1.noarch
texlive-pst-eps-svn15878.1.0-33.fc25.1.noarch
texlive-pst-text-svn15878.1.00-33.fc25.1.noarch
texlive-rotating-svn16832.2.16b-33.fc25.1.noarch
texlive-pdfpages-svn40638-33.fc25.1.noarch
texlive-cm-super-svn15878.0-33.fc25.1.noarch
texlive-xetex-svn41438-33.fc25.1.noarch
dnf-yum-1.1.10-6.fc25.noarch
libseccomp-devel-2.3.2-1.fc25.s390x
gpgme-1.8.0-10.fc25.s390x
apr-util-1.5.4-3.fc24.s390x
jbigkit-libs-2.1-5.fc24.s390x
pixman-0.34.0-2.fc24.s390x
dwz-0.12-2.fc24.s390x
expect-5.45-22.fc24.s390x
libsigsegv-2.10-10.fc24.s390x
fakeroot-libs-1.20.2-4.fc24.s390x
m17n-lib-1.7.0-5.fc24.s390x
libverto-0.2.6-6.fc24.s390x
libXmu-1.1.2-4.fc24.s390x
libXcursor-1.1.14-6.fc24.s390x
python-kitchen-1.2.4-2.fc24.noarch
fakeroot-1.20.2-4.fc24.s390x
blktrace-1.1.0-3.fc24.s390x
usermode-1.111-8.fc24.s390x
kbd-2.0.3-3.fc24.s390x
libaio-devel-0.3.110-6.fc24.s390x
web-assets-filesystem-5-4.fc24.noarch
libgpg-error-1.24-1.fc25.s390x
findutils-4.6.0-8.fc25.s390x
libassuan-2.4.3-1.fc25.s390x
libusbx-1.0.21-1.fc25.s390x
libxslt-1.1.28-13.fc25.s390x
libmetalink-0.1.3-1.fc25.s390x
perl-MIME-Base64-3.15-365.fc25.s390x
ncurses-6.0-6.20160709.fc25.s390x
libwayland-server-1.12.0-1.fc25.s390x
perl-Fedora-VSP-0.001-4.fc25.noarch
perl-libintl-perl-1.26-1.fc25.s390x
shadow-utils-4.2.1-11.fc25.s390x
atk-2.22.0-1.fc25.s390x
pam-1.3.0-1.fc25.s390x
harfbuzz-icu-1.3.2-1.fc25.s390x
libsecret-0.18.5-2.fc25.s390x
s390utils-iucvterm-1.36.0-1.fc25.s390x
python3-requests-2.10.0-4.fc25.noarch
pyusb-1.0.0-2.fc25.noarch
python-enum34-1.0.4-6.fc25.noarch
pyxattr-0.5.3-8.fc25.s390x
libbabeltrace-1.4.0-3.fc25.s390x
libthai-0.1.25-1.fc25.s390x
deltarpm-3.6-17.fc25.s390x
s390utils-mon_statd-1.36.0-1.fc25.s390x
device-mapper-multipath-0.4.9-83.fc25.s390x
python3-pygpgme-0.3-18.fc25.s390x
libreport-filesystem-2.8.0-1.fc25.s390x
ghc-srpm-macros-1.4.2-4.fc25.noarch
rpmdevtools-8.9-1.fc25.noarch
python-dnf-plugins-extras-migrate-0.0.12-4.fc25.noarch
perl-IO-Socket-SSL-2.038-1.fc25.noarch
perl-File-ShareDir-1.102-7.fc25.noarch
tcl-8.6.6-1.fc25.s390x
bzip2-1.0.6-21.fc25.s390x
libss-1.43.3-1.fc25.s390x
libselinux-utils-2.5-13.fc25.s390x
python3-enchant-1.6.8-1.fc25.noarch
python2-dockerfile-parse-0.0.5-7.fc25.noarch
systemd-bootchart-231-2.fc25.s390x
gcc-objc-6.3.1-1.fc25.s390x
e2fsprogs-1.43.3-1.fc25.s390x
libstdc++-static-6.3.1-1.fc25.s390x
libpng-devel-1.6.27-1.fc25.s390x
perl-XML-Parser-2.44-5.fc25.s390x
lttng-ust-2.8.1-2.fc25.s390x
libXfixes-devel-5.0.3-1.fc25.s390x
libXcomposite-devel-0.4.4-8.fc24.s390x
python3-javapackages-4.7.0-6.1.fc25.noarch
libcephfs_jni-devel-10.2.4-2.fc25.s390x
keyutils-libs-devel-1.5.9-8.fc24.s390x
harfbuzz-devel-1.3.2-1.fc25.s390x
libidn-devel-1.33-1.fc25.s390x
libnfs-1.9.8-2.fc24.s390x
libssh2-devel-1.8.0-1.fc25.s390x
qemu-sanity-check-nodeps-1.1.5-5.fc24.s390x
alsa-lib-devel-1.1.1-2.fc25.s390x
libpsl-0.17.0-1.fc25.s390x
libseccomp-2.3.2-1.fc25.s390x
copy-jdk-configs-2.2-2.fc25.noarch
json-glib-1.2.6-1.fc25.s390x
python2-dnf-1.1.10-6.fc25.noarch
python2-GitPython-2.1.3-1.fc25.noarch
texlive-tetex-bin-svn36770.0-33.20160520.fc25.1.noarch
texlive-amsfonts-svn29208.3.04-33.fc25.1.noarch
texlive-babel-svn40706-33.fc25.1.noarch
texlive-colortbl-svn29803.v1.0a-33.fc25.1.noarch
texlive-babelbib-svn25245.1.31-33.fc25.1.noarch
texlive-footmisc-svn23330.5.5b-33.fc25.1.noarch
texlive-makeindex-svn40768-33.fc25.1.noarch
texlive-plain-svn40274-33.fc25.1.noarch
texlive-texconfig-bin-svn29741.0-33.20160520.fc25.1.noarch
texlive-zapfding-svn31835.0-33.fc25.1.noarch
texlive-microtype-svn41127-33.fc25.1.noarch
texlive-bookman-svn31835.0-33.fc25.1.noarch
texlive-dvisvgm-def-svn41011-33.fc25.1.noarch
texlive-finstrut-svn21719.0.5-33.fc25.1.noarch
texlive-hyph-utf8-svn41189-33.fc25.1.noarch
texlive-lualibs-svn40370-33.fc25.1.noarch
kernel-modules-4.10.8-200.fc25.s390x
python2-hawkey-0.6.4-3.fc25.s390x
glibc-2.24-9.fc25.s390x
elfutils-libelf-0.169-1.fc25.s390x
libsoup-2.56.0-3.fc25.s390x
libnl3-3.2.29-3.fc25.s390x
gstreamer1-1.10.5-1.fc25.s390x
polkit-libs-0.113-8.fc25.s390x
libtirpc-1.0.2-0.fc25.s390x
emacs-common-25.2-2.fc25.s390x
libteam-1.27-1.fc25.s390x
python3-3.5.3-6.fc25.s390x
python3-pyasn1-0.2.3-1.fc25.noarch
perl-File-Path-2.12-366.fc25.noarch
mesa-libwayland-egl-devel-17.0.5-3.fc25.s390x
libacl-devel-2.2.52-13.fc25.s390x
lua-libs-5.3.4-3.fc25.s390x
pcre2-utf32-10.23-8.fc25.s390x
quota-nls-4.03-8.fc25.noarch
gtk3-3.22.16-1.fc25.s390x
ghostscript-x11-9.20-9.fc25.s390x
systemd-231-17.fc25.s390x
glusterfs-api-3.10.4-1.fc25.s390x
glusterfs-extra-xlators-3.10.4-1.fc25.s390x
glusterfs-server-3.10.4-1.fc25.s390x
java-1.8.0-openjdk-headless-1.8.0.131-1.b12.fc25.s390x
git-2.9.4-1.fc25.s390x
dhcp-common-4.3.5-3.fc25.noarch
python2-rpkg-1.49-5.fc25.noarch
vte291-devel-0.46.2-1.fc25.s390x
python-devel-2.7.13-2.fc25.s390x
elfutils-0.169-1.fc25.s390x
lua-5.3.4-3.fc25.s390x
python3-beautifulsoup4-4.6.0-1.fc25.noarch
libmicrohttpd-0.9.55-1.fc25.s390x
screen-4.6.1-1.fc25.s390x
strace-4.18-1.fc25.s390x
texlive-mparhack-svn15878.1.4-33.fc25.1.noarch
texlive-pspicture-svn15878.0-33.fc25.1.noarch
texlive-soul-svn15878.2.4-33.fc25.1.noarch
texlive-trimspaces-svn15878.1.1-33.fc25.1.noarch
texlive-varwidth-svn24104.0.92-33.fc25.1.noarch
texlive-geometry-svn19716.5.6-33.fc25.1.noarch
texlive-memoir-svn41203-33.fc25.1.noarch
texlive-pgf-svn40966-33.fc25.1.noarch
texlive-pst-coil-svn37377.1.07-33.fc25.1.noarch
texlive-pst-plot-svn41242-33.fc25.1.noarch
texlive-latex-bin-svn41438-33.fc25.1.noarch
texlive-ucs-svn35853.2.2-33.fc25.1.noarch
texlive-ae-svn15878.1.4-33.fc25.1.noarch
texlive-xetex-bin-svn41091-33.20160520.fc25.1.s390x
fedora-upgrade-26.1-1.fc25.noarch
fedpkg-1.28-1.fc25.noarch
perl-Thread-Queue-3.12-1.fc25.noarch
cdparanoia-libs-10.2-21.fc24.s390x
ustr-1.0.4-21.fc24.s390x
libusb-0.1.5-7.fc24.s390x
readline-devel-6.3-8.fc24.s390x
chkconfig-1.8-1.fc25.s390x
avahi-libs-0.6.32-4.fc25.s390x
perl-Unicode-Normalize-1.25-365.fc25.s390x
perl-libnet-3.10-1.fc25.noarch
perl-podlators-4.09-1.fc25.noarch
dbus-python-1.2.4-2.fc25.s390x
libgnome-keyring-3.12.0-7.fc25.s390x
python-backports-1.0-8.fc25.s390x
python-pycparser-2.14-7.fc25.noarch
plymouth-scripts-0.9.3-0.6.20160620git0e65b86c.fc25.s390x
cronie-1.5.1-2.fc25.s390x
python2-librepo-1.7.18-3.fc25.s390x
libXv-1.0.11-1.fc25.s390x
python2-ndg_httpsclient-0.4.0-4.fc25.noarch
btrfs-progs-4.6.1-1.fc25.s390x
libgcc-6.3.1-1.fc25.s390x
libgomp-6.3.1-1.fc25.s390x
perl-Encode-2.88-5.fc25.s390x
cracklib-2.9.6-4.fc25.s390x
libobjc-6.3.1-1.fc25.s390x
gcc-6.3.1-1.fc25.s390x
python3-dnf-plugin-system-upgrade-0.7.1-4.fc25.noarch
boost-random-1.60.0-10.fc25.s390x
libref_array-0.1.5-29.fc25.s390x
libXrender-devel-0.9.10-1.fc25.s390x
javapackages-tools-4.7.0-6.1.fc25.noarch
keyutils-1.5.9-8.fc24.s390x
libcom_err-devel-1.43.3-1.fc25.s390x
lzo-minilzo-2.08-8.fc24.s390x
libusbx-devel-1.0.21-1.fc25.s390x
virglrenderer-devel-0.5.0-1.20160411git61846f92f.fc25.s390x
acpica-tools-20160831-1.fc25.s390x
grep-2.27-2.fc25.s390x
dnf-conf-1.1.10-6.fc25.noarch
crypto-policies-20160921-4.gitf3018dd.fc25.noarch
rpm-build-libs-4.13.0.1-1.fc25.s390x
libnfsidmap-0.27-1.fc25.s390x
SDL2-2.0.5-3.fc25.s390x
texlive-etex-pkg-svn39355-33.fc25.1.noarch
texlive-multido-svn18302.1.42-33.fc25.1.noarch
texlive-gsftopk-svn40768-33.fc25.1.noarch
texlive-pst-ovl-svn40873-33.fc25.1.noarch
texlive-ltabptch-svn17533.1.74d-33.fc25.1.noarch
texlive-cite-svn36428.5.5-33.fc25.1.noarch
texlive-fpl-svn15878.1.002-33.fc25.1.noarch
texlive-mathpazo-svn15878.1.003-33.fc25.1.noarch
texlive-rcs-svn15878.0-33.fc25.1.noarch
texlive-type1cm-svn21820.0-33.fc25.1.noarch
texlive-l3kernel-svn41246-33.fc25.1.noarch
texlive-hyperref-svn41396-33.fc25.1.noarch
texlive-pst-tree-svn24142.1.12-33.fc25.1.noarch
texlive-sansmathaccent-svn30187.0-33.fc25.1.noarch
texlive-dvipdfmx-bin-svn40273-33.20160520.fc25.1.s390x
texlive-zapfchan-svn31835.0-33.fc25.1.noarch
glib2-static-2.50.3-1.fc25.s390x
bash-completion-2.5-1.fc25.noarch
glusterfs-libs-3.10.4-1.fc25.s390x
p11-kit-0.23.2-4.fc25.s390x
hyphen-2.8.8-4.fc24.s390x
gnutls-c++-3.5.14-1.fc25.s390x
python3-idna-2.5-1.fc25.noarch
nss-softokn-devel-3.30.2-1.0.fc25.s390x
less-481-7.fc25.s390x
rpmconf-base-1.0.19-1.fc25.noarch
gtk2-2.24.31-2.fc25.s390x
mesa-libgbm-17.0.5-3.fc25.s390x
nfs-utils-2.1.1-5.rc4.fc25.s390x
perl-Git-2.9.4-1.fc25.noarch
mock-1.4.2-1.fc25.noarch
mc-4.8.19-5.fc25.s390x
pcre-static-8.41-1.fc25.s390x
bind-libs-9.10.5-2.P2.fc25.s390x
libproxy-0.4.15-2.fc25.s390x
gpg-pubkey-a29cb19c-53bcbba6
m4-1.4.17-9.fc24.s390x
liblockfile-1.09-4.fc24.s390x
sg3_utils-1.41-3.fc24.s390x
libXinerama-1.1.3-6.fc24.s390x
libXft-2.3.2-4.fc24.s390x
tcp_wrappers-libs-7.6-83.fc25.s390x
perl-Text-Tabs+Wrap-2013.0523-365.fc25.noarch
perl-Error-0.17024-7.fc25.noarch
perl-Term-Cap-1.17-365.fc25.noarch
perl-Pod-Usage-1.69-1.fc25.noarch
device-mapper-persistent-data-0.6.3-1.fc25.s390x
net-snmp-libs-5.7.3-13.fc25.s390x
python3-six-1.10.0-3.fc25.noarch
python3-pysocks-1.5.6-5.fc25.noarch
python-chardet-2.3.0-1.fc25.noarch
python2-cffi-1.7.0-2.fc25.s390x
gc-devel-7.4.4-1.fc25.s390x
plymouth-0.9.3-0.6.20160620git0e65b86c.fc25.s390x
ebtables-2.0.10-21.fc25.s390x
python3-librepo-1.7.18-3.fc25.s390x
net-snmp-5.7.3-13.fc25.s390x
at-spi2-atk-2.22.0-1.fc25.s390x
avahi-autoipd-0.6.32-4.fc25.s390x
libgo-6.3.1-1.fc25.s390x
cpp-6.3.1-1.fc25.s390x
pyparsing-2.1.10-1.fc25.noarch
python3-pyparsing-2.1.10-1.fc25.noarch
libcollection-0.7.0-29.fc25.s390x
libcephfs-devel-10.2.4-2.fc25.s390x
libXdamage-devel-1.1.4-8.fc24.s390x
libverto-devel-0.2.6-6.fc24.s390x
snappy-1.1.3-2.fc24.s390x
cairo-gobject-devel-1.14.8-1.fc25.s390x
cyrus-sasl-devel-2.1.26-26.2.fc24.s390x
libXi-1.7.9-1.fc25.s390x
texlive-base-2016-33.20160520.fc25.noarch
python3-rpm-4.13.0.1-1.fc25.s390x
texlive-booktabs-svn40846-33.fc25.1.noarch
texlive-lm-svn28119.2.004-33.fc25.1.noarch
texlive-gsftopk-bin-svn40473-33.20160520.fc25.1.s390x
texlive-tex-svn40793-33.fc25.1.noarch
texlive-fancyref-svn15878.0.9c-33.fc25.1.noarch
texlive-chngcntr-svn17157.1.0a-33.fc25.1.noarch
texlive-fix2col-svn38770-33.fc25.1.noarch
texlive-marginnote-svn41382-33.fc25.1.noarch
texlive-pxfonts-svn15878.0-33.fc25.1.noarch
texlive-txfonts-svn15878.0-33.fc25.1.noarch
texlive-l3packages-svn41246-33.fc25.1.noarch
texlive-oberdiek-svn41346-33.fc25.1.noarch
texlive-pst-tools-svn34067.0.05-33.fc25.1.noarch
texlive-tex-gyre-svn18651.2.004-33.fc25.1.noarch
texlive-dvipdfmx-svn41149-33.fc25.1.noarch
texlive-collection-fontsrecommended-svn35830.0-33.20160520.fc25.1.noarch
libcacard-devel-2.5.3-1.fc25.s390x
ykpers-1.18.0-2.fc25.s390x
python2-idna-2.5-1.fc25.noarch
file-libs-5.29-4.fc25.s390x
policycoreutils-2.5-20.fc25.s390x
libgcrypt-1.7.8-1.fc25.s390x
pcre-8.41-1.fc25.s390x
GeoIP-1.6.11-1.fc25.s390x
ghostscript-core-9.20-9.fc25.s390x
python3-cffi-1.7.0-2.fc25.s390x
nss-softokn-freebl-devel-3.30.2-1.0.fc25.s390x
json-c-0.12.1-2.fc25.s390x
vim-common-8.0.705-1.fc25.s390x
vte291-0.46.2-1.fc25.s390x
libdrm-devel-2.4.81-1.fc25.s390x
gssproxy-0.7.0-9.fc25.s390x
git-core-doc-2.9.4-1.fc25.s390x
systemtap-3.1-5.fc25.s390x
mesa-libgbm-devel-17.0.5-3.fc25.s390x
vim-enhanced-8.0.705-1.fc25.s390x
glibc-static-2.24-9.fc25.s390x
libgusb-0.2.10-1.fc25.s390x
python-async-0.6.1-9.fc22.s390x
dejavu-sans-mono-fonts-2.35-3.fc24.noarch
popt-1.16-7.fc24.s390x
cyrus-sasl-lib-2.1.26-26.2.fc24.s390x
xz-5.2.2-2.fc24.s390x
libpipeline-1.4.1-2.fc24.s390x
pinentry-0.9.7-2.fc24.s390x
pth-2.0.7-27.fc24.s390x
libsepol-2.5-10.fc25.s390x
libxcb-1.12-1.fc25.s390x
perl-Getopt-Long-2.49.1-1.fc25.noarch
avahi-glib-0.6.32-4.fc25.s390x
python3-pip-8.1.2-2.fc25.noarch
python3-libcomps-0.1.7-5.fc25.s390x
python-slip-0.6.4-4.fc25.noarch
python2-libcomps-0.1.7-5.fc25.s390x
gc-7.4.4-1.fc25.s390x
s390utils-cmsfs-1.36.0-1.fc25.s390x
newt-python-0.52.19-2.fc25.s390x
qt5-srpm-macros-5.7.1-1.fc25.noarch
device-mapper-event-1.02.136-3.fc25.s390x
perl-Class-Inspector-1.31-2.fc25.noarch
libbasicobjects-0.1.1-29.fc25.s390x
libradosstriper1-10.2.4-2.fc25.s390x
libXxf86vm-devel-1.1.4-3.fc24.s390x
zziplib-0.13.62-7.fc24.s390x
libpaper-1.1.24-12.fc24.s390x
libini_config-1.3.0-29.fc25.s390x
snappy-devel-1.1.3-2.fc24.s390x
libcap-ng-devel-0.7.8-1.fc25.s390x
libxkbcommon-devel-0.7.1-1.fc25.s390x
openssl-libs-1.0.2k-1.fc25.s390x
libkadm5-1.14.4-7.fc25.s390x
rpm-libs-4.13.0.1-1.fc25.s390x
util-linux-2.28.2-2.fc25.s390x
texlive-etoolbox-svn38031.2.2a-33.fc25.1.noarch
texlive-dvips-svn41149-33.fc25.1.noarch
texlive-latexconfig-svn40274-33.fc25.1.noarch
texlive-tex-ini-files-svn40533-33.fc25.1.noarch
texlive-qstest-svn15878.0-33.fc25.1.noarch
texlive-cmap-svn41168-33.fc25.1.noarch
texlive-luatex-bin-svn41091-33.20160520.fc25.1.s390x
texlive-mflogo-svn38628-33.fc25.1.noarch
texlive-sansmath-svn17997.1.1-33.fc25.1.noarch
texlive-unicode-data-svn39808-33.fc25.1.noarch
texlive-luaotfload-bin-svn34647.0-33.20160520.fc25.1.noarch
texlive-listings-svn37534.1.6-33.fc25.1.noarch
texlive-pstricks-svn41321-33.fc25.1.noarch
texlive-metalogo-svn18611.0.12-33.fc25.1.noarch
texlive-collection-latex-svn41011-33.20160520.fc25.1.noarch
kernel-4.10.5-200.fc25.s390x
python2-dnf-plugins-core-0.1.21-5.fc25.noarch
xkeyboard-config-2.20-2.fc25.noarch
file-5.29-4.fc25.s390x
perl-Test-Harness-3.39-1.fc25.noarch
systemd-libs-231-17.fc25.s390x
webkitgtk4-jsc-2.16.5-1.fc25.s390x
gtk-update-icon-cache-3.22.16-1.fc25.s390x
glibc-devel-2.24-9.fc25.s390x
python3-pycparser-2.14-7.fc25.noarch
kernel-devel-4.11.10-200.fc25.s390x
gsm-1.0.17-1.fc25.s390x
vim-filesystem-8.0.705-1.fc25.s390x
webkitgtk4-2.16.5-1.fc25.s390x
python-2.7.13-2.fc25.s390x
glusterfs-fuse-3.10.4-1.fc25.s390x
git-core-2.9.4-1.fc25.s390x
selinux-policy-targeted-3.13.1-225.18.fc25.noarch
kernel-4.11.10-200.fc25.s390x
rpmconf-1.0.19-1.fc25.noarch
teamd-1.27-1.fc25.s390x
jasper-libs-1.900.13-4.fc25.s390x
libattr-2.4.47-16.fc24.s390x
libvisual-0.4.0-20.fc24.s390x
libpcap-1.7.4-2.fc24.s390x
libutempter-1.1.6-8.fc24.s390x
libgudev-230-3.fc24.s390x
popt-devel-1.16-7.fc24.s390x
hicolor-icon-theme-0.15-3.fc24.noarch
setup-2.10.4-1.fc25.noarch
bash-4.3.43-4.fc25.s390x
libjpeg-turbo-1.5.1-0.fc25.s390x
perl-Socket-2.024-1.fc25.s390x
perl-HTTP-Tiny-0.070-1.fc25.noarch
ipset-6.29-1.fc25.s390x
python2-setuptools-25.1.1-1.fc25.noarch
gsettings-desktop-schemas-3.22.0-1.fc25.s390x
python3-setuptools-25.1.1-1.fc25.noarch
python-slip-dbus-0.6.4-4.fc25.noarch
python2-ply-3.8-2.fc25.noarch
dtc-1.4.2-1.fc25.s390x
guile-2.0.13-1.fc25.s390x
cronie-anacron-1.5.1-2.fc25.s390x
libXtst-1.2.3-1.fc25.s390x
iso-codes-3.70-1.fc25.noarch
s390utils-1.36.0-1.fc25.s390x
python-backports-ssl_match_hostname-3.5.0.1-3.fc25.noarch
fedora-cert-0.6.0.1-1.fc25.noarch
libstdc++-6.3.1-1.fc25.s390x
subversion-libs-1.9.5-1.fc25.s390x
libgfortran-6.3.1-1.fc25.s390x
dnf-plugin-system-upgrade-0.7.1-4.fc25.noarch
lvm2-2.02.167-3.fc25.s390x
libselinux-devel-2.5-13.fc25.s390x
perl-Time-Local-1.250-1.fc25.noarch
libradosstriper-devel-10.2.4-2.fc25.s390x
flac-libs-1.3.2-1.fc25.s390x
perl-Digest-1.17-366.fc25.noarch
teckit-2.5.1-15.fc24.s390x
libpath_utils-0.2.1-29.fc25.s390x
attr-2.4.47-16.fc24.s390x
usbredir-0.7.1-2.fc24.s390x
cairo-devel-1.14.8-1.fc25.s390x
lzo-devel-2.08-8.fc24.s390x
libcap-devel-2.25-2.fc25.s390x
krb5-devel-1.14.4-7.fc25.s390x
rpm-4.13.0.1-1.fc25.s390x
kernel-devel-4.10.5-200.fc25.s390x
libbsd-0.8.3-1.fc25.s390x
texlive-url-svn32528.3.4-33.fc25.1.noarch
texlive-dvips-bin-svn40987-33.20160520.fc25.1.s390x
texlive-index-svn24099.4.1beta-33.fc25.1.noarch
texlive-setspace-svn24881.6.7a-33.fc25.1.noarch
texlive-mathtools-svn38833-33.fc25.1.noarch
texlive-cm-svn32865.0-33.fc25.1.noarch
texlive-graphics-def-svn41879-33.fc25.1.noarch
texlive-mdwtools-svn15878.1.05.4-33.fc25.1.noarch
texlive-rsfs-svn15878.0-33.fc25.1.noarch
texlive-ucharcat-svn38907-33.fc25.1.noarch
texlive-fontspec-svn41262-33.fc25.1.noarch
texlive-showexpl-svn32737.v0.3l-33.fc25.1.noarch
texlive-pstricks-add-svn40744-33.fc25.1.noarch
texlive-beamer-svn36461.3.36-33.fc25.1.noarch
texlive-collection-basic-svn41149-33.20160520.fc25.1.noarch
rpm-build-4.13.0.1-1.fc25.s390x
xemacs-filesystem-21.5.34-20.20170124hgf412e9f093d4.fc25.noarch
hawkey-0.6.4-3.fc25.s390x
gdk-pixbuf2-modules-2.36.6-1.fc25.s390x
bluez-libs-5.44-1.fc25.s390x
audit-libs-2.7.7-1.fc25.s390x
iproute-4.11.0-1.fc25.s390x
libICE-1.0.9-9.fc25.s390x
glibc-headers-2.24-9.fc25.s390x
python3-ply-3.8-2.fc25.noarch
perl-5.24.2-387.fc25.s390x
graphite2-1.3.10-1.fc25.s390x
vte-profile-0.46.2-1.fc25.s390x
gtk3-devel-3.22.16-1.fc25.s390x
python-libs-2.7.13-2.fc25.s390x
mesa-libGL-17.0.5-3.fc25.s390x
python2-pycurl-7.43.0-6.fc25.s390x
NetworkManager-1.4.4-5.fc25.s390x
mesa-libEGL-devel-17.0.5-3.fc25.s390x
mariadb-libs-10.1.24-3.fc25.s390x
mesa-libGLES-devel-17.0.5-3.fc25.s390x
hostname-3.15-8.fc25.s390x
gpg-pubkey-a0a7badb-52844296
readline-6.3-8.fc24.s390x
cpio-2.12-3.fc24.s390x
libXcomposite-0.4.4-8.fc24.s390x
procps-ng-3.3.10-11.fc24.s390x
GConf2-3.2.6-16.fc24.s390x
xz-devel-5.2.2-2.fc24.s390x
fedora-logos-22.0.0-3.fc24.s390x
gpg-pubkey-e372e838-56fd7943
kmod-libs-23-1.fc25.s390x
perl-parent-0.236-1.fc25.noarch
perl-TermReadKey-2.37-1.fc25.s390x
ncurses-c++-libs-6.0-6.20160709.fc25.s390x
gzip-1.8-1.fc25.s390x
python3-gobject-base-3.22.0-1.fc25.s390x
python2-yubico-1.3.2-3.fc25.noarch
s390utils-ziomon-1.36.0-1.fc25.s390x
librepo-1.7.18-3.fc25.s390x
librsvg2-2.40.16-2.fc25.s390x
gnat-srpm-macros-4-1.fc25.noarch
python-decoratortools-1.8-12.fc25.noarch
m17n-db-1.7.0-7.fc25.noarch
e2fsprogs-libs-1.43.3-1.fc25.s390x
libvorbis-1.3.5-1.fc25.s390x
gcc-gdb-plugin-6.3.1-1.fc25.s390x
npth-1.3-1.fc25.s390x
libcephfs1-10.2.4-2.fc25.s390x
wayland-devel-1.12.0-1.fc25.s390x
libxcb-devel-1.12-1.fc25.s390x
perl-encoding-2.19-5.fc25.s390x
python3-cssselect-0.9.2-1.fc25.noarch
gettext-libs-0.19.8.1-3.fc25.s390x
at-spi2-atk-devel-2.22.0-1.fc25.s390x
virglrenderer-0.5.0-1.20160411git61846f92f.fc25.s390x
pixman-devel-0.34.0-2.fc24.s390x
libnfs-devel-1.9.8-2.fc24.s390x
libblkid-2.28.2-2.fc25.s390x
glib2-devel-2.50.3-1.fc25.s390x
texlive-ifxetex-svn19685.0.5-33.fc25.1.noarch
texlive-caption-svn41409-33.fc25.1.noarch
texlive-float-svn15878.1.3d-33.fc25.1.noarch
texlive-pdftex-def-svn22653.0.06d-33.fc25.1.noarch
texlive-xdvi-bin-svn40750-33.20160520.fc25.1.s390x
texlive-beton-svn15878.0-33.fc25.1.noarch
texlive-filecontents-svn24250.1.3-33.fc25.1.noarch
texlive-lm-math-svn36915.1.959-33.fc25.1.noarch
texlive-pslatex-svn16416.0-33.fc25.1.noarch
texlive-times-svn35058.0-33.fc25.1.noarch
texlive-breakurl-svn29901.1.40-33.fc25.1.noarch
texlive-filehook-svn24280.0.5d-33.fc25.1.noarch
texlive-pst-pdf-svn31660.1.1v-33.fc25.1.noarch
texlive-seminar-svn34011.1.62-33.fc25.1.noarch
texlive-xetexconfig-svn41133-33.fc25.1.noarch
python-rpm-macros-3-12.fc25.noarch
rpm-devel-4.13.0.1-1.fc25.s390x
nss-pem-1.0.3-3.fc25.s390x
at-spi2-core-2.22.1-1.fc25.s390x
GeoIP-GeoLite-data-2017.04-1.fc25.noarch
kernel-devel-4.10.8-200.fc25.s390x
dbus-libs-1.11.14-1.fc25.s390x
perl-Scalar-List-Utils-1.48-1.fc25.s390x
libidn2-2.0.2-1.fc25.s390x
libtasn1-devel-4.12-1.fc25.s390x
python3-koji-1.13.0-2.fc25.noarch
glusterfs-cli-3.10.4-1.fc25.s390x
opus-1.1.5-1.fc25.s390x
mariadb-common-10.1.24-3.fc25.s390x
elfutils-libs-0.169-1.fc25.s390x
kernel-core-4.11.10-200.fc25.s390x
gnutls-dane-3.5.14-1.fc25.s390x
systemd-container-231-17.fc25.s390x
sudo-1.8.20p2-1.fc25.s390x
dbus-devel-1.11.14-1.fc25.s390x
perl-Module-CoreList-5.20170621-1.fc25.noarch
libicu-devel-57.1-5.fc25.s390x
js-jquery-2.2.4-3.fc25.noarch
=== TEST BEGIN ===
Using CC: /home/fam/bin/cc
Install prefix    /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install
BIOS directory    /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/share/qemu
binary directory  /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/bin
library directory /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/lib
module directory  /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/lib/qemu
libexec directory /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/libexec
include directory /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/include
config directory  /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/etc
local state directory   /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/var
Manual directory  /var/tmp/patchew-tester-tmp-hm2hdbnw/src/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path       /var/tmp/patchew-tester-tmp-hm2hdbnw/src
C compiler        /home/fam/bin/cc
Host C compiler   cc
C++ compiler      c++
Objective-C compiler /home/fam/bin/cc
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/include/pixman-1  -Werror -DHAS_LIBSSH2_SFTP_FSYNC -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DNCURSES_WIDECHAR -D_GNU_SOURCE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/include/p11-kit-1    -I/usr/include/libpng16 -I/usr/include/libdrm  -I/usr/include/cacard -I/usr/include/nss3 -I/usr/include/nspr4 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/libusb-1.0 
LDFLAGS           -Wl,--warn-common -m64 -g 
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          s390x
host big endian   yes
target list       aarch64-softmmu alpha-softmmu arm-softmmu cris-softmmu i386-softmmu lm32-softmmu m68k-softmmu microblazeel-softmmu microblaze-softmmu mips64el-softmmu mips64-softmmu mipsel-softmmu mips-softmmu moxie-softmmu nios2-softmmu or1k-softmmu ppc64-softmmu ppcemb-softmmu ppc-softmmu s390x-softmmu sh4eb-softmmu sh4-softmmu sparc64-softmmu sparc-softmmu tricore-softmmu unicore32-softmmu x86_64-softmmu xtensaeb-softmmu xtensa-softmmu aarch64-linux-user alpha-linux-user armeb-linux-user arm-linux-user cris-linux-user hppa-linux-user i386-linux-user m68k-linux-user microblazeel-linux-user microblaze-linux-user mips64el-linux-user mips64-linux-user mipsel-linux-user mips-linux-user mipsn32el-linux-user mipsn32-linux-user nios2-linux-user or1k-linux-user ppc64abi32-linux-user ppc64le-linux-user ppc64-linux-user ppc-linux-user s390x-linux-user sh4eb-linux-user sh4-linux-user sparc32plus-linux-user sparc64-linux-user sparc-linux-user tilegx-linux-user x86_64-linux-user
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
pixman            system
SDL support       yes (2.0.5)
GTK support       yes (3.22.16)
GTK GL support    yes
VTE support       yes (0.46.2)
TLS priority      NORMAL
GNUTLS support    yes
GNUTLS rnd        yes
libgcrypt         no
libgcrypt kdf     no
nettle            yes (3.3)
nettle kdf        yes
libtasn1          yes
curses support    yes
virgl support     yes
curl support      yes
mingw32 support   no
Audio drivers     oss
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    yes
VNC support       yes
VNC SASL support  yes
VNC JPEG support  yes
VNC PNG support   yes
xen support       no
brlapi support    yes
bluez  support    yes
Documentation     yes
PIE               no
vde support       no
netmap support    no
Linux AIO support yes
ATTR/XATTR support yes
Install blobs     yes
KVM support       yes
HAX support       no
TCG support       yes
TCG debug enabled no
TCG interpreter   no
RDMA support      no
fdt support       yes
preadv support    yes
fdatasync         yes
madvise           yes
posix_madvise     yes
libcap-ng support yes
vhost-net support yes
vhost-scsi support yes
vhost-vsock support yes
vhost-user support yes
Trace backends    log
spice support     no 
rbd support       yes
xfsctl support    no
smartcard support yes
libusb            yes
usb net redir     yes
OpenGL support    yes
OpenGL dmabufs    yes
libiscsi support  yes
libnfs support    yes
build guest agent yes
QGA VSS support   no
QGA w32 disk info no
QGA MSI support   no
seccomp support   no
coroutine backend ucontext
coroutine pool    yes
debug stack usage no
crypto afalg      no
GlusterFS support yes
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   yes
TPM passthrough   no
QOM debugging     yes
Live block migration yes
lzo support       yes
snappy support    yes
bzip2 support     yes
NUMA host support no
tcmalloc support  no
jemalloc support  no
avx2 optimization no
replication support yes
VxHS block device no
  GEN     aarch64-softmmu/config-devices.mak.tmp
  GEN     cris-softmmu/config-devices.mak.tmp
  GEN     alpha-softmmu/config-devices.mak.tmp
  GEN     arm-softmmu/config-devices.mak.tmp
  GEN     cris-softmmu/config-devices.mak
  GEN     i386-softmmu/config-devices.mak.tmp
  GEN     alpha-softmmu/config-devices.mak
  GEN     arm-softmmu/config-devices.mak
  GEN     lm32-softmmu/config-devices.mak.tmp
  GEN     m68k-softmmu/config-devices.mak.tmp
  GEN     aarch64-softmmu/config-devices.mak
  GEN     microblazeel-softmmu/config-devices.mak.tmp
  GEN     lm32-softmmu/config-devices.mak
  GEN     i386-softmmu/config-devices.mak
  GEN     m68k-softmmu/config-devices.mak
  GEN     microblaze-softmmu/config-devices.mak.tmp
  GEN     mips64el-softmmu/config-devices.mak.tmp
  GEN     mips64-softmmu/config-devices.mak.tmp
  GEN     microblazeel-softmmu/config-devices.mak
  GEN     microblaze-softmmu/config-devices.mak
  GEN     mipsel-softmmu/config-devices.mak.tmp
  GEN     mips-softmmu/config-devices.mak.tmp
  GEN     mips64-softmmu/config-devices.mak
  GEN     mips64el-softmmu/config-devices.mak
  GEN     nios2-softmmu/config-devices.mak.tmp
  GEN     moxie-softmmu/config-devices.mak.tmp
  GEN     mipsel-softmmu/config-devices.mak
  GEN     moxie-softmmu/config-devices.mak
  GEN     nios2-softmmu/config-devices.mak
  GEN     mips-softmmu/config-devices.mak
  GEN     ppc64-softmmu/config-devices.mak.tmp
  GEN     ppcemb-softmmu/config-devices.mak.tmp
  GEN     or1k-softmmu/config-devices.mak.tmp
  GEN     ppc-softmmu/config-devices.mak.tmp
  GEN     or1k-softmmu/config-devices.mak
  GEN     s390x-softmmu/config-devices.mak.tmp
  GEN     ppc64-softmmu/config-devices.mak
  GEN     ppcemb-softmmu/config-devices.mak
  GEN     sh4eb-softmmu/config-devices.mak.tmp
  GEN     ppc-softmmu/config-devices.mak
  GEN     sh4-softmmu/config-devices.mak.tmp
  GEN     sparc64-softmmu/config-devices.mak.tmp
  GEN     s390x-softmmu/config-devices.mak
  GEN     sparc-softmmu/config-devices.mak.tmp
  GEN     sh4eb-softmmu/config-devices.mak
  GEN     sh4-softmmu/config-devices.mak
  GEN     tricore-softmmu/config-devices.mak.tmp
  GEN     sparc64-softmmu/config-devices.mak
  GEN     unicore32-softmmu/config-devices.mak.tmp
  GEN     x86_64-softmmu/config-devices.mak.tmp
  GEN     sparc-softmmu/config-devices.mak
  GEN     xtensaeb-softmmu/config-devices.mak.tmp
  GEN     unicore32-softmmu/config-devices.mak
  GEN     tricore-softmmu/config-devices.mak
  GEN     xtensa-softmmu/config-devices.mak.tmp
  GEN     aarch64-linux-user/config-devices.mak.tmp
  GEN     xtensaeb-softmmu/config-devices.mak
  GEN     alpha-linux-user/config-devices.mak.tmp
  GEN     xtensa-softmmu/config-devices.mak
  GEN     x86_64-softmmu/config-devices.mak
  GEN     aarch64-linux-user/config-devices.mak
  GEN     armeb-linux-user/config-devices.mak.tmp
  GEN     alpha-linux-user/config-devices.mak
  GEN     arm-linux-user/config-devices.mak.tmp
  GEN     cris-linux-user/config-devices.mak.tmp
  GEN     hppa-linux-user/config-devices.mak.tmp
  GEN     armeb-linux-user/config-devices.mak
  GEN     arm-linux-user/config-devices.mak
  GEN     cris-linux-user/config-devices.mak
  GEN     i386-linux-user/config-devices.mak.tmp
  GEN     m68k-linux-user/config-devices.mak.tmp
  GEN     microblazeel-linux-user/config-devices.mak.tmp
  GEN     hppa-linux-user/config-devices.mak
  GEN     microblaze-linux-user/config-devices.mak.tmp
  GEN     i386-linux-user/config-devices.mak
  GEN     m68k-linux-user/config-devices.mak
  GEN     microblazeel-linux-user/config-devices.mak
  GEN     mips64el-linux-user/config-devices.mak.tmp
  GEN     mips64-linux-user/config-devices.mak.tmp
  GEN     mipsel-linux-user/config-devices.mak.tmp
  GEN     microblaze-linux-user/config-devices.mak
  GEN     mips64el-linux-user/config-devices.mak
  GEN     mips-linux-user/config-devices.mak.tmp
  GEN     mipsel-linux-user/config-devices.mak
  GEN     mipsn32el-linux-user/config-devices.mak.tmp
  GEN     mips64-linux-user/config-devices.mak
  GEN     mipsn32-linux-user/config-devices.mak.tmp
  GEN     mips-linux-user/config-devices.mak
  GEN     or1k-linux-user/config-devices.mak.tmp
  GEN     nios2-linux-user/config-devices.mak.tmp
  GEN     mipsn32el-linux-user/config-devices.mak
  GEN     mipsn32-linux-user/config-devices.mak
  GEN     ppc64le-linux-user/config-devices.mak.tmp
  GEN     ppc64abi32-linux-user/config-devices.mak.tmp
  GEN     nios2-linux-user/config-devices.mak
  GEN     or1k-linux-user/config-devices.mak
  GEN     ppc64-linux-user/config-devices.mak.tmp
  GEN     ppc-linux-user/config-devices.mak.tmp
  GEN     ppc64abi32-linux-user/config-devices.mak
  GEN     ppc64le-linux-user/config-devices.mak
  GEN     sh4eb-linux-user/config-devices.mak.tmp
  GEN     s390x-linux-user/config-devices.mak.tmp
  GEN     ppc64-linux-user/config-devices.mak
  GEN     ppc-linux-user/config-devices.mak
  GEN     sh4-linux-user/config-devices.mak.tmp
  GEN     sh4eb-linux-user/config-devices.mak
  GEN     sparc32plus-linux-user/config-devices.mak.tmp
  GEN     sparc64-linux-user/config-devices.mak.tmp
  GEN     s390x-linux-user/config-devices.mak
  GEN     sh4-linux-user/config-devices.mak
  GEN     sparc32plus-linux-user/config-devices.mak
  GEN     sparc-linux-user/config-devices.mak.tmp
  GEN     tilegx-linux-user/config-devices.mak.tmp
  GEN     x86_64-linux-user/config-devices.mak.tmp
  GEN     sparc64-linux-user/config-devices.mak
  GEN     sparc-linux-user/config-devices.mak
  GEN     config-host.h
  GEN     tilegx-linux-user/config-devices.mak
  GEN     qemu-options.def
  GEN     x86_64-linux-user/config-devices.mak
  GEN     qmp-commands.h
  GEN     qapi-types.h
  GEN     qapi-visit.h
  GEN     qapi-event.h
  GEN     qmp-marshal.c
  GEN     qapi-types.c
  GEN     qapi-visit.c
  GEN     qapi-event.c
  GEN     qmp-introspect.h
  GEN     qmp-introspect.c
  GEN     trace/generated-tcg-tracers.h
  GEN     trace/generated-helpers-wrappers.h
  GEN     trace/generated-helpers.c
  GEN     trace/generated-helpers.h
  GEN     module_block.h
  GEN     tests/test-qapi-types.h
  GEN     tests/test-qapi-visit.h
  GEN     tests/test-qmp-commands.h
  GEN     tests/test-qmp-introspect.h
  GEN     tests/test-qapi-event.h
  GEN     trace-root.h
  GEN     crypto/trace.h
  GEN     io/trace.h
  GEN     util/trace.h
  GEN     migration/trace.h
  GEN     block/trace.h
  GEN     chardev/trace.h
  GEN     hw/block/trace.h
  GEN     hw/block/dataplane/trace.h
  GEN     hw/char/trace.h
  GEN     hw/intc/trace.h
  GEN     hw/net/trace.h
  GEN     hw/virtio/trace.h
  GEN     hw/audio/trace.h
  GEN     hw/misc/trace.h
  GEN     hw/usb/trace.h
  GEN     hw/nvram/trace.h
  GEN     hw/scsi/trace.h
  GEN     hw/display/trace.h
  GEN     hw/input/trace.h
  GEN     hw/timer/trace.h
  GEN     hw/dma/trace.h
  GEN     hw/sparc/trace.h
  GEN     hw/isa/trace.h
  GEN     hw/sd/trace.h
  GEN     hw/mem/trace.h
  GEN     hw/i386/trace.h
  GEN     hw/i386/xen/trace.h
  GEN     hw/ppc/trace.h
  GEN     hw/pci/trace.h
  GEN     hw/9pfs/trace.h
  GEN     hw/s390x/trace.h
  GEN     hw/vfio/trace.h
  GEN     hw/acpi/trace.h
  GEN     hw/arm/trace.h
  GEN     hw/alpha/trace.h
  GEN     hw/xen/trace.h
  GEN     ui/trace.h
  GEN     audio/trace.h
  GEN     net/trace.h
  GEN     target/arm/trace.h
  GEN     target/i386/trace.h
  GEN     target/mips/trace.h
  GEN     target/s390x/trace.h
  GEN     target/ppc/trace.h
  GEN     target/sparc/trace.h
  GEN     qom/trace.h
  GEN     linux-user/trace.h
  GEN     qapi/trace.h
  GEN     accel/tcg/trace.h
  GEN     accel/kvm/trace.h
  GEN     trace-root.c
  GEN     nbd/trace.h
  GEN     util/trace.c
  GEN     crypto/trace.c
  GEN     io/trace.c
  GEN     migration/trace.c
  GEN     chardev/trace.c
  GEN     block/trace.c
  GEN     hw/block/trace.c
  GEN     hw/block/dataplane/trace.c
  GEN     hw/char/trace.c
  GEN     hw/intc/trace.c
  GEN     hw/net/trace.c
  GEN     hw/virtio/trace.c
  GEN     hw/audio/trace.c
  GEN     hw/misc/trace.c
  GEN     hw/usb/trace.c
  GEN     hw/scsi/trace.c
  GEN     hw/nvram/trace.c
  GEN     hw/display/trace.c
  GEN     hw/input/trace.c
  GEN     hw/dma/trace.c
  GEN     hw/timer/trace.c
  GEN     hw/sparc/trace.c
  GEN     hw/isa/trace.c
  GEN     hw/sd/trace.c
  GEN     hw/mem/trace.c
  GEN     hw/i386/trace.c
  GEN     hw/i386/xen/trace.c
  GEN     hw/9pfs/trace.c
  GEN     hw/ppc/trace.c
  GEN     hw/pci/trace.c
  GEN     hw/s390x/trace.c
  GEN     hw/vfio/trace.c
  GEN     hw/acpi/trace.c
  GEN     hw/arm/trace.c
  GEN     hw/alpha/trace.c
  GEN     ui/trace.c
  GEN     hw/xen/trace.c
  GEN     audio/trace.c
  GEN     net/trace.c
  GEN     target/arm/trace.c
  GEN     target/i386/trace.c
  GEN     target/mips/trace.c
  GEN     target/sparc/trace.c
  GEN     target/s390x/trace.c
  GEN     target/ppc/trace.c
  GEN     qom/trace.c
  GEN     linux-user/trace.c
  GEN     qapi/trace.c
  GEN     accel/tcg/trace.c
  GEN     accel/kvm/trace.c
  GEN     config-all-devices.mak
  GEN     nbd/trace.c
  GEN     docs/version.texi
  GEN     qemu-monitor.texi
  GEN     qemu-options.texi
  CC      tests/qemu-iotests/socket_scm_helper.o
  GEN     qemu-img-cmds.texi
  GEN     qemu-monitor-info.texi
  GEN     qemu-img.1
  GEN     qemu-nbd.8
  GEN     qemu-ga.8
  GEN     docs/interop/qemu-qmp-qapi.texi
  GEN     docs/interop/qemu-ga-qapi.texi
  GEN     fsdev/virtfs-proxy-helper.1
  GEN     qga/qapi-generated/qga-qapi-types.h
  GEN     qga/qapi-generated/qga-qapi-visit.h
  GEN     qga/qapi-generated/qga-qapi-types.c
  GEN     qga/qapi-generated/qga-qapi-visit.c
  GEN     qga/qapi-generated/qga-qmp-commands.h
  CC      qmp-introspect.o
  GEN     qga/qapi-generated/qga-qmp-marshal.c
  CC      qapi-types.o
  CC      qapi-event.o
  CC      qapi-visit.o
  CC      qapi/qapi-visit-core.o
  CC      qapi/qapi-dealloc-visitor.o
  CC      qapi/qobject-input-visitor.o
  CC      qapi/qobject-output-visitor.o
  CC      qapi/qmp-registry.o
  CC      qapi/qmp-dispatch.o
  CC      qapi/string-input-visitor.o
  CC      qapi/string-output-visitor.o
  CC      qapi/opts-visitor.o
  CC      qapi/qapi-clone-visitor.o
  CC      qapi/qmp-event.o
  CC      qapi/qapi-util.o
  CC      qobject/qnull.o
  CC      qobject/qnum.o
  CC      qobject/qstring.o
  CC      qobject/qdict.o
  CC      qobject/qlist.o
  CC      qobject/qbool.o
  CC      qobject/qjson.o
  CC      qobject/qobject.o
  CC      qobject/json-lexer.o
  CC      qobject/json-streamer.o
  CC      qobject/json-parser.o
  CC      trace/control.o
  CC      trace/qmp.o
  CC      util/osdep.o
  CC      util/cutils.o
  CC      util/unicode.o
  CC      util/qemu-timer-common.o
  CC      util/bufferiszero.o
  CC      util/lockcnt.o
  CC      util/aiocb.o
  CC      util/thread-pool.o
  CC      util/async.o
  CC      util/qemu-timer.o
  CC      util/iohandler.o
  CC      util/main-loop.o
  CC      util/aio-posix.o
  CC      util/compatfd.o
  CC      util/event_notifier-posix.o
  CC      util/mmap-alloc.o
  CC      util/oslib-posix.o
  CC      util/qemu-openpty.o
  CC      util/qemu-thread-posix.o
  CC      util/memfd.o
  CC      util/envlist.o
  CC      util/path.o
  CC      util/module.o
  CC      util/host-utils.o
  CC      util/bitops.o
  CC      util/bitmap.o
  CC      util/hbitmap.o
  CC      util/fifo8.o
  CC      util/acl.o
  CC      util/cacheinfo.o
  CC      util/error.o
  CC      util/qemu-error.o
  CC      util/id.o
  CC      util/iov.o
  CC      util/qemu-config.o
  CC      util/qemu-sockets.o
  CC      util/notify.o
  CC      util/uri.o
  CC      util/qemu-option.o
  CC      util/qemu-progress.o
  CC      util/keyval.o
  CC      util/hexdump.o
  CC      util/crc32c.o
  CC      util/uuid.o
  CC      util/throttle.o
  CC      util/getauxval.o
  CC      util/readline.o
  CC      util/rcu.o
  CC      util/qemu-coroutine.o
  CC      util/qemu-coroutine-lock.o
  CC      util/qemu-coroutine-io.o
  CC      util/coroutine-ucontext.o
  CC      util/qemu-coroutine-sleep.o
  CC      util/buffer.o
  CC      util/timed-average.o
  CC      util/base64.o
  CC      util/qdist.o
  CC      util/log.o
  CC      util/qht.o
  CC      util/range.o
  CC      util/stats64.o
  CC      util/systemd.o
  CC      trace-root.o
  CC      util/trace.o
  CC      crypto/trace.o
  CC      io/trace.o
  CC      migration/trace.o
  CC      block/trace.o
  CC      chardev/trace.o
  CC      hw/block/trace.o
  CC      hw/block/dataplane/trace.o
  CC      hw/char/trace.o
  CC      hw/intc/trace.o
  CC      hw/net/trace.o
  CC      hw/virtio/trace.o
  CC      hw/audio/trace.o
  CC      hw/misc/trace.o
  CC      hw/usb/trace.o
  CC      hw/scsi/trace.o
  CC      hw/nvram/trace.o
  CC      hw/display/trace.o
  CC      hw/input/trace.o
  CC      hw/timer/trace.o
  CC      hw/dma/trace.o
  CC      hw/sparc/trace.o
  CC      hw/sd/trace.o
  CC      hw/isa/trace.o
  CC      hw/mem/trace.o
  CC      hw/i386/trace.o
  CC      hw/i386/xen/trace.o
  CC      hw/9pfs/trace.o
  CC      hw/ppc/trace.o
  CC      hw/pci/trace.o
  CC      hw/s390x/trace.o
  CC      hw/vfio/trace.o
  CC      hw/acpi/trace.o
  CC      hw/arm/trace.o
  CC      hw/alpha/trace.o
  CC      hw/xen/trace.o
  CC      ui/trace.o
  CC      audio/trace.o
  CC      net/trace.o
  CC      target/i386/trace.o
  CC      target/arm/trace.o
  CC      target/mips/trace.o
  CC      target/s390x/trace.o
  CC      target/sparc/trace.o
  CC      target/ppc/trace.o
  CC      qom/trace.o
  CC      qapi/trace.o
  CC      linux-user/trace.o
  CC      accel/tcg/trace.o
  CC      accel/kvm/trace.o
  CC      nbd/trace.o
  CC      crypto/pbkdf-stub.o
  CC      stubs/arch-query-cpu-def.o
  CC      stubs/arch-query-cpu-model-expansion.o
  CC      stubs/arch-query-cpu-model-comparison.o
  CC      stubs/arch-query-cpu-model-baseline.o
  CC      stubs/bdrv-next-monitor-owned.o
  CC      stubs/blk-commit-all.o
  CC      stubs/clock-warp.o
  CC      stubs/blockdev-close-all-bdrv-states.o
  CC      stubs/cpu-get-clock.o
  CC      stubs/cpu-get-icount.o
  CC      stubs/dump.o
  CC      stubs/error-printf.o
  CC      stubs/fdset.o
  CC      stubs/gdbstub.o
  CC      stubs/iothread.o
  CC      stubs/get-vm-name.o
  CC      stubs/iothread-lock.o
  CC      stubs/is-daemonized.o
  CC      stubs/linux-aio.o
  CC      stubs/machine-init-done.o
  CC      stubs/migr-blocker.o
  CC      stubs/change-state-handler.o
  CC      stubs/monitor.o
  CC      stubs/notify-event.o
  CC      stubs/qtest.o
  CC      stubs/runstate-check.o
  CC      stubs/replay.o
  CC      stubs/set-fd-handler.o
  CC      stubs/slirp.o
  CC      stubs/sysbus.o
  CC      stubs/trace-control.o
  CC      stubs/uuid.o
  CC      stubs/vm-stop.o
  CC      stubs/vmstate.o
  CC      stubs/qmp_pc_dimm_device_list.o
  CC      stubs/target-monitor-defs.o
  CC      stubs/target-get-monitor-def.o
  CC      stubs/pc_madt_cpu_entry.o
  CC      stubs/vmgenid.o
  CC      stubs/xen-common.o
  CC      stubs/xen-hvm.o
  CC      contrib/ivshmem-client/ivshmem-client.o
  CC      contrib/ivshmem-client/main.o
  CC      contrib/ivshmem-server/ivshmem-server.o
  CC      contrib/ivshmem-server/main.o
  CC      qemu-nbd.o
  CC      block.o
  CC      blockjob.o
  CC      qemu-io-cmds.o
  CC      replication.o
  CC      block/raw-format.o
  CC      block/qcow.o
  CC      block/vdi.o
  CC      block/vmdk.o
  CC      block/cloop.o
  CC      block/bochs.o
  CC      block/vpc.o
  CC      block/vvfat.o
  CC      block/dmg.o
  CC      block/qcow2.o
  CC      block/qcow2-refcount.o
  CC      block/qcow2-cluster.o
  CC      block/qcow2-snapshot.o
  CC      block/qcow2-cache.o
  CC      block/qcow2-bitmap.o
  CC      block/qed.o
  CC      block/qed-l2-cache.o
  CC      block/qed-table.o
  CC      block/qed-cluster.o
  CC      block/qed-check.o
  CC      block/vhdx.o
  CC      block/vhdx-endian.o
  CC      block/vhdx-log.o
  CC      block/quorum.o
  CC      block/parallels.o
  CC      block/blkverify.o
  CC      block/blkdebug.o
  CC      block/blkreplay.o
  CC      block/block-backend.o
  CC      block/snapshot.o
  CC      block/qapi.o
  CC      block/file-posix.o
  CC      block/linux-aio.o
  CC      block/null.o
  CC      block/mirror.o
  CC      block/commit.o
  CC      block/io.o
  CC      block/throttle-groups.o
  CC      block/nbd.o
  CC      block/nbd-client.o
  CC      block/sheepdog.o
  CC      block/iscsi-opts.o
  CC      block/accounting.o
  CC      block/write-threshold.o
  CC      block/dirty-bitmap.o
  CC      block/backup.o
  CC      block/replication.o
  CC      block/crypto.o
  CC      nbd/server.o
  CC      nbd/client.o
  CC      nbd/common.o
  CC      block/iscsi.o
  CC      block/nfs.o
  CC      block/curl.o
  CC      block/rbd.o
  CC      block/gluster.o
  CC      block/ssh.o
  CC      block/dmg-bz2.o
  CC      crypto/init.o
  CC      crypto/hash.o
  CC      crypto/hash-nettle.o
  CC      crypto/hmac.o
  CC      crypto/aes.o
  CC      crypto/hmac-nettle.o
  CC      crypto/desrfb.o
  CC      crypto/cipher.o
  CC      crypto/tlscreds.o
  CC      crypto/tlscredsanon.o
  CC      crypto/tlscredsx509.o
  CC      crypto/tlssession.o
  CC      crypto/secret.o
  CC      crypto/random-gnutls.o
  CC      crypto/pbkdf.o
  CC      crypto/pbkdf-nettle.o
  CC      crypto/ivgen.o
  CC      crypto/ivgen-essiv.o
  CC      crypto/ivgen-plain64.o
  CC      crypto/ivgen-plain.o
  CC      crypto/afsplit.o
  CC      crypto/xts.o
  CC      crypto/block.o
  CC      crypto/block-qcow.o
  CC      io/channel.o
  CC      crypto/block-luks.o
  CC      io/channel-buffer.o
  CC      io/channel-command.o
  CC      io/channel-file.o
  CC      io/channel-socket.o
  CC      io/channel-tls.o
  CC      io/channel-websock.o
  CC      io/channel-watch.o
  CC      io/channel-util.o
  CC      io/dns-resolver.o
  CC      io/task.o
  CC      qom/object.o
  CC      qom/container.o
  CC      qom/qom-qobject.o
  CC      qom/object_interfaces.o
  GEN     qemu-img-cmds.h
  CC      qemu-io.o
  CC      fsdev/virtfs-proxy-helper.o
  CC      fsdev/9p-marshal.o
  CC      fsdev/9p-iov-marshal.o
  CC      qemu-bridge-helper.o
  CC      blockdev.o
  CC      blockdev-nbd.o
  CC      bootdevice.o
  CC      iothread.o
  CC      qdev-monitor.o
  CC      device-hotplug.o
  CC      os-posix.o
  CC      bt-host.o
  CC      bt-vhci.o
  CC      dma-helpers.o
  CC      vl.o
  CC      tpm.o
  CC      device_tree.o
  CC      qmp-marshal.o
  CC      qmp.o
  CC      hmp.o
  CC      cpus-common.o
  CC      audio/audio.o
  CC      audio/noaudio.o
  CC      audio/wavaudio.o
  CC      audio/mixeng.o
  CC      audio/sdlaudio.o
  CC      audio/ossaudio.o
  CC      audio/wavcapture.o
  CC      backends/rng-egd.o
  CC      backends/rng.o
  CC      backends/rng-random.o
  CC      backends/tpm.o
  CC      backends/hostmem.o
  CC      backends/hostmem-ram.o
  CC      backends/hostmem-file.o
  CC      backends/cryptodev.o
  CC      backends/cryptodev-builtin.o
  CC      block/stream.o
  CC      chardev/msmouse.o
  CC      chardev/wctablet.o
  CC      chardev/testdev.o
  CC      chardev/baum.o
  CC      disas/alpha.o
  CC      disas/arm.o
  CXX     disas/arm-a64.o
  CC      disas/cris.o
  CC      disas/hppa.o
  CC      disas/i386.o
  CC      disas/m68k.o
  CC      disas/microblaze.o
  CC      disas/mips.o
  CC      disas/nios2.o
  CC      disas/moxie.o
  CC      disas/ppc.o
  CC      disas/s390.o
  CC      disas/sh4.o
  CC      disas/sparc.o
  CC      disas/lm32.o
  CXX     disas/libvixl/vixl/utils.o
  CXX     disas/libvixl/vixl/compiler-intrinsics.o
  CXX     disas/libvixl/vixl/a64/decoder-a64.o
  CXX     disas/libvixl/vixl/a64/instructions-a64.o
  CXX     disas/libvixl/vixl/a64/disasm-a64.o
  CC      fsdev/qemu-fsdev.o
  CC      fsdev/qemu-fsdev-opts.o
  CC      fsdev/qemu-fsdev-throttle.o
  CC      fsdev/qemu-fsdev-dummy.o
  CC      hw/9pfs/9p.o
  CC      hw/9pfs/9p-util.o
  CC      hw/9pfs/9p-local.o
  CC      hw/9pfs/9p-xattr.o
  CC      hw/9pfs/9p-xattr-user.o
  CC      hw/9pfs/9p-posix-acl.o
  CC      hw/9pfs/cofs.o
  CC      hw/9pfs/coth.o
  CC      hw/9pfs/codir.o
  CC      hw/9pfs/cofile.o
  CC      hw/9pfs/9p-synth.o
  CC      hw/9pfs/coxattr.o
  CC      hw/9pfs/9p-proxy.o
  CC      hw/9pfs/9p-handle.o
  CC      hw/acpi/core.o
  CC      hw/acpi/piix4.o
  CC      hw/acpi/pcihp.o
  CC      hw/acpi/ich9.o
  CC      hw/acpi/tco.o
  CC      hw/acpi/cpu_hotplug.o
  CC      hw/acpi/memory_hotplug.o
  CC      hw/acpi/cpu.o
  CC      hw/acpi/nvdimm.o
  CC      hw/acpi/vmgenid.o
  CC      hw/acpi/hest_ghes.o
  CC      hw/acpi/acpi_interface.o
  CC      hw/acpi/bios-linker-loader.o
  CC      hw/acpi/aml-build.o
  CC      hw/acpi/ipmi.o
  CC      hw/acpi/acpi-stub.o
  CC      hw/acpi/ipmi-stub.o
  CC      hw/audio/sb16.o
  CC      hw/audio/es1370.o
  CC      hw/audio/ac97.o
  CC      hw/audio/fmopl.o
  CC      hw/audio/adlib.o
  CC      hw/audio/gus.o
  CC      hw/audio/gusemu_hal.o
  CC      hw/audio/gusemu_mixer.o
  CC      hw/audio/cs4231a.o
  CC      hw/audio/intel-hda.o
  CC      hw/audio/hda-codec.o
  CC      hw/audio/wm8750.o
  CC      hw/audio/pcspk.o
  CC      hw/audio/pl041.o
  CC      hw/audio/lm4549.o
  CC      hw/audio/cs4231.o
  CC      hw/audio/marvell_88w8618.o
  CC      hw/audio/milkymist-ac97.o
  CC      hw/audio/soundhw.o
  CC      hw/block/block.o
  CC      hw/block/cdrom.o
  CC      hw/block/hd-geometry.o
  CC      hw/block/fdc.o
  CC      hw/block/m25p80.o
  CC      hw/block/nand.o
  CC      hw/block/pflash_cfi01.o
  CC      hw/block/pflash_cfi02.o
  CC      hw/block/ecc.o
  CC      hw/block/onenand.o
  CC      hw/block/nvme.o
  CC      hw/bt/l2cap.o
  CC      hw/bt/core.o
  CC      hw/bt/sdp.o
  CC      hw/bt/hci.o
  CC      hw/bt/hid.o
  CC      hw/bt/hci-csr.o
  CC      hw/char/ipoctal232.o
  CC      hw/char/escc.o
  CC      hw/char/parallel.o
  CC      hw/char/pl011.o
  CC      hw/char/serial.o
  CC      hw/char/serial-isa.o
  CC      hw/char/serial-pci.o
  CC      hw/char/virtio-console.o
  CC      hw/char/xilinx_uartlite.o
  CC      hw/char/cadence_uart.o
  CC      hw/char/cmsdk-apb-uart.o
  CC      hw/char/etraxfs_ser.o
  CC      hw/char/debugcon.o
  CC      hw/char/grlib_apbuart.o
  CC      hw/char/lm32_juart.o
  CC      hw/char/imx_serial.o
  CC      hw/char/lm32_uart.o
  CC      hw/char/milkymist-uart.o
  CC      hw/char/sclpconsole.o
  CC      hw/char/sclpconsole-lm.o
  CC      hw/core/qdev.o
  CC      hw/core/qdev-properties.o
  CC      hw/core/bus.o
  CC      hw/core/reset.o
  CC      hw/core/fw-path-provider.o
  CC      hw/core/irq.o
  CC      hw/core/hotplug.o
  CC      hw/core/nmi.o
  CC      hw/core/empty_slot.o
  CC      hw/core/ptimer.o
  CC      hw/core/stream.o
  CC      hw/core/sysbus.o
  CC      hw/core/machine.o
  CC      hw/core/loader-fit.o
  CC      hw/core/loader.o
  CC      hw/core/qdev-properties-system.o
  CC      hw/core/register.o
  CC      hw/core/or-irq.o
  CC      hw/core/platform-bus.o
  CC      hw/cpu/core.o
  CC      hw/display/ads7846.o
  CC      hw/display/g364fb.o
  CC      hw/display/cirrus_vga.o
  CC      hw/display/jazz_led.o
  CC      hw/display/pl110.o
  CC      hw/display/ssd0303.o
  CC      hw/display/ssd0323.o
  CC      hw/display/vga-pci.o
  CC      hw/display/vga-isa.o
  CC      hw/display/vga-isa-mm.o
  CC      hw/display/vmware_vga.o
  CC      hw/display/blizzard.o
  CC      hw/display/exynos4210_fimd.o
  CC      hw/display/framebuffer.o
  CC      hw/display/milkymist-vgafb.o
  CC      hw/display/tc6393xb.o
  CC      hw/display/milkymist-tmu2.o
  CC      hw/dma/puv3_dma.o
  CC      hw/dma/rc4030.o
  CC      hw/dma/pl080.o
  CC      hw/dma/pl330.o
  CC      hw/dma/i82374.o
  CC      hw/dma/i8257.o
  CC      hw/dma/xilinx_axidma.o
  CC      hw/dma/xlnx-zynq-devcfg.o
  CC      hw/dma/etraxfs_dma.o
  CC      hw/dma/sparc32_dma.o
  CC      hw/dma/sun4m_iommu.o
  CC      hw/gpio/max7310.o
  CC      hw/gpio/pl061.o
  CC      hw/gpio/puv3_gpio.o
  CC      hw/gpio/zaurus.o
  CC      hw/gpio/gpio_key.o
  CC      hw/gpio/mpc8xxx.o
  CC      hw/i2c/core.o
  CC      hw/i2c/smbus.o
  CC      hw/i2c/smbus_eeprom.o
  CC      hw/i2c/i2c-ddc.o
  CC      hw/i2c/versatile_i2c.o
  CC      hw/i2c/smbus_ich9.o
  CC      hw/i2c/pm_smbus.o
  CC      hw/i2c/bitbang_i2c.o
  CC      hw/i2c/exynos4210_i2c.o
  CC      hw/i2c/imx_i2c.o
  CC      hw/i2c/aspeed_i2c.o
  CC      hw/ide/core.o
  CC      hw/ide/atapi.o
  CC      hw/ide/qdev.o
  CC      hw/ide/pci.o
  CC      hw/ide/isa.o
  CC      hw/ide/piix.o
  CC      hw/ide/cmd646.o
  CC      hw/ide/macio.o
  CC      hw/ide/via.o
  CC      hw/ide/mmio.o
  CC      hw/ide/microdrive.o
  CC      hw/ide/ahci.o
  CC      hw/input/adb.o
  CC      hw/input/hid.o
  CC      hw/ide/ich.o
  CC      hw/input/lm832x.o
  CC      hw/input/pckbd.o
  CC      hw/input/pl050.o
  CC      hw/input/ps2.o
  CC      hw/input/stellaris_input.o
  CC      hw/input/tsc2005.o
  CC      hw/input/vmmouse.o
  CC      hw/input/virtio-input.o
  CC      hw/input/virtio-input-hid.o
  CC      hw/input/virtio-input-host.o
  CC      hw/intc/heathrow_pic.o
  CC      hw/intc/i8259_common.o
  CC      hw/intc/i8259.o
  CC      hw/intc/pl190.o
  CC      hw/intc/puv3_intc.o
  CC      hw/intc/xilinx_intc.o
  CC      hw/intc/imx_avic.o
  CC      hw/intc/etraxfs_pic.o
  CC      hw/intc/realview_gic.o
  CC      hw/intc/lm32_pic.o
  CC      hw/intc/slavio_intctl.o
  CC      hw/intc/ioapic_common.o
  CC      hw/intc/arm_gic_common.o
  CC      hw/intc/arm_gic.o
  CC      hw/intc/arm_gicv2m.o
  CC      hw/intc/arm_gicv3_common.o
  CC      hw/intc/arm_gicv3.o
  CC      hw/intc/arm_gicv3_dist.o
  CC      hw/intc/arm_gicv3_redist.o
  CC      hw/intc/arm_gicv3_its_common.o
  CC      hw/intc/openpic.o
  CC      hw/intc/intc.o
  CC      hw/ipack/ipack.o
  CC      hw/ipack/tpci200.o
  CC      hw/ipmi/ipmi.o
  CC      hw/ipmi/ipmi_bmc_sim.o
  CC      hw/ipmi/ipmi_bmc_extern.o
  CC      hw/ipmi/isa_ipmi_kcs.o
  CC      hw/ipmi/isa_ipmi_bt.o
  CC      hw/isa/isa-bus.o
  CC      hw/isa/apm.o
  CC      hw/isa/i82378.o
  CC      hw/isa/pc87312.o
  CC      hw/isa/piix4.o
  CC      hw/isa/vt82c686.o
  CC      hw/mem/pc-dimm.o
  CC      hw/mem/nvdimm.o
  CC      hw/misc/applesmc.o
  CC      hw/misc/max111x.o
  CC      hw/misc/tmp105.o
  CC      hw/misc/tmp421.o
  CC      hw/misc/debugexit.o
  CC      hw/misc/sga.o
  CC      hw/misc/pc-testdev.o
  CC      hw/misc/pci-testdev.o
  CC      hw/misc/edu.o
  CC      hw/misc/arm_l2x0.o
  CC      hw/misc/unimp.o
  CC      hw/misc/arm_integrator_debug.o
  CC      hw/misc/a9scu.o
  CC      hw/misc/arm11scu.o
  CC      hw/misc/puv3_pm.o
  CC      hw/misc/macio/macio.o
  CC      hw/misc/macio/cuda.o
  CC      hw/misc/macio/mac_dbdma.o
  CC      hw/net/dp8393x.o
  CC      hw/net/ne2000.o
  CC      hw/net/eepro100.o
  CC      hw/net/pcnet-pci.o
  CC      hw/net/pcnet.o
  CC      hw/net/e1000.o
  CC      hw/net/e1000x_common.o
  CC      hw/net/net_tx_pkt.o
  CC      hw/net/net_rx_pkt.o
  CC      hw/net/e1000e.o
  CC      hw/net/e1000e_core.o
  CC      hw/net/rtl8139.o
  CC      hw/net/vmxnet3.o
  CC      hw/net/smc91c111.o
  CC      hw/net/lan9118.o
  CC      hw/net/ne2000-isa.o
  CC      hw/net/opencores_eth.o
  CC      hw/net/xgmac.o
  CC      hw/net/mipsnet.o
  CC      hw/net/xilinx_axienet.o
  CC      hw/net/allwinner_emac.o
  CC      hw/net/imx_fec.o
  CC      hw/net/cadence_gem.o
  CC      hw/net/stellaris_enet.o
  CC      hw/net/lance.o
  CC      hw/net/ftgmac100.o
  CC      hw/net/rocker/rocker.o
  CC      hw/net/rocker/rocker_fp.o
  CC      hw/net/rocker/rocker_desc.o
  CC      hw/net/rocker/rocker_world.o
  CC      hw/net/rocker/rocker_of_dpa.o
  CC      hw/nvram/ds1225y.o
  CC      hw/nvram/eeprom93xx.o
  CC      hw/nvram/fw_cfg.o
  CC      hw/nvram/chrp_nvram.o
  CC      hw/nvram/mac_nvram.o
  CC      hw/pci-bridge/pci_bridge_dev.o
  CC      hw/pci-bridge/pcie_root_port.o
  CC      hw/pci-bridge/gen_pcie_root_port.o
  CC      hw/pci-bridge/pci_expander_bridge.o
  CC      hw/pci-bridge/xio3130_upstream.o
  CC      hw/pci-bridge/xio3130_downstream.o
  CC      hw/pci-bridge/ioh3420.o
  CC      hw/pci-bridge/i82801b11.o
  CC      hw/pci-bridge/dec.o
  CC      hw/pci-host/pam.o
  CC      hw/pci-host/prep.o
  CC      hw/pci-host/grackle.o
  CC      hw/pci-host/uninorth.o
  CC      hw/pci-host/ppce500.o
  CC      hw/pci-host/apb.o
  CC      hw/pci-host/versatile.o
  CC      hw/pci-host/bonito.o
  CC      hw/pci-host/piix.o
  CC      hw/pci-host/q35.o
  CC      hw/pci-host/gpex.o
  CC      hw/pci-host/xilinx-pcie.o
  CC      hw/pci/pci.o
  CC      hw/pci/pci_bridge.o
  CC      hw/pci/msix.o
  CC      hw/pci/msi.o
  CC      hw/pci/shpc.o
  CC      hw/pci/slotid_cap.o
  CC      hw/pci/pci_host.o
  CC      hw/pci/pcie.o
  CC      hw/pci/pcie_host.o
  CC      hw/pci/pcie_aer.o
  CC      hw/pci/pcie_port.o
  CC      hw/pci/pci-stub.o
  CC      hw/pcmcia/pcmcia.o
  CC      hw/scsi/scsi-disk.o
  CC      hw/scsi/scsi-generic.o
  CC      hw/scsi/scsi-bus.o
  CC      hw/scsi/lsi53c895a.o
  CC      hw/scsi/mptsas.o
  CC      hw/scsi/mptconfig.o
  CC      hw/scsi/mptendian.o
  CC      hw/scsi/vmw_pvscsi.o
  CC      hw/scsi/megasas.o
  CC      hw/scsi/esp.o
  CC      hw/scsi/esp-pci.o
  CC      hw/sd/ssi-sd.o
  CC      hw/sd/pl181.o
  CC      hw/sd/sd.o
  CC      hw/sd/core.o
  CC      hw/sd/sdhci.o
  CC      hw/smbios/smbios.o
  CC      hw/smbios/smbios_type_38.o
  CC      hw/smbios/smbios-stub.o
  CC      hw/smbios/smbios_type_38-stub.o
  CC      hw/ssi/pl022.o
  CC      hw/ssi/ssi.o
  CC      hw/ssi/xilinx_spips.o
  CC      hw/ssi/xilinx_spi.o
  CC      hw/ssi/aspeed_smc.o
  CC      hw/ssi/stm32f2xx_spi.o
  CC      hw/timer/arm_timer.o
  CC      hw/timer/arm_mptimer.o
  CC      hw/timer/armv7m_systick.o
  CC      hw/timer/a9gtimer.o
  CC      hw/timer/cadence_ttc.o
  CC      hw/timer/ds1338.o
  CC      hw/timer/hpet.o
  CC      hw/timer/i8254.o
  CC      hw/timer/m48t59.o
  CC      hw/timer/i8254_common.o
  CC      hw/timer/m48t59-isa.o
  CC      hw/timer/pl031.o
  CC      hw/timer/puv3_ost.o
  CC      hw/timer/twl92230.o
  CC      hw/timer/xilinx_timer.o
  CC      hw/timer/etraxfs_timer.o
  CC      hw/timer/slavio_timer.o
  CC      hw/timer/grlib_gptimer.o
  CC      hw/timer/imx_epit.o
  CC      hw/timer/imx_gpt.o
  CC      hw/timer/lm32_timer.o
  CC      hw/timer/milkymist-sysctl.o
  CC      hw/timer/stm32f2xx_timer.o
  CC      hw/timer/aspeed_timer.o
  CC      hw/timer/cmsdk-apb-timer.o
  CC      hw/timer/sun4v-rtc.o
  CC      hw/tpm/tpm_tis.o
  CC      hw/usb/core.o
  CC      hw/usb/combined-packet.o
  CC      hw/usb/bus.o
  CC      hw/usb/libhw.o
  CC      hw/usb/desc.o
  CC      hw/usb/desc-msos.o
  CC      hw/usb/hcd-uhci.o
  CC      hw/usb/hcd-ohci.o
  CC      hw/usb/hcd-ehci.o
  CC      hw/usb/hcd-ehci-pci.o
  CC      hw/usb/hcd-ehci-sysbus.o
  CC      hw/usb/hcd-xhci.o
  CC      hw/usb/hcd-xhci-nec.o
  CC      hw/usb/hcd-musb.o
  CC      hw/usb/dev-hub.o
  CC      hw/usb/dev-hid.o
  CC      hw/usb/dev-wacom.o
  CC      hw/usb/dev-storage.o
  CC      hw/usb/dev-uas.o
  CC      hw/usb/dev-audio.o
  CC      hw/usb/dev-serial.o
  CC      hw/usb/dev-network.o
  CC      hw/usb/dev-bluetooth.o
  CC      hw/usb/dev-smartcard-reader.o
  CC      hw/usb/ccid-card-passthru.o
  CC      hw/usb/ccid-card-emulated.o
  CC      hw/usb/dev-mtp.o
  CC      hw/usb/redirect.o
  CC      hw/usb/quirks.o
  CC      hw/usb/host-libusb.o
  CC      hw/usb/host-legacy.o
  CC      hw/virtio/virtio-rng.o
  CC      hw/virtio/virtio-pci.o
  CC      hw/virtio/virtio-bus.o
  CC      hw/virtio/virtio-mmio.o
  CC      hw/virtio/vhost-stub.o
  CC      hw/watchdog/watchdog.o
  CC      hw/watchdog/wdt_i6300esb.o
  CC      hw/watchdog/wdt_ib700.o
  CC      hw/watchdog/wdt_diag288.o
  CC      hw/watchdog/wdt_aspeed.o
  CC      migration/migration.o
  CC      migration/fd.o
  CC      migration/socket.o
  CC      migration/exec.o
  CC      migration/tls.o
  CC      migration/channel.o
  CC      migration/savevm.o
  CC      migration/colo-comm.o
  CC      migration/colo.o
  CC      migration/colo-failover.o
  CC      migration/vmstate-types.o
  CC      migration/vmstate.o
  CC      migration/page_cache.o
  CC      migration/qemu-file.o
  CC      migration/global_state.o
  CC      migration/qemu-file-channel.o
  CC      migration/xbzrle.o
  CC      migration/postcopy-ram.o
  CC      migration/qjson.o
  CC      migration/block.o
  CC      net/net.o
  CC      net/queue.o
  CC      net/checksum.o
  CC      net/util.o
  CC      net/hub.o
  CC      net/socket.o
  CC      net/dump.o
  CC      net/eth.o
  CC      net/l2tpv3.o
  CC      net/vhost-user.o
  CC      net/slirp.o
  CC      net/filter.o
  CC      net/filter-buffer.o
  CC      net/colo-compare.o
  CC      net/filter-mirror.o
  CC      net/colo.o
  CC      net/filter-rewriter.o
  CC      net/filter-replay.o
  CC      net/tap.o
  CC      net/tap-linux.o
  CC      qom/cpu.o
  CC      replay/replay.o
  CC      replay/replay-internal.o
  CC      replay/replay-events.o
  CC      replay/replay-time.o
  CC      replay/replay-input.o
  CC      replay/replay-snapshot.o
  CC      replay/replay-char.o
  CC      replay/replay-net.o
  CC      replay/replay-audio.o
  CC      slirp/cksum.o
  CC      slirp/ip_icmp.o
  CC      slirp/if.o
  CC      slirp/ip6_icmp.o
  CC      slirp/ip6_input.o
  CC      slirp/ip_input.o
  CC      slirp/ip6_output.o
  CC      slirp/ip_output.o
  CC      slirp/dnssearch.o
  CC      slirp/dhcpv6.o
  CC      slirp/slirp.o
  CC      slirp/mbuf.o
  CC      slirp/sbuf.o
  CC      slirp/misc.o
  CC      slirp/socket.o
  CC      slirp/tcp_input.o
  CC      slirp/tcp_output.o
  CC      slirp/tcp_subr.o
  CC      slirp/tcp_timer.o
  CC      slirp/udp.o
  CC      slirp/udp6.o
  CC      slirp/bootp.o
  CC      slirp/tftp.o
  CC      slirp/arp_table.o
  CC      slirp/ndp_table.o
  CC      slirp/ncsi.o
  CC      ui/keymaps.o
  CC      ui/console.o
  CC      ui/cursor.o
  CC      ui/qemu-pixman.o
  CC      ui/input.o
  CC      ui/input-keymap.o
  CC      ui/input-legacy.o
  CC      ui/input-linux.o
  CC      ui/sdl2.o
  CC      ui/sdl2-input.o
  CC      ui/sdl2-2d.o
  CC      ui/sdl2-gl.o
  CC      ui/x_keymap.o
  CC      ui/curses.o
  CC      ui/vnc-enc-zlib.o
  CC      ui/vnc.o
  CC      ui/vnc-enc-hextile.o
  CC      ui/vnc-palette.o
  CC      ui/vnc-enc-tight.o
  CC      ui/vnc-enc-zrle.o
  CC      ui/vnc-auth-vencrypt.o
  CC      ui/vnc-auth-sasl.o
  CC      ui/vnc-ws.o
  CC      ui/vnc-jobs.o
  CC      ui/gtk.o
  CC      ui/shader.o
  VERT    ui/shader/texture-blit-vert.h
  FRAG    ui/shader/texture-blit-frag.h
  CC      ui/egl-helpers.o
  CC      ui/egl-context.o
  CC      ui/egl-headless.o
  CC      ui/gtk-gl-area.o
  CC      chardev/char.o
  CC      chardev/char-fd.o
  CC      chardev/char-fe.o
  CC      chardev/char-file.o
  CC      chardev/char-io.o
  CC      chardev/char-mux.o
  CC      chardev/char-null.o
  CC      chardev/char-parallel.o
  CC      chardev/char-pipe.o
  CC      chardev/char-pty.o
  CC      chardev/char-ringbuf.o
  CC      chardev/char-serial.o
  CC      chardev/char-socket.o
  CC      chardev/char-stdio.o
  CC      chardev/char-udp.o
  LINK    tests/qemu-iotests/socket_scm_helper
  CCAS    s390-ccw/start.o
  GEN     qemu-doc.html
  CC      s390-ccw/main.o
  CC      s390-ccw/bootmap.o
  CC      s390-ccw/sclp.o
  CC      s390-ccw/virtio.o
  CC      s390-ccw/virtio-scsi.o
  GEN     qemu-doc.txt
  CC      s390-ccw/virtio-blkdev.o
s390-netboot.img not built since roms/SLOF/ is not available.
  GEN     qemu.1
  BUILD   s390-ccw/s390-ccw.elf
  STRIP   s390-ccw/s390-ccw.img
  GEN     docs/interop/qemu-qmp-ref.html
  GEN     docs/interop/qemu-qmp-ref.txt
  GEN     docs/interop/qemu-qmp-ref.7
  GEN     docs/interop/qemu-ga-ref.html
  GEN     docs/interop/qemu-ga-ref.txt
  GEN     docs/interop/qemu-ga-ref.7
  CC      qga/commands.o
  CC      qga/guest-agent-command-state.o
  CC      qga/main.o
  CC      qga/channel-posix.o
  CC      qga/commands-posix.o
  CC      qga/qapi-generated/qga-qapi-types.o
  CC      qga/qapi-generated/qga-qapi-visit.o
  AR      libqemuutil.a
  CC      qga/qapi-generated/qga-qmp-marshal.o
  AR      libqemustub.a
  CC      qemu-img.o
  LINK    qemu-io
  LINK    fsdev/virtfs-proxy-helper
  LINK    qemu-bridge-helper
  CC      ui/console-gl.o
  LINK    qemu-ga
  LINK    ivshmem-client
  LINK    ivshmem-server
  LINK    qemu-nbd
  GEN     alpha-softmmu/hmp-commands.h
  GEN     alpha-softmmu/hmp-commands-info.h
  GEN     alpha-softmmu/config-target.h
  GEN     cris-softmmu/hmp-commands.h
  GEN     cris-softmmu/hmp-commands-info.h
  GEN     cris-softmmu/config-target.h
  CC      alpha-softmmu/exec.o
  CC      cris-softmmu/exec.o
  GEN     aarch64-softmmu/hmp-commands.h
  GEN     arm-softmmu/hmp-commands.h
  GEN     aarch64-softmmu/hmp-commands-info.h
  GEN     arm-softmmu/hmp-commands-info.h
  GEN     arm-softmmu/config-target.h
  GEN     aarch64-softmmu/config-target.h
  CC      arm-softmmu/exec.o
  CC      aarch64-softmmu/exec.o
  CC      arm-softmmu/tcg/tcg.o
  CC      alpha-softmmu/tcg/tcg.o
  CC      cris-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/tcg/tcg.o
  CC      cris-softmmu/tcg/tcg-op.o
  CC      arm-softmmu/tcg/tcg-op.o
  CC      alpha-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/tcg/tcg-op.o
  CC      cris-softmmu/tcg/optimize.o
  CC      arm-softmmu/tcg/optimize.o
  CC      cris-softmmu/tcg/tcg-common.o
  CC      alpha-softmmu/tcg/optimize.o
  CC      cris-softmmu/tcg/tcg-runtime.o
  CC      arm-softmmu/tcg/tcg-common.o
  CC      alpha-softmmu/tcg/tcg-common.o
  CC      cris-softmmu/fpu/softfloat.o
  CC      arm-softmmu/tcg/tcg-runtime.o
  CC      alpha-softmmu/tcg/tcg-runtime.o
  CC      aarch64-softmmu/tcg/optimize.o
  CC      arm-softmmu/fpu/softfloat.o
  CC      alpha-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/tcg/tcg-runtime.o
  CC      cris-softmmu/disas.o
  CC      aarch64-softmmu/fpu/softfloat.o
  CC      cris-softmmu/hax-stub.o
  CC      arm-softmmu/disas.o
  CC      cris-softmmu/arch_init.o
  CC      cris-softmmu/cpus.o
  CC      alpha-softmmu/disas.o
  GEN     arm-softmmu/gdbstub-xml.c
  CC      alpha-softmmu/hax-stub.o
  CC      cris-softmmu/monitor.o
  CC      alpha-softmmu/arch_init.o
  CC      arm-softmmu/hax-stub.o
  CC      arm-softmmu/arch_init.o
  CC      alpha-softmmu/cpus.o
  CC      arm-softmmu/cpus.o
  CC      alpha-softmmu/monitor.o
  CC      aarch64-softmmu/disas.o
  CC      cris-softmmu/gdbstub.o
  GEN     aarch64-softmmu/gdbstub-xml.c
  CC      arm-softmmu/monitor.o
  CC      aarch64-softmmu/hax-stub.o
  CC      cris-softmmu/balloon.o
  CC      aarch64-softmmu/arch_init.o
  CC      cris-softmmu/ioport.o
  CC      alpha-softmmu/gdbstub.o
  CC      aarch64-softmmu/cpus.o
  CC      cris-softmmu/numa.o
  CC      arm-softmmu/gdbstub.o
  CC      aarch64-softmmu/monitor.o
  CC      arm-softmmu/balloon.o
  CC      cris-softmmu/qtest.o
  CC      alpha-softmmu/balloon.o
  CC      arm-softmmu/ioport.o
  CC      cris-softmmu/memory.o
  CC      alpha-softmmu/ioport.o
  CC      arm-softmmu/numa.o
  CC      alpha-softmmu/numa.o
  CC      arm-softmmu/qtest.o
  CC      alpha-softmmu/qtest.o
  CC      aarch64-softmmu/gdbstub.o
  CC      cris-softmmu/memory_mapping.o
  CC      arm-softmmu/memory.o
  CC      alpha-softmmu/memory.o
  CC      aarch64-softmmu/balloon.o
  CC      cris-softmmu/dump.o
  CC      aarch64-softmmu/ioport.o
  CC      cris-softmmu/migration/ram.o
  CC      aarch64-softmmu/numa.o
  CC      arm-softmmu/memory_mapping.o
  CC      alpha-softmmu/memory_mapping.o
  CC      alpha-softmmu/dump.o
  CC      cris-softmmu/accel/accel.o
  CC      aarch64-softmmu/qtest.o
  CC      arm-softmmu/dump.o
  CC      cris-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/memory.o
  CC      alpha-softmmu/migration/ram.o
  CC      cris-softmmu/accel/tcg/tcg-all.o
  CC      arm-softmmu/migration/ram.o
  CC      cris-softmmu/accel/tcg/cputlb.o
  CC      alpha-softmmu/accel/accel.o
  CC      aarch64-softmmu/memory_mapping.o
  CC      arm-softmmu/accel/accel.o
  CC      alpha-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/dump.o
  CC      alpha-softmmu/accel/tcg/tcg-all.o
  CC      arm-softmmu/accel/stubs/kvm-stub.o
  CC      alpha-softmmu/accel/tcg/cputlb.o
  CC      cris-softmmu/accel/tcg/cpu-exec.o
  CC      arm-softmmu/accel/tcg/tcg-all.o
  CC      aarch64-softmmu/migration/ram.o
  CC      arm-softmmu/accel/tcg/cputlb.o
  CC      cris-softmmu/accel/tcg/cpu-exec-common.o
  CC      cris-softmmu/accel/tcg/translate-all.o
  CC      aarch64-softmmu/accel/accel.o
  CC      alpha-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/accel/stubs/kvm-stub.o
  CC      cris-softmmu/hw/core/generic-loader.o
  CC      alpha-softmmu/accel/tcg/cpu-exec-common.o
  CC      aarch64-softmmu/accel/tcg/tcg-all.o
  CC      cris-softmmu/hw/core/null-machine.o
  CC      alpha-softmmu/accel/tcg/translate-all.o
  CC      arm-softmmu/accel/tcg/cpu-exec.o
  CC      cris-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/accel/tcg/cputlb.o
  CC      cris-softmmu/hw/net/etraxfs_eth.o
  CC      alpha-softmmu/hw/9pfs/virtio-9p-device.o
  CC      cris-softmmu/hw/net/vhost_net.o
  CC      arm-softmmu/accel/tcg/cpu-exec-common.o
  CC      alpha-softmmu/hw/block/virtio-blk.o
  CC      cris-softmmu/hw/net/rocker/qmp-norocker.o
  CC      arm-softmmu/accel/tcg/translate-all.o
  CC      cris-softmmu/hw/vfio/common.o
  CC      alpha-softmmu/hw/block/dataplane/virtio-blk.o
  CC      arm-softmmu/hw/9pfs/virtio-9p-device.o
  CC      alpha-softmmu/hw/char/virtio-serial-bus.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec.o
  CC      cris-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/adc/stm32f2xx_adc.o
  CC      cris-softmmu/hw/vfio/spapr.o
  CC      alpha-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/block/virtio-blk.o
  CC      alpha-softmmu/hw/core/null-machine.o
  CC      cris-softmmu/hw/cris/boot.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec-common.o
  CC      cris-softmmu/hw/cris/axis_dev88.o
  CC      arm-softmmu/hw/block/dataplane/virtio-blk.o
  CC      alpha-softmmu/hw/display/vga.o
  CC      aarch64-softmmu/accel/tcg/translate-all.o
  CC      arm-softmmu/hw/char/exynos4210_uart.o
  CC      cris-softmmu/target/cris/translate.o
  CC      arm-softmmu/hw/char/omap_uart.o
  CC      aarch64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      arm-softmmu/hw/char/digic-uart.o
  CC      alpha-softmmu/hw/display/virtio-gpu.o
  CC      arm-softmmu/hw/char/stm32f2xx_usart.o
  CC      aarch64-softmmu/hw/adc/stm32f2xx_adc.o
  CC      arm-softmmu/hw/char/bcm2835_aux.o
  CC      aarch64-softmmu/hw/block/virtio-blk.o
  CC      arm-softmmu/hw/char/virtio-serial-bus.o
  CC      alpha-softmmu/hw/display/virtio-gpu-3d.o
  CC      cris-softmmu/target/cris/op_helper.o
  CC      arm-softmmu/hw/core/generic-loader.o
  CC      aarch64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      cris-softmmu/target/cris/helper.o
  CC      arm-softmmu/hw/core/null-machine.o
  CC      alpha-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-softmmu/hw/char/exynos4210_uart.o
  CC      cris-softmmu/target/cris/cpu.o
  CC      arm-softmmu/hw/cpu/arm11mpcore.o
  CC      aarch64-softmmu/hw/char/omap_uart.o
  CC      cris-softmmu/target/cris/gdbstub.o
  CC      alpha-softmmu/hw/misc/ivshmem.o
  CC      aarch64-softmmu/hw/char/digic-uart.o
  CC      arm-softmmu/hw/cpu/realview_mpcore.o
  CC      cris-softmmu/target/cris/mmu.o
  CC      alpha-softmmu/hw/misc/mmio_interface.o
  CC      arm-softmmu/hw/cpu/a9mpcore.o
  CC      aarch64-softmmu/hw/char/stm32f2xx_usart.o
  CC      alpha-softmmu/hw/net/virtio-net.o
  CC      cris-softmmu/target/cris/machine.o
  CC      arm-softmmu/hw/cpu/a15mpcore.o
  CC      aarch64-softmmu/hw/char/bcm2835_aux.o
  GEN     trace/generated-helpers.c
  CC      cris-softmmu/trace/control-target.o
  CC      arm-softmmu/hw/display/omap_dss.o
  CC      aarch64-softmmu/hw/char/virtio-serial-bus.o
  CC      alpha-softmmu/hw/net/vhost_net.o
  CC      cris-softmmu/trace/generated-helpers.o
  CC      arm-softmmu/hw/display/omap_lcdc.o
  CC      alpha-softmmu/hw/scsi/virtio-scsi.o
  LINK    cris-softmmu/qemu-system-cris
  CC      aarch64-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/display/pxa2xx_lcd.o
  CC      alpha-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/hw/cpu/arm11mpcore.o
  CC      alpha-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/cpu/realview_mpcore.o
  CC      alpha-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-softmmu/hw/cpu/a9mpcore.o
  CC      arm-softmmu/hw/display/bcm2835_fb.o
  CC      alpha-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/cpu/a15mpcore.o
  CC      alpha-softmmu/hw/timer/mc146818rtc.o
  GEN     i386-softmmu/hmp-commands.h
  CC      aarch64-softmmu/hw/display/omap_dss.o
  GEN     i386-softmmu/hmp-commands-info.h
  GEN     i386-softmmu/config-target.h
  CC      arm-softmmu/hw/display/vga.o
  CC      i386-softmmu/exec.o
  CC      alpha-softmmu/hw/vfio/common.o
  CC      aarch64-softmmu/hw/display/omap_lcdc.o
  CC      arm-softmmu/hw/display/virtio-gpu.o
  CC      aarch64-softmmu/hw/display/pxa2xx_lcd.o
  CC      alpha-softmmu/hw/vfio/pci.o
  CC      i386-softmmu/tcg/tcg.o
  CC      arm-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/display/bcm2835_fb.o
  CC      alpha-softmmu/hw/vfio/pci-quirks.o
  CC      aarch64-softmmu/hw/display/vga.o
  CC      arm-softmmu/hw/display/virtio-gpu-pci.o
  CC      alpha-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/dma/omap_dma.o
  CC      alpha-softmmu/hw/vfio/spapr.o
  CC      i386-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/hw/display/virtio-gpu.o
  CC      alpha-softmmu/hw/virtio/virtio.o
  CC      arm-softmmu/hw/dma/soc_dma.o
  CC      arm-softmmu/hw/dma/pxa2xx_dma.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-3d.o
  CC      alpha-softmmu/hw/virtio/virtio-balloon.o
  CC      arm-softmmu/hw/dma/bcm2835_dma.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-pci.o
  CC      alpha-softmmu/hw/virtio/vhost.o
  CC      arm-softmmu/hw/gpio/omap_gpio.o
  CC      arm-softmmu/hw/gpio/imx_gpio.o
  CC      aarch64-softmmu/hw/display/dpcd.o
  CC      i386-softmmu/tcg/optimize.o
  CC      arm-softmmu/hw/gpio/bcm2835_gpio.o
  CC      aarch64-softmmu/hw/display/xlnx_dp.o
  CC      alpha-softmmu/hw/virtio/vhost-backend.o
  CC      arm-softmmu/hw/i2c/omap_i2c.o
  CC      alpha-softmmu/hw/virtio/vhost-user.o
  CC      aarch64-softmmu/hw/dma/xlnx_dpdma.o
  CC      arm-softmmu/hw/input/pxa2xx_keypad.o
  CC      alpha-softmmu/hw/virtio/vhost-vsock.o
  CC      i386-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/hw/dma/omap_dma.o
  CC      arm-softmmu/hw/input/tsc210x.o
  CC      alpha-softmmu/hw/virtio/virtio-crypto.o
  CC      i386-softmmu/tcg/tcg-runtime.o
  CC      alpha-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      aarch64-softmmu/hw/dma/soc_dma.o
  CC      i386-softmmu/fpu/softfloat.o
  CC      arm-softmmu/hw/intc/armv7m_nvic.o
  CC      aarch64-softmmu/hw/dma/pxa2xx_dma.o
  CC      alpha-softmmu/hw/alpha/dp264.o
  CC      arm-softmmu/hw/intc/exynos4210_gic.o
  CC      aarch64-softmmu/hw/dma/bcm2835_dma.o
  CC      alpha-softmmu/hw/alpha/pci.o
  CC      aarch64-softmmu/hw/gpio/omap_gpio.o
  CC      arm-softmmu/hw/intc/exynos4210_combiner.o
  CC      alpha-softmmu/hw/alpha/typhoon.o
  CC      aarch64-softmmu/hw/gpio/imx_gpio.o
  CC      arm-softmmu/hw/intc/omap_intc.o
  CC      aarch64-softmmu/hw/gpio/bcm2835_gpio.o
  CC      aarch64-softmmu/hw/i2c/omap_i2c.o
  CC      alpha-softmmu/target/alpha/machine.o
  CC      arm-softmmu/hw/intc/bcm2835_ic.o
  CC      aarch64-softmmu/hw/input/pxa2xx_keypad.o
  CC      arm-softmmu/hw/intc/bcm2836_control.o
  CC      alpha-softmmu/target/alpha/translate.o
  CC      aarch64-softmmu/hw/input/tsc210x.o
  CC      arm-softmmu/hw/intc/allwinner-a10-pic.o
  CC      i386-softmmu/disas.o
  CC      arm-softmmu/hw/intc/aspeed_vic.o
  GEN     i386-softmmu/gdbstub-xml.c
  CC      aarch64-softmmu/hw/intc/armv7m_nvic.o
  CC      arm-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      alpha-softmmu/target/alpha/helper.o
  CC      i386-softmmu/hax-stub.o
  CC      aarch64-softmmu/hw/intc/exynos4210_gic.o
  CC      alpha-softmmu/target/alpha/cpu.o
  CC      i386-softmmu/arch_init.o
  CC      aarch64-softmmu/hw/intc/exynos4210_combiner.o
  CC      alpha-softmmu/target/alpha/int_helper.o
  CC      i386-softmmu/cpus.o
  CC      arm-softmmu/hw/misc/ivshmem.o
  CC      aarch64-softmmu/hw/intc/omap_intc.o
  CC      alpha-softmmu/target/alpha/fpu_helper.o
  CC      i386-softmmu/monitor.o
  CC      alpha-softmmu/target/alpha/vax_helper.o
  CC      arm-softmmu/hw/misc/arm_sysctl.o
  CC      aarch64-softmmu/hw/intc/bcm2835_ic.o
  CC      alpha-softmmu/target/alpha/sys_helper.o
  CC      aarch64-softmmu/hw/intc/bcm2836_control.o
  CC      arm-softmmu/hw/misc/cbus.o
  CC      alpha-softmmu/target/alpha/mem_helper.o
  CC      aarch64-softmmu/hw/intc/allwinner-a10-pic.o
  CC      arm-softmmu/hw/misc/exynos4210_pmu.o
  CC      alpha-softmmu/target/alpha/gdbstub.o
  CC      arm-softmmu/hw/misc/exynos4210_clk.o
  GEN     trace/generated-helpers.c
  CC      alpha-softmmu/trace/control-target.o
  CC      aarch64-softmmu/hw/intc/aspeed_vic.o
  CC      arm-softmmu/hw/misc/exynos4210_rng.o
  CC      i386-softmmu/gdbstub.o
  CC      aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      arm-softmmu/hw/misc/imx_ccm.o
  CC      alpha-softmmu/trace/generated-helpers.o
  CC      arm-softmmu/hw/misc/imx31_ccm.o
  LINK    alpha-softmmu/qemu-system-alpha
  CC      i386-softmmu/balloon.o
  CC      arm-softmmu/hw/misc/imx25_ccm.o
  CC      i386-softmmu/ioport.o
  CC      arm-softmmu/hw/misc/imx6_ccm.o
  CC      aarch64-softmmu/hw/misc/ivshmem.o
  CC      i386-softmmu/numa.o
  CC      arm-softmmu/hw/misc/imx6_src.o
  CC      aarch64-softmmu/hw/misc/arm_sysctl.o
  CC      arm-softmmu/hw/misc/mst_fpga.o
  CC      aarch64-softmmu/hw/misc/cbus.o
  CC      i386-softmmu/qtest.o
  CC      arm-softmmu/hw/misc/omap_clk.o
  CC      aarch64-softmmu/hw/misc/exynos4210_pmu.o
  CC      aarch64-softmmu/hw/misc/exynos4210_clk.o
  CC      i386-softmmu/memory.o
  CC      arm-softmmu/hw/misc/omap_gpmc.o
  GEN     lm32-softmmu/hmp-commands.h
  GEN     lm32-softmmu/hmp-commands-info.h
  GEN     lm32-softmmu/config-target.h
  CC      lm32-softmmu/exec.o
  CC      aarch64-softmmu/hw/misc/exynos4210_rng.o
  CC      arm-softmmu/hw/misc/omap_l4.o
  CC      arm-softmmu/hw/misc/omap_sdrc.o
  CC      aarch64-softmmu/hw/misc/imx_ccm.o
  CC      arm-softmmu/hw/misc/omap_tap.o
  CC      aarch64-softmmu/hw/misc/imx31_ccm.o
  CC      arm-softmmu/hw/misc/bcm2835_mbox.o
  CC      i386-softmmu/memory_mapping.o
  CC      aarch64-softmmu/hw/misc/imx25_ccm.o
  CC      arm-softmmu/hw/misc/bcm2835_property.o
  CC      i386-softmmu/dump.o
  CC      aarch64-softmmu/hw/misc/imx6_ccm.o
  CC      arm-softmmu/hw/misc/bcm2835_rng.o
  CC      lm32-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/hw/misc/imx6_src.o
  CC      arm-softmmu/hw/misc/zynq_slcr.o
  CC      i386-softmmu/migration/ram.o
  CC      aarch64-softmmu/hw/misc/mst_fpga.o
  CC      arm-softmmu/hw/misc/zynq-xadc.o
  CC      aarch64-softmmu/hw/misc/omap_clk.o
  CC      arm-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      i386-softmmu/accel/accel.o
  CC      aarch64-softmmu/hw/misc/omap_gpmc.o
  CC      i386-softmmu/accel/stubs/kvm-stub.o
  CC      arm-softmmu/hw/misc/mps2-scc.o
  CC      lm32-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/hw/misc/omap_l4.o
  CC      arm-softmmu/hw/misc/aspeed_scu.o
  CC      i386-softmmu/accel/tcg/tcg-all.o
  CC      aarch64-softmmu/hw/misc/omap_sdrc.o
  CC      i386-softmmu/accel/tcg/cputlb.o
  CC      arm-softmmu/hw/misc/aspeed_sdmc.o
  CC      aarch64-softmmu/hw/misc/omap_tap.o
  CC      arm-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/misc/bcm2835_mbox.o
  CC      aarch64-softmmu/hw/misc/bcm2835_property.o
  CC      arm-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/net/vhost_net.o
  CC      lm32-softmmu/tcg/optimize.o
  CC      aarch64-softmmu/hw/misc/bcm2835_rng.o
  CC      arm-softmmu/hw/pcmcia/pxa2xx.o
  CC      aarch64-softmmu/hw/misc/zynq_slcr.o
  CC      i386-softmmu/accel/tcg/cpu-exec.o
  CC      arm-softmmu/hw/scsi/virtio-scsi.o
  CC      aarch64-softmmu/hw/misc/zynq-xadc.o
  CC      i386-softmmu/accel/tcg/cpu-exec-common.o
  CC      aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      lm32-softmmu/tcg/tcg-common.o
  CC      arm-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-softmmu/hw/misc/mps2-scc.o
  CC      i386-softmmu/accel/tcg/translate-all.o
  CC      lm32-softmmu/tcg/tcg-runtime.o
  CC      aarch64-softmmu/hw/misc/auxbus.o
  CC      arm-softmmu/hw/scsi/vhost-scsi-common.o
  CC      lm32-softmmu/fpu/softfloat.o
  CC      i386-softmmu/hw/9pfs/virtio-9p-device.o
  CC      aarch64-softmmu/hw/misc/aspeed_scu.o
  CC      arm-softmmu/hw/scsi/vhost-scsi.o
  CC      i386-softmmu/hw/block/virtio-blk.o
  CC      aarch64-softmmu/hw/misc/aspeed_sdmc.o
  CC      arm-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/misc/mmio_interface.o
  CC      i386-softmmu/hw/block/dataplane/virtio-blk.o
  CC      aarch64-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/sd/omap_mmc.o
  CC      i386-softmmu/hw/char/virtio-serial-bus.o
  CC      arm-softmmu/hw/sd/pxa2xx_mmci.o
  CC      i386-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/sd/bcm2835_sdhost.o
  CC      aarch64-softmmu/hw/net/vhost_net.o
  CC      i386-softmmu/hw/core/null-machine.o
  CC      arm-softmmu/hw/ssi/omap_spi.o
  CC      aarch64-softmmu/hw/pcmcia/pxa2xx.o
  CC      i386-softmmu/hw/display/vga.o
  CC      lm32-softmmu/disas.o
  CC      arm-softmmu/hw/ssi/imx_spi.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi.o
  CC      lm32-softmmu/hax-stub.o
  CC      arm-softmmu/hw/timer/exynos4210_mct.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      lm32-softmmu/arch_init.o
  CC      arm-softmmu/hw/timer/exynos4210_pwm.o
  CC      lm32-softmmu/cpus.o
  CC      i386-softmmu/hw/display/virtio-gpu.o
  CC      arm-softmmu/hw/timer/exynos4210_rtc.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/scsi/vhost-scsi.o
  CC      arm-softmmu/hw/timer/omap_gptimer.o
  CC      lm32-softmmu/monitor.o
  CC      i386-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      arm-softmmu/hw/timer/omap_synctimer.o
  CC      aarch64-softmmu/hw/sd/omap_mmc.o
  CC      arm-softmmu/hw/timer/pxa2xx_timer.o
  CC      aarch64-softmmu/hw/sd/pxa2xx_mmci.o
  CC      i386-softmmu/hw/display/virtio-gpu-pci.o
  CC      arm-softmmu/hw/timer/digic-timer.o
  CC      aarch64-softmmu/hw/sd/bcm2835_sdhost.o
  CC      lm32-softmmu/gdbstub.o
  CC      i386-softmmu/hw/display/virtio-vga.o
  CC      arm-softmmu/hw/timer/allwinner-a10-pit.o
  CC      aarch64-softmmu/hw/ssi/omap_spi.o
  CC      arm-softmmu/hw/usb/tusb6010.o
  CC      aarch64-softmmu/hw/ssi/imx_spi.o
  CC      i386-softmmu/hw/intc/apic.o
  CC      lm32-softmmu/balloon.o
  CC      lm32-softmmu/ioport.o
  CC      aarch64-softmmu/hw/timer/exynos4210_mct.o
  CC      arm-softmmu/hw/vfio/common.o
  CC      lm32-softmmu/numa.o
  CC      i386-softmmu/hw/intc/apic_common.o
  CC      aarch64-softmmu/hw/timer/exynos4210_pwm.o
  CC      arm-softmmu/hw/vfio/pci.o
  CC      lm32-softmmu/qtest.o
  CC      i386-softmmu/hw/intc/ioapic.o
  CC      aarch64-softmmu/hw/timer/exynos4210_rtc.o
  CC      aarch64-softmmu/hw/timer/omap_gptimer.o
  CC      lm32-softmmu/memory.o
  CC      i386-softmmu/hw/isa/lpc_ich9.o
  CC      aarch64-softmmu/hw/timer/omap_synctimer.o
  CC      arm-softmmu/hw/vfio/pci-quirks.o
  CC      i386-softmmu/hw/misc/vmport.o
  CC      aarch64-softmmu/hw/timer/pxa2xx_timer.o
  CC      i386-softmmu/hw/misc/ivshmem.o
  CC      arm-softmmu/hw/vfio/platform.o
  CC      lm32-softmmu/memory_mapping.o
  CC      aarch64-softmmu/hw/timer/digic-timer.o
  CC      i386-softmmu/hw/misc/pvpanic.o
  CC      aarch64-softmmu/hw/timer/allwinner-a10-pit.o
  CC      arm-softmmu/hw/vfio/calxeda-xgmac.o
  CC      lm32-softmmu/dump.o
  CC      i386-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/usb/tusb6010.o
  CC      arm-softmmu/hw/vfio/amd-xgbe.o
  CC      i386-softmmu/hw/net/virtio-net.o
  CC      arm-softmmu/hw/vfio/spapr.o
  CC      aarch64-softmmu/hw/vfio/common.o
  CC      lm32-softmmu/migration/ram.o
  CC      arm-softmmu/hw/virtio/virtio.o
  CC      i386-softmmu/hw/net/vhost_net.o
  CC      aarch64-softmmu/hw/vfio/pci.o
  CC      lm32-softmmu/accel/accel.o
  CC      i386-softmmu/hw/scsi/virtio-scsi.o
  CC      lm32-softmmu/accel/stubs/kvm-stub.o
  CC      arm-softmmu/hw/virtio/virtio-balloon.o
  CC      i386-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      i386-softmmu/hw/scsi/vhost-scsi-common.o
  CC      aarch64-softmmu/hw/vfio/pci-quirks.o
  CC      lm32-softmmu/accel/tcg/tcg-all.o
  CC      arm-softmmu/hw/virtio/vhost.o
  CC      i386-softmmu/hw/scsi/vhost-scsi.o
  CC      lm32-softmmu/accel/tcg/cputlb.o
  CC      aarch64-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/virtio/vhost-backend.o
  CC      i386-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-softmmu/hw/vfio/calxeda-xgmac.o
  CC      arm-softmmu/hw/virtio/vhost-user.o
  CC      i386-softmmu/hw/timer/mc146818rtc.o
  CC      aarch64-softmmu/hw/vfio/amd-xgbe.o
  CC      arm-softmmu/hw/virtio/vhost-vsock.o
  CC      lm32-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/hw/vfio/spapr.o
  CC      i386-softmmu/hw/vfio/common.o
  CC      arm-softmmu/hw/virtio/virtio-crypto.o
  CC      aarch64-softmmu/hw/virtio/virtio.o
  CC      lm32-softmmu/accel/tcg/cpu-exec-common.o
  CC      arm-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      i386-softmmu/hw/vfio/pci.o
  CC      lm32-softmmu/accel/tcg/translate-all.o
  CC      arm-softmmu/hw/arm/boot.o
  CC      aarch64-softmmu/hw/virtio/virtio-balloon.o
  CC      lm32-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/hw/arm/collie.o
  CC      i386-softmmu/hw/vfio/pci-quirks.o
  CC      lm32-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/hw/virtio/vhost.o
  CC      arm-softmmu/hw/arm/exynos4_boards.o
  CC      lm32-softmmu/hw/input/milkymist-softusb.o
  CC      arm-softmmu/hw/arm/gumstix.o
  CC      aarch64-softmmu/hw/virtio/vhost-backend.o
  CC      i386-softmmu/hw/vfio/platform.o
  CC      lm32-softmmu/hw/misc/milkymist-hpdmc.o
  CC      arm-softmmu/hw/arm/highbank.o
  CC      lm32-softmmu/hw/misc/milkymist-pfpu.o
  CC      aarch64-softmmu/hw/virtio/vhost-user.o
  CC      i386-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/hw/arm/digic_boards.o
  CC      lm32-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/virtio/vhost-vsock.o
  CC      i386-softmmu/hw/virtio/virtio.o
  CC      arm-softmmu/hw/arm/integratorcp.o
  CC      lm32-softmmu/hw/net/milkymist-minimac2.o
  CC      arm-softmmu/hw/arm/mainstone.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto.o
  CC      lm32-softmmu/hw/net/vhost_net.o
  CC      lm32-softmmu/hw/net/rocker/qmp-norocker.o
  CC      arm-softmmu/hw/arm/musicpal.o
  CC      i386-softmmu/hw/virtio/virtio-balloon.o
  CC      aarch64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      lm32-softmmu/hw/sd/milkymist-memcard.o
  CC      i386-softmmu/hw/virtio/vhost.o
  CC      lm32-softmmu/hw/vfio/common.o
  CC      aarch64-softmmu/hw/arm/boot.o
  CC      arm-softmmu/hw/arm/nseries.o
  CC      i386-softmmu/hw/virtio/vhost-backend.o
  CC      aarch64-softmmu/hw/arm/collie.o
  CC      lm32-softmmu/hw/vfio/platform.o
  CC      arm-softmmu/hw/arm/omap_sx1.o
  CC      i386-softmmu/hw/virtio/vhost-user.o
  CC      aarch64-softmmu/hw/arm/exynos4_boards.o
  CC      lm32-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/hw/arm/palm.o
  CC      aarch64-softmmu/hw/arm/gumstix.o
  CC      i386-softmmu/hw/virtio/vhost-vsock.o
  CC      aarch64-softmmu/hw/arm/highbank.o
  CC      lm32-softmmu/hw/lm32/lm32_boards.o
  CC      arm-softmmu/hw/arm/realview.o
  CC      i386-softmmu/hw/virtio/virtio-crypto.o
  CC      lm32-softmmu/hw/lm32/milkymist.o
  CC      aarch64-softmmu/hw/arm/digic_boards.o
  CC      arm-softmmu/hw/arm/spitz.o
  CC      aarch64-softmmu/hw/arm/integratorcp.o
  CC      i386-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      lm32-softmmu/target/lm32/translate.o
  CC      aarch64-softmmu/hw/arm/mainstone.o
  CC      arm-softmmu/hw/arm/stellaris.o
  CC      aarch64-softmmu/hw/arm/musicpal.o
  CC      i386-softmmu/hw/i386/multiboot.o
  CC      lm32-softmmu/target/lm32/op_helper.o
  CC      lm32-softmmu/target/lm32/helper.o
  CC      i386-softmmu/hw/i386/pc.o
  CC      aarch64-softmmu/hw/arm/nseries.o
  CC      arm-softmmu/hw/arm/tosa.o
  CC      lm32-softmmu/target/lm32/cpu.o
  CC      arm-softmmu/hw/arm/versatilepb.o
  CC      lm32-softmmu/target/lm32/gdbstub.o
  CC      aarch64-softmmu/hw/arm/omap_sx1.o
  CC      arm-softmmu/hw/arm/vexpress.o
  CC      lm32-softmmu/target/lm32/lm32-semi.o
  CC      i386-softmmu/hw/i386/pc_piix.o
  CC      lm32-softmmu/target/lm32/machine.o
  CC      arm-softmmu/hw/arm/virt.o
  CC      aarch64-softmmu/hw/arm/palm.o
  GEN     trace/generated-helpers.c
  CC      lm32-softmmu/trace/control-target.o
  CC      i386-softmmu/hw/i386/pc_q35.o
  CC      aarch64-softmmu/hw/arm/realview.o
  CC      lm32-softmmu/trace/generated-helpers.o
  CC      arm-softmmu/hw/arm/xilinx_zynq.o
  CC      i386-softmmu/hw/i386/pc_sysfw.o
  CC      aarch64-softmmu/hw/arm/spitz.o
  LINK    lm32-softmmu/qemu-system-lm32
  CC      arm-softmmu/hw/arm/z2.o
  CC      i386-softmmu/hw/i386/x86-iommu.o
  CC      i386-softmmu/hw/i386/intel_iommu.o
  CC      arm-softmmu/hw/arm/virt-acpi-build.o
  CC      aarch64-softmmu/hw/arm/stellaris.o
  CC      arm-softmmu/hw/arm/netduino2.o
  CC      arm-softmmu/hw/arm/sysbus-fdt.o
  CC      aarch64-softmmu/hw/arm/tosa.o
  CC      arm-softmmu/hw/arm/armv7m.o
  CC      aarch64-softmmu/hw/arm/versatilepb.o
  CC      i386-softmmu/hw/i386/amd_iommu.o
  GEN     m68k-softmmu/hmp-commands.h
  CC      arm-softmmu/hw/arm/exynos4210.o
  GEN     m68k-softmmu/hmp-commands-info.h
  GEN     m68k-softmmu/config-target.h
  CC      aarch64-softmmu/hw/arm/vexpress.o
  CC      m68k-softmmu/exec.o
  CC      arm-softmmu/hw/arm/pxa2xx.o
  CC      i386-softmmu/hw/i386/kvmvapic.o
  CC      aarch64-softmmu/hw/arm/virt.o
  CC      arm-softmmu/hw/arm/pxa2xx_gpio.o
  CC      aarch64-softmmu/hw/arm/xilinx_zynq.o
  CC      i386-softmmu/hw/i386/acpi-build.o
  CC      arm-softmmu/hw/arm/pxa2xx_pic.o
  CC      aarch64-softmmu/hw/arm/z2.o
  CC      arm-softmmu/hw/arm/digic.o
  CC      i386-softmmu/hw/i386/pci-assign-load-rom.o
  CC      arm-softmmu/hw/arm/omap1.o
  CC      m68k-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/hw/arm/virt-acpi-build.o
  CC      i386-softmmu/target/i386/helper.o
  CC      aarch64-softmmu/hw/arm/netduino2.o
  CC      aarch64-softmmu/hw/arm/sysbus-fdt.o
  CC      i386-softmmu/target/i386/cpu.o
  CC      arm-softmmu/hw/arm/omap2.o
  CC      aarch64-softmmu/hw/arm/armv7m.o
  CC      m68k-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/hw/arm/exynos4210.o
  CC      i386-softmmu/target/i386/gdbstub.o
  CC      arm-softmmu/hw/arm/strongarm.o
  CC      aarch64-softmmu/hw/arm/pxa2xx.o
  CC      i386-softmmu/target/i386/xsave_helper.o
  CC      arm-softmmu/hw/arm/allwinner-a10.o
  CC      i386-softmmu/target/i386/translate.o
  CC      arm-softmmu/hw/arm/cubieboard.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_gpio.o
  CC      arm-softmmu/hw/arm/bcm2835_peripherals.o
  CC      aarch64-softmmu/hw/arm/pxa2xx_pic.o
  CC      m68k-softmmu/tcg/optimize.o
  CC      arm-softmmu/hw/arm/bcm2836.o
  CC      aarch64-softmmu/hw/arm/digic.o
  CC      aarch64-softmmu/hw/arm/omap1.o
  CC      arm-softmmu/hw/arm/raspi.o
  CC      m68k-softmmu/tcg/tcg-common.o
  CC      m68k-softmmu/tcg/tcg-runtime.o
  CC      arm-softmmu/hw/arm/stm32f205_soc.o
  CC      m68k-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/hw/arm/omap2.o
  CC      arm-softmmu/hw/arm/fsl-imx25.o
  CC      arm-softmmu/hw/arm/imx25_pdk.o
  CC      aarch64-softmmu/hw/arm/strongarm.o
  CC      arm-softmmu/hw/arm/fsl-imx31.o
  CC      arm-softmmu/hw/arm/kzm.o
  CC      aarch64-softmmu/hw/arm/allwinner-a10.o
  CC      arm-softmmu/hw/arm/fsl-imx6.o
  CC      arm-softmmu/hw/arm/sabrelite.o
  CC      aarch64-softmmu/hw/arm/cubieboard.o
  CC      arm-softmmu/hw/arm/aspeed_soc.o
  CC      m68k-softmmu/disas.o
  CC      aarch64-softmmu/hw/arm/bcm2835_peripherals.o
  CC      i386-softmmu/target/i386/bpt_helper.o
  CC      arm-softmmu/hw/arm/aspeed.o
  GEN     m68k-softmmu/gdbstub-xml.c
  CC      i386-softmmu/target/i386/cc_helper.o
  CC      m68k-softmmu/hax-stub.o
  CC      arm-softmmu/hw/arm/mps2.o
  CC      aarch64-softmmu/hw/arm/bcm2836.o
  CC      i386-softmmu/target/i386/excp_helper.o
  CC      arm-softmmu/target/arm/arm-semi.o
  CC      m68k-softmmu/arch_init.o
  CC      aarch64-softmmu/hw/arm/raspi.o
  CC      i386-softmmu/target/i386/fpu_helper.o
  CC      m68k-softmmu/cpus.o
  CC      arm-softmmu/target/arm/machine.o
  CC      aarch64-softmmu/hw/arm/stm32f205_soc.o
  CC      arm-softmmu/target/arm/psci.o
  CC      m68k-softmmu/monitor.o
  CC      aarch64-softmmu/hw/arm/xlnx-zynqmp.o
  CC      arm-softmmu/target/arm/arch_dump.o
  CC      arm-softmmu/target/arm/monitor.o
  CC      aarch64-softmmu/hw/arm/xlnx-ep108.o
  CC      arm-softmmu/target/arm/kvm-stub.o
  CC      aarch64-softmmu/hw/arm/fsl-imx25.o
  CC      arm-softmmu/target/arm/translate.o
  CC      m68k-softmmu/gdbstub.o
  CC      aarch64-softmmu/hw/arm/imx25_pdk.o
  CC      i386-softmmu/target/i386/int_helper.o
  CC      aarch64-softmmu/hw/arm/fsl-imx31.o
  CC      i386-softmmu/target/i386/mem_helper.o
  CC      m68k-softmmu/balloon.o
  CC      aarch64-softmmu/hw/arm/kzm.o
  CC      i386-softmmu/target/i386/misc_helper.o
  CC      m68k-softmmu/ioport.o
  CC      aarch64-softmmu/hw/arm/fsl-imx6.o
  CC      m68k-softmmu/numa.o
  CC      i386-softmmu/target/i386/mpx_helper.o
  CC      aarch64-softmmu/hw/arm/sabrelite.o
  CC      m68k-softmmu/qtest.o
  CC      aarch64-softmmu/hw/arm/aspeed_soc.o
  CC      i386-softmmu/target/i386/seg_helper.o
  CC      m68k-softmmu/memory.o
  CC      aarch64-softmmu/hw/arm/aspeed.o
  CC      aarch64-softmmu/hw/arm/mps2.o
  CC      m68k-softmmu/memory_mapping.o
  CC      aarch64-softmmu/target/arm/arm-semi.o
  CC      m68k-softmmu/dump.o
  CC      aarch64-softmmu/target/arm/machine.o
  CC      i386-softmmu/target/i386/smm_helper.o
  CC      aarch64-softmmu/target/arm/psci.o
  CC      m68k-softmmu/migration/ram.o
  CC      i386-softmmu/target/i386/svm_helper.o
  CC      aarch64-softmmu/target/arm/arch_dump.o
  CC      i386-softmmu/target/i386/machine.o
  CC      arm-softmmu/target/arm/op_helper.o
  CC      aarch64-softmmu/target/arm/monitor.o
  CC      i386-softmmu/target/i386/arch_memory_mapping.o
  CC      m68k-softmmu/accel/accel.o
  CC      aarch64-softmmu/target/arm/kvm-stub.o
  CC      i386-softmmu/target/i386/arch_dump.o
  CC      m68k-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/target/arm/translate.o
  CC      arm-softmmu/target/arm/helper.o
  CC      i386-softmmu/target/i386/monitor.o
  CC      m68k-softmmu/accel/tcg/tcg-all.o
  CC      i386-softmmu/target/i386/kvm-stub.o
  CC      m68k-softmmu/accel/tcg/cputlb.o
  GEN     trace/generated-helpers.c
  CC      i386-softmmu/trace/control-target.o
  CC      i386-softmmu/gdbstub-xml.o
  CC      i386-softmmu/trace/generated-helpers.o
  CC      m68k-softmmu/accel/tcg/cpu-exec.o
  LINK    i386-softmmu/qemu-system-i386
  CC      m68k-softmmu/accel/tcg/cpu-exec-common.o
  CC      arm-softmmu/target/arm/cpu.o
  CC      m68k-softmmu/accel/tcg/translate-all.o
  CC      m68k-softmmu/hw/char/mcf_uart.o
  CC      m68k-softmmu/hw/core/generic-loader.o
  CC      arm-softmmu/target/arm/neon_helper.o
  CC      m68k-softmmu/hw/core/null-machine.o
  CC      aarch64-softmmu/target/arm/op_helper.o
  CC      m68k-softmmu/hw/misc/mmio_interface.o
  CC      m68k-softmmu/hw/net/mcf_fec.o
  CC      arm-softmmu/target/arm/iwmmxt_helper.o
  CC      m68k-softmmu/hw/net/vhost_net.o
  CC      m68k-softmmu/hw/net/rocker/qmp-norocker.o
  CC      aarch64-softmmu/target/arm/helper.o
  CC      m68k-softmmu/hw/vfio/common.o
  CC      m68k-softmmu/hw/vfio/platform.o
  CC      m68k-softmmu/hw/vfio/spapr.o
  CC      arm-softmmu/target/arm/gdbstub.o
  CC      aarch64-softmmu/target/arm/cpu.o
  CC      m68k-softmmu/hw/m68k/an5206.o
  CC      arm-softmmu/target/arm/crypto_helper.o
  CC      m68k-softmmu/hw/m68k/mcf5208.o
  CC      arm-softmmu/target/arm/arm-powerctl.o
  GEN     trace/generated-helpers.c
  CC      arm-softmmu/trace/control-target.o
  CC      m68k-softmmu/hw/m68k/mcf5206.o
  CC      arm-softmmu/gdbstub-xml.o
  CC      aarch64-softmmu/target/arm/neon_helper.o
  CC      arm-softmmu/trace/generated-helpers.o
  CC      m68k-softmmu/hw/m68k/mcf_intc.o
  LINK    arm-softmmu/qemu-system-arm
  CC      m68k-softmmu/target/m68k/m68k-semi.o
  CC      aarch64-softmmu/target/arm/iwmmxt_helper.o
  CC      m68k-softmmu/target/m68k/translate.o
  CC      aarch64-softmmu/target/arm/gdbstub.o
  CC      m68k-softmmu/target/m68k/op_helper.o
  CC      aarch64-softmmu/target/arm/cpu64.o
  CC      aarch64-softmmu/target/arm/translate-a64.o
  CC      aarch64-softmmu/target/arm/helper-a64.o
  CC      aarch64-softmmu/target/arm/gdbstub64.o
  CC      m68k-softmmu/target/m68k/helper.o
  CC      m68k-softmmu/target/m68k/cpu.o
  CC      m68k-softmmu/target/m68k/fpu_helper.o
  CC      m68k-softmmu/target/m68k/gdbstub.o
  CC      aarch64-softmmu/target/arm/crypto_helper.o
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/target/arm/arm-powerctl.o
  CC      m68k-softmmu/trace/control-target.o
  CC      m68k-softmmu/gdbstub-xml.o
  CC      m68k-softmmu/trace/generated-helpers.o
  GEN     microblazeel-softmmu/hmp-commands.h
  GEN     microblaze-softmmu/hmp-commands.h
  GEN     microblazeel-softmmu/hmp-commands-info.h
  GEN     microblazeel-softmmu/config-target.h
  GEN     microblaze-softmmu/hmp-commands-info.h
  GEN     microblaze-softmmu/config-target.h
  CC      microblaze-softmmu/exec.o
  CC      microblazeel-softmmu/exec.o
  LINK    m68k-softmmu/qemu-system-m68k
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/trace/control-target.o
  CC      microblazeel-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/gdbstub-xml.o
  CC      aarch64-softmmu/trace/generated-helpers.o
  CC      microblazeel-softmmu/tcg/tcg-op.o
  CC      microblaze-softmmu/tcg/tcg.o
  LINK    aarch64-softmmu/qemu-system-aarch64
  CC      microblazeel-softmmu/tcg/optimize.o
  CC      microblazeel-softmmu/tcg/tcg-common.o
  CC      microblaze-softmmu/tcg/tcg-op.o
  CC      microblazeel-softmmu/tcg/tcg-runtime.o
  GEN     mips64el-softmmu/hmp-commands.h
  GEN     mips64el-softmmu/hmp-commands-info.h
  GEN     mips64el-softmmu/config-target.h
  CC      mips64el-softmmu/exec.o
  CC      microblazeel-softmmu/fpu/softfloat.o
  GEN     mips64-softmmu/hmp-commands.h
  GEN     mips64-softmmu/hmp-commands-info.h
  GEN     mips64-softmmu/config-target.h
  CC      mips64-softmmu/exec.o
  CC      microblaze-softmmu/tcg/optimize.o
  CC      mips64el-softmmu/tcg/tcg.o
  CC      microblazeel-softmmu/disas.o
  CC      microblaze-softmmu/tcg/tcg-common.o
  CC      microblazeel-softmmu/hax-stub.o
  CC      microblaze-softmmu/tcg/tcg-runtime.o
  CC      microblazeel-softmmu/arch_init.o
  CC      microblazeel-softmmu/cpus.o
  CC      microblaze-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/tcg/tcg.o
  CC      microblazeel-softmmu/monitor.o
  CC      mips64el-softmmu/tcg/tcg-op.o
  CC      mips64-softmmu/tcg/tcg-op.o
  CC      microblaze-softmmu/disas.o
  CC      microblazeel-softmmu/gdbstub.o
  CC      microblaze-softmmu/hax-stub.o
  CC      microblaze-softmmu/arch_init.o
  CC      microblaze-softmmu/cpus.o
  CC      microblazeel-softmmu/balloon.o
  CC      mips64el-softmmu/tcg/optimize.o
  CC      microblazeel-softmmu/ioport.o
  CC      microblaze-softmmu/monitor.o
  CC      microblazeel-softmmu/numa.o
  CC      microblazeel-softmmu/qtest.o
  CC      mips64el-softmmu/tcg/tcg-common.o
  CC      mips64-softmmu/tcg/optimize.o
  CC      mips64el-softmmu/tcg/tcg-runtime.o
  CC      microblazeel-softmmu/memory.o
  CC      microblaze-softmmu/gdbstub.o
  CC      mips64el-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/tcg/tcg-common.o
  CC      microblaze-softmmu/balloon.o
  CC      mips64-softmmu/tcg/tcg-runtime.o
  CC      microblazeel-softmmu/memory_mapping.o
  CC      microblaze-softmmu/ioport.o
  CC      microblazeel-softmmu/dump.o
  CC      microblaze-softmmu/numa.o
  CC      mips64-softmmu/fpu/softfloat.o
  CC      microblaze-softmmu/qtest.o
  CC      microblazeel-softmmu/migration/ram.o
  CC      microblaze-softmmu/memory.o
  CC      mips64el-softmmu/disas.o
  CC      microblazeel-softmmu/accel/accel.o
  CC      mips64el-softmmu/hax-stub.o
  CC      microblazeel-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/accel/tcg/tcg-all.o
  CC      mips64el-softmmu/arch_init.o
  CC      microblaze-softmmu/memory_mapping.o
  CC      mips64el-softmmu/cpus.o
  CC      microblazeel-softmmu/accel/tcg/cputlb.o
  CC      mips64-softmmu/disas.o
  CC      microblaze-softmmu/dump.o
  CC      mips64-softmmu/hax-stub.o
  CC      mips64el-softmmu/monitor.o
  CC      mips64-softmmu/arch_init.o
  CC      microblaze-softmmu/migration/ram.o
  CC      mips64-softmmu/cpus.o
  CC      microblazeel-softmmu/accel/tcg/cpu-exec.o
  CC      mips64-softmmu/monitor.o
  CC      microblaze-softmmu/accel/accel.o
  CC      mips64el-softmmu/gdbstub.o
  CC      microblazeel-softmmu/accel/tcg/cpu-exec-common.o
  CC      microblaze-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/accel/tcg/translate-all.o
  CC      microblaze-softmmu/accel/tcg/tcg-all.o
  CC      mips64el-softmmu/balloon.o
  CC      mips64el-softmmu/ioport.o
  CC      microblaze-softmmu/accel/tcg/cputlb.o
  CC      microblazeel-softmmu/hw/core/generic-loader.o
  CC      mips64el-softmmu/numa.o
  CC      microblazeel-softmmu/hw/core/null-machine.o
  CC      mips64-softmmu/gdbstub.o
  CC      mips64el-softmmu/qtest.o
  CC      microblazeel-softmmu/hw/misc/mmio_interface.o
  CC      microblazeel-softmmu/hw/net/xilinx_ethlite.o
  CC      mips64-softmmu/balloon.o
  CC      mips64el-softmmu/memory.o
  CC      microblazeel-softmmu/hw/net/vhost_net.o
  CC      mips64-softmmu/ioport.o
  CC      microblaze-softmmu/accel/tcg/cpu-exec.o
  CC      microblazeel-softmmu/hw/net/rocker/qmp-norocker.o
  CC      mips64-softmmu/numa.o
  CC      microblaze-softmmu/accel/tcg/cpu-exec-common.o
  CC      microblazeel-softmmu/hw/vfio/common.o
  CC      microblaze-softmmu/accel/tcg/translate-all.o
  CC      mips64-softmmu/qtest.o
  CC      mips64el-softmmu/memory_mapping.o
  CC      microblazeel-softmmu/hw/vfio/platform.o
  CC      mips64el-softmmu/dump.o
  CC      mips64-softmmu/memory.o
  CC      microblaze-softmmu/hw/core/generic-loader.o
  CC      microblaze-softmmu/hw/core/null-machine.o
  CC      microblazeel-softmmu/hw/vfio/spapr.o
  CC      mips64el-softmmu/migration/ram.o
  CC      microblaze-softmmu/hw/misc/mmio_interface.o
  CC      microblazeel-softmmu/hw/microblaze/petalogix_s3adsp1800_mmu.o
  CC      microblaze-softmmu/hw/net/xilinx_ethlite.o
  CC      microblazeel-softmmu/hw/microblaze/petalogix_ml605_mmu.o
  CC      mips64-softmmu/memory_mapping.o
  CC      microblaze-softmmu/hw/net/vhost_net.o
  CC      mips64el-softmmu/accel/accel.o
  CC      mips64-softmmu/dump.o
  CC      microblazeel-softmmu/hw/microblaze/boot.o
  CC      microblaze-softmmu/hw/net/rocker/qmp-norocker.o
  CC      microblaze-softmmu/hw/vfio/common.o
  CC      mips64el-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/target/microblaze/translate.o
  CC      mips64el-softmmu/accel/tcg/tcg-all.o
  CC      mips64-softmmu/migration/ram.o
  CC      mips64el-softmmu/accel/tcg/cputlb.o
  CC      microblaze-softmmu/hw/vfio/platform.o
  CC      microblazeel-softmmu/target/microblaze/op_helper.o
  CC      microblaze-softmmu/hw/vfio/spapr.o
  CC      mips64-softmmu/accel/accel.o
  CC      mips64-softmmu/accel/stubs/kvm-stub.o
  CC      microblazeel-softmmu/target/microblaze/helper.o
  CC      microblaze-softmmu/hw/microblaze/petalogix_s3adsp1800_mmu.o
  CC      mips64el-softmmu/accel/tcg/cpu-exec.o
  CC      mips64-softmmu/accel/tcg/tcg-all.o
  CC      microblaze-softmmu/hw/microblaze/petalogix_ml605_mmu.o
  CC      microblazeel-softmmu/target/microblaze/cpu.o
  CC      mips64-softmmu/accel/tcg/cputlb.o
  CC      mips64el-softmmu/accel/tcg/cpu-exec-common.o
  CC      microblaze-softmmu/hw/microblaze/boot.o
  CC      microblazeel-softmmu/target/microblaze/gdbstub.o
  CC      mips64el-softmmu/accel/tcg/translate-all.o
  CC      microblaze-softmmu/target/microblaze/translate.o
  CC      microblazeel-softmmu/target/microblaze/mmu.o
  GEN     trace/generated-helpers.c
  CC      microblazeel-softmmu/trace/control-target.o
  CC      mips64el-softmmu/hw/9pfs/virtio-9p-device.o
  CC      microblazeel-softmmu/trace/generated-helpers.o
  CC      mips64el-softmmu/hw/block/virtio-blk.o
  CC      mips64-softmmu/accel/tcg/cpu-exec.o
  LINK    microblazeel-softmmu/qemu-system-microblazeel
  CC      microblaze-softmmu/target/microblaze/op_helper.o
  CC      mips64-softmmu/accel/tcg/cpu-exec-common.o
  CC      mips64el-softmmu/hw/block/dataplane/virtio-blk.o
  CC      microblaze-softmmu/target/microblaze/helper.o
  CC      microblaze-softmmu/target/microblaze/cpu.o
  CC      mips64-softmmu/accel/tcg/translate-all.o
  CC      mips64el-softmmu/hw/char/virtio-serial-bus.o
  CC      microblaze-softmmu/target/microblaze/gdbstub.o
  CC      microblaze-softmmu/target/microblaze/mmu.o
  CC      mips64el-softmmu/hw/core/generic-loader.o
  CC      mips64-softmmu/hw/9pfs/virtio-9p-device.o
  GEN     trace/generated-helpers.c
  CC      microblaze-softmmu/trace/control-target.o
  CC      mips64-softmmu/hw/block/virtio-blk.o
  CC      microblaze-softmmu/trace/generated-helpers.o
  CC      mips64el-softmmu/hw/core/null-machine.o
  CC      mips64el-softmmu/hw/display/vga.o
  LINK    microblaze-softmmu/qemu-system-microblaze
  CC      mips64el-softmmu/hw/display/virtio-gpu.o
  CC      mips64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mips64-softmmu/hw/char/virtio-serial-bus.o
  CC      mips64el-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64-softmmu/hw/core/generic-loader.o
  GEN     mipsel-softmmu/hmp-commands.h
  CC      mips64-softmmu/hw/core/null-machine.o
  GEN     mipsel-softmmu/hmp-commands-info.h
  GEN     mipsel-softmmu/config-target.h
  CC      mips64el-softmmu/hw/intc/mips_gic.o
  CC      mipsel-softmmu/exec.o
  CC      mips64-softmmu/hw/display/vga.o
  CC      mips64el-softmmu/hw/misc/ivshmem.o
  CC      mips64el-softmmu/hw/misc/mips_cmgcr.o
  CC      mips64el-softmmu/hw/misc/mips_cpc.o
  CC      mips64-softmmu/hw/display/virtio-gpu.o
  GEN     mips-softmmu/hmp-commands.h
  GEN     mips-softmmu/hmp-commands-info.h
  GEN     mips-softmmu/config-target.h
  CC      mips64el-softmmu/hw/misc/mips_itu.o
  CC      mips-softmmu/exec.o
  CC      mips64el-softmmu/hw/misc/mmio_interface.o
  CC      mips64-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/hw/net/virtio-net.o
  CC      mipsel-softmmu/tcg/tcg.o
  CC      mips64el-softmmu/hw/net/vhost_net.o
  CC      mips64-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64el-softmmu/hw/scsi/virtio-scsi.o
  CC      mips64-softmmu/hw/intc/mips_gic.o
  CC      mips-softmmu/tcg/tcg.o
  CC      mips64el-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips64-softmmu/hw/misc/ivshmem.o
  CC      mips64el-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mips64-softmmu/hw/misc/mips_cmgcr.o
  CC      mipsel-softmmu/tcg/tcg-op.o
  CC      mips64el-softmmu/hw/scsi/vhost-scsi.o
  CC      mips64-softmmu/hw/misc/mips_cpc.o
  CC      mips64el-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mips64-softmmu/hw/misc/mips_itu.o
  CC      mips64el-softmmu/hw/timer/mips_gictimer.o
  CC      mips-softmmu/tcg/tcg-op.o
  CC      mips64-softmmu/hw/misc/mmio_interface.o
  CC      mips64el-softmmu/hw/timer/mc146818rtc.o
  CC      mips64-softmmu/hw/net/virtio-net.o
  CC      mips64el-softmmu/hw/vfio/common.o
  CC      mips64-softmmu/hw/net/vhost_net.o
  CC      mips64-softmmu/hw/scsi/virtio-scsi.o
  CC      mipsel-softmmu/tcg/optimize.o
  CC      mips64el-softmmu/hw/vfio/pci.o
  CC      mips64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips-softmmu/tcg/optimize.o
  CC      mips64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mipsel-softmmu/tcg/tcg-common.o
  CC      mips64el-softmmu/hw/vfio/pci-quirks.o
  CC      mips64-softmmu/hw/scsi/vhost-scsi.o
  CC      mipsel-softmmu/tcg/tcg-runtime.o
  CC      mips64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mips-softmmu/tcg/tcg-common.o
  CC      mips64el-softmmu/hw/vfio/platform.o
  CC      mipsel-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/hw/timer/mips_gictimer.o
  CC      mips64-softmmu/hw/timer/mc146818rtc.o
  CC      mips-softmmu/tcg/tcg-runtime.o
  CC      mips64el-softmmu/hw/vfio/spapr.o
  CC      mips64el-softmmu/hw/virtio/virtio.o
  CC      mips-softmmu/fpu/softfloat.o
  CC      mips64-softmmu/hw/vfio/common.o
  CC      mips64-softmmu/hw/vfio/pci.o
  CC      mips64el-softmmu/hw/virtio/virtio-balloon.o
  CC      mips64el-softmmu/hw/virtio/vhost.o
  CC      mips64-softmmu/hw/vfio/pci-quirks.o
  CC      mipsel-softmmu/disas.o
  CC      mips64el-softmmu/hw/virtio/vhost-backend.o
  CC      mipsel-softmmu/hax-stub.o
  CC      mips64-softmmu/hw/vfio/platform.o
  CC      mips-softmmu/disas.o
  CC      mipsel-softmmu/arch_init.o
  CC      mips64el-softmmu/hw/virtio/vhost-user.o
  CC      mips64-softmmu/hw/vfio/spapr.o
  CC      mips-softmmu/hax-stub.o
  CC      mipsel-softmmu/cpus.o
  CC      mips64el-softmmu/hw/virtio/vhost-vsock.o
  CC      mips-softmmu/arch_init.o
  CC      mips64-softmmu/hw/virtio/virtio.o
  CC      mips-softmmu/cpus.o
  CC      mips64el-softmmu/hw/virtio/virtio-crypto.o
  CC      mipsel-softmmu/monitor.o
  CC      mips64el-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      mips-softmmu/monitor.o
  CC      mips64-softmmu/hw/virtio/virtio-balloon.o
  CC      mips64el-softmmu/hw/mips/mips_r4k.o
  CC      mips64el-softmmu/hw/mips/mips_malta.o
  CC      mipsel-softmmu/gdbstub.o
  CC      mips64-softmmu/hw/virtio/vhost.o
  CC      mips64el-softmmu/hw/mips/mips_mipssim.o
  CC      mipsel-softmmu/balloon.o
  CC      mips-softmmu/gdbstub.o
  CC      mips64-softmmu/hw/virtio/vhost-backend.o
  CC      mips64el-softmmu/hw/mips/addr.o
  CC      mipsel-softmmu/ioport.o
  CC      mips64-softmmu/hw/virtio/vhost-user.o
  CC      mips64el-softmmu/hw/mips/cputimer.o
  CC      mips-softmmu/balloon.o
  CC      mipsel-softmmu/numa.o
  CC      mips-softmmu/ioport.o
  CC      mips64-softmmu/hw/virtio/vhost-vsock.o
  CC      mips64el-softmmu/hw/mips/mips_int.o
  CC      mips-softmmu/numa.o
  CC      mipsel-softmmu/qtest.o
  CC      mips64-softmmu/hw/virtio/virtio-crypto.o
  CC      mips64el-softmmu/hw/mips/mips_jazz.o
  CC      mips64el-softmmu/hw/mips/mips_fulong2e.o
  CC      mips-softmmu/qtest.o
  CC      mips64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      mipsel-softmmu/memory.o
  CC      mips64el-softmmu/hw/mips/gt64xxx_pci.o
  CC      mips-softmmu/memory.o
  CC      mips64-softmmu/hw/mips/mips_r4k.o
  CC      mips64el-softmmu/hw/mips/cps.o
  CC      mips64-softmmu/hw/mips/mips_malta.o
  CC      mips64-softmmu/hw/mips/mips_mipssim.o
  CC      mips64el-softmmu/hw/mips/boston.o
  CC      mipsel-softmmu/memory_mapping.o
  CC      mips-softmmu/memory_mapping.o
  CC      mips64el-softmmu/target/mips/translate.o
  CC      mips64-softmmu/hw/mips/addr.o
  CC      mipsel-softmmu/dump.o
  CC      mips64-softmmu/hw/mips/cputimer.o
  CC      mips-softmmu/dump.o
  CC      mips64-softmmu/hw/mips/mips_int.o
  CC      mipsel-softmmu/migration/ram.o
  CC      mips-softmmu/migration/ram.o
  CC      mips64-softmmu/hw/mips/mips_jazz.o
  CC      mips64-softmmu/hw/mips/gt64xxx_pci.o
  CC      mipsel-softmmu/accel/accel.o
  CC      mips64-softmmu/hw/mips/cps.o
  CC      mips-softmmu/accel/accel.o
  CC      mipsel-softmmu/accel/stubs/kvm-stub.o
  CC      mips64-softmmu/target/mips/translate.o
  CC      mips-softmmu/accel/stubs/kvm-stub.o
  CC      mipsel-softmmu/accel/tcg/tcg-all.o
  CC      mipsel-softmmu/accel/tcg/cputlb.o
  CC      mips-softmmu/accel/tcg/tcg-all.o
  CC      mips-softmmu/accel/tcg/cputlb.o
  CC      mips-softmmu/accel/tcg/cpu-exec.o
  CC      mipsel-softmmu/accel/tcg/cpu-exec.o
  CC      mipsel-softmmu/accel/tcg/cpu-exec-common.o
  CC      mips-softmmu/accel/tcg/cpu-exec-common.o
  CC      mipsel-softmmu/accel/tcg/translate-all.o
  CC      mips-softmmu/accel/tcg/translate-all.o
  CC      mipsel-softmmu/hw/9pfs/virtio-9p-device.o
  CC      mipsel-softmmu/hw/block/virtio-blk.o
  CC      mips-softmmu/hw/9pfs/virtio-9p-device.o
  CC      mips-softmmu/hw/block/virtio-blk.o
  CC      mipsel-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mips64el-softmmu/target/mips/dsp_helper.o
  CC      mipsel-softmmu/hw/char/virtio-serial-bus.o
  CC      mips-softmmu/hw/block/dataplane/virtio-blk.o
  CC      mipsel-softmmu/hw/core/generic-loader.o
  CC      mipsel-softmmu/hw/core/null-machine.o
  CC      mips-softmmu/hw/char/virtio-serial-bus.o
  CC      mipsel-softmmu/hw/display/vga.o
  CC      mips-softmmu/hw/core/generic-loader.o
  CC      mips64el-softmmu/target/mips/op_helper.o
  CC      mips64-softmmu/target/mips/dsp_helper.o
  CC      mips-softmmu/hw/core/null-machine.o
  CC      mips-softmmu/hw/display/vga.o
  CC      mipsel-softmmu/hw/display/virtio-gpu.o
  CC      mips-softmmu/hw/display/virtio-gpu.o
  CC      mips64-softmmu/target/mips/op_helper.o
  CC      mipsel-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/target/mips/lmi_helper.o
  CC      mips-softmmu/hw/display/virtio-gpu-3d.o
  CC      mips64el-softmmu/target/mips/helper.o
  CC      mipsel-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64el-softmmu/target/mips/cpu.o
  CC      mipsel-softmmu/hw/intc/mips_gic.o
  CC      mips-softmmu/hw/display/virtio-gpu-pci.o
  CC      mips64el-softmmu/target/mips/gdbstub.o
  CC      mipsel-softmmu/hw/misc/ivshmem.o
  CC      mips64el-softmmu/target/mips/msa_helper.o
  CC      mips-softmmu/hw/intc/mips_gic.o
  CC      mips-softmmu/hw/misc/ivshmem.o
  CC      mipsel-softmmu/hw/misc/mips_cmgcr.o
  CC      mips64-softmmu/target/mips/lmi_helper.o
  CC      mipsel-softmmu/hw/misc/mips_cpc.o
  CC      mips-softmmu/hw/misc/mips_cmgcr.o
  CC      mipsel-softmmu/hw/misc/mips_itu.o
  CC      mips64-softmmu/target/mips/helper.o
  CC      mips-softmmu/hw/misc/mips_cpc.o
  CC      mipsel-softmmu/hw/misc/mmio_interface.o
  CC      mipsel-softmmu/hw/net/virtio-net.o
  CC      mips64-softmmu/target/mips/cpu.o
  CC      mips-softmmu/hw/misc/mips_itu.o
  CC      mips64-softmmu/target/mips/gdbstub.o
  CC      mips-softmmu/hw/misc/mmio_interface.o
  CC      mips64-softmmu/target/mips/msa_helper.o
  CC      mipsel-softmmu/hw/net/vhost_net.o
  CC      mips-softmmu/hw/net/virtio-net.o
  CC      mipsel-softmmu/hw/scsi/virtio-scsi.o
  CC      mips-softmmu/hw/net/vhost_net.o
  CC      mips-softmmu/hw/scsi/virtio-scsi.o
  CC      mipsel-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips64el-softmmu/target/mips/mips-semi.o
  CC      mipsel-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mips-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      mips64el-softmmu/target/mips/machine.o
  CC      mipsel-softmmu/hw/scsi/vhost-scsi.o
  GEN     trace/generated-helpers.c
  CC      mips-softmmu/hw/scsi/vhost-scsi-common.o
  CC      mips64el-softmmu/trace/control-target.o
  CC      mipsel-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mips-softmmu/hw/scsi/vhost-scsi.o
  CC      mips64el-softmmu/trace/generated-helpers.o
  CC      mips-softmmu/hw/scsi/vhost-user-scsi.o
  CC      mipsel-softmmu/hw/timer/mips_gictimer.o
  CC      mips-softmmu/hw/timer/mips_gictimer.o
  LINK    mips64el-softmmu/qemu-system-mips64el
  CC      mipsel-softmmu/hw/timer/mc146818rtc.o
  CC      mips-softmmu/hw/timer/mc146818rtc.o
  CC      mips64-softmmu/target/mips/mips-semi.o
  CC      mips-softmmu/hw/vfio/common.o
  CC      mipsel-softmmu/hw/vfio/common.o
  CC      mips64-softmmu/target/mips/machine.o
  CC      mips-softmmu/hw/vfio/pci.o
  CC      mipsel-softmmu/hw/vfio/pci.o
  GEN     trace/generated-helpers.c
  CC      mips64-softmmu/trace/control-target.o
  CC      mips64-softmmu/trace/generated-helpers.o
  CC      mips-softmmu/hw/vfio/pci-quirks.o
  GEN     moxie-softmmu/hmp-commands.h
  GEN     moxie-softmmu/hmp-commands-info.h
  LINK    mips64-softmmu/qemu-system-mips64
  GEN     moxie-softmmu/config-target.h
  CC      moxie-softmmu/exec.o
  CC      mips-softmmu/hw/vfio/platform.o
  CC      mipsel-softmmu/hw/vfio/pci-quirks.o
  CC      mips-softmmu/hw/vfio/spapr.o
  CC      mipsel-softmmu/hw/vfio/platform.o
  CC      mipsel-softmmu/hw/vfio/spapr.o
  CC      mips-softmmu/hw/virtio/virtio.o
  CC      mipsel-softmmu/hw/virtio/virtio.o
  CC      moxie-softmmu/tcg/tcg.o
  CC      mipsel-softmmu/hw/virtio/virtio-balloon.o
  CC      mips-softmmu/hw/virtio/virtio-balloon.o
  CC      moxie-softmmu/tcg/tcg-op.o
  CC      mips-softmmu/hw/virtio/vhost.o
  CC      mipsel-softmmu/hw/virtio/vhost.o
  CC      mipsel-softmmu/hw/virtio/vhost-backend.o
  CC      mipsel-softmmu/hw/virtio/vhost-user.o
  CC      mips-softmmu/hw/virtio/vhost-backend.o
  CC      mipsel-softmmu/hw/virtio/vhost-vsock.o
  CC      mips-softmmu/hw/virtio/vhost-user.o
  CC      mipsel-softmmu/hw/virtio/virtio-crypto.o
  CC      mipsel-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      mips-softmmu/hw/virtio/vhost-vsock.o
  CC      mipsel-softmmu/hw/mips/mips_r4k.o
  GEN     nios2-softmmu/hmp-commands.h
  GEN     nios2-softmmu/hmp-commands-info.h
  GEN     nios2-softmmu/config-target.h
  CC      mips-softmmu/hw/virtio/virtio-crypto.o
  CC      nios2-softmmu/exec.o
  CC      mipsel-softmmu/hw/mips/mips_malta.o
  CC      mips-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      moxie-softmmu/tcg/optimize.o
  CC      mips-softmmu/hw/mips/mips_r4k.o
  CC      mipsel-softmmu/hw/mips/mips_mipssim.o
  CC      mipsel-softmmu/hw/mips/addr.o
  CC      mips-softmmu/hw/mips/mips_malta.o
  CC      mipsel-softmmu/hw/mips/cputimer.o
  CC      moxie-softmmu/tcg/tcg-common.o
  CC      nios2-softmmu/tcg/tcg.o
  CC      moxie-softmmu/tcg/tcg-runtime.o
  CC      mipsel-softmmu/hw/mips/mips_int.o
  CC      mips-softmmu/hw/mips/mips_mipssim.o
  CC      mipsel-softmmu/hw/mips/gt64xxx_pci.o
  CC      moxie-softmmu/fpu/softfloat.o
  CC      mips-softmmu/hw/mips/addr.o
  CC      mips-softmmu/hw/mips/cputimer.o
  CC      mips-softmmu/hw/mips/mips_int.o
  CC      mipsel-softmmu/hw/mips/cps.o
  CC      mips-softmmu/hw/mips/gt64xxx_pci.o
  CC      mipsel-softmmu/target/mips/translate.o
  CC      nios2-softmmu/tcg/tcg-op.o
  CC      mips-softmmu/hw/mips/cps.o
  CC      mips-softmmu/target/mips/translate.o
  CC      moxie-softmmu/disas.o
  CC      moxie-softmmu/hax-stub.o
  CC      moxie-softmmu/arch_init.o
  CC      nios2-softmmu/tcg/optimize.o
  CC      moxie-softmmu/cpus.o
  CC      nios2-softmmu/tcg/tcg-common.o
  CC      moxie-softmmu/monitor.o
  CC      nios2-softmmu/tcg/tcg-runtime.o
  CC      nios2-softmmu/fpu/softfloat.o
  CC      moxie-softmmu/gdbstub.o
  CC      mipsel-softmmu/target/mips/dsp_helper.o
  CC      mipsel-softmmu/target/mips/op_helper.o
  CC      moxie-softmmu/balloon.o
  CC      moxie-softmmu/ioport.o
  CC      mips-softmmu/target/mips/dsp_helper.o
  CC      moxie-softmmu/numa.o
  CC      nios2-softmmu/disas.o
  CC      moxie-softmmu/qtest.o
  CC      nios2-softmmu/hax-stub.o
  CC      mips-softmmu/target/mips/op_helper.o
  CC      nios2-softmmu/arch_init.o
  CC      moxie-softmmu/memory.o
  CC      nios2-softmmu/cpus.o
  CC      mipsel-softmmu/target/mips/lmi_helper.o
  CC      nios2-softmmu/monitor.o
  CC      moxie-softmmu/memory_mapping.o
  CC      mipsel-softmmu/target/mips/helper.o
  CC      moxie-softmmu/dump.o
  CC      mipsel-softmmu/target/mips/cpu.o
  CC      mipsel-softmmu/target/mips/gdbstub.o
  CC      mips-softmmu/target/mips/lmi_helper.o
  CC      moxie-softmmu/migration/ram.o
  CC      nios2-softmmu/gdbstub.o
  CC      mipsel-softmmu/target/mips/msa_helper.o
  CC      mips-softmmu/target/mips/helper.o
  CC      moxie-softmmu/accel/accel.o
  CC      nios2-softmmu/balloon.o
  CC      mips-softmmu/target/mips/cpu.o
  CC      nios2-softmmu/ioport.o
  CC      mips-softmmu/target/mips/gdbstub.o
  CC      moxie-softmmu/accel/stubs/kvm-stub.o
  CC      nios2-softmmu/numa.o
  CC      mips-softmmu/target/mips/msa_helper.o
  CC      moxie-softmmu/accel/tcg/tcg-all.o
  CC      nios2-softmmu/qtest.o
  CC      moxie-softmmu/accel/tcg/cputlb.o
  CC      nios2-softmmu/memory.o
  CC      moxie-softmmu/accel/tcg/cpu-exec.o
  CC      moxie-softmmu/accel/tcg/cpu-exec-common.o
  CC      mipsel-softmmu/target/mips/mips-semi.o
  CC      moxie-softmmu/accel/tcg/translate-all.o
  CC      nios2-softmmu/memory_mapping.o
  CC      mipsel-softmmu/target/mips/machine.o
  CC      moxie-softmmu/hw/core/generic-loader.o
  CC      nios2-softmmu/dump.o
  CC      mips-softmmu/target/mips/mips-semi.o
  GEN     trace/generated-helpers.c
  CC      mipsel-softmmu/trace/control-target.o
  CC      moxie-softmmu/hw/core/null-machine.o
  CC      nios2-softmmu/migration/ram.o
  CC      moxie-softmmu/hw/display/vga.o
  CC      mips-softmmu/target/mips/machine.o
  CC      mipsel-softmmu/trace/generated-helpers.o
  GEN     trace/generated-helpers.c
  CC      mips-softmmu/trace/control-target.o
  LINK    mipsel-softmmu/qemu-system-mipsel
  CC      mips-softmmu/trace/generated-helpers.o
  CC      nios2-softmmu/accel/accel.o
  CC      nios2-softmmu/accel/stubs/kvm-stub.o
  CC      moxie-softmmu/hw/misc/mmio_interface.o
  LINK    mips-softmmu/qemu-system-mips
  CC      nios2-softmmu/accel/tcg/tcg-all.o
  CC      moxie-softmmu/hw/net/vhost_net.o
  CC      nios2-softmmu/accel/tcg/cputlb.o
  CC      moxie-softmmu/hw/net/rocker/qmp-norocker.o
  CC      moxie-softmmu/hw/timer/mc146818rtc.o
  CC      moxie-softmmu/hw/vfio/common.o
  CC      moxie-softmmu/hw/vfio/platform.o
  CC      moxie-softmmu/hw/vfio/spapr.o
  CC      moxie-softmmu/hw/moxie/moxiesim.o
  CC      moxie-softmmu/target/moxie/translate.o
  CC      nios2-softmmu/accel/tcg/cpu-exec.o
  GEN     or1k-softmmu/hmp-commands.h
  GEN     or1k-softmmu/hmp-commands-info.h
  GEN     or1k-softmmu/config-target.h
  CC      nios2-softmmu/accel/tcg/cpu-exec-common.o
  CC      or1k-softmmu/exec.o
  CC      moxie-softmmu/target/moxie/helper.o
  GEN     ppc64-softmmu/hmp-commands.h
  CC      nios2-softmmu/accel/tcg/translate-all.o
  GEN     ppc64-softmmu/hmp-commands-info.h
  CC      moxie-softmmu/target/moxie/machine.o
  GEN     ppc64-softmmu/config-target.h
  CC      ppc64-softmmu/exec.o
  CC      moxie-softmmu/target/moxie/cpu.o
  CC      nios2-softmmu/hw/core/generic-loader.o
  CC      moxie-softmmu/target/moxie/mmu.o
  CC      nios2-softmmu/hw/core/null-machine.o
  GEN     trace/generated-helpers.c
  CC      moxie-softmmu/trace/control-target.o
  CC      nios2-softmmu/hw/intc/nios2_iic.o
  CC      nios2-softmmu/hw/misc/mmio_interface.o
  CC      moxie-softmmu/trace/generated-helpers.o
  CC      or1k-softmmu/tcg/tcg.o
  CC      nios2-softmmu/hw/net/vhost_net.o
  LINK    moxie-softmmu/qemu-system-moxie
  CC      nios2-softmmu/hw/net/rocker/qmp-norocker.o
  CC      nios2-softmmu/hw/timer/altera_timer.o
  CC      nios2-softmmu/hw/vfio/common.o
  CC      ppc64-softmmu/tcg/tcg.o
  CC      or1k-softmmu/tcg/tcg-op.o
  CC      nios2-softmmu/hw/vfio/platform.o
  CC      nios2-softmmu/hw/vfio/spapr.o
  CC      nios2-softmmu/hw/nios2/boot.o
  GEN     ppcemb-softmmu/hmp-commands.h
  CC      nios2-softmmu/hw/nios2/cpu_pic.o
  GEN     ppcemb-softmmu/hmp-commands-info.h
  GEN     ppcemb-softmmu/config-target.h
  CC      nios2-softmmu/hw/nios2/10m50_devboard.o
  CC      ppcemb-softmmu/exec.o
  CC      nios2-softmmu/target/nios2/translate.o
  CC      ppc64-softmmu/tcg/tcg-op.o
  CC      or1k-softmmu/tcg/optimize.o
  CC      nios2-softmmu/target/nios2/op_helper.o
  CC      nios2-softmmu/target/nios2/helper.o
  CC      or1k-softmmu/tcg/tcg-common.o
  CC      or1k-softmmu/tcg/tcg-runtime.o
  CC      nios2-softmmu/target/nios2/cpu.o
  CC      ppcemb-softmmu/tcg/tcg.o
  CC      nios2-softmmu/target/nios2/mmu.o
  CC      or1k-softmmu/fpu/softfloat.o
  CC      nios2-softmmu/target/nios2/monitor.o
  GEN     trace/generated-helpers.c
  CC      ppc64-softmmu/tcg/optimize.o
  CC      nios2-softmmu/trace/control-target.o
  CC      nios2-softmmu/trace/generated-helpers.o
  CC      ppcemb-softmmu/tcg/tcg-op.o
  CC      ppc64-softmmu/tcg/tcg-common.o
  LINK    nios2-softmmu/qemu-system-nios2
  CC      ppc64-softmmu/tcg/tcg-runtime.o
  CC      ppc64-softmmu/fpu/softfloat.o
  CC      or1k-softmmu/disas.o
  CC      ppcemb-softmmu/tcg/optimize.o
  CC      or1k-softmmu/hax-stub.o
  CC      or1k-softmmu/arch_init.o
  CC      ppc64-softmmu/disas.o
  CC      or1k-softmmu/cpus.o
  CC      ppcemb-softmmu/tcg/tcg-common.o
  CC      ppcemb-softmmu/tcg/tcg-runtime.o
  CC      or1k-softmmu/monitor.o
  CC      ppcemb-softmmu/fpu/softfloat.o
  GEN     ppc64-softmmu/gdbstub-xml.c
  CC      ppcemb-softmmu/disas.o
  CC      or1k-softmmu/gdbstub.o
  CC      ppc64-softmmu/hax-stub.o
  CC      or1k-softmmu/balloon.o
  CC      or1k-softmmu/ioport.o
  GEN     ppcemb-softmmu/gdbstub-xml.c
  CC      ppc64-softmmu/libdecnumber/decContext.o
  CC      or1k-softmmu/numa.o
  CC      ppc64-softmmu/libdecnumber/decNumber.o
  CC      ppcemb-softmmu/hax-stub.o
  CC      or1k-softmmu/memory.o
  CC      or1k-softmmu/qtest.o
  CC      ppcemb-softmmu/libdecnumber/decContext.o
  CC      or1k-softmmu/memory_mapping.o
  CC      ppcemb-softmmu/libdecnumber/decNumber.o
  CC      or1k-softmmu/dump.o
  CC      ppc64-softmmu/libdecnumber/dpd/decimal32.o
  CC      ppc64-softmmu/libdecnumber/dpd/decimal64.o
  CC      or1k-softmmu/migration/ram.o
  GEN     ppc-softmmu/hmp-commands.h
  CC      ppc64-softmmu/libdecnumber/dpd/decimal128.o
  GEN     ppc-softmmu/hmp-commands-info.h
  GEN     ppc-softmmu/config-target.h
  CC      ppc-softmmu/exec.o
  CC      ppc64-softmmu/arch_init.o
  CC      ppcemb-softmmu/libdecnumber/dpd/decimal32.o
  CC      ppc64-softmmu/cpus.o
  CC      or1k-softmmu/accel/accel.o
  CC      ppcemb-softmmu/libdecnumber/dpd/decimal64.o
  CC      ppcemb-softmmu/libdecnumber/dpd/decimal128.o
  CC      ppc64-softmmu/monitor.o
  CC      or1k-softmmu/accel/stubs/kvm-stub.o
  CC      ppcemb-softmmu/arch_init.o
  CC      or1k-softmmu/accel/tcg/tcg-all.o
  CC      ppcemb-softmmu/cpus.o
  CC      or1k-softmmu/accel/tcg/cputlb.o
  CC      ppc-softmmu/tcg/tcg.o
  CC      ppcemb-softmmu/monitor.o
  CC      ppc64-softmmu/gdbstub.o
  CC      or1k-softmmu/accel/tcg/cpu-exec.o
  CC      ppc64-softmmu/balloon.o
  CC      ppcemb-softmmu/gdbstub.o
  CC      ppc-softmmu/tcg/tcg-op.o
  CC      ppc64-softmmu/ioport.o
  CC      or1k-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppc64-softmmu/numa.o
  CC      ppcemb-softmmu/balloon.o
  CC      or1k-softmmu/accel/tcg/translate-all.o
  CC      ppcemb-softmmu/ioport.o
  CC      ppc64-softmmu/qtest.o
  CC      or1k-softmmu/hw/core/generic-loader.o
  CC      ppcemb-softmmu/numa.o
  CC      or1k-softmmu/hw/core/null-machine.o
  CC      ppc64-softmmu/memory.o
  CC      or1k-softmmu/hw/misc/mmio_interface.o
  CC      ppcemb-softmmu/qtest.o
  CC      or1k-softmmu/hw/net/vhost_net.o
  CC      ppcemb-softmmu/memory.o
  CC      ppc-softmmu/tcg/optimize.o
  CC      or1k-softmmu/hw/net/rocker/qmp-norocker.o
  CC      ppc64-softmmu/memory_mapping.o
  CC      or1k-softmmu/hw/vfio/common.o
  CC      ppc64-softmmu/dump.o
  CC      ppc64-softmmu/migration/ram.o
  CC      ppcemb-softmmu/memory_mapping.o
  CC      or1k-softmmu/hw/vfio/platform.o
  CC      ppc-softmmu/tcg/tcg-common.o
  CC      ppcemb-softmmu/dump.o
  CC      ppc-softmmu/tcg/tcg-runtime.o
  CC      or1k-softmmu/hw/vfio/spapr.o
  CC      ppc-softmmu/fpu/softfloat.o
  CC      ppc64-softmmu/accel/accel.o
  CC      or1k-softmmu/hw/openrisc/pic_cpu.o
  CC      ppcemb-softmmu/migration/ram.o
  CC      ppc64-softmmu/accel/stubs/kvm-stub.o
  CC      or1k-softmmu/hw/openrisc/cputimer.o
  CC      ppc64-softmmu/accel/tcg/tcg-all.o
  CC      or1k-softmmu/hw/openrisc/openrisc_sim.o
  CC      ppc64-softmmu/accel/tcg/cputlb.o
  CC      ppcemb-softmmu/accel/accel.o
  CC      or1k-softmmu/target/openrisc/machine.o
  CC      ppcemb-softmmu/accel/stubs/kvm-stub.o
  CC      or1k-softmmu/target/openrisc/cpu.o
  CC      or1k-softmmu/target/openrisc/exception.o
  CC      ppcemb-softmmu/accel/tcg/tcg-all.o
  CC      or1k-softmmu/target/openrisc/interrupt.o
  CC      ppcemb-softmmu/accel/tcg/cputlb.o
  CC      or1k-softmmu/target/openrisc/mmu.o
  CC      ppc64-softmmu/accel/tcg/cpu-exec.o
  CC      ppc-softmmu/disas.o
  CC      or1k-softmmu/target/openrisc/translate.o
  CC      ppc64-softmmu/accel/tcg/cpu-exec-common.o
  GEN     ppc-softmmu/gdbstub-xml.c
  CC      ppc64-softmmu/accel/tcg/translate-all.o
  CC      ppcemb-softmmu/accel/tcg/cpu-exec.o
  CC      or1k-softmmu/target/openrisc/exception_helper.o
  CC      or1k-softmmu/target/openrisc/fpu_helper.o
  CC      ppc-softmmu/hax-stub.o
  CC      ppcemb-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppc64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppcemb-softmmu/accel/tcg/translate-all.o
  CC      ppc-softmmu/libdecnumber/decContext.o
  CC      ppc64-softmmu/hw/block/virtio-blk.o
  CC      or1k-softmmu/target/openrisc/interrupt_helper.o
  CC      ppc-softmmu/libdecnumber/decNumber.o
  CC      or1k-softmmu/target/openrisc/mmu_helper.o
  CC      ppcemb-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppc64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      or1k-softmmu/target/openrisc/sys_helper.o
  CC      ppcemb-softmmu/hw/block/virtio-blk.o
  CC      or1k-softmmu/target/openrisc/gdbstub.o
  CC      ppc64-softmmu/hw/char/spapr_vty.o
  GEN     trace/generated-helpers.c
  CC      or1k-softmmu/trace/control-target.o
  CC      ppcemb-softmmu/hw/block/dataplane/virtio-blk.o
  CC      ppc64-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc-softmmu/libdecnumber/dpd/decimal32.o
  CC      or1k-softmmu/trace/generated-helpers.o
  CC      ppcemb-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc64-softmmu/hw/core/generic-loader.o
  CC      ppc-softmmu/libdecnumber/dpd/decimal64.o
  LINK    or1k-softmmu/qemu-system-or1k
  CC      ppc64-softmmu/hw/core/null-machine.o
  CC      ppc-softmmu/libdecnumber/dpd/decimal128.o
  CC      ppcemb-softmmu/hw/core/generic-loader.o
  CC      ppc64-softmmu/hw/display/sm501.o
  CC      ppc-softmmu/arch_init.o
  CC      ppcemb-softmmu/hw/core/null-machine.o
  CC      ppcemb-softmmu/hw/display/sm501.o
  CC      ppc-softmmu/cpus.o
  CC      ppc64-softmmu/hw/display/vga.o
  CC      ppc-softmmu/monitor.o
  CC      ppcemb-softmmu/hw/display/vga.o
  CC      ppc64-softmmu/hw/display/virtio-gpu.o
  CC      ppc-softmmu/gdbstub.o
  GEN     s390x-softmmu/hmp-commands.h
  GEN     s390x-softmmu/hmp-commands-info.h
  CC      s390x-softmmu/gen-features
  GEN     s390x-softmmu/config-target.h
  GEN     s390x-softmmu/gen-features.h
  CC      ppcemb-softmmu/hw/display/virtio-gpu.o
  CC      s390x-softmmu/exec.o
  CC      ppc64-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppc-softmmu/balloon.o
  CC      ppc64-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppc-softmmu/ioport.o
  CC      ppcemb-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppc64-softmmu/hw/display/virtio-vga.o
  CC      ppc-softmmu/numa.o
  CC      ppc64-softmmu/hw/intc/xics.o
  CC      ppcemb-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppc-softmmu/qtest.o
  CC      ppcemb-softmmu/hw/misc/ivshmem.o
  CC      ppc64-softmmu/hw/intc/xics_spapr.o
  CC      s390x-softmmu/tcg/tcg.o
  CC      ppc-softmmu/memory.o
  CC      ppcemb-softmmu/hw/misc/mmio_interface.o
  CC      ppc64-softmmu/hw/intc/xics_pnv.o
  CC      ppcemb-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc64-softmmu/hw/misc/ivshmem.o
  CC      ppcemb-softmmu/hw/net/virtio-net.o
  CC      ppc-softmmu/memory_mapping.o
  CC      ppc64-softmmu/hw/misc/mmio_interface.o
  CC      s390x-softmmu/tcg/tcg-op.o
  CC      ppc-softmmu/dump.o
  CC      ppc64-softmmu/hw/net/spapr_llan.o
  CC      ppcemb-softmmu/hw/net/vhost_net.o
  CC      ppc64-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc-softmmu/migration/ram.o
  CC      ppcemb-softmmu/hw/scsi/virtio-scsi.o
  CC      ppc64-softmmu/hw/net/virtio-net.o
  CC      ppcemb-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      ppc64-softmmu/hw/net/vhost_net.o
  CC      ppcemb-softmmu/hw/scsi/vhost-scsi-common.o
  CC      ppc-softmmu/accel/accel.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/etsec.o
  CC      ppcemb-softmmu/hw/scsi/vhost-scsi.o
  CC      ppc-softmmu/accel/stubs/kvm-stub.o
  CC      s390x-softmmu/tcg/optimize.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/registers.o
  CC      ppcemb-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc-softmmu/accel/tcg/tcg-all.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/rings.o
  CC      ppcemb-softmmu/hw/vfio/common.o
  CC      ppc-softmmu/accel/tcg/cputlb.o
  CC      ppc64-softmmu/hw/net/fsl_etsec/miim.o
  CC      ppcemb-softmmu/hw/vfio/pci.o
  CC      s390x-softmmu/tcg/tcg-common.o
  CC      ppc64-softmmu/hw/nvram/spapr_nvram.o
  CC      s390x-softmmu/tcg/tcg-runtime.o
  CC      ppc64-softmmu/hw/scsi/spapr_vscsi.o
  CC      s390x-softmmu/fpu/softfloat.o
  CC      ppcemb-softmmu/hw/vfio/pci-quirks.o
  CC      ppc-softmmu/accel/tcg/cpu-exec.o
  CC      ppc64-softmmu/hw/scsi/virtio-scsi.o
  CC      ppc-softmmu/accel/tcg/cpu-exec-common.o
  CC      ppc64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      ppcemb-softmmu/hw/vfio/platform.o
  CC      ppc-softmmu/accel/tcg/translate-all.o
  CC      ppc64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      ppcemb-softmmu/hw/vfio/spapr.o
  CC      ppc64-softmmu/hw/scsi/vhost-scsi.o
  CC      ppc-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppcemb-softmmu/hw/virtio/virtio.o
  CC      ppc64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc-softmmu/hw/block/virtio-blk.o
  CC      ppc64-softmmu/hw/timer/mc146818rtc.o
  CC      s390x-softmmu/disas.o
  CC      ppcemb-softmmu/hw/virtio/virtio-balloon.o
  CC      ppc-softmmu/hw/block/dataplane/virtio-blk.o
  CC      ppc64-softmmu/hw/vfio/common.o
  GEN     s390x-softmmu/gdbstub-xml.c
  CC      ppcemb-softmmu/hw/virtio/vhost.o
  CC      ppc-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc64-softmmu/hw/vfio/pci.o
  CC      s390x-softmmu/hax-stub.o
  CC      ppcemb-softmmu/hw/virtio/vhost-backend.o
  CC      ppc-softmmu/hw/core/generic-loader.o
  CC      s390x-softmmu/arch_init.o
  CC      ppc-softmmu/hw/core/null-machine.o
  CC      ppcemb-softmmu/hw/virtio/vhost-user.o
  CC      s390x-softmmu/cpus.o
  CC      ppc-softmmu/hw/display/sm501.o
  CC      ppcemb-softmmu/hw/virtio/vhost-vsock.o
  CC      ppc64-softmmu/hw/vfio/pci-quirks.o
  CC      ppcemb-softmmu/hw/virtio/virtio-crypto.o
  CC      ppc-softmmu/hw/display/vga.o
  CC      s390x-softmmu/monitor.o
  CC      ppc64-softmmu/hw/vfio/platform.o
  CC      ppcemb-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      ppc64-softmmu/hw/vfio/spapr.o
  CC      ppcemb-softmmu/hw/ppc/ppc.o
  CC      ppc64-softmmu/hw/virtio/virtio.o
  CC      ppc-softmmu/hw/display/virtio-gpu.o
  CC      ppcemb-softmmu/hw/ppc/ppc_booke.o
  CC      s390x-softmmu/gdbstub.o
  CC      ppcemb-softmmu/hw/ppc/fdt.o
  CC      ppc-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppcemb-softmmu/hw/ppc/ppc405_boards.o
  CC      ppc64-softmmu/hw/virtio/virtio-balloon.o
  CC      s390x-softmmu/balloon.o
  CC      ppcemb-softmmu/hw/ppc/ppc4xx_devs.o
  CC      s390x-softmmu/ioport.o
  CC      ppc64-softmmu/hw/virtio/vhost.o
  CC      ppc-softmmu/hw/display/virtio-gpu-pci.o
  CC      s390x-softmmu/numa.o
  CC      ppcemb-softmmu/hw/ppc/ppc405_uc.o
  CC      ppc-softmmu/hw/misc/ivshmem.o
  CC      ppc64-softmmu/hw/virtio/vhost-backend.o
  CC      s390x-softmmu/qtest.o
  CC      ppcemb-softmmu/hw/ppc/ppc440_bamboo.o
  CC      ppc64-softmmu/hw/virtio/vhost-user.o
  CC      ppc-softmmu/hw/misc/mmio_interface.o
  CC      s390x-softmmu/memory.o
  CC      ppcemb-softmmu/hw/ppc/ppc4xx_pci.o
  CC      ppc-softmmu/hw/net/xilinx_ethlite.o
  CC      ppc64-softmmu/hw/virtio/vhost-vsock.o
  CC      ppcemb-softmmu/hw/ppc/virtex_ml507.o
  CC      ppc-softmmu/hw/net/virtio-net.o
  CC      ppc64-softmmu/hw/virtio/virtio-crypto.o
  CC      ppcemb-softmmu/target/ppc/cpu-models.o
  CC      ppc64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      s390x-softmmu/memory_mapping.o
  CC      s390x-softmmu/dump.o
  CC      ppc-softmmu/hw/net/vhost_net.o
  CC      ppc64-softmmu/hw/ppc/ppc.o
  CC      ppcemb-softmmu/target/ppc/cpu.o
  CC      ppc-softmmu/hw/net/fsl_etsec/etsec.o
  CC      ppcemb-softmmu/target/ppc/translate.o
  CC      s390x-softmmu/migration/ram.o
  CC      ppc64-softmmu/hw/ppc/ppc_booke.o
  CC      ppc-softmmu/hw/net/fsl_etsec/registers.o
  CC      ppc-softmmu/hw/net/fsl_etsec/rings.o
  CC      ppc64-softmmu/hw/ppc/fdt.o
  CC      ppc-softmmu/hw/net/fsl_etsec/miim.o
  CC      ppc64-softmmu/hw/ppc/spapr.o
  CC      s390x-softmmu/accel/accel.o
  CC      ppc-softmmu/hw/scsi/virtio-scsi.o
  CC      s390x-softmmu/accel/kvm/kvm-all.o
  CC      ppc64-softmmu/hw/ppc/spapr_vio.o
  CC      ppc-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      s390x-softmmu/accel/tcg/tcg-all.o
  CC      ppc64-softmmu/hw/ppc/spapr_events.o
  CC      ppc-softmmu/hw/scsi/vhost-scsi-common.o
  CC      s390x-softmmu/accel/tcg/cputlb.o
  CC      ppc-softmmu/hw/scsi/vhost-scsi.o
  CC      ppc64-softmmu/hw/ppc/spapr_hcall.o
  CC      ppc-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc64-softmmu/hw/ppc/spapr_iommu.o
  CC      ppc-softmmu/hw/timer/mc146818rtc.o
  CC      ppc64-softmmu/hw/ppc/spapr_rtas.o
  CC      s390x-softmmu/accel/tcg/cpu-exec.o
  CC      ppc-softmmu/hw/vfio/common.o
  CC      ppc64-softmmu/hw/ppc/spapr_pci.o
  CC      s390x-softmmu/accel/tcg/cpu-exec-common.o
  CC      s390x-softmmu/accel/tcg/translate-all.o
  CC      ppc-softmmu/hw/vfio/pci.o
  CC      ppc64-softmmu/hw/ppc/spapr_rtc.o
  CC      s390x-softmmu/hw/9pfs/virtio-9p-device.o
  CC      ppc64-softmmu/hw/ppc/spapr_drc.o
  CC      ppc-softmmu/hw/vfio/pci-quirks.o
  CC      s390x-softmmu/hw/block/virtio-blk.o
  CC      ppc-softmmu/hw/vfio/platform.o
  CC      ppc64-softmmu/hw/ppc/spapr_rng.o
  CC      s390x-softmmu/hw/block/dataplane/virtio-blk.o
  CC      ppc64-softmmu/hw/ppc/spapr_cpu_core.o
  CC      s390x-softmmu/hw/char/virtio-serial-bus.o
  CC      ppc-softmmu/hw/vfio/spapr.o
  CC      ppc-softmmu/hw/virtio/virtio.o
  CC      ppc64-softmmu/hw/ppc/spapr_ovec.o
  CC      s390x-softmmu/hw/char/terminal3270.o
  CC      ppc64-softmmu/hw/ppc/pnv.o
  CC      s390x-softmmu/hw/core/generic-loader.o
  CC      ppc-softmmu/hw/virtio/virtio-balloon.o
  CC      s390x-softmmu/hw/core/null-machine.o
  CC      ppc64-softmmu/hw/ppc/pnv_xscom.o
  CC      s390x-softmmu/hw/display/virtio-gpu.o
  CC      ppc64-softmmu/hw/ppc/pnv_core.o
  CC      ppc-softmmu/hw/virtio/vhost.o
  CC      ppc64-softmmu/hw/ppc/pnv_lpc.o
  CC      s390x-softmmu/hw/display/virtio-gpu-3d.o
  CC      ppc64-softmmu/hw/ppc/pnv_psi.o
  CC      ppc-softmmu/hw/virtio/vhost-backend.o
  CC      ppc64-softmmu/hw/ppc/pnv_occ.o
  CC      s390x-softmmu/hw/display/virtio-gpu-pci.o
  CC      ppc-softmmu/hw/virtio/vhost-user.o
  CC      ppc64-softmmu/hw/ppc/pnv_bmc.o
  CC      ppc64-softmmu/hw/ppc/spapr_pci_vfio.o
  CC      ppc-softmmu/hw/virtio/vhost-vsock.o
  CC      s390x-softmmu/hw/intc/s390_flic.o
  CC      ppc64-softmmu/hw/ppc/spapr_rtas_ddw.o
  CC      ppc-softmmu/hw/virtio/virtio-crypto.o
  CC      s390x-softmmu/hw/intc/s390_flic_kvm.o
  CC      ppc64-softmmu/hw/ppc/ppc405_boards.o
  CC      ppcemb-softmmu/target/ppc/machine.o
  CC      ppc-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      s390x-softmmu/hw/misc/mmio_interface.o
  CC      ppc64-softmmu/hw/ppc/ppc4xx_devs.o
  CC      ppcemb-softmmu/target/ppc/mmu_helper.o
  CC      s390x-softmmu/hw/net/virtio-net.o
  CC      ppc-softmmu/hw/ppc/ppc.o
  CC      ppc64-softmmu/hw/ppc/ppc405_uc.o
  CC      ppc-softmmu/hw/ppc/ppc_booke.o
  CC      s390x-softmmu/hw/net/vhost_net.o
  CC      ppc-softmmu/hw/ppc/fdt.o
  CC      ppcemb-softmmu/target/ppc/mmu-hash32.o
  CC      ppc64-softmmu/hw/ppc/ppc440_bamboo.o
  CC      s390x-softmmu/hw/net/rocker/qmp-norocker.o
  CC      ppc-softmmu/hw/ppc/ppc405_boards.o
  CC      ppcemb-softmmu/target/ppc/monitor.o
  CC      s390x-softmmu/hw/scsi/virtio-scsi.o
  CC      ppc-softmmu/hw/ppc/ppc4xx_devs.o
  CC      ppc64-softmmu/hw/ppc/ppc4xx_pci.o
  CC      ppcemb-softmmu/target/ppc/arch_dump.o
  CC      ppc64-softmmu/hw/ppc/prep.o
  CC      s390x-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      ppcemb-softmmu/target/ppc/kvm-stub.o
  CC      ppc-softmmu/hw/ppc/ppc405_uc.o
  CC      ppcemb-softmmu/target/ppc/dfp_helper.o
  CC      s390x-softmmu/hw/scsi/vhost-scsi-common.o
  CC      ppc64-softmmu/hw/ppc/prep_systemio.o
  CC      s390x-softmmu/hw/scsi/vhost-scsi.o
  CC      ppc-softmmu/hw/ppc/ppc440_bamboo.o
  CC      s390x-softmmu/hw/scsi/vhost-user-scsi.o
  CC      ppc64-softmmu/hw/ppc/rs6000_mc.o
  CC      ppcemb-softmmu/target/ppc/excp_helper.o
  CC      s390x-softmmu/hw/vfio/common.o
  CC      ppc-softmmu/hw/ppc/ppc4xx_pci.o
  CC      ppc64-softmmu/hw/ppc/mac_oldworld.o
  CC      ppc64-softmmu/hw/ppc/mac_newworld.o
  CC      ppc-softmmu/hw/ppc/prep.o
  CC      ppcemb-softmmu/target/ppc/fpu_helper.o
  CC      s390x-softmmu/hw/vfio/pci.o
  CC      ppc64-softmmu/hw/ppc/e500.o
  CC      ppc-softmmu/hw/ppc/prep_systemio.o
  CC      ppc64-softmmu/hw/ppc/mpc8544ds.o
  CC      ppc-softmmu/hw/ppc/rs6000_mc.o
  CC      ppc64-softmmu/hw/ppc/e500plat.o
  CC      s390x-softmmu/hw/vfio/pci-quirks.o
  CC      ppc-softmmu/hw/ppc/mac_oldworld.o
  CC      ppc64-softmmu/hw/ppc/mpc8544_guts.o
  CC      ppc64-softmmu/hw/ppc/ppce500_spin.o
  CC      ppc-softmmu/hw/ppc/mac_newworld.o
  CC      s390x-softmmu/hw/vfio/ccw.o
  CC      ppc64-softmmu/hw/ppc/virtex_ml507.o
  CC      ppc-softmmu/hw/ppc/e500.o
  CC      s390x-softmmu/hw/vfio/platform.o
  CC      ppc64-softmmu/target/ppc/cpu-models.o
  CC      ppc-softmmu/hw/ppc/mpc8544ds.o
  CC      s390x-softmmu/hw/vfio/spapr.o
  CC      ppc-softmmu/hw/ppc/e500plat.o
  CC      s390x-softmmu/hw/virtio/virtio.o
  CC      ppc-softmmu/hw/ppc/mpc8544_guts.o
  CC      ppcemb-softmmu/target/ppc/int_helper.o
  CC      ppc-softmmu/hw/ppc/ppce500_spin.o
  CC      ppc64-softmmu/target/ppc/cpu.o
  CC      ppc-softmmu/hw/ppc/virtex_ml507.o
  CC      ppc-softmmu/target/ppc/cpu-models.o
  CC      s390x-softmmu/hw/virtio/virtio-balloon.o
  CC      ppc64-softmmu/target/ppc/translate.o
  CC      s390x-softmmu/hw/virtio/vhost.o
  CC      ppcemb-softmmu/target/ppc/timebase_helper.o
  CC      s390x-softmmu/hw/virtio/vhost-backend.o
  CC      ppc-softmmu/target/ppc/cpu.o
  CC      ppcemb-softmmu/target/ppc/misc_helper.o
  CC      s390x-softmmu/hw/virtio/vhost-user.o
  CC      ppc-softmmu/target/ppc/translate.o
  CC      s390x-softmmu/hw/virtio/vhost-vsock.o
  CC      ppcemb-softmmu/target/ppc/mem_helper.o
  CC      s390x-softmmu/hw/virtio/virtio-crypto.o
  CC      ppcemb-softmmu/target/ppc/gdbstub.o
  CC      s390x-softmmu/hw/virtio/virtio-crypto-pci.o
  GEN     trace/generated-helpers.c
  CC      ppcemb-softmmu/trace/control-target.o
  CC      s390x-softmmu/hw/s390x/s390-virtio.o
  CC      ppcemb-softmmu/gdbstub-xml.o
  CC      s390x-softmmu/hw/s390x/s390-virtio-hcall.o
  CC      ppcemb-softmmu/trace/generated-helpers.o
  CC      s390x-softmmu/hw/s390x/sclp.o
  LINK    ppcemb-softmmu/qemu-system-ppcemb
  CC      s390x-softmmu/hw/s390x/event-facility.o
  CC      s390x-softmmu/hw/s390x/sclpquiesce.o
  CC      s390x-softmmu/hw/s390x/sclpcpu.o
  CC      ppc-softmmu/target/ppc/machine.o
  CC      s390x-softmmu/hw/s390x/ipl.o
  CC      ppc64-softmmu/target/ppc/machine.o
  CC      s390x-softmmu/hw/s390x/css.o
  CC      ppc64-softmmu/target/ppc/mmu_helper.o
  CC      s390x-softmmu/hw/s390x/s390-virtio-ccw.o
  CC      s390x-softmmu/hw/s390x/3270-ccw.o
  GEN     sh4eb-softmmu/hmp-commands.h
  GEN     sh4eb-softmmu/hmp-commands-info.h
  GEN     sh4eb-softmmu/config-target.h
  CC      sh4eb-softmmu/exec.o
  CC      s390x-softmmu/hw/s390x/virtio-ccw.o
  CC      s390x-softmmu/hw/s390x/css-bridge.o
  CC      s390x-softmmu/hw/s390x/ccw-device.o
  CC      sh4eb-softmmu/tcg/tcg.o
  CC      s390x-softmmu/hw/s390x/s390-pci-bus.o
  CC      s390x-softmmu/hw/s390x/s390-pci-inst.o
  CC      sh4eb-softmmu/tcg/tcg-op.o
  CC      s390x-softmmu/hw/s390x/s390-skeys.o
  CC      ppc-softmmu/target/ppc/mmu_helper.o
  CC      s390x-softmmu/hw/s390x/s390-stattrib.o
  CC      s390x-softmmu/hw/s390x/s390-skeys-kvm.o
  CC      ppc64-softmmu/target/ppc/mmu-hash32.o
  CC      s390x-softmmu/hw/s390x/s390-stattrib-kvm.o
  CC      ppc-softmmu/target/ppc/mmu-hash32.o
  CC      ppc64-softmmu/target/ppc/monitor.o
  CC      s390x-softmmu/hw/s390x/s390-ccw.o
  CC      ppc-softmmu/target/ppc/monitor.o
  CC      ppc64-softmmu/target/ppc/arch_dump.o
  CC      ppc-softmmu/target/ppc/arch_dump.o
  CC      s390x-softmmu/target/s390x/cpu.o
  CC      ppc64-softmmu/target/ppc/mmu-hash64.o
  CC      sh4eb-softmmu/tcg/optimize.o
  CC      ppc-softmmu/target/ppc/kvm-stub.o
  CC      s390x-softmmu/target/s390x/cpu_models.o
  CC      ppc64-softmmu/target/ppc/mmu-book3s-v3.o
  CC      ppc-softmmu/target/ppc/dfp_helper.o
  CC      ppc64-softmmu/target/ppc/compat.o
  CC      s390x-softmmu/target/s390x/cpu_features.o
  CC      ppc64-softmmu/target/ppc/mmu-radix64.o
  CC      sh4eb-softmmu/tcg/tcg-common.o
  CC      s390x-softmmu/target/s390x/gdbstub.o
  CC      sh4eb-softmmu/tcg/tcg-runtime.o
  CC      ppc-softmmu/target/ppc/excp_helper.o
  CC      ppc64-softmmu/target/ppc/kvm-stub.o
  CC      sh4eb-softmmu/fpu/softfloat.o
  CC      s390x-softmmu/target/s390x/interrupt.o
  CC      ppc64-softmmu/target/ppc/dfp_helper.o
  CC      ppc-softmmu/target/ppc/fpu_helper.o
  CC      s390x-softmmu/target/s390x/helper.o
  CC      ppc64-softmmu/target/ppc/excp_helper.o
  CC      s390x-softmmu/target/s390x/translate.o
  CC      ppc64-softmmu/target/ppc/fpu_helper.o
  CC      sh4eb-softmmu/disas.o
  CC      sh4eb-softmmu/hax-stub.o
  CC      sh4eb-softmmu/arch_init.o
  CC      s390x-softmmu/target/s390x/cc_helper.o
  CC      ppc-softmmu/target/ppc/int_helper.o
  CC      sh4eb-softmmu/cpus.o
  CC      s390x-softmmu/target/s390x/excp_helper.o
  CC      s390x-softmmu/target/s390x/fpu_helper.o
  CC      sh4eb-softmmu/monitor.o
  CC      s390x-softmmu/target/s390x/int_helper.o
  CC      ppc64-softmmu/target/ppc/int_helper.o
  CC      s390x-softmmu/target/s390x/mem_helper.o
  CC      ppc-softmmu/target/ppc/timebase_helper.o
  CC      ppc-softmmu/target/ppc/misc_helper.o
  CC      sh4eb-softmmu/gdbstub.o
  CC      ppc-softmmu/target/ppc/mem_helper.o
  CC      sh4eb-softmmu/balloon.o
  CC      ppc64-softmmu/target/ppc/timebase_helper.o
  CC      ppc-softmmu/target/ppc/gdbstub.o
  CC      sh4eb-softmmu/ioport.o
  CC      ppc64-softmmu/target/ppc/misc_helper.o
  GEN     trace/generated-helpers.c
  CC      ppc-softmmu/trace/control-target.o
  CC      sh4eb-softmmu/numa.o
  CC      ppc64-softmmu/target/ppc/mem_helper.o
  CC      s390x-softmmu/target/s390x/misc_helper.o
  CC      ppc-softmmu/gdbstub-xml.o
  CC      sh4eb-softmmu/qtest.o
  CC      s390x-softmmu/target/s390x/machine.o
  CC      ppc-softmmu/trace/generated-helpers.o
  CC      ppc64-softmmu/target/ppc/gdbstub.o
  CC      sh4eb-softmmu/memory.o
  CC      s390x-softmmu/target/s390x/ioinst.o
  GEN     trace/generated-helpers.c
  LINK    ppc-softmmu/qemu-system-ppc
  CC      ppc64-softmmu/trace/control-target.o
  CC      s390x-softmmu/target/s390x/arch_dump.o
  CC      ppc64-softmmu/gdbstub-xml.o
  CC      ppc64-softmmu/trace/generated-helpers.o
  CC      s390x-softmmu/target/s390x/mmu_helper.o
  CC      sh4eb-softmmu/memory_mapping.o
  LINK    ppc64-softmmu/qemu-system-ppc64
  CC      sh4eb-softmmu/dump.o
  CC      s390x-softmmu/target/s390x/diag.o
  CC      s390x-softmmu/target/s390x/kvm.o
  CC      sh4eb-softmmu/migration/ram.o
  GEN     trace/generated-helpers.c
  CC      s390x-softmmu/trace/control-target.o
  CC      s390x-softmmu/gdbstub-xml.o
  CC      sh4eb-softmmu/accel/accel.o
  CC      s390x-softmmu/trace/generated-helpers.o
  CC      sh4eb-softmmu/accel/stubs/kvm-stub.o
  CC      sh4eb-softmmu/accel/tcg/tcg-all.o
  CC      sh4eb-softmmu/accel/tcg/cputlb.o
  CC      sh4eb-softmmu/accel/tcg/cpu-exec.o
  LINK    s390x-softmmu/qemu-system-s390x
  CC      sh4eb-softmmu/accel/tcg/cpu-exec-common.o
  CC      sh4eb-softmmu/accel/tcg/translate-all.o
  CC      sh4eb-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sh4eb-softmmu/hw/block/tc58128.o
  CC      sh4eb-softmmu/hw/block/virtio-blk.o
  CC      sh4eb-softmmu/hw/block/dataplane/virtio-blk.o
  GEN     sh4-softmmu/hmp-commands.h
  GEN     sh4-softmmu/hmp-commands-info.h
  GEN     sh4-softmmu/config-target.h
  CC      sh4eb-softmmu/hw/char/sh_serial.o
  GEN     sparc64-softmmu/hmp-commands.h
  GEN     sparc64-softmmu/hmp-commands-info.h
  GEN     sparc64-softmmu/config-target.h
  CC      sparc64-softmmu/exec.o
  CC      sh4-softmmu/exec.o
  CC      sparc64-softmmu/tcg/tcg.o
  CC      sh4eb-softmmu/hw/char/virtio-serial-bus.o
  CC      sh4eb-softmmu/hw/core/generic-loader.o
  CC      sh4eb-softmmu/hw/core/null-machine.o
  CC      sh4eb-softmmu/hw/display/sm501.o
  CC      sh4-softmmu/tcg/tcg.o
  GEN     sparc-softmmu/hmp-commands.h
  GEN     sparc-softmmu/hmp-commands-info.h
  GEN     sparc-softmmu/config-target.h
  CC      sparc-softmmu/exec.o
  CC      sparc64-softmmu/tcg/tcg-op.o
  CC      sh4eb-softmmu/hw/display/vga.o
  CC      sh4-softmmu/tcg/tcg-op.o
  CC      sh4eb-softmmu/hw/display/virtio-gpu.o
  CC      sparc-softmmu/tcg/tcg.o
  CC      sh4eb-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc64-softmmu/tcg/optimize.o
  CC      sparc-softmmu/tcg/tcg-op.o
  CC      sh4eb-softmmu/hw/display/virtio-gpu-pci.o
  CC      sparc64-softmmu/tcg/tcg-common.o
  CC      sh4-softmmu/tcg/optimize.o
  CC      sh4eb-softmmu/hw/intc/sh_intc.o
  CC      sparc64-softmmu/tcg/tcg-runtime.o
  CC      sh4eb-softmmu/hw/misc/ivshmem.o
  CC      sh4-softmmu/tcg/tcg-common.o
  CC      sparc64-softmmu/fpu/softfloat.o
  CC      sh4eb-softmmu/hw/misc/mmio_interface.o
  CC      sh4-softmmu/tcg/tcg-runtime.o
  CC      sh4eb-softmmu/hw/net/virtio-net.o
  CC      sh4-softmmu/fpu/softfloat.o
  CC      sparc-softmmu/tcg/optimize.o
  CC      sh4eb-softmmu/hw/net/vhost_net.o
  CC      sparc-softmmu/tcg/tcg-common.o
  CC      sh4eb-softmmu/hw/scsi/virtio-scsi.o
  CC      sparc64-softmmu/disas.o
  CC      sparc-softmmu/tcg/tcg-runtime.o
  CC      sh4eb-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      sparc64-softmmu/hax-stub.o
  CC      sparc-softmmu/fpu/softfloat.o
  CC      sparc64-softmmu/arch_init.o
  CC      sh4eb-softmmu/hw/scsi/vhost-scsi-common.o
  CC      sh4-softmmu/disas.o
  CC      sparc64-softmmu/cpus.o
  CC      sh4eb-softmmu/hw/scsi/vhost-scsi.o
  CC      sh4-softmmu/hax-stub.o
  CC      sparc64-softmmu/monitor.o
  CC      sh4-softmmu/arch_init.o
  CC      sh4eb-softmmu/hw/scsi/vhost-user-scsi.o
  CC      sh4-softmmu/cpus.o
  CC      sh4eb-softmmu/hw/timer/sh_timer.o
  CC      sh4-softmmu/monitor.o
  CC      sh4eb-softmmu/hw/timer/mc146818rtc.o
  CC      sparc64-softmmu/gdbstub.o
  CC      sparc-softmmu/disas.o
  CC      sh4eb-softmmu/hw/vfio/common.o
  CC      sparc-softmmu/hax-stub.o
  CC      sparc64-softmmu/balloon.o
  CC      sh4-softmmu/gdbstub.o
  CC      sparc-softmmu/arch_init.o
  CC      sh4eb-softmmu/hw/vfio/pci.o
  CC      sparc64-softmmu/ioport.o
  CC      sparc-softmmu/cpus.o
  CC      sh4-softmmu/balloon.o
  CC      sparc64-softmmu/numa.o
  CC      sparc-softmmu/monitor.o
  CC      sh4-softmmu/ioport.o
  CC      sh4eb-softmmu/hw/vfio/pci-quirks.o
  CC      sparc64-softmmu/qtest.o
  CC      sh4-softmmu/numa.o
  CC      sh4eb-softmmu/hw/vfio/platform.o
  CC      sparc64-softmmu/memory.o
  CC      sh4-softmmu/qtest.o
  CC      sh4eb-softmmu/hw/vfio/spapr.o
  CC      sh4eb-softmmu/hw/virtio/virtio.o
  CC      sparc-softmmu/gdbstub.o
  CC      sh4-softmmu/memory.o
  CC      sparc64-softmmu/memory_mapping.o
  CC      sh4eb-softmmu/hw/virtio/virtio-balloon.o
  CC      sparc-softmmu/balloon.o
  CC      sparc64-softmmu/dump.o
  CC      sparc-softmmu/ioport.o
  CC      sh4eb-softmmu/hw/virtio/vhost.o
  CC      sparc-softmmu/numa.o
  CC      sh4-softmmu/memory_mapping.o
  CC      sparc64-softmmu/migration/ram.o
  CC      sparc-softmmu/qtest.o
  CC      sh4-softmmu/dump.o
  CC      sh4eb-softmmu/hw/virtio/vhost-backend.o
  CC      sparc-softmmu/memory.o
  CC      sparc64-softmmu/accel/accel.o
  CC      sh4eb-softmmu/hw/virtio/vhost-user.o
  CC      sh4-softmmu/migration/ram.o
  CC      sparc64-softmmu/accel/stubs/kvm-stub.o
  CC      sh4eb-softmmu/hw/virtio/vhost-vsock.o
  CC      sh4eb-softmmu/hw/virtio/virtio-crypto.o
  CC      sparc64-softmmu/accel/tcg/tcg-all.o
  CC      sh4-softmmu/accel/accel.o
  CC      sparc-softmmu/memory_mapping.o
  CC      sparc64-softmmu/accel/tcg/cputlb.o
  CC      sh4eb-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      sh4-softmmu/accel/stubs/kvm-stub.o
  CC      sparc-softmmu/dump.o
  CC      sh4-softmmu/accel/tcg/tcg-all.o
  CC      sh4eb-softmmu/hw/sh4/shix.o
  CC      sh4-softmmu/accel/tcg/cputlb.o
  CC      sh4eb-softmmu/hw/sh4/r2d.o
  CC      sparc-softmmu/migration/ram.o
  CC      sh4eb-softmmu/hw/sh4/sh7750.o
  CC      sh4eb-softmmu/hw/sh4/sh7750_regnames.o
  CC      sparc64-softmmu/accel/tcg/cpu-exec.o
  CC      sh4eb-softmmu/hw/sh4/sh_pci.o
  CC      sh4eb-softmmu/target/sh4/translate.o
  CC      sparc-softmmu/accel/accel.o
  CC      sparc64-softmmu/accel/tcg/cpu-exec-common.o
  CC      sh4-softmmu/accel/tcg/cpu-exec.o
  CC      sparc-softmmu/accel/stubs/kvm-stub.o
  CC      sparc64-softmmu/accel/tcg/translate-all.o
  CC      sh4-softmmu/accel/tcg/cpu-exec-common.o
  CC      sparc-softmmu/accel/tcg/tcg-all.o
  CC      sparc64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sparc-softmmu/accel/tcg/cputlb.o
  CC      sh4-softmmu/accel/tcg/translate-all.o
  CC      sparc64-softmmu/hw/block/virtio-blk.o
  CC      sh4eb-softmmu/target/sh4/op_helper.o
  CC      sh4-softmmu/hw/9pfs/virtio-9p-device.o
  CC      sh4eb-softmmu/target/sh4/helper.o
  CC      sparc64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      sh4-softmmu/hw/block/tc58128.o
  CC      sparc64-softmmu/hw/char/virtio-serial-bus.o
  CC      sh4eb-softmmu/target/sh4/cpu.o
  CC      sparc-softmmu/accel/tcg/cpu-exec.o
  CC      sh4-softmmu/hw/block/virtio-blk.o
  CC      sh4eb-softmmu/target/sh4/monitor.o
  CC      sparc-softmmu/accel/tcg/cpu-exec-common.o
  CC      sparc64-softmmu/hw/core/generic-loader.o
  CC      sh4eb-softmmu/target/sh4/gdbstub.o
  CC      sparc-softmmu/accel/tcg/translate-all.o
  CC      sh4-softmmu/hw/block/dataplane/virtio-blk.o
  GEN     trace/generated-helpers.c
  CC      sparc64-softmmu/hw/core/null-machine.o
  CC      sh4eb-softmmu/trace/control-target.o
  CC      sh4-softmmu/hw/char/sh_serial.o
  CC      sparc64-softmmu/hw/display/vga.o
  CC      sparc-softmmu/hw/core/generic-loader.o
  CC      sh4eb-softmmu/trace/generated-helpers.o
  CC      sh4-softmmu/hw/char/virtio-serial-bus.o
  CC      sparc-softmmu/hw/core/null-machine.o
  LINK    sh4eb-softmmu/qemu-system-sh4eb
  CC      sparc-softmmu/hw/display/tcx.o
  CC      sh4-softmmu/hw/core/generic-loader.o
  CC      sparc64-softmmu/hw/display/virtio-gpu.o
  CC      sh4-softmmu/hw/core/null-machine.o
  CC      sparc-softmmu/hw/display/cg3.o
  CC      sh4-softmmu/hw/display/sm501.o
  CC      sparc-softmmu/hw/intc/grlib_irqmp.o
  CC      sparc64-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc-softmmu/hw/misc/eccmemctl.o
  CC      sh4-softmmu/hw/display/vga.o
  CC      sparc-softmmu/hw/misc/slavio_misc.o
  CC      sparc64-softmmu/hw/display/virtio-gpu-pci.o
  CC      sparc-softmmu/hw/misc/mmio_interface.o
  GEN     tricore-softmmu/hmp-commands.h
  GEN     tricore-softmmu/hmp-commands-info.h
  GEN     tricore-softmmu/config-target.h
  CC      sparc64-softmmu/hw/misc/ivshmem.o
  CC      sparc-softmmu/hw/net/vhost_net.o
  CC      tricore-softmmu/exec.o
  CC      sh4-softmmu/hw/display/virtio-gpu.o
  CC      sparc-softmmu/hw/net/rocker/qmp-norocker.o
  CC      sparc64-softmmu/hw/misc/mmio_interface.o
  CC      sparc-softmmu/hw/vfio/common.o
  CC      sparc64-softmmu/hw/net/virtio-net.o
  CC      sh4-softmmu/hw/display/virtio-gpu-3d.o
  CC      sparc-softmmu/hw/vfio/platform.o
  CC      sparc-softmmu/hw/vfio/spapr.o
  CC      sparc64-softmmu/hw/net/vhost_net.o
  CC      sparc64-softmmu/hw/scsi/virtio-scsi.o
  CC      sh4-softmmu/hw/display/virtio-gpu-pci.o
  CC      sparc-softmmu/hw/sparc/sun4m.o
  CC      tricore-softmmu/tcg/tcg.o
  CC      sh4-softmmu/hw/intc/sh_intc.o
  CC      sparc64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      sparc64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      sparc-softmmu/hw/sparc/leon3.o
  CC      sh4-softmmu/hw/misc/ivshmem.o
  CC      sparc-softmmu/target/sparc/machine.o
  CC      sparc64-softmmu/hw/scsi/vhost-scsi.o
  CC      sparc-softmmu/target/sparc/monitor.o
  CC      sh4-softmmu/hw/misc/mmio_interface.o
  CC      sparc64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      sparc-softmmu/target/sparc/translate.o
  CC      sh4-softmmu/hw/net/virtio-net.o
  CC      sparc64-softmmu/hw/timer/mc146818rtc.o
  CC      tricore-softmmu/tcg/tcg-op.o
  CC      sparc64-softmmu/hw/vfio/common.o
  CC      sh4-softmmu/hw/net/vhost_net.o
  CC      sparc-softmmu/target/sparc/helper.o
  CC      sh4-softmmu/hw/scsi/virtio-scsi.o
  CC      sparc64-softmmu/hw/vfio/pci.o
  CC      sparc-softmmu/target/sparc/cpu.o
  CC      sh4-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      sparc-softmmu/target/sparc/fop_helper.o
  CC      sh4-softmmu/hw/scsi/vhost-scsi-common.o
  CC      sh4-softmmu/hw/scsi/vhost-scsi.o
  CC      sparc-softmmu/target/sparc/cc_helper.o
  CC      sparc64-softmmu/hw/vfio/pci-quirks.o
  CC      tricore-softmmu/tcg/optimize.o
  CC      sh4-softmmu/hw/scsi/vhost-user-scsi.o
  CC      sparc-softmmu/target/sparc/win_helper.o
  CC      sh4-softmmu/hw/timer/sh_timer.o
  CC      sparc-softmmu/target/sparc/mmu_helper.o
  CC      sparc64-softmmu/hw/vfio/platform.o
  CC      sh4-softmmu/hw/timer/mc146818rtc.o
  CC      sparc-softmmu/target/sparc/ldst_helper.o
  CC      tricore-softmmu/tcg/tcg-common.o
  CC      sparc64-softmmu/hw/vfio/spapr.o
  CC      tricore-softmmu/tcg/tcg-runtime.o
  CC      sparc-softmmu/target/sparc/int32_helper.o
  CC      sh4-softmmu/hw/vfio/common.o
  CC      tricore-softmmu/fpu/softfloat.o
  CC      sparc64-softmmu/hw/virtio/virtio.o
  CC      sparc-softmmu/target/sparc/gdbstub.o
  GEN     trace/generated-helpers.c
  CC      sh4-softmmu/hw/vfio/pci.o
  CC      sparc-softmmu/trace/control-target.o
  CC      sparc64-softmmu/hw/virtio/virtio-balloon.o
  CC      sparc-softmmu/trace/generated-helpers.o
  CC      sparc64-softmmu/hw/virtio/vhost.o
  LINK    sparc-softmmu/qemu-system-sparc
  CC      sh4-softmmu/hw/vfio/pci-quirks.o
  CC      sparc64-softmmu/hw/virtio/vhost-backend.o
  CC      tricore-softmmu/disas.o
  CC      sh4-softmmu/hw/vfio/platform.o
  CC      sparc64-softmmu/hw/virtio/vhost-user.o
  CC      tricore-softmmu/hax-stub.o
  CC      tricore-softmmu/arch_init.o
  CC      sh4-softmmu/hw/vfio/spapr.o
  CC      tricore-softmmu/cpus.o
  CC      sh4-softmmu/hw/virtio/virtio.o
  CC      sparc64-softmmu/hw/virtio/vhost-vsock.o
  CC      sparc64-softmmu/hw/virtio/virtio-crypto.o
  CC      sh4-softmmu/hw/virtio/virtio-balloon.o
  CC      tricore-softmmu/monitor.o
  CC      sparc64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      sh4-softmmu/hw/virtio/vhost.o
  CC      sparc64-softmmu/hw/sparc64/sparc64.o
  CC      sh4-softmmu/hw/virtio/vhost-backend.o
  CC      sparc64-softmmu/hw/sparc64/sun4u.o
  CC      sparc64-softmmu/hw/sparc64/niagara.o
  CC      sh4-softmmu/hw/virtio/vhost-user.o
  CC      sh4-softmmu/hw/virtio/vhost-vsock.o
  CC      tricore-softmmu/gdbstub.o
  CC      sparc64-softmmu/target/sparc/machine.o
  CC      sparc64-softmmu/target/sparc/monitor.o
  CC      sh4-softmmu/hw/virtio/virtio-crypto.o
  CC      tricore-softmmu/balloon.o
  CC      sparc64-softmmu/target/sparc/translate.o
  CC      sh4-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      tricore-softmmu/ioport.o
  GEN     unicore32-softmmu/hmp-commands.h
  GEN     unicore32-softmmu/hmp-commands-info.h
  GEN     unicore32-softmmu/config-target.h
  CC      sh4-softmmu/hw/sh4/shix.o
  CC      unicore32-softmmu/exec.o
  CC      tricore-softmmu/numa.o
  CC      sh4-softmmu/hw/sh4/r2d.o
  CC      tricore-softmmu/qtest.o
  CC      sh4-softmmu/hw/sh4/sh7750.o
  CC      tricore-softmmu/memory.o
  CC      sh4-softmmu/hw/sh4/sh7750_regnames.o
  CC      unicore32-softmmu/tcg/tcg.o
  CC      sparc64-softmmu/target/sparc/helper.o
  CC      sh4-softmmu/hw/sh4/sh_pci.o
  CC      sparc64-softmmu/target/sparc/cpu.o
  CC      tricore-softmmu/memory_mapping.o
  CC      sh4-softmmu/target/sh4/translate.o
  CC      tricore-softmmu/dump.o
  CC      sparc64-softmmu/target/sparc/fop_helper.o
  CC      sparc64-softmmu/target/sparc/cc_helper.o
  CC      unicore32-softmmu/tcg/tcg-op.o
  CC      tricore-softmmu/migration/ram.o
  CC      sparc64-softmmu/target/sparc/win_helper.o
  CC      sh4-softmmu/target/sh4/op_helper.o
  CC      sparc64-softmmu/target/sparc/mmu_helper.o
  CC      tricore-softmmu/accel/accel.o
  CC      sh4-softmmu/target/sh4/helper.o
  CC      sparc64-softmmu/target/sparc/ldst_helper.o
  CC      tricore-softmmu/accel/stubs/kvm-stub.o
  CC      sparc64-softmmu/target/sparc/int64_helper.o
  CC      tricore-softmmu/accel/tcg/tcg-all.o
  CC      sh4-softmmu/target/sh4/cpu.o
  CC      tricore-softmmu/accel/tcg/cputlb.o
  CC      sparc64-softmmu/target/sparc/vis_helper.o
  CC      sh4-softmmu/target/sh4/monitor.o
  CC      unicore32-softmmu/tcg/optimize.o
  CC      sparc64-softmmu/target/sparc/gdbstub.o
  CC      sh4-softmmu/target/sh4/gdbstub.o
  GEN     trace/generated-helpers.c
  CC      sparc64-softmmu/trace/control-target.o
  GEN     trace/generated-helpers.c
  CC      sh4-softmmu/trace/control-target.o
  CC      sparc64-softmmu/trace/generated-helpers.o
  CC      unicore32-softmmu/tcg/tcg-common.o
  CC      sh4-softmmu/trace/generated-helpers.o
  CC      tricore-softmmu/accel/tcg/cpu-exec.o
  CC      unicore32-softmmu/tcg/tcg-runtime.o
  LINK    sparc64-softmmu/qemu-system-sparc64
  CC      tricore-softmmu/accel/tcg/cpu-exec-common.o
  LINK    sh4-softmmu/qemu-system-sh4
  CC      tricore-softmmu/accel/tcg/translate-all.o
  CC      unicore32-softmmu/fpu/softfloat.o
  CC      tricore-softmmu/hw/core/generic-loader.o
  CC      tricore-softmmu/hw/core/null-machine.o
  CC      unicore32-softmmu/disas.o
  CC      tricore-softmmu/hw/misc/mmio_interface.o
  CC      tricore-softmmu/hw/net/vhost_net.o
  GEN     x86_64-softmmu/hmp-commands.h
  GEN     xtensaeb-softmmu/hmp-commands.h
  GEN     x86_64-softmmu/hmp-commands-info.h
  GEN     x86_64-softmmu/config-target.h
  CC      tricore-softmmu/hw/net/rocker/qmp-norocker.o
  GEN     xtensaeb-softmmu/hmp-commands-info.h
  GEN     xtensaeb-softmmu/config-target.h
  CC      x86_64-softmmu/exec.o
  CC      xtensaeb-softmmu/exec.o
  CC      tricore-softmmu/hw/vfio/common.o
  CC      unicore32-softmmu/hax-stub.o
  CC      tricore-softmmu/hw/vfio/platform.o
  CC      unicore32-softmmu/arch_init.o
  CC      tricore-softmmu/hw/vfio/spapr.o
  CC      unicore32-softmmu/cpus.o
  CC      tricore-softmmu/hw/tricore/tricore_testboard.o
  CC      x86_64-softmmu/tcg/tcg.o
  CC      xtensaeb-softmmu/tcg/tcg.o
  CC      tricore-softmmu/target/tricore/translate.o
  CC      unicore32-softmmu/monitor.o
  CC      xtensaeb-softmmu/tcg/tcg-op.o
  CC      unicore32-softmmu/gdbstub.o
  CC      x86_64-softmmu/tcg/tcg-op.o
  CC      unicore32-softmmu/balloon.o
  CC      unicore32-softmmu/ioport.o
  CC      unicore32-softmmu/numa.o
  CC      tricore-softmmu/target/tricore/helper.o
  CC      xtensaeb-softmmu/tcg/optimize.o
  CC      tricore-softmmu/target/tricore/cpu.o
  CC      unicore32-softmmu/qtest.o
  CC      x86_64-softmmu/tcg/optimize.o
  CC      tricore-softmmu/target/tricore/op_helper.o
  CC      unicore32-softmmu/memory.o
  CC      xtensaeb-softmmu/tcg/tcg-common.o
  CC      xtensaeb-softmmu/tcg/tcg-runtime.o
  CC      x86_64-softmmu/tcg/tcg-common.o
  CC      x86_64-softmmu/tcg/tcg-runtime.o
  CC      unicore32-softmmu/memory_mapping.o
  CC      xtensaeb-softmmu/fpu/softfloat.o
  CC      tricore-softmmu/target/tricore/fpu_helper.o
  CC      x86_64-softmmu/fpu/softfloat.o
  CC      unicore32-softmmu/dump.o
  GEN     trace/generated-helpers.c
  CC      tricore-softmmu/trace/control-target.o
  CC      unicore32-softmmu/migration/ram.o
  CC      tricore-softmmu/trace/generated-helpers.o
  CC      unicore32-softmmu/accel/accel.o
  LINK    tricore-softmmu/qemu-system-tricore
  CC      unicore32-softmmu/accel/stubs/kvm-stub.o
  CC      xtensaeb-softmmu/disas.o
  CC      unicore32-softmmu/accel/tcg/tcg-all.o
  CC      x86_64-softmmu/disas.o
  GEN     x86_64-softmmu/gdbstub-xml.c
  CC      xtensaeb-softmmu/hax-stub.o
  CC      unicore32-softmmu/accel/tcg/cputlb.o
  CC      x86_64-softmmu/hax-stub.o
  CC      xtensaeb-softmmu/arch_init.o
  GEN     xtensa-softmmu/hmp-commands.h
  GEN     xtensa-softmmu/hmp-commands-info.h
  GEN     xtensa-softmmu/config-target.h
  CC      x86_64-softmmu/arch_init.o
  CC      xtensa-softmmu/exec.o
  CC      xtensaeb-softmmu/cpus.o
  CC      x86_64-softmmu/cpus.o
  CC      xtensaeb-softmmu/monitor.o
  CC      unicore32-softmmu/accel/tcg/cpu-exec.o
  CC      x86_64-softmmu/monitor.o
  CC      unicore32-softmmu/accel/tcg/cpu-exec-common.o
  CC      unicore32-softmmu/accel/tcg/translate-all.o
  CC      xtensa-softmmu/tcg/tcg.o
  CC      xtensaeb-softmmu/gdbstub.o
  CC      unicore32-softmmu/hw/core/generic-loader.o
  CC      unicore32-softmmu/hw/core/null-machine.o
  CC      xtensaeb-softmmu/balloon.o
  CC      x86_64-softmmu/gdbstub.o
  CC      unicore32-softmmu/hw/misc/mmio_interface.o
  CC      xtensaeb-softmmu/ioport.o
  CC      unicore32-softmmu/hw/net/vhost_net.o
  CC      xtensa-softmmu/tcg/tcg-op.o
  CC      xtensaeb-softmmu/numa.o
  CC      x86_64-softmmu/balloon.o
  CC      unicore32-softmmu/hw/net/rocker/qmp-norocker.o
  CC      x86_64-softmmu/ioport.o
  CC      xtensaeb-softmmu/qtest.o
  CC      unicore32-softmmu/hw/vfio/common.o
  CC      x86_64-softmmu/numa.o
  CC      xtensaeb-softmmu/memory.o
  CC      unicore32-softmmu/hw/vfio/platform.o
  CC      x86_64-softmmu/qtest.o
  CC      unicore32-softmmu/hw/vfio/spapr.o
  CC      xtensaeb-softmmu/memory_mapping.o
  CC      x86_64-softmmu/memory.o
  CC      unicore32-softmmu/hw/unicore32/puv3.o
  CC      xtensaeb-softmmu/dump.o
  CC      xtensa-softmmu/tcg/optimize.o
  CC      unicore32-softmmu/target/unicore32/translate.o
  CC      xtensaeb-softmmu/migration/ram.o
  CC      x86_64-softmmu/memory_mapping.o
  CC      xtensa-softmmu/tcg/tcg-common.o
  CC      x86_64-softmmu/dump.o
  CC      xtensa-softmmu/tcg/tcg-runtime.o
  CC      unicore32-softmmu/target/unicore32/op_helper.o
  CC      xtensaeb-softmmu/accel/accel.o
  CC      xtensa-softmmu/fpu/softfloat.o
  CC      unicore32-softmmu/target/unicore32/helper.o
  CC      xtensaeb-softmmu/accel/stubs/kvm-stub.o
  CC      x86_64-softmmu/migration/ram.o
  CC      unicore32-softmmu/target/unicore32/cpu.o
  CC      xtensaeb-softmmu/accel/tcg/tcg-all.o
  CC      unicore32-softmmu/target/unicore32/ucf64_helper.o
  CC      xtensaeb-softmmu/accel/tcg/cputlb.o
  CC      unicore32-softmmu/target/unicore32/softmmu.o
  CC      x86_64-softmmu/accel/accel.o
  CC      x86_64-softmmu/accel/stubs/kvm-stub.o
  GEN     trace/generated-helpers.c
  CC      unicore32-softmmu/trace/control-target.o
  CC      x86_64-softmmu/accel/tcg/tcg-all.o
  CC      unicore32-softmmu/trace/generated-helpers.o
  CC      x86_64-softmmu/accel/tcg/cputlb.o
  CC      xtensaeb-softmmu/accel/tcg/cpu-exec.o
  CC      xtensa-softmmu/disas.o
  LINK    unicore32-softmmu/qemu-system-unicore32
  CC      xtensa-softmmu/hax-stub.o
  CC      xtensaeb-softmmu/accel/tcg/cpu-exec-common.o
  CC      xtensa-softmmu/arch_init.o
  CC      xtensaeb-softmmu/accel/tcg/translate-all.o
  CC      xtensa-softmmu/cpus.o
  CC      x86_64-softmmu/accel/tcg/cpu-exec.o
  CC      xtensaeb-softmmu/hw/core/generic-loader.o
  CC      xtensaeb-softmmu/hw/core/null-machine.o
  GEN     aarch64-linux-user/config-target.h
  CC      aarch64-linux-user/exec.o
  CC      xtensa-softmmu/monitor.o
  CC      x86_64-softmmu/accel/tcg/cpu-exec-common.o
  CC      xtensaeb-softmmu/hw/misc/mmio_interface.o
  CC      x86_64-softmmu/accel/tcg/translate-all.o
  CC      aarch64-linux-user/tcg/tcg.o
  CC      xtensaeb-softmmu/hw/net/vhost_net.o
  CC      x86_64-softmmu/hw/9pfs/virtio-9p-device.o
  CC      xtensaeb-softmmu/hw/net/rocker/qmp-norocker.o
  CC      xtensaeb-softmmu/hw/vfio/common.o
  CC      x86_64-softmmu/hw/block/virtio-blk.o
  CC      xtensa-softmmu/gdbstub.o
  CC      xtensa-softmmu/balloon.o
  CC      xtensaeb-softmmu/hw/vfio/platform.o
  CC      x86_64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      xtensa-softmmu/ioport.o
  CC      aarch64-linux-user/tcg/tcg-op.o
  CC      x86_64-softmmu/hw/char/virtio-serial-bus.o
  CC      xtensaeb-softmmu/hw/vfio/spapr.o
  CC      xtensa-softmmu/numa.o
  CC      xtensaeb-softmmu/hw/xtensa/pic_cpu.o
  CC      x86_64-softmmu/hw/core/generic-loader.o
  CC      xtensa-softmmu/qtest.o
  CC      xtensaeb-softmmu/hw/xtensa/sim.o
  CC      x86_64-softmmu/hw/core/null-machine.o
  CC      xtensaeb-softmmu/hw/xtensa/xtfpga.o
  CC      xtensa-softmmu/memory.o
  CC      x86_64-softmmu/hw/display/vga.o
  CC      xtensaeb-softmmu/target/xtensa/xtensa-semi.o
  CC      xtensaeb-softmmu/target/xtensa/core-dc232b.o
  CC      x86_64-softmmu/hw/display/virtio-gpu.o
  CC      xtensa-softmmu/memory_mapping.o
  CC      aarch64-linux-user/tcg/optimize.o
  CC      xtensaeb-softmmu/target/xtensa/core-dc233c.o
  CC      xtensa-softmmu/dump.o
  CC      xtensaeb-softmmu/target/xtensa/core-fsf.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-3d.o
  CC      xtensaeb-softmmu/target/xtensa/monitor.o
  CC      xtensa-softmmu/migration/ram.o
  CC      aarch64-linux-user/tcg/tcg-common.o
  CC      xtensaeb-softmmu/target/xtensa/translate.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-linux-user/tcg/tcg-runtime.o
  CC      xtensa-softmmu/accel/accel.o
  CC      x86_64-softmmu/hw/display/virtio-vga.o
  CC      xtensa-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-linux-user/fpu/softfloat.o
  CC      x86_64-softmmu/hw/intc/apic.o
  CC      xtensa-softmmu/accel/tcg/tcg-all.o
  CC      xtensa-softmmu/accel/tcg/cputlb.o
  CC      xtensaeb-softmmu/target/xtensa/op_helper.o
  CC      x86_64-softmmu/hw/intc/apic_common.o
  CC      xtensaeb-softmmu/target/xtensa/helper.o
  CC      x86_64-softmmu/hw/intc/ioapic.o
  CC      xtensaeb-softmmu/target/xtensa/cpu.o
  CC      x86_64-softmmu/hw/isa/lpc_ich9.o
  CC      xtensa-softmmu/accel/tcg/cpu-exec.o
  CC      xtensaeb-softmmu/target/xtensa/gdbstub.o
  CC      x86_64-softmmu/hw/misc/vmport.o
  GEN     trace/generated-helpers.c
  CC      xtensaeb-softmmu/trace/control-target.o
  CC      xtensa-softmmu/accel/tcg/cpu-exec-common.o
  CC      x86_64-softmmu/hw/misc/ivshmem.o
  CC      aarch64-linux-user/disas.o
  CC      xtensaeb-softmmu/trace/generated-helpers.o
  CC      xtensa-softmmu/accel/tcg/translate-all.o
  CC      x86_64-softmmu/hw/misc/pvpanic.o
  GEN     aarch64-linux-user/gdbstub-xml.c
  LINK    xtensaeb-softmmu/qemu-system-xtensaeb
  CC      aarch64-linux-user/hax-stub.o
  CC      xtensa-softmmu/hw/core/generic-loader.o
  CC      x86_64-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-linux-user/gdbstub.o
  CC      x86_64-softmmu/hw/net/virtio-net.o
  CC      xtensa-softmmu/hw/core/null-machine.o
  CC      xtensa-softmmu/hw/misc/mmio_interface.o
  CC      x86_64-softmmu/hw/net/vhost_net.o
  CC      aarch64-linux-user/thunk.o
  CC      xtensa-softmmu/hw/net/vhost_net.o
  CC      xtensa-softmmu/hw/net/rocker/qmp-norocker.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi.o
  CC      xtensa-softmmu/hw/vfio/common.o
  CC      xtensa-softmmu/hw/vfio/platform.o
  CC      aarch64-linux-user/user-exec.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      aarch64-linux-user/user-exec-stub.o
  CC      x86_64-softmmu/hw/scsi/vhost-scsi-common.o
  CC      xtensa-softmmu/hw/vfio/spapr.o
  CC      x86_64-softmmu/hw/scsi/vhost-scsi.o
  CC      aarch64-linux-user/accel/stubs/kvm-stub.o
  CC      xtensa-softmmu/hw/xtensa/pic_cpu.o
  CC      x86_64-softmmu/hw/scsi/vhost-user-scsi.o
  CC      aarch64-linux-user/accel/tcg/cpu-exec.o
  CC      x86_64-softmmu/hw/timer/mc146818rtc.o
  CC      xtensa-softmmu/hw/xtensa/sim.o
  CC      xtensa-softmmu/hw/xtensa/xtfpga.o
  CC      x86_64-softmmu/hw/vfio/common.o
  CC      xtensa-softmmu/target/xtensa/xtensa-semi.o
  CC      aarch64-linux-user/accel/tcg/cpu-exec-common.o
  CC      x86_64-softmmu/hw/vfio/pci.o
  CC      aarch64-linux-user/accel/tcg/translate-all.o
  CC      xtensa-softmmu/target/xtensa/core-dc232b.o
  CC      xtensa-softmmu/target/xtensa/core-dc233c.o
  CC      aarch64-linux-user/linux-user/main.o
  CC      xtensa-softmmu/target/xtensa/core-fsf.o
  CC      x86_64-softmmu/hw/vfio/pci-quirks.o
  CC      xtensa-softmmu/target/xtensa/monitor.o
  GEN     alpha-linux-user/config-target.h
  CC      alpha-linux-user/exec.o
  CC      aarch64-linux-user/linux-user/syscall.o
  CC      xtensa-softmmu/target/xtensa/translate.o
  CC      x86_64-softmmu/hw/vfio/platform.o
  CC      alpha-linux-user/tcg/tcg.o
  CC      x86_64-softmmu/hw/vfio/spapr.o
  CC      x86_64-softmmu/hw/virtio/virtio.o
  CC      alpha-linux-user/tcg/tcg-op.o
  CC      xtensa-softmmu/target/xtensa/op_helper.o
  CC      x86_64-softmmu/hw/virtio/virtio-balloon.o
  CC      xtensa-softmmu/target/xtensa/helper.o
  CC      aarch64-linux-user/linux-user/strace.o
  CC      x86_64-softmmu/hw/virtio/vhost.o
  CC      xtensa-softmmu/target/xtensa/cpu.o
  CC      aarch64-linux-user/linux-user/mmap.o
  CC      xtensa-softmmu/target/xtensa/gdbstub.o
  CC      alpha-linux-user/tcg/optimize.o
  CC      aarch64-linux-user/linux-user/signal.o
  CC      x86_64-softmmu/hw/virtio/vhost-backend.o
  GEN     trace/generated-helpers.c
  CC      x86_64-softmmu/hw/virtio/vhost-user.o
  CC      xtensa-softmmu/trace/control-target.o
  CC      aarch64-linux-user/linux-user/elfload.o
  CC      xtensa-softmmu/trace/generated-helpers.o
  CC      alpha-linux-user/tcg/tcg-common.o
  CC      x86_64-softmmu/hw/virtio/vhost-vsock.o
  CC      alpha-linux-user/tcg/tcg-runtime.o
  LINK    xtensa-softmmu/qemu-system-xtensa
  CC      aarch64-linux-user/linux-user/linuxload.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto.o
  CC      alpha-linux-user/fpu/softfloat.o
  CC      aarch64-linux-user/linux-user/uaccess.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      alpha-linux-user/disas.o
  CC      aarch64-linux-user/linux-user/uname.o
  CCAS    aarch64-linux-user/linux-user/safe-syscall.o
  GEN     armeb-linux-user/config-target.h
  CC      x86_64-softmmu/hw/i386/multiboot.o
  CC      armeb-linux-user/exec.o
  CC      aarch64-linux-user/linux-user/flatload.o
  CC      x86_64-softmmu/hw/i386/pc.o
  CC      armeb-linux-user/tcg/tcg.o
  CC      aarch64-linux-user/target/arm/arm-semi.o
  CC      aarch64-linux-user/target/arm/kvm-stub.o
  CC      x86_64-softmmu/hw/i386/pc_piix.o
In file included from /var/tmp/patchew-tester-tmp-hm2hdbnw/src/target/arm/kvm-stub.c:15:0:
/var/tmp/patchew-tester-tmp-hm2hdbnw/src/target/arm/kvm_arm.h:291:28: error: unknown type name ‘ram_addr_t’
 void kvm_hwpoison_page_add(ram_addr_t ram_addr);
                            ^~~~~~~~~~
/var/tmp/patchew-tester-tmp-hm2hdbnw/src/rules.mak:66: recipe for target 'target/arm/kvm-stub.o' failed
make[1]: *** [target/arm/kvm-stub.o] Error 1
Makefile:326: recipe for target 'subdir-aarch64-linux-user' failed
make: *** [subdir-aarch64-linux-user] Error 2
make: *** Waiting for unfinished jobs....
  CC      alpha-linux-user/hax-stub.o
  CC      x86_64-softmmu/hw/i386/pc_q35.o
  CC      alpha-linux-user/gdbstub.o
  CC      armeb-linux-user/tcg/tcg-op.o
  CC      x86_64-softmmu/hw/i386/pc_sysfw.o
  CC      x86_64-softmmu/hw/i386/x86-iommu.o
  CC      alpha-linux-user/thunk.o
  CC      armeb-linux-user/tcg/optimize.o
  CC      alpha-linux-user/user-exec.o
  CC      x86_64-softmmu/hw/i386/intel_iommu.o
  CC      alpha-linux-user/user-exec-stub.o
  CC      x86_64-softmmu/hw/i386/amd_iommu.o
  CC      alpha-linux-user/accel/stubs/kvm-stub.o
  CC      alpha-linux-user/accel/tcg/cpu-exec.o
  CC      x86_64-softmmu/hw/i386/kvmvapic.o
  CC      x86_64-softmmu/hw/i386/acpi-build.o
  CC      armeb-linux-user/tcg/tcg-common.o
  CC      alpha-linux-user/accel/tcg/cpu-exec-common.o
  CC      alpha-linux-user/accel/tcg/translate-all.o
  CC      armeb-linux-user/tcg/tcg-runtime.o
  CC      x86_64-softmmu/hw/i386/pci-assign-load-rom.o
  CC      alpha-linux-user/linux-user/main.o
  CC      armeb-linux-user/fpu/softfloat.o
  CC      alpha-linux-user/linux-user/syscall.o
  CC      x86_64-softmmu/target/i386/helper.o
  CC      alpha-linux-user/linux-user/strace.o
  CC      x86_64-softmmu/target/i386/cpu.o
  CC      x86_64-softmmu/target/i386/gdbstub.o
  CC      armeb-linux-user/disas.o
  CC      x86_64-softmmu/target/i386/xsave_helper.o
  CC      x86_64-softmmu/target/i386/translate.o
  GEN     armeb-linux-user/gdbstub-xml.c
  CC      alpha-linux-user/linux-user/mmap.o
  CC      x86_64-softmmu/target/i386/bpt_helper.o
  CC      armeb-linux-user/hax-stub.o
  CC      alpha-linux-user/linux-user/signal.o
  CC      alpha-linux-user/linux-user/elfload.o
  CC      armeb-linux-user/gdbstub.o
  CC      alpha-linux-user/linux-user/linuxload.o
  CC      x86_64-softmmu/target/i386/cc_helper.o
  CC      armeb-linux-user/thunk.o
  CC      alpha-linux-user/linux-user/uaccess.o
  CC      alpha-linux-user/linux-user/uname.o
  CC      armeb-linux-user/user-exec.o
  CCAS    alpha-linux-user/linux-user/safe-syscall.o
  CC      alpha-linux-user/target/alpha/translate.o
  CC      alpha-linux-user/target/alpha/helper.o
  CC      armeb-linux-user/user-exec-stub.o
  CC      x86_64-softmmu/target/i386/excp_helper.o
  CC      armeb-linux-user/accel/stubs/kvm-stub.o
  CC      alpha-linux-user/target/alpha/cpu.o
  CC      armeb-linux-user/accel/tcg/cpu-exec.o
  CC      alpha-linux-user/target/alpha/int_helper.o
  CC      x86_64-softmmu/target/i386/fpu_helper.o
  CC      armeb-linux-user/accel/tcg/cpu-exec-common.o
  CC      alpha-linux-user/target/alpha/fpu_helper.o
  CC      alpha-linux-user/target/alpha/vax_helper.o
  CC      armeb-linux-user/accel/tcg/translate-all.o
  CC      alpha-linux-user/target/alpha/sys_helper.o
  CC      alpha-linux-user/target/alpha/mem_helper.o
  CC      armeb-linux-user/linux-user/main.o
  CC      armeb-linux-user/linux-user/syscall.o
  CC      alpha-linux-user/target/alpha/gdbstub.o
  CC      armeb-linux-user/linux-user/strace.o
  GEN     trace/generated-helpers.c
  CC      alpha-linux-user/trace/control-target.o
  CC      alpha-linux-user/trace/generated-helpers.o
  CC      x86_64-softmmu/target/i386/int_helper.o
  CC      x86_64-softmmu/target/i386/mem_helper.o
  LINK    alpha-linux-user/qemu-alpha
  CC      armeb-linux-user/linux-user/mmap.o
  CC      x86_64-softmmu/target/i386/misc_helper.o
  CC      armeb-linux-user/linux-user/signal.o
  CC      x86_64-softmmu/target/i386/mpx_helper.o
  CC      x86_64-softmmu/target/i386/seg_helper.o
  CC      x86_64-softmmu/target/i386/smm_helper.o
  CC      x86_64-softmmu/target/i386/svm_helper.o
  CC      x86_64-softmmu/target/i386/machine.o
  CC      x86_64-softmmu/target/i386/arch_memory_mapping.o
  CC      x86_64-softmmu/target/i386/arch_dump.o
  CC      x86_64-softmmu/target/i386/monitor.o
  CC      x86_64-softmmu/target/i386/kvm-stub.o
  GEN     trace/generated-helpers.c
  CC      x86_64-softmmu/trace/control-target.o
  CC      x86_64-softmmu/gdbstub-xml.o
  CC      armeb-linux-user/linux-user/elfload.o
  CC      x86_64-softmmu/trace/generated-helpers.o
  CC      armeb-linux-user/linux-user/linuxload.o
  CC      armeb-linux-user/linux-user/uaccess.o
  CC      armeb-linux-user/linux-user/uname.o
  LINK    x86_64-softmmu/qemu-system-x86_64
  CCAS    armeb-linux-user/linux-user/safe-syscall.o
  CC      armeb-linux-user/linux-user/flatload.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11_cpdo.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11_cpdt.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpa11_cprt.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/fpopcode.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/double_cpdo.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/single_cpdo.o
  CC      armeb-linux-user/linux-user/arm/nwfpe/extended_cpdo.o
  CC      armeb-linux-user/target/arm/arm-semi.o
  CC      armeb-linux-user/target/arm/kvm-stub.o
  CC      armeb-linux-user/target/arm/translate.o
In file included from /var/tmp/patchew-tester-tmp-hm2hdbnw/src/target/arm/kvm-stub.c:15:0:
/var/tmp/patchew-tester-tmp-hm2hdbnw/src/target/arm/kvm_arm.h:291:28: error: unknown type name ‘ram_addr_t’
 void kvm_hwpoison_page_add(ram_addr_t ram_addr);
                            ^~~~~~~~~~
/var/tmp/patchew-tester-tmp-hm2hdbnw/src/rules.mak:66: recipe for target 'target/arm/kvm-stub.o' failed
make[1]: *** [target/arm/kvm-stub.o] Error 1
make[1]: *** Waiting for unfinished jobs....
Makefile:326: recipe for target 'subdir-armeb-linux-user' failed
make: *** [subdir-armeb-linux-user] Error 2
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
@ 2017-08-18 14:23 ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

In the armv8 platform, the mainly processor hardware error notification
type are synchronous external abort(SEA) and SError Interrupt (SEI), For
the ARMv8 SEA/SEI, KVM or host kernel will deliver SIGBUS or use other
interface to notify user space. After user space gets the notification,
it will record the CPER to simulate GHES for guest OS and inject the a
exception(SEA/SEI) to KVM. 

This series patch has two parts, one part handles synchronous external
abort(SEA) exception and SError Interrupt (SEI) exception; another part
is generating APEI table when guest OS boot up, and dynamically record
CPER for the guest OS about the generic hardware errors. Currently the
userspace only handles the memory section hardware errors. Before Qemu
record the CPER, it needs to check the ACK value written by the guest
OS to avoid read-write race condition. In the simulated APEI/GHESV2/CPER
table, the max number of error soure is 11, which is classified by
notification type, now only enable the SEA/SEI notification type error
source to avoid OS boot warning.


About the whole solution we ever discuessed it in here before:
https://patchwork.kernel.org/patch/9633105/

Below is the APEI/GHESV2/CPER table layout, the max number of error soure is 11:

       etc/acpi/tables                               etc/hardware_errors
    ====================                    ==========================================
+ +--------------------------+            +------------------+
| | HEST                     |            |    address       |              +--------------+
| +--------------------------+            |    registers     |              | Error Status |
| | GHES0                    |            | +----------------+              | Data Block 0 |
| +--------------------------+ +--------->| |status_address0 |------------->| +------------+
| | .................        | |          | +----------------+              | |  CPER      |
| | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
| | .................        |   |        | +----------------+          |   | |  ....      |
| | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
| | read_ack_preserve        | | |        +------------------+          |   | +------------+
| | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
+ +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
| | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
+ +--------------------------+   | |      | +----------------+        |     | |  CPER      |
| | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
| | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
| | .................        |     | |    | |  ............. |        |     | |  CPER      |
| | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
| | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
| | read_ack_write           |     |   |  | +----------------+        |     | +------------+
+ +--------------------------|     |   |                              |     | Error Status |
| | ...............          |     |   |                              |     | Data Block 10|
+ +--------------------------+     |   |                              +---->| +------------+
| | GHES10                   |     |   |                                    | |  CPER      |
+ +--------------------------+     |   |                                    | |  CPER      |
| | .................        |     |   |                                    | |  ....      |
| | error_status_address-----+-----+   |                                    | |  CPER      |
| | .................        |         |                                    +-+------------+
| | read_ack_register--------+---------+
| | read_ack_preserve        |
| | read_ack_write           |
+ +--------------------------+


----------------------------------------------------------------------------------------------
How to test guest OS do SEA/SEI recovery:

1. In the guest OS, trigger a SEA or SEI.
2. Then you will see below error log that printed by the memory failure
3. Memory failure will do the recovery for the error.

Such as the below shown kernel log:
[   21.101216] Synchronous External Abort: synchronous external abort (0x96000010) at 0xffffff8008064018
[   21.104969] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 8
[   21.106918] {1}[Hardware Error]: event severity: recoverable
[   21.109027] {1}[Hardware Error]:  Error 0, type: recoverable
[   21.110362] {1}[Hardware Error]:   section_type: memory error
[   21.111705] {1}[Hardware Error]:   physical_address: 0x000000007a200000
[   21.113255] {1}[Hardware Error]:   error_type: 3, multi-bit ECC
[   21.118528] Internal error: : 96000010 [#1] SMP
[   21.119587] Modules linked in:
[   21.120307] CPU: 0 PID: 509 Comm: devmem Not tainted 4.12.0-rc4ajb-00990-g954379b-dirty #67
[   21.122307] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[   21.123915] task: ffffffc03da32900 task.stack: ffffffc03dbbc000
[   21.125302] PC is at __do_user_fault+0x58/0x110
[   21.126370] LR is at __do_user_fault+0x54/0x110
[   21.127433] pc : [<ffffff8008097528>] lr : [<ffffff8008097524>] pstate: 80000145
[   21.129164] sp : ffffffc03dbbfd20
[   21.129940] x29: ffffffc03dbbfd20 x28: ffffffc03da32900
[   21.131204] x27: 0000000000000000 x26: 0000007f7edc5001
[   21.132439] x25: ffffff8008648438 x24: ffffffc03dbbfec0
[   21.133689] x23: 0000000000030001 x22: 0000007f7edc5001
[   21.134934] x21: 0000000000000007 x20: 0000000092000021
[   21.136195] x19: ffffffc03da32900 x18: 0000007fdd4c18f0
[   21.137439] x17: 0000007f7ecb9ebc x16: 0000000000412058

------------------------------------------------------------------------------------------------
how to test guest OS APTI/GHES:
1. In the guest OS, use this command to dump the APEI table: 
    "iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST"
2. And find the address for the generic error status block
   according to the notification type
3. then find the CPER record through the generic error status block.

For example(notification type is SEA):

(1) root@genericarmv8:~# iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST
(2) root@genericarmv8:~# cat HEST.dsl
    /*
     * Intel ACPI Component Architecture
     * AML/ASL+ Disassembler version 20170728 (64-bit version)
     * Copyright (c) 2000 - 2017 Intel Corporation
     *
     * Disassembly of /sys/firmware/acpi/tables/HEST, Mon Sep  5 07:59:17 2016
     *
     * ACPI Data Table [HEST]
     *
     * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
     */

    ..................................................................................
    [308h 0776   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
    [30Ah 0778   2]                    Source Id : 0008
    [30Ch 0780   2]            Related Source Id : FFFF
    [30Eh 0782   1]                     Reserved : 00
    [30Fh 0783   1]                      Enabled : 01
    [310h 0784   4]       Records To Preallocate : 00000001
    [314h 0788   4]      Max Sections Per Record : 00000001
    [318h 0792   4]          Max Raw Data Length : 00001000

    [31Ch 0796  12]         Error Status Address : [Generic Address Structure]
    [31Ch 0796   1]                     Space ID : 00 [SystemMemory]
    [31Dh 0797   1]                    Bit Width : 40
    [31Eh 0798   1]                   Bit Offset : 00
    [31Fh 0799   1]         Encoded Access Width : 04 [QWord Access:64]
    [320h 0800   8]                      Address : 00000000785D0040

    [328h 0808  28]                       Notify : [Hardware Error Notification Structure]
    [328h 0808   1]                  Notify Type : 08 [SEA]
    [329h 0809   1]                Notify Length : 1C
    [32Ah 0810   2]   Configuration Write Enable : 0000
    [32Ch 0812   4]                 PollInterval : 00000000
    [330h 0816   4]                       Vector : 00000000
    [334h 0820   4]      Polling Threshold Value : 00000000
    [338h 0824   4]     Polling Threshold Window : 00000000
    [33Ch 0828   4]        Error Threshold Value : 00000000
    [340h 0832   4]       Error Threshold Window : 00000000

    [344h 0836   4]    Error Status Block Length : 00001000
    [348h 0840  12]            Read Ack Register : [Generic Address Structure]
    [348h 0840   1]                     Space ID : 00 [SystemMemory]
    [349h 0841   1]                    Bit Width : 40
    [34Ah 0842   1]                   Bit Offset : 00
    [34Bh 0843   1]         Encoded Access Width : 04 [QWord Access:64]
    [34Ch 0844   8]                      Address : 00000000785D0098

    [354h 0852   8]            Read Ack Preserve : 00000000FFFFFFFE
    [35Ch 0860   8]               Read Ack Write : 0000000000000001

    [364h 0868   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
    [366h 0870   2]                    Source Id : 0009
    [368h 0872   2]            Related Source Id : FFFF
    [36Ah 0874   1]                     Reserved : 00
    [36Bh 0875   1]                      Enabled : 01
    [36Ch 0876   4]       Records To Preallocate : 00000001
    [370h 0880   4]      Max Sections Per Record : 00000001
    [374h 0884   4]          Max Raw Data Length : 00001000

    [378h 0888  12]         Error Status Address : [Generic Address Structure]
    [378h 0888   1]                     Space ID : 00 [SystemMemory]
    [379h 0889   1]                    Bit Width : 40
    [37Ah 0890   1]                   Bit Offset : 00
    [37Bh 0891   1]         Encoded Access Width : 04 [QWord Access:64]
    [37Ch 0892   8]                      Address : 00000000785D0048

    [384h 0900  28]                       Notify : [Hardware Error Notification Structure]
    [384h 0900   1]                  Notify Type : 09 [SEI]
    [385h 0901   1]                Notify Length : 1C
    [386h 0902   2]   Configuration Write Enable : 0000
    [388h 0904   4]                 PollInterval : 00000000
    [38Ch 0908   4]                       Vector : 00000000
    [390h 0912   4]      Polling Threshold Value : 00000000
    [394h 0916   4]     Polling Threshold Window : 00000000
    [398h 0920   4]        Error Threshold Value : 00000000
    [39Ch 0924   4]       Error Threshold Window : 00000000

    [3A0h 0928   4]    Error Status Block Length : 00001000
    [3A4h 0932  12]            Read Ack Register : [Generic Address Structure]
    [3A4h 0932   1]                     Space ID : 00 [SystemMemory]
    [3A5h 0933   1]                    Bit Width : 40
    [3A6h 0934   1]                   Bit Offset : 00
    [3A7h 0935   1]         Encoded Access Width : 04 [QWord Access:64]
    [3A8h 0936   8]                      Address : 00000000785D00A0

    [3B0h 0944   8]            Read Ack Preserve : 00000000FFFFFFFE
    [3B8h 0952   8]               Read Ack Write : 0000000000000001
    .....................................................................................
(3) according to above table, the address that contains the physical address of a block
    of memory that holds the error status data for SEA notification error source is 0x00000000785D0040
(4) the address for SEA notification error source is 0x785d8108
    (qemu) xp /1 0x00000000785D0040
    00000000785d0040: 0x785d80b0

(5) check the content of generic error status block and generic error data entry
    (qemu) xp /100x 0x785d80b0
    00000000785d80b0: 0x00000001 0x00000000 0x00000000 0x00000098
    00000000785d80c0: 0x00000000 0xa5bc1114 0x4ede6f64 0x833e63b8
    00000000785d80d0: 0xb1837ced 0x00000000 0x00000300 0x00000050
    00000000785d80e0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d80f0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8100: 0x00000000 0x00000000 0x00000000 0x00004002
    00000000785d8110: 0x00000000 0x00000000 0x00000000 0x00001111
    00000000785d8120: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8130: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8140: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8150: 0x00000000 0x00000003 0x00000000 0x00000000
    00000000785d8160: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8170: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8180: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8190: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81a0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81b0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81c0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81d0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81e0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81f0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8200: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8210: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8220: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8230: 0x00000000 0x00000000 0x00000000 0x00000000
(6) check the OSPM's ACK value(for example SEA)
    /* Before OSPM acknowledges the error, check the ACK value */
    (qemu) xp /1 0x00000000785D0098
    00000000785d00f0: 0x00000000

    /* After OSPM acknowledges the error, check the ACK value */
    (qemu) xp /1 0x00000000785D0098
    00000000785d00f0: 0x00000001

Dongjiu Geng (6):
  ACPI: add APEI/HEST/CPER structures and macros
  ACPI: Add APEI GHES Table Generation support
  ACPI: build and enable APEI GHES in the Makefile and configuration
  target-arm: kvm64: detect guest RAS EXTENSION feature
  target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
  target-arm: kvm64: Handle SError interrupt from the guest OS

 default-configs/arm-softmmu.mak |   1 +
 hw/acpi/Makefile.objs           |   1 +
 hw/acpi/aml-build.c             |   2 +
 hw/acpi/hest_ghes.c             | 345 ++++++++++++++++++++++++++++++++++++++++
 hw/arm/virt-acpi-build.c        |   6 +
 include/hw/acpi/acpi-defs.h     | 193 ++++++++++++++++++++++
 include/hw/acpi/aml-build.h     |   1 +
 include/hw/acpi/hest_ghes.h     |  47 ++++++
 include/sysemu/kvm.h            |   2 +-
 linux-headers/asm-arm64/kvm.h   |   5 +
 linux-headers/linux/kvm.h       |   2 +
 target/arm/cpu.h                |   3 +
 target/arm/internals.h          |  14 ++
 target/arm/kvm.c                |  34 ++++
 target/arm/kvm64.c              | 186 ++++++++++++++++++++++
 target/arm/kvm_arm.h            |   1 +
 16 files changed, 842 insertions(+), 1 deletion(-)
 create mode 100644 hw/acpi/hest_ghes.c
 create mode 100644 include/hw/acpi/hest_ghes.h

-- 
1.8.3.1

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
@ 2017-08-18 14:23 ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

In the armv8 platform, the mainly processor hardware error notification
type are synchronous external abort(SEA) and SError Interrupt (SEI), For
the ARMv8 SEA/SEI, KVM or host kernel will deliver SIGBUS or use other
interface to notify user space. After user space gets the notification,
it will record the CPER to simulate GHES for guest OS and inject the a
exception(SEA/SEI) to KVM. 

This series patch has two parts, one part handles synchronous external
abort(SEA) exception and SError Interrupt (SEI) exception; another part
is generating APEI table when guest OS boot up, and dynamically record
CPER for the guest OS about the generic hardware errors. Currently the
userspace only handles the memory section hardware errors. Before Qemu
record the CPER, it needs to check the ACK value written by the guest
OS to avoid read-write race condition. In the simulated APEI/GHESV2/CPER
table, the max number of error soure is 11, which is classified by
notification type, now only enable the SEA/SEI notification type error
source to avoid OS boot warning.


About the whole solution we ever discuessed it in here before:
https://patchwork.kernel.org/patch/9633105/

Below is the APEI/GHESV2/CPER table layout, the max number of error soure is 11:

       etc/acpi/tables                               etc/hardware_errors
    ====================                    ==========================================
+ +--------------------------+            +------------------+
| | HEST                     |            |    address       |              +--------------+
| +--------------------------+            |    registers     |              | Error Status |
| | GHES0                    |            | +----------------+              | Data Block 0 |
| +--------------------------+ +--------->| |status_address0 |------------->| +------------+
| | .................        | |          | +----------------+              | |  CPER      |
| | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
| | .................        |   |        | +----------------+          |   | |  ....      |
| | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
| | read_ack_preserve        | | |        +------------------+          |   | +------------+
| | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
+ +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
| | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
+ +--------------------------+   | |      | +----------------+        |     | |  CPER      |
| | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
| | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
| | .................        |     | |    | |  ............. |        |     | |  CPER      |
| | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
| | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
| | read_ack_write           |     |   |  | +----------------+        |     | +------------+
+ +--------------------------|     |   |                              |     | Error Status |
| | ...............          |     |   |                              |     | Data Block 10|
+ +--------------------------+     |   |                              +---->| +------------+
| | GHES10                   |     |   |                                    | |  CPER      |
+ +--------------------------+     |   |                                    | |  CPER      |
| | .................        |     |   |                                    | |  ....      |
| | error_status_address-----+-----+   |                                    | |  CPER      |
| | .................        |         |                                    +-+------------+
| | read_ack_register--------+---------+
| | read_ack_preserve        |
| | read_ack_write           |
+ +--------------------------+


----------------------------------------------------------------------------------------------
How to test guest OS do SEA/SEI recovery:

1. In the guest OS, trigger a SEA or SEI.
2. Then you will see below error log that printed by the memory failure
3. Memory failure will do the recovery for the error.

Such as the below shown kernel log:
[   21.101216] Synchronous External Abort: synchronous external abort (0x96000010) at 0xffffff8008064018
[   21.104969] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 8
[   21.106918] {1}[Hardware Error]: event severity: recoverable
[   21.109027] {1}[Hardware Error]:  Error 0, type: recoverable
[   21.110362] {1}[Hardware Error]:   section_type: memory error
[   21.111705] {1}[Hardware Error]:   physical_address: 0x000000007a200000
[   21.113255] {1}[Hardware Error]:   error_type: 3, multi-bit ECC
[   21.118528] Internal error: : 96000010 [#1] SMP
[   21.119587] Modules linked in:
[   21.120307] CPU: 0 PID: 509 Comm: devmem Not tainted 4.12.0-rc4ajb-00990-g954379b-dirty #67
[   21.122307] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[   21.123915] task: ffffffc03da32900 task.stack: ffffffc03dbbc000
[   21.125302] PC is at __do_user_fault+0x58/0x110
[   21.126370] LR is at __do_user_fault+0x54/0x110
[   21.127433] pc : [<ffffff8008097528>] lr : [<ffffff8008097524>] pstate: 80000145
[   21.129164] sp : ffffffc03dbbfd20
[   21.129940] x29: ffffffc03dbbfd20 x28: ffffffc03da32900
[   21.131204] x27: 0000000000000000 x26: 0000007f7edc5001
[   21.132439] x25: ffffff8008648438 x24: ffffffc03dbbfec0
[   21.133689] x23: 0000000000030001 x22: 0000007f7edc5001
[   21.134934] x21: 0000000000000007 x20: 0000000092000021
[   21.136195] x19: ffffffc03da32900 x18: 0000007fdd4c18f0
[   21.137439] x17: 0000007f7ecb9ebc x16: 0000000000412058

------------------------------------------------------------------------------------------------
how to test guest OS APTI/GHES:
1. In the guest OS, use this command to dump the APEI table: 
    "iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST"
2. And find the address for the generic error status block
   according to the notification type
3. then find the CPER record through the generic error status block.

For example(notification type is SEA):

(1) root@genericarmv8:~# iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST
(2) root@genericarmv8:~# cat HEST.dsl
    /*
     * Intel ACPI Component Architecture
     * AML/ASL+ Disassembler version 20170728 (64-bit version)
     * Copyright (c) 2000 - 2017 Intel Corporation
     *
     * Disassembly of /sys/firmware/acpi/tables/HEST, Mon Sep  5 07:59:17 2016
     *
     * ACPI Data Table [HEST]
     *
     * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
     */

    ..................................................................................
    [308h 0776   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
    [30Ah 0778   2]                    Source Id : 0008
    [30Ch 0780   2]            Related Source Id : FFFF
    [30Eh 0782   1]                     Reserved : 00
    [30Fh 0783   1]                      Enabled : 01
    [310h 0784   4]       Records To Preallocate : 00000001
    [314h 0788   4]      Max Sections Per Record : 00000001
    [318h 0792   4]          Max Raw Data Length : 00001000

    [31Ch 0796  12]         Error Status Address : [Generic Address Structure]
    [31Ch 0796   1]                     Space ID : 00 [SystemMemory]
    [31Dh 0797   1]                    Bit Width : 40
    [31Eh 0798   1]                   Bit Offset : 00
    [31Fh 0799   1]         Encoded Access Width : 04 [QWord Access:64]
    [320h 0800   8]                      Address : 00000000785D0040

    [328h 0808  28]                       Notify : [Hardware Error Notification Structure]
    [328h 0808   1]                  Notify Type : 08 [SEA]
    [329h 0809   1]                Notify Length : 1C
    [32Ah 0810   2]   Configuration Write Enable : 0000
    [32Ch 0812   4]                 PollInterval : 00000000
    [330h 0816   4]                       Vector : 00000000
    [334h 0820   4]      Polling Threshold Value : 00000000
    [338h 0824   4]     Polling Threshold Window : 00000000
    [33Ch 0828   4]        Error Threshold Value : 00000000
    [340h 0832   4]       Error Threshold Window : 00000000

    [344h 0836   4]    Error Status Block Length : 00001000
    [348h 0840  12]            Read Ack Register : [Generic Address Structure]
    [348h 0840   1]                     Space ID : 00 [SystemMemory]
    [349h 0841   1]                    Bit Width : 40
    [34Ah 0842   1]                   Bit Offset : 00
    [34Bh 0843   1]         Encoded Access Width : 04 [QWord Access:64]
    [34Ch 0844   8]                      Address : 00000000785D0098

    [354h 0852   8]            Read Ack Preserve : 00000000FFFFFFFE
    [35Ch 0860   8]               Read Ack Write : 0000000000000001

    [364h 0868   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
    [366h 0870   2]                    Source Id : 0009
    [368h 0872   2]            Related Source Id : FFFF
    [36Ah 0874   1]                     Reserved : 00
    [36Bh 0875   1]                      Enabled : 01
    [36Ch 0876   4]       Records To Preallocate : 00000001
    [370h 0880   4]      Max Sections Per Record : 00000001
    [374h 0884   4]          Max Raw Data Length : 00001000

    [378h 0888  12]         Error Status Address : [Generic Address Structure]
    [378h 0888   1]                     Space ID : 00 [SystemMemory]
    [379h 0889   1]                    Bit Width : 40
    [37Ah 0890   1]                   Bit Offset : 00
    [37Bh 0891   1]         Encoded Access Width : 04 [QWord Access:64]
    [37Ch 0892   8]                      Address : 00000000785D0048

    [384h 0900  28]                       Notify : [Hardware Error Notification Structure]
    [384h 0900   1]                  Notify Type : 09 [SEI]
    [385h 0901   1]                Notify Length : 1C
    [386h 0902   2]   Configuration Write Enable : 0000
    [388h 0904   4]                 PollInterval : 00000000
    [38Ch 0908   4]                       Vector : 00000000
    [390h 0912   4]      Polling Threshold Value : 00000000
    [394h 0916   4]     Polling Threshold Window : 00000000
    [398h 0920   4]        Error Threshold Value : 00000000
    [39Ch 0924   4]       Error Threshold Window : 00000000

    [3A0h 0928   4]    Error Status Block Length : 00001000
    [3A4h 0932  12]            Read Ack Register : [Generic Address Structure]
    [3A4h 0932   1]                     Space ID : 00 [SystemMemory]
    [3A5h 0933   1]                    Bit Width : 40
    [3A6h 0934   1]                   Bit Offset : 00
    [3A7h 0935   1]         Encoded Access Width : 04 [QWord Access:64]
    [3A8h 0936   8]                      Address : 00000000785D00A0

    [3B0h 0944   8]            Read Ack Preserve : 00000000FFFFFFFE
    [3B8h 0952   8]               Read Ack Write : 0000000000000001
    .....................................................................................
(3) according to above table, the address that contains the physical address of a block
    of memory that holds the error status data for SEA notification error source is 0x00000000785D0040
(4) the address for SEA notification error source is 0x785d8108
    (qemu) xp /1 0x00000000785D0040
    00000000785d0040: 0x785d80b0

(5) check the content of generic error status block and generic error data entry
    (qemu) xp /100x 0x785d80b0
    00000000785d80b0: 0x00000001 0x00000000 0x00000000 0x00000098
    00000000785d80c0: 0x00000000 0xa5bc1114 0x4ede6f64 0x833e63b8
    00000000785d80d0: 0xb1837ced 0x00000000 0x00000300 0x00000050
    00000000785d80e0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d80f0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8100: 0x00000000 0x00000000 0x00000000 0x00004002
    00000000785d8110: 0x00000000 0x00000000 0x00000000 0x00001111
    00000000785d8120: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8130: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8140: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8150: 0x00000000 0x00000003 0x00000000 0x00000000
    00000000785d8160: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8170: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8180: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8190: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81a0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81b0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81c0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81d0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81e0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81f0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8200: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8210: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8220: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8230: 0x00000000 0x00000000 0x00000000 0x00000000
(6) check the OSPM's ACK value(for example SEA)
    /* Before OSPM acknowledges the error, check the ACK value */
    (qemu) xp /1 0x00000000785D0098
    00000000785d00f0: 0x00000000

    /* After OSPM acknowledges the error, check the ACK value */
    (qemu) xp /1 0x00000000785D0098
    00000000785d00f0: 0x00000001

Dongjiu Geng (6):
  ACPI: add APEI/HEST/CPER structures and macros
  ACPI: Add APEI GHES Table Generation support
  ACPI: build and enable APEI GHES in the Makefile and configuration
  target-arm: kvm64: detect guest RAS EXTENSION feature
  target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
  target-arm: kvm64: Handle SError interrupt from the guest OS

 default-configs/arm-softmmu.mak |   1 +
 hw/acpi/Makefile.objs           |   1 +
 hw/acpi/aml-build.c             |   2 +
 hw/acpi/hest_ghes.c             | 345 ++++++++++++++++++++++++++++++++++++++++
 hw/arm/virt-acpi-build.c        |   6 +
 include/hw/acpi/acpi-defs.h     | 193 ++++++++++++++++++++++
 include/hw/acpi/aml-build.h     |   1 +
 include/hw/acpi/hest_ghes.h     |  47 ++++++
 include/sysemu/kvm.h            |   2 +-
 linux-headers/asm-arm64/kvm.h   |   5 +
 linux-headers/linux/kvm.h       |   2 +
 target/arm/cpu.h                |   3 +
 target/arm/internals.h          |  14 ++
 target/arm/kvm.c                |  34 ++++
 target/arm/kvm64.c              | 186 ++++++++++++++++++++++
 target/arm/kvm_arm.h            |   1 +
 16 files changed, 842 insertions(+), 1 deletion(-)
 create mode 100644 hw/acpi/hest_ghes.c
 create mode 100644 include/hw/acpi/hest_ghes.h

-- 
1.8.3.1

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [Qemu-devel] [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
@ 2017-08-18 14:23 ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

In the armv8 platform, the mainly processor hardware error notification
type are synchronous external abort(SEA) and SError Interrupt (SEI), For
the ARMv8 SEA/SEI, KVM or host kernel will deliver SIGBUS or use other
interface to notify user space. After user space gets the notification,
it will record the CPER to simulate GHES for guest OS and inject the a
exception(SEA/SEI) to KVM. 

This series patch has two parts, one part handles synchronous external
abort(SEA) exception and SError Interrupt (SEI) exception; another part
is generating APEI table when guest OS boot up, and dynamically record
CPER for the guest OS about the generic hardware errors. Currently the
userspace only handles the memory section hardware errors. Before Qemu
record the CPER, it needs to check the ACK value written by the guest
OS to avoid read-write race condition. In the simulated APEI/GHESV2/CPER
table, the max number of error soure is 11, which is classified by
notification type, now only enable the SEA/SEI notification type error
source to avoid OS boot warning.


About the whole solution we ever discuessed it in here before:
https://patchwork.kernel.org/patch/9633105/

Below is the APEI/GHESV2/CPER table layout, the max number of error soure is 11:

       etc/acpi/tables                               etc/hardware_errors
    ====================                    ==========================================
+ +--------------------------+            +------------------+
| | HEST                     |            |    address       |              +--------------+
| +--------------------------+            |    registers     |              | Error Status |
| | GHES0                    |            | +----------------+              | Data Block 0 |
| +--------------------------+ +--------->| |status_address0 |------------->| +------------+
| | .................        | |          | +----------------+              | |  CPER      |
| | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
| | .................        |   |        | +----------------+          |   | |  ....      |
| | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
| | read_ack_preserve        | | |        +------------------+          |   | +------------+
| | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
+ +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
| | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
+ +--------------------------+   | |      | +----------------+        |     | |  CPER      |
| | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
| | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
| | .................        |     | |    | |  ............. |        |     | |  CPER      |
| | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
| | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
| | read_ack_write           |     |   |  | +----------------+        |     | +------------+
+ +--------------------------|     |   |                              |     | Error Status |
| | ...............          |     |   |                              |     | Data Block 10|
+ +--------------------------+     |   |                              +---->| +------------+
| | GHES10                   |     |   |                                    | |  CPER      |
+ +--------------------------+     |   |                                    | |  CPER      |
| | .................        |     |   |                                    | |  ....      |
| | error_status_address-----+-----+   |                                    | |  CPER      |
| | .................        |         |                                    +-+------------+
| | read_ack_register--------+---------+
| | read_ack_preserve        |
| | read_ack_write           |
+ +--------------------------+


----------------------------------------------------------------------------------------------
How to test guest OS do SEA/SEI recovery:

1. In the guest OS, trigger a SEA or SEI.
2. Then you will see below error log that printed by the memory failure
3. Memory failure will do the recovery for the error.

Such as the below shown kernel log:
[   21.101216] Synchronous External Abort: synchronous external abort (0x96000010) at 0xffffff8008064018
[   21.104969] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 8
[   21.106918] {1}[Hardware Error]: event severity: recoverable
[   21.109027] {1}[Hardware Error]:  Error 0, type: recoverable
[   21.110362] {1}[Hardware Error]:   section_type: memory error
[   21.111705] {1}[Hardware Error]:   physical_address: 0x000000007a200000
[   21.113255] {1}[Hardware Error]:   error_type: 3, multi-bit ECC
[   21.118528] Internal error: : 96000010 [#1] SMP
[   21.119587] Modules linked in:
[   21.120307] CPU: 0 PID: 509 Comm: devmem Not tainted 4.12.0-rc4ajb-00990-g954379b-dirty #67
[   21.122307] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[   21.123915] task: ffffffc03da32900 task.stack: ffffffc03dbbc000
[   21.125302] PC is at __do_user_fault+0x58/0x110
[   21.126370] LR is at __do_user_fault+0x54/0x110
[   21.127433] pc : [<ffffff8008097528>] lr : [<ffffff8008097524>] pstate: 80000145
[   21.129164] sp : ffffffc03dbbfd20
[   21.129940] x29: ffffffc03dbbfd20 x28: ffffffc03da32900
[   21.131204] x27: 0000000000000000 x26: 0000007f7edc5001
[   21.132439] x25: ffffff8008648438 x24: ffffffc03dbbfec0
[   21.133689] x23: 0000000000030001 x22: 0000007f7edc5001
[   21.134934] x21: 0000000000000007 x20: 0000000092000021
[   21.136195] x19: ffffffc03da32900 x18: 0000007fdd4c18f0
[   21.137439] x17: 0000007f7ecb9ebc x16: 0000000000412058

------------------------------------------------------------------------------------------------
how to test guest OS APTI/GHES:
1. In the guest OS, use this command to dump the APEI table: 
    "iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST"
2. And find the address for the generic error status block
   according to the notification type
3. then find the CPER record through the generic error status block.

For example(notification type is SEA):

(1) root@genericarmv8:~# iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST
(2) root@genericarmv8:~# cat HEST.dsl
    /*
     * Intel ACPI Component Architecture
     * AML/ASL+ Disassembler version 20170728 (64-bit version)
     * Copyright (c) 2000 - 2017 Intel Corporation
     *
     * Disassembly of /sys/firmware/acpi/tables/HEST, Mon Sep  5 07:59:17 2016
     *
     * ACPI Data Table [HEST]
     *
     * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
     */

    ..................................................................................
    [308h 0776   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
    [30Ah 0778   2]                    Source Id : 0008
    [30Ch 0780   2]            Related Source Id : FFFF
    [30Eh 0782   1]                     Reserved : 00
    [30Fh 0783   1]                      Enabled : 01
    [310h 0784   4]       Records To Preallocate : 00000001
    [314h 0788   4]      Max Sections Per Record : 00000001
    [318h 0792   4]          Max Raw Data Length : 00001000

    [31Ch 0796  12]         Error Status Address : [Generic Address Structure]
    [31Ch 0796   1]                     Space ID : 00 [SystemMemory]
    [31Dh 0797   1]                    Bit Width : 40
    [31Eh 0798   1]                   Bit Offset : 00
    [31Fh 0799   1]         Encoded Access Width : 04 [QWord Access:64]
    [320h 0800   8]                      Address : 00000000785D0040

    [328h 0808  28]                       Notify : [Hardware Error Notification Structure]
    [328h 0808   1]                  Notify Type : 08 [SEA]
    [329h 0809   1]                Notify Length : 1C
    [32Ah 0810   2]   Configuration Write Enable : 0000
    [32Ch 0812   4]                 PollInterval : 00000000
    [330h 0816   4]                       Vector : 00000000
    [334h 0820   4]      Polling Threshold Value : 00000000
    [338h 0824   4]     Polling Threshold Window : 00000000
    [33Ch 0828   4]        Error Threshold Value : 00000000
    [340h 0832   4]       Error Threshold Window : 00000000

    [344h 0836   4]    Error Status Block Length : 00001000
    [348h 0840  12]            Read Ack Register : [Generic Address Structure]
    [348h 0840   1]                     Space ID : 00 [SystemMemory]
    [349h 0841   1]                    Bit Width : 40
    [34Ah 0842   1]                   Bit Offset : 00
    [34Bh 0843   1]         Encoded Access Width : 04 [QWord Access:64]
    [34Ch 0844   8]                      Address : 00000000785D0098

    [354h 0852   8]            Read Ack Preserve : 00000000FFFFFFFE
    [35Ch 0860   8]               Read Ack Write : 0000000000000001

    [364h 0868   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
    [366h 0870   2]                    Source Id : 0009
    [368h 0872   2]            Related Source Id : FFFF
    [36Ah 0874   1]                     Reserved : 00
    [36Bh 0875   1]                      Enabled : 01
    [36Ch 0876   4]       Records To Preallocate : 00000001
    [370h 0880   4]      Max Sections Per Record : 00000001
    [374h 0884   4]          Max Raw Data Length : 00001000

    [378h 0888  12]         Error Status Address : [Generic Address Structure]
    [378h 0888   1]                     Space ID : 00 [SystemMemory]
    [379h 0889   1]                    Bit Width : 40
    [37Ah 0890   1]                   Bit Offset : 00
    [37Bh 0891   1]         Encoded Access Width : 04 [QWord Access:64]
    [37Ch 0892   8]                      Address : 00000000785D0048

    [384h 0900  28]                       Notify : [Hardware Error Notification Structure]
    [384h 0900   1]                  Notify Type : 09 [SEI]
    [385h 0901   1]                Notify Length : 1C
    [386h 0902   2]   Configuration Write Enable : 0000
    [388h 0904   4]                 PollInterval : 00000000
    [38Ch 0908   4]                       Vector : 00000000
    [390h 0912   4]      Polling Threshold Value : 00000000
    [394h 0916   4]     Polling Threshold Window : 00000000
    [398h 0920   4]        Error Threshold Value : 00000000
    [39Ch 0924   4]       Error Threshold Window : 00000000

    [3A0h 0928   4]    Error Status Block Length : 00001000
    [3A4h 0932  12]            Read Ack Register : [Generic Address Structure]
    [3A4h 0932   1]                     Space ID : 00 [SystemMemory]
    [3A5h 0933   1]                    Bit Width : 40
    [3A6h 0934   1]                   Bit Offset : 00
    [3A7h 0935   1]         Encoded Access Width : 04 [QWord Access:64]
    [3A8h 0936   8]                      Address : 00000000785D00A0

    [3B0h 0944   8]            Read Ack Preserve : 00000000FFFFFFFE
    [3B8h 0952   8]               Read Ack Write : 0000000000000001
    .....................................................................................
(3) according to above table, the address that contains the physical address of a block
    of memory that holds the error status data for SEA notification error source is 0x00000000785D0040
(4) the address for SEA notification error source is 0x785d8108
    (qemu) xp /1 0x00000000785D0040
    00000000785d0040: 0x785d80b0

(5) check the content of generic error status block and generic error data entry
    (qemu) xp /100x 0x785d80b0
    00000000785d80b0: 0x00000001 0x00000000 0x00000000 0x00000098
    00000000785d80c0: 0x00000000 0xa5bc1114 0x4ede6f64 0x833e63b8
    00000000785d80d0: 0xb1837ced 0x00000000 0x00000300 0x00000050
    00000000785d80e0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d80f0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8100: 0x00000000 0x00000000 0x00000000 0x00004002
    00000000785d8110: 0x00000000 0x00000000 0x00000000 0x00001111
    00000000785d8120: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8130: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8140: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8150: 0x00000000 0x00000003 0x00000000 0x00000000
    00000000785d8160: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8170: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8180: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8190: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81a0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81b0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81c0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81d0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81e0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81f0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8200: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8210: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8220: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8230: 0x00000000 0x00000000 0x00000000 0x00000000
(6) check the OSPM's ACK value(for example SEA)
    /* Before OSPM acknowledges the error, check the ACK value */
    (qemu) xp /1 0x00000000785D0098
    00000000785d00f0: 0x00000000

    /* After OSPM acknowledges the error, check the ACK value */
    (qemu) xp /1 0x00000000785D0098
    00000000785d00f0: 0x00000001

Dongjiu Geng (6):
  ACPI: add APEI/HEST/CPER structures and macros
  ACPI: Add APEI GHES Table Generation support
  ACPI: build and enable APEI GHES in the Makefile and configuration
  target-arm: kvm64: detect guest RAS EXTENSION feature
  target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
  target-arm: kvm64: Handle SError interrupt from the guest OS

 default-configs/arm-softmmu.mak |   1 +
 hw/acpi/Makefile.objs           |   1 +
 hw/acpi/aml-build.c             |   2 +
 hw/acpi/hest_ghes.c             | 345 ++++++++++++++++++++++++++++++++++++++++
 hw/arm/virt-acpi-build.c        |   6 +
 include/hw/acpi/acpi-defs.h     | 193 ++++++++++++++++++++++
 include/hw/acpi/aml-build.h     |   1 +
 include/hw/acpi/hest_ghes.h     |  47 ++++++
 include/sysemu/kvm.h            |   2 +-
 linux-headers/asm-arm64/kvm.h   |   5 +
 linux-headers/linux/kvm.h       |   2 +
 target/arm/cpu.h                |   3 +
 target/arm/internals.h          |  14 ++
 target/arm/kvm.c                |  34 ++++
 target/arm/kvm64.c              | 186 ++++++++++++++++++++++
 target/arm/kvm_arm.h            |   1 +
 16 files changed, 842 insertions(+), 1 deletion(-)
 create mode 100644 hw/acpi/hest_ghes.c
 create mode 100644 include/hw/acpi/hest_ghes.h

-- 
1.8.3.1

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
@ 2017-08-18 14:23 ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: linux-arm-kernel

In the armv8 platform, the mainly processor hardware error notification
type are synchronous external abort(SEA) and SError Interrupt (SEI), For
the ARMv8 SEA/SEI, KVM or host kernel will deliver SIGBUS or use other
interface to notify user space. After user space gets the notification,
it will record the CPER to simulate GHES for guest OS and inject the a
exception(SEA/SEI) to KVM. 

This series patch has two parts, one part handles synchronous external
abort(SEA) exception and SError Interrupt (SEI) exception; another part
is generating APEI table when guest OS boot up, and dynamically record
CPER for the guest OS about the generic hardware errors. Currently the
userspace only handles the memory section hardware errors. Before Qemu
record the CPER, it needs to check the ACK value written by the guest
OS to avoid read-write race condition. In the simulated APEI/GHESV2/CPER
table, the max number of error soure is 11, which is classified by
notification type, now only enable the SEA/SEI notification type error
source to avoid OS boot warning.


About the whole solution we ever discuessed it in here before:
https://patchwork.kernel.org/patch/9633105/

Below is the APEI/GHESV2/CPER table layout, the max number of error soure is 11:

       etc/acpi/tables                               etc/hardware_errors
    ====================                    ==========================================
+ +--------------------------+            +------------------+
| | HEST                     |            |    address       |              +--------------+
| +--------------------------+            |    registers     |              | Error Status |
| | GHES0                    |            | +----------------+              | Data Block 0 |
| +--------------------------+ +--------->| |status_address0 |------------->| +------------+
| | .................        | |          | +----------------+              | |  CPER      |
| | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
| | .................        |   |        | +----------------+          |   | |  ....      |
| | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
| | read_ack_preserve        | | |        +------------------+          |   | +------------+
| | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
+ +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
| | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
+ +--------------------------+   | |      | +----------------+        |     | |  CPER      |
| | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
| | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
| | .................        |     | |    | |  ............. |        |     | |  CPER      |
| | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
| | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
| | read_ack_write           |     |   |  | +----------------+        |     | +------------+
+ +--------------------------|     |   |                              |     | Error Status |
| | ...............          |     |   |                              |     | Data Block 10|
+ +--------------------------+     |   |                              +---->| +------------+
| | GHES10                   |     |   |                                    | |  CPER      |
+ +--------------------------+     |   |                                    | |  CPER      |
| | .................        |     |   |                                    | |  ....      |
| | error_status_address-----+-----+   |                                    | |  CPER      |
| | .................        |         |                                    +-+------------+
| | read_ack_register--------+---------+
| | read_ack_preserve        |
| | read_ack_write           |
+ +--------------------------+


----------------------------------------------------------------------------------------------
How to test guest OS do SEA/SEI recovery:

1. In the guest OS, trigger a SEA or SEI.
2. Then you will see below error log that printed by the memory failure
3. Memory failure will do the recovery for the error.

Such as the below shown kernel log:
[   21.101216] Synchronous External Abort: synchronous external abort (0x96000010) at 0xffffff8008064018
[   21.104969] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 8
[   21.106918] {1}[Hardware Error]: event severity: recoverable
[   21.109027] {1}[Hardware Error]:  Error 0, type: recoverable
[   21.110362] {1}[Hardware Error]:   section_type: memory error
[   21.111705] {1}[Hardware Error]:   physical_address: 0x000000007a200000
[   21.113255] {1}[Hardware Error]:   error_type: 3, multi-bit ECC
[   21.118528] Internal error: : 96000010 [#1] SMP
[   21.119587] Modules linked in:
[   21.120307] CPU: 0 PID: 509 Comm: devmem Not tainted 4.12.0-rc4ajb-00990-g954379b-dirty #67
[   21.122307] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[   21.123915] task: ffffffc03da32900 task.stack: ffffffc03dbbc000
[   21.125302] PC is at __do_user_fault+0x58/0x110
[   21.126370] LR is at __do_user_fault+0x54/0x110
[   21.127433] pc : [<ffffff8008097528>] lr : [<ffffff8008097524>] pstate: 80000145
[   21.129164] sp : ffffffc03dbbfd20
[   21.129940] x29: ffffffc03dbbfd20 x28: ffffffc03da32900
[   21.131204] x27: 0000000000000000 x26: 0000007f7edc5001
[   21.132439] x25: ffffff8008648438 x24: ffffffc03dbbfec0
[   21.133689] x23: 0000000000030001 x22: 0000007f7edc5001
[   21.134934] x21: 0000000000000007 x20: 0000000092000021
[   21.136195] x19: ffffffc03da32900 x18: 0000007fdd4c18f0
[   21.137439] x17: 0000007f7ecb9ebc x16: 0000000000412058

------------------------------------------------------------------------------------------------
how to test guest OS APTI/GHES:
1. In the guest OS, use this command to dump the APEI table: 
    "iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST"
2. And find the address for the generic error status block
   according to the notification type
3. then find the CPER record through the generic error status block.

For example(notification type is SEA):

(1) root at genericarmv8:~# iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST
(2) root at genericarmv8:~# cat HEST.dsl
    /*
     * Intel ACPI Component Architecture
     * AML/ASL+ Disassembler version 20170728 (64-bit version)
     * Copyright (c) 2000 - 2017 Intel Corporation
     *
     * Disassembly of /sys/firmware/acpi/tables/HEST, Mon Sep  5 07:59:17 2016
     *
     * ACPI Data Table [HEST]
     *
     * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
     */

    ..................................................................................
    [308h 0776   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
    [30Ah 0778   2]                    Source Id : 0008
    [30Ch 0780   2]            Related Source Id : FFFF
    [30Eh 0782   1]                     Reserved : 00
    [30Fh 0783   1]                      Enabled : 01
    [310h 0784   4]       Records To Preallocate : 00000001
    [314h 0788   4]      Max Sections Per Record : 00000001
    [318h 0792   4]          Max Raw Data Length : 00001000

    [31Ch 0796  12]         Error Status Address : [Generic Address Structure]
    [31Ch 0796   1]                     Space ID : 00 [SystemMemory]
    [31Dh 0797   1]                    Bit Width : 40
    [31Eh 0798   1]                   Bit Offset : 00
    [31Fh 0799   1]         Encoded Access Width : 04 [QWord Access:64]
    [320h 0800   8]                      Address : 00000000785D0040

    [328h 0808  28]                       Notify : [Hardware Error Notification Structure]
    [328h 0808   1]                  Notify Type : 08 [SEA]
    [329h 0809   1]                Notify Length : 1C
    [32Ah 0810   2]   Configuration Write Enable : 0000
    [32Ch 0812   4]                 PollInterval : 00000000
    [330h 0816   4]                       Vector : 00000000
    [334h 0820   4]      Polling Threshold Value : 00000000
    [338h 0824   4]     Polling Threshold Window : 00000000
    [33Ch 0828   4]        Error Threshold Value : 00000000
    [340h 0832   4]       Error Threshold Window : 00000000

    [344h 0836   4]    Error Status Block Length : 00001000
    [348h 0840  12]            Read Ack Register : [Generic Address Structure]
    [348h 0840   1]                     Space ID : 00 [SystemMemory]
    [349h 0841   1]                    Bit Width : 40
    [34Ah 0842   1]                   Bit Offset : 00
    [34Bh 0843   1]         Encoded Access Width : 04 [QWord Access:64]
    [34Ch 0844   8]                      Address : 00000000785D0098

    [354h 0852   8]            Read Ack Preserve : 00000000FFFFFFFE
    [35Ch 0860   8]               Read Ack Write : 0000000000000001

    [364h 0868   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
    [366h 0870   2]                    Source Id : 0009
    [368h 0872   2]            Related Source Id : FFFF
    [36Ah 0874   1]                     Reserved : 00
    [36Bh 0875   1]                      Enabled : 01
    [36Ch 0876   4]       Records To Preallocate : 00000001
    [370h 0880   4]      Max Sections Per Record : 00000001
    [374h 0884   4]          Max Raw Data Length : 00001000

    [378h 0888  12]         Error Status Address : [Generic Address Structure]
    [378h 0888   1]                     Space ID : 00 [SystemMemory]
    [379h 0889   1]                    Bit Width : 40
    [37Ah 0890   1]                   Bit Offset : 00
    [37Bh 0891   1]         Encoded Access Width : 04 [QWord Access:64]
    [37Ch 0892   8]                      Address : 00000000785D0048

    [384h 0900  28]                       Notify : [Hardware Error Notification Structure]
    [384h 0900   1]                  Notify Type : 09 [SEI]
    [385h 0901   1]                Notify Length : 1C
    [386h 0902   2]   Configuration Write Enable : 0000
    [388h 0904   4]                 PollInterval : 00000000
    [38Ch 0908   4]                       Vector : 00000000
    [390h 0912   4]      Polling Threshold Value : 00000000
    [394h 0916   4]     Polling Threshold Window : 00000000
    [398h 0920   4]        Error Threshold Value : 00000000
    [39Ch 0924   4]       Error Threshold Window : 00000000

    [3A0h 0928   4]    Error Status Block Length : 00001000
    [3A4h 0932  12]            Read Ack Register : [Generic Address Structure]
    [3A4h 0932   1]                     Space ID : 00 [SystemMemory]
    [3A5h 0933   1]                    Bit Width : 40
    [3A6h 0934   1]                   Bit Offset : 00
    [3A7h 0935   1]         Encoded Access Width : 04 [QWord Access:64]
    [3A8h 0936   8]                      Address : 00000000785D00A0

    [3B0h 0944   8]            Read Ack Preserve : 00000000FFFFFFFE
    [3B8h 0952   8]               Read Ack Write : 0000000000000001
    .....................................................................................
(3) according to above table, the address that contains the physical address of a block
    of memory that holds the error status data for SEA notification error source is 0x00000000785D0040
(4) the address for SEA notification error source is 0x785d8108
    (qemu) xp /1 0x00000000785D0040
    00000000785d0040: 0x785d80b0

(5) check the content of generic error status block and generic error data entry
    (qemu) xp /100x 0x785d80b0
    00000000785d80b0: 0x00000001 0x00000000 0x00000000 0x00000098
    00000000785d80c0: 0x00000000 0xa5bc1114 0x4ede6f64 0x833e63b8
    00000000785d80d0: 0xb1837ced 0x00000000 0x00000300 0x00000050
    00000000785d80e0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d80f0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8100: 0x00000000 0x00000000 0x00000000 0x00004002
    00000000785d8110: 0x00000000 0x00000000 0x00000000 0x00001111
    00000000785d8120: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8130: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8140: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8150: 0x00000000 0x00000003 0x00000000 0x00000000
    00000000785d8160: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8170: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8180: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8190: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81a0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81b0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81c0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81d0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81e0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d81f0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8200: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8210: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8220: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8230: 0x00000000 0x00000000 0x00000000 0x00000000
(6) check the OSPM's ACK value(for example SEA)
    /* Before OSPM acknowledges the error, check the ACK value */
    (qemu) xp /1 0x00000000785D0098
    00000000785d00f0: 0x00000000

    /* After OSPM acknowledges the error, check the ACK value */
    (qemu) xp /1 0x00000000785D0098
    00000000785d00f0: 0x00000001

Dongjiu Geng (6):
  ACPI: add APEI/HEST/CPER structures and macros
  ACPI: Add APEI GHES Table Generation support
  ACPI: build and enable APEI GHES in the Makefile and configuration
  target-arm: kvm64: detect guest RAS EXTENSION feature
  target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
  target-arm: kvm64: Handle SError interrupt from the guest OS

 default-configs/arm-softmmu.mak |   1 +
 hw/acpi/Makefile.objs           |   1 +
 hw/acpi/aml-build.c             |   2 +
 hw/acpi/hest_ghes.c             | 345 ++++++++++++++++++++++++++++++++++++++++
 hw/arm/virt-acpi-build.c        |   6 +
 include/hw/acpi/acpi-defs.h     | 193 ++++++++++++++++++++++
 include/hw/acpi/aml-build.h     |   1 +
 include/hw/acpi/hest_ghes.h     |  47 ++++++
 include/sysemu/kvm.h            |   2 +-
 linux-headers/asm-arm64/kvm.h   |   5 +
 linux-headers/linux/kvm.h       |   2 +
 target/arm/cpu.h                |   3 +
 target/arm/internals.h          |  14 ++
 target/arm/kvm.c                |  34 ++++
 target/arm/kvm64.c              | 186 ++++++++++++++++++++++
 target/arm/kvm_arm.h            |   1 +
 16 files changed, 842 insertions(+), 1 deletion(-)
 create mode 100644 hw/acpi/hest_ghes.c
 create mode 100644 include/hw/acpi/hest_ghes.h

-- 
1.8.3.1

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
  2017-08-18 14:23 ` Dongjiu Geng
                     ` (2 preceding siblings ...)
  (?)
@ 2017-08-18 14:23   ` Dongjiu Geng
  -1 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: zhengqiang10, wuquanming, huangshaoyu, linuxarm, gengdongjiu

(1) Add related APEI/HEST table structures and  macros, these
    definition refer to ACPI 6.1 and UEFI 2.6 spec.
(2) Add generic error status block and CPER memory section
    definition, user space only handle memory section errors.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 193 insertions(+)

diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
index 72be675..3b4bad7 100644
--- a/include/hw/acpi/acpi-defs.h
+++ b/include/hw/acpi/acpi-defs.h
@@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
 #define ACPI_APIC_GENERIC_TRANSLATOR    15
 #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
 
+/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
+#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
+#define UEFI_CPER_MEM_VALID_PA               0x0002
+#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
+#define UEFI_CPER_MEM_VALID_NODE             0x0008
+#define UEFI_CPER_MEM_VALID_CARD             0x0010
+#define UEFI_CPER_MEM_VALID_MODULE           0x0020
+#define UEFI_CPER_MEM_VALID_BANK             0x0040
+#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
+#define UEFI_CPER_MEM_VALID_ROW              0x0100
+#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
+#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
+#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
+#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
+#define UEFI_CPER_MEM_VALID_TARGET           0x2000
+#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
+#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
+#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
+#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
+#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
+
+/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
+
+enum AcpiHestNotifyType {
+    ACPI_HEST_NOTIFY_POLLED = 0,
+    ACPI_HEST_NOTIFY_EXTERNAL = 1,
+    ACPI_HEST_NOTIFY_LOCAL = 2,
+    ACPI_HEST_NOTIFY_SCI = 3,
+    ACPI_HEST_NOTIFY_NMI = 4,
+    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
+    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
+    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
+    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
+};
+
 /*
  * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
  */
@@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
 } QEMU_PACKED;
 typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
 
+/* Hardware Error Notification, from the ACPI 6.1
+ * spec, "18.3.2.9 Hardware Error Notification"
+ */
+struct AcpiHestNotify {
+    uint8_t type;
+    uint8_t length;
+    uint16_t config_write_enable;
+    uint32_t poll_interval;
+    uint32_t vector;
+    uint32_t polling_threshold_value;
+    uint32_t polling_threshold_window;
+    uint32_t error_threshold_value;
+    uint32_t error_threshold_window;
+} QEMU_PACKED;
+typedef struct AcpiHestNotify AcpiHestNotify;
+
+/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
+ * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
+ */
+enum AcpiHestSourceType {
+    ACPI_HEST_SOURCE_IA32_CHECK = 0,
+    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
+    ACPI_HEST_SOURCE_IA32_NMI = 2,
+    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
+    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
+    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
+    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
+    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
+    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
+};
+
+/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
+#define ACPI_GEBS_UNCORRECTABLE             (1)
+#define ACPI_GEBS_CORRECTABLE               (1 << 1)
+#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
+#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
+/* 10 bits, error data entry count */
+#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
+
+/* Generic Hardware Error Source Structure, refer to ACPI 6.1
+ * "18.3.2.7 Generic Hardware Error Source". in this struct the
+ * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
+ */
+
+struct AcpiGenericHardwareErrorSource {
+    uint16_t type;
+    uint16_t source_id;
+    uint16_t related_source_id;
+    uint8_t flags;
+    uint8_t enabled;
+    uint32_t number_of_records;
+    uint32_t max_sections_per_record;
+    uint32_t max_raw_data_length;
+    struct AcpiGenericAddress error_status_address;
+    struct AcpiHestNotify notify;
+    uint32_t error_status_block_length;
+} QEMU_PACKED;
+typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
+
+/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
+ * Hardware Error Source version 2", in this struct the "type" field has to
+ * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
+ */
+struct AcpiGenericHardwareErrorSourceV2 {
+    uint16_t type;
+    uint16_t source_id;
+    uint16_t related_source_id;
+    uint8_t flags;
+    uint8_t enabled;
+    uint32_t number_of_records;
+    uint32_t max_sections_per_record;
+    uint32_t max_raw_data_length;
+    struct AcpiGenericAddress error_status_address;
+    struct AcpiHestNotify notify;
+    uint32_t error_status_block_length;
+    struct AcpiGenericAddress read_ack_register;
+    uint64_t read_ack_preserve;
+    uint64_t read_ack_write;
+} QEMU_PACKED;
+typedef struct AcpiGenericHardwareErrorSourceV2
+            AcpiGenericHardwareErrorSourceV2;
+
+/* Generic Error Status block, from ACPI 6.1,
+ * "18.3.2.7.1 Generic Error Data"
+ */
+struct AcpiGenericErrorStatus {
+    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
+    uint32_t block_status;
+    uint32_t raw_data_offset;
+    uint32_t raw_data_length;
+    uint32_t data_length;
+    uint32_t error_severity;
+} QEMU_PACKED;
+typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
+
+enum AcpiGenericErrorSeverity {
+    ACPI_CPER_SEV_RECOVERABLE,
+    ACPI_CPER_SEV_FATAL,
+    ACPI_CPER_SEV_CORRECTED,
+    ACPI_CPER_SEV_NONE,
+};
+
+/* Generic Error Data entry, revision number is 0x0300,
+ * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
+ */
+struct AcpiGenericErrorData {
+    uint8_t section_type_le[16];
+    /* The "error_severity" fields that they take their
+     * values from AcpiGenericErrorSeverity
+     */
+    uint32_t error_severity;
+    uint16_t revision;
+    uint8_t validation_bits;
+    uint8_t flags;
+    uint32_t error_data_length;
+    uint8_t fru_id[16];
+    uint8_t fru_text[20];
+    uint64_t time_stamp;
+} QEMU_PACKED;
+typedef struct AcpiGenericErrorData AcpiGenericErrorData;
+
+/* From UEFI 2.6, "N.2.5 Memory Error Section" */
+struct UefiCperSecMemErr {
+    uint64_t    validation_bits;
+    uint64_t    error_status;
+    uint64_t    physical_addr;
+    uint64_t    physical_addr_mask;
+    uint16_t    node;
+    uint16_t    card;
+    uint16_t    module;
+    uint16_t    bank;
+    uint16_t    device;
+    uint16_t    row;
+    uint16_t    column;
+    uint16_t    bit_pos;
+    uint64_t    requestor_id;
+    uint64_t    responder_id;
+    uint64_t    target_id;
+    uint8_t     error_type;
+    uint8_t     reserved;
+    uint16_t    rank;
+    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
+    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
+} QEMU_PACKED;
+typedef struct UefiCperSecMemErr UefiCperSecMemErr;
+
+/*
+ * HEST Description Table
+ */
+struct AcpiHardwareErrorSourceTable {
+    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
+    uint32_t           error_source_count;
+} QEMU_PACKED;
+typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
+
 #define ACPI_SRAT_PROCESSOR_APIC     0
 #define ACPI_SRAT_MEMORY             1
 #define ACPI_SRAT_PROCESSOR_x2APIC   2
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

(1) Add related APEI/HEST table structures and  macros, these
    definition refer to ACPI 6.1 and UEFI 2.6 spec.
(2) Add generic error status block and CPER memory section
    definition, user space only handle memory section errors.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 193 insertions(+)

diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
index 72be675..3b4bad7 100644
--- a/include/hw/acpi/acpi-defs.h
+++ b/include/hw/acpi/acpi-defs.h
@@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
 #define ACPI_APIC_GENERIC_TRANSLATOR    15
 #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
 
+/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
+#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
+#define UEFI_CPER_MEM_VALID_PA               0x0002
+#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
+#define UEFI_CPER_MEM_VALID_NODE             0x0008
+#define UEFI_CPER_MEM_VALID_CARD             0x0010
+#define UEFI_CPER_MEM_VALID_MODULE           0x0020
+#define UEFI_CPER_MEM_VALID_BANK             0x0040
+#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
+#define UEFI_CPER_MEM_VALID_ROW              0x0100
+#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
+#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
+#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
+#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
+#define UEFI_CPER_MEM_VALID_TARGET           0x2000
+#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
+#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
+#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
+#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
+#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
+
+/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
+
+enum AcpiHestNotifyType {
+    ACPI_HEST_NOTIFY_POLLED = 0,
+    ACPI_HEST_NOTIFY_EXTERNAL = 1,
+    ACPI_HEST_NOTIFY_LOCAL = 2,
+    ACPI_HEST_NOTIFY_SCI = 3,
+    ACPI_HEST_NOTIFY_NMI = 4,
+    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
+    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
+    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
+    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
+};
+
 /*
  * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
  */
@@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
 } QEMU_PACKED;
 typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
 
+/* Hardware Error Notification, from the ACPI 6.1
+ * spec, "18.3.2.9 Hardware Error Notification"
+ */
+struct AcpiHestNotify {
+    uint8_t type;
+    uint8_t length;
+    uint16_t config_write_enable;
+    uint32_t poll_interval;
+    uint32_t vector;
+    uint32_t polling_threshold_value;
+    uint32_t polling_threshold_window;
+    uint32_t error_threshold_value;
+    uint32_t error_threshold_window;
+} QEMU_PACKED;
+typedef struct AcpiHestNotify AcpiHestNotify;
+
+/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
+ * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
+ */
+enum AcpiHestSourceType {
+    ACPI_HEST_SOURCE_IA32_CHECK = 0,
+    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
+    ACPI_HEST_SOURCE_IA32_NMI = 2,
+    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
+    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
+    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
+    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
+    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
+    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
+};
+
+/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
+#define ACPI_GEBS_UNCORRECTABLE             (1)
+#define ACPI_GEBS_CORRECTABLE               (1 << 1)
+#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
+#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
+/* 10 bits, error data entry count */
+#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
+
+/* Generic Hardware Error Source Structure, refer to ACPI 6.1
+ * "18.3.2.7 Generic Hardware Error Source". in this struct the
+ * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
+ */
+
+struct AcpiGenericHardwareErrorSource {
+    uint16_t type;
+    uint16_t source_id;
+    uint16_t related_source_id;
+    uint8_t flags;
+    uint8_t enabled;
+    uint32_t number_of_records;
+    uint32_t max_sections_per_record;
+    uint32_t max_raw_data_length;
+    struct AcpiGenericAddress error_status_address;
+    struct AcpiHestNotify notify;
+    uint32_t error_status_block_length;
+} QEMU_PACKED;
+typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
+
+/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
+ * Hardware Error Source version 2", in this struct the "type" field has to
+ * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
+ */
+struct AcpiGenericHardwareErrorSourceV2 {
+    uint16_t type;
+    uint16_t source_id;
+    uint16_t related_source_id;
+    uint8_t flags;
+    uint8_t enabled;
+    uint32_t number_of_records;
+    uint32_t max_sections_per_record;
+    uint32_t max_raw_data_length;
+    struct AcpiGenericAddress error_status_address;
+    struct AcpiHestNotify notify;
+    uint32_t error_status_block_length;
+    struct AcpiGenericAddress read_ack_register;
+    uint64_t read_ack_preserve;
+    uint64_t read_ack_write;
+} QEMU_PACKED;
+typedef struct AcpiGenericHardwareErrorSourceV2
+            AcpiGenericHardwareErrorSourceV2;
+
+/* Generic Error Status block, from ACPI 6.1,
+ * "18.3.2.7.1 Generic Error Data"
+ */
+struct AcpiGenericErrorStatus {
+    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
+    uint32_t block_status;
+    uint32_t raw_data_offset;
+    uint32_t raw_data_length;
+    uint32_t data_length;
+    uint32_t error_severity;
+} QEMU_PACKED;
+typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
+
+enum AcpiGenericErrorSeverity {
+    ACPI_CPER_SEV_RECOVERABLE,
+    ACPI_CPER_SEV_FATAL,
+    ACPI_CPER_SEV_CORRECTED,
+    ACPI_CPER_SEV_NONE,
+};
+
+/* Generic Error Data entry, revision number is 0x0300,
+ * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
+ */
+struct AcpiGenericErrorData {
+    uint8_t section_type_le[16];
+    /* The "error_severity" fields that they take their
+     * values from AcpiGenericErrorSeverity
+     */
+    uint32_t error_severity;
+    uint16_t revision;
+    uint8_t validation_bits;
+    uint8_t flags;
+    uint32_t error_data_length;
+    uint8_t fru_id[16];
+    uint8_t fru_text[20];
+    uint64_t time_stamp;
+} QEMU_PACKED;
+typedef struct AcpiGenericErrorData AcpiGenericErrorData;
+
+/* From UEFI 2.6, "N.2.5 Memory Error Section" */
+struct UefiCperSecMemErr {
+    uint64_t    validation_bits;
+    uint64_t    error_status;
+    uint64_t    physical_addr;
+    uint64_t    physical_addr_mask;
+    uint16_t    node;
+    uint16_t    card;
+    uint16_t    module;
+    uint16_t    bank;
+    uint16_t    device;
+    uint16_t    row;
+    uint16_t    column;
+    uint16_t    bit_pos;
+    uint64_t    requestor_id;
+    uint64_t    responder_id;
+    uint64_t    target_id;
+    uint8_t     error_type;
+    uint8_t     reserved;
+    uint16_t    rank;
+    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
+    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
+} QEMU_PACKED;
+typedef struct UefiCperSecMemErr UefiCperSecMemErr;
+
+/*
+ * HEST Description Table
+ */
+struct AcpiHardwareErrorSourceTable {
+    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
+    uint32_t           error_source_count;
+} QEMU_PACKED;
+typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
+
 #define ACPI_SRAT_PROCESSOR_APIC     0
 #define ACPI_SRAT_MEMORY             1
 #define ACPI_SRAT_PROCESSOR_x2APIC   2
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: zhengqiang10, wuquanming, huangshaoyu, linuxarm, gengdongjiu

(1) Add related APEI/HEST table structures and  macros, these
    definition refer to ACPI 6.1 and UEFI 2.6 spec.
(2) Add generic error status block and CPER memory section
    definition, user space only handle memory section errors.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 193 insertions(+)

diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
index 72be675..3b4bad7 100644
--- a/include/hw/acpi/acpi-defs.h
+++ b/include/hw/acpi/acpi-defs.h
@@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
 #define ACPI_APIC_GENERIC_TRANSLATOR    15
 #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
 
+/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
+#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
+#define UEFI_CPER_MEM_VALID_PA               0x0002
+#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
+#define UEFI_CPER_MEM_VALID_NODE             0x0008
+#define UEFI_CPER_MEM_VALID_CARD             0x0010
+#define UEFI_CPER_MEM_VALID_MODULE           0x0020
+#define UEFI_CPER_MEM_VALID_BANK             0x0040
+#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
+#define UEFI_CPER_MEM_VALID_ROW              0x0100
+#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
+#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
+#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
+#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
+#define UEFI_CPER_MEM_VALID_TARGET           0x2000
+#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
+#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
+#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
+#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
+#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
+
+/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
+
+enum AcpiHestNotifyType {
+    ACPI_HEST_NOTIFY_POLLED = 0,
+    ACPI_HEST_NOTIFY_EXTERNAL = 1,
+    ACPI_HEST_NOTIFY_LOCAL = 2,
+    ACPI_HEST_NOTIFY_SCI = 3,
+    ACPI_HEST_NOTIFY_NMI = 4,
+    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
+    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
+    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
+    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
+};
+
 /*
  * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
  */
@@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
 } QEMU_PACKED;
 typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
 
+/* Hardware Error Notification, from the ACPI 6.1
+ * spec, "18.3.2.9 Hardware Error Notification"
+ */
+struct AcpiHestNotify {
+    uint8_t type;
+    uint8_t length;
+    uint16_t config_write_enable;
+    uint32_t poll_interval;
+    uint32_t vector;
+    uint32_t polling_threshold_value;
+    uint32_t polling_threshold_window;
+    uint32_t error_threshold_value;
+    uint32_t error_threshold_window;
+} QEMU_PACKED;
+typedef struct AcpiHestNotify AcpiHestNotify;
+
+/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
+ * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
+ */
+enum AcpiHestSourceType {
+    ACPI_HEST_SOURCE_IA32_CHECK = 0,
+    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
+    ACPI_HEST_SOURCE_IA32_NMI = 2,
+    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
+    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
+    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
+    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
+    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
+    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
+};
+
+/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
+#define ACPI_GEBS_UNCORRECTABLE             (1)
+#define ACPI_GEBS_CORRECTABLE               (1 << 1)
+#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
+#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
+/* 10 bits, error data entry count */
+#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
+
+/* Generic Hardware Error Source Structure, refer to ACPI 6.1
+ * "18.3.2.7 Generic Hardware Error Source". in this struct the
+ * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
+ */
+
+struct AcpiGenericHardwareErrorSource {
+    uint16_t type;
+    uint16_t source_id;
+    uint16_t related_source_id;
+    uint8_t flags;
+    uint8_t enabled;
+    uint32_t number_of_records;
+    uint32_t max_sections_per_record;
+    uint32_t max_raw_data_length;
+    struct AcpiGenericAddress error_status_address;
+    struct AcpiHestNotify notify;
+    uint32_t error_status_block_length;
+} QEMU_PACKED;
+typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
+
+/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
+ * Hardware Error Source version 2", in this struct the "type" field has to
+ * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
+ */
+struct AcpiGenericHardwareErrorSourceV2 {
+    uint16_t type;
+    uint16_t source_id;
+    uint16_t related_source_id;
+    uint8_t flags;
+    uint8_t enabled;
+    uint32_t number_of_records;
+    uint32_t max_sections_per_record;
+    uint32_t max_raw_data_length;
+    struct AcpiGenericAddress error_status_address;
+    struct AcpiHestNotify notify;
+    uint32_t error_status_block_length;
+    struct AcpiGenericAddress read_ack_register;
+    uint64_t read_ack_preserve;
+    uint64_t read_ack_write;
+} QEMU_PACKED;
+typedef struct AcpiGenericHardwareErrorSourceV2
+            AcpiGenericHardwareErrorSourceV2;
+
+/* Generic Error Status block, from ACPI 6.1,
+ * "18.3.2.7.1 Generic Error Data"
+ */
+struct AcpiGenericErrorStatus {
+    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
+    uint32_t block_status;
+    uint32_t raw_data_offset;
+    uint32_t raw_data_length;
+    uint32_t data_length;
+    uint32_t error_severity;
+} QEMU_PACKED;
+typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
+
+enum AcpiGenericErrorSeverity {
+    ACPI_CPER_SEV_RECOVERABLE,
+    ACPI_CPER_SEV_FATAL,
+    ACPI_CPER_SEV_CORRECTED,
+    ACPI_CPER_SEV_NONE,
+};
+
+/* Generic Error Data entry, revision number is 0x0300,
+ * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
+ */
+struct AcpiGenericErrorData {
+    uint8_t section_type_le[16];
+    /* The "error_severity" fields that they take their
+     * values from AcpiGenericErrorSeverity
+     */
+    uint32_t error_severity;
+    uint16_t revision;
+    uint8_t validation_bits;
+    uint8_t flags;
+    uint32_t error_data_length;
+    uint8_t fru_id[16];
+    uint8_t fru_text[20];
+    uint64_t time_stamp;
+} QEMU_PACKED;
+typedef struct AcpiGenericErrorData AcpiGenericErrorData;
+
+/* From UEFI 2.6, "N.2.5 Memory Error Section" */
+struct UefiCperSecMemErr {
+    uint64_t    validation_bits;
+    uint64_t    error_status;
+    uint64_t    physical_addr;
+    uint64_t    physical_addr_mask;
+    uint16_t    node;
+    uint16_t    card;
+    uint16_t    module;
+    uint16_t    bank;
+    uint16_t    device;
+    uint16_t    row;
+    uint16_t    column;
+    uint16_t    bit_pos;
+    uint64_t    requestor_id;
+    uint64_t    responder_id;
+    uint64_t    target_id;
+    uint8_t     error_type;
+    uint8_t     reserved;
+    uint16_t    rank;
+    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
+    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
+} QEMU_PACKED;
+typedef struct UefiCperSecMemErr UefiCperSecMemErr;
+
+/*
+ * HEST Description Table
+ */
+struct AcpiHardwareErrorSourceTable {
+    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
+    uint32_t           error_source_count;
+} QEMU_PACKED;
+typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
+
 #define ACPI_SRAT_PROCESSOR_APIC     0
 #define ACPI_SRAT_MEMORY             1
 #define ACPI_SRAT_PROCESSOR_x2APIC   2
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [Qemu-devel] [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

(1) Add related APEI/HEST table structures and  macros, these
    definition refer to ACPI 6.1 and UEFI 2.6 spec.
(2) Add generic error status block and CPER memory section
    definition, user space only handle memory section errors.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 193 insertions(+)

diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
index 72be675..3b4bad7 100644
--- a/include/hw/acpi/acpi-defs.h
+++ b/include/hw/acpi/acpi-defs.h
@@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
 #define ACPI_APIC_GENERIC_TRANSLATOR    15
 #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
 
+/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
+#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
+#define UEFI_CPER_MEM_VALID_PA               0x0002
+#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
+#define UEFI_CPER_MEM_VALID_NODE             0x0008
+#define UEFI_CPER_MEM_VALID_CARD             0x0010
+#define UEFI_CPER_MEM_VALID_MODULE           0x0020
+#define UEFI_CPER_MEM_VALID_BANK             0x0040
+#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
+#define UEFI_CPER_MEM_VALID_ROW              0x0100
+#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
+#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
+#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
+#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
+#define UEFI_CPER_MEM_VALID_TARGET           0x2000
+#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
+#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
+#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
+#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
+#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
+
+/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
+
+enum AcpiHestNotifyType {
+    ACPI_HEST_NOTIFY_POLLED = 0,
+    ACPI_HEST_NOTIFY_EXTERNAL = 1,
+    ACPI_HEST_NOTIFY_LOCAL = 2,
+    ACPI_HEST_NOTIFY_SCI = 3,
+    ACPI_HEST_NOTIFY_NMI = 4,
+    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
+    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
+    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
+    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
+};
+
 /*
  * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
  */
@@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
 } QEMU_PACKED;
 typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
 
+/* Hardware Error Notification, from the ACPI 6.1
+ * spec, "18.3.2.9 Hardware Error Notification"
+ */
+struct AcpiHestNotify {
+    uint8_t type;
+    uint8_t length;
+    uint16_t config_write_enable;
+    uint32_t poll_interval;
+    uint32_t vector;
+    uint32_t polling_threshold_value;
+    uint32_t polling_threshold_window;
+    uint32_t error_threshold_value;
+    uint32_t error_threshold_window;
+} QEMU_PACKED;
+typedef struct AcpiHestNotify AcpiHestNotify;
+
+/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
+ * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
+ */
+enum AcpiHestSourceType {
+    ACPI_HEST_SOURCE_IA32_CHECK = 0,
+    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
+    ACPI_HEST_SOURCE_IA32_NMI = 2,
+    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
+    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
+    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
+    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
+    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
+    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
+};
+
+/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
+#define ACPI_GEBS_UNCORRECTABLE             (1)
+#define ACPI_GEBS_CORRECTABLE               (1 << 1)
+#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
+#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
+/* 10 bits, error data entry count */
+#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
+
+/* Generic Hardware Error Source Structure, refer to ACPI 6.1
+ * "18.3.2.7 Generic Hardware Error Source". in this struct the
+ * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
+ */
+
+struct AcpiGenericHardwareErrorSource {
+    uint16_t type;
+    uint16_t source_id;
+    uint16_t related_source_id;
+    uint8_t flags;
+    uint8_t enabled;
+    uint32_t number_of_records;
+    uint32_t max_sections_per_record;
+    uint32_t max_raw_data_length;
+    struct AcpiGenericAddress error_status_address;
+    struct AcpiHestNotify notify;
+    uint32_t error_status_block_length;
+} QEMU_PACKED;
+typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
+
+/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
+ * Hardware Error Source version 2", in this struct the "type" field has to
+ * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
+ */
+struct AcpiGenericHardwareErrorSourceV2 {
+    uint16_t type;
+    uint16_t source_id;
+    uint16_t related_source_id;
+    uint8_t flags;
+    uint8_t enabled;
+    uint32_t number_of_records;
+    uint32_t max_sections_per_record;
+    uint32_t max_raw_data_length;
+    struct AcpiGenericAddress error_status_address;
+    struct AcpiHestNotify notify;
+    uint32_t error_status_block_length;
+    struct AcpiGenericAddress read_ack_register;
+    uint64_t read_ack_preserve;
+    uint64_t read_ack_write;
+} QEMU_PACKED;
+typedef struct AcpiGenericHardwareErrorSourceV2
+            AcpiGenericHardwareErrorSourceV2;
+
+/* Generic Error Status block, from ACPI 6.1,
+ * "18.3.2.7.1 Generic Error Data"
+ */
+struct AcpiGenericErrorStatus {
+    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
+    uint32_t block_status;
+    uint32_t raw_data_offset;
+    uint32_t raw_data_length;
+    uint32_t data_length;
+    uint32_t error_severity;
+} QEMU_PACKED;
+typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
+
+enum AcpiGenericErrorSeverity {
+    ACPI_CPER_SEV_RECOVERABLE,
+    ACPI_CPER_SEV_FATAL,
+    ACPI_CPER_SEV_CORRECTED,
+    ACPI_CPER_SEV_NONE,
+};
+
+/* Generic Error Data entry, revision number is 0x0300,
+ * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
+ */
+struct AcpiGenericErrorData {
+    uint8_t section_type_le[16];
+    /* The "error_severity" fields that they take their
+     * values from AcpiGenericErrorSeverity
+     */
+    uint32_t error_severity;
+    uint16_t revision;
+    uint8_t validation_bits;
+    uint8_t flags;
+    uint32_t error_data_length;
+    uint8_t fru_id[16];
+    uint8_t fru_text[20];
+    uint64_t time_stamp;
+} QEMU_PACKED;
+typedef struct AcpiGenericErrorData AcpiGenericErrorData;
+
+/* From UEFI 2.6, "N.2.5 Memory Error Section" */
+struct UefiCperSecMemErr {
+    uint64_t    validation_bits;
+    uint64_t    error_status;
+    uint64_t    physical_addr;
+    uint64_t    physical_addr_mask;
+    uint16_t    node;
+    uint16_t    card;
+    uint16_t    module;
+    uint16_t    bank;
+    uint16_t    device;
+    uint16_t    row;
+    uint16_t    column;
+    uint16_t    bit_pos;
+    uint64_t    requestor_id;
+    uint64_t    responder_id;
+    uint64_t    target_id;
+    uint8_t     error_type;
+    uint8_t     reserved;
+    uint16_t    rank;
+    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
+    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
+} QEMU_PACKED;
+typedef struct UefiCperSecMemErr UefiCperSecMemErr;
+
+/*
+ * HEST Description Table
+ */
+struct AcpiHardwareErrorSourceTable {
+    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
+    uint32_t           error_source_count;
+} QEMU_PACKED;
+typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
+
 #define ACPI_SRAT_PROCESSOR_APIC     0
 #define ACPI_SRAT_MEMORY             1
 #define ACPI_SRAT_PROCESSOR_x2APIC   2
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: linux-arm-kernel

(1) Add related APEI/HEST table structures and  macros, these
    definition refer to ACPI 6.1 and UEFI 2.6 spec.
(2) Add generic error status block and CPER memory section
    definition, user space only handle memory section errors.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 193 insertions(+)

diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
index 72be675..3b4bad7 100644
--- a/include/hw/acpi/acpi-defs.h
+++ b/include/hw/acpi/acpi-defs.h
@@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
 #define ACPI_APIC_GENERIC_TRANSLATOR    15
 #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
 
+/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
+#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
+#define UEFI_CPER_MEM_VALID_PA               0x0002
+#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
+#define UEFI_CPER_MEM_VALID_NODE             0x0008
+#define UEFI_CPER_MEM_VALID_CARD             0x0010
+#define UEFI_CPER_MEM_VALID_MODULE           0x0020
+#define UEFI_CPER_MEM_VALID_BANK             0x0040
+#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
+#define UEFI_CPER_MEM_VALID_ROW              0x0100
+#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
+#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
+#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
+#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
+#define UEFI_CPER_MEM_VALID_TARGET           0x2000
+#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
+#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
+#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
+#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
+#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
+
+/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
+
+enum AcpiHestNotifyType {
+    ACPI_HEST_NOTIFY_POLLED = 0,
+    ACPI_HEST_NOTIFY_EXTERNAL = 1,
+    ACPI_HEST_NOTIFY_LOCAL = 2,
+    ACPI_HEST_NOTIFY_SCI = 3,
+    ACPI_HEST_NOTIFY_NMI = 4,
+    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
+    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
+    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
+    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
+    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
+};
+
 /*
  * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
  */
@@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
 } QEMU_PACKED;
 typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
 
+/* Hardware Error Notification, from the ACPI 6.1
+ * spec, "18.3.2.9 Hardware Error Notification"
+ */
+struct AcpiHestNotify {
+    uint8_t type;
+    uint8_t length;
+    uint16_t config_write_enable;
+    uint32_t poll_interval;
+    uint32_t vector;
+    uint32_t polling_threshold_value;
+    uint32_t polling_threshold_window;
+    uint32_t error_threshold_value;
+    uint32_t error_threshold_window;
+} QEMU_PACKED;
+typedef struct AcpiHestNotify AcpiHestNotify;
+
+/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
+ * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
+ */
+enum AcpiHestSourceType {
+    ACPI_HEST_SOURCE_IA32_CHECK = 0,
+    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
+    ACPI_HEST_SOURCE_IA32_NMI = 2,
+    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
+    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
+    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
+    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
+    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
+    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
+};
+
+/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
+#define ACPI_GEBS_UNCORRECTABLE             (1)
+#define ACPI_GEBS_CORRECTABLE               (1 << 1)
+#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
+#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
+/* 10 bits, error data entry count */
+#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
+
+/* Generic Hardware Error Source Structure, refer to ACPI 6.1
+ * "18.3.2.7 Generic Hardware Error Source". in this struct the
+ * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
+ */
+
+struct AcpiGenericHardwareErrorSource {
+    uint16_t type;
+    uint16_t source_id;
+    uint16_t related_source_id;
+    uint8_t flags;
+    uint8_t enabled;
+    uint32_t number_of_records;
+    uint32_t max_sections_per_record;
+    uint32_t max_raw_data_length;
+    struct AcpiGenericAddress error_status_address;
+    struct AcpiHestNotify notify;
+    uint32_t error_status_block_length;
+} QEMU_PACKED;
+typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
+
+/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
+ * Hardware Error Source version 2", in this struct the "type" field has to
+ * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
+ */
+struct AcpiGenericHardwareErrorSourceV2 {
+    uint16_t type;
+    uint16_t source_id;
+    uint16_t related_source_id;
+    uint8_t flags;
+    uint8_t enabled;
+    uint32_t number_of_records;
+    uint32_t max_sections_per_record;
+    uint32_t max_raw_data_length;
+    struct AcpiGenericAddress error_status_address;
+    struct AcpiHestNotify notify;
+    uint32_t error_status_block_length;
+    struct AcpiGenericAddress read_ack_register;
+    uint64_t read_ack_preserve;
+    uint64_t read_ack_write;
+} QEMU_PACKED;
+typedef struct AcpiGenericHardwareErrorSourceV2
+            AcpiGenericHardwareErrorSourceV2;
+
+/* Generic Error Status block, from ACPI 6.1,
+ * "18.3.2.7.1 Generic Error Data"
+ */
+struct AcpiGenericErrorStatus {
+    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
+    uint32_t block_status;
+    uint32_t raw_data_offset;
+    uint32_t raw_data_length;
+    uint32_t data_length;
+    uint32_t error_severity;
+} QEMU_PACKED;
+typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
+
+enum AcpiGenericErrorSeverity {
+    ACPI_CPER_SEV_RECOVERABLE,
+    ACPI_CPER_SEV_FATAL,
+    ACPI_CPER_SEV_CORRECTED,
+    ACPI_CPER_SEV_NONE,
+};
+
+/* Generic Error Data entry, revision number is 0x0300,
+ * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
+ */
+struct AcpiGenericErrorData {
+    uint8_t section_type_le[16];
+    /* The "error_severity" fields that they take their
+     * values from AcpiGenericErrorSeverity
+     */
+    uint32_t error_severity;
+    uint16_t revision;
+    uint8_t validation_bits;
+    uint8_t flags;
+    uint32_t error_data_length;
+    uint8_t fru_id[16];
+    uint8_t fru_text[20];
+    uint64_t time_stamp;
+} QEMU_PACKED;
+typedef struct AcpiGenericErrorData AcpiGenericErrorData;
+
+/* From UEFI 2.6, "N.2.5 Memory Error Section" */
+struct UefiCperSecMemErr {
+    uint64_t    validation_bits;
+    uint64_t    error_status;
+    uint64_t    physical_addr;
+    uint64_t    physical_addr_mask;
+    uint16_t    node;
+    uint16_t    card;
+    uint16_t    module;
+    uint16_t    bank;
+    uint16_t    device;
+    uint16_t    row;
+    uint16_t    column;
+    uint16_t    bit_pos;
+    uint64_t    requestor_id;
+    uint64_t    responder_id;
+    uint64_t    target_id;
+    uint8_t     error_type;
+    uint8_t     reserved;
+    uint16_t    rank;
+    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
+    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
+} QEMU_PACKED;
+typedef struct UefiCperSecMemErr UefiCperSecMemErr;
+
+/*
+ * HEST Description Table
+ */
+struct AcpiHardwareErrorSourceTable {
+    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
+    uint32_t           error_source_count;
+} QEMU_PACKED;
+typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
+
 #define ACPI_SRAT_PROCESSOR_APIC     0
 #define ACPI_SRAT_MEMORY             1
 #define ACPI_SRAT_PROCESSOR_x2APIC   2
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
  2017-08-18 14:23 ` Dongjiu Geng
  (?)
  (?)
@ 2017-08-18 14:23   ` Dongjiu Geng
  -1 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

This implements APEI GHES Table by passing the error CPER info
to the guest via a fw_cfg_blob. After a CPER info is recorded, an
SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
will be injected into the guest OS.

Below is the table layout, the max number of error soure is 11,
which is classified by notification type.

     etc/acpi/tables                               etc/hardware_errors
    ====================                    ==========================================
+ +--------------------------+            +------------------+
| | HEST                     |            |    address       |              +--------------+
| +--------------------------+            |    registers     |              | Error Status |
| | GHES0                    |            | +----------------+              | Data Block 0 |
| +--------------------------+ +--------->| |status_address0 |------------->| +------------+
| | .................        | |          | +----------------+              | |  CPER      |
| | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
| | .................        |   |        | +----------------+          |   | |  ....      |
| | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
| | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
| | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
+ +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
| | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
+ +--------------------------+   | |      | +----------------+        |     | |  CPER      |
| | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
| | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
| | .................        |     | |    | |  ............. |        |     | |  CPER      |
| | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
| | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
| | read_ack_write           |     |   |  | +----------------+        |     | +------------+
+ +--------------------------|     |   |                              |     | Error Status |
| | ...............          |     |   |                              |     | Data Block 10|
+ +--------------------------+     |   |                              +---->| +------------+
| | GHES10                   |     |   |                                    | |  CPER      |
+ +--------------------------+     |   |                                    | |  CPER      |
| | .................        |     |   |                                    | |  ....      |
| | error_status_address-----+-----+   |                                    | |  CPER      |
| | .................        |         |                                    +-+------------+
| | read_ack_register--------+---------+
| | read_ack_preserve        |
| | read_ack_write           |
+ +--------------------------+

For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
so user space must check the ack value to avoid read-write race condition.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 hw/acpi/aml-build.c         |   2 +
 hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
 hw/arm/virt-acpi-build.c    |   6 +
 include/hw/acpi/aml-build.h |   1 +
 include/hw/acpi/hest_ghes.h |  47 ++++++
 5 files changed, 401 insertions(+)
 create mode 100644 hw/acpi/hest_ghes.c
 create mode 100644 include/hw/acpi/hest_ghes.h

diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
index 36a6cc4..6849e5f 100644
--- a/hw/acpi/aml-build.c
+++ b/hw/acpi/aml-build.c
@@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
     tables->table_data = g_array_new(false, true /* clear */, 1);
     tables->tcpalog = g_array_new(false, true /* clear */, 1);
     tables->vmgenid = g_array_new(false, true /* clear */, 1);
+    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
     tables->linker = bios_linker_loader_init();
 }
 
@@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
     g_array_free(tables->table_data, true);
     g_array_free(tables->tcpalog, mfre);
     g_array_free(tables->vmgenid, mfre);
+    g_array_free(tables->hardware_errors, mfre);
 }
 
 /* Build rsdt table */
diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
new file mode 100644
index 0000000..ff6b5ef
--- /dev/null
+++ b/hw/acpi/hest_ghes.c
@@ -0,0 +1,345 @@
+/*
+ *  APEI GHES table Generation
+ *
+ *  Copyright (C) 2017 huawei.
+ *
+ *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "qmp-commands.h"
+#include "hw/acpi/acpi.h"
+#include "hw/acpi/aml-build.h"
+#include "hw/acpi/hest_ghes.h"
+#include "hw/nvram/fw_cfg.h"
+#include "sysemu/sysemu.h"
+#include "qemu/error-report.h"
+
+/* The structure that stands for the layout
+ * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
+ *
+ *           etc/hardware_errors
+ * ==========================================
+ * +------------------+
+ * |    address       |              +--------------+
+ * |    registers     |              | Error Status |
+ * | +----------------+              | Data Block 0 |
+ * | |status_address0 |------------->| +------------+
+ * | +----------------+              | |  CPER      |
+ * | |status_address1 |----------+   | |  CPER      |
+ * | +----------------+          |   | |  ....      |
+ * | |.............   |          |   | |  CPER      |
+ * | +----------------+          |   | +------------+
+ * | |status_address10|-----+    |   | Error Status |
+ * | +----------------+     |    |   | Data Block 1 |
+ * | |ack_value0      |     |    +-->| +------------+
+ * | +----------------+     |        | |  CPER      |
+ * | |ack_value1      |     |        | |  CPER      |
+ * | +----------------+     |        | |  ....      |
+ * | | .............  |     |        | |  CPER      |
+ * | +----------------+     |        +-+------------+
+ * | |ack_value10     |     |        | |..........  |
+ * | +----------------+     |        | +------------+
+ *                          |        | Error Status |
+ *                          |        | Data Block10 |
+ *                          +------->+------------+
+ *                                   | |  CPER      |
+ *                                   | |  CPER      |
+ *                                   | |  ....      |
+ *                                   | |  CPER      |
+ *                                   +-+------------+
+ */
+struct hardware_errors_buffer {
+    /* Generic Error Status Block register */
+    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
+    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
+    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
+};
+
+static int ghes_record_cper(uint64_t error_block_address,
+                                    uint64_t error_physical_addr)
+{
+    AcpiGenericErrorStatus block;
+    AcpiGenericErrorData *gdata;
+    UefiCperSecMemErr *mem_err;
+    uint64_t current_block_length;
+    unsigned char *buffer;
+    /* memory section */
+    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
+                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
+                              0x83, 0xB1};
+
+    cpu_physical_memory_read(error_block_address, &block,
+                                sizeof(AcpiGenericErrorStatus));
+
+    /* Get the current generic error status block length */
+    current_block_length = sizeof(AcpiGenericErrorStatus) +
+        le32_to_cpu(block.data_length);
+
+    /* If the Generic Error Status Block is NULL, update
+     * the block header
+     */
+    if (!block.block_status) {
+        block.block_status = ACPI_GEBS_UNCORRECTABLE;
+        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
+    }
+
+    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
+    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
+
+    /* check whether it runs out of the preallocated memory */
+    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
+       GHES_MAX_RAW_DATA_LENGTH) {
+        error_report("Record CPER out of boundary!!!");
+        return GHES_CPER_FAIL;
+    }
+
+    /* Write back the Generic Error Status Block to guest memory */
+    cpu_physical_memory_write(error_block_address, &block,
+        sizeof(AcpiGenericErrorStatus));
+
+    /* Fill in Generic Error Data Entry */
+    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
+                       sizeof(UefiCperSecMemErr));
+
+
+    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
+    gdata = (AcpiGenericErrorData *)buffer;
+
+    /* Memory section */
+    memcpy(&(gdata->section_type_le), &mem_section_id_le,
+            sizeof(mem_section_id_le));
+
+    /* error severity is recoverable */
+    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
+    gdata->revision = 0x300; /* the revision number is 0x300 */
+    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
+
+    mem_err = (UefiCperSecMemErr *) (gdata + 1);
+
+    /* User space only handle the memory section CPER */
+
+    /* Hard code to Multi-bit ECC error */
+    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
+    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
+
+    /* Record the physical address at which the memory error occurred */
+    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
+    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
+
+    /* Write back the Generic Error Data Entry to guest memory */
+    cpu_physical_memory_write(error_block_address + current_block_length,
+        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
+
+    g_free(buffer);
+    return GHES_CPER_OK;
+}
+
+static void
+build_address(GArray *table_data, BIOSLinker *linker,
+    uint32_t dst_patched_offset, uint32_t src_offset,
+    uint8_t address_space_id , uint8_t  register_bit_width,
+    uint8_t register_bit_offset, uint8_t access_size)
+{
+    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
+        offsetof(struct AcpiGenericAddress, address);
+
+    /* Address space */
+    build_append_int_noprefix(table_data, address_space_id, 1);
+    /* register bit width */
+    build_append_int_noprefix(table_data, register_bit_width, 1);
+    /* register bit offset */
+    build_append_int_noprefix(table_data, register_bit_offset, 1);
+    /* access size */
+    build_append_int_noprefix(table_data, access_size, 1);
+    acpi_data_push(table_data, address_size);
+
+    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
+     * can retrieve and read it. the address size is 64 bits.
+     */
+    bios_linker_loader_add_pointer(linker,
+        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
+        GHES_ERRORS_FW_CFG_FILE, src_offset);
+}
+
+void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
+                                            BIOSLinker *linker)
+{
+    uint32_t ghes_start = table_data->len;
+    uint32_t address_size, error_status_address_offset;
+    uint32_t read_ack_register_offset, i;
+
+    address_size = sizeof(struct AcpiGenericAddress) -
+        offsetof(struct AcpiGenericAddress, address);
+
+    error_status_address_offset = ghes_start +
+        sizeof(AcpiHardwareErrorSourceTable) +
+        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
+        offsetof(struct AcpiGenericAddress, address);
+
+    read_ack_register_offset = ghes_start +
+        sizeof(AcpiHardwareErrorSourceTable) +
+        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
+        offsetof(struct AcpiGenericAddress, address);
+
+    acpi_data_push(hardware_error,
+        offsetof(struct hardware_errors_buffer, ack_value));
+    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
+        /* Initialize read ack register */
+        build_append_int_noprefix((void *)hardware_error, 1, 8);
+
+    /* Reserved the total size for ERRORS fw_cfg blob
+     */
+    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
+
+    /* Allocate guest memory for the Data fw_cfg blob */
+    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
+                            1, false);
+    /* Reserve table header size */
+    acpi_data_push(table_data, sizeof(AcpiTableHeader));
+
+    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
+
+    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
+        build_append_int_noprefix(table_data,
+            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
+        /* source id */
+        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
+        /* related source id */
+        build_append_int_noprefix(table_data, 0xffff, 2);
+        build_append_int_noprefix(table_data, 0, 1); /* flags */
+
+        /* Currently only enable SEA notification type to avoid the kernel
+         * warning, reserve the space for other notification error source
+         */
+        if (i == ACPI_HEST_NOTIFY_SEA) {
+            build_append_int_noprefix(table_data, 1, 1); /* enabled */
+        } else {
+            build_append_int_noprefix(table_data, 0, 1); /* enabled */
+        }
+
+        /* The number of error status block per generic hardware error source */
+        build_append_int_noprefix(table_data, 1, 4);
+        /* Max sections per record */
+        build_append_int_noprefix(table_data, 1, 4);
+        /* Max raw data length */
+        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
+
+        /* Build error status address*/
+        build_address(table_data, linker, error_status_address_offset + i *
+            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
+            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
+
+        /* Hardware error notification structure */
+        build_append_int_noprefix(table_data, i, 1); /* type */
+        /* length */
+        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
+        build_append_int_noprefix(table_data, 0, 26);
+
+        /* Error Status Block Length */
+        build_append_int_noprefix(table_data,
+            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
+
+        /* Build read ack register */
+        build_address(table_data, linker, read_ack_register_offset + i *
+            sizeof(AcpiGenericHardwareErrorSourceV2),
+            offsetof(struct hardware_errors_buffer, ack_value) +
+            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
+            4 /* QWord access */);
+
+        /* Read ack preserve */
+        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
+
+        /* Read ack write */
+        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
+    }
+
+    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
+        /* Patch address of generic error status block into
+         * the address register so OSPM can retrieve and read it.
+         */
+        bios_linker_loader_add_pointer(linker,
+            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
+            GHES_ERRORS_FW_CFG_FILE,
+            offsetof(struct hardware_errors_buffer, gesb) +
+            i * GHES_MAX_RAW_DATA_LENGTH);
+
+    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
+     * so QEMU can write the ERRORS there. The address is expected to be
+     * < 4GB, but write 64 bits anyway.
+     */
+    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
+        0, address_size, GHES_ERRORS_FW_CFG_FILE,
+        offsetof(struct hardware_errors_buffer, gesb));
+
+    build_header(linker, table_data,
+        (void *)(table_data->data + ghes_start), "HEST",
+        table_data->len - ghes_start, 1, NULL, "GHES");
+}
+
+static GhesState ges;
+void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
+{
+
+    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
+    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
+
+    /* Create a read-only fw_cfg file for GHES */
+    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
+                    size);
+    /* Create a read-write fw_cfg file for Address */
+    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
+        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
+}
+
+bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
+{
+    uint64_t error_block_addr;
+    uint64_t ack_value_addr, ack_value = 0;
+    int loop = 0, ack_value_size;
+    bool ret = GHES_CPER_FAIL;
+
+    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
+        offsetof(struct hardware_errors_buffer, ack_value)) /
+            GHES_ACPI_HEST_NOTIFY_RESERVED;
+
+    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
+        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
+        error_block_addr = le32_to_cpu(error_block_addr);
+
+        ack_value_addr = ges.ghes_addr_le -
+            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
+retry:
+        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
+        if (!ack_value) {
+            if (loop < 3) {
+                usleep(100 * 1000);
+                loop++;
+                goto retry;
+            } else {
+                error_report("Last time OSPM does not acknowledge the error,"
+                    " record CPER failed this time, set the ack value to"
+                    " avoid blocking next time CPER record! exit");
+                ack_value = 1;
+                cpu_physical_memory_write(ack_value_addr,
+                    &ack_value, ack_value_size);
+                return ret;
+            }
+        } else {
+            /* A zero value in ghes_addr means that BIOS has not yet written
+             * the address
+             */
+            if (error_block_addr) {
+                ack_value = 0;
+                cpu_physical_memory_write(ack_value_addr,
+                    &ack_value, ack_value_size);
+                ret = ghes_record_cper(error_block_addr, physical_address);
+            }
+        }
+    }
+    return ret;
+}
diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
index 3d78ff6..def1ec1 100644
--- a/hw/arm/virt-acpi-build.c
+++ b/hw/arm/virt-acpi-build.c
@@ -45,6 +45,7 @@
 #include "hw/arm/virt.h"
 #include "sysemu/numa.h"
 #include "kvm_arm.h"
+#include "hw/acpi/hest_ghes.h"
 
 #define ARM_SPI_BASE 32
 #define ACPI_POWER_BUTTON_DEVICE "PWRB"
@@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
     acpi_add_table(table_offsets, tables_blob);
     build_spcr(tables_blob, tables->linker, vms);
 
+    acpi_add_table(table_offsets, tables_blob);
+    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
+
     if (nb_numa_nodes > 0) {
         acpi_add_table(table_offsets, tables_blob);
         build_srat(tables_blob, tables->linker, vms);
@@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
     fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
                     acpi_data_len(tables.tcpalog));
 
+    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
+
     build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
                                               ACPI_BUILD_RSDP_FILE, 0);
 
diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
index 88d0738..7f7b55c 100644
--- a/include/hw/acpi/aml-build.h
+++ b/include/hw/acpi/aml-build.h
@@ -211,6 +211,7 @@ struct AcpiBuildTables {
     GArray *rsdp;
     GArray *tcpalog;
     GArray *vmgenid;
+    GArray *hardware_errors;
     BIOSLinker *linker;
 } AcpiBuildTables;
 
diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
new file mode 100644
index 0000000..0772756
--- /dev/null
+++ b/include/hw/acpi/hest_ghes.h
@@ -0,0 +1,47 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * Authors:
+ *   Dongjiu Geng <gengdongjiu@huawei.com>
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef ACPI_GHES_H
+#define ACPI_GHES_H
+
+#include "hw/acpi/bios-linker-loader.h"
+
+#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
+#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
+
+#define GHES_GAS_ADDRESS_OFFSET              4
+#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
+#define GHES_NOTIFICATION_STRUCTURE          32
+
+#define GHES_CPER_OK   1
+#define GHES_CPER_FAIL 0
+
+#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
+/* The max size in Bytes for one error block */
+#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
+
+
+typedef struct GhesState {
+    uint64_t ghes_addr_le;
+} GhesState;
+
+void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
+                            BIOSLinker *linker);
+void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
+bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
+#endif
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

This implements APEI GHES Table by passing the error CPER info
to the guest via a fw_cfg_blob. After a CPER info is recorded, an
SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
will be injected into the guest OS.

Below is the table layout, the max number of error soure is 11,
which is classified by notification type.

     etc/acpi/tables                               etc/hardware_errors
    ====================                    ==========================================
+ +--------------------------+            +------------------+
| | HEST                     |            |    address       |              +--------------+
| +--------------------------+            |    registers     |              | Error Status |
| | GHES0                    |            | +----------------+              | Data Block 0 |
| +--------------------------+ +--------->| |status_address0 |------------->| +------------+
| | .................        | |          | +----------------+              | |  CPER      |
| | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
| | .................        |   |        | +----------------+          |   | |  ....      |
| | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
| | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
| | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
+ +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
| | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
+ +--------------------------+   | |      | +----------------+        |     | |  CPER      |
| | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
| | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
| | .................        |     | |    | |  ............. |        |     | |  CPER      |
| | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
| | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
| | read_ack_write           |     |   |  | +----------------+        |     | +------------+
+ +--------------------------|     |   |                              |     | Error Status |
| | ...............          |     |   |                              |     | Data Block 10|
+ +--------------------------+     |   |                              +---->| +------------+
| | GHES10                   |     |   |                                    | |  CPER      |
+ +--------------------------+     |   |                                    | |  CPER      |
| | .................        |     |   |                                    | |  ....      |
| | error_status_address-----+-----+   |                                    | |  CPER      |
| | .................        |         |                                    +-+------------+
| | read_ack_register--------+---------+
| | read_ack_preserve        |
| | read_ack_write           |
+ +--------------------------+

For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
so user space must check the ack value to avoid read-write race condition.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 hw/acpi/aml-build.c         |   2 +
 hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
 hw/arm/virt-acpi-build.c    |   6 +
 include/hw/acpi/aml-build.h |   1 +
 include/hw/acpi/hest_ghes.h |  47 ++++++
 5 files changed, 401 insertions(+)
 create mode 100644 hw/acpi/hest_ghes.c
 create mode 100644 include/hw/acpi/hest_ghes.h

diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
index 36a6cc4..6849e5f 100644
--- a/hw/acpi/aml-build.c
+++ b/hw/acpi/aml-build.c
@@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
     tables->table_data = g_array_new(false, true /* clear */, 1);
     tables->tcpalog = g_array_new(false, true /* clear */, 1);
     tables->vmgenid = g_array_new(false, true /* clear */, 1);
+    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
     tables->linker = bios_linker_loader_init();
 }
 
@@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
     g_array_free(tables->table_data, true);
     g_array_free(tables->tcpalog, mfre);
     g_array_free(tables->vmgenid, mfre);
+    g_array_free(tables->hardware_errors, mfre);
 }
 
 /* Build rsdt table */
diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
new file mode 100644
index 0000000..ff6b5ef
--- /dev/null
+++ b/hw/acpi/hest_ghes.c
@@ -0,0 +1,345 @@
+/*
+ *  APEI GHES table Generation
+ *
+ *  Copyright (C) 2017 huawei.
+ *
+ *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "qmp-commands.h"
+#include "hw/acpi/acpi.h"
+#include "hw/acpi/aml-build.h"
+#include "hw/acpi/hest_ghes.h"
+#include "hw/nvram/fw_cfg.h"
+#include "sysemu/sysemu.h"
+#include "qemu/error-report.h"
+
+/* The structure that stands for the layout
+ * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
+ *
+ *           etc/hardware_errors
+ * ==========================================
+ * +------------------+
+ * |    address       |              +--------------+
+ * |    registers     |              | Error Status |
+ * | +----------------+              | Data Block 0 |
+ * | |status_address0 |------------->| +------------+
+ * | +----------------+              | |  CPER      |
+ * | |status_address1 |----------+   | |  CPER      |
+ * | +----------------+          |   | |  ....      |
+ * | |.............   |          |   | |  CPER      |
+ * | +----------------+          |   | +------------+
+ * | |status_address10|-----+    |   | Error Status |
+ * | +----------------+     |    |   | Data Block 1 |
+ * | |ack_value0      |     |    +-->| +------------+
+ * | +----------------+     |        | |  CPER      |
+ * | |ack_value1      |     |        | |  CPER      |
+ * | +----------------+     |        | |  ....      |
+ * | | .............  |     |        | |  CPER      |
+ * | +----------------+     |        +-+------------+
+ * | |ack_value10     |     |        | |..........  |
+ * | +----------------+     |        | +------------+
+ *                          |        | Error Status |
+ *                          |        | Data Block10 |
+ *                          +------->+------------+
+ *                                   | |  CPER      |
+ *                                   | |  CPER      |
+ *                                   | |  ....      |
+ *                                   | |  CPER      |
+ *                                   +-+------------+
+ */
+struct hardware_errors_buffer {
+    /* Generic Error Status Block register */
+    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
+    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
+    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
+};
+
+static int ghes_record_cper(uint64_t error_block_address,
+                                    uint64_t error_physical_addr)
+{
+    AcpiGenericErrorStatus block;
+    AcpiGenericErrorData *gdata;
+    UefiCperSecMemErr *mem_err;
+    uint64_t current_block_length;
+    unsigned char *buffer;
+    /* memory section */
+    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
+                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
+                              0x83, 0xB1};
+
+    cpu_physical_memory_read(error_block_address, &block,
+                                sizeof(AcpiGenericErrorStatus));
+
+    /* Get the current generic error status block length */
+    current_block_length = sizeof(AcpiGenericErrorStatus) +
+        le32_to_cpu(block.data_length);
+
+    /* If the Generic Error Status Block is NULL, update
+     * the block header
+     */
+    if (!block.block_status) {
+        block.block_status = ACPI_GEBS_UNCORRECTABLE;
+        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
+    }
+
+    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
+    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
+
+    /* check whether it runs out of the preallocated memory */
+    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
+       GHES_MAX_RAW_DATA_LENGTH) {
+        error_report("Record CPER out of boundary!!!");
+        return GHES_CPER_FAIL;
+    }
+
+    /* Write back the Generic Error Status Block to guest memory */
+    cpu_physical_memory_write(error_block_address, &block,
+        sizeof(AcpiGenericErrorStatus));
+
+    /* Fill in Generic Error Data Entry */
+    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
+                       sizeof(UefiCperSecMemErr));
+
+
+    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
+    gdata = (AcpiGenericErrorData *)buffer;
+
+    /* Memory section */
+    memcpy(&(gdata->section_type_le), &mem_section_id_le,
+            sizeof(mem_section_id_le));
+
+    /* error severity is recoverable */
+    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
+    gdata->revision = 0x300; /* the revision number is 0x300 */
+    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
+
+    mem_err = (UefiCperSecMemErr *) (gdata + 1);
+
+    /* User space only handle the memory section CPER */
+
+    /* Hard code to Multi-bit ECC error */
+    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
+    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
+
+    /* Record the physical address at which the memory error occurred */
+    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
+    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
+
+    /* Write back the Generic Error Data Entry to guest memory */
+    cpu_physical_memory_write(error_block_address + current_block_length,
+        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
+
+    g_free(buffer);
+    return GHES_CPER_OK;
+}
+
+static void
+build_address(GArray *table_data, BIOSLinker *linker,
+    uint32_t dst_patched_offset, uint32_t src_offset,
+    uint8_t address_space_id , uint8_t  register_bit_width,
+    uint8_t register_bit_offset, uint8_t access_size)
+{
+    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
+        offsetof(struct AcpiGenericAddress, address);
+
+    /* Address space */
+    build_append_int_noprefix(table_data, address_space_id, 1);
+    /* register bit width */
+    build_append_int_noprefix(table_data, register_bit_width, 1);
+    /* register bit offset */
+    build_append_int_noprefix(table_data, register_bit_offset, 1);
+    /* access size */
+    build_append_int_noprefix(table_data, access_size, 1);
+    acpi_data_push(table_data, address_size);
+
+    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
+     * can retrieve and read it. the address size is 64 bits.
+     */
+    bios_linker_loader_add_pointer(linker,
+        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
+        GHES_ERRORS_FW_CFG_FILE, src_offset);
+}
+
+void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
+                                            BIOSLinker *linker)
+{
+    uint32_t ghes_start = table_data->len;
+    uint32_t address_size, error_status_address_offset;
+    uint32_t read_ack_register_offset, i;
+
+    address_size = sizeof(struct AcpiGenericAddress) -
+        offsetof(struct AcpiGenericAddress, address);
+
+    error_status_address_offset = ghes_start +
+        sizeof(AcpiHardwareErrorSourceTable) +
+        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
+        offsetof(struct AcpiGenericAddress, address);
+
+    read_ack_register_offset = ghes_start +
+        sizeof(AcpiHardwareErrorSourceTable) +
+        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
+        offsetof(struct AcpiGenericAddress, address);
+
+    acpi_data_push(hardware_error,
+        offsetof(struct hardware_errors_buffer, ack_value));
+    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
+        /* Initialize read ack register */
+        build_append_int_noprefix((void *)hardware_error, 1, 8);
+
+    /* Reserved the total size for ERRORS fw_cfg blob
+     */
+    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
+
+    /* Allocate guest memory for the Data fw_cfg blob */
+    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
+                            1, false);
+    /* Reserve table header size */
+    acpi_data_push(table_data, sizeof(AcpiTableHeader));
+
+    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
+
+    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
+        build_append_int_noprefix(table_data,
+            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
+        /* source id */
+        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
+        /* related source id */
+        build_append_int_noprefix(table_data, 0xffff, 2);
+        build_append_int_noprefix(table_data, 0, 1); /* flags */
+
+        /* Currently only enable SEA notification type to avoid the kernel
+         * warning, reserve the space for other notification error source
+         */
+        if (i == ACPI_HEST_NOTIFY_SEA) {
+            build_append_int_noprefix(table_data, 1, 1); /* enabled */
+        } else {
+            build_append_int_noprefix(table_data, 0, 1); /* enabled */
+        }
+
+        /* The number of error status block per generic hardware error source */
+        build_append_int_noprefix(table_data, 1, 4);
+        /* Max sections per record */
+        build_append_int_noprefix(table_data, 1, 4);
+        /* Max raw data length */
+        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
+
+        /* Build error status address*/
+        build_address(table_data, linker, error_status_address_offset + i *
+            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
+            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
+
+        /* Hardware error notification structure */
+        build_append_int_noprefix(table_data, i, 1); /* type */
+        /* length */
+        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
+        build_append_int_noprefix(table_data, 0, 26);
+
+        /* Error Status Block Length */
+        build_append_int_noprefix(table_data,
+            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
+
+        /* Build read ack register */
+        build_address(table_data, linker, read_ack_register_offset + i *
+            sizeof(AcpiGenericHardwareErrorSourceV2),
+            offsetof(struct hardware_errors_buffer, ack_value) +
+            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
+            4 /* QWord access */);
+
+        /* Read ack preserve */
+        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
+
+        /* Read ack write */
+        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
+    }
+
+    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
+        /* Patch address of generic error status block into
+         * the address register so OSPM can retrieve and read it.
+         */
+        bios_linker_loader_add_pointer(linker,
+            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
+            GHES_ERRORS_FW_CFG_FILE,
+            offsetof(struct hardware_errors_buffer, gesb) +
+            i * GHES_MAX_RAW_DATA_LENGTH);
+
+    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
+     * so QEMU can write the ERRORS there. The address is expected to be
+     * < 4GB, but write 64 bits anyway.
+     */
+    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
+        0, address_size, GHES_ERRORS_FW_CFG_FILE,
+        offsetof(struct hardware_errors_buffer, gesb));
+
+    build_header(linker, table_data,
+        (void *)(table_data->data + ghes_start), "HEST",
+        table_data->len - ghes_start, 1, NULL, "GHES");
+}
+
+static GhesState ges;
+void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
+{
+
+    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
+    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
+
+    /* Create a read-only fw_cfg file for GHES */
+    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
+                    size);
+    /* Create a read-write fw_cfg file for Address */
+    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
+        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
+}
+
+bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
+{
+    uint64_t error_block_addr;
+    uint64_t ack_value_addr, ack_value = 0;
+    int loop = 0, ack_value_size;
+    bool ret = GHES_CPER_FAIL;
+
+    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
+        offsetof(struct hardware_errors_buffer, ack_value)) /
+            GHES_ACPI_HEST_NOTIFY_RESERVED;
+
+    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
+        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
+        error_block_addr = le32_to_cpu(error_block_addr);
+
+        ack_value_addr = ges.ghes_addr_le -
+            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
+retry:
+        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
+        if (!ack_value) {
+            if (loop < 3) {
+                usleep(100 * 1000);
+                loop++;
+                goto retry;
+            } else {
+                error_report("Last time OSPM does not acknowledge the error,"
+                    " record CPER failed this time, set the ack value to"
+                    " avoid blocking next time CPER record! exit");
+                ack_value = 1;
+                cpu_physical_memory_write(ack_value_addr,
+                    &ack_value, ack_value_size);
+                return ret;
+            }
+        } else {
+            /* A zero value in ghes_addr means that BIOS has not yet written
+             * the address
+             */
+            if (error_block_addr) {
+                ack_value = 0;
+                cpu_physical_memory_write(ack_value_addr,
+                    &ack_value, ack_value_size);
+                ret = ghes_record_cper(error_block_addr, physical_address);
+            }
+        }
+    }
+    return ret;
+}
diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
index 3d78ff6..def1ec1 100644
--- a/hw/arm/virt-acpi-build.c
+++ b/hw/arm/virt-acpi-build.c
@@ -45,6 +45,7 @@
 #include "hw/arm/virt.h"
 #include "sysemu/numa.h"
 #include "kvm_arm.h"
+#include "hw/acpi/hest_ghes.h"
 
 #define ARM_SPI_BASE 32
 #define ACPI_POWER_BUTTON_DEVICE "PWRB"
@@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
     acpi_add_table(table_offsets, tables_blob);
     build_spcr(tables_blob, tables->linker, vms);
 
+    acpi_add_table(table_offsets, tables_blob);
+    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
+
     if (nb_numa_nodes > 0) {
         acpi_add_table(table_offsets, tables_blob);
         build_srat(tables_blob, tables->linker, vms);
@@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
     fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
                     acpi_data_len(tables.tcpalog));
 
+    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
+
     build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
                                               ACPI_BUILD_RSDP_FILE, 0);
 
diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
index 88d0738..7f7b55c 100644
--- a/include/hw/acpi/aml-build.h
+++ b/include/hw/acpi/aml-build.h
@@ -211,6 +211,7 @@ struct AcpiBuildTables {
     GArray *rsdp;
     GArray *tcpalog;
     GArray *vmgenid;
+    GArray *hardware_errors;
     BIOSLinker *linker;
 } AcpiBuildTables;
 
diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
new file mode 100644
index 0000000..0772756
--- /dev/null
+++ b/include/hw/acpi/hest_ghes.h
@@ -0,0 +1,47 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * Authors:
+ *   Dongjiu Geng <gengdongjiu@huawei.com>
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef ACPI_GHES_H
+#define ACPI_GHES_H
+
+#include "hw/acpi/bios-linker-loader.h"
+
+#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
+#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
+
+#define GHES_GAS_ADDRESS_OFFSET              4
+#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
+#define GHES_NOTIFICATION_STRUCTURE          32
+
+#define GHES_CPER_OK   1
+#define GHES_CPER_FAIL 0
+
+#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
+/* The max size in Bytes for one error block */
+#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
+
+
+typedef struct GhesState {
+    uint64_t ghes_addr_le;
+} GhesState;
+
+void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
+                            BIOSLinker *linker);
+void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
+bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
+#endif
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

This implements APEI GHES Table by passing the error CPER info
to the guest via a fw_cfg_blob. After a CPER info is recorded, an
SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
will be injected into the guest OS.

Below is the table layout, the max number of error soure is 11,
which is classified by notification type.

     etc/acpi/tables                               etc/hardware_errors
    ====================                    ==========================================
+ +--------------------------+            +------------------+
| | HEST                     |            |    address       |              +--------------+
| +--------------------------+            |    registers     |              | Error Status |
| | GHES0                    |            | +----------------+              | Data Block 0 |
| +--------------------------+ +--------->| |status_address0 |------------->| +------------+
| | .................        | |          | +----------------+              | |  CPER      |
| | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
| | .................        |   |        | +----------------+          |   | |  ....      |
| | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
| | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
| | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
+ +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
| | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
+ +--------------------------+   | |      | +----------------+        |     | |  CPER      |
| | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
| | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
| | .................        |     | |    | |  ............. |        |     | |  CPER      |
| | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
| | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
| | read_ack_write           |     |   |  | +----------------+        |     | +------------+
+ +--------------------------|     |   |                              |     | Error Status |
| | ...............          |     |   |                              |     | Data Block 10|
+ +--------------------------+     |   |                              +---->| +------------+
| | GHES10                   |     |   |                                    | |  CPER      |
+ +--------------------------+     |   |                                    | |  CPER      |
| | .................        |     |   |                                    | |  ....      |
| | error_status_address-----+-----+   |                                    | |  CPER      |
| | .................        |         |                                    +-+------------+
| | read_ack_register--------+---------+
| | read_ack_preserve        |
| | read_ack_write           |
+ +--------------------------+

For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
so user space must check the ack value to avoid read-write race condition.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 hw/acpi/aml-build.c         |   2 +
 hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
 hw/arm/virt-acpi-build.c    |   6 +
 include/hw/acpi/aml-build.h |   1 +
 include/hw/acpi/hest_ghes.h |  47 ++++++
 5 files changed, 401 insertions(+)
 create mode 100644 hw/acpi/hest_ghes.c
 create mode 100644 include/hw/acpi/hest_ghes.h

diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
index 36a6cc4..6849e5f 100644
--- a/hw/acpi/aml-build.c
+++ b/hw/acpi/aml-build.c
@@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
     tables->table_data = g_array_new(false, true /* clear */, 1);
     tables->tcpalog = g_array_new(false, true /* clear */, 1);
     tables->vmgenid = g_array_new(false, true /* clear */, 1);
+    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
     tables->linker = bios_linker_loader_init();
 }
 
@@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
     g_array_free(tables->table_data, true);
     g_array_free(tables->tcpalog, mfre);
     g_array_free(tables->vmgenid, mfre);
+    g_array_free(tables->hardware_errors, mfre);
 }
 
 /* Build rsdt table */
diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
new file mode 100644
index 0000000..ff6b5ef
--- /dev/null
+++ b/hw/acpi/hest_ghes.c
@@ -0,0 +1,345 @@
+/*
+ *  APEI GHES table Generation
+ *
+ *  Copyright (C) 2017 huawei.
+ *
+ *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "qmp-commands.h"
+#include "hw/acpi/acpi.h"
+#include "hw/acpi/aml-build.h"
+#include "hw/acpi/hest_ghes.h"
+#include "hw/nvram/fw_cfg.h"
+#include "sysemu/sysemu.h"
+#include "qemu/error-report.h"
+
+/* The structure that stands for the layout
+ * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
+ *
+ *           etc/hardware_errors
+ * ==========================================
+ * +------------------+
+ * |    address       |              +--------------+
+ * |    registers     |              | Error Status |
+ * | +----------------+              | Data Block 0 |
+ * | |status_address0 |------------->| +------------+
+ * | +----------------+              | |  CPER      |
+ * | |status_address1 |----------+   | |  CPER      |
+ * | +----------------+          |   | |  ....      |
+ * | |.............   |          |   | |  CPER      |
+ * | +----------------+          |   | +------------+
+ * | |status_address10|-----+    |   | Error Status |
+ * | +----------------+     |    |   | Data Block 1 |
+ * | |ack_value0      |     |    +-->| +------------+
+ * | +----------------+     |        | |  CPER      |
+ * | |ack_value1      |     |        | |  CPER      |
+ * | +----------------+     |        | |  ....      |
+ * | | .............  |     |        | |  CPER      |
+ * | +----------------+     |        +-+------------+
+ * | |ack_value10     |     |        | |..........  |
+ * | +----------------+     |        | +------------+
+ *                          |        | Error Status |
+ *                          |        | Data Block10 |
+ *                          +------->+------------+
+ *                                   | |  CPER      |
+ *                                   | |  CPER      |
+ *                                   | |  ....      |
+ *                                   | |  CPER      |
+ *                                   +-+------------+
+ */
+struct hardware_errors_buffer {
+    /* Generic Error Status Block register */
+    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
+    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
+    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
+};
+
+static int ghes_record_cper(uint64_t error_block_address,
+                                    uint64_t error_physical_addr)
+{
+    AcpiGenericErrorStatus block;
+    AcpiGenericErrorData *gdata;
+    UefiCperSecMemErr *mem_err;
+    uint64_t current_block_length;
+    unsigned char *buffer;
+    /* memory section */
+    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
+                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
+                              0x83, 0xB1};
+
+    cpu_physical_memory_read(error_block_address, &block,
+                                sizeof(AcpiGenericErrorStatus));
+
+    /* Get the current generic error status block length */
+    current_block_length = sizeof(AcpiGenericErrorStatus) +
+        le32_to_cpu(block.data_length);
+
+    /* If the Generic Error Status Block is NULL, update
+     * the block header
+     */
+    if (!block.block_status) {
+        block.block_status = ACPI_GEBS_UNCORRECTABLE;
+        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
+    }
+
+    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
+    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
+
+    /* check whether it runs out of the preallocated memory */
+    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
+       GHES_MAX_RAW_DATA_LENGTH) {
+        error_report("Record CPER out of boundary!!!");
+        return GHES_CPER_FAIL;
+    }
+
+    /* Write back the Generic Error Status Block to guest memory */
+    cpu_physical_memory_write(error_block_address, &block,
+        sizeof(AcpiGenericErrorStatus));
+
+    /* Fill in Generic Error Data Entry */
+    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
+                       sizeof(UefiCperSecMemErr));
+
+
+    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
+    gdata = (AcpiGenericErrorData *)buffer;
+
+    /* Memory section */
+    memcpy(&(gdata->section_type_le), &mem_section_id_le,
+            sizeof(mem_section_id_le));
+
+    /* error severity is recoverable */
+    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
+    gdata->revision = 0x300; /* the revision number is 0x300 */
+    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
+
+    mem_err = (UefiCperSecMemErr *) (gdata + 1);
+
+    /* User space only handle the memory section CPER */
+
+    /* Hard code to Multi-bit ECC error */
+    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
+    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
+
+    /* Record the physical address at which the memory error occurred */
+    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
+    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
+
+    /* Write back the Generic Error Data Entry to guest memory */
+    cpu_physical_memory_write(error_block_address + current_block_length,
+        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
+
+    g_free(buffer);
+    return GHES_CPER_OK;
+}
+
+static void
+build_address(GArray *table_data, BIOSLinker *linker,
+    uint32_t dst_patched_offset, uint32_t src_offset,
+    uint8_t address_space_id , uint8_t  register_bit_width,
+    uint8_t register_bit_offset, uint8_t access_size)
+{
+    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
+        offsetof(struct AcpiGenericAddress, address);
+
+    /* Address space */
+    build_append_int_noprefix(table_data, address_space_id, 1);
+    /* register bit width */
+    build_append_int_noprefix(table_data, register_bit_width, 1);
+    /* register bit offset */
+    build_append_int_noprefix(table_data, register_bit_offset, 1);
+    /* access size */
+    build_append_int_noprefix(table_data, access_size, 1);
+    acpi_data_push(table_data, address_size);
+
+    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
+     * can retrieve and read it. the address size is 64 bits.
+     */
+    bios_linker_loader_add_pointer(linker,
+        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
+        GHES_ERRORS_FW_CFG_FILE, src_offset);
+}
+
+void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
+                                            BIOSLinker *linker)
+{
+    uint32_t ghes_start = table_data->len;
+    uint32_t address_size, error_status_address_offset;
+    uint32_t read_ack_register_offset, i;
+
+    address_size = sizeof(struct AcpiGenericAddress) -
+        offsetof(struct AcpiGenericAddress, address);
+
+    error_status_address_offset = ghes_start +
+        sizeof(AcpiHardwareErrorSourceTable) +
+        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
+        offsetof(struct AcpiGenericAddress, address);
+
+    read_ack_register_offset = ghes_start +
+        sizeof(AcpiHardwareErrorSourceTable) +
+        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
+        offsetof(struct AcpiGenericAddress, address);
+
+    acpi_data_push(hardware_error,
+        offsetof(struct hardware_errors_buffer, ack_value));
+    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
+        /* Initialize read ack register */
+        build_append_int_noprefix((void *)hardware_error, 1, 8);
+
+    /* Reserved the total size for ERRORS fw_cfg blob
+     */
+    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
+
+    /* Allocate guest memory for the Data fw_cfg blob */
+    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
+                            1, false);
+    /* Reserve table header size */
+    acpi_data_push(table_data, sizeof(AcpiTableHeader));
+
+    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
+
+    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
+        build_append_int_noprefix(table_data,
+            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
+        /* source id */
+        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
+        /* related source id */
+        build_append_int_noprefix(table_data, 0xffff, 2);
+        build_append_int_noprefix(table_data, 0, 1); /* flags */
+
+        /* Currently only enable SEA notification type to avoid the kernel
+         * warning, reserve the space for other notification error source
+         */
+        if (i == ACPI_HEST_NOTIFY_SEA) {
+            build_append_int_noprefix(table_data, 1, 1); /* enabled */
+        } else {
+            build_append_int_noprefix(table_data, 0, 1); /* enabled */
+        }
+
+        /* The number of error status block per generic hardware error source */
+        build_append_int_noprefix(table_data, 1, 4);
+        /* Max sections per record */
+        build_append_int_noprefix(table_data, 1, 4);
+        /* Max raw data length */
+        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
+
+        /* Build error status address*/
+        build_address(table_data, linker, error_status_address_offset + i *
+            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
+            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
+
+        /* Hardware error notification structure */
+        build_append_int_noprefix(table_data, i, 1); /* type */
+        /* length */
+        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
+        build_append_int_noprefix(table_data, 0, 26);
+
+        /* Error Status Block Length */
+        build_append_int_noprefix(table_data,
+            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
+
+        /* Build read ack register */
+        build_address(table_data, linker, read_ack_register_offset + i *
+            sizeof(AcpiGenericHardwareErrorSourceV2),
+            offsetof(struct hardware_errors_buffer, ack_value) +
+            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
+            4 /* QWord access */);
+
+        /* Read ack preserve */
+        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
+
+        /* Read ack write */
+        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
+    }
+
+    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
+        /* Patch address of generic error status block into
+         * the address register so OSPM can retrieve and read it.
+         */
+        bios_linker_loader_add_pointer(linker,
+            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
+            GHES_ERRORS_FW_CFG_FILE,
+            offsetof(struct hardware_errors_buffer, gesb) +
+            i * GHES_MAX_RAW_DATA_LENGTH);
+
+    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
+     * so QEMU can write the ERRORS there. The address is expected to be
+     * < 4GB, but write 64 bits anyway.
+     */
+    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
+        0, address_size, GHES_ERRORS_FW_CFG_FILE,
+        offsetof(struct hardware_errors_buffer, gesb));
+
+    build_header(linker, table_data,
+        (void *)(table_data->data + ghes_start), "HEST",
+        table_data->len - ghes_start, 1, NULL, "GHES");
+}
+
+static GhesState ges;
+void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
+{
+
+    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
+    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
+
+    /* Create a read-only fw_cfg file for GHES */
+    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
+                    size);
+    /* Create a read-write fw_cfg file for Address */
+    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
+        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
+}
+
+bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
+{
+    uint64_t error_block_addr;
+    uint64_t ack_value_addr, ack_value = 0;
+    int loop = 0, ack_value_size;
+    bool ret = GHES_CPER_FAIL;
+
+    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
+        offsetof(struct hardware_errors_buffer, ack_value)) /
+            GHES_ACPI_HEST_NOTIFY_RESERVED;
+
+    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
+        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
+        error_block_addr = le32_to_cpu(error_block_addr);
+
+        ack_value_addr = ges.ghes_addr_le -
+            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
+retry:
+        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
+        if (!ack_value) {
+            if (loop < 3) {
+                usleep(100 * 1000);
+                loop++;
+                goto retry;
+            } else {
+                error_report("Last time OSPM does not acknowledge the error,"
+                    " record CPER failed this time, set the ack value to"
+                    " avoid blocking next time CPER record! exit");
+                ack_value = 1;
+                cpu_physical_memory_write(ack_value_addr,
+                    &ack_value, ack_value_size);
+                return ret;
+            }
+        } else {
+            /* A zero value in ghes_addr means that BIOS has not yet written
+             * the address
+             */
+            if (error_block_addr) {
+                ack_value = 0;
+                cpu_physical_memory_write(ack_value_addr,
+                    &ack_value, ack_value_size);
+                ret = ghes_record_cper(error_block_addr, physical_address);
+            }
+        }
+    }
+    return ret;
+}
diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
index 3d78ff6..def1ec1 100644
--- a/hw/arm/virt-acpi-build.c
+++ b/hw/arm/virt-acpi-build.c
@@ -45,6 +45,7 @@
 #include "hw/arm/virt.h"
 #include "sysemu/numa.h"
 #include "kvm_arm.h"
+#include "hw/acpi/hest_ghes.h"
 
 #define ARM_SPI_BASE 32
 #define ACPI_POWER_BUTTON_DEVICE "PWRB"
@@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
     acpi_add_table(table_offsets, tables_blob);
     build_spcr(tables_blob, tables->linker, vms);
 
+    acpi_add_table(table_offsets, tables_blob);
+    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
+
     if (nb_numa_nodes > 0) {
         acpi_add_table(table_offsets, tables_blob);
         build_srat(tables_blob, tables->linker, vms);
@@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
     fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
                     acpi_data_len(tables.tcpalog));
 
+    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
+
     build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
                                               ACPI_BUILD_RSDP_FILE, 0);
 
diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
index 88d0738..7f7b55c 100644
--- a/include/hw/acpi/aml-build.h
+++ b/include/hw/acpi/aml-build.h
@@ -211,6 +211,7 @@ struct AcpiBuildTables {
     GArray *rsdp;
     GArray *tcpalog;
     GArray *vmgenid;
+    GArray *hardware_errors;
     BIOSLinker *linker;
 } AcpiBuildTables;
 
diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
new file mode 100644
index 0000000..0772756
--- /dev/null
+++ b/include/hw/acpi/hest_ghes.h
@@ -0,0 +1,47 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * Authors:
+ *   Dongjiu Geng <gengdongjiu@huawei.com>
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef ACPI_GHES_H
+#define ACPI_GHES_H
+
+#include "hw/acpi/bios-linker-loader.h"
+
+#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
+#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
+
+#define GHES_GAS_ADDRESS_OFFSET              4
+#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
+#define GHES_NOTIFICATION_STRUCTURE          32
+
+#define GHES_CPER_OK   1
+#define GHES_CPER_FAIL 0
+
+#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
+/* The max size in Bytes for one error block */
+#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
+
+
+typedef struct GhesState {
+    uint64_t ghes_addr_le;
+} GhesState;
+
+void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
+                            BIOSLinker *linker);
+void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
+bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
+#endif
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: linux-arm-kernel

This implements APEI GHES Table by passing the error CPER info
to the guest via a fw_cfg_blob. After a CPER info is recorded, an
SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
will be injected into the guest OS.

Below is the table layout, the max number of error soure is 11,
which is classified by notification type.

     etc/acpi/tables                               etc/hardware_errors
    ====================                    ==========================================
+ +--------------------------+            +------------------+
| | HEST                     |            |    address       |              +--------------+
| +--------------------------+            |    registers     |              | Error Status |
| | GHES0                    |            | +----------------+              | Data Block 0 |
| +--------------------------+ +--------->| |status_address0 |------------->| +------------+
| | .................        | |          | +----------------+              | |  CPER      |
| | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
| | .................        |   |        | +----------------+          |   | |  ....      |
| | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
| | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
| | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
+ +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
| | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
+ +--------------------------+   | |      | +----------------+        |     | |  CPER      |
| | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
| | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
| | .................        |     | |    | |  ............. |        |     | |  CPER      |
| | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
| | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
| | read_ack_write           |     |   |  | +----------------+        |     | +------------+
+ +--------------------------|     |   |                              |     | Error Status |
| | ...............          |     |   |                              |     | Data Block 10|
+ +--------------------------+     |   |                              +---->| +------------+
| | GHES10                   |     |   |                                    | |  CPER      |
+ +--------------------------+     |   |                                    | |  CPER      |
| | .................        |     |   |                                    | |  ....      |
| | error_status_address-----+-----+   |                                    | |  CPER      |
| | .................        |         |                                    +-+------------+
| | read_ack_register--------+---------+
| | read_ack_preserve        |
| | read_ack_write           |
+ +--------------------------+

For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
so user space must check the ack value to avoid read-write race condition.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 hw/acpi/aml-build.c         |   2 +
 hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
 hw/arm/virt-acpi-build.c    |   6 +
 include/hw/acpi/aml-build.h |   1 +
 include/hw/acpi/hest_ghes.h |  47 ++++++
 5 files changed, 401 insertions(+)
 create mode 100644 hw/acpi/hest_ghes.c
 create mode 100644 include/hw/acpi/hest_ghes.h

diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
index 36a6cc4..6849e5f 100644
--- a/hw/acpi/aml-build.c
+++ b/hw/acpi/aml-build.c
@@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
     tables->table_data = g_array_new(false, true /* clear */, 1);
     tables->tcpalog = g_array_new(false, true /* clear */, 1);
     tables->vmgenid = g_array_new(false, true /* clear */, 1);
+    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
     tables->linker = bios_linker_loader_init();
 }
 
@@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
     g_array_free(tables->table_data, true);
     g_array_free(tables->tcpalog, mfre);
     g_array_free(tables->vmgenid, mfre);
+    g_array_free(tables->hardware_errors, mfre);
 }
 
 /* Build rsdt table */
diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
new file mode 100644
index 0000000..ff6b5ef
--- /dev/null
+++ b/hw/acpi/hest_ghes.c
@@ -0,0 +1,345 @@
+/*
+ *  APEI GHES table Generation
+ *
+ *  Copyright (C) 2017 huawei.
+ *
+ *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "qmp-commands.h"
+#include "hw/acpi/acpi.h"
+#include "hw/acpi/aml-build.h"
+#include "hw/acpi/hest_ghes.h"
+#include "hw/nvram/fw_cfg.h"
+#include "sysemu/sysemu.h"
+#include "qemu/error-report.h"
+
+/* The structure that stands for the layout
+ * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
+ *
+ *           etc/hardware_errors
+ * ==========================================
+ * +------------------+
+ * |    address       |              +--------------+
+ * |    registers     |              | Error Status |
+ * | +----------------+              | Data Block 0 |
+ * | |status_address0 |------------->| +------------+
+ * | +----------------+              | |  CPER      |
+ * | |status_address1 |----------+   | |  CPER      |
+ * | +----------------+          |   | |  ....      |
+ * | |.............   |          |   | |  CPER      |
+ * | +----------------+          |   | +------------+
+ * | |status_address10|-----+    |   | Error Status |
+ * | +----------------+     |    |   | Data Block 1 |
+ * | |ack_value0      |     |    +-->| +------------+
+ * | +----------------+     |        | |  CPER      |
+ * | |ack_value1      |     |        | |  CPER      |
+ * | +----------------+     |        | |  ....      |
+ * | | .............  |     |        | |  CPER      |
+ * | +----------------+     |        +-+------------+
+ * | |ack_value10     |     |        | |..........  |
+ * | +----------------+     |        | +------------+
+ *                          |        | Error Status |
+ *                          |        | Data Block10 |
+ *                          +------->+------------+
+ *                                   | |  CPER      |
+ *                                   | |  CPER      |
+ *                                   | |  ....      |
+ *                                   | |  CPER      |
+ *                                   +-+------------+
+ */
+struct hardware_errors_buffer {
+    /* Generic Error Status Block register */
+    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
+    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
+    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
+};
+
+static int ghes_record_cper(uint64_t error_block_address,
+                                    uint64_t error_physical_addr)
+{
+    AcpiGenericErrorStatus block;
+    AcpiGenericErrorData *gdata;
+    UefiCperSecMemErr *mem_err;
+    uint64_t current_block_length;
+    unsigned char *buffer;
+    /* memory section */
+    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
+                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
+                              0x83, 0xB1};
+
+    cpu_physical_memory_read(error_block_address, &block,
+                                sizeof(AcpiGenericErrorStatus));
+
+    /* Get the current generic error status block length */
+    current_block_length = sizeof(AcpiGenericErrorStatus) +
+        le32_to_cpu(block.data_length);
+
+    /* If the Generic Error Status Block is NULL, update
+     * the block header
+     */
+    if (!block.block_status) {
+        block.block_status = ACPI_GEBS_UNCORRECTABLE;
+        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
+    }
+
+    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
+    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
+
+    /* check whether it runs out of the preallocated memory */
+    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
+       GHES_MAX_RAW_DATA_LENGTH) {
+        error_report("Record CPER out of boundary!!!");
+        return GHES_CPER_FAIL;
+    }
+
+    /* Write back the Generic Error Status Block to guest memory */
+    cpu_physical_memory_write(error_block_address, &block,
+        sizeof(AcpiGenericErrorStatus));
+
+    /* Fill in Generic Error Data Entry */
+    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
+                       sizeof(UefiCperSecMemErr));
+
+
+    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
+    gdata = (AcpiGenericErrorData *)buffer;
+
+    /* Memory section */
+    memcpy(&(gdata->section_type_le), &mem_section_id_le,
+            sizeof(mem_section_id_le));
+
+    /* error severity is recoverable */
+    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
+    gdata->revision = 0x300; /* the revision number is 0x300 */
+    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
+
+    mem_err = (UefiCperSecMemErr *) (gdata + 1);
+
+    /* User space only handle the memory section CPER */
+
+    /* Hard code to Multi-bit ECC error */
+    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
+    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
+
+    /* Record the physical address at which the memory error occurred */
+    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
+    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
+
+    /* Write back the Generic Error Data Entry to guest memory */
+    cpu_physical_memory_write(error_block_address + current_block_length,
+        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
+
+    g_free(buffer);
+    return GHES_CPER_OK;
+}
+
+static void
+build_address(GArray *table_data, BIOSLinker *linker,
+    uint32_t dst_patched_offset, uint32_t src_offset,
+    uint8_t address_space_id , uint8_t  register_bit_width,
+    uint8_t register_bit_offset, uint8_t access_size)
+{
+    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
+        offsetof(struct AcpiGenericAddress, address);
+
+    /* Address space */
+    build_append_int_noprefix(table_data, address_space_id, 1);
+    /* register bit width */
+    build_append_int_noprefix(table_data, register_bit_width, 1);
+    /* register bit offset */
+    build_append_int_noprefix(table_data, register_bit_offset, 1);
+    /* access size */
+    build_append_int_noprefix(table_data, access_size, 1);
+    acpi_data_push(table_data, address_size);
+
+    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
+     * can retrieve and read it. the address size is 64 bits.
+     */
+    bios_linker_loader_add_pointer(linker,
+        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
+        GHES_ERRORS_FW_CFG_FILE, src_offset);
+}
+
+void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
+                                            BIOSLinker *linker)
+{
+    uint32_t ghes_start = table_data->len;
+    uint32_t address_size, error_status_address_offset;
+    uint32_t read_ack_register_offset, i;
+
+    address_size = sizeof(struct AcpiGenericAddress) -
+        offsetof(struct AcpiGenericAddress, address);
+
+    error_status_address_offset = ghes_start +
+        sizeof(AcpiHardwareErrorSourceTable) +
+        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
+        offsetof(struct AcpiGenericAddress, address);
+
+    read_ack_register_offset = ghes_start +
+        sizeof(AcpiHardwareErrorSourceTable) +
+        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
+        offsetof(struct AcpiGenericAddress, address);
+
+    acpi_data_push(hardware_error,
+        offsetof(struct hardware_errors_buffer, ack_value));
+    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
+        /* Initialize read ack register */
+        build_append_int_noprefix((void *)hardware_error, 1, 8);
+
+    /* Reserved the total size for ERRORS fw_cfg blob
+     */
+    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
+
+    /* Allocate guest memory for the Data fw_cfg blob */
+    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
+                            1, false);
+    /* Reserve table header size */
+    acpi_data_push(table_data, sizeof(AcpiTableHeader));
+
+    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
+
+    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
+        build_append_int_noprefix(table_data,
+            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
+        /* source id */
+        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
+        /* related source id */
+        build_append_int_noprefix(table_data, 0xffff, 2);
+        build_append_int_noprefix(table_data, 0, 1); /* flags */
+
+        /* Currently only enable SEA notification type to avoid the kernel
+         * warning, reserve the space for other notification error source
+         */
+        if (i == ACPI_HEST_NOTIFY_SEA) {
+            build_append_int_noprefix(table_data, 1, 1); /* enabled */
+        } else {
+            build_append_int_noprefix(table_data, 0, 1); /* enabled */
+        }
+
+        /* The number of error status block per generic hardware error source */
+        build_append_int_noprefix(table_data, 1, 4);
+        /* Max sections per record */
+        build_append_int_noprefix(table_data, 1, 4);
+        /* Max raw data length */
+        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
+
+        /* Build error status address*/
+        build_address(table_data, linker, error_status_address_offset + i *
+            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
+            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
+
+        /* Hardware error notification structure */
+        build_append_int_noprefix(table_data, i, 1); /* type */
+        /* length */
+        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
+        build_append_int_noprefix(table_data, 0, 26);
+
+        /* Error Status Block Length */
+        build_append_int_noprefix(table_data,
+            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
+
+        /* Build read ack register */
+        build_address(table_data, linker, read_ack_register_offset + i *
+            sizeof(AcpiGenericHardwareErrorSourceV2),
+            offsetof(struct hardware_errors_buffer, ack_value) +
+            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
+            4 /* QWord access */);
+
+        /* Read ack preserve */
+        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
+
+        /* Read ack write */
+        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
+    }
+
+    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
+        /* Patch address of generic error status block into
+         * the address register so OSPM can retrieve and read it.
+         */
+        bios_linker_loader_add_pointer(linker,
+            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
+            GHES_ERRORS_FW_CFG_FILE,
+            offsetof(struct hardware_errors_buffer, gesb) +
+            i * GHES_MAX_RAW_DATA_LENGTH);
+
+    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
+     * so QEMU can write the ERRORS there. The address is expected to be
+     * < 4GB, but write 64 bits anyway.
+     */
+    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
+        0, address_size, GHES_ERRORS_FW_CFG_FILE,
+        offsetof(struct hardware_errors_buffer, gesb));
+
+    build_header(linker, table_data,
+        (void *)(table_data->data + ghes_start), "HEST",
+        table_data->len - ghes_start, 1, NULL, "GHES");
+}
+
+static GhesState ges;
+void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
+{
+
+    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
+    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
+
+    /* Create a read-only fw_cfg file for GHES */
+    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
+                    size);
+    /* Create a read-write fw_cfg file for Address */
+    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
+        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
+}
+
+bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
+{
+    uint64_t error_block_addr;
+    uint64_t ack_value_addr, ack_value = 0;
+    int loop = 0, ack_value_size;
+    bool ret = GHES_CPER_FAIL;
+
+    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
+        offsetof(struct hardware_errors_buffer, ack_value)) /
+            GHES_ACPI_HEST_NOTIFY_RESERVED;
+
+    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
+        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
+        error_block_addr = le32_to_cpu(error_block_addr);
+
+        ack_value_addr = ges.ghes_addr_le -
+            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
+retry:
+        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
+        if (!ack_value) {
+            if (loop < 3) {
+                usleep(100 * 1000);
+                loop++;
+                goto retry;
+            } else {
+                error_report("Last time OSPM does not acknowledge the error,"
+                    " record CPER failed this time, set the ack value to"
+                    " avoid blocking next time CPER record! exit");
+                ack_value = 1;
+                cpu_physical_memory_write(ack_value_addr,
+                    &ack_value, ack_value_size);
+                return ret;
+            }
+        } else {
+            /* A zero value in ghes_addr means that BIOS has not yet written
+             * the address
+             */
+            if (error_block_addr) {
+                ack_value = 0;
+                cpu_physical_memory_write(ack_value_addr,
+                    &ack_value, ack_value_size);
+                ret = ghes_record_cper(error_block_addr, physical_address);
+            }
+        }
+    }
+    return ret;
+}
diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
index 3d78ff6..def1ec1 100644
--- a/hw/arm/virt-acpi-build.c
+++ b/hw/arm/virt-acpi-build.c
@@ -45,6 +45,7 @@
 #include "hw/arm/virt.h"
 #include "sysemu/numa.h"
 #include "kvm_arm.h"
+#include "hw/acpi/hest_ghes.h"
 
 #define ARM_SPI_BASE 32
 #define ACPI_POWER_BUTTON_DEVICE "PWRB"
@@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
     acpi_add_table(table_offsets, tables_blob);
     build_spcr(tables_blob, tables->linker, vms);
 
+    acpi_add_table(table_offsets, tables_blob);
+    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
+
     if (nb_numa_nodes > 0) {
         acpi_add_table(table_offsets, tables_blob);
         build_srat(tables_blob, tables->linker, vms);
@@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
     fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
                     acpi_data_len(tables.tcpalog));
 
+    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
+
     build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
                                               ACPI_BUILD_RSDP_FILE, 0);
 
diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
index 88d0738..7f7b55c 100644
--- a/include/hw/acpi/aml-build.h
+++ b/include/hw/acpi/aml-build.h
@@ -211,6 +211,7 @@ struct AcpiBuildTables {
     GArray *rsdp;
     GArray *tcpalog;
     GArray *vmgenid;
+    GArray *hardware_errors;
     BIOSLinker *linker;
 } AcpiBuildTables;
 
diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
new file mode 100644
index 0000000..0772756
--- /dev/null
+++ b/include/hw/acpi/hest_ghes.h
@@ -0,0 +1,47 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * Authors:
+ *   Dongjiu Geng <gengdongjiu@huawei.com>
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef ACPI_GHES_H
+#define ACPI_GHES_H
+
+#include "hw/acpi/bios-linker-loader.h"
+
+#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
+#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
+
+#define GHES_GAS_ADDRESS_OFFSET              4
+#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
+#define GHES_NOTIFICATION_STRUCTURE          32
+
+#define GHES_CPER_OK   1
+#define GHES_CPER_FAIL 0
+
+#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
+/* The max size in Bytes for one error block */
+#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
+
+
+typedef struct GhesState {
+    uint64_t ghes_addr_le;
+} GhesState;
+
+void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
+                            BIOSLinker *linker);
+void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
+bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
+#endif
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
  2017-08-18 14:23 ` Dongjiu Geng
  (?)
  (?)
@ 2017-08-18 14:23   ` Dongjiu Geng
  -1 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

Add CONFIG_ACPI_APEI configuration in the Makefile and
enable it in the arm-softmmu.mak

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 default-configs/arm-softmmu.mak | 1 +
 hw/acpi/Makefile.objs           | 1 +
 2 files changed, 2 insertions(+)

diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
index bbdd3c1..c362113 100644
--- a/default-configs/arm-softmmu.mak
+++ b/default-configs/arm-softmmu.mak
@@ -129,3 +129,4 @@ CONFIG_ACPI=y
 CONFIG_SMBIOS=y
 CONFIG_ASPEED_SOC=y
 CONFIG_GPIO_KEY=y
+CONFIG_ACPI_APEI=y
diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
index 11c35bc..bafb148 100644
--- a/hw/acpi/Makefile.objs
+++ b/hw/acpi/Makefile.objs
@@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
 common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
 common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
 common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
+common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
 common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
 
 common-obj-y += acpi_interface.o
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

Add CONFIG_ACPI_APEI configuration in the Makefile and
enable it in the arm-softmmu.mak

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 default-configs/arm-softmmu.mak | 1 +
 hw/acpi/Makefile.objs           | 1 +
 2 files changed, 2 insertions(+)

diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
index bbdd3c1..c362113 100644
--- a/default-configs/arm-softmmu.mak
+++ b/default-configs/arm-softmmu.mak
@@ -129,3 +129,4 @@ CONFIG_ACPI=y
 CONFIG_SMBIOS=y
 CONFIG_ASPEED_SOC=y
 CONFIG_GPIO_KEY=y
+CONFIG_ACPI_APEI=y
diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
index 11c35bc..bafb148 100644
--- a/hw/acpi/Makefile.objs
+++ b/hw/acpi/Makefile.objs
@@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
 common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
 common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
 common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
+common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
 common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
 
 common-obj-y += acpi_interface.o
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [Qemu-devel] [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

Add CONFIG_ACPI_APEI configuration in the Makefile and
enable it in the arm-softmmu.mak

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 default-configs/arm-softmmu.mak | 1 +
 hw/acpi/Makefile.objs           | 1 +
 2 files changed, 2 insertions(+)

diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
index bbdd3c1..c362113 100644
--- a/default-configs/arm-softmmu.mak
+++ b/default-configs/arm-softmmu.mak
@@ -129,3 +129,4 @@ CONFIG_ACPI=y
 CONFIG_SMBIOS=y
 CONFIG_ASPEED_SOC=y
 CONFIG_GPIO_KEY=y
+CONFIG_ACPI_APEI=y
diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
index 11c35bc..bafb148 100644
--- a/hw/acpi/Makefile.objs
+++ b/hw/acpi/Makefile.objs
@@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
 common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
 common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
 common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
+common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
 common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
 
 common-obj-y += acpi_interface.o
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: linux-arm-kernel

Add CONFIG_ACPI_APEI configuration in the Makefile and
enable it in the arm-softmmu.mak

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 default-configs/arm-softmmu.mak | 1 +
 hw/acpi/Makefile.objs           | 1 +
 2 files changed, 2 insertions(+)

diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
index bbdd3c1..c362113 100644
--- a/default-configs/arm-softmmu.mak
+++ b/default-configs/arm-softmmu.mak
@@ -129,3 +129,4 @@ CONFIG_ACPI=y
 CONFIG_SMBIOS=y
 CONFIG_ASPEED_SOC=y
 CONFIG_GPIO_KEY=y
+CONFIG_ACPI_APEI=y
diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
index 11c35bc..bafb148 100644
--- a/hw/acpi/Makefile.objs
+++ b/hw/acpi/Makefile.objs
@@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
 common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
 common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
 common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
+common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
 common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
 
 common-obj-y += acpi_interface.o
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
  2017-08-18 14:23 ` Dongjiu Geng
                     ` (2 preceding siblings ...)
  (?)
@ 2017-08-18 14:23   ` Dongjiu Geng
  -1 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: zhengqiang10, wuquanming, huangshaoyu, linuxarm, gengdongjiu

check if kvm supports guest RAS EXTENSION. if so, set
corresponding feature bit for vcpu.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 linux-headers/linux/kvm.h | 1 +
 target/arm/cpu.h          | 3 +++
 target/arm/kvm64.c        | 8 ++++++++
 3 files changed, 12 insertions(+)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 7971a4f..2aa176e 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_PPC_SMT_POSSIBLE 147
 #define KVM_CAP_HYPERV_SYNIC2 148
 #define KVM_CAP_HYPERV_VP_INDEX 149
+#define KVM_CAP_ARM_RAS_EXTENSION 150
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index b39d64a..6b0961b 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -611,6 +611,8 @@ struct ARMCPU {
 
     /* CPU has memory protection unit */
     bool has_mpu;
+    /* CPU has ras extension unit */
+    bool has_ras_extension;
     /* PMSAv7 MPU number of supported regions */
     uint32_t pmsav7_dregion;
 
@@ -1229,6 +1231,7 @@ enum arm_features {
     ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
     ARM_FEATURE_PMU, /* has PMU support */
     ARM_FEATURE_VBAR, /* has cp15 VBAR */
+    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */
 };
 
 static inline int arm_feature(CPUARMState *env, int feature)
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index a16abc8..0781367 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
         unset_feature(&env->features, ARM_FEATURE_PMU);
     }
 
+    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
+        cpu->has_ras_extension = true;
+        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
+    } else {
+        cpu->has_ras_extension = false;
+        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
+    }
+
     /* Do KVM_ARM_VCPU_INIT ioctl */
     ret = kvm_arm_vcpu_init(cs);
     if (ret) {
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

check if kvm supports guest RAS EXTENSION. if so, set
corresponding feature bit for vcpu.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 linux-headers/linux/kvm.h | 1 +
 target/arm/cpu.h          | 3 +++
 target/arm/kvm64.c        | 8 ++++++++
 3 files changed, 12 insertions(+)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 7971a4f..2aa176e 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_PPC_SMT_POSSIBLE 147
 #define KVM_CAP_HYPERV_SYNIC2 148
 #define KVM_CAP_HYPERV_VP_INDEX 149
+#define KVM_CAP_ARM_RAS_EXTENSION 150
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index b39d64a..6b0961b 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -611,6 +611,8 @@ struct ARMCPU {
 
     /* CPU has memory protection unit */
     bool has_mpu;
+    /* CPU has ras extension unit */
+    bool has_ras_extension;
     /* PMSAv7 MPU number of supported regions */
     uint32_t pmsav7_dregion;
 
@@ -1229,6 +1231,7 @@ enum arm_features {
     ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
     ARM_FEATURE_PMU, /* has PMU support */
     ARM_FEATURE_VBAR, /* has cp15 VBAR */
+    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */
 };
 
 static inline int arm_feature(CPUARMState *env, int feature)
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index a16abc8..0781367 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
         unset_feature(&env->features, ARM_FEATURE_PMU);
     }
 
+    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
+        cpu->has_ras_extension = true;
+        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
+    } else {
+        cpu->has_ras_extension = false;
+        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
+    }
+
     /* Do KVM_ARM_VCPU_INIT ioctl */
     ret = kvm_arm_vcpu_init(cs);
     if (ret) {
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: zhengqiang10, wuquanming, huangshaoyu, linuxarm, gengdongjiu

check if kvm supports guest RAS EXTENSION. if so, set
corresponding feature bit for vcpu.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 linux-headers/linux/kvm.h | 1 +
 target/arm/cpu.h          | 3 +++
 target/arm/kvm64.c        | 8 ++++++++
 3 files changed, 12 insertions(+)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 7971a4f..2aa176e 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_PPC_SMT_POSSIBLE 147
 #define KVM_CAP_HYPERV_SYNIC2 148
 #define KVM_CAP_HYPERV_VP_INDEX 149
+#define KVM_CAP_ARM_RAS_EXTENSION 150
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index b39d64a..6b0961b 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -611,6 +611,8 @@ struct ARMCPU {
 
     /* CPU has memory protection unit */
     bool has_mpu;
+    /* CPU has ras extension unit */
+    bool has_ras_extension;
     /* PMSAv7 MPU number of supported regions */
     uint32_t pmsav7_dregion;
 
@@ -1229,6 +1231,7 @@ enum arm_features {
     ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
     ARM_FEATURE_PMU, /* has PMU support */
     ARM_FEATURE_VBAR, /* has cp15 VBAR */
+    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */
 };
 
 static inline int arm_feature(CPUARMState *env, int feature)
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index a16abc8..0781367 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
         unset_feature(&env->features, ARM_FEATURE_PMU);
     }
 
+    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
+        cpu->has_ras_extension = true;
+        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
+    } else {
+        cpu->has_ras_extension = false;
+        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
+    }
+
     /* Do KVM_ARM_VCPU_INIT ioctl */
     ret = kvm_arm_vcpu_init(cs);
     if (ret) {
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [Qemu-devel] [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

check if kvm supports guest RAS EXTENSION. if so, set
corresponding feature bit for vcpu.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 linux-headers/linux/kvm.h | 1 +
 target/arm/cpu.h          | 3 +++
 target/arm/kvm64.c        | 8 ++++++++
 3 files changed, 12 insertions(+)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 7971a4f..2aa176e 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_PPC_SMT_POSSIBLE 147
 #define KVM_CAP_HYPERV_SYNIC2 148
 #define KVM_CAP_HYPERV_VP_INDEX 149
+#define KVM_CAP_ARM_RAS_EXTENSION 150
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index b39d64a..6b0961b 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -611,6 +611,8 @@ struct ARMCPU {
 
     /* CPU has memory protection unit */
     bool has_mpu;
+    /* CPU has ras extension unit */
+    bool has_ras_extension;
     /* PMSAv7 MPU number of supported regions */
     uint32_t pmsav7_dregion;
 
@@ -1229,6 +1231,7 @@ enum arm_features {
     ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
     ARM_FEATURE_PMU, /* has PMU support */
     ARM_FEATURE_VBAR, /* has cp15 VBAR */
+    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */
 };
 
 static inline int arm_feature(CPUARMState *env, int feature)
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index a16abc8..0781367 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
         unset_feature(&env->features, ARM_FEATURE_PMU);
     }
 
+    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
+        cpu->has_ras_extension = true;
+        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
+    } else {
+        cpu->has_ras_extension = false;
+        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
+    }
+
     /* Do KVM_ARM_VCPU_INIT ioctl */
     ret = kvm_arm_vcpu_init(cs);
     if (ret) {
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: linux-arm-kernel

check if kvm supports guest RAS EXTENSION. if so, set
corresponding feature bit for vcpu.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
---
 linux-headers/linux/kvm.h | 1 +
 target/arm/cpu.h          | 3 +++
 target/arm/kvm64.c        | 8 ++++++++
 3 files changed, 12 insertions(+)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 7971a4f..2aa176e 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_PPC_SMT_POSSIBLE 147
 #define KVM_CAP_HYPERV_SYNIC2 148
 #define KVM_CAP_HYPERV_VP_INDEX 149
+#define KVM_CAP_ARM_RAS_EXTENSION 150
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index b39d64a..6b0961b 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -611,6 +611,8 @@ struct ARMCPU {
 
     /* CPU has memory protection unit */
     bool has_mpu;
+    /* CPU has ras extension unit */
+    bool has_ras_extension;
     /* PMSAv7 MPU number of supported regions */
     uint32_t pmsav7_dregion;
 
@@ -1229,6 +1231,7 @@ enum arm_features {
     ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
     ARM_FEATURE_PMU, /* has PMU support */
     ARM_FEATURE_VBAR, /* has cp15 VBAR */
+    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */
 };
 
 static inline int arm_feature(CPUARMState *env, int feature)
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index a16abc8..0781367 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
         unset_feature(&env->features, ARM_FEATURE_PMU);
     }
 
+    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
+        cpu->has_ras_extension = true;
+        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
+    } else {
+        cpu->has_ras_extension = false;
+        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
+    }
+
     /* Do KVM_ARM_VCPU_INIT ioctl */
     ret = kvm_arm_vcpu_init(cs);
     if (ret) {
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 5/6] target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
  2017-08-18 14:23 ` Dongjiu Geng
  (?)
  (?)
@ 2017-08-18 14:23   ` Dongjiu Geng
  -1 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

Add SIGBUS signal handler. In this handler, it checks
the exception type, translates the host VA which is
delivered by host or KVM to guest PA, then fills this
PA to CPER, finally injects a Error to guest OS through
KVM.

Add synchronous external abort injection logic, setup
spsr_elx, esr_elx, PSTATE, far_elx, elr_elx etc, when
switch to guest OS, it will jump to the synchronous
external abort vector table entry.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
Signed-off-by: Quanming Wu <wuquanming@huawei.com>
---
 include/sysemu/kvm.h          |   2 +-
 linux-headers/asm-arm64/kvm.h |   5 ++
 target/arm/internals.h        |  13 ++++
 target/arm/kvm.c              |  34 ++++++++++
 target/arm/kvm64.c            | 150 ++++++++++++++++++++++++++++++++++++++++++
 target/arm/kvm_arm.h          |   1 +
 6 files changed, 204 insertions(+), 1 deletion(-)

diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 3a458f5..90c1605 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -361,7 +361,7 @@ bool kvm_vcpu_id_is_valid(int vcpu_id);
 /* Returns VCPU ID to be used on KVM_CREATE_VCPU ioctl() */
 unsigned long kvm_arch_vcpu_id(CPUState *cpu);
 
-#ifdef TARGET_I386
+#if defined(TARGET_I386) || defined(TARGET_AARCH64)
 #define KVM_HAVE_MCE_INJECTION 1
 void kvm_arch_on_sigbus_vcpu(CPUState *cpu, int code, void *addr);
 #endif
diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h
index d254700..5909c30 100644
--- a/linux-headers/asm-arm64/kvm.h
+++ b/linux-headers/asm-arm64/kvm.h
@@ -181,6 +181,11 @@ struct kvm_arch_memory_slot {
 #define KVM_REG_ARM64_SYSREG_OP2_MASK	0x0000000000000007
 #define KVM_REG_ARM64_SYSREG_OP2_SHIFT	0
 
+/* AArch64 fault registers */
+#define KVM_REG_ARM64_FAULT             (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
+#define KVM_REG_ARM64_FAULT_ESR_EC      (0)
+#define KVM_REG_ARM64_FAULT_FAR         (1)
+
 #define ARM64_SYS_REG_SHIFT_MASK(x,n) \
 	(((x) << KVM_REG_ARM64_SYSREG_ ## n ## _SHIFT) & \
 	KVM_REG_ARM64_SYSREG_ ## n ## _MASK)
diff --git a/target/arm/internals.h b/target/arm/internals.h
index 1f6efef..fc0ad6d 100644
--- a/target/arm/internals.h
+++ b/target/arm/internals.h
@@ -235,6 +235,19 @@ enum arm_exception_class {
 #define ARM_EL_ISV_SHIFT 24
 #define ARM_EL_IL (1 << ARM_EL_IL_SHIFT)
 #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
+#define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
+#define ARM_EL_FSC_TYPE (0x3C)
+
+#define FSC_SEA         (0x10)
+#define FSC_SEA_TTW0    (0x14)
+#define FSC_SEA_TTW1    (0x15)
+#define FSC_SEA_TTW2    (0x16)
+#define FSC_SEA_TTW3    (0x17)
+#define FSC_SECC        (0x18)
+#define FSC_SECC_TTW0   (0x1c)
+#define FSC_SECC_TTW1   (0x1d)
+#define FSC_SECC_TTW2   (0x1e)
+#define FSC_SECC_TTW3   (0x1f)
 
 /* Utility functions for constructing various kinds of syndrome value.
  * Note that in general we follow the AArch64 syndrome values; in a
diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index 7c17f0d..2e1716a 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -129,6 +129,39 @@ void kvm_arm_destroy_scratch_host_vcpu(int *fdarray)
     }
 }
 
+typedef struct HWPoisonPage {
+    ram_addr_t ram_addr;
+    QLIST_ENTRY(HWPoisonPage) list;
+} HWPoisonPage;
+
+static QLIST_HEAD(, HWPoisonPage) hwpoison_page_list =
+    QLIST_HEAD_INITIALIZER(hwpoison_page_list);
+
+static void kvm_unpoison_all(void *param)
+{
+    HWPoisonPage *page, *next_page;
+
+    QLIST_FOREACH_SAFE(page, &hwpoison_page_list, list, next_page) {
+        QLIST_REMOVE(page, list);
+        qemu_ram_remap(page->ram_addr, TARGET_PAGE_SIZE);
+        g_free(page);
+    }
+}
+
+void kvm_hwpoison_page_add(ram_addr_t ram_addr)
+{
+    HWPoisonPage *page;
+
+    QLIST_FOREACH(page, &hwpoison_page_list, list) {
+        if (page->ram_addr == ram_addr) {
+            return;
+        }
+    }
+    page = g_new(HWPoisonPage, 1);
+    page->ram_addr = ram_addr;
+    QLIST_INSERT_HEAD(&hwpoison_page_list, page, list);
+}
+
 static void kvm_arm_host_cpu_class_init(ObjectClass *oc, void *data)
 {
     ARMHostCPUClass *ahcc = ARM_HOST_CPU_CLASS(oc);
@@ -182,6 +215,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
 
     cap_has_mp_state = kvm_check_extension(s, KVM_CAP_MP_STATE);
 
+    qemu_register_reset(kvm_unpoison_all, NULL);
     type_register_static(&host_arm_cpu_type_info);
 
     return 0;
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index 0781367..d3bdab2 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -27,6 +27,8 @@
 #include "kvm_arm.h"
 #include "internals.h"
 #include "hw/arm/arm.h"
+#include "hw/acpi/acpi-defs.h"
+#include "hw/acpi/hest_ghes.h"
 
 static bool have_guest_debug;
 
@@ -590,6 +592,79 @@ int kvm_arm_cpreg_level(uint64_t regidx)
     return KVM_PUT_RUNTIME_STATE;
 }
 
+static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
+{
+    int i;
+
+    for (i = 0; i < cpu->cpreg_array_len; i++) {
+        uint32_t regidx = kvm_to_cpreg_id(cpu->cpreg_indexes[i]);
+        const ARMCPRegInfo *ri;
+        ri = get_arm_cp_reginfo(cpu->cp_regs, regidx);
+        if (!ri) {
+            continue;
+        }
+
+        if (ri->type & ARM_CP_NO_RAW) {
+            continue;
+        }
+
+        if (ri->fieldoffset == fieldoffset) {
+            cpu->cpreg_values[i] = read_raw_cp_reg(&cpu->env, ri);
+            return 0;
+        }
+    }
+    return -EINVAL;
+}
+
+/* Inject synchronous external abort */
+static int kvm_inject_arm_sea(CPUState *c)
+{
+    ARMCPU *cpu = ARM_CPU(c);
+    CPUARMState *env = &cpu->env;
+    unsigned long cpsr = pstate_read(env);
+    uint32_t esr = 0;
+    int ret;
+
+    c->exception_index = EXCP_DATA_ABORT;
+    /* Inject the exception to El1 */
+    env->exception.target_el = 1;
+    CPUClass *cc = CPU_GET_CLASS(c);
+
+    esr |= (EC_DATAABORT << ARM_EL_EC_SHIFT);
+    /* This exception syndrome includes {I,D}FSC in the bits [5:0]
+     */
+    esr |= (env->exception.syndrome & 0x3f);
+
+    /* This exception is EL0 or EL1 fault. */
+    if ((cpsr & 0xf) == PSTATE_MODE_EL0t) {
+        esr |= (EC_INSNABORT << ARM_EL_EC_SHIFT);
+    } else {
+        esr |= (EC_INSNABORT_SAME_EL << ARM_EL_EC_SHIFT);
+    }
+
+    /* In the aarch64, there is only 32-bit instruction*/
+    esr |= ARM_EL_IL;
+    env->exception.syndrome = esr;
+    cc->do_interrupt(c);
+
+    /* set ESR_EL1 */
+    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.esr_el[1]));
+
+    if (ret) {
+        fprintf(stderr, "<%s> failed to set esr_el1\n", __func__);
+        abort();
+    }
+
+    /* set FAR_EL1 */
+    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.far_el[1]));
+    if (ret) {
+        fprintf(stderr, "<%s> failed to set far_el1\n", __func__);
+        abort();
+    }
+
+    return 0;
+}
+
 #define AARCH64_CORE_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
                  KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
 
@@ -599,6 +674,9 @@ int kvm_arm_cpreg_level(uint64_t regidx)
 #define AARCH64_SIMD_CTRL_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \
                  KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
 
+#define AARCH64_FAULT_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
+                 KVM_REG_ARM64_FAULT | (x))
+
 int kvm_arch_put_registers(CPUState *cs, int level)
 {
     struct kvm_one_reg reg;
@@ -873,6 +951,22 @@ int kvm_arch_get_registers(CPUState *cs)
     }
     vfp_set_fpcr(env, fpr);
 
+    if (is_a64(env)) {
+        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_ESR_EC);
+        reg.addr = (uintptr_t)(&env->exception.syndrome);
+        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+        if (ret) {
+            return ret;
+        }
+
+        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_FAR);
+        reg.addr = (uintptr_t)(&env->exception.vaddress);
+        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+        if (ret) {
+            return ret;
+        }
+    }
+
     if (!write_kvmstate_to_list(cpu)) {
         return EINVAL;
     }
@@ -887,6 +981,62 @@ int kvm_arch_get_registers(CPUState *cs)
     return ret;
 }
 
+static bool is_abort_sea(unsigned long syndrome)
+{
+    unsigned long fault_status;
+    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
+    if ((ec != EC_INSNABORT) && (ec != EC_DATAABORT)) {
+        return false;
+    }
+
+    fault_status = syndrome & ARM_EL_FSC_TYPE;
+    switch (fault_status) {
+    case FSC_SEA:
+    case FSC_SEA_TTW0:
+    case FSC_SEA_TTW1:
+    case FSC_SEA_TTW2:
+    case FSC_SEA_TTW3:
+    case FSC_SECC:
+    case FSC_SECC_TTW0:
+    case FSC_SECC_TTW1:
+    case FSC_SECC_TTW2:
+    case FSC_SECC_TTW3:
+        return true;
+    default:
+        return false;
+    }
+}
+
+void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
+{
+    ram_addr_t ram_addr;
+    hwaddr paddr;
+
+    ARMCPU *cpu = ARM_CPU(c);
+    CPUARMState *env = &cpu->env;
+    assert(code == BUS_MCEERR_AR || code == BUS_MCEERR_AO);
+    if (addr) {
+        ram_addr = qemu_ram_addr_from_host(addr);
+        if (ram_addr != RAM_ADDR_INVALID &&
+            kvm_physical_memory_addr_from_host(c->kvm_state, addr, &paddr)) {
+            kvm_cpu_synchronize_state(c);
+            kvm_hwpoison_page_add(ram_addr);
+            if (is_abort_sea(env->exception.syndrome)) {
+                ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
+                kvm_inject_arm_sea(c);
+            }
+            return;
+        }
+        fprintf(stderr, "Hardware memory error for memory used by "
+                "QEMU itself instead of guest system!\n");
+    }
+
+    if (code == BUS_MCEERR_AR) {
+        fprintf(stderr, "Hardware memory error!\n");
+        exit(1);
+    }
+}
+
 /* C6.6.29 BRK instruction */
 static const uint32_t brk_insn = 0xd4200000;
 
diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
index 633d088..7cdde97 100644
--- a/target/arm/kvm_arm.h
+++ b/target/arm/kvm_arm.h
@@ -288,4 +288,5 @@ static inline const char *its_class_name(void)
     }
 }
 
+void kvm_hwpoison_page_add(ram_addr_t ram_addr);
 #endif
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 5/6] target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

Add SIGBUS signal handler. In this handler, it checks
the exception type, translates the host VA which is
delivered by host or KVM to guest PA, then fills this
PA to CPER, finally injects a Error to guest OS through
KVM.

Add synchronous external abort injection logic, setup
spsr_elx, esr_elx, PSTATE, far_elx, elr_elx etc, when
switch to guest OS, it will jump to the synchronous
external abort vector table entry.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
Signed-off-by: Quanming Wu <wuquanming@huawei.com>
---
 include/sysemu/kvm.h          |   2 +-
 linux-headers/asm-arm64/kvm.h |   5 ++
 target/arm/internals.h        |  13 ++++
 target/arm/kvm.c              |  34 ++++++++++
 target/arm/kvm64.c            | 150 ++++++++++++++++++++++++++++++++++++++++++
 target/arm/kvm_arm.h          |   1 +
 6 files changed, 204 insertions(+), 1 deletion(-)

diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 3a458f5..90c1605 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -361,7 +361,7 @@ bool kvm_vcpu_id_is_valid(int vcpu_id);
 /* Returns VCPU ID to be used on KVM_CREATE_VCPU ioctl() */
 unsigned long kvm_arch_vcpu_id(CPUState *cpu);
 
-#ifdef TARGET_I386
+#if defined(TARGET_I386) || defined(TARGET_AARCH64)
 #define KVM_HAVE_MCE_INJECTION 1
 void kvm_arch_on_sigbus_vcpu(CPUState *cpu, int code, void *addr);
 #endif
diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h
index d254700..5909c30 100644
--- a/linux-headers/asm-arm64/kvm.h
+++ b/linux-headers/asm-arm64/kvm.h
@@ -181,6 +181,11 @@ struct kvm_arch_memory_slot {
 #define KVM_REG_ARM64_SYSREG_OP2_MASK	0x0000000000000007
 #define KVM_REG_ARM64_SYSREG_OP2_SHIFT	0
 
+/* AArch64 fault registers */
+#define KVM_REG_ARM64_FAULT             (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
+#define KVM_REG_ARM64_FAULT_ESR_EC      (0)
+#define KVM_REG_ARM64_FAULT_FAR         (1)
+
 #define ARM64_SYS_REG_SHIFT_MASK(x,n) \
 	(((x) << KVM_REG_ARM64_SYSREG_ ## n ## _SHIFT) & \
 	KVM_REG_ARM64_SYSREG_ ## n ## _MASK)
diff --git a/target/arm/internals.h b/target/arm/internals.h
index 1f6efef..fc0ad6d 100644
--- a/target/arm/internals.h
+++ b/target/arm/internals.h
@@ -235,6 +235,19 @@ enum arm_exception_class {
 #define ARM_EL_ISV_SHIFT 24
 #define ARM_EL_IL (1 << ARM_EL_IL_SHIFT)
 #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
+#define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
+#define ARM_EL_FSC_TYPE (0x3C)
+
+#define FSC_SEA         (0x10)
+#define FSC_SEA_TTW0    (0x14)
+#define FSC_SEA_TTW1    (0x15)
+#define FSC_SEA_TTW2    (0x16)
+#define FSC_SEA_TTW3    (0x17)
+#define FSC_SECC        (0x18)
+#define FSC_SECC_TTW0   (0x1c)
+#define FSC_SECC_TTW1   (0x1d)
+#define FSC_SECC_TTW2   (0x1e)
+#define FSC_SECC_TTW3   (0x1f)
 
 /* Utility functions for constructing various kinds of syndrome value.
  * Note that in general we follow the AArch64 syndrome values; in a
diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index 7c17f0d..2e1716a 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -129,6 +129,39 @@ void kvm_arm_destroy_scratch_host_vcpu(int *fdarray)
     }
 }
 
+typedef struct HWPoisonPage {
+    ram_addr_t ram_addr;
+    QLIST_ENTRY(HWPoisonPage) list;
+} HWPoisonPage;
+
+static QLIST_HEAD(, HWPoisonPage) hwpoison_page_list =
+    QLIST_HEAD_INITIALIZER(hwpoison_page_list);
+
+static void kvm_unpoison_all(void *param)
+{
+    HWPoisonPage *page, *next_page;
+
+    QLIST_FOREACH_SAFE(page, &hwpoison_page_list, list, next_page) {
+        QLIST_REMOVE(page, list);
+        qemu_ram_remap(page->ram_addr, TARGET_PAGE_SIZE);
+        g_free(page);
+    }
+}
+
+void kvm_hwpoison_page_add(ram_addr_t ram_addr)
+{
+    HWPoisonPage *page;
+
+    QLIST_FOREACH(page, &hwpoison_page_list, list) {
+        if (page->ram_addr == ram_addr) {
+            return;
+        }
+    }
+    page = g_new(HWPoisonPage, 1);
+    page->ram_addr = ram_addr;
+    QLIST_INSERT_HEAD(&hwpoison_page_list, page, list);
+}
+
 static void kvm_arm_host_cpu_class_init(ObjectClass *oc, void *data)
 {
     ARMHostCPUClass *ahcc = ARM_HOST_CPU_CLASS(oc);
@@ -182,6 +215,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
 
     cap_has_mp_state = kvm_check_extension(s, KVM_CAP_MP_STATE);
 
+    qemu_register_reset(kvm_unpoison_all, NULL);
     type_register_static(&host_arm_cpu_type_info);
 
     return 0;
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index 0781367..d3bdab2 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -27,6 +27,8 @@
 #include "kvm_arm.h"
 #include "internals.h"
 #include "hw/arm/arm.h"
+#include "hw/acpi/acpi-defs.h"
+#include "hw/acpi/hest_ghes.h"
 
 static bool have_guest_debug;
 
@@ -590,6 +592,79 @@ int kvm_arm_cpreg_level(uint64_t regidx)
     return KVM_PUT_RUNTIME_STATE;
 }
 
+static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
+{
+    int i;
+
+    for (i = 0; i < cpu->cpreg_array_len; i++) {
+        uint32_t regidx = kvm_to_cpreg_id(cpu->cpreg_indexes[i]);
+        const ARMCPRegInfo *ri;
+        ri = get_arm_cp_reginfo(cpu->cp_regs, regidx);
+        if (!ri) {
+            continue;
+        }
+
+        if (ri->type & ARM_CP_NO_RAW) {
+            continue;
+        }
+
+        if (ri->fieldoffset == fieldoffset) {
+            cpu->cpreg_values[i] = read_raw_cp_reg(&cpu->env, ri);
+            return 0;
+        }
+    }
+    return -EINVAL;
+}
+
+/* Inject synchronous external abort */
+static int kvm_inject_arm_sea(CPUState *c)
+{
+    ARMCPU *cpu = ARM_CPU(c);
+    CPUARMState *env = &cpu->env;
+    unsigned long cpsr = pstate_read(env);
+    uint32_t esr = 0;
+    int ret;
+
+    c->exception_index = EXCP_DATA_ABORT;
+    /* Inject the exception to El1 */
+    env->exception.target_el = 1;
+    CPUClass *cc = CPU_GET_CLASS(c);
+
+    esr |= (EC_DATAABORT << ARM_EL_EC_SHIFT);
+    /* This exception syndrome includes {I,D}FSC in the bits [5:0]
+     */
+    esr |= (env->exception.syndrome & 0x3f);
+
+    /* This exception is EL0 or EL1 fault. */
+    if ((cpsr & 0xf) == PSTATE_MODE_EL0t) {
+        esr |= (EC_INSNABORT << ARM_EL_EC_SHIFT);
+    } else {
+        esr |= (EC_INSNABORT_SAME_EL << ARM_EL_EC_SHIFT);
+    }
+
+    /* In the aarch64, there is only 32-bit instruction*/
+    esr |= ARM_EL_IL;
+    env->exception.syndrome = esr;
+    cc->do_interrupt(c);
+
+    /* set ESR_EL1 */
+    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.esr_el[1]));
+
+    if (ret) {
+        fprintf(stderr, "<%s> failed to set esr_el1\n", __func__);
+        abort();
+    }
+
+    /* set FAR_EL1 */
+    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.far_el[1]));
+    if (ret) {
+        fprintf(stderr, "<%s> failed to set far_el1\n", __func__);
+        abort();
+    }
+
+    return 0;
+}
+
 #define AARCH64_CORE_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
                  KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
 
@@ -599,6 +674,9 @@ int kvm_arm_cpreg_level(uint64_t regidx)
 #define AARCH64_SIMD_CTRL_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \
                  KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
 
+#define AARCH64_FAULT_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
+                 KVM_REG_ARM64_FAULT | (x))
+
 int kvm_arch_put_registers(CPUState *cs, int level)
 {
     struct kvm_one_reg reg;
@@ -873,6 +951,22 @@ int kvm_arch_get_registers(CPUState *cs)
     }
     vfp_set_fpcr(env, fpr);
 
+    if (is_a64(env)) {
+        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_ESR_EC);
+        reg.addr = (uintptr_t)(&env->exception.syndrome);
+        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+        if (ret) {
+            return ret;
+        }
+
+        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_FAR);
+        reg.addr = (uintptr_t)(&env->exception.vaddress);
+        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+        if (ret) {
+            return ret;
+        }
+    }
+
     if (!write_kvmstate_to_list(cpu)) {
         return EINVAL;
     }
@@ -887,6 +981,62 @@ int kvm_arch_get_registers(CPUState *cs)
     return ret;
 }
 
+static bool is_abort_sea(unsigned long syndrome)
+{
+    unsigned long fault_status;
+    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
+    if ((ec != EC_INSNABORT) && (ec != EC_DATAABORT)) {
+        return false;
+    }
+
+    fault_status = syndrome & ARM_EL_FSC_TYPE;
+    switch (fault_status) {
+    case FSC_SEA:
+    case FSC_SEA_TTW0:
+    case FSC_SEA_TTW1:
+    case FSC_SEA_TTW2:
+    case FSC_SEA_TTW3:
+    case FSC_SECC:
+    case FSC_SECC_TTW0:
+    case FSC_SECC_TTW1:
+    case FSC_SECC_TTW2:
+    case FSC_SECC_TTW3:
+        return true;
+    default:
+        return false;
+    }
+}
+
+void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
+{
+    ram_addr_t ram_addr;
+    hwaddr paddr;
+
+    ARMCPU *cpu = ARM_CPU(c);
+    CPUARMState *env = &cpu->env;
+    assert(code == BUS_MCEERR_AR || code == BUS_MCEERR_AO);
+    if (addr) {
+        ram_addr = qemu_ram_addr_from_host(addr);
+        if (ram_addr != RAM_ADDR_INVALID &&
+            kvm_physical_memory_addr_from_host(c->kvm_state, addr, &paddr)) {
+            kvm_cpu_synchronize_state(c);
+            kvm_hwpoison_page_add(ram_addr);
+            if (is_abort_sea(env->exception.syndrome)) {
+                ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
+                kvm_inject_arm_sea(c);
+            }
+            return;
+        }
+        fprintf(stderr, "Hardware memory error for memory used by "
+                "QEMU itself instead of guest system!\n");
+    }
+
+    if (code == BUS_MCEERR_AR) {
+        fprintf(stderr, "Hardware memory error!\n");
+        exit(1);
+    }
+}
+
 /* C6.6.29 BRK instruction */
 static const uint32_t brk_insn = 0xd4200000;
 
diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
index 633d088..7cdde97 100644
--- a/target/arm/kvm_arm.h
+++ b/target/arm/kvm_arm.h
@@ -288,4 +288,5 @@ static inline const char *its_class_name(void)
     }
 }
 
+void kvm_hwpoison_page_add(ram_addr_t ram_addr);
 #endif
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [Qemu-devel] [PATCH v11 5/6] target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

Add SIGBUS signal handler. In this handler, it checks
the exception type, translates the host VA which is
delivered by host or KVM to guest PA, then fills this
PA to CPER, finally injects a Error to guest OS through
KVM.

Add synchronous external abort injection logic, setup
spsr_elx, esr_elx, PSTATE, far_elx, elr_elx etc, when
switch to guest OS, it will jump to the synchronous
external abort vector table entry.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
Signed-off-by: Quanming Wu <wuquanming@huawei.com>
---
 include/sysemu/kvm.h          |   2 +-
 linux-headers/asm-arm64/kvm.h |   5 ++
 target/arm/internals.h        |  13 ++++
 target/arm/kvm.c              |  34 ++++++++++
 target/arm/kvm64.c            | 150 ++++++++++++++++++++++++++++++++++++++++++
 target/arm/kvm_arm.h          |   1 +
 6 files changed, 204 insertions(+), 1 deletion(-)

diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 3a458f5..90c1605 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -361,7 +361,7 @@ bool kvm_vcpu_id_is_valid(int vcpu_id);
 /* Returns VCPU ID to be used on KVM_CREATE_VCPU ioctl() */
 unsigned long kvm_arch_vcpu_id(CPUState *cpu);
 
-#ifdef TARGET_I386
+#if defined(TARGET_I386) || defined(TARGET_AARCH64)
 #define KVM_HAVE_MCE_INJECTION 1
 void kvm_arch_on_sigbus_vcpu(CPUState *cpu, int code, void *addr);
 #endif
diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h
index d254700..5909c30 100644
--- a/linux-headers/asm-arm64/kvm.h
+++ b/linux-headers/asm-arm64/kvm.h
@@ -181,6 +181,11 @@ struct kvm_arch_memory_slot {
 #define KVM_REG_ARM64_SYSREG_OP2_MASK	0x0000000000000007
 #define KVM_REG_ARM64_SYSREG_OP2_SHIFT	0
 
+/* AArch64 fault registers */
+#define KVM_REG_ARM64_FAULT             (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
+#define KVM_REG_ARM64_FAULT_ESR_EC      (0)
+#define KVM_REG_ARM64_FAULT_FAR         (1)
+
 #define ARM64_SYS_REG_SHIFT_MASK(x,n) \
 	(((x) << KVM_REG_ARM64_SYSREG_ ## n ## _SHIFT) & \
 	KVM_REG_ARM64_SYSREG_ ## n ## _MASK)
diff --git a/target/arm/internals.h b/target/arm/internals.h
index 1f6efef..fc0ad6d 100644
--- a/target/arm/internals.h
+++ b/target/arm/internals.h
@@ -235,6 +235,19 @@ enum arm_exception_class {
 #define ARM_EL_ISV_SHIFT 24
 #define ARM_EL_IL (1 << ARM_EL_IL_SHIFT)
 #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
+#define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
+#define ARM_EL_FSC_TYPE (0x3C)
+
+#define FSC_SEA         (0x10)
+#define FSC_SEA_TTW0    (0x14)
+#define FSC_SEA_TTW1    (0x15)
+#define FSC_SEA_TTW2    (0x16)
+#define FSC_SEA_TTW3    (0x17)
+#define FSC_SECC        (0x18)
+#define FSC_SECC_TTW0   (0x1c)
+#define FSC_SECC_TTW1   (0x1d)
+#define FSC_SECC_TTW2   (0x1e)
+#define FSC_SECC_TTW3   (0x1f)
 
 /* Utility functions for constructing various kinds of syndrome value.
  * Note that in general we follow the AArch64 syndrome values; in a
diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index 7c17f0d..2e1716a 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -129,6 +129,39 @@ void kvm_arm_destroy_scratch_host_vcpu(int *fdarray)
     }
 }
 
+typedef struct HWPoisonPage {
+    ram_addr_t ram_addr;
+    QLIST_ENTRY(HWPoisonPage) list;
+} HWPoisonPage;
+
+static QLIST_HEAD(, HWPoisonPage) hwpoison_page_list =
+    QLIST_HEAD_INITIALIZER(hwpoison_page_list);
+
+static void kvm_unpoison_all(void *param)
+{
+    HWPoisonPage *page, *next_page;
+
+    QLIST_FOREACH_SAFE(page, &hwpoison_page_list, list, next_page) {
+        QLIST_REMOVE(page, list);
+        qemu_ram_remap(page->ram_addr, TARGET_PAGE_SIZE);
+        g_free(page);
+    }
+}
+
+void kvm_hwpoison_page_add(ram_addr_t ram_addr)
+{
+    HWPoisonPage *page;
+
+    QLIST_FOREACH(page, &hwpoison_page_list, list) {
+        if (page->ram_addr == ram_addr) {
+            return;
+        }
+    }
+    page = g_new(HWPoisonPage, 1);
+    page->ram_addr = ram_addr;
+    QLIST_INSERT_HEAD(&hwpoison_page_list, page, list);
+}
+
 static void kvm_arm_host_cpu_class_init(ObjectClass *oc, void *data)
 {
     ARMHostCPUClass *ahcc = ARM_HOST_CPU_CLASS(oc);
@@ -182,6 +215,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
 
     cap_has_mp_state = kvm_check_extension(s, KVM_CAP_MP_STATE);
 
+    qemu_register_reset(kvm_unpoison_all, NULL);
     type_register_static(&host_arm_cpu_type_info);
 
     return 0;
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index 0781367..d3bdab2 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -27,6 +27,8 @@
 #include "kvm_arm.h"
 #include "internals.h"
 #include "hw/arm/arm.h"
+#include "hw/acpi/acpi-defs.h"
+#include "hw/acpi/hest_ghes.h"
 
 static bool have_guest_debug;
 
@@ -590,6 +592,79 @@ int kvm_arm_cpreg_level(uint64_t regidx)
     return KVM_PUT_RUNTIME_STATE;
 }
 
+static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
+{
+    int i;
+
+    for (i = 0; i < cpu->cpreg_array_len; i++) {
+        uint32_t regidx = kvm_to_cpreg_id(cpu->cpreg_indexes[i]);
+        const ARMCPRegInfo *ri;
+        ri = get_arm_cp_reginfo(cpu->cp_regs, regidx);
+        if (!ri) {
+            continue;
+        }
+
+        if (ri->type & ARM_CP_NO_RAW) {
+            continue;
+        }
+
+        if (ri->fieldoffset == fieldoffset) {
+            cpu->cpreg_values[i] = read_raw_cp_reg(&cpu->env, ri);
+            return 0;
+        }
+    }
+    return -EINVAL;
+}
+
+/* Inject synchronous external abort */
+static int kvm_inject_arm_sea(CPUState *c)
+{
+    ARMCPU *cpu = ARM_CPU(c);
+    CPUARMState *env = &cpu->env;
+    unsigned long cpsr = pstate_read(env);
+    uint32_t esr = 0;
+    int ret;
+
+    c->exception_index = EXCP_DATA_ABORT;
+    /* Inject the exception to El1 */
+    env->exception.target_el = 1;
+    CPUClass *cc = CPU_GET_CLASS(c);
+
+    esr |= (EC_DATAABORT << ARM_EL_EC_SHIFT);
+    /* This exception syndrome includes {I,D}FSC in the bits [5:0]
+     */
+    esr |= (env->exception.syndrome & 0x3f);
+
+    /* This exception is EL0 or EL1 fault. */
+    if ((cpsr & 0xf) == PSTATE_MODE_EL0t) {
+        esr |= (EC_INSNABORT << ARM_EL_EC_SHIFT);
+    } else {
+        esr |= (EC_INSNABORT_SAME_EL << ARM_EL_EC_SHIFT);
+    }
+
+    /* In the aarch64, there is only 32-bit instruction*/
+    esr |= ARM_EL_IL;
+    env->exception.syndrome = esr;
+    cc->do_interrupt(c);
+
+    /* set ESR_EL1 */
+    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.esr_el[1]));
+
+    if (ret) {
+        fprintf(stderr, "<%s> failed to set esr_el1\n", __func__);
+        abort();
+    }
+
+    /* set FAR_EL1 */
+    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.far_el[1]));
+    if (ret) {
+        fprintf(stderr, "<%s> failed to set far_el1\n", __func__);
+        abort();
+    }
+
+    return 0;
+}
+
 #define AARCH64_CORE_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
                  KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
 
@@ -599,6 +674,9 @@ int kvm_arm_cpreg_level(uint64_t regidx)
 #define AARCH64_SIMD_CTRL_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \
                  KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
 
+#define AARCH64_FAULT_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
+                 KVM_REG_ARM64_FAULT | (x))
+
 int kvm_arch_put_registers(CPUState *cs, int level)
 {
     struct kvm_one_reg reg;
@@ -873,6 +951,22 @@ int kvm_arch_get_registers(CPUState *cs)
     }
     vfp_set_fpcr(env, fpr);
 
+    if (is_a64(env)) {
+        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_ESR_EC);
+        reg.addr = (uintptr_t)(&env->exception.syndrome);
+        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+        if (ret) {
+            return ret;
+        }
+
+        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_FAR);
+        reg.addr = (uintptr_t)(&env->exception.vaddress);
+        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+        if (ret) {
+            return ret;
+        }
+    }
+
     if (!write_kvmstate_to_list(cpu)) {
         return EINVAL;
     }
@@ -887,6 +981,62 @@ int kvm_arch_get_registers(CPUState *cs)
     return ret;
 }
 
+static bool is_abort_sea(unsigned long syndrome)
+{
+    unsigned long fault_status;
+    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
+    if ((ec != EC_INSNABORT) && (ec != EC_DATAABORT)) {
+        return false;
+    }
+
+    fault_status = syndrome & ARM_EL_FSC_TYPE;
+    switch (fault_status) {
+    case FSC_SEA:
+    case FSC_SEA_TTW0:
+    case FSC_SEA_TTW1:
+    case FSC_SEA_TTW2:
+    case FSC_SEA_TTW3:
+    case FSC_SECC:
+    case FSC_SECC_TTW0:
+    case FSC_SECC_TTW1:
+    case FSC_SECC_TTW2:
+    case FSC_SECC_TTW3:
+        return true;
+    default:
+        return false;
+    }
+}
+
+void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
+{
+    ram_addr_t ram_addr;
+    hwaddr paddr;
+
+    ARMCPU *cpu = ARM_CPU(c);
+    CPUARMState *env = &cpu->env;
+    assert(code == BUS_MCEERR_AR || code == BUS_MCEERR_AO);
+    if (addr) {
+        ram_addr = qemu_ram_addr_from_host(addr);
+        if (ram_addr != RAM_ADDR_INVALID &&
+            kvm_physical_memory_addr_from_host(c->kvm_state, addr, &paddr)) {
+            kvm_cpu_synchronize_state(c);
+            kvm_hwpoison_page_add(ram_addr);
+            if (is_abort_sea(env->exception.syndrome)) {
+                ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
+                kvm_inject_arm_sea(c);
+            }
+            return;
+        }
+        fprintf(stderr, "Hardware memory error for memory used by "
+                "QEMU itself instead of guest system!\n");
+    }
+
+    if (code == BUS_MCEERR_AR) {
+        fprintf(stderr, "Hardware memory error!\n");
+        exit(1);
+    }
+}
+
 /* C6.6.29 BRK instruction */
 static const uint32_t brk_insn = 0xd4200000;
 
diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
index 633d088..7cdde97 100644
--- a/target/arm/kvm_arm.h
+++ b/target/arm/kvm_arm.h
@@ -288,4 +288,5 @@ static inline const char *its_class_name(void)
     }
 }
 
+void kvm_hwpoison_page_add(ram_addr_t ram_addr);
 #endif
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 5/6] target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: linux-arm-kernel

Add SIGBUS signal handler. In this handler, it checks
the exception type, translates the host VA which is
delivered by host or KVM to guest PA, then fills this
PA to CPER, finally injects a Error to guest OS through
KVM.

Add synchronous external abort injection logic, setup
spsr_elx, esr_elx, PSTATE, far_elx, elr_elx etc, when
switch to guest OS, it will jump to the synchronous
external abort vector table entry.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
Signed-off-by: Quanming Wu <wuquanming@huawei.com>
---
 include/sysemu/kvm.h          |   2 +-
 linux-headers/asm-arm64/kvm.h |   5 ++
 target/arm/internals.h        |  13 ++++
 target/arm/kvm.c              |  34 ++++++++++
 target/arm/kvm64.c            | 150 ++++++++++++++++++++++++++++++++++++++++++
 target/arm/kvm_arm.h          |   1 +
 6 files changed, 204 insertions(+), 1 deletion(-)

diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 3a458f5..90c1605 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -361,7 +361,7 @@ bool kvm_vcpu_id_is_valid(int vcpu_id);
 /* Returns VCPU ID to be used on KVM_CREATE_VCPU ioctl() */
 unsigned long kvm_arch_vcpu_id(CPUState *cpu);
 
-#ifdef TARGET_I386
+#if defined(TARGET_I386) || defined(TARGET_AARCH64)
 #define KVM_HAVE_MCE_INJECTION 1
 void kvm_arch_on_sigbus_vcpu(CPUState *cpu, int code, void *addr);
 #endif
diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h
index d254700..5909c30 100644
--- a/linux-headers/asm-arm64/kvm.h
+++ b/linux-headers/asm-arm64/kvm.h
@@ -181,6 +181,11 @@ struct kvm_arch_memory_slot {
 #define KVM_REG_ARM64_SYSREG_OP2_MASK	0x0000000000000007
 #define KVM_REG_ARM64_SYSREG_OP2_SHIFT	0
 
+/* AArch64 fault registers */
+#define KVM_REG_ARM64_FAULT             (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
+#define KVM_REG_ARM64_FAULT_ESR_EC      (0)
+#define KVM_REG_ARM64_FAULT_FAR         (1)
+
 #define ARM64_SYS_REG_SHIFT_MASK(x,n) \
 	(((x) << KVM_REG_ARM64_SYSREG_ ## n ## _SHIFT) & \
 	KVM_REG_ARM64_SYSREG_ ## n ## _MASK)
diff --git a/target/arm/internals.h b/target/arm/internals.h
index 1f6efef..fc0ad6d 100644
--- a/target/arm/internals.h
+++ b/target/arm/internals.h
@@ -235,6 +235,19 @@ enum arm_exception_class {
 #define ARM_EL_ISV_SHIFT 24
 #define ARM_EL_IL (1 << ARM_EL_IL_SHIFT)
 #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
+#define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
+#define ARM_EL_FSC_TYPE (0x3C)
+
+#define FSC_SEA         (0x10)
+#define FSC_SEA_TTW0    (0x14)
+#define FSC_SEA_TTW1    (0x15)
+#define FSC_SEA_TTW2    (0x16)
+#define FSC_SEA_TTW3    (0x17)
+#define FSC_SECC        (0x18)
+#define FSC_SECC_TTW0   (0x1c)
+#define FSC_SECC_TTW1   (0x1d)
+#define FSC_SECC_TTW2   (0x1e)
+#define FSC_SECC_TTW3   (0x1f)
 
 /* Utility functions for constructing various kinds of syndrome value.
  * Note that in general we follow the AArch64 syndrome values; in a
diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index 7c17f0d..2e1716a 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -129,6 +129,39 @@ void kvm_arm_destroy_scratch_host_vcpu(int *fdarray)
     }
 }
 
+typedef struct HWPoisonPage {
+    ram_addr_t ram_addr;
+    QLIST_ENTRY(HWPoisonPage) list;
+} HWPoisonPage;
+
+static QLIST_HEAD(, HWPoisonPage) hwpoison_page_list =
+    QLIST_HEAD_INITIALIZER(hwpoison_page_list);
+
+static void kvm_unpoison_all(void *param)
+{
+    HWPoisonPage *page, *next_page;
+
+    QLIST_FOREACH_SAFE(page, &hwpoison_page_list, list, next_page) {
+        QLIST_REMOVE(page, list);
+        qemu_ram_remap(page->ram_addr, TARGET_PAGE_SIZE);
+        g_free(page);
+    }
+}
+
+void kvm_hwpoison_page_add(ram_addr_t ram_addr)
+{
+    HWPoisonPage *page;
+
+    QLIST_FOREACH(page, &hwpoison_page_list, list) {
+        if (page->ram_addr == ram_addr) {
+            return;
+        }
+    }
+    page = g_new(HWPoisonPage, 1);
+    page->ram_addr = ram_addr;
+    QLIST_INSERT_HEAD(&hwpoison_page_list, page, list);
+}
+
 static void kvm_arm_host_cpu_class_init(ObjectClass *oc, void *data)
 {
     ARMHostCPUClass *ahcc = ARM_HOST_CPU_CLASS(oc);
@@ -182,6 +215,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
 
     cap_has_mp_state = kvm_check_extension(s, KVM_CAP_MP_STATE);
 
+    qemu_register_reset(kvm_unpoison_all, NULL);
     type_register_static(&host_arm_cpu_type_info);
 
     return 0;
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index 0781367..d3bdab2 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -27,6 +27,8 @@
 #include "kvm_arm.h"
 #include "internals.h"
 #include "hw/arm/arm.h"
+#include "hw/acpi/acpi-defs.h"
+#include "hw/acpi/hest_ghes.h"
 
 static bool have_guest_debug;
 
@@ -590,6 +592,79 @@ int kvm_arm_cpreg_level(uint64_t regidx)
     return KVM_PUT_RUNTIME_STATE;
 }
 
+static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
+{
+    int i;
+
+    for (i = 0; i < cpu->cpreg_array_len; i++) {
+        uint32_t regidx = kvm_to_cpreg_id(cpu->cpreg_indexes[i]);
+        const ARMCPRegInfo *ri;
+        ri = get_arm_cp_reginfo(cpu->cp_regs, regidx);
+        if (!ri) {
+            continue;
+        }
+
+        if (ri->type & ARM_CP_NO_RAW) {
+            continue;
+        }
+
+        if (ri->fieldoffset == fieldoffset) {
+            cpu->cpreg_values[i] = read_raw_cp_reg(&cpu->env, ri);
+            return 0;
+        }
+    }
+    return -EINVAL;
+}
+
+/* Inject synchronous external abort */
+static int kvm_inject_arm_sea(CPUState *c)
+{
+    ARMCPU *cpu = ARM_CPU(c);
+    CPUARMState *env = &cpu->env;
+    unsigned long cpsr = pstate_read(env);
+    uint32_t esr = 0;
+    int ret;
+
+    c->exception_index = EXCP_DATA_ABORT;
+    /* Inject the exception to El1 */
+    env->exception.target_el = 1;
+    CPUClass *cc = CPU_GET_CLASS(c);
+
+    esr |= (EC_DATAABORT << ARM_EL_EC_SHIFT);
+    /* This exception syndrome includes {I,D}FSC in the bits [5:0]
+     */
+    esr |= (env->exception.syndrome & 0x3f);
+
+    /* This exception is EL0 or EL1 fault. */
+    if ((cpsr & 0xf) == PSTATE_MODE_EL0t) {
+        esr |= (EC_INSNABORT << ARM_EL_EC_SHIFT);
+    } else {
+        esr |= (EC_INSNABORT_SAME_EL << ARM_EL_EC_SHIFT);
+    }
+
+    /* In the aarch64, there is only 32-bit instruction*/
+    esr |= ARM_EL_IL;
+    env->exception.syndrome = esr;
+    cc->do_interrupt(c);
+
+    /* set ESR_EL1 */
+    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.esr_el[1]));
+
+    if (ret) {
+        fprintf(stderr, "<%s> failed to set esr_el1\n", __func__);
+        abort();
+    }
+
+    /* set FAR_EL1 */
+    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.far_el[1]));
+    if (ret) {
+        fprintf(stderr, "<%s> failed to set far_el1\n", __func__);
+        abort();
+    }
+
+    return 0;
+}
+
 #define AARCH64_CORE_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
                  KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
 
@@ -599,6 +674,9 @@ int kvm_arm_cpreg_level(uint64_t regidx)
 #define AARCH64_SIMD_CTRL_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \
                  KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
 
+#define AARCH64_FAULT_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
+                 KVM_REG_ARM64_FAULT | (x))
+
 int kvm_arch_put_registers(CPUState *cs, int level)
 {
     struct kvm_one_reg reg;
@@ -873,6 +951,22 @@ int kvm_arch_get_registers(CPUState *cs)
     }
     vfp_set_fpcr(env, fpr);
 
+    if (is_a64(env)) {
+        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_ESR_EC);
+        reg.addr = (uintptr_t)(&env->exception.syndrome);
+        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+        if (ret) {
+            return ret;
+        }
+
+        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_FAR);
+        reg.addr = (uintptr_t)(&env->exception.vaddress);
+        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+        if (ret) {
+            return ret;
+        }
+    }
+
     if (!write_kvmstate_to_list(cpu)) {
         return EINVAL;
     }
@@ -887,6 +981,62 @@ int kvm_arch_get_registers(CPUState *cs)
     return ret;
 }
 
+static bool is_abort_sea(unsigned long syndrome)
+{
+    unsigned long fault_status;
+    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
+    if ((ec != EC_INSNABORT) && (ec != EC_DATAABORT)) {
+        return false;
+    }
+
+    fault_status = syndrome & ARM_EL_FSC_TYPE;
+    switch (fault_status) {
+    case FSC_SEA:
+    case FSC_SEA_TTW0:
+    case FSC_SEA_TTW1:
+    case FSC_SEA_TTW2:
+    case FSC_SEA_TTW3:
+    case FSC_SECC:
+    case FSC_SECC_TTW0:
+    case FSC_SECC_TTW1:
+    case FSC_SECC_TTW2:
+    case FSC_SECC_TTW3:
+        return true;
+    default:
+        return false;
+    }
+}
+
+void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
+{
+    ram_addr_t ram_addr;
+    hwaddr paddr;
+
+    ARMCPU *cpu = ARM_CPU(c);
+    CPUARMState *env = &cpu->env;
+    assert(code == BUS_MCEERR_AR || code == BUS_MCEERR_AO);
+    if (addr) {
+        ram_addr = qemu_ram_addr_from_host(addr);
+        if (ram_addr != RAM_ADDR_INVALID &&
+            kvm_physical_memory_addr_from_host(c->kvm_state, addr, &paddr)) {
+            kvm_cpu_synchronize_state(c);
+            kvm_hwpoison_page_add(ram_addr);
+            if (is_abort_sea(env->exception.syndrome)) {
+                ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
+                kvm_inject_arm_sea(c);
+            }
+            return;
+        }
+        fprintf(stderr, "Hardware memory error for memory used by "
+                "QEMU itself instead of guest system!\n");
+    }
+
+    if (code == BUS_MCEERR_AR) {
+        fprintf(stderr, "Hardware memory error!\n");
+        exit(1);
+    }
+}
+
 /* C6.6.29 BRK instruction */
 static const uint32_t brk_insn = 0xd4200000;
 
diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
index 633d088..7cdde97 100644
--- a/target/arm/kvm_arm.h
+++ b/target/arm/kvm_arm.h
@@ -288,4 +288,5 @@ static inline const char *its_class_name(void)
     }
 }
 
+void kvm_hwpoison_page_add(ram_addr_t ram_addr);
 #endif
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 6/6] target-arm: kvm64: Handle SError interrupt for the guest OS
  2017-08-18 14:23 ` Dongjiu Geng
                     ` (2 preceding siblings ...)
  (?)
@ 2017-08-18 14:23   ` Dongjiu Geng
  -1 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: zhengqiang10, wuquanming, huangshaoyu, linuxarm, gengdongjiu

When guest OS happens SError interrupt(SEI), it will trap to host.
Host firstly calls memory failure to deal with this error and decide
whether it needs to deliver SIGBUS signal to userspace. The advantage
that using signal to notify is that it can make the notification method
is general, non-KVM user can also use it. when userspace gets this
signal and knows this is SError interrupt, it will translate the
delivered host VA to PA and record this PA to GHES.

Because ARMv8.2 adds an extension to RAS to allow system software insert
implicit Error Synchronization Barrier operations to isolate the error and
allow passes specified syndrome to guest OS, so after record the CPER, user
space calls IOCTL to pass a specified syndrome to KVM, then switch to guest
OS, guest OS can use the recorded CPER record and syndrome information to
do the recovery.

The steps are shown below:
1. translate the host VA to guest OS PA and record this error PA to HEST table.
2. set specified virtual SError syndrome and pass the value to KVM.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
Signed-off-by: Quanming Wu <wuquanming@huawei.com>
---
 linux-headers/linux/kvm.h |  1 +
 target/arm/internals.h    |  1 +
 target/arm/kvm64.c        | 28 ++++++++++++++++++++++++++++
 3 files changed, 30 insertions(+)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 2aa176e..10dfcab 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -1356,6 +1356,7 @@ struct kvm_s390_ucas_mapping {
 /* Available with KVM_CAP_S390_CMMA_MIGRATION */
 #define KVM_S390_GET_CMMA_BITS      _IOWR(KVMIO, 0xb8, struct kvm_s390_cmma_log)
 #define KVM_S390_SET_CMMA_BITS      _IOW(KVMIO, 0xb9, struct kvm_s390_cmma_log)
+#define KVM_ARM_SEI                 _IO(KVMIO,   0xb10)
 
 #define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1 << 0)
 #define KVM_DEV_ASSIGN_PCI_2_3		(1 << 1)
diff --git a/target/arm/internals.h b/target/arm/internals.h
index fc0ad6d..18b1cbc 100644
--- a/target/arm/internals.h
+++ b/target/arm/internals.h
@@ -237,6 +237,7 @@ enum arm_exception_class {
 #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
 #define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
 #define ARM_EL_FSC_TYPE (0x3C)
+#define ARM_EL_ISS_MASK ((1 << ARM_EL_IL_SHIFT) - 1)
 
 #define FSC_SEA         (0x10)
 #define FSC_SEA_TTW0    (0x14)
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index d3bdab2..b84cb49 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -616,6 +616,22 @@ static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
     return -EINVAL;
 }
 
+static int kvm_inject_arm_sei(CPUState *cs)
+{
+    ARMCPU *cpu = ARM_CPU(cs);
+    CPUARMState *env = &cpu->env;
+
+    unsigned long syndrome = env->exception.vaddress;
+    /* set virtual SError syndrome */
+    if (arm_feature(env, ARM_FEATURE_RAS_EXTENSION)) {
+        syndrome = syndrome & ARM_EL_ISS_MASK;
+    } else {
+        syndrome = 0;
+    }
+
+    return  kvm_vcpu_ioctl(CPU(cpu), KVM_ARM_SEI, &syndrome);
+}
+
 /* Inject synchronous external abort */
 static int kvm_inject_arm_sea(CPUState *c)
 {
@@ -1007,6 +1023,15 @@ static bool is_abort_sea(unsigned long syndrome)
     }
 }
 
+static bool is_abort_sei(unsigned long syndrome)
+{
+    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
+    if ((ec != EC_SERROR))
+        return false;
+    else
+        return true;
+}
+
 void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
 {
     ram_addr_t ram_addr;
@@ -1024,6 +1049,9 @@ void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
             if (is_abort_sea(env->exception.syndrome)) {
                 ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
                 kvm_inject_arm_sea(c);
+            } else if (is_abort_sei(env->exception.syndrome)) {
+                ghes_update_guest(ACPI_HEST_NOTIFY_SEI, paddr);
+                kvm_inject_arm_sei(c);
             }
             return;
         }
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 6/6] target-arm: kvm64: Handle SError interrupt for the guest OS
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

When guest OS happens SError interrupt(SEI), it will trap to host.
Host firstly calls memory failure to deal with this error and decide
whether it needs to deliver SIGBUS signal to userspace. The advantage
that using signal to notify is that it can make the notification method
is general, non-KVM user can also use it. when userspace gets this
signal and knows this is SError interrupt, it will translate the
delivered host VA to PA and record this PA to GHES.

Because ARMv8.2 adds an extension to RAS to allow system software insert
implicit Error Synchronization Barrier operations to isolate the error and
allow passes specified syndrome to guest OS, so after record the CPER, user
space calls IOCTL to pass a specified syndrome to KVM, then switch to guest
OS, guest OS can use the recorded CPER record and syndrome information to
do the recovery.

The steps are shown below:
1. translate the host VA to guest OS PA and record this error PA to HEST table.
2. set specified virtual SError syndrome and pass the value to KVM.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
Signed-off-by: Quanming Wu <wuquanming@huawei.com>
---
 linux-headers/linux/kvm.h |  1 +
 target/arm/internals.h    |  1 +
 target/arm/kvm64.c        | 28 ++++++++++++++++++++++++++++
 3 files changed, 30 insertions(+)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 2aa176e..10dfcab 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -1356,6 +1356,7 @@ struct kvm_s390_ucas_mapping {
 /* Available with KVM_CAP_S390_CMMA_MIGRATION */
 #define KVM_S390_GET_CMMA_BITS      _IOWR(KVMIO, 0xb8, struct kvm_s390_cmma_log)
 #define KVM_S390_SET_CMMA_BITS      _IOW(KVMIO, 0xb9, struct kvm_s390_cmma_log)
+#define KVM_ARM_SEI                 _IO(KVMIO,   0xb10)
 
 #define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1 << 0)
 #define KVM_DEV_ASSIGN_PCI_2_3		(1 << 1)
diff --git a/target/arm/internals.h b/target/arm/internals.h
index fc0ad6d..18b1cbc 100644
--- a/target/arm/internals.h
+++ b/target/arm/internals.h
@@ -237,6 +237,7 @@ enum arm_exception_class {
 #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
 #define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
 #define ARM_EL_FSC_TYPE (0x3C)
+#define ARM_EL_ISS_MASK ((1 << ARM_EL_IL_SHIFT) - 1)
 
 #define FSC_SEA         (0x10)
 #define FSC_SEA_TTW0    (0x14)
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index d3bdab2..b84cb49 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -616,6 +616,22 @@ static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
     return -EINVAL;
 }
 
+static int kvm_inject_arm_sei(CPUState *cs)
+{
+    ARMCPU *cpu = ARM_CPU(cs);
+    CPUARMState *env = &cpu->env;
+
+    unsigned long syndrome = env->exception.vaddress;
+    /* set virtual SError syndrome */
+    if (arm_feature(env, ARM_FEATURE_RAS_EXTENSION)) {
+        syndrome = syndrome & ARM_EL_ISS_MASK;
+    } else {
+        syndrome = 0;
+    }
+
+    return  kvm_vcpu_ioctl(CPU(cpu), KVM_ARM_SEI, &syndrome);
+}
+
 /* Inject synchronous external abort */
 static int kvm_inject_arm_sea(CPUState *c)
 {
@@ -1007,6 +1023,15 @@ static bool is_abort_sea(unsigned long syndrome)
     }
 }
 
+static bool is_abort_sei(unsigned long syndrome)
+{
+    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
+    if ((ec != EC_SERROR))
+        return false;
+    else
+        return true;
+}
+
 void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
 {
     ram_addr_t ram_addr;
@@ -1024,6 +1049,9 @@ void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
             if (is_abort_sea(env->exception.syndrome)) {
                 ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
                 kvm_inject_arm_sea(c);
+            } else if (is_abort_sei(env->exception.syndrome)) {
+                ghes_update_guest(ACPI_HEST_NOTIFY_SEI, paddr);
+                kvm_inject_arm_sei(c);
             }
             return;
         }
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 6/6] target-arm: kvm64: Handle SError interrupt for the guest OS
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: zhengqiang10, wuquanming, huangshaoyu, linuxarm, gengdongjiu

When guest OS happens SError interrupt(SEI), it will trap to host.
Host firstly calls memory failure to deal with this error and decide
whether it needs to deliver SIGBUS signal to userspace. The advantage
that using signal to notify is that it can make the notification method
is general, non-KVM user can also use it. when userspace gets this
signal and knows this is SError interrupt, it will translate the
delivered host VA to PA and record this PA to GHES.

Because ARMv8.2 adds an extension to RAS to allow system software insert
implicit Error Synchronization Barrier operations to isolate the error and
allow passes specified syndrome to guest OS, so after record the CPER, user
space calls IOCTL to pass a specified syndrome to KVM, then switch to guest
OS, guest OS can use the recorded CPER record and syndrome information to
do the recovery.

The steps are shown below:
1. translate the host VA to guest OS PA and record this error PA to HEST table.
2. set specified virtual SError syndrome and pass the value to KVM.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
Signed-off-by: Quanming Wu <wuquanming@huawei.com>
---
 linux-headers/linux/kvm.h |  1 +
 target/arm/internals.h    |  1 +
 target/arm/kvm64.c        | 28 ++++++++++++++++++++++++++++
 3 files changed, 30 insertions(+)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 2aa176e..10dfcab 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -1356,6 +1356,7 @@ struct kvm_s390_ucas_mapping {
 /* Available with KVM_CAP_S390_CMMA_MIGRATION */
 #define KVM_S390_GET_CMMA_BITS      _IOWR(KVMIO, 0xb8, struct kvm_s390_cmma_log)
 #define KVM_S390_SET_CMMA_BITS      _IOW(KVMIO, 0xb9, struct kvm_s390_cmma_log)
+#define KVM_ARM_SEI                 _IO(KVMIO,   0xb10)
 
 #define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1 << 0)
 #define KVM_DEV_ASSIGN_PCI_2_3		(1 << 1)
diff --git a/target/arm/internals.h b/target/arm/internals.h
index fc0ad6d..18b1cbc 100644
--- a/target/arm/internals.h
+++ b/target/arm/internals.h
@@ -237,6 +237,7 @@ enum arm_exception_class {
 #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
 #define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
 #define ARM_EL_FSC_TYPE (0x3C)
+#define ARM_EL_ISS_MASK ((1 << ARM_EL_IL_SHIFT) - 1)
 
 #define FSC_SEA         (0x10)
 #define FSC_SEA_TTW0    (0x14)
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index d3bdab2..b84cb49 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -616,6 +616,22 @@ static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
     return -EINVAL;
 }
 
+static int kvm_inject_arm_sei(CPUState *cs)
+{
+    ARMCPU *cpu = ARM_CPU(cs);
+    CPUARMState *env = &cpu->env;
+
+    unsigned long syndrome = env->exception.vaddress;
+    /* set virtual SError syndrome */
+    if (arm_feature(env, ARM_FEATURE_RAS_EXTENSION)) {
+        syndrome = syndrome & ARM_EL_ISS_MASK;
+    } else {
+        syndrome = 0;
+    }
+
+    return  kvm_vcpu_ioctl(CPU(cpu), KVM_ARM_SEI, &syndrome);
+}
+
 /* Inject synchronous external abort */
 static int kvm_inject_arm_sea(CPUState *c)
 {
@@ -1007,6 +1023,15 @@ static bool is_abort_sea(unsigned long syndrome)
     }
 }
 
+static bool is_abort_sei(unsigned long syndrome)
+{
+    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
+    if ((ec != EC_SERROR))
+        return false;
+    else
+        return true;
+}
+
 void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
 {
     ram_addr_t ram_addr;
@@ -1024,6 +1049,9 @@ void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
             if (is_abort_sea(env->exception.syndrome)) {
                 ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
                 kvm_inject_arm_sea(c);
+            } else if (is_abort_sei(env->exception.syndrome)) {
+                ghes_update_guest(ACPI_HEST_NOTIFY_SEI, paddr);
+                kvm_inject_arm_sei(c);
             }
             return;
         }
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [Qemu-devel] [PATCH v11 6/6] target-arm: kvm64: Handle SError interrupt for the guest OS
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: mst, imammedo, zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, gengdongjiu, zhengqiang10

When guest OS happens SError interrupt(SEI), it will trap to host.
Host firstly calls memory failure to deal with this error and decide
whether it needs to deliver SIGBUS signal to userspace. The advantage
that using signal to notify is that it can make the notification method
is general, non-KVM user can also use it. when userspace gets this
signal and knows this is SError interrupt, it will translate the
delivered host VA to PA and record this PA to GHES.

Because ARMv8.2 adds an extension to RAS to allow system software insert
implicit Error Synchronization Barrier operations to isolate the error and
allow passes specified syndrome to guest OS, so after record the CPER, user
space calls IOCTL to pass a specified syndrome to KVM, then switch to guest
OS, guest OS can use the recorded CPER record and syndrome information to
do the recovery.

The steps are shown below:
1. translate the host VA to guest OS PA and record this error PA to HEST table.
2. set specified virtual SError syndrome and pass the value to KVM.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
Signed-off-by: Quanming Wu <wuquanming@huawei.com>
---
 linux-headers/linux/kvm.h |  1 +
 target/arm/internals.h    |  1 +
 target/arm/kvm64.c        | 28 ++++++++++++++++++++++++++++
 3 files changed, 30 insertions(+)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 2aa176e..10dfcab 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -1356,6 +1356,7 @@ struct kvm_s390_ucas_mapping {
 /* Available with KVM_CAP_S390_CMMA_MIGRATION */
 #define KVM_S390_GET_CMMA_BITS      _IOWR(KVMIO, 0xb8, struct kvm_s390_cmma_log)
 #define KVM_S390_SET_CMMA_BITS      _IOW(KVMIO, 0xb9, struct kvm_s390_cmma_log)
+#define KVM_ARM_SEI                 _IO(KVMIO,   0xb10)
 
 #define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1 << 0)
 #define KVM_DEV_ASSIGN_PCI_2_3		(1 << 1)
diff --git a/target/arm/internals.h b/target/arm/internals.h
index fc0ad6d..18b1cbc 100644
--- a/target/arm/internals.h
+++ b/target/arm/internals.h
@@ -237,6 +237,7 @@ enum arm_exception_class {
 #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
 #define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
 #define ARM_EL_FSC_TYPE (0x3C)
+#define ARM_EL_ISS_MASK ((1 << ARM_EL_IL_SHIFT) - 1)
 
 #define FSC_SEA         (0x10)
 #define FSC_SEA_TTW0    (0x14)
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index d3bdab2..b84cb49 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -616,6 +616,22 @@ static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
     return -EINVAL;
 }
 
+static int kvm_inject_arm_sei(CPUState *cs)
+{
+    ARMCPU *cpu = ARM_CPU(cs);
+    CPUARMState *env = &cpu->env;
+
+    unsigned long syndrome = env->exception.vaddress;
+    /* set virtual SError syndrome */
+    if (arm_feature(env, ARM_FEATURE_RAS_EXTENSION)) {
+        syndrome = syndrome & ARM_EL_ISS_MASK;
+    } else {
+        syndrome = 0;
+    }
+
+    return  kvm_vcpu_ioctl(CPU(cpu), KVM_ARM_SEI, &syndrome);
+}
+
 /* Inject synchronous external abort */
 static int kvm_inject_arm_sea(CPUState *c)
 {
@@ -1007,6 +1023,15 @@ static bool is_abort_sea(unsigned long syndrome)
     }
 }
 
+static bool is_abort_sei(unsigned long syndrome)
+{
+    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
+    if ((ec != EC_SERROR))
+        return false;
+    else
+        return true;
+}
+
 void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
 {
     ram_addr_t ram_addr;
@@ -1024,6 +1049,9 @@ void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
             if (is_abort_sea(env->exception.syndrome)) {
                 ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
                 kvm_inject_arm_sea(c);
+            } else if (is_abort_sei(env->exception.syndrome)) {
+                ghes_update_guest(ACPI_HEST_NOTIFY_SEI, paddr);
+                kvm_inject_arm_sei(c);
             }
             return;
         }
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* [PATCH v11 6/6] target-arm: kvm64: Handle SError interrupt for the guest OS
@ 2017-08-18 14:23   ` Dongjiu Geng
  0 siblings, 0 replies; 129+ messages in thread
From: Dongjiu Geng @ 2017-08-18 14:23 UTC (permalink / raw)
  To: linux-arm-kernel

When guest OS happens SError interrupt(SEI), it will trap to host.
Host firstly calls memory failure to deal with this error and decide
whether it needs to deliver SIGBUS signal to userspace. The advantage
that using signal to notify is that it can make the notification method
is general, non-KVM user can also use it. when userspace gets this
signal and knows this is SError interrupt, it will translate the
delivered host VA to PA and record this PA to GHES.

Because ARMv8.2 adds an extension to RAS to allow system software insert
implicit Error Synchronization Barrier operations to isolate the error and
allow passes specified syndrome to guest OS, so after record the CPER, user
space calls IOCTL to pass a specified syndrome to KVM, then switch to guest
OS, guest OS can use the recorded CPER record and syndrome information to
do the recovery.

The steps are shown below:
1. translate the host VA to guest OS PA and record this error PA to HEST table.
2. set specified virtual SError syndrome and pass the value to KVM.

Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
Signed-off-by: Quanming Wu <wuquanming@huawei.com>
---
 linux-headers/linux/kvm.h |  1 +
 target/arm/internals.h    |  1 +
 target/arm/kvm64.c        | 28 ++++++++++++++++++++++++++++
 3 files changed, 30 insertions(+)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 2aa176e..10dfcab 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -1356,6 +1356,7 @@ struct kvm_s390_ucas_mapping {
 /* Available with KVM_CAP_S390_CMMA_MIGRATION */
 #define KVM_S390_GET_CMMA_BITS      _IOWR(KVMIO, 0xb8, struct kvm_s390_cmma_log)
 #define KVM_S390_SET_CMMA_BITS      _IOW(KVMIO, 0xb9, struct kvm_s390_cmma_log)
+#define KVM_ARM_SEI                 _IO(KVMIO,   0xb10)
 
 #define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1 << 0)
 #define KVM_DEV_ASSIGN_PCI_2_3		(1 << 1)
diff --git a/target/arm/internals.h b/target/arm/internals.h
index fc0ad6d..18b1cbc 100644
--- a/target/arm/internals.h
+++ b/target/arm/internals.h
@@ -237,6 +237,7 @@ enum arm_exception_class {
 #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
 #define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
 #define ARM_EL_FSC_TYPE (0x3C)
+#define ARM_EL_ISS_MASK ((1 << ARM_EL_IL_SHIFT) - 1)
 
 #define FSC_SEA         (0x10)
 #define FSC_SEA_TTW0    (0x14)
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index d3bdab2..b84cb49 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -616,6 +616,22 @@ static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
     return -EINVAL;
 }
 
+static int kvm_inject_arm_sei(CPUState *cs)
+{
+    ARMCPU *cpu = ARM_CPU(cs);
+    CPUARMState *env = &cpu->env;
+
+    unsigned long syndrome = env->exception.vaddress;
+    /* set virtual SError syndrome */
+    if (arm_feature(env, ARM_FEATURE_RAS_EXTENSION)) {
+        syndrome = syndrome & ARM_EL_ISS_MASK;
+    } else {
+        syndrome = 0;
+    }
+
+    return  kvm_vcpu_ioctl(CPU(cpu), KVM_ARM_SEI, &syndrome);
+}
+
 /* Inject synchronous external abort */
 static int kvm_inject_arm_sea(CPUState *c)
 {
@@ -1007,6 +1023,15 @@ static bool is_abort_sea(unsigned long syndrome)
     }
 }
 
+static bool is_abort_sei(unsigned long syndrome)
+{
+    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
+    if ((ec != EC_SERROR))
+        return false;
+    else
+        return true;
+}
+
 void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
 {
     ram_addr_t ram_addr;
@@ -1024,6 +1049,9 @@ void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
             if (is_abort_sea(env->exception.syndrome)) {
                 ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
                 kvm_inject_arm_sea(c);
+            } else if (is_abort_sei(env->exception.syndrome)) {
+                ghes_update_guest(ACPI_HEST_NOTIFY_SEI, paddr);
+                kvm_inject_arm_sei(c);
             }
             return;
         }
-- 
1.8.3.1

^ permalink raw reply related	[flat|nested] 129+ messages in thread

* 答复: [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
  2017-08-18 14:23 ` Dongjiu Geng
@ 2017-08-18 17:17   ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-18 17:17 UTC (permalink / raw)
  To: mst, imammedo, Zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, Shiju Jose, Laszlo Ersek, zjzhang@codeaurora.org
  Cc: Zhengqiang (turing), Wuquanming, Huangshaoyu, Linuxarm

Loop Laszlo.


> -----邮件原件-----
> 发件人: gengdongjiu
> 发送时间: 2017年8月18日 22:24
> 收件人: mst@redhat.com; imammedo@redhat.com; Zhaoshenglong <zhaoshenglong@huawei.com>; peter.maydell@linaro.org;
> pbonzini@redhat.com; qemu-devel@nongnu.org; qemu-arm@nongnu.org; kvm@vger.kernel.org; edk2-devel@lists.01.org;
> christoffer.dall@linaro.org; marc.zyngier@arm.com; will.deacon@arm.com; james.morse@arm.com; tbaicar@codeaurora.org;
> ard.biesheuvel@linaro.org; mingo@kernel.org; bp@suse.de; Shiju Jose <shiju.jose@huawei.com>; zjzhang@codeaurora.org;
> linux-arm-kernel@lists.infradead.org; kvmarm@lists.cs.columbia.edu; linux-kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> devel@acpica.org; John Garry <john.garry@huawei.com>; Jonathan Cameron <jonathan.cameron@huawei.com>; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; huangdaode <huangdaode@hisilicon.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> 抄送: Huangshaoyu <huangshaoyu@huawei.com>; Wuquanming <wuquanming@huawei.com>; Linuxarm <linuxarm@huawei.com>;
> gengdongjiu <gengdongjiu@huawei.com>; Zhengqiang (turing) <zhengqiang10@huawei.com>
> 主题: [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
> 
> In the armv8 platform, the mainly processor hardware error notification type are synchronous external abort(SEA) and SError Interrupt (SEI),
> For the ARMv8 SEA/SEI, KVM or host kernel will deliver SIGBUS or use other interface to notify user space. After user space gets the
> notification, it will record the CPER to simulate GHES for guest OS and inject the a
> exception(SEA/SEI) to KVM.
> 
> This series patch has two parts, one part handles synchronous external
> abort(SEA) exception and SError Interrupt (SEI) exception; another part is generating APEI table when guest OS boot up, and dynamically
> record CPER for the guest OS about the generic hardware errors. Currently the userspace only handles the memory section hardware
> errors. Before Qemu record the CPER, it needs to check the ACK value written by the guest OS to avoid read-write race condition. In the
> simulated APEI/GHESV2/CPER table, the max number of error soure is 11, which is classified by notification type, now only enable the
> SEA/SEI notification type error source to avoid OS boot warning.
> 
> 
> About the whole solution we ever discuessed it in here before:
> https://patchwork.kernel.org/patch/9633105/
> 
> Below is the APEI/GHESV2/CPER table layout, the max number of error soure is 11:
> 
>        etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0
> | +--------------------------+ +--------->| ||------------->|
> | +--------------------------+ +--------->| |+------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
> 
> 
> ----------------------------------------------------------------------------------------------
> How to test guest OS do SEA/SEI recovery:
> 
> 1. In the guest OS, trigger a SEA or SEI.
> 2. Then you will see below error log that printed by the memory failure 3. Memory failure will do the recovery for the error.
> 
> Such as the below shown kernel log:
> [   21.101216] Synchronous External Abort: synchronous external abort (0x96000010) at 0xffffff8008064018
> [   21.104969] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 8
> [   21.106918] {1}[Hardware Error]: event severity: recoverable
> [   21.109027] {1}[Hardware Error]:  Error 0, type: recoverable
> [   21.110362] {1}[Hardware Error]:   section_type: memory error
> [   21.111705] {1}[Hardware Error]:   physical_address: 0x000000007a200000
> [   21.113255] {1}[Hardware Error]:   error_type: 3, multi-bit ECC
> [   21.118528] Internal error: : 96000010 [#1] SMP
> [   21.119587] Modules linked in:
> [   21.120307] CPU: 0 PID: 509 Comm: devmem Not tainted 4.12.0-rc4ajb-00990-g954379b-dirty #67
> [   21.122307] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
> [   21.123915] task: ffffffc03da32900 task.stack: ffffffc03dbbc000
> [   21.125302] PC is at __do_user_fault+0x58/0x110
> [   21.126370] LR is at __do_user_fault+0x54/0x110
> [   21.127433] pc : [<ffffff8008097528>] lr : [<ffffff8008097524>] pstate: 80000145
> [   21.129164] sp : ffffffc03dbbfd20
> [   21.129940] x29: ffffffc03dbbfd20 x28: ffffffc03da32900
> [   21.131204] x27: 0000000000000000 x26: 0000007f7edc5001
> [   21.132439] x25: ffffff8008648438 x24: ffffffc03dbbfec0
> [   21.133689] x23: 0000000000030001 x22: 0000007f7edc5001
> [   21.134934] x21: 0000000000000007 x20: 0000000092000021
> [   21.136195] x19: ffffffc03da32900 x18: 0000007fdd4c18f0
> [   21.137439] x17: 0000007f7ecb9ebc x16: 0000000000412058
> 
> ------------------------------------------------------------------------------------------------
> how to test guest OS APTI/GHES:
> 1. In the guest OS, use this command to dump the APEI table:
>     "iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST"
> 2. And find the address for the generic error status block
>    according to the notification type
> 3. then find the CPER record through the generic error status block.
> 
> For example(notification type is SEA):
> 
> (1) root@genericarmv8:~# iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST
> (2) root@genericarmv8:~# cat HEST.dsl
>     /*
>      * Intel ACPI Component Architecture
>      * AML/ASL+ Disassembler version 20170728 (64-bit version)
>      * Copyright (c) 2000 - 2017 Intel Corporation
>      *
>      * Disassembly of /sys/firmware/acpi/tables/HEST, Mon Sep  5 07:59:17 2016
>      *
>      * ACPI Data Table [HEST]
>      *
>      * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
>      */
> 
>     ..................................................................................
>     [308h 0776   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
>     [30Ah 0778   2]                    Source Id : 0008
>     [30Ch 0780   2]            Related Source Id : FFFF
>     [30Eh 0782   1]                     Reserved : 00
>     [30Fh 0783   1]                      Enabled : 01
>     [310h 0784   4]       Records To Preallocate : 00000001
>     [314h 0788   4]      Max Sections Per Record : 00000001
>     [318h 0792   4]          Max Raw Data Length : 00001000
> 
>     [31Ch 0796  12]         Error Status Address : [Generic Address Structure]
>     [31Ch 0796   1]                     Space ID : 00 [SystemMemory]
>     [31Dh 0797   1]                    Bit Width : 40
>     [31Eh 0798   1]                   Bit Offset : 00
>     [31Fh 0799   1]         Encoded Access Width : 04 [QWord Access:64]
>     [320h 0800   8]                      Address : 00000000785D0040
> 
>     [328h 0808  28]                       Notify : [Hardware Error Notification Structure]
>     [328h 0808   1]                  Notify Type : 08 [SEA]
>     [329h 0809   1]                Notify Length : 1C
>     [32Ah 0810   2]   Configuration Write Enable : 0000
>     [32Ch 0812   4]                 PollInterval : 00000000
>     [330h 0816   4]                       Vector : 00000000
>     [334h 0820   4]      Polling Threshold Value : 00000000
>     [338h 0824   4]     Polling Threshold Window : 00000000
>     [33Ch 0828   4]        Error Threshold Value : 00000000
>     [340h 0832   4]       Error Threshold Window : 00000000
> 
>     [344h 0836   4]    Error Status Block Length : 00001000
>     [348h 0840  12]            Read Ack Register : [Generic Address Structure]
>     [348h 0840   1]                     Space ID : 00 [SystemMemory]
>     [349h 0841   1]                    Bit Width : 40
>     [34Ah 0842   1]                   Bit Offset : 00
>     [34Bh 0843   1]         Encoded Access Width : 04 [QWord Access:64]
>     [34Ch 0844   8]                      Address : 00000000785D0098
> 
>     [354h 0852   8]            Read Ack Preserve : 00000000FFFFFFFE
>     [35Ch 0860   8]               Read Ack Write : 0000000000000001
> 
>     [364h 0868   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
>     [366h 0870   2]                    Source Id : 0009
>     [368h 0872   2]            Related Source Id : FFFF
>     [36Ah 0874   1]                     Reserved : 00
>     [36Bh 0875   1]                      Enabled : 01
>     [36Ch 0876   4]       Records To Preallocate : 00000001
>     [370h 0880   4]      Max Sections Per Record : 00000001
>     [374h 0884   4]          Max Raw Data Length : 00001000
> 
>     [378h 0888  12]         Error Status Address : [Generic Address Structure]
>     [378h 0888   1]                     Space ID : 00 [SystemMemory]
>     [379h 0889   1]                    Bit Width : 40
>     [37Ah 0890   1]                   Bit Offset : 00
>     [37Bh 0891   1]         Encoded Access Width : 04 [QWord Access:64]
>     [37Ch 0892   8]                      Address : 00000000785D0048
> 
>     [384h 0900  28]                       Notify : [Hardware Error Notification Structure]
>     [384h 0900   1]                  Notify Type : 09 [SEI]
>     [385h 0901   1]                Notify Length : 1C
>     [386h 0902   2]   Configuration Write Enable : 0000
>     [388h 0904   4]                 PollInterval : 00000000
>     [38Ch 0908   4]                       Vector : 00000000
>     [390h 0912   4]      Polling Threshold Value : 00000000
>     [394h 0916   4]     Polling Threshold Window : 00000000
>     [398h 0920   4]        Error Threshold Value : 00000000
>     [39Ch 0924   4]       Error Threshold Window : 00000000
> 
>     [3A0h 0928   4]    Error Status Block Length : 00001000
>     [3A4h 0932  12]            Read Ack Register : [Generic Address Structure]
>     [3A4h 0932   1]                     Space ID : 00 [SystemMemory]
>     [3A5h 0933   1]                    Bit Width : 40
>     [3A6h 0934   1]                   Bit Offset : 00
>     [3A7h 0935   1]         Encoded Access Width : 04 [QWord Access:64]
>     [3A8h 0936   8]                      Address : 00000000785D00A0
> 
>     [3B0h 0944   8]            Read Ack Preserve : 00000000FFFFFFFE
>     [3B8h 0952   8]               Read Ack Write : 0000000000000001
>     .....................................................................................
> (3) according to above table, the address that contains the physical address of a block
>     of memory that holds the error status data for SEA notification error source is 0x00000000785D0040
> (4) the address for SEA notification error source is 0x785d8108
>     (qemu) xp /1 0x00000000785D0040
>     00000000785d0040: 0x785d80b0
> 
> (5) check the content of generic error status block and generic error data entry
>     (qemu) xp /100x 0x785d80b0
>     00000000785d80b0: 0x00000001 0x00000000 0x00000000 0x00000098
>     00000000785d80c0: 0x00000000 0xa5bc1114 0x4ede6f64 0x833e63b8
>     00000000785d80d0: 0xb1837ced 0x00000000 0x00000300 0x00000050
>     00000000785d80e0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d80f0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8100: 0x00000000 0x00000000 0x00000000 0x00004002
>     00000000785d8110: 0x00000000 0x00000000 0x00000000 0x00001111
>     00000000785d8120: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8130: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8140: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8150: 0x00000000 0x00000003 0x00000000 0x00000000
>     00000000785d8160: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8170: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8180: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8190: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81a0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81b0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81c0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81d0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81e0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81f0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8200: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8210: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8220: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8230: 0x00000000 0x00000000 0x00000000 0x00000000
> (6) check the OSPM's ACK value(for example SEA)
>     /* Before OSPM acknowledges the error, check the ACK value */
>     (qemu) xp /1 0x00000000785D0098
>     00000000785d00f0: 0x00000000
> 
>     /* After OSPM acknowledges the error, check the ACK value */
>     (qemu) xp /1 0x00000000785D0098
>     00000000785d00f0: 0x00000001
> 
> Dongjiu Geng (6):
>   ACPI: add APEI/HEST/CPER structures and macros
>   ACPI: Add APEI GHES Table Generation support
>   ACPI: build and enable APEI GHES in the Makefile and configuration
>   target-arm: kvm64: detect guest RAS EXTENSION feature
>   target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
>   target-arm: kvm64: Handle SError interrupt from the guest OS
> 
>  default-configs/arm-softmmu.mak |   1 +
>  hw/acpi/Makefile.objs           |   1 +
>  hw/acpi/aml-build.c             |   2 +
>  hw/acpi/hest_ghes.c             | 345 ++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c        |   6 +
>  include/hw/acpi/acpi-defs.h     | 193 ++++++++++++++++++++++
>  include/hw/acpi/aml-build.h     |   1 +
>  include/hw/acpi/hest_ghes.h     |  47 ++++++
>  include/sysemu/kvm.h            |   2 +-
>  linux-headers/asm-arm64/kvm.h   |   5 +
>  linux-headers/linux/kvm.h       |   2 +
>  target/arm/cpu.h                |   3 +
>  target/arm/internals.h          |  14 ++
>  target/arm/kvm.c                |  34 ++++
>  target/arm/kvm64.c              | 186 ++++++++++++++++++++++
>  target/arm/kvm_arm.h            |   1 +
>  16 files changed, 842 insertions(+), 1 deletion(-)  create mode 100644 hw/acpi/hest_ghes.c  create mode 100644
> include/hw/acpi/hest_ghes.h
> 
> --
> 1.8.3.1

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 129+ messages in thread

* 答复: [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
  2017-08-18 14:23 ` Dongjiu Geng
                   ` (10 preceding siblings ...)
  (?)
@ 2017-08-18 17:17 ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-18 17:17 UTC (permalink / raw)
  To: mst, imammedo, Zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, Shiju Jose, Laszlo Ersek
  Cc: Zhengqiang (turing), Wuquanming, Huangshaoyu, Linuxarm

Loop Laszlo.


> -----邮件原件-----
> 发件人: gengdongjiu
> 发送时间: 2017年8月18日 22:24
> 收件人: mst@redhat.com; imammedo@redhat.com; Zhaoshenglong <zhaoshenglong@huawei.com>; peter.maydell@linaro.org;
> pbonzini@redhat.com; qemu-devel@nongnu.org; qemu-arm@nongnu.org; kvm@vger.kernel.org; edk2-devel@lists.01.org;
> christoffer.dall@linaro.org; marc.zyngier@arm.com; will.deacon@arm.com; james.morse@arm.com; tbaicar@codeaurora.org;
> ard.biesheuvel@linaro.org; mingo@kernel.org; bp@suse.de; Shiju Jose <shiju.jose@huawei.com>; zjzhang@codeaurora.org;
> linux-arm-kernel@lists.infradead.org; kvmarm@lists.cs.columbia.edu; linux-kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> devel@acpica.org; John Garry <john.garry@huawei.com>; Jonathan Cameron <jonathan.cameron@huawei.com>; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; huangdaode <huangdaode@hisilicon.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> 抄送: Huangshaoyu <huangshaoyu@huawei.com>; Wuquanming <wuquanming@huawei.com>; Linuxarm <linuxarm@huawei.com>;
> gengdongjiu <gengdongjiu@huawei.com>; Zhengqiang (turing) <zhengqiang10@huawei.com>
> 主题: [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
> 
> In the armv8 platform, the mainly processor hardware error notification type are synchronous external abort(SEA) and SError Interrupt (SEI),
> For the ARMv8 SEA/SEI, KVM or host kernel will deliver SIGBUS or use other interface to notify user space. After user space gets the
> notification, it will record the CPER to simulate GHES for guest OS and inject the a
> exception(SEA/SEI) to KVM.
> 
> This series patch has two parts, one part handles synchronous external
> abort(SEA) exception and SError Interrupt (SEI) exception; another part is generating APEI table when guest OS boot up, and dynamically
> record CPER for the guest OS about the generic hardware errors. Currently the userspace only handles the memory section hardware
> errors. Before Qemu record the CPER, it needs to check the ACK value written by the guest OS to avoid read-write race condition. In the
> simulated APEI/GHESV2/CPER table, the max number of error soure is 11, which is classified by notification type, now only enable the
> SEA/SEI notification type error source to avoid OS boot warning.
> 
> 
> About the whole solution we ever discuessed it in here before:
> https://patchwork.kernel.org/patch/9633105/
> 
> Below is the APEI/GHESV2/CPER table layout, the max number of error soure is 11:
> 
>        etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0
> | +--------------------------+ +--------->| ||------------->|
> | +--------------------------+ +--------->| |+------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
> 
> 
> ----------------------------------------------------------------------------------------------
> How to test guest OS do SEA/SEI recovery:
> 
> 1. In the guest OS, trigger a SEA or SEI.
> 2. Then you will see below error log that printed by the memory failure 3. Memory failure will do the recovery for the error.
> 
> Such as the below shown kernel log:
> [   21.101216] Synchronous External Abort: synchronous external abort (0x96000010) at 0xffffff8008064018
> [   21.104969] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 8
> [   21.106918] {1}[Hardware Error]: event severity: recoverable
> [   21.109027] {1}[Hardware Error]:  Error 0, type: recoverable
> [   21.110362] {1}[Hardware Error]:   section_type: memory error
> [   21.111705] {1}[Hardware Error]:   physical_address: 0x000000007a200000
> [   21.113255] {1}[Hardware Error]:   error_type: 3, multi-bit ECC
> [   21.118528] Internal error: : 96000010 [#1] SMP
> [   21.119587] Modules linked in:
> [   21.120307] CPU: 0 PID: 509 Comm: devmem Not tainted 4.12.0-rc4ajb-00990-g954379b-dirty #67
> [   21.122307] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
> [   21.123915] task: ffffffc03da32900 task.stack: ffffffc03dbbc000
> [   21.125302] PC is at __do_user_fault+0x58/0x110
> [   21.126370] LR is at __do_user_fault+0x54/0x110
> [   21.127433] pc : [<ffffff8008097528>] lr : [<ffffff8008097524>] pstate: 80000145
> [   21.129164] sp : ffffffc03dbbfd20
> [   21.129940] x29: ffffffc03dbbfd20 x28: ffffffc03da32900
> [   21.131204] x27: 0000000000000000 x26: 0000007f7edc5001
> [   21.132439] x25: ffffff8008648438 x24: ffffffc03dbbfec0
> [   21.133689] x23: 0000000000030001 x22: 0000007f7edc5001
> [   21.134934] x21: 0000000000000007 x20: 0000000092000021
> [   21.136195] x19: ffffffc03da32900 x18: 0000007fdd4c18f0
> [   21.137439] x17: 0000007f7ecb9ebc x16: 0000000000412058
> 
> ------------------------------------------------------------------------------------------------
> how to test guest OS APTI/GHES:
> 1. In the guest OS, use this command to dump the APEI table:
>     "iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST"
> 2. And find the address for the generic error status block
>    according to the notification type
> 3. then find the CPER record through the generic error status block.
> 
> For example(notification type is SEA):
> 
> (1) root@genericarmv8:~# iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST
> (2) root@genericarmv8:~# cat HEST.dsl
>     /*
>      * Intel ACPI Component Architecture
>      * AML/ASL+ Disassembler version 20170728 (64-bit version)
>      * Copyright (c) 2000 - 2017 Intel Corporation
>      *
>      * Disassembly of /sys/firmware/acpi/tables/HEST, Mon Sep  5 07:59:17 2016
>      *
>      * ACPI Data Table [HEST]
>      *
>      * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
>      */
> 
>     ..................................................................................
>     [308h 0776   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
>     [30Ah 0778   2]                    Source Id : 0008
>     [30Ch 0780   2]            Related Source Id : FFFF
>     [30Eh 0782   1]                     Reserved : 00
>     [30Fh 0783   1]                      Enabled : 01
>     [310h 0784   4]       Records To Preallocate : 00000001
>     [314h 0788   4]      Max Sections Per Record : 00000001
>     [318h 0792   4]          Max Raw Data Length : 00001000
> 
>     [31Ch 0796  12]         Error Status Address : [Generic Address Structure]
>     [31Ch 0796   1]                     Space ID : 00 [SystemMemory]
>     [31Dh 0797   1]                    Bit Width : 40
>     [31Eh 0798   1]                   Bit Offset : 00
>     [31Fh 0799   1]         Encoded Access Width : 04 [QWord Access:64]
>     [320h 0800   8]                      Address : 00000000785D0040
> 
>     [328h 0808  28]                       Notify : [Hardware Error Notification Structure]
>     [328h 0808   1]                  Notify Type : 08 [SEA]
>     [329h 0809   1]                Notify Length : 1C
>     [32Ah 0810   2]   Configuration Write Enable : 0000
>     [32Ch 0812   4]                 PollInterval : 00000000
>     [330h 0816   4]                       Vector : 00000000
>     [334h 0820   4]      Polling Threshold Value : 00000000
>     [338h 0824   4]     Polling Threshold Window : 00000000
>     [33Ch 0828   4]        Error Threshold Value : 00000000
>     [340h 0832   4]       Error Threshold Window : 00000000
> 
>     [344h 0836   4]    Error Status Block Length : 00001000
>     [348h 0840  12]            Read Ack Register : [Generic Address Structure]
>     [348h 0840   1]                     Space ID : 00 [SystemMemory]
>     [349h 0841   1]                    Bit Width : 40
>     [34Ah 0842   1]                   Bit Offset : 00
>     [34Bh 0843   1]         Encoded Access Width : 04 [QWord Access:64]
>     [34Ch 0844   8]                      Address : 00000000785D0098
> 
>     [354h 0852   8]            Read Ack Preserve : 00000000FFFFFFFE
>     [35Ch 0860   8]               Read Ack Write : 0000000000000001
> 
>     [364h 0868   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
>     [366h 0870   2]                    Source Id : 0009
>     [368h 0872   2]            Related Source Id : FFFF
>     [36Ah 0874   1]                     Reserved : 00
>     [36Bh 0875   1]                      Enabled : 01
>     [36Ch 0876   4]       Records To Preallocate : 00000001
>     [370h 0880   4]      Max Sections Per Record : 00000001
>     [374h 0884   4]          Max Raw Data Length : 00001000
> 
>     [378h 0888  12]         Error Status Address : [Generic Address Structure]
>     [378h 0888   1]                     Space ID : 00 [SystemMemory]
>     [379h 0889   1]                    Bit Width : 40
>     [37Ah 0890   1]                   Bit Offset : 00
>     [37Bh 0891   1]         Encoded Access Width : 04 [QWord Access:64]
>     [37Ch 0892   8]                      Address : 00000000785D0048
> 
>     [384h 0900  28]                       Notify : [Hardware Error Notification Structure]
>     [384h 0900   1]                  Notify Type : 09 [SEI]
>     [385h 0901   1]                Notify Length : 1C
>     [386h 0902   2]   Configuration Write Enable : 0000
>     [388h 0904   4]                 PollInterval : 00000000
>     [38Ch 0908   4]                       Vector : 00000000
>     [390h 0912   4]      Polling Threshold Value : 00000000
>     [394h 0916   4]     Polling Threshold Window : 00000000
>     [398h 0920   4]        Error Threshold Value : 00000000
>     [39Ch 0924   4]       Error Threshold Window : 00000000
> 
>     [3A0h 0928   4]    Error Status Block Length : 00001000
>     [3A4h 0932  12]            Read Ack Register : [Generic Address Structure]
>     [3A4h 0932   1]                     Space ID : 00 [SystemMemory]
>     [3A5h 0933   1]                    Bit Width : 40
>     [3A6h 0934   1]                   Bit Offset : 00
>     [3A7h 0935   1]         Encoded Access Width : 04 [QWord Access:64]
>     [3A8h 0936   8]                      Address : 00000000785D00A0
> 
>     [3B0h 0944   8]            Read Ack Preserve : 00000000FFFFFFFE
>     [3B8h 0952   8]               Read Ack Write : 0000000000000001
>     .....................................................................................
> (3) according to above table, the address that contains the physical address of a block
>     of memory that holds the error status data for SEA notification error source is 0x00000000785D0040
> (4) the address for SEA notification error source is 0x785d8108
>     (qemu) xp /1 0x00000000785D0040
>     00000000785d0040: 0x785d80b0
> 
> (5) check the content of generic error status block and generic error data entry
>     (qemu) xp /100x 0x785d80b0
>     00000000785d80b0: 0x00000001 0x00000000 0x00000000 0x00000098
>     00000000785d80c0: 0x00000000 0xa5bc1114 0x4ede6f64 0x833e63b8
>     00000000785d80d0: 0xb1837ced 0x00000000 0x00000300 0x00000050
>     00000000785d80e0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d80f0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8100: 0x00000000 0x00000000 0x00000000 0x00004002
>     00000000785d8110: 0x00000000 0x00000000 0x00000000 0x00001111
>     00000000785d8120: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8130: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8140: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8150: 0x00000000 0x00000003 0x00000000 0x00000000
>     00000000785d8160: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8170: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8180: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8190: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81a0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81b0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81c0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81d0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81e0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81f0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8200: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8210: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8220: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8230: 0x00000000 0x00000000 0x00000000 0x00000000
> (6) check the OSPM's ACK value(for example SEA)
>     /* Before OSPM acknowledges the error, check the ACK value */
>     (qemu) xp /1 0x00000000785D0098
>     00000000785d00f0: 0x00000000
> 
>     /* After OSPM acknowledges the error, check the ACK value */
>     (qemu) xp /1 0x00000000785D0098
>     00000000785d00f0: 0x00000001
> 
> Dongjiu Geng (6):
>   ACPI: add APEI/HEST/CPER structures and macros
>   ACPI: Add APEI GHES Table Generation support
>   ACPI: build and enable APEI GHES in the Makefile and configuration
>   target-arm: kvm64: detect guest RAS EXTENSION feature
>   target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
>   target-arm: kvm64: Handle SError interrupt from the guest OS
> 
>  default-configs/arm-softmmu.mak |   1 +
>  hw/acpi/Makefile.objs           |   1 +
>  hw/acpi/aml-build.c             |   2 +
>  hw/acpi/hest_ghes.c             | 345 ++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c        |   6 +
>  include/hw/acpi/acpi-defs.h     | 193 ++++++++++++++++++++++
>  include/hw/acpi/aml-build.h     |   1 +
>  include/hw/acpi/hest_ghes.h     |  47 ++++++
>  include/sysemu/kvm.h            |   2 +-
>  linux-headers/asm-arm64/kvm.h   |   5 +
>  linux-headers/linux/kvm.h       |   2 +
>  target/arm/cpu.h                |   3 +
>  target/arm/internals.h          |  14 ++
>  target/arm/kvm.c                |  34 ++++
>  target/arm/kvm64.c              | 186 ++++++++++++++++++++++
>  target/arm/kvm_arm.h            |   1 +
>  16 files changed, 842 insertions(+), 1 deletion(-)  create mode 100644 hw/acpi/hest_ghes.c  create mode 100644
> include/hw/acpi/hest_ghes.h
> 
> --
> 1.8.3.1

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 129+ messages in thread

* 答复: [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
@ 2017-08-18 17:17   ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-18 17:17 UTC (permalink / raw)
  To: linux-arm-kernel

Loop Laszlo.


> -----????-----
> ???: gengdongjiu
> ????: 2017?8?18? 22:24
> ???: mst at redhat.com; imammedo at redhat.com; Zhaoshenglong <zhaoshenglong@huawei.com>; peter.maydell at linaro.org;
> pbonzini at redhat.com; qemu-devel at nongnu.org; qemu-arm at nongnu.org; kvm at vger.kernel.org; edk2-devel at lists.01.org;
> christoffer.dall at linaro.org; marc.zyngier at arm.com; will.deacon at arm.com; james.morse at arm.com; tbaicar at codeaurora.org;
> ard.biesheuvel at linaro.org; mingo at kernel.org; bp at suse.de; Shiju Jose <shiju.jose@huawei.com>; zjzhang at codeaurora.org;
> linux-arm-kernel at lists.infradead.org; kvmarm at lists.cs.columbia.edu; linux-kernel at vger.kernel.org; linux-acpi at vger.kernel.org;
> devel at acpica.org; John Garry <john.garry@huawei.com>; Jonathan Cameron <jonathan.cameron@huawei.com>; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; huangdaode <huangdaode@hisilicon.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> ??: Huangshaoyu <huangshaoyu@huawei.com>; Wuquanming <wuquanming@huawei.com>; Linuxarm <linuxarm@huawei.com>;
> gengdongjiu <gengdongjiu@huawei.com>; Zhengqiang (turing) <zhengqiang10@huawei.com>
> ??: [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI
> 
> In the armv8 platform, the mainly processor hardware error notification type are synchronous external abort(SEA) and SError Interrupt (SEI),
> For the ARMv8 SEA/SEI, KVM or host kernel will deliver SIGBUS or use other interface to notify user space. After user space gets the
> notification, it will record the CPER to simulate GHES for guest OS and inject the a
> exception(SEA/SEI) to KVM.
> 
> This series patch has two parts, one part handles synchronous external
> abort(SEA) exception and SError Interrupt (SEI) exception; another part is generating APEI table when guest OS boot up, and dynamically
> record CPER for the guest OS about the generic hardware errors. Currently the userspace only handles the memory section hardware
> errors. Before Qemu record the CPER, it needs to check the ACK value written by the guest OS to avoid read-write race condition. In the
> simulated APEI/GHESV2/CPER table, the max number of error soure is 11, which is classified by notification type, now only enable the
> SEA/SEI notification type error source to avoid OS boot warning.
> 
> 
> About the whole solution we ever discuessed it in here before:
> https://patchwork.kernel.org/patch/9633105/
> 
> Below is the APEI/GHESV2/CPER table layout, the max number of error soure is 11:
> 
>        etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0
> | +--------------------------+ +--------->| ||------------->|
> | +--------------------------+ +--------->| |+------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
> 
> 
> ----------------------------------------------------------------------------------------------
> How to test guest OS do SEA/SEI recovery:
> 
> 1. In the guest OS, trigger a SEA or SEI.
> 2. Then you will see below error log that printed by the memory failure 3. Memory failure will do the recovery for the error.
> 
> Such as the below shown kernel log:
> [   21.101216] Synchronous External Abort: synchronous external abort (0x96000010) at 0xffffff8008064018
> [   21.104969] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 8
> [   21.106918] {1}[Hardware Error]: event severity: recoverable
> [   21.109027] {1}[Hardware Error]:  Error 0, type: recoverable
> [   21.110362] {1}[Hardware Error]:   section_type: memory error
> [   21.111705] {1}[Hardware Error]:   physical_address: 0x000000007a200000
> [   21.113255] {1}[Hardware Error]:   error_type: 3, multi-bit ECC
> [   21.118528] Internal error: : 96000010 [#1] SMP
> [   21.119587] Modules linked in:
> [   21.120307] CPU: 0 PID: 509 Comm: devmem Not tainted 4.12.0-rc4ajb-00990-g954379b-dirty #67
> [   21.122307] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
> [   21.123915] task: ffffffc03da32900 task.stack: ffffffc03dbbc000
> [   21.125302] PC is at __do_user_fault+0x58/0x110
> [   21.126370] LR is at __do_user_fault+0x54/0x110
> [   21.127433] pc : [<ffffff8008097528>] lr : [<ffffff8008097524>] pstate: 80000145
> [   21.129164] sp : ffffffc03dbbfd20
> [   21.129940] x29: ffffffc03dbbfd20 x28: ffffffc03da32900
> [   21.131204] x27: 0000000000000000 x26: 0000007f7edc5001
> [   21.132439] x25: ffffff8008648438 x24: ffffffc03dbbfec0
> [   21.133689] x23: 0000000000030001 x22: 0000007f7edc5001
> [   21.134934] x21: 0000000000000007 x20: 0000000092000021
> [   21.136195] x19: ffffffc03da32900 x18: 0000007fdd4c18f0
> [   21.137439] x17: 0000007f7ecb9ebc x16: 0000000000412058
> 
> ------------------------------------------------------------------------------------------------
> how to test guest OS APTI/GHES:
> 1. In the guest OS, use this command to dump the APEI table:
>     "iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST"
> 2. And find the address for the generic error status block
>    according to the notification type
> 3. then find the CPER record through the generic error status block.
> 
> For example(notification type is SEA):
> 
> (1) root at genericarmv8:~# iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST
> (2) root at genericarmv8:~# cat HEST.dsl
>     /*
>      * Intel ACPI Component Architecture
>      * AML/ASL+ Disassembler version 20170728 (64-bit version)
>      * Copyright (c) 2000 - 2017 Intel Corporation
>      *
>      * Disassembly of /sys/firmware/acpi/tables/HEST, Mon Sep  5 07:59:17 2016
>      *
>      * ACPI Data Table [HEST]
>      *
>      * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
>      */
> 
>     ..................................................................................
>     [308h 0776   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
>     [30Ah 0778   2]                    Source Id : 0008
>     [30Ch 0780   2]            Related Source Id : FFFF
>     [30Eh 0782   1]                     Reserved : 00
>     [30Fh 0783   1]                      Enabled : 01
>     [310h 0784   4]       Records To Preallocate : 00000001
>     [314h 0788   4]      Max Sections Per Record : 00000001
>     [318h 0792   4]          Max Raw Data Length : 00001000
> 
>     [31Ch 0796  12]         Error Status Address : [Generic Address Structure]
>     [31Ch 0796   1]                     Space ID : 00 [SystemMemory]
>     [31Dh 0797   1]                    Bit Width : 40
>     [31Eh 0798   1]                   Bit Offset : 00
>     [31Fh 0799   1]         Encoded Access Width : 04 [QWord Access:64]
>     [320h 0800   8]                      Address : 00000000785D0040
> 
>     [328h 0808  28]                       Notify : [Hardware Error Notification Structure]
>     [328h 0808   1]                  Notify Type : 08 [SEA]
>     [329h 0809   1]                Notify Length : 1C
>     [32Ah 0810   2]   Configuration Write Enable : 0000
>     [32Ch 0812   4]                 PollInterval : 00000000
>     [330h 0816   4]                       Vector : 00000000
>     [334h 0820   4]      Polling Threshold Value : 00000000
>     [338h 0824   4]     Polling Threshold Window : 00000000
>     [33Ch 0828   4]        Error Threshold Value : 00000000
>     [340h 0832   4]       Error Threshold Window : 00000000
> 
>     [344h 0836   4]    Error Status Block Length : 00001000
>     [348h 0840  12]            Read Ack Register : [Generic Address Structure]
>     [348h 0840   1]                     Space ID : 00 [SystemMemory]
>     [349h 0841   1]                    Bit Width : 40
>     [34Ah 0842   1]                   Bit Offset : 00
>     [34Bh 0843   1]         Encoded Access Width : 04 [QWord Access:64]
>     [34Ch 0844   8]                      Address : 00000000785D0098
> 
>     [354h 0852   8]            Read Ack Preserve : 00000000FFFFFFFE
>     [35Ch 0860   8]               Read Ack Write : 0000000000000001
> 
>     [364h 0868   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
>     [366h 0870   2]                    Source Id : 0009
>     [368h 0872   2]            Related Source Id : FFFF
>     [36Ah 0874   1]                     Reserved : 00
>     [36Bh 0875   1]                      Enabled : 01
>     [36Ch 0876   4]       Records To Preallocate : 00000001
>     [370h 0880   4]      Max Sections Per Record : 00000001
>     [374h 0884   4]          Max Raw Data Length : 00001000
> 
>     [378h 0888  12]         Error Status Address : [Generic Address Structure]
>     [378h 0888   1]                     Space ID : 00 [SystemMemory]
>     [379h 0889   1]                    Bit Width : 40
>     [37Ah 0890   1]                   Bit Offset : 00
>     [37Bh 0891   1]         Encoded Access Width : 04 [QWord Access:64]
>     [37Ch 0892   8]                      Address : 00000000785D0048
> 
>     [384h 0900  28]                       Notify : [Hardware Error Notification Structure]
>     [384h 0900   1]                  Notify Type : 09 [SEI]
>     [385h 0901   1]                Notify Length : 1C
>     [386h 0902   2]   Configuration Write Enable : 0000
>     [388h 0904   4]                 PollInterval : 00000000
>     [38Ch 0908   4]                       Vector : 00000000
>     [390h 0912   4]      Polling Threshold Value : 00000000
>     [394h 0916   4]     Polling Threshold Window : 00000000
>     [398h 0920   4]        Error Threshold Value : 00000000
>     [39Ch 0924   4]       Error Threshold Window : 00000000
> 
>     [3A0h 0928   4]    Error Status Block Length : 00001000
>     [3A4h 0932  12]            Read Ack Register : [Generic Address Structure]
>     [3A4h 0932   1]                     Space ID : 00 [SystemMemory]
>     [3A5h 0933   1]                    Bit Width : 40
>     [3A6h 0934   1]                   Bit Offset : 00
>     [3A7h 0935   1]         Encoded Access Width : 04 [QWord Access:64]
>     [3A8h 0936   8]                      Address : 00000000785D00A0
> 
>     [3B0h 0944   8]            Read Ack Preserve : 00000000FFFFFFFE
>     [3B8h 0952   8]               Read Ack Write : 0000000000000001
>     .....................................................................................
> (3) according to above table, the address that contains the physical address of a block
>     of memory that holds the error status data for SEA notification error source is 0x00000000785D0040
> (4) the address for SEA notification error source is 0x785d8108
>     (qemu) xp /1 0x00000000785D0040
>     00000000785d0040: 0x785d80b0
> 
> (5) check the content of generic error status block and generic error data entry
>     (qemu) xp /100x 0x785d80b0
>     00000000785d80b0: 0x00000001 0x00000000 0x00000000 0x00000098
>     00000000785d80c0: 0x00000000 0xa5bc1114 0x4ede6f64 0x833e63b8
>     00000000785d80d0: 0xb1837ced 0x00000000 0x00000300 0x00000050
>     00000000785d80e0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d80f0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8100: 0x00000000 0x00000000 0x00000000 0x00004002
>     00000000785d8110: 0x00000000 0x00000000 0x00000000 0x00001111
>     00000000785d8120: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8130: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8140: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8150: 0x00000000 0x00000003 0x00000000 0x00000000
>     00000000785d8160: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8170: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8180: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8190: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81a0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81b0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81c0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81d0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81e0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d81f0: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8200: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8210: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8220: 0x00000000 0x00000000 0x00000000 0x00000000
>     00000000785d8230: 0x00000000 0x00000000 0x00000000 0x00000000
> (6) check the OSPM's ACK value(for example SEA)
>     /* Before OSPM acknowledges the error, check the ACK value */
>     (qemu) xp /1 0x00000000785D0098
>     00000000785d00f0: 0x00000000
> 
>     /* After OSPM acknowledges the error, check the ACK value */
>     (qemu) xp /1 0x00000000785D0098
>     00000000785d00f0: 0x00000001
> 
> Dongjiu Geng (6):
>   ACPI: add APEI/HEST/CPER structures and macros
>   ACPI: Add APEI GHES Table Generation support
>   ACPI: build and enable APEI GHES in the Makefile and configuration
>   target-arm: kvm64: detect guest RAS EXTENSION feature
>   target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
>   target-arm: kvm64: Handle SError interrupt from the guest OS
> 
>  default-configs/arm-softmmu.mak |   1 +
>  hw/acpi/Makefile.objs           |   1 +
>  hw/acpi/aml-build.c             |   2 +
>  hw/acpi/hest_ghes.c             | 345 ++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c        |   6 +
>  include/hw/acpi/acpi-defs.h     | 193 ++++++++++++++++++++++
>  include/hw/acpi/aml-build.h     |   1 +
>  include/hw/acpi/hest_ghes.h     |  47 ++++++
>  include/sysemu/kvm.h            |   2 +-
>  linux-headers/asm-arm64/kvm.h   |   5 +
>  linux-headers/linux/kvm.h       |   2 +
>  target/arm/cpu.h                |   3 +
>  target/arm/internals.h          |  14 ++
>  target/arm/kvm.c                |  34 ++++
>  target/arm/kvm64.c              | 186 ++++++++++++++++++++++
>  target/arm/kvm_arm.h            |   1 +
>  16 files changed, 842 insertions(+), 1 deletion(-)  create mode 100644 hw/acpi/hest_ghes.c  create mode 100644
> include/hw/acpi/hest_ghes.h
> 
> --
> 1.8.3.1

^ permalink raw reply	[flat|nested] 129+ messages in thread

* 答复: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
  2017-08-18 14:23   ` Dongjiu Geng
@ 2017-08-18 17:18     ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-18 17:18 UTC (permalink / raw)
  To: mst, imammedo, Zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, Laszlo Ersek,
	christoffer.dall, marc.zyngier, will.deacon, james.morse,
	tbaicar, ard.biesheuvel, mingo, bp, Shiju Jose,
	zjzhang@codeaurora.org
  Cc: Zhengqiang (turing), Wuquanming, Huangshaoyu, Linuxarm

Loop Laszlo

> -----邮件原件-----
> 发件人: gengdongjiu
> 发送时间: 2017年8月18日 22:24
> 收件人: mst@redhat.com; imammedo@redhat.com; Zhaoshenglong <zhaoshenglong@huawei.com>; peter.maydell@linaro.org;
> pbonzini@redhat.com; qemu-devel@nongnu.org; qemu-arm@nongnu.org; kvm@vger.kernel.org; edk2-devel@lists.01.org;
> christoffer.dall@linaro.org; marc.zyngier@arm.com; will.deacon@arm.com; james.morse@arm.com; tbaicar@codeaurora.org;
> ard.biesheuvel@linaro.org; mingo@kernel.org; bp@suse.de; Shiju Jose <shiju.jose@huawei.com>; zjzhang@codeaurora.org;
> linux-arm-kernel@lists.infradead.org; kvmarm@lists.cs.columbia.edu; linux-kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> devel@acpica.org; John Garry <john.garry@huawei.com>; Jonathan Cameron <jonathan.cameron@huawei.com>; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; huangdaode <huangdaode@hisilicon.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> 抄送: Huangshaoyu <huangshaoyu@huawei.com>; Wuquanming <wuquanming@huawei.com>; Linuxarm <linuxarm@huawei.com>;
> gengdongjiu <gengdongjiu@huawei.com>; Zhengqiang (turing) <zhengqiang10@huawei.com>
> 主题: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
> 
> (1) Add related APEI/HEST table structures and  macros, these
>     definition refer to ACPI 6.1 and UEFI 2.6 spec.
> (2) Add generic error status block and CPER memory section
>     definition, user space only handle memory section errors.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 193 insertions(+)
> 
> diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h index 72be675..3b4bad7 100644
> --- a/include/hw/acpi/acpi-defs.h
> +++ b/include/hw/acpi/acpi-defs.h
> @@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
>  #define ACPI_APIC_GENERIC_TRANSLATOR    15
>  #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
> 
> +/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
> +#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
> +#define UEFI_CPER_MEM_VALID_PA               0x0002
> +#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
> +#define UEFI_CPER_MEM_VALID_NODE             0x0008
> +#define UEFI_CPER_MEM_VALID_CARD             0x0010
> +#define UEFI_CPER_MEM_VALID_MODULE           0x0020
> +#define UEFI_CPER_MEM_VALID_BANK             0x0040
> +#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
> +#define UEFI_CPER_MEM_VALID_ROW              0x0100
> +#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
> +#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
> +#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
> +#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
> +#define UEFI_CPER_MEM_VALID_TARGET           0x2000
> +#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
> +#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
> +#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
> +#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
> +#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
> +
> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
> +
> +enum AcpiHestNotifyType {
> +    ACPI_HEST_NOTIFY_POLLED = 0,
> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
> +    ACPI_HEST_NOTIFY_LOCAL = 2,
> +    ACPI_HEST_NOTIFY_SCI = 3,
> +    ACPI_HEST_NOTIFY_NMI = 4,
> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
> +};
> +
>  /*
>   * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
>   */
> @@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {  } QEMU_PACKED;  typedef struct AcpiSystemResourceAffinityTable
> AcpiSystemResourceAffinityTable;
> 
> +/* Hardware Error Notification, from the ACPI 6.1
> + * spec, "18.3.2.9 Hardware Error Notification"
> + */
> +struct AcpiHestNotify {
> +    uint8_t type;
> +    uint8_t length;
> +    uint16_t config_write_enable;
> +    uint32_t poll_interval;
> +    uint32_t vector;
> +    uint32_t polling_threshold_value;
> +    uint32_t polling_threshold_window;
> +    uint32_t error_threshold_value;
> +    uint32_t error_threshold_window;
> +} QEMU_PACKED;
> +typedef struct AcpiHestNotify AcpiHestNotify;
> +
> +/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
> + * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
> + */
> +enum AcpiHestSourceType {
> +    ACPI_HEST_SOURCE_IA32_CHECK = 0,
> +    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
> +    ACPI_HEST_SOURCE_IA32_NMI = 2,
> +    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
> +    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
> +    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
> +    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
> +};
> +
> +/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
> +#define ACPI_GEBS_UNCORRECTABLE             (1)
> +#define ACPI_GEBS_CORRECTABLE               (1 << 1)
> +#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
> +#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
> +/* 10 bits, error data entry count */
> +#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
> +
> +/* Generic Hardware Error Source Structure, refer to ACPI 6.1
> + * "18.3.2.7 Generic Hardware Error Source". in this struct the
> + * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR  */
> +
> +struct AcpiGenericHardwareErrorSource {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length; } QEMU_PACKED; typedef struct
> +AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
> +
> +/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8
> +Generic
> + * Hardware Error Source version 2", in this struct the "type" field
> +has to
> + * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2  */ struct
> +AcpiGenericHardwareErrorSourceV2 {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length;
> +    struct AcpiGenericAddress read_ack_register;
> +    uint64_t read_ack_preserve;
> +    uint64_t read_ack_write;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericHardwareErrorSourceV2
> +            AcpiGenericHardwareErrorSourceV2;
> +
> +/* Generic Error Status block, from ACPI 6.1,
> + * "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorStatus {
> +    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
> +    uint32_t block_status;
> +    uint32_t raw_data_offset;
> +    uint32_t raw_data_length;
> +    uint32_t data_length;
> +    uint32_t error_severity;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
> +
> +enum AcpiGenericErrorSeverity {
> +    ACPI_CPER_SEV_RECOVERABLE,
> +    ACPI_CPER_SEV_FATAL,
> +    ACPI_CPER_SEV_CORRECTED,
> +    ACPI_CPER_SEV_NONE,
> +};
> +
> +/* Generic Error Data entry, revision number is 0x0300,
> + * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorData {
> +    uint8_t section_type_le[16];
> +    /* The "error_severity" fields that they take their
> +     * values from AcpiGenericErrorSeverity
> +     */
> +    uint32_t error_severity;
> +    uint16_t revision;
> +    uint8_t validation_bits;
> +    uint8_t flags;
> +    uint32_t error_data_length;
> +    uint8_t fru_id[16];
> +    uint8_t fru_text[20];
> +    uint64_t time_stamp;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorData AcpiGenericErrorData;
> +
> +/* From UEFI 2.6, "N.2.5 Memory Error Section" */ struct
> +UefiCperSecMemErr {
> +    uint64_t    validation_bits;
> +    uint64_t    error_status;
> +    uint64_t    physical_addr;
> +    uint64_t    physical_addr_mask;
> +    uint16_t    node;
> +    uint16_t    card;
> +    uint16_t    module;
> +    uint16_t    bank;
> +    uint16_t    device;
> +    uint16_t    row;
> +    uint16_t    column;
> +    uint16_t    bit_pos;
> +    uint64_t    requestor_id;
> +    uint64_t    responder_id;
> +    uint64_t    target_id;
> +    uint8_t     error_type;
> +    uint8_t     reserved;
> +    uint16_t    rank;
> +    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
> +    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
> +} QEMU_PACKED;
> +typedef struct UefiCperSecMemErr UefiCperSecMemErr;
> +
> +/*
> + * HEST Description Table
> + */
> +struct AcpiHardwareErrorSourceTable {
> +    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
> +    uint32_t           error_source_count;
> +} QEMU_PACKED;
> +typedef struct AcpiHardwareErrorSourceTable
> +AcpiHardwareErrorSourceTable;
> +
>  #define ACPI_SRAT_PROCESSOR_APIC     0
>  #define ACPI_SRAT_MEMORY             1
>  #define ACPI_SRAT_PROCESSOR_x2APIC   2
> --
> 1.8.3.1

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 129+ messages in thread

* 答复: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
  2017-08-18 14:23   ` Dongjiu Geng
                     ` (3 preceding siblings ...)
  (?)
@ 2017-08-18 17:18   ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-18 17:18 UTC (permalink / raw)
  To: mst, imammedo, Zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, Laszlo Ersek,
	christoffer.dall, marc.zyngier, will.deacon, james.morse,
	tbaicar, ard.biesheuvel, mingo, bp, Shiju Jose
  Cc: Zhengqiang (turing), Wuquanming, Huangshaoyu, Linuxarm

Loop Laszlo

> -----邮件原件-----
> 发件人: gengdongjiu
> 发送时间: 2017年8月18日 22:24
> 收件人: mst@redhat.com; imammedo@redhat.com; Zhaoshenglong <zhaoshenglong@huawei.com>; peter.maydell@linaro.org;
> pbonzini@redhat.com; qemu-devel@nongnu.org; qemu-arm@nongnu.org; kvm@vger.kernel.org; edk2-devel@lists.01.org;
> christoffer.dall@linaro.org; marc.zyngier@arm.com; will.deacon@arm.com; james.morse@arm.com; tbaicar@codeaurora.org;
> ard.biesheuvel@linaro.org; mingo@kernel.org; bp@suse.de; Shiju Jose <shiju.jose@huawei.com>; zjzhang@codeaurora.org;
> linux-arm-kernel@lists.infradead.org; kvmarm@lists.cs.columbia.edu; linux-kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> devel@acpica.org; John Garry <john.garry@huawei.com>; Jonathan Cameron <jonathan.cameron@huawei.com>; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; huangdaode <huangdaode@hisilicon.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> 抄送: Huangshaoyu <huangshaoyu@huawei.com>; Wuquanming <wuquanming@huawei.com>; Linuxarm <linuxarm@huawei.com>;
> gengdongjiu <gengdongjiu@huawei.com>; Zhengqiang (turing) <zhengqiang10@huawei.com>
> 主题: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
> 
> (1) Add related APEI/HEST table structures and  macros, these
>     definition refer to ACPI 6.1 and UEFI 2.6 spec.
> (2) Add generic error status block and CPER memory section
>     definition, user space only handle memory section errors.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 193 insertions(+)
> 
> diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h index 72be675..3b4bad7 100644
> --- a/include/hw/acpi/acpi-defs.h
> +++ b/include/hw/acpi/acpi-defs.h
> @@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
>  #define ACPI_APIC_GENERIC_TRANSLATOR    15
>  #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
> 
> +/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
> +#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
> +#define UEFI_CPER_MEM_VALID_PA               0x0002
> +#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
> +#define UEFI_CPER_MEM_VALID_NODE             0x0008
> +#define UEFI_CPER_MEM_VALID_CARD             0x0010
> +#define UEFI_CPER_MEM_VALID_MODULE           0x0020
> +#define UEFI_CPER_MEM_VALID_BANK             0x0040
> +#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
> +#define UEFI_CPER_MEM_VALID_ROW              0x0100
> +#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
> +#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
> +#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
> +#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
> +#define UEFI_CPER_MEM_VALID_TARGET           0x2000
> +#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
> +#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
> +#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
> +#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
> +#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
> +
> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
> +
> +enum AcpiHestNotifyType {
> +    ACPI_HEST_NOTIFY_POLLED = 0,
> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
> +    ACPI_HEST_NOTIFY_LOCAL = 2,
> +    ACPI_HEST_NOTIFY_SCI = 3,
> +    ACPI_HEST_NOTIFY_NMI = 4,
> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
> +};
> +
>  /*
>   * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
>   */
> @@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {  } QEMU_PACKED;  typedef struct AcpiSystemResourceAffinityTable
> AcpiSystemResourceAffinityTable;
> 
> +/* Hardware Error Notification, from the ACPI 6.1
> + * spec, "18.3.2.9 Hardware Error Notification"
> + */
> +struct AcpiHestNotify {
> +    uint8_t type;
> +    uint8_t length;
> +    uint16_t config_write_enable;
> +    uint32_t poll_interval;
> +    uint32_t vector;
> +    uint32_t polling_threshold_value;
> +    uint32_t polling_threshold_window;
> +    uint32_t error_threshold_value;
> +    uint32_t error_threshold_window;
> +} QEMU_PACKED;
> +typedef struct AcpiHestNotify AcpiHestNotify;
> +
> +/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
> + * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
> + */
> +enum AcpiHestSourceType {
> +    ACPI_HEST_SOURCE_IA32_CHECK = 0,
> +    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
> +    ACPI_HEST_SOURCE_IA32_NMI = 2,
> +    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
> +    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
> +    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
> +    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
> +};
> +
> +/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
> +#define ACPI_GEBS_UNCORRECTABLE             (1)
> +#define ACPI_GEBS_CORRECTABLE               (1 << 1)
> +#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
> +#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
> +/* 10 bits, error data entry count */
> +#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
> +
> +/* Generic Hardware Error Source Structure, refer to ACPI 6.1
> + * "18.3.2.7 Generic Hardware Error Source". in this struct the
> + * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR  */
> +
> +struct AcpiGenericHardwareErrorSource {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length; } QEMU_PACKED; typedef struct
> +AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
> +
> +/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8
> +Generic
> + * Hardware Error Source version 2", in this struct the "type" field
> +has to
> + * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2  */ struct
> +AcpiGenericHardwareErrorSourceV2 {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length;
> +    struct AcpiGenericAddress read_ack_register;
> +    uint64_t read_ack_preserve;
> +    uint64_t read_ack_write;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericHardwareErrorSourceV2
> +            AcpiGenericHardwareErrorSourceV2;
> +
> +/* Generic Error Status block, from ACPI 6.1,
> + * "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorStatus {
> +    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
> +    uint32_t block_status;
> +    uint32_t raw_data_offset;
> +    uint32_t raw_data_length;
> +    uint32_t data_length;
> +    uint32_t error_severity;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
> +
> +enum AcpiGenericErrorSeverity {
> +    ACPI_CPER_SEV_RECOVERABLE,
> +    ACPI_CPER_SEV_FATAL,
> +    ACPI_CPER_SEV_CORRECTED,
> +    ACPI_CPER_SEV_NONE,
> +};
> +
> +/* Generic Error Data entry, revision number is 0x0300,
> + * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorData {
> +    uint8_t section_type_le[16];
> +    /* The "error_severity" fields that they take their
> +     * values from AcpiGenericErrorSeverity
> +     */
> +    uint32_t error_severity;
> +    uint16_t revision;
> +    uint8_t validation_bits;
> +    uint8_t flags;
> +    uint32_t error_data_length;
> +    uint8_t fru_id[16];
> +    uint8_t fru_text[20];
> +    uint64_t time_stamp;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorData AcpiGenericErrorData;
> +
> +/* From UEFI 2.6, "N.2.5 Memory Error Section" */ struct
> +UefiCperSecMemErr {
> +    uint64_t    validation_bits;
> +    uint64_t    error_status;
> +    uint64_t    physical_addr;
> +    uint64_t    physical_addr_mask;
> +    uint16_t    node;
> +    uint16_t    card;
> +    uint16_t    module;
> +    uint16_t    bank;
> +    uint16_t    device;
> +    uint16_t    row;
> +    uint16_t    column;
> +    uint16_t    bit_pos;
> +    uint64_t    requestor_id;
> +    uint64_t    responder_id;
> +    uint64_t    target_id;
> +    uint8_t     error_type;
> +    uint8_t     reserved;
> +    uint16_t    rank;
> +    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
> +    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
> +} QEMU_PACKED;
> +typedef struct UefiCperSecMemErr UefiCperSecMemErr;
> +
> +/*
> + * HEST Description Table
> + */
> +struct AcpiHardwareErrorSourceTable {
> +    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
> +    uint32_t           error_source_count;
> +} QEMU_PACKED;
> +typedef struct AcpiHardwareErrorSourceTable
> +AcpiHardwareErrorSourceTable;
> +
>  #define ACPI_SRAT_PROCESSOR_APIC     0
>  #define ACPI_SRAT_MEMORY             1
>  #define ACPI_SRAT_PROCESSOR_x2APIC   2
> --
> 1.8.3.1

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 129+ messages in thread

* 答复: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-18 17:18     ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-18 17:18 UTC (permalink / raw)
  To: linux-arm-kernel

Loop Laszlo

> -----????-----
> ???: gengdongjiu
> ????: 2017?8?18? 22:24
> ???: mst at redhat.com; imammedo at redhat.com; Zhaoshenglong <zhaoshenglong@huawei.com>; peter.maydell at linaro.org;
> pbonzini at redhat.com; qemu-devel at nongnu.org; qemu-arm at nongnu.org; kvm at vger.kernel.org; edk2-devel at lists.01.org;
> christoffer.dall at linaro.org; marc.zyngier at arm.com; will.deacon at arm.com; james.morse at arm.com; tbaicar at codeaurora.org;
> ard.biesheuvel at linaro.org; mingo at kernel.org; bp at suse.de; Shiju Jose <shiju.jose@huawei.com>; zjzhang at codeaurora.org;
> linux-arm-kernel at lists.infradead.org; kvmarm at lists.cs.columbia.edu; linux-kernel at vger.kernel.org; linux-acpi at vger.kernel.org;
> devel at acpica.org; John Garry <john.garry@huawei.com>; Jonathan Cameron <jonathan.cameron@huawei.com>; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; huangdaode <huangdaode@hisilicon.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> ??: Huangshaoyu <huangshaoyu@huawei.com>; Wuquanming <wuquanming@huawei.com>; Linuxarm <linuxarm@huawei.com>;
> gengdongjiu <gengdongjiu@huawei.com>; Zhengqiang (turing) <zhengqiang10@huawei.com>
> ??: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
> 
> (1) Add related APEI/HEST table structures and  macros, these
>     definition refer to ACPI 6.1 and UEFI 2.6 spec.
> (2) Add generic error status block and CPER memory section
>     definition, user space only handle memory section errors.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 193 insertions(+)
> 
> diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h index 72be675..3b4bad7 100644
> --- a/include/hw/acpi/acpi-defs.h
> +++ b/include/hw/acpi/acpi-defs.h
> @@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
>  #define ACPI_APIC_GENERIC_TRANSLATOR    15
>  #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
> 
> +/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
> +#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
> +#define UEFI_CPER_MEM_VALID_PA               0x0002
> +#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
> +#define UEFI_CPER_MEM_VALID_NODE             0x0008
> +#define UEFI_CPER_MEM_VALID_CARD             0x0010
> +#define UEFI_CPER_MEM_VALID_MODULE           0x0020
> +#define UEFI_CPER_MEM_VALID_BANK             0x0040
> +#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
> +#define UEFI_CPER_MEM_VALID_ROW              0x0100
> +#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
> +#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
> +#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
> +#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
> +#define UEFI_CPER_MEM_VALID_TARGET           0x2000
> +#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
> +#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
> +#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
> +#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
> +#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
> +
> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
> +
> +enum AcpiHestNotifyType {
> +    ACPI_HEST_NOTIFY_POLLED = 0,
> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
> +    ACPI_HEST_NOTIFY_LOCAL = 2,
> +    ACPI_HEST_NOTIFY_SCI = 3,
> +    ACPI_HEST_NOTIFY_NMI = 4,
> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
> +};
> +
>  /*
>   * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
>   */
> @@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {  } QEMU_PACKED;  typedef struct AcpiSystemResourceAffinityTable
> AcpiSystemResourceAffinityTable;
> 
> +/* Hardware Error Notification, from the ACPI 6.1
> + * spec, "18.3.2.9 Hardware Error Notification"
> + */
> +struct AcpiHestNotify {
> +    uint8_t type;
> +    uint8_t length;
> +    uint16_t config_write_enable;
> +    uint32_t poll_interval;
> +    uint32_t vector;
> +    uint32_t polling_threshold_value;
> +    uint32_t polling_threshold_window;
> +    uint32_t error_threshold_value;
> +    uint32_t error_threshold_window;
> +} QEMU_PACKED;
> +typedef struct AcpiHestNotify AcpiHestNotify;
> +
> +/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
> + * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
> + */
> +enum AcpiHestSourceType {
> +    ACPI_HEST_SOURCE_IA32_CHECK = 0,
> +    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
> +    ACPI_HEST_SOURCE_IA32_NMI = 2,
> +    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
> +    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
> +    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
> +    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
> +};
> +
> +/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
> +#define ACPI_GEBS_UNCORRECTABLE             (1)
> +#define ACPI_GEBS_CORRECTABLE               (1 << 1)
> +#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
> +#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
> +/* 10 bits, error data entry count */
> +#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
> +
> +/* Generic Hardware Error Source Structure, refer to ACPI 6.1
> + * "18.3.2.7 Generic Hardware Error Source". in this struct the
> + * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR  */
> +
> +struct AcpiGenericHardwareErrorSource {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length; } QEMU_PACKED; typedef struct
> +AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
> +
> +/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8
> +Generic
> + * Hardware Error Source version 2", in this struct the "type" field
> +has to
> + * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2  */ struct
> +AcpiGenericHardwareErrorSourceV2 {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length;
> +    struct AcpiGenericAddress read_ack_register;
> +    uint64_t read_ack_preserve;
> +    uint64_t read_ack_write;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericHardwareErrorSourceV2
> +            AcpiGenericHardwareErrorSourceV2;
> +
> +/* Generic Error Status block, from ACPI 6.1,
> + * "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorStatus {
> +    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
> +    uint32_t block_status;
> +    uint32_t raw_data_offset;
> +    uint32_t raw_data_length;
> +    uint32_t data_length;
> +    uint32_t error_severity;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
> +
> +enum AcpiGenericErrorSeverity {
> +    ACPI_CPER_SEV_RECOVERABLE,
> +    ACPI_CPER_SEV_FATAL,
> +    ACPI_CPER_SEV_CORRECTED,
> +    ACPI_CPER_SEV_NONE,
> +};
> +
> +/* Generic Error Data entry, revision number is 0x0300,
> + * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorData {
> +    uint8_t section_type_le[16];
> +    /* The "error_severity" fields that they take their
> +     * values from AcpiGenericErrorSeverity
> +     */
> +    uint32_t error_severity;
> +    uint16_t revision;
> +    uint8_t validation_bits;
> +    uint8_t flags;
> +    uint32_t error_data_length;
> +    uint8_t fru_id[16];
> +    uint8_t fru_text[20];
> +    uint64_t time_stamp;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorData AcpiGenericErrorData;
> +
> +/* From UEFI 2.6, "N.2.5 Memory Error Section" */ struct
> +UefiCperSecMemErr {
> +    uint64_t    validation_bits;
> +    uint64_t    error_status;
> +    uint64_t    physical_addr;
> +    uint64_t    physical_addr_mask;
> +    uint16_t    node;
> +    uint16_t    card;
> +    uint16_t    module;
> +    uint16_t    bank;
> +    uint16_t    device;
> +    uint16_t    row;
> +    uint16_t    column;
> +    uint16_t    bit_pos;
> +    uint64_t    requestor_id;
> +    uint64_t    responder_id;
> +    uint64_t    target_id;
> +    uint8_t     error_type;
> +    uint8_t     reserved;
> +    uint16_t    rank;
> +    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
> +    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
> +} QEMU_PACKED;
> +typedef struct UefiCperSecMemErr UefiCperSecMemErr;
> +
> +/*
> + * HEST Description Table
> + */
> +struct AcpiHardwareErrorSourceTable {
> +    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
> +    uint32_t           error_source_count;
> +} QEMU_PACKED;
> +typedef struct AcpiHardwareErrorSourceTable
> +AcpiHardwareErrorSourceTable;
> +
>  #define ACPI_SRAT_PROCESSOR_APIC     0
>  #define ACPI_SRAT_MEMORY             1
>  #define ACPI_SRAT_PROCESSOR_x2APIC   2
> --
> 1.8.3.1

^ permalink raw reply	[flat|nested] 129+ messages in thread

* 答复: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
  2017-08-18 14:23   ` Dongjiu Geng
  (?)
@ 2017-08-18 17:19     ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-18 17:19 UTC (permalink / raw)
  To: mst, imammedo, Zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, Shiju Jose, zjzhang,
  Cc: Zhengqiang (turing), Wuquanming, Huangshaoyu, Linuxarm

Loop Laszlo


> -----邮件原件-----
> 发件人: gengdongjiu
> 发送时间: 2017年8月18日 22:24
> 收件人: mst@redhat.com; imammedo@redhat.com; Zhaoshenglong <zhaoshenglong@huawei.com>; peter.maydell@linaro.org;
> pbonzini@redhat.com; qemu-devel@nongnu.org; qemu-arm@nongnu.org; kvm@vger.kernel.org; edk2-devel@lists.01.org;
> christoffer.dall@linaro.org; marc.zyngier@arm.com; will.deacon@arm.com; james.morse@arm.com; tbaicar@codeaurora.org;
> ard.biesheuvel@linaro.org; mingo@kernel.org; bp@suse.de; Shiju Jose <shiju.jose@huawei.com>; zjzhang@codeaurora.org;
> linux-arm-kernel@lists.infradead.org; kvmarm@lists.cs.columbia.edu; linux-kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> devel@acpica.org; John Garry <john.garry@huawei.com>; Jonathan Cameron <jonathan.cameron@huawei.com>; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; huangdaode <huangdaode@hisilicon.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> 抄送: Huangshaoyu <huangshaoyu@huawei.com>; Wuquanming <wuquanming@huawei.com>; Linuxarm <linuxarm@huawei.com>;
> gengdongjiu <gengdongjiu@huawei.com>; Zhengqiang (turing) <zhengqiang10@huawei.com>
> 主题: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
> 
> This implements APEI GHES Table by passing the error CPER info to the guest via a fw_cfg_blob. After a CPER info is recorded, an
> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception will be injected into the guest OS.
> 
> Below is the table layout, the max number of error soure is 11, which is classified by notification type.
> 
>      etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0
> | +--------------------------+ +--------->| ||------------->|
> | +--------------------------+ +--------->| |+------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
> 
> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
> so user space must check the ack value to avoid read-write race condition.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  hw/acpi/aml-build.c         |   2 +
>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c    |   6 +
>  include/hw/acpi/aml-build.h |   1 +
>  include/hw/acpi/hest_ghes.h |  47 ++++++
>  5 files changed, 401 insertions(+)
>  create mode 100644 hw/acpi/hest_ghes.c
>  create mode 100644 include/hw/acpi/hest_ghes.h
> 
> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c index 36a6cc4..6849e5f 100644
> --- a/hw/acpi/aml-build.c
> +++ b/hw/acpi/aml-build.c
> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>      tables->table_data = g_array_new(false, true /* clear */, 1);
>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>      tables->linker = bios_linker_loader_init();  }
> 
> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>      g_array_free(tables->table_data, true);
>      g_array_free(tables->tcpalog, mfre);
>      g_array_free(tables->vmgenid, mfre);
> +    g_array_free(tables->hardware_errors, mfre);
>  }
> 
>  /* Build rsdt table */
> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c new file mode 100644 index 0000000..ff6b5ef
> --- /dev/null
> +++ b/hw/acpi/hest_ghes.c
> @@ -0,0 +1,345 @@
> +/*
> + *  APEI GHES table Generation
> + *
> + *  Copyright (C) 2017 huawei.
> + *
> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qmp-commands.h"
> +#include "hw/acpi/acpi.h"
> +#include "hw/acpi/aml-build.h"
> +#include "hw/acpi/hest_ghes.h"
> +#include "hw/nvram/fw_cfg.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/error-report.h"
> +
> +/* The structure that stands for the layout
> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
> + *
> + *           etc/hardware_errors
> + * ==========================================
> + * +------------------+
> + * |    address       |              +--------------+
> + * |    registers     |              | Error Status |
> + * | +----------------+              | Data Block 0 |
> + * | |status_address0 |------------->| +------------+
> + * | +----------------+              | |  CPER      |
> + * | |status_address1 |----------+   | |  CPER      |
> + * | +----------------+          |   | |  ....      |
> + * | |.............   |          |   | |  CPER      |
> + * | +----------------+          |   | +------------+
> + * | |status_address10|-----+    |   | Error Status |
> + * | +----------------+     |    |   | Data Block 1 |
> + * | |ack_value0      |     |    +-->| +------------+
> + * | +----------------+     |        | |  CPER      |
> + * | |ack_value1      |     |        | |  CPER      |
> + * | +----------------+     |        | |  ....      |
> + * | | .............  |     |        | |  CPER      |
> + * | +----------------+     |        +-+------------+
> + * | |ack_value10     |     |        | |..........  |
> + * | +----------------+     |        | +------------+
> + *                          |        | Error Status |
> + *                          |        | Data Block10 |
> + *                          +------->+------------+
> + *                                   | |  CPER      |
> + *                                   | |  CPER      |
> + *                                   | |  ....      |
> + *                                   | |  CPER      |
> + *                                   +-+------------+
> + */
> +struct hardware_errors_buffer {
> +    /* Generic Error Status Block register */
> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    char
> +gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
> +};
> +
> +static int ghes_record_cper(uint64_t error_block_address,
> +                                    uint64_t error_physical_addr) {
> +    AcpiGenericErrorStatus block;
> +    AcpiGenericErrorData *gdata;
> +    UefiCperSecMemErr *mem_err;
> +    uint64_t current_block_length;
> +    unsigned char *buffer;
> +    /* memory section */
> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
> +                              0x83, 0xB1};
> +
> +    cpu_physical_memory_read(error_block_address, &block,
> +                                sizeof(AcpiGenericErrorStatus));
> +
> +    /* Get the current generic error status block length */
> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
> +        le32_to_cpu(block.data_length);
> +
> +    /* If the Generic Error Status Block is NULL, update
> +     * the block header
> +     */
> +    if (!block.block_status) {
> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    }
> +
> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    /* check whether it runs out of the preallocated memory */
> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
> +       GHES_MAX_RAW_DATA_LENGTH) {
> +        error_report("Record CPER out of boundary!!!");
> +        return GHES_CPER_FAIL;
> +    }
> +
> +    /* Write back the Generic Error Status Block to guest memory */
> +    cpu_physical_memory_write(error_block_address, &block,
> +        sizeof(AcpiGenericErrorStatus));
> +
> +    /* Fill in Generic Error Data Entry */
> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
> +                       sizeof(UefiCperSecMemErr));
> +
> +
> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +    gdata = (AcpiGenericErrorData *)buffer;
> +
> +    /* Memory section */
> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
> +            sizeof(mem_section_id_le));
> +
> +    /* error severity is recoverable */
> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    gdata->revision = 0x300; /* the revision number is 0x300 */
> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
> +
> +    /* User space only handle the memory section CPER */
> +
> +    /* Hard code to Multi-bit ECC error */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
> +    mem_err->error_type =
> + cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
> +
> +    /* Record the physical address at which the memory error occurred */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> +
> +    /* Write back the Generic Error Data Entry to guest memory */
> +    cpu_physical_memory_write(error_block_address + current_block_length,
> +        buffer, sizeof(AcpiGenericErrorData) +
> + sizeof(UefiCperSecMemErr));
> +
> +    g_free(buffer);
> +    return GHES_CPER_OK;
> +}
> +
> +static void
> +build_address(GArray *table_data, BIOSLinker *linker,
> +    uint32_t dst_patched_offset, uint32_t src_offset,
> +    uint8_t address_space_id , uint8_t  register_bit_width,
> +    uint8_t register_bit_offset, uint8_t access_size) {
> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    /* Address space */
> +    build_append_int_noprefix(table_data, address_space_id, 1);
> +    /* register bit width */
> +    build_append_int_noprefix(table_data, register_bit_width, 1);
> +    /* register bit offset */
> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
> +    /* access size */
> +    build_append_int_noprefix(table_data, access_size, 1);
> +    acpi_data_push(table_data, address_size);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
> +     * can retrieve and read it. the address size is 64 bits.
> +     */
> +    bios_linker_loader_add_pointer(linker,
> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
> +        GHES_ERRORS_FW_CFG_FILE, src_offset); }
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                                            BIOSLinker *linker) {
> +    uint32_t ghes_start = table_data->len;
> +    uint32_t address_size, error_status_address_offset;
> +    uint32_t read_ack_register_offset, i;
> +
> +    address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    error_status_address_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    read_ack_register_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    acpi_data_push(hardware_error,
> +        offsetof(struct hardware_errors_buffer, ack_value));
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Initialize read ack register */
> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
> +
> +    /* Reserved the total size for ERRORS fw_cfg blob
> +     */
> +    acpi_data_push(hardware_error, sizeof(struct
> + hardware_errors_buffer));
> +
> +    /* Allocate guest memory for the Data fw_cfg blob */
> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
> +                            1, false);
> +    /* Reserve table header size */
> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
> +
> +    build_append_int_noprefix(table_data,
> + GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
> +        build_append_int_noprefix(table_data,
> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
> +        /* source id */
> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
> +        /* related source id */
> +        build_append_int_noprefix(table_data, 0xffff, 2);
> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
> +
> +        /* Currently only enable SEA notification type to avoid the kernel
> +         * warning, reserve the space for other notification error source
> +         */
> +        if (i == ACPI_HEST_NOTIFY_SEA) {
> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
> +        } else {
> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
> +        }
> +
> +        /* The number of error status block per generic hardware error source */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max sections per record */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max raw data length */
> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH,
> + 4);
> +
> +        /* Build error status address*/
> +        build_address(table_data, linker, error_status_address_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> +
> +        /* Hardware error notification structure */
> +        build_append_int_noprefix(table_data, i, 1); /* type */
> +        /* length */
> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
> +        build_append_int_noprefix(table_data, 0, 26);
> +
> +        /* Error Status Block Length */
> +        build_append_int_noprefix(table_data,
> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
> +
> +        /* Build read ack register */
> +        build_address(table_data, linker, read_ack_register_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2),
> +            offsetof(struct hardware_errors_buffer, ack_value) +
> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
> +            4 /* QWord access */);
> +
> +        /* Read ack preserve */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe),
> + 8);
> +
> +        /* Read ack write */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
> +    }
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Patch address of generic error status block into
> +         * the address register so OSPM can retrieve and read it.
> +         */
> +        bios_linker_loader_add_pointer(linker,
> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
> +            GHES_ERRORS_FW_CFG_FILE,
> +            offsetof(struct hardware_errors_buffer, gesb) +
> +            i * GHES_MAX_RAW_DATA_LENGTH);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
> +     * so QEMU can write the ERRORS there. The address is expected to be
> +     * < 4GB, but write 64 bits anyway.
> +     */
> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
> +        offsetof(struct hardware_errors_buffer, gesb));
> +
> +    build_header(linker, table_data,
> +        (void *)(table_data->data + ghes_start), "HEST",
> +        table_data->len - ghes_start, 1, NULL, "GHES"); }
> +
> +static GhesState ges;
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error) {
> +
> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
> +
> +    /* Create a read-only fw_cfg file for GHES */
> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
> +                    size);
> +    /* Create a read-write fw_cfg file for Address */
> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false); }
> +
> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address) {
> +    uint64_t error_block_addr;
> +    uint64_t ack_value_addr, ack_value = 0;
> +    int loop = 0, ack_value_size;
> +    bool ret = GHES_CPER_FAIL;
> +
> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
> +        offsetof(struct hardware_errors_buffer, ack_value)) /
> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
> +
> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
> +        error_block_addr = le32_to_cpu(error_block_addr);
> +
> +        ack_value_addr = ges.ghes_addr_le -
> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
> +retry:
> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
> +        if (!ack_value) {
> +            if (loop < 3) {
> +                usleep(100 * 1000);
> +                loop++;
> +                goto retry;
> +            } else {
> +                error_report("Last time OSPM does not acknowledge the error,"
> +                    " record CPER failed this time, set the ack value to"
> +                    " avoid blocking next time CPER record! exit");
> +                ack_value = 1;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                return ret;
> +            }
> +        } else {
> +            /* A zero value in ghes_addr means that BIOS has not yet written
> +             * the address
> +             */
> +            if (error_block_addr) {
> +                ack_value = 0;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                ret = ghes_record_cper(error_block_addr, physical_address);
> +            }
> +        }
> +    }
> +    return ret;
> +}
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index 3d78ff6..def1ec1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -45,6 +45,7 @@
>  #include "hw/arm/virt.h"
>  #include "sysemu/numa.h"
>  #include "kvm_arm.h"
> +#include "hw/acpi/hest_ghes.h"
> 
>  #define ARM_SPI_BASE 32
>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>      acpi_add_table(table_offsets, tables_blob);
>      build_spcr(tables_blob, tables->linker, vms);
> 
> +    acpi_add_table(table_offsets, tables_blob);
> +    ghes_build_acpi(tables_blob, tables->hardware_errors,
> + tables->linker);
> +
>      if (nb_numa_nodes > 0) {
>          acpi_add_table(table_offsets, tables_blob);
>          build_srat(tables_blob, tables->linker, vms); @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>                      acpi_data_len(tables.tcpalog));
> 
> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
> +
>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>                                                ACPI_BUILD_RSDP_FILE, 0);
> 
> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h index 88d0738..7f7b55c 100644
> --- a/include/hw/acpi/aml-build.h
> +++ b/include/hw/acpi/aml-build.h
> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>      GArray *rsdp;
>      GArray *tcpalog;
>      GArray *vmgenid;
> +    GArray *hardware_errors;
>      BIOSLinker *linker;
>  } AcpiBuildTables;
> 
> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h new file mode 100644 index 0000000..0772756
> --- /dev/null
> +++ b/include/hw/acpi/hest_ghes.h
> @@ -0,0 +1,47 @@
> +/*
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> +
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * Authors:
> + *   Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * You should have received a copy of the GNU General Public License
> + along
> + * with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef ACPI_GHES_H
> +#define ACPI_GHES_H
> +
> +#include "hw/acpi/bios-linker-loader.h"
> +
> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
> +
> +#define GHES_GAS_ADDRESS_OFFSET              4
> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
> +#define GHES_NOTIFICATION_STRUCTURE          32
> +
> +#define GHES_CPER_OK   1
> +#define GHES_CPER_FAIL 0
> +
> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
> +/* The max size in Bytes for one error block */
> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
> +
> +
> +typedef struct GhesState {
> +    uint64_t ghes_addr_le;
> +} GhesState;
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                            BIOSLinker *linker); void
> +ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors); bool
> +ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
> +#endif
> --
> 1.8.3.1

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 129+ messages in thread

* 答复: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-18 17:19     ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-18 17:19 UTC (permalink / raw)
  To: mst, imammedo, Zhaoshenglong, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, Shiju Jose, zjzhang, linux-ar
  Cc: Zhengqiang (turing), Wuquanming, Huangshaoyu, Linuxarm

Loop Laszlo


> -----邮件原件-----
> 发件人: gengdongjiu
> 发送时间: 2017年8月18日 22:24
> 收件人: mst@redhat.com; imammedo@redhat.com; Zhaoshenglong <zhaoshenglong@huawei.com>; peter.maydell@linaro.org;
> pbonzini@redhat.com; qemu-devel@nongnu.org; qemu-arm@nongnu.org; kvm@vger.kernel.org; edk2-devel@lists.01.org;
> christoffer.dall@linaro.org; marc.zyngier@arm.com; will.deacon@arm.com; james.morse@arm.com; tbaicar@codeaurora.org;
> ard.biesheuvel@linaro.org; mingo@kernel.org; bp@suse.de; Shiju Jose <shiju.jose@huawei.com>; zjzhang@codeaurora.org;
> linux-arm-kernel@lists.infradead.org; kvmarm@lists.cs.columbia.edu; linux-kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> devel@acpica.org; John Garry <john.garry@huawei.com>; Jonathan Cameron <jonathan.cameron@huawei.com>; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; huangdaode <huangdaode@hisilicon.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> 抄送: Huangshaoyu <huangshaoyu@huawei.com>; Wuquanming <wuquanming@huawei.com>; Linuxarm <linuxarm@huawei.com>;
> gengdongjiu <gengdongjiu@huawei.com>; Zhengqiang (turing) <zhengqiang10@huawei.com>
> 主题: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
> 
> This implements APEI GHES Table by passing the error CPER info to the guest via a fw_cfg_blob. After a CPER info is recorded, an
> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception will be injected into the guest OS.
> 
> Below is the table layout, the max number of error soure is 11, which is classified by notification type.
> 
>      etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0
> | +--------------------------+ +--------->| ||------------->|
> | +--------------------------+ +--------->| |+------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
> 
> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
> so user space must check the ack value to avoid read-write race condition.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  hw/acpi/aml-build.c         |   2 +
>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c    |   6 +
>  include/hw/acpi/aml-build.h |   1 +
>  include/hw/acpi/hest_ghes.h |  47 ++++++
>  5 files changed, 401 insertions(+)
>  create mode 100644 hw/acpi/hest_ghes.c
>  create mode 100644 include/hw/acpi/hest_ghes.h
> 
> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c index 36a6cc4..6849e5f 100644
> --- a/hw/acpi/aml-build.c
> +++ b/hw/acpi/aml-build.c
> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>      tables->table_data = g_array_new(false, true /* clear */, 1);
>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>      tables->linker = bios_linker_loader_init();  }
> 
> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>      g_array_free(tables->table_data, true);
>      g_array_free(tables->tcpalog, mfre);
>      g_array_free(tables->vmgenid, mfre);
> +    g_array_free(tables->hardware_errors, mfre);
>  }
> 
>  /* Build rsdt table */
> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c new file mode 100644 index 0000000..ff6b5ef
> --- /dev/null
> +++ b/hw/acpi/hest_ghes.c
> @@ -0,0 +1,345 @@
> +/*
> + *  APEI GHES table Generation
> + *
> + *  Copyright (C) 2017 huawei.
> + *
> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qmp-commands.h"
> +#include "hw/acpi/acpi.h"
> +#include "hw/acpi/aml-build.h"
> +#include "hw/acpi/hest_ghes.h"
> +#include "hw/nvram/fw_cfg.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/error-report.h"
> +
> +/* The structure that stands for the layout
> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
> + *
> + *           etc/hardware_errors
> + * ==========================================
> + * +------------------+
> + * |    address       |              +--------------+
> + * |    registers     |              | Error Status |
> + * | +----------------+              | Data Block 0 |
> + * | |status_address0 |------------->| +------------+
> + * | +----------------+              | |  CPER      |
> + * | |status_address1 |----------+   | |  CPER      |
> + * | +----------------+          |   | |  ....      |
> + * | |.............   |          |   | |  CPER      |
> + * | +----------------+          |   | +------------+
> + * | |status_address10|-----+    |   | Error Status |
> + * | +----------------+     |    |   | Data Block 1 |
> + * | |ack_value0      |     |    +-->| +------------+
> + * | +----------------+     |        | |  CPER      |
> + * | |ack_value1      |     |        | |  CPER      |
> + * | +----------------+     |        | |  ....      |
> + * | | .............  |     |        | |  CPER      |
> + * | +----------------+     |        +-+------------+
> + * | |ack_value10     |     |        | |..........  |
> + * | +----------------+     |        | +------------+
> + *                          |        | Error Status |
> + *                          |        | Data Block10 |
> + *                          +------->+------------+
> + *                                   | |  CPER      |
> + *                                   | |  CPER      |
> + *                                   | |  ....      |
> + *                                   | |  CPER      |
> + *                                   +-+------------+
> + */
> +struct hardware_errors_buffer {
> +    /* Generic Error Status Block register */
> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    char
> +gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
> +};
> +
> +static int ghes_record_cper(uint64_t error_block_address,
> +                                    uint64_t error_physical_addr) {
> +    AcpiGenericErrorStatus block;
> +    AcpiGenericErrorData *gdata;
> +    UefiCperSecMemErr *mem_err;
> +    uint64_t current_block_length;
> +    unsigned char *buffer;
> +    /* memory section */
> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
> +                              0x83, 0xB1};
> +
> +    cpu_physical_memory_read(error_block_address, &block,
> +                                sizeof(AcpiGenericErrorStatus));
> +
> +    /* Get the current generic error status block length */
> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
> +        le32_to_cpu(block.data_length);
> +
> +    /* If the Generic Error Status Block is NULL, update
> +     * the block header
> +     */
> +    if (!block.block_status) {
> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    }
> +
> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    /* check whether it runs out of the preallocated memory */
> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
> +       GHES_MAX_RAW_DATA_LENGTH) {
> +        error_report("Record CPER out of boundary!!!");
> +        return GHES_CPER_FAIL;
> +    }
> +
> +    /* Write back the Generic Error Status Block to guest memory */
> +    cpu_physical_memory_write(error_block_address, &block,
> +        sizeof(AcpiGenericErrorStatus));
> +
> +    /* Fill in Generic Error Data Entry */
> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
> +                       sizeof(UefiCperSecMemErr));
> +
> +
> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +    gdata = (AcpiGenericErrorData *)buffer;
> +
> +    /* Memory section */
> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
> +            sizeof(mem_section_id_le));
> +
> +    /* error severity is recoverable */
> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    gdata->revision = 0x300; /* the revision number is 0x300 */
> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
> +
> +    /* User space only handle the memory section CPER */
> +
> +    /* Hard code to Multi-bit ECC error */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
> +    mem_err->error_type =
> + cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
> +
> +    /* Record the physical address at which the memory error occurred */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> +
> +    /* Write back the Generic Error Data Entry to guest memory */
> +    cpu_physical_memory_write(error_block_address + current_block_length,
> +        buffer, sizeof(AcpiGenericErrorData) +
> + sizeof(UefiCperSecMemErr));
> +
> +    g_free(buffer);
> +    return GHES_CPER_OK;
> +}
> +
> +static void
> +build_address(GArray *table_data, BIOSLinker *linker,
> +    uint32_t dst_patched_offset, uint32_t src_offset,
> +    uint8_t address_space_id , uint8_t  register_bit_width,
> +    uint8_t register_bit_offset, uint8_t access_size) {
> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    /* Address space */
> +    build_append_int_noprefix(table_data, address_space_id, 1);
> +    /* register bit width */
> +    build_append_int_noprefix(table_data, register_bit_width, 1);
> +    /* register bit offset */
> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
> +    /* access size */
> +    build_append_int_noprefix(table_data, access_size, 1);
> +    acpi_data_push(table_data, address_size);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
> +     * can retrieve and read it. the address size is 64 bits.
> +     */
> +    bios_linker_loader_add_pointer(linker,
> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
> +        GHES_ERRORS_FW_CFG_FILE, src_offset); }
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                                            BIOSLinker *linker) {
> +    uint32_t ghes_start = table_data->len;
> +    uint32_t address_size, error_status_address_offset;
> +    uint32_t read_ack_register_offset, i;
> +
> +    address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    error_status_address_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    read_ack_register_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    acpi_data_push(hardware_error,
> +        offsetof(struct hardware_errors_buffer, ack_value));
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Initialize read ack register */
> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
> +
> +    /* Reserved the total size for ERRORS fw_cfg blob
> +     */
> +    acpi_data_push(hardware_error, sizeof(struct
> + hardware_errors_buffer));
> +
> +    /* Allocate guest memory for the Data fw_cfg blob */
> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
> +                            1, false);
> +    /* Reserve table header size */
> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
> +
> +    build_append_int_noprefix(table_data,
> + GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
> +        build_append_int_noprefix(table_data,
> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
> +        /* source id */
> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
> +        /* related source id */
> +        build_append_int_noprefix(table_data, 0xffff, 2);
> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
> +
> +        /* Currently only enable SEA notification type to avoid the kernel
> +         * warning, reserve the space for other notification error source
> +         */
> +        if (i == ACPI_HEST_NOTIFY_SEA) {
> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
> +        } else {
> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
> +        }
> +
> +        /* The number of error status block per generic hardware error source */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max sections per record */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max raw data length */
> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH,
> + 4);
> +
> +        /* Build error status address*/
> +        build_address(table_data, linker, error_status_address_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> +
> +        /* Hardware error notification structure */
> +        build_append_int_noprefix(table_data, i, 1); /* type */
> +        /* length */
> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
> +        build_append_int_noprefix(table_data, 0, 26);
> +
> +        /* Error Status Block Length */
> +        build_append_int_noprefix(table_data,
> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
> +
> +        /* Build read ack register */
> +        build_address(table_data, linker, read_ack_register_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2),
> +            offsetof(struct hardware_errors_buffer, ack_value) +
> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
> +            4 /* QWord access */);
> +
> +        /* Read ack preserve */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe),
> + 8);
> +
> +        /* Read ack write */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
> +    }
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Patch address of generic error status block into
> +         * the address register so OSPM can retrieve and read it.
> +         */
> +        bios_linker_loader_add_pointer(linker,
> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
> +            GHES_ERRORS_FW_CFG_FILE,
> +            offsetof(struct hardware_errors_buffer, gesb) +
> +            i * GHES_MAX_RAW_DATA_LENGTH);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
> +     * so QEMU can write the ERRORS there. The address is expected to be
> +     * < 4GB, but write 64 bits anyway.
> +     */
> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
> +        offsetof(struct hardware_errors_buffer, gesb));
> +
> +    build_header(linker, table_data,
> +        (void *)(table_data->data + ghes_start), "HEST",
> +        table_data->len - ghes_start, 1, NULL, "GHES"); }
> +
> +static GhesState ges;
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error) {
> +
> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
> +
> +    /* Create a read-only fw_cfg file for GHES */
> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
> +                    size);
> +    /* Create a read-write fw_cfg file for Address */
> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false); }
> +
> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address) {
> +    uint64_t error_block_addr;
> +    uint64_t ack_value_addr, ack_value = 0;
> +    int loop = 0, ack_value_size;
> +    bool ret = GHES_CPER_FAIL;
> +
> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
> +        offsetof(struct hardware_errors_buffer, ack_value)) /
> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
> +
> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
> +        error_block_addr = le32_to_cpu(error_block_addr);
> +
> +        ack_value_addr = ges.ghes_addr_le -
> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
> +retry:
> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
> +        if (!ack_value) {
> +            if (loop < 3) {
> +                usleep(100 * 1000);
> +                loop++;
> +                goto retry;
> +            } else {
> +                error_report("Last time OSPM does not acknowledge the error,"
> +                    " record CPER failed this time, set the ack value to"
> +                    " avoid blocking next time CPER record! exit");
> +                ack_value = 1;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                return ret;
> +            }
> +        } else {
> +            /* A zero value in ghes_addr means that BIOS has not yet written
> +             * the address
> +             */
> +            if (error_block_addr) {
> +                ack_value = 0;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                ret = ghes_record_cper(error_block_addr, physical_address);
> +            }
> +        }
> +    }
> +    return ret;
> +}
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index 3d78ff6..def1ec1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -45,6 +45,7 @@
>  #include "hw/arm/virt.h"
>  #include "sysemu/numa.h"
>  #include "kvm_arm.h"
> +#include "hw/acpi/hest_ghes.h"
> 
>  #define ARM_SPI_BASE 32
>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>      acpi_add_table(table_offsets, tables_blob);
>      build_spcr(tables_blob, tables->linker, vms);
> 
> +    acpi_add_table(table_offsets, tables_blob);
> +    ghes_build_acpi(tables_blob, tables->hardware_errors,
> + tables->linker);
> +
>      if (nb_numa_nodes > 0) {
>          acpi_add_table(table_offsets, tables_blob);
>          build_srat(tables_blob, tables->linker, vms); @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>                      acpi_data_len(tables.tcpalog));
> 
> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
> +
>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>                                                ACPI_BUILD_RSDP_FILE, 0);
> 
> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h index 88d0738..7f7b55c 100644
> --- a/include/hw/acpi/aml-build.h
> +++ b/include/hw/acpi/aml-build.h
> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>      GArray *rsdp;
>      GArray *tcpalog;
>      GArray *vmgenid;
> +    GArray *hardware_errors;
>      BIOSLinker *linker;
>  } AcpiBuildTables;
> 
> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h new file mode 100644 index 0000000..0772756
> --- /dev/null
> +++ b/include/hw/acpi/hest_ghes.h
> @@ -0,0 +1,47 @@
> +/*
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> +
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * Authors:
> + *   Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * You should have received a copy of the GNU General Public License
> + along
> + * with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef ACPI_GHES_H
> +#define ACPI_GHES_H
> +
> +#include "hw/acpi/bios-linker-loader.h"
> +
> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
> +
> +#define GHES_GAS_ADDRESS_OFFSET              4
> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
> +#define GHES_NOTIFICATION_STRUCTURE          32
> +
> +#define GHES_CPER_OK   1
> +#define GHES_CPER_FAIL 0
> +
> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
> +/* The max size in Bytes for one error block */
> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
> +
> +
> +typedef struct GhesState {
> +    uint64_t ghes_addr_le;
> +} GhesState;
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                            BIOSLinker *linker); void
> +ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors); bool
> +ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
> +#endif
> --
> 1.8.3.1

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 129+ messages in thread

* 答复: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-18 17:19     ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-18 17:19 UTC (permalink / raw)
  To: linux-arm-kernel

Loop Laszlo


> -----????-----
> ???: gengdongjiu
> ????: 2017?8?18? 22:24
> ???: mst at redhat.com; imammedo at redhat.com; Zhaoshenglong <zhaoshenglong@huawei.com>; peter.maydell at linaro.org;
> pbonzini at redhat.com; qemu-devel at nongnu.org; qemu-arm at nongnu.org; kvm at vger.kernel.org; edk2-devel at lists.01.org;
> christoffer.dall at linaro.org; marc.zyngier at arm.com; will.deacon at arm.com; james.morse at arm.com; tbaicar at codeaurora.org;
> ard.biesheuvel at linaro.org; mingo at kernel.org; bp at suse.de; Shiju Jose <shiju.jose@huawei.com>; zjzhang at codeaurora.org;
> linux-arm-kernel at lists.infradead.org; kvmarm at lists.cs.columbia.edu; linux-kernel at vger.kernel.org; linux-acpi at vger.kernel.org;
> devel at acpica.org; John Garry <john.garry@huawei.com>; Jonathan Cameron <jonathan.cameron@huawei.com>; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; huangdaode <huangdaode@hisilicon.com>; Wangzhou (B) <wangzhou1@hisilicon.com>
> ??: Huangshaoyu <huangshaoyu@huawei.com>; Wuquanming <wuquanming@huawei.com>; Linuxarm <linuxarm@huawei.com>;
> gengdongjiu <gengdongjiu@huawei.com>; Zhengqiang (turing) <zhengqiang10@huawei.com>
> ??: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
> 
> This implements APEI GHES Table by passing the error CPER info to the guest via a fw_cfg_blob. After a CPER info is recorded, an
> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception will be injected into the guest OS.
> 
> Below is the table layout, the max number of error soure is 11, which is classified by notification type.
> 
>      etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0
> | +--------------------------+ +--------->| ||------------->|
> | +--------------------------+ +--------->| |+------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
> 
> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
> so user space must check the ack value to avoid read-write race condition.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  hw/acpi/aml-build.c         |   2 +
>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c    |   6 +
>  include/hw/acpi/aml-build.h |   1 +
>  include/hw/acpi/hest_ghes.h |  47 ++++++
>  5 files changed, 401 insertions(+)
>  create mode 100644 hw/acpi/hest_ghes.c
>  create mode 100644 include/hw/acpi/hest_ghes.h
> 
> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c index 36a6cc4..6849e5f 100644
> --- a/hw/acpi/aml-build.c
> +++ b/hw/acpi/aml-build.c
> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>      tables->table_data = g_array_new(false, true /* clear */, 1);
>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>      tables->linker = bios_linker_loader_init();  }
> 
> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>      g_array_free(tables->table_data, true);
>      g_array_free(tables->tcpalog, mfre);
>      g_array_free(tables->vmgenid, mfre);
> +    g_array_free(tables->hardware_errors, mfre);
>  }
> 
>  /* Build rsdt table */
> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c new file mode 100644 index 0000000..ff6b5ef
> --- /dev/null
> +++ b/hw/acpi/hest_ghes.c
> @@ -0,0 +1,345 @@
> +/*
> + *  APEI GHES table Generation
> + *
> + *  Copyright (C) 2017 huawei.
> + *
> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qmp-commands.h"
> +#include "hw/acpi/acpi.h"
> +#include "hw/acpi/aml-build.h"
> +#include "hw/acpi/hest_ghes.h"
> +#include "hw/nvram/fw_cfg.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/error-report.h"
> +
> +/* The structure that stands for the layout
> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
> + *
> + *           etc/hardware_errors
> + * ==========================================
> + * +------------------+
> + * |    address       |              +--------------+
> + * |    registers     |              | Error Status |
> + * | +----------------+              | Data Block 0 |
> + * | |status_address0 |------------->| +------------+
> + * | +----------------+              | |  CPER      |
> + * | |status_address1 |----------+   | |  CPER      |
> + * | +----------------+          |   | |  ....      |
> + * | |.............   |          |   | |  CPER      |
> + * | +----------------+          |   | +------------+
> + * | |status_address10|-----+    |   | Error Status |
> + * | +----------------+     |    |   | Data Block 1 |
> + * | |ack_value0      |     |    +-->| +------------+
> + * | +----------------+     |        | |  CPER      |
> + * | |ack_value1      |     |        | |  CPER      |
> + * | +----------------+     |        | |  ....      |
> + * | | .............  |     |        | |  CPER      |
> + * | +----------------+     |        +-+------------+
> + * | |ack_value10     |     |        | |..........  |
> + * | +----------------+     |        | +------------+
> + *                          |        | Error Status |
> + *                          |        | Data Block10 |
> + *                          +------->+------------+
> + *                                   | |  CPER      |
> + *                                   | |  CPER      |
> + *                                   | |  ....      |
> + *                                   | |  CPER      |
> + *                                   +-+------------+
> + */
> +struct hardware_errors_buffer {
> +    /* Generic Error Status Block register */
> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    char
> +gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
> +};
> +
> +static int ghes_record_cper(uint64_t error_block_address,
> +                                    uint64_t error_physical_addr) {
> +    AcpiGenericErrorStatus block;
> +    AcpiGenericErrorData *gdata;
> +    UefiCperSecMemErr *mem_err;
> +    uint64_t current_block_length;
> +    unsigned char *buffer;
> +    /* memory section */
> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
> +                              0x83, 0xB1};
> +
> +    cpu_physical_memory_read(error_block_address, &block,
> +                                sizeof(AcpiGenericErrorStatus));
> +
> +    /* Get the current generic error status block length */
> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
> +        le32_to_cpu(block.data_length);
> +
> +    /* If the Generic Error Status Block is NULL, update
> +     * the block header
> +     */
> +    if (!block.block_status) {
> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    }
> +
> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    /* check whether it runs out of the preallocated memory */
> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
> +       GHES_MAX_RAW_DATA_LENGTH) {
> +        error_report("Record CPER out of boundary!!!");
> +        return GHES_CPER_FAIL;
> +    }
> +
> +    /* Write back the Generic Error Status Block to guest memory */
> +    cpu_physical_memory_write(error_block_address, &block,
> +        sizeof(AcpiGenericErrorStatus));
> +
> +    /* Fill in Generic Error Data Entry */
> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
> +                       sizeof(UefiCperSecMemErr));
> +
> +
> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +    gdata = (AcpiGenericErrorData *)buffer;
> +
> +    /* Memory section */
> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
> +            sizeof(mem_section_id_le));
> +
> +    /* error severity is recoverable */
> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    gdata->revision = 0x300; /* the revision number is 0x300 */
> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
> +
> +    /* User space only handle the memory section CPER */
> +
> +    /* Hard code to Multi-bit ECC error */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
> +    mem_err->error_type =
> + cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
> +
> +    /* Record the physical address at which the memory error occurred */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> +
> +    /* Write back the Generic Error Data Entry to guest memory */
> +    cpu_physical_memory_write(error_block_address + current_block_length,
> +        buffer, sizeof(AcpiGenericErrorData) +
> + sizeof(UefiCperSecMemErr));
> +
> +    g_free(buffer);
> +    return GHES_CPER_OK;
> +}
> +
> +static void
> +build_address(GArray *table_data, BIOSLinker *linker,
> +    uint32_t dst_patched_offset, uint32_t src_offset,
> +    uint8_t address_space_id , uint8_t  register_bit_width,
> +    uint8_t register_bit_offset, uint8_t access_size) {
> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    /* Address space */
> +    build_append_int_noprefix(table_data, address_space_id, 1);
> +    /* register bit width */
> +    build_append_int_noprefix(table_data, register_bit_width, 1);
> +    /* register bit offset */
> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
> +    /* access size */
> +    build_append_int_noprefix(table_data, access_size, 1);
> +    acpi_data_push(table_data, address_size);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
> +     * can retrieve and read it. the address size is 64 bits.
> +     */
> +    bios_linker_loader_add_pointer(linker,
> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
> +        GHES_ERRORS_FW_CFG_FILE, src_offset); }
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                                            BIOSLinker *linker) {
> +    uint32_t ghes_start = table_data->len;
> +    uint32_t address_size, error_status_address_offset;
> +    uint32_t read_ack_register_offset, i;
> +
> +    address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    error_status_address_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    read_ack_register_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    acpi_data_push(hardware_error,
> +        offsetof(struct hardware_errors_buffer, ack_value));
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Initialize read ack register */
> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
> +
> +    /* Reserved the total size for ERRORS fw_cfg blob
> +     */
> +    acpi_data_push(hardware_error, sizeof(struct
> + hardware_errors_buffer));
> +
> +    /* Allocate guest memory for the Data fw_cfg blob */
> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
> +                            1, false);
> +    /* Reserve table header size */
> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
> +
> +    build_append_int_noprefix(table_data,
> + GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
> +        build_append_int_noprefix(table_data,
> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
> +        /* source id */
> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
> +        /* related source id */
> +        build_append_int_noprefix(table_data, 0xffff, 2);
> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
> +
> +        /* Currently only enable SEA notification type to avoid the kernel
> +         * warning, reserve the space for other notification error source
> +         */
> +        if (i == ACPI_HEST_NOTIFY_SEA) {
> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
> +        } else {
> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
> +        }
> +
> +        /* The number of error status block per generic hardware error source */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max sections per record */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max raw data length */
> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH,
> + 4);
> +
> +        /* Build error status address*/
> +        build_address(table_data, linker, error_status_address_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> +
> +        /* Hardware error notification structure */
> +        build_append_int_noprefix(table_data, i, 1); /* type */
> +        /* length */
> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
> +        build_append_int_noprefix(table_data, 0, 26);
> +
> +        /* Error Status Block Length */
> +        build_append_int_noprefix(table_data,
> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
> +
> +        /* Build read ack register */
> +        build_address(table_data, linker, read_ack_register_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2),
> +            offsetof(struct hardware_errors_buffer, ack_value) +
> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
> +            4 /* QWord access */);
> +
> +        /* Read ack preserve */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe),
> + 8);
> +
> +        /* Read ack write */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
> +    }
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Patch address of generic error status block into
> +         * the address register so OSPM can retrieve and read it.
> +         */
> +        bios_linker_loader_add_pointer(linker,
> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
> +            GHES_ERRORS_FW_CFG_FILE,
> +            offsetof(struct hardware_errors_buffer, gesb) +
> +            i * GHES_MAX_RAW_DATA_LENGTH);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
> +     * so QEMU can write the ERRORS there. The address is expected to be
> +     * < 4GB, but write 64 bits anyway.
> +     */
> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
> +        offsetof(struct hardware_errors_buffer, gesb));
> +
> +    build_header(linker, table_data,
> +        (void *)(table_data->data + ghes_start), "HEST",
> +        table_data->len - ghes_start, 1, NULL, "GHES"); }
> +
> +static GhesState ges;
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error) {
> +
> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
> +
> +    /* Create a read-only fw_cfg file for GHES */
> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
> +                    size);
> +    /* Create a read-write fw_cfg file for Address */
> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false); }
> +
> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address) {
> +    uint64_t error_block_addr;
> +    uint64_t ack_value_addr, ack_value = 0;
> +    int loop = 0, ack_value_size;
> +    bool ret = GHES_CPER_FAIL;
> +
> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
> +        offsetof(struct hardware_errors_buffer, ack_value)) /
> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
> +
> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
> +        error_block_addr = le32_to_cpu(error_block_addr);
> +
> +        ack_value_addr = ges.ghes_addr_le -
> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
> +retry:
> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
> +        if (!ack_value) {
> +            if (loop < 3) {
> +                usleep(100 * 1000);
> +                loop++;
> +                goto retry;
> +            } else {
> +                error_report("Last time OSPM does not acknowledge the error,"
> +                    " record CPER failed this time, set the ack value to"
> +                    " avoid blocking next time CPER record! exit");
> +                ack_value = 1;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                return ret;
> +            }
> +        } else {
> +            /* A zero value in ghes_addr means that BIOS has not yet written
> +             * the address
> +             */
> +            if (error_block_addr) {
> +                ack_value = 0;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                ret = ghes_record_cper(error_block_addr, physical_address);
> +            }
> +        }
> +    }
> +    return ret;
> +}
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index 3d78ff6..def1ec1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -45,6 +45,7 @@
>  #include "hw/arm/virt.h"
>  #include "sysemu/numa.h"
>  #include "kvm_arm.h"
> +#include "hw/acpi/hest_ghes.h"
> 
>  #define ARM_SPI_BASE 32
>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>      acpi_add_table(table_offsets, tables_blob);
>      build_spcr(tables_blob, tables->linker, vms);
> 
> +    acpi_add_table(table_offsets, tables_blob);
> +    ghes_build_acpi(tables_blob, tables->hardware_errors,
> + tables->linker);
> +
>      if (nb_numa_nodes > 0) {
>          acpi_add_table(table_offsets, tables_blob);
>          build_srat(tables_blob, tables->linker, vms); @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>                      acpi_data_len(tables.tcpalog));
> 
> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
> +
>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>                                                ACPI_BUILD_RSDP_FILE, 0);
> 
> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h index 88d0738..7f7b55c 100644
> --- a/include/hw/acpi/aml-build.h
> +++ b/include/hw/acpi/aml-build.h
> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>      GArray *rsdp;
>      GArray *tcpalog;
>      GArray *vmgenid;
> +    GArray *hardware_errors;
>      BIOSLinker *linker;
>  } AcpiBuildTables;
> 
> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h new file mode 100644 index 0000000..0772756
> --- /dev/null
> +++ b/include/hw/acpi/hest_ghes.h
> @@ -0,0 +1,47 @@
> +/*
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> +
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * Authors:
> + *   Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * You should have received a copy of the GNU General Public License
> + along
> + * with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef ACPI_GHES_H
> +#define ACPI_GHES_H
> +
> +#include "hw/acpi/bios-linker-loader.h"
> +
> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
> +
> +#define GHES_GAS_ADDRESS_OFFSET              4
> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
> +#define GHES_NOTIFICATION_STRUCTURE          32
> +
> +#define GHES_CPER_OK   1
> +#define GHES_CPER_FAIL 0
> +
> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
> +/* The max size in Bytes for one error block */
> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
> +
> +
> +typedef struct GhesState {
> +    uint64_t ghes_addr_le;
> +} GhesState;
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                            BIOSLinker *linker); void
> +ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors); bool
> +ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
> +#endif
> --
> 1.8.3.1

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
  2017-08-18 14:23   ` Dongjiu Geng
  (?)
  (?)
@ 2017-08-24 12:33     ` Shannon Zhao
  -1 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-24 12:33 UTC (permalink / raw)
  To: Dongjiu Geng, mst, imammedo, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/18 22:23, Dongjiu Geng wrote:
> (1) Add related APEI/HEST table structures and  macros, these
>     definition refer to ACPI 6.1 and UEFI 2.6 spec.
> (2) Add generic error status block and CPER memory section
>     definition, user space only handle memory section errors.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 193 insertions(+)
> 
> diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
> index 72be675..3b4bad7 100644
> --- a/include/hw/acpi/acpi-defs.h
> +++ b/include/hw/acpi/acpi-defs.h
> @@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
>  #define ACPI_APIC_GENERIC_TRANSLATOR    15
>  #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
>  
> +/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
missing "

> +#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
> +#define UEFI_CPER_MEM_VALID_PA               0x0002
> +#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
> +#define UEFI_CPER_MEM_VALID_NODE             0x0008
> +#define UEFI_CPER_MEM_VALID_CARD             0x0010
> +#define UEFI_CPER_MEM_VALID_MODULE           0x0020
> +#define UEFI_CPER_MEM_VALID_BANK             0x0040
> +#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
> +#define UEFI_CPER_MEM_VALID_ROW              0x0100
> +#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
> +#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
> +#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
> +#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
> +#define UEFI_CPER_MEM_VALID_TARGET           0x2000
> +#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
> +#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
> +#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
> +#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
> +#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
> +
> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
> +
It's better to refer to the first spec version of this structure and
same with others you define.

> +enum AcpiHestNotifyType {
> +    ACPI_HEST_NOTIFY_POLLED = 0,
> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
> +    ACPI_HEST_NOTIFY_LOCAL = 2,
> +    ACPI_HEST_NOTIFY_SCI = 3,
> +    ACPI_HEST_NOTIFY_NMI = 4,
> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
your patchset?

> +};
> +
>  /*
>   * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
>   */
> @@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
>  } QEMU_PACKED;
>  typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
>  
> +/* Hardware Error Notification, from the ACPI 6.1
> + * spec, "18.3.2.9 Hardware Error Notification"
> + */
Use below style for multiple comment lines
/*
 * XXX
 */

> +struct AcpiHestNotify {
> +    uint8_t type;
> +    uint8_t length;
> +    uint16_t config_write_enable;
> +    uint32_t poll_interval;
> +    uint32_t vector;
> +    uint32_t polling_threshold_value;
> +    uint32_t polling_threshold_window;
> +    uint32_t error_threshold_value;
> +    uint32_t error_threshold_window;
> +} QEMU_PACKED;
> +typedef struct AcpiHestNotify AcpiHestNotify;
> +
> +/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
> + * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
> + */
> +enum AcpiHestSourceType {
> +    ACPI_HEST_SOURCE_IA32_CHECK = 0,
> +    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
> +    ACPI_HEST_SOURCE_IA32_NMI = 2,
What's 3, 4, 5 for?

> +    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
> +    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
> +    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
> +    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
> +};
> +
> +/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
> +#define ACPI_GEBS_UNCORRECTABLE             (1)
> +#define ACPI_GEBS_CORRECTABLE               (1 << 1)
> +#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
> +#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
> +/* 10 bits, error data entry count */
> +#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
> +
> +/* Generic Hardware Error Source Structure, refer to ACPI 6.1
> + * "18.3.2.7 Generic Hardware Error Source". in this struct the
> + * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
> + */
> +
> +struct AcpiGenericHardwareErrorSource {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
> +
> +/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
> + * Hardware Error Source version 2", in this struct the "type" field has to
> + * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
> + */
> +struct AcpiGenericHardwareErrorSourceV2 {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length;
> +    struct AcpiGenericAddress read_ack_register;
> +    uint64_t read_ack_preserve;
> +    uint64_t read_ack_write;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericHardwareErrorSourceV2
> +            AcpiGenericHardwareErrorSourceV2;
> +
> +/* Generic Error Status block, from ACPI 6.1,
> + * "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorStatus {
> +    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
> +    uint32_t block_status;
> +    uint32_t raw_data_offset;
> +    uint32_t raw_data_length;
> +    uint32_t data_length;
> +    uint32_t error_severity;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
> +
> +enum AcpiGenericErrorSeverity {
> +    ACPI_CPER_SEV_RECOVERABLE,
> +    ACPI_CPER_SEV_FATAL,
> +    ACPI_CPER_SEV_CORRECTED,
> +    ACPI_CPER_SEV_NONE,
> +};
> +
> +/* Generic Error Data entry, revision number is 0x0300,
> + * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorData {
> +    uint8_t section_type_le[16];
> +    /* The "error_severity" fields that they take their
> +     * values from AcpiGenericErrorSeverity
> +     */
> +    uint32_t error_severity;
> +    uint16_t revision;
> +    uint8_t validation_bits;
> +    uint8_t flags;
> +    uint32_t error_data_length;
> +    uint8_t fru_id[16];
> +    uint8_t fru_text[20];
> +    uint64_t time_stamp;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorData AcpiGenericErrorData;
> +
> +/* From UEFI 2.6, "N.2.5 Memory Error Section" */
> +struct UefiCperSecMemErr {
> +    uint64_t    validation_bits;
> +    uint64_t    error_status;
> +    uint64_t    physical_addr;
> +    uint64_t    physical_addr_mask;
> +    uint16_t    node;
> +    uint16_t    card;
> +    uint16_t    module;
> +    uint16_t    bank;
> +    uint16_t    device;
> +    uint16_t    row;
> +    uint16_t    column;
> +    uint16_t    bit_pos;
> +    uint64_t    requestor_id;
> +    uint64_t    responder_id;
> +    uint64_t    target_id;
> +    uint8_t     error_type;
> +    uint8_t     reserved;
> +    uint16_t    rank;
> +    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
> +    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
> +} QEMU_PACKED;
> +typedef struct UefiCperSecMemErr UefiCperSecMemErr;
> +
> +/*
> + * HEST Description Table
> + */
> +struct AcpiHardwareErrorSourceTable {
> +    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
> +    uint32_t           error_source_count;
> +} QEMU_PACKED;
> +typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
> +
>  #define ACPI_SRAT_PROCESSOR_APIC     0
>  #define ACPI_SRAT_MEMORY             1
>  #define ACPI_SRAT_PROCESSOR_x2APIC   2
> 

-- 
Shannon


^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-24 12:33     ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-24 12:33 UTC (permalink / raw)
  To: Dongjiu Geng, mst, imammedo, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/18 22:23, Dongjiu Geng wrote:
> (1) Add related APEI/HEST table structures and  macros, these
>     definition refer to ACPI 6.1 and UEFI 2.6 spec.
> (2) Add generic error status block and CPER memory section
>     definition, user space only handle memory section errors.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 193 insertions(+)
> 
> diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
> index 72be675..3b4bad7 100644
> --- a/include/hw/acpi/acpi-defs.h
> +++ b/include/hw/acpi/acpi-defs.h
> @@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
>  #define ACPI_APIC_GENERIC_TRANSLATOR    15
>  #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
>  
> +/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
missing "

> +#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
> +#define UEFI_CPER_MEM_VALID_PA               0x0002
> +#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
> +#define UEFI_CPER_MEM_VALID_NODE             0x0008
> +#define UEFI_CPER_MEM_VALID_CARD             0x0010
> +#define UEFI_CPER_MEM_VALID_MODULE           0x0020
> +#define UEFI_CPER_MEM_VALID_BANK             0x0040
> +#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
> +#define UEFI_CPER_MEM_VALID_ROW              0x0100
> +#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
> +#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
> +#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
> +#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
> +#define UEFI_CPER_MEM_VALID_TARGET           0x2000
> +#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
> +#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
> +#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
> +#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
> +#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
> +
> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
> +
It's better to refer to the first spec version of this structure and
same with others you define.

> +enum AcpiHestNotifyType {
> +    ACPI_HEST_NOTIFY_POLLED = 0,
> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
> +    ACPI_HEST_NOTIFY_LOCAL = 2,
> +    ACPI_HEST_NOTIFY_SCI = 3,
> +    ACPI_HEST_NOTIFY_NMI = 4,
> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
your patchset?

> +};
> +
>  /*
>   * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
>   */
> @@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
>  } QEMU_PACKED;
>  typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
>  
> +/* Hardware Error Notification, from the ACPI 6.1
> + * spec, "18.3.2.9 Hardware Error Notification"
> + */
Use below style for multiple comment lines
/*
 * XXX
 */

> +struct AcpiHestNotify {
> +    uint8_t type;
> +    uint8_t length;
> +    uint16_t config_write_enable;
> +    uint32_t poll_interval;
> +    uint32_t vector;
> +    uint32_t polling_threshold_value;
> +    uint32_t polling_threshold_window;
> +    uint32_t error_threshold_value;
> +    uint32_t error_threshold_window;
> +} QEMU_PACKED;
> +typedef struct AcpiHestNotify AcpiHestNotify;
> +
> +/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
> + * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
> + */
> +enum AcpiHestSourceType {
> +    ACPI_HEST_SOURCE_IA32_CHECK = 0,
> +    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
> +    ACPI_HEST_SOURCE_IA32_NMI = 2,
What's 3, 4, 5 for?

> +    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
> +    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
> +    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
> +    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
> +};
> +
> +/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
> +#define ACPI_GEBS_UNCORRECTABLE             (1)
> +#define ACPI_GEBS_CORRECTABLE               (1 << 1)
> +#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
> +#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
> +/* 10 bits, error data entry count */
> +#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
> +
> +/* Generic Hardware Error Source Structure, refer to ACPI 6.1
> + * "18.3.2.7 Generic Hardware Error Source". in this struct the
> + * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
> + */
> +
> +struct AcpiGenericHardwareErrorSource {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
> +
> +/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
> + * Hardware Error Source version 2", in this struct the "type" field has to
> + * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
> + */
> +struct AcpiGenericHardwareErrorSourceV2 {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length;
> +    struct AcpiGenericAddress read_ack_register;
> +    uint64_t read_ack_preserve;
> +    uint64_t read_ack_write;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericHardwareErrorSourceV2
> +            AcpiGenericHardwareErrorSourceV2;
> +
> +/* Generic Error Status block, from ACPI 6.1,
> + * "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorStatus {
> +    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
> +    uint32_t block_status;
> +    uint32_t raw_data_offset;
> +    uint32_t raw_data_length;
> +    uint32_t data_length;
> +    uint32_t error_severity;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
> +
> +enum AcpiGenericErrorSeverity {
> +    ACPI_CPER_SEV_RECOVERABLE,
> +    ACPI_CPER_SEV_FATAL,
> +    ACPI_CPER_SEV_CORRECTED,
> +    ACPI_CPER_SEV_NONE,
> +};
> +
> +/* Generic Error Data entry, revision number is 0x0300,
> + * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorData {
> +    uint8_t section_type_le[16];
> +    /* The "error_severity" fields that they take their
> +     * values from AcpiGenericErrorSeverity
> +     */
> +    uint32_t error_severity;
> +    uint16_t revision;
> +    uint8_t validation_bits;
> +    uint8_t flags;
> +    uint32_t error_data_length;
> +    uint8_t fru_id[16];
> +    uint8_t fru_text[20];
> +    uint64_t time_stamp;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorData AcpiGenericErrorData;
> +
> +/* From UEFI 2.6, "N.2.5 Memory Error Section" */
> +struct UefiCperSecMemErr {
> +    uint64_t    validation_bits;
> +    uint64_t    error_status;
> +    uint64_t    physical_addr;
> +    uint64_t    physical_addr_mask;
> +    uint16_t    node;
> +    uint16_t    card;
> +    uint16_t    module;
> +    uint16_t    bank;
> +    uint16_t    device;
> +    uint16_t    row;
> +    uint16_t    column;
> +    uint16_t    bit_pos;
> +    uint64_t    requestor_id;
> +    uint64_t    responder_id;
> +    uint64_t    target_id;
> +    uint8_t     error_type;
> +    uint8_t     reserved;
> +    uint16_t    rank;
> +    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
> +    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
> +} QEMU_PACKED;
> +typedef struct UefiCperSecMemErr UefiCperSecMemErr;
> +
> +/*
> + * HEST Description Table
> + */
> +struct AcpiHardwareErrorSourceTable {
> +    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
> +    uint32_t           error_source_count;
> +} QEMU_PACKED;
> +typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
> +
>  #define ACPI_SRAT_PROCESSOR_APIC     0
>  #define ACPI_SRAT_MEMORY             1
>  #define ACPI_SRAT_PROCESSOR_x2APIC   2
> 

-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-24 12:33     ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-24 12:33 UTC (permalink / raw)
  To: Dongjiu Geng, mst, imammedo, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/18 22:23, Dongjiu Geng wrote:
> (1) Add related APEI/HEST table structures and  macros, these
>     definition refer to ACPI 6.1 and UEFI 2.6 spec.
> (2) Add generic error status block and CPER memory section
>     definition, user space only handle memory section errors.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 193 insertions(+)
> 
> diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
> index 72be675..3b4bad7 100644
> --- a/include/hw/acpi/acpi-defs.h
> +++ b/include/hw/acpi/acpi-defs.h
> @@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
>  #define ACPI_APIC_GENERIC_TRANSLATOR    15
>  #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
>  
> +/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
missing "

> +#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
> +#define UEFI_CPER_MEM_VALID_PA               0x0002
> +#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
> +#define UEFI_CPER_MEM_VALID_NODE             0x0008
> +#define UEFI_CPER_MEM_VALID_CARD             0x0010
> +#define UEFI_CPER_MEM_VALID_MODULE           0x0020
> +#define UEFI_CPER_MEM_VALID_BANK             0x0040
> +#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
> +#define UEFI_CPER_MEM_VALID_ROW              0x0100
> +#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
> +#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
> +#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
> +#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
> +#define UEFI_CPER_MEM_VALID_TARGET           0x2000
> +#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
> +#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
> +#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
> +#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
> +#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
> +
> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
> +
It's better to refer to the first spec version of this structure and
same with others you define.

> +enum AcpiHestNotifyType {
> +    ACPI_HEST_NOTIFY_POLLED = 0,
> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
> +    ACPI_HEST_NOTIFY_LOCAL = 2,
> +    ACPI_HEST_NOTIFY_SCI = 3,
> +    ACPI_HEST_NOTIFY_NMI = 4,
> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
your patchset?

> +};
> +
>  /*
>   * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
>   */
> @@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
>  } QEMU_PACKED;
>  typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
>  
> +/* Hardware Error Notification, from the ACPI 6.1
> + * spec, "18.3.2.9 Hardware Error Notification"
> + */
Use below style for multiple comment lines
/*
 * XXX
 */

> +struct AcpiHestNotify {
> +    uint8_t type;
> +    uint8_t length;
> +    uint16_t config_write_enable;
> +    uint32_t poll_interval;
> +    uint32_t vector;
> +    uint32_t polling_threshold_value;
> +    uint32_t polling_threshold_window;
> +    uint32_t error_threshold_value;
> +    uint32_t error_threshold_window;
> +} QEMU_PACKED;
> +typedef struct AcpiHestNotify AcpiHestNotify;
> +
> +/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
> + * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
> + */
> +enum AcpiHestSourceType {
> +    ACPI_HEST_SOURCE_IA32_CHECK = 0,
> +    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
> +    ACPI_HEST_SOURCE_IA32_NMI = 2,
What's 3, 4, 5 for?

> +    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
> +    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
> +    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
> +    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
> +};
> +
> +/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
> +#define ACPI_GEBS_UNCORRECTABLE             (1)
> +#define ACPI_GEBS_CORRECTABLE               (1 << 1)
> +#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
> +#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
> +/* 10 bits, error data entry count */
> +#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
> +
> +/* Generic Hardware Error Source Structure, refer to ACPI 6.1
> + * "18.3.2.7 Generic Hardware Error Source". in this struct the
> + * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
> + */
> +
> +struct AcpiGenericHardwareErrorSource {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
> +
> +/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
> + * Hardware Error Source version 2", in this struct the "type" field has to
> + * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
> + */
> +struct AcpiGenericHardwareErrorSourceV2 {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length;
> +    struct AcpiGenericAddress read_ack_register;
> +    uint64_t read_ack_preserve;
> +    uint64_t read_ack_write;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericHardwareErrorSourceV2
> +            AcpiGenericHardwareErrorSourceV2;
> +
> +/* Generic Error Status block, from ACPI 6.1,
> + * "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorStatus {
> +    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
> +    uint32_t block_status;
> +    uint32_t raw_data_offset;
> +    uint32_t raw_data_length;
> +    uint32_t data_length;
> +    uint32_t error_severity;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
> +
> +enum AcpiGenericErrorSeverity {
> +    ACPI_CPER_SEV_RECOVERABLE,
> +    ACPI_CPER_SEV_FATAL,
> +    ACPI_CPER_SEV_CORRECTED,
> +    ACPI_CPER_SEV_NONE,
> +};
> +
> +/* Generic Error Data entry, revision number is 0x0300,
> + * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorData {
> +    uint8_t section_type_le[16];
> +    /* The "error_severity" fields that they take their
> +     * values from AcpiGenericErrorSeverity
> +     */
> +    uint32_t error_severity;
> +    uint16_t revision;
> +    uint8_t validation_bits;
> +    uint8_t flags;
> +    uint32_t error_data_length;
> +    uint8_t fru_id[16];
> +    uint8_t fru_text[20];
> +    uint64_t time_stamp;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorData AcpiGenericErrorData;
> +
> +/* From UEFI 2.6, "N.2.5 Memory Error Section" */
> +struct UefiCperSecMemErr {
> +    uint64_t    validation_bits;
> +    uint64_t    error_status;
> +    uint64_t    physical_addr;
> +    uint64_t    physical_addr_mask;
> +    uint16_t    node;
> +    uint16_t    card;
> +    uint16_t    module;
> +    uint16_t    bank;
> +    uint16_t    device;
> +    uint16_t    row;
> +    uint16_t    column;
> +    uint16_t    bit_pos;
> +    uint64_t    requestor_id;
> +    uint64_t    responder_id;
> +    uint64_t    target_id;
> +    uint8_t     error_type;
> +    uint8_t     reserved;
> +    uint16_t    rank;
> +    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
> +    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
> +} QEMU_PACKED;
> +typedef struct UefiCperSecMemErr UefiCperSecMemErr;
> +
> +/*
> + * HEST Description Table
> + */
> +struct AcpiHardwareErrorSourceTable {
> +    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
> +    uint32_t           error_source_count;
> +} QEMU_PACKED;
> +typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
> +
>  #define ACPI_SRAT_PROCESSOR_APIC     0
>  #define ACPI_SRAT_MEMORY             1
>  #define ACPI_SRAT_PROCESSOR_x2APIC   2
> 

-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-24 12:33     ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-24 12:33 UTC (permalink / raw)
  To: linux-arm-kernel



On 2017/8/18 22:23, Dongjiu Geng wrote:
> (1) Add related APEI/HEST table structures and  macros, these
>     definition refer to ACPI 6.1 and UEFI 2.6 spec.
> (2) Add generic error status block and CPER memory section
>     definition, user space only handle memory section errors.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 193 insertions(+)
> 
> diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
> index 72be675..3b4bad7 100644
> --- a/include/hw/acpi/acpi-defs.h
> +++ b/include/hw/acpi/acpi-defs.h
> @@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
>  #define ACPI_APIC_GENERIC_TRANSLATOR    15
>  #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
>  
> +/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
missing "

> +#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
> +#define UEFI_CPER_MEM_VALID_PA               0x0002
> +#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
> +#define UEFI_CPER_MEM_VALID_NODE             0x0008
> +#define UEFI_CPER_MEM_VALID_CARD             0x0010
> +#define UEFI_CPER_MEM_VALID_MODULE           0x0020
> +#define UEFI_CPER_MEM_VALID_BANK             0x0040
> +#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
> +#define UEFI_CPER_MEM_VALID_ROW              0x0100
> +#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
> +#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
> +#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
> +#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
> +#define UEFI_CPER_MEM_VALID_TARGET           0x2000
> +#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
> +#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
> +#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
> +#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
> +#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
> +
> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
> +
It's better to refer to the first spec version of this structure and
same with others you define.

> +enum AcpiHestNotifyType {
> +    ACPI_HEST_NOTIFY_POLLED = 0,
> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
> +    ACPI_HEST_NOTIFY_LOCAL = 2,
> +    ACPI_HEST_NOTIFY_SCI = 3,
> +    ACPI_HEST_NOTIFY_NMI = 4,
> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
your patchset?

> +};
> +
>  /*
>   * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
>   */
> @@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
>  } QEMU_PACKED;
>  typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
>  
> +/* Hardware Error Notification, from the ACPI 6.1
> + * spec, "18.3.2.9 Hardware Error Notification"
> + */
Use below style for multiple comment lines
/*
 * XXX
 */

> +struct AcpiHestNotify {
> +    uint8_t type;
> +    uint8_t length;
> +    uint16_t config_write_enable;
> +    uint32_t poll_interval;
> +    uint32_t vector;
> +    uint32_t polling_threshold_value;
> +    uint32_t polling_threshold_window;
> +    uint32_t error_threshold_value;
> +    uint32_t error_threshold_window;
> +} QEMU_PACKED;
> +typedef struct AcpiHestNotify AcpiHestNotify;
> +
> +/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
> + * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
> + */
> +enum AcpiHestSourceType {
> +    ACPI_HEST_SOURCE_IA32_CHECK = 0,
> +    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
> +    ACPI_HEST_SOURCE_IA32_NMI = 2,
What's 3, 4, 5 for?

> +    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
> +    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
> +    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
> +    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
> +    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
> +};
> +
> +/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
> +#define ACPI_GEBS_UNCORRECTABLE             (1)
> +#define ACPI_GEBS_CORRECTABLE               (1 << 1)
> +#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
> +#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
> +/* 10 bits, error data entry count */
> +#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
> +
> +/* Generic Hardware Error Source Structure, refer to ACPI 6.1
> + * "18.3.2.7 Generic Hardware Error Source". in this struct the
> + * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
> + */
> +
> +struct AcpiGenericHardwareErrorSource {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
> +
> +/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
> + * Hardware Error Source version 2", in this struct the "type" field has to
> + * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
> + */
> +struct AcpiGenericHardwareErrorSourceV2 {
> +    uint16_t type;
> +    uint16_t source_id;
> +    uint16_t related_source_id;
> +    uint8_t flags;
> +    uint8_t enabled;
> +    uint32_t number_of_records;
> +    uint32_t max_sections_per_record;
> +    uint32_t max_raw_data_length;
> +    struct AcpiGenericAddress error_status_address;
> +    struct AcpiHestNotify notify;
> +    uint32_t error_status_block_length;
> +    struct AcpiGenericAddress read_ack_register;
> +    uint64_t read_ack_preserve;
> +    uint64_t read_ack_write;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericHardwareErrorSourceV2
> +            AcpiGenericHardwareErrorSourceV2;
> +
> +/* Generic Error Status block, from ACPI 6.1,
> + * "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorStatus {
> +    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
> +    uint32_t block_status;
> +    uint32_t raw_data_offset;
> +    uint32_t raw_data_length;
> +    uint32_t data_length;
> +    uint32_t error_severity;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
> +
> +enum AcpiGenericErrorSeverity {
> +    ACPI_CPER_SEV_RECOVERABLE,
> +    ACPI_CPER_SEV_FATAL,
> +    ACPI_CPER_SEV_CORRECTED,
> +    ACPI_CPER_SEV_NONE,
> +};
> +
> +/* Generic Error Data entry, revision number is 0x0300,
> + * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
> + */
> +struct AcpiGenericErrorData {
> +    uint8_t section_type_le[16];
> +    /* The "error_severity" fields that they take their
> +     * values from AcpiGenericErrorSeverity
> +     */
> +    uint32_t error_severity;
> +    uint16_t revision;
> +    uint8_t validation_bits;
> +    uint8_t flags;
> +    uint32_t error_data_length;
> +    uint8_t fru_id[16];
> +    uint8_t fru_text[20];
> +    uint64_t time_stamp;
> +} QEMU_PACKED;
> +typedef struct AcpiGenericErrorData AcpiGenericErrorData;
> +
> +/* From UEFI 2.6, "N.2.5 Memory Error Section" */
> +struct UefiCperSecMemErr {
> +    uint64_t    validation_bits;
> +    uint64_t    error_status;
> +    uint64_t    physical_addr;
> +    uint64_t    physical_addr_mask;
> +    uint16_t    node;
> +    uint16_t    card;
> +    uint16_t    module;
> +    uint16_t    bank;
> +    uint16_t    device;
> +    uint16_t    row;
> +    uint16_t    column;
> +    uint16_t    bit_pos;
> +    uint64_t    requestor_id;
> +    uint64_t    responder_id;
> +    uint64_t    target_id;
> +    uint8_t     error_type;
> +    uint8_t     reserved;
> +    uint16_t    rank;
> +    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
> +    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
> +} QEMU_PACKED;
> +typedef struct UefiCperSecMemErr UefiCperSecMemErr;
> +
> +/*
> + * HEST Description Table
> + */
> +struct AcpiHardwareErrorSourceTable {
> +    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
> +    uint32_t           error_source_count;
> +} QEMU_PACKED;
> +typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
> +
>  #define ACPI_SRAT_PROCESSOR_APIC     0
>  #define ACPI_SRAT_MEMORY             1
>  #define ACPI_SRAT_PROCESSOR_x2APIC   2
> 

-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
  2017-08-18 14:23   ` Dongjiu Geng
  (?)
  (?)
@ 2017-08-24 13:03     ` Shannon Zhao
  -1 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-24 13:03 UTC (permalink / raw)
  To: Dongjiu Geng, mst, imammedo, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/18 22:23, Dongjiu Geng wrote:
> This implements APEI GHES Table by passing the error CPER info
> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
> will be injected into the guest OS.
> 
> Below is the table layout, the max number of error soure is 11,
> which is classified by notification type.
> 
>      etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
> 
> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
> so user space must check the ack value to avoid read-write race condition.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  hw/acpi/aml-build.c         |   2 +
>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c    |   6 +
>  include/hw/acpi/aml-build.h |   1 +
>  include/hw/acpi/hest_ghes.h |  47 ++++++
>  5 files changed, 401 insertions(+)
>  create mode 100644 hw/acpi/hest_ghes.c
Don't need to add the new file to hw/acpi/Makefile.objs?

>  create mode 100644 include/hw/acpi/hest_ghes.h
> 
> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
> index 36a6cc4..6849e5f 100644
> --- a/hw/acpi/aml-build.c
> +++ b/hw/acpi/aml-build.c
> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>      tables->table_data = g_array_new(false, true /* clear */, 1);
>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>      tables->linker = bios_linker_loader_init();
>  }
>  
> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>      g_array_free(tables->table_data, true);
>      g_array_free(tables->tcpalog, mfre);
>      g_array_free(tables->vmgenid, mfre);
> +    g_array_free(tables->hardware_errors, mfre);
>  }
>  
>  /* Build rsdt table */
> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
> new file mode 100644
> index 0000000..ff6b5ef
> --- /dev/null
> +++ b/hw/acpi/hest_ghes.c
> @@ -0,0 +1,345 @@
> +/*
> + *  APEI GHES table Generation
> + *
> + *  Copyright (C) 2017 huawei.
> + *
> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
Please unify this of this file and hest_ghes.h by refering to other files.

> +
> +#include "qemu/osdep.h"
> +#include "qmp-commands.h"
unnecessary including

> +#include "hw/acpi/acpi.h"
> +#include "hw/acpi/aml-build.h"
> +#include "hw/acpi/hest_ghes.h"
> +#include "hw/nvram/fw_cfg.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/error-report.h"
> +
> +/* The structure that stands for the layout
> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
> + *
> + *           etc/hardware_errors
> + * ==========================================
> + * +------------------+
> + * |    address       |              +--------------+
> + * |    registers     |              | Error Status |
> + * | +----------------+              | Data Block 0 |
> + * | |status_address0 |------------->| +------------+
> + * | +----------------+              | |  CPER      |
> + * | |status_address1 |----------+   | |  CPER      |
> + * | +----------------+          |   | |  ....      |
> + * | |.............   |          |   | |  CPER      |
> + * | +----------------+          |   | +------------+
> + * | |status_address10|-----+    |   | Error Status |
> + * | +----------------+     |    |   | Data Block 1 |
> + * | |ack_value0      |     |    +-->| +------------+
> + * | +----------------+     |        | |  CPER      |
> + * | |ack_value1      |     |        | |  CPER      |
> + * | +----------------+     |        | |  ....      |
> + * | | .............  |     |        | |  CPER      |
> + * | +----------------+     |        +-+------------+
> + * | |ack_value10     |     |        | |..........  |
> + * | +----------------+     |        | +------------+
> + *                          |        | Error Status |
> + *                          |        | Data Block10 |
> + *                          +------->+------------+
> + *                                   | |  CPER      |
> + *                                   | |  CPER      |
> + *                                   | |  ....      |
> + *                                   | |  CPER      |
> + *                                   +-+------------+
> + */
> +struct hardware_errors_buffer {
> +    /* Generic Error Status Block register */
> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
> +};
> +
> +static int ghes_record_cper(uint64_t error_block_address,
> +                                    uint64_t error_physical_addr)
> +{
> +    AcpiGenericErrorStatus block;
> +    AcpiGenericErrorData *gdata;
> +    UefiCperSecMemErr *mem_err;
> +    uint64_t current_block_length;
> +    unsigned char *buffer;
> +    /* memory section */
> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
> +                              0x83, 0xB1};
> +
> +    cpu_physical_memory_read(error_block_address, &block,
> +                                sizeof(AcpiGenericErrorStatus));
> +
> +    /* Get the current generic error status block length */
> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
> +        le32_to_cpu(block.data_length);
> +
> +    /* If the Generic Error Status Block is NULL, update
> +     * the block header
> +     */
> +    if (!block.block_status) {
> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    }
> +
> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    /* check whether it runs out of the preallocated memory */
> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
> +       GHES_MAX_RAW_DATA_LENGTH) {
> +        error_report("Record CPER out of boundary!!!");
> +        return GHES_CPER_FAIL;
> +    }
> +
> +    /* Write back the Generic Error Status Block to guest memory */
> +    cpu_physical_memory_write(error_block_address, &block,
> +        sizeof(AcpiGenericErrorStatus));
> +
> +    /* Fill in Generic Error Data Entry */
> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
> +                       sizeof(UefiCperSecMemErr));
> +
> +
> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +    gdata = (AcpiGenericErrorData *)buffer;
> +
> +    /* Memory section */
> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
> +            sizeof(mem_section_id_le));
> +
> +    /* error severity is recoverable */
> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    gdata->revision = 0x300; /* the revision number is 0x300 */
> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
> +
> +    /* User space only handle the memory section CPER */
> +
> +    /* Hard code to Multi-bit ECC error */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
> +
> +    /* Record the physical address at which the memory error occurred */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> +
> +    /* Write back the Generic Error Data Entry to guest memory */
> +    cpu_physical_memory_write(error_block_address + current_block_length,
> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +
> +    g_free(buffer);
> +    return GHES_CPER_OK;
> +}
> +
> +static void
> +build_address(GArray *table_data, BIOSLinker *linker,
> +    uint32_t dst_patched_offset, uint32_t src_offset,
> +    uint8_t address_space_id , uint8_t  register_bit_width,
> +    uint8_t register_bit_offset, uint8_t access_size)
> +{
> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    /* Address space */
> +    build_append_int_noprefix(table_data, address_space_id, 1);
> +    /* register bit width */
> +    build_append_int_noprefix(table_data, register_bit_width, 1);
> +    /* register bit offset */
> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
> +    /* access size */
> +    build_append_int_noprefix(table_data, access_size, 1);
> +    acpi_data_push(table_data, address_size);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
> +     * can retrieve and read it. the address size is 64 bits.
> +     */
> +    bios_linker_loader_add_pointer(linker,
> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
> +}
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                                            BIOSLinker *linker)
> +{
> +    uint32_t ghes_start = table_data->len;
> +    uint32_t address_size, error_status_address_offset;
> +    uint32_t read_ack_register_offset, i;
> +
> +    address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    error_status_address_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    read_ack_register_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    acpi_data_push(hardware_error,
> +        offsetof(struct hardware_errors_buffer, ack_value));
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Initialize read ack register */
> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
> +
> +    /* Reserved the total size for ERRORS fw_cfg blob
> +     */
> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
> +
> +    /* Allocate guest memory for the Data fw_cfg blob */
> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
> +                            1, false);
> +    /* Reserve table header size */
> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
> +
> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
> +        build_append_int_noprefix(table_data,
> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
> +        /* source id */
> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
> +        /* related source id */
> +        build_append_int_noprefix(table_data, 0xffff, 2);
> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
> +
> +        /* Currently only enable SEA notification type to avoid the kernel
> +         * warning, reserve the space for other notification error source
> +         */
> +        if (i == ACPI_HEST_NOTIFY_SEA) {
> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
> +        } else {
> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
> +        }
> +
> +        /* The number of error status block per generic hardware error source */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max sections per record */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max raw data length */
> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
> +
> +        /* Build error status address*/
> +        build_address(table_data, linker, error_status_address_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> +
> +        /* Hardware error notification structure */
> +        build_append_int_noprefix(table_data, i, 1); /* type */
> +        /* length */
> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
> +        build_append_int_noprefix(table_data, 0, 26);
> +
> +        /* Error Status Block Length */
> +        build_append_int_noprefix(table_data,
> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
> +
> +        /* Build read ack register */
> +        build_address(table_data, linker, read_ack_register_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2),
> +            offsetof(struct hardware_errors_buffer, ack_value) +
> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
> +            4 /* QWord access */);
> +
> +        /* Read ack preserve */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
> +
> +        /* Read ack write */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
> +    }
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Patch address of generic error status block into
> +         * the address register so OSPM can retrieve and read it.
> +         */
> +        bios_linker_loader_add_pointer(linker,
> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
> +            GHES_ERRORS_FW_CFG_FILE,
> +            offsetof(struct hardware_errors_buffer, gesb) +
> +            i * GHES_MAX_RAW_DATA_LENGTH);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
> +     * so QEMU can write the ERRORS there. The address is expected to be
> +     * < 4GB, but write 64 bits anyway.
> +     */
> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
> +        offsetof(struct hardware_errors_buffer, gesb));
> +
> +    build_header(linker, table_data,
> +        (void *)(table_data->data + ghes_start), "HEST",
> +        table_data->len - ghes_start, 1, NULL, "GHES");
> +}
> +
> +static GhesState ges;
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
> +{
> +
> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
> +
> +    /* Create a read-only fw_cfg file for GHES */
> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
> +                    size);
> +    /* Create a read-write fw_cfg file for Address */
> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
> +}
> +
> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
> +{
> +    uint64_t error_block_addr;
> +    uint64_t ack_value_addr, ack_value = 0;
> +    int loop = 0, ack_value_size;
> +    bool ret = GHES_CPER_FAIL;
> +
> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
> +        offsetof(struct hardware_errors_buffer, ack_value)) /
> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
> +
> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
> +        error_block_addr = le32_to_cpu(error_block_addr);
> +
> +        ack_value_addr = ges.ghes_addr_le -
> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
> +retry:
> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
> +        if (!ack_value) {
> +            if (loop < 3) {
> +                usleep(100 * 1000);
> +                loop++;
> +                goto retry;
> +            } else {
> +                error_report("Last time OSPM does not acknowledge the error,"
> +                    " record CPER failed this time, set the ack value to"
> +                    " avoid blocking next time CPER record! exit");
> +                ack_value = 1;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                return ret;
> +            }
> +        } else {
> +            /* A zero value in ghes_addr means that BIOS has not yet written
> +             * the address
> +             */
> +            if (error_block_addr) {
> +                ack_value = 0;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                ret = ghes_record_cper(error_block_addr, physical_address);
> +            }
> +        }
> +    }
> +    return ret;
> +}
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> index 3d78ff6..def1ec1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -45,6 +45,7 @@
>  #include "hw/arm/virt.h"
>  #include "sysemu/numa.h"
>  #include "kvm_arm.h"
> +#include "hw/acpi/hest_ghes.h"
>  
>  #define ARM_SPI_BASE 32
>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>      acpi_add_table(table_offsets, tables_blob);
>      build_spcr(tables_blob, tables->linker, vms);
>  
> +    acpi_add_table(table_offsets, tables_blob);
> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
> +
So we add this table unconditionally. Is there any bad impact if QEMU
runs on old kvm? Does it need to check whether KVM supports RAS?

>      if (nb_numa_nodes > 0) {
>          acpi_add_table(table_offsets, tables_blob);
>          build_srat(tables_blob, tables->linker, vms);
> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>                      acpi_data_len(tables.tcpalog));
>  
> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
> +
>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>                                                ACPI_BUILD_RSDP_FILE, 0);
>  
> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
> index 88d0738..7f7b55c 100644
> --- a/include/hw/acpi/aml-build.h
> +++ b/include/hw/acpi/aml-build.h
> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>      GArray *rsdp;
>      GArray *tcpalog;
>      GArray *vmgenid;
> +    GArray *hardware_errors;
>      BIOSLinker *linker;
>  } AcpiBuildTables;
>  
> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
> new file mode 100644
> index 0000000..0772756
> --- /dev/null
> +++ b/include/hw/acpi/hest_ghes.h
> @@ -0,0 +1,47 @@
> +/*
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> +
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * Authors:
> + *   Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef ACPI_GHES_H
> +#define ACPI_GHES_H
> +
> +#include "hw/acpi/bios-linker-loader.h"
> +
> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
> +
> +#define GHES_GAS_ADDRESS_OFFSET              4
> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
> +#define GHES_NOTIFICATION_STRUCTURE          32
> +
> +#define GHES_CPER_OK   1
> +#define GHES_CPER_FAIL 0
> +
> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
> +/* The max size in Bytes for one error block */
> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
> +
> +
> +typedef struct GhesState {
> +    uint64_t ghes_addr_le;
> +} GhesState;
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                            BIOSLinker *linker);
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
> +#endif
> 

-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-24 13:03     ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-24 13:03 UTC (permalink / raw)
  To: Dongjiu Geng, mst, imammedo, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/18 22:23, Dongjiu Geng wrote:
> This implements APEI GHES Table by passing the error CPER info
> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
> will be injected into the guest OS.
> 
> Below is the table layout, the max number of error soure is 11,
> which is classified by notification type.
> 
>      etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
> 
> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
> so user space must check the ack value to avoid read-write race condition.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  hw/acpi/aml-build.c         |   2 +
>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c    |   6 +
>  include/hw/acpi/aml-build.h |   1 +
>  include/hw/acpi/hest_ghes.h |  47 ++++++
>  5 files changed, 401 insertions(+)
>  create mode 100644 hw/acpi/hest_ghes.c
Don't need to add the new file to hw/acpi/Makefile.objs?

>  create mode 100644 include/hw/acpi/hest_ghes.h
> 
> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
> index 36a6cc4..6849e5f 100644
> --- a/hw/acpi/aml-build.c
> +++ b/hw/acpi/aml-build.c
> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>      tables->table_data = g_array_new(false, true /* clear */, 1);
>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>      tables->linker = bios_linker_loader_init();
>  }
>  
> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>      g_array_free(tables->table_data, true);
>      g_array_free(tables->tcpalog, mfre);
>      g_array_free(tables->vmgenid, mfre);
> +    g_array_free(tables->hardware_errors, mfre);
>  }
>  
>  /* Build rsdt table */
> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
> new file mode 100644
> index 0000000..ff6b5ef
> --- /dev/null
> +++ b/hw/acpi/hest_ghes.c
> @@ -0,0 +1,345 @@
> +/*
> + *  APEI GHES table Generation
> + *
> + *  Copyright (C) 2017 huawei.
> + *
> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
Please unify this of this file and hest_ghes.h by refering to other files.

> +
> +#include "qemu/osdep.h"
> +#include "qmp-commands.h"
unnecessary including

> +#include "hw/acpi/acpi.h"
> +#include "hw/acpi/aml-build.h"
> +#include "hw/acpi/hest_ghes.h"
> +#include "hw/nvram/fw_cfg.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/error-report.h"
> +
> +/* The structure that stands for the layout
> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
> + *
> + *           etc/hardware_errors
> + * ==========================================
> + * +------------------+
> + * |    address       |              +--------------+
> + * |    registers     |              | Error Status |
> + * | +----------------+              | Data Block 0 |
> + * | |status_address0 |------------->| +------------+
> + * | +----------------+              | |  CPER      |
> + * | |status_address1 |----------+   | |  CPER      |
> + * | +----------------+          |   | |  ....      |
> + * | |.............   |          |   | |  CPER      |
> + * | +----------------+          |   | +------------+
> + * | |status_address10|-----+    |   | Error Status |
> + * | +----------------+     |    |   | Data Block 1 |
> + * | |ack_value0      |     |    +-->| +------------+
> + * | +----------------+     |        | |  CPER      |
> + * | |ack_value1      |     |        | |  CPER      |
> + * | +----------------+     |        | |  ....      |
> + * | | .............  |     |        | |  CPER      |
> + * | +----------------+     |        +-+------------+
> + * | |ack_value10     |     |        | |..........  |
> + * | +----------------+     |        | +------------+
> + *                          |        | Error Status |
> + *                          |        | Data Block10 |
> + *                          +------->+------------+
> + *                                   | |  CPER      |
> + *                                   | |  CPER      |
> + *                                   | |  ....      |
> + *                                   | |  CPER      |
> + *                                   +-+------------+
> + */
> +struct hardware_errors_buffer {
> +    /* Generic Error Status Block register */
> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
> +};
> +
> +static int ghes_record_cper(uint64_t error_block_address,
> +                                    uint64_t error_physical_addr)
> +{
> +    AcpiGenericErrorStatus block;
> +    AcpiGenericErrorData *gdata;
> +    UefiCperSecMemErr *mem_err;
> +    uint64_t current_block_length;
> +    unsigned char *buffer;
> +    /* memory section */
> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
> +                              0x83, 0xB1};
> +
> +    cpu_physical_memory_read(error_block_address, &block,
> +                                sizeof(AcpiGenericErrorStatus));
> +
> +    /* Get the current generic error status block length */
> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
> +        le32_to_cpu(block.data_length);
> +
> +    /* If the Generic Error Status Block is NULL, update
> +     * the block header
> +     */
> +    if (!block.block_status) {
> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    }
> +
> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    /* check whether it runs out of the preallocated memory */
> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
> +       GHES_MAX_RAW_DATA_LENGTH) {
> +        error_report("Record CPER out of boundary!!!");
> +        return GHES_CPER_FAIL;
> +    }
> +
> +    /* Write back the Generic Error Status Block to guest memory */
> +    cpu_physical_memory_write(error_block_address, &block,
> +        sizeof(AcpiGenericErrorStatus));
> +
> +    /* Fill in Generic Error Data Entry */
> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
> +                       sizeof(UefiCperSecMemErr));
> +
> +
> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +    gdata = (AcpiGenericErrorData *)buffer;
> +
> +    /* Memory section */
> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
> +            sizeof(mem_section_id_le));
> +
> +    /* error severity is recoverable */
> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    gdata->revision = 0x300; /* the revision number is 0x300 */
> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
> +
> +    /* User space only handle the memory section CPER */
> +
> +    /* Hard code to Multi-bit ECC error */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
> +
> +    /* Record the physical address at which the memory error occurred */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> +
> +    /* Write back the Generic Error Data Entry to guest memory */
> +    cpu_physical_memory_write(error_block_address + current_block_length,
> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +
> +    g_free(buffer);
> +    return GHES_CPER_OK;
> +}
> +
> +static void
> +build_address(GArray *table_data, BIOSLinker *linker,
> +    uint32_t dst_patched_offset, uint32_t src_offset,
> +    uint8_t address_space_id , uint8_t  register_bit_width,
> +    uint8_t register_bit_offset, uint8_t access_size)
> +{
> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    /* Address space */
> +    build_append_int_noprefix(table_data, address_space_id, 1);
> +    /* register bit width */
> +    build_append_int_noprefix(table_data, register_bit_width, 1);
> +    /* register bit offset */
> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
> +    /* access size */
> +    build_append_int_noprefix(table_data, access_size, 1);
> +    acpi_data_push(table_data, address_size);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
> +     * can retrieve and read it. the address size is 64 bits.
> +     */
> +    bios_linker_loader_add_pointer(linker,
> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
> +}
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                                            BIOSLinker *linker)
> +{
> +    uint32_t ghes_start = table_data->len;
> +    uint32_t address_size, error_status_address_offset;
> +    uint32_t read_ack_register_offset, i;
> +
> +    address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    error_status_address_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    read_ack_register_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    acpi_data_push(hardware_error,
> +        offsetof(struct hardware_errors_buffer, ack_value));
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Initialize read ack register */
> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
> +
> +    /* Reserved the total size for ERRORS fw_cfg blob
> +     */
> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
> +
> +    /* Allocate guest memory for the Data fw_cfg blob */
> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
> +                            1, false);
> +    /* Reserve table header size */
> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
> +
> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
> +        build_append_int_noprefix(table_data,
> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
> +        /* source id */
> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
> +        /* related source id */
> +        build_append_int_noprefix(table_data, 0xffff, 2);
> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
> +
> +        /* Currently only enable SEA notification type to avoid the kernel
> +         * warning, reserve the space for other notification error source
> +         */
> +        if (i == ACPI_HEST_NOTIFY_SEA) {
> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
> +        } else {
> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
> +        }
> +
> +        /* The number of error status block per generic hardware error source */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max sections per record */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max raw data length */
> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
> +
> +        /* Build error status address*/
> +        build_address(table_data, linker, error_status_address_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> +
> +        /* Hardware error notification structure */
> +        build_append_int_noprefix(table_data, i, 1); /* type */
> +        /* length */
> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
> +        build_append_int_noprefix(table_data, 0, 26);
> +
> +        /* Error Status Block Length */
> +        build_append_int_noprefix(table_data,
> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
> +
> +        /* Build read ack register */
> +        build_address(table_data, linker, read_ack_register_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2),
> +            offsetof(struct hardware_errors_buffer, ack_value) +
> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
> +            4 /* QWord access */);
> +
> +        /* Read ack preserve */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
> +
> +        /* Read ack write */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
> +    }
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Patch address of generic error status block into
> +         * the address register so OSPM can retrieve and read it.
> +         */
> +        bios_linker_loader_add_pointer(linker,
> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
> +            GHES_ERRORS_FW_CFG_FILE,
> +            offsetof(struct hardware_errors_buffer, gesb) +
> +            i * GHES_MAX_RAW_DATA_LENGTH);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
> +     * so QEMU can write the ERRORS there. The address is expected to be
> +     * < 4GB, but write 64 bits anyway.
> +     */
> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
> +        offsetof(struct hardware_errors_buffer, gesb));
> +
> +    build_header(linker, table_data,
> +        (void *)(table_data->data + ghes_start), "HEST",
> +        table_data->len - ghes_start, 1, NULL, "GHES");
> +}
> +
> +static GhesState ges;
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
> +{
> +
> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
> +
> +    /* Create a read-only fw_cfg file for GHES */
> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
> +                    size);
> +    /* Create a read-write fw_cfg file for Address */
> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
> +}
> +
> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
> +{
> +    uint64_t error_block_addr;
> +    uint64_t ack_value_addr, ack_value = 0;
> +    int loop = 0, ack_value_size;
> +    bool ret = GHES_CPER_FAIL;
> +
> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
> +        offsetof(struct hardware_errors_buffer, ack_value)) /
> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
> +
> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
> +        error_block_addr = le32_to_cpu(error_block_addr);
> +
> +        ack_value_addr = ges.ghes_addr_le -
> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
> +retry:
> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
> +        if (!ack_value) {
> +            if (loop < 3) {
> +                usleep(100 * 1000);
> +                loop++;
> +                goto retry;
> +            } else {
> +                error_report("Last time OSPM does not acknowledge the error,"
> +                    " record CPER failed this time, set the ack value to"
> +                    " avoid blocking next time CPER record! exit");
> +                ack_value = 1;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                return ret;
> +            }
> +        } else {
> +            /* A zero value in ghes_addr means that BIOS has not yet written
> +             * the address
> +             */
> +            if (error_block_addr) {
> +                ack_value = 0;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                ret = ghes_record_cper(error_block_addr, physical_address);
> +            }
> +        }
> +    }
> +    return ret;
> +}
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> index 3d78ff6..def1ec1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -45,6 +45,7 @@
>  #include "hw/arm/virt.h"
>  #include "sysemu/numa.h"
>  #include "kvm_arm.h"
> +#include "hw/acpi/hest_ghes.h"
>  
>  #define ARM_SPI_BASE 32
>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>      acpi_add_table(table_offsets, tables_blob);
>      build_spcr(tables_blob, tables->linker, vms);
>  
> +    acpi_add_table(table_offsets, tables_blob);
> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
> +
So we add this table unconditionally. Is there any bad impact if QEMU
runs on old kvm? Does it need to check whether KVM supports RAS?

>      if (nb_numa_nodes > 0) {
>          acpi_add_table(table_offsets, tables_blob);
>          build_srat(tables_blob, tables->linker, vms);
> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>                      acpi_data_len(tables.tcpalog));
>  
> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
> +
>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>                                                ACPI_BUILD_RSDP_FILE, 0);
>  
> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
> index 88d0738..7f7b55c 100644
> --- a/include/hw/acpi/aml-build.h
> +++ b/include/hw/acpi/aml-build.h
> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>      GArray *rsdp;
>      GArray *tcpalog;
>      GArray *vmgenid;
> +    GArray *hardware_errors;
>      BIOSLinker *linker;
>  } AcpiBuildTables;
>  
> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
> new file mode 100644
> index 0000000..0772756
> --- /dev/null
> +++ b/include/hw/acpi/hest_ghes.h
> @@ -0,0 +1,47 @@
> +/*
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> +
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * Authors:
> + *   Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef ACPI_GHES_H
> +#define ACPI_GHES_H
> +
> +#include "hw/acpi/bios-linker-loader.h"
> +
> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
> +
> +#define GHES_GAS_ADDRESS_OFFSET              4
> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
> +#define GHES_NOTIFICATION_STRUCTURE          32
> +
> +#define GHES_CPER_OK   1
> +#define GHES_CPER_FAIL 0
> +
> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
> +/* The max size in Bytes for one error block */
> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
> +
> +
> +typedef struct GhesState {
> +    uint64_t ghes_addr_le;
> +} GhesState;
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                            BIOSLinker *linker);
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
> +#endif
> 

-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-24 13:03     ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-24 13:03 UTC (permalink / raw)
  To: Dongjiu Geng, mst, imammedo, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/18 22:23, Dongjiu Geng wrote:
> This implements APEI GHES Table by passing the error CPER info
> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
> will be injected into the guest OS.
> 
> Below is the table layout, the max number of error soure is 11,
> which is classified by notification type.
> 
>      etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
> 
> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
> so user space must check the ack value to avoid read-write race condition.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  hw/acpi/aml-build.c         |   2 +
>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c    |   6 +
>  include/hw/acpi/aml-build.h |   1 +
>  include/hw/acpi/hest_ghes.h |  47 ++++++
>  5 files changed, 401 insertions(+)
>  create mode 100644 hw/acpi/hest_ghes.c
Don't need to add the new file to hw/acpi/Makefile.objs?

>  create mode 100644 include/hw/acpi/hest_ghes.h
> 
> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
> index 36a6cc4..6849e5f 100644
> --- a/hw/acpi/aml-build.c
> +++ b/hw/acpi/aml-build.c
> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>      tables->table_data = g_array_new(false, true /* clear */, 1);
>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>      tables->linker = bios_linker_loader_init();
>  }
>  
> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>      g_array_free(tables->table_data, true);
>      g_array_free(tables->tcpalog, mfre);
>      g_array_free(tables->vmgenid, mfre);
> +    g_array_free(tables->hardware_errors, mfre);
>  }
>  
>  /* Build rsdt table */
> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
> new file mode 100644
> index 0000000..ff6b5ef
> --- /dev/null
> +++ b/hw/acpi/hest_ghes.c
> @@ -0,0 +1,345 @@
> +/*
> + *  APEI GHES table Generation
> + *
> + *  Copyright (C) 2017 huawei.
> + *
> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
Please unify this of this file and hest_ghes.h by refering to other files.

> +
> +#include "qemu/osdep.h"
> +#include "qmp-commands.h"
unnecessary including

> +#include "hw/acpi/acpi.h"
> +#include "hw/acpi/aml-build.h"
> +#include "hw/acpi/hest_ghes.h"
> +#include "hw/nvram/fw_cfg.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/error-report.h"
> +
> +/* The structure that stands for the layout
> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
> + *
> + *           etc/hardware_errors
> + * ==========================================
> + * +------------------+
> + * |    address       |              +--------------+
> + * |    registers     |              | Error Status |
> + * | +----------------+              | Data Block 0 |
> + * | |status_address0 |------------->| +------------+
> + * | +----------------+              | |  CPER      |
> + * | |status_address1 |----------+   | |  CPER      |
> + * | +----------------+          |   | |  ....      |
> + * | |.............   |          |   | |  CPER      |
> + * | +----------------+          |   | +------------+
> + * | |status_address10|-----+    |   | Error Status |
> + * | +----------------+     |    |   | Data Block 1 |
> + * | |ack_value0      |     |    +-->| +------------+
> + * | +----------------+     |        | |  CPER      |
> + * | |ack_value1      |     |        | |  CPER      |
> + * | +----------------+     |        | |  ....      |
> + * | | .............  |     |        | |  CPER      |
> + * | +----------------+     |        +-+------------+
> + * | |ack_value10     |     |        | |..........  |
> + * | +----------------+     |        | +------------+
> + *                          |        | Error Status |
> + *                          |        | Data Block10 |
> + *                          +------->+------------+
> + *                                   | |  CPER      |
> + *                                   | |  CPER      |
> + *                                   | |  ....      |
> + *                                   | |  CPER      |
> + *                                   +-+------------+
> + */
> +struct hardware_errors_buffer {
> +    /* Generic Error Status Block register */
> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
> +};
> +
> +static int ghes_record_cper(uint64_t error_block_address,
> +                                    uint64_t error_physical_addr)
> +{
> +    AcpiGenericErrorStatus block;
> +    AcpiGenericErrorData *gdata;
> +    UefiCperSecMemErr *mem_err;
> +    uint64_t current_block_length;
> +    unsigned char *buffer;
> +    /* memory section */
> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
> +                              0x83, 0xB1};
> +
> +    cpu_physical_memory_read(error_block_address, &block,
> +                                sizeof(AcpiGenericErrorStatus));
> +
> +    /* Get the current generic error status block length */
> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
> +        le32_to_cpu(block.data_length);
> +
> +    /* If the Generic Error Status Block is NULL, update
> +     * the block header
> +     */
> +    if (!block.block_status) {
> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    }
> +
> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    /* check whether it runs out of the preallocated memory */
> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
> +       GHES_MAX_RAW_DATA_LENGTH) {
> +        error_report("Record CPER out of boundary!!!");
> +        return GHES_CPER_FAIL;
> +    }
> +
> +    /* Write back the Generic Error Status Block to guest memory */
> +    cpu_physical_memory_write(error_block_address, &block,
> +        sizeof(AcpiGenericErrorStatus));
> +
> +    /* Fill in Generic Error Data Entry */
> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
> +                       sizeof(UefiCperSecMemErr));
> +
> +
> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +    gdata = (AcpiGenericErrorData *)buffer;
> +
> +    /* Memory section */
> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
> +            sizeof(mem_section_id_le));
> +
> +    /* error severity is recoverable */
> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    gdata->revision = 0x300; /* the revision number is 0x300 */
> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
> +
> +    /* User space only handle the memory section CPER */
> +
> +    /* Hard code to Multi-bit ECC error */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
> +
> +    /* Record the physical address at which the memory error occurred */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> +
> +    /* Write back the Generic Error Data Entry to guest memory */
> +    cpu_physical_memory_write(error_block_address + current_block_length,
> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +
> +    g_free(buffer);
> +    return GHES_CPER_OK;
> +}
> +
> +static void
> +build_address(GArray *table_data, BIOSLinker *linker,
> +    uint32_t dst_patched_offset, uint32_t src_offset,
> +    uint8_t address_space_id , uint8_t  register_bit_width,
> +    uint8_t register_bit_offset, uint8_t access_size)
> +{
> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    /* Address space */
> +    build_append_int_noprefix(table_data, address_space_id, 1);
> +    /* register bit width */
> +    build_append_int_noprefix(table_data, register_bit_width, 1);
> +    /* register bit offset */
> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
> +    /* access size */
> +    build_append_int_noprefix(table_data, access_size, 1);
> +    acpi_data_push(table_data, address_size);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
> +     * can retrieve and read it. the address size is 64 bits.
> +     */
> +    bios_linker_loader_add_pointer(linker,
> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
> +}
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                                            BIOSLinker *linker)
> +{
> +    uint32_t ghes_start = table_data->len;
> +    uint32_t address_size, error_status_address_offset;
> +    uint32_t read_ack_register_offset, i;
> +
> +    address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    error_status_address_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    read_ack_register_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    acpi_data_push(hardware_error,
> +        offsetof(struct hardware_errors_buffer, ack_value));
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Initialize read ack register */
> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
> +
> +    /* Reserved the total size for ERRORS fw_cfg blob
> +     */
> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
> +
> +    /* Allocate guest memory for the Data fw_cfg blob */
> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
> +                            1, false);
> +    /* Reserve table header size */
> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
> +
> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
> +        build_append_int_noprefix(table_data,
> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
> +        /* source id */
> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
> +        /* related source id */
> +        build_append_int_noprefix(table_data, 0xffff, 2);
> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
> +
> +        /* Currently only enable SEA notification type to avoid the kernel
> +         * warning, reserve the space for other notification error source
> +         */
> +        if (i == ACPI_HEST_NOTIFY_SEA) {
> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
> +        } else {
> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
> +        }
> +
> +        /* The number of error status block per generic hardware error source */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max sections per record */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max raw data length */
> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
> +
> +        /* Build error status address*/
> +        build_address(table_data, linker, error_status_address_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> +
> +        /* Hardware error notification structure */
> +        build_append_int_noprefix(table_data, i, 1); /* type */
> +        /* length */
> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
> +        build_append_int_noprefix(table_data, 0, 26);
> +
> +        /* Error Status Block Length */
> +        build_append_int_noprefix(table_data,
> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
> +
> +        /* Build read ack register */
> +        build_address(table_data, linker, read_ack_register_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2),
> +            offsetof(struct hardware_errors_buffer, ack_value) +
> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
> +            4 /* QWord access */);
> +
> +        /* Read ack preserve */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
> +
> +        /* Read ack write */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
> +    }
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Patch address of generic error status block into
> +         * the address register so OSPM can retrieve and read it.
> +         */
> +        bios_linker_loader_add_pointer(linker,
> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
> +            GHES_ERRORS_FW_CFG_FILE,
> +            offsetof(struct hardware_errors_buffer, gesb) +
> +            i * GHES_MAX_RAW_DATA_LENGTH);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
> +     * so QEMU can write the ERRORS there. The address is expected to be
> +     * < 4GB, but write 64 bits anyway.
> +     */
> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
> +        offsetof(struct hardware_errors_buffer, gesb));
> +
> +    build_header(linker, table_data,
> +        (void *)(table_data->data + ghes_start), "HEST",
> +        table_data->len - ghes_start, 1, NULL, "GHES");
> +}
> +
> +static GhesState ges;
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
> +{
> +
> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
> +
> +    /* Create a read-only fw_cfg file for GHES */
> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
> +                    size);
> +    /* Create a read-write fw_cfg file for Address */
> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
> +}
> +
> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
> +{
> +    uint64_t error_block_addr;
> +    uint64_t ack_value_addr, ack_value = 0;
> +    int loop = 0, ack_value_size;
> +    bool ret = GHES_CPER_FAIL;
> +
> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
> +        offsetof(struct hardware_errors_buffer, ack_value)) /
> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
> +
> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
> +        error_block_addr = le32_to_cpu(error_block_addr);
> +
> +        ack_value_addr = ges.ghes_addr_le -
> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
> +retry:
> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
> +        if (!ack_value) {
> +            if (loop < 3) {
> +                usleep(100 * 1000);
> +                loop++;
> +                goto retry;
> +            } else {
> +                error_report("Last time OSPM does not acknowledge the error,"
> +                    " record CPER failed this time, set the ack value to"
> +                    " avoid blocking next time CPER record! exit");
> +                ack_value = 1;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                return ret;
> +            }
> +        } else {
> +            /* A zero value in ghes_addr means that BIOS has not yet written
> +             * the address
> +             */
> +            if (error_block_addr) {
> +                ack_value = 0;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                ret = ghes_record_cper(error_block_addr, physical_address);
> +            }
> +        }
> +    }
> +    return ret;
> +}
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> index 3d78ff6..def1ec1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -45,6 +45,7 @@
>  #include "hw/arm/virt.h"
>  #include "sysemu/numa.h"
>  #include "kvm_arm.h"
> +#include "hw/acpi/hest_ghes.h"
>  
>  #define ARM_SPI_BASE 32
>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>      acpi_add_table(table_offsets, tables_blob);
>      build_spcr(tables_blob, tables->linker, vms);
>  
> +    acpi_add_table(table_offsets, tables_blob);
> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
> +
So we add this table unconditionally. Is there any bad impact if QEMU
runs on old kvm? Does it need to check whether KVM supports RAS?

>      if (nb_numa_nodes > 0) {
>          acpi_add_table(table_offsets, tables_blob);
>          build_srat(tables_blob, tables->linker, vms);
> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>                      acpi_data_len(tables.tcpalog));
>  
> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
> +
>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>                                                ACPI_BUILD_RSDP_FILE, 0);
>  
> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
> index 88d0738..7f7b55c 100644
> --- a/include/hw/acpi/aml-build.h
> +++ b/include/hw/acpi/aml-build.h
> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>      GArray *rsdp;
>      GArray *tcpalog;
>      GArray *vmgenid;
> +    GArray *hardware_errors;
>      BIOSLinker *linker;
>  } AcpiBuildTables;
>  
> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
> new file mode 100644
> index 0000000..0772756
> --- /dev/null
> +++ b/include/hw/acpi/hest_ghes.h
> @@ -0,0 +1,47 @@
> +/*
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> +
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * Authors:
> + *   Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef ACPI_GHES_H
> +#define ACPI_GHES_H
> +
> +#include "hw/acpi/bios-linker-loader.h"
> +
> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
> +
> +#define GHES_GAS_ADDRESS_OFFSET              4
> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
> +#define GHES_NOTIFICATION_STRUCTURE          32
> +
> +#define GHES_CPER_OK   1
> +#define GHES_CPER_FAIL 0
> +
> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
> +/* The max size in Bytes for one error block */
> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
> +
> +
> +typedef struct GhesState {
> +    uint64_t ghes_addr_le;
> +} GhesState;
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                            BIOSLinker *linker);
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
> +#endif
> 

-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-24 13:03     ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-24 13:03 UTC (permalink / raw)
  To: linux-arm-kernel



On 2017/8/18 22:23, Dongjiu Geng wrote:
> This implements APEI GHES Table by passing the error CPER info
> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
> will be injected into the guest OS.
> 
> Below is the table layout, the max number of error soure is 11,
> which is classified by notification type.
> 
>      etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
> 
> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
> so user space must check the ack value to avoid read-write race condition.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  hw/acpi/aml-build.c         |   2 +
>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c    |   6 +
>  include/hw/acpi/aml-build.h |   1 +
>  include/hw/acpi/hest_ghes.h |  47 ++++++
>  5 files changed, 401 insertions(+)
>  create mode 100644 hw/acpi/hest_ghes.c
Don't need to add the new file to hw/acpi/Makefile.objs?

>  create mode 100644 include/hw/acpi/hest_ghes.h
> 
> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
> index 36a6cc4..6849e5f 100644
> --- a/hw/acpi/aml-build.c
> +++ b/hw/acpi/aml-build.c
> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>      tables->table_data = g_array_new(false, true /* clear */, 1);
>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>      tables->linker = bios_linker_loader_init();
>  }
>  
> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>      g_array_free(tables->table_data, true);
>      g_array_free(tables->tcpalog, mfre);
>      g_array_free(tables->vmgenid, mfre);
> +    g_array_free(tables->hardware_errors, mfre);
>  }
>  
>  /* Build rsdt table */
> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
> new file mode 100644
> index 0000000..ff6b5ef
> --- /dev/null
> +++ b/hw/acpi/hest_ghes.c
> @@ -0,0 +1,345 @@
> +/*
> + *  APEI GHES table Generation
> + *
> + *  Copyright (C) 2017 huawei.
> + *
> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
Please unify this of this file and hest_ghes.h by refering to other files.

> +
> +#include "qemu/osdep.h"
> +#include "qmp-commands.h"
unnecessary including

> +#include "hw/acpi/acpi.h"
> +#include "hw/acpi/aml-build.h"
> +#include "hw/acpi/hest_ghes.h"
> +#include "hw/nvram/fw_cfg.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/error-report.h"
> +
> +/* The structure that stands for the layout
> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
> + *
> + *           etc/hardware_errors
> + * ==========================================
> + * +------------------+
> + * |    address       |              +--------------+
> + * |    registers     |              | Error Status |
> + * | +----------------+              | Data Block 0 |
> + * | |status_address0 |------------->| +------------+
> + * | +----------------+              | |  CPER      |
> + * | |status_address1 |----------+   | |  CPER      |
> + * | +----------------+          |   | |  ....      |
> + * | |.............   |          |   | |  CPER      |
> + * | +----------------+          |   | +------------+
> + * | |status_address10|-----+    |   | Error Status |
> + * | +----------------+     |    |   | Data Block 1 |
> + * | |ack_value0      |     |    +-->| +------------+
> + * | +----------------+     |        | |  CPER      |
> + * | |ack_value1      |     |        | |  CPER      |
> + * | +----------------+     |        | |  ....      |
> + * | | .............  |     |        | |  CPER      |
> + * | +----------------+     |        +-+------------+
> + * | |ack_value10     |     |        | |..........  |
> + * | +----------------+     |        | +------------+
> + *                          |        | Error Status |
> + *                          |        | Data Block10 |
> + *                          +------->+------------+
> + *                                   | |  CPER      |
> + *                                   | |  CPER      |
> + *                                   | |  ....      |
> + *                                   | |  CPER      |
> + *                                   +-+------------+
> + */
> +struct hardware_errors_buffer {
> +    /* Generic Error Status Block register */
> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
> +};
> +
> +static int ghes_record_cper(uint64_t error_block_address,
> +                                    uint64_t error_physical_addr)
> +{
> +    AcpiGenericErrorStatus block;
> +    AcpiGenericErrorData *gdata;
> +    UefiCperSecMemErr *mem_err;
> +    uint64_t current_block_length;
> +    unsigned char *buffer;
> +    /* memory section */
> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
> +                              0x83, 0xB1};
> +
> +    cpu_physical_memory_read(error_block_address, &block,
> +                                sizeof(AcpiGenericErrorStatus));
> +
> +    /* Get the current generic error status block length */
> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
> +        le32_to_cpu(block.data_length);
> +
> +    /* If the Generic Error Status Block is NULL, update
> +     * the block header
> +     */
> +    if (!block.block_status) {
> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    }
> +
> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    /* check whether it runs out of the preallocated memory */
> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
> +       GHES_MAX_RAW_DATA_LENGTH) {
> +        error_report("Record CPER out of boundary!!!");
> +        return GHES_CPER_FAIL;
> +    }
> +
> +    /* Write back the Generic Error Status Block to guest memory */
> +    cpu_physical_memory_write(error_block_address, &block,
> +        sizeof(AcpiGenericErrorStatus));
> +
> +    /* Fill in Generic Error Data Entry */
> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
> +                       sizeof(UefiCperSecMemErr));
> +
> +
> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +    gdata = (AcpiGenericErrorData *)buffer;
> +
> +    /* Memory section */
> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
> +            sizeof(mem_section_id_le));
> +
> +    /* error severity is recoverable */
> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    gdata->revision = 0x300; /* the revision number is 0x300 */
> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
> +
> +    /* User space only handle the memory section CPER */
> +
> +    /* Hard code to Multi-bit ECC error */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
> +
> +    /* Record the physical address at which the memory error occurred */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> +
> +    /* Write back the Generic Error Data Entry to guest memory */
> +    cpu_physical_memory_write(error_block_address + current_block_length,
> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +
> +    g_free(buffer);
> +    return GHES_CPER_OK;
> +}
> +
> +static void
> +build_address(GArray *table_data, BIOSLinker *linker,
> +    uint32_t dst_patched_offset, uint32_t src_offset,
> +    uint8_t address_space_id , uint8_t  register_bit_width,
> +    uint8_t register_bit_offset, uint8_t access_size)
> +{
> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    /* Address space */
> +    build_append_int_noprefix(table_data, address_space_id, 1);
> +    /* register bit width */
> +    build_append_int_noprefix(table_data, register_bit_width, 1);
> +    /* register bit offset */
> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
> +    /* access size */
> +    build_append_int_noprefix(table_data, access_size, 1);
> +    acpi_data_push(table_data, address_size);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
> +     * can retrieve and read it. the address size is 64 bits.
> +     */
> +    bios_linker_loader_add_pointer(linker,
> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
> +}
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                                            BIOSLinker *linker)
> +{
> +    uint32_t ghes_start = table_data->len;
> +    uint32_t address_size, error_status_address_offset;
> +    uint32_t read_ack_register_offset, i;
> +
> +    address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    error_status_address_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    read_ack_register_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    acpi_data_push(hardware_error,
> +        offsetof(struct hardware_errors_buffer, ack_value));
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Initialize read ack register */
> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
> +
> +    /* Reserved the total size for ERRORS fw_cfg blob
> +     */
> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
> +
> +    /* Allocate guest memory for the Data fw_cfg blob */
> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
> +                            1, false);
> +    /* Reserve table header size */
> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
> +
> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
> +        build_append_int_noprefix(table_data,
> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
> +        /* source id */
> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
> +        /* related source id */
> +        build_append_int_noprefix(table_data, 0xffff, 2);
> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
> +
> +        /* Currently only enable SEA notification type to avoid the kernel
> +         * warning, reserve the space for other notification error source
> +         */
> +        if (i == ACPI_HEST_NOTIFY_SEA) {
> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
> +        } else {
> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
> +        }
> +
> +        /* The number of error status block per generic hardware error source */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max sections per record */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max raw data length */
> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
> +
> +        /* Build error status address*/
> +        build_address(table_data, linker, error_status_address_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> +
> +        /* Hardware error notification structure */
> +        build_append_int_noprefix(table_data, i, 1); /* type */
> +        /* length */
> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
> +        build_append_int_noprefix(table_data, 0, 26);
> +
> +        /* Error Status Block Length */
> +        build_append_int_noprefix(table_data,
> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
> +
> +        /* Build read ack register */
> +        build_address(table_data, linker, read_ack_register_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2),
> +            offsetof(struct hardware_errors_buffer, ack_value) +
> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
> +            4 /* QWord access */);
> +
> +        /* Read ack preserve */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
> +
> +        /* Read ack write */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
> +    }
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Patch address of generic error status block into
> +         * the address register so OSPM can retrieve and read it.
> +         */
> +        bios_linker_loader_add_pointer(linker,
> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
> +            GHES_ERRORS_FW_CFG_FILE,
> +            offsetof(struct hardware_errors_buffer, gesb) +
> +            i * GHES_MAX_RAW_DATA_LENGTH);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
> +     * so QEMU can write the ERRORS there. The address is expected to be
> +     * < 4GB, but write 64 bits anyway.
> +     */
> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
> +        offsetof(struct hardware_errors_buffer, gesb));
> +
> +    build_header(linker, table_data,
> +        (void *)(table_data->data + ghes_start), "HEST",
> +        table_data->len - ghes_start, 1, NULL, "GHES");
> +}
> +
> +static GhesState ges;
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
> +{
> +
> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
> +
> +    /* Create a read-only fw_cfg file for GHES */
> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
> +                    size);
> +    /* Create a read-write fw_cfg file for Address */
> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
> +}
> +
> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
> +{
> +    uint64_t error_block_addr;
> +    uint64_t ack_value_addr, ack_value = 0;
> +    int loop = 0, ack_value_size;
> +    bool ret = GHES_CPER_FAIL;
> +
> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
> +        offsetof(struct hardware_errors_buffer, ack_value)) /
> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
> +
> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
> +        error_block_addr = le32_to_cpu(error_block_addr);
> +
> +        ack_value_addr = ges.ghes_addr_le -
> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
> +retry:
> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
> +        if (!ack_value) {
> +            if (loop < 3) {
> +                usleep(100 * 1000);
> +                loop++;
> +                goto retry;
> +            } else {
> +                error_report("Last time OSPM does not acknowledge the error,"
> +                    " record CPER failed this time, set the ack value to"
> +                    " avoid blocking next time CPER record! exit");
> +                ack_value = 1;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                return ret;
> +            }
> +        } else {
> +            /* A zero value in ghes_addr means that BIOS has not yet written
> +             * the address
> +             */
> +            if (error_block_addr) {
> +                ack_value = 0;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                ret = ghes_record_cper(error_block_addr, physical_address);
> +            }
> +        }
> +    }
> +    return ret;
> +}
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> index 3d78ff6..def1ec1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -45,6 +45,7 @@
>  #include "hw/arm/virt.h"
>  #include "sysemu/numa.h"
>  #include "kvm_arm.h"
> +#include "hw/acpi/hest_ghes.h"
>  
>  #define ARM_SPI_BASE 32
>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>      acpi_add_table(table_offsets, tables_blob);
>      build_spcr(tables_blob, tables->linker, vms);
>  
> +    acpi_add_table(table_offsets, tables_blob);
> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
> +
So we add this table unconditionally. Is there any bad impact if QEMU
runs on old kvm? Does it need to check whether KVM supports RAS?

>      if (nb_numa_nodes > 0) {
>          acpi_add_table(table_offsets, tables_blob);
>          build_srat(tables_blob, tables->linker, vms);
> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>                      acpi_data_len(tables.tcpalog));
>  
> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
> +
>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>                                                ACPI_BUILD_RSDP_FILE, 0);
>  
> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
> index 88d0738..7f7b55c 100644
> --- a/include/hw/acpi/aml-build.h
> +++ b/include/hw/acpi/aml-build.h
> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>      GArray *rsdp;
>      GArray *tcpalog;
>      GArray *vmgenid;
> +    GArray *hardware_errors;
>      BIOSLinker *linker;
>  } AcpiBuildTables;
>  
> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
> new file mode 100644
> index 0000000..0772756
> --- /dev/null
> +++ b/include/hw/acpi/hest_ghes.h
> @@ -0,0 +1,47 @@
> +/*
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> +
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * Authors:
> + *   Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef ACPI_GHES_H
> +#define ACPI_GHES_H
> +
> +#include "hw/acpi/bios-linker-loader.h"
> +
> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
> +
> +#define GHES_GAS_ADDRESS_OFFSET              4
> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
> +#define GHES_NOTIFICATION_STRUCTURE          32
> +
> +#define GHES_CPER_OK   1
> +#define GHES_CPER_FAIL 0
> +
> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
> +/* The max size in Bytes for one error block */
> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
> +
> +
> +typedef struct GhesState {
> +    uint64_t ghes_addr_le;
> +} GhesState;
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                            BIOSLinker *linker);
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
> +#endif
> 

-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
  2017-08-18 14:23   ` Dongjiu Geng
  (?)
  (?)
@ 2017-08-24 13:04     ` Shannon Zhao
  -1 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-24 13:04 UTC (permalink / raw)
  To: Dongjiu Geng, mst, imammedo, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/18 22:23, Dongjiu Geng wrote:
> Add CONFIG_ACPI_APEI configuration in the Makefile and
> enable it in the arm-softmmu.mak
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  default-configs/arm-softmmu.mak | 1 +
>  hw/acpi/Makefile.objs           | 1 +
>  2 files changed, 2 insertions(+)
> 
> diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
> index bbdd3c1..c362113 100644
> --- a/default-configs/arm-softmmu.mak
> +++ b/default-configs/arm-softmmu.mak
> @@ -129,3 +129,4 @@ CONFIG_ACPI=y
>  CONFIG_SMBIOS=y
>  CONFIG_ASPEED_SOC=y
>  CONFIG_GPIO_KEY=y
> +CONFIG_ACPI_APEI=y
> diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
> index 11c35bc..bafb148 100644
> --- a/hw/acpi/Makefile.objs
> +++ b/hw/acpi/Makefile.objs
> @@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
>  common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
>  common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
>  common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
> +common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
>  common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
>  
>  common-obj-y += acpi_interface.o
> 
Fold this patch into previous one.

Thanks,
-- 
Shannon


^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
@ 2017-08-24 13:04     ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-24 13:04 UTC (permalink / raw)
  To: Dongjiu Geng, mst, imammedo, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/18 22:23, Dongjiu Geng wrote:
> Add CONFIG_ACPI_APEI configuration in the Makefile and
> enable it in the arm-softmmu.mak
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  default-configs/arm-softmmu.mak | 1 +
>  hw/acpi/Makefile.objs           | 1 +
>  2 files changed, 2 insertions(+)
> 
> diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
> index bbdd3c1..c362113 100644
> --- a/default-configs/arm-softmmu.mak
> +++ b/default-configs/arm-softmmu.mak
> @@ -129,3 +129,4 @@ CONFIG_ACPI=y
>  CONFIG_SMBIOS=y
>  CONFIG_ASPEED_SOC=y
>  CONFIG_GPIO_KEY=y
> +CONFIG_ACPI_APEI=y
> diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
> index 11c35bc..bafb148 100644
> --- a/hw/acpi/Makefile.objs
> +++ b/hw/acpi/Makefile.objs
> @@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
>  common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
>  common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
>  common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
> +common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
>  common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
>  
>  common-obj-y += acpi_interface.o
> 
Fold this patch into previous one.

Thanks,
-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
@ 2017-08-24 13:04     ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-24 13:04 UTC (permalink / raw)
  To: Dongjiu Geng, mst, imammedo, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/18 22:23, Dongjiu Geng wrote:
> Add CONFIG_ACPI_APEI configuration in the Makefile and
> enable it in the arm-softmmu.mak
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  default-configs/arm-softmmu.mak | 1 +
>  hw/acpi/Makefile.objs           | 1 +
>  2 files changed, 2 insertions(+)
> 
> diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
> index bbdd3c1..c362113 100644
> --- a/default-configs/arm-softmmu.mak
> +++ b/default-configs/arm-softmmu.mak
> @@ -129,3 +129,4 @@ CONFIG_ACPI=y
>  CONFIG_SMBIOS=y
>  CONFIG_ASPEED_SOC=y
>  CONFIG_GPIO_KEY=y
> +CONFIG_ACPI_APEI=y
> diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
> index 11c35bc..bafb148 100644
> --- a/hw/acpi/Makefile.objs
> +++ b/hw/acpi/Makefile.objs
> @@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
>  common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
>  common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
>  common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
> +common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
>  common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
>  
>  common-obj-y += acpi_interface.o
> 
Fold this patch into previous one.

Thanks,
-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
@ 2017-08-24 13:04     ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-24 13:04 UTC (permalink / raw)
  To: linux-arm-kernel



On 2017/8/18 22:23, Dongjiu Geng wrote:
> Add CONFIG_ACPI_APEI configuration in the Makefile and
> enable it in the arm-softmmu.mak
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  default-configs/arm-softmmu.mak | 1 +
>  hw/acpi/Makefile.objs           | 1 +
>  2 files changed, 2 insertions(+)
> 
> diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
> index bbdd3c1..c362113 100644
> --- a/default-configs/arm-softmmu.mak
> +++ b/default-configs/arm-softmmu.mak
> @@ -129,3 +129,4 @@ CONFIG_ACPI=y
>  CONFIG_SMBIOS=y
>  CONFIG_ASPEED_SOC=y
>  CONFIG_GPIO_KEY=y
> +CONFIG_ACPI_APEI=y
> diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
> index 11c35bc..bafb148 100644
> --- a/hw/acpi/Makefile.objs
> +++ b/hw/acpi/Makefile.objs
> @@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
>  common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
>  common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
>  common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
> +common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
>  common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
>  
>  common-obj-y += acpi_interface.o
> 
Fold this patch into previous one.

Thanks,
-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
  2017-08-24 12:33     ` Shannon Zhao
                         ` (2 preceding siblings ...)
  (?)
@ 2017-08-25 10:37       ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 10:37 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10

Shannon,
   Thanks for the review. please see my reply.

On 2017/8/24 20:33, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> (1) Add related APEI/HEST table structures and  macros, these
>>     definition refer to ACPI 6.1 and UEFI 2.6 spec.
>> (2) Add generic error status block and CPER memory section
>>     definition, user space only handle memory section errors.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 193 insertions(+)
>>
>> diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
>> index 72be675..3b4bad7 100644
>> --- a/include/hw/acpi/acpi-defs.h
>> +++ b/include/hw/acpi/acpi-defs.h
>> @@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
>>  #define ACPI_APIC_GENERIC_TRANSLATOR    15
>>  #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
>>  
>> +/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
> missing "
  thanks for the pointing out.

> 
>> +#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
>> +#define UEFI_CPER_MEM_VALID_PA               0x0002
>> +#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
>> +#define UEFI_CPER_MEM_VALID_NODE             0x0008
>> +#define UEFI_CPER_MEM_VALID_CARD             0x0010
>> +#define UEFI_CPER_MEM_VALID_MODULE           0x0020
>> +#define UEFI_CPER_MEM_VALID_BANK             0x0040
>> +#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
>> +#define UEFI_CPER_MEM_VALID_ROW              0x0100
>> +#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
>> +#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
>> +#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
>> +#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
>> +#define UEFI_CPER_MEM_VALID_TARGET           0x2000
>> +#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
>> +#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
>> +#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
>> +#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
>> +#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
>> +
>> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>> +
> It's better to refer to the first spec version of this structure and
> same with others you define.
 do you mean which spec version? the definition is aligned with the linux kernel.
> 
>> +enum AcpiHestNotifyType {
>> +    ACPI_HEST_NOTIFY_POLLED = 0,
>> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>> +    ACPI_HEST_NOTIFY_SCI = 3,
>> +    ACPI_HEST_NOTIFY_NMI = 4,
>> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
> In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
> your patchset?
  it is usefull, for all the error source, I reserved the space for them.
Because the space is allocated one time, is not dynamically allocated.
so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.


> 
>> +};
>> +
>>  /*
>>   * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
>>   */
>> @@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
>>  } QEMU_PACKED;
>>  typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
>>  
>> +/* Hardware Error Notification, from the ACPI 6.1
>> + * spec, "18.3.2.9 Hardware Error Notification"
>> + */
> Use below style for multiple comment lines
> /*
>  * XXX
>  */
you are right, thanks for the pointing out.

> 
>> +struct AcpiHestNotify {
>> +    uint8_t type;
>> +    uint8_t length;
>> +    uint16_t config_write_enable;
>> +    uint32_t poll_interval;
>> +    uint32_t vector;
>> +    uint32_t polling_threshold_value;
>> +    uint32_t polling_threshold_window;
>> +    uint32_t error_threshold_value;
>> +    uint32_t error_threshold_window;
>> +} QEMU_PACKED;
>> +typedef struct AcpiHestNotify AcpiHestNotify;
>> +
>> +/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
>> + * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
>> + */
>> +enum AcpiHestSourceType {
>> +    ACPI_HEST_SOURCE_IA32_CHECK = 0,
>> +    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
>> +    ACPI_HEST_SOURCE_IA32_NMI = 2,
> What's 3, 4, 5 for?
   the ACPI spec do not use 3, 4, 5, so we not define them.

> 
>> +    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
>> +    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
>> +    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
>> +    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
>> +    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
>> +    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
>> +};
>> +
>> +/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
>> +#define ACPI_GEBS_UNCORRECTABLE             (1)
>> +#define ACPI_GEBS_CORRECTABLE               (1 << 1)
>> +#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
>> +#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
>> +/* 10 bits, error data entry count */
>> +#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
>> +
>> +/* Generic Hardware Error Source Structure, refer to ACPI 6.1
>> + * "18.3.2.7 Generic Hardware Error Source". in this struct the
>> + * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
>> + */
>> +
>> +struct AcpiGenericHardwareErrorSource {
>> +    uint16_t type;
>> +    uint16_t source_id;
>> +    uint16_t related_source_id;
>> +    uint8_t flags;
>> +    uint8_t enabled;
>> +    uint32_t number_of_records;
>> +    uint32_t max_sections_per_record;
>> +    uint32_t max_raw_data_length;
>> +    struct AcpiGenericAddress error_status_address;
>> +    struct AcpiHestNotify notify;
>> +    uint32_t error_status_block_length;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
>> +
>> +/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
>> + * Hardware Error Source version 2", in this struct the "type" field has to
>> + * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
>> + */
>> +struct AcpiGenericHardwareErrorSourceV2 {
>> +    uint16_t type;
>> +    uint16_t source_id;
>> +    uint16_t related_source_id;
>> +    uint8_t flags;
>> +    uint8_t enabled;
>> +    uint32_t number_of_records;
>> +    uint32_t max_sections_per_record;
>> +    uint32_t max_raw_data_length;
>> +    struct AcpiGenericAddress error_status_address;
>> +    struct AcpiHestNotify notify;
>> +    uint32_t error_status_block_length;
>> +    struct AcpiGenericAddress read_ack_register;
>> +    uint64_t read_ack_preserve;
>> +    uint64_t read_ack_write;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericHardwareErrorSourceV2
>> +            AcpiGenericHardwareErrorSourceV2;
>> +
>> +/* Generic Error Status block, from ACPI 6.1,
>> + * "18.3.2.7.1 Generic Error Data"
>> + */
>> +struct AcpiGenericErrorStatus {
>> +    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
>> +    uint32_t block_status;
>> +    uint32_t raw_data_offset;
>> +    uint32_t raw_data_length;
>> +    uint32_t data_length;
>> +    uint32_t error_severity;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
>> +
>> +enum AcpiGenericErrorSeverity {
>> +    ACPI_CPER_SEV_RECOVERABLE,
>> +    ACPI_CPER_SEV_FATAL,
>> +    ACPI_CPER_SEV_CORRECTED,
>> +    ACPI_CPER_SEV_NONE,
>> +};
>> +
>> +/* Generic Error Data entry, revision number is 0x0300,
>> + * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
>> + */
>> +struct AcpiGenericErrorData {
>> +    uint8_t section_type_le[16];
>> +    /* The "error_severity" fields that they take their
>> +     * values from AcpiGenericErrorSeverity
>> +     */
>> +    uint32_t error_severity;
>> +    uint16_t revision;
>> +    uint8_t validation_bits;
>> +    uint8_t flags;
>> +    uint32_t error_data_length;
>> +    uint8_t fru_id[16];
>> +    uint8_t fru_text[20];
>> +    uint64_t time_stamp;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericErrorData AcpiGenericErrorData;
>> +
>> +/* From UEFI 2.6, "N.2.5 Memory Error Section" */
>> +struct UefiCperSecMemErr {
>> +    uint64_t    validation_bits;
>> +    uint64_t    error_status;
>> +    uint64_t    physical_addr;
>> +    uint64_t    physical_addr_mask;
>> +    uint16_t    node;
>> +    uint16_t    card;
>> +    uint16_t    module;
>> +    uint16_t    bank;
>> +    uint16_t    device;
>> +    uint16_t    row;
>> +    uint16_t    column;
>> +    uint16_t    bit_pos;
>> +    uint64_t    requestor_id;
>> +    uint64_t    responder_id;
>> +    uint64_t    target_id;
>> +    uint8_t     error_type;
>> +    uint8_t     reserved;
>> +    uint16_t    rank;
>> +    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
>> +    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
>> +} QEMU_PACKED;
>> +typedef struct UefiCperSecMemErr UefiCperSecMemErr;
>> +
>> +/*
>> + * HEST Description Table
>> + */
>> +struct AcpiHardwareErrorSourceTable {
>> +    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
>> +    uint32_t           error_source_count;
>> +} QEMU_PACKED;
>> +typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
>> +
>>  #define ACPI_SRAT_PROCESSOR_APIC     0
>>  #define ACPI_SRAT_MEMORY             1
>>  #define ACPI_SRAT_PROCESSOR_x2APIC   2
>>
> 


^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-25 10:37       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 10:37 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10

Shannon,
   Thanks for the review. please see my reply.

On 2017/8/24 20:33, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> (1) Add related APEI/HEST table structures and  macros, these
>>     definition refer to ACPI 6.1 and UEFI 2.6 spec.
>> (2) Add generic error status block and CPER memory section
>>     definition, user space only handle memory section errors.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 193 insertions(+)
>>
>> diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
>> index 72be675..3b4bad7 100644
>> --- a/include/hw/acpi/acpi-defs.h
>> +++ b/include/hw/acpi/acpi-defs.h
>> @@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
>>  #define ACPI_APIC_GENERIC_TRANSLATOR    15
>>  #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
>>  
>> +/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
> missing "
  thanks for the pointing out.

> 
>> +#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
>> +#define UEFI_CPER_MEM_VALID_PA               0x0002
>> +#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
>> +#define UEFI_CPER_MEM_VALID_NODE             0x0008
>> +#define UEFI_CPER_MEM_VALID_CARD             0x0010
>> +#define UEFI_CPER_MEM_VALID_MODULE           0x0020
>> +#define UEFI_CPER_MEM_VALID_BANK             0x0040
>> +#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
>> +#define UEFI_CPER_MEM_VALID_ROW              0x0100
>> +#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
>> +#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
>> +#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
>> +#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
>> +#define UEFI_CPER_MEM_VALID_TARGET           0x2000
>> +#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
>> +#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
>> +#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
>> +#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
>> +#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
>> +
>> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>> +
> It's better to refer to the first spec version of this structure and
> same with others you define.
 do you mean which spec version? the definition is aligned with the linux kernel.
> 
>> +enum AcpiHestNotifyType {
>> +    ACPI_HEST_NOTIFY_POLLED = 0,
>> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>> +    ACPI_HEST_NOTIFY_SCI = 3,
>> +    ACPI_HEST_NOTIFY_NMI = 4,
>> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
> In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
> your patchset?
  it is usefull, for all the error source, I reserved the space for them.
Because the space is allocated one time, is not dynamically allocated.
so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.


> 
>> +};
>> +
>>  /*
>>   * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
>>   */
>> @@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
>>  } QEMU_PACKED;
>>  typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
>>  
>> +/* Hardware Error Notification, from the ACPI 6.1
>> + * spec, "18.3.2.9 Hardware Error Notification"
>> + */
> Use below style for multiple comment lines
> /*
>  * XXX
>  */
you are right, thanks for the pointing out.

> 
>> +struct AcpiHestNotify {
>> +    uint8_t type;
>> +    uint8_t length;
>> +    uint16_t config_write_enable;
>> +    uint32_t poll_interval;
>> +    uint32_t vector;
>> +    uint32_t polling_threshold_value;
>> +    uint32_t polling_threshold_window;
>> +    uint32_t error_threshold_value;
>> +    uint32_t error_threshold_window;
>> +} QEMU_PACKED;
>> +typedef struct AcpiHestNotify AcpiHestNotify;
>> +
>> +/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
>> + * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
>> + */
>> +enum AcpiHestSourceType {
>> +    ACPI_HEST_SOURCE_IA32_CHECK = 0,
>> +    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
>> +    ACPI_HEST_SOURCE_IA32_NMI = 2,
> What's 3, 4, 5 for?
   the ACPI spec do not use 3, 4, 5, so we not define them.

> 
>> +    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
>> +    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
>> +    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
>> +    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
>> +    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
>> +    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
>> +};
>> +
>> +/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
>> +#define ACPI_GEBS_UNCORRECTABLE             (1)
>> +#define ACPI_GEBS_CORRECTABLE               (1 << 1)
>> +#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
>> +#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
>> +/* 10 bits, error data entry count */
>> +#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
>> +
>> +/* Generic Hardware Error Source Structure, refer to ACPI 6.1
>> + * "18.3.2.7 Generic Hardware Error Source". in this struct the
>> + * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
>> + */
>> +
>> +struct AcpiGenericHardwareErrorSource {
>> +    uint16_t type;
>> +    uint16_t source_id;
>> +    uint16_t related_source_id;
>> +    uint8_t flags;
>> +    uint8_t enabled;
>> +    uint32_t number_of_records;
>> +    uint32_t max_sections_per_record;
>> +    uint32_t max_raw_data_length;
>> +    struct AcpiGenericAddress error_status_address;
>> +    struct AcpiHestNotify notify;
>> +    uint32_t error_status_block_length;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
>> +
>> +/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
>> + * Hardware Error Source version 2", in this struct the "type" field has to
>> + * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
>> + */
>> +struct AcpiGenericHardwareErrorSourceV2 {
>> +    uint16_t type;
>> +    uint16_t source_id;
>> +    uint16_t related_source_id;
>> +    uint8_t flags;
>> +    uint8_t enabled;
>> +    uint32_t number_of_records;
>> +    uint32_t max_sections_per_record;
>> +    uint32_t max_raw_data_length;
>> +    struct AcpiGenericAddress error_status_address;
>> +    struct AcpiHestNotify notify;
>> +    uint32_t error_status_block_length;
>> +    struct AcpiGenericAddress read_ack_register;
>> +    uint64_t read_ack_preserve;
>> +    uint64_t read_ack_write;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericHardwareErrorSourceV2
>> +            AcpiGenericHardwareErrorSourceV2;
>> +
>> +/* Generic Error Status block, from ACPI 6.1,
>> + * "18.3.2.7.1 Generic Error Data"
>> + */
>> +struct AcpiGenericErrorStatus {
>> +    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
>> +    uint32_t block_status;
>> +    uint32_t raw_data_offset;
>> +    uint32_t raw_data_length;
>> +    uint32_t data_length;
>> +    uint32_t error_severity;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
>> +
>> +enum AcpiGenericErrorSeverity {
>> +    ACPI_CPER_SEV_RECOVERABLE,
>> +    ACPI_CPER_SEV_FATAL,
>> +    ACPI_CPER_SEV_CORRECTED,
>> +    ACPI_CPER_SEV_NONE,
>> +};
>> +
>> +/* Generic Error Data entry, revision number is 0x0300,
>> + * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
>> + */
>> +struct AcpiGenericErrorData {
>> +    uint8_t section_type_le[16];
>> +    /* The "error_severity" fields that they take their
>> +     * values from AcpiGenericErrorSeverity
>> +     */
>> +    uint32_t error_severity;
>> +    uint16_t revision;
>> +    uint8_t validation_bits;
>> +    uint8_t flags;
>> +    uint32_t error_data_length;
>> +    uint8_t fru_id[16];
>> +    uint8_t fru_text[20];
>> +    uint64_t time_stamp;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericErrorData AcpiGenericErrorData;
>> +
>> +/* From UEFI 2.6, "N.2.5 Memory Error Section" */
>> +struct UefiCperSecMemErr {
>> +    uint64_t    validation_bits;
>> +    uint64_t    error_status;
>> +    uint64_t    physical_addr;
>> +    uint64_t    physical_addr_mask;
>> +    uint16_t    node;
>> +    uint16_t    card;
>> +    uint16_t    module;
>> +    uint16_t    bank;
>> +    uint16_t    device;
>> +    uint16_t    row;
>> +    uint16_t    column;
>> +    uint16_t    bit_pos;
>> +    uint64_t    requestor_id;
>> +    uint64_t    responder_id;
>> +    uint64_t    target_id;
>> +    uint8_t     error_type;
>> +    uint8_t     reserved;
>> +    uint16_t    rank;
>> +    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
>> +    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
>> +} QEMU_PACKED;
>> +typedef struct UefiCperSecMemErr UefiCperSecMemErr;
>> +
>> +/*
>> + * HEST Description Table
>> + */
>> +struct AcpiHardwareErrorSourceTable {
>> +    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
>> +    uint32_t           error_source_count;
>> +} QEMU_PACKED;
>> +typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
>> +
>>  #define ACPI_SRAT_PROCESSOR_APIC     0
>>  #define ACPI_SRAT_MEMORY             1
>>  #define ACPI_SRAT_PROCESSOR_x2APIC   2
>>
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-25 10:37       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 10:37 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10

Shannon,
   Thanks for the review. please see my reply.

On 2017/8/24 20:33, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> (1) Add related APEI/HEST table structures and  macros, these
>>     definition refer to ACPI 6.1 and UEFI 2.6 spec.
>> (2) Add generic error status block and CPER memory section
>>     definition, user space only handle memory section errors.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 193 insertions(+)
>>
>> diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
>> index 72be675..3b4bad7 100644
>> --- a/include/hw/acpi/acpi-defs.h
>> +++ b/include/hw/acpi/acpi-defs.h
>> @@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
>>  #define ACPI_APIC_GENERIC_TRANSLATOR    15
>>  #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
>>  
>> +/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
> missing "
  thanks for the pointing out.

> 
>> +#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
>> +#define UEFI_CPER_MEM_VALID_PA               0x0002
>> +#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
>> +#define UEFI_CPER_MEM_VALID_NODE             0x0008
>> +#define UEFI_CPER_MEM_VALID_CARD             0x0010
>> +#define UEFI_CPER_MEM_VALID_MODULE           0x0020
>> +#define UEFI_CPER_MEM_VALID_BANK             0x0040
>> +#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
>> +#define UEFI_CPER_MEM_VALID_ROW              0x0100
>> +#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
>> +#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
>> +#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
>> +#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
>> +#define UEFI_CPER_MEM_VALID_TARGET           0x2000
>> +#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
>> +#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
>> +#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
>> +#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
>> +#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
>> +
>> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>> +
> It's better to refer to the first spec version of this structure and
> same with others you define.
 do you mean which spec version? the definition is aligned with the linux kernel.
> 
>> +enum AcpiHestNotifyType {
>> +    ACPI_HEST_NOTIFY_POLLED = 0,
>> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>> +    ACPI_HEST_NOTIFY_SCI = 3,
>> +    ACPI_HEST_NOTIFY_NMI = 4,
>> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
> In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
> your patchset?
  it is usefull, for all the error source, I reserved the space for them.
Because the space is allocated one time, is not dynamically allocated.
so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.


> 
>> +};
>> +
>>  /*
>>   * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
>>   */
>> @@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
>>  } QEMU_PACKED;
>>  typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
>>  
>> +/* Hardware Error Notification, from the ACPI 6.1
>> + * spec, "18.3.2.9 Hardware Error Notification"
>> + */
> Use below style for multiple comment lines
> /*
>  * XXX
>  */
you are right, thanks for the pointing out.

> 
>> +struct AcpiHestNotify {
>> +    uint8_t type;
>> +    uint8_t length;
>> +    uint16_t config_write_enable;
>> +    uint32_t poll_interval;
>> +    uint32_t vector;
>> +    uint32_t polling_threshold_value;
>> +    uint32_t polling_threshold_window;
>> +    uint32_t error_threshold_value;
>> +    uint32_t error_threshold_window;
>> +} QEMU_PACKED;
>> +typedef struct AcpiHestNotify AcpiHestNotify;
>> +
>> +/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
>> + * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
>> + */
>> +enum AcpiHestSourceType {
>> +    ACPI_HEST_SOURCE_IA32_CHECK = 0,
>> +    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
>> +    ACPI_HEST_SOURCE_IA32_NMI = 2,
> What's 3, 4, 5 for?
   the ACPI spec do not use 3, 4, 5, so we not define them.

> 
>> +    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
>> +    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
>> +    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
>> +    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
>> +    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
>> +    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
>> +};
>> +
>> +/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
>> +#define ACPI_GEBS_UNCORRECTABLE             (1)
>> +#define ACPI_GEBS_CORRECTABLE               (1 << 1)
>> +#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
>> +#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
>> +/* 10 bits, error data entry count */
>> +#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
>> +
>> +/* Generic Hardware Error Source Structure, refer to ACPI 6.1
>> + * "18.3.2.7 Generic Hardware Error Source". in this struct the
>> + * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
>> + */
>> +
>> +struct AcpiGenericHardwareErrorSource {
>> +    uint16_t type;
>> +    uint16_t source_id;
>> +    uint16_t related_source_id;
>> +    uint8_t flags;
>> +    uint8_t enabled;
>> +    uint32_t number_of_records;
>> +    uint32_t max_sections_per_record;
>> +    uint32_t max_raw_data_length;
>> +    struct AcpiGenericAddress error_status_address;
>> +    struct AcpiHestNotify notify;
>> +    uint32_t error_status_block_length;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
>> +
>> +/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
>> + * Hardware Error Source version 2", in this struct the "type" field has to
>> + * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
>> + */
>> +struct AcpiGenericHardwareErrorSourceV2 {
>> +    uint16_t type;
>> +    uint16_t source_id;
>> +    uint16_t related_source_id;
>> +    uint8_t flags;
>> +    uint8_t enabled;
>> +    uint32_t number_of_records;
>> +    uint32_t max_sections_per_record;
>> +    uint32_t max_raw_data_length;
>> +    struct AcpiGenericAddress error_status_address;
>> +    struct AcpiHestNotify notify;
>> +    uint32_t error_status_block_length;
>> +    struct AcpiGenericAddress read_ack_register;
>> +    uint64_t read_ack_preserve;
>> +    uint64_t read_ack_write;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericHardwareErrorSourceV2
>> +            AcpiGenericHardwareErrorSourceV2;
>> +
>> +/* Generic Error Status block, from ACPI 6.1,
>> + * "18.3.2.7.1 Generic Error Data"
>> + */
>> +struct AcpiGenericErrorStatus {
>> +    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
>> +    uint32_t block_status;
>> +    uint32_t raw_data_offset;
>> +    uint32_t raw_data_length;
>> +    uint32_t data_length;
>> +    uint32_t error_severity;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
>> +
>> +enum AcpiGenericErrorSeverity {
>> +    ACPI_CPER_SEV_RECOVERABLE,
>> +    ACPI_CPER_SEV_FATAL,
>> +    ACPI_CPER_SEV_CORRECTED,
>> +    ACPI_CPER_SEV_NONE,
>> +};
>> +
>> +/* Generic Error Data entry, revision number is 0x0300,
>> + * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
>> + */
>> +struct AcpiGenericErrorData {
>> +    uint8_t section_type_le[16];
>> +    /* The "error_severity" fields that they take their
>> +     * values from AcpiGenericErrorSeverity
>> +     */
>> +    uint32_t error_severity;
>> +    uint16_t revision;
>> +    uint8_t validation_bits;
>> +    uint8_t flags;
>> +    uint32_t error_data_length;
>> +    uint8_t fru_id[16];
>> +    uint8_t fru_text[20];
>> +    uint64_t time_stamp;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericErrorData AcpiGenericErrorData;
>> +
>> +/* From UEFI 2.6, "N.2.5 Memory Error Section" */
>> +struct UefiCperSecMemErr {
>> +    uint64_t    validation_bits;
>> +    uint64_t    error_status;
>> +    uint64_t    physical_addr;
>> +    uint64_t    physical_addr_mask;
>> +    uint16_t    node;
>> +    uint16_t    card;
>> +    uint16_t    module;
>> +    uint16_t    bank;
>> +    uint16_t    device;
>> +    uint16_t    row;
>> +    uint16_t    column;
>> +    uint16_t    bit_pos;
>> +    uint64_t    requestor_id;
>> +    uint64_t    responder_id;
>> +    uint64_t    target_id;
>> +    uint8_t     error_type;
>> +    uint8_t     reserved;
>> +    uint16_t    rank;
>> +    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
>> +    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
>> +} QEMU_PACKED;
>> +typedef struct UefiCperSecMemErr UefiCperSecMemErr;
>> +
>> +/*
>> + * HEST Description Table
>> + */
>> +struct AcpiHardwareErrorSourceTable {
>> +    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
>> +    uint32_t           error_source_count;
>> +} QEMU_PACKED;
>> +typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
>> +
>>  #define ACPI_SRAT_PROCESSOR_APIC     0
>>  #define ACPI_SRAT_MEMORY             1
>>  #define ACPI_SRAT_PROCESSOR_x2APIC   2
>>
> 


^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-25 10:37       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 10:37 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10

Shannon,
   Thanks for the review. please see my reply.

On 2017/8/24 20:33, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> (1) Add related APEI/HEST table structures and  macros, these
>>     definition refer to ACPI 6.1 and UEFI 2.6 spec.
>> (2) Add generic error status block and CPER memory section
>>     definition, user space only handle memory section errors.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 193 insertions(+)
>>
>> diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
>> index 72be675..3b4bad7 100644
>> --- a/include/hw/acpi/acpi-defs.h
>> +++ b/include/hw/acpi/acpi-defs.h
>> @@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
>>  #define ACPI_APIC_GENERIC_TRANSLATOR    15
>>  #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
>>  
>> +/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
> missing "
  thanks for the pointing out.

> 
>> +#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
>> +#define UEFI_CPER_MEM_VALID_PA               0x0002
>> +#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
>> +#define UEFI_CPER_MEM_VALID_NODE             0x0008
>> +#define UEFI_CPER_MEM_VALID_CARD             0x0010
>> +#define UEFI_CPER_MEM_VALID_MODULE           0x0020
>> +#define UEFI_CPER_MEM_VALID_BANK             0x0040
>> +#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
>> +#define UEFI_CPER_MEM_VALID_ROW              0x0100
>> +#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
>> +#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
>> +#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
>> +#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
>> +#define UEFI_CPER_MEM_VALID_TARGET           0x2000
>> +#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
>> +#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
>> +#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
>> +#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
>> +#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
>> +
>> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>> +
> It's better to refer to the first spec version of this structure and
> same with others you define.
 do you mean which spec version? the definition is aligned with the linux kernel.
> 
>> +enum AcpiHestNotifyType {
>> +    ACPI_HEST_NOTIFY_POLLED = 0,
>> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>> +    ACPI_HEST_NOTIFY_SCI = 3,
>> +    ACPI_HEST_NOTIFY_NMI = 4,
>> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
> In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
> your patchset?
  it is usefull, for all the error source, I reserved the space for them.
Because the space is allocated one time, is not dynamically allocated.
so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.


> 
>> +};
>> +
>>  /*
>>   * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
>>   */
>> @@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
>>  } QEMU_PACKED;
>>  typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
>>  
>> +/* Hardware Error Notification, from the ACPI 6.1
>> + * spec, "18.3.2.9 Hardware Error Notification"
>> + */
> Use below style for multiple comment lines
> /*
>  * XXX
>  */
you are right, thanks for the pointing out.

> 
>> +struct AcpiHestNotify {
>> +    uint8_t type;
>> +    uint8_t length;
>> +    uint16_t config_write_enable;
>> +    uint32_t poll_interval;
>> +    uint32_t vector;
>> +    uint32_t polling_threshold_value;
>> +    uint32_t polling_threshold_window;
>> +    uint32_t error_threshold_value;
>> +    uint32_t error_threshold_window;
>> +} QEMU_PACKED;
>> +typedef struct AcpiHestNotify AcpiHestNotify;
>> +
>> +/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
>> + * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
>> + */
>> +enum AcpiHestSourceType {
>> +    ACPI_HEST_SOURCE_IA32_CHECK = 0,
>> +    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
>> +    ACPI_HEST_SOURCE_IA32_NMI = 2,
> What's 3, 4, 5 for?
   the ACPI spec do not use 3, 4, 5, so we not define them.

> 
>> +    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
>> +    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
>> +    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
>> +    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
>> +    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
>> +    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
>> +};
>> +
>> +/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
>> +#define ACPI_GEBS_UNCORRECTABLE             (1)
>> +#define ACPI_GEBS_CORRECTABLE               (1 << 1)
>> +#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
>> +#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
>> +/* 10 bits, error data entry count */
>> +#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
>> +
>> +/* Generic Hardware Error Source Structure, refer to ACPI 6.1
>> + * "18.3.2.7 Generic Hardware Error Source". in this struct the
>> + * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
>> + */
>> +
>> +struct AcpiGenericHardwareErrorSource {
>> +    uint16_t type;
>> +    uint16_t source_id;
>> +    uint16_t related_source_id;
>> +    uint8_t flags;
>> +    uint8_t enabled;
>> +    uint32_t number_of_records;
>> +    uint32_t max_sections_per_record;
>> +    uint32_t max_raw_data_length;
>> +    struct AcpiGenericAddress error_status_address;
>> +    struct AcpiHestNotify notify;
>> +    uint32_t error_status_block_length;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
>> +
>> +/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
>> + * Hardware Error Source version 2", in this struct the "type" field has to
>> + * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
>> + */
>> +struct AcpiGenericHardwareErrorSourceV2 {
>> +    uint16_t type;
>> +    uint16_t source_id;
>> +    uint16_t related_source_id;
>> +    uint8_t flags;
>> +    uint8_t enabled;
>> +    uint32_t number_of_records;
>> +    uint32_t max_sections_per_record;
>> +    uint32_t max_raw_data_length;
>> +    struct AcpiGenericAddress error_status_address;
>> +    struct AcpiHestNotify notify;
>> +    uint32_t error_status_block_length;
>> +    struct AcpiGenericAddress read_ack_register;
>> +    uint64_t read_ack_preserve;
>> +    uint64_t read_ack_write;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericHardwareErrorSourceV2
>> +            AcpiGenericHardwareErrorSourceV2;
>> +
>> +/* Generic Error Status block, from ACPI 6.1,
>> + * "18.3.2.7.1 Generic Error Data"
>> + */
>> +struct AcpiGenericErrorStatus {
>> +    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
>> +    uint32_t block_status;
>> +    uint32_t raw_data_offset;
>> +    uint32_t raw_data_length;
>> +    uint32_t data_length;
>> +    uint32_t error_severity;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
>> +
>> +enum AcpiGenericErrorSeverity {
>> +    ACPI_CPER_SEV_RECOVERABLE,
>> +    ACPI_CPER_SEV_FATAL,
>> +    ACPI_CPER_SEV_CORRECTED,
>> +    ACPI_CPER_SEV_NONE,
>> +};
>> +
>> +/* Generic Error Data entry, revision number is 0x0300,
>> + * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
>> + */
>> +struct AcpiGenericErrorData {
>> +    uint8_t section_type_le[16];
>> +    /* The "error_severity" fields that they take their
>> +     * values from AcpiGenericErrorSeverity
>> +     */
>> +    uint32_t error_severity;
>> +    uint16_t revision;
>> +    uint8_t validation_bits;
>> +    uint8_t flags;
>> +    uint32_t error_data_length;
>> +    uint8_t fru_id[16];
>> +    uint8_t fru_text[20];
>> +    uint64_t time_stamp;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericErrorData AcpiGenericErrorData;
>> +
>> +/* From UEFI 2.6, "N.2.5 Memory Error Section" */
>> +struct UefiCperSecMemErr {
>> +    uint64_t    validation_bits;
>> +    uint64_t    error_status;
>> +    uint64_t    physical_addr;
>> +    uint64_t    physical_addr_mask;
>> +    uint16_t    node;
>> +    uint16_t    card;
>> +    uint16_t    module;
>> +    uint16_t    bank;
>> +    uint16_t    device;
>> +    uint16_t    row;
>> +    uint16_t    column;
>> +    uint16_t    bit_pos;
>> +    uint64_t    requestor_id;
>> +    uint64_t    responder_id;
>> +    uint64_t    target_id;
>> +    uint8_t     error_type;
>> +    uint8_t     reserved;
>> +    uint16_t    rank;
>> +    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
>> +    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
>> +} QEMU_PACKED;
>> +typedef struct UefiCperSecMemErr UefiCperSecMemErr;
>> +
>> +/*
>> + * HEST Description Table
>> + */
>> +struct AcpiHardwareErrorSourceTable {
>> +    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
>> +    uint32_t           error_source_count;
>> +} QEMU_PACKED;
>> +typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
>> +
>>  #define ACPI_SRAT_PROCESSOR_APIC     0
>>  #define ACPI_SRAT_MEMORY             1
>>  #define ACPI_SRAT_PROCESSOR_x2APIC   2
>>
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-25 10:37       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 10:37 UTC (permalink / raw)
  To: linux-arm-kernel

Shannon,
   Thanks for the review. please see my reply.

On 2017/8/24 20:33, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> (1) Add related APEI/HEST table structures and  macros, these
>>     definition refer to ACPI 6.1 and UEFI 2.6 spec.
>> (2) Add generic error status block and CPER memory section
>>     definition, user space only handle memory section errors.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  include/hw/acpi/acpi-defs.h | 193 ++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 193 insertions(+)
>>
>> diff --git a/include/hw/acpi/acpi-defs.h b/include/hw/acpi/acpi-defs.h
>> index 72be675..3b4bad7 100644
>> --- a/include/hw/acpi/acpi-defs.h
>> +++ b/include/hw/acpi/acpi-defs.h
>> @@ -297,6 +297,44 @@ typedef struct AcpiMultipleApicTable AcpiMultipleApicTable;
>>  #define ACPI_APIC_GENERIC_TRANSLATOR    15
>>  #define ACPI_APIC_RESERVED              16   /* 16 and greater are reserved */
>>  
>> +/* UEFI Spec 2.6, "N.2.5 Memory Error Section */
> missing "
  thanks for the pointing out.

> 
>> +#define UEFI_CPER_MEM_VALID_ERROR_STATUS     0x0001
>> +#define UEFI_CPER_MEM_VALID_PA               0x0002
>> +#define UEFI_CPER_MEM_VALID_PA_MASK          0x0004
>> +#define UEFI_CPER_MEM_VALID_NODE             0x0008
>> +#define UEFI_CPER_MEM_VALID_CARD             0x0010
>> +#define UEFI_CPER_MEM_VALID_MODULE           0x0020
>> +#define UEFI_CPER_MEM_VALID_BANK             0x0040
>> +#define UEFI_CPER_MEM_VALID_DEVICE           0x0080
>> +#define UEFI_CPER_MEM_VALID_ROW              0x0100
>> +#define UEFI_CPER_MEM_VALID_COLUMN           0x0200
>> +#define UEFI_CPER_MEM_VALID_BIT_POSITION     0x0400
>> +#define UEFI_CPER_MEM_VALID_REQUESTOR        0x0800
>> +#define UEFI_CPER_MEM_VALID_RESPONDER        0x1000
>> +#define UEFI_CPER_MEM_VALID_TARGET           0x2000
>> +#define UEFI_CPER_MEM_VALID_ERROR_TYPE       0x4000
>> +#define UEFI_CPER_MEM_VALID_RANK_NUMBER      0x8000
>> +#define UEFI_CPER_MEM_VALID_CARD_HANDLE      0x10000
>> +#define UEFI_CPER_MEM_VALID_MODULE_HANDLE    0x20000
>> +#define UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC   3
>> +
>> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>> +
> It's better to refer to the first spec version of this structure and
> same with others you define.
 do you mean which spec version? the definition is aligned with the linux kernel.
> 
>> +enum AcpiHestNotifyType {
>> +    ACPI_HEST_NOTIFY_POLLED = 0,
>> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>> +    ACPI_HEST_NOTIFY_SCI = 3,
>> +    ACPI_HEST_NOTIFY_NMI = 4,
>> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
> In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
> your patchset?
  it is usefull, for all the error source, I reserved the space for them.
Because the space is allocated one time, is not dynamically allocated.
so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.


> 
>> +};
>> +
>>  /*
>>   * MADT sub-structures (Follow MULTIPLE_APIC_DESCRIPTION_TABLE)
>>   */
>> @@ -474,6 +512,161 @@ struct AcpiSystemResourceAffinityTable {
>>  } QEMU_PACKED;
>>  typedef struct AcpiSystemResourceAffinityTable AcpiSystemResourceAffinityTable;
>>  
>> +/* Hardware Error Notification, from the ACPI 6.1
>> + * spec, "18.3.2.9 Hardware Error Notification"
>> + */
> Use below style for multiple comment lines
> /*
>  * XXX
>  */
you are right, thanks for the pointing out.

> 
>> +struct AcpiHestNotify {
>> +    uint8_t type;
>> +    uint8_t length;
>> +    uint16_t config_write_enable;
>> +    uint32_t poll_interval;
>> +    uint32_t vector;
>> +    uint32_t polling_threshold_value;
>> +    uint32_t polling_threshold_window;
>> +    uint32_t error_threshold_value;
>> +    uint32_t error_threshold_window;
>> +} QEMU_PACKED;
>> +typedef struct AcpiHestNotify AcpiHestNotify;
>> +
>> +/* From ACPI 6.1, sections "18.3.2.1 IA-32 Architecture Machine
>> + * Check Exception" through "18.3.2.8 Generic Hardware Error Source version 2".
>> + */
>> +enum AcpiHestSourceType {
>> +    ACPI_HEST_SOURCE_IA32_CHECK = 0,
>> +    ACPI_HEST_SOURCE_IA32_CORRECTED_CHECK = 1,
>> +    ACPI_HEST_SOURCE_IA32_NMI = 2,
> What's 3, 4, 5 for?
   the ACPI spec do not use 3, 4, 5, so we not define them.

> 
>> +    ACPI_HEST_SOURCE_AER_ROOT_PORT = 6,
>> +    ACPI_HEST_SOURCE_AER_ENDPOINT = 7,
>> +    ACPI_HEST_SOURCE_AER_BRIDGE = 8,
>> +    ACPI_HEST_SOURCE_GENERIC_ERROR = 9,
>> +    ACPI_HEST_SOURCE_GENERIC_ERROR_V2 = 10,
>> +    ACPI_HEST_SOURCE_RESERVED = 11    /* 11 and greater are reserved */
>> +};
>> +
>> +/* Block status bitmasks from ACPI 6.1, "18.3.2.7.1 Generic Error Data" */
>> +#define ACPI_GEBS_UNCORRECTABLE             (1)
>> +#define ACPI_GEBS_CORRECTABLE               (1 << 1)
>> +#define ACPI_GEBS_MULTIPLE_UNCORRECTABLE    (1 << 2)
>> +#define ACPI_GEBS_MULTIPLE_CORRECTABLE      (1 << 3)
>> +/* 10 bits, error data entry count */
>> +#define ACPI_GEBS_ERROR_ENTRY_COUNT         (0x3FF << 4)
>> +
>> +/* Generic Hardware Error Source Structure, refer to ACPI 6.1
>> + * "18.3.2.7 Generic Hardware Error Source". in this struct the
>> + * "type" field has to be ACPI_HEST_SOURCE_GENERIC_ERROR
>> + */
>> +
>> +struct AcpiGenericHardwareErrorSource {
>> +    uint16_t type;
>> +    uint16_t source_id;
>> +    uint16_t related_source_id;
>> +    uint8_t flags;
>> +    uint8_t enabled;
>> +    uint32_t number_of_records;
>> +    uint32_t max_sections_per_record;
>> +    uint32_t max_raw_data_length;
>> +    struct AcpiGenericAddress error_status_address;
>> +    struct AcpiHestNotify notify;
>> +    uint32_t error_status_block_length;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericHardwareErrorSource AcpiGenericHardwareErrorSource;
>> +
>> +/* Generic Hardware Error Source, version 2, ACPI 6.1, "18.3.2.8 Generic
>> + * Hardware Error Source version 2", in this struct the "type" field has to
>> + * be ACPI_HEST_SOURCE_GENERIC_ERROR_V2
>> + */
>> +struct AcpiGenericHardwareErrorSourceV2 {
>> +    uint16_t type;
>> +    uint16_t source_id;
>> +    uint16_t related_source_id;
>> +    uint8_t flags;
>> +    uint8_t enabled;
>> +    uint32_t number_of_records;
>> +    uint32_t max_sections_per_record;
>> +    uint32_t max_raw_data_length;
>> +    struct AcpiGenericAddress error_status_address;
>> +    struct AcpiHestNotify notify;
>> +    uint32_t error_status_block_length;
>> +    struct AcpiGenericAddress read_ack_register;
>> +    uint64_t read_ack_preserve;
>> +    uint64_t read_ack_write;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericHardwareErrorSourceV2
>> +            AcpiGenericHardwareErrorSourceV2;
>> +
>> +/* Generic Error Status block, from ACPI 6.1,
>> + * "18.3.2.7.1 Generic Error Data"
>> + */
>> +struct AcpiGenericErrorStatus {
>> +    /* It is a bitmask composed of ACPI_GEBS_xxx macros */
>> +    uint32_t block_status;
>> +    uint32_t raw_data_offset;
>> +    uint32_t raw_data_length;
>> +    uint32_t data_length;
>> +    uint32_t error_severity;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericErrorStatus AcpiGenericErrorStatus;
>> +
>> +enum AcpiGenericErrorSeverity {
>> +    ACPI_CPER_SEV_RECOVERABLE,
>> +    ACPI_CPER_SEV_FATAL,
>> +    ACPI_CPER_SEV_CORRECTED,
>> +    ACPI_CPER_SEV_NONE,
>> +};
>> +
>> +/* Generic Error Data entry, revision number is 0x0300,
>> + * ACPI 6.1, "18.3.2.7.1 Generic Error Data"
>> + */
>> +struct AcpiGenericErrorData {
>> +    uint8_t section_type_le[16];
>> +    /* The "error_severity" fields that they take their
>> +     * values from AcpiGenericErrorSeverity
>> +     */
>> +    uint32_t error_severity;
>> +    uint16_t revision;
>> +    uint8_t validation_bits;
>> +    uint8_t flags;
>> +    uint32_t error_data_length;
>> +    uint8_t fru_id[16];
>> +    uint8_t fru_text[20];
>> +    uint64_t time_stamp;
>> +} QEMU_PACKED;
>> +typedef struct AcpiGenericErrorData AcpiGenericErrorData;
>> +
>> +/* From UEFI 2.6, "N.2.5 Memory Error Section" */
>> +struct UefiCperSecMemErr {
>> +    uint64_t    validation_bits;
>> +    uint64_t    error_status;
>> +    uint64_t    physical_addr;
>> +    uint64_t    physical_addr_mask;
>> +    uint16_t    node;
>> +    uint16_t    card;
>> +    uint16_t    module;
>> +    uint16_t    bank;
>> +    uint16_t    device;
>> +    uint16_t    row;
>> +    uint16_t    column;
>> +    uint16_t    bit_pos;
>> +    uint64_t    requestor_id;
>> +    uint64_t    responder_id;
>> +    uint64_t    target_id;
>> +    uint8_t     error_type;
>> +    uint8_t     reserved;
>> +    uint16_t    rank;
>> +    uint16_t    mem_array_handle;   /* card handle in UEFI 2.4 */
>> +    uint16_t    mem_dev_handle;     /* module handle in UEFI 2.4 */
>> +} QEMU_PACKED;
>> +typedef struct UefiCperSecMemErr UefiCperSecMemErr;
>> +
>> +/*
>> + * HEST Description Table
>> + */
>> +struct AcpiHardwareErrorSourceTable {
>> +    ACPI_TABLE_HEADER_DEF                    /* ACPI common table header */
>> +    uint32_t           error_source_count;
>> +} QEMU_PACKED;
>> +typedef struct AcpiHardwareErrorSourceTable AcpiHardwareErrorSourceTable;
>> +
>>  #define ACPI_SRAT_PROCESSOR_APIC     0
>>  #define ACPI_SRAT_MEMORY             1
>>  #define ACPI_SRAT_PROCESSOR_x2APIC   2
>>
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
  2017-08-24 13:03     ` Shannon Zhao
                         ` (2 preceding siblings ...)
  (?)
@ 2017-08-25 11:20       ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 11:20 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10

Hi Shannon,

On 2017/8/24 21:03, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
>>
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
>>
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  hw/acpi/aml-build.c         |   2 +
>>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>>  hw/arm/virt-acpi-build.c    |   6 +
>>  include/hw/acpi/aml-build.h |   1 +
>>  include/hw/acpi/hest_ghes.h |  47 ++++++
>>  5 files changed, 401 insertions(+)
>>  create mode 100644 hw/acpi/hest_ghes.c
> Don't need to add the new file to hw/acpi/Makefile.objs?
  I modified the Makefile.objs in another patch.

> 
>>  create mode 100644 include/hw/acpi/hest_ghes.h
>>
>> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
>> index 36a6cc4..6849e5f 100644
>> --- a/hw/acpi/aml-build.c
>> +++ b/hw/acpi/aml-build.c
>> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>>      tables->table_data = g_array_new(false, true /* clear */, 1);
>>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
>> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>>      tables->linker = bios_linker_loader_init();
>>  }
>>  
>> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>>      g_array_free(tables->table_data, true);
>>      g_array_free(tables->tcpalog, mfre);
>>      g_array_free(tables->vmgenid, mfre);
>> +    g_array_free(tables->hardware_errors, mfre);
>>  }
>>  
>>  /* Build rsdt table */
>> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
>> new file mode 100644
>> index 0000000..ff6b5ef
>> --- /dev/null
>> +++ b/hw/acpi/hest_ghes.c
>> @@ -0,0 +1,345 @@
>> +/*
>> + *  APEI GHES table Generation
>> + *
>> + *  Copyright (C) 2017 huawei.
>> + *
>> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + *
>> + */
> Please unify this of this file and hest_ghes.h by refering to other files.
Ok, thanks.


> 
>> +
>> +#include "qemu/osdep.h"
>> +#include "qmp-commands.h"
> unnecessary including
I will remove it.

> 
>> +#include "hw/acpi/acpi.h"
>> +#include "hw/acpi/aml-build.h"
>> +#include "hw/acpi/hest_ghes.h"
>> +#include "hw/nvram/fw_cfg.h"
>> +#include "sysemu/sysemu.h"
>> +#include "qemu/error-report.h"
>> +
>> +/* The structure that stands for the layout
>> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
>> + *
>> + *           etc/hardware_errors
>> + * ==========================================
>> + * +------------------+
>> + * |    address       |              +--------------+
>> + * |    registers     |              | Error Status |
>> + * | +----------------+              | Data Block 0 |
>> + * | |status_address0 |------------->| +------------+
>> + * | +----------------+              | |  CPER      |
>> + * | |status_address1 |----------+   | |  CPER      |
>> + * | +----------------+          |   | |  ....      |
>> + * | |.............   |          |   | |  CPER      |
>> + * | +----------------+          |   | +------------+
>> + * | |status_address10|-----+    |   | Error Status |
>> + * | +----------------+     |    |   | Data Block 1 |
>> + * | |ack_value0      |     |    +-->| +------------+
>> + * | +----------------+     |        | |  CPER      |
>> + * | |ack_value1      |     |        | |  CPER      |
>> + * | +----------------+     |        | |  ....      |
>> + * | | .............  |     |        | |  CPER      |
>> + * | +----------------+     |        +-+------------+
>> + * | |ack_value10     |     |        | |..........  |
>> + * | +----------------+     |        | +------------+
>> + *                          |        | Error Status |
>> + *                          |        | Data Block10 |
>> + *                          +------->+------------+
>> + *                                   | |  CPER      |
>> + *                                   | |  CPER      |
>> + *                                   | |  ....      |
>> + *                                   | |  CPER      |
>> + *                                   +-+------------+
>> + */
>> +struct hardware_errors_buffer {
>> +    /* Generic Error Status Block register */
>> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +};
>> +
>> +static int ghes_record_cper(uint64_t error_block_address,
>> +                                    uint64_t error_physical_addr)
>> +{
>> +    AcpiGenericErrorStatus block;
>> +    AcpiGenericErrorData *gdata;
>> +    UefiCperSecMemErr *mem_err;
>> +    uint64_t current_block_length;
>> +    unsigned char *buffer;
>> +    /* memory section */
>> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
>> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
>> +                              0x83, 0xB1};
>> +
>> +    cpu_physical_memory_read(error_block_address, &block,
>> +                                sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Get the current generic error status block length */
>> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
>> +        le32_to_cpu(block.data_length);
>> +
>> +    /* If the Generic Error Status Block is NULL, update
>> +     * the block header
>> +     */
>> +    if (!block.block_status) {
>> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
>> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    }
>> +
>> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
>> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    /* check whether it runs out of the preallocated memory */
>> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
>> +       GHES_MAX_RAW_DATA_LENGTH) {
>> +        error_report("Record CPER out of boundary!!!");
>> +        return GHES_CPER_FAIL;
>> +    }
>> +
>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
>> +        /* Build read ack register */
>> +        build_address(table_data, linker, read_ack_register_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2),
>> +            offsetof(struct hardware_errors_buffer, ack_value) +
>> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
>> +            4 /* QWord access */);
>> +
>> +        /* Read ack preserve */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
>> +
>> +        /* Read ack write */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
>> +    }
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Patch address of generic error status block into
>> +         * the address register so OSPM can retrieve and read it.
>> +         */
>> +        bios_linker_loader_add_pointer(linker,
>> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
>> +            GHES_ERRORS_FW_CFG_FILE,
>> +            offsetof(struct hardware_errors_buffer, gesb) +
>> +            i * GHES_MAX_RAW_DATA_LENGTH);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
>> +     * so QEMU can write the ERRORS there. The address is expected to be
>> +     * < 4GB, but write 64 bits anyway.
>> +     */
>> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
>> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
>> +        offsetof(struct hardware_errors_buffer, gesb));
>> +
>> +    build_header(linker, table_data,
>> +        (void *)(table_data->data + ghes_start), "HEST",
>> +        table_data->len - ghes_start, 1, NULL, "GHES");
>> +}
>> +
>> +static GhesState ges;
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
>> +{
>> +
>> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
>> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
>> +
>> +    /* Create a read-only fw_cfg file for GHES */
>> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
>> +                    size);
>> +    /* Create a read-write fw_cfg file for Address */
>> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
>> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
>> +}
>> +
>> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
>> +{
>> +    uint64_t error_block_addr;
>> +    uint64_t ack_value_addr, ack_value = 0;
>> +    int loop = 0, ack_value_size;
>> +    bool ret = GHES_CPER_FAIL;
>> +
>> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
>> +        offsetof(struct hardware_errors_buffer, ack_value)) /
>> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
>> +
>> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
>> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
>> +        error_block_addr = le32_to_cpu(error_block_addr);
>> +
>> +        ack_value_addr = ges.ghes_addr_le -
>> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
>> +retry:
>> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
>> +        if (!ack_value) {
>> +            if (loop < 3) {
>> +                usleep(100 * 1000);
>> +                loop++;
>> +                goto retry;
>> +            } else {
>> +                error_report("Last time OSPM does not acknowledge the error,"
>> +                    " record CPER failed this time, set the ack value to"
>> +                    " avoid blocking next time CPER record! exit");
>> +                ack_value = 1;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                return ret;
>> +            }
>> +        } else {
>> +            /* A zero value in ghes_addr means that BIOS has not yet written
>> +             * the address
>> +             */
>> +            if (error_block_addr) {
>> +                ack_value = 0;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                ret = ghes_record_cper(error_block_addr, physical_address);
>> +            }
>> +        }
>> +    }
>> +    return ret;
>> +}
>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>> index 3d78ff6..def1ec1 100644
>> --- a/hw/arm/virt-acpi-build.c
>> +++ b/hw/arm/virt-acpi-build.c
>> @@ -45,6 +45,7 @@
>>  #include "hw/arm/virt.h"
>>  #include "sysemu/numa.h"
>>  #include "kvm_arm.h"
>> +#include "hw/acpi/hest_ghes.h"
>>  
>>  #define ARM_SPI_BASE 32
>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>      acpi_add_table(table_offsets, tables_blob);
>>      build_spcr(tables_blob, tables->linker, vms);
>>  
>> +    acpi_add_table(table_offsets, tables_blob);
>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>> +
> So we add this table unconditionally. Is there any bad impact if QEMU
> runs on old kvm? Does it need to check whether KVM supports RAS?
this table is added before guest OS boot. so can not use KVM to check it.
if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
May be we can make it as device? if this device is enabled in the qemu
boot parameters, then we will add this table?


> 
>>      if (nb_numa_nodes > 0) {
>>          acpi_add_table(table_offsets, tables_blob);
>>          build_srat(tables_blob, tables->linker, vms);
>> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>>                      acpi_data_len(tables.tcpalog));
>>  
>> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
>> +
>>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>>                                                ACPI_BUILD_RSDP_FILE, 0);
>>  
>> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
>> index 88d0738..7f7b55c 100644
>> --- a/include/hw/acpi/aml-build.h
>> +++ b/include/hw/acpi/aml-build.h
>> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>>      GArray *rsdp;
>>      GArray *tcpalog;
>>      GArray *vmgenid;
>> +    GArray *hardware_errors;
>>      BIOSLinker *linker;
>>  } AcpiBuildTables;
>>  
>> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
>> new file mode 100644
>> index 0000000..0772756
>> --- /dev/null
>> +++ b/include/hw/acpi/hest_ghes.h
>> @@ -0,0 +1,47 @@
>> +/*
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> +
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * Authors:
>> + *   Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * You should have received a copy of the GNU General Public License along
>> + * with this program; if not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#ifndef ACPI_GHES_H
>> +#define ACPI_GHES_H
>> +
>> +#include "hw/acpi/bios-linker-loader.h"
>> +
>> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
>> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
>> +
>> +#define GHES_GAS_ADDRESS_OFFSET              4
>> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
>> +#define GHES_NOTIFICATION_STRUCTURE          32
>> +
>> +#define GHES_CPER_OK   1
>> +#define GHES_CPER_FAIL 0
>> +
>> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
>> +/* The max size in Bytes for one error block */
>> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
>> +
>> +
>> +typedef struct GhesState {
>> +    uint64_t ghes_addr_le;
>> +} GhesState;
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                            BIOSLinker *linker);
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
>> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
>> +#endif
>>
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-25 11:20       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 11:20 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10

Hi Shannon,

On 2017/8/24 21:03, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
>>
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
>>
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  hw/acpi/aml-build.c         |   2 +
>>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>>  hw/arm/virt-acpi-build.c    |   6 +
>>  include/hw/acpi/aml-build.h |   1 +
>>  include/hw/acpi/hest_ghes.h |  47 ++++++
>>  5 files changed, 401 insertions(+)
>>  create mode 100644 hw/acpi/hest_ghes.c
> Don't need to add the new file to hw/acpi/Makefile.objs?
  I modified the Makefile.objs in another patch.

> 
>>  create mode 100644 include/hw/acpi/hest_ghes.h
>>
>> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
>> index 36a6cc4..6849e5f 100644
>> --- a/hw/acpi/aml-build.c
>> +++ b/hw/acpi/aml-build.c
>> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>>      tables->table_data = g_array_new(false, true /* clear */, 1);
>>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
>> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>>      tables->linker = bios_linker_loader_init();
>>  }
>>  
>> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>>      g_array_free(tables->table_data, true);
>>      g_array_free(tables->tcpalog, mfre);
>>      g_array_free(tables->vmgenid, mfre);
>> +    g_array_free(tables->hardware_errors, mfre);
>>  }
>>  
>>  /* Build rsdt table */
>> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
>> new file mode 100644
>> index 0000000..ff6b5ef
>> --- /dev/null
>> +++ b/hw/acpi/hest_ghes.c
>> @@ -0,0 +1,345 @@
>> +/*
>> + *  APEI GHES table Generation
>> + *
>> + *  Copyright (C) 2017 huawei.
>> + *
>> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + *
>> + */
> Please unify this of this file and hest_ghes.h by refering to other files.
Ok, thanks.


> 
>> +
>> +#include "qemu/osdep.h"
>> +#include "qmp-commands.h"
> unnecessary including
I will remove it.

> 
>> +#include "hw/acpi/acpi.h"
>> +#include "hw/acpi/aml-build.h"
>> +#include "hw/acpi/hest_ghes.h"
>> +#include "hw/nvram/fw_cfg.h"
>> +#include "sysemu/sysemu.h"
>> +#include "qemu/error-report.h"
>> +
>> +/* The structure that stands for the layout
>> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
>> + *
>> + *           etc/hardware_errors
>> + * ==========================================
>> + * +------------------+
>> + * |    address       |              +--------------+
>> + * |    registers     |              | Error Status |
>> + * | +----------------+              | Data Block 0 |
>> + * | |status_address0 |------------->| +------------+
>> + * | +----------------+              | |  CPER      |
>> + * | |status_address1 |----------+   | |  CPER      |
>> + * | +----------------+          |   | |  ....      |
>> + * | |.............   |          |   | |  CPER      |
>> + * | +----------------+          |   | +------------+
>> + * | |status_address10|-----+    |   | Error Status |
>> + * | +----------------+     |    |   | Data Block 1 |
>> + * | |ack_value0      |     |    +-->| +------------+
>> + * | +----------------+     |        | |  CPER      |
>> + * | |ack_value1      |     |        | |  CPER      |
>> + * | +----------------+     |        | |  ....      |
>> + * | | .............  |     |        | |  CPER      |
>> + * | +----------------+     |        +-+------------+
>> + * | |ack_value10     |     |        | |..........  |
>> + * | +----------------+     |        | +------------+
>> + *                          |        | Error Status |
>> + *                          |        | Data Block10 |
>> + *                          +------->+------------+
>> + *                                   | |  CPER      |
>> + *                                   | |  CPER      |
>> + *                                   | |  ....      |
>> + *                                   | |  CPER      |
>> + *                                   +-+------------+
>> + */
>> +struct hardware_errors_buffer {
>> +    /* Generic Error Status Block register */
>> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +};
>> +
>> +static int ghes_record_cper(uint64_t error_block_address,
>> +                                    uint64_t error_physical_addr)
>> +{
>> +    AcpiGenericErrorStatus block;
>> +    AcpiGenericErrorData *gdata;
>> +    UefiCperSecMemErr *mem_err;
>> +    uint64_t current_block_length;
>> +    unsigned char *buffer;
>> +    /* memory section */
>> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
>> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
>> +                              0x83, 0xB1};
>> +
>> +    cpu_physical_memory_read(error_block_address, &block,
>> +                                sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Get the current generic error status block length */
>> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
>> +        le32_to_cpu(block.data_length);
>> +
>> +    /* If the Generic Error Status Block is NULL, update
>> +     * the block header
>> +     */
>> +    if (!block.block_status) {
>> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
>> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    }
>> +
>> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
>> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    /* check whether it runs out of the preallocated memory */
>> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
>> +       GHES_MAX_RAW_DATA_LENGTH) {
>> +        error_report("Record CPER out of boundary!!!");
>> +        return GHES_CPER_FAIL;
>> +    }
>> +
>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
>> +        /* Build read ack register */
>> +        build_address(table_data, linker, read_ack_register_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2),
>> +            offsetof(struct hardware_errors_buffer, ack_value) +
>> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
>> +            4 /* QWord access */);
>> +
>> +        /* Read ack preserve */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
>> +
>> +        /* Read ack write */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
>> +    }
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Patch address of generic error status block into
>> +         * the address register so OSPM can retrieve and read it.
>> +         */
>> +        bios_linker_loader_add_pointer(linker,
>> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
>> +            GHES_ERRORS_FW_CFG_FILE,
>> +            offsetof(struct hardware_errors_buffer, gesb) +
>> +            i * GHES_MAX_RAW_DATA_LENGTH);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
>> +     * so QEMU can write the ERRORS there. The address is expected to be
>> +     * < 4GB, but write 64 bits anyway.
>> +     */
>> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
>> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
>> +        offsetof(struct hardware_errors_buffer, gesb));
>> +
>> +    build_header(linker, table_data,
>> +        (void *)(table_data->data + ghes_start), "HEST",
>> +        table_data->len - ghes_start, 1, NULL, "GHES");
>> +}
>> +
>> +static GhesState ges;
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
>> +{
>> +
>> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
>> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
>> +
>> +    /* Create a read-only fw_cfg file for GHES */
>> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
>> +                    size);
>> +    /* Create a read-write fw_cfg file for Address */
>> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
>> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
>> +}
>> +
>> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
>> +{
>> +    uint64_t error_block_addr;
>> +    uint64_t ack_value_addr, ack_value = 0;
>> +    int loop = 0, ack_value_size;
>> +    bool ret = GHES_CPER_FAIL;
>> +
>> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
>> +        offsetof(struct hardware_errors_buffer, ack_value)) /
>> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
>> +
>> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
>> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
>> +        error_block_addr = le32_to_cpu(error_block_addr);
>> +
>> +        ack_value_addr = ges.ghes_addr_le -
>> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
>> +retry:
>> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
>> +        if (!ack_value) {
>> +            if (loop < 3) {
>> +                usleep(100 * 1000);
>> +                loop++;
>> +                goto retry;
>> +            } else {
>> +                error_report("Last time OSPM does not acknowledge the error,"
>> +                    " record CPER failed this time, set the ack value to"
>> +                    " avoid blocking next time CPER record! exit");
>> +                ack_value = 1;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                return ret;
>> +            }
>> +        } else {
>> +            /* A zero value in ghes_addr means that BIOS has not yet written
>> +             * the address
>> +             */
>> +            if (error_block_addr) {
>> +                ack_value = 0;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                ret = ghes_record_cper(error_block_addr, physical_address);
>> +            }
>> +        }
>> +    }
>> +    return ret;
>> +}
>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>> index 3d78ff6..def1ec1 100644
>> --- a/hw/arm/virt-acpi-build.c
>> +++ b/hw/arm/virt-acpi-build.c
>> @@ -45,6 +45,7 @@
>>  #include "hw/arm/virt.h"
>>  #include "sysemu/numa.h"
>>  #include "kvm_arm.h"
>> +#include "hw/acpi/hest_ghes.h"
>>  
>>  #define ARM_SPI_BASE 32
>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>      acpi_add_table(table_offsets, tables_blob);
>>      build_spcr(tables_blob, tables->linker, vms);
>>  
>> +    acpi_add_table(table_offsets, tables_blob);
>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>> +
> So we add this table unconditionally. Is there any bad impact if QEMU
> runs on old kvm? Does it need to check whether KVM supports RAS?
this table is added before guest OS boot. so can not use KVM to check it.
if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
May be we can make it as device? if this device is enabled in the qemu
boot parameters, then we will add this table?


> 
>>      if (nb_numa_nodes > 0) {
>>          acpi_add_table(table_offsets, tables_blob);
>>          build_srat(tables_blob, tables->linker, vms);
>> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>>                      acpi_data_len(tables.tcpalog));
>>  
>> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
>> +
>>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>>                                                ACPI_BUILD_RSDP_FILE, 0);
>>  
>> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
>> index 88d0738..7f7b55c 100644
>> --- a/include/hw/acpi/aml-build.h
>> +++ b/include/hw/acpi/aml-build.h
>> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>>      GArray *rsdp;
>>      GArray *tcpalog;
>>      GArray *vmgenid;
>> +    GArray *hardware_errors;
>>      BIOSLinker *linker;
>>  } AcpiBuildTables;
>>  
>> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
>> new file mode 100644
>> index 0000000..0772756
>> --- /dev/null
>> +++ b/include/hw/acpi/hest_ghes.h
>> @@ -0,0 +1,47 @@
>> +/*
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> +
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * Authors:
>> + *   Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * You should have received a copy of the GNU General Public License along
>> + * with this program; if not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#ifndef ACPI_GHES_H
>> +#define ACPI_GHES_H
>> +
>> +#include "hw/acpi/bios-linker-loader.h"
>> +
>> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
>> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
>> +
>> +#define GHES_GAS_ADDRESS_OFFSET              4
>> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
>> +#define GHES_NOTIFICATION_STRUCTURE          32
>> +
>> +#define GHES_CPER_OK   1
>> +#define GHES_CPER_FAIL 0
>> +
>> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
>> +/* The max size in Bytes for one error block */
>> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
>> +
>> +
>> +typedef struct GhesState {
>> +    uint64_t ghes_addr_le;
>> +} GhesState;
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                            BIOSLinker *linker);
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
>> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
>> +#endif
>>
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-25 11:20       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 11:20 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10

Hi Shannon,

On 2017/8/24 21:03, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
>>
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
>>
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  hw/acpi/aml-build.c         |   2 +
>>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>>  hw/arm/virt-acpi-build.c    |   6 +
>>  include/hw/acpi/aml-build.h |   1 +
>>  include/hw/acpi/hest_ghes.h |  47 ++++++
>>  5 files changed, 401 insertions(+)
>>  create mode 100644 hw/acpi/hest_ghes.c
> Don't need to add the new file to hw/acpi/Makefile.objs?
  I modified the Makefile.objs in another patch.

> 
>>  create mode 100644 include/hw/acpi/hest_ghes.h
>>
>> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
>> index 36a6cc4..6849e5f 100644
>> --- a/hw/acpi/aml-build.c
>> +++ b/hw/acpi/aml-build.c
>> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>>      tables->table_data = g_array_new(false, true /* clear */, 1);
>>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
>> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>>      tables->linker = bios_linker_loader_init();
>>  }
>>  
>> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>>      g_array_free(tables->table_data, true);
>>      g_array_free(tables->tcpalog, mfre);
>>      g_array_free(tables->vmgenid, mfre);
>> +    g_array_free(tables->hardware_errors, mfre);
>>  }
>>  
>>  /* Build rsdt table */
>> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
>> new file mode 100644
>> index 0000000..ff6b5ef
>> --- /dev/null
>> +++ b/hw/acpi/hest_ghes.c
>> @@ -0,0 +1,345 @@
>> +/*
>> + *  APEI GHES table Generation
>> + *
>> + *  Copyright (C) 2017 huawei.
>> + *
>> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + *
>> + */
> Please unify this of this file and hest_ghes.h by refering to other files.
Ok, thanks.


> 
>> +
>> +#include "qemu/osdep.h"
>> +#include "qmp-commands.h"
> unnecessary including
I will remove it.

> 
>> +#include "hw/acpi/acpi.h"
>> +#include "hw/acpi/aml-build.h"
>> +#include "hw/acpi/hest_ghes.h"
>> +#include "hw/nvram/fw_cfg.h"
>> +#include "sysemu/sysemu.h"
>> +#include "qemu/error-report.h"
>> +
>> +/* The structure that stands for the layout
>> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
>> + *
>> + *           etc/hardware_errors
>> + * ==========================================
>> + * +------------------+
>> + * |    address       |              +--------------+
>> + * |    registers     |              | Error Status |
>> + * | +----------------+              | Data Block 0 |
>> + * | |status_address0 |------------->| +------------+
>> + * | +----------------+              | |  CPER      |
>> + * | |status_address1 |----------+   | |  CPER      |
>> + * | +----------------+          |   | |  ....      |
>> + * | |.............   |          |   | |  CPER      |
>> + * | +----------------+          |   | +------------+
>> + * | |status_address10|-----+    |   | Error Status |
>> + * | +----------------+     |    |   | Data Block 1 |
>> + * | |ack_value0      |     |    +-->| +------------+
>> + * | +----------------+     |        | |  CPER      |
>> + * | |ack_value1      |     |        | |  CPER      |
>> + * | +----------------+     |        | |  ....      |
>> + * | | .............  |     |        | |  CPER      |
>> + * | +----------------+     |        +-+------------+
>> + * | |ack_value10     |     |        | |..........  |
>> + * | +----------------+     |        | +------------+
>> + *                          |        | Error Status |
>> + *                          |        | Data Block10 |
>> + *                          +------->+------------+
>> + *                                   | |  CPER      |
>> + *                                   | |  CPER      |
>> + *                                   | |  ....      |
>> + *                                   | |  CPER      |
>> + *                                   +-+------------+
>> + */
>> +struct hardware_errors_buffer {
>> +    /* Generic Error Status Block register */
>> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +};
>> +
>> +static int ghes_record_cper(uint64_t error_block_address,
>> +                                    uint64_t error_physical_addr)
>> +{
>> +    AcpiGenericErrorStatus block;
>> +    AcpiGenericErrorData *gdata;
>> +    UefiCperSecMemErr *mem_err;
>> +    uint64_t current_block_length;
>> +    unsigned char *buffer;
>> +    /* memory section */
>> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
>> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
>> +                              0x83, 0xB1};
>> +
>> +    cpu_physical_memory_read(error_block_address, &block,
>> +                                sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Get the current generic error status block length */
>> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
>> +        le32_to_cpu(block.data_length);
>> +
>> +    /* If the Generic Error Status Block is NULL, update
>> +     * the block header
>> +     */
>> +    if (!block.block_status) {
>> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
>> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    }
>> +
>> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
>> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    /* check whether it runs out of the preallocated memory */
>> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
>> +       GHES_MAX_RAW_DATA_LENGTH) {
>> +        error_report("Record CPER out of boundary!!!");
>> +        return GHES_CPER_FAIL;
>> +    }
>> +
>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
>> +        /* Build read ack register */
>> +        build_address(table_data, linker, read_ack_register_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2),
>> +            offsetof(struct hardware_errors_buffer, ack_value) +
>> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
>> +            4 /* QWord access */);
>> +
>> +        /* Read ack preserve */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
>> +
>> +        /* Read ack write */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
>> +    }
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Patch address of generic error status block into
>> +         * the address register so OSPM can retrieve and read it.
>> +         */
>> +        bios_linker_loader_add_pointer(linker,
>> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
>> +            GHES_ERRORS_FW_CFG_FILE,
>> +            offsetof(struct hardware_errors_buffer, gesb) +
>> +            i * GHES_MAX_RAW_DATA_LENGTH);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
>> +     * so QEMU can write the ERRORS there. The address is expected to be
>> +     * < 4GB, but write 64 bits anyway.
>> +     */
>> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
>> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
>> +        offsetof(struct hardware_errors_buffer, gesb));
>> +
>> +    build_header(linker, table_data,
>> +        (void *)(table_data->data + ghes_start), "HEST",
>> +        table_data->len - ghes_start, 1, NULL, "GHES");
>> +}
>> +
>> +static GhesState ges;
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
>> +{
>> +
>> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
>> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
>> +
>> +    /* Create a read-only fw_cfg file for GHES */
>> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
>> +                    size);
>> +    /* Create a read-write fw_cfg file for Address */
>> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
>> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
>> +}
>> +
>> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
>> +{
>> +    uint64_t error_block_addr;
>> +    uint64_t ack_value_addr, ack_value = 0;
>> +    int loop = 0, ack_value_size;
>> +    bool ret = GHES_CPER_FAIL;
>> +
>> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
>> +        offsetof(struct hardware_errors_buffer, ack_value)) /
>> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
>> +
>> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
>> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
>> +        error_block_addr = le32_to_cpu(error_block_addr);
>> +
>> +        ack_value_addr = ges.ghes_addr_le -
>> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
>> +retry:
>> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
>> +        if (!ack_value) {
>> +            if (loop < 3) {
>> +                usleep(100 * 1000);
>> +                loop++;
>> +                goto retry;
>> +            } else {
>> +                error_report("Last time OSPM does not acknowledge the error,"
>> +                    " record CPER failed this time, set the ack value to"
>> +                    " avoid blocking next time CPER record! exit");
>> +                ack_value = 1;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                return ret;
>> +            }
>> +        } else {
>> +            /* A zero value in ghes_addr means that BIOS has not yet written
>> +             * the address
>> +             */
>> +            if (error_block_addr) {
>> +                ack_value = 0;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                ret = ghes_record_cper(error_block_addr, physical_address);
>> +            }
>> +        }
>> +    }
>> +    return ret;
>> +}
>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>> index 3d78ff6..def1ec1 100644
>> --- a/hw/arm/virt-acpi-build.c
>> +++ b/hw/arm/virt-acpi-build.c
>> @@ -45,6 +45,7 @@
>>  #include "hw/arm/virt.h"
>>  #include "sysemu/numa.h"
>>  #include "kvm_arm.h"
>> +#include "hw/acpi/hest_ghes.h"
>>  
>>  #define ARM_SPI_BASE 32
>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>      acpi_add_table(table_offsets, tables_blob);
>>      build_spcr(tables_blob, tables->linker, vms);
>>  
>> +    acpi_add_table(table_offsets, tables_blob);
>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>> +
> So we add this table unconditionally. Is there any bad impact if QEMU
> runs on old kvm? Does it need to check whether KVM supports RAS?
this table is added before guest OS boot. so can not use KVM to check it.
if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
May be we can make it as device? if this device is enabled in the qemu
boot parameters, then we will add this table?


> 
>>      if (nb_numa_nodes > 0) {
>>          acpi_add_table(table_offsets, tables_blob);
>>          build_srat(tables_blob, tables->linker, vms);
>> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>>                      acpi_data_len(tables.tcpalog));
>>  
>> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
>> +
>>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>>                                                ACPI_BUILD_RSDP_FILE, 0);
>>  
>> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
>> index 88d0738..7f7b55c 100644
>> --- a/include/hw/acpi/aml-build.h
>> +++ b/include/hw/acpi/aml-build.h
>> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>>      GArray *rsdp;
>>      GArray *tcpalog;
>>      GArray *vmgenid;
>> +    GArray *hardware_errors;
>>      BIOSLinker *linker;
>>  } AcpiBuildTables;
>>  
>> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
>> new file mode 100644
>> index 0000000..0772756
>> --- /dev/null
>> +++ b/include/hw/acpi/hest_ghes.h
>> @@ -0,0 +1,47 @@
>> +/*
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> +
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * Authors:
>> + *   Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * You should have received a copy of the GNU General Public License along
>> + * with this program; if not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#ifndef ACPI_GHES_H
>> +#define ACPI_GHES_H
>> +
>> +#include "hw/acpi/bios-linker-loader.h"
>> +
>> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
>> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
>> +
>> +#define GHES_GAS_ADDRESS_OFFSET              4
>> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
>> +#define GHES_NOTIFICATION_STRUCTURE          32
>> +
>> +#define GHES_CPER_OK   1
>> +#define GHES_CPER_FAIL 0
>> +
>> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
>> +/* The max size in Bytes for one error block */
>> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
>> +
>> +
>> +typedef struct GhesState {
>> +    uint64_t ghes_addr_le;
>> +} GhesState;
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                            BIOSLinker *linker);
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
>> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
>> +#endif
>>
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-25 11:20       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 11:20 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10

Hi Shannon,

On 2017/8/24 21:03, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
>>
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
>>
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  hw/acpi/aml-build.c         |   2 +
>>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>>  hw/arm/virt-acpi-build.c    |   6 +
>>  include/hw/acpi/aml-build.h |   1 +
>>  include/hw/acpi/hest_ghes.h |  47 ++++++
>>  5 files changed, 401 insertions(+)
>>  create mode 100644 hw/acpi/hest_ghes.c
> Don't need to add the new file to hw/acpi/Makefile.objs?
  I modified the Makefile.objs in another patch.

> 
>>  create mode 100644 include/hw/acpi/hest_ghes.h
>>
>> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
>> index 36a6cc4..6849e5f 100644
>> --- a/hw/acpi/aml-build.c
>> +++ b/hw/acpi/aml-build.c
>> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>>      tables->table_data = g_array_new(false, true /* clear */, 1);
>>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
>> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>>      tables->linker = bios_linker_loader_init();
>>  }
>>  
>> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>>      g_array_free(tables->table_data, true);
>>      g_array_free(tables->tcpalog, mfre);
>>      g_array_free(tables->vmgenid, mfre);
>> +    g_array_free(tables->hardware_errors, mfre);
>>  }
>>  
>>  /* Build rsdt table */
>> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
>> new file mode 100644
>> index 0000000..ff6b5ef
>> --- /dev/null
>> +++ b/hw/acpi/hest_ghes.c
>> @@ -0,0 +1,345 @@
>> +/*
>> + *  APEI GHES table Generation
>> + *
>> + *  Copyright (C) 2017 huawei.
>> + *
>> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + *
>> + */
> Please unify this of this file and hest_ghes.h by refering to other files.
Ok, thanks.


> 
>> +
>> +#include "qemu/osdep.h"
>> +#include "qmp-commands.h"
> unnecessary including
I will remove it.

> 
>> +#include "hw/acpi/acpi.h"
>> +#include "hw/acpi/aml-build.h"
>> +#include "hw/acpi/hest_ghes.h"
>> +#include "hw/nvram/fw_cfg.h"
>> +#include "sysemu/sysemu.h"
>> +#include "qemu/error-report.h"
>> +
>> +/* The structure that stands for the layout
>> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
>> + *
>> + *           etc/hardware_errors
>> + * ==========================================
>> + * +------------------+
>> + * |    address       |              +--------------+
>> + * |    registers     |              | Error Status |
>> + * | +----------------+              | Data Block 0 |
>> + * | |status_address0 |------------->| +------------+
>> + * | +----------------+              | |  CPER      |
>> + * | |status_address1 |----------+   | |  CPER      |
>> + * | +----------------+          |   | |  ....      |
>> + * | |.............   |          |   | |  CPER      |
>> + * | +----------------+          |   | +------------+
>> + * | |status_address10|-----+    |   | Error Status |
>> + * | +----------------+     |    |   | Data Block 1 |
>> + * | |ack_value0      |     |    +-->| +------------+
>> + * | +----------------+     |        | |  CPER      |
>> + * | |ack_value1      |     |        | |  CPER      |
>> + * | +----------------+     |        | |  ....      |
>> + * | | .............  |     |        | |  CPER      |
>> + * | +----------------+     |        +-+------------+
>> + * | |ack_value10     |     |        | |..........  |
>> + * | +----------------+     |        | +------------+
>> + *                          |        | Error Status |
>> + *                          |        | Data Block10 |
>> + *                          +------->+------------+
>> + *                                   | |  CPER      |
>> + *                                   | |  CPER      |
>> + *                                   | |  ....      |
>> + *                                   | |  CPER      |
>> + *                                   +-+------------+
>> + */
>> +struct hardware_errors_buffer {
>> +    /* Generic Error Status Block register */
>> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +};
>> +
>> +static int ghes_record_cper(uint64_t error_block_address,
>> +                                    uint64_t error_physical_addr)
>> +{
>> +    AcpiGenericErrorStatus block;
>> +    AcpiGenericErrorData *gdata;
>> +    UefiCperSecMemErr *mem_err;
>> +    uint64_t current_block_length;
>> +    unsigned char *buffer;
>> +    /* memory section */
>> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
>> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
>> +                              0x83, 0xB1};
>> +
>> +    cpu_physical_memory_read(error_block_address, &block,
>> +                                sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Get the current generic error status block length */
>> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
>> +        le32_to_cpu(block.data_length);
>> +
>> +    /* If the Generic Error Status Block is NULL, update
>> +     * the block header
>> +     */
>> +    if (!block.block_status) {
>> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
>> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    }
>> +
>> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
>> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    /* check whether it runs out of the preallocated memory */
>> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
>> +       GHES_MAX_RAW_DATA_LENGTH) {
>> +        error_report("Record CPER out of boundary!!!");
>> +        return GHES_CPER_FAIL;
>> +    }
>> +
>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
>> +        /* Build read ack register */
>> +        build_address(table_data, linker, read_ack_register_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2),
>> +            offsetof(struct hardware_errors_buffer, ack_value) +
>> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
>> +            4 /* QWord access */);
>> +
>> +        /* Read ack preserve */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
>> +
>> +        /* Read ack write */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
>> +    }
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Patch address of generic error status block into
>> +         * the address register so OSPM can retrieve and read it.
>> +         */
>> +        bios_linker_loader_add_pointer(linker,
>> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
>> +            GHES_ERRORS_FW_CFG_FILE,
>> +            offsetof(struct hardware_errors_buffer, gesb) +
>> +            i * GHES_MAX_RAW_DATA_LENGTH);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
>> +     * so QEMU can write the ERRORS there. The address is expected to be
>> +     * < 4GB, but write 64 bits anyway.
>> +     */
>> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
>> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
>> +        offsetof(struct hardware_errors_buffer, gesb));
>> +
>> +    build_header(linker, table_data,
>> +        (void *)(table_data->data + ghes_start), "HEST",
>> +        table_data->len - ghes_start, 1, NULL, "GHES");
>> +}
>> +
>> +static GhesState ges;
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
>> +{
>> +
>> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
>> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
>> +
>> +    /* Create a read-only fw_cfg file for GHES */
>> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
>> +                    size);
>> +    /* Create a read-write fw_cfg file for Address */
>> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
>> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
>> +}
>> +
>> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
>> +{
>> +    uint64_t error_block_addr;
>> +    uint64_t ack_value_addr, ack_value = 0;
>> +    int loop = 0, ack_value_size;
>> +    bool ret = GHES_CPER_FAIL;
>> +
>> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
>> +        offsetof(struct hardware_errors_buffer, ack_value)) /
>> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
>> +
>> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
>> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
>> +        error_block_addr = le32_to_cpu(error_block_addr);
>> +
>> +        ack_value_addr = ges.ghes_addr_le -
>> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
>> +retry:
>> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
>> +        if (!ack_value) {
>> +            if (loop < 3) {
>> +                usleep(100 * 1000);
>> +                loop++;
>> +                goto retry;
>> +            } else {
>> +                error_report("Last time OSPM does not acknowledge the error,"
>> +                    " record CPER failed this time, set the ack value to"
>> +                    " avoid blocking next time CPER record! exit");
>> +                ack_value = 1;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                return ret;
>> +            }
>> +        } else {
>> +            /* A zero value in ghes_addr means that BIOS has not yet written
>> +             * the address
>> +             */
>> +            if (error_block_addr) {
>> +                ack_value = 0;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                ret = ghes_record_cper(error_block_addr, physical_address);
>> +            }
>> +        }
>> +    }
>> +    return ret;
>> +}
>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>> index 3d78ff6..def1ec1 100644
>> --- a/hw/arm/virt-acpi-build.c
>> +++ b/hw/arm/virt-acpi-build.c
>> @@ -45,6 +45,7 @@
>>  #include "hw/arm/virt.h"
>>  #include "sysemu/numa.h"
>>  #include "kvm_arm.h"
>> +#include "hw/acpi/hest_ghes.h"
>>  
>>  #define ARM_SPI_BASE 32
>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>      acpi_add_table(table_offsets, tables_blob);
>>      build_spcr(tables_blob, tables->linker, vms);
>>  
>> +    acpi_add_table(table_offsets, tables_blob);
>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>> +
> So we add this table unconditionally. Is there any bad impact if QEMU
> runs on old kvm? Does it need to check whether KVM supports RAS?
this table is added before guest OS boot. so can not use KVM to check it.
if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
May be we can make it as device? if this device is enabled in the qemu
boot parameters, then we will add this table?


> 
>>      if (nb_numa_nodes > 0) {
>>          acpi_add_table(table_offsets, tables_blob);
>>          build_srat(tables_blob, tables->linker, vms);
>> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>>                      acpi_data_len(tables.tcpalog));
>>  
>> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
>> +
>>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>>                                                ACPI_BUILD_RSDP_FILE, 0);
>>  
>> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
>> index 88d0738..7f7b55c 100644
>> --- a/include/hw/acpi/aml-build.h
>> +++ b/include/hw/acpi/aml-build.h
>> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>>      GArray *rsdp;
>>      GArray *tcpalog;
>>      GArray *vmgenid;
>> +    GArray *hardware_errors;
>>      BIOSLinker *linker;
>>  } AcpiBuildTables;
>>  
>> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
>> new file mode 100644
>> index 0000000..0772756
>> --- /dev/null
>> +++ b/include/hw/acpi/hest_ghes.h
>> @@ -0,0 +1,47 @@
>> +/*
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> +
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * Authors:
>> + *   Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * You should have received a copy of the GNU General Public License along
>> + * with this program; if not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#ifndef ACPI_GHES_H
>> +#define ACPI_GHES_H
>> +
>> +#include "hw/acpi/bios-linker-loader.h"
>> +
>> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
>> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
>> +
>> +#define GHES_GAS_ADDRESS_OFFSET              4
>> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
>> +#define GHES_NOTIFICATION_STRUCTURE          32
>> +
>> +#define GHES_CPER_OK   1
>> +#define GHES_CPER_FAIL 0
>> +
>> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
>> +/* The max size in Bytes for one error block */
>> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
>> +
>> +
>> +typedef struct GhesState {
>> +    uint64_t ghes_addr_le;
>> +} GhesState;
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                            BIOSLinker *linker);
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
>> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
>> +#endif
>>
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-25 11:20       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 11:20 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Shannon,

On 2017/8/24 21:03, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
>>
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
>>
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  hw/acpi/aml-build.c         |   2 +
>>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>>  hw/arm/virt-acpi-build.c    |   6 +
>>  include/hw/acpi/aml-build.h |   1 +
>>  include/hw/acpi/hest_ghes.h |  47 ++++++
>>  5 files changed, 401 insertions(+)
>>  create mode 100644 hw/acpi/hest_ghes.c
> Don't need to add the new file to hw/acpi/Makefile.objs?
  I modified the Makefile.objs in another patch.

> 
>>  create mode 100644 include/hw/acpi/hest_ghes.h
>>
>> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
>> index 36a6cc4..6849e5f 100644
>> --- a/hw/acpi/aml-build.c
>> +++ b/hw/acpi/aml-build.c
>> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>>      tables->table_data = g_array_new(false, true /* clear */, 1);
>>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
>> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>>      tables->linker = bios_linker_loader_init();
>>  }
>>  
>> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>>      g_array_free(tables->table_data, true);
>>      g_array_free(tables->tcpalog, mfre);
>>      g_array_free(tables->vmgenid, mfre);
>> +    g_array_free(tables->hardware_errors, mfre);
>>  }
>>  
>>  /* Build rsdt table */
>> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
>> new file mode 100644
>> index 0000000..ff6b5ef
>> --- /dev/null
>> +++ b/hw/acpi/hest_ghes.c
>> @@ -0,0 +1,345 @@
>> +/*
>> + *  APEI GHES table Generation
>> + *
>> + *  Copyright (C) 2017 huawei.
>> + *
>> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + *
>> + */
> Please unify this of this file and hest_ghes.h by refering to other files.
Ok, thanks.


> 
>> +
>> +#include "qemu/osdep.h"
>> +#include "qmp-commands.h"
> unnecessary including
I will remove it.

> 
>> +#include "hw/acpi/acpi.h"
>> +#include "hw/acpi/aml-build.h"
>> +#include "hw/acpi/hest_ghes.h"
>> +#include "hw/nvram/fw_cfg.h"
>> +#include "sysemu/sysemu.h"
>> +#include "qemu/error-report.h"
>> +
>> +/* The structure that stands for the layout
>> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
>> + *
>> + *           etc/hardware_errors
>> + * ==========================================
>> + * +------------------+
>> + * |    address       |              +--------------+
>> + * |    registers     |              | Error Status |
>> + * | +----------------+              | Data Block 0 |
>> + * | |status_address0 |------------->| +------------+
>> + * | +----------------+              | |  CPER      |
>> + * | |status_address1 |----------+   | |  CPER      |
>> + * | +----------------+          |   | |  ....      |
>> + * | |.............   |          |   | |  CPER      |
>> + * | +----------------+          |   | +------------+
>> + * | |status_address10|-----+    |   | Error Status |
>> + * | +----------------+     |    |   | Data Block 1 |
>> + * | |ack_value0      |     |    +-->| +------------+
>> + * | +----------------+     |        | |  CPER      |
>> + * | |ack_value1      |     |        | |  CPER      |
>> + * | +----------------+     |        | |  ....      |
>> + * | | .............  |     |        | |  CPER      |
>> + * | +----------------+     |        +-+------------+
>> + * | |ack_value10     |     |        | |..........  |
>> + * | +----------------+     |        | +------------+
>> + *                          |        | Error Status |
>> + *                          |        | Data Block10 |
>> + *                          +------->+------------+
>> + *                                   | |  CPER      |
>> + *                                   | |  CPER      |
>> + *                                   | |  ....      |
>> + *                                   | |  CPER      |
>> + *                                   +-+------------+
>> + */
>> +struct hardware_errors_buffer {
>> +    /* Generic Error Status Block register */
>> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +};
>> +
>> +static int ghes_record_cper(uint64_t error_block_address,
>> +                                    uint64_t error_physical_addr)
>> +{
>> +    AcpiGenericErrorStatus block;
>> +    AcpiGenericErrorData *gdata;
>> +    UefiCperSecMemErr *mem_err;
>> +    uint64_t current_block_length;
>> +    unsigned char *buffer;
>> +    /* memory section */
>> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
>> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
>> +                              0x83, 0xB1};
>> +
>> +    cpu_physical_memory_read(error_block_address, &block,
>> +                                sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Get the current generic error status block length */
>> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
>> +        le32_to_cpu(block.data_length);
>> +
>> +    /* If the Generic Error Status Block is NULL, update
>> +     * the block header
>> +     */
>> +    if (!block.block_status) {
>> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
>> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    }
>> +
>> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
>> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    /* check whether it runs out of the preallocated memory */
>> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
>> +       GHES_MAX_RAW_DATA_LENGTH) {
>> +        error_report("Record CPER out of boundary!!!");
>> +        return GHES_CPER_FAIL;
>> +    }
>> +
>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
>> +        /* Build read ack register */
>> +        build_address(table_data, linker, read_ack_register_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2),
>> +            offsetof(struct hardware_errors_buffer, ack_value) +
>> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
>> +            4 /* QWord access */);
>> +
>> +        /* Read ack preserve */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
>> +
>> +        /* Read ack write */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
>> +    }
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Patch address of generic error status block into
>> +         * the address register so OSPM can retrieve and read it.
>> +         */
>> +        bios_linker_loader_add_pointer(linker,
>> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
>> +            GHES_ERRORS_FW_CFG_FILE,
>> +            offsetof(struct hardware_errors_buffer, gesb) +
>> +            i * GHES_MAX_RAW_DATA_LENGTH);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
>> +     * so QEMU can write the ERRORS there. The address is expected to be
>> +     * < 4GB, but write 64 bits anyway.
>> +     */
>> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
>> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
>> +        offsetof(struct hardware_errors_buffer, gesb));
>> +
>> +    build_header(linker, table_data,
>> +        (void *)(table_data->data + ghes_start), "HEST",
>> +        table_data->len - ghes_start, 1, NULL, "GHES");
>> +}
>> +
>> +static GhesState ges;
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
>> +{
>> +
>> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
>> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
>> +
>> +    /* Create a read-only fw_cfg file for GHES */
>> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
>> +                    size);
>> +    /* Create a read-write fw_cfg file for Address */
>> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
>> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
>> +}
>> +
>> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
>> +{
>> +    uint64_t error_block_addr;
>> +    uint64_t ack_value_addr, ack_value = 0;
>> +    int loop = 0, ack_value_size;
>> +    bool ret = GHES_CPER_FAIL;
>> +
>> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
>> +        offsetof(struct hardware_errors_buffer, ack_value)) /
>> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
>> +
>> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
>> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
>> +        error_block_addr = le32_to_cpu(error_block_addr);
>> +
>> +        ack_value_addr = ges.ghes_addr_le -
>> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
>> +retry:
>> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
>> +        if (!ack_value) {
>> +            if (loop < 3) {
>> +                usleep(100 * 1000);
>> +                loop++;
>> +                goto retry;
>> +            } else {
>> +                error_report("Last time OSPM does not acknowledge the error,"
>> +                    " record CPER failed this time, set the ack value to"
>> +                    " avoid blocking next time CPER record! exit");
>> +                ack_value = 1;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                return ret;
>> +            }
>> +        } else {
>> +            /* A zero value in ghes_addr means that BIOS has not yet written
>> +             * the address
>> +             */
>> +            if (error_block_addr) {
>> +                ack_value = 0;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                ret = ghes_record_cper(error_block_addr, physical_address);
>> +            }
>> +        }
>> +    }
>> +    return ret;
>> +}
>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>> index 3d78ff6..def1ec1 100644
>> --- a/hw/arm/virt-acpi-build.c
>> +++ b/hw/arm/virt-acpi-build.c
>> @@ -45,6 +45,7 @@
>>  #include "hw/arm/virt.h"
>>  #include "sysemu/numa.h"
>>  #include "kvm_arm.h"
>> +#include "hw/acpi/hest_ghes.h"
>>  
>>  #define ARM_SPI_BASE 32
>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>      acpi_add_table(table_offsets, tables_blob);
>>      build_spcr(tables_blob, tables->linker, vms);
>>  
>> +    acpi_add_table(table_offsets, tables_blob);
>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>> +
> So we add this table unconditionally. Is there any bad impact if QEMU
> runs on old kvm? Does it need to check whether KVM supports RAS?
this table is added before guest OS boot. so can not use KVM to check it.
if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
May be we can make it as device? if this device is enabled in the qemu
boot parameters, then we will add this table?


> 
>>      if (nb_numa_nodes > 0) {
>>          acpi_add_table(table_offsets, tables_blob);
>>          build_srat(tables_blob, tables->linker, vms);
>> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>>                      acpi_data_len(tables.tcpalog));
>>  
>> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
>> +
>>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>>                                                ACPI_BUILD_RSDP_FILE, 0);
>>  
>> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
>> index 88d0738..7f7b55c 100644
>> --- a/include/hw/acpi/aml-build.h
>> +++ b/include/hw/acpi/aml-build.h
>> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>>      GArray *rsdp;
>>      GArray *tcpalog;
>>      GArray *vmgenid;
>> +    GArray *hardware_errors;
>>      BIOSLinker *linker;
>>  } AcpiBuildTables;
>>  
>> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
>> new file mode 100644
>> index 0000000..0772756
>> --- /dev/null
>> +++ b/include/hw/acpi/hest_ghes.h
>> @@ -0,0 +1,47 @@
>> +/*
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> +
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * Authors:
>> + *   Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * You should have received a copy of the GNU General Public License along
>> + * with this program; if not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#ifndef ACPI_GHES_H
>> +#define ACPI_GHES_H
>> +
>> +#include "hw/acpi/bios-linker-loader.h"
>> +
>> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
>> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
>> +
>> +#define GHES_GAS_ADDRESS_OFFSET              4
>> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
>> +#define GHES_NOTIFICATION_STRUCTURE          32
>> +
>> +#define GHES_CPER_OK   1
>> +#define GHES_CPER_FAIL 0
>> +
>> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
>> +/* The max size in Bytes for one error block */
>> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
>> +
>> +
>> +typedef struct GhesState {
>> +    uint64_t ghes_addr_le;
>> +} GhesState;
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                            BIOSLinker *linker);
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
>> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
>> +#endif
>>
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
  2017-08-24 13:04     ` Shannon Zhao
                         ` (2 preceding siblings ...)
  (?)
@ 2017-08-25 11:20       ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 11:20 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/24 21:04, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> Add CONFIG_ACPI_APEI configuration in the Makefile and
>> enable it in the arm-softmmu.mak
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  default-configs/arm-softmmu.mak | 1 +
>>  hw/acpi/Makefile.objs           | 1 +
>>  2 files changed, 2 insertions(+)
>>
>> diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
>> index bbdd3c1..c362113 100644
>> --- a/default-configs/arm-softmmu.mak
>> +++ b/default-configs/arm-softmmu.mak
>> @@ -129,3 +129,4 @@ CONFIG_ACPI=y
>>  CONFIG_SMBIOS=y
>>  CONFIG_ASPEED_SOC=y
>>  CONFIG_GPIO_KEY=y
>> +CONFIG_ACPI_APEI=y
>> diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
>> index 11c35bc..bafb148 100644
>> --- a/hw/acpi/Makefile.objs
>> +++ b/hw/acpi/Makefile.objs
>> @@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
>>  common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
>>  common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
>>  common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
>> +common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
>>  common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
>>  
>>  common-obj-y += acpi_interface.o
>>
> Fold this patch into previous one.
Ok, thanks

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
@ 2017-08-25 11:20       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 11:20 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/24 21:04, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> Add CONFIG_ACPI_APEI configuration in the Makefile and
>> enable it in the arm-softmmu.mak
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  default-configs/arm-softmmu.mak | 1 +
>>  hw/acpi/Makefile.objs           | 1 +
>>  2 files changed, 2 insertions(+)
>>
>> diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
>> index bbdd3c1..c362113 100644
>> --- a/default-configs/arm-softmmu.mak
>> +++ b/default-configs/arm-softmmu.mak
>> @@ -129,3 +129,4 @@ CONFIG_ACPI=y
>>  CONFIG_SMBIOS=y
>>  CONFIG_ASPEED_SOC=y
>>  CONFIG_GPIO_KEY=y
>> +CONFIG_ACPI_APEI=y
>> diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
>> index 11c35bc..bafb148 100644
>> --- a/hw/acpi/Makefile.objs
>> +++ b/hw/acpi/Makefile.objs
>> @@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
>>  common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
>>  common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
>>  common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
>> +common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
>>  common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
>>  
>>  common-obj-y += acpi_interface.o
>>
> Fold this patch into previous one.
Ok, thanks

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
@ 2017-08-25 11:20       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 11:20 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/24 21:04, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> Add CONFIG_ACPI_APEI configuration in the Makefile and
>> enable it in the arm-softmmu.mak
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  default-configs/arm-softmmu.mak | 1 +
>>  hw/acpi/Makefile.objs           | 1 +
>>  2 files changed, 2 insertions(+)
>>
>> diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
>> index bbdd3c1..c362113 100644
>> --- a/default-configs/arm-softmmu.mak
>> +++ b/default-configs/arm-softmmu.mak
>> @@ -129,3 +129,4 @@ CONFIG_ACPI=y
>>  CONFIG_SMBIOS=y
>>  CONFIG_ASPEED_SOC=y
>>  CONFIG_GPIO_KEY=y
>> +CONFIG_ACPI_APEI=y
>> diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
>> index 11c35bc..bafb148 100644
>> --- a/hw/acpi/Makefile.objs
>> +++ b/hw/acpi/Makefile.objs
>> @@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
>>  common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
>>  common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
>>  common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
>> +common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
>>  common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
>>  
>>  common-obj-y += acpi_interface.o
>>
> Fold this patch into previous one.
Ok, thanks

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
@ 2017-08-25 11:20       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 11:20 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/24 21:04, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> Add CONFIG_ACPI_APEI configuration in the Makefile and
>> enable it in the arm-softmmu.mak
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  default-configs/arm-softmmu.mak | 1 +
>>  hw/acpi/Makefile.objs           | 1 +
>>  2 files changed, 2 insertions(+)
>>
>> diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
>> index bbdd3c1..c362113 100644
>> --- a/default-configs/arm-softmmu.mak
>> +++ b/default-configs/arm-softmmu.mak
>> @@ -129,3 +129,4 @@ CONFIG_ACPI=y
>>  CONFIG_SMBIOS=y
>>  CONFIG_ASPEED_SOC=y
>>  CONFIG_GPIO_KEY=y
>> +CONFIG_ACPI_APEI=y
>> diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
>> index 11c35bc..bafb148 100644
>> --- a/hw/acpi/Makefile.objs
>> +++ b/hw/acpi/Makefile.objs
>> @@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
>>  common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
>>  common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
>>  common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
>> +common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
>>  common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
>>  
>>  common-obj-y += acpi_interface.o
>>
> Fold this patch into previous one.
Ok, thanks

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration
@ 2017-08-25 11:20       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-25 11:20 UTC (permalink / raw)
  To: linux-arm-kernel



On 2017/8/24 21:04, Shannon Zhao wrote:
> 
> 
> On 2017/8/18 22:23, Dongjiu Geng wrote:
>> Add CONFIG_ACPI_APEI configuration in the Makefile and
>> enable it in the arm-softmmu.mak
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  default-configs/arm-softmmu.mak | 1 +
>>  hw/acpi/Makefile.objs           | 1 +
>>  2 files changed, 2 insertions(+)
>>
>> diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
>> index bbdd3c1..c362113 100644
>> --- a/default-configs/arm-softmmu.mak
>> +++ b/default-configs/arm-softmmu.mak
>> @@ -129,3 +129,4 @@ CONFIG_ACPI=y
>>  CONFIG_SMBIOS=y
>>  CONFIG_ASPEED_SOC=y
>>  CONFIG_GPIO_KEY=y
>> +CONFIG_ACPI_APEI=y
>> diff --git a/hw/acpi/Makefile.objs b/hw/acpi/Makefile.objs
>> index 11c35bc..bafb148 100644
>> --- a/hw/acpi/Makefile.objs
>> +++ b/hw/acpi/Makefile.objs
>> @@ -6,6 +6,7 @@ common-obj-$(CONFIG_ACPI_MEMORY_HOTPLUG) += memory_hotplug.o
>>  common-obj-$(CONFIG_ACPI_CPU_HOTPLUG) += cpu.o
>>  common-obj-$(CONFIG_ACPI_NVDIMM) += nvdimm.o
>>  common-obj-$(CONFIG_ACPI_VMGENID) += vmgenid.o
>> +common-obj-$(CONFIG_ACPI_APEI) += hest_ghes.o
>>  common-obj-$(call lnot,$(CONFIG_ACPI_X86)) += acpi-stub.o
>>  
>>  common-obj-y += acpi_interface.o
>>
> Fold this patch into previous one.
Ok, thanks

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
  2017-08-25 10:37       ` gengdongjiu
                           ` (2 preceding siblings ...)
  (?)
@ 2017-08-26  1:00         ` Shannon Zhao
  -1 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-26  1:00 UTC (permalink / raw)
  To: gengdongjiu, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/25 18:37, gengdongjiu wrote:
>>> +
>>> >> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>>> >> +
>> > It's better to refer to the first spec version of this structure and
>> > same with others you define.
>  do you mean which spec version? the definition is aligned with the linux kernel.
What I mean here is that it's better to refer to the ACPI spec version
which introduces Hardware Error Notification first time.

>> > 
>>> >> +enum AcpiHestNotifyType {
>>> >> +    ACPI_HEST_NOTIFY_POLLED = 0,
>>> >> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>>> >> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>>> >> +    ACPI_HEST_NOTIFY_SCI = 3,
>>> >> +    ACPI_HEST_NOTIFY_NMI = 4,
>>> >> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>>> >> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>>> >> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>>> >> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
>> > In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
>> > your patchset?
>   it is usefull, for all the error source, I reserved the space for them.
> Because the space is allocated one time, is not dynamically allocated.
> so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.
> 
I mean whether the new type Software Delegated Exception is useful for
RAS. If so, we could add this new type here.

Thanks,
-- 
Shannon


^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-26  1:00         ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-26  1:00 UTC (permalink / raw)
  To: gengdongjiu, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/25 18:37, gengdongjiu wrote:
>>> +
>>> >> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>>> >> +
>> > It's better to refer to the first spec version of this structure and
>> > same with others you define.
>  do you mean which spec version? the definition is aligned with the linux kernel.
What I mean here is that it's better to refer to the ACPI spec version
which introduces Hardware Error Notification first time.

>> > 
>>> >> +enum AcpiHestNotifyType {
>>> >> +    ACPI_HEST_NOTIFY_POLLED = 0,
>>> >> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>>> >> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>>> >> +    ACPI_HEST_NOTIFY_SCI = 3,
>>> >> +    ACPI_HEST_NOTIFY_NMI = 4,
>>> >> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>>> >> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>>> >> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>>> >> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
>> > In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
>> > your patchset?
>   it is usefull, for all the error source, I reserved the space for them.
> Because the space is allocated one time, is not dynamically allocated.
> so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.
> 
I mean whether the new type Software Delegated Exception is useful for
RAS. If so, we could add this new type here.

Thanks,
-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-26  1:00         ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-26  1:00 UTC (permalink / raw)
  To: gengdongjiu, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/25 18:37, gengdongjiu wrote:
>>> +
>>> >> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>>> >> +
>> > It's better to refer to the first spec version of this structure and
>> > same with others you define.
>  do you mean which spec version? the definition is aligned with the linux kernel.
What I mean here is that it's better to refer to the ACPI spec version
which introduces Hardware Error Notification first time.

>> > 
>>> >> +enum AcpiHestNotifyType {
>>> >> +    ACPI_HEST_NOTIFY_POLLED = 0,
>>> >> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>>> >> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>>> >> +    ACPI_HEST_NOTIFY_SCI = 3,
>>> >> +    ACPI_HEST_NOTIFY_NMI = 4,
>>> >> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>>> >> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>>> >> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>>> >> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
>> > In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
>> > your patchset?
>   it is usefull, for all the error source, I reserved the space for them.
> Because the space is allocated one time, is not dynamically allocated.
> so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.
> 
I mean whether the new type Software Delegated Exception is useful for
RAS. If so, we could add this new type here.

Thanks,
-- 
Shannon


^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-26  1:00         ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-26  1:00 UTC (permalink / raw)
  To: gengdongjiu, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/25 18:37, gengdongjiu wrote:
>>> +
>>> >> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>>> >> +
>> > It's better to refer to the first spec version of this structure and
>> > same with others you define.
>  do you mean which spec version? the definition is aligned with the linux kernel.
What I mean here is that it's better to refer to the ACPI spec version
which introduces Hardware Error Notification first time.

>> > 
>>> >> +enum AcpiHestNotifyType {
>>> >> +    ACPI_HEST_NOTIFY_POLLED = 0,
>>> >> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>>> >> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>>> >> +    ACPI_HEST_NOTIFY_SCI = 3,
>>> >> +    ACPI_HEST_NOTIFY_NMI = 4,
>>> >> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>>> >> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>>> >> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>>> >> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
>> > In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
>> > your patchset?
>   it is usefull, for all the error source, I reserved the space for them.
> Because the space is allocated one time, is not dynamically allocated.
> so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.
> 
I mean whether the new type Software Delegated Exception is useful for
RAS. If so, we could add this new type here.

Thanks,
-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-26  1:00         ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-26  1:00 UTC (permalink / raw)
  To: linux-arm-kernel



On 2017/8/25 18:37, gengdongjiu wrote:
>>> +
>>> >> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>>> >> +
>> > It's better to refer to the first spec version of this structure and
>> > same with others you define.
>  do you mean which spec version? the definition is aligned with the linux kernel.
What I mean here is that it's better to refer to the ACPI spec version
which introduces Hardware Error Notification first time.

>> > 
>>> >> +enum AcpiHestNotifyType {
>>> >> +    ACPI_HEST_NOTIFY_POLLED = 0,
>>> >> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>>> >> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>>> >> +    ACPI_HEST_NOTIFY_SCI = 3,
>>> >> +    ACPI_HEST_NOTIFY_NMI = 4,
>>> >> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>>> >> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>>> >> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>>> >> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>>> >> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
>> > In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
>> > your patchset?
>   it is usefull, for all the error source, I reserved the space for them.
> Because the space is allocated one time, is not dynamically allocated.
> so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.
> 
I mean whether the new type Software Delegated Exception is useful for
RAS. If so, we could add this new type here.

Thanks,
-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
  2017-08-25 11:20       ` gengdongjiu
                           ` (2 preceding siblings ...)
  (?)
@ 2017-08-26  1:08         ` Shannon Zhao
  -1 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-26  1:08 UTC (permalink / raw)
  To: gengdongjiu, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/25 19:20, gengdongjiu wrote:
>>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>>> >> index 3d78ff6..def1ec1 100644
>>> >> --- a/hw/arm/virt-acpi-build.c
>>> >> +++ b/hw/arm/virt-acpi-build.c
>>> >> @@ -45,6 +45,7 @@
>>> >>  #include "hw/arm/virt.h"
>>> >>  #include "sysemu/numa.h"
>>> >>  #include "kvm_arm.h"
>>> >> +#include "hw/acpi/hest_ghes.h"
>>> >>  
>>> >>  #define ARM_SPI_BASE 32
>>> >>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>>> >> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>> >>      acpi_add_table(table_offsets, tables_blob);
>>> >>      build_spcr(tables_blob, tables->linker, vms);
>>> >>  
>>> >> +    acpi_add_table(table_offsets, tables_blob);
>>> >> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>>> >> +
>> > So we add this table unconditionally. Is there any bad impact if QEMU
>> > runs on old kvm? Does it need to check whether KVM supports RAS?
> this table is added before guest OS boot. so can not use KVM to check it.
No, we can check the RAS capability when we create vcpus like you done
in another patch ans can use that in table generation.

> if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
> May be we can make it as device? if this device is enabled in the qemu
> boot parameters, then we will add this table?
> 

And you need to add a option to virt machine for (migration)
compatibility. On new virt machine it's on by default while off for old
ones.

Thanks,
-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-26  1:08         ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-26  1:08 UTC (permalink / raw)
  To: gengdongjiu, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/25 19:20, gengdongjiu wrote:
>>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>>> >> index 3d78ff6..def1ec1 100644
>>> >> --- a/hw/arm/virt-acpi-build.c
>>> >> +++ b/hw/arm/virt-acpi-build.c
>>> >> @@ -45,6 +45,7 @@
>>> >>  #include "hw/arm/virt.h"
>>> >>  #include "sysemu/numa.h"
>>> >>  #include "kvm_arm.h"
>>> >> +#include "hw/acpi/hest_ghes.h"
>>> >>  
>>> >>  #define ARM_SPI_BASE 32
>>> >>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>>> >> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>> >>      acpi_add_table(table_offsets, tables_blob);
>>> >>      build_spcr(tables_blob, tables->linker, vms);
>>> >>  
>>> >> +    acpi_add_table(table_offsets, tables_blob);
>>> >> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>>> >> +
>> > So we add this table unconditionally. Is there any bad impact if QEMU
>> > runs on old kvm? Does it need to check whether KVM supports RAS?
> this table is added before guest OS boot. so can not use KVM to check it.
No, we can check the RAS capability when we create vcpus like you done
in another patch ans can use that in table generation.

> if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
> May be we can make it as device? if this device is enabled in the qemu
> boot parameters, then we will add this table?
> 

And you need to add a option to virt machine for (migration)
compatibility. On new virt machine it's on by default while off for old
ones.

Thanks,
-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-26  1:08         ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-26  1:08 UTC (permalink / raw)
  To: gengdongjiu, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/25 19:20, gengdongjiu wrote:
>>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>>> >> index 3d78ff6..def1ec1 100644
>>> >> --- a/hw/arm/virt-acpi-build.c
>>> >> +++ b/hw/arm/virt-acpi-build.c
>>> >> @@ -45,6 +45,7 @@
>>> >>  #include "hw/arm/virt.h"
>>> >>  #include "sysemu/numa.h"
>>> >>  #include "kvm_arm.h"
>>> >> +#include "hw/acpi/hest_ghes.h"
>>> >>  
>>> >>  #define ARM_SPI_BASE 32
>>> >>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>>> >> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>> >>      acpi_add_table(table_offsets, tables_blob);
>>> >>      build_spcr(tables_blob, tables->linker, vms);
>>> >>  
>>> >> +    acpi_add_table(table_offsets, tables_blob);
>>> >> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>>> >> +
>> > So we add this table unconditionally. Is there any bad impact if QEMU
>> > runs on old kvm? Does it need to check whether KVM supports RAS?
> this table is added before guest OS boot. so can not use KVM to check it.
No, we can check the RAS capability when we create vcpus like you done
in another patch ans can use that in table generation.

> if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
> May be we can make it as device? if this device is enabled in the qemu
> boot parameters, then we will add this table?
> 

And you need to add a option to virt machine for (migration)
compatibility. On new virt machine it's on by default while off for old
ones.

Thanks,
-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-26  1:08         ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-26  1:08 UTC (permalink / raw)
  To: gengdongjiu, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/25 19:20, gengdongjiu wrote:
>>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>>> >> index 3d78ff6..def1ec1 100644
>>> >> --- a/hw/arm/virt-acpi-build.c
>>> >> +++ b/hw/arm/virt-acpi-build.c
>>> >> @@ -45,6 +45,7 @@
>>> >>  #include "hw/arm/virt.h"
>>> >>  #include "sysemu/numa.h"
>>> >>  #include "kvm_arm.h"
>>> >> +#include "hw/acpi/hest_ghes.h"
>>> >>  
>>> >>  #define ARM_SPI_BASE 32
>>> >>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>>> >> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>> >>      acpi_add_table(table_offsets, tables_blob);
>>> >>      build_spcr(tables_blob, tables->linker, vms);
>>> >>  
>>> >> +    acpi_add_table(table_offsets, tables_blob);
>>> >> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>>> >> +
>> > So we add this table unconditionally. Is there any bad impact if QEMU
>> > runs on old kvm? Does it need to check whether KVM supports RAS?
> this table is added before guest OS boot. so can not use KVM to check it.
No, we can check the RAS capability when we create vcpus like you done
in another patch ans can use that in table generation.

> if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
> May be we can make it as device? if this device is enabled in the qemu
> boot parameters, then we will add this table?
> 

And you need to add a option to virt machine for (migration)
compatibility. On new virt machine it's on by default while off for old
ones.

Thanks,
-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-26  1:08         ` Shannon Zhao
  0 siblings, 0 replies; 129+ messages in thread
From: Shannon Zhao @ 2017-08-26  1:08 UTC (permalink / raw)
  To: linux-arm-kernel



On 2017/8/25 19:20, gengdongjiu wrote:
>>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>>> >> index 3d78ff6..def1ec1 100644
>>> >> --- a/hw/arm/virt-acpi-build.c
>>> >> +++ b/hw/arm/virt-acpi-build.c
>>> >> @@ -45,6 +45,7 @@
>>> >>  #include "hw/arm/virt.h"
>>> >>  #include "sysemu/numa.h"
>>> >>  #include "kvm_arm.h"
>>> >> +#include "hw/acpi/hest_ghes.h"
>>> >>  
>>> >>  #define ARM_SPI_BASE 32
>>> >>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>>> >> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>> >>      acpi_add_table(table_offsets, tables_blob);
>>> >>      build_spcr(tables_blob, tables->linker, vms);
>>> >>  
>>> >> +    acpi_add_table(table_offsets, tables_blob);
>>> >> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>>> >> +
>> > So we add this table unconditionally. Is there any bad impact if QEMU
>> > runs on old kvm? Does it need to check whether KVM supports RAS?
> this table is added before guest OS boot. so can not use KVM to check it.
No, we can check the RAS capability when we create vcpus like you done
in another patch ans can use that in table generation.

> if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
> May be we can make it as device? if this device is enabled in the qemu
> boot parameters, then we will add this table?
> 

And you need to add a option to virt machine for (migration)
compatibility. On new virt machine it's on by default while off for old
ones.

Thanks,
-- 
Shannon

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
  2017-08-26  1:00         ` Shannon Zhao
                             ` (2 preceding siblings ...)
  (?)
@ 2017-08-26  1:45           ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-26  1:45 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/26 9:00, Shannon Zhao wrote:
> 
> 
> On 2017/8/25 18:37, gengdongjiu wrote:
>>>> +
>>>>>> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>>>>>> +
>>>> It's better to refer to the first spec version of this structure and
>>>> same with others you define.
>>  do you mean which spec version? the definition is aligned with the linux kernel.
> What I mean here is that it's better to refer to the ACPI spec version
> which introduces Hardware Error Notification first time.
Ok, I basically understand your meaning. I will clear that. thanks.

> 
>>>>
>>>>>> +enum AcpiHestNotifyType {
>>>>>> +    ACPI_HEST_NOTIFY_POLLED = 0,
>>>>>> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>>>>>> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>>>>>> +    ACPI_HEST_NOTIFY_SCI = 3,
>>>>>> +    ACPI_HEST_NOTIFY_NMI = 4,
>>>>>> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>>>>>> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>>>>>> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>>>>>> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
>>>> In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
>>>> your patchset?
>>   it is usefull, for all the error source, I reserved the space for them.
>> Because the space is allocated one time, is not dynamically allocated.
>> so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.
>>
> I mean whether the new type Software Delegated Exception is useful for
> RAS. If so, we could add this new type here.
Just now I check the ACPI 6.2 spec, it indeed introduced the new type SDEI.

currently we do not use the type Software Delegated Exception which introduced by ACPI 6.2,
so may not need to add a new type.

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-26  1:45           ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-26  1:45 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/26 9:00, Shannon Zhao wrote:
> 
> 
> On 2017/8/25 18:37, gengdongjiu wrote:
>>>> +
>>>>>> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>>>>>> +
>>>> It's better to refer to the first spec version of this structure and
>>>> same with others you define.
>>  do you mean which spec version? the definition is aligned with the linux kernel.
> What I mean here is that it's better to refer to the ACPI spec version
> which introduces Hardware Error Notification first time.
Ok, I basically understand your meaning. I will clear that. thanks.

> 
>>>>
>>>>>> +enum AcpiHestNotifyType {
>>>>>> +    ACPI_HEST_NOTIFY_POLLED = 0,
>>>>>> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>>>>>> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>>>>>> +    ACPI_HEST_NOTIFY_SCI = 3,
>>>>>> +    ACPI_HEST_NOTIFY_NMI = 4,
>>>>>> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>>>>>> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>>>>>> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>>>>>> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
>>>> In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
>>>> your patchset?
>>   it is usefull, for all the error source, I reserved the space for them.
>> Because the space is allocated one time, is not dynamically allocated.
>> so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.
>>
> I mean whether the new type Software Delegated Exception is useful for
> RAS. If so, we could add this new type here.
Just now I check the ACPI 6.2 spec, it indeed introduced the new type SDEI.

currently we do not use the type Software Delegated Exception which introduced by ACPI 6.2,
so may not need to add a new type.

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-26  1:45           ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-26  1:45 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/26 9:00, Shannon Zhao wrote:
> 
> 
> On 2017/8/25 18:37, gengdongjiu wrote:
>>>> +
>>>>>> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>>>>>> +
>>>> It's better to refer to the first spec version of this structure and
>>>> same with others you define.
>>  do you mean which spec version? the definition is aligned with the linux kernel.
> What I mean here is that it's better to refer to the ACPI spec version
> which introduces Hardware Error Notification first time.
Ok, I basically understand your meaning. I will clear that. thanks.

> 
>>>>
>>>>>> +enum AcpiHestNotifyType {
>>>>>> +    ACPI_HEST_NOTIFY_POLLED = 0,
>>>>>> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>>>>>> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>>>>>> +    ACPI_HEST_NOTIFY_SCI = 3,
>>>>>> +    ACPI_HEST_NOTIFY_NMI = 4,
>>>>>> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>>>>>> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>>>>>> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>>>>>> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
>>>> In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
>>>> your patchset?
>>   it is usefull, for all the error source, I reserved the space for them.
>> Because the space is allocated one time, is not dynamically allocated.
>> so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.
>>
> I mean whether the new type Software Delegated Exception is useful for
> RAS. If so, we could add this new type here.
Just now I check the ACPI 6.2 spec, it indeed introduced the new type SDEI.

currently we do not use the type Software Delegated Exception which introduced by ACPI 6.2,
so may not need to add a new type.

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-26  1:45           ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-26  1:45 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10



On 2017/8/26 9:00, Shannon Zhao wrote:
> 
> 
> On 2017/8/25 18:37, gengdongjiu wrote:
>>>> +
>>>>>> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>>>>>> +
>>>> It's better to refer to the first spec version of this structure and
>>>> same with others you define.
>>  do you mean which spec version? the definition is aligned with the linux kernel.
> What I mean here is that it's better to refer to the ACPI spec version
> which introduces Hardware Error Notification first time.
Ok, I basically understand your meaning. I will clear that. thanks.

> 
>>>>
>>>>>> +enum AcpiHestNotifyType {
>>>>>> +    ACPI_HEST_NOTIFY_POLLED = 0,
>>>>>> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>>>>>> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>>>>>> +    ACPI_HEST_NOTIFY_SCI = 3,
>>>>>> +    ACPI_HEST_NOTIFY_NMI = 4,
>>>>>> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>>>>>> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>>>>>> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>>>>>> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
>>>> In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
>>>> your patchset?
>>   it is usefull, for all the error source, I reserved the space for them.
>> Because the space is allocated one time, is not dynamically allocated.
>> so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.
>>
> I mean whether the new type Software Delegated Exception is useful for
> RAS. If so, we could add this new type here.
Just now I check the ACPI 6.2 spec, it indeed introduced the new type SDEI.

currently we do not use the type Software Delegated Exception which introduced by ACPI 6.2,
so may not need to add a new type.

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros
@ 2017-08-26  1:45           ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-26  1:45 UTC (permalink / raw)
  To: linux-arm-kernel



On 2017/8/26 9:00, Shannon Zhao wrote:
> 
> 
> On 2017/8/25 18:37, gengdongjiu wrote:
>>>> +
>>>>>> +/* From the ACPI 6.1 spec, "18.3.2.9 Hardware Error Notification" */
>>>>>> +
>>>> It's better to refer to the first spec version of this structure and
>>>> same with others you define.
>>  do you mean which spec version? the definition is aligned with the linux kernel.
> What I mean here is that it's better to refer to the ACPI spec version
> which introduces Hardware Error Notification first time.
Ok, I basically understand your meaning. I will clear that. thanks.

> 
>>>>
>>>>>> +enum AcpiHestNotifyType {
>>>>>> +    ACPI_HEST_NOTIFY_POLLED = 0,
>>>>>> +    ACPI_HEST_NOTIFY_EXTERNAL = 1,
>>>>>> +    ACPI_HEST_NOTIFY_LOCAL = 2,
>>>>>> +    ACPI_HEST_NOTIFY_SCI = 3,
>>>>>> +    ACPI_HEST_NOTIFY_NMI = 4,
>>>>>> +    ACPI_HEST_NOTIFY_CMCI = 5,  /* ACPI 5.0 */
>>>>>> +    ACPI_HEST_NOTIFY_MCE = 6,   /* ACPI 5.0 */
>>>>>> +    ACPI_HEST_NOTIFY_GPIO = 7,  /* ACPI 6.0 */
>>>>>> +    ACPI_HEST_NOTIFY_SEA = 8,   /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_SEI = 9,   /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_GSIV = 10, /* ACPI 6.1 */
>>>>>> +    ACPI_HEST_NOTIFY_RESERVED = 11  /* 11 and greater are reserved */
>>>> In ACPI 6.2, 11 is for Software Delegated Exception, is this useful for
>>>> your patchset?
>>   it is usefull, for all the error source, I reserved the space for them.
>> Because the space is allocated one time, is not dynamically allocated.
>> so I use the ACPI_HEST_NOTIFY_RESERVED to specify that there is 11 error source.
>>
> I mean whether the new type Software Delegated Exception is useful for
> RAS. If so, we could add this new type here.
Just now I check the ACPI 6.2 spec, it indeed introduced the new type SDEI.

currently we do not use the type Software Delegated Exception which introduced by ACPI 6.2,
so may not need to add a new type.

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
  2017-08-26  1:08         ` Shannon Zhao
                             ` (2 preceding siblings ...)
  (?)
@ 2017-08-26  2:49           ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-26  2:49 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10

Hi Shannon,

On 2017/8/26 9:08, Shannon Zhao wrote:
> 
> 
> On 2017/8/25 19:20, gengdongjiu wrote:
>>>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>>>>>> index 3d78ff6..def1ec1 100644
>>>>>> --- a/hw/arm/virt-acpi-build.c
>>>>>> +++ b/hw/arm/virt-acpi-build.c
>>>>>> @@ -45,6 +45,7 @@
>>>>>>  #include "hw/arm/virt.h"
>>>>>>  #include "sysemu/numa.h"
>>>>>>  #include "kvm_arm.h"
>>>>>> +#include "hw/acpi/hest_ghes.h"
>>>>>>  
>>>>>>  #define ARM_SPI_BASE 32
>>>>>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>>>>>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>>>>>      acpi_add_table(table_offsets, tables_blob);
>>>>>>      build_spcr(tables_blob, tables->linker, vms);
>>>>>>  
>>>>>> +    acpi_add_table(table_offsets, tables_blob);
>>>>>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>>>>>> +
>>>> So we add this table unconditionally. Is there any bad impact if QEMU
>>>> runs on old kvm? Does it need to check whether KVM supports RAS?
>> this table is added before guest OS boot. so can not use KVM to check it.
> No, we can check the RAS capability when we create vcpus like you done
> in another patch ans can use that in table generation.
understand your meaning.

ARM James ever have below comments about the table generation.
----------------------------------------------------------------------------------
But you can use APEI in a guest on CPUs without the RAS extensions: the host may
signal memory errors to Qemu for any number of reasons, user-space shouldn't
care how it knows. Examples are PCI-AER, any APEI event notified by polling or
one of the flavours of irq.

I would expect Qemu to generate a HEST based on its abilities, i.e. if it
supports any mechanism of notifying the guest about errors. Choosing the
mechanism then depends on the type of error.

Ideally the Qemu code for HEST/GHES/CPER generation code using some of the irqs
and polling could be shared with x86, as these should be possible using common
KVM APIs.
-----------------------------------------------------------------------------------

He means we can use APEI on CPUs without RAS and may be share this code with x86,
if Qemu can support any mechanism of notifying the guest about errors, it should be
generate the table.
Now we depend on the macro KVM_HAVE_MCE_INJECTION to decide whether Qemu can support
notifying the guest.

what do you think which we should be dependent on to generate the table?

> 
>> if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
>> May be we can make it as device? if this device is enabled in the qemu
>> boot parameters, then we will add this table?
>>
> 
> And you need to add a option to virt machine for (migration)
> compatibility. On new virt machine it's on by default while off for old
> ones.
ok.

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-26  2:49           ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-26  2:49 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10

Hi Shannon,

On 2017/8/26 9:08, Shannon Zhao wrote:
> 
> 
> On 2017/8/25 19:20, gengdongjiu wrote:
>>>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>>>>>> index 3d78ff6..def1ec1 100644
>>>>>> --- a/hw/arm/virt-acpi-build.c
>>>>>> +++ b/hw/arm/virt-acpi-build.c
>>>>>> @@ -45,6 +45,7 @@
>>>>>>  #include "hw/arm/virt.h"
>>>>>>  #include "sysemu/numa.h"
>>>>>>  #include "kvm_arm.h"
>>>>>> +#include "hw/acpi/hest_ghes.h"
>>>>>>  
>>>>>>  #define ARM_SPI_BASE 32
>>>>>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>>>>>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>>>>>      acpi_add_table(table_offsets, tables_blob);
>>>>>>      build_spcr(tables_blob, tables->linker, vms);
>>>>>>  
>>>>>> +    acpi_add_table(table_offsets, tables_blob);
>>>>>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>>>>>> +
>>>> So we add this table unconditionally. Is there any bad impact if QEMU
>>>> runs on old kvm? Does it need to check whether KVM supports RAS?
>> this table is added before guest OS boot. so can not use KVM to check it.
> No, we can check the RAS capability when we create vcpus like you done
> in another patch ans can use that in table generation.
understand your meaning.

ARM James ever have below comments about the table generation.
----------------------------------------------------------------------------------
But you can use APEI in a guest on CPUs without the RAS extensions: the host may
signal memory errors to Qemu for any number of reasons, user-space shouldn't
care how it knows. Examples are PCI-AER, any APEI event notified by polling or
one of the flavours of irq.

I would expect Qemu to generate a HEST based on its abilities, i.e. if it
supports any mechanism of notifying the guest about errors. Choosing the
mechanism then depends on the type of error.

Ideally the Qemu code for HEST/GHES/CPER generation code using some of the irqs
and polling could be shared with x86, as these should be possible using common
KVM APIs.
-----------------------------------------------------------------------------------

He means we can use APEI on CPUs without RAS and may be share this code with x86,
if Qemu can support any mechanism of notifying the guest about errors, it should be
generate the table.
Now we depend on the macro KVM_HAVE_MCE_INJECTION to decide whether Qemu can support
notifying the guest.

what do you think which we should be dependent on to generate the table?

> 
>> if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
>> May be we can make it as device? if this device is enabled in the qemu
>> boot parameters, then we will add this table?
>>
> 
> And you need to add a option to virt machine for (migration)
> compatibility. On new virt machine it's on by default while off for old
> ones.
ok.

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-26  2:49           ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-26  2:49 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10

Hi Shannon,

On 2017/8/26 9:08, Shannon Zhao wrote:
> 
> 
> On 2017/8/25 19:20, gengdongjiu wrote:
>>>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>>>>>> index 3d78ff6..def1ec1 100644
>>>>>> --- a/hw/arm/virt-acpi-build.c
>>>>>> +++ b/hw/arm/virt-acpi-build.c
>>>>>> @@ -45,6 +45,7 @@
>>>>>>  #include "hw/arm/virt.h"
>>>>>>  #include "sysemu/numa.h"
>>>>>>  #include "kvm_arm.h"
>>>>>> +#include "hw/acpi/hest_ghes.h"
>>>>>>  
>>>>>>  #define ARM_SPI_BASE 32
>>>>>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>>>>>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>>>>>      acpi_add_table(table_offsets, tables_blob);
>>>>>>      build_spcr(tables_blob, tables->linker, vms);
>>>>>>  
>>>>>> +    acpi_add_table(table_offsets, tables_blob);
>>>>>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>>>>>> +
>>>> So we add this table unconditionally. Is there any bad impact if QEMU
>>>> runs on old kvm? Does it need to check whether KVM supports RAS?
>> this table is added before guest OS boot. so can not use KVM to check it.
> No, we can check the RAS capability when we create vcpus like you done
> in another patch ans can use that in table generation.
understand your meaning.

ARM James ever have below comments about the table generation.
----------------------------------------------------------------------------------
But you can use APEI in a guest on CPUs without the RAS extensions: the host may
signal memory errors to Qemu for any number of reasons, user-space shouldn't
care how it knows. Examples are PCI-AER, any APEI event notified by polling or
one of the flavours of irq.

I would expect Qemu to generate a HEST based on its abilities, i.e. if it
supports any mechanism of notifying the guest about errors. Choosing the
mechanism then depends on the type of error.

Ideally the Qemu code for HEST/GHES/CPER generation code using some of the irqs
and polling could be shared with x86, as these should be possible using common
KVM APIs.
-----------------------------------------------------------------------------------

He means we can use APEI on CPUs without RAS and may be share this code with x86,
if Qemu can support any mechanism of notifying the guest about errors, it should be
generate the table.
Now we depend on the macro KVM_HAVE_MCE_INJECTION to decide whether Qemu can support
notifying the guest.

what do you think which we should be dependent on to generate the table?

> 
>> if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
>> May be we can make it as device? if this device is enabled in the qemu
>> boot parameters, then we will add this table?
>>
> 
> And you need to add a option to virt machine for (migration)
> compatibility. On new virt machine it's on by default while off for old
> ones.
ok.

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-26  2:49           ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-26  2:49 UTC (permalink / raw)
  To: Shannon Zhao, lersek, mst, imammedo, peter.maydell, pbonzini,
	qemu-devel, qemu-arm, kvm, edk2-devel, christoffer.dall,
	marc.zyngier, will.deacon, james.morse, tbaicar, ard.biesheuvel,
	mingo, bp, shiju.jose, zjzhang, linux-arm-kernel, kvmarm,
	linux-kernel, linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1
  Cc: huangshaoyu, wuquanming, linuxarm, zhengqiang10

Hi Shannon,

On 2017/8/26 9:08, Shannon Zhao wrote:
> 
> 
> On 2017/8/25 19:20, gengdongjiu wrote:
>>>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>>>>>> index 3d78ff6..def1ec1 100644
>>>>>> --- a/hw/arm/virt-acpi-build.c
>>>>>> +++ b/hw/arm/virt-acpi-build.c
>>>>>> @@ -45,6 +45,7 @@
>>>>>>  #include "hw/arm/virt.h"
>>>>>>  #include "sysemu/numa.h"
>>>>>>  #include "kvm_arm.h"
>>>>>> +#include "hw/acpi/hest_ghes.h"
>>>>>>  
>>>>>>  #define ARM_SPI_BASE 32
>>>>>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>>>>>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>>>>>      acpi_add_table(table_offsets, tables_blob);
>>>>>>      build_spcr(tables_blob, tables->linker, vms);
>>>>>>  
>>>>>> +    acpi_add_table(table_offsets, tables_blob);
>>>>>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>>>>>> +
>>>> So we add this table unconditionally. Is there any bad impact if QEMU
>>>> runs on old kvm? Does it need to check whether KVM supports RAS?
>> this table is added before guest OS boot. so can not use KVM to check it.
> No, we can check the RAS capability when we create vcpus like you done
> in another patch ans can use that in table generation.
understand your meaning.

ARM James ever have below comments about the table generation.
----------------------------------------------------------------------------------
But you can use APEI in a guest on CPUs without the RAS extensions: the host may
signal memory errors to Qemu for any number of reasons, user-space shouldn't
care how it knows. Examples are PCI-AER, any APEI event notified by polling or
one of the flavours of irq.

I would expect Qemu to generate a HEST based on its abilities, i.e. if it
supports any mechanism of notifying the guest about errors. Choosing the
mechanism then depends on the type of error.

Ideally the Qemu code for HEST/GHES/CPER generation code using some of the irqs
and polling could be shared with x86, as these should be possible using common
KVM APIs.
-----------------------------------------------------------------------------------

He means we can use APEI on CPUs without RAS and may be share this code with x86,
if Qemu can support any mechanism of notifying the guest about errors, it should be
generate the table.
Now we depend on the macro KVM_HAVE_MCE_INJECTION to decide whether Qemu can support
notifying the guest.

what do you think which we should be dependent on to generate the table?

> 
>> if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
>> May be we can make it as device? if this device is enabled in the qemu
>> boot parameters, then we will add this table?
>>
> 
> And you need to add a option to virt machine for (migration)
> compatibility. On new virt machine it's on by default while off for old
> ones.
ok.

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-26  2:49           ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-26  2:49 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Shannon,

On 2017/8/26 9:08, Shannon Zhao wrote:
> 
> 
> On 2017/8/25 19:20, gengdongjiu wrote:
>>>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>>>>>> index 3d78ff6..def1ec1 100644
>>>>>> --- a/hw/arm/virt-acpi-build.c
>>>>>> +++ b/hw/arm/virt-acpi-build.c
>>>>>> @@ -45,6 +45,7 @@
>>>>>>  #include "hw/arm/virt.h"
>>>>>>  #include "sysemu/numa.h"
>>>>>>  #include "kvm_arm.h"
>>>>>> +#include "hw/acpi/hest_ghes.h"
>>>>>>  
>>>>>>  #define ARM_SPI_BASE 32
>>>>>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>>>>>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>>>>>      acpi_add_table(table_offsets, tables_blob);
>>>>>>      build_spcr(tables_blob, tables->linker, vms);
>>>>>>  
>>>>>> +    acpi_add_table(table_offsets, tables_blob);
>>>>>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>>>>>> +
>>>> So we add this table unconditionally. Is there any bad impact if QEMU
>>>> runs on old kvm? Does it need to check whether KVM supports RAS?
>> this table is added before guest OS boot. so can not use KVM to check it.
> No, we can check the RAS capability when we create vcpus like you done
> in another patch ans can use that in table generation.
understand your meaning.

ARM James ever have below comments about the table generation.
----------------------------------------------------------------------------------
But you can use APEI in a guest on CPUs without the RAS extensions: the host may
signal memory errors to Qemu for any number of reasons, user-space shouldn't
care how it knows. Examples are PCI-AER, any APEI event notified by polling or
one of the flavours of irq.

I would expect Qemu to generate a HEST based on its abilities, i.e. if it
supports any mechanism of notifying the guest about errors. Choosing the
mechanism then depends on the type of error.

Ideally the Qemu code for HEST/GHES/CPER generation code using some of the irqs
and polling could be shared with x86, as these should be possible using common
KVM APIs.
-----------------------------------------------------------------------------------

He means we can use APEI on CPUs without RAS and may be share this code with x86,
if Qemu can support any mechanism of notifying the guest about errors, it should be
generate the table.
Now we depend on the macro KVM_HAVE_MCE_INJECTION to decide whether Qemu can support
notifying the guest.

what do you think which we should be dependent on to generate the table?

> 
>> if the old kvm does not support RAS, it does not have bad impact. only waste table memory.
>> May be we can make it as device? if this device is enabled in the qemu
>> boot parameters, then we will add this table?
>>
> 
> And you need to add a option to virt machine for (migration)
> compatibility. On new virt machine it's on by default while off for old
> ones.
ok.

> 
> Thanks,
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
  2017-08-18 14:23   ` Dongjiu Geng
                       ` (2 preceding siblings ...)
  (?)
@ 2017-08-29 10:20     ` Igor Mammedov
  -1 siblings, 0 replies; 129+ messages in thread
From: Igor Mammedov @ 2017-08-29 10:20 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: mst, zhaoshenglong, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, zhengqiang10,
	wuquanming

On Fri, 18 Aug 2017 22:23:43 +0800
Dongjiu Geng <gengdongjiu@huawei.com> wrote:

> This implements APEI GHES Table by passing the error CPER info
> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
> will be injected into the guest OS.

it's a bit complex patch/functionality so I've just mosty skimmed and
commented only on structure of the patch and changes I'd like to see
so it would be more structured and review-able.

I'd suggest to add doc patch first which will describe how it's
supposed to work between QEMU/firmware/guest OS with expected
flows.
 
> Below is the table layout, the max number of error soure is 11,
> which is classified by notification type.
> 
>      etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
these diagram shows relations between tables which not necessarily bad
but as layout it's useless.
 * Probably there is not much sense to have HEST table here, it's described
   well enough in spec. You might just put reference here.
 * these diagrams should go into doc/spec patch
 * when you describe layout you need to show what and at what offsets
   in which order in which blob/file is located. See ACPI spec for example
   and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.

> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
> so user space must check the ack value to avoid read-write race condition.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  hw/acpi/aml-build.c         |   2 +
>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c    |   6 +
>  include/hw/acpi/aml-build.h |   1 +
>  include/hw/acpi/hest_ghes.h |  47 ++++++
>  5 files changed, 401 insertions(+)
>  create mode 100644 hw/acpi/hest_ghes.c
>  create mode 100644 include/hw/acpi/hest_ghes.h
> 
> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
> index 36a6cc4..6849e5f 100644
> --- a/hw/acpi/aml-build.c
> +++ b/hw/acpi/aml-build.c
> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>      tables->table_data = g_array_new(false, true /* clear */, 1);
>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>      tables->linker = bios_linker_loader_init();
>  }
>  
> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>      g_array_free(tables->table_data, true);
>      g_array_free(tables->tcpalog, mfre);
>      g_array_free(tables->vmgenid, mfre);
> +    g_array_free(tables->hardware_errors, mfre);
>  }
>  
>  /* Build rsdt table */
> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
> new file mode 100644
> index 0000000..ff6b5ef
> --- /dev/null
> +++ b/hw/acpi/hest_ghes.c
> @@ -0,0 +1,345 @@
> +/*
> + *  APEI GHES table Generation
> + *
> + *  Copyright (C) 2017 huawei.
> + *
> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qmp-commands.h"
> +#include "hw/acpi/acpi.h"
> +#include "hw/acpi/aml-build.h"
> +#include "hw/acpi/hest_ghes.h"
> +#include "hw/nvram/fw_cfg.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/error-report.h"
> +
> +/* The structure that stands for the layout
> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
> + *
> + *           etc/hardware_errors
> + * ==========================================
> + * +------------------+
> + * |    address       |              +--------------+
> + * |    registers     |              | Error Status |
> + * | +----------------+              | Data Block 0 |
> + * | |status_address0 |------------->| +------------+
> + * | +----------------+              | |  CPER      |
> + * | |status_address1 |----------+   | |  CPER      |
> + * | +----------------+          |   | |  ....      |
> + * | |.............   |          |   | |  CPER      |
> + * | +----------------+          |   | +------------+
> + * | |status_address10|-----+    |   | Error Status |
> + * | +----------------+     |    |   | Data Block 1 |
> + * | |ack_value0      |     |    +-->| +------------+
> + * | +----------------+     |        | |  CPER      |
> + * | |ack_value1      |     |        | |  CPER      |
> + * | +----------------+     |        | |  ....      |
> + * | | .............  |     |        | |  CPER      |
> + * | +----------------+     |        +-+------------+
> + * | |ack_value10     |     |        | |..........  |
> + * | +----------------+     |        | +------------+
> + *                          |        | Error Status |
> + *                          |        | Data Block10 |
> + *                          +------->+------------+
> + *                                   | |  CPER      |
> + *                                   | |  CPER      |
> + *                                   | |  ....      |
> + *                                   | |  CPER      |
> + *                                   +-+------------+
> + */
> +struct hardware_errors_buffer {
> +    /* Generic Error Status Block register */
> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
> +};
> +
> +static int ghes_record_cper(uint64_t error_block_address,
> +                                    uint64_t error_physical_addr)
> +{
> +    AcpiGenericErrorStatus block;
> +    AcpiGenericErrorData *gdata;
> +    UefiCperSecMemErr *mem_err;
> +    uint64_t current_block_length;
> +    unsigned char *buffer;
> +    /* memory section */
> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
> +                              0x83, 0xB1};
> +
> +    cpu_physical_memory_read(error_block_address, &block,
> +                                sizeof(AcpiGenericErrorStatus));
> +
> +    /* Get the current generic error status block length */
> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
> +        le32_to_cpu(block.data_length);
> +
> +    /* If the Generic Error Status Block is NULL, update
> +     * the block header
> +     */
> +    if (!block.block_status) {
> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    }
> +
> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    /* check whether it runs out of the preallocated memory */
> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
> +       GHES_MAX_RAW_DATA_LENGTH) {
> +        error_report("Record CPER out of boundary!!!");
> +        return GHES_CPER_FAIL;
> +    }
> +
> +    /* Write back the Generic Error Status Block to guest memory */
> +    cpu_physical_memory_write(error_block_address, &block,
> +        sizeof(AcpiGenericErrorStatus));
> +
> +    /* Fill in Generic Error Data Entry */
> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
> +                       sizeof(UefiCperSecMemErr));
> +
> +
> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
looks redundant, g_malloc0 does it for you

> +    gdata = (AcpiGenericErrorData *)buffer;
> +
> +    /* Memory section */
> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
> +            sizeof(mem_section_id_le));
> +
> +    /* error severity is recoverable */
> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    gdata->revision = 0x300; /* the revision number is 0x300 */
> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
> +
> +    /* User space only handle the memory section CPER */
> +
> +    /* Hard code to Multi-bit ECC error */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
> +
> +    /* Record the physical address at which the memory error occurred */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);

I'd prefer for you to use build_append_int_noprefix() API to compose
whole error status block

and try to get rid of most structures you introduce in patch 1/6,
as they will be left unused after that.

> +
> +    /* Write back the Generic Error Data Entry to guest memory */
> +    cpu_physical_memory_write(error_block_address + current_block_length,
> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +
> +    g_free(buffer);
> +    return GHES_CPER_OK;
> +}
> +
> +static void
> +build_address(GArray *table_data, BIOSLinker *linker,
> +    uint32_t dst_patched_offset, uint32_t src_offset,
> +    uint8_t address_space_id , uint8_t  register_bit_width,
> +    uint8_t register_bit_offset, uint8_t access_size)
> +{
> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    /* Address space */
> +    build_append_int_noprefix(table_data, address_space_id, 1);
> +    /* register bit width */
> +    build_append_int_noprefix(table_data, register_bit_width, 1);
> +    /* register bit offset */
> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
> +    /* access size */
> +    build_append_int_noprefix(table_data, access_size, 1);
> +    acpi_data_push(table_data, address_size);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
> +     * can retrieve and read it. the address size is 64 bits.
> +     */
> +    bios_linker_loader_add_pointer(linker,
> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
> +}
It's mostly generic GAS structure with linker addition.
I'd suggest to reuse something like
 https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().

> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                                            BIOSLinker *linker)
> +{
> +    uint32_t ghes_start = table_data->len;
> +    uint32_t address_size, error_status_address_offset;
> +    uint32_t read_ack_register_offset, i;
> +
> +    address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
it's confusing name for var,
AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
also, I'm not sure why it's needed at all.

> +
> +    error_status_address_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    read_ack_register_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
> +        offsetof(struct AcpiGenericAddress, address);
it's really hard to get why you use offsetof() so much in this function,
to me above code totally unreadable.

> +    acpi_data_push(hardware_error,
> +        offsetof(struct hardware_errors_buffer, ack_value));
it looks like you are trying to build several tables within one function,
so it's hard to get what's going on.
I'd suggest to build separate table independently where it's possible.

i.e. build independent tables first
and only then build dependent tables passing to it pointers
to previously build table if necessary.

> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Initialize read ack register */
> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
> +
> +    /* Reserved the total size for ERRORS fw_cfg blob
> +     */
> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
> +
> +    /* Allocate guest memory for the Data fw_cfg blob */
> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
> +                            1, false);
> +    /* Reserve table header size */
> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
> +
> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
I'd suggest to use spec field name wit table prefix, ex:
ACPI_HEST_ERROR_SOURCE_COUNT

also, beside build_append_int_noprefix() you need to at least
add comment that exactly matches field from spec.

the same applies to other fields you are adding in this patch

> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
> +        build_append_int_noprefix(table_data,
> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
> +        /* source id */
> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
> +        /* related source id */
> +        build_append_int_noprefix(table_data, 0xffff, 2);
> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
> +
> +        /* Currently only enable SEA notification type to avoid the kernel
> +         * warning, reserve the space for other notification error source
> +         */
> +        if (i == ACPI_HEST_NOTIFY_SEA) {
> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
> +        } else {
> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
> +        }
> +
> +        /* The number of error status block per generic hardware error source */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max sections per record */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max raw data length */
> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
> +
> +        /* Build error status address*/
> +        build_address(table_data, linker, error_status_address_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
just do something like this instead of build_address():
build_append_gas()
bios_linker_loader_add_pointer()

also register width 0x40 looks suspicious, where does it come from?
While at it do you have a real hardware which has HEST table that you re trying to model after?
I'd like to see HEST and other related tables from it.

> +
> +        /* Hardware error notification structure */
> +        build_append_int_noprefix(table_data, i, 1); /* type */
> +        /* length */
> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
> +        build_append_int_noprefix(table_data, 0, 26);
> +
> +        /* Error Status Block Length */
> +        build_append_int_noprefix(table_data,
> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
> +
> +        /* Build read ack register */
> +        build_address(table_data, linker, read_ack_register_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2),
> +            offsetof(struct hardware_errors_buffer, ack_value) +
> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
> +            4 /* QWord access */);
> +
> +        /* Read ack preserve */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
> +
> +        /* Read ack write */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
> +    }
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Patch address of generic error status block into
> +         * the address register so OSPM can retrieve and read it.
> +         */
> +        bios_linker_loader_add_pointer(linker,
> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
> +            GHES_ERRORS_FW_CFG_FILE,
> +            offsetof(struct hardware_errors_buffer, gesb) +
> +            i * GHES_MAX_RAW_DATA_LENGTH);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
> +     * so QEMU can write the ERRORS there. The address is expected to be
> +     * < 4GB, but write 64 bits anyway.
> +     */
> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
> +        offsetof(struct hardware_errors_buffer, gesb));
> +
> +    build_header(linker, table_data,
> +        (void *)(table_data->data + ghes_start), "HEST",
> +        table_data->len - ghes_start, 1, NULL, "GHES");
> +}
> +
> +static GhesState ges;
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
> +{
> +
> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
> +
> +    /* Create a read-only fw_cfg file for GHES */
> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
> +                    size);
> +    /* Create a read-write fw_cfg file for Address */
> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
> +}
> +
> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
> +{
> +    uint64_t error_block_addr;
> +    uint64_t ack_value_addr, ack_value = 0;
> +    int loop = 0, ack_value_size;
> +    bool ret = GHES_CPER_FAIL;
> +
> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
> +        offsetof(struct hardware_errors_buffer, ack_value)) /
> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
> +
> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
> +        error_block_addr = le32_to_cpu(error_block_addr);
> +
> +        ack_value_addr = ges.ghes_addr_le -
> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
> +retry:
> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
> +        if (!ack_value) {
> +            if (loop < 3) {
> +                usleep(100 * 1000);
> +                loop++;
> +                goto retry;
> +            } else {
> +                error_report("Last time OSPM does not acknowledge the error,"
> +                    " record CPER failed this time, set the ack value to"
> +                    " avoid blocking next time CPER record! exit");
> +                ack_value = 1;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                return ret;
> +            }
> +        } else {
> +            /* A zero value in ghes_addr means that BIOS has not yet written
> +             * the address
> +             */
> +            if (error_block_addr) {
> +                ack_value = 0;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                ret = ghes_record_cper(error_block_addr, physical_address);
> +            }
> +        }
> +    }
> +    return ret;
> +}
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> index 3d78ff6..def1ec1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -45,6 +45,7 @@
>  #include "hw/arm/virt.h"
>  #include "sysemu/numa.h"
>  #include "kvm_arm.h"
> +#include "hw/acpi/hest_ghes.h"
>  
>  #define ARM_SPI_BASE 32
>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>      acpi_add_table(table_offsets, tables_blob);
>      build_spcr(tables_blob, tables->linker, vms);
>  
> +    acpi_add_table(table_offsets, tables_blob);
> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
> +
>      if (nb_numa_nodes > 0) {
>          acpi_add_table(table_offsets, tables_blob);
>          build_srat(tables_blob, tables->linker, vms);
> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>                      acpi_data_len(tables.tcpalog));
>  
> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
> +
>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>                                                ACPI_BUILD_RSDP_FILE, 0);
>  
> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
> index 88d0738..7f7b55c 100644
> --- a/include/hw/acpi/aml-build.h
> +++ b/include/hw/acpi/aml-build.h
> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>      GArray *rsdp;
>      GArray *tcpalog;
>      GArray *vmgenid;
> +    GArray *hardware_errors;
>      BIOSLinker *linker;
>  } AcpiBuildTables;
>  
> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
> new file mode 100644
> index 0000000..0772756
> --- /dev/null
> +++ b/include/hw/acpi/hest_ghes.h
> @@ -0,0 +1,47 @@
> +/*
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> +
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * Authors:
> + *   Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef ACPI_GHES_H
> +#define ACPI_GHES_H
> +
> +#include "hw/acpi/bios-linker-loader.h"
> +
> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
> +
> +#define GHES_GAS_ADDRESS_OFFSET              4
> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
> +#define GHES_NOTIFICATION_STRUCTURE          32
> +
> +#define GHES_CPER_OK   1
> +#define GHES_CPER_FAIL 0
> +
> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
> +/* The max size in Bytes for one error block */
> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
> +
> +
> +typedef struct GhesState {
> +    uint64_t ghes_addr_le;
> +} GhesState;
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                            BIOSLinker *linker);
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
> +#endif

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-29 10:20     ` Igor Mammedov
  0 siblings, 0 replies; 129+ messages in thread
From: Igor Mammedov @ 2017-08-29 10:20 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: mst, zhaoshenglong, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, zhengqiang10,
	wuquanming, huangshaoyu, linuxarm

On Fri, 18 Aug 2017 22:23:43 +0800
Dongjiu Geng <gengdongjiu@huawei.com> wrote:

> This implements APEI GHES Table by passing the error CPER info
> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
> will be injected into the guest OS.

it's a bit complex patch/functionality so I've just mosty skimmed and
commented only on structure of the patch and changes I'd like to see
so it would be more structured and review-able.

I'd suggest to add doc patch first which will describe how it's
supposed to work between QEMU/firmware/guest OS with expected
flows.
 
> Below is the table layout, the max number of error soure is 11,
> which is classified by notification type.
> 
>      etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
these diagram shows relations between tables which not necessarily bad
but as layout it's useless.
 * Probably there is not much sense to have HEST table here, it's described
   well enough in spec. You might just put reference here.
 * these diagrams should go into doc/spec patch
 * when you describe layout you need to show what and at what offsets
   in which order in which blob/file is located. See ACPI spec for example
   and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.

> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
> so user space must check the ack value to avoid read-write race condition.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  hw/acpi/aml-build.c         |   2 +
>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c    |   6 +
>  include/hw/acpi/aml-build.h |   1 +
>  include/hw/acpi/hest_ghes.h |  47 ++++++
>  5 files changed, 401 insertions(+)
>  create mode 100644 hw/acpi/hest_ghes.c
>  create mode 100644 include/hw/acpi/hest_ghes.h
> 
> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
> index 36a6cc4..6849e5f 100644
> --- a/hw/acpi/aml-build.c
> +++ b/hw/acpi/aml-build.c
> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>      tables->table_data = g_array_new(false, true /* clear */, 1);
>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>      tables->linker = bios_linker_loader_init();
>  }
>  
> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>      g_array_free(tables->table_data, true);
>      g_array_free(tables->tcpalog, mfre);
>      g_array_free(tables->vmgenid, mfre);
> +    g_array_free(tables->hardware_errors, mfre);
>  }
>  
>  /* Build rsdt table */
> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
> new file mode 100644
> index 0000000..ff6b5ef
> --- /dev/null
> +++ b/hw/acpi/hest_ghes.c
> @@ -0,0 +1,345 @@
> +/*
> + *  APEI GHES table Generation
> + *
> + *  Copyright (C) 2017 huawei.
> + *
> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qmp-commands.h"
> +#include "hw/acpi/acpi.h"
> +#include "hw/acpi/aml-build.h"
> +#include "hw/acpi/hest_ghes.h"
> +#include "hw/nvram/fw_cfg.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/error-report.h"
> +
> +/* The structure that stands for the layout
> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
> + *
> + *           etc/hardware_errors
> + * ==========================================
> + * +------------------+
> + * |    address       |              +--------------+
> + * |    registers     |              | Error Status |
> + * | +----------------+              | Data Block 0 |
> + * | |status_address0 |------------->| +------------+
> + * | +----------------+              | |  CPER      |
> + * | |status_address1 |----------+   | |  CPER      |
> + * | +----------------+          |   | |  ....      |
> + * | |.............   |          |   | |  CPER      |
> + * | +----------------+          |   | +------------+
> + * | |status_address10|-----+    |   | Error Status |
> + * | +----------------+     |    |   | Data Block 1 |
> + * | |ack_value0      |     |    +-->| +------------+
> + * | +----------------+     |        | |  CPER      |
> + * | |ack_value1      |     |        | |  CPER      |
> + * | +----------------+     |        | |  ....      |
> + * | | .............  |     |        | |  CPER      |
> + * | +----------------+     |        +-+------------+
> + * | |ack_value10     |     |        | |..........  |
> + * | +----------------+     |        | +------------+
> + *                          |        | Error Status |
> + *                          |        | Data Block10 |
> + *                          +------->+------------+
> + *                                   | |  CPER      |
> + *                                   | |  CPER      |
> + *                                   | |  ....      |
> + *                                   | |  CPER      |
> + *                                   +-+------------+
> + */
> +struct hardware_errors_buffer {
> +    /* Generic Error Status Block register */
> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
> +};
> +
> +static int ghes_record_cper(uint64_t error_block_address,
> +                                    uint64_t error_physical_addr)
> +{
> +    AcpiGenericErrorStatus block;
> +    AcpiGenericErrorData *gdata;
> +    UefiCperSecMemErr *mem_err;
> +    uint64_t current_block_length;
> +    unsigned char *buffer;
> +    /* memory section */
> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
> +                              0x83, 0xB1};
> +
> +    cpu_physical_memory_read(error_block_address, &block,
> +                                sizeof(AcpiGenericErrorStatus));
> +
> +    /* Get the current generic error status block length */
> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
> +        le32_to_cpu(block.data_length);
> +
> +    /* If the Generic Error Status Block is NULL, update
> +     * the block header
> +     */
> +    if (!block.block_status) {
> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    }
> +
> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    /* check whether it runs out of the preallocated memory */
> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
> +       GHES_MAX_RAW_DATA_LENGTH) {
> +        error_report("Record CPER out of boundary!!!");
> +        return GHES_CPER_FAIL;
> +    }
> +
> +    /* Write back the Generic Error Status Block to guest memory */
> +    cpu_physical_memory_write(error_block_address, &block,
> +        sizeof(AcpiGenericErrorStatus));
> +
> +    /* Fill in Generic Error Data Entry */
> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
> +                       sizeof(UefiCperSecMemErr));
> +
> +
> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
looks redundant, g_malloc0 does it for you

> +    gdata = (AcpiGenericErrorData *)buffer;
> +
> +    /* Memory section */
> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
> +            sizeof(mem_section_id_le));
> +
> +    /* error severity is recoverable */
> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    gdata->revision = 0x300; /* the revision number is 0x300 */
> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
> +
> +    /* User space only handle the memory section CPER */
> +
> +    /* Hard code to Multi-bit ECC error */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
> +
> +    /* Record the physical address at which the memory error occurred */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);

I'd prefer for you to use build_append_int_noprefix() API to compose
whole error status block

and try to get rid of most structures you introduce in patch 1/6,
as they will be left unused after that.

> +
> +    /* Write back the Generic Error Data Entry to guest memory */
> +    cpu_physical_memory_write(error_block_address + current_block_length,
> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +
> +    g_free(buffer);
> +    return GHES_CPER_OK;
> +}
> +
> +static void
> +build_address(GArray *table_data, BIOSLinker *linker,
> +    uint32_t dst_patched_offset, uint32_t src_offset,
> +    uint8_t address_space_id , uint8_t  register_bit_width,
> +    uint8_t register_bit_offset, uint8_t access_size)
> +{
> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    /* Address space */
> +    build_append_int_noprefix(table_data, address_space_id, 1);
> +    /* register bit width */
> +    build_append_int_noprefix(table_data, register_bit_width, 1);
> +    /* register bit offset */
> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
> +    /* access size */
> +    build_append_int_noprefix(table_data, access_size, 1);
> +    acpi_data_push(table_data, address_size);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
> +     * can retrieve and read it. the address size is 64 bits.
> +     */
> +    bios_linker_loader_add_pointer(linker,
> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
> +}
It's mostly generic GAS structure with linker addition.
I'd suggest to reuse something like
 https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().

> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                                            BIOSLinker *linker)
> +{
> +    uint32_t ghes_start = table_data->len;
> +    uint32_t address_size, error_status_address_offset;
> +    uint32_t read_ack_register_offset, i;
> +
> +    address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
it's confusing name for var,
AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
also, I'm not sure why it's needed at all.

> +
> +    error_status_address_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    read_ack_register_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
> +        offsetof(struct AcpiGenericAddress, address);
it's really hard to get why you use offsetof() so much in this function,
to me above code totally unreadable.

> +    acpi_data_push(hardware_error,
> +        offsetof(struct hardware_errors_buffer, ack_value));
it looks like you are trying to build several tables within one function,
so it's hard to get what's going on.
I'd suggest to build separate table independently where it's possible.

i.e. build independent tables first
and only then build dependent tables passing to it pointers
to previously build table if necessary.

> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Initialize read ack register */
> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
> +
> +    /* Reserved the total size for ERRORS fw_cfg blob
> +     */
> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
> +
> +    /* Allocate guest memory for the Data fw_cfg blob */
> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
> +                            1, false);
> +    /* Reserve table header size */
> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
> +
> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
I'd suggest to use spec field name wit table prefix, ex:
ACPI_HEST_ERROR_SOURCE_COUNT

also, beside build_append_int_noprefix() you need to at least
add comment that exactly matches field from spec.

the same applies to other fields you are adding in this patch

> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
> +        build_append_int_noprefix(table_data,
> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
> +        /* source id */
> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
> +        /* related source id */
> +        build_append_int_noprefix(table_data, 0xffff, 2);
> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
> +
> +        /* Currently only enable SEA notification type to avoid the kernel
> +         * warning, reserve the space for other notification error source
> +         */
> +        if (i == ACPI_HEST_NOTIFY_SEA) {
> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
> +        } else {
> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
> +        }
> +
> +        /* The number of error status block per generic hardware error source */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max sections per record */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max raw data length */
> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
> +
> +        /* Build error status address*/
> +        build_address(table_data, linker, error_status_address_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
just do something like this instead of build_address():
build_append_gas()
bios_linker_loader_add_pointer()

also register width 0x40 looks suspicious, where does it come from?
While at it do you have a real hardware which has HEST table that you re trying to model after?
I'd like to see HEST and other related tables from it.

> +
> +        /* Hardware error notification structure */
> +        build_append_int_noprefix(table_data, i, 1); /* type */
> +        /* length */
> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
> +        build_append_int_noprefix(table_data, 0, 26);
> +
> +        /* Error Status Block Length */
> +        build_append_int_noprefix(table_data,
> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
> +
> +        /* Build read ack register */
> +        build_address(table_data, linker, read_ack_register_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2),
> +            offsetof(struct hardware_errors_buffer, ack_value) +
> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
> +            4 /* QWord access */);
> +
> +        /* Read ack preserve */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
> +
> +        /* Read ack write */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
> +    }
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Patch address of generic error status block into
> +         * the address register so OSPM can retrieve and read it.
> +         */
> +        bios_linker_loader_add_pointer(linker,
> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
> +            GHES_ERRORS_FW_CFG_FILE,
> +            offsetof(struct hardware_errors_buffer, gesb) +
> +            i * GHES_MAX_RAW_DATA_LENGTH);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
> +     * so QEMU can write the ERRORS there. The address is expected to be
> +     * < 4GB, but write 64 bits anyway.
> +     */
> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
> +        offsetof(struct hardware_errors_buffer, gesb));
> +
> +    build_header(linker, table_data,
> +        (void *)(table_data->data + ghes_start), "HEST",
> +        table_data->len - ghes_start, 1, NULL, "GHES");
> +}
> +
> +static GhesState ges;
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
> +{
> +
> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
> +
> +    /* Create a read-only fw_cfg file for GHES */
> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
> +                    size);
> +    /* Create a read-write fw_cfg file for Address */
> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
> +}
> +
> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
> +{
> +    uint64_t error_block_addr;
> +    uint64_t ack_value_addr, ack_value = 0;
> +    int loop = 0, ack_value_size;
> +    bool ret = GHES_CPER_FAIL;
> +
> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
> +        offsetof(struct hardware_errors_buffer, ack_value)) /
> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
> +
> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
> +        error_block_addr = le32_to_cpu(error_block_addr);
> +
> +        ack_value_addr = ges.ghes_addr_le -
> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
> +retry:
> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
> +        if (!ack_value) {
> +            if (loop < 3) {
> +                usleep(100 * 1000);
> +                loop++;
> +                goto retry;
> +            } else {
> +                error_report("Last time OSPM does not acknowledge the error,"
> +                    " record CPER failed this time, set the ack value to"
> +                    " avoid blocking next time CPER record! exit");
> +                ack_value = 1;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                return ret;
> +            }
> +        } else {
> +            /* A zero value in ghes_addr means that BIOS has not yet written
> +             * the address
> +             */
> +            if (error_block_addr) {
> +                ack_value = 0;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                ret = ghes_record_cper(error_block_addr, physical_address);
> +            }
> +        }
> +    }
> +    return ret;
> +}
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> index 3d78ff6..def1ec1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -45,6 +45,7 @@
>  #include "hw/arm/virt.h"
>  #include "sysemu/numa.h"
>  #include "kvm_arm.h"
> +#include "hw/acpi/hest_ghes.h"
>  
>  #define ARM_SPI_BASE 32
>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>      acpi_add_table(table_offsets, tables_blob);
>      build_spcr(tables_blob, tables->linker, vms);
>  
> +    acpi_add_table(table_offsets, tables_blob);
> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
> +
>      if (nb_numa_nodes > 0) {
>          acpi_add_table(table_offsets, tables_blob);
>          build_srat(tables_blob, tables->linker, vms);
> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>                      acpi_data_len(tables.tcpalog));
>  
> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
> +
>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>                                                ACPI_BUILD_RSDP_FILE, 0);
>  
> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
> index 88d0738..7f7b55c 100644
> --- a/include/hw/acpi/aml-build.h
> +++ b/include/hw/acpi/aml-build.h
> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>      GArray *rsdp;
>      GArray *tcpalog;
>      GArray *vmgenid;
> +    GArray *hardware_errors;
>      BIOSLinker *linker;
>  } AcpiBuildTables;
>  
> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
> new file mode 100644
> index 0000000..0772756
> --- /dev/null
> +++ b/include/hw/acpi/hest_ghes.h
> @@ -0,0 +1,47 @@
> +/*
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> +
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * Authors:
> + *   Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef ACPI_GHES_H
> +#define ACPI_GHES_H
> +
> +#include "hw/acpi/bios-linker-loader.h"
> +
> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
> +
> +#define GHES_GAS_ADDRESS_OFFSET              4
> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
> +#define GHES_NOTIFICATION_STRUCTURE          32
> +
> +#define GHES_CPER_OK   1
> +#define GHES_CPER_FAIL 0
> +
> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
> +/* The max size in Bytes for one error block */
> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
> +
> +
> +typedef struct GhesState {
> +    uint64_t ghes_addr_le;
> +} GhesState;
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                            BIOSLinker *linker);
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
> +#endif

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-29 10:20     ` Igor Mammedov
  0 siblings, 0 replies; 129+ messages in thread
From: Igor Mammedov @ 2017-08-29 10:20 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: mst, zhaoshenglong, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, zhengqiang10,
	wuquanming

On Fri, 18 Aug 2017 22:23:43 +0800
Dongjiu Geng <gengdongjiu@huawei.com> wrote:

> This implements APEI GHES Table by passing the error CPER info
> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
> will be injected into the guest OS.

it's a bit complex patch/functionality so I've just mosty skimmed and
commented only on structure of the patch and changes I'd like to see
so it would be more structured and review-able.

I'd suggest to add doc patch first which will describe how it's
supposed to work between QEMU/firmware/guest OS with expected
flows.
 
> Below is the table layout, the max number of error soure is 11,
> which is classified by notification type.
> 
>      etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
these diagram shows relations between tables which not necessarily bad
but as layout it's useless.
 * Probably there is not much sense to have HEST table here, it's described
   well enough in spec. You might just put reference here.
 * these diagrams should go into doc/spec patch
 * when you describe layout you need to show what and at what offsets
   in which order in which blob/file is located. See ACPI spec for example
   and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.

> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
> so user space must check the ack value to avoid read-write race condition.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  hw/acpi/aml-build.c         |   2 +
>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c    |   6 +
>  include/hw/acpi/aml-build.h |   1 +
>  include/hw/acpi/hest_ghes.h |  47 ++++++
>  5 files changed, 401 insertions(+)
>  create mode 100644 hw/acpi/hest_ghes.c
>  create mode 100644 include/hw/acpi/hest_ghes.h
> 
> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
> index 36a6cc4..6849e5f 100644
> --- a/hw/acpi/aml-build.c
> +++ b/hw/acpi/aml-build.c
> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>      tables->table_data = g_array_new(false, true /* clear */, 1);
>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>      tables->linker = bios_linker_loader_init();
>  }
>  
> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>      g_array_free(tables->table_data, true);
>      g_array_free(tables->tcpalog, mfre);
>      g_array_free(tables->vmgenid, mfre);
> +    g_array_free(tables->hardware_errors, mfre);
>  }
>  
>  /* Build rsdt table */
> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
> new file mode 100644
> index 0000000..ff6b5ef
> --- /dev/null
> +++ b/hw/acpi/hest_ghes.c
> @@ -0,0 +1,345 @@
> +/*
> + *  APEI GHES table Generation
> + *
> + *  Copyright (C) 2017 huawei.
> + *
> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qmp-commands.h"
> +#include "hw/acpi/acpi.h"
> +#include "hw/acpi/aml-build.h"
> +#include "hw/acpi/hest_ghes.h"
> +#include "hw/nvram/fw_cfg.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/error-report.h"
> +
> +/* The structure that stands for the layout
> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
> + *
> + *           etc/hardware_errors
> + * ==========================================
> + * +------------------+
> + * |    address       |              +--------------+
> + * |    registers     |              | Error Status |
> + * | +----------------+              | Data Block 0 |
> + * | |status_address0 |------------->| +------------+
> + * | +----------------+              | |  CPER      |
> + * | |status_address1 |----------+   | |  CPER      |
> + * | +----------------+          |   | |  ....      |
> + * | |.............   |          |   | |  CPER      |
> + * | +----------------+          |   | +------------+
> + * | |status_address10|-----+    |   | Error Status |
> + * | +----------------+     |    |   | Data Block 1 |
> + * | |ack_value0      |     |    +-->| +------------+
> + * | +----------------+     |        | |  CPER      |
> + * | |ack_value1      |     |        | |  CPER      |
> + * | +----------------+     |        | |  ....      |
> + * | | .............  |     |        | |  CPER      |
> + * | +----------------+     |        +-+------------+
> + * | |ack_value10     |     |        | |..........  |
> + * | +----------------+     |        | +------------+
> + *                          |        | Error Status |
> + *                          |        | Data Block10 |
> + *                          +------->+------------+
> + *                                   | |  CPER      |
> + *                                   | |  CPER      |
> + *                                   | |  ....      |
> + *                                   | |  CPER      |
> + *                                   +-+------------+
> + */
> +struct hardware_errors_buffer {
> +    /* Generic Error Status Block register */
> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
> +};
> +
> +static int ghes_record_cper(uint64_t error_block_address,
> +                                    uint64_t error_physical_addr)
> +{
> +    AcpiGenericErrorStatus block;
> +    AcpiGenericErrorData *gdata;
> +    UefiCperSecMemErr *mem_err;
> +    uint64_t current_block_length;
> +    unsigned char *buffer;
> +    /* memory section */
> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
> +                              0x83, 0xB1};
> +
> +    cpu_physical_memory_read(error_block_address, &block,
> +                                sizeof(AcpiGenericErrorStatus));
> +
> +    /* Get the current generic error status block length */
> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
> +        le32_to_cpu(block.data_length);
> +
> +    /* If the Generic Error Status Block is NULL, update
> +     * the block header
> +     */
> +    if (!block.block_status) {
> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    }
> +
> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    /* check whether it runs out of the preallocated memory */
> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
> +       GHES_MAX_RAW_DATA_LENGTH) {
> +        error_report("Record CPER out of boundary!!!");
> +        return GHES_CPER_FAIL;
> +    }
> +
> +    /* Write back the Generic Error Status Block to guest memory */
> +    cpu_physical_memory_write(error_block_address, &block,
> +        sizeof(AcpiGenericErrorStatus));
> +
> +    /* Fill in Generic Error Data Entry */
> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
> +                       sizeof(UefiCperSecMemErr));
> +
> +
> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
looks redundant, g_malloc0 does it for you

> +    gdata = (AcpiGenericErrorData *)buffer;
> +
> +    /* Memory section */
> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
> +            sizeof(mem_section_id_le));
> +
> +    /* error severity is recoverable */
> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    gdata->revision = 0x300; /* the revision number is 0x300 */
> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
> +
> +    /* User space only handle the memory section CPER */
> +
> +    /* Hard code to Multi-bit ECC error */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
> +
> +    /* Record the physical address at which the memory error occurred */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);

I'd prefer for you to use build_append_int_noprefix() API to compose
whole error status block

and try to get rid of most structures you introduce in patch 1/6,
as they will be left unused after that.

> +
> +    /* Write back the Generic Error Data Entry to guest memory */
> +    cpu_physical_memory_write(error_block_address + current_block_length,
> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +
> +    g_free(buffer);
> +    return GHES_CPER_OK;
> +}
> +
> +static void
> +build_address(GArray *table_data, BIOSLinker *linker,
> +    uint32_t dst_patched_offset, uint32_t src_offset,
> +    uint8_t address_space_id , uint8_t  register_bit_width,
> +    uint8_t register_bit_offset, uint8_t access_size)
> +{
> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    /* Address space */
> +    build_append_int_noprefix(table_data, address_space_id, 1);
> +    /* register bit width */
> +    build_append_int_noprefix(table_data, register_bit_width, 1);
> +    /* register bit offset */
> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
> +    /* access size */
> +    build_append_int_noprefix(table_data, access_size, 1);
> +    acpi_data_push(table_data, address_size);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
> +     * can retrieve and read it. the address size is 64 bits.
> +     */
> +    bios_linker_loader_add_pointer(linker,
> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
> +}
It's mostly generic GAS structure with linker addition.
I'd suggest to reuse something like
 https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().

> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                                            BIOSLinker *linker)
> +{
> +    uint32_t ghes_start = table_data->len;
> +    uint32_t address_size, error_status_address_offset;
> +    uint32_t read_ack_register_offset, i;
> +
> +    address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
it's confusing name for var,
AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
also, I'm not sure why it's needed at all.

> +
> +    error_status_address_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    read_ack_register_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
> +        offsetof(struct AcpiGenericAddress, address);
it's really hard to get why you use offsetof() so much in this function,
to me above code totally unreadable.

> +    acpi_data_push(hardware_error,
> +        offsetof(struct hardware_errors_buffer, ack_value));
it looks like you are trying to build several tables within one function,
so it's hard to get what's going on.
I'd suggest to build separate table independently where it's possible.

i.e. build independent tables first
and only then build dependent tables passing to it pointers
to previously build table if necessary.

> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Initialize read ack register */
> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
> +
> +    /* Reserved the total size for ERRORS fw_cfg blob
> +     */
> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
> +
> +    /* Allocate guest memory for the Data fw_cfg blob */
> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
> +                            1, false);
> +    /* Reserve table header size */
> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
> +
> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
I'd suggest to use spec field name wit table prefix, ex:
ACPI_HEST_ERROR_SOURCE_COUNT

also, beside build_append_int_noprefix() you need to at least
add comment that exactly matches field from spec.

the same applies to other fields you are adding in this patch

> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
> +        build_append_int_noprefix(table_data,
> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
> +        /* source id */
> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
> +        /* related source id */
> +        build_append_int_noprefix(table_data, 0xffff, 2);
> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
> +
> +        /* Currently only enable SEA notification type to avoid the kernel
> +         * warning, reserve the space for other notification error source
> +         */
> +        if (i == ACPI_HEST_NOTIFY_SEA) {
> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
> +        } else {
> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
> +        }
> +
> +        /* The number of error status block per generic hardware error source */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max sections per record */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max raw data length */
> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
> +
> +        /* Build error status address*/
> +        build_address(table_data, linker, error_status_address_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
just do something like this instead of build_address():
build_append_gas()
bios_linker_loader_add_pointer()

also register width 0x40 looks suspicious, where does it come from?
While at it do you have a real hardware which has HEST table that you re trying to model after?
I'd like to see HEST and other related tables from it.

> +
> +        /* Hardware error notification structure */
> +        build_append_int_noprefix(table_data, i, 1); /* type */
> +        /* length */
> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
> +        build_append_int_noprefix(table_data, 0, 26);
> +
> +        /* Error Status Block Length */
> +        build_append_int_noprefix(table_data,
> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
> +
> +        /* Build read ack register */
> +        build_address(table_data, linker, read_ack_register_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2),
> +            offsetof(struct hardware_errors_buffer, ack_value) +
> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
> +            4 /* QWord access */);
> +
> +        /* Read ack preserve */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
> +
> +        /* Read ack write */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
> +    }
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Patch address of generic error status block into
> +         * the address register so OSPM can retrieve and read it.
> +         */
> +        bios_linker_loader_add_pointer(linker,
> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
> +            GHES_ERRORS_FW_CFG_FILE,
> +            offsetof(struct hardware_errors_buffer, gesb) +
> +            i * GHES_MAX_RAW_DATA_LENGTH);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
> +     * so QEMU can write the ERRORS there. The address is expected to be
> +     * < 4GB, but write 64 bits anyway.
> +     */
> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
> +        offsetof(struct hardware_errors_buffer, gesb));
> +
> +    build_header(linker, table_data,
> +        (void *)(table_data->data + ghes_start), "HEST",
> +        table_data->len - ghes_start, 1, NULL, "GHES");
> +}
> +
> +static GhesState ges;
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
> +{
> +
> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
> +
> +    /* Create a read-only fw_cfg file for GHES */
> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
> +                    size);
> +    /* Create a read-write fw_cfg file for Address */
> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
> +}
> +
> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
> +{
> +    uint64_t error_block_addr;
> +    uint64_t ack_value_addr, ack_value = 0;
> +    int loop = 0, ack_value_size;
> +    bool ret = GHES_CPER_FAIL;
> +
> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
> +        offsetof(struct hardware_errors_buffer, ack_value)) /
> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
> +
> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
> +        error_block_addr = le32_to_cpu(error_block_addr);
> +
> +        ack_value_addr = ges.ghes_addr_le -
> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
> +retry:
> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
> +        if (!ack_value) {
> +            if (loop < 3) {
> +                usleep(100 * 1000);
> +                loop++;
> +                goto retry;
> +            } else {
> +                error_report("Last time OSPM does not acknowledge the error,"
> +                    " record CPER failed this time, set the ack value to"
> +                    " avoid blocking next time CPER record! exit");
> +                ack_value = 1;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                return ret;
> +            }
> +        } else {
> +            /* A zero value in ghes_addr means that BIOS has not yet written
> +             * the address
> +             */
> +            if (error_block_addr) {
> +                ack_value = 0;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                ret = ghes_record_cper(error_block_addr, physical_address);
> +            }
> +        }
> +    }
> +    return ret;
> +}
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> index 3d78ff6..def1ec1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -45,6 +45,7 @@
>  #include "hw/arm/virt.h"
>  #include "sysemu/numa.h"
>  #include "kvm_arm.h"
> +#include "hw/acpi/hest_ghes.h"
>  
>  #define ARM_SPI_BASE 32
>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>      acpi_add_table(table_offsets, tables_blob);
>      build_spcr(tables_blob, tables->linker, vms);
>  
> +    acpi_add_table(table_offsets, tables_blob);
> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
> +
>      if (nb_numa_nodes > 0) {
>          acpi_add_table(table_offsets, tables_blob);
>          build_srat(tables_blob, tables->linker, vms);
> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>                      acpi_data_len(tables.tcpalog));
>  
> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
> +
>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>                                                ACPI_BUILD_RSDP_FILE, 0);
>  
> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
> index 88d0738..7f7b55c 100644
> --- a/include/hw/acpi/aml-build.h
> +++ b/include/hw/acpi/aml-build.h
> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>      GArray *rsdp;
>      GArray *tcpalog;
>      GArray *vmgenid;
> +    GArray *hardware_errors;
>      BIOSLinker *linker;
>  } AcpiBuildTables;
>  
> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
> new file mode 100644
> index 0000000..0772756
> --- /dev/null
> +++ b/include/hw/acpi/hest_ghes.h
> @@ -0,0 +1,47 @@
> +/*
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> +
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * Authors:
> + *   Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef ACPI_GHES_H
> +#define ACPI_GHES_H
> +
> +#include "hw/acpi/bios-linker-loader.h"
> +
> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
> +
> +#define GHES_GAS_ADDRESS_OFFSET              4
> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
> +#define GHES_NOTIFICATION_STRUCTURE          32
> +
> +#define GHES_CPER_OK   1
> +#define GHES_CPER_FAIL 0
> +
> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
> +/* The max size in Bytes for one error block */
> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
> +
> +
> +typedef struct GhesState {
> +    uint64_t ghes_addr_le;
> +} GhesState;
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                            BIOSLinker *linker);
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
> +#endif

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-29 10:20     ` Igor Mammedov
  0 siblings, 0 replies; 129+ messages in thread
From: Igor Mammedov @ 2017-08-29 10:20 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: mst, zhaoshenglong, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, zhengqiang10,
	wuquanming, huangshaoyu, linuxarm

On Fri, 18 Aug 2017 22:23:43 +0800
Dongjiu Geng <gengdongjiu@huawei.com> wrote:

> This implements APEI GHES Table by passing the error CPER info
> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
> will be injected into the guest OS.

it's a bit complex patch/functionality so I've just mosty skimmed and
commented only on structure of the patch and changes I'd like to see
so it would be more structured and review-able.

I'd suggest to add doc patch first which will describe how it's
supposed to work between QEMU/firmware/guest OS with expected
flows.
 
> Below is the table layout, the max number of error soure is 11,
> which is classified by notification type.
> 
>      etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
these diagram shows relations between tables which not necessarily bad
but as layout it's useless.
 * Probably there is not much sense to have HEST table here, it's described
   well enough in spec. You might just put reference here.
 * these diagrams should go into doc/spec patch
 * when you describe layout you need to show what and at what offsets
   in which order in which blob/file is located. See ACPI spec for example
   and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.

> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
> so user space must check the ack value to avoid read-write race condition.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  hw/acpi/aml-build.c         |   2 +
>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c    |   6 +
>  include/hw/acpi/aml-build.h |   1 +
>  include/hw/acpi/hest_ghes.h |  47 ++++++
>  5 files changed, 401 insertions(+)
>  create mode 100644 hw/acpi/hest_ghes.c
>  create mode 100644 include/hw/acpi/hest_ghes.h
> 
> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
> index 36a6cc4..6849e5f 100644
> --- a/hw/acpi/aml-build.c
> +++ b/hw/acpi/aml-build.c
> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>      tables->table_data = g_array_new(false, true /* clear */, 1);
>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>      tables->linker = bios_linker_loader_init();
>  }
>  
> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>      g_array_free(tables->table_data, true);
>      g_array_free(tables->tcpalog, mfre);
>      g_array_free(tables->vmgenid, mfre);
> +    g_array_free(tables->hardware_errors, mfre);
>  }
>  
>  /* Build rsdt table */
> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
> new file mode 100644
> index 0000000..ff6b5ef
> --- /dev/null
> +++ b/hw/acpi/hest_ghes.c
> @@ -0,0 +1,345 @@
> +/*
> + *  APEI GHES table Generation
> + *
> + *  Copyright (C) 2017 huawei.
> + *
> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qmp-commands.h"
> +#include "hw/acpi/acpi.h"
> +#include "hw/acpi/aml-build.h"
> +#include "hw/acpi/hest_ghes.h"
> +#include "hw/nvram/fw_cfg.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/error-report.h"
> +
> +/* The structure that stands for the layout
> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
> + *
> + *           etc/hardware_errors
> + * ==========================================
> + * +------------------+
> + * |    address       |              +--------------+
> + * |    registers     |              | Error Status |
> + * | +----------------+              | Data Block 0 |
> + * | |status_address0 |------------->| +------------+
> + * | +----------------+              | |  CPER      |
> + * | |status_address1 |----------+   | |  CPER      |
> + * | +----------------+          |   | |  ....      |
> + * | |.............   |          |   | |  CPER      |
> + * | +----------------+          |   | +------------+
> + * | |status_address10|-----+    |   | Error Status |
> + * | +----------------+     |    |   | Data Block 1 |
> + * | |ack_value0      |     |    +-->| +------------+
> + * | +----------------+     |        | |  CPER      |
> + * | |ack_value1      |     |        | |  CPER      |
> + * | +----------------+     |        | |  ....      |
> + * | | .............  |     |        | |  CPER      |
> + * | +----------------+     |        +-+------------+
> + * | |ack_value10     |     |        | |..........  |
> + * | +----------------+     |        | +------------+
> + *                          |        | Error Status |
> + *                          |        | Data Block10 |
> + *                          +------->+------------+
> + *                                   | |  CPER      |
> + *                                   | |  CPER      |
> + *                                   | |  ....      |
> + *                                   | |  CPER      |
> + *                                   +-+------------+
> + */
> +struct hardware_errors_buffer {
> +    /* Generic Error Status Block register */
> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
> +};
> +
> +static int ghes_record_cper(uint64_t error_block_address,
> +                                    uint64_t error_physical_addr)
> +{
> +    AcpiGenericErrorStatus block;
> +    AcpiGenericErrorData *gdata;
> +    UefiCperSecMemErr *mem_err;
> +    uint64_t current_block_length;
> +    unsigned char *buffer;
> +    /* memory section */
> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
> +                              0x83, 0xB1};
> +
> +    cpu_physical_memory_read(error_block_address, &block,
> +                                sizeof(AcpiGenericErrorStatus));
> +
> +    /* Get the current generic error status block length */
> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
> +        le32_to_cpu(block.data_length);
> +
> +    /* If the Generic Error Status Block is NULL, update
> +     * the block header
> +     */
> +    if (!block.block_status) {
> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    }
> +
> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    /* check whether it runs out of the preallocated memory */
> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
> +       GHES_MAX_RAW_DATA_LENGTH) {
> +        error_report("Record CPER out of boundary!!!");
> +        return GHES_CPER_FAIL;
> +    }
> +
> +    /* Write back the Generic Error Status Block to guest memory */
> +    cpu_physical_memory_write(error_block_address, &block,
> +        sizeof(AcpiGenericErrorStatus));
> +
> +    /* Fill in Generic Error Data Entry */
> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
> +                       sizeof(UefiCperSecMemErr));
> +
> +
> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
looks redundant, g_malloc0 does it for you

> +    gdata = (AcpiGenericErrorData *)buffer;
> +
> +    /* Memory section */
> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
> +            sizeof(mem_section_id_le));
> +
> +    /* error severity is recoverable */
> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    gdata->revision = 0x300; /* the revision number is 0x300 */
> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
> +
> +    /* User space only handle the memory section CPER */
> +
> +    /* Hard code to Multi-bit ECC error */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
> +
> +    /* Record the physical address at which the memory error occurred */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);

I'd prefer for you to use build_append_int_noprefix() API to compose
whole error status block

and try to get rid of most structures you introduce in patch 1/6,
as they will be left unused after that.

> +
> +    /* Write back the Generic Error Data Entry to guest memory */
> +    cpu_physical_memory_write(error_block_address + current_block_length,
> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +
> +    g_free(buffer);
> +    return GHES_CPER_OK;
> +}
> +
> +static void
> +build_address(GArray *table_data, BIOSLinker *linker,
> +    uint32_t dst_patched_offset, uint32_t src_offset,
> +    uint8_t address_space_id , uint8_t  register_bit_width,
> +    uint8_t register_bit_offset, uint8_t access_size)
> +{
> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    /* Address space */
> +    build_append_int_noprefix(table_data, address_space_id, 1);
> +    /* register bit width */
> +    build_append_int_noprefix(table_data, register_bit_width, 1);
> +    /* register bit offset */
> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
> +    /* access size */
> +    build_append_int_noprefix(table_data, access_size, 1);
> +    acpi_data_push(table_data, address_size);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
> +     * can retrieve and read it. the address size is 64 bits.
> +     */
> +    bios_linker_loader_add_pointer(linker,
> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
> +}
It's mostly generic GAS structure with linker addition.
I'd suggest to reuse something like
 https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().

> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                                            BIOSLinker *linker)
> +{
> +    uint32_t ghes_start = table_data->len;
> +    uint32_t address_size, error_status_address_offset;
> +    uint32_t read_ack_register_offset, i;
> +
> +    address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
it's confusing name for var,
AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
also, I'm not sure why it's needed at all.

> +
> +    error_status_address_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    read_ack_register_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
> +        offsetof(struct AcpiGenericAddress, address);
it's really hard to get why you use offsetof() so much in this function,
to me above code totally unreadable.

> +    acpi_data_push(hardware_error,
> +        offsetof(struct hardware_errors_buffer, ack_value));
it looks like you are trying to build several tables within one function,
so it's hard to get what's going on.
I'd suggest to build separate table independently where it's possible.

i.e. build independent tables first
and only then build dependent tables passing to it pointers
to previously build table if necessary.

> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Initialize read ack register */
> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
> +
> +    /* Reserved the total size for ERRORS fw_cfg blob
> +     */
> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
> +
> +    /* Allocate guest memory for the Data fw_cfg blob */
> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
> +                            1, false);
> +    /* Reserve table header size */
> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
> +
> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
I'd suggest to use spec field name wit table prefix, ex:
ACPI_HEST_ERROR_SOURCE_COUNT

also, beside build_append_int_noprefix() you need to at least
add comment that exactly matches field from spec.

the same applies to other fields you are adding in this patch

> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
> +        build_append_int_noprefix(table_data,
> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
> +        /* source id */
> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
> +        /* related source id */
> +        build_append_int_noprefix(table_data, 0xffff, 2);
> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
> +
> +        /* Currently only enable SEA notification type to avoid the kernel
> +         * warning, reserve the space for other notification error source
> +         */
> +        if (i == ACPI_HEST_NOTIFY_SEA) {
> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
> +        } else {
> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
> +        }
> +
> +        /* The number of error status block per generic hardware error source */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max sections per record */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max raw data length */
> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
> +
> +        /* Build error status address*/
> +        build_address(table_data, linker, error_status_address_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
just do something like this instead of build_address():
build_append_gas()
bios_linker_loader_add_pointer()

also register width 0x40 looks suspicious, where does it come from?
While at it do you have a real hardware which has HEST table that you re trying to model after?
I'd like to see HEST and other related tables from it.

> +
> +        /* Hardware error notification structure */
> +        build_append_int_noprefix(table_data, i, 1); /* type */
> +        /* length */
> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
> +        build_append_int_noprefix(table_data, 0, 26);
> +
> +        /* Error Status Block Length */
> +        build_append_int_noprefix(table_data,
> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
> +
> +        /* Build read ack register */
> +        build_address(table_data, linker, read_ack_register_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2),
> +            offsetof(struct hardware_errors_buffer, ack_value) +
> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
> +            4 /* QWord access */);
> +
> +        /* Read ack preserve */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
> +
> +        /* Read ack write */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
> +    }
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Patch address of generic error status block into
> +         * the address register so OSPM can retrieve and read it.
> +         */
> +        bios_linker_loader_add_pointer(linker,
> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
> +            GHES_ERRORS_FW_CFG_FILE,
> +            offsetof(struct hardware_errors_buffer, gesb) +
> +            i * GHES_MAX_RAW_DATA_LENGTH);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
> +     * so QEMU can write the ERRORS there. The address is expected to be
> +     * < 4GB, but write 64 bits anyway.
> +     */
> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
> +        offsetof(struct hardware_errors_buffer, gesb));
> +
> +    build_header(linker, table_data,
> +        (void *)(table_data->data + ghes_start), "HEST",
> +        table_data->len - ghes_start, 1, NULL, "GHES");
> +}
> +
> +static GhesState ges;
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
> +{
> +
> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
> +
> +    /* Create a read-only fw_cfg file for GHES */
> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
> +                    size);
> +    /* Create a read-write fw_cfg file for Address */
> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
> +}
> +
> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
> +{
> +    uint64_t error_block_addr;
> +    uint64_t ack_value_addr, ack_value = 0;
> +    int loop = 0, ack_value_size;
> +    bool ret = GHES_CPER_FAIL;
> +
> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
> +        offsetof(struct hardware_errors_buffer, ack_value)) /
> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
> +
> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
> +        error_block_addr = le32_to_cpu(error_block_addr);
> +
> +        ack_value_addr = ges.ghes_addr_le -
> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
> +retry:
> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
> +        if (!ack_value) {
> +            if (loop < 3) {
> +                usleep(100 * 1000);
> +                loop++;
> +                goto retry;
> +            } else {
> +                error_report("Last time OSPM does not acknowledge the error,"
> +                    " record CPER failed this time, set the ack value to"
> +                    " avoid blocking next time CPER record! exit");
> +                ack_value = 1;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                return ret;
> +            }
> +        } else {
> +            /* A zero value in ghes_addr means that BIOS has not yet written
> +             * the address
> +             */
> +            if (error_block_addr) {
> +                ack_value = 0;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                ret = ghes_record_cper(error_block_addr, physical_address);
> +            }
> +        }
> +    }
> +    return ret;
> +}
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> index 3d78ff6..def1ec1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -45,6 +45,7 @@
>  #include "hw/arm/virt.h"
>  #include "sysemu/numa.h"
>  #include "kvm_arm.h"
> +#include "hw/acpi/hest_ghes.h"
>  
>  #define ARM_SPI_BASE 32
>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>      acpi_add_table(table_offsets, tables_blob);
>      build_spcr(tables_blob, tables->linker, vms);
>  
> +    acpi_add_table(table_offsets, tables_blob);
> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
> +
>      if (nb_numa_nodes > 0) {
>          acpi_add_table(table_offsets, tables_blob);
>          build_srat(tables_blob, tables->linker, vms);
> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>                      acpi_data_len(tables.tcpalog));
>  
> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
> +
>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>                                                ACPI_BUILD_RSDP_FILE, 0);
>  
> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
> index 88d0738..7f7b55c 100644
> --- a/include/hw/acpi/aml-build.h
> +++ b/include/hw/acpi/aml-build.h
> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>      GArray *rsdp;
>      GArray *tcpalog;
>      GArray *vmgenid;
> +    GArray *hardware_errors;
>      BIOSLinker *linker;
>  } AcpiBuildTables;
>  
> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
> new file mode 100644
> index 0000000..0772756
> --- /dev/null
> +++ b/include/hw/acpi/hest_ghes.h
> @@ -0,0 +1,47 @@
> +/*
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> +
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * Authors:
> + *   Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef ACPI_GHES_H
> +#define ACPI_GHES_H
> +
> +#include "hw/acpi/bios-linker-loader.h"
> +
> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
> +
> +#define GHES_GAS_ADDRESS_OFFSET              4
> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
> +#define GHES_NOTIFICATION_STRUCTURE          32
> +
> +#define GHES_CPER_OK   1
> +#define GHES_CPER_FAIL 0
> +
> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
> +/* The max size in Bytes for one error block */
> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
> +
> +
> +typedef struct GhesState {
> +    uint64_t ghes_addr_le;
> +} GhesState;
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                            BIOSLinker *linker);
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
> +#endif

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-29 10:20     ` Igor Mammedov
  0 siblings, 0 replies; 129+ messages in thread
From: Igor Mammedov @ 2017-08-29 10:20 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, 18 Aug 2017 22:23:43 +0800
Dongjiu Geng <gengdongjiu@huawei.com> wrote:

> This implements APEI GHES Table by passing the error CPER info
> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
> will be injected into the guest OS.

it's a bit complex patch/functionality so I've just mosty skimmed and
commented only on structure of the patch and changes I'd like to see
so it would be more structured and review-able.

I'd suggest to add doc patch first which will describe how it's
supposed to work between QEMU/firmware/guest OS with expected
flows.
 
> Below is the table layout, the max number of error soure is 11,
> which is classified by notification type.
> 
>      etc/acpi/tables                               etc/hardware_errors
>     ====================                    ==========================================
> + +--------------------------+            +------------------+
> | | HEST                     |            |    address       |              +--------------+
> | +--------------------------+            |    registers     |              | Error Status |
> | | GHES0                    |            | +----------------+              | Data Block 0 |
> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
> | | .................        | |          | +----------------+              | |  CPER      |
> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
> | | .................        |   |        | +----------------+          |   | |  ....      |
> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
> + +--------------------------|     |   |                              |     | Error Status |
> | | ...............          |     |   |                              |     | Data Block 10|
> + +--------------------------+     |   |                              +---->| +------------+
> | | GHES10                   |     |   |                                    | |  CPER      |
> + +--------------------------+     |   |                                    | |  CPER      |
> | | .................        |     |   |                                    | |  ....      |
> | | error_status_address-----+-----+   |                                    | |  CPER      |
> | | .................        |         |                                    +-+------------+
> | | read_ack_register--------+---------+
> | | read_ack_preserve        |
> | | read_ack_write           |
> + +--------------------------+
these diagram shows relations between tables which not necessarily bad
but as layout it's useless.
 * Probably there is not much sense to have HEST table here, it's described
   well enough in spec. You might just put reference here.
 * these diagrams should go into doc/spec patch
 * when you describe layout you need to show what and at what offsets
   in which order in which blob/file is located. See ACPI spec for example
   and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.

> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
> so user space must check the ack value to avoid read-write race condition.
> 
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  hw/acpi/aml-build.c         |   2 +
>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/arm/virt-acpi-build.c    |   6 +
>  include/hw/acpi/aml-build.h |   1 +
>  include/hw/acpi/hest_ghes.h |  47 ++++++
>  5 files changed, 401 insertions(+)
>  create mode 100644 hw/acpi/hest_ghes.c
>  create mode 100644 include/hw/acpi/hest_ghes.h
> 
> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
> index 36a6cc4..6849e5f 100644
> --- a/hw/acpi/aml-build.c
> +++ b/hw/acpi/aml-build.c
> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>      tables->table_data = g_array_new(false, true /* clear */, 1);
>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>      tables->linker = bios_linker_loader_init();
>  }
>  
> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>      g_array_free(tables->table_data, true);
>      g_array_free(tables->tcpalog, mfre);
>      g_array_free(tables->vmgenid, mfre);
> +    g_array_free(tables->hardware_errors, mfre);
>  }
>  
>  /* Build rsdt table */
> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
> new file mode 100644
> index 0000000..ff6b5ef
> --- /dev/null
> +++ b/hw/acpi/hest_ghes.c
> @@ -0,0 +1,345 @@
> +/*
> + *  APEI GHES table Generation
> + *
> + *  Copyright (C) 2017 huawei.
> + *
> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qmp-commands.h"
> +#include "hw/acpi/acpi.h"
> +#include "hw/acpi/aml-build.h"
> +#include "hw/acpi/hest_ghes.h"
> +#include "hw/nvram/fw_cfg.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/error-report.h"
> +
> +/* The structure that stands for the layout
> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
> + *
> + *           etc/hardware_errors
> + * ==========================================
> + * +------------------+
> + * |    address       |              +--------------+
> + * |    registers     |              | Error Status |
> + * | +----------------+              | Data Block 0 |
> + * | |status_address0 |------------->| +------------+
> + * | +----------------+              | |  CPER      |
> + * | |status_address1 |----------+   | |  CPER      |
> + * | +----------------+          |   | |  ....      |
> + * | |.............   |          |   | |  CPER      |
> + * | +----------------+          |   | +------------+
> + * | |status_address10|-----+    |   | Error Status |
> + * | +----------------+     |    |   | Data Block 1 |
> + * | |ack_value0      |     |    +-->| +------------+
> + * | +----------------+     |        | |  CPER      |
> + * | |ack_value1      |     |        | |  CPER      |
> + * | +----------------+     |        | |  ....      |
> + * | | .............  |     |        | |  CPER      |
> + * | +----------------+     |        +-+------------+
> + * | |ack_value10     |     |        | |..........  |
> + * | +----------------+     |        | +------------+
> + *                          |        | Error Status |
> + *                          |        | Data Block10 |
> + *                          +------->+------------+
> + *                                   | |  CPER      |
> + *                                   | |  CPER      |
> + *                                   | |  ....      |
> + *                                   | |  CPER      |
> + *                                   +-+------------+
> + */
> +struct hardware_errors_buffer {
> +    /* Generic Error Status Block register */
> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
> +};
> +
> +static int ghes_record_cper(uint64_t error_block_address,
> +                                    uint64_t error_physical_addr)
> +{
> +    AcpiGenericErrorStatus block;
> +    AcpiGenericErrorData *gdata;
> +    UefiCperSecMemErr *mem_err;
> +    uint64_t current_block_length;
> +    unsigned char *buffer;
> +    /* memory section */
> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
> +                              0x83, 0xB1};
> +
> +    cpu_physical_memory_read(error_block_address, &block,
> +                                sizeof(AcpiGenericErrorStatus));
> +
> +    /* Get the current generic error status block length */
> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
> +        le32_to_cpu(block.data_length);
> +
> +    /* If the Generic Error Status Block is NULL, update
> +     * the block header
> +     */
> +    if (!block.block_status) {
> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    }
> +
> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    /* check whether it runs out of the preallocated memory */
> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
> +       GHES_MAX_RAW_DATA_LENGTH) {
> +        error_report("Record CPER out of boundary!!!");
> +        return GHES_CPER_FAIL;
> +    }
> +
> +    /* Write back the Generic Error Status Block to guest memory */
> +    cpu_physical_memory_write(error_block_address, &block,
> +        sizeof(AcpiGenericErrorStatus));
> +
> +    /* Fill in Generic Error Data Entry */
> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
> +                       sizeof(UefiCperSecMemErr));
> +
> +
> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
looks redundant, g_malloc0 does it for you

> +    gdata = (AcpiGenericErrorData *)buffer;
> +
> +    /* Memory section */
> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
> +            sizeof(mem_section_id_le));
> +
> +    /* error severity is recoverable */
> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
> +    gdata->revision = 0x300; /* the revision number is 0x300 */
> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
> +
> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
> +
> +    /* User space only handle the memory section CPER */
> +
> +    /* Hard code to Multi-bit ECC error */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
> +
> +    /* Record the physical address at which the memory error occurred */
> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);

I'd prefer for you to use build_append_int_noprefix() API to compose
whole error status block

and try to get rid of most structures you introduce in patch 1/6,
as they will be left unused after that.

> +
> +    /* Write back the Generic Error Data Entry to guest memory */
> +    cpu_physical_memory_write(error_block_address + current_block_length,
> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> +
> +    g_free(buffer);
> +    return GHES_CPER_OK;
> +}
> +
> +static void
> +build_address(GArray *table_data, BIOSLinker *linker,
> +    uint32_t dst_patched_offset, uint32_t src_offset,
> +    uint8_t address_space_id , uint8_t  register_bit_width,
> +    uint8_t register_bit_offset, uint8_t access_size)
> +{
> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    /* Address space */
> +    build_append_int_noprefix(table_data, address_space_id, 1);
> +    /* register bit width */
> +    build_append_int_noprefix(table_data, register_bit_width, 1);
> +    /* register bit offset */
> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
> +    /* access size */
> +    build_append_int_noprefix(table_data, access_size, 1);
> +    acpi_data_push(table_data, address_size);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
> +     * can retrieve and read it. the address size is 64 bits.
> +     */
> +    bios_linker_loader_add_pointer(linker,
> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
> +}
It's mostly generic GAS structure with linker addition.
I'd suggest to reuse something like
 https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().

> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                                            BIOSLinker *linker)
> +{
> +    uint32_t ghes_start = table_data->len;
> +    uint32_t address_size, error_status_address_offset;
> +    uint32_t read_ack_register_offset, i;
> +
> +    address_size = sizeof(struct AcpiGenericAddress) -
> +        offsetof(struct AcpiGenericAddress, address);
it's confusing name for var,
AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
also, I'm not sure why it's needed at all.

> +
> +    error_status_address_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
> +        offsetof(struct AcpiGenericAddress, address);
> +
> +    read_ack_register_offset = ghes_start +
> +        sizeof(AcpiHardwareErrorSourceTable) +
> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
> +        offsetof(struct AcpiGenericAddress, address);
it's really hard to get why you use offsetof() so much in this function,
to me above code totally unreadable.

> +    acpi_data_push(hardware_error,
> +        offsetof(struct hardware_errors_buffer, ack_value));
it looks like you are trying to build several tables within one function,
so it's hard to get what's going on.
I'd suggest to build separate table independently where it's possible.

i.e. build independent tables first
and only then build dependent tables passing to it pointers
to previously build table if necessary.

> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Initialize read ack register */
> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
> +
> +    /* Reserved the total size for ERRORS fw_cfg blob
> +     */
> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
> +
> +    /* Allocate guest memory for the Data fw_cfg blob */
> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
> +                            1, false);
> +    /* Reserve table header size */
> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
> +
> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
I'd suggest to use spec field name wit table prefix, ex:
ACPI_HEST_ERROR_SOURCE_COUNT

also, beside build_append_int_noprefix() you need to at least
add comment that exactly matches field from spec.

the same applies to other fields you are adding in this patch

> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
> +        build_append_int_noprefix(table_data,
> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
> +        /* source id */
> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
> +        /* related source id */
> +        build_append_int_noprefix(table_data, 0xffff, 2);
> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
> +
> +        /* Currently only enable SEA notification type to avoid the kernel
> +         * warning, reserve the space for other notification error source
> +         */
> +        if (i == ACPI_HEST_NOTIFY_SEA) {
> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
> +        } else {
> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
> +        }
> +
> +        /* The number of error status block per generic hardware error source */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max sections per record */
> +        build_append_int_noprefix(table_data, 1, 4);
> +        /* Max raw data length */
> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
> +
> +        /* Build error status address*/
> +        build_address(table_data, linker, error_status_address_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
just do something like this instead of build_address():
build_append_gas()
bios_linker_loader_add_pointer()

also register width 0x40 looks suspicious, where does it come from?
While at it do you have a real hardware which has HEST table that you re trying to model after?
I'd like to see HEST and other related tables from it.

> +
> +        /* Hardware error notification structure */
> +        build_append_int_noprefix(table_data, i, 1); /* type */
> +        /* length */
> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
> +        build_append_int_noprefix(table_data, 0, 26);
> +
> +        /* Error Status Block Length */
> +        build_append_int_noprefix(table_data,
> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
> +
> +        /* Build read ack register */
> +        build_address(table_data, linker, read_ack_register_offset + i *
> +            sizeof(AcpiGenericHardwareErrorSourceV2),
> +            offsetof(struct hardware_errors_buffer, ack_value) +
> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
> +            4 /* QWord access */);
> +
> +        /* Read ack preserve */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
> +
> +        /* Read ack write */
> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
> +    }
> +
> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
> +        /* Patch address of generic error status block into
> +         * the address register so OSPM can retrieve and read it.
> +         */
> +        bios_linker_loader_add_pointer(linker,
> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
> +            GHES_ERRORS_FW_CFG_FILE,
> +            offsetof(struct hardware_errors_buffer, gesb) +
> +            i * GHES_MAX_RAW_DATA_LENGTH);
> +
> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
> +     * so QEMU can write the ERRORS there. The address is expected to be
> +     * < 4GB, but write 64 bits anyway.
> +     */
> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
> +        offsetof(struct hardware_errors_buffer, gesb));
> +
> +    build_header(linker, table_data,
> +        (void *)(table_data->data + ghes_start), "HEST",
> +        table_data->len - ghes_start, 1, NULL, "GHES");
> +}
> +
> +static GhesState ges;
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
> +{
> +
> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
> +
> +    /* Create a read-only fw_cfg file for GHES */
> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
> +                    size);
> +    /* Create a read-write fw_cfg file for Address */
> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
> +}
> +
> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
> +{
> +    uint64_t error_block_addr;
> +    uint64_t ack_value_addr, ack_value = 0;
> +    int loop = 0, ack_value_size;
> +    bool ret = GHES_CPER_FAIL;
> +
> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
> +        offsetof(struct hardware_errors_buffer, ack_value)) /
> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
> +
> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
> +        error_block_addr = le32_to_cpu(error_block_addr);
> +
> +        ack_value_addr = ges.ghes_addr_le -
> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
> +retry:
> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
> +        if (!ack_value) {
> +            if (loop < 3) {
> +                usleep(100 * 1000);
> +                loop++;
> +                goto retry;
> +            } else {
> +                error_report("Last time OSPM does not acknowledge the error,"
> +                    " record CPER failed this time, set the ack value to"
> +                    " avoid blocking next time CPER record! exit");
> +                ack_value = 1;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                return ret;
> +            }
> +        } else {
> +            /* A zero value in ghes_addr means that BIOS has not yet written
> +             * the address
> +             */
> +            if (error_block_addr) {
> +                ack_value = 0;
> +                cpu_physical_memory_write(ack_value_addr,
> +                    &ack_value, ack_value_size);
> +                ret = ghes_record_cper(error_block_addr, physical_address);
> +            }
> +        }
> +    }
> +    return ret;
> +}
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> index 3d78ff6..def1ec1 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -45,6 +45,7 @@
>  #include "hw/arm/virt.h"
>  #include "sysemu/numa.h"
>  #include "kvm_arm.h"
> +#include "hw/acpi/hest_ghes.h"
>  
>  #define ARM_SPI_BASE 32
>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>      acpi_add_table(table_offsets, tables_blob);
>      build_spcr(tables_blob, tables->linker, vms);
>  
> +    acpi_add_table(table_offsets, tables_blob);
> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
> +
>      if (nb_numa_nodes > 0) {
>          acpi_add_table(table_offsets, tables_blob);
>          build_srat(tables_blob, tables->linker, vms);
> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>                      acpi_data_len(tables.tcpalog));
>  
> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
> +
>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>                                                ACPI_BUILD_RSDP_FILE, 0);
>  
> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
> index 88d0738..7f7b55c 100644
> --- a/include/hw/acpi/aml-build.h
> +++ b/include/hw/acpi/aml-build.h
> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>      GArray *rsdp;
>      GArray *tcpalog;
>      GArray *vmgenid;
> +    GArray *hardware_errors;
>      BIOSLinker *linker;
>  } AcpiBuildTables;
>  
> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
> new file mode 100644
> index 0000000..0772756
> --- /dev/null
> +++ b/include/hw/acpi/hest_ghes.h
> @@ -0,0 +1,47 @@
> +/*
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> +
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * Authors:
> + *   Dongjiu Geng <gengdongjiu@huawei.com>
> + *
> + * You should have received a copy of the GNU General Public License along
> + * with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef ACPI_GHES_H
> +#define ACPI_GHES_H
> +
> +#include "hw/acpi/bios-linker-loader.h"
> +
> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
> +
> +#define GHES_GAS_ADDRESS_OFFSET              4
> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
> +#define GHES_NOTIFICATION_STRUCTURE          32
> +
> +#define GHES_CPER_OK   1
> +#define GHES_CPER_FAIL 0
> +
> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
> +/* The max size in Bytes for one error block */
> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
> +
> +
> +typedef struct GhesState {
> +    uint64_t ghes_addr_le;
> +} GhesState;
> +
> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> +                            BIOSLinker *linker);
> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
> +#endif

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
  2017-08-29 10:20     ` Igor Mammedov
                         ` (2 preceding siblings ...)
  (?)
@ 2017-08-29 11:15       ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-29 11:15 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: mst, zhaoshenglong, lersek, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, zhengqiang10

Igor,
  Thank you very much for your review and comments, I will check your comments in detail and reply to you.


On 2017/8/29 18:20, Igor Mammedov wrote:
> On Fri, 18 Aug 2017 22:23:43 +0800
> Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> 
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
> 
> it's a bit complex patch/functionality so I've just mosty skimmed and
> commented only on structure of the patch and changes I'd like to see
> so it would be more structured and review-able.
> 
> I'd suggest to add doc patch first which will describe how it's
> supposed to work between QEMU/firmware/guest OS with expected
> flows.
>  
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
> these diagram shows relations between tables which not necessarily bad
> but as layout it's useless.
>  * Probably there is not much sense to have HEST table here, it's described
>    well enough in spec. You might just put reference here.
>  * these diagrams should go into doc/spec patch
>  * when you describe layout you need to show what and at what offsets
>    in which order in which blob/file is located. See ACPI spec for example
>    and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.
> 
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  hw/acpi/aml-build.c         |   2 +
>>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>>  hw/arm/virt-acpi-build.c    |   6 +
>>  include/hw/acpi/aml-build.h |   1 +
>>  include/hw/acpi/hest_ghes.h |  47 ++++++
>>  5 files changed, 401 insertions(+)
>>  create mode 100644 hw/acpi/hest_ghes.c
>>  create mode 100644 include/hw/acpi/hest_ghes.h
>>
>> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
>> index 36a6cc4..6849e5f 100644
>> --- a/hw/acpi/aml-build.c
>> +++ b/hw/acpi/aml-build.c
>> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>>      tables->table_data = g_array_new(false, true /* clear */, 1);
>>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
>> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>>      tables->linker = bios_linker_loader_init();
>>  }
>>  
>> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>>      g_array_free(tables->table_data, true);
>>      g_array_free(tables->tcpalog, mfre);
>>      g_array_free(tables->vmgenid, mfre);
>> +    g_array_free(tables->hardware_errors, mfre);
>>  }
>>  
>>  /* Build rsdt table */
>> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
>> new file mode 100644
>> index 0000000..ff6b5ef
>> --- /dev/null
>> +++ b/hw/acpi/hest_ghes.c
>> @@ -0,0 +1,345 @@
>> +/*
>> + *  APEI GHES table Generation
>> + *
>> + *  Copyright (C) 2017 huawei.
>> + *
>> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + *
>> + */
>> +
>> +#include "qemu/osdep.h"
>> +#include "qmp-commands.h"
>> +#include "hw/acpi/acpi.h"
>> +#include "hw/acpi/aml-build.h"
>> +#include "hw/acpi/hest_ghes.h"
>> +#include "hw/nvram/fw_cfg.h"
>> +#include "sysemu/sysemu.h"
>> +#include "qemu/error-report.h"
>> +
>> +/* The structure that stands for the layout
>> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
>> + *
>> + *           etc/hardware_errors
>> + * ==========================================
>> + * +------------------+
>> + * |    address       |              +--------------+
>> + * |    registers     |              | Error Status |
>> + * | +----------------+              | Data Block 0 |
>> + * | |status_address0 |------------->| +------------+
>> + * | +----------------+              | |  CPER      |
>> + * | |status_address1 |----------+   | |  CPER      |
>> + * | +----------------+          |   | |  ....      |
>> + * | |.............   |          |   | |  CPER      |
>> + * | +----------------+          |   | +------------+
>> + * | |status_address10|-----+    |   | Error Status |
>> + * | +----------------+     |    |   | Data Block 1 |
>> + * | |ack_value0      |     |    +-->| +------------+
>> + * | +----------------+     |        | |  CPER      |
>> + * | |ack_value1      |     |        | |  CPER      |
>> + * | +----------------+     |        | |  ....      |
>> + * | | .............  |     |        | |  CPER      |
>> + * | +----------------+     |        +-+------------+
>> + * | |ack_value10     |     |        | |..........  |
>> + * | +----------------+     |        | +------------+
>> + *                          |        | Error Status |
>> + *                          |        | Data Block10 |
>> + *                          +------->+------------+
>> + *                                   | |  CPER      |
>> + *                                   | |  CPER      |
>> + *                                   | |  ....      |
>> + *                                   | |  CPER      |
>> + *                                   +-+------------+
>> + */
>> +struct hardware_errors_buffer {
>> +    /* Generic Error Status Block register */
>> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +};
>> +
>> +static int ghes_record_cper(uint64_t error_block_address,
>> +                                    uint64_t error_physical_addr)
>> +{
>> +    AcpiGenericErrorStatus block;
>> +    AcpiGenericErrorData *gdata;
>> +    UefiCperSecMemErr *mem_err;
>> +    uint64_t current_block_length;
>> +    unsigned char *buffer;
>> +    /* memory section */
>> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
>> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
>> +                              0x83, 0xB1};
>> +
>> +    cpu_physical_memory_read(error_block_address, &block,
>> +                                sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Get the current generic error status block length */
>> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
>> +        le32_to_cpu(block.data_length);
>> +
>> +    /* If the Generic Error Status Block is NULL, update
>> +     * the block header
>> +     */
>> +    if (!block.block_status) {
>> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
>> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    }
>> +
>> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
>> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    /* check whether it runs out of the preallocated memory */
>> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
>> +       GHES_MAX_RAW_DATA_LENGTH) {
>> +        error_report("Record CPER out of boundary!!!");
>> +        return GHES_CPER_FAIL;
>> +    }
>> +
>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> looks redundant, g_malloc0 does it for you
> 
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> 
> I'd prefer for you to use build_append_int_noprefix() API to compose
> whole error status block
> 
> and try to get rid of most structures you introduce in patch 1/6,
> as they will be left unused after that.
> 
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
> It's mostly generic GAS structure with linker addition.
> I'd suggest to reuse something like
>  https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
> to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().
> 
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
> it's confusing name for var,
> AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> also, I'm not sure why it's needed at all.
> 
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
> it's really hard to get why you use offsetof() so much in this function,
> to me above code totally unreadable.
> 
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
> it looks like you are trying to build several tables within one function,
> so it's hard to get what's going on.
> I'd suggest to build separate table independently where it's possible.
> 
> i.e. build independent tables first
> and only then build dependent tables passing to it pointers
> to previously build table if necessary.
> 
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
> I'd suggest to use spec field name wit table prefix, ex:
> ACPI_HEST_ERROR_SOURCE_COUNT
> 
> also, beside build_append_int_noprefix() you need to at least
> add comment that exactly matches field from spec.
> 
> the same applies to other fields you are adding in this patch
> 
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> just do something like this instead of build_address():
> build_append_gas()
> bios_linker_loader_add_pointer()
> 
> also register width 0x40 looks suspicious, where does it come from?
> While at it do you have a real hardware which has HEST table that you re trying to model after?
> I'd like to see HEST and other related tables from it.
> 
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
>> +        /* Build read ack register */
>> +        build_address(table_data, linker, read_ack_register_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2),
>> +            offsetof(struct hardware_errors_buffer, ack_value) +
>> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
>> +            4 /* QWord access */);
>> +
>> +        /* Read ack preserve */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
>> +
>> +        /* Read ack write */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
>> +    }
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Patch address of generic error status block into
>> +         * the address register so OSPM can retrieve and read it.
>> +         */
>> +        bios_linker_loader_add_pointer(linker,
>> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
>> +            GHES_ERRORS_FW_CFG_FILE,
>> +            offsetof(struct hardware_errors_buffer, gesb) +
>> +            i * GHES_MAX_RAW_DATA_LENGTH);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
>> +     * so QEMU can write the ERRORS there. The address is expected to be
>> +     * < 4GB, but write 64 bits anyway.
>> +     */
>> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
>> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
>> +        offsetof(struct hardware_errors_buffer, gesb));
>> +
>> +    build_header(linker, table_data,
>> +        (void *)(table_data->data + ghes_start), "HEST",
>> +        table_data->len - ghes_start, 1, NULL, "GHES");
>> +}
>> +
>> +static GhesState ges;
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
>> +{
>> +
>> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
>> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
>> +
>> +    /* Create a read-only fw_cfg file for GHES */
>> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
>> +                    size);
>> +    /* Create a read-write fw_cfg file for Address */
>> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
>> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
>> +}
>> +
>> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
>> +{
>> +    uint64_t error_block_addr;
>> +    uint64_t ack_value_addr, ack_value = 0;
>> +    int loop = 0, ack_value_size;
>> +    bool ret = GHES_CPER_FAIL;
>> +
>> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
>> +        offsetof(struct hardware_errors_buffer, ack_value)) /
>> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
>> +
>> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
>> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
>> +        error_block_addr = le32_to_cpu(error_block_addr);
>> +
>> +        ack_value_addr = ges.ghes_addr_le -
>> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
>> +retry:
>> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
>> +        if (!ack_value) {
>> +            if (loop < 3) {
>> +                usleep(100 * 1000);
>> +                loop++;
>> +                goto retry;
>> +            } else {
>> +                error_report("Last time OSPM does not acknowledge the error,"
>> +                    " record CPER failed this time, set the ack value to"
>> +                    " avoid blocking next time CPER record! exit");
>> +                ack_value = 1;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                return ret;
>> +            }
>> +        } else {
>> +            /* A zero value in ghes_addr means that BIOS has not yet written
>> +             * the address
>> +             */
>> +            if (error_block_addr) {
>> +                ack_value = 0;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                ret = ghes_record_cper(error_block_addr, physical_address);
>> +            }
>> +        }
>> +    }
>> +    return ret;
>> +}
>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>> index 3d78ff6..def1ec1 100644
>> --- a/hw/arm/virt-acpi-build.c
>> +++ b/hw/arm/virt-acpi-build.c
>> @@ -45,6 +45,7 @@
>>  #include "hw/arm/virt.h"
>>  #include "sysemu/numa.h"
>>  #include "kvm_arm.h"
>> +#include "hw/acpi/hest_ghes.h"
>>  
>>  #define ARM_SPI_BASE 32
>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>      acpi_add_table(table_offsets, tables_blob);
>>      build_spcr(tables_blob, tables->linker, vms);
>>  
>> +    acpi_add_table(table_offsets, tables_blob);
>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>> +
>>      if (nb_numa_nodes > 0) {
>>          acpi_add_table(table_offsets, tables_blob);
>>          build_srat(tables_blob, tables->linker, vms);
>> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>>                      acpi_data_len(tables.tcpalog));
>>  
>> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
>> +
>>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>>                                                ACPI_BUILD_RSDP_FILE, 0);
>>  
>> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
>> index 88d0738..7f7b55c 100644
>> --- a/include/hw/acpi/aml-build.h
>> +++ b/include/hw/acpi/aml-build.h
>> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>>      GArray *rsdp;
>>      GArray *tcpalog;
>>      GArray *vmgenid;
>> +    GArray *hardware_errors;
>>      BIOSLinker *linker;
>>  } AcpiBuildTables;
>>  
>> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
>> new file mode 100644
>> index 0000000..0772756
>> --- /dev/null
>> +++ b/include/hw/acpi/hest_ghes.h
>> @@ -0,0 +1,47 @@
>> +/*
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> +
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * Authors:
>> + *   Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * You should have received a copy of the GNU General Public License along
>> + * with this program; if not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#ifndef ACPI_GHES_H
>> +#define ACPI_GHES_H
>> +
>> +#include "hw/acpi/bios-linker-loader.h"
>> +
>> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
>> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
>> +
>> +#define GHES_GAS_ADDRESS_OFFSET              4
>> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
>> +#define GHES_NOTIFICATION_STRUCTURE          32
>> +
>> +#define GHES_CPER_OK   1
>> +#define GHES_CPER_FAIL 0
>> +
>> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
>> +/* The max size in Bytes for one error block */
>> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
>> +
>> +
>> +typedef struct GhesState {
>> +    uint64_t ghes_addr_le;
>> +} GhesState;
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                            BIOSLinker *linker);
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
>> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
>> +#endif
> 
> 
> .
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-29 11:15       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-29 11:15 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: mst, zhaoshenglong, lersek, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, zhengqiang10,
	wuquanming, huangshaoyu, linuxarm

Igor,
  Thank you very much for your review and comments, I will check your comments in detail and reply to you.


On 2017/8/29 18:20, Igor Mammedov wrote:
> On Fri, 18 Aug 2017 22:23:43 +0800
> Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> 
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
> 
> it's a bit complex patch/functionality so I've just mosty skimmed and
> commented only on structure of the patch and changes I'd like to see
> so it would be more structured and review-able.
> 
> I'd suggest to add doc patch first which will describe how it's
> supposed to work between QEMU/firmware/guest OS with expected
> flows.
>  
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
> these diagram shows relations between tables which not necessarily bad
> but as layout it's useless.
>  * Probably there is not much sense to have HEST table here, it's described
>    well enough in spec. You might just put reference here.
>  * these diagrams should go into doc/spec patch
>  * when you describe layout you need to show what and at what offsets
>    in which order in which blob/file is located. See ACPI spec for example
>    and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.
> 
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  hw/acpi/aml-build.c         |   2 +
>>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>>  hw/arm/virt-acpi-build.c    |   6 +
>>  include/hw/acpi/aml-build.h |   1 +
>>  include/hw/acpi/hest_ghes.h |  47 ++++++
>>  5 files changed, 401 insertions(+)
>>  create mode 100644 hw/acpi/hest_ghes.c
>>  create mode 100644 include/hw/acpi/hest_ghes.h
>>
>> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
>> index 36a6cc4..6849e5f 100644
>> --- a/hw/acpi/aml-build.c
>> +++ b/hw/acpi/aml-build.c
>> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>>      tables->table_data = g_array_new(false, true /* clear */, 1);
>>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
>> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>>      tables->linker = bios_linker_loader_init();
>>  }
>>  
>> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>>      g_array_free(tables->table_data, true);
>>      g_array_free(tables->tcpalog, mfre);
>>      g_array_free(tables->vmgenid, mfre);
>> +    g_array_free(tables->hardware_errors, mfre);
>>  }
>>  
>>  /* Build rsdt table */
>> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
>> new file mode 100644
>> index 0000000..ff6b5ef
>> --- /dev/null
>> +++ b/hw/acpi/hest_ghes.c
>> @@ -0,0 +1,345 @@
>> +/*
>> + *  APEI GHES table Generation
>> + *
>> + *  Copyright (C) 2017 huawei.
>> + *
>> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + *
>> + */
>> +
>> +#include "qemu/osdep.h"
>> +#include "qmp-commands.h"
>> +#include "hw/acpi/acpi.h"
>> +#include "hw/acpi/aml-build.h"
>> +#include "hw/acpi/hest_ghes.h"
>> +#include "hw/nvram/fw_cfg.h"
>> +#include "sysemu/sysemu.h"
>> +#include "qemu/error-report.h"
>> +
>> +/* The structure that stands for the layout
>> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
>> + *
>> + *           etc/hardware_errors
>> + * ==========================================
>> + * +------------------+
>> + * |    address       |              +--------------+
>> + * |    registers     |              | Error Status |
>> + * | +----------------+              | Data Block 0 |
>> + * | |status_address0 |------------->| +------------+
>> + * | +----------------+              | |  CPER      |
>> + * | |status_address1 |----------+   | |  CPER      |
>> + * | +----------------+          |   | |  ....      |
>> + * | |.............   |          |   | |  CPER      |
>> + * | +----------------+          |   | +------------+
>> + * | |status_address10|-----+    |   | Error Status |
>> + * | +----------------+     |    |   | Data Block 1 |
>> + * | |ack_value0      |     |    +-->| +------------+
>> + * | +----------------+     |        | |  CPER      |
>> + * | |ack_value1      |     |        | |  CPER      |
>> + * | +----------------+     |        | |  ....      |
>> + * | | .............  |     |        | |  CPER      |
>> + * | +----------------+     |        +-+------------+
>> + * | |ack_value10     |     |        | |..........  |
>> + * | +----------------+     |        | +------------+
>> + *                          |        | Error Status |
>> + *                          |        | Data Block10 |
>> + *                          +------->+------------+
>> + *                                   | |  CPER      |
>> + *                                   | |  CPER      |
>> + *                                   | |  ....      |
>> + *                                   | |  CPER      |
>> + *                                   +-+------------+
>> + */
>> +struct hardware_errors_buffer {
>> +    /* Generic Error Status Block register */
>> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +};
>> +
>> +static int ghes_record_cper(uint64_t error_block_address,
>> +                                    uint64_t error_physical_addr)
>> +{
>> +    AcpiGenericErrorStatus block;
>> +    AcpiGenericErrorData *gdata;
>> +    UefiCperSecMemErr *mem_err;
>> +    uint64_t current_block_length;
>> +    unsigned char *buffer;
>> +    /* memory section */
>> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
>> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
>> +                              0x83, 0xB1};
>> +
>> +    cpu_physical_memory_read(error_block_address, &block,
>> +                                sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Get the current generic error status block length */
>> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
>> +        le32_to_cpu(block.data_length);
>> +
>> +    /* If the Generic Error Status Block is NULL, update
>> +     * the block header
>> +     */
>> +    if (!block.block_status) {
>> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
>> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    }
>> +
>> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
>> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    /* check whether it runs out of the preallocated memory */
>> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
>> +       GHES_MAX_RAW_DATA_LENGTH) {
>> +        error_report("Record CPER out of boundary!!!");
>> +        return GHES_CPER_FAIL;
>> +    }
>> +
>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> looks redundant, g_malloc0 does it for you
> 
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> 
> I'd prefer for you to use build_append_int_noprefix() API to compose
> whole error status block
> 
> and try to get rid of most structures you introduce in patch 1/6,
> as they will be left unused after that.
> 
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
> It's mostly generic GAS structure with linker addition.
> I'd suggest to reuse something like
>  https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
> to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().
> 
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
> it's confusing name for var,
> AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> also, I'm not sure why it's needed at all.
> 
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
> it's really hard to get why you use offsetof() so much in this function,
> to me above code totally unreadable.
> 
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
> it looks like you are trying to build several tables within one function,
> so it's hard to get what's going on.
> I'd suggest to build separate table independently where it's possible.
> 
> i.e. build independent tables first
> and only then build dependent tables passing to it pointers
> to previously build table if necessary.
> 
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
> I'd suggest to use spec field name wit table prefix, ex:
> ACPI_HEST_ERROR_SOURCE_COUNT
> 
> also, beside build_append_int_noprefix() you need to at least
> add comment that exactly matches field from spec.
> 
> the same applies to other fields you are adding in this patch
> 
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> just do something like this instead of build_address():
> build_append_gas()
> bios_linker_loader_add_pointer()
> 
> also register width 0x40 looks suspicious, where does it come from?
> While at it do you have a real hardware which has HEST table that you re trying to model after?
> I'd like to see HEST and other related tables from it.
> 
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
>> +        /* Build read ack register */
>> +        build_address(table_data, linker, read_ack_register_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2),
>> +            offsetof(struct hardware_errors_buffer, ack_value) +
>> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
>> +            4 /* QWord access */);
>> +
>> +        /* Read ack preserve */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
>> +
>> +        /* Read ack write */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
>> +    }
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Patch address of generic error status block into
>> +         * the address register so OSPM can retrieve and read it.
>> +         */
>> +        bios_linker_loader_add_pointer(linker,
>> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
>> +            GHES_ERRORS_FW_CFG_FILE,
>> +            offsetof(struct hardware_errors_buffer, gesb) +
>> +            i * GHES_MAX_RAW_DATA_LENGTH);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
>> +     * so QEMU can write the ERRORS there. The address is expected to be
>> +     * < 4GB, but write 64 bits anyway.
>> +     */
>> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
>> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
>> +        offsetof(struct hardware_errors_buffer, gesb));
>> +
>> +    build_header(linker, table_data,
>> +        (void *)(table_data->data + ghes_start), "HEST",
>> +        table_data->len - ghes_start, 1, NULL, "GHES");
>> +}
>> +
>> +static GhesState ges;
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
>> +{
>> +
>> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
>> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
>> +
>> +    /* Create a read-only fw_cfg file for GHES */
>> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
>> +                    size);
>> +    /* Create a read-write fw_cfg file for Address */
>> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
>> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
>> +}
>> +
>> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
>> +{
>> +    uint64_t error_block_addr;
>> +    uint64_t ack_value_addr, ack_value = 0;
>> +    int loop = 0, ack_value_size;
>> +    bool ret = GHES_CPER_FAIL;
>> +
>> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
>> +        offsetof(struct hardware_errors_buffer, ack_value)) /
>> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
>> +
>> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
>> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
>> +        error_block_addr = le32_to_cpu(error_block_addr);
>> +
>> +        ack_value_addr = ges.ghes_addr_le -
>> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
>> +retry:
>> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
>> +        if (!ack_value) {
>> +            if (loop < 3) {
>> +                usleep(100 * 1000);
>> +                loop++;
>> +                goto retry;
>> +            } else {
>> +                error_report("Last time OSPM does not acknowledge the error,"
>> +                    " record CPER failed this time, set the ack value to"
>> +                    " avoid blocking next time CPER record! exit");
>> +                ack_value = 1;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                return ret;
>> +            }
>> +        } else {
>> +            /* A zero value in ghes_addr means that BIOS has not yet written
>> +             * the address
>> +             */
>> +            if (error_block_addr) {
>> +                ack_value = 0;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                ret = ghes_record_cper(error_block_addr, physical_address);
>> +            }
>> +        }
>> +    }
>> +    return ret;
>> +}
>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>> index 3d78ff6..def1ec1 100644
>> --- a/hw/arm/virt-acpi-build.c
>> +++ b/hw/arm/virt-acpi-build.c
>> @@ -45,6 +45,7 @@
>>  #include "hw/arm/virt.h"
>>  #include "sysemu/numa.h"
>>  #include "kvm_arm.h"
>> +#include "hw/acpi/hest_ghes.h"
>>  
>>  #define ARM_SPI_BASE 32
>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>      acpi_add_table(table_offsets, tables_blob);
>>      build_spcr(tables_blob, tables->linker, vms);
>>  
>> +    acpi_add_table(table_offsets, tables_blob);
>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>> +
>>      if (nb_numa_nodes > 0) {
>>          acpi_add_table(table_offsets, tables_blob);
>>          build_srat(tables_blob, tables->linker, vms);
>> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>>                      acpi_data_len(tables.tcpalog));
>>  
>> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
>> +
>>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>>                                                ACPI_BUILD_RSDP_FILE, 0);
>>  
>> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
>> index 88d0738..7f7b55c 100644
>> --- a/include/hw/acpi/aml-build.h
>> +++ b/include/hw/acpi/aml-build.h
>> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>>      GArray *rsdp;
>>      GArray *tcpalog;
>>      GArray *vmgenid;
>> +    GArray *hardware_errors;
>>      BIOSLinker *linker;
>>  } AcpiBuildTables;
>>  
>> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
>> new file mode 100644
>> index 0000000..0772756
>> --- /dev/null
>> +++ b/include/hw/acpi/hest_ghes.h
>> @@ -0,0 +1,47 @@
>> +/*
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> +
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * Authors:
>> + *   Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * You should have received a copy of the GNU General Public License along
>> + * with this program; if not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#ifndef ACPI_GHES_H
>> +#define ACPI_GHES_H
>> +
>> +#include "hw/acpi/bios-linker-loader.h"
>> +
>> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
>> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
>> +
>> +#define GHES_GAS_ADDRESS_OFFSET              4
>> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
>> +#define GHES_NOTIFICATION_STRUCTURE          32
>> +
>> +#define GHES_CPER_OK   1
>> +#define GHES_CPER_FAIL 0
>> +
>> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
>> +/* The max size in Bytes for one error block */
>> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
>> +
>> +
>> +typedef struct GhesState {
>> +    uint64_t ghes_addr_le;
>> +} GhesState;
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                            BIOSLinker *linker);
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
>> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
>> +#endif
> 
> 
> .
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-29 11:15       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-29 11:15 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: mst, zhaoshenglong, lersek, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, zhengqiang10

Igor,
  Thank you very much for your review and comments, I will check your comments in detail and reply to you.


On 2017/8/29 18:20, Igor Mammedov wrote:
> On Fri, 18 Aug 2017 22:23:43 +0800
> Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> 
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
> 
> it's a bit complex patch/functionality so I've just mosty skimmed and
> commented only on structure of the patch and changes I'd like to see
> so it would be more structured and review-able.
> 
> I'd suggest to add doc patch first which will describe how it's
> supposed to work between QEMU/firmware/guest OS with expected
> flows.
>  
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
> these diagram shows relations between tables which not necessarily bad
> but as layout it's useless.
>  * Probably there is not much sense to have HEST table here, it's described
>    well enough in spec. You might just put reference here.
>  * these diagrams should go into doc/spec patch
>  * when you describe layout you need to show what and at what offsets
>    in which order in which blob/file is located. See ACPI spec for example
>    and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.
> 
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  hw/acpi/aml-build.c         |   2 +
>>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>>  hw/arm/virt-acpi-build.c    |   6 +
>>  include/hw/acpi/aml-build.h |   1 +
>>  include/hw/acpi/hest_ghes.h |  47 ++++++
>>  5 files changed, 401 insertions(+)
>>  create mode 100644 hw/acpi/hest_ghes.c
>>  create mode 100644 include/hw/acpi/hest_ghes.h
>>
>> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
>> index 36a6cc4..6849e5f 100644
>> --- a/hw/acpi/aml-build.c
>> +++ b/hw/acpi/aml-build.c
>> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>>      tables->table_data = g_array_new(false, true /* clear */, 1);
>>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
>> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>>      tables->linker = bios_linker_loader_init();
>>  }
>>  
>> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>>      g_array_free(tables->table_data, true);
>>      g_array_free(tables->tcpalog, mfre);
>>      g_array_free(tables->vmgenid, mfre);
>> +    g_array_free(tables->hardware_errors, mfre);
>>  }
>>  
>>  /* Build rsdt table */
>> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
>> new file mode 100644
>> index 0000000..ff6b5ef
>> --- /dev/null
>> +++ b/hw/acpi/hest_ghes.c
>> @@ -0,0 +1,345 @@
>> +/*
>> + *  APEI GHES table Generation
>> + *
>> + *  Copyright (C) 2017 huawei.
>> + *
>> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + *
>> + */
>> +
>> +#include "qemu/osdep.h"
>> +#include "qmp-commands.h"
>> +#include "hw/acpi/acpi.h"
>> +#include "hw/acpi/aml-build.h"
>> +#include "hw/acpi/hest_ghes.h"
>> +#include "hw/nvram/fw_cfg.h"
>> +#include "sysemu/sysemu.h"
>> +#include "qemu/error-report.h"
>> +
>> +/* The structure that stands for the layout
>> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
>> + *
>> + *           etc/hardware_errors
>> + * ==========================================
>> + * +------------------+
>> + * |    address       |              +--------------+
>> + * |    registers     |              | Error Status |
>> + * | +----------------+              | Data Block 0 |
>> + * | |status_address0 |------------->| +------------+
>> + * | +----------------+              | |  CPER      |
>> + * | |status_address1 |----------+   | |  CPER      |
>> + * | +----------------+          |   | |  ....      |
>> + * | |.............   |          |   | |  CPER      |
>> + * | +----------------+          |   | +------------+
>> + * | |status_address10|-----+    |   | Error Status |
>> + * | +----------------+     |    |   | Data Block 1 |
>> + * | |ack_value0      |     |    +-->| +------------+
>> + * | +----------------+     |        | |  CPER      |
>> + * | |ack_value1      |     |        | |  CPER      |
>> + * | +----------------+     |        | |  ....      |
>> + * | | .............  |     |        | |  CPER      |
>> + * | +----------------+     |        +-+------------+
>> + * | |ack_value10     |     |        | |..........  |
>> + * | +----------------+     |        | +------------+
>> + *                          |        | Error Status |
>> + *                          |        | Data Block10 |
>> + *                          +------->+------------+
>> + *                                   | |  CPER      |
>> + *                                   | |  CPER      |
>> + *                                   | |  ....      |
>> + *                                   | |  CPER      |
>> + *                                   +-+------------+
>> + */
>> +struct hardware_errors_buffer {
>> +    /* Generic Error Status Block register */
>> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +};
>> +
>> +static int ghes_record_cper(uint64_t error_block_address,
>> +                                    uint64_t error_physical_addr)
>> +{
>> +    AcpiGenericErrorStatus block;
>> +    AcpiGenericErrorData *gdata;
>> +    UefiCperSecMemErr *mem_err;
>> +    uint64_t current_block_length;
>> +    unsigned char *buffer;
>> +    /* memory section */
>> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
>> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
>> +                              0x83, 0xB1};
>> +
>> +    cpu_physical_memory_read(error_block_address, &block,
>> +                                sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Get the current generic error status block length */
>> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
>> +        le32_to_cpu(block.data_length);
>> +
>> +    /* If the Generic Error Status Block is NULL, update
>> +     * the block header
>> +     */
>> +    if (!block.block_status) {
>> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
>> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    }
>> +
>> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
>> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    /* check whether it runs out of the preallocated memory */
>> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
>> +       GHES_MAX_RAW_DATA_LENGTH) {
>> +        error_report("Record CPER out of boundary!!!");
>> +        return GHES_CPER_FAIL;
>> +    }
>> +
>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> looks redundant, g_malloc0 does it for you
> 
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> 
> I'd prefer for you to use build_append_int_noprefix() API to compose
> whole error status block
> 
> and try to get rid of most structures you introduce in patch 1/6,
> as they will be left unused after that.
> 
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
> It's mostly generic GAS structure with linker addition.
> I'd suggest to reuse something like
>  https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
> to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().
> 
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
> it's confusing name for var,
> AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> also, I'm not sure why it's needed at all.
> 
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
> it's really hard to get why you use offsetof() so much in this function,
> to me above code totally unreadable.
> 
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
> it looks like you are trying to build several tables within one function,
> so it's hard to get what's going on.
> I'd suggest to build separate table independently where it's possible.
> 
> i.e. build independent tables first
> and only then build dependent tables passing to it pointers
> to previously build table if necessary.
> 
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
> I'd suggest to use spec field name wit table prefix, ex:
> ACPI_HEST_ERROR_SOURCE_COUNT
> 
> also, beside build_append_int_noprefix() you need to at least
> add comment that exactly matches field from spec.
> 
> the same applies to other fields you are adding in this patch
> 
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> just do something like this instead of build_address():
> build_append_gas()
> bios_linker_loader_add_pointer()
> 
> also register width 0x40 looks suspicious, where does it come from?
> While at it do you have a real hardware which has HEST table that you re trying to model after?
> I'd like to see HEST and other related tables from it.
> 
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
>> +        /* Build read ack register */
>> +        build_address(table_data, linker, read_ack_register_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2),
>> +            offsetof(struct hardware_errors_buffer, ack_value) +
>> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
>> +            4 /* QWord access */);
>> +
>> +        /* Read ack preserve */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
>> +
>> +        /* Read ack write */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
>> +    }
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Patch address of generic error status block into
>> +         * the address register so OSPM can retrieve and read it.
>> +         */
>> +        bios_linker_loader_add_pointer(linker,
>> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
>> +            GHES_ERRORS_FW_CFG_FILE,
>> +            offsetof(struct hardware_errors_buffer, gesb) +
>> +            i * GHES_MAX_RAW_DATA_LENGTH);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
>> +     * so QEMU can write the ERRORS there. The address is expected to be
>> +     * < 4GB, but write 64 bits anyway.
>> +     */
>> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
>> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
>> +        offsetof(struct hardware_errors_buffer, gesb));
>> +
>> +    build_header(linker, table_data,
>> +        (void *)(table_data->data + ghes_start), "HEST",
>> +        table_data->len - ghes_start, 1, NULL, "GHES");
>> +}
>> +
>> +static GhesState ges;
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
>> +{
>> +
>> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
>> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
>> +
>> +    /* Create a read-only fw_cfg file for GHES */
>> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
>> +                    size);
>> +    /* Create a read-write fw_cfg file for Address */
>> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
>> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
>> +}
>> +
>> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
>> +{
>> +    uint64_t error_block_addr;
>> +    uint64_t ack_value_addr, ack_value = 0;
>> +    int loop = 0, ack_value_size;
>> +    bool ret = GHES_CPER_FAIL;
>> +
>> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
>> +        offsetof(struct hardware_errors_buffer, ack_value)) /
>> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
>> +
>> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
>> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
>> +        error_block_addr = le32_to_cpu(error_block_addr);
>> +
>> +        ack_value_addr = ges.ghes_addr_le -
>> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
>> +retry:
>> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
>> +        if (!ack_value) {
>> +            if (loop < 3) {
>> +                usleep(100 * 1000);
>> +                loop++;
>> +                goto retry;
>> +            } else {
>> +                error_report("Last time OSPM does not acknowledge the error,"
>> +                    " record CPER failed this time, set the ack value to"
>> +                    " avoid blocking next time CPER record! exit");
>> +                ack_value = 1;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                return ret;
>> +            }
>> +        } else {
>> +            /* A zero value in ghes_addr means that BIOS has not yet written
>> +             * the address
>> +             */
>> +            if (error_block_addr) {
>> +                ack_value = 0;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                ret = ghes_record_cper(error_block_addr, physical_address);
>> +            }
>> +        }
>> +    }
>> +    return ret;
>> +}
>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>> index 3d78ff6..def1ec1 100644
>> --- a/hw/arm/virt-acpi-build.c
>> +++ b/hw/arm/virt-acpi-build.c
>> @@ -45,6 +45,7 @@
>>  #include "hw/arm/virt.h"
>>  #include "sysemu/numa.h"
>>  #include "kvm_arm.h"
>> +#include "hw/acpi/hest_ghes.h"
>>  
>>  #define ARM_SPI_BASE 32
>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>      acpi_add_table(table_offsets, tables_blob);
>>      build_spcr(tables_blob, tables->linker, vms);
>>  
>> +    acpi_add_table(table_offsets, tables_blob);
>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>> +
>>      if (nb_numa_nodes > 0) {
>>          acpi_add_table(table_offsets, tables_blob);
>>          build_srat(tables_blob, tables->linker, vms);
>> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>>                      acpi_data_len(tables.tcpalog));
>>  
>> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
>> +
>>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>>                                                ACPI_BUILD_RSDP_FILE, 0);
>>  
>> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
>> index 88d0738..7f7b55c 100644
>> --- a/include/hw/acpi/aml-build.h
>> +++ b/include/hw/acpi/aml-build.h
>> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>>      GArray *rsdp;
>>      GArray *tcpalog;
>>      GArray *vmgenid;
>> +    GArray *hardware_errors;
>>      BIOSLinker *linker;
>>  } AcpiBuildTables;
>>  
>> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
>> new file mode 100644
>> index 0000000..0772756
>> --- /dev/null
>> +++ b/include/hw/acpi/hest_ghes.h
>> @@ -0,0 +1,47 @@
>> +/*
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> +
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * Authors:
>> + *   Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * You should have received a copy of the GNU General Public License along
>> + * with this program; if not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#ifndef ACPI_GHES_H
>> +#define ACPI_GHES_H
>> +
>> +#include "hw/acpi/bios-linker-loader.h"
>> +
>> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
>> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
>> +
>> +#define GHES_GAS_ADDRESS_OFFSET              4
>> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
>> +#define GHES_NOTIFICATION_STRUCTURE          32
>> +
>> +#define GHES_CPER_OK   1
>> +#define GHES_CPER_FAIL 0
>> +
>> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
>> +/* The max size in Bytes for one error block */
>> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
>> +
>> +
>> +typedef struct GhesState {
>> +    uint64_t ghes_addr_le;
>> +} GhesState;
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                            BIOSLinker *linker);
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
>> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
>> +#endif
> 
> 
> .
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-29 11:15       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-29 11:15 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: mst, zhaoshenglong, lersek, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, tbaicar, ard.biesheuvel, mingo, bp,
	shiju.jose, zjzhang, linux-arm-kernel, kvmarm, linux-kernel,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, zhengqiang10,
	wuquanming, huangshaoyu, linuxarm

Igor,
  Thank you very much for your review and comments, I will check your comments in detail and reply to you.


On 2017/8/29 18:20, Igor Mammedov wrote:
> On Fri, 18 Aug 2017 22:23:43 +0800
> Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> 
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
> 
> it's a bit complex patch/functionality so I've just mosty skimmed and
> commented only on structure of the patch and changes I'd like to see
> so it would be more structured and review-able.
> 
> I'd suggest to add doc patch first which will describe how it's
> supposed to work between QEMU/firmware/guest OS with expected
> flows.
>  
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
> these diagram shows relations between tables which not necessarily bad
> but as layout it's useless.
>  * Probably there is not much sense to have HEST table here, it's described
>    well enough in spec. You might just put reference here.
>  * these diagrams should go into doc/spec patch
>  * when you describe layout you need to show what and at what offsets
>    in which order in which blob/file is located. See ACPI spec for example
>    and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.
> 
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  hw/acpi/aml-build.c         |   2 +
>>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>>  hw/arm/virt-acpi-build.c    |   6 +
>>  include/hw/acpi/aml-build.h |   1 +
>>  include/hw/acpi/hest_ghes.h |  47 ++++++
>>  5 files changed, 401 insertions(+)
>>  create mode 100644 hw/acpi/hest_ghes.c
>>  create mode 100644 include/hw/acpi/hest_ghes.h
>>
>> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
>> index 36a6cc4..6849e5f 100644
>> --- a/hw/acpi/aml-build.c
>> +++ b/hw/acpi/aml-build.c
>> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>>      tables->table_data = g_array_new(false, true /* clear */, 1);
>>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
>> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>>      tables->linker = bios_linker_loader_init();
>>  }
>>  
>> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>>      g_array_free(tables->table_data, true);
>>      g_array_free(tables->tcpalog, mfre);
>>      g_array_free(tables->vmgenid, mfre);
>> +    g_array_free(tables->hardware_errors, mfre);
>>  }
>>  
>>  /* Build rsdt table */
>> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
>> new file mode 100644
>> index 0000000..ff6b5ef
>> --- /dev/null
>> +++ b/hw/acpi/hest_ghes.c
>> @@ -0,0 +1,345 @@
>> +/*
>> + *  APEI GHES table Generation
>> + *
>> + *  Copyright (C) 2017 huawei.
>> + *
>> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + *
>> + */
>> +
>> +#include "qemu/osdep.h"
>> +#include "qmp-commands.h"
>> +#include "hw/acpi/acpi.h"
>> +#include "hw/acpi/aml-build.h"
>> +#include "hw/acpi/hest_ghes.h"
>> +#include "hw/nvram/fw_cfg.h"
>> +#include "sysemu/sysemu.h"
>> +#include "qemu/error-report.h"
>> +
>> +/* The structure that stands for the layout
>> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
>> + *
>> + *           etc/hardware_errors
>> + * ==========================================
>> + * +------------------+
>> + * |    address       |              +--------------+
>> + * |    registers     |              | Error Status |
>> + * | +----------------+              | Data Block 0 |
>> + * | |status_address0 |------------->| +------------+
>> + * | +----------------+              | |  CPER      |
>> + * | |status_address1 |----------+   | |  CPER      |
>> + * | +----------------+          |   | |  ....      |
>> + * | |.............   |          |   | |  CPER      |
>> + * | +----------------+          |   | +------------+
>> + * | |status_address10|-----+    |   | Error Status |
>> + * | +----------------+     |    |   | Data Block 1 |
>> + * | |ack_value0      |     |    +-->| +------------+
>> + * | +----------------+     |        | |  CPER      |
>> + * | |ack_value1      |     |        | |  CPER      |
>> + * | +----------------+     |        | |  ....      |
>> + * | | .............  |     |        | |  CPER      |
>> + * | +----------------+     |        +-+------------+
>> + * | |ack_value10     |     |        | |..........  |
>> + * | +----------------+     |        | +------------+
>> + *                          |        | Error Status |
>> + *                          |        | Data Block10 |
>> + *                          +------->+------------+
>> + *                                   | |  CPER      |
>> + *                                   | |  CPER      |
>> + *                                   | |  ....      |
>> + *                                   | |  CPER      |
>> + *                                   +-+------------+
>> + */
>> +struct hardware_errors_buffer {
>> +    /* Generic Error Status Block register */
>> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +};
>> +
>> +static int ghes_record_cper(uint64_t error_block_address,
>> +                                    uint64_t error_physical_addr)
>> +{
>> +    AcpiGenericErrorStatus block;
>> +    AcpiGenericErrorData *gdata;
>> +    UefiCperSecMemErr *mem_err;
>> +    uint64_t current_block_length;
>> +    unsigned char *buffer;
>> +    /* memory section */
>> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
>> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
>> +                              0x83, 0xB1};
>> +
>> +    cpu_physical_memory_read(error_block_address, &block,
>> +                                sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Get the current generic error status block length */
>> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
>> +        le32_to_cpu(block.data_length);
>> +
>> +    /* If the Generic Error Status Block is NULL, update
>> +     * the block header
>> +     */
>> +    if (!block.block_status) {
>> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
>> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    }
>> +
>> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
>> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    /* check whether it runs out of the preallocated memory */
>> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
>> +       GHES_MAX_RAW_DATA_LENGTH) {
>> +        error_report("Record CPER out of boundary!!!");
>> +        return GHES_CPER_FAIL;
>> +    }
>> +
>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> looks redundant, g_malloc0 does it for you
> 
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> 
> I'd prefer for you to use build_append_int_noprefix() API to compose
> whole error status block
> 
> and try to get rid of most structures you introduce in patch 1/6,
> as they will be left unused after that.
> 
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
> It's mostly generic GAS structure with linker addition.
> I'd suggest to reuse something like
>  https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
> to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().
> 
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
> it's confusing name for var,
> AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> also, I'm not sure why it's needed at all.
> 
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
> it's really hard to get why you use offsetof() so much in this function,
> to me above code totally unreadable.
> 
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
> it looks like you are trying to build several tables within one function,
> so it's hard to get what's going on.
> I'd suggest to build separate table independently where it's possible.
> 
> i.e. build independent tables first
> and only then build dependent tables passing to it pointers
> to previously build table if necessary.
> 
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
> I'd suggest to use spec field name wit table prefix, ex:
> ACPI_HEST_ERROR_SOURCE_COUNT
> 
> also, beside build_append_int_noprefix() you need to at least
> add comment that exactly matches field from spec.
> 
> the same applies to other fields you are adding in this patch
> 
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> just do something like this instead of build_address():
> build_append_gas()
> bios_linker_loader_add_pointer()
> 
> also register width 0x40 looks suspicious, where does it come from?
> While at it do you have a real hardware which has HEST table that you re trying to model after?
> I'd like to see HEST and other related tables from it.
> 
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
>> +        /* Build read ack register */
>> +        build_address(table_data, linker, read_ack_register_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2),
>> +            offsetof(struct hardware_errors_buffer, ack_value) +
>> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
>> +            4 /* QWord access */);
>> +
>> +        /* Read ack preserve */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
>> +
>> +        /* Read ack write */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
>> +    }
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Patch address of generic error status block into
>> +         * the address register so OSPM can retrieve and read it.
>> +         */
>> +        bios_linker_loader_add_pointer(linker,
>> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
>> +            GHES_ERRORS_FW_CFG_FILE,
>> +            offsetof(struct hardware_errors_buffer, gesb) +
>> +            i * GHES_MAX_RAW_DATA_LENGTH);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
>> +     * so QEMU can write the ERRORS there. The address is expected to be
>> +     * < 4GB, but write 64 bits anyway.
>> +     */
>> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
>> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
>> +        offsetof(struct hardware_errors_buffer, gesb));
>> +
>> +    build_header(linker, table_data,
>> +        (void *)(table_data->data + ghes_start), "HEST",
>> +        table_data->len - ghes_start, 1, NULL, "GHES");
>> +}
>> +
>> +static GhesState ges;
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
>> +{
>> +
>> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
>> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
>> +
>> +    /* Create a read-only fw_cfg file for GHES */
>> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
>> +                    size);
>> +    /* Create a read-write fw_cfg file for Address */
>> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
>> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
>> +}
>> +
>> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
>> +{
>> +    uint64_t error_block_addr;
>> +    uint64_t ack_value_addr, ack_value = 0;
>> +    int loop = 0, ack_value_size;
>> +    bool ret = GHES_CPER_FAIL;
>> +
>> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
>> +        offsetof(struct hardware_errors_buffer, ack_value)) /
>> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
>> +
>> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
>> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
>> +        error_block_addr = le32_to_cpu(error_block_addr);
>> +
>> +        ack_value_addr = ges.ghes_addr_le -
>> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
>> +retry:
>> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
>> +        if (!ack_value) {
>> +            if (loop < 3) {
>> +                usleep(100 * 1000);
>> +                loop++;
>> +                goto retry;
>> +            } else {
>> +                error_report("Last time OSPM does not acknowledge the error,"
>> +                    " record CPER failed this time, set the ack value to"
>> +                    " avoid blocking next time CPER record! exit");
>> +                ack_value = 1;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                return ret;
>> +            }
>> +        } else {
>> +            /* A zero value in ghes_addr means that BIOS has not yet written
>> +             * the address
>> +             */
>> +            if (error_block_addr) {
>> +                ack_value = 0;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                ret = ghes_record_cper(error_block_addr, physical_address);
>> +            }
>> +        }
>> +    }
>> +    return ret;
>> +}
>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>> index 3d78ff6..def1ec1 100644
>> --- a/hw/arm/virt-acpi-build.c
>> +++ b/hw/arm/virt-acpi-build.c
>> @@ -45,6 +45,7 @@
>>  #include "hw/arm/virt.h"
>>  #include "sysemu/numa.h"
>>  #include "kvm_arm.h"
>> +#include "hw/acpi/hest_ghes.h"
>>  
>>  #define ARM_SPI_BASE 32
>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>      acpi_add_table(table_offsets, tables_blob);
>>      build_spcr(tables_blob, tables->linker, vms);
>>  
>> +    acpi_add_table(table_offsets, tables_blob);
>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>> +
>>      if (nb_numa_nodes > 0) {
>>          acpi_add_table(table_offsets, tables_blob);
>>          build_srat(tables_blob, tables->linker, vms);
>> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>>                      acpi_data_len(tables.tcpalog));
>>  
>> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
>> +
>>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>>                                                ACPI_BUILD_RSDP_FILE, 0);
>>  
>> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
>> index 88d0738..7f7b55c 100644
>> --- a/include/hw/acpi/aml-build.h
>> +++ b/include/hw/acpi/aml-build.h
>> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>>      GArray *rsdp;
>>      GArray *tcpalog;
>>      GArray *vmgenid;
>> +    GArray *hardware_errors;
>>      BIOSLinker *linker;
>>  } AcpiBuildTables;
>>  
>> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
>> new file mode 100644
>> index 0000000..0772756
>> --- /dev/null
>> +++ b/include/hw/acpi/hest_ghes.h
>> @@ -0,0 +1,47 @@
>> +/*
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> +
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * Authors:
>> + *   Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * You should have received a copy of the GNU General Public License along
>> + * with this program; if not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#ifndef ACPI_GHES_H
>> +#define ACPI_GHES_H
>> +
>> +#include "hw/acpi/bios-linker-loader.h"
>> +
>> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
>> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
>> +
>> +#define GHES_GAS_ADDRESS_OFFSET              4
>> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
>> +#define GHES_NOTIFICATION_STRUCTURE          32
>> +
>> +#define GHES_CPER_OK   1
>> +#define GHES_CPER_FAIL 0
>> +
>> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
>> +/* The max size in Bytes for one error block */
>> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
>> +
>> +
>> +typedef struct GhesState {
>> +    uint64_t ghes_addr_le;
>> +} GhesState;
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                            BIOSLinker *linker);
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
>> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
>> +#endif
> 
> 
> .
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-08-29 11:15       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-08-29 11:15 UTC (permalink / raw)
  To: linux-arm-kernel

Igor,
  Thank you very much for your review and comments, I will check your comments in detail and reply to you.


On 2017/8/29 18:20, Igor Mammedov wrote:
> On Fri, 18 Aug 2017 22:23:43 +0800
> Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> 
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
> 
> it's a bit complex patch/functionality so I've just mosty skimmed and
> commented only on structure of the patch and changes I'd like to see
> so it would be more structured and review-able.
> 
> I'd suggest to add doc patch first which will describe how it's
> supposed to work between QEMU/firmware/guest OS with expected
> flows.
>  
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
> these diagram shows relations between tables which not necessarily bad
> but as layout it's useless.
>  * Probably there is not much sense to have HEST table here, it's described
>    well enough in spec. You might just put reference here.
>  * these diagrams should go into doc/spec patch
>  * when you describe layout you need to show what and at what offsets
>    in which order in which blob/file is located. See ACPI spec for example
>    and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.
> 
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  hw/acpi/aml-build.c         |   2 +
>>  hw/acpi/hest_ghes.c         | 345 ++++++++++++++++++++++++++++++++++++++++++++
>>  hw/arm/virt-acpi-build.c    |   6 +
>>  include/hw/acpi/aml-build.h |   1 +
>>  include/hw/acpi/hest_ghes.h |  47 ++++++
>>  5 files changed, 401 insertions(+)
>>  create mode 100644 hw/acpi/hest_ghes.c
>>  create mode 100644 include/hw/acpi/hest_ghes.h
>>
>> diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
>> index 36a6cc4..6849e5f 100644
>> --- a/hw/acpi/aml-build.c
>> +++ b/hw/acpi/aml-build.c
>> @@ -1561,6 +1561,7 @@ void acpi_build_tables_init(AcpiBuildTables *tables)
>>      tables->table_data = g_array_new(false, true /* clear */, 1);
>>      tables->tcpalog = g_array_new(false, true /* clear */, 1);
>>      tables->vmgenid = g_array_new(false, true /* clear */, 1);
>> +    tables->hardware_errors = g_array_new(false, true /* clear */, 1);
>>      tables->linker = bios_linker_loader_init();
>>  }
>>  
>> @@ -1571,6 +1572,7 @@ void acpi_build_tables_cleanup(AcpiBuildTables *tables, bool mfre)
>>      g_array_free(tables->table_data, true);
>>      g_array_free(tables->tcpalog, mfre);
>>      g_array_free(tables->vmgenid, mfre);
>> +    g_array_free(tables->hardware_errors, mfre);
>>  }
>>  
>>  /* Build rsdt table */
>> diff --git a/hw/acpi/hest_ghes.c b/hw/acpi/hest_ghes.c
>> new file mode 100644
>> index 0000000..ff6b5ef
>> --- /dev/null
>> +++ b/hw/acpi/hest_ghes.c
>> @@ -0,0 +1,345 @@
>> +/*
>> + *  APEI GHES table Generation
>> + *
>> + *  Copyright (C) 2017 huawei.
>> + *
>> + *  Author: Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + *
>> + */
>> +
>> +#include "qemu/osdep.h"
>> +#include "qmp-commands.h"
>> +#include "hw/acpi/acpi.h"
>> +#include "hw/acpi/aml-build.h"
>> +#include "hw/acpi/hest_ghes.h"
>> +#include "hw/nvram/fw_cfg.h"
>> +#include "sysemu/sysemu.h"
>> +#include "qemu/error-report.h"
>> +
>> +/* The structure that stands for the layout
>> + * GHES_ERRORS_FW_CFG_FILE fw_cfg blob
>> + *
>> + *           etc/hardware_errors
>> + * ==========================================
>> + * +------------------+
>> + * |    address       |              +--------------+
>> + * |    registers     |              | Error Status |
>> + * | +----------------+              | Data Block 0 |
>> + * | |status_address0 |------------->| +------------+
>> + * | +----------------+              | |  CPER      |
>> + * | |status_address1 |----------+   | |  CPER      |
>> + * | +----------------+          |   | |  ....      |
>> + * | |.............   |          |   | |  CPER      |
>> + * | +----------------+          |   | +------------+
>> + * | |status_address10|-----+    |   | Error Status |
>> + * | +----------------+     |    |   | Data Block 1 |
>> + * | |ack_value0      |     |    +-->| +------------+
>> + * | +----------------+     |        | |  CPER      |
>> + * | |ack_value1      |     |        | |  CPER      |
>> + * | +----------------+     |        | |  ....      |
>> + * | | .............  |     |        | |  CPER      |
>> + * | +----------------+     |        +-+------------+
>> + * | |ack_value10     |     |        | |..........  |
>> + * | +----------------+     |        | +------------+
>> + *                          |        | Error Status |
>> + *                          |        | Data Block10 |
>> + *                          +------->+------------+
>> + *                                   | |  CPER      |
>> + *                                   | |  CPER      |
>> + *                                   | |  ....      |
>> + *                                   | |  CPER      |
>> + *                                   +-+------------+
>> + */
>> +struct hardware_errors_buffer {
>> +    /* Generic Error Status Block register */
>> +    uint64_t gesb_address[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    uint64_t ack_value[GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +    char gesb[GHES_MAX_RAW_DATA_LENGTH][GHES_ACPI_HEST_NOTIFY_RESERVED];
>> +};
>> +
>> +static int ghes_record_cper(uint64_t error_block_address,
>> +                                    uint64_t error_physical_addr)
>> +{
>> +    AcpiGenericErrorStatus block;
>> +    AcpiGenericErrorData *gdata;
>> +    UefiCperSecMemErr *mem_err;
>> +    uint64_t current_block_length;
>> +    unsigned char *buffer;
>> +    /* memory section */
>> +    char mem_section_id_le[] = {0x14, 0x11, 0xBC, 0xA5, 0x64, 0x6F, 0xDE,
>> +                              0x4E, 0xB8, 0x63, 0x3E, 0x83, 0xED, 0x7C,
>> +                              0x83, 0xB1};
>> +
>> +    cpu_physical_memory_read(error_block_address, &block,
>> +                                sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Get the current generic error status block length */
>> +    current_block_length = sizeof(AcpiGenericErrorStatus) +
>> +        le32_to_cpu(block.data_length);
>> +
>> +    /* If the Generic Error Status Block is NULL, update
>> +     * the block header
>> +     */
>> +    if (!block.block_status) {
>> +        block.block_status = ACPI_GEBS_UNCORRECTABLE;
>> +        block.error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    }
>> +
>> +    block.data_length += cpu_to_le32(sizeof(AcpiGenericErrorData));
>> +    block.data_length += cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    /* check whether it runs out of the preallocated memory */
>> +    if ((le32_to_cpu(block.data_length) + sizeof(AcpiGenericErrorStatus)) >
>> +       GHES_MAX_RAW_DATA_LENGTH) {
>> +        error_report("Record CPER out of boundary!!!");
>> +        return GHES_CPER_FAIL;
>> +    }
>> +
>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> looks redundant, g_malloc0 does it for you
> 
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> 
> I'd prefer for you to use build_append_int_noprefix() API to compose
> whole error status block
> 
> and try to get rid of most structures you introduce in patch 1/6,
> as they will be left unused after that.
> 
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
> It's mostly generic GAS structure with linker addition.
> I'd suggest to reuse something like
>  https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
> to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().
> 
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
> it's confusing name for var,
> AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> also, I'm not sure why it's needed at all.
> 
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
> it's really hard to get why you use offsetof() so much in this function,
> to me above code totally unreadable.
> 
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
> it looks like you are trying to build several tables within one function,
> so it's hard to get what's going on.
> I'd suggest to build separate table independently where it's possible.
> 
> i.e. build independent tables first
> and only then build dependent tables passing to it pointers
> to previously build table if necessary.
> 
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
> I'd suggest to use spec field name wit table prefix, ex:
> ACPI_HEST_ERROR_SOURCE_COUNT
> 
> also, beside build_append_int_noprefix() you need to at least
> add comment that exactly matches field from spec.
> 
> the same applies to other fields you are adding in this patch
> 
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> just do something like this instead of build_address():
> build_append_gas()
> bios_linker_loader_add_pointer()
> 
> also register width 0x40 looks suspicious, where does it come from?
> While at it do you have a real hardware which has HEST table that you re trying to model after?
> I'd like to see HEST and other related tables from it.
> 
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
>> +        /* Build read ack register */
>> +        build_address(table_data, linker, read_ack_register_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2),
>> +            offsetof(struct hardware_errors_buffer, ack_value) +
>> +            i * address_size, AML_SYSTEM_MEMORY, 0x40, 0,
>> +            4 /* QWord access */);
>> +
>> +        /* Read ack preserve */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0xfffffffe), 8);
>> +
>> +        /* Read ack write */
>> +        build_append_int_noprefix(table_data, cpu_to_le64(0x1), 8);
>> +    }
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Patch address of generic error status block into
>> +         * the address register so OSPM can retrieve and read it.
>> +         */
>> +        bios_linker_loader_add_pointer(linker,
>> +            GHES_ERRORS_FW_CFG_FILE, address_size * i, address_size,
>> +            GHES_ERRORS_FW_CFG_FILE,
>> +            offsetof(struct hardware_errors_buffer, gesb) +
>> +            i * GHES_MAX_RAW_DATA_LENGTH);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the ADDR fw_cfg blob
>> +     * so QEMU can write the ERRORS there. The address is expected to be
>> +     * < 4GB, but write 64 bits anyway.
>> +     */
>> +    bios_linker_loader_write_pointer(linker, GHES_DATA_ADDR_FW_CFG_FILE,
>> +        0, address_size, GHES_ERRORS_FW_CFG_FILE,
>> +        offsetof(struct hardware_errors_buffer, gesb));
>> +
>> +    build_header(linker, table_data,
>> +        (void *)(table_data->data + ghes_start), "HEST",
>> +        table_data->len - ghes_start, 1, NULL, "GHES");
>> +}
>> +
>> +static GhesState ges;
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_error)
>> +{
>> +
>> +    size_t request_block_size = sizeof(uint64_t) + GHES_MAX_RAW_DATA_LENGTH;
>> +    size_t size = GHES_ACPI_HEST_NOTIFY_RESERVED * request_block_size;
>> +
>> +    /* Create a read-only fw_cfg file for GHES */
>> +    fw_cfg_add_file(s, GHES_ERRORS_FW_CFG_FILE, hardware_error->data,
>> +                    size);
>> +    /* Create a read-write fw_cfg file for Address */
>> +    fw_cfg_add_file_callback(s, GHES_DATA_ADDR_FW_CFG_FILE, NULL, NULL,
>> +        &ges.ghes_addr_le, sizeof(ges.ghes_addr_le), false);
>> +}
>> +
>> +bool ghes_update_guest(uint32_t notify, uint64_t physical_address)
>> +{
>> +    uint64_t error_block_addr;
>> +    uint64_t ack_value_addr, ack_value = 0;
>> +    int loop = 0, ack_value_size;
>> +    bool ret = GHES_CPER_FAIL;
>> +
>> +    ack_value_size = (offsetof(struct hardware_errors_buffer, gesb) -
>> +        offsetof(struct hardware_errors_buffer, ack_value)) /
>> +            GHES_ACPI_HEST_NOTIFY_RESERVED;
>> +
>> +    if (physical_address && notify < GHES_ACPI_HEST_NOTIFY_RESERVED) {
>> +        error_block_addr = ges.ghes_addr_le + notify * GHES_MAX_RAW_DATA_LENGTH;
>> +        error_block_addr = le32_to_cpu(error_block_addr);
>> +
>> +        ack_value_addr = ges.ghes_addr_le -
>> +            (GHES_ACPI_HEST_NOTIFY_RESERVED - notify) * ack_value_size;
>> +retry:
>> +        cpu_physical_memory_read(ack_value_addr, &ack_value, ack_value_size);
>> +        if (!ack_value) {
>> +            if (loop < 3) {
>> +                usleep(100 * 1000);
>> +                loop++;
>> +                goto retry;
>> +            } else {
>> +                error_report("Last time OSPM does not acknowledge the error,"
>> +                    " record CPER failed this time, set the ack value to"
>> +                    " avoid blocking next time CPER record! exit");
>> +                ack_value = 1;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                return ret;
>> +            }
>> +        } else {
>> +            /* A zero value in ghes_addr means that BIOS has not yet written
>> +             * the address
>> +             */
>> +            if (error_block_addr) {
>> +                ack_value = 0;
>> +                cpu_physical_memory_write(ack_value_addr,
>> +                    &ack_value, ack_value_size);
>> +                ret = ghes_record_cper(error_block_addr, physical_address);
>> +            }
>> +        }
>> +    }
>> +    return ret;
>> +}
>> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
>> index 3d78ff6..def1ec1 100644
>> --- a/hw/arm/virt-acpi-build.c
>> +++ b/hw/arm/virt-acpi-build.c
>> @@ -45,6 +45,7 @@
>>  #include "hw/arm/virt.h"
>>  #include "sysemu/numa.h"
>>  #include "kvm_arm.h"
>> +#include "hw/acpi/hest_ghes.h"
>>  
>>  #define ARM_SPI_BASE 32
>>  #define ACPI_POWER_BUTTON_DEVICE "PWRB"
>> @@ -771,6 +772,9 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
>>      acpi_add_table(table_offsets, tables_blob);
>>      build_spcr(tables_blob, tables->linker, vms);
>>  
>> +    acpi_add_table(table_offsets, tables_blob);
>> +    ghes_build_acpi(tables_blob, tables->hardware_errors, tables->linker);
>> +
>>      if (nb_numa_nodes > 0) {
>>          acpi_add_table(table_offsets, tables_blob);
>>          build_srat(tables_blob, tables->linker, vms);
>> @@ -887,6 +891,8 @@ void virt_acpi_setup(VirtMachineState *vms)
>>      fw_cfg_add_file(vms->fw_cfg, ACPI_BUILD_TPMLOG_FILE, tables.tcpalog->data,
>>                      acpi_data_len(tables.tcpalog));
>>  
>> +    ghes_add_fw_cfg(vms->fw_cfg, tables.hardware_errors);
>> +
>>      build_state->rsdp_mr = acpi_add_rom_blob(build_state, tables.rsdp,
>>                                                ACPI_BUILD_RSDP_FILE, 0);
>>  
>> diff --git a/include/hw/acpi/aml-build.h b/include/hw/acpi/aml-build.h
>> index 88d0738..7f7b55c 100644
>> --- a/include/hw/acpi/aml-build.h
>> +++ b/include/hw/acpi/aml-build.h
>> @@ -211,6 +211,7 @@ struct AcpiBuildTables {
>>      GArray *rsdp;
>>      GArray *tcpalog;
>>      GArray *vmgenid;
>> +    GArray *hardware_errors;
>>      BIOSLinker *linker;
>>  } AcpiBuildTables;
>>  
>> diff --git a/include/hw/acpi/hest_ghes.h b/include/hw/acpi/hest_ghes.h
>> new file mode 100644
>> index 0000000..0772756
>> --- /dev/null
>> +++ b/include/hw/acpi/hest_ghes.h
>> @@ -0,0 +1,47 @@
>> +/*
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> +
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * Authors:
>> + *   Dongjiu Geng <gengdongjiu@huawei.com>
>> + *
>> + * You should have received a copy of the GNU General Public License along
>> + * with this program; if not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#ifndef ACPI_GHES_H
>> +#define ACPI_GHES_H
>> +
>> +#include "hw/acpi/bios-linker-loader.h"
>> +
>> +#define GHES_ERRORS_FW_CFG_FILE      "etc/hardware_errors"
>> +#define GHES_DATA_ADDR_FW_CFG_FILE      "etc/hardware_errors_addr"
>> +
>> +#define GHES_GAS_ADDRESS_OFFSET              4
>> +#define GHES_ERROR_STATUS_ADDRESS_OFFSET     20
>> +#define GHES_NOTIFICATION_STRUCTURE          32
>> +
>> +#define GHES_CPER_OK   1
>> +#define GHES_CPER_FAIL 0
>> +
>> +#define GHES_ACPI_HEST_NOTIFY_RESERVED           11
>> +/* The max size in Bytes for one error block */
>> +#define GHES_MAX_RAW_DATA_LENGTH                 0x1000
>> +
>> +
>> +typedef struct GhesState {
>> +    uint64_t ghes_addr_le;
>> +} GhesState;
>> +
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                            BIOSLinker *linker);
>> +void ghes_add_fw_cfg(FWCfgState *s, GArray *hardware_errors);
>> +bool ghes_update_guest(uint32_t notify, uint64_t error_physical_addr);
>> +#endif
> 
> 
> .
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
  2017-08-29 10:20     ` Igor Mammedov
                         ` (2 preceding siblings ...)
  (?)
@ 2017-09-01  9:58       ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-09-01  9:58 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: mst, zhaoshenglong, lersek, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, ard.biesheuvel, mingo, bp, shiju.jose,
	zjzhang, linux-arm-kernel, kvmarm, linux-kernel, linux-acpi,
	devel, john.garry, jonathan.cameron, shameerali.kolothum.thodi,
	huangdaode, wangzhou1, zhengqiang10, wuquanming

Hi Igor,

On 2017/8/29 18:20, Igor Mammedov wrote:
> On Fri, 18 Aug 2017 22:23:43 +0800
> Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> 
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
> 
> it's a bit complex patch/functionality so I've just mosty skimmed and
> commented only on structure of the patch and changes I'd like to see
> so it would be more structured and review-able.

I will make it more structured and review-able, so that it is easily readable.

> 
> I'd suggest to add doc patch first which will describe how it's
> supposed to work between QEMU/firmware/guest OS with expected
> flows.
It is sure, I will add a doc patch.

>  
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
> these diagram shows relations between tables which not necessarily bad
> but as layout it's useless.
Maybe not yet. it shows how the table is generated and how the CPER is recorded through fw_cfg blob
between Qemu and guest firmware. for example: etc/acpi/tables and etc/hardware_errors
anyway I will move it to the spec/doc.

>  * Probably there is not much sense to have HEST table here, it's described
>    well enough in spec. You might just put reference here.
>  * these diagrams should go into doc/spec patch
I will move these diagrams to doc/spec patch

>  * when you describe layout you need to show what and at what offsets
>    in which order in which blob/file is located. See ACPI spec for example
>    and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.
Ok, thanks for the suggestion.

> 
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---

cut----

>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> looks redundant, g_malloc0 does it for you
   I will remove it.

> 
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> 
> I'd prefer for you to use build_append_int_noprefix() API to compose
> whole error status block
  I will use build_append_int_noprefix() API to compose the block status

> 
> and try to get rid of most structures you introduce in patch 1/6,
> as they will be left unused after that.
I will clear the structures and remove the unused structures that this patch does not used.

> 
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
> It's mostly generic GAS structure with linker addition.
> I'd suggest to reuse something like
>  https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
> to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().
thanks, OK.

> 
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
> it's confusing name for var,
> AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> also, I'm not sure why it's needed at all.
 it is because other people have concern about where does the "unsigned 64 bit integer"
 come from, they are confused about the "unsigned 64 bit integer"
 so they suggested use sizeof. anyway I will directly use unsigned 64 bit integer.

> 
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
> it's really hard to get why you use offsetof() so much in this function,
> to me above code totally unreadable.
I will find method to make it easily readable.

> 
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
> it looks like you are trying to build several tables within one function,
> so it's hard to get what's going on.
> I'd suggest to build separate table independently where it's possible.
> 
> i.e. build independent tables first
> and only then build dependent tables passing to it pointers
> to previously build table if necessary.
thanks for the suggestion, I will make the table independently as far as possible

> 
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
> I'd suggest to use spec field name wit table prefix, ex:
> ACPI_HEST_ERROR_SOURCE_COUNT
thanks for the suggestion, I will use your suggested name.

> 
> also, beside build_append_int_noprefix() you need to at least
> add comment that exactly matches field from spec.
> 
> the same applies to other fields you are adding in this patch
  OK, will add it.

> 
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> just do something like this instead of build_address():
> build_append_gas()
> bios_linker_loader_add_pointer()
Thanks for the suggestion.

> 
> also register width 0x40 looks suspicious, where does it come from?
> While at it do you have a real hardware which has HEST table that you re trying to model after?
> I'd like to see HEST and other related tables from it.

Igor, what is your suspicious point? The register width 0x40 come from our host BIOS record to the System Memory space.

For the SEA/SEI, the Qemu(user space) only handle the memory section hardware error, not include processor error. so it may not
involve other Address Space, such as System I/O space

it is sure we have hardware. I share our BIOS code that generate HEST and other related table here.
https://github.com/hisilicon/uefi.git
Now we have also submitted the host BIOS code to open source for review, this code generated the HEST and other related table
thanks.

> 
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
cut -----


^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-09-01  9:58       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-09-01  9:58 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: mst, zhaoshenglong, lersek, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, ard.biesheuvel, mingo, bp, shiju.jose,
	zjzhang, linux-arm-kernel, kvmarm, linux-kernel, linux-acpi,
	devel, john.garry, jonathan.cameron, shameerali.kolothum.thodi,
	huangdaode, wangzhou1, zhengqiang10, wuquanming, huangshaoyu,
	linuxarm

Hi Igor,

On 2017/8/29 18:20, Igor Mammedov wrote:
> On Fri, 18 Aug 2017 22:23:43 +0800
> Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> 
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
> 
> it's a bit complex patch/functionality so I've just mosty skimmed and
> commented only on structure of the patch and changes I'd like to see
> so it would be more structured and review-able.

I will make it more structured and review-able, so that it is easily readable.

> 
> I'd suggest to add doc patch first which will describe how it's
> supposed to work between QEMU/firmware/guest OS with expected
> flows.
It is sure, I will add a doc patch.

>  
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
> these diagram shows relations between tables which not necessarily bad
> but as layout it's useless.
Maybe not yet. it shows how the table is generated and how the CPER is recorded through fw_cfg blob
between Qemu and guest firmware. for example: etc/acpi/tables and etc/hardware_errors
anyway I will move it to the spec/doc.

>  * Probably there is not much sense to have HEST table here, it's described
>    well enough in spec. You might just put reference here.
>  * these diagrams should go into doc/spec patch
I will move these diagrams to doc/spec patch

>  * when you describe layout you need to show what and at what offsets
>    in which order in which blob/file is located. See ACPI spec for example
>    and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.
Ok, thanks for the suggestion.

> 
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---

cut----

>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> looks redundant, g_malloc0 does it for you
   I will remove it.

> 
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> 
> I'd prefer for you to use build_append_int_noprefix() API to compose
> whole error status block
  I will use build_append_int_noprefix() API to compose the block status

> 
> and try to get rid of most structures you introduce in patch 1/6,
> as they will be left unused after that.
I will clear the structures and remove the unused structures that this patch does not used.

> 
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
> It's mostly generic GAS structure with linker addition.
> I'd suggest to reuse something like
>  https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
> to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().
thanks, OK.

> 
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
> it's confusing name for var,
> AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> also, I'm not sure why it's needed at all.
 it is because other people have concern about where does the "unsigned 64 bit integer"
 come from, they are confused about the "unsigned 64 bit integer"
 so they suggested use sizeof. anyway I will directly use unsigned 64 bit integer.

> 
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
> it's really hard to get why you use offsetof() so much in this function,
> to me above code totally unreadable.
I will find method to make it easily readable.

> 
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
> it looks like you are trying to build several tables within one function,
> so it's hard to get what's going on.
> I'd suggest to build separate table independently where it's possible.
> 
> i.e. build independent tables first
> and only then build dependent tables passing to it pointers
> to previously build table if necessary.
thanks for the suggestion, I will make the table independently as far as possible

> 
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
> I'd suggest to use spec field name wit table prefix, ex:
> ACPI_HEST_ERROR_SOURCE_COUNT
thanks for the suggestion, I will use your suggested name.

> 
> also, beside build_append_int_noprefix() you need to at least
> add comment that exactly matches field from spec.
> 
> the same applies to other fields you are adding in this patch
  OK, will add it.

> 
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> just do something like this instead of build_address():
> build_append_gas()
> bios_linker_loader_add_pointer()
Thanks for the suggestion.

> 
> also register width 0x40 looks suspicious, where does it come from?
> While at it do you have a real hardware which has HEST table that you re trying to model after?
> I'd like to see HEST and other related tables from it.

Igor, what is your suspicious point? The register width 0x40 come from our host BIOS record to the System Memory space.

For the SEA/SEI, the Qemu(user space) only handle the memory section hardware error, not include processor error. so it may not
involve other Address Space, such as System I/O space

it is sure we have hardware. I share our BIOS code that generate HEST and other related table here.
https://github.com/hisilicon/uefi.git
Now we have also submitted the host BIOS code to open source for review, this code generated the HEST and other related table
thanks.

> 
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
cut -----

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-09-01  9:58       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-09-01  9:58 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: mst, zhaoshenglong, lersek, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, ard.biesheuvel, mingo, bp, shiju.jose,
	zjzhang, linux-arm-kernel, kvmarm, linux-kernel, linux-acpi,
	devel, john.garry, jonathan.cameron, shameerali.kolothum.thodi,
	huangdaode, wangzhou1, zhengqiang10, wuquanming

Hi Igor,

On 2017/8/29 18:20, Igor Mammedov wrote:
> On Fri, 18 Aug 2017 22:23:43 +0800
> Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> 
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
> 
> it's a bit complex patch/functionality so I've just mosty skimmed and
> commented only on structure of the patch and changes I'd like to see
> so it would be more structured and review-able.

I will make it more structured and review-able, so that it is easily readable.

> 
> I'd suggest to add doc patch first which will describe how it's
> supposed to work between QEMU/firmware/guest OS with expected
> flows.
It is sure, I will add a doc patch.

>  
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
> these diagram shows relations between tables which not necessarily bad
> but as layout it's useless.
Maybe not yet. it shows how the table is generated and how the CPER is recorded through fw_cfg blob
between Qemu and guest firmware. for example: etc/acpi/tables and etc/hardware_errors
anyway I will move it to the spec/doc.

>  * Probably there is not much sense to have HEST table here, it's described
>    well enough in spec. You might just put reference here.
>  * these diagrams should go into doc/spec patch
I will move these diagrams to doc/spec patch

>  * when you describe layout you need to show what and at what offsets
>    in which order in which blob/file is located. See ACPI spec for example
>    and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.
Ok, thanks for the suggestion.

> 
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---

cut----

>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> looks redundant, g_malloc0 does it for you
   I will remove it.

> 
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> 
> I'd prefer for you to use build_append_int_noprefix() API to compose
> whole error status block
  I will use build_append_int_noprefix() API to compose the block status

> 
> and try to get rid of most structures you introduce in patch 1/6,
> as they will be left unused after that.
I will clear the structures and remove the unused structures that this patch does not used.

> 
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
> It's mostly generic GAS structure with linker addition.
> I'd suggest to reuse something like
>  https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
> to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().
thanks, OK.

> 
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
> it's confusing name for var,
> AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> also, I'm not sure why it's needed at all.
 it is because other people have concern about where does the "unsigned 64 bit integer"
 come from, they are confused about the "unsigned 64 bit integer"
 so they suggested use sizeof. anyway I will directly use unsigned 64 bit integer.

> 
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
> it's really hard to get why you use offsetof() so much in this function,
> to me above code totally unreadable.
I will find method to make it easily readable.

> 
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
> it looks like you are trying to build several tables within one function,
> so it's hard to get what's going on.
> I'd suggest to build separate table independently where it's possible.
> 
> i.e. build independent tables first
> and only then build dependent tables passing to it pointers
> to previously build table if necessary.
thanks for the suggestion, I will make the table independently as far as possible

> 
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
> I'd suggest to use spec field name wit table prefix, ex:
> ACPI_HEST_ERROR_SOURCE_COUNT
thanks for the suggestion, I will use your suggested name.

> 
> also, beside build_append_int_noprefix() you need to at least
> add comment that exactly matches field from spec.
> 
> the same applies to other fields you are adding in this patch
  OK, will add it.

> 
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> just do something like this instead of build_address():
> build_append_gas()
> bios_linker_loader_add_pointer()
Thanks for the suggestion.

> 
> also register width 0x40 looks suspicious, where does it come from?
> While at it do you have a real hardware which has HEST table that you re trying to model after?
> I'd like to see HEST and other related tables from it.

Igor, what is your suspicious point? The register width 0x40 come from our host BIOS record to the System Memory space.

For the SEA/SEI, the Qemu(user space) only handle the memory section hardware error, not include processor error. so it may not
involve other Address Space, such as System I/O space

it is sure we have hardware. I share our BIOS code that generate HEST and other related table here.
https://github.com/hisilicon/uefi.git
Now we have also submitted the host BIOS code to open source for review, this code generated the HEST and other related table
thanks.

> 
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
cut -----


^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-09-01  9:58       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-09-01  9:58 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: mst, zhaoshenglong, lersek, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, ard.biesheuvel, mingo, bp, shiju.jose,
	zjzhang, linux-arm-kernel, kvmarm, linux-kernel, linux-acpi,
	devel, john.garry, jonathan.cameron, shameerali.kolothum.thodi,
	huangdaode, wangzhou1, zhengqiang10, wuquanming, huangshaoyu,
	linuxarm

Hi Igor,

On 2017/8/29 18:20, Igor Mammedov wrote:
> On Fri, 18 Aug 2017 22:23:43 +0800
> Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> 
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
> 
> it's a bit complex patch/functionality so I've just mosty skimmed and
> commented only on structure of the patch and changes I'd like to see
> so it would be more structured and review-able.

I will make it more structured and review-able, so that it is easily readable.

> 
> I'd suggest to add doc patch first which will describe how it's
> supposed to work between QEMU/firmware/guest OS with expected
> flows.
It is sure, I will add a doc patch.

>  
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
> these diagram shows relations between tables which not necessarily bad
> but as layout it's useless.
Maybe not yet. it shows how the table is generated and how the CPER is recorded through fw_cfg blob
between Qemu and guest firmware. for example: etc/acpi/tables and etc/hardware_errors
anyway I will move it to the spec/doc.

>  * Probably there is not much sense to have HEST table here, it's described
>    well enough in spec. You might just put reference here.
>  * these diagrams should go into doc/spec patch
I will move these diagrams to doc/spec patch

>  * when you describe layout you need to show what and at what offsets
>    in which order in which blob/file is located. See ACPI spec for example
>    and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.
Ok, thanks for the suggestion.

> 
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---

cut----

>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> looks redundant, g_malloc0 does it for you
   I will remove it.

> 
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> 
> I'd prefer for you to use build_append_int_noprefix() API to compose
> whole error status block
  I will use build_append_int_noprefix() API to compose the block status

> 
> and try to get rid of most structures you introduce in patch 1/6,
> as they will be left unused after that.
I will clear the structures and remove the unused structures that this patch does not used.

> 
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
> It's mostly generic GAS structure with linker addition.
> I'd suggest to reuse something like
>  https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
> to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().
thanks, OK.

> 
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
> it's confusing name for var,
> AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> also, I'm not sure why it's needed at all.
 it is because other people have concern about where does the "unsigned 64 bit integer"
 come from, they are confused about the "unsigned 64 bit integer"
 so they suggested use sizeof. anyway I will directly use unsigned 64 bit integer.

> 
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
> it's really hard to get why you use offsetof() so much in this function,
> to me above code totally unreadable.
I will find method to make it easily readable.

> 
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
> it looks like you are trying to build several tables within one function,
> so it's hard to get what's going on.
> I'd suggest to build separate table independently where it's possible.
> 
> i.e. build independent tables first
> and only then build dependent tables passing to it pointers
> to previously build table if necessary.
thanks for the suggestion, I will make the table independently as far as possible

> 
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
> I'd suggest to use spec field name wit table prefix, ex:
> ACPI_HEST_ERROR_SOURCE_COUNT
thanks for the suggestion, I will use your suggested name.

> 
> also, beside build_append_int_noprefix() you need to at least
> add comment that exactly matches field from spec.
> 
> the same applies to other fields you are adding in this patch
  OK, will add it.

> 
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> just do something like this instead of build_address():
> build_append_gas()
> bios_linker_loader_add_pointer()
Thanks for the suggestion.

> 
> also register width 0x40 looks suspicious, where does it come from?
> While at it do you have a real hardware which has HEST table that you re trying to model after?
> I'd like to see HEST and other related tables from it.

Igor, what is your suspicious point? The register width 0x40 come from our host BIOS record to the System Memory space.

For the SEA/SEI, the Qemu(user space) only handle the memory section hardware error, not include processor error. so it may not
involve other Address Space, such as System I/O space

it is sure we have hardware. I share our BIOS code that generate HEST and other related table here.
https://github.com/hisilicon/uefi.git
Now we have also submitted the host BIOS code to open source for review, this code generated the HEST and other related table
thanks.

> 
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
cut -----

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-09-01  9:58       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-09-01  9:58 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Igor,

On 2017/8/29 18:20, Igor Mammedov wrote:
> On Fri, 18 Aug 2017 22:23:43 +0800
> Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> 
>> This implements APEI GHES Table by passing the error CPER info
>> to the guest via a fw_cfg_blob. After a CPER info is recorded, an
>> SEA(Synchronous External Abort)/SEI(SError Interrupt) exception
>> will be injected into the guest OS.
> 
> it's a bit complex patch/functionality so I've just mosty skimmed and
> commented only on structure of the patch and changes I'd like to see
> so it would be more structured and review-able.

I will make it more structured and review-able, so that it is easily readable.

> 
> I'd suggest to add doc patch first which will describe how it's
> supposed to work between QEMU/firmware/guest OS with expected
> flows.
It is sure, I will add a doc patch.

>  
>> Below is the table layout, the max number of error soure is 11,
>> which is classified by notification type.
>>
>>      etc/acpi/tables                               etc/hardware_errors
>>     ====================                    ==========================================
>> + +--------------------------+            +------------------+
>> | | HEST                     |            |    address       |              +--------------+
>> | +--------------------------+            |    registers     |              | Error Status |
>> | | GHES0                    |            | +----------------+              | Data Block 0 |
>> | +--------------------------+ +--------->| |status_address0 |------------->| +------------+
>> | | .................        | |          | +----------------+              | |  CPER      |
>> | | error_status_address-----+-+ +------->| |status_address1 |----------+   | |  CPER      |
>> | | .................        |   |        | +----------------+          |   | |  ....      |
>> | | read_ack_register--------+-+ |        |  .............   |          |   | |  CPER      |
>> | | read_ack_preserve        | | |        +------------------+          |   | +-+------------+
>> | | read_ack_write           | | | +----->| |status_address10|--------+ |   | Error Status |
>> + +--------------------------+ | | |      | +----------------+        | |   | Data Block 1 |
>> | | GHES1                    | +-+-+----->| | ack_value0     |        | +-->| +------------+
>> + +--------------------------+   | |      | +----------------+        |     | |  CPER      |
>> | | .................        |   | | +--->| | ack_value1     |        |     | |  CPER      |
>> | | error_status_address-----+---+ | |    | +----------------+        |     | |  ....      |
>> | | .................        |     | |    | |  ............. |        |     | |  CPER      |
>> | | read_ack_register--------+-----+-+    | +----------------+        |     +-+------------+
>> | | read_ack_preserve        |     |   +->| | ack_value10    |        |     | |..........  |
>> | | read_ack_write           |     |   |  | +----------------+        |     | +------------+
>> + +--------------------------|     |   |                              |     | Error Status |
>> | | ...............          |     |   |                              |     | Data Block 10|
>> + +--------------------------+     |   |                              +---->| +------------+
>> | | GHES10                   |     |   |                                    | |  CPER      |
>> + +--------------------------+     |   |                                    | |  CPER      |
>> | | .................        |     |   |                                    | |  ....      |
>> | | error_status_address-----+-----+   |                                    | |  CPER      |
>> | | .................        |         |                                    +-+------------+
>> | | read_ack_register--------+---------+
>> | | read_ack_preserve        |
>> | | read_ack_write           |
>> + +--------------------------+
> these diagram shows relations between tables which not necessarily bad
> but as layout it's useless.
Maybe not yet. it shows how the table is generated and how the CPER is recorded through fw_cfg blob
between Qemu and guest firmware. for example: etc/acpi/tables and etc/hardware_errors
anyway I will move it to the spec/doc.

>  * Probably there is not much sense to have HEST table here, it's described
>    well enough in spec. You might just put reference here.
>  * these diagrams should go into doc/spec patch
I will move these diagrams to doc/spec patch

>  * when you describe layout you need to show what and at what offsets
>    in which order in which blob/file is located. See ACPI spec for example
>    and/or docs/specs/acpi_nvdimm.txt docs/specs/acpi_mem_hotplug.txt for inspiration.
Ok, thanks for the suggestion.

> 
>> For GHESv2 error source, the OSPM must acknowledges the error via Read Ack register.
>> so user space must check the ack value to avoid read-write race condition.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---

cut----

>> +    /* Write back the Generic Error Status Block to guest memory */
>> +    cpu_physical_memory_write(error_block_address, &block,
>> +        sizeof(AcpiGenericErrorStatus));
>> +
>> +    /* Fill in Generic Error Data Entry */
>> +    buffer = g_malloc0(sizeof(AcpiGenericErrorData) +
>> +                       sizeof(UefiCperSecMemErr));
>> +
>> +
>> +    memset(buffer, 0, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
> looks redundant, g_malloc0 does it for you
   I will remove it.

> 
>> +    gdata = (AcpiGenericErrorData *)buffer;
>> +
>> +    /* Memory section */
>> +    memcpy(&(gdata->section_type_le), &mem_section_id_le,
>> +            sizeof(mem_section_id_le));
>> +
>> +    /* error severity is recoverable */
>> +    gdata->error_severity = ACPI_CPER_SEV_RECOVERABLE;
>> +    gdata->revision = 0x300; /* the revision number is 0x300 */
>> +    gdata->error_data_length = cpu_to_le32(sizeof(UefiCperSecMemErr));
>> +
>> +    mem_err = (UefiCperSecMemErr *) (gdata + 1);
>> +
>> +    /* User space only handle the memory section CPER */
>> +
>> +    /* Hard code to Multi-bit ECC error */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_ERROR_TYPE);
>> +    mem_err->error_type = cpu_to_le32(UEFI_CPER_MEM_ERROR_TYPE_MULTI_ECC);
>> +
>> +    /* Record the physical address at which the memory error occurred */
>> +    mem_err->validation_bits |= cpu_to_le32(UEFI_CPER_MEM_VALID_PA);
>> +    mem_err->physical_addr = cpu_to_le32(error_physical_addr);
> 
> I'd prefer for you to use build_append_int_noprefix() API to compose
> whole error status block
  I will use build_append_int_noprefix() API to compose the block status

> 
> and try to get rid of most structures you introduce in patch 1/6,
> as they will be left unused after that.
I will clear the structures and remove the unused structures that this patch does not used.

> 
>> +
>> +    /* Write back the Generic Error Data Entry to guest memory */
>> +    cpu_physical_memory_write(error_block_address + current_block_length,
>> +        buffer, sizeof(AcpiGenericErrorData) + sizeof(UefiCperSecMemErr));
>> +
>> +    g_free(buffer);
>> +    return GHES_CPER_OK;
>> +}
>> +
>> +static void
>> +build_address(GArray *table_data, BIOSLinker *linker,
>> +    uint32_t dst_patched_offset, uint32_t src_offset,
>> +    uint8_t address_space_id , uint8_t  register_bit_width,
>> +    uint8_t register_bit_offset, uint8_t access_size)
>> +{
>> +    uint32_t address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    /* Address space */
>> +    build_append_int_noprefix(table_data, address_space_id, 1);
>> +    /* register bit width */
>> +    build_append_int_noprefix(table_data, register_bit_width, 1);
>> +    /* register bit offset */
>> +    build_append_int_noprefix(table_data, register_bit_offset, 1);
>> +    /* access size */
>> +    build_append_int_noprefix(table_data, access_size, 1);
>> +    acpi_data_push(table_data, address_size);
>> +
>> +    /* Patch address of ERRORS fw_cfg blob into the TABLE fw_cfg blob so OSPM
>> +     * can retrieve and read it. the address size is 64 bits.
>> +     */
>> +    bios_linker_loader_add_pointer(linker,
>> +        ACPI_BUILD_TABLE_FILE, dst_patched_offset, sizeof(uint64_t),
>> +        GHES_ERRORS_FW_CFG_FILE, src_offset);
>> +}
> It's mostly generic GAS structure with linker addition.
> I'd suggest to reuse something like
>  https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
> to build GAS and use bios_linker_loader_add_pointer() directly in ghes_build_acpi().
thanks, OK.

> 
>> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
>> +                                            BIOSLinker *linker)
>> +{
>> +    uint32_t ghes_start = table_data->len;
>> +    uint32_t address_size, error_status_address_offset;
>> +    uint32_t read_ack_register_offset, i;
>> +
>> +    address_size = sizeof(struct AcpiGenericAddress) -
>> +        offsetof(struct AcpiGenericAddress, address);
> it's confusing name for var,
> AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> also, I'm not sure why it's needed at all.
 it is because other people have concern about where does the "unsigned 64 bit integer"
 come from, they are confused about the "unsigned 64 bit integer"
 so they suggested use sizeof. anyway I will directly use unsigned 64 bit integer.

> 
>> +
>> +    error_status_address_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, error_status_address) +
>> +        offsetof(struct AcpiGenericAddress, address);
>> +
>> +    read_ack_register_offset = ghes_start +
>> +        sizeof(AcpiHardwareErrorSourceTable) +
>> +        offsetof(AcpiGenericHardwareErrorSourceV2, read_ack_register) +
>> +        offsetof(struct AcpiGenericAddress, address);
> it's really hard to get why you use offsetof() so much in this function,
> to me above code totally unreadable.
I will find method to make it easily readable.

> 
>> +    acpi_data_push(hardware_error,
>> +        offsetof(struct hardware_errors_buffer, ack_value));
> it looks like you are trying to build several tables within one function,
> so it's hard to get what's going on.
> I'd suggest to build separate table independently where it's possible.
> 
> i.e. build independent tables first
> and only then build dependent tables passing to it pointers
> to previously build table if necessary.
thanks for the suggestion, I will make the table independently as far as possible

> 
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++)
>> +        /* Initialize read ack register */
>> +        build_append_int_noprefix((void *)hardware_error, 1, 8);
>> +
>> +    /* Reserved the total size for ERRORS fw_cfg blob
>> +     */
>> +    acpi_data_push(hardware_error, sizeof(struct hardware_errors_buffer));
>> +
>> +    /* Allocate guest memory for the Data fw_cfg blob */
>> +    bios_linker_loader_alloc(linker, GHES_ERRORS_FW_CFG_FILE, hardware_error,
>> +                            1, false);
>> +    /* Reserve table header size */
>> +    acpi_data_push(table_data, sizeof(AcpiTableHeader));
>> +
>> +    build_append_int_noprefix(table_data, GHES_ACPI_HEST_NOTIFY_RESERVED, 4);
> GHES_ACPI_HEST_NOTIFY_RESERVED - name doesn't actually tell what it is
> I'd suggest to use spec field name wit table prefix, ex:
> ACPI_HEST_ERROR_SOURCE_COUNT
thanks for the suggestion, I will use your suggested name.

> 
> also, beside build_append_int_noprefix() you need to at least
> add comment that exactly matches field from spec.
> 
> the same applies to other fields you are adding in this patch
  OK, will add it.

> 
>> +
>> +    for (i = 0; i < GHES_ACPI_HEST_NOTIFY_RESERVED; i++) {
>> +        build_append_int_noprefix(table_data,
>> +            ACPI_HEST_SOURCE_GENERIC_ERROR_V2, 2); /* type */
>> +        /* source id */
>> +        build_append_int_noprefix(table_data, cpu_to_le16(i), 2);
>> +        /* related source id */
>> +        build_append_int_noprefix(table_data, 0xffff, 2);
>> +        build_append_int_noprefix(table_data, 0, 1); /* flags */
>> +
>> +        /* Currently only enable SEA notification type to avoid the kernel
>> +         * warning, reserve the space for other notification error source
>> +         */
>> +        if (i == ACPI_HEST_NOTIFY_SEA) {
>> +            build_append_int_noprefix(table_data, 1, 1); /* enabled */
>> +        } else {
>> +            build_append_int_noprefix(table_data, 0, 1); /* enabled */
>> +        }
>> +
>> +        /* The number of error status block per generic hardware error source */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max sections per record */
>> +        build_append_int_noprefix(table_data, 1, 4);
>> +        /* Max raw data length */
>> +        build_append_int_noprefix(table_data, GHES_MAX_RAW_DATA_LENGTH, 4);
>> +
>> +        /* Build error status address*/
>> +        build_address(table_data, linker, error_status_address_offset + i *
>> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
>> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);
> just do something like this instead of build_address():
> build_append_gas()
> bios_linker_loader_add_pointer()
Thanks for the suggestion.

> 
> also register width 0x40 looks suspicious, where does it come from?
> While at it do you have a real hardware which has HEST table that you re trying to model after?
> I'd like to see HEST and other related tables from it.

Igor, what is your suspicious point? The register width 0x40 come from our host BIOS record to the System Memory space.

For the SEA/SEI, the Qemu(user space) only handle the memory section hardware error, not include processor error. so it may not
involve other Address Space, such as System I/O space

it is sure we have hardware. I share our BIOS code that generate HEST and other related table here.
https://github.com/hisilicon/uefi.git
Now we have also submitted the host BIOS code to open source for review, this code generated the HEST and other related table
thanks.

> 
>> +
>> +        /* Hardware error notification structure */
>> +        build_append_int_noprefix(table_data, i, 1); /* type */
>> +        /* length */
>> +        build_append_int_noprefix(table_data, sizeof(AcpiHestNotify), 1);
>> +        build_append_int_noprefix(table_data, 0, 26);
>> +
>> +        /* Error Status Block Length */
>> +        build_append_int_noprefix(table_data,
>> +            cpu_to_le32(GHES_MAX_RAW_DATA_LENGTH), 4);
>> +
cut -----

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
  2017-09-01  9:58       ` gengdongjiu
  (?)
  (?)
@ 2017-09-01 11:51         ` Igor Mammedov
  -1 siblings, 0 replies; 129+ messages in thread
From: Igor Mammedov @ 2017-09-01 11:51 UTC (permalink / raw)
  To: gengdongjiu
  Cc: kvm, mst, will.deacon, qemu-devel, kvmarm, zjzhang, mingo,
	linux-acpi, huangshaoyu, huangdaode, bp, lersek, wuquanming,
	marc.zyngier, edk2-devel, john.garry, linuxarm, qemu-arm,
	jonathan.cameron, zhengqiang10, linux-arm-kernel, devel,
	ard.biesheuvel, linux-kernel, wangzhou1, pbonzini, shiju.jose

On Fri, 1 Sep 2017 17:58:55 +0800
gengdongjiu <gengdongjiu@huawei.com> wrote:

> Hi Igor,
> 
> On 2017/8/29 18:20, Igor Mammedov wrote:
> > On Fri, 18 Aug 2017 22:23:43 +0800
> > Dongjiu Geng <gengdongjiu@huawei.com> wrote:
[...]
> >   
> >> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> >> +                                            BIOSLinker *linker)
> >> +{
> >> +    uint32_t ghes_start = table_data->len;
> >> +    uint32_t address_size, error_status_address_offset;
> >> +    uint32_t read_ack_register_offset, i;
> >> +
> >> +    address_size = sizeof(struct AcpiGenericAddress) -
> >> +        offsetof(struct AcpiGenericAddress, address);  
> > it's confusing name for var,
> > AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> > also, I'm not sure why it's needed at all.  
>  it is because other people have concern about where does the "unsigned 64 bit integer"
>  come from, they are confused about the "unsigned 64 bit integer"
>  so they suggested use sizeof. anyway I will directly use unsigned 64 bit integer.
Maybe properly named macro instead of sizeof(foo) would do the job


[...]
> >> +
> >> +        /* Build error status address*/
> >> +        build_address(table_data, linker, error_status_address_offset + i *
> >> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> >> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);  
> > just do something like this instead of build_address():
> > build_append_gas()
> > bios_linker_loader_add_pointer()  
> Thanks for the suggestion.
> 
> > 
> > also register width 0x40 looks suspicious, where does it come from?
> > While at it do you have a real hardware which has HEST table that you re trying to model after?
> > I'd like to see HEST and other related tables from it.  
> 
> Igor, what is your suspicious point? The register width 0x40 come from our host BIOS record to the System Memory space.

maybe s/0x40/ERROR_STATUS_BLOCK_POINTER_SIZE/

[...]

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-09-01 11:51         ` Igor Mammedov
  0 siblings, 0 replies; 129+ messages in thread
From: Igor Mammedov @ 2017-09-01 11:51 UTC (permalink / raw)
  To: gengdongjiu
  Cc: mst, zhaoshenglong, lersek, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, ard.biesheuvel, mingo, bp, shiju.jose,
	zjzhang, linux-arm-kernel, kvmarm, linux-kernel, linux-acpi,
	devel, john.garry, jonathan.cameron, shameerali.kolothum.thodi,
	huangdaode, wangzhou1, zhengqiang10, wuquanming, huangshaoyu,
	linuxarm

On Fri, 1 Sep 2017 17:58:55 +0800
gengdongjiu <gengdongjiu@huawei.com> wrote:

> Hi Igor,
> 
> On 2017/8/29 18:20, Igor Mammedov wrote:
> > On Fri, 18 Aug 2017 22:23:43 +0800
> > Dongjiu Geng <gengdongjiu@huawei.com> wrote:
[...]
> >   
> >> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> >> +                                            BIOSLinker *linker)
> >> +{
> >> +    uint32_t ghes_start = table_data->len;
> >> +    uint32_t address_size, error_status_address_offset;
> >> +    uint32_t read_ack_register_offset, i;
> >> +
> >> +    address_size = sizeof(struct AcpiGenericAddress) -
> >> +        offsetof(struct AcpiGenericAddress, address);  
> > it's confusing name for var,
> > AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> > also, I'm not sure why it's needed at all.  
>  it is because other people have concern about where does the "unsigned 64 bit integer"
>  come from, they are confused about the "unsigned 64 bit integer"
>  so they suggested use sizeof. anyway I will directly use unsigned 64 bit integer.
Maybe properly named macro instead of sizeof(foo) would do the job


[...]
> >> +
> >> +        /* Build error status address*/
> >> +        build_address(table_data, linker, error_status_address_offset + i *
> >> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> >> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);  
> > just do something like this instead of build_address():
> > build_append_gas()
> > bios_linker_loader_add_pointer()  
> Thanks for the suggestion.
> 
> > 
> > also register width 0x40 looks suspicious, where does it come from?
> > While at it do you have a real hardware which has HEST table that you re trying to model after?
> > I'd like to see HEST and other related tables from it.  
> 
> Igor, what is your suspicious point? The register width 0x40 come from our host BIOS record to the System Memory space.

maybe s/0x40/ERROR_STATUS_BLOCK_POINTER_SIZE/

[...]

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-09-01 11:51         ` Igor Mammedov
  0 siblings, 0 replies; 129+ messages in thread
From: Igor Mammedov @ 2017-09-01 11:51 UTC (permalink / raw)
  To: gengdongjiu
  Cc: mst, zhaoshenglong, lersek, peter.maydell, pbonzini, qemu-devel,
	qemu-arm, kvm, edk2-devel, christoffer.dall, marc.zyngier,
	will.deacon, james.morse, ard.biesheuvel, mingo, bp, shiju.jose,
	zjzhang, linux-arm-kernel, kvmarm, linux-kernel, linux-acpi,
	devel, john.garry, jonathan.cameron, shameerali.kolothum.thodi,
	huangdaode, wangzhou1, zhengqiang10, wuquanming, huangshaoyu,
	linuxarm

On Fri, 1 Sep 2017 17:58:55 +0800
gengdongjiu <gengdongjiu@huawei.com> wrote:

> Hi Igor,
> 
> On 2017/8/29 18:20, Igor Mammedov wrote:
> > On Fri, 18 Aug 2017 22:23:43 +0800
> > Dongjiu Geng <gengdongjiu@huawei.com> wrote:
[...]
> >   
> >> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> >> +                                            BIOSLinker *linker)
> >> +{
> >> +    uint32_t ghes_start = table_data->len;
> >> +    uint32_t address_size, error_status_address_offset;
> >> +    uint32_t read_ack_register_offset, i;
> >> +
> >> +    address_size = sizeof(struct AcpiGenericAddress) -
> >> +        offsetof(struct AcpiGenericAddress, address);  
> > it's confusing name for var,
> > AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> > also, I'm not sure why it's needed at all.  
>  it is because other people have concern about where does the "unsigned 64 bit integer"
>  come from, they are confused about the "unsigned 64 bit integer"
>  so they suggested use sizeof. anyway I will directly use unsigned 64 bit integer.
Maybe properly named macro instead of sizeof(foo) would do the job


[...]
> >> +
> >> +        /* Build error status address*/
> >> +        build_address(table_data, linker, error_status_address_offset + i *
> >> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> >> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);  
> > just do something like this instead of build_address():
> > build_append_gas()
> > bios_linker_loader_add_pointer()  
> Thanks for the suggestion.
> 
> > 
> > also register width 0x40 looks suspicious, where does it come from?
> > While at it do you have a real hardware which has HEST table that you re trying to model after?
> > I'd like to see HEST and other related tables from it.  
> 
> Igor, what is your suspicious point? The register width 0x40 come from our host BIOS record to the System Memory space.

maybe s/0x40/ERROR_STATUS_BLOCK_POINTER_SIZE/

[...]

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [Qemu-devel] [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support
@ 2017-09-01 11:51         ` Igor Mammedov
  0 siblings, 0 replies; 129+ messages in thread
From: Igor Mammedov @ 2017-09-01 11:51 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, 1 Sep 2017 17:58:55 +0800
gengdongjiu <gengdongjiu@huawei.com> wrote:

> Hi Igor,
> 
> On 2017/8/29 18:20, Igor Mammedov wrote:
> > On Fri, 18 Aug 2017 22:23:43 +0800
> > Dongjiu Geng <gengdongjiu@huawei.com> wrote:
[...]
> >   
> >> +void ghes_build_acpi(GArray *table_data, GArray *hardware_error,
> >> +                                            BIOSLinker *linker)
> >> +{
> >> +    uint32_t ghes_start = table_data->len;
> >> +    uint32_t address_size, error_status_address_offset;
> >> +    uint32_t read_ack_register_offset, i;
> >> +
> >> +    address_size = sizeof(struct AcpiGenericAddress) -
> >> +        offsetof(struct AcpiGenericAddress, address);  
> > it's confusing name for var,
> > AcpiGenericAddress::address is fixed unsigned 64 bit integer per spec
> > also, I'm not sure why it's needed at all.  
>  it is because other people have concern about where does the "unsigned 64 bit integer"
>  come from, they are confused about the "unsigned 64 bit integer"
>  so they suggested use sizeof. anyway I will directly use unsigned 64 bit integer.
Maybe properly named macro instead of sizeof(foo) would do the job


[...]
> >> +
> >> +        /* Build error status address*/
> >> +        build_address(table_data, linker, error_status_address_offset + i *
> >> +            sizeof(AcpiGenericHardwareErrorSourceV2), i * address_size,
> >> +            AML_SYSTEM_MEMORY, 0x40, 0, 4 /* QWord access */);  
> > just do something like this instead of build_address():
> > build_append_gas()
> > bios_linker_loader_add_pointer()  
> Thanks for the suggestion.
> 
> > 
> > also register width 0x40 looks suspicious, where does it come from?
> > While at it do you have a real hardware which has HEST table that you re trying to model after?
> > I'd like to see HEST and other related tables from it.  
> 
> Igor, what is your suspicious point? The register width 0x40 come from our host BIOS record to the System Memory space.

maybe s/0x40/ERROR_STATUS_BLOCK_POINTER_SIZE/

[...]

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
  2017-08-18 14:23   ` Dongjiu Geng
                       ` (2 preceding siblings ...)
  (?)
@ 2017-09-05 17:26     ` Peter Maydell
  -1 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:26 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> check if kvm supports guest RAS EXTENSION. if so, set
> corresponding feature bit for vcpu.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  linux-headers/linux/kvm.h | 1 +
>  target/arm/cpu.h          | 3 +++
>  target/arm/kvm64.c        | 8 ++++++++
>  3 files changed, 12 insertions(+)
>
> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
> index 7971a4f..2aa176e 100644
> --- a/linux-headers/linux/kvm.h
> +++ b/linux-headers/linux/kvm.h
> @@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
>  #define KVM_CAP_PPC_SMT_POSSIBLE 147
>  #define KVM_CAP_HYPERV_SYNIC2 148
>  #define KVM_CAP_HYPERV_VP_INDEX 149
> +#define KVM_CAP_ARM_RAS_EXTENSION 150
>
>  #ifdef KVM_CAP_IRQ_ROUTING
>

Hi. Changes to linux-headers need to be done as a patch of their
own created using scripts/update-linux-headers.sh run against a
mainline kernel tree (and with a commit message that quotes the
kernel commit hash used). This ensures that we have a consistent
set of headers that don't diverge from the kernel copy.

> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> index b39d64a..6b0961b 100644
> --- a/target/arm/cpu.h
> +++ b/target/arm/cpu.h
> @@ -611,6 +611,8 @@ struct ARMCPU {
>
>      /* CPU has memory protection unit */
>      bool has_mpu;
> +    /* CPU has ras extension unit */
> +    bool has_ras_extension;
>      /* PMSAv7 MPU number of supported regions */
>      uint32_t pmsav7_dregion;
>
> @@ -1229,6 +1231,7 @@ enum arm_features {
>      ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
>      ARM_FEATURE_PMU, /* has PMU support */
>      ARM_FEATURE_VBAR, /* has cp15 VBAR */
> +    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */

Missing space after '/*' ?

>  };
>
>  static inline int arm_feature(CPUARMState *env, int feature)
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index a16abc8..0781367 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
>          unset_feature(&env->features, ARM_FEATURE_PMU);
>      }
>
> +    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
> +        cpu->has_ras_extension = true;
> +        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
> +    } else {
> +        cpu->has_ras_extension = false;
> +        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
> +    }
> +

Shouldn't we need to also tell the kernel that we actually want
it to expose RAS to the guest? Compare the PMU code in this
function, where we set a kvm_init_features bit to do this.
(This suggests that your ABI for the kernel part of this feature
may not be correct?)

You should also not be calling set_feature() here -- if the
CPU features bit doesn't say "this CPU should have the RAS
extensions" we shouldn't create a CPU with them. Instead
you should set it in kvm_arm_get_host_cpu_features() (again,
compare the PMU code).

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
@ 2017-09-05 17:26     ` Peter Maydell
  0 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:26 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm, lkml - Kernel Mailing List,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, Huangshaoyu,
	wuquanming, Linuxarm, zhengqiang10

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> check if kvm supports guest RAS EXTENSION. if so, set
> corresponding feature bit for vcpu.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  linux-headers/linux/kvm.h | 1 +
>  target/arm/cpu.h          | 3 +++
>  target/arm/kvm64.c        | 8 ++++++++
>  3 files changed, 12 insertions(+)
>
> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
> index 7971a4f..2aa176e 100644
> --- a/linux-headers/linux/kvm.h
> +++ b/linux-headers/linux/kvm.h
> @@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
>  #define KVM_CAP_PPC_SMT_POSSIBLE 147
>  #define KVM_CAP_HYPERV_SYNIC2 148
>  #define KVM_CAP_HYPERV_VP_INDEX 149
> +#define KVM_CAP_ARM_RAS_EXTENSION 150
>
>  #ifdef KVM_CAP_IRQ_ROUTING
>

Hi. Changes to linux-headers need to be done as a patch of their
own created using scripts/update-linux-headers.sh run against a
mainline kernel tree (and with a commit message that quotes the
kernel commit hash used). This ensures that we have a consistent
set of headers that don't diverge from the kernel copy.

> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> index b39d64a..6b0961b 100644
> --- a/target/arm/cpu.h
> +++ b/target/arm/cpu.h
> @@ -611,6 +611,8 @@ struct ARMCPU {
>
>      /* CPU has memory protection unit */
>      bool has_mpu;
> +    /* CPU has ras extension unit */
> +    bool has_ras_extension;
>      /* PMSAv7 MPU number of supported regions */
>      uint32_t pmsav7_dregion;
>
> @@ -1229,6 +1231,7 @@ enum arm_features {
>      ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
>      ARM_FEATURE_PMU, /* has PMU support */
>      ARM_FEATURE_VBAR, /* has cp15 VBAR */
> +    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */

Missing space after '/*' ?

>  };
>
>  static inline int arm_feature(CPUARMState *env, int feature)
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index a16abc8..0781367 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
>          unset_feature(&env->features, ARM_FEATURE_PMU);
>      }
>
> +    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
> +        cpu->has_ras_extension = true;
> +        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
> +    } else {
> +        cpu->has_ras_extension = false;
> +        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
> +    }
> +

Shouldn't we need to also tell the kernel that we actually want
it to expose RAS to the guest? Compare the PMU code in this
function, where we set a kvm_init_features bit to do this.
(This suggests that your ABI for the kernel part of this feature
may not be correct?)

You should also not be calling set_feature() here -- if the
CPU features bit doesn't say "this CPU should have the RAS
extensions" we shouldn't create a CPU with them. Instead
you should set it in kvm_arm_get_host_cpu_features() (again,
compare the PMU code).

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
@ 2017-09-05 17:26     ` Peter Maydell
  0 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:26 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm, l

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> check if kvm supports guest RAS EXTENSION. if so, set
> corresponding feature bit for vcpu.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  linux-headers/linux/kvm.h | 1 +
>  target/arm/cpu.h          | 3 +++
>  target/arm/kvm64.c        | 8 ++++++++
>  3 files changed, 12 insertions(+)
>
> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
> index 7971a4f..2aa176e 100644
> --- a/linux-headers/linux/kvm.h
> +++ b/linux-headers/linux/kvm.h
> @@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
>  #define KVM_CAP_PPC_SMT_POSSIBLE 147
>  #define KVM_CAP_HYPERV_SYNIC2 148
>  #define KVM_CAP_HYPERV_VP_INDEX 149
> +#define KVM_CAP_ARM_RAS_EXTENSION 150
>
>  #ifdef KVM_CAP_IRQ_ROUTING
>

Hi. Changes to linux-headers need to be done as a patch of their
own created using scripts/update-linux-headers.sh run against a
mainline kernel tree (and with a commit message that quotes the
kernel commit hash used). This ensures that we have a consistent
set of headers that don't diverge from the kernel copy.

> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> index b39d64a..6b0961b 100644
> --- a/target/arm/cpu.h
> +++ b/target/arm/cpu.h
> @@ -611,6 +611,8 @@ struct ARMCPU {
>
>      /* CPU has memory protection unit */
>      bool has_mpu;
> +    /* CPU has ras extension unit */
> +    bool has_ras_extension;
>      /* PMSAv7 MPU number of supported regions */
>      uint32_t pmsav7_dregion;
>
> @@ -1229,6 +1231,7 @@ enum arm_features {
>      ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
>      ARM_FEATURE_PMU, /* has PMU support */
>      ARM_FEATURE_VBAR, /* has cp15 VBAR */
> +    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */

Missing space after '/*' ?

>  };
>
>  static inline int arm_feature(CPUARMState *env, int feature)
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index a16abc8..0781367 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
>          unset_feature(&env->features, ARM_FEATURE_PMU);
>      }
>
> +    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
> +        cpu->has_ras_extension = true;
> +        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
> +    } else {
> +        cpu->has_ras_extension = false;
> +        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
> +    }
> +

Shouldn't we need to also tell the kernel that we actually want
it to expose RAS to the guest? Compare the PMU code in this
function, where we set a kvm_init_features bit to do this.
(This suggests that your ABI for the kernel part of this feature
may not be correct?)

You should also not be calling set_feature() here -- if the
CPU features bit doesn't say "this CPU should have the RAS
extensions" we shouldn't create a CPU with them. Instead
you should set it in kvm_arm_get_host_cpu_features() (again,
compare the PMU code).

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
@ 2017-09-05 17:26     ` Peter Maydell
  0 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:26 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm, lkml - Kernel Mailing List,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, Huangshaoyu,
	wuquanming, Linuxarm, zhengqiang10

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> check if kvm supports guest RAS EXTENSION. if so, set
> corresponding feature bit for vcpu.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  linux-headers/linux/kvm.h | 1 +
>  target/arm/cpu.h          | 3 +++
>  target/arm/kvm64.c        | 8 ++++++++
>  3 files changed, 12 insertions(+)
>
> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
> index 7971a4f..2aa176e 100644
> --- a/linux-headers/linux/kvm.h
> +++ b/linux-headers/linux/kvm.h
> @@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
>  #define KVM_CAP_PPC_SMT_POSSIBLE 147
>  #define KVM_CAP_HYPERV_SYNIC2 148
>  #define KVM_CAP_HYPERV_VP_INDEX 149
> +#define KVM_CAP_ARM_RAS_EXTENSION 150
>
>  #ifdef KVM_CAP_IRQ_ROUTING
>

Hi. Changes to linux-headers need to be done as a patch of their
own created using scripts/update-linux-headers.sh run against a
mainline kernel tree (and with a commit message that quotes the
kernel commit hash used). This ensures that we have a consistent
set of headers that don't diverge from the kernel copy.

> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> index b39d64a..6b0961b 100644
> --- a/target/arm/cpu.h
> +++ b/target/arm/cpu.h
> @@ -611,6 +611,8 @@ struct ARMCPU {
>
>      /* CPU has memory protection unit */
>      bool has_mpu;
> +    /* CPU has ras extension unit */
> +    bool has_ras_extension;
>      /* PMSAv7 MPU number of supported regions */
>      uint32_t pmsav7_dregion;
>
> @@ -1229,6 +1231,7 @@ enum arm_features {
>      ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
>      ARM_FEATURE_PMU, /* has PMU support */
>      ARM_FEATURE_VBAR, /* has cp15 VBAR */
> +    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */

Missing space after '/*' ?

>  };
>
>  static inline int arm_feature(CPUARMState *env, int feature)
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index a16abc8..0781367 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
>          unset_feature(&env->features, ARM_FEATURE_PMU);
>      }
>
> +    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
> +        cpu->has_ras_extension = true;
> +        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
> +    } else {
> +        cpu->has_ras_extension = false;
> +        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
> +    }
> +

Shouldn't we need to also tell the kernel that we actually want
it to expose RAS to the guest? Compare the PMU code in this
function, where we set a kvm_init_features bit to do this.
(This suggests that your ABI for the kernel part of this feature
may not be correct?)

You should also not be calling set_feature() here -- if the
CPU features bit doesn't say "this CPU should have the RAS
extensions" we shouldn't create a CPU with them. Instead
you should set it in kvm_arm_get_host_cpu_features() (again,
compare the PMU code).

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
@ 2017-09-05 17:26     ` Peter Maydell
  0 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:26 UTC (permalink / raw)
  To: linux-arm-kernel

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> check if kvm supports guest RAS EXTENSION. if so, set
> corresponding feature bit for vcpu.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> ---
>  linux-headers/linux/kvm.h | 1 +
>  target/arm/cpu.h          | 3 +++
>  target/arm/kvm64.c        | 8 ++++++++
>  3 files changed, 12 insertions(+)
>
> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
> index 7971a4f..2aa176e 100644
> --- a/linux-headers/linux/kvm.h
> +++ b/linux-headers/linux/kvm.h
> @@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
>  #define KVM_CAP_PPC_SMT_POSSIBLE 147
>  #define KVM_CAP_HYPERV_SYNIC2 148
>  #define KVM_CAP_HYPERV_VP_INDEX 149
> +#define KVM_CAP_ARM_RAS_EXTENSION 150
>
>  #ifdef KVM_CAP_IRQ_ROUTING
>

Hi. Changes to linux-headers need to be done as a patch of their
own created using scripts/update-linux-headers.sh run against a
mainline kernel tree (and with a commit message that quotes the
kernel commit hash used). This ensures that we have a consistent
set of headers that don't diverge from the kernel copy.

> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> index b39d64a..6b0961b 100644
> --- a/target/arm/cpu.h
> +++ b/target/arm/cpu.h
> @@ -611,6 +611,8 @@ struct ARMCPU {
>
>      /* CPU has memory protection unit */
>      bool has_mpu;
> +    /* CPU has ras extension unit */
> +    bool has_ras_extension;
>      /* PMSAv7 MPU number of supported regions */
>      uint32_t pmsav7_dregion;
>
> @@ -1229,6 +1231,7 @@ enum arm_features {
>      ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
>      ARM_FEATURE_PMU, /* has PMU support */
>      ARM_FEATURE_VBAR, /* has cp15 VBAR */
> +    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */

Missing space after '/*' ?

>  };
>
>  static inline int arm_feature(CPUARMState *env, int feature)
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index a16abc8..0781367 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
>          unset_feature(&env->features, ARM_FEATURE_PMU);
>      }
>
> +    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
> +        cpu->has_ras_extension = true;
> +        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
> +    } else {
> +        cpu->has_ras_extension = false;
> +        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
> +    }
> +

Shouldn't we need to also tell the kernel that we actually want
it to expose RAS to the guest? Compare the PMU code in this
function, where we set a kvm_init_features bit to do this.
(This suggests that your ABI for the kernel part of this feature
may not be correct?)

You should also not be calling set_feature() here -- if the
CPU features bit doesn't say "this CPU should have the RAS
extensions" we shouldn't create a CPU with them. Instead
you should set it in kvm_arm_get_host_cpu_features() (again,
compare the PMU code).

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 5/6] target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
  2017-08-18 14:23   ` Dongjiu Geng
                       ` (2 preceding siblings ...)
  (?)
@ 2017-09-05 17:46     ` Peter Maydell
  -1 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:46 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> Add SIGBUS signal handler. In this handler, it checks
> the exception type, translates the host VA which is
> delivered by host or KVM to guest PA, then fills this
> PA to CPER, finally injects a Error to guest OS through
> KVM.
>
> Add synchronous external abort injection logic, setup
> spsr_elx, esr_elx, PSTATE, far_elx, elr_elx etc, when
> switch to guest OS, it will jump to the synchronous
> external abort vector table entry.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> Signed-off-by: Quanming Wu <wuquanming@huawei.com>
> ---
>  include/sysemu/kvm.h          |   2 +-
>  linux-headers/asm-arm64/kvm.h |   5 ++
>  target/arm/internals.h        |  13 ++++
>  target/arm/kvm.c              |  34 ++++++++++
>  target/arm/kvm64.c            | 150 ++++++++++++++++++++++++++++++++++++++++++
>  target/arm/kvm_arm.h          |   1 +
>  6 files changed, 204 insertions(+), 1 deletion(-)

Have you tested whether this patchset builds OK on aarch32 ?

> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index 3a458f5..90c1605 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -361,7 +361,7 @@ bool kvm_vcpu_id_is_valid(int vcpu_id);
>  /* Returns VCPU ID to be used on KVM_CREATE_VCPU ioctl() */
>  unsigned long kvm_arch_vcpu_id(CPUState *cpu);
>
> -#ifdef TARGET_I386
> +#if defined(TARGET_I386) || defined(TARGET_AARCH64)
>  #define KVM_HAVE_MCE_INJECTION 1
>  void kvm_arch_on_sigbus_vcpu(CPUState *cpu, int code, void *addr);
>  #endif
> diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h
> index d254700..5909c30 100644
> --- a/linux-headers/asm-arm64/kvm.h
> +++ b/linux-headers/asm-arm64/kvm.h
> @@ -181,6 +181,11 @@ struct kvm_arch_memory_slot {
>  #define KVM_REG_ARM64_SYSREG_OP2_MASK  0x0000000000000007
>  #define KVM_REG_ARM64_SYSREG_OP2_SHIFT 0
>
> +/* AArch64 fault registers */
> +#define KVM_REG_ARM64_FAULT             (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
> +#define KVM_REG_ARM64_FAULT_ESR_EC      (0)
> +#define KVM_REG_ARM64_FAULT_FAR         (1)
> +
>  #define ARM64_SYS_REG_SHIFT_MASK(x,n) \
>         (((x) << KVM_REG_ARM64_SYSREG_ ## n ## _SHIFT) & \
>         KVM_REG_ARM64_SYSREG_ ## n ## _MASK)

Again, linux-headers changes need to go in their own header sync patch.

> diff --git a/target/arm/internals.h b/target/arm/internals.h
> index 1f6efef..fc0ad6d 100644
> --- a/target/arm/internals.h
> +++ b/target/arm/internals.h
> @@ -235,6 +235,19 @@ enum arm_exception_class {
>  #define ARM_EL_ISV_SHIFT 24
>  #define ARM_EL_IL (1 << ARM_EL_IL_SHIFT)
>  #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
> +#define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
> +#define ARM_EL_FSC_TYPE (0x3C)
> +
> +#define FSC_SEA         (0x10)
> +#define FSC_SEA_TTW0    (0x14)
> +#define FSC_SEA_TTW1    (0x15)
> +#define FSC_SEA_TTW2    (0x16)
> +#define FSC_SEA_TTW3    (0x17)
> +#define FSC_SECC        (0x18)
> +#define FSC_SECC_TTW0   (0x1c)
> +#define FSC_SECC_TTW1   (0x1d)
> +#define FSC_SECC_TTW2   (0x1e)
> +#define FSC_SECC_TTW3   (0x1f)
>
>  /* Utility functions for constructing various kinds of syndrome value.
>   * Note that in general we follow the AArch64 syndrome values; in a
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index 7c17f0d..2e1716a 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -129,6 +129,39 @@ void kvm_arm_destroy_scratch_host_vcpu(int *fdarray)
>      }
>  }
>
> +typedef struct HWPoisonPage {
> +    ram_addr_t ram_addr;
> +    QLIST_ENTRY(HWPoisonPage) list;
> +} HWPoisonPage;
> +
> +static QLIST_HEAD(, HWPoisonPage) hwpoison_page_list =
> +    QLIST_HEAD_INITIALIZER(hwpoison_page_list);
> +
> +static void kvm_unpoison_all(void *param)
> +{
> +    HWPoisonPage *page, *next_page;
> +
> +    QLIST_FOREACH_SAFE(page, &hwpoison_page_list, list, next_page) {
> +        QLIST_REMOVE(page, list);
> +        qemu_ram_remap(page->ram_addr, TARGET_PAGE_SIZE);
> +        g_free(page);
> +    }
> +}
> +
> +void kvm_hwpoison_page_add(ram_addr_t ram_addr)
> +{
> +    HWPoisonPage *page;
> +
> +    QLIST_FOREACH(page, &hwpoison_page_list, list) {
> +        if (page->ram_addr == ram_addr) {
> +            return;
> +        }
> +    }
> +    page = g_new(HWPoisonPage, 1);
> +    page->ram_addr = ram_addr;
> +    QLIST_INSERT_HEAD(&hwpoison_page_list, page, list);
> +}

This code has all just been copied-and-pasted from target/i386/kvm.c.
Please instead abstract it out properly into a cpu-independent
source file.

>  static void kvm_arm_host_cpu_class_init(ObjectClass *oc, void *data)
>  {
>      ARMHostCPUClass *ahcc = ARM_HOST_CPU_CLASS(oc);
> @@ -182,6 +215,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
>
>      cap_has_mp_state = kvm_check_extension(s, KVM_CAP_MP_STATE);
>
> +    qemu_register_reset(kvm_unpoison_all, NULL);
>      type_register_static(&host_arm_cpu_type_info);
>
>      return 0;
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index 0781367..d3bdab2 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -27,6 +27,8 @@
>  #include "kvm_arm.h"
>  #include "internals.h"
>  #include "hw/arm/arm.h"
> +#include "hw/acpi/acpi-defs.h"
> +#include "hw/acpi/hest_ghes.h"
>
>  static bool have_guest_debug;
>
> @@ -590,6 +592,79 @@ int kvm_arm_cpreg_level(uint64_t regidx)
>      return KVM_PUT_RUNTIME_STATE;
>  }
>
> +static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
> +{
> +    int i;
> +
> +    for (i = 0; i < cpu->cpreg_array_len; i++) {
> +        uint32_t regidx = kvm_to_cpreg_id(cpu->cpreg_indexes[i]);
> +        const ARMCPRegInfo *ri;
> +        ri = get_arm_cp_reginfo(cpu->cp_regs, regidx);
> +        if (!ri) {
> +            continue;
> +        }
> +
> +        if (ri->type & ARM_CP_NO_RAW) {
> +            continue;
> +        }
> +
> +        if (ri->fieldoffset == fieldoffset) {
> +            cpu->cpreg_values[i] = read_raw_cp_reg(&cpu->env, ri);
> +            return 0;
> +        }
> +    }
> +    return -EINVAL;

What is this ??? You should never need to look up things in
the cpreg arrays by fieldoffset.

The code for handling debug exits (software step, watchpoint, etc)
is probably a good place to look for how to deal with register state.

> +}
> +
> +/* Inject synchronous external abort */
> +static int kvm_inject_arm_sea(CPUState *c)
> +{
> +    ARMCPU *cpu = ARM_CPU(c);
> +    CPUARMState *env = &cpu->env;
> +    unsigned long cpsr = pstate_read(env);
> +    uint32_t esr = 0;
> +    int ret;
> +
> +    c->exception_index = EXCP_DATA_ABORT;
> +    /* Inject the exception to El1 */
> +    env->exception.target_el = 1;
> +    CPUClass *cc = CPU_GET_CLASS(c);
> +
> +    esr |= (EC_DATAABORT << ARM_EL_EC_SHIFT);

We have functions in internals.h for constructing ESR values,
please use them.

> +    /* This exception syndrome includes {I,D}FSC in the bits [5:0]
> +     */
> +    esr |= (env->exception.syndrome & 0x3f);
> +
> +    /* This exception is EL0 or EL1 fault. */
> +    if ((cpsr & 0xf) == PSTATE_MODE_EL0t) {
> +        esr |= (EC_INSNABORT << ARM_EL_EC_SHIFT);
> +    } else {
> +        esr |= (EC_INSNABORT_SAME_EL << ARM_EL_EC_SHIFT);
> +    }
> +
> +    /* In the aarch64, there is only 32-bit instruction*/
> +    esr |= ARM_EL_IL;
> +    env->exception.syndrome = esr;
> +    cc->do_interrupt(c);
> +
> +    /* set ESR_EL1 */
> +    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.esr_el[1]));
> +
> +    if (ret) {
> +        fprintf(stderr, "<%s> failed to set esr_el1\n", __func__);
> +        abort();
> +    }
> +
> +    /* set FAR_EL1 */
> +    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.far_el[1]));
> +    if (ret) {
> +        fprintf(stderr, "<%s> failed to set far_el1\n", __func__);
> +        abort();
> +    }
> +
> +    return 0;
> +}
> +
>  #define AARCH64_CORE_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
>                   KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
>
> @@ -599,6 +674,9 @@ int kvm_arm_cpreg_level(uint64_t regidx)
>  #define AARCH64_SIMD_CTRL_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \
>                   KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
>
> +#define AARCH64_FAULT_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
> +                 KVM_REG_ARM64_FAULT | (x))
> +
>  int kvm_arch_put_registers(CPUState *cs, int level)
>  {
>      struct kvm_one_reg reg;
> @@ -873,6 +951,22 @@ int kvm_arch_get_registers(CPUState *cs)
>      }
>      vfp_set_fpcr(env, fpr);
>
> +    if (is_a64(env)) {
> +        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_ESR_EC);
> +        reg.addr = (uintptr_t)(&env->exception.syndrome);
> +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> +        if (ret) {
> +            return ret;
> +        }
> +
> +        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_FAR);
> +        reg.addr = (uintptr_t)(&env->exception.vaddress);
> +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);

This looks dubious. exception.syndrome and exception.vaddress
are just internal information QEMU uses, not guest visible things.
And only synchronizing them if the CPU happens to be in AArch64
at the point when this function is called is also odd.

> +        if (ret) {
> +            return ret;
> +        }
> +    }
> +
>      if (!write_kvmstate_to_list(cpu)) {
>          return EINVAL;
>      }
> @@ -887,6 +981,62 @@ int kvm_arch_get_registers(CPUState *cs)
>      return ret;
>  }
>
> +static bool is_abort_sea(unsigned long syndrome)
> +{
> +    unsigned long fault_status;

Don't use "unsigned long" when you really mean uint32_t.

> +    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
> +    if ((ec != EC_INSNABORT) && (ec != EC_DATAABORT)) {
> +        return false;
> +    }
> +
> +    fault_status = syndrome & ARM_EL_FSC_TYPE;
> +    switch (fault_status) {
> +    case FSC_SEA:
> +    case FSC_SEA_TTW0:
> +    case FSC_SEA_TTW1:
> +    case FSC_SEA_TTW2:
> +    case FSC_SEA_TTW3:
> +    case FSC_SECC:
> +    case FSC_SECC_TTW0:
> +    case FSC_SECC_TTW1:
> +    case FSC_SECC_TTW2:
> +    case FSC_SECC_TTW3:
> +        return true;
> +    default:
> +        return false;
> +    }
> +}
> +
> +void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
> +{
> +    ram_addr_t ram_addr;
> +    hwaddr paddr;
> +
> +    ARMCPU *cpu = ARM_CPU(c);
> +    CPUARMState *env = &cpu->env;
> +    assert(code == BUS_MCEERR_AR || code == BUS_MCEERR_AO);
> +    if (addr) {
> +        ram_addr = qemu_ram_addr_from_host(addr);
> +        if (ram_addr != RAM_ADDR_INVALID &&
> +            kvm_physical_memory_addr_from_host(c->kvm_state, addr, &paddr)) {
> +            kvm_cpu_synchronize_state(c);
> +            kvm_hwpoison_page_add(ram_addr);
> +            if (is_abort_sea(env->exception.syndrome)) {
> +                ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
> +                kvm_inject_arm_sea(c);
> +            }

This looks a bit odd. There are cases where we hwpoison the page
but don't tell the guest about it? When would we get this kind
of sigbus when the exception syndrome wasn't the right kind ?

Are we guaranteed not to get this kind of signal if we told
the kernel not to expose RAS to the guest ?

> +            return;
> +        }
> +        fprintf(stderr, "Hardware memory error for memory used by "
> +                "QEMU itself instead of guest system!\n");
> +    }
> +
> +    if (code == BUS_MCEERR_AR) {
> +        fprintf(stderr, "Hardware memory error!\n");
> +        exit(1);
> +    }
> +}
> +
>  /* C6.6.29 BRK instruction */
>  static const uint32_t brk_insn = 0xd4200000;
>
> diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
> index 633d088..7cdde97 100644
> --- a/target/arm/kvm_arm.h
> +++ b/target/arm/kvm_arm.h
> @@ -288,4 +288,5 @@ static inline const char *its_class_name(void)
>      }
>  }
>
> +void kvm_hwpoison_page_add(ram_addr_t ram_addr);

Any new globally-visible function prototype in a header should
have a doc-comment formatted documentation comment, please.

>  #endif
> --
> 1.8.3.1

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 5/6] target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
@ 2017-09-05 17:46     ` Peter Maydell
  0 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:46 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm, lkml - Kernel Mailing List,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, Huangshaoyu,
	wuquanming, Linuxarm, zhengqiang10

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> Add SIGBUS signal handler. In this handler, it checks
> the exception type, translates the host VA which is
> delivered by host or KVM to guest PA, then fills this
> PA to CPER, finally injects a Error to guest OS through
> KVM.
>
> Add synchronous external abort injection logic, setup
> spsr_elx, esr_elx, PSTATE, far_elx, elr_elx etc, when
> switch to guest OS, it will jump to the synchronous
> external abort vector table entry.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> Signed-off-by: Quanming Wu <wuquanming@huawei.com>
> ---
>  include/sysemu/kvm.h          |   2 +-
>  linux-headers/asm-arm64/kvm.h |   5 ++
>  target/arm/internals.h        |  13 ++++
>  target/arm/kvm.c              |  34 ++++++++++
>  target/arm/kvm64.c            | 150 ++++++++++++++++++++++++++++++++++++++++++
>  target/arm/kvm_arm.h          |   1 +
>  6 files changed, 204 insertions(+), 1 deletion(-)

Have you tested whether this patchset builds OK on aarch32 ?

> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index 3a458f5..90c1605 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -361,7 +361,7 @@ bool kvm_vcpu_id_is_valid(int vcpu_id);
>  /* Returns VCPU ID to be used on KVM_CREATE_VCPU ioctl() */
>  unsigned long kvm_arch_vcpu_id(CPUState *cpu);
>
> -#ifdef TARGET_I386
> +#if defined(TARGET_I386) || defined(TARGET_AARCH64)
>  #define KVM_HAVE_MCE_INJECTION 1
>  void kvm_arch_on_sigbus_vcpu(CPUState *cpu, int code, void *addr);
>  #endif
> diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h
> index d254700..5909c30 100644
> --- a/linux-headers/asm-arm64/kvm.h
> +++ b/linux-headers/asm-arm64/kvm.h
> @@ -181,6 +181,11 @@ struct kvm_arch_memory_slot {
>  #define KVM_REG_ARM64_SYSREG_OP2_MASK  0x0000000000000007
>  #define KVM_REG_ARM64_SYSREG_OP2_SHIFT 0
>
> +/* AArch64 fault registers */
> +#define KVM_REG_ARM64_FAULT             (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
> +#define KVM_REG_ARM64_FAULT_ESR_EC      (0)
> +#define KVM_REG_ARM64_FAULT_FAR         (1)
> +
>  #define ARM64_SYS_REG_SHIFT_MASK(x,n) \
>         (((x) << KVM_REG_ARM64_SYSREG_ ## n ## _SHIFT) & \
>         KVM_REG_ARM64_SYSREG_ ## n ## _MASK)

Again, linux-headers changes need to go in their own header sync patch.

> diff --git a/target/arm/internals.h b/target/arm/internals.h
> index 1f6efef..fc0ad6d 100644
> --- a/target/arm/internals.h
> +++ b/target/arm/internals.h
> @@ -235,6 +235,19 @@ enum arm_exception_class {
>  #define ARM_EL_ISV_SHIFT 24
>  #define ARM_EL_IL (1 << ARM_EL_IL_SHIFT)
>  #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
> +#define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
> +#define ARM_EL_FSC_TYPE (0x3C)
> +
> +#define FSC_SEA         (0x10)
> +#define FSC_SEA_TTW0    (0x14)
> +#define FSC_SEA_TTW1    (0x15)
> +#define FSC_SEA_TTW2    (0x16)
> +#define FSC_SEA_TTW3    (0x17)
> +#define FSC_SECC        (0x18)
> +#define FSC_SECC_TTW0   (0x1c)
> +#define FSC_SECC_TTW1   (0x1d)
> +#define FSC_SECC_TTW2   (0x1e)
> +#define FSC_SECC_TTW3   (0x1f)
>
>  /* Utility functions for constructing various kinds of syndrome value.
>   * Note that in general we follow the AArch64 syndrome values; in a
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index 7c17f0d..2e1716a 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -129,6 +129,39 @@ void kvm_arm_destroy_scratch_host_vcpu(int *fdarray)
>      }
>  }
>
> +typedef struct HWPoisonPage {
> +    ram_addr_t ram_addr;
> +    QLIST_ENTRY(HWPoisonPage) list;
> +} HWPoisonPage;
> +
> +static QLIST_HEAD(, HWPoisonPage) hwpoison_page_list =
> +    QLIST_HEAD_INITIALIZER(hwpoison_page_list);
> +
> +static void kvm_unpoison_all(void *param)
> +{
> +    HWPoisonPage *page, *next_page;
> +
> +    QLIST_FOREACH_SAFE(page, &hwpoison_page_list, list, next_page) {
> +        QLIST_REMOVE(page, list);
> +        qemu_ram_remap(page->ram_addr, TARGET_PAGE_SIZE);
> +        g_free(page);
> +    }
> +}
> +
> +void kvm_hwpoison_page_add(ram_addr_t ram_addr)
> +{
> +    HWPoisonPage *page;
> +
> +    QLIST_FOREACH(page, &hwpoison_page_list, list) {
> +        if (page->ram_addr == ram_addr) {
> +            return;
> +        }
> +    }
> +    page = g_new(HWPoisonPage, 1);
> +    page->ram_addr = ram_addr;
> +    QLIST_INSERT_HEAD(&hwpoison_page_list, page, list);
> +}

This code has all just been copied-and-pasted from target/i386/kvm.c.
Please instead abstract it out properly into a cpu-independent
source file.

>  static void kvm_arm_host_cpu_class_init(ObjectClass *oc, void *data)
>  {
>      ARMHostCPUClass *ahcc = ARM_HOST_CPU_CLASS(oc);
> @@ -182,6 +215,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
>
>      cap_has_mp_state = kvm_check_extension(s, KVM_CAP_MP_STATE);
>
> +    qemu_register_reset(kvm_unpoison_all, NULL);
>      type_register_static(&host_arm_cpu_type_info);
>
>      return 0;
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index 0781367..d3bdab2 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -27,6 +27,8 @@
>  #include "kvm_arm.h"
>  #include "internals.h"
>  #include "hw/arm/arm.h"
> +#include "hw/acpi/acpi-defs.h"
> +#include "hw/acpi/hest_ghes.h"
>
>  static bool have_guest_debug;
>
> @@ -590,6 +592,79 @@ int kvm_arm_cpreg_level(uint64_t regidx)
>      return KVM_PUT_RUNTIME_STATE;
>  }
>
> +static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
> +{
> +    int i;
> +
> +    for (i = 0; i < cpu->cpreg_array_len; i++) {
> +        uint32_t regidx = kvm_to_cpreg_id(cpu->cpreg_indexes[i]);
> +        const ARMCPRegInfo *ri;
> +        ri = get_arm_cp_reginfo(cpu->cp_regs, regidx);
> +        if (!ri) {
> +            continue;
> +        }
> +
> +        if (ri->type & ARM_CP_NO_RAW) {
> +            continue;
> +        }
> +
> +        if (ri->fieldoffset == fieldoffset) {
> +            cpu->cpreg_values[i] = read_raw_cp_reg(&cpu->env, ri);
> +            return 0;
> +        }
> +    }
> +    return -EINVAL;

What is this ??? You should never need to look up things in
the cpreg arrays by fieldoffset.

The code for handling debug exits (software step, watchpoint, etc)
is probably a good place to look for how to deal with register state.

> +}
> +
> +/* Inject synchronous external abort */
> +static int kvm_inject_arm_sea(CPUState *c)
> +{
> +    ARMCPU *cpu = ARM_CPU(c);
> +    CPUARMState *env = &cpu->env;
> +    unsigned long cpsr = pstate_read(env);
> +    uint32_t esr = 0;
> +    int ret;
> +
> +    c->exception_index = EXCP_DATA_ABORT;
> +    /* Inject the exception to El1 */
> +    env->exception.target_el = 1;
> +    CPUClass *cc = CPU_GET_CLASS(c);
> +
> +    esr |= (EC_DATAABORT << ARM_EL_EC_SHIFT);

We have functions in internals.h for constructing ESR values,
please use them.

> +    /* This exception syndrome includes {I,D}FSC in the bits [5:0]
> +     */
> +    esr |= (env->exception.syndrome & 0x3f);
> +
> +    /* This exception is EL0 or EL1 fault. */
> +    if ((cpsr & 0xf) == PSTATE_MODE_EL0t) {
> +        esr |= (EC_INSNABORT << ARM_EL_EC_SHIFT);
> +    } else {
> +        esr |= (EC_INSNABORT_SAME_EL << ARM_EL_EC_SHIFT);
> +    }
> +
> +    /* In the aarch64, there is only 32-bit instruction*/
> +    esr |= ARM_EL_IL;
> +    env->exception.syndrome = esr;
> +    cc->do_interrupt(c);
> +
> +    /* set ESR_EL1 */
> +    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.esr_el[1]));
> +
> +    if (ret) {
> +        fprintf(stderr, "<%s> failed to set esr_el1\n", __func__);
> +        abort();
> +    }
> +
> +    /* set FAR_EL1 */
> +    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.far_el[1]));
> +    if (ret) {
> +        fprintf(stderr, "<%s> failed to set far_el1\n", __func__);
> +        abort();
> +    }
> +
> +    return 0;
> +}
> +
>  #define AARCH64_CORE_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
>                   KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
>
> @@ -599,6 +674,9 @@ int kvm_arm_cpreg_level(uint64_t regidx)
>  #define AARCH64_SIMD_CTRL_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \
>                   KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
>
> +#define AARCH64_FAULT_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
> +                 KVM_REG_ARM64_FAULT | (x))
> +
>  int kvm_arch_put_registers(CPUState *cs, int level)
>  {
>      struct kvm_one_reg reg;
> @@ -873,6 +951,22 @@ int kvm_arch_get_registers(CPUState *cs)
>      }
>      vfp_set_fpcr(env, fpr);
>
> +    if (is_a64(env)) {
> +        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_ESR_EC);
> +        reg.addr = (uintptr_t)(&env->exception.syndrome);
> +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> +        if (ret) {
> +            return ret;
> +        }
> +
> +        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_FAR);
> +        reg.addr = (uintptr_t)(&env->exception.vaddress);
> +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);

This looks dubious. exception.syndrome and exception.vaddress
are just internal information QEMU uses, not guest visible things.
And only synchronizing them if the CPU happens to be in AArch64
at the point when this function is called is also odd.

> +        if (ret) {
> +            return ret;
> +        }
> +    }
> +
>      if (!write_kvmstate_to_list(cpu)) {
>          return EINVAL;
>      }
> @@ -887,6 +981,62 @@ int kvm_arch_get_registers(CPUState *cs)
>      return ret;
>  }
>
> +static bool is_abort_sea(unsigned long syndrome)
> +{
> +    unsigned long fault_status;

Don't use "unsigned long" when you really mean uint32_t.

> +    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
> +    if ((ec != EC_INSNABORT) && (ec != EC_DATAABORT)) {
> +        return false;
> +    }
> +
> +    fault_status = syndrome & ARM_EL_FSC_TYPE;
> +    switch (fault_status) {
> +    case FSC_SEA:
> +    case FSC_SEA_TTW0:
> +    case FSC_SEA_TTW1:
> +    case FSC_SEA_TTW2:
> +    case FSC_SEA_TTW3:
> +    case FSC_SECC:
> +    case FSC_SECC_TTW0:
> +    case FSC_SECC_TTW1:
> +    case FSC_SECC_TTW2:
> +    case FSC_SECC_TTW3:
> +        return true;
> +    default:
> +        return false;
> +    }
> +}
> +
> +void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
> +{
> +    ram_addr_t ram_addr;
> +    hwaddr paddr;
> +
> +    ARMCPU *cpu = ARM_CPU(c);
> +    CPUARMState *env = &cpu->env;
> +    assert(code == BUS_MCEERR_AR || code == BUS_MCEERR_AO);
> +    if (addr) {
> +        ram_addr = qemu_ram_addr_from_host(addr);
> +        if (ram_addr != RAM_ADDR_INVALID &&
> +            kvm_physical_memory_addr_from_host(c->kvm_state, addr, &paddr)) {
> +            kvm_cpu_synchronize_state(c);
> +            kvm_hwpoison_page_add(ram_addr);
> +            if (is_abort_sea(env->exception.syndrome)) {
> +                ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
> +                kvm_inject_arm_sea(c);
> +            }

This looks a bit odd. There are cases where we hwpoison the page
but don't tell the guest about it? When would we get this kind
of sigbus when the exception syndrome wasn't the right kind ?

Are we guaranteed not to get this kind of signal if we told
the kernel not to expose RAS to the guest ?

> +            return;
> +        }
> +        fprintf(stderr, "Hardware memory error for memory used by "
> +                "QEMU itself instead of guest system!\n");
> +    }
> +
> +    if (code == BUS_MCEERR_AR) {
> +        fprintf(stderr, "Hardware memory error!\n");
> +        exit(1);
> +    }
> +}
> +
>  /* C6.6.29 BRK instruction */
>  static const uint32_t brk_insn = 0xd4200000;
>
> diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
> index 633d088..7cdde97 100644
> --- a/target/arm/kvm_arm.h
> +++ b/target/arm/kvm_arm.h
> @@ -288,4 +288,5 @@ static inline const char *its_class_name(void)
>      }
>  }
>
> +void kvm_hwpoison_page_add(ram_addr_t ram_addr);

Any new globally-visible function prototype in a header should
have a doc-comment formatted documentation comment, please.

>  #endif
> --
> 1.8.3.1

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 5/6] target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
@ 2017-09-05 17:46     ` Peter Maydell
  0 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:46 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm, l

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> Add SIGBUS signal handler. In this handler, it checks
> the exception type, translates the host VA which is
> delivered by host or KVM to guest PA, then fills this
> PA to CPER, finally injects a Error to guest OS through
> KVM.
>
> Add synchronous external abort injection logic, setup
> spsr_elx, esr_elx, PSTATE, far_elx, elr_elx etc, when
> switch to guest OS, it will jump to the synchronous
> external abort vector table entry.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> Signed-off-by: Quanming Wu <wuquanming@huawei.com>
> ---
>  include/sysemu/kvm.h          |   2 +-
>  linux-headers/asm-arm64/kvm.h |   5 ++
>  target/arm/internals.h        |  13 ++++
>  target/arm/kvm.c              |  34 ++++++++++
>  target/arm/kvm64.c            | 150 ++++++++++++++++++++++++++++++++++++++++++
>  target/arm/kvm_arm.h          |   1 +
>  6 files changed, 204 insertions(+), 1 deletion(-)

Have you tested whether this patchset builds OK on aarch32 ?

> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index 3a458f5..90c1605 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -361,7 +361,7 @@ bool kvm_vcpu_id_is_valid(int vcpu_id);
>  /* Returns VCPU ID to be used on KVM_CREATE_VCPU ioctl() */
>  unsigned long kvm_arch_vcpu_id(CPUState *cpu);
>
> -#ifdef TARGET_I386
> +#if defined(TARGET_I386) || defined(TARGET_AARCH64)
>  #define KVM_HAVE_MCE_INJECTION 1
>  void kvm_arch_on_sigbus_vcpu(CPUState *cpu, int code, void *addr);
>  #endif
> diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h
> index d254700..5909c30 100644
> --- a/linux-headers/asm-arm64/kvm.h
> +++ b/linux-headers/asm-arm64/kvm.h
> @@ -181,6 +181,11 @@ struct kvm_arch_memory_slot {
>  #define KVM_REG_ARM64_SYSREG_OP2_MASK  0x0000000000000007
>  #define KVM_REG_ARM64_SYSREG_OP2_SHIFT 0
>
> +/* AArch64 fault registers */
> +#define KVM_REG_ARM64_FAULT             (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
> +#define KVM_REG_ARM64_FAULT_ESR_EC      (0)
> +#define KVM_REG_ARM64_FAULT_FAR         (1)
> +
>  #define ARM64_SYS_REG_SHIFT_MASK(x,n) \
>         (((x) << KVM_REG_ARM64_SYSREG_ ## n ## _SHIFT) & \
>         KVM_REG_ARM64_SYSREG_ ## n ## _MASK)

Again, linux-headers changes need to go in their own header sync patch.

> diff --git a/target/arm/internals.h b/target/arm/internals.h
> index 1f6efef..fc0ad6d 100644
> --- a/target/arm/internals.h
> +++ b/target/arm/internals.h
> @@ -235,6 +235,19 @@ enum arm_exception_class {
>  #define ARM_EL_ISV_SHIFT 24
>  #define ARM_EL_IL (1 << ARM_EL_IL_SHIFT)
>  #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
> +#define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
> +#define ARM_EL_FSC_TYPE (0x3C)
> +
> +#define FSC_SEA         (0x10)
> +#define FSC_SEA_TTW0    (0x14)
> +#define FSC_SEA_TTW1    (0x15)
> +#define FSC_SEA_TTW2    (0x16)
> +#define FSC_SEA_TTW3    (0x17)
> +#define FSC_SECC        (0x18)
> +#define FSC_SECC_TTW0   (0x1c)
> +#define FSC_SECC_TTW1   (0x1d)
> +#define FSC_SECC_TTW2   (0x1e)
> +#define FSC_SECC_TTW3   (0x1f)
>
>  /* Utility functions for constructing various kinds of syndrome value.
>   * Note that in general we follow the AArch64 syndrome values; in a
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index 7c17f0d..2e1716a 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -129,6 +129,39 @@ void kvm_arm_destroy_scratch_host_vcpu(int *fdarray)
>      }
>  }
>
> +typedef struct HWPoisonPage {
> +    ram_addr_t ram_addr;
> +    QLIST_ENTRY(HWPoisonPage) list;
> +} HWPoisonPage;
> +
> +static QLIST_HEAD(, HWPoisonPage) hwpoison_page_list =
> +    QLIST_HEAD_INITIALIZER(hwpoison_page_list);
> +
> +static void kvm_unpoison_all(void *param)
> +{
> +    HWPoisonPage *page, *next_page;
> +
> +    QLIST_FOREACH_SAFE(page, &hwpoison_page_list, list, next_page) {
> +        QLIST_REMOVE(page, list);
> +        qemu_ram_remap(page->ram_addr, TARGET_PAGE_SIZE);
> +        g_free(page);
> +    }
> +}
> +
> +void kvm_hwpoison_page_add(ram_addr_t ram_addr)
> +{
> +    HWPoisonPage *page;
> +
> +    QLIST_FOREACH(page, &hwpoison_page_list, list) {
> +        if (page->ram_addr == ram_addr) {
> +            return;
> +        }
> +    }
> +    page = g_new(HWPoisonPage, 1);
> +    page->ram_addr = ram_addr;
> +    QLIST_INSERT_HEAD(&hwpoison_page_list, page, list);
> +}

This code has all just been copied-and-pasted from target/i386/kvm.c.
Please instead abstract it out properly into a cpu-independent
source file.

>  static void kvm_arm_host_cpu_class_init(ObjectClass *oc, void *data)
>  {
>      ARMHostCPUClass *ahcc = ARM_HOST_CPU_CLASS(oc);
> @@ -182,6 +215,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
>
>      cap_has_mp_state = kvm_check_extension(s, KVM_CAP_MP_STATE);
>
> +    qemu_register_reset(kvm_unpoison_all, NULL);
>      type_register_static(&host_arm_cpu_type_info);
>
>      return 0;
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index 0781367..d3bdab2 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -27,6 +27,8 @@
>  #include "kvm_arm.h"
>  #include "internals.h"
>  #include "hw/arm/arm.h"
> +#include "hw/acpi/acpi-defs.h"
> +#include "hw/acpi/hest_ghes.h"
>
>  static bool have_guest_debug;
>
> @@ -590,6 +592,79 @@ int kvm_arm_cpreg_level(uint64_t regidx)
>      return KVM_PUT_RUNTIME_STATE;
>  }
>
> +static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
> +{
> +    int i;
> +
> +    for (i = 0; i < cpu->cpreg_array_len; i++) {
> +        uint32_t regidx = kvm_to_cpreg_id(cpu->cpreg_indexes[i]);
> +        const ARMCPRegInfo *ri;
> +        ri = get_arm_cp_reginfo(cpu->cp_regs, regidx);
> +        if (!ri) {
> +            continue;
> +        }
> +
> +        if (ri->type & ARM_CP_NO_RAW) {
> +            continue;
> +        }
> +
> +        if (ri->fieldoffset == fieldoffset) {
> +            cpu->cpreg_values[i] = read_raw_cp_reg(&cpu->env, ri);
> +            return 0;
> +        }
> +    }
> +    return -EINVAL;

What is this ??? You should never need to look up things in
the cpreg arrays by fieldoffset.

The code for handling debug exits (software step, watchpoint, etc)
is probably a good place to look for how to deal with register state.

> +}
> +
> +/* Inject synchronous external abort */
> +static int kvm_inject_arm_sea(CPUState *c)
> +{
> +    ARMCPU *cpu = ARM_CPU(c);
> +    CPUARMState *env = &cpu->env;
> +    unsigned long cpsr = pstate_read(env);
> +    uint32_t esr = 0;
> +    int ret;
> +
> +    c->exception_index = EXCP_DATA_ABORT;
> +    /* Inject the exception to El1 */
> +    env->exception.target_el = 1;
> +    CPUClass *cc = CPU_GET_CLASS(c);
> +
> +    esr |= (EC_DATAABORT << ARM_EL_EC_SHIFT);

We have functions in internals.h for constructing ESR values,
please use them.

> +    /* This exception syndrome includes {I,D}FSC in the bits [5:0]
> +     */
> +    esr |= (env->exception.syndrome & 0x3f);
> +
> +    /* This exception is EL0 or EL1 fault. */
> +    if ((cpsr & 0xf) == PSTATE_MODE_EL0t) {
> +        esr |= (EC_INSNABORT << ARM_EL_EC_SHIFT);
> +    } else {
> +        esr |= (EC_INSNABORT_SAME_EL << ARM_EL_EC_SHIFT);
> +    }
> +
> +    /* In the aarch64, there is only 32-bit instruction*/
> +    esr |= ARM_EL_IL;
> +    env->exception.syndrome = esr;
> +    cc->do_interrupt(c);
> +
> +    /* set ESR_EL1 */
> +    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.esr_el[1]));
> +
> +    if (ret) {
> +        fprintf(stderr, "<%s> failed to set esr_el1\n", __func__);
> +        abort();
> +    }
> +
> +    /* set FAR_EL1 */
> +    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.far_el[1]));
> +    if (ret) {
> +        fprintf(stderr, "<%s> failed to set far_el1\n", __func__);
> +        abort();
> +    }
> +
> +    return 0;
> +}
> +
>  #define AARCH64_CORE_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
>                   KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
>
> @@ -599,6 +674,9 @@ int kvm_arm_cpreg_level(uint64_t regidx)
>  #define AARCH64_SIMD_CTRL_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \
>                   KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
>
> +#define AARCH64_FAULT_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
> +                 KVM_REG_ARM64_FAULT | (x))
> +
>  int kvm_arch_put_registers(CPUState *cs, int level)
>  {
>      struct kvm_one_reg reg;
> @@ -873,6 +951,22 @@ int kvm_arch_get_registers(CPUState *cs)
>      }
>      vfp_set_fpcr(env, fpr);
>
> +    if (is_a64(env)) {
> +        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_ESR_EC);
> +        reg.addr = (uintptr_t)(&env->exception.syndrome);
> +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> +        if (ret) {
> +            return ret;
> +        }
> +
> +        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_FAR);
> +        reg.addr = (uintptr_t)(&env->exception.vaddress);
> +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);

This looks dubious. exception.syndrome and exception.vaddress
are just internal information QEMU uses, not guest visible things.
And only synchronizing them if the CPU happens to be in AArch64
at the point when this function is called is also odd.

> +        if (ret) {
> +            return ret;
> +        }
> +    }
> +
>      if (!write_kvmstate_to_list(cpu)) {
>          return EINVAL;
>      }
> @@ -887,6 +981,62 @@ int kvm_arch_get_registers(CPUState *cs)
>      return ret;
>  }
>
> +static bool is_abort_sea(unsigned long syndrome)
> +{
> +    unsigned long fault_status;

Don't use "unsigned long" when you really mean uint32_t.

> +    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
> +    if ((ec != EC_INSNABORT) && (ec != EC_DATAABORT)) {
> +        return false;
> +    }
> +
> +    fault_status = syndrome & ARM_EL_FSC_TYPE;
> +    switch (fault_status) {
> +    case FSC_SEA:
> +    case FSC_SEA_TTW0:
> +    case FSC_SEA_TTW1:
> +    case FSC_SEA_TTW2:
> +    case FSC_SEA_TTW3:
> +    case FSC_SECC:
> +    case FSC_SECC_TTW0:
> +    case FSC_SECC_TTW1:
> +    case FSC_SECC_TTW2:
> +    case FSC_SECC_TTW3:
> +        return true;
> +    default:
> +        return false;
> +    }
> +}
> +
> +void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
> +{
> +    ram_addr_t ram_addr;
> +    hwaddr paddr;
> +
> +    ARMCPU *cpu = ARM_CPU(c);
> +    CPUARMState *env = &cpu->env;
> +    assert(code == BUS_MCEERR_AR || code == BUS_MCEERR_AO);
> +    if (addr) {
> +        ram_addr = qemu_ram_addr_from_host(addr);
> +        if (ram_addr != RAM_ADDR_INVALID &&
> +            kvm_physical_memory_addr_from_host(c->kvm_state, addr, &paddr)) {
> +            kvm_cpu_synchronize_state(c);
> +            kvm_hwpoison_page_add(ram_addr);
> +            if (is_abort_sea(env->exception.syndrome)) {
> +                ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
> +                kvm_inject_arm_sea(c);
> +            }

This looks a bit odd. There are cases where we hwpoison the page
but don't tell the guest about it? When would we get this kind
of sigbus when the exception syndrome wasn't the right kind ?

Are we guaranteed not to get this kind of signal if we told
the kernel not to expose RAS to the guest ?

> +            return;
> +        }
> +        fprintf(stderr, "Hardware memory error for memory used by "
> +                "QEMU itself instead of guest system!\n");
> +    }
> +
> +    if (code == BUS_MCEERR_AR) {
> +        fprintf(stderr, "Hardware memory error!\n");
> +        exit(1);
> +    }
> +}
> +
>  /* C6.6.29 BRK instruction */
>  static const uint32_t brk_insn = 0xd4200000;
>
> diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
> index 633d088..7cdde97 100644
> --- a/target/arm/kvm_arm.h
> +++ b/target/arm/kvm_arm.h
> @@ -288,4 +288,5 @@ static inline const char *its_class_name(void)
>      }
>  }
>
> +void kvm_hwpoison_page_add(ram_addr_t ram_addr);

Any new globally-visible function prototype in a header should
have a doc-comment formatted documentation comment, please.

>  #endif
> --
> 1.8.3.1

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 5/6] target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
@ 2017-09-05 17:46     ` Peter Maydell
  0 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:46 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm, lkml - Kernel Mailing List,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, Huangshaoyu,
	wuquanming, Linuxarm, zhengqiang10

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> Add SIGBUS signal handler. In this handler, it checks
> the exception type, translates the host VA which is
> delivered by host or KVM to guest PA, then fills this
> PA to CPER, finally injects a Error to guest OS through
> KVM.
>
> Add synchronous external abort injection logic, setup
> spsr_elx, esr_elx, PSTATE, far_elx, elr_elx etc, when
> switch to guest OS, it will jump to the synchronous
> external abort vector table entry.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> Signed-off-by: Quanming Wu <wuquanming@huawei.com>
> ---
>  include/sysemu/kvm.h          |   2 +-
>  linux-headers/asm-arm64/kvm.h |   5 ++
>  target/arm/internals.h        |  13 ++++
>  target/arm/kvm.c              |  34 ++++++++++
>  target/arm/kvm64.c            | 150 ++++++++++++++++++++++++++++++++++++++++++
>  target/arm/kvm_arm.h          |   1 +
>  6 files changed, 204 insertions(+), 1 deletion(-)

Have you tested whether this patchset builds OK on aarch32 ?

> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index 3a458f5..90c1605 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -361,7 +361,7 @@ bool kvm_vcpu_id_is_valid(int vcpu_id);
>  /* Returns VCPU ID to be used on KVM_CREATE_VCPU ioctl() */
>  unsigned long kvm_arch_vcpu_id(CPUState *cpu);
>
> -#ifdef TARGET_I386
> +#if defined(TARGET_I386) || defined(TARGET_AARCH64)
>  #define KVM_HAVE_MCE_INJECTION 1
>  void kvm_arch_on_sigbus_vcpu(CPUState *cpu, int code, void *addr);
>  #endif
> diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h
> index d254700..5909c30 100644
> --- a/linux-headers/asm-arm64/kvm.h
> +++ b/linux-headers/asm-arm64/kvm.h
> @@ -181,6 +181,11 @@ struct kvm_arch_memory_slot {
>  #define KVM_REG_ARM64_SYSREG_OP2_MASK  0x0000000000000007
>  #define KVM_REG_ARM64_SYSREG_OP2_SHIFT 0
>
> +/* AArch64 fault registers */
> +#define KVM_REG_ARM64_FAULT             (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
> +#define KVM_REG_ARM64_FAULT_ESR_EC      (0)
> +#define KVM_REG_ARM64_FAULT_FAR         (1)
> +
>  #define ARM64_SYS_REG_SHIFT_MASK(x,n) \
>         (((x) << KVM_REG_ARM64_SYSREG_ ## n ## _SHIFT) & \
>         KVM_REG_ARM64_SYSREG_ ## n ## _MASK)

Again, linux-headers changes need to go in their own header sync patch.

> diff --git a/target/arm/internals.h b/target/arm/internals.h
> index 1f6efef..fc0ad6d 100644
> --- a/target/arm/internals.h
> +++ b/target/arm/internals.h
> @@ -235,6 +235,19 @@ enum arm_exception_class {
>  #define ARM_EL_ISV_SHIFT 24
>  #define ARM_EL_IL (1 << ARM_EL_IL_SHIFT)
>  #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
> +#define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
> +#define ARM_EL_FSC_TYPE (0x3C)
> +
> +#define FSC_SEA         (0x10)
> +#define FSC_SEA_TTW0    (0x14)
> +#define FSC_SEA_TTW1    (0x15)
> +#define FSC_SEA_TTW2    (0x16)
> +#define FSC_SEA_TTW3    (0x17)
> +#define FSC_SECC        (0x18)
> +#define FSC_SECC_TTW0   (0x1c)
> +#define FSC_SECC_TTW1   (0x1d)
> +#define FSC_SECC_TTW2   (0x1e)
> +#define FSC_SECC_TTW3   (0x1f)
>
>  /* Utility functions for constructing various kinds of syndrome value.
>   * Note that in general we follow the AArch64 syndrome values; in a
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index 7c17f0d..2e1716a 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -129,6 +129,39 @@ void kvm_arm_destroy_scratch_host_vcpu(int *fdarray)
>      }
>  }
>
> +typedef struct HWPoisonPage {
> +    ram_addr_t ram_addr;
> +    QLIST_ENTRY(HWPoisonPage) list;
> +} HWPoisonPage;
> +
> +static QLIST_HEAD(, HWPoisonPage) hwpoison_page_list =
> +    QLIST_HEAD_INITIALIZER(hwpoison_page_list);
> +
> +static void kvm_unpoison_all(void *param)
> +{
> +    HWPoisonPage *page, *next_page;
> +
> +    QLIST_FOREACH_SAFE(page, &hwpoison_page_list, list, next_page) {
> +        QLIST_REMOVE(page, list);
> +        qemu_ram_remap(page->ram_addr, TARGET_PAGE_SIZE);
> +        g_free(page);
> +    }
> +}
> +
> +void kvm_hwpoison_page_add(ram_addr_t ram_addr)
> +{
> +    HWPoisonPage *page;
> +
> +    QLIST_FOREACH(page, &hwpoison_page_list, list) {
> +        if (page->ram_addr == ram_addr) {
> +            return;
> +        }
> +    }
> +    page = g_new(HWPoisonPage, 1);
> +    page->ram_addr = ram_addr;
> +    QLIST_INSERT_HEAD(&hwpoison_page_list, page, list);
> +}

This code has all just been copied-and-pasted from target/i386/kvm.c.
Please instead abstract it out properly into a cpu-independent
source file.

>  static void kvm_arm_host_cpu_class_init(ObjectClass *oc, void *data)
>  {
>      ARMHostCPUClass *ahcc = ARM_HOST_CPU_CLASS(oc);
> @@ -182,6 +215,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
>
>      cap_has_mp_state = kvm_check_extension(s, KVM_CAP_MP_STATE);
>
> +    qemu_register_reset(kvm_unpoison_all, NULL);
>      type_register_static(&host_arm_cpu_type_info);
>
>      return 0;
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index 0781367..d3bdab2 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -27,6 +27,8 @@
>  #include "kvm_arm.h"
>  #include "internals.h"
>  #include "hw/arm/arm.h"
> +#include "hw/acpi/acpi-defs.h"
> +#include "hw/acpi/hest_ghes.h"
>
>  static bool have_guest_debug;
>
> @@ -590,6 +592,79 @@ int kvm_arm_cpreg_level(uint64_t regidx)
>      return KVM_PUT_RUNTIME_STATE;
>  }
>
> +static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
> +{
> +    int i;
> +
> +    for (i = 0; i < cpu->cpreg_array_len; i++) {
> +        uint32_t regidx = kvm_to_cpreg_id(cpu->cpreg_indexes[i]);
> +        const ARMCPRegInfo *ri;
> +        ri = get_arm_cp_reginfo(cpu->cp_regs, regidx);
> +        if (!ri) {
> +            continue;
> +        }
> +
> +        if (ri->type & ARM_CP_NO_RAW) {
> +            continue;
> +        }
> +
> +        if (ri->fieldoffset == fieldoffset) {
> +            cpu->cpreg_values[i] = read_raw_cp_reg(&cpu->env, ri);
> +            return 0;
> +        }
> +    }
> +    return -EINVAL;

What is this ??? You should never need to look up things in
the cpreg arrays by fieldoffset.

The code for handling debug exits (software step, watchpoint, etc)
is probably a good place to look for how to deal with register state.

> +}
> +
> +/* Inject synchronous external abort */
> +static int kvm_inject_arm_sea(CPUState *c)
> +{
> +    ARMCPU *cpu = ARM_CPU(c);
> +    CPUARMState *env = &cpu->env;
> +    unsigned long cpsr = pstate_read(env);
> +    uint32_t esr = 0;
> +    int ret;
> +
> +    c->exception_index = EXCP_DATA_ABORT;
> +    /* Inject the exception to El1 */
> +    env->exception.target_el = 1;
> +    CPUClass *cc = CPU_GET_CLASS(c);
> +
> +    esr |= (EC_DATAABORT << ARM_EL_EC_SHIFT);

We have functions in internals.h for constructing ESR values,
please use them.

> +    /* This exception syndrome includes {I,D}FSC in the bits [5:0]
> +     */
> +    esr |= (env->exception.syndrome & 0x3f);
> +
> +    /* This exception is EL0 or EL1 fault. */
> +    if ((cpsr & 0xf) == PSTATE_MODE_EL0t) {
> +        esr |= (EC_INSNABORT << ARM_EL_EC_SHIFT);
> +    } else {
> +        esr |= (EC_INSNABORT_SAME_EL << ARM_EL_EC_SHIFT);
> +    }
> +
> +    /* In the aarch64, there is only 32-bit instruction*/
> +    esr |= ARM_EL_IL;
> +    env->exception.syndrome = esr;
> +    cc->do_interrupt(c);
> +
> +    /* set ESR_EL1 */
> +    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.esr_el[1]));
> +
> +    if (ret) {
> +        fprintf(stderr, "<%s> failed to set esr_el1\n", __func__);
> +        abort();
> +    }
> +
> +    /* set FAR_EL1 */
> +    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.far_el[1]));
> +    if (ret) {
> +        fprintf(stderr, "<%s> failed to set far_el1\n", __func__);
> +        abort();
> +    }
> +
> +    return 0;
> +}
> +
>  #define AARCH64_CORE_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
>                   KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
>
> @@ -599,6 +674,9 @@ int kvm_arm_cpreg_level(uint64_t regidx)
>  #define AARCH64_SIMD_CTRL_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \
>                   KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
>
> +#define AARCH64_FAULT_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
> +                 KVM_REG_ARM64_FAULT | (x))
> +
>  int kvm_arch_put_registers(CPUState *cs, int level)
>  {
>      struct kvm_one_reg reg;
> @@ -873,6 +951,22 @@ int kvm_arch_get_registers(CPUState *cs)
>      }
>      vfp_set_fpcr(env, fpr);
>
> +    if (is_a64(env)) {
> +        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_ESR_EC);
> +        reg.addr = (uintptr_t)(&env->exception.syndrome);
> +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> +        if (ret) {
> +            return ret;
> +        }
> +
> +        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_FAR);
> +        reg.addr = (uintptr_t)(&env->exception.vaddress);
> +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);

This looks dubious. exception.syndrome and exception.vaddress
are just internal information QEMU uses, not guest visible things.
And only synchronizing them if the CPU happens to be in AArch64
at the point when this function is called is also odd.

> +        if (ret) {
> +            return ret;
> +        }
> +    }
> +
>      if (!write_kvmstate_to_list(cpu)) {
>          return EINVAL;
>      }
> @@ -887,6 +981,62 @@ int kvm_arch_get_registers(CPUState *cs)
>      return ret;
>  }
>
> +static bool is_abort_sea(unsigned long syndrome)
> +{
> +    unsigned long fault_status;

Don't use "unsigned long" when you really mean uint32_t.

> +    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
> +    if ((ec != EC_INSNABORT) && (ec != EC_DATAABORT)) {
> +        return false;
> +    }
> +
> +    fault_status = syndrome & ARM_EL_FSC_TYPE;
> +    switch (fault_status) {
> +    case FSC_SEA:
> +    case FSC_SEA_TTW0:
> +    case FSC_SEA_TTW1:
> +    case FSC_SEA_TTW2:
> +    case FSC_SEA_TTW3:
> +    case FSC_SECC:
> +    case FSC_SECC_TTW0:
> +    case FSC_SECC_TTW1:
> +    case FSC_SECC_TTW2:
> +    case FSC_SECC_TTW3:
> +        return true;
> +    default:
> +        return false;
> +    }
> +}
> +
> +void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
> +{
> +    ram_addr_t ram_addr;
> +    hwaddr paddr;
> +
> +    ARMCPU *cpu = ARM_CPU(c);
> +    CPUARMState *env = &cpu->env;
> +    assert(code == BUS_MCEERR_AR || code == BUS_MCEERR_AO);
> +    if (addr) {
> +        ram_addr = qemu_ram_addr_from_host(addr);
> +        if (ram_addr != RAM_ADDR_INVALID &&
> +            kvm_physical_memory_addr_from_host(c->kvm_state, addr, &paddr)) {
> +            kvm_cpu_synchronize_state(c);
> +            kvm_hwpoison_page_add(ram_addr);
> +            if (is_abort_sea(env->exception.syndrome)) {
> +                ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
> +                kvm_inject_arm_sea(c);
> +            }

This looks a bit odd. There are cases where we hwpoison the page
but don't tell the guest about it? When would we get this kind
of sigbus when the exception syndrome wasn't the right kind ?

Are we guaranteed not to get this kind of signal if we told
the kernel not to expose RAS to the guest ?

> +            return;
> +        }
> +        fprintf(stderr, "Hardware memory error for memory used by "
> +                "QEMU itself instead of guest system!\n");
> +    }
> +
> +    if (code == BUS_MCEERR_AR) {
> +        fprintf(stderr, "Hardware memory error!\n");
> +        exit(1);
> +    }
> +}
> +
>  /* C6.6.29 BRK instruction */
>  static const uint32_t brk_insn = 0xd4200000;
>
> diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
> index 633d088..7cdde97 100644
> --- a/target/arm/kvm_arm.h
> +++ b/target/arm/kvm_arm.h
> @@ -288,4 +288,5 @@ static inline const char *its_class_name(void)
>      }
>  }
>
> +void kvm_hwpoison_page_add(ram_addr_t ram_addr);

Any new globally-visible function prototype in a header should
have a doc-comment formatted documentation comment, please.

>  #endif
> --
> 1.8.3.1

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 5/6] target-arm: kvm64: handle SIGBUS signal for synchronous External Abort
@ 2017-09-05 17:46     ` Peter Maydell
  0 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:46 UTC (permalink / raw)
  To: linux-arm-kernel

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> Add SIGBUS signal handler. In this handler, it checks
> the exception type, translates the host VA which is
> delivered by host or KVM to guest PA, then fills this
> PA to CPER, finally injects a Error to guest OS through
> KVM.
>
> Add synchronous external abort injection logic, setup
> spsr_elx, esr_elx, PSTATE, far_elx, elr_elx etc, when
> switch to guest OS, it will jump to the synchronous
> external abort vector table entry.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> Signed-off-by: Quanming Wu <wuquanming@huawei.com>
> ---
>  include/sysemu/kvm.h          |   2 +-
>  linux-headers/asm-arm64/kvm.h |   5 ++
>  target/arm/internals.h        |  13 ++++
>  target/arm/kvm.c              |  34 ++++++++++
>  target/arm/kvm64.c            | 150 ++++++++++++++++++++++++++++++++++++++++++
>  target/arm/kvm_arm.h          |   1 +
>  6 files changed, 204 insertions(+), 1 deletion(-)

Have you tested whether this patchset builds OK on aarch32 ?

> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index 3a458f5..90c1605 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -361,7 +361,7 @@ bool kvm_vcpu_id_is_valid(int vcpu_id);
>  /* Returns VCPU ID to be used on KVM_CREATE_VCPU ioctl() */
>  unsigned long kvm_arch_vcpu_id(CPUState *cpu);
>
> -#ifdef TARGET_I386
> +#if defined(TARGET_I386) || defined(TARGET_AARCH64)
>  #define KVM_HAVE_MCE_INJECTION 1
>  void kvm_arch_on_sigbus_vcpu(CPUState *cpu, int code, void *addr);
>  #endif
> diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h
> index d254700..5909c30 100644
> --- a/linux-headers/asm-arm64/kvm.h
> +++ b/linux-headers/asm-arm64/kvm.h
> @@ -181,6 +181,11 @@ struct kvm_arch_memory_slot {
>  #define KVM_REG_ARM64_SYSREG_OP2_MASK  0x0000000000000007
>  #define KVM_REG_ARM64_SYSREG_OP2_SHIFT 0
>
> +/* AArch64 fault registers */
> +#define KVM_REG_ARM64_FAULT             (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
> +#define KVM_REG_ARM64_FAULT_ESR_EC      (0)
> +#define KVM_REG_ARM64_FAULT_FAR         (1)
> +
>  #define ARM64_SYS_REG_SHIFT_MASK(x,n) \
>         (((x) << KVM_REG_ARM64_SYSREG_ ## n ## _SHIFT) & \
>         KVM_REG_ARM64_SYSREG_ ## n ## _MASK)

Again, linux-headers changes need to go in their own header sync patch.

> diff --git a/target/arm/internals.h b/target/arm/internals.h
> index 1f6efef..fc0ad6d 100644
> --- a/target/arm/internals.h
> +++ b/target/arm/internals.h
> @@ -235,6 +235,19 @@ enum arm_exception_class {
>  #define ARM_EL_ISV_SHIFT 24
>  #define ARM_EL_IL (1 << ARM_EL_IL_SHIFT)
>  #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
> +#define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
> +#define ARM_EL_FSC_TYPE (0x3C)
> +
> +#define FSC_SEA         (0x10)
> +#define FSC_SEA_TTW0    (0x14)
> +#define FSC_SEA_TTW1    (0x15)
> +#define FSC_SEA_TTW2    (0x16)
> +#define FSC_SEA_TTW3    (0x17)
> +#define FSC_SECC        (0x18)
> +#define FSC_SECC_TTW0   (0x1c)
> +#define FSC_SECC_TTW1   (0x1d)
> +#define FSC_SECC_TTW2   (0x1e)
> +#define FSC_SECC_TTW3   (0x1f)
>
>  /* Utility functions for constructing various kinds of syndrome value.
>   * Note that in general we follow the AArch64 syndrome values; in a
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index 7c17f0d..2e1716a 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -129,6 +129,39 @@ void kvm_arm_destroy_scratch_host_vcpu(int *fdarray)
>      }
>  }
>
> +typedef struct HWPoisonPage {
> +    ram_addr_t ram_addr;
> +    QLIST_ENTRY(HWPoisonPage) list;
> +} HWPoisonPage;
> +
> +static QLIST_HEAD(, HWPoisonPage) hwpoison_page_list =
> +    QLIST_HEAD_INITIALIZER(hwpoison_page_list);
> +
> +static void kvm_unpoison_all(void *param)
> +{
> +    HWPoisonPage *page, *next_page;
> +
> +    QLIST_FOREACH_SAFE(page, &hwpoison_page_list, list, next_page) {
> +        QLIST_REMOVE(page, list);
> +        qemu_ram_remap(page->ram_addr, TARGET_PAGE_SIZE);
> +        g_free(page);
> +    }
> +}
> +
> +void kvm_hwpoison_page_add(ram_addr_t ram_addr)
> +{
> +    HWPoisonPage *page;
> +
> +    QLIST_FOREACH(page, &hwpoison_page_list, list) {
> +        if (page->ram_addr == ram_addr) {
> +            return;
> +        }
> +    }
> +    page = g_new(HWPoisonPage, 1);
> +    page->ram_addr = ram_addr;
> +    QLIST_INSERT_HEAD(&hwpoison_page_list, page, list);
> +}

This code has all just been copied-and-pasted from target/i386/kvm.c.
Please instead abstract it out properly into a cpu-independent
source file.

>  static void kvm_arm_host_cpu_class_init(ObjectClass *oc, void *data)
>  {
>      ARMHostCPUClass *ahcc = ARM_HOST_CPU_CLASS(oc);
> @@ -182,6 +215,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
>
>      cap_has_mp_state = kvm_check_extension(s, KVM_CAP_MP_STATE);
>
> +    qemu_register_reset(kvm_unpoison_all, NULL);
>      type_register_static(&host_arm_cpu_type_info);
>
>      return 0;
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index 0781367..d3bdab2 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -27,6 +27,8 @@
>  #include "kvm_arm.h"
>  #include "internals.h"
>  #include "hw/arm/arm.h"
> +#include "hw/acpi/acpi-defs.h"
> +#include "hw/acpi/hest_ghes.h"
>
>  static bool have_guest_debug;
>
> @@ -590,6 +592,79 @@ int kvm_arm_cpreg_level(uint64_t regidx)
>      return KVM_PUT_RUNTIME_STATE;
>  }
>
> +static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
> +{
> +    int i;
> +
> +    for (i = 0; i < cpu->cpreg_array_len; i++) {
> +        uint32_t regidx = kvm_to_cpreg_id(cpu->cpreg_indexes[i]);
> +        const ARMCPRegInfo *ri;
> +        ri = get_arm_cp_reginfo(cpu->cp_regs, regidx);
> +        if (!ri) {
> +            continue;
> +        }
> +
> +        if (ri->type & ARM_CP_NO_RAW) {
> +            continue;
> +        }
> +
> +        if (ri->fieldoffset == fieldoffset) {
> +            cpu->cpreg_values[i] = read_raw_cp_reg(&cpu->env, ri);
> +            return 0;
> +        }
> +    }
> +    return -EINVAL;

What is this ??? You should never need to look up things in
the cpreg arrays by fieldoffset.

The code for handling debug exits (software step, watchpoint, etc)
is probably a good place to look for how to deal with register state.

> +}
> +
> +/* Inject synchronous external abort */
> +static int kvm_inject_arm_sea(CPUState *c)
> +{
> +    ARMCPU *cpu = ARM_CPU(c);
> +    CPUARMState *env = &cpu->env;
> +    unsigned long cpsr = pstate_read(env);
> +    uint32_t esr = 0;
> +    int ret;
> +
> +    c->exception_index = EXCP_DATA_ABORT;
> +    /* Inject the exception to El1 */
> +    env->exception.target_el = 1;
> +    CPUClass *cc = CPU_GET_CLASS(c);
> +
> +    esr |= (EC_DATAABORT << ARM_EL_EC_SHIFT);

We have functions in internals.h for constructing ESR values,
please use them.

> +    /* This exception syndrome includes {I,D}FSC in the bits [5:0]
> +     */
> +    esr |= (env->exception.syndrome & 0x3f);
> +
> +    /* This exception is EL0 or EL1 fault. */
> +    if ((cpsr & 0xf) == PSTATE_MODE_EL0t) {
> +        esr |= (EC_INSNABORT << ARM_EL_EC_SHIFT);
> +    } else {
> +        esr |= (EC_INSNABORT_SAME_EL << ARM_EL_EC_SHIFT);
> +    }
> +
> +    /* In the aarch64, there is only 32-bit instruction*/
> +    esr |= ARM_EL_IL;
> +    env->exception.syndrome = esr;
> +    cc->do_interrupt(c);
> +
> +    /* set ESR_EL1 */
> +    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.esr_el[1]));
> +
> +    if (ret) {
> +        fprintf(stderr, "<%s> failed to set esr_el1\n", __func__);
> +        abort();
> +    }
> +
> +    /* set FAR_EL1 */
> +    ret = kvm_arm_cpreg_value(cpu, offsetof(CPUARMState, cp15.far_el[1]));
> +    if (ret) {
> +        fprintf(stderr, "<%s> failed to set far_el1\n", __func__);
> +        abort();
> +    }
> +
> +    return 0;
> +}
> +
>  #define AARCH64_CORE_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
>                   KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
>
> @@ -599,6 +674,9 @@ int kvm_arm_cpreg_level(uint64_t regidx)
>  #define AARCH64_SIMD_CTRL_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \
>                   KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
>
> +#define AARCH64_FAULT_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
> +                 KVM_REG_ARM64_FAULT | (x))
> +
>  int kvm_arch_put_registers(CPUState *cs, int level)
>  {
>      struct kvm_one_reg reg;
> @@ -873,6 +951,22 @@ int kvm_arch_get_registers(CPUState *cs)
>      }
>      vfp_set_fpcr(env, fpr);
>
> +    if (is_a64(env)) {
> +        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_ESR_EC);
> +        reg.addr = (uintptr_t)(&env->exception.syndrome);
> +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> +        if (ret) {
> +            return ret;
> +        }
> +
> +        reg.id = AARCH64_FAULT_REG(KVM_REG_ARM64_FAULT_FAR);
> +        reg.addr = (uintptr_t)(&env->exception.vaddress);
> +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);

This looks dubious. exception.syndrome and exception.vaddress
are just internal information QEMU uses, not guest visible things.
And only synchronizing them if the CPU happens to be in AArch64
at the point when this function is called is also odd.

> +        if (ret) {
> +            return ret;
> +        }
> +    }
> +
>      if (!write_kvmstate_to_list(cpu)) {
>          return EINVAL;
>      }
> @@ -887,6 +981,62 @@ int kvm_arch_get_registers(CPUState *cs)
>      return ret;
>  }
>
> +static bool is_abort_sea(unsigned long syndrome)
> +{
> +    unsigned long fault_status;

Don't use "unsigned long" when you really mean uint32_t.

> +    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);
> +    if ((ec != EC_INSNABORT) && (ec != EC_DATAABORT)) {
> +        return false;
> +    }
> +
> +    fault_status = syndrome & ARM_EL_FSC_TYPE;
> +    switch (fault_status) {
> +    case FSC_SEA:
> +    case FSC_SEA_TTW0:
> +    case FSC_SEA_TTW1:
> +    case FSC_SEA_TTW2:
> +    case FSC_SEA_TTW3:
> +    case FSC_SECC:
> +    case FSC_SECC_TTW0:
> +    case FSC_SECC_TTW1:
> +    case FSC_SECC_TTW2:
> +    case FSC_SECC_TTW3:
> +        return true;
> +    default:
> +        return false;
> +    }
> +}
> +
> +void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
> +{
> +    ram_addr_t ram_addr;
> +    hwaddr paddr;
> +
> +    ARMCPU *cpu = ARM_CPU(c);
> +    CPUARMState *env = &cpu->env;
> +    assert(code == BUS_MCEERR_AR || code == BUS_MCEERR_AO);
> +    if (addr) {
> +        ram_addr = qemu_ram_addr_from_host(addr);
> +        if (ram_addr != RAM_ADDR_INVALID &&
> +            kvm_physical_memory_addr_from_host(c->kvm_state, addr, &paddr)) {
> +            kvm_cpu_synchronize_state(c);
> +            kvm_hwpoison_page_add(ram_addr);
> +            if (is_abort_sea(env->exception.syndrome)) {
> +                ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
> +                kvm_inject_arm_sea(c);
> +            }

This looks a bit odd. There are cases where we hwpoison the page
but don't tell the guest about it? When would we get this kind
of sigbus when the exception syndrome wasn't the right kind ?

Are we guaranteed not to get this kind of signal if we told
the kernel not to expose RAS to the guest ?

> +            return;
> +        }
> +        fprintf(stderr, "Hardware memory error for memory used by "
> +                "QEMU itself instead of guest system!\n");
> +    }
> +
> +    if (code == BUS_MCEERR_AR) {
> +        fprintf(stderr, "Hardware memory error!\n");
> +        exit(1);
> +    }
> +}
> +
>  /* C6.6.29 BRK instruction */
>  static const uint32_t brk_insn = 0xd4200000;
>
> diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
> index 633d088..7cdde97 100644
> --- a/target/arm/kvm_arm.h
> +++ b/target/arm/kvm_arm.h
> @@ -288,4 +288,5 @@ static inline const char *its_class_name(void)
>      }
>  }
>
> +void kvm_hwpoison_page_add(ram_addr_t ram_addr);

Any new globally-visible function prototype in a header should
have a doc-comment formatted documentation comment, please.

>  #endif
> --
> 1.8.3.1

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 6/6] target-arm: kvm64: Handle SError interrupt for the guest OS
  2017-08-18 14:23   ` Dongjiu Geng
                       ` (2 preceding siblings ...)
  (?)
@ 2017-09-05 17:50     ` Peter Maydell
  -1 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:50 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> When guest OS happens SError interrupt(SEI), it will trap to host.
> Host firstly calls memory failure to deal with this error and decide
> whether it needs to deliver SIGBUS signal to userspace. The advantage
> that using signal to notify is that it can make the notification method
> is general, non-KVM user can also use it. when userspace gets this
> signal and knows this is SError interrupt, it will translate the
> delivered host VA to PA and record this PA to GHES.
>
> Because ARMv8.2 adds an extension to RAS to allow system software insert
> implicit Error Synchronization Barrier operations to isolate the error and
> allow passes specified syndrome to guest OS, so after record the CPER, user
> space calls IOCTL to pass a specified syndrome to KVM, then switch to guest
> OS, guest OS can use the recorded CPER record and syndrome information to
> do the recovery.
>
> The steps are shown below:
> 1. translate the host VA to guest OS PA and record this error PA to HEST table.
> 2. set specified virtual SError syndrome and pass the value to KVM.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> Signed-off-by: Quanming Wu <wuquanming@huawei.com>
> ---
>  linux-headers/linux/kvm.h |  1 +
>  target/arm/internals.h    |  1 +
>  target/arm/kvm64.c        | 28 ++++++++++++++++++++++++++++
>  3 files changed, 30 insertions(+)
>
> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
> index 2aa176e..10dfcab 100644
> --- a/linux-headers/linux/kvm.h
> +++ b/linux-headers/linux/kvm.h
> @@ -1356,6 +1356,7 @@ struct kvm_s390_ucas_mapping {
>  /* Available with KVM_CAP_S390_CMMA_MIGRATION */
>  #define KVM_S390_GET_CMMA_BITS      _IOWR(KVMIO, 0xb8, struct kvm_s390_cmma_log)
>  #define KVM_S390_SET_CMMA_BITS      _IOW(KVMIO, 0xb9, struct kvm_s390_cmma_log)
> +#define KVM_ARM_SEI                 _IO(KVMIO,   0xb10)
>
>  #define KVM_DEV_ASSIGN_ENABLE_IOMMU    (1 << 0)
>  #define KVM_DEV_ASSIGN_PCI_2_3         (1 << 1)
> diff --git a/target/arm/internals.h b/target/arm/internals.h
> index fc0ad6d..18b1cbc 100644
> --- a/target/arm/internals.h
> +++ b/target/arm/internals.h
> @@ -237,6 +237,7 @@ enum arm_exception_class {
>  #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
>  #define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
>  #define ARM_EL_FSC_TYPE (0x3C)
> +#define ARM_EL_ISS_MASK ((1 << ARM_EL_IL_SHIFT) - 1)
>
>  #define FSC_SEA         (0x10)
>  #define FSC_SEA_TTW0    (0x14)
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index d3bdab2..b84cb49 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -616,6 +616,22 @@ static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
>      return -EINVAL;
>  }
>
> +static int kvm_inject_arm_sei(CPUState *cs)
> +{
> +    ARMCPU *cpu = ARM_CPU(cs);
> +    CPUARMState *env = &cpu->env;
> +
> +    unsigned long syndrome = env->exception.vaddress;
> +    /* set virtual SError syndrome */
> +    if (arm_feature(env, ARM_FEATURE_RAS_EXTENSION)) {
> +        syndrome = syndrome & ARM_EL_ISS_MASK;
> +    } else {
> +        syndrome = 0;
> +    }
> +
> +    return  kvm_vcpu_ioctl(CPU(cpu), KVM_ARM_SEI, &syndrome);

This looks odd. If we don't have the RAS extension why do
we need to do anything at all here ?

> +}
> +
>  /* Inject synchronous external abort */
>  static int kvm_inject_arm_sea(CPUState *c)
>  {
> @@ -1007,6 +1023,15 @@ static bool is_abort_sea(unsigned long syndrome)
>      }
>  }
>
> +static bool is_abort_sei(unsigned long syndrome)
> +{
> +    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);

You don't need to bother masking here -- in other places in
QEMU we assume that the EC field is at the top of the word,
so just "syndrome >> ARM_EL_EC_SHIFT" is sufficient.

> +    if ((ec != EC_SERROR))
> +        return false;
> +    else
> +        return true;

scripts/checkpatch.pl should tell you that this if needs braces
(it's good to get in the habit of running it on all patches; it
is not always correct, so judgement is required, but it will flag
up some common mistakes).

In this particular case, you should just
   return ec == EC_SERROR;
though.

> +}
> +
>  void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
>  {
>      ram_addr_t ram_addr;
> @@ -1024,6 +1049,9 @@ void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
>              if (is_abort_sea(env->exception.syndrome)) {
>                  ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
>                  kvm_inject_arm_sea(c);
> +            } else if (is_abort_sei(env->exception.syndrome)) {
> +                ghes_update_guest(ACPI_HEST_NOTIFY_SEI, paddr);
> +                kvm_inject_arm_sei(c);
>              }
>              return;
>          }
> --
> 1.8.3.1

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 6/6] target-arm: kvm64: Handle SError interrupt for the guest OS
@ 2017-09-05 17:50     ` Peter Maydell
  0 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:50 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm, lkml - Kernel Mailing List,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, Huangshaoyu,
	wuquanming, Linuxarm, zhengqiang10

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> When guest OS happens SError interrupt(SEI), it will trap to host.
> Host firstly calls memory failure to deal with this error and decide
> whether it needs to deliver SIGBUS signal to userspace. The advantage
> that using signal to notify is that it can make the notification method
> is general, non-KVM user can also use it. when userspace gets this
> signal and knows this is SError interrupt, it will translate the
> delivered host VA to PA and record this PA to GHES.
>
> Because ARMv8.2 adds an extension to RAS to allow system software insert
> implicit Error Synchronization Barrier operations to isolate the error and
> allow passes specified syndrome to guest OS, so after record the CPER, user
> space calls IOCTL to pass a specified syndrome to KVM, then switch to guest
> OS, guest OS can use the recorded CPER record and syndrome information to
> do the recovery.
>
> The steps are shown below:
> 1. translate the host VA to guest OS PA and record this error PA to HEST table.
> 2. set specified virtual SError syndrome and pass the value to KVM.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> Signed-off-by: Quanming Wu <wuquanming@huawei.com>
> ---
>  linux-headers/linux/kvm.h |  1 +
>  target/arm/internals.h    |  1 +
>  target/arm/kvm64.c        | 28 ++++++++++++++++++++++++++++
>  3 files changed, 30 insertions(+)
>
> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
> index 2aa176e..10dfcab 100644
> --- a/linux-headers/linux/kvm.h
> +++ b/linux-headers/linux/kvm.h
> @@ -1356,6 +1356,7 @@ struct kvm_s390_ucas_mapping {
>  /* Available with KVM_CAP_S390_CMMA_MIGRATION */
>  #define KVM_S390_GET_CMMA_BITS      _IOWR(KVMIO, 0xb8, struct kvm_s390_cmma_log)
>  #define KVM_S390_SET_CMMA_BITS      _IOW(KVMIO, 0xb9, struct kvm_s390_cmma_log)
> +#define KVM_ARM_SEI                 _IO(KVMIO,   0xb10)
>
>  #define KVM_DEV_ASSIGN_ENABLE_IOMMU    (1 << 0)
>  #define KVM_DEV_ASSIGN_PCI_2_3         (1 << 1)
> diff --git a/target/arm/internals.h b/target/arm/internals.h
> index fc0ad6d..18b1cbc 100644
> --- a/target/arm/internals.h
> +++ b/target/arm/internals.h
> @@ -237,6 +237,7 @@ enum arm_exception_class {
>  #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
>  #define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
>  #define ARM_EL_FSC_TYPE (0x3C)
> +#define ARM_EL_ISS_MASK ((1 << ARM_EL_IL_SHIFT) - 1)
>
>  #define FSC_SEA         (0x10)
>  #define FSC_SEA_TTW0    (0x14)
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index d3bdab2..b84cb49 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -616,6 +616,22 @@ static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
>      return -EINVAL;
>  }
>
> +static int kvm_inject_arm_sei(CPUState *cs)
> +{
> +    ARMCPU *cpu = ARM_CPU(cs);
> +    CPUARMState *env = &cpu->env;
> +
> +    unsigned long syndrome = env->exception.vaddress;
> +    /* set virtual SError syndrome */
> +    if (arm_feature(env, ARM_FEATURE_RAS_EXTENSION)) {
> +        syndrome = syndrome & ARM_EL_ISS_MASK;
> +    } else {
> +        syndrome = 0;
> +    }
> +
> +    return  kvm_vcpu_ioctl(CPU(cpu), KVM_ARM_SEI, &syndrome);

This looks odd. If we don't have the RAS extension why do
we need to do anything at all here ?

> +}
> +
>  /* Inject synchronous external abort */
>  static int kvm_inject_arm_sea(CPUState *c)
>  {
> @@ -1007,6 +1023,15 @@ static bool is_abort_sea(unsigned long syndrome)
>      }
>  }
>
> +static bool is_abort_sei(unsigned long syndrome)
> +{
> +    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);

You don't need to bother masking here -- in other places in
QEMU we assume that the EC field is at the top of the word,
so just "syndrome >> ARM_EL_EC_SHIFT" is sufficient.

> +    if ((ec != EC_SERROR))
> +        return false;
> +    else
> +        return true;

scripts/checkpatch.pl should tell you that this if needs braces
(it's good to get in the habit of running it on all patches; it
is not always correct, so judgement is required, but it will flag
up some common mistakes).

In this particular case, you should just
   return ec == EC_SERROR;
though.

> +}
> +
>  void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
>  {
>      ram_addr_t ram_addr;
> @@ -1024,6 +1049,9 @@ void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
>              if (is_abort_sea(env->exception.syndrome)) {
>                  ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
>                  kvm_inject_arm_sea(c);
> +            } else if (is_abort_sei(env->exception.syndrome)) {
> +                ghes_update_guest(ACPI_HEST_NOTIFY_SEI, paddr);
> +                kvm_inject_arm_sei(c);
>              }
>              return;
>          }
> --
> 1.8.3.1

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 6/6] target-arm: kvm64: Handle SError interrupt for the guest OS
@ 2017-09-05 17:50     ` Peter Maydell
  0 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:50 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm, l

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> When guest OS happens SError interrupt(SEI), it will trap to host.
> Host firstly calls memory failure to deal with this error and decide
> whether it needs to deliver SIGBUS signal to userspace. The advantage
> that using signal to notify is that it can make the notification method
> is general, non-KVM user can also use it. when userspace gets this
> signal and knows this is SError interrupt, it will translate the
> delivered host VA to PA and record this PA to GHES.
>
> Because ARMv8.2 adds an extension to RAS to allow system software insert
> implicit Error Synchronization Barrier operations to isolate the error and
> allow passes specified syndrome to guest OS, so after record the CPER, user
> space calls IOCTL to pass a specified syndrome to KVM, then switch to guest
> OS, guest OS can use the recorded CPER record and syndrome information to
> do the recovery.
>
> The steps are shown below:
> 1. translate the host VA to guest OS PA and record this error PA to HEST table.
> 2. set specified virtual SError syndrome and pass the value to KVM.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> Signed-off-by: Quanming Wu <wuquanming@huawei.com>
> ---
>  linux-headers/linux/kvm.h |  1 +
>  target/arm/internals.h    |  1 +
>  target/arm/kvm64.c        | 28 ++++++++++++++++++++++++++++
>  3 files changed, 30 insertions(+)
>
> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
> index 2aa176e..10dfcab 100644
> --- a/linux-headers/linux/kvm.h
> +++ b/linux-headers/linux/kvm.h
> @@ -1356,6 +1356,7 @@ struct kvm_s390_ucas_mapping {
>  /* Available with KVM_CAP_S390_CMMA_MIGRATION */
>  #define KVM_S390_GET_CMMA_BITS      _IOWR(KVMIO, 0xb8, struct kvm_s390_cmma_log)
>  #define KVM_S390_SET_CMMA_BITS      _IOW(KVMIO, 0xb9, struct kvm_s390_cmma_log)
> +#define KVM_ARM_SEI                 _IO(KVMIO,   0xb10)
>
>  #define KVM_DEV_ASSIGN_ENABLE_IOMMU    (1 << 0)
>  #define KVM_DEV_ASSIGN_PCI_2_3         (1 << 1)
> diff --git a/target/arm/internals.h b/target/arm/internals.h
> index fc0ad6d..18b1cbc 100644
> --- a/target/arm/internals.h
> +++ b/target/arm/internals.h
> @@ -237,6 +237,7 @@ enum arm_exception_class {
>  #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
>  #define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
>  #define ARM_EL_FSC_TYPE (0x3C)
> +#define ARM_EL_ISS_MASK ((1 << ARM_EL_IL_SHIFT) - 1)
>
>  #define FSC_SEA         (0x10)
>  #define FSC_SEA_TTW0    (0x14)
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index d3bdab2..b84cb49 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -616,6 +616,22 @@ static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
>      return -EINVAL;
>  }
>
> +static int kvm_inject_arm_sei(CPUState *cs)
> +{
> +    ARMCPU *cpu = ARM_CPU(cs);
> +    CPUARMState *env = &cpu->env;
> +
> +    unsigned long syndrome = env->exception.vaddress;
> +    /* set virtual SError syndrome */
> +    if (arm_feature(env, ARM_FEATURE_RAS_EXTENSION)) {
> +        syndrome = syndrome & ARM_EL_ISS_MASK;
> +    } else {
> +        syndrome = 0;
> +    }
> +
> +    return  kvm_vcpu_ioctl(CPU(cpu), KVM_ARM_SEI, &syndrome);

This looks odd. If we don't have the RAS extension why do
we need to do anything at all here ?

> +}
> +
>  /* Inject synchronous external abort */
>  static int kvm_inject_arm_sea(CPUState *c)
>  {
> @@ -1007,6 +1023,15 @@ static bool is_abort_sea(unsigned long syndrome)
>      }
>  }
>
> +static bool is_abort_sei(unsigned long syndrome)
> +{
> +    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);

You don't need to bother masking here -- in other places in
QEMU we assume that the EC field is at the top of the word,
so just "syndrome >> ARM_EL_EC_SHIFT" is sufficient.

> +    if ((ec != EC_SERROR))
> +        return false;
> +    else
> +        return true;

scripts/checkpatch.pl should tell you that this if needs braces
(it's good to get in the habit of running it on all patches; it
is not always correct, so judgement is required, but it will flag
up some common mistakes).

In this particular case, you should just
   return ec == EC_SERROR;
though.

> +}
> +
>  void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
>  {
>      ram_addr_t ram_addr;
> @@ -1024,6 +1049,9 @@ void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
>              if (is_abort_sea(env->exception.syndrome)) {
>                  ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
>                  kvm_inject_arm_sea(c);
> +            } else if (is_abort_sei(env->exception.syndrome)) {
> +                ghes_update_guest(ACPI_HEST_NOTIFY_SEI, paddr);
> +                kvm_inject_arm_sei(c);
>              }
>              return;
>          }
> --
> 1.8.3.1

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 6/6] target-arm: kvm64: Handle SError interrupt for the guest OS
@ 2017-09-05 17:50     ` Peter Maydell
  0 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:50 UTC (permalink / raw)
  To: Dongjiu Geng
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm, lkml - Kernel Mailing List,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, Huangshaoyu,
	wuquanming, Linuxarm, zhengqiang10

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> When guest OS happens SError interrupt(SEI), it will trap to host.
> Host firstly calls memory failure to deal with this error and decide
> whether it needs to deliver SIGBUS signal to userspace. The advantage
> that using signal to notify is that it can make the notification method
> is general, non-KVM user can also use it. when userspace gets this
> signal and knows this is SError interrupt, it will translate the
> delivered host VA to PA and record this PA to GHES.
>
> Because ARMv8.2 adds an extension to RAS to allow system software insert
> implicit Error Synchronization Barrier operations to isolate the error and
> allow passes specified syndrome to guest OS, so after record the CPER, user
> space calls IOCTL to pass a specified syndrome to KVM, then switch to guest
> OS, guest OS can use the recorded CPER record and syndrome information to
> do the recovery.
>
> The steps are shown below:
> 1. translate the host VA to guest OS PA and record this error PA to HEST table.
> 2. set specified virtual SError syndrome and pass the value to KVM.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> Signed-off-by: Quanming Wu <wuquanming@huawei.com>
> ---
>  linux-headers/linux/kvm.h |  1 +
>  target/arm/internals.h    |  1 +
>  target/arm/kvm64.c        | 28 ++++++++++++++++++++++++++++
>  3 files changed, 30 insertions(+)
>
> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
> index 2aa176e..10dfcab 100644
> --- a/linux-headers/linux/kvm.h
> +++ b/linux-headers/linux/kvm.h
> @@ -1356,6 +1356,7 @@ struct kvm_s390_ucas_mapping {
>  /* Available with KVM_CAP_S390_CMMA_MIGRATION */
>  #define KVM_S390_GET_CMMA_BITS      _IOWR(KVMIO, 0xb8, struct kvm_s390_cmma_log)
>  #define KVM_S390_SET_CMMA_BITS      _IOW(KVMIO, 0xb9, struct kvm_s390_cmma_log)
> +#define KVM_ARM_SEI                 _IO(KVMIO,   0xb10)
>
>  #define KVM_DEV_ASSIGN_ENABLE_IOMMU    (1 << 0)
>  #define KVM_DEV_ASSIGN_PCI_2_3         (1 << 1)
> diff --git a/target/arm/internals.h b/target/arm/internals.h
> index fc0ad6d..18b1cbc 100644
> --- a/target/arm/internals.h
> +++ b/target/arm/internals.h
> @@ -237,6 +237,7 @@ enum arm_exception_class {
>  #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
>  #define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
>  #define ARM_EL_FSC_TYPE (0x3C)
> +#define ARM_EL_ISS_MASK ((1 << ARM_EL_IL_SHIFT) - 1)
>
>  #define FSC_SEA         (0x10)
>  #define FSC_SEA_TTW0    (0x14)
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index d3bdab2..b84cb49 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -616,6 +616,22 @@ static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
>      return -EINVAL;
>  }
>
> +static int kvm_inject_arm_sei(CPUState *cs)
> +{
> +    ARMCPU *cpu = ARM_CPU(cs);
> +    CPUARMState *env = &cpu->env;
> +
> +    unsigned long syndrome = env->exception.vaddress;
> +    /* set virtual SError syndrome */
> +    if (arm_feature(env, ARM_FEATURE_RAS_EXTENSION)) {
> +        syndrome = syndrome & ARM_EL_ISS_MASK;
> +    } else {
> +        syndrome = 0;
> +    }
> +
> +    return  kvm_vcpu_ioctl(CPU(cpu), KVM_ARM_SEI, &syndrome);

This looks odd. If we don't have the RAS extension why do
we need to do anything at all here ?

> +}
> +
>  /* Inject synchronous external abort */
>  static int kvm_inject_arm_sea(CPUState *c)
>  {
> @@ -1007,6 +1023,15 @@ static bool is_abort_sea(unsigned long syndrome)
>      }
>  }
>
> +static bool is_abort_sei(unsigned long syndrome)
> +{
> +    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);

You don't need to bother masking here -- in other places in
QEMU we assume that the EC field is at the top of the word,
so just "syndrome >> ARM_EL_EC_SHIFT" is sufficient.

> +    if ((ec != EC_SERROR))
> +        return false;
> +    else
> +        return true;

scripts/checkpatch.pl should tell you that this if needs braces
(it's good to get in the habit of running it on all patches; it
is not always correct, so judgement is required, but it will flag
up some common mistakes).

In this particular case, you should just
   return ec == EC_SERROR;
though.

> +}
> +
>  void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
>  {
>      ram_addr_t ram_addr;
> @@ -1024,6 +1049,9 @@ void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
>              if (is_abort_sea(env->exception.syndrome)) {
>                  ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
>                  kvm_inject_arm_sea(c);
> +            } else if (is_abort_sei(env->exception.syndrome)) {
> +                ghes_update_guest(ACPI_HEST_NOTIFY_SEI, paddr);
> +                kvm_inject_arm_sei(c);
>              }
>              return;
>          }
> --
> 1.8.3.1

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 6/6] target-arm: kvm64: Handle SError interrupt for the guest OS
@ 2017-09-05 17:50     ` Peter Maydell
  0 siblings, 0 replies; 129+ messages in thread
From: Peter Maydell @ 2017-09-05 17:50 UTC (permalink / raw)
  To: linux-arm-kernel

On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
> When guest OS happens SError interrupt(SEI), it will trap to host.
> Host firstly calls memory failure to deal with this error and decide
> whether it needs to deliver SIGBUS signal to userspace. The advantage
> that using signal to notify is that it can make the notification method
> is general, non-KVM user can also use it. when userspace gets this
> signal and knows this is SError interrupt, it will translate the
> delivered host VA to PA and record this PA to GHES.
>
> Because ARMv8.2 adds an extension to RAS to allow system software insert
> implicit Error Synchronization Barrier operations to isolate the error and
> allow passes specified syndrome to guest OS, so after record the CPER, user
> space calls IOCTL to pass a specified syndrome to KVM, then switch to guest
> OS, guest OS can use the recorded CPER record and syndrome information to
> do the recovery.
>
> The steps are shown below:
> 1. translate the host VA to guest OS PA and record this error PA to HEST table.
> 2. set specified virtual SError syndrome and pass the value to KVM.
>
> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
> Signed-off-by: Quanming Wu <wuquanming@huawei.com>
> ---
>  linux-headers/linux/kvm.h |  1 +
>  target/arm/internals.h    |  1 +
>  target/arm/kvm64.c        | 28 ++++++++++++++++++++++++++++
>  3 files changed, 30 insertions(+)
>
> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
> index 2aa176e..10dfcab 100644
> --- a/linux-headers/linux/kvm.h
> +++ b/linux-headers/linux/kvm.h
> @@ -1356,6 +1356,7 @@ struct kvm_s390_ucas_mapping {
>  /* Available with KVM_CAP_S390_CMMA_MIGRATION */
>  #define KVM_S390_GET_CMMA_BITS      _IOWR(KVMIO, 0xb8, struct kvm_s390_cmma_log)
>  #define KVM_S390_SET_CMMA_BITS      _IOW(KVMIO, 0xb9, struct kvm_s390_cmma_log)
> +#define KVM_ARM_SEI                 _IO(KVMIO,   0xb10)
>
>  #define KVM_DEV_ASSIGN_ENABLE_IOMMU    (1 << 0)
>  #define KVM_DEV_ASSIGN_PCI_2_3         (1 << 1)
> diff --git a/target/arm/internals.h b/target/arm/internals.h
> index fc0ad6d..18b1cbc 100644
> --- a/target/arm/internals.h
> +++ b/target/arm/internals.h
> @@ -237,6 +237,7 @@ enum arm_exception_class {
>  #define ARM_EL_ISV (1 << ARM_EL_ISV_SHIFT)
>  #define ARM_EL_EC_MASK  ((0x3F) << ARM_EL_EC_SHIFT)
>  #define ARM_EL_FSC_TYPE (0x3C)
> +#define ARM_EL_ISS_MASK ((1 << ARM_EL_IL_SHIFT) - 1)
>
>  #define FSC_SEA         (0x10)
>  #define FSC_SEA_TTW0    (0x14)
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index d3bdab2..b84cb49 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -616,6 +616,22 @@ static int kvm_arm_cpreg_value(ARMCPU *cpu, ptrdiff_t fieldoffset)
>      return -EINVAL;
>  }
>
> +static int kvm_inject_arm_sei(CPUState *cs)
> +{
> +    ARMCPU *cpu = ARM_CPU(cs);
> +    CPUARMState *env = &cpu->env;
> +
> +    unsigned long syndrome = env->exception.vaddress;
> +    /* set virtual SError syndrome */
> +    if (arm_feature(env, ARM_FEATURE_RAS_EXTENSION)) {
> +        syndrome = syndrome & ARM_EL_ISS_MASK;
> +    } else {
> +        syndrome = 0;
> +    }
> +
> +    return  kvm_vcpu_ioctl(CPU(cpu), KVM_ARM_SEI, &syndrome);

This looks odd. If we don't have the RAS extension why do
we need to do anything at all here ?

> +}
> +
>  /* Inject synchronous external abort */
>  static int kvm_inject_arm_sea(CPUState *c)
>  {
> @@ -1007,6 +1023,15 @@ static bool is_abort_sea(unsigned long syndrome)
>      }
>  }
>
> +static bool is_abort_sei(unsigned long syndrome)
> +{
> +    uint8_t ec = ((syndrome & ARM_EL_EC_MASK) >> ARM_EL_EC_SHIFT);

You don't need to bother masking here -- in other places in
QEMU we assume that the EC field is at the top of the word,
so just "syndrome >> ARM_EL_EC_SHIFT" is sufficient.

> +    if ((ec != EC_SERROR))
> +        return false;
> +    else
> +        return true;

scripts/checkpatch.pl should tell you that this if needs braces
(it's good to get in the habit of running it on all patches; it
is not always correct, so judgement is required, but it will flag
up some common mistakes).

In this particular case, you should just
   return ec == EC_SERROR;
though.

> +}
> +
>  void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
>  {
>      ram_addr_t ram_addr;
> @@ -1024,6 +1049,9 @@ void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
>              if (is_abort_sea(env->exception.syndrome)) {
>                  ghes_update_guest(ACPI_HEST_NOTIFY_SEA, paddr);
>                  kvm_inject_arm_sea(c);
> +            } else if (is_abort_sei(env->exception.syndrome)) {
> +                ghes_update_guest(ACPI_HEST_NOTIFY_SEI, paddr);
> +                kvm_inject_arm_sei(c);
>              }
>              return;
>          }
> --
> 1.8.3.1

thanks
-- PMM

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
  2017-09-05 17:26     ` Peter Maydell
                         ` (2 preceding siblings ...)
  (?)
@ 2017-09-06  9:35       ` gengdongjiu
  -1 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-09-06  9:35 UTC (permalink / raw)
  To: Peter Maydell
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm

Hi Peter,
  Thanks very much for your review, I will check your comments in detail and reply.


On 2017/9/6 1:26, Peter Maydell wrote:
> On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
>> check if kvm supports guest RAS EXTENSION. if so, set
>> corresponding feature bit for vcpu.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  linux-headers/linux/kvm.h | 1 +
>>  target/arm/cpu.h          | 3 +++
>>  target/arm/kvm64.c        | 8 ++++++++
>>  3 files changed, 12 insertions(+)
>>
>> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
>> index 7971a4f..2aa176e 100644
>> --- a/linux-headers/linux/kvm.h
>> +++ b/linux-headers/linux/kvm.h
>> @@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
>>  #define KVM_CAP_PPC_SMT_POSSIBLE 147
>>  #define KVM_CAP_HYPERV_SYNIC2 148
>>  #define KVM_CAP_HYPERV_VP_INDEX 149
>> +#define KVM_CAP_ARM_RAS_EXTENSION 150
>>
>>  #ifdef KVM_CAP_IRQ_ROUTING
>>
> 
> Hi. Changes to linux-headers need to be done as a patch of their
> own created using scripts/update-linux-headers.sh run against a
> mainline kernel tree (and with a commit message that quotes the
> kernel commit hash used). This ensures that we have a consistent
> set of headers that don't diverge from the kernel copy.
> 
>> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
>> index b39d64a..6b0961b 100644
>> --- a/target/arm/cpu.h
>> +++ b/target/arm/cpu.h
>> @@ -611,6 +611,8 @@ struct ARMCPU {
>>
>>      /* CPU has memory protection unit */
>>      bool has_mpu;
>> +    /* CPU has ras extension unit */
>> +    bool has_ras_extension;
>>      /* PMSAv7 MPU number of supported regions */
>>      uint32_t pmsav7_dregion;
>>
>> @@ -1229,6 +1231,7 @@ enum arm_features {
>>      ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
>>      ARM_FEATURE_PMU, /* has PMU support */
>>      ARM_FEATURE_VBAR, /* has cp15 VBAR */
>> +    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */
> 
> Missing space after '/*' ?
> 
>>  };
>>
>>  static inline int arm_feature(CPUARMState *env, int feature)
>> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
>> index a16abc8..0781367 100644
>> --- a/target/arm/kvm64.c
>> +++ b/target/arm/kvm64.c
>> @@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>          unset_feature(&env->features, ARM_FEATURE_PMU);
>>      }
>>
>> +    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
>> +        cpu->has_ras_extension = true;
>> +        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
>> +    } else {
>> +        cpu->has_ras_extension = false;
>> +        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
>> +    }
>> +
> 
> Shouldn't we need to also tell the kernel that we actually want
> it to expose RAS to the guest? Compare the PMU code in this
> function, where we set a kvm_init_features bit to do this.
> (This suggests that your ABI for the kernel part of this feature
> may not be correct?)
> 
> You should also not be calling set_feature() here -- if the
> CPU features bit doesn't say "this CPU should have the RAS
> extensions" we shouldn't create a CPU with them. Instead
> you should set it in kvm_arm_get_host_cpu_features() (again,
> compare the PMU code).
> 
> thanks
> -- PMM
> 
> .
> 


^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
@ 2017-09-06  9:35       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-09-06  9:35 UTC (permalink / raw)
  To: Peter Maydell
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm, lkml - Kernel Mailing List,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, Huangshaoyu,
	wuquanming, Linuxarm, zhengqiang10

Hi Peter,
  Thanks very much for your review, I will check your comments in detail and reply.


On 2017/9/6 1:26, Peter Maydell wrote:
> On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
>> check if kvm supports guest RAS EXTENSION. if so, set
>> corresponding feature bit for vcpu.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  linux-headers/linux/kvm.h | 1 +
>>  target/arm/cpu.h          | 3 +++
>>  target/arm/kvm64.c        | 8 ++++++++
>>  3 files changed, 12 insertions(+)
>>
>> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
>> index 7971a4f..2aa176e 100644
>> --- a/linux-headers/linux/kvm.h
>> +++ b/linux-headers/linux/kvm.h
>> @@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
>>  #define KVM_CAP_PPC_SMT_POSSIBLE 147
>>  #define KVM_CAP_HYPERV_SYNIC2 148
>>  #define KVM_CAP_HYPERV_VP_INDEX 149
>> +#define KVM_CAP_ARM_RAS_EXTENSION 150
>>
>>  #ifdef KVM_CAP_IRQ_ROUTING
>>
> 
> Hi. Changes to linux-headers need to be done as a patch of their
> own created using scripts/update-linux-headers.sh run against a
> mainline kernel tree (and with a commit message that quotes the
> kernel commit hash used). This ensures that we have a consistent
> set of headers that don't diverge from the kernel copy.
> 
>> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
>> index b39d64a..6b0961b 100644
>> --- a/target/arm/cpu.h
>> +++ b/target/arm/cpu.h
>> @@ -611,6 +611,8 @@ struct ARMCPU {
>>
>>      /* CPU has memory protection unit */
>>      bool has_mpu;
>> +    /* CPU has ras extension unit */
>> +    bool has_ras_extension;
>>      /* PMSAv7 MPU number of supported regions */
>>      uint32_t pmsav7_dregion;
>>
>> @@ -1229,6 +1231,7 @@ enum arm_features {
>>      ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
>>      ARM_FEATURE_PMU, /* has PMU support */
>>      ARM_FEATURE_VBAR, /* has cp15 VBAR */
>> +    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */
> 
> Missing space after '/*' ?
> 
>>  };
>>
>>  static inline int arm_feature(CPUARMState *env, int feature)
>> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
>> index a16abc8..0781367 100644
>> --- a/target/arm/kvm64.c
>> +++ b/target/arm/kvm64.c
>> @@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>          unset_feature(&env->features, ARM_FEATURE_PMU);
>>      }
>>
>> +    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
>> +        cpu->has_ras_extension = true;
>> +        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
>> +    } else {
>> +        cpu->has_ras_extension = false;
>> +        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
>> +    }
>> +
> 
> Shouldn't we need to also tell the kernel that we actually want
> it to expose RAS to the guest? Compare the PMU code in this
> function, where we set a kvm_init_features bit to do this.
> (This suggests that your ABI for the kernel part of this feature
> may not be correct?)
> 
> You should also not be calling set_feature() here -- if the
> CPU features bit doesn't say "this CPU should have the RAS
> extensions" we shouldn't create a CPU with them. Instead
> you should set it in kvm_arm_get_host_cpu_features() (again,
> compare the PMU code).
> 
> thanks
> -- PMM
> 
> .
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
@ 2017-09-06  9:35       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-09-06  9:35 UTC (permalink / raw)
  To: Peter Maydell
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm

Hi Peter,
  Thanks very much for your review, I will check your comments in detail and reply.


On 2017/9/6 1:26, Peter Maydell wrote:
> On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
>> check if kvm supports guest RAS EXTENSION. if so, set
>> corresponding feature bit for vcpu.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  linux-headers/linux/kvm.h | 1 +
>>  target/arm/cpu.h          | 3 +++
>>  target/arm/kvm64.c        | 8 ++++++++
>>  3 files changed, 12 insertions(+)
>>
>> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
>> index 7971a4f..2aa176e 100644
>> --- a/linux-headers/linux/kvm.h
>> +++ b/linux-headers/linux/kvm.h
>> @@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
>>  #define KVM_CAP_PPC_SMT_POSSIBLE 147
>>  #define KVM_CAP_HYPERV_SYNIC2 148
>>  #define KVM_CAP_HYPERV_VP_INDEX 149
>> +#define KVM_CAP_ARM_RAS_EXTENSION 150
>>
>>  #ifdef KVM_CAP_IRQ_ROUTING
>>
> 
> Hi. Changes to linux-headers need to be done as a patch of their
> own created using scripts/update-linux-headers.sh run against a
> mainline kernel tree (and with a commit message that quotes the
> kernel commit hash used). This ensures that we have a consistent
> set of headers that don't diverge from the kernel copy.
> 
>> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
>> index b39d64a..6b0961b 100644
>> --- a/target/arm/cpu.h
>> +++ b/target/arm/cpu.h
>> @@ -611,6 +611,8 @@ struct ARMCPU {
>>
>>      /* CPU has memory protection unit */
>>      bool has_mpu;
>> +    /* CPU has ras extension unit */
>> +    bool has_ras_extension;
>>      /* PMSAv7 MPU number of supported regions */
>>      uint32_t pmsav7_dregion;
>>
>> @@ -1229,6 +1231,7 @@ enum arm_features {
>>      ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
>>      ARM_FEATURE_PMU, /* has PMU support */
>>      ARM_FEATURE_VBAR, /* has cp15 VBAR */
>> +    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */
> 
> Missing space after '/*' ?
> 
>>  };
>>
>>  static inline int arm_feature(CPUARMState *env, int feature)
>> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
>> index a16abc8..0781367 100644
>> --- a/target/arm/kvm64.c
>> +++ b/target/arm/kvm64.c
>> @@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>          unset_feature(&env->features, ARM_FEATURE_PMU);
>>      }
>>
>> +    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
>> +        cpu->has_ras_extension = true;
>> +        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
>> +    } else {
>> +        cpu->has_ras_extension = false;
>> +        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
>> +    }
>> +
> 
> Shouldn't we need to also tell the kernel that we actually want
> it to expose RAS to the guest? Compare the PMU code in this
> function, where we set a kvm_init_features bit to do this.
> (This suggests that your ABI for the kernel part of this feature
> may not be correct?)
> 
> You should also not be calling set_feature() here -- if the
> CPU features bit doesn't say "this CPU should have the RAS
> extensions" we shouldn't create a CPU with them. Instead
> you should set it in kvm_arm_get_host_cpu_features() (again,
> compare the PMU code).
> 
> thanks
> -- PMM
> 
> .
> 


^ permalink raw reply	[flat|nested] 129+ messages in thread

* Re: [Qemu-devel] [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
@ 2017-09-06  9:35       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-09-06  9:35 UTC (permalink / raw)
  To: Peter Maydell
  Cc: Michael S. Tsirkin, Igor Mammedov, Shannon Zhao, Paolo Bonzini,
	QEMU Developers, qemu-arm, kvm-devel, edk2-devel,
	Christoffer Dall, Marc Zyngier, Will Deacon, James Morse,
	Tyler Baicar, Ard Biesheuvel, Ingo Molnar, bp, shiju.jose,
	zjzhang, arm-mail-list, kvmarm, lkml - Kernel Mailing List,
	linux-acpi, devel, john.garry, jonathan.cameron,
	shameerali.kolothum.thodi, huangdaode, wangzhou1, Huangshaoyu,
	wuquanming, Linuxarm, zhengqiang10

Hi Peter,
  Thanks very much for your review, I will check your comments in detail and reply.


On 2017/9/6 1:26, Peter Maydell wrote:
> On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
>> check if kvm supports guest RAS EXTENSION. if so, set
>> corresponding feature bit for vcpu.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  linux-headers/linux/kvm.h | 1 +
>>  target/arm/cpu.h          | 3 +++
>>  target/arm/kvm64.c        | 8 ++++++++
>>  3 files changed, 12 insertions(+)
>>
>> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
>> index 7971a4f..2aa176e 100644
>> --- a/linux-headers/linux/kvm.h
>> +++ b/linux-headers/linux/kvm.h
>> @@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
>>  #define KVM_CAP_PPC_SMT_POSSIBLE 147
>>  #define KVM_CAP_HYPERV_SYNIC2 148
>>  #define KVM_CAP_HYPERV_VP_INDEX 149
>> +#define KVM_CAP_ARM_RAS_EXTENSION 150
>>
>>  #ifdef KVM_CAP_IRQ_ROUTING
>>
> 
> Hi. Changes to linux-headers need to be done as a patch of their
> own created using scripts/update-linux-headers.sh run against a
> mainline kernel tree (and with a commit message that quotes the
> kernel commit hash used). This ensures that we have a consistent
> set of headers that don't diverge from the kernel copy.
> 
>> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
>> index b39d64a..6b0961b 100644
>> --- a/target/arm/cpu.h
>> +++ b/target/arm/cpu.h
>> @@ -611,6 +611,8 @@ struct ARMCPU {
>>
>>      /* CPU has memory protection unit */
>>      bool has_mpu;
>> +    /* CPU has ras extension unit */
>> +    bool has_ras_extension;
>>      /* PMSAv7 MPU number of supported regions */
>>      uint32_t pmsav7_dregion;
>>
>> @@ -1229,6 +1231,7 @@ enum arm_features {
>>      ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
>>      ARM_FEATURE_PMU, /* has PMU support */
>>      ARM_FEATURE_VBAR, /* has cp15 VBAR */
>> +    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */
> 
> Missing space after '/*' ?
> 
>>  };
>>
>>  static inline int arm_feature(CPUARMState *env, int feature)
>> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
>> index a16abc8..0781367 100644
>> --- a/target/arm/kvm64.c
>> +++ b/target/arm/kvm64.c
>> @@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>          unset_feature(&env->features, ARM_FEATURE_PMU);
>>      }
>>
>> +    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
>> +        cpu->has_ras_extension = true;
>> +        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
>> +    } else {
>> +        cpu->has_ras_extension = false;
>> +        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
>> +    }
>> +
> 
> Shouldn't we need to also tell the kernel that we actually want
> it to expose RAS to the guest? Compare the PMU code in this
> function, where we set a kvm_init_features bit to do this.
> (This suggests that your ABI for the kernel part of this feature
> may not be correct?)
> 
> You should also not be calling set_feature() here -- if the
> CPU features bit doesn't say "this CPU should have the RAS
> extensions" we shouldn't create a CPU with them. Instead
> you should set it in kvm_arm_get_host_cpu_features() (again,
> compare the PMU code).
> 
> thanks
> -- PMM
> 
> .
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

* [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature
@ 2017-09-06  9:35       ` gengdongjiu
  0 siblings, 0 replies; 129+ messages in thread
From: gengdongjiu @ 2017-09-06  9:35 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Peter,
  Thanks very much for your review, I will check your comments in detail and reply.


On 2017/9/6 1:26, Peter Maydell wrote:
> On 18 August 2017 at 15:23, Dongjiu Geng <gengdongjiu@huawei.com> wrote:
>> check if kvm supports guest RAS EXTENSION. if so, set
>> corresponding feature bit for vcpu.
>>
>> Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com>
>> ---
>>  linux-headers/linux/kvm.h | 1 +
>>  target/arm/cpu.h          | 3 +++
>>  target/arm/kvm64.c        | 8 ++++++++
>>  3 files changed, 12 insertions(+)
>>
>> diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
>> index 7971a4f..2aa176e 100644
>> --- a/linux-headers/linux/kvm.h
>> +++ b/linux-headers/linux/kvm.h
>> @@ -929,6 +929,7 @@ struct kvm_ppc_resize_hpt {
>>  #define KVM_CAP_PPC_SMT_POSSIBLE 147
>>  #define KVM_CAP_HYPERV_SYNIC2 148
>>  #define KVM_CAP_HYPERV_VP_INDEX 149
>> +#define KVM_CAP_ARM_RAS_EXTENSION 150
>>
>>  #ifdef KVM_CAP_IRQ_ROUTING
>>
> 
> Hi. Changes to linux-headers need to be done as a patch of their
> own created using scripts/update-linux-headers.sh run against a
> mainline kernel tree (and with a commit message that quotes the
> kernel commit hash used). This ensures that we have a consistent
> set of headers that don't diverge from the kernel copy.
> 
>> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
>> index b39d64a..6b0961b 100644
>> --- a/target/arm/cpu.h
>> +++ b/target/arm/cpu.h
>> @@ -611,6 +611,8 @@ struct ARMCPU {
>>
>>      /* CPU has memory protection unit */
>>      bool has_mpu;
>> +    /* CPU has ras extension unit */
>> +    bool has_ras_extension;
>>      /* PMSAv7 MPU number of supported regions */
>>      uint32_t pmsav7_dregion;
>>
>> @@ -1229,6 +1231,7 @@ enum arm_features {
>>      ARM_FEATURE_THUMB_DSP, /* DSP insns supported in the Thumb encodings */
>>      ARM_FEATURE_PMU, /* has PMU support */
>>      ARM_FEATURE_VBAR, /* has cp15 VBAR */
>> +    ARM_FEATURE_RAS_EXTENSION, /*has RAS extension support */
> 
> Missing space after '/*' ?
> 
>>  };
>>
>>  static inline int arm_feature(CPUARMState *env, int feature)
>> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
>> index a16abc8..0781367 100644
>> --- a/target/arm/kvm64.c
>> +++ b/target/arm/kvm64.c
>> @@ -518,6 +518,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>          unset_feature(&env->features, ARM_FEATURE_PMU);
>>      }
>>
>> +    if (kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_RAS_EXTENSION)) {
>> +        cpu->has_ras_extension = true;
>> +        set_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
>> +    } else {
>> +        cpu->has_ras_extension = false;
>> +        unset_feature(&env->features, ARM_FEATURE_RAS_EXTENSION);
>> +    }
>> +
> 
> Shouldn't we need to also tell the kernel that we actually want
> it to expose RAS to the guest? Compare the PMU code in this
> function, where we set a kvm_init_features bit to do this.
> (This suggests that your ABI for the kernel part of this feature
> may not be correct?)
> 
> You should also not be calling set_feature() here -- if the
> CPU features bit doesn't say "this CPU should have the RAS
> extensions" we shouldn't create a CPU with them. Instead
> you should set it in kvm_arm_get_host_cpu_features() (again,
> compare the PMU code).
> 
> thanks
> -- PMM
> 
> .
> 

^ permalink raw reply	[flat|nested] 129+ messages in thread

end of thread, other threads:[~2017-09-06  9:36 UTC | newest]

Thread overview: 129+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-18 14:23 [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI Dongjiu Geng
2017-08-18 14:23 ` Dongjiu Geng
2017-08-18 14:23 ` [Qemu-devel] " Dongjiu Geng
2017-08-18 14:23 ` Dongjiu Geng
2017-08-18 14:21 ` [Qemu-devel] " no-reply
2017-08-18 14:21   ` no-reply
2017-08-18 14:21   ` no-reply
2017-08-18 14:23 ` [PATCH v11 1/6] ACPI: add APEI/HEST/CPER structures and macros Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-08-18 14:23   ` [Qemu-devel] " Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-08-18 17:18   ` 答复: " gengdongjiu
2017-08-18 17:18   ` gengdongjiu
2017-08-18 17:18     ` gengdongjiu
2017-08-24 12:33   ` Shannon Zhao
2017-08-24 12:33     ` Shannon Zhao
2017-08-24 12:33     ` [Qemu-devel] " Shannon Zhao
2017-08-24 12:33     ` Shannon Zhao
2017-08-25 10:37     ` gengdongjiu
2017-08-25 10:37       ` gengdongjiu
2017-08-25 10:37       ` [Qemu-devel] " gengdongjiu
2017-08-25 10:37       ` gengdongjiu
2017-08-25 10:37       ` gengdongjiu
2017-08-26  1:00       ` Shannon Zhao
2017-08-26  1:00         ` Shannon Zhao
2017-08-26  1:00         ` [Qemu-devel] " Shannon Zhao
2017-08-26  1:00         ` Shannon Zhao
2017-08-26  1:00         ` Shannon Zhao
2017-08-26  1:45         ` gengdongjiu
2017-08-26  1:45           ` gengdongjiu
2017-08-26  1:45           ` [Qemu-devel] " gengdongjiu
2017-08-26  1:45           ` gengdongjiu
2017-08-26  1:45           ` gengdongjiu
2017-08-18 14:23 ` [PATCH v11 2/6] ACPI: Add APEI GHES Table Generation support Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-08-18 14:23   ` [Qemu-devel] " Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-08-18 17:19   ` 答复: " gengdongjiu
2017-08-18 17:19     ` gengdongjiu
2017-08-18 17:19     ` gengdongjiu
2017-08-24 13:03   ` Shannon Zhao
2017-08-24 13:03     ` Shannon Zhao
2017-08-24 13:03     ` [Qemu-devel] " Shannon Zhao
2017-08-24 13:03     ` Shannon Zhao
2017-08-25 11:20     ` gengdongjiu
2017-08-25 11:20       ` gengdongjiu
2017-08-25 11:20       ` [Qemu-devel] " gengdongjiu
2017-08-25 11:20       ` gengdongjiu
2017-08-25 11:20       ` gengdongjiu
2017-08-26  1:08       ` Shannon Zhao
2017-08-26  1:08         ` Shannon Zhao
2017-08-26  1:08         ` [Qemu-devel] " Shannon Zhao
2017-08-26  1:08         ` Shannon Zhao
2017-08-26  1:08         ` Shannon Zhao
2017-08-26  2:49         ` gengdongjiu
2017-08-26  2:49           ` gengdongjiu
2017-08-26  2:49           ` [Qemu-devel] " gengdongjiu
2017-08-26  2:49           ` gengdongjiu
2017-08-26  2:49           ` gengdongjiu
2017-08-29 10:20   ` [Qemu-devel] " Igor Mammedov
2017-08-29 10:20     ` Igor Mammedov
2017-08-29 10:20     ` Igor Mammedov
2017-08-29 10:20     ` Igor Mammedov
2017-08-29 10:20     ` Igor Mammedov
2017-08-29 11:15     ` gengdongjiu
2017-08-29 11:15       ` gengdongjiu
2017-08-29 11:15       ` gengdongjiu
2017-08-29 11:15       ` gengdongjiu
2017-08-29 11:15       ` gengdongjiu
2017-09-01  9:58     ` gengdongjiu
2017-09-01  9:58       ` gengdongjiu
2017-09-01  9:58       ` gengdongjiu
2017-09-01  9:58       ` gengdongjiu
2017-09-01  9:58       ` gengdongjiu
2017-09-01 11:51       ` Igor Mammedov
2017-09-01 11:51         ` Igor Mammedov
2017-09-01 11:51         ` Igor Mammedov
2017-09-01 11:51         ` Igor Mammedov
2017-08-18 14:23 ` [PATCH v11 3/6] ACPI: build and enable APEI GHES in the Makefile and configuration Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-08-18 14:23   ` [Qemu-devel] " Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-08-24 13:04   ` Shannon Zhao
2017-08-24 13:04     ` Shannon Zhao
2017-08-24 13:04     ` [Qemu-devel] " Shannon Zhao
2017-08-24 13:04     ` Shannon Zhao
2017-08-25 11:20     ` gengdongjiu
2017-08-25 11:20       ` gengdongjiu
2017-08-25 11:20       ` [Qemu-devel] " gengdongjiu
2017-08-25 11:20       ` gengdongjiu
2017-08-25 11:20       ` gengdongjiu
2017-08-18 14:23 ` [PATCH v11 4/6] target-arm: kvm64: detect guest RAS EXTENSION feature Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-08-18 14:23   ` [Qemu-devel] " Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-09-05 17:26   ` Peter Maydell
2017-09-05 17:26     ` Peter Maydell
2017-09-05 17:26     ` [Qemu-devel] " Peter Maydell
2017-09-05 17:26     ` Peter Maydell
2017-09-05 17:26     ` Peter Maydell
2017-09-06  9:35     ` gengdongjiu
2017-09-06  9:35       ` gengdongjiu
2017-09-06  9:35       ` [Qemu-devel] " gengdongjiu
2017-09-06  9:35       ` gengdongjiu
2017-09-06  9:35       ` gengdongjiu
2017-08-18 14:23 ` [PATCH v11 5/6] target-arm: kvm64: handle SIGBUS signal for synchronous External Abort Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-08-18 14:23   ` [Qemu-devel] " Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-09-05 17:46   ` Peter Maydell
2017-09-05 17:46     ` Peter Maydell
2017-09-05 17:46     ` [Qemu-devel] " Peter Maydell
2017-09-05 17:46     ` Peter Maydell
2017-09-05 17:46     ` Peter Maydell
2017-08-18 14:23 ` [PATCH v11 6/6] target-arm: kvm64: Handle SError interrupt for the guest OS Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-08-18 14:23   ` [Qemu-devel] " Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-08-18 14:23   ` Dongjiu Geng
2017-09-05 17:50   ` Peter Maydell
2017-09-05 17:50     ` Peter Maydell
2017-09-05 17:50     ` [Qemu-devel] " Peter Maydell
2017-09-05 17:50     ` Peter Maydell
2017-09-05 17:50     ` Peter Maydell
2017-08-18 17:17 ` 答复: [PATCH v11 0/6] Add RAS virtualization support for armv8 SEA and SEI gengdongjiu
2017-08-18 17:17   ` gengdongjiu
2017-08-18 17:17 ` gengdongjiu

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.