[PULL v2 00/11] capstone + disassembler patch queue

[PULL v2 00/11] capstone + disassembler patch queue

[Richard Henderson]

Version 2 retains a dummy capstone/all makefile target, to avoid
the build failure that Peter saw.


r~


The following changes since commit dd8c1e808f1ca311e1f50bff218c3ee3198b1f02:

  Merge remote-tracking branch 'remotes/cohuck/tags/s390x-20201002' into staging (2020-10-02 14:29:49 +0100)

are available in the Git repository at:

  https://gitlab.com/rth7680/qemu.git tags/pull-cap-20201002

for you to fetch changes up to 94816249a1e14f90f56a2f6e1d566e959e9bc26d:

  disas/capstone: Add skipdata hook for s390x (2020-10-02 11:05:07 -0500)

----------------------------------------------------------------
Update capstone submodule from v3.0.5 to v5 ("next").
Convert submodule build to meson.
Enable capstone disassembly for s390x.
Code cleanups in disas.c

----------------------------------------------------------------
Richard Henderson (11):
      capstone: Convert Makefile bits to meson bits
      capstone: Update to upstream "next" branch
      capstone: Require version 4.0 from a system library
      disas: Move host asm annotations to tb_gen_code
      disas: Clean up CPUDebug initialization
      disas: Use qemu/bswap.h for bfd endian loads
      disas: Cleanup plugin_disas
      disas: Configure capstone for aarch64 host without libvixl
      disas: Split out capstone code to disas/capstone.c
      disas: Enable capstone disassembly for s390x
      disas/capstone: Add skipdata hook for s390x

 configure                 |  68 +----
 Makefile                  |  18 +-
 meson.build               | 124 +++++++-
 include/disas/dis-asm.h   | 104 +++----
 include/disas/disas.h     |   2 +-
 include/exec/log.h        |   4 +-
 accel/tcg/translate-all.c |  24 +-
 disas.c                   | 707 +++++++++++-----------------------------------
 disas/capstone.c          | 326 +++++++++++++++++++++
 target/s390x/cpu.c        |   4 +
 tcg/tcg.c                 |   4 +-
 capstone                  |   2 +-
 disas/meson.build         |   1 +
 meson_options.txt         |   4 +
 14 files changed, 687 insertions(+), 705 deletions(-)
 create mode 100644 disas/capstone.c

[PULL v2 01/11] capstone: Convert Makefile bits to meson bits

[Richard Henderson]

There are better ways to do this, e.g. meson cmake subproject,
but that requires cmake 3.7 and some of our CI environments
only provide cmake 3.5.

Nor can we add a meson.build file to capstone/, because the git
submodule would then always report "untracked files".  Fixing that
would require creating our own branch on the qemu git mirror, at
which point we could just as easily create a native meson subproject.

Instead, build the library via the main meson.build.

This improves the current state of affairs in that we will re-link
the qemu executables against a changed libcapstone.a, which we wouldn't
do before-hand.  In addition, the use of the configuration header file
instead of command-line -DEFINES means that we will rebuild the
capstone objects with changes to meson.build.

Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 configure         |  68 ++++------------------------
 Makefile          |  18 ++------
 meson.build       | 111 +++++++++++++++++++++++++++++++++++++++++++---
 meson_options.txt |   4 ++
 4 files changed, 121 insertions(+), 80 deletions(-)

diff --git a/configure b/configure
index a5841241be..f46f433649 100755
--- a/configure
+++ b/configure
@@ -478,7 +478,7 @@ opengl=""
 opengl_dmabuf="no"
 cpuid_h="no"
 avx2_opt=""
-capstone=""
+capstone="auto"
 lzo=""
 snappy=""
 bzip2=""
@@ -1575,11 +1575,11 @@ for opt do
   ;;
   --enable-vhost-kernel) vhost_kernel="yes"
   ;;
-  --disable-capstone) capstone="no"
+  --disable-capstone) capstone="disabled"
   ;;
-  --enable-capstone) capstone="yes"
+  --enable-capstone) capstone="enabled"
   ;;
-  --enable-capstone=git) capstone="git"
+  --enable-capstone=git) capstone="internal"
   ;;
   --enable-capstone=system) capstone="system"
   ;;
@@ -5017,51 +5017,11 @@ fi
 # capstone
 
 case "$capstone" in
-  "" | yes)
-    if $pkg_config capstone; then
-      capstone=system
-    elif test -e "${source_path}/.git" && test $git_update = 'yes' ; then
-      capstone=git
-    elif test -e "${source_path}/capstone/Makefile" ; then
-      capstone=internal
-    elif test -z "$capstone" ; then
-      capstone=no
-    else
-      feature_not_found "capstone" "Install capstone devel or git submodule"
-    fi
-    ;;
-
-  system)
-    if ! $pkg_config capstone; then
-      feature_not_found "capstone" "Install capstone devel"
-    fi
-    ;;
-esac
-
-case "$capstone" in
-  git | internal)
-    if test "$capstone" = git; then
+  auto | enabled | internal)
+    # Simpler to always update submodule, even if not needed.
+    if test -e "${source_path}/.git" && test $git_update = 'yes' ; then
       git_submodules="${git_submodules} capstone"
     fi
-    mkdir -p capstone
-    if test "$mingw32" = "yes"; then
-      LIBCAPSTONE=capstone.lib
-    else
-      LIBCAPSTONE=libcapstone.a
-    fi
-    capstone_libs="-Lcapstone -lcapstone"
-    capstone_cflags="-I${source_path}/capstone/include"
-    ;;
-
-  system)
-    capstone_libs="$($pkg_config --libs capstone)"
-    capstone_cflags="$($pkg_config --cflags capstone)"
-    ;;
-
-  no)
-    ;;
-  *)
-    error_exit "Unknown state for capstone: $capstone"
     ;;
 esac
 
@@ -7142,11 +7102,6 @@ fi
 if test "$ivshmem" = "yes" ; then
   echo "CONFIG_IVSHMEM=y" >> $config_host_mak
 fi
-if test "$capstone" != "no" ; then
-  echo "CONFIG_CAPSTONE=y" >> $config_host_mak
-  echo "CAPSTONE_CFLAGS=$capstone_cflags" >> $config_host_mak
-  echo "CAPSTONE_LIBS=$capstone_libs" >> $config_host_mak
-fi
 if test "$debug_mutex" = "yes" ; then
   echo "CONFIG_DEBUG_MUTEX=y" >> $config_host_mak
 fi
@@ -7664,13 +7619,7 @@ done # for target in $targets
 if [ "$fdt" = "git" ]; then
   subdirs="$subdirs dtc"
 fi
-if [ "$capstone" = "git" -o "$capstone" = "internal" ]; then
-  subdirs="$subdirs capstone"
-fi
 echo "SUBDIRS=$subdirs" >> $config_host_mak
-if test -n "$LIBCAPSTONE"; then
-  echo "LIBCAPSTONE=$LIBCAPSTONE" >> $config_host_mak
-fi
 
 if test "$numa" = "yes"; then
   echo "CONFIG_NUMA=y" >> $config_host_mak
@@ -7846,7 +7795,8 @@ NINJA=${ninja:-$PWD/ninjatool} $meson setup \
 	-Dmalloc=$malloc -Dmalloc_trim=$malloc_trim \
 	-Dcocoa=$cocoa -Dmpath=$mpath -Dsdl=$sdl -Dsdl_image=$sdl_image \
 	-Dvnc=$vnc -Dvnc_sasl=$vnc_sasl -Dvnc_jpeg=$vnc_jpeg -Dvnc_png=$vnc_png \
-	-Dgettext=$gettext -Dxkbcommon=$xkbcommon -Du2f=$u2f\
+	-Dgettext=$gettext -Dxkbcommon=$xkbcommon -Du2f=$u2f \
+	-Dcapstone=$capstone \
         $cross_arg \
         "$PWD" "$source_path"
 
diff --git a/Makefile b/Makefile
index 54fc1a9d10..f27bd4b2eb 100644
--- a/Makefile
+++ b/Makefile
@@ -156,21 +156,11 @@ dtc/all: .git-submodule-status dtc/libfdt
 dtc/%: .git-submodule-status
 	@mkdir -p $@
 
-# Overriding CFLAGS causes us to lose defines added in the sub-makefile.
-# Not overriding CFLAGS leads to mis-matches between compilation modes.
-# Therefore we replicate some of the logic in the sub-makefile.
-# Remove all the extra -Warning flags that QEMU uses that Capstone doesn't;
-# no need to annoy QEMU developers with such things.
-CAP_CFLAGS = $(patsubst -W%,,$(CFLAGS) $(QEMU_CFLAGS)) $(CAPSTONE_CFLAGS)
-CAP_CFLAGS += -DCAPSTONE_USE_SYS_DYN_MEM
-CAP_CFLAGS += -DCAPSTONE_HAS_ARM
-CAP_CFLAGS += -DCAPSTONE_HAS_ARM64
-CAP_CFLAGS += -DCAPSTONE_HAS_POWERPC
-CAP_CFLAGS += -DCAPSTONE_HAS_X86
-
+# Retain for a while so that incremental build across this patch
+# does not raise an error for missing target "capstone/all", which
+# comes from the saved SUBDIRS value.
 .PHONY: capstone/all
-capstone/all: .git-submodule-status
-	$(call quiet-command,$(MAKE) -C $(SRC_PATH)/capstone CAPSTONE_SHARED=no BUILDDIR="$(BUILD_DIR)/capstone" CC="$(CC)" AR="$(AR)" LD="$(LD)" RANLIB="$(RANLIB)" CFLAGS="$(CAP_CFLAGS)" $(SUBDIR_MAKEFLAGS) $(BUILD_DIR)/capstone/$(LIBCAPSTONE))
+capstone/all:
 
 .PHONY: slirp/all
 slirp/all: .git-submodule-status
diff --git a/meson.build b/meson.build
index 3161c1f037..cd01da48ca 100644
--- a/meson.build
+++ b/meson.build
@@ -10,6 +10,7 @@ else
   keyval = import('unstable-keyval')
 endif
 ss = import('sourceset')
+fs = import('fs')
 
 sh = find_program('sh')
 cc = meson.get_compiler('c')
@@ -495,11 +496,6 @@ if 'CONFIG_USB_LIBUSB' in config_host
   libusb = declare_dependency(compile_args: config_host['LIBUSB_CFLAGS'].split(),
                               link_args: config_host['LIBUSB_LIBS'].split())
 endif
-capstone = not_found
-if 'CONFIG_CAPSTONE' in config_host
-  capstone = declare_dependency(compile_args: config_host['CAPSTONE_CFLAGS'].split(),
-                                link_args: config_host['CAPSTONE_LIBS'].split())
-endif
 libpmem = not_found
 if 'CONFIG_LIBPMEM' in config_host
   libpmem = declare_dependency(compile_args: config_host['LIBPMEM_CFLAGS'].split(),
@@ -581,7 +577,6 @@ foreach k, v: config_host
     config_host_data.set(k, v == 'y' ? 1 : v)
   endif
 endforeach
-genh += configure_file(output: 'config-host.h', configuration: config_host_data)
 
 minikconf = find_program('scripts/minikconf.py')
 config_all = {}
@@ -736,6 +731,108 @@ config_all += {
   'CONFIG_ALL': true,
 }
 
+# Submodules
+
+capstone = not_found
+capstone_opt = get_option('capstone')
+if capstone_opt == 'disabled'
+  capstone_opt = false
+elif capstone_opt in ['enabled', 'auto', 'system']
+  have_internal = fs.exists('capstone/Makefile')
+  capstone = dependency('capstone', static: enable_static,
+                        required: capstone_opt == 'system' or
+                                  capstone_opt == 'enabled' and not have_internal)
+  if capstone.found()
+    capstone_opt = 'system'
+  elif have_internal
+    capstone_opt = 'internal'
+  else
+    capstone_opt = false
+  endif
+endif
+if capstone_opt == 'internal'
+  capstone_data = configuration_data()
+  capstone_data.set('CAPSTONE_USE_SYS_DYN_MEM', '1')
+
+  capstone_files = files(
+    'capstone/cs.c',
+    'capstone/MCInst.c',
+    'capstone/MCInstrDesc.c',
+    'capstone/MCRegisterInfo.c',
+    'capstone/SStream.c',
+    'capstone/utils.c'
+  )
+
+  if 'CONFIG_ARM_DIS' in config_all_disas
+    capstone_data.set('CAPSTONE_HAS_ARM', '1')
+    capstone_files += files(
+      'capstone/arch/ARM/ARMDisassembler.c',
+      'capstone/arch/ARM/ARMInstPrinter.c',
+      'capstone/arch/ARM/ARMMapping.c',
+      'capstone/arch/ARM/ARMModule.c'
+    )
+  endif
+
+  # FIXME: This config entry currently depends on a c++ compiler.
+  # Which is needed for building libvixl, but not for capstone.
+  if 'CONFIG_ARM_A64_DIS' in config_all_disas
+    capstone_data.set('CAPSTONE_HAS_ARM64', '1')
+    capstone_files += files(
+      'capstone/arch/AArch64/AArch64BaseInfo.c',
+      'capstone/arch/AArch64/AArch64Disassembler.c',
+      'capstone/arch/AArch64/AArch64InstPrinter.c',
+      'capstone/arch/AArch64/AArch64Mapping.c',
+      'capstone/arch/AArch64/AArch64Module.c'
+    )
+  endif
+
+  if 'CONFIG_PPC_DIS' in config_all_disas
+    capstone_data.set('CAPSTONE_HAS_POWERPC', '1')
+    capstone_files += files(
+      'capstone/arch/PowerPC/PPCDisassembler.c',
+      'capstone/arch/PowerPC/PPCInstPrinter.c',
+      'capstone/arch/PowerPC/PPCMapping.c',
+      'capstone/arch/PowerPC/PPCModule.c'
+    )
+  endif
+
+  if 'CONFIG_I386_DIS' in config_all_disas
+    capstone_data.set('CAPSTONE_HAS_X86', 1)
+    capstone_files += files(
+      'capstone/arch/X86/X86Disassembler.c',
+      'capstone/arch/X86/X86DisassemblerDecoder.c',
+      'capstone/arch/X86/X86ATTInstPrinter.c',
+      'capstone/arch/X86/X86IntelInstPrinter.c',
+      'capstone/arch/X86/X86Mapping.c',
+      'capstone/arch/X86/X86Module.c'
+    )
+  endif
+
+  configure_file(output: 'capstone-defs.h', configuration: capstone_data)
+
+  capstone_cargs = [
+    # FIXME: There does not seem to be a way to completely replace the c_args
+    # that come from add_project_arguments() -- we can only add to them.
+    # So: disable all warnings with a big hammer.
+    '-Wno-error', '-w',
+
+    # Include all configuration defines via a header file, which will wind up
+    # as a dependency on the object file, and thus changes here will result
+    # in a rebuild.
+    '-include', 'capstone-defs.h'
+  ]
+
+  libcapstone = static_library('capstone',
+                               sources: capstone_files,
+                               c_args: capstone_cargs,
+                               include_directories: 'capstone/include')
+  capstone = declare_dependency(link_with: libcapstone,
+                                include_directories: 'capstone/include')
+endif
+config_host_data.set('CONFIG_CAPSTONE', capstone.found())
+
+genh += configure_file(output: 'config-host.h', configuration: config_host_data)
+
 # Generators
 
 hxtool = find_program('scripts/hxtool')
@@ -1566,7 +1663,7 @@ summary_info += {'vvfat support':     config_host.has_key('CONFIG_VVFAT')}
 summary_info += {'qed support':       config_host.has_key('CONFIG_QED')}
 summary_info += {'parallels support': config_host.has_key('CONFIG_PARALLELS')}
 summary_info += {'sheepdog support':  config_host.has_key('CONFIG_SHEEPDOG')}
-summary_info += {'capstone':          config_host.has_key('CONFIG_CAPSTONE')}
+summary_info += {'capstone':          capstone_opt}
 summary_info += {'libpmem support':   config_host.has_key('CONFIG_LIBPMEM')}
 summary_info += {'libdaxctl support': config_host.has_key('CONFIG_LIBDAXCTL')}
 summary_info += {'libudev':           libudev.found()}
diff --git a/meson_options.txt b/meson_options.txt
index 46ea1d889a..a1228d29a9 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -31,3 +31,7 @@ option('vnc_sasl', type : 'feature', value : 'auto',
        description: 'SASL authentication for VNC server')
 option('xkbcommon', type : 'feature', value : 'auto',
        description: 'xkbcommon support')
+
+option('capstone', type: 'combo', value: 'auto',
+       choices: ['disabled', 'enabled', 'auto', 'system', 'internal'],
+       description: 'Whether and how to find the capstone library')
-- 
2.25.1

Re: [PULL v2 00/11] capstone + disassembler patch queue

[Peter Maydell]

On Fri, 2 Oct 2020 at 17:51, Richard Henderson
<richard.henderson@linaro.org> wrote:
>
> Version 2 retains a dummy capstone/all makefile target, to avoid
> the build failure that Peter saw.
>
>
> r~
>
>
> The following changes since commit dd8c1e808f1ca311e1f50bff218c3ee3198b1f02:
>
>   Merge remote-tracking branch 'remotes/cohuck/tags/s390x-20201002' into staging (2020-10-02 14:29:49 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/rth7680/qemu.git tags/pull-cap-20201002
>
> for you to fetch changes up to 94816249a1e14f90f56a2f6e1d566e959e9bc26d:
>
>   disas/capstone: Add skipdata hook for s390x (2020-10-02 11:05:07 -0500)
>
> ----------------------------------------------------------------
> Update capstone submodule from v3.0.5 to v5 ("next").
> Convert submodule build to meson.
> Enable capstone disassembly for s390x.
> Code cleanups in disas.c
>

Meson warning on the BSDs:

Configuring sparc-bsd-user-config-target.h using configuration
Configuring sparc64-bsd-user-config-target.h using configuration
Configuring x86_64-bsd-user-config-target.h using configuration
Did not find CMake 'cmake'
Found CMake: NO
Run-time dependency capstone found: NO (tried pkgconfig and cmake)
../src/meson.build:753: WARNING: Trying to compare values of different
types (bool, str) using ==.
The result of this is undefined and will become a hard error in a
future Meson release.
Configuring config-host.h using configuration
Program scripts/hxtool found: YES
Program scripts/shaderinclude.pl found: YES
Program scripts/qapi-gen.py found: YES
Program scripts/qemu-version.sh found: YES
Run-time dependency threads found: YES
Program keycodemapdb/tools/keymap-gen found: YES
Program scripts/decodetree.py found: YES

Warning from ppc64be box (gcc compilefarm one):

Configuring sh4eb-linux-user-config-target.h using configuration
Configuring sparc-linux-user-config-target.h using configuration
Configuring sparc32plus-linux-user-config-target.h using configuration
Configuring sparc64-linux-user-config-target.h using configuration
Configuring x86_64-linux-user-config-target.h using configuration
Configuring xtensa-linux-user-config-target.h using configuration
Configuring xtensaeb-linux-user-config-target.h using configuration
Found CMake: /usr/bin/cmake (2.8.12.2)
WARNING: The version of CMake /usr/bin/cmake is 2.8.12.2 but version
>=3.4 is required
Run-time dependency capstone found: NO (tried pkgconfig and cmake)
Configuring capstone-defs.h using configuration
Configuring config-host.h using configuration

We shouldn't be looking for or using cmake at all.

thanks
-- PMM

Re: [PULL v2 00/11] capstone + disassembler patch queue

[Richard Henderson]

On 10/2/20 3:37 PM, Peter Maydell wrote:
> Meson warning on the BSDs:
> 
> Configuring sparc-bsd-user-config-target.h using configuration
> Configuring sparc64-bsd-user-config-target.h using configuration
> Configuring x86_64-bsd-user-config-target.h using configuration
> Did not find CMake 'cmake'
> Found CMake: NO
> Run-time dependency capstone found: NO (tried pkgconfig and cmake)
> ../src/meson.build:753: WARNING: Trying to compare values of different
> types (bool, str) using ==.
> The result of this is undefined and will become a hard error in a
> future Meson release.
> Configuring config-host.h using configuration
> Program scripts/hxtool found: YES
> Program scripts/shaderinclude.pl found: YES
> Program scripts/qapi-gen.py found: YES
> Program scripts/qemu-version.sh found: YES
> Run-time dependency threads found: YES
> Program keycodemapdb/tools/keymap-gen found: YES
> Program scripts/decodetree.py found: YES
> 
> Warning from ppc64be box (gcc compilefarm one):
> 
> Configuring sh4eb-linux-user-config-target.h using configuration
> Configuring sparc-linux-user-config-target.h using configuration
> Configuring sparc32plus-linux-user-config-target.h using configuration
> Configuring sparc64-linux-user-config-target.h using configuration
> Configuring x86_64-linux-user-config-target.h using configuration
> Configuring xtensa-linux-user-config-target.h using configuration
> Configuring xtensaeb-linux-user-config-target.h using configuration
> Found CMake: /usr/bin/cmake (2.8.12.2)
> WARNING: The version of CMake /usr/bin/cmake is 2.8.12.2 but version
>> =3.4 is required
> Run-time dependency capstone found: NO (tried pkgconfig and cmake)
> Configuring capstone-defs.h using configuration
> Configuring config-host.h using configuration
> 
> We shouldn't be looking for or using cmake at all.

Huh.  I hadn't noticed that before.
I think it's coming from meson internally, but I'm not sure what's causing.
Certainly it is not something I asked for.

So it's a warning.  Does the build succeed?


r~

Re: [PULL v2 00/11] capstone + disassembler patch queue

[Paolo Bonzini]

On 02/10/20 22:37, Peter Maydell wrote:
> Found CMake: /usr/bin/cmake (2.8.12.2)
> WARNING: The version of CMake /usr/bin/cmake is 2.8.12.2 but version
>> =3.4 is required
> Run-time dependency capstone found: NO (tried pkgconfig and cmake)
> Configuring capstone-defs.h using configuration
> Configuring config-host.h using configuration
> 
> We shouldn't be looking for or using cmake at all.

This is a missing "method: 'pkg-config'".

Paolo

Re: [PULL v2 00/11] capstone + disassembler patch queue

[Paolo Bonzini]

On 02/10/20 22:37, Peter Maydell wrote:
> ../src/meson.build:753: WARNING: Trying to compare values of different
> types (bool, str) using ==.
> The result of this is undefined and will become a hard error in a
> future Meson release.
> Configuring config-host.h using configuration
> Program scripts/hxtool found: YES
> Program scripts/shaderinclude.pl found: YES
> Program scripts/qapi-gen.py found: YES
> Program scripts/qemu-version.sh found: YES
> Run-time dependency threads found: YES
> Program keycodemapdb/tools/keymap-gen found: YES
> Program scripts/decodetree.py found: YES

This can be rewritten like

-if capstone_opt == 'disabled'
-  capstone_opt = false
-elif capstone_opt in ['enabled', 'auto', 'system']
+if capstone_opt in ['enabled', 'auto', 'system']
   have_internal = fs.exists('capstone/Makefile')
   capstone = dependency('capstone', static: enable_static,
                         required: capstone_opt == 'system' or
                                   capstone_opt == 'enabled' and not have_internal)
   if capstone.found()
     capstone_opt = 'system'
   elif have_internal
     capstone_opt = 'internal'
   else
-    capstone_opt = false
+    capstone_opt = 'disabled'
   endif
 endif
 if capstone_opt == 'internal'

...
-summary_info += {'capstone':          capstone_opt}
+summary_info += {'capstone':          capstone_opt == 'disabled' ? false : capstone_opt}


That said, this also showed a bug which can be fixed like this:

-  have_internal = fs.exists('capstone/Makefile')
+  have_internal = fs.exists(meson.current_source_dir() / 'capstone/Makefile')

Paolo

Re: [PULL v2 00/11] capstone + disassembler patch queue

[Richard Henderson]

On 10/3/20 2:06 AM, Paolo Bonzini wrote:
> On 02/10/20 22:37, Peter Maydell wrote:
>> Found CMake: /usr/bin/cmake (2.8.12.2)
>> WARNING: The version of CMake /usr/bin/cmake is 2.8.12.2 but version
>>> =3.4 is required
>> Run-time dependency capstone found: NO (tried pkgconfig and cmake)
>> Configuring capstone-defs.h using configuration
>> Configuring config-host.h using configuration
>>
>> We shouldn't be looking for or using cmake at all.
> 
> This is a missing "method: 'pkg-config'".

Ah, thanks.


r~