[Qemu-devel] [Bug 1701798] Re: dynamically linked binaries crash for big-endian targets

[Bruno Haible <bruno@clisp.org>]

Re #4:
>  2) files like /etc/ld.so.cache (and other things the dynamic linker uses) are not in the -L directory but are in the host
>  3) the ld.so.cache format is not endian-agnostic
>  4) glibc's dynamic linker code does not ignore a wrong-endian ld.so.cache but crashes instead

Indeed the problem is with /etc/ld.so.cache, and ONLY /etc/ld.so.cache.
When I hack the do_openat function in linux-user/syscall.c, to pretend
that no /etc/ld.so.cache exists, - see attached hide-ld.so.cache.diff -
the dynamically-linked binaries work (except the s390x case, which you
identified as a different issue):

$ QEMU_LD_PREFIX=/usr/sparc64-linux-gnu ~/inst-qemu/2.10+-20171107/bin/qemu-sparc64 hello.sparc64
Hello world
$ QEMU_LD_PREFIX=/usr/mips-linux-gnu ~/inst-qemu/2.10+-20171107/bin/qemu-mips hello.mips
Hello world
$ QEMU_LD_PREFIX=/usr/mips64-linux-gnuabi64 ~/inst-qemu/2.10+-20171107/bin/qemu-mips64 hello.mips64
Hello world
$ QEMU_LD_PREFIX=/usr/powerpc-linux-gnu ~/inst-qemu/2.10+-20171107/bin/qemu-ppc hello.powerpc
Hello world
$ QEMU_LD_PREFIX=/usr/powerpc64-linux-gnu ~/inst-qemu/2.10+-20171107/bin/qemu-ppc64 hello.powerpc64
Hello world
$ QEMU_LD_PREFIX=/usr/s390x-linux-gnu ~/inst-qemu/2.10+-20171107/bin/qemu-s390x hello.s390x
Killed
$ QEMU_LD_PREFIX=/usr/hppa-linux-gnu ~/inst-qemu/2.10+-20171107/bin/qemu-hppa hello.hppa
Hello world
$ QEMU_LD_PREFIX=/usr/m68k-linux-gnu QEMU_CPU=m68020 ~/inst-qemu/2.10+-20171107/bin/qemu-m68k hello.m68k
Hello world

Hurray! This bug that has seriously limited the value of linux-user
emulation is now gone for me!

> Can you check whether these work if you copy the QEMU and the
dynamically linked target binary into a chroot

This is way too cumbersome for me:
  - Need to copy my workspaces into specific file locations on the disk,
  - Need to use 'chroot' command before anything else,
  - Need to use a statically-linked qemu.

> Personally I think that (4) is a glibc bug...

Maybe, but if you can fix it in 5 to 10 lines code in qemu, I doubt it's
worth reporting it to the glibc people.

Small improvement: In my hack, I just pretended /etc/ld.so.cache is
absent. Possibly it's better to map it to
$QEMU_LD_PREFIX/etc/ld.so.cache .

** Patch added: "hide-ld.so.cache.diff"
   https://bugs.launchpad.net/qemu/+bug/1701798/+attachment/5005583/+files/hide-ld.so.cache.diff

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1701798

Title:
  dynamically linked binaries crash for big-endian targets

Status in QEMU:
  New

Bug description:
  On the targets
    hppa
    m68k
    mips
    mips64
    powerpc
    powerpc64
    s390x
    sparc64
  dynamically linked binaries crash, but statically linked binaries work.
  On the targets
    aarch64
    alpha
    armhf
    powerpc64le
    sh4
  both dynamically linked and statically linked binaries work.

  How to reproduce:

  1) On Ubuntu 16.04, install the packages
  g++-5-aarch64-linux-gnu
  g++-5-alpha-linux-gnu
  g++-5-arm-linux-gnueabihf
  g++-5-hppa-linux-gnu
  g++-5-m68k-linux-gnu
  g++-5-mips-linux-gnu
  g++-5-mips64-linux-gnuabi64
  g++-5-powerpc-linux-gnu
  g++-5-powerpc64-linux-gnu
  g++-5-powerpc64le-linux-gnu
  g++-5-s390x-linux-gnu
  g++-5-sh4-linux-gnu
  g++-5-sparc64-linux-gnu

  2) Install qemu 2.9.0 from source (for m68k, use the 2.7.0-m68k
  code from https://github.com/vivier/qemu-m68k.git):
  $ ../configure --prefix=/home/bruno/inst-qemu/2.9.0 --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,i386-softmmu,m68k-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sh4-softmmu,sparc-softmmu,sparc64-softmmu,x86_64-softmmu,aarch64-linux-user,alpha-linux-user,arm-linux-user,hppa-linux-user,m68k-linux-user,mips-linux-user,mipsel-linux-user,mips64-linux-user,mips64el-linux-user,ppc-linux-user,ppc64-linux-user,ppc64le-linux-user,s390x-linux-user,sh4-linux-user,sparc-linux-user,sparc64-linux-user --disable-strip --disable-werror --enable-gtk --enable-vnc
  $ make
  $ make install

  3) Cross-compile the programs:

  $ aarch64-linux-gnu-gcc-5 -O hello.c -o hello.aarch64
  $ alpha-linux-gnu-gcc-5 -O hello.c -o hello.alpha
  $ arm-linux-gnueabihf-gcc-5 -O hello.c -o hello.armhf
  $ hppa-linux-gnu-gcc-5 -O hello.c -o hello.hppa
  $ m68k-linux-gnu-gcc-5 -O hello.c -o hello.m68k
  $ mips-linux-gnu-gcc-5 -O hello.c -o hello.mips
  $ mips64-linux-gnuabi64-gcc-5 -O hello.c -o hello.mips64
  $ powerpc-linux-gnu-gcc-5 -O hello.c -o hello.powerpc
  $ powerpc64-linux-gnu-gcc-5 -O hello.c -o hello.powerpc64
  $ powerpc64le-linux-gnu-gcc-5 -O hello.c -o hello.powerpc64le
  $ s390x-linux-gnu-gcc-5 -O hello.c -o hello.s390x
  $ sh4-linux-gnu-gcc-5 -O hello.c -o hello.sh4
  $ sparc64-linux-gnu-gcc-5 -O hello.c -o hello.sparc64

  4) Run the programs:

  * aarch64 works:
  $ QEMU_LD_PREFIX=/usr/aarch64-linux-gnu ~/inst-qemu/2.9.0/bin/qemu-aarch64 hello.aarch64
  Hello world

  * alpha works:
  $ QEMU_LD_PREFIX=/usr/alpha-linux-gnu ~/inst-qemu/2.9.0/bin/qemu-alpha hello.alpha 
  Hello world

  * armhf works:
  $ QEMU_LD_PREFIX=/usr/arm-linux-gnueabihf ~/inst-qemu/2.9.0/bin/qemu-arm hello.armhf
  Hello world

  * powerpc64le works:
  $ QEMU_LD_PREFIX=/usr/powerpc64le-linux-gnu ~/inst-qemu/2.9.0/bin/qemu-ppc64le hello.powerpc64le
  Hello world

  * sh4 works:
  $ QEMU_LD_PREFIX=/usr/sh4-linux-gnu ~/inst-qemu/2.9.0/bin/qemu-sh4 hello.sh4
  Hello world

  * ===== sparc64 does not work:
  $ QEMU_LD_PREFIX=/usr/sparc64-linux-gnu ~/inst-qemu/2.9.0/bin/qemu-sparc64 hello.sparc64
  Segmentation fault (core dumped)

  When I copy the file to a machine with `uname -srm` = "Linux 4.5.0-2-sparc64 sparc64",
  it works:
  $ ./hello.sparc64
  Hello world

  When I copy the file and its execution environment /usr/sparc64-linux-gnu to the
  same machine and run the binary in a chroot environment:
  # /bin/hello.sparc64 
  Hello world

  * ===== mips does not work:
  $ QEMU_LD_PREFIX=/usr/mips-linux-gnu ~/inst-qemu/2.9.0/bin/qemu-mips hello.mips
  qemu: uncaught target signal 11 (Segmentation fault) - core dumped

  When I copy the file to a machine with `uname -srm` = "Linux 3.16.0-4-4kc-malta mips",
  it works:
  $ ./hello.mips
  Hello world

  When I copy the file and its execution environment /usr/mips-linux-gnu to the
  same machine and run the binary in a chroot environment:
  # /bin/hello.mips 
  Hello world

  * ===== mips64 does not work:
  $ QEMU_LD_PREFIX=/usr/mips64-linux-gnuabi64 ~/inst-qemu/2.9.0/bin/qemu-mips64 hello.mips64
  qemu: uncaught target signal 11 (Segmentation fault) - core dumped

  When I copy the file to a machine with `uname -srm` = "Linux 3.16.0-4-5kc-malta mips64",
  it works:
  $ ./hello.mips64
  Hello world

  * ===== powerpc does not work:
  $ QEMU_LD_PREFIX=/usr/powerpc-linux-gnu ~/inst-qemu/2.9.0/bin/qemu-ppc hello.powerpc
  qemu: uncaught target signal 11 (Segmentation fault) - core dumped

  When I copy the file to a machine with `uname -srm` = "Linux 3.17.2-200.fc20.ppc64p7 ppc64",
  it works:
  $ ./hello.powerpc
  Hello world

  * ===== powerpc64 does not work:
  $ QEMU_LD_PREFIX=/usr/powerpc64-linux-gnu ~/inst-qemu/2.9.0/bin/qemu-ppc64 hello.powerpc64
  qemu: uncaught target signal 11 (Segmentation fault) - core dumped

  When I copy the file to a machine with `uname -srm` = "Linux 3.17.2-200.fc20.ppc64p7 ppc64",
  it works:
  $ ./hello.powerpc64
  Hello world

  * ===== s390x does not work:
  $ QEMU_LD_PREFIX=/usr/s390x-linux-gnu ~/inst-qemu/2.9.0/bin/qemu-s390x hello.s390x
  <hangs>
  $ QEMU_LD_PREFIX=/usr/s390x-linux-gnu ~/inst-qemu/2.8.1/bin/qemu-s390x hello.s390x
  qemu-s390x: /media/develdata/devel/build/qemu-2.8.1/translate-all.c:175: tb_lock: Assertion `!have_tb_lock' failed.
  Segmentation fault (core dumped)

  When I copy the file to a machine with `uname -srm` = "Linux 3.16.0-4-s390x s390x",
  it works:
  $ ./hello.s390x
  Hello world

  * ===== hppa does not work:
  $ QEMU_LD_PREFIX=/usr/hppa-linux-gnu ~/inst-qemu/2.9.0/bin/qemu-hppa hello.hppa
  Segmentation fault (core dumped)

  * ===== m68k does not work:
  $ QEMU_LD_PREFIX=/usr/m68k-linux-gnu QEMU_CPU=m68020 ~/inst-qemu/2.9.0/bin/qemu-m68k hello.m68k
  qemu: uncaught target signal 4 (Illegal instruction) - core dumped
  $ QEMU_LD_PREFIX=/usr/m68k-linux-gnu QEMU_CPU=m68020 ~/inst-qemu/2.7.0-m68k/bin/qemu-m68k hello.m68k
  qemu: uncaught target signal 11 (Segmentation fault) - core dumped

  
  The set of targets where it does not work is exactly the big-endian targets.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1701798/+subscriptions