[Qemu-devel] [PATCH 0/3] OpenBSD fixes

[Qemu-devel] [PATCH 0/3] OpenBSD fixes

[Philippe Mathieu-Daudé]

Fixes I encountered while trying to run QEMU test suite on OpenBSD.
More to come, but please review.

Regards,

Phil.

Philippe Mathieu-Daudé (3):
  configure: Disable W^X on OpenBSD
  XXX oslib-posix: Ignore fcntl("/dev/null", F_SETFL, O_NONBLOCK)
    failure
  WIP tests/vm: Run tests on OpenBSD

 configure          | 11 +++++++++++
 tests/vm/openbsd   |  4 +---
 util/oslib-posix.c |  2 ++
 3 files changed, 14 insertions(+), 3 deletions(-)

-- 
2.20.1

[Qemu-devel] [PATCH 1/3] configure: Disable W^X on OpenBSD

[Philippe Mathieu-Daudé]

Since OpenBSD 6.0 [1], W^X is enforced by default [2].
TCG requires WX access. Disable W^X if it is available.
This fixes:

  # lm32-softmmu/qemu-system-lm32
  Could not allocate dynamic translator buffer

  # sysctl kern.wxabort=1
  kern.wxabort: 0 -> 1
  # lm32-softmmu/qemu-system-lm32
  mmap: Not supported
  Abort trap (core dumped)
  # gdb -q lm32-softmmu/qemu-system-lm32 qemu-system-lm32.core
  (gdb) bt
  #0  0x000017e3c156c50a in _thread_sys___syscall () at {standard input}:5
  #1  0x000017e3c15e5d7a in *_libc_mmap (addr=Variable "addr" is not available.) at /usr/src/lib/libc/sys/mmap.c:47
  #2  0x000017e17d9abc8b in alloc_code_gen_buffer () at /usr/src/qemu/accel/tcg/translate-all.c:1064
  #3  0x000017e17d9abd04 in code_gen_alloc (tb_size=0) at /usr/src/qemu/accel/tcg/translate-all.c:1112
  #4  0x000017e17d9abe81 in tcg_exec_init (tb_size=0) at /usr/src/qemu/accel/tcg/translate-all.c:1149
  #5  0x000017e17d9897e9 in tcg_init (ms=0x17e45e456800) at /usr/src/qemu/accel/tcg/tcg-all.c:66
  #6  0x000017e17d9891b8 in accel_init_machine (acc=0x17e3c3f50800, ms=0x17e45e456800) at /usr/src/qemu/accel/accel.c:63
  #7  0x000017e17d989312 in configure_accelerator (ms=0x17e45e456800, progname=0x7f7fffff07b0 "lm32-softmmu/qemu-system-lm32") at /usr/src/qemu/accel/accel.c:111
  #8  0x000017e17d9d8616 in main (argc=1, argv=0x7f7fffff06b8, envp=0x7f7fffff06c8) at vl.c:4325

[1] https://www.openbsd.org/faq/upgrade60.html
[2] https://undeadly.org/cgi?action=article&sid=20160527203200

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 configure | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/configure b/configure
index b18281c61f..f6acc028a7 100755
--- a/configure
+++ b/configure
@@ -5795,6 +5795,17 @@ if test "$mingw32" = "yes" ; then
     done
 fi
 
+# Disable W^X if available
+if test "$tcg" = "yes" -a "$targetos" = "OpenBSD"; then
+    cat > $TMPC <<EOF
+    int main(void) { return 0; }
+EOF
+    wx_ldflags="-Wl,-z,wxneeded"
+    if compile_prog "" "$wx_ldflags"; then
+        QEMU_LDFLAGS="$QEMU_LDFLAGS -Wl,-z,wxneeded"
+    fi
+fi
+
 qemu_confdir=$sysconfdir$confsuffix
 qemu_moddir=$libdir$confsuffix
 qemu_datadir=$datadir$confsuffix
-- 
2.20.1

Re: [Qemu-devel] [PATCH 1/3] configure: Disable W^X on OpenBSD

[Thomas Huth]

On 2019-01-25 20:27, Philippe Mathieu-Daudé wrote:
> Since OpenBSD 6.0 [1], W^X is enforced by default [2].
> TCG requires WX access. Disable W^X if it is available.
> This fixes:
> 
>   # lm32-softmmu/qemu-system-lm32
>   Could not allocate dynamic translator buffer
> 
>   # sysctl kern.wxabort=1
>   kern.wxabort: 0 -> 1
>   # lm32-softmmu/qemu-system-lm32
>   mmap: Not supported
>   Abort trap (core dumped)
>   # gdb -q lm32-softmmu/qemu-system-lm32 qemu-system-lm32.core
>   (gdb) bt
>   #0  0x000017e3c156c50a in _thread_sys___syscall () at {standard input}:5
>   #1  0x000017e3c15e5d7a in *_libc_mmap (addr=Variable "addr" is not available.) at /usr/src/lib/libc/sys/mmap.c:47
>   #2  0x000017e17d9abc8b in alloc_code_gen_buffer () at /usr/src/qemu/accel/tcg/translate-all.c:1064
>   #3  0x000017e17d9abd04 in code_gen_alloc (tb_size=0) at /usr/src/qemu/accel/tcg/translate-all.c:1112
>   #4  0x000017e17d9abe81 in tcg_exec_init (tb_size=0) at /usr/src/qemu/accel/tcg/translate-all.c:1149
>   #5  0x000017e17d9897e9 in tcg_init (ms=0x17e45e456800) at /usr/src/qemu/accel/tcg/tcg-all.c:66
>   #6  0x000017e17d9891b8 in accel_init_machine (acc=0x17e3c3f50800, ms=0x17e45e456800) at /usr/src/qemu/accel/accel.c:63
>   #7  0x000017e17d989312 in configure_accelerator (ms=0x17e45e456800, progname=0x7f7fffff07b0 "lm32-softmmu/qemu-system-lm32") at /usr/src/qemu/accel/accel.c:111
>   #8  0x000017e17d9d8616 in main (argc=1, argv=0x7f7fffff06b8, envp=0x7f7fffff06c8) at vl.c:4325
> 
> [1] https://www.openbsd.org/faq/upgrade60.html
> [2] https://undeadly.org/cgi?action=article&sid=20160527203200
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  configure | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/configure b/configure
> index b18281c61f..f6acc028a7 100755
> --- a/configure
> +++ b/configure
> @@ -5795,6 +5795,17 @@ if test "$mingw32" = "yes" ; then
>      done
>  fi
>  
> +# Disable W^X if available

I'd like to suggest to mention OpenBSD in the comment.

> +if test "$tcg" = "yes" -a "$targetos" = "OpenBSD"; then
> +    cat > $TMPC <<EOF
> +    int main(void) { return 0; }
> +EOF
> +    wx_ldflags="-Wl,-z,wxneeded"
> +    if compile_prog "" "$wx_ldflags"; then
> +        QEMU_LDFLAGS="$QEMU_LDFLAGS -Wl,-z,wxneeded"

Why do you introduce the wx_ldflags variable above, just to use it one
time? I'd suggest to either use it in the QEMU_LDFLAGS line, too, or to
get rid of the variable completely and always use -Wl,-z,wxneeded directly.

 Thomas

> +    fi
> +fi
> +
>  qemu_confdir=$sysconfdir$confsuffix
>  qemu_moddir=$libdir$confsuffix
>  qemu_datadir=$datadir$confsuffix

Re: [Qemu-devel] [PATCH 1/3] configure: Disable W^X on OpenBSD

[Philippe Mathieu-Daudé]

On 1/28/19 9:43 AM, Thomas Huth wrote:
> On 2019-01-25 20:27, Philippe Mathieu-Daudé wrote:
>> Since OpenBSD 6.0 [1], W^X is enforced by default [2].
>> TCG requires WX access. Disable W^X if it is available.
>> This fixes:
>>
>>   # lm32-softmmu/qemu-system-lm32
>>   Could not allocate dynamic translator buffer
>>
>>   # sysctl kern.wxabort=1
>>   kern.wxabort: 0 -> 1
>>   # lm32-softmmu/qemu-system-lm32
>>   mmap: Not supported
>>   Abort trap (core dumped)
>>   # gdb -q lm32-softmmu/qemu-system-lm32 qemu-system-lm32.core
>>   (gdb) bt
>>   #0  0x000017e3c156c50a in _thread_sys___syscall () at {standard input}:5
>>   #1  0x000017e3c15e5d7a in *_libc_mmap (addr=Variable "addr" is not available.) at /usr/src/lib/libc/sys/mmap.c:47
>>   #2  0x000017e17d9abc8b in alloc_code_gen_buffer () at /usr/src/qemu/accel/tcg/translate-all.c:1064
>>   #3  0x000017e17d9abd04 in code_gen_alloc (tb_size=0) at /usr/src/qemu/accel/tcg/translate-all.c:1112
>>   #4  0x000017e17d9abe81 in tcg_exec_init (tb_size=0) at /usr/src/qemu/accel/tcg/translate-all.c:1149
>>   #5  0x000017e17d9897e9 in tcg_init (ms=0x17e45e456800) at /usr/src/qemu/accel/tcg/tcg-all.c:66
>>   #6  0x000017e17d9891b8 in accel_init_machine (acc=0x17e3c3f50800, ms=0x17e45e456800) at /usr/src/qemu/accel/accel.c:63
>>   #7  0x000017e17d989312 in configure_accelerator (ms=0x17e45e456800, progname=0x7f7fffff07b0 "lm32-softmmu/qemu-system-lm32") at /usr/src/qemu/accel/accel.c:111
>>   #8  0x000017e17d9d8616 in main (argc=1, argv=0x7f7fffff06b8, envp=0x7f7fffff06c8) at vl.c:4325
>>
>> [1] https://www.openbsd.org/faq/upgrade60.html
>> [2] https://undeadly.org/cgi?action=article&sid=20160527203200
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>  configure | 11 +++++++++++
>>  1 file changed, 11 insertions(+)
>>
>> diff --git a/configure b/configure
>> index b18281c61f..f6acc028a7 100755
>> --- a/configure
>> +++ b/configure
>> @@ -5795,6 +5795,17 @@ if test "$mingw32" = "yes" ; then
>>      done
>>  fi
>>  
>> +# Disable W^X if available
> 
> I'd like to suggest to mention OpenBSD in the comment.

Good idea.

>> +if test "$tcg" = "yes" -a "$targetos" = "OpenBSD"; then
>> +    cat > $TMPC <<EOF
>> +    int main(void) { return 0; }
>> +EOF
>> +    wx_ldflags="-Wl,-z,wxneeded"
>> +    if compile_prog "" "$wx_ldflags"; then
>> +        QEMU_LDFLAGS="$QEMU_LDFLAGS -Wl,-z,wxneeded"
> 
> Why do you introduce the wx_ldflags variable above, just to use it one
> time? I'd suggest to either use it in the QEMU_LDFLAGS line, too, or to
> get rid of the variable completely and always use -Wl,-z,wxneeded directly.

I did not notice :) I'll remove the variable to avoid creating variables
used once.

Thanks!

Phil.

> 
>  Thomas
> 
>> +    fi
>> +fi
>> +
>>  qemu_confdir=$sysconfdir$confsuffix
>>  qemu_moddir=$libdir$confsuffix
>>  qemu_datadir=$datadir$confsuffix

[Qemu-devel] [PATCH 2/3] XXX oslib-posix: Ignore fcntl("/dev/null", F_SETFL, O_NONBLOCK) failure

[Philippe Mathieu-Daudé]

Previous to OpenBSD 6.3 [1], fcntl(F_SETFL) is not permitted on memory
devices. Do not assert fcntl failures on OpenBSD.
This fixes:

  $ lm32-softmmu/qemu-system-lm32
  assertion "f != -1" failed: file "util/oslib-posix.c", line 247, function "qemu_set_nonblock"
  Abort trap (core dumped)

[1] The fix seems https://github.com/openbsd/src/commit/c2a35b387f9d3c
  "fcntl(F_SETFL) invokes the FIONBIO and FIOASYNC ioctls internally, so
  the memory devices (/dev/null, /dev/zero, etc) need to permit them."

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 util/oslib-posix.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/util/oslib-posix.c b/util/oslib-posix.c
index 4ce1ba9ca4..064c3ae2f7 100644
--- a/util/oslib-posix.c
+++ b/util/oslib-posix.c
@@ -244,7 +244,9 @@ void qemu_set_nonblock(int fd)
     f = fcntl(fd, F_GETFL);
     assert(f != -1);
     f = fcntl(fd, F_SETFL, f | O_NONBLOCK);
+#ifndef __OpenBSD__
     assert(f != -1);
+#endif
 }
 
 int socket_set_fast_reuse(int fd)
-- 
2.20.1

Re: [Qemu-devel] [PATCH 2/3] XXX oslib-posix: Ignore fcntl("/dev/null", F_SETFL, O_NONBLOCK) failure

[Markus Armbruster]

Is the XXX in the subject meant to go on permanent record?

Re: [Qemu-devel] [PATCH 2/3] XXX oslib-posix: Ignore fcntl("/dev/null", F_SETFL, O_NONBLOCK) failure

[Philippe Mathieu-Daudé]

On 1/28/19 7:22 AM, Markus Armbruster wrote:
> Is the XXX in the subject meant to go on permanent record?

The original plan was to rename it as NOTFORMERGE before sending but I
forgot (late Friday). So the whole content of this patch isn't meant to
go on perm record ;)

Re: [Qemu-devel] [PATCH 2/3] XXX oslib-posix: Ignore fcntl("/dev/null", F_SETFL, O_NONBLOCK) failure

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> Previous to OpenBSD 6.3 [1], fcntl(F_SETFL) is not permitted on memory
> devices. Do not assert fcntl failures on OpenBSD.
> This fixes:
>
>   $ lm32-softmmu/qemu-system-lm32
>   assertion "f != -1" failed: file "util/oslib-posix.c", line 247, function "qemu_set_nonblock"
>   Abort trap (core dumped)
>
> [1] The fix seems https://github.com/openbsd/src/commit/c2a35b387f9d3c
>   "fcntl(F_SETFL) invokes the FIONBIO and FIOASYNC ioctls internally, so
>   the memory devices (/dev/null, /dev/zero, etc) need to permit them."

I assume set_nonblock is called on more than just these special devices?
Is there anyway to check this on OpenBSD or is it just an anonymous fd
at this point?

>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  util/oslib-posix.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/util/oslib-posix.c b/util/oslib-posix.c
> index 4ce1ba9ca4..064c3ae2f7 100644
> --- a/util/oslib-posix.c
> +++ b/util/oslib-posix.c
> @@ -244,7 +244,9 @@ void qemu_set_nonblock(int fd)
>      f = fcntl(fd, F_GETFL);
>      assert(f != -1);
>      f = fcntl(fd, F_SETFL, f | O_NONBLOCK);
> +#ifndef __OpenBSD__
>      assert(f != -1);
> +#endif
>  }
>
>  int socket_set_fast_reuse(int fd)


--
Alex Bennée

Re: [Qemu-devel] [PATCH 2/3] XXX oslib-posix: Ignore fcntl("/dev/null", F_SETFL, O_NONBLOCK) failure

[Philippe Mathieu-Daudé]

On 1/28/19 10:47 AM, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
> 
>> Previous to OpenBSD 6.3 [1], fcntl(F_SETFL) is not permitted on memory
>> devices. Do not assert fcntl failures on OpenBSD.
>> This fixes:
>>
>>   $ lm32-softmmu/qemu-system-lm32
>>   assertion "f != -1" failed: file "util/oslib-posix.c", line 247, function "qemu_set_nonblock"
>>   Abort trap (core dumped)
>>
>> [1] The fix seems https://github.com/openbsd/src/commit/c2a35b387f9d3c
>>   "fcntl(F_SETFL) invokes the FIONBIO and FIOASYNC ioctls internally, so
>>   the memory devices (/dev/null, /dev/zero, etc) need to permit them."
> 
> I assume set_nonblock is called on more than just these special devices?
> Is there anyway to check this on OpenBSD or is it just an anonymous fd
> at this point?

I'll let an OpenBSD expert to answer that.

I forgot to comment, the assert() was added one month ago (da93b82079d),
Michael said we can also revert this change.

>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>  util/oslib-posix.c | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/util/oslib-posix.c b/util/oslib-posix.c
>> index 4ce1ba9ca4..064c3ae2f7 100644
>> --- a/util/oslib-posix.c
>> +++ b/util/oslib-posix.c
>> @@ -244,7 +244,9 @@ void qemu_set_nonblock(int fd)
>>      f = fcntl(fd, F_GETFL);
>>      assert(f != -1);
>>      f = fcntl(fd, F_SETFL, f | O_NONBLOCK);
>> +#ifndef __OpenBSD__
>>      assert(f != -1);
>> +#endif
>>  }
>>
>>  int socket_set_fast_reuse(int fd)
> 
> 
> --
> Alex Bennée
> 

Re: [Qemu-devel] [PATCH 2/3] XXX oslib-posix: Ignore fcntl("/dev/null", F_SETFL, O_NONBLOCK) failure

[Paolo Bonzini]

On 28/01/19 10:47, Alex Bennée wrote:
>>
>> [1] The fix seems https://github.com/openbsd/src/commit/c2a35b387f9d3c
>>   "fcntl(F_SETFL) invokes the FIONBIO and FIOASYNC ioctls internally, so
>>   the memory devices (/dev/null, /dev/zero, etc) need to permit them."
> I assume set_nonblock is called on more than just these special devices?
> Is there anyway to check this on OpenBSD or is it just an anonymous fd
> at this point?
> 

Perhaps on OpenBSD we should just assert that we don't get EBADF?

Paolo

Re: [Qemu-devel] [PATCH 2/3] XXX oslib-posix: Ignore fcntl("/dev/null", F_SETFL, O_NONBLOCK) failure

[Philippe Mathieu-Daudé]

On 1/28/19 12:03 PM, Paolo Bonzini wrote:
> On 28/01/19 10:47, Alex Bennée wrote:
>>>
>>> [1] The fix seems https://github.com/openbsd/src/commit/c2a35b387f9d3c
>>>   "fcntl(F_SETFL) invokes the FIONBIO and FIOASYNC ioctls internally, so
>>>   the memory devices (/dev/null, /dev/zero, etc) need to permit them."
>> I assume set_nonblock is called on more than just these special devices?
>> Is there anyway to check this on OpenBSD or is it just an anonymous fd
>> at this point?
>>
> 
> Perhaps on OpenBSD we should just assert that we don't get EBADF?

We get ENODEV for "not a memory device":

    19 ENODEV Operation not supported by device.
    An attempt was made to apply an inappropriate function to a device,
    for example, trying to read a write-only device such as a printer.

I'll respin with your suggestion.

Thanks!

Phil.

[Qemu-devel] [PATCH 3/3] WIP tests/vm: Run tests on OpenBSD

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/vm/openbsd | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/tests/vm/openbsd b/tests/vm/openbsd
index cfe0572c59..de907dd21c 100755
--- a/tests/vm/openbsd
+++ b/tests/vm/openbsd
@@ -25,9 +25,7 @@ class OpenBSDVM(basevm.BaseVM):
         cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
         tar -xf /dev/rsd1c;
         ./configure --cc=x86_64-unknown-openbsd6.1-gcc-4.9.4 --python=python2.7 {configure_opts};
-        gmake --output-sync -j{jobs} {verbose};
-        # XXX: "gmake check" seems to always hang or fail
-        #gmake --output-sync -j{jobs} check {verbose};
+        gmake --output-sync -j{jobs} check {verbose};
     """
 
     def build_image(self, img):
-- 
2.20.1

Re: [Qemu-devel] [PATCH 3/3] WIP tests/vm: Run tests on OpenBSD

[Thomas Huth]

On 2019-01-25 20:27, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/vm/openbsd | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
> 
> diff --git a/tests/vm/openbsd b/tests/vm/openbsd
> index cfe0572c59..de907dd21c 100755
> --- a/tests/vm/openbsd
> +++ b/tests/vm/openbsd
> @@ -25,9 +25,7 @@ class OpenBSDVM(basevm.BaseVM):
>          cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
>          tar -xf /dev/rsd1c;
>          ./configure --cc=x86_64-unknown-openbsd6.1-gcc-4.9.4 --python=python2.7 {configure_opts};
> -        gmake --output-sync -j{jobs} {verbose};
> -        # XXX: "gmake check" seems to always hang or fail
> -        #gmake --output-sync -j{jobs} check {verbose};
> +        gmake --output-sync -j{jobs} check {verbose};
>      """
>  
>      def build_image(self, img):
> 

Please add a proper patch description and remove the WIP from the subject.

 Thanks,
  Thomas

Re: [Qemu-devel] [PATCH 3/3] WIP tests/vm: Run tests on OpenBSD

[Philippe Mathieu-Daudé]

On 1/28/19 9:44 AM, Thomas Huth wrote:
> On 2019-01-25 20:27, Philippe Mathieu-Daudé wrote:
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>  tests/vm/openbsd | 4 +---
>>  1 file changed, 1 insertion(+), 3 deletions(-)
>>
>> diff --git a/tests/vm/openbsd b/tests/vm/openbsd
>> index cfe0572c59..de907dd21c 100755
>> --- a/tests/vm/openbsd
>> +++ b/tests/vm/openbsd
>> @@ -25,9 +25,7 @@ class OpenBSDVM(basevm.BaseVM):
>>          cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
>>          tar -xf /dev/rsd1c;
>>          ./configure --cc=x86_64-unknown-openbsd6.1-gcc-4.9.4 --python=python2.7 {configure_opts};
>> -        gmake --output-sync -j{jobs} {verbose};
>> -        # XXX: "gmake check" seems to always hang or fail
>> -        #gmake --output-sync -j{jobs} check {verbose};
>> +        gmake --output-sync -j{jobs} check {verbose};
>>      """
>>  
>>      def build_image(self, img):
>>
> 
> Please add a proper patch description and remove the WIP from the subject.

Oops :/ I thought about renaming this also NOTFORMERGE.

The description could be:

  OpenBSD tests are still failing, but this patch help to test
  the improvement of the previous 2 patches (disable W^X, disable
  assert(fcntl)).