Re: [PULL 00/18] Block patches

[Max Reitz <mreitz@redhat.com>]

[-- Attachment #1.1: Type: text/plain, Size: 15994 bytes --]

On 23.06.20 14:55, Peter Maydell wrote:
> On Mon, 22 Jun 2020 at 16:11, Max Reitz <mreitz@redhat.com> wrote:
>>
>> The following changes since commit bae31bfa48b9caecee25da3d5333901a126a06b4:
>>
>>   Merge remote-tracking branch 'remotes/kraxel/tags/audio-20200619-pull-request' into staging (2020-06-19 22:56:59 +0100)
>>
>> are available in the Git repository at:
>>
>>   https://github.com/XanClic/qemu.git tags/pull-block-2020-06-22
>>
>> for you to fetch changes up to 74c55e4142a7bb835c38d3770c74210cbb1e4fab:
>>
>>   iotests: don't test qcow2.py inside 291 (2020-06-22 16:05:23 +0200)
>>
>> ----------------------------------------------------------------
>> Block patches:
>> - Support modifying a LUKS-encrypted image's keyslots
>> - iotest fixes
>>
>> ----------------------------------------------------------------
> 
> Hi; I see various iotest failures, different things on
> PPC64 Linux, OpenBSD and FreeBSD, and on an AArch32 build
> that happens to not have optional crypto libs installed.

OK.  Sorry.  None of this looks easily fixable, so I suppose I’ll have
to drop everything but the last two patches for now.

> On PPC64 Linux, lots of iotests fail like this:
> 
>   TEST    iotest-qcow2: 001 [fail]
> QEMU          --
> "/home/pm215/qemu/build/all/tests/qemu-iotests/../../ppc64-softmmu/qemu-system-ppc64"
> -nodefaults -display none -accel qtest
> QEMU_IMG      -- "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-io"  --cache
> writeback --aio threads -f qcow2
> QEMU_NBD      -- "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- Linux/ppc64 gcc1-power7 3.10.0-862.14.4.el7.ppc64
> TEST_DIR      -- /home/pm215/qemu/build/all/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.vvBdnkatyZ
> SOCKET_SCM_HELPER --
> /home/pm215/qemu/build/all/tests/qemu-iotests/socket_scm_helper
> 
> --- /home/pm215/qemu/tests/qemu-iotests/001.out 2015-04-08
> 18:43:24.908449234 +0000
> +++ /home/pm215/qemu/build/all/tests/qemu-iotests/001.out.bad
> 2020-06-23 10:42:29.262626202 +0000
> @@ -1,5 +1,6 @@
>  QA output created by 001
> -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
> +./common.filter: line 128: readarray: -d: invalid option
> +readarray: usage: readarray [-n count] [-O origin] [-s count] [-t]
> [-u fd] [-C callback] [-c quantum] [array]
> 
>  == reading whole image ==
>  read 134217728/134217728 bytes at offset 0
>   TEST    iotest-qcow2: 002 [fail]
> QEMU          --
> "/home/pm215/qemu/build/all/tests/qemu-iotests/../../ppc64-softmmu/qemu-system-ppc64"
> -nodefaults -display none -accel qtest
> QEMU_IMG      -- "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-io"  --cache
> writeback --aio threads -f qcow2
> QEMU_NBD      -- "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- Linux/ppc64 gcc1-power7 3.10.0-862.14.4.el7.ppc64
> TEST_DIR      -- /home/pm215/qemu/build/all/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.vvBdnkatyZ
> SOCKET_SCM_HELPER --
> /home/pm215/qemu/build/all/tests/qemu-iotests/socket_scm_helper
> 
> Looks like you're trying to use a readarray option that doesn't
> exist (maybe only exists in newer shells?)

Yes, I am.  Well, that’s a real shame. :/

I wasn’t aware that readarray’s -d option was only introduced at some
later point.  Looks like that happened in 4.4, which was only released
2016.  So that’s probably too new indeed.

> iotests failures on aarch32 which happens to not have some
> optional crypto lib dependency installed I guess; these
> iotests ought to be made to skip if the functionality they're
> testing isn't compiled into this QEMU:

Right.  Maxim, can you do that?

>   TEST    iotest-qcow2: 293 [fail]
> QEMU          --
> "/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
> -nodefaults -display none -machine virt -accel qtest
> QEMU_IMG      --
> "/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-io"
>  --cache writeback --aio threads -f qcow2
> QEMU_NBD      --
> "/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- Linux/aarch64 mustang-maydell 4.15.0-101-generic
> TEST_DIR      --
> /home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.tjBWiNDvED
> SOCKET_SCM_HELPER --
> /home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/socket_scm_helper
> 
> --- /home/peter.maydell/qemu/tests/qemu-iotests/293.out 2020-06-23
> 10:38:50.091867725 +0000
> +++ /home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/293.out.bad
>       2020-06-23 10:59:09.027627162 +0000
> @@ -1,5 +1,6 @@
>  QA output created by 293
>  == creating a test image ==
> +qemu-img: TEST_DIR/t.IMGFMT: No crypto library supporting PBKDF in
> this build: Function not implemented
>  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432
> 
>  == test that key 0 opens the image ==
> @@ -7,10 +8,15 @@
>  4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> 
>  == adding a password to slot 4 ==
> +qemu-img: Can't amend encryption options - encryption not present
>  == adding a password to slot 1 ==
> +qemu-img: Can't amend encryption options - encryption not present
>  == adding a password to slot 3 ==
> +qemu-img: Can't amend encryption options - encryption not present
>  == adding a password to slot 2 ==
> +qemu-img: Can't amend encryption options - encryption not present
>  == erase slot 4 ==
> +qemu-img: Can't amend encryption options - encryption not present
> 
>  == all secrets should work ==
>  read 4096/4096 bytes at offset 0
> @@ -23,19 +29,29 @@
>  4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> 
>  == erase slot 0 and try it ==
> -qemu-io: can't open: Invalid password, cannot unlock any keyslot
> +qemu-img: Can't amend encryption options - encryption not present
> +read 4096/4096 bytes at offset 0
> +4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> [etc]
> 
> 
> On OpenBSD: iotest 293 fails trying to use "seq" when it doesn't exist:

Well that’s nice.  Maxim, I think we’ll need a for (()) loop then (like
e.g. tests/qemu-iotests/common.pattern:25 does it).

>   TEST    iotest-qcow2: 293 [fail]
> QEMU          --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
> -nodef
> aults -display none -machine virt -accel qtest
> QEMU_IMG      --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-io"
> --cache writeback --aio threads -
> f qcow2
> QEMU_NBD      --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- OpenBSD/amd64 openbsd 6.6
> TEST_DIR      -- /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.o4s3SCLIsx
> SOCKET_SCM_HELPER --
> 
> --- /home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/293.out  Tue
> Jun 23 10:38:33 2020
> +++ /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/293.out.bad
>  Tue Jun 23 11:06:19 2020
> @@ -1,6 +1,6 @@
>  QA output created by 293
>  == creating a test image ==
> -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432
> +Formatting 'TEST_DIR/t.IMGFMT'x0IMGFMT encrypt.format=luks
> encrypt.key-secret=sec0 encrypt.iter-time=10 cluster_size=65
> 536 compression_type=zlib size=33554432 compat=1.1 lazy_refcounts=off
> refcount_bits=16
> 
>  == test that key 0 opens the image ==
>  read 4096/4096 bytes at offset 0
> @@ -29,25 +29,26 @@
>  qemu-io: can't open: Invalid password, cannot unlock any keyslot
> 
>  == filling  4 slots with secret 2 ==
> +./293: line 111: seq: command not found
> 
>  == adding secret 0 ==
> 
>  == adding secret 3 (last slot) ==
> 
>  == trying to add another slot (should fail) ==
> -qemu-img: Can't add a keyslot - all keyslots are in use
> +qemu-img: Could not open
> 'driver=qcow2,file.filename=/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/scratch/t.qco
> w2,encrypt.key-secret=sec2': Invalid password, cannot unlock any keyslot
> 
>  == all secrets should work again ==
>  read 4096/4096 bytes at offset 0
>  4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>  read 4096/4096 bytes at offset 0
>  4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +qemu-io: can't open: Invalid password, cannot unlock any keyslot
>  read 4096/4096 bytes at offset 0
>  4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> -read 4096/4096 bytes at offset 0
> -4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> 
>  == erase all keys of secret 2==
> +qemu-img: No keyslots match given (old) password for erase operation
>  == erase all keys of secret 1==
>  == erase all keys of secret 0==
>  == erasing secret3 will fail now since it is the only secret (in 3 slots) ==
> 
> Also OpenBSD, iotest 295 failed because the QEMU dumped core:

I assume the SIGKILL occurred because 295 somewhere waits on some QMP
event that never happens (probably because the encryption is again not
supported), so the iotest kills it on timeout.

(Maxim, I think that we’ll need something similar to what we need in 293
to skip the test if there’s no crypto library.)

>   TEST    iotest-qcow2: 295 [fail]
> QEMU          --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
> -nodef
> aults -display none -machine virt -accel qtest
> QEMU_IMG      --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-io"
> --cache writeback --aio threads -
> f qcow2
> QEMU_NBD      --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- OpenBSD/amd64 openbsd 6.6
> TEST_DIR      -- /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.o4s3SCLIsx
> SOCKET_SCM_HELPER --
> 
> --- /home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/295.out  Tue
> Jun 23 10:38:33 2020
> +++ /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/295.out.bad
>  Tue Jun 23 11:09:01 2020
> @@ -4,19 +4,11 @@
>  {"return": {}}
>  {"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
>  {"return": {}}
> +WARNING:qemu.machine:qemu received signal 9:
> /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64
> -display none -vga none -chardev
> socket,id=mon,path=/tmp/tmp.o4s3SCLIsx/qemu-36824-monitor.sock -mon
> chardev=mon,mode=control -qtest
> unix:path=/tmp/tmp.o4s3SCLIsx/qemu-36824-qtest.sock -accel qtest
> -nodefaults -display none -machine virt -accel qtest
>  {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
>  {"return": {}}
> -{"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
> -{"return": {}}
> -Job failed: Invalid password, cannot unlock any keyslot
>  {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
>  {"return": {}}
> -{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
> -{"return": {}}
> -{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
> -{"return": {}}
> -{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
> -{"return": {}}
>  Job failed: Refusing to overwrite active keyslot 2 - please erase it first
>  {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
>  {"return": {}}
> @@ -33,8 +25,37 @@
>  {"return": {}}
>  {"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
>  {"return": {}}
> -...
> +.E.
> +======================================================================
> +ERROR: testOldPassword (__main__.EncryptionSetupTestCase)
>  ----------------------------------------------------------------------
> +Traceback (most recent call last):
> +  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/qmp.py",
> line 139, in __get_events
> +    ret = self.__json_read(only_event=True)
> +  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/qmp.py",
> line 99, in __json_read
> +    data = self.__sockfile.readline()
> +  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
> +    return self._sock.recv_into(b)
> +socket.timeout: timed out
> +
> +During handling of the above exception, another exception occurred:
> +
> +Traceback (most recent call last):
> +  File "295", line 234, in testOldPassword
> +    self.addKeyQmp("testdev", new_secret = self.secrets[1])
> +  File "295", line 164, in addKeyQmp
> +    self.vm.run_job('job_add_key')
> +  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/iotests.py",
> line 648, in run_job
> +    ev = filter_qmp_event(self.events_wait(events, timeout=wait))
> +  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/machine.py",
> line 518, in events_wait
> +    event = self._qmp.pull_event(wait=timeout)
> +  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/qmp.py",
> line 254, in pull_event
> +    self.__get_events(wait)
> +  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/qmp.py",
> line 141, in __get_events
> +    raise QMPTimeoutError("Timeout waiting for event")
> +qemu.qmp.QMPTimeoutError: Timeout waiting for event
> +
> +----------------------------------------------------------------------
>  Ran 3 tests
> 
> -OK
> +FAILED (errors=1)
> 
> FreeBSD: lots of iotest failures, looks like some sed or other
> command trying to sanitize output is broken on this host OS
> (assuming gnu sed or whatever, maybe?)

It’s assuming that \x0 produces a NUL character.  It’s a shame it
doesn’t, but I suppose it doesn’t even really matter without readarray
supporting -d.

>   TEST    iotest-qcow2: 001 [fail]
> QEMU          --
> "/home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
> -nodefaults -display none -machine virt -accel qtest
> QEMU_IMG      --
> "/home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/../../qemu-io"
> --cache writeback --aio threads -f qcow2
> QEMU_NBD      --
> "/home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- FreeBSD/amd64 freebsd 12.1-RELEASE
> TEST_DIR      -- /home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.l5ksaQ9N
> SOCKET_SCM_HELPER --
> 
> --- /home/qemu/qemu-test.RNYtim/src/tests/qemu-iotests/001.out
> 2020-06-23 10:38:33.000000000 +0000
> +++ /home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/001.out.bad
>  2020-06-23 10:56:47.833020000 +0000
> @@ -1,5 +1,5 @@
>  QA output created by 001
> -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
> +Formatting 'TEST_DIR/t.IMGFMT'x0IMGFMT cluster_size=65536
> compression_type=zlib size=134217728 compat=1.1 lazy_refcounts=off
> refcount_bits=16
> 
>  == reading whole image ==
>  read 134217728/134217728 bytes at offset 0
> 
> 
> thanks

Thank you!

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]