From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
To: Max Reitz <mreitz@redhat.com>, qemu-block@nongnu.org
Cc: kwolf@redhat.com, wencongyang2@huawei.com,
xiechanglong.d@gmail.com, qemu-devel@nongnu.org,
armbru@redhat.com, den@openvz.org
Subject: Re: [PATCH v4 09/23] job: call job_enter from job_pause
Date: Wed, 21 Apr 2021 11:31:02 +0300 [thread overview]
Message-ID: <97630ba3-af42-9c33-30b3-bdd9aeb0a5d9@virtuozzo.com> (raw)
In-Reply-To: <062f350d-5194-393b-5725-953740ad4b0f@virtuozzo.com>
07.04.2021 14:38, Vladimir Sementsov-Ogievskiy wrote:
> 07.04.2021 14:19, Max Reitz wrote:
>> On 16.01.21 22:46, Vladimir Sementsov-Ogievskiy wrote:
>>> If main job coroutine called job_yield (while some background process
>>> is in progress), we should give it a chance to call job_pause_point().
>>> It will be used in backup, when moved on async block-copy.
>>>
>>> Note, that job_user_pause is not enough: we want to handle
>>> child_job_drained_begin() as well, which call job_pause().
>>>
>>> Still, if job is already in job_do_yield() in job_pause_point() we
>>> should not enter it.
>>>
>>> iotest 109 output is modified: on stop we do bdrv_drain_all() which now
>>> triggers job pause immediately (and pause after ready is standby).
>>>
>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>> ---
>>> job.c | 3 +++
>>> tests/qemu-iotests/109.out | 24 ++++++++++++++++++++++++
>>> 2 files changed, 27 insertions(+)
>>
>> While looking into
>>
>> https://lists.gnu.org/archive/html/qemu-block/2021-04/msg00035.html
>>
>> I noticed this:
>>
>> $ ./qemu-img create -f raw src.img 1G
>> $ ./qemu-img create -f raw dst.img 1G
>>
>> $ (echo '
>> {"execute":"qmp_capabilities"}
>> {"execute":"blockdev-mirror",
>> "arguments":{"job-id":"mirror",
>> "device":"source",
>> "target":"target",
>> "sync":"full",
>> "filter-node-name":"mirror-top"}}
>> '; sleep 3; echo '
>> {"execute":"human-monitor-command",
>> "arguments":{"command-line":
>> "qemu-io mirror-top \"write 0 1G\""}}') \
>> | x86_64-softmmu/qemu-system-x86_64 \
>> -qmp stdio \
>> -blockdev file,node-name=source,filename=src.img \
>> -blockdev file,node-name=target,filename=dst.img \
>> -object iothread,id=iothr0 \
>> -device virtio-blk,drive=source,iothread=iothr0
>>
>> Before this commit, qemu-io returned an error that there is a permission conflict with virtio-blk. After this commit, there is an abort (“qemu: qemu_mutex_unlock_impl: Operation not permitted”):
>>
>> #0 0x00007f8445a4eef5 in raise () at /usr/lib/libc.so.6
>> #1 0x00007f8445a38862 in abort () at /usr/lib/libc.so.6
>> #2 0x000055fbb14a36bf in error_exit
>> (err=<optimized out>, msg=msg@entry=0x55fbb1634790 <__func__.27> "qemu_mutex_unlock_impl")
>> at ../util/qemu-thread-posix.c:37
>> #3 0x000055fbb14a3bc3 in qemu_mutex_unlock_impl
>> (mutex=mutex@entry=0x55fbb25ab6e0, file=file@entry=0x55fbb1636957 "../util/async.c", line=line@entry=650)
>> at ../util/qemu-thread-posix.c:109
>> #4 0x000055fbb14b2e75 in aio_context_release (ctx=ctx@entry=0x55fbb25ab680) at ../util/async.c:650
>> #5 0x000055fbb13d2029 in bdrv_do_drained_begin
>> (bs=bs@entry=0x55fbb3a87000, recursive=recursive@entry=false, parent=parent@entry=0x0, ignore_bds_parents=ignore_bds_parents@entry=false, poll=poll@entry=true) at ../block/io.c:441
>> #6 0x000055fbb13d2192 in bdrv_do_drained_begin
>> (poll=true, ignore_bds_parents=false, parent=0x0, recursive=false, bs=0x55fbb3a87000) at ../block/io.c:448
>> #7 0x000055fbb13c71a7 in blk_drain (blk=0x55fbb26c5a00) at ../block/block-backend.c:1718
>> #8 0x000055fbb13c8bbd in blk_unref (blk=0x55fbb26c5a00) at ../block/block-backend.c:498
>> #9 blk_unref (blk=0x55fbb26c5a00) at ../block/block-backend.c:491
>> #10 0x000055fbb1024863 in hmp_qemu_io (mon=0x7fffaf3fc7d0, qdict=<optimized out>)
>> at ../block/monitor/block-hmp-cmds.c:628
>>
>> Can you make anything out of this?
>>
>
> Hmm.. Interesting.
>
> man pthread_mutex_unlock
> ...
> EPERM The mutex type is PTHREAD_MUTEX_ERRORCHECK or PTHREAD_MUTEX_RECURSIVE, or the mutex is a
> robust mutex, and the current thread does not own the mutex.
>
> So, thread doesn't own the mutex.. We have an iothread here.
>
> AIO_WAIT_WHILE() documents that ctx must be acquired exactly once by caller.. But I don't see, where is it acquired in the call stack?
>
> The other question, is why permission conflict is lost with the commit. Strange. I ss that hmp_qemu_io creates blk with perm=0 and shread=BLK_PERM_ALL.. How could it conflict even before the considered commit?
>
Sorry, I've answered and forgot about this thread. Now, looking through my series I find this again. Seems that problem is really in lacking aio-context locking around blk_unref(). I'll send patch now.
--
Best regards,
Vladimir
next prev parent reply other threads:[~2021-04-21 8:32 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-16 21:46 [PATCH v4 00/23] backup performance: block_status + async Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 01/23] qapi: backup: add perf.use-copy-range parameter Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 02/23] block/block-copy: More explicit call_state Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 03/23] block/block-copy: implement block_copy_async Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 04/23] block/block-copy: add max_chunk and max_workers parameters Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 05/23] block/block-copy: add list of all call-states Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 06/23] block/block-copy: add ratelimit to block-copy Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 07/23] block/block-copy: add block_copy_cancel Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 08/23] blockjob: add set_speed to BlockJobDriver Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 09/23] job: call job_enter from job_pause Vladimir Sementsov-Ogievskiy
2021-01-18 13:45 ` Max Reitz
2021-01-18 14:20 ` Max Reitz
2021-04-07 11:19 ` Max Reitz
2021-04-07 11:38 ` Vladimir Sementsov-Ogievskiy
2021-04-21 8:31 ` Vladimir Sementsov-Ogievskiy [this message]
2021-01-16 21:46 ` [PATCH v4 10/23] qapi: backup: add max-chunk and max-workers to x-perf struct Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 11/23] iotests: 56: prepare for backup over block-copy Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 12/23] iotests: 185: " Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 13/23] iotests: 219: " Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 14/23] iotests: 257: " Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 15/23] block/block-copy: make progress_bytes_callback optional Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 16/23] block/backup: drop extra gotos from backup_run() Vladimir Sementsov-Ogievskiy
2021-01-16 21:46 ` [PATCH v4 17/23] backup: move to block-copy Vladimir Sementsov-Ogievskiy
2021-01-16 21:47 ` [PATCH v4 18/23] qapi: backup: disable copy_range by default Vladimir Sementsov-Ogievskiy
2021-01-16 21:47 ` [PATCH v4 19/23] block/block-copy: drop unused block_copy_set_progress_callback() Vladimir Sementsov-Ogievskiy
2021-01-16 21:47 ` [PATCH v4 20/23] block/block-copy: drop unused argument of block_copy() Vladimir Sementsov-Ogievskiy
2021-01-16 21:47 ` [PATCH v4 21/23] simplebench/bench_block_job: use correct shebang line with python3 Vladimir Sementsov-Ogievskiy
2021-01-16 21:47 ` [PATCH v4 22/23] simplebench: bench_block_job: add cmd_options argument Vladimir Sementsov-Ogievskiy
2021-01-16 21:47 ` [PATCH v4 23/23] simplebench: add bench-backup.py Vladimir Sementsov-Ogievskiy
2021-01-18 14:01 ` Max Reitz
2021-01-18 15:07 ` [PATCH v4 00/23] backup performance: block_status + async Max Reitz
2021-01-18 17:04 ` Vladimir Sementsov-Ogievskiy
2021-01-19 18:40 ` Max Reitz
2021-01-19 19:22 ` Vladimir Sementsov-Ogievskiy
2021-01-19 19:29 ` Vladimir Sementsov-Ogievskiy
2021-01-20 10:39 ` Max Reitz
2021-01-20 13:50 ` Max Reitz
2021-01-20 14:34 ` Max Reitz
2021-01-20 14:44 ` Max Reitz
2021-01-20 15:53 ` Max Reitz
2021-01-20 16:00 ` Max Reitz
2021-01-20 16:04 ` Daniel P. Berrangé
2021-01-20 16:40 ` Max Reitz
2021-01-20 10:20 ` [PATCH v4 11.5/23] iotests/129: Limit backup's max-chunk/max-workers Max Reitz
2021-01-20 11:24 ` Vladimir Sementsov-Ogievskiy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=97630ba3-af42-9c33-30b3-bdd9aeb0a5d9@virtuozzo.com \
--to=vsementsov@virtuozzo.com \
--cc=armbru@redhat.com \
--cc=den@openvz.org \
--cc=kwolf@redhat.com \
--cc=mreitz@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
--cc=wencongyang2@huawei.com \
--cc=xiechanglong.d@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).