From: Dima Stepanov <dimastep@yandex-team.ru>
To: Jason Wang <jasowang@redhat.com>
Cc: fam@euphon.net, kwolf@redhat.com, stefanha@redhat.com,
qemu-block@nongnu.org, mst@redhat.com, qemu-devel@nongnu.org,
dgilbert@redhat.com, arei.gonglei@huawei.com, fengli@smartx.com,
yc-core@yandex-team.ru, pbonzini@redhat.com,
marcandre.lureau@redhat.com, raphael.norwitz@nutanix.com,
mreitz@redhat.com
Subject: Re: [PATCH v2 5/5] vhost: add device started check in migration set log
Date: Wed, 13 May 2020 12:39:43 +0300 [thread overview]
Message-ID: <20200513093934.GA15883@dimastep-nix> (raw)
In-Reply-To: <5afd8f14-ed9f-d3e1-3e7f-649a5140db28@redhat.com>
On Wed, May 13, 2020 at 11:20:50AM +0800, Jason Wang wrote:
>
> On 2020/5/12 下午5:35, Dima Stepanov wrote:
> >On Tue, May 12, 2020 at 11:32:50AM +0800, Jason Wang wrote:
> >>On 2020/5/11 下午5:25, Dima Stepanov wrote:
> >>>On Mon, May 11, 2020 at 11:15:53AM +0800, Jason Wang wrote:
> >>>>On 2020/4/30 下午9:36, Dima Stepanov wrote:
> >>>>>If vhost-user daemon is used as a backend for the vhost device, then we
> >>>>>should consider a possibility of disconnect at any moment. If such
> >>>>>disconnect happened in the vhost_migration_log() routine the vhost
> >>>>>device structure will be clean up.
> >>>>>At the start of the vhost_migration_log() function there is a check:
> >>>>> if (!dev->started) {
> >>>>> dev->log_enabled = enable;
> >>>>> return 0;
> >>>>> }
> >>>>>To be consistent with this check add the same check after calling the
> >>>>>vhost_dev_set_log() routine. This in general help not to break a
> >>>>>migration due the assert() message. But it looks like that this code
> >>>>>should be revised to handle these errors more carefully.
> >>>>>
> >>>>>In case of vhost-user device backend the fail paths should consider the
> >>>>>state of the device. In this case we should skip some function calls
> >>>>>during rollback on the error paths, so not to get the NULL dereference
> >>>>>errors.
> >>>>>
> >>>>>Signed-off-by: Dima Stepanov<dimastep@yandex-team.ru>
> >>>>>---
> >>>>> hw/virtio/vhost.c | 39 +++++++++++++++++++++++++++++++++++----
> >>>>> 1 file changed, 35 insertions(+), 4 deletions(-)
> >>>>>
> >>>>>diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> >>>>>index 3ee50c4..d5ab96d 100644
> >>>>>--- a/hw/virtio/vhost.c
> >>>>>+++ b/hw/virtio/vhost.c
> >>>>>@@ -787,6 +787,17 @@ static int vhost_dev_set_features(struct vhost_dev *dev,
> >>>>> static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
> >>>>> {
> >>>>> int r, i, idx;
> >>>>>+
> >>>>>+ if (!dev->started) {
> >>>>>+ /*
> >>>>>+ * If vhost-user daemon is used as a backend for the
> >>>>>+ * device and the connection is broken, then the vhost_dev
> >>>>>+ * structure will be reset all its values to 0.
> >>>>>+ * Add additional check for the device state.
> >>>>>+ */
> >>>>>+ return -1;
> >>>>>+ }
> >>>>>+
> >>>>> r = vhost_dev_set_features(dev, enable_log);
> >>>>> if (r < 0) {
> >>>>> goto err_features;
> >>>>>@@ -801,12 +812,19 @@ static int vhost_dev_set_log(struct vhost_dev *dev, bool enable_log)
> >>>>> }
> >>>>> return 0;
> >>>>> err_vq:
> >>>>>- for (; i >= 0; --i) {
> >>>>>+ /*
> >>>>>+ * Disconnect with the vhost-user daemon can lead to the
> >>>>>+ * vhost_dev_cleanup() call which will clean up vhost_dev
> >>>>>+ * structure.
> >>>>>+ */
> >>>>>+ for (; dev->started && (i >= 0); --i) {
> >>>>> idx = dev->vhost_ops->vhost_get_vq_index(
> >>>>Why need the check of dev->started here, can started be modified outside
> >>>>mainloop? If yes, I don't get the check of !dev->started in the beginning of
> >>>>this function.
> >>>>
> >>>No dev->started can't change outside the mainloop. The main problem is
> >>>only for the vhost_user_blk daemon. Consider the case when we
> >>>successfully pass the dev->started check at the beginning of the
> >>>function, but after it we hit the disconnect on the next call on the
> >>>second or third iteration:
> >>> r = vhost_virtqueue_set_addr(dev, dev->vqs + i, idx, enable_log);
> >>>The unix socket backend device will call the disconnect routine for this
> >>>device and reset the structure. So the structure will be reset (and
> >>>dev->started set to false) inside this set_addr() call.
> >>I still don't get here. I think the disconnect can not happen in the middle
> >>of vhost_dev_set_log() since both of them were running in mainloop. And even
> >>if it can, we probably need other synchronization mechanism other than
> >>simple check here.
> >Disconnect isn't happened in the separate thread it is happened in this
> >routine inside vhost_dev_set_log. When for instance vhost_user_write()
> >call failed:
> > vhost_user_set_log_base()
> > vhost_user_write()
> > vhost_user_blk_disconnect()
> > vhost_dev_cleanup()
> > vhost_user_backend_cleanup()
> >So the point is that if we somehow got a disconnect with the
> >vhost-user-blk daemon before the vhost_user_write() call then it will
> >continue clean up by running vhost_user_blk_disconnect() function. I
> >wrote a more detailed backtrace stack in the separate thread, which is
> >pretty similar to what we have here:
> > Re: [PATCH v2 4/5] vhost: check vring address before calling unmap
> >The places are different but the problem is pretty similar.
>
>
> Yes.
>
>
> >
> >So if vhost-user commands handshake then everything is fine and
> >reconnect will work as expected. The only problem is how to handle
> >reconnect properly between vhost-user command send/receive.
> >
> >As i wrote we have a test:
> > - run src VM with vhost-usr-blk daemon used
> > - run fio inside it
> > - perform reconnect every X seconds (just kill and restart daemon),
> > X is random
> > - run dst VM
> > - perform migration
> > - fio should complete in dst VM
> >And we cycle this test like forever.
> >So it fails once per ~25 iteration. By adding some delays inside qemu we
> >were able to make the race window larger.
>
>
> It would be better if we can draft some qtest for this.
Yes, i'm in process of figuring out how to make/reproduce it in the
qtest framework instead of our internal one.
>
> Thanks
>
>
> >
>
next prev parent reply other threads:[~2020-05-13 9:40 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-04-30 13:36 [PATCH v2 0/5] vhost-user reconnect issues during vhost initialization Dima Stepanov
2020-04-30 13:36 ` [PATCH v2 1/5] char-socket: return -1 in case of disconnect during tcp_chr_write Dima Stepanov
2020-05-06 8:54 ` Li Feng
2020-05-06 9:46 ` Marc-André Lureau
2020-04-30 13:36 ` [PATCH v2 2/5] vhost: introduce wrappers to set guest notifiers for virtio device Dima Stepanov
2020-05-04 0:36 ` Raphael Norwitz
2020-05-06 8:54 ` Dima Stepanov
2020-05-11 3:03 ` Jason Wang
2020-05-11 8:55 ` Dima Stepanov
2020-04-30 13:36 ` [PATCH v2 3/5] vhost-user-blk: add mechanism to track the guest notifiers init state Dima Stepanov
2020-05-04 1:06 ` Raphael Norwitz
2020-05-06 8:51 ` Dima Stepanov
2020-04-30 13:36 ` [PATCH v2 4/5] vhost: check vring address before calling unmap Dima Stepanov
2020-05-04 1:13 ` Raphael Norwitz
2020-05-11 3:05 ` Jason Wang
2020-05-11 9:11 ` Dima Stepanov
2020-05-12 3:26 ` Jason Wang
2020-05-12 9:08 ` Dima Stepanov
2020-05-13 3:00 ` Jason Wang
2020-05-13 9:36 ` Dima Stepanov
2020-05-14 7:28 ` Jason Wang
2020-04-30 13:36 ` [PATCH v2 5/5] vhost: add device started check in migration set log Dima Stepanov
2020-05-06 22:08 ` Raphael Norwitz
2020-05-07 7:15 ` Michael S. Tsirkin
2020-05-07 15:35 ` Dima Stepanov
2020-05-11 0:03 ` Raphael Norwitz
2020-05-11 9:43 ` Dima Stepanov
2020-05-11 3:15 ` Jason Wang
2020-05-11 9:25 ` Dima Stepanov
2020-05-12 3:32 ` Jason Wang
2020-05-12 3:47 ` Li Feng
2020-05-12 9:23 ` Dima Stepanov
2020-05-12 9:35 ` Dima Stepanov
2020-05-13 3:20 ` Jason Wang
2020-05-13 9:39 ` Dima Stepanov [this message]
2020-05-13 4:15 ` Michael S. Tsirkin
2020-05-13 5:56 ` Jason Wang
2020-05-13 9:47 ` Dima Stepanov
2020-05-14 7:34 ` Jason Wang
2020-05-15 16:54 ` Dima Stepanov
2020-05-16 3:20 ` Li Feng
2020-05-18 2:52 ` Jason Wang
2020-05-18 9:33 ` Dima Stepanov
2020-05-18 9:27 ` Dima Stepanov
2020-05-18 2:50 ` Jason Wang
2020-05-18 9:41 ` Dima Stepanov
2020-05-18 9:53 ` Dr. David Alan Gilbert
2020-05-19 9:07 ` Dima Stepanov
2020-05-19 10:24 ` Dr. David Alan Gilbert
2020-05-19 9:59 ` Michael S. Tsirkin
2020-05-19 9:13 ` Dima Stepanov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200513093934.GA15883@dimastep-nix \
--to=dimastep@yandex-team.ru \
--cc=arei.gonglei@huawei.com \
--cc=dgilbert@redhat.com \
--cc=fam@euphon.net \
--cc=fengli@smartx.com \
--cc=jasowang@redhat.com \
--cc=kwolf@redhat.com \
--cc=marcandre.lureau@redhat.com \
--cc=mreitz@redhat.com \
--cc=mst@redhat.com \
--cc=pbonzini@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
--cc=raphael.norwitz@nutanix.com \
--cc=stefanha@redhat.com \
--cc=yc-core@yandex-team.ru \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).