From: "Alex Bennée" <alex.bennee@linaro.org>
To: qemu-devel@nongnu.org,
"rust-vmm@lists.opendev.org" <rust-vmm@lists.opendev.org>
Cc: "Marc-André Lureau" <marcandre.lureau@redhat.com>,
"Dr. David Alan Gilbert" <dgilbert@redhat.com>,
"Sergio Lopez" <slp@redhat.com>,
"Stefan Hajnoczi" <stefanha@redhat.com>,
"Michael S. Tsirkin" <mst@redhat.com>
Subject: vhost reply_ack negotiation (a.k.a differences in vhost-user behaviour with libvhost-user and vhost-user-backend.rs)
Date: Fri, 19 Feb 2021 16:04:34 +0000 [thread overview]
Message-ID: <8735xskm7j.fsf@linaro.org> (raw)
Hi,
I finally got a chance to get down into the guts of vhost-user while
attempting to port my original C RPMB daemon to Rust using the
vhost-user-backend and related crates. I ended up with this hang during
negotiation:
startup
vhost_user_write req:1 flags:0x1
vhost_user_read_start
vhost_user_read req:1 flags:0x5
vhost_user_backend_init: we got 170000000
vhost_user_write req:15 flags:0x1
vhost_user_read_start
vhost_user_read req:15 flags:0x5
vhost_user_set_protocol_features: 2008
vhost_user_write req:16 flags:0x1
vhost_user_write req:3 flags:0x1
vhost_user_write req:1 flags:0x1
vhost_user_read_start
vhost_user_read req:1 flags:0x5
vhost_user_write req:13 flags:0x1
kernel initialises device
virtio_rpmb virtio1: init done!
vhost_user_write req:13 flags:0x1
vhost_dev_set_features: 130000000
vhost_user_set_features: 130000000
vhost_user_write req:2 flags:0x1
vhost_user_write req:5 flags:0x9
vhost_user_read_start
The proximate cause is the vhost crate handling:
MasterReq::SET_MEM_TABLE => {
let res = self.set_mem_table(&hdr, size, &buf, rfds);
self.send_ack_message(&hdr, res)?;
}
which gates on the replay_ack_enabled flag:
fn send_ack_message(
&mut self,
req: &VhostUserMsgHeader<MasterReq>,
res: Result<()>,
) -> Result<()> {
if dbg!(self.reply_ack_enabled) {
let hdr = self.new_reply_header::<VhostUserU64>(req, 0)?;
let val = match res {
Ok(_) => 0,
Err(_) => 1,
};
let msg = VhostUserU64::new(val);
self.main_sock.send_message(&hdr, &msg, None)?;
}
Ok(())
}
which is only set when we have all the appropriate acknowledged flags:
fn update_reply_ack_flag(&mut self) {
let vflag = VhostUserVirtioFeatures::PROTOCOL_FEATURES.bits();
let pflag = VhostUserProtocolFeatures::REPLY_ACK;
if (self.virtio_features & vflag) != 0
&& (self.acked_virtio_features & vflag) != 0
&& self.protocol_features.contains(pflag)
&& (self.acked_protocol_features & pflag.bits()) != 0
{
self.reply_ack_enabled = true;
} else {
self.reply_ack_enabled = false;
}
}
which from above you can see QEMU helpfully dropped those bits in the
reply. It does however work in the C/libvhost version:
virtio_rpmb virtio1: init done!
vhost_user_write req:13 flags:0x1
vhost_dev_set_features: 130000000
vhost_user_set_features: 130000000
vhost_user_write req:2 flags:0x1
vhost_user_write req:37 flags:0x9
vhost_user_read_start
vhost_user_read req:37 flags:0x5
vhost_user_write req:8 flags:0x1
vhost_user_write req:10 flags:0x1
vhost_user_write req:9 flags:0x1
vhost_user_write req:12 flags:0x1
vhost_user_write req:13 flags:0x1
albeit with a slightly different message sequence
(VHOST_USER_ADD_MEM_REG instead of VHOST_USER_SET_MEM_TABLE). Reading
the C code you can see why:
need_reply = vmsg.flags & VHOST_USER_NEED_REPLY_MASK;
reply_requested = vu_process_message(dev, &vmsg);
if (!reply_requested && need_reply) {
vmsg_set_reply_u64(&vmsg, 0);
reply_requested = 1;
}
So regardless of what may have been negotiated it will always reply with
something if the master requested it do so. This points us at the
specification which reads:
- Bit 3 is the need_reply flag - see :ref:`REPLY_ACK <reply_ack>` for
details.
which says in VHOST_USER_PROTOCOL_F_REPLY_ACK that this bit should only
be honoured when the feature has been negotiated. Which brings us to a
series of questions:
- Should QEMU have preserved VhostUserVirtioFeatures::PROTOCOL_FEATURES
when doing the eventual VHOST_USER_SET_FEATURES reply?
- Is vhost.rs being to strict or libvhost-user too lax in interpreting
the negotiated features before processing the ``need_reply`` [Bit 3]
field of the messages?
- are VHOST_USER_SET_MEM_TABLE to VHOST_USER_SET_INFLIGHT_FD included
in the "list of the ones that do" require replies or do they only
reply when REPLY_ACK has been negotiated as the ambiguous "seealso::"
box out seems to imply?
Currently I have some hacks in:
https://github.com/stsquad/vhost/tree/my-hacks
which gets my daemon booting up to the point we actually need to do a
transaction. However I won't submit a PR until I've worked out exactly
where the problems are.
--
Alex Bennée
next reply other threads:[~2021-02-19 17:22 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-19 16:04 Alex Bennée [this message]
2021-02-22 13:06 ` vhost reply_ack negotiation (a.k.a differences in vhost-user behaviour with libvhost-user and vhost-user-backend.rs) Dr. David Alan Gilbert
2021-02-22 13:21 ` Alex Bennée
2021-02-22 13:27 ` Dr. David Alan Gilbert
2021-02-23 10:23 ` [Rust-VMM] " Jiang Liu
2021-02-26 19:58 ` Raphael Norwitz
2021-02-26 21:25 ` Raphael Norwitz
2021-02-27 12:23 ` Alex Bennée
2021-02-23 11:44 ` Michael S. Tsirkin
2021-02-25 4:19 ` [Rust-VMM] " Dylan Reid
2021-02-25 6:36 ` Keiichi Watanabe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8735xskm7j.fsf@linaro.org \
--to=alex.bennee@linaro.org \
--cc=dgilbert@redhat.com \
--cc=marcandre.lureau@redhat.com \
--cc=mst@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=rust-vmm@lists.opendev.org \
--cc=slp@redhat.com \
--cc=stefanha@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).