Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

["Wang, Wei W" <wei.w.wang@intel.com>]

On Tuesday, February 6, 2018 5:32 PM, Stefan Hajnoczi wrote:
> On Tue, Feb 06, 2018 at 01:28:25AM +0000, Wang, Wei W wrote:
> > On Tuesday, February 6, 2018 12:26 AM, Stefan Hajnoczi wrote:
> > > On Fri, Feb 02, 2018 at 09:08:44PM +0800, Wei Wang wrote:
> > > > On 02/02/2018 01:08 AM, Michael S. Tsirkin wrote:
> > > > > On Tue, Jan 30, 2018 at 08:09:19PM +0800, Wei Wang wrote:
> > > > > > Issues:
> > > > > > Suppose we have both the vhost and virtio-net set up, and 
> > > > > > vhost pmd <-> virtio-net pmd communication works well. Now, 
> > > > > > vhost pmd exits (virtio-net pmd is still there). Some time 
> > > > > > later, we re-run vhost pmd, the vhost pmd doesn't know the 
> > > > > > virtqueue addresses of the virtio-net pmd, unless the 
> > > > > > virtio-net pmd reloads to start the 2nd phase of the 
> > > > > > vhost-user protocol. So the second run of the vhost
> > > pmd won't work.
> > > > > >
> > > > > > Any thoughts?
> > > > > >
> > > > > > Best,
> > > > > > Wei
> > > > > So vhost in qemu must resend all configuration on reconnect.
> > > > > Does this address the issues?
> > > > >
> > > >
> > > > Yes, but the issues are
> > > > 1) there is no reconnecting when a pmd exits (the socket 
> > > > connection seems still on at the device layer);
> > >
> > > This is how real hardware works too.  If the driver suddenly stops 
> > > running then the device remains operational.  When the driver is 
> > > started again it resets the device and initializes it.
> > >
> > > > 2) If we find a way to break the QEMU layer socket connection 
> > > > when pmd exits and get it reconnect, virtio-net device still 
> > > > won't send all the configure when reconnecting, because socket 
> > > > connecting only triggers phase 1 of vhost-user negotiation (i.e.
> > > > vhost_user_init). Phase 2 is triggered after the driver loads 
> > > > (i.e. vhost_net_start). If the virtio-net pmd doesn't reload, 
> > > > there are no phase 2 messages (like virtqueue addresses which 
> > > > are allocated by the pmd). I think we need to think more about 
> > > > this before
> moving forward.
> > >
> > > Marc-André: How does vhost-user reconnect work when the master 
> > > goes away and a new master comes online?  Wei found that the QEMU 
> > > slave implementation only does partial vhost-user initialization 
> > > upon reconnect, so the new master doesn't get the virtqueue 
> > > address and
> related information.
> > > Is this a QEMU bug?
> >
> > Actually we are discussing the slave (vhost is the slave, right?) going away.
> When a slave exits and some moment later a new slave runs, the master
> (virtio-net) won't send the virtqueue addresses to the new vhost slave.
> 
> Yes, apologies for the typo.  s/QEMU slave/QEMU master/
> 
> Yesterday I asked Marc-André for help on IRC and we found the code 
> path where the QEMU master performs phase 2 negotiation upon 
> reconnect.  It's not obvious but the qmp_set_link() calls in net_vhost_user_event() will do it.
> 
> I'm going to try to reproduce the issue you're seeing now.  Will let 
> you know what I find.
> 

OK. Thanks. I observed no messages after re-run virtio-vhost-user pmd, and found there is no re-connection event happening in the device side. 

I also tried to switch the role of client/server - virtio-net to run a server socket, and virtio-vhost-user to run the client, and it seems the current code fails to run that way. The reason is the virtio-net side vhost_user_get_features() doesn't return. On the vhost side, I don't see virtio_vhost_user_deliver_m2s being invoked to deliver the GET_FEATURES message. I'll come back to continue later.

Best,
Wei