Hi Stefan, Thanks for the explanation. So reconnection for vhost-user is not a well defined behavior, and QEMU is doing its best to retry when possible, depending on each device. The guest does not know about it, so it's never notified that the device needs to be reset. But what about the vhost-user backend initialization? Does QEMU go again through initializing memory table, vrings, etc... since it can't assume anything from the backend? Thanks, Sebastien ________________________________ From: Stefan Hajnoczi Sent: Tuesday, May 11, 2021 2:45 PM To: Boeuf, Sebastien Cc: virtio-fs@redhat.com; qemu-devel@nongnu.org Subject: vhost-user reconnection and crash recovery Hi Sebastien, On #virtio-fs IRC you asked: I have a vhost-user question regarding disconnection/reconnection. How should this be handled? Let's say the vhost-user backend disconnects, and reconnects later on, does QEMU reset the virtio device by notifying the guest? Or does it simply reconnects to the backend without letting the guest know about what happened? The vhost-user protocol does not have a generic reconnection solution. Reconnection is handled on a case-by-case basis because device-specific and implementation-specific state is involved. The vhost-user-fs-pci device in QEMU has not been tested with reconnection as far as I know. The ideal reconnection behavior is to resume the device from its previous state without disrupting the guest. Device state must survive reconnection in order for this to work. Neither QEMU virtiofsd nor virtiofsd-rs implement this today. virtiofs has a lot of state, making it particularly difficult to support either DEVICE_NEEDS_RESET or transparent vhost-user reconnection. We have discussed virtiofs crash recovery on the bi-weekly virtiofs call (https://etherpad.opendev.org/p/virtiofs-external-meeting). If you want to work on this then joining the call would be a good starting point to coordinate with others. One approach for transparent crash recovery is for virtiofsd to keep its state in tmpfs (e.g. inode/fd mappings) and open fds shared with a clone(2) process via CLONE_FILES. This way the virtiofsd process can terminate but its state persists in memory thanks to its clone process. The clone can then be used to launch the new virtiofsd process from the old state. This would allow the device to resume transparently with QEMU only reconnecting the vhost-user UNIX domain socket. This is an idea that we discussed in the bi-weekly virtiofs call. You mentioned device reset. VIRTIO 1.1 has the Device Status Field DEVICE_NEEDS_RESET flat that the device can use to tell the driver that a reset is necessary. This feature is present in the specification but not implemented in the Linux guest drivers. Again the reason is that handling it requires driver-specific logic for restoring state after reset...otherwise the device reset would be visible to userspace. Stefan --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.