On Fri, Apr 24, 2020 at 03:25:40PM +0900, Chirantan Ekbote wrote:
> Use simple round-robin scheduling based on the `unique` field of the
> fuse request to spread requests across multiple queues, if supported by
> the device.

Multiqueue is not intended to be used this way and this patch will
reduce performance*.  I don't think it should be merged.

* I know it increases performance for you :) but hear me out:

The purpose of multiqueue is for SMP scalability.  It allows queues to
be processed with CPU/NUMA affinity to the vCPU that submitted the
request (i.e. the virtqueue processing thread runs on a sibling physical
CPU core).  Each queue has its own MSI-X interrupt so that completion
interrupts can be processed on the same vCPU that submitted the request.

Spreading requests across queues defeats all this.  Virtqueue processing
threads that are located in the wrong place will now process the
requests.  Completion interrupts will wake up a vCPU that did not submit
the request and IPIs are necessary to notify the vCPU that originally
submitted the request.

Even if you don't care about SMP performance, using multiqueue as a
workaround for missing request parallelism still won't yield the best
results.  The guest should be able to submit up to the maximum queue
depth of the physical storage device.  Many Linux block drivers have max
queue depths of 64.  This would require 64 virtqueues (plus the queue
selection algorithm would have to utilize each one) and shows how
wasteful this approach is.

Instead of modifying the guest driver, please implement request
parallelism in your device implementation.  Device implementations
should pop a virtqueue element, submit I/O, and then move on to the next
virtqueue element instead of waiting for the I/O to complete.  This can
be done with a thread pool, coroutines, async I/O APIs, etc.

The C implementation of virtiofsd has request parallelism and there is
work underway to do it in Rust for cloud-hypervisor too.  Perhaps you
can try the cloud-hypervisor code, I have CCed Sergio Lopez?

Thanks,
Stefan