Re: Follow up on UBD discussion

From: Ming Lei <ming.lei@redhat.com>
To: Gabriel Krisman Bertazi <krisman@collabora.com>
Cc: linux-block@vger.kernel.org
Subject: Re: Follow up on UBD discussion
Date: Tue, 3 May 2022 16:02:42 +0800	[thread overview]
Message-ID: <YnDhorlKgOKiWkiz@T590> (raw)
In-Reply-To: <874k27rfwm.fsf@collabora.com>

Hello Gabriel,

CC linux-block and hope you don't mind, :-)

On Mon, May 02, 2022 at 01:41:13PM -0400, Gabriel Krisman Bertazi wrote:
> 
> Hi Ming,
> 
> First of all, I hope I didn't put you on the spot too much during the
> discussion.  My original proposal was to propose my design, but your
> implementation quite solved the questions I had. :)

I think that is open source, then we can put efforts together to make things
better.

> 
> I'd like to follow up to restart the communication and see
> where I can help more with UBD.  As I said during the talk, I've
> done some fio runs and I was able to saturate NBD much faster than UBD:
> 
> https://people.collabora.com/~krisman/mingl-ubd/bw.png

Yeah, that is true since NBD has extra socket communication cost which
can't be efficient as io_uring.

> 
> I've also wrote some fixes to the initialization path, which I
> planned to send to you as soon as you published your code, but I think
> you might want to take a look already and see if you want to just squash
> it into your code base.
> 
> I pushed those fixes here:
> 
>   https://gitlab.collabora.com/krisman/linux/-/tree/mingl-ubd

I have added the 1st fix and 3rd patch into my tree:

https://github.com/ming1/linux/commits/v5.17-ubd-dev

The added check in 2nd patch is done lockless, which may not be reliable
enough, so I didn't add it. Also adding device is in slow path, and no
necessary to improve in that code path.

I also cleaned up ubd driver a bit: debug code cleanup, remove zero copy
code, remove command of UBD_IO_GET_DATA and always make ubd driver
builtin.

ubdsrv part has been cleaned up too:

https://github.com/ming1/ubdsrv

> 
> I'm looking into adding support for multiple driver queues next, and
> should be able to share some patches on that shortly.

OK, please post them on linux-block so that more eyes can look at the
code, meantime the ubdsrv side needs to handle MQ too.

Sooner or later, the single ubdsrv task may be saturated by copying data and
io_uring command communication only, which can be shown by running io on
ubd-null target. In my lattop, the ubdsrv cpu utilization is close to
90% when IOPS is > 500K. So MQ may help some fast backing cases.

Thanks,
Ming