Re: [PATCHSET v6b 0/11] Turn single segment imports into ITER_UBUF

[Linus Torvalds <torvalds@linux-foundation.org>]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

On Thu, Mar 30, 2023 at 10:33 AM Jens Axboe <axboe@kernel.dk> wrote:
>
> That said, there might be things to improve here. But that's a task
> for another time.

So I ended up looking at this, and funnily enough, the *compat*
version of the "copy iovec from user" is actually written to be a lot
more efficient than the "native" version.

The reason is that the compat version has to load the data one field
at a time anyway to do the conversion, so it open-codes the loop. And
it does it all using the efficient "user_access_begin()" etc, so it
generates good code.

In contrast, the native version just does a "copy_from_user()" and
then loops over the result to verify it. And that's actually pretty
horrid. Doing the open-coded loop that fetches and verifies the iov
entries one at a time should be much better.

I dunno. That's my gut feel, at least. And it may explain why your
"readv()" benchmark has "_copy_from_user()" much higher up than the
"read()" case.

Something like the attached *may* help.

Untested - I only checked the generated assembly to see that it seems
to be sane, but I might have done something stupid. I basically copied
the compat code, fixed it up for non-compat types, and then massaged
it a bit more.

                 Linus

[-- Attachment #2: patch.diff --]
[-- Type: text/x-patch, Size: 1567 bytes --]

 lib/iov_iter.c | 35 ++++++++++++++++++++++++++---------
 1 file changed, 26 insertions(+), 9 deletions(-)

diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 274014e4eafe..e793d6ca299c 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -1731,18 +1731,35 @@ static int copy_compat_iovec_from_user(struct iovec *iov,
 }
 
 static int copy_iovec_from_user(struct iovec *iov,
-		const struct iovec __user *uvec, unsigned long nr_segs)
+		const struct iovec __user *uiov, unsigned long nr_segs)
 {
-	unsigned long seg;
+	int ret = -EFAULT;
 
-	if (copy_from_user(iov, uvec, nr_segs * sizeof(*uvec)))
+	if (!user_access_begin(uiov, nr_segs * sizeof(*uiov)))
 		return -EFAULT;
-	for (seg = 0; seg < nr_segs; seg++) {
-		if ((ssize_t)iov[seg].iov_len < 0)
-			return -EINVAL;
-	}
 
-	return 0;
+	do {
+		void __user *buf;
+		ssize_t len;
+
+		unsafe_get_user(len, &uiov->iov_len, uaccess_end);
+		unsafe_get_user(buf, &uiov->iov_base, uaccess_end);
+
+		/* check for size_t not fitting in ssize_t .. */
+		if (unlikely(len < 0)) {
+			ret = -EINVAL;
+			goto uaccess_end;
+		}
+		iov->iov_base = buf;
+		iov->iov_len = len;
+
+		uiov++; iov++;
+	} while (--nr_segs);
+
+	ret = 0;
+uaccess_end:
+	user_access_end();
+	return ret;
 }
 
 struct iovec *iovec_from_user(const struct iovec __user *uvec,
@@ -1767,7 +1784,7 @@ struct iovec *iovec_from_user(const struct iovec __user *uvec,
 			return ERR_PTR(-ENOMEM);
 	}
 
-	if (compat)
+	if (unlikely(compat))
 		ret = copy_compat_iovec_from_user(iov, uvec, nr_segs);
 	else
 		ret = copy_iovec_from_user(iov, uvec, nr_segs);