stable.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: David Laight <David.Laight@ACULAB.COM>
To: 'Matt Fleming' <matt@codeblueprint.co.uk>,
	Ingo Molnar <mingo@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Thomas Gleixner <tglx@linutronix.de>
Cc: Alexey Dobriyan <adobriyan@gmail.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"Grimm, Jon" <Jon.Grimm@amd.com>,
	"Kumar, Venkataramanan" <Venkataramanan.Kumar@amd.com>,
	Jan Kara <jack@suse.cz>,
	"stable@vger.kernel.org" <stable@vger.kernel.org>
Subject: RE: [PATCH] x86/asm/64: Align start of __clear_user() loop to 16-bytes
Date: Thu, 18 Jun 2020 10:48:05 +0000	[thread overview]
Message-ID: <39f8304b75094f87a54ace7732708d30@AcuMS.aculab.com> (raw)
In-Reply-To: <20200618102002.30034-1-matt@codeblueprint.co.uk>

From: Matt Fleming
> Sent: 18 June 2020 11:20
> x86 CPUs can suffer severe performance drops if a tight loop, such as
> the ones in __clear_user(), straddles a 16-byte instruction fetch
> window, or worse, a 64-byte cacheline. This issues was discovered in the
> SUSE kernel with the following commit,
> 
>   1153933703d9 ("x86/asm/64: Micro-optimize __clear_user() - Use immediate constants")
> 
> which increased the code object size from 10 bytes to 15 bytes and
> caused the 8-byte copy loop in __clear_user() to be split across a
> 64-byte cacheline.
> 
> Aligning the start of the loop to 16-bytes makes this fit neatly inside
> a single instruction fetch window again and restores the performance of
> __clear_user() which is used heavily when reading from /dev/zero.
> 
> Here are some numbers from running libmicro's read_z* and pread_z*
> microbenchmarks which read from /dev/zero:
> 
>   Zen 1 (Naples)
> 
>   libmicro-file
>                                         5.7.0-rc6              5.7.0-rc6              5.7.0-rc6
>                                                     revert-1153933703d9+               align16+
>   Time mean95-pread_z100k       9.9195 (   0.00%)      5.9856 (  39.66%)      5.9938 (  39.58%)
>   Time mean95-pread_z10k        1.1378 (   0.00%)      0.7450 (  34.52%)      0.7467 (  34.38%)
>   Time mean95-pread_z1k         0.2623 (   0.00%)      0.2251 (  14.18%)      0.2252 (  14.15%)
>   Time mean95-pread_zw100k      9.9974 (   0.00%)      6.0648 (  39.34%)      6.0756 (  39.23%)
>   Time mean95-read_z100k        9.8940 (   0.00%)      5.9885 (  39.47%)      5.9994 (  39.36%)
>   Time mean95-read_z10k         1.1394 (   0.00%)      0.7483 (  34.33%)      0.7482 (  34.33%)
> 
> Note that this doesn't affect Haswell or Broadwell microarchitectures
> which seem to avoid the alignment issue by executing the loop straight
> out of the Loop Stream Detector (verified using perf events).

Which cpu was affected?
At least one source (www.agner.org/optimize) implies that both ivy
bridge and sandy bridge have uop caches that mean (If I've read it
correctly) the loop shouldn't be affected by the alignment).

> diff --git a/arch/x86/lib/usercopy_64.c b/arch/x86/lib/usercopy_64.c
> index fff28c6f73a2..b0dfac3d3df7 100644
> --- a/arch/x86/lib/usercopy_64.c
> +++ b/arch/x86/lib/usercopy_64.c
> @@ -24,6 +24,7 @@ unsigned long __clear_user(void __user *addr, unsigned long size)
>  	asm volatile(
>  		"	testq  %[size8],%[size8]\n"
>  		"	jz     4f\n"
> +		"	.align 16\n"
>  		"0:	movq $0,(%[dst])\n"
>  		"	addq   $8,%[dst]\n"
>  		"	decl %%ecx ; jnz   0b\n"

You can do better that that loop.
Change 'dst' to point to the end of the buffer, negate the count
and divide by 8 and you get:
		"0:	movq $0,($[dst],%%ecx,8)\n"
		"	add $1,%%ecx"
		"	jnz 0b\n"
which might run at one iteration per clock especially on cpu that pair
the add and jnz into a single uop.
(You need to use add not inc.)

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)


  reply	other threads:[~2020-06-18 10:48 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-18 10:20 [PATCH] x86/asm/64: Align start of __clear_user() loop to 16-bytes Matt Fleming
2020-06-18 10:48 ` David Laight [this message]
2020-06-18 13:16   ` Alexey Dobriyan
2020-06-18 16:39     ` David Laight
2020-06-18 21:01       ` Alexey Dobriyan
2020-06-19 16:40 ` [tip: x86/urgent] " tip-bot2 for Matt Fleming

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=39f8304b75094f87a54ace7732708d30@AcuMS.aculab.com \
    --to=david.laight@aculab.com \
    --cc=Jon.Grimm@amd.com \
    --cc=Venkataramanan.Kumar@amd.com \
    --cc=adobriyan@gmail.com \
    --cc=jack@suse.cz \
    --cc=linux-kernel@vger.kernel.org \
    --cc=matt@codeblueprint.co.uk \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=stable@vger.kernel.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).