From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1752581AbbCJOJw (ORCPT <rfc822;w@1wt.eu>);
	Tue, 10 Mar 2015 10:09:52 -0400
Received: from mail-qg0-f42.google.com ([209.85.192.42]:41068 "EHLO
	mail-qg0-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1751879AbbCJOJu (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Tue, 10 Mar 2015 10:09:50 -0400
MIME-Version: 1.0
In-Reply-To: <CALCETrV=n2AYVObh0URscYbRb2o+wBS9KSYOq==V92UXRmpj0Q@mail.gmail.com>
References: <1425926364-9526-1-git-send-email-dvlasenk@redhat.com>
 <1425926364-9526-4-git-send-email-dvlasenk@redhat.com> <20150310125151.GB21686@gmail.com>
 <54FEEF0D.5080505@redhat.com> <20150310132147.GB26185@gmail.com>
 <CALCETrW6-ToMwhfSzA79K2zz6iPNoz0EjPa_wHK7jiJLip8zag@mail.gmail.com>
 <CAK1hOcOjNfyrb3k5euNHDQdc-2kKKi2DkBev+LLkcVd4VUszyQ@mail.gmail.com> <CALCETrV=n2AYVObh0URscYbRb2o+wBS9KSYOq==V92UXRmpj0Q@mail.gmail.com>
From: Denys Vlasenko <vda.linux@googlemail.com>
Date: Tue, 10 Mar 2015 15:09:27 +0100
Message-ID: <CAK1hOcPRtEXayNdJ=djR1tDCf9OLb7yC27fsgezd2hxMOxwJNw@mail.gmail.com>
Subject: Re: [PATCH 3/4] x86: save user rsp in pt_regs->sp on SYSCALL64 fastpath
To: Andy Lutomirski <luto@amacapital.net>
Cc: Ingo Molnar <mingo@kernel.org>, Denys Vlasenko <dvlasenk@redhat.com>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Steven Rostedt <rostedt@goodmis.org>, Borislav Petkov <bp@alien8.de>,
        "H. Peter Anvin" <hpa@zytor.com>, Oleg Nesterov <oleg@redhat.com>,
        Frederic Weisbecker <fweisbec@gmail.com>,
        Alexei Starovoitov <ast@plumgrid.com>, Will Drewry <wad@chromium.org>,
        Kees Cook <keescook@chromium.org>, X86 ML <x86@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset=UTF-8
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Mar 10, 2015 at 3:02 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Tue, Mar 10, 2015 at 7:00 AM, Denys Vlasenko
> <vda.linux@googlemail.com> wrote:
>> On Tue, Mar 10, 2015 at 2:26 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>>> usersp is IMO tolerable.  The nasty thing is the FIXUP_TOP_OF_STACK /
>>> RESTORE_TOP_OF_STACK garbage, and this patch is the main step toward
>>> killing that off completely.  I've still never convinced myself that
>>> there aren't ptrace-related info leaks in there.
>>>
>>> Denys, did you ever benchmark what happens if we use push instead of
>>> mov?  I bet that we get that cycle back and more, not to mention much
>>> less icache usage.
>>
>> Yes, I did.
>> Push conversion seems to perform the same as current, MOV-based code.
>>
>> The expected win there that we lose two huge 12-byte insns
>> which store __USER_CS and __USER_DS in iret frame.
>>
>> MOVQ imm,ofs(%rsp) has a very unfortunate encoding in x86:
>> - needs REX prefix
>> - no sing-extending imm8 form exists for it
>> - ofs in our case can't fit into 8 bits
>> - (%esp) requires SIB byte
>>
>> In my tests, each such instruction adds one cycle.
>>
>> Compare this to PUSH imm8, which is 2 bytes only.
>
> Does that mean that using push on top of this patch gets us our cycle back?

Maybe. I can't be sure about it.

In general I see a jitter of 1-2, sometimes 3 cycles even when I do changes
which merely change code size (e.g. replacing equivalent insns).
This may be caused by jump targets getting aligned differently
wrt cacheline boundaries. If second/third/fourth insn after current one
is not fetched because it did not fit into the cacheline,
then some insn decoders don't get anything to chew on.