From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751474AbdFAHA2 (ORCPT ); Thu, 1 Jun 2017 03:00:28 -0400 Received: from mail.fireflyinternet.com ([109.228.58.192]:64415 "EHLO fireflyinternet.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751013AbdFAHA1 (ORCPT ); Thu, 1 Jun 2017 03:00:27 -0400 X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; From: Chris Wilson To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, intel-gfx@lists.freedesktop.org, Chris Wilson , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" Subject: [PATCH 1/3] x86-32: Teach copy_from_user to unroll .size=6/8 Date: Thu, 1 Jun 2017 07:58:41 +0100 Message-Id: <20170601065843.2392-2-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170601065843.2392-1-chris@chris-wilson.co.uk> References: <20170601065843.2392-1-chris@chris-wilson.co.uk> X-Originating-IP: 78.156.65.138 X-Country: code=GB country="United Kingdom" ip=78.156.65.138 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Two exception handling register moves are faster to inline than a call to __copy_user_ll(). We already apply the conversion for a get_user() call, so for symmetry we should also apply the optimisation to copy_from_user. Signed-off-by: Chris Wilson Cc: Thomas Gleixner Cc: Ingo Molnar Cc: "H. Peter Anvin" --- arch/x86/include/asm/uaccess_32.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/uaccess_32.h b/arch/x86/include/asm/uaccess_32.h index aeda9bb8af50..44d17d1ab07c 100644 --- a/arch/x86/include/asm/uaccess_32.h +++ b/arch/x86/include/asm/uaccess_32.h @@ -23,30 +23,47 @@ static __always_inline unsigned long raw_copy_from_user(void *to, const void __user *from, unsigned long n) { if (__builtin_constant_p(n)) { - unsigned long ret; + unsigned long ret = 0; switch (n) { case 1: - ret = 0; __uaccess_begin(); __get_user_asm_nozero(*(u8 *)to, from, ret, "b", "b", "=q", 1); __uaccess_end(); return ret; case 2: - ret = 0; __uaccess_begin(); __get_user_asm_nozero(*(u16 *)to, from, ret, "w", "w", "=r", 2); __uaccess_end(); return ret; case 4: - ret = 0; __uaccess_begin(); __get_user_asm_nozero(*(u32 *)to, from, ret, "l", "k", "=r", 4); __uaccess_end(); return ret; + case 6: + __uaccess_begin(); + __get_user_asm_nozero(*(u32 *)to, from, ret, + "l", "k", "=r", 6); + if (likely(!ret)) + __get_user_asm_nozero(*(u16 *)(4 + (char *)to), + (u16 __user *)(4 + (char __user *)from), + ret, "w", "w", "=r", 2); + __uaccess_end(); + return ret; + case 8: + __uaccess_begin(); + __get_user_asm_nozero(*(u32 *)to, from, ret, + "l", "k", "=r", 8); + if (likely(!ret)) + __get_user_asm_nozero(*(u32 *)(4 + (char *)to), + (u32 __user *)(4 + (char __user *)from), + ret, "l", "k", "=r", 4); + __uaccess_end(); + return ret; } } return __copy_user_ll(to, (__force const void *)from, n); -- 2.11.0 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Wilson Subject: [PATCH 1/3] x86-32: Teach copy_from_user to unroll .size=6/8 Date: Thu, 1 Jun 2017 07:58:41 +0100 Message-ID: <20170601065843.2392-2-chris@chris-wilson.co.uk> References: <20170601065843.2392-1-chris@chris-wilson.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7B6DF6E2F4 for ; Thu, 1 Jun 2017 07:00:20 +0000 (UTC) In-Reply-To: <20170601065843.2392-1-chris@chris-wilson.co.uk> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To: linux-kernel@vger.kernel.org Cc: intel-gfx@lists.freedesktop.org, x86@kernel.org, Ingo Molnar , "H. Peter Anvin" , Thomas Gleixner List-Id: intel-gfx@lists.freedesktop.org VHdvIGV4Y2VwdGlvbiBoYW5kbGluZyByZWdpc3RlciBtb3ZlcyBhcmUgZmFzdGVyIHRvIGlubGlu ZSB0aGFuIGEgY2FsbAp0byBfX2NvcHlfdXNlcl9sbCgpLiBXZSBhbHJlYWR5IGFwcGx5IHRoZSBj b252ZXJzaW9uIGZvciBhIGdldF91c2VyKCkKY2FsbCwgc28gZm9yIHN5bW1ldHJ5IHdlIHNob3Vs ZCBhbHNvIGFwcGx5IHRoZSBvcHRpbWlzYXRpb24gdG8KY29weV9mcm9tX3VzZXIuCgpTaWduZWQt b2ZmLWJ5OiBDaHJpcyBXaWxzb24gPGNocmlzQGNocmlzLXdpbHNvbi5jby51az4KQ2M6IFRob21h cyBHbGVpeG5lciA8dGdseEBsaW51dHJvbml4LmRlPgpDYzogSW5nbyBNb2xuYXIgPG1pbmdvQHJl ZGhhdC5jb20+CkNjOiAiSC4gUGV0ZXIgQW52aW4iIDxocGFAenl0b3IuY29tPgotLS0KIGFyY2gv eDg2L2luY2x1ZGUvYXNtL3VhY2Nlc3NfMzIuaCB8IDI1ICsrKysrKysrKysrKysrKysrKysrKy0t LS0KIDEgZmlsZSBjaGFuZ2VkLCAyMSBpbnNlcnRpb25zKCspLCA0IGRlbGV0aW9ucygtKQoKZGlm ZiAtLWdpdCBhL2FyY2gveDg2L2luY2x1ZGUvYXNtL3VhY2Nlc3NfMzIuaCBiL2FyY2gveDg2L2lu Y2x1ZGUvYXNtL3VhY2Nlc3NfMzIuaAppbmRleCBhZWRhOWJiOGFmNTAuLjQ0ZDE3ZDFhYjA3YyAx MDA2NDQKLS0tIGEvYXJjaC94ODYvaW5jbHVkZS9hc20vdWFjY2Vzc18zMi5oCisrKyBiL2FyY2gv eDg2L2luY2x1ZGUvYXNtL3VhY2Nlc3NfMzIuaApAQCAtMjMsMzAgKzIzLDQ3IEBAIHN0YXRpYyBf X2Fsd2F5c19pbmxpbmUgdW5zaWduZWQgbG9uZwogcmF3X2NvcHlfZnJvbV91c2VyKHZvaWQgKnRv LCBjb25zdCB2b2lkIF9fdXNlciAqZnJvbSwgdW5zaWduZWQgbG9uZyBuKQogewogCWlmIChfX2J1 aWx0aW5fY29uc3RhbnRfcChuKSkgewotCQl1bnNpZ25lZCBsb25nIHJldDsKKwkJdW5zaWduZWQg bG9uZyByZXQgPSAwOwogCiAJCXN3aXRjaCAobikgewogCQljYXNlIDE6Ci0JCQlyZXQgPSAwOwog CQkJX191YWNjZXNzX2JlZ2luKCk7CiAJCQlfX2dldF91c2VyX2FzbV9ub3plcm8oKih1OCAqKXRv LCBmcm9tLCByZXQsCiAJCQkJCSAgICAgICJiIiwgImIiLCAiPXEiLCAxKTsKIAkJCV9fdWFjY2Vz c19lbmQoKTsKIAkJCXJldHVybiByZXQ7CiAJCWNhc2UgMjoKLQkJCXJldCA9IDA7CiAJCQlfX3Vh Y2Nlc3NfYmVnaW4oKTsKIAkJCV9fZ2V0X3VzZXJfYXNtX25vemVybygqKHUxNiAqKXRvLCBmcm9t LCByZXQsCiAJCQkJCSAgICAgICJ3IiwgInciLCAiPXIiLCAyKTsKIAkJCV9fdWFjY2Vzc19lbmQo KTsKIAkJCXJldHVybiByZXQ7CiAJCWNhc2UgNDoKLQkJCXJldCA9IDA7CiAJCQlfX3VhY2Nlc3Nf YmVnaW4oKTsKIAkJCV9fZ2V0X3VzZXJfYXNtX25vemVybygqKHUzMiAqKXRvLCBmcm9tLCByZXQs CiAJCQkJCSAgICAgICJsIiwgImsiLCAiPXIiLCA0KTsKIAkJCV9fdWFjY2Vzc19lbmQoKTsKIAkJ CXJldHVybiByZXQ7CisJCWNhc2UgNjoKKwkJCV9fdWFjY2Vzc19iZWdpbigpOworCQkJX19nZXRf dXNlcl9hc21fbm96ZXJvKCoodTMyICopdG8sIGZyb20sIHJldCwKKwkJCQkJICAgICAgImwiLCAi ayIsICI9ciIsIDYpOworCQkJaWYgKGxpa2VseSghcmV0KSkKKwkJCQlfX2dldF91c2VyX2FzbV9u b3plcm8oKih1MTYgKikoNCArIChjaGFyICopdG8pLAorCQkJCQkJICAgICAgKHUxNiBfX3VzZXIg KikoNCArIChjaGFyIF9fdXNlciAqKWZyb20pLAorCQkJCQkJICAgICAgcmV0LCAidyIsICJ3Iiwg Ij1yIiwgMik7CisJCQlfX3VhY2Nlc3NfZW5kKCk7CisJCQlyZXR1cm4gcmV0OworCQljYXNlIDg6 CisJCQlfX3VhY2Nlc3NfYmVnaW4oKTsKKwkJCV9fZ2V0X3VzZXJfYXNtX25vemVybygqKHUzMiAq KXRvLCBmcm9tLCByZXQsCisJCQkJCSAgICAgICJsIiwgImsiLCAiPXIiLCA4KTsKKwkJCWlmIChs aWtlbHkoIXJldCkpCisJCQkJX19nZXRfdXNlcl9hc21fbm96ZXJvKCoodTMyICopKDQgKyAoY2hh ciAqKXRvKSwKKwkJCQkJCSAgICAgICh1MzIgX191c2VyICopKDQgKyAoY2hhciBfX3VzZXIgKilm cm9tKSwKKwkJCQkJCSAgICAgIHJldCwgImwiLCAiayIsICI9ciIsIDQpOworCQkJX191YWNjZXNz X2VuZCgpOworCQkJcmV0dXJuIHJldDsKIAkJfQogCX0KIAlyZXR1cm4gX19jb3B5X3VzZXJfbGwo dG8sIChfX2ZvcmNlIGNvbnN0IHZvaWQgKilmcm9tLCBuKTsKLS0gCjIuMTEuMAoKX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KSW50ZWwtZ2Z4IG1haWxpbmcg bGlzdApJbnRlbC1nZnhAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRl c2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vaW50ZWwtZ2Z4Cg==