From mboxrd@z Thu Jan 1 00:00:00 1970 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754077AbeAKX6Y (ORCPT + 1 other); Thu, 11 Jan 2018 18:58:24 -0500 Received: from mail-bl2nam02on0043.outbound.protection.outlook.com ([104.47.38.43]:11904 "EHLO NAM02-BL2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750941AbeAKX6W (ORCPT ); Thu, 11 Jan 2018 18:58:22 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Thomas.Lendacky@amd.com; Subject: Re: [PATCH v8 03/12] x86/retpoline: Add initial retpoline support To: David Woodhouse , Andi Kleen Cc: Paul Turner , LKML , Linus Torvalds , Greg Kroah-Hartman , Tim Chen , Dave Hansen , tglx@linutronix.de, Kees Cook , Rik van Riel , Peter Zijlstra , Andy Lutomirski , Jiri Kosina , gnomes@lxorguk.ukuu.org.uk, x86@kernel.org, Josh Poimboeuf References: <1515707194-20531-1-git-send-email-dwmw@amazon.co.uk> <1515707194-20531-4-git-send-email-dwmw@amazon.co.uk> From: Tom Lendacky Message-ID: Date: Thu, 11 Jan 2018 17:58:16 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: <1515707194-20531-4-git-send-email-dwmw@amazon.co.uk> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.78.1] X-ClientProxiedBy: DM5PR06CA0070.namprd06.prod.outlook.com (10.168.110.160) To DM5PR12MB1148.namprd12.prod.outlook.com (10.168.236.143) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 206eab08-4058-432a-7570-08d5594f30c1 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020073)(4652020)(48565401081)(5600026)(4604075)(2017052603307)(7153060)(7193020);SRVR:DM5PR12MB1148; X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1148;3:zDVF1tvV/TVDhJcTUCWzM/IsQEAjblf/tJWvCfhBsU+Kpefr7wwKL9zvfWCd4xpT++/yVSHtvGvluNBgGqa+cLkeTQiDS9XOzsM+r/idSckWKO00zskjr87dM7bfcwX/oiXMYdE1tqV0rfruKLNaBhd0a7AgfKN6OwA91kJyF7Krar1/7D0Y/Or1cQwdrb8l8uzmhWwVZM8D2ZS897q93sOepcYN2lwJpDmPh1pwh9Cr3RFkoNuaEcy32TGdoo0B;25:a5YoIOQmAnaIB3fbPPfxyV/68gfIJwmKE6ORCbNaaslMvn9swVFPlauA3B4jg3U/pSrgrZzgJuHhlnTg5UGdGWrxLssplIOh6BoJgTNdX084W5fzdfLsbdkx8NvARGA1slGcesnEbrIiWLCkU61rvI3WlgK0ux6mOUtDemQug2aX0Kvm+SuRxb3fOZRLbiWiAHURM+fAJPcZbYDooKRmp9pQXXFlEN/bFrkYL8M32qK7ZQN/QxxYrI8GQa9fUYkMRXPhF+k4bq7yew9NIGkcZe+Wa+SlHJeNWs1BIpPmOFDntlgmZIxhxVtVHmLgBUA/Yy/biEnU3lTePTlwdkBToQ==;31:tfZBvi793jtvTulAa9MU7Un91TvG9DQSE6WR0sMbVppisHMYAsbnZGqckXvBB3R7hHvZzNhOn8jbjhNX8XMDtE1F9JwxdQwstn+MPhSP6N0bXXV5FEAS9WBGTCLrOYgHQLZqkVaxUZ2BX55qkba45U0mts5z9y7ZTrVBwHzfCQrSKqPMzkOG6Fhqg5Apd0UwtTyeRSuJ4N4se36hQAYOJiaAE67Fu/xaeDTiHQCpIWA= X-MS-TrafficTypeDiagnostic: DM5PR12MB1148: X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1148;20:c3AWU6yyvKaPVaGauIFFiAcG81vt4CguhkBLORMx+LbeOfNTlDnGUTasiWL+F901Oqsr4hmyh2sET1ovwfBcKhRDPdIlBw0NKBBwRbVyxCKpR3SjmxKsIqTFqPagiMjTWD96eHlfRaC3ZBgDHpwJellIRw+0UbDL8RJImrRfhshiQo/kPjkld1U6BXPXpRFV/NRISnD8l/LQHgMvaiJY6Z5nL7GsI+o5qEFiMv7AVRHk/RMAlryPFLsGz3tQZYGdV8alDiXIY6OPCmiIUzOotmT44hNyt/vFRk5tiLMxX4nC93Ga6deZr9Ikk1Coz7KJH3KbKB+g5KB753k7jBY9xp+UHp/dkaP5UfJf7CtBzHx6fK6CWjDJuc/0JNojKEN+os1HfcqEAft3uc2pTQU4N0G8ouq9rNkhiBjj5Gk1fjt4HmXf4bZB2N25gQfIqvCyCeEbPSJz0HCNvl4JWJt0R/6ZS38oCFe5+siXWpQkZcmH288WdfBm5VwG5aR6K9rb X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(211936372134217)(42068640409301)(153496737603132)(146755900322472)(228905959029699); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040470)(2401047)(5005006)(8121501046)(3002001)(93006095)(93001095)(10201501046)(3231023)(944501137)(6055026)(6041268)(20161123560045)(20161123564045)(20161123558120)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011);SRVR:DM5PR12MB1148;BCL:0;PCL:0;RULEID:(100000803101)(100110400095);SRVR:DM5PR12MB1148; X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1148;4:36LRSn94YvfnEVVEapLCW9QhBjR1R0ujHOF9+t8Aj4mhTYOYp6dt1E2X+EegEEiFLYynaxc7X3svM+mxTPEkUYW7s3guRwEKMjfpg48CtWWCvJ7Z1Q4ElWmk/8H515uVUJCjEb+j6qwzcXCkI22Rn6OohEI6pF26cz/W+O4Y9o1nE4rrJHMlRn38c+jMj76olPkdgpTMU7lbMhQrnFW4fEIZ4sgzIUtgxHRuM3we+ALO16i6UjEXdOKdWTrW1dT+XXLK2PB2R2qCAiKgNSBhSMoiL+8/wa+FOR9khf2jqWePhP0QqMGV0zS+flGustuDX2vn/+fabTNTa79wd2IPQ8TdGbcyrHUgsf7tXwEIab7io5Vp6fGuD24L+kJy83ogS2xEwDlqFSLOwp3cj+XTGkGFeirTUH1M4EoQzDl7ncOs2BWl0rMy2iKsDV1L+EhBap2hZINsUkuzSkT8NxqTYA== X-Forefront-PRVS: 0549E6FD50 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6049001)(396003)(39860400002)(376002)(346002)(366004)(39380400002)(189003)(199004)(24454002)(8676002)(72206003)(110136005)(6116002)(58126008)(229853002)(16526018)(3260700006)(305945005)(3846002)(90366009)(16576012)(25786009)(106356001)(54906003)(83506002)(81156014)(6246003)(7416002)(6486002)(65956001)(230700001)(47776003)(36756003)(65806001)(77096006)(81166006)(97736004)(53546011)(7736002)(478600001)(316002)(65826007)(50466002)(8936002)(4326008)(68736007)(386003)(105586002)(31686004)(31696002)(52146003)(966005)(59450400001)(66066001)(76176011)(6666003)(53936002)(5660300001)(6306002)(52116002)(64126003)(86362001)(23676004)(2486003)(2950100002)(2906002);DIR:OUT;SFP:1101;SCL:1;SRVR:DM5PR12MB1148;H:[10.236.65.116];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtETTVQUjEyTUIxMTQ4OzIzOmMwSlBJMmYxT3JpNDh3bGNLNDFIL2Y4bjFG?= =?utf-8?B?QnRZWGsvWjlLRHNCdS9CKytzdGIvUlk2ejlIc2kwQTVKamNiQkVGRW5IMWhL?= =?utf-8?B?aWpqMUpHTXFkRkN1STZUK0JCNSs4NUR2Y0E5Z0tEbjkvdEdyV2NxbkppdDNC?= =?utf-8?B?T3NmelhCZE5QbTVvT1lQdEl4QmpNYzNBUlNmZDM0bVZuUmNzU0xkbDd4cG0w?= =?utf-8?B?RndJV2x1QUE1UnNnUlI4QVFzOTFxZ2gwZm9mb2dERWZKOWZ4OGRjbThSWDQ3?= =?utf-8?B?VGR2VjRyMkZsYnpQajFGdEZud2xRWVpiVytFUmttclFxYlc2VmZrNWRURjAr?= =?utf-8?B?cWhJVG9HdElFM0hnU1RCSmk1U3BGSnFTMTdDTFBCbFNsZE5DWmFNRVNta0w0?= =?utf-8?B?TnBNYkJvbjBwSTcxVTdsandadFNzYUtTajNZandNNXpuaEtiOXliblFMNE9Z?= =?utf-8?B?c0tYeUxKTi9HaHhNMDduQUptU3lqajdnalBrcW9WMEJrN3R3NTJleDVwYTNQ?= =?utf-8?B?QkRaSFQ4cjM0M2pCUG9qQjZFdlVwTlFzUzN0M003VEpxZS9zSXVVa1J6LzFY?= =?utf-8?B?bnNEZm9XVGhOWXVKaGdqZ1RvU3FaSWJEWngzbkJNaUtHZjFSUjQ5ckpWQzBZ?= =?utf-8?B?SmVBa2I3Q3l6eU9nWE11b2ZHRnZhb0QvcFQ1WHN3TnAxcDFHRDRKT1huKytT?= =?utf-8?B?dTgzdmdHcGpmYi9pdnZaNmpCdXYzN3M1MTQ0dTBUeXdhbzFXTXAweGl6ZW5V?= =?utf-8?B?cnE1Ukp5b0NpWHl5c1NpbFg4VmxLNXhxNnU3RVpteThBNG1iaGllM1Babk1u?= =?utf-8?B?WkQzaC9mVVBUY0kwbjFIZFVyWE1ES1B2bldWTnVNNkRVU2FKanpXL0Y0U0pP?= =?utf-8?B?V1BqQTlYTURnMkxKMHFsS3REMTZxeWhKZ0VaaUhyeFE2OTFUVkZ0cTM2V2hv?= =?utf-8?B?aXBtV2EvMnpDOHRkN2hrbFhuUjB0SEpabDFzTEEzKzB1L3ZGQTVQRWNvb01v?= =?utf-8?B?T2VzMFhsUGtPbU5BYThNNjZsWG1ZcUY4Y1RaVXpmdVhid044dXRQYWYxQm5Q?= =?utf-8?B?U1RUV2tud2oxaDlzbUFPVE5jYkdvNERRSTZHSEI4R04yaE91RVpxR2hsem9k?= =?utf-8?B?Nk95dVFaMWY0M0xwV1ZtL2kvcklxLytXOVpIbXZUYy9RVXl3QjdDbzNhMUF5?= =?utf-8?B?MUdxakpKckx2Z0h3MUh5UkxhZnovZHVuNGRhZVVCWjFTWm1yME4ralZkSEJp?= =?utf-8?B?MG5zUzh4ZmIveGhJbGVhSzlMRysxZS9zV2wycjJJbjZJWmIxTDdaNnEydlBj?= =?utf-8?B?akwyQ2diaWdlaEpKZ0c5TitWTnhPU3JqSFRFNHF4ejY2UzJVMjVXeTY2Q29P?= =?utf-8?B?Z3RBZmEvMWRSK2xYZi9JaFZIdmdYcUxqZnQyMHMwNlZjeUluZitzZ2pJeVBn?= =?utf-8?B?UmtDRC9GaVFOcks5VHJnS1NDNmx0T24rUWRnbUZsRDNnY2lobWRnUXFVVStU?= =?utf-8?B?cEI2WEJOOW9IQjRyOUF3Ly9NL3ZiSkcwa2o4YTZaYWF1MlNwbGdEUVFmaHZ1?= =?utf-8?B?QTdzRlJ0REg4OUphUEVoN3lYNWROeGNyN1haUmJaNDRWZ2pCWU93TE5aNGtv?= =?utf-8?B?ZkVYNmlPaXcwR0NWeTBlVVR6VXVTZ3JKdGxsU3JLT1RGTXhuSm5hTFBkajFi?= =?utf-8?B?eFMzK2h3enZ2WmhMWHNYU1BqNEZ3Vlp4aUx5cFhaOTZ0STVXUGc5ajVFNXJD?= =?utf-8?B?NGZVREhHMm53QkhVYmU4TVM0VjNJZUFxR1NnQkJwV0kzMlNQanNpcEw4OWU1?= =?utf-8?B?ZXJPbW1tdXFSaTBZVktnWmlvZ2ppTnpSakhZY0pwQldQdlFiQytQVzhQV2Z4?= =?utf-8?B?NWlVSEo2V1Rwckhqc3MwMklTU3VOdkNtWitESnhCRXBnZUdIcHQwRlJlTVpw?= =?utf-8?B?OU1HUzIxZnVPR1dzL3JJdUw0OUVzYk4xSUJDN3pPQlBzTkFDSERPYmZvUXBC?= =?utf-8?B?emhjNVlvWUdVSVNVc01QVzB2WnZXdU9sdzh6VVp5L05lMklocWtWWHhOM0h6?= =?utf-8?Q?g6iO54SRRdBFjFkUDWZk9/rDV?= X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1148;6:Q1u5YM8njABOe1PxsEWyTe5X5hcA3v7ysL7AWMYOYYlGkoo8jQ/wr/CNUejHrQApqnCLLgVt2BggFnLCU86PPsAhxWterfS8PKVC2+9nT4pxfPv6nSUTmmZu1spAmcsrXPQJZd2FXNGn1dO9MkTqq79P8thrVdlsr//1urP+ha0DpH+IL02Dc7lXpw3GE62cslWUoRvbE1xCWqdGkpFIiKNXb7f5C9xQDBSXdyN4ZLbffdFgclotDVzFfB9BPSMGMXdEv/KGdrotGUBVpHk80RhnIL7IyqgqqsSqUt4GyNLNlNhUmOWHsmjiUNq7vilOgvZtkmZKuX1EYxerNCQ7NhV7rT5PshoTCzK0VMvHkwg=;5:a6FqcnAQsHooEDmxxEhAcBHqksG7pQRukA1iBtpGv5pCIVNQTiZ5vX+bDdJhUpw8AAhAnkJXOnfVNOnDD11i7RbOQrIp13NxyWIXSaIunRA0KBXLVlIJMuASqP6hlEGMQsDU8Yq/BBcunqrcR4QhQMLQ7wi92xPzzJVUVvvST78=;24:JYIGAihCuog3zeEL2CfuxBptebJ/rA1meaHYDAP0BR9joj5/q+BTvFWBAlZ1hX9L0lXiLpm9BBZk01tZgQxOcA2jBuboCTpz/pcbSdJjtmU=;7:96crjhl7csIuKyiLiiriI8n0Ul5wY9h6Caj5GANUecJSHiSLUmgOkmnQTgkWuAL/UR8LMvSJL+6Y0gsegRIIF+XLJDfmAhuKiFlzKJ6xS8Ei4fNUIc+ILZSb2hIuniuWDpVobg1mwcN4B8IYRzDqoszoS1uxfTuNS7e22jljdxKOvumgVx5CfwsKP6c1bqJ7kxZ5fUlA+7SnA4jt182WoTvRsESMI8EPLSKJYTdPHYJmDYkJNUakIY70mQkq2b3C SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1148;20:k/naibqvSwWj87Coo3BbYVZCmf31TUwpP2k4jP8Te+MNNHPvT1RyioNZeDonaY83EWuffr0GUSDRTCCfOTNsLL2120pcdfLIRAErUuEax9v15or/td6VK8cNePDr/St8OS+M3LNfjg3sdVRFstyW9E9uiGfktmnn/5Nu1m7dtdKZYZ01o66uZcRayV/3OfpOw4Oe5GPKnUYXPlVLSOcDldvSGvu2JDOgPzieeUfVS6IFNuvbRnTVVK/9I2j1MDb4 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jan 2018 23:58:19.1660 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 206eab08-4058-432a-7570-08d5594f30c1 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1148 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Return-Path: On 1/11/2018 3:46 PM, David Woodhouse wrote: > Enable the use of -mindirect-branch=thunk-extern in newer GCC, and provide > the corresponding thunks. Provide assembler macros for invoking the thunks > in the same way that GCC does, from native and inline assembler. > > This adds X86_FEATURE_RETPOLINE and sets it by default on all CPUs. In > some circumstances, IBRS microcode features may be used instead, and the > retpoline can be disabled. > > On AMD CPUs if lfence is serialising, the retpoline can be dramatically > simplified to a simple "lfence; jmp *\reg". A future patch, after it has > been verified that lfence really is serialising in all circumstances, can > enable this by setting the X86_FEATURE_RETPOLINE_AMD feature bit in addition > to X86_FEATURE_RETPOLINE. > > Do not align the retpoline in the altinstr section, because there is no > guarantee that it stays aligned when it's copied over the oldinstr during > alternative patching. > > [ Andi Kleen: Rename the macros, add CONFIG_RETPOLINE option, export thunks] > [ tglx: Put actual function CALL/JMP in front of the macros, convert to > symbolic labels ] > [ dwmw2: Convert back to numeric labels, merge objtool fixes ] > > Signed-off-by: David Woodhouse > Signed-off-by: Thomas Gleixner > Acked-by: Arjan van de Ven > Acked-by: Ingo Molnar > Cc: gnomes@lxorguk.ukuu.org.uk > Cc: Rik van Riel > Cc: Andi Kleen > Cc: Peter Zijlstra > Cc: Linus Torvalds > Cc: Jiri Kosina > Cc: Andy Lutomirski > Cc: Dave Hansen > Cc: Kees Cook > Cc: Tim Chen > Cc: Greg Kroah-Hartman > Cc: Paul Turner > Link: https://lkml.kernel.org/r/1515508997-6154-2-git-send-email-dwmw@amazon.co.uk > --- > arch/x86/Kconfig | 13 ++++ > arch/x86/Makefile | 10 +++ > arch/x86/include/asm/asm-prototypes.h | 25 +++++++ > arch/x86/include/asm/cpufeatures.h | 2 + > arch/x86/include/asm/nospec-branch.h | 128 ++++++++++++++++++++++++++++++++++ > arch/x86/kernel/cpu/common.c | 4 ++ > arch/x86/lib/Makefile | 1 + > arch/x86/lib/retpoline.S | 48 +++++++++++++ > 8 files changed, 231 insertions(+) > create mode 100644 arch/x86/include/asm/nospec-branch.h > create mode 100644 arch/x86/lib/retpoline.S > ... > diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h > new file mode 100644 > index 0000000..e20e92e > --- /dev/null > +++ b/arch/x86/include/asm/nospec-branch.h > @@ -0,0 +1,128 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > + > +#ifndef __NOSPEC_BRANCH_H__ > +#define __NOSPEC_BRANCH_H__ > + > +#include > +#include > +#include > + > +#ifdef __ASSEMBLY__ > + > +/* > + * This should be used immediately before a retpoline alternative. It tells > + * objtool where the retpolines are so that it can make sense of the control > + * flow by just reading the original instruction(s) and ignoring the > + * alternatives. > + */ > +.macro ANNOTATE_NOSPEC_ALTERNATIVE > + .Lannotate_\@: > + .pushsection .discard.nospec > + .long .Lannotate_\@ - . > + .popsection > +.endm > + > +/* > + * These are the bare retpoline primitives for indirect jmp and call. > + * Do not use these directly; they only exist to make the ALTERNATIVE > + * invocation below less ugly. > + */ > +.macro RETPOLINE_JMP reg:req > + call .Ldo_rop_\@ > +.Lspec_trap_\@: > + pause Talked with our engineers some more on using pause vs. lfence. Pause is not serializing on AMD, so the pause/jmp loop will use power as it is speculated over waiting for return to mispredict to the correct target. Can this be changed back to lfence? It looked like a very small difference in cycles/time. Thanks, Tom > + jmp .Lspec_trap_\@ > +.Ldo_rop_\@: > + mov \reg, (%_ASM_SP) > + ret > +.endm > + > +/* > + * This is a wrapper around RETPOLINE_JMP so the called function in reg > + * returns to the instruction after the macro. > + */ > +.macro RETPOLINE_CALL reg:req > + jmp .Ldo_call_\@ > +.Ldo_retpoline_jmp_\@: > + RETPOLINE_JMP \reg > +.Ldo_call_\@: > + call .Ldo_retpoline_jmp_\@ > +.endm > + > +/* > + * JMP_NOSPEC and CALL_NOSPEC macros can be used instead of a simple > + * indirect jmp/call which may be susceptible to the Spectre variant 2 > + * attack. > + */ > +.macro JMP_NOSPEC reg:req > +#ifdef CONFIG_RETPOLINE > + ANNOTATE_NOSPEC_ALTERNATIVE > + ALTERNATIVE_2 __stringify(jmp *\reg), \ > + __stringify(RETPOLINE_JMP \reg), X86_FEATURE_RETPOLINE, \ > + __stringify(lfence; jmp *\reg), X86_FEATURE_RETPOLINE_AMD > +#else > + jmp *\reg > +#endif > +.endm > + > +.macro CALL_NOSPEC reg:req > +#ifdef CONFIG_RETPOLINE > + ANNOTATE_NOSPEC_ALTERNATIVE > + ALTERNATIVE_2 __stringify(call *\reg), \ > + __stringify(RETPOLINE_CALL \reg), X86_FEATURE_RETPOLINE,\ > + __stringify(lfence; call *\reg), X86_FEATURE_RETPOLINE_AMD > +#else > + call *\reg > +#endif > +.endm > + > +#else /* __ASSEMBLY__ */ > + > +#define ANNOTATE_NOSPEC_ALTERNATIVE \ > + "999:\n\t" \ > + ".pushsection .discard.nospec\n\t" \ > + ".long 999b - .\n\t" \ > + ".popsection\n\t" > + > +#if defined(CONFIG_X86_64) && defined(RETPOLINE) > + > +/* > + * Since the inline asm uses the %V modifier which is only in newer GCC, > + * the 64-bit one is dependent on RETPOLINE not CONFIG_RETPOLINE. > + */ > +# define CALL_NOSPEC \ > + ANNOTATE_NOSPEC_ALTERNATIVE \ > + ALTERNATIVE( \ > + "call *%[thunk_target]\n", \ > + "call __x86_indirect_thunk_%V[thunk_target]\n", \ > + X86_FEATURE_RETPOLINE) > +# define THUNK_TARGET(addr) [thunk_target] "r" (addr) > + > +#elif defined(CONFIG_X86_32) && defined(CONFIG_RETPOLINE) > +/* > + * For i386 we use the original ret-equivalent retpoline, because > + * otherwise we'll run out of registers. We don't care about CET > + * here, anyway. > + */ > +# define CALL_NOSPEC ALTERNATIVE("call *%[thunk_target]\n", \ > + " jmp 904f;\n" \ > + " .align 16\n" \ > + "901: call 903f;\n" \ > + "902: pause;\n" \ > + " jmp 902b;\n" \ > + " .align 16\n" \ > + "903: addl $4, %%esp;\n" \ > + " pushl %[thunk_target];\n" \ > + " ret;\n" \ > + " .align 16\n" \ > + "904: call 901b;\n", \ > + X86_FEATURE_RETPOLINE) > + > +# define THUNK_TARGET(addr) [thunk_target] "rm" (addr) > +#else /* No retpoline */ > +# define CALL_NOSPEC "call *%[thunk_target]\n" > +# define THUNK_TARGET(addr) [thunk_target] "rm" (addr) > +#endif > + > +#endif /* __ASSEMBLY__ */ > +#endif /* __NOSPEC_BRANCH_H__ */ > diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c > index 372ba3f..7a671d1 100644 > --- a/arch/x86/kernel/cpu/common.c > +++ b/arch/x86/kernel/cpu/common.c > @@ -905,6 +905,10 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c) > setup_force_cpu_bug(X86_BUG_SPECTRE_V1); > setup_force_cpu_bug(X86_BUG_SPECTRE_V2); > > +#ifdef CONFIG_RETPOLINE > + setup_force_cpu_cap(X86_FEATURE_RETPOLINE); > +#endif > + > fpu__init_system(c); > > #ifdef CONFIG_X86_32 > diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile > index 457f681..d435c89 100644 > --- a/arch/x86/lib/Makefile > +++ b/arch/x86/lib/Makefile > @@ -26,6 +26,7 @@ lib-y += memcpy_$(BITS).o > lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o > lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o > lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o > +lib-$(CONFIG_RETPOLINE) += retpoline.o > > obj-y += msr.o msr-reg.o msr-reg-export.o hweight.o > > diff --git a/arch/x86/lib/retpoline.S b/arch/x86/lib/retpoline.S > new file mode 100644 > index 0000000..cb45c6c > --- /dev/null > +++ b/arch/x86/lib/retpoline.S > @@ -0,0 +1,48 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +.macro THUNK reg > + .section .text.__x86.indirect_thunk.\reg > + > +ENTRY(__x86_indirect_thunk_\reg) > + CFI_STARTPROC > + JMP_NOSPEC %\reg > + CFI_ENDPROC > +ENDPROC(__x86_indirect_thunk_\reg) > +.endm > + > +/* > + * Despite being an assembler file we can't just use .irp here > + * because __KSYM_DEPS__ only uses the C preprocessor and would > + * only see one instance of "__x86_indirect_thunk_\reg" rather > + * than one per register with the correct names. So we do it > + * the simple and nasty way... > + */ > +#define EXPORT_THUNK(reg) EXPORT_SYMBOL(__x86_indirect_thunk_ ## reg) > +#define GENERATE_THUNK(reg) THUNK reg ; EXPORT_THUNK(reg) > + > +GENERATE_THUNK(_ASM_AX) > +GENERATE_THUNK(_ASM_BX) > +GENERATE_THUNK(_ASM_CX) > +GENERATE_THUNK(_ASM_DX) > +GENERATE_THUNK(_ASM_SI) > +GENERATE_THUNK(_ASM_DI) > +GENERATE_THUNK(_ASM_BP) > +GENERATE_THUNK(_ASM_SP) > +#ifdef CONFIG_64BIT > +GENERATE_THUNK(r8) > +GENERATE_THUNK(r9) > +GENERATE_THUNK(r10) > +GENERATE_THUNK(r11) > +GENERATE_THUNK(r12) > +GENERATE_THUNK(r13) > +GENERATE_THUNK(r14) > +GENERATE_THUNK(r15) > +#endif >