From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49CB3C54EBE for ; Thu, 12 Jan 2023 19:57:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D25AC8E000B; Thu, 12 Jan 2023 14:57:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A668F8E0005; Thu, 12 Jan 2023 14:57:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24B468E0007; Thu, 12 Jan 2023 14:57:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 02A4B8E0007 for ; Thu, 12 Jan 2023 14:57:26 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id CD5A71C6067 for ; Thu, 12 Jan 2023 19:57:25 +0000 (UTC) X-FDA: 80347206450.09.8E03993 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf12.hostedemail.com (Postfix) with ESMTP id 1A7A64000B for ; Thu, 12 Jan 2023 19:57:23 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=sUhXWkR9; spf=none (imf12.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673553444; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=2yvU/+KlibzuaJax7jV4X0So1xAo1cwX7MytP6HDPxE=; b=rHNRmdK/w/rz8lZAnS/9KK/YxuhtXRhLt3Xn76SQv5hfg/0sNNS4ph+7zpi+Q0YXSWLdpk NfQ1nDr4n5U208+2WKWgUJYD4dQy/RQ4ahQ0eeVt3PTo6aJnETuTN9NsWOgEgHMrsO45Qm UFdu6dVtTKZOuY2Mh2ABtgRaIhDNr78= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=sUhXWkR9; spf=none (imf12.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673553444; a=rsa-sha256; cv=none; b=BZKpQefq4/7+gl1cpBSvIBn2GeR6vJFsMPhGaLNkmvj8t3OHsCEbg7XS8j6aC1VjjXoHEA 1NkIQsNsRJtbuHJwVTkLEuAn295owp07AahmF4n7etlWycMcssQK9JlYWNSNpD/HA7DWdY MXF/MA9t5QxjYOw6RPvn5lxNpz9x5no= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=2yvU/+KlibzuaJax7jV4X0So1xAo1cwX7MytP6HDPxE=; b=sUhXWkR989HxDuMz0IgNt58RkK nqTN5ZbI/duQQhxiZIzMawumh/twojZHHcaDqklE1ZF5s9tPOmCTfdy15HjBg8WXgPVucBjQvW43I QASwgw9LJ/VCj67WQV8Kbc/Lcf7g3wIR57WUJTSoSCSVZinq9B6HixK86IFvy4TlcucW6OHR2E3Ei tIHujO7fCf/vd7x/FNnvOJuOoXwmDT/3a+HM+YQFyYM+KFp4K7hYwgSSHbdVvWDfVyJkmrNJb8/L7 PjCgd3meHxowC0uYDYr89IC++ERZO4mhecEGYeyICb5FH0IsLoWD/gnKtstWsY8OMoYFxQfEP1FB/ ssGUXlaA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pG3hb-005Oef-UL; Thu, 12 Jan 2023 19:57:32 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 9301630343B; Thu, 12 Jan 2023 20:57:13 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 194442CCF62AF; Thu, 12 Jan 2023 20:57:08 +0100 (CET) Message-ID: <20230112195540.988741683@infradead.org> User-Agent: quilt/0.66 Date: Thu, 12 Jan 2023 20:43:41 +0100 From: Peter Zijlstra To: peterz@infradead.org Cc: richard.henderson@linaro.org, ink@jurassic.park.msu.ru, mattst88@gmail.com, vgupta@kernel.org, linux@armlinux.org.uk, nsekhar@ti.com, brgl@bgdev.pl, ulli.kroll@googlemail.com, linus.walleij@linaro.org, shawnguo@kernel.org, Sascha Hauer , kernel@pengutronix.de, festevam@gmail.com, linux-imx@nxp.com, tony@atomide.com, khilman@kernel.org, krzysztof.kozlowski@linaro.org, alim.akhtar@samsung.com, catalin.marinas@arm.com, will@kernel.org, guoren@kernel.org, bcain@quicinc.com, chenhuacai@kernel.org, kernel@xen0n.name, geert@linux-m68k.org, sammy@sammy.net, monstr@monstr.eu, tsbogend@alpha.franken.de, dinguyen@kernel.org, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, shorne@gmail.com, James.Bottomley@HansenPartnership.com, deller@gmx.de, mpe@ellerman.id.au, npiggin@gmail.com, christophe.leroy@csgroup.eu, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, ysato@users.sourceforge.jp, dalias@libc.org, davem@davemloft.net, richard@nod.at, anton.ivanov@cambridgegreys.com, johannes@sipsolutions.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, jgross@suse.com, srivatsa@csail.mit.edu, amakhalov@vmware.com, pv-drivers@vmware.com, boris.ostrovsky@oracle.com, chris@zankel.net, jcmvbkbc@gmail.com, rafael@kernel.org, lenb@kernel.org, pavel@ucw.cz, gregkh@linuxfoundation.org, mturquette@baylibre.com, sboyd@kernel.org, daniel.lezcano@linaro.org, lpieralisi@kernel.org, sudeep.holla@arm.com, agross@kernel.org, andersson@kernel.org, konrad.dybcio@linaro.org, anup@brainfault.org, thierry.reding@gmail.com, jonathanh@nvidia.com, jacob.jun.pan@linux.intel.com, atishp@atishpatra.org, Arnd Bergmann , yury.norov@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk, dennis@kernel.org, tj@kernel.org, cl@linux.com, rostedt@goodmis.org, mhiramat@kernel.org, frederic@kernel.org, paulmck@kernel.org, pmladek@suse.com, senozhatsky@chromium.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, ryabinin.a.a@gmail.com, glider@google.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, Andrew Morton , jpoimboe@kernel.org, linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-omap@vger.kernel.org, linux-samsung-soc@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, linux-perf-users@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-xtensa@linux-xtensa.org, linux-acpi@vger.kernel.org, linux-pm@vger.kernel.org, linux-clk@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kasan-dev@googlegroups.com, "Rafael J. Wysocki" , Ulf Hansson Subject: [PATCH v3 27/51] cpuidle,sched: Remove annotations from TIF_{POLLING_NRFLAG,NEED_RESCHED} References: <20230112194314.845371875@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Rspamd-Queue-Id: 1A7A64000B X-Stat-Signature: 3pcg1xi8iyqzhrpy6hcnjauwedtfd9y9 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1673553443-529133 X-HE-Meta: U2FsdGVkX19p/C/dqoquOjxEA36PEqqYHmVSDrZZ7htSntLiJQsFGNUZkvCxvEPXH65vDExVJVCv+Fn+ITleG5er2op23fFnfe5h+pwv1lBD6uzFSxyFNF/Uu1/tm6OifZbBqi25Xsv6UaV+3ttbec7vCP3mK43JtlMVm/sfttVlYMIKSYHKlgSgvpggCFR90H3K7arIdLM187iNFOHoh3f4Q8fTIegy5Iov1+gxhvQ0pzQdvvRyLtYYGzLNAwpih9sig3B+HejkmhxRBtZeJxqp32eBj8T5kEc433jUcYo0yN+5hLWLNhAxX769uNn8onWmXhkHpgbmoXd1og4mxFuI9if7qO454b7wlPArsAwAm6OkHHVsVDPQ+c7cL8/V01ek72OxB0U4q8ZJKz41EPsC+IUswJjwKz/KNeZuKgTkjfG70PNCjw6KvVBMiqxntx213W6aOiryfPNMEdbTUQsMZOEP0t7AlWds9B8ZXF+oxIGEEfVIC351TEMsJAbjfDbHmCYaRYfDgIdZh7n3OJeu1p6hwuDxQUedGskijkgkiH11ReB2oULMRyoEFoc7LE5GH2uWzAXGZmY3jAnksgFZMhok1Jg3zgU4OyJH/A1FLjfn8msHrvjz8ZG1cyB4OzFigpGGIs0l3rPAeMwCGh6AIAVlzZ0Lzj7LpbNbJ66M5HsGUbhLorimRJGNvHBJ2q/CdDvboxN7dsd3MY7YuFXDLmy+HKNKtHODJbbp6VJYb0naVAyGPN/UiNnYFVdT2vsSTwhd2pUwbM32esyKSZWEXHcsec8gip4tDyJ5LU5xR8ZZFRgEcJ6KwWmTCPqn7ILQSg4bbXLRE5rn58a6k9rwxiDwUDGeFryufQtULDmkCcUYpIvNhAPMD61MAAB+qcDLGkx/8ctz3e7zagJtmmpnIVRILPV8u5jGVNHSICgsY5mx7RFgRbHKDzcwy59c5aWm3HvW4q8pVUpcvLN CzeyM2Rw fAjQy7TgTrE8RUXduyuDD/v+YTVp+sbCGJZ9WL23pVPsWQqkx5ICodhG+TjV34TeJwoP2caulvLewZpEop/SLLM4OWHymiURJoEHzqrgywdaTQEOW0fuRViQ3wJvalWboGNaQ7/aSlYSfkjbIcXccEZOjUGoTgkqukLPJhqc8HmaSp+XNOCK3yj32K1WV0ZtJ+/R909rZycvJV+1vHpuagL76owrjADJmiV4Ze80868jgf4om3K1MNPq/rdrn+9z/GvCiPz6zVlwzVe3mWsg5E96VGzTnnBv712TbMwsuh1XJ1E5G1RXtkJxrB8N6yONXfgbcNBzbURZuWMeTkWGBWnpOQ98AzJ1Zd4TaUkWojOit8kW7ZuYU5nTIfvgbpxHS8UpxuGpaTpnxVAYihO9trH4yO7STDKa/cqJ//oJr8gtVDEDf9d+jkwPHJIPiS3BXnfkTs/URIAAIx60/UZmP6segPqCTzkNHZMroXbBbR+Z067WStPjaVErdSjTrYy8ZvokepHidAlxr57u9Ql0ucao2NBl9X0SnaGMf7nMRT4tomYFb2a24OmkXNuIEjTrAz1UvAqFDdKs5vJFPmlrP8dBx48Z9SjEMsQr616akwAnkQe912my/XUOJ0W7Vk3TYVqxMMfdVrSI9CC+XJAKqKXBX/23BvhpAAaA+N6AQiC4A1T1kAl4Tlp9yNogUIBFsmrQ4bIwHKcrDua2AHIwiZ5zPhkt+jx5mDRFUMDnqQT+wxSPTNTYtWWuD8PmdQOOUdZqa16vFS70GQ16U4JPC/eWlQX9nFPrbQ7nVhFnTHF5B/OtMX0b/eBazxTtyNNnQf18f74e/AXl/+LiJCxTKzH9XeSZ2BQWhPNhZfQ+47AnfYsgH9wOkA9dWlpkqwu67hil0rUxc25YTkthxh87+tfvCkto4H8HA/yMrrP9i9NmAABv89eAtihjBqiTiMgJSJssKH2f8qex4N8vdV5XyzlpUAS9c IchkP1X9 ccPs5/oyd5LF2VMt/rVleMla9/Bh/B1Mb0XWjtSDtixQcz2yframqY/62GQIodRIhCiqFLHlkxB/FV9On82J6F4ylz0+CUrcVlMoiuiGwN10bTT/a4jvy2hUzg8Le7usTqW6TgI40ijD2McgF4i0lI+cVx8yohJPPhbB7jhUDS+tIMeEyaynT0yFMO/Hyl1LdxTM8eMBawXXQW48pULzUO7tiCTv6BqJY45Lo2ppFykmz7S4k5syTi5SPDcqR2z0Ufcfj97tWp8BiILt2YYiwda/wvtnjb7ekGo6e9xlyLJ/KpEvj1+vHNx4jKeibSlG3bght+KSXvNfllZIZwuMXPFd9E6owofkuTAu0WguoIrBKcKkJSfcGusN+ws9OR6+kNtR7b93uKlHjAfMnD0SXhnf69jre/NV2wlUrtj6AgJYfFJAM6ZlP+8H4E13fwqOWR7ez4+kwMjPtEXYCpaHZtw1COS5eKl71G4N0Xi2uvwD5lLCG/Zu2Xv99DUBpCJ1MgRQXGF0/yeeOXL3rkHh7PsniT7xufhp4ZsTjaD933zJM73WnRUujOg7iuL0iiK+15JCI4Va4SVWLJ9J4RiVVNgcR6+hpQvlEKSeZOl0hKmmVthgBXkZQ7RnDj8QpzMDg9aBeebiN6rF+vkQe7gzduMzXO4nAHEtPamh2hPYy6x9IwnY/3Yqzq82703F71/qO64gCpoLrylam65oXxv5qsF16zfC9bLFo828Oxy/zGR1TlpWIV6kBEc60KnEe6m/9u72cipRAtpVr7rjU6nYGgcrx1j5q0GbijcIX8LtiIoj1+N4ugIwrrGqw22MQ2+8g39H45RmgQp+y5YYNmTjZYp2+SLaAKhEwDe4irraA9RVY1/UEtoroOnt7y7NNTJMiIZG9FL8PBtuSpUbn5Pu9dd4jXSzF/cHdhbia/AJJEs88vuNSUuswZAZ3tG7MvX7d1JN1bU+ZQUDiEOujo/WF3PT+8tGg avEawGJ1 ap2lQkpX8BDTQphgGEATDYprA0ormZY2e0dBjI62Lqwpf65jLy0b5p4oHsazr2JbIDqcdf9HWMjM8VtpKDpnFhtLYyDBSw243ChlbvOLMuHI94wT8EyilnfQ7dqz9NMP8HyQAn0zDGQhUsKE4dO3+92InrEWEqMo1iXSDfGYWAA240E6BlxhTLtS57xGKc8f+VbzvntKXAVlKYla3LTUycqwJBy6jIMsA9pqdHV+zRxc6Nglnz09qW4//B471Vhdqh9x0gFffwt7xoWT1fTRx637M55U58fGp43cZiKERoiOJ5Iwdro58jxQwqNc6PnVOirjvFI3P8P45b6m8MEivFhoc/zCFUSyt75xousHsPI56cMKbDJh0oSmbjqnibeplFyo/Ybvp6tmG2nc0Bw2TRgQY/QnKB6synprWoQjO1qc/juufHajCw2Cn6nSG87S1pZ6u7oqdE+aRoIoYIbn+1GPwsFSRXNVdqX6TDne/TjH5jao7ZBGqwDSB5vpIab5cwBA8P7Fr5ry/u4NiR3FtdegJox1NsyicU98nPSiieltpg+WGzSW/tghrNYkNayFtYSx82UqcA8RUooSuEKk7iLWnBTiJXNH5rvL9ZA7I5YW7hHuf+NEEHy2SqEak1qR/CCQfcaiX78h2vTxn0NqieOlyOv2RniEmIMHXcNOnnLIhJKUGKB/bSK5lKcB66a23zsY/SEicRJYfFTf5feEmx6mu1QjpTDQU75rxqI2jZ9FnQKNRsP7wgKG4QD9w3CwZGcTevMYHMv0irGQvkxIPOKnApptptuZUDUciPK8re5mjnLUsth/0heSsij30AQbIFi6LrHtJNa3VtPzk1B3L4mJ3jcNM6Fi5L7ygEXYWs/+gZtmq/UhqA2ZSuuLuFUXMkrmLjr9zIk6apVaBPg7Y40karvxztK+C7yBdr36ezxod4aEwSkyCYyCMSouJtvGebPtjvYBY9ytRURw2VTb1T5Bd8+CS 1y5UrAse E3gn0qmU6lzgdFC6+RBt+O848gSxAcsRT+0oL0Uyyg74J3v2zygZQkZl4YA1ndIOuKazU4RQycGFBqudsEkMV3GgigmQtmlfleLK+7ZKzOwLdLNVlK9jgwJx31pkEZDd4IbYebJDzVsp6FPtJTmuq3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vmlinux.o: warning: objtool: mwait_idle+0x5: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0xc5: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x73: call to test_ti_thread_flag() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0xbc: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0xea: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0xb4: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0xa6: call to current_clr_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0xbf: call to current_clr_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0xa1: call to current_clr_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: mwait_idle+0xe: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0xc5: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x73: call to test_ti_thread_flag() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0xbc: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0xea: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0xb4: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x73: call to test_ti_thread_flag() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0x73: call to test_ti_thread_flag.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0x91: call to test_ti_thread_flag.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0x78: call to test_ti_thread_flag.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_safe_halt+0xf: call to test_ti_thread_flag.constprop.0() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) Acked-by: Rafael J. Wysocki Acked-by: Frederic Weisbecker Tested-by: Tony Lindgren Tested-by: Ulf Hansson --- include/linux/sched/idle.h | 40 ++++++++++++++++++++++++++++++---------- include/linux/thread_info.h | 18 +++++++++++++++++- 2 files changed, 47 insertions(+), 11 deletions(-) --- a/include/linux/sched/idle.h +++ b/include/linux/sched/idle.h @@ -23,12 +23,37 @@ static inline void wake_up_if_idle(int c */ #ifdef TIF_POLLING_NRFLAG -static inline void __current_set_polling(void) +#ifdef _ASM_GENERIC_BITOPS_INSTRUMENTED_ATOMIC_H + +static __always_inline void __current_set_polling(void) { - set_thread_flag(TIF_POLLING_NRFLAG); + arch_set_bit(TIF_POLLING_NRFLAG, + (unsigned long *)(¤t_thread_info()->flags)); } -static inline bool __must_check current_set_polling_and_test(void) +static __always_inline void __current_clr_polling(void) +{ + arch_clear_bit(TIF_POLLING_NRFLAG, + (unsigned long *)(¤t_thread_info()->flags)); +} + +#else + +static __always_inline void __current_set_polling(void) +{ + set_bit(TIF_POLLING_NRFLAG, + (unsigned long *)(¤t_thread_info()->flags)); +} + +static __always_inline void __current_clr_polling(void) +{ + clear_bit(TIF_POLLING_NRFLAG, + (unsigned long *)(¤t_thread_info()->flags)); +} + +#endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_ATOMIC_H */ + +static __always_inline bool __must_check current_set_polling_and_test(void) { __current_set_polling(); @@ -41,12 +66,7 @@ static inline bool __must_check current_ return unlikely(tif_need_resched()); } -static inline void __current_clr_polling(void) -{ - clear_thread_flag(TIF_POLLING_NRFLAG); -} - -static inline bool __must_check current_clr_polling_and_test(void) +static __always_inline bool __must_check current_clr_polling_and_test(void) { __current_clr_polling(); @@ -73,7 +93,7 @@ static inline bool __must_check current_ } #endif -static inline void current_clr_polling(void) +static __always_inline void current_clr_polling(void) { __current_clr_polling(); --- a/include/linux/thread_info.h +++ b/include/linux/thread_info.h @@ -177,7 +177,23 @@ static __always_inline unsigned long rea clear_ti_thread_flag(task_thread_info(t), TIF_##fl) #endif /* !CONFIG_GENERIC_ENTRY */ -#define tif_need_resched() test_thread_flag(TIF_NEED_RESCHED) +#ifdef _ASM_GENERIC_BITOPS_INSTRUMENTED_NON_ATOMIC_H + +static __always_inline bool tif_need_resched(void) +{ + return arch_test_bit(TIF_NEED_RESCHED, + (unsigned long *)(¤t_thread_info()->flags)); +} + +#else + +static __always_inline bool tif_need_resched(void) +{ + return test_bit(TIF_NEED_RESCHED, + (unsigned long *)(¤t_thread_info()->flags)); +} + +#endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_NON_ATOMIC_H */ #ifndef CONFIG_HAVE_ARCH_WITHIN_STACK_FRAMES static inline int arch_within_stack_frames(const void * const stack,