* [PATCH v8 00/14] Function Granular KASLR
@ 2021-12-02 22:32 Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 01/14] x86: Makefile: Add build and config option for CONFIG_FG_KASLR Alexander Lobakin
` (14 more replies)
0 siblings, 15 replies; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
This is a massive rework and a respin of Kristen Accardi's marvellous
FG-KASLR series (v5).
The major differences since v5 [0]:
* You can now tune the number of functions per each section to
achieve the preferable vmlinux size or protection level. Default
is still as one section per function.
This can be handy for storage-constrained systems. 4-8 fps are
still strong, but reduce the size of the final vmlinu{x,z}
significantly (see the comparison below);
* I don't use orphan sections anymore. It's not reliable at all /
may differ from linker to linker, and also conflicts with
CONFIG_LD_ORPHAN_WARN which is great for catching random bugs ->
* All the .text.* sections are now being described explicitly in the
linker script. A Perl script is used to take the original LDS, the
original object file, read a list of input sections from it and
generate the resulting LDS.
This costs a bit of linking time as LD tends to think hard when
processing scripts > 1 Mb (a subject for future BFD and LLD
patches). It adds about 60-80 seconds to the whole linking process
(BTF step, 2-3 kallsyms steps and the final step), but "better
safe than sorry".
In addition, that approach allows to reserve some space at the end
of text (8-12 Kb, no impact on vmlinux size as THP-aligned (2 Mb)
rodata goes right after it) and add some link-time assertions ->
* Input .text section now must be empty, otherwise the linkage will
be stopped. This is implemented by the size assertion in the
resulting LD script and is designed to plug the potentional layout
leakage. This also means that ->
* "Regular" ASM functions are now being placed into unique separate
functions the same way compiler does this for C functions. This is
achieved by introducing and using several new macros which take
the symbol name as a base for its new section name.
This gives a better opportunity to LTO, DCE and FG-KASLR, as ASM
code can now also be randomized or garbage-collected;
* It's now fully compatible with ClangLTO, ClangCFI,
CONFIG_LD_ORPHAN_WARN and some all the rest stuff landed since the
last revision has been published;
* Includes several fixes: relocations inside .altinstr_replacement
code and minor style issues found and/or suggested by LKP robot.
The series has been compile-time and runtime tested on the following
setups with no issues:
- x86_64, GCC 11, Binutils 2.35;
- x86_64, Clang/LLVM 13, ClangLTO + ClangCFI (from Sami's tree).
Some numbers for comparison:
* make -j65 -- time of the full kernel compilation with the named
option enabled (and -j$(($(nproc) + 1))), give to see mainly how
linkers choke on big LD scripts;
* boot -- time elapsed from starting the kernel by the bootloader
to login prompt, affected mostly by the main FG-KASLR preboot
loop which shuffles function sections;
* vmlinux.o -- the size of the final vmlinux.o, altered by relocs
and -ffunction-sections;
* vmlinux -- the size of the final vmlinux, depends directly on the
number of (function) sections;
* bzImage -- the size of the final compressed kernel, same as with
vmlinux;
* bogoops/s -- stress-ng -c$(nproc) results on the kernel with the
named feature enabled;
* fps -- the number of functions per section, controlled by
CONFIG_FG_KASLR_SHIFT and CONFIG_MODULE_FG_KASLR_SHIFT.
16 fps means shift = 4, 8 fps on shift = 2, 1 fps for shift = 0.
feat make -j65 boot vmlinux.o vmlinux bzImage bogoops/s
Relocatable 4m38.478s 24.440s 72014208 58579520 9396192 57640.39
KASLR 4m39.344s 24.204s 72020624 87805776 9740352 57393.80
FG-K 16 fps 6m16.493s 25.429s 83759856 87194160 10885632 57784.76
FG-K 8 fps 6m20.190s 25.094s 83759856 88741328 10985248 56625.84
FG-K 1 fps 7m09.611s 25.922s 83759856 95681128 11352192 56953.99
From v7 (unreleased):
- rebase on top of 5.16-rc3, notably:
- drop 4 patches already taken in mainline;
- adopt to the new exception handlers logics;
- changed two new x86 ASM crypto module to generate function
sections. Also:
- improve generate_text_sections.pl script to address changes in
Clang 13 emitting __cfi_check_fail() only on final linking;
- retest on the latest stable Clang/LLVM stack (13);
- add missing .lds rule to the top .gitignore.
From v6 [1]:
- rebase on top of 5.15-rc1 and
db2b0c5d7b6f ("objtool: Support pv_opsindirect calls for noinstr")
from tip's objtool/core as there is plenty of counter-intuitive
conflicts between these two;
- change livepatch bit (#12) logics from forced overrride to exit
with errno and a error message to make it more clear to the users
(Miroslav);
- expand the cover letter a bit, add some build-time and runtime
numbers (Kees, Kristen).
The series is also available here: [2]
[0] https://lore.kernel.org/kernel-hardening/20200923173905.11219-1-kristen@linux.intel.com
[1] https://lore.kernel.org/kernel-hardening/20210831144114.154-1-alexandr.lobakin@intel.com
[2] https://github.com/alobakin/linux/pull/3
The original v5 cover letter:
Function Granular Kernel Address Space Layout Randomization (fgkaslr)
---------------------------------------------------------------------
This patch set is an implementation of finer grained kernel address space
randomization. It rearranges your kernel code at load time
on a per-function level granularity, with only around a second added to
boot time.
Changes in v5:
--------------
* fixed a bug in the code which increases boot heap size for
CONFIG_FG_KASLR which prevented the boot heap from being increased
for CONFIG_FG_KASLR when using bzip2 compression. Thanks to Andy Lavr
for finding the problem and identifying the solution.
* changed the adjustment of the orc_unwind_ip table at boot time to
disregard relocs associated with this table, and instead inspect the
entries separately. Relocs are not able to be used since they are
no longer correct once the table is resorted at buildtime.
* changed how orc_unwind_ip addresses in randomized sections are identified
to include the byte immediately after the end of the section.
* updated module code to use kvmalloc/kvfree based on suggestions from
Evgenii Shatokhin <eshatokhin@virtuozzo.com>.
* changed kernel commandline to disable fgkaslr to simply "nofgkaslr" to
match the nokaslr option. fgkaslr="X" can be added at a later date
if it is needed.
* Added a patch to force livepatch to require symbols to be unique if
using while fgkaslr either for core or modules.
Changes in v4:
-------------
* dropped the patch to split out change to STATIC definition in
x86/boot/compressed/misc.c and replaced with a patch authored
by Kees Cook to avoid the duplicate malloc definitions
* Added a section to Documentation/admin-guide/kernel-parameters.txt
to document the fgkaslr boot option.
* redesigned the patch to hide the new layout when reading
/proc/kallsyms. The previous implementation utilized a dynamically
allocated linked list to display the kernel and module symbols
in alphabetical order. The new implementation uses a randomly
shuffled index array to display the kernel and module symbols
in a random order.
Changes in v3:
-------------
* Makefile changes to accommodate CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
* removal of extraneous ALIGN_PAGE from _etext changes
* changed variable names in x86/tools/relocs to be less confusing
* split out change to STATIC definition in x86/boot/compressed/misc.c
* Updates to Documentation to make it more clear what is preserved in .text
* much more detailed commit message for function granular KASLR patch
* minor tweaks and changes that make for more readable code
* this cover letter updated slightly to add additional details
Changes in v2:
--------------
* Fix to address i386 build failure
* Allow module reordering patch to be configured separately so that
arm (or other non-x86_64 arches) can take advantage of module function
reordering. This support has not be tested by me, but smoke tested by
Ard Biesheuvel <ardb@kernel.org> on arm.
* Fix build issue when building on arm as reported by
Ard Biesheuvel <ardb@kernel.org>
Patches to objtool are included because they are dependencies for this
patchset, however they have been submitted by their maintainer separately.
Background
----------
KASLR was merged into the kernel with the objective of increasing the
difficulty of code reuse attacks. Code reuse attacks reused existing code
snippets to get around existing memory protections. They exploit software bugs
which expose addresses of useful code snippets to control the flow of
execution for their own nefarious purposes. KASLR moves the entire kernel
code text as a unit at boot time in order to make addresses less predictable.
The order of the code within the segment is unchanged - only the base address
is shifted. There are a few shortcomings to this algorithm.
1. Low Entropy - there are only so many locations the kernel can fit in. This
means an attacker could guess without too much trouble.
2. Knowledge of a single address can reveal the offset of the base address,
exposing all other locations for a published/known kernel image.
3. Info leaks abound.
Finer grained ASLR has been proposed as a way to make ASLR more resistant
to info leaks. It is not a new concept at all, and there are many variations
possible. Function reordering is an implementation of finer grained ASLR
which randomizes the layout of an address space on a function level
granularity. We use the term "fgkaslr" in this document to refer to the
technique of function reordering when used with KASLR, as well as finer grained
KASLR in general.
Proposed Improvement
--------------------
This patch set proposes adding function reordering on top of the existing
KASLR base address randomization. The over-arching objective is incremental
improvement over what we already have. It is designed to work in combination
with the existing solution. The implementation is really pretty simple, and
there are 2 main area where changes occur:
* Build time
GCC has had an option to place functions into individual .text sections for
many years now. This option can be used to implement function reordering at
load time. The final compiled vmlinux retains all the section headers, which
can be used to help find the address ranges of each function. Using this
information and an expanded table of relocation addresses, individual text
sections can be suffled immediately after decompression. Some data tables
inside the kernel that have assumptions about order require re-sorting
after being updated when applying relocations. In order to modify these tables,
a few key symbols are excluded from the objcopy symbol stripping process for
use after shuffling the text segments.
Some highlights from the build time changes to look for:
The top level kernel Makefile was modified to add the gcc flag if it
is supported. Currently, I am applying this flag to everything it is
possible to randomize. Anything that is written in C and not present in a
special input section is randomized. The final binary segment 0 retains a
consolidated .text section, as well as all the individual .text.* sections.
Future work could turn off this flags for selected files or even entire
subsystems, although obviously at the cost of security.
The relocs tool is updated to add relative relocations. This information
previously wasn't included because it wasn't necessary when moving the
entire .text segment as a unit.
A new file was created to contain a list of symbols that objcopy should
keep. We use those symbols at load time as described below.
* Load time
The boot kernel was modified to parse the vmlinux elf file after
decompression to check for our interesting symbols that we kept, and to
look for any .text.* sections to randomize. The consolidated .text section
is skipped and not moved. The sections are shuffled randomly, and copied
into memory following the .text section in a new random order. The existing
code which updated relocation addresses was modified to account for
not just a fixed delta from the load address, but the offset that the function
section was moved to. This requires inspection of each address to see if
it was impacted by a randomization. We use a bsearch to make this less
horrible on performance. Any tables that need to be modified with new
addresses or resorted are updated using the symbol addresses parsed from the
elf symbol table.
In order to hide our new layout, symbols reported through /proc/kallsyms
will be displayed in a random order.
Security Considerations
-----------------------
The objective of this patch set is to improve a technology that is already
merged into the kernel (KASLR). This code will not prevent all attacks,
but should instead be considered as one of several tools that can be used.
In particular, this code is meant to make KASLR more effective in the presence
of info leaks.
How much entropy we are adding to the existing entropy of standard KASLR will
depend on a few variables. Firstly and most obviously, the number of functions
that are randomized matters. This implementation keeps the existing .text
section for code that cannot be randomized - for example, because it was
assembly code. The less sections to randomize, the less entropy. In addition,
due to alignment (16 bytes for x86_64), the number of bits in a address that
the attacker needs to guess is reduced, as the lower bits are identical.
Performance Impact
------------------
There are two areas where function reordering can impact performance: boot
time latency, and run time performance.
* Boot time latency
This implementation of finer grained KASLR impacts the boot time of the kernel
in several places. It requires additional parsing of the kernel ELF file to
obtain the section headers of the sections to be randomized. It calls the
random number generator for each section to be randomized to determine that
section's new memory location. It copies the decompressed kernel into a new
area of memory to avoid corruption when laying out the newly randomized
sections. It increases the number of relocations the kernel has to perform at
boot time vs. standard KASLR, and it also requires a lookup on each address
that needs to be relocated to see if it was in a randomized section and needs
to be adjusted by a new offset. Finally, it re-sorts a few data tables that
are required to be sorted by address.
Booting a test VM on a modern, well appointed system showed an increase in
latency of approximately 1 second.
* Run time
The performance impact at run-time of function reordering varies by workload.
Using kcbench, a kernel compilation benchmark, the performance of a kernel
build with finer grained KASLR was about 1% slower than a kernel with standard
KASLR. Analysis with perf showed a slightly higher percentage of
L1-icache-load-misses. Other workloads were examined as well, with varied
results. Some workloads performed significantly worse under FGKASLR, while
others stayed the same or were mysteriously better. In general, it will
depend on the code flow whether or not finer grained KASLR will impact
your workload, and how the underlying code was designed. Because the layout
changes per boot, each time a system is rebooted the performance of a workload
may change.
Future work could identify hot areas that may not be randomized and either
leave them in the .text section or group them together into a single section
that may be randomized. If grouping things together helps, one other thing to
consider is that if we could identify text blobs that should be grouped together
to benefit a particular code flow, it could be interesting to explore
whether this security feature could be also be used as a performance
feature if you are interested in optimizing your kernel layout for a
particular workload at boot time. Optimizing function layout for a particular
workload has been researched and proven effective - for more information
read the Facebook paper "Optimizing Function Placement for Large-Scale
Data-Center Applications" (see references section below).
Image Size
----------
Adding additional section headers as a result of compiling with
-ffunction-sections will increase the size of the vmlinux ELF file.
With a standard distro config, the resulting vmlinux was increased by
about 3%. The compressed image is also increased due to the header files,
as well as the extra relocations that must be added. You can expect fgkaslr
to increase the size of the compressed image by about 15%.
Memory Usage
------------
fgkaslr increases the amount of heap that is required at boot time,
although this extra memory is released when the kernel has finished
decompression. As a result, it may not be appropriate to use this feature on
systems without much memory.
Building
--------
To enable fine grained KASLR, you need to have the following config options
set (including all the ones you would use to build normal KASLR)
CONFIG_FG_KASLR=y
In addition, fgkaslr is only supported for the X86_64 architecture.
Modules
-------
Modules are randomized similarly to the rest of the kernel by shuffling
the sections at load time prior to moving them into memory. The module must
also have been build with the -ffunction-sections compiler option.
Although fgkaslr for the kernel is only supported for the X86_64 architecture,
it is possible to use fgkaslr with modules on other architectures. To enable
this feature, select
CONFIG_MODULE_FG_KASLR=y
This option is selected automatically for X86_64 when CONFIG_FG_KASLR is set.
Disabling
---------
Disabling normal KASLR using the nokaslr command line option also disables
fgkaslr. It is also possible to disable fgkaslr separately by booting with
nofgkaslr on the commandline.
References
----------
There are a lot of academic papers which explore finer grained ASLR.
This paper in particular contributed the most to my implementation design
as well as my overall understanding of the problem space:
Selfrando: Securing the Tor Browser against De-anonymization Exploits,
M. Conti, S. Crane, T. Frassetto, et al.
For more information on how function layout impacts performance, see:
Optimizing Function Placement for Large-Scale Data-Center Applications,
G. Ottoni, B. Maher
Alexander Lobakin (8):
linkage: add macros for putting ASM functions into own sections
x86: conditionally place regular ASM functions into separate sections
FG-KASLR: use a scripted approach to handle .text.* sections
livepatch: only match unique symbols when using FG-KASLR
x86/boot: allow FG-KASLR to be selected
arm64/crypto: conditionally place ASM functions into separate sections
module: use a scripted approach for FG-KASLR
maintainers: add MAINTAINERS entry for FG-KASLR
Kristen Carlson Accardi (6):
x86: Makefile: Add build and config option for CONFIG_FG_KASLR
x86/tools: Add relative relocs for randomized functions
x86: Add support for function granular KASLR
kallsyms: Hide layout
module: Reorder functions
Documentation: add documentation for FG-KASLR
.gitignore | 1 +
.../admin-guide/kernel-parameters.txt | 6 +
Documentation/security/fgkaslr.rst | 172 ++++
Documentation/security/index.rst | 1 +
MAINTAINERS | 12 +
Makefile | 17 +-
arch/Kconfig | 3 +
arch/arm64/crypto/aes-ce-ccm-core.S | 16 +-
arch/arm64/crypto/aes-ce-core.S | 16 +-
arch/arm64/crypto/aes-ce.S | 4 +-
arch/arm64/crypto/aes-cipher-core.S | 8 +-
arch/arm64/crypto/aes-modes.S | 16 +-
arch/arm64/crypto/aes-neon.S | 4 +-
arch/arm64/crypto/aes-neonbs-core.S | 38 +-
arch/arm64/crypto/chacha-neon-core.S | 18 +-
arch/arm64/crypto/crct10dif-ce-core.S | 14 +-
arch/arm64/crypto/ghash-ce-core.S | 24 +-
arch/arm64/crypto/nh-neon-core.S | 4 +-
arch/arm64/crypto/poly1305-armv8.pl | 17 +
arch/arm64/crypto/sha1-ce-core.S | 4 +-
arch/arm64/crypto/sha2-ce-core.S | 4 +-
arch/arm64/crypto/sha3-ce-core.S | 4 +-
arch/arm64/crypto/sha512-armv8.pl | 11 +
arch/arm64/crypto/sha512-ce-core.S | 4 +-
arch/arm64/crypto/sm3-ce-core.S | 4 +-
arch/arm64/crypto/sm4-ce-core.S | 4 +-
arch/x86/Kconfig | 1 +
arch/x86/boot/compressed/Makefile | 9 +-
arch/x86/boot/compressed/fgkaslr.c | 893 ++++++++++++++++++
arch/x86/boot/compressed/misc.c | 154 ++-
arch/x86/boot/compressed/misc.h | 28 +
arch/x86/boot/compressed/utils.c | 13 +
arch/x86/boot/compressed/vmlinux.symbols | 19 +
arch/x86/crypto/aegis128-aesni-asm.S | 36 +-
arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 12 +-
arch/x86/crypto/aesni-intel_asm.S | 116 ++-
arch/x86/crypto/aesni-intel_avx-x86_64.S | 32 +-
arch/x86/crypto/blake2s-core.S | 8 +-
arch/x86/crypto/blowfish-x86_64-asm_64.S | 16 +-
arch/x86/crypto/camellia-aesni-avx-asm_64.S | 28 +-
arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 28 +-
arch/x86/crypto/camellia-x86_64-asm_64.S | 16 +-
arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 24 +-
arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 20 +-
arch/x86/crypto/chacha-avx2-x86_64.S | 12 +-
arch/x86/crypto/chacha-avx512vl-x86_64.S | 12 +-
arch/x86/crypto/chacha-ssse3-x86_64.S | 16 +-
arch/x86/crypto/crc32-pclmul_asm.S | 4 +-
arch/x86/crypto/crc32c-pcl-intel-asm_64.S | 4 +-
arch/x86/crypto/crct10dif-pcl-asm_64.S | 4 +-
arch/x86/crypto/des3_ede-asm_64.S | 8 +-
arch/x86/crypto/ghash-clmulni-intel_asm.S | 12 +-
arch/x86/crypto/nh-avx2-x86_64.S | 4 +-
arch/x86/crypto/nh-sse2-x86_64.S | 4 +-
arch/x86/crypto/poly1305-x86_64-cryptogams.pl | 8 +-
arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 20 +-
arch/x86/crypto/serpent-avx2-asm_64.S | 20 +-
arch/x86/crypto/serpent-sse2-i586-asm_32.S | 8 +-
arch/x86/crypto/serpent-sse2-x86_64-asm_64.S | 8 +-
arch/x86/crypto/sha1_avx2_x86_64_asm.S | 4 +-
arch/x86/crypto/sha1_ni_asm.S | 4 +-
arch/x86/crypto/sha1_ssse3_asm.S | 4 +-
arch/x86/crypto/sha256-avx-asm.S | 4 +-
arch/x86/crypto/sha256-avx2-asm.S | 4 +-
arch/x86/crypto/sha256-ssse3-asm.S | 4 +-
arch/x86/crypto/sha256_ni_asm.S | 4 +-
arch/x86/crypto/sha512-avx-asm.S | 4 +-
arch/x86/crypto/sha512-avx2-asm.S | 4 +-
arch/x86/crypto/sha512-ssse3-asm.S | 4 +-
arch/x86/crypto/sm4-aesni-avx-asm_64.S | 24 +-
arch/x86/crypto/sm4-aesni-avx2-asm_64.S | 16 +-
arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 20 +-
arch/x86/crypto/twofish-i586-asm_32.S | 8 +-
arch/x86/crypto/twofish-x86_64-asm_64-3way.S | 8 +-
arch/x86/crypto/twofish-x86_64-asm_64.S | 8 +-
arch/x86/entry/entry_32.S | 24 +-
arch/x86/entry/entry_64.S | 18 +-
arch/x86/entry/thunk_32.S | 4 +-
arch/x86/entry/thunk_64.S | 8 +-
arch/x86/include/asm/boot.h | 13 +-
arch/x86/include/asm/paravirt.h | 5 +-
arch/x86/include/asm/qspinlock_paravirt.h | 2 +-
arch/x86/kernel/acpi/wakeup_32.S | 9 +-
arch/x86/kernel/acpi/wakeup_64.S | 10 +-
arch/x86/kernel/ftrace_32.S | 19 +-
arch/x86/kernel/ftrace_64.S | 24 +-
arch/x86/kernel/kprobes/core.c | 3 +-
arch/x86/kernel/kvm.c | 2 +-
arch/x86/kernel/relocate_kernel_32.S | 2 +
arch/x86/kernel/relocate_kernel_64.S | 2 +
arch/x86/kernel/vmlinux.lds.S | 6 +-
arch/x86/kvm/emulate.c | 2 +-
arch/x86/kvm/vmx/vmenter.S | 8 +-
arch/x86/lib/clear_page_64.S | 12 +-
arch/x86/lib/cmpxchg16b_emu.S | 4 +-
arch/x86/lib/copy_mc_64.S | 8 +-
arch/x86/lib/copy_page_64.S | 7 +-
arch/x86/lib/copy_user_64.S | 18 +-
arch/x86/lib/csum-copy_64.S | 4 +-
arch/x86/lib/error-inject.c | 3 +-
arch/x86/lib/getuser.S | 37 +-
arch/x86/lib/hweight.S | 9 +-
arch/x86/lib/iomap_copy_64.S | 4 +-
arch/x86/lib/memmove_64.S | 4 +-
arch/x86/lib/memset_64.S | 12 +-
arch/x86/lib/msr-reg.S | 8 +-
arch/x86/lib/putuser.S | 18 +-
arch/x86/mm/mem_encrypt_boot.S | 8 +-
arch/x86/platform/efi/efi_stub_64.S | 4 +-
arch/x86/platform/efi/efi_thunk_64.S | 4 +-
arch/x86/power/hibernate_asm_32.S | 14 +-
arch/x86/power/hibernate_asm_64.S | 14 +-
arch/x86/tools/relocs.c | 32 +-
arch/x86/tools/relocs.h | 4 +-
arch/x86/tools/relocs_common.c | 14 +-
arch/x86/xen/xen-asm.S | 24 +-
arch/x86/xen/xen-head.S | 6 +-
include/asm-generic/vmlinux.lds.h | 38 +-
include/linux/linkage.h | 82 ++
include/uapi/linux/elf.h | 1 +
init/Kconfig | 51 +
kernel/kallsyms.c | 158 +++-
kernel/livepatch/core.c | 13 +
kernel/module.c | 91 +-
scripts/Makefile.modfinal | 19 +-
scripts/generate_text_sections.pl | 163 ++++
scripts/link-vmlinux.sh | 25 +-
scripts/module.lds.S | 14 +-
128 files changed, 2609 insertions(+), 626 deletions(-)
create mode 100644 Documentation/security/fgkaslr.rst
create mode 100644 arch/x86/boot/compressed/fgkaslr.c
create mode 100644 arch/x86/boot/compressed/utils.c
create mode 100644 arch/x86/boot/compressed/vmlinux.symbols
create mode 100755 scripts/generate_text_sections.pl
--
2.33.1
^ permalink raw reply [flat|nested] 40+ messages in thread
* [PATCH v8 01/14] x86: Makefile: Add build and config option for CONFIG_FG_KASLR
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 02/14] x86/tools: Add relative relocs for randomized functions Alexander Lobakin
` (13 subsequent siblings)
14 siblings, 0 replies; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
From: Kristen Carlson Accardi <kristen@linux.intel.com>
Allow user to select CONFIG_FG_KASLR if dependencies are met. Change
the make file to build with -ffunction-sections if CONFIG_FG_KASLR.
While the only architecture that supports CONFIG_FG_KASLR does not
currently enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION, make sure these
2 features play nicely together for the future by ensuring that if
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is selected when used with
CONFIG_FG_KASLR the function sections will not be consolidated back
into .text. Thanks to Kees Cook for the dead code elimination changes.
Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Tested-by: Tony Luck <tony.luck@intel.com>
[ alobakin:
- improve cflags management in the top Makefile
- move ARCH_HAS_FG_KASLR to the top arch/Kconfig
- add symtab_shndx to the list of known sections ]
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
Makefile | 13 ++++++++++++-
arch/Kconfig | 3 +++
include/asm-generic/vmlinux.lds.h | 20 ++++++++++++++++++--
init/Kconfig | 12 ++++++++++++
4 files changed, 45 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile
index 0a6ecc8bb2d2..a4d2eac5f81f 100644
--- a/Makefile
+++ b/Makefile
@@ -882,8 +882,19 @@ ifdef CONFIG_DEBUG_SECTION_MISMATCH
KBUILD_CFLAGS += -fno-inline-functions-called-once
endif
+# ClangLTO implies -ffunction-sections -fdata-sections, no need
+# to specify them manually and trigger a pointless full rebuild
+ifndef CONFIG_LTO_CLANG
+ifneq ($(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION)$(CONFIG_FG_KASLR),)
+KBUILD_CFLAGS_KERNEL += -ffunction-sections
+endif
+
+ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
+KBUILD_CFLAGS_KERNEL += -fdata-sections
+endif
+endif # CONFIG_LTO_CLANG
+
ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
-KBUILD_CFLAGS_KERNEL += -ffunction-sections -fdata-sections
LDFLAGS_vmlinux += --gc-sections
endif
diff --git a/arch/Kconfig b/arch/Kconfig
index d3c4ab249e9c..602b67162e53 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -1312,6 +1312,9 @@ config ARCH_HAS_PARANOID_L1D_FLUSH
config DYNAMIC_SIGFRAME
bool
+config ARCH_HAS_FG_KASLR
+ bool
+
source "kernel/gcov/Kconfig"
source "scripts/gcc-plugins/Kconfig"
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 42f3866bca69..96fbedcbf7c8 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -97,14 +97,12 @@
* sections to be brought in with rodata.
*/
#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CLANG)
-#define TEXT_MAIN .text .text.[0-9a-zA-Z_]*
#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..L* .data..compoundliteral* .data.$__unnamed_* .data.$L*
#define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]*
#define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* .rodata..L*
#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..compoundliteral*
#define SBSS_MAIN .sbss .sbss.[0-9a-zA-Z_]*
#else
-#define TEXT_MAIN .text
#define DATA_MAIN .data
#define SDATA_MAIN .sdata
#define RODATA_MAIN .rodata
@@ -112,6 +110,23 @@
#define SBSS_MAIN .sbss
#endif
+/*
+ * LTO_CLANG, LD_DEAD_CODE_DATA_ELIMINATION and FG_KASLR options enable
+ * -ffunction-sections, which produces separately named .text sections. In
+ * the case of CONFIG_FG_KASLR, they need to stay distict so they can be
+ * separately randomized. Without CONFIG_FG_KASLR, the separate .text
+ * sections can be collected back into a common section, which makes the
+ * resulting image slightly smaller
+ */
+#if (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+ defined(CONFIG_LTO_CLANG)) && !defined(CONFIG_FG_KASLR)
+#define TEXT_MAIN .text .text.[0-9a-zA-Z_]*
+#elif defined(CONFIG_FG_KASLR)
+#define TEXT_MAIN .text.__unused__
+#else
+#define TEXT_MAIN .text
+#endif
+
/*
* GCC 4.5 and later have a 32 bytes section alignment for structures.
* Except GCC 4.9, that feels the need to align on 64 bytes.
@@ -840,6 +855,7 @@
#define ELF_DETAILS \
.comment 0 : { *(.comment) } \
.symtab 0 : { *(.symtab) } \
+ .symtab_shndx 0 : { *(.symtab_shndx) } \
.strtab 0 : { *(.strtab) } \
.shstrtab 0 : { *(.shstrtab) }
diff --git a/init/Kconfig b/init/Kconfig
index 4b7bac10c72d..5cb8f8230915 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2051,6 +2051,18 @@ config PROFILING
config TRACEPOINTS
bool
+config FG_KASLR
+ bool "Function Granular Kernel Address Space Layout Randomization"
+ depends on ARCH_HAS_FG_KASLR
+ help
+ This option improves the randomness of the kernel text
+ over basic Kernel Address Space Layout Randomization (KASLR)
+ by reordering the kernel text at boot time. This feature
+ uses information generated at compile time to re-layout the
+ kernel text section at boot time at function level granularity.
+
+ If unsure, say N.
+
endmenu # General setup
source "arch/Kconfig"
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 02/14] x86/tools: Add relative relocs for randomized functions
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 01/14] x86: Makefile: Add build and config option for CONFIG_FG_KASLR Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 03/14] x86: Add support for function granular KASLR Alexander Lobakin
` (12 subsequent siblings)
14 siblings, 0 replies; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
From: Kristen Carlson Accardi <kristen@linux.intel.com>
When reordering functions, the relative offsets for relocs that
are either in the randomized sections, or refer to the randomized
sections will need to be adjusted. Add code to detect whether a
reloc satisfies these cases, and if so, add them to the appropriate
reloc list.
Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Tested-by: Tony Luck <tony.luck@intel.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
[ alobakin: don't split relocs' usage string across lines ]
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
arch/x86/boot/compressed/Makefile | 7 ++++++-
arch/x86/tools/relocs.c | 32 +++++++++++++++++++++++++++----
arch/x86/tools/relocs.h | 4 ++--
arch/x86/tools/relocs_common.c | 14 +++++++++-----
4 files changed, 45 insertions(+), 12 deletions(-)
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 431bf7f846c3..c31a24161fbf 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -106,6 +106,11 @@ $(obj)/vmlinux: $(vmlinux-objs-y) $(efi-obj-y) FORCE
$(call if_changed,ld)
OBJCOPYFLAGS_vmlinux.bin := -R .comment -S
+
+ifdef CONFIG_FG_KASLR
+RELOCS_ARGS += --fg-kaslr
+endif
+
$(obj)/vmlinux.bin: vmlinux FORCE
$(call if_changed,objcopy)
@@ -113,7 +118,7 @@ targets += $(patsubst $(obj)/%,%,$(vmlinux-objs-y)) vmlinux.bin.all vmlinux.relo
CMD_RELOCS = arch/x86/tools/relocs
quiet_cmd_relocs = RELOCS $@
- cmd_relocs = $(CMD_RELOCS) $< > $@;$(CMD_RELOCS) --abs-relocs $<
+ cmd_relocs = $(CMD_RELOCS) $(RELOCS_ARGS) $< > $@;$(CMD_RELOCS) $(RELOCS_ARGS) --abs-relocs $<
$(obj)/vmlinux.relocs: vmlinux FORCE
$(call if_changed,relocs)
diff --git a/arch/x86/tools/relocs.c b/arch/x86/tools/relocs.c
index c736cf2ac76b..8aa1f39be561 100644
--- a/arch/x86/tools/relocs.c
+++ b/arch/x86/tools/relocs.c
@@ -45,6 +45,8 @@ struct section {
};
static struct section *secs;
+static int fgkaslr_mode;
+
static const char * const sym_regex_kernel[S_NSYMTYPES] = {
/*
* Following symbols have been audited. There values are constant and do
@@ -823,6 +825,24 @@ static int is_percpu_sym(ElfW(Sym) *sym, const char *symname)
strncmp(symname, "init_per_cpu_", 13);
}
+static int is_function_section(struct section *sec)
+{
+ if (!fgkaslr_mode)
+ return 0;
+
+ return !strncmp(sec_name(sec->shdr.sh_info), ".text.", 6);
+}
+
+static int is_randomized_sym(ElfW(Sym) *sym)
+{
+ if (!fgkaslr_mode)
+ return 0;
+
+ if (sym->st_shndx > shnum)
+ return 0;
+
+ return !strncmp(sec_name(sym_index(sym)), ".text.", 6);
+}
static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,
const char *symname)
@@ -848,12 +868,15 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,
case R_X86_64_PC32:
case R_X86_64_PLT32:
/*
- * PC relative relocations don't need to be adjusted unless
- * referencing a percpu symbol.
+ * we need to keep pc relative relocations for sections which
+ * might be randomized, and for the percpu section.
+ * We also need to keep relocations for any offset which might
+ * reference an address in a section which has been randomized.
*
* NB: R_X86_64_PLT32 can be treated as R_X86_64_PC32.
*/
- if (is_percpu_sym(sym, symname))
+ if (is_function_section(sec) || is_randomized_sym(sym) ||
+ is_percpu_sym(sym, symname))
add_reloc(&relocs32neg, offset);
break;
@@ -1168,8 +1191,9 @@ static void print_reloc_info(void)
void process(FILE *fp, int use_real_mode, int as_text,
int show_absolute_syms, int show_absolute_relocs,
- int show_reloc_info)
+ int show_reloc_info, int fgkaslr)
{
+ fgkaslr_mode = fgkaslr;
regex_init(use_real_mode);
read_ehdr(fp);
read_shdrs(fp);
diff --git a/arch/x86/tools/relocs.h b/arch/x86/tools/relocs.h
index 4c49c82446eb..269db511b243 100644
--- a/arch/x86/tools/relocs.h
+++ b/arch/x86/tools/relocs.h
@@ -32,8 +32,8 @@ enum symtype {
void process_32(FILE *fp, int use_real_mode, int as_text,
int show_absolute_syms, int show_absolute_relocs,
- int show_reloc_info);
+ int show_reloc_info, int fgkaslr);
void process_64(FILE *fp, int use_real_mode, int as_text,
int show_absolute_syms, int show_absolute_relocs,
- int show_reloc_info);
+ int show_reloc_info, int fgkaslr);
#endif /* RELOCS_H */
diff --git a/arch/x86/tools/relocs_common.c b/arch/x86/tools/relocs_common.c
index 6634352a20bc..d6acda36575a 100644
--- a/arch/x86/tools/relocs_common.c
+++ b/arch/x86/tools/relocs_common.c
@@ -12,14 +12,13 @@ void die(char *fmt, ...)
static void usage(void)
{
- die("relocs [--abs-syms|--abs-relocs|--reloc-info|--text|--realmode]" \
- " vmlinux\n");
+ die("relocs [--abs-syms|--abs-relocs|--reloc-info|--text|--realmode|--fg-kaslr] vmlinux\n");
}
int main(int argc, char **argv)
{
int show_absolute_syms, show_absolute_relocs, show_reloc_info;
- int as_text, use_real_mode;
+ int as_text, use_real_mode, fgkaslr_opt;
const char *fname;
FILE *fp;
int i;
@@ -30,6 +29,7 @@ int main(int argc, char **argv)
show_reloc_info = 0;
as_text = 0;
use_real_mode = 0;
+ fgkaslr_opt = 0;
fname = NULL;
for (i = 1; i < argc; i++) {
char *arg = argv[i];
@@ -54,6 +54,10 @@ int main(int argc, char **argv)
use_real_mode = 1;
continue;
}
+ if (strcmp(arg, "--fg-kaslr") == 0) {
+ fgkaslr_opt = 1;
+ continue;
+ }
}
else if (!fname) {
fname = arg;
@@ -75,11 +79,11 @@ int main(int argc, char **argv)
if (e_ident[EI_CLASS] == ELFCLASS64)
process_64(fp, use_real_mode, as_text,
show_absolute_syms, show_absolute_relocs,
- show_reloc_info);
+ show_reloc_info, fgkaslr_opt);
else
process_32(fp, use_real_mode, as_text,
show_absolute_syms, show_absolute_relocs,
- show_reloc_info);
+ show_reloc_info, fgkaslr_opt);
fclose(fp);
return 0;
}
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 03/14] x86: Add support for function granular KASLR
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 01/14] x86: Makefile: Add build and config option for CONFIG_FG_KASLR Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 02/14] x86/tools: Add relative relocs for randomized functions Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-03 9:18 ` Peter Zijlstra
2021-12-03 10:34 ` Peter Zijlstra
2021-12-02 22:32 ` [PATCH v8 04/14] linkage: add macros for putting ASM functions into own sections Alexander Lobakin
` (11 subsequent siblings)
14 siblings, 2 replies; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
From: Kristen Carlson Accardi <kristen@linux.intel.com>
This commit contains the changes required to re-layout the kernel text
sections generated by -ffunction-sections shortly after decompression.
Documentation of the feature is also added.
After decompression, the decompressed image's elf headers are parsed.
In order to manually update certain data structures that are built with
relative offsets during the kernel build process, certain symbols are
not stripped by objdump and their location is retained in the elf symbol
tables. These addresses are saved.
If the image was built with -ffunction-sections, there will be ELF section
headers present which contain information about the address range of each
section. Anything that is not broken out into function sections (i.e. is
consolidated into .text) is left in it's original location, but any other
executable section which begins with ".text." is located and shuffled
randomly within the remaining text segment address range.
After the sections have been copied to their new locations, but before
relocations have been applied, the kallsyms tables must be updated to
reflect the new symbol locations. Because it is expected that these tables
will be sorted by address, the kallsyms tables will need to be sorted
after the update.
When applying relocations, the address of the relocation needs to be
adjusted by the offset from the original location of the section that was
randomized to it's new location. In addition, if a value at that relocation
was a location in the text segment that was randomized, it's value will be
adjusted to a new location.
After relocations have been applied, the exception table must be updated
with new symbol locations, and then re-sorted by the new address. The
orc table will have been updated as part of applying relocations, but since
it is expected to be sorted by address, it will need to be resorted.
Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Tested-by: Tony Luck <tony.luck@intel.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Reported-by: kernel test robot <lkp@intel.com> # #if -> #ifdef
[ alobakin: fix .altinstr_replacement relocations ]
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
arch/x86/boot/compressed/Makefile | 2 +
arch/x86/boot/compressed/fgkaslr.c | 893 +++++++++++++++++++++++
arch/x86/boot/compressed/misc.c | 154 +++-
arch/x86/boot/compressed/misc.h | 28 +
arch/x86/boot/compressed/utils.c | 13 +
arch/x86/boot/compressed/vmlinux.symbols | 19 +
arch/x86/include/asm/boot.h | 13 +-
arch/x86/kernel/vmlinux.lds.S | 2 +
include/uapi/linux/elf.h | 1 +
9 files changed, 1098 insertions(+), 27 deletions(-)
create mode 100644 arch/x86/boot/compressed/fgkaslr.c
create mode 100644 arch/x86/boot/compressed/utils.c
create mode 100644 arch/x86/boot/compressed/vmlinux.symbols
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index c31a24161fbf..e12fb0c8f4fc 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -89,6 +89,7 @@ vmlinux-objs-y := $(obj)/vmlinux.lds $(obj)/kernel_info.o $(obj)/head_$(BITS).o
vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o
vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o
+vmlinux-objs-$(CONFIG_FG_KASLR) += $(obj)/fgkaslr.o $(obj)/utils.o
ifdef CONFIG_X86_64
vmlinux-objs-y += $(obj)/ident_map_64.o
vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
@@ -108,6 +109,7 @@ $(obj)/vmlinux: $(vmlinux-objs-y) $(efi-obj-y) FORCE
OBJCOPYFLAGS_vmlinux.bin := -R .comment -S
ifdef CONFIG_FG_KASLR
+OBJCOPYFLAGS += --keep-symbols=$(srctree)/$(src)/vmlinux.symbols
RELOCS_ARGS += --fg-kaslr
endif
diff --git a/arch/x86/boot/compressed/fgkaslr.c b/arch/x86/boot/compressed/fgkaslr.c
new file mode 100644
index 000000000000..265630266f46
--- /dev/null
+++ b/arch/x86/boot/compressed/fgkaslr.c
@@ -0,0 +1,893 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * This contains the routines needed to reorder the kernel text section
+ * at boot time.
+ *
+ * Copyright (C) 2020-2021, Intel Corporation.
+ * Author: Kristen Carlson Accardi <kristen@linux.intel.com>
+ */
+
+#include "misc.h"
+#include "error.h"
+#include "pgtable.h"
+#include "../string.h"
+#include "../voffset.h"
+#include <linux/sort.h>
+#include <linux/bsearch.h>
+#include "../../include/asm/extable.h"
+#include "../../include/asm/orc_types.h"
+
+/*
+ * Longest parameter of 'fgkaslr=' is 'off' right now, plus an extra '\0'
+ * for termination.
+ */
+#define MAX_FGKASLR_ARG_LENGTH 4
+static int nofgkaslr;
+
+/*
+ * Use normal definitions of mem*() from string.c. There are already
+ * included header files which expect a definition of memset() and by
+ * the time we define memset macro, it is too late.
+ */
+#undef memcpy
+#undef memset
+#define memzero(s, n) memset((s), 0, (n))
+#define memmove memmove
+
+void *memmove(void *dest, const void *src, size_t n);
+
+static unsigned long percpu_start;
+static unsigned long percpu_end;
+
+static long addr_kallsyms_names;
+static long addr_kallsyms_offsets;
+static long addr_kallsyms_num_syms;
+static long addr_kallsyms_relative_base;
+static long addr_kallsyms_markers;
+static long addr___start___ex_table;
+static long addr___stop___ex_table;
+static long addr___altinstr_replacement;
+static long addr___altinstr_replacement_end;
+static long addr__stext;
+static long addr__etext;
+static long addr__sinittext;
+static long addr__einittext;
+static long addr___start_orc_unwind_ip;
+static long addr___stop_orc_unwind_ip;
+static long addr___start_orc_unwind;
+
+/* addresses in mapped address space */
+static int *base;
+static u8 *names;
+static unsigned long relative_base;
+static unsigned int *markers_addr;
+
+struct kallsyms_name {
+ u8 len;
+ u8 indecis[256];
+};
+
+static struct kallsyms_name *names_table;
+
+static struct orc_entry *cur_orc_table;
+static int *cur_orc_ip_table;
+
+/* Array of pointers to sections headers for randomized sections */
+Elf_Shdr **sections;
+
+/* Number of elements in the randomized section header array (sections) */
+static int sections_size;
+
+/* Array of all section headers, randomized or otherwise */
+static Elf_Shdr *sechdrs;
+
+static bool is_orc_unwind(long addr)
+{
+ if (addr >= addr___start_orc_unwind_ip &&
+ addr < addr___stop_orc_unwind_ip)
+ return true;
+ return false;
+}
+
+static bool is_text(long addr)
+{
+ if ((addr >= addr__stext && addr < addr__etext) ||
+ (addr >= addr__sinittext && addr < addr__einittext) ||
+ (addr >= addr___altinstr_replacement &&
+ addr <= addr___altinstr_replacement_end))
+ return true;
+ return false;
+}
+
+bool is_percpu_addr(long pc, long offset)
+{
+ unsigned long ptr;
+ long address;
+
+ address = pc + offset + 4;
+
+ ptr = (unsigned long)address;
+
+ if (ptr >= percpu_start && ptr < percpu_end)
+ return true;
+
+ return false;
+}
+
+static int cmp_section_addr(const void *a, const void *b)
+{
+ unsigned long ptr = (unsigned long)a;
+ Elf_Shdr *s = *(Elf_Shdr **)b;
+ unsigned long end = s->sh_addr + s->sh_size;
+
+ if (ptr >= s->sh_addr && ptr < end)
+ return 0;
+
+ if (ptr < s->sh_addr)
+ return -1;
+
+ return 1;
+}
+
+static int cmp_section_addr_orc(const void *a, const void *b)
+{
+ unsigned long ptr = (unsigned long)a;
+ Elf_Shdr *s = *(Elf_Shdr **)b;
+ unsigned long end = s->sh_addr + s->sh_size;
+
+ /* orc relocations can be one past the end of the section */
+ if (ptr >= s->sh_addr && ptr <= end)
+ return 0;
+
+ if (ptr < s->sh_addr)
+ return -1;
+
+ return 1;
+}
+
+/*
+ * Discover if the orc_unwind address is in a randomized section and if so,
+ * adjust by the saved offset.
+ */
+Elf_Shdr *adjust_address_orc(long *address)
+{
+ Elf_Shdr **s;
+ Elf_Shdr *shdr;
+
+ if (nofgkaslr)
+ return NULL;
+
+ s = bsearch((const void *)*address, sections, sections_size, sizeof(*s),
+ cmp_section_addr_orc);
+ if (s) {
+ shdr = *s;
+ *address += shdr->sh_offset;
+ return shdr;
+ }
+
+ return NULL;
+}
+
+/*
+ * Discover if the address is in a randomized section and if so, adjust
+ * by the saved offset.
+ */
+Elf_Shdr *adjust_address(long *address)
+{
+ Elf_Shdr **s;
+ Elf_Shdr *shdr;
+
+ if (nofgkaslr)
+ return NULL;
+
+ s = bsearch((const void *)*address, sections, sections_size, sizeof(*s),
+ cmp_section_addr);
+ if (s) {
+ shdr = *s;
+ *address += shdr->sh_offset;
+ return shdr;
+ }
+
+ return NULL;
+}
+
+void adjust_relative_offset(long pc, long *value, Elf_Shdr *section)
+{
+ Elf_Shdr *s;
+ long address;
+
+ if (nofgkaslr)
+ return;
+
+ /*
+ * sometimes we are updating a relative offset that would
+ * normally be relative to the next instruction (such as a call).
+ * In this case to calculate the target, you need to add 32bits to
+ * the pc to get the next instruction value. However, sometimes
+ * targets are just data that was stored in a table such as ksymtab
+ * or cpu alternatives. In this case our target is not relative to
+ * the next instruction.
+ */
+
+ /* Calculate the address that this offset would call. */
+ if (!is_text(pc))
+ address = pc + *value;
+ else
+ address = pc + *value + 4;
+
+ /*
+ * orc ip addresses are sorted at build time after relocs have
+ * been applied, making the relocs no longer valid. Skip any
+ * relocs for the orc_unwind_ip table. These will be updated
+ * separately.
+ */
+ if (is_orc_unwind(pc))
+ return;
+
+ s = adjust_address(&address);
+
+ /*
+ * if the address is in section that was randomized,
+ * we need to adjust the offset.
+ */
+ if (s)
+ *value += s->sh_offset;
+
+ /*
+ * If the PC that this offset was calculated for was in a section
+ * that has been randomized, the value needs to be adjusted by the
+ * same amount as the randomized section was adjusted from it's original
+ * location.
+ */
+ if (section)
+ *value -= section->sh_offset;
+}
+
+static void kallsyms_swp(void *a, void *b, int size)
+{
+ int idx1, idx2;
+ int temp;
+ struct kallsyms_name name_a;
+
+ /* Determine our index into the array. */
+ idx1 = (int *)a - base;
+ idx2 = (int *)b - base;
+ temp = base[idx1];
+ base[idx1] = base[idx2];
+ base[idx2] = temp;
+
+ /* Swap the names table. */
+ memcpy(&name_a, &names_table[idx1], sizeof(name_a));
+ memcpy(&names_table[idx1], &names_table[idx2],
+ sizeof(struct kallsyms_name));
+ memcpy(&names_table[idx2], &name_a, sizeof(struct kallsyms_name));
+}
+
+static int kallsyms_cmp(const void *a, const void *b)
+{
+ int addr_a, addr_b;
+ unsigned long uaddr_a, uaddr_b;
+
+ addr_a = *(int *)a;
+ addr_b = *(int *)b;
+
+ if (addr_a >= 0)
+ uaddr_a = addr_a;
+ if (addr_b >= 0)
+ uaddr_b = addr_b;
+
+ if (addr_a < 0)
+ uaddr_a = relative_base - 1 - addr_a;
+ if (addr_b < 0)
+ uaddr_b = relative_base - 1 - addr_b;
+
+ if (uaddr_b > uaddr_a)
+ return -1;
+
+ return 0;
+}
+
+static void deal_with_names(int num_syms)
+{
+ int num_bytes;
+ int i, j;
+ int offset;
+
+ /* we should have num_syms kallsyms_name entries */
+ num_bytes = num_syms * sizeof(*names_table);
+ names_table = malloc(num_syms * sizeof(*names_table));
+ if (!names_table) {
+ debug_putstr("\nbytes requested: ");
+ debug_puthex(num_bytes);
+ error("\nunable to allocate space for names table\n");
+ }
+
+ /* read all the names entries */
+ offset = 0;
+ for (i = 0; i < num_syms; i++) {
+ names_table[i].len = names[offset];
+ offset++;
+ for (j = 0; j < names_table[i].len; j++) {
+ names_table[i].indecis[j] = names[offset];
+ offset++;
+ }
+ }
+}
+
+static void write_sorted_names(int num_syms)
+{
+ int i, j;
+ int offset = 0;
+ unsigned int *markers;
+
+ /*
+ * we are going to need to regenerate the markers table, which is a
+ * table of offsets into the compressed stream every 256 symbols.
+ * this code copied almost directly from scripts/kallsyms.c
+ */
+ markers = malloc(sizeof(unsigned int) * ((num_syms + 255) / 256));
+ if (!markers) {
+ debug_putstr("\nfailed to allocate heap space of ");
+ debug_puthex(((num_syms + 255) / 256));
+ debug_putstr(" bytes\n");
+ error("Unable to allocate space for markers table");
+ }
+
+ for (i = 0; i < num_syms; i++) {
+ if ((i & 0xFF) == 0)
+ markers[i >> 8] = offset;
+
+ names[offset] = (u8)names_table[i].len;
+ offset++;
+ for (j = 0; j < names_table[i].len; j++) {
+ names[offset] = (u8)names_table[i].indecis[j];
+ offset++;
+ }
+ }
+
+ /* write new markers table over old one */
+ for (i = 0; i < ((num_syms + 255) >> 8); i++)
+ markers_addr[i] = markers[i];
+
+ free(markers);
+ free(names_table);
+}
+
+static void sort_kallsyms(unsigned long map)
+{
+ int num_syms;
+ int i;
+
+ debug_putstr("\nRe-sorting kallsyms...\n");
+
+ num_syms = *(int *)(addr_kallsyms_num_syms + map);
+ base = (int *)(addr_kallsyms_offsets + map);
+ relative_base = *(unsigned long *)(addr_kallsyms_relative_base + map);
+ markers_addr = (unsigned int *)(addr_kallsyms_markers + map);
+ names = (u8 *)(addr_kallsyms_names + map);
+
+ /*
+ * the kallsyms table was generated prior to any randomization.
+ * it is a bunch of offsets from "relative base". In order for
+ * us to check if a symbol has an address that was in a randomized
+ * section, we need to reconstruct the address to it's original
+ * value prior to handle_relocations.
+ */
+ for (i = 0; i < num_syms; i++) {
+ unsigned long addr;
+ int new_base;
+
+ /*
+ * according to kernel/kallsyms.c, positive offsets are absolute
+ * values and negative offsets are relative to the base.
+ */
+ if (base[i] >= 0)
+ addr = base[i];
+ else
+ addr = relative_base - 1 - base[i];
+
+ if (adjust_address(&addr)) {
+ /* here we need to recalcuate the offset */
+ new_base = relative_base - 1 - addr;
+ base[i] = new_base;
+ }
+ }
+
+ /*
+ * here we need to read in all the kallsyms_names info
+ * so that we can regenerate it.
+ */
+ deal_with_names(num_syms);
+
+ sort(base, num_syms, sizeof(int), kallsyms_cmp, kallsyms_swp);
+
+ /* write the newly sorted names table over the old one */
+ write_sorted_names(num_syms);
+}
+
+/*
+ * We need to include this file here rather than in utils.c because
+ * some of the helper functions in extable.c are used to update
+ * the extable below and are defined as "static" in extable.c
+ */
+#include "../../../../lib/extable.c"
+
+static inline unsigned long
+ex_fixup_addr(const struct exception_table_entry *x)
+{
+ return ((unsigned long)&x->fixup + x->fixup);
+}
+
+static void update_ex_table(unsigned long map)
+{
+ struct exception_table_entry *start_ex_table =
+ (struct exception_table_entry *)(addr___start___ex_table + map);
+ struct exception_table_entry *stop_ex_table =
+ (struct exception_table_entry *)(addr___stop___ex_table + map);
+ int num_entries =
+ (addr___stop___ex_table - addr___start___ex_table) /
+ sizeof(struct exception_table_entry);
+ int i;
+
+ debug_putstr("\nUpdating exception table...");
+ for (i = 0; i < num_entries; i++) {
+ unsigned long insn = ex_to_insn(&start_ex_table[i]);
+ unsigned long fixup = ex_fixup_addr(&start_ex_table[i]);
+ unsigned long addr;
+ Elf_Shdr *s;
+
+ /* check each address to see if it needs adjusting */
+ addr = insn - map;
+ s = adjust_address(&addr);
+ if (s)
+ start_ex_table[i].insn += s->sh_offset;
+
+ addr = fixup - map;
+ s = adjust_address(&addr);
+ if (s)
+ start_ex_table[i].fixup += s->sh_offset;
+ }
+}
+
+static void sort_ex_table(unsigned long map)
+{
+ struct exception_table_entry *start_ex_table =
+ (struct exception_table_entry *)(addr___start___ex_table + map);
+ struct exception_table_entry *stop_ex_table =
+ (struct exception_table_entry *)(addr___stop___ex_table + map);
+
+ debug_putstr("\nRe-sorting exception table...");
+
+ sort_extable(start_ex_table, stop_ex_table);
+}
+
+static inline unsigned long orc_ip(const int *ip)
+{
+ return (unsigned long)ip + *ip;
+}
+
+static void orc_sort_swap(void *_a, void *_b, int size)
+{
+ struct orc_entry *orc_a, *orc_b;
+ struct orc_entry orc_tmp;
+ int *a = _a, *b = _b, tmp;
+ int delta = _b - _a;
+
+ /* Swap the .orc_unwind_ip entries: */
+ tmp = *a;
+ *a = *b + delta;
+ *b = tmp - delta;
+
+ /* Swap the corresponding .orc_unwind entries: */
+ orc_a = cur_orc_table + (a - cur_orc_ip_table);
+ orc_b = cur_orc_table + (b - cur_orc_ip_table);
+ orc_tmp = *orc_a;
+ *orc_a = *orc_b;
+ *orc_b = orc_tmp;
+}
+
+static int orc_sort_cmp(const void *_a, const void *_b)
+{
+ struct orc_entry *orc_a;
+ const int *a = _a, *b = _b;
+ unsigned long a_val = orc_ip(a);
+ unsigned long b_val = orc_ip(b);
+
+ if (a_val > b_val)
+ return 1;
+ if (a_val < b_val)
+ return -1;
+
+ /*
+ * The "weak" section terminator entries need to always be on the left
+ * to ensure the lookup code skips them in favor of real entries.
+ * These terminator entries exist to handle any gaps created by
+ * whitelisted .o files which didn't get objtool generation.
+ */
+ orc_a = cur_orc_table + (a - cur_orc_ip_table);
+ return orc_a->sp_reg == ORC_REG_UNDEFINED && !orc_a->end ? -1 : 1;
+}
+
+static void update_orc_table(unsigned long map)
+{
+ int i;
+ int num_entries =
+ (addr___stop_orc_unwind_ip - addr___start_orc_unwind_ip) / sizeof(int);
+
+ cur_orc_ip_table = (int *)(addr___start_orc_unwind_ip + map);
+ cur_orc_table = (struct orc_entry *)(addr___start_orc_unwind + map);
+
+ debug_putstr("\nUpdating orc tables...\n");
+ for (i = 0; i < num_entries; i++) {
+ unsigned long ip = orc_ip(&cur_orc_ip_table[i]);
+ Elf_Shdr *s;
+
+ /* check each address to see if it needs adjusting */
+ ip = ip - map;
+
+ /*
+ * objtool places terminator entries just outside the end of
+ * the section. To identify an orc_unwind_ip address that might
+ * need adjusting, the address should be compared differently
+ * than a normal address.
+ */
+ s = adjust_address_orc(&ip);
+ if (s)
+ cur_orc_ip_table[i] += s->sh_offset;
+ }
+}
+
+static void sort_orc_table(unsigned long map)
+{
+ int num_entries =
+ (addr___stop_orc_unwind_ip - addr___start_orc_unwind_ip) / sizeof(int);
+
+ cur_orc_ip_table = (int *)(addr___start_orc_unwind_ip + map);
+ cur_orc_table = (struct orc_entry *)(addr___start_orc_unwind + map);
+
+ debug_putstr("\nRe-sorting orc tables...\n");
+ sort(cur_orc_ip_table, num_entries, sizeof(int), orc_sort_cmp,
+ orc_sort_swap);
+}
+
+void post_relocations_cleanup(unsigned long map)
+{
+ if (!nofgkaslr) {
+ update_ex_table(map);
+ sort_ex_table(map);
+ update_orc_table(map);
+ sort_orc_table(map);
+ }
+
+ /*
+ * maybe one day free will do something. So, we "free" this memory
+ * in either case
+ */
+ free(sections);
+ free(sechdrs);
+}
+
+void pre_relocations_cleanup(unsigned long map)
+{
+ if (nofgkaslr)
+ return;
+
+ sort_kallsyms(map);
+}
+
+static void shuffle_sections(int *list, int size)
+{
+ int i;
+ unsigned long j;
+ int temp;
+
+ for (i = size - 1; i > 0; i--) {
+ j = kaslr_get_random_long(NULL) % (i + 1);
+
+ temp = list[i];
+ list[i] = list[j];
+ list[j] = temp;
+ }
+}
+
+static void move_text(int num_sections, char *secstrings, Elf_Shdr *text,
+ void *source, void *dest, Elf64_Phdr *phdr)
+{
+ unsigned long adjusted_addr;
+ int copy_bytes;
+ void *stash;
+ Elf_Shdr **sorted_sections;
+ int *index_list;
+ int i, j;
+
+ memmove(dest, source + text->sh_offset, text->sh_size);
+ copy_bytes = text->sh_size;
+ dest += text->sh_size;
+ adjusted_addr = text->sh_addr + text->sh_size;
+
+ /*
+ * we leave the sections sorted in their original order
+ * by s->sh_addr, but shuffle the indexes in a random
+ * order for copying.
+ */
+ index_list = malloc(sizeof(int) * num_sections);
+ if (!index_list)
+ error("Failed to allocate space for index list");
+
+ for (i = 0; i < num_sections; i++)
+ index_list[i] = i;
+
+ shuffle_sections(index_list, num_sections);
+
+ /*
+ * to avoid overwriting earlier sections before they can get
+ * copied to dest, stash everything into a buffer first.
+ * this will cause our source address to be off by
+ * phdr->p_offset though, so we'll adjust s->sh_offset below.
+ *
+ * TBD: ideally we'd simply decompress higher up so that our
+ * copy wasn't in danger of overwriting anything important.
+ */
+ stash = malloc(phdr->p_filesz);
+ if (!stash)
+ error("Failed to allocate space for text stash");
+
+ memcpy(stash, source + phdr->p_offset, phdr->p_filesz);
+
+ /* now we'd walk through the sections. */
+ for (j = 0; j < num_sections; j++) {
+ unsigned long aligned_addr;
+ Elf_Shdr *s;
+ const char *sname;
+ void *src;
+ int pad_bytes;
+
+ s = sections[index_list[j]];
+
+ sname = secstrings + s->sh_name;
+
+ /* align addr for this section */
+ aligned_addr = ALIGN(adjusted_addr, s->sh_addralign);
+
+ /*
+ * copy out of stash, so adjust offset
+ */
+ src = stash + s->sh_offset - phdr->p_offset;
+
+ /*
+ * Fill any space between sections with int3
+ */
+ pad_bytes = aligned_addr - adjusted_addr;
+ memset(dest, 0xcc, pad_bytes);
+
+ dest = (void *)ALIGN((unsigned long)dest, s->sh_addralign);
+
+ memmove(dest, src, s->sh_size);
+
+ dest += s->sh_size;
+ copy_bytes += s->sh_size + pad_bytes;
+ adjusted_addr = aligned_addr + s->sh_size;
+
+ /* we can blow away sh_offset for our own uses */
+ s->sh_offset = aligned_addr - s->sh_addr;
+ }
+
+ free(index_list);
+
+ /*
+ * move remainder of text segment. Ok to just use original source
+ * here since this area is untouched.
+ */
+ memmove(dest, source + text->sh_offset + copy_bytes,
+ phdr->p_filesz - copy_bytes);
+ free(stash);
+}
+
+#define GET_SYM(name) \
+ do { \
+ if (!addr_ ## name) { \
+ if (strcmp(#name, strtab + sym->st_name) == 0) {\
+ addr_ ## name = sym->st_value; \
+ continue; \
+ } \
+ } \
+ } while (0)
+
+static void parse_symtab(Elf64_Sym *symtab, char *strtab, long num_syms)
+{
+ Elf64_Sym *sym;
+
+ if (!symtab || !strtab)
+ return;
+
+ debug_putstr("\nLooking for symbols... ");
+
+ /*
+ * walk through the symbol table looking for the symbols
+ * that we care about.
+ */
+ for (sym = symtab; --num_syms >= 0; sym++) {
+ if (!sym->st_name)
+ continue;
+
+ GET_SYM(kallsyms_num_syms);
+ GET_SYM(kallsyms_offsets);
+ GET_SYM(kallsyms_relative_base);
+ GET_SYM(kallsyms_names);
+ GET_SYM(kallsyms_markers);
+ GET_SYM(__altinstr_replacement);
+ GET_SYM(__altinstr_replacement_end);
+ GET_SYM(_stext);
+ GET_SYM(_etext);
+ GET_SYM(_sinittext);
+ GET_SYM(_einittext);
+ GET_SYM(__start_orc_unwind_ip);
+ GET_SYM(__stop_orc_unwind_ip);
+ GET_SYM(__start_orc_unwind);
+ GET_SYM(__start___ex_table);
+ GET_SYM(__stop___ex_table);
+ }
+}
+
+void layout_randomized_image(void *output, Elf64_Ehdr *ehdr, Elf64_Phdr *phdrs)
+{
+ Elf64_Phdr *phdr;
+ Elf_Shdr *s;
+ Elf_Shdr *text = NULL;
+ Elf_Shdr *percpu = NULL;
+ char *secstrings;
+ const char *sname;
+ int num_sections = 0;
+ Elf64_Sym *symtab = NULL;
+ char *strtab = NULL;
+ long num_syms = 0;
+ void *dest;
+ int i;
+ char arg[MAX_FGKASLR_ARG_LENGTH];
+ Elf_Shdr shdr;
+ unsigned long shnum;
+ unsigned int shstrndx;
+
+ debug_putstr("\nParsing ELF section headers... ");
+
+ /*
+ * Even though fgkaslr may have been disabled, we still
+ * need to parse through the section headers to get the
+ * start and end of the percpu section. This is because
+ * if we were built with CONFIG_FG_KASLR, there are more
+ * relative relocations present in vmlinux.relocs than
+ * just the percpu, and only the percpu relocs need to be
+ * adjusted when using just normal base address kaslr.
+ */
+ if (cmdline_find_option_bool("nofgkaslr")) {
+ warn("FG_KASLR disabled on cmdline.");
+ nofgkaslr = 1;
+ }
+
+ /* read the first section header */
+ shnum = ehdr->e_shnum;
+ shstrndx = ehdr->e_shstrndx;
+ if (shnum == SHN_UNDEF || shstrndx == SHN_XINDEX) {
+ memcpy(&shdr, output + ehdr->e_shoff, sizeof(shdr));
+ if (shnum == SHN_UNDEF)
+ shnum = shdr.sh_size;
+ if (shstrndx == SHN_XINDEX)
+ shstrndx = shdr.sh_link;
+ }
+
+ /* we are going to need to allocate space for the section headers */
+ sechdrs = malloc(sizeof(*sechdrs) * shnum);
+ if (!sechdrs)
+ error("Failed to allocate space for shdrs");
+
+ sections = malloc(sizeof(*sections) * shnum);
+ if (!sections)
+ error("Failed to allocate space for section pointers");
+
+ memcpy(sechdrs, output + ehdr->e_shoff,
+ sizeof(*sechdrs) * shnum);
+
+ /* we need to allocate space for the section string table */
+ s = &sechdrs[shstrndx];
+
+ secstrings = malloc(s->sh_size);
+ if (!secstrings)
+ error("Failed to allocate space for shstr");
+
+ memcpy(secstrings, output + s->sh_offset, s->sh_size);
+
+ /*
+ * now we need to walk through the section headers and collect the
+ * sizes of the .text sections to be randomized.
+ */
+ for (i = 0; i < shnum; i++) {
+ s = &sechdrs[i];
+ sname = secstrings + s->sh_name;
+
+ if (s->sh_type == SHT_SYMTAB) {
+ /* only one symtab per image */
+ if (symtab)
+ error("Unexpected duplicate symtab");
+
+ symtab = malloc(s->sh_size);
+ if (!symtab)
+ error("Failed to allocate space for symtab");
+
+ memcpy(symtab, output + s->sh_offset, s->sh_size);
+ num_syms = s->sh_size / sizeof(*symtab);
+ continue;
+ }
+
+ if (s->sh_type == SHT_STRTAB && i != ehdr->e_shstrndx) {
+ if (strtab)
+ error("Unexpected duplicate strtab");
+
+ strtab = malloc(s->sh_size);
+ if (!strtab)
+ error("Failed to allocate space for strtab");
+
+ memcpy(strtab, output + s->sh_offset, s->sh_size);
+ }
+
+ if (!strcmp(sname, ".text")) {
+ if (text)
+ error("Unexpected duplicate .text section");
+
+ text = s;
+ continue;
+ }
+
+ if (!strcmp(sname, ".data..percpu")) {
+ /* get start addr for later */
+ percpu = s;
+ continue;
+ }
+
+ if (!(s->sh_flags & SHF_ALLOC) ||
+ !(s->sh_flags & SHF_EXECINSTR) ||
+ !(strstarts(sname, ".text")))
+ continue;
+
+ sections[num_sections] = s;
+
+ num_sections++;
+ }
+ sections[num_sections] = NULL;
+ sections_size = num_sections;
+
+ parse_symtab(symtab, strtab, num_syms);
+
+ for (i = 0; i < ehdr->e_phnum; i++) {
+ phdr = &phdrs[i];
+
+ switch (phdr->p_type) {
+ case PT_LOAD:
+ if ((phdr->p_align % 0x200000) != 0)
+ error("Alignment of LOAD segment isn't multiple of 2MB");
+ dest = output;
+ dest += (phdr->p_paddr - LOAD_PHYSICAL_ADDR);
+ if (!nofgkaslr &&
+ (text && phdr->p_offset == text->sh_offset)) {
+ move_text(num_sections, secstrings, text,
+ output, dest, phdr);
+ } else {
+ if (percpu &&
+ phdr->p_offset == percpu->sh_offset) {
+ percpu_start = percpu->sh_addr;
+ percpu_end = percpu_start +
+ phdr->p_filesz;
+ }
+ memmove(dest, output + phdr->p_offset,
+ phdr->p_filesz);
+ }
+ break;
+ default: /* Ignore other PT_* */
+ break;
+ }
+ }
+
+ /* we need to keep the section info to redo relocs */
+ free(secstrings);
+
+ free(phdrs);
+}
diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c
index a4339cb2d247..34b2b3174727 100644
--- a/arch/x86/boot/compressed/misc.c
+++ b/arch/x86/boot/compressed/misc.c
@@ -207,10 +207,21 @@ static void handle_relocations(void *output, unsigned long output_len,
if (IS_ENABLED(CONFIG_X86_64))
delta = virt_addr - LOAD_PHYSICAL_ADDR;
- if (!delta) {
- debug_putstr("No relocation needed... ");
- return;
+ /*
+ * it is possible to have delta be zero and still have enabled
+ * fg kaslr. We need to perform relocations for fgkaslr regardless
+ * of whether the base address has moved.
+ */
+ if (!IS_ENABLED(CONFIG_FG_KASLR) ||
+ cmdline_find_option_bool("nokaslr")) {
+ if (!delta) {
+ debug_putstr("No relocation needed... ");
+ return;
+ }
}
+
+ pre_relocations_cleanup(map);
+
debug_putstr("Performing relocations... ");
/*
@@ -234,35 +245,106 @@ static void handle_relocations(void *output, unsigned long output_len,
*/
for (reloc = output + output_len - sizeof(*reloc); *reloc; reloc--) {
long extended = *reloc;
+ long value;
+
+ /*
+ * if using fgkaslr, we might have moved the address
+ * of the relocation. Check it to see if it needs adjusting
+ * from the original address.
+ */
+ adjust_address(&extended);
+
extended += map;
ptr = (unsigned long)extended;
if (ptr < min_addr || ptr > max_addr)
error("32-bit relocation outside of kernel!\n");
- *(uint32_t *)ptr += delta;
+ value = *(int32_t *)ptr;
+
+ /*
+ * If using fgkaslr, the value of the relocation
+ * might need to be changed because it referred
+ * to an address that has moved.
+ */
+ adjust_address(&value);
+
+ value += delta;
+
+ *(uint32_t *)ptr = value;
}
#ifdef CONFIG_X86_64
while (*--reloc) {
long extended = *reloc;
+ long value;
+ long oldvalue;
+ Elf64_Shdr *s;
+
+ /*
+ * if using fgkaslr, we might have moved the address
+ * of the relocation. Check it to see if it needs adjusting
+ * from the original address.
+ */
+ s = adjust_address(&extended);
+
extended += map;
ptr = (unsigned long)extended;
if (ptr < min_addr || ptr > max_addr)
error("inverse 32-bit relocation outside of kernel!\n");
- *(int32_t *)ptr -= delta;
+ value = *(int32_t *)ptr;
+ oldvalue = value;
+
+ /*
+ * If using fgkaslr, these relocs will contain
+ * relative offsets which might need to be
+ * changed because it referred
+ * to an address that has moved.
+ */
+ adjust_relative_offset(*reloc, &value, s);
+
+ /*
+ * only percpu symbols need to have their values adjusted for
+ * base address kaslr since relative offsets within the .text
+ * and .text.* sections are ok wrt each other.
+ */
+ if (is_percpu_addr(*reloc, oldvalue))
+ value -= delta;
+
+ *(int32_t *)ptr = value;
}
for (reloc--; *reloc; reloc--) {
long extended = *reloc;
+ long value;
+
+ /*
+ * if using fgkaslr, we might have moved the address
+ * of the relocation. Check it to see if it needs adjusting
+ * from the original address.
+ */
+ adjust_address(&extended);
+
extended += map;
ptr = (unsigned long)extended;
if (ptr < min_addr || ptr > max_addr)
error("64-bit relocation outside of kernel!\n");
- *(uint64_t *)ptr += delta;
+ value = *(int64_t *)ptr;
+
+ /*
+ * If using fgkaslr, the value of the relocation
+ * might need to be changed because it referred
+ * to an address that has moved.
+ */
+ adjust_address(&value);
+
+ value += delta;
+
+ *(uint64_t *)ptr = value;
}
+ post_relocations_cleanup(map);
#endif
}
#else
@@ -271,6 +353,35 @@ static inline void handle_relocations(void *output, unsigned long output_len,
{ }
#endif
+static void layout_image(void *output, Elf_Ehdr *ehdr, Elf_Phdr *phdrs)
+{
+ int i;
+ void *dest;
+ Elf_Phdr *phdr;
+
+ for (i = 0; i < ehdr->e_phnum; i++) {
+ phdr = &phdrs[i];
+
+ switch (phdr->p_type) {
+ case PT_LOAD:
+#ifdef CONFIG_X86_64
+ if ((phdr->p_align % 0x200000) != 0)
+ error("Alignment of LOAD segment isn't multiple of 2MB");
+#endif
+#ifdef CONFIG_RELOCATABLE
+ dest = output;
+ dest += (phdr->p_paddr - LOAD_PHYSICAL_ADDR);
+#else
+ dest = (void *)(phdr->p_paddr);
+#endif
+ memmove(dest, output + phdr->p_offset, phdr->p_filesz);
+ break;
+ default: /* Ignore other PT_* */
+ break;
+ }
+ }
+}
+
static void parse_elf(void *output)
{
#ifdef CONFIG_X86_64
@@ -282,6 +393,7 @@ static void parse_elf(void *output)
#endif
void *dest;
int i;
+ int nokaslr;
memcpy(&ehdr, output, sizeof(ehdr));
if (ehdr.e_ident[EI_MAG0] != ELFMAG0 ||
@@ -292,6 +404,12 @@ static void parse_elf(void *output)
return;
}
+ if (IS_ENABLED(CONFIG_FG_KASLR)) {
+ nokaslr = cmdline_find_option_bool("nokaslr");
+ if (nokaslr)
+ warn("FG_KASLR disabled: 'nokaslr' on cmdline.");
+ }
+
debug_putstr("Parsing ELF... ");
phdrs = malloc(sizeof(*phdrs) * ehdr.e_phnum);
@@ -300,26 +418,10 @@ static void parse_elf(void *output)
memcpy(phdrs, output + ehdr.e_phoff, sizeof(*phdrs) * ehdr.e_phnum);
- for (i = 0; i < ehdr.e_phnum; i++) {
- phdr = &phdrs[i];
-
- switch (phdr->p_type) {
- case PT_LOAD:
-#ifdef CONFIG_X86_64
- if ((phdr->p_align % 0x200000) != 0)
- error("Alignment of LOAD segment isn't multiple of 2MB");
-#endif
-#ifdef CONFIG_RELOCATABLE
- dest = output;
- dest += (phdr->p_paddr - LOAD_PHYSICAL_ADDR);
-#else
- dest = (void *)(phdr->p_paddr);
-#endif
- memmove(dest, output + phdr->p_offset, phdr->p_filesz);
- break;
- default: /* Ignore other PT_* */ break;
- }
- }
+ if (IS_ENABLED(CONFIG_FG_KASLR) && !nokaslr)
+ layout_randomized_image(output, &ehdr, phdrs);
+ else
+ layout_image(output, &ehdr, phdrs);
free(phdrs);
}
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 16ed360b6692..1315a101c1c9 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -83,6 +83,34 @@ struct mem_vector {
u64 size;
};
+#ifdef CONFIG_X86_64
+#define Elf_Ehdr Elf64_Ehdr
+#define Elf_Phdr Elf64_Phdr
+#define Elf_Shdr Elf64_Shdr
+#else
+#define Elf_Ehdr Elf32_Ehdr
+#define Elf_Phdr Elf32_Phdr
+#define Elf_Shdr Elf32_Shdr
+#endif
+
+#ifdef CONFIG_FG_KASLR
+void layout_randomized_image(void *output, Elf_Ehdr *ehdr, Elf_Phdr *phdrs);
+void pre_relocations_cleanup(unsigned long map);
+void post_relocations_cleanup(unsigned long map);
+Elf_Shdr *adjust_address(long *address);
+void adjust_relative_offset(long pc, long *value, Elf_Shdr *section);
+bool is_percpu_addr(long pc, long offset);
+#else
+static inline void layout_randomized_image(void *output, Elf_Ehdr *ehdr,
+ Elf_Phdr *phdrs) { }
+static inline void pre_relocations_cleanup(unsigned long map) { }
+static inline void post_relocations_cleanup(unsigned long map) { }
+static inline Elf_Shdr *adjust_address(long *address) { return NULL; }
+static inline void adjust_relative_offset(long pc, long *value,
+ Elf_Shdr *section) { }
+static inline bool is_percpu_addr(long pc, long offset) { return true; }
+#endif
+
#ifdef CONFIG_RANDOMIZE_BASE
/* kaslr.c */
void choose_random_location(unsigned long input,
diff --git a/arch/x86/boot/compressed/utils.c b/arch/x86/boot/compressed/utils.c
new file mode 100644
index 000000000000..7c3c745f6251
--- /dev/null
+++ b/arch/x86/boot/compressed/utils.c
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * This contains various libraries that are needed for FG-KASLR.
+ *
+ * Copyright (C) 2020-2021, Intel Corporation.
+ * Author: Kristen Carlson Accardi <kristen@linux.intel.com>
+ */
+
+#define _LINUX_KPROBES_H
+#define NOKPROBE_SYMBOL(fname)
+
+#include "../../../../lib/sort.c"
+#include "../../../../lib/bsearch.c"
diff --git a/arch/x86/boot/compressed/vmlinux.symbols b/arch/x86/boot/compressed/vmlinux.symbols
new file mode 100644
index 000000000000..da41f3ee153c
--- /dev/null
+++ b/arch/x86/boot/compressed/vmlinux.symbols
@@ -0,0 +1,19 @@
+kallsyms_offsets
+kallsyms_addresses
+kallsyms_num_syms
+kallsyms_relative_base
+kallsyms_names
+kallsyms_token_table
+kallsyms_token_index
+kallsyms_markers
+__start___ex_table
+__stop___ex_table
+__altinstr_replacement
+__altinstr_replacement_end
+_sinittext
+_einittext
+_stext
+_etext
+__start_orc_unwind_ip
+__stop_orc_unwind_ip
+__start_orc_unwind
diff --git a/arch/x86/include/asm/boot.h b/arch/x86/include/asm/boot.h
index 9191280d9ea3..ce5fdee49046 100644
--- a/arch/x86/include/asm/boot.h
+++ b/arch/x86/include/asm/boot.h
@@ -24,7 +24,18 @@
# error "Invalid value for CONFIG_PHYSICAL_ALIGN"
#endif
-#if defined(CONFIG_KERNEL_BZIP2)
+#ifdef CONFIG_FG_KASLR
+/*
+ * We need extra boot heap when using fgkaslr because we make a copy
+ * of the original decompressed kernel to avoid issues with writing
+ * over ourselves when shuffling the sections. We also need extra
+ * space for resorting kallsyms after shuffling. This value could
+ * be decreased if free() would release memory properly, or if we
+ * could avoid the kernel copy. It would need to be increased if we
+ * find additional tables that need to be resorted.
+ */
+# define BOOT_HEAP_SIZE 0x4800000
+#elif defined(CONFIG_KERNEL_BZIP2)
# define BOOT_HEAP_SIZE 0x400000
#elif defined(CONFIG_KERNEL_ZSTD)
/*
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 3d6dc12d198f..e1db74452485 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -304,7 +304,9 @@ SECTIONS
* get the address and the length of them to patch the kernel safely.
*/
.altinstr_replacement : AT(ADDR(.altinstr_replacement) - LOAD_OFFSET) {
+ __altinstr_replacement = .;
*(.altinstr_replacement)
+ __altinstr_replacement_end = .;
}
/*
diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h
index 61bf4774b8f2..1c74d9594919 100644
--- a/include/uapi/linux/elf.h
+++ b/include/uapi/linux/elf.h
@@ -299,6 +299,7 @@ typedef struct elf64_phdr {
#define SHN_LIVEPATCH 0xff20
#define SHN_ABS 0xfff1
#define SHN_COMMON 0xfff2
+#define SHN_XINDEX 0xffff
#define SHN_HIRESERVE 0xffff
typedef struct elf32_shdr {
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 04/14] linkage: add macros for putting ASM functions into own sections
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
` (2 preceding siblings ...)
2021-12-02 22:32 ` [PATCH v8 03/14] x86: Add support for function granular KASLR Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-03 9:31 ` Peter Zijlstra
2021-12-02 22:32 ` [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections Alexander Lobakin
` (10 subsequent siblings)
14 siblings, 1 reply; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
With ClangLTO or -ffunction-sections (DCE, FG-KASLR), compiler
places C functions into separate sections by default.
However, this doesn't happen with ASM functions which are still
being placed into .text.
Introduce a pack of macros which generate a new unique section
for the describing function named in the same fashion
(.text.<func_name>).
This will be needed to make input .text section empty to harden
the kernel even more.
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
include/linux/linkage.h | 82 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 82 insertions(+)
diff --git a/include/linux/linkage.h b/include/linux/linkage.h
index dbf8506decca..baaab7dece08 100644
--- a/include/linux/linkage.h
+++ b/include/linux/linkage.h
@@ -355,4 +355,86 @@
#endif /* __ASSEMBLY__ */
+/*
+ * Allow ASM symbols to have their own unique sections if they are being
+ * generated by the compiler for C functions (DCE, FG-KASLR, LTO).
+ */
+#if (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) && !defined(MODULE)) || \
+ (defined(CONFIG_FG_KASLR) && !defined(MODULE)) || \
+ (defined(CONFIG_MODULE_FG_KASLR) && defined(MODULE)) || \
+ (defined(CONFIG_LTO_CLANG))
+
+#define SYM_TEXT_SECTION(name) \
+ .pushsection .text.##name, "ax"
+
+#define ASM_TEXT_SECTION(name) \
+ ".text." #name
+
+#define ASM_PUSH_SECTION(name) \
+ ".pushsection .text." #name ", \"ax\""
+
+#else /* just .text */
+
+#define SYM_TEXT_SECTION(name) \
+ .pushsection .text, "ax"
+
+#define ASM_TEXT_SECTION(name) \
+ ".text"
+
+#define ASM_PUSH_SECTION(name) \
+ ".pushsection .text, \"ax\""
+
+#endif /* just .text */
+
+#ifdef __ASSEMBLY__
+
+#define SYM_TEXT_END_SECTION \
+ .popsection
+
+#define SYM_FUNC_START_LOCAL_ALIAS_SECTION(name) \
+ SYM_TEXT_SECTION(name) ASM_NL \
+ SYM_FUNC_START_LOCAL_ALIAS(name)
+
+#define SYM_FUNC_START_LOCAL_SECTION(name) \
+ SYM_TEXT_SECTION(name) ASM_NL \
+ SYM_FUNC_START_LOCAL(name)
+
+#define SYM_FUNC_START_NOALIGN_SECTION(name) \
+ SYM_TEXT_SECTION(name) ASM_NL \
+ SYM_FUNC_START_NOALIGN(name)
+
+#define SYM_FUNC_START_WEAK_SECTION(name) \
+ SYM_TEXT_SECTION(name) ASM_NL \
+ SYM_FUNC_START_WEAK(name)
+
+#define SYM_FUNC_START_SECTION(name) \
+ SYM_TEXT_SECTION(name) ASM_NL \
+ SYM_FUNC_START(name)
+
+#define SYM_CODE_START_LOCAL_NOALIGN_SECTION(name) \
+ SYM_TEXT_SECTION(name) ASM_NL \
+ SYM_CODE_START_LOCAL_NOALIGN(name)
+
+#define SYM_CODE_START_NOALIGN_SECTION(name) \
+ SYM_TEXT_SECTION(name) ASM_NL \
+ SYM_CODE_START_NOALIGN(name)
+
+#define SYM_CODE_START_SECTION(name) \
+ SYM_TEXT_SECTION(name) ASM_NL \
+ SYM_CODE_START(name)
+
+#define SYM_FUNC_END_ALIAS_SECTION(name) \
+ SYM_FUNC_END_ALIAS(name) ASM_NL \
+ SYM_TEXT_END_SECTION
+
+#define SYM_FUNC_END_SECTION(name) \
+ SYM_FUNC_END(name) ASM_NL \
+ SYM_TEXT_END_SECTION
+
+#define SYM_CODE_END_SECTION(name) \
+ SYM_CODE_END(name) ASM_NL \
+ SYM_TEXT_END_SECTION
+
+#endif /* __ASSEMBLY__ */
+
#endif /* _LINUX_LINKAGE_H */
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
` (3 preceding siblings ...)
2021-12-02 22:32 ` [PATCH v8 04/14] linkage: add macros for putting ASM functions into own sections Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-03 9:44 ` Peter Zijlstra
2021-12-02 22:32 ` [PATCH v8 06/14] FG-KASLR: use a scripted approach to handle .text.* sections Alexander Lobakin
` (9 subsequent siblings)
14 siblings, 1 reply; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
Use the newly introduces macros to create unique separate sections
for (almost) every "regular" ASM function (i.e. for those which
aren't explicitly put into a specific one).
There should be no leftovers as input .text will be size-asserted
in the LD script generated for FG-KASLR.
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
arch/x86/crypto/aegis128-aesni-asm.S | 36 +++---
arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 12 +-
arch/x86/crypto/aesni-intel_asm.S | 116 +++++++++---------
arch/x86/crypto/aesni-intel_avx-x86_64.S | 32 ++---
arch/x86/crypto/blake2s-core.S | 8 +-
arch/x86/crypto/blowfish-x86_64-asm_64.S | 16 +--
arch/x86/crypto/camellia-aesni-avx-asm_64.S | 28 ++---
arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 28 ++---
arch/x86/crypto/camellia-x86_64-asm_64.S | 16 +--
arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 24 ++--
arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 20 +--
arch/x86/crypto/chacha-avx2-x86_64.S | 12 +-
arch/x86/crypto/chacha-avx512vl-x86_64.S | 12 +-
arch/x86/crypto/chacha-ssse3-x86_64.S | 16 +--
arch/x86/crypto/crc32-pclmul_asm.S | 4 +-
arch/x86/crypto/crc32c-pcl-intel-asm_64.S | 4 +-
arch/x86/crypto/crct10dif-pcl-asm_64.S | 4 +-
arch/x86/crypto/des3_ede-asm_64.S | 8 +-
arch/x86/crypto/ghash-clmulni-intel_asm.S | 12 +-
arch/x86/crypto/nh-avx2-x86_64.S | 4 +-
arch/x86/crypto/nh-sse2-x86_64.S | 4 +-
arch/x86/crypto/poly1305-x86_64-cryptogams.pl | 8 +-
arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 20 +--
arch/x86/crypto/serpent-avx2-asm_64.S | 20 +--
arch/x86/crypto/serpent-sse2-i586-asm_32.S | 8 +-
arch/x86/crypto/serpent-sse2-x86_64-asm_64.S | 8 +-
arch/x86/crypto/sha1_avx2_x86_64_asm.S | 4 +-
arch/x86/crypto/sha1_ni_asm.S | 4 +-
arch/x86/crypto/sha1_ssse3_asm.S | 4 +-
arch/x86/crypto/sha256-avx-asm.S | 4 +-
arch/x86/crypto/sha256-avx2-asm.S | 4 +-
arch/x86/crypto/sha256-ssse3-asm.S | 4 +-
arch/x86/crypto/sha256_ni_asm.S | 4 +-
arch/x86/crypto/sha512-avx-asm.S | 4 +-
arch/x86/crypto/sha512-avx2-asm.S | 4 +-
arch/x86/crypto/sha512-ssse3-asm.S | 4 +-
arch/x86/crypto/sm4-aesni-avx-asm_64.S | 24 ++--
arch/x86/crypto/sm4-aesni-avx2-asm_64.S | 16 +--
arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 20 +--
arch/x86/crypto/twofish-i586-asm_32.S | 8 +-
arch/x86/crypto/twofish-x86_64-asm_64-3way.S | 8 +-
arch/x86/crypto/twofish-x86_64-asm_64.S | 8 +-
arch/x86/entry/entry_32.S | 24 ++--
arch/x86/entry/entry_64.S | 18 +--
arch/x86/entry/thunk_32.S | 4 +-
arch/x86/entry/thunk_64.S | 8 +-
arch/x86/include/asm/paravirt.h | 5 +-
arch/x86/include/asm/qspinlock_paravirt.h | 2 +-
arch/x86/kernel/acpi/wakeup_32.S | 9 +-
arch/x86/kernel/acpi/wakeup_64.S | 10 +-
arch/x86/kernel/ftrace_32.S | 19 +--
arch/x86/kernel/ftrace_64.S | 24 ++--
arch/x86/kernel/kprobes/core.c | 3 +-
arch/x86/kernel/kvm.c | 2 +-
arch/x86/kernel/relocate_kernel_32.S | 2 +
arch/x86/kernel/relocate_kernel_64.S | 2 +
arch/x86/kvm/emulate.c | 2 +-
arch/x86/kvm/vmx/vmenter.S | 8 +-
arch/x86/lib/clear_page_64.S | 12 +-
arch/x86/lib/cmpxchg16b_emu.S | 4 +-
arch/x86/lib/copy_mc_64.S | 8 +-
arch/x86/lib/copy_page_64.S | 7 +-
arch/x86/lib/copy_user_64.S | 18 +--
arch/x86/lib/csum-copy_64.S | 4 +-
arch/x86/lib/error-inject.c | 3 +-
arch/x86/lib/getuser.S | 37 +++---
arch/x86/lib/hweight.S | 9 +-
arch/x86/lib/iomap_copy_64.S | 4 +-
arch/x86/lib/memmove_64.S | 4 +-
arch/x86/lib/memset_64.S | 12 +-
arch/x86/lib/msr-reg.S | 8 +-
arch/x86/lib/putuser.S | 18 +--
arch/x86/mm/mem_encrypt_boot.S | 8 +-
arch/x86/platform/efi/efi_stub_64.S | 4 +-
arch/x86/platform/efi/efi_thunk_64.S | 4 +-
arch/x86/power/hibernate_asm_32.S | 14 ++-
arch/x86/power/hibernate_asm_64.S | 14 ++-
arch/x86/xen/xen-asm.S | 24 ++--
arch/x86/xen/xen-head.S | 6 +-
79 files changed, 486 insertions(+), 482 deletions(-)
diff --git a/arch/x86/crypto/aegis128-aesni-asm.S b/arch/x86/crypto/aegis128-aesni-asm.S
index 51d46d93efbc..0f2d5d77fc6e 100644
--- a/arch/x86/crypto/aegis128-aesni-asm.S
+++ b/arch/x86/crypto/aegis128-aesni-asm.S
@@ -71,7 +71,7 @@
* %r8
* %r9
*/
-SYM_FUNC_START_LOCAL(__load_partial)
+SYM_FUNC_START_LOCAL_SECTION(__load_partial)
xor %r9d, %r9d
pxor MSG, MSG
@@ -123,7 +123,7 @@ SYM_FUNC_START_LOCAL(__load_partial)
.Lld_partial_8:
ret
-SYM_FUNC_END(__load_partial)
+SYM_FUNC_END_SECTION(__load_partial)
/*
* __store_partial: internal ABI
@@ -137,7 +137,7 @@ SYM_FUNC_END(__load_partial)
* %r9
* %r10
*/
-SYM_FUNC_START_LOCAL(__store_partial)
+SYM_FUNC_START_LOCAL_SECTION(__store_partial)
mov LEN, %r8
mov DST, %r9
@@ -181,12 +181,12 @@ SYM_FUNC_START_LOCAL(__store_partial)
.Lst_partial_1:
ret
-SYM_FUNC_END(__store_partial)
+SYM_FUNC_END_SECTION(__store_partial)
/*
* void crypto_aegis128_aesni_init(void *state, const void *key, const void *iv);
*/
-SYM_FUNC_START(crypto_aegis128_aesni_init)
+SYM_FUNC_START_SECTION(crypto_aegis128_aesni_init)
FRAME_BEGIN
/* load IV: */
@@ -226,13 +226,13 @@ SYM_FUNC_START(crypto_aegis128_aesni_init)
FRAME_END
ret
-SYM_FUNC_END(crypto_aegis128_aesni_init)
+SYM_FUNC_END_SECTION(crypto_aegis128_aesni_init)
/*
* void crypto_aegis128_aesni_ad(void *state, unsigned int length,
* const void *data);
*/
-SYM_FUNC_START(crypto_aegis128_aesni_ad)
+SYM_FUNC_START_SECTION(crypto_aegis128_aesni_ad)
FRAME_BEGIN
cmp $0x10, LEN
@@ -378,7 +378,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_ad)
.Lad_out:
FRAME_END
ret
-SYM_FUNC_END(crypto_aegis128_aesni_ad)
+SYM_FUNC_END_SECTION(crypto_aegis128_aesni_ad)
.macro encrypt_block a s0 s1 s2 s3 s4 i
movdq\a (\i * 0x10)(SRC), MSG
@@ -402,7 +402,7 @@ SYM_FUNC_END(crypto_aegis128_aesni_ad)
* void crypto_aegis128_aesni_enc(void *state, unsigned int length,
* const void *src, void *dst);
*/
-SYM_FUNC_START(crypto_aegis128_aesni_enc)
+SYM_FUNC_START_SECTION(crypto_aegis128_aesni_enc)
FRAME_BEGIN
cmp $0x10, LEN
@@ -493,13 +493,13 @@ SYM_FUNC_START(crypto_aegis128_aesni_enc)
.Lenc_out:
FRAME_END
ret
-SYM_FUNC_END(crypto_aegis128_aesni_enc)
+SYM_FUNC_END_SECTION(crypto_aegis128_aesni_enc)
/*
* void crypto_aegis128_aesni_enc_tail(void *state, unsigned int length,
* const void *src, void *dst);
*/
-SYM_FUNC_START(crypto_aegis128_aesni_enc_tail)
+SYM_FUNC_START_SECTION(crypto_aegis128_aesni_enc_tail)
FRAME_BEGIN
/* load the state: */
@@ -533,7 +533,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_enc_tail)
FRAME_END
ret
-SYM_FUNC_END(crypto_aegis128_aesni_enc_tail)
+SYM_FUNC_END_SECTION(crypto_aegis128_aesni_enc_tail)
.macro decrypt_block a s0 s1 s2 s3 s4 i
movdq\a (\i * 0x10)(SRC), MSG
@@ -556,7 +556,7 @@ SYM_FUNC_END(crypto_aegis128_aesni_enc_tail)
* void crypto_aegis128_aesni_dec(void *state, unsigned int length,
* const void *src, void *dst);
*/
-SYM_FUNC_START(crypto_aegis128_aesni_dec)
+SYM_FUNC_START_SECTION(crypto_aegis128_aesni_dec)
FRAME_BEGIN
cmp $0x10, LEN
@@ -647,13 +647,13 @@ SYM_FUNC_START(crypto_aegis128_aesni_dec)
.Ldec_out:
FRAME_END
ret
-SYM_FUNC_END(crypto_aegis128_aesni_dec)
+SYM_FUNC_END_SECTION(crypto_aegis128_aesni_dec)
/*
* void crypto_aegis128_aesni_dec_tail(void *state, unsigned int length,
* const void *src, void *dst);
*/
-SYM_FUNC_START(crypto_aegis128_aesni_dec_tail)
+SYM_FUNC_START_SECTION(crypto_aegis128_aesni_dec_tail)
FRAME_BEGIN
/* load the state: */
@@ -697,13 +697,13 @@ SYM_FUNC_START(crypto_aegis128_aesni_dec_tail)
FRAME_END
ret
-SYM_FUNC_END(crypto_aegis128_aesni_dec_tail)
+SYM_FUNC_END_SECTION(crypto_aegis128_aesni_dec_tail)
/*
* void crypto_aegis128_aesni_final(void *state, void *tag_xor,
* u64 assoclen, u64 cryptlen);
*/
-SYM_FUNC_START(crypto_aegis128_aesni_final)
+SYM_FUNC_START_SECTION(crypto_aegis128_aesni_final)
FRAME_BEGIN
/* load the state: */
@@ -744,4 +744,4 @@ SYM_FUNC_START(crypto_aegis128_aesni_final)
FRAME_END
ret
-SYM_FUNC_END(crypto_aegis128_aesni_final)
+SYM_FUNC_END_SECTION(crypto_aegis128_aesni_final)
diff --git a/arch/x86/crypto/aes_ctrby8_avx-x86_64.S b/arch/x86/crypto/aes_ctrby8_avx-x86_64.S
index 3f0fc7dd87d7..b9968403c05c 100644
--- a/arch/x86/crypto/aes_ctrby8_avx-x86_64.S
+++ b/arch/x86/crypto/aes_ctrby8_avx-x86_64.S
@@ -535,11 +535,11 @@ ddq_add_8:
* aes_ctr_enc_128_avx_by8(void *in, void *iv, void *keys, void *out,
* unsigned int num_bytes)
*/
-SYM_FUNC_START(aes_ctr_enc_128_avx_by8)
+SYM_FUNC_START_SECTION(aes_ctr_enc_128_avx_by8)
/* call the aes main loop */
do_aes_ctrmain KEY_128
-SYM_FUNC_END(aes_ctr_enc_128_avx_by8)
+SYM_FUNC_END_SECTION(aes_ctr_enc_128_avx_by8)
/*
* routine to do AES192 CTR enc/decrypt "by8"
@@ -548,11 +548,11 @@ SYM_FUNC_END(aes_ctr_enc_128_avx_by8)
* aes_ctr_enc_192_avx_by8(void *in, void *iv, void *keys, void *out,
* unsigned int num_bytes)
*/
-SYM_FUNC_START(aes_ctr_enc_192_avx_by8)
+SYM_FUNC_START_SECTION(aes_ctr_enc_192_avx_by8)
/* call the aes main loop */
do_aes_ctrmain KEY_192
-SYM_FUNC_END(aes_ctr_enc_192_avx_by8)
+SYM_FUNC_END_SECTION(aes_ctr_enc_192_avx_by8)
/*
* routine to do AES256 CTR enc/decrypt "by8"
@@ -561,8 +561,8 @@ SYM_FUNC_END(aes_ctr_enc_192_avx_by8)
* aes_ctr_enc_256_avx_by8(void *in, void *iv, void *keys, void *out,
* unsigned int num_bytes)
*/
-SYM_FUNC_START(aes_ctr_enc_256_avx_by8)
+SYM_FUNC_START_SECTION(aes_ctr_enc_256_avx_by8)
/* call the aes main loop */
do_aes_ctrmain KEY_256
-SYM_FUNC_END(aes_ctr_enc_256_avx_by8)
+SYM_FUNC_END_SECTION(aes_ctr_enc_256_avx_by8)
diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-intel_asm.S
index 4e3972570916..5c1076b12df8 100644
--- a/arch/x86/crypto/aesni-intel_asm.S
+++ b/arch/x86/crypto/aesni-intel_asm.S
@@ -1587,7 +1587,7 @@ _esb_loop_\@:
* poly = x^128 + x^127 + x^126 + x^121 + 1
*
*****************************************************************************/
-SYM_FUNC_START(aesni_gcm_dec)
+SYM_FUNC_START_SECTION(aesni_gcm_dec)
FUNC_SAVE
GCM_INIT %arg6, arg7, arg8, arg9
@@ -1595,8 +1595,7 @@ SYM_FUNC_START(aesni_gcm_dec)
GCM_COMPLETE arg10, arg11
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_dec)
-
+SYM_FUNC_END_SECTION(aesni_gcm_dec)
/*****************************************************************************
* void aesni_gcm_enc(void *aes_ctx, // AES Key schedule. Starts on a 16 byte boundary.
@@ -1675,7 +1674,7 @@ SYM_FUNC_END(aesni_gcm_dec)
*
* poly = x^128 + x^127 + x^126 + x^121 + 1
***************************************************************************/
-SYM_FUNC_START(aesni_gcm_enc)
+SYM_FUNC_START_SECTION(aesni_gcm_enc)
FUNC_SAVE
GCM_INIT %arg6, arg7, arg8, arg9
@@ -1684,7 +1683,7 @@ SYM_FUNC_START(aesni_gcm_enc)
GCM_COMPLETE arg10, arg11
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_enc)
+SYM_FUNC_END_SECTION(aesni_gcm_enc)
/*****************************************************************************
* void aesni_gcm_init(void *aes_ctx, // AES Key schedule. Starts on a 16 byte boundary.
@@ -1697,12 +1696,12 @@ SYM_FUNC_END(aesni_gcm_enc)
* const u8 *aad, // Additional Authentication Data (AAD)
* u64 aad_len) // Length of AAD in bytes.
*/
-SYM_FUNC_START(aesni_gcm_init)
+SYM_FUNC_START_SECTION(aesni_gcm_init)
FUNC_SAVE
GCM_INIT %arg3, %arg4,%arg5, %arg6
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_init)
+SYM_FUNC_END_SECTION(aesni_gcm_init)
/*****************************************************************************
* void aesni_gcm_enc_update(void *aes_ctx, // AES Key schedule. Starts on a 16 byte boundary.
@@ -1712,12 +1711,12 @@ SYM_FUNC_END(aesni_gcm_init)
* const u8 *in, // Plaintext input
* u64 plaintext_len, // Length of data in bytes for encryption.
*/
-SYM_FUNC_START(aesni_gcm_enc_update)
+SYM_FUNC_START_SECTION(aesni_gcm_enc_update)
FUNC_SAVE
GCM_ENC_DEC enc
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_enc_update)
+SYM_FUNC_END_SECTION(aesni_gcm_enc_update)
/*****************************************************************************
* void aesni_gcm_dec_update(void *aes_ctx, // AES Key schedule. Starts on a 16 byte boundary.
@@ -1727,12 +1726,12 @@ SYM_FUNC_END(aesni_gcm_enc_update)
* const u8 *in, // Plaintext input
* u64 plaintext_len, // Length of data in bytes for encryption.
*/
-SYM_FUNC_START(aesni_gcm_dec_update)
+SYM_FUNC_START_SECTION(aesni_gcm_dec_update)
FUNC_SAVE
GCM_ENC_DEC dec
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_dec_update)
+SYM_FUNC_END_SECTION(aesni_gcm_dec_update)
/*****************************************************************************
* void aesni_gcm_finalize(void *aes_ctx, // AES Key schedule. Starts on a 16 byte boundary.
@@ -1742,17 +1741,15 @@ SYM_FUNC_END(aesni_gcm_dec_update)
* u64 auth_tag_len); // Authenticated Tag Length in bytes. Valid values are 16 (most likely),
* // 12 or 8.
*/
-SYM_FUNC_START(aesni_gcm_finalize)
+SYM_FUNC_START_SECTION(aesni_gcm_finalize)
FUNC_SAVE
GCM_COMPLETE %arg3 %arg4
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_finalize)
-
+SYM_FUNC_END_SECTION(aesni_gcm_finalize)
#endif
-
-SYM_FUNC_START_LOCAL_ALIAS(_key_expansion_128)
+SYM_FUNC_START_LOCAL_ALIAS_SECTION(_key_expansion_128)
SYM_FUNC_START_LOCAL(_key_expansion_256a)
pshufd $0b11111111, %xmm1, %xmm1
shufps $0b00010000, %xmm0, %xmm4
@@ -1764,9 +1761,9 @@ SYM_FUNC_START_LOCAL(_key_expansion_256a)
add $0x10, TKEYP
ret
SYM_FUNC_END(_key_expansion_256a)
-SYM_FUNC_END_ALIAS(_key_expansion_128)
+SYM_FUNC_END_ALIAS_SECTION(_key_expansion_128)
-SYM_FUNC_START_LOCAL(_key_expansion_192a)
+SYM_FUNC_START_LOCAL_SECTION(_key_expansion_192a)
pshufd $0b01010101, %xmm1, %xmm1
shufps $0b00010000, %xmm0, %xmm4
pxor %xmm4, %xmm0
@@ -1788,9 +1785,9 @@ SYM_FUNC_START_LOCAL(_key_expansion_192a)
movaps %xmm1, 0x10(TKEYP)
add $0x20, TKEYP
ret
-SYM_FUNC_END(_key_expansion_192a)
+SYM_FUNC_END_SECTION(_key_expansion_192a)
-SYM_FUNC_START_LOCAL(_key_expansion_192b)
+SYM_FUNC_START_LOCAL_SECTION(_key_expansion_192b)
pshufd $0b01010101, %xmm1, %xmm1
shufps $0b00010000, %xmm0, %xmm4
pxor %xmm4, %xmm0
@@ -1807,9 +1804,9 @@ SYM_FUNC_START_LOCAL(_key_expansion_192b)
movaps %xmm0, (TKEYP)
add $0x10, TKEYP
ret
-SYM_FUNC_END(_key_expansion_192b)
+SYM_FUNC_END_SECTION(_key_expansion_192b)
-SYM_FUNC_START_LOCAL(_key_expansion_256b)
+SYM_FUNC_START_LOCAL_SECTION(_key_expansion_256b)
pshufd $0b10101010, %xmm1, %xmm1
shufps $0b00010000, %xmm2, %xmm4
pxor %xmm4, %xmm2
@@ -1819,13 +1816,13 @@ SYM_FUNC_START_LOCAL(_key_expansion_256b)
movaps %xmm2, (TKEYP)
add $0x10, TKEYP
ret
-SYM_FUNC_END(_key_expansion_256b)
+SYM_FUNC_END_SECTION(_key_expansion_256b)
/*
* int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
* unsigned int key_len)
*/
-SYM_FUNC_START(aesni_set_key)
+SYM_FUNC_START_SECTION(aesni_set_key)
FRAME_BEGIN
#ifndef __x86_64__
pushl KEYP
@@ -1934,12 +1931,12 @@ SYM_FUNC_START(aesni_set_key)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_set_key)
+SYM_FUNC_END_SECTION(aesni_set_key)
/*
* void aesni_enc(const void *ctx, u8 *dst, const u8 *src)
*/
-SYM_FUNC_START(aesni_enc)
+SYM_FUNC_START_SECTION(aesni_enc)
FRAME_BEGIN
#ifndef __x86_64__
pushl KEYP
@@ -1958,7 +1955,7 @@ SYM_FUNC_START(aesni_enc)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_enc)
+SYM_FUNC_END_SECTION(aesni_enc)
/*
* _aesni_enc1: internal ABI
@@ -1972,7 +1969,7 @@ SYM_FUNC_END(aesni_enc)
* KEY
* TKEYP (T1)
*/
-SYM_FUNC_START_LOCAL(_aesni_enc1)
+SYM_FUNC_START_LOCAL_SECTION(_aesni_enc1)
movaps (KEYP), KEY # key
mov KEYP, TKEYP
pxor KEY, STATE # round 0
@@ -2015,7 +2012,7 @@ SYM_FUNC_START_LOCAL(_aesni_enc1)
movaps 0x70(TKEYP), KEY
aesenclast KEY, STATE
ret
-SYM_FUNC_END(_aesni_enc1)
+SYM_FUNC_END_SECTION(_aesni_enc1)
/*
* _aesni_enc4: internal ABI
@@ -2035,7 +2032,7 @@ SYM_FUNC_END(_aesni_enc1)
* KEY
* TKEYP (T1)
*/
-SYM_FUNC_START_LOCAL(_aesni_enc4)
+SYM_FUNC_START_LOCAL_SECTION(_aesni_enc4)
movaps (KEYP), KEY # key
mov KEYP, TKEYP
pxor KEY, STATE1 # round 0
@@ -2123,12 +2120,12 @@ SYM_FUNC_START_LOCAL(_aesni_enc4)
aesenclast KEY, STATE3
aesenclast KEY, STATE4
ret
-SYM_FUNC_END(_aesni_enc4)
+SYM_FUNC_END_SECTION(_aesni_enc4)
/*
* void aesni_dec (const void *ctx, u8 *dst, const u8 *src)
*/
-SYM_FUNC_START(aesni_dec)
+SYM_FUNC_START_SECTION(aesni_dec)
FRAME_BEGIN
#ifndef __x86_64__
pushl KEYP
@@ -2148,7 +2145,7 @@ SYM_FUNC_START(aesni_dec)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_dec)
+SYM_FUNC_END_SECTION(aesni_dec)
/*
* _aesni_dec1: internal ABI
@@ -2162,7 +2159,7 @@ SYM_FUNC_END(aesni_dec)
* KEY
* TKEYP (T1)
*/
-SYM_FUNC_START_LOCAL(_aesni_dec1)
+SYM_FUNC_START_LOCAL_SECTION(_aesni_dec1)
movaps (KEYP), KEY # key
mov KEYP, TKEYP
pxor KEY, STATE # round 0
@@ -2205,7 +2202,7 @@ SYM_FUNC_START_LOCAL(_aesni_dec1)
movaps 0x70(TKEYP), KEY
aesdeclast KEY, STATE
ret
-SYM_FUNC_END(_aesni_dec1)
+SYM_FUNC_END_SECTION(_aesni_dec1)
/*
* _aesni_dec4: internal ABI
@@ -2225,7 +2222,7 @@ SYM_FUNC_END(_aesni_dec1)
* KEY
* TKEYP (T1)
*/
-SYM_FUNC_START_LOCAL(_aesni_dec4)
+SYM_FUNC_START_LOCAL_SECTION(_aesni_dec4)
movaps (KEYP), KEY # key
mov KEYP, TKEYP
pxor KEY, STATE1 # round 0
@@ -2313,13 +2310,13 @@ SYM_FUNC_START_LOCAL(_aesni_dec4)
aesdeclast KEY, STATE3
aesdeclast KEY, STATE4
ret
-SYM_FUNC_END(_aesni_dec4)
+SYM_FUNC_END_SECTION(_aesni_dec4)
/*
* void aesni_ecb_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
* size_t len)
*/
-SYM_FUNC_START(aesni_ecb_enc)
+SYM_FUNC_START_SECTION(aesni_ecb_enc)
FRAME_BEGIN
#ifndef __x86_64__
pushl LEN
@@ -2373,13 +2370,13 @@ SYM_FUNC_START(aesni_ecb_enc)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_ecb_enc)
+SYM_FUNC_END_SECTION(aesni_ecb_enc)
/*
* void aesni_ecb_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
* size_t len);
*/
-SYM_FUNC_START(aesni_ecb_dec)
+SYM_FUNC_START_SECTION(aesni_ecb_dec)
FRAME_BEGIN
#ifndef __x86_64__
pushl LEN
@@ -2434,13 +2431,13 @@ SYM_FUNC_START(aesni_ecb_dec)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_ecb_dec)
+SYM_FUNC_END_SECTION(aesni_ecb_dec)
/*
* void aesni_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
* size_t len, u8 *iv)
*/
-SYM_FUNC_START(aesni_cbc_enc)
+SYM_FUNC_START_SECTION(aesni_cbc_enc)
FRAME_BEGIN
#ifndef __x86_64__
pushl IVP
@@ -2478,13 +2475,13 @@ SYM_FUNC_START(aesni_cbc_enc)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_cbc_enc)
+SYM_FUNC_END_SECTION(aesni_cbc_enc)
/*
* void aesni_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
* size_t len, u8 *iv)
*/
-SYM_FUNC_START(aesni_cbc_dec)
+SYM_FUNC_START_SECTION(aesni_cbc_dec)
FRAME_BEGIN
#ifndef __x86_64__
pushl IVP
@@ -2571,13 +2568,13 @@ SYM_FUNC_START(aesni_cbc_dec)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_cbc_dec)
+SYM_FUNC_END_SECTION(aesni_cbc_dec)
/*
* void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
* size_t len, u8 *iv)
*/
-SYM_FUNC_START(aesni_cts_cbc_enc)
+SYM_FUNC_START_SECTION(aesni_cts_cbc_enc)
FRAME_BEGIN
#ifndef __x86_64__
pushl IVP
@@ -2628,13 +2625,13 @@ SYM_FUNC_START(aesni_cts_cbc_enc)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_cts_cbc_enc)
+SYM_FUNC_END_SECTION(aesni_cts_cbc_enc)
/*
* void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
* size_t len, u8 *iv)
*/
-SYM_FUNC_START(aesni_cts_cbc_dec)
+SYM_FUNC_START_SECTION(aesni_cts_cbc_dec)
FRAME_BEGIN
#ifndef __x86_64__
pushl IVP
@@ -2689,7 +2686,7 @@ SYM_FUNC_START(aesni_cts_cbc_dec)
#endif
FRAME_END
ret
-SYM_FUNC_END(aesni_cts_cbc_dec)
+SYM_FUNC_END_SECTION(aesni_cts_cbc_dec)
.pushsection .rodata
.align 16
@@ -2718,7 +2715,7 @@ SYM_FUNC_END(aesni_cts_cbc_dec)
* INC: == 1, in little endian
* BSWAP_MASK == endian swapping mask
*/
-SYM_FUNC_START_LOCAL(_aesni_inc_init)
+SYM_FUNC_START_LOCAL_SECTION(_aesni_inc_init)
movaps .Lbswap_mask, BSWAP_MASK
movaps IV, CTR
pshufb BSWAP_MASK, CTR
@@ -2726,7 +2723,7 @@ SYM_FUNC_START_LOCAL(_aesni_inc_init)
movq TCTR_LOW, INC
movq CTR, TCTR_LOW
ret
-SYM_FUNC_END(_aesni_inc_init)
+SYM_FUNC_END_SECTION(_aesni_inc_init)
/*
* _aesni_inc: internal ABI
@@ -2743,7 +2740,7 @@ SYM_FUNC_END(_aesni_inc_init)
* CTR: == output IV, in little endian
* TCTR_LOW: == lower qword of CTR
*/
-SYM_FUNC_START_LOCAL(_aesni_inc)
+SYM_FUNC_START_LOCAL_SECTION(_aesni_inc)
paddq INC, CTR
add $1, TCTR_LOW
jnc .Linc_low
@@ -2754,13 +2751,13 @@ SYM_FUNC_START_LOCAL(_aesni_inc)
movaps CTR, IV
pshufb BSWAP_MASK, IV
ret
-SYM_FUNC_END(_aesni_inc)
+SYM_FUNC_END_SECTION(_aesni_inc)
/*
* void aesni_ctr_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
* size_t len, u8 *iv)
*/
-SYM_FUNC_START(aesni_ctr_enc)
+SYM_FUNC_START_SECTION(aesni_ctr_enc)
FRAME_BEGIN
cmp $16, LEN
jb .Lctr_enc_just_ret
@@ -2817,8 +2814,7 @@ SYM_FUNC_START(aesni_ctr_enc)
.Lctr_enc_just_ret:
FRAME_END
ret
-SYM_FUNC_END(aesni_ctr_enc)
-
+SYM_FUNC_END_SECTION(aesni_ctr_enc)
#endif
.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16
@@ -2849,7 +2845,7 @@ SYM_FUNC_END(aesni_ctr_enc)
* void aesni_xts_encrypt(const struct crypto_aes_ctx *ctx, u8 *dst,
* const u8 *src, unsigned int len, le128 *iv)
*/
-SYM_FUNC_START(aesni_xts_encrypt)
+SYM_FUNC_START_SECTION(aesni_xts_encrypt)
FRAME_BEGIN
#ifndef __x86_64__
pushl IVP
@@ -2998,13 +2994,13 @@ SYM_FUNC_START(aesni_xts_encrypt)
movups STATE, (OUTP)
jmp .Lxts_enc_ret
-SYM_FUNC_END(aesni_xts_encrypt)
+SYM_FUNC_END_SECTION(aesni_xts_encrypt)
/*
* void aesni_xts_decrypt(const struct crypto_aes_ctx *ctx, u8 *dst,
* const u8 *src, unsigned int len, le128 *iv)
*/
-SYM_FUNC_START(aesni_xts_decrypt)
+SYM_FUNC_START_SECTION(aesni_xts_decrypt)
FRAME_BEGIN
#ifndef __x86_64__
pushl IVP
@@ -3160,4 +3156,4 @@ SYM_FUNC_START(aesni_xts_decrypt)
movups STATE, (OUTP)
jmp .Lxts_dec_ret
-SYM_FUNC_END(aesni_xts_decrypt)
+SYM_FUNC_END_SECTION(aesni_xts_decrypt)
diff --git a/arch/x86/crypto/aesni-intel_avx-x86_64.S b/arch/x86/crypto/aesni-intel_avx-x86_64.S
index 98e3552b6e03..1a116980d273 100644
--- a/arch/x86/crypto/aesni-intel_avx-x86_64.S
+++ b/arch/x86/crypto/aesni-intel_avx-x86_64.S
@@ -1763,12 +1763,12 @@ _initial_blocks_done\@:
# const u8 *aad, /* Additional Authentication Data (AAD)*/
# u64 aad_len) /* Length of AAD in bytes. With RFC4106 this is going to be 8 or 12 Bytes */
#############################################################
-SYM_FUNC_START(aesni_gcm_init_avx_gen2)
+SYM_FUNC_START_SECTION(aesni_gcm_init_avx_gen2)
FUNC_SAVE
INIT GHASH_MUL_AVX, PRECOMPUTE_AVX
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_init_avx_gen2)
+SYM_FUNC_END_SECTION(aesni_gcm_init_avx_gen2)
###############################################################################
#void aesni_gcm_enc_update_avx_gen2(
@@ -1778,7 +1778,7 @@ SYM_FUNC_END(aesni_gcm_init_avx_gen2)
# const u8 *in, /* Plaintext input */
# u64 plaintext_len) /* Length of data in Bytes for encryption. */
###############################################################################
-SYM_FUNC_START(aesni_gcm_enc_update_avx_gen2)
+SYM_FUNC_START_SECTION(aesni_gcm_enc_update_avx_gen2)
FUNC_SAVE
mov keysize, %eax
cmp $32, %eax
@@ -1797,7 +1797,7 @@ key_256_enc_update:
GCM_ENC_DEC INITIAL_BLOCKS_AVX, GHASH_8_ENCRYPT_8_PARALLEL_AVX, GHASH_LAST_8_AVX, GHASH_MUL_AVX, ENC, 13
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_enc_update_avx_gen2)
+SYM_FUNC_END_SECTION(aesni_gcm_enc_update_avx_gen2)
###############################################################################
#void aesni_gcm_dec_update_avx_gen2(
@@ -1807,7 +1807,7 @@ SYM_FUNC_END(aesni_gcm_enc_update_avx_gen2)
# const u8 *in, /* Ciphertext input */
# u64 plaintext_len) /* Length of data in Bytes for encryption. */
###############################################################################
-SYM_FUNC_START(aesni_gcm_dec_update_avx_gen2)
+SYM_FUNC_START_SECTION(aesni_gcm_dec_update_avx_gen2)
FUNC_SAVE
mov keysize,%eax
cmp $32, %eax
@@ -1826,7 +1826,7 @@ key_256_dec_update:
GCM_ENC_DEC INITIAL_BLOCKS_AVX, GHASH_8_ENCRYPT_8_PARALLEL_AVX, GHASH_LAST_8_AVX, GHASH_MUL_AVX, DEC, 13
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_dec_update_avx_gen2)
+SYM_FUNC_END_SECTION(aesni_gcm_dec_update_avx_gen2)
###############################################################################
#void aesni_gcm_finalize_avx_gen2(
@@ -1836,7 +1836,7 @@ SYM_FUNC_END(aesni_gcm_dec_update_avx_gen2)
# u64 auth_tag_len)# /* Authenticated Tag Length in bytes.
# Valid values are 16 (most likely), 12 or 8. */
###############################################################################
-SYM_FUNC_START(aesni_gcm_finalize_avx_gen2)
+SYM_FUNC_START_SECTION(aesni_gcm_finalize_avx_gen2)
FUNC_SAVE
mov keysize,%eax
cmp $32, %eax
@@ -1855,7 +1855,7 @@ key_256_finalize:
GCM_COMPLETE GHASH_MUL_AVX, 13, arg3, arg4
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_finalize_avx_gen2)
+SYM_FUNC_END_SECTION(aesni_gcm_finalize_avx_gen2)
###############################################################################
# GHASH_MUL MACRO to implement: Data*HashKey mod (128,127,126,121,0)
@@ -2731,12 +2731,12 @@ _initial_blocks_done\@:
# const u8 *aad, /* Additional Authentication Data (AAD)*/
# u64 aad_len) /* Length of AAD in bytes. With RFC4106 this is going to be 8 or 12 Bytes */
#############################################################
-SYM_FUNC_START(aesni_gcm_init_avx_gen4)
+SYM_FUNC_START_SECTION(aesni_gcm_init_avx_gen4)
FUNC_SAVE
INIT GHASH_MUL_AVX2, PRECOMPUTE_AVX2
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_init_avx_gen4)
+SYM_FUNC_END_SECTION(aesni_gcm_init_avx_gen4)
###############################################################################
#void aesni_gcm_enc_avx_gen4(
@@ -2746,7 +2746,7 @@ SYM_FUNC_END(aesni_gcm_init_avx_gen4)
# const u8 *in, /* Plaintext input */
# u64 plaintext_len) /* Length of data in Bytes for encryption. */
###############################################################################
-SYM_FUNC_START(aesni_gcm_enc_update_avx_gen4)
+SYM_FUNC_START_SECTION(aesni_gcm_enc_update_avx_gen4)
FUNC_SAVE
mov keysize,%eax
cmp $32, %eax
@@ -2765,7 +2765,7 @@ key_256_enc_update4:
GCM_ENC_DEC INITIAL_BLOCKS_AVX2, GHASH_8_ENCRYPT_8_PARALLEL_AVX2, GHASH_LAST_8_AVX2, GHASH_MUL_AVX2, ENC, 13
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_enc_update_avx_gen4)
+SYM_FUNC_END_SECTION(aesni_gcm_enc_update_avx_gen4)
###############################################################################
#void aesni_gcm_dec_update_avx_gen4(
@@ -2775,7 +2775,7 @@ SYM_FUNC_END(aesni_gcm_enc_update_avx_gen4)
# const u8 *in, /* Ciphertext input */
# u64 plaintext_len) /* Length of data in Bytes for encryption. */
###############################################################################
-SYM_FUNC_START(aesni_gcm_dec_update_avx_gen4)
+SYM_FUNC_START_SECTION(aesni_gcm_dec_update_avx_gen4)
FUNC_SAVE
mov keysize,%eax
cmp $32, %eax
@@ -2794,7 +2794,7 @@ key_256_dec_update4:
GCM_ENC_DEC INITIAL_BLOCKS_AVX2, GHASH_8_ENCRYPT_8_PARALLEL_AVX2, GHASH_LAST_8_AVX2, GHASH_MUL_AVX2, DEC, 13
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_dec_update_avx_gen4)
+SYM_FUNC_END_SECTION(aesni_gcm_dec_update_avx_gen4)
###############################################################################
#void aesni_gcm_finalize_avx_gen4(
@@ -2804,7 +2804,7 @@ SYM_FUNC_END(aesni_gcm_dec_update_avx_gen4)
# u64 auth_tag_len)# /* Authenticated Tag Length in bytes.
# Valid values are 16 (most likely), 12 or 8. */
###############################################################################
-SYM_FUNC_START(aesni_gcm_finalize_avx_gen4)
+SYM_FUNC_START_SECTION(aesni_gcm_finalize_avx_gen4)
FUNC_SAVE
mov keysize,%eax
cmp $32, %eax
@@ -2823,4 +2823,4 @@ key_256_finalize4:
GCM_COMPLETE GHASH_MUL_AVX2, 13, arg3, arg4
FUNC_RESTORE
ret
-SYM_FUNC_END(aesni_gcm_finalize_avx_gen4)
+SYM_FUNC_END_SECTION(aesni_gcm_finalize_avx_gen4)
diff --git a/arch/x86/crypto/blake2s-core.S b/arch/x86/crypto/blake2s-core.S
index 2ca79974f819..557169b98d3d 100644
--- a/arch/x86/crypto/blake2s-core.S
+++ b/arch/x86/crypto/blake2s-core.S
@@ -46,7 +46,7 @@ SIGMA2:
#endif /* CONFIG_AS_AVX512 */
.text
-SYM_FUNC_START(blake2s_compress_ssse3)
+SYM_FUNC_START_SECTION(blake2s_compress_ssse3)
testq %rdx,%rdx
je .Lendofloop
movdqu (%rdi),%xmm0
@@ -172,10 +172,10 @@ SYM_FUNC_START(blake2s_compress_ssse3)
movdqu %xmm14,0x20(%rdi)
.Lendofloop:
ret
-SYM_FUNC_END(blake2s_compress_ssse3)
+SYM_FUNC_END_SECTION(blake2s_compress_ssse3)
#ifdef CONFIG_AS_AVX512
-SYM_FUNC_START(blake2s_compress_avx512)
+SYM_FUNC_START_SECTION(blake2s_compress_avx512)
vmovdqu (%rdi),%xmm0
vmovdqu 0x10(%rdi),%xmm1
vmovdqu 0x20(%rdi),%xmm4
@@ -252,5 +252,5 @@ SYM_FUNC_START(blake2s_compress_avx512)
vmovdqu %xmm4,0x20(%rdi)
vzeroupper
retq
-SYM_FUNC_END(blake2s_compress_avx512)
+SYM_FUNC_END_SECTION(blake2s_compress_avx512)
#endif /* CONFIG_AS_AVX512 */
diff --git a/arch/x86/crypto/blowfish-x86_64-asm_64.S b/arch/x86/crypto/blowfish-x86_64-asm_64.S
index 4222ac6d6584..d95e7bbd4d6c 100644
--- a/arch/x86/crypto/blowfish-x86_64-asm_64.S
+++ b/arch/x86/crypto/blowfish-x86_64-asm_64.S
@@ -103,7 +103,7 @@
bswapq RX0; \
xorq RX0, (RIO);
-SYM_FUNC_START(__blowfish_enc_blk)
+SYM_FUNC_START_SECTION(__blowfish_enc_blk)
/* input:
* %rdi: ctx
* %rsi: dst
@@ -139,9 +139,9 @@ SYM_FUNC_START(__blowfish_enc_blk)
.L__enc_xor:
xor_block();
ret;
-SYM_FUNC_END(__blowfish_enc_blk)
+SYM_FUNC_END_SECTION(__blowfish_enc_blk)
-SYM_FUNC_START(blowfish_dec_blk)
+SYM_FUNC_START_SECTION(blowfish_dec_blk)
/* input:
* %rdi: ctx
* %rsi: dst
@@ -171,7 +171,7 @@ SYM_FUNC_START(blowfish_dec_blk)
movq %r11, %r12;
ret;
-SYM_FUNC_END(blowfish_dec_blk)
+SYM_FUNC_END_SECTION(blowfish_dec_blk)
/**********************************************************************
4-way blowfish, four blocks parallel
@@ -283,7 +283,7 @@ SYM_FUNC_END(blowfish_dec_blk)
bswapq RX3; \
xorq RX3, 24(RIO);
-SYM_FUNC_START(__blowfish_enc_blk_4way)
+SYM_FUNC_START_SECTION(__blowfish_enc_blk_4way)
/* input:
* %rdi: ctx
* %rsi: dst
@@ -330,9 +330,9 @@ SYM_FUNC_START(__blowfish_enc_blk_4way)
popq %rbx;
popq %r12;
ret;
-SYM_FUNC_END(__blowfish_enc_blk_4way)
+SYM_FUNC_END_SECTION(__blowfish_enc_blk_4way)
-SYM_FUNC_START(blowfish_dec_blk_4way)
+SYM_FUNC_START_SECTION(blowfish_dec_blk_4way)
/* input:
* %rdi: ctx
* %rsi: dst
@@ -365,4 +365,4 @@ SYM_FUNC_START(blowfish_dec_blk_4way)
popq %r12;
ret;
-SYM_FUNC_END(blowfish_dec_blk_4way)
+SYM_FUNC_END_SECTION(blowfish_dec_blk_4way)
diff --git a/arch/x86/crypto/camellia-aesni-avx-asm_64.S b/arch/x86/crypto/camellia-aesni-avx-asm_64.S
index e2a0e0f4bf9d..33f0ae08bf88 100644
--- a/arch/x86/crypto/camellia-aesni-avx-asm_64.S
+++ b/arch/x86/crypto/camellia-aesni-avx-asm_64.S
@@ -187,21 +187,21 @@
* Size optimization... with inlined roundsm16, binary would be over 5 times
* larger and would only be 0.5% faster (on sandy-bridge).
*/
+SYM_FUNC_START_LOCAL_SECTION(roundsm16_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd)
.align 8
-SYM_FUNC_START_LOCAL(roundsm16_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd)
roundsm16(%xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7,
%xmm8, %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm15,
%rcx, (%r9));
ret;
-SYM_FUNC_END(roundsm16_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd)
+SYM_FUNC_END_SECTION(roundsm16_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd)
+SYM_FUNC_START_LOCAL_SECTION(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab)
.align 8
-SYM_FUNC_START_LOCAL(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab)
roundsm16(%xmm4, %xmm5, %xmm6, %xmm7, %xmm0, %xmm1, %xmm2, %xmm3,
%xmm12, %xmm13, %xmm14, %xmm15, %xmm8, %xmm9, %xmm10, %xmm11,
%rax, (%r9));
ret;
-SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab)
+SYM_FUNC_END_SECTION(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab)
/*
* IN/OUT:
@@ -712,8 +712,8 @@ SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab)
.text
+SYM_FUNC_START_LOCAL_SECTION(__camellia_enc_blk16)
.align 8
-SYM_FUNC_START_LOCAL(__camellia_enc_blk16)
/* input:
* %rdi: ctx, CTX
* %rax: temporary storage, 256 bytes
@@ -797,10 +797,10 @@ SYM_FUNC_START_LOCAL(__camellia_enc_blk16)
%xmm15, %rax, %rcx, 24);
jmp .Lenc_done;
-SYM_FUNC_END(__camellia_enc_blk16)
+SYM_FUNC_END_SECTION(__camellia_enc_blk16)
+SYM_FUNC_START_LOCAL_SECTION(__camellia_dec_blk16)
.align 8
-SYM_FUNC_START_LOCAL(__camellia_dec_blk16)
/* input:
* %rdi: ctx, CTX
* %rax: temporary storage, 256 bytes
@@ -882,9 +882,9 @@ SYM_FUNC_START_LOCAL(__camellia_dec_blk16)
((key_table + (24) * 8) + 4)(CTX));
jmp .Ldec_max24;
-SYM_FUNC_END(__camellia_dec_blk16)
+SYM_FUNC_END_SECTION(__camellia_dec_blk16)
-SYM_FUNC_START(camellia_ecb_enc_16way)
+SYM_FUNC_START_SECTION(camellia_ecb_enc_16way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst (16 blocks)
@@ -907,9 +907,9 @@ SYM_FUNC_START(camellia_ecb_enc_16way)
FRAME_END
ret;
-SYM_FUNC_END(camellia_ecb_enc_16way)
+SYM_FUNC_END_SECTION(camellia_ecb_enc_16way)
-SYM_FUNC_START(camellia_ecb_dec_16way)
+SYM_FUNC_START_SECTION(camellia_ecb_dec_16way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst (16 blocks)
@@ -937,9 +937,9 @@ SYM_FUNC_START(camellia_ecb_dec_16way)
FRAME_END
ret;
-SYM_FUNC_END(camellia_ecb_dec_16way)
+SYM_FUNC_END_SECTION(camellia_ecb_dec_16way)
-SYM_FUNC_START(camellia_cbc_dec_16way)
+SYM_FUNC_START_SECTION(camellia_cbc_dec_16way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst (16 blocks)
@@ -988,4 +988,4 @@ SYM_FUNC_START(camellia_cbc_dec_16way)
FRAME_END
ret;
-SYM_FUNC_END(camellia_cbc_dec_16way)
+SYM_FUNC_END_SECTION(camellia_cbc_dec_16way)
diff --git a/arch/x86/crypto/camellia-aesni-avx2-asm_64.S b/arch/x86/crypto/camellia-aesni-avx2-asm_64.S
index 706f70829a07..d2cd829cd1c1 100644
--- a/arch/x86/crypto/camellia-aesni-avx2-asm_64.S
+++ b/arch/x86/crypto/camellia-aesni-avx2-asm_64.S
@@ -221,21 +221,21 @@
* Size optimization... with inlined roundsm32 binary would be over 5 times
* larger and would only marginally faster.
*/
+SYM_FUNC_START_LOCAL_SECTION(roundsm32_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd)
.align 8
-SYM_FUNC_START_LOCAL(roundsm32_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd)
roundsm32(%ymm0, %ymm1, %ymm2, %ymm3, %ymm4, %ymm5, %ymm6, %ymm7,
%ymm8, %ymm9, %ymm10, %ymm11, %ymm12, %ymm13, %ymm14, %ymm15,
%rcx, (%r9));
ret;
-SYM_FUNC_END(roundsm32_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd)
+SYM_FUNC_END_SECTION(roundsm32_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd)
+SYM_FUNC_START_LOCAL_SECTION(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab)
.align 8
-SYM_FUNC_START_LOCAL(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab)
roundsm32(%ymm4, %ymm5, %ymm6, %ymm7, %ymm0, %ymm1, %ymm2, %ymm3,
%ymm12, %ymm13, %ymm14, %ymm15, %ymm8, %ymm9, %ymm10, %ymm11,
%rax, (%r9));
ret;
-SYM_FUNC_END(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab)
+SYM_FUNC_END_SECTION(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab)
/*
* IN/OUT:
@@ -748,8 +748,8 @@ SYM_FUNC_END(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab)
.text
+SYM_FUNC_START_LOCAL_SECTION(__camellia_enc_blk32)
.align 8
-SYM_FUNC_START_LOCAL(__camellia_enc_blk32)
/* input:
* %rdi: ctx, CTX
* %rax: temporary storage, 512 bytes
@@ -833,10 +833,10 @@ SYM_FUNC_START_LOCAL(__camellia_enc_blk32)
%ymm15, %rax, %rcx, 24);
jmp .Lenc_done;
-SYM_FUNC_END(__camellia_enc_blk32)
+SYM_FUNC_END_SECTION(__camellia_enc_blk32)
+SYM_FUNC_START_LOCAL_SECTION(__camellia_dec_blk32)
.align 8
-SYM_FUNC_START_LOCAL(__camellia_dec_blk32)
/* input:
* %rdi: ctx, CTX
* %rax: temporary storage, 512 bytes
@@ -918,9 +918,9 @@ SYM_FUNC_START_LOCAL(__camellia_dec_blk32)
((key_table + (24) * 8) + 4)(CTX));
jmp .Ldec_max24;
-SYM_FUNC_END(__camellia_dec_blk32)
+SYM_FUNC_END_SECTION(__camellia_dec_blk32)
-SYM_FUNC_START(camellia_ecb_enc_32way)
+SYM_FUNC_START_SECTION(camellia_ecb_enc_32way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst (32 blocks)
@@ -947,9 +947,9 @@ SYM_FUNC_START(camellia_ecb_enc_32way)
FRAME_END
ret;
-SYM_FUNC_END(camellia_ecb_enc_32way)
+SYM_FUNC_END_SECTION(camellia_ecb_enc_32way)
-SYM_FUNC_START(camellia_ecb_dec_32way)
+SYM_FUNC_START_SECTION(camellia_ecb_dec_32way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst (32 blocks)
@@ -981,9 +981,9 @@ SYM_FUNC_START(camellia_ecb_dec_32way)
FRAME_END
ret;
-SYM_FUNC_END(camellia_ecb_dec_32way)
+SYM_FUNC_END_SECTION(camellia_ecb_dec_32way)
-SYM_FUNC_START(camellia_cbc_dec_32way)
+SYM_FUNC_START_SECTION(camellia_cbc_dec_32way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst (32 blocks)
@@ -1048,4 +1048,4 @@ SYM_FUNC_START(camellia_cbc_dec_32way)
addq $(16 * 32), %rsp;
FRAME_END
ret;
-SYM_FUNC_END(camellia_cbc_dec_32way)
+SYM_FUNC_END_SECTION(camellia_cbc_dec_32way)
diff --git a/arch/x86/crypto/camellia-x86_64-asm_64.S b/arch/x86/crypto/camellia-x86_64-asm_64.S
index 1372e6408850..8b4dd32b752d 100644
--- a/arch/x86/crypto/camellia-x86_64-asm_64.S
+++ b/arch/x86/crypto/camellia-x86_64-asm_64.S
@@ -175,7 +175,7 @@
bswapq RAB0; \
movq RAB0, 4*2(RIO);
-SYM_FUNC_START(__camellia_enc_blk)
+SYM_FUNC_START_SECTION(__camellia_enc_blk)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -220,9 +220,9 @@ SYM_FUNC_START(__camellia_enc_blk)
movq RR12, %r12;
ret;
-SYM_FUNC_END(__camellia_enc_blk)
+SYM_FUNC_END_SECTION(__camellia_enc_blk)
-SYM_FUNC_START(camellia_dec_blk)
+SYM_FUNC_START_SECTION(camellia_dec_blk)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -258,7 +258,7 @@ SYM_FUNC_START(camellia_dec_blk)
movq RR12, %r12;
ret;
-SYM_FUNC_END(camellia_dec_blk)
+SYM_FUNC_END_SECTION(camellia_dec_blk)
/**********************************************************************
2-way camellia
@@ -409,7 +409,7 @@ SYM_FUNC_END(camellia_dec_blk)
bswapq RAB1; \
movq RAB1, 12*2(RIO);
-SYM_FUNC_START(__camellia_enc_blk_2way)
+SYM_FUNC_START_SECTION(__camellia_enc_blk_2way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -456,9 +456,9 @@ SYM_FUNC_START(__camellia_enc_blk_2way)
movq RR12, %r12;
popq %rbx;
ret;
-SYM_FUNC_END(__camellia_enc_blk_2way)
+SYM_FUNC_END_SECTION(__camellia_enc_blk_2way)
-SYM_FUNC_START(camellia_dec_blk_2way)
+SYM_FUNC_START_SECTION(camellia_dec_blk_2way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -496,4 +496,4 @@ SYM_FUNC_START(camellia_dec_blk_2way)
movq RR12, %r12;
movq RXOR, %rbx;
ret;
-SYM_FUNC_END(camellia_dec_blk_2way)
+SYM_FUNC_END_SECTION(camellia_dec_blk_2way)
diff --git a/arch/x86/crypto/cast5-avx-x86_64-asm_64.S b/arch/x86/crypto/cast5-avx-x86_64-asm_64.S
index 8a6181b08b59..971bd5e2f13d 100644
--- a/arch/x86/crypto/cast5-avx-x86_64-asm_64.S
+++ b/arch/x86/crypto/cast5-avx-x86_64-asm_64.S
@@ -208,8 +208,8 @@
.text
+SYM_FUNC_START_LOCAL_SECTION(__cast5_enc_blk16)
.align 16
-SYM_FUNC_START_LOCAL(__cast5_enc_blk16)
/* input:
* %rdi: ctx
* RL1: blocks 1 and 2
@@ -280,10 +280,10 @@ SYM_FUNC_START_LOCAL(__cast5_enc_blk16)
outunpack_blocks(RR4, RL4, RTMP, RX, RKM);
ret;
-SYM_FUNC_END(__cast5_enc_blk16)
+SYM_FUNC_END_SECTION(__cast5_enc_blk16)
+SYM_FUNC_START_LOCAL_SECTION(__cast5_dec_blk16)
.align 16
-SYM_FUNC_START_LOCAL(__cast5_dec_blk16)
/* input:
* %rdi: ctx
* RL1: encrypted blocks 1 and 2
@@ -357,9 +357,9 @@ SYM_FUNC_START_LOCAL(__cast5_dec_blk16)
.L__skip_dec:
vpsrldq $4, RKR, RKR;
jmp .L__dec_tail;
-SYM_FUNC_END(__cast5_dec_blk16)
+SYM_FUNC_END_SECTION(__cast5_dec_blk16)
-SYM_FUNC_START(cast5_ecb_enc_16way)
+SYM_FUNC_START_SECTION(cast5_ecb_enc_16way)
/* input:
* %rdi: ctx
* %rsi: dst
@@ -394,9 +394,9 @@ SYM_FUNC_START(cast5_ecb_enc_16way)
popq %r15;
FRAME_END
ret;
-SYM_FUNC_END(cast5_ecb_enc_16way)
+SYM_FUNC_END_SECTION(cast5_ecb_enc_16way)
-SYM_FUNC_START(cast5_ecb_dec_16way)
+SYM_FUNC_START_SECTION(cast5_ecb_dec_16way)
/* input:
* %rdi: ctx
* %rsi: dst
@@ -432,9 +432,9 @@ SYM_FUNC_START(cast5_ecb_dec_16way)
popq %r15;
FRAME_END
ret;
-SYM_FUNC_END(cast5_ecb_dec_16way)
+SYM_FUNC_END_SECTION(cast5_ecb_dec_16way)
-SYM_FUNC_START(cast5_cbc_dec_16way)
+SYM_FUNC_START_SECTION(cast5_cbc_dec_16way)
/* input:
* %rdi: ctx
* %rsi: dst
@@ -484,9 +484,9 @@ SYM_FUNC_START(cast5_cbc_dec_16way)
popq %r12;
FRAME_END
ret;
-SYM_FUNC_END(cast5_cbc_dec_16way)
+SYM_FUNC_END_SECTION(cast5_cbc_dec_16way)
-SYM_FUNC_START(cast5_ctr_16way)
+SYM_FUNC_START_SECTION(cast5_ctr_16way)
/* input:
* %rdi: ctx
* %rsi: dst
@@ -560,4 +560,4 @@ SYM_FUNC_START(cast5_ctr_16way)
popq %r12;
FRAME_END
ret;
-SYM_FUNC_END(cast5_ctr_16way)
+SYM_FUNC_END_SECTION(cast5_ctr_16way)
diff --git a/arch/x86/crypto/cast6-avx-x86_64-asm_64.S b/arch/x86/crypto/cast6-avx-x86_64-asm_64.S
index fbddcecc3e3f..0da62a1c5159 100644
--- a/arch/x86/crypto/cast6-avx-x86_64-asm_64.S
+++ b/arch/x86/crypto/cast6-avx-x86_64-asm_64.S
@@ -244,8 +244,8 @@
.text
+SYM_FUNC_START_LOCAL_SECTION(__cast6_enc_blk8)
.align 8
-SYM_FUNC_START_LOCAL(__cast6_enc_blk8)
/* input:
* %rdi: ctx
* RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: blocks
@@ -290,10 +290,10 @@ SYM_FUNC_START_LOCAL(__cast6_enc_blk8)
outunpack_blocks(RA2, RB2, RC2, RD2, RTMP, RX, RKRF, RKM);
ret;
-SYM_FUNC_END(__cast6_enc_blk8)
+SYM_FUNC_END_SECTION(__cast6_enc_blk8)
+SYM_FUNC_START_LOCAL_SECTION(__cast6_dec_blk8)
.align 8
-SYM_FUNC_START_LOCAL(__cast6_dec_blk8)
/* input:
* %rdi: ctx
* RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: encrypted blocks
@@ -337,9 +337,9 @@ SYM_FUNC_START_LOCAL(__cast6_dec_blk8)
outunpack_blocks(RA2, RB2, RC2, RD2, RTMP, RX, RKRF, RKM);
ret;
-SYM_FUNC_END(__cast6_dec_blk8)
+SYM_FUNC_END_SECTION(__cast6_dec_blk8)
-SYM_FUNC_START(cast6_ecb_enc_8way)
+SYM_FUNC_START_SECTION(cast6_ecb_enc_8way)
/* input:
* %rdi: ctx
* %rsi: dst
@@ -360,9 +360,9 @@ SYM_FUNC_START(cast6_ecb_enc_8way)
popq %r15;
FRAME_END
ret;
-SYM_FUNC_END(cast6_ecb_enc_8way)
+SYM_FUNC_END_SECTION(cast6_ecb_enc_8way)
-SYM_FUNC_START(cast6_ecb_dec_8way)
+SYM_FUNC_START_SECTION(cast6_ecb_dec_8way)
/* input:
* %rdi: ctx
* %rsi: dst
@@ -383,9 +383,9 @@ SYM_FUNC_START(cast6_ecb_dec_8way)
popq %r15;
FRAME_END
ret;
-SYM_FUNC_END(cast6_ecb_dec_8way)
+SYM_FUNC_END_SECTION(cast6_ecb_dec_8way)
-SYM_FUNC_START(cast6_cbc_dec_8way)
+SYM_FUNC_START_SECTION(cast6_cbc_dec_8way)
/* input:
* %rdi: ctx
* %rsi: dst
@@ -409,4 +409,4 @@ SYM_FUNC_START(cast6_cbc_dec_8way)
popq %r12;
FRAME_END
ret;
-SYM_FUNC_END(cast6_cbc_dec_8way)
+SYM_FUNC_END_SECTION(cast6_cbc_dec_8way)
diff --git a/arch/x86/crypto/chacha-avx2-x86_64.S b/arch/x86/crypto/chacha-avx2-x86_64.S
index ee9a40ab4109..1e66a2d462c4 100644
--- a/arch/x86/crypto/chacha-avx2-x86_64.S
+++ b/arch/x86/crypto/chacha-avx2-x86_64.S
@@ -34,7 +34,7 @@ CTR4BL: .octa 0x00000000000000000000000000000002
.text
-SYM_FUNC_START(chacha_2block_xor_avx2)
+SYM_FUNC_START_SECTION(chacha_2block_xor_avx2)
# %rdi: Input state matrix, s
# %rsi: up to 2 data blocks output, o
# %rdx: up to 2 data blocks input, i
@@ -224,9 +224,9 @@ SYM_FUNC_START(chacha_2block_xor_avx2)
lea -8(%r10),%rsp
jmp .Ldone2
-SYM_FUNC_END(chacha_2block_xor_avx2)
+SYM_FUNC_END_SECTION(chacha_2block_xor_avx2)
-SYM_FUNC_START(chacha_4block_xor_avx2)
+SYM_FUNC_START_SECTION(chacha_4block_xor_avx2)
# %rdi: Input state matrix, s
# %rsi: up to 4 data blocks output, o
# %rdx: up to 4 data blocks input, i
@@ -529,9 +529,9 @@ SYM_FUNC_START(chacha_4block_xor_avx2)
lea -8(%r10),%rsp
jmp .Ldone4
-SYM_FUNC_END(chacha_4block_xor_avx2)
+SYM_FUNC_END_SECTION(chacha_4block_xor_avx2)
-SYM_FUNC_START(chacha_8block_xor_avx2)
+SYM_FUNC_START_SECTION(chacha_8block_xor_avx2)
# %rdi: Input state matrix, s
# %rsi: up to 8 data blocks output, o
# %rdx: up to 8 data blocks input, i
@@ -1018,4 +1018,4 @@ SYM_FUNC_START(chacha_8block_xor_avx2)
jmp .Ldone8
-SYM_FUNC_END(chacha_8block_xor_avx2)
+SYM_FUNC_END_SECTION(chacha_8block_xor_avx2)
diff --git a/arch/x86/crypto/chacha-avx512vl-x86_64.S b/arch/x86/crypto/chacha-avx512vl-x86_64.S
index bb193fde123a..da5ab2897d80 100644
--- a/arch/x86/crypto/chacha-avx512vl-x86_64.S
+++ b/arch/x86/crypto/chacha-avx512vl-x86_64.S
@@ -24,7 +24,7 @@ CTR8BL: .octa 0x00000003000000020000000100000000
.text
-SYM_FUNC_START(chacha_2block_xor_avx512vl)
+SYM_FUNC_START_SECTION(chacha_2block_xor_avx512vl)
# %rdi: Input state matrix, s
# %rsi: up to 2 data blocks output, o
# %rdx: up to 2 data blocks input, i
@@ -187,9 +187,9 @@ SYM_FUNC_START(chacha_2block_xor_avx512vl)
jmp .Ldone2
-SYM_FUNC_END(chacha_2block_xor_avx512vl)
+SYM_FUNC_END_SECTION(chacha_2block_xor_avx512vl)
-SYM_FUNC_START(chacha_4block_xor_avx512vl)
+SYM_FUNC_START_SECTION(chacha_4block_xor_avx512vl)
# %rdi: Input state matrix, s
# %rsi: up to 4 data blocks output, o
# %rdx: up to 4 data blocks input, i
@@ -453,9 +453,9 @@ SYM_FUNC_START(chacha_4block_xor_avx512vl)
jmp .Ldone4
-SYM_FUNC_END(chacha_4block_xor_avx512vl)
+SYM_FUNC_END_SECTION(chacha_4block_xor_avx512vl)
-SYM_FUNC_START(chacha_8block_xor_avx512vl)
+SYM_FUNC_START_SECTION(chacha_8block_xor_avx512vl)
# %rdi: Input state matrix, s
# %rsi: up to 8 data blocks output, o
# %rdx: up to 8 data blocks input, i
@@ -833,4 +833,4 @@ SYM_FUNC_START(chacha_8block_xor_avx512vl)
jmp .Ldone8
-SYM_FUNC_END(chacha_8block_xor_avx512vl)
+SYM_FUNC_END_SECTION(chacha_8block_xor_avx512vl)
diff --git a/arch/x86/crypto/chacha-ssse3-x86_64.S b/arch/x86/crypto/chacha-ssse3-x86_64.S
index ca1788bfee16..4c66f51dfd61 100644
--- a/arch/x86/crypto/chacha-ssse3-x86_64.S
+++ b/arch/x86/crypto/chacha-ssse3-x86_64.S
@@ -33,7 +33,7 @@ CTRINC: .octa 0x00000003000000020000000100000000
*
* Clobbers: %r8d, %xmm4-%xmm7
*/
-SYM_FUNC_START_LOCAL(chacha_permute)
+SYM_FUNC_START_LOCAL_SECTION(chacha_permute)
movdqa ROT8(%rip),%xmm4
movdqa ROT16(%rip),%xmm5
@@ -109,9 +109,9 @@ SYM_FUNC_START_LOCAL(chacha_permute)
jnz .Ldoubleround
ret
-SYM_FUNC_END(chacha_permute)
+SYM_FUNC_END_SECTION(chacha_permute)
-SYM_FUNC_START(chacha_block_xor_ssse3)
+SYM_FUNC_START_SECTION(chacha_block_xor_ssse3)
# %rdi: Input state matrix, s
# %rsi: up to 1 data block output, o
# %rdx: up to 1 data block input, i
@@ -197,9 +197,9 @@ SYM_FUNC_START(chacha_block_xor_ssse3)
lea -8(%r10),%rsp
jmp .Ldone
-SYM_FUNC_END(chacha_block_xor_ssse3)
+SYM_FUNC_END_SECTION(chacha_block_xor_ssse3)
-SYM_FUNC_START(hchacha_block_ssse3)
+SYM_FUNC_START_SECTION(hchacha_block_ssse3)
# %rdi: Input state matrix, s
# %rsi: output (8 32-bit words)
# %edx: nrounds
@@ -218,9 +218,9 @@ SYM_FUNC_START(hchacha_block_ssse3)
FRAME_END
ret
-SYM_FUNC_END(hchacha_block_ssse3)
+SYM_FUNC_END_SECTION(hchacha_block_ssse3)
-SYM_FUNC_START(chacha_4block_xor_ssse3)
+SYM_FUNC_START_SECTION(chacha_4block_xor_ssse3)
# %rdi: Input state matrix, s
# %rsi: up to 4 data blocks output, o
# %rdx: up to 4 data blocks input, i
@@ -788,4 +788,4 @@ SYM_FUNC_START(chacha_4block_xor_ssse3)
jmp .Ldone4
-SYM_FUNC_END(chacha_4block_xor_ssse3)
+SYM_FUNC_END_SECTION(chacha_4block_xor_ssse3)
diff --git a/arch/x86/crypto/crc32-pclmul_asm.S b/arch/x86/crypto/crc32-pclmul_asm.S
index 6e7d4c4d3208..21b19a2f4556 100644
--- a/arch/x86/crypto/crc32-pclmul_asm.S
+++ b/arch/x86/crypto/crc32-pclmul_asm.S
@@ -102,7 +102,7 @@
* size_t len, uint crc32)
*/
-SYM_FUNC_START(crc32_pclmul_le_16) /* buffer and buffer size are 16 bytes aligned */
+SYM_FUNC_START_SECTION(crc32_pclmul_le_16) /* buffer and buffer size are 16 bytes aligned */
movdqa (BUF), %xmm1
movdqa 0x10(BUF), %xmm2
movdqa 0x20(BUF), %xmm3
@@ -237,4 +237,4 @@ fold_64:
pextrd $0x01, %xmm1, %eax
ret
-SYM_FUNC_END(crc32_pclmul_le_16)
+SYM_FUNC_END_SECTION(crc32_pclmul_le_16)
diff --git a/arch/x86/crypto/crc32c-pcl-intel-asm_64.S b/arch/x86/crypto/crc32c-pcl-intel-asm_64.S
index ac1f303eed0f..342f09adf41c 100644
--- a/arch/x86/crypto/crc32c-pcl-intel-asm_64.S
+++ b/arch/x86/crypto/crc32c-pcl-intel-asm_64.S
@@ -73,7 +73,7 @@
# unsigned int crc_pcl(u8 *buffer, int len, unsigned int crc_init);
.text
-SYM_FUNC_START(crc_pcl)
+SYM_FUNC_START_SECTION(crc_pcl)
#define bufp rdi
#define bufp_dw %edi
#define bufp_w %di
@@ -307,7 +307,7 @@ do_return:
popq %rdi
popq %rbx
ret
-SYM_FUNC_END(crc_pcl)
+SYM_FUNC_END_SECTION(crc_pcl)
.section .rodata, "a", @progbits
################################################################
diff --git a/arch/x86/crypto/crct10dif-pcl-asm_64.S b/arch/x86/crypto/crct10dif-pcl-asm_64.S
index b2533d63030e..b6d35088fbfc 100644
--- a/arch/x86/crypto/crct10dif-pcl-asm_64.S
+++ b/arch/x86/crypto/crct10dif-pcl-asm_64.S
@@ -94,8 +94,8 @@
#
# Assumes len >= 16.
#
+SYM_FUNC_START_SECTION(crc_t10dif_pcl)
.align 16
-SYM_FUNC_START(crc_t10dif_pcl)
movdqa .Lbswap_mask(%rip), BSWAP_MASK
@@ -280,7 +280,7 @@ SYM_FUNC_START(crc_t10dif_pcl)
jge .Lfold_16_bytes_loop # 32 <= len <= 255
add $16, len
jmp .Lhandle_partial_segment # 17 <= len <= 31
-SYM_FUNC_END(crc_t10dif_pcl)
+SYM_FUNC_END_SECTION(crc_t10dif_pcl)
.section .rodata, "a", @progbits
.align 16
diff --git a/arch/x86/crypto/des3_ede-asm_64.S b/arch/x86/crypto/des3_ede-asm_64.S
index fac0fdc3f25d..116947d2e097 100644
--- a/arch/x86/crypto/des3_ede-asm_64.S
+++ b/arch/x86/crypto/des3_ede-asm_64.S
@@ -162,7 +162,7 @@
movl left##d, (io); \
movl right##d, 4(io);
-SYM_FUNC_START(des3_ede_x86_64_crypt_blk)
+SYM_FUNC_START_SECTION(des3_ede_x86_64_crypt_blk)
/* input:
* %rdi: round keys, CTX
* %rsi: dst
@@ -244,7 +244,7 @@ SYM_FUNC_START(des3_ede_x86_64_crypt_blk)
popq %rbx;
ret;
-SYM_FUNC_END(des3_ede_x86_64_crypt_blk)
+SYM_FUNC_END_SECTION(des3_ede_x86_64_crypt_blk)
/***********************************************************************
* 3-way 3DES
@@ -418,7 +418,7 @@ SYM_FUNC_END(des3_ede_x86_64_crypt_blk)
#define __movq(src, dst) \
movq src, dst;
-SYM_FUNC_START(des3_ede_x86_64_crypt_blk_3way)
+SYM_FUNC_START_SECTION(des3_ede_x86_64_crypt_blk_3way)
/* input:
* %rdi: ctx, round keys
* %rsi: dst (3 blocks)
@@ -529,7 +529,7 @@ SYM_FUNC_START(des3_ede_x86_64_crypt_blk_3way)
popq %rbx;
ret;
-SYM_FUNC_END(des3_ede_x86_64_crypt_blk_3way)
+SYM_FUNC_END_SECTION(des3_ede_x86_64_crypt_blk_3way)
.section .rodata, "a", @progbits
.align 16
diff --git a/arch/x86/crypto/ghash-clmulni-intel_asm.S b/arch/x86/crypto/ghash-clmulni-intel_asm.S
index 99ac25e18e09..719cb550ed7a 100644
--- a/arch/x86/crypto/ghash-clmulni-intel_asm.S
+++ b/arch/x86/crypto/ghash-clmulni-intel_asm.S
@@ -43,7 +43,7 @@
* T2
* T3
*/
-SYM_FUNC_START_LOCAL(__clmul_gf128mul_ble)
+SYM_FUNC_START_LOCAL_SECTION(__clmul_gf128mul_ble)
movaps DATA, T1
pshufd $0b01001110, DATA, T2
pshufd $0b01001110, SHASH, T3
@@ -86,10 +86,10 @@ SYM_FUNC_START_LOCAL(__clmul_gf128mul_ble)
pxor T2, T1
pxor T1, DATA
ret
-SYM_FUNC_END(__clmul_gf128mul_ble)
+SYM_FUNC_END_SECTION(__clmul_gf128mul_ble)
/* void clmul_ghash_mul(char *dst, const u128 *shash) */
-SYM_FUNC_START(clmul_ghash_mul)
+SYM_FUNC_START_SECTION(clmul_ghash_mul)
FRAME_BEGIN
movups (%rdi), DATA
movups (%rsi), SHASH
@@ -100,13 +100,13 @@ SYM_FUNC_START(clmul_ghash_mul)
movups DATA, (%rdi)
FRAME_END
ret
-SYM_FUNC_END(clmul_ghash_mul)
+SYM_FUNC_END_SECTION(clmul_ghash_mul)
/*
* void clmul_ghash_update(char *dst, const char *src, unsigned int srclen,
* const u128 *shash);
*/
-SYM_FUNC_START(clmul_ghash_update)
+SYM_FUNC_START_SECTION(clmul_ghash_update)
FRAME_BEGIN
cmp $16, %rdx
jb .Lupdate_just_ret # check length
@@ -129,4 +129,4 @@ SYM_FUNC_START(clmul_ghash_update)
.Lupdate_just_ret:
FRAME_END
ret
-SYM_FUNC_END(clmul_ghash_update)
+SYM_FUNC_END_SECTION(clmul_ghash_update)
diff --git a/arch/x86/crypto/nh-avx2-x86_64.S b/arch/x86/crypto/nh-avx2-x86_64.S
index b22c7b936272..6df795f28706 100644
--- a/arch/x86/crypto/nh-avx2-x86_64.S
+++ b/arch/x86/crypto/nh-avx2-x86_64.S
@@ -69,7 +69,7 @@
*
* It's guaranteed that message_len % 16 == 0.
*/
-SYM_FUNC_START(nh_avx2)
+SYM_FUNC_START_SECTION(nh_avx2)
vmovdqu 0x00(KEY), K0
vmovdqu 0x10(KEY), K1
@@ -154,4 +154,4 @@ SYM_FUNC_START(nh_avx2)
vpaddq T4, T0, T0
vmovdqu T0, (HASH)
ret
-SYM_FUNC_END(nh_avx2)
+SYM_FUNC_END_SECTION(nh_avx2)
diff --git a/arch/x86/crypto/nh-sse2-x86_64.S b/arch/x86/crypto/nh-sse2-x86_64.S
index d7ae22dd6683..5113a5a081fd 100644
--- a/arch/x86/crypto/nh-sse2-x86_64.S
+++ b/arch/x86/crypto/nh-sse2-x86_64.S
@@ -71,7 +71,7 @@
*
* It's guaranteed that message_len % 16 == 0.
*/
-SYM_FUNC_START(nh_sse2)
+SYM_FUNC_START_SECTION(nh_sse2)
movdqu 0x00(KEY), K0
movdqu 0x10(KEY), K1
@@ -120,4 +120,4 @@ SYM_FUNC_START(nh_sse2)
movdqu T0, 0x00(HASH)
movdqu T1, 0x10(HASH)
ret
-SYM_FUNC_END(nh_sse2)
+SYM_FUNC_END_SECTION(nh_sse2)
diff --git a/arch/x86/crypto/poly1305-x86_64-cryptogams.pl b/arch/x86/crypto/poly1305-x86_64-cryptogams.pl
index 71fae5a09e56..c8797e902793 100644
--- a/arch/x86/crypto/poly1305-x86_64-cryptogams.pl
+++ b/arch/x86/crypto/poly1305-x86_64-cryptogams.pl
@@ -108,8 +108,8 @@ if (!$kernel) {
sub declare_function() {
my ($name, $align, $nargs) = @_;
if($kernel) {
+ $code .= "SYM_FUNC_START_SECTION($name)\n";
$code .= ".align $align\n";
- $code .= "SYM_FUNC_START($name)\n";
$code .= ".L$name:\n";
} else {
$code .= ".globl $name\n";
@@ -122,7 +122,7 @@ sub declare_function() {
sub end_function() {
my ($name) = @_;
if($kernel) {
- $code .= "SYM_FUNC_END($name)\n";
+ $code .= "SYM_FUNC_END_SECTION($name)\n";
} else {
$code .= ".size $name,.-$name\n";
}
@@ -421,6 +421,7 @@ my ($H0,$H1,$H2,$H3,$H4, $T0,$T1,$T2,$T3,$T4, $D0,$D1,$D2,$D3,$D4, $MASK) =
map("%xmm$_",(0..15));
$code.=<<___;
+SYM_TEXT_SECTION(__poly1305_block)
.type __poly1305_block,\@abi-omnipotent
.align 32
__poly1305_block:
@@ -431,7 +432,9 @@ $code.=<<___;
pop $ctx
ret
.size __poly1305_block,.-__poly1305_block
+SYM_TEXT_END_SECTION
+SYM_TEXT_SECTION(__poly1305_init_avx)
.type __poly1305_init_avx,\@abi-omnipotent
.align 32
__poly1305_init_avx:
@@ -596,6 +599,7 @@ __poly1305_init_avx:
pop %rbp
ret
.size __poly1305_init_avx,.-__poly1305_init_avx
+SYM_TEXT_END_SECTION
___
&declare_function("poly1305_blocks_avx", 32, 4);
diff --git a/arch/x86/crypto/serpent-avx-x86_64-asm_64.S b/arch/x86/crypto/serpent-avx-x86_64-asm_64.S
index b7ee24df7fba..072f26f4bc9b 100644
--- a/arch/x86/crypto/serpent-avx-x86_64-asm_64.S
+++ b/arch/x86/crypto/serpent-avx-x86_64-asm_64.S
@@ -550,8 +550,8 @@
#define write_blocks(x0, x1, x2, x3, t0, t1, t2) \
transpose_4x4(x0, x1, x2, x3, t0, t1, t2)
+SYM_FUNC_START_LOCAL_SECTION(__serpent_enc_blk8_avx)
.align 8
-SYM_FUNC_START_LOCAL(__serpent_enc_blk8_avx)
/* input:
* %rdi: ctx, CTX
* RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: blocks
@@ -602,10 +602,10 @@ SYM_FUNC_START_LOCAL(__serpent_enc_blk8_avx)
write_blocks(RA2, RB2, RC2, RD2, RK0, RK1, RK2);
ret;
-SYM_FUNC_END(__serpent_enc_blk8_avx)
+SYM_FUNC_END_SECTION(__serpent_enc_blk8_avx)
+SYM_FUNC_START_LOCAL_SECTION(__serpent_dec_blk8_avx)
.align 8
-SYM_FUNC_START_LOCAL(__serpent_dec_blk8_avx)
/* input:
* %rdi: ctx, CTX
* RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: encrypted blocks
@@ -656,9 +656,9 @@ SYM_FUNC_START_LOCAL(__serpent_dec_blk8_avx)
write_blocks(RC2, RD2, RB2, RE2, RK0, RK1, RK2);
ret;
-SYM_FUNC_END(__serpent_dec_blk8_avx)
+SYM_FUNC_END_SECTION(__serpent_dec_blk8_avx)
-SYM_FUNC_START(serpent_ecb_enc_8way_avx)
+SYM_FUNC_START_SECTION(serpent_ecb_enc_8way_avx)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -674,9 +674,9 @@ SYM_FUNC_START(serpent_ecb_enc_8way_avx)
FRAME_END
ret;
-SYM_FUNC_END(serpent_ecb_enc_8way_avx)
+SYM_FUNC_END_SECTION(serpent_ecb_enc_8way_avx)
-SYM_FUNC_START(serpent_ecb_dec_8way_avx)
+SYM_FUNC_START_SECTION(serpent_ecb_dec_8way_avx)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -692,9 +692,9 @@ SYM_FUNC_START(serpent_ecb_dec_8way_avx)
FRAME_END
ret;
-SYM_FUNC_END(serpent_ecb_dec_8way_avx)
+SYM_FUNC_END_SECTION(serpent_ecb_dec_8way_avx)
-SYM_FUNC_START(serpent_cbc_dec_8way_avx)
+SYM_FUNC_START_SECTION(serpent_cbc_dec_8way_avx)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -710,4 +710,4 @@ SYM_FUNC_START(serpent_cbc_dec_8way_avx)
FRAME_END
ret;
-SYM_FUNC_END(serpent_cbc_dec_8way_avx)
+SYM_FUNC_END_SECTION(serpent_cbc_dec_8way_avx)
diff --git a/arch/x86/crypto/serpent-avx2-asm_64.S b/arch/x86/crypto/serpent-avx2-asm_64.S
index 9161b6e441f3..57b25c3ab45b 100644
--- a/arch/x86/crypto/serpent-avx2-asm_64.S
+++ b/arch/x86/crypto/serpent-avx2-asm_64.S
@@ -550,8 +550,8 @@
#define write_blocks(x0, x1, x2, x3, t0, t1, t2) \
transpose_4x4(x0, x1, x2, x3, t0, t1, t2)
+SYM_FUNC_START_LOCAL_SECTION(__serpent_enc_blk16)
.align 8
-SYM_FUNC_START_LOCAL(__serpent_enc_blk16)
/* input:
* %rdi: ctx, CTX
* RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: plaintext
@@ -602,10 +602,10 @@ SYM_FUNC_START_LOCAL(__serpent_enc_blk16)
write_blocks(RA2, RB2, RC2, RD2, RK0, RK1, RK2);
ret;
-SYM_FUNC_END(__serpent_enc_blk16)
+SYM_FUNC_END_SECTION(__serpent_enc_blk16)
+SYM_FUNC_START_LOCAL_SECTION(__serpent_dec_blk16)
.align 8
-SYM_FUNC_START_LOCAL(__serpent_dec_blk16)
/* input:
* %rdi: ctx, CTX
* RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: ciphertext
@@ -656,9 +656,9 @@ SYM_FUNC_START_LOCAL(__serpent_dec_blk16)
write_blocks(RC2, RD2, RB2, RE2, RK0, RK1, RK2);
ret;
-SYM_FUNC_END(__serpent_dec_blk16)
+SYM_FUNC_END_SECTION(__serpent_dec_blk16)
-SYM_FUNC_START(serpent_ecb_enc_16way)
+SYM_FUNC_START_SECTION(serpent_ecb_enc_16way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -678,9 +678,9 @@ SYM_FUNC_START(serpent_ecb_enc_16way)
FRAME_END
ret;
-SYM_FUNC_END(serpent_ecb_enc_16way)
+SYM_FUNC_END_SECTION(serpent_ecb_enc_16way)
-SYM_FUNC_START(serpent_ecb_dec_16way)
+SYM_FUNC_START_SECTION(serpent_ecb_dec_16way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -700,9 +700,9 @@ SYM_FUNC_START(serpent_ecb_dec_16way)
FRAME_END
ret;
-SYM_FUNC_END(serpent_ecb_dec_16way)
+SYM_FUNC_END_SECTION(serpent_ecb_dec_16way)
-SYM_FUNC_START(serpent_cbc_dec_16way)
+SYM_FUNC_START_SECTION(serpent_cbc_dec_16way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -723,4 +723,4 @@ SYM_FUNC_START(serpent_cbc_dec_16way)
FRAME_END
ret;
-SYM_FUNC_END(serpent_cbc_dec_16way)
+SYM_FUNC_END_SECTION(serpent_cbc_dec_16way)
diff --git a/arch/x86/crypto/serpent-sse2-i586-asm_32.S b/arch/x86/crypto/serpent-sse2-i586-asm_32.S
index 6379b99cb722..c74cfb225d81 100644
--- a/arch/x86/crypto/serpent-sse2-i586-asm_32.S
+++ b/arch/x86/crypto/serpent-sse2-i586-asm_32.S
@@ -497,7 +497,7 @@
pxor t0, x3; \
movdqu x3, (3*4*4)(out);
-SYM_FUNC_START(__serpent_enc_blk_4way)
+SYM_FUNC_START_SECTION(__serpent_enc_blk_4way)
/* input:
* arg_ctx(%esp): ctx, CTX
* arg_dst(%esp): dst
@@ -559,9 +559,9 @@ SYM_FUNC_START(__serpent_enc_blk_4way)
xor_blocks(%eax, RA, RB, RC, RD, RT0, RT1, RE);
ret;
-SYM_FUNC_END(__serpent_enc_blk_4way)
+SYM_FUNC_END_SECTION(__serpent_enc_blk_4way)
-SYM_FUNC_START(serpent_dec_blk_4way)
+SYM_FUNC_START_SECTION(serpent_dec_blk_4way)
/* input:
* arg_ctx(%esp): ctx, CTX
* arg_dst(%esp): dst
@@ -613,4 +613,4 @@ SYM_FUNC_START(serpent_dec_blk_4way)
write_blocks(%eax, RC, RD, RB, RE, RT0, RT1, RA);
ret;
-SYM_FUNC_END(serpent_dec_blk_4way)
+SYM_FUNC_END_SECTION(serpent_dec_blk_4way)
diff --git a/arch/x86/crypto/serpent-sse2-x86_64-asm_64.S b/arch/x86/crypto/serpent-sse2-x86_64-asm_64.S
index efb6dc17dc90..772b3ddfe0f2 100644
--- a/arch/x86/crypto/serpent-sse2-x86_64-asm_64.S
+++ b/arch/x86/crypto/serpent-sse2-x86_64-asm_64.S
@@ -619,7 +619,7 @@
pxor t0, x3; \
movdqu x3, (3*4*4)(out);
-SYM_FUNC_START(__serpent_enc_blk_8way)
+SYM_FUNC_START_SECTION(__serpent_enc_blk_8way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -682,9 +682,9 @@ SYM_FUNC_START(__serpent_enc_blk_8way)
xor_blocks(%rax, RA2, RB2, RC2, RD2, RK0, RK1, RK2);
ret;
-SYM_FUNC_END(__serpent_enc_blk_8way)
+SYM_FUNC_END_SECTION(__serpent_enc_blk_8way)
-SYM_FUNC_START(serpent_dec_blk_8way)
+SYM_FUNC_START_SECTION(serpent_dec_blk_8way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -736,4 +736,4 @@ SYM_FUNC_START(serpent_dec_blk_8way)
write_blocks(%rax, RC2, RD2, RB2, RE2, RK0, RK1, RK2);
ret;
-SYM_FUNC_END(serpent_dec_blk_8way)
+SYM_FUNC_END_SECTION(serpent_dec_blk_8way)
diff --git a/arch/x86/crypto/sha1_avx2_x86_64_asm.S b/arch/x86/crypto/sha1_avx2_x86_64_asm.S
index 5eed620f4676..b3f2b06121f5 100644
--- a/arch/x86/crypto/sha1_avx2_x86_64_asm.S
+++ b/arch/x86/crypto/sha1_avx2_x86_64_asm.S
@@ -634,7 +634,7 @@ _loop3:
* param: function's name
*/
.macro SHA1_VECTOR_ASM name
- SYM_FUNC_START(\name)
+ SYM_FUNC_START_SECTION(\name)
push %rbx
push %r12
@@ -676,7 +676,7 @@ _loop3:
ret
- SYM_FUNC_END(\name)
+ SYM_FUNC_END_SECTION(\name)
.endm
.section .rodata
diff --git a/arch/x86/crypto/sha1_ni_asm.S b/arch/x86/crypto/sha1_ni_asm.S
index 5d8415f482bd..55fdacd4931f 100644
--- a/arch/x86/crypto/sha1_ni_asm.S
+++ b/arch/x86/crypto/sha1_ni_asm.S
@@ -92,8 +92,8 @@
* numBlocks: Number of blocks to process
*/
.text
+SYM_FUNC_START_SECTION(sha1_ni_transform)
.align 32
-SYM_FUNC_START(sha1_ni_transform)
push %rbp
mov %rsp, %rbp
sub $FRAME_SIZE, %rsp
@@ -291,7 +291,7 @@ SYM_FUNC_START(sha1_ni_transform)
pop %rbp
ret
-SYM_FUNC_END(sha1_ni_transform)
+SYM_FUNC_END_SECTION(sha1_ni_transform)
.section .rodata.cst16.PSHUFFLE_BYTE_FLIP_MASK, "aM", @progbits, 16
.align 16
diff --git a/arch/x86/crypto/sha1_ssse3_asm.S b/arch/x86/crypto/sha1_ssse3_asm.S
index d25668d2a1e9..936cba71c6ee 100644
--- a/arch/x86/crypto/sha1_ssse3_asm.S
+++ b/arch/x86/crypto/sha1_ssse3_asm.S
@@ -67,7 +67,7 @@
* param: function's name
*/
.macro SHA1_VECTOR_ASM name
- SYM_FUNC_START(\name)
+ SYM_FUNC_START_SECTION(\name)
push %rbx
push %r12
@@ -101,7 +101,7 @@
pop %rbx
ret
- SYM_FUNC_END(\name)
+ SYM_FUNC_END_SECTION(\name)
.endm
/*
diff --git a/arch/x86/crypto/sha256-avx-asm.S b/arch/x86/crypto/sha256-avx-asm.S
index 4739cd31b9db..cfa45b4319c0 100644
--- a/arch/x86/crypto/sha256-avx-asm.S
+++ b/arch/x86/crypto/sha256-avx-asm.S
@@ -346,7 +346,7 @@ a = TMP_
## arg 3 : Num blocks
########################################################################
.text
-SYM_FUNC_START(sha256_transform_avx)
+SYM_FUNC_START_SECTION(sha256_transform_avx)
.align 32
pushq %rbx
pushq %r12
@@ -459,7 +459,7 @@ done_hash:
popq %r12
popq %rbx
ret
-SYM_FUNC_END(sha256_transform_avx)
+SYM_FUNC_END_SECTION(sha256_transform_avx)
.section .rodata.cst256.K256, "aM", @progbits, 256
.align 64
diff --git a/arch/x86/crypto/sha256-avx2-asm.S b/arch/x86/crypto/sha256-avx2-asm.S
index 4087f7432a7e..84b4f0bcc5a4 100644
--- a/arch/x86/crypto/sha256-avx2-asm.S
+++ b/arch/x86/crypto/sha256-avx2-asm.S
@@ -523,7 +523,7 @@ STACK_SIZE = _CTX + _CTX_SIZE
## arg 3 : Num blocks
########################################################################
.text
-SYM_FUNC_START(sha256_transform_rorx)
+SYM_FUNC_START_SECTION(sha256_transform_rorx)
.align 32
pushq %rbx
pushq %r12
@@ -711,7 +711,7 @@ done_hash:
popq %r12
popq %rbx
ret
-SYM_FUNC_END(sha256_transform_rorx)
+SYM_FUNC_END_SECTION(sha256_transform_rorx)
.section .rodata.cst512.K256, "aM", @progbits, 512
.align 64
diff --git a/arch/x86/crypto/sha256-ssse3-asm.S b/arch/x86/crypto/sha256-ssse3-asm.S
index ddfa863b4ee3..cc70ae3d02c0 100644
--- a/arch/x86/crypto/sha256-ssse3-asm.S
+++ b/arch/x86/crypto/sha256-ssse3-asm.S
@@ -355,7 +355,7 @@ a = TMP_
## arg 3 : Num blocks
########################################################################
.text
-SYM_FUNC_START(sha256_transform_ssse3)
+SYM_FUNC_START_SECTION(sha256_transform_ssse3)
.align 32
pushq %rbx
pushq %r12
@@ -473,7 +473,7 @@ done_hash:
popq %rbx
ret
-SYM_FUNC_END(sha256_transform_ssse3)
+SYM_FUNC_END_SECTION(sha256_transform_ssse3)
.section .rodata.cst256.K256, "aM", @progbits, 256
.align 64
diff --git a/arch/x86/crypto/sha256_ni_asm.S b/arch/x86/crypto/sha256_ni_asm.S
index 7abade04a3a3..c4049c4a4ef7 100644
--- a/arch/x86/crypto/sha256_ni_asm.S
+++ b/arch/x86/crypto/sha256_ni_asm.S
@@ -96,8 +96,8 @@
*/
.text
+SYM_FUNC_START_SECTION(sha256_ni_transform)
.align 32
-SYM_FUNC_START(sha256_ni_transform)
shl $6, NUM_BLKS /* convert to bytes */
jz .Ldone_hash
@@ -327,7 +327,7 @@ SYM_FUNC_START(sha256_ni_transform)
.Ldone_hash:
ret
-SYM_FUNC_END(sha256_ni_transform)
+SYM_FUNC_END_SECTION(sha256_ni_transform)
.section .rodata.cst256.K256, "aM", @progbits, 256
.align 64
diff --git a/arch/x86/crypto/sha512-avx-asm.S b/arch/x86/crypto/sha512-avx-asm.S
index 3d8f0fd4eea8..ab128785d19d 100644
--- a/arch/x86/crypto/sha512-avx-asm.S
+++ b/arch/x86/crypto/sha512-avx-asm.S
@@ -273,7 +273,7 @@ frame_size = frame_WK + WK_SIZE
# of SHA512 message blocks.
# "blocks" is the message length in SHA512 blocks
########################################################################
-SYM_FUNC_START(sha512_transform_avx)
+SYM_FUNC_START_SECTION(sha512_transform_avx)
test msglen, msglen
je nowork
@@ -362,7 +362,7 @@ updateblock:
nowork:
ret
-SYM_FUNC_END(sha512_transform_avx)
+SYM_FUNC_END_SECTION(sha512_transform_avx)
########################################################################
### Binary Data
diff --git a/arch/x86/crypto/sha512-avx2-asm.S b/arch/x86/crypto/sha512-avx2-asm.S
index 072cb0f0deae..e0b2faa3eac7 100644
--- a/arch/x86/crypto/sha512-avx2-asm.S
+++ b/arch/x86/crypto/sha512-avx2-asm.S
@@ -565,7 +565,7 @@ frame_size = frame_CTX + CTX_SIZE
# of SHA512 message blocks.
# "blocks" is the message length in SHA512 blocks
########################################################################
-SYM_FUNC_START(sha512_transform_rorx)
+SYM_FUNC_START_SECTION(sha512_transform_rorx)
# Save GPRs
push %rbx
push %r12
@@ -680,7 +680,7 @@ done_hash:
pop %rbx
ret
-SYM_FUNC_END(sha512_transform_rorx)
+SYM_FUNC_END_SECTION(sha512_transform_rorx)
########################################################################
### Binary Data
diff --git a/arch/x86/crypto/sha512-ssse3-asm.S b/arch/x86/crypto/sha512-ssse3-asm.S
index bd51c9070bed..a3d7c69b8085 100644
--- a/arch/x86/crypto/sha512-ssse3-asm.S
+++ b/arch/x86/crypto/sha512-ssse3-asm.S
@@ -274,7 +274,7 @@ frame_size = frame_WK + WK_SIZE
# of SHA512 message blocks.
# "blocks" is the message length in SHA512 blocks.
########################################################################
-SYM_FUNC_START(sha512_transform_ssse3)
+SYM_FUNC_START_SECTION(sha512_transform_ssse3)
test msglen, msglen
je nowork
@@ -364,7 +364,7 @@ updateblock:
nowork:
ret
-SYM_FUNC_END(sha512_transform_ssse3)
+SYM_FUNC_END_SECTION(sha512_transform_ssse3)
########################################################################
### Binary Data
diff --git a/arch/x86/crypto/sm4-aesni-avx-asm_64.S b/arch/x86/crypto/sm4-aesni-avx-asm_64.S
index 1cc72b4804fa..021064aa0d2e 100644
--- a/arch/x86/crypto/sm4-aesni-avx-asm_64.S
+++ b/arch/x86/crypto/sm4-aesni-avx-asm_64.S
@@ -145,8 +145,8 @@
* void sm4_aesni_avx_crypt4(const u32 *rk, u8 *dst,
* const u8 *src, int nblocks)
*/
+SYM_FUNC_START_SECTION(sm4_aesni_avx_crypt4)
.align 8
-SYM_FUNC_START(sm4_aesni_avx_crypt4)
/* input:
* %rdi: round key array, CTX
* %rsi: dst (1..4 blocks)
@@ -247,10 +247,10 @@ SYM_FUNC_START(sm4_aesni_avx_crypt4)
vzeroall;
FRAME_END
ret;
-SYM_FUNC_END(sm4_aesni_avx_crypt4)
+SYM_FUNC_END_SECTION(sm4_aesni_avx_crypt4)
+SYM_FUNC_START_LOCAL_SECTION(__sm4_crypt_blk8)
.align 8
-SYM_FUNC_START_LOCAL(__sm4_crypt_blk8)
/* input:
* %rdi: round key array, CTX
* RA0, RA1, RA2, RA3, RB0, RB1, RB2, RB3: eight parallel
@@ -357,14 +357,14 @@ SYM_FUNC_START_LOCAL(__sm4_crypt_blk8)
FRAME_END
ret;
-SYM_FUNC_END(__sm4_crypt_blk8)
+SYM_FUNC_END_SECTION(__sm4_crypt_blk8)
/*
* void sm4_aesni_avx_crypt8(const u32 *rk, u8 *dst,
* const u8 *src, int nblocks)
*/
+SYM_FUNC_START_SECTION(sm4_aesni_avx_crypt8)
.align 8
-SYM_FUNC_START(sm4_aesni_avx_crypt8)
/* input:
* %rdi: round key array, CTX
* %rsi: dst (1..8 blocks)
@@ -413,14 +413,14 @@ SYM_FUNC_START(sm4_aesni_avx_crypt8)
vzeroall;
FRAME_END
ret;
-SYM_FUNC_END(sm4_aesni_avx_crypt8)
+SYM_FUNC_END_SECTION(sm4_aesni_avx_crypt8)
/*
* void sm4_aesni_avx_ctr_enc_blk8(const u32 *rk, u8 *dst,
* const u8 *src, u8 *iv)
*/
+SYM_FUNC_START_SECTION(sm4_aesni_avx_ctr_enc_blk8)
.align 8
-SYM_FUNC_START(sm4_aesni_avx_ctr_enc_blk8)
/* input:
* %rdi: round key array, CTX
* %rsi: dst (8 blocks)
@@ -488,14 +488,14 @@ SYM_FUNC_START(sm4_aesni_avx_ctr_enc_blk8)
vzeroall;
FRAME_END
ret;
-SYM_FUNC_END(sm4_aesni_avx_ctr_enc_blk8)
+SYM_FUNC_END_SECTION(sm4_aesni_avx_ctr_enc_blk8)
/*
* void sm4_aesni_avx_cbc_dec_blk8(const u32 *rk, u8 *dst,
* const u8 *src, u8 *iv)
*/
+SYM_FUNC_START_SECTION(sm4_aesni_avx_cbc_dec_blk8)
.align 8
-SYM_FUNC_START(sm4_aesni_avx_cbc_dec_blk8)
/* input:
* %rdi: round key array, CTX
* %rsi: dst (8 blocks)
@@ -538,14 +538,14 @@ SYM_FUNC_START(sm4_aesni_avx_cbc_dec_blk8)
vzeroall;
FRAME_END
ret;
-SYM_FUNC_END(sm4_aesni_avx_cbc_dec_blk8)
+SYM_FUNC_END_SECTION(sm4_aesni_avx_cbc_dec_blk8)
/*
* void sm4_aesni_avx_cfb_dec_blk8(const u32 *rk, u8 *dst,
* const u8 *src, u8 *iv)
*/
+SYM_FUNC_START_SECTION(sm4_aesni_avx_cfb_dec_blk8)
.align 8
-SYM_FUNC_START(sm4_aesni_avx_cfb_dec_blk8)
/* input:
* %rdi: round key array, CTX
* %rsi: dst (8 blocks)
@@ -591,4 +591,4 @@ SYM_FUNC_START(sm4_aesni_avx_cfb_dec_blk8)
vzeroall;
FRAME_END
ret;
-SYM_FUNC_END(sm4_aesni_avx_cfb_dec_blk8)
+SYM_FUNC_END_SECTION(sm4_aesni_avx_cfb_dec_blk8)
diff --git a/arch/x86/crypto/sm4-aesni-avx2-asm_64.S b/arch/x86/crypto/sm4-aesni-avx2-asm_64.S
index 9c5d3f3ad45a..3cc3116815f1 100644
--- a/arch/x86/crypto/sm4-aesni-avx2-asm_64.S
+++ b/arch/x86/crypto/sm4-aesni-avx2-asm_64.S
@@ -155,8 +155,8 @@
.text
.align 16
+SYM_FUNC_START_LOCAL_SECTION(__sm4_crypt_blk16)
.align 8
-SYM_FUNC_START_LOCAL(__sm4_crypt_blk16)
/* input:
* %rdi: round key array, CTX
* RA0, RA1, RA2, RA3, RB0, RB1, RB2, RB3: sixteen parallel
@@ -269,7 +269,7 @@ SYM_FUNC_START_LOCAL(__sm4_crypt_blk16)
FRAME_END
ret;
-SYM_FUNC_END(__sm4_crypt_blk16)
+SYM_FUNC_END_SECTION(__sm4_crypt_blk16)
#define inc_le128(x, minus_one, tmp) \
vpcmpeqq minus_one, x, tmp; \
@@ -281,8 +281,8 @@ SYM_FUNC_END(__sm4_crypt_blk16)
* void sm4_aesni_avx2_ctr_enc_blk16(const u32 *rk, u8 *dst,
* const u8 *src, u8 *iv)
*/
+SYM_FUNC_START_SECTION(sm4_aesni_avx2_ctr_enc_blk16)
.align 8
-SYM_FUNC_START(sm4_aesni_avx2_ctr_enc_blk16)
/* input:
* %rdi: round key array, CTX
* %rsi: dst (16 blocks)
@@ -388,14 +388,14 @@ SYM_FUNC_START(sm4_aesni_avx2_ctr_enc_blk16)
vzeroall;
FRAME_END
ret;
-SYM_FUNC_END(sm4_aesni_avx2_ctr_enc_blk16)
+SYM_FUNC_END_SECTION(sm4_aesni_avx2_ctr_enc_blk16)
/*
* void sm4_aesni_avx2_cbc_dec_blk16(const u32 *rk, u8 *dst,
* const u8 *src, u8 *iv)
*/
+SYM_FUNC_START_SECTION(sm4_aesni_avx2_cbc_dec_blk16)
.align 8
-SYM_FUNC_START(sm4_aesni_avx2_cbc_dec_blk16)
/* input:
* %rdi: round key array, CTX
* %rsi: dst (16 blocks)
@@ -442,14 +442,14 @@ SYM_FUNC_START(sm4_aesni_avx2_cbc_dec_blk16)
vzeroall;
FRAME_END
ret;
-SYM_FUNC_END(sm4_aesni_avx2_cbc_dec_blk16)
+SYM_FUNC_END_SECTION(sm4_aesni_avx2_cbc_dec_blk16)
/*
* void sm4_aesni_avx2_cfb_dec_blk16(const u32 *rk, u8 *dst,
* const u8 *src, u8 *iv)
*/
+SYM_FUNC_START_SECTION(sm4_aesni_avx2_cfb_dec_blk16)
.align 8
-SYM_FUNC_START(sm4_aesni_avx2_cfb_dec_blk16)
/* input:
* %rdi: round key array, CTX
* %rsi: dst (16 blocks)
@@ -498,4 +498,4 @@ SYM_FUNC_START(sm4_aesni_avx2_cfb_dec_blk16)
vzeroall;
FRAME_END
ret;
-SYM_FUNC_END(sm4_aesni_avx2_cfb_dec_blk16)
+SYM_FUNC_END_SECTION(sm4_aesni_avx2_cfb_dec_blk16)
diff --git a/arch/x86/crypto/twofish-avx-x86_64-asm_64.S b/arch/x86/crypto/twofish-avx-x86_64-asm_64.S
index 37e63b3c664e..b3808a5c9003 100644
--- a/arch/x86/crypto/twofish-avx-x86_64-asm_64.S
+++ b/arch/x86/crypto/twofish-avx-x86_64-asm_64.S
@@ -228,8 +228,8 @@
vpxor x2, wkey, x2; \
vpxor x3, wkey, x3;
+SYM_FUNC_START_LOCAL_SECTION(__twofish_enc_blk8)
.align 8
-SYM_FUNC_START_LOCAL(__twofish_enc_blk8)
/* input:
* %rdi: ctx, CTX
* RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2: blocks
@@ -268,10 +268,10 @@ SYM_FUNC_START_LOCAL(__twofish_enc_blk8)
outunpack_blocks(RC2, RD2, RA2, RB2, RK1, RX0, RY0, RK2);
ret;
-SYM_FUNC_END(__twofish_enc_blk8)
+SYM_FUNC_END_SECTION(__twofish_enc_blk8)
+SYM_FUNC_START_LOCAL_SECTION(__twofish_dec_blk8)
.align 8
-SYM_FUNC_START_LOCAL(__twofish_dec_blk8)
/* input:
* %rdi: ctx, CTX
* RC1, RD1, RA1, RB1, RC2, RD2, RA2, RB2: encrypted blocks
@@ -308,9 +308,9 @@ SYM_FUNC_START_LOCAL(__twofish_dec_blk8)
outunpack_blocks(RA2, RB2, RC2, RD2, RK1, RX0, RY0, RK2);
ret;
-SYM_FUNC_END(__twofish_dec_blk8)
+SYM_FUNC_END_SECTION(__twofish_dec_blk8)
-SYM_FUNC_START(twofish_ecb_enc_8way)
+SYM_FUNC_START_SECTION(twofish_ecb_enc_8way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -328,9 +328,9 @@ SYM_FUNC_START(twofish_ecb_enc_8way)
FRAME_END
ret;
-SYM_FUNC_END(twofish_ecb_enc_8way)
+SYM_FUNC_END_SECTION(twofish_ecb_enc_8way)
-SYM_FUNC_START(twofish_ecb_dec_8way)
+SYM_FUNC_START_SECTION(twofish_ecb_dec_8way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -348,9 +348,9 @@ SYM_FUNC_START(twofish_ecb_dec_8way)
FRAME_END
ret;
-SYM_FUNC_END(twofish_ecb_dec_8way)
+SYM_FUNC_END_SECTION(twofish_ecb_dec_8way)
-SYM_FUNC_START(twofish_cbc_dec_8way)
+SYM_FUNC_START_SECTION(twofish_cbc_dec_8way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -373,4 +373,4 @@ SYM_FUNC_START(twofish_cbc_dec_8way)
FRAME_END
ret;
-SYM_FUNC_END(twofish_cbc_dec_8way)
+SYM_FUNC_END_SECTION(twofish_cbc_dec_8way)
diff --git a/arch/x86/crypto/twofish-i586-asm_32.S b/arch/x86/crypto/twofish-i586-asm_32.S
index a6f09e4f2e46..cf633600ed20 100644
--- a/arch/x86/crypto/twofish-i586-asm_32.S
+++ b/arch/x86/crypto/twofish-i586-asm_32.S
@@ -207,7 +207,7 @@
xor %esi, d ## D;\
ror $1, d ## D;
-SYM_FUNC_START(twofish_enc_blk)
+SYM_FUNC_START_SECTION(twofish_enc_blk)
push %ebp /* save registers according to calling convention*/
push %ebx
push %esi
@@ -261,9 +261,9 @@ SYM_FUNC_START(twofish_enc_blk)
pop %ebp
mov $1, %eax
ret
-SYM_FUNC_END(twofish_enc_blk)
+SYM_FUNC_END_SECTION(twofish_enc_blk)
-SYM_FUNC_START(twofish_dec_blk)
+SYM_FUNC_START_SECTION(twofish_dec_blk)
push %ebp /* save registers according to calling convention*/
push %ebx
push %esi
@@ -318,4 +318,4 @@ SYM_FUNC_START(twofish_dec_blk)
pop %ebp
mov $1, %eax
ret
-SYM_FUNC_END(twofish_dec_blk)
+SYM_FUNC_END_SECTION(twofish_dec_blk)
diff --git a/arch/x86/crypto/twofish-x86_64-asm_64-3way.S b/arch/x86/crypto/twofish-x86_64-asm_64-3way.S
index bca4cea757ce..1878b26473fb 100644
--- a/arch/x86/crypto/twofish-x86_64-asm_64-3way.S
+++ b/arch/x86/crypto/twofish-x86_64-asm_64-3way.S
@@ -220,7 +220,7 @@
rorq $32, RAB2; \
outunpack3(mov, RIO, 2, RAB, 2);
-SYM_FUNC_START(__twofish_enc_blk_3way)
+SYM_FUNC_START_SECTION(__twofish_enc_blk_3way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -267,9 +267,9 @@ SYM_FUNC_START(__twofish_enc_blk_3way)
popq %r12;
popq %r13;
ret;
-SYM_FUNC_END(__twofish_enc_blk_3way)
+SYM_FUNC_END_SECTION(__twofish_enc_blk_3way)
-SYM_FUNC_START(twofish_dec_blk_3way)
+SYM_FUNC_START_SECTION(twofish_dec_blk_3way)
/* input:
* %rdi: ctx, CTX
* %rsi: dst
@@ -302,4 +302,4 @@ SYM_FUNC_START(twofish_dec_blk_3way)
popq %r12;
popq %r13;
ret;
-SYM_FUNC_END(twofish_dec_blk_3way)
+SYM_FUNC_END_SECTION(twofish_dec_blk_3way)
diff --git a/arch/x86/crypto/twofish-x86_64-asm_64.S b/arch/x86/crypto/twofish-x86_64-asm_64.S
index d2e56232494a..87013b60e9d1 100644
--- a/arch/x86/crypto/twofish-x86_64-asm_64.S
+++ b/arch/x86/crypto/twofish-x86_64-asm_64.S
@@ -202,7 +202,7 @@
xor %r8d, d ## D;\
ror $1, d ## D;
-SYM_FUNC_START(twofish_enc_blk)
+SYM_FUNC_START_SECTION(twofish_enc_blk)
pushq R1
/* %rdi contains the ctx address */
@@ -253,9 +253,9 @@ SYM_FUNC_START(twofish_enc_blk)
popq R1
movl $1,%eax
ret
-SYM_FUNC_END(twofish_enc_blk)
+SYM_FUNC_END_SECTION(twofish_enc_blk)
-SYM_FUNC_START(twofish_dec_blk)
+SYM_FUNC_START_SECTION(twofish_dec_blk)
pushq R1
/* %rdi contains the ctx address */
@@ -305,4 +305,4 @@ SYM_FUNC_START(twofish_dec_blk)
popq R1
movl $1,%eax
ret
-SYM_FUNC_END(twofish_dec_blk)
+SYM_FUNC_END_SECTION(twofish_dec_blk)
diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S
index ccb9d32768f3..0447fac4e558 100644
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -675,8 +675,7 @@ __irqentry_text_end:
* %eax: prev task
* %edx: next task
*/
-.pushsection .text, "ax"
-SYM_CODE_START(__switch_to_asm)
+SYM_CODE_START_SECTION(__switch_to_asm)
/*
* Save callee-saved registers
* This must match the order in struct inactive_task_frame
@@ -721,8 +720,7 @@ SYM_CODE_START(__switch_to_asm)
popl %ebp
jmp __switch_to
-SYM_CODE_END(__switch_to_asm)
-.popsection
+SYM_CODE_END_SECTION(__switch_to_asm)
/*
* The unwinder expects the last frame on the stack to always be at the same
@@ -731,8 +729,7 @@ SYM_CODE_END(__switch_to_asm)
* asmlinkage function so its argument has to be pushed on the stack. This
* wrapper creates a proper "end of stack" frame header before the call.
*/
-.pushsection .text, "ax"
-SYM_FUNC_START(schedule_tail_wrapper)
+SYM_FUNC_START_SECTION(schedule_tail_wrapper)
FRAME_BEGIN
pushl %eax
@@ -741,8 +738,7 @@ SYM_FUNC_START(schedule_tail_wrapper)
FRAME_END
ret
-SYM_FUNC_END(schedule_tail_wrapper)
-.popsection
+SYM_FUNC_END_SECTION(schedule_tail_wrapper)
/*
* A newly forked process directly context switches into this address.
@@ -751,8 +747,7 @@ SYM_FUNC_END(schedule_tail_wrapper)
* ebx: kernel thread func (NULL for user thread)
* edi: kernel thread arg
*/
-.pushsection .text, "ax"
-SYM_CODE_START(ret_from_fork)
+SYM_CODE_START_SECTION(ret_from_fork)
call schedule_tail_wrapper
testl %ebx, %ebx
@@ -774,8 +769,7 @@ SYM_CODE_START(ret_from_fork)
*/
movl $0, PT_EAX(%esp)
jmp 2b
-SYM_CODE_END(ret_from_fork)
-.popsection
+SYM_CODE_END_SECTION(ret_from_fork)
SYM_ENTRY(__begin_SYSENTER_singlestep_region, SYM_L_GLOBAL, SYM_A_NONE)
/*
@@ -1247,8 +1241,7 @@ SYM_CODE_START(asm_exc_nmi)
#endif
SYM_CODE_END(asm_exc_nmi)
-.pushsection .text, "ax"
-SYM_CODE_START(rewind_stack_do_exit)
+SYM_CODE_START_SECTION(rewind_stack_do_exit)
/* Prevent any naive code from trying to unwind to our caller. */
xorl %ebp, %ebp
@@ -1257,5 +1250,4 @@ SYM_CODE_START(rewind_stack_do_exit)
call do_exit
1: jmp 1b
-SYM_CODE_END(rewind_stack_do_exit)
-.popsection
+SYM_CODE_END_SECTION(rewind_stack_do_exit)
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index e38a4cf795d9..815e9f979467 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -221,8 +221,7 @@ SYM_CODE_END(entry_SYSCALL_64)
* %rdi: prev task
* %rsi: next task
*/
-.pushsection .text, "ax"
-SYM_FUNC_START(__switch_to_asm)
+SYM_FUNC_START_SECTION(__switch_to_asm)
/*
* Save callee-saved registers
* This must match the order in inactive_task_frame
@@ -263,8 +262,7 @@ SYM_FUNC_START(__switch_to_asm)
popq %rbp
jmp __switch_to
-SYM_FUNC_END(__switch_to_asm)
-.popsection
+SYM_FUNC_END_SECTION(__switch_to_asm)
/*
* A newly forked process directly context switches into this address.
@@ -273,8 +271,7 @@ SYM_FUNC_END(__switch_to_asm)
* rbx: kernel thread func (NULL for user thread)
* r12: kernel thread arg
*/
-.pushsection .text, "ax"
-SYM_CODE_START(ret_from_fork)
+SYM_CODE_START_SECTION(ret_from_fork)
UNWIND_HINT_EMPTY
movq %rax, %rdi
call schedule_tail /* rdi: 'prev' task parameter */
@@ -300,8 +297,7 @@ SYM_CODE_START(ret_from_fork)
*/
movq $0, RAX(%rsp)
jmp 2b
-SYM_CODE_END(ret_from_fork)
-.popsection
+SYM_CODE_END_SECTION(ret_from_fork)
.macro DEBUG_ENTRY_ASSERT_IRQS_OFF
#ifdef CONFIG_DEBUG_ENTRY
@@ -1428,8 +1424,7 @@ SYM_CODE_START(ignore_sysret)
SYM_CODE_END(ignore_sysret)
#endif
-.pushsection .text, "ax"
-SYM_CODE_START(rewind_stack_do_exit)
+SYM_CODE_START_SECTION(rewind_stack_do_exit)
UNWIND_HINT_FUNC
/* Prevent any naive code from trying to unwind to our caller. */
xorl %ebp, %ebp
@@ -1439,5 +1434,4 @@ SYM_CODE_START(rewind_stack_do_exit)
UNWIND_HINT_REGS
call do_exit
-SYM_CODE_END(rewind_stack_do_exit)
-.popsection
+SYM_CODE_END_SECTION(rewind_stack_do_exit)
diff --git a/arch/x86/entry/thunk_32.S b/arch/x86/entry/thunk_32.S
index f1f96d4d8cd6..0149feb5d7a4 100644
--- a/arch/x86/entry/thunk_32.S
+++ b/arch/x86/entry/thunk_32.S
@@ -10,7 +10,7 @@
/* put return address in eax (arg1) */
.macro THUNK name, func, put_ret_addr_in_eax=0
-SYM_CODE_START_NOALIGN(\name)
+SYM_CODE_START_NOALIGN_SECTION(\name)
pushl %eax
pushl %ecx
pushl %edx
@@ -26,7 +26,7 @@ SYM_CODE_START_NOALIGN(\name)
popl %eax
ret
_ASM_NOKPROBE(\name)
-SYM_CODE_END(\name)
+SYM_CODE_END_SECTION(\name)
.endm
#ifdef CONFIG_PREEMPTION
diff --git a/arch/x86/entry/thunk_64.S b/arch/x86/entry/thunk_64.S
index 496b11ec469d..04c495e06020 100644
--- a/arch/x86/entry/thunk_64.S
+++ b/arch/x86/entry/thunk_64.S
@@ -11,7 +11,7 @@
/* rdi: arg1 ... normal C conventions. rax is saved/restored. */
.macro THUNK name, func
-SYM_FUNC_START_NOALIGN(\name)
+SYM_FUNC_START_NOALIGN_SECTION(\name)
pushq %rbp
movq %rsp, %rbp
@@ -27,7 +27,7 @@ SYM_FUNC_START_NOALIGN(\name)
call \func
jmp __thunk_restore
-SYM_FUNC_END(\name)
+SYM_FUNC_END_SECTION(\name)
_ASM_NOKPROBE(\name)
.endm
@@ -39,7 +39,7 @@ SYM_FUNC_END(\name)
#endif
#ifdef CONFIG_PREEMPTION
-SYM_CODE_START_LOCAL_NOALIGN(__thunk_restore)
+SYM_CODE_START_LOCAL_NOALIGN_SECTION(__thunk_restore)
popq %r11
popq %r10
popq %r9
@@ -52,5 +52,5 @@ SYM_CODE_START_LOCAL_NOALIGN(__thunk_restore)
popq %rbp
ret
_ASM_NOKPROBE(__thunk_restore)
-SYM_CODE_END(__thunk_restore)
+SYM_CODE_END_SECTION(__thunk_restore)
#endif
diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index 21c4a694ca11..9c16fa753b46 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -675,8 +675,9 @@ bool __raw_callee_save___native_vcpu_is_preempted(long cpu);
".size " PV_THUNK_NAME(func) ", .-" PV_THUNK_NAME(func) ";" \
".popsection")
-#define PV_CALLEE_SAVE_REGS_THUNK(func) \
- __PV_CALLEE_SAVE_REGS_THUNK(func, ".text")
+#define PV_CALLEE_SAVE_REGS_THUNK(func) \
+ __PV_CALLEE_SAVE_REGS_THUNK(func, \
+ ASM_TEXT_SECTION(__raw_callee_save_##func))
/* Get a reference to a callee-save function */
#define PV_CALLEE_SAVE(func) \
diff --git a/arch/x86/include/asm/qspinlock_paravirt.h b/arch/x86/include/asm/qspinlock_paravirt.h
index 159622ee0674..29558fff2453 100644
--- a/arch/x86/include/asm/qspinlock_paravirt.h
+++ b/arch/x86/include/asm/qspinlock_paravirt.h
@@ -34,7 +34,7 @@ PV_CALLEE_SAVE_REGS_THUNK(__pv_queued_spin_unlock_slowpath);
* rsi = lockval (second argument)
* rdx = internal variable (set to 0)
*/
-asm (".pushsection .text;"
+asm (ASM_PUSH_SECTION(__raw_callee_save___pv_queued_spin_unlock) ";"
".globl " PV_UNLOCK ";"
".type " PV_UNLOCK ", @function;"
".align 4,0x90;"
diff --git a/arch/x86/kernel/acpi/wakeup_32.S b/arch/x86/kernel/acpi/wakeup_32.S
index daf88f8143c5..4553c09bb891 100644
--- a/arch/x86/kernel/acpi/wakeup_32.S
+++ b/arch/x86/kernel/acpi/wakeup_32.S
@@ -7,9 +7,9 @@
# Copyright 2003, 2008 Pavel Machek <pavel@suse.cz
.code32
+SYM_CODE_START_SECTION(wakeup_pmode_return)
ALIGN
-SYM_CODE_START(wakeup_pmode_return)
movw $__KERNEL_DS, %ax
movw %ax, %ss
movw %ax, %fs
@@ -42,9 +42,9 @@ SYM_CODE_END(wakeup_pmode_return)
bogus_magic:
jmp bogus_magic
+SYM_TEXT_END_SECTION
-
-
+SYM_TEXT_SECTION(do_suspend_lowlevel)
save_registers:
sidt saved_idt
sldt saved_ldt
@@ -87,7 +87,7 @@ ret_point:
call restore_registers
call restore_processor_state
ret
-SYM_CODE_END(do_suspend_lowlevel)
+SYM_CODE_END_SECTION(do_suspend_lowlevel)
.data
ALIGN
@@ -98,4 +98,3 @@ saved_eip: .long 0
saved_idt: .long 0,0
saved_ldt: .long 0
saved_tss: .long 0
-
diff --git a/arch/x86/kernel/acpi/wakeup_64.S b/arch/x86/kernel/acpi/wakeup_64.S
index d5d8a352eafa..0a0e42e45d7a 100644
--- a/arch/x86/kernel/acpi/wakeup_64.S
+++ b/arch/x86/kernel/acpi/wakeup_64.S
@@ -16,7 +16,7 @@
/*
* Hooray, we are in Long 64-bit mode (but still running in low memory)
*/
-SYM_FUNC_START(wakeup_long64)
+SYM_FUNC_START_SECTION(wakeup_long64)
movq saved_magic, %rax
movq $0x123456789abcdef0, %rdx
cmpq %rdx, %rax
@@ -28,7 +28,7 @@ SYM_FUNC_START(wakeup_long64)
jmp 1b
2:
movw $__KERNEL_DS, %ax
- movw %ax, %ss
+ movw %ax, %ss
movw %ax, %ds
movw %ax, %es
movw %ax, %fs
@@ -43,9 +43,9 @@ SYM_FUNC_START(wakeup_long64)
movq saved_rip, %rax
ANNOTATE_RETPOLINE_SAFE
jmp *%rax
-SYM_FUNC_END(wakeup_long64)
+SYM_FUNC_END_SECTION(wakeup_long64)
-SYM_FUNC_START(do_suspend_lowlevel)
+SYM_FUNC_START_SECTION(do_suspend_lowlevel)
FRAME_BEGIN
subq $8, %rsp
xorl %eax, %eax
@@ -128,7 +128,7 @@ SYM_FUNC_START(do_suspend_lowlevel)
addq $8, %rsp
FRAME_END
jmp restore_processor_state
-SYM_FUNC_END(do_suspend_lowlevel)
+SYM_FUNC_END_SECTION(do_suspend_lowlevel)
STACK_FRAME_NON_STANDARD do_suspend_lowlevel
.data
diff --git a/arch/x86/kernel/ftrace_32.S b/arch/x86/kernel/ftrace_32.S
index e405fe1a8bf4..f79b2f436fde 100644
--- a/arch/x86/kernel/ftrace_32.S
+++ b/arch/x86/kernel/ftrace_32.S
@@ -18,13 +18,12 @@
# define MCOUNT_FRAME 0 /* using frame = false */
#endif
-SYM_FUNC_START(__fentry__)
+SYM_FUNC_START_SECTION(__fentry__)
ret
-SYM_FUNC_END(__fentry__)
+SYM_FUNC_END_SECTION(__fentry__)
EXPORT_SYMBOL(__fentry__)
-SYM_CODE_START(ftrace_caller)
-
+SYM_CODE_START_SECTION(ftrace_caller)
#ifdef CONFIG_FRAME_POINTER
/*
* Frame pointers are of ip followed by bp.
@@ -85,9 +84,9 @@ ftrace_graph_call:
/* This is weak to keep gas from relaxing the jumps */
SYM_INNER_LABEL_ALIGN(ftrace_stub, SYM_L_WEAK)
ret
-SYM_CODE_END(ftrace_caller)
+SYM_CODE_END_SECTION(ftrace_caller)
-SYM_CODE_START(ftrace_regs_caller)
+SYM_CODE_START_SECTION(ftrace_regs_caller)
/*
* We're here from an mcount/fentry CALL, and the stack frame looks like:
*
@@ -161,10 +160,10 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL)
popl %eax
jmp .Lftrace_ret
-SYM_CODE_END(ftrace_regs_caller)
+SYM_CODE_END_SECTION(ftrace_regs_caller)
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
-SYM_CODE_START(ftrace_graph_caller)
+SYM_CODE_START_SECTION(ftrace_graph_caller)
pushl %eax
pushl %ecx
pushl %edx
@@ -178,8 +177,9 @@ SYM_CODE_START(ftrace_graph_caller)
popl %ecx
popl %eax
ret
-SYM_CODE_END(ftrace_graph_caller)
+SYM_CODE_END_SECTION(ftrace_graph_caller)
+SYM_TEXT_SECTION(return_to_handler)
.globl return_to_handler
return_to_handler:
pushl %eax
@@ -190,4 +190,5 @@ return_to_handler:
popl %edx
popl %eax
JMP_NOSPEC ecx
+SYM_TEXT_END_SECTION
#endif
diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S
index 7a879901f103..703b6abc4948 100644
--- a/arch/x86/kernel/ftrace_64.S
+++ b/arch/x86/kernel/ftrace_64.S
@@ -131,12 +131,12 @@
#ifdef CONFIG_DYNAMIC_FTRACE
-SYM_FUNC_START(__fentry__)
+SYM_FUNC_START_SECTION(__fentry__)
retq
-SYM_FUNC_END(__fentry__)
+SYM_FUNC_END_SECTION(__fentry__)
EXPORT_SYMBOL(__fentry__)
-SYM_FUNC_START(ftrace_caller)
+SYM_FUNC_START_SECTION(ftrace_caller)
/* save_mcount_regs fills in first two parameters */
save_mcount_regs
@@ -171,9 +171,9 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)
SYM_INNER_LABEL(ftrace_caller_end, SYM_L_GLOBAL)
jmp ftrace_epilogue
-SYM_FUNC_END(ftrace_caller);
+SYM_FUNC_END_SECTION(ftrace_caller)
-SYM_FUNC_START(ftrace_epilogue)
+SYM_FUNC_START_SECTION(ftrace_epilogue)
/*
* This is weak to keep gas from relaxing the jumps.
* It is also used to copy the retq for trampolines.
@@ -181,9 +181,9 @@ SYM_FUNC_START(ftrace_epilogue)
SYM_INNER_LABEL_ALIGN(ftrace_stub, SYM_L_WEAK)
UNWIND_HINT_FUNC
retq
-SYM_FUNC_END(ftrace_epilogue)
+SYM_FUNC_END_SECTION(ftrace_epilogue)
-SYM_FUNC_START(ftrace_regs_caller)
+SYM_FUNC_START_SECTION(ftrace_regs_caller)
/* Save the current flags before any operations that can change them */
pushfq
@@ -274,12 +274,12 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL)
UNWIND_HINT_FUNC
jmp ftrace_epilogue
-SYM_FUNC_END(ftrace_regs_caller)
+SYM_FUNC_END_SECTION(ftrace_regs_caller)
#else /* ! CONFIG_DYNAMIC_FTRACE */
-SYM_FUNC_START(__fentry__)
+SYM_FUNC_START_SECTION(__fentry__)
cmpq $ftrace_stub, ftrace_trace_function
jnz trace
@@ -301,12 +301,12 @@ trace:
restore_mcount_regs
jmp ftrace_stub
-SYM_FUNC_END(__fentry__)
+SYM_FUNC_END_SECTION(__fentry__)
EXPORT_SYMBOL(__fentry__)
#endif /* CONFIG_DYNAMIC_FTRACE */
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
-SYM_FUNC_START(return_to_handler)
+SYM_FUNC_START_SECTION(return_to_handler)
subq $24, %rsp
/* Save the return values */
@@ -321,5 +321,5 @@ SYM_FUNC_START(return_to_handler)
movq (%rsp), %rax
addq $24, %rsp
JMP_NOSPEC rdi
-SYM_FUNC_END(return_to_handler)
+SYM_FUNC_END_SECTION(return_to_handler)
#endif
diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index fce99e249d61..f7b044ecd449 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -1018,7 +1018,7 @@ NOKPROBE_SYMBOL(kprobe_int3_handler);
* calls trampoline_handler() runs, which calls the kretprobe's handler.
*/
asm(
- ".text\n"
+ ASM_PUSH_SECTION(__kretprobe_trampoline) "\n"
".global __kretprobe_trampoline\n"
".type __kretprobe_trampoline, @function\n"
"__kretprobe_trampoline:\n"
@@ -1053,6 +1053,7 @@ asm(
#endif
" ret\n"
".size __kretprobe_trampoline, .-__kretprobe_trampoline\n"
+ ".popsection\n"
);
NOKPROBE_SYMBOL(__kretprobe_trampoline);
/*
diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index 59abbdad7729..6119c4d8807c 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -1020,7 +1020,7 @@ extern bool __raw_callee_save___kvm_vcpu_is_preempted(long);
* restoring to/from the stack.
*/
asm(
-".pushsection .text;"
+ASM_PUSH_SECTION(__raw_callee_save___kvm_vcpu_is_preempted) ";"
".global __raw_callee_save___kvm_vcpu_is_preempted;"
".type __raw_callee_save___kvm_vcpu_is_preempted, @function;"
"__raw_callee_save___kvm_vcpu_is_preempted:"
diff --git a/arch/x86/kernel/relocate_kernel_32.S b/arch/x86/kernel/relocate_kernel_32.S
index f469153eca8a..541f28254fa4 100644
--- a/arch/x86/kernel/relocate_kernel_32.S
+++ b/arch/x86/kernel/relocate_kernel_32.S
@@ -35,6 +35,7 @@
#define CP_PA_BACKUP_PAGES_MAP DATA(0x1c)
.text
+SYM_TEXT_SECTION(kexec_control_code)
SYM_CODE_START_NOALIGN(relocate_kernel)
/* Save the CPU context, used for jumping back */
@@ -276,3 +277,4 @@ SYM_CODE_END(swap_pages)
.globl kexec_control_code_size
.set kexec_control_code_size, . - relocate_kernel
+SYM_TEXT_END_SECTION
diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S
index c8fe74a28143..3f68fb738385 100644
--- a/arch/x86/kernel/relocate_kernel_64.S
+++ b/arch/x86/kernel/relocate_kernel_64.S
@@ -38,6 +38,7 @@
#define CP_PA_BACKUP_PAGES_MAP DATA(0x30)
.text
+SYM_TEXT_SECTION(kexec_control_code)
.align PAGE_SIZE
.code64
SYM_CODE_START_NOALIGN(relocate_kernel)
@@ -293,3 +294,4 @@ SYM_CODE_END(swap_pages)
.globl kexec_control_code_size
.set kexec_control_code_size, . - relocate_kernel
+SYM_TEXT_END_SECTION
diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index 28b1a4e57827..9000242f23b4 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -323,7 +323,7 @@ static int fastop(struct x86_emulate_ctxt *ctxt, fastop_t fop);
#define FOP_START(op) \
extern void em_##op(struct fastop *fake); \
- asm(".pushsection .text, \"ax\" \n\t" \
+ asm(ASM_PUSH_SECTION(em_##op) "\n\t" \
".global em_" #op " \n\t" \
".align " __stringify(FASTOP_SIZE) " \n\t" \
"em_" #op ":\n\t"
diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S
index 3a6461694fc2..06f4f77d9ca5 100644
--- a/arch/x86/kvm/vmx/vmenter.S
+++ b/arch/x86/kvm/vmx/vmenter.S
@@ -246,7 +246,7 @@ SYM_FUNC_END(__vmx_vcpu_run)
* Save and restore volatile registers across a call to vmread_error(). Note,
* all parameters are passed on the stack.
*/
-SYM_FUNC_START(vmread_error_trampoline)
+SYM_FUNC_START_SECTION(vmread_error_trampoline)
push %_ASM_BP
mov %_ASM_SP, %_ASM_BP
@@ -294,9 +294,9 @@ SYM_FUNC_START(vmread_error_trampoline)
pop %_ASM_BP
ret
-SYM_FUNC_END(vmread_error_trampoline)
+SYM_FUNC_END_SECTION(vmread_error_trampoline)
-SYM_FUNC_START(vmx_do_interrupt_nmi_irqoff)
+SYM_FUNC_START_SECTION(vmx_do_interrupt_nmi_irqoff)
/*
* Unconditionally create a stack frame, getting the correct RSP on the
* stack (for x86-64) would take two instructions anyways, and RBP can
@@ -327,4 +327,4 @@ SYM_FUNC_START(vmx_do_interrupt_nmi_irqoff)
mov %_ASM_BP, %_ASM_SP
pop %_ASM_BP
ret
-SYM_FUNC_END(vmx_do_interrupt_nmi_irqoff)
+SYM_FUNC_END_SECTION(vmx_do_interrupt_nmi_irqoff)
diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S
index c4c7dd115953..64e0e5c3528d 100644
--- a/arch/x86/lib/clear_page_64.S
+++ b/arch/x86/lib/clear_page_64.S
@@ -13,15 +13,15 @@
* Zero a page.
* %rdi - page
*/
-SYM_FUNC_START(clear_page_rep)
+SYM_FUNC_START_SECTION(clear_page_rep)
movl $4096/8,%ecx
xorl %eax,%eax
rep stosq
ret
-SYM_FUNC_END(clear_page_rep)
+SYM_FUNC_END_SECTION(clear_page_rep)
EXPORT_SYMBOL_GPL(clear_page_rep)
-SYM_FUNC_START(clear_page_orig)
+SYM_FUNC_START_SECTION(clear_page_orig)
xorl %eax,%eax
movl $4096/64,%ecx
.p2align 4
@@ -40,13 +40,13 @@ SYM_FUNC_START(clear_page_orig)
jnz .Lloop
nop
ret
-SYM_FUNC_END(clear_page_orig)
+SYM_FUNC_END_SECTION(clear_page_orig)
EXPORT_SYMBOL_GPL(clear_page_orig)
-SYM_FUNC_START(clear_page_erms)
+SYM_FUNC_START_SECTION(clear_page_erms)
movl $4096,%ecx
xorl %eax,%eax
rep stosb
ret
-SYM_FUNC_END(clear_page_erms)
+SYM_FUNC_END_SECTION(clear_page_erms)
EXPORT_SYMBOL_GPL(clear_page_erms)
diff --git a/arch/x86/lib/cmpxchg16b_emu.S b/arch/x86/lib/cmpxchg16b_emu.S
index 3542502faa3b..0ee7b541f39e 100644
--- a/arch/x86/lib/cmpxchg16b_emu.S
+++ b/arch/x86/lib/cmpxchg16b_emu.S
@@ -13,7 +13,7 @@
* %rcx : high 64 bits of new value
* %al : Operation successful
*/
-SYM_FUNC_START(this_cpu_cmpxchg16b_emu)
+SYM_FUNC_START_SECTION(this_cpu_cmpxchg16b_emu)
#
# Emulate 'cmpxchg16b %gs:(%rsi)' except we return the result in %al not
@@ -44,4 +44,4 @@ SYM_FUNC_START(this_cpu_cmpxchg16b_emu)
xor %al,%al
ret
-SYM_FUNC_END(this_cpu_cmpxchg16b_emu)
+SYM_FUNC_END_SECTION(this_cpu_cmpxchg16b_emu)
diff --git a/arch/x86/lib/copy_mc_64.S b/arch/x86/lib/copy_mc_64.S
index 7334055157ba..aeea772b5d5f 100644
--- a/arch/x86/lib/copy_mc_64.S
+++ b/arch/x86/lib/copy_mc_64.S
@@ -17,7 +17,7 @@
* boundary. The non-fragile version is equivalent to memcpy()
* regardless of CPU machine-check-recovery capability.
*/
-SYM_FUNC_START(copy_mc_fragile)
+SYM_FUNC_START_SECTION(copy_mc_fragile)
cmpl $8, %edx
/* Less than 8 bytes? Go to byte copy loop */
jb .L_no_whole_words
@@ -78,7 +78,7 @@ SYM_FUNC_START(copy_mc_fragile)
xorl %eax, %eax
.L_done:
ret
-SYM_FUNC_END(copy_mc_fragile)
+SYM_FUNC_END_SECTION(copy_mc_fragile)
.section .fixup, "ax"
/*
@@ -125,7 +125,7 @@ SYM_FUNC_END(copy_mc_fragile)
* machine check recovery support this version should be no slower than
* standard memcpy.
*/
-SYM_FUNC_START(copy_mc_enhanced_fast_string)
+SYM_FUNC_START_SECTION(copy_mc_enhanced_fast_string)
movq %rdi, %rax
movq %rdx, %rcx
.L_copy:
@@ -133,7 +133,7 @@ SYM_FUNC_START(copy_mc_enhanced_fast_string)
/* Copy successful. Return zero */
xorl %eax, %eax
ret
-SYM_FUNC_END(copy_mc_enhanced_fast_string)
+SYM_FUNC_END_SECTION(copy_mc_enhanced_fast_string)
.section .fixup, "ax"
.E_copy:
diff --git a/arch/x86/lib/copy_page_64.S b/arch/x86/lib/copy_page_64.S
index db4b4f9197c7..f212b9dfd469 100644
--- a/arch/x86/lib/copy_page_64.S
+++ b/arch/x86/lib/copy_page_64.S
@@ -12,6 +12,8 @@
* copy unless the CPU indicates X86_FEATURE_REP_GOOD. Could vary the
* prefetch distance based on SMP/UP.
*/
+
+SYM_TEXT_SECTION(copy_page)
ALIGN
SYM_FUNC_START(copy_page)
ALTERNATIVE "jmp copy_page_regs", "", X86_FEATURE_REP_GOOD
@@ -19,9 +21,10 @@ SYM_FUNC_START(copy_page)
rep movsq
ret
SYM_FUNC_END(copy_page)
+SYM_TEXT_END_SECTION
EXPORT_SYMBOL(copy_page)
-SYM_FUNC_START_LOCAL(copy_page_regs)
+SYM_FUNC_START_LOCAL_SECTION(copy_page_regs)
subq $2*8, %rsp
movq %rbx, (%rsp)
movq %r12, 1*8(%rsp)
@@ -86,4 +89,4 @@ SYM_FUNC_START_LOCAL(copy_page_regs)
movq 1*8(%rsp), %r12
addq $2*8, %rsp
ret
-SYM_FUNC_END(copy_page_regs)
+SYM_FUNC_END_SECTION(copy_page_regs)
diff --git a/arch/x86/lib/copy_user_64.S b/arch/x86/lib/copy_user_64.S
index 2797e630b9b1..dbd9382e54d8 100644
--- a/arch/x86/lib/copy_user_64.S
+++ b/arch/x86/lib/copy_user_64.S
@@ -54,7 +54,7 @@
* Output:
* eax uncopied bytes or 0 if successful.
*/
-SYM_FUNC_START(copy_user_generic_unrolled)
+SYM_FUNC_START_SECTION(copy_user_generic_unrolled)
ASM_STAC
cmpl $8,%edx
jb 20f /* less then 8 bytes, go to byte copy loop */
@@ -137,7 +137,7 @@ SYM_FUNC_START(copy_user_generic_unrolled)
_ASM_EXTABLE_CPY(19b, 40b)
_ASM_EXTABLE_CPY(21b, 50b)
_ASM_EXTABLE_CPY(22b, 50b)
-SYM_FUNC_END(copy_user_generic_unrolled)
+SYM_FUNC_END_SECTION(copy_user_generic_unrolled)
EXPORT_SYMBOL(copy_user_generic_unrolled)
/* Some CPUs run faster using the string copy instructions.
@@ -158,7 +158,7 @@ EXPORT_SYMBOL(copy_user_generic_unrolled)
* Output:
* eax uncopied bytes or 0 if successful.
*/
-SYM_FUNC_START(copy_user_generic_string)
+SYM_FUNC_START_SECTION(copy_user_generic_string)
ASM_STAC
cmpl $8,%edx
jb 2f /* less than 8 bytes, go to byte copy loop */
@@ -183,7 +183,7 @@ SYM_FUNC_START(copy_user_generic_string)
_ASM_EXTABLE_CPY(1b, 11b)
_ASM_EXTABLE_CPY(3b, 12b)
-SYM_FUNC_END(copy_user_generic_string)
+SYM_FUNC_END_SECTION(copy_user_generic_string)
EXPORT_SYMBOL(copy_user_generic_string)
/*
@@ -198,7 +198,7 @@ EXPORT_SYMBOL(copy_user_generic_string)
* Output:
* eax uncopied bytes or 0 if successful.
*/
-SYM_FUNC_START(copy_user_enhanced_fast_string)
+SYM_FUNC_START_SECTION(copy_user_enhanced_fast_string)
ASM_STAC
cmpl $64,%edx
jb .L_copy_short_string /* less then 64 bytes, avoid the costly 'rep' */
@@ -215,7 +215,7 @@ SYM_FUNC_START(copy_user_enhanced_fast_string)
.previous
_ASM_EXTABLE_CPY(1b, 12b)
-SYM_FUNC_END(copy_user_enhanced_fast_string)
+SYM_FUNC_END_SECTION(copy_user_enhanced_fast_string)
EXPORT_SYMBOL(copy_user_enhanced_fast_string)
/*
@@ -232,6 +232,7 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string)
* Output:
* eax uncopied bytes or 0 if successful.
*/
+SYM_TEXT_SECTION(copy_user_handle_tail)
SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail)
movl %edx,%ecx
1: rep movsb
@@ -241,6 +242,7 @@ SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail)
_ASM_EXTABLE_CPY(1b, 2b)
SYM_CODE_END(.Lcopy_user_handle_tail)
+SYM_TEXT_END_SECTION
/*
* copy_user_nocache - Uncached memory copy with exception handling
@@ -251,7 +253,7 @@ SYM_CODE_END(.Lcopy_user_handle_tail)
* - Require 8-byte alignment when size is 8 bytes or larger.
* - Require 4-byte alignment when size is 4 bytes.
*/
-SYM_FUNC_START(__copy_user_nocache)
+SYM_FUNC_START_SECTION(__copy_user_nocache)
ASM_STAC
/* If size is less than 8 bytes, go to 4-byte copy */
@@ -390,5 +392,5 @@ SYM_FUNC_START(__copy_user_nocache)
_ASM_EXTABLE_CPY(31b, .L_fixup_4b_copy)
_ASM_EXTABLE_CPY(40b, .L_fixup_1b_copy)
_ASM_EXTABLE_CPY(41b, .L_fixup_1b_copy)
-SYM_FUNC_END(__copy_user_nocache)
+SYM_FUNC_END_SECTION(__copy_user_nocache)
EXPORT_SYMBOL(__copy_user_nocache)
diff --git a/arch/x86/lib/csum-copy_64.S b/arch/x86/lib/csum-copy_64.S
index 1fbd8ee9642d..051e043df32a 100644
--- a/arch/x86/lib/csum-copy_64.S
+++ b/arch/x86/lib/csum-copy_64.S
@@ -36,7 +36,7 @@
_ASM_EXTABLE_UA(20b, .Lfault)
.endm
-SYM_FUNC_START(csum_partial_copy_generic)
+SYM_FUNC_START_SECTION(csum_partial_copy_generic)
subq $5*8, %rsp
movq %rbx, 0*8(%rsp)
movq %r12, 1*8(%rsp)
@@ -253,4 +253,4 @@ SYM_FUNC_START(csum_partial_copy_generic)
.Lfault:
xorl %eax, %eax
jmp .Lout
-SYM_FUNC_END(csum_partial_copy_generic)
+SYM_FUNC_END_SECTION(csum_partial_copy_generic)
diff --git a/arch/x86/lib/error-inject.c b/arch/x86/lib/error-inject.c
index be5b5fb1598b..eddc8ab73a98 100644
--- a/arch/x86/lib/error-inject.c
+++ b/arch/x86/lib/error-inject.c
@@ -6,12 +6,13 @@
asmlinkage void just_return_func(void);
asm(
- ".text\n"
+ ASM_PUSH_SECTION(just_return_func) "\n"
".type just_return_func, @function\n"
".globl just_return_func\n"
"just_return_func:\n"
" ret\n"
".size just_return_func, .-just_return_func\n"
+ ".popsection\n"
);
void override_function_with_return(struct pt_regs *regs)
diff --git a/arch/x86/lib/getuser.S b/arch/x86/lib/getuser.S
index fa1bc2104b32..d8a98dbcb310 100644
--- a/arch/x86/lib/getuser.S
+++ b/arch/x86/lib/getuser.S
@@ -47,7 +47,7 @@
#endif
.text
-SYM_FUNC_START(__get_user_1)
+SYM_FUNC_START_SECTION(__get_user_1)
LOAD_TASK_SIZE_MINUS_N(0)
cmp %_ASM_DX,%_ASM_AX
jae bad_get_user
@@ -58,10 +58,10 @@ SYM_FUNC_START(__get_user_1)
xor %eax,%eax
ASM_CLAC
ret
-SYM_FUNC_END(__get_user_1)
+SYM_FUNC_END_SECTION(__get_user_1)
EXPORT_SYMBOL(__get_user_1)
-SYM_FUNC_START(__get_user_2)
+SYM_FUNC_START_SECTION(__get_user_2)
LOAD_TASK_SIZE_MINUS_N(1)
cmp %_ASM_DX,%_ASM_AX
jae bad_get_user
@@ -72,10 +72,10 @@ SYM_FUNC_START(__get_user_2)
xor %eax,%eax
ASM_CLAC
ret
-SYM_FUNC_END(__get_user_2)
+SYM_FUNC_END_SECTION(__get_user_2)
EXPORT_SYMBOL(__get_user_2)
-SYM_FUNC_START(__get_user_4)
+SYM_FUNC_START_SECTION(__get_user_4)
LOAD_TASK_SIZE_MINUS_N(3)
cmp %_ASM_DX,%_ASM_AX
jae bad_get_user
@@ -86,10 +86,10 @@ SYM_FUNC_START(__get_user_4)
xor %eax,%eax
ASM_CLAC
ret
-SYM_FUNC_END(__get_user_4)
+SYM_FUNC_END_SECTION(__get_user_4)
EXPORT_SYMBOL(__get_user_4)
-SYM_FUNC_START(__get_user_8)
+SYM_FUNC_START_SECTION(__get_user_8)
#ifdef CONFIG_X86_64
LOAD_TASK_SIZE_MINUS_N(7)
cmp %_ASM_DX,%_ASM_AX
@@ -114,41 +114,41 @@ SYM_FUNC_START(__get_user_8)
ASM_CLAC
ret
#endif
-SYM_FUNC_END(__get_user_8)
+SYM_FUNC_END_SECTION(__get_user_8)
EXPORT_SYMBOL(__get_user_8)
/* .. and the same for __get_user, just without the range checks */
-SYM_FUNC_START(__get_user_nocheck_1)
+SYM_FUNC_START_SECTION(__get_user_nocheck_1)
ASM_STAC
ASM_BARRIER_NOSPEC
6: movzbl (%_ASM_AX),%edx
xor %eax,%eax
ASM_CLAC
ret
-SYM_FUNC_END(__get_user_nocheck_1)
+SYM_FUNC_END_SECTION(__get_user_nocheck_1)
EXPORT_SYMBOL(__get_user_nocheck_1)
-SYM_FUNC_START(__get_user_nocheck_2)
+SYM_FUNC_START_SECTION(__get_user_nocheck_2)
ASM_STAC
ASM_BARRIER_NOSPEC
7: movzwl (%_ASM_AX),%edx
xor %eax,%eax
ASM_CLAC
ret
-SYM_FUNC_END(__get_user_nocheck_2)
+SYM_FUNC_END_SECTION(__get_user_nocheck_2)
EXPORT_SYMBOL(__get_user_nocheck_2)
-SYM_FUNC_START(__get_user_nocheck_4)
+SYM_FUNC_START_SECTION(__get_user_nocheck_4)
ASM_STAC
ASM_BARRIER_NOSPEC
8: movl (%_ASM_AX),%edx
xor %eax,%eax
ASM_CLAC
ret
-SYM_FUNC_END(__get_user_nocheck_4)
+SYM_FUNC_END_SECTION(__get_user_nocheck_4)
EXPORT_SYMBOL(__get_user_nocheck_4)
-SYM_FUNC_START(__get_user_nocheck_8)
+SYM_FUNC_START_SECTION(__get_user_nocheck_8)
ASM_STAC
ASM_BARRIER_NOSPEC
#ifdef CONFIG_X86_64
@@ -160,10 +160,10 @@ SYM_FUNC_START(__get_user_nocheck_8)
xor %eax,%eax
ASM_CLAC
ret
-SYM_FUNC_END(__get_user_nocheck_8)
+SYM_FUNC_END_SECTION(__get_user_nocheck_8)
EXPORT_SYMBOL(__get_user_nocheck_8)
-
+SYM_TEXT_SECTION(bad_get_user)
SYM_CODE_START_LOCAL(.Lbad_get_user_clac)
ASM_CLAC
bad_get_user:
@@ -171,8 +171,10 @@ bad_get_user:
mov $(-EFAULT),%_ASM_AX
ret
SYM_CODE_END(.Lbad_get_user_clac)
+SYM_TEXT_END_SECTION
#ifdef CONFIG_X86_32
+SYM_TEXT_SECTION(bad_get_user_8)
SYM_CODE_START_LOCAL(.Lbad_get_user_8_clac)
ASM_CLAC
bad_get_user_8:
@@ -181,6 +183,7 @@ bad_get_user_8:
mov $(-EFAULT),%_ASM_AX
ret
SYM_CODE_END(.Lbad_get_user_8_clac)
+SYM_TEXT_END_SECTION
#endif
/* get_user */
diff --git a/arch/x86/lib/hweight.S b/arch/x86/lib/hweight.S
index dbf8cc97b7f5..679afb4ccee0 100644
--- a/arch/x86/lib/hweight.S
+++ b/arch/x86/lib/hweight.S
@@ -8,8 +8,7 @@
* unsigned int __sw_hweight32(unsigned int w)
* %rdi: w
*/
-SYM_FUNC_START(__sw_hweight32)
-
+SYM_FUNC_START_SECTION(__sw_hweight32)
#ifdef CONFIG_X86_64
movl %edi, %eax # w
#endif
@@ -33,10 +32,10 @@ SYM_FUNC_START(__sw_hweight32)
shrl $24, %eax # w = w_tmp >> 24
__ASM_SIZE(pop,) %__ASM_REG(dx)
ret
-SYM_FUNC_END(__sw_hweight32)
+SYM_FUNC_END_SECTION(__sw_hweight32)
EXPORT_SYMBOL(__sw_hweight32)
-SYM_FUNC_START(__sw_hweight64)
+SYM_FUNC_START_SECTION(__sw_hweight64)
#ifdef CONFIG_X86_64
pushq %rdi
pushq %rdx
@@ -79,5 +78,5 @@ SYM_FUNC_START(__sw_hweight64)
popl %ecx
ret
#endif
-SYM_FUNC_END(__sw_hweight64)
+SYM_FUNC_END_SECTION(__sw_hweight64)
EXPORT_SYMBOL(__sw_hweight64)
diff --git a/arch/x86/lib/iomap_copy_64.S b/arch/x86/lib/iomap_copy_64.S
index cb5a1964506b..0b96616a6f13 100644
--- a/arch/x86/lib/iomap_copy_64.S
+++ b/arch/x86/lib/iomap_copy_64.S
@@ -8,8 +8,8 @@
/*
* override generic version in lib/iomap_copy.c
*/
-SYM_FUNC_START(__iowrite32_copy)
+SYM_FUNC_START_SECTION(__iowrite32_copy)
movl %edx,%ecx
rep movsd
ret
-SYM_FUNC_END(__iowrite32_copy)
+SYM_FUNC_END_SECTION(__iowrite32_copy)
diff --git a/arch/x86/lib/memmove_64.S b/arch/x86/lib/memmove_64.S
index 64801010d312..ae41caf90afd 100644
--- a/arch/x86/lib/memmove_64.S
+++ b/arch/x86/lib/memmove_64.S
@@ -24,7 +24,7 @@
* Output:
* rax: dest
*/
-SYM_FUNC_START_WEAK(memmove)
+SYM_FUNC_START_WEAK_SECTION(memmove)
SYM_FUNC_START(__memmove)
mov %rdi, %rax
@@ -207,6 +207,6 @@ SYM_FUNC_START(__memmove)
13:
retq
SYM_FUNC_END(__memmove)
-SYM_FUNC_END_ALIAS(memmove)
+SYM_FUNC_END_ALIAS_SECTION(memmove)
EXPORT_SYMBOL(__memmove)
EXPORT_SYMBOL(memmove)
diff --git a/arch/x86/lib/memset_64.S b/arch/x86/lib/memset_64.S
index 9827ae267f96..466e943d363f 100644
--- a/arch/x86/lib/memset_64.S
+++ b/arch/x86/lib/memset_64.S
@@ -17,7 +17,7 @@
*
* rax original destination
*/
-SYM_FUNC_START_WEAK(memset)
+SYM_FUNC_START_WEAK_SECTION(memset)
SYM_FUNC_START(__memset)
/*
* Some CPUs support enhanced REP MOVSB/STOSB feature. It is recommended
@@ -42,7 +42,7 @@ SYM_FUNC_START(__memset)
movq %r9,%rax
ret
SYM_FUNC_END(__memset)
-SYM_FUNC_END_ALIAS(memset)
+SYM_FUNC_END_ALIAS_SECTION(memset)
EXPORT_SYMBOL(memset)
EXPORT_SYMBOL(__memset)
@@ -57,16 +57,16 @@ EXPORT_SYMBOL(__memset)
*
* rax original destination
*/
-SYM_FUNC_START_LOCAL(memset_erms)
+SYM_FUNC_START_LOCAL_SECTION(memset_erms)
movq %rdi,%r9
movb %sil,%al
movq %rdx,%rcx
rep stosb
movq %r9,%rax
ret
-SYM_FUNC_END(memset_erms)
+SYM_FUNC_END_SECTION(memset_erms)
-SYM_FUNC_START_LOCAL(memset_orig)
+SYM_FUNC_START_LOCAL_SECTION(memset_orig)
movq %rdi,%r10
/* expand byte value */
@@ -137,4 +137,4 @@ SYM_FUNC_START_LOCAL(memset_orig)
subq %r8,%rdx
jmp .Lafter_bad_alignment
.Lfinal:
-SYM_FUNC_END(memset_orig)
+SYM_FUNC_END_SECTION(memset_orig)
diff --git a/arch/x86/lib/msr-reg.S b/arch/x86/lib/msr-reg.S
index a2b9caa5274c..7e17eba68445 100644
--- a/arch/x86/lib/msr-reg.S
+++ b/arch/x86/lib/msr-reg.S
@@ -12,7 +12,7 @@
*
*/
.macro op_safe_regs op
-SYM_FUNC_START(\op\()_safe_regs)
+SYM_FUNC_START_SECTION(\op\()_safe_regs)
pushq %rbx
pushq %r12
movq %rdi, %r10 /* Save pointer */
@@ -41,13 +41,13 @@ SYM_FUNC_START(\op\()_safe_regs)
jmp 2b
_ASM_EXTABLE(1b, 3b)
-SYM_FUNC_END(\op\()_safe_regs)
+SYM_FUNC_END_SECTION(\op\()_safe_regs)
.endm
#else /* X86_32 */
.macro op_safe_regs op
-SYM_FUNC_START(\op\()_safe_regs)
+SYM_FUNC_START_SECTION(\op\()_safe_regs)
pushl %ebx
pushl %ebp
pushl %esi
@@ -83,7 +83,7 @@ SYM_FUNC_START(\op\()_safe_regs)
jmp 2b
_ASM_EXTABLE(1b, 3b)
-SYM_FUNC_END(\op\()_safe_regs)
+SYM_FUNC_END_SECTION(\op\()_safe_regs)
.endm
#endif
diff --git a/arch/x86/lib/putuser.S b/arch/x86/lib/putuser.S
index 0ea344c5ea43..2f0281d57ef7 100644
--- a/arch/x86/lib/putuser.S
+++ b/arch/x86/lib/putuser.S
@@ -43,7 +43,7 @@
#endif
.text
-SYM_FUNC_START(__put_user_1)
+SYM_FUNC_START_SECTION(__put_user_1)
LOAD_TASK_SIZE_MINUS_N(0)
cmp %_ASM_BX,%_ASM_CX
jae .Lbad_put_user
@@ -53,11 +53,11 @@ SYM_INNER_LABEL(__put_user_nocheck_1, SYM_L_GLOBAL)
xor %ecx,%ecx
ASM_CLAC
ret
-SYM_FUNC_END(__put_user_1)
+SYM_FUNC_END_SECTION(__put_user_1)
EXPORT_SYMBOL(__put_user_1)
EXPORT_SYMBOL(__put_user_nocheck_1)
-SYM_FUNC_START(__put_user_2)
+SYM_FUNC_START_SECTION(__put_user_2)
LOAD_TASK_SIZE_MINUS_N(1)
cmp %_ASM_BX,%_ASM_CX
jae .Lbad_put_user
@@ -67,11 +67,11 @@ SYM_INNER_LABEL(__put_user_nocheck_2, SYM_L_GLOBAL)
xor %ecx,%ecx
ASM_CLAC
ret
-SYM_FUNC_END(__put_user_2)
+SYM_FUNC_END_SECTION(__put_user_2)
EXPORT_SYMBOL(__put_user_2)
EXPORT_SYMBOL(__put_user_nocheck_2)
-SYM_FUNC_START(__put_user_4)
+SYM_FUNC_START_SECTION(__put_user_4)
LOAD_TASK_SIZE_MINUS_N(3)
cmp %_ASM_BX,%_ASM_CX
jae .Lbad_put_user
@@ -81,11 +81,11 @@ SYM_INNER_LABEL(__put_user_nocheck_4, SYM_L_GLOBAL)
xor %ecx,%ecx
ASM_CLAC
ret
-SYM_FUNC_END(__put_user_4)
+SYM_FUNC_END_SECTION(__put_user_4)
EXPORT_SYMBOL(__put_user_4)
EXPORT_SYMBOL(__put_user_nocheck_4)
-SYM_FUNC_START(__put_user_8)
+SYM_FUNC_START_SECTION(__put_user_8)
LOAD_TASK_SIZE_MINUS_N(7)
cmp %_ASM_BX,%_ASM_CX
jae .Lbad_put_user
@@ -98,16 +98,18 @@ SYM_INNER_LABEL(__put_user_nocheck_8, SYM_L_GLOBAL)
xor %ecx,%ecx
ASM_CLAC
RET
-SYM_FUNC_END(__put_user_8)
+SYM_FUNC_END_SECTION(__put_user_8)
EXPORT_SYMBOL(__put_user_8)
EXPORT_SYMBOL(__put_user_nocheck_8)
+SYM_TEXT_SECTION(bad_put_user)
SYM_CODE_START_LOCAL(.Lbad_put_user_clac)
ASM_CLAC
.Lbad_put_user:
movl $-EFAULT,%ecx
RET
SYM_CODE_END(.Lbad_put_user_clac)
+SYM_TEXT_END_SECTION
_ASM_EXTABLE_UA(1b, .Lbad_put_user_clac)
_ASM_EXTABLE_UA(2b, .Lbad_put_user_clac)
diff --git a/arch/x86/mm/mem_encrypt_boot.S b/arch/x86/mm/mem_encrypt_boot.S
index 17d292b7072f..58ff3fd7f65c 100644
--- a/arch/x86/mm/mem_encrypt_boot.S
+++ b/arch/x86/mm/mem_encrypt_boot.S
@@ -16,7 +16,7 @@
.text
.code64
-SYM_FUNC_START(sme_encrypt_execute)
+SYM_FUNC_START_SECTION(sme_encrypt_execute)
/*
* Entry parameters:
@@ -66,9 +66,9 @@ SYM_FUNC_START(sme_encrypt_execute)
pop %rbp
ret
-SYM_FUNC_END(sme_encrypt_execute)
+SYM_FUNC_END_SECTION(sme_encrypt_execute)
-SYM_FUNC_START(__enc_copy)
+SYM_FUNC_START_SECTION(__enc_copy)
/*
* Routine used to encrypt memory in place.
* This routine must be run outside of the kernel proper since
@@ -153,4 +153,4 @@ SYM_FUNC_START(__enc_copy)
ret
.L__enc_copy_end:
-SYM_FUNC_END(__enc_copy)
+SYM_FUNC_END_SECTION(__enc_copy)
diff --git a/arch/x86/platform/efi/efi_stub_64.S b/arch/x86/platform/efi/efi_stub_64.S
index 90380a17ab23..8841e757e11b 100644
--- a/arch/x86/platform/efi/efi_stub_64.S
+++ b/arch/x86/platform/efi/efi_stub_64.S
@@ -10,7 +10,7 @@
#include <linux/linkage.h>
#include <asm/nospec-branch.h>
-SYM_FUNC_START(__efi_call)
+SYM_FUNC_START_SECTION(__efi_call)
pushq %rbp
movq %rsp, %rbp
and $~0xf, %rsp
@@ -24,4 +24,4 @@ SYM_FUNC_START(__efi_call)
CALL_NOSPEC rdi
leave
ret
-SYM_FUNC_END(__efi_call)
+SYM_FUNC_END_SECTION(__efi_call)
diff --git a/arch/x86/platform/efi/efi_thunk_64.S b/arch/x86/platform/efi/efi_thunk_64.S
index fd3dd1708eba..a9d0f5be37ac 100644
--- a/arch/x86/platform/efi/efi_thunk_64.S
+++ b/arch/x86/platform/efi/efi_thunk_64.S
@@ -25,7 +25,7 @@
.text
.code64
-SYM_CODE_START(__efi64_thunk)
+SYM_CODE_START_SECTION(__efi64_thunk)
push %rbp
push %rbx
@@ -69,7 +69,7 @@ SYM_CODE_START(__efi64_thunk)
2: pushl $__KERNEL_CS
pushl %ebp
lret
-SYM_CODE_END(__efi64_thunk)
+SYM_CODE_END_SECTION(__efi64_thunk)
.bss
.balign 8
diff --git a/arch/x86/power/hibernate_asm_32.S b/arch/x86/power/hibernate_asm_32.S
index 8786653ad3c0..37c5df83b629 100644
--- a/arch/x86/power/hibernate_asm_32.S
+++ b/arch/x86/power/hibernate_asm_32.S
@@ -16,7 +16,7 @@
.text
-SYM_FUNC_START(swsusp_arch_suspend)
+SYM_FUNC_START_SECTION(swsusp_arch_suspend)
movl %esp, saved_context_esp
movl %ebx, saved_context_ebx
movl %ebp, saved_context_ebp
@@ -33,9 +33,9 @@ SYM_FUNC_START(swsusp_arch_suspend)
call swsusp_save
FRAME_END
ret
-SYM_FUNC_END(swsusp_arch_suspend)
+SYM_FUNC_END_SECTION(swsusp_arch_suspend)
-SYM_CODE_START(restore_image)
+SYM_CODE_START_SECTION(restore_image)
/* prepare to jump to the image kernel */
movl restore_jump_address, %ebx
movl restore_cr3, %ebp
@@ -45,10 +45,10 @@ SYM_CODE_START(restore_image)
/* jump to relocated restore code */
movl relocated_restore_code, %eax
jmpl *%eax
-SYM_CODE_END(restore_image)
+SYM_CODE_END_SECTION(restore_image)
/* code below has been relocated to a safe page */
-SYM_CODE_START(core_restore_code)
+SYM_CODE_START_SECTION(core_restore_code)
movl temp_pgt, %eax
movl %eax, %cr3
@@ -78,8 +78,9 @@ copy_loop:
done:
jmpl *%ebx
-SYM_CODE_END(core_restore_code)
+SYM_CODE_END_SECTION(core_restore_code)
+SYM_TEXT_SECTION(restore_registers)
/* code below belongs to the image kernel */
.align PAGE_SIZE
SYM_FUNC_START(restore_registers)
@@ -110,3 +111,4 @@ SYM_FUNC_START(restore_registers)
ret
SYM_FUNC_END(restore_registers)
+SYM_TEXT_END_SECTION
diff --git a/arch/x86/power/hibernate_asm_64.S b/arch/x86/power/hibernate_asm_64.S
index d9bed596d849..1318c660215b 100644
--- a/arch/x86/power/hibernate_asm_64.S
+++ b/arch/x86/power/hibernate_asm_64.S
@@ -23,6 +23,7 @@
#include <asm/frame.h>
#include <asm/nospec-branch.h>
+SYM_TEXT_SECTION(restore_registers)
/* code below belongs to the image kernel */
.align PAGE_SIZE
SYM_FUNC_START(restore_registers)
@@ -68,8 +69,9 @@ SYM_FUNC_START(restore_registers)
ret
SYM_FUNC_END(restore_registers)
+SYM_TEXT_END_SECTION
-SYM_FUNC_START(swsusp_arch_suspend)
+SYM_FUNC_START_SECTION(swsusp_arch_suspend)
movq $saved_context, %rax
movq %rsp, pt_regs_sp(%rax)
movq %rbp, pt_regs_bp(%rax)
@@ -97,9 +99,9 @@ SYM_FUNC_START(swsusp_arch_suspend)
call swsusp_save
FRAME_END
ret
-SYM_FUNC_END(swsusp_arch_suspend)
+SYM_FUNC_END_SECTION(swsusp_arch_suspend)
-SYM_FUNC_START(restore_image)
+SYM_FUNC_START_SECTION(restore_image)
/* prepare to jump to the image kernel */
movq restore_jump_address(%rip), %r8
movq restore_cr3(%rip), %r9
@@ -115,10 +117,10 @@ SYM_FUNC_START(restore_image)
movq relocated_restore_code(%rip), %rcx
ANNOTATE_RETPOLINE_SAFE
jmpq *%rcx
-SYM_FUNC_END(restore_image)
+SYM_FUNC_END_SECTION(restore_image)
/* code below has been relocated to a safe page */
-SYM_FUNC_START(core_restore_code)
+SYM_FUNC_START_SECTION(core_restore_code)
/* switch to temporary page tables */
movq %rax, %cr3
/* flush TLB */
@@ -147,4 +149,4 @@ SYM_FUNC_START(core_restore_code)
/* jump to the restore_registers address from the image header */
ANNOTATE_RETPOLINE_SAFE
jmpq *%r8
-SYM_FUNC_END(core_restore_code)
+SYM_FUNC_END_SECTION(core_restore_code)
diff --git a/arch/x86/xen/xen-asm.S b/arch/x86/xen/xen-asm.S
index 220dd9678494..e123e8d4babc 100644
--- a/arch/x86/xen/xen-asm.S
+++ b/arch/x86/xen/xen-asm.S
@@ -119,12 +119,12 @@ SYM_FUNC_END(xen_read_cr2_direct);
.popsection
.macro xen_pv_trap name
-SYM_CODE_START(xen_\name)
+SYM_CODE_START_SECTION(xen_\name)
UNWIND_HINT_EMPTY
pop %rcx
pop %r11
jmp \name
-SYM_CODE_END(xen_\name)
+SYM_CODE_END_SECTION(xen_\name)
_ASM_NOKPROBE(xen_\name)
.endm
@@ -186,11 +186,11 @@ hypercall_iret = hypercall_page + __HYPERVISOR_iret * 32
* r11 }<-- pushed by hypercall page
* rsp->rax }
*/
-SYM_CODE_START(xen_iret)
+SYM_CODE_START_SECTION(xen_iret)
UNWIND_HINT_EMPTY
pushq $0
jmp hypercall_iret
-SYM_CODE_END(xen_iret)
+SYM_CODE_END_SECTION(xen_iret)
/*
* Xen handles syscall callbacks much like ordinary exceptions, which
@@ -208,7 +208,7 @@ SYM_CODE_END(xen_iret)
*/
/* Normal 64-bit system call target */
-SYM_CODE_START(xen_syscall_target)
+SYM_CODE_START_SECTION(xen_syscall_target)
UNWIND_HINT_EMPTY
popq %rcx
popq %r11
@@ -222,12 +222,12 @@ SYM_CODE_START(xen_syscall_target)
movq $__USER_CS, 1*8(%rsp)
jmp entry_SYSCALL_64_after_hwframe
-SYM_CODE_END(xen_syscall_target)
+SYM_CODE_END_SECTION(xen_syscall_target)
#ifdef CONFIG_IA32_EMULATION
/* 32-bit compat syscall target */
-SYM_CODE_START(xen_syscall32_target)
+SYM_CODE_START_SECTION(xen_syscall32_target)
UNWIND_HINT_EMPTY
popq %rcx
popq %r11
@@ -241,10 +241,10 @@ SYM_CODE_START(xen_syscall32_target)
movq $__USER32_CS, 1*8(%rsp)
jmp entry_SYSCALL_compat_after_hwframe
-SYM_CODE_END(xen_syscall32_target)
+SYM_CODE_END_SECTION(xen_syscall32_target)
/* 32-bit compat sysenter target */
-SYM_CODE_START(xen_sysenter_target)
+SYM_CODE_START_SECTION(xen_sysenter_target)
UNWIND_HINT_EMPTY
/*
* NB: Xen is polite and clears TF from EFLAGS for us. This means
@@ -262,11 +262,11 @@ SYM_CODE_START(xen_sysenter_target)
movq $__USER32_CS, 1*8(%rsp)
jmp entry_SYSENTER_compat_after_hwframe
-SYM_CODE_END(xen_sysenter_target)
+SYM_CODE_END_SECTION(xen_sysenter_target)
#else /* !CONFIG_IA32_EMULATION */
-SYM_CODE_START(xen_syscall32_target)
+SYM_CODE_START_SECTION(xen_syscall32_target)
SYM_CODE_START(xen_sysenter_target)
UNWIND_HINT_EMPTY
lea 16(%rsp), %rsp /* strip %rcx, %r11 */
@@ -274,6 +274,6 @@ SYM_CODE_START(xen_sysenter_target)
pushq $0
jmp hypercall_iret
SYM_CODE_END(xen_sysenter_target)
-SYM_CODE_END(xen_syscall32_target)
+SYM_CODE_END_SECTION(xen_syscall32_target)
#endif /* CONFIG_IA32_EMULATION */
diff --git a/arch/x86/xen/xen-head.S b/arch/x86/xen/xen-head.S
index 6a64496edefb..6cb0ffb02670 100644
--- a/arch/x86/xen/xen-head.S
+++ b/arch/x86/xen/xen-head.S
@@ -71,13 +71,11 @@ SYM_CODE_END(startup_xen)
__FINIT
#ifdef CONFIG_XEN_PV_SMP
-.pushsection .text
-SYM_CODE_START(asm_cpu_bringup_and_idle)
+SYM_CODE_START_SECTION(asm_cpu_bringup_and_idle)
UNWIND_HINT_EMPTY
call cpu_bringup_and_idle
-SYM_CODE_END(asm_cpu_bringup_and_idle)
-.popsection
+SYM_CODE_END_SECTION(asm_cpu_bringup_and_idle)
#endif
#endif
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 06/14] FG-KASLR: use a scripted approach to handle .text.* sections
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
` (4 preceding siblings ...)
2021-12-02 22:32 ` [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-03 9:54 ` Peter Zijlstra
2021-12-02 22:32 ` [PATCH v8 07/14] kallsyms: Hide layout Alexander Lobakin
` (8 subsequent siblings)
14 siblings, 1 reply; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
Instead of relying on the linker and his heuristics about where to
place (orphan) .text.* section, use a script to read vmlinux.o and
generate a new .tmp_vmlinux.lds which will contain an entry for
each of them. It relies on a magic marker inside the preprocessed
vmlinux.lds (which is harmless in case FG-KASLR is disabled) and
injects a list of input text sections there.
As a bonus, this approach allows us to configure FG-KASLR in terms
of number of functions per each section. The zero value means one
section per each functions, it is the strongest choice, but the
resulting vmlinux also has the biggest size here, as well as the
total number of sections and the boottime delay (which is still
barely noticeable). The values of 4-8 are still strong enough and
allows to save some space, and so on.
We also keep tracking the maximal alignment we found while
traversing through the readelf output and the number of times we
spotted it. It's actual only for values >= 64 and is required to
reserve some space between the last .text.* section and the _etext
marker.
The reason is that e.g. x86 has at least 3 ASM sections (4 with
ClangCFI) aligned to 4096, and when mixing them with the small
sections, we could go past the _etext and render the kernel
unbootable. This reserved space ensures this won't happen.
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
arch/x86/kernel/vmlinux.lds.S | 4 +-
include/asm-generic/vmlinux.lds.h | 6 ++
init/Kconfig | 14 +++
scripts/generate_text_sections.pl | 156 ++++++++++++++++++++++++++++++
scripts/link-vmlinux.sh | 25 ++++-
5 files changed, 203 insertions(+), 2 deletions(-)
create mode 100755 scripts/generate_text_sections.pl
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index e1db74452485..532daa7fc888 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -147,9 +147,11 @@ SECTIONS
#endif
} :text =0xcccc
+ TEXT_FG_KASLR
+
/* End of text section, which should occupy whole number of pages */
- _etext = .;
. = ALIGN(PAGE_SIZE);
+ _etext = .;
X86_ALIGN_RODATA_BEGIN
RO_DATA(PAGE_SIZE)
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 96fbedcbf7c8..8ddc08baf50c 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -127,6 +127,12 @@
#define TEXT_MAIN .text
#endif
+/*
+ * Used by scripts/generate_text_sections.pl to inject text sections,
+ * harmless if FG-KASLR is disabled.
+ */
+#define TEXT_FG_KASLR __fg_kaslr_magic = .;
+
/*
* GCC 4.5 and later have a 32 bytes section alignment for structures.
* Except GCC 4.9, that feels the need to align on 64 bytes.
diff --git a/init/Kconfig b/init/Kconfig
index 5cb8f8230915..8baeaef382c9 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2063,6 +2063,20 @@ config FG_KASLR
If unsure, say N.
+config FG_KASLR_SHIFT
+ int "FG-KASLR granularity (number of functions per section shift)"
+ depends on FG_KASLR
+ range 0 16
+ default 0
+ help
+ This sets the number of functions that will be put in each section
+ as a power of two.
+ Decreasing the value increases the randomization, but also increases
+ the size of the final kernel/vmlinux due to the amount of sections.
+ 0 means that a separate section will be created for each function.
+ 16 almost disables the randomization, leaving only the manual
+ separation.
+
endmenu # General setup
source "arch/Kconfig"
diff --git a/scripts/generate_text_sections.pl b/scripts/generate_text_sections.pl
new file mode 100755
index 000000000000..6871045fb7a6
--- /dev/null
+++ b/scripts/generate_text_sections.pl
@@ -0,0 +1,156 @@
+#!/usr/bin/env perl
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Generates a new LD script with every .text.* section described for FG-KASLR
+# to avoid orphan/heuristic section placement and double-checks we don't have
+# any symbols in plain .text section.
+#
+# Copyright (C) 2021, Intel Corporation.
+# Author: Alexander Lobakin <alexandr.lobakin@intel.com>
+#
+
+use strict;
+use warnings;
+
+## parameters
+my $expecting = 0;
+my $shift = 0;
+my $file;
+
+foreach (@ARGV) {
+ if ($_ eq '-s') {
+ $expecting = 1;
+ } elsif ($expecting) {
+ $shift = $_ + 0;
+ if ($shift < 0) {
+ $shift = 0;
+ } elsif ($shift > 16) {
+ $shift = 16;
+ }
+ $expecting = 0;
+ } elsif (!defined($file)) {
+ $file = $_;
+ } else {
+ die "$0: usage: $0 [-s shift] binary < linker script";
+ }
+}
+
+if (!defined($file)) {
+ die "$0: usage: $0 [-s shift] binary < linker script";
+}
+
+## environment
+my $readelf = $ENV{'READELF'} || die "$0: ERROR: READELF not set?";
+
+## text sections array
+my @sections = ();
+my $has_ccf = 0;
+
+## max alignment found to reserve some space
+my $max_align = 64;
+my $count = 0;
+
+sub read_sections {
+ open(my $fh, "\"$readelf\" -SW \"$file\" 2>/dev/null |")
+ or die "$0: ERROR: failed to execute \"$readelf\": $!";
+
+ while (<$fh>) {
+ my $name;
+ my $align;
+ chomp;
+
+ ($name, $align) = $_ =~ /^\s*\[[\s0-9]*\]\s*(\.\S*)\s*[A-Z]*\s*[0-9a-f]{16}\s*[0-9a-f]*\s*[0-9a-f]*\s*[0-9a-f]*\s*[0-9a-f]{2}\s*[A-Z]{2}\s*[0-9]\s*[0-9]\s*([0-9]*)$/;
+
+ if (!defined($name)) {
+ next;
+ }
+
+ ## Clang 13 onwards emits __cfi_check_fail only on final
+ ## linking, so it won't appear in .o files and will be
+ ## missing in @sections. Add it manually to prevent
+ ## spawning orphans.
+ if ($name eq ".text.__cfi_check_fail") {
+ $has_ccf = 1;
+ }
+
+ if (!($name =~ /^\.text\.[0-9a-zA-Z_]*((\.constprop|\.isra|\.part)\.[0-9])*(|\.[0-9cfi]*)$/)) {
+ next;
+ }
+
+ if ($align > $max_align) {
+ $max_align = $align;
+ $count = 1;
+ } elsif ($align == $max_align) {
+ $count++;
+ }
+
+ push(@sections, $name);
+ }
+
+ close($fh);
+
+ if (!$has_ccf) {
+ push(@sections, ".text.__cfi_check_fail");
+ }
+
+ @sections = sort @sections;
+}
+
+sub print_sections {
+ my $fps = 1 << $shift;
+ my $counter = 1;
+
+ print "\t.text.0 : ALIGN(16) {\n";
+ print "\t\t*(.text)\n";
+ print "\t}\n";
+
+ print "\tASSERT(SIZEOF(.text.0) == 0, \"Plain .text is not empty!\")\n\n";
+
+ if (!@sections) {
+ return;
+ }
+
+ while () {
+ print "\t.text.$counter : ALIGN(16) {\n";
+
+ my @a = (($counter - 1) * $fps .. ($counter * $fps) - 1);
+ for (@a) {
+ print "\t\t*($sections[$_])\n";
+
+ if ($sections[$_] eq $sections[-1]) {
+ print "\t}\n";
+ return;
+ }
+ }
+
+ print "\t}\n";
+ $counter++;
+ }
+}
+
+sub print_reserve {
+ ## If we have text sections aligned with 64 bytes or more, make
+ ## sure we reserve some space for them to not overlap _etext
+ ## while shuffling sections.
+ if (!$count) {
+ return;
+ }
+
+ print "\n\t. += $max_align * $count;\n";
+}
+
+sub print_lds {
+ while (<STDIN>) {
+ if ($_ =~ /^\s*__fg_kaslr_magic = \.;$/) {
+ print_sections();
+ print_reserve();
+ } else {
+ print $_;
+ }
+ }
+}
+
+## main
+
+read_sections();
+print_lds();
diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index 5cdd9bc5c385..9245351a38d9 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -66,6 +66,18 @@ gen_symversions()
done
}
+# If CONFIG_FG_KASLR is selected, generate a linker script which will
+# declare all custom text sections for future boottime shuffling
+gen_text_sections()
+{
+ info GEN .tmp_vmlinux.lds
+
+ ${PERL} ${srctree}/scripts/generate_text_sections.pl \
+ -s "${CONFIG_FG_KASLR_SHIFT}" vmlinux.o \
+ < "${objtree}/${KBUILD_LDS}" \
+ > .tmp_vmlinux.lds
+}
+
# Link of vmlinux.o used for section mismatch analysis
# ${1} output file
modpost_link()
@@ -155,12 +167,19 @@ vmlinux_link()
local ld
local ldflags
local ldlibs
+ local lds
info LD ${output}
# skip output file argument
shift
+ if [ -n "${CONFIG_FG_KASLR}" ]; then
+ lds=".tmp_vmlinux.lds"
+ else
+ lds="${objtree}/${KBUILD_LDS}"
+ fi
+
if [ -n "${CONFIG_LTO_CLANG}" ]; then
# Use vmlinux.o instead of performing the slow LTO link again.
objs=vmlinux.o
@@ -182,7 +201,7 @@ vmlinux_link()
ldlibs=
fi
- ldflags="${ldflags} ${wl}--script=${objtree}/${KBUILD_LDS}"
+ ldflags="${ldflags} ${wl}--script=${lds}"
# The kallsyms linking does not need debug symbols included.
if [ "$output" != "${output#.tmp_vmlinux.kallsyms}" ] ; then
@@ -342,6 +361,10 @@ info GEN modules.builtin
tr '\0' '\n' < modules.builtin.modinfo | sed -n 's/^[[:alnum:]:_]*\.file=//p' |
tr ' ' '\n' | uniq | sed -e 's:^:kernel/:' -e 's/$/.ko/' > modules.builtin
+if [ -n "${CONFIG_FG_KASLR}" ]; then
+ gen_text_sections
+fi
+
btf_vmlinux_bin_o=""
if [ -n "${CONFIG_DEBUG_INFO_BTF}" ]; then
btf_vmlinux_bin_o=.btf.vmlinux.bin.o
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 07/14] kallsyms: Hide layout
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
` (5 preceding siblings ...)
2021-12-02 22:32 ` [PATCH v8 06/14] FG-KASLR: use a scripted approach to handle .text.* sections Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-03 10:00 ` Peter Zijlstra
2021-12-02 22:32 ` [PATCH v8 08/14] livepatch: only match unique symbols when using FG-KASLR Alexander Lobakin
` (7 subsequent siblings)
14 siblings, 1 reply; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
From: Kristen Carlson Accardi <kristen@linux.intel.com>
This patch makes /proc/kallsyms display in a random order, rather
than sorted by address in order to hide the newly randomized address
layout.
Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Tested-by: Tony Luck <tony.luck@intel.com>
Reported-by: kernel test robot <lkp@intel.com> # swap.cocci
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
kernel/kallsyms.c | 158 +++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 157 insertions(+), 1 deletion(-)
diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c
index 3011bc33a5ba..ff9d8b651966 100644
--- a/kernel/kallsyms.c
+++ b/kernel/kallsyms.c
@@ -583,6 +583,12 @@ struct kallsym_iter {
int show_value;
};
+struct kallsyms_shuffled_iter {
+ struct kallsym_iter iter;
+ loff_t total_syms;
+ loff_t shuffled_index[];
+};
+
int __weak arch_get_kallsym(unsigned int symnum, unsigned long *value,
char *type, char *name)
{
@@ -830,7 +836,7 @@ bool kallsyms_show_value(const struct cred *cred)
}
}
-static int kallsyms_open(struct inode *inode, struct file *file)
+static int __kallsyms_open(struct inode *inode, struct file *file)
{
/*
* We keep iterator in m->private, since normal case is to
@@ -851,6 +857,156 @@ static int kallsyms_open(struct inode *inode, struct file *file)
return 0;
}
+/*
+ * When function granular kaslr is enabled, we need to print out the symbols
+ * at random so we don't reveal the new layout.
+ */
+#ifdef CONFIG_FG_KASLR
+static int update_random_pos(struct kallsyms_shuffled_iter *s_iter,
+ loff_t pos, loff_t *new_pos)
+{
+ loff_t new;
+
+ if (pos >= s_iter->total_syms)
+ return 0;
+
+ new = s_iter->shuffled_index[pos];
+
+ /*
+ * normally this would be done as part of update_iter, however,
+ * we want to avoid triggering this in the event that new is
+ * zero since we don't want to blow away our pos end indicators.
+ */
+ if (new == 0) {
+ s_iter->iter.name[0] = '\0';
+ s_iter->iter.nameoff = get_symbol_offset(new);
+ s_iter->iter.pos = new;
+ }
+
+ *new_pos = new;
+ return 1;
+}
+
+static void *shuffled_start(struct seq_file *m, loff_t *pos)
+{
+ struct kallsyms_shuffled_iter *s_iter = m->private;
+ loff_t new_pos;
+
+ if (!update_random_pos(s_iter, *pos, &new_pos))
+ return NULL;
+
+ return s_start(m, &new_pos);
+}
+
+static void *shuffled_next(struct seq_file *m, void *p, loff_t *pos)
+{
+ struct kallsyms_shuffled_iter *s_iter = m->private;
+ loff_t new_pos;
+
+ (*pos)++;
+
+ if (!update_random_pos(s_iter, *pos, &new_pos))
+ return NULL;
+
+ if (!update_iter(m->private, new_pos))
+ return NULL;
+
+ return p;
+}
+
+/*
+ * shuffle_index_list()
+ * Use a Fisher Yates algorithm to shuffle a list of text sections.
+ */
+static void shuffle_index_list(loff_t *indexes, loff_t size)
+{
+ u32 i, j;
+
+ for (i = size - 1; i > 0; i--) {
+ /* pick a random index from 0 to i */
+ j = get_random_u32() % (i + 1);
+
+ swap(indexes[i], indexes[j]);
+ }
+}
+
+static const struct seq_operations kallsyms_shuffled_op = {
+ .start = shuffled_start,
+ .next = shuffled_next,
+ .stop = s_stop,
+ .show = s_show
+};
+
+static int kallsyms_random_open(struct inode *inode, struct file *file)
+{
+ loff_t pos;
+ struct kallsyms_shuffled_iter *shuffled_iter;
+ struct kallsym_iter iter;
+ bool show_value;
+
+ /*
+ * If privileged, go ahead and use the normal algorithm for
+ * displaying symbols
+ */
+ show_value = kallsyms_show_value(file->f_cred);
+ if (show_value)
+ return __kallsyms_open(inode, file);
+
+ /*
+ * we need to figure out how many extra symbols there are
+ * to print out past kallsyms_num_syms
+ */
+ pos = kallsyms_num_syms;
+ reset_iter(&iter, 0);
+ do {
+ if (!update_iter(&iter, pos))
+ break;
+ pos++;
+ } while (1);
+
+ /*
+ * add storage space for an array of loff_t equal to the size
+ * of the total number of symbols we need to print
+ */
+ shuffled_iter = __seq_open_private(file, &kallsyms_shuffled_op,
+ sizeof(*shuffled_iter) +
+ (sizeof(pos) * pos));
+ if (!shuffled_iter)
+ return -ENOMEM;
+
+ reset_iter(&shuffled_iter->iter, 0);
+ shuffled_iter->iter.show_value = show_value;
+ shuffled_iter->total_syms = pos;
+
+ /*
+ * the existing update_iter algorithm requires that we
+ * are either moving along increasing pos sequentially,
+ * or that these values are correct. Since these values
+ * were discovered above, initialize our new iter so we
+ * can use update_iter non-sequentially.
+ */
+ shuffled_iter->iter.pos_arch_end = iter.pos_arch_end;
+ shuffled_iter->iter.pos_mod_end = iter.pos_mod_end;
+ shuffled_iter->iter.pos_ftrace_mod_end = iter.pos_ftrace_mod_end;
+
+ /*
+ * initialize the array with all possible pos values, then
+ * shuffle the array so that the values will display in a random
+ * order.
+ */
+ for (pos = 0; pos < shuffled_iter->total_syms; pos++)
+ shuffled_iter->shuffled_index[pos] = pos;
+
+ shuffle_index_list(shuffled_iter->shuffled_index, shuffled_iter->total_syms);
+
+ return 0;
+}
+
+#define kallsyms_open kallsyms_random_open
+#else
+#define kallsyms_open __kallsyms_open
+#endif /* !CONFIG_FG_KASLR */
+
#ifdef CONFIG_KGDB_KDB
const char *kdb_walk_kallsyms(loff_t *pos)
{
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 08/14] livepatch: only match unique symbols when using FG-KASLR
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
` (6 preceding siblings ...)
2021-12-02 22:32 ` [PATCH v8 07/14] kallsyms: Hide layout Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-03 10:05 ` Peter Zijlstra
2021-12-02 22:32 ` [PATCH v8 09/14] x86/boot: allow FG-KASLR to be selected Alexander Lobakin
` (6 subsequent siblings)
14 siblings, 1 reply; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
If any type of function granular randomization is enabled, the sympos
algorithm will fail, as it will be impossible to resolve symbols when
there are duplicates using the previous symbol position.
We could override sympos to 0, but make it more clear to the user
and bail out if the symbol is not unique.
Suggested-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
kernel/livepatch/core.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 335d988bd811..10ea75111057 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -169,6 +169,17 @@ static int klp_find_object_symbol(const char *objname, const char *name,
else
kallsyms_on_each_symbol(klp_find_callback, &args);
+ /*
+ * If function granular randomization is enabled, it is impossible
+ * to resolve symbols when there are duplicates using the previous
+ * symbol position (i.e. sympos != 0).
+ */
+ if (IS_ENABLED(CONFIG_FG_KASLR) && sympos) {
+ pr_err("FG-KASLR is enabled, specifying symbol position %lu for symbol '%s' in object '%s' does not work\n",
+ sympos, name, objname ? objname : "vmlinux");
+ goto out_err;
+ }
+
/*
* Ensure an address was found. If sympos is 0, ensure symbol is unique;
* otherwise ensure the symbol position count matches sympos.
@@ -186,6 +197,7 @@ static int klp_find_object_symbol(const char *objname, const char *name,
return 0;
}
+out_err:
*addr = 0;
return -EINVAL;
}
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 09/14] x86/boot: allow FG-KASLR to be selected
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
` (7 preceding siblings ...)
2021-12-02 22:32 ` [PATCH v8 08/14] livepatch: only match unique symbols when using FG-KASLR Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 10/14] arm64/crypto: conditionally place ASM functions into separate sections Alexander Lobakin
` (5 subsequent siblings)
14 siblings, 0 replies; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
Now that we have full support of FG-KASLR from both kernel core
and x86 code, allow FG-KASLR to be enabled for x86_64 if the
"regular" KASLR is also turned on.
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
arch/x86/Kconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 7399327d1eff..970a03c7588d 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -75,6 +75,7 @@ config X86
select ARCH_HAS_EARLY_DEBUG if KGDB
select ARCH_HAS_ELF_RANDOMIZE
select ARCH_HAS_FAST_MULTIPLIER
+ select ARCH_HAS_FG_KASLR if X86_64 && RANDOMIZE_BASE
select ARCH_HAS_FILTER_PGPROT
select ARCH_HAS_FORTIFY_SOURCE
select ARCH_HAS_GCOV_PROFILE_ALL
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 10/14] arm64/crypto: conditionally place ASM functions into separate sections
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
` (8 preceding siblings ...)
2021-12-02 22:32 ` [PATCH v8 09/14] x86/boot: allow FG-KASLR to be selected Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 11/14] module: Reorder functions Alexander Lobakin
` (4 subsequent siblings)
14 siblings, 0 replies; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
The resulting LD script generated by FG-KASLR script contains a size
assertion for the input .text function. In case if it's not empty,
the build will stop plug a potentional layout leakage.
As FG-KASLR for modules tends to be arch-independent, we should take
care of the modular ASM code of every architecture to not break the
build.
This is the ARM64 part.
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
arch/arm64/crypto/aes-ce-ccm-core.S | 16 +++++------
arch/arm64/crypto/aes-ce-core.S | 16 +++++------
arch/arm64/crypto/aes-ce.S | 4 +--
arch/arm64/crypto/aes-cipher-core.S | 8 +++---
arch/arm64/crypto/aes-modes.S | 16 +++++------
arch/arm64/crypto/aes-neon.S | 4 +--
arch/arm64/crypto/aes-neonbs-core.S | 38 +++++++++++++--------------
arch/arm64/crypto/chacha-neon-core.S | 18 ++++++-------
arch/arm64/crypto/crct10dif-ce-core.S | 14 +++++-----
arch/arm64/crypto/ghash-ce-core.S | 24 ++++++++---------
arch/arm64/crypto/nh-neon-core.S | 4 +--
arch/arm64/crypto/poly1305-armv8.pl | 17 ++++++++++++
arch/arm64/crypto/sha1-ce-core.S | 4 +--
arch/arm64/crypto/sha2-ce-core.S | 4 +--
arch/arm64/crypto/sha3-ce-core.S | 4 +--
arch/arm64/crypto/sha512-armv8.pl | 11 ++++++++
arch/arm64/crypto/sha512-ce-core.S | 4 +--
arch/arm64/crypto/sm3-ce-core.S | 4 +--
arch/arm64/crypto/sm4-ce-core.S | 4 +--
19 files changed, 121 insertions(+), 93 deletions(-)
diff --git a/arch/arm64/crypto/aes-ce-ccm-core.S b/arch/arm64/crypto/aes-ce-ccm-core.S
index b03f7f71f893..03c8606f45b4 100644
--- a/arch/arm64/crypto/aes-ce-ccm-core.S
+++ b/arch/arm64/crypto/aes-ce-ccm-core.S
@@ -15,7 +15,7 @@
* u32 ce_aes_ccm_auth_data(u8 mac[], u8 const in[], u32 abytes,
* u32 macp, u8 const rk[], u32 rounds);
*/
-SYM_FUNC_START(ce_aes_ccm_auth_data)
+SYM_FUNC_START_SECTION(ce_aes_ccm_auth_data)
ld1 {v0.16b}, [x0] /* load mac */
cbz w3, 1f
sub w3, w3, #16
@@ -80,13 +80,13 @@ SYM_FUNC_START(ce_aes_ccm_auth_data)
st1 {v0.16b}, [x0]
10: mov w0, w3
ret
-SYM_FUNC_END(ce_aes_ccm_auth_data)
+SYM_FUNC_END_SECTION(ce_aes_ccm_auth_data)
/*
* void ce_aes_ccm_final(u8 mac[], u8 const ctr[], u8 const rk[],
* u32 rounds);
*/
-SYM_FUNC_START(ce_aes_ccm_final)
+SYM_FUNC_START_SECTION(ce_aes_ccm_final)
ld1 {v3.4s}, [x2], #16 /* load first round key */
ld1 {v0.16b}, [x0] /* load mac */
cmp w3, #12 /* which key size? */
@@ -120,7 +120,7 @@ SYM_FUNC_START(ce_aes_ccm_final)
eor v0.16b, v0.16b, v1.16b /* en-/decrypt the mac */
st1 {v0.16b}, [x0] /* store result */
ret
-SYM_FUNC_END(ce_aes_ccm_final)
+SYM_FUNC_END_SECTION(ce_aes_ccm_final)
.macro aes_ccm_do_crypt,enc
cbz x2, 5f
@@ -212,10 +212,10 @@ CPU_LE( rev x8, x8 )
* u8 const rk[], u32 rounds, u8 mac[],
* u8 ctr[]);
*/
-SYM_FUNC_START(ce_aes_ccm_encrypt)
+SYM_FUNC_START_SECTION(ce_aes_ccm_encrypt)
aes_ccm_do_crypt 1
-SYM_FUNC_END(ce_aes_ccm_encrypt)
+SYM_FUNC_END_SECTION(ce_aes_ccm_encrypt)
-SYM_FUNC_START(ce_aes_ccm_decrypt)
+SYM_FUNC_START_SECTION(ce_aes_ccm_decrypt)
aes_ccm_do_crypt 0
-SYM_FUNC_END(ce_aes_ccm_decrypt)
+SYM_FUNC_END_SECTION(ce_aes_ccm_decrypt)
diff --git a/arch/arm64/crypto/aes-ce-core.S b/arch/arm64/crypto/aes-ce-core.S
index e52e13eb8fdb..abe6ee0501bf 100644
--- a/arch/arm64/crypto/aes-ce-core.S
+++ b/arch/arm64/crypto/aes-ce-core.S
@@ -8,7 +8,7 @@
.arch armv8-a+crypto
-SYM_FUNC_START(__aes_ce_encrypt)
+SYM_FUNC_START_SECTION(__aes_ce_encrypt)
sub w3, w3, #2
ld1 {v0.16b}, [x2]
ld1 {v1.4s}, [x0], #16
@@ -34,9 +34,9 @@ SYM_FUNC_START(__aes_ce_encrypt)
eor v0.16b, v0.16b, v3.16b
st1 {v0.16b}, [x1]
ret
-SYM_FUNC_END(__aes_ce_encrypt)
+SYM_FUNC_END_SECTION(__aes_ce_encrypt)
-SYM_FUNC_START(__aes_ce_decrypt)
+SYM_FUNC_START_SECTION(__aes_ce_decrypt)
sub w3, w3, #2
ld1 {v0.16b}, [x2]
ld1 {v1.4s}, [x0], #16
@@ -62,23 +62,23 @@ SYM_FUNC_START(__aes_ce_decrypt)
eor v0.16b, v0.16b, v3.16b
st1 {v0.16b}, [x1]
ret
-SYM_FUNC_END(__aes_ce_decrypt)
+SYM_FUNC_END_SECTION(__aes_ce_decrypt)
/*
* __aes_ce_sub() - use the aese instruction to perform the AES sbox
* substitution on each byte in 'input'
*/
-SYM_FUNC_START(__aes_ce_sub)
+SYM_FUNC_START_SECTION(__aes_ce_sub)
dup v1.4s, w0
movi v0.16b, #0
aese v0.16b, v1.16b
umov w0, v0.s[0]
ret
-SYM_FUNC_END(__aes_ce_sub)
+SYM_FUNC_END_SECTION(__aes_ce_sub)
-SYM_FUNC_START(__aes_ce_invert)
+SYM_FUNC_START_SECTION(__aes_ce_invert)
ld1 {v0.4s}, [x1]
aesimc v1.16b, v0.16b
st1 {v1.4s}, [x0]
ret
-SYM_FUNC_END(__aes_ce_invert)
+SYM_FUNC_END_SECTION(__aes_ce_invert)
diff --git a/arch/arm64/crypto/aes-ce.S b/arch/arm64/crypto/aes-ce.S
index 1dc5bbbfeed2..909d2dcf0907 100644
--- a/arch/arm64/crypto/aes-ce.S
+++ b/arch/arm64/crypto/aes-ce.S
@@ -9,8 +9,8 @@
#include <linux/linkage.h>
#include <asm/assembler.h>
-#define AES_FUNC_START(func) SYM_FUNC_START(ce_ ## func)
-#define AES_FUNC_END(func) SYM_FUNC_END(ce_ ## func)
+#define AES_FUNC_START(func) SYM_FUNC_START_SECTION(ce_ ## func)
+#define AES_FUNC_END(func) SYM_FUNC_END_SECTION(ce_ ## func)
.arch armv8-a+crypto
diff --git a/arch/arm64/crypto/aes-cipher-core.S b/arch/arm64/crypto/aes-cipher-core.S
index c9d6955f8404..e47c0aef7a7d 100644
--- a/arch/arm64/crypto/aes-cipher-core.S
+++ b/arch/arm64/crypto/aes-cipher-core.S
@@ -122,11 +122,11 @@ CPU_BE( rev w7, w7 )
ret
.endm
-SYM_FUNC_START(__aes_arm64_encrypt)
+SYM_FUNC_START_SECTION(__aes_arm64_encrypt)
do_crypt fround, crypto_ft_tab, crypto_ft_tab + 1, 2
-SYM_FUNC_END(__aes_arm64_encrypt)
+SYM_FUNC_END_SECTION(__aes_arm64_encrypt)
+SYM_FUNC_START_SECTION(__aes_arm64_decrypt)
.align 5
-SYM_FUNC_START(__aes_arm64_decrypt)
do_crypt iround, crypto_it_tab, crypto_aes_inv_sbox, 0
-SYM_FUNC_END(__aes_arm64_decrypt)
+SYM_FUNC_END_SECTION(__aes_arm64_decrypt)
diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S
index b495de22bb38..5f7a43fa8438 100644
--- a/arch/arm64/crypto/aes-modes.S
+++ b/arch/arm64/crypto/aes-modes.S
@@ -22,26 +22,26 @@
#define ST5(x...) x
#endif
-SYM_FUNC_START_LOCAL(aes_encrypt_block4x)
+SYM_FUNC_START_LOCAL_SECTION(aes_encrypt_block4x)
encrypt_block4x v0, v1, v2, v3, w3, x2, x8, w7
ret
-SYM_FUNC_END(aes_encrypt_block4x)
+SYM_FUNC_END_SECTION(aes_encrypt_block4x)
-SYM_FUNC_START_LOCAL(aes_decrypt_block4x)
+SYM_FUNC_START_LOCAL_SECTION(aes_decrypt_block4x)
decrypt_block4x v0, v1, v2, v3, w3, x2, x8, w7
ret
-SYM_FUNC_END(aes_decrypt_block4x)
+SYM_FUNC_END_SECTION(aes_decrypt_block4x)
#if MAX_STRIDE == 5
-SYM_FUNC_START_LOCAL(aes_encrypt_block5x)
+SYM_FUNC_START_LOCAL_SECTION(aes_encrypt_block5x)
encrypt_block5x v0, v1, v2, v3, v4, w3, x2, x8, w7
ret
-SYM_FUNC_END(aes_encrypt_block5x)
+SYM_FUNC_END_SECTION(aes_encrypt_block5x)
-SYM_FUNC_START_LOCAL(aes_decrypt_block5x)
+SYM_FUNC_START_LOCAL_SECTION(aes_decrypt_block5x)
decrypt_block5x v0, v1, v2, v3, v4, w3, x2, x8, w7
ret
-SYM_FUNC_END(aes_decrypt_block5x)
+SYM_FUNC_END_SECTION(aes_decrypt_block5x)
#endif
/*
diff --git a/arch/arm64/crypto/aes-neon.S b/arch/arm64/crypto/aes-neon.S
index e47d3ec2cfb4..9c8d6cccd2cd 100644
--- a/arch/arm64/crypto/aes-neon.S
+++ b/arch/arm64/crypto/aes-neon.S
@@ -8,8 +8,8 @@
#include <linux/linkage.h>
#include <asm/assembler.h>
-#define AES_FUNC_START(func) SYM_FUNC_START(neon_ ## func)
-#define AES_FUNC_END(func) SYM_FUNC_END(neon_ ## func)
+#define AES_FUNC_START(func) SYM_FUNC_START_SECTION(neon_ ## func)
+#define AES_FUNC_END(func) SYM_FUNC_END_SECTION(neon_ ## func)
xtsmask .req v7
cbciv .req v7
diff --git a/arch/arm64/crypto/aes-neonbs-core.S b/arch/arm64/crypto/aes-neonbs-core.S
index a3405b8c344b..582343f18ad0 100644
--- a/arch/arm64/crypto/aes-neonbs-core.S
+++ b/arch/arm64/crypto/aes-neonbs-core.S
@@ -380,7 +380,7 @@ ISRM0: .octa 0x0306090c00070a0d01040b0e0205080f
/*
* void aesbs_convert_key(u8 out[], u32 const rk[], int rounds)
*/
-SYM_FUNC_START(aesbs_convert_key)
+SYM_FUNC_START_SECTION(aesbs_convert_key)
ld1 {v7.4s}, [x1], #16 // load round 0 key
ld1 {v17.4s}, [x1], #16 // load round 1 key
@@ -425,10 +425,10 @@ SYM_FUNC_START(aesbs_convert_key)
eor v17.16b, v17.16b, v7.16b
str q17, [x0]
ret
-SYM_FUNC_END(aesbs_convert_key)
+SYM_FUNC_END_SECTION(aesbs_convert_key)
+SYM_FUNC_START_LOCAL_SECTION(aesbs_encrypt8)
.align 4
-SYM_FUNC_START_LOCAL(aesbs_encrypt8)
ldr q9, [bskey], #16 // round 0 key
ldr q8, M0SR
ldr q24, SR
@@ -488,10 +488,10 @@ SYM_FUNC_START_LOCAL(aesbs_encrypt8)
eor v2.16b, v2.16b, v12.16b
eor v5.16b, v5.16b, v12.16b
ret
-SYM_FUNC_END(aesbs_encrypt8)
+SYM_FUNC_END_SECTION(aesbs_encrypt8)
+SYM_FUNC_START_LOCAL_SECTION(aesbs_decrypt8)
.align 4
-SYM_FUNC_START_LOCAL(aesbs_decrypt8)
lsl x9, rounds, #7
add bskey, bskey, x9
@@ -553,7 +553,7 @@ SYM_FUNC_START_LOCAL(aesbs_decrypt8)
eor v3.16b, v3.16b, v12.16b
eor v5.16b, v5.16b, v12.16b
ret
-SYM_FUNC_END(aesbs_decrypt8)
+SYM_FUNC_END_SECTION(aesbs_decrypt8)
/*
* aesbs_ecb_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
@@ -619,13 +619,13 @@ SYM_FUNC_END(aesbs_decrypt8)
ret
.endm
+SYM_FUNC_START_SECTION(aesbs_ecb_encrypt)
.align 4
-SYM_FUNC_START(aesbs_ecb_encrypt)
__ecb_crypt aesbs_encrypt8, v0, v1, v4, v6, v3, v7, v2, v5
-SYM_FUNC_END(aesbs_ecb_encrypt)
+SYM_FUNC_END_SECTION(aesbs_ecb_encrypt)
+SYM_FUNC_START_SECTION(aesbs_ecb_decrypt)
.align 4
-SYM_FUNC_START(aesbs_ecb_decrypt)
__ecb_crypt aesbs_decrypt8, v0, v1, v6, v4, v2, v7, v3, v5
SYM_FUNC_END(aesbs_ecb_decrypt)
@@ -633,8 +633,8 @@ SYM_FUNC_END(aesbs_ecb_decrypt)
* aesbs_cbc_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
* int blocks, u8 iv[])
*/
+SYM_FUNC_START_SECTION(aesbs_cbc_decrypt)
.align 4
-SYM_FUNC_START(aesbs_cbc_decrypt)
frame_push 6
mov x19, x0
@@ -718,7 +718,7 @@ SYM_FUNC_START(aesbs_cbc_decrypt)
2: frame_pop
ret
-SYM_FUNC_END(aesbs_cbc_decrypt)
+SYM_FUNC_END_SECTION(aesbs_cbc_decrypt)
.macro next_tweak, out, in, const, tmp
sshr \tmp\().2d, \in\().2d, #63
@@ -734,7 +734,7 @@ SYM_FUNC_END(aesbs_cbc_decrypt)
* aesbs_xts_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
* int blocks, u8 iv[])
*/
-SYM_FUNC_START_LOCAL(__xts_crypt8)
+SYM_FUNC_START_LOCAL_SECTION(__xts_crypt8)
mov x6, #1
lsl x6, x6, x23
subs w23, w23, #8
@@ -787,7 +787,7 @@ SYM_FUNC_START_LOCAL(__xts_crypt8)
0: mov bskey, x21
mov rounds, x22
br x16
-SYM_FUNC_END(__xts_crypt8)
+SYM_FUNC_END_SECTION(__xts_crypt8)
.macro __xts_crypt, do8, o0, o1, o2, o3, o4, o5, o6, o7
frame_push 6, 64
@@ -851,13 +851,13 @@ SYM_FUNC_END(__xts_crypt8)
ret
.endm
-SYM_FUNC_START(aesbs_xts_encrypt)
+SYM_FUNC_START_SECTION(aesbs_xts_encrypt)
__xts_crypt aesbs_encrypt8, v0, v1, v4, v6, v3, v7, v2, v5
-SYM_FUNC_END(aesbs_xts_encrypt)
+SYM_FUNC_END_SECTION(aesbs_xts_encrypt)
-SYM_FUNC_START(aesbs_xts_decrypt)
+SYM_FUNC_START_SECTION(aesbs_xts_decrypt)
__xts_crypt aesbs_decrypt8, v0, v1, v6, v4, v2, v7, v3, v5
-SYM_FUNC_END(aesbs_xts_decrypt)
+SYM_FUNC_END_SECTION(aesbs_xts_decrypt)
.macro next_ctr, v
mov \v\().d[1], x8
@@ -871,7 +871,7 @@ SYM_FUNC_END(aesbs_xts_decrypt)
* aesbs_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[],
* int rounds, int blocks, u8 iv[], u8 final[])
*/
-SYM_FUNC_START(aesbs_ctr_encrypt)
+SYM_FUNC_START_SECTION(aesbs_ctr_encrypt)
frame_push 8
mov x19, x0
@@ -998,4 +998,4 @@ CPU_LE( rev x8, x8 )
7: cbz x25, 8b
st1 {v5.16b}, [x25]
b 8b
-SYM_FUNC_END(aesbs_ctr_encrypt)
+SYM_FUNC_END_SECTION(aesbs_ctr_encrypt)
diff --git a/arch/arm64/crypto/chacha-neon-core.S b/arch/arm64/crypto/chacha-neon-core.S
index b70ac76f2610..34a3087055f8 100644
--- a/arch/arm64/crypto/chacha-neon-core.S
+++ b/arch/arm64/crypto/chacha-neon-core.S
@@ -23,7 +23,6 @@
#include <asm/cache.h>
.text
- .align 6
/*
* chacha_permute - permute one block
@@ -36,7 +35,8 @@
*
* Clobbers: w3, x10, v4, v12
*/
-SYM_FUNC_START_LOCAL(chacha_permute)
+SYM_FUNC_START_LOCAL_SECTION(chacha_permute)
+ .align 6
adr_l x10, ROT8
ld1 {v12.4s}, [x10]
@@ -104,9 +104,9 @@ SYM_FUNC_START_LOCAL(chacha_permute)
b.ne .Ldoubleround
ret
-SYM_FUNC_END(chacha_permute)
+SYM_FUNC_END_SECTION(chacha_permute)
-SYM_FUNC_START(chacha_block_xor_neon)
+SYM_FUNC_START_SECTION(chacha_block_xor_neon)
// x0: Input state matrix, s
// x1: 1 data block output, o
// x2: 1 data block input, i
@@ -143,9 +143,9 @@ SYM_FUNC_START(chacha_block_xor_neon)
ldp x29, x30, [sp], #16
ret
-SYM_FUNC_END(chacha_block_xor_neon)
+SYM_FUNC_END_SECTION(chacha_block_xor_neon)
-SYM_FUNC_START(hchacha_block_neon)
+SYM_FUNC_START_SECTION(hchacha_block_neon)
// x0: Input state matrix, s
// x1: output (8 32-bit words)
// w2: nrounds
@@ -163,7 +163,7 @@ SYM_FUNC_START(hchacha_block_neon)
ldp x29, x30, [sp], #16
ret
-SYM_FUNC_END(hchacha_block_neon)
+SYM_FUNC_END_SECTION(hchacha_block_neon)
a0 .req w12
a1 .req w13
@@ -182,8 +182,8 @@ SYM_FUNC_END(hchacha_block_neon)
a14 .req w27
a15 .req w28
+SYM_FUNC_START_SECTION(chacha_4block_xor_neon)
.align 6
-SYM_FUNC_START(chacha_4block_xor_neon)
frame_push 10
// x0: Input state matrix, s
@@ -790,7 +790,7 @@ CPU_BE( rev a15, a15 )
st1 {v28.16b-v31.16b}, [x7] // overlapping stores
3: st1 {v24.16b-v27.16b}, [x1]
b .Lout
-SYM_FUNC_END(chacha_4block_xor_neon)
+SYM_FUNC_END_SECTION(chacha_4block_xor_neon)
.section ".rodata", "a", %progbits
.align L1_CACHE_SHIFT
diff --git a/arch/arm64/crypto/crct10dif-ce-core.S b/arch/arm64/crypto/crct10dif-ce-core.S
index dce6dcebfca1..54e121a56895 100644
--- a/arch/arm64/crypto/crct10dif-ce-core.S
+++ b/arch/arm64/crypto/crct10dif-ce-core.S
@@ -131,7 +131,7 @@
tbl bd4.16b, {\bd\().16b}, perm4.16b
.endm
-SYM_FUNC_START_LOCAL(__pmull_p8_core)
+SYM_FUNC_START_LOCAL_SECTION(__pmull_p8_core)
.L__pmull_p8_core:
ext t4.8b, ad.8b, ad.8b, #1 // A1
ext t5.8b, ad.8b, ad.8b, #2 // A2
@@ -194,7 +194,7 @@ SYM_FUNC_START_LOCAL(__pmull_p8_core)
eor t4.16b, t4.16b, t5.16b
eor t6.16b, t6.16b, t3.16b
ret
-SYM_FUNC_END(__pmull_p8_core)
+SYM_FUNC_END_SECTION(__pmull_p8_core)
.macro __pmull_p8, rq, ad, bd, i
.ifnc \bd, fold_consts
@@ -465,21 +465,21 @@ CPU_LE( ext v7.16b, v7.16b, v7.16b, #8 )
//
// Assumes len >= 16.
//
-SYM_FUNC_START(crc_t10dif_pmull_p8)
+SYM_FUNC_START_SECTION(crc_t10dif_pmull_p8)
stp x29, x30, [sp, #-16]!
mov x29, sp
crc_t10dif_pmull p8
-SYM_FUNC_END(crc_t10dif_pmull_p8)
+SYM_FUNC_END_SECTION(crc_t10dif_pmull_p8)
- .align 5
//
// u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len);
//
// Assumes len >= 16.
//
-SYM_FUNC_START(crc_t10dif_pmull_p64)
+SYM_FUNC_START_SECTION(crc_t10dif_pmull_p64)
+ .align 5
crc_t10dif_pmull p64
-SYM_FUNC_END(crc_t10dif_pmull_p64)
+SYM_FUNC_END_SECTION(crc_t10dif_pmull_p64)
.section ".rodata", "a"
.align 4
diff --git a/arch/arm64/crypto/ghash-ce-core.S b/arch/arm64/crypto/ghash-ce-core.S
index 7868330dd54e..a69c1d4479db 100644
--- a/arch/arm64/crypto/ghash-ce-core.S
+++ b/arch/arm64/crypto/ghash-ce-core.S
@@ -350,13 +350,13 @@ CPU_LE( rev64 T1.16b, T1.16b )
* void pmull_ghash_update(int blocks, u64 dg[], const char *src,
* struct ghash_key const *k, const char *head)
*/
-SYM_FUNC_START(pmull_ghash_update_p64)
+SYM_FUNC_START_SECTION(pmull_ghash_update_p64)
__pmull_ghash p64
-SYM_FUNC_END(pmull_ghash_update_p64)
+SYM_FUNC_END_SECTION(pmull_ghash_update_p64)
-SYM_FUNC_START(pmull_ghash_update_p8)
+SYM_FUNC_START_SECTION(pmull_ghash_update_p8)
__pmull_ghash p8
-SYM_FUNC_END(pmull_ghash_update_p8)
+SYM_FUNC_END_SECTION(pmull_ghash_update_p8)
KS0 .req v8
KS1 .req v9
@@ -602,20 +602,20 @@ CPU_LE( rev w8, w8 )
* struct ghash_key const *k, u64 dg[], u8 ctr[],
* int rounds, u8 tag)
*/
-SYM_FUNC_START(pmull_gcm_encrypt)
+SYM_FUNC_START_SECTION(pmull_gcm_encrypt)
pmull_gcm_do_crypt 1
-SYM_FUNC_END(pmull_gcm_encrypt)
+SYM_FUNC_END_SECTION(pmull_gcm_encrypt)
/*
* void pmull_gcm_decrypt(int blocks, u8 dst[], const u8 src[],
* struct ghash_key const *k, u64 dg[], u8 ctr[],
* int rounds, u8 tag)
*/
-SYM_FUNC_START(pmull_gcm_decrypt)
+SYM_FUNC_START_SECTION(pmull_gcm_decrypt)
pmull_gcm_do_crypt 0
-SYM_FUNC_END(pmull_gcm_decrypt)
+SYM_FUNC_END_SECTION(pmull_gcm_decrypt)
-SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
+SYM_FUNC_START_LOCAL_SECTION(pmull_gcm_ghash_4x)
movi MASK.16b, #0xe1
shl MASK.2d, MASK.2d, #57
@@ -696,9 +696,9 @@ SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
eor XL.16b, XL.16b, T2.16b
ret
-SYM_FUNC_END(pmull_gcm_ghash_4x)
+SYM_FUNC_END_SECTION(pmull_gcm_ghash_4x)
-SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x)
+SYM_FUNC_START_LOCAL_SECTION(pmull_gcm_enc_4x)
ld1 {KS0.16b}, [x5] // load upper counter
sub w10, w8, #4
sub w11, w8, #3
@@ -761,7 +761,7 @@ SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x)
eor INP3.16b, INP3.16b, KS3.16b
ret
-SYM_FUNC_END(pmull_gcm_enc_4x)
+SYM_FUNC_END_SECTION(pmull_gcm_enc_4x)
.section ".rodata", "a"
.align 6
diff --git a/arch/arm64/crypto/nh-neon-core.S b/arch/arm64/crypto/nh-neon-core.S
index 51c0a534ef87..cb354d3f7e7b 100644
--- a/arch/arm64/crypto/nh-neon-core.S
+++ b/arch/arm64/crypto/nh-neon-core.S
@@ -62,7 +62,7 @@
*
* It's guaranteed that message_len % 16 == 0.
*/
-SYM_FUNC_START(nh_neon)
+SYM_FUNC_START_SECTION(nh_neon)
ld1 {K0.4s,K1.4s}, [KEY], #32
movi PASS0_SUMS.2d, #0
@@ -100,4 +100,4 @@ SYM_FUNC_START(nh_neon)
addp T1.2d, PASS2_SUMS.2d, PASS3_SUMS.2d
st1 {T0.16b,T1.16b}, [HASH]
ret
-SYM_FUNC_END(nh_neon)
+SYM_FUNC_END_SECTION(nh_neon)
diff --git a/arch/arm64/crypto/poly1305-armv8.pl b/arch/arm64/crypto/poly1305-armv8.pl
index cbc980fb02e3..039e6a9ce68c 100644
--- a/arch/arm64/crypto/poly1305-armv8.pl
+++ b/arch/arm64/crypto/poly1305-armv8.pl
@@ -48,8 +48,12 @@ my ($h0,$h1,$h2,$r0,$r1,$s1,$t0,$t1,$d0,$d1,$d2) = map("x$_",(4..14));
$code.=<<___;
#ifndef __KERNEL__
+# define SYM_TEXT_SECTION()
+# define SYM_TEXT_END_SECTION
# include "arm_arch.h"
.extern OPENSSL_armcap_P
+#else
+# include <linux/linkage.h>
#endif
.text
@@ -58,6 +62,7 @@ $code.=<<___;
.globl poly1305_blocks
.globl poly1305_emit
+SYM_TEXT_SECTION(poly1305_init)
.globl poly1305_init
.type poly1305_init,%function
.align 5
@@ -107,7 +112,9 @@ poly1305_init:
.Lno_key:
ret
.size poly1305_init,.-poly1305_init
+SYM_TEXT_END_SECTION
+SYM_TEXT_SECTION(poly1305_blocks)
.type poly1305_blocks,%function
.align 5
poly1305_blocks:
@@ -198,7 +205,9 @@ poly1305_blocks:
.Lno_data:
ret
.size poly1305_blocks,.-poly1305_blocks
+SYM_TEXT_END_SECTION
+SYM_TEXT_SECTION(poly1305_emit)
.type poly1305_emit,%function
.align 5
poly1305_emit:
@@ -258,6 +267,7 @@ poly1305_emit:
ret
.size poly1305_emit,.-poly1305_emit
+SYM_TEXT_END_SECTION
___
my ($R0,$R1,$S1,$R2,$S2,$R3,$S3,$R4,$S4) = map("v$_.4s",(0..8));
my ($IN01_0,$IN01_1,$IN01_2,$IN01_3,$IN01_4) = map("v$_.2s",(9..13));
@@ -270,6 +280,7 @@ my ($in2,$zeros)=("x16","x17");
my $is_base2_26 = $zeros; # borrow
$code.=<<___;
+SYM_TEXT_SECTION(poly1305_mult)
.type poly1305_mult,%function
.align 5
poly1305_mult:
@@ -306,7 +317,9 @@ poly1305_mult:
ret
.size poly1305_mult,.-poly1305_mult
+SYM_TEXT_END_SECTION
+SYM_TEXT_SECTION(poly1305_splat)
.type poly1305_splat,%function
.align 4
poly1305_splat:
@@ -333,7 +346,9 @@ poly1305_splat:
ret
.size poly1305_splat,.-poly1305_splat
+SYM_TEXT_END_SECTION
+SYM_TEXT_SECTION(poly1305_blocks_neon)
#ifdef __KERNEL__
.globl poly1305_blocks_neon
#endif
@@ -888,6 +903,8 @@ poly1305_blocks_neon:
.align 5
.Lzeros:
.long 0,0,0,0,0,0,0,0
+SYM_TEXT_END_SECTION
+
.asciz "Poly1305 for ARMv8, CRYPTOGAMS by \@dot-asm"
.align 2
#if !defined(__KERNEL__) && !defined(_WIN64)
diff --git a/arch/arm64/crypto/sha1-ce-core.S b/arch/arm64/crypto/sha1-ce-core.S
index 889ca0f8972b..2ba5f8ea39fc 100644
--- a/arch/arm64/crypto/sha1-ce-core.S
+++ b/arch/arm64/crypto/sha1-ce-core.S
@@ -65,7 +65,7 @@
* int sha1_ce_transform(struct sha1_ce_state *sst, u8 const *src,
* int blocks)
*/
-SYM_FUNC_START(sha1_ce_transform)
+SYM_FUNC_START_SECTION(sha1_ce_transform)
/* load round constants */
loadrc k0.4s, 0x5a827999, w6
loadrc k1.4s, 0x6ed9eba1, w6
@@ -147,4 +147,4 @@ CPU_LE( rev32 v11.16b, v11.16b )
str dgb, [x0, #16]
mov w0, w2
ret
-SYM_FUNC_END(sha1_ce_transform)
+SYM_FUNC_END_SECTION(sha1_ce_transform)
diff --git a/arch/arm64/crypto/sha2-ce-core.S b/arch/arm64/crypto/sha2-ce-core.S
index 491179922f49..6c1a4a128355 100644
--- a/arch/arm64/crypto/sha2-ce-core.S
+++ b/arch/arm64/crypto/sha2-ce-core.S
@@ -75,7 +75,7 @@
* int blocks)
*/
.text
-SYM_FUNC_START(sha2_ce_transform)
+SYM_FUNC_START_SECTION(sha2_ce_transform)
/* load round constants */
adr_l x8, .Lsha2_rcon
ld1 { v0.4s- v3.4s}, [x8], #64
@@ -154,4 +154,4 @@ CPU_LE( rev32 v19.16b, v19.16b )
3: st1 {dgav.4s, dgbv.4s}, [x0]
mov w0, w2
ret
-SYM_FUNC_END(sha2_ce_transform)
+SYM_FUNC_END_SECTION(sha2_ce_transform)
diff --git a/arch/arm64/crypto/sha3-ce-core.S b/arch/arm64/crypto/sha3-ce-core.S
index 9c77313f5a60..6105cc815c9a 100644
--- a/arch/arm64/crypto/sha3-ce-core.S
+++ b/arch/arm64/crypto/sha3-ce-core.S
@@ -40,7 +40,7 @@
* int sha3_ce_transform(u64 *st, const u8 *data, int blocks, int dg_size)
*/
.text
-SYM_FUNC_START(sha3_ce_transform)
+SYM_FUNC_START_SECTION(sha3_ce_transform)
/* load state */
add x8, x0, #32
ld1 { v0.1d- v3.1d}, [x0]
@@ -197,7 +197,7 @@ SYM_FUNC_START(sha3_ce_transform)
st1 {v24.1d}, [x0]
mov w0, w2
ret
-SYM_FUNC_END(sha3_ce_transform)
+SYM_FUNC_END_SECTION(sha3_ce_transform)
.section ".rodata", "a"
.align 8
diff --git a/arch/arm64/crypto/sha512-armv8.pl b/arch/arm64/crypto/sha512-armv8.pl
index 2d8655d5b1af..7952696d3c88 100644
--- a/arch/arm64/crypto/sha512-armv8.pl
+++ b/arch/arm64/crypto/sha512-armv8.pl
@@ -195,11 +195,16 @@ ___
$code.=<<___;
#ifndef __KERNEL__
# include "arm_arch.h"
+# define SYM_TEXT_SECTION()
+# define SYM_TEXT_END_SECTION
+#else
+# include <linux/linkage.h>
#endif
.text
.extern OPENSSL_armcap_P
+SYM_TEXT_SECTION($func)
.globl $func
.type $func,%function
.align 6
@@ -285,7 +290,9 @@ $code.=<<___;
ldp x29,x30,[sp],#128
ret
.size $func,.-$func
+SYM_TEXT_END_SECTION
+SYM_TEXT_SECTION(K$BITS)
.align 6
.type .LK$BITS,%object
.LK$BITS:
@@ -354,6 +361,8 @@ $code.=<<___ if ($SZ==4);
___
$code.=<<___;
.size .LK$BITS,.-.LK$BITS
+SYM_TEXT_END_SECTION
+
#ifndef __KERNEL__
.align 3
.LOPENSSL_armcap_P:
@@ -637,6 +646,7 @@ sub body_00_15 () {
}
$code.=<<___;
+SYM_TEXT_SECTION(sha256_block_neon)
#ifdef __KERNEL__
.globl sha256_block_neon
#endif
@@ -736,6 +746,7 @@ $code.=<<___;
add sp,sp,#16*4+16
ret
.size sha256_block_neon,.-sha256_block_neon
+SYM_TEXT_END_SECTION
___
}
diff --git a/arch/arm64/crypto/sha512-ce-core.S b/arch/arm64/crypto/sha512-ce-core.S
index b6a3a36e15f5..7d34aabb3daa 100644
--- a/arch/arm64/crypto/sha512-ce-core.S
+++ b/arch/arm64/crypto/sha512-ce-core.S
@@ -106,7 +106,7 @@
* int blocks)
*/
.text
-SYM_FUNC_START(sha512_ce_transform)
+SYM_FUNC_START_SECTION(sha512_ce_transform)
/* load state */
ld1 {v8.2d-v11.2d}, [x0]
@@ -203,4 +203,4 @@ CPU_LE( rev64 v19.16b, v19.16b )
3: st1 {v8.2d-v11.2d}, [x0]
mov w0, w2
ret
-SYM_FUNC_END(sha512_ce_transform)
+SYM_FUNC_END_SECTION(sha512_ce_transform)
diff --git a/arch/arm64/crypto/sm3-ce-core.S b/arch/arm64/crypto/sm3-ce-core.S
index ef97d3187cb7..7be60c41e36d 100644
--- a/arch/arm64/crypto/sm3-ce-core.S
+++ b/arch/arm64/crypto/sm3-ce-core.S
@@ -73,7 +73,7 @@
* int blocks)
*/
.text
-SYM_FUNC_START(sm3_ce_transform)
+SYM_FUNC_START_SECTION(sm3_ce_transform)
/* load state */
ld1 {v8.4s-v9.4s}, [x0]
rev64 v8.4s, v8.4s
@@ -131,7 +131,7 @@ CPU_LE( rev32 v3.16b, v3.16b )
ext v9.16b, v9.16b, v9.16b, #8
st1 {v8.4s-v9.4s}, [x0]
ret
-SYM_FUNC_END(sm3_ce_transform)
+SYM_FUNC_END_SECTION(sm3_ce_transform)
.section ".rodata", "a"
.align 3
diff --git a/arch/arm64/crypto/sm4-ce-core.S b/arch/arm64/crypto/sm4-ce-core.S
index 4ac6cfbc5797..5f64ed209a26 100644
--- a/arch/arm64/crypto/sm4-ce-core.S
+++ b/arch/arm64/crypto/sm4-ce-core.S
@@ -15,7 +15,7 @@
* void sm4_ce_do_crypt(const u32 *rk, u32 *out, const u32 *in);
*/
.text
-SYM_FUNC_START(sm4_ce_do_crypt)
+SYM_FUNC_START_SECTION(sm4_ce_do_crypt)
ld1 {v8.4s}, [x2]
ld1 {v0.4s-v3.4s}, [x0], #64
CPU_LE( rev32 v8.16b, v8.16b )
@@ -33,4 +33,4 @@ CPU_LE( rev32 v8.16b, v8.16b )
CPU_LE( rev32 v8.16b, v8.16b )
st1 {v8.4s}, [x1]
ret
-SYM_FUNC_END(sm4_ce_do_crypt)
+SYM_FUNC_END_SECTION(sm4_ce_do_crypt)
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 11/14] module: Reorder functions
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
` (9 preceding siblings ...)
2021-12-02 22:32 ` [PATCH v8 10/14] arm64/crypto: conditionally place ASM functions into separate sections Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-03 10:23 ` Peter Zijlstra
2021-12-02 22:32 ` [PATCH v8 12/14] module: use a scripted approach for FG-KASLR Alexander Lobakin
` (3 subsequent siblings)
14 siblings, 1 reply; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
From: Kristen Carlson Accardi <kristen@linux.intel.com>
Introduce a new config option to allow modules to be re-ordered
by function. This option can be enabled independently of the
kernel text KASLR or FG_KASLR settings so that it can be used
by architectures that do not support either of these features.
This option will be selected by default if CONFIG_FG_KASLR is
selected.
If a module has functions split out into separate text sections
(i.e. compiled with the -ffunction-sections flag), reorder the
functions to provide some code diversification to modules.
Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Tested-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Tested-by: Tony Luck <tony.luck@intel.com>
Acked-by: Jessica Yu <jeyu@kernel.org>
Tested-by: Jessica Yu <jeyu@kernel.org>
Reported-by: kernel test robot <lkp@intel.com> # swap.cocci
[ alobakin: make it work with ClangCFI ]
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
Makefile | 4 ++
init/Kconfig | 12 ++++++
kernel/kallsyms.c | 4 +-
kernel/livepatch/core.c | 3 +-
kernel/module.c | 91 ++++++++++++++++++++++++++++++++++++++++-
5 files changed, 109 insertions(+), 5 deletions(-)
diff --git a/Makefile b/Makefile
index a4d2eac5f81f..649c309d10e2 100644
--- a/Makefile
+++ b/Makefile
@@ -885,6 +885,10 @@ endif
# ClangLTO implies -ffunction-sections -fdata-sections, no need
# to specify them manually and trigger a pointless full rebuild
ifndef CONFIG_LTO_CLANG
+ifdef CONFIG_MODULE_FG_KASLR
+KBUILD_CFLAGS_MODULE += -ffunction-sections
+endif
+
ifneq ($(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION)$(CONFIG_FG_KASLR),)
KBUILD_CFLAGS_KERNEL += -ffunction-sections
endif
diff --git a/init/Kconfig b/init/Kconfig
index 8baeaef382c9..1f7e57d323bb 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2360,6 +2360,18 @@ config UNUSED_KSYMS_WHITELIST
one per line. The path can be absolute, or relative to the kernel
source tree.
+config MODULE_FG_KASLR
+ bool "Module Function Granular Layout Randomization"
+ default FG_KASLR
+ depends on BROKEN
+ help
+ This option randomizes the module text section by reordering the text
+ section by function at module load time. In order to use this
+ feature, the module must have been compiled with the
+ -ffunction-sections compiler flag.
+
+ If unsure, say N.
+
endif # MODULES
config MODULES_TREE_LOOKUP
diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c
index ff9d8b651966..4ff14a94d1bd 100644
--- a/kernel/kallsyms.c
+++ b/kernel/kallsyms.c
@@ -861,7 +861,7 @@ static int __kallsyms_open(struct inode *inode, struct file *file)
* When function granular kaslr is enabled, we need to print out the symbols
* at random so we don't reveal the new layout.
*/
-#ifdef CONFIG_FG_KASLR
+#if defined(CONFIG_FG_KASLR) || defined(CONFIG_MODULE_FG_KASLR)
static int update_random_pos(struct kallsyms_shuffled_iter *s_iter,
loff_t pos, loff_t *new_pos)
{
@@ -1005,7 +1005,7 @@ static int kallsyms_random_open(struct inode *inode, struct file *file)
#define kallsyms_open kallsyms_random_open
#else
#define kallsyms_open __kallsyms_open
-#endif /* !CONFIG_FG_KASLR */
+#endif /* !CONFIG_FG_KASLR && !CONFIG_MODULE_FG_KASLR */
#ifdef CONFIG_KGDB_KDB
const char *kdb_walk_kallsyms(loff_t *pos)
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 10ea75111057..8a5240b5eb5e 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -174,7 +174,8 @@ static int klp_find_object_symbol(const char *objname, const char *name,
* to resolve symbols when there are duplicates using the previous
* symbol position (i.e. sympos != 0).
*/
- if (IS_ENABLED(CONFIG_FG_KASLR) && sympos) {
+ if ((IS_ENABLED(CONFIG_FG_KASLR) || IS_ENABLED(CONFIG_MODULE_FG_KASLR)) &&
+ sympos) {
pr_err("FG-KASLR is enabled, specifying symbol position %lu for symbol '%s' in object '%s' does not work\n",
sympos, name, objname ? objname : "vmlinux");
goto out_err;
diff --git a/kernel/module.c b/kernel/module.c
index 84a9141a5e15..48d5919d09dd 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -57,6 +57,7 @@
#include <linux/bsearch.h>
#include <linux/dynamic_debug.h>
#include <linux/audit.h>
+#include <linux/random.h>
#include <uapi/linux/module.h>
#include "module-internal.h"
@@ -1527,7 +1528,7 @@ static void free_sect_attrs(struct module_sect_attrs *sect_attrs)
for (section = 0; section < sect_attrs->nsections; section++)
kfree(sect_attrs->attrs[section].battr.attr.name);
- kfree(sect_attrs);
+ kvfree(sect_attrs);
}
static void add_sect_attrs(struct module *mod, const struct load_info *info)
@@ -1544,7 +1545,7 @@ static void add_sect_attrs(struct module *mod, const struct load_info *info)
size[0] = ALIGN(struct_size(sect_attrs, attrs, nloaded),
sizeof(sect_attrs->grp.bin_attrs[0]));
size[1] = (nloaded + 1) * sizeof(sect_attrs->grp.bin_attrs[0]);
- sect_attrs = kzalloc(size[0] + size[1], GFP_KERNEL);
+ sect_attrs = kvzalloc(size[0] + size[1], GFP_KERNEL);
if (sect_attrs == NULL)
return;
@@ -2416,6 +2417,89 @@ static bool module_init_layout_section(const char *sname)
return module_init_section(sname);
}
+/*
+ * shuffle_text_list()
+ * Use a Fisher Yates algorithm to shuffle a list of text sections.
+ */
+static void shuffle_text_list(Elf_Shdr **list, int size)
+{
+ u32 i, j;
+
+ for (i = size - 1; i > 0; i--) {
+ /*
+ * pick a random index from 0 to i
+ */
+ j = get_random_u32() % (i + 1);
+
+ swap(list[i], list[j]);
+ }
+}
+
+/*
+ * randomize_text()
+ * Look through the core section looking for executable code sections.
+ * Store sections in an array and then shuffle the sections
+ * to reorder the functions.
+ */
+static void randomize_text(struct module *mod, struct load_info *info)
+{
+ int max_sections = info->hdr->e_shnum;
+ int num_text_sections = 0;
+ Elf_Shdr **text_list;
+ int i, size;
+
+ text_list = kvmalloc_array(max_sections, sizeof(*text_list), GFP_KERNEL);
+ if (!text_list)
+ return;
+
+ for (i = 0; i < max_sections; i++) {
+ Elf_Shdr *shdr = &info->sechdrs[i];
+ const char *sname = info->secstrings + shdr->sh_name;
+
+ if (!(shdr->sh_flags & SHF_ALLOC) ||
+ !(shdr->sh_flags & SHF_EXECINSTR) ||
+ (shdr->sh_flags & ARCH_SHF_SMALL) ||
+ module_init_layout_section(sname))
+ continue;
+
+ /*
+ * With CONFIG_CFI_CLANG, .text with __cfi_check() must come
+ * before any other text sections, and be aligned to PAGE_SIZE.
+ * Don't include it in the shuffle list.
+ */
+ if (IS_ENABLED(CONFIG_CFI_CLANG) && !strcmp(sname, ".text"))
+ continue;
+
+ if (!num_text_sections)
+ size = shdr->sh_entsize;
+
+ text_list[num_text_sections] = shdr;
+ num_text_sections++;
+ }
+
+ if (!num_text_sections)
+ goto exit;
+
+ shuffle_text_list(text_list, num_text_sections);
+
+ for (i = 0; i < num_text_sections; i++) {
+ Elf_Shdr *shdr = text_list[i];
+
+ /*
+ * get_offset has a section index for it's last
+ * argument, that is only used by arch_mod_section_prepend(),
+ * which is only defined by parisc. Since this type
+ * of randomization isn't supported on parisc, we can
+ * safely pass in zero as the last argument, as it is
+ * ignored.
+ */
+ shdr->sh_entsize = get_offset(mod, &size, shdr, 0);
+ }
+
+exit:
+ kvfree(text_list);
+}
+
/*
* Lay out the SHF_ALLOC sections in a way not dissimilar to how ld
* might -- code, read-only data, read-write data, small data. Tally
@@ -2510,6 +2594,9 @@ static void layout_sections(struct module *mod, struct load_info *info)
break;
}
}
+
+ if (IS_ENABLED(CONFIG_MODULE_FG_KASLR))
+ randomize_text(mod, info);
}
static void set_license(struct module *mod, const char *license)
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 12/14] module: use a scripted approach for FG-KASLR
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
` (10 preceding siblings ...)
2021-12-02 22:32 ` [PATCH v8 11/14] module: Reorder functions Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 13/14] Documentation: add documentation " Alexander Lobakin
` (2 subsequent siblings)
14 siblings, 0 replies; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
Use the same methods and scripts to generate an LD script for every
module containing all the output text sections.
The only difference there is that we don't need to reserve any space
as the memory for every section is being allocated dynamically.
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
.gitignore | 1 +
include/asm-generic/vmlinux.lds.h | 12 ++++++++++++
init/Kconfig | 15 ++++++++++++++-
scripts/Makefile.modfinal | 19 ++++++++++++++++---
scripts/generate_text_sections.pl | 9 ++++++++-
scripts/module.lds.S | 14 +++++++++++++-
6 files changed, 64 insertions(+), 6 deletions(-)
diff --git a/.gitignore b/.gitignore
index 7afd412dadd2..a39d0eb87395 100644
--- a/.gitignore
+++ b/.gitignore
@@ -26,6 +26,7 @@
*.gz
*.i
*.ko
+*.lds
*.lex.c
*.ll
*.lst
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 8ddc08baf50c..13718807c027 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -127,6 +127,18 @@
#define TEXT_MAIN .text
#endif
+/*
+ * Same for modules. However, LD_DEAD_CODE_DATA_ELIMINATION doesn't touch
+ * them, so no need to check for it here.
+ */
+#if defined(CONFIG_LTO_CLANG) && !defined(CONFIG_MODULE_FG_KASLR)
+#define TEXT_MAIN_MODULE .text .text.[0-9a-zA-Z_]*
+#elif defined(CONFIG_MODULE_FG_KASLR)
+#define TEXT_MAIN_MODULE .text.__unused__
+#else
+#define TEXT_MAIN_MODULE .text
+#endif
+
/*
* Used by scripts/generate_text_sections.pl to inject text sections,
* harmless if FG-KASLR is disabled.
diff --git a/init/Kconfig b/init/Kconfig
index 1f7e57d323bb..1cbd0ffcb6c0 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2363,7 +2363,6 @@ config UNUSED_KSYMS_WHITELIST
config MODULE_FG_KASLR
bool "Module Function Granular Layout Randomization"
default FG_KASLR
- depends on BROKEN
help
This option randomizes the module text section by reordering the text
section by function at module load time. In order to use this
@@ -2372,6 +2371,20 @@ config MODULE_FG_KASLR
If unsure, say N.
+config MODULE_FG_KASLR_SHIFT
+ int "Module FG-KASLR granularity (functions per section shift)"
+ depends on MODULE_FG_KASLR
+ range 0 16
+ default 0
+ help
+ This sets the number of functions that will be put in each section
+ as a power of two.
+ Decreasing the value increases the randomization, but also increases
+ the size of the final kernel module due to the amount of sections.
+ 0 means that a separate section will be created for each function.
+ 16 almost disables the randomization, leaving only the manual
+ separation.
+
endif # MODULES
config MODULES_TREE_LOOKUP
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index 7f39599e9fae..9353ce78a74e 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -28,13 +28,24 @@ quiet_cmd_cc_o_c = CC [M] $@
%.mod.o: %.mod.c FORCE
$(call if_changed_dep,cc_o_c)
+ifdef CONFIG_MODULE_FG_KASLR
+quiet_cmd_gen_modules_lds = GEN [M] $@
+ cmd_gen_modules_lds = \
+ $(PERL) $(srctree)/scripts/generate_text_sections.pl \
+ -s $(CONFIG_MODULE_FG_KASLR_SHIFT) $(filter %.o, $^) \
+ < $(filter %.lds, $^) > $@
+
+%.lds: %$(mod-prelink-ext).o scripts/module.lds FORCE
+ $(call if_changed,gen_modules_lds)
+endif
+
ARCH_POSTLINK := $(wildcard $(srctree)/arch/$(SRCARCH)/Makefile.postlink)
quiet_cmd_ld_ko_o = LD [M] $@
cmd_ld_ko_o += \
$(LD) -r $(KBUILD_LDFLAGS) \
$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE) \
- -T scripts/module.lds -o $@ $(filter %.o, $^); \
+ -T $(filter %.lds, $^) -o $@ $(filter %.o, $^); \
$(if $(ARCH_POSTLINK), $(MAKE) -f $(ARCH_POSTLINK) $@, true)
quiet_cmd_btf_ko = BTF [M] $@
@@ -56,13 +67,15 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check), \
# Re-generate module BTFs if either module's .ko or vmlinux changed
-$(modules): %.ko: %$(mod-prelink-ext).o %.mod.o scripts/module.lds $(if $(KBUILD_BUILTIN),vmlinux) FORCE
+$(modules): %.ko: %$(mod-prelink-ext).o %.mod.o
+$(modules): %.ko: $(if $(CONFIG_MODULE_FG_KASLR),%.lds,scripts/module.lds)
+$(modules): %.ko: $(if $(KBUILD_BUILTIN),vmlinux) FORCE
+$(call if_changed_except,ld_ko_o,vmlinux)
ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+$(if $(newer-prereqs),$(call cmd,btf_ko))
endif
-targets += $(modules) $(modules:.ko=.mod.o)
+targets += $(modules) $(modules:.ko=.mod.o) $(if $(CONFIG_MODULE_FG_KASLR),$(modules:.ko=.lds))
# Add FORCE to the prequisites of a target to force it to be always rebuilt.
# ---------------------------------------------------------------------------
diff --git a/scripts/generate_text_sections.pl b/scripts/generate_text_sections.pl
index 6871045fb7a6..d4c5614d9481 100755
--- a/scripts/generate_text_sections.pl
+++ b/scripts/generate_text_sections.pl
@@ -45,6 +45,7 @@ my $readelf = $ENV{'READELF'} || die "$0: ERROR: READELF not set?";
## text sections array
my @sections = ();
my $has_ccf = 0;
+my $vmlinux = 0;
## max alignment found to reserve some space
my $max_align = 64;
@@ -73,6 +74,12 @@ sub read_sections {
$has_ccf = 1;
}
+ ## If we're processing a module, don't reserve any space
+ ## at the end as its sections are being allocated separately.
+ if ($name eq ".sched.text") {
+ $vmlinux = 1;
+ }
+
if (!($name =~ /^\.text\.[0-9a-zA-Z_]*((\.constprop|\.isra|\.part)\.[0-9])*(|\.[0-9cfi]*)$/)) {
next;
}
@@ -132,7 +139,7 @@ sub print_reserve {
## If we have text sections aligned with 64 bytes or more, make
## sure we reserve some space for them to not overlap _etext
## while shuffling sections.
- if (!$count) {
+ if (!$vmlinux or !$count) {
return;
}
diff --git a/scripts/module.lds.S b/scripts/module.lds.S
index 1d0e1e4dc3d2..6e957aa614b1 100644
--- a/scripts/module.lds.S
+++ b/scripts/module.lds.S
@@ -3,6 +3,11 @@
* Archs are free to supply their own linker scripts. ld will
* combine them automatically.
*/
+
+#include <asm-generic/vmlinux.lds.h>
+
+#undef SANITIZER_DISCARDS
+
#ifdef CONFIG_CFI_CLANG
# include <asm/page.h>
# define ALIGN_CFI ALIGN(PAGE_SIZE)
@@ -58,9 +63,16 @@ SECTIONS {
*/
.text : ALIGN_CFI {
*(.text.__cfi_check)
- *(.text .text.[0-9a-zA-Z_]* .text..L.cfi*)
+ *(TEXT_MAIN_MODULE)
+ *(.text..L.cfi.jumptable .text..L.cfi.jumptable.*)
+ }
+#elif defined(CONFIG_MODULE_FG_KASLR)
+ .text : {
+ *(TEXT_MAIN_MODULE)
}
#endif
+
+ TEXT_FG_KASLR
}
/* bring in arch-specific sections */
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 13/14] Documentation: add documentation for FG-KASLR
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
` (11 preceding siblings ...)
2021-12-02 22:32 ` [PATCH v8 12/14] module: use a scripted approach for FG-KASLR Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 14/14] maintainers: add MAINTAINERS entry " Alexander Lobakin
2021-12-03 10:38 ` [PATCH v8 00/14] Function Granular KASLR Peter Zijlstra
14 siblings, 0 replies; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
From: Kristen Carlson Accardi <kristen@linux.intel.com>
Describe the main principles behind the FG-KASLR hardening feature
in a new doc section.
Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
.../admin-guide/kernel-parameters.txt | 6 +
Documentation/security/fgkaslr.rst | 172 ++++++++++++++++++
Documentation/security/index.rst | 1 +
3 files changed, 179 insertions(+)
create mode 100644 Documentation/security/fgkaslr.rst
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 9725c546a0d4..3940b8610140 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -2224,6 +2224,12 @@
kernel and module base offset ASLR (Address Space
Layout Randomization).
+ nofgkaslr [KNL]
+ When CONFIG_FG_KASLR is set, this parameter
+ disables kernel function granular ASLR
+ (Address Space Layout Randomization).
+ See Documentation/security/fgkaslr.rst.
+
kasan_multi_shot
[KNL] Enforce KASAN (Kernel Address Sanitizer) to print
report on every invalid memory access. Without this
diff --git a/Documentation/security/fgkaslr.rst b/Documentation/security/fgkaslr.rst
new file mode 100644
index 000000000000..50dc24f675b5
--- /dev/null
+++ b/Documentation/security/fgkaslr.rst
@@ -0,0 +1,172 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=====================================================================
+Function Granular Kernel Address Space Layout Randomization (fgkaslr)
+=====================================================================
+
+:Date: 6 April 2020
+:Author: Kristen Accardi
+
+Kernel Address Space Layout Randomization (KASLR) was merged into the kernel
+with the objective of increasing the difficulty of code reuse attacks. Code
+reuse attacks reused existing code snippets to get around existing memory
+protections. They exploit software bugs which expose addresses of useful code
+snippets to control the flow of execution for their own nefarious purposes.
+KASLR as it was originally implemented moves the entire kernel code text as a
+unit at boot time in order to make addresses less predictable. The order of the
+code within the segment is unchanged - only the base address is shifted. There
+are a few shortcomings to this algorithm.
+
+1. Low Entropy - there are only so many locations the kernel can fit in. This
+ means an attacker could guess without too much trouble.
+2. Knowledge of a single address can reveal the offset of the base address,
+ exposing all other locations for a published/known kernel image.
+3. Info leaks abound.
+
+Finer grained ASLR has been proposed as a way to make ASLR more resistant
+to info leaks. It is not a new concept at all, and there are many variations
+possible. Function reordering is an implementation of finer grained ASLR
+which randomizes the layout of an address space on a function level
+granularity. The term "fgkaslr" is used in this document to refer to the
+technique of function reordering when used with KASLR, as well as finer grained
+KASLR in general.
+
+The objective of this patch set is to improve a technology that is already
+merged into the kernel (KASLR). This code will not prevent all code reuse
+attacks, and should be considered as one of several tools that can be used.
+
+Implementation Details
+======================
+
+The over-arching objective of the fgkaslr implementation is incremental
+improvement over the existing KASLR algorithm. It is designed to work with
+the existing solution, and there are two main area where code changes occur:
+Build time, and Load time.
+
+Build time
+----------
+
+GCC has had an option to place functions into individual .text sections
+for many years now (-ffunction-sections). This option is used to implement
+function reordering at load time. The final compiled vmlinux retains all the
+section headers, which can be used to help find the address ranges of each
+function. Using this information and an expanded table of relocation addresses,
+individual text sections can be shuffled immediately after decompression.
+Some data tables inside the kernel that have assumptions about order
+require sorting after the update. In order to modify these tables,
+a few key symbols from the objcopy symbol stripping process are preserved
+for use after shuffling the text segments. Any special input sections which are
+defined by the kernel build process and collected into the .text output
+segment are left unmodified and will still be present inside the .text segment,
+unrandomized other than normal base address randomization.
+
+Load time
+---------
+
+The boot kernel was modified to parse the vmlinux elf file after
+decompression to check for symbols for modifying data tables, and to
+look for any .text.* sections to randomize. The sections are then shuffled,
+and tables are updated or resorted. The existing code which updated relocation
+addresses was modified to account for not just a fixed delta from the load
+address, but the offset that the function section was moved to. This requires
+inspection of each address to see if it was impacted by a randomization.
+
+In order to hide the new layout, symbols reported through /proc/kallsyms will
+be displayed in a random order.
+
+Performance Impact
+==================
+
+There are two areas where function reordering can impact performance: boot
+time latency, and run time performance.
+
+Boot time latency
+-----------------
+
+This implementation of finer grained KASLR impacts the boot time of the kernel
+in several places. It requires additional parsing of the kernel ELF file to
+obtain the section headers of the sections to be randomized. It calls the
+random number generator for each section to be randomized to determine that
+section's new memory location. It copies the decompressed kernel into a new
+area of memory to avoid corruption when laying out the newly randomized
+sections. It increases the number of relocations the kernel has to perform at
+boot time vs. standard KASLR, and it also requires a lookup on each address
+that needs to be relocated to see if it was in a randomized section and needs
+to be adjusted by a new offset. Finally, it re-sorts a few data tables that
+are required to be sorted by address.
+
+Booting a test VM on a modern, well appointed system showed an increase in
+latency of approximately 1 second.
+
+Run time
+--------
+
+The performance impact at run-time of function reordering varies by workload.
+Randomly reordering the functions will cause an increase in cache misses
+for some workloads. Some workloads perform significantly worse under FGKASLR,
+while others stay the same or even improve. In general, it will depend on the
+code flow whether or not finer grained KASLR will impact a workload, and how
+the underlying code was designed. Because the layout changes per boot, each
+time a system is rebooted the performance of a workload may change.
+
+Image Size
+==========
+
+fgkaslr increases the size of the kernel binary due to the extra section
+headers that are included, as well as the extra relocations that need to
+be added. You can expect fgkaslr to increase the size of the resulting
+vmlinux by about 3%, and the compressed image (bzImage) by 15%.
+
+Memory Usage
+============
+
+fgkaslr increases the amount of heap that is required at boot time,
+although this extra memory is released when the kernel has finished
+decompression. As a result, it may not be appropriate to use this feature
+on systems without much memory.
+
+Building
+========
+
+To enable fine grained KASLR, you need to have the following config options
+set (including all the ones you would use to build normal KASLR)
+
+``CONFIG_FG_KASLR=y``
+
+fgkaslr for the kernel is only supported for the X86_64 architecture.
+
+Modules
+=======
+
+Modules are randomized similarly to the rest of the kernel by shuffling
+the sections at load time prior to moving them into memory. The module must
+also have been build with the -ffunction-sections compiler option.
+
+Although fgkaslr for the kernel is only supported for the X86_64 architecture,
+it is possible to use fgkaslr with modules on other architectures. To enable
+this feature, select the following config option:
+
+``CONFIG_MODULE_FG_KASLR``
+
+This option is selected automatically for X86_64 when CONFIG_FG_KASLR is set.
+
+Disabling
+=========
+
+Disabling normal kaslr using the nokaslr command line option also disables
+fgkaslr. In addition, it is possible to disable fgkaslr separately by booting
+with "nofgkaslr" on the commandline.
+
+Further Information
+===================
+
+There are a lot of academic papers which explore finer grained ASLR.
+This paper in particular contributed significantly to the implementation design.
+
+Selfrando: Securing the Tor Browser against De-anonymization Exploits,
+M. Conti, S. Crane, T. Frassetto, et al.
+
+For more information on how function layout impacts performance, see:
+
+Optimizing Function Placement for Large-Scale Data-Center Applications,
+G. Ottoni, B. Maher
diff --git a/Documentation/security/index.rst b/Documentation/security/index.rst
index 16335de04e8c..41444124090f 100644
--- a/Documentation/security/index.rst
+++ b/Documentation/security/index.rst
@@ -7,6 +7,7 @@ Security Documentation
credentials
IMA-templates
+ fgkaslr
keys/index
lsm
lsm-development
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v8 14/14] maintainers: add MAINTAINERS entry for FG-KASLR
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
` (12 preceding siblings ...)
2021-12-02 22:32 ` [PATCH v8 13/14] Documentation: add documentation " Alexander Lobakin
@ 2021-12-02 22:32 ` Alexander Lobakin
2021-12-03 10:38 ` [PATCH v8 00/14] Function Granular KASLR Peter Zijlstra
14 siblings, 0 replies; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-02 22:32 UTC (permalink / raw)
To: linux-hardening, x86
Cc: Alexander Lobakin, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnd Bergmann, Josh Poimboeuf, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
Add an entry for FG-KASLR containing the maintainers, reviewers,
public mailing lists, files and so on.
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
MAINTAINERS | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 360e9aa0205d..336cae4c08b4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7853,6 +7853,18 @@ L: platform-driver-x86@vger.kernel.org
S: Maintained
F: drivers/platform/x86/fujitsu-tablet.c
+FUNCTION-GRAINED KASLR (FG-KASLR)
+M: Alexander Lobakin <alexandr.lobakin@intel.com>
+R: Kristen Carlson Accardi <kristen@linux.intel.com>
+R: Kees Cook <keescook@chromium.org>
+L: linux-hardening@vger.kernel.org
+S: Supported
+F: Documentation/security/fgkaslr.rst
+F: arch/x86/boot/compressed/fgkaslr.c
+F: arch/x86/boot/compressed/utils.c
+F: arch/x86/boot/compressed/vmlinux.symbols
+F: scripts/generate_text_sections.pl
+
FUSE: FILESYSTEM IN USERSPACE
M: Miklos Szeredi <miklos@szeredi.hu>
L: linux-fsdevel@vger.kernel.org
--
2.33.1
^ permalink raw reply related [flat|nested] 40+ messages in thread
* Re: [PATCH v8 03/14] x86: Add support for function granular KASLR
2021-12-02 22:32 ` [PATCH v8 03/14] x86: Add support for function granular KASLR Alexander Lobakin
@ 2021-12-03 9:18 ` Peter Zijlstra
2021-12-03 13:57 ` Alexander Lobakin
2021-12-03 10:34 ` Peter Zijlstra
1 sibling, 1 reply; 40+ messages in thread
From: Peter Zijlstra @ 2021-12-03 9:18 UTC (permalink / raw)
To: Alexander Lobakin
Cc: linux-hardening, x86, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, linux-kernel, linux-kbuild,
linux-arch, live-patching, llvm
On Thu, Dec 02, 2021 at 11:32:03PM +0100, Alexander Lobakin wrote:
> From: Kristen Carlson Accardi <kristen@linux.intel.com>
>
> This commit contains the changes required to re-layout the kernel text
> sections generated by -ffunction-sections shortly after decompression.
> Documentation of the feature is also added.
>
> After decompression, the decompressed image's elf headers are parsed.
> In order to manually update certain data structures that are built with
> relative offsets during the kernel build process, certain symbols are
> not stripped by objdump and their location is retained in the elf symbol
> tables. These addresses are saved.
>
> If the image was built with -ffunction-sections, there will be ELF section
> headers present which contain information about the address range of each
> section. Anything that is not broken out into function sections (i.e. is
> consolidated into .text) is left in it's original location, but any other
> executable section which begins with ".text." is located and shuffled
> randomly within the remaining text segment address range.
>
> After the sections have been copied to their new locations, but before
> relocations have been applied, the kallsyms tables must be updated to
> reflect the new symbol locations. Because it is expected that these tables
> will be sorted by address, the kallsyms tables will need to be sorted
> after the update.
>
> When applying relocations, the address of the relocation needs to be
> adjusted by the offset from the original location of the section that was
> randomized to it's new location. In addition, if a value at that relocation
> was a location in the text segment that was randomized, it's value will be
> adjusted to a new location.
>
> After relocations have been applied, the exception table must be updated
> with new symbol locations, and then re-sorted by the new address. The
> orc table will have been updated as part of applying relocations, but since
> it is expected to be sorted by address, it will need to be resorted.
> +static long addr_kallsyms_names;
> +static long addr_kallsyms_offsets;
> +static long addr_kallsyms_num_syms;
> +static long addr_kallsyms_relative_base;
> +static long addr_kallsyms_markers;
> +static long addr___start___ex_table;
> +static long addr___stop___ex_table;
> +static long addr___altinstr_replacement;
> +static long addr___altinstr_replacement_end;
> +static long addr__stext;
> +static long addr__etext;
> +static long addr__sinittext;
> +static long addr__einittext;
> +static long addr___start_orc_unwind_ip;
> +static long addr___stop_orc_unwind_ip;
> +static long addr___start_orc_unwind;
> +void post_relocations_cleanup(unsigned long map)
> +{
> + if (!nofgkaslr) {
> + update_ex_table(map);
> + sort_ex_table(map);
> + update_orc_table(map);
> + sort_orc_table(map);
> + }
> +
> + /*
> + * maybe one day free will do something. So, we "free" this memory
> + * in either case
> + */
> + free(sections);
> + free(sechdrs);
> +}
> +
> +void pre_relocations_cleanup(unsigned long map)
> +{
> + if (nofgkaslr)
> + return;
> +
> + sort_kallsyms(map);
> +}
> diff --git a/arch/x86/boot/compressed/vmlinux.symbols b/arch/x86/boot/compressed/vmlinux.symbols
> new file mode 100644
> index 000000000000..da41f3ee153c
> --- /dev/null
> +++ b/arch/x86/boot/compressed/vmlinux.symbols
> @@ -0,0 +1,19 @@
> +kallsyms_offsets
> +kallsyms_addresses
> +kallsyms_num_syms
> +kallsyms_relative_base
> +kallsyms_names
> +kallsyms_token_table
> +kallsyms_token_index
> +kallsyms_markers
> +__start___ex_table
> +__stop___ex_table
> +__altinstr_replacement
> +__altinstr_replacement_end
> +_sinittext
> +_einittext
> +_stext
> +_etext
> +__start_orc_unwind_ip
> +__stop_orc_unwind_ip
> +__start_orc_unwind
So please don't make it hard to add sections; the above has far too much
duplication. For example you can trivially generate the addr_ symbol and
the .symbol file from a common include file and a bit of macro wrappery,
ideally that macro wrappery would also specify the sort location and
function such that you can also generate those pre_ and post_ functions.
And this is only for sections that need to be sorted right? There's
currently a patch in flight to also pre-sort the ftrace table.
All unsorted or runtime sorted sections are fine since they're fixed up
by the relocations?
Is it at all feasible to share the comparison functions between the
various sorters?
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 04/14] linkage: add macros for putting ASM functions into own sections
2021-12-02 22:32 ` [PATCH v8 04/14] linkage: add macros for putting ASM functions into own sections Alexander Lobakin
@ 2021-12-03 9:31 ` Peter Zijlstra
2021-12-03 14:08 ` Alexander Lobakin
0 siblings, 1 reply; 40+ messages in thread
From: Peter Zijlstra @ 2021-12-03 9:31 UTC (permalink / raw)
To: Alexander Lobakin
Cc: linux-hardening, x86, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, linux-kernel, linux-kbuild,
linux-arch, live-patching, llvm
On Thu, Dec 02, 2021 at 11:32:04PM +0100, Alexander Lobakin wrote:
> diff --git a/include/linux/linkage.h b/include/linux/linkage.h
> index dbf8506decca..baaab7dece08 100644
> --- a/include/linux/linkage.h
> +++ b/include/linux/linkage.h
> @@ -355,4 +355,86 @@
>
> #endif /* __ASSEMBLY__ */
>
> +/*
> + * Allow ASM symbols to have their own unique sections if they are being
> + * generated by the compiler for C functions (DCE, FG-KASLR, LTO).
> + */
> +#if (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) && !defined(MODULE)) || \
> + (defined(CONFIG_FG_KASLR) && !defined(MODULE)) || \
> + (defined(CONFIG_MODULE_FG_KASLR) && defined(MODULE)) || \
> + (defined(CONFIG_LTO_CLANG))
> +
> +#define SYM_TEXT_SECTION(name) \
> + .pushsection .text.##name, "ax"
> +
> +#define ASM_TEXT_SECTION(name) \
> + ".text." #name
> +
> +#define ASM_PUSH_SECTION(name) \
> + ".pushsection .text." #name ", \"ax\""
> +
> +#else /* just .text */
> +
> +#define SYM_TEXT_SECTION(name) \
> + .pushsection .text, "ax"
> +
> +#define ASM_TEXT_SECTION(name) \
> + ".text"
> +
> +#define ASM_PUSH_SECTION(name) \
> + ".pushsection .text, \"ax\""
> +
> +#endif /* just .text */
That's terribly inconsistent, SYM_TEXT_SECTION is in fact
PUSH_TEXT_SECTION, ASM_PUSH_SECTION is in fact ASM_PUSH_TEXT_SECTION and
should be stringify(PUSH_TEXT_SECTION()) or something, and they're all
repeating that ASM_TEXT_SECTION thing :/
> +
> +#ifdef __ASSEMBLY__
> +
> +#define SYM_TEXT_END_SECTION \
> + .popsection
> +
> +#define SYM_FUNC_START_LOCAL_ALIAS_SECTION(name) \
> + SYM_TEXT_SECTION(name) ASM_NL \
> + SYM_FUNC_START_LOCAL_ALIAS(name)
> +
> +#define SYM_FUNC_START_LOCAL_SECTION(name) \
> + SYM_TEXT_SECTION(name) ASM_NL \
> + SYM_FUNC_START_LOCAL(name)
> +
> +#define SYM_FUNC_START_NOALIGN_SECTION(name) \
> + SYM_TEXT_SECTION(name) ASM_NL \
> + SYM_FUNC_START_NOALIGN(name)
> +
> +#define SYM_FUNC_START_WEAK_SECTION(name) \
> + SYM_TEXT_SECTION(name) ASM_NL \
> + SYM_FUNC_START_WEAK(name)
> +
> +#define SYM_FUNC_START_SECTION(name) \
> + SYM_TEXT_SECTION(name) ASM_NL \
> + SYM_FUNC_START(name)
> +
> +#define SYM_CODE_START_LOCAL_NOALIGN_SECTION(name) \
> + SYM_TEXT_SECTION(name) ASM_NL \
> + SYM_CODE_START_LOCAL_NOALIGN(name)
> +
> +#define SYM_CODE_START_NOALIGN_SECTION(name) \
> + SYM_TEXT_SECTION(name) ASM_NL \
> + SYM_CODE_START_NOALIGN(name)
> +
> +#define SYM_CODE_START_SECTION(name) \
> + SYM_TEXT_SECTION(name) ASM_NL \
> + SYM_CODE_START(name)
> +
> +#define SYM_FUNC_END_ALIAS_SECTION(name) \
> + SYM_FUNC_END_ALIAS(name) ASM_NL \
> + SYM_TEXT_END_SECTION
> +
> +#define SYM_FUNC_END_SECTION(name) \
> + SYM_FUNC_END(name) ASM_NL \
> + SYM_TEXT_END_SECTION
> +
> +#define SYM_CODE_END_SECTION(name) \
> + SYM_CODE_END(name) ASM_NL \
> + SYM_TEXT_END_SECTION
> +
> +#endif /* __ASSEMBLY__ */
*URGH* why do we have to have new macros for this? SYM_FUNC_START*()
already takes the name as argument.
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections
2021-12-02 22:32 ` [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections Alexander Lobakin
@ 2021-12-03 9:44 ` Peter Zijlstra
2021-12-03 14:10 ` Alexander Lobakin
2021-12-10 11:01 ` Alexander Lobakin
0 siblings, 2 replies; 40+ messages in thread
From: Peter Zijlstra @ 2021-12-03 9:44 UTC (permalink / raw)
To: Alexander Lobakin
Cc: linux-hardening, x86, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, linux-kernel, linux-kbuild,
linux-arch, live-patching, llvm, hjl.tools
On Thu, Dec 02, 2021 at 11:32:05PM +0100, Alexander Lobakin wrote:
> Use the newly introduces macros to create unique separate sections
> for (almost) every "regular" ASM function (i.e. for those which
> aren't explicitly put into a specific one).
> There should be no leftovers as input .text will be size-asserted
> in the LD script generated for FG-KASLR.
*groan*...
Please, can't we do something like:
#define SYM_PUSH_SECTION(name) \
.if section == .text \
.push_section .text.##name \
.else \
.push_section .text \
.endif
#define SYM_POP_SECTION() \
.pop_section
and wrap that inside the existing SYM_FUNC_START*() SYM_FUNC_END()
macros.
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 06/14] FG-KASLR: use a scripted approach to handle .text.* sections
2021-12-02 22:32 ` [PATCH v8 06/14] FG-KASLR: use a scripted approach to handle .text.* sections Alexander Lobakin
@ 2021-12-03 9:54 ` Peter Zijlstra
0 siblings, 0 replies; 40+ messages in thread
From: Peter Zijlstra @ 2021-12-03 9:54 UTC (permalink / raw)
To: Alexander Lobakin
Cc: linux-hardening, x86, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, linux-kernel, linux-kbuild,
linux-arch, live-patching, llvm
On Thu, Dec 02, 2021 at 11:32:06PM +0100, Alexander Lobakin wrote:
> +#!/usr/bin/env perl
> + ($name, $align) = $_ =~ /^\s*\[[\s0-9]*\]\s*(\.\S*)\s*[A-Z]*\s*[0-9a-f]{16}\s*[0-9a-f]*\s*[0-9a-f]*\s*[0-9a-f]*\s*[0-9a-f]{2}\s*[A-Z]{2}\s*[0-9]\s*[0-9]\s*([0-9]*)$/;
> + if (!($name =~ /^\.text\.[0-9a-zA-Z_]*((\.constprop|\.isra|\.part)\.[0-9])*(|\.[0-9cfi]*)$/)) {
:-(
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 07/14] kallsyms: Hide layout
2021-12-02 22:32 ` [PATCH v8 07/14] kallsyms: Hide layout Alexander Lobakin
@ 2021-12-03 10:00 ` Peter Zijlstra
2021-12-03 10:03 ` Ard Biesheuvel
0 siblings, 1 reply; 40+ messages in thread
From: Peter Zijlstra @ 2021-12-03 10:00 UTC (permalink / raw)
To: Alexander Lobakin
Cc: linux-hardening, x86, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, linux-kernel, linux-kbuild,
linux-arch, live-patching, llvm
On Thu, Dec 02, 2021 at 11:32:07PM +0100, Alexander Lobakin wrote:
> From: Kristen Carlson Accardi <kristen@linux.intel.com>
>
> This patch makes /proc/kallsyms display in a random order, rather
> than sorted by address in order to hide the newly randomized address
> layout.
Is there a reason to not always do this? That is, why are we keeping two
copies of this code around? Less code is more better etc..
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 07/14] kallsyms: Hide layout
2021-12-03 10:00 ` Peter Zijlstra
@ 2021-12-03 10:03 ` Ard Biesheuvel
2021-12-07 5:31 ` Josh Poimboeuf
0 siblings, 1 reply; 40+ messages in thread
From: Ard Biesheuvel @ 2021-12-03 10:03 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Alexander Lobakin, linux-hardening, X86 ML, Jesse Brandeburg,
Kristen Carlson Accardi, Kees Cook, Miklos Szeredi, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, Linux Kernel Mailing List,
Linux Kbuild mailing list, linux-arch, live-patching, llvm
On Fri, 3 Dec 2021 at 11:01, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Thu, Dec 02, 2021 at 11:32:07PM +0100, Alexander Lobakin wrote:
> > From: Kristen Carlson Accardi <kristen@linux.intel.com>
> >
> > This patch makes /proc/kallsyms display in a random order, rather
> > than sorted by address in order to hide the newly randomized address
> > layout.
>
> Is there a reason to not always do this? That is, why are we keeping two
> copies of this code around? Less code is more better etc..
+1.
IIRC I made the exact same point when this patch was sent out by
Kristen a while ago.
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 08/14] livepatch: only match unique symbols when using FG-KASLR
2021-12-02 22:32 ` [PATCH v8 08/14] livepatch: only match unique symbols when using FG-KASLR Alexander Lobakin
@ 2021-12-03 10:05 ` Peter Zijlstra
2021-12-03 14:14 ` Alexander Lobakin
2021-12-06 6:03 ` Josh Poimboeuf
0 siblings, 2 replies; 40+ messages in thread
From: Peter Zijlstra @ 2021-12-03 10:05 UTC (permalink / raw)
To: Alexander Lobakin
Cc: linux-hardening, x86, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, linux-kernel, linux-kbuild,
linux-arch, live-patching, llvm
On Thu, Dec 02, 2021 at 11:32:08PM +0100, Alexander Lobakin wrote:
> If any type of function granular randomization is enabled, the sympos
> algorithm will fail, as it will be impossible to resolve symbols when
> there are duplicates using the previous symbol position.
>
> We could override sympos to 0, but make it more clear to the user
> and bail out if the symbol is not unique.
Since we're going lots of horrendous things already, why can't we fix
this duplicate nonsense too?
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 11/14] module: Reorder functions
2021-12-02 22:32 ` [PATCH v8 11/14] module: Reorder functions Alexander Lobakin
@ 2021-12-03 10:23 ` Peter Zijlstra
0 siblings, 0 replies; 40+ messages in thread
From: Peter Zijlstra @ 2021-12-03 10:23 UTC (permalink / raw)
To: Alexander Lobakin
Cc: linux-hardening, x86, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, linux-kernel, linux-kbuild,
linux-arch, live-patching, llvm
On Thu, Dec 02, 2021 at 11:32:11PM +0100, Alexander Lobakin wrote:
> +/*
> + * shuffle_text_list()
> + * Use a Fisher Yates algorithm to shuffle a list of text sections.
> + */
> +static void shuffle_text_list(Elf_Shdr **list, int size)
> +{
> + u32 i, j;
> +
> + for (i = size - 1; i > 0; i--) {
> + /*
> + * pick a random index from 0 to i
> + */
> + j = get_random_u32() % (i + 1);
> +
> + swap(list[i], list[j]);
> + }
> +}
I'm sure I've seen pretty much that exact function earlier in this
series; does we really need two of them?
#define shuffle_me_harder(_base, _size, _nr) \
do { \
struct { unsigned char _[_size]; } _t, *_a = (void *)(_base); \
int _i, _j; \
for (_i = (_nr)-1; _i > 0; _i--) { \
_j = get_random_u32() % (_i + 1); \
_t = _a[_i]; \
_a[_i] = _a[_j]; \
_a[_j] = _t; \
} \
} while (0)
#define shuffle_array(_array) shuffle_me_harder(_array, sizeof(_array[0]), sizeof(_array)/sizeof(_array[0]))
Or something like that...
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 03/14] x86: Add support for function granular KASLR
2021-12-02 22:32 ` [PATCH v8 03/14] x86: Add support for function granular KASLR Alexander Lobakin
2021-12-03 9:18 ` Peter Zijlstra
@ 2021-12-03 10:34 ` Peter Zijlstra
1 sibling, 0 replies; 40+ messages in thread
From: Peter Zijlstra @ 2021-12-03 10:34 UTC (permalink / raw)
To: Alexander Lobakin
Cc: linux-hardening, x86, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, linux-kernel, linux-kbuild,
linux-arch, live-patching, llvm
On Thu, Dec 02, 2021 at 11:32:03PM +0100, Alexander Lobakin wrote:
> +static struct orc_entry *cur_orc_table;
> +static int *cur_orc_ip_table;
> +static int cmp_section_addr_orc(const void *a, const void *b)
> +{
> + unsigned long ptr = (unsigned long)a;
> + Elf_Shdr *s = *(Elf_Shdr **)b;
> + unsigned long end = s->sh_addr + s->sh_size;
> +
> + /* orc relocations can be one past the end of the section */
> + if (ptr >= s->sh_addr && ptr <= end)
> + return 0;
> +
> + if (ptr < s->sh_addr)
> + return -1;
> +
> + return 1;
> +}
> +
> +/*
> + * Discover if the orc_unwind address is in a randomized section and if so,
> + * adjust by the saved offset.
> + */
> +Elf_Shdr *adjust_address_orc(long *address)
> +{
> + Elf_Shdr **s;
> + Elf_Shdr *shdr;
> +
> + if (nofgkaslr)
> + return NULL;
> +
> + s = bsearch((const void *)*address, sections, sections_size, sizeof(*s),
> + cmp_section_addr_orc);
> + if (s) {
> + shdr = *s;
> + *address += shdr->sh_offset;
> + return shdr;
> + }
> +
> + return NULL;
> +}
> +static inline unsigned long orc_ip(const int *ip)
> +{
> + return (unsigned long)ip + *ip;
> +}
> +
> +static void orc_sort_swap(void *_a, void *_b, int size)
> +{
> + struct orc_entry *orc_a, *orc_b;
> + struct orc_entry orc_tmp;
> + int *a = _a, *b = _b, tmp;
> + int delta = _b - _a;
> +
> + /* Swap the .orc_unwind_ip entries: */
> + tmp = *a;
> + *a = *b + delta;
> + *b = tmp - delta;
> +
> + /* Swap the corresponding .orc_unwind entries: */
> + orc_a = cur_orc_table + (a - cur_orc_ip_table);
> + orc_b = cur_orc_table + (b - cur_orc_ip_table);
> + orc_tmp = *orc_a;
> + *orc_a = *orc_b;
> + *orc_b = orc_tmp;
> +}
> +
> +static int orc_sort_cmp(const void *_a, const void *_b)
> +{
> + struct orc_entry *orc_a;
> + const int *a = _a, *b = _b;
> + unsigned long a_val = orc_ip(a);
> + unsigned long b_val = orc_ip(b);
> +
> + if (a_val > b_val)
> + return 1;
> + if (a_val < b_val)
> + return -1;
> +
> + /*
> + * The "weak" section terminator entries need to always be on the left
> + * to ensure the lookup code skips them in favor of real entries.
> + * These terminator entries exist to handle any gaps created by
> + * whitelisted .o files which didn't get objtool generation.
> + */
> + orc_a = cur_orc_table + (a - cur_orc_ip_table);
> + return orc_a->sp_reg == ORC_REG_UNDEFINED && !orc_a->end ? -1 : 1;
> +}
> +
> +static void update_orc_table(unsigned long map)
> +{
> + int i;
> + int num_entries =
> + (addr___stop_orc_unwind_ip - addr___start_orc_unwind_ip) / sizeof(int);
> +
> + cur_orc_ip_table = (int *)(addr___start_orc_unwind_ip + map);
> + cur_orc_table = (struct orc_entry *)(addr___start_orc_unwind + map);
> +
> + debug_putstr("\nUpdating orc tables...\n");
> + for (i = 0; i < num_entries; i++) {
> + unsigned long ip = orc_ip(&cur_orc_ip_table[i]);
> + Elf_Shdr *s;
> +
> + /* check each address to see if it needs adjusting */
> + ip = ip - map;
> +
> + /*
> + * objtool places terminator entries just outside the end of
> + * the section. To identify an orc_unwind_ip address that might
> + * need adjusting, the address should be compared differently
> + * than a normal address.
> + */
> + s = adjust_address_orc(&ip);
> + if (s)
> + cur_orc_ip_table[i] += s->sh_offset;
> + }
> +}
> +
> +static void sort_orc_table(unsigned long map)
> +{
> + int num_entries =
> + (addr___stop_orc_unwind_ip - addr___start_orc_unwind_ip) / sizeof(int);
> +
> + cur_orc_ip_table = (int *)(addr___start_orc_unwind_ip + map);
> + cur_orc_table = (struct orc_entry *)(addr___start_orc_unwind + map);
> +
> + debug_putstr("\nRe-sorting orc tables...\n");
> + sort(cur_orc_ip_table, num_entries, sizeof(int), orc_sort_cmp,
> + orc_sort_swap);
> +}
Is this somehow different from what we already have in
arch/x86/kernel/unwind_orc.c for module support? Do we really need two
copies of all that?
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 00/14] Function Granular KASLR
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
` (13 preceding siblings ...)
2021-12-02 22:32 ` [PATCH v8 14/14] maintainers: add MAINTAINERS entry " Alexander Lobakin
@ 2021-12-03 10:38 ` Peter Zijlstra
2021-12-03 14:41 ` Alexander Lobakin
14 siblings, 1 reply; 40+ messages in thread
From: Peter Zijlstra @ 2021-12-03 10:38 UTC (permalink / raw)
To: Alexander Lobakin
Cc: linux-hardening, x86, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, linux-kernel, linux-kbuild,
linux-arch, live-patching, llvm
On Thu, Dec 02, 2021 at 11:32:00PM +0100, Alexander Lobakin wrote:
> feat make -j65 boot vmlinux.o vmlinux bzImage bogoops/s
> Relocatable 4m38.478s 24.440s 72014208 58579520 9396192 57640.39
> KASLR 4m39.344s 24.204s 72020624 87805776 9740352 57393.80
> FG-K 16 fps 6m16.493s 25.429s 83759856 87194160 10885632 57784.76
> FG-K 8 fps 6m20.190s 25.094s 83759856 88741328 10985248 56625.84
> FG-K 1 fps 7m09.611s 25.922s 83759856 95681128 11352192 56953.99
:sadface: so at best it makes my kernel compiles ~50% slower. Who would
ever consider doing that? It's like retpolines weren't bad enough; lets
heap on the fail?
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 03/14] x86: Add support for function granular KASLR
2021-12-03 9:18 ` Peter Zijlstra
@ 2021-12-03 13:57 ` Alexander Lobakin
0 siblings, 0 replies; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-03 13:57 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Alexander Lobakin, linux-hardening, x86, Jesse Brandeburg,
Kristen Carlson Accardi, Kees Cook, Miklos Szeredi,
Ard Biesheuvel, Tony Luck, Bruce Schlobohm, Jessica Yu,
kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, Nick Desaulniers,
Herbert Xu, David S. Miller, Thomas Gleixner, Will Deacon,
Ingo Molnar, Borislav Petkov, Dave Hansen, H. Peter Anvin,
Andy Lutomirski, Arnd Bergmann, Josh Poimboeuf,
Nathan Chancellor, Masami Hiramatsu, Marios Pomonis,
Sami Tolvanen, linux-kernel, linux-kbuild, linux-arch,
live-patching, llvm
From: Peter Zijlstra <peterz@infradead.org>
Date: Fri, 3 Dec 2021 10:18:47 +0100
> On Thu, Dec 02, 2021 at 11:32:03PM +0100, Alexander Lobakin wrote:
> > From: Kristen Carlson Accardi <kristen@linux.intel.com>
> >
> > This commit contains the changes required to re-layout the kernel text
> > sections generated by -ffunction-sections shortly after decompression.
> > Documentation of the feature is also added.
> >
> > After decompression, the decompressed image's elf headers are parsed.
> > In order to manually update certain data structures that are built with
> > relative offsets during the kernel build process, certain symbols are
> > not stripped by objdump and their location is retained in the elf symbol
> > tables. These addresses are saved.
> >
> > If the image was built with -ffunction-sections, there will be ELF section
> > headers present which contain information about the address range of each
> > section. Anything that is not broken out into function sections (i.e. is
> > consolidated into .text) is left in it's original location, but any other
> > executable section which begins with ".text." is located and shuffled
> > randomly within the remaining text segment address range.
> >
> > After the sections have been copied to their new locations, but before
> > relocations have been applied, the kallsyms tables must be updated to
> > reflect the new symbol locations. Because it is expected that these tables
> > will be sorted by address, the kallsyms tables will need to be sorted
> > after the update.
> >
> > When applying relocations, the address of the relocation needs to be
> > adjusted by the offset from the original location of the section that was
> > randomized to it's new location. In addition, if a value at that relocation
> > was a location in the text segment that was randomized, it's value will be
> > adjusted to a new location.
> >
> > After relocations have been applied, the exception table must be updated
> > with new symbol locations, and then re-sorted by the new address. The
> > orc table will have been updated as part of applying relocations, but since
> > it is expected to be sorted by address, it will need to be resorted.
>
>
> > +static long addr_kallsyms_names;
> > +static long addr_kallsyms_offsets;
> > +static long addr_kallsyms_num_syms;
> > +static long addr_kallsyms_relative_base;
> > +static long addr_kallsyms_markers;
> > +static long addr___start___ex_table;
> > +static long addr___stop___ex_table;
> > +static long addr___altinstr_replacement;
> > +static long addr___altinstr_replacement_end;
> > +static long addr__stext;
> > +static long addr__etext;
> > +static long addr__sinittext;
> > +static long addr__einittext;
> > +static long addr___start_orc_unwind_ip;
> > +static long addr___stop_orc_unwind_ip;
> > +static long addr___start_orc_unwind;
>
> > +void post_relocations_cleanup(unsigned long map)
> > +{
> > + if (!nofgkaslr) {
> > + update_ex_table(map);
> > + sort_ex_table(map);
> > + update_orc_table(map);
> > + sort_orc_table(map);
> > + }
> > +
> > + /*
> > + * maybe one day free will do something. So, we "free" this memory
> > + * in either case
> > + */
> > + free(sections);
> > + free(sechdrs);
> > +}
> > +
> > +void pre_relocations_cleanup(unsigned long map)
> > +{
> > + if (nofgkaslr)
> > + return;
> > +
> > + sort_kallsyms(map);
> > +}
>
> > diff --git a/arch/x86/boot/compressed/vmlinux.symbols b/arch/x86/boot/compressed/vmlinux.symbols
> > new file mode 100644
> > index 000000000000..da41f3ee153c
> > --- /dev/null
> > +++ b/arch/x86/boot/compressed/vmlinux.symbols
> > @@ -0,0 +1,19 @@
> > +kallsyms_offsets
> > +kallsyms_addresses
> > +kallsyms_num_syms
> > +kallsyms_relative_base
> > +kallsyms_names
> > +kallsyms_token_table
> > +kallsyms_token_index
> > +kallsyms_markers
> > +__start___ex_table
> > +__stop___ex_table
> > +__altinstr_replacement
> > +__altinstr_replacement_end
> > +_sinittext
> > +_einittext
> > +_stext
> > +_etext
> > +__start_orc_unwind_ip
> > +__stop_orc_unwind_ip
> > +__start_orc_unwind
>
> So please don't make it hard to add sections; the above has far too much
> duplication. For example you can trivially generate the addr_ symbol and
> the .symbol file from a common include file and a bit of macro wrappery,
> ideally that macro wrappery would also specify the sort location and
> function such that you can also generate those pre_ and post_ functions.
Re automatical generation using some wrappery -- sounds nice.
I mostly was only doing makeup for Kristen's commits so didn't pay
attention to that duplication.
> And this is only for sections that need to be sorted right? There's
> currently a patch in flight to also pre-sort the ftrace table.
Kallsyms, ORC tables and extable are getting sorted. text, inittext
and altinstr_replacement related symbols are needed to perform
shuffling (text) and relocation fixups (altinsr, inittext).
> All unsorted or runtime sorted sections are fine since they're fixed up
> by the relocations?
Right.
> Is it at all feasible to share the comparison functions between the
> various sorters?
They look very similar, I think it'd be fine to merge them (seems
like not only cmp, but adjust functions as well).
Thanks,
Al
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 04/14] linkage: add macros for putting ASM functions into own sections
2021-12-03 9:31 ` Peter Zijlstra
@ 2021-12-03 14:08 ` Alexander Lobakin
0 siblings, 0 replies; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-03 14:08 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Alexander Lobakin, linux-hardening, x86, Jesse Brandeburg,
Kristen Carlson Accardi, Kees Cook, Miklos Szeredi,
Ard Biesheuvel, Tony Luck, Bruce Schlobohm, Jessica Yu,
kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, Nick Desaulniers,
Herbert Xu, David S. Miller, Thomas Gleixner, Will Deacon,
Ingo Molnar, Borislav Petkov, Dave Hansen, H. Peter Anvin,
Andy Lutomirski, Arnd Bergmann, Josh Poimboeuf,
Nathan Chancellor, Masami Hiramatsu, Marios Pomonis,
Sami Tolvanen, linux-kernel, linux-kbuild, linux-arch,
live-patching, llvm
From: Peter Zijlstra <peterz@infradead.org>
Date: Fri, 3 Dec 2021 10:31:30 +0100
> On Thu, Dec 02, 2021 at 11:32:04PM +0100, Alexander Lobakin wrote:
>
> > diff --git a/include/linux/linkage.h b/include/linux/linkage.h
> > index dbf8506decca..baaab7dece08 100644
> > --- a/include/linux/linkage.h
> > +++ b/include/linux/linkage.h
> > @@ -355,4 +355,86 @@
> >
> > #endif /* __ASSEMBLY__ */
> >
> > +/*
> > + * Allow ASM symbols to have their own unique sections if they are being
> > + * generated by the compiler for C functions (DCE, FG-KASLR, LTO).
> > + */
> > +#if (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) && !defined(MODULE)) || \
> > + (defined(CONFIG_FG_KASLR) && !defined(MODULE)) || \
> > + (defined(CONFIG_MODULE_FG_KASLR) && defined(MODULE)) || \
> > + (defined(CONFIG_LTO_CLANG))
> > +
> > +#define SYM_TEXT_SECTION(name) \
> > + .pushsection .text.##name, "ax"
> > +
> > +#define ASM_TEXT_SECTION(name) \
> > + ".text." #name
> > +
> > +#define ASM_PUSH_SECTION(name) \
> > + ".pushsection .text." #name ", \"ax\""
> > +
> > +#else /* just .text */
> > +
> > +#define SYM_TEXT_SECTION(name) \
> > + .pushsection .text, "ax"
> > +
> > +#define ASM_TEXT_SECTION(name) \
> > + ".text"
> > +
> > +#define ASM_PUSH_SECTION(name) \
> > + ".pushsection .text, \"ax\""
> > +
> > +#endif /* just .text */
>
> That's terribly inconsistent, SYM_TEXT_SECTION is in fact
> PUSH_TEXT_SECTION, ASM_PUSH_SECTION is in fact ASM_PUSH_TEXT_SECTION and
> should be stringify(PUSH_TEXT_SECTION()) or something, and they're all
> repeating that ASM_TEXT_SECTION thing :/
Right. In fact I was waiting for someone to review them to pick more
fitting names, so I'll change them for sure.
> > +
> > +#ifdef __ASSEMBLY__
> > +
> > +#define SYM_TEXT_END_SECTION \
> > + .popsection
> > +
> > +#define SYM_FUNC_START_LOCAL_ALIAS_SECTION(name) \
> > + SYM_TEXT_SECTION(name) ASM_NL \
> > + SYM_FUNC_START_LOCAL_ALIAS(name)
> > +
> > +#define SYM_FUNC_START_LOCAL_SECTION(name) \
> > + SYM_TEXT_SECTION(name) ASM_NL \
> > + SYM_FUNC_START_LOCAL(name)
> > +
> > +#define SYM_FUNC_START_NOALIGN_SECTION(name) \
> > + SYM_TEXT_SECTION(name) ASM_NL \
> > + SYM_FUNC_START_NOALIGN(name)
> > +
> > +#define SYM_FUNC_START_WEAK_SECTION(name) \
> > + SYM_TEXT_SECTION(name) ASM_NL \
> > + SYM_FUNC_START_WEAK(name)
> > +
> > +#define SYM_FUNC_START_SECTION(name) \
> > + SYM_TEXT_SECTION(name) ASM_NL \
> > + SYM_FUNC_START(name)
> > +
> > +#define SYM_CODE_START_LOCAL_NOALIGN_SECTION(name) \
> > + SYM_TEXT_SECTION(name) ASM_NL \
> > + SYM_CODE_START_LOCAL_NOALIGN(name)
> > +
> > +#define SYM_CODE_START_NOALIGN_SECTION(name) \
> > + SYM_TEXT_SECTION(name) ASM_NL \
> > + SYM_CODE_START_NOALIGN(name)
> > +
> > +#define SYM_CODE_START_SECTION(name) \
> > + SYM_TEXT_SECTION(name) ASM_NL \
> > + SYM_CODE_START(name)
> > +
> > +#define SYM_FUNC_END_ALIAS_SECTION(name) \
> > + SYM_FUNC_END_ALIAS(name) ASM_NL \
> > + SYM_TEXT_END_SECTION
> > +
> > +#define SYM_FUNC_END_SECTION(name) \
> > + SYM_FUNC_END(name) ASM_NL \
> > + SYM_TEXT_END_SECTION
> > +
> > +#define SYM_CODE_END_SECTION(name) \
> > + SYM_CODE_END(name) ASM_NL \
> > + SYM_TEXT_END_SECTION
> > +
> > +#endif /* __ASSEMBLY__ */
>
> *URGH* why do we have to have new macros for this? SYM_FUNC_START*()
> already takes the name as argument.
I'm not sure whether we should have new macros for this.
I introduced them since there's plenty of code which use
SYM_FUNC_*(), but does .{push,pop}section outside of it, e.g. to
place something to .noinstr or .init. If I'd replace them, those
would be broken.
So I'm fine with the replacement, but those cases need to be
adressed somehow then. Are there any tricks to check if we're
already outside of .text to not push it into a new section?
Al
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections
2021-12-03 9:44 ` Peter Zijlstra
@ 2021-12-03 14:10 ` Alexander Lobakin
2021-12-03 16:34 ` Peter Zijlstra
2021-12-10 11:01 ` Alexander Lobakin
1 sibling, 1 reply; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-03 14:10 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Alexander Lobakin, linux-hardening, x86, Jesse Brandeburg,
Kristen Carlson Accardi, Kees Cook, Miklos Szeredi,
Ard Biesheuvel, Tony Luck, Bruce Schlobohm, Jessica Yu,
kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, Nick Desaulniers,
Herbert Xu, David S. Miller, Thomas Gleixner, Will Deacon,
Ingo Molnar, Borislav Petkov, Dave Hansen, H. Peter Anvin,
Andy Lutomirski, Arnd Bergmann, Josh Poimboeuf,
Nathan Chancellor, Masami Hiramatsu, Marios Pomonis,
Sami Tolvanen, linux-kernel, linux-kbuild, linux-arch,
live-patching, llvm, hjl.tools
From: Peter Zijlstra <peterz@infradead.org>
Date: Fri, 3 Dec 2021 10:44:10 +0100
> On Thu, Dec 02, 2021 at 11:32:05PM +0100, Alexander Lobakin wrote:
> > Use the newly introduces macros to create unique separate sections
> > for (almost) every "regular" ASM function (i.e. for those which
> > aren't explicitly put into a specific one).
> > There should be no leftovers as input .text will be size-asserted
> > in the LD script generated for FG-KASLR.
>
> *groan*...
>
> Please, can't we do something like:
>
> #define SYM_PUSH_SECTION(name) \
> .if section == .text \
> .push_section .text.##name \
> .else \
> .push_section .text \
> .endif
>
> #define SYM_POP_SECTION() \
> .pop_section
>
> and wrap that inside the existing SYM_FUNC_START*() SYM_FUNC_END()
> macros.
Ah I see. I asked about this in my previous mail and you replied
already (: Cool stuff, I'll use it, it simplifies things a lot.
Thanks!
Al
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 08/14] livepatch: only match unique symbols when using FG-KASLR
2021-12-03 10:05 ` Peter Zijlstra
@ 2021-12-03 14:14 ` Alexander Lobakin
2021-12-06 6:03 ` Josh Poimboeuf
1 sibling, 0 replies; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-03 14:14 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Alexander Lobakin, linux-hardening, x86, Jesse Brandeburg,
Kristen Carlson Accardi, Kees Cook, Miklos Szeredi,
Ard Biesheuvel, Tony Luck, Bruce Schlobohm, Jessica Yu,
kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, Nick Desaulniers,
Herbert Xu, David S. Miller, Thomas Gleixner, Will Deacon,
Ingo Molnar, Borislav Petkov, Dave Hansen, H. Peter Anvin,
Andy Lutomirski, Arnd Bergmann, Josh Poimboeuf,
Nathan Chancellor, Masami Hiramatsu, Marios Pomonis,
Sami Tolvanen, linux-kernel, linux-kbuild, linux-arch,
live-patching, llvm
From: Peter Zijlstra <peterz@infradead.org>
Date: Fri, 3 Dec 2021 11:05:54 +0100
> On Thu, Dec 02, 2021 at 11:32:08PM +0100, Alexander Lobakin wrote:
> > If any type of function granular randomization is enabled, the sympos
> > algorithm will fail, as it will be impossible to resolve symbols when
> > there are duplicates using the previous symbol position.
> >
> > We could override sympos to 0, but make it more clear to the user
> > and bail out if the symbol is not unique.
>
> Since we're going lots of horrendous things already, why can't we fix
> this duplicate nonsense too?
Oh, I see a ton of code duplication here in Kristen's code as well.
I'll address it.
Al
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 00/14] Function Granular KASLR
2021-12-03 10:38 ` [PATCH v8 00/14] Function Granular KASLR Peter Zijlstra
@ 2021-12-03 14:41 ` Alexander Lobakin
2021-12-03 16:32 ` Peter Zijlstra
0 siblings, 1 reply; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-03 14:41 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Alexander Lobakin, linux-hardening, x86, Jesse Brandeburg,
Kristen Carlson Accardi, Kees Cook, Miklos Szeredi,
Ard Biesheuvel, Tony Luck, Bruce Schlobohm, Jessica Yu,
kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, Nick Desaulniers,
Herbert Xu, David S. Miller, Thomas Gleixner, Will Deacon,
Ingo Molnar, Borislav Petkov, Dave Hansen, H. Peter Anvin,
Andy Lutomirski, Arnd Bergmann, Josh Poimboeuf,
Nathan Chancellor, Masami Hiramatsu, Marios Pomonis,
Sami Tolvanen, linux-kernel, linux-kbuild, linux-arch,
live-patching, llvm
From: Peter Zijlstra <peterz@infradead.org>
Date: Fri, 3 Dec 2021 11:38:30 +0100
> On Thu, Dec 02, 2021 at 11:32:00PM +0100, Alexander Lobakin wrote:
>
> > feat make -j65 boot vmlinux.o vmlinux bzImage bogoops/s
> > Relocatable 4m38.478s 24.440s 72014208 58579520 9396192 57640.39
> > KASLR 4m39.344s 24.204s 72020624 87805776 9740352 57393.80
> > FG-K 16 fps 6m16.493s 25.429s 83759856 87194160 10885632 57784.76
> > FG-K 8 fps 6m20.190s 25.094s 83759856 88741328 10985248 56625.84
> > FG-K 1 fps 7m09.611s 25.922s 83759856 95681128 11352192 56953.99
>
> :sadface: so at best it makes my kernel compiles ~50% slower. Who would
> ever consider doing that? It's like retpolines weren't bad enough; lets
> heap on the fail?
I was waiting for that :D
I know it's horrible for now, but there are some points to consider:
- folks who are placing hardening over everything don't mind
compile times most likely;
- linkers choking on huge LD scripts is actually a bug in their
code. They process 40k sections as orphans (without a generated
LD script) for a split second, so they're likely able to do the
same with it. Our position here is that after FG-KASLR landing
we'll report it and probably look into linkers' code to see if
that can be addressed (Kees et al are on this AFAIU);
- ClangLTO (at least "Fat", not sure about Thin as I didn't used
it) thinks on vmlinux.o for ~5 minutes on 8-core Skylake. Still,
it is here in mainline and is widely (relatively) used.
I know FG-KASLR stuff is way more exotic, but anyways.
- And the last one: I wouldn't consider FG-KASLR production ready
as Kees would like to see it. Apart from compilation time, you
get random performance {in,de}creases here-and-there all over
the kernel and modules you can't predict at all.
I guess it would become better later on when/if we introduce
profiling-based function placement (there are some discussions
around that and one related article is referred in the orig
cover letter), but dunno for now.
There's one issue in the current code as well -- PTI switching
code is in .entry.text which doesn't currently get randomized.
So it can probably be hunted using gadget collectors I guess?
Ok, so here's a summary of TODOs (not including sadfaces
unfortunately):
* generate vmlinux.symbols and the corresponding variables
on-the-go;
* unify comparison and adjustment functions, probably reuse
some of the already existing ones;
* don't introduce new macros in linkage.h, just use fancy
'section == .text' to decide in-place;
* change new macros' names (those which shouldn' be wiped out)
to make them more consistent;
* look over for several code dups.
Am I missing anything else?
One more quest, what could I do with this infinitely long regexp
in gen_text_sections.pl script? Just try to wrap over or it can
be simplified somehow?
Al
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 00/14] Function Granular KASLR
2021-12-03 14:41 ` Alexander Lobakin
@ 2021-12-03 16:32 ` Peter Zijlstra
0 siblings, 0 replies; 40+ messages in thread
From: Peter Zijlstra @ 2021-12-03 16:32 UTC (permalink / raw)
To: Alexander Lobakin
Cc: linux-hardening, x86, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, linux-kernel, linux-kbuild,
linux-arch, live-patching, llvm
On Fri, Dec 03, 2021 at 03:41:36PM +0100, Alexander Lobakin wrote:
> From: Peter Zijlstra <peterz@infradead.org>
> Date: Fri, 3 Dec 2021 11:38:30 +0100
>
> > On Thu, Dec 02, 2021 at 11:32:00PM +0100, Alexander Lobakin wrote:
> >
> > > feat make -j65 boot vmlinux.o vmlinux bzImage bogoops/s
> > > Relocatable 4m38.478s 24.440s 72014208 58579520 9396192 57640.39
> > > KASLR 4m39.344s 24.204s 72020624 87805776 9740352 57393.80
> > > FG-K 16 fps 6m16.493s 25.429s 83759856 87194160 10885632 57784.76
> > > FG-K 8 fps 6m20.190s 25.094s 83759856 88741328 10985248 56625.84
> > > FG-K 1 fps 7m09.611s 25.922s 83759856 95681128 11352192 56953.99
> >
> > :sadface: so at best it makes my kernel compiles ~50% slower. Who would
> > ever consider doing that? It's like retpolines weren't bad enough; lets
> > heap on the fail?
>
> I was waiting for that :D
>
> I know it's horrible for now, but there are some points to consider:
> - folks who are placing hardening over everything don't mind
> compile times most likely;
> - linkers choking on huge LD scripts is actually a bug in their
> code. They process 40k sections as orphans (without a generated
> LD script) for a split second, so they're likely able to do the
> same with it. Our position here is that after FG-KASLR landing
> we'll report it and probably look into linkers' code to see if
> that can be addressed (Kees et al are on this AFAIU);
> - ClangLTO (at least "Fat", not sure about Thin as I didn't used
> it) thinks on vmlinux.o for ~5 minutes on 8-core Skylake. Still,
> it is here in mainline and is widely (relatively) used.
> I know FG-KASLR stuff is way more exotic, but anyways.
> - And the last one: I wouldn't consider FG-KASLR production ready
> as Kees would like to see it. Apart from compilation time, you
> get random performance {in,de}creases here-and-there all over
> the kernel and modules you can't predict at all.
> I guess it would become better later on when/if we introduce
> profiling-based function placement (there are some discussions
> around that and one related article is referred in the orig
> cover letter), but dunno for now.
> There's one issue in the current code as well -- PTI switching
> code is in .entry.text which doesn't currently get randomized.
> So it can probably be hunted using gadget collectors I guess?
Oooh, so those compile times are not, as one would expect the compile
times for a set .config but with different kernel, but instead for a
varying .config on the same kernel?
IOW, they don't represent the run-time overhead of this thing, but
merely the toolchain overhead of all this.
So what is the actual runtime overhead of all this?
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections
2021-12-03 14:10 ` Alexander Lobakin
@ 2021-12-03 16:34 ` Peter Zijlstra
2021-12-03 16:51 ` H.J. Lu
2021-12-03 19:46 ` Nicolas Pitre
0 siblings, 2 replies; 40+ messages in thread
From: Peter Zijlstra @ 2021-12-03 16:34 UTC (permalink / raw)
To: Alexander Lobakin
Cc: linux-hardening, x86, Jesse Brandeburg, Kristen Carlson Accardi,
Kees Cook, Miklos Szeredi, Ard Biesheuvel, Tony Luck,
Bruce Schlobohm, Jessica Yu, kernel test robot, Miroslav Benes,
Evgenii Shatokhin, Jonathan Corbet, Masahiro Yamada,
Michal Marek, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, linux-kernel, linux-kbuild,
linux-arch, live-patching, llvm, hjl.tools
On Fri, Dec 03, 2021 at 03:10:51PM +0100, Alexander Lobakin wrote:
> From: Peter Zijlstra <peterz@infradead.org>
> Date: Fri, 3 Dec 2021 10:44:10 +0100
>
> > On Thu, Dec 02, 2021 at 11:32:05PM +0100, Alexander Lobakin wrote:
> > > Use the newly introduces macros to create unique separate sections
> > > for (almost) every "regular" ASM function (i.e. for those which
> > > aren't explicitly put into a specific one).
> > > There should be no leftovers as input .text will be size-asserted
> > > in the LD script generated for FG-KASLR.
> >
> > *groan*...
> >
> > Please, can't we do something like:
> >
> > #define SYM_PUSH_SECTION(name) \
> > .if section == .text \
> > .push_section .text.##name \
> > .else \
> > .push_section .text \
> > .endif
> >
> > #define SYM_POP_SECTION() \
> > .pop_section
> >
> > and wrap that inside the existing SYM_FUNC_START*() SYM_FUNC_END()
> > macros.
>
> Ah I see. I asked about this in my previous mail and you replied
> already (: Cool stuff, I'll use it, it simplifies things a lot.
Note, I've no idea if it works. GAS and me aren't really on speaking
terms. It would be my luck for that to be totally impossible, hjl?
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections
2021-12-03 16:34 ` Peter Zijlstra
@ 2021-12-03 16:51 ` H.J. Lu
2021-12-03 19:46 ` Nicolas Pitre
1 sibling, 0 replies; 40+ messages in thread
From: H.J. Lu @ 2021-12-03 16:51 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Alexander Lobakin, linux-hardening, the arch/x86 maintainers,
Jesse Brandeburg, Kristen Carlson Accardi, Kees Cook,
Miklos Szeredi, Ard Biesheuvel, Tony Luck, Bruce Schlobohm,
Jessica Yu, kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, Nick Desaulniers,
Herbert Xu, David S. Miller, Thomas Gleixner, Will Deacon,
Ingo Molnar, Borislav Petkov, Dave Hansen, H. Peter Anvin,
Andy Lutomirski, Arnd Bergmann, Josh Poimboeuf,
Nathan Chancellor, Masami Hiramatsu, Marios Pomonis,
Sami Tolvanen, LKML, Linux Kbuild mailing list, linux-arch,
live-patching, llvm
On Fri, Dec 3, 2021 at 8:34 AM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Fri, Dec 03, 2021 at 03:10:51PM +0100, Alexander Lobakin wrote:
> > From: Peter Zijlstra <peterz@infradead.org>
> > Date: Fri, 3 Dec 2021 10:44:10 +0100
> >
> > > On Thu, Dec 02, 2021 at 11:32:05PM +0100, Alexander Lobakin wrote:
> > > > Use the newly introduces macros to create unique separate sections
> > > > for (almost) every "regular" ASM function (i.e. for those which
> > > > aren't explicitly put into a specific one).
> > > > There should be no leftovers as input .text will be size-asserted
> > > > in the LD script generated for FG-KASLR.
> > >
> > > *groan*...
> > >
> > > Please, can't we do something like:
> > >
> > > #define SYM_PUSH_SECTION(name) \
> > > .if section == .text \
> > > .push_section .text.##name \
> > > .else \
> > > .push_section .text \
> > > .endif
> > >
> > > #define SYM_POP_SECTION() \
> > > .pop_section
> > >
> > > and wrap that inside the existing SYM_FUNC_START*() SYM_FUNC_END()
> > > macros.
> >
> > Ah I see. I asked about this in my previous mail and you replied
> > already (: Cool stuff, I'll use it, it simplifies things a lot.
>
> Note, I've no idea if it works. GAS and me aren't really on speaking
> terms. It would be my luck for that to be totally impossible, hjl?
What exactly do you want assembler to do?
--
H.J.
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections
2021-12-03 16:34 ` Peter Zijlstra
2021-12-03 16:51 ` H.J. Lu
@ 2021-12-03 19:46 ` Nicolas Pitre
2021-12-04 8:31 ` Ard Biesheuvel
2021-12-04 8:55 ` Peter Zijlstra
1 sibling, 2 replies; 40+ messages in thread
From: Nicolas Pitre @ 2021-12-03 19:46 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Alexander Lobakin, linux-hardening, x86, Jesse Brandeburg,
Kristen Carlson Accardi, Kees Cook, Miklos Szeredi,
Ard Biesheuvel, Tony Luck, Bruce Schlobohm, Jessica Yu,
kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, Nick Desaulniers,
Herbert Xu, David S. Miller, Thomas Gleixner, Will Deacon,
Ingo Molnar, Borislav Petkov, Dave Hansen, H. Peter Anvin,
Andy Lutomirski, Arnd Bergmann, Josh Poimboeuf,
Nathan Chancellor, Masami Hiramatsu, Marios Pomonis,
Sami Tolvanen, linux-kernel, linux-kbuild, linux-arch,
live-patching, llvm, hjl.tools
On Fri, 3 Dec 2021, Peter Zijlstra wrote:
> On Fri, Dec 03, 2021 at 03:10:51PM +0100, Alexander Lobakin wrote:
> > From: Peter Zijlstra <peterz@infradead.org>
> > Date: Fri, 3 Dec 2021 10:44:10 +0100
> >
> > > On Thu, Dec 02, 2021 at 11:32:05PM +0100, Alexander Lobakin wrote:
> > > > Use the newly introduces macros to create unique separate sections
> > > > for (almost) every "regular" ASM function (i.e. for those which
> > > > aren't explicitly put into a specific one).
> > > > There should be no leftovers as input .text will be size-asserted
> > > > in the LD script generated for FG-KASLR.
> > >
> > > *groan*...
> > >
> > > Please, can't we do something like:
> > >
> > > #define SYM_PUSH_SECTION(name) \
> > > .if section == .text \
> > > .push_section .text.##name \
> > > .else \
> > > .push_section .text \
> > > .endif
> > >
> > > #define SYM_POP_SECTION() \
> > > .pop_section
> > >
> > > and wrap that inside the existing SYM_FUNC_START*() SYM_FUNC_END()
> > > macros.
> >
> > Ah I see. I asked about this in my previous mail and you replied
> > already (: Cool stuff, I'll use it, it simplifies things a lot.
>
> Note, I've no idea if it works. GAS and me aren't really on speaking
> terms. It would be my luck for that to be totally impossible, hjl?
Surely this would do it:
http://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=451133cefa839104
Nicolas
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections
2021-12-03 19:46 ` Nicolas Pitre
@ 2021-12-04 8:31 ` Ard Biesheuvel
2021-12-04 8:55 ` Peter Zijlstra
1 sibling, 0 replies; 40+ messages in thread
From: Ard Biesheuvel @ 2021-12-04 8:31 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Peter Zijlstra, Alexander Lobakin, linux-hardening, X86 ML,
Jesse Brandeburg, Kristen Carlson Accardi, Kees Cook,
Miklos Szeredi, Tony Luck, Bruce Schlobohm, Jessica Yu,
kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, Nick Desaulniers,
Herbert Xu, David S. Miller, Thomas Gleixner, Will Deacon,
Ingo Molnar, Borislav Petkov, Dave Hansen, H. Peter Anvin,
Andy Lutomirski, Arnd Bergmann, Josh Poimboeuf,
Nathan Chancellor, Masami Hiramatsu, Marios Pomonis,
Sami Tolvanen, Linux Kernel Mailing List,
Linux Kbuild mailing list, linux-arch, live-patching, llvm,
hjl.tools
On Fri, 3 Dec 2021 at 20:46, Nicolas Pitre <nico@fluxnic.net> wrote:
>
> On Fri, 3 Dec 2021, Peter Zijlstra wrote:
>
> > On Fri, Dec 03, 2021 at 03:10:51PM +0100, Alexander Lobakin wrote:
> > > From: Peter Zijlstra <peterz@infradead.org>
> > > Date: Fri, 3 Dec 2021 10:44:10 +0100
> > >
> > > > On Thu, Dec 02, 2021 at 11:32:05PM +0100, Alexander Lobakin wrote:
> > > > > Use the newly introduces macros to create unique separate sections
> > > > > for (almost) every "regular" ASM function (i.e. for those which
> > > > > aren't explicitly put into a specific one).
> > > > > There should be no leftovers as input .text will be size-asserted
> > > > > in the LD script generated for FG-KASLR.
> > > >
> > > > *groan*...
> > > >
> > > > Please, can't we do something like:
> > > >
> > > > #define SYM_PUSH_SECTION(name) \
> > > > .if section == .text \
> > > > .push_section .text.##name \
> > > > .else \
> > > > .push_section .text \
> > > > .endif
> > > >
> > > > #define SYM_POP_SECTION() \
> > > > .pop_section
> > > >
> > > > and wrap that inside the existing SYM_FUNC_START*() SYM_FUNC_END()
> > > > macros.
> > >
> > > Ah I see. I asked about this in my previous mail and you replied
> > > already (: Cool stuff, I'll use it, it simplifies things a lot.
> >
> > Note, I've no idea if it works. GAS and me aren't really on speaking
> > terms. It would be my luck for that to be totally impossible, hjl?
>
> Surely this would do it:
>
> http://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=451133cefa839104
>
That seems rather useful, actually. It will also fix a problem with
subsections, which are sometimes difficult to construct from a macro,
as they cannot be created using pushsection/popsection unless you know
the current section name, and the alternative syntax (.subsection /
.previous) does not permit nesting. This makes their use from a macro
risky, given that it may not be obvious to the macro's caller that it
uses a subsection under the hood.
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections
2021-12-03 19:46 ` Nicolas Pitre
2021-12-04 8:31 ` Ard Biesheuvel
@ 2021-12-04 8:55 ` Peter Zijlstra
1 sibling, 0 replies; 40+ messages in thread
From: Peter Zijlstra @ 2021-12-04 8:55 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Alexander Lobakin, linux-hardening, x86, Jesse Brandeburg,
Kristen Carlson Accardi, Kees Cook, Miklos Szeredi,
Ard Biesheuvel, Tony Luck, Bruce Schlobohm, Jessica Yu,
kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, Nick Desaulniers,
Herbert Xu, David S. Miller, Thomas Gleixner, Will Deacon,
Ingo Molnar, Borislav Petkov, Dave Hansen, H. Peter Anvin,
Andy Lutomirski, Arnd Bergmann, Josh Poimboeuf,
Nathan Chancellor, Masami Hiramatsu, Marios Pomonis,
Sami Tolvanen, linux-kernel, linux-kbuild, linux-arch,
live-patching, llvm, hjl.tools
On Fri, Dec 03, 2021 at 02:46:39PM -0500, Nicolas Pitre wrote:
> Surely this would do it:
>
> http://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=451133cefa839104
Ooh, yes, excellent that, thanks!
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 08/14] livepatch: only match unique symbols when using FG-KASLR
2021-12-03 10:05 ` Peter Zijlstra
2021-12-03 14:14 ` Alexander Lobakin
@ 2021-12-06 6:03 ` Josh Poimboeuf
1 sibling, 0 replies; 40+ messages in thread
From: Josh Poimboeuf @ 2021-12-06 6:03 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Alexander Lobakin, linux-hardening, x86, Jesse Brandeburg,
Kristen Carlson Accardi, Kees Cook, Miklos Szeredi,
Ard Biesheuvel, Tony Luck, Bruce Schlobohm, Jessica Yu,
kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, Nick Desaulniers,
Herbert Xu, David S. Miller, Thomas Gleixner, Will Deacon,
Ingo Molnar, Borislav Petkov, Dave Hansen, H. Peter Anvin,
Andy Lutomirski, Arnd Bergmann, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen, linux-kernel,
linux-kbuild, linux-arch, live-patching, llvm
On Fri, Dec 03, 2021 at 11:05:54AM +0100, Peter Zijlstra wrote:
> On Thu, Dec 02, 2021 at 11:32:08PM +0100, Alexander Lobakin wrote:
> > If any type of function granular randomization is enabled, the sympos
> > algorithm will fail, as it will be impossible to resolve symbols when
> > there are duplicates using the previous symbol position.
> >
> > We could override sympos to 0, but make it more clear to the user
> > and bail out if the symbol is not unique.
>
> Since we're going lots of horrendous things already, why can't we fix
> this duplicate nonsense too?
I assume you mean using this new linker flag: "-z unique-symbol"
https://sourceware.org/bugzilla/show_bug.cgi?id=26391
--
Josh
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 07/14] kallsyms: Hide layout
2021-12-03 10:03 ` Ard Biesheuvel
@ 2021-12-07 5:31 ` Josh Poimboeuf
0 siblings, 0 replies; 40+ messages in thread
From: Josh Poimboeuf @ 2021-12-07 5:31 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Peter Zijlstra, Alexander Lobakin, linux-hardening, X86 ML,
Jesse Brandeburg, Kristen Carlson Accardi, Kees Cook,
Miklos Szeredi, Tony Luck, Bruce Schlobohm, Jessica Yu,
kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, Nick Desaulniers,
Herbert Xu, David S. Miller, Thomas Gleixner, Will Deacon,
Ingo Molnar, Borislav Petkov, Dave Hansen, H. Peter Anvin,
Andy Lutomirski, Arnd Bergmann, Nathan Chancellor,
Masami Hiramatsu, Marios Pomonis, Sami Tolvanen,
Linux Kernel Mailing List, Linux Kbuild mailing list, linux-arch,
live-patching, llvm
On Fri, Dec 03, 2021 at 11:03:35AM +0100, Ard Biesheuvel wrote:
> On Fri, 3 Dec 2021 at 11:01, Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Thu, Dec 02, 2021 at 11:32:07PM +0100, Alexander Lobakin wrote:
> > > From: Kristen Carlson Accardi <kristen@linux.intel.com>
> > >
> > > This patch makes /proc/kallsyms display in a random order, rather
> > > than sorted by address in order to hide the newly randomized address
> > > layout.
> >
> > Is there a reason to not always do this? That is, why are we keeping two
> > copies of this code around? Less code is more better etc..
>
> +1.
>
> IIRC I made the exact same point when this patch was sent out by
> Kristen a while ago.
Yes. Alexander, I'd recommend going back to the review comments from
Kristen's last posting, they may have been missed.
--
Josh
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections
2021-12-03 9:44 ` Peter Zijlstra
2021-12-03 14:10 ` Alexander Lobakin
@ 2021-12-10 11:01 ` Alexander Lobakin
2021-12-10 13:20 ` Nicolas Pitre
1 sibling, 1 reply; 40+ messages in thread
From: Alexander Lobakin @ 2021-12-10 11:01 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Alexander Lobakin, linux-hardening, x86, Jesse Brandeburg,
Kristen Carlson Accardi, Kees Cook, Miklos Szeredi,
Ard Biesheuvel, Tony Luck, Bruce Schlobohm, Jessica Yu,
kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, H . J . Lu,
Nicolas Pitre, Nick Desaulniers, Herbert Xu, David S. Miller,
Thomas Gleixner, Will Deacon, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Arnd Bergmann,
Josh Poimboeuf, Nathan Chancellor, Masami Hiramatsu,
Marios Pomonis, Sami Tolvanen, linux-kernel, linux-kbuild,
linux-arch, live-patching, llvm
From: Peter Zijlstra <peterz@infradead.org>
Date: Fri, 3 Dec 2021 10:44:10 +0100
> On Thu, Dec 02, 2021 at 11:32:05PM +0100, Alexander Lobakin wrote:
> > Use the newly introduces macros to create unique separate sections
> > for (almost) every "regular" ASM function (i.e. for those which
> > aren't explicitly put into a specific one).
> > There should be no leftovers as input .text will be size-asserted
> > in the LD script generated for FG-KASLR.
>
> *groan*...
>
> Please, can't we do something like:
>
> #define SYM_PUSH_SECTION(name) \
> .if section == .text \
> .push_section .text.##name \
> .else \
> .push_section .text \
> .endif
This condition
.pushsection .text
.if section == .text
# do something
.endif
.popsection
doesn't really works. `do something` doesn't happen.
This works only when
.pushsection .text
.equ section, .text
but it's not really okayish I'd say to find all .{,push}section
occurences and replace them with a macro (which would also do .equ).
I don't really know how %S with --sectname-subst should help me as
.if %S == .text
# do something
.endif
doesn't work at all (syntax error) -- and it shouldn't, %S is
supposed to work only inside .{,push}section directives.
I could do unconditional
.pushsection %S.##name
^^^^^^ function name
but this would involve changing LDS scripts (and vmlinux.lds.h) to
let's say replace *(.noinstr.text) with *(.noinstr.text*).
So I hope there is a way to get current section name? If not, then
the last option is the least harmful I suppose.
At least not as harmful as current approach with alternative macros,
far from it lol.
>
> #define SYM_POP_SECTION() \
> .pop_section
>
> and wrap that inside the existing SYM_FUNC_START*() SYM_FUNC_END()
> macros.
Thanks,
Al
^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections
2021-12-10 11:01 ` Alexander Lobakin
@ 2021-12-10 13:20 ` Nicolas Pitre
0 siblings, 0 replies; 40+ messages in thread
From: Nicolas Pitre @ 2021-12-10 13:20 UTC (permalink / raw)
To: Alexander Lobakin
Cc: Peter Zijlstra, linux-hardening, x86, Jesse Brandeburg,
Kristen Carlson Accardi, Kees Cook, Miklos Szeredi,
Ard Biesheuvel, Tony Luck, Bruce Schlobohm, Jessica Yu,
kernel test robot, Miroslav Benes, Evgenii Shatokhin,
Jonathan Corbet, Masahiro Yamada, Michal Marek, H . J . Lu,
Nick Desaulniers, Herbert Xu, David S. Miller, Thomas Gleixner,
Will Deacon, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Andy Lutomirski, Arnd Bergmann, Josh Poimboeuf,
Nathan Chancellor, Masami Hiramatsu, Marios Pomonis,
Sami Tolvanen, linux-kernel, linux-kbuild, linux-arch,
live-patching, llvm
On Fri, 10 Dec 2021, Alexander Lobakin wrote:
> I could do unconditional
>
> .pushsection %S.##name
> ^^^^^^ function name
>
> but this would involve changing LDS scripts (and vmlinux.lds.h) to
> let's say replace *(.noinstr.text) with *(.noinstr.text*).
Yes, this is meant to be used with a linker script that expects the new
name.
Nicolas
^ permalink raw reply [flat|nested] 40+ messages in thread
end of thread, other threads:[~2021-12-10 13:28 UTC | newest]
Thread overview: 40+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-02 22:32 [PATCH v8 00/14] Function Granular KASLR Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 01/14] x86: Makefile: Add build and config option for CONFIG_FG_KASLR Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 02/14] x86/tools: Add relative relocs for randomized functions Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 03/14] x86: Add support for function granular KASLR Alexander Lobakin
2021-12-03 9:18 ` Peter Zijlstra
2021-12-03 13:57 ` Alexander Lobakin
2021-12-03 10:34 ` Peter Zijlstra
2021-12-02 22:32 ` [PATCH v8 04/14] linkage: add macros for putting ASM functions into own sections Alexander Lobakin
2021-12-03 9:31 ` Peter Zijlstra
2021-12-03 14:08 ` Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 05/14] x86: conditionally place regular ASM functions into separate sections Alexander Lobakin
2021-12-03 9:44 ` Peter Zijlstra
2021-12-03 14:10 ` Alexander Lobakin
2021-12-03 16:34 ` Peter Zijlstra
2021-12-03 16:51 ` H.J. Lu
2021-12-03 19:46 ` Nicolas Pitre
2021-12-04 8:31 ` Ard Biesheuvel
2021-12-04 8:55 ` Peter Zijlstra
2021-12-10 11:01 ` Alexander Lobakin
2021-12-10 13:20 ` Nicolas Pitre
2021-12-02 22:32 ` [PATCH v8 06/14] FG-KASLR: use a scripted approach to handle .text.* sections Alexander Lobakin
2021-12-03 9:54 ` Peter Zijlstra
2021-12-02 22:32 ` [PATCH v8 07/14] kallsyms: Hide layout Alexander Lobakin
2021-12-03 10:00 ` Peter Zijlstra
2021-12-03 10:03 ` Ard Biesheuvel
2021-12-07 5:31 ` Josh Poimboeuf
2021-12-02 22:32 ` [PATCH v8 08/14] livepatch: only match unique symbols when using FG-KASLR Alexander Lobakin
2021-12-03 10:05 ` Peter Zijlstra
2021-12-03 14:14 ` Alexander Lobakin
2021-12-06 6:03 ` Josh Poimboeuf
2021-12-02 22:32 ` [PATCH v8 09/14] x86/boot: allow FG-KASLR to be selected Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 10/14] arm64/crypto: conditionally place ASM functions into separate sections Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 11/14] module: Reorder functions Alexander Lobakin
2021-12-03 10:23 ` Peter Zijlstra
2021-12-02 22:32 ` [PATCH v8 12/14] module: use a scripted approach for FG-KASLR Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 13/14] Documentation: add documentation " Alexander Lobakin
2021-12-02 22:32 ` [PATCH v8 14/14] maintainers: add MAINTAINERS entry " Alexander Lobakin
2021-12-03 10:38 ` [PATCH v8 00/14] Function Granular KASLR Peter Zijlstra
2021-12-03 14:41 ` Alexander Lobakin
2021-12-03 16:32 ` Peter Zijlstra
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.