linux-hardening.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v6 kspp-next 00/22] Function Granular KASLR
@ 2021-08-31 14:40 Alexander Lobakin
  2021-08-31 14:40 ` [PATCH v6 kspp-next 01/22] kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG Alexander Lobakin
                   ` (22 more replies)
  0 siblings, 23 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:40 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

This is a massive rework and a respin of Kristen Accardi's marvellous
FG-KASLR series (v5).

The major differences since v5 [0]:
* You can now tune the number of functions per each section to
  achieve the preferable vmlinux size or protection level. Default
  is still as one section per each function.
  This can be handy for storage-constrained systems. 4-8 fps are
  still strong, but reduce the size of the final vmlinu{x,z}
  significantly;
* I don't use orphan sections anymore. It's not reliable at all /
  may differ from linker to linker, and also conflicts with
  CONFIG_LD_ORPHAN_WARN which is great for catching random bugs ->
* All the .text.* sections are now being described explicitly in the
  linker script. A Perl script is used to take the original LDS, the
  original object file, read a list of input sections from it and
  generate the resulting LDS.
  This costs a bit of linking time as LD tends to think hard when
  processing scripts > 1 Mb. It adds about 40-60 seconds to the
  whole linking process (BTF step, 2-3 kallsyms steps and the final
  step), but "better safe than sorry".
  In addition, that approach allows to reserve some space at the end
  and add some link assertions ->
* Input .text section now must be empty, otherwise the linkage will
  be stopped. This is implemented by the size assertion in the
  resulting LD script and is designed to plug the potentional layout
  leakage. This also means that ->
* "Regular" ASM functions are now being placed into unique separate
  functions the same way compiler does this for C functions. This is
  achieved by introducing and using several new macros which take
  the symbol name as a base for its new section name.
  This gives a better opportunity to both DCE and FG-KASLR, as ASM
  code now can also be randomized or garbage-collected;
* It's now fully compatible with ClangLTO, ClangCFI,
  CONFIG_LD_ORPHAN_WARN and some more stuff landed since the last
  revision was published;
* Includes several fixes: relocations inside .altinstr_replacement
  code and minor issues found and/or suggested by LKP robot.

The series was compile-time and runtime tested on the following
setups with no issues:
- x86_64, GCC 11, Binutils 2.35;
- x86_64, Clang/LLVM 12, ClangLTO + ClangCFI (from Sami's tree).

The first 4 patches are from the linux-kbuild tree and included
to avoid merge conflicts and non-intuitive resolving of them.

The series is also available here: [1]

[0] https://lore.kernel.org/kernel-hardening/20200923173905.11219-1-kristen@linux.intel.com
[1] https://github.com/alobakin/linux/pull/3

The original v5 cover letter:

Function Granular Kernel Address Space Layout Randomization (fgkaslr)
---------------------------------------------------------------------

This patch set is an implementation of finer grained kernel address space
randomization. It rearranges your kernel code at load time 
on a per-function level granularity, with only around a second added to
boot time.

Changes in v5:
--------------
* fixed a bug in the code which increases boot heap size for
  CONFIG_FG_KASLR which prevented the boot heap from being increased
  for CONFIG_FG_KASLR when using bzip2 compression. Thanks to Andy Lavr
  for finding the problem and identifying the solution.
* changed the adjustment of the orc_unwind_ip table at boot time to
  disregard relocs associated with this table, and instead inspect the
  entries separately. Relocs are not able to be used since they are
  no longer correct once the table is resorted at buildtime.
* changed how orc_unwind_ip addresses in randomized sections are identified
  to include the byte immediately after the end of the section.
* updated module code to use kvmalloc/kvfree based on suggestions from
  Evgenii Shatokhin <eshatokhin@virtuozzo.com>.
* changed kernel commandline to disable fgkaslr to simply "nofgkaslr" to
  match the nokaslr option. fgkaslr="X" can be added at a later date
  if it is needed.
* Added a patch to force livepatch to require symbols to be unique if
  using while fgkaslr either for core or modules.

Changes in v4:
-------------
* dropped the patch to split out change to STATIC definition in
  x86/boot/compressed/misc.c and replaced with a patch authored
  by Kees Cook to avoid the duplicate malloc definitions
* Added a section to Documentation/admin-guide/kernel-parameters.txt
  to document the fgkaslr boot option.
* redesigned the patch to hide the new layout when reading
  /proc/kallsyms. The previous implementation utilized a dynamically
  allocated linked list to display the kernel and module symbols
  in alphabetical order. The new implementation uses a randomly
  shuffled index array to display the kernel and module symbols
  in a random order.

Changes in v3:
-------------
* Makefile changes to accommodate CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
* removal of extraneous ALIGN_PAGE from _etext changes
* changed variable names in x86/tools/relocs to be less confusing
* split out change to STATIC definition in x86/boot/compressed/misc.c
* Updates to Documentation to make it more clear what is preserved in .text
* much more detailed commit message for function granular KASLR patch
* minor tweaks and changes that make for more readable code
* this cover letter updated slightly to add additional details

Changes in v2:
--------------
* Fix to address i386 build failure
* Allow module reordering patch to be configured separately so that
  arm (or other non-x86_64 arches) can take advantage of module function
  reordering. This support has not be tested by me, but smoke tested by
  Ard Biesheuvel <ardb@kernel.org> on arm.
* Fix build issue when building on arm as reported by
  Ard Biesheuvel <ardb@kernel.org> 

Patches to objtool are included because they are dependencies for this
patchset, however they have been submitted by their maintainer separately.

Background
----------
KASLR was merged into the kernel with the objective of increasing the
difficulty of code reuse attacks. Code reuse attacks reused existing code
snippets to get around existing memory protections. They exploit software bugs
which expose addresses of useful code snippets to control the flow of
execution for their own nefarious purposes. KASLR moves the entire kernel
code text as a unit at boot time in order to make addresses less predictable.
The order of the code within the segment is unchanged - only the base address
is shifted. There are a few shortcomings to this algorithm.

1. Low Entropy - there are only so many locations the kernel can fit in. This
   means an attacker could guess without too much trouble.
2. Knowledge of a single address can reveal the offset of the base address,
   exposing all other locations for a published/known kernel image.
3. Info leaks abound.

Finer grained ASLR has been proposed as a way to make ASLR more resistant
to info leaks. It is not a new concept at all, and there are many variations
possible. Function reordering is an implementation of finer grained ASLR
which randomizes the layout of an address space on a function level
granularity. We use the term "fgkaslr" in this document to refer to the
technique of function reordering when used with KASLR, as well as finer grained
KASLR in general.

Proposed Improvement
--------------------
This patch set proposes adding function reordering on top of the existing
KASLR base address randomization. The over-arching objective is incremental
improvement over what we already have. It is designed to work in combination
with the existing solution. The implementation is really pretty simple, and
there are 2 main area where changes occur:

* Build time

GCC has had an option to place functions into individual .text sections for
many years now. This option can be used to implement function reordering at
load time. The final compiled vmlinux retains all the section headers, which
can be used to help find the address ranges of each function. Using this
information and an expanded table of relocation addresses, individual text
sections can be suffled immediately after decompression. Some data tables
inside the kernel that have assumptions about order require re-sorting
after being updated when applying relocations. In order to modify these tables,
a few key symbols are excluded from the objcopy symbol stripping process for
use after shuffling the text segments.

Some highlights from the build time changes to look for:

The top level kernel Makefile was modified to add the gcc flag if it
is supported. Currently, I am applying this flag to everything it is
possible to randomize. Anything that is written in C and not present in a
special input section is randomized. The final binary segment 0 retains a
consolidated .text section, as well as all the individual .text.* sections.
Future work could turn off this flags for selected files or even entire
subsystems, although obviously at the cost of security.

The relocs tool is updated to add relative relocations. This information
previously wasn't included because it wasn't necessary when moving the
entire .text segment as a unit. 

A new file was created to contain a list of symbols that objcopy should
keep. We use those symbols at load time as described below.

* Load time

The boot kernel was modified to parse the vmlinux elf file after
decompression to check for our interesting symbols that we kept, and to
look for any .text.* sections to randomize. The consolidated .text section
is skipped and not moved. The sections are shuffled randomly, and copied
into memory following the .text section in a new random order. The existing
code which updated relocation addresses was modified to account for
not just a fixed delta from the load address, but the offset that the function
section was moved to. This requires inspection of each address to see if
it was impacted by a randomization. We use a bsearch to make this less
horrible on performance. Any tables that need to be modified with new
addresses or resorted are updated using the symbol addresses parsed from the
elf symbol table.

In order to hide our new layout, symbols reported through /proc/kallsyms
will be displayed in a random order.

Security Considerations
-----------------------
The objective of this patch set is to improve a technology that is already
merged into the kernel (KASLR). This code will not prevent all attacks,
but should instead be considered as one of several tools that can be used.
In particular, this code is meant to make KASLR more effective in the presence
of info leaks.

How much entropy we are adding to the existing entropy of standard KASLR will
depend on a few variables. Firstly and most obviously, the number of functions
that are randomized matters. This implementation keeps the existing .text
section for code that cannot be randomized - for example, because it was
assembly code. The less sections to randomize, the less entropy. In addition,
due to alignment (16 bytes for x86_64), the number of bits in a address that
the attacker needs to guess is reduced, as the lower bits are identical.

Performance Impact
------------------
There are two areas where function reordering can impact performance: boot
time latency, and run time performance.

* Boot time latency
This implementation of finer grained KASLR impacts the boot time of the kernel
in several places. It requires additional parsing of the kernel ELF file to
obtain the section headers of the sections to be randomized. It calls the
random number generator for each section to be randomized to determine that
section's new memory location. It copies the decompressed kernel into a new
area of memory to avoid corruption when laying out the newly randomized
sections. It increases the number of relocations the kernel has to perform at
boot time vs. standard KASLR, and it also requires a lookup on each address
that needs to be relocated to see if it was in a randomized section and needs
to be adjusted by a new offset. Finally, it re-sorts a few data tables that
are required to be sorted by address.

Booting a test VM on a modern, well appointed system showed an increase in
latency of approximately 1 second.

* Run time
The performance impact at run-time of function reordering varies by workload.
Using kcbench, a kernel compilation benchmark, the performance of a kernel
build with finer grained KASLR was about 1% slower than a kernel with standard
KASLR. Analysis with perf showed a slightly higher percentage of 
L1-icache-load-misses. Other workloads were examined as well, with varied
results. Some workloads performed significantly worse under FGKASLR, while
others stayed the same or were mysteriously better. In general, it will
depend on the code flow whether or not finer grained KASLR will impact
your workload, and how the underlying code was designed. Because the layout
changes per boot, each time a system is rebooted the performance of a workload
may change.

Future work could identify hot areas that may not be randomized and either
leave them in the .text section or group them together into a single section
that may be randomized. If grouping things together helps, one other thing to
consider is that if we could identify text blobs that should be grouped together
to benefit a particular code flow, it could be interesting to explore
whether this security feature could be also be used as a performance
feature if you are interested in optimizing your kernel layout for a
particular workload at boot time. Optimizing function layout for a particular
workload has been researched and proven effective - for more information
read the Facebook paper "Optimizing Function Placement for Large-Scale
Data-Center Applications" (see references section below).

Image Size
----------
Adding additional section headers as a result of compiling with
-ffunction-sections will increase the size of the vmlinux ELF file.
With a standard distro config, the resulting vmlinux was increased by
about 3%. The compressed image is also increased due to the header files,
as well as the extra relocations that must be added. You can expect fgkaslr
to increase the size of the compressed image by about 15%.

Memory Usage
------------
fgkaslr increases the amount of heap that is required at boot time,
although this extra memory is released when the kernel has finished
decompression. As a result, it may not be appropriate to use this feature on
systems without much memory.

Building
--------
To enable fine grained KASLR, you need to have the following config options
set (including all the ones you would use to build normal KASLR)

CONFIG_FG_KASLR=y

In addition, fgkaslr is only supported for the X86_64 architecture.

Modules
-------
Modules are randomized similarly to the rest of the kernel by shuffling
the sections at load time prior to moving them into memory. The module must
also have been build with the -ffunction-sections compiler option.

Although fgkaslr for the kernel is only supported for the X86_64 architecture,
it is possible to use fgkaslr with modules on other architectures. To enable
this feature, select

CONFIG_MODULE_FG_KASLR=y

This option is selected automatically for X86_64 when CONFIG_FG_KASLR is set.

Disabling
---------
Disabling normal KASLR using the nokaslr command line option also disables
fgkaslr. It is also possible to disable fgkaslr separately by booting with
nofgkaslr on the commandline.

References
----------
There are a lot of academic papers which explore finer grained ASLR.
This paper in particular contributed the most to my implementation design
as well as my overall understanding of the problem space:

Selfrando: Securing the Tor Browser against De-anonymization Exploits,
M. Conti, S. Crane, T. Frassetto, et al.

For more information on how function layout impacts performance, see:

Optimizing Function Placement for Large-Scale Data-Center Applications,
G. Ottoni, B. Maher

Alexander Lobakin (7):
  linkage: add macros for putting ASM functions into own sections
  x86: conditionally place regular ASM functions into separate sections
  FG-KASLR: use a scripted approach to handle .text.* sections
  x86/boot: allow FG-KASLR to be selected
  arm64/crypto: conditionally place ASM functions into separate sections
  module: use a scripted approach for FG-KASLR
  maintainers: add MAINTAINERS entry for FG-KASLR

Kees Cook (2):
  x86/boot: Allow a "silent" kaslr random byte fetch
  x86/boot/compressed: Avoid duplicate malloc() implementations

Kristen Carlson Accardi (9):
  x86: tools/relocs: Support >64K section headers
  x86: Makefile: Add build and config option for CONFIG_FG_KASLR
  Make sure ORC lookup covers the entire _etext - _stext
  x86/tools: Add relative relocs for randomized functions
  x86: Add support for function granular KASLR
  kallsyms: Hide layout
  livepatch: only match unique symbols when using fgkaslr
  module: Reorder functions
  Documentation: add a documentation for FG-KASLR

Masahiro Yamada (3):
  kbuild: merge vmlinux_link() between the ordinary link and Clang LTO
  kbuild: do not remove 'linux' link in scripts/link-vmlinux.sh
  kbuild: merge vmlinux_link() between ARCH=um and other architectures

Sami Tolvanen (1):
  kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG

 .../admin-guide/kernel-parameters.txt         |   6 +
 Documentation/security/fgkaslr.rst            | 172 ++++
 Documentation/security/index.rst              |   1 +
 MAINTAINERS                                   |  12 +
 Makefile                                      |  17 +-
 arch/Kconfig                                  |   3 +
 arch/arm64/crypto/aes-ce-ccm-core.S           |  16 +-
 arch/arm64/crypto/aes-ce-core.S               |  16 +-
 arch/arm64/crypto/aes-ce.S                    |   4 +-
 arch/arm64/crypto/aes-cipher-core.S           |   8 +-
 arch/arm64/crypto/aes-modes.S                 |  16 +-
 arch/arm64/crypto/aes-neon.S                  |   4 +-
 arch/arm64/crypto/aes-neonbs-core.S           |  38 +-
 arch/arm64/crypto/chacha-neon-core.S          |  18 +-
 arch/arm64/crypto/crct10dif-ce-core.S         |  14 +-
 arch/arm64/crypto/ghash-ce-core.S             |  24 +-
 arch/arm64/crypto/nh-neon-core.S              |   4 +-
 arch/arm64/crypto/poly1305-armv8.pl           |  17 +
 arch/arm64/crypto/sha1-ce-core.S              |   4 +-
 arch/arm64/crypto/sha2-ce-core.S              |   4 +-
 arch/arm64/crypto/sha3-ce-core.S              |   4 +-
 arch/arm64/crypto/sha512-armv8.pl             |  11 +
 arch/arm64/crypto/sha512-ce-core.S            |   4 +-
 arch/arm64/crypto/sm3-ce-core.S               |   4 +-
 arch/arm64/crypto/sm4-ce-core.S               |   4 +-
 arch/x86/Kconfig                              |   1 +
 arch/x86/boot/compressed/Makefile             |   9 +-
 arch/x86/boot/compressed/fgkaslr.c            | 905 ++++++++++++++++++
 arch/x86/boot/compressed/kaslr.c              |   4 -
 arch/x86/boot/compressed/misc.c               | 157 ++-
 arch/x86/boot/compressed/misc.h               |  30 +
 arch/x86/boot/compressed/utils.c              |  13 +
 arch/x86/boot/compressed/vmlinux.symbols      |  19 +
 arch/x86/crypto/aegis128-aesni-asm.S          |  36 +-
 arch/x86/crypto/aes_ctrby8_avx-x86_64.S       |  12 +-
 arch/x86/crypto/aesni-intel_asm.S             | 116 ++-
 arch/x86/crypto/aesni-intel_avx-x86_64.S      |  32 +-
 arch/x86/crypto/blake2s-core.S                |   8 +-
 arch/x86/crypto/blowfish-x86_64-asm_64.S      |  16 +-
 arch/x86/crypto/camellia-aesni-avx-asm_64.S   |  28 +-
 arch/x86/crypto/camellia-aesni-avx2-asm_64.S  |  28 +-
 arch/x86/crypto/camellia-x86_64-asm_64.S      |  16 +-
 arch/x86/crypto/cast5-avx-x86_64-asm_64.S     |  24 +-
 arch/x86/crypto/cast6-avx-x86_64-asm_64.S     |  20 +-
 arch/x86/crypto/chacha-avx2-x86_64.S          |  12 +-
 arch/x86/crypto/chacha-avx512vl-x86_64.S      |  12 +-
 arch/x86/crypto/chacha-ssse3-x86_64.S         |  16 +-
 arch/x86/crypto/crc32-pclmul_asm.S            |   4 +-
 arch/x86/crypto/crc32c-pcl-intel-asm_64.S     |   4 +-
 arch/x86/crypto/crct10dif-pcl-asm_64.S        |   4 +-
 arch/x86/crypto/des3_ede-asm_64.S             |   8 +-
 arch/x86/crypto/ghash-clmulni-intel_asm.S     |  12 +-
 arch/x86/crypto/nh-avx2-x86_64.S              |   4 +-
 arch/x86/crypto/nh-sse2-x86_64.S              |   4 +-
 arch/x86/crypto/poly1305-x86_64-cryptogams.pl |   8 +-
 arch/x86/crypto/serpent-avx-x86_64-asm_64.S   |  20 +-
 arch/x86/crypto/serpent-avx2-asm_64.S         |  20 +-
 arch/x86/crypto/serpent-sse2-i586-asm_32.S    |   8 +-
 arch/x86/crypto/serpent-sse2-x86_64-asm_64.S  |   8 +-
 arch/x86/crypto/sha1_avx2_x86_64_asm.S        |   4 +-
 arch/x86/crypto/sha1_ni_asm.S                 |   4 +-
 arch/x86/crypto/sha1_ssse3_asm.S              |   4 +-
 arch/x86/crypto/sha256-avx-asm.S              |   4 +-
 arch/x86/crypto/sha256-avx2-asm.S             |   4 +-
 arch/x86/crypto/sha256-ssse3-asm.S            |   4 +-
 arch/x86/crypto/sha256_ni_asm.S               |   4 +-
 arch/x86/crypto/sha512-avx-asm.S              |   4 +-
 arch/x86/crypto/sha512-avx2-asm.S             |   4 +-
 arch/x86/crypto/sha512-ssse3-asm.S            |   4 +-
 arch/x86/crypto/twofish-avx-x86_64-asm_64.S   |  20 +-
 arch/x86/crypto/twofish-i586-asm_32.S         |   8 +-
 arch/x86/crypto/twofish-x86_64-asm_64-3way.S  |   8 +-
 arch/x86/crypto/twofish-x86_64-asm_64.S       |   8 +-
 arch/x86/entry/entry_32.S                     |  24 +-
 arch/x86/entry/entry_64.S                     |  18 +-
 arch/x86/entry/thunk_32.S                     |   4 +-
 arch/x86/entry/thunk_64.S                     |   8 +-
 arch/x86/include/asm/boot.h                   |  13 +-
 arch/x86/include/asm/paravirt.h               |   2 +-
 arch/x86/include/asm/qspinlock_paravirt.h     |   2 +-
 arch/x86/kernel/acpi/wakeup_32.S              |   9 +-
 arch/x86/kernel/acpi/wakeup_64.S              |  10 +-
 arch/x86/kernel/ftrace_32.S                   |  19 +-
 arch/x86/kernel/ftrace_64.S                   |  28 +-
 arch/x86/kernel/irqflags.S                    |   4 +-
 arch/x86/kernel/kprobes/core.c                |   3 +-
 arch/x86/kernel/kvm.c                         |   2 +-
 arch/x86/kernel/relocate_kernel_32.S          |   2 +
 arch/x86/kernel/relocate_kernel_64.S          |   2 +
 arch/x86/kernel/vmlinux.lds.S                 |   6 +-
 arch/x86/kvm/emulate.c                        |   2 +-
 arch/x86/kvm/vmx/vmenter.S                    |   8 +-
 arch/x86/lib/clear_page_64.S                  |  12 +-
 arch/x86/lib/cmpxchg16b_emu.S                 |   4 +-
 arch/x86/lib/copy_mc_64.S                     |   8 +-
 arch/x86/lib/copy_page_64.S                   |   7 +-
 arch/x86/lib/copy_user_64.S                   |  18 +-
 arch/x86/lib/csum-copy_64.S                   |   4 +-
 arch/x86/lib/error-inject.c                   |   3 +-
 arch/x86/lib/getuser.S                        |  37 +-
 arch/x86/lib/hweight.S                        |   9 +-
 arch/x86/lib/iomap_copy_64.S                  |   4 +-
 arch/x86/lib/kaslr.c                          |  18 +-
 arch/x86/lib/memmove_64.S                     |   4 +-
 arch/x86/lib/memset_64.S                      |  12 +-
 arch/x86/lib/msr-reg.S                        |   8 +-
 arch/x86/lib/putuser.S                        |  18 +-
 arch/x86/mm/mem_encrypt_boot.S                |   8 +-
 arch/x86/platform/efi/efi_stub_64.S           |   4 +-
 arch/x86/platform/efi/efi_thunk_64.S          |   4 +-
 arch/x86/power/hibernate_asm_32.S             |  14 +-
 arch/x86/power/hibernate_asm_64.S             |  14 +-
 arch/x86/tools/relocs.c                       | 135 ++-
 arch/x86/tools/relocs.h                       |   4 +-
 arch/x86/tools/relocs_common.c                |  15 +-
 arch/x86/xen/xen-asm.S                        |  49 +-
 arch/x86/xen/xen-head.S                       |  10 +-
 include/asm-generic/vmlinux.lds.h             |  41 +-
 include/linux/decompress/mm.h                 |  12 +-
 include/linux/linkage.h                       |  76 ++
 include/uapi/linux/elf.h                      |   1 +
 init/Kconfig                                  |  51 +
 kernel/kallsyms.c                             | 158 ++-
 kernel/livepatch/core.c                       |  11 +
 kernel/module.c                               |  91 +-
 scripts/Makefile.build                        |  27 +-
 scripts/Makefile.lib                          |   7 +
 scripts/Makefile.modfinal                     |  36 +-
 scripts/Makefile.modpost                      |  22 +-
 scripts/gen_autoksyms.sh                      |  12 -
 scripts/generate_text_sections.pl             | 149 +++
 scripts/link-vmlinux.sh                       | 104 +-
 scripts/module.lds.S                          |  14 +-
 133 files changed, 2771 insertions(+), 757 deletions(-)
 create mode 100644 Documentation/security/fgkaslr.rst
 create mode 100644 arch/x86/boot/compressed/fgkaslr.c
 create mode 100644 arch/x86/boot/compressed/utils.c
 create mode 100644 arch/x86/boot/compressed/vmlinux.symbols
 create mode 100755 scripts/generate_text_sections.pl

-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 01/22] kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
@ 2021-08-31 14:40 ` Alexander Lobakin
  2021-08-31 14:40 ` [PATCH v6 kspp-next 02/22] kbuild: merge vmlinux_link() between the ordinary link and Clang LTO Alexander Lobakin
                   ` (21 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:40 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux, Alexander Lobakin

From: Sami Tolvanen <samitolvanen@google.com>

With CONFIG_LTO_CLANG, we currently link modules into native
code just before modpost, which means with TRIM_UNUSED_KSYMS
enabled, we still look at the LLVM bitcode in the .o files when
generating the list of used symbols. As the bitcode doesn't
yet have calls to compiler intrinsics and llvm-nm doesn't see
function references that only exist in function-level inline
assembly, we currently need a whitelist for TRIM_UNUSED_KSYMS to
work with LTO.

This change moves module LTO linking to happen earlier, and
thus avoids the issue with LLVM bitcode and TRIM_UNUSED_KSYMS
entirely, allowing us to also drop the whitelist from
gen_autoksyms.sh.

Link: https://github.com/ClangBuiltLinux/linux/issues/1369
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Reviewed-by: Alexander Lobakin <alobakin@pm.me>
Tested-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---
 scripts/Makefile.build    | 27 ++++++++++++++++++++++++++-
 scripts/Makefile.lib      |  7 +++++++
 scripts/Makefile.modfinal | 21 ++-------------------
 scripts/Makefile.modpost  | 22 +++-------------------
 scripts/gen_autoksyms.sh  | 12 ------------
 5 files changed, 38 insertions(+), 51 deletions(-)

diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 02197cb8e3a7..ea549579bfb7 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -88,6 +88,10 @@ endif
 
 targets-for-modules := $(patsubst %.o, %.mod, $(filter %.o, $(obj-m)))
 
+ifdef CONFIG_LTO_CLANG
+targets-for-modules += $(patsubst %.o, %.lto.o, $(filter %.o, $(obj-m)))
+endif
+
 ifdef need-modorder
 targets-for-modules += $(obj)/modules.order
 endif
@@ -271,12 +275,33 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) $$(objtool_dep) FORCE
 	$(call if_changed_rule,cc_o_c)
 	$(call cmd,force_checksrc)
 
+ifdef CONFIG_LTO_CLANG
+# Module .o files may contain LLVM bitcode, compile them into native code
+# before ELF processing
+quiet_cmd_cc_lto_link_modules = LTO [M] $@
+cmd_cc_lto_link_modules =						\
+	$(LD) $(ld_flags) -r -o $@					\
+		$(shell [ -s $(@:.lto.o=.o.symversions) ] &&		\
+			echo -T $(@:.lto.o=.o.symversions))		\
+		--whole-archive $(filter-out FORCE,$^)
+
+ifdef CONFIG_STACK_VALIDATION
+# objtool was skipped for LLVM bitcode, run it now that we have compiled
+# modules into native code
+cmd_cc_lto_link_modules += ;						\
+	$(objtree)/tools/objtool/objtool $(objtool_args) --module $@
+endif
+
+$(obj)/%.lto.o: $(obj)/%.o FORCE
+	$(call if_changed,cc_lto_link_modules)
+endif
+
 cmd_mod = { \
 	echo $(if $($*-objs)$($*-y)$($*-m), $(addprefix $(obj)/, $($*-objs) $($*-y) $($*-m)), $(@:.mod=.o)); \
 	$(undefined_syms) echo; \
 	} > $@
 
-$(obj)/%.mod: $(obj)/%.o FORCE
+$(obj)/%.mod: $(obj)/%$(mod-prelink-ext).o FORCE
 	$(call if_changed,mod)
 
 quiet_cmd_cc_lst_c = MKLST   $@
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 10950559b223..af1c920a585c 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -225,6 +225,13 @@ dtc_cpp_flags  = -Wp,-MMD,$(depfile).pre.tmp -nostdinc                    \
 		 $(addprefix -I,$(DTC_INCLUDE))                          \
 		 -undef -D__DTS__
 
+ifeq ($(CONFIG_LTO_CLANG),y)
+# With CONFIG_LTO_CLANG, .o files in modules might be LLVM bitcode, so we
+# need to run LTO to compile them into native code (.lto.o) before further
+# processing.
+mod-prelink-ext := .lto
+endif
+
 # Objtool arguments are also needed for modfinal with LTO, so we define
 # then here to avoid duplication.
 objtool_args =								\
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index 5e9b8057fb24..ff805777431c 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -9,7 +9,7 @@ __modfinal:
 include include/config/auto.conf
 include $(srctree)/scripts/Kbuild.include
 
-# for c_flags and objtool_args
+# for c_flags and mod-prelink-ext
 include $(srctree)/scripts/Makefile.lib
 
 # find all modules listed in modules.order
@@ -30,23 +30,6 @@ quiet_cmd_cc_o_c = CC [M]  $@
 
 ARCH_POSTLINK := $(wildcard $(srctree)/arch/$(SRCARCH)/Makefile.postlink)
 
-ifdef CONFIG_LTO_CLANG
-# With CONFIG_LTO_CLANG, reuse the object file we compiled for modpost to
-# avoid a second slow LTO link
-prelink-ext := .lto
-
-# ELF processing was skipped earlier because we didn't have native code,
-# so let's now process the prelinked binary before we link the module.
-
-ifdef CONFIG_STACK_VALIDATION
-cmd_ld_ko_o +=								\
-	$(objtree)/tools/objtool/objtool $(objtool_args)		\
-		$(@:.ko=$(prelink-ext).o);
-
-endif # CONFIG_STACK_VALIDATION
-
-endif # CONFIG_LTO_CLANG
-
 quiet_cmd_ld_ko_o = LD [M]  $@
       cmd_ld_ko_o +=							\
 	$(LD) -r $(KBUILD_LDFLAGS)					\
@@ -72,7 +55,7 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check),      \
 
 
 # Re-generate module BTFs if either module's .ko or vmlinux changed
-$(modules): %.ko: %$(prelink-ext).o %.mod.o scripts/module.lds $(if $(KBUILD_BUILTIN),vmlinux) FORCE
+$(modules): %.ko: %$(mod-prelink-ext).o %.mod.o scripts/module.lds $(if $(KBUILD_BUILTIN),vmlinux) FORCE
 	+$(call if_changed_except,ld_ko_o,vmlinux)
 ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 	+$(if $(newer-prereqs),$(call cmd,btf_ko))
diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost
index c383ba33d837..eef56d629799 100644
--- a/scripts/Makefile.modpost
+++ b/scripts/Makefile.modpost
@@ -41,7 +41,7 @@ __modpost:
 include include/config/auto.conf
 include $(srctree)/scripts/Kbuild.include
 
-# for ld_flags
+# for mod-prelink-ext
 include $(srctree)/scripts/Makefile.lib
 
 MODPOST = scripts/mod/modpost								\
@@ -118,22 +118,6 @@ $(input-symdump):
 	@echo >&2 '         Modules may not have dependencies or modversions.'
 	@echo >&2 '         You may get many unresolved symbol warnings.'
 
-ifdef CONFIG_LTO_CLANG
-# With CONFIG_LTO_CLANG, .o files might be LLVM bitcode, so we need to run
-# LTO to compile them into native code before running modpost
-prelink-ext := .lto
-
-quiet_cmd_cc_lto_link_modules = LTO [M] $@
-cmd_cc_lto_link_modules =						\
-	$(LD) $(ld_flags) -r -o $@					\
-		$(shell [ -s $(@:.lto.o=.o.symversions) ] &&		\
-			echo -T $(@:.lto.o=.o.symversions))		\
-		--whole-archive $^
-
-%.lto.o: %.o
-	$(call if_changed,cc_lto_link_modules)
-endif
-
 modules := $(sort $(shell cat $(MODORDER)))
 
 # KBUILD_MODPOST_WARN can be set to avoid error out in case of undefined symbols
@@ -144,9 +128,9 @@ endif
 # Read out modules.order to pass in modpost.
 # Otherwise, allmodconfig would fail with "Argument list too long".
 quiet_cmd_modpost = MODPOST $@
-      cmd_modpost = sed 's/\.ko$$/$(prelink-ext)\.o/' $< | $(MODPOST) -T -
+      cmd_modpost = sed 's/\.ko$$/$(mod-prelink-ext)\.o/' $< | $(MODPOST) -T -
 
-$(output-symdump): $(MODORDER) $(input-symdump) $(modules:.ko=$(prelink-ext).o) FORCE
+$(output-symdump): $(MODORDER) $(input-symdump) $(modules:.ko=$(mod-prelink-ext).o) FORCE
 	$(call if_changed,modpost)
 
 targets += $(output-symdump)
diff --git a/scripts/gen_autoksyms.sh b/scripts/gen_autoksyms.sh
index da320151e7c3..6ed0d225c8b1 100755
--- a/scripts/gen_autoksyms.sh
+++ b/scripts/gen_autoksyms.sh
@@ -26,18 +26,6 @@ if [ -n "$CONFIG_MODVERSIONS" ]; then
 	needed_symbols="$needed_symbols module_layout"
 fi
 
-# With CONFIG_LTO_CLANG, LLVM bitcode has not yet been compiled into a binary
-# when the .mod files are generated, which means they don't yet contain
-# references to certain symbols that will be present in the final binaries.
-if [ -n "$CONFIG_LTO_CLANG" ]; then
-	# intrinsic functions
-	needed_symbols="$needed_symbols memcpy memmove memset"
-	# ftrace
-	needed_symbols="$needed_symbols _mcount"
-	# stack protector symbols
-	needed_symbols="$needed_symbols __stack_chk_fail __stack_chk_guard"
-fi
-
 ksym_wl=
 if [ -n "$CONFIG_UNUSED_KSYMS_WHITELIST" ]; then
 	# Use 'eval' to expand the whitelist path and check if it is relative
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 02/22] kbuild: merge vmlinux_link() between the ordinary link and Clang LTO
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
  2021-08-31 14:40 ` [PATCH v6 kspp-next 01/22] kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG Alexander Lobakin
@ 2021-08-31 14:40 ` Alexander Lobakin
  2021-08-31 14:40 ` [PATCH v6 kspp-next 03/22] kbuild: do not remove 'linux' link in scripts/link-vmlinux.sh Alexander Lobakin
                   ` (20 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:40 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

From: Masahiro Yamada <masahiroy@kernel.org>

When Clang LTO is enabled, vmlinux_link() reuses vmlinux.o instead of
re-linking ${KBUILD_VMLINUX_OBJS} and ${KBUILD_VMLINUX_LIBS}.

That is the only difference here, so merge the similar code.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
 scripts/link-vmlinux.sh | 30 ++++++++++++++----------------
 1 file changed, 14 insertions(+), 16 deletions(-)

diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index 36ef7b37fc5d..a6c4d0bce3ba 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -154,12 +154,23 @@ vmlinux_link()
 	local objects
 	local strip_debug
 	local map_option
+	local objs
+	local libs
 
 	info LD ${output}
 
 	# skip output file argument
 	shift
 
+	if [ -n "${CONFIG_LTO_CLANG}" ]; then
+		# Use vmlinux.o instead of performing the slow LTO link again.
+		objs=vmlinux.o
+		libs=
+	else
+		objs="${KBUILD_VMLINUX_OBJS}"
+		libs="${KBUILD_VMLINUX_LIBS}"
+	fi
+
 	# The kallsyms linking does not need debug symbols included.
 	if [ "$output" != "${output#.tmp_vmlinux.kallsyms}" ] ; then
 		strip_debug=-Wl,--strip-debug
@@ -170,22 +181,9 @@ vmlinux_link()
 	fi
 
 	if [ "${SRCARCH}" != "um" ]; then
-		if [ -n "${CONFIG_LTO_CLANG}" ]; then
-			# Use vmlinux.o instead of performing the slow LTO
-			# link again.
-			objects="--whole-archive		\
-				vmlinux.o 			\
-				--no-whole-archive		\
-				${@}"
-		else
-			objects="--whole-archive		\
-				${KBUILD_VMLINUX_OBJS}		\
-				--no-whole-archive		\
-				--start-group			\
-				${KBUILD_VMLINUX_LIBS}		\
-				--end-group			\
-				${@}"
-		fi
+		objects="--whole-archive ${objs} --no-whole-archive	\
+			 --start-group ${libs} --end-group		\
+			 $@"
 
 		${LD} ${KBUILD_LDFLAGS} ${LDFLAGS_vmlinux}	\
 			${strip_debug#-Wl,}			\
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 03/22] kbuild: do not remove 'linux' link in scripts/link-vmlinux.sh
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
  2021-08-31 14:40 ` [PATCH v6 kspp-next 01/22] kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG Alexander Lobakin
  2021-08-31 14:40 ` [PATCH v6 kspp-next 02/22] kbuild: merge vmlinux_link() between the ordinary link and Clang LTO Alexander Lobakin
@ 2021-08-31 14:40 ` Alexander Lobakin
  2021-08-31 14:40 ` [PATCH v6 kspp-next 04/22] kbuild: merge vmlinux_link() between ARCH=um and other architectures Alexander Lobakin
                   ` (19 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:40 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

From: Masahiro Yamada <masahiroy@kernel.org>

arch/um/Makefile passes the -f option to the ln command:

  linux: vmlinux
          @echo '  LINK $@'
          $(Q)ln -f $< $@

So, the hard link is always re-created, and the old one is removed
anyway.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
 scripts/link-vmlinux.sh | 1 -
 1 file changed, 1 deletion(-)

diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index a6c4d0bce3ba..7b9c62e4d54a 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -206,7 +206,6 @@ vmlinux_link()
 			-Wl,-T,${lds}				\
 			${objects}				\
 			-lutil -lrt -lpthread
-		rm -f linux
 	fi
 }
 
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 04/22] kbuild: merge vmlinux_link() between ARCH=um and other architectures
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (2 preceding siblings ...)
  2021-08-31 14:40 ` [PATCH v6 kspp-next 03/22] kbuild: do not remove 'linux' link in scripts/link-vmlinux.sh Alexander Lobakin
@ 2021-08-31 14:40 ` Alexander Lobakin
  2021-08-31 14:40 ` [PATCH v6 kspp-next 05/22] x86: tools/relocs: Support >64K section headers Alexander Lobakin
                   ` (18 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:40 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

From: Masahiro Yamada <masahiroy@kernel.org>

For ARCH=um, ${CC} is used as the linker driver. Hence, the linker
options are prefixed with -Wl, .

Merge the similar code.

I replaced the -T option with the long option --script= so that it
works well with/without ${wl}.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
 scripts/link-vmlinux.sh | 56 +++++++++++++++++------------------------
 1 file changed, 23 insertions(+), 33 deletions(-)

diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index 7b9c62e4d54a..d74cee5c4326 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -149,13 +149,12 @@ objtool_link()
 # ${2}, ${3}, ... - optional extra .o files
 vmlinux_link()
 {
-	local lds="${objtree}/${KBUILD_LDS}"
 	local output=${1}
-	local objects
-	local strip_debug
-	local map_option
 	local objs
 	local libs
+	local ld
+	local ldflags
+	local ldlibs
 
 	info LD ${output}
 
@@ -171,42 +170,33 @@ vmlinux_link()
 		libs="${KBUILD_VMLINUX_LIBS}"
 	fi
 
+	if [ "${SRCARCH}" = "um" ]; then
+		wl=-Wl,
+		ld="${CC}"
+		ldflags="${CFLAGS_vmlinux}"
+		ldlibs="-lutil -lrt -lpthread"
+	else
+		wl=
+		ld="${LD}"
+		ldflags="${KBUILD_LDFLAGS} ${LDFLAGS_vmlinux}"
+		ldlibs=
+	fi
+
+	ldflags="${ldflags} ${wl}--script=${objtree}/${KBUILD_LDS}"
+
 	# The kallsyms linking does not need debug symbols included.
 	if [ "$output" != "${output#.tmp_vmlinux.kallsyms}" ] ; then
-		strip_debug=-Wl,--strip-debug
+		ldflags="${ldflags} ${wl}--strip-debug"
 	fi
 
 	if [ -n "${CONFIG_VMLINUX_MAP}" ]; then
-		map_option="-Map=${output}.map"
+		ldflags="${ldflags} ${wl}-Map=${output}.map"
 	fi
 
-	if [ "${SRCARCH}" != "um" ]; then
-		objects="--whole-archive ${objs} --no-whole-archive	\
-			 --start-group ${libs} --end-group		\
-			 $@"
-
-		${LD} ${KBUILD_LDFLAGS} ${LDFLAGS_vmlinux}	\
-			${strip_debug#-Wl,}			\
-			-o ${output}				\
-			${map_option}				\
-			-T ${lds} ${objects}
-	else
-		objects="-Wl,--whole-archive			\
-			${KBUILD_VMLINUX_OBJS}			\
-			-Wl,--no-whole-archive			\
-			-Wl,--start-group			\
-			${KBUILD_VMLINUX_LIBS}			\
-			-Wl,--end-group				\
-			${@}"
-
-		${CC} ${CFLAGS_vmlinux}				\
-			${strip_debug}				\
-			-o ${output}				\
-			${map_option:+-Wl,${map_option}}	\
-			-Wl,-T,${lds}				\
-			${objects}				\
-			-lutil -lrt -lpthread
-	fi
+	${ld} ${ldflags} -o ${output}					\
+		${wl}--whole-archive ${objs} ${wl}--no-whole-archive	\
+		${wl}--start-group ${libs} ${wl}--end-group		\
+		$@ ${ldlibs}
 }
 
 # generate .BTF typeinfo from DWARF debuginfo
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 05/22] x86: tools/relocs: Support >64K section headers
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (3 preceding siblings ...)
  2021-08-31 14:40 ` [PATCH v6 kspp-next 04/22] kbuild: merge vmlinux_link() between ARCH=um and other architectures Alexander Lobakin
@ 2021-08-31 14:40 ` Alexander Lobakin
  2021-08-31 14:40 ` [PATCH v6 kspp-next 06/22] x86/boot: Allow a "silent" kaslr random byte fetch Alexander Lobakin
                   ` (17 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:40 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

From: Kristen Carlson Accardi <kristen@linux.intel.com>

While the relocs tool already supports finding the total number
of section headers if vmlinux exceeds 64K sections, it fails to
read the extended symbol table to get section header indexes for symbols,
causing incorrect symbol table indexes to be used when there are > 64K
symbols.

Parse the elf file to read the extended symbol table info, and then
replace all direct references to st_shndx with calls to sym_index(),
which will determine whether the value can be read directly or
whether the value should be pulled out of the extended table.

Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Tested-by: Tony Luck <tony.luck@intel.com>
Acked-by: H. Peter Anvin (Intel) <hpa@zytor.com>
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 arch/x86/tools/relocs.c | 103 ++++++++++++++++++++++++++++++----------
 1 file changed, 78 insertions(+), 25 deletions(-)

diff --git a/arch/x86/tools/relocs.c b/arch/x86/tools/relocs.c
index 9ba700dc47de..ec50dfad407c 100644
--- a/arch/x86/tools/relocs.c
+++ b/arch/x86/tools/relocs.c
@@ -14,6 +14,10 @@
 static Elf_Ehdr		ehdr;
 static unsigned long	shnum;
 static unsigned int	shstrndx;
+static unsigned int	shsymtabndx;
+static unsigned int	shxsymtabndx;
+
+static int sym_index(Elf_Sym *sym);
 
 struct relocs {
 	uint32_t	*offset;
@@ -32,6 +36,7 @@ struct section {
 	Elf_Shdr       shdr;
 	struct section *link;
 	Elf_Sym        *symtab;
+	Elf32_Word     *xsymtab;
 	Elf_Rel        *reltab;
 	char           *strtab;
 };
@@ -265,7 +270,7 @@ static const char *sym_name(const char *sym_strtab, Elf_Sym *sym)
 		name = sym_strtab + sym->st_name;
 	}
 	else {
-		name = sec_name(sym->st_shndx);
+		name = sec_name(sym_index(sym));
 	}
 	return name;
 }
@@ -335,6 +340,23 @@ static uint64_t elf64_to_cpu(uint64_t val)
 #define elf_xword_to_cpu(x)	elf32_to_cpu(x)
 #endif
 
+static int sym_index(Elf_Sym *sym)
+{
+	Elf_Sym *symtab = secs[shsymtabndx].symtab;
+	Elf32_Word *xsymtab = secs[shxsymtabndx].xsymtab;
+	unsigned long offset;
+	int index;
+
+	if (sym->st_shndx != SHN_XINDEX)
+		return sym->st_shndx;
+
+	/* calculate offset of sym from head of table. */
+	offset = (unsigned long)sym - (unsigned long)symtab;
+	index = offset / sizeof(*sym);
+
+	return elf32_to_cpu(xsymtab[index]);
+}
+
 static void read_ehdr(FILE *fp)
 {
 	if (fread(&ehdr, sizeof(ehdr), 1, fp) != 1) {
@@ -468,31 +490,60 @@ static void read_strtabs(FILE *fp)
 static void read_symtabs(FILE *fp)
 {
 	int i,j;
+
 	for (i = 0; i < shnum; i++) {
 		struct section *sec = &secs[i];
-		if (sec->shdr.sh_type != SHT_SYMTAB) {
+		int num_syms;
+
+		switch (sec->shdr.sh_type) {
+		case SHT_SYMTAB_SHNDX:
+			sec->xsymtab = malloc(sec->shdr.sh_size);
+			if (!sec->xsymtab) {
+				die("malloc of %d bytes for xsymtab failed\n",
+				    sec->shdr.sh_size);
+			}
+			if (fseek(fp, sec->shdr.sh_offset, SEEK_SET) < 0) {
+				die("Seek to %d failed: %s\n",
+				    sec->shdr.sh_offset, strerror(errno));
+			}
+			if (fread(sec->xsymtab, 1, sec->shdr.sh_size, fp)
+			    != sec->shdr.sh_size) {
+				die("Cannot read extended symbol table: %s\n",
+				    strerror(errno));
+			}
+			shxsymtabndx = i;
+			continue;
+
+		case SHT_SYMTAB:
+			num_syms = sec->shdr.sh_size / sizeof(Elf_Sym);
+
+			sec->symtab = malloc(sec->shdr.sh_size);
+			if (!sec->symtab) {
+				die("malloc of %d bytes for symtab failed\n",
+				    sec->shdr.sh_size);
+			}
+			if (fseek(fp, sec->shdr.sh_offset, SEEK_SET) < 0) {
+				die("Seek to %d failed: %s\n",
+				    sec->shdr.sh_offset, strerror(errno));
+			}
+			if (fread(sec->symtab, 1, sec->shdr.sh_size, fp)
+			    != sec->shdr.sh_size) {
+				die("Cannot read symbol table: %s\n",
+				    strerror(errno));
+			}
+			for (j = 0; j < num_syms; j++) {
+				Elf_Sym *sym = &sec->symtab[j];
+
+				sym->st_name  = elf_word_to_cpu(sym->st_name);
+				sym->st_value = elf_addr_to_cpu(sym->st_value);
+				sym->st_size  = elf_xword_to_cpu(sym->st_size);
+				sym->st_shndx = elf_half_to_cpu(sym->st_shndx);
+			}
+			shsymtabndx = i;
+			continue;
+
+		default:
 			continue;
-		}
-		sec->symtab = malloc(sec->shdr.sh_size);
-		if (!sec->symtab) {
-			die("malloc of %d bytes for symtab failed\n",
-				sec->shdr.sh_size);
-		}
-		if (fseek(fp, sec->shdr.sh_offset, SEEK_SET) < 0) {
-			die("Seek to %d failed: %s\n",
-				sec->shdr.sh_offset, strerror(errno));
-		}
-		if (fread(sec->symtab, 1, sec->shdr.sh_size, fp)
-		    != sec->shdr.sh_size) {
-			die("Cannot read symbol table: %s\n",
-				strerror(errno));
-		}
-		for (j = 0; j < sec->shdr.sh_size/sizeof(Elf_Sym); j++) {
-			Elf_Sym *sym = &sec->symtab[j];
-			sym->st_name  = elf_word_to_cpu(sym->st_name);
-			sym->st_value = elf_addr_to_cpu(sym->st_value);
-			sym->st_size  = elf_xword_to_cpu(sym->st_size);
-			sym->st_shndx = elf_half_to_cpu(sym->st_shndx);
 		}
 	}
 }
@@ -759,7 +810,9 @@ static void percpu_init(void)
  */
 static int is_percpu_sym(ElfW(Sym) *sym, const char *symname)
 {
-	return (sym->st_shndx == per_cpu_shndx) &&
+	int shndx = sym_index(sym);
+
+	return (shndx == per_cpu_shndx) &&
 		strcmp(symname, "__init_begin") &&
 		strcmp(symname, "__per_cpu_load") &&
 		strncmp(symname, "init_per_cpu_", 13);
@@ -1092,7 +1145,7 @@ static int do_reloc_info(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,
 		sec_name(sec->shdr.sh_info),
 		rel_type(ELF_R_TYPE(rel->r_info)),
 		symname,
-		sec_name(sym->st_shndx));
+		sec_name(sym_index(sym)));
 	return 0;
 }
 
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 06/22] x86/boot: Allow a "silent" kaslr random byte fetch
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (4 preceding siblings ...)
  2021-08-31 14:40 ` [PATCH v6 kspp-next 05/22] x86: tools/relocs: Support >64K section headers Alexander Lobakin
@ 2021-08-31 14:40 ` Alexander Lobakin
  2021-08-31 14:40 ` [PATCH v6 kspp-next 07/22] x86: Makefile: Add build and config option for CONFIG_FG_KASLR Alexander Lobakin
                   ` (16 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:40 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

From: Kees Cook <keescook@chromium.org>

Under earlyprintk, each RNG call produces a debug report line. When
shuffling hundreds of functions, this is not useful information (each
line is identical and tells us nothing new). Instead, allow for a NULL
"purpose" to suppress the debug reporting.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 arch/x86/lib/kaslr.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/arch/x86/lib/kaslr.c b/arch/x86/lib/kaslr.c
index a53665116458..2b3eb8c948a3 100644
--- a/arch/x86/lib/kaslr.c
+++ b/arch/x86/lib/kaslr.c
@@ -56,11 +56,14 @@ unsigned long kaslr_get_random_long(const char *purpose)
 	unsigned long raw, random = get_boot_seed();
 	bool use_i8254 = true;
 
-	debug_putstr(purpose);
-	debug_putstr(" KASLR using");
+	if (purpose) {
+		debug_putstr(purpose);
+		debug_putstr(" KASLR using");
+	}
 
 	if (has_cpuflag(X86_FEATURE_RDRAND)) {
-		debug_putstr(" RDRAND");
+		if (purpose)
+			debug_putstr(" RDRAND");
 		if (rdrand_long(&raw)) {
 			random ^= raw;
 			use_i8254 = false;
@@ -68,7 +71,8 @@ unsigned long kaslr_get_random_long(const char *purpose)
 	}
 
 	if (has_cpuflag(X86_FEATURE_TSC)) {
-		debug_putstr(" RDTSC");
+		if (purpose)
+			debug_putstr(" RDTSC");
 		raw = rdtsc();
 
 		random ^= raw;
@@ -76,7 +80,8 @@ unsigned long kaslr_get_random_long(const char *purpose)
 	}
 
 	if (use_i8254) {
-		debug_putstr(" i8254");
+		if (purpose)
+			debug_putstr(" i8254");
 		random ^= i8254();
 	}
 
@@ -86,7 +91,8 @@ unsigned long kaslr_get_random_long(const char *purpose)
 	    : "a" (random), "rm" (mix_const));
 	random += raw;
 
-	debug_putstr("...\n");
+	if (purpose)
+		debug_putstr("...\n");
 
 	return random;
 }
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 07/22] x86: Makefile: Add build and config option for CONFIG_FG_KASLR
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (5 preceding siblings ...)
  2021-08-31 14:40 ` [PATCH v6 kspp-next 06/22] x86/boot: Allow a "silent" kaslr random byte fetch Alexander Lobakin
@ 2021-08-31 14:40 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 08/22] Make sure ORC lookup covers the entire _etext - _stext Alexander Lobakin
                   ` (15 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:40 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

From: Kristen Carlson Accardi <kristen@linux.intel.com>

Allow user to select CONFIG_FG_KASLR if dependencies are met. Change
the make file to build with -ffunction-sections if CONFIG_FG_KASLR.

While the only architecture that supports CONFIG_FG_KASLR does not
currently enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION, make sure these
2 features play nicely together for the future by ensuring that if
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is selected when used with
CONFIG_FG_KASLR the function sections will not be consolidated back
into .text. Thanks to Kees Cook for the dead code elimination changes.

Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Tested-by: Tony Luck <tony.luck@intel.com>
[ alobakin:
 - improve cflags management in the top Makefile
 - move ARCH_HAS_FG_KASLR to the top arch/Kconfig
 - add symtab_shndx to the list of known sections ]
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 Makefile                          | 13 ++++++++++++-
 arch/Kconfig                      |  3 +++
 include/asm-generic/vmlinux.lds.h | 20 ++++++++++++++++++--
 init/Kconfig                      | 12 ++++++++++++
 4 files changed, 45 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 61741e9d9c6e..283876e170f7 100644
--- a/Makefile
+++ b/Makefile
@@ -918,8 +918,19 @@ ifdef CONFIG_DEBUG_SECTION_MISMATCH
 KBUILD_CFLAGS += $(call cc-option, -fno-inline-functions-called-once)
 endif
 
+# ClangLTO implies -ffunction-sections -fdata-sections, no need
+# to specify them manually and trigger a pointless full rebuild
+ifndef CONFIG_LTO_CLANG
+ifneq ($(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION)$(CONFIG_FG_KASLR),)
+KBUILD_CFLAGS_KERNEL += -ffunction-sections
+endif
+
+ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
+KBUILD_CFLAGS_KERNEL += -fdata-sections
+endif
+endif # CONFIG_LTO_CLANG
+
 ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
-KBUILD_CFLAGS_KERNEL += -ffunction-sections -fdata-sections
 LDFLAGS_vmlinux += --gc-sections
 endif
 
diff --git a/arch/Kconfig b/arch/Kconfig
index 129df498a8e1..e7a9a43eee90 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -1282,6 +1282,9 @@ config ARCH_SPLIT_ARG64
 config ARCH_HAS_ELFCORE_COMPAT
 	bool
 
+config ARCH_HAS_FG_KASLR
+	bool
+
 source "kernel/gcov/Kconfig"
 
 source "scripts/gcc-plugins/Kconfig"
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 62669b36a772..5d6da19b02bc 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -97,14 +97,12 @@
  * sections to be brought in with rodata.
  */
 #if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CLANG)
-#define TEXT_MAIN .text .text.[0-9a-zA-Z_]*
 #define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..L* .data..compoundliteral* .data.$__unnamed_* .data.$L*
 #define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]*
 #define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* .rodata..L*
 #define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..compoundliteral*
 #define SBSS_MAIN .sbss .sbss.[0-9a-zA-Z_]*
 #else
-#define TEXT_MAIN .text
 #define DATA_MAIN .data
 #define SDATA_MAIN .sdata
 #define RODATA_MAIN .rodata
@@ -112,6 +110,23 @@
 #define SBSS_MAIN .sbss
 #endif
 
+/*
+ * LTO_CLANG, LD_DEAD_CODE_DATA_ELIMINATION and FG_KASLR options enable
+ * -ffunction-sections, which produces separately named .text sections. In
+ * the case of CONFIG_FG_KASLR, they need to stay distict so they can be
+ * separately randomized. Without CONFIG_FG_KASLR, the separate .text
+ * sections can be collected back into a common section, which makes the
+ * resulting image slightly smaller
+ */
+#if (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+     defined(CONFIG_LTO_CLANG)) && !defined(CONFIG_FG_KASLR)
+#define TEXT_MAIN		.text .text.[0-9a-zA-Z_]*
+#elif defined(CONFIG_FG_KASLR)
+#define TEXT_MAIN		.text.__unused__
+#else
+#define TEXT_MAIN		.text
+#endif
+
 /*
  * GCC 4.5 and later have a 32 bytes section alignment for structures.
  * Except GCC 4.9, that feels the need to align on 64 bytes.
@@ -842,6 +857,7 @@
 #define ELF_DETAILS							\
 		.comment 0 : { *(.comment) }				\
 		.symtab 0 : { *(.symtab) }				\
+		.symtab_shndx 0 : { *(.symtab_shndx) }			\
 		.strtab 0 : { *(.strtab) }				\
 		.shstrtab 0 : { *(.shstrtab) }
 
diff --git a/init/Kconfig b/init/Kconfig
index 55f9f7738ebb..cd1440b6a566 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2021,6 +2021,18 @@ config PROFILING
 config TRACEPOINTS
 	bool
 
+config FG_KASLR
+	bool "Function Granular Kernel Address Space Layout Randomization"
+	depends on ARCH_HAS_FG_KASLR
+	help
+	  This option improves the randomness of the kernel text
+	  over basic Kernel Address Space Layout Randomization (KASLR)
+	  by reordering the kernel text at boot time. This feature
+	  uses information generated at compile time to re-layout the
+	  kernel text section at boot time at function level granularity.
+
+	  If unsure, say N.
+
 endmenu		# General setup
 
 source "arch/Kconfig"
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 08/22] Make sure ORC lookup covers the entire _etext - _stext
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (6 preceding siblings ...)
  2021-08-31 14:40 ` [PATCH v6 kspp-next 07/22] x86: Makefile: Add build and config option for CONFIG_FG_KASLR Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 09/22] x86/tools: Add relative relocs for randomized functions Alexander Lobakin
                   ` (14 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

From: Kristen Carlson Accardi <kristen@linux.intel.com>

When using -ffunction-sections to place each function in
it's own text section so it can be randomized at load time, the
linker will place most of the functions into separate .text.*
sections. SIZEOF(.text) won't work here for calculating the ORC
lookup table size, the total text size must be calculated to
include .text AND all .text.*.

Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Tested-by: Tony Luck <tony.luck@intel.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
[ alobakin: move it to vmlinux.lds.h and make arch-indep ]
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 include/asm-generic/vmlinux.lds.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 5d6da19b02bc..01fdeb5dd216 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -887,10 +887,11 @@
 		KEEP(*(.orc_unwind))					\
 		__stop_orc_unwind = .;					\
 	}								\
+	text_size = _etext - _stext;					\
 	. = ALIGN(4);							\
 	.orc_lookup : AT(ADDR(.orc_lookup) - LOAD_OFFSET) {		\
 		orc_lookup = .;						\
-		. += (((SIZEOF(.text) + LOOKUP_BLOCK_SIZE - 1) /	\
+		. += (((text_size + LOOKUP_BLOCK_SIZE - 1) /		\
 			LOOKUP_BLOCK_SIZE) + 1) * 4;			\
 		orc_lookup_end = .;					\
 	}
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 09/22] x86/tools: Add relative relocs for randomized functions
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (7 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 08/22] Make sure ORC lookup covers the entire _etext - _stext Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 10/22] x86/boot/compressed: Avoid duplicate malloc() implementations Alexander Lobakin
                   ` (13 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

From: Kristen Carlson Accardi <kristen@linux.intel.com>

When reordering functions, the relative offsets for relocs that
are either in the randomized sections, or refer to the randomized
sections will need to be adjusted. Add code to detect whether a
reloc satisfies these cases, and if so, add them to the appropriate
reloc list.

Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Tested-by: Tony Luck <tony.luck@intel.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 arch/x86/boot/compressed/Makefile |  7 ++++++-
 arch/x86/tools/relocs.c           | 32 +++++++++++++++++++++++++++----
 arch/x86/tools/relocs.h           |  4 ++--
 arch/x86/tools/relocs_common.c    | 15 ++++++++++-----
 4 files changed, 46 insertions(+), 12 deletions(-)

diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 431bf7f846c3..c31a24161fbf 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -106,6 +106,11 @@ $(obj)/vmlinux: $(vmlinux-objs-y) $(efi-obj-y) FORCE
 	$(call if_changed,ld)
 
 OBJCOPYFLAGS_vmlinux.bin :=  -R .comment -S
+
+ifdef CONFIG_FG_KASLR
+RELOCS_ARGS += --fg-kaslr
+endif
+
 $(obj)/vmlinux.bin: vmlinux FORCE
 	$(call if_changed,objcopy)
 
@@ -113,7 +118,7 @@ targets += $(patsubst $(obj)/%,%,$(vmlinux-objs-y)) vmlinux.bin.all vmlinux.relo
 
 CMD_RELOCS = arch/x86/tools/relocs
 quiet_cmd_relocs = RELOCS  $@
-      cmd_relocs = $(CMD_RELOCS) $< > $@;$(CMD_RELOCS) --abs-relocs $<
+      cmd_relocs = $(CMD_RELOCS) $(RELOCS_ARGS) $< > $@;$(CMD_RELOCS) $(RELOCS_ARGS) --abs-relocs $<
 $(obj)/vmlinux.relocs: vmlinux FORCE
 	$(call if_changed,relocs)
 
diff --git a/arch/x86/tools/relocs.c b/arch/x86/tools/relocs.c
index ec50dfad407c..5ae6d1b8ea03 100644
--- a/arch/x86/tools/relocs.c
+++ b/arch/x86/tools/relocs.c
@@ -42,6 +42,8 @@ struct section {
 };
 static struct section *secs;
 
+static int fgkaslr_mode;
+
 static const char * const sym_regex_kernel[S_NSYMTYPES] = {
 /*
  * Following symbols have been audited. There values are constant and do
@@ -818,6 +820,24 @@ static int is_percpu_sym(ElfW(Sym) *sym, const char *symname)
 		strncmp(symname, "init_per_cpu_", 13);
 }
 
+static int is_function_section(struct section *sec)
+{
+	if (!fgkaslr_mode)
+		return 0;
+
+	return !strncmp(sec_name(sec->shdr.sh_info), ".text.", 6);
+}
+
+static int is_randomized_sym(ElfW(Sym) *sym)
+{
+	if (!fgkaslr_mode)
+		return 0;
+
+	if (sym->st_shndx > shnum)
+		return 0;
+
+	return !strncmp(sec_name(sym_index(sym)), ".text.", 6);
+}
 
 static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,
 		      const char *symname)
@@ -843,12 +863,15 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,
 	case R_X86_64_PC32:
 	case R_X86_64_PLT32:
 		/*
-		 * PC relative relocations don't need to be adjusted unless
-		 * referencing a percpu symbol.
+		 * we need to keep pc relative relocations for sections which
+		 * might be randomized, and for the percpu section.
+		 * We also need to keep relocations for any offset which might
+		 * reference an address in a section which has been randomized.
 		 *
 		 * NB: R_X86_64_PLT32 can be treated as R_X86_64_PC32.
 		 */
-		if (is_percpu_sym(sym, symname))
+		if (is_function_section(sec) || is_randomized_sym(sym) ||
+		    is_percpu_sym(sym, symname))
 			add_reloc(&relocs32neg, offset);
 		break;
 
@@ -1163,8 +1186,9 @@ static void print_reloc_info(void)
 
 void process(FILE *fp, int use_real_mode, int as_text,
 	     int show_absolute_syms, int show_absolute_relocs,
-	     int show_reloc_info)
+	     int show_reloc_info, int fgkaslr)
 {
+	fgkaslr_mode = fgkaslr;
 	regex_init(use_real_mode);
 	read_ehdr(fp);
 	read_shdrs(fp);
diff --git a/arch/x86/tools/relocs.h b/arch/x86/tools/relocs.h
index 43c83c0fd22c..f582895c04dd 100644
--- a/arch/x86/tools/relocs.h
+++ b/arch/x86/tools/relocs.h
@@ -31,8 +31,8 @@ enum symtype {
 
 void process_32(FILE *fp, int use_real_mode, int as_text,
 		int show_absolute_syms, int show_absolute_relocs,
-		int show_reloc_info);
+		int show_reloc_info, int fgkaslr);
 void process_64(FILE *fp, int use_real_mode, int as_text,
 		int show_absolute_syms, int show_absolute_relocs,
-		int show_reloc_info);
+		int show_reloc_info, int fgkaslr);
 #endif /* RELOCS_H */
diff --git a/arch/x86/tools/relocs_common.c b/arch/x86/tools/relocs_common.c
index 6634352a20bc..b1072e63175f 100644
--- a/arch/x86/tools/relocs_common.c
+++ b/arch/x86/tools/relocs_common.c
@@ -12,14 +12,14 @@ void die(char *fmt, ...)
 
 static void usage(void)
 {
-	die("relocs [--abs-syms|--abs-relocs|--reloc-info|--text|--realmode]" \
-	    " vmlinux\n");
+	die("relocs [--abs-syms|--abs-relocs|--reloc-info|--text|--realmode|"
+	    "--fg-kaslr] vmlinux\n");
 }
 
 int main(int argc, char **argv)
 {
 	int show_absolute_syms, show_absolute_relocs, show_reloc_info;
-	int as_text, use_real_mode;
+	int as_text, use_real_mode, fgkaslr_opt;
 	const char *fname;
 	FILE *fp;
 	int i;
@@ -30,6 +30,7 @@ int main(int argc, char **argv)
 	show_reloc_info = 0;
 	as_text = 0;
 	use_real_mode = 0;
+	fgkaslr_opt = 0;
 	fname = NULL;
 	for (i = 1; i < argc; i++) {
 		char *arg = argv[i];
@@ -54,6 +55,10 @@ int main(int argc, char **argv)
 				use_real_mode = 1;
 				continue;
 			}
+			if (strcmp(arg, "--fg-kaslr") == 0) {
+				fgkaslr_opt = 1;
+				continue;
+			}
 		}
 		else if (!fname) {
 			fname = arg;
@@ -75,11 +80,11 @@ int main(int argc, char **argv)
 	if (e_ident[EI_CLASS] == ELFCLASS64)
 		process_64(fp, use_real_mode, as_text,
 			   show_absolute_syms, show_absolute_relocs,
-			   show_reloc_info);
+			   show_reloc_info, fgkaslr_opt);
 	else
 		process_32(fp, use_real_mode, as_text,
 			   show_absolute_syms, show_absolute_relocs,
-			   show_reloc_info);
+			   show_reloc_info, fgkaslr_opt);
 	fclose(fp);
 	return 0;
 }
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 10/22] x86/boot/compressed: Avoid duplicate malloc() implementations
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (8 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 09/22] x86/tools: Add relative relocs for randomized functions Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 11/22] x86: Add support for function granular KASLR Alexander Lobakin
                   ` (12 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

From: Kees Cook <keescook@chromium.org>

The preboot malloc() (and free()) implementation in
include/linux/decompress/mm.h (which is also included by the
static decompressors) is static. This is fine when the only thing
interested in using malloc() is the decompression code, but the
x86 preboot environment uses malloc() in a couple places, leading to a
potential collision when the static copies of the available memory
region ("malloc_ptr") gets reset to the global "free_mem_ptr" value.
As it happens, the existing usage pattern happened to be safe because each
user did 1 malloc() and 1 free() before returning and were not nested:

extract_kernel() (misc.c)
	choose_random_location() (kaslr.c)
		mem_avoid_init()
			handle_mem_options()
				malloc()
				...
				free()
	...
	parse_elf() (misc.c)
		malloc()
		...
		free()

Adding FGKASLR, however, will insert additional malloc() calls local to
fgkaslr.c in the middle of parse_elf()'s malloc()/free() pair:

	parse_elf() (misc.c)
		malloc()
		if (...) {
			layout_randomized_image(output, &ehdr, phdrs);
				malloc() <- boom
				...
		else
			layout_image(output, &ehdr, phdrs);
		free()

To avoid collisions, there must be a single implementation of malloc().
Adjust include/linux/decompress/mm.h so that visibility can be
controlled, provide prototypes in misc.h, and implement the functions in
misc.c. This also results in a small size savings:

$ size vmlinux.before vmlinux.after
   text    data     bss     dec     hex filename
8842314     468  178320 9021102  89a6ae vmlinux.before
8842240     468  178320 9021028  89a664 vmlinux.after

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 arch/x86/boot/compressed/kaslr.c |  4 ----
 arch/x86/boot/compressed/misc.c  |  3 +++
 arch/x86/boot/compressed/misc.h  |  2 ++
 include/linux/decompress/mm.h    | 12 ++++++++++--
 4 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c
index e36690778497..7d94f95c84dd 100644
--- a/arch/x86/boot/compressed/kaslr.c
+++ b/arch/x86/boot/compressed/kaslr.c
@@ -32,10 +32,6 @@
 #include <generated/utsrelease.h>
 #include <asm/efi.h>
 
-/* Macros used by the included decompressor code below. */
-#define STATIC
-#include <linux/decompress/mm.h>
-
 #define _SETUP
 #include <asm/setup.h>	/* For COMMAND_LINE_SIZE */
 #undef _SETUP
diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c
index 743f13ea25c1..a4339cb2d247 100644
--- a/arch/x86/boot/compressed/misc.c
+++ b/arch/x86/boot/compressed/misc.c
@@ -28,6 +28,9 @@
 
 /* Macros used by the included decompressor code below. */
 #define STATIC		static
+/* Define an externally visible malloc()/free(). */
+#define MALLOC_VISIBLE
+#include <linux/decompress/mm.h>
 
 /*
  * Provide definitions of memzero and memmove as some of the decompressors will
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 31139256859f..1a2e422dc357 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -44,6 +44,8 @@ extern char _head[], _end[];
 /* misc.c */
 extern memptr free_mem_ptr;
 extern memptr free_mem_end_ptr;
+extern void *malloc(int size);
+extern void free(void *where);
 extern struct boot_params *boot_params;
 void __putstr(const char *s);
 void __puthex(unsigned long value);
diff --git a/include/linux/decompress/mm.h b/include/linux/decompress/mm.h
index 868e9eacd69e..9192986b1a73 100644
--- a/include/linux/decompress/mm.h
+++ b/include/linux/decompress/mm.h
@@ -25,13 +25,21 @@
 #define STATIC_RW_DATA static
 #endif
 
+/*
+ * When an architecture needs to share the malloc()/free() implementation
+ * between compilation units, it needs to have non-local visibility.
+ */
+#ifndef MALLOC_VISIBLE
+#define MALLOC_VISIBLE static
+#endif
+
 /* A trivial malloc implementation, adapted from
  *  malloc by Hannu Savolainen 1993 and Matthias Urlichs 1994
  */
 STATIC_RW_DATA unsigned long malloc_ptr;
 STATIC_RW_DATA int malloc_count;
 
-static void *malloc(int size)
+MALLOC_VISIBLE void *malloc(int size)
 {
 	void *p;
 
@@ -52,7 +60,7 @@ static void *malloc(int size)
 	return p;
 }
 
-static void free(void *where)
+MALLOC_VISIBLE void free(void *where)
 {
 	malloc_count--;
 	if (!malloc_count)
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 11/22] x86: Add support for function granular KASLR
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (9 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 10/22] x86/boot/compressed: Avoid duplicate malloc() implementations Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 12/22] linkage: add macros for putting ASM functions into own sections Alexander Lobakin
                   ` (11 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux, kernel test robot

From: Kristen Carlson Accardi <kristen@linux.intel.com>

This commit contains the changes required to re-layout the kernel text
sections generated by -ffunction-sections shortly after decompression.
Documentation of the feature is also added.

After decompression, the decompressed image's elf headers are parsed.
In order to manually update certain data structures that are built with
relative offsets during the kernel build process, certain symbols are
not stripped by objdump and their location is retained in the elf symbol
tables. These addresses are saved.

If the image was built with -ffunction-sections, there will be ELF section
headers present which contain information about the address range of each
section. Anything that is not broken out into function sections (i.e. is
consolidated into .text) is left in it's original location, but any other
executable section which begins with ".text." is located and shuffled
randomly within the remaining text segment address range.

After the sections have been copied to their new locations, but before
relocations have been applied, the kallsyms tables must be updated to
reflect the new symbol locations. Because it is expected that these tables
will be sorted by address, the kallsyms tables will need to be sorted
after the update.

When applying relocations, the address of the relocation needs to be
adjusted by the offset from the original location of the section that was
randomized to it's new location. In addition, if a value at that relocation
was a location in the text segment that was randomized, it's value will be
adjusted to a new location.

After relocations have been applied, the exception table must be updated
with new symbol locations, and then re-sorted by the new address. The
orc table will have been updated as part of applying relocations, but since
it is expected to be sorted by address, it will need to be resorted.

Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Tested-by: Tony Luck <tony.luck@intel.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Reported-by: kernel test robot <lkp@intel.com> # #if -> #ifdef
[ alobakin: fix .altinstr_replacement relocations ]
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 arch/x86/boot/compressed/Makefile        |   2 +
 arch/x86/boot/compressed/fgkaslr.c       | 905 +++++++++++++++++++++++
 arch/x86/boot/compressed/misc.c          | 154 +++-
 arch/x86/boot/compressed/misc.h          |  28 +
 arch/x86/boot/compressed/utils.c         |  13 +
 arch/x86/boot/compressed/vmlinux.symbols |  19 +
 arch/x86/include/asm/boot.h              |  13 +-
 arch/x86/kernel/vmlinux.lds.S            |   2 +
 include/uapi/linux/elf.h                 |   1 +
 9 files changed, 1110 insertions(+), 27 deletions(-)
 create mode 100644 arch/x86/boot/compressed/fgkaslr.c
 create mode 100644 arch/x86/boot/compressed/utils.c
 create mode 100644 arch/x86/boot/compressed/vmlinux.symbols

diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index c31a24161fbf..e12fb0c8f4fc 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -89,6 +89,7 @@ vmlinux-objs-y := $(obj)/vmlinux.lds $(obj)/kernel_info.o $(obj)/head_$(BITS).o
 
 vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o
 vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o
+vmlinux-objs-$(CONFIG_FG_KASLR) += $(obj)/fgkaslr.o $(obj)/utils.o
 ifdef CONFIG_X86_64
 	vmlinux-objs-y += $(obj)/ident_map_64.o
 	vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
@@ -108,6 +109,7 @@ $(obj)/vmlinux: $(vmlinux-objs-y) $(efi-obj-y) FORCE
 OBJCOPYFLAGS_vmlinux.bin :=  -R .comment -S
 
 ifdef CONFIG_FG_KASLR
+OBJCOPYFLAGS += --keep-symbols=$(srctree)/$(src)/vmlinux.symbols
 RELOCS_ARGS += --fg-kaslr
 endif
 
diff --git a/arch/x86/boot/compressed/fgkaslr.c b/arch/x86/boot/compressed/fgkaslr.c
new file mode 100644
index 000000000000..0de99af5fc8d
--- /dev/null
+++ b/arch/x86/boot/compressed/fgkaslr.c
@@ -0,0 +1,905 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * This contains the routines needed to reorder the kernel text section
+ * at boot time.
+ *
+ * Copyright (C) 2020-2021, Intel Corporation.
+ * Author: Kristen Carlson Accardi <kristen@linux.intel.com>
+ */
+
+#include "misc.h"
+#include "error.h"
+#include "pgtable.h"
+#include "../string.h"
+#include "../voffset.h"
+#include <linux/sort.h>
+#include <linux/bsearch.h>
+#include "../../include/asm/extable.h"
+#include "../../include/asm/orc_types.h"
+
+/*
+ * Longest parameter of 'fgkaslr=' is 'off' right now, plus an extra '\0'
+ * for termination.
+ */
+#define MAX_FGKASLR_ARG_LENGTH 4
+static int nofgkaslr;
+
+/*
+ * Use normal definitions of mem*() from string.c. There are already
+ * included header files which expect a definition of memset() and by
+ * the time we define memset macro, it is too late.
+ */
+#undef memcpy
+#undef memset
+#define memzero(s, n)	memset((s), 0, (n))
+#define memmove		memmove
+
+void *memmove(void *dest, const void *src, size_t n);
+
+static unsigned long percpu_start;
+static unsigned long percpu_end;
+
+static long addr_kallsyms_names;
+static long addr_kallsyms_offsets;
+static long addr_kallsyms_num_syms;
+static long addr_kallsyms_relative_base;
+static long addr_kallsyms_markers;
+static long addr___start___ex_table;
+static long addr___stop___ex_table;
+static long addr___altinstr_replacement;
+static long addr___altinstr_replacement_end;
+static long addr__stext;
+static long addr__etext;
+static long addr__sinittext;
+static long addr__einittext;
+static long addr___start_orc_unwind_ip;
+static long addr___stop_orc_unwind_ip;
+static long addr___start_orc_unwind;
+
+/* addresses in mapped address space */
+static int *base;
+static u8 *names;
+static unsigned long relative_base;
+static unsigned int *markers_addr;
+
+struct kallsyms_name {
+	u8 len;
+	u8 indecis[256];
+};
+
+static struct kallsyms_name *names_table;
+
+static struct orc_entry *cur_orc_table;
+static int *cur_orc_ip_table;
+
+/* Array of pointers to sections headers for randomized sections */
+Elf_Shdr **sections;
+
+/* Number of elements in the randomized section header array (sections) */
+static int sections_size;
+
+/* Array of all section headers, randomized or otherwise */
+static Elf_Shdr *sechdrs;
+
+static bool is_orc_unwind(long addr)
+{
+	if (addr >= addr___start_orc_unwind_ip &&
+	    addr < addr___stop_orc_unwind_ip)
+		return true;
+	return false;
+}
+
+static bool is_text(long addr)
+{
+	if ((addr >= addr__stext && addr < addr__etext) ||
+	    (addr >= addr__sinittext && addr < addr__einittext) ||
+	    (addr >= addr___altinstr_replacement &&
+	     addr <= addr___altinstr_replacement_end))
+		return true;
+	return false;
+}
+
+bool is_percpu_addr(long pc, long offset)
+{
+	unsigned long ptr;
+	long address;
+
+	address = pc + offset + 4;
+
+	ptr = (unsigned long)address;
+
+	if (ptr >= percpu_start && ptr < percpu_end)
+		return true;
+
+	return false;
+}
+
+static int cmp_section_addr(const void *a, const void *b)
+{
+	unsigned long ptr = (unsigned long)a;
+	Elf_Shdr *s = *(Elf_Shdr **)b;
+	unsigned long end = s->sh_addr + s->sh_size;
+
+	if (ptr >= s->sh_addr && ptr < end)
+		return 0;
+
+	if (ptr < s->sh_addr)
+		return -1;
+
+	return 1;
+}
+
+static int cmp_section_addr_orc(const void *a, const void *b)
+{
+	unsigned long ptr = (unsigned long)a;
+	Elf_Shdr *s = *(Elf_Shdr **)b;
+	unsigned long end = s->sh_addr + s->sh_size;
+
+	/* orc relocations can be one past the end of the section */
+	if (ptr >= s->sh_addr && ptr <= end)
+		return 0;
+
+	if (ptr < s->sh_addr)
+		return -1;
+
+	return 1;
+}
+
+/*
+ * Discover if the orc_unwind address is in a randomized section and if so,
+ * adjust by the saved offset.
+ */
+Elf_Shdr *adjust_address_orc(long *address)
+{
+	Elf_Shdr **s;
+	Elf_Shdr *shdr;
+
+	if (nofgkaslr)
+		return NULL;
+
+	s = bsearch((const void *)*address, sections, sections_size, sizeof(*s),
+		    cmp_section_addr_orc);
+	if (s) {
+		shdr = *s;
+		*address += shdr->sh_offset;
+		return shdr;
+	}
+
+	return NULL;
+}
+
+/*
+ * Discover if the address is in a randomized section and if so, adjust
+ * by the saved offset.
+ */
+Elf_Shdr *adjust_address(long *address)
+{
+	Elf_Shdr **s;
+	Elf_Shdr *shdr;
+
+	if (nofgkaslr)
+		return NULL;
+
+	s = bsearch((const void *)*address, sections, sections_size, sizeof(*s),
+		    cmp_section_addr);
+	if (s) {
+		shdr = *s;
+		*address += shdr->sh_offset;
+		return shdr;
+	}
+
+	return NULL;
+}
+
+void adjust_relative_offset(long pc, long *value, Elf_Shdr *section)
+{
+	Elf_Shdr *s;
+	long address;
+
+	if (nofgkaslr)
+		return;
+
+	/*
+	 * sometimes we are updating a relative offset that would
+	 * normally be relative to the next instruction (such as a call).
+	 * In this case to calculate the target, you need to add 32bits to
+	 * the pc to get the next instruction value. However, sometimes
+	 * targets are just data that was stored in a table such as ksymtab
+	 * or cpu alternatives. In this case our target is not relative to
+	 * the next instruction.
+	 */
+
+	/* Calculate the address that this offset would call. */
+	if (!is_text(pc))
+		address = pc + *value;
+	else
+		address = pc + *value + 4;
+
+	/*
+	 * orc ip addresses are sorted at build time after relocs have
+	 * been applied, making the relocs no longer valid. Skip any
+	 * relocs for the orc_unwind_ip table. These will be updated
+	 * separately.
+	 */
+	if (is_orc_unwind(pc))
+		return;
+
+	s = adjust_address(&address);
+
+	/*
+	 * if the address is in section that was randomized,
+	 * we need to adjust the offset.
+	 */
+	if (s)
+		*value += s->sh_offset;
+
+	/*
+	 * If the PC that this offset was calculated for was in a section
+	 * that has been randomized, the value needs to be adjusted by the
+	 * same amount as the randomized section was adjusted from it's original
+	 * location.
+	 */
+	if (section)
+		*value -= section->sh_offset;
+}
+
+static void kallsyms_swp(void *a, void *b, int size)
+{
+	int idx1, idx2;
+	int temp;
+	struct kallsyms_name name_a;
+
+	/* Determine our index into the array. */
+	idx1 = (int *)a - base;
+	idx2 = (int *)b - base;
+	temp = base[idx1];
+	base[idx1] = base[idx2];
+	base[idx2] = temp;
+
+	/* Swap the names table. */
+	memcpy(&name_a, &names_table[idx1], sizeof(name_a));
+	memcpy(&names_table[idx1], &names_table[idx2],
+	       sizeof(struct kallsyms_name));
+	memcpy(&names_table[idx2], &name_a, sizeof(struct kallsyms_name));
+}
+
+static int kallsyms_cmp(const void *a, const void *b)
+{
+	int addr_a, addr_b;
+	unsigned long uaddr_a, uaddr_b;
+
+	addr_a = *(int *)a;
+	addr_b = *(int *)b;
+
+	if (addr_a >= 0)
+		uaddr_a = addr_a;
+	if (addr_b >= 0)
+		uaddr_b = addr_b;
+
+	if (addr_a < 0)
+		uaddr_a = relative_base - 1 - addr_a;
+	if (addr_b < 0)
+		uaddr_b = relative_base - 1 - addr_b;
+
+	if (uaddr_b > uaddr_a)
+		return -1;
+
+	return 0;
+}
+
+static void deal_with_names(int num_syms)
+{
+	int num_bytes;
+	int i, j;
+	int offset;
+
+	/* we should have num_syms kallsyms_name entries */
+	num_bytes = num_syms * sizeof(*names_table);
+	names_table = malloc(num_syms * sizeof(*names_table));
+	if (!names_table) {
+		debug_putstr("\nbytes requested: ");
+		debug_puthex(num_bytes);
+		error("\nunable to allocate space for names table\n");
+	}
+
+	/* read all the names entries */
+	offset = 0;
+	for (i = 0; i < num_syms; i++) {
+		names_table[i].len = names[offset];
+		offset++;
+		for (j = 0; j < names_table[i].len; j++) {
+			names_table[i].indecis[j] = names[offset];
+			offset++;
+		}
+	}
+}
+
+static void write_sorted_names(int num_syms)
+{
+	int i, j;
+	int offset = 0;
+	unsigned int *markers;
+
+	/*
+	 * we are going to need to regenerate the markers table, which is a
+	 * table of offsets into the compressed stream every 256 symbols.
+	 * this code copied almost directly from scripts/kallsyms.c
+	 */
+	markers = malloc(sizeof(unsigned int) * ((num_syms + 255) / 256));
+	if (!markers) {
+		debug_putstr("\nfailed to allocate heap space of ");
+		debug_puthex(((num_syms + 255) / 256));
+		debug_putstr(" bytes\n");
+		error("Unable to allocate space for markers table");
+	}
+
+	for (i = 0; i < num_syms; i++) {
+		if ((i & 0xFF) == 0)
+			markers[i >> 8] = offset;
+
+		names[offset] = (u8)names_table[i].len;
+		offset++;
+		for (j = 0; j < names_table[i].len; j++) {
+			names[offset] = (u8)names_table[i].indecis[j];
+			offset++;
+		}
+	}
+
+	/* write new markers table over old one */
+	for (i = 0; i < ((num_syms + 255) >> 8); i++)
+		markers_addr[i] = markers[i];
+
+	free(markers);
+	free(names_table);
+}
+
+static void sort_kallsyms(unsigned long map)
+{
+	int num_syms;
+	int i;
+
+	debug_putstr("\nRe-sorting kallsyms...\n");
+
+	num_syms = *(int *)(addr_kallsyms_num_syms + map);
+	base = (int *)(addr_kallsyms_offsets + map);
+	relative_base = *(unsigned long *)(addr_kallsyms_relative_base + map);
+	markers_addr = (unsigned int *)(addr_kallsyms_markers + map);
+	names = (u8 *)(addr_kallsyms_names + map);
+
+	/*
+	 * the kallsyms table was generated prior to any randomization.
+	 * it is a bunch of offsets from "relative base". In order for
+	 * us to check if a symbol has an address that was in a randomized
+	 * section, we need to reconstruct the address to it's original
+	 * value prior to handle_relocations.
+	 */
+	for (i = 0; i < num_syms; i++) {
+		unsigned long addr;
+		int new_base;
+
+		/*
+		 * according to kernel/kallsyms.c, positive offsets are absolute
+		 * values and negative offsets are relative to the base.
+		 */
+		if (base[i] >= 0)
+			addr = base[i];
+		else
+			addr = relative_base - 1 - base[i];
+
+		if (adjust_address(&addr)) {
+			/* here we need to recalcuate the offset */
+			new_base = relative_base - 1 - addr;
+			base[i] = new_base;
+		}
+	}
+
+	/*
+	 * here we need to read in all the kallsyms_names info
+	 * so that we can regenerate it.
+	 */
+	deal_with_names(num_syms);
+
+	sort(base, num_syms, sizeof(int), kallsyms_cmp, kallsyms_swp);
+
+	/* write the newly sorted names table over the old one */
+	write_sorted_names(num_syms);
+}
+
+/*
+ * We need to include this file here rather than in utils.c because
+ * some of the helper functions in extable.c are used to update
+ * the extable below and are defined as "static" in extable.c
+ */
+#include "../../../../lib/extable.c"
+
+static inline unsigned long
+ex_fixup_handler(const struct exception_table_entry *x)
+{
+	return ((unsigned long)&x->handler + x->handler);
+}
+
+static inline unsigned long
+ex_fixup_addr(const struct exception_table_entry *x)
+{
+	return ((unsigned long)&x->fixup + x->fixup);
+}
+
+static void update_ex_table(unsigned long map)
+{
+	struct exception_table_entry *start_ex_table =
+		(struct exception_table_entry *)(addr___start___ex_table + map);
+	struct exception_table_entry *stop_ex_table =
+		(struct exception_table_entry *)(addr___stop___ex_table + map);
+	int num_entries =
+		(addr___stop___ex_table - addr___start___ex_table) /
+		sizeof(struct exception_table_entry);
+	int i;
+
+	debug_putstr("\nUpdating exception table...");
+	for (i = 0; i < num_entries; i++) {
+		unsigned long insn = ex_to_insn(&start_ex_table[i]);
+		unsigned long fixup = ex_fixup_addr(&start_ex_table[i]);
+		unsigned long handler = ex_fixup_handler(&start_ex_table[i]);
+		unsigned long addr;
+		Elf_Shdr *s;
+
+		/* check each address to see if it needs adjusting */
+		addr = insn - map;
+		s = adjust_address(&addr);
+		if (s)
+			start_ex_table[i].insn += s->sh_offset;
+
+		addr = fixup - map;
+		s = adjust_address(&addr);
+		if (s)
+			start_ex_table[i].fixup += s->sh_offset;
+
+		addr = handler - map;
+		s = adjust_address(&addr);
+		if (s)
+			start_ex_table[i].handler += s->sh_offset;
+	}
+}
+
+static void sort_ex_table(unsigned long map)
+{
+	struct exception_table_entry *start_ex_table =
+		(struct exception_table_entry *)(addr___start___ex_table + map);
+	struct exception_table_entry *stop_ex_table =
+		(struct exception_table_entry *)(addr___stop___ex_table + map);
+
+	debug_putstr("\nRe-sorting exception table...");
+
+	sort_extable(start_ex_table, stop_ex_table);
+}
+
+static inline unsigned long orc_ip(const int *ip)
+{
+	return (unsigned long)ip + *ip;
+}
+
+static void orc_sort_swap(void *_a, void *_b, int size)
+{
+	struct orc_entry *orc_a, *orc_b;
+	struct orc_entry orc_tmp;
+	int *a = _a, *b = _b, tmp;
+	int delta = _b - _a;
+
+	/* Swap the .orc_unwind_ip entries: */
+	tmp = *a;
+	*a = *b + delta;
+	*b = tmp - delta;
+
+	/* Swap the corresponding .orc_unwind entries: */
+	orc_a = cur_orc_table + (a - cur_orc_ip_table);
+	orc_b = cur_orc_table + (b - cur_orc_ip_table);
+	orc_tmp = *orc_a;
+	*orc_a = *orc_b;
+	*orc_b = orc_tmp;
+}
+
+static int orc_sort_cmp(const void *_a, const void *_b)
+{
+	struct orc_entry *orc_a;
+	const int *a = _a, *b = _b;
+	unsigned long a_val = orc_ip(a);
+	unsigned long b_val = orc_ip(b);
+
+	if (a_val > b_val)
+		return 1;
+	if (a_val < b_val)
+		return -1;
+
+	/*
+	 * The "weak" section terminator entries need to always be on the left
+	 * to ensure the lookup code skips them in favor of real entries.
+	 * These terminator entries exist to handle any gaps created by
+	 * whitelisted .o files which didn't get objtool generation.
+	 */
+	orc_a = cur_orc_table + (a - cur_orc_ip_table);
+	return orc_a->sp_reg == ORC_REG_UNDEFINED && !orc_a->end ? -1 : 1;
+}
+
+static void update_orc_table(unsigned long map)
+{
+	int i;
+	int num_entries =
+		(addr___stop_orc_unwind_ip - addr___start_orc_unwind_ip) / sizeof(int);
+
+	cur_orc_ip_table = (int *)(addr___start_orc_unwind_ip + map);
+	cur_orc_table = (struct orc_entry *)(addr___start_orc_unwind + map);
+
+	debug_putstr("\nUpdating orc tables...\n");
+	for (i = 0; i < num_entries; i++) {
+		unsigned long ip = orc_ip(&cur_orc_ip_table[i]);
+		Elf_Shdr *s;
+
+		/* check each address to see if it needs adjusting */
+		ip = ip - map;
+
+		/*
+		 * objtool places terminator entries just outside the end of
+		 * the section. To identify an orc_unwind_ip address that might
+		 * need adjusting, the address should be compared differently
+		 * than a normal address.
+		 */
+		s = adjust_address_orc(&ip);
+		if (s)
+			cur_orc_ip_table[i] += s->sh_offset;
+	}
+}
+
+static void sort_orc_table(unsigned long map)
+{
+	int num_entries =
+		(addr___stop_orc_unwind_ip - addr___start_orc_unwind_ip) / sizeof(int);
+
+	cur_orc_ip_table = (int *)(addr___start_orc_unwind_ip + map);
+	cur_orc_table = (struct orc_entry *)(addr___start_orc_unwind + map);
+
+	debug_putstr("\nRe-sorting orc tables...\n");
+	sort(cur_orc_ip_table, num_entries, sizeof(int), orc_sort_cmp,
+	     orc_sort_swap);
+}
+
+void post_relocations_cleanup(unsigned long map)
+{
+	if (!nofgkaslr) {
+		update_ex_table(map);
+		sort_ex_table(map);
+		update_orc_table(map);
+		sort_orc_table(map);
+	}
+
+	/*
+	 * maybe one day free will do something. So, we "free" this memory
+	 * in either case
+	 */
+	free(sections);
+	free(sechdrs);
+}
+
+void pre_relocations_cleanup(unsigned long map)
+{
+	if (nofgkaslr)
+		return;
+
+	sort_kallsyms(map);
+}
+
+static void shuffle_sections(int *list, int size)
+{
+	int i;
+	unsigned long j;
+	int temp;
+
+	for (i = size - 1; i > 0; i--) {
+		j = kaslr_get_random_long(NULL) % (i + 1);
+
+		temp = list[i];
+		list[i] = list[j];
+		list[j] = temp;
+	}
+}
+
+static void move_text(int num_sections, char *secstrings, Elf_Shdr *text,
+		      void *source, void *dest, Elf64_Phdr *phdr)
+{
+	unsigned long adjusted_addr;
+	int copy_bytes;
+	void *stash;
+	Elf_Shdr **sorted_sections;
+	int *index_list;
+	int i, j;
+
+	memmove(dest, source + text->sh_offset, text->sh_size);
+	copy_bytes = text->sh_size;
+	dest += text->sh_size;
+	adjusted_addr = text->sh_addr + text->sh_size;
+
+	/*
+	 * we leave the sections sorted in their original order
+	 * by s->sh_addr, but shuffle the indexes in a random
+	 * order for copying.
+	 */
+	index_list = malloc(sizeof(int) * num_sections);
+	if (!index_list)
+		error("Failed to allocate space for index list");
+
+	for (i = 0; i < num_sections; i++)
+		index_list[i] = i;
+
+	shuffle_sections(index_list, num_sections);
+
+	/*
+	 * to avoid overwriting earlier sections before they can get
+	 * copied to dest, stash everything into a buffer first.
+	 * this will cause our source address to be off by
+	 * phdr->p_offset though, so we'll adjust s->sh_offset below.
+	 *
+	 * TBD: ideally we'd simply decompress higher up so that our
+	 * copy wasn't in danger of overwriting anything important.
+	 */
+	stash = malloc(phdr->p_filesz);
+	if (!stash)
+		error("Failed to allocate space for text stash");
+
+	memcpy(stash, source + phdr->p_offset, phdr->p_filesz);
+
+	/* now we'd walk through the sections. */
+	for (j = 0; j < num_sections; j++) {
+		unsigned long aligned_addr;
+		Elf_Shdr *s;
+		const char *sname;
+		void *src;
+		int pad_bytes;
+
+		s = sections[index_list[j]];
+
+		sname = secstrings + s->sh_name;
+
+		/* align addr for this section */
+		aligned_addr = ALIGN(adjusted_addr, s->sh_addralign);
+
+		/*
+		 * copy out of stash, so adjust offset
+		 */
+		src = stash + s->sh_offset - phdr->p_offset;
+
+		/*
+		 * Fill any space between sections with int3
+		 */
+		pad_bytes = aligned_addr - adjusted_addr;
+		memset(dest, 0xcc, pad_bytes);
+
+		dest = (void *)ALIGN((unsigned long)dest, s->sh_addralign);
+
+		memmove(dest, src, s->sh_size);
+
+		dest += s->sh_size;
+		copy_bytes += s->sh_size + pad_bytes;
+		adjusted_addr = aligned_addr + s->sh_size;
+
+		/* we can blow away sh_offset for our own uses */
+		s->sh_offset = aligned_addr - s->sh_addr;
+	}
+
+	free(index_list);
+
+	/*
+	 * move remainder of text segment. Ok to just use original source
+	 * here since this area is untouched.
+	 */
+	memmove(dest, source + text->sh_offset + copy_bytes,
+		phdr->p_filesz - copy_bytes);
+	free(stash);
+}
+
+#define GET_SYM(name)							\
+	do {								\
+		if (!addr_ ## name) {					\
+			if (strcmp(#name, strtab + sym->st_name) == 0) {\
+				addr_ ## name = sym->st_value;		\
+				continue;				\
+			}						\
+		}							\
+	} while (0)
+
+static void parse_symtab(Elf64_Sym *symtab, char *strtab, long num_syms)
+{
+	Elf64_Sym *sym;
+
+	if (!symtab || !strtab)
+		return;
+
+	debug_putstr("\nLooking for symbols... ");
+
+	/*
+	 * walk through the symbol table looking for the symbols
+	 * that we care about.
+	 */
+	for (sym = symtab; --num_syms >= 0; sym++) {
+		if (!sym->st_name)
+			continue;
+
+		GET_SYM(kallsyms_num_syms);
+		GET_SYM(kallsyms_offsets);
+		GET_SYM(kallsyms_relative_base);
+		GET_SYM(kallsyms_names);
+		GET_SYM(kallsyms_markers);
+		GET_SYM(__altinstr_replacement);
+		GET_SYM(__altinstr_replacement_end);
+		GET_SYM(_stext);
+		GET_SYM(_etext);
+		GET_SYM(_sinittext);
+		GET_SYM(_einittext);
+		GET_SYM(__start_orc_unwind_ip);
+		GET_SYM(__stop_orc_unwind_ip);
+		GET_SYM(__start_orc_unwind);
+		GET_SYM(__start___ex_table);
+		GET_SYM(__stop___ex_table);
+	}
+}
+
+void layout_randomized_image(void *output, Elf64_Ehdr *ehdr, Elf64_Phdr *phdrs)
+{
+	Elf64_Phdr *phdr;
+	Elf_Shdr *s;
+	Elf_Shdr *text = NULL;
+	Elf_Shdr *percpu = NULL;
+	char *secstrings;
+	const char *sname;
+	int num_sections = 0;
+	Elf64_Sym *symtab = NULL;
+	char *strtab = NULL;
+	long num_syms = 0;
+	void *dest;
+	int i;
+	char arg[MAX_FGKASLR_ARG_LENGTH];
+	Elf_Shdr shdr;
+	unsigned long shnum;
+	unsigned int shstrndx;
+
+	debug_putstr("\nParsing ELF section headers... ");
+
+	/*
+	 * Even though fgkaslr may have been disabled, we still
+	 * need to parse through the section headers to get the
+	 * start and end of the percpu section. This is because
+	 * if we were built with CONFIG_FG_KASLR, there are more
+	 * relative relocations present in vmlinux.relocs than
+	 * just the percpu, and only the percpu relocs need to be
+	 * adjusted when using just normal base address kaslr.
+	 */
+	if (cmdline_find_option_bool("nofgkaslr")) {
+		warn("FG_KASLR disabled on cmdline.");
+		nofgkaslr = 1;
+	}
+
+	/* read the first section header */
+	shnum = ehdr->e_shnum;
+	shstrndx = ehdr->e_shstrndx;
+	if (shnum == SHN_UNDEF || shstrndx == SHN_XINDEX) {
+		memcpy(&shdr, output + ehdr->e_shoff, sizeof(shdr));
+		if (shnum == SHN_UNDEF)
+			shnum = shdr.sh_size;
+		if (shstrndx == SHN_XINDEX)
+			shstrndx = shdr.sh_link;
+	}
+
+	/* we are going to need to allocate space for the section headers */
+	sechdrs = malloc(sizeof(*sechdrs) * shnum);
+	if (!sechdrs)
+		error("Failed to allocate space for shdrs");
+
+	sections = malloc(sizeof(*sections) * shnum);
+	if (!sections)
+		error("Failed to allocate space for section pointers");
+
+	memcpy(sechdrs, output + ehdr->e_shoff,
+	       sizeof(*sechdrs) * shnum);
+
+	/* we need to allocate space for the section string table */
+	s = &sechdrs[shstrndx];
+
+	secstrings = malloc(s->sh_size);
+	if (!secstrings)
+		error("Failed to allocate space for shstr");
+
+	memcpy(secstrings, output + s->sh_offset, s->sh_size);
+
+	/*
+	 * now we need to walk through the section headers and collect the
+	 * sizes of the .text sections to be randomized.
+	 */
+	for (i = 0; i < shnum; i++) {
+		s = &sechdrs[i];
+		sname = secstrings + s->sh_name;
+
+		if (s->sh_type == SHT_SYMTAB) {
+			/* only one symtab per image */
+			if (symtab)
+				error("Unexpected duplicate symtab");
+
+			symtab = malloc(s->sh_size);
+			if (!symtab)
+				error("Failed to allocate space for symtab");
+
+			memcpy(symtab, output + s->sh_offset, s->sh_size);
+			num_syms = s->sh_size / sizeof(*symtab);
+			continue;
+		}
+
+		if (s->sh_type == SHT_STRTAB && i != ehdr->e_shstrndx) {
+			if (strtab)
+				error("Unexpected duplicate strtab");
+
+			strtab = malloc(s->sh_size);
+			if (!strtab)
+				error("Failed to allocate space for strtab");
+
+			memcpy(strtab, output + s->sh_offset, s->sh_size);
+		}
+
+		if (!strcmp(sname, ".text")) {
+			if (text)
+				error("Unexpected duplicate .text section");
+
+			text = s;
+			continue;
+		}
+
+		if (!strcmp(sname, ".data..percpu")) {
+			/* get start addr for later */
+			percpu = s;
+			continue;
+		}
+
+		if (!(s->sh_flags & SHF_ALLOC) ||
+		    !(s->sh_flags & SHF_EXECINSTR) ||
+		    !(strstarts(sname, ".text")))
+			continue;
+
+		sections[num_sections] = s;
+
+		num_sections++;
+	}
+	sections[num_sections] = NULL;
+	sections_size = num_sections;
+
+	parse_symtab(symtab, strtab, num_syms);
+
+	for (i = 0; i < ehdr->e_phnum; i++) {
+		phdr = &phdrs[i];
+
+		switch (phdr->p_type) {
+		case PT_LOAD:
+			if ((phdr->p_align % 0x200000) != 0)
+				error("Alignment of LOAD segment isn't multiple of 2MB");
+			dest = output;
+			dest += (phdr->p_paddr - LOAD_PHYSICAL_ADDR);
+			if (!nofgkaslr &&
+			    (text && phdr->p_offset == text->sh_offset)) {
+				move_text(num_sections, secstrings, text,
+					  output, dest, phdr);
+			} else {
+				if (percpu &&
+				    phdr->p_offset == percpu->sh_offset) {
+					percpu_start = percpu->sh_addr;
+					percpu_end = percpu_start +
+							phdr->p_filesz;
+				}
+				memmove(dest, output + phdr->p_offset,
+					phdr->p_filesz);
+			}
+			break;
+		default: /* Ignore other PT_* */
+			break;
+		}
+	}
+
+	/* we need to keep the section info to redo relocs */
+	free(secstrings);
+
+	free(phdrs);
+}
diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c
index a4339cb2d247..34b2b3174727 100644
--- a/arch/x86/boot/compressed/misc.c
+++ b/arch/x86/boot/compressed/misc.c
@@ -207,10 +207,21 @@ static void handle_relocations(void *output, unsigned long output_len,
 	if (IS_ENABLED(CONFIG_X86_64))
 		delta = virt_addr - LOAD_PHYSICAL_ADDR;
 
-	if (!delta) {
-		debug_putstr("No relocation needed... ");
-		return;
+	/*
+	 * it is possible to have delta be zero and still have enabled
+	 * fg kaslr. We need to perform relocations for fgkaslr regardless
+	 * of whether the base address has moved.
+	 */
+	if (!IS_ENABLED(CONFIG_FG_KASLR) ||
+	    cmdline_find_option_bool("nokaslr")) {
+		if (!delta) {
+			debug_putstr("No relocation needed... ");
+			return;
+		}
 	}
+
+	pre_relocations_cleanup(map);
+
 	debug_putstr("Performing relocations... ");
 
 	/*
@@ -234,35 +245,106 @@ static void handle_relocations(void *output, unsigned long output_len,
 	 */
 	for (reloc = output + output_len - sizeof(*reloc); *reloc; reloc--) {
 		long extended = *reloc;
+		long value;
+
+		/*
+		 * if using fgkaslr, we might have moved the address
+		 * of the relocation. Check it to see if it needs adjusting
+		 * from the original address.
+		 */
+		adjust_address(&extended);
+
 		extended += map;
 
 		ptr = (unsigned long)extended;
 		if (ptr < min_addr || ptr > max_addr)
 			error("32-bit relocation outside of kernel!\n");
 
-		*(uint32_t *)ptr += delta;
+		value = *(int32_t *)ptr;
+
+		/*
+		 * If using fgkaslr, the value of the relocation
+		 * might need to be changed because it referred
+		 * to an address that has moved.
+		 */
+		adjust_address(&value);
+
+		value += delta;
+
+		*(uint32_t *)ptr = value;
 	}
 #ifdef CONFIG_X86_64
 	while (*--reloc) {
 		long extended = *reloc;
+		long value;
+		long oldvalue;
+		Elf64_Shdr *s;
+
+		/*
+		 * if using fgkaslr, we might have moved the address
+		 * of the relocation. Check it to see if it needs adjusting
+		 * from the original address.
+		 */
+		s = adjust_address(&extended);
+
 		extended += map;
 
 		ptr = (unsigned long)extended;
 		if (ptr < min_addr || ptr > max_addr)
 			error("inverse 32-bit relocation outside of kernel!\n");
 
-		*(int32_t *)ptr -= delta;
+		value = *(int32_t *)ptr;
+		oldvalue = value;
+
+		/*
+		 * If using fgkaslr, these relocs will contain
+		 * relative offsets which might need to be
+		 * changed because it referred
+		 * to an address that has moved.
+		 */
+		adjust_relative_offset(*reloc, &value, s);
+
+		/*
+		 * only percpu symbols need to have their values adjusted for
+		 * base address kaslr since relative offsets within the .text
+		 * and .text.* sections are ok wrt each other.
+		 */
+		if (is_percpu_addr(*reloc, oldvalue))
+			value -= delta;
+
+		*(int32_t *)ptr = value;
 	}
 	for (reloc--; *reloc; reloc--) {
 		long extended = *reloc;
+		long value;
+
+		/*
+		 * if using fgkaslr, we might have moved the address
+		 * of the relocation. Check it to see if it needs adjusting
+		 * from the original address.
+		 */
+		adjust_address(&extended);
+
 		extended += map;
 
 		ptr = (unsigned long)extended;
 		if (ptr < min_addr || ptr > max_addr)
 			error("64-bit relocation outside of kernel!\n");
 
-		*(uint64_t *)ptr += delta;
+		value = *(int64_t *)ptr;
+
+		/*
+		 * If using fgkaslr, the value of the relocation
+		 * might need to be changed because it referred
+		 * to an address that has moved.
+		 */
+		adjust_address(&value);
+
+		value += delta;
+
+		*(uint64_t *)ptr = value;
 	}
+	post_relocations_cleanup(map);
 #endif
 }
 #else
@@ -271,6 +353,35 @@ static inline void handle_relocations(void *output, unsigned long output_len,
 { }
 #endif
 
+static void layout_image(void *output, Elf_Ehdr *ehdr, Elf_Phdr *phdrs)
+{
+	int i;
+	void *dest;
+	Elf_Phdr *phdr;
+
+	for (i = 0; i < ehdr->e_phnum; i++) {
+		phdr = &phdrs[i];
+
+		switch (phdr->p_type) {
+		case PT_LOAD:
+#ifdef CONFIG_X86_64
+			if ((phdr->p_align % 0x200000) != 0)
+				error("Alignment of LOAD segment isn't multiple of 2MB");
+#endif
+#ifdef CONFIG_RELOCATABLE
+			dest = output;
+			dest += (phdr->p_paddr - LOAD_PHYSICAL_ADDR);
+#else
+			dest = (void *)(phdr->p_paddr);
+#endif
+			memmove(dest, output + phdr->p_offset, phdr->p_filesz);
+			break;
+		default: /* Ignore other PT_* */
+			break;
+		}
+	}
+}
+
 static void parse_elf(void *output)
 {
 #ifdef CONFIG_X86_64
@@ -282,6 +393,7 @@ static void parse_elf(void *output)
 #endif
 	void *dest;
 	int i;
+	int nokaslr;
 
 	memcpy(&ehdr, output, sizeof(ehdr));
 	if (ehdr.e_ident[EI_MAG0] != ELFMAG0 ||
@@ -292,6 +404,12 @@ static void parse_elf(void *output)
 		return;
 	}
 
+	if (IS_ENABLED(CONFIG_FG_KASLR)) {
+		nokaslr = cmdline_find_option_bool("nokaslr");
+		if (nokaslr)
+			warn("FG_KASLR disabled: 'nokaslr' on cmdline.");
+	}
+
 	debug_putstr("Parsing ELF... ");
 
 	phdrs = malloc(sizeof(*phdrs) * ehdr.e_phnum);
@@ -300,26 +418,10 @@ static void parse_elf(void *output)
 
 	memcpy(phdrs, output + ehdr.e_phoff, sizeof(*phdrs) * ehdr.e_phnum);
 
-	for (i = 0; i < ehdr.e_phnum; i++) {
-		phdr = &phdrs[i];
-
-		switch (phdr->p_type) {
-		case PT_LOAD:
-#ifdef CONFIG_X86_64
-			if ((phdr->p_align % 0x200000) != 0)
-				error("Alignment of LOAD segment isn't multiple of 2MB");
-#endif
-#ifdef CONFIG_RELOCATABLE
-			dest = output;
-			dest += (phdr->p_paddr - LOAD_PHYSICAL_ADDR);
-#else
-			dest = (void *)(phdr->p_paddr);
-#endif
-			memmove(dest, output + phdr->p_offset, phdr->p_filesz);
-			break;
-		default: /* Ignore other PT_* */ break;
-		}
-	}
+	if (IS_ENABLED(CONFIG_FG_KASLR) && !nokaslr)
+		layout_randomized_image(output, &ehdr, phdrs);
+	else
+		layout_image(output, &ehdr, phdrs);
 
 	free(phdrs);
 }
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 1a2e422dc357..5368894a0dce 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -81,6 +81,34 @@ struct mem_vector {
 	u64 size;
 };
 
+#ifdef CONFIG_X86_64
+#define Elf_Ehdr Elf64_Ehdr
+#define Elf_Phdr Elf64_Phdr
+#define Elf_Shdr Elf64_Shdr
+#else
+#define Elf_Ehdr Elf32_Ehdr
+#define Elf_Phdr Elf32_Phdr
+#define Elf_Shdr Elf32_Shdr
+#endif
+
+#ifdef CONFIG_FG_KASLR
+void layout_randomized_image(void *output, Elf_Ehdr *ehdr, Elf_Phdr *phdrs);
+void pre_relocations_cleanup(unsigned long map);
+void post_relocations_cleanup(unsigned long map);
+Elf_Shdr *adjust_address(long *address);
+void adjust_relative_offset(long pc, long *value, Elf_Shdr *section);
+bool is_percpu_addr(long pc, long offset);
+#else
+static inline void layout_randomized_image(void *output, Elf_Ehdr *ehdr,
+					   Elf_Phdr *phdrs) { }
+static inline void pre_relocations_cleanup(unsigned long map) { }
+static inline void post_relocations_cleanup(unsigned long map) { }
+static inline Elf_Shdr *adjust_address(long *address) { return NULL; }
+static inline void adjust_relative_offset(long pc, long *value,
+					  Elf_Shdr *section) { }
+static inline bool is_percpu_addr(long pc, long offset) { return true; }
+#endif
+
 #ifdef CONFIG_RANDOMIZE_BASE
 /* kaslr.c */
 void choose_random_location(unsigned long input,
diff --git a/arch/x86/boot/compressed/utils.c b/arch/x86/boot/compressed/utils.c
new file mode 100644
index 000000000000..7c3c745f6251
--- /dev/null
+++ b/arch/x86/boot/compressed/utils.c
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * This contains various libraries that are needed for FG-KASLR.
+ *
+ * Copyright (C) 2020-2021, Intel Corporation.
+ * Author: Kristen Carlson Accardi <kristen@linux.intel.com>
+ */
+
+#define _LINUX_KPROBES_H
+#define NOKPROBE_SYMBOL(fname)
+
+#include "../../../../lib/sort.c"
+#include "../../../../lib/bsearch.c"
diff --git a/arch/x86/boot/compressed/vmlinux.symbols b/arch/x86/boot/compressed/vmlinux.symbols
new file mode 100644
index 000000000000..da41f3ee153c
--- /dev/null
+++ b/arch/x86/boot/compressed/vmlinux.symbols
@@ -0,0 +1,19 @@
+kallsyms_offsets
+kallsyms_addresses
+kallsyms_num_syms
+kallsyms_relative_base
+kallsyms_names
+kallsyms_token_table
+kallsyms_token_index
+kallsyms_markers
+__start___ex_table
+__stop___ex_table
+__altinstr_replacement
+__altinstr_replacement_end
+_sinittext
+_einittext
+_stext
+_etext
+__start_orc_unwind_ip
+__stop_orc_unwind_ip
+__start_orc_unwind
diff --git a/arch/x86/include/asm/boot.h b/arch/x86/include/asm/boot.h
index 9191280d9ea3..ce5fdee49046 100644
--- a/arch/x86/include/asm/boot.h
+++ b/arch/x86/include/asm/boot.h
@@ -24,7 +24,18 @@
 # error "Invalid value for CONFIG_PHYSICAL_ALIGN"
 #endif
 
-#if defined(CONFIG_KERNEL_BZIP2)
+#ifdef CONFIG_FG_KASLR
+/*
+ * We need extra boot heap when using fgkaslr because we make a copy
+ * of the original decompressed kernel to avoid issues with writing
+ * over ourselves when shuffling the sections. We also need extra
+ * space for resorting kallsyms after shuffling. This value could
+ * be decreased if free() would release memory properly, or if we
+ * could avoid the kernel copy. It would need to be increased if we
+ * find additional tables that need to be resorted.
+ */
+# define BOOT_HEAP_SIZE		0x4800000
+#elif defined(CONFIG_KERNEL_BZIP2)
 # define BOOT_HEAP_SIZE		0x400000
 #elif defined(CONFIG_KERNEL_ZSTD)
 /*
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index efd9e9ea17f2..9692e990145b 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -290,7 +290,9 @@ SECTIONS
 	 * get the address and the length of them to patch the kernel safely.
 	 */
 	.altinstr_replacement : AT(ADDR(.altinstr_replacement) - LOAD_OFFSET) {
+		__altinstr_replacement = .;
 		*(.altinstr_replacement)
+		__altinstr_replacement_end = .;
 	}
 
 	/*
diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h
index 61bf4774b8f2..1c74d9594919 100644
--- a/include/uapi/linux/elf.h
+++ b/include/uapi/linux/elf.h
@@ -299,6 +299,7 @@ typedef struct elf64_phdr {
 #define SHN_LIVEPATCH	0xff20
 #define SHN_ABS		0xfff1
 #define SHN_COMMON	0xfff2
+#define SHN_XINDEX	0xffff
 #define SHN_HIRESERVE	0xffff
  
 typedef struct elf32_shdr {
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 12/22] linkage: add macros for putting ASM functions into own sections
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (10 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 11/22] x86: Add support for function granular KASLR Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 14/22] FG-KASLR: use a scripted approach to handle .text.* sections Alexander Lobakin
                   ` (10 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

With ClangLTO or -ffunction-sections (DCE, FG-KASLR), compiler
places C functions into separate sections by default.
However, this doesn't happen with ASM functions which are still
being placed into .text.
Introduce a pack of macros which generate a new unique section
for the describing function named in the same fashion
(.text.<func_name>).
This will be needed to make input .text section empty to harden
the kernel even more.

Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 include/linux/linkage.h | 76 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)

diff --git a/include/linux/linkage.h b/include/linux/linkage.h
index dbf8506decca..ef8641a7dc1b 100644
--- a/include/linux/linkage.h
+++ b/include/linux/linkage.h
@@ -355,4 +355,80 @@
 
 #endif /* __ASSEMBLY__ */
 
+/*
+ * Allow ASM symbols to have their own unique sections if they are being
+ * generated by the compiler for C functions (DCE, FG-KASLR, LTO).
+ */
+#if (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) && !defined(MODULE)) || \
+    (defined(CONFIG_FG_KASLR) && !defined(MODULE)) || \
+    (defined(CONFIG_MODULE_FG_KASLR) && defined(MODULE)) || \
+    (defined(CONFIG_LTO_CLANG))
+
+#define ASM_PUSH_SECTION(name)				\
+	".pushsection .text." #name
+
+#define SYM_TEXT_SECTION(name)				\
+	.pushsection .text.##name
+
+#else /* just .text */
+
+#define ASM_PUSH_SECTION(name)				\
+	".pushsection .text"
+
+#define SYM_TEXT_SECTION(name)				\
+	.pushsection .text
+
+#endif /* just .text */
+
+#ifdef __ASSEMBLY__
+
+#define SYM_TEXT_END_SECTION				\
+	.popsection
+
+#define SYM_FUNC_START_LOCAL_ALIAS_SECTION(name)	\
+	SYM_TEXT_SECTION(name) ASM_NL			\
+	SYM_FUNC_START_LOCAL_ALIAS(name)
+
+#define SYM_FUNC_START_LOCAL_SECTION(name)		\
+	SYM_TEXT_SECTION(name) ASM_NL			\
+	SYM_FUNC_START_LOCAL(name)
+
+#define SYM_FUNC_START_NOALIGN_SECTION(name)		\
+	SYM_TEXT_SECTION(name) ASM_NL			\
+	SYM_FUNC_START_NOALIGN(name)
+
+#define SYM_FUNC_START_WEAK_SECTION(name)		\
+	SYM_TEXT_SECTION(name) ASM_NL			\
+	SYM_FUNC_START_WEAK(name)
+
+#define SYM_FUNC_START_SECTION(name)			\
+	SYM_TEXT_SECTION(name) ASM_NL			\
+	SYM_FUNC_START(name)
+
+#define SYM_CODE_START_LOCAL_NOALIGN_SECTION(name)	\
+	SYM_TEXT_SECTION(name) ASM_NL			\
+	SYM_CODE_START_LOCAL_NOALIGN(name)
+
+#define SYM_CODE_START_NOALIGN_SECTION(name)		\
+	SYM_TEXT_SECTION(name) ASM_NL			\
+	SYM_CODE_START_NOALIGN(name)
+
+#define SYM_CODE_START_SECTION(name)			\
+	SYM_TEXT_SECTION(name) ASM_NL			\
+	SYM_CODE_START(name)
+
+#define SYM_FUNC_END_ALIAS_SECTION(name)		\
+	SYM_FUNC_END_ALIAS(name) ASM_NL			\
+	SYM_TEXT_END_SECTION
+
+#define SYM_FUNC_END_SECTION(name)			\
+	SYM_FUNC_END(name) ASM_NL			\
+	SYM_TEXT_END_SECTION
+
+#define SYM_CODE_END_SECTION(name)			\
+	SYM_CODE_END(name) ASM_NL			\
+	SYM_TEXT_END_SECTION
+
+#endif /* __ASSEMBLY__ */
+
 #endif /* _LINUX_LINKAGE_H */
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 14/22] FG-KASLR: use a scripted approach to handle .text.* sections
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (11 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 12/22] linkage: add macros for putting ASM functions into own sections Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 15/22] kallsyms: Hide layout Alexander Lobakin
                   ` (9 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

Instead of relying on the linker and his heuristics about where to
place (orphan) .text.* section, use a script to read vmlinux.o and
generate a new .tmp_vmlinux.lds which will contain an entry for
each of them. It relies on a magic marker inside the preprocessed
vmlinux.lds (which is harmless in case FG-KASLR is disabled) and
injects a list of input text sections there.

As a bonus, this approach allows us to configure FG-KASLR in terms
of number of functions per each section. The zero value means one
section per each functions, it is the strongest choice, but the
resulting vmlinux also has the biggest size here, as well as the
total number of sections and the boottime delay (which is still
barely noticeable). The values of 4-8 are still strong enough and
allows to save some space, and so on.

We also keep tracking the maximal alignment we found while
traversing through the readelf output and the number of times we
spotted it. It's actual only for values >= 64 and is required to
reserve some space between the last .text.* section and the _etext
marker.
The reason is that e.g. x86 has at least 3 ASM sections (4 with
ClangCFI) aligned to 4096, and when mixing them with the small
sections, we could go past the _etext and render the kernel
unbootable. This reserved space ensures this won't happen.

Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 arch/x86/kernel/vmlinux.lds.S     |   4 +-
 include/asm-generic/vmlinux.lds.h |   6 ++
 init/Kconfig                      |  14 +++
 scripts/generate_text_sections.pl | 144 ++++++++++++++++++++++++++++++
 scripts/link-vmlinux.sh           |  25 +++++-
 5 files changed, 191 insertions(+), 2 deletions(-)
 create mode 100755 scripts/generate_text_sections.pl

diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 9692e990145b..4ed3b3b2b0e7 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -147,9 +147,11 @@ SECTIONS
 #endif
 	} :text =0xcccc
 
+	TEXT_FG_KASLR
+
 	/* End of text section, which should occupy whole number of pages */
-	_etext = .;
 	. = ALIGN(PAGE_SIZE);
+	_etext = .;
 
 	X86_ALIGN_RODATA_BEGIN
 	RO_DATA(PAGE_SIZE)
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 01fdeb5dd216..70fac18c786e 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -127,6 +127,12 @@
 #define TEXT_MAIN		.text
 #endif
 
+/*
+ * Used by scripts/generate_text_sections.pl to inject text sections,
+ * harmless if FG-KASLR is disabled.
+ */
+#define TEXT_FG_KASLR		__fg_kaslr_magic = .;
+
 /*
  * GCC 4.5 and later have a 32 bytes section alignment for structures.
  * Except GCC 4.9, that feels the need to align on 64 bytes.
diff --git a/init/Kconfig b/init/Kconfig
index cd1440b6a566..e72633f4f8a9 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2033,6 +2033,20 @@ config FG_KASLR
 
 	  If unsure, say N.
 
+config FG_KASLR_SHIFT
+	int "FG-KASLR granularity (number of functions per section shift)"
+	depends on FG_KASLR
+	range 0 16
+	default 0
+	help
+	  This sets the number of functions that will be put in each section
+	  as a power of two.
+	  Decreasing the value increases the randomization, but also increases
+	  the size of the final kernel/vmlinux due to the amount of sections.
+	  0 means that a separate section will be created for each function.
+	  16 almost disables the randomization, leaving only the manual
+	  separation.
+
 endmenu		# General setup
 
 source "arch/Kconfig"
diff --git a/scripts/generate_text_sections.pl b/scripts/generate_text_sections.pl
new file mode 100755
index 000000000000..5f3ece2ee0ea
--- /dev/null
+++ b/scripts/generate_text_sections.pl
@@ -0,0 +1,144 @@
+#!/usr/bin/env perl
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Generates a new LD script with every .text.* section described for FG-KASLR
+# to avoid orphan/heuristic section placement and double-checks we don't have
+# any symbols in plain .text section.
+#
+# Copyright (C) 2021, Intel Corporation.
+# Author: Alexander Lobakin <alexandr.lobakin@intel.com>
+#
+
+use strict;
+use warnings;
+
+## parameters
+my $expecting = 0;
+my $shift = 0;
+my $file;
+
+foreach (@ARGV) {
+	if ($_ eq '-s') {
+		$expecting = 1;
+	} elsif ($expecting) {
+		$shift = $_ + 0;
+		if ($shift < 0) {
+			$shift = 0;
+		} elsif ($shift > 16) {
+			$shift = 16;
+		}
+		$expecting = 0;
+	} elsif (!defined($file)) {
+		$file = $_;
+	} else {
+		die "$0: usage: $0 [-s shift] binary < linker script";
+	}
+}
+
+if (!defined($file)) {
+	die "$0: usage: $0 [-s shift] binary < linker script";
+}
+
+## environment
+my $readelf = $ENV{'READELF'} || die "$0: ERROR: READELF not set?";
+
+## text sections array
+my @sections = ();
+
+## max alignment found to reserve some space
+my $max_align = 64;
+my $count = 0;
+
+sub read_sections {
+	open(my $fh, "\"$readelf\" -SW \"$file\" 2>/dev/null |")
+		or die "$0: ERROR: failed to execute \"$readelf\": $!";
+
+	while (<$fh>) {
+		my $name;
+		my $align;
+		chomp;
+
+		($name, $align) = $_ =~ /^\s*\[[\s0-9]*\]\s*(\.\S*)\s*[A-Z]*\s*[0-9a-f]{16}\s*[0-9a-f]*\s*[0-9a-f]*\s*[0-9a-f]*\s*[0-9a-f]{2}\s*[A-Z]{2}\s*[0-9]\s*[0-9]\s*([0-9]*)$/;
+
+		if (!defined($name)) {
+			next;
+		}
+
+		if (!($name =~ /^\.text\.[0-9a-zA-Z_]*((\.constprop|\.isra|\.part)\.[0-9])*(|\.[0-9cfi]*)$/)) {
+			next;
+		}
+
+		if ($align > $max_align) {
+			$max_align = $align;
+			$count = 1;
+		} elsif ($align == $max_align) {
+			$count++;
+		}
+
+		push(@sections, $name);
+	}
+
+	close($fh);
+
+	@sections = sort @sections;
+}
+
+sub print_sections {
+	my $fps = 1 << $shift;
+	my $counter = 1;
+
+	print "\t.text.0 : ALIGN(16) {\n";
+	print "\t\t*(.text)\n";
+	print "\t}\n";
+
+	print "\tASSERT(SIZEOF(.text.0) == 0, \"Plain .text is not empty!\")\n\n";
+
+	if (!@sections) {
+		return;
+	}
+
+	while () {
+		print "\t.text.$counter : ALIGN(16) {\n";
+
+		my @a = (($counter - 1) * $fps .. ($counter * $fps) - 1);
+		for (@a) {
+			print "\t\t*($sections[$_])\n";
+
+			if ($sections[$_] eq $sections[-1]) {
+				print "\t}\n";
+				return;
+			}
+		}
+
+		print "\t}\n";
+		$counter++;
+	}
+}
+
+sub print_reserve {
+	## If we have text sections aligned with 64 bytes or more, make sure
+	## we reserve some space for them to not overlap _etext while shuffling
+	## sections
+
+	if (!$count) {
+		return;
+	}
+
+	print "\n\t. += $max_align * $count;\n";
+}
+
+sub print_lds {
+	while (<STDIN>) {
+		if ($_ =~ /^\s*__fg_kaslr_magic = \.;$/) {
+			print_sections();
+			print_reserve();
+		} else {
+			print $_;
+		}
+	}
+}
+
+## main
+
+read_sections();
+print_lds();
diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index d74cee5c4326..b4e6578371bc 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -66,6 +66,18 @@ gen_symversions()
 	done
 }
 
+# If CONFIG_FG_KASLR is selected, generate a linker script which will
+# declare all custom text sections for future boottime shuffling
+gen_text_sections()
+{
+	info GEN .tmp_vmlinux.lds
+
+	${PERL} ${srctree}/scripts/generate_text_sections.pl	\
+		-s "${CONFIG_FG_KASLR_SHIFT}" vmlinux.o		\
+		< "${objtree}/${KBUILD_LDS}"			\
+		> .tmp_vmlinux.lds
+}
+
 # Link of vmlinux.o used for section mismatch analysis
 # ${1} output file
 modpost_link()
@@ -155,12 +167,19 @@ vmlinux_link()
 	local ld
 	local ldflags
 	local ldlibs
+	local lds
 
 	info LD ${output}
 
 	# skip output file argument
 	shift
 
+	if [ -n "${CONFIG_FG_KASLR}" ]; then
+		lds=".tmp_vmlinux.lds"
+	else
+		lds="${objtree}/${KBUILD_LDS}"
+	fi
+
 	if [ -n "${CONFIG_LTO_CLANG}" ]; then
 		# Use vmlinux.o instead of performing the slow LTO link again.
 		objs=vmlinux.o
@@ -182,7 +201,7 @@ vmlinux_link()
 		ldlibs=
 	fi
 
-	ldflags="${ldflags} ${wl}--script=${objtree}/${KBUILD_LDS}"
+	ldflags="${ldflags} ${wl}--script=${lds}"
 
 	# The kallsyms linking does not need debug symbols included.
 	if [ "$output" != "${output#.tmp_vmlinux.kallsyms}" ] ; then
@@ -351,6 +370,10 @@ info GEN modules.builtin
 tr '\0' '\n' < modules.builtin.modinfo | sed -n 's/^[[:alnum:]:_]*\.file=//p' |
 	tr ' ' '\n' | uniq | sed -e 's:^:kernel/:' -e 's/$/.ko/' > modules.builtin
 
+if [ -n "${CONFIG_FG_KASLR}" ]; then
+	gen_text_sections
+fi
+
 btf_vmlinux_bin_o=""
 if [ -n "${CONFIG_DEBUG_INFO_BTF}" ]; then
 	btf_vmlinux_bin_o=.btf.vmlinux.bin.o
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 15/22] kallsyms: Hide layout
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (12 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 14/22] FG-KASLR: use a scripted approach to handle .text.* sections Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 16/22] livepatch: only match unique symbols when using fgkaslr Alexander Lobakin
                   ` (8 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux, kernel test robot

From: Kristen Carlson Accardi <kristen@linux.intel.com>

This patch makes /proc/kallsyms display in a random order, rather
than sorted by address in order to hide the newly randomized address
layout.

Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Tested-by: Tony Luck <tony.luck@intel.com>
Reported-by: kernel test robot <lkp@intel.com> # swap.cocci
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 kernel/kallsyms.c | 158 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 157 insertions(+), 1 deletion(-)

diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c
index 0ba87982d017..5ffdcc2fb88e 100644
--- a/kernel/kallsyms.c
+++ b/kernel/kallsyms.c
@@ -563,6 +563,12 @@ struct kallsym_iter {
 	int show_value;
 };
 
+struct kallsyms_shuffled_iter {
+	struct kallsym_iter iter;
+	loff_t total_syms;
+	loff_t shuffled_index[];
+};
+
 int __weak arch_get_kallsym(unsigned int symnum, unsigned long *value,
 			    char *type, char *name)
 {
@@ -810,7 +816,7 @@ bool kallsyms_show_value(const struct cred *cred)
 	}
 }
 
-static int kallsyms_open(struct inode *inode, struct file *file)
+static int __kallsyms_open(struct inode *inode, struct file *file)
 {
 	/*
 	 * We keep iterator in m->private, since normal case is to
@@ -831,6 +837,156 @@ static int kallsyms_open(struct inode *inode, struct file *file)
 	return 0;
 }
 
+/*
+ * When function granular kaslr is enabled, we need to print out the symbols
+ * at random so we don't reveal the new layout.
+ */
+#ifdef CONFIG_FG_KASLR
+static int update_random_pos(struct kallsyms_shuffled_iter *s_iter,
+			     loff_t pos, loff_t *new_pos)
+{
+	loff_t new;
+
+	if (pos >= s_iter->total_syms)
+		return 0;
+
+	new = s_iter->shuffled_index[pos];
+
+	/*
+	 * normally this would be done as part of update_iter, however,
+	 * we want to avoid triggering this in the event that new is
+	 * zero since we don't want to blow away our pos end indicators.
+	 */
+	if (new == 0) {
+		s_iter->iter.name[0] = '\0';
+		s_iter->iter.nameoff = get_symbol_offset(new);
+		s_iter->iter.pos = new;
+	}
+
+	*new_pos = new;
+	return 1;
+}
+
+static void *shuffled_start(struct seq_file *m, loff_t *pos)
+{
+	struct kallsyms_shuffled_iter *s_iter = m->private;
+	loff_t new_pos;
+
+	if (!update_random_pos(s_iter, *pos, &new_pos))
+		return NULL;
+
+	return s_start(m, &new_pos);
+}
+
+static void *shuffled_next(struct seq_file *m, void *p, loff_t *pos)
+{
+	struct kallsyms_shuffled_iter *s_iter = m->private;
+	loff_t new_pos;
+
+	(*pos)++;
+
+	if (!update_random_pos(s_iter, *pos, &new_pos))
+		return NULL;
+
+	if (!update_iter(m->private, new_pos))
+		return NULL;
+
+	return p;
+}
+
+/*
+ * shuffle_index_list()
+ * Use a Fisher Yates algorithm to shuffle a list of text sections.
+ */
+static void shuffle_index_list(loff_t *indexes, loff_t size)
+{
+	u32 i, j;
+
+	for (i = size - 1; i > 0; i--) {
+		/* pick a random index from 0 to i */
+		j = get_random_u32() % (i + 1);
+
+		swap(indexes[i], indexes[j]);
+	}
+}
+
+static const struct seq_operations kallsyms_shuffled_op = {
+	.start = shuffled_start,
+	.next = shuffled_next,
+	.stop = s_stop,
+	.show = s_show
+};
+
+static int kallsyms_random_open(struct inode *inode, struct file *file)
+{
+	loff_t pos;
+	struct kallsyms_shuffled_iter *shuffled_iter;
+	struct kallsym_iter iter;
+	bool show_value;
+
+	/*
+	 * If privileged, go ahead and use the normal algorithm for
+	 * displaying symbols
+	 */
+	show_value = kallsyms_show_value(file->f_cred);
+	if (show_value)
+		return __kallsyms_open(inode, file);
+
+	/*
+	 * we need to figure out how many extra symbols there are
+	 * to print out past kallsyms_num_syms
+	 */
+	pos = kallsyms_num_syms;
+	reset_iter(&iter, 0);
+	do {
+		if (!update_iter(&iter, pos))
+			break;
+		pos++;
+	} while (1);
+
+	/*
+	 * add storage space for an array of loff_t equal to the size
+	 * of the total number of symbols we need to print
+	 */
+	shuffled_iter = __seq_open_private(file, &kallsyms_shuffled_op,
+					   sizeof(*shuffled_iter) +
+					   (sizeof(pos) * pos));
+	if (!shuffled_iter)
+		return -ENOMEM;
+
+	reset_iter(&shuffled_iter->iter, 0);
+	shuffled_iter->iter.show_value = show_value;
+	shuffled_iter->total_syms = pos;
+
+	/*
+	 * the existing update_iter algorithm requires that we
+	 * are either moving along increasing pos sequentially,
+	 * or that these values are correct. Since these values
+	 * were discovered above, initialize our new iter so we
+	 * can use update_iter non-sequentially.
+	 */
+	shuffled_iter->iter.pos_arch_end = iter.pos_arch_end;
+	shuffled_iter->iter.pos_mod_end = iter.pos_mod_end;
+	shuffled_iter->iter.pos_ftrace_mod_end = iter.pos_ftrace_mod_end;
+
+	/*
+	 * initialize the array with all possible pos values, then
+	 * shuffle the array so that the values will display in a random
+	 * order.
+	 */
+	for (pos = 0; pos < shuffled_iter->total_syms; pos++)
+		shuffled_iter->shuffled_index[pos] = pos;
+
+	shuffle_index_list(shuffled_iter->shuffled_index, shuffled_iter->total_syms);
+
+	return 0;
+}
+
+#define kallsyms_open kallsyms_random_open
+#else
+#define kallsyms_open __kallsyms_open
+#endif /* !CONFIG_FG_KASLR */
+
 #ifdef	CONFIG_KGDB_KDB
 const char *kdb_walk_kallsyms(loff_t *pos)
 {
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 16/22] livepatch: only match unique symbols when using fgkaslr
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (13 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 15/22] kallsyms: Hide layout Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-09-09 11:53   ` Miroslav Benes
  2021-08-31 14:41 ` [PATCH v6 kspp-next 17/22] x86/boot: allow FG-KASLR to be selected Alexander Lobakin
                   ` (7 subsequent siblings)
  22 siblings, 1 reply; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

From: Kristen Carlson Accardi <kristen@linux.intel.com>

If any type of function granular randomization is enabled, the sympos
algorithm will fail, as it will be impossible to resolve symbols when
there are duplicates using the previous symbol position.

Override the value of sympos to always be zero if fgkaslr is enabled for
either the core kernel or modules, forcing the algorithm
to require that only unique symbols are allowed to be patched.

Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 kernel/livepatch/core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 335d988bd811..852bbfa9da7b 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -169,6 +169,17 @@ static int klp_find_object_symbol(const char *objname, const char *name,
 	else
 		kallsyms_on_each_symbol(klp_find_callback, &args);
 
+	/*
+	 * If any type of function granular randomization is enabled, it
+	 * will be impossible to resolve symbols when there are duplicates
+	 * using the previous symbol position (i.e. sympos != 0). Override
+	 * the value of sympos to always be zero in this case. This will
+	 * force the algorithm to require that only unique symbols are
+	 * allowed to be patched.
+	 */
+	if (IS_ENABLED(CONFIG_FG_KASLR))
+		sympos = 0;
+
 	/*
 	 * Ensure an address was found. If sympos is 0, ensure symbol is unique;
 	 * otherwise ensure the symbol position count matches sympos.
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 17/22] x86/boot: allow FG-KASLR to be selected
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (14 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 16/22] livepatch: only match unique symbols when using fgkaslr Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 18/22] arm64/crypto: conditionally place ASM functions into separate sections Alexander Lobakin
                   ` (6 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

Now that we have full support of FG-KASLR from both kernel core
and x86 code, allow FG-KASLR to be enabled for x86_64 if the
"regular" KASLR is also turned on.

Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 88fb922c23a0..bba561053557 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -74,6 +74,7 @@ config X86
 	select ARCH_HAS_EARLY_DEBUG		if KGDB
 	select ARCH_HAS_ELF_RANDOMIZE
 	select ARCH_HAS_FAST_MULTIPLIER
+	select ARCH_HAS_FG_KASLR		if X86_64 && RANDOMIZE_BASE
 	select ARCH_HAS_FILTER_PGPROT
 	select ARCH_HAS_FORTIFY_SOURCE
 	select ARCH_HAS_GCOV_PROFILE_ALL
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 18/22] arm64/crypto: conditionally place ASM functions into separate sections
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (15 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 17/22] x86/boot: allow FG-KASLR to be selected Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 19/22] module: Reorder functions Alexander Lobakin
                   ` (5 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

The resulting LD script generated by FG-KASLR script contains a size
assertion for the input .text function. In case if it's not empty,
the build will stop plug a potentional layout leakage.
As FG-KASLR for modules tends to be arch-independent, we should take
care of the modular ASM code of every architecture to not break the
build.
This is the ARM64 part.

Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 arch/arm64/crypto/aes-ce-ccm-core.S   | 16 +++++------
 arch/arm64/crypto/aes-ce-core.S       | 16 +++++------
 arch/arm64/crypto/aes-ce.S            |  4 +--
 arch/arm64/crypto/aes-cipher-core.S   |  8 +++---
 arch/arm64/crypto/aes-modes.S         | 16 +++++------
 arch/arm64/crypto/aes-neon.S          |  4 +--
 arch/arm64/crypto/aes-neonbs-core.S   | 38 +++++++++++++--------------
 arch/arm64/crypto/chacha-neon-core.S  | 18 ++++++-------
 arch/arm64/crypto/crct10dif-ce-core.S | 14 +++++-----
 arch/arm64/crypto/ghash-ce-core.S     | 24 ++++++++---------
 arch/arm64/crypto/nh-neon-core.S      |  4 +--
 arch/arm64/crypto/poly1305-armv8.pl   | 17 ++++++++++++
 arch/arm64/crypto/sha1-ce-core.S      |  4 +--
 arch/arm64/crypto/sha2-ce-core.S      |  4 +--
 arch/arm64/crypto/sha3-ce-core.S      |  4 +--
 arch/arm64/crypto/sha512-armv8.pl     | 11 ++++++++
 arch/arm64/crypto/sha512-ce-core.S    |  4 +--
 arch/arm64/crypto/sm3-ce-core.S       |  4 +--
 arch/arm64/crypto/sm4-ce-core.S       |  4 +--
 19 files changed, 121 insertions(+), 93 deletions(-)

diff --git a/arch/arm64/crypto/aes-ce-ccm-core.S b/arch/arm64/crypto/aes-ce-ccm-core.S
index 99a028e298ed..e9a7c7bfecda 100644
--- a/arch/arm64/crypto/aes-ce-ccm-core.S
+++ b/arch/arm64/crypto/aes-ce-ccm-core.S
@@ -15,7 +15,7 @@
 	 * void ce_aes_ccm_auth_data(u8 mac[], u8 const in[], u32 abytes,
 	 *			     u32 *macp, u8 const rk[], u32 rounds);
 	 */
-SYM_FUNC_START(ce_aes_ccm_auth_data)
+SYM_FUNC_START_SECTION(ce_aes_ccm_auth_data)
 	ldr	w8, [x3]			/* leftover from prev round? */
 	ld1	{v0.16b}, [x0]			/* load mac */
 	cbz	w8, 1f
@@ -81,13 +81,13 @@ SYM_FUNC_START(ce_aes_ccm_auth_data)
 	st1	{v0.16b}, [x0]
 10:	str	w8, [x3]
 	ret
-SYM_FUNC_END(ce_aes_ccm_auth_data)
+SYM_FUNC_END_SECTION(ce_aes_ccm_auth_data)
 
 	/*
 	 * void ce_aes_ccm_final(u8 mac[], u8 const ctr[], u8 const rk[],
 	 * 			 u32 rounds);
 	 */
-SYM_FUNC_START(ce_aes_ccm_final)
+SYM_FUNC_START_SECTION(ce_aes_ccm_final)
 	ld1	{v3.4s}, [x2], #16		/* load first round key */
 	ld1	{v0.16b}, [x0]			/* load mac */
 	cmp	w3, #12				/* which key size? */
@@ -121,7 +121,7 @@ SYM_FUNC_START(ce_aes_ccm_final)
 	eor	v0.16b, v0.16b, v1.16b		/* en-/decrypt the mac */
 	st1	{v0.16b}, [x0]			/* store result */
 	ret
-SYM_FUNC_END(ce_aes_ccm_final)
+SYM_FUNC_END_SECTION(ce_aes_ccm_final)
 
 	.macro	aes_ccm_do_crypt,enc
 	ldr	x8, [x6, #8]			/* load lower ctr */
@@ -212,10 +212,10 @@ CPU_LE(	rev	x8, x8			)
 	 * 			   u8 const rk[], u32 rounds, u8 mac[],
 	 * 			   u8 ctr[]);
 	 */
-SYM_FUNC_START(ce_aes_ccm_encrypt)
+SYM_FUNC_START_SECTION(ce_aes_ccm_encrypt)
 	aes_ccm_do_crypt	1
-SYM_FUNC_END(ce_aes_ccm_encrypt)
+SYM_FUNC_END_SECTION(ce_aes_ccm_encrypt)
 
-SYM_FUNC_START(ce_aes_ccm_decrypt)
+SYM_FUNC_START_SECTION(ce_aes_ccm_decrypt)
 	aes_ccm_do_crypt	0
-SYM_FUNC_END(ce_aes_ccm_decrypt)
+SYM_FUNC_END_SECTION(ce_aes_ccm_decrypt)
diff --git a/arch/arm64/crypto/aes-ce-core.S b/arch/arm64/crypto/aes-ce-core.S
index e52e13eb8fdb..abe6ee0501bf 100644
--- a/arch/arm64/crypto/aes-ce-core.S
+++ b/arch/arm64/crypto/aes-ce-core.S
@@ -8,7 +8,7 @@
 
 	.arch		armv8-a+crypto
 
-SYM_FUNC_START(__aes_ce_encrypt)
+SYM_FUNC_START_SECTION(__aes_ce_encrypt)
 	sub		w3, w3, #2
 	ld1		{v0.16b}, [x2]
 	ld1		{v1.4s}, [x0], #16
@@ -34,9 +34,9 @@ SYM_FUNC_START(__aes_ce_encrypt)
 	eor		v0.16b, v0.16b, v3.16b
 	st1		{v0.16b}, [x1]
 	ret
-SYM_FUNC_END(__aes_ce_encrypt)
+SYM_FUNC_END_SECTION(__aes_ce_encrypt)
 
-SYM_FUNC_START(__aes_ce_decrypt)
+SYM_FUNC_START_SECTION(__aes_ce_decrypt)
 	sub		w3, w3, #2
 	ld1		{v0.16b}, [x2]
 	ld1		{v1.4s}, [x0], #16
@@ -62,23 +62,23 @@ SYM_FUNC_START(__aes_ce_decrypt)
 	eor		v0.16b, v0.16b, v3.16b
 	st1		{v0.16b}, [x1]
 	ret
-SYM_FUNC_END(__aes_ce_decrypt)
+SYM_FUNC_END_SECTION(__aes_ce_decrypt)
 
 /*
  * __aes_ce_sub() - use the aese instruction to perform the AES sbox
  *                  substitution on each byte in 'input'
  */
-SYM_FUNC_START(__aes_ce_sub)
+SYM_FUNC_START_SECTION(__aes_ce_sub)
 	dup		v1.4s, w0
 	movi		v0.16b, #0
 	aese		v0.16b, v1.16b
 	umov		w0, v0.s[0]
 	ret
-SYM_FUNC_END(__aes_ce_sub)
+SYM_FUNC_END_SECTION(__aes_ce_sub)
 
-SYM_FUNC_START(__aes_ce_invert)
+SYM_FUNC_START_SECTION(__aes_ce_invert)
 	ld1		{v0.4s}, [x1]
 	aesimc		v1.16b, v0.16b
 	st1		{v1.4s}, [x0]
 	ret
-SYM_FUNC_END(__aes_ce_invert)
+SYM_FUNC_END_SECTION(__aes_ce_invert)
diff --git a/arch/arm64/crypto/aes-ce.S b/arch/arm64/crypto/aes-ce.S
index 1dc5bbbfeed2..909d2dcf0907 100644
--- a/arch/arm64/crypto/aes-ce.S
+++ b/arch/arm64/crypto/aes-ce.S
@@ -9,8 +9,8 @@
 #include <linux/linkage.h>
 #include <asm/assembler.h>
 
-#define AES_FUNC_START(func)		SYM_FUNC_START(ce_ ## func)
-#define AES_FUNC_END(func)		SYM_FUNC_END(ce_ ## func)
+#define AES_FUNC_START(func)		SYM_FUNC_START_SECTION(ce_ ## func)
+#define AES_FUNC_END(func)		SYM_FUNC_END_SECTION(ce_ ## func)
 
 	.arch		armv8-a+crypto
 
diff --git a/arch/arm64/crypto/aes-cipher-core.S b/arch/arm64/crypto/aes-cipher-core.S
index c9d6955f8404..e47c0aef7a7d 100644
--- a/arch/arm64/crypto/aes-cipher-core.S
+++ b/arch/arm64/crypto/aes-cipher-core.S
@@ -122,11 +122,11 @@ CPU_BE(	rev		w7, w7		)
 	ret
 	.endm
 
-SYM_FUNC_START(__aes_arm64_encrypt)
+SYM_FUNC_START_SECTION(__aes_arm64_encrypt)
 	do_crypt	fround, crypto_ft_tab, crypto_ft_tab + 1, 2
-SYM_FUNC_END(__aes_arm64_encrypt)
+SYM_FUNC_END_SECTION(__aes_arm64_encrypt)
 
+SYM_FUNC_START_SECTION(__aes_arm64_decrypt)
 	.align		5
-SYM_FUNC_START(__aes_arm64_decrypt)
 	do_crypt	iround, crypto_it_tab, crypto_aes_inv_sbox, 0
-SYM_FUNC_END(__aes_arm64_decrypt)
+SYM_FUNC_END_SECTION(__aes_arm64_decrypt)
diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S
index b495de22bb38..5f7a43fa8438 100644
--- a/arch/arm64/crypto/aes-modes.S
+++ b/arch/arm64/crypto/aes-modes.S
@@ -22,26 +22,26 @@
 #define ST5(x...) x
 #endif
 
-SYM_FUNC_START_LOCAL(aes_encrypt_block4x)
+SYM_FUNC_START_LOCAL_SECTION(aes_encrypt_block4x)
 	encrypt_block4x	v0, v1, v2, v3, w3, x2, x8, w7
 	ret
-SYM_FUNC_END(aes_encrypt_block4x)
+SYM_FUNC_END_SECTION(aes_encrypt_block4x)
 
-SYM_FUNC_START_LOCAL(aes_decrypt_block4x)
+SYM_FUNC_START_LOCAL_SECTION(aes_decrypt_block4x)
 	decrypt_block4x	v0, v1, v2, v3, w3, x2, x8, w7
 	ret
-SYM_FUNC_END(aes_decrypt_block4x)
+SYM_FUNC_END_SECTION(aes_decrypt_block4x)
 
 #if MAX_STRIDE == 5
-SYM_FUNC_START_LOCAL(aes_encrypt_block5x)
+SYM_FUNC_START_LOCAL_SECTION(aes_encrypt_block5x)
 	encrypt_block5x	v0, v1, v2, v3, v4, w3, x2, x8, w7
 	ret
-SYM_FUNC_END(aes_encrypt_block5x)
+SYM_FUNC_END_SECTION(aes_encrypt_block5x)
 
-SYM_FUNC_START_LOCAL(aes_decrypt_block5x)
+SYM_FUNC_START_LOCAL_SECTION(aes_decrypt_block5x)
 	decrypt_block5x	v0, v1, v2, v3, v4, w3, x2, x8, w7
 	ret
-SYM_FUNC_END(aes_decrypt_block5x)
+SYM_FUNC_END_SECTION(aes_decrypt_block5x)
 #endif
 
 	/*
diff --git a/arch/arm64/crypto/aes-neon.S b/arch/arm64/crypto/aes-neon.S
index e47d3ec2cfb4..9c8d6cccd2cd 100644
--- a/arch/arm64/crypto/aes-neon.S
+++ b/arch/arm64/crypto/aes-neon.S
@@ -8,8 +8,8 @@
 #include <linux/linkage.h>
 #include <asm/assembler.h>
 
-#define AES_FUNC_START(func)		SYM_FUNC_START(neon_ ## func)
-#define AES_FUNC_END(func)		SYM_FUNC_END(neon_ ## func)
+#define AES_FUNC_START(func)		SYM_FUNC_START_SECTION(neon_ ## func)
+#define AES_FUNC_END(func)		SYM_FUNC_END_SECTION(neon_ ## func)
 
 	xtsmask		.req	v7
 	cbciv		.req	v7
diff --git a/arch/arm64/crypto/aes-neonbs-core.S b/arch/arm64/crypto/aes-neonbs-core.S
index a3405b8c344b..582343f18ad0 100644
--- a/arch/arm64/crypto/aes-neonbs-core.S
+++ b/arch/arm64/crypto/aes-neonbs-core.S
@@ -380,7 +380,7 @@ ISRM0:	.octa		0x0306090c00070a0d01040b0e0205080f
 	/*
 	 * void aesbs_convert_key(u8 out[], u32 const rk[], int rounds)
 	 */
-SYM_FUNC_START(aesbs_convert_key)
+SYM_FUNC_START_SECTION(aesbs_convert_key)
 	ld1		{v7.4s}, [x1], #16		// load round 0 key
 	ld1		{v17.4s}, [x1], #16		// load round 1 key
 
@@ -425,10 +425,10 @@ SYM_FUNC_START(aesbs_convert_key)
 	eor		v17.16b, v17.16b, v7.16b
 	str		q17, [x0]
 	ret
-SYM_FUNC_END(aesbs_convert_key)
+SYM_FUNC_END_SECTION(aesbs_convert_key)
 
+SYM_FUNC_START_LOCAL_SECTION(aesbs_encrypt8)
 	.align		4
-SYM_FUNC_START_LOCAL(aesbs_encrypt8)
 	ldr		q9, [bskey], #16		// round 0 key
 	ldr		q8, M0SR
 	ldr		q24, SR
@@ -488,10 +488,10 @@ SYM_FUNC_START_LOCAL(aesbs_encrypt8)
 	eor		v2.16b, v2.16b, v12.16b
 	eor		v5.16b, v5.16b, v12.16b
 	ret
-SYM_FUNC_END(aesbs_encrypt8)
+SYM_FUNC_END_SECTION(aesbs_encrypt8)
 
+SYM_FUNC_START_LOCAL_SECTION(aesbs_decrypt8)
 	.align		4
-SYM_FUNC_START_LOCAL(aesbs_decrypt8)
 	lsl		x9, rounds, #7
 	add		bskey, bskey, x9
 
@@ -553,7 +553,7 @@ SYM_FUNC_START_LOCAL(aesbs_decrypt8)
 	eor		v3.16b, v3.16b, v12.16b
 	eor		v5.16b, v5.16b, v12.16b
 	ret
-SYM_FUNC_END(aesbs_decrypt8)
+SYM_FUNC_END_SECTION(aesbs_decrypt8)
 
 	/*
 	 * aesbs_ecb_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
@@ -619,13 +619,13 @@ SYM_FUNC_END(aesbs_decrypt8)
 	ret
 	.endm
 
+SYM_FUNC_START_SECTION(aesbs_ecb_encrypt)
 	.align		4
-SYM_FUNC_START(aesbs_ecb_encrypt)
 	__ecb_crypt	aesbs_encrypt8, v0, v1, v4, v6, v3, v7, v2, v5
-SYM_FUNC_END(aesbs_ecb_encrypt)
+SYM_FUNC_END_SECTION(aesbs_ecb_encrypt)
 
+SYM_FUNC_START_SECTION(aesbs_ecb_decrypt)
 	.align		4
-SYM_FUNC_START(aesbs_ecb_decrypt)
 	__ecb_crypt	aesbs_decrypt8, v0, v1, v6, v4, v2, v7, v3, v5
 SYM_FUNC_END(aesbs_ecb_decrypt)
 
@@ -633,8 +633,8 @@ SYM_FUNC_END(aesbs_ecb_decrypt)
 	 * aesbs_cbc_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
 	 *		     int blocks, u8 iv[])
 	 */
+SYM_FUNC_START_SECTION(aesbs_cbc_decrypt)
 	.align		4
-SYM_FUNC_START(aesbs_cbc_decrypt)
 	frame_push	6
 
 	mov		x19, x0
@@ -718,7 +718,7 @@ SYM_FUNC_START(aesbs_cbc_decrypt)
 
 2:	frame_pop
 	ret
-SYM_FUNC_END(aesbs_cbc_decrypt)
+SYM_FUNC_END_SECTION(aesbs_cbc_decrypt)
 
 	.macro		next_tweak, out, in, const, tmp
 	sshr		\tmp\().2d,  \in\().2d,   #63
@@ -734,7 +734,7 @@ SYM_FUNC_END(aesbs_cbc_decrypt)
 	 * aesbs_xts_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
 	 *		     int blocks, u8 iv[])
 	 */
-SYM_FUNC_START_LOCAL(__xts_crypt8)
+SYM_FUNC_START_LOCAL_SECTION(__xts_crypt8)
 	mov		x6, #1
 	lsl		x6, x6, x23
 	subs		w23, w23, #8
@@ -787,7 +787,7 @@ SYM_FUNC_START_LOCAL(__xts_crypt8)
 0:	mov		bskey, x21
 	mov		rounds, x22
 	br		x16
-SYM_FUNC_END(__xts_crypt8)
+SYM_FUNC_END_SECTION(__xts_crypt8)
 
 	.macro		__xts_crypt, do8, o0, o1, o2, o3, o4, o5, o6, o7
 	frame_push	6, 64
@@ -851,13 +851,13 @@ SYM_FUNC_END(__xts_crypt8)
 	ret
 	.endm
 
-SYM_FUNC_START(aesbs_xts_encrypt)
+SYM_FUNC_START_SECTION(aesbs_xts_encrypt)
 	__xts_crypt	aesbs_encrypt8, v0, v1, v4, v6, v3, v7, v2, v5
-SYM_FUNC_END(aesbs_xts_encrypt)
+SYM_FUNC_END_SECTION(aesbs_xts_encrypt)
 
-SYM_FUNC_START(aesbs_xts_decrypt)
+SYM_FUNC_START_SECTION(aesbs_xts_decrypt)
 	__xts_crypt	aesbs_decrypt8, v0, v1, v6, v4, v2, v7, v3, v5
-SYM_FUNC_END(aesbs_xts_decrypt)
+SYM_FUNC_END_SECTION(aesbs_xts_decrypt)
 
 	.macro		next_ctr, v
 	mov		\v\().d[1], x8
@@ -871,7 +871,7 @@ SYM_FUNC_END(aesbs_xts_decrypt)
 	 * aesbs_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[],
 	 *		     int rounds, int blocks, u8 iv[], u8 final[])
 	 */
-SYM_FUNC_START(aesbs_ctr_encrypt)
+SYM_FUNC_START_SECTION(aesbs_ctr_encrypt)
 	frame_push	8
 
 	mov		x19, x0
@@ -998,4 +998,4 @@ CPU_LE(	rev		x8, x8		)
 7:	cbz		x25, 8b
 	st1		{v5.16b}, [x25]
 	b		8b
-SYM_FUNC_END(aesbs_ctr_encrypt)
+SYM_FUNC_END_SECTION(aesbs_ctr_encrypt)
diff --git a/arch/arm64/crypto/chacha-neon-core.S b/arch/arm64/crypto/chacha-neon-core.S
index b70ac76f2610..34a3087055f8 100644
--- a/arch/arm64/crypto/chacha-neon-core.S
+++ b/arch/arm64/crypto/chacha-neon-core.S
@@ -23,7 +23,6 @@
 #include <asm/cache.h>
 
 	.text
-	.align		6
 
 /*
  * chacha_permute - permute one block
@@ -36,7 +35,8 @@
  *
  * Clobbers: w3, x10, v4, v12
  */
-SYM_FUNC_START_LOCAL(chacha_permute)
+SYM_FUNC_START_LOCAL_SECTION(chacha_permute)
+	.align		6
 
 	adr_l		x10, ROT8
 	ld1		{v12.4s}, [x10]
@@ -104,9 +104,9 @@ SYM_FUNC_START_LOCAL(chacha_permute)
 	b.ne		.Ldoubleround
 
 	ret
-SYM_FUNC_END(chacha_permute)
+SYM_FUNC_END_SECTION(chacha_permute)
 
-SYM_FUNC_START(chacha_block_xor_neon)
+SYM_FUNC_START_SECTION(chacha_block_xor_neon)
 	// x0: Input state matrix, s
 	// x1: 1 data block output, o
 	// x2: 1 data block input, i
@@ -143,9 +143,9 @@ SYM_FUNC_START(chacha_block_xor_neon)
 
 	ldp		x29, x30, [sp], #16
 	ret
-SYM_FUNC_END(chacha_block_xor_neon)
+SYM_FUNC_END_SECTION(chacha_block_xor_neon)
 
-SYM_FUNC_START(hchacha_block_neon)
+SYM_FUNC_START_SECTION(hchacha_block_neon)
 	// x0: Input state matrix, s
 	// x1: output (8 32-bit words)
 	// w2: nrounds
@@ -163,7 +163,7 @@ SYM_FUNC_START(hchacha_block_neon)
 
 	ldp		x29, x30, [sp], #16
 	ret
-SYM_FUNC_END(hchacha_block_neon)
+SYM_FUNC_END_SECTION(hchacha_block_neon)
 
 	a0		.req	w12
 	a1		.req	w13
@@ -182,8 +182,8 @@ SYM_FUNC_END(hchacha_block_neon)
 	a14		.req	w27
 	a15		.req	w28
 
+SYM_FUNC_START_SECTION(chacha_4block_xor_neon)
 	.align		6
-SYM_FUNC_START(chacha_4block_xor_neon)
 	frame_push	10
 
 	// x0: Input state matrix, s
@@ -790,7 +790,7 @@ CPU_BE(	  rev		a15, a15	)
 	st1		{v28.16b-v31.16b}, [x7]		// overlapping stores
 3:	st1		{v24.16b-v27.16b}, [x1]
 	b		.Lout
-SYM_FUNC_END(chacha_4block_xor_neon)
+SYM_FUNC_END_SECTION(chacha_4block_xor_neon)
 
 	.section	".rodata", "a", %progbits
 	.align		L1_CACHE_SHIFT
diff --git a/arch/arm64/crypto/crct10dif-ce-core.S b/arch/arm64/crypto/crct10dif-ce-core.S
index dce6dcebfca1..54e121a56895 100644
--- a/arch/arm64/crypto/crct10dif-ce-core.S
+++ b/arch/arm64/crypto/crct10dif-ce-core.S
@@ -131,7 +131,7 @@
 	tbl		bd4.16b, {\bd\().16b}, perm4.16b
 	.endm
 
-SYM_FUNC_START_LOCAL(__pmull_p8_core)
+SYM_FUNC_START_LOCAL_SECTION(__pmull_p8_core)
 .L__pmull_p8_core:
 	ext		t4.8b, ad.8b, ad.8b, #1			// A1
 	ext		t5.8b, ad.8b, ad.8b, #2			// A2
@@ -194,7 +194,7 @@ SYM_FUNC_START_LOCAL(__pmull_p8_core)
 	eor		t4.16b, t4.16b, t5.16b
 	eor		t6.16b, t6.16b, t3.16b
 	ret
-SYM_FUNC_END(__pmull_p8_core)
+SYM_FUNC_END_SECTION(__pmull_p8_core)
 
 	.macro		__pmull_p8, rq, ad, bd, i
 	.ifnc		\bd, fold_consts
@@ -465,21 +465,21 @@ CPU_LE(	ext		v7.16b, v7.16b, v7.16b, #8	)
 //
 // Assumes len >= 16.
 //
-SYM_FUNC_START(crc_t10dif_pmull_p8)
+SYM_FUNC_START_SECTION(crc_t10dif_pmull_p8)
 	stp		x29, x30, [sp, #-16]!
 	mov		x29, sp
 	crc_t10dif_pmull p8
-SYM_FUNC_END(crc_t10dif_pmull_p8)
+SYM_FUNC_END_SECTION(crc_t10dif_pmull_p8)
 
-	.align		5
 //
 // u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len);
 //
 // Assumes len >= 16.
 //
-SYM_FUNC_START(crc_t10dif_pmull_p64)
+SYM_FUNC_START_SECTION(crc_t10dif_pmull_p64)
+	.align			5
 	crc_t10dif_pmull	p64
-SYM_FUNC_END(crc_t10dif_pmull_p64)
+SYM_FUNC_END_SECTION(crc_t10dif_pmull_p64)
 
 	.section	".rodata", "a"
 	.align		4
diff --git a/arch/arm64/crypto/ghash-ce-core.S b/arch/arm64/crypto/ghash-ce-core.S
index 7868330dd54e..a69c1d4479db 100644
--- a/arch/arm64/crypto/ghash-ce-core.S
+++ b/arch/arm64/crypto/ghash-ce-core.S
@@ -350,13 +350,13 @@ CPU_LE(	rev64		T1.16b, T1.16b	)
 	 * void pmull_ghash_update(int blocks, u64 dg[], const char *src,
 	 *			   struct ghash_key const *k, const char *head)
 	 */
-SYM_FUNC_START(pmull_ghash_update_p64)
+SYM_FUNC_START_SECTION(pmull_ghash_update_p64)
 	__pmull_ghash	p64
-SYM_FUNC_END(pmull_ghash_update_p64)
+SYM_FUNC_END_SECTION(pmull_ghash_update_p64)
 
-SYM_FUNC_START(pmull_ghash_update_p8)
+SYM_FUNC_START_SECTION(pmull_ghash_update_p8)
 	__pmull_ghash	p8
-SYM_FUNC_END(pmull_ghash_update_p8)
+SYM_FUNC_END_SECTION(pmull_ghash_update_p8)
 
 	KS0		.req	v8
 	KS1		.req	v9
@@ -602,20 +602,20 @@ CPU_LE(	rev		w8, w8		)
 	 *			  struct ghash_key const *k, u64 dg[], u8 ctr[],
 	 *			  int rounds, u8 tag)
 	 */
-SYM_FUNC_START(pmull_gcm_encrypt)
+SYM_FUNC_START_SECTION(pmull_gcm_encrypt)
 	pmull_gcm_do_crypt	1
-SYM_FUNC_END(pmull_gcm_encrypt)
+SYM_FUNC_END_SECTION(pmull_gcm_encrypt)
 
 	/*
 	 * void pmull_gcm_decrypt(int blocks, u8 dst[], const u8 src[],
 	 *			  struct ghash_key const *k, u64 dg[], u8 ctr[],
 	 *			  int rounds, u8 tag)
 	 */
-SYM_FUNC_START(pmull_gcm_decrypt)
+SYM_FUNC_START_SECTION(pmull_gcm_decrypt)
 	pmull_gcm_do_crypt	0
-SYM_FUNC_END(pmull_gcm_decrypt)
+SYM_FUNC_END_SECTION(pmull_gcm_decrypt)
 
-SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
+SYM_FUNC_START_LOCAL_SECTION(pmull_gcm_ghash_4x)
 	movi		MASK.16b, #0xe1
 	shl		MASK.2d, MASK.2d, #57
 
@@ -696,9 +696,9 @@ SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
 	eor		XL.16b, XL.16b, T2.16b
 
 	ret
-SYM_FUNC_END(pmull_gcm_ghash_4x)
+SYM_FUNC_END_SECTION(pmull_gcm_ghash_4x)
 
-SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x)
+SYM_FUNC_START_LOCAL_SECTION(pmull_gcm_enc_4x)
 	ld1		{KS0.16b}, [x5]			// load upper counter
 	sub		w10, w8, #4
 	sub		w11, w8, #3
@@ -761,7 +761,7 @@ SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x)
 	eor		INP3.16b, INP3.16b, KS3.16b
 
 	ret
-SYM_FUNC_END(pmull_gcm_enc_4x)
+SYM_FUNC_END_SECTION(pmull_gcm_enc_4x)
 
 	.section	".rodata", "a"
 	.align		6
diff --git a/arch/arm64/crypto/nh-neon-core.S b/arch/arm64/crypto/nh-neon-core.S
index 51c0a534ef87..cb354d3f7e7b 100644
--- a/arch/arm64/crypto/nh-neon-core.S
+++ b/arch/arm64/crypto/nh-neon-core.S
@@ -62,7 +62,7 @@
  *
  * It's guaranteed that message_len % 16 == 0.
  */
-SYM_FUNC_START(nh_neon)
+SYM_FUNC_START_SECTION(nh_neon)
 
 	ld1		{K0.4s,K1.4s}, [KEY], #32
 	  movi		PASS0_SUMS.2d, #0
@@ -100,4 +100,4 @@ SYM_FUNC_START(nh_neon)
 	addp		T1.2d, PASS2_SUMS.2d, PASS3_SUMS.2d
 	st1		{T0.16b,T1.16b}, [HASH]
 	ret
-SYM_FUNC_END(nh_neon)
+SYM_FUNC_END_SECTION(nh_neon)
diff --git a/arch/arm64/crypto/poly1305-armv8.pl b/arch/arm64/crypto/poly1305-armv8.pl
index cbc980fb02e3..039e6a9ce68c 100644
--- a/arch/arm64/crypto/poly1305-armv8.pl
+++ b/arch/arm64/crypto/poly1305-armv8.pl
@@ -48,8 +48,12 @@ my ($h0,$h1,$h2,$r0,$r1,$s1,$t0,$t1,$d0,$d1,$d2) = map("x$_",(4..14));
 
 $code.=<<___;
 #ifndef __KERNEL__
+# define SYM_TEXT_SECTION()
+# define SYM_TEXT_END_SECTION
 # include "arm_arch.h"
 .extern	OPENSSL_armcap_P
+#else
+# include <linux/linkage.h>
 #endif
 
 .text
@@ -58,6 +62,7 @@ $code.=<<___;
 .globl	poly1305_blocks
 .globl	poly1305_emit
 
+SYM_TEXT_SECTION(poly1305_init)
 .globl	poly1305_init
 .type	poly1305_init,%function
 .align	5
@@ -107,7 +112,9 @@ poly1305_init:
 .Lno_key:
 	ret
 .size	poly1305_init,.-poly1305_init
+SYM_TEXT_END_SECTION
 
+SYM_TEXT_SECTION(poly1305_blocks)
 .type	poly1305_blocks,%function
 .align	5
 poly1305_blocks:
@@ -198,7 +205,9 @@ poly1305_blocks:
 .Lno_data:
 	ret
 .size	poly1305_blocks,.-poly1305_blocks
+SYM_TEXT_END_SECTION
 
+SYM_TEXT_SECTION(poly1305_emit)
 .type	poly1305_emit,%function
 .align	5
 poly1305_emit:
@@ -258,6 +267,7 @@ poly1305_emit:
 
 	ret
 .size	poly1305_emit,.-poly1305_emit
+SYM_TEXT_END_SECTION
 ___
 my ($R0,$R1,$S1,$R2,$S2,$R3,$S3,$R4,$S4) = map("v$_.4s",(0..8));
 my ($IN01_0,$IN01_1,$IN01_2,$IN01_3,$IN01_4) = map("v$_.2s",(9..13));
@@ -270,6 +280,7 @@ my ($in2,$zeros)=("x16","x17");
 my $is_base2_26 = $zeros;		# borrow
 
 $code.=<<___;
+SYM_TEXT_SECTION(poly1305_mult)
 .type	poly1305_mult,%function
 .align	5
 poly1305_mult:
@@ -306,7 +317,9 @@ poly1305_mult:
 
 	ret
 .size	poly1305_mult,.-poly1305_mult
+SYM_TEXT_END_SECTION
 
+SYM_TEXT_SECTION(poly1305_splat)
 .type	poly1305_splat,%function
 .align	4
 poly1305_splat:
@@ -333,7 +346,9 @@ poly1305_splat:
 
 	ret
 .size	poly1305_splat,.-poly1305_splat
+SYM_TEXT_END_SECTION
 
+SYM_TEXT_SECTION(poly1305_blocks_neon)
 #ifdef	__KERNEL__
 .globl	poly1305_blocks_neon
 #endif
@@ -888,6 +903,8 @@ poly1305_blocks_neon:
 .align	5
 .Lzeros:
 .long	0,0,0,0,0,0,0,0
+SYM_TEXT_END_SECTION
+
 .asciz	"Poly1305 for ARMv8, CRYPTOGAMS by \@dot-asm"
 .align	2
 #if !defined(__KERNEL__) && !defined(_WIN64)
diff --git a/arch/arm64/crypto/sha1-ce-core.S b/arch/arm64/crypto/sha1-ce-core.S
index 889ca0f8972b..2ba5f8ea39fc 100644
--- a/arch/arm64/crypto/sha1-ce-core.S
+++ b/arch/arm64/crypto/sha1-ce-core.S
@@ -65,7 +65,7 @@
 	 * int sha1_ce_transform(struct sha1_ce_state *sst, u8 const *src,
 	 *			 int blocks)
 	 */
-SYM_FUNC_START(sha1_ce_transform)
+SYM_FUNC_START_SECTION(sha1_ce_transform)
 	/* load round constants */
 	loadrc		k0.4s, 0x5a827999, w6
 	loadrc		k1.4s, 0x6ed9eba1, w6
@@ -147,4 +147,4 @@ CPU_LE(	rev32		v11.16b, v11.16b	)
 	str		dgb, [x0, #16]
 	mov		w0, w2
 	ret
-SYM_FUNC_END(sha1_ce_transform)
+SYM_FUNC_END_SECTION(sha1_ce_transform)
diff --git a/arch/arm64/crypto/sha2-ce-core.S b/arch/arm64/crypto/sha2-ce-core.S
index 491179922f49..6c1a4a128355 100644
--- a/arch/arm64/crypto/sha2-ce-core.S
+++ b/arch/arm64/crypto/sha2-ce-core.S
@@ -75,7 +75,7 @@
 	 *			  int blocks)
 	 */
 	.text
-SYM_FUNC_START(sha2_ce_transform)
+SYM_FUNC_START_SECTION(sha2_ce_transform)
 	/* load round constants */
 	adr_l		x8, .Lsha2_rcon
 	ld1		{ v0.4s- v3.4s}, [x8], #64
@@ -154,4 +154,4 @@ CPU_LE(	rev32		v19.16b, v19.16b	)
 3:	st1		{dgav.4s, dgbv.4s}, [x0]
 	mov		w0, w2
 	ret
-SYM_FUNC_END(sha2_ce_transform)
+SYM_FUNC_END_SECTION(sha2_ce_transform)
diff --git a/arch/arm64/crypto/sha3-ce-core.S b/arch/arm64/crypto/sha3-ce-core.S
index 9c77313f5a60..6105cc815c9a 100644
--- a/arch/arm64/crypto/sha3-ce-core.S
+++ b/arch/arm64/crypto/sha3-ce-core.S
@@ -40,7 +40,7 @@
 	 * int sha3_ce_transform(u64 *st, const u8 *data, int blocks, int dg_size)
 	 */
 	.text
-SYM_FUNC_START(sha3_ce_transform)
+SYM_FUNC_START_SECTION(sha3_ce_transform)
 	/* load state */
 	add	x8, x0, #32
 	ld1	{ v0.1d- v3.1d}, [x0]
@@ -197,7 +197,7 @@ SYM_FUNC_START(sha3_ce_transform)
 	st1	{v24.1d}, [x0]
 	mov	w0, w2
 	ret
-SYM_FUNC_END(sha3_ce_transform)
+SYM_FUNC_END_SECTION(sha3_ce_transform)
 
 	.section	".rodata", "a"
 	.align		8
diff --git a/arch/arm64/crypto/sha512-armv8.pl b/arch/arm64/crypto/sha512-armv8.pl
index 2d8655d5b1af..7952696d3c88 100644
--- a/arch/arm64/crypto/sha512-armv8.pl
+++ b/arch/arm64/crypto/sha512-armv8.pl
@@ -195,11 +195,16 @@ ___
 $code.=<<___;
 #ifndef	__KERNEL__
 # include "arm_arch.h"
+# define SYM_TEXT_SECTION()
+# define SYM_TEXT_END_SECTION
+#else
+# include <linux/linkage.h>
 #endif
 
 .text
 
 .extern	OPENSSL_armcap_P
+SYM_TEXT_SECTION($func)
 .globl	$func
 .type	$func,%function
 .align	6
@@ -285,7 +290,9 @@ $code.=<<___;
 	ldp	x29,x30,[sp],#128
 	ret
 .size	$func,.-$func
+SYM_TEXT_END_SECTION
 
+SYM_TEXT_SECTION(K$BITS)
 .align	6
 .type	.LK$BITS,%object
 .LK$BITS:
@@ -354,6 +361,8 @@ $code.=<<___ if ($SZ==4);
 ___
 $code.=<<___;
 .size	.LK$BITS,.-.LK$BITS
+SYM_TEXT_END_SECTION
+
 #ifndef	__KERNEL__
 .align	3
 .LOPENSSL_armcap_P:
@@ -637,6 +646,7 @@ sub body_00_15 () {
 }
 
 $code.=<<___;
+SYM_TEXT_SECTION(sha256_block_neon)
 #ifdef	__KERNEL__
 .globl	sha256_block_neon
 #endif
@@ -736,6 +746,7 @@ $code.=<<___;
 	add	sp,sp,#16*4+16
 	ret
 .size	sha256_block_neon,.-sha256_block_neon
+SYM_TEXT_END_SECTION
 ___
 }
 
diff --git a/arch/arm64/crypto/sha512-ce-core.S b/arch/arm64/crypto/sha512-ce-core.S
index b6a3a36e15f5..7d34aabb3daa 100644
--- a/arch/arm64/crypto/sha512-ce-core.S
+++ b/arch/arm64/crypto/sha512-ce-core.S
@@ -106,7 +106,7 @@
 	 *			  int blocks)
 	 */
 	.text
-SYM_FUNC_START(sha512_ce_transform)
+SYM_FUNC_START_SECTION(sha512_ce_transform)
 	/* load state */
 	ld1		{v8.2d-v11.2d}, [x0]
 
@@ -203,4 +203,4 @@ CPU_LE(	rev64		v19.16b, v19.16b	)
 3:	st1		{v8.2d-v11.2d}, [x0]
 	mov		w0, w2
 	ret
-SYM_FUNC_END(sha512_ce_transform)
+SYM_FUNC_END_SECTION(sha512_ce_transform)
diff --git a/arch/arm64/crypto/sm3-ce-core.S b/arch/arm64/crypto/sm3-ce-core.S
index ef97d3187cb7..7be60c41e36d 100644
--- a/arch/arm64/crypto/sm3-ce-core.S
+++ b/arch/arm64/crypto/sm3-ce-core.S
@@ -73,7 +73,7 @@
 	 *                       int blocks)
 	 */
 	.text
-SYM_FUNC_START(sm3_ce_transform)
+SYM_FUNC_START_SECTION(sm3_ce_transform)
 	/* load state */
 	ld1		{v8.4s-v9.4s}, [x0]
 	rev64		v8.4s, v8.4s
@@ -131,7 +131,7 @@ CPU_LE(	rev32		v3.16b, v3.16b		)
 	ext		v9.16b, v9.16b, v9.16b, #8
 	st1		{v8.4s-v9.4s}, [x0]
 	ret
-SYM_FUNC_END(sm3_ce_transform)
+SYM_FUNC_END_SECTION(sm3_ce_transform)
 
 	.section	".rodata", "a"
 	.align		3
diff --git a/arch/arm64/crypto/sm4-ce-core.S b/arch/arm64/crypto/sm4-ce-core.S
index 4ac6cfbc5797..5f64ed209a26 100644
--- a/arch/arm64/crypto/sm4-ce-core.S
+++ b/arch/arm64/crypto/sm4-ce-core.S
@@ -15,7 +15,7 @@
 	 * void sm4_ce_do_crypt(const u32 *rk, u32 *out, const u32 *in);
 	 */
 	.text
-SYM_FUNC_START(sm4_ce_do_crypt)
+SYM_FUNC_START_SECTION(sm4_ce_do_crypt)
 	ld1		{v8.4s}, [x2]
 	ld1		{v0.4s-v3.4s}, [x0], #64
 CPU_LE(	rev32		v8.16b, v8.16b		)
@@ -33,4 +33,4 @@ CPU_LE(	rev32		v8.16b, v8.16b		)
 CPU_LE(	rev32		v8.16b, v8.16b		)
 	st1		{v8.4s}, [x1]
 	ret
-SYM_FUNC_END(sm4_ce_do_crypt)
+SYM_FUNC_END_SECTION(sm4_ce_do_crypt)
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 19/22] module: Reorder functions
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (16 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 18/22] arm64/crypto: conditionally place ASM functions into separate sections Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 20/22] module: use a scripted approach for FG-KASLR Alexander Lobakin
                   ` (4 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux, kernel test robot

From: Kristen Carlson Accardi <kristen@linux.intel.com>

Introduce a new config option to allow modules to be re-ordered
by function. This option can be enabled independently of the
kernel text KASLR or FG_KASLR settings so that it can be used
by architectures that do not support either of these features.
This option will be selected by default if CONFIG_FG_KASLR is
selected.

If a module has functions split out into separate text sections
(i.e. compiled with the -ffunction-sections flag), reorder the
functions to provide some code diversification to modules.

Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Tested-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Tested-by: Tony Luck <tony.luck@intel.com>
Acked-by: Jessica Yu <jeyu@kernel.org>
Tested-by: Jessica Yu <jeyu@kernel.org>
Reported-by: kernel test robot <lkp@intel.com> # swap.cocci
[ alobakin: make it work with ClangCFI ]
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 Makefile                |  4 ++
 init/Kconfig            | 12 ++++++
 kernel/kallsyms.c       |  4 +-
 kernel/livepatch/core.c |  2 +-
 kernel/module.c         | 91 ++++++++++++++++++++++++++++++++++++++++-
 5 files changed, 108 insertions(+), 5 deletions(-)

diff --git a/Makefile b/Makefile
index 283876e170f7..2bde66addb89 100644
--- a/Makefile
+++ b/Makefile
@@ -921,6 +921,10 @@ endif
 # ClangLTO implies -ffunction-sections -fdata-sections, no need
 # to specify them manually and trigger a pointless full rebuild
 ifndef CONFIG_LTO_CLANG
+ifdef CONFIG_MODULE_FG_KASLR
+KBUILD_CFLAGS_MODULE += -ffunction-sections
+endif
+
 ifneq ($(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION)$(CONFIG_FG_KASLR),)
 KBUILD_CFLAGS_KERNEL += -ffunction-sections
 endif
diff --git a/init/Kconfig b/init/Kconfig
index e72633f4f8a9..e8158c256ee9 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2330,6 +2330,18 @@ config UNUSED_KSYMS_WHITELIST
 	  one per line. The path can be absolute, or relative to the kernel
 	  source tree.
 
+config MODULE_FG_KASLR
+	bool "Module Function Granular Layout Randomization"
+	default FG_KASLR
+	depends on BROKEN
+	help
+	  This option randomizes the module text section by reordering the text
+	  section by function at module load time. In order to use this
+	  feature, the module must have been compiled with the
+	  -ffunction-sections compiler flag.
+
+	  If unsure, say N.
+
 endif # MODULES
 
 config MODULES_TREE_LOOKUP
diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c
index 5ffdcc2fb88e..6906cc726149 100644
--- a/kernel/kallsyms.c
+++ b/kernel/kallsyms.c
@@ -841,7 +841,7 @@ static int __kallsyms_open(struct inode *inode, struct file *file)
  * When function granular kaslr is enabled, we need to print out the symbols
  * at random so we don't reveal the new layout.
  */
-#ifdef CONFIG_FG_KASLR
+#if defined(CONFIG_FG_KASLR) || defined(CONFIG_MODULE_FG_KASLR)
 static int update_random_pos(struct kallsyms_shuffled_iter *s_iter,
 			     loff_t pos, loff_t *new_pos)
 {
@@ -985,7 +985,7 @@ static int kallsyms_random_open(struct inode *inode, struct file *file)
 #define kallsyms_open kallsyms_random_open
 #else
 #define kallsyms_open __kallsyms_open
-#endif /* !CONFIG_FG_KASLR */
+#endif /* !CONFIG_FG_KASLR && !CONFIG_MODULE_FG_KASLR */
 
 #ifdef	CONFIG_KGDB_KDB
 const char *kdb_walk_kallsyms(loff_t *pos)
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 852bbfa9da7b..15c4cd25592d 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -177,7 +177,7 @@ static int klp_find_object_symbol(const char *objname, const char *name,
 	 * force the algorithm to require that only unique symbols are
 	 * allowed to be patched.
 	 */
-	if (IS_ENABLED(CONFIG_FG_KASLR))
+	if (IS_ENABLED(CONFIG_FG_KASLR) || IS_ENABLED(CONFIG_MODULE_FG_KASLR))
 		sympos = 0;
 
 	/*
diff --git a/kernel/module.c b/kernel/module.c
index ed13917ea5f3..08747e5f2442 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -57,6 +57,7 @@
 #include <linux/bsearch.h>
 #include <linux/dynamic_debug.h>
 #include <linux/audit.h>
+#include <linux/random.h>
 #include <uapi/linux/module.h>
 #include "module-internal.h"
 
@@ -1527,7 +1528,7 @@ static void free_sect_attrs(struct module_sect_attrs *sect_attrs)
 
 	for (section = 0; section < sect_attrs->nsections; section++)
 		kfree(sect_attrs->attrs[section].battr.attr.name);
-	kfree(sect_attrs);
+	kvfree(sect_attrs);
 }
 
 static void add_sect_attrs(struct module *mod, const struct load_info *info)
@@ -1544,7 +1545,7 @@ static void add_sect_attrs(struct module *mod, const struct load_info *info)
 	size[0] = ALIGN(struct_size(sect_attrs, attrs, nloaded),
 			sizeof(sect_attrs->grp.bin_attrs[0]));
 	size[1] = (nloaded + 1) * sizeof(sect_attrs->grp.bin_attrs[0]);
-	sect_attrs = kzalloc(size[0] + size[1], GFP_KERNEL);
+	sect_attrs = kvzalloc(size[0] + size[1], GFP_KERNEL);
 	if (sect_attrs == NULL)
 		return;
 
@@ -2416,6 +2417,89 @@ static bool module_init_layout_section(const char *sname)
 	return module_init_section(sname);
 }
 
+/*
+ * shuffle_text_list()
+ * Use a Fisher Yates algorithm to shuffle a list of text sections.
+ */
+static void shuffle_text_list(Elf_Shdr **list, int size)
+{
+	u32 i, j;
+
+	for (i = size - 1; i > 0; i--) {
+		/*
+		 * pick a random index from 0 to i
+		 */
+		j = get_random_u32() % (i + 1);
+
+		swap(list[i], list[j]);
+	}
+}
+
+/*
+ * randomize_text()
+ * Look through the core section looking for executable code sections.
+ * Store sections in an array and then shuffle the sections
+ * to reorder the functions.
+ */
+static void randomize_text(struct module *mod, struct load_info *info)
+{
+	int max_sections = info->hdr->e_shnum;
+	int num_text_sections = 0;
+	Elf_Shdr **text_list;
+	int i, size;
+
+	text_list = kvmalloc_array(max_sections, sizeof(*text_list), GFP_KERNEL);
+	if (!text_list)
+		return;
+
+	for (i = 0; i < max_sections; i++) {
+		Elf_Shdr *shdr = &info->sechdrs[i];
+		const char *sname = info->secstrings + shdr->sh_name;
+
+		if (!(shdr->sh_flags & SHF_ALLOC) ||
+		    !(shdr->sh_flags & SHF_EXECINSTR) ||
+		    (shdr->sh_flags & ARCH_SHF_SMALL) ||
+		    module_init_layout_section(sname))
+			continue;
+
+		/*
+		 * With CONFIG_CFI_CLANG, .text with __cfi_check() must come
+		 * before any other text sections, and be aligned to PAGE_SIZE.
+		 * Don't include it in the shuffle list.
+		 */
+		if (IS_ENABLED(CONFIG_CFI_CLANG) && !strcmp(sname, ".text"))
+			continue;
+
+		if (!num_text_sections)
+			size = shdr->sh_entsize;
+
+		text_list[num_text_sections] = shdr;
+		num_text_sections++;
+	}
+
+	if (!num_text_sections)
+		goto exit;
+
+	shuffle_text_list(text_list, num_text_sections);
+
+	for (i = 0; i < num_text_sections; i++) {
+		Elf_Shdr *shdr = text_list[i];
+
+		/*
+		 * get_offset has a section index for it's last
+		 * argument, that is only used by arch_mod_section_prepend(),
+		 * which is only defined by parisc. Since this type
+		 * of randomization isn't supported on parisc, we can
+		 * safely pass in zero as the last argument, as it is
+		 * ignored.
+		 */
+		shdr->sh_entsize = get_offset(mod, &size, shdr, 0);
+	}
+
+exit:
+	kvfree(text_list);
+}
+
 /*
  * Lay out the SHF_ALLOC sections in a way not dissimilar to how ld
  * might -- code, read-only data, read-write data, small data.  Tally
@@ -2510,6 +2594,9 @@ static void layout_sections(struct module *mod, struct load_info *info)
 			break;
 		}
 	}
+
+	if (IS_ENABLED(CONFIG_MODULE_FG_KASLR))
+		randomize_text(mod, info);
 }
 
 static void set_license(struct module *mod, const char *license)
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 20/22] module: use a scripted approach for FG-KASLR
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (17 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 19/22] module: Reorder functions Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 21/22] Documentation: add a documentation " Alexander Lobakin
                   ` (3 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

Use the same methods and scripts to generate an LD script for every
module containing all the output text sections.
The only difference there is that we don't need to reserve any space
as the memory for every section is being allocated dynamically.

Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 include/asm-generic/vmlinux.lds.h | 12 ++++++++++++
 init/Kconfig                      | 15 ++++++++++++++-
 scripts/Makefile.modfinal         | 19 ++++++++++++++++---
 scripts/generate_text_sections.pl |  7 ++++++-
 scripts/module.lds.S              | 14 +++++++++++++-
 5 files changed, 61 insertions(+), 6 deletions(-)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 70fac18c786e..561f3ef06745 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -127,6 +127,18 @@
 #define TEXT_MAIN		.text
 #endif
 
+/*
+ * Same for modules. However, LD_DEAD_CODE_DATA_ELIMINATION doesn't touch
+ * them, so no need to check for it here.
+ */
+#if defined(CONFIG_LTO_CLANG) && !defined(CONFIG_MODULE_FG_KASLR)
+#define TEXT_MAIN_MODULE	.text .text.[0-9a-zA-Z_]*
+#elif defined(CONFIG_MODULE_FG_KASLR)
+#define TEXT_MAIN_MODULE	.text.__unused__
+#else
+#define TEXT_MAIN_MODULE	.text
+#endif
+
 /*
  * Used by scripts/generate_text_sections.pl to inject text sections,
  * harmless if FG-KASLR is disabled.
diff --git a/init/Kconfig b/init/Kconfig
index e8158c256ee9..8e0b5973fb72 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2333,7 +2333,6 @@ config UNUSED_KSYMS_WHITELIST
 config MODULE_FG_KASLR
 	bool "Module Function Granular Layout Randomization"
 	default FG_KASLR
-	depends on BROKEN
 	help
 	  This option randomizes the module text section by reordering the text
 	  section by function at module load time. In order to use this
@@ -2342,6 +2341,20 @@ config MODULE_FG_KASLR
 
 	  If unsure, say N.
 
+config MODULE_FG_KASLR_SHIFT
+	int "Module FG-KASLR granularity (functions per section shift)"
+	depends on MODULE_FG_KASLR
+	range 0 16
+	default 0
+	help
+	  This sets the number of functions that will be put in each section
+	  as a power of two.
+	  Decreasing the value increases the randomization, but also increases
+	  the size of the final kernel module due to the amount of sections.
+	  0 means that a separate section will be created for each function.
+	  16 almost disables the randomization, leaving only the manual
+	  separation.
+
 endif # MODULES
 
 config MODULES_TREE_LOOKUP
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index ff805777431c..ac1b8415519f 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -28,13 +28,24 @@ quiet_cmd_cc_o_c = CC [M]  $@
 %.mod.o: %.mod.c FORCE
 	$(call if_changed_dep,cc_o_c)
 
+ifdef CONFIG_MODULE_FG_KASLR
+quiet_cmd_gen_modules_lds = GEN [M] $@
+      cmd_gen_modules_lds =						\
+	$(PERL) $(srctree)/scripts/generate_text_sections.pl		\
+		-s $(CONFIG_MODULE_FG_KASLR_SHIFT) $(filter %.o, $^)	\
+		< $(filter %.lds, $^) > $@
+
+%.lds: %$(mod-prelink-ext).o scripts/module.lds FORCE
+	$(call if_changed,gen_modules_lds)
+endif
+
 ARCH_POSTLINK := $(wildcard $(srctree)/arch/$(SRCARCH)/Makefile.postlink)
 
 quiet_cmd_ld_ko_o = LD [M]  $@
       cmd_ld_ko_o +=							\
 	$(LD) -r $(KBUILD_LDFLAGS)					\
 		$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE)		\
-		-T scripts/module.lds -o $@ $(filter %.o, $^);		\
+		-T $(filter %.lds, $^) -o $@ $(filter %.o, $^);		\
 	$(if $(ARCH_POSTLINK), $(MAKE) -f $(ARCH_POSTLINK) $@, true)
 
 quiet_cmd_btf_ko = BTF [M] $@
@@ -55,13 +66,15 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check),      \
 
 
 # Re-generate module BTFs if either module's .ko or vmlinux changed
-$(modules): %.ko: %$(mod-prelink-ext).o %.mod.o scripts/module.lds $(if $(KBUILD_BUILTIN),vmlinux) FORCE
+$(modules): %.ko: %$(mod-prelink-ext).o %.mod.o
+$(modules): %.ko: $(if $(CONFIG_MODULE_FG_KASLR),%.lds,scripts/module.lds)
+$(modules): %.ko: $(if $(KBUILD_BUILTIN),vmlinux) FORCE
 	+$(call if_changed_except,ld_ko_o,vmlinux)
 ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 	+$(if $(newer-prereqs),$(call cmd,btf_ko))
 endif
 
-targets += $(modules) $(modules:.ko=.mod.o)
+targets += $(modules) $(modules:.ko=.mod.o) $(if $(CONFIG_MODULE_FG_KASLR),$(modules:.ko=.lds))
 
 # Add FORCE to the prequisites of a target to force it to be always rebuilt.
 # ---------------------------------------------------------------------------
diff --git a/scripts/generate_text_sections.pl b/scripts/generate_text_sections.pl
index 5f3ece2ee0ea..d5b16057b9ff 100755
--- a/scripts/generate_text_sections.pl
+++ b/scripts/generate_text_sections.pl
@@ -44,6 +44,7 @@ my $readelf = $ENV{'READELF'} || die "$0: ERROR: READELF not set?";
 
 ## text sections array
 my @sections = ();
+my $vmlinux = 0;
 
 ## max alignment found to reserve some space
 my $max_align = 64;
@@ -64,6 +65,10 @@ sub read_sections {
 			next;
 		}
 
+		if ($name eq ".sched.text") {
+			$vmlinux = 1;
+		}
+
 		if (!($name =~ /^\.text\.[0-9a-zA-Z_]*((\.constprop|\.isra|\.part)\.[0-9])*(|\.[0-9cfi]*)$/)) {
 			next;
 		}
@@ -120,7 +125,7 @@ sub print_reserve {
 	## we reserve some space for them to not overlap _etext while shuffling
 	## sections
 
-	if (!$count) {
+	if (!$vmlinux or !$count) {
 		return;
 	}
 
diff --git a/scripts/module.lds.S b/scripts/module.lds.S
index 04c5685c25cf..f32437783edf 100644
--- a/scripts/module.lds.S
+++ b/scripts/module.lds.S
@@ -3,6 +3,11 @@
  * Archs are free to supply their own linker scripts.  ld will
  * combine them automatically.
  */
+
+#include <asm-generic/vmlinux.lds.h>
+
+#undef SANITIZER_DISCARDS
+
 #ifdef CONFIG_CFI_CLANG
 # include <asm/page.h>
 # define ALIGN_CFI 		ALIGN(PAGE_SIZE)
@@ -57,9 +62,16 @@ SECTIONS {
 	 */
 	.text : ALIGN_CFI {
 		*(.text.__cfi_check)
-		*(.text .text.[0-9a-zA-Z_]* .text..L.cfi*)
+		*(TEXT_MAIN_MODULE)
+		*(.text..L.cfi.jumptable .text..L.cfi.jumptable.*)
+	}
+#elif defined(CONFIG_MODULE_FG_KASLR)
+	.text : {
+		*(TEXT_MAIN_MODULE)
 	}
 #endif
+
+	TEXT_FG_KASLR
 }
 
 /* bring in arch-specific sections */
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 21/22] Documentation: add a documentation for FG-KASLR
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (18 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 20/22] module: use a scripted approach for FG-KASLR Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 14:41 ` [PATCH v6 kspp-next 22/22] maintainers: add MAINTAINERS entry " Alexander Lobakin
                   ` (2 subsequent siblings)
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

From: Kristen Carlson Accardi <kristen@linux.intel.com>

Describe the main principles behind the FG-KASLR hardening feature
in a new doc section.

Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 .../admin-guide/kernel-parameters.txt         |   6 +
 Documentation/security/fgkaslr.rst            | 172 ++++++++++++++++++
 Documentation/security/index.rst              |   1 +
 3 files changed, 179 insertions(+)
 create mode 100644 Documentation/security/fgkaslr.rst

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index bdb22006f713..f63175a13147 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -2211,6 +2211,12 @@
 			kernel and module base offset ASLR (Address Space
 			Layout Randomization).
 
+	nofgkaslr	[KNL]
+			When CONFIG_FG_KASLR is set, this parameter
+			disables kernel function granular ASLR
+			(Address Space Layout Randomization).
+			See Documentation/security/fgkaslr.rst.
+
 	kasan_multi_shot
 			[KNL] Enforce KASAN (Kernel Address Sanitizer) to print
 			report on every invalid memory access. Without this
diff --git a/Documentation/security/fgkaslr.rst b/Documentation/security/fgkaslr.rst
new file mode 100644
index 000000000000..50dc24f675b5
--- /dev/null
+++ b/Documentation/security/fgkaslr.rst
@@ -0,0 +1,172 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=====================================================================
+Function Granular Kernel Address Space Layout Randomization (fgkaslr)
+=====================================================================
+
+:Date: 6 April 2020
+:Author: Kristen Accardi
+
+Kernel Address Space Layout Randomization (KASLR) was merged into the kernel
+with the objective of increasing the difficulty of code reuse attacks. Code
+reuse attacks reused existing code snippets to get around existing memory
+protections. They exploit software bugs which expose addresses of useful code
+snippets to control the flow of execution for their own nefarious purposes.
+KASLR as it was originally implemented moves the entire kernel code text as a
+unit at boot time in order to make addresses less predictable. The order of the
+code within the segment is unchanged - only the base address is shifted. There
+are a few shortcomings to this algorithm.
+
+1. Low Entropy - there are only so many locations the kernel can fit in. This
+   means an attacker could guess without too much trouble.
+2. Knowledge of a single address can reveal the offset of the base address,
+   exposing all other locations for a published/known kernel image.
+3. Info leaks abound.
+
+Finer grained ASLR has been proposed as a way to make ASLR more resistant
+to info leaks. It is not a new concept at all, and there are many variations
+possible. Function reordering is an implementation of finer grained ASLR
+which randomizes the layout of an address space on a function level
+granularity. The term "fgkaslr" is used in this document to refer to the
+technique of function reordering when used with KASLR, as well as finer grained
+KASLR in general.
+
+The objective of this patch set is to improve a technology that is already
+merged into the kernel (KASLR). This code will not prevent all code reuse
+attacks, and should be considered as one of several tools that can be used.
+
+Implementation Details
+======================
+
+The over-arching objective of the fgkaslr implementation is incremental
+improvement over the existing KASLR algorithm. It is designed to work with
+the existing solution, and there are two main area where code changes occur:
+Build time, and Load time.
+
+Build time
+----------
+
+GCC has had an option to place functions into individual .text sections
+for many years now (-ffunction-sections). This option is used to implement
+function reordering at load time. The final compiled vmlinux retains all the
+section headers, which can be used to help find the address ranges of each
+function. Using this information and an expanded table of relocation addresses,
+individual text sections can be shuffled immediately after decompression.
+Some data tables inside the kernel that have assumptions about order
+require sorting after the update. In order to modify these tables,
+a few key symbols from the objcopy symbol stripping process are preserved
+for use after shuffling the text segments. Any special input sections which are
+defined by the kernel build process and collected into the .text output
+segment are left unmodified and will still be present inside the .text segment,
+unrandomized other than normal base address randomization.
+
+Load time
+---------
+
+The boot kernel was modified to parse the vmlinux elf file after
+decompression to check for symbols for modifying data tables, and to
+look for any .text.* sections to randomize. The sections are then shuffled,
+and tables are updated or resorted. The existing code which updated relocation
+addresses was modified to account for not just a fixed delta from the load
+address, but the offset that the function section was moved to. This requires
+inspection of each address to see if it was impacted by a randomization.
+
+In order to hide the new layout, symbols reported through /proc/kallsyms will
+be displayed in a random order.
+
+Performance Impact
+==================
+
+There are two areas where function reordering can impact performance: boot
+time latency, and run time performance.
+
+Boot time latency
+-----------------
+
+This implementation of finer grained KASLR impacts the boot time of the kernel
+in several places. It requires additional parsing of the kernel ELF file to
+obtain the section headers of the sections to be randomized. It calls the
+random number generator for each section to be randomized to determine that
+section's new memory location. It copies the decompressed kernel into a new
+area of memory to avoid corruption when laying out the newly randomized
+sections. It increases the number of relocations the kernel has to perform at
+boot time vs. standard KASLR, and it also requires a lookup on each address
+that needs to be relocated to see if it was in a randomized section and needs
+to be adjusted by a new offset. Finally, it re-sorts a few data tables that
+are required to be sorted by address.
+
+Booting a test VM on a modern, well appointed system showed an increase in
+latency of approximately 1 second.
+
+Run time
+--------
+
+The performance impact at run-time of function reordering varies by workload.
+Randomly reordering the functions will cause an increase in cache misses
+for some workloads. Some workloads perform significantly worse under FGKASLR,
+while others stay the same or even improve. In general, it will depend on the
+code flow whether or not finer grained KASLR will impact a workload, and how
+the underlying code was designed. Because the layout changes per boot, each
+time a system is rebooted the performance of a workload may change.
+
+Image Size
+==========
+
+fgkaslr increases the size of the kernel binary due to the extra section
+headers that are included, as well as the extra relocations that need to
+be added. You can expect fgkaslr to increase the size of the resulting
+vmlinux by about 3%, and the compressed image (bzImage) by 15%.
+
+Memory Usage
+============
+
+fgkaslr increases the amount of heap that is required at boot time,
+although this extra memory is released when the kernel has finished
+decompression. As a result, it may not be appropriate to use this feature
+on systems without much memory.
+
+Building
+========
+
+To enable fine grained KASLR, you need to have the following config options
+set (including all the ones you would use to build normal KASLR)
+
+``CONFIG_FG_KASLR=y``
+
+fgkaslr for the kernel is only supported for the X86_64 architecture.
+
+Modules
+=======
+
+Modules are randomized similarly to the rest of the kernel by shuffling
+the sections at load time prior to moving them into memory. The module must
+also have been build with the -ffunction-sections compiler option.
+
+Although fgkaslr for the kernel is only supported for the X86_64 architecture,
+it is possible to use fgkaslr with modules on other architectures. To enable
+this feature, select the following config option:
+
+``CONFIG_MODULE_FG_KASLR``
+
+This option is selected automatically for X86_64 when CONFIG_FG_KASLR is set.
+
+Disabling
+=========
+
+Disabling normal kaslr using the nokaslr command line option also disables
+fgkaslr. In addition, it is possible to disable fgkaslr separately by booting
+with "nofgkaslr" on the commandline.
+
+Further Information
+===================
+
+There are a lot of academic papers which explore finer grained ASLR.
+This paper in particular contributed significantly to the implementation design.
+
+Selfrando: Securing the Tor Browser against De-anonymization Exploits,
+M. Conti, S. Crane, T. Frassetto, et al.
+
+For more information on how function layout impacts performance, see:
+
+Optimizing Function Placement for Large-Scale Data-Center Applications,
+G. Ottoni, B. Maher
diff --git a/Documentation/security/index.rst b/Documentation/security/index.rst
index 16335de04e8c..41444124090f 100644
--- a/Documentation/security/index.rst
+++ b/Documentation/security/index.rst
@@ -7,6 +7,7 @@ Security Documentation
 
    credentials
    IMA-templates
+   fgkaslr
    keys/index
    lsm
    lsm-development
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* [PATCH v6 kspp-next 22/22] maintainers: add MAINTAINERS entry for FG-KASLR
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (19 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 21/22] Documentation: add a documentation " Alexander Lobakin
@ 2021-08-31 14:41 ` Alexander Lobakin
  2021-08-31 17:27 ` [PATCH v6 kspp-next 00/22] Function Granular KASLR Kees Cook
  2021-09-01 17:16 ` Accardi, Kristen C
  22 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-08-31 14:41 UTC (permalink / raw)
  To: linux-hardening
  Cc: Kristen C Accardi, Kristen Carlson Accardi, Kees Cook,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	Alexander Lobakin, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

Add an entry for FG-KASLR containing the maintainers, reviewers,
public mailing lists, files and so on.

Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
---
 MAINTAINERS | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index d7b4f32875a9..9040d17e6e70 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7637,6 +7637,18 @@ L:	platform-driver-x86@vger.kernel.org
 S:	Maintained
 F:	drivers/platform/x86/fujitsu-tablet.c
 
+FUNCTION-GRAINED KASLR (FG-KASLR)
+M:	Alexander Lobakin <alexandr.lobakin@intel.com>
+R:	Kristen Carlson Accardi <kristen@linux.intel.com>
+R:	Kees Cook <keescook@chromium.org>
+L:	linux-hardening@vger.kernel.org
+S:	Supported
+F:	Documentation/security/fgkaslr.rst
+F:	arch/x86/boot/compressed/fgkaslr.c
+F:	arch/x86/boot/compressed/utils.c
+F:	arch/x86/boot/compressed/vmlinux.symbols
+F:	scripts/generate_text_sections.pl
+
 FUSE: FILESYSTEM IN USERSPACE
 M:	Miklos Szeredi <miklos@szeredi.hu>
 L:	linux-fsdevel@vger.kernel.org
-- 
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v6 kspp-next 00/22] Function Granular KASLR
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (20 preceding siblings ...)
  2021-08-31 14:41 ` [PATCH v6 kspp-next 22/22] maintainers: add MAINTAINERS entry " Alexander Lobakin
@ 2021-08-31 17:27 ` Kees Cook
  2021-09-01 10:36   ` Alexander Lobakin
  2021-09-01 17:16 ` Accardi, Kristen C
  22 siblings, 1 reply; 29+ messages in thread
From: Kees Cook @ 2021-08-31 17:27 UTC (permalink / raw)
  To: Alexander Lobakin
  Cc: linux-hardening, Kristen C Accardi, Kristen Carlson Accardi,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	linux-kbuild, linux-arch, linux-kernel, clang-built-linux

On Tue, Aug 31, 2021 at 04:40:52PM +0200, Alexander Lobakin wrote:
> This is a massive rework and a respin of Kristen Accardi's marvellous
> FG-KASLR series (v5).

Thanks for working on this! I know Marios has been looking at some of
this as well. I think he tracked down a kretprobes bug and has a fixed
prepared.

> The major differences since v5 [0]:
> * You can now tune the number of functions per each section to
>   achieve the preferable vmlinux size or protection level. Default
>   is still as one section per each function.
>   This can be handy for storage-constrained systems. 4-8 fps are
>   still strong, but reduce the size of the final vmlinu{x,z}
>   significantly;

Interesting, but I'm not sure what the size issue is. v5's on-disk
image size issues were related to the large relocation table that was
used during decompress and layout, but would get discarded. The final
in-core image size was roughly the same size as a non-FGKASLR kernel
(since functions were already aligned even without -ffunction-sections).
How does the functions-per-section knob change image size?

> * I don't use orphan sections anymore. It's not reliable at all /
>   may differ from linker to linker, and also conflicts with
>   CONFIG_LD_ORPHAN_WARN which is great for catching random bugs ->
> * All the .text.* sections are now being described explicitly in the
>   linker script. A Perl script is used to take the original LDS, the
>   original object file, read a list of input sections from it and
>   generate the resulting LDS.
>   This costs a bit of linking time as LD tends to think hard when
>   processing scripts > 1 Mb. It adds about 40-60 seconds to the
>   whole linking process (BTF step, 2-3 kallsyms steps and the final
>   step), but "better safe than sorry".
>   In addition, that approach allows to reserve some space at the end
>   and add some link assertions ->

Yeah, this "hope that orphan handling does it right" bugged me too, but my
attempts to solve it looked much like yours: creating a linker file that
named all the sections. I found this to be prohibitively expensive at link
time (and that seems backed by your own measurements of an extra minute
or so at link time). If that's still the result of using a generated
linker file, we just need to depend on orphan handling. LD_ORPHAN_WARN
will still exist for non-FGKASLR builds, so the benefits will continue
to exist -- I think the correct solution is to have the linker grow a
"pass through" special target like "DISCARD", which just maps given
input section patterns into same-named output sections.

> * Input .text section now must be empty, otherwise the linkage will
>   be stopped. This is implemented by the size assertion in the
>   resulting LD script and is designed to plug the potentional layout
>   leakage. This also means that ->

I worry this will create unexpected problems for named sections that
weren't originally being randomized with the v5 FGKASLR.

> * "Regular" ASM functions are now being placed into unique separate
>   functions the same way compiler does this for C functions. This is
>   achieved by introducing and using several new macros which take
>   the symbol name as a base for its new section name.
>   This gives a better opportunity to both DCE and FG-KASLR, as ASM
>   code now can also be randomized or garbage-collected;

This is interesting! I think it'd be a good evolutionary step on top of
"basic FGKASLR".

> * It's now fully compatible with ClangLTO, ClangCFI,
>   CONFIG_LD_ORPHAN_WARN and some more stuff landed since the last
>   revision was published;

FWIW, v5 was was too. :) I didn't have to do anything to v5 to make it
work with ClangLTO and ClangCFI.

> * Includes several fixes: relocations inside .altinstr_replacement
>   code and minor issues found and/or suggested by LKP robot.

Excellent!

> The series was compile-time and runtime tested on the following
> setups with no issues:
> - x86_64, GCC 11, Binutils 2.35;
> - x86_64, Clang/LLVM 12, ClangLTO + ClangCFI (from Sami's tree).

Great, this is a good start. One place we saw problems in the past was
with i386 build gotchas, so that'll need testing too.

> The first 4 patches are from the linux-kbuild tree and included
> to avoid merge conflicts and non-intuitive resolving of them.

Sounds good. It might be easier to base the series on linux-next, so a
smaller series. Though given the merge window just opened, it might make
more sense for a v7 to be based on v5.15-rc2 in three weeks.

> The series is also available here: [1]
> 
> [0] https://lore.kernel.org/kernel-hardening/20200923173905.11219-1-kristen@linux.intel.com
> [1] https://github.com/alobakin/linux/pull/3
> 
> The original v5 cover letter:

More notes below...

> 
> Function Granular Kernel Address Space Layout Randomization (fgkaslr)
> ---------------------------------------------------------------------
> 
> This patch set is an implementation of finer grained kernel address space
> randomization. It rearranges your kernel code at load time 
> on a per-function level granularity, with only around a second added to
> boot time.
> 
> Changes in v5:
> --------------
> * fixed a bug in the code which increases boot heap size for
>   CONFIG_FG_KASLR which prevented the boot heap from being increased
>   for CONFIG_FG_KASLR when using bzip2 compression. Thanks to Andy Lavr
>   for finding the problem and identifying the solution.
> * changed the adjustment of the orc_unwind_ip table at boot time to
>   disregard relocs associated with this table, and instead inspect the
>   entries separately. Relocs are not able to be used since they are
>   no longer correct once the table is resorted at buildtime.
> * changed how orc_unwind_ip addresses in randomized sections are identified
>   to include the byte immediately after the end of the section.
> * updated module code to use kvmalloc/kvfree based on suggestions from
>   Evgenii Shatokhin <eshatokhin@virtuozzo.com>.
> * changed kernel commandline to disable fgkaslr to simply "nofgkaslr" to
>   match the nokaslr option. fgkaslr="X" can be added at a later date
>   if it is needed.
> * Added a patch to force livepatch to require symbols to be unique if
>   using while fgkaslr either for core or modules.
> 
> Changes in v4:
> -------------
> * dropped the patch to split out change to STATIC definition in
>   x86/boot/compressed/misc.c and replaced with a patch authored
>   by Kees Cook to avoid the duplicate malloc definitions
> * Added a section to Documentation/admin-guide/kernel-parameters.txt
>   to document the fgkaslr boot option.
> * redesigned the patch to hide the new layout when reading
>   /proc/kallsyms. The previous implementation utilized a dynamically
>   allocated linked list to display the kernel and module symbols
>   in alphabetical order. The new implementation uses a randomly
>   shuffled index array to display the kernel and module symbols
>   in a random order.
> 
> Changes in v3:
> -------------
> * Makefile changes to accommodate CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
> * removal of extraneous ALIGN_PAGE from _etext changes
> * changed variable names in x86/tools/relocs to be less confusing
> * split out change to STATIC definition in x86/boot/compressed/misc.c
> * Updates to Documentation to make it more clear what is preserved in .text
> * much more detailed commit message for function granular KASLR patch
> * minor tweaks and changes that make for more readable code
> * this cover letter updated slightly to add additional details
> 
> Changes in v2:
> --------------
> * Fix to address i386 build failure
> * Allow module reordering patch to be configured separately so that
>   arm (or other non-x86_64 arches) can take advantage of module function
>   reordering. This support has not be tested by me, but smoke tested by
>   Ard Biesheuvel <ardb@kernel.org> on arm.
> * Fix build issue when building on arm as reported by
>   Ard Biesheuvel <ardb@kernel.org> 
> 
> Patches to objtool are included because they are dependencies for this
> patchset, however they have been submitted by their maintainer separately.
> 
> Background
> ----------
> KASLR was merged into the kernel with the objective of increasing the
> difficulty of code reuse attacks. Code reuse attacks reused existing code
> snippets to get around existing memory protections. They exploit software bugs
> which expose addresses of useful code snippets to control the flow of
> execution for their own nefarious purposes. KASLR moves the entire kernel
> code text as a unit at boot time in order to make addresses less predictable.
> The order of the code within the segment is unchanged - only the base address
> is shifted. There are a few shortcomings to this algorithm.
> 
> 1. Low Entropy - there are only so many locations the kernel can fit in. This
>    means an attacker could guess without too much trouble.
> 2. Knowledge of a single address can reveal the offset of the base address,
>    exposing all other locations for a published/known kernel image.
> 3. Info leaks abound.
> 
> Finer grained ASLR has been proposed as a way to make ASLR more resistant
> to info leaks. It is not a new concept at all, and there are many variations
> possible. Function reordering is an implementation of finer grained ASLR
> which randomizes the layout of an address space on a function level
> granularity. We use the term "fgkaslr" in this document to refer to the
> technique of function reordering when used with KASLR, as well as finer grained
> KASLR in general.
> 
> Proposed Improvement
> --------------------
> This patch set proposes adding function reordering on top of the existing
> KASLR base address randomization. The over-arching objective is incremental
> improvement over what we already have. It is designed to work in combination
> with the existing solution. The implementation is really pretty simple, and
> there are 2 main area where changes occur:
> 
> * Build time
> 
> GCC has had an option to place functions into individual .text sections for
> many years now. This option can be used to implement function reordering at
> load time. The final compiled vmlinux retains all the section headers, which
> can be used to help find the address ranges of each function. Using this
> information and an expanded table of relocation addresses, individual text
> sections can be suffled immediately after decompression. Some data tables
> inside the kernel that have assumptions about order require re-sorting
> after being updated when applying relocations. In order to modify these tables,
> a few key symbols are excluded from the objcopy symbol stripping process for
> use after shuffling the text segments.
> 
> Some highlights from the build time changes to look for:
> 
> The top level kernel Makefile was modified to add the gcc flag if it
> is supported. Currently, I am applying this flag to everything it is
> possible to randomize. Anything that is written in C and not present in a
> special input section is randomized. The final binary segment 0 retains a
> consolidated .text section, as well as all the individual .text.* sections.
> Future work could turn off this flags for selected files or even entire
> subsystems, although obviously at the cost of security.
> 
> The relocs tool is updated to add relative relocations. This information
> previously wasn't included because it wasn't necessary when moving the
> entire .text segment as a unit. 
> 
> A new file was created to contain a list of symbols that objcopy should
> keep. We use those symbols at load time as described below.
> 
> * Load time
> 
> The boot kernel was modified to parse the vmlinux elf file after
> decompression to check for our interesting symbols that we kept, and to
> look for any .text.* sections to randomize. The consolidated .text section
> is skipped and not moved. The sections are shuffled randomly, and copied
> into memory following the .text section in a new random order. The existing
> code which updated relocation addresses was modified to account for
> not just a fixed delta from the load address, but the offset that the function
> section was moved to. This requires inspection of each address to see if
> it was impacted by a randomization. We use a bsearch to make this less
> horrible on performance. Any tables that need to be modified with new
> addresses or resorted are updated using the symbol addresses parsed from the
> elf symbol table.
> 
> In order to hide our new layout, symbols reported through /proc/kallsyms
> will be displayed in a random order.
> 
> Security Considerations
> -----------------------
> The objective of this patch set is to improve a technology that is already
> merged into the kernel (KASLR). This code will not prevent all attacks,
> but should instead be considered as one of several tools that can be used.
> In particular, this code is meant to make KASLR more effective in the presence
> of info leaks.
> 
> How much entropy we are adding to the existing entropy of standard KASLR will
> depend on a few variables. Firstly and most obviously, the number of functions
> that are randomized matters. This implementation keeps the existing .text
> section for code that cannot be randomized - for example, because it was
> assembly code. The less sections to randomize, the less entropy. In addition,
> due to alignment (16 bytes for x86_64), the number of bits in a address that
> the attacker needs to guess is reduced, as the lower bits are identical.
> 
> Performance Impact
> ------------------
> There are two areas where function reordering can impact performance: boot
> time latency, and run time performance.
> 
> * Boot time latency
> This implementation of finer grained KASLR impacts the boot time of the kernel
> in several places. It requires additional parsing of the kernel ELF file to
> obtain the section headers of the sections to be randomized. It calls the
> random number generator for each section to be randomized to determine that
> section's new memory location. It copies the decompressed kernel into a new
> area of memory to avoid corruption when laying out the newly randomized
> sections. It increases the number of relocations the kernel has to perform at
> boot time vs. standard KASLR, and it also requires a lookup on each address
> that needs to be relocated to see if it was in a randomized section and needs
> to be adjusted by a new offset. Finally, it re-sorts a few data tables that
> are required to be sorted by address.
> 
> Booting a test VM on a modern, well appointed system showed an increase in
> latency of approximately 1 second.
> 
> * Run time
> The performance impact at run-time of function reordering varies by workload.
> Using kcbench, a kernel compilation benchmark, the performance of a kernel
> build with finer grained KASLR was about 1% slower than a kernel with standard
> KASLR. Analysis with perf showed a slightly higher percentage of 
> L1-icache-load-misses. Other workloads were examined as well, with varied
> results. Some workloads performed significantly worse under FGKASLR, while
> others stayed the same or were mysteriously better. In general, it will
> depend on the code flow whether or not finer grained KASLR will impact
> your workload, and how the underlying code was designed. Because the layout
> changes per boot, each time a system is rebooted the performance of a workload
> may change.
> 
> Future work could identify hot areas that may not be randomized and either
> leave them in the .text section or group them together into a single section
> that may be randomized. If grouping things together helps, one other thing to
> consider is that if we could identify text blobs that should be grouped together
> to benefit a particular code flow, it could be interesting to explore
> whether this security feature could be also be used as a performance
> feature if you are interested in optimizing your kernel layout for a
> particular workload at boot time. Optimizing function layout for a particular
> workload has been researched and proven effective - for more information
> read the Facebook paper "Optimizing Function Placement for Large-Scale
> Data-Center Applications" (see references section below).
> 
> Image Size
> ----------
> Adding additional section headers as a result of compiling with
> -ffunction-sections will increase the size of the vmlinux ELF file.
> With a standard distro config, the resulting vmlinux was increased by
> about 3%. The compressed image is also increased due to the header files,
> as well as the extra relocations that must be added. You can expect fgkaslr
> to increase the size of the compressed image by about 15%.
> 
> Memory Usage
> ------------
> fgkaslr increases the amount of heap that is required at boot time,
> although this extra memory is released when the kernel has finished
> decompression. As a result, it may not be appropriate to use this feature on
> systems without much memory.
> 
> Building
> --------
> To enable fine grained KASLR, you need to have the following config options
> set (including all the ones you would use to build normal KASLR)
> 
> CONFIG_FG_KASLR=y
> 
> In addition, fgkaslr is only supported for the X86_64 architecture.
> 
> Modules
> -------
> Modules are randomized similarly to the rest of the kernel by shuffling
> the sections at load time prior to moving them into memory. The module must
> also have been build with the -ffunction-sections compiler option.
> 
> Although fgkaslr for the kernel is only supported for the X86_64 architecture,
> it is possible to use fgkaslr with modules on other architectures. To enable
> this feature, select
> 
> CONFIG_MODULE_FG_KASLR=y
> 
> This option is selected automatically for X86_64 when CONFIG_FG_KASLR is set.
> 
> Disabling
> ---------
> Disabling normal KASLR using the nokaslr command line option also disables
> fgkaslr. It is also possible to disable fgkaslr separately by booting with
> nofgkaslr on the commandline.
> 
> References
> ----------
> There are a lot of academic papers which explore finer grained ASLR.
> This paper in particular contributed the most to my implementation design
> as well as my overall understanding of the problem space:
> 
> Selfrando: Securing the Tor Browser against De-anonymization Exploits,
> M. Conti, S. Crane, T. Frassetto, et al.
> 
> For more information on how function layout impacts performance, see:
> 
> Optimizing Function Placement for Large-Scale Data-Center Applications,
> G. Ottoni, B. Maher
> 
> Alexander Lobakin (7):
>   linkage: add macros for putting ASM functions into own sections
>   x86: conditionally place regular ASM functions into separate sections
>   FG-KASLR: use a scripted approach to handle .text.* sections
>   x86/boot: allow FG-KASLR to be selected
>   arm64/crypto: conditionally place ASM functions into separate sections
>   module: use a scripted approach for FG-KASLR
>   maintainers: add MAINTAINERS entry for FG-KASLR
> 
> Kees Cook (2):
>   x86/boot: Allow a "silent" kaslr random byte fetch
>   x86/boot/compressed: Avoid duplicate malloc() implementations

These two can get landed right away -- they're standalone fixes that
can safely go in -tip.

> 
> Kristen Carlson Accardi (9):
>   x86: tools/relocs: Support >64K section headers

Same for this.

>   x86: Makefile: Add build and config option for CONFIG_FG_KASLR
>   Make sure ORC lookup covers the entire _etext - _stext
>   x86/tools: Add relative relocs for randomized functions
>   x86: Add support for function granular KASLR
>   kallsyms: Hide layout
>   livepatch: only match unique symbols when using fgkaslr
>   module: Reorder functions
>   Documentation: add a documentation for FG-KASLR

I suspect it'll still be easier to review this series as a rebase v5
followed by the evolutionary improvements, since the "basic FGKASLR" has
been reviewed in the past, and is fairly noninvasive. The changes for
ASM, new .text rules, etc, make a lot more changes that I think would be
nice to have separate so reasonable a/b testing can be done.

I'll try to go through the individual patches soon, though I'm currently
pretty swamped. :)

I'm looking forward to having this feature finally landed; it's a nice
complement to future eXecute-Only memory work too.

-Kees

> 
> Masahiro Yamada (3):
>   kbuild: merge vmlinux_link() between the ordinary link and Clang LTO
>   kbuild: do not remove 'linux' link in scripts/link-vmlinux.sh
>   kbuild: merge vmlinux_link() between ARCH=um and other architectures
> 
> Sami Tolvanen (1):
>   kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG
> 
>  .../admin-guide/kernel-parameters.txt         |   6 +
>  Documentation/security/fgkaslr.rst            | 172 ++++
>  Documentation/security/index.rst              |   1 +
>  MAINTAINERS                                   |  12 +
>  Makefile                                      |  17 +-
>  arch/Kconfig                                  |   3 +
>  arch/arm64/crypto/aes-ce-ccm-core.S           |  16 +-
>  arch/arm64/crypto/aes-ce-core.S               |  16 +-
>  arch/arm64/crypto/aes-ce.S                    |   4 +-
>  arch/arm64/crypto/aes-cipher-core.S           |   8 +-
>  arch/arm64/crypto/aes-modes.S                 |  16 +-
>  arch/arm64/crypto/aes-neon.S                  |   4 +-
>  arch/arm64/crypto/aes-neonbs-core.S           |  38 +-
>  arch/arm64/crypto/chacha-neon-core.S          |  18 +-
>  arch/arm64/crypto/crct10dif-ce-core.S         |  14 +-
>  arch/arm64/crypto/ghash-ce-core.S             |  24 +-
>  arch/arm64/crypto/nh-neon-core.S              |   4 +-
>  arch/arm64/crypto/poly1305-armv8.pl           |  17 +
>  arch/arm64/crypto/sha1-ce-core.S              |   4 +-
>  arch/arm64/crypto/sha2-ce-core.S              |   4 +-
>  arch/arm64/crypto/sha3-ce-core.S              |   4 +-
>  arch/arm64/crypto/sha512-armv8.pl             |  11 +
>  arch/arm64/crypto/sha512-ce-core.S            |   4 +-
>  arch/arm64/crypto/sm3-ce-core.S               |   4 +-
>  arch/arm64/crypto/sm4-ce-core.S               |   4 +-
>  arch/x86/Kconfig                              |   1 +
>  arch/x86/boot/compressed/Makefile             |   9 +-
>  arch/x86/boot/compressed/fgkaslr.c            | 905 ++++++++++++++++++
>  arch/x86/boot/compressed/kaslr.c              |   4 -
>  arch/x86/boot/compressed/misc.c               | 157 ++-
>  arch/x86/boot/compressed/misc.h               |  30 +
>  arch/x86/boot/compressed/utils.c              |  13 +
>  arch/x86/boot/compressed/vmlinux.symbols      |  19 +
>  arch/x86/crypto/aegis128-aesni-asm.S          |  36 +-
>  arch/x86/crypto/aes_ctrby8_avx-x86_64.S       |  12 +-
>  arch/x86/crypto/aesni-intel_asm.S             | 116 ++-
>  arch/x86/crypto/aesni-intel_avx-x86_64.S      |  32 +-
>  arch/x86/crypto/blake2s-core.S                |   8 +-
>  arch/x86/crypto/blowfish-x86_64-asm_64.S      |  16 +-
>  arch/x86/crypto/camellia-aesni-avx-asm_64.S   |  28 +-
>  arch/x86/crypto/camellia-aesni-avx2-asm_64.S  |  28 +-
>  arch/x86/crypto/camellia-x86_64-asm_64.S      |  16 +-
>  arch/x86/crypto/cast5-avx-x86_64-asm_64.S     |  24 +-
>  arch/x86/crypto/cast6-avx-x86_64-asm_64.S     |  20 +-
>  arch/x86/crypto/chacha-avx2-x86_64.S          |  12 +-
>  arch/x86/crypto/chacha-avx512vl-x86_64.S      |  12 +-
>  arch/x86/crypto/chacha-ssse3-x86_64.S         |  16 +-
>  arch/x86/crypto/crc32-pclmul_asm.S            |   4 +-
>  arch/x86/crypto/crc32c-pcl-intel-asm_64.S     |   4 +-
>  arch/x86/crypto/crct10dif-pcl-asm_64.S        |   4 +-
>  arch/x86/crypto/des3_ede-asm_64.S             |   8 +-
>  arch/x86/crypto/ghash-clmulni-intel_asm.S     |  12 +-
>  arch/x86/crypto/nh-avx2-x86_64.S              |   4 +-
>  arch/x86/crypto/nh-sse2-x86_64.S              |   4 +-
>  arch/x86/crypto/poly1305-x86_64-cryptogams.pl |   8 +-
>  arch/x86/crypto/serpent-avx-x86_64-asm_64.S   |  20 +-
>  arch/x86/crypto/serpent-avx2-asm_64.S         |  20 +-
>  arch/x86/crypto/serpent-sse2-i586-asm_32.S    |   8 +-
>  arch/x86/crypto/serpent-sse2-x86_64-asm_64.S  |   8 +-
>  arch/x86/crypto/sha1_avx2_x86_64_asm.S        |   4 +-
>  arch/x86/crypto/sha1_ni_asm.S                 |   4 +-
>  arch/x86/crypto/sha1_ssse3_asm.S              |   4 +-
>  arch/x86/crypto/sha256-avx-asm.S              |   4 +-
>  arch/x86/crypto/sha256-avx2-asm.S             |   4 +-
>  arch/x86/crypto/sha256-ssse3-asm.S            |   4 +-
>  arch/x86/crypto/sha256_ni_asm.S               |   4 +-
>  arch/x86/crypto/sha512-avx-asm.S              |   4 +-
>  arch/x86/crypto/sha512-avx2-asm.S             |   4 +-
>  arch/x86/crypto/sha512-ssse3-asm.S            |   4 +-
>  arch/x86/crypto/twofish-avx-x86_64-asm_64.S   |  20 +-
>  arch/x86/crypto/twofish-i586-asm_32.S         |   8 +-
>  arch/x86/crypto/twofish-x86_64-asm_64-3way.S  |   8 +-
>  arch/x86/crypto/twofish-x86_64-asm_64.S       |   8 +-
>  arch/x86/entry/entry_32.S                     |  24 +-
>  arch/x86/entry/entry_64.S                     |  18 +-
>  arch/x86/entry/thunk_32.S                     |   4 +-
>  arch/x86/entry/thunk_64.S                     |   8 +-
>  arch/x86/include/asm/boot.h                   |  13 +-
>  arch/x86/include/asm/paravirt.h               |   2 +-
>  arch/x86/include/asm/qspinlock_paravirt.h     |   2 +-
>  arch/x86/kernel/acpi/wakeup_32.S              |   9 +-
>  arch/x86/kernel/acpi/wakeup_64.S              |  10 +-
>  arch/x86/kernel/ftrace_32.S                   |  19 +-
>  arch/x86/kernel/ftrace_64.S                   |  28 +-
>  arch/x86/kernel/irqflags.S                    |   4 +-
>  arch/x86/kernel/kprobes/core.c                |   3 +-
>  arch/x86/kernel/kvm.c                         |   2 +-
>  arch/x86/kernel/relocate_kernel_32.S          |   2 +
>  arch/x86/kernel/relocate_kernel_64.S          |   2 +
>  arch/x86/kernel/vmlinux.lds.S                 |   6 +-
>  arch/x86/kvm/emulate.c                        |   2 +-
>  arch/x86/kvm/vmx/vmenter.S                    |   8 +-
>  arch/x86/lib/clear_page_64.S                  |  12 +-
>  arch/x86/lib/cmpxchg16b_emu.S                 |   4 +-
>  arch/x86/lib/copy_mc_64.S                     |   8 +-
>  arch/x86/lib/copy_page_64.S                   |   7 +-
>  arch/x86/lib/copy_user_64.S                   |  18 +-
>  arch/x86/lib/csum-copy_64.S                   |   4 +-
>  arch/x86/lib/error-inject.c                   |   3 +-
>  arch/x86/lib/getuser.S                        |  37 +-
>  arch/x86/lib/hweight.S                        |   9 +-
>  arch/x86/lib/iomap_copy_64.S                  |   4 +-
>  arch/x86/lib/kaslr.c                          |  18 +-
>  arch/x86/lib/memmove_64.S                     |   4 +-
>  arch/x86/lib/memset_64.S                      |  12 +-
>  arch/x86/lib/msr-reg.S                        |   8 +-
>  arch/x86/lib/putuser.S                        |  18 +-
>  arch/x86/mm/mem_encrypt_boot.S                |   8 +-
>  arch/x86/platform/efi/efi_stub_64.S           |   4 +-
>  arch/x86/platform/efi/efi_thunk_64.S          |   4 +-
>  arch/x86/power/hibernate_asm_32.S             |  14 +-
>  arch/x86/power/hibernate_asm_64.S             |  14 +-
>  arch/x86/tools/relocs.c                       | 135 ++-
>  arch/x86/tools/relocs.h                       |   4 +-
>  arch/x86/tools/relocs_common.c                |  15 +-
>  arch/x86/xen/xen-asm.S                        |  49 +-
>  arch/x86/xen/xen-head.S                       |  10 +-
>  include/asm-generic/vmlinux.lds.h             |  41 +-
>  include/linux/decompress/mm.h                 |  12 +-
>  include/linux/linkage.h                       |  76 ++
>  include/uapi/linux/elf.h                      |   1 +
>  init/Kconfig                                  |  51 +
>  kernel/kallsyms.c                             | 158 ++-
>  kernel/livepatch/core.c                       |  11 +
>  kernel/module.c                               |  91 +-
>  scripts/Makefile.build                        |  27 +-
>  scripts/Makefile.lib                          |   7 +
>  scripts/Makefile.modfinal                     |  36 +-
>  scripts/Makefile.modpost                      |  22 +-
>  scripts/gen_autoksyms.sh                      |  12 -
>  scripts/generate_text_sections.pl             | 149 +++
>  scripts/link-vmlinux.sh                       | 104 +-
>  scripts/module.lds.S                          |  14 +-
>  133 files changed, 2771 insertions(+), 757 deletions(-)
>  create mode 100644 Documentation/security/fgkaslr.rst
>  create mode 100644 arch/x86/boot/compressed/fgkaslr.c
>  create mode 100644 arch/x86/boot/compressed/utils.c
>  create mode 100644 arch/x86/boot/compressed/vmlinux.symbols
>  create mode 100755 scripts/generate_text_sections.pl
> 
> -- 
> 2.31.1
> 

-- 
Kees Cook

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v6 kspp-next 00/22] Function Granular KASLR
  2021-08-31 17:27 ` [PATCH v6 kspp-next 00/22] Function Granular KASLR Kees Cook
@ 2021-09-01 10:36   ` Alexander Lobakin
  2021-09-02  1:36     ` Kees Cook
  0 siblings, 1 reply; 29+ messages in thread
From: Alexander Lobakin @ 2021-09-01 10:36 UTC (permalink / raw)
  To: Kees Cook
  Cc: Alexander Lobakin, linux-hardening, Kristen C Accardi,
	Kristen Carlson Accardi, Masahiro Yamada, H. Peter Anvin,
	Jessica Yu, Nathan Chancellor, Nick Desaulniers, Marios Pomonis,
	Sami Tolvanen, Tony Luck, Ard Biesheuvel, Jesse Brandeburg,
	Lukasz Czapnik, Marta A Plantykow, Michal Kubiak,
	Michal Swiatkowski, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

From: Kees Cook <keescook@chromium.org>
Date: Tue, 31 Aug 2021 10:27:45 -0700

> On Tue, Aug 31, 2021 at 04:40:52PM +0200, Alexander Lobakin wrote:
> > This is a massive rework and a respin of Kristen Accardi's marvellous
> > FG-KASLR series (v5).
> 
> Thanks for working on this! I know Marios has been looking at some of
> this as well. I think he tracked down a kretprobes bug and has a fixed
> prepared.

I was waiting for the fix to be landed in our discussion, but it
hasn't appeared there, so I queued the series without it. Will be
glad to finally see the fix and include it in v7.

> > The major differences since v5 [0]:
> > * You can now tune the number of functions per each section to
> >   achieve the preferable vmlinux size or protection level. Default
> >   is still as one section per each function.
> >   This can be handy for storage-constrained systems. 4-8 fps are
> >   still strong, but reduce the size of the final vmlinu{x,z}
> >   significantly;
> 
> Interesting, but I'm not sure what the size issue is. v5's on-disk
> image size issues were related to the large relocation table that was
> used during decompress and layout, but would get discarded. The final
> in-core image size was roughly the same size as a non-FGKASLR kernel
> (since functions were already aligned even without -ffunction-sections).
> How does the functions-per-section knob change image size?

Without FG-KASLR, we have only one .text section, and the total
section number is relatively small.
With FG-KASLR enabled, we have 40K+ separate text sections (I have
40K on a setup with ClangLTO and ClangCFI and about 48K on a
"regular" one) and each of them is described in the ELF header. Plus
a separate .rela.text section for every single of them. That's the
main reason of the size increases.

> > * I don't use orphan sections anymore. It's not reliable at all /
> >   may differ from linker to linker, and also conflicts with
> >   CONFIG_LD_ORPHAN_WARN which is great for catching random bugs ->
> > * All the .text.* sections are now being described explicitly in the
> >   linker script. A Perl script is used to take the original LDS, the
> >   original object file, read a list of input sections from it and
> >   generate the resulting LDS.
> >   This costs a bit of linking time as LD tends to think hard when
> >   processing scripts > 1 Mb. It adds about 40-60 seconds to the
> >   whole linking process (BTF step, 2-3 kallsyms steps and the final
> >   step), but "better safe than sorry".
> >   In addition, that approach allows to reserve some space at the end
> >   and add some link assertions ->
> 
> Yeah, this "hope that orphan handling does it right" bugged me too, but my
> attempts to solve it looked much like yours: creating a linker file that
> named all the sections. I found this to be prohibitively expensive at link
> time (and that seems backed by your own measurements of an extra minute
> or so at link time). If that's still the result of using a generated
> linker file, we just need to depend on orphan handling. LD_ORPHAN_WARN
> will still exist for non-FGKASLR builds, so the benefits will continue
> to exist -- I think the correct solution is to have the linker grow a
> "pass through" special target like "DISCARD", which just maps given
> input section patterns into same-named output sections.

We still have LD_ORPHAN_WARN on non-FG-KASLR builds, but we also
have a rather different set of sections with FG-KASLR enabled. For
example, I noticed the appearing of .symtab_shndx section only in
virtue of LD_ORPHAN_WARN. So it's kinda not the same.
I don't see a problem in this extra minute. FG-KASLR is all about
security, and you often pay something for this. We already have a
size increase, and a small delay while booting, and we can't get
rid of them. With orphan sections you leave a space for potentional
flaws of the code, linker and/or linker script, which is really
unwanted in case of a security feature.
After all, ClangLTO increases the linking time at lot, and
TRIM_UNUSED_KSYMS builds almost the entire kernel two times in a
row, but nobody complains about this as there's nothing we can do
with it and it's the price you pay for the optimizations, so again,
I don't see a problem here.
I'll be glad to see approaches with no link time penalties and still
without "grey zones" like orphans and stuff, but I could come only
with this. This can be a room for future patches and optimizations.

> > * Input .text section now must be empty, otherwise the linkage will
> >   be stopped. This is implemented by the size assertion in the
> >   resulting LD script and is designed to plug the potentional layout
> >   leakage. This also means that ->
> 
> I worry this will create unexpected problems for named sections that
> weren't originally being randomized with the v5 FGKASLR.

1. Input .text just contained a bunch of ASM functions (described
   below), none of them required any kind of special handling.
2. This was tested a lot.
3. We have plenty of time to test on a wide variety of setups since
   we miss 5.15 window.

> > * "Regular" ASM functions are now being placed into unique separate
> >   functions the same way compiler does this for C functions. This is
> >   achieved by introducing and using several new macros which take
> >   the symbol name as a base for its new section name.
> >   This gives a better opportunity to both DCE and FG-KASLR, as ASM
> >   code now can also be randomized or garbage-collected;
> 
> This is interesting! I think it'd be a good evolutionary step on top of
> "basic FGKASLR".

I still don't get why you're trying to split this series into two.
It's been almost a year since v5 was published, I doubt you can get
"basic FG-KASLR" accepted quickly just because it was reviewed back
then.
I prefer to provide a full picture of what I'm trying to bring, so
the community could review it all and throw much more ideas and
stuff.

> > * It's now fully compatible with ClangLTO, ClangCFI,
> >   CONFIG_LD_ORPHAN_WARN and some more stuff landed since the last
> >   revision was published;
> 
> FWIW, v5 was was too. :) I didn't have to do anything to v5 to make it
> work with ClangLTO and ClangCFI.

Once again, repeating the thing I wrote earlier in our discussion:
ClangCFI, at least shadowed implementation, requires the first text
section of the module to be page-aligned and contain __cfi_check()
at the very beginning of this section. With FG-KASLR and without
special handling, this section gets randomized along with the
others, and ClangCFI either rejects almost all modules or panics
the kernel.

> > * Includes several fixes: relocations inside .altinstr_replacement
> >   code and minor issues found and/or suggested by LKP robot.
> 
> Excellent!
> 
> > The series was compile-time and runtime tested on the following
> > setups with no issues:
> > - x86_64, GCC 11, Binutils 2.35;
> > - x86_64, Clang/LLVM 12, ClangLTO + ClangCFI (from Sami's tree).
> 
> Great, this is a good start. One place we saw problems in the past was
> with i386 build gotchas, so that'll need testing too.

For now, FG_KASLR for x86 depends on X86_64. We might relax this
dependency later after enough testing or whatsoever (like it's been
done for ClangLTO).

> > The first 4 patches are from the linux-kbuild tree and included
> > to avoid merge conflicts and non-intuitive resolving of them.
> 
> Sounds good. It might be easier to base the series on linux-next, so a
> smaller series. Though given the merge window just opened, it might make
> more sense for a v7 to be based on v5.15-rc2 in three weeks.

I don't usually base any series on linux-next, because it contains
all the changes from all "for-next" branches and repos, while the
series finally gets accepted to the specific repo based on just
v5.x-rc1 (sometimes on -rc2). This may bring additional apply/merge
problems.

> > The series is also available here: [1]
> > 
> > [0] https://lore.kernel.org/kernel-hardening/20200923173905.11219-1-kristen@linux.intel.com
> > [1] https://github.com/alobakin/linux/pull/3
> > 
> > The original v5 cover letter:
> 
> More notes below...
> 
> > 
> > Function Granular Kernel Address Space Layout Randomization (fgkaslr)
> > ---------------------------------------------------------------------
> > 
> > This patch set is an implementation of finer grained kernel address space
> > randomization. It rearranges your kernel code at load time 
> > on a per-function level granularity, with only around a second added to
> > boot time.
> > 
> > Changes in v5:
> > --------------
> > * fixed a bug in the code which increases boot heap size for
> >   CONFIG_FG_KASLR which prevented the boot heap from being increased
> >   for CONFIG_FG_KASLR when using bzip2 compression. Thanks to Andy Lavr
> >   for finding the problem and identifying the solution.
> > * changed the adjustment of the orc_unwind_ip table at boot time to
> >   disregard relocs associated with this table, and instead inspect the
> >   entries separately. Relocs are not able to be used since they are
> >   no longer correct once the table is resorted at buildtime.
> > * changed how orc_unwind_ip addresses in randomized sections are identified
> >   to include the byte immediately after the end of the section.
> > * updated module code to use kvmalloc/kvfree based on suggestions from
> >   Evgenii Shatokhin <eshatokhin@virtuozzo.com>.
> > * changed kernel commandline to disable fgkaslr to simply "nofgkaslr" to
> >   match the nokaslr option. fgkaslr="X" can be added at a later date
> >   if it is needed.
> > * Added a patch to force livepatch to require symbols to be unique if
> >   using while fgkaslr either for core or modules.
> > 
> > Changes in v4:
> > -------------
> > * dropped the patch to split out change to STATIC definition in
> >   x86/boot/compressed/misc.c and replaced with a patch authored
> >   by Kees Cook to avoid the duplicate malloc definitions
> > * Added a section to Documentation/admin-guide/kernel-parameters.txt
> >   to document the fgkaslr boot option.
> > * redesigned the patch to hide the new layout when reading
> >   /proc/kallsyms. The previous implementation utilized a dynamically
> >   allocated linked list to display the kernel and module symbols
> >   in alphabetical order. The new implementation uses a randomly
> >   shuffled index array to display the kernel and module symbols
> >   in a random order.
> > 
> > Changes in v3:
> > -------------
> > * Makefile changes to accommodate CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
> > * removal of extraneous ALIGN_PAGE from _etext changes
> > * changed variable names in x86/tools/relocs to be less confusing
> > * split out change to STATIC definition in x86/boot/compressed/misc.c
> > * Updates to Documentation to make it more clear what is preserved in .text
> > * much more detailed commit message for function granular KASLR patch
> > * minor tweaks and changes that make for more readable code
> > * this cover letter updated slightly to add additional details
> > 
> > Changes in v2:
> > --------------
> > * Fix to address i386 build failure
> > * Allow module reordering patch to be configured separately so that
> >   arm (or other non-x86_64 arches) can take advantage of module function
> >   reordering. This support has not be tested by me, but smoke tested by
> >   Ard Biesheuvel <ardb@kernel.org> on arm.
> > * Fix build issue when building on arm as reported by
> >   Ard Biesheuvel <ardb@kernel.org> 
> > 
> > Patches to objtool are included because they are dependencies for this
> > patchset, however they have been submitted by their maintainer separately.
> > 
> > Background
> > ----------
> > KASLR was merged into the kernel with the objective of increasing the
> > difficulty of code reuse attacks. Code reuse attacks reused existing code
> > snippets to get around existing memory protections. They exploit software bugs
> > which expose addresses of useful code snippets to control the flow of
> > execution for their own nefarious purposes. KASLR moves the entire kernel
> > code text as a unit at boot time in order to make addresses less predictable.
> > The order of the code within the segment is unchanged - only the base address
> > is shifted. There are a few shortcomings to this algorithm.
> > 
> > 1. Low Entropy - there are only so many locations the kernel can fit in. This
> >    means an attacker could guess without too much trouble.
> > 2. Knowledge of a single address can reveal the offset of the base address,
> >    exposing all other locations for a published/known kernel image.
> > 3. Info leaks abound.
> > 
> > Finer grained ASLR has been proposed as a way to make ASLR more resistant
> > to info leaks. It is not a new concept at all, and there are many variations
> > possible. Function reordering is an implementation of finer grained ASLR
> > which randomizes the layout of an address space on a function level
> > granularity. We use the term "fgkaslr" in this document to refer to the
> > technique of function reordering when used with KASLR, as well as finer grained
> > KASLR in general.
> > 
> > Proposed Improvement
> > --------------------
> > This patch set proposes adding function reordering on top of the existing
> > KASLR base address randomization. The over-arching objective is incremental
> > improvement over what we already have. It is designed to work in combination
> > with the existing solution. The implementation is really pretty simple, and
> > there are 2 main area where changes occur:
> > 
> > * Build time
> > 
> > GCC has had an option to place functions into individual .text sections for
> > many years now. This option can be used to implement function reordering at
> > load time. The final compiled vmlinux retains all the section headers, which
> > can be used to help find the address ranges of each function. Using this
> > information and an expanded table of relocation addresses, individual text
> > sections can be suffled immediately after decompression. Some data tables
> > inside the kernel that have assumptions about order require re-sorting
> > after being updated when applying relocations. In order to modify these tables,
> > a few key symbols are excluded from the objcopy symbol stripping process for
> > use after shuffling the text segments.
> > 
> > Some highlights from the build time changes to look for:
> > 
> > The top level kernel Makefile was modified to add the gcc flag if it
> > is supported. Currently, I am applying this flag to everything it is
> > possible to randomize. Anything that is written in C and not present in a
> > special input section is randomized. The final binary segment 0 retains a
> > consolidated .text section, as well as all the individual .text.* sections.
> > Future work could turn off this flags for selected files or even entire
> > subsystems, although obviously at the cost of security.
> > 
> > The relocs tool is updated to add relative relocations. This information
> > previously wasn't included because it wasn't necessary when moving the
> > entire .text segment as a unit. 
> > 
> > A new file was created to contain a list of symbols that objcopy should
> > keep. We use those symbols at load time as described below.
> > 
> > * Load time
> > 
> > The boot kernel was modified to parse the vmlinux elf file after
> > decompression to check for our interesting symbols that we kept, and to
> > look for any .text.* sections to randomize. The consolidated .text section
> > is skipped and not moved. The sections are shuffled randomly, and copied
> > into memory following the .text section in a new random order. The existing
> > code which updated relocation addresses was modified to account for
> > not just a fixed delta from the load address, but the offset that the function
> > section was moved to. This requires inspection of each address to see if
> > it was impacted by a randomization. We use a bsearch to make this less
> > horrible on performance. Any tables that need to be modified with new
> > addresses or resorted are updated using the symbol addresses parsed from the
> > elf symbol table.
> > 
> > In order to hide our new layout, symbols reported through /proc/kallsyms
> > will be displayed in a random order.
> > 
> > Security Considerations
> > -----------------------
> > The objective of this patch set is to improve a technology that is already
> > merged into the kernel (KASLR). This code will not prevent all attacks,
> > but should instead be considered as one of several tools that can be used.
> > In particular, this code is meant to make KASLR more effective in the presence
> > of info leaks.
> > 
> > How much entropy we are adding to the existing entropy of standard KASLR will
> > depend on a few variables. Firstly and most obviously, the number of functions
> > that are randomized matters. This implementation keeps the existing .text
> > section for code that cannot be randomized - for example, because it was
> > assembly code. The less sections to randomize, the less entropy. In addition,
> > due to alignment (16 bytes for x86_64), the number of bits in a address that
> > the attacker needs to guess is reduced, as the lower bits are identical.
> > 
> > Performance Impact
> > ------------------
> > There are two areas where function reordering can impact performance: boot
> > time latency, and run time performance.
> > 
> > * Boot time latency
> > This implementation of finer grained KASLR impacts the boot time of the kernel
> > in several places. It requires additional parsing of the kernel ELF file to
> > obtain the section headers of the sections to be randomized. It calls the
> > random number generator for each section to be randomized to determine that
> > section's new memory location. It copies the decompressed kernel into a new
> > area of memory to avoid corruption when laying out the newly randomized
> > sections. It increases the number of relocations the kernel has to perform at
> > boot time vs. standard KASLR, and it also requires a lookup on each address
> > that needs to be relocated to see if it was in a randomized section and needs
> > to be adjusted by a new offset. Finally, it re-sorts a few data tables that
> > are required to be sorted by address.
> > 
> > Booting a test VM on a modern, well appointed system showed an increase in
> > latency of approximately 1 second.
> > 
> > * Run time
> > The performance impact at run-time of function reordering varies by workload.
> > Using kcbench, a kernel compilation benchmark, the performance of a kernel
> > build with finer grained KASLR was about 1% slower than a kernel with standard
> > KASLR. Analysis with perf showed a slightly higher percentage of 
> > L1-icache-load-misses. Other workloads were examined as well, with varied
> > results. Some workloads performed significantly worse under FGKASLR, while
> > others stayed the same or were mysteriously better. In general, it will
> > depend on the code flow whether or not finer grained KASLR will impact
> > your workload, and how the underlying code was designed. Because the layout
> > changes per boot, each time a system is rebooted the performance of a workload
> > may change.
> > 
> > Future work could identify hot areas that may not be randomized and either
> > leave them in the .text section or group them together into a single section
> > that may be randomized. If grouping things together helps, one other thing to
> > consider is that if we could identify text blobs that should be grouped together
> > to benefit a particular code flow, it could be interesting to explore
> > whether this security feature could be also be used as a performance
> > feature if you are interested in optimizing your kernel layout for a
> > particular workload at boot time. Optimizing function layout for a particular
> > workload has been researched and proven effective - for more information
> > read the Facebook paper "Optimizing Function Placement for Large-Scale
> > Data-Center Applications" (see references section below).
> > 
> > Image Size
> > ----------
> > Adding additional section headers as a result of compiling with
> > -ffunction-sections will increase the size of the vmlinux ELF file.
> > With a standard distro config, the resulting vmlinux was increased by
> > about 3%. The compressed image is also increased due to the header files,
> > as well as the extra relocations that must be added. You can expect fgkaslr
> > to increase the size of the compressed image by about 15%.
> > 
> > Memory Usage
> > ------------
> > fgkaslr increases the amount of heap that is required at boot time,
> > although this extra memory is released when the kernel has finished
> > decompression. As a result, it may not be appropriate to use this feature on
> > systems without much memory.
> > 
> > Building
> > --------
> > To enable fine grained KASLR, you need to have the following config options
> > set (including all the ones you would use to build normal KASLR)
> > 
> > CONFIG_FG_KASLR=y
> > 
> > In addition, fgkaslr is only supported for the X86_64 architecture.
> > 
> > Modules
> > -------
> > Modules are randomized similarly to the rest of the kernel by shuffling
> > the sections at load time prior to moving them into memory. The module must
> > also have been build with the -ffunction-sections compiler option.
> > 
> > Although fgkaslr for the kernel is only supported for the X86_64 architecture,
> > it is possible to use fgkaslr with modules on other architectures. To enable
> > this feature, select
> > 
> > CONFIG_MODULE_FG_KASLR=y
> > 
> > This option is selected automatically for X86_64 when CONFIG_FG_KASLR is set.
> > 
> > Disabling
> > ---------
> > Disabling normal KASLR using the nokaslr command line option also disables
> > fgkaslr. It is also possible to disable fgkaslr separately by booting with
> > nofgkaslr on the commandline.
> > 
> > References
> > ----------
> > There are a lot of academic papers which explore finer grained ASLR.
> > This paper in particular contributed the most to my implementation design
> > as well as my overall understanding of the problem space:
> > 
> > Selfrando: Securing the Tor Browser against De-anonymization Exploits,
> > M. Conti, S. Crane, T. Frassetto, et al.
> > 
> > For more information on how function layout impacts performance, see:
> > 
> > Optimizing Function Placement for Large-Scale Data-Center Applications,
> > G. Ottoni, B. Maher
> > 
> > Alexander Lobakin (7):
> >   linkage: add macros for putting ASM functions into own sections
> >   x86: conditionally place regular ASM functions into separate sections
> >   FG-KASLR: use a scripted approach to handle .text.* sections
> >   x86/boot: allow FG-KASLR to be selected
> >   arm64/crypto: conditionally place ASM functions into separate sections
> >   module: use a scripted approach for FG-KASLR
> >   maintainers: add MAINTAINERS entry for FG-KASLR
> > 
> > Kees Cook (2):
> >   x86/boot: Allow a "silent" kaslr random byte fetch
> >   x86/boot/compressed: Avoid duplicate malloc() implementations
> 
> These two can get landed right away -- they're standalone fixes that
> can safely go in -tip.
> 
> > 
> > Kristen Carlson Accardi (9):
> >   x86: tools/relocs: Support >64K section headers
> 
> Same for this.

They make little to no sense for non-FG-KASLR systems. And none of
them are "pure" fixes.
The same could be said about e.g. ORC lookup patch, but again, it
makes no sense right now.

> >   x86: Makefile: Add build and config option for CONFIG_FG_KASLR
> >   Make sure ORC lookup covers the entire _etext - _stext
> >   x86/tools: Add relative relocs for randomized functions
> >   x86: Add support for function granular KASLR
> >   kallsyms: Hide layout
> >   livepatch: only match unique symbols when using fgkaslr
> >   module: Reorder functions
> >   Documentation: add a documentation for FG-KASLR
> 
> I suspect it'll still be easier to review this series as a rebase v5
> followed by the evolutionary improvements, since the "basic FGKASLR" has
> been reviewed in the past, and is fairly noninvasive. The changes for
> ASM, new .text rules, etc, make a lot more changes that I think would be
> nice to have separate so reasonable a/b testing can be done.

I don't see a point in testing it two times instead of just one, as
well as in delivering this feature in two halves. It sounds like
"let's introduce ClangLTO, but firstly only for modules, as LTO for
vmlinux requires changes in objtool code and a special handling for
the initcalls".
The changes you mentioned only seem invasive, in fact, they can
carry way less harm than the "basic FG-KASLR" itself.

> I'll try to go through the individual patches soon, though I'm currently
> pretty swamped. :)
> 
> I'm looking forward to having this feature finally landed; it's a nice
> complement to future eXecute-Only memory work too.
> 
> -Kees
> 
> > 
> > Masahiro Yamada (3):
> >   kbuild: merge vmlinux_link() between the ordinary link and Clang LTO
> >   kbuild: do not remove 'linux' link in scripts/link-vmlinux.sh
> >   kbuild: merge vmlinux_link() between ARCH=um and other architectures
> > 
> > Sami Tolvanen (1):
> >   kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG
> > 
> >  .../admin-guide/kernel-parameters.txt         |   6 +
> >  Documentation/security/fgkaslr.rst            | 172 ++++
> >  Documentation/security/index.rst              |   1 +
> >  MAINTAINERS                                   |  12 +
> >  Makefile                                      |  17 +-
> >  arch/Kconfig                                  |   3 +
> >  arch/arm64/crypto/aes-ce-ccm-core.S           |  16 +-
> >  arch/arm64/crypto/aes-ce-core.S               |  16 +-
> >  arch/arm64/crypto/aes-ce.S                    |   4 +-
> >  arch/arm64/crypto/aes-cipher-core.S           |   8 +-
> >  arch/arm64/crypto/aes-modes.S                 |  16 +-
> >  arch/arm64/crypto/aes-neon.S                  |   4 +-
> >  arch/arm64/crypto/aes-neonbs-core.S           |  38 +-
> >  arch/arm64/crypto/chacha-neon-core.S          |  18 +-
> >  arch/arm64/crypto/crct10dif-ce-core.S         |  14 +-
> >  arch/arm64/crypto/ghash-ce-core.S             |  24 +-
> >  arch/arm64/crypto/nh-neon-core.S              |   4 +-
> >  arch/arm64/crypto/poly1305-armv8.pl           |  17 +
> >  arch/arm64/crypto/sha1-ce-core.S              |   4 +-
> >  arch/arm64/crypto/sha2-ce-core.S              |   4 +-
> >  arch/arm64/crypto/sha3-ce-core.S              |   4 +-
> >  arch/arm64/crypto/sha512-armv8.pl             |  11 +
> >  arch/arm64/crypto/sha512-ce-core.S            |   4 +-
> >  arch/arm64/crypto/sm3-ce-core.S               |   4 +-
> >  arch/arm64/crypto/sm4-ce-core.S               |   4 +-
> >  arch/x86/Kconfig                              |   1 +
> >  arch/x86/boot/compressed/Makefile             |   9 +-
> >  arch/x86/boot/compressed/fgkaslr.c            | 905 ++++++++++++++++++
> >  arch/x86/boot/compressed/kaslr.c              |   4 -
> >  arch/x86/boot/compressed/misc.c               | 157 ++-
> >  arch/x86/boot/compressed/misc.h               |  30 +
> >  arch/x86/boot/compressed/utils.c              |  13 +
> >  arch/x86/boot/compressed/vmlinux.symbols      |  19 +
> >  arch/x86/crypto/aegis128-aesni-asm.S          |  36 +-
> >  arch/x86/crypto/aes_ctrby8_avx-x86_64.S       |  12 +-
> >  arch/x86/crypto/aesni-intel_asm.S             | 116 ++-
> >  arch/x86/crypto/aesni-intel_avx-x86_64.S      |  32 +-
> >  arch/x86/crypto/blake2s-core.S                |   8 +-
> >  arch/x86/crypto/blowfish-x86_64-asm_64.S      |  16 +-
> >  arch/x86/crypto/camellia-aesni-avx-asm_64.S   |  28 +-
> >  arch/x86/crypto/camellia-aesni-avx2-asm_64.S  |  28 +-
> >  arch/x86/crypto/camellia-x86_64-asm_64.S      |  16 +-
> >  arch/x86/crypto/cast5-avx-x86_64-asm_64.S     |  24 +-
> >  arch/x86/crypto/cast6-avx-x86_64-asm_64.S     |  20 +-
> >  arch/x86/crypto/chacha-avx2-x86_64.S          |  12 +-
> >  arch/x86/crypto/chacha-avx512vl-x86_64.S      |  12 +-
> >  arch/x86/crypto/chacha-ssse3-x86_64.S         |  16 +-
> >  arch/x86/crypto/crc32-pclmul_asm.S            |   4 +-
> >  arch/x86/crypto/crc32c-pcl-intel-asm_64.S     |   4 +-
> >  arch/x86/crypto/crct10dif-pcl-asm_64.S        |   4 +-
> >  arch/x86/crypto/des3_ede-asm_64.S             |   8 +-
> >  arch/x86/crypto/ghash-clmulni-intel_asm.S     |  12 +-
> >  arch/x86/crypto/nh-avx2-x86_64.S              |   4 +-
> >  arch/x86/crypto/nh-sse2-x86_64.S              |   4 +-
> >  arch/x86/crypto/poly1305-x86_64-cryptogams.pl |   8 +-
> >  arch/x86/crypto/serpent-avx-x86_64-asm_64.S   |  20 +-
> >  arch/x86/crypto/serpent-avx2-asm_64.S         |  20 +-
> >  arch/x86/crypto/serpent-sse2-i586-asm_32.S    |   8 +-
> >  arch/x86/crypto/serpent-sse2-x86_64-asm_64.S  |   8 +-
> >  arch/x86/crypto/sha1_avx2_x86_64_asm.S        |   4 +-
> >  arch/x86/crypto/sha1_ni_asm.S                 |   4 +-
> >  arch/x86/crypto/sha1_ssse3_asm.S              |   4 +-
> >  arch/x86/crypto/sha256-avx-asm.S              |   4 +-
> >  arch/x86/crypto/sha256-avx2-asm.S             |   4 +-
> >  arch/x86/crypto/sha256-ssse3-asm.S            |   4 +-
> >  arch/x86/crypto/sha256_ni_asm.S               |   4 +-
> >  arch/x86/crypto/sha512-avx-asm.S              |   4 +-
> >  arch/x86/crypto/sha512-avx2-asm.S             |   4 +-
> >  arch/x86/crypto/sha512-ssse3-asm.S            |   4 +-
> >  arch/x86/crypto/twofish-avx-x86_64-asm_64.S   |  20 +-
> >  arch/x86/crypto/twofish-i586-asm_32.S         |   8 +-
> >  arch/x86/crypto/twofish-x86_64-asm_64-3way.S  |   8 +-
> >  arch/x86/crypto/twofish-x86_64-asm_64.S       |   8 +-
> >  arch/x86/entry/entry_32.S                     |  24 +-
> >  arch/x86/entry/entry_64.S                     |  18 +-
> >  arch/x86/entry/thunk_32.S                     |   4 +-
> >  arch/x86/entry/thunk_64.S                     |   8 +-
> >  arch/x86/include/asm/boot.h                   |  13 +-
> >  arch/x86/include/asm/paravirt.h               |   2 +-
> >  arch/x86/include/asm/qspinlock_paravirt.h     |   2 +-
> >  arch/x86/kernel/acpi/wakeup_32.S              |   9 +-
> >  arch/x86/kernel/acpi/wakeup_64.S              |  10 +-
> >  arch/x86/kernel/ftrace_32.S                   |  19 +-
> >  arch/x86/kernel/ftrace_64.S                   |  28 +-
> >  arch/x86/kernel/irqflags.S                    |   4 +-
> >  arch/x86/kernel/kprobes/core.c                |   3 +-
> >  arch/x86/kernel/kvm.c                         |   2 +-
> >  arch/x86/kernel/relocate_kernel_32.S          |   2 +
> >  arch/x86/kernel/relocate_kernel_64.S          |   2 +
> >  arch/x86/kernel/vmlinux.lds.S                 |   6 +-
> >  arch/x86/kvm/emulate.c                        |   2 +-
> >  arch/x86/kvm/vmx/vmenter.S                    |   8 +-
> >  arch/x86/lib/clear_page_64.S                  |  12 +-
> >  arch/x86/lib/cmpxchg16b_emu.S                 |   4 +-
> >  arch/x86/lib/copy_mc_64.S                     |   8 +-
> >  arch/x86/lib/copy_page_64.S                   |   7 +-
> >  arch/x86/lib/copy_user_64.S                   |  18 +-
> >  arch/x86/lib/csum-copy_64.S                   |   4 +-
> >  arch/x86/lib/error-inject.c                   |   3 +-
> >  arch/x86/lib/getuser.S                        |  37 +-
> >  arch/x86/lib/hweight.S                        |   9 +-
> >  arch/x86/lib/iomap_copy_64.S                  |   4 +-
> >  arch/x86/lib/kaslr.c                          |  18 +-
> >  arch/x86/lib/memmove_64.S                     |   4 +-
> >  arch/x86/lib/memset_64.S                      |  12 +-
> >  arch/x86/lib/msr-reg.S                        |   8 +-
> >  arch/x86/lib/putuser.S                        |  18 +-
> >  arch/x86/mm/mem_encrypt_boot.S                |   8 +-
> >  arch/x86/platform/efi/efi_stub_64.S           |   4 +-
> >  arch/x86/platform/efi/efi_thunk_64.S          |   4 +-
> >  arch/x86/power/hibernate_asm_32.S             |  14 +-
> >  arch/x86/power/hibernate_asm_64.S             |  14 +-
> >  arch/x86/tools/relocs.c                       | 135 ++-
> >  arch/x86/tools/relocs.h                       |   4 +-
> >  arch/x86/tools/relocs_common.c                |  15 +-
> >  arch/x86/xen/xen-asm.S                        |  49 +-
> >  arch/x86/xen/xen-head.S                       |  10 +-
> >  include/asm-generic/vmlinux.lds.h             |  41 +-
> >  include/linux/decompress/mm.h                 |  12 +-
> >  include/linux/linkage.h                       |  76 ++
> >  include/uapi/linux/elf.h                      |   1 +
> >  init/Kconfig                                  |  51 +
> >  kernel/kallsyms.c                             | 158 ++-
> >  kernel/livepatch/core.c                       |  11 +
> >  kernel/module.c                               |  91 +-
> >  scripts/Makefile.build                        |  27 +-
> >  scripts/Makefile.lib                          |   7 +
> >  scripts/Makefile.modfinal                     |  36 +-
> >  scripts/Makefile.modpost                      |  22 +-
> >  scripts/gen_autoksyms.sh                      |  12 -
> >  scripts/generate_text_sections.pl             | 149 +++
> >  scripts/link-vmlinux.sh                       | 104 +-
> >  scripts/module.lds.S                          |  14 +-
> >  133 files changed, 2771 insertions(+), 757 deletions(-)
> >  create mode 100644 Documentation/security/fgkaslr.rst
> >  create mode 100644 arch/x86/boot/compressed/fgkaslr.c
> >  create mode 100644 arch/x86/boot/compressed/utils.c
> >  create mode 100644 arch/x86/boot/compressed/vmlinux.symbols
> >  create mode 100755 scripts/generate_text_sections.pl
> > 
> > -- 
> > 2.31.1
> > 
> 
> -- 
> Kees Cook

Thanks,
Al

^ permalink raw reply	[flat|nested] 29+ messages in thread

* RE: [PATCH v6 kspp-next 00/22] Function Granular KASLR
  2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
                   ` (21 preceding siblings ...)
  2021-08-31 17:27 ` [PATCH v6 kspp-next 00/22] Function Granular KASLR Kees Cook
@ 2021-09-01 17:16 ` Accardi, Kristen C
  22 siblings, 0 replies; 29+ messages in thread
From: Accardi, Kristen C @ 2021-09-01 17:16 UTC (permalink / raw)
  To: Lobakin, Alexandr, linux-hardening
  Cc: Kristen Carlson Accardi, Luck, Tony, Brandeburg, Jesse, Czapnik,
	Lukasz, Plantykow, Marta A, Kubiak, Michal, Swiatkowski, Michal

(dropping all external emails)

Hey, going external was pretty premature here. We do have a process of internal review required for any patches that touch x86 subsystem that requires getting signed off by x86 reviewers - with a major revision such as this you should have waited until one of the x86 reviewers gave you a reviewed by before going external. I appreciate that this is an own time kind of project for you, but the community just sees one intel and we all need to follow the same standards and practices. Please read the process. 

 https://intelpedia.intel.com/Intel_Patch_Review_Process#Review_Process_for_x86_submissions

-----Original Message-----
From: Lobakin, Alexandr <alexandr.lobakin@intel.com> 
Sent: Tuesday, August 31, 2021 7:41 AM
To: linux-hardening@vger.kernel.org
Cc: Accardi, Kristen C <kristen.c.accardi@intel.com>; Kristen Carlson Accardi <kristen@linux.intel.com>; Kees Cook <keescook@chromium.org>; Masahiro Yamada <masahiroy@kernel.org>; H. Peter Anvin <hpa@zytor.com>; Jessica Yu <jeyu@kernel.org>; Nathan Chancellor <nathan@kernel.org>; Nick Desaulniers <ndesaulniers@google.com>; Marios Pomonis <pomonis@google.com>; Sami Tolvanen <samitolvanen@google.com>; Luck, Tony <tony.luck@intel.com>; Ard Biesheuvel <ardb@kernel.org>; Brandeburg, Jesse <jesse.brandeburg@intel.com>; Czapnik, Lukasz <lukasz.czapnik@intel.com>; Plantykow, Marta A <marta.a.plantykow@intel.com>; Kubiak, Michal <michal.kubiak@intel.com>; Swiatkowski, Michal <michal.swiatkowski@intel.com>; Lobakin, Alexandr <alexandr.lobakin@intel.com>; linux-kbuild@vger.kernel.org; linux-arch@vger.kernel.org; linux-kernel@vger.kernel.org; clang-built-linux@googlegroups.com
Subject: [PATCH v6 kspp-next 00/22] Function Granular KASLR

This is a massive rework and a respin of Kristen Accardi's marvellous FG-KASLR series (v5).

The major differences since v5 [0]:
* You can now tune the number of functions per each section to
  achieve the preferable vmlinux size or protection level. Default
  is still as one section per each function.
  This can be handy for storage-constrained systems. 4-8 fps are
  still strong, but reduce the size of the final vmlinu{x,z}
  significantly;
* I don't use orphan sections anymore. It's not reliable at all /
  may differ from linker to linker, and also conflicts with
  CONFIG_LD_ORPHAN_WARN which is great for catching random bugs ->
* All the .text.* sections are now being described explicitly in the
  linker script. A Perl script is used to take the original LDS, the
  original object file, read a list of input sections from it and
  generate the resulting LDS.
  This costs a bit of linking time as LD tends to think hard when
  processing scripts > 1 Mb. It adds about 40-60 seconds to the
  whole linking process (BTF step, 2-3 kallsyms steps and the final
  step), but "better safe than sorry".
  In addition, that approach allows to reserve some space at the end
  and add some link assertions ->
* Input .text section now must be empty, otherwise the linkage will
  be stopped. This is implemented by the size assertion in the
  resulting LD script and is designed to plug the potentional layout
  leakage. This also means that ->
* "Regular" ASM functions are now being placed into unique separate
  functions the same way compiler does this for C functions. This is
  achieved by introducing and using several new macros which take
  the symbol name as a base for its new section name.
  This gives a better opportunity to both DCE and FG-KASLR, as ASM
  code now can also be randomized or garbage-collected;
* It's now fully compatible with ClangLTO, ClangCFI,
  CONFIG_LD_ORPHAN_WARN and some more stuff landed since the last
  revision was published;
* Includes several fixes: relocations inside .altinstr_replacement
  code and minor issues found and/or suggested by LKP robot.

The series was compile-time and runtime tested on the following setups with no issues:
- x86_64, GCC 11, Binutils 2.35;
- x86_64, Clang/LLVM 12, ClangLTO + ClangCFI (from Sami's tree).

The first 4 patches are from the linux-kbuild tree and included to avoid merge conflicts and non-intuitive resolving of them.

The series is also available here: [1]

[0] https://lore.kernel.org/kernel-hardening/20200923173905.11219-1-kristen@linux.intel.com
[1] https://github.com/alobakin/linux/pull/3

The original v5 cover letter:

Function Granular Kernel Address Space Layout Randomization (fgkaslr)
---------------------------------------------------------------------

This patch set is an implementation of finer grained kernel address space randomization. It rearranges your kernel code at load time on a per-function level granularity, with only around a second added to boot time.

Changes in v5:
--------------
* fixed a bug in the code which increases boot heap size for
  CONFIG_FG_KASLR which prevented the boot heap from being increased
  for CONFIG_FG_KASLR when using bzip2 compression. Thanks to Andy Lavr
  for finding the problem and identifying the solution.
* changed the adjustment of the orc_unwind_ip table at boot time to
  disregard relocs associated with this table, and instead inspect the
  entries separately. Relocs are not able to be used since they are
  no longer correct once the table is resorted at buildtime.
* changed how orc_unwind_ip addresses in randomized sections are identified
  to include the byte immediately after the end of the section.
* updated module code to use kvmalloc/kvfree based on suggestions from
  Evgenii Shatokhin <eshatokhin@virtuozzo.com>.
* changed kernel commandline to disable fgkaslr to simply "nofgkaslr" to
  match the nokaslr option. fgkaslr="X" can be added at a later date
  if it is needed.
* Added a patch to force livepatch to require symbols to be unique if
  using while fgkaslr either for core or modules.

Changes in v4:
-------------
* dropped the patch to split out change to STATIC definition in
  x86/boot/compressed/misc.c and replaced with a patch authored
  by Kees Cook to avoid the duplicate malloc definitions
* Added a section to Documentation/admin-guide/kernel-parameters.txt
  to document the fgkaslr boot option.
* redesigned the patch to hide the new layout when reading
  /proc/kallsyms. The previous implementation utilized a dynamically
  allocated linked list to display the kernel and module symbols
  in alphabetical order. The new implementation uses a randomly
  shuffled index array to display the kernel and module symbols
  in a random order.

Changes in v3:
-------------
* Makefile changes to accommodate CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
* removal of extraneous ALIGN_PAGE from _etext changes
* changed variable names in x86/tools/relocs to be less confusing
* split out change to STATIC definition in x86/boot/compressed/misc.c
* Updates to Documentation to make it more clear what is preserved in .text
* much more detailed commit message for function granular KASLR patch
* minor tweaks and changes that make for more readable code
* this cover letter updated slightly to add additional details

Changes in v2:
--------------
* Fix to address i386 build failure
* Allow module reordering patch to be configured separately so that
  arm (or other non-x86_64 arches) can take advantage of module function
  reordering. This support has not be tested by me, but smoke tested by
  Ard Biesheuvel <ardb@kernel.org> on arm.
* Fix build issue when building on arm as reported by
  Ard Biesheuvel <ardb@kernel.org> 

Patches to objtool are included because they are dependencies for this patchset, however they have been submitted by their maintainer separately.

Background
----------
KASLR was merged into the kernel with the objective of increasing the difficulty of code reuse attacks. Code reuse attacks reused existing code snippets to get around existing memory protections. They exploit software bugs which expose addresses of useful code snippets to control the flow of execution for their own nefarious purposes. KASLR moves the entire kernel code text as a unit at boot time in order to make addresses less predictable.
The order of the code within the segment is unchanged - only the base address is shifted. There are a few shortcomings to this algorithm.

1. Low Entropy - there are only so many locations the kernel can fit in. This
   means an attacker could guess without too much trouble.
2. Knowledge of a single address can reveal the offset of the base address,
   exposing all other locations for a published/known kernel image.
3. Info leaks abound.

Finer grained ASLR has been proposed as a way to make ASLR more resistant to info leaks. It is not a new concept at all, and there are many variations possible. Function reordering is an implementation of finer grained ASLR which randomizes the layout of an address space on a function level granularity. We use the term "fgkaslr" in this document to refer to the technique of function reordering when used with KASLR, as well as finer grained KASLR in general.

Proposed Improvement
--------------------
This patch set proposes adding function reordering on top of the existing KASLR base address randomization. The over-arching objective is incremental improvement over what we already have. It is designed to work in combination with the existing solution. The implementation is really pretty simple, and there are 2 main area where changes occur:

* Build time

GCC has had an option to place functions into individual .text sections for many years now. This option can be used to implement function reordering at load time. The final compiled vmlinux retains all the section headers, which can be used to help find the address ranges of each function. Using this information and an expanded table of relocation addresses, individual text sections can be suffled immediately after decompression. Some data tables inside the kernel that have assumptions about order require re-sorting after being updated when applying relocations. In order to modify these tables, a few key symbols are excluded from the objcopy symbol stripping process for use after shuffling the text segments.

Some highlights from the build time changes to look for:

The top level kernel Makefile was modified to add the gcc flag if it is supported. Currently, I am applying this flag to everything it is possible to randomize. Anything that is written in C and not present in a special input section is randomized. The final binary segment 0 retains a consolidated .text section, as well as all the individual .text.* sections.
Future work could turn off this flags for selected files or even entire subsystems, although obviously at the cost of security.

The relocs tool is updated to add relative relocations. This information previously wasn't included because it wasn't necessary when moving the entire .text segment as a unit. 

A new file was created to contain a list of symbols that objcopy should keep. We use those symbols at load time as described below.

* Load time

The boot kernel was modified to parse the vmlinux elf file after decompression to check for our interesting symbols that we kept, and to look for any .text.* sections to randomize. The consolidated .text section is skipped and not moved. The sections are shuffled randomly, and copied into memory following the .text section in a new random order. The existing code which updated relocation addresses was modified to account for not just a fixed delta from the load address, but the offset that the function section was moved to. This requires inspection of each address to see if it was impacted by a randomization. We use a bsearch to make this less horrible on performance. Any tables that need to be modified with new addresses or resorted are updated using the symbol addresses parsed from the elf symbol table.

In order to hide our new layout, symbols reported through /proc/kallsyms will be displayed in a random order.

Security Considerations
-----------------------
The objective of this patch set is to improve a technology that is already merged into the kernel (KASLR). This code will not prevent all attacks, but should instead be considered as one of several tools that can be used.
In particular, this code is meant to make KASLR more effective in the presence of info leaks.

How much entropy we are adding to the existing entropy of standard KASLR will depend on a few variables. Firstly and most obviously, the number of functions that are randomized matters. This implementation keeps the existing .text section for code that cannot be randomized - for example, because it was assembly code. The less sections to randomize, the less entropy. In addition, due to alignment (16 bytes for x86_64), the number of bits in a address that the attacker needs to guess is reduced, as the lower bits are identical.

Performance Impact
------------------
There are two areas where function reordering can impact performance: boot time latency, and run time performance.

* Boot time latency
This implementation of finer grained KASLR impacts the boot time of the kernel in several places. It requires additional parsing of the kernel ELF file to obtain the section headers of the sections to be randomized. It calls the random number generator for each section to be randomized to determine that section's new memory location. It copies the decompressed kernel into a new area of memory to avoid corruption when laying out the newly randomized sections. It increases the number of relocations the kernel has to perform at boot time vs. standard KASLR, and it also requires a lookup on each address that needs to be relocated to see if it was in a randomized section and needs to be adjusted by a new offset. Finally, it re-sorts a few data tables that are required to be sorted by address.

Booting a test VM on a modern, well appointed system showed an increase in latency of approximately 1 second.

* Run time
The performance impact at run-time of function reordering varies by workload.
Using kcbench, a kernel compilation benchmark, the performance of a kernel build with finer grained KASLR was about 1% slower than a kernel with standard KASLR. Analysis with perf showed a slightly higher percentage of L1-icache-load-misses. Other workloads were examined as well, with varied results. Some workloads performed significantly worse under FGKASLR, while others stayed the same or were mysteriously better. In general, it will depend on the code flow whether or not finer grained KASLR will impact your workload, and how the underlying code was designed. Because the layout changes per boot, each time a system is rebooted the performance of a workload may change.

Future work could identify hot areas that may not be randomized and either leave them in the .text section or group them together into a single section that may be randomized. If grouping things together helps, one other thing to consider is that if we could identify text blobs that should be grouped together to benefit a particular code flow, it could be interesting to explore whether this security feature could be also be used as a performance feature if you are interested in optimizing your kernel layout for a particular workload at boot time. Optimizing function layout for a particular workload has been researched and proven effective - for more information read the Facebook paper "Optimizing Function Placement for Large-Scale Data-Center Applications" (see references section below).

Image Size
----------
Adding additional section headers as a result of compiling with -ffunction-sections will increase the size of the vmlinux ELF file.
With a standard distro config, the resulting vmlinux was increased by about 3%. The compressed image is also increased due to the header files, as well as the extra relocations that must be added. You can expect fgkaslr to increase the size of the compressed image by about 15%.

Memory Usage
------------
fgkaslr increases the amount of heap that is required at boot time, although this extra memory is released when the kernel has finished decompression. As a result, it may not be appropriate to use this feature on systems without much memory.

Building
--------
To enable fine grained KASLR, you need to have the following config options set (including all the ones you would use to build normal KASLR)

CONFIG_FG_KASLR=y

In addition, fgkaslr is only supported for the X86_64 architecture.

Modules
-------
Modules are randomized similarly to the rest of the kernel by shuffling the sections at load time prior to moving them into memory. The module must also have been build with the -ffunction-sections compiler option.

Although fgkaslr for the kernel is only supported for the X86_64 architecture, it is possible to use fgkaslr with modules on other architectures. To enable this feature, select

CONFIG_MODULE_FG_KASLR=y

This option is selected automatically for X86_64 when CONFIG_FG_KASLR is set.

Disabling
---------
Disabling normal KASLR using the nokaslr command line option also disables fgkaslr. It is also possible to disable fgkaslr separately by booting with nofgkaslr on the commandline.

References
----------
There are a lot of academic papers which explore finer grained ASLR.
This paper in particular contributed the most to my implementation design as well as my overall understanding of the problem space:

Selfrando: Securing the Tor Browser against De-anonymization Exploits, M. Conti, S. Crane, T. Frassetto, et al.

For more information on how function layout impacts performance, see:

Optimizing Function Placement for Large-Scale Data-Center Applications, G. Ottoni, B. Maher

Alexander Lobakin (7):
  linkage: add macros for putting ASM functions into own sections
  x86: conditionally place regular ASM functions into separate sections
  FG-KASLR: use a scripted approach to handle .text.* sections
  x86/boot: allow FG-KASLR to be selected
  arm64/crypto: conditionally place ASM functions into separate sections
  module: use a scripted approach for FG-KASLR
  maintainers: add MAINTAINERS entry for FG-KASLR

Kees Cook (2):
  x86/boot: Allow a "silent" kaslr random byte fetch
  x86/boot/compressed: Avoid duplicate malloc() implementations

Kristen Carlson Accardi (9):
  x86: tools/relocs: Support >64K section headers
  x86: Makefile: Add build and config option for CONFIG_FG_KASLR
  Make sure ORC lookup covers the entire _etext - _stext
  x86/tools: Add relative relocs for randomized functions
  x86: Add support for function granular KASLR
  kallsyms: Hide layout
  livepatch: only match unique symbols when using fgkaslr
  module: Reorder functions
  Documentation: add a documentation for FG-KASLR

Masahiro Yamada (3):
  kbuild: merge vmlinux_link() between the ordinary link and Clang LTO
  kbuild: do not remove 'linux' link in scripts/link-vmlinux.sh
  kbuild: merge vmlinux_link() between ARCH=um and other architectures

Sami Tolvanen (1):
  kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG

 .../admin-guide/kernel-parameters.txt         |   6 +
 Documentation/security/fgkaslr.rst            | 172 ++++
 Documentation/security/index.rst              |   1 +
 MAINTAINERS                                   |  12 +
 Makefile                                      |  17 +-
 arch/Kconfig                                  |   3 +
 arch/arm64/crypto/aes-ce-ccm-core.S           |  16 +-
 arch/arm64/crypto/aes-ce-core.S               |  16 +-
 arch/arm64/crypto/aes-ce.S                    |   4 +-
 arch/arm64/crypto/aes-cipher-core.S           |   8 +-
 arch/arm64/crypto/aes-modes.S                 |  16 +-
 arch/arm64/crypto/aes-neon.S                  |   4 +-
 arch/arm64/crypto/aes-neonbs-core.S           |  38 +-
 arch/arm64/crypto/chacha-neon-core.S          |  18 +-
 arch/arm64/crypto/crct10dif-ce-core.S         |  14 +-
 arch/arm64/crypto/ghash-ce-core.S             |  24 +-
 arch/arm64/crypto/nh-neon-core.S              |   4 +-
 arch/arm64/crypto/poly1305-armv8.pl           |  17 +
 arch/arm64/crypto/sha1-ce-core.S              |   4 +-
 arch/arm64/crypto/sha2-ce-core.S              |   4 +-
 arch/arm64/crypto/sha3-ce-core.S              |   4 +-
 arch/arm64/crypto/sha512-armv8.pl             |  11 +
 arch/arm64/crypto/sha512-ce-core.S            |   4 +-
 arch/arm64/crypto/sm3-ce-core.S               |   4 +-
 arch/arm64/crypto/sm4-ce-core.S               |   4 +-
 arch/x86/Kconfig                              |   1 +
 arch/x86/boot/compressed/Makefile             |   9 +-
 arch/x86/boot/compressed/fgkaslr.c            | 905 ++++++++++++++++++
 arch/x86/boot/compressed/kaslr.c              |   4 -
 arch/x86/boot/compressed/misc.c               | 157 ++-
 arch/x86/boot/compressed/misc.h               |  30 +
 arch/x86/boot/compressed/utils.c              |  13 +
 arch/x86/boot/compressed/vmlinux.symbols      |  19 +
 arch/x86/crypto/aegis128-aesni-asm.S          |  36 +-
 arch/x86/crypto/aes_ctrby8_avx-x86_64.S       |  12 +-
 arch/x86/crypto/aesni-intel_asm.S             | 116 ++-
 arch/x86/crypto/aesni-intel_avx-x86_64.S      |  32 +-
 arch/x86/crypto/blake2s-core.S                |   8 +-
 arch/x86/crypto/blowfish-x86_64-asm_64.S      |  16 +-
 arch/x86/crypto/camellia-aesni-avx-asm_64.S   |  28 +-
 arch/x86/crypto/camellia-aesni-avx2-asm_64.S  |  28 +-
 arch/x86/crypto/camellia-x86_64-asm_64.S      |  16 +-
 arch/x86/crypto/cast5-avx-x86_64-asm_64.S     |  24 +-
 arch/x86/crypto/cast6-avx-x86_64-asm_64.S     |  20 +-
 arch/x86/crypto/chacha-avx2-x86_64.S          |  12 +-
 arch/x86/crypto/chacha-avx512vl-x86_64.S      |  12 +-
 arch/x86/crypto/chacha-ssse3-x86_64.S         |  16 +-
 arch/x86/crypto/crc32-pclmul_asm.S            |   4 +-
 arch/x86/crypto/crc32c-pcl-intel-asm_64.S     |   4 +-
 arch/x86/crypto/crct10dif-pcl-asm_64.S        |   4 +-
 arch/x86/crypto/des3_ede-asm_64.S             |   8 +-
 arch/x86/crypto/ghash-clmulni-intel_asm.S     |  12 +-
 arch/x86/crypto/nh-avx2-x86_64.S              |   4 +-
 arch/x86/crypto/nh-sse2-x86_64.S              |   4 +-
 arch/x86/crypto/poly1305-x86_64-cryptogams.pl |   8 +-
 arch/x86/crypto/serpent-avx-x86_64-asm_64.S   |  20 +-
 arch/x86/crypto/serpent-avx2-asm_64.S         |  20 +-
 arch/x86/crypto/serpent-sse2-i586-asm_32.S    |   8 +-
 arch/x86/crypto/serpent-sse2-x86_64-asm_64.S  |   8 +-
 arch/x86/crypto/sha1_avx2_x86_64_asm.S        |   4 +-
 arch/x86/crypto/sha1_ni_asm.S                 |   4 +-
 arch/x86/crypto/sha1_ssse3_asm.S              |   4 +-
 arch/x86/crypto/sha256-avx-asm.S              |   4 +-
 arch/x86/crypto/sha256-avx2-asm.S             |   4 +-
 arch/x86/crypto/sha256-ssse3-asm.S            |   4 +-
 arch/x86/crypto/sha256_ni_asm.S               |   4 +-
 arch/x86/crypto/sha512-avx-asm.S              |   4 +-
 arch/x86/crypto/sha512-avx2-asm.S             |   4 +-
 arch/x86/crypto/sha512-ssse3-asm.S            |   4 +-
 arch/x86/crypto/twofish-avx-x86_64-asm_64.S   |  20 +-
 arch/x86/crypto/twofish-i586-asm_32.S         |   8 +-
 arch/x86/crypto/twofish-x86_64-asm_64-3way.S  |   8 +-
 arch/x86/crypto/twofish-x86_64-asm_64.S       |   8 +-
 arch/x86/entry/entry_32.S                     |  24 +-
 arch/x86/entry/entry_64.S                     |  18 +-
 arch/x86/entry/thunk_32.S                     |   4 +-
 arch/x86/entry/thunk_64.S                     |   8 +-
 arch/x86/include/asm/boot.h                   |  13 +-
 arch/x86/include/asm/paravirt.h               |   2 +-
 arch/x86/include/asm/qspinlock_paravirt.h     |   2 +-
 arch/x86/kernel/acpi/wakeup_32.S              |   9 +-
 arch/x86/kernel/acpi/wakeup_64.S              |  10 +-
 arch/x86/kernel/ftrace_32.S                   |  19 +-
 arch/x86/kernel/ftrace_64.S                   |  28 +-
 arch/x86/kernel/irqflags.S                    |   4 +-
 arch/x86/kernel/kprobes/core.c                |   3 +-
 arch/x86/kernel/kvm.c                         |   2 +-
 arch/x86/kernel/relocate_kernel_32.S          |   2 +
 arch/x86/kernel/relocate_kernel_64.S          |   2 +
 arch/x86/kernel/vmlinux.lds.S                 |   6 +-
 arch/x86/kvm/emulate.c                        |   2 +-
 arch/x86/kvm/vmx/vmenter.S                    |   8 +-
 arch/x86/lib/clear_page_64.S                  |  12 +-
 arch/x86/lib/cmpxchg16b_emu.S                 |   4 +-
 arch/x86/lib/copy_mc_64.S                     |   8 +-
 arch/x86/lib/copy_page_64.S                   |   7 +-
 arch/x86/lib/copy_user_64.S                   |  18 +-
 arch/x86/lib/csum-copy_64.S                   |   4 +-
 arch/x86/lib/error-inject.c                   |   3 +-
 arch/x86/lib/getuser.S                        |  37 +-
 arch/x86/lib/hweight.S                        |   9 +-
 arch/x86/lib/iomap_copy_64.S                  |   4 +-
 arch/x86/lib/kaslr.c                          |  18 +-
 arch/x86/lib/memmove_64.S                     |   4 +-
 arch/x86/lib/memset_64.S                      |  12 +-
 arch/x86/lib/msr-reg.S                        |   8 +-
 arch/x86/lib/putuser.S                        |  18 +-
 arch/x86/mm/mem_encrypt_boot.S                |   8 +-
 arch/x86/platform/efi/efi_stub_64.S           |   4 +-
 arch/x86/platform/efi/efi_thunk_64.S          |   4 +-
 arch/x86/power/hibernate_asm_32.S             |  14 +-
 arch/x86/power/hibernate_asm_64.S             |  14 +-
 arch/x86/tools/relocs.c                       | 135 ++-
 arch/x86/tools/relocs.h                       |   4 +-
 arch/x86/tools/relocs_common.c                |  15 +-
 arch/x86/xen/xen-asm.S                        |  49 +-
 arch/x86/xen/xen-head.S                       |  10 +-
 include/asm-generic/vmlinux.lds.h             |  41 +-
 include/linux/decompress/mm.h                 |  12 +-
 include/linux/linkage.h                       |  76 ++
 include/uapi/linux/elf.h                      |   1 +
 init/Kconfig                                  |  51 +
 kernel/kallsyms.c                             | 158 ++-
 kernel/livepatch/core.c                       |  11 +
 kernel/module.c                               |  91 +-
 scripts/Makefile.build                        |  27 +-
 scripts/Makefile.lib                          |   7 +
 scripts/Makefile.modfinal                     |  36 +-
 scripts/Makefile.modpost                      |  22 +-
 scripts/gen_autoksyms.sh                      |  12 -
 scripts/generate_text_sections.pl             | 149 +++
 scripts/link-vmlinux.sh                       | 104 +-
 scripts/module.lds.S                          |  14 +-
 133 files changed, 2771 insertions(+), 757 deletions(-)  create mode 100644 Documentation/security/fgkaslr.rst
 create mode 100644 arch/x86/boot/compressed/fgkaslr.c
 create mode 100644 arch/x86/boot/compressed/utils.c  create mode 100644 arch/x86/boot/compressed/vmlinux.symbols
 create mode 100755 scripts/generate_text_sections.pl

--
2.31.1


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v6 kspp-next 00/22] Function Granular KASLR
  2021-09-01 10:36   ` Alexander Lobakin
@ 2021-09-02  1:36     ` Kees Cook
  2021-09-03 11:19       ` Alexander Lobakin
  0 siblings, 1 reply; 29+ messages in thread
From: Kees Cook @ 2021-09-02  1:36 UTC (permalink / raw)
  To: Alexander Lobakin
  Cc: linux-hardening, Kristen C Accardi, Kristen Carlson Accardi,
	Masahiro Yamada, H. Peter Anvin, Jessica Yu, Nathan Chancellor,
	Nick Desaulniers, Marios Pomonis, Sami Tolvanen, Tony Luck,
	Ard Biesheuvel, Jesse Brandeburg, Lukasz Czapnik,
	Marta A Plantykow, Michal Kubiak, Michal Swiatkowski,
	linux-kbuild, linux-arch, linux-kernel, clang-built-linux

On Wed, Sep 01, 2021 at 12:36:58PM +0200, Alexander Lobakin wrote:
> Without FG-KASLR, we have only one .text section, and the total
> section number is relatively small.
> With FG-KASLR enabled, we have 40K+ separate text sections (I have
> 40K on a setup with ClangLTO and ClangCFI and about 48K on a
> "regular" one) and each of them is described in the ELF header. Plus
> a separate .rela.text section for every single of them. That's the
> main reason of the size increases.

If you have the size comparisons handy, I'd love to see them. My memory
from v5 was that none of that end up in-core. And in that case, why
limit the entropy of the resulting layout?

> We still have LD_ORPHAN_WARN on non-FG-KASLR builds, but we also
> have a rather different set of sections with FG-KASLR enabled. For
> example, I noticed the appearing of .symtab_shndx section only in
> virtue of LD_ORPHAN_WARN. So it's kinda not the same.

Agreed: I'd rather have LD_ORPHAN_WARN always enabled.

> I don't see a problem in this extra minute. FG-KASLR is all about

But not at this cost. Maybe the x86 maintainers will disagree, but I see
this as a prohibitive cost to doing development work under FGKASLR, and
if we expect this to become the default in distros, no one is going to
be happy with that change. Link time dominates the partial rebuild time,
so my opinion is that it should not be so inflated if not absolutely
needed. Perhaps once the link time bugs in ld.bfd and ld.lld get fixed,
but not now.

> security, and you often pay something for this. We already have a
> size increase, and a small delay while booting, and we can't get
> rid of them. With orphan sections you leave a space for potentional

There's a difference between development time costs and run time costs.
I don't think the LD_ORPHAN_WARN coverage is worth it in this case.

Either way, we need to fix the linker.

> flaws of the code, linker and/or linker script, which is really
> unwanted in case of a security feature.
> After all, ClangLTO increases the linking time at lot, and
> TRIM_UNUSED_KSYMS builds almost the entire kernel two times in a
> row, but nobody complains about this as there's nothing we can do
> with it and it's the price you pay for the optimizations, so again,
> I don't see a problem here.

I get what you mean with regard to getting the perfect situation, but
the kernel went 29 years without LD_ORPHAN_WARN. :) Anyway, we'll see
what other folks think, I guess.

> I still don't get why you're trying to split this series into two.
> It's been almost a year since v5 was published, I doubt you can get
> "basic FG-KASLR" accepted quickly just because it was reviewed back
> then.

Well, because it was blocked then by a single bug, and everything else
you've described are distinct improvements on v5, so to me it makes
sense to have it separated into those phases. I don't mean split the
series, I mean rearrange the series so that a rebased v5 is at the
start, and the improvements follow.

> I prefer to provide a full picture of what I'm trying to bring, so
> the community could review it all and throw much more ideas and
> stuff.

Understood. I am suggesting some ideas about how it might help with
review. :)

> > > * It's now fully compatible with ClangLTO, ClangCFI,
> > >   CONFIG_LD_ORPHAN_WARN and some more stuff landed since the last
> > >   revision was published;
> > 
> > FWIW, v5 was was too. :) I didn't have to do anything to v5 to make it
> > work with ClangLTO and ClangCFI.
> 
> Once again, repeating the thing I wrote earlier in our discussion:
> ClangCFI, at least shadowed implementation, requires the first text
> section of the module to be page-aligned and contain __cfi_check()
> at the very beginning of this section. With FG-KASLR and without
> special handling, this section gets randomized along with the
> others, and ClangCFI either rejects almost all modules or panics
> the kernel.

Ah-ha, thanks. I must have missed your answer to this earlier. I had
probably done my initial v5 testing without modules.

> > Great, this is a good start. One place we saw problems in the past was
> > with i386 build gotchas, so that'll need testing too.
> 
> For now, FG_KASLR for x86 depends on X86_64. We might relax this
> dependency later after enough testing or whatsoever (like it's been
> done for ClangLTO).

Yes, but we've had a history of making big patches that do _intend_ to
break the i386 build, but they do anyway. Hence my question.

> > Sounds good. It might be easier to base the series on linux-next, so a
> > smaller series. Though given the merge window just opened, it might make
> > more sense for a v7 to be based on v5.15-rc2 in three weeks.
> 
> I don't usually base any series on linux-next, because it contains
> all the changes from all "for-next" branches and repos, while the
> series finally gets accepted to the specific repo based on just
> v5.x-rc1 (sometimes on -rc2). This may bring additional apply/merge
> problems.

Understood. I just find it confusing to include patches on lkml that
already exist in a -next branch. Perhaps base on kbuild -next?

> > > Kees Cook (2):
> > >   x86/boot: Allow a "silent" kaslr random byte fetch
> > >   x86/boot/compressed: Avoid duplicate malloc() implementations
> > 
> > These two can get landed right away -- they're standalone fixes that
> > can safely go in -tip.
> > 
> > > 
> > > Kristen Carlson Accardi (9):
> > >   x86: tools/relocs: Support >64K section headers
> > 
> > Same for this.
> 
> They make little to no sense for non-FG-KASLR systems. And none of
> them are "pure" fixes.
> The same could be said about e.g. ORC lookup patch, but again, it
> makes no sense right now.

*shrug* They're trivial changes that have been reviewed before, so it
seems like we can avoid resending them every time.

> > I suspect it'll still be easier to review this series as a rebase v5
> > followed by the evolutionary improvements, since the "basic FGKASLR" has
> > been reviewed in the past, and is fairly noninvasive. The changes for
> > ASM, new .text rules, etc, make a lot more changes that I think would be
> > nice to have separate so reasonable a/b testing can be done.
> 
> I don't see a point in testing it two times instead of just one, as
> well as in delivering this feature in two halves. It sounds like
> "let's introduce ClangLTO, but firstly only for modules, as LTO for
> vmlinux requires changes in objtool code and a special handling for
> the initcalls".
> The changes you mentioned only seem invasive, in fact, they can
> carry way less harm than the "basic FG-KASLR" itself.

Mostly it's a question of building on prior testing (v5 worked), so that
new changes can be debugged if they cause problems. Regardless, it's
been so long, perhaps it won't matter to other reviewers and they'll
want to just start over from scratch.

-Kees

-- 
Kees Cook

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v6 kspp-next 00/22] Function Granular KASLR
  2021-09-02  1:36     ` Kees Cook
@ 2021-09-03 11:19       ` Alexander Lobakin
  0 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-09-03 11:19 UTC (permalink / raw)
  To: Kees Cook
  Cc: Alexander Lobakin, linux-hardening, Kristen C Accardi,
	Kristen Carlson Accardi, Masahiro Yamada, H. Peter Anvin,
	Jessica Yu, Nathan Chancellor, Nick Desaulniers, Marios Pomonis,
	Sami Tolvanen, Tony Luck, Ard Biesheuvel, Jesse Brandeburg,
	Lukasz Czapnik, Marta A Plantykow, Michal Kubiak,
	Michal Swiatkowski, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux

> From: Kees Cook <keescook@chromium.org>
> Date: Wed, 1 Sep 2021 18:36:59 -0700
> 
> On Wed, Sep 01, 2021 at 12:36:58PM +0200, Alexander Lobakin wrote:
> > Without FG-KASLR, we have only one .text section, and the total
> > section number is relatively small.
> > With FG-KASLR enabled, we have 40K+ separate text sections (I have
> > 40K on a setup with ClangLTO and ClangCFI and about 48K on a
> > "regular" one) and each of them is described in the ELF header. Plus
> > a separate .rela.text section for every single of them. That's the
> > main reason of the size increases.
> 
> If you have the size comparisons handy, I'd love to see them. My memory
> from v5 was that none of that end up in-core. And in that case, why
> limit the entropy of the resulting layout?

My testing machine is down for now, but I could send a size
comparison later. It's something about 10 Mb of uncompressed kernel
between 1 and 4 fps or so.

> > We still have LD_ORPHAN_WARN on non-FG-KASLR builds, but we also
> > have a rather different set of sections with FG-KASLR enabled. For
> > example, I noticed the appearing of .symtab_shndx section only in
> > virtue of LD_ORPHAN_WARN. So it's kinda not the same.
> 
> Agreed: I'd rather have LD_ORPHAN_WARN always enabled.
> 
> > I don't see a problem in this extra minute. FG-KASLR is all about
> 
> But not at this cost. Maybe the x86 maintainers will disagree, but I see
> this as a prohibitive cost to doing development work under FGKASLR, and
> if we expect this to become the default in distros, no one is going to
> be happy with that change. Link time dominates the partial rebuild time,
> so my opinion is that it should not be so inflated if not absolutely
> needed. Perhaps once the link time bugs in ld.bfd and ld.lld get fixed,
> but not now.

I don't think FG-KASLR will be enabled by default in distros. Apart
from linking time, it also increases cache misses a lot, and when
it comes to performance critical usecases like high-speed servers
and datacenters, I don't believe their maintainers would consider
FG-KASLR.
Speaking about distros, almost no build systems to my knowledge use
partial building, so this is only a downside for developers.

> > security, and you often pay something for this. We already have a
> > size increase, and a small delay while booting, and we can't get
> > rid of them. With orphan sections you leave a space for potentional
> 
> There's a difference between development time costs and run time costs.
> I don't think the LD_ORPHAN_WARN coverage is worth it in this case.
> 
> Either way, we need to fix the linker.

I agree on that, I was surprised both BFD and LLD choke on big LD
scripts.

> > flaws of the code, linker and/or linker script, which is really
> > unwanted in case of a security feature.
> > After all, ClangLTO increases the linking time at lot, and
> > TRIM_UNUSED_KSYMS builds almost the entire kernel two times in a
> > row, but nobody complains about this as there's nothing we can do
> > with it and it's the price you pay for the optimizations, so again,
> > I don't see a problem here.
> 
> I get what you mean with regard to getting the perfect situation, but
> the kernel went 29 years without LD_ORPHAN_WARN. :) Anyway, we'll see
> what other folks think, I guess.

Also agree, let's wait for more opinions on that, I'm open to
everything.

> > I still don't get why you're trying to split this series into two.
> > It's been almost a year since v5 was published, I doubt you can get
> > "basic FG-KASLR" accepted quickly just because it was reviewed back
> > then.
> 
> Well, because it was blocked then by a single bug, and everything else
> you've described are distinct improvements on v5, so to me it makes
> sense to have it separated into those phases. I don't mean split the
> series, I mean rearrange the series so that a rebased v5 is at the
> start, and the improvements follow.
> 
> > I prefer to provide a full picture of what I'm trying to bring, so
> > the community could review it all and throw much more ideas and
> > stuff.
> 
> Understood. I am suggesting some ideas about how it might help with
> review. :)
> 
> > > > * It's now fully compatible with ClangLTO, ClangCFI,
> > > >   CONFIG_LD_ORPHAN_WARN and some more stuff landed since the last
> > > >   revision was published;
> > > 
> > > FWIW, v5 was was too. :) I didn't have to do anything to v5 to make it
> > > work with ClangLTO and ClangCFI.
> > 
> > Once again, repeating the thing I wrote earlier in our discussion:
> > ClangCFI, at least shadowed implementation, requires the first text
> > section of the module to be page-aligned and contain __cfi_check()
> > at the very beginning of this section. With FG-KASLR and without
> > special handling, this section gets randomized along with the
> > others, and ClangCFI either rejects almost all modules or panics
> > the kernel.
> 
> Ah-ha, thanks. I must have missed your answer to this earlier. I had
> probably done my initial v5 testing without modules.
> 
> > > Great, this is a good start. One place we saw problems in the past was
> > > with i386 build gotchas, so that'll need testing too.
> > 
> > For now, FG_KASLR for x86 depends on X86_64. We might relax this
> > dependency later after enough testing or whatsoever (like it's been
> > done for ClangLTO).
> 
> Yes, but we've had a history of making big patches that do _intend_ to
> break the i386 build, but they do anyway. Hence my question.
> 
> > > Sounds good. It might be easier to base the series on linux-next, so a
> > > smaller series. Though given the merge window just opened, it might make
> > > more sense for a v7 to be based on v5.15-rc2 in three weeks.
> > 
> > I don't usually base any series on linux-next, because it contains
> > all the changes from all "for-next" branches and repos, while the
> > series finally gets accepted to the specific repo based on just
> > v5.x-rc1 (sometimes on -rc2). This may bring additional apply/merge
> > problems.
> 
> Understood. I just find it confusing to include patches on lkml that
> already exist in a -next branch. Perhaps base on kbuild -next?

That's not a problem anymore I believe, since it doesn't hit 5.15
window, so the rebased v7 will be on top of 5.15-rc1 which will
already contain those Kbuild fixes.

> > > > Kees Cook (2):
> > > >   x86/boot: Allow a "silent" kaslr random byte fetch
> > > >   x86/boot/compressed: Avoid duplicate malloc() implementations
> > > 
> > > These two can get landed right away -- they're standalone fixes that
> > > can safely go in -tip.
> > > 
> > > > 
> > > > Kristen Carlson Accardi (9):
> > > >   x86: tools/relocs: Support >64K section headers
> > > 
> > > Same for this.
> > 
> > They make little to no sense for non-FG-KASLR systems. And none of
> > them are "pure" fixes.
> > The same could be said about e.g. ORC lookup patch, but again, it
> > makes no sense right now.
> 
> *shrug* They're trivial changes that have been reviewed before, so it
> seems like we can avoid resending them every time.
> 
> > > I suspect it'll still be easier to review this series as a rebase v5
> > > followed by the evolutionary improvements, since the "basic FGKASLR" has
> > > been reviewed in the past, and is fairly noninvasive. The changes for
> > > ASM, new .text rules, etc, make a lot more changes that I think would be
> > > nice to have separate so reasonable a/b testing can be done.
> > 
> > I don't see a point in testing it two times instead of just one, as
> > well as in delivering this feature in two halves. It sounds like
> > "let's introduce ClangLTO, but firstly only for modules, as LTO for
> > vmlinux requires changes in objtool code and a special handling for
> > the initcalls".
> > The changes you mentioned only seem invasive, in fact, they can
> > carry way less harm than the "basic FG-KASLR" itself.
> 
> Mostly it's a question of building on prior testing (v5 worked), so that
> new changes can be debugged if they cause problems. Regardless, it's
> been so long, perhaps it won't matter to other reviewers and they'll
> want to just start over from scratch.
> 
> -Kees
> 
> -- 
> Kees Cook

Thanks,
Al

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v6 kspp-next 16/22] livepatch: only match unique symbols when using fgkaslr
  2021-08-31 14:41 ` [PATCH v6 kspp-next 16/22] livepatch: only match unique symbols when using fgkaslr Alexander Lobakin
@ 2021-09-09 11:53   ` Miroslav Benes
  2021-09-10 12:29     ` Alexander Lobakin
  0 siblings, 1 reply; 29+ messages in thread
From: Miroslav Benes @ 2021-09-09 11:53 UTC (permalink / raw)
  To: Alexander Lobakin
  Cc: linux-hardening, Kristen C Accardi, Kristen Carlson Accardi,
	Kees Cook, Masahiro Yamada, H. Peter Anvin, Jessica Yu,
	Nathan Chancellor, Nick Desaulniers, Marios Pomonis,
	Sami Tolvanen, Tony Luck, Ard Biesheuvel, Jesse Brandeburg,
	Lukasz Czapnik, Marta A Plantykow, Michal Kubiak,
	Michal Swiatkowski, linux-kbuild, linux-arch, linux-kernel,
	clang-built-linux, live-patching

Hi,

On Tue, 31 Aug 2021, Alexander Lobakin wrote:

> From: Kristen Carlson Accardi <kristen@linux.intel.com>
> 
> If any type of function granular randomization is enabled, the sympos
> algorithm will fail, as it will be impossible to resolve symbols when
> there are duplicates using the previous symbol position.
> 
> Override the value of sympos to always be zero if fgkaslr is enabled for
> either the core kernel or modules, forcing the algorithm
> to require that only unique symbols are allowed to be patched.
> 
> Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
> Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
> ---
>  kernel/livepatch/core.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> index 335d988bd811..852bbfa9da7b 100644
> --- a/kernel/livepatch/core.c
> +++ b/kernel/livepatch/core.c
> @@ -169,6 +169,17 @@ static int klp_find_object_symbol(const char *objname, const char *name,
>  	else
>  		kallsyms_on_each_symbol(klp_find_callback, &args);
>  
> +	/*
> +	 * If any type of function granular randomization is enabled, it
> +	 * will be impossible to resolve symbols when there are duplicates
> +	 * using the previous symbol position (i.e. sympos != 0). Override
> +	 * the value of sympos to always be zero in this case. This will
> +	 * force the algorithm to require that only unique symbols are
> +	 * allowed to be patched.
> +	 */
> +	if (IS_ENABLED(CONFIG_FG_KASLR))
> +		sympos = 0;
> +

I ran the live patching tests and no problem occurred, which is great. We 
do not have a test for old_sympos, which makes the testing less telling, 
but at least nothing blows up with the section randomization itself.

However, I want to reiterate what I wrote for the same patch in v5 
series.

The above hunk should work, but I wonder if we should make it more 
explicit. With the change the user will get the error with "unresolvable 
ambiguity for symbol..." if they specify sympos and the symbol is not 
unique. It could confuse them.

So, how about it making it something like

if (IS_ENABLED(CONFIG_FG_KASLR) || IS_ENABLED(CONFIG_MODULE_FG_KASLR))
        if (sympos) {
                pr_err("fgkaslr is enabled, specifying sympos for symbol '%s' in object '%s' does not work.\n",
                        name, objname);
                *addr = 0;
                return -EINVAL;
        }

? (there could be goto to the error out at the end of the function to 
save copy-pasting).

In that case, if sympos is not specified, the user will get the message 
which matches the reality. If the user specifies it, they will get the 
error in case of fgkaslr (no matter if the symbol is found or not).

What do you think?

Miroslav

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: [PATCH v6 kspp-next 16/22] livepatch: only match unique symbols when using fgkaslr
  2021-09-09 11:53   ` Miroslav Benes
@ 2021-09-10 12:29     ` Alexander Lobakin
  0 siblings, 0 replies; 29+ messages in thread
From: Alexander Lobakin @ 2021-09-10 12:29 UTC (permalink / raw)
  To: Miroslav Benes
  Cc: Alexander Lobakin, linux-hardening, Kristen C Accardi,
	Kristen Carlson Accardi, Kees Cook, Masahiro Yamada,
	H. Peter Anvin, Jessica Yu, Nathan Chancellor, Nick Desaulniers,
	Marios Pomonis, Sami Tolvanen, Tony Luck, Ard Biesheuvel,
	Jesse Brandeburg, Lukasz Czapnik, Marta A Plantykow,
	Michal Kubiak, Michal Swiatkowski, linux-kbuild, linux-arch,
	linux-kernel, clang-built-linux, live-patching

From: Miroslav Benes <mbenes@suse.cz>
Date: Thu, 9 Sep 2021 13:53:35 +0200 (CEST)

> Hi,

Hi!

> On Tue, 31 Aug 2021, Alexander Lobakin wrote:
> 
> > From: Kristen Carlson Accardi <kristen@linux.intel.com>
> > 
> > If any type of function granular randomization is enabled, the sympos
> > algorithm will fail, as it will be impossible to resolve symbols when
> > there are duplicates using the previous symbol position.
> > 
> > Override the value of sympos to always be zero if fgkaslr is enabled for
> > either the core kernel or modules, forcing the algorithm
> > to require that only unique symbols are allowed to be patched.
> > 
> > Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
> > Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com>
> > ---
> >  kernel/livepatch/core.c | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
> > 
> > diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
> > index 335d988bd811..852bbfa9da7b 100644
> > --- a/kernel/livepatch/core.c
> > +++ b/kernel/livepatch/core.c
> > @@ -169,6 +169,17 @@ static int klp_find_object_symbol(const char *objname, const char *name,
> >  	else
> >  		kallsyms_on_each_symbol(klp_find_callback, &args);
> >  
> > +	/*
> > +	 * If any type of function granular randomization is enabled, it
> > +	 * will be impossible to resolve symbols when there are duplicates
> > +	 * using the previous symbol position (i.e. sympos != 0). Override
> > +	 * the value of sympos to always be zero in this case. This will
> > +	 * force the algorithm to require that only unique symbols are
> > +	 * allowed to be patched.
> > +	 */
> > +	if (IS_ENABLED(CONFIG_FG_KASLR))
> > +		sympos = 0;
> > +
> 
> I ran the live patching tests and no problem occurred, which is great. We 
> do not have a test for old_sympos, which makes the testing less telling, 
> but at least nothing blows up with the section randomization itself.

Great, thanks!

> However, I want to reiterate what I wrote for the same patch in v5 
> series.
> 
> The above hunk should work, but I wonder if we should make it more 
> explicit. With the change the user will get the error with "unresolvable 
> ambiguity for symbol..." if they specify sympos and the symbol is not 
> unique. It could confuse them.
> 
> So, how about it making it something like
> 
> if (IS_ENABLED(CONFIG_FG_KASLR) || IS_ENABLED(CONFIG_MODULE_FG_KASLR))
>         if (sympos) {
>                 pr_err("fgkaslr is enabled, specifying sympos for symbol '%s' in object '%s' does not work.\n",
>                         name, objname);
>                 *addr = 0;
>                 return -EINVAL;
>         }
> 
> ? (there could be goto to the error out at the end of the function to 
> save copy-pasting).
> 
> In that case, if sympos is not specified, the user will get the message 
> which matches the reality. If the user specifies it, they will get the 
> error in case of fgkaslr (no matter if the symbol is found or not).

Not familiar with livepatching unfortunately, hope Kristen and/or
Kees will comment on this. Looks fine for me anyways.

> What do you think?
> 
> Miroslav

Thanks,
Al

^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2021-09-10 12:30 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-31 14:40 [PATCH v6 kspp-next 00/22] Function Granular KASLR Alexander Lobakin
2021-08-31 14:40 ` [PATCH v6 kspp-next 01/22] kbuild: Fix TRIM_UNUSED_KSYMS with LTO_CLANG Alexander Lobakin
2021-08-31 14:40 ` [PATCH v6 kspp-next 02/22] kbuild: merge vmlinux_link() between the ordinary link and Clang LTO Alexander Lobakin
2021-08-31 14:40 ` [PATCH v6 kspp-next 03/22] kbuild: do not remove 'linux' link in scripts/link-vmlinux.sh Alexander Lobakin
2021-08-31 14:40 ` [PATCH v6 kspp-next 04/22] kbuild: merge vmlinux_link() between ARCH=um and other architectures Alexander Lobakin
2021-08-31 14:40 ` [PATCH v6 kspp-next 05/22] x86: tools/relocs: Support >64K section headers Alexander Lobakin
2021-08-31 14:40 ` [PATCH v6 kspp-next 06/22] x86/boot: Allow a "silent" kaslr random byte fetch Alexander Lobakin
2021-08-31 14:40 ` [PATCH v6 kspp-next 07/22] x86: Makefile: Add build and config option for CONFIG_FG_KASLR Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 08/22] Make sure ORC lookup covers the entire _etext - _stext Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 09/22] x86/tools: Add relative relocs for randomized functions Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 10/22] x86/boot/compressed: Avoid duplicate malloc() implementations Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 11/22] x86: Add support for function granular KASLR Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 12/22] linkage: add macros for putting ASM functions into own sections Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 14/22] FG-KASLR: use a scripted approach to handle .text.* sections Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 15/22] kallsyms: Hide layout Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 16/22] livepatch: only match unique symbols when using fgkaslr Alexander Lobakin
2021-09-09 11:53   ` Miroslav Benes
2021-09-10 12:29     ` Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 17/22] x86/boot: allow FG-KASLR to be selected Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 18/22] arm64/crypto: conditionally place ASM functions into separate sections Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 19/22] module: Reorder functions Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 20/22] module: use a scripted approach for FG-KASLR Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 21/22] Documentation: add a documentation " Alexander Lobakin
2021-08-31 14:41 ` [PATCH v6 kspp-next 22/22] maintainers: add MAINTAINERS entry " Alexander Lobakin
2021-08-31 17:27 ` [PATCH v6 kspp-next 00/22] Function Granular KASLR Kees Cook
2021-09-01 10:36   ` Alexander Lobakin
2021-09-02  1:36     ` Kees Cook
2021-09-03 11:19       ` Alexander Lobakin
2021-09-01 17:16 ` Accardi, Kristen C

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).