Linux v5.19-rc6: Building and testing x86/retbleed

Linux v5.19-rc6: Building and testing x86/retbleed

[Sedat Dilek]

Hi,

as far as I can see to fully support x86/retbleed you need LLVM-15?

Due to:

[ arch/x86/Kconfig ]
config CC_HAS_RETURN_THUNK
def_bool $(cc-option,-mfunction-return=thunk-extern)

Has someone a commit-id which supports above and is stable/safe?

Looks like apt.llvm.org ships new clang-15 packages, again (see P.S.).

Has anyone numbers w/ and w/o enabling x86/retbleed?

Thanks.

Regards,
-Sedat-

P.S.: List of available clang-15 packages in Debian AMD64 world.

$ LC_ALL=C apt-cache policy clang-15
clang-15:
 Installed: (none)
 Candidate: 1:15~++20220712100758+8fe076ffe090-1~exp1~20220712220909.616
 Version table:
    1:15~++20220712100758+8fe076ffe090-1~exp1~20220712220909.616 99
        99 https://apt.llvm.org/unstable llvm-toolchain/main amd64 Packages
    1:15~++20220625103012+3d37e785c77a-1~exp1 99
        99 https://ftp.debian.org/debian unstable/main amd64 Packages
        99 https://deb.debian.org/debian unstable/main amd64 Packages
    1:15~++20220309105819+8bef17ed59aa-1~exp1 99
        99 https://ftp.debian.org/debian experimental/main amd64 Packages
        99 https://deb.debian.org/debian experimental/main amd64 Packages

-EOT-

Re: Linux v5.19-rc6: Building and testing x86/retbleed

[Nathan Chancellor]

Hi Sedat!

On Thu, Jul 14, 2022 at 11:49:33AM +0200, Sedat Dilek wrote:
> Hi,
> 
> as far as I can see to fully support x86/retbleed you need LLVM-15?

> Due to:
> 
> [ arch/x86/Kconfig ]
> config CC_HAS_RETURN_THUNK
> def_bool $(cc-option,-mfunction-return=thunk-extern)

Per the documentation, rethunks are only effective for certain AMD Zen
platforms so unless you have one of those CPUs, you don't need a new
compiler for the retbleed mitigations. Additionally, you might have an
old enough CPU that doesn't require them anyways, the affected models
can be seen in the cpu_vuln_blacklist variable in
arch/x86/kernel/cpu/common.c.

For the record, here is the output of the retbleed sysfs node on all of
my x86 hardware:

AMD Threadripper 3990X: Mitigation: untrained return thunk; SMT enabled with STIBP protection
Intel Core i7-11700: Mitigation: Enhanced IBRS
AMD Ryzen 3 4300G: Mitigation: untrained return thunk; SMT enabled with STIBP protection
Intel Core i5-4210U: Not affected

> Has someone a commit-id which supports above and is stable/safe?

This is the latest compiler I built and everything appears to work fine:

$ clang --version | head -1
ClangBuiltLinux clang version 15.0.0 (https://github.com/llvm/llvm-project 0406c0cda675f3cb7d294a3e65eb4f19c9efe98b)

> Has anyone numbers w/ and w/o enabling x86/retbleed?

I haven't noticed any major slow downs while compiling kernels, which is
the main use case for my workstation, but it is a Zen platform so I am
not paying the IBPB penalty, which will be worse. Some stats of my full
set of builds against linux-next and mainline:

Before:

linux-next: 2h 29m 44s
mainline: 2h 21m 49s

After:

linux-next: 2h 34m 04s
mainline: 2h 23m 53s

I would say that is within the noise of thermal throttling and other
variables.

Cheers,
Nathan

Re: Linux v5.19-rc6: Building and testing x86/retbleed

[Sedat Dilek]

On Thu, Jul 14, 2022 at 5:55 PM Nathan Chancellor <nathan@kernel.org> wrote:
>

Hi Nathan,

today, I had some time for testing.

> Hi Sedat!
>
> On Thu, Jul 14, 2022 at 11:49:33AM +0200, Sedat Dilek wrote:
> > Hi,
> >
> > as far as I can see to fully support x86/retbleed you need LLVM-15?
>
> > Due to:
> >
> > [ arch/x86/Kconfig ]
> > config CC_HAS_RETURN_THUNK
> > def_bool $(cc-option,-mfunction-return=thunk-extern)
>
> Per the documentation, rethunks are only effective for certain AMD Zen
> platforms so unless you have one of those CPUs, you don't need a new
> compiler for the retbleed mitigations. Additionally, you might have an
> old enough CPU that doesn't require them anyways, the affected models
> can be seen in the cpu_vuln_blacklist variable in
> arch/x86/kernel/cpu/common.c.
>
> For the record, here is the output of the retbleed sysfs node on all of
> my x86 hardware:
>
> AMD Threadripper 3990X: Mitigation: untrained return thunk; SMT enabled with STIBP protection
> Intel Core i7-11700: Mitigation: Enhanced IBRS
> AMD Ryzen 3 4300G: Mitigation: untrained return thunk; SMT enabled with STIBP protection
> Intel Core i5-4210U: Not affected
>

Thanks for these details, but my question was about compile-ability.

> > Has someone a commit-id which supports above and is stable/safe?
>
> This is the latest compiler I built and everything appears to work fine:
>
> $ clang --version | head -1
> ClangBuiltLinux clang version 15.0.0 (https://github.com/llvm/llvm-project 0406c0cda675f3cb7d294a3e65eb4f19c9efe98b)
>

I was able to build and boot...

# cat /proc/version
Linux version 5.19.0-rc6-1-amd64-clang15-lto
(sedat.dilek@gmail.com@iniza) (Debian clang version
15.0.0-++20220716100712+45067f8fbf61-1~exp1~20220716220809.624, Debian
LLD 15.0.0) #1~bookworm+dileks1 SMP PREEMPT_DYNAMIC 2022-07-17

[ DIFFCONFIG ]

$ scripts/diffconfig ../configs/config-5.19.0-rc6-1-amd64-clang14-lto
/boot/config-5.19.0-rc6-1-amd64-clang15-lto | egrep -v
'XEN_VIRTIO|XEN_GRANT_DMA_OPS|INTEL_IFS'
AS_VERSION 140005 -> 150000
BUILD_SALT "5.19.0-rc6-1-amd64-clang14-lto" -> "5.19.0-rc6-3-amd64-clang15-lto"
CC_VERSION_TEXT "dileks clang version 14.0.5
(https://github.com/llvm/llvm-project.git
c12386ae247c0d46e1d513942e322e3a0510b126)" -> "Debian clang version
15.0.0-++20220716100712+45067f8fbf61-1~exp1~20220716220809.624"
CLANG_VERSION 140005 -> 150000
LLD_VERSION 140005 -> 150000
+CC_HAS_RANDSTRUCT y
+CC_HAS_RETURN_THUNK y
+CC_HAS_SLS y
+CC_HAS_ZERO_CALL_USED_REGS y
+CPU_IBPB_ENTRY y
+CPU_IBRS_ENTRY y
+CPU_UNRET_ENTRY y
+RANDSTRUCT_FULL n
+RETHUNK y
+SLS y
+SPECULATION_MITIGATIONS y
+ZERO_CALL_USED_REGS n

Have set SLS y (as Debian enabled it) but cannot say what is with
ZERO_CALL_USED_REGS and RANDSTRUCT_FULL Kconfigs - disabled for now.

[ WARNINGS ]

Known warnings:

$ grep warning: build-log_5.19.0-rc6-1-amd64-clang15-lto.txt
59:dpkg-architecture: warning: specified GNU system type
x86_64-linux-gnu does not match CC system type x86_64-pc-linux-gnu,
try setting a correct CC environment variable
45362:drivers/net/wireless/realtek/rtlwifi/rtl8192cu/rtl8192cu.o:
warning: objtool: .text.rtl92cu_hw_init: unexpected end of section

^^ https://github.com/ClangBuiltLinux/linux/issues/1613
https://github.com/ClangBuiltLinux/linux/issues/1613#issuecomment-1077949287

New warnings:

61600:vmlinux.o: warning: objtool:
.text.seg6_local_fill_encap.llvm.12745529623442014871: unexpected end
of section
61601:vmlinux.o: warning: objtool:
.text.seg6_local_cmp_encap.llvm.12745529623442014871: unexpected end
of section

[ MITIGATIONS ]

# for v in $( ls /sys/devices/system/cpu/vulnerabilities/ ) ; do echo
[ $v ] ; cat /sys/devices/system/cpu/vulnerabilities/$v ; done
[ itlb_multihit ]
KVM: Mitigation: VMX disabled
[ l1tf ]
Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable
[ mds ]
Mitigation: Clear CPU buffers; SMT vulnerable
[ meltdown ]
Mitigation: PTI
[ mmio_stale_data ]
Not affected
[ retbleed ]
Not affected
[ spec_store_bypass ]
Mitigation: Speculative Store Bypass disabled via prctl
[ spectre_v1 ]
Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[ spectre_v2 ]
Mitigation: Retpolines, IBPB: conditional, IBRS_FW, STIBP:
conditional, RSB filling
[ srbds ]
Not affected
[ tsx_async_abort ]
Not affected

CPU: Intel SandyBridge

> > Has anyone numbers w/ and w/o enabling x86/retbleed?
>
> I haven't noticed any major slow downs while compiling kernels, which is
> the main use case for my workstation, but it is a Zen platform so I am
> not paying the IBPB penalty, which will be worse. Some stats of my full
> set of builds against linux-next and mainline:
>
> Before:
>
> linux-next: 2h 29m 44s
> mainline: 2h 21m 49s
>
> After:
>
> linux-next: 2h 34m 04s
> mainline: 2h 23m 53s
>
> I would say that is within the noise of thermal throttling and other
> variables.
>

Thanks for the numbers, Nathan.

Regards,
-Sedat-

[ x86/retbleed: Add call depth tracking mitigation ] (Skylake)
https://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git/log/?h=depthtracking