[PATCH] vmlinux.lds.h: Handle clang's module.{c,d}tor sections

[PATCH] vmlinux.lds.h: Handle clang's module.{c,d}tor sections

[Nathan Chancellor]

A recent change in LLVM causes module_{c,d}tor sections to appear when
CONFIG_K{A,C}SAN are enabled, which results in orphan section warnings
because these are not handled anywhere:

ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_ctor) is being placed in '.text.asan.module_ctor'
ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_dtor) is being placed in '.text.asan.module_dtor'
ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.tsan.module_ctor) is being placed in '.text.tsan.module_ctor'

Place them in the TEXT_TEXT section so that these technologies continue
to work with the newer compiler versions. All of the KASAN and KCSAN
KUnit tests continue to pass after this change.

Cc: stable@vger.kernel.org
Link: https://github.com/ClangBuiltLinux/linux/issues/1432
Link: https://github.com/llvm/llvm-project/commit/7b789562244ee941b7bf2cefeb3fc08a59a01865
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
---
 include/asm-generic/vmlinux.lds.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 17325416e2de..3b79b1e76556 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -586,6 +586,7 @@
 		NOINSTR_TEXT						\
 		*(.text..refcount)					\
 		*(.ref.text)						\
+		*(.text.asan .text.asan.*)				\
 		TEXT_CFI_JT						\
 	MEM_KEEP(init.text*)						\
 	MEM_KEEP(exit.text*)						\

base-commit: 4669e13cd67f8532be12815ed3d37e775a9bdc16
-- 
2.32.0.264.g75ae10bc75

Re: [PATCH] vmlinux.lds.h: Handle clang's module.{c,d}tor sections

[Nick Desaulniers]

On Fri, Jul 30, 2021 at 3:38 PM Nathan Chancellor <nathan@kernel.org> wrote:
>
> A recent change in LLVM causes module_{c,d}tor sections to appear when
> CONFIG_K{A,C}SAN are enabled, which results in orphan section warnings
> because these are not handled anywhere:
>
> ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_ctor) is being placed in '.text.asan.module_ctor'
> ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_dtor) is being placed in '.text.asan.module_dtor'
> ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.tsan.module_ctor) is being placed in '.text.tsan.module_ctor'

^ .text.tsan.*

>
> Place them in the TEXT_TEXT section so that these technologies continue
> to work with the newer compiler versions. All of the KASAN and KCSAN
> KUnit tests continue to pass after this change.
>
> Cc: stable@vger.kernel.org
> Link: https://github.com/ClangBuiltLinux/linux/issues/1432
> Link: https://github.com/llvm/llvm-project/commit/7b789562244ee941b7bf2cefeb3fc08a59a01865
> Signed-off-by: Nathan Chancellor <nathan@kernel.org>
> ---
>  include/asm-generic/vmlinux.lds.h | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index 17325416e2de..3b79b1e76556 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -586,6 +586,7 @@
>                 NOINSTR_TEXT                                            \
>                 *(.text..refcount)                                      \
>                 *(.ref.text)                                            \
> +               *(.text.asan .text.asan.*)                              \

Will this match .text.tsan.module_ctor?

Do we want to add these conditionally on
CONFIG_KASAN_GENERIC/CONFIG_KCSAN like we do for SANITIZER_DISCARDS?

>                 TEXT_CFI_JT                                             \
>         MEM_KEEP(init.text*)                                            \
>         MEM_KEEP(exit.text*)                                            \
>
> base-commit: 4669e13cd67f8532be12815ed3d37e775a9bdc16
> --


-- 
Thanks,
~Nick Desaulniers

Re: [PATCH] vmlinux.lds.h: Handle clang's module.{c,d}tor sections

[Fangrui Song]

On 2021-07-30, Nick Desaulniers wrote:
>On Fri, Jul 30, 2021 at 3:38 PM Nathan Chancellor <nathan@kernel.org> wrote:
>>
>> A recent change in LLVM causes module_{c,d}tor sections to appear when
>> CONFIG_K{A,C}SAN are enabled, which results in orphan section warnings
>> because these are not handled anywhere:
>>
>> ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_ctor) is being placed in '.text.asan.module_ctor'
>> ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_dtor) is being placed in '.text.asan.module_dtor'
>> ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.tsan.module_ctor) is being placed in '.text.tsan.module_ctor'
>
>^ .text.tsan.*

I was wondering why the orphan section warning only arose recently.
Now I see: the function asan.module_ctor has the SHF_GNU_RETAIN flag, so
it is in a separate section even with -fno-function-sections (default).

It seems that with -ffunction-sections the issue should have been caught
much earlier.

>>
>> Place them in the TEXT_TEXT section so that these technologies continue
>> to work with the newer compiler versions. All of the KASAN and KCSAN
>> KUnit tests continue to pass after this change.
>>
>> Cc: stable@vger.kernel.org
>> Link: https://github.com/ClangBuiltLinux/linux/issues/1432
>> Link: https://github.com/llvm/llvm-project/commit/7b789562244ee941b7bf2cefeb3fc08a59a01865
>> Signed-off-by: Nathan Chancellor <nathan@kernel.org>
>> ---
>>  include/asm-generic/vmlinux.lds.h | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
>> index 17325416e2de..3b79b1e76556 100644
>> --- a/include/asm-generic/vmlinux.lds.h
>> +++ b/include/asm-generic/vmlinux.lds.h
>> @@ -586,6 +586,7 @@
>>                 NOINSTR_TEXT                                            \
>>                 *(.text..refcount)                                      \
>>                 *(.ref.text)                                            \
>> +               *(.text.asan .text.asan.*)                              \
>
>Will this match .text.tsan.module_ctor?

asan.module_ctor is the only function AddressSanitizer synthesizes in the instrumented translation unit.
There is no function called "asan".

(Even if a function "asan" exists due to -ffunction-sections
-funique-section-names, TEXT_MAIN will match .text.asan, so the
.text.asan pattern will match nothing.)

>Do we want to add these conditionally on
>CONFIG_KASAN_GENERIC/CONFIG_KCSAN like we do for SANITIZER_DISCARDS?
>
>>                 TEXT_CFI_JT                                             \
>>         MEM_KEEP(init.text*)                                            \
>>         MEM_KEEP(exit.text*)                                            \
>>
>> base-commit: 4669e13cd67f8532be12815ed3d37e775a9bdc16
>> --
>
>
>-- 
>Thanks,
>~Nick Desaulniers

Re: [PATCH] vmlinux.lds.h: Handle clang's module.{c,d}tor sections

[Nathan Chancellor]

On 7/30/2021 3:59 PM, Fangrui Song wrote:
> On 2021-07-30, Nick Desaulniers wrote:
>> On Fri, Jul 30, 2021 at 3:38 PM Nathan Chancellor <nathan@kernel.org> 
>> wrote:
>>>
>>> A recent change in LLVM causes module_{c,d}tor sections to appear when
>>> CONFIG_K{A,C}SAN are enabled, which results in orphan section warnings
>>> because these are not handled anywhere:
>>>
>>> ld.lld: warning: 
>>> arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_ctor) is being 
>>> placed in '.text.asan.module_ctor'
>>> ld.lld: warning: 
>>> arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_dtor) is being 
>>> placed in '.text.asan.module_dtor'
>>> ld.lld: warning: 
>>> arch/x86/pci/built-in.a(legacy.o):(.text.tsan.module_ctor) is being 
>>> placed in '.text.tsan.module_ctor'
>>
>> ^ .text.tsan.*
> 
> I was wondering why the orphan section warning only arose recently.
> Now I see: the function asan.module_ctor has the SHF_GNU_RETAIN flag, so
> it is in a separate section even with -fno-function-sections (default).

Thanks for the explanation, I will add this to the commit message.

> It seems that with -ffunction-sections the issue should have been caught
> much earlier.
> 
>>>
>>> Place them in the TEXT_TEXT section so that these technologies continue
>>> to work with the newer compiler versions. All of the KASAN and KCSAN
>>> KUnit tests continue to pass after this change.
>>>
>>> Cc: stable@vger.kernel.org
>>> Link: https://github.com/ClangBuiltLinux/linux/issues/1432
>>> Link: 
>>> https://github.com/llvm/llvm-project/commit/7b789562244ee941b7bf2cefeb3fc08a59a01865 
>>>
>>> Signed-off-by: Nathan Chancellor <nathan@kernel.org>
>>> ---
>>>  include/asm-generic/vmlinux.lds.h | 1 +
>>>  1 file changed, 1 insertion(+)
>>>
>>> diff --git a/include/asm-generic/vmlinux.lds.h 
>>> b/include/asm-generic/vmlinux.lds.h
>>> index 17325416e2de..3b79b1e76556 100644
>>> --- a/include/asm-generic/vmlinux.lds.h
>>> +++ b/include/asm-generic/vmlinux.lds.h
>>> @@ -586,6 +586,7 @@
>>>                 
>>> NOINSTR_TEXT                                            \
>>>                 
>>> *(.text..refcount)                                      \
>>>                 
>>> *(.ref.text)                                            \
>>> +               *(.text.asan 
>>> .text.asan.*)                              \
>>
>> Will this match .text.tsan.module_ctor?

No, I forgot to test CONFIG_KCSAN with this version, rather than the 
prior one I had on GitHub so I will send v2 shortly.

> asan.module_ctor is the only function AddressSanitizer synthesizes in 
> the instrumented translation unit.
> There is no function called "asan".
> 
> (Even if a function "asan" exists due to -ffunction-sections
> -funique-section-names, TEXT_MAIN will match .text.asan, so the
> .text.asan pattern will match nothing.)

Sounds good, I will update it to remove the .text.asan and replace it 
with .text.tsan.*

>> Do we want to add these conditionally on
>> CONFIG_KASAN_GENERIC/CONFIG_KCSAN like we do for SANITIZER_DISCARDS?

I do not think there is a point in doing so but I can if others feel 
strongly.

Thank you both for the comments for the comments!

Cheers,
Nathan

[PATCH v2] vmlinux.lds.h: Handle clang's module.{c,d}tor sections

[Nathan Chancellor]

A recent change in LLVM causes module_{c,d}tor sections to appear when
CONFIG_K{A,C}SAN are enabled, which results in orphan section warnings
because these are not handled anywhere:

ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_ctor) is being placed in '.text.asan.module_ctor'
ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_dtor) is being placed in '.text.asan.module_dtor'
ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.tsan.module_ctor) is being placed in '.text.tsan.module_ctor'

Fangrui explains: "the function asan.module_ctor has the SHF_GNU_RETAIN
flag, so it is in a separate section even with -fno-function-sections
(default)".

Place them in the TEXT_TEXT section so that these technologies continue
to work with the newer compiler versions. All of the KASAN and KCSAN
KUnit tests continue to pass after this change.

Cc: stable@vger.kernel.org
Link: https://github.com/ClangBuiltLinux/linux/issues/1432
Link: https://github.com/llvm/llvm-project/commit/7b789562244ee941b7bf2cefeb3fc08a59a01865
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
---

v1 -> v2:

* Fix inclusion of .text.tsan.* (Nick)

* Drop .text.asan as it does not exist plus it would be handled by a
  different line (Fangrui)

* Add Fangrui's explanation about why the LLVM commit caused these
  sections to appear.

 include/asm-generic/vmlinux.lds.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 17325416e2de..62669b36a772 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -586,6 +586,7 @@
 		NOINSTR_TEXT						\
 		*(.text..refcount)					\
 		*(.ref.text)						\
+		*(.text.asan.* .text.tsan.*)				\
 		TEXT_CFI_JT						\
 	MEM_KEEP(init.text*)						\
 	MEM_KEEP(exit.text*)						\

base-commit: 4669e13cd67f8532be12815ed3d37e775a9bdc16
-- 
2.32.0.264.g75ae10bc75

Re: [PATCH v2] vmlinux.lds.h: Handle clang's module.{c,d}tor sections

[Fangrui Song]

Reviewed-by: Fangrui Song <maskray@google.com>

On 2021-07-30, Nathan Chancellor wrote:
>A recent change in LLVM causes module_{c,d}tor sections to appear when
>CONFIG_K{A,C}SAN are enabled, which results in orphan section warnings
>because these are not handled anywhere:
>
>ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_ctor) is being placed in '.text.asan.module_ctor'
>ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_dtor) is being placed in '.text.asan.module_dtor'
>ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.tsan.module_ctor) is being placed in '.text.tsan.module_ctor'
>
>Fangrui explains: "the function asan.module_ctor has the SHF_GNU_RETAIN
>flag, so it is in a separate section even with -fno-function-sections
>(default)".

If my theory is true, we should see orphan section warning with
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
before my sanitizer change.

>Place them in the TEXT_TEXT section so that these technologies continue
>to work with the newer compiler versions. All of the KASAN and KCSAN
>KUnit tests continue to pass after this change.
>
>Cc: stable@vger.kernel.org
>Link: https://github.com/ClangBuiltLinux/linux/issues/1432
>Link: https://github.com/llvm/llvm-project/commit/7b789562244ee941b7bf2cefeb3fc08a59a01865
>Signed-off-by: Nathan Chancellor <nathan@kernel.org>
>---
>
>v1 -> v2:
>
>* Fix inclusion of .text.tsan.* (Nick)
>
>* Drop .text.asan as it does not exist plus it would be handled by a
>  different line (Fangrui)
>
>* Add Fangrui's explanation about why the LLVM commit caused these
>  sections to appear.
>
> include/asm-generic/vmlinux.lds.h | 1 +
> 1 file changed, 1 insertion(+)
>
>diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
>index 17325416e2de..62669b36a772 100644
>--- a/include/asm-generic/vmlinux.lds.h
>+++ b/include/asm-generic/vmlinux.lds.h
>@@ -586,6 +586,7 @@
> 		NOINSTR_TEXT						\
> 		*(.text..refcount)					\
> 		*(.ref.text)						\
>+		*(.text.asan.* .text.tsan.*)				\

When kmsan is upstreamed, we may need to add .text.msan.* :)

(
I wondered why we cannot just change the TEXT_MAIN pattern to .text.*

For large userspace applications, separating .text.unlikely .text.hot can help
do things like hugepage and mlock, which can improve instruction cache
localize and reduce instruction TLB miss rates,,, but not sure this
helps much for the kernel.

Or perhaps some .text.FOOBAR has special usage which cannot be placed
into the output .text
)


> 		TEXT_CFI_JT						\
> 	MEM_KEEP(init.text*)						\
> 	MEM_KEEP(exit.text*)						\
>
>base-commit: 4669e13cd67f8532be12815ed3d37e775a9bdc16
>-- 
>2.32.0.264.g75ae10bc75
>

Re: [PATCH v2] vmlinux.lds.h: Handle clang's module.{c,d}tor sections

[Marco Elver]

On Sat, 31 Jul 2021 at 04:33, Nathan Chancellor <nathan@kernel.org> wrote:
> A recent change in LLVM causes module_{c,d}tor sections to appear when
> CONFIG_K{A,C}SAN are enabled, which results in orphan section warnings
> because these are not handled anywhere:
>
> ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_ctor) is being placed in '.text.asan.module_ctor'
> ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_dtor) is being placed in '.text.asan.module_dtor'
> ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.tsan.module_ctor) is being placed in '.text.tsan.module_ctor'
>
> Fangrui explains: "the function asan.module_ctor has the SHF_GNU_RETAIN
> flag, so it is in a separate section even with -fno-function-sections
> (default)".
>
> Place them in the TEXT_TEXT section so that these technologies continue
> to work with the newer compiler versions. All of the KASAN and KCSAN
> KUnit tests continue to pass after this change.
>
> Cc: stable@vger.kernel.org
> Link: https://github.com/ClangBuiltLinux/linux/issues/1432
> Link: https://github.com/llvm/llvm-project/commit/7b789562244ee941b7bf2cefeb3fc08a59a01865
> Signed-off-by: Nathan Chancellor <nathan@kernel.org>

Acked-by: Marco Elver <elver@google.com>

For KASAN module_ctors are very much required to support detecting
globals out-of-bounds: https://reviews.llvm.org/D81390
For KASAN the test would have revealed that at the latest.

KCSAN does not yet have much use for the module_ctors, but it may
change in future, so keeping them all was the right call.

Thanks,
-- Marco

> ---
>
> v1 -> v2:
>
> * Fix inclusion of .text.tsan.* (Nick)
>
> * Drop .text.asan as it does not exist plus it would be handled by a
>   different line (Fangrui)
>
> * Add Fangrui's explanation about why the LLVM commit caused these
>   sections to appear.
>
>  include/asm-generic/vmlinux.lds.h | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index 17325416e2de..62669b36a772 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -586,6 +586,7 @@
>                 NOINSTR_TEXT                                            \
>                 *(.text..refcount)                                      \
>                 *(.ref.text)                                            \
> +               *(.text.asan.* .text.tsan.*)                            \
>                 TEXT_CFI_JT                                             \
>         MEM_KEEP(init.text*)                                            \
>         MEM_KEEP(exit.text*)                                            \
>
> base-commit: 4669e13cd67f8532be12815ed3d37e775a9bdc16
> --
> 2.32.0.264.g75ae10bc75
>

Re: [PATCH v2] vmlinux.lds.h: Handle clang's module.{c,d}tor sections

[Nick Desaulniers]

On Fri, Jul 30, 2021 at 7:33 PM Nathan Chancellor <nathan@kernel.org> wrote:
>
> A recent change in LLVM causes module_{c,d}tor sections to appear when
> CONFIG_K{A,C}SAN are enabled, which results in orphan section warnings
> because these are not handled anywhere:
>
> ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_ctor) is being placed in '.text.asan.module_ctor'
> ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.asan.module_dtor) is being placed in '.text.asan.module_dtor'
> ld.lld: warning: arch/x86/pci/built-in.a(legacy.o):(.text.tsan.module_ctor) is being placed in '.text.tsan.module_ctor'
>
> Fangrui explains: "the function asan.module_ctor has the SHF_GNU_RETAIN
> flag, so it is in a separate section even with -fno-function-sections
> (default)".
>
> Place them in the TEXT_TEXT section so that these technologies continue
> to work with the newer compiler versions. All of the KASAN and KCSAN
> KUnit tests continue to pass after this change.
>
> Cc: stable@vger.kernel.org
> Link: https://github.com/ClangBuiltLinux/linux/issues/1432
> Link: https://github.com/llvm/llvm-project/commit/7b789562244ee941b7bf2cefeb3fc08a59a01865
> Signed-off-by: Nathan Chancellor <nathan@kernel.org>

Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>

> ---
>
> v1 -> v2:
>
> * Fix inclusion of .text.tsan.* (Nick)
>
> * Drop .text.asan as it does not exist plus it would be handled by a
>   different line (Fangrui)
>
> * Add Fangrui's explanation about why the LLVM commit caused these
>   sections to appear.
>
>  include/asm-generic/vmlinux.lds.h | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index 17325416e2de..62669b36a772 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -586,6 +586,7 @@
>                 NOINSTR_TEXT                                            \
>                 *(.text..refcount)                                      \
>                 *(.ref.text)                                            \
> +               *(.text.asan.* .text.tsan.*)                            \
>                 TEXT_CFI_JT                                             \
>         MEM_KEEP(init.text*)                                            \
>         MEM_KEEP(exit.text*)                                            \
>
> base-commit: 4669e13cd67f8532be12815ed3d37e775a9bdc16
> --
> 2.32.0.264.g75ae10bc75
>


-- 
Thanks,
~Nick Desaulniers