Makefile kernel address tag sanitizer.

Makefile kernel address tag sanitizer.

[Matthew Malcomson]

Hello,

If this is the wrong list & person to ask I'd appreciate being shown who
to ask.

I'm working on implementing hwasan (software tagging address sanitizer)
for GCC (most recent upstream version here
https://gcc.gnu.org/ml/gcc-patches/2019-09/msg00387.html).

I have a working implementation of hwasan for userspace and am now
looking at trying CONFIG_KASAN_SW_TAGS compiled with gcc (only with
CONFIG_KASAN_OUTLINE for now).

I notice the current scripts/Makefile.kasan hard-codes the parameter
`-mllvm -hwasan-instrument-stack=0` to avoid instrumenting stack
variables, and found an email mentioning that stack instrumentation is
not yet supported.
https://lore.kernel.org/linux-arm-kernel/cover.1544099024.git.andreyknvl@google.com/


What is the support that to be added for stack instrumentation?


Thanks,
Matthew
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Makefile kernel address tag sanitizer.

[Andrey Konovalov]

On Wed, Oct 16, 2019 at 3:12 PM Matthew Malcomson
<Matthew.Malcomson@arm.com> wrote:
>
> Hello,
>
> If this is the wrong list & person to ask I'd appreciate being shown who
> to ask.
>
> I'm working on implementing hwasan (software tagging address sanitizer)
> for GCC (most recent upstream version here
> https://gcc.gnu.org/ml/gcc-patches/2019-09/msg00387.html).
>
> I have a working implementation of hwasan for userspace and am now
> looking at trying CONFIG_KASAN_SW_TAGS compiled with gcc (only with
> CONFIG_KASAN_OUTLINE for now).
>
> I notice the current scripts/Makefile.kasan hard-codes the parameter
> `-mllvm -hwasan-instrument-stack=0` to avoid instrumenting stack
> variables, and found an email mentioning that stack instrumentation is
> not yet supported.
> https://lore.kernel.org/linux-arm-kernel/cover.1544099024.git.andreyknvl@google.com/
>
>
> What is the support that to be added for stack instrumentation?

Hi Matthew,

The plan was to upstream tag-based KASAN without stack instrumentation
first, and then enable stack instrumentation as a separate effort. I
didn't yet get to this last part. I remember when I tried enabling
stack instrumentation I was getting what looked like false-positive
reports coming from the printk related code. I didn't investigate them
though. It's possible that some tweaks to the runtime implementation
will be required.

Thanks!

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Makefile kernel address tag sanitizer.

[Matthew Malcomson]

Hi Andrey,

Thanks for the clarification on that bit, could I ask another question?

I seem to have non-stack compiling with GCC running ok, but would like 
to have some better testing than I've managed so far.

I'm running on an instrumented kernel, but haven't seen a crash yet.

Is there a KASAN testsuite to run somewhere so I can proove that bad 
accesses would be caught?

Cheers,
Matthew

On 16/10/19 14:47, Andrey Konovalov wrote:
> On Wed, Oct 16, 2019 at 3:12 PM Matthew Malcomson
> <Matthew.Malcomson@arm.com> wrote:
>>
>> Hello,
>>
>> If this is the wrong list & person to ask I'd appreciate being shown who
>> to ask.
>>
>> I'm working on implementing hwasan (software tagging address sanitizer)
>> for GCC (most recent upstream version here
>> https://gcc.gnu.org/ml/gcc-patches/2019-09/msg00387.html).
>>
>> I have a working implementation of hwasan for userspace and am now
>> looking at trying CONFIG_KASAN_SW_TAGS compiled with gcc (only with
>> CONFIG_KASAN_OUTLINE for now).
>>
>> I notice the current scripts/Makefile.kasan hard-codes the parameter
>> `-mllvm -hwasan-instrument-stack=0` to avoid instrumenting stack
>> variables, and found an email mentioning that stack instrumentation is
>> not yet supported.
>> https://lore.kernel.org/linux-arm-kernel/cover.1544099024.git.andreyknvl@google.com/
>>
>>
>> What is the support that to be added for stack instrumentation?
> 
> Hi Matthew,
> 
> The plan was to upstream tag-based KASAN without stack instrumentation
> first, and then enable stack instrumentation as a separate effort. I
> didn't yet get to this last part. I remember when I tried enabling
> stack instrumentation I was getting what looked like false-positive
> reports coming from the printk related code. I didn't investigate them
> though. It's possible that some tweaks to the runtime implementation
> will be required.
> 
> Thanks!
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Makefile kernel address tag sanitizer.

[Andrey Konovalov]

On Tue, Oct 29, 2019 at 6:45 PM Matthew Malcomson
<Matthew.Malcomson@arm.com> wrote:
>
> Hi Andrey,

Hi Matthew,

>
> Thanks for the clarification on that bit, could I ask another question?
>
> I seem to have non-stack compiling with GCC running ok, but would like
> to have some better testing than I've managed so far.

Great! =)

>
> I'm running on an instrumented kernel, but haven't seen a crash yet.
>
> Is there a KASAN testsuite to run somewhere so I can proove that bad
> accesses would be caught?

Kind of. There's CONFIG_TEST_KASAN which produces lib/test_kasan.ko,
which you can insmod and it will do all kinds of bad accesses.
Unfortunately there's no automated checker for it, so you'll need to
look through the reports manually and check if they make sense.

Thanks!

>
> Cheers,
> Matthew
>
> On 16/10/19 14:47, Andrey Konovalov wrote:
> > On Wed, Oct 16, 2019 at 3:12 PM Matthew Malcomson
> > <Matthew.Malcomson@arm.com> wrote:
> >>
> >> Hello,
> >>
> >> If this is the wrong list & person to ask I'd appreciate being shown who
> >> to ask.
> >>
> >> I'm working on implementing hwasan (software tagging address sanitizer)
> >> for GCC (most recent upstream version here
> >> https://gcc.gnu.org/ml/gcc-patches/2019-09/msg00387.html).
> >>
> >> I have a working implementation of hwasan for userspace and am now
> >> looking at trying CONFIG_KASAN_SW_TAGS compiled with gcc (only with
> >> CONFIG_KASAN_OUTLINE for now).
> >>
> >> I notice the current scripts/Makefile.kasan hard-codes the parameter
> >> `-mllvm -hwasan-instrument-stack=0` to avoid instrumenting stack
> >> variables, and found an email mentioning that stack instrumentation is
> >> not yet supported.
> >> https://lore.kernel.org/linux-arm-kernel/cover.1544099024.git.andreyknvl@google.com/
> >>
> >>
> >> What is the support that to be added for stack instrumentation?
> >
> > Hi Matthew,
> >
> > The plan was to upstream tag-based KASAN without stack instrumentation
> > first, and then enable stack instrumentation as a separate effort. I
> > didn't yet get to this last part. I remember when I tried enabling
> > stack instrumentation I was getting what looked like false-positive
> > reports coming from the printk related code. I didn't investigate them
> > though. It's possible that some tweaks to the runtime implementation
> > will be required.
> >
> > Thanks!
> >
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Makefile kernel address tag sanitizer.

[Matthew Malcomson]

On 29/10/19 17:50, Andrey Konovalov wrote:
> On Tue, Oct 29, 2019 at 6:45 PM Matthew Malcomson
> <Matthew.Malcomson@arm.com> wrote:
>>
>> Hi Andrey,
> 
> Hi Matthew,
> 
>>
>> Thanks for the clarification on that bit, could I ask another question?
>>
>> I seem to have non-stack compiling with GCC running ok, but would like
>> to have some better testing than I've managed so far.
> 
> Great! =)
> 
>>
>> I'm running on an instrumented kernel, but haven't seen a crash yet.
>>
>> Is there a KASAN testsuite to run somewhere so I can proove that bad
>> accesses would be caught?
> 
> Kind of. There's CONFIG_TEST_KASAN which produces lib/test_kasan.ko,
> which you can insmod and it will do all kinds of bad accesses.
> Unfortunately there's no automated checker for it, so you'll need to
> look through the reports manually and check if they make sense.

Great, that was really useful!

I found one issue in my instrumentation through using these tests -- I 
haven't defined `__SANITIZE_ADDRESS__` (which means memset calls aren't 
sanitized here since a macro replaces them with __memset).

Looking at the current kernel code it seems that for clang you use 
`__SANITIZE_ADDRESS__`, for either hwasan or asan.  (commit 2bd926b4).

Do you (or anyone else) have any objections to using 
`__SANITIZE_HWADDRESS__` to indicate tagging address sanitizer so they 
can be distinguished?

I can provide a patch to the kernel to account for the compiler 
behaviour if it's acceptable.



Similarly, I'm thinking I'll add no_sanitize_hwaddress as the hwasan 
equivalent of no_sanitize_address, which will require an update in the 
kernel given it seems you want KASAN to be used the same whether using 
tags or not.

Cheers,
Matthew

> 
> Thanks!
> 
>>
>> Cheers,
>> Matthew
>>
>> On 16/10/19 14:47, Andrey Konovalov wrote:
>>> On Wed, Oct 16, 2019 at 3:12 PM Matthew Malcomson
>>> <Matthew.Malcomson@arm.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> If this is the wrong list & person to ask I'd appreciate being shown who
>>>> to ask.
>>>>
>>>> I'm working on implementing hwasan (software tagging address sanitizer)
>>>> for GCC (most recent upstream version here
>>>> https://gcc.gnu.org/ml/gcc-patches/2019-09/msg00387.html).
>>>>
>>>> I have a working implementation of hwasan for userspace and am now
>>>> looking at trying CONFIG_KASAN_SW_TAGS compiled with gcc (only with
>>>> CONFIG_KASAN_OUTLINE for now).
>>>>
>>>> I notice the current scripts/Makefile.kasan hard-codes the parameter
>>>> `-mllvm -hwasan-instrument-stack=0` to avoid instrumenting stack
>>>> variables, and found an email mentioning that stack instrumentation is
>>>> not yet supported.
>>>> https://lore.kernel.org/linux-arm-kernel/cover.1544099024.git.andreyknvl@google.com/
>>>>
>>>>
>>>> What is the support that to be added for stack instrumentation?
>>>
>>> Hi Matthew,
>>>
>>> The plan was to upstream tag-based KASAN without stack instrumentation
>>> first, and then enable stack instrumentation as a separate effort. I
>>> didn't yet get to this last part. I remember when I tried enabling
>>> stack instrumentation I was getting what looked like false-positive
>>> reports coming from the printk related code. I didn't investigate them
>>> though. It's possible that some tweaks to the runtime implementation
>>> will be required.
>>>
>>> Thanks!
>>>
>>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Makefile kernel address tag sanitizer.

[Andrey Konovalov]

On Wed, Oct 30, 2019 at 2:30 PM Matthew Malcomson
<Matthew.Malcomson@arm.com> wrote:
>
> On 29/10/19 17:50, Andrey Konovalov wrote:
> > On Tue, Oct 29, 2019 at 6:45 PM Matthew Malcomson
> > <Matthew.Malcomson@arm.com> wrote:
> >>
> >> Hi Andrey,
> >
> > Hi Matthew,
> >
> >>
> >> Thanks for the clarification on that bit, could I ask another question?
> >>
> >> I seem to have non-stack compiling with GCC running ok, but would like
> >> to have some better testing than I've managed so far.
> >
> > Great! =)
> >
> >>
> >> I'm running on an instrumented kernel, but haven't seen a crash yet.
> >>
> >> Is there a KASAN testsuite to run somewhere so I can proove that bad
> >> accesses would be caught?
> >
> > Kind of. There's CONFIG_TEST_KASAN which produces lib/test_kasan.ko,
> > which you can insmod and it will do all kinds of bad accesses.
> > Unfortunately there's no automated checker for it, so you'll need to
> > look through the reports manually and check if they make sense.
>
> Great, that was really useful!
>
> I found one issue in my instrumentation through using these tests -- I
> haven't defined `__SANITIZE_ADDRESS__` (which means memset calls aren't
> sanitized here since a macro replaces them with __memset).
>
> Looking at the current kernel code it seems that for clang you use
> `__SANITIZE_ADDRESS__`, for either hwasan or asan.  (commit 2bd926b4).
>
> Do you (or anyone else) have any objections to using
> `__SANITIZE_HWADDRESS__` to indicate tagging address sanitizer so they
> can be distinguished?
>
> I can provide a patch to the kernel to account for the compiler
> behaviour if it's acceptable.
>
>
>
> Similarly, I'm thinking I'll add no_sanitize_hwaddress as the hwasan
> equivalent of no_sanitize_address, which will require an update in the
> kernel given it seems you want KASAN to be used the same whether using
> tags or not.

We have intentionally reused the same macros to simplify things. Is
there any reason to use separate macros for GCC? Are there places
where we need to use specifically no_sanitize_hwaddress and
__SANITIZE_HWADDRESS__, but not no_sanitize_address and
__SANITIZE_ADDRESS__?

>
> Cheers,
> Matthew
>
> >
> > Thanks!
> >
> >>
> >> Cheers,
> >> Matthew
> >>
> >> On 16/10/19 14:47, Andrey Konovalov wrote:
> >>> On Wed, Oct 16, 2019 at 3:12 PM Matthew Malcomson
> >>> <Matthew.Malcomson@arm.com> wrote:
> >>>>
> >>>> Hello,
> >>>>
> >>>> If this is the wrong list & person to ask I'd appreciate being shown who
> >>>> to ask.
> >>>>
> >>>> I'm working on implementing hwasan (software tagging address sanitizer)
> >>>> for GCC (most recent upstream version here
> >>>> https://gcc.gnu.org/ml/gcc-patches/2019-09/msg00387.html).
> >>>>
> >>>> I have a working implementation of hwasan for userspace and am now
> >>>> looking at trying CONFIG_KASAN_SW_TAGS compiled with gcc (only with
> >>>> CONFIG_KASAN_OUTLINE for now).
> >>>>
> >>>> I notice the current scripts/Makefile.kasan hard-codes the parameter
> >>>> `-mllvm -hwasan-instrument-stack=0` to avoid instrumenting stack
> >>>> variables, and found an email mentioning that stack instrumentation is
> >>>> not yet supported.
> >>>> https://lore.kernel.org/linux-arm-kernel/cover.1544099024.git.andreyknvl@google.com/
> >>>>
> >>>>
> >>>> What is the support that to be added for stack instrumentation?
> >>>
> >>> Hi Matthew,
> >>>
> >>> The plan was to upstream tag-based KASAN without stack instrumentation
> >>> first, and then enable stack instrumentation as a separate effort. I
> >>> didn't yet get to this last part. I remember when I tried enabling
> >>> stack instrumentation I was getting what looked like false-positive
> >>> reports coming from the printk related code. I didn't investigate them
> >>> though. It's possible that some tweaks to the runtime implementation
> >>> will be required.
> >>>
> >>> Thanks!
> >>>
> >>
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Makefile kernel address tag sanitizer.

[Matthew Malcomson]

On 30/10/19 13:56, Andrey Konovalov wrote:
> On Wed, Oct 30, 2019 at 2:30 PM Matthew Malcomson
> <Matthew.Malcomson@arm.com> wrote:
>>
>> On 29/10/19 17:50, Andrey Konovalov wrote:
>>> On Tue, Oct 29, 2019 at 6:45 PM Matthew Malcomson
>>> <Matthew.Malcomson@arm.com> wrote:
>>>>
>>>> Hi Andrey,
>>>
>>> Hi Matthew,
>>>
>>>>
>>>> Thanks for the clarification on that bit, could I ask another question?
>>>>
>>>> I seem to have non-stack compiling with GCC running ok, but would like
>>>> to have some better testing than I've managed so far.
>>>
>>> Great! =)
>>>
>>>>
>>>> I'm running on an instrumented kernel, but haven't seen a crash yet.
>>>>
>>>> Is there a KASAN testsuite to run somewhere so I can proove that bad
>>>> accesses would be caught?
>>>
>>> Kind of. There's CONFIG_TEST_KASAN which produces lib/test_kasan.ko,
>>> which you can insmod and it will do all kinds of bad accesses.
>>> Unfortunately there's no automated checker for it, so you'll need to
>>> look through the reports manually and check if they make sense.
>>
>> Great, that was really useful!
>>
>> I found one issue in my instrumentation through using these tests -- I
>> haven't defined `__SANITIZE_ADDRESS__` (which means memset calls aren't
>> sanitized here since a macro replaces them with __memset).
>>
>> Looking at the current kernel code it seems that for clang you use
>> `__SANITIZE_ADDRESS__`, for either hwasan or asan.  (commit 2bd926b4).
>>
>> Do you (or anyone else) have any objections to using
>> `__SANITIZE_HWADDRESS__` to indicate tagging address sanitizer so they
>> can be distinguished?
>>
>> I can provide a patch to the kernel to account for the compiler
>> behaviour if it's acceptable.
>>
>>
>>
>> Similarly, I'm thinking I'll add no_sanitize_hwaddress as the hwasan
>> equivalent of no_sanitize_address, which will require an update in the
>> kernel given it seems you want KASAN to be used the same whether using
>> tags or not.
> 
> We have intentionally reused the same macros to simplify things. Is
> there any reason to use separate macros for GCC? Are there places
> where we need to use specifically no_sanitize_hwaddress and
> __SANITIZE_HWADDRESS__, but not no_sanitize_address and
> __SANITIZE_ADDRESS__?
> 

Honestly, I'm not sure ...

I think I'll come back after asking a bit of feedback from GCC upstream 
with this in mind.
A discussion with a colleague has already brought up a few different 
options.

>>
>> Cheers,
>> Matthew
>>
>>>
>>> Thanks!
>>>
>>>>
>>>> Cheers,
>>>> Matthew
>>>>
>>>> On 16/10/19 14:47, Andrey Konovalov wrote:
>>>>> On Wed, Oct 16, 2019 at 3:12 PM Matthew Malcomson
>>>>> <Matthew.Malcomson@arm.com> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> If this is the wrong list & person to ask I'd appreciate being shown who
>>>>>> to ask.
>>>>>>
>>>>>> I'm working on implementing hwasan (software tagging address sanitizer)
>>>>>> for GCC (most recent upstream version here
>>>>>> https://gcc.gnu.org/ml/gcc-patches/2019-09/msg00387.html).
>>>>>>
>>>>>> I have a working implementation of hwasan for userspace and am now
>>>>>> looking at trying CONFIG_KASAN_SW_TAGS compiled with gcc (only with
>>>>>> CONFIG_KASAN_OUTLINE for now).
>>>>>>
>>>>>> I notice the current scripts/Makefile.kasan hard-codes the parameter
>>>>>> `-mllvm -hwasan-instrument-stack=0` to avoid instrumenting stack
>>>>>> variables, and found an email mentioning that stack instrumentation is
>>>>>> not yet supported.
>>>>>> https://lore.kernel.org/linux-arm-kernel/cover.1544099024.git.andreyknvl@google.com/
>>>>>>
>>>>>>
>>>>>> What is the support that to be added for stack instrumentation?
>>>>>
>>>>> Hi Matthew,
>>>>>
>>>>> The plan was to upstream tag-based KASAN without stack instrumentation
>>>>> first, and then enable stack instrumentation as a separate effort. I
>>>>> didn't yet get to this last part. I remember when I tried enabling
>>>>> stack instrumentation I was getting what looked like false-positive
>>>>> reports coming from the printk related code. I didn't investigate them
>>>>> though. It's possible that some tweaks to the runtime implementation
>>>>> will be required.
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel