32-bit arm unwind info for PLTs

32-bit arm unwind info for PLTs

[Kursad Oney]

Hi,

I have a large kernel module that gets loaded to vmalloc via
ARM_MODULE_PLTS. When I either use perf, or enable (yet unmerged)
KASAN-arm changes, I see occasional warnings like this:

unwind: Index not found pc:f3aa8d6c

The address f3aa8d6c is in the .plt section. I printed the backtrace
in unwind_frame where the warning is printed, and the backtrace looks
like this:

NMI backtrace for cpu 0
CPU: 0 PID: 7193 Comm: hostapd Tainted: P                  4.19.100 #7
Hardware name: Generic DT based system
[<c02144b8>] (unwind_backtrace) from [<c020dac8>] (show_stack+0x10/0x14)
[<c020dac8>] (show_stack) from [<c083b238>] (dump_stack+0x9c/0xb0)
[<c083b238>] (dump_stack) from [<c0842dfc>] (nmi_cpu_backtrace+0xb4/0xe8)
[<c0842dfc>] (nmi_cpu_backtrace) from [<c0842fb0>] (nmi_trigger_cpumask_backtrace+0x180/0x1d4)
[<c0842fb0>] (nmi_trigger_cpumask_backtrace) from [<c0214428>] (unwind_frame+0x650/0x6e0)
[<c0214428>] (unwind_frame) from [<c020d53c>] (walk_stackframe+0x30/0x3c)
[<c020d53c>] (walk_stackframe) from [<c020d778>] (__save_stack_trace+0x100/0x108)
[<c020d778>] (__save_stack_trace) from [<c03369ec>] (__kasan_slab_free+0x124/0x1f8)
[<c03369ec>] (__kasan_slab_free) from [<c033396c>] (kmem_cache_free+0x5c/0x19c)
[<c033396c>] (kmem_cache_free) from [<c029138c>] (rcu_process_callbacks+0x360/0x604)
[<c029138c>] (rcu_process_callbacks) from [<c020295c>] (__do_softirq+0x174/0x374)
[<c020295c>] (__do_softirq) from [<c022b07c>] (irq_exit+0xd0/0xf8)
[<c022b07c>] (irq_exit) from [<c027ce3c>] (__handle_domain_irq+0x7c/0xd4)
[<c027ce3c>] (__handle_domain_irq) from [<c04dfc00>] (gic_handle_irq+0x4c/0x90)
[<c04dfc00>] (gic_handle_irq) from [<c02020cc>] (__irq_svc+0x6c/0xac)
Exception stack(0xe4c3b400 to 0xe4c3b448)
b400: dca01404 f3d63a80 bd7ac750 00000000 f3d63a84 f0b6c164 f3bb6434 dca0126c
b420: f3d665d0 dca01408 dca01404 0c002c24 00000000 e4c3b450 f37ed5a4 f3aa8d6c
b440: 600b0013 ffffffff
[<c02020cc>] (__irq_svc) from [<f3aa8d6c>] (wl_module_exit+0xf64/0x21f8 [wl])

To me it looks like while the CPU is executing an instruction in the
PLT, it gets an interrupt. If we call save_stack_trace() or any
function that eventually gets to unwind_frame() from that context,
then the unwinder doesn't know how to unwind the PLT. Does this sound
right? Any idea how the unwinder code should deal with this situation?

Thanks!
kursad

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: 32-bit arm unwind info for PLTs

[Ard Biesheuvel]

On Thu, 2 Apr 2020 at 17:45, Kursad Oney <kursad.oney@broadcom.com> wrote:
>
> Hi,
>
> I have a large kernel module that gets loaded to vmalloc via
> ARM_MODULE_PLTS. When I either use perf, or enable (yet unmerged)
> KASAN-arm changes, I see occasional warnings like this:
>
> unwind: Index not found pc:f3aa8d6c
>
> The address f3aa8d6c is in the .plt section. I printed the backtrace
> in unwind_frame where the warning is printed, and the backtrace looks
> like this:
>
> NMI backtrace for cpu 0
> CPU: 0 PID: 7193 Comm: hostapd Tainted: P                  4.19.100 #7
> Hardware name: Generic DT based system
> [<c02144b8>] (unwind_backtrace) from [<c020dac8>] (show_stack+0x10/0x14)
> [<c020dac8>] (show_stack) from [<c083b238>] (dump_stack+0x9c/0xb0)
> [<c083b238>] (dump_stack) from [<c0842dfc>] (nmi_cpu_backtrace+0xb4/0xe8)
> [<c0842dfc>] (nmi_cpu_backtrace) from [<c0842fb0>] (nmi_trigger_cpumask_backtrace+0x180/0x1d4)
> [<c0842fb0>] (nmi_trigger_cpumask_backtrace) from [<c0214428>] (unwind_frame+0x650/0x6e0)
> [<c0214428>] (unwind_frame) from [<c020d53c>] (walk_stackframe+0x30/0x3c)
> [<c020d53c>] (walk_stackframe) from [<c020d778>] (__save_stack_trace+0x100/0x108)
> [<c020d778>] (__save_stack_trace) from [<c03369ec>] (__kasan_slab_free+0x124/0x1f8)
> [<c03369ec>] (__kasan_slab_free) from [<c033396c>] (kmem_cache_free+0x5c/0x19c)
> [<c033396c>] (kmem_cache_free) from [<c029138c>] (rcu_process_callbacks+0x360/0x604)
> [<c029138c>] (rcu_process_callbacks) from [<c020295c>] (__do_softirq+0x174/0x374)
> [<c020295c>] (__do_softirq) from [<c022b07c>] (irq_exit+0xd0/0xf8)
> [<c022b07c>] (irq_exit) from [<c027ce3c>] (__handle_domain_irq+0x7c/0xd4)
> [<c027ce3c>] (__handle_domain_irq) from [<c04dfc00>] (gic_handle_irq+0x4c/0x90)
> [<c04dfc00>] (gic_handle_irq) from [<c02020cc>] (__irq_svc+0x6c/0xac)
> Exception stack(0xe4c3b400 to 0xe4c3b448)
> b400: dca01404 f3d63a80 bd7ac750 00000000 f3d63a84 f0b6c164 f3bb6434 dca0126c
> b420: f3d665d0 dca01408 dca01404 0c002c24 00000000 e4c3b450 f37ed5a4 f3aa8d6c
> b440: 600b0013 ffffffff
> [<c02020cc>] (__irq_svc) from [<f3aa8d6c>] (wl_module_exit+0xf64/0x21f8 [wl])
>
> To me it looks like while the CPU is executing an instruction in the
> PLT, it gets an interrupt. If we call save_stack_trace() or any
> function that eventually gets to unwind_frame() from that context,
> then the unwinder doesn't know how to unwind the PLT. Does this sound
> right? Any idea how the unwinder code should deal with this situation?
>

It is going to be infeasible to emit CFI directives on the fly, and
expose them via a separate .eh_frame section in the module. So what we
should probably be doing is teaching the unwinder about these entries,
and simply ignore them, which is reasonable, given that they only set
PC using a PC+immediate LDR, and so they don't affect the state of the
stack or the register file. That means we'll need to use the value of
LR to locate the next stack frame.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: 32-bit arm unwind info for PLTs

[Kursad Oney]

Hi Ard,

On Mon, Apr 6, 2020 at 8:28 AM Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Thu, 2 Apr 2020 at 17:45, Kursad Oney <kursad.oney@broadcom.com> wrote:
> >
> > Hi,
> >
> > I have a large kernel module that gets loaded to vmalloc via
> > ARM_MODULE_PLTS. When I either use perf, or enable (yet unmerged)
> > KASAN-arm changes, I see occasional warnings like this:
> >
> > unwind: Index not found pc:f3aa8d6c
> >
> > The address f3aa8d6c is in the .plt section. I printed the backtrace
> > in unwind_frame where the warning is printed, and the backtrace looks
> > like this:
> >
> > NMI backtrace for cpu 0
> > CPU: 0 PID: 7193 Comm: hostapd Tainted: P                  4.19.100 #7
> > Hardware name: Generic DT based system
> > [<c02144b8>] (unwind_backtrace) from [<c020dac8>] (show_stack+0x10/0x14)
> > [<c020dac8>] (show_stack) from [<c083b238>] (dump_stack+0x9c/0xb0)
> > [<c083b238>] (dump_stack) from [<c0842dfc>] (nmi_cpu_backtrace+0xb4/0xe8)
> > [<c0842dfc>] (nmi_cpu_backtrace) from [<c0842fb0>] (nmi_trigger_cpumask_backtrace+0x180/0x1d4)
> > [<c0842fb0>] (nmi_trigger_cpumask_backtrace) from [<c0214428>] (unwind_frame+0x650/0x6e0)
> > [<c0214428>] (unwind_frame) from [<c020d53c>] (walk_stackframe+0x30/0x3c)
> > [<c020d53c>] (walk_stackframe) from [<c020d778>] (__save_stack_trace+0x100/0x108)
> > [<c020d778>] (__save_stack_trace) from [<c03369ec>] (__kasan_slab_free+0x124/0x1f8)
> > [<c03369ec>] (__kasan_slab_free) from [<c033396c>] (kmem_cache_free+0x5c/0x19c)
> > [<c033396c>] (kmem_cache_free) from [<c029138c>] (rcu_process_callbacks+0x360/0x604)
> > [<c029138c>] (rcu_process_callbacks) from [<c020295c>] (__do_softirq+0x174/0x374)
> > [<c020295c>] (__do_softirq) from [<c022b07c>] (irq_exit+0xd0/0xf8)
> > [<c022b07c>] (irq_exit) from [<c027ce3c>] (__handle_domain_irq+0x7c/0xd4)
> > [<c027ce3c>] (__handle_domain_irq) from [<c04dfc00>] (gic_handle_irq+0x4c/0x90)
> > [<c04dfc00>] (gic_handle_irq) from [<c02020cc>] (__irq_svc+0x6c/0xac)
> > Exception stack(0xe4c3b400 to 0xe4c3b448)
> > b400: dca01404 f3d63a80 bd7ac750 00000000 f3d63a84 f0b6c164 f3bb6434 dca0126c
> > b420: f3d665d0 dca01408 dca01404 0c002c24 00000000 e4c3b450 f37ed5a4 f3aa8d6c
> > b440: 600b0013 ffffffff
> > [<c02020cc>] (__irq_svc) from [<f3aa8d6c>] (wl_module_exit+0xf64/0x21f8 [wl])
> >
> > To me it looks like while the CPU is executing an instruction in the
> > PLT, it gets an interrupt. If we call save_stack_trace() or any
> > function that eventually gets to unwind_frame() from that context,
> > then the unwinder doesn't know how to unwind the PLT. Does this sound
> > right? Any idea how the unwinder code should deal with this situation?
> >
>
> It is going to be infeasible to emit CFI directives on the fly, and
> expose them via a separate .eh_frame section in the module. So what we
> should probably be doing is teaching the unwinder about these entries,
> and simply ignore them, which is reasonable, given that they only set
> PC using a PC+immediate LDR, and so they don't affect the state of the
> stack or the register file. That means we'll need to use the value of
> LR to locate the next stack frame.

Thanks for the response. Given an address, what is a good way to teach
the unwinder about these entries? Since before the interrupt we can be
in any module's PLT region, do we search all modules or is there a
thing we can directly check on the address/instruction itself?

Also I understand the idea of using LR to locate the next stack frame,
but I have no idea how to do it.

Lastly, as far as I can tell, we don't seem to be losing any important
information here with the current code (but I am certainly not
considering all possible cases). Would it be reasonable to emit the
"unwind: Index not found" message only if we are not in PLT and not
unwind the rest of the stack?

Thanks!
kursad

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: 32-bit arm unwind info for PLTs

[Ard Biesheuvel]

On Mon, 6 Apr 2020 at 23:55, Kursad Oney <kursad.oney@broadcom.com> wrote:
>
> Hi Ard,
>
> On Mon, Apr 6, 2020 at 8:28 AM Ard Biesheuvel <ardb@kernel.org> wrote:
> >
> > On Thu, 2 Apr 2020 at 17:45, Kursad Oney <kursad.oney@broadcom.com> wrote:
> > >
> > > Hi,
> > >
> > > I have a large kernel module that gets loaded to vmalloc via
> > > ARM_MODULE_PLTS. When I either use perf, or enable (yet unmerged)
> > > KASAN-arm changes, I see occasional warnings like this:
> > >
> > > unwind: Index not found pc:f3aa8d6c
> > >
> > > The address f3aa8d6c is in the .plt section. I printed the backtrace
> > > in unwind_frame where the warning is printed, and the backtrace looks
> > > like this:
> > >
> > > NMI backtrace for cpu 0
> > > CPU: 0 PID: 7193 Comm: hostapd Tainted: P                  4.19.100 #7
> > > Hardware name: Generic DT based system
> > > [<c02144b8>] (unwind_backtrace) from [<c020dac8>] (show_stack+0x10/0x14)
> > > [<c020dac8>] (show_stack) from [<c083b238>] (dump_stack+0x9c/0xb0)
> > > [<c083b238>] (dump_stack) from [<c0842dfc>] (nmi_cpu_backtrace+0xb4/0xe8)
> > > [<c0842dfc>] (nmi_cpu_backtrace) from [<c0842fb0>] (nmi_trigger_cpumask_backtrace+0x180/0x1d4)
> > > [<c0842fb0>] (nmi_trigger_cpumask_backtrace) from [<c0214428>] (unwind_frame+0x650/0x6e0)
> > > [<c0214428>] (unwind_frame) from [<c020d53c>] (walk_stackframe+0x30/0x3c)
> > > [<c020d53c>] (walk_stackframe) from [<c020d778>] (__save_stack_trace+0x100/0x108)
> > > [<c020d778>] (__save_stack_trace) from [<c03369ec>] (__kasan_slab_free+0x124/0x1f8)
> > > [<c03369ec>] (__kasan_slab_free) from [<c033396c>] (kmem_cache_free+0x5c/0x19c)
> > > [<c033396c>] (kmem_cache_free) from [<c029138c>] (rcu_process_callbacks+0x360/0x604)
> > > [<c029138c>] (rcu_process_callbacks) from [<c020295c>] (__do_softirq+0x174/0x374)
> > > [<c020295c>] (__do_softirq) from [<c022b07c>] (irq_exit+0xd0/0xf8)
> > > [<c022b07c>] (irq_exit) from [<c027ce3c>] (__handle_domain_irq+0x7c/0xd4)
> > > [<c027ce3c>] (__handle_domain_irq) from [<c04dfc00>] (gic_handle_irq+0x4c/0x90)
> > > [<c04dfc00>] (gic_handle_irq) from [<c02020cc>] (__irq_svc+0x6c/0xac)
> > > Exception stack(0xe4c3b400 to 0xe4c3b448)
> > > b400: dca01404 f3d63a80 bd7ac750 00000000 f3d63a84 f0b6c164 f3bb6434 dca0126c
> > > b420: f3d665d0 dca01408 dca01404 0c002c24 00000000 e4c3b450 f37ed5a4 f3aa8d6c
> > > b440: 600b0013 ffffffff
> > > [<c02020cc>] (__irq_svc) from [<f3aa8d6c>] (wl_module_exit+0xf64/0x21f8 [wl])
> > >
> > > To me it looks like while the CPU is executing an instruction in the
> > > PLT, it gets an interrupt. If we call save_stack_trace() or any
> > > function that eventually gets to unwind_frame() from that context,
> > > then the unwinder doesn't know how to unwind the PLT. Does this sound
> > > right? Any idea how the unwinder code should deal with this situation?
> > >
> >
> > It is going to be infeasible to emit CFI directives on the fly, and
> > expose them via a separate .eh_frame section in the module. So what we
> > should probably be doing is teaching the unwinder about these entries,
> > and simply ignore them, which is reasonable, given that they only set
> > PC using a PC+immediate LDR, and so they don't affect the state of the
> > stack or the register file. That means we'll need to use the value of
> > LR to locate the next stack frame.
>
> Thanks for the response. Given an address, what is a good way to teach
> the unwinder about these entries? Since before the interrupt we can be
> in any module's PLT region, do we search all modules or is there a
> thing we can directly check on the address/instruction itself?
>

You could use __module_address() to retrieve the module, and cross
reference the address with its PLT sections.

> Also I understand the idea of using LR to locate the next stack frame,
> but I have no idea how to do it.
>

The unwinder should have access to a struct pt_regs at each level, and
the value of LR can simply be read from there.

> Lastly, as far as I can tell, we don't seem to be losing any important
> information here with the current code (but I am certainly not
> considering all possible cases). Would it be reasonable to emit the
> "unwind: Index not found" message only if we are not in PLT and not
> unwind the rest of the stack?
>

What would that fix? The unwind already terminates at this point, and
the warning only appears occasionally, and is factually correct.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: 32-bit arm unwind info for PLTs

[Kursad Oney]

Hi Ard,

On Tue, Apr 7, 2020 at 3:12 AM Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Mon, 6 Apr 2020 at 23:55, Kursad Oney <kursad.oney@broadcom.com> wrote:
> >
> > Hi Ard,
> >
> > On Mon, Apr 6, 2020 at 8:28 AM Ard Biesheuvel <ardb@kernel.org> wrote:
> > >
> > > On Thu, 2 Apr 2020 at 17:45, Kursad Oney <kursad.oney@broadcom.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I have a large kernel module that gets loaded to vmalloc via
> > > > ARM_MODULE_PLTS. When I either use perf, or enable (yet unmerged)
> > > > KASAN-arm changes, I see occasional warnings like this:
> > > >
> > > > unwind: Index not found pc:f3aa8d6c
> > > >
> > > > The address f3aa8d6c is in the .plt section. I printed the backtrace
> > > > in unwind_frame where the warning is printed, and the backtrace looks
> > > > like this:
> > > >
> > > > NMI backtrace for cpu 0
> > > > CPU: 0 PID: 7193 Comm: hostapd Tainted: P                  4.19.100 #7
> > > > Hardware name: Generic DT based system
> > > > [<c02144b8>] (unwind_backtrace) from [<c020dac8>] (show_stack+0x10/0x14)
> > > > [<c020dac8>] (show_stack) from [<c083b238>] (dump_stack+0x9c/0xb0)
> > > > [<c083b238>] (dump_stack) from [<c0842dfc>] (nmi_cpu_backtrace+0xb4/0xe8)
> > > > [<c0842dfc>] (nmi_cpu_backtrace) from [<c0842fb0>] (nmi_trigger_cpumask_backtrace+0x180/0x1d4)
> > > > [<c0842fb0>] (nmi_trigger_cpumask_backtrace) from [<c0214428>] (unwind_frame+0x650/0x6e0)
> > > > [<c0214428>] (unwind_frame) from [<c020d53c>] (walk_stackframe+0x30/0x3c)
> > > > [<c020d53c>] (walk_stackframe) from [<c020d778>] (__save_stack_trace+0x100/0x108)
> > > > [<c020d778>] (__save_stack_trace) from [<c03369ec>] (__kasan_slab_free+0x124/0x1f8)
> > > > [<c03369ec>] (__kasan_slab_free) from [<c033396c>] (kmem_cache_free+0x5c/0x19c)
> > > > [<c033396c>] (kmem_cache_free) from [<c029138c>] (rcu_process_callbacks+0x360/0x604)
> > > > [<c029138c>] (rcu_process_callbacks) from [<c020295c>] (__do_softirq+0x174/0x374)
> > > > [<c020295c>] (__do_softirq) from [<c022b07c>] (irq_exit+0xd0/0xf8)
> > > > [<c022b07c>] (irq_exit) from [<c027ce3c>] (__handle_domain_irq+0x7c/0xd4)
> > > > [<c027ce3c>] (__handle_domain_irq) from [<c04dfc00>] (gic_handle_irq+0x4c/0x90)
> > > > [<c04dfc00>] (gic_handle_irq) from [<c02020cc>] (__irq_svc+0x6c/0xac)
> > > > Exception stack(0xe4c3b400 to 0xe4c3b448)
> > > > b400: dca01404 f3d63a80 bd7ac750 00000000 f3d63a84 f0b6c164 f3bb6434 dca0126c
> > > > b420: f3d665d0 dca01408 dca01404 0c002c24 00000000 e4c3b450 f37ed5a4 f3aa8d6c
> > > > b440: 600b0013 ffffffff
> > > > [<c02020cc>] (__irq_svc) from [<f3aa8d6c>] (wl_module_exit+0xf64/0x21f8 [wl])
> > > >
> > > > To me it looks like while the CPU is executing an instruction in the
> > > > PLT, it gets an interrupt. If we call save_stack_trace() or any
> > > > function that eventually gets to unwind_frame() from that context,
> > > > then the unwinder doesn't know how to unwind the PLT. Does this sound
> > > > right? Any idea how the unwinder code should deal with this situation?
> > > >
> > >
> > > It is going to be infeasible to emit CFI directives on the fly, and
> > > expose them via a separate .eh_frame section in the module. So what we
> > > should probably be doing is teaching the unwinder about these entries,
> > > and simply ignore them, which is reasonable, given that they only set
> > > PC using a PC+immediate LDR, and so they don't affect the state of the
> > > stack or the register file. That means we'll need to use the value of
> > > LR to locate the next stack frame.
> >
> > Thanks for the response. Given an address, what is a good way to teach
> > the unwinder about these entries? Since before the interrupt we can be
> > in any module's PLT region, do we search all modules or is there a
> > thing we can directly check on the address/instruction itself?
> >
>
> You could use __module_address() to retrieve the module, and cross
> reference the address with its PLT sections.
>
Thanks, I will do that.

> > Also I understand the idea of using LR to locate the next stack frame,
> > but I have no idea how to do it.
> >
>
> The unwinder should have access to a struct pt_regs at each level, and
> the value of LR can simply be read from there.
>
> > Lastly, as far as I can tell, we don't seem to be losing any important
> > information here with the current code (but I am certainly not
> > considering all possible cases). Would it be reasonable to emit the
> > "unwind: Index not found" message only if we are not in PLT and not
> > unwind the rest of the stack?
> >
>
> What would that fix? The unwind already terminates at this point, and
> the warning only appears occasionally, and is factually correct.

OK I understand. Thanks for the answers.
kursad

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: 32-bit arm unwind info for PLTs

[Alexander Sverdlin]

Hello Kursad!

On 02/04/2020 17:38, Kursad Oney wrote:
> I have a large kernel module that gets loaded to vmalloc via
> ARM_MODULE_PLTS. When I either use perf, or enable (yet unmerged)
> KASAN-arm changes, I see occasional warnings like this:
> 
> unwind: Index not found pc:f3aa8d6c
> 
> The address f3aa8d6c is in the .plt section. I printed the backtrace
> in unwind_frame where the warning is printed, and the backtrace looks
> like this:
> 
> NMI backtrace for cpu 0
> CPU: 0 PID: 7193 Comm: hostapd Tainted: P                  4.19.100 #7
> Hardware name: Generic DT based system
> [<c02144b8>] (unwind_backtrace) from [<c020dac8>] (show_stack+0x10/0x14)
> [<c020dac8>] (show_stack) from [<c083b238>] (dump_stack+0x9c/0xb0)
> [<c083b238>] (dump_stack) from [<c0842dfc>] (nmi_cpu_backtrace+0xb4/0xe8)
> [<c0842dfc>] (nmi_cpu_backtrace) from [<c0842fb0>] (nmi_trigger_cpumask_backtrace+0x180/0x1d4)
> [<c0842fb0>] (nmi_trigger_cpumask_backtrace) from [<c0214428>] (unwind_frame+0x650/0x6e0)
> [<c0214428>] (unwind_frame) from [<c020d53c>] (walk_stackframe+0x30/0x3c)
> [<c020d53c>] (walk_stackframe) from [<c020d778>] (__save_stack_trace+0x100/0x108)
> [<c020d778>] (__save_stack_trace) from [<c03369ec>] (__kasan_slab_free+0x124/0x1f8)
> [<c03369ec>] (__kasan_slab_free) from [<c033396c>] (kmem_cache_free+0x5c/0x19c)
> [<c033396c>] (kmem_cache_free) from [<c029138c>] (rcu_process_callbacks+0x360/0x604)
> [<c029138c>] (rcu_process_callbacks) from [<c020295c>] (__do_softirq+0x174/0x374)
> [<c020295c>] (__do_softirq) from [<c022b07c>] (irq_exit+0xd0/0xf8)
> [<c022b07c>] (irq_exit) from [<c027ce3c>] (__handle_domain_irq+0x7c/0xd4)
> [<c027ce3c>] (__handle_domain_irq) from [<c04dfc00>] (gic_handle_irq+0x4c/0x90)
> [<c04dfc00>] (gic_handle_irq) from [<c02020cc>] (__irq_svc+0x6c/0xac)
> Exception stack(0xe4c3b400 to 0xe4c3b448)
> b400: dca01404 f3d63a80 bd7ac750 00000000 f3d63a84 f0b6c164 f3bb6434 dca0126c
> b420: f3d665d0 dca01408 dca01404 0c002c24 00000000 e4c3b450 f37ed5a4 f3aa8d6c
> b440: 600b0013 ffffffff
> [<c02020cc>] (__irq_svc) from [<f3aa8d6c>] (wl_module_exit+0xf64/0x21f8 [wl])
> 
> To me it looks like while the CPU is executing an instruction in the
> PLT, it gets an interrupt. If we call save_stack_trace() or any
> function that eventually gets to unwind_frame() from that context,
> then the unwinder doesn't know how to unwind the PLT. Does this sound
> right? Any idea how the unwinder code should deal with this situation?

have you been able to fix the issue?

Would you like to test the following patch?

diff --git a/arch/arm/include/asm/module.h b/arch/arm/include/asm/module.h
index 5546c97..07c51a3 100644
--- a/arch/arm/include/asm/module.h
+++ b/arch/arm/include/asm/module.h
@@ -37,6 +37,11 @@ struct mod_arch_specific {
 
 struct module;
 u32 get_module_plt(struct module *mod, unsigned long loc, Elf32_Addr val);
+#ifdef CONFIG_ARM_MODULE_PLTS
+bool in_module_plt(unsigned long loc);
+#else
+static inline bool in_module_plt(unsigned long loc) { return false; }
+#endif
 
 #ifdef CONFIG_THUMB2_KERNEL
 #define HAVE_ARCH_KALLSYMS_SYMBOL_VALUE
diff --git a/arch/arm/kernel/module-plts.c b/arch/arm/kernel/module-plts.c
index 1fc309b..a5351bf 100644
--- a/arch/arm/kernel/module-plts.c
+++ b/arch/arm/kernel/module-plts.c
@@ -284,3 +284,17 @@ int module_frob_arch_sections(Elf_Ehdr *ehdr, Elf_Shdr *sechdrs,
 		 mod->arch.core.plt->sh_size, mod->arch.init.plt->sh_size);
 	return 0;
 }
+
+bool in_module_plt(unsigned long loc)
+{
+	struct module *mod;
+	bool ret;
+
+	preempt_disable();
+	mod = __module_text_address(loc);
+	ret = mod && (loc - (u32)mod->arch.core.plt_ent < mod->arch.core.plt_count * PLT_ENT_SIZE ||
+		      loc - (u32)mod->arch.init.plt_ent < mod->arch.init.plt_count * PLT_ENT_SIZE);
+	preempt_enable();
+
+	return ret;
+}
diff --git a/arch/arm/kernel/unwind.c b/arch/arm/kernel/unwind.c
index 9232901..dd6c8dd 100644
--- a/arch/arm/kernel/unwind.c
+++ b/arch/arm/kernel/unwind.c
@@ -31,6 +31,7 @@
 #include <linux/slab.h>
 #include <linux/spinlock.h>
 #include <linux/list.h>
+#include <linux/module.h>
 
 #include <asm/stacktrace.h>
 #include <asm/traps.h>
@@ -394,8 +395,18 @@ int unwind_frame(struct stackframe *frame)
 
 	idx = unwind_find_idx(frame->pc);
 	if (!idx) {
-		if (frame->pc && kernel_text_address(frame->pc))
+		if (frame->pc && kernel_text_address(frame->pc)) {
+			if (in_module_plt(frame->pc) && frame->pc != frame->lr) {
+				/*
+				 * Quoting Ard: Veneers only set PC using a
+				 * PC+immediate LDR, and so they don't affect
+				 * the state of the stack or the register file
+				 */
+				frame->pc = frame->lr;
+				return URC_OK;
+			}
 			pr_warn("unwind: Index not found %08lx\n", frame->pc);
+		}
 		return -URC_FAILURE;
 	}
 
-- 

-- 
Best regards,
Alexander Sverdlin.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: 32-bit arm unwind info for PLTs

[Kursad Oney]

[-- Attachment #1.1: Type: text/plain, Size: 5767 bytes --]

Hi Alexander,

On Wed, Jul 6, 2022 at 4:47 AM Alexander Sverdlin
<alexander.sverdlin@nokia.com> wrote:
>
> Hello Kursad!
>
> On 02/04/2020 17:38, Kursad Oney wrote:
> > I have a large kernel module that gets loaded to vmalloc via
> > ARM_MODULE_PLTS. When I either use perf, or enable (yet unmerged)
> > KASAN-arm changes, I see occasional warnings like this:
> >
> > unwind: Index not found pc:f3aa8d6c
> >
> > The address f3aa8d6c is in the .plt section. I printed the backtrace
> > in unwind_frame where the warning is printed, and the backtrace looks
> > like this:
> >
> > NMI backtrace for cpu 0
> > CPU: 0 PID: 7193 Comm: hostapd Tainted: P                  4.19.100 #7
> > Hardware name: Generic DT based system
> > [<c02144b8>] (unwind_backtrace) from [<c020dac8>] (show_stack+0x10/0x14)
> > [<c020dac8>] (show_stack) from [<c083b238>] (dump_stack+0x9c/0xb0)
> > [<c083b238>] (dump_stack) from [<c0842dfc>] (nmi_cpu_backtrace+0xb4/0xe8)
> > [<c0842dfc>] (nmi_cpu_backtrace) from [<c0842fb0>] (nmi_trigger_cpumask_backtrace+0x180/0x1d4)
> > [<c0842fb0>] (nmi_trigger_cpumask_backtrace) from [<c0214428>] (unwind_frame+0x650/0x6e0)
> > [<c0214428>] (unwind_frame) from [<c020d53c>] (walk_stackframe+0x30/0x3c)
> > [<c020d53c>] (walk_stackframe) from [<c020d778>] (__save_stack_trace+0x100/0x108)
> > [<c020d778>] (__save_stack_trace) from [<c03369ec>] (__kasan_slab_free+0x124/0x1f8)
> > [<c03369ec>] (__kasan_slab_free) from [<c033396c>] (kmem_cache_free+0x5c/0x19c)
> > [<c033396c>] (kmem_cache_free) from [<c029138c>] (rcu_process_callbacks+0x360/0x604)
> > [<c029138c>] (rcu_process_callbacks) from [<c020295c>] (__do_softirq+0x174/0x374)
> > [<c020295c>] (__do_softirq) from [<c022b07c>] (irq_exit+0xd0/0xf8)
> > [<c022b07c>] (irq_exit) from [<c027ce3c>] (__handle_domain_irq+0x7c/0xd4)
> > [<c027ce3c>] (__handle_domain_irq) from [<c04dfc00>] (gic_handle_irq+0x4c/0x90)
> > [<c04dfc00>] (gic_handle_irq) from [<c02020cc>] (__irq_svc+0x6c/0xac)
> > Exception stack(0xe4c3b400 to 0xe4c3b448)
> > b400: dca01404 f3d63a80 bd7ac750 00000000 f3d63a84 f0b6c164 f3bb6434 dca0126c
> > b420: f3d665d0 dca01408 dca01404 0c002c24 00000000 e4c3b450 f37ed5a4 f3aa8d6c
> > b440: 600b0013 ffffffff
> > [<c02020cc>] (__irq_svc) from [<f3aa8d6c>] (wl_module_exit+0xf64/0x21f8 [wl])
> >
> > To me it looks like while the CPU is executing an instruction in the
> > PLT, it gets an interrupt. If we call save_stack_trace() or any
> > function that eventually gets to unwind_frame() from that context,
> > then the unwinder doesn't know how to unwind the PLT. Does this sound
> > right? Any idea how the unwinder code should deal with this situation?
>
> have you been able to fix the issue?

At the time, I implemented a workaround but haven't had a chance to
clean it up and submit upstream.

>
> Would you like to test the following patch?

I just did and it works for me. Thanks for the patch! You can include
my Tested-by tag (below) if you submit it upstream.

>
> diff --git a/arch/arm/include/asm/module.h b/arch/arm/include/asm/module.h
> index 5546c97..07c51a3 100644
> --- a/arch/arm/include/asm/module.h
> +++ b/arch/arm/include/asm/module.h
> @@ -37,6 +37,11 @@ struct mod_arch_specific {
>
>  struct module;
>  u32 get_module_plt(struct module *mod, unsigned long loc, Elf32_Addr val);
> +#ifdef CONFIG_ARM_MODULE_PLTS
> +bool in_module_plt(unsigned long loc);
> +#else
> +static inline bool in_module_plt(unsigned long loc) { return false; }
> +#endif
>
>  #ifdef CONFIG_THUMB2_KERNEL
>  #define HAVE_ARCH_KALLSYMS_SYMBOL_VALUE
> diff --git a/arch/arm/kernel/module-plts.c b/arch/arm/kernel/module-plts.c
> index 1fc309b..a5351bf 100644
> --- a/arch/arm/kernel/module-plts.c
> +++ b/arch/arm/kernel/module-plts.c
> @@ -284,3 +284,17 @@ int module_frob_arch_sections(Elf_Ehdr *ehdr, Elf_Shdr *sechdrs,
>                  mod->arch.core.plt->sh_size, mod->arch.init.plt->sh_size);
>         return 0;
>  }
> +
> +bool in_module_plt(unsigned long loc)
> +{
> +       struct module *mod;
> +       bool ret;
> +
> +       preempt_disable();
> +       mod = __module_text_address(loc);
> +       ret = mod && (loc - (u32)mod->arch.core.plt_ent < mod->arch.core.plt_count * PLT_ENT_SIZE ||
> +                     loc - (u32)mod->arch.init.plt_ent < mod->arch.init.plt_count * PLT_ENT_SIZE);
> +       preempt_enable();
> +
> +       return ret;
> +}
> diff --git a/arch/arm/kernel/unwind.c b/arch/arm/kernel/unwind.c
> index 9232901..dd6c8dd 100644
> --- a/arch/arm/kernel/unwind.c
> +++ b/arch/arm/kernel/unwind.c
> @@ -31,6 +31,7 @@
>  #include <linux/slab.h>
>  #include <linux/spinlock.h>
>  #include <linux/list.h>
> +#include <linux/module.h>
>
>  #include <asm/stacktrace.h>
>  #include <asm/traps.h>
> @@ -394,8 +395,18 @@ int unwind_frame(struct stackframe *frame)
>
>         idx = unwind_find_idx(frame->pc);
>         if (!idx) {
> -               if (frame->pc && kernel_text_address(frame->pc))
> +               if (frame->pc && kernel_text_address(frame->pc)) {
> +                       if (in_module_plt(frame->pc) && frame->pc != frame->lr) {
> +                               /*
> +                                * Quoting Ard: Veneers only set PC using a
> +                                * PC+immediate LDR, and so they don't affect
> +                                * the state of the stack or the register file
> +                                */
> +                               frame->pc = frame->lr;
> +                               return URC_OK;
> +                       }
>                         pr_warn("unwind: Index not found %08lx\n", frame->pc);
> +               }
>                 return -URC_FAILURE;
>         }
>
> --

Tested-by: Kursad Oney <kursad.oney@broadcom.com>

>
> --
> Best regards,
> Alexander Sverdlin.

[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4206 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: 32-bit arm unwind info for PLTs

[Alexander Sverdlin]

Hello Kursad,

On 06/07/2022 17:47, Kursad Oney wrote:
>>> To me it looks like while the CPU is executing an instruction in the
>>> PLT, it gets an interrupt. If we call save_stack_trace() or any
>>> function that eventually gets to unwind_frame() from that context,
>>> then the unwinder doesn't know how to unwind the PLT. Does this sound
>>> right? Any idea how the unwinder code should deal with this situation?
>> have you been able to fix the issue?
> At the time, I implemented a workaround but haven't had a chance to
> clean it up and submit upstream.
> 
>> Would you like to test the following patch?
> I just did and it works for me. Thanks for the patch! You can include
> my Tested-by tag (below) if you submit it upstream.

thanks a lot for the quick test!
I'll properly send it soon.

-- 
Best regards,
Alexander Sverdlin.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel