* Issue using faddr2line on kernel modules
@ 2022-01-17 4:57 Kaiwan N Billimoria
2022-01-17 19:48 ` Josh Poimboeuf
0 siblings, 1 reply; 15+ messages in thread
From: Kaiwan N Billimoria @ 2022-01-17 4:57 UTC (permalink / raw)
To: Linux Kernel Mailing List, jpoimboe; +Cc: Chi-Thanh Hoang
Hi there,
Am researching using the cool faddr2line script to help debug Oops'es
from kernel modules..
I find it works just fine when used against the unstripped vmlinux
with debug symbols.
My use case is for a kernel module which Oopses, though. Here's my scenario:
I built a module on a custom debug kernel (5.10.60) with most debug
options enabled...
KASLR is enabled by default as well.
A test kernel module Oopses on my x86_64 guest running this kernel with:
RIP: 0010:do_the_work+0x15b/0x174 [oops_tryv2]
So, i try this:
$ <...>/linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x15b/0x174
bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
$
(It works fine with addr2line though!).
Now I think I've traced the faddr2line script's failure to locate
anything down to this:
...
done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 ==
fn { found=1; line =$0; start=$1; next } found == 1 { found=0;
print line, "0x"$1 } END {if (found == 1) print line, end; }')
The nm output is:
$ nm -n ./oops_tryv2.ko |grep -i do_the_work
0000000000000000 t do_the_work
$
nm shows the text addr as 0x0; this is obviously incorrect (same 0x0
with objdump -d on the module).
Am I missing something? Any suggestions as to what I can try, to get
faddr2line working?
TIA,
Regards,
Kaiwan.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-01-17 4:57 Issue using faddr2line on kernel modules Kaiwan N Billimoria
@ 2022-01-17 19:48 ` Josh Poimboeuf
2022-01-17 20:27 ` Josh Poimboeuf
0 siblings, 1 reply; 15+ messages in thread
From: Josh Poimboeuf @ 2022-01-17 19:48 UTC (permalink / raw)
To: Kaiwan N Billimoria; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
On Mon, Jan 17, 2022 at 10:27:14AM +0530, Kaiwan N Billimoria wrote:
> Hi there,
>
> Am researching using the cool faddr2line script to help debug Oops'es
> from kernel modules..
> I find it works just fine when used against the unstripped vmlinux
> with debug symbols.
>
> My use case is for a kernel module which Oopses, though. Here's my scenario:
> I built a module on a custom debug kernel (5.10.60) with most debug
> options enabled...
> KASLR is enabled by default as well.
>
> A test kernel module Oopses on my x86_64 guest running this kernel with:
> RIP: 0010:do_the_work+0x15b/0x174 [oops_tryv2]
>
> So, i try this:
>
> $ <...>/linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x15b/0x174
> bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> $
>
> (It works fine with addr2line though!).
> Now I think I've traced the faddr2line script's failure to locate
> anything down to this:
> ...
> done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 ==
> fn { found=1; line =$0; start=$1; next } found == 1 { found=0;
> print line, "0x"$1 } END {if (found == 1) print line, end; }')
>
> The nm output is:
> $ nm -n ./oops_tryv2.ko |grep -i do_the_work
> 0000000000000000 t do_the_work
> $
>
> nm shows the text addr as 0x0; this is obviously incorrect (same 0x0
> with objdump -d on the module).
> Am I missing something? Any suggestions as to what I can try, to get
> faddr2line working?
Hi Kaiwan,
Thanks for reporting this issue. The module text address of 0x0 is not
necessarily incorrect, as the address is relative the the module, where
all text usually starts at zero.
I was able to recreate this problem using a module which only has a
single function in .text. Does this fix it?
diff --git a/scripts/faddr2line b/scripts/faddr2line
index 6c6439f69a72..cd6e812b9d06 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -111,21 +111,11 @@ __faddr2line() {
# Go through each of the object's symbols which match the func name.
# In rare cases there might be duplicates.
- file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
while read symbol; do
local fields=($symbol)
local sym_base=0x${fields[0]}
- local sym_type=${fields[1]}
- local sym_end=${fields[3]}
-
- # calculate the size
- local sym_size=$(($sym_end - $sym_base))
- if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
- warn "bad symbol size: base: $sym_base end: $sym_end"
- DONE=1
- return
- fi
- sym_size=0x$(printf %x $sym_size)
+ local sym_size=0x${fields[1]}
+ local sym_type=${fields[2]}
# calculate the address
local addr=$(($sym_base + $offset))
@@ -189,7 +179,7 @@ __faddr2line() {
DONE=1
- done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
+ done < <(${NM} -nS $objfile | awk -v fn=$func '$4 == fn')
}
[[ $# -lt 2 ]] && usage
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-01-17 19:48 ` Josh Poimboeuf
@ 2022-01-17 20:27 ` Josh Poimboeuf
2022-01-18 2:40 ` Kaiwan N Billimoria
0 siblings, 1 reply; 15+ messages in thread
From: Josh Poimboeuf @ 2022-01-17 20:27 UTC (permalink / raw)
To: Kaiwan N Billimoria; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
On Mon, Jan 17, 2022 at 11:48:39AM -0800, Josh Poimboeuf wrote:
> On Mon, Jan 17, 2022 at 10:27:14AM +0530, Kaiwan N Billimoria wrote:
> > Hi there,
> >
> > Am researching using the cool faddr2line script to help debug Oops'es
> > from kernel modules..
> > I find it works just fine when used against the unstripped vmlinux
> > with debug symbols.
> >
> > My use case is for a kernel module which Oopses, though. Here's my scenario:
> > I built a module on a custom debug kernel (5.10.60) with most debug
> > options enabled...
> > KASLR is enabled by default as well.
> >
> > A test kernel module Oopses on my x86_64 guest running this kernel with:
> > RIP: 0010:do_the_work+0x15b/0x174 [oops_tryv2]
> >
> > So, i try this:
> >
> > $ <...>/linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x15b/0x174
> > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> > $
> >
> > (It works fine with addr2line though!).
> > Now I think I've traced the faddr2line script's failure to locate
> > anything down to this:
> > ...
> > done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 ==
> > fn { found=1; line =$0; start=$1; next } found == 1 { found=0;
> > print line, "0x"$1 } END {if (found == 1) print line, end; }')
> >
> > The nm output is:
> > $ nm -n ./oops_tryv2.ko |grep -i do_the_work
> > 0000000000000000 t do_the_work
> > $
> >
> > nm shows the text addr as 0x0; this is obviously incorrect (same 0x0
> > with objdump -d on the module).
> > Am I missing something? Any suggestions as to what I can try, to get
> > faddr2line working?
>
> Hi Kaiwan,
>
> Thanks for reporting this issue. The module text address of 0x0 is not
> necessarily incorrect, as the address is relative the the module, where
> all text usually starts at zero.
>
> I was able to recreate this problem using a module which only has a
> single function in .text. Does this fix it?
Actually, that patch has other problems. Try this one?
----
From: Josh Poimboeuf <jpoimboe@redhat.com>
Subject: [PATCH] scripts/faddr2line: Only look for text symbols when
calculating function size
With the following commit:
efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
... it was discovered that faddr2line can't just read a function's ELF
size, because that wouldn't match the kallsyms function size which is
printed in the stack trace. The kallsyms size includes any padding
after the function, whereas the ELF size does not.
So faddr2line has to manually calculate the size of a function similar
to how kallsyms does. It does so by starting with a sorted list of
symbols and subtracting the function address from the subsequent
symbol's address.
That calculation is broken in the case where the function is the last
(or only) symbol in the .text section. The next symbol in the sorted
list might actually be a data symbol, which can break the function size
detection:
$ scripts/faddr2line sound/soundcore.ko sound_devnode+0x5/0x35
bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
Similar breakage can occur when reading from a .o file.
Fix it by only looking for text symbols.
Fixes: efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
Reported-by: Kaiwan N Billimoria <kaiwan.billimoria@gmail.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
---
scripts/faddr2line | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/faddr2line b/scripts/faddr2line
index 6c6439f69a72..2a130134f1e6 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -189,7 +189,7 @@ __faddr2line() {
DONE=1
- done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
+ done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$2 !~ /[Tt]/ {next} $3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
}
[[ $# -lt 2 ]] && usage
--
2.31.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-01-17 20:27 ` Josh Poimboeuf
@ 2022-01-18 2:40 ` Kaiwan N Billimoria
2022-01-18 17:52 ` Josh Poimboeuf
0 siblings, 1 reply; 15+ messages in thread
From: Kaiwan N Billimoria @ 2022-01-18 2:40 UTC (permalink / raw)
To: Josh Poimboeuf; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
Hi Josh,
Actually your first patch - the one you mentioned had other issues -
worked perfectly when applied:
scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
do_the_work+0x16f/0x0000000000000194:
do_the_work at <...>/oops_tryv2/oops_tryv2.c:62
The second one still failed in the same manner:
scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
So, is it possible to fixup issues with the first version?
What are these issues?
Thanks,
Kaiwan.
On Tue, Jan 18, 2022 at 1:57 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
>
> On Mon, Jan 17, 2022 at 11:48:39AM -0800, Josh Poimboeuf wrote:
> > On Mon, Jan 17, 2022 at 10:27:14AM +0530, Kaiwan N Billimoria wrote:
> > > Hi there,
> > >
> > > Am researching using the cool faddr2line script to help debug Oops'es
> > > from kernel modules..
> > > I find it works just fine when used against the unstripped vmlinux
> > > with debug symbols.
> > >
> > > My use case is for a kernel module which Oopses, though. Here's my scenario:
> > > I built a module on a custom debug kernel (5.10.60) with most debug
> > > options enabled...
> > > KASLR is enabled by default as well.
> > >
> > > A test kernel module Oopses on my x86_64 guest running this kernel with:
> > > RIP: 0010:do_the_work+0x15b/0x174 [oops_tryv2]
> > >
> > > So, i try this:
> > >
> > > $ <...>/linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x15b/0x174
> > > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> > > $
> > >
> > > (It works fine with addr2line though!).
> > > Now I think I've traced the faddr2line script's failure to locate
> > > anything down to this:
> > > ...
> > > done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 ==
> > > fn { found=1; line =$0; start=$1; next } found == 1 { found=0;
> > > print line, "0x"$1 } END {if (found == 1) print line, end; }')
> > >
> > > The nm output is:
> > > $ nm -n ./oops_tryv2.ko |grep -i do_the_work
> > > 0000000000000000 t do_the_work
> > > $
> > >
> > > nm shows the text addr as 0x0; this is obviously incorrect (same 0x0
> > > with objdump -d on the module).
> > > Am I missing something? Any suggestions as to what I can try, to get
> > > faddr2line working?
> >
> > Hi Kaiwan,
> >
> > Thanks for reporting this issue. The module text address of 0x0 is not
> > necessarily incorrect, as the address is relative the the module, where
> > all text usually starts at zero.
> >
> > I was able to recreate this problem using a module which only has a
> > single function in .text. Does this fix it?
>
> Actually, that patch has other problems. Try this one?
>
> ----
>
> From: Josh Poimboeuf <jpoimboe@redhat.com>
> Subject: [PATCH] scripts/faddr2line: Only look for text symbols when
> calculating function size
>
> With the following commit:
>
> efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
>
> ... it was discovered that faddr2line can't just read a function's ELF
> size, because that wouldn't match the kallsyms function size which is
> printed in the stack trace. The kallsyms size includes any padding
> after the function, whereas the ELF size does not.
>
> So faddr2line has to manually calculate the size of a function similar
> to how kallsyms does. It does so by starting with a sorted list of
> symbols and subtracting the function address from the subsequent
> symbol's address.
>
> That calculation is broken in the case where the function is the last
> (or only) symbol in the .text section. The next symbol in the sorted
> list might actually be a data symbol, which can break the function size
> detection:
>
> $ scripts/faddr2line sound/soundcore.ko sound_devnode+0x5/0x35
> bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
>
> Similar breakage can occur when reading from a .o file.
>
> Fix it by only looking for text symbols.
>
> Fixes: efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
> Reported-by: Kaiwan N Billimoria <kaiwan.billimoria@gmail.com>
> Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
> ---
> scripts/faddr2line | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/faddr2line b/scripts/faddr2line
> index 6c6439f69a72..2a130134f1e6 100755
> --- a/scripts/faddr2line
> +++ b/scripts/faddr2line
> @@ -189,7 +189,7 @@ __faddr2line() {
>
> DONE=1
>
> - done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> + done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$2 !~ /[Tt]/ {next} $3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> }
>
> [[ $# -lt 2 ]] && usage
> --
> 2.31.1
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-01-18 2:40 ` Kaiwan N Billimoria
@ 2022-01-18 17:52 ` Josh Poimboeuf
2022-01-19 3:57 ` Kaiwan N Billimoria
0 siblings, 1 reply; 15+ messages in thread
From: Josh Poimboeuf @ 2022-01-18 17:52 UTC (permalink / raw)
To: Kaiwan N Billimoria; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
On Tue, Jan 18, 2022 at 08:10:28AM +0530, Kaiwan N Billimoria wrote:
> Hi Josh,
>
> Actually your first patch - the one you mentioned had other issues -
> worked perfectly when applied:
>
> scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
> do_the_work+0x16f/0x0000000000000194:
> do_the_work at <...>/oops_tryv2/oops_tryv2.c:62
>
> The second one still failed in the same manner:
>
> scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
> bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
>
> So, is it possible to fixup issues with the first version?
> What are these issues?
The first patch basically reverts the fix in commit efdb4167e676
("scripts/faddr2line: Fix "size mismatch" error"). That would be nice
as it's simpler and more robust, but unfortunately it would cause a lot
of "size mismatch" errors with vmlinux symbols.
Can you give the output of 'nm -n ./oops_tryv2.ko'? There must be some
text symbol immediately after the do_the_work() symbol which is either
out of order, or part of another section.
Is do_the_work() in the .text section?
> On Tue, Jan 18, 2022 at 1:57 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> >
> > On Mon, Jan 17, 2022 at 11:48:39AM -0800, Josh Poimboeuf wrote:
> > > On Mon, Jan 17, 2022 at 10:27:14AM +0530, Kaiwan N Billimoria wrote:
> > > > Hi there,
> > > >
> > > > Am researching using the cool faddr2line script to help debug Oops'es
> > > > from kernel modules..
> > > > I find it works just fine when used against the unstripped vmlinux
> > > > with debug symbols.
> > > >
> > > > My use case is for a kernel module which Oopses, though. Here's my scenario:
> > > > I built a module on a custom debug kernel (5.10.60) with most debug
> > > > options enabled...
> > > > KASLR is enabled by default as well.
> > > >
> > > > A test kernel module Oopses on my x86_64 guest running this kernel with:
> > > > RIP: 0010:do_the_work+0x15b/0x174 [oops_tryv2]
> > > >
> > > > So, i try this:
> > > >
> > > > $ <...>/linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x15b/0x174
> > > > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> > > > $
> > > >
> > > > (It works fine with addr2line though!).
> > > > Now I think I've traced the faddr2line script's failure to locate
> > > > anything down to this:
> > > > ...
> > > > done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 ==
> > > > fn { found=1; line =$0; start=$1; next } found == 1 { found=0;
> > > > print line, "0x"$1 } END {if (found == 1) print line, end; }')
> > > >
> > > > The nm output is:
> > > > $ nm -n ./oops_tryv2.ko |grep -i do_the_work
> > > > 0000000000000000 t do_the_work
> > > > $
> > > >
> > > > nm shows the text addr as 0x0; this is obviously incorrect (same 0x0
> > > > with objdump -d on the module).
> > > > Am I missing something? Any suggestions as to what I can try, to get
> > > > faddr2line working?
> > >
> > > Hi Kaiwan,
> > >
> > > Thanks for reporting this issue. The module text address of 0x0 is not
> > > necessarily incorrect, as the address is relative the the module, where
> > > all text usually starts at zero.
> > >
> > > I was able to recreate this problem using a module which only has a
> > > single function in .text. Does this fix it?
> >
> > Actually, that patch has other problems. Try this one?
> >
> > ----
> >
> > From: Josh Poimboeuf <jpoimboe@redhat.com>
> > Subject: [PATCH] scripts/faddr2line: Only look for text symbols when
> > calculating function size
> >
> > With the following commit:
> >
> > efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
> >
> > ... it was discovered that faddr2line can't just read a function's ELF
> > size, because that wouldn't match the kallsyms function size which is
> > printed in the stack trace. The kallsyms size includes any padding
> > after the function, whereas the ELF size does not.
> >
> > So faddr2line has to manually calculate the size of a function similar
> > to how kallsyms does. It does so by starting with a sorted list of
> > symbols and subtracting the function address from the subsequent
> > symbol's address.
> >
> > That calculation is broken in the case where the function is the last
> > (or only) symbol in the .text section. The next symbol in the sorted
> > list might actually be a data symbol, which can break the function size
> > detection:
> >
> > $ scripts/faddr2line sound/soundcore.ko sound_devnode+0x5/0x35
> > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> >
> > Similar breakage can occur when reading from a .o file.
> >
> > Fix it by only looking for text symbols.
> >
> > Fixes: efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
> > Reported-by: Kaiwan N Billimoria <kaiwan.billimoria@gmail.com>
> > Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
> > ---
> > scripts/faddr2line | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/scripts/faddr2line b/scripts/faddr2line
> > index 6c6439f69a72..2a130134f1e6 100755
> > --- a/scripts/faddr2line
> > +++ b/scripts/faddr2line
> > @@ -189,7 +189,7 @@ __faddr2line() {
> >
> > DONE=1
> >
> > - done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> > + done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$2 !~ /[Tt]/ {next} $3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> > }
> >
> > [[ $# -lt 2 ]] && usage
> > --
> > 2.31.1
> >
>
--
Josh
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-01-18 17:52 ` Josh Poimboeuf
@ 2022-01-19 3:57 ` Kaiwan N Billimoria
2022-01-19 21:21 ` Josh Poimboeuf
0 siblings, 1 reply; 15+ messages in thread
From: Kaiwan N Billimoria @ 2022-01-19 3:57 UTC (permalink / raw)
To: Josh Poimboeuf; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
On Tue, Jan 18, 2022 at 11:22 PM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
>
> On Tue, Jan 18, 2022 at 08:10:28AM +0530, Kaiwan N Billimoria wrote:
> > Hi Josh,
> >
> > Actually your first patch - the one you mentioned had other issues -
> > worked perfectly when applied:
> >
> > scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
> > do_the_work+0x16f/0x0000000000000194:
> > do_the_work at <...>/oops_tryv2/oops_tryv2.c:62
> >
> > The second one still failed in the same manner:
> >
> > scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
> > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> >
> > So, is it possible to fixup issues with the first version?
> > What are these issues?
>
> The first patch basically reverts the fix in commit efdb4167e676
> ("scripts/faddr2line: Fix "size mismatch" error"). That would be nice
> as it's simpler and more robust, but unfortunately it would cause a lot
> of "size mismatch" errors with vmlinux symbols.
I see... if it comes to it, could there be a workaround where the
script detects a vmlinux image and works in a given manner,
else in this manner for other images - in effect, for modules?
>
> Can you give the output of 'nm -n ./oops_tryv2.ko'? There must be some
> text symbol immediately after the do_the_work() symbol which is either
> out of order, or part of another section.
>
> Is do_the_work() in the .text section?
Yes...
Here's the output I get:
$ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
0000000000000000 r __func__.24215
0000000000000000 r __param_bug_in_workq
0000000000000000 D __this_module
0000000000000000 r _note_7
0000000000000000 T cleanup_module
0000000000000000 t do_the_work
0000000000000000 t do_the_work.cold
0000000000000000 b gctx
0000000000000000 T init_module
0000000000000000 t try_oops_exit
0000000000000000 t try_oops_init
0000000000000008 b t1
$
BTW, here's the code:
https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2
Thanks,
Kaiwan.
>
> > On Tue, Jan 18, 2022 at 1:57 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> > >
> > > On Mon, Jan 17, 2022 at 11:48:39AM -0800, Josh Poimboeuf wrote:
> > > > On Mon, Jan 17, 2022 at 10:27:14AM +0530, Kaiwan N Billimoria wrote:
> > > > > Hi there,
> > > > >
> > > > > Am researching using the cool faddr2line script to help debug Oops'es
> > > > > from kernel modules..
> > > > > I find it works just fine when used against the unstripped vmlinux
> > > > > with debug symbols.
> > > > >
> > > > > My use case is for a kernel module which Oopses, though. Here's my scenario:
> > > > > I built a module on a custom debug kernel (5.10.60) with most debug
> > > > > options enabled...
> > > > > KASLR is enabled by default as well.
> > > > >
> > > > > A test kernel module Oopses on my x86_64 guest running this kernel with:
> > > > > RIP: 0010:do_the_work+0x15b/0x174 [oops_tryv2]
> > > > >
> > > > > So, i try this:
> > > > >
> > > > > $ <...>/linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x15b/0x174
> > > > > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> > > > > $
> > > > >
> > > > > (It works fine with addr2line though!).
> > > > > Now I think I've traced the faddr2line script's failure to locate
> > > > > anything down to this:
> > > > > ...
> > > > > done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 ==
> > > > > fn { found=1; line =$0; start=$1; next } found == 1 { found=0;
> > > > > print line, "0x"$1 } END {if (found == 1) print line, end; }')
> > > > >
> > > > > The nm output is:
> > > > > $ nm -n ./oops_tryv2.ko |grep -i do_the_work
> > > > > 0000000000000000 t do_the_work
> > > > > $
> > > > >
> > > > > nm shows the text addr as 0x0; this is obviously incorrect (same 0x0
> > > > > with objdump -d on the module).
> > > > > Am I missing something? Any suggestions as to what I can try, to get
> > > > > faddr2line working?
> > > >
> > > > Hi Kaiwan,
> > > >
> > > > Thanks for reporting this issue. The module text address of 0x0 is not
> > > > necessarily incorrect, as the address is relative the the module, where
> > > > all text usually starts at zero.
> > > >
> > > > I was able to recreate this problem using a module which only has a
> > > > single function in .text. Does this fix it?
> > >
> > > Actually, that patch has other problems. Try this one?
> > >
> > > ----
> > >
> > > From: Josh Poimboeuf <jpoimboe@redhat.com>
> > > Subject: [PATCH] scripts/faddr2line: Only look for text symbols when
> > > calculating function size
> > >
> > > With the following commit:
> > >
> > > efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
> > >
> > > ... it was discovered that faddr2line can't just read a function's ELF
> > > size, because that wouldn't match the kallsyms function size which is
> > > printed in the stack trace. The kallsyms size includes any padding
> > > after the function, whereas the ELF size does not.
> > >
> > > So faddr2line has to manually calculate the size of a function similar
> > > to how kallsyms does. It does so by starting with a sorted list of
> > > symbols and subtracting the function address from the subsequent
> > > symbol's address.
> > >
> > > That calculation is broken in the case where the function is the last
> > > (or only) symbol in the .text section. The next symbol in the sorted
> > > list might actually be a data symbol, which can break the function size
> > > detection:
> > >
> > > $ scripts/faddr2line sound/soundcore.ko sound_devnode+0x5/0x35
> > > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> > >
> > > Similar breakage can occur when reading from a .o file.
> > >
> > > Fix it by only looking for text symbols.
> > >
> > > Fixes: efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
> > > Reported-by: Kaiwan N Billimoria <kaiwan.billimoria@gmail.com>
> > > Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
> > > ---
> > > scripts/faddr2line | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/scripts/faddr2line b/scripts/faddr2line
> > > index 6c6439f69a72..2a130134f1e6 100755
> > > --- a/scripts/faddr2line
> > > +++ b/scripts/faddr2line
> > > @@ -189,7 +189,7 @@ __faddr2line() {
> > >
> > > DONE=1
> > >
> > > - done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> > > + done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$2 !~ /[Tt]/ {next} $3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> > > }
> > >
> > > [[ $# -lt 2 ]] && usage
> > > --
> > > 2.31.1
> > >
> >
>
> --
> Josh
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-01-19 3:57 ` Kaiwan N Billimoria
@ 2022-01-19 21:21 ` Josh Poimboeuf
2022-01-19 23:43 ` Josh Poimboeuf
0 siblings, 1 reply; 15+ messages in thread
From: Josh Poimboeuf @ 2022-01-19 21:21 UTC (permalink / raw)
To: Kaiwan N Billimoria; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
On Wed, Jan 19, 2022 at 09:27:18AM +0530, Kaiwan N Billimoria wrote:
> On Tue, Jan 18, 2022 at 11:22 PM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> >
> > On Tue, Jan 18, 2022 at 08:10:28AM +0530, Kaiwan N Billimoria wrote:
> > > Hi Josh,
> > >
> > > Actually your first patch - the one you mentioned had other issues -
> > > worked perfectly when applied:
> > >
> > > scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
> > > do_the_work+0x16f/0x0000000000000194:
> > > do_the_work at <...>/oops_tryv2/oops_tryv2.c:62
> > >
> > > The second one still failed in the same manner:
> > >
> > > scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
> > > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> > >
> > > So, is it possible to fixup issues with the first version?
> > > What are these issues?
> >
> > The first patch basically reverts the fix in commit efdb4167e676
> > ("scripts/faddr2line: Fix "size mismatch" error"). That would be nice
> > as it's simpler and more robust, but unfortunately it would cause a lot
> > of "size mismatch" errors with vmlinux symbols.
>
> I see... if it comes to it, could there be a workaround where the
> script detects a vmlinux image and works in a given manner,
> else in this manner for other images - in effect, for modules?
I think I misspoke about the vmlinux behavior being different, so I
don't think that would work.
> > Can you give the output of 'nm -n ./oops_tryv2.ko'? There must be some
> > text symbol immediately after the do_the_work() symbol which is either
> > out of order, or part of another section.
> >
> > Is do_the_work() in the .text section?
> Yes...
> Here's the output I get:
>
> $ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
> 0000000000000000 r __func__.24215
> 0000000000000000 r __param_bug_in_workq
> 0000000000000000 D __this_module
> 0000000000000000 r _note_7
> 0000000000000000 T cleanup_module
> 0000000000000000 t do_the_work
> 0000000000000000 t do_the_work.cold
> 0000000000000000 b gctx
> 0000000000000000 T init_module
> 0000000000000000 t try_oops_exit
> 0000000000000000 t try_oops_init
> 0000000000000008 b t1
> $
>
> BTW, here's the code:
> https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2
Ok, it looks like the symbols aren't sorted like the code expects. I
need to do a more robust fix.
--
Josh
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-01-19 21:21 ` Josh Poimboeuf
@ 2022-01-19 23:43 ` Josh Poimboeuf
2022-01-20 5:14 ` Kaiwan N Billimoria
0 siblings, 1 reply; 15+ messages in thread
From: Josh Poimboeuf @ 2022-01-19 23:43 UTC (permalink / raw)
To: Kaiwan N Billimoria; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
On Wed, Jan 19, 2022 at 01:21:29PM -0800, Josh Poimboeuf wrote:
> > $ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
> > 0000000000000000 r __func__.24215
> > 0000000000000000 r __param_bug_in_workq
> > 0000000000000000 D __this_module
> > 0000000000000000 r _note_7
> > 0000000000000000 T cleanup_module
> > 0000000000000000 t do_the_work
> > 0000000000000000 t do_the_work.cold
> > 0000000000000000 b gctx
> > 0000000000000000 T init_module
> > 0000000000000000 t try_oops_exit
> > 0000000000000000 t try_oops_init
> > 0000000000000008 b t1
> > $
> >
> > BTW, here's the code:
> > https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2
>
> Ok, it looks like the symbols aren't sorted like the code expects. I
> need to do a more robust fix.
Ok, please try this instead. This takes a much more robust approach to
the function size calculation, using readelf to confine the symbol
search to the section matching the original symbol.
This actually has multiple fixes and cleanups, so it'll eventually be
split up into a patch set.
----
diff --git a/scripts/faddr2line b/scripts/faddr2line
index 6c6439f69a72..1acb68927977 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -97,10 +97,11 @@ __faddr2line() {
local dir_prefix=$3
local print_warnings=$4
+ local first=1
local func=${func_addr%+*}
local offset=${func_addr#*+}
offset=${offset%/*}
- local size=
+ local user_size=
[[ $func_addr =~ "/" ]] && size=${func_addr#*/}
if [[ -z $func ]] || [[ -z $offset ]] || [[ $func = $func_addr ]]; then
@@ -110,73 +111,87 @@ __faddr2line() {
fi
# Go through each of the object's symbols which match the func name.
- # In rare cases there might be duplicates.
- file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
+ # In rare cases there might be duplicates, in which case we print both.
while read symbol; do
local fields=($symbol)
- local sym_base=0x${fields[0]}
- local sym_type=${fields[1]}
- local sym_end=${fields[3]}
-
- # calculate the size
- local sym_size=$(($sym_end - $sym_base))
- if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
- warn "bad symbol size: base: $sym_base end: $sym_end"
+ local sym_addr=0x${fields[1]}
+ local sym_size=${fields[2]}
+ local sym_sec=${fields[6]}
+
+ # Get the section size:
+ local sec_size=$(${READELF} --section-headers --wide $objfile |
+ sed 's/\[ /\[/' |
+ awk -v sec=$sym_sec '$1 == "[" sec "]" { print "0x" $6; exit }')
+
+ if [[ -z $sec_size ]]; then
+ warn "bad section size: section: $sym_sec"
DONE=1
return
fi
- sym_size=0x$(printf %x $sym_size)
- # calculate the address
- local addr=$(($sym_base + $offset))
- if [[ -z $addr ]] || [[ $addr = 0 ]]; then
- warn "bad address: $sym_base + $offset"
+ # Calculate the symbol size:
+ #
+ # We can't use the ELF size, because kallsyms also includes the
+ # padding bytes in its size calculation. For kallsyms, the
+ # size calculation is the distance between the symbol and the
+ # next symbol in a sorted list.
+ local size=$(${READELF} --symbols --wide $objfile |
+ awk -v sec=$sym_sec '$7 == sec' |
+ sort --key=2 |
+ awk --bignum -v sym_addr=${sym_addr#0x} -v sym_size=${sym_size} -v sym_name=${func} -v sec_size=${sec_size} \
+ '$2 == sym_addr && $3 == sym_size && $8 == sym_name { found = 1; next } \
+ found == 1 { size = strtonum("0x" $2) - strtonum("0x" sym_addr); if (size < sym_size) next; found = 2; print size; exit } \
+ END { if (found == 1) print strtonum(sec_size) }')
+
+ if [[ -z $size ]]; then
+ warn "bad symbol size: addr: $sym_addr"
DONE=1
return
fi
- addr=0x$(printf %x $addr)
- # weed out non-function symbols
- if [[ $sym_type != t ]] && [[ $sym_type != T ]]; then
- [[ $print_warnings = 1 ]] &&
- echo "skipping $func address at $addr due to non-function symbol of type '$sym_type'"
- continue
+ # Calculate the specified address:
+ local addr=$(($sym_addr + $offset))
+ if [[ -z $addr ]] || [[ $addr = 0 ]]; then
+ warn "bad address: $sym_addr + $offset"
+ DONE=1
+ return
fi
+ addr=0x$(printf %x $addr)
- # if the user provided a size, make sure it matches the symbol's size
- if [[ -n $size ]] && [[ $size -ne $sym_size ]]; then
+ # If the user provided a size, make sure it matches the symbol's size:
+ if [[ -n $user_size ]] && [[ $user_size -ne $size ]]; then
[[ $print_warnings = 1 ]] &&
- echo "skipping $func address at $addr due to size mismatch ($size != $sym_size)"
+ echo "skipping $func address at $addr due to size mismatch ($user_size != $size)"
continue;
fi
- # make sure the provided offset is within the symbol's range
- if [[ $offset -gt $sym_size ]]; then
+ # Make sure the provided offset is within the symbol's range:
+ if [[ $offset -gt $size ]]; then
[[ $print_warnings = 1 ]] &&
- echo "skipping $func address at $addr due to size mismatch ($offset > $sym_size)"
+ echo "skipping $func address at $addr due to size mismatch ($offset > $size)"
continue
fi
- # separate multiple entries with a blank line
- [[ $FIRST = 0 ]] && echo
- FIRST=0
+ # In case of duplicates, separate multiple entries with a blank line:
+ [[ $first = 0 ]] && echo
+ first=0
- # pass real address to addr2line
- echo "$func+$offset/$sym_size:"
- local file_lines=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
- [[ -z $file_lines ]] && return
+ # Pass full address to addr2line:
+ echo "$func+$offset/$size:"
+ local output=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
+ [[ -z $output ]] && continue
if [[ $LIST = 0 ]]; then
- echo "$file_lines" | while read -r line
+ echo "$output" | while read -r line
do
echo $line
done
DONE=1;
- return
+ continue
fi
- # show each line with context
- echo "$file_lines" | while read -r line
+ # If --list was specified, show each line with context:
+ echo "$output" | while read -r line
do
echo
echo $line
@@ -189,7 +204,7 @@ __faddr2line() {
DONE=1
- done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
+ done < <(${READELF} --symbols --wide $objfile | awk -v fn=$func '$4 == "FUNC" && $8 == fn')
}
[[ $# -lt 2 ]] && usage
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-01-19 23:43 ` Josh Poimboeuf
@ 2022-01-20 5:14 ` Kaiwan N Billimoria
2022-01-20 17:17 ` Josh Poimboeuf
0 siblings, 1 reply; 15+ messages in thread
From: Kaiwan N Billimoria @ 2022-01-20 5:14 UTC (permalink / raw)
To: Josh Poimboeuf; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
On Thu, Jan 20, 2022 at 5:13 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
>
> On Wed, Jan 19, 2022 at 01:21:29PM -0800, Josh Poimboeuf wrote:
> > > $ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
> > > 0000000000000000 r __func__.24215
> > > 0000000000000000 r __param_bug_in_workq
> > > 0000000000000000 D __this_module
> > > 0000000000000000 r _note_7
> > > 0000000000000000 T cleanup_module
> > > 0000000000000000 t do_the_work
> > > 0000000000000000 t do_the_work.cold
> > > 0000000000000000 b gctx
> > > 0000000000000000 T init_module
> > > 0000000000000000 t try_oops_exit
> > > 0000000000000000 t try_oops_init
> > > 0000000000000008 b t1
> > > $
> > >
> > > BTW, here's the code:
> > > https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2
> >
> > Ok, it looks like the symbols aren't sorted like the code expects. I
> > need to do a more robust fix.
>
> Ok, please try this instead. This takes a much more robust approach to
> the function size calculation, using readelf to confine the symbol
> search to the section matching the original symbol.
>
> This actually has multiple fixes and cleanups, so it'll eventually be
> split up into a patch set.
Great. Am facing a few issues though:
1. Not sure if it's me but am having issues applying your patches; it
seems to fail:
$ patch -p1 --dry-run < ./fa1.patch
checking file scripts/faddr2line
Hunk #1 FAILED at 97.
Hunk #2 FAILED at 110.
Hunk #3 FAILED at 189.
3 out of 3 hunks FAILED
$
Against which ver are you generating the patch?
(Am using 5.10.60; i noticed the last commit in mainline was f5f67cc
in Nov 2018, so vanilla 5.10.60 should be fine, no?).
2. I manually applied your patch :) This is the output I then get:
$ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
awk: not an option: --bignum
bad symbol size: addr: 0x0000000000000000
$
(Not sure if this helps but am running on Ubuntu 20.04; my awk is mawk
v 1.3.4.20200120-2).
Thanks,
Kaiwan.
>
> ----
>
> diff --git a/scripts/faddr2line b/scripts/faddr2line
> index 6c6439f69a72..1acb68927977 100755
> --- a/scripts/faddr2line
> +++ b/scripts/faddr2line
> @@ -97,10 +97,11 @@ __faddr2line() {
> local dir_prefix=$3
> local print_warnings=$4
>
> + local first=1
> local func=${func_addr%+*}
> local offset=${func_addr#*+}
> offset=${offset%/*}
> - local size=
> + local user_size=
> [[ $func_addr =~ "/" ]] && size=${func_addr#*/}
>
> if [[ -z $func ]] || [[ -z $offset ]] || [[ $func = $func_addr ]]; then
> @@ -110,73 +111,87 @@ __faddr2line() {
> fi
>
> # Go through each of the object's symbols which match the func name.
> - # In rare cases there might be duplicates.
> - file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
> + # In rare cases there might be duplicates, in which case we print both.
> while read symbol; do
> local fields=($symbol)
> - local sym_base=0x${fields[0]}
> - local sym_type=${fields[1]}
> - local sym_end=${fields[3]}
> -
> - # calculate the size
> - local sym_size=$(($sym_end - $sym_base))
> - if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
> - warn "bad symbol size: base: $sym_base end: $sym_end"
> + local sym_addr=0x${fields[1]}
> + local sym_size=${fields[2]}
> + local sym_sec=${fields[6]}
> +
> + # Get the section size:
> + local sec_size=$(${READELF} --section-headers --wide $objfile |
> + sed 's/\[ /\[/' |
> + awk -v sec=$sym_sec '$1 == "[" sec "]" { print "0x" $6; exit }')
> +
> + if [[ -z $sec_size ]]; then
> + warn "bad section size: section: $sym_sec"
> DONE=1
> return
> fi
> - sym_size=0x$(printf %x $sym_size)
>
> - # calculate the address
> - local addr=$(($sym_base + $offset))
> - if [[ -z $addr ]] || [[ $addr = 0 ]]; then
> - warn "bad address: $sym_base + $offset"
> + # Calculate the symbol size:
> + #
> + # We can't use the ELF size, because kallsyms also includes the
> + # padding bytes in its size calculation. For kallsyms, the
> + # size calculation is the distance between the symbol and the
> + # next symbol in a sorted list.
> + local size=$(${READELF} --symbols --wide $objfile |
> + awk -v sec=$sym_sec '$7 == sec' |
> + sort --key=2 |
> + awk --bignum -v sym_addr=${sym_addr#0x} -v sym_size=${sym_size} -v sym_name=${func} -v sec_size=${sec_size} \
> + '$2 == sym_addr && $3 == sym_size && $8 == sym_name { found = 1; next } \
> + found == 1 { size = strtonum("0x" $2) - strtonum("0x" sym_addr); if (size < sym_size) next; found = 2; print size; exit } \
> + END { if (found == 1) print strtonum(sec_size) }')
> +
> + if [[ -z $size ]]; then
> + warn "bad symbol size: addr: $sym_addr"
> DONE=1
> return
> fi
> - addr=0x$(printf %x $addr)
>
> - # weed out non-function symbols
> - if [[ $sym_type != t ]] && [[ $sym_type != T ]]; then
> - [[ $print_warnings = 1 ]] &&
> - echo "skipping $func address at $addr due to non-function symbol of type '$sym_type'"
> - continue
> + # Calculate the specified address:
> + local addr=$(($sym_addr + $offset))
> + if [[ -z $addr ]] || [[ $addr = 0 ]]; then
> + warn "bad address: $sym_addr + $offset"
> + DONE=1
> + return
> fi
> + addr=0x$(printf %x $addr)
>
> - # if the user provided a size, make sure it matches the symbol's size
> - if [[ -n $size ]] && [[ $size -ne $sym_size ]]; then
> + # If the user provided a size, make sure it matches the symbol's size:
> + if [[ -n $user_size ]] && [[ $user_size -ne $size ]]; then
> [[ $print_warnings = 1 ]] &&
> - echo "skipping $func address at $addr due to size mismatch ($size != $sym_size)"
> + echo "skipping $func address at $addr due to size mismatch ($user_size != $size)"
> continue;
> fi
>
> - # make sure the provided offset is within the symbol's range
> - if [[ $offset -gt $sym_size ]]; then
> + # Make sure the provided offset is within the symbol's range:
> + if [[ $offset -gt $size ]]; then
> [[ $print_warnings = 1 ]] &&
> - echo "skipping $func address at $addr due to size mismatch ($offset > $sym_size)"
> + echo "skipping $func address at $addr due to size mismatch ($offset > $size)"
> continue
> fi
>
> - # separate multiple entries with a blank line
> - [[ $FIRST = 0 ]] && echo
> - FIRST=0
> + # In case of duplicates, separate multiple entries with a blank line:
> + [[ $first = 0 ]] && echo
> + first=0
>
> - # pass real address to addr2line
> - echo "$func+$offset/$sym_size:"
> - local file_lines=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
> - [[ -z $file_lines ]] && return
> + # Pass full address to addr2line:
> + echo "$func+$offset/$size:"
> + local output=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
> + [[ -z $output ]] && continue
>
> if [[ $LIST = 0 ]]; then
> - echo "$file_lines" | while read -r line
> + echo "$output" | while read -r line
> do
> echo $line
> done
> DONE=1;
> - return
> + continue
> fi
>
> - # show each line with context
> - echo "$file_lines" | while read -r line
> + # If --list was specified, show each line with context:
> + echo "$output" | while read -r line
> do
> echo
> echo $line
> @@ -189,7 +204,7 @@ __faddr2line() {
>
> DONE=1
>
> - done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> + done < <(${READELF} --symbols --wide $objfile | awk -v fn=$func '$4 == "FUNC" && $8 == fn')
> }
>
> [[ $# -lt 2 ]] && usage
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-01-20 5:14 ` Kaiwan N Billimoria
@ 2022-01-20 17:17 ` Josh Poimboeuf
2022-01-20 20:05 ` Kaiwan N Billimoria
0 siblings, 1 reply; 15+ messages in thread
From: Josh Poimboeuf @ 2022-01-20 17:17 UTC (permalink / raw)
To: Kaiwan N Billimoria; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
[-- Attachment #1: Type: text/plain, Size: 8639 bytes --]
On Thu, Jan 20, 2022 at 10:44:44AM +0530, Kaiwan N Billimoria wrote:
> On Thu, Jan 20, 2022 at 5:13 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> >
> > On Wed, Jan 19, 2022 at 01:21:29PM -0800, Josh Poimboeuf wrote:
> > > > $ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
> > > > 0000000000000000 r __func__.24215
> > > > 0000000000000000 r __param_bug_in_workq
> > > > 0000000000000000 D __this_module
> > > > 0000000000000000 r _note_7
> > > > 0000000000000000 T cleanup_module
> > > > 0000000000000000 t do_the_work
> > > > 0000000000000000 t do_the_work.cold
> > > > 0000000000000000 b gctx
> > > > 0000000000000000 T init_module
> > > > 0000000000000000 t try_oops_exit
> > > > 0000000000000000 t try_oops_init
> > > > 0000000000000008 b t1
> > > > $
> > > >
> > > > BTW, here's the code:
> > > > https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2
> > >
> > > Ok, it looks like the symbols aren't sorted like the code expects. I
> > > need to do a more robust fix.
> >
> > Ok, please try this instead. This takes a much more robust approach to
> > the function size calculation, using readelf to confine the symbol
> > search to the section matching the original symbol.
> >
> > This actually has multiple fixes and cleanups, so it'll eventually be
> > split up into a patch set.
>
> Great. Am facing a few issues though:
>
> 1. Not sure if it's me but am having issues applying your patches; it
> seems to fail:
>
> $ patch -p1 --dry-run < ./fa1.patch
> checking file scripts/faddr2line
> Hunk #1 FAILED at 97.
> Hunk #2 FAILED at 110.
> Hunk #3 FAILED at 189.
> 3 out of 3 hunks FAILED
> $
Hm, did you happen to have the older patch already applied? This should
be applied instead of that one. Though it's definitely possible I
messed the patch up somehow.
When saving the patch, make sure to preserve the original whitespace.
I'm adding the new patch here as an attachment, if that helps.
> Against which ver are you generating the patch?
> (Am using 5.10.60; i noticed the last commit in mainline was f5f67cc
> in Nov 2018, so vanilla 5.10.60 should be fine, no?).
It was against a newer kernel, but yes, applying it against 5.10.60
should also work.
>
> 2. I manually applied your patch :) This is the output I then get:
>
> $ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko
> do_the_work+0x16f/0x194 awk: not an option: --bignum bad symbol size:
> addr: 0x0000000000000000 $
>
> (Not sure if this helps but am running on Ubuntu 20.04; my awk is mawk
> v 1.3.4.20200120-2).
Sorry, I didn't realize mawk was still a thing :-)
Try this one?
----
diff --git a/scripts/faddr2line b/scripts/faddr2line
index 6c6439f69a72..01f28e52f3fc 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -97,86 +97,127 @@ __faddr2line() {
local dir_prefix=$3
local print_warnings=$4
- local func=${func_addr%+*}
+ local first=1
+ local sym_name=${func_addr%+*}
local offset=${func_addr#*+}
offset=${offset%/*}
- local size=
- [[ $func_addr =~ "/" ]] && size=${func_addr#*/}
+ local user_size=
+ [[ $func_addr =~ "/" ]] && user_size=${func_addr#*/}
- if [[ -z $func ]] || [[ -z $offset ]] || [[ $func = $func_addr ]]; then
+ if [[ -z $sym_name ]] || [[ -z $offset ]] || [[ $sym_name = $func_addr ]]; then
warn "bad func+offset $func_addr"
DONE=1
return
fi
# Go through each of the object's symbols which match the func name.
- # In rare cases there might be duplicates.
- file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
- while read symbol; do
- local fields=($symbol)
- local sym_base=0x${fields[0]}
- local sym_type=${fields[1]}
- local sym_end=${fields[3]}
-
- # calculate the size
- local sym_size=$(($sym_end - $sym_base))
- if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
- warn "bad symbol size: base: $sym_base end: $sym_end"
+ # In rare cases there might be duplicates, in which case we print all
+ # matches.
+ while read line; do
+ local fields=($line)
+ local sym_addr=0x${fields[1]}
+ local sym_size=${fields[2]}
+ local sym_sec=${fields[6]}
+
+ # Get the section size:
+ local sec_size=$(${READELF} --section-headers --wide $objfile |
+ sed 's/\[ /\[/' |
+ awk -v sec=$sym_sec '$1 == "[" sec "]" { print "0x" $6; exit }')
+
+ if [[ -z $sec_size ]]; then
+ warn "bad section size: section: $sym_sec"
DONE=1
return
fi
- sym_size=0x$(printf %x $sym_size)
- # calculate the address
- local addr=$(($sym_base + $offset))
- if [[ -z $addr ]] || [[ $addr = 0 ]]; then
- warn "bad address: $sym_base + $offset"
+ # Calculate the symbol size:
+ #
+ # We can't use the ELF size, because kallsyms also includes the
+ # padding bytes in its size calculation. For kallsyms, the
+ # size calculation is the distance between the symbol and the
+ # next symbol in a sorted list.
+ local size
+ local next_sym_addr
+ local found=0
+ while read line; do
+ local fields=($line)
+ next_sym_addr=0x${fields[1]}
+ local next_sym_size=${fields[2]}
+ local next_sym_name=${fields[7]:-}
+
+ if [[ $next_sym_addr = $sym_addr ]] &&
+ [[ $next_sym_size = $sym_size ]] &&
+ [[ $next_sym_name = $sym_name ]]; then
+ found=1
+ continue
+ fi
+
+ if [[ $found = 1 ]]; then
+ size=$(($next_sym_addr - $sym_addr))
+ [[ $size -lt $sym_size ]] && continue;
+ found=2
+ break
+ fi
+ done < <(${READELF} --symbols --wide $objfile | awk -v sec=$sym_sec '$7 == sec' | sort --key=2)
+
+ if [[ $found = 0 ]]; then
+ warn "can't find symbol: sym_name: $sym_name sym_sec=$sym_sec sym_addr: $sym_addr sym_size: $sym_size"
DONE=1
return
fi
- addr=0x$(printf %x $addr)
- # weed out non-function symbols
- if [[ $sym_type != t ]] && [[ $sym_type != T ]]; then
- [[ $print_warnings = 1 ]] &&
- echo "skipping $func address at $addr due to non-function symbol of type '$sym_type'"
- continue
+ [[ $found = 1 ]] && size=$(($sec_size - $sym_addr))
+
+ if [[ -z $size ]] || [[ $size -le 0 ]]; then
+ warn "bad symbol size: sym_addr: $sym_addr next_sym_addr: $next_sym_addr"
+ DONE=1
+ return
fi
+ size=0x$(printf %x $size)
- # if the user provided a size, make sure it matches the symbol's size
- if [[ -n $size ]] && [[ $size -ne $sym_size ]]; then
+ # Calculate the specified address:
+ local addr=$(($sym_addr + $offset))
+ if [[ -z $addr ]] || [[ $addr = 0 ]]; then
+ warn "bad address: $sym_addr + $offset"
+ DONE=1
+ return
+ fi
+ addr=0x$(printf %x $addr)
+
+ # If the user provided a size, make sure it matches the symbol's size:
+ if [[ -n $user_size ]] && [[ $user_size -ne $size ]]; then
[[ $print_warnings = 1 ]] &&
- echo "skipping $func address at $addr due to size mismatch ($size != $sym_size)"
+ echo "skipping $sym_name address at $addr due to size mismatch ($user_size != $size)"
continue;
fi
- # make sure the provided offset is within the symbol's range
- if [[ $offset -gt $sym_size ]]; then
+ # Make sure the provided offset is within the symbol's range:
+ if [[ $offset -gt $size ]]; then
[[ $print_warnings = 1 ]] &&
- echo "skipping $func address at $addr due to size mismatch ($offset > $sym_size)"
+ echo "skipping $sym_name address at $addr due to size mismatch ($offset > $size)"
continue
fi
- # separate multiple entries with a blank line
- [[ $FIRST = 0 ]] && echo
- FIRST=0
+ # In case of duplicates, separate multiple entries with a blank line:
+ [[ $first = 0 ]] && echo
+ first=0
- # pass real address to addr2line
- echo "$func+$offset/$sym_size:"
- local file_lines=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
- [[ -z $file_lines ]] && return
+ # Pass full address to addr2line:
+ echo "$sym_name+$offset/$size:"
+ local output=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
+ [[ -z $output ]] && continue
if [[ $LIST = 0 ]]; then
- echo "$file_lines" | while read -r line
+ echo "$output" | while read -r line
do
echo $line
done
DONE=1;
- return
+ continue
fi
- # show each line with context
- echo "$file_lines" | while read -r line
+ # If --list was specified, show each line with context:
+ echo "$output" | while read -r line
do
echo
echo $line
@@ -189,7 +230,7 @@ __faddr2line() {
DONE=1
- done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
+ done < <(${READELF} --symbols --wide $objfile | awk -v fn=$sym_name '$4 == "FUNC" && $8 == fn')
}
[[ $# -lt 2 ]] && usage
[-- Attachment #2: faddr2line-fix.patch --]
[-- Type: text/plain, Size: 6025 bytes --]
diff --git a/scripts/faddr2line b/scripts/faddr2line
index 6c6439f69a72..01f28e52f3fc 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -97,86 +97,127 @@ __faddr2line() {
local dir_prefix=$3
local print_warnings=$4
- local func=${func_addr%+*}
+ local first=1
+ local sym_name=${func_addr%+*}
local offset=${func_addr#*+}
offset=${offset%/*}
- local size=
- [[ $func_addr =~ "/" ]] && size=${func_addr#*/}
+ local user_size=
+ [[ $func_addr =~ "/" ]] && user_size=${func_addr#*/}
- if [[ -z $func ]] || [[ -z $offset ]] || [[ $func = $func_addr ]]; then
+ if [[ -z $sym_name ]] || [[ -z $offset ]] || [[ $sym_name = $func_addr ]]; then
warn "bad func+offset $func_addr"
DONE=1
return
fi
# Go through each of the object's symbols which match the func name.
- # In rare cases there might be duplicates.
- file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
- while read symbol; do
- local fields=($symbol)
- local sym_base=0x${fields[0]}
- local sym_type=${fields[1]}
- local sym_end=${fields[3]}
-
- # calculate the size
- local sym_size=$(($sym_end - $sym_base))
- if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
- warn "bad symbol size: base: $sym_base end: $sym_end"
+ # In rare cases there might be duplicates, in which case we print all
+ # matches.
+ while read line; do
+ local fields=($line)
+ local sym_addr=0x${fields[1]}
+ local sym_size=${fields[2]}
+ local sym_sec=${fields[6]}
+
+ # Get the section size:
+ local sec_size=$(${READELF} --section-headers --wide $objfile |
+ sed 's/\[ /\[/' |
+ awk -v sec=$sym_sec '$1 == "[" sec "]" { print "0x" $6; exit }')
+
+ if [[ -z $sec_size ]]; then
+ warn "bad section size: section: $sym_sec"
DONE=1
return
fi
- sym_size=0x$(printf %x $sym_size)
- # calculate the address
- local addr=$(($sym_base + $offset))
- if [[ -z $addr ]] || [[ $addr = 0 ]]; then
- warn "bad address: $sym_base + $offset"
+ # Calculate the symbol size:
+ #
+ # We can't use the ELF size, because kallsyms also includes the
+ # padding bytes in its size calculation. For kallsyms, the
+ # size calculation is the distance between the symbol and the
+ # next symbol in a sorted list.
+ local size
+ local next_sym_addr
+ local found=0
+ while read line; do
+ local fields=($line)
+ next_sym_addr=0x${fields[1]}
+ local next_sym_size=${fields[2]}
+ local next_sym_name=${fields[7]:-}
+
+ if [[ $next_sym_addr = $sym_addr ]] &&
+ [[ $next_sym_size = $sym_size ]] &&
+ [[ $next_sym_name = $sym_name ]]; then
+ found=1
+ continue
+ fi
+
+ if [[ $found = 1 ]]; then
+ size=$(($next_sym_addr - $sym_addr))
+ [[ $size -lt $sym_size ]] && continue;
+ found=2
+ break
+ fi
+ done < <(${READELF} --symbols --wide $objfile | awk -v sec=$sym_sec '$7 == sec' | sort --key=2)
+
+ if [[ $found = 0 ]]; then
+ warn "can't find symbol: sym_name: $sym_name sym_sec=$sym_sec sym_addr: $sym_addr sym_size: $sym_size"
DONE=1
return
fi
- addr=0x$(printf %x $addr)
- # weed out non-function symbols
- if [[ $sym_type != t ]] && [[ $sym_type != T ]]; then
- [[ $print_warnings = 1 ]] &&
- echo "skipping $func address at $addr due to non-function symbol of type '$sym_type'"
- continue
+ [[ $found = 1 ]] && size=$(($sec_size - $sym_addr))
+
+ if [[ -z $size ]] || [[ $size -le 0 ]]; then
+ warn "bad symbol size: sym_addr: $sym_addr next_sym_addr: $next_sym_addr"
+ DONE=1
+ return
fi
+ size=0x$(printf %x $size)
- # if the user provided a size, make sure it matches the symbol's size
- if [[ -n $size ]] && [[ $size -ne $sym_size ]]; then
+ # Calculate the specified address:
+ local addr=$(($sym_addr + $offset))
+ if [[ -z $addr ]] || [[ $addr = 0 ]]; then
+ warn "bad address: $sym_addr + $offset"
+ DONE=1
+ return
+ fi
+ addr=0x$(printf %x $addr)
+
+ # If the user provided a size, make sure it matches the symbol's size:
+ if [[ -n $user_size ]] && [[ $user_size -ne $size ]]; then
[[ $print_warnings = 1 ]] &&
- echo "skipping $func address at $addr due to size mismatch ($size != $sym_size)"
+ echo "skipping $sym_name address at $addr due to size mismatch ($user_size != $size)"
continue;
fi
- # make sure the provided offset is within the symbol's range
- if [[ $offset -gt $sym_size ]]; then
+ # Make sure the provided offset is within the symbol's range:
+ if [[ $offset -gt $size ]]; then
[[ $print_warnings = 1 ]] &&
- echo "skipping $func address at $addr due to size mismatch ($offset > $sym_size)"
+ echo "skipping $sym_name address at $addr due to size mismatch ($offset > $size)"
continue
fi
- # separate multiple entries with a blank line
- [[ $FIRST = 0 ]] && echo
- FIRST=0
+ # In case of duplicates, separate multiple entries with a blank line:
+ [[ $first = 0 ]] && echo
+ first=0
- # pass real address to addr2line
- echo "$func+$offset/$sym_size:"
- local file_lines=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
- [[ -z $file_lines ]] && return
+ # Pass full address to addr2line:
+ echo "$sym_name+$offset/$size:"
+ local output=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
+ [[ -z $output ]] && continue
if [[ $LIST = 0 ]]; then
- echo "$file_lines" | while read -r line
+ echo "$output" | while read -r line
do
echo $line
done
DONE=1;
- return
+ continue
fi
- # show each line with context
- echo "$file_lines" | while read -r line
+ # If --list was specified, show each line with context:
+ echo "$output" | while read -r line
do
echo
echo $line
@@ -189,7 +230,7 @@ __faddr2line() {
DONE=1
- done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
+ done < <(${READELF} --symbols --wide $objfile | awk -v fn=$sym_name '$4 == "FUNC" && $8 == fn')
}
[[ $# -lt 2 ]] && usage
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-01-20 17:17 ` Josh Poimboeuf
@ 2022-01-20 20:05 ` Kaiwan N Billimoria
2022-01-27 11:24 ` Kaiwan N Billimoria
2022-02-06 20:28 ` Josh Poimboeuf
0 siblings, 2 replies; 15+ messages in thread
From: Kaiwan N Billimoria @ 2022-01-20 20:05 UTC (permalink / raw)
To: Josh Poimboeuf; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
On Thu, Jan 20, 2022 at 10:48 PM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
>
> On Thu, Jan 20, 2022 at 10:44:44AM +0530, Kaiwan N Billimoria wrote:
> > On Thu, Jan 20, 2022 at 5:13 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> > >
> > > On Wed, Jan 19, 2022 at 01:21:29PM -0800, Josh Poimboeuf wrote:
> > > > > $ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
> > > > > 0000000000000000 r __func__.24215
> > > > > 0000000000000000 r __param_bug_in_workq
> > > > > 0000000000000000 D __this_module
> > > > > 0000000000000000 r _note_7
> > > > > 0000000000000000 T cleanup_module
> > > > > 0000000000000000 t do_the_work
> > > > > 0000000000000000 t do_the_work.cold
> > > > > 0000000000000000 b gctx
> > > > > 0000000000000000 T init_module
> > > > > 0000000000000000 t try_oops_exit
> > > > > 0000000000000000 t try_oops_init
> > > > > 0000000000000008 b t1
> > > > > $
> > > > >
> > > > > BTW, here's the code:
> > > > > https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2
> > > >
> > > > Ok, it looks like the symbols aren't sorted like the code expects. I
> > > > need to do a more robust fix.
> > >
> > > Ok, please try this instead. This takes a much more robust approach to
> > > the function size calculation, using readelf to confine the symbol
> > > search to the section matching the original symbol.
> > >
> > > This actually has multiple fixes and cleanups, so it'll eventually be
> > > split up into a patch set.
> >
> > Great. Am facing a few issues though:
> >
> > 1. Not sure if it's me but am having issues applying your patches; it
> > seems to fail:
> >
> > $ patch -p1 --dry-run < ./fa1.patch
> > checking file scripts/faddr2line
> > Hunk #1 FAILED at 97.
> > Hunk #2 FAILED at 110.
> > Hunk #3 FAILED at 189.
> > 3 out of 3 hunks FAILED
> > $
>
> Hm, did you happen to have the older patch already applied? This should
> be applied instead of that one. Though it's definitely possible I
> messed the patch up somehow.
>
> When saving the patch, make sure to preserve the original whitespace.
>
> I'm adding the new patch here as an attachment, if that helps.
That helped! I think my stubbornly using Gmail perhaps caused issues
when i copy-pasted the earlier patches... thanks.
>
> > Against which ver are you generating the patch?
> > (Am using 5.10.60; i noticed the last commit in mainline was f5f67cc
> > in Nov 2018, so vanilla 5.10.60 should be fine, no?).
>
> It was against a newer kernel, but yes, applying it against 5.10.60
> should also work.
Cool.
>...
> Sorry, I didn't realize mawk was still a thing :-)
>
> Try this one?
Definitely better !
This is the o/p i now get:
$ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko
do_the_work.cold+0x68/0x126
do_the_work.cold+0x68/0x126:
delay_sec at <...>/oops_tryv2/../../convenient.h:279
$
The source file+line-no doesn't show up here though... (as it's a foo.cold()?).
(I'm unsure why the line delay_sec at <...>convenient.h:279 shows up
; i don't call this delay_sec() func directly... anyway, that's my
problem.)
Thanks.
>
> ----
>
> diff --git a/scripts/faddr2line b/scripts/faddr2line
> index 6c6439f69a72..01f28e52f3fc 100755
> --- a/scripts/faddr2line
> +++ b/scripts/faddr2line
> @@ -97,86 +97,127 @@ __faddr2line() {
> local dir_prefix=$3
> local print_warnings=$4
>
> - local func=${func_addr%+*}
> + local first=1
> + local sym_name=${func_addr%+*}
> local offset=${func_addr#*+}
> offset=${offset%/*}
> - local size=
> - [[ $func_addr =~ "/" ]] && size=${func_addr#*/}
> + local user_size=
> + [[ $func_addr =~ "/" ]] && user_size=${func_addr#*/}
>
> - if [[ -z $func ]] || [[ -z $offset ]] || [[ $func = $func_addr ]]; then
> + if [[ -z $sym_name ]] || [[ -z $offset ]] || [[ $sym_name = $func_addr ]]; then
> warn "bad func+offset $func_addr"
> DONE=1
> return
> fi
>
> # Go through each of the object's symbols which match the func name.
> - # In rare cases there might be duplicates.
> - file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
> - while read symbol; do
> - local fields=($symbol)
> - local sym_base=0x${fields[0]}
> - local sym_type=${fields[1]}
> - local sym_end=${fields[3]}
> -
> - # calculate the size
> - local sym_size=$(($sym_end - $sym_base))
> - if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
> - warn "bad symbol size: base: $sym_base end: $sym_end"
> + # In rare cases there might be duplicates, in which case we print all
> + # matches.
> + while read line; do
> + local fields=($line)
> + local sym_addr=0x${fields[1]}
> + local sym_size=${fields[2]}
> + local sym_sec=${fields[6]}
> +
> + # Get the section size:
> + local sec_size=$(${READELF} --section-headers --wide $objfile |
> + sed 's/\[ /\[/' |
> + awk -v sec=$sym_sec '$1 == "[" sec "]" { print "0x" $6; exit }')
> +
> + if [[ -z $sec_size ]]; then
> + warn "bad section size: section: $sym_sec"
> DONE=1
> return
> fi
> - sym_size=0x$(printf %x $sym_size)
>
> - # calculate the address
> - local addr=$(($sym_base + $offset))
> - if [[ -z $addr ]] || [[ $addr = 0 ]]; then
> - warn "bad address: $sym_base + $offset"
> + # Calculate the symbol size:
> + #
> + # We can't use the ELF size, because kallsyms also includes the
> + # padding bytes in its size calculation. For kallsyms, the
> + # size calculation is the distance between the symbol and the
> + # next symbol in a sorted list.
> + local size
> + local next_sym_addr
> + local found=0
> + while read line; do
> + local fields=($line)
> + next_sym_addr=0x${fields[1]}
> + local next_sym_size=${fields[2]}
> + local next_sym_name=${fields[7]:-}
> +
> + if [[ $next_sym_addr = $sym_addr ]] &&
> + [[ $next_sym_size = $sym_size ]] &&
> + [[ $next_sym_name = $sym_name ]]; then
> + found=1
> + continue
> + fi
> +
> + if [[ $found = 1 ]]; then
> + size=$(($next_sym_addr - $sym_addr))
> + [[ $size -lt $sym_size ]] && continue;
> + found=2
> + break
> + fi
> + done < <(${READELF} --symbols --wide $objfile | awk -v sec=$sym_sec '$7 == sec' | sort --key=2)
> +
> + if [[ $found = 0 ]]; then
> + warn "can't find symbol: sym_name: $sym_name sym_sec=$sym_sec sym_addr: $sym_addr sym_size: $sym_size"
> DONE=1
> return
> fi
> - addr=0x$(printf %x $addr)
>
> - # weed out non-function symbols
> - if [[ $sym_type != t ]] && [[ $sym_type != T ]]; then
> - [[ $print_warnings = 1 ]] &&
> - echo "skipping $func address at $addr due to non-function symbol of type '$sym_type'"
> - continue
> + [[ $found = 1 ]] && size=$(($sec_size - $sym_addr))
> +
> + if [[ -z $size ]] || [[ $size -le 0 ]]; then
> + warn "bad symbol size: sym_addr: $sym_addr next_sym_addr: $next_sym_addr"
> + DONE=1
> + return
> fi
> + size=0x$(printf %x $size)
>
> - # if the user provided a size, make sure it matches the symbol's size
> - if [[ -n $size ]] && [[ $size -ne $sym_size ]]; then
> + # Calculate the specified address:
> + local addr=$(($sym_addr + $offset))
> + if [[ -z $addr ]] || [[ $addr = 0 ]]; then
> + warn "bad address: $sym_addr + $offset"
> + DONE=1
> + return
> + fi
> + addr=0x$(printf %x $addr)
> +
> + # If the user provided a size, make sure it matches the symbol's size:
> + if [[ -n $user_size ]] && [[ $user_size -ne $size ]]; then
> [[ $print_warnings = 1 ]] &&
> - echo "skipping $func address at $addr due to size mismatch ($size != $sym_size)"
> + echo "skipping $sym_name address at $addr due to size mismatch ($user_size != $size)"
> continue;
> fi
>
> - # make sure the provided offset is within the symbol's range
> - if [[ $offset -gt $sym_size ]]; then
> + # Make sure the provided offset is within the symbol's range:
> + if [[ $offset -gt $size ]]; then
> [[ $print_warnings = 1 ]] &&
> - echo "skipping $func address at $addr due to size mismatch ($offset > $sym_size)"
> + echo "skipping $sym_name address at $addr due to size mismatch ($offset > $size)"
> continue
> fi
>
> - # separate multiple entries with a blank line
> - [[ $FIRST = 0 ]] && echo
> - FIRST=0
> + # In case of duplicates, separate multiple entries with a blank line:
> + [[ $first = 0 ]] && echo
> + first=0
>
> - # pass real address to addr2line
> - echo "$func+$offset/$sym_size:"
> - local file_lines=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
> - [[ -z $file_lines ]] && return
> + # Pass full address to addr2line:
> + echo "$sym_name+$offset/$size:"
> + local output=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
> + [[ -z $output ]] && continue
>
> if [[ $LIST = 0 ]]; then
> - echo "$file_lines" | while read -r line
> + echo "$output" | while read -r line
> do
> echo $line
> done
> DONE=1;
> - return
> + continue
> fi
>
> - # show each line with context
> - echo "$file_lines" | while read -r line
> + # If --list was specified, show each line with context:
> + echo "$output" | while read -r line
> do
> echo
> echo $line
> @@ -189,7 +230,7 @@ __faddr2line() {
>
> DONE=1
>
> - done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> + done < <(${READELF} --symbols --wide $objfile | awk -v fn=$sym_name '$4 == "FUNC" && $8 == fn')
> }
>
> [[ $# -lt 2 ]] && usage
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-01-20 20:05 ` Kaiwan N Billimoria
@ 2022-01-27 11:24 ` Kaiwan N Billimoria
2022-02-06 20:28 ` Josh Poimboeuf
1 sibling, 0 replies; 15+ messages in thread
From: Kaiwan N Billimoria @ 2022-01-27 11:24 UTC (permalink / raw)
To: Josh Poimboeuf; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
Hi Josh,
<gentle ping>
I understand you must be busy, just checking...
any update on the faddr2line script?
Thanks,
Kaiwan.
On Fri, Jan 21, 2022 at 1:35 AM Kaiwan N Billimoria
<kaiwan.billimoria@gmail.com> wrote:
>
> On Thu, Jan 20, 2022 at 10:48 PM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> >
> > On Thu, Jan 20, 2022 at 10:44:44AM +0530, Kaiwan N Billimoria wrote:
> > > On Thu, Jan 20, 2022 at 5:13 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> > > >
> > > > On Wed, Jan 19, 2022 at 01:21:29PM -0800, Josh Poimboeuf wrote:
> > > > > > $ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
> > > > > > 0000000000000000 r __func__.24215
> > > > > > 0000000000000000 r __param_bug_in_workq
> > > > > > 0000000000000000 D __this_module
> > > > > > 0000000000000000 r _note_7
> > > > > > 0000000000000000 T cleanup_module
> > > > > > 0000000000000000 t do_the_work
> > > > > > 0000000000000000 t do_the_work.cold
> > > > > > 0000000000000000 b gctx
> > > > > > 0000000000000000 T init_module
> > > > > > 0000000000000000 t try_oops_exit
> > > > > > 0000000000000000 t try_oops_init
> > > > > > 0000000000000008 b t1
> > > > > > $
> > > > > >
> > > > > > BTW, here's the code:
> > > > > > https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2
> > > > >
> > > > > Ok, it looks like the symbols aren't sorted like the code expects. I
> > > > > need to do a more robust fix.
> > > >
> > > > Ok, please try this instead. This takes a much more robust approach to
> > > > the function size calculation, using readelf to confine the symbol
> > > > search to the section matching the original symbol.
> > > >
> > > > This actually has multiple fixes and cleanups, so it'll eventually be
> > > > split up into a patch set.
> > >
> > > Great. Am facing a few issues though:
> > >
> > > 1. Not sure if it's me but am having issues applying your patches; it
> > > seems to fail:
> > >
> > > $ patch -p1 --dry-run < ./fa1.patch
> > > checking file scripts/faddr2line
> > > Hunk #1 FAILED at 97.
> > > Hunk #2 FAILED at 110.
> > > Hunk #3 FAILED at 189.
> > > 3 out of 3 hunks FAILED
> > > $
> >
> > Hm, did you happen to have the older patch already applied? This should
> > be applied instead of that one. Though it's definitely possible I
> > messed the patch up somehow.
> >
> > When saving the patch, make sure to preserve the original whitespace.
> >
> > I'm adding the new patch here as an attachment, if that helps.
>
> That helped! I think my stubbornly using Gmail perhaps caused issues
> when i copy-pasted the earlier patches... thanks.
> >
> > > Against which ver are you generating the patch?
> > > (Am using 5.10.60; i noticed the last commit in mainline was f5f67cc
> > > in Nov 2018, so vanilla 5.10.60 should be fine, no?).
> >
> > It was against a newer kernel, but yes, applying it against 5.10.60
> > should also work.
> Cool.
> >...
> > Sorry, I didn't realize mawk was still a thing :-)
> >
> > Try this one?
>
> Definitely better !
> This is the o/p i now get:
>
> $ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko
> do_the_work.cold+0x68/0x126
> do_the_work.cold+0x68/0x126:
> delay_sec at <...>/oops_tryv2/../../convenient.h:279
> $
>
> The source file+line-no doesn't show up here though... (as it's a foo.cold()?).
> (I'm unsure why the line delay_sec at <...>convenient.h:279 shows up
> ; i don't call this delay_sec() func directly... anyway, that's my
> problem.)
>
> Thanks.
> >
> > ----
> >
> > diff --git a/scripts/faddr2line b/scripts/faddr2line
> > index 6c6439f69a72..01f28e52f3fc 100755
> > --- a/scripts/faddr2line
> > +++ b/scripts/faddr2line
> > @@ -97,86 +97,127 @@ __faddr2line() {
> > local dir_prefix=$3
> > local print_warnings=$4
> >
> > - local func=${func_addr%+*}
> > + local first=1
> > + local sym_name=${func_addr%+*}
> > local offset=${func_addr#*+}
> > offset=${offset%/*}
> > - local size=
> > - [[ $func_addr =~ "/" ]] && size=${func_addr#*/}
> > + local user_size=
> > + [[ $func_addr =~ "/" ]] && user_size=${func_addr#*/}
> >
> > - if [[ -z $func ]] || [[ -z $offset ]] || [[ $func = $func_addr ]]; then
> > + if [[ -z $sym_name ]] || [[ -z $offset ]] || [[ $sym_name = $func_addr ]]; then
> > warn "bad func+offset $func_addr"
> > DONE=1
> > return
> > fi
> >
> > # Go through each of the object's symbols which match the func name.
> > - # In rare cases there might be duplicates.
> > - file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
> > - while read symbol; do
> > - local fields=($symbol)
> > - local sym_base=0x${fields[0]}
> > - local sym_type=${fields[1]}
> > - local sym_end=${fields[3]}
> > -
> > - # calculate the size
> > - local sym_size=$(($sym_end - $sym_base))
> > - if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
> > - warn "bad symbol size: base: $sym_base end: $sym_end"
> > + # In rare cases there might be duplicates, in which case we print all
> > + # matches.
> > + while read line; do
> > + local fields=($line)
> > + local sym_addr=0x${fields[1]}
> > + local sym_size=${fields[2]}
> > + local sym_sec=${fields[6]}
> > +
> > + # Get the section size:
> > + local sec_size=$(${READELF} --section-headers --wide $objfile |
> > + sed 's/\[ /\[/' |
> > + awk -v sec=$sym_sec '$1 == "[" sec "]" { print "0x" $6; exit }')
> > +
> > + if [[ -z $sec_size ]]; then
> > + warn "bad section size: section: $sym_sec"
> > DONE=1
> > return
> > fi
> > - sym_size=0x$(printf %x $sym_size)
> >
> > - # calculate the address
> > - local addr=$(($sym_base + $offset))
> > - if [[ -z $addr ]] || [[ $addr = 0 ]]; then
> > - warn "bad address: $sym_base + $offset"
> > + # Calculate the symbol size:
> > + #
> > + # We can't use the ELF size, because kallsyms also includes the
> > + # padding bytes in its size calculation. For kallsyms, the
> > + # size calculation is the distance between the symbol and the
> > + # next symbol in a sorted list.
> > + local size
> > + local next_sym_addr
> > + local found=0
> > + while read line; do
> > + local fields=($line)
> > + next_sym_addr=0x${fields[1]}
> > + local next_sym_size=${fields[2]}
> > + local next_sym_name=${fields[7]:-}
> > +
> > + if [[ $next_sym_addr = $sym_addr ]] &&
> > + [[ $next_sym_size = $sym_size ]] &&
> > + [[ $next_sym_name = $sym_name ]]; then
> > + found=1
> > + continue
> > + fi
> > +
> > + if [[ $found = 1 ]]; then
> > + size=$(($next_sym_addr - $sym_addr))
> > + [[ $size -lt $sym_size ]] && continue;
> > + found=2
> > + break
> > + fi
> > + done < <(${READELF} --symbols --wide $objfile | awk -v sec=$sym_sec '$7 == sec' | sort --key=2)
> > +
> > + if [[ $found = 0 ]]; then
> > + warn "can't find symbol: sym_name: $sym_name sym_sec=$sym_sec sym_addr: $sym_addr sym_size: $sym_size"
> > DONE=1
> > return
> > fi
> > - addr=0x$(printf %x $addr)
> >
> > - # weed out non-function symbols
> > - if [[ $sym_type != t ]] && [[ $sym_type != T ]]; then
> > - [[ $print_warnings = 1 ]] &&
> > - echo "skipping $func address at $addr due to non-function symbol of type '$sym_type'"
> > - continue
> > + [[ $found = 1 ]] && size=$(($sec_size - $sym_addr))
> > +
> > + if [[ -z $size ]] || [[ $size -le 0 ]]; then
> > + warn "bad symbol size: sym_addr: $sym_addr next_sym_addr: $next_sym_addr"
> > + DONE=1
> > + return
> > fi
> > + size=0x$(printf %x $size)
> >
> > - # if the user provided a size, make sure it matches the symbol's size
> > - if [[ -n $size ]] && [[ $size -ne $sym_size ]]; then
> > + # Calculate the specified address:
> > + local addr=$(($sym_addr + $offset))
> > + if [[ -z $addr ]] || [[ $addr = 0 ]]; then
> > + warn "bad address: $sym_addr + $offset"
> > + DONE=1
> > + return
> > + fi
> > + addr=0x$(printf %x $addr)
> > +
> > + # If the user provided a size, make sure it matches the symbol's size:
> > + if [[ -n $user_size ]] && [[ $user_size -ne $size ]]; then
> > [[ $print_warnings = 1 ]] &&
> > - echo "skipping $func address at $addr due to size mismatch ($size != $sym_size)"
> > + echo "skipping $sym_name address at $addr due to size mismatch ($user_size != $size)"
> > continue;
> > fi
> >
> > - # make sure the provided offset is within the symbol's range
> > - if [[ $offset -gt $sym_size ]]; then
> > + # Make sure the provided offset is within the symbol's range:
> > + if [[ $offset -gt $size ]]; then
> > [[ $print_warnings = 1 ]] &&
> > - echo "skipping $func address at $addr due to size mismatch ($offset > $sym_size)"
> > + echo "skipping $sym_name address at $addr due to size mismatch ($offset > $size)"
> > continue
> > fi
> >
> > - # separate multiple entries with a blank line
> > - [[ $FIRST = 0 ]] && echo
> > - FIRST=0
> > + # In case of duplicates, separate multiple entries with a blank line:
> > + [[ $first = 0 ]] && echo
> > + first=0
> >
> > - # pass real address to addr2line
> > - echo "$func+$offset/$sym_size:"
> > - local file_lines=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
> > - [[ -z $file_lines ]] && return
> > + # Pass full address to addr2line:
> > + echo "$sym_name+$offset/$size:"
> > + local output=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
> > + [[ -z $output ]] && continue
> >
> > if [[ $LIST = 0 ]]; then
> > - echo "$file_lines" | while read -r line
> > + echo "$output" | while read -r line
> > do
> > echo $line
> > done
> > DONE=1;
> > - return
> > + continue
> > fi
> >
> > - # show each line with context
> > - echo "$file_lines" | while read -r line
> > + # If --list was specified, show each line with context:
> > + echo "$output" | while read -r line
> > do
> > echo
> > echo $line
> > @@ -189,7 +230,7 @@ __faddr2line() {
> >
> > DONE=1
> >
> > - done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> > + done < <(${READELF} --symbols --wide $objfile | awk -v fn=$sym_name '$4 == "FUNC" && $8 == fn')
> > }
> >
> > [[ $# -lt 2 ]] && usage
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-01-20 20:05 ` Kaiwan N Billimoria
2022-01-27 11:24 ` Kaiwan N Billimoria
@ 2022-02-06 20:28 ` Josh Poimboeuf
2022-02-07 2:23 ` Kaiwan N Billimoria
1 sibling, 1 reply; 15+ messages in thread
From: Josh Poimboeuf @ 2022-02-06 20:28 UTC (permalink / raw)
To: Kaiwan N Billimoria; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
On Fri, Jan 21, 2022 at 01:35:13AM +0530, Kaiwan N Billimoria wrote:
> > Try this one?
>
> Definitely better !
> This is the o/p i now get:
>
> $ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko
> do_the_work.cold+0x68/0x126
> do_the_work.cold+0x68/0x126:
> delay_sec at <...>/oops_tryv2/../../convenient.h:279
> $
>
> The source file+line-no doesn't show up here though... (as it's a foo.cold()?).
> (I'm unsure why the line delay_sec at <...>convenient.h:279 shows up
> ; i don't call this delay_sec() func directly... anyway, that's my
> problem.)
Hm... do you get the same result with addr2line?
--
Josh
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-02-06 20:28 ` Josh Poimboeuf
@ 2022-02-07 2:23 ` Kaiwan N Billimoria
2022-02-11 3:31 ` Kaiwan N Billimoria
0 siblings, 1 reply; 15+ messages in thread
From: Kaiwan N Billimoria @ 2022-02-07 2:23 UTC (permalink / raw)
To: Josh Poimboeuf; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
On Mon, Feb 7, 2022 at 1:59 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
>
> On Fri, Jan 21, 2022 at 01:35:13AM +0530, Kaiwan N Billimoria wrote:
> > > Try this one?
> >
> > Definitely better !
> > This is the o/p i now get:
> >
> > $ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko
> > do_the_work.cold+0x68/0x126
> > do_the_work.cold+0x68/0x126:
> > delay_sec at <...>/oops_tryv2/../../convenient.h:279
> > $
> >
> > The source file+line-no doesn't show up here though... (as it's a foo.cold()?).
> > (I'm unsure why the line delay_sec at <...>convenient.h:279 shows up
> > ; i don't call this delay_sec() func directly... anyway, that's my
> > problem.)
>
> Hm... do you get the same result with addr2line?
addr2line works correctly.
And, so does the patched faddr2line, when i tried it out now on
another VM! Lemme check some more and get back, thanks.
- Kaiwan.
>
>
> --
> Josh
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Issue using faddr2line on kernel modules
2022-02-07 2:23 ` Kaiwan N Billimoria
@ 2022-02-11 3:31 ` Kaiwan N Billimoria
0 siblings, 0 replies; 15+ messages in thread
From: Kaiwan N Billimoria @ 2022-02-11 3:31 UTC (permalink / raw)
To: Josh Poimboeuf; +Cc: Linux Kernel Mailing List, Chi-Thanh Hoang
It works !
$ .../linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x124
do_the_work+0x124/0x15e:
do_the_work at .../oops_tryv2.c:62
$
Will you now upstream this patch?
Thanks!
On Mon, Feb 7, 2022 at 7:53 AM Kaiwan N Billimoria
<kaiwan.billimoria@gmail.com> wrote:
>
> On Mon, Feb 7, 2022 at 1:59 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> >
> > On Fri, Jan 21, 2022 at 01:35:13AM +0530, Kaiwan N Billimoria wrote:
> > > > Try this one?
> > >
> > > Definitely better !
> > > This is the o/p i now get:
> > >
> > > $ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko
> > > do_the_work.cold+0x68/0x126
> > > do_the_work.cold+0x68/0x126:
> > > delay_sec at <...>/oops_tryv2/../../convenient.h:279
> > > $
> > >
> > > The source file+line-no doesn't show up here though... (as it's a foo.cold()?).
> > > (I'm unsure why the line delay_sec at <...>convenient.h:279 shows up
> > > ; i don't call this delay_sec() func directly... anyway, that's my
> > > problem.)
> >
> > Hm... do you get the same result with addr2line?
>
> addr2line works correctly.
> And, so does the patched faddr2line, when i tried it out now on
> another VM! Lemme check some more and get back, thanks.
> - Kaiwan.
> >
> >
> > --
> > Josh
> >
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2022-02-11 3:32 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-17 4:57 Issue using faddr2line on kernel modules Kaiwan N Billimoria
2022-01-17 19:48 ` Josh Poimboeuf
2022-01-17 20:27 ` Josh Poimboeuf
2022-01-18 2:40 ` Kaiwan N Billimoria
2022-01-18 17:52 ` Josh Poimboeuf
2022-01-19 3:57 ` Kaiwan N Billimoria
2022-01-19 21:21 ` Josh Poimboeuf
2022-01-19 23:43 ` Josh Poimboeuf
2022-01-20 5:14 ` Kaiwan N Billimoria
2022-01-20 17:17 ` Josh Poimboeuf
2022-01-20 20:05 ` Kaiwan N Billimoria
2022-01-27 11:24 ` Kaiwan N Billimoria
2022-02-06 20:28 ` Josh Poimboeuf
2022-02-07 2:23 ` Kaiwan N Billimoria
2022-02-11 3:31 ` Kaiwan N Billimoria
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).