[PATCH 0/2] Expose kallsyms data in vmcoreinfo note

[PATCH 0/2] Expose kallsyms data in vmcoreinfo note

[Stephen Brennan]

The kernel can be configured to contain a lot of introspection or
debugging information built-in, such as ORC for unwinding stack traces,
BTF for type information, and of course kallsyms. Debuggers could use
this information to navigate a core dump or live system, but they need
to be able to find it.

This patch series adds the necessary symbols into vmcoreinfo, which
would allow a debugger to find and interpret the kallsyms table. Using
the kallsyms data, the debugger can then lookup any symbol, allowing it
to find ORC, BTF, or any other useful data.

This would allow a live kernel, or core dump, to be debugged without
any DWARF debuginfo. This is useful for many cases: the debuginfo may
not have been generated, or you may not want to deploy the large files
everywhere you need them.

I've demonstrated a proof of concept for this at LSF/MM+BPF during a
lighting talk. Using a work-in-progress branch of the drgn debugger, and
an extended set of BTF generated by a patched version of dwarves, I've
been able to open a core dump without any DWARF info and do basic tasks
such as enumerating slab caches, block devices, tasks, and doing
backtraces. I hope this series can be a first step toward a new
possibility of "DWARFless debugging".

Related discussion around the BTF side of this:
https://lore.kernel.org/bpf/586a6288-704a-f7a7-b256-e18a675927df at oracle.com/T/#u

Some work-in-progress branches using this feature:
https://github.com/brenns10/dwarves/tree/remove_percpu_restriction_1
https://github.com/brenns10/drgn/tree/kallsyms_plus_btf

Stephen Brennan (2):
  kallsyms: Move declarations to internal header
  vmcoreinfo: Include kallsyms symbols

 kernel/crash_core.c        | 14 ++++++++++++++
 kernel/kallsyms.c          | 23 +----------------------
 kernel/kallsyms_internal.h | 30 ++++++++++++++++++++++++++++++
 3 files changed, 45 insertions(+), 22 deletions(-)
 create mode 100644 kernel/kallsyms_internal.h

-- 
2.30.2

[PATCH 1/2] kallsyms: Move declarations to internal header

[Stephen Brennan]

To include kallsyms data in the vmcoreinfo note, we must make the symbol
declarations visible outside of kallsyms.c. Move these to a new internal
header file.

Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
---
 kernel/kallsyms.c          | 23 +----------------------
 kernel/kallsyms_internal.h | 30 ++++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 22 deletions(-)
 create mode 100644 kernel/kallsyms_internal.h

diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c
index 79f2eb617a62..42373ff69cac 100644
--- a/kernel/kallsyms.c
+++ b/kernel/kallsyms.c
@@ -30,28 +30,7 @@
 #include <linux/module.h>
 #include <linux/kernel.h>
 
-/*
- * These will be re-linked against their real values
- * during the second link stage.
- */
-extern const unsigned long kallsyms_addresses[] __weak;
-extern const int kallsyms_offsets[] __weak;
-extern const u8 kallsyms_names[] __weak;
-
-/*
- * Tell the compiler that the count isn't in the small data section if the arch
- * has one (eg: FRV).
- */
-extern const unsigned int kallsyms_num_syms
-__section(".rodata") __attribute__((weak));
-
-extern const unsigned long kallsyms_relative_base
-__section(".rodata") __attribute__((weak));
-
-extern const char kallsyms_token_table[] __weak;
-extern const u16 kallsyms_token_index[] __weak;
-
-extern const unsigned int kallsyms_markers[] __weak;
+#include "kallsyms_internal.h"
 
 /*
  * Expand a compressed symbol data into the resulting uncompressed string,
diff --git a/kernel/kallsyms_internal.h b/kernel/kallsyms_internal.h
new file mode 100644
index 000000000000..2d0c6f2f0243
--- /dev/null
+++ b/kernel/kallsyms_internal.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef LINUX_KALLSYMS_INTERNAL_H_
+#define LINUX_KALLSYMS_INTERNAL_H_
+
+#include <linux/types.h>
+
+/*
+ * These will be re-linked against their real values
+ * during the second link stage.
+ */
+extern const unsigned long kallsyms_addresses[] __weak;
+extern const int kallsyms_offsets[] __weak;
+extern const u8 kallsyms_names[] __weak;
+
+/*
+ * Tell the compiler that the count isn't in the small data section if the arch
+ * has one (eg: FRV).
+ */
+extern const unsigned int kallsyms_num_syms
+__section(".rodata") __attribute__((weak));
+
+extern const unsigned long kallsyms_relative_base
+__section(".rodata") __attribute__((weak));
+
+extern const char kallsyms_token_table[] __weak;
+extern const u16 kallsyms_token_index[] __weak;
+
+extern const unsigned int kallsyms_markers[] __weak;
+
+#endif // LINUX_KALLSYMS_INTERNAL_H_
-- 
2.30.2

[PATCH 2/2] vmcoreinfo: Include kallsyms symbols

[Stephen Brennan]

The internal kallsyms tables contain information which could be quite
useful to a debugging tool in the absence of other debuginfo. If
kallsyms is enabled, then a debugging tool could parse it and use it as
a fallback symbol table. Combined with BTF data, live & post-mortem
debuggers can support basic operations without needing a large DWARF
debuginfo file available. As many as five symbols are necessary to
properly parse kallsyms names and addresses.  Add these to the
vmcoreinfo note.

CONFIG_KALLSYMS_ABSOLUTE_PERCPU does impact the computation of symbol
addresses. However, a debugger can infer this configuration value by
comparing the address of _stext in the vmcoreinfo with the address
computed via kallsyms. So there's no need to include information about
this config value in the vmcoreinfo note.

To verify that we're still well below the maximum of 4096 bytes, I
created a script[1] to compute a rough upper bound on the possible size
of vmcoreinfo. On v5.18-rc7, the script reports 3106 bytes, and with
this patch, the maximum become 3370 bytes.

[1]: https://github.com/brenns10/kernel_stuff/blob/master/vmcoreinfosize/

Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
---
 kernel/crash_core.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/kernel/crash_core.c b/kernel/crash_core.c
index 256cf6db573c..b9c89b80856d 100644
--- a/kernel/crash_core.c
+++ b/kernel/crash_core.c
@@ -15,6 +15,8 @@
 
 #include <crypto/sha1.h>
 
+#include "kallsyms_internal.h"
+
 /* vmcoreinfo stuff */
 unsigned char *vmcoreinfo_data;
 size_t vmcoreinfo_size;
@@ -484,6 +486,18 @@ static int __init crash_save_vmcoreinfo_init(void)
 	VMCOREINFO_NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE);
 #endif
 
+#ifdef CONFIG_KALLSYMS
+	VMCOREINFO_SYMBOL(kallsyms_names);
+	VMCOREINFO_SYMBOL(kallsyms_token_table);
+	VMCOREINFO_SYMBOL(kallsyms_token_index);
+#ifdef CONFIG_KALLSYMS_BASE_RELATIVE
+	VMCOREINFO_SYMBOL(kallsyms_offsets);
+	VMCOREINFO_SYMBOL(kallsyms_relative_base);
+#else
+	VMCOREINFO_SYMBOL(kallsyms_addresses);
+#endif /* CONFIG_KALLSYMS_BASE_RELATIVE */
+#endif /* CONFIG_KALLSYMS */
+
 	arch_crash_save_vmcoreinfo();
 	update_vmcoreinfo_note();
 
-- 
2.30.2

[PATCH 0/2] Expose kallsyms data in vmcoreinfo note

[Baoquan He]

On 05/16/22 at 05:05pm, Stephen Brennan wrote:
> The kernel can be configured to contain a lot of introspection or
> debugging information built-in, such as ORC for unwinding stack traces,
> BTF for type information, and of course kallsyms. Debuggers could use
> this information to navigate a core dump or live system, but they need
> to be able to find it.
> 
> This patch series adds the necessary symbols into vmcoreinfo, which
> would allow a debugger to find and interpret the kallsyms table. Using
> the kallsyms data, the debugger can then lookup any symbol, allowing it
> to find ORC, BTF, or any other useful data.
> 
> This would allow a live kernel, or core dump, to be debugged without
> any DWARF debuginfo. This is useful for many cases: the debuginfo may
> not have been generated, or you may not want to deploy the large files
> everywhere you need them.
> 
> I've demonstrated a proof of concept for this at LSF/MM+BPF during a
> lighting talk. Using a work-in-progress branch of the drgn debugger, and
> an extended set of BTF generated by a patched version of dwarves, I've
> been able to open a core dump without any DWARF info and do basic tasks
> such as enumerating slab caches, block devices, tasks, and doing
> backtraces. I hope this series can be a first step toward a new
> possibility of "DWARFless debugging".

Thanks. Seems no reason to reject, even though I haven't tried drgn.
And hope it has no security issue, e.g info leakage, at least I don't
see it has. So,

Acked-by: Baoquan He <bhe@redhat.com>

[PATCH 0/2] Expose kallsyms data in vmcoreinfo note

[Stephen Brennan]

Baoquan He <bhe@redhat.com> writes:
> On 05/16/22 at 05:05pm, Stephen Brennan wrote:
>> The kernel can be configured to contain a lot of introspection or
>> debugging information built-in, such as ORC for unwinding stack traces,
>> BTF for type information, and of course kallsyms. Debuggers could use
>> this information to navigate a core dump or live system, but they need
>> to be able to find it.
>> 
>> This patch series adds the necessary symbols into vmcoreinfo, which
>> would allow a debugger to find and interpret the kallsyms table. Using
>> the kallsyms data, the debugger can then lookup any symbol, allowing it
>> to find ORC, BTF, or any other useful data.
>> 
>> This would allow a live kernel, or core dump, to be debugged without
>> any DWARF debuginfo. This is useful for many cases: the debuginfo may
>> not have been generated, or you may not want to deploy the large files
>> everywhere you need them.
>> 
>> I've demonstrated a proof of concept for this at LSF/MM+BPF during a
>> lighting talk. Using a work-in-progress branch of the drgn debugger, and
>> an extended set of BTF generated by a patched version of dwarves, I've
>> been able to open a core dump without any DWARF info and do basic tasks
>> such as enumerating slab caches, block devices, tasks, and doing
>> backtraces. I hope this series can be a first step toward a new
>> possibility of "DWARFless debugging".
>
> Thanks. Seems no reason to reject, even though I haven't tried drgn.
> And hope it has no security issue, e.g info leakage, at least I don't
> see it has. So,
>
> Acked-by: Baoquan He <bhe@redhat.com>

Thanks Baoquan! I don't believe we have any worries regarding security,
since the kallsyms data itself is already available to root via
/proc/kallsyms, and core dumps should already be treated as highly
sensitive data by anybody handling them.

Do you know which tree this patch will go through?

Thanks,
Stephen

[PATCH 0/2] Expose kallsyms data in vmcoreinfo note

[Baoquan He]

On 05/23/22 at 11:00am, Stephen Brennan wrote:
> Baoquan He <bhe@redhat.com> writes:
> > On 05/16/22 at 05:05pm, Stephen Brennan wrote:
> >> The kernel can be configured to contain a lot of introspection or
> >> debugging information built-in, such as ORC for unwinding stack traces,
> >> BTF for type information, and of course kallsyms. Debuggers could use
> >> this information to navigate a core dump or live system, but they need
> >> to be able to find it.
> >> 
> >> This patch series adds the necessary symbols into vmcoreinfo, which
> >> would allow a debugger to find and interpret the kallsyms table. Using
> >> the kallsyms data, the debugger can then lookup any symbol, allowing it
> >> to find ORC, BTF, or any other useful data.
> >> 
> >> This would allow a live kernel, or core dump, to be debugged without
> >> any DWARF debuginfo. This is useful for many cases: the debuginfo may
> >> not have been generated, or you may not want to deploy the large files
> >> everywhere you need them.
> >> 
> >> I've demonstrated a proof of concept for this at LSF/MM+BPF during a
> >> lighting talk. Using a work-in-progress branch of the drgn debugger, and
> >> an extended set of BTF generated by a patched version of dwarves, I've
> >> been able to open a core dump without any DWARF info and do basic tasks
> >> such as enumerating slab caches, block devices, tasks, and doing
> >> backtraces. I hope this series can be a first step toward a new
> >> possibility of "DWARFless debugging".
> >
> > Thanks. Seems no reason to reject, even though I haven't tried drgn.
> > And hope it has no security issue, e.g info leakage, at least I don't
> > see it has. So,
> >
> > Acked-by: Baoquan He <bhe@redhat.com>
> 
> Thanks Baoquan! I don't believe we have any worries regarding security,
> since the kallsyms data itself is already available to root via
> /proc/kallsyms, and core dumps should already be treated as highly
> sensitive data by anybody handling them.
> 
> Do you know which tree this patch will go through?

I would like to ask Andrew to help check and pick this if no concern.

Thanks
Baoquan

Re: [PATCH 0/2] Expose kallsyms data in vmcoreinfo note

[Stephen Brennan]

Hi Andrew,

Baoquan He <bhe@redhat.com> writes:
> On 05/23/22 at 11:00am, Stephen Brennan wrote:
>> Do you know which tree this patch will go through?
>
> I would like to ask Andrew to help check and pick this if no concern.

Just wanted to check if you would be willing to take a look at this
series and possibly queue it up. I can resend the series directly to you
if you need.

Thanks,
Stephen

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2] Expose kallsyms data in vmcoreinfo note

[Andrew Morton]

On Mon, 16 May 2022 17:05:06 -0700 Stephen Brennan <stephen.s.brennan@oracle.com> wrote:

> The kernel can be configured to contain a lot of introspection or
> debugging information built-in, such as ORC for unwinding stack traces,
> BTF for type information, and of course kallsyms. Debuggers could use
> this information to navigate a core dump or live system, but they need
> to be able to find it.
> 
> This patch series adds the necessary symbols into vmcoreinfo, which
> would allow a debugger to find and interpret the kallsyms table. Using
> the kallsyms data, the debugger can then lookup any symbol, allowing it
> to find ORC, BTF, or any other useful data.
> 
> This would allow a live kernel, or core dump, to be debugged without
> any DWARF debuginfo. This is useful for many cases: the debuginfo may
> not have been generated, or you may not want to deploy the large files
> everywhere you need them.

Am trying to understand the value of all of this.  Can you explain
further why carrying the dwarf info is problematic?  How problematic
are these large files?

> I've demonstrated a proof of concept for this at LSF/MM+BPF during a
> lighting talk. Using a work-in-progress branch of the drgn debugger, and
> an extended set of BTF generated by a patched version of dwarves, I've
> been able to open a core dump without any DWARF info and do basic tasks
> such as enumerating slab caches, block devices, tasks, and doing
> backtraces. I hope this series can be a first step toward a new
> possibility of "DWARFless debugging".
> 
> Related discussion around the BTF side of this:
> https://lore.kernel.org/bpf/586a6288-704a-f7a7-b256-e18a675927df@oracle.com/T/#u
> 
> Some work-in-progress branches using this feature:
> https://github.com/brenns10/dwarves/tree/remove_percpu_restriction_1
> https://github.com/brenns10/drgn/tree/kallsyms_plus_btf

What's the story on using gdb with this?

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2] Expose kallsyms data in vmcoreinfo note

[Stephen Brennan]

Andrew Morton <akpm@linux-foundation.org> writes:
> On Mon, 16 May 2022 17:05:06 -0700 Stephen Brennan <stephen.s.brennan@oracle.com> wrote:
>
>> The kernel can be configured to contain a lot of introspection or
>> debugging information built-in, such as ORC for unwinding stack traces,
>> BTF for type information, and of course kallsyms. Debuggers could use
>> this information to navigate a core dump or live system, but they need
>> to be able to find it.
>> 
>> This patch series adds the necessary symbols into vmcoreinfo, which
>> would allow a debugger to find and interpret the kallsyms table. Using
>> the kallsyms data, the debugger can then lookup any symbol, allowing it
>> to find ORC, BTF, or any other useful data.
>> 
>> This would allow a live kernel, or core dump, to be debugged without
>> any DWARF debuginfo. This is useful for many cases: the debuginfo may
>> not have been generated, or you may not want to deploy the large files
>> everywhere you need them.
>
> Am trying to understand the value of all of this.  Can you explain
> further why carrying the dwarf info is problematic?  How problematic
> are these large files?

The primary value I see in this is that it unlocks data that's already
present in core dumps. Most distribution kernels already have
CONFIG_KALLSYMS_ALL, but you can't reliably find the symbol table in a
core dump without a change like this. Most distribution kernels already
have a fair amount of type information via BTF already built-in (though
as I mention below, there's a bit more that would be useful). So this
change can unlock that data, and make core dumps self-describing: you
can open one up at any time without extra debuginfo, and start reading
symbols and data values out of memory.

As far as the DWARF, I do find it unwieldy, we're talking about file
sizes ranging from 500 MiB to a 1 GiB or maybe a bit more. This isn't
huge for on-disk storage. But for things like OS images, it's pretty
hefty just for the off-chance that you need debugging.

A bigger concern than the size of the files is that, we frequently deal
with customers who, due to some sort of requirements, have a policy
against installing debuginfo packages on production machine. When they
inevitably ask folks like me to help debug the kernel on their "no
debuginfo" machines, I'd like to be able to just get my job done using
the info already there, rather than battling them over debuginfo.

There's also some corner cases. I've definitely taken a core dump on my
personal machine, only to find that the debuginfo had been deleted due
to a kernel package upgrade, so the core dump was useless. And some
distributions build kernels with no debuginfo packages. This can suit
all of those possibilities without undue overhead.

>> I've demonstrated a proof of concept for this at LSF/MM+BPF during a
>> lighting talk. Using a work-in-progress branch of the drgn debugger, and
>> an extended set of BTF generated by a patched version of dwarves, I've
>> been able to open a core dump without any DWARF info and do basic tasks
>> such as enumerating slab caches, block devices, tasks, and doing
>> backtraces. I hope this series can be a first step toward a new
>> possibility of "DWARFless debugging".
>> 
>> Related discussion around the BTF side of this:
>> https://lore.kernel.org/bpf/586a6288-704a-f7a7-b256-e18a675927df@oracle.com/T/#u
>> 
>> Some work-in-progress branches using this feature:
>> https://github.com/brenns10/dwarves/tree/remove_percpu_restriction_1
>> https://github.com/brenns10/drgn/tree/kallsyms_plus_btf
>
> What's the story on using gdb with this?

There is no story with GDB as of yet. I was already familiar with the
code of drgn when I started down this path, so that's what I used. Drgn
happens to have a very extensible type system which made it quite simple
to do. I'd love to see support for doing this with GDB, and might look
into the feasibility of it, but it's not on my roadmap right now.

That said, the drgn work is moving ahead nicely, much of this code has
moved from a "work-in-progress branch" to pull requests which are in
review. I have every intention of seeing those changes through.

Thanks for taking a look at this. I'm happy to answer more questions or
clarify anytihng you want.

Stephen

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2] Expose kallsyms data in vmcoreinfo note

[Andrew Morton]

On Mon, 13 Jun 2022 14:59:44 -0700 Stephen Brennan <stephen.s.brennan@oracle.com> wrote:

> >> Related discussion around the BTF side of this:
> >> https://lore.kernel.org/bpf/586a6288-704a-f7a7-b256-e18a675927df@oracle.com/T/#u
> >> 
> >> Some work-in-progress branches using this feature:
> >> https://github.com/brenns10/dwarves/tree/remove_percpu_restriction_1
> >> https://github.com/brenns10/drgn/tree/kallsyms_plus_btf
> >
> > What's the story on using gdb with this?
> 
> There is no story with GDB as of yet. I was already familiar with the
> code of drgn when I started down this path, so that's what I used. Drgn
> happens to have a very extensible type system which made it quite simple
> to do. I'd love to see support for doing this with GDB, and might look
> into the feasibility of it, but it's not on my roadmap right now.

Naive question - could some standalone tool take this kallsyms-based
info, combine it with a core image and create a minimally-dwarfified
file which any debugger can munch on?

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2] Expose kallsyms data in vmcoreinfo note

[Stephen Brennan]

Andrew Morton <akpm@linux-foundation.org> writes:
> On Mon, 13 Jun 2022 14:59:44 -0700 Stephen Brennan <stephen.s.brennan@oracle.com> wrote:
>> >> Related discussion around the BTF side of this:
>> >> https://lore.kernel.org/bpf/586a6288-704a-f7a7-b256-e18a675927df@oracle.com/T/#u
>> >> 
>> >> Some work-in-progress branches using this feature:
>> >> https://github.com/brenns10/dwarves/tree/remove_percpu_restriction_1
>> >> https://github.com/brenns10/drgn/tree/kallsyms_plus_btf
>> >
>> > What's the story on using gdb with this?
>> 
>> There is no story with GDB as of yet. I was already familiar with the
>> code of drgn when I started down this path, so that's what I used. Drgn
>> happens to have a very extensible type system which made it quite simple
>> to do. I'd love to see support for doing this with GDB, and might look
>> into the feasibility of it, but it's not on my roadmap right now.
>
> Naive question - could some standalone tool take this kallsyms-based
> info, combine it with a core image and create a minimally-dwarfified
> file which any debugger can munch on?

I'm not too familiar with the guts of DWARF, so honestly my guess may
not be any better than yours.

One thing that strikes me is that DWARF is typically included in an ELF
section, whereas x86_64 kernels are typically a bzImage, so I'm not
entirely sure how you'd get back an ELF file suitable to stick the DWARF
into. And from there, I really have no guess about the DWARF.

I will say that Compact Type Format (CTF) [1] is rather similar to BTF
in scope, and it already has support in the GNU Binutils. I'd imagine
that there's much more of a fighting chance of converting BTF to CTF and
getting GDB to use that instead.

[1] https://lwn.net/Articles/795384/

Stephen

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec