* [PATCH v6 00/13] Add build ID to stacktraces @ 2021-05-11 0:38 Stephen Boyd 2021-05-11 0:38 ` [PATCH v6 05/13] module: Add printk formats to add module " Stephen Boyd ` (2 more replies) 0 siblings, 3 replies; 11+ messages in thread From: Stephen Boyd @ 2021-05-11 0:38 UTC (permalink / raw) To: Andrew Morton, Petr Mladek Cc: linux-kernel, Alexei Starovoitov, Andy Shevchenko, Baoquan He, Borislav Petkov, Catalin Marinas, Dave Young, Evan Green, Hsin-Yi Wang, Ingo Molnar, Jessica Yu, Jiri Olsa, kexec, Konstantin Khlebnikov, linux-arm-kernel, linux-doc, Matthew Wilcox, Rasmus Villemoes, Sasha Levin, Sergey Senozhatsky, Steven Rostedt, Thomas Gleixner, Vivek Goyal, Will Deacon, x86, Christoph Hellwig, peter enderborg This series adds the kernel's build ID[1] to the stacktrace header printed in oops messages, warnings, etc. and the build ID for any module that appears in the stacktrace after the module name. The goal is to make the stacktrace more self-contained and descriptive by including the relevant build IDs in the kernel logs when something goes wrong. This can be used by post processing tools like script/decode_stacktrace.sh and kernel developers to easily locate the debug info associated with a kernel crash and line up what line and file things started falling apart at. To show how this can be used I've included a patch to decode_stacktrace.sh that downloads the debuginfo from a debuginfod server. This also includes some patches to make the buildid.c file use more const arguments and consolidate logic into buildid.c from kdump. These are left to the end as they were mostly cleanup patches. I don't know who exactly maintains this so I guess Andrew is the best option to merge all this code. Otherwise, Petr mentioned it could possibly go through the printk tree given that it touches mostly printk things. Here's an example lkdtm stacktrace on arm64. WARNING: CPU: 4 PID: 3255 at drivers/misc/lkdtm/bugs.c:83 lkdtm_WARNING+0x28/0x30 [lkdtm] Modules linked in: lkdtm rfcomm algif_hash algif_skcipher af_alg xt_cgroup uinput xt_MASQUERADE CPU: 4 PID: 3255 Comm: bash Not tainted 5.11 #3 aa23f7a1231c229de205662d5a9e0d4c580f19a1 Hardware name: Google Lazor (rev3+) with KB Backlight (DT) pstate: 00400009 (nzcv daif +PAN -UAO -TCO BTYPE=--) pc : lkdtm_WARNING+0x28/0x30 [lkdtm] lr : lkdtm_do_action+0x24/0x40 [lkdtm] sp : ffffffc0134fbca0 x29: ffffffc0134fbca0 x28: ffffff92d53ba240 x27: 0000000000000000 x26: 0000000000000000 x25: 0000000000000000 x24: ffffffe3622352c0 x23: 0000000000000020 x22: ffffffe362233366 x21: ffffffe3622352e0 x20: ffffffc0134fbde0 x19: 0000000000000008 x18: 0000000000000000 x17: ffffff929b6536fc x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000012 x13: ffffffe380ed892c x12: ffffffe381d05068 x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000001 x8 : ffffffe362237000 x7 : aaaaaaaaaaaaaaaa x6 : 0000000000000000 x5 : 0000000000000000 x4 : 0000000000000001 x3 : 0000000000000008 x2 : ffffff93fef25a70 x1 : ffffff93fef15788 x0 : ffffffe3622352e0 Call trace: lkdtm_WARNING+0x28/0x30 [lkdtm ed5019fdf5e53be37cb1ba7899292d7e143b259e] direct_entry+0x16c/0x1b4 [lkdtm ed5019fdf5e53be37cb1ba7899292d7e143b259e] full_proxy_write+0x74/0xa4 vfs_write+0xec/0x2e8 ksys_write+0x84/0xf0 __arm64_sys_write+0x24/0x30 el0_svc_common+0xf4/0x1c0 do_el0_svc_compat+0x28/0x3c el0_svc_compat+0x10/0x1c el0_sync_compat_handler+0xa8/0xcc el0_sync_compat+0x178/0x180 ---[ end trace 3d95032303e59e68 ]--- Changes from v5 (https://lore.kernel.org/r/20210420215003.3510247-1-swboyd@chromium.org): * Rebased onto v5.12 * Moved logic for when to include build ID code to kdump patch * Simplified commit text to show before/after details Changes from v4 (https://lore.kernel.org/r/20210410015300.3764485-1-swboyd@chromium.org): * Stubbed out more code when CONFIG_STACKTRACE_BUILD_ID=n * Use static_assert instead of BUILD_BUG_ON() * Dropped bad printk change to IP on x86 Changes from v3 (https://lore.kernel.org/r/20210331030520.3816265-1-swboyd@chromium.org): * Fixed compilation warnings due to config changes * Fixed kernel-doc on init_vmlinx_build_id() * Totally removed add_build_id_vmcoreinfo() * Added another printk format %pBb to help x86 print backtraces * Some BUILD_BUG_ON() checks to make sure the buildid doesn't get bigger or smaller Changes from v2 (https://lore.kernel.org/r/20210324020443.1815557-1-swboyd@chromium.org): * Renamed symbol printing function to indicate build IDness * Put build ID information behind Kconfig knob * Build ID for vmlinux is calculated in early init instead of on demand * printk format is %pS[R]b Changes from v1 (https://lore.kernel.org/r/20210301174749.1269154-1-swboyd@chromium.org): * New printk format %pSb and %pSr * Return binary format instead of hex format string from build ID APIs * Some new patches to cleanup buildid/decode_stacktrace.sh * A new patch to decode_stacktrace.sh to parse output [1] https://fedoraproject.org/wiki/Releases/FeatureBuildId Cc: Alexei Starovoitov <ast@kernel.org> Cc: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Cc: Baoquan He <bhe@redhat.com> Cc: Borislav Petkov <bp@alien8.de> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Dave Young <dyoung@redhat.com> Cc: Evan Green <evgreen@chromium.org> Cc: Hsin-Yi Wang <hsinyi@chromium.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Jessica Yu <jeyu@kernel.org> Cc: Jiri Olsa <jolsa@kernel.org> Cc: <kexec@lists.infradead.org> Cc: Konstantin Khlebnikov <khlebnikov@yandex-team.ru> Cc: <linux-arm-kernel@lists.infradead.org> Cc: <linux-doc@vger.kernel.org> Cc: <linux-kernel@vger.kernel.org> Cc: Matthew Wilcox <willy@infradead.org> Cc: Petr Mladek <pmladek@suse.com> Cc: Rasmus Villemoes <linux@rasmusvillemoes.dk> Cc: Sasha Levin <sashal@kernel.org> Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vivek Goyal <vgoyal@redhat.com> Cc: Will Deacon <will@kernel.org> Cc: <x86@kernel.org> Cc: Christoph Hellwig <hch@infradead.org> Cc: peter enderborg <peter.enderborg@sony.com> Stephen Boyd (13): buildid: Only consider GNU notes for build ID parsing buildid: Add API to parse build ID out of buffer buildid: Stash away kernels build ID on init dump_stack: Add vmlinux build ID to stack traces module: Add printk formats to add module build ID to stacktraces arm64: stacktrace: Use %pSb for backtrace printing x86/dumpstack: Use %pSb/%pBb for backtrace printing scripts/decode_stacktrace.sh: Support debuginfod scripts/decode_stacktrace.sh: Silence stderr messages from addr2line/nm scripts/decode_stacktrace.sh: Indicate 'auto' can be used for base path buildid: Mark some arguments const buildid: Fix kernel-doc notation kdump: Use vmlinux_build_id to simplify Documentation/core-api/printk-formats.rst | 11 +++ arch/arm64/kernel/stacktrace.c | 2 +- arch/x86/kernel/dumpstack.c | 2 +- include/linux/buildid.h | 8 ++ include/linux/crash_core.h | 12 +-- include/linux/kallsyms.h | 20 ++++- include/linux/module.h | 8 +- init/main.c | 1 + kernel/crash_core.c | 50 +---------- kernel/kallsyms.c | 101 +++++++++++++++++----- kernel/module.c | 31 ++++++- lib/Kconfig.debug | 11 +++ lib/buildid.c | 74 ++++++++++++---- lib/dump_stack.c | 13 ++- lib/vsprintf.c | 8 +- scripts/decode_stacktrace.sh | 89 +++++++++++++++---- 16 files changed, 327 insertions(+), 114 deletions(-) base-commit: 9f4ad9e425a1d3b6a34617b8ea226d56a119a717 -- https://chromeos.dev ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v6 05/13] module: Add printk formats to add module build ID to stacktraces 2021-05-11 0:38 [PATCH v6 00/13] Add build ID to stacktraces Stephen Boyd @ 2021-05-11 0:38 ` Stephen Boyd 2021-05-11 11:35 ` Petr Mladek 2021-05-11 11:48 ` [PATCH v6 00/13] Add " Petr Mladek 2021-05-11 12:36 ` David Laight 2 siblings, 1 reply; 11+ messages in thread From: Stephen Boyd @ 2021-05-11 0:38 UTC (permalink / raw) To: Andrew Morton, Petr Mladek Cc: linux-kernel, Jiri Olsa, Alexei Starovoitov, Jessica Yu, Evan Green, Hsin-Yi Wang, Steven Rostedt, Sergey Senozhatsky, Andy Shevchenko, Rasmus Villemoes, linux-doc, Matthew Wilcox Let's make kernel stacktraces easier to identify by including the build ID[1] of a module if the stacktrace is printing a symbol from a module. This makes it simpler for developers to locate a kernel module's full debuginfo for a particular stacktrace. Combined with scripts/decode_stracktrace.sh, a developer can download the matching debuginfo from a debuginfod[2] server and find the exact file and line number for the functions plus offsets in a stacktrace that match the module. This is especially useful for pstore crash debugging where the kernel crashes are recorded in something like console-ramoops and the recovery kernel/modules are different or the debuginfo doesn't exist on the device due to space concerns (the debuginfo can be too large for space limited devices). Originally, I put this on the %pS format, but that was quickly rejected given that %pS is used in other places such as ftrace where build IDs aren't meaningful. There was some discussions on the list to put every module build ID into the "Modules linked in:" section of the stacktrace message but that quickly becomes very hard to read once you have more than three or four modules linked in. It also provides too much information when we don't expect each module to be traversed in a stacktrace. Having the build ID for modules that aren't important just makes things messy. Splitting it to multiple lines for each module quickly explodes the number of lines printed in an oops too, possibly wrapping the warning off the console. And finally, trying to stash away each module used in a callstack to provide the ID of each symbol printed is cumbersome and would require changes to each architecture to stash away modules and return their build IDs once unwinding has completed. Instead, we opt for the simpler approach of introducing new printk formats '%pS[R]b' for "pointer symbolic backtrace with module build ID" and '%pBb' for "pointer backtrace with module build ID" and then updating the few places in the architecture layer where the stacktrace is printed to use this new format. Before: Call trace: lkdtm_WARNING+0x28/0x30 [lkdtm] direct_entry+0x16c/0x1b4 [lkdtm] full_proxy_write+0x74/0xa4 vfs_write+0xec/0x2e8 After: Call trace: lkdtm_WARNING+0x28/0x30 [lkdtm 6c2215028606bda50de823490723dc4bc5bf46f9] direct_entry+0x16c/0x1b4 [lkdtm 6c2215028606bda50de823490723dc4bc5bf46f9] full_proxy_write+0x74/0xa4 vfs_write+0xec/0x2e8 Cc: Jiri Olsa <jolsa@kernel.org> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Jessica Yu <jeyu@kernel.org> Cc: Evan Green <evgreen@chromium.org> Cc: Hsin-Yi Wang <hsinyi@chromium.org> Cc: Petr Mladek <pmladek@suse.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com> Cc: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Cc: Rasmus Villemoes <linux@rasmusvillemoes.dk> Cc: <linux-doc@vger.kernel.org> Cc: Matthew Wilcox <willy@infradead.org> Link: https://fedoraproject.org/wiki/Releases/FeatureBuildId [1] Link: https://sourceware.org/elfutils/Debuginfod.html [2] Signed-off-by: Stephen Boyd <swboyd@chromium.org> --- Documentation/core-api/printk-formats.rst | 11 +++ include/linux/kallsyms.h | 20 ++++- include/linux/module.h | 8 +- kernel/kallsyms.c | 101 +++++++++++++++++----- kernel/module.c | 31 ++++++- lib/vsprintf.c | 8 +- 6 files changed, 154 insertions(+), 25 deletions(-) diff --git a/Documentation/core-api/printk-formats.rst b/Documentation/core-api/printk-formats.rst index 160e710d992f..5f60533f2a56 100644 --- a/Documentation/core-api/printk-formats.rst +++ b/Documentation/core-api/printk-formats.rst @@ -114,6 +114,17 @@ used when printing stack backtraces. The specifier takes into consideration the effect of compiler optimisations which may occur when tail-calls are used and marked with the noreturn GCC attribute. +If the pointer is within a module, the module name and optionally build ID is +printed after the symbol name with an extra ``b`` appended to the end of the +specifier. + +:: + %pS versatile_init+0x0/0x110 [module_name] + %pSb versatile_init+0x0/0x110 [module_name ed5019fdf5e53be37cb1ba7899292d7e143b259e] + %pSRb versatile_init+0x9/0x110 [module_name ed5019fdf5e53be37cb1ba7899292d7e143b259e] + (with __builtin_extract_return_addr() translation) + %pBb prev_fn_of_versatile_init+0x88/0x88 [module_name ed5019fdf5e53be37cb1ba7899292d7e143b259e] + Probed Pointers from BPF / tracing ---------------------------------- diff --git a/include/linux/kallsyms.h b/include/linux/kallsyms.h index 465060acc981..f760cb839775 100644 --- a/include/linux/kallsyms.h +++ b/include/linux/kallsyms.h @@ -7,6 +7,7 @@ #define _LINUX_KALLSYMS_H #include <linux/errno.h> +#include <linux/buildid.h> #include <linux/kernel.h> #include <linux/stddef.h> #include <linux/mm.h> @@ -15,8 +16,9 @@ #include <asm/sections.h> #define KSYM_NAME_LEN 128 -#define KSYM_SYMBOL_LEN (sizeof("%s+%#lx/%#lx [%s]") + (KSYM_NAME_LEN - 1) + \ - 2*(BITS_PER_LONG*3/10) + (MODULE_NAME_LEN - 1) + 1) +#define KSYM_SYMBOL_LEN (sizeof("%s+%#lx/%#lx [%s %s]") + (KSYM_NAME_LEN - 1) + \ + 2*(BITS_PER_LONG*3/10) + (MODULE_NAME_LEN - 1) + \ + (BUILD_ID_SIZE_MAX * 2) + 1) struct cred; struct module; @@ -91,8 +93,10 @@ const char *kallsyms_lookup(unsigned long addr, /* Look up a kernel symbol and return it in a text buffer. */ extern int sprint_symbol(char *buffer, unsigned long address); +extern int sprint_symbol_build_id(char *buffer, unsigned long address); extern int sprint_symbol_no_offset(char *buffer, unsigned long address); extern int sprint_backtrace(char *buffer, unsigned long address); +extern int sprint_backtrace_build_id(char *buffer, unsigned long address); int lookup_symbol_name(unsigned long addr, char *symname); int lookup_symbol_attrs(unsigned long addr, unsigned long *size, unsigned long *offset, char *modname, char *name); @@ -128,6 +132,12 @@ static inline int sprint_symbol(char *buffer, unsigned long addr) return 0; } +static inline int sprint_symbol_build_id(char *buffer, unsigned long address) +{ + *buffer = '\0'; + return 0; +} + static inline int sprint_symbol_no_offset(char *buffer, unsigned long addr) { *buffer = '\0'; @@ -140,6 +150,12 @@ static inline int sprint_backtrace(char *buffer, unsigned long addr) return 0; } +static inline int sprint_backtrace_build_id(char *buffer, unsigned long addr) +{ + *buffer = '\0'; + return 0; +} + static inline int lookup_symbol_name(unsigned long addr, char *symname) { return -ERANGE; diff --git a/include/linux/module.h b/include/linux/module.h index da4b6fbe8ebe..4145a53ff9d8 100644 --- a/include/linux/module.h +++ b/include/linux/module.h @@ -11,6 +11,7 @@ #include <linux/list.h> #include <linux/stat.h> +#include <linux/buildid.h> #include <linux/compiler.h> #include <linux/cache.h> #include <linux/kmod.h> @@ -364,6 +365,11 @@ struct module { /* Unique handle for this module */ char name[MODULE_NAME_LEN]; +#ifdef CONFIG_STACKTRACE_BUILD_ID + /* Module build ID */ + unsigned char build_id[BUILD_ID_SIZE_MAX]; +#endif + /* Sysfs stuff. */ struct module_kobject mkobj; struct module_attribute *modinfo_attrs; @@ -627,7 +633,7 @@ void *dereference_module_function_descriptor(struct module *mod, void *ptr); const char *module_address_lookup(unsigned long addr, unsigned long *symbolsize, unsigned long *offset, - char **modname, + char **modname, const unsigned char **modbuildid, char *namebuf); int lookup_module_symbol_name(unsigned long addr, char *symname); int lookup_module_symbol_attrs(unsigned long addr, unsigned long *size, unsigned long *offset, char *modname, char *name); diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c index 8043a90aa50e..b1ad3df3d71b 100644 --- a/kernel/kallsyms.c +++ b/kernel/kallsyms.c @@ -25,7 +25,10 @@ #include <linux/filter.h> #include <linux/ftrace.h> #include <linux/kprobes.h> +#include <linux/build_bug.h> #include <linux/compiler.h> +#include <linux/module.h> +#include <linux/kernel.h> /* * These will be re-linked against their real values @@ -273,21 +276,13 @@ int kallsyms_lookup_size_offset(unsigned long addr, unsigned long *symbolsize, get_symbol_pos(addr, symbolsize, offset); return 1; } - return !!module_address_lookup(addr, symbolsize, offset, NULL, namebuf) || + return !!module_address_lookup(addr, symbolsize, offset, NULL, NULL, namebuf) || !!__bpf_address_lookup(addr, symbolsize, offset, namebuf); } -/* - * Lookup an address - * - modname is set to NULL if it's in the kernel. - * - We guarantee that the returned name is valid until we reschedule even if. - * It resides in a module. - * - We also guarantee that modname will be valid until rescheduled. - */ -const char *kallsyms_lookup(unsigned long addr, - unsigned long *symbolsize, - unsigned long *offset, - char **modname, char *namebuf) +const char *kallsyms_lookup_buildid(unsigned long addr, unsigned long *symbolsize, + unsigned long *offset, char **modname, + const unsigned char **modbuildid, char *namebuf) { const char *ret; @@ -303,12 +298,14 @@ const char *kallsyms_lookup(unsigned long addr, namebuf, KSYM_NAME_LEN); if (modname) *modname = NULL; + if (modbuildid) + *modbuildid = NULL; return namebuf; } /* See if it's in a module or a BPF JITed image. */ ret = module_address_lookup(addr, symbolsize, offset, - modname, namebuf); + modname, modbuildid, namebuf); if (!ret) ret = bpf_address_lookup(addr, symbolsize, offset, modname, namebuf); @@ -319,6 +316,22 @@ const char *kallsyms_lookup(unsigned long addr, return ret; } +/* + * Lookup an address + * - modname is set to NULL if it's in the kernel. + * - We guarantee that the returned name is valid until we reschedule even if. + * It resides in a module. + * - We also guarantee that modname will be valid until rescheduled. + */ +const char *kallsyms_lookup(unsigned long addr, + unsigned long *symbolsize, + unsigned long *offset, + char **modname, char *namebuf) +{ + return kallsyms_lookup_buildid(addr, symbolsize, offset, modname, + NULL, namebuf); +} + int lookup_symbol_name(unsigned long addr, char *symname) { symname[0] = '\0'; @@ -359,15 +372,17 @@ int lookup_symbol_attrs(unsigned long addr, unsigned long *size, /* Look up a kernel symbol and return it in a text buffer. */ static int __sprint_symbol(char *buffer, unsigned long address, - int symbol_offset, int add_offset) + int symbol_offset, int add_offset, int add_buildid) { char *modname; + const unsigned char *buildid; const char *name; unsigned long offset, size; int len; address += symbol_offset; - name = kallsyms_lookup(address, &size, &offset, &modname, buffer); + name = kallsyms_lookup_buildid(address, &size, &offset, &modname, &buildid, + buffer); if (!name) return sprintf(buffer, "0x%lx", address - symbol_offset); @@ -379,8 +394,17 @@ static int __sprint_symbol(char *buffer, unsigned long address, if (add_offset) len += sprintf(buffer + len, "+%#lx/%#lx", offset, size); - if (modname) - len += sprintf(buffer + len, " [%s]", modname); + if (modname) { + len += sprintf(buffer + len, " [%s", modname); +#if IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID) + if (add_buildid && buildid) { + /* build ID should match length of sprintf */ + static_assert(sizeof(typeof_member(struct module, build_id)) == 20); + len += sprintf(buffer + len, " %20phN", buildid); + } +#endif + len += sprintf(buffer + len, "]"); + } return len; } @@ -398,10 +422,27 @@ static int __sprint_symbol(char *buffer, unsigned long address, */ int sprint_symbol(char *buffer, unsigned long address) { - return __sprint_symbol(buffer, address, 0, 1); + return __sprint_symbol(buffer, address, 0, 1, 0); } EXPORT_SYMBOL_GPL(sprint_symbol); +/** + * sprint_symbol_build_id - Look up a kernel symbol and return it in a text buffer + * @buffer: buffer to be stored + * @address: address to lookup + * + * This function looks up a kernel symbol with @address and stores its name, + * offset, size, module name and module build ID to @buffer if possible. If no + * symbol was found, just saves its @address as is. + * + * This function returns the number of bytes stored in @buffer. + */ +int sprint_symbol_build_id(char *buffer, unsigned long address) +{ + return __sprint_symbol(buffer, address, 0, 1, 1); +} +EXPORT_SYMBOL_GPL(sprint_symbol_build_id); + /** * sprint_symbol_no_offset - Look up a kernel symbol and return it in a text buffer * @buffer: buffer to be stored @@ -415,7 +456,7 @@ EXPORT_SYMBOL_GPL(sprint_symbol); */ int sprint_symbol_no_offset(char *buffer, unsigned long address) { - return __sprint_symbol(buffer, address, 0, 0); + return __sprint_symbol(buffer, address, 0, 0, 0); } EXPORT_SYMBOL_GPL(sprint_symbol_no_offset); @@ -435,7 +476,27 @@ EXPORT_SYMBOL_GPL(sprint_symbol_no_offset); */ int sprint_backtrace(char *buffer, unsigned long address) { - return __sprint_symbol(buffer, address, -1, 1); + return __sprint_symbol(buffer, address, -1, 1, 0); +} + +/** + * sprint_backtrace_build_id - Look up a backtrace symbol and return it in a text buffer + * @buffer: buffer to be stored + * @address: address to lookup + * + * This function is for stack backtrace and does the same thing as + * sprint_symbol() but with modified/decreased @address. If there is a + * tail-call to the function marked "noreturn", gcc optimized out code after + * the call so that the stack-saved return address could point outside of the + * caller. This function ensures that kallsyms will find the original caller + * by decreasing @address. This function also appends the module build ID to + * the @buffer if @address is within a kernel module. + * + * This function returns the number of bytes stored in @buffer. + */ +int sprint_backtrace_build_id(char *buffer, unsigned long address) +{ + return __sprint_symbol(buffer, address, -1, 1, 1); } /* To avoid using get_symbol_offset for every symbol, we carry prefix along. */ diff --git a/kernel/module.c b/kernel/module.c index 30479355ab85..a5eaf5188a83 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -13,6 +13,7 @@ #include <linux/trace_events.h> #include <linux/init.h> #include <linux/kallsyms.h> +#include <linux/buildid.h> #include <linux/file.h> #include <linux/fs.h> #include <linux/sysfs.h> @@ -2780,6 +2781,26 @@ static void add_kallsyms(struct module *mod, const struct load_info *info) } #endif /* CONFIG_KALLSYMS */ +#if IS_ENABLED(CONFIG_KALLSYMS) && IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID) +static void init_build_id(struct module *mod, const struct load_info *info) +{ + const Elf_Shdr *sechdr; + unsigned int i; + + for (i = 0; i < info->hdr->e_shnum; i++) { + sechdr = &info->sechdrs[i]; + if (!sect_empty(sechdr) && sechdr->sh_type == SHT_NOTE && + !build_id_parse_buf((void *)sechdr->sh_addr, mod->build_id, + sechdr->sh_size)) + break; + } +} +#else +static void init_build_id(struct module *mod, const struct load_info *info) +{ +} +#endif + static void dynamic_debug_setup(struct module *mod, struct _ddebug *debug, unsigned int num) { if (!debug) @@ -4004,6 +4025,7 @@ static int load_module(struct load_info *info, const char __user *uargs, goto free_arch_cleanup; } + init_build_id(mod, info); dynamic_debug_setup(mod, info->debug, info->num_debug); /* Ftrace init must be called in the MODULE_STATE_UNFORMED state */ @@ -4235,7 +4257,7 @@ void * __weak dereference_module_function_descriptor(struct module *mod, const char *module_address_lookup(unsigned long addr, unsigned long *size, unsigned long *offset, - char **modname, + char **modname, const unsigned char **modbuildid, char *namebuf) { const char *ret = NULL; @@ -4246,6 +4268,13 @@ const char *module_address_lookup(unsigned long addr, if (mod) { if (modname) *modname = mod->name; + if (modbuildid) { +#if IS_ENABLED(CONFIG_STACKTRACE_BUILD_ID) + *modbuildid = mod->build_id; +#else + *modbuildid = NULL; +#endif + } ret = find_kallsyms_symbol(mod, addr, size, offset); } diff --git a/lib/vsprintf.c b/lib/vsprintf.c index 41ddc353ebb8..fee5282ebb0a 100644 --- a/lib/vsprintf.c +++ b/lib/vsprintf.c @@ -966,8 +966,12 @@ char *symbol_string(char *buf, char *end, void *ptr, value = (unsigned long)ptr; #ifdef CONFIG_KALLSYMS - if (*fmt == 'B') + if (*fmt == 'B' && fmt[1] == 'b') + sprint_backtrace_build_id(sym, value); + else if (*fmt == 'B') sprint_backtrace(sym, value); + else if (*fmt == 'S' && (fmt[1] == 'b' || (fmt[1] == 'R' && fmt[2] == 'b'))) + sprint_symbol_build_id(sym, value); else if (*fmt != 's') sprint_symbol(sym, value); else @@ -2129,9 +2133,11 @@ early_param("no_hash_pointers", no_hash_pointers_enable); * - 'S' For symbolic direct pointers (or function descriptors) with offset * - 's' For symbolic direct pointers (or function descriptors) without offset * - '[Ss]R' as above with __builtin_extract_return_addr() translation + * - 'S[R]b' as above with module build ID (for use in backtraces) * - '[Ff]' %pf and %pF were obsoleted and later removed in favor of * %ps and %pS. Be careful when re-using these specifiers. * - 'B' For backtraced symbolic direct pointers with offset + * - 'Bb' as above with module build ID (for use in backtraces) * - 'R' For decoded struct resource, e.g., [mem 0x0-0x1f 64bit pref] * - 'r' For raw struct resource, e.g., [mem 0x0-0x1f flags 0x201] * - 'b[l]' For a bitmap, the number of bits is determined by the field -- https://chromeos.dev ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH v6 05/13] module: Add printk formats to add module build ID to stacktraces 2021-05-11 0:38 ` [PATCH v6 05/13] module: Add printk formats to add module " Stephen Boyd @ 2021-05-11 11:35 ` Petr Mladek 2021-05-11 16:51 ` Stephen Boyd 0 siblings, 1 reply; 11+ messages in thread From: Petr Mladek @ 2021-05-11 11:35 UTC (permalink / raw) To: Stephen Boyd Cc: Andrew Morton, linux-kernel, Jiri Olsa, Alexei Starovoitov, Jessica Yu, Evan Green, Hsin-Yi Wang, Steven Rostedt, Sergey Senozhatsky, Andy Shevchenko, Rasmus Villemoes, linux-doc, Matthew Wilcox On Mon 2021-05-10 17:38:37, Stephen Boyd wrote: > Let's make kernel stacktraces easier to identify by including the build > ID[1] of a module if the stacktrace is printing a symbol from a module. > --- a/include/linux/module.h > +++ b/include/linux/module.h > @@ -11,6 +11,7 @@ > > #include <linux/list.h> > #include <linux/stat.h> > +#include <linux/buildid.h> > #include <linux/compiler.h> > #include <linux/cache.h> > #include <linux/kmod.h> > @@ -364,6 +365,11 @@ struct module { > /* Unique handle for this module */ > char name[MODULE_NAME_LEN]; > > +#ifdef CONFIG_STACKTRACE_BUILD_ID > + /* Module build ID */ > + unsigned char build_id[BUILD_ID_SIZE_MAX]; > +#endif > + > /* Sysfs stuff. */ > struct module_kobject mkobj; > struct module_attribute *modinfo_attrs; > @@ -627,7 +633,7 @@ void *dereference_module_function_descriptor(struct module *mod, void *ptr); > const char *module_address_lookup(unsigned long addr, > unsigned long *symbolsize, > unsigned long *offset, > - char **modname, > + char **modname, const unsigned char **modbuildid, Nit: It would have been better to follow the existing style and put only one parameter per-line. Please, fix it in the next version if there is any. But no need to resend it just because of this unless Jessica asks for it. > char *namebuf); > int lookup_module_symbol_name(unsigned long addr, char *symname); > int lookup_module_symbol_attrs(unsigned long addr, unsigned long *size, unsigned long *offset, char *modname, char *name); > diff --git a/kernel/module.c b/kernel/module.c > index 30479355ab85..a5eaf5188a83 100644 > --- a/kernel/module.c > +++ b/kernel/module.c > @@ -4235,7 +4257,7 @@ void * __weak dereference_module_function_descriptor(struct module *mod, > const char *module_address_lookup(unsigned long addr, > unsigned long *size, > unsigned long *offset, > - char **modname, > + char **modname, const unsigned char **modbuildid, Same here. > char *namebuf) > { > const char *ret = NULL; Otherwise, it looks good. Reviewed-by: Petr Mladek <pmladek@suse.com> Best Regards, Petr ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v6 05/13] module: Add printk formats to add module build ID to stacktraces 2021-05-11 11:35 ` Petr Mladek @ 2021-05-11 16:51 ` Stephen Boyd 0 siblings, 0 replies; 11+ messages in thread From: Stephen Boyd @ 2021-05-11 16:51 UTC (permalink / raw) To: Petr Mladek Cc: Andrew Morton, linux-kernel, Jiri Olsa, Alexei Starovoitov, Jessica Yu, Evan Green, Hsin-Yi Wang, Steven Rostedt, Sergey Senozhatsky, Andy Shevchenko, Rasmus Villemoes, linux-doc, Matthew Wilcox Quoting Petr Mladek (2021-05-11 04:35:21) > On Mon 2021-05-10 17:38:37, Stephen Boyd wrote: > > Let's make kernel stacktraces easier to identify by including the build > > ID[1] of a module if the stacktrace is printing a symbol from a module. > > > --- a/include/linux/module.h > > +++ b/include/linux/module.h > > @@ -11,6 +11,7 @@ > > > > #include <linux/list.h> > > #include <linux/stat.h> > > +#include <linux/buildid.h> > > #include <linux/compiler.h> > > #include <linux/cache.h> > > #include <linux/kmod.h> > > @@ -364,6 +365,11 @@ struct module { > > /* Unique handle for this module */ > > char name[MODULE_NAME_LEN]; > > > > +#ifdef CONFIG_STACKTRACE_BUILD_ID > > + /* Module build ID */ > > + unsigned char build_id[BUILD_ID_SIZE_MAX]; > > +#endif > > + > > /* Sysfs stuff. */ > > struct module_kobject mkobj; > > struct module_attribute *modinfo_attrs; > > @@ -627,7 +633,7 @@ void *dereference_module_function_descriptor(struct module *mod, void *ptr); > > const char *module_address_lookup(unsigned long addr, > > unsigned long *symbolsize, > > unsigned long *offset, > > - char **modname, > > + char **modname, const unsigned char **modbuildid, > > Nit: It would have been better to follow the existing style and put only one > parameter per-line. > > Please, fix it in the next version if there is any. But no need > to resend it just because of this unless Jessica asks for it. Thanks. akpm already picked this series up and fixed this problem. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v6 00/13] Add build ID to stacktraces 2021-05-11 0:38 [PATCH v6 00/13] Add build ID to stacktraces Stephen Boyd 2021-05-11 0:38 ` [PATCH v6 05/13] module: Add printk formats to add module " Stephen Boyd @ 2021-05-11 11:48 ` Petr Mladek 2021-05-11 12:36 ` David Laight 2 siblings, 0 replies; 11+ messages in thread From: Petr Mladek @ 2021-05-11 11:48 UTC (permalink / raw) To: Stephen Boyd Cc: Andrew Morton, linux-kernel, Alexei Starovoitov, Andy Shevchenko, Baoquan He, Borislav Petkov, Catalin Marinas, Dave Young, Evan Green, Hsin-Yi Wang, Ingo Molnar, Jessica Yu, Jiri Olsa, kexec, Konstantin Khlebnikov, linux-arm-kernel, linux-doc, Matthew Wilcox, Rasmus Villemoes, Sasha Levin, Sergey Senozhatsky, Steven Rostedt, Thomas Gleixner, Vivek Goyal, Will Deacon, x86, Christoph Hellwig, peter enderborg On Mon 2021-05-10 17:38:32, Stephen Boyd wrote: > This series adds the kernel's build ID[1] to the stacktrace header > printed in oops messages, warnings, etc. and the build ID for any module > that appears in the stacktrace after the module name. The goal is to > make the stacktrace more self-contained and descriptive by including the > relevant build IDs in the kernel logs when something goes wrong. This > can be used by post processing tools like script/decode_stacktrace.sh > and kernel developers to easily locate the debug info associated with a > kernel crash and line up what line and file things started falling apart > at. The entire series looks good to me. I reviewed carefully only the 5th patch touching printk/kallsyms/module code. I just scanned over the other patches touching kernel code and did not notice any obvious problem. I did not check the changes in decode_stacktrace.sh at all. I tried to get stacktraces on x86_64 and it worked as expected. Best Regards, Petr ^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH v6 00/13] Add build ID to stacktraces 2021-05-11 0:38 [PATCH v6 00/13] Add build ID to stacktraces Stephen Boyd 2021-05-11 0:38 ` [PATCH v6 05/13] module: Add printk formats to add module " Stephen Boyd 2021-05-11 11:48 ` [PATCH v6 00/13] Add " Petr Mladek @ 2021-05-11 12:36 ` David Laight 2021-05-11 12:52 ` Steven Rostedt 2 siblings, 1 reply; 11+ messages in thread From: David Laight @ 2021-05-11 12:36 UTC (permalink / raw) To: 'Stephen Boyd', Andrew Morton, Petr Mladek Cc: linux-kernel, Alexei Starovoitov, Andy Shevchenko, Baoquan He, Borislav Petkov, Catalin Marinas, Dave Young, Evan Green, Hsin-Yi Wang, Ingo Molnar, Jessica Yu, Jiri Olsa, kexec, Konstantin Khlebnikov, linux-arm-kernel, linux-doc, Matthew Wilcox, Rasmus Villemoes, Sasha Levin, Sergey Senozhatsky, Steven Rostedt, Thomas Gleixner, Vivek Goyal, Will Deacon, x86, Christoph Hellwig, peter enderborg From: Stephen Boyd > Sent: 11 May 2021 01:39 > > This series adds the kernel's build ID[1] to the stacktrace header > printed in oops messages, warnings, etc. and the build ID for any module > that appears in the stacktrace after the module name. The goal is to > make the stacktrace more self-contained and descriptive by including the > relevant build IDs in the kernel logs when something goes wrong. This > can be used by post processing tools like script/decode_stacktrace.sh > and kernel developers to easily locate the debug info associated with a > kernel crash and line up what line and file things started falling apart > at. > > To show how this can be used I've included a patch to > decode_stacktrace.sh that downloads the debuginfo from a debuginfod > server. ... > Here's an example lkdtm stacktrace on arm64. > > WARNING: CPU: 4 PID: 3255 at drivers/misc/lkdtm/bugs.c:83 lkdtm_WARNING+0x28/0x30 [lkdtm] > Modules linked in: lkdtm rfcomm algif_hash algif_skcipher af_alg xt_cgroup uinput xt_MASQUERADE > CPU: 4 PID: 3255 Comm: bash Not tainted 5.11 #3 aa23f7a1231c229de205662d5a9e0d4c580f19a1 > Hardware name: Google Lazor (rev3+) with KB Backlight (DT) ... > x1 : ffffff93fef15788 x0 : ffffffe3622352e0 > Call trace: > lkdtm_WARNING+0x28/0x30 [lkdtm ed5019fdf5e53be37cb1ba7899292d7e143b259e] > direct_entry+0x16c/0x1b4 [lkdtm ed5019fdf5e53be37cb1ba7899292d7e143b259e] > full_proxy_write+0x74/0xa4 Is there any way to get it to print each module ID only once? David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales) ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v6 00/13] Add build ID to stacktraces 2021-05-11 12:36 ` David Laight @ 2021-05-11 12:52 ` Steven Rostedt 2021-05-11 12:58 ` David Laight 0 siblings, 1 reply; 11+ messages in thread From: Steven Rostedt @ 2021-05-11 12:52 UTC (permalink / raw) To: David Laight Cc: 'Stephen Boyd', Andrew Morton, Petr Mladek, linux-kernel, Alexei Starovoitov, Andy Shevchenko, Baoquan He, Borislav Petkov, Catalin Marinas, Dave Young, Evan Green, Hsin-Yi Wang, Ingo Molnar, Jessica Yu, Jiri Olsa, kexec, Konstantin Khlebnikov, linux-arm-kernel, linux-doc, Matthew Wilcox, Rasmus Villemoes, Sasha Levin, Sergey Senozhatsky, Thomas Gleixner, Vivek Goyal, Will Deacon, x86, Christoph Hellwig, peter enderborg On Tue, 11 May 2021 12:36:06 +0000 David Laight <David.Laight@ACULAB.COM> wrote: > > x1 : ffffff93fef15788 x0 : ffffffe3622352e0 > > Call trace: > > lkdtm_WARNING+0x28/0x30 [lkdtm ed5019fdf5e53be37cb1ba7899292d7e143b259e] > > direct_entry+0x16c/0x1b4 [lkdtm ed5019fdf5e53be37cb1ba7899292d7e143b259e] > > full_proxy_write+0x74/0xa4 > > Is there any way to get it to print each module ID only once? If there's a trivial way to do that, then perhaps it should be done, but for now, this patch series isn't as obnoxious as the previous versions. It only affects stack traces, and I'm fine with that. -- Steve ^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH v6 00/13] Add build ID to stacktraces 2021-05-11 12:52 ` Steven Rostedt @ 2021-05-11 12:58 ` David Laight 2021-05-11 14:21 ` Petr Mladek 0 siblings, 1 reply; 11+ messages in thread From: David Laight @ 2021-05-11 12:58 UTC (permalink / raw) To: 'Steven Rostedt' Cc: 'Stephen Boyd', Andrew Morton, Petr Mladek, linux-kernel, Alexei Starovoitov, Andy Shevchenko, Baoquan He, Borislav Petkov, Catalin Marinas, Dave Young, Evan Green, Hsin-Yi Wang, Ingo Molnar, Jessica Yu, Jiri Olsa, kexec, Konstantin Khlebnikov, linux-arm-kernel, linux-doc, Matthew Wilcox, Rasmus Villemoes, Sasha Levin, Sergey Senozhatsky, Thomas Gleixner, Vivek Goyal, Will Deacon, x86, Christoph Hellwig, peter enderborg From: Steven Rostedt > Sent: 11 May 2021 13:53 > > On Tue, 11 May 2021 12:36:06 +0000 > David Laight <David.Laight@ACULAB.COM> wrote: > > > > x1 : ffffff93fef15788 x0 : ffffffe3622352e0 > > > Call trace: > > > lkdtm_WARNING+0x28/0x30 [lkdtm ed5019fdf5e53be37cb1ba7899292d7e143b259e] > > > direct_entry+0x16c/0x1b4 [lkdtm ed5019fdf5e53be37cb1ba7899292d7e143b259e] > > > full_proxy_write+0x74/0xa4 > > > > Is there any way to get it to print each module ID only once? > > If there's a trivial way to do that, then perhaps it should be done, but for > now, this patch series isn't as obnoxious as the previous versions. It only > affects stack traces, and I'm fine with that. True. Printing the id in the module list was horrid. The real downside is all the extra text that will overflow the in-kernel buffer. At least it shouldn't be extra lines causing screen wrap. Unless the variable names are long - hi rust :-) David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales) ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v6 00/13] Add build ID to stacktraces 2021-05-11 12:58 ` David Laight @ 2021-05-11 14:21 ` Petr Mladek 2021-05-11 14:31 ` David Laight 0 siblings, 1 reply; 11+ messages in thread From: Petr Mladek @ 2021-05-11 14:21 UTC (permalink / raw) To: David Laight Cc: 'Steven Rostedt', 'Stephen Boyd', Andrew Morton, linux-kernel, Alexei Starovoitov, Andy Shevchenko, Baoquan He, Borislav Petkov, Catalin Marinas, Dave Young, Evan Green, Hsin-Yi Wang, Ingo Molnar, Jessica Yu, Jiri Olsa, kexec, Konstantin Khlebnikov, linux-arm-kernel, linux-doc, Matthew Wilcox, Rasmus Villemoes, Sasha Levin, Sergey Senozhatsky, Thomas Gleixner, Vivek Goyal, Will Deacon, x86, Christoph Hellwig, peter enderborg On Tue 2021-05-11 12:58:47, David Laight wrote: > From: Steven Rostedt > > Sent: 11 May 2021 13:53 > > > > On Tue, 11 May 2021 12:36:06 +0000 > > David Laight <David.Laight@ACULAB.COM> wrote: > > > > > > x1 : ffffff93fef15788 x0 : ffffffe3622352e0 > > > > Call trace: > > > > lkdtm_WARNING+0x28/0x30 [lkdtm ed5019fdf5e53be37cb1ba7899292d7e143b259e] > > > > direct_entry+0x16c/0x1b4 [lkdtm ed5019fdf5e53be37cb1ba7899292d7e143b259e] > > > > full_proxy_write+0x74/0xa4 > > > > > > Is there any way to get it to print each module ID only once? > > > > If there's a trivial way to do that, then perhaps it should be done, but for > > now, this patch series isn't as obnoxious as the previous versions. It only > > affects stack traces, and I'm fine with that. > > True. Printing the id in the module list was horrid. > > The real downside is all the extra text that will overflow the > in-kernel buffer. > At least it shouldn't be extra lines causing screen wrap. > Unless the variable names are long - hi rust :-) Note that the ID is printed only when CONFIG_STACKTRACE_BUILD_ID is enabled. It will be used only by some distros/vendors that use it to download the debuginfo packages. Best Regards, Petr ^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH v6 00/13] Add build ID to stacktraces 2021-05-11 14:21 ` Petr Mladek @ 2021-05-11 14:31 ` David Laight 2021-05-11 14:35 ` Matthew Wilcox 0 siblings, 1 reply; 11+ messages in thread From: David Laight @ 2021-05-11 14:31 UTC (permalink / raw) To: 'Petr Mladek' Cc: 'Steven Rostedt', 'Stephen Boyd', Andrew Morton, linux-kernel, Alexei Starovoitov, Andy Shevchenko, Baoquan He, Borislav Petkov, Catalin Marinas, Dave Young, Evan Green, Hsin-Yi Wang, Ingo Molnar, Jessica Yu, Jiri Olsa, kexec, Konstantin Khlebnikov, linux-arm-kernel, linux-doc, Matthew Wilcox, Rasmus Villemoes, Sasha Levin, Sergey Senozhatsky, Thomas Gleixner, Vivek Goyal, Will Deacon, x86, Christoph Hellwig, peter enderborg From: Petr Mladek > Sent: 11 May 2021 15:22 > > On Tue 2021-05-11 12:58:47, David Laight wrote: > > From: Steven Rostedt > > > Sent: 11 May 2021 13:53 > > > > > > On Tue, 11 May 2021 12:36:06 +0000 > > > David Laight <David.Laight@ACULAB.COM> wrote: > > > > > > > > x1 : ffffff93fef15788 x0 : ffffffe3622352e0 > > > > > Call trace: > > > > > lkdtm_WARNING+0x28/0x30 [lkdtm ed5019fdf5e53be37cb1ba7899292d7e143b259e] > > > > > direct_entry+0x16c/0x1b4 [lkdtm ed5019fdf5e53be37cb1ba7899292d7e143b259e] > > > > > full_proxy_write+0x74/0xa4 > > > > > > > > Is there any way to get it to print each module ID only once? > > > > > > If there's a trivial way to do that, then perhaps it should be done, but for > > > now, this patch series isn't as obnoxious as the previous versions. It only > > > affects stack traces, and I'm fine with that. > > > > True. Printing the id in the module list was horrid. > > > > The real downside is all the extra text that will overflow the > > in-kernel buffer. > > At least it shouldn't be extra lines causing screen wrap. > > Unless the variable names are long - hi rust :-) > > Note that the ID is printed only when CONFIG_STACKTRACE_BUILD_ID > is enabled. It will be used only by some distros/vendors that > use it to download the debuginfo packages. Until Ubuntu decide to turn it on :-) Actually, for the use case, the id could be trimmed significantly. It is only trying to differentiate between builds of a specific module. So even 8 digits would be plenty. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales) ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v6 00/13] Add build ID to stacktraces 2021-05-11 14:31 ` David Laight @ 2021-05-11 14:35 ` Matthew Wilcox 0 siblings, 0 replies; 11+ messages in thread From: Matthew Wilcox @ 2021-05-11 14:35 UTC (permalink / raw) To: David Laight Cc: 'Petr Mladek', 'Steven Rostedt', 'Stephen Boyd', Andrew Morton, linux-kernel, Alexei Starovoitov, Andy Shevchenko, Baoquan He, Borislav Petkov, Catalin Marinas, Dave Young, Evan Green, Hsin-Yi Wang, Ingo Molnar, Jessica Yu, Jiri Olsa, kexec, Konstantin Khlebnikov, linux-arm-kernel, linux-doc, Rasmus Villemoes, Sasha Levin, Sergey Senozhatsky, Thomas Gleixner, Vivek Goyal, Will Deacon, x86, Christoph Hellwig, peter enderborg On Tue, May 11, 2021 at 02:31:38PM +0000, David Laight wrote: > Actually, for the use case, the id could be trimmed significantly. > It is only trying to differentiate between builds of a specific module. > So even 8 digits would be plenty. asked and answered. please review the bidding. ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-05-11 16:52 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-05-11 0:38 [PATCH v6 00/13] Add build ID to stacktraces Stephen Boyd 2021-05-11 0:38 ` [PATCH v6 05/13] module: Add printk formats to add module " Stephen Boyd 2021-05-11 11:35 ` Petr Mladek 2021-05-11 16:51 ` Stephen Boyd 2021-05-11 11:48 ` [PATCH v6 00/13] Add " Petr Mladek 2021-05-11 12:36 ` David Laight 2021-05-11 12:52 ` Steven Rostedt 2021-05-11 12:58 ` David Laight 2021-05-11 14:21 ` Petr Mladek 2021-05-11 14:31 ` David Laight 2021-05-11 14:35 ` Matthew Wilcox
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).