unwind_init() takes 100 ms

unwind_init() takes 100 ms

[Paul Menzel]

Dear Josh, dear Linux folks,


Trying to decrease the boot time of the 64-bit Linux kernel (Linux 
4.19-rc7 (0238df64)) on a Asus F2A85-M PRO with an AMD processor, I 
noticed `unwind_init()` called from `setup_arch()` 
`arch/x86/kernel/setup.c` takes over 100 ms to initialize according to 
Linux time stamps (which are present for early boot now).

```
[…]
[    0.047650] ACPI: HPET id: 0x10228210 base: 0xfed00000
[    0.047653] smpboot: Allowing 4 CPUs, 2 hotplug CPUs
[    0.047670] [mem 0x8f000000-0xfebfffff] available for PCI devices
[    0.047672] clocksource: refined-jiffies: mask: 0xffffffff 
max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.151426] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:4 
nr_node_ids:1
[    0.151892] percpu: Embedded 44 pages/cpu @(____ptrval____) s142552 
r8192 d29480 u524288
[…]
```

That’s a third of the time to get to unpacking the initramfs.

Therefore, I have several questions.

1.  Can you reproduce that on your systems?
2.  Can this be decreased or moved out of the “hot path”?
3.  If not, how can this be disabled? I was unable to do so in the 
configuration menu. I could always select an unwinder under *Kernel 
hacking*, and was not able to disable it. I
4.  Would a command line parameter be reasonable `disable_unwind`, so 
people could decrease their boot time with distribution kernels, and 
easily turn it back on, when they need a stacktrace without having to 
rebuild the Linux kernel?


Kind regards,

Paul

Re: unwind_init() takes 100 ms

[Josh Poimboeuf]

On Mon, Oct 08, 2018 at 11:52:46AM +0200, Paul Menzel wrote:
> Dear Josh, dear Linux folks,
> 
> 
> Trying to decrease the boot time of the 64-bit Linux kernel (Linux 4.19-rc7
> (0238df64)) on a Asus F2A85-M PRO with an AMD processor, I noticed
> `unwind_init()` called from `setup_arch()` `arch/x86/kernel/setup.c` takes
> over 100 ms to initialize according to Linux time stamps (which are present
> for early boot now).
> 
> ```
> […]
> [    0.047650] ACPI: HPET id: 0x10228210 base: 0xfed00000
> [    0.047653] smpboot: Allowing 4 CPUs, 2 hotplug CPUs
> [    0.047670] [mem 0x8f000000-0xfebfffff] available for PCI devices
> [    0.047672] clocksource: refined-jiffies: mask: 0xffffffff max_cycles:
> 0xffffffff, max_idle_ns: 1910969940391419 ns
> [    0.151426] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:4
> nr_node_ids:1
> [    0.151892] percpu: Embedded 44 pages/cpu @(____ptrval____) s142552 r8192
> d29480 u524288
> […]
> ```
> 
> That’s a third of the time to get to unpacking the initramfs.
> 
> Therefore, I have several questions.
> 
> 1.  Can you reproduce that on your systems?

I haven't tried to reproduce, but it wouldn't surprise me because it
sorts the big ORC table during boot.

> 2.  Can this be decreased or moved out of the “hot path”?

It would be better to sort it at build time, but I haven't had a chance
to implement that yet.  I'll try to get to it one of these days...

> 3.  If not, how can this be disabled? I was unable to do so in the
> configuration menu. I could always select an unwinder under *Kernel
> hacking*, and was not able to disable it.

If you select CONFIG_UNWINDER_FRAME_POINTER, the boot time penalty
should go away.

Or do CONFIG_EXPERT and CONFIG_UNWINDER_GUESS if you don't want the
general slowdown caused by frame pointers.

> 4.  Would a command line parameter be reasonable `disable_unwind`, so people
> could decrease their boot time with distribution kernels, and easily turn it
> back on, when they need a stacktrace without having to rebuild the Linux
> kernel?

I think a boot cmdline option to disable ORC would be ok.  However,
disabling ORC would need to fall back to the "guess" unwinder.
Otherwise it would make debugging impossible.  That shouldn't be too
hard, but it would require restructuring the code a bit.

I'm not sure about a runtime toggle though.  It might be racy and I'd
rather avoid that completely, unless there's a solid justification for
it.

-- 
Josh

Re: unwind_init() takes 100 ms

[Josh Poimboeuf]

On Mon, Oct 08, 2018 at 12:34:17PM -0500, Josh Poimboeuf wrote:
> > 4.  Would a command line parameter be reasonable `disable_unwind`, so people
> > could decrease their boot time with distribution kernels, and easily turn it
> > back on, when they need a stacktrace without having to rebuild the Linux
> > kernel?
> 
> I think a boot cmdline option to disable ORC would be ok.  However,
> disabling ORC would need to fall back to the "guess" unwinder.
> Otherwise it would make debugging impossible.  That shouldn't be too
> hard, but it would require restructuring the code a bit.

But I should also clarify that this option wouldn't be worth it.
Sorting at build time is the way to go.

-- 
Josh

Re: unwind_init() takes 100 ms

[Ingo Molnar]

* Josh Poimboeuf <jpoimboe@redhat.com> wrote:

> > 4.  Would a command line parameter be reasonable `disable_unwind`, so people
> > could decrease their boot time with distribution kernels, and easily turn it
> > back on, when they need a stacktrace without having to rebuild the Linux
> > kernel?
> 
> I think a boot cmdline option to disable ORC would be ok.

I don't agree with such a kludge: 100 msecs isn't the end of the world for a
distro kernel bootup which takes a lot longer than that typically, and we'd like to
see the build time sorting implemented as well.

Let's fix this for real intstead of a workaround that is going to decrease
the debuggability of the kernel *significantly*.

> I'm not sure about a runtime toggle though.  It might be racy and I'd
> rather avoid that completely, unless there's a solid justification for
> it.

No, just no ...

Debug facilities need to be robust, dependable and as simple as possible.

Thanks,

	Ingo

Re: unwind_init() takes 100 ms

[Ingo Molnar]

* Josh Poimboeuf <jpoimboe@redhat.com> wrote:

> On Mon, Oct 08, 2018 at 12:34:17PM -0500, Josh Poimboeuf wrote:
> > > 4.  Would a command line parameter be reasonable `disable_unwind`, so people
> > > could decrease their boot time with distribution kernels, and easily turn it
> > > back on, when they need a stacktrace without having to rebuild the Linux
> > > kernel?
> > 
> > I think a boot cmdline option to disable ORC would be ok.  However,
> > disabling ORC would need to fall back to the "guess" unwinder.
> > Otherwise it would make debugging impossible.  That shouldn't be too
> > hard, but it would require restructuring the code a bit.
> 
> But I should also clarify that this option wouldn't be worth it.
> Sorting at build time is the way to go.

Yep, very much so.

Thanks,

	Ingo