Dramatic lockdep slowdown in 4.14

Dramatic lockdep slowdown in 4.14

[Johan Hovold]

Hi,

I had noticed that the BeagleBone Black boot time appeared to have
increased significantly with 4.14 and yesterday I finally had time to
investigate it.

Boot time (from "Linux version" to login prompt) had in fact doubled
since 4.13 where it took 17 seconds (with my current config) compared to
the 35 seconds I now see with 4.14-rc4.

I quick bisect pointed to lockdep and specifically the following commit:

	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
	               of a crosslock")

which I've verified is the commit which doubled the boot time (compared
to 28a903f63ec0^) (added by lockdep crossrelease series [1]).

I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
brought boot time down to about 14 seconds.

Now since it's lockdep I guess this can't really be considered a
regression if these changes did improve lockdep correctness, but still,
this dramatic slow down essentially forces me to disable PROVE_LOCKING
by default on this system.

Is this lockdep slowdown expected and desirable?

Johan

[1] https://lkml.kernel.org/r/1502089981-21272-1-git-send-email-byungchul.park@lge.com

Dramatic lockdep slowdown in 4.14

[Johan Hovold]

Hi,

I had noticed that the BeagleBone Black boot time appeared to have
increased significantly with 4.14 and yesterday I finally had time to
investigate it.

Boot time (from "Linux version" to login prompt) had in fact doubled
since 4.13 where it took 17 seconds (with my current config) compared to
the 35 seconds I now see with 4.14-rc4.

I quick bisect pointed to lockdep and specifically the following commit:

	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
	               of a crosslock")

which I've verified is the commit which doubled the boot time (compared
to 28a903f63ec0^) (added by lockdep crossrelease series [1]).

I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
brought boot time down to about 14 seconds.

Now since it's lockdep I guess this can't really be considered a
regression if these changes did improve lockdep correctness, but still,
this dramatic slow down essentially forces me to disable PROVE_LOCKING
by default on this system.

Is this lockdep slowdown expected and desirable?

Johan

[1] https://lkml.kernel.org/r/1502089981-21272-1-git-send-email-byungchul.park@lge.com

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Dramatic lockdep slowdown in 4.14

[Johan Hovold]

Hi,

I had noticed that the BeagleBone Black boot time appeared to have
increased significantly with 4.14 and yesterday I finally had time to
investigate it.

Boot time (from "Linux version" to login prompt) had in fact doubled
since 4.13 where it took 17 seconds (with my current config) compared to
the 35 seconds I now see with 4.14-rc4.

I quick bisect pointed to lockdep and specifically the following commit:

	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
	               of a crosslock")

which I've verified is the commit which doubled the boot time (compared
to 28a903f63ec0^) (added by lockdep crossrelease series [1]).

I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
brought boot time down to about 14 seconds.

Now since it's lockdep I guess this can't really be considered a
regression if these changes did improve lockdep correctness, but still,
this dramatic slow down essentially forces me to disable PROVE_LOCKING
by default on this system.

Is this lockdep slowdown expected and desirable?

Johan

[1] https://lkml.kernel.org/r/1502089981-21272-1-git-send-email-byungchul.park at lge.com

Re: Dramatic lockdep slowdown in 4.14

[Peter Zijlstra]

On Fri, Oct 13, 2017 at 11:03:33AM +0200, Johan Hovold wrote:
> Hi,
> 
> I had noticed that the BeagleBone Black boot time appeared to have
> increased significantly with 4.14 and yesterday I finally had time to
> investigate it.
> 
> Boot time (from "Linux version" to login prompt) had in fact doubled
> since 4.13 where it took 17 seconds (with my current config) compared to
> the 35 seconds I now see with 4.14-rc4.
> 
> I quick bisect pointed to lockdep and specifically the following commit:
> 
> 	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
> 	               of a crosslock")
> 
> which I've verified is the commit which doubled the boot time (compared
> to 28a903f63ec0^) (added by lockdep crossrelease series [1]).
> 
> I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
> brought boot time down to about 14 seconds.
> 
> Now since it's lockdep I guess this can't really be considered a
> regression if these changes did improve lockdep correctness, but still,
> this dramatic slow down essentially forces me to disable PROVE_LOCKING
> by default on this system.
> 
> Is this lockdep slowdown expected and desirable?

Expected yes, desirable not so much. Its the save_stack_trace() in
add_xhlock() (IIRC).

I've not yet had time to figure out what to do about that.

Re: Dramatic lockdep slowdown in 4.14

[Peter Zijlstra]

On Fri, Oct 13, 2017 at 11:03:33AM +0200, Johan Hovold wrote:
> Hi,
> 
> I had noticed that the BeagleBone Black boot time appeared to have
> increased significantly with 4.14 and yesterday I finally had time to
> investigate it.
> 
> Boot time (from "Linux version" to login prompt) had in fact doubled
> since 4.13 where it took 17 seconds (with my current config) compared to
> the 35 seconds I now see with 4.14-rc4.
> 
> I quick bisect pointed to lockdep and specifically the following commit:
> 
> 	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
> 	               of a crosslock")
> 
> which I've verified is the commit which doubled the boot time (compared
> to 28a903f63ec0^) (added by lockdep crossrelease series [1]).
> 
> I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
> brought boot time down to about 14 seconds.
> 
> Now since it's lockdep I guess this can't really be considered a
> regression if these changes did improve lockdep correctness, but still,
> this dramatic slow down essentially forces me to disable PROVE_LOCKING
> by default on this system.
> 
> Is this lockdep slowdown expected and desirable?

Expected yes, desirable not so much. Its the save_stack_trace() in
add_xhlock() (IIRC).

I've not yet had time to figure out what to do about that.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Dramatic lockdep slowdown in 4.14

[Peter Zijlstra]

On Fri, Oct 13, 2017 at 11:03:33AM +0200, Johan Hovold wrote:
> Hi,
> 
> I had noticed that the BeagleBone Black boot time appeared to have
> increased significantly with 4.14 and yesterday I finally had time to
> investigate it.
> 
> Boot time (from "Linux version" to login prompt) had in fact doubled
> since 4.13 where it took 17 seconds (with my current config) compared to
> the 35 seconds I now see with 4.14-rc4.
> 
> I quick bisect pointed to lockdep and specifically the following commit:
> 
> 	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
> 	               of a crosslock")
> 
> which I've verified is the commit which doubled the boot time (compared
> to 28a903f63ec0^) (added by lockdep crossrelease series [1]).
> 
> I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
> brought boot time down to about 14 seconds.
> 
> Now since it's lockdep I guess this can't really be considered a
> regression if these changes did improve lockdep correctness, but still,
> this dramatic slow down essentially forces me to disable PROVE_LOCKING
> by default on this system.
> 
> Is this lockdep slowdown expected and desirable?

Expected yes, desirable not so much. Its the save_stack_trace() in
add_xhlock() (IIRC).

I've not yet had time to figure out what to do about that.

Re: Dramatic lockdep slowdown in 4.14

[Johan Hovold]

On Fri, Oct 13, 2017 at 11:07:44AM +0200, Peter Zijlstra wrote:
> On Fri, Oct 13, 2017 at 11:03:33AM +0200, Johan Hovold wrote:
> > Hi,
> > 
> > I had noticed that the BeagleBone Black boot time appeared to have
> > increased significantly with 4.14 and yesterday I finally had time to
> > investigate it.
> > 
> > Boot time (from "Linux version" to login prompt) had in fact doubled
> > since 4.13 where it took 17 seconds (with my current config) compared to
> > the 35 seconds I now see with 4.14-rc4.
> > 
> > I quick bisect pointed to lockdep and specifically the following commit:
> > 
> > 	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
> > 	               of a crosslock")
> > 
> > which I've verified is the commit which doubled the boot time (compared
> > to 28a903f63ec0^) (added by lockdep crossrelease series [1]).
> > 
> > I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
> > brought boot time down to about 14 seconds.
> > 
> > Now since it's lockdep I guess this can't really be considered a
> > regression if these changes did improve lockdep correctness, but still,
> > this dramatic slow down essentially forces me to disable PROVE_LOCKING
> > by default on this system.
> > 
> > Is this lockdep slowdown expected and desirable?
> 
> Expected yes, desirable not so much. Its the save_stack_trace() in
> add_xhlock() (IIRC).
> 
> I've not yet had time to figure out what to do about that.

Thanks for confirming. Do you think it makes sense to track this as a
4.14 regression to avoid having others spend time on tracking this down
meanwhile? (Adding Thorsten on CC.)

Johan

Re: Dramatic lockdep slowdown in 4.14

[Johan Hovold]

On Fri, Oct 13, 2017 at 11:07:44AM +0200, Peter Zijlstra wrote:
> On Fri, Oct 13, 2017 at 11:03:33AM +0200, Johan Hovold wrote:
> > Hi,
> > 
> > I had noticed that the BeagleBone Black boot time appeared to have
> > increased significantly with 4.14 and yesterday I finally had time to
> > investigate it.
> > 
> > Boot time (from "Linux version" to login prompt) had in fact doubled
> > since 4.13 where it took 17 seconds (with my current config) compared to
> > the 35 seconds I now see with 4.14-rc4.
> > 
> > I quick bisect pointed to lockdep and specifically the following commit:
> > 
> > 	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
> > 	               of a crosslock")
> > 
> > which I've verified is the commit which doubled the boot time (compared
> > to 28a903f63ec0^) (added by lockdep crossrelease series [1]).
> > 
> > I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
> > brought boot time down to about 14 seconds.
> > 
> > Now since it's lockdep I guess this can't really be considered a
> > regression if these changes did improve lockdep correctness, but still,
> > this dramatic slow down essentially forces me to disable PROVE_LOCKING
> > by default on this system.
> > 
> > Is this lockdep slowdown expected and desirable?
> 
> Expected yes, desirable not so much. Its the save_stack_trace() in
> add_xhlock() (IIRC).
> 
> I've not yet had time to figure out what to do about that.

Thanks for confirming. Do you think it makes sense to track this as a
4.14 regression to avoid having others spend time on tracking this down
meanwhile? (Adding Thorsten on CC.)

Johan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Dramatic lockdep slowdown in 4.14

[Johan Hovold]

On Fri, Oct 13, 2017 at 11:07:44AM +0200, Peter Zijlstra wrote:
> On Fri, Oct 13, 2017 at 11:03:33AM +0200, Johan Hovold wrote:
> > Hi,
> > 
> > I had noticed that the BeagleBone Black boot time appeared to have
> > increased significantly with 4.14 and yesterday I finally had time to
> > investigate it.
> > 
> > Boot time (from "Linux version" to login prompt) had in fact doubled
> > since 4.13 where it took 17 seconds (with my current config) compared to
> > the 35 seconds I now see with 4.14-rc4.
> > 
> > I quick bisect pointed to lockdep and specifically the following commit:
> > 
> > 	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
> > 	               of a crosslock")
> > 
> > which I've verified is the commit which doubled the boot time (compared
> > to 28a903f63ec0^) (added by lockdep crossrelease series [1]).
> > 
> > I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
> > brought boot time down to about 14 seconds.
> > 
> > Now since it's lockdep I guess this can't really be considered a
> > regression if these changes did improve lockdep correctness, but still,
> > this dramatic slow down essentially forces me to disable PROVE_LOCKING
> > by default on this system.
> > 
> > Is this lockdep slowdown expected and desirable?
> 
> Expected yes, desirable not so much. Its the save_stack_trace() in
> add_xhlock() (IIRC).
> 
> I've not yet had time to figure out what to do about that.

Thanks for confirming. Do you think it makes sense to track this as a
4.14 regression to avoid having others spend time on tracking this down
meanwhile? (Adding Thorsten on CC.)

Johan

Re: Dramatic lockdep slowdown in 4.14

[Ingo Molnar]

* Johan Hovold <johan@kernel.org> wrote:

> Hi,
> 
> I had noticed that the BeagleBone Black boot time appeared to have
> increased significantly with 4.14 and yesterday I finally had time to
> investigate it.
> 
> Boot time (from "Linux version" to login prompt) had in fact doubled
> since 4.13 where it took 17 seconds (with my current config) compared to
> the 35 seconds I now see with 4.14-rc4.
> 
> I quick bisect pointed to lockdep and specifically the following commit:
> 
> 	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
> 	               of a crosslock")
> 
> which I've verified is the commit which doubled the boot time (compared
> to 28a903f63ec0^) (added by lockdep crossrelease series [1]).
> 
> I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
> brought boot time down to about 14 seconds.
> 
> Now since it's lockdep I guess this can't really be considered a
> regression if these changes did improve lockdep correctness, but still,
> this dramatic slow down essentially forces me to disable PROVE_LOCKING
> by default on this system.
> 
> Is this lockdep slowdown expected and desirable?

It's not desirable at all.

Does the patch below fix the regression for you - or does the introduction and 
handling of ->nr_acquire hurt as well?

Thanks,

	Ingo

====================>
 lib/Kconfig.debug | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index c6401d325b0e..f5b40c1668ea 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1138,8 +1138,8 @@ config PROVE_LOCKING
 	select DEBUG_MUTEXES
 	select DEBUG_RT_MUTEXES if RT_MUTEXES
 	select DEBUG_LOCK_ALLOC
-	select LOCKDEP_CROSSRELEASE
-	select LOCKDEP_COMPLETIONS
+#	select LOCKDEP_CROSSRELEASE
+#	select LOCKDEP_COMPLETIONS
 	select TRACE_IRQFLAGS
 	default n
 	help

Re: Dramatic lockdep slowdown in 4.14

[Ingo Molnar]

* Johan Hovold <johan@kernel.org> wrote:

> Hi,
> 
> I had noticed that the BeagleBone Black boot time appeared to have
> increased significantly with 4.14 and yesterday I finally had time to
> investigate it.
> 
> Boot time (from "Linux version" to login prompt) had in fact doubled
> since 4.13 where it took 17 seconds (with my current config) compared to
> the 35 seconds I now see with 4.14-rc4.
> 
> I quick bisect pointed to lockdep and specifically the following commit:
> 
> 	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
> 	               of a crosslock")
> 
> which I've verified is the commit which doubled the boot time (compared
> to 28a903f63ec0^) (added by lockdep crossrelease series [1]).
> 
> I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
> brought boot time down to about 14 seconds.
> 
> Now since it's lockdep I guess this can't really be considered a
> regression if these changes did improve lockdep correctness, but still,
> this dramatic slow down essentially forces me to disable PROVE_LOCKING
> by default on this system.
> 
> Is this lockdep slowdown expected and desirable?

It's not desirable at all.

Does the patch below fix the regression for you - or does the introduction and 
handling of ->nr_acquire hurt as well?

Thanks,

	Ingo

====================>
 lib/Kconfig.debug | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index c6401d325b0e..f5b40c1668ea 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1138,8 +1138,8 @@ config PROVE_LOCKING
 	select DEBUG_MUTEXES
 	select DEBUG_RT_MUTEXES if RT_MUTEXES
 	select DEBUG_LOCK_ALLOC
-	select LOCKDEP_CROSSRELEASE
-	select LOCKDEP_COMPLETIONS
+#	select LOCKDEP_CROSSRELEASE
+#	select LOCKDEP_COMPLETIONS
 	select TRACE_IRQFLAGS
 	default n
 	help

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Dramatic lockdep slowdown in 4.14

[Ingo Molnar]

* Johan Hovold <johan@kernel.org> wrote:

> Hi,
> 
> I had noticed that the BeagleBone Black boot time appeared to have
> increased significantly with 4.14 and yesterday I finally had time to
> investigate it.
> 
> Boot time (from "Linux version" to login prompt) had in fact doubled
> since 4.13 where it took 17 seconds (with my current config) compared to
> the 35 seconds I now see with 4.14-rc4.
> 
> I quick bisect pointed to lockdep and specifically the following commit:
> 
> 	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
> 	               of a crosslock")
> 
> which I've verified is the commit which doubled the boot time (compared
> to 28a903f63ec0^) (added by lockdep crossrelease series [1]).
> 
> I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
> brought boot time down to about 14 seconds.
> 
> Now since it's lockdep I guess this can't really be considered a
> regression if these changes did improve lockdep correctness, but still,
> this dramatic slow down essentially forces me to disable PROVE_LOCKING
> by default on this system.
> 
> Is this lockdep slowdown expected and desirable?

It's not desirable at all.

Does the patch below fix the regression for you - or does the introduction and 
handling of ->nr_acquire hurt as well?

Thanks,

	Ingo

====================>
 lib/Kconfig.debug | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index c6401d325b0e..f5b40c1668ea 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1138,8 +1138,8 @@ config PROVE_LOCKING
 	select DEBUG_MUTEXES
 	select DEBUG_RT_MUTEXES if RT_MUTEXES
 	select DEBUG_LOCK_ALLOC
-	select LOCKDEP_CROSSRELEASE
-	select LOCKDEP_COMPLETIONS
+#	select LOCKDEP_CROSSRELEASE
+#	select LOCKDEP_COMPLETIONS
 	select TRACE_IRQFLAGS
 	default n
 	help

Re: Dramatic lockdep slowdown in 4.14

[Johan Hovold]

On Sat, Oct 14, 2017 at 09:26:59AM +0200, Ingo Molnar wrote:
> 
> * Johan Hovold <johan@kernel.org> wrote:
> 
> > Hi,
> > 
> > I had noticed that the BeagleBone Black boot time appeared to have
> > increased significantly with 4.14 and yesterday I finally had time to
> > investigate it.
> > 
> > Boot time (from "Linux version" to login prompt) had in fact doubled
> > since 4.13 where it took 17 seconds (with my current config) compared to
> > the 35 seconds I now see with 4.14-rc4.
> > 
> > I quick bisect pointed to lockdep and specifically the following commit:
> > 
> > 	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
> > 	               of a crosslock")
> > 
> > which I've verified is the commit which doubled the boot time (compared
> > to 28a903f63ec0^) (added by lockdep crossrelease series [1]).
> > 
> > I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
> > brought boot time down to about 14 seconds.
> > 
> > Now since it's lockdep I guess this can't really be considered a
> > regression if these changes did improve lockdep correctness, but still,
> > this dramatic slow down essentially forces me to disable PROVE_LOCKING
> > by default on this system.
> > 
> > Is this lockdep slowdown expected and desirable?
> 
> It's not desirable at all.
> 
> Does the patch below fix the regression for you - or does the introduction and 
> handling of ->nr_acquire hurt as well?

> -	select LOCKDEP_CROSSRELEASE
> -	select LOCKDEP_COMPLETIONS
> +#	select LOCKDEP_CROSSRELEASE
> +#	select LOCKDEP_COMPLETIONS

Disabling these options this way gives me a about boot time of 17
seconds again, so yes, that fixes the problem.

Thanks,
Johan

Re: Dramatic lockdep slowdown in 4.14

[Johan Hovold]

On Sat, Oct 14, 2017 at 09:26:59AM +0200, Ingo Molnar wrote:
> 
> * Johan Hovold <johan@kernel.org> wrote:
> 
> > Hi,
> > 
> > I had noticed that the BeagleBone Black boot time appeared to have
> > increased significantly with 4.14 and yesterday I finally had time to
> > investigate it.
> > 
> > Boot time (from "Linux version" to login prompt) had in fact doubled
> > since 4.13 where it took 17 seconds (with my current config) compared to
> > the 35 seconds I now see with 4.14-rc4.
> > 
> > I quick bisect pointed to lockdep and specifically the following commit:
> > 
> > 	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
> > 	               of a crosslock")
> > 
> > which I've verified is the commit which doubled the boot time (compared
> > to 28a903f63ec0^) (added by lockdep crossrelease series [1]).
> > 
> > I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
> > brought boot time down to about 14 seconds.
> > 
> > Now since it's lockdep I guess this can't really be considered a
> > regression if these changes did improve lockdep correctness, but still,
> > this dramatic slow down essentially forces me to disable PROVE_LOCKING
> > by default on this system.
> > 
> > Is this lockdep slowdown expected and desirable?
> 
> It's not desirable at all.
> 
> Does the patch below fix the regression for you - or does the introduction and 
> handling of ->nr_acquire hurt as well?

> -	select LOCKDEP_CROSSRELEASE
> -	select LOCKDEP_COMPLETIONS
> +#	select LOCKDEP_CROSSRELEASE
> +#	select LOCKDEP_COMPLETIONS

Disabling these options this way gives me a about boot time of 17
seconds again, so yes, that fixes the problem.

Thanks,
Johan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Dramatic lockdep slowdown in 4.14

[Johan Hovold]

On Sat, Oct 14, 2017 at 09:26:59AM +0200, Ingo Molnar wrote:
> 
> * Johan Hovold <johan@kernel.org> wrote:
> 
> > Hi,
> > 
> > I had noticed that the BeagleBone Black boot time appeared to have
> > increased significantly with 4.14 and yesterday I finally had time to
> > investigate it.
> > 
> > Boot time (from "Linux version" to login prompt) had in fact doubled
> > since 4.13 where it took 17 seconds (with my current config) compared to
> > the 35 seconds I now see with 4.14-rc4.
> > 
> > I quick bisect pointed to lockdep and specifically the following commit:
> > 
> > 	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition
> > 	               of a crosslock")
> > 
> > which I've verified is the commit which doubled the boot time (compared
> > to 28a903f63ec0^) (added by lockdep crossrelease series [1]).
> > 
> > I also verified that simply disabling CONFIG_PROVE_LOCKING on 4.14-rc4
> > brought boot time down to about 14 seconds.
> > 
> > Now since it's lockdep I guess this can't really be considered a
> > regression if these changes did improve lockdep correctness, but still,
> > this dramatic slow down essentially forces me to disable PROVE_LOCKING
> > by default on this system.
> > 
> > Is this lockdep slowdown expected and desirable?
> 
> It's not desirable at all.
> 
> Does the patch below fix the regression for you - or does the introduction and 
> handling of ->nr_acquire hurt as well?

> -	select LOCKDEP_CROSSRELEASE
> -	select LOCKDEP_COMPLETIONS
> +#	select LOCKDEP_CROSSRELEASE
> +#	select LOCKDEP_COMPLETIONS

Disabling these options this way gives me a about boot time of 17
seconds again, so yes, that fixes the problem.

Thanks,
Johan

[tip:locking/urgent] locking/lockdep: Disable cross-release features for now

[tip-bot for Ingo Molnar]

Commit-ID:  b483cf3bc249d7af706390efa63d6671e80d1c09
Gitweb:     https://git.kernel.org/tip/b483cf3bc249d7af706390efa63d6671e80d1c09
Author:     Ingo Molnar <mingo@kernel.org>
AuthorDate: Sat, 14 Oct 2017 09:26:59 +0200
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Sat, 14 Oct 2017 12:50:26 +0200

locking/lockdep: Disable cross-release features for now

Johan Hovold reported a big lockdep slowdown on his system, caused by lockdep:

> I had noticed that the BeagleBone Black boot time appeared to have
> increased significantly with 4.14 and yesterday I finally had time to
> investigate it.
>
> Boot time (from "Linux version" to login prompt) had in fact doubled
> since 4.13 where it took 17 seconds (with my current config) compared to
> the 35 seconds I now see with 4.14-rc4.
>
> I quick bisect pointed to lockdep and specifically the following commit:
>
>	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition of a crosslock")

Because the final v4.14 release is close, disable the cross-release lockdep
features for now.

Bisected-by: Johan Hovold <johan@kernel.org>
Debugged-by: Johan Hovold <johan@kernel.org>
Reported-by: Johan Hovold <johan@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Byungchul Park <byungchul.park@lge.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Lindgren <tony@atomide.com>
Cc: kernel-team@lge.com
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-mm@kvack.org
Cc: linux-omap@vger.kernel.org
Link: http://lkml.kernel.org/r/20171014072659.f2yr6mhm5ha3eou7@gmail.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 lib/Kconfig.debug | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 2689b7c..e270584 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1092,8 +1092,8 @@ config PROVE_LOCKING
 	select DEBUG_MUTEXES
 	select DEBUG_RT_MUTEXES if RT_MUTEXES
 	select DEBUG_LOCK_ALLOC
-	select LOCKDEP_CROSSRELEASE
-	select LOCKDEP_COMPLETIONS
+	select LOCKDEP_CROSSRELEASE if BROKEN
+	select LOCKDEP_COMPLETIONS if BROKEN
 	select TRACE_IRQFLAGS
 	default n
 	help

Re: [tip:locking/urgent] locking/lockdep: Disable cross-release features for now

[Byungchul Park]

On Sat, Oct 14, 2017 at 04:36:51AM -0700, tip-bot for Ingo Molnar wrote:
> Commit-ID:  b483cf3bc249d7af706390efa63d6671e80d1c09
> Gitweb:     https://git.kernel.org/tip/b483cf3bc249d7af706390efa63d6671e80d1c09
> Author:     Ingo Molnar <mingo@kernel.org>
> AuthorDate: Sat, 14 Oct 2017 09:26:59 +0200
> Committer:  Ingo Molnar <mingo@kernel.org>
> CommitDate: Sat, 14 Oct 2017 12:50:26 +0200
> 
> locking/lockdep: Disable cross-release features for now
> 
> Johan Hovold reported a big lockdep slowdown on his system, caused by lockdep:
> 
> > I had noticed that the BeagleBone Black boot time appeared to have
> > increased significantly with 4.14 and yesterday I finally had time to
> > investigate it.
> >
> > Boot time (from "Linux version" to login prompt) had in fact doubled
> > since 4.13 where it took 17 seconds (with my current config) compared to
> > the 35 seconds I now see with 4.14-rc4.
> >
> > I quick bisect pointed to lockdep and specifically the following commit:
> >
> >	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition of a crosslock")
> 
> Because the final v4.14 release is close, disable the cross-release lockdep
> features for now.
> 
> Bisected-by: Johan Hovold <johan@kernel.org>
> Debugged-by: Johan Hovold <johan@kernel.org>
> Reported-by: Johan Hovold <johan@kernel.org>
> Cc: Arnd Bergmann <arnd@arndb.de>
> Cc: Byungchul Park <byungchul.park@lge.com>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Tony Lindgren <tony@atomide.com>
> Cc: kernel-team@lge.com
> Cc: linux-arm-kernel@lists.infradead.org
> Cc: linux-mm@kvack.org
> Cc: linux-omap@vger.kernel.org
> Link: http://lkml.kernel.org/r/20171014072659.f2yr6mhm5ha3eou7@gmail.com
> Signed-off-by: Ingo Molnar <mingo@kernel.org>
> ---
>  lib/Kconfig.debug | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> index 2689b7c..e270584 100644
> --- a/lib/Kconfig.debug
> +++ b/lib/Kconfig.debug
> @@ -1092,8 +1092,8 @@ config PROVE_LOCKING
>  	select DEBUG_MUTEXES
>  	select DEBUG_RT_MUTEXES if RT_MUTEXES
>  	select DEBUG_LOCK_ALLOC
> -	select LOCKDEP_CROSSRELEASE
> -	select LOCKDEP_COMPLETIONS
> +	select LOCKDEP_CROSSRELEASE if BROKEN
> +	select LOCKDEP_COMPLETIONS if BROKEN

I agree with disabling crossrelease as default, becasue of regression,
as I originally did.

However, it's expected to spend more time once it's enabled. Is the
following acceptable?

----->8-----
>From d796b36bbed5783a22879d745e4d332bb02697a5 Mon Sep 17 00:00:00 2001
From: Byungchul Park <byungchul.park@lge.com>
Date: Mon, 16 Oct 2017 10:54:20 +0900
Subject: [PATCH] lockdep: Diable LOCKDEP_CROSSRELEASE as default

Lockdep slowdeon has been reported becasue of LOCKDEP_CROSSRELEASE. The
feature needs more time to work, but it's not desirable as default. So
disable the Kconfig as default.

Signed-off-by: Byungchul Park <byungchul.park@lge.com>
---
 lib/Kconfig.debug | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 3db9167..fa45f78 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1138,8 +1138,6 @@ config PROVE_LOCKING
 	select DEBUG_MUTEXES
 	select DEBUG_RT_MUTEXES if RT_MUTEXES
 	select DEBUG_LOCK_ALLOC
-	select LOCKDEP_CROSSRELEASE if BROKEN
-	select LOCKDEP_COMPLETIONS if BROKEN
 	select TRACE_IRQFLAGS
 	default n
 	help
@@ -1210,7 +1208,9 @@ config LOCK_STAT
 	 (CONFIG_LOCKDEP defines "acquire" and "release" events.)
 
 config LOCKDEP_CROSSRELEASE
-	bool
+	bool "Lock debugging: enable cross-locking checks in lockdep"
+	depends on PROVE_LOCKING
+	default n
 	help
 	 This makes lockdep work for crosslock which is a lock allowed to
 	 be released in a different context from the acquisition context.
@@ -1220,7 +1220,9 @@ config LOCKDEP_CROSSRELEASE
 	 detector, lockdep.
 
 config LOCKDEP_COMPLETIONS
-	bool
+	bool "Lock debugging: allow completions to use deadlock detector"
+	depends on LOCKDEP_CROSSRELEASE
+	default n
 	help
 	 A deadlock caused by wait_for_completion() and complete() can be
 	 detected by lockdep using crossrelease feature.
-- 
1.9.1

Re: [tip:locking/urgent] locking/lockdep: Disable cross-release features for now

[Ingo Molnar]

* Byungchul Park <byungchul.park@lge.com> wrote:

> On Sat, Oct 14, 2017 at 04:36:51AM -0700, tip-bot for Ingo Molnar wrote:
> > Commit-ID:  b483cf3bc249d7af706390efa63d6671e80d1c09
> > Gitweb:     https://git.kernel.org/tip/b483cf3bc249d7af706390efa63d6671e80d1c09
> > Author:     Ingo Molnar <mingo@kernel.org>
> > AuthorDate: Sat, 14 Oct 2017 09:26:59 +0200
> > Committer:  Ingo Molnar <mingo@kernel.org>
> > CommitDate: Sat, 14 Oct 2017 12:50:26 +0200
> > 
> > locking/lockdep: Disable cross-release features for now
> > 
> > Johan Hovold reported a big lockdep slowdown on his system, caused by lockdep:
> > 
> > > I had noticed that the BeagleBone Black boot time appeared to have
> > > increased significantly with 4.14 and yesterday I finally had time to
> > > investigate it.
> > >
> > > Boot time (from "Linux version" to login prompt) had in fact doubled
> > > since 4.13 where it took 17 seconds (with my current config) compared to
> > > the 35 seconds I now see with 4.14-rc4.
> > >
> > > I quick bisect pointed to lockdep and specifically the following commit:
> > >
> > >	28a903f63ec0 ("locking/lockdep: Handle non(or multi)-acquisition of a crosslock")
> > 
> > Because the final v4.14 release is close, disable the cross-release lockdep
> > features for now.
> > 
> > Bisected-by: Johan Hovold <johan@kernel.org>
> > Debugged-by: Johan Hovold <johan@kernel.org>
> > Reported-by: Johan Hovold <johan@kernel.org>
> > Cc: Arnd Bergmann <arnd@arndb.de>
> > Cc: Byungchul Park <byungchul.park@lge.com>
> > Cc: Linus Torvalds <torvalds@linux-foundation.org>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Tony Lindgren <tony@atomide.com>
> > Cc: kernel-team@lge.com
> > Cc: linux-arm-kernel@lists.infradead.org
> > Cc: linux-mm@kvack.org
> > Cc: linux-omap@vger.kernel.org
> > Link: http://lkml.kernel.org/r/20171014072659.f2yr6mhm5ha3eou7@gmail.com
> > Signed-off-by: Ingo Molnar <mingo@kernel.org>
> > ---
> >  lib/Kconfig.debug | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> > index 2689b7c..e270584 100644
> > --- a/lib/Kconfig.debug
> > +++ b/lib/Kconfig.debug
> > @@ -1092,8 +1092,8 @@ config PROVE_LOCKING
> >  	select DEBUG_MUTEXES
> >  	select DEBUG_RT_MUTEXES if RT_MUTEXES
> >  	select DEBUG_LOCK_ALLOC
> > -	select LOCKDEP_CROSSRELEASE
> > -	select LOCKDEP_COMPLETIONS
> > +	select LOCKDEP_CROSSRELEASE if BROKEN
> > +	select LOCKDEP_COMPLETIONS if BROKEN
> 
> I agree with disabling crossrelease as default, becasue of regression,
> as I originally did.
> 
> However, it's expected to spend more time once it's enabled. Is the
> following acceptable?

No, please fix performance.

Thanks,

	Ingo

Re: [tip:locking/urgent] locking/lockdep: Disable cross-release features for now

[Thomas Gleixner]

On Tue, 17 Oct 2017, Ingo Molnar wrote:
> * Byungchul Park <byungchul.park@lge.com> wrote:
> > > diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> > > index 2689b7c..e270584 100644
> > > --- a/lib/Kconfig.debug
> > > +++ b/lib/Kconfig.debug
> > > @@ -1092,8 +1092,8 @@ config PROVE_LOCKING
> > >  	select DEBUG_MUTEXES
> > >  	select DEBUG_RT_MUTEXES if RT_MUTEXES
> > >  	select DEBUG_LOCK_ALLOC
> > > -	select LOCKDEP_CROSSRELEASE
> > > -	select LOCKDEP_COMPLETIONS
> > > +	select LOCKDEP_CROSSRELEASE if BROKEN
> > > +	select LOCKDEP_COMPLETIONS if BROKEN
> > 
> > I agree with disabling crossrelease as default, becasue of regression,
> > as I originally did.
> > 
> > However, it's expected to spend more time once it's enabled. Is the
> > following acceptable?
> 
> No, please fix performance.

You know very well that with the cross release stuff we have to take the
performance hit of stack unwinding because we have no idea whether there
will show up a new lock relation later or not. And there is not much you
can do in that respect.

OTOH, the cross release feature unearthed real deadlocks already so it is a
valuable debug feature and having an explicit config switch which defaults
to N is well worth it.

Thanks,

	tglx

Re: [tip:locking/urgent] locking/lockdep: Disable cross-release features for now

[Ingo Molnar]

* Thomas Gleixner <tglx@linutronix.de> wrote:

> On Tue, 17 Oct 2017, Ingo Molnar wrote:
> > * Byungchul Park <byungchul.park@lge.com> wrote:
> > > > diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> > > > index 2689b7c..e270584 100644
> > > > --- a/lib/Kconfig.debug
> > > > +++ b/lib/Kconfig.debug
> > > > @@ -1092,8 +1092,8 @@ config PROVE_LOCKING
> > > >  	select DEBUG_MUTEXES
> > > >  	select DEBUG_RT_MUTEXES if RT_MUTEXES
> > > >  	select DEBUG_LOCK_ALLOC
> > > > -	select LOCKDEP_CROSSRELEASE
> > > > -	select LOCKDEP_COMPLETIONS
> > > > +	select LOCKDEP_CROSSRELEASE if BROKEN
> > > > +	select LOCKDEP_COMPLETIONS if BROKEN
> > > 
> > > I agree with disabling crossrelease as default, becasue of regression,
> > > as I originally did.
> > > 
> > > However, it's expected to spend more time once it's enabled. Is the
> > > following acceptable?
> > 
> > No, please fix performance.
> 
> You know very well that with the cross release stuff we have to take the
> performance hit of stack unwinding because we have no idea whether there
> will show up a new lock relation later or not. And there is not much you
> can do in that respect.
> 
> OTOH, the cross release feature unearthed real deadlocks already so it is a
> valuable debug feature and having an explicit config switch which defaults
> to N is well worth it.

I disagree, because even if that's correct, the choices are not binary. The 
performance regression was a slowdown of around 7x: lockdep boot overhead on that 
particula system went from +3 seconds to +21 seconds...

As a response to the performance regression I haven't seen _any_ attempt to 
measure, profile and generally quantify the performance impact, which would at 
least make it more believable that the overhead cannot be reduced. That really 
makes me worry about the code on a higher level than just whether it can be 
enabled by default or not.

Caring about the performance of debug features very much matters, _especially_ 
when they are expensive.

Thanks,

	Ingo

Re: [tip:locking/urgent] locking/lockdep: Disable cross-release features for now

[Thomas Gleixner]

On Tue, 17 Oct 2017, Ingo Molnar wrote:
> * Thomas Gleixner <tglx@linutronix.de> wrote:
> > On Tue, 17 Oct 2017, Ingo Molnar wrote:
> > > No, please fix performance.
> > 
> > You know very well that with the cross release stuff we have to take the
> > performance hit of stack unwinding because we have no idea whether there
> > will show up a new lock relation later or not. And there is not much you
> > can do in that respect.
> > 
> > OTOH, the cross release feature unearthed real deadlocks already so it is a
> > valuable debug feature and having an explicit config switch which defaults
> > to N is well worth it.
> 
> I disagree, because even if that's correct, the choices are not binary. The 
> performance regression was a slowdown of around 7x: lockdep boot overhead on that 
> particula system went from +3 seconds to +21 seconds...

Hmm, I might have missed something, but what I've seen in this thread is:

> > > Boot time (from "Linux version" to login prompt) had in fact doubled
> > > since 4.13 where it took 17 seconds (with my current config) compared to
> > > the 35 seconds I now see with 4.14-rc4.

So that's 2x not 7x. On one of my main test machines it's about ~1.4 so I
did not even really notice until this thread came up. Probably I have no
expectations on boot time and performance when lockdep is on :)

> As a response to the performance regression I haven't seen _any_ attempt to 
> measure, profile and generally quantify the performance impact, which would at 
> least make it more believable that the overhead cannot be reduced. That really 
> makes me worry about the code on a higher level than just whether it can be 
> enabled by default or not.

I did some quick perf top analysis, not in detail though, and what really
dominates with that feature is the unwinder, which needs to be
unconditional due to the nature of the problem.

I have not spend a huge amount of time to think about ways to improve that,
but I could not come up with anything smart so far.

The only thing I thought about was making the unwind short and only record
one or two call levels (if at all) instead of following the full call
chain. That makes it less useful for a quick test, but once you hit a splat
you can enable full depth recording for full analysis. In the full analysis
case performance is the least of your worries.

> Caring about the performance of debug features very much matters, _especially_ 
> when they are expensive.

I'm not disagreeing. I'm just trying to understand why this is marked
BROKEN where I think it should be marked TOO_EXPENSIVE.

Thanks,

	tglx

Re: [tip:locking/urgent] locking/lockdep: Disable cross-release features for now

[Ingo Molnar]

* Thomas Gleixner <tglx@linutronix.de> wrote:

> On Tue, 17 Oct 2017, Ingo Molnar wrote:
> > * Thomas Gleixner <tglx@linutronix.de> wrote:
> > > On Tue, 17 Oct 2017, Ingo Molnar wrote:
> > > > No, please fix performance.
> > > 
> > > You know very well that with the cross release stuff we have to take the
> > > performance hit of stack unwinding because we have no idea whether there
> > > will show up a new lock relation later or not. And there is not much you
> > > can do in that respect.
> > > 
> > > OTOH, the cross release feature unearthed real deadlocks already so it is a
> > > valuable debug feature and having an explicit config switch which defaults
> > > to N is well worth it.
> > 
> > I disagree, because even if that's correct, the choices are not binary. The 
> > performance regression was a slowdown of around 7x: lockdep boot overhead on that 
> > particula system went from +3 seconds to +21 seconds...
> 
> Hmm, I might have missed something, but what I've seen in this thread is:
> 
> > > > Boot time (from "Linux version" to login prompt) had in fact doubled
> > > > since 4.13 where it took 17 seconds (with my current config) compared to
> > > > the 35 seconds I now see with 4.14-rc4.
> 
> So that's 2x not 7x. [...]

Yeah, so what you missed I think is that the no-lockdep bootup time is 14 seconds. 
So we have:

   vanilla:                        14 secs
   lockdep:                        17 secs (+3 secs)
   lockdep+crossrelease:           35 secs (+21 secs)

So lockdep overhead got 7x worse on this system.

Thanks,

	Ingo

Re: [tip:locking/urgent] locking/lockdep: Disable cross-release features for now

[Byungchul Park]

On Tue, Oct 17, 2017 at 09:12:02AM +0200, Ingo Molnar wrote:
> > > diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> > > index 2689b7c..e270584 100644
> > > --- a/lib/Kconfig.debug
> > > +++ b/lib/Kconfig.debug
> > > @@ -1092,8 +1092,8 @@ config PROVE_LOCKING
> > >  	select DEBUG_MUTEXES
> > >  	select DEBUG_RT_MUTEXES if RT_MUTEXES
> > >  	select DEBUG_LOCK_ALLOC
> > > -	select LOCKDEP_CROSSRELEASE
> > > -	select LOCKDEP_COMPLETIONS
> > > +	select LOCKDEP_CROSSRELEASE if BROKEN
> > > +	select LOCKDEP_COMPLETIONS if BROKEN
> > 
> > I agree with disabling crossrelease as default, becasue of regression,
> > as I originally did.
> > 
> > However, it's expected to spend more time once it's enabled. Is the
> > following acceptable?
> 
> No, please fix performance.

OK. I will fx the performance problem by making the unwinding optional.

Re: [tip:locking/urgent] locking/lockdep: Disable cross-release features for now

[Byungchul Park]

On Tue, Oct 17, 2017 at 05:03:40PM +0200, Thomas Gleixner wrote:
> On Tue, 17 Oct 2017, Ingo Molnar wrote:
> > * Thomas Gleixner <tglx@linutronix.de> wrote:
> > > On Tue, 17 Oct 2017, Ingo Molnar wrote:
> > > > No, please fix performance.
> > > 
> > > You know very well that with the cross release stuff we have to take the
> > > performance hit of stack unwinding because we have no idea whether there
> > > will show up a new lock relation later or not. And there is not much you
> > > can do in that respect.
> > > 
> > > OTOH, the cross release feature unearthed real deadlocks already so it is a
> > > valuable debug feature and having an explicit config switch which defaults
> > > to N is well worth it.
> > 
> > I disagree, because even if that's correct, the choices are not binary. The 
> > performance regression was a slowdown of around 7x: lockdep boot overhead on that 
> > particula system went from +3 seconds to +21 seconds...
> 
> Hmm, I might have missed something, but what I've seen in this thread is:
> 
> > > > Boot time (from "Linux version" to login prompt) had in fact doubled
> > > > since 4.13 where it took 17 seconds (with my current config) compared to
> > > > the 35 seconds I now see with 4.14-rc4.
> 
> So that's 2x not 7x. On one of my main test machines it's about ~1.4 so I
> did not even really notice until this thread came up. Probably I have no
> expectations on boot time and performance when lockdep is on :)
> 
> > As a response to the performance regression I haven't seen _any_ attempt to 
> > measure, profile and generally quantify the performance impact, which would at 
> > least make it more believable that the overhead cannot be reduced. That really 
> > makes me worry about the code on a higher level than just whether it can be 
> > enabled by default or not.
> 
> I did some quick perf top analysis, not in detail though, and what really
> dominates with that feature is the unwinder, which needs to be
> unconditional due to the nature of the problem.
> 
> I have not spend a huge amount of time to think about ways to improve that,
> but I could not come up with anything smart so far.
> 
> The only thing I thought about was making the unwind short and only record
> one or two call levels (if at all) instead of following the full call

Yes, I think that's the best option I can do.

Thank you very much.

> chain. That makes it less useful for a quick test, but once you hit a splat
> you can enable full depth recording for full analysis. In the full analysis
> case performance is the least of your worries.
> 
> > Caring about the performance of debug features very much matters, _especially_ 
> > when they are expensive.
> 
> I'm not disagreeing. I'm just trying to understand why this is marked
> BROKEN where I think it should be marked TOO_EXPENSIVE.
> 
> Thanks,
> 
> 	tglx