Detailed Stack Information Patch [0/3]

Detailed Stack Information Patch [0/3]

[Stefani Seibold]

Hi,

this is a patch which give you a better overview of the userland
application stack usage, especially for embedded linux.

Currently you are only able to dump the main process/thread stack usage
which is showed in proc/pid/status by the "VmStk" Value. But you get no
information about the consumed stack memory of the the threads.

For some reasons it becomes important to know how much memory is
consumed by each thread:

- Get out of virtual memory by creating a lot of threads
 (f.e. the developer did assign each of them the default size)
- Misuse the thread stack for big temporary data buffers
- Thread stack overruns

So this patch gives the developer an important tool to figure out if there
is one of this issues.

The patch is splitted in three parts.

Part 1 :
--------

 fs/exec.c             |    4 ++++
 fs/proc/array.c       |   22 ++++++++++++++++++++++
 fs/proc/task_mmu.c    |   12 ++++++++++++
 include/linux/sched.h |    3 +++
 init/Kconfig          |   12 ++++++++++++
 kernel/fork.c         |    5 +++++
 6 files changed, 58 insertions(+)

This is an enhancement in the /proc/<pid>/tasks/<tid>/maps and
smaps which marks the mapping where the thread stack pointer reside with
"[thread stack]".

Also there is a new entry "stack usage" in proc/pid/status, which will
you give the current stack usage.

This feature will be enabled the which "enable /proc/<pid> stack
monitoring" under "General setup-->"


Part 2:
-------

 fs/proc/Makefile   |    1 
 fs/proc/stackmon.c |  254 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 init/Kconfig       |   10 ++
 3 files changed, 265 insertions(+)

This enable a new /proc/stackmon file entry. A cat /proc/stackmon
will produce a output like:

   bytes   pages maxpages vm_start vm_end   processid threadid  name
     436       1       1  afdbf000-afdd4000 pid:  409 tid:  409 syslogd
    1168       1       1  afd12000-afd27000 pid:  411 tid:  411 sh
     516       1       1  afe6c000-afe81000 pid:  412 tid:  412 getty
    4580       2       2  af918000-af92d000 pid:  419 tid:  419 cat
The first value is the current effektive stack usage in bytes.

The second value is the current real stack usage in pages. This means
how many pages are really occupied by the stack.

The thrird value is the maximum real stack usage in pages. This value
will be determinate by the difference of the highest used page in the
mapping and the page of stack start address.

The fourth value is the start- and end- address of the mapping where the
stack currently reside.

The fifth value process id.

The sixth value is thread id.

And the seventh and last entry is the name of the process.

This feature will be enabled the which "enable /proc/<pid> stack
monitoring" under "General setup-->".


Part 3:
-------

 init/Kconfig  |   13 ++
 mm/Makefile   |    1 
 mm/stackdbg.c |  319 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 332 insertions(+)

There is also an additional stack monitor which can be enabled by boot
time or by the /sys filesystem. Which this you are able to detect if a
application use more stack than a given value. If the application
exceeds this value, it will receive a SIGTRAP signal. 

If there is a debugger attached at this time, there is the ability to
examinate the stack usage. Otherwise the application will be terminated.
In both cases a kernel log entry "pid:%d (%s) tid:%d stack size %lu
exceeds max stack size." will be written.

There are following entries under /sys/kernel/stackmon to control the
monitor.

mode:
 Setting this to an value not equal zero will start the stack monitoring
thread. Default is 0. Setting it to zero will stop the kernel thread.

stacksize:
 This value is the stack size in kb which triggers a SIGTRAP to the
application, if the stack usage is equal or more. Default is 256 kb.
 
ticks:
 Number of ticks between the monitoring invocation. A higher value will
give a less change to trigger a stack over usage, but will also result
in a less CPU usage. Default is 1.

All this parameters can also set at boot time with the kernel parameter
"stackmon=<stacksize>:<mode>:<ticks>".

The stack monitor can also be compiled as a module.



This patch is against 2.6.29. The patch is cpu independent, so it
should work on all linux supported architectures, it was tested under
x86 and powerpc. There is no dependency to a library: glibc, uclibc
and all other should work.

I hope you like it and want ask for inclusion into the kernel or linux-next? 
Please give it a try.

If you have ideas how to do things in a better way, please let me know.

Have a nice day,
Stefani

Re: Detailed Stack Information Patch [0/3]

[Andi Kleen]

Stefani Seibold <stefani@seibold.net> writes:
>
> - Get out of virtual memory by creating a lot of threads
>  (f.e. the developer did assign each of them the default size)

The application just fails then? I don't think that needs
a new monitoring tool.

> - Misuse the thread stack for big temporary data buffers

That would be better checked for at compile time
(except for alloca, but that is quite rare)

> - Thread stack overruns

Your method would be racy at best to determine this because
you don't keep track of the worst case, only the current case.

So e.g. if you monitoring app checks once per second the stack
could overflow between your monitoring intervals, but already
have bounced back before the checker comes in.

gcc has support to generate stack overflow checking code,
that would be more reliable. Alternatively you could keep
track of consumption in the VMA that has the stack,  but
that can't handle very large jumps (like f() { char x[1<<30]; } )
The later can only be handled well by the compiler.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: Detailed Stack Information Patch [0/3]

[Stefani Seibold]

Hi Andi,

Am Dienstag, den 31.03.2009, 17:49 +0200 schrieb Andi Kleen:
> Stefani Seibold <stefani@seibold.net> writes:
> >
> > - Get out of virtual memory by creating a lot of threads
> >  (f.e. the developer did assign each of them the default size)
> 
> The application just fails then? I don't think that needs
> a new monitoring tool.
> 

First, this patch is not only a monitoring tool. Only the last part 3/3
is the monitoring tool.

Patch 1/3 enhance the the proc/<pid>/task/<tid>/maps by the marking the
thread stack.

Patch 2/3 gives you an overview of the current process/thread stack
usage with the /proc/stackmon entry.

> > - Misuse the thread stack for big temporary data buffers
> 
> That would be better checked for at compile time
> (except for alloca, but that is quite rare)

Fine but it did not work for functions like:

void foo(int n)
{
	char buf[n*1024];

}

This is valid with gcc.

> 
> > - Thread stack overruns
> 
> Your method would be racy at best to determine this because
> you don't keep track of the worst case, only the current case.
> 
> So e.g. if you monitoring app checks once per second the stack
> could overflow between your monitoring intervals, but already
> have bounced back before the checker comes in.
> 

The Monitor is part 3/3. And you are right it is not a complete rock
solid solution. But it works in many cases and thats is what counts.

> Alternatively you could keep
> track of consumption in the VMA that has the stack,  but
> that can't handle very large jumps (like f() { char x[1<<30]; } )
> The later can only be handled well by the compiler.

Thats is exactly what i am doing, i walk through the pages of the thread
stack mapped memory and keep track of the highest access page. So i have
the high water mark of the used stack.

The patches are not intrusive, especially part 1.

> 

Stefani

Re: Detailed Stack Information Patch [0/3]

[Jörn Engel]

On Tue, 31 March 2009 20:22:15 +0200, Stefani Seibold wrote:
> Am Dienstag, den 31.03.2009, 17:49 +0200 schrieb Andi Kleen:
> > Stefani Seibold <stefani@seibold.net> writes:
> 
> > > - Misuse the thread stack for big temporary data buffers
> > 
> > That would be better checked for at compile time
> > (except for alloca, but that is quite rare)
> 
> Fine but it did not work for functions like:
> 
> void foo(int n)
> {
> 	char buf[n*1024];
> 
> }
> 
> This is valid with gcc.

Good call.  checkstack should look for those as well.  It is certainly
possible to detect statically and warn about:

  10:   29 c4                   sub    %eax,%esp

Runaway recursions are a different matter, though.  The code I once had
to detect them depends on an old version of smatch, which in turn
depends on gcc 3.1.  And even assuming this was in a reasonable shape, I
still don't know what to do about it.  The kernel has thousands of
recursions and trying to work out how deep each one may stack is a
never-ending project.

Jörn

-- 
A quarrel is quickly settled when deserted by one party; there is
no battle unless there be two.
-- Seneca

Re: Detailed Stack Information Patch [0/3]

[Andi Kleen]

On Tue, Mar 31, 2009 at 08:22:15PM +0200, Stefani Seibold wrote:
> Hi Andi,
> 
> Am Dienstag, den 31.03.2009, 17:49 +0200 schrieb Andi Kleen:
> > Stefani Seibold <stefani@seibold.net> writes:
> > >
> > > - Get out of virtual memory by creating a lot of threads
> > >  (f.e. the developer did assign each of them the default size)
> > 
> > The application just fails then? I don't think that needs
> > a new monitoring tool.
> > 
> 
> First, this patch is not only a monitoring tool. Only the last part 3/3
> is the monitoring tool.
> 
> Patch 1/3 enhance the the proc/<pid>/task/<tid>/maps by the marking the
> thread stack.

Well some implementation of it. There are certainly runtimes that
switch stacks. For example what happens when someone uses sigaltstack()?

You'll probably need some more sanity checks, otherwise a monitor
will occasionally just report crap in legitimate cases. Unfortunately
it's difficult to distingush the legitimate cases from the bugs.

> > > - Misuse the thread stack for big temporary data buffers
> > 
> > That would be better checked for at compile time
> > (except for alloca, but that is quite rare)
> 
> Fine but it did not work for functions like:

That's the alloca() case, but you can disable both with the right options.
There's still the "recursive function" case.

> > > - Thread stack overruns
> > 
> > Your method would be racy at best to determine this because
> > you don't keep track of the worst case, only the current case.
> > 
> > So e.g. if you monitoring app checks once per second the stack
> > could overflow between your monitoring intervals, but already
> > have bounced back before the checker comes in.
> > 
> 
> The Monitor is part 3/3. And you are right it is not a complete rock
> solid solution. But it works in many cases and thats is what counts.

For stack overflow one would think a rock solid solution
is needed?  After all you'll crash if you miss a case.

> > track of consumption in the VMA that has the stack,  but
> > that can't handle very large jumps (like f() { char x[1<<30]; } )
> > The later can only be handled well by the compiler.
> 
> Thats is exactly what i am doing, i walk through the pages of the thread
> stack mapped memory and keep track of the highest access page. So i have
> the high water mark of the used stack.

Ok. Of course it doesn't work for really large allocations. there used
to be special patches around to add a special gap to catch those,
but it's problematic with the tight address space on 32bit and also
again not fully bullet proof. On 64bit the solution is typical
to just use a large stack.

> The patches are not intrusive, especially part 1.

To be honest it seems too much like a special case hack to me
to include by default. It could be probably done with a systemtap
script in the same way, but I would really recommend to just
build with gcc's stack overflow checker while testing together
with static checking.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: Detailed Stack Information Patch [0/3]

[Stefani Seibold]

Hi Andi,

stop complaining about the monitor. This is only an additional
functionality.

The main purpose are part 1 and 2.

> 
> Well some implementation of it. There are certainly runtimes that
> switch stacks. For example what happens when someone uses sigaltstack()?
> 

What should happen with sigaltstack? This is complete independent from
the process and thread stack. So it works.


> That's the alloca() case, but you can disable both with the right options.
> There's still the "recursive function" case.
> 

And no idea ;-) 

> > 
> > The Monitor is part 3/3. And you are right it is not a complete rock
> > solid solution. But it works in many cases and thats is what counts.
> 
> For stack overflow one would think a rock solid solution
> is needed?  After all you'll crash if you miss a case.
> 

Again, the monitor is the only a part of the patch and i know that this
is a issue.

The first two patches will also work without the monitor and if you
don't like the monitor, no problem. It is a CONFIG_... parameter.

> To be honest it seems too much like a special case hack to me
> to include by default. It could be probably done with a systemtap
> script in the same way, but I would really recommend to just
> build with gcc's stack overflow checker while testing together
> with static checking.
> 

Thanks for the hack - I am not sure if you really had a look at my first
posting nor had a look into my code.

We discus about complete different things. You have from user land no
possibility to figure out where is the thread stack locate nor what was
the highest used thread stack address.

That is a simple debug information which can provide very easily which
the first two patches.

Stefani

Re: Detailed Stack Information Patch [0/3]

[Jörn Engel]

http://en.wikipedia.org/wiki/Posting_style

On Thu, 22 January 2009 22:39:28 +0100, Stefani Seibold wrote:
> 
> First, i had explained what is the reason for the patch.

You did explain your solution.  The problem you are trying to solve was
not clear to me.  It may be my mistake.  If you have the stack
information, what can you do that you couldn't do without?  Notice
overly large stacks and fix the application?  I would guess so, but you
surely know better than me.

> Second, the number of #ifdef is not more or less than other features
> which extends the task_struct on demand.
> 
> There is no way to do this without #ifdef's, only if i add this feature
> without a CONFIG_.... option.

True.  The comment was not about removing them completely but moving
them where they hurt less.  When I'm reading through some generic code
and I read something like this:

	f->f_pos = 0;
	f->f_op = fops_get(inode->i_fop);
	file_move(f, &inode->i_sb->s_files);

	error = security_dentry_open(f);
	if (error)
		goto cleanup_all;

I only get slightly distracted by the security_dentry_open() that I
personally may not care about one bit.  Much nicer than having this:

	f->f_pos = 0;
	f->f_op = fops_get(inode->i_fop);
	file_move(f, &inode->i_sb->s_files);

#ifdef CONFIG_SECURITY
	error = security_dentry_open(f);
	if (error)
		goto cleanup_all;
#endif

See the difference?

> But this patch did nit solve a problem, it is a feature like
> PROC_PAGE_MONITOR or similar. It will help to figure out, how much stack
> will be consumed by a particular process or thread.
> 
> It also not a patch which cares Joe Kernelhacker, it cares Jane
> Userlandhacker!

Fair enough.  So assuming I wanted to reduce or limit stack consumption,
what would I do with this information.  I know the responsible process,
but not the callchain.  kill -QUIT 12345 would give me a core dump and I
could analyse that.

Is that what you try to achieve?  I was only guessing and may have
missed a more important point.

Jörn

-- 
Everything should be made as simple as possible, but not simpler.
-- Albert Einstein

Re: Detailed Stack Information Patch [0/3]

[Stefani Seibold]

First, i had explained what is the reason for the patch.

Second, the number of #ifdef is not more or less than other features
which extends the task_struct on demand.

There is no way to do this without #ifdef's, only if i add this feature
without a CONFIG_.... option.

I think the main reason why i get low responses is, because i posted it
to the wrong mail list. kernel-mm would be a better place for this.

So i will wait until 2.6.29 is out, then move my patch to this version,
enhance it, add a diffstat and try it again which are more detailed
reason why this patch should be included.

But this patch did nit solve a problem, it is a feature like
PROC_PAGE_MONITOR or similar. It will help to figure out, how much stack
will be consumed by a particular process or thread.

It also not a patch which cares Joe Kernelhacker, it cares Jane
Userlandhacker!

Thnx,
Steffi

Am Donnerstag, den 22.01.2009, 20:41 +0100 schrieb Jörn Engel:
> On Tue, 20 January 2009 11:16:37 +0100, Stefani Seibold wrote:
> > 
> > this is a patch which give you a better overview of the userland
> > application stack usage, especially for embedded linux.
> > 
> > Currently you are only able to dump the main process/thread stack usage
> > which is showed in proc/pid/status by the "VmStk" Value. But you get no
> > information about the consumed stack memory of the the threads.
> >
> > [...]
> > 
> > This patch is against 2.6.28.1. The patch is cpu independent, so it
> > should work on all linux supported architectures, it was tested under
> > x86 and powerpc. Also there is not dependency a library: glibc, uclibc
> > and all other should work.
> > 
> > I hope you like it and want ask what is necessary for inclusion into the
> > main stream kernel or linux-next? If you have ideas how to do things in
> > a better way, please let me know.
> 
> First goal would be to get people interested.  Why would Joe
> Kernelhacker care about this, what problem would it solve for him?  Next
> goal is to prove to akpm that the solved problems are worth the
> maintenance burden this code brings.
> 
> It would be nice to have diffstat added to each patch to give people
> a quick overview.  More importantly, the number of #ifdef's in the
> patches may raise a red flag.  You should try to remove them from common
> code and have a single one in the headers:
> 
> #ifdef CONFIG_NEW_FEATURE
> 
> void handle_this(int foo, long bar);
> 
> #else
> 
> static inline void handle_this(int foo, long bar)
> {
> }
> #endif
> 
> Not sure what else to say.  I'm still wondering whether it will solve a
> problem for me.
> 
> Jörn
> 

Re: Detailed Stack Information Patch [0/3]

[Jörn Engel]

On Tue, 20 January 2009 11:16:37 +0100, Stefani Seibold wrote:
> 
> this is a patch which give you a better overview of the userland
> application stack usage, especially for embedded linux.
> 
> Currently you are only able to dump the main process/thread stack usage
> which is showed in proc/pid/status by the "VmStk" Value. But you get no
> information about the consumed stack memory of the the threads.
>
> [...]
> 
> This patch is against 2.6.28.1. The patch is cpu independent, so it
> should work on all linux supported architectures, it was tested under
> x86 and powerpc. Also there is not dependency a library: glibc, uclibc
> and all other should work.
> 
> I hope you like it and want ask what is necessary for inclusion into the
> main stream kernel or linux-next? If you have ideas how to do things in
> a better way, please let me know.

First goal would be to get people interested.  Why would Joe
Kernelhacker care about this, what problem would it solve for him?  Next
goal is to prove to akpm that the solved problems are worth the
maintenance burden this code brings.

It would be nice to have diffstat added to each patch to give people
a quick overview.  More importantly, the number of #ifdef's in the
patches may raise a red flag.  You should try to remove them from common
code and have a single one in the headers:

#ifdef CONFIG_NEW_FEATURE

void handle_this(int foo, long bar);

#else

static inline void handle_this(int foo, long bar)
{
}
#endif

Not sure what else to say.  I'm still wondering whether it will solve a
problem for me.

Jörn

-- 
Joern's library part 4:
http://www.paulgraham.com/spam.html

Detailed Stack Information Patch [0/3]

[Stefani Seibold]

Hi,

this is a patch which give you a better overview of the userland
application stack usage, especially for embedded linux.

Currently you are only able to dump the main process/thread stack usage
which is showed in proc/pid/status by the "VmStk" Value. But you get no
information about the consumed stack memory of the the threads.

The patch is splitted in three parts.


Part 1 :
--------

This is an enhancement in the /proc/<pid>/tasks/<tid>/maps and
smaps which marks the mapping where the thread stack pointer reside with
"[thread stack]".

Also there is a new entry "stack usage" in proc/pid/status, which will
you give the current stack usage.

This feature will be enabled the which "enable /proc/<pid> stack
monitoring" under "General setup-->"


Part 2:
-------

This enable a new /proc/stackmon file entry. A cat /proc/stackmon
will produce a output like:

   bytes   pages maxpages vm_start vm_end   processid threadid  name
     436       1       1  afdbf000-afdd4000 pid:  409 tid:  409 syslogd
    1168       1       1  afd12000-afd27000 pid:  411 tid:  411 sh
     516       1       1  afe6c000-afe81000 pid:  412 tid:  412 getty
    4580       2       2  af918000-af92d000 pid:  419 tid:  419 cat
The first value is the current effektive stack usage in bytes.

The second value is the current real stack usage in pages. This means
how many pages are really occupied by the stack.

The thrird value is the maximum real stack usage in pages. This value
will be determinate by the difference of the highest used page in the
mapping and the page of stack start address.

The fourth value is the start- and end- address of the mapping where the
stack currently reside.

The fifth value process id.

The sixth value is thread id.

And the seventh and last entry is the name of the process.

This feature will be enabled the which "enable /proc/<pid> stack
monitoring" under "General setup-->".


Part 3:
-------

There is also an additional stack monitor which can be enabled by boot
time or by the /sys filesystem. Which this you are able to detect if a
application use more stack than a given value. If the application
exceeds this value, it will receive a SIGTRAP signal. 

If there is a debugger attached at this time, there is the ability to
examinate the stack usage. Otherwise the application will be terminated.
In both cases a kernel log entry "pid:%d (%s) tid:%d stack size %lu
exceeds max stack size." will be written.

There are following entries under /sys/kernel/stackmon to control the
monitor.

mode:
 Setting this to an value not equal zero will start the stack monitoring
thread. Default is 0. Setting it to zero will stop the kernel thread.

stacksize:
 This value is the stack size in kb which triggers a SIGTRAP to the
application, if the stack usage is equal or more. Default is 256 kb.
 
ticks:
 Number of ticks between the monitoring invocation. A higher value will
give a less change to trigger a stack over usage, but will also result
in a less CPU usage. Default is 1.

All this parameters can also set at boot time with the kernel parameter
"stackmon=<stacksize>:<mode>:<ticks>".

The monitor can also compiled as a module.


This patch is against 2.6.28.1. The patch is cpu independent, so it
should work on all linux supported architectures, it was tested under
x86 and powerpc. Also there is not dependency a library: glibc, uclibc
and all other should work.

I hope you like it and want ask what is necessary for inclusion into the
main stream kernel or linux-next? If you have ideas how to do things in
a better way, please let me know.

Have a nice day,
Stefani