TASK_UNMAPPED_BASE & stack location

TASK_UNMAPPED_BASE & stack location

[Martin J. Bligh]

Is there any good reason we can't remove TASK_UNMAPPED_BASE, and just shove
libraries directly above the program text? Red Hat seems to have patches to
dynamically tune it on a per-processes basis anyway ...

Moreover, can we put the stack back where it's meant to be, below the
program text, in that wasted 128MB of virtual space? Who really wants 
> 128MB of stack anyway (and can't fix their app)?

I'm sure there's some horrible reason we can't do this ... would just like
to know what it is. If it's "standards compilance" I don't really believe
it - we don't comply with the standard now anyway ...

M.

PS. Motivation is creating large shmem segments for DBs.

Re: TASK_UNMAPPED_BASE & stack location

[H. Peter Anvin]

Followup to:  <459930000.1051302738@[10.10.2.4]>
By author:    "Martin J. Bligh" <mbligh@aracnet.com>
In newsgroup: linux.dev.kernel
>
> Is there any good reason we can't remove TASK_UNMAPPED_BASE, and just shove
> libraries directly above the program text? Red Hat seems to have patches to
> dynamically tune it on a per-processes basis anyway ...
> 
> Moreover, can we put the stack back where it's meant to be, below the
> program text, in that wasted 128MB of virtual space? Who really wants 
> > 128MB of stack anyway (and can't fix their app)?
> 

That space is NULL pointer trap zone.  NULL pointer trapping -> good.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: TASK_UNMAPPED_BASE & stack location

[Martin J. Bligh]

>> Is there any good reason we can't remove TASK_UNMAPPED_BASE, and just
>> shove libraries directly above the program text? Red Hat seems to have
>> patches to dynamically tune it on a per-processes basis anyway ...
>> 
>> Moreover, can we put the stack back where it's meant to be, below the
>> program text, in that wasted 128MB of virtual space? Who really wants 
>> > 128MB of stack anyway (and can't fix their app)?
> 
> That space is NULL pointer trap zone.  NULL pointer trapping -> good.

128Mb of it? The bottom page, or even a few Mb, sure ... 
but 128Mb seems somewhat excessive ...

M.

Re: TASK_UNMAPPED_BASE & stack location

[H. Peter Anvin]

Martin J. Bligh wrote:
>>>Is there any good reason we can't remove TASK_UNMAPPED_BASE, and just
>>>shove libraries directly above the program text? Red Hat seems to have
>>>patches to dynamically tune it on a per-processes basis anyway ...
>>>
>>>Moreover, can we put the stack back where it's meant to be, below the
>>>program text, in that wasted 128MB of virtual space? Who really wants 
>>>
>>>>128MB of stack anyway (and can't fix their app)?
>>
>>That space is NULL pointer trap zone.  NULL pointer trapping -> good.
> 
> 
> 128Mb of it? The bottom page, or even a few Mb, sure ... 
> but 128Mb seems somewhat excessive ...
> 

Perhaps, but large data structures can easily generate reasonably large
values when indirected.

	-hpa

Re: TASK_UNMAPPED_BASE & stack location

[Timothy Miller]

Martin J. Bligh wrote:

>>>Is there any good reason we can't remove TASK_UNMAPPED_BASE, and just
>>>shove libraries directly above the program text? Red Hat seems to have
>>>patches to dynamically tune it on a per-processes basis anyway ...
>>>
>>>Moreover, can we put the stack back where it's meant to be, below the
>>>program text, in that wasted 128MB of virtual space? Who really wants 
>>>      
>>>
>>>>128MB of stack anyway (and can't fix their app)?
>>>>        
>>>>
>>That space is NULL pointer trap zone.  NULL pointer trapping -> good.
>>    
>>
>
>128Mb of it? The bottom page, or even a few Mb, sure ... 
>but 128Mb seems somewhat excessive ..
>  
>
Considering that your process space is 4gig, and that that 128Mb doesn't 
really exist anywhere (no RAM, no page table entries, nothing), it's 
really not excessive.  If you're so strapped for process space that you 
need that extra 128Mb, then you probably shouldn't be using a 32-bit 
processor.

I understand that the stack exists somewhere high up in the address 
space.  And there's some other things up there (mmap space, etc).  What 
happens if the heap grows so much that it collides with one of those 
upper address spaces?  Out of memory?

Re: TASK_UNMAPPED_BASE & stack location

[Martin J. Bligh]

>> 128Mb of it? The bottom page, or even a few Mb, sure ... 
>> but 128Mb seems somewhat excessive ..
>>  
>> 
> Considering that your process space is 4gig, and that that 128Mb doesn't
> really exist anywhere (no RAM, no page table entries, nothing), it's
> really not excessive.  

I need the virtual space.

> If you're so strapped for process space that you
> need that extra 128Mb, then you probably shouldn't be using a 32-bit
> processor.

Point me at a cheap 32 cpu 64-bit machine. Market realities dicate
otherwise.

M.

Re: TASK_UNMAPPED_BASE & stack location

[H. Peter Anvin]

Martin J. Bligh wrote:
>>>128Mb of it? The bottom page, or even a few Mb, sure ... 
>>>but 128Mb seems somewhat excessive ..
>>> 
>>>
>>
>>Considering that your process space is 4gig, and that that 128Mb doesn't
>>really exist anywhere (no RAM, no page table entries, nothing), it's
>>really not excessive.  
> 
> 
> I need the virtual space.
> 

As far as I know, you're perfectly free to move the stack whereever you
feel like, as long as you map the space first, copy the contents and set
%esp.  mremap() is useful for this.

	-hpa

Re: TASK_UNMAPPED_BASE & stack location

[Martin J. Bligh]

>>>> 128Mb of it? The bottom page, or even a few Mb, sure ... 
>>>> but 128Mb seems somewhat excessive ..
>>> 
>>> Considering that your process space is 4gig, and that that 128Mb doesn't
>>> really exist anywhere (no RAM, no page table entries, nothing), it's
>>> really not excessive.  
>> 
>> I need the virtual space.
> 
> As far as I know, you're perfectly free to move the stack whereever you
> feel like, as long as you map the space first, copy the contents and set
> %esp.  mremap() is useful for this.

Right ... I was just suggesting changing it by default. Was wondering if
anyone knew of anything it'd break. Your issue with NULL pointers is
interesting, but I'm not sure this would really affect that much ... the
stack is normally tiny in comparison.

M.

Re: TASK_UNMAPPED_BASE & stack location

[badari]

Martin,

Only problem with moving TASK_UNMAPPED_BASE right above
text would be - limiting the malloc() space. malloc() is clever enough
to mmap() and do the right thing. Once I moved TASK_UNMAPPED_BASE
to 0x10000000 and I could not run some of the programs with large
data segments.

Moving stacks below text would be tricky. pthread library knows
the placement of stack. It uses this to distinguish between
threads and pthreads manager.

I don't know what other librarys/apps depend on this kind of stuff.

Thanks,
Badari

static inline pthread_descr thread_self (void)
{
#ifdef THREAD_SELF
  return THREAD_SELF;
#else
  char *sp = CURRENT_STACK_FRAME;
  if (sp >= __pthread_initial_thread_bos)
    return &__pthread_initial_thread;
  else if (sp >= __pthread_manager_thread_bos
           && sp < __pthread_manager_thread_tos)
    return &__pthread_manager_thread;
  else if (__pthread_nonstandard_stacks)
    return __pthread_find_self();
  else
#ifdef _STACK_GROWS_DOWN
    return (pthread_descr)(((unsigned long)sp | (STACK_SIZE-1))+1) - 1;
#else
    return (pthread_descr)((unsigned long)sp &~ (STACK_SIZE-1));
#endif
#endif
}


"Martin J. Bligh" wrote:

> Is there any good reason we can't remove TASK_UNMAPPED_BASE, and just shove
> libraries directly above the program text? Red Hat seems to have patches to
> dynamically tune it on a per-processes basis anyway ...
>
> Moreover, can we put the stack back where it's meant to be, below the
> program text, in that wasted 128MB of virtual space? Who really wants
> > 128MB of stack anyway (and can't fix their app)?
>
> I'm sure there's some horrible reason we can't do this ... would just like
> to know what it is. If it's "standards compilance" I don't really believe
> it - we don't comply with the standard now anyway ...
>
> M.
>
> PS. Motivation is creating large shmem segments for DBs.
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: TASK_UNMAPPED_BASE & stack location

[William Lee Irwin III]

On Fri, Apr 25, 2003 at 04:52:05PM -0700, badari wrote:
> Only problem with moving TASK_UNMAPPED_BASE right above
> text would be - limiting the malloc() space. malloc() is clever enough
> to mmap() and do the right thing. Once I moved TASK_UNMAPPED_BASE
> to 0x10000000 and I could not run some of the programs with large
> data segments.
> Moving stacks below text would be tricky. pthread library knows
> the placement of stack. It uses this to distinguish between
> threads and pthreads manager.
> I don't know what other librarys/apps depend on this kind of stuff.

STACK_TOP is easy to change to see what goes wrong; it's a single
#define in include/asm-i386/a.out.h

Someone should spin it up and see how well pthreads copes.


-- wli

Re: TASK_UNMAPPED_BASE & stack location

[H. Peter Anvin]

Martin J. Bligh wrote:
> 
> Right ... I was just suggesting changing it by default. Was wondering if
> anyone knew of anything it'd break. Your issue with NULL pointers is
> interesting, but I'm not sure this would really affect that much ... the
> stack is normally tiny in comparison.
> 

Normally, yes, but there are applications which use very large stacks.

	-hpa

Re: TASK_UNMAPPED_BASE & stack location

[Hugh Dickins]

On Fri, 25 Apr 2003, Martin J. Bligh wrote:
> >> 128Mb of it? The bottom page, or even a few Mb, sure ... 
> >> but 128Mb seems somewhat excessive ..

Yes.

> > Considering that your process space is 4gig, and that that 128Mb doesn't
> > really exist anywhere (no RAM, no page table entries, nothing), it's
> > really not excessive.  
> 
> I need the virtual space.

Plus you would (very often) get more physical.  i386 ELF text typically
begins at 0x08048000: putting stack just below text in many cases shares
page table between stack+text+data, and saves the page table at top of
user address space.

Hugh

Re: TASK_UNMAPPED_BASE & stack location

[Martin J. Bligh]

>> >> 128Mb of it? The bottom page, or even a few Mb, sure ... 
>> >> but 128Mb seems somewhat excessive ..
> 
> Yes.
> 
>> > Considering that your process space is 4gig, and that that 128Mb
>> > doesn't really exist anywhere (no RAM, no page table entries,
>> > nothing), it's really not excessive.  
>> 
>> I need the virtual space.
> 
> Plus you would (very often) get more physical.  i386 ELF text typically
> begins at 0x08048000: putting stack just below text in many cases shares
> page table between stack+text+data, and saves the page table at top of
> user address space.

A fine point - I hadn't thought of that. 
It just seems tidier all around ....

M.

Re: TASK_UNMAPPED_BASE & stack location

[jlnance]

On Fri, Apr 25, 2003 at 03:06:23PM -0700, Martin J. Bligh wrote:
> > If you're so strapped for process space that you
> > need that extra 128Mb, then you probably shouldn't be using a 32-bit
> > processor.
> 
> Point me at a cheap 32 cpu 64-bit machine. Market realities dicate
> otherwise.

Do you have a cheap 32 cpu 32-bit machine?  SMP?

Re: TASK_UNMAPPED_BASE & stack location

[Rik van Riel]

On Fri, 25 Apr 2003, Martin J. Bligh wrote:

> Is there any good reason we can't remove TASK_UNMAPPED_BASE, and just shove
> libraries directly above the program text? Red Hat seems to have patches to
> dynamically tune it on a per-processes basis anyway ...

What could be done is leave the stack where it is, but have
malloc() space and mmap() space grow towards each other:

0                                            3G
| |prog | malloc -->         <-- mmap | stack |

The stack will get the stack size ulimit size and the space
between where malloc and mmap start should be about 2.7 GB.

That 2.7 GB will of course by divided between malloc and mmap,
but the division will be done dynamically based on whoever
needs the space.  Much better than the current static 1:1.7
division...

Re: TASK_UNMAPPED_BASE & stack location

[William Lee Irwin III]

On Fri, 25 Apr 2003, Martin J. Bligh wrote:
>> Is there any good reason we can't remove TASK_UNMAPPED_BASE, and just shove
>> libraries directly above the program text? Red Hat seems to have patches to
>> dynamically tune it on a per-processes basis anyway ...

On Sat, Apr 26, 2003 at 10:37:11AM -0400, Rik van Riel wrote:
> What could be done is leave the stack where it is, but have
> malloc() space and mmap() space grow towards each other:
> 0                                            3G
> | |prog | malloc -->         <-- mmap | stack |
> The stack will get the stack size ulimit size and the space
> between where malloc and mmap start should be about 2.7 GB.
> That 2.7 GB will of course by divided between malloc and mmap,
> but the division will be done dynamically based on whoever
> needs the space.  Much better than the current static 1:1.7
> division...

My internal proposals (backed by code) already include this in addition
to relocating the stack (whose kernel side is trivial).


-- wli

Re: TASK_UNMAPPED_BASE & stack location

[Martin J. Bligh]

>> > If you're so strapped for process space that you
>> > need that extra 128Mb, then you probably shouldn't be using a 32-bit
>> > processor.
>> 
>> Point me at a cheap 32 cpu 64-bit machine. Market realities dicate
>> otherwise.
> 
> Do you have a cheap 32 cpu 32-bit machine?  SMP?

Comparitively, yes. They're about 1/8th of the price. Technically speaking
it's SMP, but NUMA, not UMA. Though I think Unisys's box is UMA (more or
less anyway).

M.

Re: TASK_UNMAPPED_BASE & stack location

[Hui Huang]

Andi Kleen wrote:
> 
>>I've moved PAGE_OFFSET around a lot (which moves the stack, as you say). 
>>Haven't seen it break anything yet ... IMHO it was broken anyway if this
>>hurts it. Obviously not something one could do in a stable kernel series,
>>but 2.5 seems like a perfect time for it to me ... unless I'm missing some
>>glibc / linker thing, it seems like a simple change.
> 
> 
> It at least broke Sun Java.
> 

That has been fixed in latest JDK.

Re: TASK_UNMAPPED_BASE & stack location

[Chuck Ebbert]

Martin J. Bligh wrote:


> Point me at a cheap 32 cpu 64-bit machine. Market realities dicate
> otherwise.


  Well it looks like the CPU is now available -- 'all' you need now
is a 32-port Hypertransport switch. :)

-------
 Chuck

Re: TASK_UNMAPPED_BASE & stack location

[Chuck Ebbert]

Andi Kleen wrote:


> (e.g. just moving __PAGE_OFFSET on amd64 to 4GB for 32bit broke some things)


  Wow.  The whole kernel is 'not there' as far as 32bit usermode
on x86-^h^h^h^hAMD64 is concerned then, right?

  Do you also get 'unlimited' space to run multiple 32-bit apps?


-------
 Chuck

Re: TASK_UNMAPPED_BASE & stack location

[Andi Kleen]

"Martin J. Bligh" <mbligh@aracnet.com> writes:

>>> Is there any good reason we can't remove TASK_UNMAPPED_BASE, and just
>>> shove libraries directly above the program text? Red Hat seems to have
>>> patches to dynamically tune it on a per-processes basis anyway ...
>> 
>> Yes. You won't get a continuous sbrk/brk heap then anymore. Not sure it
>> is a  big problem though.
>
> Me no understand. I think this *makes* it a contiguous space. The way I see
> it, we currently allocate from TASK_UNMAPPED_BASE up to the top, then start
> again above program text. Which seems a bit silly.

The space for brk is not completely continuous anymore, especially
when you use mmap() too.

Same with mmap.

Basically mmap() and brk/sbrk (=malloc) will fragment each other.

wli's suggestion of making it a personality makes sense.

> I've moved PAGE_OFFSET around a lot (which moves the stack, as you say). 
> Haven't seen it break anything yet ... IMHO it was broken anyway if this
> hurts it. Obviously not something one could do in a stable kernel series,
> but 2.5 seems like a perfect time for it to me ... unless I'm missing some
> glibc / linker thing, it seems like a simple change.

It at least broke Sun Java.

-Andi

Re: TASK_UNMAPPED_BASE & stack location

[Martin J. Bligh]

>> Is there any good reason we can't remove TASK_UNMAPPED_BASE, and just
>> shove libraries directly above the program text? Red Hat seems to have
>> patches to dynamically tune it on a per-processes basis anyway ...
> 
> Yes. You won't get a continuous sbrk/brk heap then anymore. Not sure it
> is a  big problem though.

Me no understand. I think this *makes* it a contiguous space. The way I see
it, we currently allocate from TASK_UNMAPPED_BASE up to the top, then start
again above program text. Which seems a bit silly.

> It's probably worth a sysctl at least.

mmm. I'd prefer a config option. Defaulting to 'y' ;-)

>> Moreover, can we put the stack back where it's meant to be, below the
>> program text, in that wasted 128MB of virtual space? Who really wants 
>>> 128MB of stack anyway (and can't fix their app)?
> 
> You could, but I bet it would break some programs
> (e.g. just moving __PAGE_OFFSET on amd64 to 4GB for 32bit broke some
> things)

I've moved PAGE_OFFSET around a lot (which moves the stack, as you say). 
Haven't seen it break anything yet ... IMHO it was broken anyway if this
hurts it. Obviously not something one could do in a stable kernel series,
but 2.5 seems like a perfect time for it to me ... unless I'm missing some
glibc / linker thing, it seems like a simple change.

M.

Re: TASK_UNMAPPED_BASE & stack location

[William Lee Irwin III]

"Martin J. Bligh" <mbligh@aracnet.com> writes:
>> Is there any good reason we can't remove TASK_UNMAPPED_BASE, and just shove
>> libraries directly above the program text? Red Hat seems to have patches to
>> dynamically tune it on a per-processes basis anyway ...

On Fri, Apr 25, 2003 at 11:54:56PM +0200, Andi Kleen wrote:
> Yes. You won't get a continuous sbrk/brk heap then anymore. Not sure it is a 
> big problem though.
> But apparently Solaris/x86 is doing that.
> It's probably worth a sysctl at least.

How about a personality? It is a very slightly different ABI.


-- wli

Re: TASK_UNMAPPED_BASE & stack location

[Andi Kleen]

"Martin J. Bligh" <mbligh@aracnet.com> writes:

> Is there any good reason we can't remove TASK_UNMAPPED_BASE, and just shove
> libraries directly above the program text? Red Hat seems to have patches to
> dynamically tune it on a per-processes basis anyway ...

Yes. You won't get a continuous sbrk/brk heap then anymore. Not sure it is a 
big problem though.

But apparently Solaris/x86 is doing that.

It's probably worth a sysctl at least.

> Moreover, can we put the stack back where it's meant to be, below the
> program text, in that wasted 128MB of virtual space? Who really wants 
>> 128MB of stack anyway (and can't fix their app)?

You could, but I bet it would break some programs
(e.g. just moving __PAGE_OFFSET on amd64 to 4GB for 32bit broke some things)

-Andi