modify_ldt returning ENOMEM on highmem machine

modify_ldt returning ENOMEM on highmem machine

[Holger Lubitz]

Hi,

I tried posting this before, but apparently it didn't get through to the
list.
The following are the final lines of a strace of a failing mplayer:

[...]
modify_ldt(0x1, 0xbffff85c, 0x10)       = -1 ENOMEM (Cannot allocate
memory)
--- SIGSEGV (Segmentation fault) ---
+++ killed by SIGSEGV +++

But I have lots of memory available:

[hal@duncan hal]$ cat /proc/meminfo 
        total:    used:    free:  shared: buffers:  cached:
Mem:  1054355456 1012023296 42332160        0 73637888 522129408
Swap: 542826496  4218880 538607616
MemTotal:      1029644 kB
MemFree:         41340 kB
MemShared:           0 kB
Buffers:         71912 kB
Cached:         505916 kB
SwapCached:       3976 kB
Active:         196216 kB
Inactive:       417156 kB
HighTotal:      131008 kB
HighFree:         2044 kB
LowTotal:       898636 kB
LowFree:         39296 kB
SwapTotal:      530104 kB
SwapFree:       525984 kB

The bug exists with 2.4.16 as well as with 2.4.17-rc1 and 2.4.13-ac8.
Under
ac8, I can work around it by copying something large to /dev/null, which
gets
some buffers freed.  With 2.4.16 and .17-rc1, usually all I can do to
get
mplayer working again is reboot.
After memory fills up again (usually one updatedb is sufficient), the
problem
reappears.

I have been unable to reproduce the problem when I use "mem=896M" or on
a
kernel with highmem support disabled, so I suspect it to be highmem
related.

Holger

Re: modify_ldt returning ENOMEM on highmem machine

[Alan Cox]

> I tried posting this before, but apparently it didn't get through to the
> list.
> The following are the final lines of a strace of a failing mplayer:

glibc 2.2 uses lots of ldt's which get vmalloced. Our vmalloc address space
is way too small on big boxes. Grab the patch posted to the kernel list
to only vmalloc large ldts and the box will be fine

Highmem isnt a direct link but large memory boxes are the most problematic