Huge dynamically linked program does not run on mips-linux

Huge dynamically linked program does not run on mips-linux

[Atsushi Nemoto]

I have a problem that my huge dynamically linked program cause
SIGSEGV or SIGBUS immediately after running from main() on mips-linux.

Digging into this problem, I found that GOT entries are corrupted.

Here are some informations from readelf and objdump:

$ mips-linux-readelf -a myapp
...
Relocation section '.rel.dyn' at offset 0xe3b20 contains 426 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
...
100c47bc  003f7f03 R_MIPS_REL32      004e4990   getrlimit
...
Symbol table '.dynsym' contains 17742 entries:
   Num:    Value  Size Type    Bind   Vis      Ndx Name
...
 16255: 004e4990     0 FUNC    GLOBAL DEFAULT  UND getrlimit
...
Symbol table '.symtab' contains 26849 entries:
   Num:    Value  Size Type    Bind   Vis      Ndx Name
...
  7664: 004e4990     0 FUNC    GLOBAL DEFAULT  UND getrlimit
...

$ mips-linux-objdump -s -j .got myapp
...
Contents of section .got:
...
 100c47b0 100c5cba 00c07c54 100c5c5c 004e4990  ..\...|T..\\.NI.
...


0x004e4990 is "stub" routine to jump into libc's getrlimit.

0x100c47bc must contain 0x004e4990 at run-time, but when the signal
sent the value is 0x009c9320.  It is 0x004e4990+0x004e4990.

The outout from objdump shows 0x004e4990 is already in .got and the
output from readelf shows 0x004e4990 will be added to it by dynamic
loader.  Is my understanding right?

Could anyone tell me why binutils generate such informations?  Or is
this dynamic loader issue?

I'm using binutils 2.14, gcc-3.3.2, glibc-2.2.5.  I tried gcc-3.3.1,
glibc-2.3.2, uClibc-0.9.21, binutils-2.14.90.0.6 and binutils-cvs but
no lock.

My program is huge enough so that older binutils causes "relocation
truncated to fit" error.

The program can work well if statically linked.  Other (relatively
small) dynamically linked programs can work well also.

Thanks.
---
Atsushi Nemoto

Re: Huge dynamically linked program does not run on mips-linux

[Atsushi Nemoto]

>>>>> On Wed, 22 Oct 2003 17:11:18 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> said:
anemo> I have a problem that my huge dynamically linked program cause
anemo> SIGSEGV or SIGBUS immediately after running from main() on
anemo> mips-linux.

anemo> Digging into this problem, I found that GOT entries are
anemo> corrupted.
...
anemo> My program is huge enough so that older binutils causes
anemo> "relocation truncated to fit" error.

More information.  My program's .got size exceeds 64K.  It seems the
corruption does not happen if .got size is smaller then 64K.

$ mips-linux-readelf -e myapp
...
Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
...
  [21] .got              PROGBITS        100b15d0 a075d0 013a04 04 WAp  0   0 16

---
Atsushi Nemoto

Re: Huge dynamically linked program does not run on mips-linux

[Jun Sun]

On Wed, Oct 29, 2003 at 04:32:01PM +0900, Atsushi Nemoto wrote:
> >>>>> On Wed, 22 Oct 2003 17:11:18 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> said:
> anemo> I have a problem that my huge dynamically linked program cause
> anemo> SIGSEGV or SIGBUS immediately after running from main() on
> anemo> mips-linux.
> 
> anemo> Digging into this problem, I found that GOT entries are
> anemo> corrupted.
> ...
> anemo> My program is huge enough so that older binutils causes
> anemo> "relocation truncated to fit" error.
> 
> More information.  My program's .got size exceeds 64K.  It seems the
> corruption does not happen if .got size is smaller then 64K.
> 
> $ mips-linux-readelf -e myapp
> ...
> Section Headers:
>   [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
> ...
>   [21] .got              PROGBITS        100b15d0 a075d0 013a04 04 WAp  0   0 16
> 

Isn't this a known problem in binutils?  IIRC, someone is working or has
added "--big-got" support.

Jun

Re: Huge dynamically linked program does not run on mips-linux

[Daniel Jacobowitz]

On Wed, Oct 29, 2003 at 10:14:00AM -0800, Jun Sun wrote:
> On Wed, Oct 29, 2003 at 04:32:01PM +0900, Atsushi Nemoto wrote:
> > >>>>> On Wed, 22 Oct 2003 17:11:18 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> said:
> > anemo> I have a problem that my huge dynamically linked program cause
> > anemo> SIGSEGV or SIGBUS immediately after running from main() on
> > anemo> mips-linux.
> > 
> > anemo> Digging into this problem, I found that GOT entries are
> > anemo> corrupted.
> > ...
> > anemo> My program is huge enough so that older binutils causes
> > anemo> "relocation truncated to fit" error.
> > 
> > More information.  My program's .got size exceeds 64K.  It seems the
> > corruption does not happen if .got size is smaller then 64K.
> > 
> > $ mips-linux-readelf -e myapp
> > ...
> > Section Headers:
> >   [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
> > ...
> >   [21] .got              PROGBITS        100b15d0 a075d0 013a04 04 WAp  0   0 16
> > 
> 
> Isn't this a known problem in binutils?  IIRC, someone is working or has
> added "--big-got" support.

Atsushi-san's program would not even link with a binutils that didn't
support multiple GOTs; I guess that something is going wrong with that
support.

I don't suppose you could provide a testcase?

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: Huge dynamically linked program does not run on mips-linux

[Thiemo Seufer]

Daniel Jacobowitz wrote:
[snip]
> > > anemo> My program is huge enough so that older binutils causes
> > > anemo> "relocation truncated to fit" error.
> > > 
> > > More information.  My program's .got size exceeds 64K.  It seems the
> > > corruption does not happen if .got size is smaller then 64K.
> > > 
> > > $ mips-linux-readelf -e myapp
> > > ...
> > > Section Headers:
> > >   [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
> > > ...
> > >   [21] .got              PROGBITS        100b15d0 a075d0 013a04 04 WAp  0   0 16
> > > 
> > 
> > Isn't this a known problem in binutils?  IIRC, someone is working or has
> > added "--big-got" support.
> 
> Atsushi-san's program would not even link with a binutils that didn't
> support multiple GOTs; I guess that something is going wrong with that
> support.

When building python-qt3 on debian unstable I get an oversize GOT and:

/usr/bin/ld: BFD 2.14.90.0.6 20030820 Debian GNU/Linux assertion fail ../../bfd/elfxx-mips.c:2287

Seems like multi-GOT is broken for this case.


Thiemo

Re: Huge dynamically linked program does not run on mips-linux

[Eric Christopher]

> When building python-qt3 on debian unstable I get an oversize GOT and:
> 
> /usr/bin/ld: BFD 2.14.90.0.6 20030820 Debian GNU/Linux assertion fail ../../bfd/elfxx-mips.c:2287
> 
> Seems like multi-GOT is broken for this case.

Judging from the line offset from today's sources to that error message
it might have been fixed with rsandifo's last patch that fixed an
uninitialized variable problem. (global_gotno)

-eric

-- 
Eric Christopher <echristo@redhat.com>

Re: Huge dynamically linked program does not run on mips-linux

[Atsushi Nemoto]

>>>>> On Wed, 29 Oct 2003 18:25:04 -0800, Eric Christopher <echristo@redhat.com> said:
>> When building python-qt3 on debian unstable I get an oversize GOT
>> and:
>> 
>> /usr/bin/ld: BFD 2.14.90.0.6 20030820 Debian GNU/Linux assertion
>> fail ../../bfd/elfxx-mips.c:2287
>> 
>> Seems like multi-GOT is broken for this case.

eric> Judging from the line offset from today's sources to that error
eric> message it might have been fixed with rsandifo's last patch that
eric> fixed an uninitialized variable problem. (global_gotno)

I tried binutils-2.14.90.0.7 (based on binutils 2003 1029 in CVS) but
my problem did no solved.  It seems something is still wrong.

---
Atsushi Nemoto

Re: Huge dynamically linked program does not run on mips-linux

[Eric Christopher]

> eric> Judging from the line offset from today's sources to that error
> eric> message it might have been fixed with rsandifo's last patch that
> eric> fixed an uninitialized variable problem. (global_gotno)
> 
> I tried binutils-2.14.90.0.7 (based on binutils 2003 1029 in CVS) but
> my problem did no solved.  It seems something is still wrong.

This was plain mips-linux? Not mips64-linux?

And where would I find the sources?

-eric

-- 
Eric Christopher <echristo@redhat.com>

Re: Huge dynamically linked program does not run on mips-linux

[Atsushi Nemoto]

>>>>> On Tue, 04 Nov 2003 00:05:56 -0800, Eric Christopher <echristo@redhat.com> said:
>> I tried binutils-2.14.90.0.7 (based on binutils 2003 1029 in CVS)
>> but my problem did no solved.  It seems something is still wrong.

eric> This was plain mips-linux? Not mips64-linux?

Yes.  mips-linux and mipsel-linux target (host is i386).  Both target
generate broken binary for my test program.

eric> And where would I find the sources?

I'm using plain binutils 2.14 and gcc 3.3.2 from gnu.org FTP site,
binutils 2.14.90.0.7 from
http://www.kernel.org/pub/linux/devel/binutils/.

---
Atsushi Nemoto

Re: Huge dynamically linked program does not run on mips-linux

[Eric Christopher]

On Tue, 2003-11-04 at 03:02, Atsushi Nemoto wrote:
> >>>>> On Tue, 04 Nov 2003 00:05:56 -0800, Eric Christopher <echristo@redhat.com> said:
> >> I tried binutils-2.14.90.0.7 (based on binutils 2003 1029 in CVS)
> >> but my problem did no solved.  It seems something is still wrong.
> 
> eric> This was plain mips-linux? Not mips64-linux?
> 
> Yes.  mips-linux and mipsel-linux target (host is i386).  Both target
> generate broken binary for my test program.
> 
> eric> And where would I find the sources?
> 
> I'm using plain binutils 2.14 and gcc 3.3.2 from gnu.org FTP site,
> binutils 2.14.90.0.7 from
> http://www.kernel.org/pub/linux/devel/binutils/.

I'm using mainline gcc, but I meant the python-qt sources you were
compiling.

-eric

-- 
Eric Christopher <echristo@redhat.com>

Re: Huge dynamically linked program does not run on mips-linux

[Atsushi Nemoto]

>>>>> On Tue, 04 Nov 2003 09:53:06 -0800, Eric Christopher <echristo@redhat.com> said:

echristo> I'm using mainline gcc, but I meant the python-qt sources
echristo> you were compiling.

The link error of phyton-qt was reported by Thiemo Seufer.  I have not
tried it.

My problem is runtime failure, not link error.  So it may be a
different problem.

---
Atsushi Nemoto

Re: Huge dynamically linked program does not run on mips-linux

[Atsushi Nemoto]

>>>>> On Fri, 07 Nov 2003 01:54:21 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> said:
anemo> My problem is runtime failure, not link error.  So it may be a
anemo> different problem.

My problem was solved by recent fix in binutils.

The patch is:

http://sources.redhat.com/ml/binutils/2003-11/msg00469.html

and is already in binutils CVS.  Thank you.

---
Atsushi Nemoto

Re: Huge dynamically linked program does not run on mips-linux

[Thiemo Seufer]

Eric Christopher wrote:
[snip]
> > Yes.  mips-linux and mipsel-linux target (host is i386).  Both target
> > generate broken binary for my test program.
> > 
> > eric> And where would I find the sources?
> > 
> > I'm using plain binutils 2.14 and gcc 3.3.2 from gnu.org FTP site,
> > binutils 2.14.90.0.7 from
> > http://www.kernel.org/pub/linux/devel/binutils/.
> 
> I'm using mainline gcc, but I meant the python-qt sources you were
> compiling.

It was python-qt-3.8 from debian unstable, compiled with
"gcc (GCC) 3.3.2 (Debian)" and binutils
"2.14.90.0.7 20031029 Debian GNU/Linux"

An attempt to link with CVS ld shows the same BFD assertion.


Thiemo

Re: Huge dynamically linked program does not run on mips-linux

[Eric Christopher]

> > I'm using mainline gcc, but I meant the python-qt sources you were
> > compiling.
> 
> It was python-qt-3.8 from debian unstable, compiled with
> "gcc (GCC) 3.3.2 (Debian)" and binutils
> "2.14.90.0.7 20031029 Debian GNU/Linux"
> 
> An attempt to link with CVS ld shows the same BFD assertion.

OK. Well, the one machine I have has current sources for gcc and
binutils so I'll be trying that :)

Pointer to a tarball with the sources?

-eric

-- 
Eric Christopher <echristo@redhat.com>

Re: Huge dynamically linked program does not run on mips-linux

[Atsushi Nemoto]

[-- Attachment #1: Type: Text/Plain, Size: 8124 bytes --]

>>>>> On Wed, 29 Oct 2003 13:15:17 -0500, Daniel Jacobowitz <dan@debian.org> said:
dan> Atsushi-san's program would not even link with a binutils that
dan> didn't support multiple GOTs; I guess that something is going
dan> wrong with that support.

Yes, I guess too.

dan> I don't suppose you could provide a testcase?

I wrote a short script to generate a testcase.  Running attached awk
script create src0.c, src1.c, ..., src3.c.

$ mips-linux-gcc -c src[0-4].c
$ mips-linux-gcc -o bigapp src[0-4].o

This bigapp program compiled with binutils-2.14, gcc-3.3.2, uClibc
0.9.21 is corrupted.

Here is some outputs from readelf:

$ mips-linux-readelf -S bigapp
There are 31 section headers, starting at offset 0x36ad78:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .interp           PROGBITS        004000f4 0000f4 00000d 00   A  0   0  1
  [ 2] .reginfo          MIPS_REGINFO    00400104 000104 000018 18   A  0   0  4
  [ 3] .dynamic          DYNAMIC         0040011c 00011c 0000d8 08   A  6   0  4
  [ 4] .hash             HASH            004001f4 0001f4 028784 04   A  5   0  4
  [ 5] .dynsym           DYNSYM          00428978 028978 061c40 10   A  6   1  4
  [ 6] .dynstr           STRTAB          0048a5b8 08a5b8 041da5 00   A  0   0  1
  [ 7] .init             PROGBITS        004cc370 0cc370 0000ac 00  AX  0   0  4
  [ 8] .text             PROGBITS        004cc420 0cc420 1b7d60 00  AX  0   0 16
  [ 9] .fini             PROGBITS        00684180 284180 00005c 00  AX  0   0  4
  [10] .rodata           PROGBITS        006841e0 2841e0 000010 00   A  0   0 16
  [11] .data             PROGBITS        10000000 285000 000010 00  WA  0   0 16
  [12] .rld_map          PROGBITS        10000010 285010 000004 00  WA  0   0  4
  [13] .eh_frame         PROGBITS        10000014 285014 000004 00   A  0   0  4
  [14] .ctors            PROGBITS        10000018 285018 000008 00  WA  0   0  4
  [15] .dtors            PROGBITS        10000020 285020 000008 00  WA  0   0  4
  [16] .jcr              PROGBITS        10000028 285028 000004 00  WA  0   0  4
  [17] .got              PROGBITS        10000030 285030 02245c 04 WAp  0   0 16
  [18] .sbss             NOBITS          1002248c 2a7490 000000 00 WAp  0   0  1
  [19] .bss              NOBITS          10022490 2a7490 000020 00  WA  0   0 16
  [20] .comment          PROGBITS        00000000 2a7490 0000a2 00      0   0  1
  [21] .debug_aranges    MIPS_DWARF      00000000 2a7538 000020 00      0   0  8
  [22] .debug_info       MIPS_DWARF      00000000 2a7558 000069 00      0   0  1
  [23] .debug_abbrev     MIPS_DWARF      00000000 2a75c1 000014 00      0   0  1
  [24] .debug_line       MIPS_DWARF      00000000 2a75d5 000044 00      0   0  1
  [25] .pdr              PROGBITS        00000000 2a761c 0c3660 00      0   0  4
  [26] .rel.dyn          REL             004cc360 0cc360 000010 08   A  5   0 16
  [27] .mdebug.abi32     PROGBITS        00000000 36ac7c 000000 00      0   0  1
  [28] .shstrtab         STRTAB          00000000 36ac7c 0000fb 00      0   0  1
  [29] .symtab           SYMTAB          00000000 36b250 0620a0 10     30  45  4
  [30] .strtab           STRTAB          00000000 3cd2f0 041ee7 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings)
  I (info), L (link order), G (group), x (unknown)
  O (extra OS processing required) o (OS specific), p (processor specific)
$ mips-linux-readelf -r bigapp

Relocation section '.rel.dyn' at offset 0xcc360 contains 2 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
00000000  00000000 R_MIPS_NONE      
1001f4f0  00314703 R_MIPS_REL32      004cc490   getpid
$ mips-linux-readelf -x 17 bigapp | grep 1001f4f0
  0x1001f4f0 004cc490 005ea570 00655ba0 0062c9e0 .L...^.p.e[..b..
$ mips-linux-readelf -s bigapp | grep getpid
 12615: 004cc490     0 FUNC    GLOBAL DEFAULT  UND getpid
  4109: 004cc490     0 FUNC    GLOBAL DEFAULT  UND getpid


Strangely the bigapp works correct if I compiled it with glibc 2.2.5.

Here is some outputs from readelf on bigapp.glibc:

$ mips-linux-readelf -S bigapp.glibc
There are 36 section headers, starting at offset 0x378644:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .interp           PROGBITS        00400114 000114 00000d 00   A  0   0  1
  [ 2] .note.ABI-tag     NOTE            00400130 000130 000020 00   A  0   0 16
  [ 3] .reginfo          MIPS_REGINFO    00400150 000150 000018 18   A  0   0  4
  [ 4] .dynamic          DYNAMIC         00400168 000168 0000f0 08   A  7   0  4
  [ 5] .hash             HASH            00400258 000258 028764 04   A  6   0  4
  [ 6] .dynsym           DYNSYM          004289bc 0289bc 061bc0 10   A  7   1  4
  [ 7] .dynstr           STRTAB          0048a57c 08a57c 041d6d 00   A  0   0  1
  [ 8] .gnu.version      VERSYM          004cc2ea 0cc2ea 00c378 02   A  6   0  2
  [ 9] .gnu.version_r    VERNEED         004d8664 0d8664 000020 00   A  7   1  4
  [10] .init             PROGBITS        004d86a0 0d86a0 0000ac 00  AX  0   0  4
  [11] .text             PROGBITS        004d8750 0d8750 1b7d40 00  AX  0   0 16
  [12] .fini             PROGBITS        00690490 290490 00005c 00  AX  0   0  4
  [13] .rodata           PROGBITS        006904f0 2904f0 000020 00   A  0   0 16
  [14] .data             PROGBITS        10000000 291000 000020 00  WA  0   0 16
  [15] .rld_map          PROGBITS        10000020 291020 000004 00  WA  0   0  4
  [16] .eh_frame         PROGBITS        10000024 291024 000004 00   A  0   0  4
  [17] .ctors            PROGBITS        10000028 291028 000008 00  WA  0   0  4
  [18] .dtors            PROGBITS        10000030 291030 000008 00  WA  0   0  4
  [19] .jcr              PROGBITS        10000038 291038 000004 00  WA  0   0  4
  [20] .got              PROGBITS        10000040 291040 022460 04 WAp  0   0 16
  [21] .sbss             NOBITS          100224a0 2b34a0 000000 00 WAp  0   0  1
  [22] .bss              NOBITS          100224a0 2b34a0 000020 00  WA  0   0 16
  [23] .comment          PROGBITS        00000000 2b34a0 0000b4 00      0   0  1
  [24] .debug_aranges    MIPS_DWARF      00000000 2b3558 000058 00      0   0  8
  [25] .debug_pubnames   MIPS_DWARF      00000000 2b35b0 000025 00      0   0  1
  [26] .debug_info       MIPS_DWARF      00000000 2b35d5 000c55 00      0   0  1
  [27] .debug_abbrev     MIPS_DWARF      00000000 2b422a 00012b 00      0   0  1
  [28] .debug_line       MIPS_DWARF      00000000 2b4355 00028e 00      0   0  1
  [29] .debug_str        MIPS_DWARF      00000000 2b45e3 000960 01  MS  0   0  1
  [30] .pdr              PROGBITS        00000000 2b4f44 0c35c0 00      0   0  4
  [31] .mdebug.abi32     PROGBITS        00000000 378504 000000 00      0   0  1
  [32] .rel.dyn          REL             004d8690 0d8690 000010 08   A  6   0 16
  [33] .shstrtab         STRTAB          00000000 378504 000140 00      0   0  1
  [34] .symtab           SYMTAB          00000000 378be4 062200 10     35  59  4
  [35] .strtab           STRTAB          00000000 3dade4 04204e 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings)
  I (info), L (link order), G (group), x (unknown)
  O (extra OS processing required) o (OS specific), p (processor specific)
$ mips-linux-readelf -r bigapp.glibc

Relocation section '.rel.dyn' at offset 0xd8690 contains 2 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
00000000  00000000 R_MIPS_NONE      
1001c3c0  00340403 R_MIPS_REL32      00000000   getpid
$ mips-linux-readelf -x 20 bigapp.glibc | grep 1001c3c0
  0x1001c3c0 00000000 0060d3e0 0063cd10 00608f70 .....`...c...`.p
$ mips-linux-readelf -s bigapp.glibc | grep getpid
 13316: 00000000     0 FUNC    GLOBAL DEFAULT  UND getpid@GLIBC_2.0 (2)
  2943: 00000000     0 FUNC    GLOBAL DEFAULT  UND getpid@@GLIBC_2.0


Is there other informations I can provide?

---
Atsushi Nemoto

[-- Attachment #2: mkbigapp --]
[-- Type: Text/Plain, Size: 688 bytes --]

#!/bin/awk -f
BEGIN {
	nfile = 5;
	nfunc = 5000;
	for (j = 0; j < nfile; j++) {
		fname = "src" j ".c";
		printf("/* %s */\n", fname) > fname;
		for (i = 0; i < nfunc; i++) {
			printf("void func%d_%d(void) {}\n", j, i, i) >> fname;
		}
		printf("int func%d(void) {\n", j) >> fname;
		for (i = 0; i < nfunc; i++) {
			printf("func%d_%d();\n", j, i) >> fname;
		}
		printf("return getpid();\n") >> fname;
		printf("}\n") >> fname;
		if (j == 0) {
			printf("int main(int argc, char **argv) {\n") > fname;
			for (i = 0; i < nfile; i++) {
				printf("printf(\"%%d\\n\", func%d());\n", i) >> fname;
			}
			printf("return 0;\n") >> fname;
			printf("}\n") >> fname;
		}
		close(fname);
	}
}

Re: Huge dynamically linked program does not run on mips-linux

[Daniel Jacobowitz]

On Thu, Oct 30, 2003 at 09:54:53PM +0900, Atsushi Nemoto wrote:
> >>>>> On Wed, 29 Oct 2003 13:15:17 -0500, Daniel Jacobowitz <dan@debian.org> said:
> dan> Atsushi-san's program would not even link with a binutils that
> dan> didn't support multiple GOTs; I guess that something is going
> dan> wrong with that support.
> 
> Yes, I guess too.
> 
> dan> I don't suppose you could provide a testcase?
> 
> I wrote a short script to generate a testcase.  Running attached awk
> script create src0.c, src1.c, ..., src3.c.
> 
> $ mips-linux-gcc -c src[0-4].c
> $ mips-linux-gcc -o bigapp src[0-4].o
> 
> This bigapp program compiled with binutils-2.14, gcc-3.3.2, uClibc
> 0.9.21 is corrupted.
> 
> Here is some outputs from readelf:
> 
> $ mips-linux-readelf -S bigapp
> There are 31 section headers, starting at offset 0x36ad78:
> 
> Section Headers:
>   [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
>   [ 0]                   NULL            00000000 000000 000000 00      0   0  0
>   [ 1] .interp           PROGBITS        004000f4 0000f4 00000d 00   A  0   0  1
>   [ 2] .reginfo          MIPS_REGINFO    00400104 000104 000018 18   A  0   0  4
>   [ 3] .dynamic          DYNAMIC         0040011c 00011c 0000d8 08   A  6   0  4
>   [ 4] .hash             HASH            004001f4 0001f4 028784 04   A  5   0  4
>   [ 5] .dynsym           DYNSYM          00428978 028978 061c40 10   A  6   1  4
>   [ 6] .dynstr           STRTAB          0048a5b8 08a5b8 041da5 00   A  0   0  1
>   [ 7] .init             PROGBITS        004cc370 0cc370 0000ac 00  AX  0   0  4
>   [ 8] .text             PROGBITS        004cc420 0cc420 1b7d60 00  AX  0   0 16
>   [ 9] .fini             PROGBITS        00684180 284180 00005c 00  AX  0   0  4
>   [10] .rodata           PROGBITS        006841e0 2841e0 000010 00   A  0   0 16
>   [11] .data             PROGBITS        10000000 285000 000010 00  WA  0   0 16
>   [12] .rld_map          PROGBITS        10000010 285010 000004 00  WA  0   0  4
>   [13] .eh_frame         PROGBITS        10000014 285014 000004 00   A  0   0  4
>   [14] .ctors            PROGBITS        10000018 285018 000008 00  WA  0   0  4
>   [15] .dtors            PROGBITS        10000020 285020 000008 00  WA  0   0  4
>   [16] .jcr              PROGBITS        10000028 285028 000004 00  WA  0   0  4
>   [17] .got              PROGBITS        10000030 285030 02245c 04 WAp  0   0 16
>   [18] .sbss             NOBITS          1002248c 2a7490 000000 00 WAp  0   0  1
>   [19] .bss              NOBITS          10022490 2a7490 000020 00  WA  0   0 16
>   [20] .comment          PROGBITS        00000000 2a7490 0000a2 00      0   0  1
>   [21] .debug_aranges    MIPS_DWARF      00000000 2a7538 000020 00      0   0  8
>   [22] .debug_info       MIPS_DWARF      00000000 2a7558 000069 00      0   0  1
>   [23] .debug_abbrev     MIPS_DWARF      00000000 2a75c1 000014 00      0   0  1
>   [24] .debug_line       MIPS_DWARF      00000000 2a75d5 000044 00      0   0  1
>   [25] .pdr              PROGBITS        00000000 2a761c 0c3660 00      0   0  4
>   [26] .rel.dyn          REL             004cc360 0cc360 000010 08   A  5   0 16
>   [27] .mdebug.abi32     PROGBITS        00000000 36ac7c 000000 00      0   0  1
>   [28] .shstrtab         STRTAB          00000000 36ac7c 0000fb 00      0   0  1
>   [29] .symtab           SYMTAB          00000000 36b250 0620a0 10     30  45  4
>   [30] .strtab           STRTAB          00000000 3cd2f0 041ee7 00      0   0  1
> Key to Flags:
>   W (write), A (alloc), X (execute), M (merge), S (strings)
>   I (info), L (link order), G (group), x (unknown)
>   O (extra OS processing required) o (OS specific), p (processor specific)
> $ mips-linux-readelf -r bigapp
> 
> Relocation section '.rel.dyn' at offset 0xcc360 contains 2 entries:
>  Offset     Info    Type            Sym.Value  Sym. Name
> 00000000  00000000 R_MIPS_NONE      
> 1001f4f0  00314703 R_MIPS_REL32      004cc490   getpid
> $ mips-linux-readelf -x 17 bigapp | grep 1001f4f0
>   0x1001f4f0 004cc490 005ea570 00655ba0 0062c9e0 .L...^.p.e[..b..
> $ mips-linux-readelf -s bigapp | grep getpid
>  12615: 004cc490     0 FUNC    GLOBAL DEFAULT  UND getpid
>   4109: 004cc490     0 FUNC    GLOBAL DEFAULT  UND getpid

Now this is very strange.  Please bear with me; I have the bad habit of
starting my email before I've fully figured out the problem...


MIPS, at least MIPS/Linux, has a very odd ELF ABI.  The following isn't
based on documentation, just on my reading of the glibc loader.  The
.got section is relocated implicitly; dynamic relocations aren't even
_used_ normally, in the single-GOT case.

First we have reserved entres; two of them.  Then local got entries;
DT_MIPS_LOCAL_GOTNO of them.  They get the library base added.  Then
global entries.  We start in the dynsym table at DT_MIPS_GOTSYM, and
count up to just under DT_MIPS_SYMTABNO; GOT entries are set to the
address of the referenced symbol.

The loader does not require or expect any dynamic relocations to refer
to the .got section.  It also doesn't seem to care what's there, at
least not for external functions - defined functions and sections are
another matter.  So it doesn't expect the relocation to be there. 


Now, here's the peculiar thing:
drow@nevyn:~/mips-got% readelf -Ds app | sort -n | grep UND
   55 3777: 00000000     0  NOTYPE   WEAK DEFAULT UND __gmon_start__
  768 1005: 00000000     0  NOTYPE   WEAK DEFAULT UND _Jv_RegisterClasses
  5231 8511: 00000000     0    FUNC GLOBAL DEFAULT UND __libc_start_main
  13350 4887: 00000000     0    FUNC GLOBAL DEFAULT UND getpid
  22116 8185: 00000000     0    FUNC GLOBAL DEFAULT UND printf
drow@nevyn:~/mips-got% readelf -Dr app                     

'REL' relocation section at offset 0x4d8700 contains 24 bytes:
 Offset     Info    Type            Sym.Value  Sym. Name
00000000  00000000 R_MIPS_NONE      
1001c53c  00342603 R_MIPS_REL32      00000000   getpid
100196c4  00566403 R_MIPS_REL32      00000000   printf


i.e. there are five undefined symbols, and I'm guessing that the ones
in the first created GOT don't get REL32 relocations emitted.

A little poking in binutils reveals that this is deliberate.  This
comment, from the multigot patch in January:
  /* A pointer to the primary got, i.e., the one that's going to get
     the implicit relocations from DT_MIPS_LOCAL_GOTNO and
     DT_MIPS_GOTSYM.  */
  struct mips_got_info *primary;

There's only one problem - it doesn't work the way that comment
suggests.  got[2] through got[DT_MIPS_LOCAL_GOTNO - 1] are relocated
by l_addr, as local GOT entries.  Then got[DT_MIPS_LOCAL_GOTNO] through
got[DT_MIPS_SYMTABNO - 1] are relocated as global GOT entries.  So
DT_MIPS_SYMTABNO is probably intended to be the number of symbols _used
by the primary GOT_.  Not the total size of .dynsym.


Reading the comments in elfxx-mips.c it looks like this shouldn't
even be happening; the global symbols should end up in the "master
GOT".  And there should be stubs.  But in my tests the stubs don't get
built, and the value in the GOT for getpid points to func0_4663.  And I
can't find code that ought to create them.  So I'm going to have to
punt to Alexandre for details.  A quick potential fix might be changing
DT_MIPS_SYMTABNO.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer