Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> So what is the verdict? I think the code looks right.
> 
> readelf -a /bin/bash shows:
>  0x0000000c (INIT)                       0x24198
> 
> This is a not an OPD, and therefore will have a local non-unique OPD
> constructed for the call.

See the same for INIT in /bin/sh.  However, the INIT value of interest
here is that for /lib/libc.so.6:

dave@mx3210:~/gnu/gcc-4.3/objdir/hppa-linux/libjava$ readelf -a /lib/libc.so.6|grep INIT
 0x0000000c (INIT)                       0x1f164

This was the value at r21 + 4.  This value also isn't an OPD.  However,
I showed before that init() didn't get called using an OPD.  As a result,
r19 was likely incorrect for the call.  See the code snippet in the
original report.

Possibly, there's a GCC optimization bug but I'm not sure.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> Out of curiosity did the space registers match between the core dump
> and the user fault printed by the kernel?

Yes.  One thing that I noticed is the illegal instruction kernel
printout lacks a header like the following:

Feb 19 00:22:42 localhost kernel: do_page_fault() pid=4255 command='ld' type=15
address=0x006cb040
Feb 19 00:22:42 localhost kernel: vm_start = 0x0007c000, vm_end = 0x001f4000

So, the cause of the printout isn't clear.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> I posted a testcase here that always crashes the kernel:
> http://lists.parisc-linux.org/pipermail/parisc-linux/2007-February/031250.html
> 
> I'm trying to build my kernel with debugging but my current
> configuration doesn't boot.

Yes, this testcase is obviously excellent progress toward understanding
this bug.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> On 2/19/07, John David Anglin <dave@hiauly1.hia.nrc.ca> wrote:
> > > John, what's the fastest non-PA8800 machine you have right now? Would a dual
> > > 750MHz j6700 help you?
> >
> > c3750 (875 MHz).  This one is is subject to the thread timing bug.
> 
> What is the "thread timing bug"?

http://lists.parisc-linux.org/pipermail/parisc-linux/2006-December/030969.html

I presume this is the same bug as you reported here:
http://lists.parisc-linux.org/pipermail/parisc-linux/2007-February/031200.html

In the java case, the tests that fail are always thread intensive.
I should note that when a test times out in the gcc testsuite (other
than in the acats suite), expect tries to kill off the processes
that have timed out.  Yet, sometimes the processes survive and killing
the initial process causes a system crash.  Also, the system often
fails to reboot when it panics.

There's some timing aspect to this bug.  I have the strong impression
that it occurs mainly on fast processors.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> John, what's the fastest non-PA8800 machine you have right now? Would a dual
> 750MHz j6700 help you?

c3750 (875 MHz).  This one is is subject to the thread timing bug.

I'm out of ethernet connections and NRC doesn't like switches, so
I can't easily add new machines.  I could swap out the c3750 for
the j6700.  It would probably would build gcc faster.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> Keep in mind James pointed out that while he fixed the primary source of
> corruption, there still might be second-order problems...

There's always bugs...

While this might be a kernel TLB management problem, there's no strong
evidence of that and I wouldn't automatically assign the problem to the
pa8800 bug black hole.  This could just as easily be a bug in the dynamic
loader.  The system now runs for weeks with only occasional problems in
UP mode.  Will see about SMP mode.

The memory corruption is very limited and appears to occur in the
initial phase of loading libc.  It's happened twice while the echo
command tried to generate a file list building libjava.  Tens of
thousands of other applications have been launched successfully.
Probably, a lot more applications would die if memory corruption
was rampant.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> What hardware are you using?

This is pa8800 (Mako) at 800 MHz.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[Kyle McMartin]

On Mon, Feb 19, 2007 at 11:56:30AM -0500, John David Anglin wrote:
> > What hardware are you using?
> 
> This is pa8800 (Mako) at 800 MHz.
> 

Keep in mind James pointed out that while he fixed the primary source of
corruption, there still might be second-order problems...

John, what's the fastest non-PA8800 machine you have right now? Would a dual
750MHz j6700 help you?

Cheers,
	Kyle M.
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> On 2/18/07, Carlos O'Donell <carlos@systemhalted.org> wrote:
> > We could add a PROC or OS specific DT_* value to solve this problem.
> > IA64 uses PLT_RESERVE for this specific purpose.
> 
> ... we are all volunteers, and this work would be *icing* on the cake.
> I'm trying to fix pthread_pop/push_cleanup in NPTL right now :-)

Here's another one:

echo ../../../gcc/libjava/classpath/lib/javax/swing/filechooser/*.class > javax/
swing/filechooser.list
/bin/sh: In elf_machine_rela_relative ELF32_R_SYM (reloc->r_info) != 0. Aborting
.make[3]: *** [javax/swing/filechooser.list] Illegal instruction (core dumped)
make[3]: *** Waiting for unfinished jobs....

Core was generated by `/bin/sh -c echo ../../../gcc/libjava/classpath/lib/javax/swing/filechooser/*.cl'.
Program terminated with signal 4, Illegal instruction.
#0  0x402e624c in _dl_relocate_object () from /lib/ld.so.1
(gdb) bt
#0  0x402e624c in _dl_relocate_object () from /lib/ld.so.1
#1  0x402e6248 in _dl_relocate_object () from /lib/ld.so.1
Previous frame identical to this frame (corrupt stack?)
(gdb) disass 0x402e623c 0x402e625c
Dump of assembler code from 0x402e623c to 0x402e625c:
0x402e623c <_dl_relocate_object+564>:   ldw 3c4(r1),r25
0x402e6240 <_dl_relocate_object+568>:   b,l 0x402e96d0 <_dl_dprintf>,rp
0x402e6244 <_dl_relocate_object+572>:   ldi 2,r26
0x402e6248 <_dl_relocate_object+576>:   copy r4,r19
0x402e624c <_dl_relocate_object+580>:   iitlbp r0,(sr0,r0)
0x402e6250 <_dl_relocate_object+584>:   cmpib,= 1,r6,0x402e64a0 <_dl_relocate_object+1176>
0x402e6254 <_dl_relocate_object+588>:   extrw,u r7,31,2,ret0
0x402e6258 <_dl_relocate_object+592>:   cmpib,>> 1,r6,0x402e64ac <_dl_relocate_object+1188>

It really seems like something is randomly messing up memory when shared
libraries are being loaded.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> On 2/18/07, Carlos O'Donell <carlos@systemhalted.org> wrote:
> > To be truthful I don't understand why we rummage around the relocs in
> > elf_machine_runtime_setup, we have the linkmap for the object we are
> > loading, there should be no need to do this.
> 
> I realized later, that DT_PLTGOT is the LTP, and we have no way to ask
> the linkmap the following questions:
> 
> 1. Where is the end of that objects PLT?
> 2. Where is the start of that objects GOT?
> 
> The only way to answer those questions is to look for the last reloc
> in the PLT and match the stub signature.

Is elf_machine_runtime_setup adjusting DT_PLTGOT?

This search seems horrific.  Shouldn't we add DT values for the
above?  There must be OS specific values that could be used.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> (gdb) p/x $r20
> $2 = 0x4086221c
> (gdb) x/2x $r20
> 0x4086221c <.LC2+84>:   0x00000003      0x40df252f

Comparing the setup of this region of memory in /lib/libc.so.6
to that in a simple testcase, it seems that the value setup for
r19 is somehow wrong.  Surrounding values are similar to what
I see in the simple testcase:

#include <stdlib.h>
int
main ()
{
  exit (0);
}

Testcase:
(gdb) x/16x 0x4046221c
0x4046221c:     0x00000003      0x40465c6c      0x00000002      0x0000147c
0x4046222c:     0x00000014      0x00000007      0x00000017      0x403397cc
0x4046223c:     0x00000007      0x4032d0ac      0x00000008      0x0000c720
0x4046224c:     0x00000009      0x0000000c      0x6ffffffc      0x00010ec0

Segv case:
(gdb) x/16x $r20
0x4086221c <.LC2+84>:   0x00000003      0x40df252f      0x00000002      0x0000147c
0x4086222c <.LC2+100>:  0x00000014      0x00000007      0x00000017      0x407397cc
0x4086223c <.LC2+116>:  0x00000007      0x4072d0ac      0x00000008      0x0000c720
0x4086224c <.LC2+132>:  0x00000009      0x0000000c      0x6ffffffc      0x00010ec0

I think '3' in the preceeding location indicates the erronious value
is a PLTGOT value (i.e., we're looking at part of the relocated dynamic
section in /lib/libc.so.6).

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> See the same for INIT in /bin/sh.  However, the INIT value of interest
> here is that for /lib/libc.so.6:
> 
> dave@mx3210:~/gnu/gcc-4.3/objdir/hppa-linux/libjava$ readelf -a /lib/libc.so.6|grep INIT
>  0x0000000c (INIT)                       0x1f164
> 
> This was the value at r21 + 4.  This value also isn't an OPD.

After I sent the last message, I realized that I had  misread the bb
condition.  The execution path actually traversed the code to setup an OPD:

(gdb) disass 0x400d7854 0x400d78a4
Dump of assembler code from 0x400d7854 to 0x400d78a4:
0x400d7854 <call_init+344>:     stw r22,-78(sp)
0x400d7858 <call_init+348>:     ldw 2c(r3),r20
0x400d785c <call_init+352>:     ldo -78(sp),r22
0x400d7860 <call_init+356>:     depwi -1,30,1,r22
0x400d7864 <call_init+360>:     ldw 4(r20),ret0
0x400d7868 <call_init+364>:     b,l 0x400d77c4 <call_init+200>,r0
0x400d786c <call_init+368>:     stw ret0,-74(sp)

(gdb) x/2x $sp - 0x78
0xc030ec08:     0x4073b164      0x40df252f
(gdb) p/x $ret0
$1 = 0x40df252f
(gdb) p/x $r20
$2 = 0x4086221c
(gdb) x/2x $r20
0x4086221c <.LC2+84>:   0x00000003      0x40df252f
(gdb) p/x $r3
$3 = 0x400010b0
(gdb) p/x $r3 + 0x2c
$4 = 0x400010dc
(gdb) x/x 0x400010dc
0x400010dc:     0x4086221c

So, the r19 value came from the OPD.  This value was loaded from
memory at 0x400d7864.  The value of r20 used in this insn was loaded
at 0x400d7858.  The values in memory and r20 are consistent.  The
values in memory and ret0 are also consistent.

(gdb) info sharedlib
>>>From        To          Syms Read   Shared Object Library
0x404dda70  0x40508088  Yes         /lib/libncurses.so.5
0x40317e44  0x40318fc0  Yes         /lib/libdl.so.2
0x4073ac50  0x40829a64  Yes         /lib/libc.so.6
0x400cccd0  0x400def18  Yes         /lib/ld.so.1

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> Both cases work correctly. So what happened in the original post?

There's two checks in call_init for the plabel bit:

0x400d77a4 <call_init+168>:     ldw 678(r1),ret0
0x400d77a8 <call_init+172>:     ldw 0(ret0),r20
0x400d77ac <call_init+176>:     bb,<,n r20,1e,0x400d7884 <call_init+392>
0x400d77b0 <call_init+180>:     cmpib,=,n 0,r21,0x400d77ec <call_init+240>
0x400d77b4 <call_init+184>:     ldw 4(r21),r20
0x400d77b8 <call_init+188>:     ldw 0(r3),ret0
0x400d77bc <call_init+192>:     add,l ret0,r20,r22
0x400d77c0 <call_init+196>:     bb,>=,n r22,1e,0x400d7854 <call_init+344>
0x400d77c4 <call_init+200>:     copy r19,r4
0x400d77c8 <call_init+204>:     fstw fr14,-10(sp)
0x400d77cc <call_init+208>:     ldw -10(sp),r26
0x400d77d0 <call_init+212>:     fstw fr13,-10(sp)
0x400d77d4 <call_init+216>:     ldw -10(sp),r25
0x400d77d8 <call_init+220>:     fstw fr12,-10(sp)
0x400d77dc <call_init+224>:     ldw -10(sp),r24
0x400d77e0 <call_init+228>:     b,l 0x400dedd4 <$$dyncall>,r31
0x400d77e4 <call_init+232>:     copy r31,rp
0x400d77e8 <call_init+236>:     copy r4,r19

It's the latter one that didn't have the plabel bit.  Didn't check
the first.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> > While it's possible that DT_INIT and DT_FINI shouldn't point at function
> > descriptors, the above hack can't be right since DL_DT_INIT_ADDRESS is used
> > for calling init() from dl-init.c:
> >
> >       init_t init = (init_t) DL_DT_INIT_ADDRESS
> >         (l, l->l_addr + l->l_info[DT_INIT]->d_un.d_ptr);
> >
> >       /* Call the function.  */
> >       init (argc, argv, env);
> 
> We might have been lucky in that init () called another function which
> setup r19 for us before we accessed data. I agree completely that r19
> should be set to the r19 of the module before calling init(), even if
> it *is* or *isn't* a function descriptor.
> 
> I wrote dl-lookupcfg.h, and the code you see there is legacy. I
> carried it forward without understanding what it meant.

It may be that special care is needed for applications that are
statically linked.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

[parisc-linux] Re: call_init in libc6 2.3.6.ds1-11

[John David Anglin]

> /* The test for "addr & 2" below is to accomodate old binaries which
>    violated the ELF ABI by pointing DT_INIT and DT_FINI at a function
>    descriptor.  */
> #define DL_DT_INIT_ADDRESS(map, addr) \
>   ((Elf32_Addr)(addr) & 2 ? (addr) : DL_AUTO_FUNCTION_ADDRESS (map, addr))

This is the ia64 define:

#define DL_DT_INIT_ADDRESS(map, addr) DL_AUTO_FUNCTION_ADDRESS (map, addr)

Calls to init on ia64 always use a function descriptor.

While it's possible that DT_INIT and DT_FINI shouldn't point at function
descriptors, the above hack can't be right since DL_DT_INIT_ADDRESS is used
for calling init() from dl-init.c:

      init_t init = (init_t) DL_DT_INIT_ADDRESS
	(l, l->l_addr + l->l_info[DT_INIT]->d_un.d_ptr);

      /* Call the function.  */
      init (argc, argv, env);

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux