3.12.8 is doing nasty things

3.12.8 is doing nasty things

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 3404 bytes --]

I reported that I saw strange things happening after upgrading to 3.12.5. 
Meanwhile I upgraded to 3.12.8 and things became worse. I regularly see 
processes to fail randomly. I can see this as I somethimes get failures in my 
nagios because the SSH connection to the watched host (only the watched one 
runs .8) fails. I then see stuff like this in dmesg:

[723945.604000] do_page_fault() pid=30516 command='sshd' type=15 
address=0x00000000
[723945.604000] CPU: 1 PID: 30516 Comm: sshd Not tainted 3.12.8 #1
[723945.604000] task: 000000023f0eb7c8 ti: 00000002397ac000 task.ti: 
00000002397ac000

[723945.604000]      YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
[723945.604000] PSW: 00000000000001001111111100001111 Not tainted
[723945.604000] r00-03  000000ff0004ff0f 0000000040c3dc64 0000000040b63027 
0000000000000000
[723945.604000] r04-07  0000000040c3d464 0000000000000000 000000000008d11c 
0000000000089ca0
[723945.604000] r08-11  000000000008d11c 000000000008991c 000000000008811c 
000000000008811c
[723945.604000] r12-15  000000000008811c 0000000000000000 000000000008811c 
00000000000888a8
[723945.604000] r16-19  000000000006eb1c 000000000006eb3c 000000000006e440 
0000000040c3d464
[723945.604000] r20-23  00000000fb014990 000000004145eb40 0000000000000000 
0000000000000095
[723945.604000] r24-27  00000000fffffff5 0000000040002890 00000000fb014988 
000000000008711c
[723945.604000] r28-31  fffffffffffffffe 0000000000000000 00000000fb014a00 
0000000040b644c3
[723945.604000] sr00-03  0000000006807000 0000000000000000 0000000000000000 
0000000006807000
[723945.604000] sr04-07  0000000006807000 0000000006807000 0000000006807000 
0000000006807000

[723945.604000]       VZOUICununcqcqcqcqcqcrmunTDVZOUI
[723945.604000] FPSR: 00000000000000000000000000000000
[723945.604000] FPER1: 00000000
[723945.604000] fr00-03  0000000000000000 0000000000000000 0000000000000000 
0000000000000000
[723945.604000] fr04-07  41d4b5f44002d0e5 0000000000000000 0000000000000000 
0000000000000000
[723945.604000] fr08-11  0000000000000000 0000000000000000 0000000000000000 
0000000000000000
[723945.604000] fr12-15  41d4b5f44002d0e5 0000000000000000 0000000000000000 
0000000000000000
[723945.604000] fr16-19  0000000000000000 0000000000000000 0000000000000000 
0000000000000000
[723945.604000] fr20-23  0000000000000000 0000000000000000 0000000000469971 
0000000600000000
[723945.604000] fr24-27  0000000034a94675 0000000000000000 0000000000000000 
0000000000000000
[723945.604000] fr28-31  0000000000000000 0000000000000000 0000000000000000 
0000000000000000

[723945.604000] IASQ: 0000000006807000 0000000006807000 IAOQ: 0000000040b63033 
0000000040b63037
[723945.604000]  IIR: 0ca0109c    ISR: 0000000006807000  IOR: 0000000000000000
[723945.604000]  CPU:        1   CR30: 00000002397ac000 CR31: ffffffffffffffff
[723945.604000]  ORIG_R28: 0000000000000000
[723945.604000]  IAOQ[0]: 0000000040b63033
[723945.604000]  IAOQ[1]: 0000000040b63037
[723945.604000]  RP(r2): 0000000040b63027

When I upgraded OpenSSL I also saw an ICE during build, which went away on the 
next try.

I also see a lot of those:

[724507.480000] INEQUIVALENT ALIASES 0x42ed0000 and 0x42ed1000 in file 
libwrap.so.0.7.6

That could be the binutils bug that was mentioned, this file is from 2010. I'll 
rebuild that now to get rid of those warnings.

Next week I'll try 3.13, but since it has the same changes I expect the same 
results.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 3.12.8 is doing nasty things

[Jeroen Roovers]

On Fri, 24 Jan 2014 21:44:51 +0100
Rolf Eike Beer <eike-kernel@sf-tec.de> wrote:

> I reported that I saw strange things happening after upgrading to
> 3.12.5. Meanwhile I upgraded to 3.12.8 and things became worse. I
> regularly see processes to fail randomly. I can see this as I
> somethimes get failures in my nagios because the SSH connection to
> the watched host (only the watched one runs .8) fails.

Confirmed with any kernel above 3.12.2 that I cared to test (I didn't
test 3.12.3). The issues vary from illegal instructions to segmentation
faults to double frees in userland, so it's basically a "random" thing.
I can't say it's become "worse" after 3.12.5 (since I haven't kept per
version statistics).


Regards,
     jer

Re: 3.12.8 is doing nasty things

[Guy Martin]

On Fri, 24 Jan 2014 22:07:23 +0100
Jeroen Roovers <jer@gentoo.org> wrote:

> On Fri, 24 Jan 2014 21:44:51 +0100
> Rolf Eike Beer <eike-kernel@sf-tec.de> wrote:
> 
> > I reported that I saw strange things happening after upgrading to
> > 3.12.5. Meanwhile I upgraded to 3.12.8 and things became worse. I
> > regularly see processes to fail randomly. I can see this as I
> > somethimes get failures in my nagios because the SSH connection to
> > the watched host (only the watched one runs .8) fails.
> 
> Confirmed with any kernel above 3.12.2 that I cared to test (I didn't
> test 3.12.3). The issues vary from illegal instructions to
> segmentation faults to double frees in userland, so it's basically a
> "random" thing. I can't say it's become "worse" after 3.12.5 (since I
> haven't kept per version statistics).


Same here, 3.12.5 was not behaving correctly and 3.12.8 was even worse.
With the later, I also experienced sshd segfault, python threading
segfault/hangs.
I quickly reverted to 3.12.4 which is far more stable.

  Guy

Re: 3.12.8 is doing nasty things

[James Bottomley]

On Fri, 2014-01-24 at 22:26 +0100, Guy Martin wrote:
> On Fri, 24 Jan 2014 22:07:23 +0100
> Jeroen Roovers <jer@gentoo.org> wrote:
> 
> > On Fri, 24 Jan 2014 21:44:51 +0100
> > Rolf Eike Beer <eike-kernel@sf-tec.de> wrote:
> > 
> > > I reported that I saw strange things happening after upgrading to
> > > 3.12.5. Meanwhile I upgraded to 3.12.8 and things became worse. I
> > > regularly see processes to fail randomly. I can see this as I
> > > somethimes get failures in my nagios because the SSH connection to
> > > the watched host (only the watched one runs .8) fails.
> > 
> > Confirmed with any kernel above 3.12.2 that I cared to test (I didn't
> > test 3.12.3). The issues vary from illegal instructions to
> > segmentation faults to double frees in userland, so it's basically a
> > "random" thing. I can't say it's become "worse" after 3.12.5 (since I
> > haven't kept per version statistics).
> 
> 
> Same here, 3.12.5 was not behaving correctly and 3.12.8 was even worse.
> With the later, I also experienced sshd segfault, python threading
> segfault/hangs.
> I quickly reverted to 3.12.4 which is far more stable.

This all sounds like a flushing problem.  I can't confirm with my
pa8800 ... but that could be because the additional pa8800 flushes are
masking the problem.  Could you try bisecting the parisc commits
between .4 and .5 and see if there's a particular cause?

Thanks,

James

Re: 3.12.8 is doing nasty things

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 1572 bytes --]

Am Samstag, 25. Januar 2014, 14:30:39 schrieben Sie:
> On Fri, 2014-01-24 at 22:26 +0100, Guy Martin wrote:
> > On Fri, 24 Jan 2014 22:07:23 +0100
> > 
> > Jeroen Roovers <jer@gentoo.org> wrote:
> > > On Fri, 24 Jan 2014 21:44:51 +0100
> > > 
> > > Rolf Eike Beer <eike-kernel@sf-tec.de> wrote:
> > > > I reported that I saw strange things happening after upgrading to
> > > > 3.12.5. Meanwhile I upgraded to 3.12.8 and things became worse. I
> > > > regularly see processes to fail randomly. I can see this as I
> > > > somethimes get failures in my nagios because the SSH connection to
> > > > the watched host (only the watched one runs .8) fails.
> > > 
> > > Confirmed with any kernel above 3.12.2 that I cared to test (I didn't
> > > test 3.12.3). The issues vary from illegal instructions to
> > > segmentation faults to double frees in userland, so it's basically a
> > > "random" thing. I can't say it's become "worse" after 3.12.5 (since I
> > > haven't kept per version statistics).
> > 
> > Same here, 3.12.5 was not behaving correctly and 3.12.8 was even worse.
> > With the later, I also experienced sshd segfault, python threading
> > segfault/hangs.
> > I quickly reverted to 3.12.4 which is far more stable.
> 
> This all sounds like a flushing problem.  I can't confirm with my
> pa8800 ... but that could be because the additional pa8800 flushes are
> masking the problem.  Could you try bisecting the parisc commits
> between .4 and .5 and see if there's a particular cause?

It's the one mmap fix, I see .5 being stable after reverting that fix.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 3.12.8 is doing nasty things

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]

> Next week I'll try 3.13, but since it has the same changes I expect the same
> results.

I'm now on 3.13.2 and things are similar. While I did an initial test with 100 
invocations of ssh which went flawlessly I now see random process errors again. 
Examples can be found here: 
http://my.cdash.org/index.php?project=Qsmtp&date=2014-02-11

Normally the reports should have arrived much earlier, but for an unknown 
reason the resolv.conf (generated by DHCP) was empty, I assume that the DHCP 
process has failed during while renewing the IP address. The segfaults during 
the build have not happened with the older kernel.

Greetings,

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 3.12.8 is doing nasty things

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

Am Dienstag, 11. Februar 2014, 16:30:08 schrieb Rolf Eike Beer:
> > Next week I'll try 3.13, but since it has the same changes I expect the
> > same results.
> 
> I'm now on 3.13.2 and things are similar. While I did an initial test with
> 100 invocations of ssh which went flawlessly I now see random process
> errors again. Examples can be found here:
> http://my.cdash.org/index.php?project=Qsmtp&date=2014-02-11
> 
> Normally the reports should have arrived much earlier, but for an unknown
> reason the resolv.conf (generated by DHCP) was empty, I assume that the DHCP
> process has failed during while renewing the IP address. The segfaults
> during the build have not happened with the older kernel.

I can now confirm that the cache coherency fixes are _not_ the reason for those 
failures I see. I have rebuild 3.13.2 without them and still see random 
segfaults in my dashboards. I'll try the mmap() patch next.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 3.12.8 is doing nasty things

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 1145 bytes --]

Am Donnerstag, 13. Februar 2014, 07:37:55 schrieb Rolf Eike Beer:
> Am Dienstag, 11. Februar 2014, 16:30:08 schrieb Rolf Eike Beer:
> > > Next week I'll try 3.13, but since it has the same changes I expect the
> > > same results.
> > 
> > I'm now on 3.13.2 and things are similar. While I did an initial test with
> > 100 invocations of ssh which went flawlessly I now see random process
> > errors again. Examples can be found here:
> > http://my.cdash.org/index.php?project=Qsmtp&date=2014-02-11
> > 
> > Normally the reports should have arrived much earlier, but for an unknown
> > reason the resolv.conf (generated by DHCP) was empty, I assume that the
> > DHCP process has failed during while renewing the IP address. The
> > segfaults during the build have not happened with the older kernel.
> 
> I can now confirm that the cache coherency fixes are _not_ the reason for
> those failures I see. I have rebuild 3.13.2 without them and still see
> random segfaults in my dashboards. I'll try the mmap() patch next.

After 2,5 days with 3.13.2 and 0576da2c08e3d332f1b0653030d28ab804585ab6 I have 
not yet seen a single random segfault.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 3.12.8 is doing nasty things

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]

Am Samstag, 15. Februar 2014, 08:47:06 schrieb Rolf Eike Beer:
> Am Donnerstag, 13. Februar 2014, 07:37:55 schrieb Rolf Eike Beer:
> > Am Dienstag, 11. Februar 2014, 16:30:08 schrieb Rolf Eike Beer:
> > > > Next week I'll try 3.13, but since it has the same changes I expect
> > > > the
> > > > same results.
> > > 
> > > I'm now on 3.13.2 and things are similar. While I did an initial test
> > > with
> > > 100 invocations of ssh which went flawlessly I now see random process
> > > errors again. Examples can be found here:
> > > http://my.cdash.org/index.php?project=Qsmtp&date=2014-02-11
> > > 
> > > Normally the reports should have arrived much earlier, but for an
> > > unknown
> > > reason the resolv.conf (generated by DHCP) was empty, I assume that the
> > > DHCP process has failed during while renewing the IP address. The
> > > segfaults during the build have not happened with the older kernel.
> > 
> > I can now confirm that the cache coherency fixes are _not_ the reason for
> > those failures I see. I have rebuild 3.13.2 without them and still see
> > random segfaults in my dashboards. I'll try the mmap() patch next.
> 
> After 2,5 days with 3.13.2 and 0576da2c08e3d332f1b0653030d28ab804585ab6 I
> have not yet seen a single random segfault.

A short update on this: I am still running on that kernel and have not seen a 
single random process crash since then. I'm willing to help in debugging this 
if anyone tells me how, but I will now just revert that patch from every new 
kernel to get a stable machine.

Greetings,

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Aw: Re: 3.12.8 is doing nasty things

[Helge Deller]

> > > > I'm now on 3.13.2 and things are similar. While I did an initial test
> > > > with
> > > > 100 invocations of ssh which went flawlessly I now see random process
> > > > errors again. Examples can be found here:
> > > > http://my.cdash.org/index.php?project=Qsmtp&date=2014-02-11
> > > > 
> > > > Normally the reports should have arrived much earlier, but for an
> > > > unknown
> > > > reason the resolv.conf (generated by DHCP) was empty, I assume that the
> > > > DHCP process has failed during while renewing the IP address. The
> > > > segfaults during the build have not happened with the older kernel.
> > > 
> > > I can now confirm that the cache coherency fixes are _not_ the reason for
> > > those failures I see. I have rebuild 3.13.2 without them and still see
> > > random segfaults in my dashboards. I'll try the mmap() patch next.
> > 
> > After 2,5 days with 3.13.2 and 0576da2c08e3d332f1b0653030d28ab804585ab6 I
> > have not yet seen a single random segfault.
> 
> A short update on this: I am still running on that kernel and have not seen a 
> single random process crash since then. I'm willing to help in debugging this 
> if anyone tells me how, but I will now just revert that patch from every new 
> kernel to get a stable machine.

On a debian-kernel linux-image-3.13-1-parisc64-smp (3.13.4-1) and the latest libc6
which Dave uploaded, I'm now seeing similiar issues which are most likely related:

lsXX:~# dpkg-reconfigure locales
Generating locales (this might take a while)...
  de_DE.ISO-8859-1...cannot map archive header: Invalid argument
 done
  de_DE.UTF-8...cannot map archive header: Invalid argument
 done
  de_DE.ISO-8859-15@euro...cannot map archive header: Invalid argument
 done
  en_US.ISO-8859-1...cannot map archive header: Invalid argument
 done
  en_US.ISO-8859-15...cannot map archive header: Invalid argument
 done
  en_US.UTF-8...cannot map archive header: Invalid argument
 done
Generation complete.

Helge

Re: Aw: Re: 3.12.8 is doing nasty things

[John David Anglin]

On 2/26/2014 9:26 AM, Helge Deller wrote:
>>>>> I'm now on 3.13.2 and things are similar. While I did an initial test
>>>>> with
>>>>> 100 invocations of ssh which went flawlessly I now see random process
>>>>> errors again. Examples can be found here:
>>>>> http://my.cdash.org/index.php?project=Qsmtp&date=2014-02-11
>>>>>
>>>>> Normally the reports should have arrived much earlier, but for an
>>>>> unknown
>>>>> reason the resolv.conf (generated by DHCP) was empty, I assume that the
>>>>> DHCP process has failed during while renewing the IP address. The
>>>>> segfaults during the build have not happened with the older kernel.
>>>> I can now confirm that the cache coherency fixes are _not_ the reason for
>>>> those failures I see. I have rebuild 3.13.2 without them and still see
>>>> random segfaults in my dashboards. I'll try the mmap() patch next.
>>> After 2,5 days with 3.13.2 and 0576da2c08e3d332f1b0653030d28ab804585ab6 I
>>> have not yet seen a single random segfault.
>> A short update on this: I am still running on that kernel and have not seen a
>> single random process crash since then. I'm willing to help in debugging this
>> if anyone tells me how, but I will now just revert that patch from every new
>> kernel to get a stable machine.
> On a debian-kernel linux-image-3.13-1-parisc64-smp (3.13.4-1) and the latest libc6
> which Dave uploaded, I'm now seeing similiar issues which are most likely related:
>
> lsXX:~# dpkg-reconfigure locales
> Generating locales (this might take a while)...
>    de_DE.ISO-8859-1...cannot map archive header: Invalid argument
In my opinion, this is a glibc bug.  The issue has been discussed before.

I'm not seeing random segvs with 3.13.2 and later in general use.  I 
haven't reverted anything.

The only random program failures that I see are thread related. There 
are a bunch of failures
in the glibc testsuite, but these are not new.  There was a message on 
the console this morning about
reaching maximum lock depth 1024.  This was probably from glibc 
testsuite.  There is a known
issue with locking in glib2.0 which sometimes causes dot to segv.

If you are testing programs that use threads intensively, then I could 
see that you might see problems.

A TLS bug affecting thread local variables was recently fixed in gcc-4.8.

Dave

-- 
John David Anglin    dave.anglin@bell.net

Re: 3.12.8 is doing nasty things

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 1741 bytes --]

Am Mittwoch 26 Februar 2014, 11:00:01 schrieben Sie:
> On 2/26/2014 9:26 AM, Helge Deller wrote:

>>>> After 2,5 days with 3.13.2 and 0576da2c08e3d332f1b0653030d28ab804585ab6
>>>> I have not yet seen a single random segfault.
> >> 
> >> A short update on this: I am still running on that kernel and have not
> >> seen a single random process crash since then. I'm willing to help in
> >> debugging this if anyone tells me how, but I will now just revert that
> >> patch from every new kernel to get a stable machine.
> > 
> > On a debian-kernel linux-image-3.13-1-parisc64-smp (3.13.4-1) and the
> > latest libc6 which Dave uploaded, I'm now seeing similiar issues which
> > are most likely related:
> > 
> > lsXX:~# dpkg-reconfigure locales
> > Generating locales (this might take a while)...
> > 
> >    de_DE.ISO-8859-1...cannot map archive header: Invalid argument
> 
> In my opinion, this is a glibc bug.  The issue has been discussed before.
> 
> I'm not seeing random segvs with 3.13.2 and later in general use.  I
> haven't reverted anything.
> 
> The only random program failures that I see are thread related. There
> are a bunch of failures
> in the glibc testsuite, but these are not new.  There was a message on
> the console this morning about
> reaching maximum lock depth 1024.  This was probably from glibc
> testsuite.  There is a known
> issue with locking in glib2.0 which sometimes causes dot to segv.
> 
> If you are testing programs that use threads intensively, then I could
> see that you might see problems.
> 
> A TLS bug affecting thread local variables was recently fixed in gcc-4.8.

I don't think this is the problem that I'm seeing. What crashed was things 
like make or ld which don't care for threads.

Eike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 3.12.8 is doing nasty things

[Jeroen Roovers]

On Wed, 26 Feb 2014 15:26:15 +0100
"Helge Deller" <deller@gmx.de> wrote:

> On a debian-kernel linux-image-3.13-1-parisc64-smp (3.13.4-1) and the
> latest libc6 which Dave uploaded, I'm now seeing similiar issues
> which are most likely related:
> 
> lsXX:~# dpkg-reconfigure locales
> Generating locales (this might take a while)...
>   de_DE.ISO-8859-1...cannot map archive header: Invalid argument
>  done

That isn't the same problem with the random segmentation faults that
this thread documents.


     jer

Re: 3.12.8 is doing nasty things

[John David Anglin]

On 2/26/2014 2:20 PM, Jeroen Roovers wrote:
> On Wed, 26 Feb 2014 15:26:15 +0100
> "Helge Deller" <deller@gmx.de> wrote:
>
>> On a debian-kernel linux-image-3.13-1-parisc64-smp (3.13.4-1) and the
>> latest libc6 which Dave uploaded, I'm now seeing similiar issues
>> which are most likely related:
>>
>> lsXX:~# dpkg-reconfigure locales
>> Generating locales (this might take a while)...
>>    de_DE.ISO-8859-1...cannot map archive header: Invalid argument
>>   done
> That isn't the same problem with the random segmentation faults that
> this thread documents.
Well the matter is confusing.  I may be wrong but Eike seems to be 
complaining that
0576da2c08e3d332f1b0653030d28ab804585ab6 is related to the random 
segmentation
fault.   This is the change to the mmap handling that causes the invalid 
argument error above.

Can this be clarified?

Dave

-- 
John David Anglin    dave.anglin@bell.net

Re: Re: 3.12.8 is doing nasty things

[Aaro Koskinen]

On Wed, Feb 26, 2014 at 03:26:15PM +0100, Helge Deller wrote:
> On a debian-kernel linux-image-3.13-1-parisc64-smp (3.13.4-1) and the latest libc6
> which Dave uploaded, I'm now seeing similiar issues which are most likely related:
> 
> lsXX:~# dpkg-reconfigure locales
> Generating locales (this might take a while)...
>   de_DE.ISO-8859-1...cannot map archive header: Invalid argument

This is caused by this: http://marc.info/?l=linux-parisc&m=138748066325722&w=2

A.