GCC crashes on aarch64 since gatesgarth

GCC crashes on aarch64 since gatesgarth

[Mike Looijmans]

When doing large builds, the GCC compiler tends to crash on random spots 
in the code. There are a few common denominators though.

It only happens when compiling for aarch64 (cortex-A53), not for 32-bit 
arm (cortex-A9)

It's random and usually happens on "big" sets like kernel, openssl, 
boost, u-boot etc.

It always reports "during GIMPLE pass: ealias" in the error, for example:

| during GIMPLE pass: ealias
| ../openssl-1.1.1i/crypto/x509v3/v3_utl.c: In function 'do_x509_check':
| ../openssl-1.1.1i/crypto/x509v3/v3_utl.c:1239:1: internal compiler 
error: Illegal instruction
| 1239 | }

Compiling the same thing again usually goes fine.

I've never experienced this with the zeus and older branches of OE.


I've already tried upgrading to the latest gatesgarth status, and 
cleaning out everything and start from scratch. I've also run "mprime" 
test on my machine (over one hour) just to be confident that the system 
itself is really okay.


Ideas to diagnose, fix or reliably reprodruce are more than welcome.

-- 
Mike Looijmans


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topicproducts.com

Please consider the environment before printing this e-mail

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Leon Woestenberg]

[-- Attachment #1: Type: text/plain, Size: 2066 bytes --]

Hello Mike,

At first sight, this does sound like memory corruption in one specific
memory area (DIMM?) to me.
Check dmesg for tripping temperatures etc.
I would reduce both the amount of bitbake tasks and Makefile parallelism to
1 on a fresh run to reduce memory pressure.

Not seen anything similar yet (MACHINE=zcu102, build host i7-10700K w/
128MB memory.)
I would assume aarch64 is widely run by the community.
Let me know if I must replicate a specific set of commits.

Regards,

Leon.

On Wed, Jan 27, 2021 at 4:10 PM Mike Looijmans <mike.looijmans@topic.nl>
wrote:

> When doing large builds, the GCC compiler tends to crash on random spots
> in the code. There are a few common denominators though.
>
> It only happens when compiling for aarch64 (cortex-A53), not for 32-bit
> arm (cortex-A9)
>
> It's random and usually happens on "big" sets like kernel, openssl,
> boost, u-boot etc.
>
> It always reports "during GIMPLE pass: ealias" in the error, for example:
>
> | during GIMPLE pass: ealias
> | ../openssl-1.1.1i/crypto/x509v3/v3_utl.c: In function 'do_x509_check':
> | ../openssl-1.1.1i/crypto/x509v3/v3_utl.c:1239:1: internal compiler
> error: Illegal instruction
> | 1239 | }
>
> Compiling the same thing again usually goes fine.
>
> I've never experienced this with the zeus and older branches of OE.
>
>
> I've already tried upgrading to the latest gatesgarth status, and
> cleaning out everything and start from scratch. I've also run "mprime"
> test on my machine (over one hour) just to be confident that the system
> itself is really okay.
>
>
> Ideas to diagnose, fix or reliably reprodruce are more than welcome.
>
> --
> Mike Looijmans
>
>
> Met vriendelijke groet / kind regards,
>
> Mike Looijmans
> System Expert
>
>
> TOPIC Embedded Products B.V.
> Materiaalweg 4, 5681 RJ Best
> The Netherlands
>
> T: +31 (0) 499 33 69 69
> E: mike.looijmans@topicproducts.com
> W: www.topicproducts.com
>
> Please consider the environment before printing this e-mail
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 2860 bytes --]

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Mike Looijmans]

Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topicproducts.com

Please consider the environment before printing this e-mail
On 27-01-2021 16:43, Leon Woestenberg wrote:
>
> Hello Mike,
>
> At first sight, this does sound like memory corruption in one specific 
> memory area (DIMM?) to me.

On itself, yes, that was my first thought too - hence running memory and 
system tests.
But it only happens with the gatesgarth branch. I've put loads of 
pressure on this machine's memory before, never got errors like that. I 
sometimes have multiple OE branches building at the same time too.

> Check dmesg for tripping temperatures etc.
> I would reduce both the amount of bitbake tasks and Makefile 
> parallelism to 1 on a fresh run to reduce memory pressure.

Reducing somewhat, e.g. 8 each, did not make a difference. Didn't want 
to go to "1", it'll take days to rebuild then...

> Not seen anything similar yet (MACHINE=zcu102, build host i7-10700K w/ 
> 128MB memory.)
>
Guess that's GB not MB :)

Mine is a Ryzen 7 1700 w/ 32GB.

CPU Temp doesn't rise above 70 degrees while building (during mprime 
testing it got up to 90, still within spec)

> I would assume aarch64 is widely run by the community.

I would, too

> Let me know if I must replicate a specific set of commits.

Don't know what you mean by that, can you explain?



>
> Regards,
>
> Leon.
>
> On Wed, Jan 27, 2021 at 4:10 PM Mike Looijmans 
> <mike.looijmans@topic.nl <mailto:mike.looijmans@topic.nl>> wrote:
>
>     When doing large builds, the GCC compiler tends to crash on random
>     spots
>     in the code. There are a few common denominators though.
>
>     It only happens when compiling for aarch64 (cortex-A53), not for
>     32-bit
>     arm (cortex-A9)
>
>     It's random and usually happens on "big" sets like kernel, openssl,
>     boost, u-boot etc.
>
>     It always reports "during GIMPLE pass: ealias" in the error, for
>     example:
>
>     | during GIMPLE pass: ealias
>     | ../openssl-1.1.1i/crypto/x509v3/v3_utl.c: In function
>     'do_x509_check':
>     | ../openssl-1.1.1i/crypto/x509v3/v3_utl.c:1239:1: internal compiler
>     error: Illegal instruction
>     | 1239 | }
>
>     Compiling the same thing again usually goes fine.
>
>     I've never experienced this with the zeus and older branches of OE.
>
>
>     I've already tried upgrading to the latest gatesgarth status, and
>     cleaning out everything and start from scratch. I've also run
>     "mprime"
>     test on my machine (over one hour) just to be confident that the
>     system
>     itself is really okay.
>
>
>     Ideas to diagnose, fix or reliably reprodruce are more than welcome.
>
>     -- 
>     Mike Looijmans
>
>
>     Met vriendelijke groet / kind regards,
>
>     Mike Looijmans
>     System Expert
>
>
>     TOPIC Embedded Products B.V.
>     Materiaalweg 4, 5681 RJ Best
>     The Netherlands
>
>     T: +31 (0) 499 33 69 69
>     E: mike.looijmans@topicproducts.com
>     <mailto:mike.looijmans@topicproducts.com>
>     W: www.topicproducts.com <http://www.topicproducts.com>
>
>     Please consider the environment before printing this e-mail
>
>     
>

-- 
Mike Looijmans

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Leon Woestenberg]

>
> > Let me know if I must replicate a specific set of commits.
>
> Don't know what you mean by that, can you explain?
>
I mean I could try to reproduce your build locally, but I would want
the specific commits of the layers you are testing against, and the
local.conf settings that trigger your faults.
(I.e. be as close to the failing setup as possible).

Leon.

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Mike Looijmans]

I've been doing quite some tests and things last week, my conclusion so 
far is that the latest gcc seems to trigger a bug particular to this 
CPU. Maybe even this particular batch.

Hours of testing found no issues with RAM.

I've also done huge amounts of builds with zeus, no problems.

Then googling for "illegal instruction gcc ryzen 1700" reveals quite a 
few threads on something that sounds suspiciously similar. Though it 
makes me wonder why I never encountered it before, since these are all 
many years old. I did try a bunch of the suggestions done there, but so 
far no luck.



Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topicproducts.com

Please consider the environment before printing this e-mail
On 27-01-2021 16:10, Mike Looijmans via lists.openembedded.org wrote:
> When doing large builds, the GCC compiler tends to crash on random 
> spots in the code. There are a few common denominators though.
>
> It only happens when compiling for aarch64 (cortex-A53), not for 
> 32-bit arm (cortex-A9)
>
> It's random and usually happens on "big" sets like kernel, openssl, 
> boost, u-boot etc.
>
> It always reports "during GIMPLE pass: ealias" in the error, for example:
>
> | during GIMPLE pass: ealias
> | ../openssl-1.1.1i/crypto/x509v3/v3_utl.c: In function 'do_x509_check':
> | ../openssl-1.1.1i/crypto/x509v3/v3_utl.c:1239:1: internal compiler 
> error: Illegal instruction
> | 1239 | }
>
> Compiling the same thing again usually goes fine.
>
> I've never experienced this with the zeus and older branches of OE.
>
>
> I've already tried upgrading to the latest gatesgarth status, and 
> cleaning out everything and start from scratch. I've also run "mprime" 
> test on my machine (over one hour) just to be confident that the 
> system itself is really okay.
>
>
> Ideas to diagnose, fix or reliably reprodruce are more than welcome. 

-- 
Mike Looijmans

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Randy MacLeod]

On 2021-02-05 2:49 a.m., Mike Looijmans wrote:
> I've been doing quite some tests and things last week, my conclusion so 
> far is that the latest gcc seems to trigger a bug particular to this 
> CPU. Maybe even this particular batch.
> 
> Hours of testing found no issues with RAM.
> 
> I've also done huge amounts of builds with zeus, no problems.
> 
> Then googling for "illegal instruction gcc ryzen 1700" reveals quite a 
> few threads on something that sounds suspiciously similar. Though it 
> makes me wonder why I never encountered it before, since these are all 
> many years old. I did try a bunch of the suggestions done there, but so 
> far no luck.
> 
> 

Any update on this bug Mike?
Have you opened a YP bug? If not please do.
I'd like to see if the problem still happens on master.

../Randy


> 
> Met vriendelijke groet / kind regards,
> 
> Mike Looijmans
> System Expert
> 
> 
> TOPIC Embedded Products B.V.
> Materiaalweg 4, 5681 RJ Best
> The Netherlands
> 
> T: +31 (0) 499 33 69 69
> E: mike.looijmans@topicproducts.com
> W: www.topicproducts.com
> 
> Please consider the environment before printing this e-mail
> On 27-01-2021 16:10, Mike Looijmans via lists.openembedded.org wrote:
>> When doing large builds, the GCC compiler tends to crash on random 
>> spots in the code. There are a few common denominators though.
>>
>> It only happens when compiling for aarch64 (cortex-A53), not for 
>> 32-bit arm (cortex-A9)
>>
>> It's random and usually happens on "big" sets like kernel, openssl, 
>> boost, u-boot etc.
>>
>> It always reports "during GIMPLE pass: ealias" in the error, for example:
>>
>> | during GIMPLE pass: ealias
>> | ../openssl-1.1.1i/crypto/x509v3/v3_utl.c: In function 'do_x509_check':
>> | ../openssl-1.1.1i/crypto/x509v3/v3_utl.c:1239:1: internal compiler 
>> error: Illegal instruction
>> | 1239 | }
>>
>> Compiling the same thing again usually goes fine.
>>
>> I've never experienced this with the zeus and older branches of OE.
>>
>>
>> I've already tried upgrading to the latest gatesgarth status, and 
>> cleaning out everything and start from scratch. I've also run "mprime" 
>> test on my machine (over one hour) just to be confident that the 
>> system itself is really okay.
>>
>>
>> Ideas to diagnose, fix or reliably reprodruce are more than welcome. 
> 
> 
> 
> 
> 


-- 
# Randy MacLeod
# Wind River Linux

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Mike Looijmans]

Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topicproducts.com

Please consider the environment before printing this e-mail
On 05-03-2021 00:32, Randy MacLeod wrote:
> On 2021-02-05 2:49 a.m., Mike Looijmans wrote:
>> I've been doing quite some tests and things last week, my conclusion so far 
>> is that the latest gcc seems to trigger a bug particular to this CPU. Maybe 
>> even this particular batch.
>>
>> Hours of testing found no issues with RAM.
>>
>> I've also done huge amounts of builds with zeus, no problems.
>>
>> Then googling for "illegal instruction gcc ryzen 1700" reveals quite a few 
>> threads on something that sounds suspiciously similar. Though it makes me 
>> wonder why I never encountered it before, since these are all many years 
>> old. I did try a bunch of the suggestions done there, but so far no luck.
>>
>>
> 
> Any update on this bug Mike?
> Have you opened a YP bug? If not please do.
> I'd like to see if the problem still happens on master.

So far I've discovered that this only happens when compiling for an aarch64. 
When targetting 32-bit ARM the compiler crash does not occur.

I've contacted AMD for an RMA for my CPU, but it's out of warranty now. I plan 
to borrow a compatible CPU to see if that makes the problem go away. If it 
does I'll just buy a new CPU...

As an example, if I take the current gatesgarth branch and meta-raspberrypi, 
then building for "raspberrypi4" runs without a hitch. If I chose 
"raspberrypi4-64" as MACHINE, the build will crash multiple times, typically 
on large components like the linux kernel, qt and boost. It happens on both 
plain C (kernel, u-boot) and on C++ code (boost, Qt). The same applies to e.g. 
the Xilinx Zynq, building for the 7-series (cortex A9) is fine but building 
for the MPSoC (cortex A53) will crash.

A silly workaround is to run "while ! bitbake ... ; do echo again; done" which 
will retry until it succeeds.

I'd be very interested if there's anyone with Ryzen setup who can reproduce 
this, or fail to, which is also interesting to know.

I also tried building in tmpfs and on a different SSD to rule out the NVME as 
a cause.

I'll try the master branch too, and report back on that (tomorrow probably).


> 
> ../Randy
> 
> 
>>
>> Met vriendelijke groet / kind regards,
>>
>> Mike Looijmans
>> System Expert
>>
>>
>> TOPIC Embedded Products B.V.
>> Materiaalweg 4, 5681 RJ Best
>> The Netherlands
>>
>> T: +31 (0) 499 33 69 69
>> E: mike.looijmans@topicproducts.com
>> W: www.topicproducts.com
>>
>> Please consider the environment before printing this e-mail
>> On 27-01-2021 16:10, Mike Looijmans via lists.openembedded.org wrote:
>>> When doing large builds, the GCC compiler tends to crash on random spots in 
>>> the code. There are a few common denominators though.
>>>
>>> It only happens when compiling for aarch64 (cortex-A53), not for 32-bit arm 
>>> (cortex-A9)
>>>
>>> It's random and usually happens on "big" sets like kernel, openssl, boost, 
>>> u-boot etc.
>>>
>>> It always reports "during GIMPLE pass: ealias" in the error, for example:
>>>
>>> | during GIMPLE pass: ealias
>>> | ../openssl-1.1.1i/crypto/x509v3/v3_utl.c: In function 'do_x509_check':
>>> | ../openssl-1.1.1i/crypto/x509v3/v3_utl.c:1239:1: internal compiler error: 
>>> Illegal instruction
>>> | 1239 | }
>>>
>>> Compiling the same thing again usually goes fine.
>>>
>>> I've never experienced this with the zeus and older branches of OE.
>>>
>>>
>>> I've already tried upgrading to the latest gatesgarth status, and cleaning 
>>> out everything and start from scratch. I've also run "mprime" test on my 
>>> machine (over one hour) just to be confident that the system itself is 
>>> really okay.
>>>
>>>
>>> Ideas to diagnose, fix or reliably reprodruce are more than welcome. 
>>
>>
>>
>> 
>>
> 
> 

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Max Krummenacher]

Hi Mike

Am Donnerstag, den 11.03.2021, 10:43 +0100 schrieb Mike Looijmans:
> Met vriendelijke groet / kind regards,
> 
> Mike Looijmans
> System Expert
> 
> 
> TOPIC Embedded Products B.V.
> Materiaalweg 4, 5681 RJ Best
> The Netherlands
> 
> T: +31 (0) 499 33 69 69
> E: mike.looijmans@topicproducts.com
> W: www.topicproducts.com
> 
> Please consider the environment before printing this e-mail
> On 05-03-2021 00:32, Randy MacLeod wrote:
> > On 2021-02-05 2:49 a.m., Mike Looijmans wrote:
> > > I've been doing quite some tests and things last week, my conclusion so far 
> > > is that the latest gcc seems to trigger a bug particular to this CPU. Maybe 
> > > even this particular batch.
> > > 
> > > Hours of testing found no issues with RAM.
> > > 
> > > I've also done huge amounts of builds with zeus, no problems.
> > > 
> > > Then googling for "illegal instruction gcc ryzen 1700" reveals quite a few 
> > > threads on something that sounds suspiciously similar. Though it makes me 
> > > wonder why I never encountered it before, since these are all many years 
> > > old. I did try a bunch of the suggestions done there, but so far no luck.
> > > 
> > > 
> > 
> > Any update on this bug Mike?
> > Have you opened a YP bug? If not please do.
> > I'd like to see if the problem still happens on master.
> 
> So far I've discovered that this only happens when compiling for an aarch64. 
> When targetting 32-bit ARM the compiler crash does not occur.
> 
> I've contacted AMD for an RMA for my CPU, but it's out of warranty now. I plan 
> to borrow a compatible CPU to see if that makes the problem go away. If it 
> does I'll just buy a new CPU...
> 
> As an example, if I take the current gatesgarth branch and meta-raspberrypi, 
> then building for "raspberrypi4" runs without a hitch. If I chose 
> "raspberrypi4-64" as MACHINE, the build will crash multiple times, typically 
> on large components like the linux kernel, qt and boost. It happens on both 
> plain C (kernel, u-boot) and on C++ code (boost, Qt). The same applies to e.g. 
> the Xilinx Zynq, building for the 7-series (cortex A9) is fine but building 
> for the MPSoC (cortex A53) will crash.
> 
> A silly workaround is to run "while ! bitbake ... ; do echo again; done" which 
> will retry until it succeeds.
> 
> I'd be very interested if there's anyone with Ryzen setup who can reproduce 
> this, or fail to, which is also interesting to know.
> 
> I also tried building in tmpfs and on a different SSD to rule out the NVME as 
> a cause.
> 
> I'll try the master branch too, and report back on that (tomorrow probably).
> 
> 
> > ../Randy
> > 
> > 
> > > Met vriendelijke groet / kind regards,
> > > 
> > > Mike Looijmans
> > > System Expert
> > > 
> > > 
> > > TOPIC Embedded Products B.V.
> > > Materiaalweg 4, 5681 RJ Best
> > > The Netherlands
> > > 
> > > T: +31 (0) 499 33 69 69
> > > E: mike.looijmans@topicproducts.com
> > > W: www.topicproducts.com
> > > 
> > > Please consider the environment before printing this e-mail
> > > On 27-01-2021 16:10, Mike Looijmans via lists.openembedded.org wrote:
> > > > When doing large builds, the GCC compiler tends to crash on random spots in 
> > > > the code. There are a few common denominators though.
> > > > 
> > > > It only happens when compiling for aarch64 (cortex-A53), not for 32-bit arm 
> > > > (cortex-A9)
> > > > 
> > > > It's random and usually happens on "big" sets like kernel, openssl, boost, 
> > > > u-boot etc.
> > > > 
> > > > It always reports "during GIMPLE pass: ealias" in the error, for example:
> > > > 
> > > > > during GIMPLE pass: ealias
> > > > > ../openssl-1.1.1i/crypto/x509v3/v3_utl.c: In function 'do_x509_check':
> > > > > ../openssl-1.1.1i/crypto/x509v3/v3_utl.c:1239:1: internal compiler error: 
> > > > Illegal instruction
> > > > > 1239 | }
> > > > 
> > > > Compiling the same thing again usually goes fine.
> > > > 
> > > > I've never experienced this with the zeus and older branches of OE.
> > > > 
> > > > 
> > > > I've already tried upgrading to the latest gatesgarth status, and cleaning 
> > > > out everything and start from scratch. I've also run "mprime" test on my 
> > > > machine (over one hour) just to be confident that the system itself is 
> > > > really okay.
> > > > 
> > > > 
> > > > Ideas to diagnose, fix or reliably reprodruce are more than welcome. 
> > > 
> > > 

For me a build of core-image-weston on a AMD Ryzen 7 3700X succeeded. The build run trough without
any hick-ups. openssl-1.1.1i is built as part of that image.
I used HEAD of today's gatesgarth.
So the issue seems to not affect all Ryzen CPUs.

Max


MACHINE=raspberrypi4-64 bitbake core-image-weston
WARNING: Host distribution "opensuseleap-15.2" has not been validated with this version of the build
system; you may possibly experience unexpected failures. It is recommended that you use a tested
distribution.
Loading cache: 100%
|                                                                                                   
                                                                            | ETA:  --:--:--
Loaded 0 entries from dependency cache.
Parsing recipes: 100% |#####################################################| Time: 0:00:43
Parsing of 2884 .bb files complete (0 cached, 2884 parsed). 4160 targets, 288 skipped, 0 masked, 0
errors.
WARNING: No recipes in default available for:
  /mnt/devel/oe-disk/oe-core_gatesgarth/build/../layers/meta-raspberrypi/recipes-
multimedia/gstreamer/gstreamer1.0-plugins-good_1.18.%.bbappend
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.49.2"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-tdx-linux"
MACHINE              = "raspberrypi4-64"
DISTRO               = "tdx-xwayland"
DISTRO_VERSION       = "0.0.0-devel-20210311125851+build.0"
TUNE_FEATURES        = "aarch64 armv8a crc crypto cortexa72"
TARGET_FPU           = ""
meta-toradex-nxp     = "HEAD:b8187ed55f204f50dea60c27f6cc79e9d2f1081b"
meta-freescale       = "gatesgarth:1acf098972f9f86b26eee815827667dbd2fcdeaf"
meta-freescale-3rdparty = "gatesgarth:efef93a0dc5be90c5bdc81dae252516887c95586"
meta-toradex-tegra   = "HEAD:627949f52b58d54ec2b5992b327bb33123cc53b2"
meta-toradex-bsp-common = "HEAD:e06c552ef433689b12caef0bf1f589d0cab180f9"
meta-raspberrypi     = "master:fdf02accfe3f68d11fa178d72eb377f9ca96411c"
meta-oe              
meta-filesystems     
meta-gnome           
meta-xfce            
meta-initramfs       
meta-networking      
meta-multimedia      
meta-python          = "gatesgarth:945f062ff098dc9c8ba8d22c5eef88adec60730d"
meta-freescale-distro = "gatesgarth:50eb2b32e7702bc435049bfe0a98fc65c864c106"
meta-toradex-demos   = "HEAD:4ffac94ed18c8701511385dcfc363a3051855fd7"
meta-qt5             = "gatesgarth:2b33a5d5e888370bb56685b86aa82b73624f19f0"
meta-toradex-distro  = "HEAD:6ddae219515fda5e6ffb0ad6a2fdc5430d9fa926"
meta-poky            = "gatesgarth:ac4a956f606d9cb4e025af80e6f4dcb74cbfd941"
meta                 = "gatesgarth:6311cb4930bd0add7aec61e5e0df6bb7ae0c4481"

Initialising tasks: 100%
|###########################################################################| Time: 0:00:04
Sstate summary: Wanted 1109 Found 142 Missed 967 Current 702 (12% match, 46% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 5358 tasks of which 2483 didn't need to be rerun and all succeeded.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 12 seconds

Summary: There were 2 WARNING messages shown.

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Mike Looijmans]

Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topicproducts.com

Please consider the environment before printing this e-mail
On 05-03-2021 00:32, Randy MacLeod wrote:
> On 2021-02-05 2:49 a.m., Mike Looijmans wrote:
>> I've been doing quite some tests and things last week, my conclusion 
>> so far is that the latest gcc seems to trigger a bug particular to 
>> this CPU. Maybe even this particular batch.
>>
>> Hours of testing found no issues with RAM.
>>
>> I've also done huge amounts of builds with zeus, no problems.
>>
>> Then googling for "illegal instruction gcc ryzen 1700" reveals quite 
>> a few threads on something that sounds suspiciously similar. Though 
>> it makes me wonder why I never encountered it before, since these are 
>> all many years old. I did try a bunch of the suggestions done there, 
>> but so far no luck.
>>
>>
>
> Any update on this bug Mike?
> Have you opened a YP bug? If not please do.
> I'd like to see if the problem still happens on master. 

Interesting.

I just built the 64-kernel using OE master and no crashes, everything 
went fine. With the gatesgarth branch, it'd crash at least two times 
during kernel compilation.

So something has been fixed since then?


-- 
Mike Looijmans

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Khem Raj]

On Thu, Mar 11, 2021 at 11:22 PM Mike Looijmans <mike.looijmans@topic.nl> wrote:
>
>
> Met vriendelijke groet / kind regards,
>
> Mike Looijmans
> System Expert
>
>
> TOPIC Embedded Products B.V.
> Materiaalweg 4, 5681 RJ Best
> The Netherlands
>
> T: +31 (0) 499 33 69 69
> E: mike.looijmans@topicproducts.com
> W: www.topicproducts.com
>
> Please consider the environment before printing this e-mail
> On 05-03-2021 00:32, Randy MacLeod wrote:
> > On 2021-02-05 2:49 a.m., Mike Looijmans wrote:
> >> I've been doing quite some tests and things last week, my conclusion
> >> so far is that the latest gcc seems to trigger a bug particular to
> >> this CPU. Maybe even this particular batch.
> >>
> >> Hours of testing found no issues with RAM.
> >>
> >> I've also done huge amounts of builds with zeus, no problems.
> >>
> >> Then googling for "illegal instruction gcc ryzen 1700" reveals quite
> >> a few threads on something that sounds suspiciously similar. Though
> >> it makes me wonder why I never encountered it before, since these are
> >> all many years old. I did try a bunch of the suggestions done there,
> >> but so far no luck.
> >>
> >>
> >
> > Any update on this bug Mike?
> > Have you opened a YP bug? If not please do.
> > I'd like to see if the problem still happens on master.
>
> Interesting.
>
> I just built the 64-kernel using OE master and no crashes, everything
> went fine. With the gatesgarth branch, it'd crash at least two times
> during kernel compilation.
>
> So something has been fixed since then?

can you try by disabling uninative ?

>
>
> --
> Mike Looijmans
>
>
> 
>

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Mike Looijmans]

Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topicproducts.com

Please consider the environment before printing this e-mail
On 12-03-2021 08:33, Khem Raj wrote:
> On Thu, Mar 11, 2021 at 11:22 PM Mike Looijmans <mike.looijmans@topic.nl> wrote:
>>
>> Met vriendelijke groet / kind regards,
>>
>> Mike Looijmans
>> System Expert
>>
>>
>> TOPIC Embedded Products B.V.
>> Materiaalweg 4, 5681 RJ Best
>> The Netherlands
>>
>> T: +31 (0) 499 33 69 69
>> E: mike.looijmans@topicproducts.com
>> W: www.topicproducts.com
>>
>> Please consider the environment before printing this e-mail
>> On 05-03-2021 00:32, Randy MacLeod wrote:
>>> On 2021-02-05 2:49 a.m., Mike Looijmans wrote:
>>>> I've been doing quite some tests and things last week, my conclusion
>>>> so far is that the latest gcc seems to trigger a bug particular to
>>>> this CPU. Maybe even this particular batch.
>>>>
>>>> Hours of testing found no issues with RAM.
>>>>
>>>> I've also done huge amounts of builds with zeus, no problems.
>>>>
>>>> Then googling for "illegal instruction gcc ryzen 1700" reveals quite
>>>> a few threads on something that sounds suspiciously similar. Though
>>>> it makes me wonder why I never encountered it before, since these are
>>>> all many years old. I did try a bunch of the suggestions done there,
>>>> but so far no luck.
>>>>
>>>>
>>> Any update on this bug Mike?
>>> Have you opened a YP bug? If not please do.
>>> I'd like to see if the problem still happens on master.
>> Interesting.
>>
>> I just built the 64-kernel using OE master and no crashes, everything
>> went fine. With the gatesgarth branch, it'd crash at least two times
>> during kernel compilation.
>>
>> So something has been fixed since then?
> can you try by disabling uninative ?

Probably, if only I knew what you mean by that?

What do I have to configure where?

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Mike Looijmans]

Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topicproducts.com

Please consider the environment before printing this e-mail
On 12-03-2021 08:37, Mike Looijmans via lists.openembedded.org wrote:
> On 12-03-2021 08:33, Khem Raj wrote:
>> On Thu, Mar 11, 2021 at 11:22 PM Mike Looijmans 
>> <mike.looijmans@topic.nl> wrote:
>>>
>>> On 05-03-2021 00:32, Randy MacLeod wrote:
>>>> On 2021-02-05 2:49 a.m., Mike Looijmans wrote:
>>>>> I've been doing quite some tests and things last week, my conclusion
>>>>> so far is that the latest gcc seems to trigger a bug particular to
>>>>> this CPU. Maybe even this particular batch.
>>>>>
>>>>> Hours of testing found no issues with RAM.
>>>>>
>>>>> I've also done huge amounts of builds with zeus, no problems.
>>>>>
>>>>> Then googling for "illegal instruction gcc ryzen 1700" reveals quite
>>>>> a few threads on something that sounds suspiciously similar. Though
>>>>> it makes me wonder why I never encountered it before, since these are
>>>>> all many years old. I did try a bunch of the suggestions done there,
>>>>> but so far no luck.
>>>>>
>>>>>
>>>> Any update on this bug Mike?
>>>> Have you opened a YP bug? If not please do.
>>>> I'd like to see if the problem still happens on master.
>>> Interesting.
>>>
>>> I just built the 64-kernel using OE master and no crashes, everything
>>> went fine. With the gatesgarth branch, it'd crash at least two times
>>> during kernel compilation.
>>>
>>> So something has been fixed since then?
>> can you try by disabling uninative ?
>
> Probably, if only I knew what you mean by that?
>
> What do I have to configure where?
>
As far as I can determine I think i'm not using unitative since I didn't 
explicitly enable it anywhere...

-- 
Mike Looijmans

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Khem Raj]

if you are using poky then its on by default you might have to disable 
it for this test

conf/distro/poky.conf:require conf/distro/include/yocto-uninative.inc
conf/distro/poky.conf:INHERIT += "uninative"


On 3/14/21 8:00 AM, Mike Looijmans wrote:
> 
> Met vriendelijke groet / kind regards,
> 
> Mike Looijmans
> System Expert
> 
> 
> TOPIC Embedded Products B.V.
> Materiaalweg 4, 5681 RJ Best
> The Netherlands
> 
> T: +31 (0) 499 33 69 69
> E: mike.looijmans@topicproducts.com
> W: www.topicproducts.com
> 
> Please consider the environment before printing this e-mail
> On 12-03-2021 08:37, Mike Looijmans via lists.openembedded.org wrote:
>> On 12-03-2021 08:33, Khem Raj wrote:
>>> On Thu, Mar 11, 2021 at 11:22 PM Mike Looijmans 
>>> <mike.looijmans@topic.nl> wrote:
>>>>
>>>> On 05-03-2021 00:32, Randy MacLeod wrote:
>>>>> On 2021-02-05 2:49 a.m., Mike Looijmans wrote:
>>>>>> I've been doing quite some tests and things last week, my conclusion
>>>>>> so far is that the latest gcc seems to trigger a bug particular to
>>>>>> this CPU. Maybe even this particular batch.
>>>>>>
>>>>>> Hours of testing found no issues with RAM.
>>>>>>
>>>>>> I've also done huge amounts of builds with zeus, no problems.
>>>>>>
>>>>>> Then googling for "illegal instruction gcc ryzen 1700" reveals quite
>>>>>> a few threads on something that sounds suspiciously similar. Though
>>>>>> it makes me wonder why I never encountered it before, since these are
>>>>>> all many years old. I did try a bunch of the suggestions done there,
>>>>>> but so far no luck.
>>>>>>
>>>>>>
>>>>> Any update on this bug Mike?
>>>>> Have you opened a YP bug? If not please do.
>>>>> I'd like to see if the problem still happens on master.
>>>> Interesting.
>>>>
>>>> I just built the 64-kernel using OE master and no crashes, everything
>>>> went fine. With the gatesgarth branch, it'd crash at least two times
>>>> during kernel compilation.
>>>>
>>>> So something has been fixed since then?
>>> can you try by disabling uninative ?
>>
>> Probably, if only I knew what you mean by that?
>>
>> What do I have to configure where?
>>
> As far as I can determine I think i'm not using unitative since I didn't 
> explicitly enable it anywhere...
> 

Re: [OE-core] GCC crashes on aarch64 since gatesgarth

[Mike Looijmans]

Ancient thread, but might be of interest to others.

A while ago the power supply died. After replacing the PSU, the error 
below hasn't occurred any more.

I'm not 100% certain, but it's very likely the issue was caused by that 
PSU on the brink of failing completely. Not the first thing to come to 
mind...

Apparently gatesgarth is more power demanding than hardknott :)

Kind regards,
Mike


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topic.nl

Please consider the environment before printing this e-mail
On 27-01-2021 16:10, Mike Looijmans via lists.openembedded.org wrote:
> When doing large builds, the GCC compiler tends to crash on random 
> spots in the code. There are a few common denominators though.
>
> It only happens when compiling for aarch64 (cortex-A53), not for 
> 32-bit arm (cortex-A9)
>
> It's random and usually happens on "big" sets like kernel, openssl, 
> boost, u-boot etc.
>
> It always reports "during GIMPLE pass: ealias" in the error, for example:
>
> | during GIMPLE pass: ealias
> | ../openssl-1.1.1i/crypto/x509v3/v3_utl.c: In function 'do_x509_check':
> | ../openssl-1.1.1i/crypto/x509v3/v3_utl.c:1239:1: internal compiler 
> error: Illegal instruction
> | 1239 | }
>
> Compiling the same thing again usually goes fine.
>
> I've never experienced this with the zeus and older branches of OE.
>
>
> I've already tried upgrading to the latest gatesgarth status, and 
> cleaning out everything and start from scratch. I've also run "mprime" 
> test on my machine (over one hour) just to be confident that the 
> system itself is really okay.
>
>
> Ideas to diagnose, fix or reliably reprodruce are more than welcome.
>
>
> 
>

-- 
Mike Looijmans