Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000]

Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000]

[Andrew Cooper]

On 18/07/2022 22:31, chris@dalessio.org wrote:
> I am trying to run Xen-4.16.1-4.fc36 on Fedora 36 on a brand new Lenovo
> ThinkStation p620, but I keep getting the following error booting the
> Xen kernel.
> 
> Panic on CPU 0:
> FATAL TRAP: vec 7, #NM[0000]
>
> Version info:
> Name        : xen
> Version     : 4.16.1
> Release     : 4.fc36

So https://koji.fedoraproject.org/koji/buildinfo?buildID=1991182 should
be the binary build in use, and looking at the debug syms, it really
does have:

ffff82d040439c80 <amd_iommu_init>:
...
ffff82d04043a00c:       0f 6e c2                movd   %edx,%mm0
ffff82d04043a00f:       0f 62 c0                punpckldq %mm0,%mm0
ffff82d04043a012:       49 89 87 c0 00 00 00    mov    %rax,0xc0(%r15)
ffff82d04043a019:       41 0f 7f 87 d0 00 00    movq   %mm0,0xd0(%r15)
ffff82d04043a020:       00

So hardware is correct - this build of Xen is nonsense.

The binary is also full of .annobin_ stuff which appears to be some kind
of GCC plugin for watermarking.

Michael: Any idea what's going on here?  Something has caused GCC to
emit some MMX logic which is ultimately why things exploded, but this
probably means that some of the build CFLAGS got dropped.

Thanks,

~Andrew

Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000]

[ChrisD]

[-- Attachment #1: Type: text/plain, Size: 1609 bytes --]

So, you think it's a problem with fc36?
-------- Original message --------From: Andrew Cooper <Andrew.Cooper3@citrix.com> Date: 7/18/22  6:25 PM  (GMT-05:00) To: chris@dalessio.org, xen-devel@lists.xenproject.org Cc: Jan Beulich <jbeulich@suse.com>, Michael Young <m.a.young@durham.ac.uk> Subject: Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000] On 18/07/2022 22:31, chris@dalessio.org wrote:> I am trying to run Xen-4.16.1-4.fc36 on Fedora 36 on a brand new Lenovo> ThinkStation p620, but I keep getting the following error booting the> Xen kernel.> > Panic on CPU 0:> FATAL TRAP: vec 7, #NM[0000]>> Version info:> Name        : xen> Version     : 4.16.1> Release     : 4.fc36So https://koji.fedoraproject.org/koji/buildinfo?buildID=1991182 shouldbe the binary build in use, and looking at the debug syms, it reallydoes have:ffff82d040439c80 <amd_iommu_init>:...ffff82d04043a00c:       0f 6e c2                movd   %edx,%mm0ffff82d04043a00f:       0f 62 c0                punpckldq %mm0,%mm0ffff82d04043a012:       49 89 87 c0 00 00 00    mov    %rax,0xc0(%r15)ffff82d04043a019:       41 0f 7f 87 d0 00 00    movq   %mm0,0xd0(%r15)ffff82d04043a020:       00So hardware is correct - this build of Xen is nonsense.The binary is also full of .annobin_ stuff which appears to be some kindof GCC plugin for watermarking.Michael: Any idea what's going on here?  Something has caused GCC toemit some MMX logic which is ultimately why things exploded, but thisprobably means that some of the build CFLAGS got dropped.Thanks,~Andrew

[-- Attachment #2: Type: text/html, Size: 2430 bytes --]

Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000]

[Jan Beulich]

On 20.07.2022 06:48, ChrisD wrote:
> So, you think it's a problem with fc36?

Well, if that's where the binary came from, then yes.

And btw please don't top-post and please don't send html mail
(see the almost unreadable stuff below that was how your mail
came through when read as plain-text).

Jan

> -------- Original message --------From: Andrew Cooper <Andrew.Cooper3@citrix.com> Date: 7/18/22  6:25 PM  (GMT-05:00) To: chris@dalessio.org, xen-devel@lists.xenproject.org Cc: Jan Beulich <jbeulich@suse.com>, Michael Young <m.a.young@durham.ac.uk> Subject: Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000] On 18/07/2022 22:31, chris@dalessio.org wrote:> I am trying to run Xen-4.16.1-4.fc36 on Fedora 36 on a brand new Lenovo> ThinkStation p620, but I keep getting the following error booting the> Xen kernel.> > Panic on CPU 0:> FATAL TRAP: vec 7, #NM[0000]>> Version info:> Name        : xen> Version     : 4.16.1> Release     : 4.fc36So https://koji.fedoraproject.org/koji/buildinfo?buildID=1991182 shouldbe the binary build in use, and looking at the debug syms, it reallydoes have:ffff82d040439c80 <amd_iommu_init>:...ffff82d04043a00c:       0f 6e c2                movd   %edx,%mm0ffff82d04043a00f:       0f 62 c0                punpckldq %mm0,%mm0ffff82d04043a012:       49 89 87 c0 00 00 00    mov    %rax,0xc0(%r15)ffff82d04043a019:       41 0f 7f 87 d0 00 00    movq   %mm0,0xd0(%r15)ffff82d04043a020:       00So hardware is correct - this build of Xen is nonsense.The binary is also full of .annobin_ stuff which appears to be some kindof GCC plugin for watermarking.Michael: Any idea what's going on here?  Something has caused GCC toemit some MMX logic which is ultimately why things exploded, but thisprobably means that some of the build CFLAGS got dropped.Thanks,~Andrew

Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000]

[Andrew Cooper]

On 20/07/2022 09:19, Jan Beulich wrote:
> On 20.07.2022 06:48, ChrisD wrote:
>> So, you think it's a problem with fc36?
> Well, if that's where the binary came from, then yes.

So
https://kojipkgs.fedoraproject.org//packages/xen/4.16.1/4.fc36/data/logs/x86_64/build.log
is the build log.

For iommu_init.c I don't see anything overly concerning, although the
quantity of nonsense on the gcc cmdline is speaks volumes.

One observation though.  We do pass -mno-sse but not -mno-mmx.  I still
can't figure out what makes the compiler think there's any SIMD to be
done in this function.

~Andrew

Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000]

[Jan Beulich]

On 20.07.2022 12:02, Andrew Cooper wrote:
> On 20/07/2022 09:19, Jan Beulich wrote:
>> On 20.07.2022 06:48, ChrisD wrote:
>>> So, you think it's a problem with fc36?
>> Well, if that's where the binary came from, then yes.
> 
> So
> https://kojipkgs.fedoraproject.org//packages/xen/4.16.1/4.fc36/data/logs/x86_64/build.log
> is the build log.
> 
> For iommu_init.c I don't see anything overly concerning, although the
> quantity of nonsense on the gcc cmdline is speaks volumes.
> 
> One observation though.  We do pass -mno-sse but not -mno-mmx.

Right, but that should be no problem - the compiler isn't supposed
to enable MMX without -mmmx.

Jan

>  I still
> can't figure out what makes the compiler think there's any SIMD to be
> done in this function.
> 
> ~Andrew

Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000]

[Jan Beulich]

On 20.07.2022 12:02, Andrew Cooper wrote:
> On 20/07/2022 09:19, Jan Beulich wrote:
>> On 20.07.2022 06:48, ChrisD wrote:
>>> So, you think it's a problem with fc36?
>> Well, if that's where the binary came from, then yes.
> 
> So
> https://kojipkgs.fedoraproject.org//packages/xen/4.16.1/4.fc36/data/logs/x86_64/build.log
> is the build log.
> 
> For iommu_init.c I don't see anything overly concerning, although the
> quantity of nonsense on the gcc cmdline is speaks volumes.

Hmm, I wouldn't call two uses of -specs= entirely unconcerning. About
anything can be enabled/disabled there.

Jan

Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000]

[Jan Beulich]

On 20.07.2022 12:02, Andrew Cooper wrote:
> One observation though.  We do pass -mno-sse but not -mno-mmx.  I still
> can't figure out what makes the compiler think there's any SIMD to be
> done in this function.

So this looks to be "optimization", done in a few more places. The pattern
is always the same: A 32-bit GPR is known to be zero, and there's nearby
code which wants to store two adjacent zeros. Hence they take those 32
bits of zero in the GPR, move to %mm0 (which already zeros the upper half),
unpack it to have the 32 bits of zero duplicated into the upper half, and
then use %mm0 to do the store of the pair of zeros. IOW they "auto-
vectorize" these two stores into a single V2SI (using the common notation)
one.

Besides this being quite the opposite of optimization, of course we didn't
tell the compiler anywhere that it might use any of the %mm<N> registers.

Jan

Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000]

[Jan Beulich]

On 20.07.2022 13:28, Jan Beulich wrote:
> On 20.07.2022 12:02, Andrew Cooper wrote:
>> On 20/07/2022 09:19, Jan Beulich wrote:
>>> On 20.07.2022 06:48, ChrisD wrote:
>>>> So, you think it's a problem with fc36?
>>> Well, if that's where the binary came from, then yes.
>>
>> So
>> https://kojipkgs.fedoraproject.org//packages/xen/4.16.1/4.fc36/data/logs/x86_64/build.log
>> is the build log.
>>
>> For iommu_init.c I don't see anything overly concerning, although the
>> quantity of nonsense on the gcc cmdline is speaks volumes.
>>
>> One observation though.  We do pass -mno-sse but not -mno-mmx.
> 
> Right, but that should be no problem - the compiler isn't supposed
> to enable MMX without -mmmx.

Or so I - wrongly - thought. Will make a patch. Nevertheless I consider
gcc behavior here close to insane.

Jan

Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000]

[Jan Beulich]

On 20.07.2022 06:48, ChrisD wrote:
> So, you think it's a problem with fc36?

Just to also state it here - it turns out we need to tell the compiler
to avoid MMX insns. A patch for this was already sent and ack-ed.

Jan

Re: Panic on CPU 0: FATAL TRAP: vec 7, #NM[0000]

[Michael Young]

On Wed, 20 Jul 2022, Jan Beulich wrote:

> On 20.07.2022 06:48, ChrisD wrote:
>> So, you think it's a problem with fc36?
>
> Just to also state it here - it turns out we need to tell the compiler
> to avoid MMX insns. A patch for this was already sent and ack-ed.

I have done a temporary build of xen for F36 at

http://koji.fedoraproject.org/koji/taskinfo?taskID=89746549

which has the proposed patch applied if anyone wants to test it.

 	Michael Young