[MODERATED] eager FPU backport for 2.6.32

[MODERATED] eager FPU backport for 2.6.32

[Paolo Bonzini]

For the lucky souls who have to backport eager FPU support to 2.6.32, 
and at the risk of being larted by Thomas :) here is my current list of 
FPU patches on top of 2.6.32.  I tried to include also those that were 
already in RHEL before I started the backport.

It's basically everything up to (and not including) Ingo's 200-patch 
cleanup.

Paolo

5b3efd500854d45d305b53c54c97db5970959980 x86, ptrace: regset extensions to support xstate
ff7fbc72e0c3ef7e94a27a3a918fd09ec9a30204 x86, ptrace: Simplify xstateregs_get()
6dbbe14f21368a45aedba7eab0221857b8ad8d16 x86, ptrace: Remove set_stopped_child_used_math() in [x]fpregs_set
	commit 5a0e3ad6af8660be21ca98a971cd00f331318c05 NOT backported (treewide cleanup)
dce8bf4e115aa44d590802ce3554e926840c9042 x86, fpu: Use the proper asm constraint in use_xsave()
c9ad488289144ae5ef53b012e15895ef1f5e4bb6 x86: Eliminate TS_XSAVE
86603283326c9e95e5ad4e9fdddeec93cac5d9ad x86: Introduce 'struct fpu' and related API
82d4150cec83b9775f84810b39a1c0b91585d429 x86, xsave: Move boot cpu initialization to xsave_init()
0e49bf66d2ca649b167428adddbbbe9d9bd4894c x86, xsave: Separate fpu and xsave initialization
c9775b4cc522e5f1b40b1366a993f0f05f600f39 x86, fpu: Use static_cpu_has() to implement use_xsave()
d6d4d4205cf4ce4ba13bc320305afbda25303496 x86, xsave: Cleanup return codes in check_for_xstate()
8e221b6db4477643fefc885a97ea9889ac733140 x86: Avoid unnecessary __clear_user() and xrstor in signal handling
a1488f8bf4d72ad724700f6e982469a1240e4264 x86, xsave: Track the offset, size of state in the xsave layout
29104e101d710dd152f807978884643a52eca8b7 x86, xsave: Sync xsave memory layout with its header for user handling
6bad06b768920e278c7cedfdda56a0b4c6a35ee9 x86, xsave: Use xsaveopt in context-switch path when supported
7aa2b5f8ec60505160df1c25398e8286c8432689 x86, xsave: Do not include asm/i387.h in asm/xsave.h
97e80a70db689fb1e876df9f12305cc72f85ca53 x86, xsave: Introduce xstate enable functions
ee813d53a8e980a3a28318efb8935d45723f5211 x86, xsave: Check cpuid level for XSTATE_CPUID (0x0d)
45c2d7f46211a0b1f6b425c59575c53145afc4b4 x86, xsave: Make init_xstate_buf static
4995b9dba908436c1611454f9bd2cb3ddf6babee x86, xsave: Add __init attribute to setup_xstate_features()
1cff92d8fdb27684308864d9cdb324bee43b40ab x86, xsave: Make xstate_enable_boot_cpu() __init, protect on CPU 0
5ee481da7b62a992b91f958bf26aaaa92354c170 x86: Export FPU API for KVM use
2d5b5a665508c60577c1088e0405850a965b6795 KVM: x86: XSAVE/XRSTOR live migration support
1f999ab5a1360afc388868cc0ef9afe8edeef3be x86, xsave: Disable xsave in i387 emulation mode
f45755b8346f1a089ca7957dce5f4c3c6cb26e6f KVM: fix poison overwritten caused by using wrong xstate size
6ac8bac2684235f4caf22a410549c582aa7327d6 x86, fpu: Merge fpu_init()
51115d4d45700fc7c08306f7ba6e68551f526ae5 x86, fpu: Merge tolerant_fwait()
bfd946cb891800d408decaae268a3480775178a3 x86, fpu: Merge __save_init_fpu()
a4d4fbc7735bba6654b20f859135f9d3f8fe7f76 x86-64, fpu: Disable preemption when using TS_USEDFPU
10c11f304986a1f84201c2261a428701f9d2dffc x86-64, fpu: Fix %cs value in convert_from_fxsr()
820241356d6aa9a895fc10def15794a5a5bfcd98 x86-64, fpu: Simplify constraints for fxsave/fxtstor
8eb91a577d7763d21628f6761045328784b1911c x86, fpu: Remove unnecessary ifdefs from i387 code.
eec73f813ab0954253e5e2168119c4555f83f07d x86, fpu: Remove PSHUFB_XMM5_* macros
58a992b9cbaf449aeebd3575c3695a9eb5d95b5e x86-32, fpu: Rewrite fpu_save_init()
b2b57fe053c9cf8b8af5a0e826a465996afed0ff x86, fpu: Merge fpu_save_init()
	commit d7acb92fea932ad2e7846480aeacddc2c03c8485 NOT backported (could be)
	commit fd35fbcdd1b2579a6e00a1545f7124e4005d0474 NOT backported (could be)
10340ae130fb70352eae1ae8a00b7906d91bf166 x86, xsave: Use alloc_bootmem_align() instead of alloc_bootmem()
e5c301428294cb8925667c9ee39f817c4ab1c2c9 KVM: Initialize fpu state in preemptible context
	commit 0d2eb44f631d9d0a826efa3156f157477fdaecf4 partially backported (treewide cleanup)
	commit 497888cf69bf607ac1fe061a6437e0a670b0022f partially backported (treewide cleanup)
be98c2cdb15ba26148cd2bd58a857d4f7759ed38 i387: math_state_restore() isn't called from asm
5b1cbac37798805c1fee18c8cebe5c0a13975b17 i387: make irq_fpu_usable() tests more robust
c38e23456278e967f094b08247ffc3711b1029b2 i387: fix sense of sanity check
15d8791cae75dca27bfda8ecfe87dca9379d6bb0 i387: fix x86-64 preemption-unsafe user stack save/restore
b6c66418dcad0fcf83cd1d0a39482db37bf4fc41 i387: move TS_USEDFPU clearing out of __save_init_fpu and into callers
6d59d7a9f5b723a7ac1925c136e93ec83c0c3043 i387: don't ever touch TS_USEDFPU directly, use helper functions
b3b0870ef3ffed72b92415423da864f440f57ad6 i387: do not preload FPU state at task switch time
4903062b5485f0e2c286a23b44c9b59d9b017d53 i387: move AMD K7/K8 fpu fxsave/fxrstor workaround from save to restore
f94edacf998516ac9d849f7bc6949a703977a7f3 i387: move TS_USEDFPU flag from thread_info to task_struct
34ddc81a230b15c0e345b6b253049db731499f7e i387: re-introduce FPU state preloading at context switch time
cea20ca3f3181fc36788a15bc65d1062b96a0a6c i387: fix up some fpu_counter confusion
80ab6f1e8c981b1b6604b2f22e36c917526235cd i387: use 'restore_fpu_checking()' directly in task switching code
	commit 7e16838d94b566a17b65231073d179bc04d590c8 minimal backport (without lazy restore support)
	commit 27e74da9800289e69ba907777df1e2085231eff7 NOT backported (no lazy restore)
8546c008924d5fd1724fa698eaa92b414bafd50d i387: Uninline the generic FP helpers that we expose to kernel modules
1361b83a13d4d92e53fbb6c877528713e118b821 i387: Split up <asm/i387.h> into exported and internal interfaces
	commit 089f9fba56faf33cc6dd2a6442b7ac92c58b8209 NOT backported (no lazy restore)
7a040a4384c7c4973deb4d58a76e1b0ee3c8aa39 x86, extable: Remove open-coded exception table entries in arch/x86/include/asm/xsave.h
	commit c6ae41e7d469f00d9c92a2b2887c7235d121c009 NOT backported (no lazy restore)
050902c011712ad4703038fa4489ec4edd87d396 x86, signal: Cleanup ifdefs and is_ia32, is_x32
d75f1b391f5ef73016d14bc6f7e4725820ebaa5b x86, xsave: remove thread_has_fpu() bug check in __sanitize_i387_state()
	commit c767a54ba0657e52e6edaa97cbe0b0a8bf1c1655 partially backported (treewide cleanup)
5c7d03e99cb1ed449328ed9fba0c632944d39e7e x86/fpu/xsave: Keep __user annotation in casts
0ca5bd0d886578ad0afeceaa83458c0f35cb3c6b x86, fpu: Consolidate inline asm routines for saving/restoring fpu state
0ff8fef4eaf252ee13a2d0b175a8c876415bd62a x86/signals: ia32_signal.c: add __user casts to fix sparse warnings
72a671ced66db6d1c2bfff1c930a101ac8d08204 x86, fpu: Unify signal handling code paths for x86 and x86_64 kernels
e962591749dfd4df9fea2c530ed7a3cfed50e5aa x86, fpu: drop_fpu() before restoring new state from sigframe
377ffbcc536a5a6666dc077395163ab149c02610 x86, fpu: remove unnecessary user_fpu_end() in save_xstate_sig()
304bceda6a18ae0b0240b8aac9a6bdf8ce2d2469 x86, fpu: use non-lazy fpu restore for processors supporting xsave
5d2bd7009f306c82afddd1ca4d9763ad8473c216 x86, fpu: decouple non-lazy/eager fpu restore from xsave
212b02125f3725127148475059a70031bd031bdb x86, fpu: enable eagerfpu by default for xsaveopt
e00229819f306b1f86134095347e9187dc346bd1 x86, fpu: make eagerfpu= boot param tri-state
	commit 63bcff2a307b9bcc712a8251eb27df8b2e117967 NOT backported (SMAP)
b1a74bf8212367be2b1d6685c11a84e056eaaaf1 x86, kvm: fix kvm's usage of kernel_fpu_begin/end()
	commit e139e95590dfebab81841bf7a3ac46500f51a47c NOT backported (SMAP)
6f5298c2139b06925037490367906f3d73955b86 x86/i387.c: Initialize thread xstate only on CPU0 only once
5187b28ff08249ab8a162e802209ed04e271ca02 x86: Allow FPU to be used at interrupt time even with eagerfpu
66463db4fc5605d51c7bb81d009d5bf30a783a2c x86, fpu: shift drop_init_fpu() from save_xstate_sig() to handle_signal()
9b6dba9e0798325dab427b9d60c61630ccc39b28 x86: Merge simd_math_error() into math_error()
a9241ea5fd709fc935dade130f4e3b2612bbe9e3 x86/fpu: Don't reset thread.fpu_counter
1a2a7f4ec8e3a7ac582dac4d01fcc7e8acd3bb30 x86/fpu: Don't do __thread_fpu_end() if use_eager_fpu()
08a744c6bfded3d5fa66f94263f81773226113d1 x86/fpu: Change math_error() to use unlazy_fpu(), kill (now) unused save_init_fpu()
731bd6a93a6e9172094a2322bd0ee964bb1f4d63 x86, fpu: Check tsk_used_math() in kernel_fpu_end() for eager FPU
14e153ef75eecae8fd0738ffb42120f4962a00cd x86, fpu: Introduce per-cpu in_kernel_fpu state
33a3ebdc077fd85f1bf4d4586eea579b297461ae x86, fpu: Don't abuse has_fpu in __kernel_fpu_begin/end()
	commit 728e53fef429a0f3c9dda3587c3ccc57ad268b70 NOT backported (no lazy FPU restore)
	commit 7aeccb83e76316b365e4b44a1dd982ee22a7d8b2 NOT backported (no lazy FPU restore)
110d7f7513bbb916b8654da9e2973ac5bed929a9 x86/fpu: Don't abuse FPU in kernel threads if use_eager_fpu()
e7f180dcd8ab48f18b20d7e8a7e9b39192bdf8e0 x86/fpu: Change xstateregs_get()/set() to use ->xsave.i387 rather than ->fxsave
1d23c4518b1f3a03c278f23333149245c178d2a6 x86/fpu: Factor out memset(xstate, 0) in fpu_finit() paths
fb14b4eadf73500d3b2104f031472a268562c047 x86/fpu: Document user_fpu_begin()
8f4d81863ba4e8dfee93bd50840f1099a296251f x86/fpu: Introduce restore_init_xstate()
9cb6ce823bbd1adbe15e30bd1435c84c2e271767 x86/fpu: Use restore_init_xstate() instead of math_state_restore() on kthread exec
f893959b0898bd876673adbeb6798bdf25c034d7 x86/fpu: Don't abuse drop_init_fpu() in flush_thread()
d2d0ac9a4644e00120bb9b7427a512a99d2cacc5 x86/fpu: Fold __drop_fpu() into its sole user
7575637ab293861a799f3bbafe0d8c597389f4e9 x86, fpu: Fix math_state_restore() race with kernel_fpu_begin()
b85e67d1483c72b77d1bdc265aa8ba91590794c1 x86/fpu: Rename drop_init_fpu() to fpu_reset_state()
c88d47480d300eaad80c213d50c9bf6077fc49bc x86/fpu: Always restore_xinit_state() when use_eager_cpu()

[MODERATED] Re: eager FPU backport for 2.6.32

[Linus Torvalds]

On Mon, 11 Jun 2018, speck for Paolo Bonzini wrote:
>
> For the lucky souls who have to backport eager FPU support to 2.6.32, 
> and at the risk of being larted by Thomas :) here is my current list of 
> FPU patches on top of 2.6.32.  I tried to include also those that were 
> already in RHEL before I started the backport.

Please, why are we doing this crazy thing?

Who really cares about 2.6.32? The likelihood that they have any security 
at all is basically NIL, so why care about it for some odd x86 hardware 
bug?

Maybe 2.6.32 is used in the embedded world, but if you allow people to run 
random binaries on it, you're f*cking insane. Why not tell people that?

             Linus

[MODERATED] Re: eager FPU backport for 2.6.32

[Peter Zijlstra]

On Mon, Jun 11, 2018 at 08:18:53AM -0700, speck for Linus Torvalds wrote:
> Please, why are we doing this crazy thing?

RHEL6 extended support contract I suspect... but yes, that's marketing
speak for what you said :-)

[MODERATED] Re: eager FPU backport for 2.6.32

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 691 bytes --]

On 11/06/2018 17:44, speck for Peter Zijlstra wrote:
> On Mon, Jun 11, 2018 at 08:18:53AM -0700, speck for Linus Torvalds wrote:
>> Please, why are we doing this crazy thing?
> RHEL6 extended support contract I suspect... but yes, that's marketing
> speak for what you said :-)

Yeah it's crazy but it helps paying my meals and quite likely at least
Oracle and SuSE care too.  It's not even a particularly hard backport,
just boring.  PTI was backported to 2.6.18 even, that was more interesting.

Perhaps Linus was thinking of upstream 2.6.32 stable, I agree that's not
particularly interesting since it doesn't even have XSAVE(OPT) support
but RHEL of course does.

Paolo

[MODERATED] Re: eager FPU backport for 2.6.32

[Josh Poimboeuf]

On Mon, Jun 11, 2018 at 05:44:15PM +0200, speck for Peter Zijlstra wrote:
> On Mon, Jun 11, 2018 at 08:18:53AM -0700, speck for Linus Torvalds wrote:
> > Please, why are we doing this crazy thing?
> 
> RHEL6 extended support contract I suspect... but yes, that's marketing
> speak for what you said :-)

Welcome to our nightmare.  And let's not even mention RHEL5...

-- 
Josh

[MODERATED] Re: eager FPU backport for 2.6.32

[Linus Torvalds]

On Mon, 11 Jun 2018, speck for Josh Poimboeuf wrote:
> 
> Welcome to our nightmare.  And let's not even mention RHEL5...

Heh.

Does the support contract really say "no package version changes"?

Because after all it _is_ called "RHEL6", not "RHEL2.6.32".

At some point, backporting stuff is just way more dangerous than just 
upgrading to a well-tested kernel. It's not like you have to go bleeding 
edge. 4.4 is at what, stable sub-version 130 by now?

Or even 3.16, for chrissake.

But whatever. I guess _I_ don't have to care [ evil cackling ], and I 
guess it gives you guys job security.

I just hope RH covers mental services too.

            Linus

[MODERATED] Re: eager FPU backport for 2.6.32

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

On 11/06/2018 17:59, speck for Linus Torvalds wrote:
> 
> On Mon, 11 Jun 2018, speck for Josh Poimboeuf wrote:
>> Welcome to our nightmare.  And let's not even mention RHEL5...
> Heh.
> 
> Does the support contract really say "no package version changes"?

No, but there are some (not full of course) guarantees about drivers ABI
(not just API), and given stable-api-nonsense.txt there aren't many ways
to do it.  Not that I disagree with stable-api-nonsense.txt, of course.

> Because after all it _is_ called "RHEL6", not "RHEL2.6.32".
> 
> At some point, backporting stuff is just way more dangerous than just 
> upgrading to a well-tested kernel.

It certainly is, but things get progressively calmer---and until they
don't, the backporting pace is crazy enough that conflicts are pretty
rare.  It's somewhat self-regulating; at some point people start seeing
conflicts, that causes push back against new features, and luckily the
pointy-haired people do listen.

It usually works out fine, except when people discover everything is
broken in your CPUs and you have to backport 300-ish core MM and arch
patches eight years into the product lifetime.  That's when the job
security part kicks in.

Paolo

[MODERATED] Re: eager FPU backport for 2.6.32

[Jiri Kosina]

On Mon, 11 Jun 2018, speck for Paolo Bonzini wrote:

> >> Please, why are we doing this crazy thing?
> > RHEL6 extended support contract I suspect... but yes, that's marketing
> > speak for what you said :-)
> 
> Yeah it's crazy but it helps paying my meals and quite likely at least
> Oracle and SuSE care too.  

Absolutely. We have 3.0 backport of eager FPU stuff ready, but as 2.6.32 
patches are super-set of it, it's probably pointless to send it here at 
this point.

> It's not even a particularly hard backport, just boring.  PTI was 
> backported to 2.6.18 even, that was more interesting.

There are people on this list who even did 2.6.16 PTI port :)

On Mon, 11 Jun 2018, speck for Paolo Bonzini wrote:

> > Does the support contract really say "no package version changes"?

> No, but there are some (not full of course) guarantees about drivers ABI 
> (not just API), and given stable-api-nonsense.txt there aren't many ways 
> to do it.  Not that I disagree with stable-api-nonsense.txt, of course.

Also, each and every time we start talking about changing major kernel 
version (or any other reasonably core package), we get pushback along the 
lines of "that would invalidate all the certifications we have, and it'd 
cost us a fortune to recertify everything with 3rd parties and such".

No matter how crazy and irelevant on technical level that sounds, welcome 
to the enterprise universe :/

-- 
Jiri Kosina
SUSE Labs

[MODERATED] Re: eager FPU backport for 2.6.32

[Jiri Kosina]

On Mon, 11 Jun 2018, speck for Paolo Bonzini wrote:

> For the lucky souls who have to backport eager FPU support to 2.6.32, 
> and at the risk of being larted by Thomas :) here is my current list of 
> FPU patches on top of 2.6.32.  I tried to include also those that were 
> already in RHEL before I started the backport.

Hi Paolo,

have have quite divergent eager FPU switching backports to 3.0 and 4.4 
(where 4.4 is basically cherry-picked fixes from upstream, while 3.0 is 
mostly a divergent minimalistic implementation), and for *both* codestream 
we're receiving reports of Oracle database segfaulting in a way that looks 
like register / memory corruption. We're not seeing any errors / reports 
with any other userspace excercising FPU.

Have you guys at RH by chance received any such reports after switching 
older kernels to eager FPU switching?

Thanks,

-- 
Jiri Kosina
SUSE Labs

[MODERATED] Re: eager FPU backport for 2.6.32

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 5521 bytes --]

On 31/07/2018 09:35, speck for Jiri Kosina wrote:
> 
>> For the lucky souls who have to backport eager FPU support to 2.6.32, 
>> and at the risk of being larted by Thomas :) here is my current list of 
>> FPU patches on top of 2.6.32.  I tried to include also those that were 
>> already in RHEL before I started the backport.
> Hi Paolo,
> 
> have have quite divergent eager FPU switching backports to 3.0 and 4.4 
> (where 4.4 is basically cherry-picked fixes from upstream, while 3.0 is 
> mostly a divergent minimalistic implementation), and for *both* codestream 
> we're receiving reports of Oracle database segfaulting in a way that looks 
> like register / memory corruption. We're not seeing any errors / reports 
> with any other userspace excercising FPU.
> 
> Have you guys at RH by chance received any such reports after switching 
> older kernels to eager FPU switching?

No, but we had bugs with signals with the patch list I sent earlier,
so here is a list of extra patches that we added on top of that list.

a9241ea5fd709fc935dade130f4e3b2612bbe9e3
    x86/fpu: Don't reset thread.fpu_counter

1a2a7f4ec8e3a7ac582dac4d01fcc7e8acd3bb30
    x86/fpu: Don't do __thread_fpu_end() if use_eager_fpu()

9b6dba9e0798325dab427b9d60c61630ccc39b28
    x86: Merge simd_math_error() into math_error()

08a744c6bfded3d5fa66f94263f81773226113d1
    x86/fpu: Change math_error() to use unlazy_fpu(), kill (now) unused save_init_fpu()

731bd6a93a6e9172094a2322bd0ee964bb1f4d63
    x86, fpu: Check tsk_used_math() in kernel_fpu_end() for eager FPU

14e153ef75eecae8fd0738ffb42120f4962a00cd
    x86, fpu: Introduce per-cpu in_kernel_fpu state

33a3ebdc077fd85f1bf4d4586eea579b297461ae
    x86, fpu: Don't abuse has_fpu in __kernel_fpu_begin/end()

4b2e762e2e53c721458a83d547b222178bb72a34
    x86/fpu: Always allow FPU in interrupt if use_eager_fpu()

e7f180dcd8ab48f18b20d7e8a7e9b39192bdf8e0
    x86/fpu: Change xstateregs_get()/set() to use ->xsave.i387 rather than ->fxsave

1d23c4518b1f3a03c278f23333149245c178d2a6
    x86/fpu: Factor out memset(xstate, 0) in fpu_finit() paths

fb14b4eadf73500d3b2104f031472a268562c047
    x86/fpu: Document user_fpu_begin()

8f4d81863ba4e8dfee93bd50840f1099a296251f
    x86/fpu: Introduce restore_init_xstate()

f893959b0898bd876673adbeb6798bdf25c034d7
    x86/fpu: Don't abuse drop_init_fpu() in flush_thread()

d2d0ac9a4644e00120bb9b7427a512a99d2cacc5
    x86/fpu: Fold __drop_fpu() into its sole user

7575637ab293861a799f3bbafe0d8c597389f4e9
    x86, fpu: Fix math_state_restore() race with kernel_fpu_begin()

b85e67d1483c72b77d1bdc265aa8ba91590794c1
    x86/fpu: Rename drop_init_fpu() to fpu_reset_state()

c88d47480d300eaad80c213d50c9bf6077fc49bc
    x86/fpu: Always restore_xinit_state() when use_eager_cpu()

ab6b52947545a5355154f64f449f97af9d05845f
    x86/fpu: Fix 32-bit signal frame handling

18ecb3bfa5a9f6fffbb3eeb4369f0b9463438ec0
    x86/fpu: Load xsave pointer *after* initialization

c447e76b4cabb49ddae8e49c5758f031f35d55fb (more or less redone from scratch)
    kvm/fpu: Enable eager restore kvm FPU for MPX

A reproducer is after my sig.  It does 10,000 iterations, but really
it almost always fails within the first 3-4, and if it doesn't it fails
within the first 100.

Also note that I didn't backport fully lazy FPU because it scared
the hell out of me. :)

Paolo

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <unistd.h>
#include <string.h>


void set(unsigned long long v)
{
        asm("movsd %[v], %%XMM0"
                        :
                        : [v] "m" (v)
                        :
           );
}

unsigned long long get(void)
{
        unsigned long long v;
        asm("movsd  %%XMM0, %[check]"
                :
                : [check] "m" (v)
                :
        );
        return v;
}

volatile int signal_cnt = 0;

void sigcld(int s, siginfo_t *si, void *ctx)
{
        ucontext_t *uc = ctx;
        mcontext_t *mc = &uc->uc_mcontext;
        fpregset_t fpr = mc->fpregs;
        //printf("in signal handler, saved xmm0 is %llx %llx\n", fpr->_xmm[0].element[0], fpr->_xmm[0].element[1]);
        signal_cnt++;
}

void try(int j)
{
        int i, status;
        int rounds = 100;

        for (i = 0; i < rounds; i++) {
                unsigned long long correct = i + 0x5713AFDB2639ECA0ULL;
                unsigned long long actual;
                signal_cnt = 0;
                set(correct);
                if (fork() == 0) {
                        exit(0);
                }
                actual = get();
                int x = signal_cnt;
                if (correct != actual) {
                        printf("xmm0 is different for %d, %d: %llx (expected %llx) signal_cnt=%d\n",
                               j, i, actual, correct, signal_cnt);
                        exit(1);
                }
        }

        for (i = 0; i < rounds; i++) {
                wait(&status);
        }

}

int main(void)
{
        int i;
        setvbuf(stdout, NULL, _IONBF, 0);
        struct sigaction sigact = { .sa_sigaction = sigcld, .sa_flags = SA_SIGINFO };
        sigaction(SIGCLD, &sigact, NULL);
        for (i = 0; i < 10000; i++) {
                if ((i % 10) == 0) putchar (((i / 10) % 10) + '0');
                try(i);
                if ((i % 100) == 99) printf(" %d\n", i+1);
        }
        return 0;
}