PV-vNUMA issue: topology is misinterpreted by the guest

PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3987 bytes --]

Hey,

This started on IRC, but it's actually appropriate to have the
conversation here.

I just discovered an issue with vNUMA, when PV guests are used. In fact,
creating a 4 vCPUs PV guest, and making up things so that all the 4
vCPUs should be busy, I see this:

root@Zhaman:~# xl vcpu-list test
Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
test                                 4     0    5   r--    1481.9  all / 0-7
test                                 4     1    2   r--    1479.4  all / 0-7
test                                 4     2   15   -b-       7.5  all / 8-15
test                                 4     3   10   -b-    1324.8  all / 8-15

Going checking inside the guest, confirms that *everything* runs on
vCPUs 0 and 1. However, using schedtool or taskset, I can force tasks to
execute on vCPUs 2 and 3.

Inspecting the guest's dmesg, I've seen this:

[    0.128416] ------------[ cut here ]------------
[    0.128416] WARNING: CPU: 2 PID: 0 at ../arch/x86/kernel/smpboot.c:317 topology_sane.isra.2+0x74/0x88()
[    0.128416] sched: CPU #2's smt-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
[    0.128416] Modules linked in:
[    0.128416] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.19.0+ #1
[    0.128416]  0000000000000009 ffff88001ee3bdd0 ffffffff81657c7b ffffffff810bbd2c
[    0.128416]  ffff88001ee3be20 ffff88001ee3be10 ffffffff81081510 ffff88001ee3bea0
[    0.128416]  ffffffff8103aa02 ffff88003ea0a001 0000000000000000 ffff88001f20a040
[    0.128416] Call Trace:
[    0.128416]  [<ffffffff81657c7b>] dump_stack+0x4f/0x7b
[    0.128416]  [<ffffffff810bbd2c>] ? up+0x39/0x3e
[    0.128416]  [<ffffffff81081510>] warn_slowpath_common+0xa1/0xbb
[    0.128416]  [<ffffffff8103aa02>] ? topology_sane.isra.2+0x74/0x88
[    0.128416]  [<ffffffff81081570>] warn_slowpath_fmt+0x46/0x48
[    0.128416]  [<ffffffff8101eeb1>] ? __cpuid.constprop.0+0x15/0x19
[    0.128416]  [<ffffffff8103aa02>] topology_sane.isra.2+0x74/0x88
[    0.128416]  [<ffffffff8103ac70>] set_cpu_sibling_map+0x21a/0x444
[    0.128416]  [<ffffffff81056ac3>] ? numa_add_cpu+0x98/0x9f
[    0.128416]  [<ffffffff8100b8f2>] cpu_bringup+0x63/0xa8
[    0.128416]  [<ffffffff8100b945>] cpu_bringup_and_idle+0xe/0x1a
[    0.128416] ---[ end trace 95bff1aef57ee1b1 ]---

So, basically, Linux is complaining that we're trying to put two vCPUs,
that looks to be SMT siblings, on different NUMA nodes. And, yes, I
think this is quite disruptive for the Linux's scheduler internal logic.

The vnuma bits of the guest config are these:

 vnuma = [ [ "pnode=0","size=512","vcpus=0-1","vdistances=10,20"  ],
           [ "pnode=1","size=512","vcpus=2-3","vdistances=20,10"  ] ]

From inside the guest, the topology looks to be like this:

root@test:~# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1
node 0 size: 475 MB
node 0 free: 382 MB
node 1 cpus: 2 3
node 1 size: 495 MB
node 1 free: 475 MB
node distances:
node   0   1 
  0:  10  10 
  1:  20  10

root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list 
0-1
root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list 
0-3
root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list 
2-3
root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list 
0-3

So the complain during boot seems to be against 'core_siblings' (which
was not what I expected, but perhaps I misremember the meaning of
"core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll
double check).

Anyway, is there anything we can do to fix or workaround things?

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Jan Beulich]

>>> On 16.07.15 at 12:32, <dario.faggioli@citrix.com> wrote:
> root@test:~# numactl --hardware
> available: 2 nodes (0-1)
> node 0 cpus: 0 1
> node 0 size: 475 MB
> node 0 free: 382 MB
> node 1 cpus: 2 3
> node 1 size: 495 MB
> node 1 free: 475 MB
> node distances:
> node   0   1 
>   0:  10  10 
>   1:  20  10
> 
> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list 
> 0-1
> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list 
> 0-3
> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list 
> 2-3
> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list 
> 0-3
> 
> So the complain during boot seems to be against 'core_siblings' (which
> was not what I expected, but perhaps I misremember the meaning of
> "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll
> double check).
> 
> Anyway, is there anything we can do to fix or workaround things?

Make the guest honor topology also at the CPUID layer. Whether
that's by not wrongly consuming the respective CPUID bits (i.e. a
guest side change) or reflecting PV state in what the hypervisor
returns I'm not sure about. While the latter might be more clean,
I'd be afraid this might get in the way of what the tool stack wants
to see.

Jan

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Andrew Cooper]

On 16/07/15 11:47, Jan Beulich wrote:
>>>> On 16.07.15 at 12:32, <dario.faggioli@citrix.com> wrote:
>> root@test:~# numactl --hardware
>> available: 2 nodes (0-1)
>> node 0 cpus: 0 1
>> node 0 size: 475 MB
>> node 0 free: 382 MB
>> node 1 cpus: 2 3
>> node 1 size: 495 MB
>> node 1 free: 475 MB
>> node distances:
>> node   0   1 
>>   0:  10  10 
>>   1:  20  10
>>
>> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list 
>> 0-1
>> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list 
>> 0-3
>> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list 
>> 2-3
>> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list 
>> 0-3
>>
>> So the complain during boot seems to be against 'core_siblings' (which
>> was not what I expected, but perhaps I misremember the meaning of
>> "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll
>> double check).
>>
>> Anyway, is there anything we can do to fix or workaround things?
> Make the guest honor topology also at the CPUID layer. Whether
> that's by not wrongly consuming the respective CPUID bits (i.e. a
> guest side change) or reflecting PV state in what the hypervisor
> returns I'm not sure about. While the latter might be more clean,
> I'd be afraid this might get in the way of what the tool stack wants
> to see.

Xen's CPUID handling currently has no concept of per-core and
per-package data in the cpuid policy.  The guest sees the information
from the pcpu which happened to be running libxc at the time the policy
was decided, with a Xen-level fudge factor applied.

There is also the regular problem of (failing to) trap cpuid
instructions in PV guests, so any "numa aware" software which doesn't
use the force override prefix will still fail in the above way.

Lots of issues in this area are discussed in
http://xenbits.xen.org/people/andrewcoop/feature-levelling/feature-levelling-E.pdf
, especially the extra work section at the end.

~Andrew

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Wei Liu]

On Thu, Jul 16, 2015 at 11:56:50AM +0100, Andrew Cooper wrote:
> On 16/07/15 11:47, Jan Beulich wrote:
> >>>> On 16.07.15 at 12:32, <dario.faggioli@citrix.com> wrote:
> >> root@test:~# numactl --hardware
> >> available: 2 nodes (0-1)
> >> node 0 cpus: 0 1
> >> node 0 size: 475 MB
> >> node 0 free: 382 MB
> >> node 1 cpus: 2 3
> >> node 1 size: 495 MB
> >> node 1 free: 475 MB
> >> node distances:
> >> node   0   1 
> >>   0:  10  10 
> >>   1:  20  10
> >>
> >> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list 
> >> 0-1
> >> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list 
> >> 0-3
> >> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list 
> >> 2-3
> >> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list 
> >> 0-3
> >>
> >> So the complain during boot seems to be against 'core_siblings' (which
> >> was not what I expected, but perhaps I misremember the meaning of
> >> "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll
> >> double check).
> >>
> >> Anyway, is there anything we can do to fix or workaround things?
> > Make the guest honor topology also at the CPUID layer. Whether
> > that's by not wrongly consuming the respective CPUID bits (i.e. a
> > guest side change) or reflecting PV state in what the hypervisor
> > returns I'm not sure about. While the latter might be more clean,
> > I'd be afraid this might get in the way of what the tool stack wants
> > to see.
> 
> Xen's CPUID handling currently has no concept of per-core and
> per-package data in the cpuid policy.  The guest sees the information

Can / Will Xen have that concept in the future?

Wei.

> from the pcpu which happened to be running libxc at the time the policy
> was decided, with a Xen-level fudge factor applied.
> 
> There is also the regular problem of (failing to) trap cpuid
> instructions in PV guests, so any "numa aware" software which doesn't
> use the force override prefix will still fail in the above way.
> 
> Lots of issues in this area are discussed in
> http://xenbits.xen.org/people/andrewcoop/feature-levelling/feature-levelling-E.pdf
> , especially the extra work section at the end.
> 
> ~Andrew

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Wei Liu]

On Thu, Jul 16, 2015 at 12:32:42PM +0200, Dario Faggioli wrote:
> Hey,
> 
> This started on IRC, but it's actually appropriate to have the
> conversation here.
> 
> I just discovered an issue with vNUMA, when PV guests are used. In fact,
> creating a 4 vCPUs PV guest, and making up things so that all the 4
> vCPUs should be busy, I see this:
> 
> root@Zhaman:~# xl vcpu-list test
> Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
> test                                 4     0    5   r--    1481.9  all / 0-7
> test                                 4     1    2   r--    1479.4  all / 0-7
> test                                 4     2   15   -b-       7.5  all / 8-15
> test                                 4     3   10   -b-    1324.8  all / 8-15
> 
> Going checking inside the guest, confirms that *everything* runs on
> vCPUs 0 and 1. However, using schedtool or taskset, I can force tasks to
> execute on vCPUs 2 and 3.
> 
> Inspecting the guest's dmesg, I've seen this:
> 
> [    0.128416] ------------[ cut here ]------------
> [    0.128416] WARNING: CPU: 2 PID: 0 at ../arch/x86/kernel/smpboot.c:317 topology_sane.isra.2+0x74/0x88()
> [    0.128416] sched: CPU #2's smt-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
> [    0.128416] Modules linked in:
> [    0.128416] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.19.0+ #1
> [    0.128416]  0000000000000009 ffff88001ee3bdd0 ffffffff81657c7b ffffffff810bbd2c
> [    0.128416]  ffff88001ee3be20 ffff88001ee3be10 ffffffff81081510 ffff88001ee3bea0
> [    0.128416]  ffffffff8103aa02 ffff88003ea0a001 0000000000000000 ffff88001f20a040
> [    0.128416] Call Trace:
> [    0.128416]  [<ffffffff81657c7b>] dump_stack+0x4f/0x7b
> [    0.128416]  [<ffffffff810bbd2c>] ? up+0x39/0x3e
> [    0.128416]  [<ffffffff81081510>] warn_slowpath_common+0xa1/0xbb
> [    0.128416]  [<ffffffff8103aa02>] ? topology_sane.isra.2+0x74/0x88
> [    0.128416]  [<ffffffff81081570>] warn_slowpath_fmt+0x46/0x48
> [    0.128416]  [<ffffffff8101eeb1>] ? __cpuid.constprop.0+0x15/0x19
> [    0.128416]  [<ffffffff8103aa02>] topology_sane.isra.2+0x74/0x88
> [    0.128416]  [<ffffffff8103ac70>] set_cpu_sibling_map+0x21a/0x444
> [    0.128416]  [<ffffffff81056ac3>] ? numa_add_cpu+0x98/0x9f
> [    0.128416]  [<ffffffff8100b8f2>] cpu_bringup+0x63/0xa8
> [    0.128416]  [<ffffffff8100b945>] cpu_bringup_and_idle+0xe/0x1a
> [    0.128416] ---[ end trace 95bff1aef57ee1b1 ]---
> 
> So, basically, Linux is complaining that we're trying to put two vCPUs,
> that looks to be SMT siblings, on different NUMA nodes. And, yes, I
> think this is quite disruptive for the Linux's scheduler internal logic.
> 
> The vnuma bits of the guest config are these:
> 
>  vnuma = [ [ "pnode=0","size=512","vcpus=0-1","vdistances=10,20"  ],
>            [ "pnode=1","size=512","vcpus=2-3","vdistances=20,10"  ] ]
> 
> From inside the guest, the topology looks to be like this:
> 
> root@test:~# numactl --hardware
> available: 2 nodes (0-1)
> node 0 cpus: 0 1
> node 0 size: 475 MB
> node 0 free: 382 MB
> node 1 cpus: 2 3
> node 1 size: 495 MB
> node 1 free: 475 MB
> node distances:
> node   0   1 
>   0:  10  10 
>   1:  20  10
> 
> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list 
> 0-1
> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list 
> 0-3
> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list 
> 2-3
> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list 
> 0-3
> 
> So the complain during boot seems to be against 'core_siblings' (which
> was not what I expected, but perhaps I misremember the meaning of
> "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll
> double check).
> 
> Anyway, is there anything we can do to fix or workaround things?
> 


IIRC Linux already consumes some bits returned by cpuid anyway, is it
possible to generate a "dummy" layout in Linux kernel according to vNUMA
information? I had this idea long ago but wasn't quite sure if it's dumb
or not.

Wei.

> Regards,
> Dario
> -- 
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Andrew Cooper]

On 16/07/15 16:25, Wei Liu wrote:
> On Thu, Jul 16, 2015 at 11:56:50AM +0100, Andrew Cooper wrote:
>> On 16/07/15 11:47, Jan Beulich wrote:
>>>>>> On 16.07.15 at 12:32, <dario.faggioli@citrix.com> wrote:
>>>> root@test:~# numactl --hardware
>>>> available: 2 nodes (0-1)
>>>> node 0 cpus: 0 1
>>>> node 0 size: 475 MB
>>>> node 0 free: 382 MB
>>>> node 1 cpus: 2 3
>>>> node 1 size: 495 MB
>>>> node 1 free: 475 MB
>>>> node distances:
>>>> node   0   1 
>>>>   0:  10  10 
>>>>   1:  20  10
>>>>
>>>> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list 
>>>> 0-1
>>>> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list 
>>>> 0-3
>>>> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list 
>>>> 2-3
>>>> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list 
>>>> 0-3
>>>>
>>>> So the complain during boot seems to be against 'core_siblings' (which
>>>> was not what I expected, but perhaps I misremember the meaning of
>>>> "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll
>>>> double check).
>>>>
>>>> Anyway, is there anything we can do to fix or workaround things?
>>> Make the guest honor topology also at the CPUID layer. Whether
>>> that's by not wrongly consuming the respective CPUID bits (i.e. a
>>> guest side change) or reflecting PV state in what the hypervisor
>>> returns I'm not sure about. While the latter might be more clean,
>>> I'd be afraid this might get in the way of what the tool stack wants
>>> to see.
>> Xen's CPUID handling currently has no concept of per-core and
>> per-package data in the cpuid policy.  The guest sees the information
> Can / Will Xen have that concept in the future?

It is certainly possible.

I plan to try and lay some ground work as part of the feature levelling
fixes, but fixing the hypervisor representation of cpuid is specifically
out of scope for the feature levelling fixes (in a deliberate attempt to
prevent the project expanding to fill more time than I have).

It is on my list of areas to tackle, but it is several nested cans of worms.

~Andrew

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/16/2015 11:45 AM, Andrew Cooper wrote:
> On 16/07/15 16:25, Wei Liu wrote:
>> On Thu, Jul 16, 2015 at 11:56:50AM +0100, Andrew Cooper wrote:
>>> On 16/07/15 11:47, Jan Beulich wrote:
>>>>>>> On 16.07.15 at 12:32, <dario.faggioli@citrix.com> wrote:
>>>>> root@test:~# numactl --hardware
>>>>> available: 2 nodes (0-1)
>>>>> node 0 cpus: 0 1
>>>>> node 0 size: 475 MB
>>>>> node 0 free: 382 MB
>>>>> node 1 cpus: 2 3
>>>>> node 1 size: 495 MB
>>>>> node 1 free: 475 MB
>>>>> node distances:
>>>>> node   0   1
>>>>>    0:  10  10
>>>>>    1:  20  10
>>>>>
>>>>> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list
>>>>> 0-1
>>>>> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list
>>>>> 0-3
>>>>> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list
>>>>> 2-3
>>>>> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list
>>>>> 0-3
>>>>>
>>>>> So the complain during boot seems to be against 'core_siblings' (which
>>>>> was not what I expected, but perhaps I misremember the meaning of
>>>>> "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll
>>>>> double check).
>>>>>
>>>>> Anyway, is there anything we can do to fix or workaround things?
>>>> Make the guest honor topology also at the CPUID layer. Whether
>>>> that's by not wrongly consuming the respective CPUID bits (i.e. a
>>>> guest side change) or reflecting PV state in what the hypervisor
>>>> returns I'm not sure about. While the latter might be more clean,
>>>> I'd be afraid this might get in the way of what the tool stack wants
>>>> to see.
>>> Xen's CPUID handling currently has no concept of per-core and
>>> per-package data in the cpuid policy.  The guest sees the information
>> Can / Will Xen have that concept in the future?
> It is certainly possible.
>
> I plan to try and lay some ground work as part of the feature levelling
> fixes, but fixing the hypervisor representation of cpuid is specifically
> out of scope for the feature levelling fixes (in a deliberate attempt to
> prevent the project expanding to fill more time than I have).
>
> It is on my list of areas to tackle, but it is several nested cans of worms.

Can't we set leaf 1's EBX[32:16] to 1?

-boris

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Jan Beulich]

>>> On 16.07.15 at 17:50, <boris.ostrovsky@oracle.com> wrote:
> Can't we set leaf 1's EBX[32:16] to 1?

I don't think we should partially overwrite the relevant parts of
CPUID output - either all or nothing (so that things at least
remain consistent).

Jan

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Andrew Cooper]

On 16/07/15 17:29, Jan Beulich wrote:
>>>> On 16.07.15 at 17:50, <boris.ostrovsky@oracle.com> wrote:
>> Can't we set leaf 1's EBX[32:16] to 1?
> I don't think we should partially overwrite the relevant parts of
> CPUID output - either all or nothing (so that things at least
> remain consistent).

Also, there are no masking/override MSRs for that feature leaf on any
hardware I am aware of, and a PV guest using plain `cpuid` will not
observe any attempt of Xen's to control the value.

~Andrew

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/16/2015 12:39 PM, Andrew Cooper wrote:
> On 16/07/15 17:29, Jan Beulich wrote:
>>>>> On 16.07.15 at 17:50, <boris.ostrovsky@oracle.com> wrote:
>>> Can't we set leaf 1's EBX[32:16] to 1?

(I obviously fat-fingered this --- I meant EBX[23:16])

>> I don't think we should partially overwrite the relevant parts of
>> CPUID output - either all or nothing (so that things at least
>> remain consistent).

Possibly more, but specifically for Dario's problem I think this could 
resolve that.

> Also, there are no masking/override MSRs for that feature leaf on any
> hardware I am aware of, and a PV guest using plain `cpuid` will not
> observe any attempt of Xen's to control the value.

True, but we already modify other CPUID bits (e.g. we clear 
X86_FEATURE_TOPOEXT) so it won't make things much worse. (And again, for 
this specific problem CPUID is queried by guest kernel which uses PV 
CPUID instruction).

And in general (both for PV and HVM) --- is there any reason to expose 
CPU topology at all? I can see it being useful if VCPUs are pinned but 
if they are not then it can make performance worse.

-boris

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Jan Beulich]

>>> On 16.07.15 at 18:59, <boris.ostrovsky@oracle.com> wrote:
> And in general (both for PV and HVM) --- is there any reason to expose 
> CPU topology at all? I can see it being useful if VCPUs are pinned but 
> if they are not then it can make performance worse.

Indeed - that's what our kernels have been doing for years, and
it seems like someone over here is now looking into whether this
could be done in pv-ops too (without too much uglification).

Jan

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1656 bytes --]

On Fri, 2015-07-17 at 07:09 +0100, Jan Beulich wrote:
> >>> On 16.07.15 at 18:59, <boris.ostrovsky@oracle.com> wrote:
> > And in general (both for PV and HVM) --- is there any reason to expose 
> > CPU topology at all? I can see it being useful if VCPUs are pinned but 
> > if they are not then it can make performance worse.
> 
> Indeed 
>
Indeed indeed. :-)

And in fact, this is even independent from vNUMA. Yet, I remember we
were discussing about this since the beginning of vNUMA work, back when
it was Elena doing it, but the it seems we all forgot... Sorry for
that! :-/

I seriously think we should do something about this as, while in a non
vNUMA setup it can certainly cause weird/inconsistent performance, in a
vNUMA one, as shown, it's quite a hige mess.

> - that's what our kernels have been doing for years, and
> it seems like someone over here is now looking into whether this
> could be done in pv-ops too (without too much uglification).
> 
That would be great, IMO. I'd be up for helping with this, but I know
next to nothing about CPUID, so that would require some setup time. If,
at least, you could keep me in the loop it would be great.

In the meanwhile, what should we do? Document this? How? "don't use
vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is
there a workaround we can put in place/suggest?

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Jan Beulich]

>>> On 17.07.15 at 09:27, <dario.faggioli@citrix.com> wrote:
> In the meanwhile, what should we do? Document this? How? "don't use
> vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is
> there a workaround we can put in place/suggest?

Use SLE / openSUSE kernels ;-) ?

Jan

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Wei Liu]

On Fri, Jul 17, 2015 at 09:27:55AM +0200, Dario Faggioli wrote:
> On Fri, 2015-07-17 at 07:09 +0100, Jan Beulich wrote:
> > >>> On 16.07.15 at 18:59, <boris.ostrovsky@oracle.com> wrote:
> > > And in general (both for PV and HVM) --- is there any reason to expose 
> > > CPU topology at all? I can see it being useful if VCPUs are pinned but 
> > > if they are not then it can make performance worse.
> > 
> > Indeed 
> >
> Indeed indeed. :-)
> 
> And in fact, this is even independent from vNUMA. Yet, I remember we
> were discussing about this since the beginning of vNUMA work, back when
> it was Elena doing it, but the it seems we all forgot... Sorry for
> that! :-/
> 
> I seriously think we should do something about this as, while in a non
> vNUMA setup it can certainly cause weird/inconsistent performance, in a
> vNUMA one, as shown, it's quite a hige mess.
> 
> > - that's what our kernels have been doing for years, and
> > it seems like someone over here is now looking into whether this
> > could be done in pv-ops too (without too much uglification).
> > 
> That would be great, IMO. I'd be up for helping with this, but I know
> next to nothing about CPUID, so that would require some setup time. If,
> at least, you could keep me in the loop it would be great.
> 
> In the meanwhile, what should we do? Document this? How? "don't use
> vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is
> there a workaround we can put in place/suggest?
> 

Don't worry. PV vNUMA kernel patch is not upstreamed. :-P

This is one of the issues that is not upstreamed. The other issue is my
own laziness...

Wei.

> Regards,
> Dario
> -- 
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Andrew Cooper]

On 16/07/15 17:59, Boris Ostrovsky wrote:
> On 07/16/2015 12:39 PM, Andrew Cooper wrote:
>> On 16/07/15 17:29, Jan Beulich wrote:
>>>>>> On 16.07.15 at 17:50, <boris.ostrovsky@oracle.com> wrote:
>>>> Can't we set leaf 1's EBX[32:16] to 1?
>
> (I obviously fat-fingered this --- I meant EBX[23:16])
>
>>> I don't think we should partially overwrite the relevant parts of
>>> CPUID output - either all or nothing (so that things at least
>>> remain consistent).
>
> Possibly more, but specifically for Dario's problem I think this could
> resolve that.
>
>> Also, there are no masking/override MSRs for that feature leaf on any
>> hardware I am aware of, and a PV guest using plain `cpuid` will not
>> observe any attempt of Xen's to control the value.
>
> True, but we already modify other CPUID bits (e.g. we clear
> X86_FEATURE_TOPOEXT) so it won't make things much worse. (And again,
> for this specific problem CPUID is queried by guest kernel which uses
> PV CPUID instruction).

The current state of cpuid handling is fragile to say the least. 
Overriding that leaf might indeed work, if configured from the toolstack
level.  It can't be clobbered in the hypervisor (like some of the
existing cpuid clobbering) as that would break migration.

>
> And in general (both for PV and HVM) --- is there any reason to expose
> CPU topology at all? I can see it being useful if VCPUs are pinned but
> if they are not then it can make performance worse.

For better or for worse, lots of software running in VMs have licensing
restrictions on CPU topology.  XenServer has options to configure the
visible topology inside the VM, to work around the issue that currently
every vcpu looks like a separate socket.

~Andrew

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/17/2015 03:27 AM, Dario Faggioli wrote:
> On Fri, 2015-07-17 at 07:09 +0100, Jan Beulich wrote:
>>>>> On 16.07.15 at 18:59, <boris.ostrovsky@oracle.com> wrote:
>>> And in general (both for PV and HVM) --- is there any reason to expose
>>> CPU topology at all? I can see it being useful if VCPUs are pinned but
>>> if they are not then it can make performance worse.
>> Indeed
>>
> Indeed indeed. :-)
>
> And in fact, this is even independent from vNUMA. Yet, I remember we
> were discussing about this since the beginning of vNUMA work, back when
> it was Elena doing it, but the it seems we all forgot... Sorry for
> that! :-/
>
> I seriously think we should do something about this as, while in a non
> vNUMA setup it can certainly cause weird/inconsistent performance, in a
> vNUMA one, as shown, it's quite a hige mess.
>
>> - that's what our kernels have been doing for years, and
>> it seems like someone over here is now looking into whether this
>> could be done in pv-ops too (without too much uglification).
>>
> That would be great, IMO. I'd be up for helping with this, but I know
> next to nothing about CPUID, so that would require some setup time. If,
> at least, you could keep me in the loop it would be great.
>
> In the meanwhile, what should we do? Document this? How? "don't use
> vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is
> there a workaround we can put in place/suggest?

I haven't been able to reproduce this on my Intel box because I think I 
have different core enumeration. Can you try adding
   cpuid=['0x1:ebx=xxxxxxxx00000001xxxxxxxxxxxxxxxx']
to your config file?

On AMD, BTW, we fail a different test so some other bits probably need 
to be tweaked. You may fail it too (the LLC sanity check).

-boris

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3777 bytes --]

On Fri, 2015-07-17 at 14:17 -0400, Boris Ostrovsky wrote:
> On 07/17/2015 03:27 AM, Dario Faggioli wrote:

> > In the meanwhile, what should we do? Document this? How? "don't use
> > vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is
> > there a workaround we can put in place/suggest?
> 
> I haven't been able to reproduce this on my Intel box because I think I 
> have different core enumeration. 
>
Yes, most likely, that's highly topology dependant. :-(

> Can you try adding
>    cpuid=['0x1:ebx=xxxxxxxx00000001xxxxxxxxxxxxxxxx']
> to your config file?
> 
Done (sorry for the delay, the testbox was busy doing other stuff).

Still no joy (.101 is the IP address of the guest, domain id 3):

root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
root@Zhaman:~# xl vcpu-list 3
Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
test                                 3     0    4   r--      23.6  all / 0-7
test                                 3     1    9   r--      19.8  all / 0-7
test                                 3     2    8   -b-       0.4  all / 8-15
test                                 3     3    4   -b-       0.2  all / 8-15

*HOWEVER* it seems to have an effect. In fact, now, topology as it is
shown in /sys/... is different:

root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list 
0
(it was 0-1)

This, OTOH, is still the same:
root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list  
0-3

Also, I now see this:

[    0.150560] ------------[ cut here ]------------
[    0.150560] WARNING: CPU: 2 PID: 0 at ../arch/x86/kernel/smpboot.c:317 topology_sane.isra.2+0x74/0x88()
[    0.150560] sched: CPU #2's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
[    0.150560] Modules linked in:
[    0.150560] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.19.0+ #1
[    0.150560]  0000000000000009 ffff88001ee2fdd0 ffffffff81657c7b ffffffff810bbd2c
[    0.150560]  ffff88001ee2fe20 ffff88001ee2fe10 ffffffff81081510 ffff88001ee2fea0
[    0.150560]  ffffffff8103aa02 ffff88003ea0a001 0000000000000000 ffff88001f20a040
[    0.150560] Call Trace:
[    0.150560]  [<ffffffff81657c7b>] dump_stack+0x4f/0x7b
[    0.150560]  [<ffffffff810bbd2c>] ? up+0x39/0x3e
[    0.150560]  [<ffffffff81081510>] warn_slowpath_common+0xa1/0xbb
[    0.150560]  [<ffffffff8103aa02>] ? topology_sane.isra.2+0x74/0x88
[    0.150560]  [<ffffffff81081570>] warn_slowpath_fmt+0x46/0x48
[    0.150560]  [<ffffffff8101eeb1>] ? __cpuid.constprop.0+0x15/0x19
[    0.150560]  [<ffffffff8103aa02>] topology_sane.isra.2+0x74/0x88
[    0.150560]  [<ffffffff8103acd0>] set_cpu_sibling_map+0x27a/0x444
[    0.150560]  [<ffffffff81056ac3>] ? numa_add_cpu+0x98/0x9f
[    0.150560]  [<ffffffff8100b8f2>] cpu_bringup+0x63/0xa8
[    0.150560]  [<ffffffff8100b945>] cpu_bringup_and_idle+0xe/0x1a
[    0.150560] ---[ end trace 63d204896cce9f68 ]---

Notice that it now says 'llc-sibling', while, before, it was saying
'smt-sibling'.

> On AMD, BTW, we fail a different test so some other bits probably need 
> to be tweaked. You may fail it too (the LLC sanity check).
> 
Yep, that's the one I guess. Should I try something more/else?

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/20/2015 10:09 AM, Dario Faggioli wrote:
> On Fri, 2015-07-17 at 14:17 -0400, Boris Ostrovsky wrote:
>> On 07/17/2015 03:27 AM, Dario Faggioli wrote:
>>> In the meanwhile, what should we do? Document this? How? "don't use
>>> vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is
>>> there a workaround we can put in place/suggest?
>> I haven't been able to reproduce this on my Intel box because I think I
>> have different core enumeration.
>>
> Yes, most likely, that's highly topology dependant. :-(
>
>> Can you try adding
>>     cpuid=['0x1:ebx=xxxxxxxx00000001xxxxxxxxxxxxxxxx']
>> to your config file?
>>
> Done (sorry for the delay, the testbox was busy doing other stuff).
>
> Still no joy (.101 is the IP address of the guest, domain id 3):
>
> root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
> root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
> root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
> root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
> root@Zhaman:~# xl vcpu-list 3
> Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
> test                                 3     0    4   r--      23.6  all / 0-7
> test                                 3     1    9   r--      19.8  all / 0-7
> test                                 3     2    8   -b-       0.4  all / 8-15
> test                                 3     3    4   -b-       0.2  all / 8-15
>
> *HOWEVER* it seems to have an effect. In fact, now, topology as it is
> shown in /sys/... is different:
>
> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list
> 0
> (it was 0-1)
>
> This, OTOH, is still the same:
> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list
> 0-3
>
> Also, I now see this:
>
> [    0.150560] ------------[ cut here ]------------
> [    0.150560] WARNING: CPU: 2 PID: 0 at ../arch/x86/kernel/smpboot.c:317 topology_sane.isra.2+0x74/0x88()
> [    0.150560] sched: CPU #2's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
> [    0.150560] Modules linked in:
> [    0.150560] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.19.0+ #1
> [    0.150560]  0000000000000009 ffff88001ee2fdd0 ffffffff81657c7b ffffffff810bbd2c
> [    0.150560]  ffff88001ee2fe20 ffff88001ee2fe10 ffffffff81081510 ffff88001ee2fea0
> [    0.150560]  ffffffff8103aa02 ffff88003ea0a001 0000000000000000 ffff88001f20a040
> [    0.150560] Call Trace:
> [    0.150560]  [<ffffffff81657c7b>] dump_stack+0x4f/0x7b
> [    0.150560]  [<ffffffff810bbd2c>] ? up+0x39/0x3e
> [    0.150560]  [<ffffffff81081510>] warn_slowpath_common+0xa1/0xbb
> [    0.150560]  [<ffffffff8103aa02>] ? topology_sane.isra.2+0x74/0x88
> [    0.150560]  [<ffffffff81081570>] warn_slowpath_fmt+0x46/0x48
> [    0.150560]  [<ffffffff8101eeb1>] ? __cpuid.constprop.0+0x15/0x19
> [    0.150560]  [<ffffffff8103aa02>] topology_sane.isra.2+0x74/0x88
> [    0.150560]  [<ffffffff8103acd0>] set_cpu_sibling_map+0x27a/0x444
> [    0.150560]  [<ffffffff81056ac3>] ? numa_add_cpu+0x98/0x9f
> [    0.150560]  [<ffffffff8100b8f2>] cpu_bringup+0x63/0xa8
> [    0.150560]  [<ffffffff8100b945>] cpu_bringup_and_idle+0xe/0x1a
> [    0.150560] ---[ end trace 63d204896cce9f68 ]---
>
> Notice that it now says 'llc-sibling', while, before, it was saying
> 'smt-sibling'.

Exactly. You are now passing the first topology test which was to see 
that threads are on the same node. And since each processor has only one 
thread (as evidenced by thread_siblings_list) we are good.

The second test checks that cores (i.e. things that share last level 
cache) are on the same node. And they are not.


>
>> On AMD, BTW, we fail a different test so some other bits probably need
>> to be tweaked. You may fail it too (the LLC sanity check).
>>
> Yep, that's the one I guess. Should I try something more/else?


I'll need to see how LLC IDs are calculated, probably also from some 
CPUID bits. The question though will be --- what do we do with how cache 
sizes (and TLB sizes for that matter) are presented to the guests. Do we 
scale them down per thread?

-boris

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/20/2015 10:43 AM, Boris Ostrovsky wrote:
> On 07/20/2015 10:09 AM, Dario Faggioli wrote:
>> On Fri, 2015-07-17 at 14:17 -0400, Boris Ostrovsky wrote:
>>> On 07/17/2015 03:27 AM, Dario Faggioli wrote:
>>>> In the meanwhile, what should we do? Document this? How? "don't use
>>>> vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is
>>>> there a workaround we can put in place/suggest?
>>> I haven't been able to reproduce this on my Intel box because I think I
>>> have different core enumeration.
>>>
>> Yes, most likely, that's highly topology dependant. :-(
>>
>>> Can you try adding
>>>     cpuid=['0x1:ebx=xxxxxxxx00000001xxxxxxxxxxxxxxxx']
>>> to your config file?
>>>
>> Done (sorry for the delay, the testbox was busy doing other stuff).
>>
>> Still no joy (.101 is the IP address of the guest, domain id 3):
>>
>> root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
>> root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
>> root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
>> root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
>> root@Zhaman:~# xl vcpu-list 3
>> Name                                ID  VCPU   CPU State Time(s) 
>> Affinity (Hard / Soft)
>> test                                 3     0    4   r-- 23.6  all / 0-7
>> test                                 3     1    9   r-- 19.8  all / 0-7
>> test                                 3     2    8   -b- 0.4  all / 8-15
>> test                                 3     3    4   -b- 0.2  all / 8-15
>>
>> *HOWEVER* it seems to have an effect. In fact, now, topology as it is
>> shown in /sys/... is different:
>>
>> root@test:~# cat 
>> /sys/devices/system/cpu/cpu0/topology/thread_siblings_list
>> 0
>> (it was 0-1)
>>
>> This, OTOH, is still the same:
>> root@test:~# cat 
>> /sys/devices/system/cpu/cpu0/topology/core_siblings_list
>> 0-3
>>
>> Also, I now see this:
>>
>> [    0.150560] ------------[ cut here ]------------
>> [    0.150560] WARNING: CPU: 2 PID: 0 at 
>> ../arch/x86/kernel/smpboot.c:317 topology_sane.isra.2+0x74/0x88()
>> [    0.150560] sched: CPU #2's llc-sibling CPU #0 is not on the same 
>> node! [node: 1 != 0]. Ignoring dependency.
>> [    0.150560] Modules linked in:
>> [    0.150560] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.19.0+ #1
>> [    0.150560]  0000000000000009 ffff88001ee2fdd0 ffffffff81657c7b 
>> ffffffff810bbd2c
>> [    0.150560]  ffff88001ee2fe20 ffff88001ee2fe10 ffffffff81081510 
>> ffff88001ee2fea0
>> [    0.150560]  ffffffff8103aa02 ffff88003ea0a001 0000000000000000 
>> ffff88001f20a040
>> [    0.150560] Call Trace:
>> [    0.150560]  [<ffffffff81657c7b>] dump_stack+0x4f/0x7b
>> [    0.150560]  [<ffffffff810bbd2c>] ? up+0x39/0x3e
>> [    0.150560]  [<ffffffff81081510>] warn_slowpath_common+0xa1/0xbb
>> [    0.150560]  [<ffffffff8103aa02>] ? topology_sane.isra.2+0x74/0x88
>> [    0.150560]  [<ffffffff81081570>] warn_slowpath_fmt+0x46/0x48
>> [    0.150560]  [<ffffffff8101eeb1>] ? __cpuid.constprop.0+0x15/0x19
>> [    0.150560]  [<ffffffff8103aa02>] topology_sane.isra.2+0x74/0x88
>> [    0.150560]  [<ffffffff8103acd0>] set_cpu_sibling_map+0x27a/0x444
>> [    0.150560]  [<ffffffff81056ac3>] ? numa_add_cpu+0x98/0x9f
>> [    0.150560]  [<ffffffff8100b8f2>] cpu_bringup+0x63/0xa8
>> [    0.150560]  [<ffffffff8100b945>] cpu_bringup_and_idle+0xe/0x1a
>> [    0.150560] ---[ end trace 63d204896cce9f68 ]---
>>
>> Notice that it now says 'llc-sibling', while, before, it was saying
>> 'smt-sibling'.
>
> Exactly. You are now passing the first topology test which was to see 
> that threads are on the same node. And since each processor has only 
> one thread (as evidenced by thread_siblings_list) we are good.
>
> The second test checks that cores (i.e. things that share last level 
> cache) are on the same node. And they are not.
>
>
>>
>>> On AMD, BTW, we fail a different test so some other bits probably need
>>> to be tweaked. You may fail it too (the LLC sanity check).
>>>
>> Yep, that's the one I guess. Should I try something more/else?
>
>
> I'll need to see how LLC IDs are calculated, probably also from some 
> CPUID bits.


No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) which 
use indexes in ECX register and xl syntax doesn't allow you to override 
CPUIDs for such leaves.

-boris

> The question though will be --- what do we do with how cache sizes 
> (and TLB sizes for that matter) are presented to the guests. Do we 
> scale them down per thread?
>
> -boris

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1415 bytes --]

On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote:
> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote:
> > On 07/20/2015 10:09 AM, Dario Faggioli wrote:

> > I'll need to see how LLC IDs are calculated, probably also from some 
> > CPUID bits.
> 
> 
> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) which 
> use indexes in ECX register and xl syntax doesn't allow you to override 
> CPUIDs for such leaves.
> 
Right. Which leaves us with the question of what should we do and/or
recommend users to do?

If there were a workaround that we could put in place, and document
somewhere, however tricky it was, I'd say to go for it, and call it
acceptable for now.

But, if there isn't, should we disable PV vnuma, or warn the user that
he may see issues? Can we identify, in Xen or in toolstack, whether an
host topology will be problematic, and disable/warn in those cases too?

I'm not sure, honestly. Disabling looks too aggressive, but it's an
issue I wouldn't like an user to be facing, without at least being
informed of the possibility... so, perhaps a (set of) warning(s)?
Thoughts?

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/22/2015 03:36 PM, Dario Faggioli wrote:
> On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote:
>> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote:
>>> On 07/20/2015 10:09 AM, Dario Faggioli wrote:
>
>>> I'll need to see how LLC IDs are calculated, probably also from some
>>> CPUID bits.
>>
>>
>> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) which
>> use indexes in ECX register and xl syntax doesn't allow you to override
>> CPUIDs for such leaves.
>>
> Right. Which leaves us with the question of what should we do and/or
> recommend users to do?
>
> If there were a workaround that we could put in place, and document
> somewhere, however tricky it was, I'd say to go for it, and call it
> acceptable for now.
>
> But, if there isn't, should we disable PV vnuma, or warn the user that
> he may see issues? Can we identify, in Xen or in toolstack, whether an
> host topology will be problematic, and disable/warn in those cases too?
>
> I'm not sure, honestly. Disabling looks too aggressive, but it's an
> issue I wouldn't like an user to be facing, without at least being
> informed of the possibility... so, perhaps a (set of) warning(s)?
> Thoughts?

I think we have 2 possible solutions:

1. Try to handle this all in the hypervisor via CPUID mangling.

2. Add PV-topology support to the guest and indicate this capability via
    elfnote; only enable PV-numa if this note is present.

I'd prefer the second solution. If you are okay with this, I'd try to do
some patches for the pvops kernel.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/22/2015 09:50 AM, Juergen Gross wrote:
> On 07/22/2015 03:36 PM, Dario Faggioli wrote:
>> On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote:
>>> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote:
>>>> On 07/20/2015 10:09 AM, Dario Faggioli wrote:
>>
>>>> I'll need to see how LLC IDs are calculated, probably also from some
>>>> CPUID bits.
>>>
>>>
>>> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) which
>>> use indexes in ECX register and xl syntax doesn't allow you to override
>>> CPUIDs for such leaves.
>>>
>> Right. Which leaves us with the question of what should we do and/or
>> recommend users to do?
>>
>> If there were a workaround that we could put in place, and document
>> somewhere, however tricky it was, I'd say to go for it, and call it
>> acceptable for now.
>>
>> But, if there isn't, should we disable PV vnuma, or warn the user that
>> he may see issues? Can we identify, in Xen or in toolstack, whether an
>> host topology will be problematic, and disable/warn in those cases too?
>>
>> I'm not sure, honestly. Disabling looks too aggressive, but it's an
>> issue I wouldn't like an user to be facing, without at least being
>> informed of the possibility... so, perhaps a (set of) warning(s)?
>> Thoughts?
>
> I think we have 2 possible solutions:
>
> 1. Try to handle this all in the hypervisor via CPUID mangling.
>
> 2. Add PV-topology support to the guest and indicate this capability via
>    elfnote; only enable PV-numa if this note is present.
>
> I'd prefer the second solution. If you are okay with this, I'd try to do
> some patches for the pvops kernel.

What if I configure a guest to follow HW topology? I.e. I pin VCPUs to 
appropriate cores/threads? With elfnote I am stuck with disabled topology.

Besides, this is not necessarily a NUMA-only issue, it's a scheduling 
one (inside the guest) as well.

-boris

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/22/2015 03:58 PM, Boris Ostrovsky wrote:
> On 07/22/2015 09:50 AM, Juergen Gross wrote:
>> On 07/22/2015 03:36 PM, Dario Faggioli wrote:
>>> On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote:
>>>> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote:
>>>>> On 07/20/2015 10:09 AM, Dario Faggioli wrote:
>>>
>>>>> I'll need to see how LLC IDs are calculated, probably also from some
>>>>> CPUID bits.
>>>>
>>>>
>>>> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) which
>>>> use indexes in ECX register and xl syntax doesn't allow you to override
>>>> CPUIDs for such leaves.
>>>>
>>> Right. Which leaves us with the question of what should we do and/or
>>> recommend users to do?
>>>
>>> If there were a workaround that we could put in place, and document
>>> somewhere, however tricky it was, I'd say to go for it, and call it
>>> acceptable for now.
>>>
>>> But, if there isn't, should we disable PV vnuma, or warn the user that
>>> he may see issues? Can we identify, in Xen or in toolstack, whether an
>>> host topology will be problematic, and disable/warn in those cases too?
>>>
>>> I'm not sure, honestly. Disabling looks too aggressive, but it's an
>>> issue I wouldn't like an user to be facing, without at least being
>>> informed of the possibility... so, perhaps a (set of) warning(s)?
>>> Thoughts?
>>
>> I think we have 2 possible solutions:
>>
>> 1. Try to handle this all in the hypervisor via CPUID mangling.
>>
>> 2. Add PV-topology support to the guest and indicate this capability via
>>    elfnote; only enable PV-numa if this note is present.
>>
>> I'd prefer the second solution. If you are okay with this, I'd try to do
>> some patches for the pvops kernel.
>
> What if I configure a guest to follow HW topology? I.e. I pin VCPUs to
> appropriate cores/threads? With elfnote I am stuck with disabled topology.

Add an option to do exactly that: follow HW topology (pin vcpus,
configure vnuma)?

Add a force flag to the vnuma configuration to ignore the elfnote?

> Besides, this is not necessarily a NUMA-only issue, it's a scheduling
> one (inside the guest) as well.

Sure. That's what Jan said regarding SUSE's xen-kernel. No toplogy info
(or a trivial one) might be better than the wrong one...

This patch for pvops should be written in any case. I'll do this, but it
would be nice to know whether PV-numa should be considered or not.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/22/2015 10:09 AM, Juergen Gross wrote:
> On 07/22/2015 03:58 PM, Boris Ostrovsky wrote:
>> On 07/22/2015 09:50 AM, Juergen Gross wrote:
>>> On 07/22/2015 03:36 PM, Dario Faggioli wrote:
>>>> On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote:
>>>>> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote:
>>>>>> On 07/20/2015 10:09 AM, Dario Faggioli wrote:
>>>>
>>>>>> I'll need to see how LLC IDs are calculated, probably also from some
>>>>>> CPUID bits.
>>>>>
>>>>>
>>>>> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) 
>>>>> which
>>>>> use indexes in ECX register and xl syntax doesn't allow you to 
>>>>> override
>>>>> CPUIDs for such leaves.
>>>>>
>>>> Right. Which leaves us with the question of what should we do and/or
>>>> recommend users to do?
>>>>
>>>> If there were a workaround that we could put in place, and document
>>>> somewhere, however tricky it was, I'd say to go for it, and call it
>>>> acceptable for now.
>>>>
>>>> But, if there isn't, should we disable PV vnuma, or warn the user that
>>>> he may see issues? Can we identify, in Xen or in toolstack, whether an
>>>> host topology will be problematic, and disable/warn in those cases 
>>>> too?
>>>>
>>>> I'm not sure, honestly. Disabling looks too aggressive, but it's an
>>>> issue I wouldn't like an user to be facing, without at least being
>>>> informed of the possibility... so, perhaps a (set of) warning(s)?
>>>> Thoughts?
>>>
>>> I think we have 2 possible solutions:
>>>
>>> 1. Try to handle this all in the hypervisor via CPUID mangling.
>>>
>>> 2. Add PV-topology support to the guest and indicate this capability 
>>> via
>>>    elfnote; only enable PV-numa if this note is present.
>>>
>>> I'd prefer the second solution. If you are okay with this, I'd try 
>>> to do
>>> some patches for the pvops kernel.

Why do you think that kernel patches are preferable to CPUID management? 
This would be all in tools, I'd think. (Well, one problem that I can 
think of is that AMD sometimes pokes at MSRs and/or Northbridge's PCI 
registers to figure out nodeID --- that we may need to have to address 
in the hypervisor)

And those patches won't help HVM guests, will they? How would they be 
useful by user processes?


-boris

>>
>> What if I configure a guest to follow HW topology? I.e. I pin VCPUs to
>> appropriate cores/threads? With elfnote I am stuck with disabled 
>> topology.
>
> Add an option to do exactly that: follow HW topology (pin vcpus,
> configure vnuma)?
>
> Add a force flag to the vnuma configuration to ignore the elfnote?
>
>> Besides, this is not necessarily a NUMA-only issue, it's a scheduling
>> one (inside the guest) as well.
>
> Sure. That's what Jan said regarding SUSE's xen-kernel. No toplogy info
> (or a trivial one) might be better than the wrong one...
>
> This patch for pvops should be written in any case. I'll do this, but it
> would be nice to know whether PV-numa should be considered or not.

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3779 bytes --]

On Wed, 2015-07-22 at 16:09 +0200, Juergen Gross wrote:
> On 07/22/2015 03:58 PM, Boris Ostrovsky wrote:

> > What if I configure a guest to follow HW topology? I.e. I pin VCPUs to
> > appropriate cores/threads? With elfnote I am stuck with disabled topology.
> 
> Add an option to do exactly that: follow HW topology (pin vcpus,
> configure vnuma)?
> 
I thought about configuring things in such a way that they match the
host topology, as Boris is suggesting, too. And in that case, I think
arranging for doing so in toolstack, if PV vNUMA is identified (as I
think Juergen is suggesting) seems a good approach.

However, when I try to do that on my box, manually, but I don't seem to
be able to.

Here's what I tried. Since I have this host topology:
cpu_topology           :
cpu:    core    socket     node
  0:       0        1        0
  1:       0        1        0
  2:       1        1        0
  3:       1        1        0
  4:       9        1        0
  5:       9        1        0
  6:      10        1        0
  7:      10        1        0
  8:       0        0        1
  9:       0        0        1
 10:       1        0        1
 11:       1        0        1
 12:       9        0        1
 13:       9        0        1
 14:      10        0        1
 15:      10        0        1

I configured the guest like this:
vcpus       = '4'
memory      = '1024'
vnuma = [ [ "pnode=0","size=512","vcpus=0-1","vdistances=10,20"  ],
          [ "pnode=1","size=512","vcpus=2-3","vdistances=20,10"  ] ]
cpus=["0","1","8","9"]

This means vcpus 0 and 1, which are assigned to vnode 0, are pinned to
pcpu 0 and 1, which are siblings, per the host topology.
Similarly, vcpus 2 and 3, assigned to vnode 1, are assigned to two
siblings pcpus on pnode 1.

This seems to be honoured:
# xl vcpu-list 4
Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
test                                 4     0    0   -b-      10.9  0 / 0-7
test                                 4     1    1   -b-       7.6  1 / 0-7
test                                 4     2    8   -b-       0.1  8 / 8-15
test                                 4     3    9   -b-       0.1  9 / 8-15

And yet, no joy:
# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
# xl vcpu-list 4
Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
test                                 4     0    0   r--      16.4  0 / 0-7
test                                 4     1    1   r--      12.5  1 / 0-7
test                                 4     2    8   -b-       0.2  8 / 8-15
test                                 4     3    9   -b-       0.1  9 / 8-15

So, what am I doing wrong at "following the hw topology"?

> > Besides, this is not necessarily a NUMA-only issue, it's a scheduling
> > one (inside the guest) as well.
> 
> Sure. That's what Jan said regarding SUSE's xen-kernel. No toplogy info
> (or a trivial one) might be better than the wrong one...
> 
Yep. Exacty. As Boris says, this is a generic scheduling issue, although
it's tru that it's only (as far as I can tell) with vNUMA that it bite
us so hard... I mean, performance are always going to be inconsistent,
but it's only in that case that you basically _loose_ some of the
vcpus! :-O

Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/22/2015 10:50 AM, Dario Faggioli wrote:
> On Wed, 2015-07-22 at 16:09 +0200, Juergen Gross wrote:
>> On 07/22/2015 03:58 PM, Boris Ostrovsky wrote:
>>> What if I configure a guest to follow HW topology? I.e. I pin VCPUs to
>>> appropriate cores/threads? With elfnote I am stuck with disabled topology.
>> Add an option to do exactly that: follow HW topology (pin vcpus,
>> configure vnuma)?
>>
> I thought about configuring things in such a way that they match the
> host topology, as Boris is suggesting, too. And in that case, I think
> arranging for doing so in toolstack, if PV vNUMA is identified (as I
> think Juergen is suggesting) seems a good approach.
>
> However, when I try to do that on my box, manually, but I don't seem to
> be able to.
>
> Here's what I tried. Since I have this host topology:
> cpu_topology           :
> cpu:    core    socket     node
>    0:       0        1        0
>    1:       0        1        0
>    2:       1        1        0
>    3:       1        1        0
>    4:       9        1        0
>    5:       9        1        0
>    6:      10        1        0
>    7:      10        1        0
>    8:       0        0        1
>    9:       0        0        1
>   10:       1        0        1
>   11:       1        0        1
>   12:       9        0        1
>   13:       9        0        1
>   14:      10        0        1
>   15:      10        0        1
>
> I configured the guest like this:
> vcpus       = '4'
> memory      = '1024'
> vnuma = [ [ "pnode=0","size=512","vcpus=0-1","vdistances=10,20"  ],
>            [ "pnode=1","size=512","vcpus=2-3","vdistances=20,10"  ] ]
> cpus=["0","1","8","9"]
>
> This means vcpus 0 and 1, which are assigned to vnode 0, are pinned to
> pcpu 0 and 1, which are siblings, per the host topology.
> Similarly, vcpus 2 and 3, assigned to vnode 1, are assigned to two
> siblings pcpus on pnode 1.
>
> This seems to be honoured:
> # xl vcpu-list 4
> Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
> test                                 4     0    0   -b-      10.9  0 / 0-7
> test                                 4     1    1   -b-       7.6  1 / 0-7
> test                                 4     2    8   -b-       0.1  8 / 8-15
> test                                 4     3    9   -b-       0.1  9 / 8-15
>
> And yet, no joy:
> # ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
> # ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
> # ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
> # ssh root@192.168.1.101 "yes > /dev/null 2>&1 &"
> # xl vcpu-list 4
> Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
> test                                 4     0    0   r--      16.4  0 / 0-7
> test                                 4     1    1   r--      12.5  1 / 0-7
> test                                 4     2    8   -b-       0.2  8 / 8-15
> test                                 4     3    9   -b-       0.1  9 / 8-15
>
> So, what am I doing wrong at "following the hw topology"?
>
>>> Besides, this is not necessarily a NUMA-only issue, it's a scheduling
>>> one (inside the guest) as well.
>> Sure. That's what Jan said regarding SUSE's xen-kernel. No toplogy info
>> (or a trivial one) might be better than the wrong one...
>>
> Yep. Exacty. As Boris says, this is a generic scheduling issue, although
> it's tru that it's only (as far as I can tell) with vNUMA that it bite
> us so hard...

I am not sure that it's only vNUMA. It's just that with vNUMA we can see 
a warning (on your system) that something goes wrong. In other cases 
(like scheduling, or sizing objects based on discovered cache sizes) we 
don't see anything in the log but system/programs are making wrong 
decisions. (And your results above may well be the example of that)

-boris


> I mean, performance are always going to be inconsistent,
> but it's only in that case that you basically _loose_ some of the
> vcpus! :-O
>
> Dario

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1830 bytes --]

On Wed, 2015-07-22 at 11:32 -0400, Boris Ostrovsky wrote:
> On 07/22/2015 10:50 AM, Dario Faggioli wrote:

> > Yep. Exacty. As Boris says, this is a generic scheduling issue, although
> > it's tru that it's only (as far as I can tell) with vNUMA that it bite
> > us so hard...
> 
> I am not sure that it's only vNUMA. It's just that with vNUMA we can see 
> a warning (on your system) that something goes wrong. In other cases 
> (like scheduling, or sizing objects based on discovered cache sizes) we 
> don't see anything in the log but system/programs are making wrong 
> decisions. 
>
I'm not questioning that the guest scheduler is put in a position where
it may make weird and/or wrong choices... Let's put it like this: on
that very test box, or any other NUMA box I've worked on, I've never
seen performance affected _so_much_ as in the vNUMA case.

In fact, of course there are other issues (like the ones you're
mentioning, caused by this), but it's only with vNUMA that I see 2 out
of 4 vcpus completely lost! :-/

> (And your results above may well be the example of that)
> 
Right. BTW, the example was meant at investigating what you suggested,
i.e., to have the guest topology follow the host topology. I tried, but
could not achieve it. Do you think I'm doing something wrong? May it be
that it's not always doable (ISTR Andrew saying that it is also the pcpu
where the vcpus are created that matters)?

I'm (re)asking because, if you think that's feasible, I can work on
that. I fit's not, well... :-/

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/22/2015 11:49 AM, Dario Faggioli wrote:
> On Wed, 2015-07-22 at 11:32 -0400, Boris Ostrovsky wrote:
>> On 07/22/2015 10:50 AM, Dario Faggioli wrote:
>>> Yep. Exacty. As Boris says, this is a generic scheduling issue, although
>>> it's tru that it's only (as far as I can tell) with vNUMA that it bite
>>> us so hard...
>> I am not sure that it's only vNUMA. It's just that with vNUMA we can see
>> a warning (on your system) that something goes wrong. In other cases
>> (like scheduling, or sizing objects based on discovered cache sizes) we
>> don't see anything in the log but system/programs are making wrong
>> decisions.
>>
> I'm not questioning that the guest scheduler is put in a position where
> it may make weird and/or wrong choices... Let's put it like this: on
> that very test box, or any other NUMA box I've worked on, I've never
> seen performance affected _so_much_ as in the vNUMA case.
>
> In fact, of course there are other issues (like the ones you're
> mentioning, caused by this), but it's only with vNUMA that I see 2 out
> of 4 vcpus completely lost! :-/

My guess would be that scheduling domains are not set up properly. And 
perhaps because your test has lots of IO the load doesn't cross 
thresholds that are needed to switch to another domain?

Does it work the same way if you run a compiled busy loop (i.e. 
'main(){while(1);}')?

>
>> (And your results above may well be the example of that)
>>
> Right. BTW, the example was meant at investigating what you suggested,
> i.e., to have the guest topology follow the host topology. I tried, but
> could not achieve it. Do you think I'm doing something wrong? May it be
> that it's not always doable (ISTR Andrew saying that it is also the pcpu
> where the vcpus are created that matters)?

I don't think this is currently doable with what we have for CPUID 
support in xl syntax. I am pretty sure we need to at least be able to 
specify all leaf 4's indexes. And we can't.

BTW, irrespective of this particular problem, adding support for indexed 
CPUID leaves would be a good idea.

-boris

>
> I'm (re)asking because, if you think that's feasible, I can work on
> that. I fit's not, well... :-/

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/22/2015 04:44 PM, Boris Ostrovsky wrote:
> On 07/22/2015 10:09 AM, Juergen Gross wrote:
>> On 07/22/2015 03:58 PM, Boris Ostrovsky wrote:
>>> On 07/22/2015 09:50 AM, Juergen Gross wrote:
>>>> On 07/22/2015 03:36 PM, Dario Faggioli wrote:
>>>>> On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote:
>>>>>> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote:
>>>>>>> On 07/20/2015 10:09 AM, Dario Faggioli wrote:
>>>>>
>>>>>>> I'll need to see how LLC IDs are calculated, probably also from some
>>>>>>> CPUID bits.
>>>>>>
>>>>>>
>>>>>> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel)
>>>>>> which
>>>>>> use indexes in ECX register and xl syntax doesn't allow you to
>>>>>> override
>>>>>> CPUIDs for such leaves.
>>>>>>
>>>>> Right. Which leaves us with the question of what should we do and/or
>>>>> recommend users to do?
>>>>>
>>>>> If there were a workaround that we could put in place, and document
>>>>> somewhere, however tricky it was, I'd say to go for it, and call it
>>>>> acceptable for now.
>>>>>
>>>>> But, if there isn't, should we disable PV vnuma, or warn the user that
>>>>> he may see issues? Can we identify, in Xen or in toolstack, whether an
>>>>> host topology will be problematic, and disable/warn in those cases
>>>>> too?
>>>>>
>>>>> I'm not sure, honestly. Disabling looks too aggressive, but it's an
>>>>> issue I wouldn't like an user to be facing, without at least being
>>>>> informed of the possibility... so, perhaps a (set of) warning(s)?
>>>>> Thoughts?
>>>>
>>>> I think we have 2 possible solutions:
>>>>
>>>> 1. Try to handle this all in the hypervisor via CPUID mangling.
>>>>
>>>> 2. Add PV-topology support to the guest and indicate this capability
>>>> via
>>>>    elfnote; only enable PV-numa if this note is present.
>>>>
>>>> I'd prefer the second solution. If you are okay with this, I'd try
>>>> to do
>>>> some patches for the pvops kernel.
>
> Why do you think that kernel patches are preferable to CPUID management?
> This would be all in tools, I'd think. (Well, one problem that I can
> think of is that AMD sometimes pokes at MSRs and/or Northbridge's PCI
> registers to figure out nodeID --- that we may need to have to address
> in the hypervisor)

Doing it via CPUID is more HW specific. Trying to fake a topology for
the guest from outside might lead to weird decisions in the guest e.g.
regarding licenses based on socket counts.

If you are doing it in the guest itself you are able to address the
different problems (scheduling, licensing) in different ways.

> And those patches won't help HVM guests, will they? How would they be
> useful by user processes?

HVM can use pv interfaces as well. It's called pv-NUMA :-)

Hmm, I didn't think of user processes. Are you aware of cases where they
are to be considered? The only case where user processes are involved I
could think of is licensing again. Depending on the licensing model
playing with CPUID is either good or bad. I can even imagine the CPUID
configuration capabilities in xl are in use today for exactly this
purpose. Using them for pv-NUMA as well will make this feature unusable
for those users.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Jan Beulich]

>>> On 22.07.15 at 20:10, <boris.ostrovsky@oracle.com> wrote:
> I don't think this is currently doable with what we have for CPUID 
> support in xl syntax. I am pretty sure we need to at least be able to 
> specify all leaf 4's indexes. And we can't.
> 
> BTW, irrespective of this particular problem, adding support for indexed 
> CPUID leaves would be a good idea.

So what is it that makes you think this doesn't work already?
xc_cpuid_policy() intentionally has "input" as a pointer,
allowing not just EAX to be specified. And libxl_cpuid.c also
has

static const char *input_names[2] = { "leaf", "subleaf" };

Jan

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Jan Beulich]

>>> On 23.07.15 at 06:43, <JGross@suse.com> wrote:
> Hmm, I didn't think of user processes. Are you aware of cases where they
> are to be considered?

Why wouldn't a sophisticated user mode program attempt to adjust
certain memory objects' sizes based on cache size?

Jan

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Andrew Cooper]

On 23/07/15 05:43, Juergen Gross wrote:
> On 07/22/2015 04:44 PM, Boris Ostrovsky wrote:
>> On 07/22/2015 10:09 AM, Juergen Gross wrote:
>>> On 07/22/2015 03:58 PM, Boris Ostrovsky wrote:
>>>> On 07/22/2015 09:50 AM, Juergen Gross wrote:
>>>>> On 07/22/2015 03:36 PM, Dario Faggioli wrote:
>>>>>> On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote:
>>>>>>> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote:
>>>>>>>> On 07/20/2015 10:09 AM, Dario Faggioli wrote:
>>>>>>
>>>>>>>> I'll need to see how LLC IDs are calculated, probably also from
>>>>>>>> some
>>>>>>>> CPUID bits.
>>>>>>>
>>>>>>>
>>>>>>> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel)
>>>>>>> which
>>>>>>> use indexes in ECX register and xl syntax doesn't allow you to
>>>>>>> override
>>>>>>> CPUIDs for such leaves.
>>>>>>>
>>>>>> Right. Which leaves us with the question of what should we do and/or
>>>>>> recommend users to do?
>>>>>>
>>>>>> If there were a workaround that we could put in place, and document
>>>>>> somewhere, however tricky it was, I'd say to go for it, and call it
>>>>>> acceptable for now.
>>>>>>
>>>>>> But, if there isn't, should we disable PV vnuma, or warn the user
>>>>>> that
>>>>>> he may see issues? Can we identify, in Xen or in toolstack,
>>>>>> whether an
>>>>>> host topology will be problematic, and disable/warn in those cases
>>>>>> too?
>>>>>>
>>>>>> I'm not sure, honestly. Disabling looks too aggressive, but it's an
>>>>>> issue I wouldn't like an user to be facing, without at least being
>>>>>> informed of the possibility... so, perhaps a (set of) warning(s)?
>>>>>> Thoughts?
>>>>>
>>>>> I think we have 2 possible solutions:
>>>>>
>>>>> 1. Try to handle this all in the hypervisor via CPUID mangling.
>>>>>
>>>>> 2. Add PV-topology support to the guest and indicate this capability
>>>>> via
>>>>>    elfnote; only enable PV-numa if this note is present.
>>>>>
>>>>> I'd prefer the second solution. If you are okay with this, I'd try
>>>>> to do
>>>>> some patches for the pvops kernel.
>>
>> Why do you think that kernel patches are preferable to CPUID management?
>> This would be all in tools, I'd think. (Well, one problem that I can
>> think of is that AMD sometimes pokes at MSRs and/or Northbridge's PCI
>> registers to figure out nodeID --- that we may need to have to address
>> in the hypervisor)
>
> Doing it via CPUID is more HW specific. Trying to fake a topology for
> the guest from outside might lead to weird decisions in the guest e.g.
> regarding licenses based on socket counts.
>
> If you are doing it in the guest itself you are able to address the
> different problems (scheduling, licensing) in different ways.
>
>> And those patches won't help HVM guests, will they? How would they be
>> useful by user processes?
>
> HVM can use pv interfaces as well. It's called pv-NUMA :-)
>
> Hmm, I didn't think of user processes. Are you aware of cases where they
> are to be considered? The only case where user processes are involved I
> could think of is licensing again. Depending on the licensing model
> playing with CPUID is either good or bad. I can even imagine the CPUID
> configuration capabilities in xl are in use today for exactly this
> purpose. Using them for pv-NUMA as well will make this feature unusable
> for those users.

Userspace can also use things like hwloc which use cpuid to calculate
efficient allocation of resources, rather than for licensing purposes.

~Andrew

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1057 bytes --]

On Wed, 2015-07-22 at 14:10 -0400, Boris Ostrovsky wrote:
> On 07/22/2015 11:49 AM, Dario Faggioli wrote:

> > In fact, of course there are other issues (like the ones you're
> > mentioning, caused by this), but it's only with vNUMA that I see 2 out
> > of 4 vcpus completely lost! :-/
> 
> My guess would be that scheduling domains are not set up properly. And 
> perhaps because your test has lots of IO the load doesn't cross 
> thresholds that are needed to switch to another domain?
> 
I also things sched domains are involved. However, my test has no IO at
all, actually.

> Does it work the same way if you run a compiled busy loop (i.e. 
> 'main(){while(1);}')?
> 
Well, I can try, but I think that 'yes > /dev/null' (a bunch of them),
is really close to that.

Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3473 bytes --]

On Thu, 2015-07-23 at 06:43 +0200, Juergen Gross wrote:
> On 07/22/2015 04:44 PM, Boris Ostrovsky wrote:
> > On 07/22/2015 10:09 AM, Juergen Gross wrote:

> >>>> I think we have 2 possible solutions:
> >>>>
> >>>> 1. Try to handle this all in the hypervisor via CPUID mangling.
> >>>>
> >>>> 2. Add PV-topology support to the guest and indicate this capability
> >>>> via
> >>>>    elfnote; only enable PV-numa if this note is present.
> >>>>
> >>>> I'd prefer the second solution. If you are okay with this, I'd try
> >>>> to do
> >>>> some patches for the pvops kernel.
> >
> > Why do you think that kernel patches are preferable to CPUID management?
> > This would be all in tools, I'd think. (Well, one problem that I can
> > think of is that AMD sometimes pokes at MSRs and/or Northbridge's PCI
> > registers to figure out nodeID --- that we may need to have to address
> > in the hypervisor)
> 
> Doing it via CPUID is more HW specific. Trying to fake a topology for
> the guest from outside might lead to weird decisions in the guest e.g.
> regarding licenses based on socket counts.
> 
I do see the value of this, I think...

> If you are doing it in the guest itself you are able to address the
> different problems (scheduling, licensing) in different ways.
> 
... but, at least in the case of vNUMA for instance, there still need to
be a correlation between the vNUMA topology, and the "CPUID topology",
and vNUMA topology is decided by toolstack.

Then, if you mean, within all the possible solutions that matches (i.e.,
that does not cause problems to!) the vNUMA setup we've been given,
let's pick up one that also is best for this xxx other purpose, then I
agree.

What I'm not sure I see, although, is how you would be specifying the
other purpose, e.g., in this case, are you thinking to another parameter
saying that we want to minimize the socket count?

> Depending on the licensing model
> playing with CPUID is either good or bad. I can even imagine the CPUID
> configuration capabilities in xl are in use today for exactly this
> purpose. Using them for pv-NUMA as well will make this feature unusable
> for those users.
>
Yeah, well... So, you want a VM with only one socket, because of
whatever reasons (say licensing), and you're using libxl's CPUID
fiddling capability to do that. Now, if you specify, for such a VM, a
vNUMA layout with 2 vnodes, well, I'd call this asking for troubles. I
know, strictly speaking, socket != (v)NUMA node. Still, I think this
will be a corner case, way less common than just a user specifying a
vNUMA topology, and getting only a fraction of the vcpus being
used/usable! :-/

In summary, I probably know too few of CPUID handling to have a clear
view on whether something like 'making it match the topology' --which
also means, if no vNUMA, CPUID should say flat, for some definition of
flat-- should leave in tools or in kernel... I just know that we need to
do something *consistent*.

FWIW, I was thinking that the kernel were a better place, as Juergen is
saying, while now I'm more convinced that tools would be more
appropriate, as Boris is saying.

Thanks and Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/23/2015 04:07 PM, Dario Faggioli wrote:
> On Thu, 2015-07-23 at 06:43 +0200, Juergen Gross wrote:
>> On 07/22/2015 04:44 PM, Boris Ostrovsky wrote:
>>> On 07/22/2015 10:09 AM, Juergen Gross wrote:
>
>>>>>> I think we have 2 possible solutions:
>>>>>>
>>>>>> 1. Try to handle this all in the hypervisor via CPUID mangling.
>>>>>>
>>>>>> 2. Add PV-topology support to the guest and indicate this capability
>>>>>> via
>>>>>>     elfnote; only enable PV-numa if this note is present.
>>>>>>
>>>>>> I'd prefer the second solution. If you are okay with this, I'd try
>>>>>> to do
>>>>>> some patches for the pvops kernel.
>>>
>>> Why do you think that kernel patches are preferable to CPUID management?
>>> This would be all in tools, I'd think. (Well, one problem that I can
>>> think of is that AMD sometimes pokes at MSRs and/or Northbridge's PCI
>>> registers to figure out nodeID --- that we may need to have to address
>>> in the hypervisor)
>>
>> Doing it via CPUID is more HW specific. Trying to fake a topology for
>> the guest from outside might lead to weird decisions in the guest e.g.
>> regarding licenses based on socket counts.
>>
> I do see the value of this, I think...
>
>> If you are doing it in the guest itself you are able to address the
>> different problems (scheduling, licensing) in different ways.
>>
> ... but, at least in the case of vNUMA for instance, there still need to
> be a correlation between the vNUMA topology, and the "CPUID topology",
> and vNUMA topology is decided by toolstack.
>
> Then, if you mean, within all the possible solutions that matches (i.e.,
> that does not cause problems to!) the vNUMA setup we've been given,
> let's pick up one that also is best for this xxx other purpose, then I
> agree.
>
> What I'm not sure I see, although, is how you would be specifying the
> other purpose, e.g., in this case, are you thinking to another parameter
> saying that we want to minimize the socket count?
>
>> Depending on the licensing model
>> playing with CPUID is either good or bad. I can even imagine the CPUID
>> configuration capabilities in xl are in use today for exactly this
>> purpose. Using them for pv-NUMA as well will make this feature unusable
>> for those users.
>>
> Yeah, well... So, you want a VM with only one socket, because of
> whatever reasons (say licensing), and you're using libxl's CPUID
> fiddling capability to do that. Now, if you specify, for such a VM, a
> vNUMA layout with 2 vnodes, well, I'd call this asking for troubles. I
> know, strictly speaking, socket != (v)NUMA node. Still, I think this
> will be a corner case, way less common than just a user specifying a
> vNUMA topology, and getting only a fraction of the vcpus being
> used/usable! :-/
>
> In summary, I probably know too few of CPUID handling to have a clear
> view on whether something like 'making it match the topology' --which
> also means, if no vNUMA, CPUID should say flat, for some definition of
> flat-- should leave in tools or in kernel... I just know that we need to
> do something *consistent*.
>
> FWIW, I was thinking that the kernel were a better place, as Juergen is
> saying, while now I'm more convinced that tools would be more
> appropriate, as Boris is saying.

I'm just about to write down all sources of information the linux kernel
is using to build topology information and for what purpose this data is
being used. This should help to decide which way to go.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/23/2015 04:07 PM, Dario Faggioli wrote:
> On Thu, 2015-07-23 at 06:43 +0200, Juergen Gross wrote:
>> On 07/22/2015 04:44 PM, Boris Ostrovsky wrote:
>>> On 07/22/2015 10:09 AM, Juergen Gross wrote:
>
>>>>>> I think we have 2 possible solutions:
>>>>>>
>>>>>> 1. Try to handle this all in the hypervisor via CPUID mangling.
>>>>>>
>>>>>> 2. Add PV-topology support to the guest and indicate this capability
>>>>>> via
>>>>>>     elfnote; only enable PV-numa if this note is present.
>>>>>>
>>>>>> I'd prefer the second solution. If you are okay with this, I'd try
>>>>>> to do
>>>>>> some patches for the pvops kernel.
>>>
>>> Why do you think that kernel patches are preferable to CPUID management?
>>> This would be all in tools, I'd think. (Well, one problem that I can
>>> think of is that AMD sometimes pokes at MSRs and/or Northbridge's PCI
>>> registers to figure out nodeID --- that we may need to have to address
>>> in the hypervisor)
>>
>> Doing it via CPUID is more HW specific. Trying to fake a topology for
>> the guest from outside might lead to weird decisions in the guest e.g.
>> regarding licenses based on socket counts.
>>
> I do see the value of this, I think...
>
>> If you are doing it in the guest itself you are able to address the
>> different problems (scheduling, licensing) in different ways.
>>
> ... but, at least in the case of vNUMA for instance, there still need to
> be a correlation between the vNUMA topology, and the "CPUID topology",
> and vNUMA topology is decided by toolstack.
>
> Then, if you mean, within all the possible solutions that matches (i.e.,
> that does not cause problems to!) the vNUMA setup we've been given,
> let's pick up one that also is best for this xxx other purpose, then I
> agree.
>
> What I'm not sure I see, although, is how you would be specifying the
> other purpose, e.g., in this case, are you thinking to another parameter
> saying that we want to minimize the socket count?
>
>> Depending on the licensing model
>> playing with CPUID is either good or bad. I can even imagine the CPUID
>> configuration capabilities in xl are in use today for exactly this
>> purpose. Using them for pv-NUMA as well will make this feature unusable
>> for those users.
>>
> Yeah, well... So, you want a VM with only one socket, because of
> whatever reasons (say licensing), and you're using libxl's CPUID
> fiddling capability to do that. Now, if you specify, for such a VM, a
> vNUMA layout with 2 vnodes, well, I'd call this asking for troubles. I
> know, strictly speaking, socket != (v)NUMA node. Still, I think this
> will be a corner case, way less common than just a user specifying a
> vNUMA topology, and getting only a fraction of the vcpus being
> used/usable! :-/
>
> In summary, I probably know too few of CPUID handling to have a clear
> view on whether something like 'making it match the topology' --which
> also means, if no vNUMA, CPUID should say flat, for some definition of
> flat-- should leave in tools or in kernel... I just know that we need to
> do something *consistent*.
>
> FWIW, I was thinking that the kernel were a better place, as Juergen is
> saying, while now I'm more convinced that tools would be more
> appropriate, as Boris is saying.

I've collected some information from the linux kernel sources as a base
for the discussion:

The complete numa information (cpu->node and memory->node relations) is
taken from the acpi tables (srat, slit for "distances").

The topology information is obtained via:
- intel:
   + cpuid leaf b with subleafs, leaf 4
   + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available
- amd:
   + cpuid leaf 8000001e, leaf 8000001d, leaf 4
   + msr c001100c
   + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available

The scheduler is aware of:
- smt siblings (from topology)
- last-level-cache siblings (from topology)
- node siblings (from numa information)
It will especially move tasks from one cpu to another first between smt
siblings, second between llc siblings, third between node siblings and
last all cpus.

Memory management does numa node aware memory allocation.

Topology and numa information are made available through /sys and /proc
filesystems.

cpuid instruction is available for user mode as well.


Anything I have missed?


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 4588 bytes --]

On Fri, 2015-07-24 at 12:28 +0200, Juergen Gross wrote:
> On 07/23/2015 04:07 PM, Dario Faggioli wrote:

> > FWIW, I was thinking that the kernel were a better place, as Juergen is
> > saying, while now I'm more convinced that tools would be more
> > appropriate, as Boris is saying.
> 
> I've collected some information from the linux kernel sources as a base
> for the discussion:
> 
That's great, thanks for this!

> The complete numa information (cpu->node and memory->node relations) is
> taken from the acpi tables (srat, slit for "distances").
> 
Ok. And I already have a question (as I lost track of things a bit).
What you just said about ACPI tables is certainly true for baremetal and
HVM guests, but for PV? At the time I was looking into it, together with
Elena, there were Linux patches being produced for the PV case, which
makes sense.
However, ISTR that both Wei and Elena mentioned recently that those
patches have not been upstreamed in Linux yet... Is that the case? Maybe
not all, but at least some of them are there? Because if not, I'm not
sure I see how a PV guest would even see a vNUMA topology (which it
does).

Of course, I can go and check, but since you just looked, you may have
it fresh and clear already. :-)

> The topology information is obtained via:
> - intel:
>    + cpuid leaf b with subleafs, leaf 4
>    + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available
> - amd:
>    + cpuid leaf 8000001e, leaf 8000001d, leaf 4
>    + msr c001100c
>    + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available
> 
> The scheduler is aware of:
> - smt siblings (from topology)
> - last-level-cache siblings (from topology)
> - node siblings (from numa information)
>
Right. So, this confirms what we were guessing: we need to "reconcile"
these two sources of information (from the guest point of view).

Both the 'in kernel' and 'in toolstack' approach should have all the
necessary information to make things match, I think. In fact, in
toolstack, we know what the vNUMA topology is (we're parsing and
actually putting it in place!). In kernel, we know it as we read it from
tables or hypercalls (isn't that so, for PV guest?).

In fact, I think that it is the topology, i.e., what comes from MSRs,
that needs to adapt, and follow vNUMA, as much as possible. Do we agree
on this?

IMO, the thing boils down to these:

 1) from where (kernel vs. toolstack) is it the most easy and effective
    to enact the CPUID fiddling? As in, can we do that in toolstack?
    (Andrew was not so sure, and Boris found issues, although Jan seems
    to think they're no show stopper.)
    I'm quite certain that we can do that from inside the kernel,
    although, how early would we need to be doing it? Do we have the
    vNUMA info already?

 2) when tweaking the values of CPUID and other MSRs, are there other
    vNUMA (and topology in general) constraints and requirements we
    should take into account? For instance, do we want, for licensing
    reasons, all (or most) of the vcpus to be siblings, rather than full
    sockets? Etc.
     2a) if yes, how and where are these constraints specified?

If looking at 1) only, it still looks to me that doing things within the
kernel would be the way to go.

When looking at 2), OTOH, toolstacks variants start to be more
appealing. Especially depending on our answer to 2a). In fact,
in case we want to give the user a way to specify this
siblings-vs-cores-vs-sockets information, it IMO would be good to deal
with that in tools, rather than having to involve Xen or Linux!

> It will especially move tasks from one cpu to another first between smt
> siblings, second between llc siblings, third between node siblings and
> last all cpus.
> 
Yep, this part, I knew.

Maybe, there is room for "fixing" this at this level, hooking up inside
the scheduler code... but I'm shooting in the dark, without having check
whether and how this could be really feasible, should I?

One thing I don't like about this approach is that it would potentially
solve vNUMA and other scheduling anomalies, but...

> cpuid instruction is available for user mode as well.
> 
...it would not do any good for other subsystems, and user level code
and apps.

Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/24/2015 04:44 PM, Dario Faggioli wrote:
> On Fri, 2015-07-24 at 12:28 +0200, Juergen Gross wrote:
>> On 07/23/2015 04:07 PM, Dario Faggioli wrote:
>
>>> FWIW, I was thinking that the kernel were a better place, as Juergen is
>>> saying, while now I'm more convinced that tools would be more
>>> appropriate, as Boris is saying.
>>
>> I've collected some information from the linux kernel sources as a base
>> for the discussion:
>>
> That's great, thanks for this!
>
>> The complete numa information (cpu->node and memory->node relations) is
>> taken from the acpi tables (srat, slit for "distances").
>>
> Ok. And I already have a question (as I lost track of things a bit).
> What you just said about ACPI tables is certainly true for baremetal and
> HVM guests, but for PV? At the time I was looking into it, together with
> Elena, there were Linux patches being produced for the PV case, which
> makes sense.
> However, ISTR that both Wei and Elena mentioned recently that those
> patches have not been upstreamed in Linux yet... Is that the case? Maybe
> not all, but at least some of them are there? Because if not, I'm not
> sure I see how a PV guest would even see a vNUMA topology (which it
> does).
>
> Of course, I can go and check, but since you just looked, you may have
> it fresh and clear already. :-)

I checked "bottom up", so when I found the acpi scan stuff I stopped
searching how the kernel obtains numa info. During my search I found no
clue of an pv-numa stuff in the kernel. And a quick "grep -i numa" in
arch/x86/xen and drivers/xen didn't reveal anything. Same for a complete
kernel source search for "vnuma".

>
>> The topology information is obtained via:
>> - intel:
>>     + cpuid leaf b with subleafs, leaf 4
>>     + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available
>> - amd:
>>     + cpuid leaf 8000001e, leaf 8000001d, leaf 4
>>     + msr c001100c
>>     + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available
>>
>> The scheduler is aware of:
>> - smt siblings (from topology)
>> - last-level-cache siblings (from topology)
>> - node siblings (from numa information)
>>
> Right. So, this confirms what we were guessing: we need to "reconcile"
> these two sources of information (from the guest point of view).
>
> Both the 'in kernel' and 'in toolstack' approach should have all the
> necessary information to make things match, I think. In fact, in
> toolstack, we know what the vNUMA topology is (we're parsing and
> actually putting it in place!). In kernel, we know it as we read it from
> tables or hypercalls (isn't that so, for PV guest?).
>
> In fact, I think that it is the topology, i.e., what comes from MSRs,
> that needs to adapt, and follow vNUMA, as much as possible. Do we agree
> on this?

I think we have to be very careful here. I see two possible scenarios:

1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will
    try to schedule the vcpus according to their numa affinity. So they
    can change pcpus at any time in case of very busy guests. I don't
    think the linux kernel should treat the cpus differently in this
    case as it will be in vane regarding the Xen scheduler's activity.
    So we should use the "null" topology in this case.

2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen
    scheduler can't move vcpus between pcpus, so the linux kernel should
    see the real topology of the used pcpus in order to optimize for this
    picture.

This only covers the scheduling aspect, of course.

>
> IMO, the thing boils down to these:
>
>   1) from where (kernel vs. toolstack) is it the most easy and effective
>      to enact the CPUID fiddling? As in, can we do that in toolstack?
>      (Andrew was not so sure, and Boris found issues, although Jan seems
>      to think they're no show stopper.)
>      I'm quite certain that we can do that from inside the kernel,
>      although, how early would we need to be doing it? Do we have the
>      vNUMA info already?
>
>   2) when tweaking the values of CPUID and other MSRs, are there other
>      vNUMA (and topology in general) constraints and requirements we
>      should take into account? For instance, do we want, for licensing
>      reasons, all (or most) of the vcpus to be siblings, rather than full
>      sockets? Etc.
>       2a) if yes, how and where are these constraints specified?
>
> If looking at 1) only, it still looks to me that doing things within the
> kernel would be the way to go.
>
> When looking at 2), OTOH, toolstacks variants start to be more
> appealing. Especially depending on our answer to 2a). In fact,
> in case we want to give the user a way to specify this
> siblings-vs-cores-vs-sockets information, it IMO would be good to deal
> with that in tools, rather than having to involve Xen or Linux!
>
>> It will especially move tasks from one cpu to another first between smt
>> siblings, second between llc siblings, third between node siblings and
>> last all cpus.
>>
> Yep, this part, I knew.
>
> Maybe, there is room for "fixing" this at this level, hooking up inside
> the scheduler code... but I'm shooting in the dark, without having check
> whether and how this could be really feasible, should I?

Uuh, I don't think a change of the scheduler on behalf of Xen is really
appreciated. :-)

I'd rather fiddle with the cpu masks on the different levels to let the
scheduler do the right thing.

> One thing I don't like about this approach is that it would potentially
> solve vNUMA and other scheduling anomalies, but...
>
>> cpuid instruction is available for user mode as well.
>>
> ...it would not do any good for other subsystems, and user level code
> and apps.

Indeed. I think the optimal solution would be two-fold: give the
scheduler the information it is needing to react correctly via a
kernel patch not relying on cpuid values and fiddle with the cpuid
values from xen tools according to any needs of other subsystems and/or
user code (e.g. licensing).


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/24/2015 05:14 PM, Juergen Gross wrote:
> On 07/24/2015 04:44 PM, Dario Faggioli wrote:
>> On Fri, 2015-07-24 at 12:28 +0200, Juergen Gross wrote:
>>> On 07/23/2015 04:07 PM, Dario Faggioli wrote:
>>
>>>> FWIW, I was thinking that the kernel were a better place, as Juergen is
>>>> saying, while now I'm more convinced that tools would be more
>>>> appropriate, as Boris is saying.
>>>
>>> I've collected some information from the linux kernel sources as a base
>>> for the discussion:
>>>
>> That's great, thanks for this!
>>
>>> The complete numa information (cpu->node and memory->node relations) is
>>> taken from the acpi tables (srat, slit for "distances").
>>>
>> Ok. And I already have a question (as I lost track of things a bit).
>> What you just said about ACPI tables is certainly true for baremetal and
>> HVM guests, but for PV? At the time I was looking into it, together with
>> Elena, there were Linux patches being produced for the PV case, which
>> makes sense.
>> However, ISTR that both Wei and Elena mentioned recently that those
>> patches have not been upstreamed in Linux yet... Is that the case? Maybe
>> not all, but at least some of them are there? Because if not, I'm not
>> sure I see how a PV guest would even see a vNUMA topology (which it
>> does).
>>
>> Of course, I can go and check, but since you just looked, you may have
>> it fresh and clear already. :-)
>
> I checked "bottom up", so when I found the acpi scan stuff I stopped
> searching how the kernel obtains numa info. During my search I found no
> clue of an pv-numa stuff in the kernel. And a quick "grep -i numa" in
> arch/x86/xen and drivers/xen didn't reveal anything. Same for a complete
> kernel source search for "vnuma".
>
>>
>>> The topology information is obtained via:
>>> - intel:
>>>     + cpuid leaf b with subleafs, leaf 4
>>>     + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available
>>> - amd:
>>>     + cpuid leaf 8000001e, leaf 8000001d, leaf 4
>>>     + msr c001100c
>>>     + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available
>>>
>>> The scheduler is aware of:
>>> - smt siblings (from topology)
>>> - last-level-cache siblings (from topology)
>>> - node siblings (from numa information)
>>>
>> Right. So, this confirms what we were guessing: we need to "reconcile"
>> these two sources of information (from the guest point of view).
>>
>> Both the 'in kernel' and 'in toolstack' approach should have all the
>> necessary information to make things match, I think. In fact, in
>> toolstack, we know what the vNUMA topology is (we're parsing and
>> actually putting it in place!). In kernel, we know it as we read it from
>> tables or hypercalls (isn't that so, for PV guest?).
>>
>> In fact, I think that it is the topology, i.e., what comes from MSRs,
>> that needs to adapt, and follow vNUMA, as much as possible. Do we agree
>> on this?
>
> I think we have to be very careful here. I see two possible scenarios:
>
> 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will
>     try to schedule the vcpus according to their numa affinity. So they
>     can change pcpus at any time in case of very busy guests. I don't
>     think the linux kernel should treat the cpus differently in this
>     case as it will be in vane regarding the Xen scheduler's activity.
>     So we should use the "null" topology in this case.

Sorry, the topology should reflect the vcpu<->numa-node relations, of
course, but nothing else (so flat topolgy in each numa node).

>
> 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen
>     scheduler can't move vcpus between pcpus, so the linux kernel should
>     see the real topology of the used pcpus in order to optimize for this
>     picture.
>
> This only covers the scheduling aspect, of course.
>
>>
>> IMO, the thing boils down to these:
>>
>>   1) from where (kernel vs. toolstack) is it the most easy and effective
>>      to enact the CPUID fiddling? As in, can we do that in toolstack?
>>      (Andrew was not so sure, and Boris found issues, although Jan seems
>>      to think they're no show stopper.)
>>      I'm quite certain that we can do that from inside the kernel,
>>      although, how early would we need to be doing it? Do we have the
>>      vNUMA info already?
>>
>>   2) when tweaking the values of CPUID and other MSRs, are there other
>>      vNUMA (and topology in general) constraints and requirements we
>>      should take into account? For instance, do we want, for licensing
>>      reasons, all (or most) of the vcpus to be siblings, rather than full
>>      sockets? Etc.
>>       2a) if yes, how and where are these constraints specified?
>>
>> If looking at 1) only, it still looks to me that doing things within the
>> kernel would be the way to go.
>>
>> When looking at 2), OTOH, toolstacks variants start to be more
>> appealing. Especially depending on our answer to 2a). In fact,
>> in case we want to give the user a way to specify this
>> siblings-vs-cores-vs-sockets information, it IMO would be good to deal
>> with that in tools, rather than having to involve Xen or Linux!
>>
>>> It will especially move tasks from one cpu to another first between smt
>>> siblings, second between llc siblings, third between node siblings and
>>> last all cpus.
>>>
>> Yep, this part, I knew.
>>
>> Maybe, there is room for "fixing" this at this level, hooking up inside
>> the scheduler code... but I'm shooting in the dark, without having check
>> whether and how this could be really feasible, should I?
>
> Uuh, I don't think a change of the scheduler on behalf of Xen is really
> appreciated. :-)
>
> I'd rather fiddle with the cpu masks on the different levels to let the
> scheduler do the right thing.
>
>> One thing I don't like about this approach is that it would potentially
>> solve vNUMA and other scheduling anomalies, but...
>>
>>> cpuid instruction is available for user mode as well.
>>>
>> ...it would not do any good for other subsystems, and user level code
>> and apps.
>
> Indeed. I think the optimal solution would be two-fold: give the
> scheduler the information it is needing to react correctly via a
> kernel patch not relying on cpuid values and fiddle with the cpuid
> values from xen tools according to any needs of other subsystems and/or
> user code (e.g. licensing).


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3832 bytes --]

On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote:
> On 07/24/2015 05:14 PM, Juergen Gross wrote:
> > On 07/24/2015 04:44 PM, Dario Faggioli wrote:

> >> In fact, I think that it is the topology, i.e., what comes from MSRs,
> >> that needs to adapt, and follow vNUMA, as much as possible. Do we agree
> >> on this?
> >
> > I think we have to be very careful here. I see two possible scenarios:
> >
> > 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will
> >     try to schedule the vcpus according to their numa affinity. So they
> >     can change pcpus at any time in case of very busy guests. I don't
> >     think the linux kernel should treat the cpus differently in this
> >     case as it will be in vane regarding the Xen scheduler's activity.
> >     So we should use the "null" topology in this case.
> 
> Sorry, the topology should reflect the vcpu<->numa-node relations, of
> course, but nothing else (so flat topolgy in each numa node).
> 
Yeah, I was replying to this point saying something like this right
now... Luckily, I've seen this email! :-P

With this semantic, I fully agree with this.

> > 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen
> >     scheduler can't move vcpus between pcpus, so the linux kernel should
> >     see the real topology of the used pcpus in order to optimize for this
> >     picture.
> >
>
Mmm... I did think about this too, but I'm not sure. I see the value of
this of course, and the reason why it makes sense. However, pinning can
change on-line, via `xl vcpu-pin' and stuff. Also migration could make
things less certain, I think. What happens if we build on top of the
initial pinning, and then things change?

To be fair, there is stuff building on top of the initial pinning
already, e.g., from which physical NUMA node we allocate the memory
relies depends exactly on that. That being said, I'm not sure I'm
comfortable with adding more of this...

Perhaps introduce an 'immutable_pinning' flag, which will prevent
affinity to be changed, and then bind the topology to pinning only if
that one is set?

> >> Maybe, there is room for "fixing" this at this level, hooking up inside
> >> the scheduler code... but I'm shooting in the dark, without having check
> >> whether and how this could be really feasible, should I?
> >
> > Uuh, I don't think a change of the scheduler on behalf of Xen is really
> > appreciated. :-)
> >
I'm sure it would (have been! :-)) a true and giant nightmare!! :-D

> >> One thing I don't like about this approach is that it would potentially
> >> solve vNUMA and other scheduling anomalies, but...
> >>
> >>> cpuid instruction is available for user mode as well.
> >>>
> >> ...it would not do any good for other subsystems, and user level code
> >> and apps.
> >
> > Indeed. I think the optimal solution would be two-fold: give the
> > scheduler the information it is needing to react correctly via a
> > kernel patch not relying on cpuid values and fiddle with the cpuid
> > values from xen tools according to any needs of other subsystems and/or
> > user code (e.g. licensing).
> 
So, just to check if I'm understanding is correct: you'd like to add an
abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
code, to hide the direct interaction with CPUID.
Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
check with Xen/match vNUMA/whatever... Is this that you are saying?

If yes, I think I like it...

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/23/2015 03:25 AM, Jan Beulich wrote:
>>>> On 22.07.15 at 20:10, <boris.ostrovsky@oracle.com> wrote:
>> I don't think this is currently doable with what we have for CPUID
>> support in xl syntax. I am pretty sure we need to at least be able to
>> specify all leaf 4's indexes. And we can't.
>>
>> BTW, irrespective of this particular problem, adding support for indexed
>> CPUID leaves would be a good idea.
> So what is it that makes you think this doesn't work already?
> xc_cpuid_policy() intentionally has "input" as a pointer,
> allowing not just EAX to be specified. And libxl_cpuid.c also
> has
>
> static const char *input_names[2] = { "leaf", "subleaf" };
>

Right. I wasn't aware of this and until I looked at 
libxl_cpuid_parse_config_xend() I wouldn't have known what the syntax 
for specifying subleaves is because we don't describe it in xl man page 
(at least I can't find it).

In which case we may be able to adjust them and see what effect they have.


-boris

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1818 bytes --]

On Fri, 2015-07-24 at 17:14 +0200, Juergen Gross wrote:
> On 07/24/2015 04:44 PM, Dario Faggioli wrote:

> > Ok. And I already have a question (as I lost track of things a bit).
> > What you just said about ACPI tables is certainly true for baremetal and
> > HVM guests, but for PV? At the time I was looking into it, together with
> > Elena, there were Linux patches being produced for the PV case, which
> > makes sense.
> > However, ISTR that both Wei and Elena mentioned recently that those
> > patches have not been upstreamed in Linux yet... Is that the case? Maybe
> > not all, but at least some of them are there? Because if not, I'm not
> > sure I see how a PV guest would even see a vNUMA topology (which it
> > does).

> I checked "bottom up", so when I found the acpi scan stuff I stopped
> searching how the kernel obtains numa info. During my search I found no
> clue of an pv-numa stuff in the kernel. And a quick "grep -i numa" in
> arch/x86/xen and drivers/xen didn't reveal anything. Same for a complete
> kernel source search for "vnuma".
> 
BTW, I've also been grepping, and I'm not seeing XENMEM_get_vnumainfo
being called anywhere either... Well, no wonder, we're seeing vNUMA
setup issues! If I did check for this before, I wouldn't even have
tested PV-vNUMA in the first place!! :-O

I actually wonder how _some_ of the vNUMA info get through!?!? :-O

IAC, all we're saying remains true, of course, and the CPUID issue will
still be there, if we don't fix it, when such a patch will be accepted.

Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Konrad Rzeszutek Wilk]

On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote:
> On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote:
> > On 07/24/2015 05:14 PM, Juergen Gross wrote:
> > > On 07/24/2015 04:44 PM, Dario Faggioli wrote:
> 
> > >> In fact, I think that it is the topology, i.e., what comes from MSRs,
> > >> that needs to adapt, and follow vNUMA, as much as possible. Do we agree
> > >> on this?
> > >
> > > I think we have to be very careful here. I see two possible scenarios:
> > >
> > > 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will
> > >     try to schedule the vcpus according to their numa affinity. So they
> > >     can change pcpus at any time in case of very busy guests. I don't
> > >     think the linux kernel should treat the cpus differently in this
> > >     case as it will be in vane regarding the Xen scheduler's activity.
> > >     So we should use the "null" topology in this case.
> > 
> > Sorry, the topology should reflect the vcpu<->numa-node relations, of
> > course, but nothing else (so flat topolgy in each numa node).
> > 
> Yeah, I was replying to this point saying something like this right
> now... Luckily, I've seen this email! :-P
> 
> With this semantic, I fully agree with this.
> 
> > > 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen
> > >     scheduler can't move vcpus between pcpus, so the linux kernel should
> > >     see the real topology of the used pcpus in order to optimize for this
> > >     picture.
> > >
> >
> Mmm... I did think about this too, but I'm not sure. I see the value of
> this of course, and the reason why it makes sense. However, pinning can
> change on-line, via `xl vcpu-pin' and stuff. Also migration could make
> things less certain, I think. What happens if we build on top of the
> initial pinning, and then things change?
> 
> To be fair, there is stuff building on top of the initial pinning
> already, e.g., from which physical NUMA node we allocate the memory
> relies depends exactly on that. That being said, I'm not sure I'm
> comfortable with adding more of this...
> 
> Perhaps introduce an 'immutable_pinning' flag, which will prevent
> affinity to be changed, and then bind the topology to pinning only if
> that one is set?
> 
> > >> Maybe, there is room for "fixing" this at this level, hooking up inside
> > >> the scheduler code... but I'm shooting in the dark, without having check
> > >> whether and how this could be really feasible, should I?
> > >
> > > Uuh, I don't think a change of the scheduler on behalf of Xen is really
> > > appreciated. :-)
> > >
> I'm sure it would (have been! :-)) a true and giant nightmare!! :-D
> 
> > >> One thing I don't like about this approach is that it would potentially
> > >> solve vNUMA and other scheduling anomalies, but...
> > >>
> > >>> cpuid instruction is available for user mode as well.
> > >>>
> > >> ...it would not do any good for other subsystems, and user level code
> > >> and apps.
> > >
> > > Indeed. I think the optimal solution would be two-fold: give the
> > > scheduler the information it is needing to react correctly via a
> > > kernel patch not relying on cpuid values and fiddle with the cpuid
> > > values from xen tools according to any needs of other subsystems and/or
> > > user code (e.g. licensing).
> > 
> So, just to check if I'm understanding is correct: you'd like to add an
> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
> code, to hide the direct interaction with CPUID.
> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
> check with Xen/match vNUMA/whatever... Is this that you are saying?
> 
> If yes, I think I like it...

I don't think this is workable. For example there are applications
which use 'cpuid' and figure out the core/thread and use it for its own
scheduling purposes.

> 
> Regards,
> Dario
> -- 
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)



> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/24/2015 05:58 PM, Dario Faggioli wrote:
> On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote:
>> On 07/24/2015 05:14 PM, Juergen Gross wrote:
>>> On 07/24/2015 04:44 PM, Dario Faggioli wrote:
>
>>>> In fact, I think that it is the topology, i.e., what comes from MSRs,
>>>> that needs to adapt, and follow vNUMA, as much as possible. Do we agree
>>>> on this?
>>>
>>> I think we have to be very careful here. I see two possible scenarios:
>>>
>>> 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will
>>>      try to schedule the vcpus according to their numa affinity. So they
>>>      can change pcpus at any time in case of very busy guests. I don't
>>>      think the linux kernel should treat the cpus differently in this
>>>      case as it will be in vane regarding the Xen scheduler's activity.
>>>      So we should use the "null" topology in this case.
>>
>> Sorry, the topology should reflect the vcpu<->numa-node relations, of
>> course, but nothing else (so flat topolgy in each numa node).
>>
> Yeah, I was replying to this point saying something like this right
> now... Luckily, I've seen this email! :-P
>
> With this semantic, I fully agree with this.
>
>>> 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen
>>>      scheduler can't move vcpus between pcpus, so the linux kernel should
>>>      see the real topology of the used pcpus in order to optimize for this
>>>      picture.
>>>
>>
> Mmm... I did think about this too, but I'm not sure. I see the value of
> this of course, and the reason why it makes sense. However, pinning can
> change on-line, via `xl vcpu-pin' and stuff. Also migration could make
> things less certain, I think. What happens if we build on top of the
> initial pinning, and then things change?

If we can fiddle with the masks on boot, we could do it in a running
system, too. Another advantage with not relying on cpuid. :-)

> To be fair, there is stuff building on top of the initial pinning
> already, e.g., from which physical NUMA node we allocate the memory
> relies depends exactly on that. That being said, I'm not sure I'm
> comfortable with adding more of this...
>
> Perhaps introduce an 'immutable_pinning' flag, which will prevent
> affinity to be changed, and then bind the topology to pinning only if
> that one is set?

Hmm, this would require disabling migration as well. Or enable it
with "--force" only.

>
>>>> Maybe, there is room for "fixing" this at this level, hooking up inside
>>>> the scheduler code... but I'm shooting in the dark, without having check
>>>> whether and how this could be really feasible, should I?
>>>
>>> Uuh, I don't think a change of the scheduler on behalf of Xen is really
>>> appreciated. :-)
>>>
> I'm sure it would (have been! :-)) a true and giant nightmare!! :-D
>
>>>> One thing I don't like about this approach is that it would potentially
>>>> solve vNUMA and other scheduling anomalies, but...
>>>>
>>>>> cpuid instruction is available for user mode as well.
>>>>>
>>>> ...it would not do any good for other subsystems, and user level code
>>>> and apps.
>>>
>>> Indeed. I think the optimal solution would be two-fold: give the
>>> scheduler the information it is needing to react correctly via a
>>> kernel patch not relying on cpuid values and fiddle with the cpuid
>>> values from xen tools according to any needs of other subsystems and/or
>>> user code (e.g. licensing).
>>
> So, just to check if I'm understanding is correct: you'd like to add an
> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
> code, to hide the direct interaction with CPUID.
> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
> check with Xen/match vNUMA/whatever... Is this that you are saying?

Sort of, yes.

I just wouldn't add it, as it is already existing (more or less). It
can deal right now with AMD and Intel, we would "just" have to add Xen.

>
> If yes, I think I like it...

I hope e.g. the KVM guys will like it, too. :-)


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1068 bytes --]

On Fri, 2015-07-24 at 12:09 -0400, Konrad Rzeszutek Wilk wrote:
> On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote:

> > So, just to check if I'm understanding is correct: you'd like to add an
> > abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
> > code, to hide the direct interaction with CPUID.
> > Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
> > check with Xen/match vNUMA/whatever... Is this that you are saying?
> > 
> > If yes, I think I like it...
> 
> I don't think this is workable. For example there are applications
> which use 'cpuid' and figure out the core/thread and use it for its own
> scheduling purposes.
> 
Naah! That pretty much only happens with databases, so, honestly, who
cares!! ;-P ;-P ;-P

Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/24/2015 06:09 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote:
>> On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote:
>>> On 07/24/2015 05:14 PM, Juergen Gross wrote:
>>>> On 07/24/2015 04:44 PM, Dario Faggioli wrote:
>>
>>>>> In fact, I think that it is the topology, i.e., what comes from MSRs,
>>>>> that needs to adapt, and follow vNUMA, as much as possible. Do we agree
>>>>> on this?
>>>>
>>>> I think we have to be very careful here. I see two possible scenarios:
>>>>
>>>> 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will
>>>>      try to schedule the vcpus according to their numa affinity. So they
>>>>      can change pcpus at any time in case of very busy guests. I don't
>>>>      think the linux kernel should treat the cpus differently in this
>>>>      case as it will be in vane regarding the Xen scheduler's activity.
>>>>      So we should use the "null" topology in this case.
>>>
>>> Sorry, the topology should reflect the vcpu<->numa-node relations, of
>>> course, but nothing else (so flat topolgy in each numa node).
>>>
>> Yeah, I was replying to this point saying something like this right
>> now... Luckily, I've seen this email! :-P
>>
>> With this semantic, I fully agree with this.
>>
>>>> 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen
>>>>      scheduler can't move vcpus between pcpus, so the linux kernel should
>>>>      see the real topology of the used pcpus in order to optimize for this
>>>>      picture.
>>>>
>>>
>> Mmm... I did think about this too, but I'm not sure. I see the value of
>> this of course, and the reason why it makes sense. However, pinning can
>> change on-line, via `xl vcpu-pin' and stuff. Also migration could make
>> things less certain, I think. What happens if we build on top of the
>> initial pinning, and then things change?
>>
>> To be fair, there is stuff building on top of the initial pinning
>> already, e.g., from which physical NUMA node we allocate the memory
>> relies depends exactly on that. That being said, I'm not sure I'm
>> comfortable with adding more of this...
>>
>> Perhaps introduce an 'immutable_pinning' flag, which will prevent
>> affinity to be changed, and then bind the topology to pinning only if
>> that one is set?
>>
>>>>> Maybe, there is room for "fixing" this at this level, hooking up inside
>>>>> the scheduler code... but I'm shooting in the dark, without having check
>>>>> whether and how this could be really feasible, should I?
>>>>
>>>> Uuh, I don't think a change of the scheduler on behalf of Xen is really
>>>> appreciated. :-)
>>>>
>> I'm sure it would (have been! :-)) a true and giant nightmare!! :-D
>>
>>>>> One thing I don't like about this approach is that it would potentially
>>>>> solve vNUMA and other scheduling anomalies, but...
>>>>>
>>>>>> cpuid instruction is available for user mode as well.
>>>>>>
>>>>> ...it would not do any good for other subsystems, and user level code
>>>>> and apps.
>>>>
>>>> Indeed. I think the optimal solution would be two-fold: give the
>>>> scheduler the information it is needing to react correctly via a
>>>> kernel patch not relying on cpuid values and fiddle with the cpuid
>>>> values from xen tools according to any needs of other subsystems and/or
>>>> user code (e.g. licensing).
>>>
>> So, just to check if I'm understanding is correct: you'd like to add an
>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
>> code, to hide the direct interaction with CPUID.
>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
>> check with Xen/match vNUMA/whatever... Is this that you are saying?
>>
>> If yes, I think I like it...
>
> I don't think this is workable. For example there are applications
> which use 'cpuid' and figure out the core/thread and use it for its own
> scheduling purposes.

Might be, yes.

The pure cpuid solution won't work for all license related issues.

Doing it via an abstraction layer in the kernel would work in more than
90% of all cases AND would still enable a user to fiddle cpuids
according to his needs (either topology or license).

I'd rather have an out-of-the-box kernel solution with special user
requirements handling than a complex solution making some user
requirements impossible to meet.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Konrad Rzeszutek Wilk]

On Fri, Jul 24, 2015 at 06:18:56PM +0200, Juergen Gross wrote:
> On 07/24/2015 06:09 PM, Konrad Rzeszutek Wilk wrote:
> >On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote:
> >>On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote:
> >>>On 07/24/2015 05:14 PM, Juergen Gross wrote:
> >>>>On 07/24/2015 04:44 PM, Dario Faggioli wrote:
> >>
> >>>>>In fact, I think that it is the topology, i.e., what comes from MSRs,
> >>>>>that needs to adapt, and follow vNUMA, as much as possible. Do we agree
> >>>>>on this?
> >>>>
> >>>>I think we have to be very careful here. I see two possible scenarios:
> >>>>
> >>>>1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will
> >>>>     try to schedule the vcpus according to their numa affinity. So they
> >>>>     can change pcpus at any time in case of very busy guests. I don't
> >>>>     think the linux kernel should treat the cpus differently in this
> >>>>     case as it will be in vane regarding the Xen scheduler's activity.
> >>>>     So we should use the "null" topology in this case.
> >>>
> >>>Sorry, the topology should reflect the vcpu<->numa-node relations, of
> >>>course, but nothing else (so flat topolgy in each numa node).
> >>>
> >>Yeah, I was replying to this point saying something like this right
> >>now... Luckily, I've seen this email! :-P
> >>
> >>With this semantic, I fully agree with this.
> >>
> >>>>2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen
> >>>>     scheduler can't move vcpus between pcpus, so the linux kernel should
> >>>>     see the real topology of the used pcpus in order to optimize for this
> >>>>     picture.
> >>>>
> >>>
> >>Mmm... I did think about this too, but I'm not sure. I see the value of
> >>this of course, and the reason why it makes sense. However, pinning can
> >>change on-line, via `xl vcpu-pin' and stuff. Also migration could make
> >>things less certain, I think. What happens if we build on top of the
> >>initial pinning, and then things change?
> >>
> >>To be fair, there is stuff building on top of the initial pinning
> >>already, e.g., from which physical NUMA node we allocate the memory
> >>relies depends exactly on that. That being said, I'm not sure I'm
> >>comfortable with adding more of this...
> >>
> >>Perhaps introduce an 'immutable_pinning' flag, which will prevent
> >>affinity to be changed, and then bind the topology to pinning only if
> >>that one is set?
> >>
> >>>>>Maybe, there is room for "fixing" this at this level, hooking up inside
> >>>>>the scheduler code... but I'm shooting in the dark, without having check
> >>>>>whether and how this could be really feasible, should I?
> >>>>
> >>>>Uuh, I don't think a change of the scheduler on behalf of Xen is really
> >>>>appreciated. :-)
> >>>>
> >>I'm sure it would (have been! :-)) a true and giant nightmare!! :-D
> >>
> >>>>>One thing I don't like about this approach is that it would potentially
> >>>>>solve vNUMA and other scheduling anomalies, but...
> >>>>>
> >>>>>>cpuid instruction is available for user mode as well.
> >>>>>>
> >>>>>...it would not do any good for other subsystems, and user level code
> >>>>>and apps.
> >>>>
> >>>>Indeed. I think the optimal solution would be two-fold: give the
> >>>>scheduler the information it is needing to react correctly via a
> >>>>kernel patch not relying on cpuid values and fiddle with the cpuid
> >>>>values from xen tools according to any needs of other subsystems and/or
> >>>>user code (e.g. licensing).
> >>>
> >>So, just to check if I'm understanding is correct: you'd like to add an
> >>abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
> >>code, to hide the direct interaction with CPUID.
> >>Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
> >>check with Xen/match vNUMA/whatever... Is this that you are saying?
> >>
> >>If yes, I think I like it...
> >
> >I don't think this is workable. For example there are applications
> >which use 'cpuid' and figure out the core/thread and use it for its own
> >scheduling purposes.
> 
> Might be, yes.

There are <cough>databases</cough> that do this.

> 
> The pure cpuid solution won't work for all license related issues.
> 
> Doing it via an abstraction layer in the kernel would work in more than
> 90% of all cases AND would still enable a user to fiddle cpuids
> according to his needs (either topology or license).
> 
> I'd rather have an out-of-the-box kernel solution with special user
> requirements handling than a complex solution making some user
> requirements impossible to meet.

I think there are two issues here - the solution you are trying to come
up with is for PV scenarios.

But the issue I described is for PVH and HVM - where the cpuid is intercepted
by the hypervisor and we can mangle it as we see fit.

I think so? Perhaps I misunderstood?

> 
> 
> Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/24/2015 06:29 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Jul 24, 2015 at 06:18:56PM +0200, Juergen Gross wrote:
>> On 07/24/2015 06:09 PM, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote:
>>>> On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote:
>>>>> On 07/24/2015 05:14 PM, Juergen Gross wrote:
>>>>>> On 07/24/2015 04:44 PM, Dario Faggioli wrote:
>>>>
>>>>>>> In fact, I think that it is the topology, i.e., what comes from MSRs,
>>>>>>> that needs to adapt, and follow vNUMA, as much as possible. Do we agree
>>>>>>> on this?
>>>>>>
>>>>>> I think we have to be very careful here. I see two possible scenarios:
>>>>>>
>>>>>> 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will
>>>>>>      try to schedule the vcpus according to their numa affinity. So they
>>>>>>      can change pcpus at any time in case of very busy guests. I don't
>>>>>>      think the linux kernel should treat the cpus differently in this
>>>>>>      case as it will be in vane regarding the Xen scheduler's activity.
>>>>>>      So we should use the "null" topology in this case.
>>>>>
>>>>> Sorry, the topology should reflect the vcpu<->numa-node relations, of
>>>>> course, but nothing else (so flat topolgy in each numa node).
>>>>>
>>>> Yeah, I was replying to this point saying something like this right
>>>> now... Luckily, I've seen this email! :-P
>>>>
>>>> With this semantic, I fully agree with this.
>>>>
>>>>>> 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen
>>>>>>      scheduler can't move vcpus between pcpus, so the linux kernel should
>>>>>>      see the real topology of the used pcpus in order to optimize for this
>>>>>>      picture.
>>>>>>
>>>>>
>>>> Mmm... I did think about this too, but I'm not sure. I see the value of
>>>> this of course, and the reason why it makes sense. However, pinning can
>>>> change on-line, via `xl vcpu-pin' and stuff. Also migration could make
>>>> things less certain, I think. What happens if we build on top of the
>>>> initial pinning, and then things change?
>>>>
>>>> To be fair, there is stuff building on top of the initial pinning
>>>> already, e.g., from which physical NUMA node we allocate the memory
>>>> relies depends exactly on that. That being said, I'm not sure I'm
>>>> comfortable with adding more of this...
>>>>
>>>> Perhaps introduce an 'immutable_pinning' flag, which will prevent
>>>> affinity to be changed, and then bind the topology to pinning only if
>>>> that one is set?
>>>>
>>>>>>> Maybe, there is room for "fixing" this at this level, hooking up inside
>>>>>>> the scheduler code... but I'm shooting in the dark, without having check
>>>>>>> whether and how this could be really feasible, should I?
>>>>>>
>>>>>> Uuh, I don't think a change of the scheduler on behalf of Xen is really
>>>>>> appreciated. :-)
>>>>>>
>>>> I'm sure it would (have been! :-)) a true and giant nightmare!! :-D
>>>>
>>>>>>> One thing I don't like about this approach is that it would potentially
>>>>>>> solve vNUMA and other scheduling anomalies, but...
>>>>>>>
>>>>>>>> cpuid instruction is available for user mode as well.
>>>>>>>>
>>>>>>> ...it would not do any good for other subsystems, and user level code
>>>>>>> and apps.
>>>>>>
>>>>>> Indeed. I think the optimal solution would be two-fold: give the
>>>>>> scheduler the information it is needing to react correctly via a
>>>>>> kernel patch not relying on cpuid values and fiddle with the cpuid
>>>>>> values from xen tools according to any needs of other subsystems and/or
>>>>>> user code (e.g. licensing).
>>>>>
>>>> So, just to check if I'm understanding is correct: you'd like to add an
>>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
>>>> code, to hide the direct interaction with CPUID.
>>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
>>>> check with Xen/match vNUMA/whatever... Is this that you are saying?
>>>>
>>>> If yes, I think I like it...
>>>
>>> I don't think this is workable. For example there are applications
>>> which use 'cpuid' and figure out the core/thread and use it for its own
>>> scheduling purposes.
>>
>> Might be, yes.
>
> There are <cough>databases</cough> that do this.

Really? How do you know? ;-)

>> The pure cpuid solution won't work for all license related issues.
>>
>> Doing it via an abstraction layer in the kernel would work in more than
>> 90% of all cases AND would still enable a user to fiddle cpuids
>> according to his needs (either topology or license).
>>
>> I'd rather have an out-of-the-box kernel solution with special user
>> requirements handling than a complex solution making some user
>> requirements impossible to meet.
>
> I think there are two issues here - the solution you are trying to come
> up with is for PV scenarios.

I'd say my solution is a paravirtualized one.

> But the issue I described is for PVH and HVM - where the cpuid is intercepted
> by the hypervisor and we can mangle it as we see fit.

I think even PVH and HVM are able to use paravirtualized interfaces.

I don't say mangling cpuids can't solve the scheduling problem. It
surely can. But it can't solve the scheduling problem without hiding
information like number of sockets or cores which might be required
for license purposes. If we don't care, fine.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/24/2015 12:10 PM, Juergen Gross wrote:
>
> If we can fiddle with the masks on boot, we could do it in a running
> system, too. Another advantage with not relying on cpuid. :-)


I am trying to catch up with this thread so I may have missed it, but I 
still don't understand why we don't want to rely on CPUID.

I think I saw Juergen said --- because it's HW-specific. But what's 
wrong with that? Hypervisor is building virtualized x86 (in this case) 
hardware and on such HW CPUID is the standard way of determining 
thread/core topology. Plus various ACPI tables and such.

And having a solution that doesn't address userspace (when there *is* a 
solution that can do it) doesn't seem like the best approach. Yes, it 
still won't cover userspace for PV guests but neither will the kernel patch.

As far as licensing is concerned --- are we sure this can't also be 
addressed by CPUID? BTW, if I was asked about who is most concerned 
about licensing my first answer would be --- databases. I.e. userspace.

(Also, I don't know whether this is still true but in the past APICID 
format was also used for topology discovery. Just to make things a bit 
more interesting ;-))


-boris

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/24/2015 12:39 PM, Juergen Gross wrote:
>
>
> I don't say mangling cpuids can't solve the scheduling problem. It
> surely can. But it can't solve the scheduling problem without hiding
> information like number of sockets or cores which might be required
> for license purposes. If we don't care, fine.
>

(this is somewhat repeating the email I just sent)

Why can's we construct socket/core info with CPUID (and *possibly* ACPI 
changes) that we present a reasonable (licensing-wise) picture?

Can you suggest an example where it will not work and then maybe we can 
figure something out?

-boris

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/24/2015 06:40 PM, Boris Ostrovsky wrote:
> On 07/24/2015 12:10 PM, Juergen Gross wrote:
>>
>> If we can fiddle with the masks on boot, we could do it in a running
>> system, too. Another advantage with not relying on cpuid. :-)
>
>
> I am trying to catch up with this thread so I may have missed it, but I
> still don't understand why we don't want to rely on CPUID.
>
> I think I saw Juergen said --- because it's HW-specific. But what's
> wrong with that? Hypervisor is building virtualized x86 (in this case)
> hardware and on such HW CPUID is the standard way of determining
> thread/core topology. Plus various ACPI tables and such.
>
> And having a solution that doesn't address userspace (when there *is* a
> solution that can do it) doesn't seem like the best approach. Yes, it
> still won't cover userspace for PV guests but neither will the kernel
> patch.
>
> As far as licensing is concerned --- are we sure this can't also be
> addressed by CPUID? BTW, if I was asked about who is most concerned
> about licensing my first answer would be --- databases. I.e. userspace.

The problem is to construct cpuids which will enable the linux
scheduler to work correct in spite of the hypervisor scheduler
moving vcpus between pcpus. The only way to do this is to emulate
single-threaded cores on the numa nodes without further grouping.
So either single-core sockets or one socket with many cores.

This might be problematic for licensing: the multi-socket solution
might require a higher license based on socket numbers. Or the
license is based on cores and will be more expensive as no hyperthreads
are detectable.

> (Also, I don't know whether this is still true but in the past APICID
> format was also used for topology discovery. Just to make things a bit
> more interesting ;-))

Another +1 for the pv-solution. :-)


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/24/2015 12:48 PM, Juergen Gross wrote:
> On 07/24/2015 06:40 PM, Boris Ostrovsky wrote:
>> On 07/24/2015 12:10 PM, Juergen Gross wrote:
>>>
>>> If we can fiddle with the masks on boot, we could do it in a running
>>> system, too. Another advantage with not relying on cpuid. :-)
>>
>>
>> I am trying to catch up with this thread so I may have missed it, but I
>> still don't understand why we don't want to rely on CPUID.
>>
>> I think I saw Juergen said --- because it's HW-specific. But what's
>> wrong with that? Hypervisor is building virtualized x86 (in this case)
>> hardware and on such HW CPUID is the standard way of determining
>> thread/core topology. Plus various ACPI tables and such.
>>
>> And having a solution that doesn't address userspace (when there *is* a
>> solution that can do it) doesn't seem like the best approach. Yes, it
>> still won't cover userspace for PV guests but neither will the kernel
>> patch.
>>
>> As far as licensing is concerned --- are we sure this can't also be
>> addressed by CPUID? BTW, if I was asked about who is most concerned
>> about licensing my first answer would be --- databases. I.e. userspace.
>
> The problem is to construct cpuids which will enable the linux
> scheduler to work correct in spite of the hypervisor scheduler
> moving vcpus between pcpus. The only way to do this is to emulate
> single-threaded cores on the numa nodes without further grouping.
> So either single-core sockets or one socket with many cores.

Right.

>
> This might be problematic for licensing: the multi-socket solution
> might require a higher license based on socket numbers. Or the
> license is based on cores and will be more expensive as no hyperthreads
> are detectable.

If we don't pin VCPUs approriately (which I think is the scenario that 
we are discussing) then CPUID can be used for find out package ID. And 
so any licensed SW will easily discover that it is running on different 
packages.


>
>> (Also, I don't know whether this is still true but in the past APICID
>> format was also used for topology discovery. Just to make things a bit
>> more interesting ;-))
>
> Another +1 for the pv-solution. :-)


Not necessarily. ACPI/CPUID changes can take care of that outside kernel.

BTW, this can be useful (in case you don't already have it):
https://software.intel.com/en-us/articles/intel-64-architecture-processor-topology-enumeration

There is a similar document for AMD but I don't remember where it lives.

-boris "the CPUID bigot"

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Elena Ufimtseva]

On Fri, Jul 24, 2015 at 04:44:36PM +0200, Dario Faggioli wrote:
> On Fri, 2015-07-24 at 12:28 +0200, Juergen Gross wrote:
> > On 07/23/2015 04:07 PM, Dario Faggioli wrote:
> 
> > > FWIW, I was thinking that the kernel were a better place, as Juergen is
> > > saying, while now I'm more convinced that tools would be more
> > > appropriate, as Boris is saying.
> > 
> > I've collected some information from the linux kernel sources as a base
> > for the discussion:
> > 
> That's great, thanks for this!
> 
> > The complete numa information (cpu->node and memory->node relations) is
> > taken from the acpi tables (srat, slit for "distances").
> > 
> Ok. And I already have a question (as I lost track of things a bit).
> What you just said about ACPI tables is certainly true for baremetal and
> HVM guests, but for PV? At the time I was looking into it, together with
> Elena, there were Linux patches being produced for the PV case, which
> makes sense.
> However, ISTR that both Wei and Elena mentioned recently that those
> patches have not been upstreamed in Linux yet... Is that the case? Maybe
> not all, but at least some of them are there? Because if not, I'm not
> sure I see how a PV guest would even see a vNUMA topology (which it
> does).

Yes, PV guest should not be able to get any vnuma info. I hope finish
with rebasnig patch and post it on Monday. Few things has changed, as well as
assignment of cpus to vnodes and the way it was done before (in latest
version on linux vnuma patch) does not work anymore.

> 
> Of course, I can go and check, but since you just looked, you may have
> it fresh and clear already. :-)
> 
> > The topology information is obtained via:
> > - intel:
> >    + cpuid leaf b with subleafs, leaf 4
> >    + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available
> > - amd:
> >    + cpuid leaf 8000001e, leaf 8000001d, leaf 4
> >    + msr c001100c
> >    + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available
> > 
> > The scheduler is aware of:
> > - smt siblings (from topology)
> > - last-level-cache siblings (from topology)
> > - node siblings (from numa information)
> >
> Right. So, this confirms what we were guessing: we need to "reconcile"
> these two sources of information (from the guest point of view).
> 
> Both the 'in kernel' and 'in toolstack' approach should have all the
> necessary information to make things match, I think. In fact, in
> toolstack, we know what the vNUMA topology is (we're parsing and
> actually putting it in place!). In kernel, we know it as we read it from
> tables or hypercalls (isn't that so, for PV guest?).
> 
> In fact, I think that it is the topology, i.e., what comes from MSRs,
> that needs to adapt, and follow vNUMA, as much as possible. Do we agree
> on this?
> 
> IMO, the thing boils down to these:
> 
>  1) from where (kernel vs. toolstack) is it the most easy and effective
>     to enact the CPUID fiddling? As in, can we do that in toolstack?
>     (Andrew was not so sure, and Boris found issues, although Jan seems
>     to think they're no show stopper.)
>     I'm quite certain that we can do that from inside the kernel,
>     although, how early would we need to be doing it? Do we have the
>     vNUMA info already?
> 
>  2) when tweaking the values of CPUID and other MSRs, are there other
>     vNUMA (and topology in general) constraints and requirements we
>     should take into account? For instance, do we want, for licensing
>     reasons, all (or most) of the vcpus to be siblings, rather than full
>     sockets? Etc.
>      2a) if yes, how and where are these constraints specified?
> 
> If looking at 1) only, it still looks to me that doing things within the
> kernel would be the way to go.
> 
> When looking at 2), OTOH, toolstacks variants start to be more
> appealing. Especially depending on our answer to 2a). In fact,
> in case we want to give the user a way to specify this
> siblings-vs-cores-vs-sockets information, it IMO would be good to deal
> with that in tools, rather than having to involve Xen or Linux!
> 
> > It will especially move tasks from one cpu to another first between smt
> > siblings, second between llc siblings, third between node siblings and
> > last all cpus.
> > 
> Yep, this part, I knew.
> 
> Maybe, there is room for "fixing" this at this level, hooking up inside
> the scheduler code... but I'm shooting in the dark, without having check
> whether and how this could be really feasible, should I?
> 
> One thing I don't like about this approach is that it would potentially
> solve vNUMA and other scheduling anomalies, but...
> 
> > cpuid instruction is available for user mode as well.
> > 
> ...it would not do any good for other subsystems, and user level code
> and apps.
> 
> Dario
> -- 
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/24/2015 06:40 PM, Boris Ostrovsky wrote:
> On 07/24/2015 12:10 PM, Juergen Gross wrote:
>>
>> If we can fiddle with the masks on boot, we could do it in a running
>> system, too. Another advantage with not relying on cpuid. :-)
>
>
> I am trying to catch up with this thread so I may have missed it, but I
> still don't understand why we don't want to rely on CPUID.
>
> I think I saw Juergen said --- because it's HW-specific. But what's
> wrong with that? Hypervisor is building virtualized x86 (in this case)
> hardware and on such HW CPUID is the standard way of determining
> thread/core topology. Plus various ACPI tables and such.

I didn't mean specific to x86, but even different between AMD and Intel.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/24/2015 06:44 PM, Boris Ostrovsky wrote:
> On 07/24/2015 12:39 PM, Juergen Gross wrote:
>>
>>
>> I don't say mangling cpuids can't solve the scheduling problem. It
>> surely can. But it can't solve the scheduling problem without hiding
>> information like number of sockets or cores which might be required
>> for license purposes. If we don't care, fine.
>>
>
> (this is somewhat repeating the email I just sent)
>
> Why can's we construct socket/core info with CPUID (and *possibly* ACPI
> changes) that we present a reasonable (licensing-wise) picture?
>
> Can you suggest an example where it will not work and then maybe we can
> figure something out?

Let's assume a software with license based on core count. You have a
system with a 2 8 core processors and hyperthreads enabled, summing up
to 32 logical processors. Your license is valid for up to 16 cores, so
running the software on bare metal on your system is fine.

Now you are running the software inside a virtual machine with 24 vcpus
in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each
processor). As we have to hide hyperthreading in order to not to have
to pin each vcpu to just a single logical processor, the topology
resulting from this picture will have to present 24 cores. The license
will not cover this hardware.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[George Dunlap]

On Fri, Jul 24, 2015 at 5:09 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote:
>> On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote:
>> > On 07/24/2015 05:14 PM, Juergen Gross wrote:
>> > > On 07/24/2015 04:44 PM, Dario Faggioli wrote:
>>
>> > >> In fact, I think that it is the topology, i.e., what comes from MSRs,
>> > >> that needs to adapt, and follow vNUMA, as much as possible. Do we agree
>> > >> on this?
>> > >
>> > > I think we have to be very careful here. I see two possible scenarios:
>> > >
>> > > 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will
>> > >     try to schedule the vcpus according to their numa affinity. So they
>> > >     can change pcpus at any time in case of very busy guests. I don't
>> > >     think the linux kernel should treat the cpus differently in this
>> > >     case as it will be in vane regarding the Xen scheduler's activity.
>> > >     So we should use the "null" topology in this case.
>> >
>> > Sorry, the topology should reflect the vcpu<->numa-node relations, of
>> > course, but nothing else (so flat topolgy in each numa node).
>> >
>> Yeah, I was replying to this point saying something like this right
>> now... Luckily, I've seen this email! :-P
>>
>> With this semantic, I fully agree with this.
>>
>> > > 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen
>> > >     scheduler can't move vcpus between pcpus, so the linux kernel should
>> > >     see the real topology of the used pcpus in order to optimize for this
>> > >     picture.
>> > >
>> >
>> Mmm... I did think about this too, but I'm not sure. I see the value of
>> this of course, and the reason why it makes sense. However, pinning can
>> change on-line, via `xl vcpu-pin' and stuff. Also migration could make
>> things less certain, I think. What happens if we build on top of the
>> initial pinning, and then things change?
>>
>> To be fair, there is stuff building on top of the initial pinning
>> already, e.g., from which physical NUMA node we allocate the memory
>> relies depends exactly on that. That being said, I'm not sure I'm
>> comfortable with adding more of this...
>>
>> Perhaps introduce an 'immutable_pinning' flag, which will prevent
>> affinity to be changed, and then bind the topology to pinning only if
>> that one is set?
>>
>> > >> Maybe, there is room for "fixing" this at this level, hooking up inside
>> > >> the scheduler code... but I'm shooting in the dark, without having check
>> > >> whether and how this could be really feasible, should I?
>> > >
>> > > Uuh, I don't think a change of the scheduler on behalf of Xen is really
>> > > appreciated. :-)
>> > >
>> I'm sure it would (have been! :-)) a true and giant nightmare!! :-D
>>
>> > >> One thing I don't like about this approach is that it would potentially
>> > >> solve vNUMA and other scheduling anomalies, but...
>> > >>
>> > >>> cpuid instruction is available for user mode as well.
>> > >>>
>> > >> ...it would not do any good for other subsystems, and user level code
>> > >> and apps.
>> > >
>> > > Indeed. I think the optimal solution would be two-fold: give the
>> > > scheduler the information it is needing to react correctly via a
>> > > kernel patch not relying on cpuid values and fiddle with the cpuid
>> > > values from xen tools according to any needs of other subsystems and/or
>> > > user code (e.g. licensing).
>> >
>> So, just to check if I'm understanding is correct: you'd like to add an
>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
>> code, to hide the direct interaction with CPUID.
>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
>> check with Xen/match vNUMA/whatever... Is this that you are saying?
>>
>> If yes, I think I like it...
>
> I don't think this is workable. For example there are applications
> which use 'cpuid' and figure out the core/thread and use it for its own
> scheduling purposes.

Can you expand a little on this?  I'm having trouble figuring out
exactly what user-space applications are reading and how they're using
it -- and, how they work currently in virtual environments, given that
they (typically) will be moved between physical processors even if
they stay on the same virtual processor.

 -George

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[George Dunlap]

On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote:
> On 07/24/2015 06:44 PM, Boris Ostrovsky wrote:
>>
>> On 07/24/2015 12:39 PM, Juergen Gross wrote:
>>>
>>>
>>>
>>> I don't say mangling cpuids can't solve the scheduling problem. It
>>> surely can. But it can't solve the scheduling problem without hiding
>>> information like number of sockets or cores which might be required
>>> for license purposes. If we don't care, fine.
>>>
>>
>> (this is somewhat repeating the email I just sent)
>>
>> Why can's we construct socket/core info with CPUID (and *possibly* ACPI
>> changes) that we present a reasonable (licensing-wise) picture?
>>
>> Can you suggest an example where it will not work and then maybe we can
>> figure something out?
>
>
> Let's assume a software with license based on core count. You have a
> system with a 2 8 core processors and hyperthreads enabled, summing up
> to 32 logical processors. Your license is valid for up to 16 cores, so
> running the software on bare metal on your system is fine.
>
> Now you are running the software inside a virtual machine with 24 vcpus
> in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each
> processor). As we have to hide hyperthreading in order to not to have
> to pin each vcpu to just a single logical processor, the topology
> resulting from this picture will have to present 24 cores. The license
> will not cover this hardware.

But how does doing a PV topology help this situation?  Because we're
telling one thing to the OS (via our PV interface) and another thing
to applications (via direct CPUID access)?

 -George

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Andrew Cooper]

On 27/07/15 11:41, George Dunlap wrote:
> On Fri, Jul 24, 2015 at 5:09 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
>> On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote:
>>> On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote:
>>>> On 07/24/2015 05:14 PM, Juergen Gross wrote:
>>>>> On 07/24/2015 04:44 PM, Dario Faggioli wrote:
>>>>>> In fact, I think that it is the topology, i.e., what comes from MSRs,
>>>>>> that needs to adapt, and follow vNUMA, as much as possible. Do we agree
>>>>>> on this?
>>>>> I think we have to be very careful here. I see two possible scenarios:
>>>>>
>>>>> 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will
>>>>>     try to schedule the vcpus according to their numa affinity. So they
>>>>>     can change pcpus at any time in case of very busy guests. I don't
>>>>>     think the linux kernel should treat the cpus differently in this
>>>>>     case as it will be in vane regarding the Xen scheduler's activity.
>>>>>     So we should use the "null" topology in this case.
>>>> Sorry, the topology should reflect the vcpu<->numa-node relations, of
>>>> course, but nothing else (so flat topolgy in each numa node).
>>>>
>>> Yeah, I was replying to this point saying something like this right
>>> now... Luckily, I've seen this email! :-P
>>>
>>> With this semantic, I fully agree with this.
>>>
>>>>> 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen
>>>>>     scheduler can't move vcpus between pcpus, so the linux kernel should
>>>>>     see the real topology of the used pcpus in order to optimize for this
>>>>>     picture.
>>>>>
>>> Mmm... I did think about this too, but I'm not sure. I see the value of
>>> this of course, and the reason why it makes sense. However, pinning can
>>> change on-line, via `xl vcpu-pin' and stuff. Also migration could make
>>> things less certain, I think. What happens if we build on top of the
>>> initial pinning, and then things change?
>>>
>>> To be fair, there is stuff building on top of the initial pinning
>>> already, e.g., from which physical NUMA node we allocate the memory
>>> relies depends exactly on that. That being said, I'm not sure I'm
>>> comfortable with adding more of this...
>>>
>>> Perhaps introduce an 'immutable_pinning' flag, which will prevent
>>> affinity to be changed, and then bind the topology to pinning only if
>>> that one is set?
>>>
>>>>>> Maybe, there is room for "fixing" this at this level, hooking up inside
>>>>>> the scheduler code... but I'm shooting in the dark, without having check
>>>>>> whether and how this could be really feasible, should I?
>>>>> Uuh, I don't think a change of the scheduler on behalf of Xen is really
>>>>> appreciated. :-)
>>>>>
>>> I'm sure it would (have been! :-)) a true and giant nightmare!! :-D
>>>
>>>>>> One thing I don't like about this approach is that it would potentially
>>>>>> solve vNUMA and other scheduling anomalies, but...
>>>>>>
>>>>>>> cpuid instruction is available for user mode as well.
>>>>>>>
>>>>>> ...it would not do any good for other subsystems, and user level code
>>>>>> and apps.
>>>>> Indeed. I think the optimal solution would be two-fold: give the
>>>>> scheduler the information it is needing to react correctly via a
>>>>> kernel patch not relying on cpuid values and fiddle with the cpuid
>>>>> values from xen tools according to any needs of other subsystems and/or
>>>>> user code (e.g. licensing).
>>> So, just to check if I'm understanding is correct: you'd like to add an
>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
>>> code, to hide the direct interaction with CPUID.
>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
>>> check with Xen/match vNUMA/whatever... Is this that you are saying?
>>>
>>> If yes, I think I like it...
>> I don't think this is workable. For example there are applications
>> which use 'cpuid' and figure out the core/thread and use it for its own
>> scheduling purposes.
> Can you expand a little on this?  I'm having trouble figuring out
> exactly what user-space applications are reading and how they're using
> it -- and, how they work currently in virtual environments, given that
> they (typically) will be moved between physical processors even if
> they stay on the same virtual processor.

There are many examples of userspace application using cpuid to modify
themselves.  Any serious application with processor optimisations will
use the cpuid feature bits to choose the most efficient algorithm.

hwloc is an perfect example which gathers all of the topology
information out of cpuid to work out how to most efficiently
pin/schedule tasks.

As to how the work, that is very much an open question.  "function"
might be a better term.

~Andrew

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Andrew Cooper]

On 27/07/15 11:43, George Dunlap wrote:
> On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote:
>> On 07/24/2015 06:44 PM, Boris Ostrovsky wrote:
>>> On 07/24/2015 12:39 PM, Juergen Gross wrote:
>>>>
>>>>
>>>> I don't say mangling cpuids can't solve the scheduling problem. It
>>>> surely can. But it can't solve the scheduling problem without hiding
>>>> information like number of sockets or cores which might be required
>>>> for license purposes. If we don't care, fine.
>>>>
>>> (this is somewhat repeating the email I just sent)
>>>
>>> Why can's we construct socket/core info with CPUID (and *possibly* ACPI
>>> changes) that we present a reasonable (licensing-wise) picture?
>>>
>>> Can you suggest an example where it will not work and then maybe we can
>>> figure something out?
>>
>> Let's assume a software with license based on core count. You have a
>> system with a 2 8 core processors and hyperthreads enabled, summing up
>> to 32 logical processors. Your license is valid for up to 16 cores, so
>> running the software on bare metal on your system is fine.
>>
>> Now you are running the software inside a virtual machine with 24 vcpus
>> in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each
>> processor). As we have to hide hyperthreading in order to not to have
>> to pin each vcpu to just a single logical processor, the topology
>> resulting from this picture will have to present 24 cores. The license
>> will not cover this hardware.
> But how does doing a PV topology help this situation?  Because we're
> telling one thing to the OS (via our PV interface) and another thing
> to applications (via direct CPUID access)?

I expressed exactly these concerns right back at the start of the vnuma
work.

The OS and its userspace can and will use cpuid.  Most examples will
only use cpuid.  The only thing worse that providing no NUMA information
at all is providing conflicting information between cpuid and vnuma.

IMO, HVM guests should get all their NUMA information from the same
sources as native hardware would provide.  PV guests are admittedly
harder as in generally we cannot hide the real topology information in
cpuid.

~Andrew

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/27/2015 12:43 PM, George Dunlap wrote:
> On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote:
>> On 07/24/2015 06:44 PM, Boris Ostrovsky wrote:
>>>
>>> On 07/24/2015 12:39 PM, Juergen Gross wrote:
>>>>
>>>>
>>>>
>>>> I don't say mangling cpuids can't solve the scheduling problem. It
>>>> surely can. But it can't solve the scheduling problem without hiding
>>>> information like number of sockets or cores which might be required
>>>> for license purposes. If we don't care, fine.
>>>>
>>>
>>> (this is somewhat repeating the email I just sent)
>>>
>>> Why can's we construct socket/core info with CPUID (and *possibly* ACPI
>>> changes) that we present a reasonable (licensing-wise) picture?
>>>
>>> Can you suggest an example where it will not work and then maybe we can
>>> figure something out?
>>
>>
>> Let's assume a software with license based on core count. You have a
>> system with a 2 8 core processors and hyperthreads enabled, summing up
>> to 32 logical processors. Your license is valid for up to 16 cores, so
>> running the software on bare metal on your system is fine.
>>
>> Now you are running the software inside a virtual machine with 24 vcpus
>> in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each
>> processor). As we have to hide hyperthreading in order to not to have
>> to pin each vcpu to just a single logical processor, the topology
>> resulting from this picture will have to present 24 cores. The license
>> will not cover this hardware.
>
> But how does doing a PV topology help this situation?  Because we're
> telling one thing to the OS (via our PV interface) and another thing
> to applications (via direct CPUID access)?

Exactly.

In my example it would even work to not modify the cpuid information at
all. The kernel wouldn't try to be extremely clever regarding scheduling
and the user land would see the cpuid information from the real hardware
(only the 12 cores it is running on, of course).


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[George Dunlap]

On 07/27/2015 11:54 AM, Juergen Gross wrote:
> On 07/27/2015 12:43 PM, George Dunlap wrote:
>> On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote:
>>> On 07/24/2015 06:44 PM, Boris Ostrovsky wrote:
>>>>
>>>> On 07/24/2015 12:39 PM, Juergen Gross wrote:
>>>>>
>>>>>
>>>>>
>>>>> I don't say mangling cpuids can't solve the scheduling problem. It
>>>>> surely can. But it can't solve the scheduling problem without hiding
>>>>> information like number of sockets or cores which might be required
>>>>> for license purposes. If we don't care, fine.
>>>>>
>>>>
>>>> (this is somewhat repeating the email I just sent)
>>>>
>>>> Why can's we construct socket/core info with CPUID (and *possibly* ACPI
>>>> changes) that we present a reasonable (licensing-wise) picture?
>>>>
>>>> Can you suggest an example where it will not work and then maybe we can
>>>> figure something out?
>>>
>>>
>>> Let's assume a software with license based on core count. You have a
>>> system with a 2 8 core processors and hyperthreads enabled, summing up
>>> to 32 logical processors. Your license is valid for up to 16 cores, so
>>> running the software on bare metal on your system is fine.
>>>
>>> Now you are running the software inside a virtual machine with 24 vcpus
>>> in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each
>>> processor). As we have to hide hyperthreading in order to not to have
>>> to pin each vcpu to just a single logical processor, the topology
>>> resulting from this picture will have to present 24 cores. The license
>>> will not cover this hardware.
>>
>> But how does doing a PV topology help this situation?  Because we're
>> telling one thing to the OS (via our PV interface) and another thing
>> to applications (via direct CPUID access)?
> 
> Exactly.
> 
> In my example it would even work to not modify the cpuid information at
> all. The kernel wouldn't try to be extremely clever regarding scheduling
> and the user land would see the cpuid information from the real hardware
> (only the 12 cores it is running on, of course).

Right; so it seems

1. Userspace applications are in the habit of reading CPUID to determine
the topology of the system they're running on

2. Many use the topology information to help themselves make better
scheduling decisions.  Because a vcpu is not typically pinned to a
specific pcpu, we may need to lie here slightly (e.g., not mention
threads) to get the optimal behavior overall.

3. Others use the topology information to implement licensing
restrictions.  Because threads are treated differently to cores, we want
to tell the truth here (i.e., make sure we mention that some of these
are threads) to get the optimal behavior overall.

Numbers #2 and #3 lead to contradictory courses of action; we cannot
optimize for both at the same time.

I think at some level we need to just try to accommodate both -- if the
user doesn't have licensing issues, or prefers performance over
licensing, then present a unified topology in PVH / HVM using CPUID,
ACPI, &c.  I think this should be the default.

If the user has licensing issues, and doesn't mind having wonky or
unreliable topology to its guests, then let the raw CPUID through.  But
it would, in this case, be good to try to give the guest OS scheduler a
hint that it shouldn't really bother trying to read the topology or do
placement as a result, as any decisions will be unreliable.

Or alternately, if the user wants to give up on the "consolidation"
aspect of virtualization, they can pin vcpus to pcpus and then pass in
the actual host topology (hyperthreads and all).

 -George

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/27/2015 12:54 PM, Andrew Cooper wrote:
> On 27/07/15 11:43, George Dunlap wrote:
>> On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote:
>>> On 07/24/2015 06:44 PM, Boris Ostrovsky wrote:
>>>> On 07/24/2015 12:39 PM, Juergen Gross wrote:
>>>>>
>>>>>
>>>>> I don't say mangling cpuids can't solve the scheduling problem. It
>>>>> surely can. But it can't solve the scheduling problem without hiding
>>>>> information like number of sockets or cores which might be required
>>>>> for license purposes. If we don't care, fine.
>>>>>
>>>> (this is somewhat repeating the email I just sent)
>>>>
>>>> Why can's we construct socket/core info with CPUID (and *possibly* ACPI
>>>> changes) that we present a reasonable (licensing-wise) picture?
>>>>
>>>> Can you suggest an example where it will not work and then maybe we can
>>>> figure something out?
>>>
>>> Let's assume a software with license based on core count. You have a
>>> system with a 2 8 core processors and hyperthreads enabled, summing up
>>> to 32 logical processors. Your license is valid for up to 16 cores, so
>>> running the software on bare metal on your system is fine.
>>>
>>> Now you are running the software inside a virtual machine with 24 vcpus
>>> in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each
>>> processor). As we have to hide hyperthreading in order to not to have
>>> to pin each vcpu to just a single logical processor, the topology
>>> resulting from this picture will have to present 24 cores. The license
>>> will not cover this hardware.
>> But how does doing a PV topology help this situation?  Because we're
>> telling one thing to the OS (via our PV interface) and another thing
>> to applications (via direct CPUID access)?
>
> I expressed exactly these concerns right back at the start of the vnuma
> work.
>
> The OS and its userspace can and will use cpuid.  Most examples will
> only use cpuid.  The only thing worse that providing no NUMA information
> at all is providing conflicting information between cpuid and vnuma.
>
> IMO, HVM guests should get all their NUMA information from the same
> sources as native hardware would provide.  PV guests are admittedly
> harder as in generally we cannot hide the real topology information in
> cpuid.

Are you aware the same is true currently even without vNUMA?

The linux kernel (and other OS's as well) will make scheduling decisions
based on cpuid data obtained during boot. The information will be
correct only by chance and the real relation between vcpus and pcpus is
changing all the time.

So without adapting the kernel to that scenario it won't run optimal.
You can either change the data to let the kernel make some sane
decisions (cpuid mangling) or you can adapt the kernel somehow, e.g.
by modifying the kernel internal tables used for making scheduling
decisions (my proposal).

Something should be done regardless of the vNUMA support.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/27/2015 01:11 PM, George Dunlap wrote:
> On 07/27/2015 11:54 AM, Juergen Gross wrote:
>> On 07/27/2015 12:43 PM, George Dunlap wrote:
>>> On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote:
>>>> On 07/24/2015 06:44 PM, Boris Ostrovsky wrote:
>>>>>
>>>>> On 07/24/2015 12:39 PM, Juergen Gross wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> I don't say mangling cpuids can't solve the scheduling problem. It
>>>>>> surely can. But it can't solve the scheduling problem without hiding
>>>>>> information like number of sockets or cores which might be required
>>>>>> for license purposes. If we don't care, fine.
>>>>>>
>>>>>
>>>>> (this is somewhat repeating the email I just sent)
>>>>>
>>>>> Why can's we construct socket/core info with CPUID (and *possibly* ACPI
>>>>> changes) that we present a reasonable (licensing-wise) picture?
>>>>>
>>>>> Can you suggest an example where it will not work and then maybe we can
>>>>> figure something out?
>>>>
>>>>
>>>> Let's assume a software with license based on core count. You have a
>>>> system with a 2 8 core processors and hyperthreads enabled, summing up
>>>> to 32 logical processors. Your license is valid for up to 16 cores, so
>>>> running the software on bare metal on your system is fine.
>>>>
>>>> Now you are running the software inside a virtual machine with 24 vcpus
>>>> in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each
>>>> processor). As we have to hide hyperthreading in order to not to have
>>>> to pin each vcpu to just a single logical processor, the topology
>>>> resulting from this picture will have to present 24 cores. The license
>>>> will not cover this hardware.
>>>
>>> But how does doing a PV topology help this situation?  Because we're
>>> telling one thing to the OS (via our PV interface) and another thing
>>> to applications (via direct CPUID access)?
>>
>> Exactly.
>>
>> In my example it would even work to not modify the cpuid information at
>> all. The kernel wouldn't try to be extremely clever regarding scheduling
>> and the user land would see the cpuid information from the real hardware
>> (only the 12 cores it is running on, of course).
>
> Right; so it seems
>
> 1. Userspace applications are in the habit of reading CPUID to determine
> the topology of the system they're running on
>
> 2. Many use the topology information to help themselves make better
> scheduling decisions.  Because a vcpu is not typically pinned to a
> specific pcpu, we may need to lie here slightly (e.g., not mention
> threads) to get the optimal behavior overall.
>
> 3. Others use the topology information to implement licensing
> restrictions.  Because threads are treated differently to cores, we want
> to tell the truth here (i.e., make sure we mention that some of these
> are threads) to get the optimal behavior overall.
>
> Numbers #2 and #3 lead to contradictory courses of action; we cannot
> optimize for both at the same time.
>
> I think at some level we need to just try to accommodate both -- if the
> user doesn't have licensing issues, or prefers performance over
> licensing, then present a unified topology in PVH / HVM using CPUID,
> ACPI, &c.  I think this should be the default.
>
> If the user has licensing issues, and doesn't mind having wonky or
> unreliable topology to its guests, then let the raw CPUID through.  But
> it would, in this case, be good to try to give the guest OS scheduler a
> hint that it shouldn't really bother trying to read the topology or do
> placement as a result, as any decisions will be unreliable.
>
> Or alternately, if the user wants to give up on the "consolidation"
> aspect of virtualization, they can pin vcpus to pcpus and then pass in
> the actual host topology (hyperthreads and all).

There would be another solution, of course:

Support hyperthreads in the Xen scheduler via gang scheduling. While
this is not a simple solution, it is a fair one. Hyperthreads on one
core can influence each other rather much. With both threads always
running vcpus of the same guest the penalty/advantage would stay in the
same domain. The guest could make really sensible scheduling decisions
and the licensing would still work as desired.

Just an idea, but maybe worth to explore further instead of tweaking
more and more bits to make the virtual system somehow act sane.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Tim Deegan]

At 14:01 +0200 on 27 Jul (1438005701), Juergen Gross wrote:
> There would be another solution, of course:
> 
> Support hyperthreads in the Xen scheduler via gang scheduling. While
> this is not a simple solution, it is a fair one. Hyperthreads on one
> core can influence each other rather much. With both threads always
> running vcpus of the same guest the penalty/advantage would stay in the
> same domain. The guest could make really sensible scheduling decisions
> and the licensing would still work as desired.

This might also be interesting for people worried about side-channels
(e.g. http://sophia.re/RECON ).

Tim.

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1488 bytes --]

On Mon, 2015-07-27 at 11:49 +0100, Andrew Cooper wrote:
> On 27/07/15 11:41, George Dunlap wrote:

> > Can you expand a little on this?  I'm having trouble figuring out
> > exactly what user-space applications are reading and how they're using
> > it -- and, how they work currently in virtual environments, given that
> > they (typically) will be moved between physical processors even if
> > they stay on the same virtual processor.
> 
> There are many examples of userspace application using cpuid to modify
> themselves.  Any serious application with processor optimisations will
> use the cpuid feature bits to choose the most efficient algorithm.
> 
> hwloc is an perfect example which gathers all of the topology
> information out of cpuid to work out how to most efficiently
> pin/schedule tasks.
> 
And all of this is broken, right now, isn't it?

Where, saying this, I'm aiming at stressing the point (as the thread is
starting to become a bit hard to follow) that we must come up with a
decent solution, working reasonably well in a bunch of (conflicting!
*sigh*) scenarios, *not* to implement something new without breaking
what we have, as what we have is broken already!

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2166 bytes --]

On Mon, 2015-07-27 at 14:01 +0200, Juergen Gross wrote:
> On 07/27/2015 01:11 PM, George Dunlap wrote:

> > Or alternately, if the user wants to give up on the "consolidation"
> > aspect of virtualization, they can pin vcpus to pcpus and then pass in
> > the actual host topology (hyperthreads and all).
> 
> There would be another solution, of course:
> 
> Support hyperthreads in the Xen scheduler via gang scheduling. While
> this is not a simple solution, it is a fair one. Hyperthreads on one
> core can influence each other rather much. With both threads always
> running vcpus of the same guest the penalty/advantage would stay in the
> same domain. The guest could make really sensible scheduling decisions
> and the licensing would still work as desired.
> 
This is interesting indeed, but I much rather see it as something
orthogonal, which may indeed bring benefits in some of the scenarios
described here, but should not be considered *the* solution.

Implementing, enabling and asking users to use something like this will
impact the system behavior and performance, in ways that may not be
desirable for all use cases.

So, while I do think that this may be something nice to have and offer,
trying to use it for solving the problem we're debating here would make
things even more complex to configure.

Also, this would take care of HT related issues, but what about cores
(as in 'should vcpus be cores of sockets or full sockets') and !HT boxes
(like AMD)?

Not to mention, as you say yourself, that it's not easy to implement.

> Just an idea, but maybe worth to explore further instead of tweaking
> more and more bits to make the virtual system somehow act sane.
> 
Sure, and it it's interesting indeed, for a bunch or reasons and
purposes (as Tim is also noting). Not so much --or at least not
necessarily-- for this one, IMO.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3869 bytes --]

On Fri, 2015-07-24 at 13:11 -0400, Boris Ostrovsky wrote:
> On 07/24/2015 12:48 PM, Juergen Gross wrote:
> > On 07/24/2015 06:40 PM, Boris Ostrovsky wrote:
> >> On 07/24/2015 12:10 PM, Juergen Gross wrote:
> >>>
> >>> If we can fiddle with the masks on boot, we could do it in a running
> >>> system, too. Another advantage with not relying on cpuid. :-)
> >>
> >>
> >> I am trying to catch up with this thread so I may have missed it, but I
> >> still don't understand why we don't want to rely on CPUID.
> >>
> >> I think I saw Juergen said --- because it's HW-specific. But what's
> >> wrong with that? Hypervisor is building virtualized x86 (in this case)
> >> hardware and on such HW CPUID is the standard way of determining
> >> thread/core topology. Plus various ACPI tables and such.
> >>
> >> And having a solution that doesn't address userspace (when there *is* a
> >> solution that can do it) doesn't seem like the best approach. Yes, it
> >> still won't cover userspace for PV guests but neither will the kernel
> >> patch.
> >>
> >> As far as licensing is concerned --- are we sure this can't also be
> >> addressed by CPUID? BTW, if I was asked about who is most concerned
> >> about licensing my first answer would be --- databases. I.e. userspace.
> >
> > The problem is to construct cpuids which will enable the linux
> > scheduler to work correct in spite of the hypervisor scheduler
> > moving vcpus between pcpus. The only way to do this is to emulate
> > single-threaded cores on the numa nodes without further grouping.
> > So either single-core sockets or one socket with many cores.
> 
> Right.
> 
So you see it now? If adhering to some guest virtual topology (for
whatever reason, e.g., performance) and licensing disagree, using CPUID
for both is always going to fail!

> > This might be problematic for licensing: the multi-socket solution
> > might require a higher license based on socket numbers. Or the
> > license is based on cores and will be more expensive as no hyperthreads
> > are detectable.
> 
> If we don't pin VCPUs approriately (which I think is the scenario that 
> we are discussing) then CPUID can be used for find out package ID. And 
> so any licensed SW will easily discover that it is running on different 
> packages.
> 
That's why Juergen is suggesting to keep the things separate,
effectively decoupling them, AFAICU his proposal.

Note that I'm still a bit puzzled by the idea of presenting different
information to the guest OS and to the guest userspace, but that has
upsides, and this decoupling is one.

In fact, in the case you're describing, i.e., not pinned vcpus, etc:
 - the Linux kernel is not relying on CPUID when building scheduling  
   domains, and everything so, at least all the user space applications
   that does not call CPUID and/or rely on that for
   scheduling/performance matters *will* *be* *fine*
 - you can pass down, via tools, a mangled CPUID, e.g., making it best 
   fit your licensing needs.

Problems arises in case you have both the following kind of applications
(or the same application doing both the following operations):
 a) applications that poke at CPUID for licensing purposes
 b) applications that poke at CPUID for placement/performance purposes

In this case, it's well possible that mangling CPUID for making a)
happy, will make b) unhappy, and vice-versa. And the fact that the
kernel does not rely on CPUID any longer may not help, as the
application is kind of bypassing it... Although, it won't harm either...

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3372 bytes --]

On Mon, 2015-07-27 at 12:11 +0100, George Dunlap wrote:

> 1. Userspace applications are in the habit of reading CPUID to determine
> the topology of the system they're running on
> 
I'd add this item here:

1b. Linux kernel uses CPUID to configure some bits of its scheduler. The
    result of that, will affect *all* the in-guest application that does
    not use CPUID for scheduling/performance purposes

Yes, I saw you mention this afterwards, but I think it really should be
here, in the analysis part.

> 2. Many use the topology information to help themselves make better
> scheduling decisions.  Because a vcpu is not typically pinned to a
> specific pcpu, we may need to lie here slightly (e.g., not mention
> threads) to get the optimal behavior overall.
> 
> 3. Others use the topology information to implement licensing
> restrictions.  Because threads are treated differently to cores, we want
> to tell the truth here (i.e., make sure we mention that some of these
> are threads) to get the optimal behavior overall.
> 
Define truth. I mean, is 'telling the truth' always good for this case?
E.g., in a 4 vcpus guest, on a 4 socket box, without any pinning, it may
be that when app X samples CPUID to check the license, each vcpu is
running on a different socket, so the truth means we need a licence for
4 sockets... I don't think this is ideal, is it?

Anyways...
> Numbers #2 and #3 lead to contradictory courses of action; we cannot
> optimize for both at the same time.
> 
...that's certainly true, IMO.

> I think at some level we need to just try to accommodate both -- if the
> user doesn't have licensing issues, or prefers performance over
> licensing, then present a unified topology in PVH / HVM using CPUID,
> ACPI, &c.  I think this should be the default.
> 
> If the user has licensing issues, and doesn't mind having wonky or
> unreliable topology to its guests, then let the raw CPUID through.  But
> it would, in this case, be good to try to give the guest OS scheduler a
> hint that it shouldn't really bother trying to read the topology or do
> placement as a result, as any decisions will be unreliable.
> 
This last sentence is basically Juergen proposal, AFAICT, and I agree
that it would be good in case #3. But, thinking more about it, would it
harm in case #2? I don't think it would.

After all, it'd boil down to make peace with the fact that something
like CPUID, in a (not only PV!) VM, is just not reliable enough to use
it for building in-kernel scheduling related data structure, like
Linux's scheduling domains. It is unreliable because its content may
conflict with vNUMA, but, really, even with no vNUMA, it is unreliable
because it depends on whether the VM's vcpus are pinned or not, and if
not, it depends on where they actually run, which is pure randomness,
from the guest point of view.

So, I'm really starting to think that a patch stopping the Linux kernel
relying on CPUID, whether original or mangled, and for all kind of
guests, would really be a nice one to have!

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/27/2015 03:23 PM, Dario Faggioli wrote:
> On Mon, 2015-07-27 at 14:01 +0200, Juergen Gross wrote:
>> On 07/27/2015 01:11 PM, George Dunlap wrote:
>
>>> Or alternately, if the user wants to give up on the "consolidation"
>>> aspect of virtualization, they can pin vcpus to pcpus and then pass in
>>> the actual host topology (hyperthreads and all).
>>
>> There would be another solution, of course:
>>
>> Support hyperthreads in the Xen scheduler via gang scheduling. While
>> this is not a simple solution, it is a fair one. Hyperthreads on one
>> core can influence each other rather much. With both threads always
>> running vcpus of the same guest the penalty/advantage would stay in the
>> same domain. The guest could make really sensible scheduling decisions
>> and the licensing would still work as desired.
>>
> This is interesting indeed, but I much rather see it as something
> orthogonal, which may indeed bring benefits in some of the scenarios
> described here, but should not be considered *the* solution.

Correct. I still think it should be done.

> Implementing, enabling and asking users to use something like this will
> impact the system behavior and performance, in ways that may not be
> desirable for all use cases.

I'd make it a scheduler parameter. So you could it enable for a
specific cpupool where you want it to be active.

> So, while I do think that this may be something nice to have and offer,
> trying to use it for solving the problem we're debating here would make
> things even more complex to configure.
>
> Also, this would take care of HT related issues, but what about cores
> (as in 'should vcpus be cores of sockets or full sockets') and !HT boxes
> (like AMD)?

!HT boxes will have no problem: We won't have to hide HT as cores...

Regarding many sockets with 1 core each or 1 socket with many cores: I
think 1 socket for the non-NUMA case is okay, we'll want multiple
sockets for NUMA.

> Not to mention, as you say yourself, that it's not easy to implement.

Yeah, but it will be fun. ;-)

>
>> Just an idea, but maybe worth to explore further instead of tweaking
>> more and more bits to make the virtual system somehow act sane.
>>
> Sure, and it it's interesting indeed, for a bunch or reasons and
> purposes (as Tim is also noting). Not so much --or at least not
> necessarily-- for this one, IMO.

It's especially interesting regarding accounting. A vcpu running for
1 second can do much more if no other vcpu is running on the same core.
This would be a problem of the guest then, like on bare metal.

For real time purposes it might be even interesting to schedule only 1
vcpu per core to have a reliable high speed of the vcpu.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1374 bytes --]

On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote:
> On 07/24/2015 05:58 PM, Dario Faggioli wrote:

> > So, just to check if I'm understanding is correct: you'd like to add an
> > abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
> > code, to hide the direct interaction with CPUID.
> > Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
> > check with Xen/match vNUMA/whatever... Is this that you are saying?
> 
> Sort of, yes.
> 
> I just wouldn't add it, as it is already existing (more or less). It
> can deal right now with AMD and Intel, we would "just" have to add Xen.
> 
So, having gone through the rest of the thread (so far), and having
given a fair amount o thinking to this, I really think that something
like this would be a good thing to have in Linux.

Of course, it's not that my opinion on where should be in Linux counts
that much! :-D   Nevertheless, I wanted to make it clear that, while
skeptic at the beginning, I now think this is (part of) the way to go,
as I said and explained in my reply to George.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/27/2015 10:09 AM, Dario Faggioli wrote:
> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote:
>> On 07/24/2015 05:58 PM, Dario Faggioli wrote:
>>> So, just to check if I'm understanding is correct: you'd like to add an
>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
>>> code, to hide the direct interaction with CPUID.
>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
>>> check with Xen/match vNUMA/whatever... Is this that you are saying?
>> Sort of, yes.
>>
>> I just wouldn't add it, as it is already existing (more or less). It
>> can deal right now with AMD and Intel, we would "just" have to add Xen.
>>
> So, having gone through the rest of the thread (so far), and having
> given a fair amount o thinking to this, I really think that something
> like this would be a good thing to have in Linux.
>
> Of course, it's not that my opinion on where should be in Linux counts
> that much! :-D   Nevertheless, I wanted to make it clear that, while
> skeptic at the beginning, I now think this is (part of) the way to go,
> as I said and explained in my reply to George.

And I continue to believe that kernel solution does not address the 
userland problem which is no less important than making kernel do proper 
scheduling decisions (and I suspect when this patch goes for review 
that's what the scheduling people are going to say).

Remember the original problem that started this thread was that kernel 
complained that topology didn't make sense and it turned off all 
topology-related decisions. Which means that kernel already has a 
solution for weird topology. Some enumeration doesn't trigger this 
warning, but we can come up with one that does. Or we can indeed have a 
patch in kernel that will, possibly silently, fail topology_sane() when 
virtualized and not pinned.

(This is what I assume kernel does when topology_sane() fails. And if it 
doesn't, that's a bug IMO)

The licensing problem that Juergen described can be solved by pining 
vcpus and exposing HT bit. Besides,  creating a guest with 24 VPCUs and 
hoping that 16-core licensing will work I think is pushing it a bit when 
you know that VCPUs will jump around cores (i.e. "on average" you are 
running on more than 16 cores -- multi-threaded or not -- which arguably 
is what licensing is trying to prevent)

-boris

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/27/2015 04:34 PM, Boris Ostrovsky wrote:
> On 07/27/2015 10:09 AM, Dario Faggioli wrote:
>> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote:
>>> On 07/24/2015 05:58 PM, Dario Faggioli wrote:
>>>> So, just to check if I'm understanding is correct: you'd like to add an
>>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
>>>> code, to hide the direct interaction with CPUID.
>>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
>>>> check with Xen/match vNUMA/whatever... Is this that you are saying?
>>> Sort of, yes.
>>>
>>> I just wouldn't add it, as it is already existing (more or less). It
>>> can deal right now with AMD and Intel, we would "just" have to add Xen.
>>>
>> So, having gone through the rest of the thread (so far), and having
>> given a fair amount o thinking to this, I really think that something
>> like this would be a good thing to have in Linux.
>>
>> Of course, it's not that my opinion on where should be in Linux counts
>> that much! :-D   Nevertheless, I wanted to make it clear that, while
>> skeptic at the beginning, I now think this is (part of) the way to go,
>> as I said and explained in my reply to George.
>
> And I continue to believe that kernel solution does not address the
> userland problem which is no less important than making kernel do proper
> scheduling decisions (and I suspect when this patch goes for review
> that's what the scheduling people are going to say).
>
> Remember the original problem that started this thread was that kernel
> complained that topology didn't make sense and it turned off all
> topology-related decisions. Which means that kernel already has a
> solution for weird topology. Some enumeration doesn't trigger this
> warning, but we can come up with one that does. Or we can indeed have a
> patch in kernel that will, possibly silently, fail topology_sane() when
> virtualized and not pinned.

How would you come up with a topology the kernel is complaining about
and user mode scheduling will use for sane decisions ?

> (This is what I assume kernel does when topology_sane() fails. And if it
> doesn't, that's a bug IMO)
>
> The licensing problem that Juergen described can be solved by pining
> vcpus and exposing HT bit. Besides,  creating a guest with 24 VPCUs and

Hmm, yes. This way you sacrifice most of the virtualization advantages.

> hoping that 16-core licensing will work I think is pushing it a bit when
> you know that VCPUs will jump around cores (i.e. "on average" you are
> running on more than 16 cores -- multi-threaded or not -- which arguably
> is what licensing is trying to prevent)

On a machine with only 16 cores running on more than 16 cores? I have
some problems to believe this. The point was: if the license is happy on
bare metal it should be so when running on the same hardware as a guest.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/27/2015 04:34 PM, Boris Ostrovsky wrote:
> On 07/27/2015 10:09 AM, Dario Faggioli wrote:
>> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote:
>>> On 07/24/2015 05:58 PM, Dario Faggioli wrote:
>>>> So, just to check if I'm understanding is correct: you'd like to add an
>>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
>>>> code, to hide the direct interaction with CPUID.
>>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
>>>> check with Xen/match vNUMA/whatever... Is this that you are saying?
>>> Sort of, yes.
>>>
>>> I just wouldn't add it, as it is already existing (more or less). It
>>> can deal right now with AMD and Intel, we would "just" have to add Xen.
>>>
>> So, having gone through the rest of the thread (so far), and having
>> given a fair amount o thinking to this, I really think that something
>> like this would be a good thing to have in Linux.
>>
>> Of course, it's not that my opinion on where should be in Linux counts
>> that much! :-D   Nevertheless, I wanted to make it clear that, while
>> skeptic at the beginning, I now think this is (part of) the way to go,
>> as I said and explained in my reply to George.
>
> And I continue to believe that kernel solution does not address the
> userland problem which is no less important than making kernel do proper
> scheduling decisions (and I suspect when this patch goes for review
> that's what the scheduling people are going to say).

I didn't say it would solve that problem. It would decouple kernel
scheduling and cpuid values in order to be free to present cpuid values
to user land needed there.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Boris Ostrovsky]

On 07/27/2015 10:43 AM, Juergen Gross wrote:
> On 07/27/2015 04:34 PM, Boris Ostrovsky wrote:
>> On 07/27/2015 10:09 AM, Dario Faggioli wrote:
>>> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote:
>>>> On 07/24/2015 05:58 PM, Dario Faggioli wrote:
>>>>> So, just to check if I'm understanding is correct: you'd like to 
>>>>> add an
>>>>> abstraction layer, in Linux, like in generic (or, perhaps, 
>>>>> scheduling)
>>>>> code, to hide the direct interaction with CPUID.
>>>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, 
>>>>> it'd
>>>>> check with Xen/match vNUMA/whatever... Is this that you are saying?
>>>> Sort of, yes.
>>>>
>>>> I just wouldn't add it, as it is already existing (more or less). It
>>>> can deal right now with AMD and Intel, we would "just" have to add 
>>>> Xen.
>>>>
>>> So, having gone through the rest of the thread (so far), and having
>>> given a fair amount o thinking to this, I really think that something
>>> like this would be a good thing to have in Linux.
>>>
>>> Of course, it's not that my opinion on where should be in Linux counts
>>> that much! :-D   Nevertheless, I wanted to make it clear that, while
>>> skeptic at the beginning, I now think this is (part of) the way to go,
>>> as I said and explained in my reply to George.
>>
>> And I continue to believe that kernel solution does not address the
>> userland problem which is no less important than making kernel do proper
>> scheduling decisions (and I suspect when this patch goes for review
>> that's what the scheduling people are going to say).
>>
>> Remember the original problem that started this thread was that kernel
>> complained that topology didn't make sense and it turned off all
>> topology-related decisions. Which means that kernel already has a
>> solution for weird topology. Some enumeration doesn't trigger this
>> warning, but we can come up with one that does. Or we can indeed have a
>> patch in kernel that will, possibly silently, fail topology_sane() when
>> virtualized and not pinned.
>
> How would you come up with a topology the kernel is complaining about
> and user mode scheduling will use for sane decisions ?

We need to understand first why Dario's box is apparently the only one 
resulting in a warning and probably then emulate that enumeration.

And again, if that is not possible then just make topology_sane() fail.

>
>> (This is what I assume kernel does when topology_sane() fails. And if it
>> doesn't, that's a bug IMO)
>>
>> The licensing problem that Juergen described can be solved by pining
>> vcpus and exposing HT bit. Besides,  creating a guest with 24 VPCUs and
>
> Hmm, yes. This way you sacrifice most of the virtualization advantages.
>
>> hoping that 16-core licensing will work I think is pushing it a bit when
>> you know that VCPUs will jump around cores (i.e. "on average" you are
>> running on more than 16 cores -- multi-threaded or not -- which arguably
>> is what licensing is trying to prevent)
>
> On a machine with only 16 cores running on more than 16 cores? I have
> some problems to believe this. The point was: if the license is happy on
> bare metal it should be so when running on the same hardware as a guest.

Ok, that's not how I should have described it. I meant that IMO asking 
for 24 VCPUs is somewhat akin to oversubscribing since you kind of know 
that you dont' have 24 PCPUs, you are just trying to fool the kernel 
into thinking that threads are cores.

-boris

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2587 bytes --]

On Mon, 2015-07-27 at 10:34 -0400, Boris Ostrovsky wrote:
> On 07/27/2015 10:09 AM, Dario Faggioli wrote:

> > Of course, it's not that my opinion on where should be in Linux counts
> > that much! :-D   Nevertheless, I wanted to make it clear that, while
> > skeptic at the beginning, I now think this is (part of) the way to go,
> > as I said and explained in my reply to George.
> 
> And I continue to believe that kernel solution does not address the 
> userland problem which is no less important than making kernel do proper 
> scheduling decisions 
>
Sure, but the key point now is, IMO, whether we recognise that we're
dealing with two problems, which I think is the case.

One problem is:
 1. CPUID in VMs is unreliable, can we rely on it as few as possible?

The other problem is:
 2. if someone wants to rely on CPUID, what should we do.

Avoiding the scheduling domains (and/or whatever else!) to relay on
something that is unreliable by definition, which is what is being
proposed, seems to me a pretty decent solution to 1.

> (and I suspect when this patch goes for review 
> that's what the scheduling people are going to say).
> 
Perhaps. However, I may be too optimistic, but I think that making them
realize that all we are doing is stopping relying on unreliable grounds
would not be impossible... Especially if the code really looks as
Juergen said, i.e., it's already quite friendly to implementing
something like this...

> Remember the original problem that started this thread was that kernel 
> complained that topology didn't make sense and it turned off all 
> topology-related decisions. Which means that kernel already has a 
> solution for weird topology. Some enumeration doesn't trigger this 
> warning, but we can come up with one that does.
>
Exactly, because it relies on CPUID, which is unreliable!! :-)

> Or we can indeed have a 
> patch in kernel that will, possibly silently, fail topology_sane() when 
> virtualized and not pinned.
> 
Well, I think that, if I were a Linux scheduler or x86 maintainer, I
would oppose much more to a similar patch, rather than to one that makes
the kernel stop relying on an instruction that gives pCPU specific
results, to build-up long lived information, in a context where pCPUs
change. :-)

Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/27/2015 04:51 PM, Boris Ostrovsky wrote:
> On 07/27/2015 10:43 AM, Juergen Gross wrote:
>> On 07/27/2015 04:34 PM, Boris Ostrovsky wrote:
>>> On 07/27/2015 10:09 AM, Dario Faggioli wrote:
>>>> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote:
>>>>> On 07/24/2015 05:58 PM, Dario Faggioli wrote:
>>>>>> So, just to check if I'm understanding is correct: you'd like to
>>>>>> add an
>>>>>> abstraction layer, in Linux, like in generic (or, perhaps,
>>>>>> scheduling)
>>>>>> code, to hide the direct interaction with CPUID.
>>>>>> Such layer, on baremetal, would just read CPUID while, on PV-ops,
>>>>>> it'd
>>>>>> check with Xen/match vNUMA/whatever... Is this that you are saying?
>>>>> Sort of, yes.
>>>>>
>>>>> I just wouldn't add it, as it is already existing (more or less). It
>>>>> can deal right now with AMD and Intel, we would "just" have to add
>>>>> Xen.
>>>>>
>>>> So, having gone through the rest of the thread (so far), and having
>>>> given a fair amount o thinking to this, I really think that something
>>>> like this would be a good thing to have in Linux.
>>>>
>>>> Of course, it's not that my opinion on where should be in Linux counts
>>>> that much! :-D   Nevertheless, I wanted to make it clear that, while
>>>> skeptic at the beginning, I now think this is (part of) the way to go,
>>>> as I said and explained in my reply to George.
>>>
>>> And I continue to believe that kernel solution does not address the
>>> userland problem which is no less important than making kernel do proper
>>> scheduling decisions (and I suspect when this patch goes for review
>>> that's what the scheduling people are going to say).
>>>
>>> Remember the original problem that started this thread was that kernel
>>> complained that topology didn't make sense and it turned off all
>>> topology-related decisions. Which means that kernel already has a
>>> solution for weird topology. Some enumeration doesn't trigger this
>>> warning, but we can come up with one that does. Or we can indeed have a
>>> patch in kernel that will, possibly silently, fail topology_sane() when
>>> virtualized and not pinned.
>>
>> How would you come up with a topology the kernel is complaining about
>> and user mode scheduling will use for sane decisions ?
>
> We need to understand first why Dario's box is apparently the only one
> resulting in a warning and probably then emulate that enumeration.

This will lead to other problems in user land e.g. with hwloc.

> And again, if that is not possible then just make topology_sane() fail.

And again: once you claim that kernel mode isn't everything and here
you fail to respect possible user land requirements.

>>> (This is what I assume kernel does when topology_sane() fails. And if it
>>> doesn't, that's a bug IMO)
>>>
>>> The licensing problem that Juergen described can be solved by pining
>>> vcpus and exposing HT bit. Besides,  creating a guest with 24 VPCUs and
>>
>> Hmm, yes. This way you sacrifice most of the virtualization advantages.
>>
>>> hoping that 16-core licensing will work I think is pushing it a bit when
>>> you know that VCPUs will jump around cores (i.e. "on average" you are
>>> running on more than 16 cores -- multi-threaded or not -- which arguably
>>> is what licensing is trying to prevent)
>>
>> On a machine with only 16 cores running on more than 16 cores? I have
>> some problems to believe this. The point was: if the license is happy on
>> bare metal it should be so when running on the same hardware as a guest.
>
> Ok, that's not how I should have described it. I meant that IMO asking
> for 24 VCPUs is somewhat akin to oversubscribing since you kind of know
> that you dont' have 24 PCPUs, you are just trying to fool the kernel
> into thinking that threads are cores.

/proc/cpuinfo on bare metal will list 32 cpus. xl info in dom0 will list
32 cpus. You have 32 entities where you can do scheduling. So what's the
problem having a domU with 24 vcpus? There are still 8 pcpus free for
e.g. dom0 then.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[David Vrabel]

On 16/07/15 11:32, Dario Faggioli wrote:
> 
> Anyway, is there anything we can do to fix or workaround things?

This thread has gotten a bit long...

For Linux I would like to see:

1. No support for NUMA in PV guests -- if you want new MM features in a
guest use HVM.

2. For HVM guests, use the existing hardware interfaces to present NUMA
topology.  i.e., CPUID, ACPI tables etc.  This will work for both kernel
and userspace and both will see the same topology.

This also has the advantage that any hypervisor/toolstack work will also
be applicable to other guests (e.g., Windows).

David

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1391 bytes --]

On Mon, 2015-07-27 at 16:13 +0100, David Vrabel wrote:
> On 16/07/15 11:32, Dario Faggioli wrote:
> > 
> > Anyway, is there anything we can do to fix or workaround things?
> 
> This thread has gotten a bit long...
> 
Yep, indeed... :-(

> For Linux I would like to see:
> 
> 1. No support for NUMA in PV guests -- if you want new MM features in a
> guest use HVM.
> 
Wow... Really? What about all the code we have in libxl and Xen to deal
exactly with that?  What about making it possible to configure vNUMA for
Dom0?

:-O

> 2. For HVM guests, use the existing hardware interfaces to present NUMA
> topology.  i.e., CPUID, ACPI tables etc.  This will work for both kernel
> and userspace and both will see the same topology.
> 
> This also has the advantage that any hypervisor/toolstack work will also
> be applicable to other guests (e.g., Windows).
> 
Yeah, indeed. That's the downside of Juergen's "Linux scheduler
approach". But the issue is there, even without taking vNUMA into
account, and I think something like that would really help (only for
Dom0, and Linux guests, of course).

Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[David Vrabel]

On 27/07/15 17:02, Dario Faggioli wrote:
> On Mon, 2015-07-27 at 16:13 +0100, David Vrabel wrote:
>> On 16/07/15 11:32, Dario Faggioli wrote:
>>>
>>> Anyway, is there anything we can do to fix or workaround things?
>>
>> This thread has gotten a bit long...
>>
> Yep, indeed... :-(
> 
>> For Linux I would like to see:
>>
>> 1. No support for NUMA in PV guests -- if you want new MM features in a
>> guest use HVM.
>>
> Wow... Really? What about all the code we have in libxl and Xen to deal
> exactly with that?  What about making it possible to configure vNUMA for
> Dom0?

I don't think there is any (much?) PV-specific code in Xen/toolstack for
this, right?  It's common between HVM and PV, yes?

I would prefer effort into making a no-dm HVM dom0 work instead because
this is a better long-term solution for dom0.

That said, if someone does the work for vNUMA in Linux PV guests and it
looks sensible and self-contained then I would probably merge it.

>> 2. For HVM guests, use the existing hardware interfaces to present NUMA
>> topology.  i.e., CPUID, ACPI tables etc.  This will work for both kernel
>> and userspace and both will see the same topology.
>>
>> This also has the advantage that any hypervisor/toolstack work will also
>> be applicable to other guests (e.g., Windows).
>>
> Yeah, indeed. That's the downside of Juergen's "Linux scheduler
> approach". But the issue is there, even without taking vNUMA into
> account, and I think something like that would really help (only for
> Dom0, and Linux guests, of course).

I disagree.  Whether we're using vNUMA or not, Xen should still ensure
that the guest kernel and userspace see a consistent and correct
topology using the native mechanisms.

David

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Andrew Cooper]

On 27/07/15 17:31, David Vrabel wrote:
>
>>> 2. For HVM guests, use the existing hardware interfaces to present NUMA
>>> topology.  i.e., CPUID, ACPI tables etc.  This will work for both kernel
>>> and userspace and both will see the same topology.
>>>
>>> This also has the advantage that any hypervisor/toolstack work will also
>>> be applicable to other guests (e.g., Windows).
>>>
>> Yeah, indeed. That's the downside of Juergen's "Linux scheduler
>> approach". But the issue is there, even without taking vNUMA into
>> account, and I think something like that would really help (only for
>> Dom0, and Linux guests, of course).
> I disagree.  Whether we're using vNUMA or not, Xen should still ensure
> that the guest kernel and userspace see a consistent and correct
> topology using the native mechanisms.

+1

~Andrew

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 4967 bytes --]

On Mon, 2015-07-27 at 17:33 +0100, Andrew Cooper wrote:
> On 27/07/15 17:31, David Vrabel wrote:
> >
> >> Yeah, indeed. That's the downside of Juergen's "Linux scheduler
> >> approach". But the issue is there, even without taking vNUMA into
> >> account, and I think something like that would really help (only for
> >> Dom0, and Linux guests, of course).
> > I disagree.  Whether we're using vNUMA or not, Xen should still ensure
> > that the guest kernel and userspace see a consistent and correct
> > topology using the native mechanisms.
> 
> +1
> 
+1 from me as well. In fact, a mechanism for making exactly such thing
happen, was what I was after when starting the thread.

Then it came up that CPUID needs to be used for at least two different
and potentially conflicting purposes, that we want to support both and
that, whether and for whatever reason it's used, Linux configures its
scheduler after it, potentially resulting in rather pathological setups.

It's at that point that some decoupling started to appear
interesting... :-P

Also, are we really being consistent? If my methodology is correct
(which might not be, please, double check, and sorry for that), I'm
seeing quite some inconsistency around:

HOST:
 root@Zhaman:~# xl info -n
 ...
 cpu_topology           :
 cpu:    core    socket     node
   0:       0        1        0
   1:       0        1        0
   2:       1        1        0
   3:       1        1        0
   4:       9        1        0
   5:       9        1        0
   6:      10        1        0
   7:      10        1        0
   8:       0        0        1
   9:       0        0        1
  10:       1        0        1
  11:       1        0        1
  12:       9        0        1
  13:       9        0        1
  14:      10        0        1
  15:      10        0        1
 ...
 root@Zhaman:~# xl vcpu-list test
 Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
 test                                 2     0    0   r--       1.5  0 / all
 test                                 2     1    1   r--       0.2  1 / all
 test                                 2     2    8   -b-       2.2  8 / all
 test                                 2     3    9   -b-       2.0  9 / all

GUEST (HVM, 4 vcpus):
 root@test:~# cpuid|grep CORE_ID
   (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0
   (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=1
   (APIC synth): PKG_ID=0 CORE_ID=0 SMT_ID=0
   (APIC synth): PKG_ID=0 CORE_ID=0 SMT_ID=1

HOST:
 root@Zhaman:~# xl vcpu-pin 2 all 0
 root@Zhaman:~# xl vcpu-list 2
 Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
 test                                 2     0    0   -b-      43.7  0 / all
 test                                 2     1    0   -b-      38.4  0 / all
 test                                 2     2    0   -b-      36.9  0 / all
 test                                 2     3    0   -b-      38.8  0 / all

GUEST:
 root@test:~# cpuid|grep CORE_ID
   (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0
   (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0
   (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0
   (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0

HOST:
 root@Zhaman:~# xl vcpu-pin 2 0 7
 root@Zhaman:~# xl vcpu-pin 2 1 7
 root@Zhaman:~# xl vcpu-pin 2 2 15
 root@Zhaman:~# xl vcpu-pin 2 3 15
 root@Zhaman:~# xl vcpu-list 2
 Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
 test                                 2     0    7   -b-      44.3  7 / all
 test                                 2     1    7   -b-      38.9  7 / all
 test                                 2     2   15   -b-      37.3  15 / all
 test                                 2     3   15   -b-      39.2  15 / all

GUEST:
 root@test:~# cpuid|grep CORE_ID
   (APIC synth): PKG_ID=0 CORE_ID=26 SMT_ID=1
   (APIC synth): PKG_ID=0 CORE_ID=26 SMT_ID=1
   (APIC synth): PKG_ID=0 CORE_ID=10 SMT_ID=1
   (APIC synth): PKG_ID=0 CORE_ID=10 SMT_ID=1

So, it looks to me that:
 1) any application using CPUID for either licensing or
    placement/performance optimization will get (potentially) random 
    results;
 2) whatever set of values the kernel used, during guest boot, to build
    up its internal scheduling data structures, has no guarantee of
    being related to any value returned by CPUID, at a later point.

Hence, I think I'm seeing inconsistency between kernel and userspace
(and between userspace and itself, over time) already... Am I
overlooking something? 

(I'll provide the same, for a PV guest, tomorrow.)

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Konrad Rzeszutek Wilk]

. snip..
> So, it looks to me that:
>  1) any application using CPUID for either licensing or
>     placement/performance optimization will get (potentially) random 
>     results;

Right, that is a bug that Andrew outlined in this leveling document I
believe. We just pluck the cpuid results on whatever PCPU we were
running when the toolstack constructs it.

>  2) whatever set of values the kernel used, during guest boot, to build
>     up its internal scheduling data structures, has no guarantee of
>     being related to any value returned by CPUID, at a later point.

Or after a migration.
> 
> Hence, I think I'm seeing inconsistency between kernel and userspace
> (and between userspace and itself, over time) already... Am I
> overlooking something? 

No. I think that is bad. We should be providing the same data - unless
pinning is used or any other mechanism is employed to change the cpuid.

By default the cpuid ought to be same for a guest throughout its life
(lets ignore PV which is 'special').

Also oddly, what is with the SMT threads?

On my HVM guests, if I allocate four CPUs (vcpu=4), I get four cores and
each core says it has four SMT threads? This is based on /proc/cpuinfo:

[konrad@build-external linux]$ cat /proc/cpuinfo |grep "core id"
core id         : 0
core id         : 1
core id         : 2
core id         : 3
[konrad@build-external linux]$ cat /proc/cpuinfo |grep "siblings"
siblings        : 4
siblings        : 4
siblings        : 4
siblings        : 4

?
> 
> (I'll provide the same, for a PV guest, tomorrow.)
> 
> Regards,
> Dario
> 
> -- 
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 5487 bytes --]

On 27/07/2015 18:42, Dario Faggioli wrote:
> On Mon, 2015-07-27 at 17:33 +0100, Andrew Cooper wrote: >> On 27/07/15 17:31, David Vrabel wrote: >>> >>>> Yeah, indeed. That's
the downside of Juergen's "Linux scheduler >>>> approach". But the issue
is there, even without taking vNUMA into >>>> account, and I think
something like that would really help (only for >>>> Dom0, and Linux
guests, of course). >>> I disagree.  Whether we're using vNUMA or not,
Xen should still ensure >>> that the guest kernel and userspace see a
consistent and correct >>> topology using the native mechanisms. >> >>
+1 >> > +1 from me as well. In fact, a mechanism for making exactly such
thing > happen, was what I was after when starting the thread. > > Then
it came up that CPUID needs to be used for at least two different > and
potentially conflicting purposes, that we want to support both and >
that, whether and for whatever reason it's used, Linux configures its >
scheduler after it, potentially resulting in rather pathological setups.

I don't see what the problem is here.  Fundamentally, "NUMA optimise" vs
"comply with licence" is a user/admin decision at boot time, and we need
not cater to both halves at the same time.

Supporting either, as chosen by the admin, is worthwhile.

> > > It's at that point that some decoupling started to appear >
interesting... :-P > > Also, are we really being consistent? If my
methodology is correct > (which might not be, please, double check, and
sorry for that), I'm > seeing quite some inconsistency around: > > HOST:
>  root@Zhaman:~# xl info -n >  ... >  cpu_topology           : > 
cpu:    core    socket     node >    0:       0        1        0 >   
1:       0        1        0 >    2:       1        1        0 >   
3:       1        1        0 >    4:       9        1        0 >   
5:       9        1        0 >    6:      10        1        0 >   
7:      10        1        0 >    8:       0        0        1 >   
9:       0        0        1 >   10:       1        0        1 >  
11:       1        0        1 >   12:       9        0        1 >  
13:       9        0        1 >   14:      10        0        1 >  
15:      10        0        1

o_O

What kind of system results in this layout?  Can you dump the ACPI
tables and make them available?

> >  ... >  root@Zhaman:~# xl vcpu-list test > 
Name                                ID  VCPU   CPU State   Time(s)
Affinity (Hard / Soft) >  test                                 2    
0    0   r--       1.5  0 / all >  test                                
2     1    1   r--       0.2  1 / all > 
test                                 2     2    8   -b-       2.2  8 /
all >  test                                 2     3    9   -b-      
2.0  9 / all > > GUEST (HVM, 4 vcpus): >  root@test:~# cpuid|grep
CORE_ID >    (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 >    (APIC
synth): PKG_ID=0 CORE_ID=16 SMT_ID=1 >    (APIC synth): PKG_ID=0
CORE_ID=0 SMT_ID=0 >    (APIC synth): PKG_ID=0 CORE_ID=0 SMT_ID=1 > >
HOST: >  root@Zhaman:~# xl vcpu-pin 2 all 0 >  root@Zhaman:~# xl
vcpu-list 2 >  Name                                ID  VCPU   CPU
State   Time(s) Affinity (Hard / Soft) > 
test                                 2     0    0   -b-      43.7  0 /
all >  test                                 2     1    0   -b-     
38.4  0 / all >  test                                 2     2    0  
-b-      36.9  0 / all >  test                                 2    
3    0   -b-      38.8  0 / all > > GUEST: >  root@test:~# cpuid|grep
CORE_ID >    (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 >    (APIC
synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 >    (APIC synth): PKG_ID=0
CORE_ID=16 SMT_ID=0 >    (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 > >
HOST: >  root@Zhaman:~# xl vcpu-pin 2 0 7 >  root@Zhaman:~# xl vcpu-pin
2 1 7 >  root@Zhaman:~# xl vcpu-pin 2 2 15 >  root@Zhaman:~# xl vcpu-pin
2 3 15 >  root@Zhaman:~# xl vcpu-list 2 > 
Name                                ID  VCPU   CPU State   Time(s)
Affinity (Hard / Soft) >  test                                 2    
0    7   -b-      44.3  7 / all >  test                                
2     1    7   -b-      38.9  7 / all > 
test                                 2     2   15   -b-      37.3  15 /
all >  test                                 2     3   15   -b-     
39.2  15 / all > > GUEST: >  root@test:~# cpuid|grep CORE_ID >    (APIC
synth): PKG_ID=0 CORE_ID=26 SMT_ID=1 >    (APIC synth): PKG_ID=0
CORE_ID=26 SMT_ID=1 >    (APIC synth): PKG_ID=0 CORE_ID=10 SMT_ID=1 >   
(APIC synth): PKG_ID=0 CORE_ID=10 SMT_ID=1 > > So, it looks to me that:
>  1) any application using CPUID for either licensing or >    
placement/performance optimization will get (potentially) random >    
results; >  2) whatever set of values the kernel used, during guest
boot, to build >     up its internal scheduling data structures, has no
guarantee of >     being related to any value returned by CPUID, at a
later point. > > Hence, I think I'm seeing inconsistency between kernel
and userspace > (and between userspace and itself, over time) already...
Am I > overlooking something?

All current CPUID values presented to guests are about as reliable as
being picked from /dev/urandom.  (This isn't strictly true - the feature
flags will be in the right ballpark if the VM has not migrated yet).

Fixing this (as described in my feature levelling design document) is
sufficiently non-trivial that it has been deferred to post
feature-levelling work.

~Andrew


[-- Attachment #1.2: Type: text/html, Size: 7657 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/28/2015 01:19 AM, Andrew Cooper wrote:
> On 27/07/2015 18:42, Dario Faggioli wrote:
>> On Mon, 2015-07-27 at 17:33 +0100, Andrew Cooper wrote:
>>> On 27/07/15 17:31, David Vrabel wrote:
>>>>
>>>>> Yeah, indeed.
>>>>> That's the downside of Juergen's "Linux scheduler
>>>>> approach". But the issue is there, even without taking vNUMA into
>>>>> account, and I  think something like that would really help (only for
>>>>> Dom0, and Linux guests, of course).
>>>> I disagree.  Whether we're using vNUMA or not, Xen should still ensure
>>>> that the guest kernel and userspace see a consistent and correct
>>>> topology using the native mechanisms.
>>>
>>> +1
>>>
>> +1 from me as well. In fact, a mechanism for making exactly such thing
>> happen, was what I was after when starting the thread.
>>
>> Then it came up that CPUID needs to be used for at least two different
>> and potentially conflicting purposes, that we want to support both and
>> that, whether and for whatever reason it's used, Linux configures its
>> scheduler after it, potentially resulting in rather pathological setups.
>>
> I don't see what the problem is here.  Fundamentally, "NUMA optimise" vs
> "comply with licence" is a user/admin decision at boot time, and we need
> not cater to both halves at the same time.
>
> Supporting either, as chosen by the admin, is worthwhile.

Wrong assumption again. *It's not only about NUMA*!

The choice is: "comply with license" against "sane scheduling". NUMA
just makes it more obvious, that the data the guest's scheduling
decisions are based on is garbage as soon as you tell the guest there
are hyperthreads without pinning the vcpus.

Right now the sibling information is more or less random leading
eventually to some vcpus thinking they have no sibling at all. As soon
as you deliver sibling information based on vcpu number you might end
up with a deterministic bad scheduling behaviour.


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/27/2015 04:09 PM, Dario Faggioli wrote:
> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote:
>> On 07/24/2015 05:58 PM, Dario Faggioli wrote:
>
>>> So, just to check if I'm understanding is correct: you'd like to add an
>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
>>> code, to hide the direct interaction with CPUID.
>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
>>> check with Xen/match vNUMA/whatever... Is this that you are saying?
>>
>> Sort of, yes.
>>
>> I just wouldn't add it, as it is already existing (more or less). It
>> can deal right now with AMD and Intel, we would "just" have to add Xen.
>>
> So, having gone through the rest of the thread (so far), and having
> given a fair amount o thinking to this, I really think that something
> like this would be a good thing to have in Linux.
>
> Of course, it's not that my opinion on where should be in Linux counts
> that much! :-D   Nevertheless, I wanted to make it clear that, while
> skeptic at the beginning, I now think this is (part of) the way to go,
> as I said and explained in my reply to George.

I think it's time to obtain some real numbers.

I'll make some performance tests on a big machine (4 sockets, 60 cores,
120 threads) regarding topology information:

- bare metal
- "random" topology (like today)
- "simple" topology (all vcpus regarded as equal)
- "real" topology with all vcpus pinned

This should show:

- how intrusive would the topology patch(es) be?
- what is the performance impact of a "wrong" scheduling data base


Juergen

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1.1: Type: text/plain, Size: 3627 bytes --]

On Tue, 2015-07-28 at 00:19 +0100, Andrew Cooper wrote:
> On 27/07/2015 18:42, Dario Faggioli wrote:
> > On Mon, 2015-07-27 at 17:33 +0100, Andrew Cooper wrote: >> On
> 27/07/15 17:31, David Vrabel wrote: >>> >>>> Yeah, indeed. That's the
> downside of Juergen's "Linux scheduler >>>> approach". But the issue
> is there, even without taking vNUMA into >>>> account, and I think
> something like that would really help (only for >>>> Dom0, and Linux
> guests, of course). >>> I disagree.  Whether we're using vNUMA or not,
> Xen should still ensure >>> that the guest kernel and userspace see a
> consistent and correct >>> topology using the native mechanisms. >> >>
> +1 >> > +1 from me as well. In fact, a mechanism for making exactly
> such thing > happen, was what I was after when starting the thread. >
> > Then it came up that CPUID needs to be used for at least two
> different > and potentially conflicting purposes, that we want to
> support both and > that, whether and for whatever reason it's used,
> Linux configures its > scheduler after it, potentially resulting in
> rather pathological setups.
> 
> I don't see what the problem is here.  Fundamentally, "NUMA optimise"
> vs "comply with licence" is a user/admin decision at boot time, and we
> need not cater to both halves at the same time.
>
> Supporting either, as chosen by the admin, is worthwhile.
>
I agree (with the added consideration that, as Juergen is saying, this
is not only related to [v]NUMA).

When optimising for placement/NUMA/scheduling/etc, I really don't see
any problem at all (once more, that's what I was after when starting the
thread!).

An issue might be that, when 'optimise for licenses', especially (but
not limiting to) if in combination with vNUMA, you risk putting the
guest scheduler in a really weird situation, hence the idea of taking
care of that in some other way.

Anyway, if Juergen manages to give us some numbers, we may have a better
picture of whether and how much this matters.

> > > > It's at that point that some decoupling started to appear >
> interesting... :-P > > Also, are we really being consistent? If my
> methodology is correct > (which might not be, please, double check,
> and sorry for that), I'm > seeing quite some inconsistency around: > >
> HOST: >  root@Zhaman:~# xl info -n >  ... >  cpu_topology           :
> >  cpu:    core    socket     node >    0:       0        1        0 >
> 1:       0        1        0 >    2:       1        1        0 >    3:
> 1        1        0 >    4:       9        1        0 >    5:       9
> 1        0 >    6:      10        1        0 >    7:      10        1
> 0 >    8:       0        0        1 >    9:       0        0        1
> >   10:       1        0        1 >   11:       1        0        1 >
> 12:       9        0        1 >   13:       9        0        1 >
> 14:      10        0        1 >   15:      10        0        1
> 
> o_O
>
> What kind of system results in this layout?  Can you dump the ACPI
> tables and make them available?
>
This is the test box I use for day to day development and testing...

I've done an acpidump (while on baremetal), and am attaching the result.
Let me know if I should do anything more or different.

If I can ask, what is it that looks weird?

Thanks and Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.1.2: acpi.log --]
[-- Type: text/x-log, Size: 199195 bytes --]

APIC @ 0x0000000000000000
  0000: 41 50 49 43 6A 02 00 00 01 F6 44 45 4C 4C 20 20  APICj.....DELL  
  0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20  B10K   .....ASL 
  0020: 61 00 00 00 00 00 E0 FE 01 00 00 00 00 08 01 20  a.............. 
  0030: 01 00 00 00 00 08 02 22 01 00 00 00 00 08 03 32  .......".......2
  0040: 01 00 00 00 00 08 04 34 01 00 00 00 00 08 05 00  .......4........
  0050: 01 00 00 00 00 08 06 02 01 00 00 00 00 08 07 12  ................
  0060: 01 00 00 00 00 08 08 14 01 00 00 00 00 08 09 21  ...............!
  0070: 01 00 00 00 00 08 0A 23 01 00 00 00 00 08 0B 33  .......#.......3
  0080: 01 00 00 00 00 08 0C 35 01 00 00 00 00 08 0D 01  .......5........
  0090: 01 00 00 00 00 08 0E 03 01 00 00 00 00 08 0F 13  ................
  00A0: 01 00 00 00 00 08 10 15 01 00 00 00 00 08 11 00  ................
  00B0: 00 00 00 00 00 08 12 00 00 00 00 00 00 08 13 00  ................
  00C0: 00 00 00 00 00 08 14 00 00 00 00 00 00 08 15 00  ................
  00D0: 00 00 00 00 00 08 16 00 00 00 00 00 00 08 17 00  ................
  00E0: 00 00 00 00 00 08 18 00 00 00 00 00 00 08 19 00  ................
  00F0: 00 00 00 00 00 08 1A 00 00 00 00 00 00 08 1B 00  ................
  0100: 00 00 00 00 00 08 1C 00 00 00 00 00 00 08 1D 00  ................
  0110: 00 00 00 00 00 08 1E 00 00 00 00 00 00 08 1F 00  ................
  0120: 00 00 00 00 00 08 20 00 00 00 00 00 00 08 21 00  ...... .......!.
  0130: 00 00 00 00 00 08 22 00 00 00 00 00 00 08 23 00  ......".......#.
  0140: 00 00 00 00 00 08 24 00 00 00 00 00 00 08 25 00  ......$.......%.
  0150: 00 00 00 00 00 08 26 00 00 00 00 00 00 08 27 00  ......&.......'.
  0160: 00 00 00 00 00 08 28 00 00 00 00 00 00 08 29 00  ......(.......).
  0170: 00 00 00 00 00 08 2A 00 00 00 00 00 00 08 2B 00  ......*.......+.
  0180: 00 00 00 00 00 08 2C 00 00 00 00 00 00 08 2D 00  ......,.......-.
  0190: 00 00 00 00 00 08 2E 00 00 00 00 00 00 08 2F 00  ............../.
  01A0: 00 00 00 00 00 08 30 00 00 00 00 00 00 08 31 00  ......0.......1.
  01B0: 00 00 00 00 00 08 32 00 00 00 00 00 00 08 33 00  ......2.......3.
  01C0: 00 00 00 00 00 08 34 00 00 00 00 00 00 08 35 00  ......4.......5.
  01D0: 00 00 00 00 00 08 36 00 00 00 00 00 00 08 37 00  ......6.......7.
  01E0: 00 00 00 00 00 08 38 00 00 00 00 00 00 08 39 00  ......8.......9.
  01F0: 00 00 00 00 00 08 3A 00 00 00 00 00 00 08 3B 00  ......:.......;.
  0200: 00 00 00 00 00 08 3C 00 00 00 00 00 00 08 3D 00  ......<.......=.
  0210: 00 00 00 00 00 08 3E 00 00 00 00 00 00 08 3F 00  ......>.......?.
  0220: 00 00 00 00 00 08 40 00 00 00 00 00 01 0C 08 00  ......@.........
  0230: 00 00 C0 FE 00 00 00 00 01 0C 09 00 00 00 C8 FE  ................
  0240: 18 00 00 00 01 0C 0A 00 00 80 C8 FE 30 00 00 00  ............0...
  0250: 02 0A 00 00 02 00 00 00 00 00 02 0A 00 09 09 00  ................
  0260: 00 00 0D 00 04 06 FF 0D 00 01                    ..........

ASF! @ 0x0000000000000000
  0000: 41 53 46 21 96 00 00 00 20 84 44 45 4C 4C 20 20  ASF!.... .DELL  
  0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20  B10K   .....ASL 
  0020: 61 00 00 00 04 00 07 00 00 01 88 01 00 2C 00 00  a............,..
  0030: 00 03 0C 89 04 01 01 05 6F 00 68 08 88 17 00 89  ........o.h.....
  0040: 04 04 04 07 6F 00 68 20 88 03 00 89 05 01 01 19  ....o.h ........
  0050: 6F 00 68 20 88 22 00 02 00 18 00 04 04 00 00 02  o.h ."..........
  0060: 88 00 01 01 88 00 02 00 88 00 03 03 88 00 04 03  ................
  0070: 00 17 00 22 18 00 00 00 03 0F 01 00 00 00 00 00  ..."............
  0080: 00 00 00 00 00 00 80 00 10 00 FF 00 6E 02 00 00  ............n...
  0090: 02 A2 00 00 00 00                                ......

BOOT @ 0x0000000000000000
  0000: 42 4F 4F 54 28 00 00 00 01 04 44 45 4C 4C 20 20  BOOT(.....DELL  
  0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20  B10K   .....ASL 
  0020: 61 00 00 00 7A 00 00 00                          a...z...

DMAR @ 0x0000000000000000
  0000: 44 4D 41 52 00 01 00 00 01 5A 44 45 4C 4C 20 20  DMAR.....ZDELL  
  0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20  B10K   .....ASL 
  0020: 61 00 00 00 27 01 00 00 00 00 00 00 00 00 00 00  a...'...........
  0030: 00 00 28 00 00 00 00 00 00 E0 FF DF 00 00 00 00  ..(.............
  0040: 02 08 00 00 00 20 07 00 02 08 00 00 00 20 09 00  ..... ....... ..
  0050: 03 08 00 00 0A 20 13 00 00 00 20 00 01 00 00 00  ..... .... .....
  0060: 00 00 DC FE 00 00 00 00 03 08 00 00 09 00 13 00  ................
  0070: 03 08 00 00 08 00 1F 07 01 00 58 00 00 00 00 00  ..........X.....
  0080: 00 80 E5 DB 00 00 00 00 FF FF E6 DB 00 00 00 00  ................
  0090: 01 08 00 00 00 00 1D 00 01 08 00 00 00 00 1D 01  ................
  00A0: 01 08 00 00 00 00 1D 02 01 08 00 00 00 00 1D 07  ................
  00B0: 01 08 00 00 00 00 1A 00 01 08 00 00 00 00 1A 01  ................
  00C0: 01 08 00 00 00 00 1A 02 01 08 00 00 00 00 1A 07  ................
  00D0: 02 00 18 00 00 00 00 00 02 08 00 00 00 00 03 00  ................
  00E0: 02 08 00 00 00 00 07 00 02 00 18 00 00 00 00 00  ................
  00F0: 02 08 00 00 00 20 07 00 02 08 00 00 00 20 09 00  ..... ....... ..

DSDT @ 0x0000000000000000
  0000: 44 53 44 54 B9 55 00 00 01 43 44 45 4C 4C 00 00  DSDT.U...CDELL..
  0010: 64 74 5F 65 78 00 00 00 00 10 00 00 49 4E 54 4C  dt_ex.......INTL
  0020: 24 06 05 20 14 07 44 42 49 4E 00 A3 10 47 05 5C  $.. ..DBIN...G.\
  0030: 00 5B 82 1F 2E 5F 53 42 5F 56 42 54 4E 08 5F 48  .[..._SB_VBTN._H
  0040: 49 44 0C 41 D0 0C 0C 08 5F 50 52 57 12 05 02 01  ID.A...._PRW....
  0050: 0A 04 5B 80 53 53 54 53 01 0B 28 08 01 5B 81 0B  ..[.SSTS..(..[..
  0060: 53 53 54 53 41 54 53 54 53 01 5B 80 53 41 43 54  SSTSATSTS.[.SACT
  0070: 01 0B 2A 08 01 5B 81 0D 53 41 43 54 41 00 02 54  ..*..[..SACTA..T
  0080: 50 4F 4C 01 10 25 5F 47 50 45 14 1F 5F 4C 30 30  POL..%_GPE.._L00
  0090: 00 70 00 54 50 4F 4C 70 01 54 53 54 53 86 5C 2E  .p.TPOLp.TSTS.\.
  00A0: 5F 53 42 5F 56 42 54 4E 0A 02 10 30 5F 53 42 5F  _SB_VBTN...0_SB_
  00B0: 14 2A 5F 49 4E 49 00 A0 23 5B 12 5F 4F 53 49 60  .*_INI..#[._OSI`
  00C0: A0 1A 5F 4F 53 49 0D 57 69 6E 64 6F 77 73 20 32  .._OSI.Windows 2
  00D0: 30 30 31 00 70 0A 04 4D 53 4F 53 10 4A 26 5C 00  001.p..MSOS.J&\.
  00E0: 08 4D 53 4F 53 00 5B 01 4D 54 58 5F 01 14 38 43  .MSOS.[.MTX_..8C
  00F0: 4D 52 44 01 5B 23 53 4D 49 4D FF FF 70 68 60 7B  MRD.[#SMIM..ph`{
  0100: 60 0A 7F 60 70 60 53 4D 49 44 70 0A 84 53 4D 49  `..`p`SMIDp..SMI
  0110: 43 70 0A 85 53 4D 49 43 70 53 4D 49 44 60 5B 27  Cp..SMICpSMID`['
  0120: 53 4D 49 4D A4 60 14 36 43 4D 57 52 02 5B 23 53  SMIM.`.6CMWR.[#S
  0130: 4D 49 4D FF FF 70 68 60 7D 60 0A 80 60 70 60 53  MIM..ph`}`..`p`S
  0140: 4D 49 44 70 0A 84 53 4D 49 43 70 69 53 4D 49 44  MIDp..SMICpiSMID
  0150: 70 0A 85 53 4D 49 43 5B 27 53 4D 49 4D 14 11 47  p..SMIC['SMIM..G
  0160: 43 4B 42 00 A4 7B 0A 20 43 4D 52 44 0A 26 00 14  CKB..{. CMRD.&..
  0170: 11 47 43 4D 53 00 A4 7B 0A 40 43 4D 52 44 0A 6F  .GCMS..{.@CMRD.o
  0180: 00 14 11 47 43 4F 4E 00 A4 7B 0A 04 43 4D 52 44  ...GCON..{..CMRD
  0190: 0A 55 00 14 14 47 43 55 43 00 7B 01 43 4D 52 44  .U...GCUC.{.CMRD
  01A0: 0A 22 60 A4 7F 01 60 00 14 11 47 43 53 31 00 A4  ."`...`...GCS1..
  01B0: 7B 0A 03 43 4D 52 44 0A 25 00 14 11 47 43 53 32  {..CMRD.%...GCS2
  01C0: 00 A4 7B 0A 0C 43 4D 52 44 0A 25 00 14 11 47 43  ..{..CMRD.%...GC
  01D0: 46 44 00 A4 7B 0A 18 43 4D 52 44 0A 55 00 14 11  FD..{..CMRD.U...
  01E0: 47 43 50 50 00 A4 7B 0A 70 43 4D 52 44 0A 23 00  GCPP..{.pCMRD.#.
  01F0: 14 11 47 43 54 50 00 A4 7B 0A 40 43 4D 52 44 0A  ..GCTP..{.@CMRD.
  0200: 22 00 14 11 47 43 4E 43 00 A4 7B 0A 03 43 4D 52  "...GCNC..{..CMR
  0210: 44 0A 59 00 14 4A 10 48 41 43 4B 00 A0 1E 93 4D  D.Y..J.HACK....M
  0220: 53 4F 53 0A 04 70 43 4D 52 44 0A 6F 60 7D 60 0A  SOS..pCMRD.o`}`.
  0230: 04 60 43 4D 57 52 0A 6F 60 A4 01 A1 15 70 43 4D  .`CMWR.o`....pCM
  0240: 52 44 0A 6F 60 7B 60 0A FB 60 43 4D 57 52 0A 6F  RD.o`{`..`CMWR.o
  0250: 60 72 87 5F 4F 53 5F 01 60 70 60 61 08 42 55 46  `r._OS_.`p`a.BUF
  0260: 30 11 02 60 08 42 55 46 31 11 02 60 08 4F 53 4E  0..`.BUF1..`.OSN
  0270: 54 01 08 4F 53 39 38 01 70 5F 4F 53 5F 42 55 46  T..OS98.p_OS_BUF
  0280: 30 70 0D 4D 69 63 72 6F 73 6F 66 74 20 57 69 6E  0p.Microsoft Win
  0290: 64 6F 77 73 20 4E 54 00 42 55 46 31 A2 1F 60 76  dows NT.BUF1..`v
  02A0: 60 A0 12 93 83 88 42 55 46 30 60 00 83 88 42 55  `.....BUF0`...BU
  02B0: 46 31 60 00 A1 07 70 00 4F 53 4E 54 A0 0E 4F 53  F1`...p.OSNT..OS
  02C0: 4E 54 70 0A 03 4D 53 4F 53 A4 01 A1 43 05 70 0D  NTp..MSOS...C.p.
  02D0: 4D 69 63 72 6F 73 6F 66 74 20 57 69 6E 64 6F 77  Microsoft Window
  02E0: 73 00 42 55 46 31 A2 1F 61 76 61 A0 12 93 83 88  s.BUF1..ava.....
  02F0: 42 55 46 30 61 00 83 88 42 55 46 31 61 00 A1 07  BUF0a...BUF1a...
  0300: 70 00 4F 53 39 38 A0 0D 4F 53 39 38 70 01 4D 53  p.OS98..OS98p.MS
  0310: 4F 53 A4 00 A1 0A 70 0A 02 4D 53 4F 53 A4 00 14  OS....p..MSOS...
  0320: 26 49 53 4C 49 00 A0 1B 5B 12 5F 4F 53 49 60 A0  &ISLI...[._OSI`.
  0330: 0E 5F 4F 53 49 0D 4C 69 6E 75 78 00 A4 01 A1 03  ._OSI.Linux.....
  0340: A4 00 A1 03 A4 00 10 05 5F 53 42 5F 5B 80 43 4D  ........_SB_[.CM
  0350: 53 5F 01 0A 70 0A 02 5B 81 10 43 4D 53 5F 01 43  S_..p..[..CMS_.C
  0360: 4D 53 49 08 43 4D 53 44 08 5B 80 53 4D 49 52 01  MSI.CMSD.[.SMIR.
  0370: 0A B2 0A 02 5B 81 10 53 4D 49 52 01 53 4D 49 43  ....[..SMIR.SMIC
  0380: 08 53 4D 49 44 08 5B 80 53 54 4F 4E 01 0A 84 01  .SMID.[.STON....
  0390: 5B 81 0B 53 54 4F 4E 01 4D 47 49 43 08 5B 80 53  [..STON.MGIC.[.S
  03A0: 49 4F 5F 01 0A 2E 0A 02 5B 81 10 53 49 4F 5F 01  IO_.....[..SIO_.
  03B0: 53 49 4F 49 08 53 49 4F 44 08 5B 80 50 4D 31 52  SIOI.SIOD.[.PM1R
  03C0: 01 0B 00 08 0A 04 5B 81 1A 50 4D 31 52 01 50 4D  ......[..PM1R.PM
  03D0: 53 31 08 50 4D 53 32 08 50 4D 45 31 08 50 4D 45  S1.PMS2.PME1.PME
  03E0: 32 08 5B 80 47 50 53 54 01 0B 20 08 01 5B 81 0B  2.[.GPST.. ..[..
  03F0: 47 50 53 54 01 47 53 54 30 08 5B 80 47 4C 42 43  GPST.GST0.[.GLBC
  0400: 01 0B 28 08 0A 06 5B 81 24 47 4C 42 43 01 54 48  ..(...[.$GLBC.TH
  0410: 52 50 08 47 4C 42 54 08 45 4F 53 5F 08 4C 49 44  RP.GLBT.EOS_.LID
  0420: 50 08 54 48 4D 45 08 52 49 45 4E 08 5B 80 47 50  P.THME.RIEN.[.GP
  0430: 45 43 01 0B 42 08 01 5B 81 12 47 50 45 43 01 54  EC..B..[..GPEC.T
  0440: 48 50 4C 01 53 57 47 50 01 00 06 5B 80 50 4D 45  HPL.SWGP...[.PME
  0450: 53 01 0B 00 0C 01 5B 81 0B 50 4D 45 53 01 47 53  S.....[..PMES.GS
  0460: 54 53 08 5B 01 53 4D 49 4D 01 5B 01 53 4D 49 58  TS.[.SMIM.[.SMIX
  0470: 01 08 53 58 58 30 11 04 0B 00 01 08 53 58 58 31  ..SXX0......SXX1
  0480: 11 03 0A 08 8B 53 58 58 31 00 53 58 58 32 8B 53  .....SXX1.SXX2.S
  0490: 58 58 31 0A 04 53 58 58 33 14 2E 53 4D 49 5F 02  XX1..SXX3..SMI_.
  04A0: 5B 23 53 4D 49 4D FF FF 70 69 53 4D 49 44 70 68  [#SMIM..piSMIDph
  04B0: 53 4D 49 43 70 4D 47 49 43 60 70 53 4D 49 44 61  SMICpMGIC`pSMIDa
  04C0: 5B 27 53 4D 49 4D A4 61 14 28 53 4D 49 31 01 5B  ['SMIM.a.(SMI1.[
  04D0: 23 53 4D 49 4D FF FF 70 68 53 4D 49 43 70 4D 47  #SMIM..phSMICpMG
  04E0: 49 43 60 70 53 4D 49 44 60 5B 27 53 4D 49 4D A4  IC`pSMID`['SMIM.
  04F0: 60 14 37 53 4D 49 32 01 5B 23 53 4D 49 4D FF FF  `.7SMI2.[#SMIM..
  0500: 70 68 53 4D 49 43 70 4D 47 49 43 60 70 53 4D 49  phSMICpMGIC`pSMI
  0510: 43 61 70 53 4D 49 44 60 79 60 0A 08 60 72 61 60  CapSMID`y`..`ra`
  0520: 60 5B 27 53 4D 49 4D A4 60 14 23 53 4D 49 34 01  `['SMIM.`.#SMI4.
  0530: 72 68 01 60 70 53 4D 49 32 68 62 70 53 4D 49 32  rh.`pSMI2hbpSMI2
  0540: 60 61 79 61 0A 10 61 72 61 62 60 A4 60 14 0D 47  `aya..arab`.`..G
  0550: 54 4D 52 00 A4 53 4D 49 34 0A 78 14 0D 47 54 4D  TMR..SMI4.x..GTM
  0560: 4C 00 A4 53 4D 49 34 0A 74 14 0D 47 54 4D 48 00  L..SMI4.t..GTMH.
  0570: A4 53 4D 49 34 0A 76 14 0D 47 54 4F 4D 00 A4 53  .SMI4.v..GTOM..S
  0580: 4D 49 34 0A 81 14 17 47 55 53 42 00 A0 07 49 53  MI4....GUSB...IS
  0590: 4C 49 A4 00 A1 08 A4 53 4D 49 32 0A BA 14 0E 47  LI.....SMI2....G
  05A0: 54 50 4D 00 A4 53 4D 49 5F 0A 9B 00 14 14 53 58  TPM..SMI_.....SX
  05B0: 31 30 00 5B 23 53 4D 49 58 FF FF 70 00 53 58 58  10.[#SMIX..p.SXX
  05C0: 32 14 31 53 58 33 30 01 70 53 58 58 32 60 75 60  2.1SX30.pSXX2`u`
  05D0: A0 22 92 94 60 87 53 58 58 30 8C 53 58 58 30 53  ."..`.SXX0.SXX0S
  05E0: 58 58 32 53 58 32 30 70 68 53 58 32 30 70 60 53  XX2SX20phSX20p`S
  05F0: 58 58 32 14 1B 53 58 33 33 02 A0 14 95 69 87 68  XX2..SX33....i.h
  0600: 8C 68 69 53 58 32 30 53 58 33 30 53 58 32 30 14  .hiSX20SX30SX20.
  0610: 16 53 58 33 34 02 70 00 60 A2 0C 95 60 69 53 58  .SX34.p.`...`iSX
  0620: 33 33 68 60 75 60 14 29 53 58 58 36 02 70 69 53  33h`u`.)SXX6.piS
  0630: 4D 49 44 70 68 53 4D 49 43 70 53 4D 49 43 60 A2  MIDphSMICpSMIC`.
  0640: 0B 92 93 60 00 70 53 4D 49 43 60 A4 53 4D 49 44  ...`.pSMIC`.SMID
  0650: 14 1D 53 58 58 35 02 A0 16 95 69 87 68 8C 68 69  ..SXX5....i.h.hi
  0660: 53 58 32 30 53 58 58 36 0A 98 53 58 32 30 14 23  SX20SXX6..SX20.#
  0670: 53 58 58 34 00 53 58 58 36 0A 96 00 70 00 60 A2  SXX4.SXX6...p.`.
  0680: 12 95 60 53 58 58 32 53 58 58 35 53 58 58 30 60  ..`SXX2SXX5SXX0`
  0690: 75 60 14 1F 53 58 58 38 02 A0 18 95 69 87 68 8C  u`..SXX8....i.h.
  06A0: 68 69 53 58 32 30 70 53 58 58 36 0A 97 00 53 58  hiSX20pSXX6...SX
  06B0: 32 30 14 23 53 58 58 37 00 70 00 60 A2 19 95 60  20.#SXX7.p.`...`
  06C0: 53 58 58 33 72 53 58 58 32 60 61 53 58 58 38 53  SXX3rSXX2`aSXX8S
  06D0: 58 58 30 61 75 60 14 42 04 53 58 31 31 00 53 58  XX0au`.B.SX11.SX
  06E0: 58 34 70 53 58 58 36 0A 99 00 53 58 58 33 72 53  X4pSXX6...SXX3rS
  06F0: 58 58 32 53 58 58 33 60 A0 1C 95 87 53 58 58 30  XX2SXX3`....SXX0
  0700: 60 70 87 53 58 58 30 60 74 60 53 58 58 32 60 70  `p.SXX0`t`SXX2`p
  0710: 60 53 58 58 33 53 58 58 37 14 32 53 58 34 30 00  `SXX3SXX7.2SX40.
  0720: 70 53 58 58 32 60 75 60 A0 21 92 94 60 87 53 58  pSXX2`u`.!..`.SX
  0730: 58 30 8C 53 58 58 30 53 58 58 32 53 58 32 30 70  X0.SXX0SXX2SX20p
  0740: 60 53 58 58 32 A4 53 58 32 30 A4 00 14 35 53 58  `SXX2.SX20...5SX
  0750: 34 31 00 70 53 58 58 32 60 72 60 0A 02 60 A0 21  41.pSXX2`r`..`.!
  0760: 92 94 60 87 53 58 58 30 8B 53 58 58 30 53 58 58  ..`.SXX0.SXX0SXX
  0770: 32 53 58 32 31 70 60 53 58 58 32 A4 53 58 32 31  2SX21p`SXX2.SX21
  0780: A4 00 14 35 53 58 34 32 00 70 53 58 58 32 60 72  ...5SX42.pSXX2`r
  0790: 60 0A 04 60 A0 21 92 94 60 87 53 58 58 30 8A 53  `..`.!..`.SXX0.S
  07A0: 58 58 30 53 58 58 32 53 58 32 32 70 60 53 58 58  XX0SXX2SX22p`SXX
  07B0: 32 A4 53 58 32 32 A4 00 14 1C 53 58 34 33 02 A0  2.SX22....SX43..
  07C0: 15 95 69 87 68 8C 68 69 53 58 32 30 70 53 58 34  ..i.h.hiSX20pSX4
  07D0: 30 53 58 32 30 14 16 53 58 34 34 02 70 00 60 A2  0SX20..SX44.p.`.
  07E0: 0C 95 60 69 53 58 34 33 68 60 75 60 14 0C 53 58  ..`iSX43h`u`..SX
  07F0: 31 32 00 5B 27 53 4D 49 58 14 15 47 4D 49 30 00  12.['SMIX..GMI0.
  0800: 70 53 4D 49 32 0A D4 60 79 60 0A 10 60 A4 60 14  pSMI2..`y`..`.`.
  0810: 21 47 4D 58 30 00 70 53 4D 49 32 0A D5 60 79 60  !GMX0.pSMI2..`y`
  0820: 0A 10 60 A0 0B 92 93 60 00 7D 60 0B FF FF 60 A4  ..`....`.}`...`.
  0830: 60 14 15 47 4D 49 31 00 70 53 4D 49 32 0A D0 60  `..GMI1.pSMI2..`
  0840: 79 60 0A 10 60 A4 60 14 21 47 4D 58 31 00 70 53  y`..`.`.!GMX1.pS
  0850: 4D 49 32 0A D1 60 79 60 0A 10 60 A0 0B 92 93 60  MI2..`y`..`....`
  0860: 00 7D 60 0B FF FF 60 A4 60 14 16 47 49 42 30 00  .}`...`.`..GIB0.
  0870: 70 53 4D 49 32 0A D6 60 7B 60 0B FF FF 60 A4 60  pSMI2..`{`...`.`
  0880: 14 16 47 49 58 30 00 70 53 4D 49 32 0A D7 60 7B  ..GIX0.pSMI2..`{
  0890: 60 0B FF FF 60 A4 60 14 16 47 49 42 31 00 70 53  `...`.`..GIB1.pS
  08A0: 4D 49 32 0A D2 60 7B 60 0B FF FF 60 A4 60 14 16  MI2..`{`...`.`..
  08B0: 47 49 58 31 00 70 53 4D 49 32 0A D3 60 7B 60 0B  GIX1.pSMI2..`{`.
  08C0: FF FF 60 A4 60 10 4A 80 5F 53 42 5F 5B 82 42 80  ..`.`.J._SB_[.B.
  08D0: 41 4D 57 30 5B 01 57 4D 49 58 01 08 5F 48 49 44  AMW0[.WMIX.._HID
  08E0: 0D 2A 70 6E 70 30 63 31 34 00 08 5F 55 49 44 00  .*pnp0c14.._UID.
  08F0: 14 13 53 54 42 59 03 8C 68 69 54 4D 50 5F 70 6A  ..STBY..hiTMP_pj
  0900: 54 4D 50 5F 14 13 53 54 57 44 03 8B 68 69 54 4D  TMP_..STWD..hiTM
  0910: 50 5F 70 6A 54 4D 50 5F 14 13 53 54 44 57 03 8A  P_pjTMP_..STDW..
  0920: 68 69 54 4D 50 5F 70 6A 54 4D 50 5F 14 18 43 4C  hiTMP_pjTMP_..CL
  0930: 42 59 01 70 00 60 A2 0E 95 60 87 68 53 54 42 59  BY.p.`...`.hSTBY
  0940: 68 60 00 75 60 08 5F 57 44 47 11 48 06 0A 64 BC  h`.u`._WDG.H..d.
  0950: DC 9D 8D 97 A9 DA 11 B0 12 B6 22 A1 EF 54 92 41  .........."..T.A
  0960: 41 01 00 CE 93 05 A8 97 A9 DA 11 B0 12 B6 22 A1  A.............".
  0970: EF 54 92 42 41 01 02 94 59 BB 9D 97 A9 DA 11 B0  .T.BA...Y.......
  0980: 12 B6 22 A1 EF 54 92 D0 00 01 08 E0 6C 77 A3 88  .."..T......lw..
  0990: 1E DB 11 A9 8B 08 00 20 0C 9A 66 42 43 01 00 21  ....... ..fBC..!
  09A0: 12 90 05 66 D5 D1 11 B2 F0 00 A0 C9 06 29 10 4D  ...f.........).M
  09B0: 4F 01 00 08 49 4E 46 4F 11 03 0A 80 08 45 43 44  O...INFO.....ECD
  09C0: 30 00 14 0E 57 45 44 30 01 70 68 45 43 44 30 A4  0...WED0.phECD0.
  09D0: 00 14 08 57 43 41 41 01 A4 00 14 48 06 57 51 41  ...WCAA....H.WQA
  09E0: 41 01 5B 23 57 4D 49 58 FF FF 43 4C 42 59 49 4E  A.[#WMIX..CLBYIN
  09F0: 46 4F A0 0B 92 93 68 00 70 49 4E 46 4F 61 A1 3C  FO....h.pINFOa.<
  0A00: 53 54 44 57 49 4E 46 4F 00 0C 44 45 4C 4C 53 54  STDWINFO..DELLST
  0A10: 44 57 49 4E 46 4F 0A 04 0C 20 57 4D 49 53 54 44  DWINFO... WMISTD
  0A20: 57 49 4E 46 4F 0A 08 01 53 54 44 57 49 4E 46 4F  WINFO...STDWINFO
  0A30: 0A 0C 0B 00 10 70 49 4E 46 4F 61 5B 27 57 4D 49  .....pINFOa['WMI
  0A40: 58 A4 61 14 08 57 53 41 41 02 A4 69 14 28 57 4D  X.a..WSAA..i.(WM
  0A50: 42 41 03 8A 6A 0A 28 57 42 55 46 72 57 42 55 46  BA..j.(WBUFrWBUF
  0A60: 0A 2C 61 A0 0F 92 94 61 0B 00 10 70 57 4D 49 5F  .,a....a...pWMI_
  0A70: 6A 61 60 A4 60 08 57 4D 49 32 11 03 0A 80 08 57  ja`.`.WMI2.....W
  0A80: 4D 49 33 00 14 1C 57 4D 49 36 00 5B 23 57 4D 49  MI3...WMI6.[#WMI
  0A90: 58 FF FF 70 00 57 4D 49 33 43 4C 42 59 57 4D 49  X..p.WMI3CLBYWMI
  0AA0: 32 14 14 57 4D 49 37 00 5B 23 57 4D 49 58 FF FF  2..WMI7.[#WMIX..
  0AB0: 70 00 57 4D 49 33 14 34 57 4D 49 38 01 70 57 4D  p.WMI3.4WMI8.pWM
  0AC0: 49 33 60 72 60 0A 02 60 A0 22 92 94 60 87 57 4D  I3`r`..`."..`.WM
  0AD0: 49 32 8B 57 4D 49 32 57 4D 49 33 57 4D 49 39 70  I2.WMI2WMI3WMI9p
  0AE0: 68 57 4D 49 39 70 60 57 4D 49 33 14 4A 06 5F 57  hWMI9p`WMI3.J._W
  0AF0: 45 44 01 57 4D 49 36 A0 16 91 92 93 68 0A D0 93  ED.WMI6.....h...
  0B00: 45 43 44 30 00 57 4D 49 37 A4 57 4D 49 32 A1 3E  ECD0.WMI7.WMI2.>
  0B10: 53 58 31 30 53 58 33 30 01 53 58 31 31 70 53 58  SX10SX30.SX11pSX
  0B20: 34 31 60 A2 20 94 60 00 57 4D 49 38 60 A2 10 94  41`. .`.WMI8`...
  0B30: 60 00 57 4D 49 38 53 58 34 31 74 60 01 60 70 53  `.WMI8SX41t`.`pS
  0B40: 58 34 31 60 57 4D 49 38 00 53 58 31 32 57 4D 49  X41`WMI8.SX12WMI
  0B50: 37 A4 57 4D 49 32 08 57 51 4D 4F 11 44 57 0B 6F  7.WMI2.WQMO.DW.o
  0B60: 05 46 4F 4D 42 01 00 00 00 5F 05 00 00 88 1C 00  .FOMB...._......
  0B70: 00 44 53 00 01 1A 7D DA 54 18 D5 8D 00 01 06 18  .DS...}.T.......
  0B80: 42 10 0F 10 22 21 04 12 01 A1 C8 2C 0C 86 10 38  B..."!.....,...8
  0B90: 2E 84 1C 40 48 1C 14 4A 08 84 FA 13 C8 AF 00 84  ...@H..J........
  0BA0: 0E 05 C8 14 60 50 80 53 04 11 F4 2A C0 A6 00 93  ....`P.S...*....
  0BB0: 02 2C 0A D0 2E C0 B2 00 DD 02 A4 C3 12 91 E0 28  .,.............(
  0BC0: 31 E0 28 9D D8 C2 0D 1B BC 50 14 CD 20 4A 82 CA  1.(......P.. J..
  0BD0: 05 F8 46 10 78 B9 02 24 4F 40 9A 05 18 16 60 5D  ..F.x..$O@....`]
  0BE0: 80 EC 21 50 A9 43 40 C9 19 02 6A 00 AD 4E 40 F8  ..!P.C@...j..N@.
  0BF0: 95 4E 09 49 10 CE 58 C5 E3 6B 16 4D CF 49 CE 31  .N.I..X..k.M.I.1
  0C00: E4 78 5C E8 41 F0 40 0A 40 58 78 08 45 80 41 49  .x\.A.@.@Xx.E.AI
  0C10: 18 0B 75 31 6A D4 48 D9 80 0C 51 DA A8 D1 03 3A  ..u1j.H...Q....:
  0C20: BF 23 39 BB A3 3B 92 04 46 3D A6 63 2C 6C 46 42  .#9..;..F=.c,lFB
  0C30: 8D D1 1C 14 81 C6 0D DA 12 61 35 AE D8 67 66 E1  .........a5..gf.
  0C40: C3 12 C6 11 1C 58 82 46 D1 34 C7 B3 0D 91 E0 20  .....X.F.4..... 
  0C50: 42 63 64 40 C8 F3 B0 05 7A E4 09 EC 1E 51 0A 11  Bcd@....z....Q..
  0C60: 34 DF 13 A9 51 80 36 0C D9 3A 1B 68 A8 B1 1A 43  4...Q.6..:.h...C
  0C70: 11 44 84 A0 51 0C 16 21 54 88 FF 7F 94 A8 A7 14  .D..Q..!T.......
  0C80: 24 6A 65 20 42 0B 66 04 66 7F 10 24 C6 99 41 87  $je B.f.f..$..A.
  0C90: 05 CB 00 91 11 41 A3 61 67 01 0F C7 33 69 7E 62  .....A.ag...3i~b
  0CA0: 1A 9C 09 C6 86 90 06 08 89 3A 38 50 02 4B 19 38  .........:8P.K.8
  0CB0: B1 3D 2E 8D EF 8C A3 86 38 F5 33 F3 3F C2 5B F0  .=......8.3.?.[.
  0CC0: 11 80 8F C1 83 3D 84 80 47 C8 CE 00 06 C4 7B 9F  .....=..G.....{.
  0CD0: 34 99 8B CF 02 30 86 0F D7 F8 28 34 1E 76 3E 60  4....0....(4.v>`
  0CE0: E3 E2 F0 3E 14 9C 70 B1 20 0A 00 21 59 E7 03 F4  ...>..p. ..!Y...
  0CF0: AC 8F 2D E0 C3 40 B3 77 08 42 F0 22 E0 A3 83 8F  ..-..@.w.B."....
  0D00: 1B 1E F7 F3 06 18 0E 07 1E 8E 4F 1B C0 65 04 5C  ..........O..e.\
  0D10: DA 93 C2 04 92 FC 04 90 18 18 D4 81 C0 07 0B B8  ................
  0D20: 92 E0 50 C3 F3 C4 1E 10 FE FF 47 79 22 2F 06 9E  ..P.......Gy"/..
  0D30: FE 63 00 8C 03 82 A7 75 52 BE 79 3C 48 78 50 61  .c.....uR.y<HxPa
  0D40: 12 F8 94 C0 D0 F8 71 03 AC A3 C6 1F 10 E0 9D 24  ......q........$
  0D50: CE AF CF 01 E8 D0 70 8A 0C E4 35 E0 A4 4F C9 E3  ......p...5..O..
  0D60: 4B E0 33 07 EC BB C1 61 1C 4C 88 08 EF 01 4F 1D  K.3....a.L....O.
  0D70: BE 6B 3C 0A 04 8A D0 DB 99 83 9E 42 8C 12 ED AC  .k<........B....
  0D80: C2 3C 70 44 F1 91 C3 08 EF 1E BE 13 3C 80 B4 36  .<pD........<..6
  0D90: 39 E1 06 7A E6 60 D1 CE 2C B2 00 A2 48 A3 41 9D  9..z.`..,...H.A.
  0DA0: 11 7C 1A F0 B4 9E 62 7C 94 30 C8 19 1E D8 73 C2  .|....b|.0....s.
  0DB0: 63 80 07 CC EE 07 3E 4E F8 5C 80 77 0D A8 19 FA  c.....>N.\.w....
  0DC0: B0 01 E7 D0 81 3F 4D E0 0F 16 F8 F1 F8 9A C3 26  .....?M........&
  0DD0: 9C C0 F2 07 81 1A 99 A1 3D CB D3 7A 0D F0 69 C7  ........=..z..i.
  0DE0: 04 3E 6F F8 FF FF CF F1 78 C0 AF F8 74 41 EE 0A  .>o.....x...tA..
  0DF0: 9E AF CF 2E CC C6 78 50 A3 F0 01 07 77 76 F1 11  ......xP....wv..
  0E00: C0 67 17 E0 39 89 67 09 F0 1E 02 7C 22 89 F7 B0  .g..9.g....|"...
  0E10: 05 63 C4 78 C8 33 AE 7A 18 BA 08 58 DD 7D 05 75  .c.x.3.z...X.}.u
  0E20: F4 02 13 D4 6B 06 EE F4 02 7C 4E 59 F0 FE FF A7  ....k....|NY....
  0E30: 2C E0 7E 55 E0 47 14 30 40 76 76 3A 11 C2 7B C9  ,.~U.G.0@vv:..{.
  0E40: 73 88 6F 57 3E 98 04 79 0E 88 F0 94 C5 EF 03 51  s.oW>..y.......Q
  0E50: 62 1E 50 A4 28 46 0C F2 84 E5 EB 49 0C 43 07 0B  b.P.(F.....I.C..
  0E60: 17 3E C2 53 16 60 F1 92 85 39 65 C1 7C 1B F8 94  .>.S.`...9e.|...
  0E70: 05 8E FF FF 29 0B 5C E3 7E CA 02 66 D2 9F 02 3E  ....).\.~..f...>
  0E80: D5 F8 09 A0 E8 07 0B 0A E3 53 16 E0 4A DE 01 01  .........S..J...
  0E90: 34 67 27 DF 16 0C 76 CC BE 64 F8 94 08 86 43 86  4g'...v..d....C.
  0EA0: EF 54 87 F2 C8 F1 14 E0 23 16 D8 E3 1C 03 74 5C  .T......#.....t\
  0EB0: F1 11 CB FF FF 23 16 C0 8D 03 08 FE D4 01 EB 1E  .....#..........
  0EC0: 10 D6 87 0E E0 21 FB 21 A0 33 8C 25 83 C8 C6 B9  .....!.!.3.%....
  0ED0: 86 8E D1 E2 17 AA 9B 42 EC 83 E1 B2 81 04 EA E1  .......B........
  0EE0: 5A 30 85 44 D1 68 34 06 26 30 82 33 88 01 9D 11  Z0.D.h4.&0.3....
  0EF0: 42 87 32 9C 8A F3 10 EA FF 4F 30 D4 8D 89 CE CE  B.2......O0.....
  0F00: F3 E7 B7 11 9F 08 0C EC 2B C4 5B 06 58 86 E5 C5  ........+.[.X...
  0F10: 3D 01 1C E3 49 26 A8 E6 58 83 9A 83 AF 02 6F 64  =...I&..X.....od
  0F20: 26 F0 15 0C 6C A7 19 8C BE 3B 01 28 80 7C 14 F0  &...l....;.(.|..
  0F30: 7D F9 6D 80 CD E2 95 D9 68 3E 7F 22 86 8E 12 33  }.m.....h>."...3
  0F40: 74 0A E2 A1 3B E8 D0 D1 C7 01 9F AC 70 C1 0E 5F  t...;.......p.._
  0F50: D0 26 77 B4 27 E6 59 78 9E B8 B9 83 E9 88 04 63  .&w.'.Yx.......c
  0F60: F0 98 C9 83 59 E0 E4 41 F1 FF 9F 3C 4C 78 4C D8  ....Y..A...<LxL.
  0F70: C3 21 3D 74 78 64 7C 9C 3E FD 30 EC D3 39 97 A2  .!=txd|.>.0..9..
  0F80: 67 A4 3B C6 33 17 06 D6 23 E7 B0 46 0B 7B C0 CF  g.;.3...#..F.{..
  0F90: 21 BE C3 F8 C0 C3 60 7D 7A 01 C7 F1 0B FE 69 00  !.....`}z.....i.
  0FA0: 3C 07 10 8F E1 05 84 1F 5F 74 FE A5 42 17 27 79  <......._t..B.'y
  0FB0: 30 A8 D3 14 E0 EA F4 06 9E AB 3E EE 2C 85 FB FF  0.........>.,...
  0FC0: 9F A5 60 1C 7E 7D 81 F7 5D CA 08 07 F2 2C 05 F6  ..`.~}..]....,..
  0FD0: D8 CF 10 1D 02 7C 96 02 F8 F1 FA F0 79 06 6E E0  .....|......y.n.
  0FE0: D3 2F D0 FA FF 9F 42 C0 7F 8C F0 11 07 77 F4 05  ./....B......w..
  0FF0: 6E 07 63 7E C6 C0 1D C7 80 C7 C9 19 77 9A F0 10  n.c~........w...
  1000: F8 00 5A 9D 1E 39 F5 9C 12 EE 38 C0 A7 84 1B 00  ..Z..9....8.....
  1010: 26 C0 19 13 50 F5 FF 3F 63 02 63 90 08 EF 0A 51  &...P..?c.c....Q
  1020: DF 91 0D F1 8C CC 70 DE 1A 1E 5F 9E 91 C1 79 C6  ......p..._...y.
  1030: 04 F8 F3 FF 3F D2 63 6F 0A C1 9F 42 80 89 F8 33  ....?.co...B...3
  1040: 26 15 7E C6 84 36 17 4F DB 67 4C 80 7B FF FF 33  &.~..6.O.gL.{..3
  1050: 26 E0 3B F2 19 13 D0 73 A4 C2 9D 31 C1 30 CE F7  &.;....s...1.0..
  1060: 27 83 3C 8F F8 40 E0 A3 0C 53 68 D3 A7 46 A3 56  '.<..@...Sh..F.V
  1070: 0D CA D4 28 D3 A0 56 9F 4A 8D 19 3B 3A 59 C4 5A  ...(..V.J..;:Y.Z
  1080: 35 58 87 5A AF 40 2C E9 89 21 10 FF FF 45 79 00  5X.Z.@,..!...Ey.
  1090: 61 71 4D 80 30 E1 AB 12 88 63 83 50 B1 3A 20 1A  aqM.0....c.P.: .
  10A0: 19 A2 41 04 E4 10 3E 80 58 24 10 81 13 25 A0 D4  ..A...>.X$...%..
  10B0: 09 08 13 BA 16 81 58 9E 17 10 16 0E 84 4A 32 03  ......X......J2.
  10C0: CE F4 81 08 C8 52 5F 21 02 B2 70 10 01 F9 FF 0F  .....R_!..p.....
  10D0: 5B 01 57 4D 49 53 01 08 57 4D 30 30 00 14 20 57  [.WMIS..WM00.. W
  10E0: 4D 30 32 00 53 58 31 30 53 58 33 30 00 53 58 31  M02.SX10SX30.SX1
  10F0: 31 70 53 58 34 32 57 4D 30 30 53 58 31 32 14 2C  1pSX42WM00SX12.,
  1100: 57 4D 30 33 03 5B 80 57 57 50 52 00 68 01 5B 81  WM03.[.WWPR.h.[.
  1110: 0B 57 57 50 52 11 4D 45 4D 57 08 8C 69 6A 57 56  .WWPR.MEMW..ijWV
  1120: 41 4C 70 57 56 41 4C 4D 45 4D 57 14 2D 57 4D 30  ALpWVALMEMW.-WM0
  1130: 35 03 5B 80 57 57 50 52 00 68 0A 04 5B 81 0B 57  5.[.WWPR.h..[..W
  1140: 57 50 52 11 4D 57 33 32 20 8A 69 6A 57 56 41 4C  WPR.MW32 .ijWVAL
  1150: 70 57 56 41 4C 4D 57 33 32 14 32 57 4D 30 34 03  pWVALMW32.2WM04.
  1160: 5B 80 57 52 50 52 00 68 01 5B 81 0B 57 52 50 52  [.WRPR.h.[..WRPR
  1170: 11 4D 45 4D 52 08 8C 69 6A 57 56 41 4C 70 4D 45  .MEMR..ijWVALpME
  1180: 4D 52 57 56 41 4C 70 00 4D 45 4D 52 14 33 57 4D  MRWVALp.MEMR.3WM
  1190: 30 36 03 5B 80 57 52 50 52 00 68 0A 04 5B 81 0B  06.[.WRPR.h..[..
  11A0: 57 52 50 52 11 4D 52 33 32 20 8A 69 6A 57 56 41  WRPR.MR32 .ijWVA
  11B0: 4C 70 4D 52 33 32 57 56 41 4C 70 00 4D 52 33 32  LpMR32WVALp.MR32
  11C0: 14 47 05 57 4D 30 37 02 A0 4D 04 92 94 69 0B 00  .G.WM07..M...i..
  11D0: 01 70 57 4D 30 30 60 70 00 61 A2 15 95 61 69 57  .pWM00`p.a...aiW
  11E0: 4D 30 35 60 68 61 72 60 0A 04 60 72 61 0A 04 61  M05`har`..`ra..a
  11F0: 53 58 58 36 0A 9A 00 70 57 4D 30 30 60 70 00 61  SXX6...pWM00`p.a
  1200: A2 15 95 61 69 57 4D 30 36 60 68 61 72 60 0A 04  ...aiWM06`har`..
  1210: 60 72 61 0A 04 61 A4 68 14 28 57 4D 49 5F 02 5B  `ra..a.h.(WMI_.[
  1220: 23 57 4D 49 53 FF FF A0 0B 93 57 4D 30 30 00 57  #WMIS.....WM00.W
  1230: 4D 30 32 57 4D 30 37 68 69 5B 27 57 4D 49 53 A4  M02WM07hi['WMIS.
  1240: 68 10 46 1A 5F 50 52 5F 5B 83 0B 43 50 55 31 01  h.F._PR_[..CPU1.
  1250: 10 08 00 00 06 5B 83 0B 43 50 55 32 02 00 00 00  .....[..CPU2....
  1260: 00 00 5B 83 0B 43 50 55 33 03 00 00 00 00 00 5B  ..[..CPU3......[
  1270: 83 0B 43 50 55 34 04 00 00 00 00 00 5B 83 0B 43  ..CPU4......[..C
  1280: 50 55 35 05 00 00 00 00 00 5B 83 0B 43 50 55 36  PU5......[..CPU6
  1290: 06 00 00 00 00 00 5B 83 0B 43 50 55 37 07 00 00  ......[..CPU7...
  12A0: 00 00 00 5B 83 0B 43 50 55 38 08 00 00 00 00 00  ...[..CPU8......
  12B0: 5B 83 0B 43 50 55 39 09 00 00 00 00 00 5B 83 0B  [..CPU9......[..
  12C0: 43 50 31 30 0A 00 00 00 00 00 5B 83 0B 43 50 31  CP10......[..CP1
  12D0: 31 0B 00 00 00 00 00 5B 83 0B 43 50 31 32 0C 00  1......[..CP12..
  12E0: 00 00 00 00 5B 83 0B 43 50 31 33 0D 00 00 00 00  ....[..CP13.....
  12F0: 00 5B 83 0B 43 50 31 34 0E 00 00 00 00 00 5B 83  .[..CP14......[.
  1300: 0B 43 50 31 35 0F 00 00 00 00 00 5B 83 0B 43 50  .CP15......[..CP
  1310: 31 36 10 00 00 00 00 00 5B 83 0B 43 50 31 37 11  16......[..CP17.
  1320: 00 00 00 00 00 5B 83 0B 43 50 31 38 12 00 00 00  .....[..CP18....
  1330: 00 00 5B 83 0B 43 50 31 39 13 00 00 00 00 00 5B  ..[..CP19......[
  1340: 83 0B 43 50 32 30 14 00 00 00 00 00 5B 83 0B 43  ..CP20......[..C
  1350: 50 32 31 15 00 00 00 00 00 5B 83 0B 43 50 32 32  P21......[..CP22
  1360: 16 00 00 00 00 00 5B 83 0B 43 50 32 33 17 00 00  ......[..CP23...
  1370: 00 00 00 5B 83 0B 43 50 32 34 18 00 00 00 00 00  ...[..CP24......
  1380: 5B 83 0B 43 50 32 35 19 00 00 00 00 00 5B 83 0B  [..CP25......[..
  1390: 43 50 32 36 1A 00 00 00 00 00 5B 83 0B 43 50 32  CP26......[..CP2
  13A0: 37 1B 00 00 00 00 00 5B 83 0B 43 50 32 38 1C 00  7......[..CP28..
  13B0: 00 00 00 00 5B 83 0B 43 50 32 39 1D 00 00 00 00  ....[..CP29.....
  13C0: 00 5B 83 0B 43 50 33 30 1E 00 00 00 00 00 5B 83  .[..CP30......[.
  13D0: 0B 43 50 33 31 1F 00 00 00 00 00 5B 83 0B 43 50  .CP31......[..CP
  13E0: 33 32 20 00 00 00 00 00 10 15 5C 00 08 53 53 54  32 .......\..SST
  13F0: 58 00 14 0B 5F 50 54 53 01 50 53 4B 4D 68 14 29  X..._PTS.PSKMh.)
  1400: 2E 5F 47 50 45 5F 4C 30 32 00 86 5C 2E 5F 53 42  ._GPE_L02..\._SB
  1410: 5F 41 4D 57 30 0A D0 70 47 53 54 30 60 7B 60 0A  _AMW0..pGST0`{`.
  1420: 04 60 70 60 47 53 54 30 14 3C 2E 5F 47 50 45 5F  .`p`GST0.<._GPE_
  1430: 4C 30 38 00 47 50 4B 4D A0 16 80 93 53 53 54 58  L08.GPKM....SSTX
  1440: 01 00 86 5C 2E 5F 53 42 5F 56 42 54 4E 0A 02 86  ...\._SB_VBTN...
  1450: 5C 2F 04 5F 53 42 5F 50 43 49 30 49 53 41 5F 4B  \/._SB_PCI0ISA_K
  1460: 42 44 5F 0A 02 14 49 04 2E 5F 47 50 45 5F 4C 31  BD_...I.._GPE_L1
  1470: 38 00 7F 4C 49 44 50 01 4C 49 44 50 7D 50 4D 45  8..LIDP.LIDP}PME
  1480: 32 01 50 4D 45 32 A0 28 93 5C 2F 04 5F 53 42 5F  2.PME2.(.\/._SB_
  1490: 50 43 49 30 50 43 49 31 58 4D 45 31 01 86 5C 2F  PCI0PCI1XME1..\/
  14A0: 03 5F 53 42 5F 50 43 49 30 50 43 49 31 0A 02 14  ._SB_PCI0PCI1...
  14B0: 32 2E 5F 47 50 45 5F 4C 31 42 00 7F 4C 49 44 50  2._GPE_L1B..LIDP
  14C0: 0A 08 4C 49 44 50 7D 50 4D 45 32 01 50 4D 45 32  ..LIDP}PME2.PME2
  14D0: 86 5C 2F 03 5F 53 42 5F 50 43 49 30 50 43 49 35  .\/._SB_PCI0PCI5
  14E0: 0A 02 14 45 07 2E 5F 47 50 45 5F 4C 31 43 00 7F  ...E.._GPE_L1C..
  14F0: 4C 49 44 50 0A 08 4C 49 44 50 7D 50 4D 45 32 01  LIDP..LIDP}PME2.
  1500: 50 4D 45 32 A0 28 93 5C 2F 04 5F 53 42 5F 50 43  PME2.(.\/._SB_PC
  1510: 49 30 50 43 49 32 58 4D 45 32 01 86 5C 2F 03 5F  I0PCI2XME2..\/._
  1520: 53 42 5F 50 43 49 30 50 43 49 32 0A 02 A1 2A A0  SB_PCI0PCI2...*.
  1530: 28 93 5C 2F 04 5F 53 42 5F 50 43 49 30 50 43 49  (.\/._SB_PCI0PCI
  1540: 33 58 4D 45 33 01 86 5C 2F 03 5F 53 42 5F 50 43  3XME3..\/._SB_PC
  1550: 49 30 50 43 49 33 0A 02 14 16 2E 5F 47 50 45 5F  I0PCI3....._GPE_
  1560: 4C 31 44 00 7F 4C 49 44 50 0A 20 4C 49 44 50 14  L1D..LIDP. LIDP.
  1570: 45 07 2E 5F 47 50 45 5F 4C 31 46 00 7F 4C 49 44  E.._GPE_L1F..LID
  1580: 50 0A 08 4C 49 44 50 7D 50 4D 45 32 01 50 4D 45  P..LIDP}PME2.PME
  1590: 32 A0 28 93 5C 2F 04 5F 53 42 5F 50 43 49 37 50  2.(.\/._SB_PCI7P
  15A0: 43 49 42 58 4D 45 42 01 86 5C 2F 03 5F 53 42 5F  CIBXMEB..\/._SB_
  15B0: 50 43 49 37 50 43 49 42 0A 02 A1 2A A0 28 93 5C  PCI7PCIB...*.(.\
  15C0: 2F 04 5F 53 42 5F 50 43 49 37 50 43 49 41 58 4D  /._SB_PCI7PCIAXM
  15D0: 45 41 01 86 5C 2F 03 5F 53 42 5F 50 43 49 37 50  EA..\/._SB_PCI7P
  15E0: 43 49 41 0A 02 14 43 07 5F 57 41 4B 01 A0 09 93  CIA...C._WAK....
  15F0: 68 0A 04 47 55 53 42 A1 17 A0 15 93 68 0A 03 A0  h..GUSB.....h...
  1600: 0F 7B 0A 20 43 4D 52 44 0A 49 00 47 55 53 42 43  .{. CMRD.I.GUSBC
  1610: 4D 57 52 0A 49 7B 0A DF 43 4D 52 44 0A 49 00 A0  MWR.I{..CMRD.I..
  1620: 37 7D 7B 50 4D 53 32 01 00 7D 7B 92 48 41 43 4B  7}{PMS2..}{.HACK
  1630: 7B 47 4C 42 54 0A 11 00 00 7B 92 48 41 43 4B 7B  {GLBT....{.HACK{
  1640: 54 48 52 50 0A 18 00 00 00 00 86 5C 2E 5F 53 42  THRP.......\._SB
  1650: 5F 56 42 54 4E 0A 02 A4 00 10 33 5F 53 49 5F 14  _VBTN.....3_SI_.
  1660: 2D 5F 53 53 54 01 70 68 53 53 54 58 A0 10 93 68  -_SST.phSSTX...h
  1670: 0A 03 7B 4C 45 44 32 0A FD 4C 45 44 32 A0 0F 93  ..{LED2..LED2...
  1680: 68 01 7D 4C 45 44 32 0A 02 4C 45 44 32 10 81 79  h.}LED2..LED2..y
  1690: 01 5F 53 42 5F 5B 82 43 CC 50 43 49 30 08 5F 48  ._SB_[.C.PCI0._H
  16A0: 49 44 0C 41 D0 0A 03 08 5F 55 49 44 0A 04 08 5F  ID.A...._UID..._
  16B0: 42 42 4E 00 08 5F 41 44 52 00 08 5F 50 52 57 12  BBN.._ADR.._PRW.
  16C0: 06 02 0A 0D 0A 05 14 08 5F 53 31 44 00 A4 01 14  ........_S1D....
  16D0: 14 5F 53 33 44 00 A0 08 48 41 43 4B A4 0A 03 A1  ._S3D...HACK....
  16E0: 04 A4 0A 02 08 50 49 43 30 12 4A 39 40 12 0B 04  .....PIC0.J9@...
  16F0: 0B FF FF 00 4C 4E 4B 41 00 12 0B 04 0B FF FF 01  ....LNKA........
  1700: 4C 4E 4B 42 00 12 0C 04 0B FF FF 0A 02 4C 4E 4B  LNKB.........LNK
  1710: 43 00 12 0C 04 0B FF FF 0A 03 4C 4E 4B 44 00 12  C.........LNKD..
  1720: 0D 04 0C FF FF 01 00 00 4C 4E 4B 41 00 12 0D 04  ........LNKA....
  1730: 0C FF FF 01 00 01 4C 4E 4B 42 00 12 0E 04 0C FF  ......LNKB......
  1740: FF 01 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF  .....LNKC.......
  1750: 01 00 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 02  ....LNKD........
  1760: 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 02 00 01  ..LNKA..........
  1770: 4C 4E 4B 42 00 12 0E 04 0C FF FF 02 00 0A 02 4C  LNKB...........L
  1780: 4E 4B 43 00 12 0E 04 0C FF FF 02 00 0A 03 4C 4E  NKC...........LN
  1790: 4B 44 00 12 0D 04 0C FF FF 03 00 00 4C 4E 4B 41  KD..........LNKA
  17A0: 00 12 0D 04 0C FF FF 03 00 01 4C 4E 4B 42 00 12  ..........LNKB..
  17B0: 0E 04 0C FF FF 03 00 0A 02 4C 4E 4B 43 00 12 0E  .........LNKC...
  17C0: 04 0C FF FF 03 00 0A 03 4C 4E 4B 44 00 12 0D 04  ........LNKD....
  17D0: 0C FF FF 04 00 00 4C 4E 4B 41 00 12 0D 04 0C FF  ......LNKA......
  17E0: FF 04 00 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 04  ....LNKB........
  17F0: 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF 04 00  ...LNKC.........
  1800: 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 05 00 00  ..LNKD..........
  1810: 4C 4E 4B 41 00 12 0D 04 0C FF FF 05 00 01 4C 4E  LNKA..........LN
  1820: 4B 42 00 12 0E 04 0C FF FF 05 00 0A 02 4C 4E 4B  KB...........LNK
  1830: 43 00 12 0E 04 0C FF FF 05 00 0A 03 4C 4E 4B 44  C...........LNKD
  1840: 00 12 0D 04 0C FF FF 06 00 00 4C 4E 4B 41 00 12  ..........LNKA..
  1850: 0D 04 0C FF FF 06 00 01 4C 4E 4B 42 00 12 0E 04  ........LNKB....
  1860: 0C FF FF 06 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C  .......LNKC.....
  1870: FF FF 06 00 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF  ......LNKD......
  1880: FF 07 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 07  ....LNKA........
  1890: 00 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 07 00 0A  ..LNKB..........
  18A0: 02 4C 4E 4B 43 00 12 0E 04 0C FF FF 07 00 0A 03  .LNKC...........
  18B0: 4C 4E 4B 44 00 12 0D 04 0C FF FF 08 00 00 4C 4E  LNKD..........LN
  18C0: 4B 41 00 12 0D 04 0C FF FF 08 00 01 4C 4E 4B 42  KA..........LNKB
  18D0: 00 12 0E 04 0C FF FF 08 00 0A 02 4C 4E 4B 43 00  ...........LNKC.
  18E0: 12 0E 04 0C FF FF 08 00 0A 03 4C 4E 4B 44 00 12  ..........LNKD..
  18F0: 0D 04 0C FF FF 09 00 00 4C 4E 4B 41 00 12 0D 04  ........LNKA....
  1900: 0C FF FF 09 00 01 4C 4E 4B 42 00 12 0E 04 0C FF  ......LNKB......
  1910: FF 09 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF  .....LNKC.......
  1920: 09 00 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 0A  ....LNKD........
  1930: 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 0A 00 01  ..LNKA..........
  1940: 4C 4E 4B 42 00 12 0E 04 0C FF FF 0A 00 0A 02 4C  LNKB...........L
  1950: 4E 4B 43 00 12 0E 04 0C FF FF 0A 00 0A 03 4C 4E  NKC...........LN
  1960: 4B 44 00 12 0D 04 0C FF FF 16 00 00 4C 4E 4B 41  KD..........LNKA
  1970: 00 12 0D 04 0C FF FF 16 00 01 4C 4E 4B 42 00 12  ..........LNKB..
  1980: 0E 04 0C FF FF 16 00 0A 02 4C 4E 4B 47 00 12 0E  .........LNKG...
  1990: 04 0C FF FF 16 00 0A 03 4C 4E 4B 44 00 12 0D 04  ........LNKD....
  19A0: 0C FF FF 1A 00 00 4C 4E 4B 41 00 12 0D 04 0C FF  ......LNKA......
  19B0: FF 1A 00 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 1A  ....LNKB........
  19C0: 00 0A 02 4C 4E 4B 47 00 12 0E 04 0C FF FF 1A 00  ...LNKG.........
  19D0: 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 1B 00 00  ..LNKD..........
  19E0: 4C 4E 4B 41 00 12 0D 04 0C FF FF 1C 00 00 4C 4E  LNKA..........LN
  19F0: 4B 41 00 12 0D 04 0C FF FF 1C 00 01 4C 4E 4B 42  KA..........LNKB
  1A00: 00 12 0E 04 0C FF FF 1C 00 0A 02 4C 4E 4B 43 00  ...........LNKC.
  1A10: 12 0E 04 0C FF FF 1C 00 0A 03 4C 4E 4B 44 00 12  ..........LNKD..
  1A20: 0D 04 0C FF FF 1D 00 00 4C 4E 4B 48 00 12 0D 04  ........LNKH....
  1A30: 0C FF FF 1D 00 01 4C 4E 4B 42 00 12 0E 04 0C FF  ......LNKB......
  1A40: FF 1D 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF  .....LNKC.......
  1A50: 1D 00 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 1F  ....LNKD........
  1A60: 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 1F 00 01  ..LNKA..........
  1A70: 4C 4E 4B 42 00 12 0E 04 0C FF FF 1F 00 0A 02 4C  LNKB...........L
  1A80: 4E 4B 45 00 08 41 50 49 30 12 4A 31 40 12 09 04  NKE..API0.J1@...
  1A90: 0B FF FF 00 00 0A 10 12 09 04 0B FF FF 01 00 0A  ................
  1AA0: 11 12 0A 04 0B FF FF 0A 02 00 0A 12 12 0A 04 0B  ................
  1AB0: FF FF 0A 03 00 0A 13 12 0B 04 0C FF FF 01 00 00  ................
  1AC0: 00 0A 10 12 0B 04 0C FF FF 01 00 01 00 0A 11 12  ................
  1AD0: 0C 04 0C FF FF 01 00 0A 02 00 0A 12 12 0C 04 0C  ................
  1AE0: FF FF 01 00 0A 03 00 0A 13 12 0B 04 0C FF FF 02  ................
  1AF0: 00 00 00 0A 10 12 0B 04 0C FF FF 02 00 01 00 0A  ................
  1B00: 11 12 0C 04 0C FF FF 02 00 0A 02 00 0A 12 12 0C  ................
  1B10: 04 0C FF FF 02 00 0A 03 00 0A 13 12 0B 04 0C FF  ................
  1B20: FF 03 00 00 00 0A 10 12 0B 04 0C FF FF 03 00 01  ................
  1B30: 00 0A 11 12 0C 04 0C FF FF 03 00 0A 02 00 0A 12  ................
  1B40: 12 0C 04 0C FF FF 03 00 0A 03 00 0A 13 12 0B 04  ................
  1B50: 0C FF FF 04 00 00 00 0A 10 12 0B 04 0C FF FF 04  ................
  1B60: 00 01 00 0A 11 12 0C 04 0C FF FF 04 00 0A 02 00  ................
  1B70: 0A 12 12 0C 04 0C FF FF 04 00 0A 03 00 0A 13 12  ................
  1B80: 0B 04 0C FF FF 05 00 00 00 0A 10 12 0B 04 0C FF  ................
  1B90: FF 05 00 01 00 0A 11 12 0C 04 0C FF FF 05 00 0A  ................
  1BA0: 02 00 0A 12 12 0C 04 0C FF FF 05 00 0A 03 00 0A  ................
  1BB0: 13 12 0B 04 0C FF FF 06 00 00 00 0A 10 12 0B 04  ................
  1BC0: 0C FF FF 06 00 01 00 0A 11 12 0C 04 0C FF FF 06  ................
  1BD0: 00 0A 02 00 0A 12 12 0C 04 0C FF FF 06 00 0A 03  ................
  1BE0: 00 0A 13 12 0B 04 0C FF FF 07 00 00 00 0A 10 12  ................
  1BF0: 0B 04 0C FF FF 07 00 01 00 0A 11 12 0C 04 0C FF  ................
  1C00: FF 07 00 0A 02 00 0A 12 12 0C 04 0C FF FF 07 00  ................
  1C10: 0A 03 00 0A 13 12 0B 04 0C FF FF 08 00 00 00 0A  ................
  1C20: 10 12 0B 04 0C FF FF 08 00 01 00 0A 11 12 0C 04  ................
  1C30: 0C FF FF 08 00 0A 02 00 0A 12 12 0C 04 0C FF FF  ................
  1C40: 08 00 0A 03 00 0A 13 12 0B 04 0C FF FF 09 00 00  ................
  1C50: 00 0A 10 12 0B 04 0C FF FF 09 00 01 00 0A 11 12  ................
  1C60: 0C 04 0C FF FF 09 00 0A 02 00 0A 12 12 0C 04 0C  ................
  1C70: FF FF 09 00 0A 03 00 0A 13 12 0B 04 0C FF FF 0A  ................
  1C80: 00 00 00 0A 10 12 0B 04 0C FF FF 0A 00 01 00 0A  ................
  1C90: 11 12 0C 04 0C FF FF 0A 00 0A 02 00 0A 12 12 0C  ................
  1CA0: 04 0C FF FF 0A 00 0A 03 00 0A 13 12 0B 04 0C FF  ................
  1CB0: FF 16 00 00 00 0A 10 12 0B 04 0C FF FF 16 00 01  ................
  1CC0: 00 0A 11 12 0C 04 0C FF FF 16 00 0A 02 00 0A 12  ................
  1CD0: 12 0C 04 0C FF FF 16 00 0A 03 00 0A 13 12 0B 04  ................
  1CE0: 0C FF FF 1A 00 00 00 0A 10 12 0B 04 0C FF FF 1A  ................
  1CF0: 00 01 00 0A 11 12 0C 04 0C FF FF 1A 00 0A 02 00  ................
  1D00: 0A 16 12 0C 04 0C FF FF 1A 00 0A 03 00 0A 13 12  ................
  1D10: 0B 04 0C FF FF 1B 00 00 00 0A 10 12 0B 04 0C FF  ................
  1D20: FF 1C 00 00 00 0A 10 12 0B 04 0C FF FF 1C 00 01  ................
  1D30: 00 0A 11 12 0C 04 0C FF FF 1C 00 0A 02 00 0A 12  ................
  1D40: 12 0C 04 0C FF FF 1C 00 0A 03 00 0A 13 12 0B 04  ................
  1D50: 0C FF FF 1D 00 00 00 0A 17 12 0B 04 0C FF FF 1D  ................
  1D60: 00 01 00 0A 11 12 0C 04 0C FF FF 1D 00 0A 02 00  ................
  1D70: 0A 12 12 0C 04 0C FF FF 1D 00 0A 03 00 0A 13 12  ................
  1D80: 0B 04 0C FF FF 1F 00 00 00 0A 10 12 0B 04 0C FF  ................
  1D90: FF 1F 00 01 00 0A 11 12 0C 04 0C FF FF 1F 00 0A  ................
  1DA0: 02 00 0A 14 5B 82 40 13 50 43 49 31 14 08 5F 53  ....[.@.PCI1.._S
  1DB0: 31 44 00 A4 01 14 14 5F 53 33 44 00 A0 08 48 41  1D....._S3D...HA
  1DC0: 43 4B A4 0A 03 A1 04 A4 0A 02 14 14 5F 53 34 44  CK.........._S4D
  1DD0: 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14  ...HACK.........
  1DE0: 14 5F 53 35 44 00 A0 08 48 41 43 4B A4 0A 03 A1  ._S5D...HACK....
  1DF0: 04 A4 0A 02 14 0F 5F 41 44 52 00 70 0C 00 00 01  ......_ADR.p....
  1E00: 00 60 A4 60 08 5F 55 49 44 0A 20 08 5F 50 52 57  .`.`._UID. ._PRW
  1E10: 12 06 02 0A 18 0A 05 5B 80 50 45 52 53 02 0A 94  .......[.PERS...
  1E20: 0A 02 5B 81 10 50 45 52 53 01 50 45 52 30 08 50  ..[..PERS.PER0.P
  1E30: 45 52 31 08 14 19 58 4D 45 31 00 70 50 45 52 31  ER1...XME1.pPER1
  1E40: 60 7B 60 0A 80 60 70 60 50 45 52 31 A4 60 08 50  `{`..`p`PER1.`.P
  1E50: 49 43 31 12 34 04 12 0B 04 0B FF FF 00 4C 4E 4B  IC1.4........LNK
  1E60: 41 00 12 0B 04 0B FF FF 01 4C 4E 4B 42 00 12 0C  A........LNKB...
  1E70: 04 0B FF FF 0A 02 4C 4E 4B 43 00 12 0C 04 0B FF  ......LNKC......
  1E80: FF 0A 03 4C 4E 4B 44 00 08 41 50 49 31 12 2C 04  ...LNKD..API1.,.
  1E90: 12 09 04 0B FF FF 00 00 0A 1C 12 09 04 0B FF FF  ................
  1EA0: 01 00 0A 28 12 0A 04 0B FF FF 0A 02 00 0A 29 12  ...(..........).
  1EB0: 0A 04 0B FF FF 0A 03 00 0A 2F 14 1B 5F 50 52 54  ........./.._PRT
  1EC0: 00 70 41 50 49 31 60 A0 0C 92 50 49 43 46 70 50  .pAPI1`...PICFpP
  1ED0: 49 43 31 60 A4 60 5B 82 49 13 50 43 49 32 14 08  IC1`.`[.I.PCI2..
  1EE0: 5F 53 31 44 00 A4 01 14 14 5F 53 33 44 00 A0 08  _S1D....._S3D...
  1EF0: 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 14 5F 53  HACK.........._S
  1F00: 34 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A  4D...HACK.......
  1F10: 02 14 14 5F 53 35 44 00 A0 08 48 41 43 4B A4 0A  ..._S5D...HACK..
  1F20: 03 A1 04 A4 0A 02 14 0F 5F 41 44 52 00 70 0C 00  ........_ADR.p..
  1F30: 00 03 00 60 A4 60 08 5F 55 49 44 0A 21 08 5F 50  ...`.`._UID.!._P
  1F40: 52 57 12 06 02 0A 1C 0A 05 5B 80 52 53 54 53 02  RW.......[.RSTS.
  1F50: 0A B0 0A 04 5B 81 1A 52 53 54 53 01 50 49 44 30  ....[..RSTS.PID0
  1F60: 08 50 49 44 31 08 50 4D 58 53 08 50 4E 55 53 08  .PID1.PMXS.PNUS.
  1F70: 14 18 58 4D 45 32 00 70 50 4D 58 53 60 7B 60 01  ..XME2.pPMXS`{`.
  1F80: 60 70 60 50 4D 58 53 A4 60 08 50 49 43 32 12 34  `p`PMXS.`.PIC2.4
  1F90: 04 12 0B 04 0B FF FF 00 4C 4E 4B 41 00 12 0B 04  ........LNKA....
  1FA0: 0B FF FF 01 4C 4E 4B 42 00 12 0C 04 0B FF FF 0A  ....LNKB........
  1FB0: 02 4C 4E 4B 43 00 12 0C 04 0B FF FF 0A 03 4C 4E  .LNKC.........LN
  1FC0: 4B 44 00 08 41 50 49 32 12 2C 04 12 09 04 0B FF  KD..API2.,......
  1FD0: FF 00 00 0A 18 12 09 04 0B FF FF 01 00 0A 22 12  ..............".
  1FE0: 0A 04 0B FF FF 0A 02 00 0A 23 12 0A 04 0B FF FF  .........#......
  1FF0: 0A 03 00 0A 24 14 1B 5F 50 52 54 00 70 41 50 49  ....$.._PRT.pAPI
  2000: 32 60 A0 0C 92 50 49 43 46 70 50 49 43 32 60 A4  2`...PICFpPIC2`.
  2010: 60 5B 82 40 14 50 43 49 33 14 08 5F 53 31 44 00  `[.@.PCI3.._S1D.
  2020: A4 01 14 14 5F 53 33 44 00 A0 08 48 41 43 4B A4  ...._S3D...HACK.
  2030: 0A 03 A1 04 A4 0A 02 14 14 5F 53 34 44 00 A0 08  ........._S4D...
  2040: 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 14 5F 53  HACK.........._S
  2050: 35 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A  5D...HACK.......
  2060: 02 14 0F 5F 41 44 52 00 70 0C 00 00 07 00 60 A4  ..._ADR.p.....`.
  2070: 60 08 5F 55 49 44 0A 22 08 5F 50 52 57 12 06 02  `._UID."._PRW...
  2080: 0A 1C 0A 05 5B 80 52 53 54 53 02 0A B0 0A 04 5B  ....[.RSTS.....[
  2090: 81 1A 52 53 54 53 01 50 49 44 30 08 50 49 44 31  ..RSTS.PID0.PID1
  20A0: 08 50 4D 58 53 08 50 4E 55 53 08 14 1F 58 4D 45  .PMXS.PNUS...XME
  20B0: 33 00 70 50 4D 58 53 60 7B 60 01 60 70 60 50 4D  3.pPMXS`{`.`p`PM
  20C0: 58 53 A0 04 60 A4 01 A1 03 A4 00 08 50 49 43 33  XS..`.......PIC3
  20D0: 12 34 04 12 0B 04 0B FF FF 00 4C 4E 4B 41 00 12  .4........LNKA..
  20E0: 0B 04 0B FF FF 01 4C 4E 4B 42 00 12 0C 04 0B FF  ......LNKB......
  20F0: FF 0A 02 4C 4E 4B 43 00 12 0C 04 0B FF FF 0A 03  ...LNKC.........
  2100: 4C 4E 4B 44 00 08 41 50 49 33 12 2C 04 12 09 04  LNKD..API3.,....
  2110: 0B FF FF 00 00 0A 1E 12 09 04 0B FF FF 01 00 0A  ................
  2120: 25 12 0A 04 0B FF FF 0A 02 00 0A 27 12 0A 04 0B  %..........'....
  2130: FF FF 0A 03 00 0A 26 14 1B 5F 50 52 54 00 70 41  ......&.._PRT.pA
  2140: 50 49 33 60 A0 0C 92 50 49 43 46 70 50 49 43 33  PI3`...PICFpPIC3
  2150: 60 A4 60 5B 82 4A 0F 50 43 49 35 14 08 5F 53 31  `.`[.J.PCI5.._S1
  2160: 44 00 A4 01 14 14 5F 53 33 44 00 A0 08 48 41 43  D....._S3D...HAC
  2170: 4B A4 0A 03 A1 04 A4 0A 02 14 14 5F 53 34 44 00  K.........._S4D.
  2180: A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 14  ..HACK..........
  2190: 5F 53 35 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04  _S5D...HACK.....
  21A0: A4 0A 02 14 0F 5F 41 44 52 00 70 0C 05 00 1C 00  ....._ADR.p.....
  21B0: 60 A4 60 08 5F 55 49 44 0A 24 08 5F 50 52 57 12  `.`._UID.$._PRW.
  21C0: 06 02 0A 1B 0A 05 5B 80 52 53 54 53 02 0A 60 0A  ......[.RSTS..`.
  21D0: 04 5B 81 1A 52 53 54 53 01 50 49 44 30 08 50 49  .[..RSTS.PID0.PI
  21E0: 44 31 08 50 4D 58 53 08 50 4E 55 53 08 14 1F 58  D1.PMXS.PNUS...X
  21F0: 4D 45 35 00 70 50 4D 58 53 60 7B 60 01 60 70 60  ME5.pPMXS`{`.`p`
  2200: 50 4D 58 53 A0 04 60 A4 01 A1 03 A4 00 08 50 49  PMXS..`.......PI
  2210: 43 35 12 0E 01 12 0B 04 0B FF FF 00 4C 4E 4B 42  C5..........LNKB
  2220: 00 08 41 50 49 35 12 0C 01 12 09 04 0B FF FF 00  ..API5..........
  2230: 00 0A 11 14 1B 5F 50 52 54 00 70 41 50 49 35 60  ....._PRT.pAPI5`
  2240: A0 0C 92 50 49 43 46 70 50 49 43 35 60 A4 60 5B  ...PICFpPIC5`.`[
  2250: 82 49 10 50 43 49 36 14 08 5F 53 31 44 00 A4 01  .I.PCI6.._S1D...
  2260: 14 14 5F 53 33 44 00 A0 08 48 41 43 4B A4 0A 03  .._S3D...HACK...
  2270: A1 04 A4 0A 02 14 14 5F 53 34 44 00 A0 08 48 41  ......._S4D...HA
  2280: 43 4B A4 0A 03 A1 04 A4 0A 02 14 14 5F 53 35 44  CK.........._S5D
  2290: 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14  ...HACK.........
  22A0: 0F 5F 41 44 52 00 70 0C 00 00 1E 00 60 A4 60 08  ._ADR.p.....`.`.
  22B0: 5F 55 49 44 0A 25 08 5F 50 52 57 12 06 02 0A 0B  _UID.%._PRW.....
  22C0: 0A 05 08 50 49 43 36 12 3C 04 12 0D 04 0C FF FF  ...PIC6.<.......
  22D0: 04 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 04 00  ...LNKA.........
  22E0: 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 04 00 0A 02  .LNKB...........
  22F0: 4C 4E 4B 43 00 12 0E 04 0C FF FF 04 00 0A 03 4C  LNKC...........L
  2300: 4E 4B 44 00 08 41 50 49 36 12 34 04 12 0B 04 0C  NKD..API6.4.....
  2310: FF FF 04 00 00 00 0A 10 12 0B 04 0C FF FF 04 00  ................
  2320: 01 00 0A 11 12 0C 04 0C FF FF 04 00 0A 02 00 0A  ................
  2330: 12 12 0C 04 0C FF FF 04 00 0A 03 00 0A 13 14 1B  ................
  2340: 5F 50 52 54 00 70 41 50 49 36 60 A0 0C 92 50 49  _PRT.pAPI6`...PI
  2350: 43 46 70 50 49 43 36 60 A4 60 5B 82 43 AC 50 43  CFpPIC6`.`[.C.PC
  2360: 49 37 08 5F 48 49 44 0C 41 D0 0A 03 08 5F 55 49  I7._HID.A...._UI
  2370: 44 0A 26 08 5F 42 42 4E 0A 20 08 5F 41 44 52 00  D.&._BBN. ._ADR.
  2380: 14 08 5F 53 31 44 00 A4 01 14 14 5F 53 33 44 00  .._S1D....._S3D.
  2390: A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 08 50  ..HACK.........P
  23A0: 49 43 37 12 43 2B 30 12 0B 04 0B FF FF 00 4C 4E  IC7.C+0.......LN
  23B0: 4B 41 00 12 0B 04 0B FF FF 01 4C 4E 4B 42 00 12  KA........LNKB..
  23C0: 0C 04 0B FF FF 0A 02 4C 4E 4B 43 00 12 0C 04 0B  .......LNKC.....
  23D0: FF FF 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 01  ....LNKD........
  23E0: 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 01 00 01  ..LNKA..........
  23F0: 4C 4E 4B 42 00 12 0E 04 0C FF FF 01 00 0A 02 4C  LNKB...........L
  2400: 4E 4B 43 00 12 0E 04 0C FF FF 01 00 0A 03 4C 4E  NKC...........LN
  2410: 4B 44 00 12 0D 04 0C FF FF 02 00 00 4C 4E 4B 41  KD..........LNKA
  2420: 00 12 0D 04 0C FF FF 02 00 01 4C 4E 4B 42 00 12  ..........LNKB..
  2430: 0E 04 0C FF FF 02 00 0A 02 4C 4E 4B 43 00 12 0E  .........LNKC...
  2440: 04 0C FF FF 02 00 0A 03 4C 4E 4B 44 00 12 0D 04  ........LNKD....
  2450: 0C FF FF 03 00 00 4C 4E 4B 41 00 12 0D 04 0C FF  ......LNKA......
  2460: FF 03 00 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 03  ....LNKB........
  2470: 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF 03 00  ...LNKC.........
  2480: 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 04 00 00  ..LNKD..........
  2490: 4C 4E 4B 41 00 12 0D 04 0C FF FF 04 00 01 4C 4E  LNKA..........LN
  24A0: 4B 42 00 12 0E 04 0C FF FF 04 00 0A 02 4C 4E 4B  KB...........LNK
  24B0: 43 00 12 0E 04 0C FF FF 04 00 0A 03 4C 4E 4B 44  C...........LNKD
  24C0: 00 12 0D 04 0C FF FF 05 00 00 4C 4E 4B 41 00 12  ..........LNKA..
  24D0: 0D 04 0C FF FF 05 00 01 4C 4E 4B 42 00 12 0E 04  ........LNKB....
  24E0: 0C FF FF 05 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C  .......LNKC.....
  24F0: FF FF 05 00 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF  ......LNKD......
  2500: FF 06 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 06  ....LNKA........
  2510: 00 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 06 00 0A  ..LNKB..........
  2520: 02 4C 4E 4B 43 00 12 0E 04 0C FF FF 06 00 0A 03  .LNKC...........
  2530: 4C 4E 4B 44 00 12 0D 04 0C FF FF 07 00 00 4C 4E  LNKD..........LN
  2540: 4B 41 00 12 0D 04 0C FF FF 07 00 01 4C 4E 4B 42  KA..........LNKB
  2550: 00 12 0E 04 0C FF FF 07 00 0A 02 4C 4E 4B 43 00  ...........LNKC.
  2560: 12 0E 04 0C FF FF 07 00 0A 03 4C 4E 4B 44 00 12  ..........LNKD..
  2570: 0D 04 0C FF FF 08 00 00 4C 4E 4B 41 00 12 0D 04  ........LNKA....
  2580: 0C FF FF 08 00 01 4C 4E 4B 42 00 12 0E 04 0C FF  ......LNKB......
  2590: FF 08 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF  .....LNKC.......
  25A0: 08 00 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 09  ....LNKD........
  25B0: 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 09 00 01  ..LNKA..........
  25C0: 4C 4E 4B 42 00 12 0E 04 0C FF FF 09 00 0A 02 4C  LNKB...........L
  25D0: 4E 4B 43 00 12 0E 04 0C FF FF 09 00 0A 03 4C 4E  NKC...........LN
  25E0: 4B 44 00 12 0D 04 0C FF FF 0A 00 00 4C 4E 4B 41  KD..........LNKA
  25F0: 00 12 0D 04 0C FF FF 0A 00 01 4C 4E 4B 42 00 12  ..........LNKB..
  2600: 0E 04 0C FF FF 0A 00 0A 02 4C 4E 4B 43 00 12 0E  .........LNKC...
  2610: 04 0C FF FF 0A 00 0A 03 4C 4E 4B 44 00 12 0D 04  ........LNKD....
  2620: 0C FF FF 16 00 00 4C 4E 4B 41 00 12 0D 04 0C FF  ......LNKA......
  2630: FF 16 00 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 16  ....LNKB........
  2640: 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF 16 00  ...LNKC.........
  2650: 0A 03 4C 4E 4B 44 00 08 41 50 49 37 12 43 25 30  ..LNKD..API7.C%0
  2660: 12 09 04 0B FF FF 00 00 0A 10 12 09 04 0B FF FF  ................
  2670: 01 00 0A 11 12 0A 04 0B FF FF 0A 02 00 0A 12 12  ................
  2680: 0A 04 0B FF FF 0A 03 00 0A 13 12 0B 04 0C FF FF  ................
  2690: 01 00 00 00 0A 10 12 0B 04 0C FF FF 01 00 01 00  ................
  26A0: 0A 11 12 0C 04 0C FF FF 01 00 0A 02 00 0A 12 12  ................
  26B0: 0C 04 0C FF FF 01 00 0A 03 00 0A 13 12 0B 04 0C  ................
  26C0: FF FF 02 00 00 00 0A 10 12 0B 04 0C FF FF 02 00  ................
  26D0: 01 00 0A 11 12 0C 04 0C FF FF 02 00 0A 02 00 0A  ................
  26E0: 12 12 0C 04 0C FF FF 02 00 0A 03 00 0A 13 12 0B  ................
  26F0: 04 0C FF FF 03 00 00 00 0A 10 12 0B 04 0C FF FF  ................
  2700: 03 00 01 00 0A 11 12 0C 04 0C FF FF 03 00 0A 02  ................
  2710: 00 0A 12 12 0C 04 0C FF FF 03 00 0A 03 00 0A 13  ................
  2720: 12 0B 04 0C FF FF 04 00 00 00 0A 10 12 0B 04 0C  ................
  2730: FF FF 04 00 01 00 0A 11 12 0C 04 0C FF FF 04 00  ................
  2740: 0A 02 00 0A 12 12 0C 04 0C FF FF 04 00 0A 03 00  ................
  2750: 0A 13 12 0B 04 0C FF FF 05 00 00 00 0A 10 12 0B  ................
  2760: 04 0C FF FF 05 00 01 00 0A 11 12 0C 04 0C FF FF  ................
  2770: 05 00 0A 02 00 0A 12 12 0C 04 0C FF FF 05 00 0A  ................
  2780: 03 00 0A 13 12 0B 04 0C FF FF 06 00 00 00 0A 10  ................
  2790: 12 0B 04 0C FF FF 06 00 01 00 0A 11 12 0C 04 0C  ................
  27A0: FF FF 06 00 0A 02 00 0A 12 12 0C 04 0C FF FF 06  ................
  27B0: 00 0A 03 00 0A 13 12 0B 04 0C FF FF 07 00 00 00  ................
  27C0: 0A 10 12 0B 04 0C FF FF 07 00 01 00 0A 11 12 0C  ................
  27D0: 04 0C FF FF 07 00 0A 02 00 0A 12 12 0C 04 0C FF  ................
  27E0: FF 07 00 0A 03 00 0A 13 12 0B 04 0C FF FF 08 00  ................
  27F0: 00 00 0A 10 12 0B 04 0C FF FF 08 00 01 00 0A 11  ................
  2800: 12 0C 04 0C FF FF 08 00 0A 02 00 0A 12 12 0C 04  ................
  2810: 0C FF FF 08 00 0A 03 00 0A 13 12 0B 04 0C FF FF  ................
  2820: 09 00 00 00 0A 10 12 0B 04 0C FF FF 09 00 01 00  ................
  2830: 0A 11 12 0C 04 0C FF FF 09 00 0A 02 00 0A 12 12  ................
  2840: 0C 04 0C FF FF 09 00 0A 03 00 0A 13 12 0B 04 0C  ................
  2850: FF FF 0A 00 00 00 0A 10 12 0B 04 0C FF FF 0A 00  ................
  2860: 01 00 0A 11 12 0C 04 0C FF FF 0A 00 0A 02 00 0A  ................
  2870: 12 12 0C 04 0C FF FF 0A 00 0A 03 00 0A 13 12 0B  ................
  2880: 04 0C FF FF 16 00 00 00 0A 10 12 0B 04 0C FF FF  ................
  2890: 16 00 01 00 0A 11 12 0C 04 0C FF FF 16 00 0A 02  ................
  28A0: 00 0A 12 12 0C 04 0C FF FF 16 00 0A 03 00 0A 13  ................
  28B0: 14 1B 5F 50 52 54 00 70 41 50 49 37 60 A0 0C 92  .._PRT.pAPI7`...
  28C0: 50 49 43 46 70 50 49 43 37 60 A4 60 14 42 1D 5F  PICFpPIC7`.`.B._
  28D0: 43 52 53 00 08 4D 45 4D 53 11 4A 07 0A 76 88 0D  CRS..MEMS.J..v..
  28E0: 00 02 0C 00 00 00 20 00 3D 00 00 00 1E 00 88 0D  ...... .=.......
  28F0: 00 01 00 03 00 00 00 00 00 00 00 00 00 00 87 17  ................
  2900: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
  2910: 00 00 00 00 00 00 00 00 88 0D 00 01 0C 03 00 00  ................
  2920: B0 03 BB 03 00 00 0C 00 88 0D 00 01 0C 03 00 00  ................
  2930: C0 03 DF 03 00 00 20 00 87 17 00 00 0C 03 00 00  ...... .........
  2940: 00 00 00 00 0A 00 FF FF 0B 00 00 00 00 00 00 00  ................
  2950: 02 00 79 00 A0 4E 04 92 93 47 4D 58 31 00 8A 4D  ..y..N...GMX1..M
  2960: 45 4D 53 0A 2A 4D 50 31 4E 8A 4D 45 4D 53 0A 2E  EMS.*MP1N.MEMS..
  2970: 4D 50 31 58 8A 4D 45 4D 53 0A 36 4D 50 31 4C 70  MP1X.MEMS.6MP1Lp
  2980: 47 4D 49 31 4D 50 31 4E 70 47 4D 58 31 4D 50 31  GMI1MP1NpGMX1MP1
  2990: 58 74 4D 50 31 58 4D 50 31 4E 4D 50 31 4C 75 4D  XtMP1XMP1NMP1LuM
  29A0: 50 31 4C A0 4E 04 92 93 47 49 58 31 00 8B 4D 45  P1L.N...GIX1..ME
  29B0: 4D 53 0A 18 49 50 31 4E 8B 4D 45 4D 53 0A 1A 49  MS..IP1N.MEMS..I
  29C0: 50 31 58 8B 4D 45 4D 53 0A 1E 49 50 31 4C 70 47  P1X.MEMS..IP1LpG
  29D0: 49 42 31 49 50 31 4E 70 47 49 58 31 49 50 31 58  IB1IP1NpGIX1IP1X
  29E0: 74 49 50 31 58 49 50 31 4E 49 50 31 4C 75 49 50  tIP1XIP1NIP1LuIP
  29F0: 31 4C 70 43 4D 52 44 0A 74 60 A0 4F 09 95 60 0A  1LpCMRD.t`.O..`.
  2A00: 20 8B 4D 45 4D 53 0A 42 56 47 31 4E 8B 4D 45 4D   .MEMS.BVG1N.MEM
  2A10: 53 0A 44 56 47 31 58 8B 4D 45 4D 53 0A 48 56 47  S.DVG1X.MEMS.HVG
  2A20: 31 4C 8B 4D 45 4D 53 0A 52 56 47 32 4E 8B 4D 45  1L.MEMS.RVG2N.ME
  2A30: 4D 53 0A 54 56 47 32 58 8B 4D 45 4D 53 0A 58 56  MS.TVG2X.MEMS.XV
  2A40: 47 32 4C 8B 4D 45 4D 53 0A 64 56 47 4D 4E 8B 4D  G2L.MEMS.dVGMN.M
  2A50: 45 4D 53 0A 68 56 47 4D 58 8B 4D 45 4D 53 0A 70  EMS.hVGMX.MEMS.p
  2A60: 56 47 4D 4C 70 00 56 47 31 4E 70 00 56 47 31 58  VGMLp.VG1Np.VG1X
  2A70: 70 00 56 47 31 4C 70 00 56 47 32 4E 70 00 56 47  p.VG1Lp.VG2Np.VG
  2A80: 32 58 70 00 56 47 32 4C 70 00 56 47 4D 4E 70 00  2Xp.VG2Lp.VGMNp.
  2A90: 56 47 4D 58 70 00 56 47 4D 4C A4 4D 45 4D 53 5B  VGMXp.VGML.MEMS[
  2AA0: 82 4A 0F 50 43 49 38 14 08 5F 53 31 44 00 A4 01  .J.PCI8.._S1D...
  2AB0: 14 14 5F 53 33 44 00 A0 08 48 41 43 4B A4 0A 03  .._S3D...HACK...
  2AC0: A1 04 A4 0A 02 14 14 5F 53 34 44 00 A0 08 48 41  ......._S4D...HA
  2AD0: 43 4B A4 0A 03 A1 04 A4 0A 02 14 14 5F 53 35 44  CK.........._S5D
  2AE0: 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14  ...HACK.........
  2AF0: 0F 5F 41 44 52 00 70 0C 00 00 01 00 60 A4 60 08  ._ADR.p.....`.`.
  2B00: 5F 55 49 44 0A 27 08 5F 50 52 57 12 06 02 0A 18  _UID.'._PRW.....
  2B10: 0A 05 5B 80 52 53 54 53 02 0A B0 0A 04 5B 81 1A  ..[.RSTS.....[..
  2B20: 52 53 54 53 01 50 49 44 30 08 50 49 44 31 08 50  RSTS.PID0.PID1.P
  2B30: 4D 58 53 08 50 4E 55 53 08 14 1F 58 4D 45 38 00  MXS.PNUS...XME8.
  2B40: 70 50 4D 58 53 60 7B 60 01 60 70 60 50 4D 58 53  pPMXS`{`.`p`PMXS
  2B50: A0 04 60 A4 01 A1 03 A4 00 08 50 49 43 38 12 0E  ..`.......PIC8..
  2B60: 01 12 0B 04 0B FF FF 00 4C 4E 4B 41 00 08 41 50  ........LNKA..AP
  2B70: 49 38 12 0C 01 12 09 04 0B FF FF 00 00 0A 34 14  I8............4.
  2B80: 1B 5F 50 52 54 00 70 41 50 49 38 60 A0 0C 92 50  ._PRT.pAPI8`...P
  2B90: 49 43 46 70 50 49 43 38 60 A4 60 5B 82 40 14 50  ICFpPIC8`.`[.@.P
  2BA0: 43 49 41 14 08 5F 53 31 44 00 A4 01 14 14 5F 53  CIA.._S1D....._S
  2BB0: 33 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A  3D...HACK.......
  2BC0: 02 14 14 5F 53 34 44 00 A0 08 48 41 43 4B A4 0A  ..._S4D...HACK..
  2BD0: 03 A1 04 A4 0A 02 14 14 5F 53 35 44 00 A0 08 48  ........_S5D...H
  2BE0: 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 0F 5F 41 44  ACK.........._AD
  2BF0: 52 00 70 0C 00 00 07 00 60 A4 60 08 5F 55 49 44  R.p.....`.`._UID
  2C00: 0A 29 08 5F 50 52 57 12 06 02 0A 1F 0A 05 5B 80  .)._PRW.......[.
  2C10: 52 53 54 53 02 0A B0 0A 04 5B 81 1A 52 53 54 53  RSTS.....[..RSTS
  2C20: 01 50 49 44 30 08 50 49 44 31 08 50 4D 58 53 08  .PID0.PID1.PMXS.
  2C30: 50 4E 55 53 08 14 1F 58 4D 45 41 00 70 50 4D 58  PNUS...XMEA.pPMX
  2C40: 53 60 7B 60 01 60 70 60 50 4D 58 53 A0 04 60 A4  S`{`.`p`PMXS..`.
  2C50: 01 A1 03 A4 00 08 50 49 43 41 12 34 04 12 0B 04  ......PICA.4....
  2C60: 0B FF FF 00 4C 4E 4B 41 00 12 0B 04 0B FF FF 01  ....LNKA........
  2C70: 4C 4E 4B 42 00 12 0C 04 0B FF FF 0A 02 4C 4E 4B  LNKB.........LNK
  2C80: 43 00 12 0C 04 0B FF FF 0A 03 4C 4E 4B 44 00 08  C.........LNKD..
  2C90: 41 50 49 41 12 2C 04 12 09 04 0B FF FF 00 00 0A  APIA.,..........
  2CA0: 36 12 09 04 0B FF FF 01 00 0A 3D 12 0A 04 0B FF  6.........=.....
  2CB0: FF 0A 02 00 0A 3F 12 0A 04 0B FF FF 0A 03 00 0A  .....?..........
  2CC0: 3E 14 1B 5F 50 52 54 00 70 41 50 49 41 60 A0 0C  >.._PRT.pAPIA`..
  2CD0: 92 50 49 43 46 70 50 49 43 41 60 A4 60 5B 82 40  .PICFpPICA`.`[.@
  2CE0: 14 50 43 49 42 14 08 5F 53 31 44 00 A4 01 14 14  .PCIB.._S1D.....
  2CF0: 5F 53 33 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04  _S3D...HACK.....
  2D00: A4 0A 02 14 14 5F 53 34 44 00 A0 08 48 41 43 4B  ....._S4D...HACK
  2D10: A4 0A 03 A1 04 A4 0A 02 14 14 5F 53 35 44 00 A0  .........._S5D..
  2D20: 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 0F 5F  .HACK.........._
  2D30: 41 44 52 00 70 0C 00 00 09 00 60 A4 60 08 5F 55  ADR.p.....`.`._U
  2D40: 49 44 0A 2A 08 5F 50 52 57 12 06 02 0A 1F 0A 05  ID.*._PRW.......
  2D50: 5B 80 52 53 54 53 02 0A B0 0A 04 5B 81 1A 52 53  [.RSTS.....[..RS
  2D60: 54 53 01 50 49 44 30 08 50 49 44 31 08 50 4D 58  TS.PID0.PID1.PMX
  2D70: 53 08 50 4E 55 53 08 14 1F 58 4D 45 42 00 70 50  S.PNUS...XMEB.pP
  2D80: 4D 58 53 60 7B 60 01 60 70 60 50 4D 58 53 A0 04  MXS`{`.`p`PMXS..
  2D90: 60 A4 01 A1 03 A4 00 08 50 49 43 42 12 34 04 12  `.......PICB.4..
  2DA0: 0B 04 0B FF FF 00 4C 4E 4B 41 00 12 0B 04 0B FF  ......LNKA......
  2DB0: FF 01 4C 4E 4B 42 00 12 0C 04 0B FF FF 0A 02 4C  ..LNKB.........L
  2DC0: 4E 4B 43 00 12 0C 04 0B FF FF 0A 03 4C 4E 4B 44  NKC.........LNKD
  2DD0: 00 08 41 50 49 42 12 2C 04 12 09 04 0B FF FF 00  ..APIB.,........
  2DE0: 00 0A 38 12 09 04 0B FF FF 01 00 0A 42 12 0A 04  ..8.........B...
  2DF0: 0B FF FF 0A 02 00 0A 47 12 0A 04 0B FF FF 0A 03  .......G........
  2E00: 00 0A 41 14 1B 5F 50 52 54 00 70 41 50 49 42 60  ..A.._PRT.pAPIB`
  2E10: A0 0C 92 50 49 43 46 70 50 49 43 42 60 A4 60 10  ...PICFpPICB`.`.
  2E20: 43 24 5C 00 14 4E 04 2E 5F 47 50 45 5F 4C 30 33  C$\..N.._GPE_L03
  2E30: 00 A0 16 80 93 53 53 54 58 01 00 86 5C 2E 5F 53  .....SSTX...\._S
  2E40: 42 5F 56 42 54 4E 0A 02 5B 22 0A 14 7F 54 48 52  B_VBTN..["...THR
  2E50: 50 0A 08 54 48 52 50 7D 50 4D 45 32 01 50 4D 45  P..THRP}PME2.PME
  2E60: 32 86 5C 2F 03 5F 53 42 5F 50 43 49 30 55 53 42  2.\/._SB_PCI0USB
  2E70: 30 0A 02 14 4E 04 2E 5F 47 50 45 5F 4C 30 34 00  0...N.._GPE_L04.
  2E80: A0 16 80 93 53 53 54 58 01 00 86 5C 2E 5F 53 42  ....SSTX...\._SB
  2E90: 5F 56 42 54 4E 0A 02 5B 22 0A 14 7F 54 48 52 50  _VBTN..["...THRP
  2EA0: 0A 10 54 48 52 50 7D 50 4D 45 32 01 50 4D 45 32  ..THRP}PME2.PME2
  2EB0: 86 5C 2F 03 5F 53 42 5F 50 43 49 30 55 53 42 31  .\/._SB_PCI0USB1
  2EC0: 0A 02 14 4E 04 2E 5F 47 50 45 5F 4C 30 43 00 A0  ...N.._GPE_L0C..
  2ED0: 16 80 93 53 53 54 58 01 00 86 5C 2E 5F 53 42 5F  ...SSTX...\._SB_
  2EE0: 56 42 54 4E 0A 02 5B 22 0A 14 7F 47 4C 42 54 0A  VBTN..["...GLBT.
  2EF0: 10 47 4C 42 54 7D 50 4D 45 32 01 50 4D 45 32 86  .GLBT}PME2.PME2.
  2F00: 5C 2F 03 5F 53 42 5F 50 43 49 30 55 53 42 32 0A  \/._SB_PCI0USB2.
  2F10: 02 14 4E 04 2E 5F 47 50 45 5F 4C 30 45 00 A0 16  ..N.._GPE_L0E...
  2F20: 80 93 53 53 54 58 01 00 86 5C 2E 5F 53 42 5F 56  ..SSTX...\._SB_V
  2F30: 42 54 4E 0A 02 5B 22 0A 14 7F 47 4C 42 54 0A 40  BTN..["...GLBT.@
  2F40: 47 4C 42 54 7D 50 4D 45 32 01 50 4D 45 32 86 5C  GLBT}PME2.PME2.\
  2F50: 2F 03 5F 53 42 5F 50 43 49 30 55 53 42 33 0A 02  /._SB_PCI0USB3..
  2F60: 14 4E 04 2E 5F 47 50 45 5F 4C 30 35 00 A0 16 80  .N.._GPE_L05....
  2F70: 93 53 53 54 58 01 00 86 5C 2E 5F 53 42 5F 56 42  .SSTX...\._SB_VB
  2F80: 54 4E 0A 02 5B 22 0A 14 7F 54 48 52 50 0A 20 54  TN..["...THRP. T
  2F90: 48 52 50 7D 50 4D 45 32 01 50 4D 45 32 86 5C 2F  HRP}PME2.PME2.\/
  2FA0: 03 5F 53 42 5F 50 43 49 30 55 53 42 34 0A 02 14  ._SB_PCI0USB4...
  2FB0: 4E 04 2E 5F 47 50 45 5F 4C 32 30 00 A0 16 80 93  N.._GPE_L20.....
  2FC0: 53 53 54 58 01 00 86 5C 2E 5F 53 42 5F 56 42 54  SSTX...\._SB_VBT
  2FD0: 4E 0A 02 5B 22 0A 14 7F 54 48 52 50 0A 20 54 48  N..["...THRP. TH
  2FE0: 52 50 7D 50 4D 45 32 01 50 4D 45 32 86 5C 2F 03  RP}PME2.PME2.\/.
  2FF0: 5F 53 42 5F 50 43 49 30 55 53 42 35 0A 02 14 31  _SB_PCI0USB5...1
  3000: 2E 5F 47 50 45 5F 4C 30 44 00 5B 22 0A 14 7F 47  ._GPE_L0D.["...G
  3010: 4C 42 54 0A 20 47 4C 42 54 7D 50 4D 45 32 01 50  LBT. GLBT}PME2.P
  3020: 4D 45 32 86 5C 2E 5F 53 42 5F 50 43 49 30 0A 02  ME2.\._SB_PCI0..
  3030: 14 32 2E 5F 47 50 45 5F 4C 30 42 00 7F 47 4C 42  .2._GPE_L0B..GLB
  3040: 54 0A 08 47 4C 42 54 7D 50 4D 45 32 01 50 4D 45  T..GLBT}PME2.PME
  3050: 32 86 5C 2F 03 5F 53 42 5F 50 43 49 30 50 43 49  2.\/._SB_PCI0PCI
  3060: 36 0A 02 10 4A 0F 2E 5F 53 42 5F 50 43 49 30 5B  6...J.._SB_PCI0[
  3070: 82 4D 0E 49 53 41 5F 08 5F 41 44 52 0C 00 00 1F  .M.ISA_._ADR....
  3080: 00 08 5F 55 49 44 0A 0A 5B 80 50 34 30 43 02 0A  .._UID..[.P40C..
  3090: 60 0A 04 5B 80 50 34 31 43 02 0A 68 0A 04 5B 82  `..[.P41C..h..[.
  30A0: 4E 0B 4D 42 49 4F 08 5F 48 49 44 0C 41 D0 0C 01  N.MBIO._HID.A...
  30B0: 08 5F 55 49 44 0A 0B 14 46 0A 5F 43 52 53 00 08  ._UID...F._CRS..
  30C0: 4D 49 4F 31 11 35 0A 32 47 01 62 00 62 00 01 02  MIO1.5.2G.b.b...
  30D0: 47 01 65 00 65 00 01 0B 47 01 E0 00 E0 00 01 10  G.e.e...G.......
  30E0: 47 01 00 08 00 08 01 60 47 01 00 0C 00 0C 01 80  G......`G.......
  30F0: 47 01 60 08 60 08 01 A0 79 00 08 4D 49 4F 32 11  G.`.`...y..MIO2.
  3100: 46 04 0A 42 47 01 60 00 60 00 01 01 47 01 64 00  F..BG.`.`...G.d.
  3110: 64 00 01 01 47 01 62 00 62 00 01 02 47 01 65 00  d...G.b.b...G.e.
  3120: 65 00 01 0B 47 01 E0 00 E0 00 01 10 47 01 00 08  e...G.......G...
  3130: 00 08 01 60 47 01 00 0C 00 0C 01 80 47 01 60 08  ...`G.......G.`.
  3140: 60 08 01 A0 79 00 A0 10 7D 47 43 4B 42 47 43 4D  `...y...}GCKBGCM
  3150: 53 00 A4 4D 49 4F 31 A1 06 A4 4D 49 4F 32 10 41  S..MIO1...MIO2.A
  3160: 38 2E 5F 53 42 5F 50 43 49 30 5B 82 45 09 55 53  8._SB_PCI0[.E.US
  3170: 42 30 08 5F 41 44 52 0C 00 00 1D 00 08 5F 55 49  B0._ADR......_UI
  3180: 44 0A 05 08 5F 50 52 57 12 06 02 0A 03 0A 03 5B  D..._PRW.......[
  3190: 80 55 50 43 31 02 0A C1 01 5B 81 0B 55 50 43 31  .UPC1....[..UPC1
  31A0: 01 4C 45 47 4B 08 14 28 5F 49 4E 49 00 A0 05 48  .LEGK..(_INI...H
  31B0: 41 43 4B A1 17 70 4C 45 47 4B 60 7B 60 0A 60 60  ACK..pLEGK`{`.``
  31C0: 7D 60 0A 20 60 70 60 4C 45 47 4B 47 55 53 42 14  }`. `p`LEGKGUSB.
  31D0: 13 5F 53 54 41 00 A0 08 47 43 55 43 A4 0A 0F A1  ._STA...GCUC....
  31E0: 03 A4 00 14 08 5F 53 31 44 00 A4 01 14 14 5F 53  ....._S1D....._S
  31F0: 33 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A  3D...HACK.......
  3200: 02 5B 82 41 09 55 53 42 31 08 5F 41 44 52 0C 01  .[.A.USB1._ADR..
  3210: 00 1D 00 08 5F 55 49 44 0A 06 08 5F 50 52 57 12  ...._UID..._PRW.
  3220: 06 02 0A 04 0A 03 5B 80 55 50 43 31 02 0A C1 01  ......[.UPC1....
  3230: 5B 81 0B 55 50 43 31 01 4C 45 47 4B 08 14 24 5F  [..UPC1.LEGK..$_
  3240: 49 4E 49 00 A0 05 48 41 43 4B A1 17 70 4C 45 47  INI...HACK..pLEG
  3250: 4B 60 7B 60 0A 60 60 7D 60 0A 20 60 70 60 4C 45  K`{`.``}`. `p`LE
  3260: 47 4B 14 13 5F 53 54 41 00 A0 08 47 43 55 43 A4  GK.._STA...GCUC.
  3270: 0A 0F A1 03 A4 00 14 08 5F 53 31 44 00 A4 01 14  ........_S1D....
  3280: 14 5F 53 33 44 00 A0 08 48 41 43 4B A4 0A 03 A1  ._S3D...HACK....
  3290: 04 A4 0A 02 5B 82 41 09 55 53 42 32 08 5F 41 44  ....[.A.USB2._AD
  32A0: 52 0C 02 00 1D 00 08 5F 55 49 44 0A 14 08 5F 50  R......_UID..._P
  32B0: 52 57 12 06 02 0A 0C 0A 03 5B 80 55 50 43 31 02  RW.......[.UPC1.
  32C0: 0A C1 01 5B 81 0B 55 50 43 31 01 4C 45 47 4B 08  ...[..UPC1.LEGK.
  32D0: 14 24 5F 49 4E 49 00 A0 05 48 41 43 4B A1 17 70  .$_INI...HACK..p
  32E0: 4C 45 47 4B 60 7B 60 0A 60 60 7D 60 0A 20 60 70  LEGK`{`.``}`. `p
  32F0: 60 4C 45 47 4B 14 13 5F 53 54 41 00 A0 08 47 43  `LEGK.._STA...GC
  3300: 55 43 A4 0A 0F A1 03 A4 00 14 08 5F 53 31 44 00  UC........._S1D.
  3310: A4 01 14 14 5F 53 33 44 00 A0 08 48 41 43 4B A4  ...._S3D...HACK.
  3320: 0A 03 A1 04 A4 0A 02 5B 82 41 09 55 53 42 33 08  .......[.A.USB3.
  3330: 5F 41 44 52 0C 00 00 1A 00 08 5F 55 49 44 0A 15  _ADR......_UID..
  3340: 08 5F 50 52 57 12 06 02 0A 0E 0A 03 5B 80 55 50  ._PRW.......[.UP
  3350: 43 31 02 0A C1 01 5B 81 0B 55 50 43 31 01 4C 45  C1....[..UPC1.LE
  3360: 47 4B 08 14 24 5F 49 4E 49 00 A0 05 48 41 43 4B  GK..$_INI...HACK
  3370: A1 17 70 4C 45 47 4B 60 7B 60 0A 60 60 7D 60 0A  ..pLEGK`{`.``}`.
  3380: 20 60 70 60 4C 45 47 4B 14 13 5F 53 54 41 00 A0   `p`LEGK.._STA..
  3390: 08 47 43 55 43 A4 0A 0F A1 03 A4 00 14 08 5F 53  .GCUC........._S
  33A0: 31 44 00 A4 01 14 14 5F 53 33 44 00 A0 08 48 41  1D....._S3D...HA
  33B0: 43 4B A4 0A 03 A1 04 A4 0A 02 5B 82 41 09 55 53  CK........[.A.US
  33C0: 42 34 08 5F 41 44 52 0C 01 00 1A 00 08 5F 55 49  B4._ADR......_UI
  33D0: 44 0A 16 08 5F 50 52 57 12 06 02 0A 05 0A 03 5B  D..._PRW.......[
  33E0: 80 55 50 43 31 02 0A C1 01 5B 81 0B 55 50 43 31  .UPC1....[..UPC1
  33F0: 01 4C 45 47 4B 08 14 24 5F 49 4E 49 00 A0 05 48  .LEGK..$_INI...H
  3400: 41 43 4B A1 17 70 4C 45 47 4B 60 7B 60 0A 60 60  ACK..pLEGK`{`.``
  3410: 7D 60 0A 20 60 70 60 4C 45 47 4B 14 13 5F 53 54  }`. `p`LEGK.._ST
  3420: 41 00 A0 08 47 43 55 43 A4 0A 0F A1 03 A4 00 14  A...GCUC........
  3430: 08 5F 53 31 44 00 A4 01 14 14 5F 53 33 44 00 A0  ._S1D....._S3D..
  3440: 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 5B 82 41  .HACK........[.A
  3450: 09 55 53 42 35 08 5F 41 44 52 0C 02 00 1A 00 08  .USB5._ADR......
  3460: 5F 55 49 44 0A 1A 08 5F 50 52 57 12 06 02 0A 20  _UID..._PRW.... 
  3470: 0A 03 5B 80 55 50 43 31 02 0A C1 01 5B 81 0B 55  ..[.UPC1....[..U
  3480: 50 43 31 01 4C 45 47 4B 08 14 24 5F 49 4E 49 00  PC1.LEGK..$_INI.
  3490: A0 05 48 41 43 4B A1 17 70 4C 45 47 4B 60 7B 60  ..HACK..pLEGK`{`
  34A0: 0A 60 60 7D 60 0A 20 60 70 60 4C 45 47 4B 14 13  .``}`. `p`LEGK..
  34B0: 5F 53 54 41 00 A0 08 47 43 55 43 A4 0A 0F A1 03  _STA...GCUC.....
  34C0: A4 00 14 08 5F 53 31 44 00 A4 01 14 14 5F 53 33  ...._S1D....._S3
  34D0: 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02  D...HACK........
  34E0: 10 16 5C 00 08 50 49 43 46 00 14 0C 5F 50 49 43  ..\..PICF..._PIC
  34F0: 01 70 68 50 49 43 46 10 49 3F 2E 5F 53 42 5F 50  .phPICF.I?._SB_P
  3500: 43 49 30 14 1B 5F 50 52 54 00 70 41 50 49 30 60  CI0.._PRT.pAPI0`
  3510: A0 0C 92 50 49 43 46 70 50 49 43 30 60 A4 60 14  ...PICFpPIC0`.`.
  3520: 41 3D 5F 43 52 53 00 5B 80 4F 52 4D 46 00 0C 6C  A=_CRS.[.ORMF..l
  3530: FF 0F 00 0A 04 5B 81 10 4F 52 4D 46 02 4F 52 47  .....[..ORMF.ORG
  3540: 50 10 4F 52 4E 44 10 08 4D 45 4D 50 11 4F 14 0B  P.ORND..MEMP.O..
  3550: 4A 01 88 0D 00 02 0C 00 00 00 00 00 1F 00 00 00  J...............
  3560: 80 00 47 01 F8 0C F8 0C 01 08 88 0D 00 01 0C 03  ..G.............
  3570: 00 00 00 00 AF 03 00 00 B0 03 88 0D 00 01 0C 03  ................
  3580: 00 00 B0 03 BB 03 00 00 0C 00 88 0D 00 01 0C 03  ................
  3590: 00 00 BC 03 BF 03 00 00 04 00 88 0D 00 01 0C 03  ................
  35A0: 00 00 C0 03 DF 03 00 00 20 00 88 0D 00 01 0C 03  ........ .......
  35B0: 00 00 E0 03 F7 0C 00 00 18 09 88 0D 00 01 00 03  ................
  35C0: 00 00 00 00 00 00 00 00 00 00 87 17 00 00 0C 03  ................
  35D0: 00 00 00 00 00 00 0A 00 FF FF 0B 00 00 00 00 00  ................
  35E0: 00 00 02 00 87 17 00 00 0C 03 00 00 00 00 00 80  ................
  35F0: 0C 00 FF FF 0D 00 00 00 00 00 00 80 01 00 87 17  ................
  3600: 00 00 0C 03 00 00 00 00 00 00 0E 00 FF FF 0F 00  ................
  3610: 00 00 00 00 00 00 02 00 87 17 00 00 00 01 00 00  ................
  3620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  3630: 00 00 87 17 00 00 00 01 00 00 00 00 00 00 00 00  ................
  3640: FF FF FF F7 00 00 00 00 00 00 00 00 87 17 00 00  ................
  3650: 00 01 00 00 00 00 00 00 98 FF FF 0F 98 FF 00 00  ................
  3660: 00 00 00 10 00 00 87 17 00 00 08 01 00 00 00 00  ................
  3670: 00 C0 97 FF FF FF 97 FF 00 00 00 00 00 40 00 00  .............@..
  3680: 87 17 00 00 0C 01 00 00 00 00 00 00 D2 FE FF FF  ................
  3690: D9 FE 00 00 00 00 00 00 08 00 79 00 8A 4D 45 4D  ..........y..MEM
  36A0: 50 0A 9C 50 4D 49 4E 8A 4D 45 4D 50 0A A0 50 4D  P..PMIN.MEMP..PM
  36B0: 41 58 8A 4D 45 4D 50 0A A8 50 4C 45 4E 8A 4D 45  AX.MEMP..PLEN.ME
  36C0: 4D 50 0A B6 42 4D 49 4E 8A 4D 45 4D 50 0A C2 42  MP..BMIN.MEMP..B
  36D0: 4C 45 4E 79 4F 52 47 50 0A 04 42 4D 49 4E 74 0C  LENyORGP..BMINt.
  36E0: 00 00 10 00 42 4D 49 4E 42 4C 45 4E 70 0C 00 00  ....BMINBLENp...
  36F0: 0C 00 50 4D 49 4E 74 42 4D 49 4E 01 50 4D 41 58  ..PMINtBMIN.PMAX
  3700: 74 42 4D 49 4E 50 4D 49 4E 50 4C 45 4E 8A 4D 45  tBMINPMINPLEN.ME
  3710: 4D 50 0A EA 4D 45 4D 4D 8A 4D 45 4D 50 0A EE 4D  MP..MEMM.MEMP..M
  3720: 45 4D 41 8A 4D 45 4D 50 0A F6 4D 45 4D 4C 70 47  EMA.MEMP..MEMLpG
  3730: 54 4F 4D 4D 45 4D 4D 74 4D 45 4D 41 4D 45 4D 4D  TOMMEMMtMEMAMEMM
  3740: 4D 45 4D 4C 75 4D 45 4D 4C 70 47 4D 49 30 60 70  MEMLuMEMLpGMI0`p
  3750: 47 4D 49 31 61 A0 0D 95 61 60 A0 08 92 93 61 00  GMI1a...a`....a.
  3760: 70 61 60 76 60 70 60 4D 45 4D 41 74 4D 45 4D 41  pa`v`p`MEMAtMEMA
  3770: 4D 45 4D 4D 4D 45 4D 4C 75 4D 45 4D 4C 8A 4D 45  MEMMMEMLuMEML.ME
  3780: 4D 50 0A D0 4D 49 4E 31 8A 4D 45 4D 50 0A D4 4D  MP..MIN1.MEMP..M
  3790: 41 58 31 8A 4D 45 4D 50 0A DC 4C 45 4E 31 70 47  AX1.MEMP..LEN1pG
  37A0: 4D 58 30 4D 41 58 31 70 47 4D 49 30 4D 49 4E 31  MX0MAX1pGMI0MIN1
  37B0: 74 4D 41 58 31 4D 49 4E 31 4C 45 4E 31 75 4C 45  tMAX1MIN1LEN1uLE
  37C0: 4E 31 8B 4D 45 4D 50 0A 70 49 52 31 4E 8B 4D 45  N1.MEMP.pIR1N.ME
  37D0: 4D 50 0A 72 49 52 31 58 8B 4D 45 4D 50 0A 76 49  MP.rIR1X.MEMP.vI
  37E0: 52 31 4C 70 0B 00 0D 49 52 31 4E 70 47 49 58 30  R1Lp...IR1NpGIX0
  37F0: 49 52 31 58 74 49 52 31 58 49 52 31 4E 49 52 31  IR1XtIR1XIR1NIR1
  3800: 4C 75 49 52 31 4C 70 43 4D 52 44 0A 74 60 A0 4F  LuIR1LpCMRD.t`.O
  3810: 09 94 60 0A 1F 8B 4D 45 4D 50 0A 30 56 47 31 4E  ..`...MEMP.0VG1N
  3820: 8B 4D 45 4D 50 0A 32 56 47 31 58 8B 4D 45 4D 50  .MEMP.2VG1X.MEMP
  3830: 0A 36 56 47 31 4C 8B 4D 45 4D 50 0A 50 56 47 32  .6VG1L.MEMP.PVG2
  3840: 4E 8B 4D 45 4D 50 0A 52 56 47 32 58 8B 4D 45 4D  N.MEMP.RVG2X.MEM
  3850: 50 0A 56 56 47 32 4C 8B 4D 45 4D 50 0A 82 56 47  P.VVG2L.MEMP..VG
  3860: 4D 4E 8B 4D 45 4D 50 0A 86 56 47 4D 58 8B 4D 45  MN.MEMP..VGMX.ME
  3870: 4D 50 0A 8E 56 47 4D 4C 70 00 56 47 31 4E 70 00  MP..VGMLp.VG1Np.
  3880: 56 47 31 58 70 00 56 47 31 4C 70 00 56 47 32 4E  VG1Xp.VG1Lp.VG2N
  3890: 70 00 56 47 32 58 70 00 56 47 32 4C 70 00 56 47  p.VG2Xp.VG2Lp.VG
  38A0: 4D 4E 70 00 56 47 4D 58 70 00 56 47 4D 4C 8A 4D  MNp.VGMXp.VGML.M
  38B0: 45 4D 50 0B 04 01 55 4D 49 4E 8A 4D 45 4D 50 0B  EMP...UMIN.MEMP.
  38C0: 08 01 55 4D 41 58 8A 4D 45 4D 50 0B 10 01 55 4C  ..UMAX.MEMP...UL
  38D0: 45 4E A0 05 47 43 55 43 A1 13 70 00 55 4D 49 4E  EN..GCUC..p.UMIN
  38E0: 70 00 55 4D 41 58 70 00 55 4C 45 4E A4 4D 45 4D  p.UMAXp.ULEN.MEM
  38F0: 50 10 44 05 2F 03 5F 53 42 5F 50 43 49 30 49 53  P.D./._SB_PCI0IS
  3900: 41 5F 5B 82 42 04 44 4D 41 5F 08 5F 48 49 44 0C  A_[.B.DMA_._HID.
  3910: 41 D0 02 00 14 31 5F 43 52 53 00 08 44 4D 41 42  A....1_CRS..DMAB
  3920: 11 20 0A 1D 47 01 80 00 80 00 01 20 47 01 00 00  . ..G...... G...
  3930: 00 00 01 20 47 01 C0 00 C0 00 01 20 2A 10 12 79  ... G...... *..y
  3940: 00 A4 44 4D 41 42 10 43 04 2F 03 5F 53 42 5F 50  ..DMAB.C./._SB_P
  3950: 43 49 30 49 53 41 5F 5B 82 31 46 50 55 5F 08 5F  CI0ISA_[.1FPU_._
  3960: 48 49 44 0C 41 D0 0C 04 14 21 5F 43 52 53 00 08  HID.A....!_CRS..
  3970: 46 50 55 42 11 10 0A 0D 47 01 F0 00 F0 00 01 10  FPUB....G.......
  3980: 22 00 20 79 00 A4 46 50 55 42 10 41 05 2F 03 5F  ". y..FPUB.A./._
  3990: 53 42 5F 50 43 49 30 49 53 41 5F 5B 82 3F 50 49  SB_PCI0ISA_[.?PI
  39A0: 43 5F 08 5F 48 49 44 0B 41 D0 14 31 5F 43 52 53  C_._HID.A..1_CRS
  39B0: 00 08 50 49 43 42 11 20 0A 1D 47 01 20 00 20 00  ..PICB. ..G. . .
  39C0: 01 20 47 01 A0 00 A0 00 01 20 47 01 D0 04 D0 04  . G...... G.....
  39D0: 01 02 22 04 00 79 00 A4 50 49 43 42 10 3F 2F 03  .."..y..PICB.?/.
  39E0: 5F 53 42 5F 50 43 49 30 49 53 41 5F 5B 82 2E 53  _SB_PCI0ISA_[..S
  39F0: 50 4B 5F 08 5F 48 49 44 0C 41 D0 08 00 14 1E 5F  PK_._HID.A....._
  3A00: 43 52 53 00 08 53 50 4B 42 11 0D 0A 0A 47 01 61  CRS..SPKB....G.a
  3A10: 00 61 00 01 01 79 00 A4 53 50 4B 42 10 43 04 2F  .a...y..SPKB.C./
  3A20: 03 5F 53 42 5F 50 43 49 30 49 53 41 5F 5B 82 31  ._SB_PCI0ISA_[.1
  3A30: 52 54 43 5F 08 5F 48 49 44 0C 41 D0 0B 00 14 21  RTC_._HID.A....!
  3A40: 5F 43 52 53 00 08 52 54 43 42 11 10 0A 0D 47 01  _CRS..RTCB....G.
  3A50: 70 00 70 00 01 10 22 00 01 79 00 A4 52 54 43 42  p.p..."..y..RTCB
  3A60: 10 43 04 2F 03 5F 53 42 5F 50 43 49 30 49 53 41  .C./._SB_PCI0ISA
  3A70: 5F 5B 82 31 54 4D 52 5F 08 5F 48 49 44 0C 41 D0  _[.1TMR_._HID.A.
  3A80: 01 00 14 21 5F 43 52 53 00 08 54 4D 52 42 11 10  ...!_CRS..TMRB..
  3A90: 0A 0D 47 01 40 00 40 00 01 20 22 01 00 79 00 A4  ..G.@.@.. "..y..
  3AA0: 54 4D 52 42 10 4A 5E 5F 53 42 5F 5B 81 24 2F 03  TMRB.J^_SB_[.$/.
  3AB0: 50 43 49 30 49 53 41 5F 50 34 30 43 01 50 52 51  PCI0ISA_P40C.PRQ
  3AC0: 30 08 50 52 51 31 08 50 52 51 32 08 50 52 51 33  0.PRQ1.PRQ2.PRQ3
  3AD0: 08 5B 81 24 2F 03 50 43 49 30 49 53 41 5F 50 34  .[.$/.PCI0ISA_P4
  3AE0: 31 43 01 50 52 51 34 08 50 52 51 35 08 50 52 51  1C.PRQ4.PRQ5.PRQ
  3AF0: 36 08 50 52 51 37 08 5B 82 41 0B 4C 4E 4B 41 08  6.PRQ7.[.A.LNKA.
  3B00: 5F 48 49 44 0C 41 D0 0C 0F 08 5F 55 49 44 0A 0C  _HID.A...._UID..
  3B10: 08 5F 50 52 53 11 09 0A 06 23 F8 9E 18 79 00 14  ._PRS....#...y..
  3B20: 1A 5F 53 54 41 00 70 0A 0B 60 A0 0D 7B 0A 80 50  ._STA.p..`..{..P
  3B30: 52 51 30 61 70 0A 09 60 A4 60 14 11 5F 44 49 53  RQ0ap..`.`.._DIS
  3B40: 00 7D 50 52 51 30 0A 80 50 52 51 30 14 41 04 5F  .}PRQ0..PRQ0.A._
  3B50: 43 52 53 00 08 50 52 52 30 11 09 0A 06 23 10 00  CRS..PRR0....#..
  3B60: 18 79 00 8B 50 52 52 30 01 49 51 52 5F 70 50 52  .y..PRR0.IQR_pPR
  3B70: 51 30 60 A0 0C 92 95 60 0A 80 70 00 49 51 52 5F  Q0`....`..p.IQR_
  3B80: A1 08 79 01 60 49 51 52 5F A4 50 52 52 30 14 1B  ..y.`IQR_.PRR0..
  3B90: 5F 53 52 53 01 8B 68 01 49 51 52 5F 82 49 51 52  _SRS..h.IQR_.IQR
  3BA0: 5F 60 76 60 70 60 50 52 51 30 5B 82 41 0B 4C 4E  _`v`p`PRQ0[.A.LN
  3BB0: 4B 42 08 5F 48 49 44 0C 41 D0 0C 0F 08 5F 55 49  KB._HID.A...._UI
  3BC0: 44 0A 0D 08 5F 50 52 53 11 09 0A 06 23 F8 9E 18  D..._PRS....#...
  3BD0: 79 00 14 1A 5F 53 54 41 00 70 0A 0B 60 A0 0D 7B  y..._STA.p..`..{
  3BE0: 0A 80 50 52 51 31 61 70 0A 09 60 A4 60 14 11 5F  ..PRQ1ap..`.`.._
  3BF0: 44 49 53 00 7D 50 52 51 31 0A 80 50 52 51 31 14  DIS.}PRQ1..PRQ1.
  3C00: 41 04 5F 43 52 53 00 08 50 52 52 31 11 09 0A 06  A._CRS..PRR1....
  3C10: 23 10 00 18 79 00 8B 50 52 52 31 01 49 51 52 5F  #...y..PRR1.IQR_
  3C20: 70 50 52 51 31 60 A0 0C 92 95 60 0A 80 70 00 49  pPRQ1`....`..p.I
  3C30: 51 52 5F A1 08 79 01 60 49 51 52 5F A4 50 52 52  QR_..y.`IQR_.PRR
  3C40: 31 14 1B 5F 53 52 53 01 8B 68 01 49 51 52 5F 82  1.._SRS..h.IQR_.
  3C50: 49 51 52 5F 60 76 60 70 60 50 52 51 31 5B 82 41  IQR_`v`p`PRQ1[.A
  3C60: 0B 4C 4E 4B 43 08 5F 48 49 44 0C 41 D0 0C 0F 08  .LNKC._HID.A....
  3C70: 5F 55 49 44 0A 0E 08 5F 50 52 53 11 09 0A 06 23  _UID..._PRS....#
  3C80: F8 9E 18 79 00 14 1A 5F 53 54 41 00 70 0A 0B 60  ...y..._STA.p..`
  3C90: A0 0D 7B 0A 80 50 52 51 32 61 70 0A 09 60 A4 60  ..{..PRQ2ap..`.`
  3CA0: 14 11 5F 44 49 53 00 7D 50 52 51 32 0A 80 50 52  .._DIS.}PRQ2..PR
  3CB0: 51 32 14 41 04 5F 43 52 53 00 08 50 52 52 32 11  Q2.A._CRS..PRR2.
  3CC0: 09 0A 06 23 10 00 18 79 00 8B 50 52 52 32 01 49  ...#...y..PRR2.I
  3CD0: 51 52 5F 70 50 52 51 32 60 A0 0C 92 95 60 0A 80  QR_pPRQ2`....`..
  3CE0: 70 00 49 51 52 5F A1 08 79 01 60 49 51 52 5F A4  p.IQR_..y.`IQR_.
  3CF0: 50 52 52 32 14 1B 5F 53 52 53 01 8B 68 01 49 51  PRR2.._SRS..h.IQ
  3D00: 52 5F 82 49 51 52 5F 60 76 60 70 60 50 52 51 32  R_.IQR_`v`p`PRQ2
  3D10: 5B 82 41 0B 4C 4E 4B 44 08 5F 48 49 44 0C 41 D0  [.A.LNKD._HID.A.
  3D20: 0C 0F 08 5F 55 49 44 0A 0F 08 5F 50 52 53 11 09  ..._UID..._PRS..
  3D30: 0A 06 23 F8 9E 18 79 00 14 1A 5F 53 54 41 00 70  ..#...y..._STA.p
  3D40: 0A 0B 60 A0 0D 7B 0A 80 50 52 51 33 61 70 0A 09  ..`..{..PRQ3ap..
  3D50: 60 A4 60 14 11 5F 44 49 53 00 7D 50 52 51 33 0A  `.`.._DIS.}PRQ3.
  3D60: 80 50 52 51 33 14 41 04 5F 43 52 53 00 08 50 52  .PRQ3.A._CRS..PR
  3D70: 52 33 11 09 0A 06 23 10 00 18 79 00 8B 50 52 52  R3....#...y..PRR
  3D80: 33 01 49 51 52 5F 70 50 52 51 33 60 A0 0C 92 95  3.IQR_pPRQ3`....
  3D90: 60 0A 80 70 00 49 51 52 5F A1 08 79 01 60 49 51  `..p.IQR_..y.`IQ
  3DA0: 52 5F A4 50 52 52 33 14 1B 5F 53 52 53 01 8B 68  R_.PRR3.._SRS..h
  3DB0: 01 49 51 52 5F 82 49 51 52 5F 60 76 60 70 60 50  .IQR_.IQR_`v`p`P
  3DC0: 52 51 33 5B 82 41 0B 4C 4E 4B 45 08 5F 48 49 44  RQ3[.A.LNKE._HID
  3DD0: 0C 41 D0 0C 0F 08 5F 55 49 44 0A 10 08 5F 50 52  .A...._UID..._PR
  3DE0: 53 11 09 0A 06 23 F8 9E 18 79 00 14 1A 5F 53 54  S....#...y..._ST
  3DF0: 41 00 70 0A 0B 60 A0 0D 7B 0A 80 50 52 51 34 61  A.p..`..{..PRQ4a
  3E00: 70 0A 09 60 A4 60 14 11 5F 44 49 53 00 7D 50 52  p..`.`.._DIS.}PR
  3E10: 51 34 0A 80 50 52 51 34 14 41 04 5F 43 52 53 00  Q4..PRQ4.A._CRS.
  3E20: 08 50 52 52 34 11 09 0A 06 23 10 00 18 79 00 8B  .PRR4....#...y..
  3E30: 50 52 52 34 01 49 51 52 5F 70 50 52 51 34 60 A0  PRR4.IQR_pPRQ4`.
  3E40: 0C 92 95 60 0A 80 70 00 49 51 52 5F A1 08 79 01  ...`..p.IQR_..y.
  3E50: 60 49 51 52 5F A4 50 52 52 34 14 1B 5F 53 52 53  `IQR_.PRR4.._SRS
  3E60: 01 8B 68 01 49 51 52 5F 82 49 51 52 5F 60 76 60  ..h.IQR_.IQR_`v`
  3E70: 70 60 50 52 51 34 5B 82 41 0B 4C 4E 4B 46 08 5F  p`PRQ4[.A.LNKF._
  3E80: 48 49 44 0C 41 D0 0C 0F 08 5F 55 49 44 0A 11 08  HID.A...._UID...
  3E90: 5F 50 52 53 11 09 0A 06 23 F8 9E 18 79 00 14 1A  _PRS....#...y...
  3EA0: 5F 53 54 41 00 70 0A 0B 60 A0 0D 7B 0A 80 50 52  _STA.p..`..{..PR
  3EB0: 51 35 61 70 0A 09 60 A4 60 14 11 5F 44 49 53 00  Q5ap..`.`.._DIS.
  3EC0: 7D 50 52 51 35 0A 80 50 52 51 35 14 41 04 5F 43  }PRQ5..PRQ5.A._C
  3ED0: 52 53 00 08 50 52 52 35 11 09 0A 06 23 10 00 18  RS..PRR5....#...
  3EE0: 79 00 8B 50 52 52 35 01 49 51 52 5F 70 50 52 51  y..PRR5.IQR_pPRQ
  3EF0: 35 60 A0 0C 92 95 60 0A 80 70 00 49 51 52 5F A1  5`....`..p.IQR_.
  3F00: 08 79 01 60 49 51 52 5F A4 50 52 52 35 14 1B 5F  .y.`IQR_.PRR5.._
  3F10: 53 52 53 01 8B 68 01 49 51 52 5F 82 49 51 52 5F  SRS..h.IQR_.IQR_
  3F20: 60 76 60 70 60 50 52 51 35 5B 82 41 0B 4C 4E 4B  `v`p`PRQ5[.A.LNK
  3F30: 47 08 5F 48 49 44 0C 41 D0 0C 0F 08 5F 55 49 44  G._HID.A...._UID
  3F40: 0A 12 08 5F 50 52 53 11 09 0A 06 23 F8 9E 18 79  ..._PRS....#...y
  3F50: 00 14 1A 5F 53 54 41 00 70 0A 0B 60 A0 0D 7B 0A  ..._STA.p..`..{.
  3F60: 80 50 52 51 36 61 70 0A 09 60 A4 60 14 11 5F 44  .PRQ6ap..`.`.._D
  3F70: 49 53 00 7D 50 52 51 36 0A 80 50 52 51 36 14 41  IS.}PRQ6..PRQ6.A
  3F80: 04 5F 43 52 53 00 08 50 52 52 36 11 09 0A 06 23  ._CRS..PRR6....#
  3F90: 10 00 18 79 00 8B 50 52 52 36 01 49 51 52 5F 70  ...y..PRR6.IQR_p
  3FA0: 50 52 51 36 60 A0 0C 92 95 60 0A 80 70 00 49 51  PRQ6`....`..p.IQ
  3FB0: 52 5F A1 08 79 01 60 49 51 52 5F A4 50 52 52 36  R_..y.`IQR_.PRR6
  3FC0: 14 1B 5F 53 52 53 01 8B 68 01 49 51 52 5F 82 49  .._SRS..h.IQR_.I
  3FD0: 51 52 5F 60 76 60 70 60 50 52 51 36 5B 82 41 0B  QR_`v`p`PRQ6[.A.
  3FE0: 4C 4E 4B 48 08 5F 48 49 44 0C 41 D0 0C 0F 08 5F  LNKH._HID.A...._
  3FF0: 55 49 44 0A 13 08 5F 50 52 53 11 09 0A 06 23 F8  UID..._PRS....#.
  4000: 9E 18 79 00 14 1A 5F 53 54 41 00 70 0A 0B 60 A0  ..y..._STA.p..`.
  4010: 0D 7B 0A 80 50 52 51 37 61 70 0A 09 60 A4 60 14  .{..PRQ7ap..`.`.
  4020: 11 5F 44 49 53 00 7D 50 52 51 37 0A 80 50 52 51  ._DIS.}PRQ7..PRQ
  4030: 37 14 41 04 5F 43 52 53 00 08 50 52 52 37 11 09  7.A._CRS..PRR7..
  4040: 0A 06 23 10 00 18 79 00 8B 50 52 52 37 01 49 51  ..#...y..PRR7.IQ
  4050: 52 5F 70 50 52 51 37 60 A0 0C 92 95 60 0A 80 70  R_pPRQ7`....`..p
  4060: 00 49 51 52 5F A1 08 79 01 60 49 51 52 5F A4 50  .IQR_..y.`IQR_.P
  4070: 52 52 37 14 1B 5F 53 52 53 01 8B 68 01 49 51 52  RR7.._SRS..h.IQR
  4080: 5F 82 49 51 52 5F 60 76 60 70 60 50 52 51 37 10  _.IQR_`v`p`PRQ7.
  4090: 39 5F 53 42 5F 5B 82 32 48 50 45 54 08 5F 48 49  9_SB_[.2HPET._HI
  40A0: 44 0C 41 D0 01 03 14 22 5F 43 52 53 00 08 48 42  D.A...."_CRS..HB
  40B0: 41 42 11 11 0A 0E 86 09 00 00 00 00 D0 FE 00 04  AB..............
  40C0: 00 00 79 00 A4 48 42 41 42 10 4A 16 5C 00 5B 80  ..y..HBAB.J.\.[.
  40D0: 57 45 4E 58 01 0B 0A 0C 0A 06 5B 81 24 57 45 4E  WENX......[.$WEN
  40E0: 58 01 57 45 4E 31 08 57 45 4E 32 08 57 45 4E 33  X.WEN1.WEN2.WEN3
  40F0: 08 57 45 4E 34 08 57 45 4E 35 08 57 45 4E 36 08  .WEN4.WEN5.WEN6.
  4100: 5B 80 57 53 54 58 01 0B 00 0C 0A 0A 5B 81 38 57  [.WSTX......[.8W
  4110: 53 54 58 01 57 53 54 30 08 50 4D 45 4A 08 50 45  STX.WST0.PMEJ.PE
  4120: 4E 41 08 50 45 4E 4A 08 57 53 54 31 08 57 53 54  NA.PENJ.WST1.WST
  4130: 32 08 57 53 54 33 08 57 53 54 34 08 57 53 54 35  2.WST3.WST4.WST5
  4140: 08 57 53 54 36 08 5B 80 53 47 50 58 01 0B 4B 0C  .WST6.[.SGPX..K.
  4150: 0A 06 5B 81 24 53 47 50 58 01 53 47 50 31 08 53  ..[.$SGPX.SGP1.S
  4160: 47 50 32 08 53 47 50 33 08 53 47 50 34 08 53 47  GP2.SGP3.SGP4.SG
  4170: 50 35 08 53 47 50 36 08 5B 80 4C 45 44 58 01 0B  P5.SGP6.[.LEDX..
  4180: 5D 0C 0A 02 5B 81 10 4C 45 44 58 01 4C 45 44 31  ]...[..LEDX.LED1
  4190: 08 4C 45 44 32 08 14 4D 06 50 53 4B 4D 01 A0 45  .LED2..M.PSKM..E
  41A0: 06 7D 93 68 0A 03 93 68 01 00 70 0A 55 53 49 4F  .}.h...h..p.USIO
  41B0: 49 70 0A 07 53 49 4F 49 70 0A 07 53 49 4F 44 70  Ip..SIOIp..SIODp
  41C0: 0A F0 53 49 4F 49 70 53 49 4F 44 60 7D 60 0A 60  ..SIOIpSIOD`}`.`
  41D0: 53 49 4F 44 70 0A AA 53 49 4F 49 70 57 53 54 31  SIODp..SIOIpWST1
  41E0: 60 7B 60 0A 18 57 53 54 31 70 47 53 54 53 60 7B  `{`..WST1pGSTS`{
  41F0: 60 01 47 53 54 53 70 57 45 4E 31 60 7D 60 0A 18  `.GSTSpWEN1`}`..
  4200: 57 45 4E 31 14 2F 47 50 4B 4D 00 70 57 45 4E 31  WEN1./GPKM.pWEN1
  4210: 60 7B 60 0A E7 57 45 4E 31 70 57 53 54 31 60 7B  `{`..WEN1pWST1`{
  4220: 60 0A 18 57 53 54 31 70 47 53 54 53 60 7B 60 01  `..WST1pGSTS`{`.
  4230: 47 53 54 53 10 41 D7 2F 03 5F 53 42 5F 50 43 49  GSTS.A./._SB_PCI
  4240: 30 49 53 41 5F 5B 80 4E 53 49 4F 01 0A 2E 0A 02  0ISA_[.NSIO.....
  4250: 5B 81 10 4E 53 49 4F 01 49 4E 44 58 08 44 41 54  [..NSIO.INDX.DAT
  4260: 41 08 5B 86 47 05 49 4E 44 58 44 41 54 41 01 00  A.[.G.INDXDATA..
  4270: 10 43 46 47 5F 08 00 20 4C 44 4E 5F 08 00 40 0C  .CFG_.. LDN_..@.
  4280: 53 49 49 44 08 00 48 07 41 43 54 52 08 00 48 17  SIID..H.ACTR..H.
  4290: 49 4F 41 48 08 49 4F 41 4C 08 00 40 07 49 4E 54  IOAH.IOAL..@.INT
  42A0: 52 08 00 18 44 4D 43 48 08 00 48 3D 4F 50 54 31  R...DMCH..H=OPT1
  42B0: 08 4F 50 54 32 08 4F 50 54 33 08 5B 80 43 4F 4D  .OPT2.OPT3.[.COM
  42C0: 5F 02 0A 80 01 5B 81 14 43 4F 4D 5F 01 4E 53 43  _....[..COM_.NSC
  42D0: 41 03 00 01 4E 53 43 42 03 00 01 5B 80 46 44 50  A...NSCB...[.FDP
  42E0: 41 02 0A 81 01 5B 81 14 46 44 50 41 01 4C 50 54  A....[..FDPA.LPT
  42F0: 4F 02 00 02 46 44 44 43 01 00 03 5B 01 4D 54 58  O...FDDC...[.MTX
  4300: 5F 01 14 18 53 49 4F 44 01 43 53 49 4F 0A 55 70  _...SIOD.CSIO.Up
  4310: 68 4C 44 4E 5F 43 53 49 4F 0A AA 14 0C 43 53 49  hLDN_CSIO....CSI
  4320: 4F 01 70 68 49 4E 44 58 5B 82 41 2C 46 44 43 5F  O.phINDX[.A,FDC_
  4330: 08 5F 48 49 44 0C 41 D0 07 00 14 41 06 5F 53 54  ._HID.A....A._ST
  4340: 41 00 08 52 45 54 5F 00 5B 23 4D 54 58 5F FF FF  A..RET_.[#MTX_..
  4350: 53 49 4F 44 00 A0 2D 47 43 46 44 43 53 49 4F 0A  SIOD..-GCFDCSIO.
  4360: 55 A0 12 41 43 54 52 43 53 49 4F 0A AA 70 0A 0F  U..ACTRCSIO..p..
  4370: 52 45 54 5F A1 0E 43 53 49 4F 0A AA 70 0A 0D 52  RET_..CSIO..p..R
  4380: 45 54 5F A1 0D 43 53 49 4F 0A AA 70 00 52 45 54  ET_..CSIO..p.RET
  4390: 5F 5B 27 4D 54 58 5F A4 52 45 54 5F 14 31 5F 44  _['MTX_.RET_.1_D
  43A0: 49 53 00 5B 23 4D 54 58 5F FF FF 53 49 4F 44 00  IS.[#MTX_..SIOD.
  43B0: 43 53 49 4F 0A 55 70 00 49 4E 54 52 70 00 41 43  CSIO.Up.INTRp.AC
  43C0: 54 52 43 53 49 4F 0A AA 5B 27 4D 54 58 5F 14 4A  TRCSIO..['MTX_.J
  43D0: 11 5F 43 52 53 00 08 46 44 42 30 11 1B 0A 18 47  ._CRS..FDB0....G
  43E0: 01 F0 03 F0 03 08 06 47 01 F7 03 F7 03 01 01 22  .......G......."
  43F0: 40 00 2A 04 08 79 00 8C 46 44 42 30 0A 02 49 4F  @.*..y..FDB0..IO
  4400: 4C 30 8C 46 44 42 30 0A 03 49 4F 48 30 8C 46 44  L0.FDB0..IOH0.FD
  4410: 42 30 0A 04 49 4F 4C 31 8C 46 44 42 30 0A 05 49  B0..IOL1.FDB0..I
  4420: 4F 48 31 8C 46 44 42 30 0A 0A 49 32 4C 30 8C 46  OH1.FDB0..I2L0.F
  4430: 44 42 30 0A 0B 49 32 48 30 8C 46 44 42 30 0A 0C  DB0..I2H0.FDB0..
  4440: 49 32 4C 31 8C 46 44 42 30 0A 0D 49 32 48 31 8B  I2L1.FDB0..I2H1.
  4450: 46 44 42 30 0A 11 49 51 52 5F 8C 46 44 42 30 0A  FDB0..IQR_.FDB0.
  4460: 14 44 41 4D 5F 5B 23 4D 54 58 5F FF FF 53 49 4F  .DAM_[#MTX_..SIO
  4470: 44 00 43 53 49 4F 0A 55 70 49 4F 41 48 49 4F 48  D.CSIO.UpIOAHIOH
  4480: 30 70 49 4F 41 48 49 4F 48 31 70 49 4F 41 4C 49  0pIOAHIOH1pIOALI
  4490: 4F 4C 30 70 49 4F 41 4C 49 4F 4C 31 70 49 4F 41  OL0pIOALIOL1pIOA
  44A0: 48 49 32 48 30 70 49 4F 41 48 49 32 48 31 72 49  HI2H0pIOAHI2H1rI
  44B0: 4F 41 4C 0A 07 49 32 4C 30 72 49 4F 41 4C 0A 07  OAL..I2L0rIOAL..
  44C0: 49 32 4C 31 79 01 49 4E 54 52 49 51 52 5F 79 01  I2L1y.INTRIQR_y.
  44D0: 44 4D 43 48 44 41 4D 5F 43 53 49 4F 0A AA 5B 27  DMCHDAM_CSIO..['
  44E0: 4D 54 58 5F A4 46 44 42 30 14 4E 05 5F 50 52 53  MTX_.FDB0.N._PRS
  44F0: 00 08 46 44 42 31 11 4C 04 0A 48 30 47 01 F0 03  ..FDB1.L..H0G...
  4500: F0 03 08 06 47 01 F7 03 F7 03 01 01 22 40 00 2A  ....G......."@.*
  4510: 04 08 30 47 01 F0 03 F0 03 08 06 47 01 F7 03 F7  ..0G.......G....
  4520: 03 01 01 22 F8 10 2A 0E 08 30 47 01 70 03 70 03  ..."..*..0G.p.p.
  4530: 08 06 47 01 77 03 77 03 01 01 22 F8 10 2A 0E 08  ..G.w.w..."..*..
  4540: 38 79 00 A4 46 44 42 31 14 42 0A 5F 53 52 53 01  8y..FDB1.B._SRS.
  4550: 8C 68 0A 02 49 4F 4C 4F 8C 68 0A 03 49 4F 48 49  .h..IOLO.h..IOHI
  4560: 8B 68 0A 11 49 51 52 5F 8C 68 0A 14 44 41 4D 5F  .h..IQR_.h..DAM_
  4570: 5B 23 4D 54 58 5F FF FF 53 49 4F 44 00 43 53 49  [#MTX_..SIOD.CSI
  4580: 4F 0A 55 70 49 4F 4C 4F 49 4F 41 4C 70 49 4F 48  O.UpIOLOIOALpIOH
  4590: 49 49 4F 41 48 43 53 49 4F 0A AA A0 0E 93 49 4F  IIOAHCSIO.....IO
  45A0: 4C 4F 0A 70 70 01 46 44 44 43 A0 0E 93 49 4F 4C  LO.pp.FDDC...IOL
  45B0: 4F 0A F0 70 00 46 44 44 43 43 53 49 4F 0A 55 82  O..p.FDDCCSIO.U.
  45C0: 49 51 52 5F 60 74 60 01 49 4E 54 52 82 44 41 4D  IQR_`t`.INTR.DAM
  45D0: 5F 60 74 60 01 44 4D 43 48 70 01 41 43 54 52 43  _`t`.DMCHp.ACTRC
  45E0: 53 49 4F 0A AA 5B 27 4D 54 58 5F 5B 82 4F 05 4B  SIO..['MTX_[.O.K
  45F0: 42 44 5F 08 5F 48 49 44 0C 41 D0 03 03 14 24 5F  BD_._HID.A....$_
  4600: 53 54 41 00 08 52 45 54 5F 00 70 00 52 45 54 5F  STA..RET_.p.RET_
  4610: A0 0C 47 43 4B 42 70 0A 0F 52 45 54 5F A4 52 45  ..GCKBp..RET_.RE
  4620: 54 5F 14 29 5F 43 52 53 00 08 4B 42 44 42 11 18  T_.)_CRS..KBDB..
  4630: 0A 15 47 01 60 00 60 00 01 01 47 01 64 00 64 00  ..G.`.`...G.d.d.
  4640: 01 01 22 02 00 79 00 A4 4B 42 44 42 5B 82 41 08  .."..y..KBDB[.A.
  4650: 4D 4F 55 5F 08 5F 48 49 44 0C 41 D0 0F 13 14 2A  MOU_._HID.A....*
  4660: 5F 53 54 41 00 08 52 45 54 5F 00 70 00 52 45 54  _STA..RET_.p.RET
  4670: 5F A0 12 47 43 4F 4E A0 0C 47 43 4D 53 70 0A 0F  _..GCON..GCMSp..
  4680: 52 45 54 5F A4 52 45 54 5F 14 45 04 5F 43 52 53  RET_.RET_.E._CRS
  4690: 00 08 4D 4F 55 42 11 08 0A 05 22 00 10 79 00 08  ..MOUB...."..y..
  46A0: 4D 4F 4B 42 11 18 0A 15 47 01 60 00 60 00 01 01  MOKB....G.`.`...
  46B0: 47 01 64 00 64 00 01 01 22 00 10 79 00 A0 0A 47  G.d.d..."..y...G
  46C0: 43 4B 42 A4 4D 4F 55 42 A1 06 A4 4D 4F 4B 42 5B  CKB.MOUB...MOKB[
  46D0: 82 49 40 50 52 54 5F 08 5F 48 49 44 0C 41 D0 04  .I@PRT_._HID.A..
  46E0: 01 14 42 06 5F 53 54 41 00 08 52 45 54 5F 00 5B  ..B._STA..RET_.[
  46F0: 23 4D 54 58 5F FF FF 53 49 4F 44 0A 03 A0 2D 47  #MTX_..SIOD...-G
  4700: 43 50 50 43 53 49 4F 0A 55 A0 12 41 43 54 52 43  CPPCSIO.U..ACTRC
  4710: 53 49 4F 0A AA 70 0A 0F 52 45 54 5F A1 0E 43 53  SIO..p..RET_..CS
  4720: 49 4F 0A AA 70 0A 0D 52 45 54 5F A1 0D 43 53 49  IO..p..RET_..CSI
  4730: 4F 0A AA 70 00 52 45 54 5F 5B 27 4D 54 58 5F A4  O..p.RET_['MTX_.
  4740: 52 45 54 5F 14 39 5F 44 49 53 00 5B 23 4D 54 58  RET_.9_DIS.[#MTX
  4750: 5F FF FF 53 49 4F 44 0A 03 43 53 49 4F 0A 55 70  _..SIOD..CSIO.Up
  4760: 00 49 4E 54 52 70 0A 04 44 4D 43 48 70 00 41 43  .INTRp..DMCHp.AC
  4770: 54 52 43 53 49 4F 0A AA 5B 27 4D 54 58 5F 14 49  TRCSIO..['MTX_.I
  4780: 18 5F 43 52 53 00 08 50 54 42 30 11 1B 0A 18 47  ._CRS..PTB0....G
  4790: 01 78 03 78 03 08 08 47 01 78 07 78 07 08 08 22  .x.x...G.x.x..."
  47A0: 80 00 2A 00 08 79 00 8C 50 54 42 30 0A 02 49 4F  ..*..y..PTB0..IO
  47B0: 4C 30 8C 50 54 42 30 0A 03 49 4F 48 30 8C 50 54  L0.PTB0..IOH0.PT
  47C0: 42 30 0A 04 49 4F 4C 31 8C 50 54 42 30 0A 05 49  B0..IOL1.PTB0..I
  47D0: 4F 48 31 8C 50 54 42 30 0A 06 4C 41 4C 4E 8C 50  OH1.PTB0..LALN.P
  47E0: 54 42 30 0A 07 4C 4C 45 4E 8C 50 54 42 30 0A 0A  TB0..LLEN.PTB0..
  47F0: 49 32 4C 30 8C 50 54 42 30 0A 0B 49 32 48 30 8C  I2L0.PTB0..I2H0.
  4800: 50 54 42 30 0A 0C 49 32 4C 31 8C 50 54 42 30 0A  PTB0..I2L1.PTB0.
  4810: 0D 49 32 48 31 8C 50 54 42 30 0A 0E 48 41 4C 4E  .I2H1.PTB0..HALN
  4820: 8C 50 54 42 30 0A 0F 48 4C 45 4E 8B 50 54 42 30  .PTB0..HLEN.PTB0
  4830: 0A 11 49 51 52 5F 8C 50 54 42 30 0A 14 44 41 4D  ..IQR_.PTB0..DAM
  4840: 5F 5B 23 4D 54 58 5F FF FF 53 49 4F 44 0A 03 43  _[#MTX_..SIOD..C
  4850: 53 49 4F 0A 55 70 49 4F 41 48 49 4F 48 30 70 49  SIO.UpIOAHIOH0pI
  4860: 4F 41 48 49 4F 48 31 70 49 4F 41 4C 49 4F 4C 30  OAHIOH1pIOALIOL0
  4870: 70 49 4F 41 4C 49 4F 4C 31 72 49 4F 41 48 0A 04  pIOALIOL1rIOAH..
  4880: 49 32 48 30 72 49 4F 41 48 0A 04 49 32 48 31 70  I2H0rIOAH..I2H1p
  4890: 49 4F 41 4C 49 32 4C 30 70 49 4F 41 4C 49 32 4C  IOALI2L0pIOALI2L
  48A0: 31 79 01 49 4E 54 52 49 51 52 5F 79 01 44 4D 43  1y.INTRIQR_y.DMC
  48B0: 48 44 41 4D 5F A0 24 93 49 4F 4C 30 0A BC 70 0A  HDAM_.$.IOL0..p.
  48C0: 04 4C 41 4C 4E 70 0A 04 4C 4C 45 4E 70 0A 04 48  .LALNp..LLENp..H
  48D0: 41 4C 4E 70 0A 04 48 4C 45 4E A0 0E 93 44 4D 43  ALNp..HLEN...DMC
  48E0: 48 0A 04 70 00 44 41 4D 5F A0 0D 93 49 4E 54 52  H..p.DAM_...INTR
  48F0: 00 70 00 49 51 52 5F 43 53 49 4F 0A AA 5B 27 4D  .p.IQR_CSIO..['M
  4900: 54 58 5F A4 50 54 42 30 14 4F 0F 5F 50 52 53 00  TX_.PTB0.O._PRS.
  4910: 08 50 54 42 31 11 4D 0E 0A E9 30 47 01 78 03 78  .PTB1.M...0G.x.x
  4920: 03 08 08 47 01 78 07 78 07 08 08 22 80 00 2A 00  ...G.x.x..."..*.
  4930: 08 30 47 01 78 02 78 02 08 08 47 01 78 06 78 06  .0G.x.x...G.x.x.
  4940: 08 08 22 20 00 2A 00 08 30 47 01 78 03 78 03 08  .." .*..0G.x.x..
  4950: 08 47 01 78 07 78 07 08 08 22 F8 10 2A 00 08 30  .G.x.x..."..*..0
  4960: 47 01 78 03 78 03 08 08 47 01 78 07 78 07 08 08  G.x.x...G.x.x...
  4970: 22 F8 10 2A 0E 08 30 47 01 78 02 78 02 08 08 47  "..*..0G.x.x...G
  4980: 01 78 06 78 06 08 08 22 F8 10 2A 00 08 30 47 01  .x.x..."..*..0G.
  4990: 78 02 78 02 08 08 47 01 78 06 78 06 08 08 22 F8  x.x...G.x.x...".
  49A0: 10 2A 0E 08 30 47 01 78 03 78 03 08 08 47 01 78  .*..0G.x.x...G.x
  49B0: 07 78 07 08 08 22 00 00 2A 00 08 30 47 01 78 02  .x..."..*..0G.x.
  49C0: 78 02 08 08 47 01 78 06 78 06 08 08 22 00 00 2A  x...G.x.x..."..*
  49D0: 00 08 30 47 01 BC 03 BC 03 04 04 47 01 BC 07 BC  ..0G.......G....
  49E0: 07 04 04 22 00 00 2A 00 08 30 47 01 BC 03 BC 03  ..."..*..0G.....
  49F0: 04 04 47 01 BC 07 BC 07 04 04 22 80 00 2A 00 08  ..G......."..*..
  4A00: 38 79 00 A4 50 54 42 31 14 41 0D 5F 53 52 53 01  8y..PTB1.A._SRS.
  4A10: 8C 68 0A 02 49 4F 4C 4F 8C 68 0A 03 49 4F 48 49  .h..IOLO.h..IOHI
  4A20: 8B 68 0A 11 49 51 52 5F 8C 68 0A 14 44 41 4D 5F  .h..IQR_.h..DAM_
  4A30: 5B 23 4D 54 58 5F FF FF 53 49 4F 44 0A 03 43 53  [#MTX_..SIOD..CS
  4A40: 49 4F 0A 55 70 49 4F 4C 4F 49 4F 41 4C 70 49 4F  IO.UpIOLOIOALpIO
  4A50: 48 49 49 4F 41 48 A0 0F 93 49 4F 4C 4F 0A BC 70  HIIOAH...IOLO..p
  4A60: 0A 02 4C 50 54 4F A0 26 93 49 4F 4C 4F 0A 78 A0  ..LPTO.&.IOLO.x.
  4A70: 0E 93 49 4F 48 49 0A 02 70 01 4C 50 54 4F A0 0E  ..IOHI..p.LPTO..
  4A80: 93 49 4F 48 49 0A 03 70 00 4C 50 54 4F A0 0D 93  .IOHI..p.LPTO...
  4A90: 49 51 52 5F 00 70 00 49 4E 54 52 A1 0E 82 49 51  IQR_.p.INTR...IQ
  4AA0: 52 5F 60 74 60 01 49 4E 54 52 A0 0E 93 44 41 4D  R_`t`.INTR...DAM
  4AB0: 5F 00 70 0A 04 44 4D 43 48 A1 0E 82 44 41 4D 5F  _.p..DMCH...DAM_
  4AC0: 60 74 60 01 44 4D 43 48 70 01 41 43 54 52 43 53  `t`.DMCHp.ACTRCS
  4AD0: 49 4F 0A AA 5B 27 4D 54 58 5F 5B 82 42 25 43 4F  IO..['MTX_[.B%CO
  4AE0: 4D 41 08 5F 48 49 44 0C 41 D0 05 01 08 5F 55 49  MA._HID.A...._UI
  4AF0: 44 01 14 42 06 5F 53 54 41 00 08 52 45 54 5F 00  D..B._STA..RET_.
  4B00: 5B 23 4D 54 58 5F FF FF 53 49 4F 44 0A 04 A0 2D  [#MTX_..SIOD...-
  4B10: 47 43 53 31 43 53 49 4F 0A 55 A0 12 41 43 54 52  GCS1CSIO.U..ACTR
  4B20: 43 53 49 4F 0A AA 70 0A 0F 52 45 54 5F A1 0E 43  CSIO..p..RET_..C
  4B30: 53 49 4F 0A AA 70 0A 0D 52 45 54 5F A1 0D 43 53  SIO..p..RET_..CS
  4B40: 49 4F 0A AA 70 00 52 45 54 5F 5B 27 4D 54 58 5F  IO..p.RET_['MTX_
  4B50: A4 52 45 54 5F 14 32 5F 44 49 53 00 5B 23 4D 54  .RET_.2_DIS.[#MT
  4B60: 58 5F FF FF 53 49 4F 44 0A 04 43 53 49 4F 0A 55  X_..SIOD..CSIO.U
  4B70: 70 00 49 4E 54 52 70 00 41 43 54 52 43 53 49 4F  p.INTRp.ACTRCSIO
  4B80: 0A AA 5B 27 4D 54 58 5F 14 47 0A 5F 43 52 53 00  ..['MTX_.G._CRS.
  4B90: 08 43 4D 41 30 11 10 0A 0D 47 01 F8 03 F8 03 08  .CMA0....G......
  4BA0: 08 22 10 00 79 00 8C 43 4D 41 30 0A 02 49 4F 4C  ."..y..CMA0..IOL
  4BB0: 30 8C 43 4D 41 30 0A 03 49 4F 48 30 8C 43 4D 41  0.CMA0..IOH0.CMA
  4BC0: 30 0A 04 49 4F 4C 31 8C 43 4D 41 30 0A 05 49 4F  0..IOL1.CMA0..IO
  4BD0: 48 31 8B 43 4D 41 30 0A 09 49 51 52 5F 5B 23 4D  H1.CMA0..IQR_[#M
  4BE0: 54 58 5F FF FF 53 49 4F 44 0A 04 43 53 49 4F 0A  TX_..SIOD..CSIO.
  4BF0: 55 70 49 4F 41 4C 49 4F 4C 30 70 49 4F 41 4C 49  UpIOALIOL0pIOALI
  4C00: 4F 4C 31 70 49 4F 41 48 49 4F 48 30 70 49 4F 41  OL1pIOAHIOH0pIOA
  4C10: 48 49 4F 48 31 79 01 49 4E 54 52 49 51 52 5F 43  HIOH1y.INTRIQR_C
  4C20: 53 49 4F 0A AA 5B 27 4D 54 58 5F A4 43 4D 41 30  SIO..['MTX_.CMA0
  4C30: 14 48 04 5F 50 52 53 00 08 43 4D 41 31 11 36 0A  .H._PRS..CMA1.6.
  4C40: 33 30 47 01 F8 03 F8 03 08 08 22 10 00 30 47 01  30G......."..0G.
  4C50: E8 03 E8 03 08 08 22 10 00 30 47 01 F8 02 F8 02  ......"..0G.....
  4C60: 08 08 22 08 00 30 47 01 E8 02 E8 02 08 08 22 08  .."..0G.......".
  4C70: 00 38 79 00 A4 43 4D 41 31 14 44 0B 5F 53 52 53  .8y..CMA1.D._SRS
  4C80: 01 8C 68 0A 02 49 4F 4C 4F 8C 68 0A 03 49 4F 48  ..h..IOLO.h..IOH
  4C90: 49 8B 68 0A 09 49 51 52 5F 5B 23 4D 54 58 5F FF  I.h..IQR_[#MTX_.
  4CA0: FF 53 49 4F 44 0A 04 43 53 49 4F 0A 55 70 49 4F  .SIOD..CSIO.UpIO
  4CB0: 4C 4F 49 4F 41 4C 70 49 4F 48 49 49 4F 41 48 82  LOIOALpIOHIIOAH.
  4CC0: 49 51 52 5F 60 74 60 01 49 4E 54 52 A0 27 93 49  IQR_`t`.INTR.'.I
  4CD0: 4F 48 49 0A 03 A0 0E 93 49 4F 4C 4F 0A F8 70 00  OHI.....IOLO..p.
  4CE0: 4E 53 43 41 A0 0F 93 49 4F 4C 4F 0A E8 70 0A 07  NSCA...IOLO..p..
  4CF0: 4E 53 43 41 A0 27 93 49 4F 48 49 0A 02 A0 0E 93  NSCA.'.IOHI.....
  4D00: 49 4F 4C 4F 0A F8 70 01 4E 53 43 41 A0 0F 93 49  IOLO..p.NSCA...I
  4D10: 4F 4C 4F 0A E8 70 0A 05 4E 53 43 41 70 01 41 43  OLO..p..NSCAp.AC
  4D20: 54 52 43 53 49 4F 0A AA 5B 27 4D 54 58 5F 5B 82  TRCSIO..['MTX_[.
  4D30: 46 27 43 4F 4D 42 08 5F 48 49 44 0C 41 D0 05 01  F'COMB._HID.A...
  4D40: 08 5F 55 49 44 0A 02 14 42 06 5F 53 54 41 00 08  ._UID...B._STA..
  4D50: 52 45 54 5F 00 5B 23 4D 54 58 5F FF FF 53 49 4F  RET_.[#MTX_..SIO
  4D60: 44 0A 05 A0 2D 47 43 53 32 43 53 49 4F 0A 55 A0  D...-GCS2CSIO.U.
  4D70: 12 41 43 54 52 43 53 49 4F 0A AA 70 0A 0F 52 45  .ACTRCSIO..p..RE
  4D80: 54 5F A1 0E 43 53 49 4F 0A AA 70 0A 0D 52 45 54  T_..CSIO..p..RET
  4D90: 5F A1 0D 43 53 49 4F 0A AA 70 00 52 45 54 5F 5B  _..CSIO..p.RET_[
  4DA0: 27 4D 54 58 5F A4 52 45 54 5F 14 44 04 5F 44 49  'MTX_.RET_.D._DI
  4DB0: 53 00 5B 23 4D 54 58 5F FF FF 53 49 4F 44 0A 05  S.[#MTX_..SIOD..
  4DC0: 43 53 49 4F 0A 55 70 00 49 4E 54 52 70 4F 50 54  CSIO.Up.INTRpOPT
  4DD0: 31 60 7D 60 0A 20 60 70 60 4F 50 54 31 70 00 41  1`}`. `p`OPT1p.A
  4DE0: 43 54 52 43 53 49 4F 0A AA 5B 27 4D 54 58 5F 14  CTRCSIO..['MTX_.
  4DF0: 47 0A 5F 43 52 53 00 08 43 4D 42 30 11 10 0A 0D  G._CRS..CMB0....
  4E00: 47 01 F8 03 F8 03 08 08 22 10 00 79 00 8C 43 4D  G......."..y..CM
  4E10: 42 30 0A 02 49 4F 4C 30 8C 43 4D 42 30 0A 03 49  B0..IOL0.CMB0..I
  4E20: 4F 48 30 8C 43 4D 42 30 0A 04 49 4F 4C 31 8C 43  OH0.CMB0..IOL1.C
  4E30: 4D 42 30 0A 05 49 4F 48 31 8B 43 4D 42 30 0A 09  MB0..IOH1.CMB0..
  4E40: 49 51 52 5F 5B 23 4D 54 58 5F FF FF 53 49 4F 44  IQR_[#MTX_..SIOD
  4E50: 0A 05 43 53 49 4F 0A 55 70 49 4F 41 4C 49 4F 4C  ..CSIO.UpIOALIOL
  4E60: 30 70 49 4F 41 4C 49 4F 4C 31 70 49 4F 41 48 49  0pIOALIOL1pIOAHI
  4E70: 4F 48 30 70 49 4F 41 48 49 4F 48 31 79 01 49 4E  OH0pIOAHIOH1y.IN
  4E80: 54 52 49 51 52 5F 43 53 49 4F 0A AA 5B 27 4D 54  TRIQR_CSIO..['MT
  4E90: 58 5F A4 43 4D 42 30 14 48 04 5F 50 52 53 00 08  X_.CMB0.H._PRS..
  4EA0: 43 4D 42 31 11 36 0A 33 30 47 01 F8 02 F8 02 08  CMB1.6.30G......
  4EB0: 08 22 08 00 30 47 01 E8 02 E8 02 08 08 22 08 00  ."..0G......."..
  4EC0: 30 47 01 F8 03 F8 03 08 08 22 10 00 30 47 01 E8  0G......."..0G..
  4ED0: 03 E8 03 08 08 22 10 00 38 79 00 A4 43 4D 42 31  ....."..8y..CMB1
  4EE0: 14 45 0C 5F 53 52 53 01 8C 68 0A 02 49 4F 4C 4F  .E._SRS..h..IOLO
  4EF0: 8C 68 0A 03 49 4F 48 49 8B 68 0A 09 49 51 52 5F  .h..IOHI.h..IQR_
  4F00: 5B 23 4D 54 58 5F FF FF 53 49 4F 44 0A 05 43 53  [#MTX_..SIOD..CS
  4F10: 49 4F 0A 55 70 49 4F 4C 4F 49 4F 41 4C 70 49 4F  IO.UpIOLOIOALpIO
  4F20: 48 49 49 4F 41 48 82 49 51 52 5F 60 74 60 01 49  HIIOAH.IQR_`t`.I
  4F30: 4E 54 52 A0 27 93 49 4F 48 49 0A 03 A0 0E 93 49  NTR.'.IOHI.....I
  4F40: 4F 4C 4F 0A F8 70 00 4E 53 43 42 A0 0F 93 49 4F  OLO..p.NSCB...IO
  4F50: 4C 4F 0A E8 70 0A 07 4E 53 43 42 A0 27 93 49 4F  LO..p..NSCB.'.IO
  4F60: 48 49 0A 02 A0 0E 93 49 4F 4C 4F 0A F8 70 01 4E  HI.....IOLO..p.N
  4F70: 53 43 42 A0 0F 93 49 4F 4C 4F 0A E8 70 0A 05 4E  SCB...IOLO..p..N
  4F80: 53 43 42 70 01 41 43 54 52 70 4F 50 54 31 60 7B  SCBp.ACTRpOPT1`{
  4F90: 60 0A DF 60 70 60 4F 50 54 31 43 53 49 4F 0A AA  `..`p`OPT1CSIO..
  4FA0: 5B 27 4D 54 58 5F 10 42 61 2F 03 5F 53 42 5F 50  ['MTX_.Ba/._SB_P
  4FB0: 43 49 30 49 53 41 5F 5B 82 48 5A 54 50 4D 5F 08  CI0ISA_[.HZTPM_.
  4FC0: 5F 48 49 44 0C 41 D0 0C 31 08 5F 43 49 44 0C 41  _HID.A..1._CID.A
  4FD0: D0 0C 31 14 1B 5F 53 54 41 00 A0 07 47 54 50 4D  ..1.._STA...GTPM
  4FE0: A4 00 A0 08 47 43 54 50 A4 0A 0F A1 03 A4 00 14  ....GCTP........
  4FF0: 33 5F 43 52 53 00 A0 07 47 54 50 4D A4 00 A0 20  3_CRS...GTPM... 
  5000: 47 43 54 50 A4 11 19 0A 16 86 09 00 01 00 00 D4  GCTP............
  5010: FE 00 50 00 00 47 01 B0 0C B0 0C 08 10 79 00 A1  ..P..G.......y..
  5020: 03 A4 00 14 42 15 5F 44 53 4D 04 A0 43 0F 93 96  ....B._DSM..C...
  5030: 68 00 11 13 0A 10 A6 FA DD 3D 1B 36 B4 4E A4 24  h........=.6.N.$
  5040: 8D 10 08 9D 16 53 08 5F 54 5F 30 00 70 99 6A 00  .....S._T_0.p.j.
  5050: 5F 54 5F 30 A0 0C 93 5F 54 5F 30 00 A4 11 03 01  _T_0..._T_0.....
  5060: 3F A1 4D 0B A0 0D 93 5F 54 5F 30 01 A4 0D 31 2E  ?.M...._T_0...1.
  5070: 30 00 A1 4C 0A A0 1B 93 5F 54 5F 30 0A 02 70 99  0..L...._T_0..p.
  5080: 6B 00 60 7B 60 0A 0F 60 43 4D 57 52 0A 7E 60 A4  k.`{`..`CMWR.~`.
  5090: 00 A1 4D 08 A0 1B 93 5F 54 5F 30 0A 03 70 43 4D  ..M...._T_0..pCM
  50A0: 52 44 0A 7E 60 7B 60 0A 0F 60 A4 50 50 52 32 60  RD.~`{`..`.PPR2`
  50B0: A1 4E 06 A0 0B 93 5F 54 5F 30 0A 04 A4 0A 02 A1  .N...._T_0......
  50C0: 4F 05 A0 29 93 5F 54 5F 30 0A 05 70 43 4D 52 44  O..)._T_0..pCMRD
  50D0: 0A 7E 60 7A 60 0A 04 60 70 43 4D 52 44 0A 7F 61  .~`z`..`pCMRD..a
  50E0: 7B 61 0A 7F 61 A4 50 50 52 33 60 61 A1 32 A0 29  {a..a.PPR3`a.2.)
  50F0: 93 5F 54 5F 30 0A 06 08 5F 54 5F 31 0D 00 70 9C  ._T_0..._T_1..p.
  5100: 6B FF 00 5F 54 5F 31 A0 0C 93 5F 54 5F 31 0D 65  k.._T_1..._T_1.e
  5110: 6E 00 A4 00 A1 03 A4 01 A1 06 A4 11 03 01 00 A0  n...............
  5120: 41 05 93 96 68 00 11 13 0A 10 ED 54 60 37 13 CC  A...h......T`7..
  5130: 75 46 90 1C 47 56 D7 F2 D4 5D 08 5F 54 5F 32 00  uF..GV...]._T_2.
  5140: 70 99 6A 00 5F 54 5F 32 A0 0C 93 5F 54 5F 32 00  p.j._T_2..._T_2.
  5150: A4 11 03 01 01 A1 1B A0 19 93 5F 54 5F 32 01 70  .........._T_2.p
  5160: 53 4D 49 5F 0A BC 7B 99 6B 00 01 00 61 A4 99 61  SMI_..{.k...a..a
  5170: 00 A4 11 03 01 00 14 4E 12 50 50 52 32 01 08 5F  .......N.PPR2.._
  5180: 54 5F 30 00 70 99 68 00 5F 54 5F 30 A0 0D 93 5F  T_0.p.h._T_0..._
  5190: 54 5F 30 00 A4 12 04 02 00 00 A1 44 10 A0 0D 93  T_0........D....
  51A0: 5F 54 5F 30 01 A4 12 04 02 00 01 A1 43 0F A0 0F  _T_0........C...
  51B0: 93 5F 54 5F 30 0A 02 A4 12 05 02 00 0A 02 A1 40  ._T_0..........@
  51C0: 0E A0 0F 93 5F 54 5F 30 0A 03 A4 12 05 02 00 0A  ...._T_0........
  51D0: 03 A1 4D 0C A0 0F 93 5F 54 5F 30 0A 04 A4 12 05  ..M...._T_0.....
  51E0: 02 00 0A 04 A1 4A 0B A0 0F 93 5F 54 5F 30 0A 05  .....J...._T_0..
  51F0: A4 12 05 02 00 0A 05 A1 47 0A A0 0F 93 5F 54 5F  ........G...._T_
  5200: 30 0A 06 A4 12 05 02 00 0A 06 A1 44 09 A0 0F 93  0..........D....
  5210: 5F 54 5F 30 0A 07 A4 12 05 02 00 0A 07 A1 41 08  _T_0..........A.
  5220: A0 0F 93 5F 54 5F 30 0A 08 A4 12 05 02 00 0A 08  ..._T_0.........
  5230: A1 4E 06 A0 0F 93 5F 54 5F 30 0A 09 A4 12 05 02  .N...._T_0......
  5240: 00 0A 09 A1 4B 05 A0 0F 93 5F 54 5F 30 0A 0A A4  ....K...._T_0...
  5250: 12 05 02 00 0A 0A A1 48 04 A0 0F 93 5F 54 5F 30  .......H...._T_0
  5260: 0A 0B A4 12 05 02 00 0A 0B A1 35 A0 0F 93 5F 54  ..........5..._T
  5270: 5F 30 0A 0C A4 12 05 02 00 0A 0C A1 23 A0 0F 93  _0..........#...
  5280: 5F 54 5F 30 0A 0D A4 12 05 02 00 0A 0D A1 11 A0  _T_0............
  5290: 0F 93 5F 54 5F 30 0A 0E A4 12 05 02 00 0A 0E A4  .._T_0..........
  52A0: 12 04 02 01 00 14 4B 2B 50 50 52 33 02 A0 4C 13  ......K+PPR3..L.
  52B0: 93 69 00 08 5F 54 5F 30 00 70 99 68 00 5F 54 5F  .i.._T_0.p.h._T_
  52C0: 30 A0 0E 93 5F 54 5F 30 00 A4 12 05 03 00 00 00  0..._T_0........
  52D0: A1 42 11 A0 0E 93 5F 54 5F 30 01 A4 12 05 03 00  .B...._T_0......
  52E0: 01 00 A1 40 10 A0 10 93 5F 54 5F 30 0A 02 A4 12  ...@...._T_0....
  52F0: 06 03 00 0A 02 00 A1 4C 0E A0 10 93 5F 54 5F 30  .......L...._T_0
  5300: 0A 03 A4 12 06 03 00 0A 03 00 A1 48 0D A0 10 93  ...........H....
  5310: 5F 54 5F 30 0A 04 A4 12 06 03 00 0A 04 00 A1 44  _T_0...........D
  5320: 0C A0 10 93 5F 54 5F 30 0A 05 A4 12 06 03 00 0A  ...._T_0........
  5330: 05 00 A1 40 0B A0 10 93 5F 54 5F 30 0A 06 A4 12  ...@...._T_0....
  5340: 06 03 00 0A 06 00 A1 4C 09 A0 10 93 5F 54 5F 30  .......L...._T_0
  5350: 0A 07 A4 12 06 03 00 0A 07 00 A1 48 08 A0 10 93  ...........H....
  5360: 5F 54 5F 30 0A 08 A4 12 06 03 00 0A 08 00 A1 44  _T_0...........D
  5370: 07 A0 10 93 5F 54 5F 30 0A 09 A4 12 06 03 00 0A  ...._T_0........
  5380: 09 00 A1 40 06 A0 10 93 5F 54 5F 30 0A 0A A4 12  ...@...._T_0....
  5390: 06 03 00 0A 0A 00 A1 4C 04 A0 10 93 5F 54 5F 30  .......L...._T_0
  53A0: 0A 0B A4 12 06 03 00 0A 0B 00 A1 38 A0 10 93 5F  ...........8..._
  53B0: 54 5F 30 0A 0C A4 12 06 03 00 0A 0C 00 A1 25 A0  T_0...........%.
  53C0: 10 93 5F 54 5F 30 0A 0D A4 12 06 03 00 0A 0D 00  .._T_0..........
  53D0: A1 12 A0 10 93 5F 54 5F 30 0A 0E A4 12 06 03 00  ....._T_0.......
  53E0: 0A 0E 00 A4 12 05 03 01 00 00 A1 46 17 08 5F 54  ...........F.._T
  53F0: 5F 31 00 70 99 68 00 5F 54 5F 31 A0 12 93 5F 54  _1.p.h._T_1..._T
  5400: 5F 31 00 A4 12 09 03 00 00 0C F0 FF FF FF A1 4B  _1.............K
  5410: 14 A0 12 93 5F 54 5F 31 01 A4 12 09 03 00 01 0C  ...._T_1........
  5420: F0 FF FF FF A1 45 13 A0 14 93 5F 54 5F 31 0A 02  .....E...._T_1..
  5430: A4 12 0A 03 00 0A 02 0C F0 FF FF FF A1 4D 11 A0  .............M..
  5440: 14 93 5F 54 5F 31 0A 03 A4 12 0A 03 00 0A 03 0C  .._T_1..........
  5450: F0 FF FF FF A1 45 10 A0 14 93 5F 54 5F 31 0A 04  .....E...._T_1..
  5460: A4 12 0A 03 00 0A 04 0C F0 FF FF FF A1 4D 0E A0  .............M..
  5470: 14 93 5F 54 5F 31 0A 05 A4 12 0A 03 00 0A 05 0C  .._T_1..........
  5480: F0 FF FF FF A1 45 0D A0 14 93 5F 54 5F 31 0A 06  .....E...._T_1..
  5490: A4 12 0A 03 00 0A 06 0C F0 FF FF FF A1 4D 0B A0  .............M..
  54A0: 14 93 5F 54 5F 31 0A 07 A4 12 0A 03 00 0A 07 0C  .._T_1..........
  54B0: F0 FF FF FF A1 45 0A A0 14 93 5F 54 5F 31 0A 08  .....E...._T_1..
  54C0: A4 12 0A 03 00 0A 08 0C F0 FF FF FF A1 4D 08 A0  .............M..
  54D0: 14 93 5F 54 5F 31 0A 09 A4 12 0A 03 00 0A 09 0C  .._T_1..........
  54E0: F0 FF FF FF A1 45 07 A0 14 93 5F 54 5F 31 0A 0A  .....E...._T_1..
  54F0: A4 12 0A 03 00 0A 0A 0C F0 FF FF FF A1 4D 05 A0  .............M..
  5500: 14 93 5F 54 5F 31 0A 0B A4 12 0A 03 00 0A 0B 0C  .._T_1..........
  5510: F0 FF FF FF A1 45 04 A0 14 93 5F 54 5F 31 0A 0C  .....E...._T_1..
  5520: A4 12 0A 03 00 0A 0C 0C F0 FF FF FF A1 2D A0 14  .............-..
  5530: 93 5F 54 5F 31 0A 0D A4 12 0A 03 00 0A 0D 0C F0  ._T_1...........
  5540: FF FF FF A1 16 A0 14 93 5F 54 5F 31 0A 0E A4 12  ........_T_1....
  5550: 0A 03 00 0A 0E 0C F0 FF FF FF A4 12 05 03 01 00  ................
  5560: 00 5B 82 46 05 54 43 4D 5F 08 5F 48 49 44 0C 69  .[.F.TCM_._HID.i
  5570: 23 01 01 14 1C 5F 53 54 41 00 A0 08 92 47 54 50  #...._STA....GTP
  5580: 4D A4 00 A0 08 47 43 54 50 A4 0A 0F A1 03 A4 00  M....GCTP.......
  5590: 14 28 5F 43 52 53 00 A0 1D 90 47 54 50 4D 47 43  .(_CRS....GTPMGC
  55A0: 54 50 A4 11 11 0A 0E 86 09 00 01 00 00 D4 FE 00  TP..............
  55B0: 50 00 00 79 00 A1 03 A4 00                       P..y.....

FACP @ 0x0000000000000000
  0000: 46 41 43 50 F4 00 00 00 03 23 44 45 4C 4C 20 20  FACP.....#DELL  
  0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20  B10K   .....ASL 
  0020: 61 00 00 00 00 9C DF DB B5 E9 E9 FF 01 05 09 00  a...............
  0030: B2 00 00 00 70 71 00 00 00 08 00 00 00 00 00 00  ....pq..........
  0040: 04 08 00 00 00 00 00 00 50 08 00 00 08 08 00 00  ........P.......
  0050: 20 08 00 00 00 00 00 00 04 02 01 04 10 00 00 00   ...............
  0060: F4 01 88 13 00 00 00 00 00 00 00 00 00 13 00 00  ................
  0070: A5 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0080: 00 00 00 00 00 9C DF DB 00 00 00 00 B5 E9 E9 FF  ................
  0090: 00 00 00 00 01 20 00 01 00 08 00 00 00 00 00 00  ..... ..........
  00A0: 00 00 00 00 00 00 00 00 00 00 00 00 01 10 00 01  ................
  00B0: 04 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  00C0: 00 00 00 00 01 08 00 01 50 08 00 00 00 00 00 00  ........P.......
  00D0: 01 20 00 01 08 08 00 00 00 00 00 00 01 40 00 01  . ...........@..
  00E0: 20 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ...............
  00F0: 00 00 00 00                                      ....

FACS @ 0x0000000000000000
  0000: 46 41 43 53 40 00 00 00 00 00 00 00 00 00 00 00  FACS@...........
  0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

HPET @ 0x0000000000000000
  0000: 48 50 45 54 38 00 00 00 01 0E 44 45 4C 4C 20 20  HPET8.....DELL  
  0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20  B10K   .....ASL 
  0020: 61 00 00 00 01 A3 86 80 00 00 00 00 00 00 D0 FE  a...............
  0030: 00 00 00 00 00 E8 03 00                          ........

MCFG @ 0x0000000000000000
  0000: 4D 43 46 47 3C 00 00 00 01 4A 44 45 4C 4C 20 20  MCFG<....JDELL  
  0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20  B10K   .....ASL 
  0020: 61 00 00 00 00 00 00 00 00 00 00 00 00 00 00 F8  a...............
  0030: 00 00 00 00 00 00 00 3F 00 00 00 00              .......?....

SLIC @ 0x0000000000000000
  0000: 53 4C 49 43 76 01 00 00 01 85 44 45 4C 4C 20 20  SLICv.....DELL  
  0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20  B10K   .....ASL 
  0020: 61 00 00 00 00 00 00 00 9C 00 00 00 06 02 00 00  a...............
  0030: 00 24 00 00 52 53 41 31 00 04 00 00 01 00 01 00  .$..RSA1........
  0040: 7F F6 C1 05 BE 5C 57 63 A5 8A 68 F3 6E 8F 06 FA  .....\Wc..h.n...
  0050: AF B4 9F 68 82 23 EC 50 40 5A 73 7F EC E4 07 CB  ...h.#.P@Zs.....
  0060: DC 25 1A 9C E3 E3 66 11 E0 A5 98 06 C5 80 0A FA  .%....f.........
  0070: 42 93 86 98 E7 D5 1B D4 D7 3A A4 0B EE E2 7D BE  B........:....}.
  0080: 5F 5B 15 0C AB D0 21 DE BF E9 B5 6E A4 57 B9 8C  _[....!....n.W..
  0090: 0C D2 BA 3A 69 30 76 94 71 A2 64 D7 4C D8 85 BF  ...:i0v.q.d.L...
  00A0: DF A5 6A C8 DC 45 D5 4D 8C B8 8C 05 2F FC 2E 23  ..j..E.M..../..#
  00B0: C4 29 C5 6F 3F 29 6C 6D 57 79 0E B6 75 ED 21 95  .).o?)lmWy..u.!.
  00C0: 01 00 00 00 B6 00 00 00 00 00 02 00 44 45 4C 4C  ............DELL
  00D0: 20 20 42 31 30 4B 20 20 20 00 57 49 4E 44 4F 57    B10K   .WINDOW
  00E0: 53 20 01 00 02 00 00 00 00 00 00 00 00 00 00 00  S ..............
  00F0: 00 00 00 00 00 00 5A 3A A5 CC F1 35 95 0B 73 96  ......Z:...5..s.
  0100: 49 3F B2 87 9F 41 2D 3D 56 46 EB 93 67 F2 87 92  I?...A-=VF..g...
  0110: F5 C0 6E D7 0B 7C 89 A0 D6 70 63 A2 AF 7C DB 56  ..n..|...pc..|.V
  0120: 3A F8 A0 D4 12 C7 8D 27 4F 86 C0 8E A6 DC 72 03  :......'O.....r.
  0130: 14 90 03 17 EC 0A A6 E0 5D BE BD 1E 8B 7B E1 0B  ........]....{..
  0140: DF 26 44 03 F5 7E B2 0A 07 A2 E9 B7 15 F5 38 8E  .&D..~........8.
  0150: EA A7 05 CC 84 0F 7C 61 63 ED CF 58 C0 E8 7B DC  ......|ac..X..{.
  0160: 5C BE 1E 3C 21 9D 4A EC 54 EC 3F 6B 92 00 28 BF  \..<!.J.T.?k..(.
  0170: E1 F2 32 26 7D E3                                ..2&}.

TCPA @ 0x0000000000000000
  0000: 54 43 50 41 32 00 00 00 01 7F 44 45 4C 4C 20 20  TCPA2.....DELL  
  0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20  B10K   .....ASL 
  0020: 61 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00  a...............
  0030: 00 00                                            ..

SRAT @ 0x0000000000000000
  0000: 53 52 41 54 70 05 00 00 01 76 44 45 4C 4C 20 20  SRATp....vDELL  
  0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20  B10K   .....ASL 
  0020: 61 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00  a...............
  0030: 00 10 00 20 01 00 00 00 00 00 00 00 00 00 00 00  ... ............
  0040: 00 10 00 22 01 00 00 00 00 00 00 00 00 00 00 00  ..."............
  0050: 00 10 00 32 01 00 00 00 00 00 00 00 00 00 00 00  ...2............
  0060: 00 10 00 34 01 00 00 00 00 00 00 00 00 00 00 00  ...4............
  0070: 00 10 01 00 01 00 00 00 00 00 00 00 00 00 00 00  ................
  0080: 00 10 01 02 01 00 00 00 00 00 00 00 00 00 00 00  ................
  0090: 00 10 01 12 01 00 00 00 00 00 00 00 00 00 00 00  ................
  00A0: 00 10 01 14 01 00 00 00 00 00 00 00 00 00 00 00  ................
  00B0: 00 10 00 21 01 00 00 00 00 00 00 00 00 00 00 00  ...!............
  00C0: 00 10 00 23 01 00 00 00 00 00 00 00 00 00 00 00  ...#............
  00D0: 00 10 00 33 01 00 00 00 00 00 00 00 00 00 00 00  ...3............
  00E0: 00 10 00 35 01 00 00 00 00 00 00 00 00 00 00 00  ...5............
  00F0: 00 10 01 01 01 00 00 00 00 00 00 00 00 00 00 00  ................
  0100: 00 10 01 03 01 00 00 00 00 00 00 00 00 00 00 00  ................
  0110: 00 10 01 13 01 00 00 00 00 00 00 00 00 00 00 00  ................
  0120: 00 10 01 15 01 00 00 00 00 00 00 00 00 00 00 00  ................
  0130: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0140: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0150: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0160: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0170: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0180: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0190: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  01A0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  01B0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  01C0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  01D0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  01E0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  01F0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0200: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0210: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0220: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0230: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0240: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0250: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0260: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0270: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0280: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0290: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  02A0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  02B0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  02C0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  02D0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  02E0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  02F0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0300: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0310: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0320: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0330: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0340: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0350: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0360: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0370: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0380: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0390: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  03A0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  03B0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  03C0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  03D0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  03E0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  03F0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0400: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0410: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0420: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0430: 01 28 01 00 00 00 00 00 00 00 00 00 00 00 00 00  .(..............
  0440: 00 00 00 DC 00 00 00 00 00 00 00 00 01 00 00 00  ................
  0450: 00 00 00 00 00 00 00 00 01 28 01 00 00 00 00 00  .........(......
  0460: 00 00 00 00 01 00 00 00 00 00 00 A4 00 00 00 00  ................
  0470: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00  ................
  0480: 01 28 00 00 00 00 00 00 00 00 00 A4 01 00 00 00  .(..............
  0490: 00 00 00 80 01 00 00 00 00 00 00 00 01 00 00 00  ................
  04A0: 00 00 00 00 00 00 00 00 01 28 00 00 00 00 00 00  .........(......
  04B0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  04C0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  04D0: 01 28 00 00 00 00 00 00 00 00 00 00 00 00 00 00  .(..............
  04E0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  04F0: 00 00 00 00 00 00 00 00 01 28 00 00 00 00 00 00  .........(......
  0500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0520: 01 28 00 00 00 00 00 00 00 00 00 00 00 00 00 00  .(..............
  0530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0540: 00 00 00 00 00 00 00 00 01 28 00 00 00 00 00 00  .........(......
  0550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  0560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

SSDT @ 0x0000000000000000
  0000: 53 53 44 54 9C 00 00 00 01 EF 44 45 4C 4C 00 00  SSDT......DELL..
  0010: 73 74 5F 65 78 00 00 00 00 10 00 00 49 4E 54 4C  st_ex.......INTL
  0020: 24 06 05 20 14 07 55 53 45 53 00 A3 10 3B 5C 00  $.. ..USES...;\.
  0030: 08 5C 5F 53 30 5F 12 06 04 00 00 00 00 08 5C 5F  .\_S0_........\_
  0040: 53 33 5F 12 08 04 0A 03 0A 03 00 00 08 5C 5F 53  S3_..........\_S
  0050: 34 5F 12 06 04 00 00 00 00 08 5C 5F 53 35 5F 12  4_........\_S5_.
  0060: 08 04 0A 02 0A 02 00 00 10 33 5C 2F 04 5F 53 42  .........3\/._SB
  0070: 5F 50 43 49 30 49 53 41 5F 4D 4F 55 5F 14 1E 5F  _PCI0ISA_MOU_.._
  0080: 50 52 57 00 A0 0D 48 41 43 4B A4 12 06 02 0A 08  PRW...HACK......
  0090: 0A 03 A1 09 A4 12 06 02 0A 08 0A 05              ............

SSDT @ 0x0000000000000000
  0000: 53 53 44 54 64 44 00 00 01 A8 49 4E 54 45 4C 00  SSDTdD....INTEL.
  0010: 50 50 4D 20 52 43 4D 20 01 00 00 80 49 4E 54 4C  PPM RCM ....INTL
  0020: 09 11 06 20 10 43 44 5C 2E 5F 50 52 5F 43 50 55  ... .CD\._PR_CPU
  0030: 31 08 54 59 50 45 0C 00 00 00 00 08 50 53 45 4E  1.TYPE......PSEN
  0040: 0C 01 00 00 00 08 43 53 45 4E 0C 01 00 00 00 08  ......CSEN......
  0050: 4D 57 4F 53 0C 01 00 00 00 08 54 53 45 4E 0C 00  MWOS......TSEN..
  0060: 00 00 00 14 49 07 5F 50 44 43 01 8A 68 0A 08 43  ....I._PDC..h..C
  0070: 41 50 41 70 43 41 50 41 54 59 50 45 8A 68 0A 00  APApCAPATYPE.h..
  0080: 52 45 56 53 8A 68 0A 04 53 49 5A 45 70 87 68 60  REVS.h..SIZEp.h`
  0090: 70 74 60 0A 08 00 61 5B 13 68 0A 40 77 61 0A 08  pt`...a[.h.@wa..
  00A0: 00 54 45 4D 50 08 53 54 53 30 11 07 0A 04 00 00  .TEMP.STS0......
  00B0: 00 00 73 53 54 53 30 54 45 4D 50 62 5F 4F 53 43  ..sSTS0TEMPb_OSC
  00C0: 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70  ......w@.).G...p
  00D0: 58 71 39 53 52 45 56 53 53 49 5A 45 62 14 4E 0F  Xq9SREVSSIZEb.N.
  00E0: 5F 4F 53 43 04 8A 6B 0A 04 43 41 50 41 70 43 41  _OSC..k..CAPApCA
  00F0: 50 41 54 59 50 45 8A 6B 0A 00 53 54 53 30 8A 6B  PATYPE.k..STS0.k
  0100: 0A 04 43 41 50 30 8A 68 0A 00 49 49 44 30 8A 68  ..CAP0.h..IID0.h
  0110: 0A 04 49 49 44 31 8A 68 0A 08 49 49 44 32 8A 68  ..IID1.h..IID2.h
  0120: 0A 0C 49 49 44 33 08 55 49 44 30 11 13 0A 10 16  ..IID3.UID0.....
  0130: A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53 8A  .w@.).G...pXq9S.
  0140: 68 0A 00 45 49 44 30 8A 68 0A 04 45 49 44 31 8A  h..EID0.h..EID1.
  0150: 68 0A 08 45 49 44 32 8A 68 0A 0C 45 49 44 33 A0  h..EID2.h..EID3.
  0160: 47 03 92 90 90 93 49 49 44 30 45 49 44 30 93 49  G.....IID0EID0.I
  0170: 49 44 31 45 49 44 31 90 93 49 49 44 32 45 49 44  ID1EID1..IID2EID
  0180: 32 93 49 49 44 33 45 49 44 33 70 0A 06 88 53 54  2.IID3EID3p...ST
  0190: 53 30 0A 00 00 A4 6B A0 44 01 92 93 69 0A 01 70  S0....k.D...i..p
  01A0: 0A 0A 88 53 54 53 30 0A 00 00 A4 6B A0 48 01 7B  ...STS0....k.H.{
  01B0: 53 54 53 30 0A 01 00 7B 43 41 50 30 0B FF 0B 43  STS0...{CAP0...C
  01C0: 41 50 30 A4 6B 7B 43 41 50 30 0B FF 0B 43 41 50  AP0.k{CAP0...CAP
  01D0: 30 70 43 41 50 30 54 59 50 45 A4 6B 14 4D 11 5F  0pCAP0TYPE.k.M._
  01E0: 50 53 53 00 A0 43 11 50 53 45 4E A4 12 4B 10 08  PSS..C.PSEN..K..
  01F0: 12 20 06 0C 5B 09 00 00 0C 80 38 01 00 0C 0A 00  . ..[.....8.....
  0200: 00 00 0C 0A 00 00 00 0C 13 00 00 00 0C 13 00 00  ................
  0210: 00 12 20 06 0C 5A 09 00 00 0C 80 38 01 00 0C 0A  .. ..Z.....8....
  0220: 00 00 00 0C 0A 00 00 00 0C 12 00 00 00 0C 12 00  ................
  0230: 00 00 12 20 06 0C D5 08 00 00 0C D0 01 01 00 0C  ... ............
  0240: 0A 00 00 00 0C 0A 00 00 00 0C 11 00 00 00 0C 11  ................
  0250: 00 00 00 12 20 06 0C 50 08 00 00 0C D8 D6 00 00  .... ..P........
  0260: 0C 0A 00 00 00 0C 0A 00 00 00 0C 10 00 00 00 0C  ................
  0270: 10 00 00 00 12 20 06 0C CB 07 00 00 0C 50 C3 00  ..... .......P..
  0280: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0F 00 00 00  ................
  0290: 0C 0F 00 00 00 12 20 06 0C 46 07 00 00 0C 28 A0  ...... ..F....(.
  02A0: 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0E 00 00  ................
  02B0: 00 0C 0E 00 00 00 12 20 06 0C C1 06 00 00 0C D0  ....... ........
  02C0: 84 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0D 00  ................
  02D0: 00 00 0C 0D 00 00 00 12 20 06 0C 3C 06 00 00 0C  ........ ..<....
  02E0: 30 75 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C  0u..............
  02F0: 00 00 00 0C 0C 00 00 00 A4 00 14 35 5F 50 43 54  ...........5_PCT
  0300: 00 A4 12 4D 02 02 11 14 0A 11 82 0C 00 7F 40 00  ...M..........@.
  0310: 00 99 01 00 00 00 00 00 00 79 00 11 14 0A 11 82  .........y......
  0320: 0C 00 7F 10 00 00 98 01 00 00 00 00 00 00 79 00  ..............y.
  0330: 14 08 5F 50 50 43 00 A4 00 14 42 03 5F 50 53 44  .._PPC....B._PSD
  0340: 00 A0 48 02 7B 54 59 50 45 0C 20 08 00 00 00 A4  ..H.{TYPE. .....
  0350: 12 49 01 01 12 15 05 0A 05 0A 00 0C 00 00 00 00  .I..............
  0360: 0C FE 00 00 00 0C 08 00 00 00 A4 00 14 4B 0F 5F  .............K._
  0370: 43 53 54 00 A0 41 0F 43 53 45 4E A0 4D 07 90 4D  CST..A.CSEN.M..M
  0380: 57 4F 53 7B 54 59 50 45 0B 00 02 00 A4 12 4B 06  WOS{TYPE......K.
  0390: 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 01 02  ....!...........
  03A0: 01 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01 00  .........y......
  03B0: 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00 7F  ......!.........
  03C0: 01 02 01 10 00 00 00 00 00 00 00 79 00 0A 02 0B  ...........y....
  03D0: 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82 0C  @.......!.......
  03E0: 00 7F 01 02 01 20 00 00 00 00 00 00 00 79 00 0A  ..... .......y..
  03F0: 03 0B 60 00 0C 5E 01 00 00 A4 12 4B 06 04 0A 03  ..`..^.....K....
  0400: 12 21 04 11 14 0A 11 82 0C 00 7F 00 00 00 00 00  .!..............
  0410: 00 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03  ......y.........
  0420: 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00  ...!............
  0430: 14 08 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C  ........y....@..
  0440: F4 01 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08  .....!..........
  0450: 00 00 15 08 00 00 00 00 00 00 79 00 0A 03 0B 60  ..........y....`
  0460: 00 0C 5E 01 00 00 A4 00 10 43 44 5C 2E 5F 50 52  ..^......CD\._PR
  0470: 5F 43 50 55 32 08 54 59 50 45 0C 00 00 00 00 08  _CPU2.TYPE......
  0480: 50 53 45 4E 0C 01 00 00 00 08 43 53 45 4E 0C 01  PSEN......CSEN..
  0490: 00 00 00 08 4D 57 4F 53 0C 01 00 00 00 08 54 53  ....MWOS......TS
  04A0: 45 4E 0C 00 00 00 00 14 49 07 5F 50 44 43 01 8A  EN......I._PDC..
  04B0: 68 0A 08 43 41 50 41 70 43 41 50 41 54 59 50 45  h..CAPApCAPATYPE
  04C0: 8A 68 0A 00 52 45 56 53 8A 68 0A 04 53 49 5A 45  .h..REVS.h..SIZE
  04D0: 70 87 68 60 70 74 60 0A 08 00 61 5B 13 68 0A 40  p.h`pt`...a[.h.@
  04E0: 77 61 0A 08 00 54 45 4D 50 08 53 54 53 30 11 07  wa...TEMP.STS0..
  04F0: 0A 04 00 00 00 00 73 53 54 53 30 54 45 4D 50 62  ......sSTS0TEMPb
  0500: 5F 4F 53 43 11 13 0A 10 16 A6 77 40 0C 29 BE 47  _OSC......w@.).G
  0510: 9E BD D8 70 58 71 39 53 52 45 56 53 53 49 5A 45  ...pXq9SREVSSIZE
  0520: 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A 04 43 41 50  b.N._OSC..k..CAP
  0530: 41 70 43 41 50 41 54 59 50 45 8A 6B 0A 00 53 54  ApCAPATYPE.k..ST
  0540: 53 30 8A 6B 0A 04 43 41 50 30 8A 68 0A 00 49 49  S0.k..CAP0.h..II
  0550: 44 30 8A 68 0A 04 49 49 44 31 8A 68 0A 08 49 49  D0.h..IID1.h..II
  0560: 44 32 8A 68 0A 0C 49 49 44 33 08 55 49 44 30 11  D2.h..IID3.UID0.
  0570: 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58  .....w@.).G...pX
  0580: 71 39 53 8A 68 0A 00 45 49 44 30 8A 68 0A 04 45  q9S.h..EID0.h..E
  0590: 49 44 31 8A 68 0A 08 45 49 44 32 8A 68 0A 0C 45  ID1.h..EID2.h..E
  05A0: 49 44 33 A0 47 03 92 90 90 93 49 49 44 30 45 49  ID3.G.....IID0EI
  05B0: 44 30 93 49 49 44 31 45 49 44 31 90 93 49 49 44  D0.IID1EID1..IID
  05C0: 32 45 49 44 32 93 49 49 44 33 45 49 44 33 70 0A  2EID2.IID3EID3p.
  05D0: 06 88 53 54 53 30 0A 00 00 A4 6B A0 44 01 92 93  ..STS0....k.D...
  05E0: 69 0A 01 70 0A 0A 88 53 54 53 30 0A 00 00 A4 6B  i..p...STS0....k
  05F0: A0 48 01 7B 53 54 53 30 0A 01 00 7B 43 41 50 30  .H.{STS0...{CAP0
  0600: 0B FF 0B 43 41 50 30 A4 6B 7B 43 41 50 30 0B FF  ...CAP0.k{CAP0..
  0610: 0B 43 41 50 30 70 43 41 50 30 54 59 50 45 A4 6B  .CAP0pCAP0TYPE.k
  0620: 14 4D 11 5F 50 53 53 00 A0 43 11 50 53 45 4E A4  .M._PSS..C.PSEN.
  0630: 12 4B 10 08 12 20 06 0C 5B 09 00 00 0C 80 38 01  .K... ..[.....8.
  0640: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 13 00 00 00  ................
  0650: 0C 13 00 00 00 12 20 06 0C 5A 09 00 00 0C 80 38  ...... ..Z.....8
  0660: 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 12 00 00  ................
  0670: 00 0C 12 00 00 00 12 20 06 0C D5 08 00 00 0C D0  ....... ........
  0680: 01 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 11 00  ................
  0690: 00 00 0C 11 00 00 00 12 20 06 0C 50 08 00 00 0C  ........ ..P....
  06A0: D8 D6 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 10  ................
  06B0: 00 00 00 0C 10 00 00 00 12 20 06 0C CB 07 00 00  ......... ......
  06C0: 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C  .P..............
  06D0: 0F 00 00 00 0C 0F 00 00 00 12 20 06 0C 46 07 00  .......... ..F..
  06E0: 00 0C 28 A0 00 00 0C 0A 00 00 00 0C 0A 00 00 00  ..(.............
  06F0: 0C 0E 00 00 00 0C 0E 00 00 00 12 20 06 0C C1 06  ........... ....
  0700: 00 00 0C D0 84 00 00 0C 0A 00 00 00 0C 0A 00 00  ................
  0710: 00 0C 0D 00 00 00 0C 0D 00 00 00 12 20 06 0C 3C  ............ ..<
  0720: 06 00 00 0C 30 75 00 00 0C 0A 00 00 00 0C 0A 00  ....0u..........
  0730: 00 00 0C 0C 00 00 00 0C 0C 00 00 00 A4 00 14 35  ...............5
  0740: 5F 50 43 54 00 A4 12 4D 02 02 11 14 0A 11 82 0C  _PCT...M........
  0750: 00 7F 40 00 00 99 01 00 00 00 00 00 00 79 00 11  ..@..........y..
  0760: 14 0A 11 82 0C 00 7F 10 00 00 98 01 00 00 00 00  ................
  0770: 00 00 79 00 14 08 5F 50 50 43 00 A4 00 14 42 03  ..y..._PPC....B.
  0780: 5F 50 53 44 00 A0 48 02 7B 54 59 50 45 0C 20 08  _PSD..H.{TYPE. .
  0790: 00 00 00 A4 12 49 01 01 12 15 05 0A 05 0A 00 0C  .....I..........
  07A0: 00 00 00 00 0C FE 00 00 00 0C 08 00 00 00 A4 00  ................
  07B0: 14 4B 0F 5F 43 53 54 00 A0 41 0F 43 53 45 4E A0  .K._CST..A.CSEN.
  07C0: 4D 07 90 4D 57 4F 53 7B 54 59 50 45 0B 00 02 00  M..MWOS{TYPE....
  07D0: A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82 0C  ..K.....!.......
  07E0: 00 7F 01 02 01 00 00 00 00 00 00 00 00 79 00 0A  .............y..
  07F0: 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A 11  ..........!.....
  0800: 82 0C 00 7F 01 02 01 10 00 00 00 00 00 00 00 79  ...............y
  0810: 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11 14  ....@.......!...
  0820: 0A 11 82 0C 00 7F 01 02 01 20 00 00 00 00 00 00  ......... ......
  0830: 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 12 4B  .y....`..^.....K
  0840: 06 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 00  .....!..........
  0850: 00 00 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01  ..........y.....
  0860: 00 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00  .......!........
  0870: 01 08 00 00 14 08 00 00 00 00 00 00 79 00 0A 02  ............y...
  0880: 0B 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82  .@.......!......
  0890: 0C 00 01 08 00 00 15 08 00 00 00 00 00 00 79 00  ..............y.
  08A0: 0A 03 0B 60 00 0C 5E 01 00 00 A4 00 10 43 44 5C  ...`..^......CD\
  08B0: 2E 5F 50 52 5F 43 50 55 33 08 54 59 50 45 0C 00  ._PR_CPU3.TYPE..
  08C0: 00 00 00 08 50 53 45 4E 0C 01 00 00 00 08 43 53  ....PSEN......CS
  08D0: 45 4E 0C 01 00 00 00 08 4D 57 4F 53 0C 01 00 00  EN......MWOS....
  08E0: 00 08 54 53 45 4E 0C 00 00 00 00 14 49 07 5F 50  ..TSEN......I._P
  08F0: 44 43 01 8A 68 0A 08 43 41 50 41 70 43 41 50 41  DC..h..CAPApCAPA
  0900: 54 59 50 45 8A 68 0A 00 52 45 56 53 8A 68 0A 04  TYPE.h..REVS.h..
  0910: 53 49 5A 45 70 87 68 60 70 74 60 0A 08 00 61 5B  SIZEp.h`pt`...a[
  0920: 13 68 0A 40 77 61 0A 08 00 54 45 4D 50 08 53 54  .h.@wa...TEMP.ST
  0930: 53 30 11 07 0A 04 00 00 00 00 73 53 54 53 30 54  S0........sSTS0T
  0940: 45 4D 50 62 5F 4F 53 43 11 13 0A 10 16 A6 77 40  EMPb_OSC......w@
  0950: 0C 29 BE 47 9E BD D8 70 58 71 39 53 52 45 56 53  .).G...pXq9SREVS
  0960: 53 49 5A 45 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A  SIZEb.N._OSC..k.
  0970: 04 43 41 50 41 70 43 41 50 41 54 59 50 45 8A 6B  .CAPApCAPATYPE.k
  0980: 0A 00 53 54 53 30 8A 6B 0A 04 43 41 50 30 8A 68  ..STS0.k..CAP0.h
  0990: 0A 00 49 49 44 30 8A 68 0A 04 49 49 44 31 8A 68  ..IID0.h..IID1.h
  09A0: 0A 08 49 49 44 32 8A 68 0A 0C 49 49 44 33 08 55  ..IID2.h..IID3.U
  09B0: 49 44 30 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E  ID0......w@.).G.
  09C0: BD D8 70 58 71 39 53 8A 68 0A 00 45 49 44 30 8A  ..pXq9S.h..EID0.
  09D0: 68 0A 04 45 49 44 31 8A 68 0A 08 45 49 44 32 8A  h..EID1.h..EID2.
  09E0: 68 0A 0C 45 49 44 33 A0 47 03 92 90 90 93 49 49  h..EID3.G.....II
  09F0: 44 30 45 49 44 30 93 49 49 44 31 45 49 44 31 90  D0EID0.IID1EID1.
  0A00: 93 49 49 44 32 45 49 44 32 93 49 49 44 33 45 49  .IID2EID2.IID3EI
  0A10: 44 33 70 0A 06 88 53 54 53 30 0A 00 00 A4 6B A0  D3p...STS0....k.
  0A20: 44 01 92 93 69 0A 01 70 0A 0A 88 53 54 53 30 0A  D...i..p...STS0.
  0A30: 00 00 A4 6B A0 48 01 7B 53 54 53 30 0A 01 00 7B  ...k.H.{STS0...{
  0A40: 43 41 50 30 0B FF 0B 43 41 50 30 A4 6B 7B 43 41  CAP0...CAP0.k{CA
  0A50: 50 30 0B FF 0B 43 41 50 30 70 43 41 50 30 54 59  P0...CAP0pCAP0TY
  0A60: 50 45 A4 6B 14 4D 11 5F 50 53 53 00 A0 43 11 50  PE.k.M._PSS..C.P
  0A70: 53 45 4E A4 12 4B 10 08 12 20 06 0C 5B 09 00 00  SEN..K... ..[...
  0A80: 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C  ..8.............
  0A90: 13 00 00 00 0C 13 00 00 00 12 20 06 0C 5A 09 00  .......... ..Z..
  0AA0: 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00  ...8............
  0AB0: 0C 12 00 00 00 0C 12 00 00 00 12 20 06 0C D5 08  ........... ....
  0AC0: 00 00 0C D0 01 01 00 0C 0A 00 00 00 0C 0A 00 00  ................
  0AD0: 00 0C 11 00 00 00 0C 11 00 00 00 12 20 06 0C 50  ............ ..P
  0AE0: 08 00 00 0C D8 D6 00 00 0C 0A 00 00 00 0C 0A 00  ................
  0AF0: 00 00 0C 10 00 00 00 0C 10 00 00 00 12 20 06 0C  ............. ..
  0B00: CB 07 00 00 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A  .....P..........
  0B10: 00 00 00 0C 0F 00 00 00 0C 0F 00 00 00 12 20 06  .............. .
  0B20: 0C 46 07 00 00 0C 28 A0 00 00 0C 0A 00 00 00 0C  .F....(.........
  0B30: 0A 00 00 00 0C 0E 00 00 00 0C 0E 00 00 00 12 20  ............... 
  0B40: 06 0C C1 06 00 00 0C D0 84 00 00 0C 0A 00 00 00  ................
  0B50: 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D 00 00 00 12  ................
  0B60: 20 06 0C 3C 06 00 00 0C 30 75 00 00 0C 0A 00 00   ..<....0u......
  0B70: 00 0C 0A 00 00 00 0C 0C 00 00 00 0C 0C 00 00 00  ................
  0B80: A4 00 14 35 5F 50 43 54 00 A4 12 4D 02 02 11 14  ...5_PCT...M....
  0B90: 0A 11 82 0C 00 7F 40 00 00 99 01 00 00 00 00 00  ......@.........
  0BA0: 00 79 00 11 14 0A 11 82 0C 00 7F 10 00 00 98 01  .y..............
  0BB0: 00 00 00 00 00 00 79 00 14 08 5F 50 50 43 00 A4  ......y..._PPC..
  0BC0: 00 14 42 03 5F 50 53 44 00 A0 48 02 7B 54 59 50  ..B._PSD..H.{TYP
  0BD0: 45 0C 20 08 00 00 00 A4 12 49 01 01 12 15 05 0A  E. ......I......
  0BE0: 05 0A 00 0C 00 00 00 00 0C FE 00 00 00 0C 08 00  ................
  0BF0: 00 00 A4 00 14 4B 0F 5F 43 53 54 00 A0 41 0F 43  .....K._CST..A.C
  0C00: 53 45 4E A0 4D 07 90 4D 57 4F 53 7B 54 59 50 45  SEN.M..MWOS{TYPE
  0C10: 0B 00 02 00 A4 12 4B 06 04 0A 03 12 21 04 11 14  ......K.....!...
  0C20: 0A 11 82 0C 00 7F 01 02 01 00 00 00 00 00 00 00  ................
  0C30: 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21 04  .y............!.
  0C40: 11 14 0A 11 82 0C 00 7F 01 02 01 10 00 00 00 00  ................
  0C50: 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00 12  ...y....@.......
  0C60: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 20 00 00  !............ ..
  0C70: 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00  .....y....`..^..
  0C80: 00 A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82  ...K.....!......
  0C90: 0C 00 7F 00 00 00 00 00 00 00 00 00 00 00 79 00  ..............y.
  0CA0: 0A 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A  ...........!....
  0CB0: 11 82 0C 00 01 08 00 00 14 08 00 00 00 00 00 00  ................
  0CC0: 79 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11  y....@.......!..
  0CD0: 14 0A 11 82 0C 00 01 08 00 00 15 08 00 00 00 00  ................
  0CE0: 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 00  ..y....`..^.....
  0CF0: 10 43 44 5C 2E 5F 50 52 5F 43 50 55 34 08 54 59  .CD\._PR_CPU4.TY
  0D00: 50 45 0C 00 00 00 00 08 50 53 45 4E 0C 01 00 00  PE......PSEN....
  0D10: 00 08 43 53 45 4E 0C 01 00 00 00 08 4D 57 4F 53  ..CSEN......MWOS
  0D20: 0C 01 00 00 00 08 54 53 45 4E 0C 00 00 00 00 14  ......TSEN......
  0D30: 49 07 5F 50 44 43 01 8A 68 0A 08 43 41 50 41 70  I._PDC..h..CAPAp
  0D40: 43 41 50 41 54 59 50 45 8A 68 0A 00 52 45 56 53  CAPATYPE.h..REVS
  0D50: 8A 68 0A 04 53 49 5A 45 70 87 68 60 70 74 60 0A  .h..SIZEp.h`pt`.
  0D60: 08 00 61 5B 13 68 0A 40 77 61 0A 08 00 54 45 4D  ..a[.h.@wa...TEM
  0D70: 50 08 53 54 53 30 11 07 0A 04 00 00 00 00 73 53  P.STS0........sS
  0D80: 54 53 30 54 45 4D 50 62 5F 4F 53 43 11 13 0A 10  TS0TEMPb_OSC....
  0D90: 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53  ..w@.).G...pXq9S
  0DA0: 52 45 56 53 53 49 5A 45 62 14 4E 0F 5F 4F 53 43  REVSSIZEb.N._OSC
  0DB0: 04 8A 6B 0A 04 43 41 50 41 70 43 41 50 41 54 59  ..k..CAPApCAPATY
  0DC0: 50 45 8A 6B 0A 00 53 54 53 30 8A 6B 0A 04 43 41  PE.k..STS0.k..CA
  0DD0: 50 30 8A 68 0A 00 49 49 44 30 8A 68 0A 04 49 49  P0.h..IID0.h..II
  0DE0: 44 31 8A 68 0A 08 49 49 44 32 8A 68 0A 0C 49 49  D1.h..IID2.h..II
  0DF0: 44 33 08 55 49 44 30 11 13 0A 10 16 A6 77 40 0C  D3.UID0......w@.
  0E00: 29 BE 47 9E BD D8 70 58 71 39 53 8A 68 0A 00 45  ).G...pXq9S.h..E
  0E10: 49 44 30 8A 68 0A 04 45 49 44 31 8A 68 0A 08 45  ID0.h..EID1.h..E
  0E20: 49 44 32 8A 68 0A 0C 45 49 44 33 A0 47 03 92 90  ID2.h..EID3.G...
  0E30: 90 93 49 49 44 30 45 49 44 30 93 49 49 44 31 45  ..IID0EID0.IID1E
  0E40: 49 44 31 90 93 49 49 44 32 45 49 44 32 93 49 49  ID1..IID2EID2.II
  0E50: 44 33 45 49 44 33 70 0A 06 88 53 54 53 30 0A 00  D3EID3p...STS0..
  0E60: 00 A4 6B A0 44 01 92 93 69 0A 01 70 0A 0A 88 53  ..k.D...i..p...S
  0E70: 54 53 30 0A 00 00 A4 6B A0 48 01 7B 53 54 53 30  TS0....k.H.{STS0
  0E80: 0A 01 00 7B 43 41 50 30 0B FF 0B 43 41 50 30 A4  ...{CAP0...CAP0.
  0E90: 6B 7B 43 41 50 30 0B FF 0B 43 41 50 30 70 43 41  k{CAP0...CAP0pCA
  0EA0: 50 30 54 59 50 45 A4 6B 14 4D 11 5F 50 53 53 00  P0TYPE.k.M._PSS.
  0EB0: A0 43 11 50 53 45 4E A4 12 4B 10 08 12 20 06 0C  .C.PSEN..K... ..
  0EC0: 5B 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A  [.....8.........
  0ED0: 00 00 00 0C 13 00 00 00 0C 13 00 00 00 12 20 06  .............. .
  0EE0: 0C 5A 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C  .Z.....8........
  0EF0: 0A 00 00 00 0C 12 00 00 00 0C 12 00 00 00 12 20  ............... 
  0F00: 06 0C D5 08 00 00 0C D0 01 01 00 0C 0A 00 00 00  ................
  0F10: 0C 0A 00 00 00 0C 11 00 00 00 0C 11 00 00 00 12  ................
  0F20: 20 06 0C 50 08 00 00 0C D8 D6 00 00 0C 0A 00 00   ..P............
  0F30: 00 0C 0A 00 00 00 0C 10 00 00 00 0C 10 00 00 00  ................
  0F40: 12 20 06 0C CB 07 00 00 0C 50 C3 00 00 0C 0A 00  . .......P......
  0F50: 00 00 0C 0A 00 00 00 0C 0F 00 00 00 0C 0F 00 00  ................
  0F60: 00 12 20 06 0C 46 07 00 00 0C 28 A0 00 00 0C 0A  .. ..F....(.....
  0F70: 00 00 00 0C 0A 00 00 00 0C 0E 00 00 00 0C 0E 00  ................
  0F80: 00 00 12 20 06 0C C1 06 00 00 0C D0 84 00 00 0C  ... ............
  0F90: 0A 00 00 00 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D  ................
  0FA0: 00 00 00 12 20 06 0C 3C 06 00 00 0C 30 75 00 00  .... ..<....0u..
  0FB0: 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C 00 00 00 0C  ................
  0FC0: 0C 00 00 00 A4 00 14 35 5F 50 43 54 00 A4 12 4D  .......5_PCT...M
  0FD0: 02 02 11 14 0A 11 82 0C 00 7F 40 00 00 99 01 00  ..........@.....
  0FE0: 00 00 00 00 00 79 00 11 14 0A 11 82 0C 00 7F 10  .....y..........
  0FF0: 00 00 98 01 00 00 00 00 00 00 79 00 14 08 5F 50  ..........y..._P
  1000: 50 43 00 A4 00 14 42 03 5F 50 53 44 00 A0 48 02  PC....B._PSD..H.
  1010: 7B 54 59 50 45 0C 20 08 00 00 00 A4 12 49 01 01  {TYPE. ......I..
  1020: 12 15 05 0A 05 0A 00 0C 00 00 00 00 0C FE 00 00  ................
  1030: 00 0C 08 00 00 00 A4 00 14 4B 0F 5F 43 53 54 00  .........K._CST.
  1040: A0 41 0F 43 53 45 4E A0 4D 07 90 4D 57 4F 53 7B  .A.CSEN.M..MWOS{
  1050: 54 59 50 45 0B 00 02 00 A4 12 4B 06 04 0A 03 12  TYPE......K.....
  1060: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 00 00 00  !...............
  1070: 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03 00  .....y..........
  1080: 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 10  ..!.............
  1090: 00 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C F4  .......y....@...
  10A0: 01 00 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02  ....!...........
  10B0: 01 20 00 00 00 00 00 00 00 79 00 0A 03 0B 60 00  . .......y....`.
  10C0: 0C 5E 01 00 00 A4 12 4B 06 04 0A 03 12 21 04 11  .^.....K.....!..
  10D0: 14 0A 11 82 0C 00 7F 00 00 00 00 00 00 00 00 00  ................
  10E0: 00 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21  ..y............!
  10F0: 04 11 14 0A 11 82 0C 00 01 08 00 00 14 08 00 00  ................
  1100: 00 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00  ....y....@......
  1110: 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00 15 08  .!..............
  1120: 00 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01  ......y....`..^.
  1130: 00 00 A4 00 10 43 44 5C 2E 5F 50 52 5F 43 50 55  .....CD\._PR_CPU
  1140: 35 08 54 59 50 45 0C 00 00 00 00 08 50 53 45 4E  5.TYPE......PSEN
  1150: 0C 01 00 00 00 08 43 53 45 4E 0C 01 00 00 00 08  ......CSEN......
  1160: 4D 57 4F 53 0C 01 00 00 00 08 54 53 45 4E 0C 00  MWOS......TSEN..
  1170: 00 00 00 14 49 07 5F 50 44 43 01 8A 68 0A 08 43  ....I._PDC..h..C
  1180: 41 50 41 70 43 41 50 41 54 59 50 45 8A 68 0A 00  APApCAPATYPE.h..
  1190: 52 45 56 53 8A 68 0A 04 53 49 5A 45 70 87 68 60  REVS.h..SIZEp.h`
  11A0: 70 74 60 0A 08 00 61 5B 13 68 0A 40 77 61 0A 08  pt`...a[.h.@wa..
  11B0: 00 54 45 4D 50 08 53 54 53 30 11 07 0A 04 00 00  .TEMP.STS0......
  11C0: 00 00 73 53 54 53 30 54 45 4D 50 62 5F 4F 53 43  ..sSTS0TEMPb_OSC
  11D0: 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70  ......w@.).G...p
  11E0: 58 71 39 53 52 45 56 53 53 49 5A 45 62 14 4E 0F  Xq9SREVSSIZEb.N.
  11F0: 5F 4F 53 43 04 8A 6B 0A 04 43 41 50 41 70 43 41  _OSC..k..CAPApCA
  1200: 50 41 54 59 50 45 8A 6B 0A 00 53 54 53 30 8A 6B  PATYPE.k..STS0.k
  1210: 0A 04 43 41 50 30 8A 68 0A 00 49 49 44 30 8A 68  ..CAP0.h..IID0.h
  1220: 0A 04 49 49 44 31 8A 68 0A 08 49 49 44 32 8A 68  ..IID1.h..IID2.h
  1230: 0A 0C 49 49 44 33 08 55 49 44 30 11 13 0A 10 16  ..IID3.UID0.....
  1240: A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53 8A  .w@.).G...pXq9S.
  1250: 68 0A 00 45 49 44 30 8A 68 0A 04 45 49 44 31 8A  h..EID0.h..EID1.
  1260: 68 0A 08 45 49 44 32 8A 68 0A 0C 45 49 44 33 A0  h..EID2.h..EID3.
  1270: 47 03 92 90 90 93 49 49 44 30 45 49 44 30 93 49  G.....IID0EID0.I
  1280: 49 44 31 45 49 44 31 90 93 49 49 44 32 45 49 44  ID1EID1..IID2EID
  1290: 32 93 49 49 44 33 45 49 44 33 70 0A 06 88 53 54  2.IID3EID3p...ST
  12A0: 53 30 0A 00 00 A4 6B A0 44 01 92 93 69 0A 01 70  S0....k.D...i..p
  12B0: 0A 0A 88 53 54 53 30 0A 00 00 A4 6B A0 48 01 7B  ...STS0....k.H.{
  12C0: 53 54 53 30 0A 01 00 7B 43 41 50 30 0B FF 0B 43  STS0...{CAP0...C
  12D0: 41 50 30 A4 6B 7B 43 41 50 30 0B FF 0B 43 41 50  AP0.k{CAP0...CAP
  12E0: 30 70 43 41 50 30 54 59 50 45 A4 6B 14 4D 11 5F  0pCAP0TYPE.k.M._
  12F0: 50 53 53 00 A0 43 11 50 53 45 4E A4 12 4B 10 08  PSS..C.PSEN..K..
  1300: 12 20 06 0C 5B 09 00 00 0C 80 38 01 00 0C 0A 00  . ..[.....8.....
  1310: 00 00 0C 0A 00 00 00 0C 13 00 00 00 0C 13 00 00  ................
  1320: 00 12 20 06 0C 5A 09 00 00 0C 80 38 01 00 0C 0A  .. ..Z.....8....
  1330: 00 00 00 0C 0A 00 00 00 0C 12 00 00 00 0C 12 00  ................
  1340: 00 00 12 20 06 0C D5 08 00 00 0C D0 01 01 00 0C  ... ............
  1350: 0A 00 00 00 0C 0A 00 00 00 0C 11 00 00 00 0C 11  ................
  1360: 00 00 00 12 20 06 0C 50 08 00 00 0C D8 D6 00 00  .... ..P........
  1370: 0C 0A 00 00 00 0C 0A 00 00 00 0C 10 00 00 00 0C  ................
  1380: 10 00 00 00 12 20 06 0C CB 07 00 00 0C 50 C3 00  ..... .......P..
  1390: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0F 00 00 00  ................
  13A0: 0C 0F 00 00 00 12 20 06 0C 46 07 00 00 0C 28 A0  ...... ..F....(.
  13B0: 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0E 00 00  ................
  13C0: 00 0C 0E 00 00 00 12 20 06 0C C1 06 00 00 0C D0  ....... ........
  13D0: 84 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0D 00  ................
  13E0: 00 00 0C 0D 00 00 00 12 20 06 0C 3C 06 00 00 0C  ........ ..<....
  13F0: 30 75 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C  0u..............
  1400: 00 00 00 0C 0C 00 00 00 A4 00 14 35 5F 50 43 54  ...........5_PCT
  1410: 00 A4 12 4D 02 02 11 14 0A 11 82 0C 00 7F 40 00  ...M..........@.
  1420: 00 99 01 00 00 00 00 00 00 79 00 11 14 0A 11 82  .........y......
  1430: 0C 00 7F 10 00 00 98 01 00 00 00 00 00 00 79 00  ..............y.
  1440: 14 08 5F 50 50 43 00 A4 00 14 42 03 5F 50 53 44  .._PPC....B._PSD
  1450: 00 A0 48 02 7B 54 59 50 45 0C 20 08 00 00 00 A4  ..H.{TYPE. .....
  1460: 12 49 01 01 12 15 05 0A 05 0A 00 0C 01 00 00 00  .I..............
  1470: 0C FE 00 00 00 0C 08 00 00 00 A4 00 14 4B 0F 5F  .............K._
  1480: 43 53 54 00 A0 41 0F 43 53 45 4E A0 4D 07 90 4D  CST..A.CSEN.M..M
  1490: 57 4F 53 7B 54 59 50 45 0B 00 02 00 A4 12 4B 06  WOS{TYPE......K.
  14A0: 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 01 02  ....!...........
  14B0: 01 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01 00  .........y......
  14C0: 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00 7F  ......!.........
  14D0: 01 02 01 10 00 00 00 00 00 00 00 79 00 0A 02 0B  ...........y....
  14E0: 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82 0C  @.......!.......
  14F0: 00 7F 01 02 01 20 00 00 00 00 00 00 00 79 00 0A  ..... .......y..
  1500: 03 0B 60 00 0C 5E 01 00 00 A4 12 4B 06 04 0A 03  ..`..^.....K....
  1510: 12 21 04 11 14 0A 11 82 0C 00 7F 00 00 00 00 00  .!..............
  1520: 00 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03  ......y.........
  1530: 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00  ...!............
  1540: 14 08 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C  ........y....@..
  1550: F4 01 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08  .....!..........
  1560: 00 00 15 08 00 00 00 00 00 00 79 00 0A 03 0B 60  ..........y....`
  1570: 00 0C 5E 01 00 00 A4 00 10 43 44 5C 2E 5F 50 52  ..^......CD\._PR
  1580: 5F 43 50 55 36 08 54 59 50 45 0C 00 00 00 00 08  _CPU6.TYPE......
  1590: 50 53 45 4E 0C 01 00 00 00 08 43 53 45 4E 0C 01  PSEN......CSEN..
  15A0: 00 00 00 08 4D 57 4F 53 0C 01 00 00 00 08 54 53  ....MWOS......TS
  15B0: 45 4E 0C 00 00 00 00 14 49 07 5F 50 44 43 01 8A  EN......I._PDC..
  15C0: 68 0A 08 43 41 50 41 70 43 41 50 41 54 59 50 45  h..CAPApCAPATYPE
  15D0: 8A 68 0A 00 52 45 56 53 8A 68 0A 04 53 49 5A 45  .h..REVS.h..SIZE
  15E0: 70 87 68 60 70 74 60 0A 08 00 61 5B 13 68 0A 40  p.h`pt`...a[.h.@
  15F0: 77 61 0A 08 00 54 45 4D 50 08 53 54 53 30 11 07  wa...TEMP.STS0..
  1600: 0A 04 00 00 00 00 73 53 54 53 30 54 45 4D 50 62  ......sSTS0TEMPb
  1610: 5F 4F 53 43 11 13 0A 10 16 A6 77 40 0C 29 BE 47  _OSC......w@.).G
  1620: 9E BD D8 70 58 71 39 53 52 45 56 53 53 49 5A 45  ...pXq9SREVSSIZE
  1630: 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A 04 43 41 50  b.N._OSC..k..CAP
  1640: 41 70 43 41 50 41 54 59 50 45 8A 6B 0A 00 53 54  ApCAPATYPE.k..ST
  1650: 53 30 8A 6B 0A 04 43 41 50 30 8A 68 0A 00 49 49  S0.k..CAP0.h..II
  1660: 44 30 8A 68 0A 04 49 49 44 31 8A 68 0A 08 49 49  D0.h..IID1.h..II
  1670: 44 32 8A 68 0A 0C 49 49 44 33 08 55 49 44 30 11  D2.h..IID3.UID0.
  1680: 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58  .....w@.).G...pX
  1690: 71 39 53 8A 68 0A 00 45 49 44 30 8A 68 0A 04 45  q9S.h..EID0.h..E
  16A0: 49 44 31 8A 68 0A 08 45 49 44 32 8A 68 0A 0C 45  ID1.h..EID2.h..E
  16B0: 49 44 33 A0 47 03 92 90 90 93 49 49 44 30 45 49  ID3.G.....IID0EI
  16C0: 44 30 93 49 49 44 31 45 49 44 31 90 93 49 49 44  D0.IID1EID1..IID
  16D0: 32 45 49 44 32 93 49 49 44 33 45 49 44 33 70 0A  2EID2.IID3EID3p.
  16E0: 06 88 53 54 53 30 0A 00 00 A4 6B A0 44 01 92 93  ..STS0....k.D...
  16F0: 69 0A 01 70 0A 0A 88 53 54 53 30 0A 00 00 A4 6B  i..p...STS0....k
  1700: A0 48 01 7B 53 54 53 30 0A 01 00 7B 43 41 50 30  .H.{STS0...{CAP0
  1710: 0B FF 0B 43 41 50 30 A4 6B 7B 43 41 50 30 0B FF  ...CAP0.k{CAP0..
  1720: 0B 43 41 50 30 70 43 41 50 30 54 59 50 45 A4 6B  .CAP0pCAP0TYPE.k
  1730: 14 4D 11 5F 50 53 53 00 A0 43 11 50 53 45 4E A4  .M._PSS..C.PSEN.
  1740: 12 4B 10 08 12 20 06 0C 5B 09 00 00 0C 80 38 01  .K... ..[.....8.
  1750: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 13 00 00 00  ................
  1760: 0C 13 00 00 00 12 20 06 0C 5A 09 00 00 0C 80 38  ...... ..Z.....8
  1770: 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 12 00 00  ................
  1780: 00 0C 12 00 00 00 12 20 06 0C D5 08 00 00 0C D0  ....... ........
  1790: 01 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 11 00  ................
  17A0: 00 00 0C 11 00 00 00 12 20 06 0C 50 08 00 00 0C  ........ ..P....
  17B0: D8 D6 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 10  ................
  17C0: 00 00 00 0C 10 00 00 00 12 20 06 0C CB 07 00 00  ......... ......
  17D0: 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C  .P..............
  17E0: 0F 00 00 00 0C 0F 00 00 00 12 20 06 0C 46 07 00  .......... ..F..
  17F0: 00 0C 28 A0 00 00 0C 0A 00 00 00 0C 0A 00 00 00  ..(.............
  1800: 0C 0E 00 00 00 0C 0E 00 00 00 12 20 06 0C C1 06  ........... ....
  1810: 00 00 0C D0 84 00 00 0C 0A 00 00 00 0C 0A 00 00  ................
  1820: 00 0C 0D 00 00 00 0C 0D 00 00 00 12 20 06 0C 3C  ............ ..<
  1830: 06 00 00 0C 30 75 00 00 0C 0A 00 00 00 0C 0A 00  ....0u..........
  1840: 00 00 0C 0C 00 00 00 0C 0C 00 00 00 A4 00 14 35  ...............5
  1850: 5F 50 43 54 00 A4 12 4D 02 02 11 14 0A 11 82 0C  _PCT...M........
  1860: 00 7F 40 00 00 99 01 00 00 00 00 00 00 79 00 11  ..@..........y..
  1870: 14 0A 11 82 0C 00 7F 10 00 00 98 01 00 00 00 00  ................
  1880: 00 00 79 00 14 08 5F 50 50 43 00 A4 00 14 42 03  ..y..._PPC....B.
  1890: 5F 50 53 44 00 A0 48 02 7B 54 59 50 45 0C 20 08  _PSD..H.{TYPE. .
  18A0: 00 00 00 A4 12 49 01 01 12 15 05 0A 05 0A 00 0C  .....I..........
  18B0: 01 00 00 00 0C FE 00 00 00 0C 08 00 00 00 A4 00  ................
  18C0: 14 4B 0F 5F 43 53 54 00 A0 41 0F 43 53 45 4E A0  .K._CST..A.CSEN.
  18D0: 4D 07 90 4D 57 4F 53 7B 54 59 50 45 0B 00 02 00  M..MWOS{TYPE....
  18E0: A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82 0C  ..K.....!.......
  18F0: 00 7F 01 02 01 00 00 00 00 00 00 00 00 79 00 0A  .............y..
  1900: 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A 11  ..........!.....
  1910: 82 0C 00 7F 01 02 01 10 00 00 00 00 00 00 00 79  ...............y
  1920: 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11 14  ....@.......!...
  1930: 0A 11 82 0C 00 7F 01 02 01 20 00 00 00 00 00 00  ......... ......
  1940: 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 12 4B  .y....`..^.....K
  1950: 06 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 00  .....!..........
  1960: 00 00 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01  ..........y.....
  1970: 00 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00  .......!........
  1980: 01 08 00 00 14 08 00 00 00 00 00 00 79 00 0A 02  ............y...
  1990: 0B 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82  .@.......!......
  19A0: 0C 00 01 08 00 00 15 08 00 00 00 00 00 00 79 00  ..............y.
  19B0: 0A 03 0B 60 00 0C 5E 01 00 00 A4 00 10 43 44 5C  ...`..^......CD\
  19C0: 2E 5F 50 52 5F 43 50 55 37 08 54 59 50 45 0C 00  ._PR_CPU7.TYPE..
  19D0: 00 00 00 08 50 53 45 4E 0C 01 00 00 00 08 43 53  ....PSEN......CS
  19E0: 45 4E 0C 01 00 00 00 08 4D 57 4F 53 0C 01 00 00  EN......MWOS....
  19F0: 00 08 54 53 45 4E 0C 00 00 00 00 14 49 07 5F 50  ..TSEN......I._P
  1A00: 44 43 01 8A 68 0A 08 43 41 50 41 70 43 41 50 41  DC..h..CAPApCAPA
  1A10: 54 59 50 45 8A 68 0A 00 52 45 56 53 8A 68 0A 04  TYPE.h..REVS.h..
  1A20: 53 49 5A 45 70 87 68 60 70 74 60 0A 08 00 61 5B  SIZEp.h`pt`...a[
  1A30: 13 68 0A 40 77 61 0A 08 00 54 45 4D 50 08 53 54  .h.@wa...TEMP.ST
  1A40: 53 30 11 07 0A 04 00 00 00 00 73 53 54 53 30 54  S0........sSTS0T
  1A50: 45 4D 50 62 5F 4F 53 43 11 13 0A 10 16 A6 77 40  EMPb_OSC......w@
  1A60: 0C 29 BE 47 9E BD D8 70 58 71 39 53 52 45 56 53  .).G...pXq9SREVS
  1A70: 53 49 5A 45 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A  SIZEb.N._OSC..k.
  1A80: 04 43 41 50 41 70 43 41 50 41 54 59 50 45 8A 6B  .CAPApCAPATYPE.k
  1A90: 0A 00 53 54 53 30 8A 6B 0A 04 43 41 50 30 8A 68  ..STS0.k..CAP0.h
  1AA0: 0A 00 49 49 44 30 8A 68 0A 04 49 49 44 31 8A 68  ..IID0.h..IID1.h
  1AB0: 0A 08 49 49 44 32 8A 68 0A 0C 49 49 44 33 08 55  ..IID2.h..IID3.U
  1AC0: 49 44 30 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E  ID0......w@.).G.
  1AD0: BD D8 70 58 71 39 53 8A 68 0A 00 45 49 44 30 8A  ..pXq9S.h..EID0.
  1AE0: 68 0A 04 45 49 44 31 8A 68 0A 08 45 49 44 32 8A  h..EID1.h..EID2.
  1AF0: 68 0A 0C 45 49 44 33 A0 47 03 92 90 90 93 49 49  h..EID3.G.....II
  1B00: 44 30 45 49 44 30 93 49 49 44 31 45 49 44 31 90  D0EID0.IID1EID1.
  1B10: 93 49 49 44 32 45 49 44 32 93 49 49 44 33 45 49  .IID2EID2.IID3EI
  1B20: 44 33 70 0A 06 88 53 54 53 30 0A 00 00 A4 6B A0  D3p...STS0....k.
  1B30: 44 01 92 93 69 0A 01 70 0A 0A 88 53 54 53 30 0A  D...i..p...STS0.
  1B40: 00 00 A4 6B A0 48 01 7B 53 54 53 30 0A 01 00 7B  ...k.H.{STS0...{
  1B50: 43 41 50 30 0B FF 0B 43 41 50 30 A4 6B 7B 43 41  CAP0...CAP0.k{CA
  1B60: 50 30 0B FF 0B 43 41 50 30 70 43 41 50 30 54 59  P0...CAP0pCAP0TY
  1B70: 50 45 A4 6B 14 4D 11 5F 50 53 53 00 A0 43 11 50  PE.k.M._PSS..C.P
  1B80: 53 45 4E A4 12 4B 10 08 12 20 06 0C 5B 09 00 00  SEN..K... ..[...
  1B90: 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C  ..8.............
  1BA0: 13 00 00 00 0C 13 00 00 00 12 20 06 0C 5A 09 00  .......... ..Z..
  1BB0: 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00  ...8............
  1BC0: 0C 12 00 00 00 0C 12 00 00 00 12 20 06 0C D5 08  ........... ....
  1BD0: 00 00 0C D0 01 01 00 0C 0A 00 00 00 0C 0A 00 00  ................
  1BE0: 00 0C 11 00 00 00 0C 11 00 00 00 12 20 06 0C 50  ............ ..P
  1BF0: 08 00 00 0C D8 D6 00 00 0C 0A 00 00 00 0C 0A 00  ................
  1C00: 00 00 0C 10 00 00 00 0C 10 00 00 00 12 20 06 0C  ............. ..
  1C10: CB 07 00 00 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A  .....P..........
  1C20: 00 00 00 0C 0F 00 00 00 0C 0F 00 00 00 12 20 06  .............. .
  1C30: 0C 46 07 00 00 0C 28 A0 00 00 0C 0A 00 00 00 0C  .F....(.........
  1C40: 0A 00 00 00 0C 0E 00 00 00 0C 0E 00 00 00 12 20  ............... 
  1C50: 06 0C C1 06 00 00 0C D0 84 00 00 0C 0A 00 00 00  ................
  1C60: 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D 00 00 00 12  ................
  1C70: 20 06 0C 3C 06 00 00 0C 30 75 00 00 0C 0A 00 00   ..<....0u......
  1C80: 00 0C 0A 00 00 00 0C 0C 00 00 00 0C 0C 00 00 00  ................
  1C90: A4 00 14 35 5F 50 43 54 00 A4 12 4D 02 02 11 14  ...5_PCT...M....
  1CA0: 0A 11 82 0C 00 7F 40 00 00 99 01 00 00 00 00 00  ......@.........
  1CB0: 00 79 00 11 14 0A 11 82 0C 00 7F 10 00 00 98 01  .y..............
  1CC0: 00 00 00 00 00 00 79 00 14 08 5F 50 50 43 00 A4  ......y..._PPC..
  1CD0: 00 14 42 03 5F 50 53 44 00 A0 48 02 7B 54 59 50  ..B._PSD..H.{TYP
  1CE0: 45 0C 20 08 00 00 00 A4 12 49 01 01 12 15 05 0A  E. ......I......
  1CF0: 05 0A 00 0C 01 00 00 00 0C FE 00 00 00 0C 08 00  ................
  1D00: 00 00 A4 00 14 4B 0F 5F 43 53 54 00 A0 41 0F 43  .....K._CST..A.C
  1D10: 53 45 4E A0 4D 07 90 4D 57 4F 53 7B 54 59 50 45  SEN.M..MWOS{TYPE
  1D20: 0B 00 02 00 A4 12 4B 06 04 0A 03 12 21 04 11 14  ......K.....!...
  1D30: 0A 11 82 0C 00 7F 01 02 01 00 00 00 00 00 00 00  ................
  1D40: 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21 04  .y............!.
  1D50: 11 14 0A 11 82 0C 00 7F 01 02 01 10 00 00 00 00  ................
  1D60: 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00 12  ...y....@.......
  1D70: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 20 00 00  !............ ..
  1D80: 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00  .....y....`..^..
  1D90: 00 A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82  ...K.....!......
  1DA0: 0C 00 7F 00 00 00 00 00 00 00 00 00 00 00 79 00  ..............y.
  1DB0: 0A 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A  ...........!....
  1DC0: 11 82 0C 00 01 08 00 00 14 08 00 00 00 00 00 00  ................
  1DD0: 79 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11  y....@.......!..
  1DE0: 14 0A 11 82 0C 00 01 08 00 00 15 08 00 00 00 00  ................
  1DF0: 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 00  ..y....`..^.....
  1E00: 10 43 44 5C 2E 5F 50 52 5F 43 50 55 38 08 54 59  .CD\._PR_CPU8.TY
  1E10: 50 45 0C 00 00 00 00 08 50 53 45 4E 0C 01 00 00  PE......PSEN....
  1E20: 00 08 43 53 45 4E 0C 01 00 00 00 08 4D 57 4F 53  ..CSEN......MWOS
  1E30: 0C 01 00 00 00 08 54 53 45 4E 0C 00 00 00 00 14  ......TSEN......
  1E40: 49 07 5F 50 44 43 01 8A 68 0A 08 43 41 50 41 70  I._PDC..h..CAPAp
  1E50: 43 41 50 41 54 59 50 45 8A 68 0A 00 52 45 56 53  CAPATYPE.h..REVS
  1E60: 8A 68 0A 04 53 49 5A 45 70 87 68 60 70 74 60 0A  .h..SIZEp.h`pt`.
  1E70: 08 00 61 5B 13 68 0A 40 77 61 0A 08 00 54 45 4D  ..a[.h.@wa...TEM
  1E80: 50 08 53 54 53 30 11 07 0A 04 00 00 00 00 73 53  P.STS0........sS
  1E90: 54 53 30 54 45 4D 50 62 5F 4F 53 43 11 13 0A 10  TS0TEMPb_OSC....
  1EA0: 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53  ..w@.).G...pXq9S
  1EB0: 52 45 56 53 53 49 5A 45 62 14 4E 0F 5F 4F 53 43  REVSSIZEb.N._OSC
  1EC0: 04 8A 6B 0A 04 43 41 50 41 70 43 41 50 41 54 59  ..k..CAPApCAPATY
  1ED0: 50 45 8A 6B 0A 00 53 54 53 30 8A 6B 0A 04 43 41  PE.k..STS0.k..CA
  1EE0: 50 30 8A 68 0A 00 49 49 44 30 8A 68 0A 04 49 49  P0.h..IID0.h..II
  1EF0: 44 31 8A 68 0A 08 49 49 44 32 8A 68 0A 0C 49 49  D1.h..IID2.h..II
  1F00: 44 33 08 55 49 44 30 11 13 0A 10 16 A6 77 40 0C  D3.UID0......w@.
  1F10: 29 BE 47 9E BD D8 70 58 71 39 53 8A 68 0A 00 45  ).G...pXq9S.h..E
  1F20: 49 44 30 8A 68 0A 04 45 49 44 31 8A 68 0A 08 45  ID0.h..EID1.h..E
  1F30: 49 44 32 8A 68 0A 0C 45 49 44 33 A0 47 03 92 90  ID2.h..EID3.G...
  1F40: 90 93 49 49 44 30 45 49 44 30 93 49 49 44 31 45  ..IID0EID0.IID1E
  1F50: 49 44 31 90 93 49 49 44 32 45 49 44 32 93 49 49  ID1..IID2EID2.II
  1F60: 44 33 45 49 44 33 70 0A 06 88 53 54 53 30 0A 00  D3EID3p...STS0..
  1F70: 00 A4 6B A0 44 01 92 93 69 0A 01 70 0A 0A 88 53  ..k.D...i..p...S
  1F80: 54 53 30 0A 00 00 A4 6B A0 48 01 7B 53 54 53 30  TS0....k.H.{STS0
  1F90: 0A 01 00 7B 43 41 50 30 0B FF 0B 43 41 50 30 A4  ...{CAP0...CAP0.
  1FA0: 6B 7B 43 41 50 30 0B FF 0B 43 41 50 30 70 43 41  k{CAP0...CAP0pCA
  1FB0: 50 30 54 59 50 45 A4 6B 14 4D 11 5F 50 53 53 00  P0TYPE.k.M._PSS.
  1FC0: A0 43 11 50 53 45 4E A4 12 4B 10 08 12 20 06 0C  .C.PSEN..K... ..
  1FD0: 5B 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A  [.....8.........
  1FE0: 00 00 00 0C 13 00 00 00 0C 13 00 00 00 12 20 06  .............. .
  1FF0: 0C 5A 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C  .Z.....8........
  2000: 0A 00 00 00 0C 12 00 00 00 0C 12 00 00 00 12 20  ............... 
  2010: 06 0C D5 08 00 00 0C D0 01 01 00 0C 0A 00 00 00  ................
  2020: 0C 0A 00 00 00 0C 11 00 00 00 0C 11 00 00 00 12  ................
  2030: 20 06 0C 50 08 00 00 0C D8 D6 00 00 0C 0A 00 00   ..P............
  2040: 00 0C 0A 00 00 00 0C 10 00 00 00 0C 10 00 00 00  ................
  2050: 12 20 06 0C CB 07 00 00 0C 50 C3 00 00 0C 0A 00  . .......P......
  2060: 00 00 0C 0A 00 00 00 0C 0F 00 00 00 0C 0F 00 00  ................
  2070: 00 12 20 06 0C 46 07 00 00 0C 28 A0 00 00 0C 0A  .. ..F....(.....
  2080: 00 00 00 0C 0A 00 00 00 0C 0E 00 00 00 0C 0E 00  ................
  2090: 00 00 12 20 06 0C C1 06 00 00 0C D0 84 00 00 0C  ... ............
  20A0: 0A 00 00 00 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D  ................
  20B0: 00 00 00 12 20 06 0C 3C 06 00 00 0C 30 75 00 00  .... ..<....0u..
  20C0: 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C 00 00 00 0C  ................
  20D0: 0C 00 00 00 A4 00 14 35 5F 50 43 54 00 A4 12 4D  .......5_PCT...M
  20E0: 02 02 11 14 0A 11 82 0C 00 7F 40 00 00 99 01 00  ..........@.....
  20F0: 00 00 00 00 00 79 00 11 14 0A 11 82 0C 00 7F 10  .....y..........
  2100: 00 00 98 01 00 00 00 00 00 00 79 00 14 08 5F 50  ..........y..._P
  2110: 50 43 00 A4 00 14 42 03 5F 50 53 44 00 A0 48 02  PC....B._PSD..H.
  2120: 7B 54 59 50 45 0C 20 08 00 00 00 A4 12 49 01 01  {TYPE. ......I..
  2130: 12 15 05 0A 05 0A 00 0C 01 00 00 00 0C FE 00 00  ................
  2140: 00 0C 08 00 00 00 A4 00 14 4B 0F 5F 43 53 54 00  .........K._CST.
  2150: A0 41 0F 43 53 45 4E A0 4D 07 90 4D 57 4F 53 7B  .A.CSEN.M..MWOS{
  2160: 54 59 50 45 0B 00 02 00 A4 12 4B 06 04 0A 03 12  TYPE......K.....
  2170: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 00 00 00  !...............
  2180: 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03 00  .....y..........
  2190: 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 10  ..!.............
  21A0: 00 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C F4  .......y....@...
  21B0: 01 00 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02  ....!...........
  21C0: 01 20 00 00 00 00 00 00 00 79 00 0A 03 0B 60 00  . .......y....`.
  21D0: 0C 5E 01 00 00 A4 12 4B 06 04 0A 03 12 21 04 11  .^.....K.....!..
  21E0: 14 0A 11 82 0C 00 7F 00 00 00 00 00 00 00 00 00  ................
  21F0: 00 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21  ..y............!
  2200: 04 11 14 0A 11 82 0C 00 01 08 00 00 14 08 00 00  ................
  2210: 00 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00  ....y....@......
  2220: 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00 15 08  .!..............
  2230: 00 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01  ......y....`..^.
  2240: 00 00 A4 00 10 43 44 5C 2E 5F 50 52 5F 43 50 55  .....CD\._PR_CPU
  2250: 39 08 54 59 50 45 0C 00 00 00 00 08 50 53 45 4E  9.TYPE......PSEN
  2260: 0C 01 00 00 00 08 43 53 45 4E 0C 01 00 00 00 08  ......CSEN......
  2270: 4D 57 4F 53 0C 01 00 00 00 08 54 53 45 4E 0C 00  MWOS......TSEN..
  2280: 00 00 00 14 49 07 5F 50 44 43 01 8A 68 0A 08 43  ....I._PDC..h..C
  2290: 41 50 41 70 43 41 50 41 54 59 50 45 8A 68 0A 00  APApCAPATYPE.h..
  22A0: 52 45 56 53 8A 68 0A 04 53 49 5A 45 70 87 68 60  REVS.h..SIZEp.h`
  22B0: 70 74 60 0A 08 00 61 5B 13 68 0A 40 77 61 0A 08  pt`...a[.h.@wa..
  22C0: 00 54 45 4D 50 08 53 54 53 30 11 07 0A 04 00 00  .TEMP.STS0......
  22D0: 00 00 73 53 54 53 30 54 45 4D 50 62 5F 4F 53 43  ..sSTS0TEMPb_OSC
  22E0: 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70  ......w@.).G...p
  22F0: 58 71 39 53 52 45 56 53 53 49 5A 45 62 14 4E 0F  Xq9SREVSSIZEb.N.
  2300: 5F 4F 53 43 04 8A 6B 0A 04 43 41 50 41 70 43 41  _OSC..k..CAPApCA
  2310: 50 41 54 59 50 45 8A 6B 0A 00 53 54 53 30 8A 6B  PATYPE.k..STS0.k
  2320: 0A 04 43 41 50 30 8A 68 0A 00 49 49 44 30 8A 68  ..CAP0.h..IID0.h
  2330: 0A 04 49 49 44 31 8A 68 0A 08 49 49 44 32 8A 68  ..IID1.h..IID2.h
  2340: 0A 0C 49 49 44 33 08 55 49 44 30 11 13 0A 10 16  ..IID3.UID0.....
  2350: A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53 8A  .w@.).G...pXq9S.
  2360: 68 0A 00 45 49 44 30 8A 68 0A 04 45 49 44 31 8A  h..EID0.h..EID1.
  2370: 68 0A 08 45 49 44 32 8A 68 0A 0C 45 49 44 33 A0  h..EID2.h..EID3.
  2380: 47 03 92 90 90 93 49 49 44 30 45 49 44 30 93 49  G.....IID0EID0.I
  2390: 49 44 31 45 49 44 31 90 93 49 49 44 32 45 49 44  ID1EID1..IID2EID
  23A0: 32 93 49 49 44 33 45 49 44 33 70 0A 06 88 53 54  2.IID3EID3p...ST
  23B0: 53 30 0A 00 00 A4 6B A0 44 01 92 93 69 0A 01 70  S0....k.D...i..p
  23C0: 0A 0A 88 53 54 53 30 0A 00 00 A4 6B A0 48 01 7B  ...STS0....k.H.{
  23D0: 53 54 53 30 0A 01 00 7B 43 41 50 30 0B FF 0B 43  STS0...{CAP0...C
  23E0: 41 50 30 A4 6B 7B 43 41 50 30 0B FF 0B 43 41 50  AP0.k{CAP0...CAP
  23F0: 30 70 43 41 50 30 54 59 50 45 A4 6B 14 4D 11 5F  0pCAP0TYPE.k.M._
  2400: 50 53 53 00 A0 43 11 50 53 45 4E A4 12 4B 10 08  PSS..C.PSEN..K..
  2410: 12 20 06 0C 5B 09 00 00 0C 80 38 01 00 0C 0A 00  . ..[.....8.....
  2420: 00 00 0C 0A 00 00 00 0C 13 00 00 00 0C 13 00 00  ................
  2430: 00 12 20 06 0C 5A 09 00 00 0C 80 38 01 00 0C 0A  .. ..Z.....8....
  2440: 00 00 00 0C 0A 00 00 00 0C 12 00 00 00 0C 12 00  ................
  2450: 00 00 12 20 06 0C D5 08 00 00 0C D0 01 01 00 0C  ... ............
  2460: 0A 00 00 00 0C 0A 00 00 00 0C 11 00 00 00 0C 11  ................
  2470: 00 00 00 12 20 06 0C 50 08 00 00 0C D8 D6 00 00  .... ..P........
  2480: 0C 0A 00 00 00 0C 0A 00 00 00 0C 10 00 00 00 0C  ................
  2490: 10 00 00 00 12 20 06 0C CB 07 00 00 0C 50 C3 00  ..... .......P..
  24A0: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0F 00 00 00  ................
  24B0: 0C 0F 00 00 00 12 20 06 0C 46 07 00 00 0C 28 A0  ...... ..F....(.
  24C0: 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0E 00 00  ................
  24D0: 00 0C 0E 00 00 00 12 20 06 0C C1 06 00 00 0C D0  ....... ........
  24E0: 84 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0D 00  ................
  24F0: 00 00 0C 0D 00 00 00 12 20 06 0C 3C 06 00 00 0C  ........ ..<....
  2500: 30 75 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C  0u..............
  2510: 00 00 00 0C 0C 00 00 00 A4 00 14 35 5F 50 43 54  ...........5_PCT
  2520: 00 A4 12 4D 02 02 11 14 0A 11 82 0C 00 7F 40 00  ...M..........@.
  2530: 00 99 01 00 00 00 00 00 00 79 00 11 14 0A 11 82  .........y......
  2540: 0C 00 7F 10 00 00 98 01 00 00 00 00 00 00 79 00  ..............y.
  2550: 14 08 5F 50 50 43 00 A4 00 14 42 03 5F 50 53 44  .._PPC....B._PSD
  2560: 00 A0 48 02 7B 54 59 50 45 0C 20 08 00 00 00 A4  ..H.{TYPE. .....
  2570: 12 49 01 01 12 15 05 0A 05 0A 00 0C 00 00 00 00  .I..............
  2580: 0C FE 00 00 00 0C 08 00 00 00 A4 00 14 4B 0F 5F  .............K._
  2590: 43 53 54 00 A0 41 0F 43 53 45 4E A0 4D 07 90 4D  CST..A.CSEN.M..M
  25A0: 57 4F 53 7B 54 59 50 45 0B 00 02 00 A4 12 4B 06  WOS{TYPE......K.
  25B0: 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 01 02  ....!...........
  25C0: 01 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01 00  .........y......
  25D0: 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00 7F  ......!.........
  25E0: 01 02 01 10 00 00 00 00 00 00 00 79 00 0A 02 0B  ...........y....
  25F0: 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82 0C  @.......!.......
  2600: 00 7F 01 02 01 20 00 00 00 00 00 00 00 79 00 0A  ..... .......y..
  2610: 03 0B 60 00 0C 5E 01 00 00 A4 12 4B 06 04 0A 03  ..`..^.....K....
  2620: 12 21 04 11 14 0A 11 82 0C 00 7F 00 00 00 00 00  .!..............
  2630: 00 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03  ......y.........
  2640: 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00  ...!............
  2650: 14 08 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C  ........y....@..
  2660: F4 01 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08  .....!..........
  2670: 00 00 15 08 00 00 00 00 00 00 79 00 0A 03 0B 60  ..........y....`
  2680: 00 0C 5E 01 00 00 A4 00 10 43 44 5C 2E 5F 50 52  ..^......CD\._PR
  2690: 5F 43 50 31 30 08 54 59 50 45 0C 00 00 00 00 08  _CP10.TYPE......
  26A0: 50 53 45 4E 0C 01 00 00 00 08 43 53 45 4E 0C 01  PSEN......CSEN..
  26B0: 00 00 00 08 4D 57 4F 53 0C 01 00 00 00 08 54 53  ....MWOS......TS
  26C0: 45 4E 0C 00 00 00 00 14 49 07 5F 50 44 43 01 8A  EN......I._PDC..
  26D0: 68 0A 08 43 41 50 41 70 43 41 50 41 54 59 50 45  h..CAPApCAPATYPE
  26E0: 8A 68 0A 00 52 45 56 53 8A 68 0A 04 53 49 5A 45  .h..REVS.h..SIZE
  26F0: 70 87 68 60 70 74 60 0A 08 00 61 5B 13 68 0A 40  p.h`pt`...a[.h.@
  2700: 77 61 0A 08 00 54 45 4D 50 08 53 54 53 30 11 07  wa...TEMP.STS0..
  2710: 0A 04 00 00 00 00 73 53 54 53 30 54 45 4D 50 62  ......sSTS0TEMPb
  2720: 5F 4F 53 43 11 13 0A 10 16 A6 77 40 0C 29 BE 47  _OSC......w@.).G
  2730: 9E BD D8 70 58 71 39 53 52 45 56 53 53 49 5A 45  ...pXq9SREVSSIZE
  2740: 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A 04 43 41 50  b.N._OSC..k..CAP
  2750: 41 70 43 41 50 41 54 59 50 45 8A 6B 0A 00 53 54  ApCAPATYPE.k..ST
  2760: 53 30 8A 6B 0A 04 43 41 50 30 8A 68 0A 00 49 49  S0.k..CAP0.h..II
  2770: 44 30 8A 68 0A 04 49 49 44 31 8A 68 0A 08 49 49  D0.h..IID1.h..II
  2780: 44 32 8A 68 0A 0C 49 49 44 33 08 55 49 44 30 11  D2.h..IID3.UID0.
  2790: 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58  .....w@.).G...pX
  27A0: 71 39 53 8A 68 0A 00 45 49 44 30 8A 68 0A 04 45  q9S.h..EID0.h..E
  27B0: 49 44 31 8A 68 0A 08 45 49 44 32 8A 68 0A 0C 45  ID1.h..EID2.h..E
  27C0: 49 44 33 A0 47 03 92 90 90 93 49 49 44 30 45 49  ID3.G.....IID0EI
  27D0: 44 30 93 49 49 44 31 45 49 44 31 90 93 49 49 44  D0.IID1EID1..IID
  27E0: 32 45 49 44 32 93 49 49 44 33 45 49 44 33 70 0A  2EID2.IID3EID3p.
  27F0: 06 88 53 54 53 30 0A 00 00 A4 6B A0 44 01 92 93  ..STS0....k.D...
  2800: 69 0A 01 70 0A 0A 88 53 54 53 30 0A 00 00 A4 6B  i..p...STS0....k
  2810: A0 48 01 7B 53 54 53 30 0A 01 00 7B 43 41 50 30  .H.{STS0...{CAP0
  2820: 0B FF 0B 43 41 50 30 A4 6B 7B 43 41 50 30 0B FF  ...CAP0.k{CAP0..
  2830: 0B 43 41 50 30 70 43 41 50 30 54 59 50 45 A4 6B  .CAP0pCAP0TYPE.k
  2840: 14 4D 11 5F 50 53 53 00 A0 43 11 50 53 45 4E A4  .M._PSS..C.PSEN.
  2850: 12 4B 10 08 12 20 06 0C 5B 09 00 00 0C 80 38 01  .K... ..[.....8.
  2860: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 13 00 00 00  ................
  2870: 0C 13 00 00 00 12 20 06 0C 5A 09 00 00 0C 80 38  ...... ..Z.....8
  2880: 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 12 00 00  ................
  2890: 00 0C 12 00 00 00 12 20 06 0C D5 08 00 00 0C D0  ....... ........
  28A0: 01 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 11 00  ................
  28B0: 00 00 0C 11 00 00 00 12 20 06 0C 50 08 00 00 0C  ........ ..P....
  28C0: D8 D6 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 10  ................
  28D0: 00 00 00 0C 10 00 00 00 12 20 06 0C CB 07 00 00  ......... ......
  28E0: 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C  .P..............
  28F0: 0F 00 00 00 0C 0F 00 00 00 12 20 06 0C 46 07 00  .......... ..F..
  2900: 00 0C 28 A0 00 00 0C 0A 00 00 00 0C 0A 00 00 00  ..(.............
  2910: 0C 0E 00 00 00 0C 0E 00 00 00 12 20 06 0C C1 06  ........... ....
  2920: 00 00 0C D0 84 00 00 0C 0A 00 00 00 0C 0A 00 00  ................
  2930: 00 0C 0D 00 00 00 0C 0D 00 00 00 12 20 06 0C 3C  ............ ..<
  2940: 06 00 00 0C 30 75 00 00 0C 0A 00 00 00 0C 0A 00  ....0u..........
  2950: 00 00 0C 0C 00 00 00 0C 0C 00 00 00 A4 00 14 35  ...............5
  2960: 5F 50 43 54 00 A4 12 4D 02 02 11 14 0A 11 82 0C  _PCT...M........
  2970: 00 7F 40 00 00 99 01 00 00 00 00 00 00 79 00 11  ..@..........y..
  2980: 14 0A 11 82 0C 00 7F 10 00 00 98 01 00 00 00 00  ................
  2990: 00 00 79 00 14 08 5F 50 50 43 00 A4 00 14 42 03  ..y..._PPC....B.
  29A0: 5F 50 53 44 00 A0 48 02 7B 54 59 50 45 0C 20 08  _PSD..H.{TYPE. .
  29B0: 00 00 00 A4 12 49 01 01 12 15 05 0A 05 0A 00 0C  .....I..........
  29C0: 00 00 00 00 0C FE 00 00 00 0C 08 00 00 00 A4 00  ................
  29D0: 14 4B 0F 5F 43 53 54 00 A0 41 0F 43 53 45 4E A0  .K._CST..A.CSEN.
  29E0: 4D 07 90 4D 57 4F 53 7B 54 59 50 45 0B 00 02 00  M..MWOS{TYPE....
  29F0: A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82 0C  ..K.....!.......
  2A00: 00 7F 01 02 01 00 00 00 00 00 00 00 00 79 00 0A  .............y..
  2A10: 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A 11  ..........!.....
  2A20: 82 0C 00 7F 01 02 01 10 00 00 00 00 00 00 00 79  ...............y
  2A30: 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11 14  ....@.......!...
  2A40: 0A 11 82 0C 00 7F 01 02 01 20 00 00 00 00 00 00  ......... ......
  2A50: 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 12 4B  .y....`..^.....K
  2A60: 06 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 00  .....!..........
  2A70: 00 00 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01  ..........y.....
  2A80: 00 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00  .......!........
  2A90: 01 08 00 00 14 08 00 00 00 00 00 00 79 00 0A 02  ............y...
  2AA0: 0B 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82  .@.......!......
  2AB0: 0C 00 01 08 00 00 15 08 00 00 00 00 00 00 79 00  ..............y.
  2AC0: 0A 03 0B 60 00 0C 5E 01 00 00 A4 00 10 43 44 5C  ...`..^......CD\
  2AD0: 2E 5F 50 52 5F 43 50 31 31 08 54 59 50 45 0C 00  ._PR_CP11.TYPE..
  2AE0: 00 00 00 08 50 53 45 4E 0C 01 00 00 00 08 43 53  ....PSEN......CS
  2AF0: 45 4E 0C 01 00 00 00 08 4D 57 4F 53 0C 01 00 00  EN......MWOS....
  2B00: 00 08 54 53 45 4E 0C 00 00 00 00 14 49 07 5F 50  ..TSEN......I._P
  2B10: 44 43 01 8A 68 0A 08 43 41 50 41 70 43 41 50 41  DC..h..CAPApCAPA
  2B20: 54 59 50 45 8A 68 0A 00 52 45 56 53 8A 68 0A 04  TYPE.h..REVS.h..
  2B30: 53 49 5A 45 70 87 68 60 70 74 60 0A 08 00 61 5B  SIZEp.h`pt`...a[
  2B40: 13 68 0A 40 77 61 0A 08 00 54 45 4D 50 08 53 54  .h.@wa...TEMP.ST
  2B50: 53 30 11 07 0A 04 00 00 00 00 73 53 54 53 30 54  S0........sSTS0T
  2B60: 45 4D 50 62 5F 4F 53 43 11 13 0A 10 16 A6 77 40  EMPb_OSC......w@
  2B70: 0C 29 BE 47 9E BD D8 70 58 71 39 53 52 45 56 53  .).G...pXq9SREVS
  2B80: 53 49 5A 45 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A  SIZEb.N._OSC..k.
  2B90: 04 43 41 50 41 70 43 41 50 41 54 59 50 45 8A 6B  .CAPApCAPATYPE.k
  2BA0: 0A 00 53 54 53 30 8A 6B 0A 04 43 41 50 30 8A 68  ..STS0.k..CAP0.h
  2BB0: 0A 00 49 49 44 30 8A 68 0A 04 49 49 44 31 8A 68  ..IID0.h..IID1.h
  2BC0: 0A 08 49 49 44 32 8A 68 0A 0C 49 49 44 33 08 55  ..IID2.h..IID3.U
  2BD0: 49 44 30 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E  ID0......w@.).G.
  2BE0: BD D8 70 58 71 39 53 8A 68 0A 00 45 49 44 30 8A  ..pXq9S.h..EID0.
  2BF0: 68 0A 04 45 49 44 31 8A 68 0A 08 45 49 44 32 8A  h..EID1.h..EID2.
  2C00: 68 0A 0C 45 49 44 33 A0 47 03 92 90 90 93 49 49  h..EID3.G.....II
  2C10: 44 30 45 49 44 30 93 49 49 44 31 45 49 44 31 90  D0EID0.IID1EID1.
  2C20: 93 49 49 44 32 45 49 44 32 93 49 49 44 33 45 49  .IID2EID2.IID3EI
  2C30: 44 33 70 0A 06 88 53 54 53 30 0A 00 00 A4 6B A0  D3p...STS0....k.
  2C40: 44 01 92 93 69 0A 01 70 0A 0A 88 53 54 53 30 0A  D...i..p...STS0.
  2C50: 00 00 A4 6B A0 48 01 7B 53 54 53 30 0A 01 00 7B  ...k.H.{STS0...{
  2C60: 43 41 50 30 0B FF 0B 43 41 50 30 A4 6B 7B 43 41  CAP0...CAP0.k{CA
  2C70: 50 30 0B FF 0B 43 41 50 30 70 43 41 50 30 54 59  P0...CAP0pCAP0TY
  2C80: 50 45 A4 6B 14 4D 11 5F 50 53 53 00 A0 43 11 50  PE.k.M._PSS..C.P
  2C90: 53 45 4E A4 12 4B 10 08 12 20 06 0C 5B 09 00 00  SEN..K... ..[...
  2CA0: 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C  ..8.............
  2CB0: 13 00 00 00 0C 13 00 00 00 12 20 06 0C 5A 09 00  .......... ..Z..
  2CC0: 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00  ...8............
  2CD0: 0C 12 00 00 00 0C 12 00 00 00 12 20 06 0C D5 08  ........... ....
  2CE0: 00 00 0C D0 01 01 00 0C 0A 00 00 00 0C 0A 00 00  ................
  2CF0: 00 0C 11 00 00 00 0C 11 00 00 00 12 20 06 0C 50  ............ ..P
  2D00: 08 00 00 0C D8 D6 00 00 0C 0A 00 00 00 0C 0A 00  ................
  2D10: 00 00 0C 10 00 00 00 0C 10 00 00 00 12 20 06 0C  ............. ..
  2D20: CB 07 00 00 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A  .....P..........
  2D30: 00 00 00 0C 0F 00 00 00 0C 0F 00 00 00 12 20 06  .............. .
  2D40: 0C 46 07 00 00 0C 28 A0 00 00 0C 0A 00 00 00 0C  .F....(.........
  2D50: 0A 00 00 00 0C 0E 00 00 00 0C 0E 00 00 00 12 20  ............... 
  2D60: 06 0C C1 06 00 00 0C D0 84 00 00 0C 0A 00 00 00  ................
  2D70: 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D 00 00 00 12  ................
  2D80: 20 06 0C 3C 06 00 00 0C 30 75 00 00 0C 0A 00 00   ..<....0u......
  2D90: 00 0C 0A 00 00 00 0C 0C 00 00 00 0C 0C 00 00 00  ................
  2DA0: A4 00 14 35 5F 50 43 54 00 A4 12 4D 02 02 11 14  ...5_PCT...M....
  2DB0: 0A 11 82 0C 00 7F 40 00 00 99 01 00 00 00 00 00  ......@.........
  2DC0: 00 79 00 11 14 0A 11 82 0C 00 7F 10 00 00 98 01  .y..............
  2DD0: 00 00 00 00 00 00 79 00 14 08 5F 50 50 43 00 A4  ......y..._PPC..
  2DE0: 00 14 42 03 5F 50 53 44 00 A0 48 02 7B 54 59 50  ..B._PSD..H.{TYP
  2DF0: 45 0C 20 08 00 00 00 A4 12 49 01 01 12 15 05 0A  E. ......I......
  2E00: 05 0A 00 0C 00 00 00 00 0C FE 00 00 00 0C 08 00  ................
  2E10: 00 00 A4 00 14 4B 0F 5F 43 53 54 00 A0 41 0F 43  .....K._CST..A.C
  2E20: 53 45 4E A0 4D 07 90 4D 57 4F 53 7B 54 59 50 45  SEN.M..MWOS{TYPE
  2E30: 0B 00 02 00 A4 12 4B 06 04 0A 03 12 21 04 11 14  ......K.....!...
  2E40: 0A 11 82 0C 00 7F 01 02 01 00 00 00 00 00 00 00  ................
  2E50: 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21 04  .y............!.
  2E60: 11 14 0A 11 82 0C 00 7F 01 02 01 10 00 00 00 00  ................
  2E70: 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00 12  ...y....@.......
  2E80: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 20 00 00  !............ ..
  2E90: 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00  .....y....`..^..
  2EA0: 00 A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82  ...K.....!......
  2EB0: 0C 00 7F 00 00 00 00 00 00 00 00 00 00 00 79 00  ..............y.
  2EC0: 0A 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A  ...........!....
  2ED0: 11 82 0C 00 01 08 00 00 14 08 00 00 00 00 00 00  ................
  2EE0: 79 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11  y....@.......!..
  2EF0: 14 0A 11 82 0C 00 01 08 00 00 15 08 00 00 00 00  ................
  2F00: 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 00  ..y....`..^.....
  2F10: 10 43 44 5C 2E 5F 50 52 5F 43 50 31 32 08 54 59  .CD\._PR_CP12.TY
  2F20: 50 45 0C 00 00 00 00 08 50 53 45 4E 0C 01 00 00  PE......PSEN....
  2F30: 00 08 43 53 45 4E 0C 01 00 00 00 08 4D 57 4F 53  ..CSEN......MWOS
  2F40: 0C 01 00 00 00 08 54 53 45 4E 0C 00 00 00 00 14  ......TSEN......
  2F50: 49 07 5F 50 44 43 01 8A 68 0A 08 43 41 50 41 70  I._PDC..h..CAPAp
  2F60: 43 41 50 41 54 59 50 45 8A 68 0A 00 52 45 56 53  CAPATYPE.h..REVS
  2F70: 8A 68 0A 04 53 49 5A 45 70 87 68 60 70 74 60 0A  .h..SIZEp.h`pt`.
  2F80: 08 00 61 5B 13 68 0A 40 77 61 0A 08 00 54 45 4D  ..a[.h.@wa...TEM
  2F90: 50 08 53 54 53 30 11 07 0A 04 00 00 00 00 73 53  P.STS0........sS
  2FA0: 54 53 30 54 45 4D 50 62 5F 4F 53 43 11 13 0A 10  TS0TEMPb_OSC....
  2FB0: 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53  ..w@.).G...pXq9S
  2FC0: 52 45 56 53 53 49 5A 45 62 14 4E 0F 5F 4F 53 43  REVSSIZEb.N._OSC
  2FD0: 04 8A 6B 0A 04 43 41 50 41 70 43 41 50 41 54 59  ..k..CAPApCAPATY
  2FE0: 50 45 8A 6B 0A 00 53 54 53 30 8A 6B 0A 04 43 41  PE.k..STS0.k..CA
  2FF0: 50 30 8A 68 0A 00 49 49 44 30 8A 68 0A 04 49 49  P0.h..IID0.h..II
  3000: 44 31 8A 68 0A 08 49 49 44 32 8A 68 0A 0C 49 49  D1.h..IID2.h..II
  3010: 44 33 08 55 49 44 30 11 13 0A 10 16 A6 77 40 0C  D3.UID0......w@.
  3020: 29 BE 47 9E BD D8 70 58 71 39 53 8A 68 0A 00 45  ).G...pXq9S.h..E
  3030: 49 44 30 8A 68 0A 04 45 49 44 31 8A 68 0A 08 45  ID0.h..EID1.h..E
  3040: 49 44 32 8A 68 0A 0C 45 49 44 33 A0 47 03 92 90  ID2.h..EID3.G...
  3050: 90 93 49 49 44 30 45 49 44 30 93 49 49 44 31 45  ..IID0EID0.IID1E
  3060: 49 44 31 90 93 49 49 44 32 45 49 44 32 93 49 49  ID1..IID2EID2.II
  3070: 44 33 45 49 44 33 70 0A 06 88 53 54 53 30 0A 00  D3EID3p...STS0..
  3080: 00 A4 6B A0 44 01 92 93 69 0A 01 70 0A 0A 88 53  ..k.D...i..p...S
  3090: 54 53 30 0A 00 00 A4 6B A0 48 01 7B 53 54 53 30  TS0....k.H.{STS0
  30A0: 0A 01 00 7B 43 41 50 30 0B FF 0B 43 41 50 30 A4  ...{CAP0...CAP0.
  30B0: 6B 7B 43 41 50 30 0B FF 0B 43 41 50 30 70 43 41  k{CAP0...CAP0pCA
  30C0: 50 30 54 59 50 45 A4 6B 14 4D 11 5F 50 53 53 00  P0TYPE.k.M._PSS.
  30D0: A0 43 11 50 53 45 4E A4 12 4B 10 08 12 20 06 0C  .C.PSEN..K... ..
  30E0: 5B 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A  [.....8.........
  30F0: 00 00 00 0C 13 00 00 00 0C 13 00 00 00 12 20 06  .............. .
  3100: 0C 5A 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C  .Z.....8........
  3110: 0A 00 00 00 0C 12 00 00 00 0C 12 00 00 00 12 20  ............... 
  3120: 06 0C D5 08 00 00 0C D0 01 01 00 0C 0A 00 00 00  ................
  3130: 0C 0A 00 00 00 0C 11 00 00 00 0C 11 00 00 00 12  ................
  3140: 20 06 0C 50 08 00 00 0C D8 D6 00 00 0C 0A 00 00   ..P............
  3150: 00 0C 0A 00 00 00 0C 10 00 00 00 0C 10 00 00 00  ................
  3160: 12 20 06 0C CB 07 00 00 0C 50 C3 00 00 0C 0A 00  . .......P......
  3170: 00 00 0C 0A 00 00 00 0C 0F 00 00 00 0C 0F 00 00  ................
  3180: 00 12 20 06 0C 46 07 00 00 0C 28 A0 00 00 0C 0A  .. ..F....(.....
  3190: 00 00 00 0C 0A 00 00 00 0C 0E 00 00 00 0C 0E 00  ................
  31A0: 00 00 12 20 06 0C C1 06 00 00 0C D0 84 00 00 0C  ... ............
  31B0: 0A 00 00 00 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D  ................
  31C0: 00 00 00 12 20 06 0C 3C 06 00 00 0C 30 75 00 00  .... ..<....0u..
  31D0: 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C 00 00 00 0C  ................
  31E0: 0C 00 00 00 A4 00 14 35 5F 50 43 54 00 A4 12 4D  .......5_PCT...M
  31F0: 02 02 11 14 0A 11 82 0C 00 7F 40 00 00 99 01 00  ..........@.....
  3200: 00 00 00 00 00 79 00 11 14 0A 11 82 0C 00 7F 10  .....y..........
  3210: 00 00 98 01 00 00 00 00 00 00 79 00 14 08 5F 50  ..........y..._P
  3220: 50 43 00 A4 00 14 42 03 5F 50 53 44 00 A0 48 02  PC....B._PSD..H.
  3230: 7B 54 59 50 45 0C 20 08 00 00 00 A4 12 49 01 01  {TYPE. ......I..
  3240: 12 15 05 0A 05 0A 00 0C 00 00 00 00 0C FE 00 00  ................
  3250: 00 0C 08 00 00 00 A4 00 14 4B 0F 5F 43 53 54 00  .........K._CST.
  3260: A0 41 0F 43 53 45 4E A0 4D 07 90 4D 57 4F 53 7B  .A.CSEN.M..MWOS{
  3270: 54 59 50 45 0B 00 02 00 A4 12 4B 06 04 0A 03 12  TYPE......K.....
  3280: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 00 00 00  !...............
  3290: 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03 00  .....y..........
  32A0: 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 10  ..!.............
  32B0: 00 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C F4  .......y....@...
  32C0: 01 00 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02  ....!...........
  32D0: 01 20 00 00 00 00 00 00 00 79 00 0A 03 0B 60 00  . .......y....`.
  32E0: 0C 5E 01 00 00 A4 12 4B 06 04 0A 03 12 21 04 11  .^.....K.....!..
  32F0: 14 0A 11 82 0C 00 7F 00 00 00 00 00 00 00 00 00  ................
  3300: 00 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21  ..y............!
  3310: 04 11 14 0A 11 82 0C 00 01 08 00 00 14 08 00 00  ................
  3320: 00 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00  ....y....@......
  3330: 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00 15 08  .!..............
  3340: 00 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01  ......y....`..^.
  3350: 00 00 A4 00 10 43 44 5C 2E 5F 50 52 5F 43 50 31  .....CD\._PR_CP1
  3360: 33 08 54 59 50 45 0C 00 00 00 00 08 50 53 45 4E  3.TYPE......PSEN
  3370: 0C 01 00 00 00 08 43 53 45 4E 0C 01 00 00 00 08  ......CSEN......
  3380: 4D 57 4F 53 0C 01 00 00 00 08 54 53 45 4E 0C 00  MWOS......TSEN..
  3390: 00 00 00 14 49 07 5F 50 44 43 01 8A 68 0A 08 43  ....I._PDC..h..C
  33A0: 41 50 41 70 43 41 50 41 54 59 50 45 8A 68 0A 00  APApCAPATYPE.h..
  33B0: 52 45 56 53 8A 68 0A 04 53 49 5A 45 70 87 68 60  REVS.h..SIZEp.h`
  33C0: 70 74 60 0A 08 00 61 5B 13 68 0A 40 77 61 0A 08  pt`...a[.h.@wa..
  33D0: 00 54 45 4D 50 08 53 54 53 30 11 07 0A 04 00 00  .TEMP.STS0......
  33E0: 00 00 73 53 54 53 30 54 45 4D 50 62 5F 4F 53 43  ..sSTS0TEMPb_OSC
  33F0: 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70  ......w@.).G...p
  3400: 58 71 39 53 52 45 56 53 53 49 5A 45 62 14 4E 0F  Xq9SREVSSIZEb.N.
  3410: 5F 4F 53 43 04 8A 6B 0A 04 43 41 50 41 70 43 41  _OSC..k..CAPApCA
  3420: 50 41 54 59 50 45 8A 6B 0A 00 53 54 53 30 8A 6B  PATYPE.k..STS0.k
  3430: 0A 04 43 41 50 30 8A 68 0A 00 49 49 44 30 8A 68  ..CAP0.h..IID0.h
  3440: 0A 04 49 49 44 31 8A 68 0A 08 49 49 44 32 8A 68  ..IID1.h..IID2.h
  3450: 0A 0C 49 49 44 33 08 55 49 44 30 11 13 0A 10 16  ..IID3.UID0.....
  3460: A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53 8A  .w@.).G...pXq9S.
  3470: 68 0A 00 45 49 44 30 8A 68 0A 04 45 49 44 31 8A  h..EID0.h..EID1.
  3480: 68 0A 08 45 49 44 32 8A 68 0A 0C 45 49 44 33 A0  h..EID2.h..EID3.
  3490: 47 03 92 90 90 93 49 49 44 30 45 49 44 30 93 49  G.....IID0EID0.I
  34A0: 49 44 31 45 49 44 31 90 93 49 49 44 32 45 49 44  ID1EID1..IID2EID
  34B0: 32 93 49 49 44 33 45 49 44 33 70 0A 06 88 53 54  2.IID3EID3p...ST
  34C0: 53 30 0A 00 00 A4 6B A0 44 01 92 93 69 0A 01 70  S0....k.D...i..p
  34D0: 0A 0A 88 53 54 53 30 0A 00 00 A4 6B A0 48 01 7B  ...STS0....k.H.{
  34E0: 53 54 53 30 0A 01 00 7B 43 41 50 30 0B FF 0B 43  STS0...{CAP0...C
  34F0: 41 50 30 A4 6B 7B 43 41 50 30 0B FF 0B 43 41 50  AP0.k{CAP0...CAP
  3500: 30 70 43 41 50 30 54 59 50 45 A4 6B 14 4D 11 5F  0pCAP0TYPE.k.M._
  3510: 50 53 53 00 A0 43 11 50 53 45 4E A4 12 4B 10 08  PSS..C.PSEN..K..
  3520: 12 20 06 0C 5B 09 00 00 0C 80 38 01 00 0C 0A 00  . ..[.....8.....
  3530: 00 00 0C 0A 00 00 00 0C 13 00 00 00 0C 13 00 00  ................
  3540: 00 12 20 06 0C 5A 09 00 00 0C 80 38 01 00 0C 0A  .. ..Z.....8....
  3550: 00 00 00 0C 0A 00 00 00 0C 12 00 00 00 0C 12 00  ................
  3560: 00 00 12 20 06 0C D5 08 00 00 0C D0 01 01 00 0C  ... ............
  3570: 0A 00 00 00 0C 0A 00 00 00 0C 11 00 00 00 0C 11  ................
  3580: 00 00 00 12 20 06 0C 50 08 00 00 0C D8 D6 00 00  .... ..P........
  3590: 0C 0A 00 00 00 0C 0A 00 00 00 0C 10 00 00 00 0C  ................
  35A0: 10 00 00 00 12 20 06 0C CB 07 00 00 0C 50 C3 00  ..... .......P..
  35B0: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0F 00 00 00  ................
  35C0: 0C 0F 00 00 00 12 20 06 0C 46 07 00 00 0C 28 A0  ...... ..F....(.
  35D0: 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0E 00 00  ................
  35E0: 00 0C 0E 00 00 00 12 20 06 0C C1 06 00 00 0C D0  ....... ........
  35F0: 84 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0D 00  ................
  3600: 00 00 0C 0D 00 00 00 12 20 06 0C 3C 06 00 00 0C  ........ ..<....
  3610: 30 75 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C  0u..............
  3620: 00 00 00 0C 0C 00 00 00 A4 00 14 35 5F 50 43 54  ...........5_PCT
  3630: 00 A4 12 4D 02 02 11 14 0A 11 82 0C 00 7F 40 00  ...M..........@.
  3640: 00 99 01 00 00 00 00 00 00 79 00 11 14 0A 11 82  .........y......
  3650: 0C 00 7F 10 00 00 98 01 00 00 00 00 00 00 79 00  ..............y.
  3660: 14 08 5F 50 50 43 00 A4 00 14 42 03 5F 50 53 44  .._PPC....B._PSD
  3670: 00 A0 48 02 7B 54 59 50 45 0C 20 08 00 00 00 A4  ..H.{TYPE. .....
  3680: 12 49 01 01 12 15 05 0A 05 0A 00 0C 01 00 00 00  .I..............
  3690: 0C FE 00 00 00 0C 08 00 00 00 A4 00 14 4B 0F 5F  .............K._
  36A0: 43 53 54 00 A0 41 0F 43 53 45 4E A0 4D 07 90 4D  CST..A.CSEN.M..M
  36B0: 57 4F 53 7B 54 59 50 45 0B 00 02 00 A4 12 4B 06  WOS{TYPE......K.
  36C0: 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 01 02  ....!...........
  36D0: 01 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01 00  .........y......
  36E0: 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00 7F  ......!.........
  36F0: 01 02 01 10 00 00 00 00 00 00 00 79 00 0A 02 0B  ...........y....
  3700: 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82 0C  @.......!.......
  3710: 00 7F 01 02 01 20 00 00 00 00 00 00 00 79 00 0A  ..... .......y..
  3720: 03 0B 60 00 0C 5E 01 00 00 A4 12 4B 06 04 0A 03  ..`..^.....K....
  3730: 12 21 04 11 14 0A 11 82 0C 00 7F 00 00 00 00 00  .!..............
  3740: 00 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03  ......y.........
  3750: 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00  ...!............
  3760: 14 08 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C  ........y....@..
  3770: F4 01 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08  .....!..........
  3780: 00 00 15 08 00 00 00 00 00 00 79 00 0A 03 0B 60  ..........y....`
  3790: 00 0C 5E 01 00 00 A4 00 10 43 44 5C 2E 5F 50 52  ..^......CD\._PR
  37A0: 5F 43 50 31 34 08 54 59 50 45 0C 00 00 00 00 08  _CP14.TYPE......
  37B0: 50 53 45 4E 0C 01 00 00 00 08 43 53 45 4E 0C 01  PSEN......CSEN..
  37C0: 00 00 00 08 4D 57 4F 53 0C 01 00 00 00 08 54 53  ....MWOS......TS
  37D0: 45 4E 0C 00 00 00 00 14 49 07 5F 50 44 43 01 8A  EN......I._PDC..
  37E0: 68 0A 08 43 41 50 41 70 43 41 50 41 54 59 50 45  h..CAPApCAPATYPE
  37F0: 8A 68 0A 00 52 45 56 53 8A 68 0A 04 53 49 5A 45  .h..REVS.h..SIZE
  3800: 70 87 68 60 70 74 60 0A 08 00 61 5B 13 68 0A 40  p.h`pt`...a[.h.@
  3810: 77 61 0A 08 00 54 45 4D 50 08 53 54 53 30 11 07  wa...TEMP.STS0..
  3820: 0A 04 00 00 00 00 73 53 54 53 30 54 45 4D 50 62  ......sSTS0TEMPb
  3830: 5F 4F 53 43 11 13 0A 10 16 A6 77 40 0C 29 BE 47  _OSC......w@.).G
  3840: 9E BD D8 70 58 71 39 53 52 45 56 53 53 49 5A 45  ...pXq9SREVSSIZE
  3850: 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A 04 43 41 50  b.N._OSC..k..CAP
  3860: 41 70 43 41 50 41 54 59 50 45 8A 6B 0A 00 53 54  ApCAPATYPE.k..ST
  3870: 53 30 8A 6B 0A 04 43 41 50 30 8A 68 0A 00 49 49  S0.k..CAP0.h..II
  3880: 44 30 8A 68 0A 04 49 49 44 31 8A 68 0A 08 49 49  D0.h..IID1.h..II
  3890: 44 32 8A 68 0A 0C 49 49 44 33 08 55 49 44 30 11  D2.h..IID3.UID0.
  38A0: 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58  .....w@.).G...pX
  38B0: 71 39 53 8A 68 0A 00 45 49 44 30 8A 68 0A 04 45  q9S.h..EID0.h..E
  38C0: 49 44 31 8A 68 0A 08 45 49 44 32 8A 68 0A 0C 45  ID1.h..EID2.h..E
  38D0: 49 44 33 A0 47 03 92 90 90 93 49 49 44 30 45 49  ID3.G.....IID0EI
  38E0: 44 30 93 49 49 44 31 45 49 44 31 90 93 49 49 44  D0.IID1EID1..IID
  38F0: 32 45 49 44 32 93 49 49 44 33 45 49 44 33 70 0A  2EID2.IID3EID3p.
  3900: 06 88 53 54 53 30 0A 00 00 A4 6B A0 44 01 92 93  ..STS0....k.D...
  3910: 69 0A 01 70 0A 0A 88 53 54 53 30 0A 00 00 A4 6B  i..p...STS0....k
  3920: A0 48 01 7B 53 54 53 30 0A 01 00 7B 43 41 50 30  .H.{STS0...{CAP0
  3930: 0B FF 0B 43 41 50 30 A4 6B 7B 43 41 50 30 0B FF  ...CAP0.k{CAP0..
  3940: 0B 43 41 50 30 70 43 41 50 30 54 59 50 45 A4 6B  .CAP0pCAP0TYPE.k
  3950: 14 4D 11 5F 50 53 53 00 A0 43 11 50 53 45 4E A4  .M._PSS..C.PSEN.
  3960: 12 4B 10 08 12 20 06 0C 5B 09 00 00 0C 80 38 01  .K... ..[.....8.
  3970: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 13 00 00 00  ................
  3980: 0C 13 00 00 00 12 20 06 0C 5A 09 00 00 0C 80 38  ...... ..Z.....8
  3990: 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 12 00 00  ................
  39A0: 00 0C 12 00 00 00 12 20 06 0C D5 08 00 00 0C D0  ....... ........
  39B0: 01 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 11 00  ................
  39C0: 00 00 0C 11 00 00 00 12 20 06 0C 50 08 00 00 0C  ........ ..P....
  39D0: D8 D6 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 10  ................
  39E0: 00 00 00 0C 10 00 00 00 12 20 06 0C CB 07 00 00  ......... ......
  39F0: 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C  .P..............
  3A00: 0F 00 00 00 0C 0F 00 00 00 12 20 06 0C 46 07 00  .......... ..F..
  3A10: 00 0C 28 A0 00 00 0C 0A 00 00 00 0C 0A 00 00 00  ..(.............
  3A20: 0C 0E 00 00 00 0C 0E 00 00 00 12 20 06 0C C1 06  ........... ....
  3A30: 00 00 0C D0 84 00 00 0C 0A 00 00 00 0C 0A 00 00  ................
  3A40: 00 0C 0D 00 00 00 0C 0D 00 00 00 12 20 06 0C 3C  ............ ..<
  3A50: 06 00 00 0C 30 75 00 00 0C 0A 00 00 00 0C 0A 00  ....0u..........
  3A60: 00 00 0C 0C 00 00 00 0C 0C 00 00 00 A4 00 14 35  ...............5
  3A70: 5F 50 43 54 00 A4 12 4D 02 02 11 14 0A 11 82 0C  _PCT...M........
  3A80: 00 7F 40 00 00 99 01 00 00 00 00 00 00 79 00 11  ..@..........y..
  3A90: 14 0A 11 82 0C 00 7F 10 00 00 98 01 00 00 00 00  ................
  3AA0: 00 00 79 00 14 08 5F 50 50 43 00 A4 00 14 42 03  ..y..._PPC....B.
  3AB0: 5F 50 53 44 00 A0 48 02 7B 54 59 50 45 0C 20 08  _PSD..H.{TYPE. .
  3AC0: 00 00 00 A4 12 49 01 01 12 15 05 0A 05 0A 00 0C  .....I..........
  3AD0: 01 00 00 00 0C FE 00 00 00 0C 08 00 00 00 A4 00  ................
  3AE0: 14 4B 0F 5F 43 53 54 00 A0 41 0F 43 53 45 4E A0  .K._CST..A.CSEN.
  3AF0: 4D 07 90 4D 57 4F 53 7B 54 59 50 45 0B 00 02 00  M..MWOS{TYPE....
  3B00: A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82 0C  ..K.....!.......
  3B10: 00 7F 01 02 01 00 00 00 00 00 00 00 00 79 00 0A  .............y..
  3B20: 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A 11  ..........!.....
  3B30: 82 0C 00 7F 01 02 01 10 00 00 00 00 00 00 00 79  ...............y
  3B40: 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11 14  ....@.......!...
  3B50: 0A 11 82 0C 00 7F 01 02 01 20 00 00 00 00 00 00  ......... ......
  3B60: 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 12 4B  .y....`..^.....K
  3B70: 06 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 00  .....!..........
  3B80: 00 00 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01  ..........y.....
  3B90: 00 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00  .......!........
  3BA0: 01 08 00 00 14 08 00 00 00 00 00 00 79 00 0A 02  ............y...
  3BB0: 0B 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82  .@.......!......
  3BC0: 0C 00 01 08 00 00 15 08 00 00 00 00 00 00 79 00  ..............y.
  3BD0: 0A 03 0B 60 00 0C 5E 01 00 00 A4 00 10 43 44 5C  ...`..^......CD\
  3BE0: 2E 5F 50 52 5F 43 50 31 35 08 54 59 50 45 0C 00  ._PR_CP15.TYPE..
  3BF0: 00 00 00 08 50 53 45 4E 0C 01 00 00 00 08 43 53  ....PSEN......CS
  3C00: 45 4E 0C 01 00 00 00 08 4D 57 4F 53 0C 01 00 00  EN......MWOS....
  3C10: 00 08 54 53 45 4E 0C 00 00 00 00 14 49 07 5F 50  ..TSEN......I._P
  3C20: 44 43 01 8A 68 0A 08 43 41 50 41 70 43 41 50 41  DC..h..CAPApCAPA
  3C30: 54 59 50 45 8A 68 0A 00 52 45 56 53 8A 68 0A 04  TYPE.h..REVS.h..
  3C40: 53 49 5A 45 70 87 68 60 70 74 60 0A 08 00 61 5B  SIZEp.h`pt`...a[
  3C50: 13 68 0A 40 77 61 0A 08 00 54 45 4D 50 08 53 54  .h.@wa...TEMP.ST
  3C60: 53 30 11 07 0A 04 00 00 00 00 73 53 54 53 30 54  S0........sSTS0T
  3C70: 45 4D 50 62 5F 4F 53 43 11 13 0A 10 16 A6 77 40  EMPb_OSC......w@
  3C80: 0C 29 BE 47 9E BD D8 70 58 71 39 53 52 45 56 53  .).G...pXq9SREVS
  3C90: 53 49 5A 45 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A  SIZEb.N._OSC..k.
  3CA0: 04 43 41 50 41 70 43 41 50 41 54 59 50 45 8A 6B  .CAPApCAPATYPE.k
  3CB0: 0A 00 53 54 53 30 8A 6B 0A 04 43 41 50 30 8A 68  ..STS0.k..CAP0.h
  3CC0: 0A 00 49 49 44 30 8A 68 0A 04 49 49 44 31 8A 68  ..IID0.h..IID1.h
  3CD0: 0A 08 49 49 44 32 8A 68 0A 0C 49 49 44 33 08 55  ..IID2.h..IID3.U
  3CE0: 49 44 30 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E  ID0......w@.).G.
  3CF0: BD D8 70 58 71 39 53 8A 68 0A 00 45 49 44 30 8A  ..pXq9S.h..EID0.
  3D00: 68 0A 04 45 49 44 31 8A 68 0A 08 45 49 44 32 8A  h..EID1.h..EID2.
  3D10: 68 0A 0C 45 49 44 33 A0 47 03 92 90 90 93 49 49  h..EID3.G.....II
  3D20: 44 30 45 49 44 30 93 49 49 44 31 45 49 44 31 90  D0EID0.IID1EID1.
  3D30: 93 49 49 44 32 45 49 44 32 93 49 49 44 33 45 49  .IID2EID2.IID3EI
  3D40: 44 33 70 0A 06 88 53 54 53 30 0A 00 00 A4 6B A0  D3p...STS0....k.
  3D50: 44 01 92 93 69 0A 01 70 0A 0A 88 53 54 53 30 0A  D...i..p...STS0.
  3D60: 00 00 A4 6B A0 48 01 7B 53 54 53 30 0A 01 00 7B  ...k.H.{STS0...{
  3D70: 43 41 50 30 0B FF 0B 43 41 50 30 A4 6B 7B 43 41  CAP0...CAP0.k{CA
  3D80: 50 30 0B FF 0B 43 41 50 30 70 43 41 50 30 54 59  P0...CAP0pCAP0TY
  3D90: 50 45 A4 6B 14 4D 11 5F 50 53 53 00 A0 43 11 50  PE.k.M._PSS..C.P
  3DA0: 53 45 4E A4 12 4B 10 08 12 20 06 0C 5B 09 00 00  SEN..K... ..[...
  3DB0: 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C  ..8.............
  3DC0: 13 00 00 00 0C 13 00 00 00 12 20 06 0C 5A 09 00  .......... ..Z..
  3DD0: 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00  ...8............
  3DE0: 0C 12 00 00 00 0C 12 00 00 00 12 20 06 0C D5 08  ........... ....
  3DF0: 00 00 0C D0 01 01 00 0C 0A 00 00 00 0C 0A 00 00  ................
  3E00: 00 0C 11 00 00 00 0C 11 00 00 00 12 20 06 0C 50  ............ ..P
  3E10: 08 00 00 0C D8 D6 00 00 0C 0A 00 00 00 0C 0A 00  ................
  3E20: 00 00 0C 10 00 00 00 0C 10 00 00 00 12 20 06 0C  ............. ..
  3E30: CB 07 00 00 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A  .....P..........
  3E40: 00 00 00 0C 0F 00 00 00 0C 0F 00 00 00 12 20 06  .............. .
  3E50: 0C 46 07 00 00 0C 28 A0 00 00 0C 0A 00 00 00 0C  .F....(.........
  3E60: 0A 00 00 00 0C 0E 00 00 00 0C 0E 00 00 00 12 20  ............... 
  3E70: 06 0C C1 06 00 00 0C D0 84 00 00 0C 0A 00 00 00  ................
  3E80: 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D 00 00 00 12  ................
  3E90: 20 06 0C 3C 06 00 00 0C 30 75 00 00 0C 0A 00 00   ..<....0u......
  3EA0: 00 0C 0A 00 00 00 0C 0C 00 00 00 0C 0C 00 00 00  ................
  3EB0: A4 00 14 35 5F 50 43 54 00 A4 12 4D 02 02 11 14  ...5_PCT...M....
  3EC0: 0A 11 82 0C 00 7F 40 00 00 99 01 00 00 00 00 00  ......@.........
  3ED0: 00 79 00 11 14 0A 11 82 0C 00 7F 10 00 00 98 01  .y..............
  3EE0: 00 00 00 00 00 00 79 00 14 08 5F 50 50 43 00 A4  ......y..._PPC..
  3EF0: 00 14 42 03 5F 50 53 44 00 A0 48 02 7B 54 59 50  ..B._PSD..H.{TYP
  3F00: 45 0C 20 08 00 00 00 A4 12 49 01 01 12 15 05 0A  E. ......I......
  3F10: 05 0A 00 0C 01 00 00 00 0C FE 00 00 00 0C 08 00  ................
  3F20: 00 00 A4 00 14 4B 0F 5F 43 53 54 00 A0 41 0F 43  .....K._CST..A.C
  3F30: 53 45 4E A0 4D 07 90 4D 57 4F 53 7B 54 59 50 45  SEN.M..MWOS{TYPE
  3F40: 0B 00 02 00 A4 12 4B 06 04 0A 03 12 21 04 11 14  ......K.....!...
  3F50: 0A 11 82 0C 00 7F 01 02 01 00 00 00 00 00 00 00  ................
  3F60: 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21 04  .y............!.
  3F70: 11 14 0A 11 82 0C 00 7F 01 02 01 10 00 00 00 00  ................
  3F80: 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00 12  ...y....@.......
  3F90: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 20 00 00  !............ ..
  3FA0: 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00  .....y....`..^..
  3FB0: 00 A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82  ...K.....!......
  3FC0: 0C 00 7F 00 00 00 00 00 00 00 00 00 00 00 79 00  ..............y.
  3FD0: 0A 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A  ...........!....
  3FE0: 11 82 0C 00 01 08 00 00 14 08 00 00 00 00 00 00  ................
  3FF0: 79 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11  y....@.......!..
  4000: 14 0A 11 82 0C 00 01 08 00 00 15 08 00 00 00 00  ................
  4010: 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 00  ..y....`..^.....
  4020: 10 43 44 5C 2E 5F 50 52 5F 43 50 31 36 08 54 59  .CD\._PR_CP16.TY
  4030: 50 45 0C 00 00 00 00 08 50 53 45 4E 0C 01 00 00  PE......PSEN....
  4040: 00 08 43 53 45 4E 0C 01 00 00 00 08 4D 57 4F 53  ..CSEN......MWOS
  4050: 0C 01 00 00 00 08 54 53 45 4E 0C 00 00 00 00 14  ......TSEN......
  4060: 49 07 5F 50 44 43 01 8A 68 0A 08 43 41 50 41 70  I._PDC..h..CAPAp
  4070: 43 41 50 41 54 59 50 45 8A 68 0A 00 52 45 56 53  CAPATYPE.h..REVS
  4080: 8A 68 0A 04 53 49 5A 45 70 87 68 60 70 74 60 0A  .h..SIZEp.h`pt`.
  4090: 08 00 61 5B 13 68 0A 40 77 61 0A 08 00 54 45 4D  ..a[.h.@wa...TEM
  40A0: 50 08 53 54 53 30 11 07 0A 04 00 00 00 00 73 53  P.STS0........sS
  40B0: 54 53 30 54 45 4D 50 62 5F 4F 53 43 11 13 0A 10  TS0TEMPb_OSC....
  40C0: 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53  ..w@.).G...pXq9S
  40D0: 52 45 56 53 53 49 5A 45 62 14 4E 0F 5F 4F 53 43  REVSSIZEb.N._OSC
  40E0: 04 8A 6B 0A 04 43 41 50 41 70 43 41 50 41 54 59  ..k..CAPApCAPATY
  40F0: 50 45 8A 6B 0A 00 53 54 53 30 8A 6B 0A 04 43 41  PE.k..STS0.k..CA
  4100: 50 30 8A 68 0A 00 49 49 44 30 8A 68 0A 04 49 49  P0.h..IID0.h..II
  4110: 44 31 8A 68 0A 08 49 49 44 32 8A 68 0A 0C 49 49  D1.h..IID2.h..II
  4120: 44 33 08 55 49 44 30 11 13 0A 10 16 A6 77 40 0C  D3.UID0......w@.
  4130: 29 BE 47 9E BD D8 70 58 71 39 53 8A 68 0A 00 45  ).G...pXq9S.h..E
  4140: 49 44 30 8A 68 0A 04 45 49 44 31 8A 68 0A 08 45  ID0.h..EID1.h..E
  4150: 49 44 32 8A 68 0A 0C 45 49 44 33 A0 47 03 92 90  ID2.h..EID3.G...
  4160: 90 93 49 49 44 30 45 49 44 30 93 49 49 44 31 45  ..IID0EID0.IID1E
  4170: 49 44 31 90 93 49 49 44 32 45 49 44 32 93 49 49  ID1..IID2EID2.II
  4180: 44 33 45 49 44 33 70 0A 06 88 53 54 53 30 0A 00  D3EID3p...STS0..
  4190: 00 A4 6B A0 44 01 92 93 69 0A 01 70 0A 0A 88 53  ..k.D...i..p...S
  41A0: 54 53 30 0A 00 00 A4 6B A0 48 01 7B 53 54 53 30  TS0....k.H.{STS0
  41B0: 0A 01 00 7B 43 41 50 30 0B FF 0B 43 41 50 30 A4  ...{CAP0...CAP0.
  41C0: 6B 7B 43 41 50 30 0B FF 0B 43 41 50 30 70 43 41  k{CAP0...CAP0pCA
  41D0: 50 30 54 59 50 45 A4 6B 14 4D 11 5F 50 53 53 00  P0TYPE.k.M._PSS.
  41E0: A0 43 11 50 53 45 4E A4 12 4B 10 08 12 20 06 0C  .C.PSEN..K... ..
  41F0: 5B 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A  [.....8.........
  4200: 00 00 00 0C 13 00 00 00 0C 13 00 00 00 12 20 06  .............. .
  4210: 0C 5A 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C  .Z.....8........
  4220: 0A 00 00 00 0C 12 00 00 00 0C 12 00 00 00 12 20  ............... 
  4230: 06 0C D5 08 00 00 0C D0 01 01 00 0C 0A 00 00 00  ................
  4240: 0C 0A 00 00 00 0C 11 00 00 00 0C 11 00 00 00 12  ................
  4250: 20 06 0C 50 08 00 00 0C D8 D6 00 00 0C 0A 00 00   ..P............
  4260: 00 0C 0A 00 00 00 0C 10 00 00 00 0C 10 00 00 00  ................
  4270: 12 20 06 0C CB 07 00 00 0C 50 C3 00 00 0C 0A 00  . .......P......
  4280: 00 00 0C 0A 00 00 00 0C 0F 00 00 00 0C 0F 00 00  ................
  4290: 00 12 20 06 0C 46 07 00 00 0C 28 A0 00 00 0C 0A  .. ..F....(.....
  42A0: 00 00 00 0C 0A 00 00 00 0C 0E 00 00 00 0C 0E 00  ................
  42B0: 00 00 12 20 06 0C C1 06 00 00 0C D0 84 00 00 0C  ... ............
  42C0: 0A 00 00 00 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D  ................
  42D0: 00 00 00 12 20 06 0C 3C 06 00 00 0C 30 75 00 00  .... ..<....0u..
  42E0: 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C 00 00 00 0C  ................
  42F0: 0C 00 00 00 A4 00 14 35 5F 50 43 54 00 A4 12 4D  .......5_PCT...M
  4300: 02 02 11 14 0A 11 82 0C 00 7F 40 00 00 99 01 00  ..........@.....
  4310: 00 00 00 00 00 79 00 11 14 0A 11 82 0C 00 7F 10  .....y..........
  4320: 00 00 98 01 00 00 00 00 00 00 79 00 14 08 5F 50  ..........y..._P
  4330: 50 43 00 A4 00 14 42 03 5F 50 53 44 00 A0 48 02  PC....B._PSD..H.
  4340: 7B 54 59 50 45 0C 20 08 00 00 00 A4 12 49 01 01  {TYPE. ......I..
  4350: 12 15 05 0A 05 0A 00 0C 01 00 00 00 0C FE 00 00  ................
  4360: 00 0C 08 00 00 00 A4 00 14 4B 0F 5F 43 53 54 00  .........K._CST.
  4370: A0 41 0F 43 53 45 4E A0 4D 07 90 4D 57 4F 53 7B  .A.CSEN.M..MWOS{
  4380: 54 59 50 45 0B 00 02 00 A4 12 4B 06 04 0A 03 12  TYPE......K.....
  4390: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 00 00 00  !...............
  43A0: 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03 00  .....y..........
  43B0: 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 10  ..!.............
  43C0: 00 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C F4  .......y....@...
  43D0: 01 00 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02  ....!...........
  43E0: 01 20 00 00 00 00 00 00 00 79 00 0A 03 0B 60 00  . .......y....`.
  43F0: 0C 5E 01 00 00 A4 12 4B 06 04 0A 03 12 21 04 11  .^.....K.....!..
  4400: 14 0A 11 82 0C 00 7F 00 00 00 00 00 00 00 00 00  ................
  4410: 00 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21  ..y............!
  4420: 04 11 14 0A 11 82 0C 00 01 08 00 00 14 08 00 00  ................
  4430: 00 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00  ....y....@......
  4440: 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00 15 08  .!..............
  4450: 00 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01  ......y....`..^.
  4460: 00 00 A4 00                                      ....


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Andrew Cooper]

On 28/07/15 04:52, Juergen Gross wrote:
> On 07/28/2015 01:19 AM, Andrew Cooper wrote:
>> On 27/07/2015 18:42, Dario Faggioli wrote:
>>> On Mon, 2015-07-27 at 17:33 +0100, Andrew Cooper wrote:
>>>> On 27/07/15 17:31, David Vrabel wrote:
>>>>>
>>>>>> Yeah, indeed.
>>>>>> That's the downside of Juergen's "Linux scheduler
>>>>>> approach". But the issue is there, even without taking vNUMA into
>>>>>> account, and I  think something like that would really help (only
>>>>>> for
>>>>>> Dom0, and Linux guests, of course).
>>>>> I disagree.  Whether we're using vNUMA or not, Xen should still
>>>>> ensure
>>>>> that the guest kernel and userspace see a consistent and correct
>>>>> topology using the native mechanisms.
>>>>
>>>> +1
>>>>
>>> +1 from me as well. In fact, a mechanism for making exactly such thing
>>> happen, was what I was after when starting the thread.
>>>
>>> Then it came up that CPUID needs to be used for at least two different
>>> and potentially conflicting purposes, that we want to support both and
>>> that, whether and for whatever reason it's used, Linux configures its
>>> scheduler after it, potentially resulting in rather pathological
>>> setups.
>>>
>> I don't see what the problem is here.  Fundamentally, "NUMA optimise" vs
>> "comply with licence" is a user/admin decision at boot time, and we need
>> not cater to both halves at the same time.
>>
>> Supporting either, as chosen by the admin, is worthwhile.
>
> Wrong assumption again. *It's not only about NUMA*!
> The choice is: "comply with license" against "sane scheduling". NUMA
> just makes it more obvious, that the data the guest's scheduling
> decisions are based on is garbage as soon as you tell the guest there
> are hyperthreads without pinning the vcpus.

Fine.  s/NUMA/scheduling/

But the point still stands.  The CPUID information should be consistent
with other sources of information.

~Andrew

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Wei Liu]

On Fri, Jul 24, 2015 at 06:05:59PM +0200, Dario Faggioli wrote:
> On Fri, 2015-07-24 at 17:14 +0200, Juergen Gross wrote:
> > On 07/24/2015 04:44 PM, Dario Faggioli wrote:
> 
> > > Ok. And I already have a question (as I lost track of things a bit).
> > > What you just said about ACPI tables is certainly true for baremetal and
> > > HVM guests, but for PV? At the time I was looking into it, together with
> > > Elena, there were Linux patches being produced for the PV case, which
> > > makes sense.
> > > However, ISTR that both Wei and Elena mentioned recently that those
> > > patches have not been upstreamed in Linux yet... Is that the case? Maybe
> > > not all, but at least some of them are there? Because if not, I'm not
> > > sure I see how a PV guest would even see a vNUMA topology (which it
> > > does).
> 
> > I checked "bottom up", so when I found the acpi scan stuff I stopped
> > searching how the kernel obtains numa info. During my search I found no
> > clue of an pv-numa stuff in the kernel. And a quick "grep -i numa" in
> > arch/x86/xen and drivers/xen didn't reveal anything. Same for a complete
> > kernel source search for "vnuma".
> > 
> BTW, I've also been grepping, and I'm not seeing XENMEM_get_vnumainfo
> being called anywhere either... Well, no wonder, we're seeing vNUMA
> setup issues! If I did check for this before, I wouldn't even have
> tested PV-vNUMA in the first place!! :-O
> 
> I actually wonder how _some_ of the vNUMA info get through!?!? :-O
> 

Maybe you're using a custom built kernel with my PV vNUMA patch in?

I sent that to you once because you wanted to test that.

Wei.

> IAC, all we're saying remains true, of course, and the CPUID issue will
> still be there, if we don't fix it, when such a patch will be accepted.
> 
> Dario
> -- 
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

[-- Attachment #1: Type: text/plain, Size: 2906 bytes --]

On 07/28/2015 06:29 AM, Juergen Gross wrote:
> On 07/27/2015 04:09 PM, Dario Faggioli wrote:
>> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote:
>>> On 07/24/2015 05:58 PM, Dario Faggioli wrote:
>>
>>>> So, just to check if I'm understanding is correct: you'd like to add an
>>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
>>>> code, to hide the direct interaction with CPUID.
>>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
>>>> check with Xen/match vNUMA/whatever... Is this that you are saying?
>>>
>>> Sort of, yes.
>>>
>>> I just wouldn't add it, as it is already existing (more or less). It
>>> can deal right now with AMD and Intel, we would "just" have to add Xen.
>>>
>> So, having gone through the rest of the thread (so far), and having
>> given a fair amount o thinking to this, I really think that something
>> like this would be a good thing to have in Linux.
>>
>> Of course, it's not that my opinion on where should be in Linux counts
>> that much! :-D   Nevertheless, I wanted to make it clear that, while
>> skeptic at the beginning, I now think this is (part of) the way to go,
>> as I said and explained in my reply to George.
>
> I think it's time to obtain some real numbers.
>
> I'll make some performance tests on a big machine (4 sockets, 60 cores,
> 120 threads) regarding topology information:
>
> - bare metal
> - "random" topology (like today)
> - "simple" topology (all vcpus regarded as equal)
> - "real" topology with all vcpus pinned
>
> This should show:
>
> - how intrusive would the topology patch(es) be?
> - what is the performance impact of a "wrong" scheduling data base

On the above box I used a pvops kernel 4.2-rc4 plus a rather small patch
(see attachment). I did 5 kernel builds in each environment:

make clean
time make -j 120

The first result of the 5 runs was always omitted as it would have to
build up buffer caches etc. The Xen cases were all done in dom0, pinning
of vcpus in the last scenario was done via dom0_vcpus_pin boot parameter
of the hypervisor.

Here are the results (everything in seconds):

                     elapsed   user   system
bare metal:            100    5770      805
"random" topology:     283    6740    20700
"simple" topology:     290    6740    22200
"real" topology:       185    7800     8040

As expected bare metal is the best. Next is "real" topology with pinned
vcpus (expected again - but system time already factor of 10 up!).
What I didn't expect is: "random" is better than "simple" topology. I
could test some other topologies (e.g. everything on one socket, or even
on one core), but I'm not sure this makes sense. I didn't check the
exact topology result of the "random" case, maybe I'll do that tomorrow
with another measurement.

BTW: the topology hack is working, as each cpu is shown to have a
sibling count of 1 in /proc/cpuinfo.


Juergen

[-- Attachment #2: topo.patch --]
[-- Type: text/x-patch, Size: 2057 bytes --]

commit 2d9cee2b37319714e07e6f8b4044c0db44cc8e7d
Author: Juergen Gross <jgross@suse.com>
Date:   Tue Jul 28 09:28:35 2015 +0200

    xen: use simple topology on demand
    
    As the cpuid information is currently taken from the physical cpu the
    current virtual cpu is running on, the topology information derived
    from the cpuid is potentially useless.
    
    In order to avoid insane scheduling decisions based on random data
    provide a possibility to set topology data to a very simple scheme
    making all cpus appear to be independent from each other.
    
    Signed-off-by: Juergen Gross <jgross@suse.com>

diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c
index 8648438..a57e816 100644
--- a/arch/x86/xen/smp.c
+++ b/arch/x86/xen/smp.c
@@ -55,6 +55,25 @@ static irqreturn_t xen_call_function_interrupt(int irq, void *dev_id);
 static irqreturn_t xen_call_function_single_interrupt(int irq, void *dev_id);
 static irqreturn_t xen_irq_work_interrupt(int irq, void *dev_id);
 
+static bool xen_simple_topology;
+static __init int xen_parse_topology(char *arg)
+{
+	xen_simple_topology = true;
+	return 0;
+}
+early_param("xen_simple_topology", xen_parse_topology);
+
+static void xen_set_cpu_sibling_map(int cpu)
+{
+	if (xen_simple_topology) {
+		cpumask_set_cpu(cpu, topology_sibling_cpumask(cpu));
+		cpumask_set_cpu(cpu, cpu_llc_shared_mask(cpu));
+		cpumask_set_cpu(cpu, topology_core_cpumask(cpu));
+	} else {
+		set_cpu_sibling_map(cpu);
+	}
+}
+
 /*
  * Reschedule call back.
  */
@@ -82,7 +101,7 @@ static void cpu_bringup(void)
 	cpu = smp_processor_id();
 	smp_store_cpu_info(cpu);
 	cpu_data(cpu).x86_max_cores = 1;
-	set_cpu_sibling_map(cpu);
+	xen_set_cpu_sibling_map(cpu);
 
 	xen_setup_cpu_clockevents();
 
@@ -333,7 +352,7 @@ static void __init xen_smp_prepare_cpus(unsigned int max_cpus)
 		zalloc_cpumask_var(&per_cpu(cpu_core_map, i), GFP_KERNEL);
 		zalloc_cpumask_var(&per_cpu(cpu_llc_shared_map, i), GFP_KERNEL);
 	}
-	set_cpu_sibling_map(0);
+	xen_set_cpu_sibling_map(0);
 
 	if (xen_smp_intr_init(0))
 		BUG();

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1812 bytes --]

On Tue, 2015-07-28 at 11:05 +0100, Wei Liu wrote:
> On Fri, Jul 24, 2015 at 06:05:59PM +0200, Dario Faggioli wrote:

> > BTW, I've also been grepping, and I'm not seeing XENMEM_get_vnumainfo
> > being called anywhere either... Well, no wonder, we're seeing vNUMA
> > setup issues! If I did check for this before, I wouldn't even have
> > tested PV-vNUMA in the first place!! :-O
> > 
> > I actually wonder how _some_ of the vNUMA info get through!?!? :-O
> > 
> 
> Maybe you're using a custom built kernel with my PV vNUMA patch in?
> 
Yep, that must be the case... I only recalled about that after sending
this email (no, I don't update Dom0/guest kernel for my testbox really
often! :-/)

> I sent that to you once because you wanted to test that.
> 
And I confirmed it. Booting the guest with Debian's stick 4.0.0, I get
this from Xen:

(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 129914):
(XEN)     Node 0: 56111
(XEN)     Node 1: 73803
(XEN) Domain 2 (total: 262144):
(XEN)     Node 0: 131072
(XEN)     Node 1: 131072
(XEN)      2 vnodes, 4 vcpus, guest physical layout:
(XEN)          0: pnode   0, vcpus 0-1 
(XEN)            0000000000000000 - 0000000020000000
(XEN)          1: pnode   1, vcpus 2-3
(XEN)            0000000020000000 - 0000000040000000


But the (PV) guest does not actually know anything about vNUMA:

root@test:~# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3
node 0 size: 993 MB
node 0 free: 918 MB
node distances:
node   0 
  0:  10 

Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 5849 bytes --]

On Tue, 2015-07-28 at 17:11 +0200, Juergen Gross wrote:
> On 07/28/2015 06:29 AM, Juergen Gross wrote:

> > I'll make some performance tests on a big machine (4 sockets, 60 cores,
> > 120 threads) regarding topology information:
> >
> > - bare metal
> > - "random" topology (like today)
> > - "simple" topology (all vcpus regarded as equal)
> > - "real" topology with all vcpus pinned
> >
> > This should show:
> >
> > - how intrusive would the topology patch(es) be?
> > - what is the performance impact of a "wrong" scheduling data base
> 
> On the above box I used a pvops kernel 4.2-rc4 plus a rather small patch
> (see attachment). I did 5 kernel builds in each environment:
> 
> make clean
> time make -j 120
> 
Right. If you have time, can you try '-j60' and '-j30' (maybe even -j45
and -j15, if you've got _a_lot_ of time! :-)).

I'm asking this because, with hyperthreading involved, I've sometimes
seen things being the worse when *not* (over)saturating the CPU
capacity.

The explanation is that, if every vcpu is busy, meaning that every
thread is busy, it does not make much difference where you schedule the
busy vcpus.

OTOH, if only 1/2 of the threads are busy, a properly setup system will
effectively spread the load in such a way that each vcpu has a full core
available; a messed up one will, when trying to do the same, end up
scheduling stuff on siblings, even if there are idle cores available.

In this case, things are a bit more tricky. In fact, I've observed the
above while looking after the Xen scheduler. In this case, it is the
guest (dom0) scheduler that we are looking at, and, e.g., if the load is
small enough, Xen's scheduler will fix things up, at least up to a
certain extent.

It's worth a try anyway, I guess, if you have time, of course.

> The first result of the 5 runs was always omitted as it would have to
> build up buffer caches etc. The Xen cases were all done in dom0, pinning
> of vcpus in the last scenario was done via dom0_vcpus_pin boot parameter
> of the hypervisor.
> 
> Here are the results (everything in seconds):
> 
>                      elapsed   user   system
> bare metal:            100    5770      805
> "random" topology:     283    6740    20700
> "simple" topology:     290    6740    22200
> "real" topology:       185    7800     8040
> 
> As expected bare metal is the best. Next is "real" topology with pinned
> vcpus (expected again - but system time already factor of 10 up!).
>
I also think that (massively) overloading biases things in favour of
pinning. In fact, pinning incurs in less overhead, as there are no
scheduling decisions involved, and no migrations of vcpus among pcpus.
With the system oversubscribed to to 200%, even in the non-pinning case
there shouldn't be much migrations, but certainly there will be some,
and they turn out to be pure overhead! In fact, they bring zero
benefits, as it's not possible that any of them will put the system in a
more advantageous state, performance wise: we're fully loaded and we
want to stay fully loaded!

> What I didn't expect is: "random" is better than "simple" topology. 
>
Weird indeed!

> I
> could test some other topologies (e.g. everything on one socket, or even
> on one core), but I'm not sure this makes sense. I didn't check the
> exact topology result of the "random" case, maybe I'll do that tomorrow
> with another measurement.
> 
So, my test box looks like this:
cpu_topology           :
cpu:    core    socket     node
  0:       0        1        0
  1:       0        1        0
  2:       1        1        0
  3:       1        1        0
  4:       9        1        0
  5:       9        1        0
  6:      10        1        0
  7:      10        1        0
  8:       0        0        1
  9:       0        0        1
 10:       1        0        1
 11:       1        0        1
 12:       9        0        1
 13:       9        0        1
 14:      10        0        1
 15:      10        0        1

In Dom0, here's what I see _without_ any pinning:

root@Zhaman:~# for i in `seq 0 15`;do cat /sys/devices/system/cpu/cpu$i/topology/thread_siblings_list ;done
0-1
0-1
2-3
2-3
4-5
4-5
6-7
6-7
8-9
8-9
10-11
10-11
12-13
12-13
14-15
14-15

root@Zhaman:~# cat /proc/cpuinfo |grep "physical id"
physical id	: 1
physical id	: 1
physical id	: 1
physical id	: 1
physical id	: 1
physical id	: 1
physical id	: 1
physical id	: 1
physical id	: 0
physical id	: 0
physical id	: 0
physical id	: 0
physical id	: 0
physical id	: 0
physical id	: 0
physical id	: 0

root@Zhaman:~# cat /proc/cpuinfo |grep "core id"
core id		: 0
core id		: 0
core id		: 1
core id		: 1
core id		: 9
core id		: 9
core id		: 10
core id		: 10
core id		: 0
core id		: 0
core id		: 1
core id		: 1
core id		: 9
core id		: 9
core id		: 10
core id		: 10

root@Zhaman:~# cat /proc/cpuinfo |grep "cpu cores"
cpu cores	: 4
<same for all cpus>

root@Zhaman:~# cat /proc/cpuinfo |grep "siblings" 
siblings	: 8
<same for all cpus>

So, basically, as far as Dom0 on my test box is concerned, "random"
actually matches the host topology.

Sure, without pinning, this looks equally wrong, as Xen's scheduler can
well execute, say, vcpu 0 and vcpu 4, which are not siblings, on the
same core. But then again, if the load is small, it just won't happen
(e.g., if there are only those two busy vcpus, Xen will send them on
!siblings core), while if it's too hugh, it won't matter... :-/

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 4297 bytes --]

On Tue, 2015-07-28 at 18:17 +0200, Dario Faggioli wrote:

> So, my test box looks like this:
> cpu_topology           :
> cpu:    core    socket     node
>   0:       0        1        0
>   1:       0        1        0
>   2:       1        1        0
>   3:       1        1        0
>   4:       9        1        0
>   5:       9        1        0
>   6:      10        1        0
>   7:      10        1        0
>   8:       0        0        1
>   9:       0        0        1
>  10:       1        0        1
>  11:       1        0        1
>  12:       9        0        1
>  13:       9        0        1
>  14:      10        0        1
>  15:      10        0        1
> 
> In Dom0, here's what I see _without_ any pinning:
> 
And now I've tried with dom0_vcpus_pin:

root@Zhaman:~# xl vcpu-list 
Name                                ID  VCPU   CPU State   Time(s) Affinity (Hard / Soft)
Domain-0                             0     0    0   -b-       6.8  0 / all
Domain-0                             0     1    1   -b-       1.6  1 / all
Domain-0                             0     2    2   -b-       2.3  2 / all
Domain-0                             0     3    3   -b-       1.5  3 / all
Domain-0                             0     4    4   -b-       3.2  4 / all
Domain-0                             0     5    5   -b-       1.5  5 / all
Domain-0                             0     6    6   -b-       2.0  6 / all
Domain-0                             0     7    7   -b-       2.2  7 / all
Domain-0                             0     8    8   -b-       1.6  8 / all
Domain-0                             0     9    9   -b-       1.6  9 / all
Domain-0                             0    10   10   r--       2.1  10 / all
Domain-0                             0    11   11   -b-       1.5  11 / all
Domain-0                             0    12   12   -b-       2.4  12 / all
Domain-0                             0    13   13   -b-       1.1  13 / all
Domain-0                             0    14   14   -b-       2.4  14 / all
Domain-0                             0    15   15   -b-       2.4  15 / all

> root@Zhaman:~# for i in `seq 0 15`;do cat /sys/devices/system/cpu/cpu$i/topology/thread_siblings_list ;done
> 0-1
> 0-1
> 2-3
> 2-3
> 4-5
> 4-5
> 6-7
> 6-7
> 8-9
> 8-9
> 10-11
> 10-11
> 12-13
> 12-13
> 14-15
> 14-15
> 
Same result.

> root@Zhaman:~# cat /proc/cpuinfo |grep "physical id"
> physical id	: 1
> physical id	: 1
> physical id	: 1
> physical id	: 1
> physical id	: 1
> physical id	: 1
> physical id	: 1
> physical id	: 1
> physical id	: 0
> physical id	: 0
> physical id	: 0
> physical id	: 0
> physical id	: 0
> physical id	: 0
> physical id	: 0
> physical id	: 0
> 
Same result.

> root@Zhaman:~# cat /proc/cpuinfo |grep "core id"
> core id		: 0
> core id		: 0
> core id		: 1
> core id		: 1
> core id		: 9
> core id		: 9
> core id		: 10
> core id		: 10
> core id		: 0
> core id		: 0
> core id		: 1
> core id		: 1
> core id		: 9
> core id		: 9
> core id		: 10
> core id		: 10
> 
Same result.

> root@Zhaman:~# cat /proc/cpuinfo |grep "cpu cores"
> cpu cores	: 4
> <same for all cpus>
> 
Same result.

> root@Zhaman:~# cat /proc/cpuinfo |grep "siblings" 
> siblings	: 8
> <same for all cpus>
> 
And same result here as well.

So, for Dom0, pinning does not make much difference as far as what
topology is guessed, which actually makes sense, considering how pcpus
are brought up in Xen, and assuming that Linux does something similar
(no, I'm not familiar with that code).

It should be making a difference in terms of whether and how much this
topology, although matching the host one, mislead the guest scheduler.
Actually, I still think it does, it's just harder to identify than we
expected, which may be seen as a good thing (it does not look like it is
a big deal, after all :-D), or a bad thing (it may start biting us
anytime, without us noticing promptly :-/).

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross]

On 07/28/2015 06:17 PM, Dario Faggioli wrote:
> On Tue, 2015-07-28 at 17:11 +0200, Juergen Gross wrote:
>> On 07/28/2015 06:29 AM, Juergen Gross wrote:
>
>>> I'll make some performance tests on a big machine (4 sockets, 60 cores,
>>> 120 threads) regarding topology information:
>>>
>>> - bare metal
>>> - "random" topology (like today)
>>> - "simple" topology (all vcpus regarded as equal)
>>> - "real" topology with all vcpus pinned
>>>
>>> This should show:
>>>
>>> - how intrusive would the topology patch(es) be?
>>> - what is the performance impact of a "wrong" scheduling data base
>>
>> On the above box I used a pvops kernel 4.2-rc4 plus a rather small patch
>> (see attachment). I did 5 kernel builds in each environment:
>>
>> make clean
>> time make -j 120
>>
> Right. If you have time, can you try '-j60' and '-j30' (maybe even -j45
> and -j15, if you've got _a_lot_ of time! :-)).

The test machine can do this without me watching, so I've just started
the first configuration...

> I'm asking this because, with hyperthreading involved, I've sometimes
> seen things being the worse when *not* (over)saturating the CPU
> capacity.

Hmm, oversaturation shouldn't happen here. I've added -j 240 to let it
happen.

...

> So, basically, as far as Dom0 on my test box is concerned, "random"
> actually matches the host topology.

Okay, have to check that on my box.

I think I'll have another try with a domU. This could be much more
"random" than dom0.

> Sure, without pinning, this looks equally wrong, as Xen's scheduler can
> well execute, say, vcpu 0 and vcpu 4, which are not siblings, on the
> same core. But then again, if the load is small, it just won't happen
> (e.g., if there are only those two busy vcpus, Xen will send them on
> !siblings core), while if it's too hugh, it won't matter... :-/

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1088 bytes --]

On Wed, 2015-07-29 at 08:04 +0200, Juergen Gross wrote:
> On 07/28/2015 06:17 PM, Dario Faggioli wrote:
> >> On 07/28/2015 06:29 AM, Juergen Gross wrote:
> >
> >>> I'll make some performance tests on a big machine (4 sockets, 60 cores,
> >>> 120 threads) regarding topology information:
> >>>
> > I'm asking this because, with hyperthreading involved, I've sometimes
> > seen things being the worse when *not* (over)saturating the CPU
> > capacity.
> 
> Hmm, oversaturation shouldn't happen here. I've added -j 240 to let it
> happen.
> 
Oh, sorry... I was doing the math wrong (I was not counting the threads,
which is funny, as this is pretty much all about threads! :-O)

> I think I'll have another try with a domU. This could be much more
> "random" than dom0.
> 
Indeed!

Thanks and Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 4528 bytes --]

On Tue, 2015-07-28 at 17:11 +0200, Juergen Gross wrote:
> On 07/28/2015 06:29 AM, Juergen Gross wrote:
> > On 07/27/2015 04:09 PM, Dario Faggioli wrote:
> >> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote:
> >>> On 07/24/2015 05:58 PM, Dario Faggioli wrote:
> >>
> >>>> So, just to check if I'm understanding is correct: you'd like to add an
> >>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
> >>>> code, to hide the direct interaction with CPUID.
> >>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
> >>>> check with Xen/match vNUMA/whatever... Is this that you are saying?
> >>>
> >>> Sort of, yes.
> >>>
> >>> I just wouldn't add it, as it is already existing (more or less). It
> >>> can deal right now with AMD and Intel, we would "just" have to add Xen.
> >>>
> >> So, having gone through the rest of the thread (so far), and having
> >> given a fair amount o thinking to this, I really think that something
> >> like this would be a good thing to have in Linux.
> >>
> >> Of course, it's not that my opinion on where should be in Linux counts
> >> that much! :-D   Nevertheless, I wanted to make it clear that, while
> >> skeptic at the beginning, I now think this is (part of) the way to go,
> >> as I said and explained in my reply to George.
> >
> > I think it's time to obtain some real numbers.
> >
> > I'll make some performance tests on a big machine (4 sockets, 60 cores,
> > 120 threads) regarding topology information:
> >
> > - bare metal
> > - "random" topology (like today)
> > - "simple" topology (all vcpus regarded as equal)
> > - "real" topology with all vcpus pinned
> >
> > This should show:
> >
> > - how intrusive would the topology patch(es) be?
> > - what is the performance impact of a "wrong" scheduling data base
> 
> On the above box I used a pvops kernel 4.2-rc4 plus a rather small patch
> (see attachment). I did 5 kernel builds in each environment:
> 
> make clean
> time make -j 120
> 
> The first result of the 5 runs was always omitted as it would have to
> build up buffer caches etc. The Xen cases were all done in dom0, pinning
> of vcpus in the last scenario was done via dom0_vcpus_pin boot parameter
> of the hypervisor.
> 
> Here are the results (everything in seconds):
> 
>                      elapsed   user   system
> bare metal:            100    5770      805
> "random" topology:     283    6740    20700
> "simple" topology:     290    6740    22200
> "real" topology:       185    7800     8040
> 
> As expected bare metal is the best. Next is "real" topology with pinned
> vcpus (expected again - but system time already factor of 10 up!).
> What I didn't expect is: "random" is better than "simple" topology. I
> could test some other topologies (e.g. everything on one socket, or even
> on one core), but I'm not sure this makes sense. I didn't check the
> exact topology result of the "random" case, maybe I'll do that tomorrow
> with another measurement.
>
> BTW: the topology hack is working, as each cpu is shown to have a
> sibling count of 1 in /proc/cpuinfo.
> 
Hey, just a 'wild' idea, as a possible explanation (or direction for
further investigations) on why 'simple' si actually worse than 'random'.

Could it be that, by setting up the topology like in 'simple', we create
more work for the Linux scheduler, and hence incur in more overhead? Or,
looking at thing the other way round, which is it the topology that
impose the less overhead, in terms of load balancing work, to the Linux
scheduler?

I think we should identify and try it as, in the absence of any vNUMA
topology being specified, that's what we want, as our flat topology (and
with vNUMA, that's what we want within virtual nodes).


I don't really know, right now, whether such topology would be 'all
cores, no SMT siblings', or 'all siblings of each others', or 'all
sockets, no core/SMT siblings'... This requires inspecting the
scheduler's and scheduling domain's code, which I can do, but not today,
as I won't be working. If you (or anyone) think it's worth and actually
have a look, let me/us know your findings. If not, I'll do it myself
tomorrow.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel