* PV-vNUMA issue: topology is misinterpreted by the guest @ 2015-07-16 10:32 Dario Faggioli 2015-07-16 10:47 ` Jan Beulich ` (2 more replies) 0 siblings, 3 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-16 10:32 UTC (permalink / raw) To: xen-devel Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 3987 bytes --] Hey, This started on IRC, but it's actually appropriate to have the conversation here. I just discovered an issue with vNUMA, when PV guests are used. In fact, creating a 4 vCPUs PV guest, and making up things so that all the 4 vCPUs should be busy, I see this: root@Zhaman:~# xl vcpu-list test Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) test 4 0 5 r-- 1481.9 all / 0-7 test 4 1 2 r-- 1479.4 all / 0-7 test 4 2 15 -b- 7.5 all / 8-15 test 4 3 10 -b- 1324.8 all / 8-15 Going checking inside the guest, confirms that *everything* runs on vCPUs 0 and 1. However, using schedtool or taskset, I can force tasks to execute on vCPUs 2 and 3. Inspecting the guest's dmesg, I've seen this: [ 0.128416] ------------[ cut here ]------------ [ 0.128416] WARNING: CPU: 2 PID: 0 at ../arch/x86/kernel/smpboot.c:317 topology_sane.isra.2+0x74/0x88() [ 0.128416] sched: CPU #2's smt-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency. [ 0.128416] Modules linked in: [ 0.128416] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.19.0+ #1 [ 0.128416] 0000000000000009 ffff88001ee3bdd0 ffffffff81657c7b ffffffff810bbd2c [ 0.128416] ffff88001ee3be20 ffff88001ee3be10 ffffffff81081510 ffff88001ee3bea0 [ 0.128416] ffffffff8103aa02 ffff88003ea0a001 0000000000000000 ffff88001f20a040 [ 0.128416] Call Trace: [ 0.128416] [<ffffffff81657c7b>] dump_stack+0x4f/0x7b [ 0.128416] [<ffffffff810bbd2c>] ? up+0x39/0x3e [ 0.128416] [<ffffffff81081510>] warn_slowpath_common+0xa1/0xbb [ 0.128416] [<ffffffff8103aa02>] ? topology_sane.isra.2+0x74/0x88 [ 0.128416] [<ffffffff81081570>] warn_slowpath_fmt+0x46/0x48 [ 0.128416] [<ffffffff8101eeb1>] ? __cpuid.constprop.0+0x15/0x19 [ 0.128416] [<ffffffff8103aa02>] topology_sane.isra.2+0x74/0x88 [ 0.128416] [<ffffffff8103ac70>] set_cpu_sibling_map+0x21a/0x444 [ 0.128416] [<ffffffff81056ac3>] ? numa_add_cpu+0x98/0x9f [ 0.128416] [<ffffffff8100b8f2>] cpu_bringup+0x63/0xa8 [ 0.128416] [<ffffffff8100b945>] cpu_bringup_and_idle+0xe/0x1a [ 0.128416] ---[ end trace 95bff1aef57ee1b1 ]--- So, basically, Linux is complaining that we're trying to put two vCPUs, that looks to be SMT siblings, on different NUMA nodes. And, yes, I think this is quite disruptive for the Linux's scheduler internal logic. The vnuma bits of the guest config are these: vnuma = [ [ "pnode=0","size=512","vcpus=0-1","vdistances=10,20" ], [ "pnode=1","size=512","vcpus=2-3","vdistances=20,10" ] ] From inside the guest, the topology looks to be like this: root@test:~# numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 node 0 size: 475 MB node 0 free: 382 MB node 1 cpus: 2 3 node 1 size: 495 MB node 1 free: 475 MB node distances: node 0 1 0: 10 10 1: 20 10 root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list 0-1 root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list 0-3 root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list 2-3 root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list 0-3 So the complain during boot seems to be against 'core_siblings' (which was not what I expected, but perhaps I misremember the meaning of "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll double check). Anyway, is there anything we can do to fix or workaround things? Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-16 10:32 PV-vNUMA issue: topology is misinterpreted by the guest Dario Faggioli @ 2015-07-16 10:47 ` Jan Beulich 2015-07-16 10:56 ` Andrew Cooper 2015-07-16 15:26 ` Wei Liu 2015-07-27 15:13 ` David Vrabel 2 siblings, 1 reply; 95+ messages in thread From: Jan Beulich @ 2015-07-16 10:47 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, xen-devel, Boris Ostrovsky >>> On 16.07.15 at 12:32, <dario.faggioli@citrix.com> wrote: > root@test:~# numactl --hardware > available: 2 nodes (0-1) > node 0 cpus: 0 1 > node 0 size: 475 MB > node 0 free: 382 MB > node 1 cpus: 2 3 > node 1 size: 495 MB > node 1 free: 475 MB > node distances: > node 0 1 > 0: 10 10 > 1: 20 10 > > root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list > 0-1 > root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list > 0-3 > root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list > 2-3 > root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list > 0-3 > > So the complain during boot seems to be against 'core_siblings' (which > was not what I expected, but perhaps I misremember the meaning of > "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll > double check). > > Anyway, is there anything we can do to fix or workaround things? Make the guest honor topology also at the CPUID layer. Whether that's by not wrongly consuming the respective CPUID bits (i.e. a guest side change) or reflecting PV state in what the hypervisor returns I'm not sure about. While the latter might be more clean, I'd be afraid this might get in the way of what the tool stack wants to see. Jan ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-16 10:47 ` Jan Beulich @ 2015-07-16 10:56 ` Andrew Cooper 2015-07-16 15:25 ` Wei Liu 0 siblings, 1 reply; 95+ messages in thread From: Andrew Cooper @ 2015-07-16 10:56 UTC (permalink / raw) To: Jan Beulich, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, David Vrabel, xen-devel, Boris Ostrovsky On 16/07/15 11:47, Jan Beulich wrote: >>>> On 16.07.15 at 12:32, <dario.faggioli@citrix.com> wrote: >> root@test:~# numactl --hardware >> available: 2 nodes (0-1) >> node 0 cpus: 0 1 >> node 0 size: 475 MB >> node 0 free: 382 MB >> node 1 cpus: 2 3 >> node 1 size: 495 MB >> node 1 free: 475 MB >> node distances: >> node 0 1 >> 0: 10 10 >> 1: 20 10 >> >> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list >> 0-1 >> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list >> 0-3 >> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list >> 2-3 >> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list >> 0-3 >> >> So the complain during boot seems to be against 'core_siblings' (which >> was not what I expected, but perhaps I misremember the meaning of >> "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll >> double check). >> >> Anyway, is there anything we can do to fix or workaround things? > Make the guest honor topology also at the CPUID layer. Whether > that's by not wrongly consuming the respective CPUID bits (i.e. a > guest side change) or reflecting PV state in what the hypervisor > returns I'm not sure about. While the latter might be more clean, > I'd be afraid this might get in the way of what the tool stack wants > to see. Xen's CPUID handling currently has no concept of per-core and per-package data in the cpuid policy. The guest sees the information from the pcpu which happened to be running libxc at the time the policy was decided, with a Xen-level fudge factor applied. There is also the regular problem of (failing to) trap cpuid instructions in PV guests, so any "numa aware" software which doesn't use the force override prefix will still fail in the above way. Lots of issues in this area are discussed in http://xenbits.xen.org/people/andrewcoop/feature-levelling/feature-levelling-E.pdf , especially the extra work section at the end. ~Andrew ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-16 10:56 ` Andrew Cooper @ 2015-07-16 15:25 ` Wei Liu 2015-07-16 15:45 ` Andrew Cooper 0 siblings, 1 reply; 95+ messages in thread From: Wei Liu @ 2015-07-16 15:25 UTC (permalink / raw) To: Andrew Cooper Cc: Elena Ufimtseva, Wei Liu, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On Thu, Jul 16, 2015 at 11:56:50AM +0100, Andrew Cooper wrote: > On 16/07/15 11:47, Jan Beulich wrote: > >>>> On 16.07.15 at 12:32, <dario.faggioli@citrix.com> wrote: > >> root@test:~# numactl --hardware > >> available: 2 nodes (0-1) > >> node 0 cpus: 0 1 > >> node 0 size: 475 MB > >> node 0 free: 382 MB > >> node 1 cpus: 2 3 > >> node 1 size: 495 MB > >> node 1 free: 475 MB > >> node distances: > >> node 0 1 > >> 0: 10 10 > >> 1: 20 10 > >> > >> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list > >> 0-1 > >> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list > >> 0-3 > >> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list > >> 2-3 > >> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list > >> 0-3 > >> > >> So the complain during boot seems to be against 'core_siblings' (which > >> was not what I expected, but perhaps I misremember the meaning of > >> "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll > >> double check). > >> > >> Anyway, is there anything we can do to fix or workaround things? > > Make the guest honor topology also at the CPUID layer. Whether > > that's by not wrongly consuming the respective CPUID bits (i.e. a > > guest side change) or reflecting PV state in what the hypervisor > > returns I'm not sure about. While the latter might be more clean, > > I'd be afraid this might get in the way of what the tool stack wants > > to see. > > Xen's CPUID handling currently has no concept of per-core and > per-package data in the cpuid policy. The guest sees the information Can / Will Xen have that concept in the future? Wei. > from the pcpu which happened to be running libxc at the time the policy > was decided, with a Xen-level fudge factor applied. > > There is also the regular problem of (failing to) trap cpuid > instructions in PV guests, so any "numa aware" software which doesn't > use the force override prefix will still fail in the above way. > > Lots of issues in this area are discussed in > http://xenbits.xen.org/people/andrewcoop/feature-levelling/feature-levelling-E.pdf > , especially the extra work section at the end. > > ~Andrew ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-16 15:25 ` Wei Liu @ 2015-07-16 15:45 ` Andrew Cooper 2015-07-16 15:50 ` Boris Ostrovsky 0 siblings, 1 reply; 95+ messages in thread From: Andrew Cooper @ 2015-07-16 15:45 UTC (permalink / raw) To: Wei Liu Cc: Elena Ufimtseva, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 16/07/15 16:25, Wei Liu wrote: > On Thu, Jul 16, 2015 at 11:56:50AM +0100, Andrew Cooper wrote: >> On 16/07/15 11:47, Jan Beulich wrote: >>>>>> On 16.07.15 at 12:32, <dario.faggioli@citrix.com> wrote: >>>> root@test:~# numactl --hardware >>>> available: 2 nodes (0-1) >>>> node 0 cpus: 0 1 >>>> node 0 size: 475 MB >>>> node 0 free: 382 MB >>>> node 1 cpus: 2 3 >>>> node 1 size: 495 MB >>>> node 1 free: 475 MB >>>> node distances: >>>> node 0 1 >>>> 0: 10 10 >>>> 1: 20 10 >>>> >>>> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list >>>> 0-1 >>>> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list >>>> 0-3 >>>> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list >>>> 2-3 >>>> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list >>>> 0-3 >>>> >>>> So the complain during boot seems to be against 'core_siblings' (which >>>> was not what I expected, but perhaps I misremember the meaning of >>>> "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll >>>> double check). >>>> >>>> Anyway, is there anything we can do to fix or workaround things? >>> Make the guest honor topology also at the CPUID layer. Whether >>> that's by not wrongly consuming the respective CPUID bits (i.e. a >>> guest side change) or reflecting PV state in what the hypervisor >>> returns I'm not sure about. While the latter might be more clean, >>> I'd be afraid this might get in the way of what the tool stack wants >>> to see. >> Xen's CPUID handling currently has no concept of per-core and >> per-package data in the cpuid policy. The guest sees the information > Can / Will Xen have that concept in the future? It is certainly possible. I plan to try and lay some ground work as part of the feature levelling fixes, but fixing the hypervisor representation of cpuid is specifically out of scope for the feature levelling fixes (in a deliberate attempt to prevent the project expanding to fill more time than I have). It is on my list of areas to tackle, but it is several nested cans of worms. ~Andrew ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-16 15:45 ` Andrew Cooper @ 2015-07-16 15:50 ` Boris Ostrovsky 2015-07-16 16:29 ` Jan Beulich 0 siblings, 1 reply; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-16 15:50 UTC (permalink / raw) To: Andrew Cooper, Wei Liu Cc: Elena Ufimtseva, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel On 07/16/2015 11:45 AM, Andrew Cooper wrote: > On 16/07/15 16:25, Wei Liu wrote: >> On Thu, Jul 16, 2015 at 11:56:50AM +0100, Andrew Cooper wrote: >>> On 16/07/15 11:47, Jan Beulich wrote: >>>>>>> On 16.07.15 at 12:32, <dario.faggioli@citrix.com> wrote: >>>>> root@test:~# numactl --hardware >>>>> available: 2 nodes (0-1) >>>>> node 0 cpus: 0 1 >>>>> node 0 size: 475 MB >>>>> node 0 free: 382 MB >>>>> node 1 cpus: 2 3 >>>>> node 1 size: 495 MB >>>>> node 1 free: 475 MB >>>>> node distances: >>>>> node 0 1 >>>>> 0: 10 10 >>>>> 1: 20 10 >>>>> >>>>> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list >>>>> 0-1 >>>>> root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list >>>>> 0-3 >>>>> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list >>>>> 2-3 >>>>> root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list >>>>> 0-3 >>>>> >>>>> So the complain during boot seems to be against 'core_siblings' (which >>>>> was not what I expected, but perhaps I misremember the meaning of >>>>> "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll >>>>> double check). >>>>> >>>>> Anyway, is there anything we can do to fix or workaround things? >>>> Make the guest honor topology also at the CPUID layer. Whether >>>> that's by not wrongly consuming the respective CPUID bits (i.e. a >>>> guest side change) or reflecting PV state in what the hypervisor >>>> returns I'm not sure about. While the latter might be more clean, >>>> I'd be afraid this might get in the way of what the tool stack wants >>>> to see. >>> Xen's CPUID handling currently has no concept of per-core and >>> per-package data in the cpuid policy. The guest sees the information >> Can / Will Xen have that concept in the future? > It is certainly possible. > > I plan to try and lay some ground work as part of the feature levelling > fixes, but fixing the hypervisor representation of cpuid is specifically > out of scope for the feature levelling fixes (in a deliberate attempt to > prevent the project expanding to fill more time than I have). > > It is on my list of areas to tackle, but it is several nested cans of worms. Can't we set leaf 1's EBX[32:16] to 1? -boris ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-16 15:50 ` Boris Ostrovsky @ 2015-07-16 16:29 ` Jan Beulich 2015-07-16 16:39 ` Andrew Cooper 0 siblings, 1 reply; 95+ messages in thread From: Jan Beulich @ 2015-07-16 16:29 UTC (permalink / raw) To: Boris Ostrovsky Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, xen-devel >>> On 16.07.15 at 17:50, <boris.ostrovsky@oracle.com> wrote: > Can't we set leaf 1's EBX[32:16] to 1? I don't think we should partially overwrite the relevant parts of CPUID output - either all or nothing (so that things at least remain consistent). Jan ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-16 16:29 ` Jan Beulich @ 2015-07-16 16:39 ` Andrew Cooper 2015-07-16 16:59 ` Boris Ostrovsky 0 siblings, 1 reply; 95+ messages in thread From: Andrew Cooper @ 2015-07-16 16:39 UTC (permalink / raw) To: Jan Beulich, Boris Ostrovsky Cc: Elena Ufimtseva, Wei Liu, Dario Faggioli, David Vrabel, xen-devel On 16/07/15 17:29, Jan Beulich wrote: >>>> On 16.07.15 at 17:50, <boris.ostrovsky@oracle.com> wrote: >> Can't we set leaf 1's EBX[32:16] to 1? > I don't think we should partially overwrite the relevant parts of > CPUID output - either all or nothing (so that things at least > remain consistent). Also, there are no masking/override MSRs for that feature leaf on any hardware I am aware of, and a PV guest using plain `cpuid` will not observe any attempt of Xen's to control the value. ~Andrew ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-16 16:39 ` Andrew Cooper @ 2015-07-16 16:59 ` Boris Ostrovsky 2015-07-17 6:09 ` Jan Beulich 2015-07-17 10:17 ` Andrew Cooper 0 siblings, 2 replies; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-16 16:59 UTC (permalink / raw) To: Andrew Cooper, Jan Beulich Cc: Elena Ufimtseva, Wei Liu, Dario Faggioli, David Vrabel, xen-devel On 07/16/2015 12:39 PM, Andrew Cooper wrote: > On 16/07/15 17:29, Jan Beulich wrote: >>>>> On 16.07.15 at 17:50, <boris.ostrovsky@oracle.com> wrote: >>> Can't we set leaf 1's EBX[32:16] to 1? (I obviously fat-fingered this --- I meant EBX[23:16]) >> I don't think we should partially overwrite the relevant parts of >> CPUID output - either all or nothing (so that things at least >> remain consistent). Possibly more, but specifically for Dario's problem I think this could resolve that. > Also, there are no masking/override MSRs for that feature leaf on any > hardware I am aware of, and a PV guest using plain `cpuid` will not > observe any attempt of Xen's to control the value. True, but we already modify other CPUID bits (e.g. we clear X86_FEATURE_TOPOEXT) so it won't make things much worse. (And again, for this specific problem CPUID is queried by guest kernel which uses PV CPUID instruction). And in general (both for PV and HVM) --- is there any reason to expose CPU topology at all? I can see it being useful if VCPUs are pinned but if they are not then it can make performance worse. -boris ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-16 16:59 ` Boris Ostrovsky @ 2015-07-17 6:09 ` Jan Beulich 2015-07-17 7:27 ` Dario Faggioli 2015-07-17 10:17 ` Andrew Cooper 1 sibling, 1 reply; 95+ messages in thread From: Jan Beulich @ 2015-07-17 6:09 UTC (permalink / raw) To: Boris Ostrovsky Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, xen-devel >>> On 16.07.15 at 18:59, <boris.ostrovsky@oracle.com> wrote: > And in general (both for PV and HVM) --- is there any reason to expose > CPU topology at all? I can see it being useful if VCPUs are pinned but > if they are not then it can make performance worse. Indeed - that's what our kernels have been doing for years, and it seems like someone over here is now looking into whether this could be done in pv-ops too (without too much uglification). Jan ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-17 6:09 ` Jan Beulich @ 2015-07-17 7:27 ` Dario Faggioli 2015-07-17 7:42 ` Jan Beulich ` (2 more replies) 0 siblings, 3 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-17 7:27 UTC (permalink / raw) To: Jan Beulich Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 1656 bytes --] On Fri, 2015-07-17 at 07:09 +0100, Jan Beulich wrote: > >>> On 16.07.15 at 18:59, <boris.ostrovsky@oracle.com> wrote: > > And in general (both for PV and HVM) --- is there any reason to expose > > CPU topology at all? I can see it being useful if VCPUs are pinned but > > if they are not then it can make performance worse. > > Indeed > Indeed indeed. :-) And in fact, this is even independent from vNUMA. Yet, I remember we were discussing about this since the beginning of vNUMA work, back when it was Elena doing it, but the it seems we all forgot... Sorry for that! :-/ I seriously think we should do something about this as, while in a non vNUMA setup it can certainly cause weird/inconsistent performance, in a vNUMA one, as shown, it's quite a hige mess. > - that's what our kernels have been doing for years, and > it seems like someone over here is now looking into whether this > could be done in pv-ops too (without too much uglification). > That would be great, IMO. I'd be up for helping with this, but I know next to nothing about CPUID, so that would require some setup time. If, at least, you could keep me in the loop it would be great. In the meanwhile, what should we do? Document this? How? "don't use vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is there a workaround we can put in place/suggest? Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-17 7:27 ` Dario Faggioli @ 2015-07-17 7:42 ` Jan Beulich 2015-07-17 8:44 ` Wei Liu 2015-07-17 18:17 ` Boris Ostrovsky 2 siblings, 0 replies; 95+ messages in thread From: Jan Beulich @ 2015-07-17 7:42 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, WeiLiu, Andrew Cooper, David Vrabel, xen-devel, Boris Ostrovsky >>> On 17.07.15 at 09:27, <dario.faggioli@citrix.com> wrote: > In the meanwhile, what should we do? Document this? How? "don't use > vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is > there a workaround we can put in place/suggest? Use SLE / openSUSE kernels ;-) ? Jan ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-17 7:27 ` Dario Faggioli 2015-07-17 7:42 ` Jan Beulich @ 2015-07-17 8:44 ` Wei Liu 2015-07-17 18:17 ` Boris Ostrovsky 2 siblings, 0 replies; 95+ messages in thread From: Wei Liu @ 2015-07-17 8:44 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On Fri, Jul 17, 2015 at 09:27:55AM +0200, Dario Faggioli wrote: > On Fri, 2015-07-17 at 07:09 +0100, Jan Beulich wrote: > > >>> On 16.07.15 at 18:59, <boris.ostrovsky@oracle.com> wrote: > > > And in general (both for PV and HVM) --- is there any reason to expose > > > CPU topology at all? I can see it being useful if VCPUs are pinned but > > > if they are not then it can make performance worse. > > > > Indeed > > > Indeed indeed. :-) > > And in fact, this is even independent from vNUMA. Yet, I remember we > were discussing about this since the beginning of vNUMA work, back when > it was Elena doing it, but the it seems we all forgot... Sorry for > that! :-/ > > I seriously think we should do something about this as, while in a non > vNUMA setup it can certainly cause weird/inconsistent performance, in a > vNUMA one, as shown, it's quite a hige mess. > > > - that's what our kernels have been doing for years, and > > it seems like someone over here is now looking into whether this > > could be done in pv-ops too (without too much uglification). > > > That would be great, IMO. I'd be up for helping with this, but I know > next to nothing about CPUID, so that would require some setup time. If, > at least, you could keep me in the loop it would be great. > > In the meanwhile, what should we do? Document this? How? "don't use > vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is > there a workaround we can put in place/suggest? > Don't worry. PV vNUMA kernel patch is not upstreamed. :-P This is one of the issues that is not upstreamed. The other issue is my own laziness... Wei. > Regards, > Dario > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-17 7:27 ` Dario Faggioli 2015-07-17 7:42 ` Jan Beulich 2015-07-17 8:44 ` Wei Liu @ 2015-07-17 18:17 ` Boris Ostrovsky 2015-07-20 14:09 ` Dario Faggioli 2 siblings, 1 reply; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-17 18:17 UTC (permalink / raw) To: Dario Faggioli, Jan Beulich Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, xen-devel On 07/17/2015 03:27 AM, Dario Faggioli wrote: > On Fri, 2015-07-17 at 07:09 +0100, Jan Beulich wrote: >>>>> On 16.07.15 at 18:59, <boris.ostrovsky@oracle.com> wrote: >>> And in general (both for PV and HVM) --- is there any reason to expose >>> CPU topology at all? I can see it being useful if VCPUs are pinned but >>> if they are not then it can make performance worse. >> Indeed >> > Indeed indeed. :-) > > And in fact, this is even independent from vNUMA. Yet, I remember we > were discussing about this since the beginning of vNUMA work, back when > it was Elena doing it, but the it seems we all forgot... Sorry for > that! :-/ > > I seriously think we should do something about this as, while in a non > vNUMA setup it can certainly cause weird/inconsistent performance, in a > vNUMA one, as shown, it's quite a hige mess. > >> - that's what our kernels have been doing for years, and >> it seems like someone over here is now looking into whether this >> could be done in pv-ops too (without too much uglification). >> > That would be great, IMO. I'd be up for helping with this, but I know > next to nothing about CPUID, so that would require some setup time. If, > at least, you could keep me in the loop it would be great. > > In the meanwhile, what should we do? Document this? How? "don't use > vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is > there a workaround we can put in place/suggest? I haven't been able to reproduce this on my Intel box because I think I have different core enumeration. Can you try adding cpuid=['0x1:ebx=xxxxxxxx00000001xxxxxxxxxxxxxxxx'] to your config file? On AMD, BTW, we fail a different test so some other bits probably need to be tweaked. You may fail it too (the LLC sanity check). -boris ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-17 18:17 ` Boris Ostrovsky @ 2015-07-20 14:09 ` Dario Faggioli 2015-07-20 14:43 ` Boris Ostrovsky 0 siblings, 1 reply; 95+ messages in thread From: Dario Faggioli @ 2015-07-20 14:09 UTC (permalink / raw) To: Boris Ostrovsky Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 3777 bytes --] On Fri, 2015-07-17 at 14:17 -0400, Boris Ostrovsky wrote: > On 07/17/2015 03:27 AM, Dario Faggioli wrote: > > In the meanwhile, what should we do? Document this? How? "don't use > > vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is > > there a workaround we can put in place/suggest? > > I haven't been able to reproduce this on my Intel box because I think I > have different core enumeration. > Yes, most likely, that's highly topology dependant. :-( > Can you try adding > cpuid=['0x1:ebx=xxxxxxxx00000001xxxxxxxxxxxxxxxx'] > to your config file? > Done (sorry for the delay, the testbox was busy doing other stuff). Still no joy (.101 is the IP address of the guest, domain id 3): root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" root@Zhaman:~# xl vcpu-list 3 Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) test 3 0 4 r-- 23.6 all / 0-7 test 3 1 9 r-- 19.8 all / 0-7 test 3 2 8 -b- 0.4 all / 8-15 test 3 3 4 -b- 0.2 all / 8-15 *HOWEVER* it seems to have an effect. In fact, now, topology as it is shown in /sys/... is different: root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list 0 (it was 0-1) This, OTOH, is still the same: root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list 0-3 Also, I now see this: [ 0.150560] ------------[ cut here ]------------ [ 0.150560] WARNING: CPU: 2 PID: 0 at ../arch/x86/kernel/smpboot.c:317 topology_sane.isra.2+0x74/0x88() [ 0.150560] sched: CPU #2's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency. [ 0.150560] Modules linked in: [ 0.150560] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.19.0+ #1 [ 0.150560] 0000000000000009 ffff88001ee2fdd0 ffffffff81657c7b ffffffff810bbd2c [ 0.150560] ffff88001ee2fe20 ffff88001ee2fe10 ffffffff81081510 ffff88001ee2fea0 [ 0.150560] ffffffff8103aa02 ffff88003ea0a001 0000000000000000 ffff88001f20a040 [ 0.150560] Call Trace: [ 0.150560] [<ffffffff81657c7b>] dump_stack+0x4f/0x7b [ 0.150560] [<ffffffff810bbd2c>] ? up+0x39/0x3e [ 0.150560] [<ffffffff81081510>] warn_slowpath_common+0xa1/0xbb [ 0.150560] [<ffffffff8103aa02>] ? topology_sane.isra.2+0x74/0x88 [ 0.150560] [<ffffffff81081570>] warn_slowpath_fmt+0x46/0x48 [ 0.150560] [<ffffffff8101eeb1>] ? __cpuid.constprop.0+0x15/0x19 [ 0.150560] [<ffffffff8103aa02>] topology_sane.isra.2+0x74/0x88 [ 0.150560] [<ffffffff8103acd0>] set_cpu_sibling_map+0x27a/0x444 [ 0.150560] [<ffffffff81056ac3>] ? numa_add_cpu+0x98/0x9f [ 0.150560] [<ffffffff8100b8f2>] cpu_bringup+0x63/0xa8 [ 0.150560] [<ffffffff8100b945>] cpu_bringup_and_idle+0xe/0x1a [ 0.150560] ---[ end trace 63d204896cce9f68 ]--- Notice that it now says 'llc-sibling', while, before, it was saying 'smt-sibling'. > On AMD, BTW, we fail a different test so some other bits probably need > to be tweaked. You may fail it too (the LLC sanity check). > Yep, that's the one I guess. Should I try something more/else? Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-20 14:09 ` Dario Faggioli @ 2015-07-20 14:43 ` Boris Ostrovsky 2015-07-21 20:00 ` Boris Ostrovsky 0 siblings, 1 reply; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-20 14:43 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/20/2015 10:09 AM, Dario Faggioli wrote: > On Fri, 2015-07-17 at 14:17 -0400, Boris Ostrovsky wrote: >> On 07/17/2015 03:27 AM, Dario Faggioli wrote: >>> In the meanwhile, what should we do? Document this? How? "don't use >>> vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is >>> there a workaround we can put in place/suggest? >> I haven't been able to reproduce this on my Intel box because I think I >> have different core enumeration. >> > Yes, most likely, that's highly topology dependant. :-( > >> Can you try adding >> cpuid=['0x1:ebx=xxxxxxxx00000001xxxxxxxxxxxxxxxx'] >> to your config file? >> > Done (sorry for the delay, the testbox was busy doing other stuff). > > Still no joy (.101 is the IP address of the guest, domain id 3): > > root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" > root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" > root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" > root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" > root@Zhaman:~# xl vcpu-list 3 > Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) > test 3 0 4 r-- 23.6 all / 0-7 > test 3 1 9 r-- 19.8 all / 0-7 > test 3 2 8 -b- 0.4 all / 8-15 > test 3 3 4 -b- 0.2 all / 8-15 > > *HOWEVER* it seems to have an effect. In fact, now, topology as it is > shown in /sys/... is different: > > root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list > 0 > (it was 0-1) > > This, OTOH, is still the same: > root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list > 0-3 > > Also, I now see this: > > [ 0.150560] ------------[ cut here ]------------ > [ 0.150560] WARNING: CPU: 2 PID: 0 at ../arch/x86/kernel/smpboot.c:317 topology_sane.isra.2+0x74/0x88() > [ 0.150560] sched: CPU #2's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency. > [ 0.150560] Modules linked in: > [ 0.150560] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.19.0+ #1 > [ 0.150560] 0000000000000009 ffff88001ee2fdd0 ffffffff81657c7b ffffffff810bbd2c > [ 0.150560] ffff88001ee2fe20 ffff88001ee2fe10 ffffffff81081510 ffff88001ee2fea0 > [ 0.150560] ffffffff8103aa02 ffff88003ea0a001 0000000000000000 ffff88001f20a040 > [ 0.150560] Call Trace: > [ 0.150560] [<ffffffff81657c7b>] dump_stack+0x4f/0x7b > [ 0.150560] [<ffffffff810bbd2c>] ? up+0x39/0x3e > [ 0.150560] [<ffffffff81081510>] warn_slowpath_common+0xa1/0xbb > [ 0.150560] [<ffffffff8103aa02>] ? topology_sane.isra.2+0x74/0x88 > [ 0.150560] [<ffffffff81081570>] warn_slowpath_fmt+0x46/0x48 > [ 0.150560] [<ffffffff8101eeb1>] ? __cpuid.constprop.0+0x15/0x19 > [ 0.150560] [<ffffffff8103aa02>] topology_sane.isra.2+0x74/0x88 > [ 0.150560] [<ffffffff8103acd0>] set_cpu_sibling_map+0x27a/0x444 > [ 0.150560] [<ffffffff81056ac3>] ? numa_add_cpu+0x98/0x9f > [ 0.150560] [<ffffffff8100b8f2>] cpu_bringup+0x63/0xa8 > [ 0.150560] [<ffffffff8100b945>] cpu_bringup_and_idle+0xe/0x1a > [ 0.150560] ---[ end trace 63d204896cce9f68 ]--- > > Notice that it now says 'llc-sibling', while, before, it was saying > 'smt-sibling'. Exactly. You are now passing the first topology test which was to see that threads are on the same node. And since each processor has only one thread (as evidenced by thread_siblings_list) we are good. The second test checks that cores (i.e. things that share last level cache) are on the same node. And they are not. > >> On AMD, BTW, we fail a different test so some other bits probably need >> to be tweaked. You may fail it too (the LLC sanity check). >> > Yep, that's the one I guess. Should I try something more/else? I'll need to see how LLC IDs are calculated, probably also from some CPUID bits. The question though will be --- what do we do with how cache sizes (and TLB sizes for that matter) are presented to the guests. Do we scale them down per thread? -boris ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-20 14:43 ` Boris Ostrovsky @ 2015-07-21 20:00 ` Boris Ostrovsky 2015-07-22 13:36 ` Dario Faggioli 0 siblings, 1 reply; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-21 20:00 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/20/2015 10:43 AM, Boris Ostrovsky wrote: > On 07/20/2015 10:09 AM, Dario Faggioli wrote: >> On Fri, 2015-07-17 at 14:17 -0400, Boris Ostrovsky wrote: >>> On 07/17/2015 03:27 AM, Dario Faggioli wrote: >>>> In the meanwhile, what should we do? Document this? How? "don't use >>>> vNUMA with PV guest in SMT enabled systems" seems a bit harsh... Is >>>> there a workaround we can put in place/suggest? >>> I haven't been able to reproduce this on my Intel box because I think I >>> have different core enumeration. >>> >> Yes, most likely, that's highly topology dependant. :-( >> >>> Can you try adding >>> cpuid=['0x1:ebx=xxxxxxxx00000001xxxxxxxxxxxxxxxx'] >>> to your config file? >>> >> Done (sorry for the delay, the testbox was busy doing other stuff). >> >> Still no joy (.101 is the IP address of the guest, domain id 3): >> >> root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" >> root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" >> root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" >> root@Zhaman:~# ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" >> root@Zhaman:~# xl vcpu-list 3 >> Name ID VCPU CPU State Time(s) >> Affinity (Hard / Soft) >> test 3 0 4 r-- 23.6 all / 0-7 >> test 3 1 9 r-- 19.8 all / 0-7 >> test 3 2 8 -b- 0.4 all / 8-15 >> test 3 3 4 -b- 0.2 all / 8-15 >> >> *HOWEVER* it seems to have an effect. In fact, now, topology as it is >> shown in /sys/... is different: >> >> root@test:~# cat >> /sys/devices/system/cpu/cpu0/topology/thread_siblings_list >> 0 >> (it was 0-1) >> >> This, OTOH, is still the same: >> root@test:~# cat >> /sys/devices/system/cpu/cpu0/topology/core_siblings_list >> 0-3 >> >> Also, I now see this: >> >> [ 0.150560] ------------[ cut here ]------------ >> [ 0.150560] WARNING: CPU: 2 PID: 0 at >> ../arch/x86/kernel/smpboot.c:317 topology_sane.isra.2+0x74/0x88() >> [ 0.150560] sched: CPU #2's llc-sibling CPU #0 is not on the same >> node! [node: 1 != 0]. Ignoring dependency. >> [ 0.150560] Modules linked in: >> [ 0.150560] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.19.0+ #1 >> [ 0.150560] 0000000000000009 ffff88001ee2fdd0 ffffffff81657c7b >> ffffffff810bbd2c >> [ 0.150560] ffff88001ee2fe20 ffff88001ee2fe10 ffffffff81081510 >> ffff88001ee2fea0 >> [ 0.150560] ffffffff8103aa02 ffff88003ea0a001 0000000000000000 >> ffff88001f20a040 >> [ 0.150560] Call Trace: >> [ 0.150560] [<ffffffff81657c7b>] dump_stack+0x4f/0x7b >> [ 0.150560] [<ffffffff810bbd2c>] ? up+0x39/0x3e >> [ 0.150560] [<ffffffff81081510>] warn_slowpath_common+0xa1/0xbb >> [ 0.150560] [<ffffffff8103aa02>] ? topology_sane.isra.2+0x74/0x88 >> [ 0.150560] [<ffffffff81081570>] warn_slowpath_fmt+0x46/0x48 >> [ 0.150560] [<ffffffff8101eeb1>] ? __cpuid.constprop.0+0x15/0x19 >> [ 0.150560] [<ffffffff8103aa02>] topology_sane.isra.2+0x74/0x88 >> [ 0.150560] [<ffffffff8103acd0>] set_cpu_sibling_map+0x27a/0x444 >> [ 0.150560] [<ffffffff81056ac3>] ? numa_add_cpu+0x98/0x9f >> [ 0.150560] [<ffffffff8100b8f2>] cpu_bringup+0x63/0xa8 >> [ 0.150560] [<ffffffff8100b945>] cpu_bringup_and_idle+0xe/0x1a >> [ 0.150560] ---[ end trace 63d204896cce9f68 ]--- >> >> Notice that it now says 'llc-sibling', while, before, it was saying >> 'smt-sibling'. > > Exactly. You are now passing the first topology test which was to see > that threads are on the same node. And since each processor has only > one thread (as evidenced by thread_siblings_list) we are good. > > The second test checks that cores (i.e. things that share last level > cache) are on the same node. And they are not. > > >> >>> On AMD, BTW, we fail a different test so some other bits probably need >>> to be tweaked. You may fail it too (the LLC sanity check). >>> >> Yep, that's the one I guess. Should I try something more/else? > > > I'll need to see how LLC IDs are calculated, probably also from some > CPUID bits. No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) which use indexes in ECX register and xl syntax doesn't allow you to override CPUIDs for such leaves. -boris > The question though will be --- what do we do with how cache sizes > (and TLB sizes for that matter) are presented to the guests. Do we > scale them down per thread? > > -boris ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-21 20:00 ` Boris Ostrovsky @ 2015-07-22 13:36 ` Dario Faggioli 2015-07-22 13:50 ` Juergen Gross 0 siblings, 1 reply; 95+ messages in thread From: Dario Faggioli @ 2015-07-22 13:36 UTC (permalink / raw) To: Boris Ostrovsky Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 1415 bytes --] On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote: > On 07/20/2015 10:43 AM, Boris Ostrovsky wrote: > > On 07/20/2015 10:09 AM, Dario Faggioli wrote: > > I'll need to see how LLC IDs are calculated, probably also from some > > CPUID bits. > > > No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) which > use indexes in ECX register and xl syntax doesn't allow you to override > CPUIDs for such leaves. > Right. Which leaves us with the question of what should we do and/or recommend users to do? If there were a workaround that we could put in place, and document somewhere, however tricky it was, I'd say to go for it, and call it acceptable for now. But, if there isn't, should we disable PV vnuma, or warn the user that he may see issues? Can we identify, in Xen or in toolstack, whether an host topology will be problematic, and disable/warn in those cases too? I'm not sure, honestly. Disabling looks too aggressive, but it's an issue I wouldn't like an user to be facing, without at least being informed of the possibility... so, perhaps a (set of) warning(s)? Thoughts? Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-22 13:36 ` Dario Faggioli @ 2015-07-22 13:50 ` Juergen Gross 2015-07-22 13:58 ` Boris Ostrovsky 0 siblings, 1 reply; 95+ messages in thread From: Juergen Gross @ 2015-07-22 13:50 UTC (permalink / raw) To: Dario Faggioli, Boris Ostrovsky Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/22/2015 03:36 PM, Dario Faggioli wrote: > On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote: >> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote: >>> On 07/20/2015 10:09 AM, Dario Faggioli wrote: > >>> I'll need to see how LLC IDs are calculated, probably also from some >>> CPUID bits. >> >> >> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) which >> use indexes in ECX register and xl syntax doesn't allow you to override >> CPUIDs for such leaves. >> > Right. Which leaves us with the question of what should we do and/or > recommend users to do? > > If there were a workaround that we could put in place, and document > somewhere, however tricky it was, I'd say to go for it, and call it > acceptable for now. > > But, if there isn't, should we disable PV vnuma, or warn the user that > he may see issues? Can we identify, in Xen or in toolstack, whether an > host topology will be problematic, and disable/warn in those cases too? > > I'm not sure, honestly. Disabling looks too aggressive, but it's an > issue I wouldn't like an user to be facing, without at least being > informed of the possibility... so, perhaps a (set of) warning(s)? > Thoughts? I think we have 2 possible solutions: 1. Try to handle this all in the hypervisor via CPUID mangling. 2. Add PV-topology support to the guest and indicate this capability via elfnote; only enable PV-numa if this note is present. I'd prefer the second solution. If you are okay with this, I'd try to do some patches for the pvops kernel. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-22 13:50 ` Juergen Gross @ 2015-07-22 13:58 ` Boris Ostrovsky 2015-07-22 14:09 ` Juergen Gross 0 siblings, 1 reply; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-22 13:58 UTC (permalink / raw) To: Juergen Gross, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/22/2015 09:50 AM, Juergen Gross wrote: > On 07/22/2015 03:36 PM, Dario Faggioli wrote: >> On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote: >>> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote: >>>> On 07/20/2015 10:09 AM, Dario Faggioli wrote: >> >>>> I'll need to see how LLC IDs are calculated, probably also from some >>>> CPUID bits. >>> >>> >>> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) which >>> use indexes in ECX register and xl syntax doesn't allow you to override >>> CPUIDs for such leaves. >>> >> Right. Which leaves us with the question of what should we do and/or >> recommend users to do? >> >> If there were a workaround that we could put in place, and document >> somewhere, however tricky it was, I'd say to go for it, and call it >> acceptable for now. >> >> But, if there isn't, should we disable PV vnuma, or warn the user that >> he may see issues? Can we identify, in Xen or in toolstack, whether an >> host topology will be problematic, and disable/warn in those cases too? >> >> I'm not sure, honestly. Disabling looks too aggressive, but it's an >> issue I wouldn't like an user to be facing, without at least being >> informed of the possibility... so, perhaps a (set of) warning(s)? >> Thoughts? > > I think we have 2 possible solutions: > > 1. Try to handle this all in the hypervisor via CPUID mangling. > > 2. Add PV-topology support to the guest and indicate this capability via > elfnote; only enable PV-numa if this note is present. > > I'd prefer the second solution. If you are okay with this, I'd try to do > some patches for the pvops kernel. What if I configure a guest to follow HW topology? I.e. I pin VCPUs to appropriate cores/threads? With elfnote I am stuck with disabled topology. Besides, this is not necessarily a NUMA-only issue, it's a scheduling one (inside the guest) as well. -boris ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-22 13:58 ` Boris Ostrovsky @ 2015-07-22 14:09 ` Juergen Gross 2015-07-22 14:44 ` Boris Ostrovsky 2015-07-22 14:50 ` Dario Faggioli 0 siblings, 2 replies; 95+ messages in thread From: Juergen Gross @ 2015-07-22 14:09 UTC (permalink / raw) To: Boris Ostrovsky, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/22/2015 03:58 PM, Boris Ostrovsky wrote: > On 07/22/2015 09:50 AM, Juergen Gross wrote: >> On 07/22/2015 03:36 PM, Dario Faggioli wrote: >>> On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote: >>>> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote: >>>>> On 07/20/2015 10:09 AM, Dario Faggioli wrote: >>> >>>>> I'll need to see how LLC IDs are calculated, probably also from some >>>>> CPUID bits. >>>> >>>> >>>> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) which >>>> use indexes in ECX register and xl syntax doesn't allow you to override >>>> CPUIDs for such leaves. >>>> >>> Right. Which leaves us with the question of what should we do and/or >>> recommend users to do? >>> >>> If there were a workaround that we could put in place, and document >>> somewhere, however tricky it was, I'd say to go for it, and call it >>> acceptable for now. >>> >>> But, if there isn't, should we disable PV vnuma, or warn the user that >>> he may see issues? Can we identify, in Xen or in toolstack, whether an >>> host topology will be problematic, and disable/warn in those cases too? >>> >>> I'm not sure, honestly. Disabling looks too aggressive, but it's an >>> issue I wouldn't like an user to be facing, without at least being >>> informed of the possibility... so, perhaps a (set of) warning(s)? >>> Thoughts? >> >> I think we have 2 possible solutions: >> >> 1. Try to handle this all in the hypervisor via CPUID mangling. >> >> 2. Add PV-topology support to the guest and indicate this capability via >> elfnote; only enable PV-numa if this note is present. >> >> I'd prefer the second solution. If you are okay with this, I'd try to do >> some patches for the pvops kernel. > > What if I configure a guest to follow HW topology? I.e. I pin VCPUs to > appropriate cores/threads? With elfnote I am stuck with disabled topology. Add an option to do exactly that: follow HW topology (pin vcpus, configure vnuma)? Add a force flag to the vnuma configuration to ignore the elfnote? > Besides, this is not necessarily a NUMA-only issue, it's a scheduling > one (inside the guest) as well. Sure. That's what Jan said regarding SUSE's xen-kernel. No toplogy info (or a trivial one) might be better than the wrong one... This patch for pvops should be written in any case. I'll do this, but it would be nice to know whether PV-numa should be considered or not. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-22 14:09 ` Juergen Gross @ 2015-07-22 14:44 ` Boris Ostrovsky 2015-07-23 4:43 ` Juergen Gross 2015-07-22 14:50 ` Dario Faggioli 1 sibling, 1 reply; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-22 14:44 UTC (permalink / raw) To: Juergen Gross, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/22/2015 10:09 AM, Juergen Gross wrote: > On 07/22/2015 03:58 PM, Boris Ostrovsky wrote: >> On 07/22/2015 09:50 AM, Juergen Gross wrote: >>> On 07/22/2015 03:36 PM, Dario Faggioli wrote: >>>> On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote: >>>>> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote: >>>>>> On 07/20/2015 10:09 AM, Dario Faggioli wrote: >>>> >>>>>> I'll need to see how LLC IDs are calculated, probably also from some >>>>>> CPUID bits. >>>>> >>>>> >>>>> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) >>>>> which >>>>> use indexes in ECX register and xl syntax doesn't allow you to >>>>> override >>>>> CPUIDs for such leaves. >>>>> >>>> Right. Which leaves us with the question of what should we do and/or >>>> recommend users to do? >>>> >>>> If there were a workaround that we could put in place, and document >>>> somewhere, however tricky it was, I'd say to go for it, and call it >>>> acceptable for now. >>>> >>>> But, if there isn't, should we disable PV vnuma, or warn the user that >>>> he may see issues? Can we identify, in Xen or in toolstack, whether an >>>> host topology will be problematic, and disable/warn in those cases >>>> too? >>>> >>>> I'm not sure, honestly. Disabling looks too aggressive, but it's an >>>> issue I wouldn't like an user to be facing, without at least being >>>> informed of the possibility... so, perhaps a (set of) warning(s)? >>>> Thoughts? >>> >>> I think we have 2 possible solutions: >>> >>> 1. Try to handle this all in the hypervisor via CPUID mangling. >>> >>> 2. Add PV-topology support to the guest and indicate this capability >>> via >>> elfnote; only enable PV-numa if this note is present. >>> >>> I'd prefer the second solution. If you are okay with this, I'd try >>> to do >>> some patches for the pvops kernel. Why do you think that kernel patches are preferable to CPUID management? This would be all in tools, I'd think. (Well, one problem that I can think of is that AMD sometimes pokes at MSRs and/or Northbridge's PCI registers to figure out nodeID --- that we may need to have to address in the hypervisor) And those patches won't help HVM guests, will they? How would they be useful by user processes? -boris >> >> What if I configure a guest to follow HW topology? I.e. I pin VCPUs to >> appropriate cores/threads? With elfnote I am stuck with disabled >> topology. > > Add an option to do exactly that: follow HW topology (pin vcpus, > configure vnuma)? > > Add a force flag to the vnuma configuration to ignore the elfnote? > >> Besides, this is not necessarily a NUMA-only issue, it's a scheduling >> one (inside the guest) as well. > > Sure. That's what Jan said regarding SUSE's xen-kernel. No toplogy info > (or a trivial one) might be better than the wrong one... > > This patch for pvops should be written in any case. I'll do this, but it > would be nice to know whether PV-numa should be considered or not. ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-22 14:44 ` Boris Ostrovsky @ 2015-07-23 4:43 ` Juergen Gross 2015-07-23 7:28 ` Jan Beulich ` (2 more replies) 0 siblings, 3 replies; 95+ messages in thread From: Juergen Gross @ 2015-07-23 4:43 UTC (permalink / raw) To: Boris Ostrovsky, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/22/2015 04:44 PM, Boris Ostrovsky wrote: > On 07/22/2015 10:09 AM, Juergen Gross wrote: >> On 07/22/2015 03:58 PM, Boris Ostrovsky wrote: >>> On 07/22/2015 09:50 AM, Juergen Gross wrote: >>>> On 07/22/2015 03:36 PM, Dario Faggioli wrote: >>>>> On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote: >>>>>> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote: >>>>>>> On 07/20/2015 10:09 AM, Dario Faggioli wrote: >>>>> >>>>>>> I'll need to see how LLC IDs are calculated, probably also from some >>>>>>> CPUID bits. >>>>>> >>>>>> >>>>>> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) >>>>>> which >>>>>> use indexes in ECX register and xl syntax doesn't allow you to >>>>>> override >>>>>> CPUIDs for such leaves. >>>>>> >>>>> Right. Which leaves us with the question of what should we do and/or >>>>> recommend users to do? >>>>> >>>>> If there were a workaround that we could put in place, and document >>>>> somewhere, however tricky it was, I'd say to go for it, and call it >>>>> acceptable for now. >>>>> >>>>> But, if there isn't, should we disable PV vnuma, or warn the user that >>>>> he may see issues? Can we identify, in Xen or in toolstack, whether an >>>>> host topology will be problematic, and disable/warn in those cases >>>>> too? >>>>> >>>>> I'm not sure, honestly. Disabling looks too aggressive, but it's an >>>>> issue I wouldn't like an user to be facing, without at least being >>>>> informed of the possibility... so, perhaps a (set of) warning(s)? >>>>> Thoughts? >>>> >>>> I think we have 2 possible solutions: >>>> >>>> 1. Try to handle this all in the hypervisor via CPUID mangling. >>>> >>>> 2. Add PV-topology support to the guest and indicate this capability >>>> via >>>> elfnote; only enable PV-numa if this note is present. >>>> >>>> I'd prefer the second solution. If you are okay with this, I'd try >>>> to do >>>> some patches for the pvops kernel. > > Why do you think that kernel patches are preferable to CPUID management? > This would be all in tools, I'd think. (Well, one problem that I can > think of is that AMD sometimes pokes at MSRs and/or Northbridge's PCI > registers to figure out nodeID --- that we may need to have to address > in the hypervisor) Doing it via CPUID is more HW specific. Trying to fake a topology for the guest from outside might lead to weird decisions in the guest e.g. regarding licenses based on socket counts. If you are doing it in the guest itself you are able to address the different problems (scheduling, licensing) in different ways. > And those patches won't help HVM guests, will they? How would they be > useful by user processes? HVM can use pv interfaces as well. It's called pv-NUMA :-) Hmm, I didn't think of user processes. Are you aware of cases where they are to be considered? The only case where user processes are involved I could think of is licensing again. Depending on the licensing model playing with CPUID is either good or bad. I can even imagine the CPUID configuration capabilities in xl are in use today for exactly this purpose. Using them for pv-NUMA as well will make this feature unusable for those users. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-23 4:43 ` Juergen Gross @ 2015-07-23 7:28 ` Jan Beulich 2015-07-23 9:42 ` Andrew Cooper 2015-07-23 14:07 ` Dario Faggioli 2 siblings, 0 replies; 95+ messages in thread From: Jan Beulich @ 2015-07-23 7:28 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, xen-devel, Boris Ostrovsky >>> On 23.07.15 at 06:43, <JGross@suse.com> wrote: > Hmm, I didn't think of user processes. Are you aware of cases where they > are to be considered? Why wouldn't a sophisticated user mode program attempt to adjust certain memory objects' sizes based on cache size? Jan ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-23 4:43 ` Juergen Gross 2015-07-23 7:28 ` Jan Beulich @ 2015-07-23 9:42 ` Andrew Cooper 2015-07-23 14:07 ` Dario Faggioli 2 siblings, 0 replies; 95+ messages in thread From: Andrew Cooper @ 2015-07-23 9:42 UTC (permalink / raw) To: Juergen Gross, Boris Ostrovsky, Dario Faggioli Cc: Elena Ufimtseva, xen-devel, Wei Liu, David Vrabel, Jan Beulich On 23/07/15 05:43, Juergen Gross wrote: > On 07/22/2015 04:44 PM, Boris Ostrovsky wrote: >> On 07/22/2015 10:09 AM, Juergen Gross wrote: >>> On 07/22/2015 03:58 PM, Boris Ostrovsky wrote: >>>> On 07/22/2015 09:50 AM, Juergen Gross wrote: >>>>> On 07/22/2015 03:36 PM, Dario Faggioli wrote: >>>>>> On Tue, 2015-07-21 at 16:00 -0400, Boris Ostrovsky wrote: >>>>>>> On 07/20/2015 10:43 AM, Boris Ostrovsky wrote: >>>>>>>> On 07/20/2015 10:09 AM, Dario Faggioli wrote: >>>>>> >>>>>>>> I'll need to see how LLC IDs are calculated, probably also from >>>>>>>> some >>>>>>>> CPUID bits. >>>>>>> >>>>>>> >>>>>>> No, can't do this: LLC is calculated from CPUID leaf 4 (on Intel) >>>>>>> which >>>>>>> use indexes in ECX register and xl syntax doesn't allow you to >>>>>>> override >>>>>>> CPUIDs for such leaves. >>>>>>> >>>>>> Right. Which leaves us with the question of what should we do and/or >>>>>> recommend users to do? >>>>>> >>>>>> If there were a workaround that we could put in place, and document >>>>>> somewhere, however tricky it was, I'd say to go for it, and call it >>>>>> acceptable for now. >>>>>> >>>>>> But, if there isn't, should we disable PV vnuma, or warn the user >>>>>> that >>>>>> he may see issues? Can we identify, in Xen or in toolstack, >>>>>> whether an >>>>>> host topology will be problematic, and disable/warn in those cases >>>>>> too? >>>>>> >>>>>> I'm not sure, honestly. Disabling looks too aggressive, but it's an >>>>>> issue I wouldn't like an user to be facing, without at least being >>>>>> informed of the possibility... so, perhaps a (set of) warning(s)? >>>>>> Thoughts? >>>>> >>>>> I think we have 2 possible solutions: >>>>> >>>>> 1. Try to handle this all in the hypervisor via CPUID mangling. >>>>> >>>>> 2. Add PV-topology support to the guest and indicate this capability >>>>> via >>>>> elfnote; only enable PV-numa if this note is present. >>>>> >>>>> I'd prefer the second solution. If you are okay with this, I'd try >>>>> to do >>>>> some patches for the pvops kernel. >> >> Why do you think that kernel patches are preferable to CPUID management? >> This would be all in tools, I'd think. (Well, one problem that I can >> think of is that AMD sometimes pokes at MSRs and/or Northbridge's PCI >> registers to figure out nodeID --- that we may need to have to address >> in the hypervisor) > > Doing it via CPUID is more HW specific. Trying to fake a topology for > the guest from outside might lead to weird decisions in the guest e.g. > regarding licenses based on socket counts. > > If you are doing it in the guest itself you are able to address the > different problems (scheduling, licensing) in different ways. > >> And those patches won't help HVM guests, will they? How would they be >> useful by user processes? > > HVM can use pv interfaces as well. It's called pv-NUMA :-) > > Hmm, I didn't think of user processes. Are you aware of cases where they > are to be considered? The only case where user processes are involved I > could think of is licensing again. Depending on the licensing model > playing with CPUID is either good or bad. I can even imagine the CPUID > configuration capabilities in xl are in use today for exactly this > purpose. Using them for pv-NUMA as well will make this feature unusable > for those users. Userspace can also use things like hwloc which use cpuid to calculate efficient allocation of resources, rather than for licensing purposes. ~Andrew ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-23 4:43 ` Juergen Gross 2015-07-23 7:28 ` Jan Beulich 2015-07-23 9:42 ` Andrew Cooper @ 2015-07-23 14:07 ` Dario Faggioli 2015-07-23 14:13 ` Juergen Gross 2015-07-24 10:28 ` Juergen Gross 2 siblings, 2 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-23 14:07 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 3473 bytes --] On Thu, 2015-07-23 at 06:43 +0200, Juergen Gross wrote: > On 07/22/2015 04:44 PM, Boris Ostrovsky wrote: > > On 07/22/2015 10:09 AM, Juergen Gross wrote: > >>>> I think we have 2 possible solutions: > >>>> > >>>> 1. Try to handle this all in the hypervisor via CPUID mangling. > >>>> > >>>> 2. Add PV-topology support to the guest and indicate this capability > >>>> via > >>>> elfnote; only enable PV-numa if this note is present. > >>>> > >>>> I'd prefer the second solution. If you are okay with this, I'd try > >>>> to do > >>>> some patches for the pvops kernel. > > > > Why do you think that kernel patches are preferable to CPUID management? > > This would be all in tools, I'd think. (Well, one problem that I can > > think of is that AMD sometimes pokes at MSRs and/or Northbridge's PCI > > registers to figure out nodeID --- that we may need to have to address > > in the hypervisor) > > Doing it via CPUID is more HW specific. Trying to fake a topology for > the guest from outside might lead to weird decisions in the guest e.g. > regarding licenses based on socket counts. > I do see the value of this, I think... > If you are doing it in the guest itself you are able to address the > different problems (scheduling, licensing) in different ways. > ... but, at least in the case of vNUMA for instance, there still need to be a correlation between the vNUMA topology, and the "CPUID topology", and vNUMA topology is decided by toolstack. Then, if you mean, within all the possible solutions that matches (i.e., that does not cause problems to!) the vNUMA setup we've been given, let's pick up one that also is best for this xxx other purpose, then I agree. What I'm not sure I see, although, is how you would be specifying the other purpose, e.g., in this case, are you thinking to another parameter saying that we want to minimize the socket count? > Depending on the licensing model > playing with CPUID is either good or bad. I can even imagine the CPUID > configuration capabilities in xl are in use today for exactly this > purpose. Using them for pv-NUMA as well will make this feature unusable > for those users. > Yeah, well... So, you want a VM with only one socket, because of whatever reasons (say licensing), and you're using libxl's CPUID fiddling capability to do that. Now, if you specify, for such a VM, a vNUMA layout with 2 vnodes, well, I'd call this asking for troubles. I know, strictly speaking, socket != (v)NUMA node. Still, I think this will be a corner case, way less common than just a user specifying a vNUMA topology, and getting only a fraction of the vcpus being used/usable! :-/ In summary, I probably know too few of CPUID handling to have a clear view on whether something like 'making it match the topology' --which also means, if no vNUMA, CPUID should say flat, for some definition of flat-- should leave in tools or in kernel... I just know that we need to do something *consistent*. FWIW, I was thinking that the kernel were a better place, as Juergen is saying, while now I'm more convinced that tools would be more appropriate, as Boris is saying. Thanks and Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-23 14:07 ` Dario Faggioli @ 2015-07-23 14:13 ` Juergen Gross 2015-07-24 10:28 ` Juergen Gross 1 sibling, 0 replies; 95+ messages in thread From: Juergen Gross @ 2015-07-23 14:13 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/23/2015 04:07 PM, Dario Faggioli wrote: > On Thu, 2015-07-23 at 06:43 +0200, Juergen Gross wrote: >> On 07/22/2015 04:44 PM, Boris Ostrovsky wrote: >>> On 07/22/2015 10:09 AM, Juergen Gross wrote: > >>>>>> I think we have 2 possible solutions: >>>>>> >>>>>> 1. Try to handle this all in the hypervisor via CPUID mangling. >>>>>> >>>>>> 2. Add PV-topology support to the guest and indicate this capability >>>>>> via >>>>>> elfnote; only enable PV-numa if this note is present. >>>>>> >>>>>> I'd prefer the second solution. If you are okay with this, I'd try >>>>>> to do >>>>>> some patches for the pvops kernel. >>> >>> Why do you think that kernel patches are preferable to CPUID management? >>> This would be all in tools, I'd think. (Well, one problem that I can >>> think of is that AMD sometimes pokes at MSRs and/or Northbridge's PCI >>> registers to figure out nodeID --- that we may need to have to address >>> in the hypervisor) >> >> Doing it via CPUID is more HW specific. Trying to fake a topology for >> the guest from outside might lead to weird decisions in the guest e.g. >> regarding licenses based on socket counts. >> > I do see the value of this, I think... > >> If you are doing it in the guest itself you are able to address the >> different problems (scheduling, licensing) in different ways. >> > ... but, at least in the case of vNUMA for instance, there still need to > be a correlation between the vNUMA topology, and the "CPUID topology", > and vNUMA topology is decided by toolstack. > > Then, if you mean, within all the possible solutions that matches (i.e., > that does not cause problems to!) the vNUMA setup we've been given, > let's pick up one that also is best for this xxx other purpose, then I > agree. > > What I'm not sure I see, although, is how you would be specifying the > other purpose, e.g., in this case, are you thinking to another parameter > saying that we want to minimize the socket count? > >> Depending on the licensing model >> playing with CPUID is either good or bad. I can even imagine the CPUID >> configuration capabilities in xl are in use today for exactly this >> purpose. Using them for pv-NUMA as well will make this feature unusable >> for those users. >> > Yeah, well... So, you want a VM with only one socket, because of > whatever reasons (say licensing), and you're using libxl's CPUID > fiddling capability to do that. Now, if you specify, for such a VM, a > vNUMA layout with 2 vnodes, well, I'd call this asking for troubles. I > know, strictly speaking, socket != (v)NUMA node. Still, I think this > will be a corner case, way less common than just a user specifying a > vNUMA topology, and getting only a fraction of the vcpus being > used/usable! :-/ > > In summary, I probably know too few of CPUID handling to have a clear > view on whether something like 'making it match the topology' --which > also means, if no vNUMA, CPUID should say flat, for some definition of > flat-- should leave in tools or in kernel... I just know that we need to > do something *consistent*. > > FWIW, I was thinking that the kernel were a better place, as Juergen is > saying, while now I'm more convinced that tools would be more > appropriate, as Boris is saying. I'm just about to write down all sources of information the linux kernel is using to build topology information and for what purpose this data is being used. This should help to decide which way to go. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-23 14:07 ` Dario Faggioli 2015-07-23 14:13 ` Juergen Gross @ 2015-07-24 10:28 ` Juergen Gross 2015-07-24 14:44 ` Dario Faggioli 1 sibling, 1 reply; 95+ messages in thread From: Juergen Gross @ 2015-07-24 10:28 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/23/2015 04:07 PM, Dario Faggioli wrote: > On Thu, 2015-07-23 at 06:43 +0200, Juergen Gross wrote: >> On 07/22/2015 04:44 PM, Boris Ostrovsky wrote: >>> On 07/22/2015 10:09 AM, Juergen Gross wrote: > >>>>>> I think we have 2 possible solutions: >>>>>> >>>>>> 1. Try to handle this all in the hypervisor via CPUID mangling. >>>>>> >>>>>> 2. Add PV-topology support to the guest and indicate this capability >>>>>> via >>>>>> elfnote; only enable PV-numa if this note is present. >>>>>> >>>>>> I'd prefer the second solution. If you are okay with this, I'd try >>>>>> to do >>>>>> some patches for the pvops kernel. >>> >>> Why do you think that kernel patches are preferable to CPUID management? >>> This would be all in tools, I'd think. (Well, one problem that I can >>> think of is that AMD sometimes pokes at MSRs and/or Northbridge's PCI >>> registers to figure out nodeID --- that we may need to have to address >>> in the hypervisor) >> >> Doing it via CPUID is more HW specific. Trying to fake a topology for >> the guest from outside might lead to weird decisions in the guest e.g. >> regarding licenses based on socket counts. >> > I do see the value of this, I think... > >> If you are doing it in the guest itself you are able to address the >> different problems (scheduling, licensing) in different ways. >> > ... but, at least in the case of vNUMA for instance, there still need to > be a correlation between the vNUMA topology, and the "CPUID topology", > and vNUMA topology is decided by toolstack. > > Then, if you mean, within all the possible solutions that matches (i.e., > that does not cause problems to!) the vNUMA setup we've been given, > let's pick up one that also is best for this xxx other purpose, then I > agree. > > What I'm not sure I see, although, is how you would be specifying the > other purpose, e.g., in this case, are you thinking to another parameter > saying that we want to minimize the socket count? > >> Depending on the licensing model >> playing with CPUID is either good or bad. I can even imagine the CPUID >> configuration capabilities in xl are in use today for exactly this >> purpose. Using them for pv-NUMA as well will make this feature unusable >> for those users. >> > Yeah, well... So, you want a VM with only one socket, because of > whatever reasons (say licensing), and you're using libxl's CPUID > fiddling capability to do that. Now, if you specify, for such a VM, a > vNUMA layout with 2 vnodes, well, I'd call this asking for troubles. I > know, strictly speaking, socket != (v)NUMA node. Still, I think this > will be a corner case, way less common than just a user specifying a > vNUMA topology, and getting only a fraction of the vcpus being > used/usable! :-/ > > In summary, I probably know too few of CPUID handling to have a clear > view on whether something like 'making it match the topology' --which > also means, if no vNUMA, CPUID should say flat, for some definition of > flat-- should leave in tools or in kernel... I just know that we need to > do something *consistent*. > > FWIW, I was thinking that the kernel were a better place, as Juergen is > saying, while now I'm more convinced that tools would be more > appropriate, as Boris is saying. I've collected some information from the linux kernel sources as a base for the discussion: The complete numa information (cpu->node and memory->node relations) is taken from the acpi tables (srat, slit for "distances"). The topology information is obtained via: - intel: + cpuid leaf b with subleafs, leaf 4 + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available - amd: + cpuid leaf 8000001e, leaf 8000001d, leaf 4 + msr c001100c + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available The scheduler is aware of: - smt siblings (from topology) - last-level-cache siblings (from topology) - node siblings (from numa information) It will especially move tasks from one cpu to another first between smt siblings, second between llc siblings, third between node siblings and last all cpus. Memory management does numa node aware memory allocation. Topology and numa information are made available through /sys and /proc filesystems. cpuid instruction is available for user mode as well. Anything I have missed? Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 10:28 ` Juergen Gross @ 2015-07-24 14:44 ` Dario Faggioli 2015-07-24 15:14 ` Juergen Gross 2015-07-24 20:27 ` Elena Ufimtseva 0 siblings, 2 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-24 14:44 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 4588 bytes --] On Fri, 2015-07-24 at 12:28 +0200, Juergen Gross wrote: > On 07/23/2015 04:07 PM, Dario Faggioli wrote: > > FWIW, I was thinking that the kernel were a better place, as Juergen is > > saying, while now I'm more convinced that tools would be more > > appropriate, as Boris is saying. > > I've collected some information from the linux kernel sources as a base > for the discussion: > That's great, thanks for this! > The complete numa information (cpu->node and memory->node relations) is > taken from the acpi tables (srat, slit for "distances"). > Ok. And I already have a question (as I lost track of things a bit). What you just said about ACPI tables is certainly true for baremetal and HVM guests, but for PV? At the time I was looking into it, together with Elena, there were Linux patches being produced for the PV case, which makes sense. However, ISTR that both Wei and Elena mentioned recently that those patches have not been upstreamed in Linux yet... Is that the case? Maybe not all, but at least some of them are there? Because if not, I'm not sure I see how a PV guest would even see a vNUMA topology (which it does). Of course, I can go and check, but since you just looked, you may have it fresh and clear already. :-) > The topology information is obtained via: > - intel: > + cpuid leaf b with subleafs, leaf 4 > + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available > - amd: > + cpuid leaf 8000001e, leaf 8000001d, leaf 4 > + msr c001100c > + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available > > The scheduler is aware of: > - smt siblings (from topology) > - last-level-cache siblings (from topology) > - node siblings (from numa information) > Right. So, this confirms what we were guessing: we need to "reconcile" these two sources of information (from the guest point of view). Both the 'in kernel' and 'in toolstack' approach should have all the necessary information to make things match, I think. In fact, in toolstack, we know what the vNUMA topology is (we're parsing and actually putting it in place!). In kernel, we know it as we read it from tables or hypercalls (isn't that so, for PV guest?). In fact, I think that it is the topology, i.e., what comes from MSRs, that needs to adapt, and follow vNUMA, as much as possible. Do we agree on this? IMO, the thing boils down to these: 1) from where (kernel vs. toolstack) is it the most easy and effective to enact the CPUID fiddling? As in, can we do that in toolstack? (Andrew was not so sure, and Boris found issues, although Jan seems to think they're no show stopper.) I'm quite certain that we can do that from inside the kernel, although, how early would we need to be doing it? Do we have the vNUMA info already? 2) when tweaking the values of CPUID and other MSRs, are there other vNUMA (and topology in general) constraints and requirements we should take into account? For instance, do we want, for licensing reasons, all (or most) of the vcpus to be siblings, rather than full sockets? Etc. 2a) if yes, how and where are these constraints specified? If looking at 1) only, it still looks to me that doing things within the kernel would be the way to go. When looking at 2), OTOH, toolstacks variants start to be more appealing. Especially depending on our answer to 2a). In fact, in case we want to give the user a way to specify this siblings-vs-cores-vs-sockets information, it IMO would be good to deal with that in tools, rather than having to involve Xen or Linux! > It will especially move tasks from one cpu to another first between smt > siblings, second between llc siblings, third between node siblings and > last all cpus. > Yep, this part, I knew. Maybe, there is room for "fixing" this at this level, hooking up inside the scheduler code... but I'm shooting in the dark, without having check whether and how this could be really feasible, should I? One thing I don't like about this approach is that it would potentially solve vNUMA and other scheduling anomalies, but... > cpuid instruction is available for user mode as well. > ...it would not do any good for other subsystems, and user level code and apps. Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 14:44 ` Dario Faggioli @ 2015-07-24 15:14 ` Juergen Gross 2015-07-24 15:24 ` Juergen Gross 2015-07-24 16:05 ` Dario Faggioli 2015-07-24 20:27 ` Elena Ufimtseva 1 sibling, 2 replies; 95+ messages in thread From: Juergen Gross @ 2015-07-24 15:14 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/24/2015 04:44 PM, Dario Faggioli wrote: > On Fri, 2015-07-24 at 12:28 +0200, Juergen Gross wrote: >> On 07/23/2015 04:07 PM, Dario Faggioli wrote: > >>> FWIW, I was thinking that the kernel were a better place, as Juergen is >>> saying, while now I'm more convinced that tools would be more >>> appropriate, as Boris is saying. >> >> I've collected some information from the linux kernel sources as a base >> for the discussion: >> > That's great, thanks for this! > >> The complete numa information (cpu->node and memory->node relations) is >> taken from the acpi tables (srat, slit for "distances"). >> > Ok. And I already have a question (as I lost track of things a bit). > What you just said about ACPI tables is certainly true for baremetal and > HVM guests, but for PV? At the time I was looking into it, together with > Elena, there were Linux patches being produced for the PV case, which > makes sense. > However, ISTR that both Wei and Elena mentioned recently that those > patches have not been upstreamed in Linux yet... Is that the case? Maybe > not all, but at least some of them are there? Because if not, I'm not > sure I see how a PV guest would even see a vNUMA topology (which it > does). > > Of course, I can go and check, but since you just looked, you may have > it fresh and clear already. :-) I checked "bottom up", so when I found the acpi scan stuff I stopped searching how the kernel obtains numa info. During my search I found no clue of an pv-numa stuff in the kernel. And a quick "grep -i numa" in arch/x86/xen and drivers/xen didn't reveal anything. Same for a complete kernel source search for "vnuma". > >> The topology information is obtained via: >> - intel: >> + cpuid leaf b with subleafs, leaf 4 >> + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available >> - amd: >> + cpuid leaf 8000001e, leaf 8000001d, leaf 4 >> + msr c001100c >> + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available >> >> The scheduler is aware of: >> - smt siblings (from topology) >> - last-level-cache siblings (from topology) >> - node siblings (from numa information) >> > Right. So, this confirms what we were guessing: we need to "reconcile" > these two sources of information (from the guest point of view). > > Both the 'in kernel' and 'in toolstack' approach should have all the > necessary information to make things match, I think. In fact, in > toolstack, we know what the vNUMA topology is (we're parsing and > actually putting it in place!). In kernel, we know it as we read it from > tables or hypercalls (isn't that so, for PV guest?). > > In fact, I think that it is the topology, i.e., what comes from MSRs, > that needs to adapt, and follow vNUMA, as much as possible. Do we agree > on this? I think we have to be very careful here. I see two possible scenarios: 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will try to schedule the vcpus according to their numa affinity. So they can change pcpus at any time in case of very busy guests. I don't think the linux kernel should treat the cpus differently in this case as it will be in vane regarding the Xen scheduler's activity. So we should use the "null" topology in this case. 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen scheduler can't move vcpus between pcpus, so the linux kernel should see the real topology of the used pcpus in order to optimize for this picture. This only covers the scheduling aspect, of course. > > IMO, the thing boils down to these: > > 1) from where (kernel vs. toolstack) is it the most easy and effective > to enact the CPUID fiddling? As in, can we do that in toolstack? > (Andrew was not so sure, and Boris found issues, although Jan seems > to think they're no show stopper.) > I'm quite certain that we can do that from inside the kernel, > although, how early would we need to be doing it? Do we have the > vNUMA info already? > > 2) when tweaking the values of CPUID and other MSRs, are there other > vNUMA (and topology in general) constraints and requirements we > should take into account? For instance, do we want, for licensing > reasons, all (or most) of the vcpus to be siblings, rather than full > sockets? Etc. > 2a) if yes, how and where are these constraints specified? > > If looking at 1) only, it still looks to me that doing things within the > kernel would be the way to go. > > When looking at 2), OTOH, toolstacks variants start to be more > appealing. Especially depending on our answer to 2a). In fact, > in case we want to give the user a way to specify this > siblings-vs-cores-vs-sockets information, it IMO would be good to deal > with that in tools, rather than having to involve Xen or Linux! > >> It will especially move tasks from one cpu to another first between smt >> siblings, second between llc siblings, third between node siblings and >> last all cpus. >> > Yep, this part, I knew. > > Maybe, there is room for "fixing" this at this level, hooking up inside > the scheduler code... but I'm shooting in the dark, without having check > whether and how this could be really feasible, should I? Uuh, I don't think a change of the scheduler on behalf of Xen is really appreciated. :-) I'd rather fiddle with the cpu masks on the different levels to let the scheduler do the right thing. > One thing I don't like about this approach is that it would potentially > solve vNUMA and other scheduling anomalies, but... > >> cpuid instruction is available for user mode as well. >> > ...it would not do any good for other subsystems, and user level code > and apps. Indeed. I think the optimal solution would be two-fold: give the scheduler the information it is needing to react correctly via a kernel patch not relying on cpuid values and fiddle with the cpuid values from xen tools according to any needs of other subsystems and/or user code (e.g. licensing). Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 15:14 ` Juergen Gross @ 2015-07-24 15:24 ` Juergen Gross 2015-07-24 15:58 ` Dario Faggioli 2015-07-24 16:05 ` Dario Faggioli 1 sibling, 1 reply; 95+ messages in thread From: Juergen Gross @ 2015-07-24 15:24 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/24/2015 05:14 PM, Juergen Gross wrote: > On 07/24/2015 04:44 PM, Dario Faggioli wrote: >> On Fri, 2015-07-24 at 12:28 +0200, Juergen Gross wrote: >>> On 07/23/2015 04:07 PM, Dario Faggioli wrote: >> >>>> FWIW, I was thinking that the kernel were a better place, as Juergen is >>>> saying, while now I'm more convinced that tools would be more >>>> appropriate, as Boris is saying. >>> >>> I've collected some information from the linux kernel sources as a base >>> for the discussion: >>> >> That's great, thanks for this! >> >>> The complete numa information (cpu->node and memory->node relations) is >>> taken from the acpi tables (srat, slit for "distances"). >>> >> Ok. And I already have a question (as I lost track of things a bit). >> What you just said about ACPI tables is certainly true for baremetal and >> HVM guests, but for PV? At the time I was looking into it, together with >> Elena, there were Linux patches being produced for the PV case, which >> makes sense. >> However, ISTR that both Wei and Elena mentioned recently that those >> patches have not been upstreamed in Linux yet... Is that the case? Maybe >> not all, but at least some of them are there? Because if not, I'm not >> sure I see how a PV guest would even see a vNUMA topology (which it >> does). >> >> Of course, I can go and check, but since you just looked, you may have >> it fresh and clear already. :-) > > I checked "bottom up", so when I found the acpi scan stuff I stopped > searching how the kernel obtains numa info. During my search I found no > clue of an pv-numa stuff in the kernel. And a quick "grep -i numa" in > arch/x86/xen and drivers/xen didn't reveal anything. Same for a complete > kernel source search for "vnuma". > >> >>> The topology information is obtained via: >>> - intel: >>> + cpuid leaf b with subleafs, leaf 4 >>> + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available >>> - amd: >>> + cpuid leaf 8000001e, leaf 8000001d, leaf 4 >>> + msr c001100c >>> + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available >>> >>> The scheduler is aware of: >>> - smt siblings (from topology) >>> - last-level-cache siblings (from topology) >>> - node siblings (from numa information) >>> >> Right. So, this confirms what we were guessing: we need to "reconcile" >> these two sources of information (from the guest point of view). >> >> Both the 'in kernel' and 'in toolstack' approach should have all the >> necessary information to make things match, I think. In fact, in >> toolstack, we know what the vNUMA topology is (we're parsing and >> actually putting it in place!). In kernel, we know it as we read it from >> tables or hypercalls (isn't that so, for PV guest?). >> >> In fact, I think that it is the topology, i.e., what comes from MSRs, >> that needs to adapt, and follow vNUMA, as much as possible. Do we agree >> on this? > > I think we have to be very careful here. I see two possible scenarios: > > 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will > try to schedule the vcpus according to their numa affinity. So they > can change pcpus at any time in case of very busy guests. I don't > think the linux kernel should treat the cpus differently in this > case as it will be in vane regarding the Xen scheduler's activity. > So we should use the "null" topology in this case. Sorry, the topology should reflect the vcpu<->numa-node relations, of course, but nothing else (so flat topolgy in each numa node). > > 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen > scheduler can't move vcpus between pcpus, so the linux kernel should > see the real topology of the used pcpus in order to optimize for this > picture. > > This only covers the scheduling aspect, of course. > >> >> IMO, the thing boils down to these: >> >> 1) from where (kernel vs. toolstack) is it the most easy and effective >> to enact the CPUID fiddling? As in, can we do that in toolstack? >> (Andrew was not so sure, and Boris found issues, although Jan seems >> to think they're no show stopper.) >> I'm quite certain that we can do that from inside the kernel, >> although, how early would we need to be doing it? Do we have the >> vNUMA info already? >> >> 2) when tweaking the values of CPUID and other MSRs, are there other >> vNUMA (and topology in general) constraints and requirements we >> should take into account? For instance, do we want, for licensing >> reasons, all (or most) of the vcpus to be siblings, rather than full >> sockets? Etc. >> 2a) if yes, how and where are these constraints specified? >> >> If looking at 1) only, it still looks to me that doing things within the >> kernel would be the way to go. >> >> When looking at 2), OTOH, toolstacks variants start to be more >> appealing. Especially depending on our answer to 2a). In fact, >> in case we want to give the user a way to specify this >> siblings-vs-cores-vs-sockets information, it IMO would be good to deal >> with that in tools, rather than having to involve Xen or Linux! >> >>> It will especially move tasks from one cpu to another first between smt >>> siblings, second between llc siblings, third between node siblings and >>> last all cpus. >>> >> Yep, this part, I knew. >> >> Maybe, there is room for "fixing" this at this level, hooking up inside >> the scheduler code... but I'm shooting in the dark, without having check >> whether and how this could be really feasible, should I? > > Uuh, I don't think a change of the scheduler on behalf of Xen is really > appreciated. :-) > > I'd rather fiddle with the cpu masks on the different levels to let the > scheduler do the right thing. > >> One thing I don't like about this approach is that it would potentially >> solve vNUMA and other scheduling anomalies, but... >> >>> cpuid instruction is available for user mode as well. >>> >> ...it would not do any good for other subsystems, and user level code >> and apps. > > Indeed. I think the optimal solution would be two-fold: give the > scheduler the information it is needing to react correctly via a > kernel patch not relying on cpuid values and fiddle with the cpuid > values from xen tools according to any needs of other subsystems and/or > user code (e.g. licensing). Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 15:24 ` Juergen Gross @ 2015-07-24 15:58 ` Dario Faggioli 2015-07-24 16:09 ` Konrad Rzeszutek Wilk 2015-07-24 16:10 ` Juergen Gross 0 siblings, 2 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-24 15:58 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 3832 bytes --] On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote: > On 07/24/2015 05:14 PM, Juergen Gross wrote: > > On 07/24/2015 04:44 PM, Dario Faggioli wrote: > >> In fact, I think that it is the topology, i.e., what comes from MSRs, > >> that needs to adapt, and follow vNUMA, as much as possible. Do we agree > >> on this? > > > > I think we have to be very careful here. I see two possible scenarios: > > > > 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will > > try to schedule the vcpus according to their numa affinity. So they > > can change pcpus at any time in case of very busy guests. I don't > > think the linux kernel should treat the cpus differently in this > > case as it will be in vane regarding the Xen scheduler's activity. > > So we should use the "null" topology in this case. > > Sorry, the topology should reflect the vcpu<->numa-node relations, of > course, but nothing else (so flat topolgy in each numa node). > Yeah, I was replying to this point saying something like this right now... Luckily, I've seen this email! :-P With this semantic, I fully agree with this. > > 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen > > scheduler can't move vcpus between pcpus, so the linux kernel should > > see the real topology of the used pcpus in order to optimize for this > > picture. > > > Mmm... I did think about this too, but I'm not sure. I see the value of this of course, and the reason why it makes sense. However, pinning can change on-line, via `xl vcpu-pin' and stuff. Also migration could make things less certain, I think. What happens if we build on top of the initial pinning, and then things change? To be fair, there is stuff building on top of the initial pinning already, e.g., from which physical NUMA node we allocate the memory relies depends exactly on that. That being said, I'm not sure I'm comfortable with adding more of this... Perhaps introduce an 'immutable_pinning' flag, which will prevent affinity to be changed, and then bind the topology to pinning only if that one is set? > >> Maybe, there is room for "fixing" this at this level, hooking up inside > >> the scheduler code... but I'm shooting in the dark, without having check > >> whether and how this could be really feasible, should I? > > > > Uuh, I don't think a change of the scheduler on behalf of Xen is really > > appreciated. :-) > > I'm sure it would (have been! :-)) a true and giant nightmare!! :-D > >> One thing I don't like about this approach is that it would potentially > >> solve vNUMA and other scheduling anomalies, but... > >> > >>> cpuid instruction is available for user mode as well. > >>> > >> ...it would not do any good for other subsystems, and user level code > >> and apps. > > > > Indeed. I think the optimal solution would be two-fold: give the > > scheduler the information it is needing to react correctly via a > > kernel patch not relying on cpuid values and fiddle with the cpuid > > values from xen tools according to any needs of other subsystems and/or > > user code (e.g. licensing). > So, just to check if I'm understanding is correct: you'd like to add an abstraction layer, in Linux, like in generic (or, perhaps, scheduling) code, to hide the direct interaction with CPUID. Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd check with Xen/match vNUMA/whatever... Is this that you are saying? If yes, I think I like it... Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 15:58 ` Dario Faggioli @ 2015-07-24 16:09 ` Konrad Rzeszutek Wilk 2015-07-24 16:14 ` Dario Faggioli ` (2 more replies) 2015-07-24 16:10 ` Juergen Gross 1 sibling, 3 replies; 95+ messages in thread From: Konrad Rzeszutek Wilk @ 2015-07-24 16:09 UTC (permalink / raw) To: Dario Faggioli Cc: Juergen Gross, Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote: > On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote: > > On 07/24/2015 05:14 PM, Juergen Gross wrote: > > > On 07/24/2015 04:44 PM, Dario Faggioli wrote: > > > >> In fact, I think that it is the topology, i.e., what comes from MSRs, > > >> that needs to adapt, and follow vNUMA, as much as possible. Do we agree > > >> on this? > > > > > > I think we have to be very careful here. I see two possible scenarios: > > > > > > 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will > > > try to schedule the vcpus according to their numa affinity. So they > > > can change pcpus at any time in case of very busy guests. I don't > > > think the linux kernel should treat the cpus differently in this > > > case as it will be in vane regarding the Xen scheduler's activity. > > > So we should use the "null" topology in this case. > > > > Sorry, the topology should reflect the vcpu<->numa-node relations, of > > course, but nothing else (so flat topolgy in each numa node). > > > Yeah, I was replying to this point saying something like this right > now... Luckily, I've seen this email! :-P > > With this semantic, I fully agree with this. > > > > 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen > > > scheduler can't move vcpus between pcpus, so the linux kernel should > > > see the real topology of the used pcpus in order to optimize for this > > > picture. > > > > > > Mmm... I did think about this too, but I'm not sure. I see the value of > this of course, and the reason why it makes sense. However, pinning can > change on-line, via `xl vcpu-pin' and stuff. Also migration could make > things less certain, I think. What happens if we build on top of the > initial pinning, and then things change? > > To be fair, there is stuff building on top of the initial pinning > already, e.g., from which physical NUMA node we allocate the memory > relies depends exactly on that. That being said, I'm not sure I'm > comfortable with adding more of this... > > Perhaps introduce an 'immutable_pinning' flag, which will prevent > affinity to be changed, and then bind the topology to pinning only if > that one is set? > > > >> Maybe, there is room for "fixing" this at this level, hooking up inside > > >> the scheduler code... but I'm shooting in the dark, without having check > > >> whether and how this could be really feasible, should I? > > > > > > Uuh, I don't think a change of the scheduler on behalf of Xen is really > > > appreciated. :-) > > > > I'm sure it would (have been! :-)) a true and giant nightmare!! :-D > > > >> One thing I don't like about this approach is that it would potentially > > >> solve vNUMA and other scheduling anomalies, but... > > >> > > >>> cpuid instruction is available for user mode as well. > > >>> > > >> ...it would not do any good for other subsystems, and user level code > > >> and apps. > > > > > > Indeed. I think the optimal solution would be two-fold: give the > > > scheduler the information it is needing to react correctly via a > > > kernel patch not relying on cpuid values and fiddle with the cpuid > > > values from xen tools according to any needs of other subsystems and/or > > > user code (e.g. licensing). > > > So, just to check if I'm understanding is correct: you'd like to add an > abstraction layer, in Linux, like in generic (or, perhaps, scheduling) > code, to hide the direct interaction with CPUID. > Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd > check with Xen/match vNUMA/whatever... Is this that you are saying? > > If yes, I think I like it... I don't think this is workable. For example there are applications which use 'cpuid' and figure out the core/thread and use it for its own scheduling purposes. > > Regards, > Dario > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:09 ` Konrad Rzeszutek Wilk @ 2015-07-24 16:14 ` Dario Faggioli 2015-07-24 16:18 ` Juergen Gross 2015-07-27 10:41 ` George Dunlap 2 siblings, 0 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-24 16:14 UTC (permalink / raw) To: Konrad Rzeszutek Wilk Cc: Juergen Gross, Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 1068 bytes --] On Fri, 2015-07-24 at 12:09 -0400, Konrad Rzeszutek Wilk wrote: > On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote: > > So, just to check if I'm understanding is correct: you'd like to add an > > abstraction layer, in Linux, like in generic (or, perhaps, scheduling) > > code, to hide the direct interaction with CPUID. > > Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd > > check with Xen/match vNUMA/whatever... Is this that you are saying? > > > > If yes, I think I like it... > > I don't think this is workable. For example there are applications > which use 'cpuid' and figure out the core/thread and use it for its own > scheduling purposes. > Naah! That pretty much only happens with databases, so, honestly, who cares!! ;-P ;-P ;-P Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:09 ` Konrad Rzeszutek Wilk 2015-07-24 16:14 ` Dario Faggioli @ 2015-07-24 16:18 ` Juergen Gross 2015-07-24 16:29 ` Konrad Rzeszutek Wilk 2015-07-27 10:41 ` George Dunlap 2 siblings, 1 reply; 95+ messages in thread From: Juergen Gross @ 2015-07-24 16:18 UTC (permalink / raw) To: Konrad Rzeszutek Wilk, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/24/2015 06:09 PM, Konrad Rzeszutek Wilk wrote: > On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote: >> On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote: >>> On 07/24/2015 05:14 PM, Juergen Gross wrote: >>>> On 07/24/2015 04:44 PM, Dario Faggioli wrote: >> >>>>> In fact, I think that it is the topology, i.e., what comes from MSRs, >>>>> that needs to adapt, and follow vNUMA, as much as possible. Do we agree >>>>> on this? >>>> >>>> I think we have to be very careful here. I see two possible scenarios: >>>> >>>> 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will >>>> try to schedule the vcpus according to their numa affinity. So they >>>> can change pcpus at any time in case of very busy guests. I don't >>>> think the linux kernel should treat the cpus differently in this >>>> case as it will be in vane regarding the Xen scheduler's activity. >>>> So we should use the "null" topology in this case. >>> >>> Sorry, the topology should reflect the vcpu<->numa-node relations, of >>> course, but nothing else (so flat topolgy in each numa node). >>> >> Yeah, I was replying to this point saying something like this right >> now... Luckily, I've seen this email! :-P >> >> With this semantic, I fully agree with this. >> >>>> 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen >>>> scheduler can't move vcpus between pcpus, so the linux kernel should >>>> see the real topology of the used pcpus in order to optimize for this >>>> picture. >>>> >>> >> Mmm... I did think about this too, but I'm not sure. I see the value of >> this of course, and the reason why it makes sense. However, pinning can >> change on-line, via `xl vcpu-pin' and stuff. Also migration could make >> things less certain, I think. What happens if we build on top of the >> initial pinning, and then things change? >> >> To be fair, there is stuff building on top of the initial pinning >> already, e.g., from which physical NUMA node we allocate the memory >> relies depends exactly on that. That being said, I'm not sure I'm >> comfortable with adding more of this... >> >> Perhaps introduce an 'immutable_pinning' flag, which will prevent >> affinity to be changed, and then bind the topology to pinning only if >> that one is set? >> >>>>> Maybe, there is room for "fixing" this at this level, hooking up inside >>>>> the scheduler code... but I'm shooting in the dark, without having check >>>>> whether and how this could be really feasible, should I? >>>> >>>> Uuh, I don't think a change of the scheduler on behalf of Xen is really >>>> appreciated. :-) >>>> >> I'm sure it would (have been! :-)) a true and giant nightmare!! :-D >> >>>>> One thing I don't like about this approach is that it would potentially >>>>> solve vNUMA and other scheduling anomalies, but... >>>>> >>>>>> cpuid instruction is available for user mode as well. >>>>>> >>>>> ...it would not do any good for other subsystems, and user level code >>>>> and apps. >>>> >>>> Indeed. I think the optimal solution would be two-fold: give the >>>> scheduler the information it is needing to react correctly via a >>>> kernel patch not relying on cpuid values and fiddle with the cpuid >>>> values from xen tools according to any needs of other subsystems and/or >>>> user code (e.g. licensing). >>> >> So, just to check if I'm understanding is correct: you'd like to add an >> abstraction layer, in Linux, like in generic (or, perhaps, scheduling) >> code, to hide the direct interaction with CPUID. >> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd >> check with Xen/match vNUMA/whatever... Is this that you are saying? >> >> If yes, I think I like it... > > I don't think this is workable. For example there are applications > which use 'cpuid' and figure out the core/thread and use it for its own > scheduling purposes. Might be, yes. The pure cpuid solution won't work for all license related issues. Doing it via an abstraction layer in the kernel would work in more than 90% of all cases AND would still enable a user to fiddle cpuids according to his needs (either topology or license). I'd rather have an out-of-the-box kernel solution with special user requirements handling than a complex solution making some user requirements impossible to meet. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:18 ` Juergen Gross @ 2015-07-24 16:29 ` Konrad Rzeszutek Wilk 2015-07-24 16:39 ` Juergen Gross 0 siblings, 1 reply; 95+ messages in thread From: Konrad Rzeszutek Wilk @ 2015-07-24 16:29 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On Fri, Jul 24, 2015 at 06:18:56PM +0200, Juergen Gross wrote: > On 07/24/2015 06:09 PM, Konrad Rzeszutek Wilk wrote: > >On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote: > >>On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote: > >>>On 07/24/2015 05:14 PM, Juergen Gross wrote: > >>>>On 07/24/2015 04:44 PM, Dario Faggioli wrote: > >> > >>>>>In fact, I think that it is the topology, i.e., what comes from MSRs, > >>>>>that needs to adapt, and follow vNUMA, as much as possible. Do we agree > >>>>>on this? > >>>> > >>>>I think we have to be very careful here. I see two possible scenarios: > >>>> > >>>>1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will > >>>> try to schedule the vcpus according to their numa affinity. So they > >>>> can change pcpus at any time in case of very busy guests. I don't > >>>> think the linux kernel should treat the cpus differently in this > >>>> case as it will be in vane regarding the Xen scheduler's activity. > >>>> So we should use the "null" topology in this case. > >>> > >>>Sorry, the topology should reflect the vcpu<->numa-node relations, of > >>>course, but nothing else (so flat topolgy in each numa node). > >>> > >>Yeah, I was replying to this point saying something like this right > >>now... Luckily, I've seen this email! :-P > >> > >>With this semantic, I fully agree with this. > >> > >>>>2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen > >>>> scheduler can't move vcpus between pcpus, so the linux kernel should > >>>> see the real topology of the used pcpus in order to optimize for this > >>>> picture. > >>>> > >>> > >>Mmm... I did think about this too, but I'm not sure. I see the value of > >>this of course, and the reason why it makes sense. However, pinning can > >>change on-line, via `xl vcpu-pin' and stuff. Also migration could make > >>things less certain, I think. What happens if we build on top of the > >>initial pinning, and then things change? > >> > >>To be fair, there is stuff building on top of the initial pinning > >>already, e.g., from which physical NUMA node we allocate the memory > >>relies depends exactly on that. That being said, I'm not sure I'm > >>comfortable with adding more of this... > >> > >>Perhaps introduce an 'immutable_pinning' flag, which will prevent > >>affinity to be changed, and then bind the topology to pinning only if > >>that one is set? > >> > >>>>>Maybe, there is room for "fixing" this at this level, hooking up inside > >>>>>the scheduler code... but I'm shooting in the dark, without having check > >>>>>whether and how this could be really feasible, should I? > >>>> > >>>>Uuh, I don't think a change of the scheduler on behalf of Xen is really > >>>>appreciated. :-) > >>>> > >>I'm sure it would (have been! :-)) a true and giant nightmare!! :-D > >> > >>>>>One thing I don't like about this approach is that it would potentially > >>>>>solve vNUMA and other scheduling anomalies, but... > >>>>> > >>>>>>cpuid instruction is available for user mode as well. > >>>>>> > >>>>>...it would not do any good for other subsystems, and user level code > >>>>>and apps. > >>>> > >>>>Indeed. I think the optimal solution would be two-fold: give the > >>>>scheduler the information it is needing to react correctly via a > >>>>kernel patch not relying on cpuid values and fiddle with the cpuid > >>>>values from xen tools according to any needs of other subsystems and/or > >>>>user code (e.g. licensing). > >>> > >>So, just to check if I'm understanding is correct: you'd like to add an > >>abstraction layer, in Linux, like in generic (or, perhaps, scheduling) > >>code, to hide the direct interaction with CPUID. > >>Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd > >>check with Xen/match vNUMA/whatever... Is this that you are saying? > >> > >>If yes, I think I like it... > > > >I don't think this is workable. For example there are applications > >which use 'cpuid' and figure out the core/thread and use it for its own > >scheduling purposes. > > Might be, yes. There are <cough>databases</cough> that do this. > > The pure cpuid solution won't work for all license related issues. > > Doing it via an abstraction layer in the kernel would work in more than > 90% of all cases AND would still enable a user to fiddle cpuids > according to his needs (either topology or license). > > I'd rather have an out-of-the-box kernel solution with special user > requirements handling than a complex solution making some user > requirements impossible to meet. I think there are two issues here - the solution you are trying to come up with is for PV scenarios. But the issue I described is for PVH and HVM - where the cpuid is intercepted by the hypervisor and we can mangle it as we see fit. I think so? Perhaps I misunderstood? > > > Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:29 ` Konrad Rzeszutek Wilk @ 2015-07-24 16:39 ` Juergen Gross 2015-07-24 16:44 ` Boris Ostrovsky 0 siblings, 1 reply; 95+ messages in thread From: Juergen Gross @ 2015-07-24 16:39 UTC (permalink / raw) To: Konrad Rzeszutek Wilk Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/24/2015 06:29 PM, Konrad Rzeszutek Wilk wrote: > On Fri, Jul 24, 2015 at 06:18:56PM +0200, Juergen Gross wrote: >> On 07/24/2015 06:09 PM, Konrad Rzeszutek Wilk wrote: >>> On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote: >>>> On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote: >>>>> On 07/24/2015 05:14 PM, Juergen Gross wrote: >>>>>> On 07/24/2015 04:44 PM, Dario Faggioli wrote: >>>> >>>>>>> In fact, I think that it is the topology, i.e., what comes from MSRs, >>>>>>> that needs to adapt, and follow vNUMA, as much as possible. Do we agree >>>>>>> on this? >>>>>> >>>>>> I think we have to be very careful here. I see two possible scenarios: >>>>>> >>>>>> 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will >>>>>> try to schedule the vcpus according to their numa affinity. So they >>>>>> can change pcpus at any time in case of very busy guests. I don't >>>>>> think the linux kernel should treat the cpus differently in this >>>>>> case as it will be in vane regarding the Xen scheduler's activity. >>>>>> So we should use the "null" topology in this case. >>>>> >>>>> Sorry, the topology should reflect the vcpu<->numa-node relations, of >>>>> course, but nothing else (so flat topolgy in each numa node). >>>>> >>>> Yeah, I was replying to this point saying something like this right >>>> now... Luckily, I've seen this email! :-P >>>> >>>> With this semantic, I fully agree with this. >>>> >>>>>> 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen >>>>>> scheduler can't move vcpus between pcpus, so the linux kernel should >>>>>> see the real topology of the used pcpus in order to optimize for this >>>>>> picture. >>>>>> >>>>> >>>> Mmm... I did think about this too, but I'm not sure. I see the value of >>>> this of course, and the reason why it makes sense. However, pinning can >>>> change on-line, via `xl vcpu-pin' and stuff. Also migration could make >>>> things less certain, I think. What happens if we build on top of the >>>> initial pinning, and then things change? >>>> >>>> To be fair, there is stuff building on top of the initial pinning >>>> already, e.g., from which physical NUMA node we allocate the memory >>>> relies depends exactly on that. That being said, I'm not sure I'm >>>> comfortable with adding more of this... >>>> >>>> Perhaps introduce an 'immutable_pinning' flag, which will prevent >>>> affinity to be changed, and then bind the topology to pinning only if >>>> that one is set? >>>> >>>>>>> Maybe, there is room for "fixing" this at this level, hooking up inside >>>>>>> the scheduler code... but I'm shooting in the dark, without having check >>>>>>> whether and how this could be really feasible, should I? >>>>>> >>>>>> Uuh, I don't think a change of the scheduler on behalf of Xen is really >>>>>> appreciated. :-) >>>>>> >>>> I'm sure it would (have been! :-)) a true and giant nightmare!! :-D >>>> >>>>>>> One thing I don't like about this approach is that it would potentially >>>>>>> solve vNUMA and other scheduling anomalies, but... >>>>>>> >>>>>>>> cpuid instruction is available for user mode as well. >>>>>>>> >>>>>>> ...it would not do any good for other subsystems, and user level code >>>>>>> and apps. >>>>>> >>>>>> Indeed. I think the optimal solution would be two-fold: give the >>>>>> scheduler the information it is needing to react correctly via a >>>>>> kernel patch not relying on cpuid values and fiddle with the cpuid >>>>>> values from xen tools according to any needs of other subsystems and/or >>>>>> user code (e.g. licensing). >>>>> >>>> So, just to check if I'm understanding is correct: you'd like to add an >>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling) >>>> code, to hide the direct interaction with CPUID. >>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd >>>> check with Xen/match vNUMA/whatever... Is this that you are saying? >>>> >>>> If yes, I think I like it... >>> >>> I don't think this is workable. For example there are applications >>> which use 'cpuid' and figure out the core/thread and use it for its own >>> scheduling purposes. >> >> Might be, yes. > > There are <cough>databases</cough> that do this. Really? How do you know? ;-) >> The pure cpuid solution won't work for all license related issues. >> >> Doing it via an abstraction layer in the kernel would work in more than >> 90% of all cases AND would still enable a user to fiddle cpuids >> according to his needs (either topology or license). >> >> I'd rather have an out-of-the-box kernel solution with special user >> requirements handling than a complex solution making some user >> requirements impossible to meet. > > I think there are two issues here - the solution you are trying to come > up with is for PV scenarios. I'd say my solution is a paravirtualized one. > But the issue I described is for PVH and HVM - where the cpuid is intercepted > by the hypervisor and we can mangle it as we see fit. I think even PVH and HVM are able to use paravirtualized interfaces. I don't say mangling cpuids can't solve the scheduling problem. It surely can. But it can't solve the scheduling problem without hiding information like number of sockets or cores which might be required for license purposes. If we don't care, fine. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:39 ` Juergen Gross @ 2015-07-24 16:44 ` Boris Ostrovsky 2015-07-27 4:35 ` Juergen Gross 0 siblings, 1 reply; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-24 16:44 UTC (permalink / raw) To: Juergen Gross, Konrad Rzeszutek Wilk Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel On 07/24/2015 12:39 PM, Juergen Gross wrote: > > > I don't say mangling cpuids can't solve the scheduling problem. It > surely can. But it can't solve the scheduling problem without hiding > information like number of sockets or cores which might be required > for license purposes. If we don't care, fine. > (this is somewhat repeating the email I just sent) Why can's we construct socket/core info with CPUID (and *possibly* ACPI changes) that we present a reasonable (licensing-wise) picture? Can you suggest an example where it will not work and then maybe we can figure something out? -boris ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:44 ` Boris Ostrovsky @ 2015-07-27 4:35 ` Juergen Gross 2015-07-27 10:43 ` George Dunlap 0 siblings, 1 reply; 95+ messages in thread From: Juergen Gross @ 2015-07-27 4:35 UTC (permalink / raw) To: Boris Ostrovsky, Konrad Rzeszutek Wilk Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel On 07/24/2015 06:44 PM, Boris Ostrovsky wrote: > On 07/24/2015 12:39 PM, Juergen Gross wrote: >> >> >> I don't say mangling cpuids can't solve the scheduling problem. It >> surely can. But it can't solve the scheduling problem without hiding >> information like number of sockets or cores which might be required >> for license purposes. If we don't care, fine. >> > > (this is somewhat repeating the email I just sent) > > Why can's we construct socket/core info with CPUID (and *possibly* ACPI > changes) that we present a reasonable (licensing-wise) picture? > > Can you suggest an example where it will not work and then maybe we can > figure something out? Let's assume a software with license based on core count. You have a system with a 2 8 core processors and hyperthreads enabled, summing up to 32 logical processors. Your license is valid for up to 16 cores, so running the software on bare metal on your system is fine. Now you are running the software inside a virtual machine with 24 vcpus in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each processor). As we have to hide hyperthreading in order to not to have to pin each vcpu to just a single logical processor, the topology resulting from this picture will have to present 24 cores. The license will not cover this hardware. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 4:35 ` Juergen Gross @ 2015-07-27 10:43 ` George Dunlap 2015-07-27 10:54 ` Andrew Cooper 2015-07-27 10:54 ` Juergen Gross 0 siblings, 2 replies; 95+ messages in thread From: George Dunlap @ 2015-07-27 10:43 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote: > On 07/24/2015 06:44 PM, Boris Ostrovsky wrote: >> >> On 07/24/2015 12:39 PM, Juergen Gross wrote: >>> >>> >>> >>> I don't say mangling cpuids can't solve the scheduling problem. It >>> surely can. But it can't solve the scheduling problem without hiding >>> information like number of sockets or cores which might be required >>> for license purposes. If we don't care, fine. >>> >> >> (this is somewhat repeating the email I just sent) >> >> Why can's we construct socket/core info with CPUID (and *possibly* ACPI >> changes) that we present a reasonable (licensing-wise) picture? >> >> Can you suggest an example where it will not work and then maybe we can >> figure something out? > > > Let's assume a software with license based on core count. You have a > system with a 2 8 core processors and hyperthreads enabled, summing up > to 32 logical processors. Your license is valid for up to 16 cores, so > running the software on bare metal on your system is fine. > > Now you are running the software inside a virtual machine with 24 vcpus > in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each > processor). As we have to hide hyperthreading in order to not to have > to pin each vcpu to just a single logical processor, the topology > resulting from this picture will have to present 24 cores. The license > will not cover this hardware. But how does doing a PV topology help this situation? Because we're telling one thing to the OS (via our PV interface) and another thing to applications (via direct CPUID access)? -George ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 10:43 ` George Dunlap @ 2015-07-27 10:54 ` Andrew Cooper 2015-07-27 11:13 ` Juergen Gross 2015-07-27 10:54 ` Juergen Gross 1 sibling, 1 reply; 95+ messages in thread From: Andrew Cooper @ 2015-07-27 10:54 UTC (permalink / raw) To: George Dunlap, Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 27/07/15 11:43, George Dunlap wrote: > On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote: >> On 07/24/2015 06:44 PM, Boris Ostrovsky wrote: >>> On 07/24/2015 12:39 PM, Juergen Gross wrote: >>>> >>>> >>>> I don't say mangling cpuids can't solve the scheduling problem. It >>>> surely can. But it can't solve the scheduling problem without hiding >>>> information like number of sockets or cores which might be required >>>> for license purposes. If we don't care, fine. >>>> >>> (this is somewhat repeating the email I just sent) >>> >>> Why can's we construct socket/core info with CPUID (and *possibly* ACPI >>> changes) that we present a reasonable (licensing-wise) picture? >>> >>> Can you suggest an example where it will not work and then maybe we can >>> figure something out? >> >> Let's assume a software with license based on core count. You have a >> system with a 2 8 core processors and hyperthreads enabled, summing up >> to 32 logical processors. Your license is valid for up to 16 cores, so >> running the software on bare metal on your system is fine. >> >> Now you are running the software inside a virtual machine with 24 vcpus >> in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each >> processor). As we have to hide hyperthreading in order to not to have >> to pin each vcpu to just a single logical processor, the topology >> resulting from this picture will have to present 24 cores. The license >> will not cover this hardware. > But how does doing a PV topology help this situation? Because we're > telling one thing to the OS (via our PV interface) and another thing > to applications (via direct CPUID access)? I expressed exactly these concerns right back at the start of the vnuma work. The OS and its userspace can and will use cpuid. Most examples will only use cpuid. The only thing worse that providing no NUMA information at all is providing conflicting information between cpuid and vnuma. IMO, HVM guests should get all their NUMA information from the same sources as native hardware would provide. PV guests are admittedly harder as in generally we cannot hide the real topology information in cpuid. ~Andrew ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 10:54 ` Andrew Cooper @ 2015-07-27 11:13 ` Juergen Gross 0 siblings, 0 replies; 95+ messages in thread From: Juergen Gross @ 2015-07-27 11:13 UTC (permalink / raw) To: Andrew Cooper, George Dunlap Cc: Elena Ufimtseva, Wei Liu, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/27/2015 12:54 PM, Andrew Cooper wrote: > On 27/07/15 11:43, George Dunlap wrote: >> On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote: >>> On 07/24/2015 06:44 PM, Boris Ostrovsky wrote: >>>> On 07/24/2015 12:39 PM, Juergen Gross wrote: >>>>> >>>>> >>>>> I don't say mangling cpuids can't solve the scheduling problem. It >>>>> surely can. But it can't solve the scheduling problem without hiding >>>>> information like number of sockets or cores which might be required >>>>> for license purposes. If we don't care, fine. >>>>> >>>> (this is somewhat repeating the email I just sent) >>>> >>>> Why can's we construct socket/core info with CPUID (and *possibly* ACPI >>>> changes) that we present a reasonable (licensing-wise) picture? >>>> >>>> Can you suggest an example where it will not work and then maybe we can >>>> figure something out? >>> >>> Let's assume a software with license based on core count. You have a >>> system with a 2 8 core processors and hyperthreads enabled, summing up >>> to 32 logical processors. Your license is valid for up to 16 cores, so >>> running the software on bare metal on your system is fine. >>> >>> Now you are running the software inside a virtual machine with 24 vcpus >>> in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each >>> processor). As we have to hide hyperthreading in order to not to have >>> to pin each vcpu to just a single logical processor, the topology >>> resulting from this picture will have to present 24 cores. The license >>> will not cover this hardware. >> But how does doing a PV topology help this situation? Because we're >> telling one thing to the OS (via our PV interface) and another thing >> to applications (via direct CPUID access)? > > I expressed exactly these concerns right back at the start of the vnuma > work. > > The OS and its userspace can and will use cpuid. Most examples will > only use cpuid. The only thing worse that providing no NUMA information > at all is providing conflicting information between cpuid and vnuma. > > IMO, HVM guests should get all their NUMA information from the same > sources as native hardware would provide. PV guests are admittedly > harder as in generally we cannot hide the real topology information in > cpuid. Are you aware the same is true currently even without vNUMA? The linux kernel (and other OS's as well) will make scheduling decisions based on cpuid data obtained during boot. The information will be correct only by chance and the real relation between vcpus and pcpus is changing all the time. So without adapting the kernel to that scenario it won't run optimal. You can either change the data to let the kernel make some sane decisions (cpuid mangling) or you can adapt the kernel somehow, e.g. by modifying the kernel internal tables used for making scheduling decisions (my proposal). Something should be done regardless of the vNUMA support. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 10:43 ` George Dunlap 2015-07-27 10:54 ` Andrew Cooper @ 2015-07-27 10:54 ` Juergen Gross 2015-07-27 11:11 ` George Dunlap 1 sibling, 1 reply; 95+ messages in thread From: Juergen Gross @ 2015-07-27 10:54 UTC (permalink / raw) To: George Dunlap Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/27/2015 12:43 PM, George Dunlap wrote: > On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote: >> On 07/24/2015 06:44 PM, Boris Ostrovsky wrote: >>> >>> On 07/24/2015 12:39 PM, Juergen Gross wrote: >>>> >>>> >>>> >>>> I don't say mangling cpuids can't solve the scheduling problem. It >>>> surely can. But it can't solve the scheduling problem without hiding >>>> information like number of sockets or cores which might be required >>>> for license purposes. If we don't care, fine. >>>> >>> >>> (this is somewhat repeating the email I just sent) >>> >>> Why can's we construct socket/core info with CPUID (and *possibly* ACPI >>> changes) that we present a reasonable (licensing-wise) picture? >>> >>> Can you suggest an example where it will not work and then maybe we can >>> figure something out? >> >> >> Let's assume a software with license based on core count. You have a >> system with a 2 8 core processors and hyperthreads enabled, summing up >> to 32 logical processors. Your license is valid for up to 16 cores, so >> running the software on bare metal on your system is fine. >> >> Now you are running the software inside a virtual machine with 24 vcpus >> in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each >> processor). As we have to hide hyperthreading in order to not to have >> to pin each vcpu to just a single logical processor, the topology >> resulting from this picture will have to present 24 cores. The license >> will not cover this hardware. > > But how does doing a PV topology help this situation? Because we're > telling one thing to the OS (via our PV interface) and another thing > to applications (via direct CPUID access)? Exactly. In my example it would even work to not modify the cpuid information at all. The kernel wouldn't try to be extremely clever regarding scheduling and the user land would see the cpuid information from the real hardware (only the 12 cores it is running on, of course). Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 10:54 ` Juergen Gross @ 2015-07-27 11:11 ` George Dunlap 2015-07-27 12:01 ` Juergen Gross 2015-07-27 14:02 ` Dario Faggioli 0 siblings, 2 replies; 95+ messages in thread From: George Dunlap @ 2015-07-27 11:11 UTC (permalink / raw) To: Juergen Gross, George Dunlap Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/27/2015 11:54 AM, Juergen Gross wrote: > On 07/27/2015 12:43 PM, George Dunlap wrote: >> On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote: >>> On 07/24/2015 06:44 PM, Boris Ostrovsky wrote: >>>> >>>> On 07/24/2015 12:39 PM, Juergen Gross wrote: >>>>> >>>>> >>>>> >>>>> I don't say mangling cpuids can't solve the scheduling problem. It >>>>> surely can. But it can't solve the scheduling problem without hiding >>>>> information like number of sockets or cores which might be required >>>>> for license purposes. If we don't care, fine. >>>>> >>>> >>>> (this is somewhat repeating the email I just sent) >>>> >>>> Why can's we construct socket/core info with CPUID (and *possibly* ACPI >>>> changes) that we present a reasonable (licensing-wise) picture? >>>> >>>> Can you suggest an example where it will not work and then maybe we can >>>> figure something out? >>> >>> >>> Let's assume a software with license based on core count. You have a >>> system with a 2 8 core processors and hyperthreads enabled, summing up >>> to 32 logical processors. Your license is valid for up to 16 cores, so >>> running the software on bare metal on your system is fine. >>> >>> Now you are running the software inside a virtual machine with 24 vcpus >>> in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each >>> processor). As we have to hide hyperthreading in order to not to have >>> to pin each vcpu to just a single logical processor, the topology >>> resulting from this picture will have to present 24 cores. The license >>> will not cover this hardware. >> >> But how does doing a PV topology help this situation? Because we're >> telling one thing to the OS (via our PV interface) and another thing >> to applications (via direct CPUID access)? > > Exactly. > > In my example it would even work to not modify the cpuid information at > all. The kernel wouldn't try to be extremely clever regarding scheduling > and the user land would see the cpuid information from the real hardware > (only the 12 cores it is running on, of course). Right; so it seems 1. Userspace applications are in the habit of reading CPUID to determine the topology of the system they're running on 2. Many use the topology information to help themselves make better scheduling decisions. Because a vcpu is not typically pinned to a specific pcpu, we may need to lie here slightly (e.g., not mention threads) to get the optimal behavior overall. 3. Others use the topology information to implement licensing restrictions. Because threads are treated differently to cores, we want to tell the truth here (i.e., make sure we mention that some of these are threads) to get the optimal behavior overall. Numbers #2 and #3 lead to contradictory courses of action; we cannot optimize for both at the same time. I think at some level we need to just try to accommodate both -- if the user doesn't have licensing issues, or prefers performance over licensing, then present a unified topology in PVH / HVM using CPUID, ACPI, &c. I think this should be the default. If the user has licensing issues, and doesn't mind having wonky or unreliable topology to its guests, then let the raw CPUID through. But it would, in this case, be good to try to give the guest OS scheduler a hint that it shouldn't really bother trying to read the topology or do placement as a result, as any decisions will be unreliable. Or alternately, if the user wants to give up on the "consolidation" aspect of virtualization, they can pin vcpus to pcpus and then pass in the actual host topology (hyperthreads and all). -George ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 11:11 ` George Dunlap @ 2015-07-27 12:01 ` Juergen Gross 2015-07-27 12:16 ` Tim Deegan 2015-07-27 13:23 ` Dario Faggioli 2015-07-27 14:02 ` Dario Faggioli 1 sibling, 2 replies; 95+ messages in thread From: Juergen Gross @ 2015-07-27 12:01 UTC (permalink / raw) To: George Dunlap, George Dunlap Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/27/2015 01:11 PM, George Dunlap wrote: > On 07/27/2015 11:54 AM, Juergen Gross wrote: >> On 07/27/2015 12:43 PM, George Dunlap wrote: >>> On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote: >>>> On 07/24/2015 06:44 PM, Boris Ostrovsky wrote: >>>>> >>>>> On 07/24/2015 12:39 PM, Juergen Gross wrote: >>>>>> >>>>>> >>>>>> >>>>>> I don't say mangling cpuids can't solve the scheduling problem. It >>>>>> surely can. But it can't solve the scheduling problem without hiding >>>>>> information like number of sockets or cores which might be required >>>>>> for license purposes. If we don't care, fine. >>>>>> >>>>> >>>>> (this is somewhat repeating the email I just sent) >>>>> >>>>> Why can's we construct socket/core info with CPUID (and *possibly* ACPI >>>>> changes) that we present a reasonable (licensing-wise) picture? >>>>> >>>>> Can you suggest an example where it will not work and then maybe we can >>>>> figure something out? >>>> >>>> >>>> Let's assume a software with license based on core count. You have a >>>> system with a 2 8 core processors and hyperthreads enabled, summing up >>>> to 32 logical processors. Your license is valid for up to 16 cores, so >>>> running the software on bare metal on your system is fine. >>>> >>>> Now you are running the software inside a virtual machine with 24 vcpus >>>> in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each >>>> processor). As we have to hide hyperthreading in order to not to have >>>> to pin each vcpu to just a single logical processor, the topology >>>> resulting from this picture will have to present 24 cores. The license >>>> will not cover this hardware. >>> >>> But how does doing a PV topology help this situation? Because we're >>> telling one thing to the OS (via our PV interface) and another thing >>> to applications (via direct CPUID access)? >> >> Exactly. >> >> In my example it would even work to not modify the cpuid information at >> all. The kernel wouldn't try to be extremely clever regarding scheduling >> and the user land would see the cpuid information from the real hardware >> (only the 12 cores it is running on, of course). > > Right; so it seems > > 1. Userspace applications are in the habit of reading CPUID to determine > the topology of the system they're running on > > 2. Many use the topology information to help themselves make better > scheduling decisions. Because a vcpu is not typically pinned to a > specific pcpu, we may need to lie here slightly (e.g., not mention > threads) to get the optimal behavior overall. > > 3. Others use the topology information to implement licensing > restrictions. Because threads are treated differently to cores, we want > to tell the truth here (i.e., make sure we mention that some of these > are threads) to get the optimal behavior overall. > > Numbers #2 and #3 lead to contradictory courses of action; we cannot > optimize for both at the same time. > > I think at some level we need to just try to accommodate both -- if the > user doesn't have licensing issues, or prefers performance over > licensing, then present a unified topology in PVH / HVM using CPUID, > ACPI, &c. I think this should be the default. > > If the user has licensing issues, and doesn't mind having wonky or > unreliable topology to its guests, then let the raw CPUID through. But > it would, in this case, be good to try to give the guest OS scheduler a > hint that it shouldn't really bother trying to read the topology or do > placement as a result, as any decisions will be unreliable. > > Or alternately, if the user wants to give up on the "consolidation" > aspect of virtualization, they can pin vcpus to pcpus and then pass in > the actual host topology (hyperthreads and all). There would be another solution, of course: Support hyperthreads in the Xen scheduler via gang scheduling. While this is not a simple solution, it is a fair one. Hyperthreads on one core can influence each other rather much. With both threads always running vcpus of the same guest the penalty/advantage would stay in the same domain. The guest could make really sensible scheduling decisions and the licensing would still work as desired. Just an idea, but maybe worth to explore further instead of tweaking more and more bits to make the virtual system somehow act sane. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 12:01 ` Juergen Gross @ 2015-07-27 12:16 ` Tim Deegan 2015-07-27 13:23 ` Dario Faggioli 1 sibling, 0 replies; 95+ messages in thread From: Tim Deegan @ 2015-07-27 12:16 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, George Dunlap, Andrew Cooper, Dario Faggioli, George Dunlap, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky At 14:01 +0200 on 27 Jul (1438005701), Juergen Gross wrote: > There would be another solution, of course: > > Support hyperthreads in the Xen scheduler via gang scheduling. While > this is not a simple solution, it is a fair one. Hyperthreads on one > core can influence each other rather much. With both threads always > running vcpus of the same guest the penalty/advantage would stay in the > same domain. The guest could make really sensible scheduling decisions > and the licensing would still work as desired. This might also be interesting for people worried about side-channels (e.g. http://sophia.re/RECON ). Tim. ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 12:01 ` Juergen Gross 2015-07-27 12:16 ` Tim Deegan @ 2015-07-27 13:23 ` Dario Faggioli 2015-07-27 14:02 ` Juergen Gross 1 sibling, 1 reply; 95+ messages in thread From: Dario Faggioli @ 2015-07-27 13:23 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, George Dunlap, Andrew Cooper, George Dunlap, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 2166 bytes --] On Mon, 2015-07-27 at 14:01 +0200, Juergen Gross wrote: > On 07/27/2015 01:11 PM, George Dunlap wrote: > > Or alternately, if the user wants to give up on the "consolidation" > > aspect of virtualization, they can pin vcpus to pcpus and then pass in > > the actual host topology (hyperthreads and all). > > There would be another solution, of course: > > Support hyperthreads in the Xen scheduler via gang scheduling. While > this is not a simple solution, it is a fair one. Hyperthreads on one > core can influence each other rather much. With both threads always > running vcpus of the same guest the penalty/advantage would stay in the > same domain. The guest could make really sensible scheduling decisions > and the licensing would still work as desired. > This is interesting indeed, but I much rather see it as something orthogonal, which may indeed bring benefits in some of the scenarios described here, but should not be considered *the* solution. Implementing, enabling and asking users to use something like this will impact the system behavior and performance, in ways that may not be desirable for all use cases. So, while I do think that this may be something nice to have and offer, trying to use it for solving the problem we're debating here would make things even more complex to configure. Also, this would take care of HT related issues, but what about cores (as in 'should vcpus be cores of sockets or full sockets') and !HT boxes (like AMD)? Not to mention, as you say yourself, that it's not easy to implement. > Just an idea, but maybe worth to explore further instead of tweaking > more and more bits to make the virtual system somehow act sane. > Sure, and it it's interesting indeed, for a bunch or reasons and purposes (as Tim is also noting). Not so much --or at least not necessarily-- for this one, IMO. Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 13:23 ` Dario Faggioli @ 2015-07-27 14:02 ` Juergen Gross 0 siblings, 0 replies; 95+ messages in thread From: Juergen Gross @ 2015-07-27 14:02 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, George Dunlap, Andrew Cooper, George Dunlap, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/27/2015 03:23 PM, Dario Faggioli wrote: > On Mon, 2015-07-27 at 14:01 +0200, Juergen Gross wrote: >> On 07/27/2015 01:11 PM, George Dunlap wrote: > >>> Or alternately, if the user wants to give up on the "consolidation" >>> aspect of virtualization, they can pin vcpus to pcpus and then pass in >>> the actual host topology (hyperthreads and all). >> >> There would be another solution, of course: >> >> Support hyperthreads in the Xen scheduler via gang scheduling. While >> this is not a simple solution, it is a fair one. Hyperthreads on one >> core can influence each other rather much. With both threads always >> running vcpus of the same guest the penalty/advantage would stay in the >> same domain. The guest could make really sensible scheduling decisions >> and the licensing would still work as desired. >> > This is interesting indeed, but I much rather see it as something > orthogonal, which may indeed bring benefits in some of the scenarios > described here, but should not be considered *the* solution. Correct. I still think it should be done. > Implementing, enabling and asking users to use something like this will > impact the system behavior and performance, in ways that may not be > desirable for all use cases. I'd make it a scheduler parameter. So you could it enable for a specific cpupool where you want it to be active. > So, while I do think that this may be something nice to have and offer, > trying to use it for solving the problem we're debating here would make > things even more complex to configure. > > Also, this would take care of HT related issues, but what about cores > (as in 'should vcpus be cores of sockets or full sockets') and !HT boxes > (like AMD)? !HT boxes will have no problem: We won't have to hide HT as cores... Regarding many sockets with 1 core each or 1 socket with many cores: I think 1 socket for the non-NUMA case is okay, we'll want multiple sockets for NUMA. > Not to mention, as you say yourself, that it's not easy to implement. Yeah, but it will be fun. ;-) > >> Just an idea, but maybe worth to explore further instead of tweaking >> more and more bits to make the virtual system somehow act sane. >> > Sure, and it it's interesting indeed, for a bunch or reasons and > purposes (as Tim is also noting). Not so much --or at least not > necessarily-- for this one, IMO. It's especially interesting regarding accounting. A vcpu running for 1 second can do much more if no other vcpu is running on the same core. This would be a problem of the guest then, like on bare metal. For real time purposes it might be even interesting to schedule only 1 vcpu per core to have a reliable high speed of the vcpu. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 11:11 ` George Dunlap 2015-07-27 12:01 ` Juergen Gross @ 2015-07-27 14:02 ` Dario Faggioli 1 sibling, 0 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-27 14:02 UTC (permalink / raw) To: George Dunlap Cc: Juergen Gross, Elena Ufimtseva, Wei Liu, George Dunlap, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 3372 bytes --] On Mon, 2015-07-27 at 12:11 +0100, George Dunlap wrote: > 1. Userspace applications are in the habit of reading CPUID to determine > the topology of the system they're running on > I'd add this item here: 1b. Linux kernel uses CPUID to configure some bits of its scheduler. The result of that, will affect *all* the in-guest application that does not use CPUID for scheduling/performance purposes Yes, I saw you mention this afterwards, but I think it really should be here, in the analysis part. > 2. Many use the topology information to help themselves make better > scheduling decisions. Because a vcpu is not typically pinned to a > specific pcpu, we may need to lie here slightly (e.g., not mention > threads) to get the optimal behavior overall. > > 3. Others use the topology information to implement licensing > restrictions. Because threads are treated differently to cores, we want > to tell the truth here (i.e., make sure we mention that some of these > are threads) to get the optimal behavior overall. > Define truth. I mean, is 'telling the truth' always good for this case? E.g., in a 4 vcpus guest, on a 4 socket box, without any pinning, it may be that when app X samples CPUID to check the license, each vcpu is running on a different socket, so the truth means we need a licence for 4 sockets... I don't think this is ideal, is it? Anyways... > Numbers #2 and #3 lead to contradictory courses of action; we cannot > optimize for both at the same time. > ...that's certainly true, IMO. > I think at some level we need to just try to accommodate both -- if the > user doesn't have licensing issues, or prefers performance over > licensing, then present a unified topology in PVH / HVM using CPUID, > ACPI, &c. I think this should be the default. > > If the user has licensing issues, and doesn't mind having wonky or > unreliable topology to its guests, then let the raw CPUID through. But > it would, in this case, be good to try to give the guest OS scheduler a > hint that it shouldn't really bother trying to read the topology or do > placement as a result, as any decisions will be unreliable. > This last sentence is basically Juergen proposal, AFAICT, and I agree that it would be good in case #3. But, thinking more about it, would it harm in case #2? I don't think it would. After all, it'd boil down to make peace with the fact that something like CPUID, in a (not only PV!) VM, is just not reliable enough to use it for building in-kernel scheduling related data structure, like Linux's scheduling domains. It is unreliable because its content may conflict with vNUMA, but, really, even with no vNUMA, it is unreliable because it depends on whether the VM's vcpus are pinned or not, and if not, it depends on where they actually run, which is pure randomness, from the guest point of view. So, I'm really starting to think that a patch stopping the Linux kernel relying on CPUID, whether original or mangled, and for all kind of guests, would really be a nice one to have! Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:09 ` Konrad Rzeszutek Wilk 2015-07-24 16:14 ` Dario Faggioli 2015-07-24 16:18 ` Juergen Gross @ 2015-07-27 10:41 ` George Dunlap 2015-07-27 10:49 ` Andrew Cooper 2 siblings, 1 reply; 95+ messages in thread From: George Dunlap @ 2015-07-27 10:41 UTC (permalink / raw) To: Konrad Rzeszutek Wilk Cc: Juergen Gross, Elena Ufimtseva, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On Fri, Jul 24, 2015 at 5:09 PM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: > On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote: >> On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote: >> > On 07/24/2015 05:14 PM, Juergen Gross wrote: >> > > On 07/24/2015 04:44 PM, Dario Faggioli wrote: >> >> > >> In fact, I think that it is the topology, i.e., what comes from MSRs, >> > >> that needs to adapt, and follow vNUMA, as much as possible. Do we agree >> > >> on this? >> > > >> > > I think we have to be very careful here. I see two possible scenarios: >> > > >> > > 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will >> > > try to schedule the vcpus according to their numa affinity. So they >> > > can change pcpus at any time in case of very busy guests. I don't >> > > think the linux kernel should treat the cpus differently in this >> > > case as it will be in vane regarding the Xen scheduler's activity. >> > > So we should use the "null" topology in this case. >> > >> > Sorry, the topology should reflect the vcpu<->numa-node relations, of >> > course, but nothing else (so flat topolgy in each numa node). >> > >> Yeah, I was replying to this point saying something like this right >> now... Luckily, I've seen this email! :-P >> >> With this semantic, I fully agree with this. >> >> > > 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen >> > > scheduler can't move vcpus between pcpus, so the linux kernel should >> > > see the real topology of the used pcpus in order to optimize for this >> > > picture. >> > > >> > >> Mmm... I did think about this too, but I'm not sure. I see the value of >> this of course, and the reason why it makes sense. However, pinning can >> change on-line, via `xl vcpu-pin' and stuff. Also migration could make >> things less certain, I think. What happens if we build on top of the >> initial pinning, and then things change? >> >> To be fair, there is stuff building on top of the initial pinning >> already, e.g., from which physical NUMA node we allocate the memory >> relies depends exactly on that. That being said, I'm not sure I'm >> comfortable with adding more of this... >> >> Perhaps introduce an 'immutable_pinning' flag, which will prevent >> affinity to be changed, and then bind the topology to pinning only if >> that one is set? >> >> > >> Maybe, there is room for "fixing" this at this level, hooking up inside >> > >> the scheduler code... but I'm shooting in the dark, without having check >> > >> whether and how this could be really feasible, should I? >> > > >> > > Uuh, I don't think a change of the scheduler on behalf of Xen is really >> > > appreciated. :-) >> > > >> I'm sure it would (have been! :-)) a true and giant nightmare!! :-D >> >> > >> One thing I don't like about this approach is that it would potentially >> > >> solve vNUMA and other scheduling anomalies, but... >> > >> >> > >>> cpuid instruction is available for user mode as well. >> > >>> >> > >> ...it would not do any good for other subsystems, and user level code >> > >> and apps. >> > > >> > > Indeed. I think the optimal solution would be two-fold: give the >> > > scheduler the information it is needing to react correctly via a >> > > kernel patch not relying on cpuid values and fiddle with the cpuid >> > > values from xen tools according to any needs of other subsystems and/or >> > > user code (e.g. licensing). >> > >> So, just to check if I'm understanding is correct: you'd like to add an >> abstraction layer, in Linux, like in generic (or, perhaps, scheduling) >> code, to hide the direct interaction with CPUID. >> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd >> check with Xen/match vNUMA/whatever... Is this that you are saying? >> >> If yes, I think I like it... > > I don't think this is workable. For example there are applications > which use 'cpuid' and figure out the core/thread and use it for its own > scheduling purposes. Can you expand a little on this? I'm having trouble figuring out exactly what user-space applications are reading and how they're using it -- and, how they work currently in virtual environments, given that they (typically) will be moved between physical processors even if they stay on the same virtual processor. -George ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 10:41 ` George Dunlap @ 2015-07-27 10:49 ` Andrew Cooper 2015-07-27 13:11 ` Dario Faggioli 0 siblings, 1 reply; 95+ messages in thread From: Andrew Cooper @ 2015-07-27 10:49 UTC (permalink / raw) To: George Dunlap, Konrad Rzeszutek Wilk Cc: Juergen Gross, Elena Ufimtseva, Wei Liu, Dario Faggioli, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 27/07/15 11:41, George Dunlap wrote: > On Fri, Jul 24, 2015 at 5:09 PM, Konrad Rzeszutek Wilk > <konrad.wilk@oracle.com> wrote: >> On Fri, Jul 24, 2015 at 05:58:29PM +0200, Dario Faggioli wrote: >>> On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote: >>>> On 07/24/2015 05:14 PM, Juergen Gross wrote: >>>>> On 07/24/2015 04:44 PM, Dario Faggioli wrote: >>>>>> In fact, I think that it is the topology, i.e., what comes from MSRs, >>>>>> that needs to adapt, and follow vNUMA, as much as possible. Do we agree >>>>>> on this? >>>>> I think we have to be very careful here. I see two possible scenarios: >>>>> >>>>> 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will >>>>> try to schedule the vcpus according to their numa affinity. So they >>>>> can change pcpus at any time in case of very busy guests. I don't >>>>> think the linux kernel should treat the cpus differently in this >>>>> case as it will be in vane regarding the Xen scheduler's activity. >>>>> So we should use the "null" topology in this case. >>>> Sorry, the topology should reflect the vcpu<->numa-node relations, of >>>> course, but nothing else (so flat topolgy in each numa node). >>>> >>> Yeah, I was replying to this point saying something like this right >>> now... Luckily, I've seen this email! :-P >>> >>> With this semantic, I fully agree with this. >>> >>>>> 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen >>>>> scheduler can't move vcpus between pcpus, so the linux kernel should >>>>> see the real topology of the used pcpus in order to optimize for this >>>>> picture. >>>>> >>> Mmm... I did think about this too, but I'm not sure. I see the value of >>> this of course, and the reason why it makes sense. However, pinning can >>> change on-line, via `xl vcpu-pin' and stuff. Also migration could make >>> things less certain, I think. What happens if we build on top of the >>> initial pinning, and then things change? >>> >>> To be fair, there is stuff building on top of the initial pinning >>> already, e.g., from which physical NUMA node we allocate the memory >>> relies depends exactly on that. That being said, I'm not sure I'm >>> comfortable with adding more of this... >>> >>> Perhaps introduce an 'immutable_pinning' flag, which will prevent >>> affinity to be changed, and then bind the topology to pinning only if >>> that one is set? >>> >>>>>> Maybe, there is room for "fixing" this at this level, hooking up inside >>>>>> the scheduler code... but I'm shooting in the dark, without having check >>>>>> whether and how this could be really feasible, should I? >>>>> Uuh, I don't think a change of the scheduler on behalf of Xen is really >>>>> appreciated. :-) >>>>> >>> I'm sure it would (have been! :-)) a true and giant nightmare!! :-D >>> >>>>>> One thing I don't like about this approach is that it would potentially >>>>>> solve vNUMA and other scheduling anomalies, but... >>>>>> >>>>>>> cpuid instruction is available for user mode as well. >>>>>>> >>>>>> ...it would not do any good for other subsystems, and user level code >>>>>> and apps. >>>>> Indeed. I think the optimal solution would be two-fold: give the >>>>> scheduler the information it is needing to react correctly via a >>>>> kernel patch not relying on cpuid values and fiddle with the cpuid >>>>> values from xen tools according to any needs of other subsystems and/or >>>>> user code (e.g. licensing). >>> So, just to check if I'm understanding is correct: you'd like to add an >>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling) >>> code, to hide the direct interaction with CPUID. >>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd >>> check with Xen/match vNUMA/whatever... Is this that you are saying? >>> >>> If yes, I think I like it... >> I don't think this is workable. For example there are applications >> which use 'cpuid' and figure out the core/thread and use it for its own >> scheduling purposes. > Can you expand a little on this? I'm having trouble figuring out > exactly what user-space applications are reading and how they're using > it -- and, how they work currently in virtual environments, given that > they (typically) will be moved between physical processors even if > they stay on the same virtual processor. There are many examples of userspace application using cpuid to modify themselves. Any serious application with processor optimisations will use the cpuid feature bits to choose the most efficient algorithm. hwloc is an perfect example which gathers all of the topology information out of cpuid to work out how to most efficiently pin/schedule tasks. As to how the work, that is very much an open question. "function" might be a better term. ~Andrew ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 10:49 ` Andrew Cooper @ 2015-07-27 13:11 ` Dario Faggioli 0 siblings, 0 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-27 13:11 UTC (permalink / raw) To: Andrew Cooper Cc: Juergen Gross, Elena Ufimtseva, Wei Liu, George Dunlap, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 1488 bytes --] On Mon, 2015-07-27 at 11:49 +0100, Andrew Cooper wrote: > On 27/07/15 11:41, George Dunlap wrote: > > Can you expand a little on this? I'm having trouble figuring out > > exactly what user-space applications are reading and how they're using > > it -- and, how they work currently in virtual environments, given that > > they (typically) will be moved between physical processors even if > > they stay on the same virtual processor. > > There are many examples of userspace application using cpuid to modify > themselves. Any serious application with processor optimisations will > use the cpuid feature bits to choose the most efficient algorithm. > > hwloc is an perfect example which gathers all of the topology > information out of cpuid to work out how to most efficiently > pin/schedule tasks. > And all of this is broken, right now, isn't it? Where, saying this, I'm aiming at stressing the point (as the thread is starting to become a bit hard to follow) that we must come up with a decent solution, working reasonably well in a bunch of (conflicting! *sigh*) scenarios, *not* to implement something new without breaking what we have, as what we have is broken already! Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 15:58 ` Dario Faggioli 2015-07-24 16:09 ` Konrad Rzeszutek Wilk @ 2015-07-24 16:10 ` Juergen Gross 2015-07-24 16:40 ` Boris Ostrovsky 2015-07-27 14:09 ` Dario Faggioli 1 sibling, 2 replies; 95+ messages in thread From: Juergen Gross @ 2015-07-24 16:10 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/24/2015 05:58 PM, Dario Faggioli wrote: > On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote: >> On 07/24/2015 05:14 PM, Juergen Gross wrote: >>> On 07/24/2015 04:44 PM, Dario Faggioli wrote: > >>>> In fact, I think that it is the topology, i.e., what comes from MSRs, >>>> that needs to adapt, and follow vNUMA, as much as possible. Do we agree >>>> on this? >>> >>> I think we have to be very careful here. I see two possible scenarios: >>> >>> 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will >>> try to schedule the vcpus according to their numa affinity. So they >>> can change pcpus at any time in case of very busy guests. I don't >>> think the linux kernel should treat the cpus differently in this >>> case as it will be in vane regarding the Xen scheduler's activity. >>> So we should use the "null" topology in this case. >> >> Sorry, the topology should reflect the vcpu<->numa-node relations, of >> course, but nothing else (so flat topolgy in each numa node). >> > Yeah, I was replying to this point saying something like this right > now... Luckily, I've seen this email! :-P > > With this semantic, I fully agree with this. > >>> 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen >>> scheduler can't move vcpus between pcpus, so the linux kernel should >>> see the real topology of the used pcpus in order to optimize for this >>> picture. >>> >> > Mmm... I did think about this too, but I'm not sure. I see the value of > this of course, and the reason why it makes sense. However, pinning can > change on-line, via `xl vcpu-pin' and stuff. Also migration could make > things less certain, I think. What happens if we build on top of the > initial pinning, and then things change? If we can fiddle with the masks on boot, we could do it in a running system, too. Another advantage with not relying on cpuid. :-) > To be fair, there is stuff building on top of the initial pinning > already, e.g., from which physical NUMA node we allocate the memory > relies depends exactly on that. That being said, I'm not sure I'm > comfortable with adding more of this... > > Perhaps introduce an 'immutable_pinning' flag, which will prevent > affinity to be changed, and then bind the topology to pinning only if > that one is set? Hmm, this would require disabling migration as well. Or enable it with "--force" only. > >>>> Maybe, there is room for "fixing" this at this level, hooking up inside >>>> the scheduler code... but I'm shooting in the dark, without having check >>>> whether and how this could be really feasible, should I? >>> >>> Uuh, I don't think a change of the scheduler on behalf of Xen is really >>> appreciated. :-) >>> > I'm sure it would (have been! :-)) a true and giant nightmare!! :-D > >>>> One thing I don't like about this approach is that it would potentially >>>> solve vNUMA and other scheduling anomalies, but... >>>> >>>>> cpuid instruction is available for user mode as well. >>>>> >>>> ...it would not do any good for other subsystems, and user level code >>>> and apps. >>> >>> Indeed. I think the optimal solution would be two-fold: give the >>> scheduler the information it is needing to react correctly via a >>> kernel patch not relying on cpuid values and fiddle with the cpuid >>> values from xen tools according to any needs of other subsystems and/or >>> user code (e.g. licensing). >> > So, just to check if I'm understanding is correct: you'd like to add an > abstraction layer, in Linux, like in generic (or, perhaps, scheduling) > code, to hide the direct interaction with CPUID. > Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd > check with Xen/match vNUMA/whatever... Is this that you are saying? Sort of, yes. I just wouldn't add it, as it is already existing (more or less). It can deal right now with AMD and Intel, we would "just" have to add Xen. > > If yes, I think I like it... I hope e.g. the KVM guys will like it, too. :-) Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:10 ` Juergen Gross @ 2015-07-24 16:40 ` Boris Ostrovsky 2015-07-24 16:48 ` Juergen Gross 2015-07-27 4:24 ` Juergen Gross 2015-07-27 14:09 ` Dario Faggioli 1 sibling, 2 replies; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-24 16:40 UTC (permalink / raw) To: Juergen Gross, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/24/2015 12:10 PM, Juergen Gross wrote: > > If we can fiddle with the masks on boot, we could do it in a running > system, too. Another advantage with not relying on cpuid. :-) I am trying to catch up with this thread so I may have missed it, but I still don't understand why we don't want to rely on CPUID. I think I saw Juergen said --- because it's HW-specific. But what's wrong with that? Hypervisor is building virtualized x86 (in this case) hardware and on such HW CPUID is the standard way of determining thread/core topology. Plus various ACPI tables and such. And having a solution that doesn't address userspace (when there *is* a solution that can do it) doesn't seem like the best approach. Yes, it still won't cover userspace for PV guests but neither will the kernel patch. As far as licensing is concerned --- are we sure this can't also be addressed by CPUID? BTW, if I was asked about who is most concerned about licensing my first answer would be --- databases. I.e. userspace. (Also, I don't know whether this is still true but in the past APICID format was also used for topology discovery. Just to make things a bit more interesting ;-)) -boris ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:40 ` Boris Ostrovsky @ 2015-07-24 16:48 ` Juergen Gross 2015-07-24 17:11 ` Boris Ostrovsky 2015-07-27 4:24 ` Juergen Gross 1 sibling, 1 reply; 95+ messages in thread From: Juergen Gross @ 2015-07-24 16:48 UTC (permalink / raw) To: Boris Ostrovsky, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/24/2015 06:40 PM, Boris Ostrovsky wrote: > On 07/24/2015 12:10 PM, Juergen Gross wrote: >> >> If we can fiddle with the masks on boot, we could do it in a running >> system, too. Another advantage with not relying on cpuid. :-) > > > I am trying to catch up with this thread so I may have missed it, but I > still don't understand why we don't want to rely on CPUID. > > I think I saw Juergen said --- because it's HW-specific. But what's > wrong with that? Hypervisor is building virtualized x86 (in this case) > hardware and on such HW CPUID is the standard way of determining > thread/core topology. Plus various ACPI tables and such. > > And having a solution that doesn't address userspace (when there *is* a > solution that can do it) doesn't seem like the best approach. Yes, it > still won't cover userspace for PV guests but neither will the kernel > patch. > > As far as licensing is concerned --- are we sure this can't also be > addressed by CPUID? BTW, if I was asked about who is most concerned > about licensing my first answer would be --- databases. I.e. userspace. The problem is to construct cpuids which will enable the linux scheduler to work correct in spite of the hypervisor scheduler moving vcpus between pcpus. The only way to do this is to emulate single-threaded cores on the numa nodes without further grouping. So either single-core sockets or one socket with many cores. This might be problematic for licensing: the multi-socket solution might require a higher license based on socket numbers. Or the license is based on cores and will be more expensive as no hyperthreads are detectable. > (Also, I don't know whether this is still true but in the past APICID > format was also used for topology discovery. Just to make things a bit > more interesting ;-)) Another +1 for the pv-solution. :-) Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:48 ` Juergen Gross @ 2015-07-24 17:11 ` Boris Ostrovsky 2015-07-27 13:40 ` Dario Faggioli 0 siblings, 1 reply; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-24 17:11 UTC (permalink / raw) To: Juergen Gross, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/24/2015 12:48 PM, Juergen Gross wrote: > On 07/24/2015 06:40 PM, Boris Ostrovsky wrote: >> On 07/24/2015 12:10 PM, Juergen Gross wrote: >>> >>> If we can fiddle with the masks on boot, we could do it in a running >>> system, too. Another advantage with not relying on cpuid. :-) >> >> >> I am trying to catch up with this thread so I may have missed it, but I >> still don't understand why we don't want to rely on CPUID. >> >> I think I saw Juergen said --- because it's HW-specific. But what's >> wrong with that? Hypervisor is building virtualized x86 (in this case) >> hardware and on such HW CPUID is the standard way of determining >> thread/core topology. Plus various ACPI tables and such. >> >> And having a solution that doesn't address userspace (when there *is* a >> solution that can do it) doesn't seem like the best approach. Yes, it >> still won't cover userspace for PV guests but neither will the kernel >> patch. >> >> As far as licensing is concerned --- are we sure this can't also be >> addressed by CPUID? BTW, if I was asked about who is most concerned >> about licensing my first answer would be --- databases. I.e. userspace. > > The problem is to construct cpuids which will enable the linux > scheduler to work correct in spite of the hypervisor scheduler > moving vcpus between pcpus. The only way to do this is to emulate > single-threaded cores on the numa nodes without further grouping. > So either single-core sockets or one socket with many cores. Right. > > This might be problematic for licensing: the multi-socket solution > might require a higher license based on socket numbers. Or the > license is based on cores and will be more expensive as no hyperthreads > are detectable. If we don't pin VCPUs approriately (which I think is the scenario that we are discussing) then CPUID can be used for find out package ID. And so any licensed SW will easily discover that it is running on different packages. > >> (Also, I don't know whether this is still true but in the past APICID >> format was also used for topology discovery. Just to make things a bit >> more interesting ;-)) > > Another +1 for the pv-solution. :-) Not necessarily. ACPI/CPUID changes can take care of that outside kernel. BTW, this can be useful (in case you don't already have it): https://software.intel.com/en-us/articles/intel-64-architecture-processor-topology-enumeration There is a similar document for AMD but I don't remember where it lives. -boris "the CPUID bigot" ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 17:11 ` Boris Ostrovsky @ 2015-07-27 13:40 ` Dario Faggioli 0 siblings, 0 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-27 13:40 UTC (permalink / raw) To: Boris Ostrovsky Cc: Juergen Gross, Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 3869 bytes --] On Fri, 2015-07-24 at 13:11 -0400, Boris Ostrovsky wrote: > On 07/24/2015 12:48 PM, Juergen Gross wrote: > > On 07/24/2015 06:40 PM, Boris Ostrovsky wrote: > >> On 07/24/2015 12:10 PM, Juergen Gross wrote: > >>> > >>> If we can fiddle with the masks on boot, we could do it in a running > >>> system, too. Another advantage with not relying on cpuid. :-) > >> > >> > >> I am trying to catch up with this thread so I may have missed it, but I > >> still don't understand why we don't want to rely on CPUID. > >> > >> I think I saw Juergen said --- because it's HW-specific. But what's > >> wrong with that? Hypervisor is building virtualized x86 (in this case) > >> hardware and on such HW CPUID is the standard way of determining > >> thread/core topology. Plus various ACPI tables and such. > >> > >> And having a solution that doesn't address userspace (when there *is* a > >> solution that can do it) doesn't seem like the best approach. Yes, it > >> still won't cover userspace for PV guests but neither will the kernel > >> patch. > >> > >> As far as licensing is concerned --- are we sure this can't also be > >> addressed by CPUID? BTW, if I was asked about who is most concerned > >> about licensing my first answer would be --- databases. I.e. userspace. > > > > The problem is to construct cpuids which will enable the linux > > scheduler to work correct in spite of the hypervisor scheduler > > moving vcpus between pcpus. The only way to do this is to emulate > > single-threaded cores on the numa nodes without further grouping. > > So either single-core sockets or one socket with many cores. > > Right. > So you see it now? If adhering to some guest virtual topology (for whatever reason, e.g., performance) and licensing disagree, using CPUID for both is always going to fail! > > This might be problematic for licensing: the multi-socket solution > > might require a higher license based on socket numbers. Or the > > license is based on cores and will be more expensive as no hyperthreads > > are detectable. > > If we don't pin VCPUs approriately (which I think is the scenario that > we are discussing) then CPUID can be used for find out package ID. And > so any licensed SW will easily discover that it is running on different > packages. > That's why Juergen is suggesting to keep the things separate, effectively decoupling them, AFAICU his proposal. Note that I'm still a bit puzzled by the idea of presenting different information to the guest OS and to the guest userspace, but that has upsides, and this decoupling is one. In fact, in the case you're describing, i.e., not pinned vcpus, etc: - the Linux kernel is not relying on CPUID when building scheduling domains, and everything so, at least all the user space applications that does not call CPUID and/or rely on that for scheduling/performance matters *will* *be* *fine* - you can pass down, via tools, a mangled CPUID, e.g., making it best fit your licensing needs. Problems arises in case you have both the following kind of applications (or the same application doing both the following operations): a) applications that poke at CPUID for licensing purposes b) applications that poke at CPUID for placement/performance purposes In this case, it's well possible that mangling CPUID for making a) happy, will make b) unhappy, and vice-versa. And the fact that the kernel does not rely on CPUID any longer may not help, as the application is kind of bypassing it... Although, it won't harm either... Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:40 ` Boris Ostrovsky 2015-07-24 16:48 ` Juergen Gross @ 2015-07-27 4:24 ` Juergen Gross 1 sibling, 0 replies; 95+ messages in thread From: Juergen Gross @ 2015-07-27 4:24 UTC (permalink / raw) To: Boris Ostrovsky, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/24/2015 06:40 PM, Boris Ostrovsky wrote: > On 07/24/2015 12:10 PM, Juergen Gross wrote: >> >> If we can fiddle with the masks on boot, we could do it in a running >> system, too. Another advantage with not relying on cpuid. :-) > > > I am trying to catch up with this thread so I may have missed it, but I > still don't understand why we don't want to rely on CPUID. > > I think I saw Juergen said --- because it's HW-specific. But what's > wrong with that? Hypervisor is building virtualized x86 (in this case) > hardware and on such HW CPUID is the standard way of determining > thread/core topology. Plus various ACPI tables and such. I didn't mean specific to x86, but even different between AMD and Intel. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:10 ` Juergen Gross 2015-07-24 16:40 ` Boris Ostrovsky @ 2015-07-27 14:09 ` Dario Faggioli 2015-07-27 14:34 ` Boris Ostrovsky 2015-07-28 4:29 ` Juergen Gross 1 sibling, 2 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-27 14:09 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 1374 bytes --] On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote: > On 07/24/2015 05:58 PM, Dario Faggioli wrote: > > So, just to check if I'm understanding is correct: you'd like to add an > > abstraction layer, in Linux, like in generic (or, perhaps, scheduling) > > code, to hide the direct interaction with CPUID. > > Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd > > check with Xen/match vNUMA/whatever... Is this that you are saying? > > Sort of, yes. > > I just wouldn't add it, as it is already existing (more or less). It > can deal right now with AMD and Intel, we would "just" have to add Xen. > So, having gone through the rest of the thread (so far), and having given a fair amount o thinking to this, I really think that something like this would be a good thing to have in Linux. Of course, it's not that my opinion on where should be in Linux counts that much! :-D Nevertheless, I wanted to make it clear that, while skeptic at the beginning, I now think this is (part of) the way to go, as I said and explained in my reply to George. Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 14:09 ` Dario Faggioli @ 2015-07-27 14:34 ` Boris Ostrovsky 2015-07-27 14:43 ` Juergen Gross ` (2 more replies) 2015-07-28 4:29 ` Juergen Gross 1 sibling, 3 replies; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-27 14:34 UTC (permalink / raw) To: Dario Faggioli, Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/27/2015 10:09 AM, Dario Faggioli wrote: > On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote: >> On 07/24/2015 05:58 PM, Dario Faggioli wrote: >>> So, just to check if I'm understanding is correct: you'd like to add an >>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling) >>> code, to hide the direct interaction with CPUID. >>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd >>> check with Xen/match vNUMA/whatever... Is this that you are saying? >> Sort of, yes. >> >> I just wouldn't add it, as it is already existing (more or less). It >> can deal right now with AMD and Intel, we would "just" have to add Xen. >> > So, having gone through the rest of the thread (so far), and having > given a fair amount o thinking to this, I really think that something > like this would be a good thing to have in Linux. > > Of course, it's not that my opinion on where should be in Linux counts > that much! :-D Nevertheless, I wanted to make it clear that, while > skeptic at the beginning, I now think this is (part of) the way to go, > as I said and explained in my reply to George. And I continue to believe that kernel solution does not address the userland problem which is no less important than making kernel do proper scheduling decisions (and I suspect when this patch goes for review that's what the scheduling people are going to say). Remember the original problem that started this thread was that kernel complained that topology didn't make sense and it turned off all topology-related decisions. Which means that kernel already has a solution for weird topology. Some enumeration doesn't trigger this warning, but we can come up with one that does. Or we can indeed have a patch in kernel that will, possibly silently, fail topology_sane() when virtualized and not pinned. (This is what I assume kernel does when topology_sane() fails. And if it doesn't, that's a bug IMO) The licensing problem that Juergen described can be solved by pining vcpus and exposing HT bit. Besides, creating a guest with 24 VPCUs and hoping that 16-core licensing will work I think is pushing it a bit when you know that VCPUs will jump around cores (i.e. "on average" you are running on more than 16 cores -- multi-threaded or not -- which arguably is what licensing is trying to prevent) -boris ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 14:34 ` Boris Ostrovsky @ 2015-07-27 14:43 ` Juergen Gross 2015-07-27 14:51 ` Boris Ostrovsky 2015-07-27 14:47 ` Juergen Gross 2015-07-27 14:58 ` Dario Faggioli 2 siblings, 1 reply; 95+ messages in thread From: Juergen Gross @ 2015-07-27 14:43 UTC (permalink / raw) To: Boris Ostrovsky, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/27/2015 04:34 PM, Boris Ostrovsky wrote: > On 07/27/2015 10:09 AM, Dario Faggioli wrote: >> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote: >>> On 07/24/2015 05:58 PM, Dario Faggioli wrote: >>>> So, just to check if I'm understanding is correct: you'd like to add an >>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling) >>>> code, to hide the direct interaction with CPUID. >>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd >>>> check with Xen/match vNUMA/whatever... Is this that you are saying? >>> Sort of, yes. >>> >>> I just wouldn't add it, as it is already existing (more or less). It >>> can deal right now with AMD and Intel, we would "just" have to add Xen. >>> >> So, having gone through the rest of the thread (so far), and having >> given a fair amount o thinking to this, I really think that something >> like this would be a good thing to have in Linux. >> >> Of course, it's not that my opinion on where should be in Linux counts >> that much! :-D Nevertheless, I wanted to make it clear that, while >> skeptic at the beginning, I now think this is (part of) the way to go, >> as I said and explained in my reply to George. > > And I continue to believe that kernel solution does not address the > userland problem which is no less important than making kernel do proper > scheduling decisions (and I suspect when this patch goes for review > that's what the scheduling people are going to say). > > Remember the original problem that started this thread was that kernel > complained that topology didn't make sense and it turned off all > topology-related decisions. Which means that kernel already has a > solution for weird topology. Some enumeration doesn't trigger this > warning, but we can come up with one that does. Or we can indeed have a > patch in kernel that will, possibly silently, fail topology_sane() when > virtualized and not pinned. How would you come up with a topology the kernel is complaining about and user mode scheduling will use for sane decisions ? > (This is what I assume kernel does when topology_sane() fails. And if it > doesn't, that's a bug IMO) > > The licensing problem that Juergen described can be solved by pining > vcpus and exposing HT bit. Besides, creating a guest with 24 VPCUs and Hmm, yes. This way you sacrifice most of the virtualization advantages. > hoping that 16-core licensing will work I think is pushing it a bit when > you know that VCPUs will jump around cores (i.e. "on average" you are > running on more than 16 cores -- multi-threaded or not -- which arguably > is what licensing is trying to prevent) On a machine with only 16 cores running on more than 16 cores? I have some problems to believe this. The point was: if the license is happy on bare metal it should be so when running on the same hardware as a guest. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 14:43 ` Juergen Gross @ 2015-07-27 14:51 ` Boris Ostrovsky 2015-07-27 15:03 ` Juergen Gross 0 siblings, 1 reply; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-27 14:51 UTC (permalink / raw) To: Juergen Gross, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/27/2015 10:43 AM, Juergen Gross wrote: > On 07/27/2015 04:34 PM, Boris Ostrovsky wrote: >> On 07/27/2015 10:09 AM, Dario Faggioli wrote: >>> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote: >>>> On 07/24/2015 05:58 PM, Dario Faggioli wrote: >>>>> So, just to check if I'm understanding is correct: you'd like to >>>>> add an >>>>> abstraction layer, in Linux, like in generic (or, perhaps, >>>>> scheduling) >>>>> code, to hide the direct interaction with CPUID. >>>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, >>>>> it'd >>>>> check with Xen/match vNUMA/whatever... Is this that you are saying? >>>> Sort of, yes. >>>> >>>> I just wouldn't add it, as it is already existing (more or less). It >>>> can deal right now with AMD and Intel, we would "just" have to add >>>> Xen. >>>> >>> So, having gone through the rest of the thread (so far), and having >>> given a fair amount o thinking to this, I really think that something >>> like this would be a good thing to have in Linux. >>> >>> Of course, it's not that my opinion on where should be in Linux counts >>> that much! :-D Nevertheless, I wanted to make it clear that, while >>> skeptic at the beginning, I now think this is (part of) the way to go, >>> as I said and explained in my reply to George. >> >> And I continue to believe that kernel solution does not address the >> userland problem which is no less important than making kernel do proper >> scheduling decisions (and I suspect when this patch goes for review >> that's what the scheduling people are going to say). >> >> Remember the original problem that started this thread was that kernel >> complained that topology didn't make sense and it turned off all >> topology-related decisions. Which means that kernel already has a >> solution for weird topology. Some enumeration doesn't trigger this >> warning, but we can come up with one that does. Or we can indeed have a >> patch in kernel that will, possibly silently, fail topology_sane() when >> virtualized and not pinned. > > How would you come up with a topology the kernel is complaining about > and user mode scheduling will use for sane decisions ? We need to understand first why Dario's box is apparently the only one resulting in a warning and probably then emulate that enumeration. And again, if that is not possible then just make topology_sane() fail. > >> (This is what I assume kernel does when topology_sane() fails. And if it >> doesn't, that's a bug IMO) >> >> The licensing problem that Juergen described can be solved by pining >> vcpus and exposing HT bit. Besides, creating a guest with 24 VPCUs and > > Hmm, yes. This way you sacrifice most of the virtualization advantages. > >> hoping that 16-core licensing will work I think is pushing it a bit when >> you know that VCPUs will jump around cores (i.e. "on average" you are >> running on more than 16 cores -- multi-threaded or not -- which arguably >> is what licensing is trying to prevent) > > On a machine with only 16 cores running on more than 16 cores? I have > some problems to believe this. The point was: if the license is happy on > bare metal it should be so when running on the same hardware as a guest. Ok, that's not how I should have described it. I meant that IMO asking for 24 VCPUs is somewhat akin to oversubscribing since you kind of know that you dont' have 24 PCPUs, you are just trying to fool the kernel into thinking that threads are cores. -boris ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 14:51 ` Boris Ostrovsky @ 2015-07-27 15:03 ` Juergen Gross 0 siblings, 0 replies; 95+ messages in thread From: Juergen Gross @ 2015-07-27 15:03 UTC (permalink / raw) To: Boris Ostrovsky, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/27/2015 04:51 PM, Boris Ostrovsky wrote: > On 07/27/2015 10:43 AM, Juergen Gross wrote: >> On 07/27/2015 04:34 PM, Boris Ostrovsky wrote: >>> On 07/27/2015 10:09 AM, Dario Faggioli wrote: >>>> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote: >>>>> On 07/24/2015 05:58 PM, Dario Faggioli wrote: >>>>>> So, just to check if I'm understanding is correct: you'd like to >>>>>> add an >>>>>> abstraction layer, in Linux, like in generic (or, perhaps, >>>>>> scheduling) >>>>>> code, to hide the direct interaction with CPUID. >>>>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, >>>>>> it'd >>>>>> check with Xen/match vNUMA/whatever... Is this that you are saying? >>>>> Sort of, yes. >>>>> >>>>> I just wouldn't add it, as it is already existing (more or less). It >>>>> can deal right now with AMD and Intel, we would "just" have to add >>>>> Xen. >>>>> >>>> So, having gone through the rest of the thread (so far), and having >>>> given a fair amount o thinking to this, I really think that something >>>> like this would be a good thing to have in Linux. >>>> >>>> Of course, it's not that my opinion on where should be in Linux counts >>>> that much! :-D Nevertheless, I wanted to make it clear that, while >>>> skeptic at the beginning, I now think this is (part of) the way to go, >>>> as I said and explained in my reply to George. >>> >>> And I continue to believe that kernel solution does not address the >>> userland problem which is no less important than making kernel do proper >>> scheduling decisions (and I suspect when this patch goes for review >>> that's what the scheduling people are going to say). >>> >>> Remember the original problem that started this thread was that kernel >>> complained that topology didn't make sense and it turned off all >>> topology-related decisions. Which means that kernel already has a >>> solution for weird topology. Some enumeration doesn't trigger this >>> warning, but we can come up with one that does. Or we can indeed have a >>> patch in kernel that will, possibly silently, fail topology_sane() when >>> virtualized and not pinned. >> >> How would you come up with a topology the kernel is complaining about >> and user mode scheduling will use for sane decisions ? > > We need to understand first why Dario's box is apparently the only one > resulting in a warning and probably then emulate that enumeration. This will lead to other problems in user land e.g. with hwloc. > And again, if that is not possible then just make topology_sane() fail. And again: once you claim that kernel mode isn't everything and here you fail to respect possible user land requirements. >>> (This is what I assume kernel does when topology_sane() fails. And if it >>> doesn't, that's a bug IMO) >>> >>> The licensing problem that Juergen described can be solved by pining >>> vcpus and exposing HT bit. Besides, creating a guest with 24 VPCUs and >> >> Hmm, yes. This way you sacrifice most of the virtualization advantages. >> >>> hoping that 16-core licensing will work I think is pushing it a bit when >>> you know that VCPUs will jump around cores (i.e. "on average" you are >>> running on more than 16 cores -- multi-threaded or not -- which arguably >>> is what licensing is trying to prevent) >> >> On a machine with only 16 cores running on more than 16 cores? I have >> some problems to believe this. The point was: if the license is happy on >> bare metal it should be so when running on the same hardware as a guest. > > Ok, that's not how I should have described it. I meant that IMO asking > for 24 VCPUs is somewhat akin to oversubscribing since you kind of know > that you dont' have 24 PCPUs, you are just trying to fool the kernel > into thinking that threads are cores. /proc/cpuinfo on bare metal will list 32 cpus. xl info in dom0 will list 32 cpus. You have 32 entities where you can do scheduling. So what's the problem having a domU with 24 vcpus? There are still 8 pcpus free for e.g. dom0 then. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 14:34 ` Boris Ostrovsky 2015-07-27 14:43 ` Juergen Gross @ 2015-07-27 14:47 ` Juergen Gross 2015-07-27 14:58 ` Dario Faggioli 2 siblings, 0 replies; 95+ messages in thread From: Juergen Gross @ 2015-07-27 14:47 UTC (permalink / raw) To: Boris Ostrovsky, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/27/2015 04:34 PM, Boris Ostrovsky wrote: > On 07/27/2015 10:09 AM, Dario Faggioli wrote: >> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote: >>> On 07/24/2015 05:58 PM, Dario Faggioli wrote: >>>> So, just to check if I'm understanding is correct: you'd like to add an >>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling) >>>> code, to hide the direct interaction with CPUID. >>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd >>>> check with Xen/match vNUMA/whatever... Is this that you are saying? >>> Sort of, yes. >>> >>> I just wouldn't add it, as it is already existing (more or less). It >>> can deal right now with AMD and Intel, we would "just" have to add Xen. >>> >> So, having gone through the rest of the thread (so far), and having >> given a fair amount o thinking to this, I really think that something >> like this would be a good thing to have in Linux. >> >> Of course, it's not that my opinion on where should be in Linux counts >> that much! :-D Nevertheless, I wanted to make it clear that, while >> skeptic at the beginning, I now think this is (part of) the way to go, >> as I said and explained in my reply to George. > > And I continue to believe that kernel solution does not address the > userland problem which is no less important than making kernel do proper > scheduling decisions (and I suspect when this patch goes for review > that's what the scheduling people are going to say). I didn't say it would solve that problem. It would decouple kernel scheduling and cpuid values in order to be free to present cpuid values to user land needed there. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 14:34 ` Boris Ostrovsky 2015-07-27 14:43 ` Juergen Gross 2015-07-27 14:47 ` Juergen Gross @ 2015-07-27 14:58 ` Dario Faggioli 2 siblings, 0 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-27 14:58 UTC (permalink / raw) To: Boris Ostrovsky Cc: Juergen Gross, Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 2587 bytes --] On Mon, 2015-07-27 at 10:34 -0400, Boris Ostrovsky wrote: > On 07/27/2015 10:09 AM, Dario Faggioli wrote: > > Of course, it's not that my opinion on where should be in Linux counts > > that much! :-D Nevertheless, I wanted to make it clear that, while > > skeptic at the beginning, I now think this is (part of) the way to go, > > as I said and explained in my reply to George. > > And I continue to believe that kernel solution does not address the > userland problem which is no less important than making kernel do proper > scheduling decisions > Sure, but the key point now is, IMO, whether we recognise that we're dealing with two problems, which I think is the case. One problem is: 1. CPUID in VMs is unreliable, can we rely on it as few as possible? The other problem is: 2. if someone wants to rely on CPUID, what should we do. Avoiding the scheduling domains (and/or whatever else!) to relay on something that is unreliable by definition, which is what is being proposed, seems to me a pretty decent solution to 1. > (and I suspect when this patch goes for review > that's what the scheduling people are going to say). > Perhaps. However, I may be too optimistic, but I think that making them realize that all we are doing is stopping relying on unreliable grounds would not be impossible... Especially if the code really looks as Juergen said, i.e., it's already quite friendly to implementing something like this... > Remember the original problem that started this thread was that kernel > complained that topology didn't make sense and it turned off all > topology-related decisions. Which means that kernel already has a > solution for weird topology. Some enumeration doesn't trigger this > warning, but we can come up with one that does. > Exactly, because it relies on CPUID, which is unreliable!! :-) > Or we can indeed have a > patch in kernel that will, possibly silently, fail topology_sane() when > virtualized and not pinned. > Well, I think that, if I were a Linux scheduler or x86 maintainer, I would oppose much more to a similar patch, rather than to one that makes the kernel stop relying on an instruction that gives pCPU specific results, to build-up long lived information, in a context where pCPUs change. :-) Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 14:09 ` Dario Faggioli 2015-07-27 14:34 ` Boris Ostrovsky @ 2015-07-28 4:29 ` Juergen Gross 2015-07-28 15:11 ` Juergen Gross 1 sibling, 1 reply; 95+ messages in thread From: Juergen Gross @ 2015-07-28 4:29 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/27/2015 04:09 PM, Dario Faggioli wrote: > On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote: >> On 07/24/2015 05:58 PM, Dario Faggioli wrote: > >>> So, just to check if I'm understanding is correct: you'd like to add an >>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling) >>> code, to hide the direct interaction with CPUID. >>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd >>> check with Xen/match vNUMA/whatever... Is this that you are saying? >> >> Sort of, yes. >> >> I just wouldn't add it, as it is already existing (more or less). It >> can deal right now with AMD and Intel, we would "just" have to add Xen. >> > So, having gone through the rest of the thread (so far), and having > given a fair amount o thinking to this, I really think that something > like this would be a good thing to have in Linux. > > Of course, it's not that my opinion on where should be in Linux counts > that much! :-D Nevertheless, I wanted to make it clear that, while > skeptic at the beginning, I now think this is (part of) the way to go, > as I said and explained in my reply to George. I think it's time to obtain some real numbers. I'll make some performance tests on a big machine (4 sockets, 60 cores, 120 threads) regarding topology information: - bare metal - "random" topology (like today) - "simple" topology (all vcpus regarded as equal) - "real" topology with all vcpus pinned This should show: - how intrusive would the topology patch(es) be? - what is the performance impact of a "wrong" scheduling data base Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-28 4:29 ` Juergen Gross @ 2015-07-28 15:11 ` Juergen Gross 2015-07-28 16:17 ` Dario Faggioli 2015-07-29 7:44 ` Dario Faggioli 0 siblings, 2 replies; 95+ messages in thread From: Juergen Gross @ 2015-07-28 15:11 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1: Type: text/plain, Size: 2906 bytes --] On 07/28/2015 06:29 AM, Juergen Gross wrote: > On 07/27/2015 04:09 PM, Dario Faggioli wrote: >> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote: >>> On 07/24/2015 05:58 PM, Dario Faggioli wrote: >> >>>> So, just to check if I'm understanding is correct: you'd like to add an >>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling) >>>> code, to hide the direct interaction with CPUID. >>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd >>>> check with Xen/match vNUMA/whatever... Is this that you are saying? >>> >>> Sort of, yes. >>> >>> I just wouldn't add it, as it is already existing (more or less). It >>> can deal right now with AMD and Intel, we would "just" have to add Xen. >>> >> So, having gone through the rest of the thread (so far), and having >> given a fair amount o thinking to this, I really think that something >> like this would be a good thing to have in Linux. >> >> Of course, it's not that my opinion on where should be in Linux counts >> that much! :-D Nevertheless, I wanted to make it clear that, while >> skeptic at the beginning, I now think this is (part of) the way to go, >> as I said and explained in my reply to George. > > I think it's time to obtain some real numbers. > > I'll make some performance tests on a big machine (4 sockets, 60 cores, > 120 threads) regarding topology information: > > - bare metal > - "random" topology (like today) > - "simple" topology (all vcpus regarded as equal) > - "real" topology with all vcpus pinned > > This should show: > > - how intrusive would the topology patch(es) be? > - what is the performance impact of a "wrong" scheduling data base On the above box I used a pvops kernel 4.2-rc4 plus a rather small patch (see attachment). I did 5 kernel builds in each environment: make clean time make -j 120 The first result of the 5 runs was always omitted as it would have to build up buffer caches etc. The Xen cases were all done in dom0, pinning of vcpus in the last scenario was done via dom0_vcpus_pin boot parameter of the hypervisor. Here are the results (everything in seconds): elapsed user system bare metal: 100 5770 805 "random" topology: 283 6740 20700 "simple" topology: 290 6740 22200 "real" topology: 185 7800 8040 As expected bare metal is the best. Next is "real" topology with pinned vcpus (expected again - but system time already factor of 10 up!). What I didn't expect is: "random" is better than "simple" topology. I could test some other topologies (e.g. everything on one socket, or even on one core), but I'm not sure this makes sense. I didn't check the exact topology result of the "random" case, maybe I'll do that tomorrow with another measurement. BTW: the topology hack is working, as each cpu is shown to have a sibling count of 1 in /proc/cpuinfo. Juergen [-- Attachment #2: topo.patch --] [-- Type: text/x-patch, Size: 2057 bytes --] commit 2d9cee2b37319714e07e6f8b4044c0db44cc8e7d Author: Juergen Gross <jgross@suse.com> Date: Tue Jul 28 09:28:35 2015 +0200 xen: use simple topology on demand As the cpuid information is currently taken from the physical cpu the current virtual cpu is running on, the topology information derived from the cpuid is potentially useless. In order to avoid insane scheduling decisions based on random data provide a possibility to set topology data to a very simple scheme making all cpus appear to be independent from each other. Signed-off-by: Juergen Gross <jgross@suse.com> diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c index 8648438..a57e816 100644 --- a/arch/x86/xen/smp.c +++ b/arch/x86/xen/smp.c @@ -55,6 +55,25 @@ static irqreturn_t xen_call_function_interrupt(int irq, void *dev_id); static irqreturn_t xen_call_function_single_interrupt(int irq, void *dev_id); static irqreturn_t xen_irq_work_interrupt(int irq, void *dev_id); +static bool xen_simple_topology; +static __init int xen_parse_topology(char *arg) +{ + xen_simple_topology = true; + return 0; +} +early_param("xen_simple_topology", xen_parse_topology); + +static void xen_set_cpu_sibling_map(int cpu) +{ + if (xen_simple_topology) { + cpumask_set_cpu(cpu, topology_sibling_cpumask(cpu)); + cpumask_set_cpu(cpu, cpu_llc_shared_mask(cpu)); + cpumask_set_cpu(cpu, topology_core_cpumask(cpu)); + } else { + set_cpu_sibling_map(cpu); + } +} + /* * Reschedule call back. */ @@ -82,7 +101,7 @@ static void cpu_bringup(void) cpu = smp_processor_id(); smp_store_cpu_info(cpu); cpu_data(cpu).x86_max_cores = 1; - set_cpu_sibling_map(cpu); + xen_set_cpu_sibling_map(cpu); xen_setup_cpu_clockevents(); @@ -333,7 +352,7 @@ static void __init xen_smp_prepare_cpus(unsigned int max_cpus) zalloc_cpumask_var(&per_cpu(cpu_core_map, i), GFP_KERNEL); zalloc_cpumask_var(&per_cpu(cpu_llc_shared_map, i), GFP_KERNEL); } - set_cpu_sibling_map(0); + xen_set_cpu_sibling_map(0); if (xen_smp_intr_init(0)) BUG(); [-- Attachment #3: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply related [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-28 15:11 ` Juergen Gross @ 2015-07-28 16:17 ` Dario Faggioli 2015-07-28 17:13 ` Dario Faggioli 2015-07-29 6:04 ` Juergen Gross 2015-07-29 7:44 ` Dario Faggioli 1 sibling, 2 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-28 16:17 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 5849 bytes --] On Tue, 2015-07-28 at 17:11 +0200, Juergen Gross wrote: > On 07/28/2015 06:29 AM, Juergen Gross wrote: > > I'll make some performance tests on a big machine (4 sockets, 60 cores, > > 120 threads) regarding topology information: > > > > - bare metal > > - "random" topology (like today) > > - "simple" topology (all vcpus regarded as equal) > > - "real" topology with all vcpus pinned > > > > This should show: > > > > - how intrusive would the topology patch(es) be? > > - what is the performance impact of a "wrong" scheduling data base > > On the above box I used a pvops kernel 4.2-rc4 plus a rather small patch > (see attachment). I did 5 kernel builds in each environment: > > make clean > time make -j 120 > Right. If you have time, can you try '-j60' and '-j30' (maybe even -j45 and -j15, if you've got _a_lot_ of time! :-)). I'm asking this because, with hyperthreading involved, I've sometimes seen things being the worse when *not* (over)saturating the CPU capacity. The explanation is that, if every vcpu is busy, meaning that every thread is busy, it does not make much difference where you schedule the busy vcpus. OTOH, if only 1/2 of the threads are busy, a properly setup system will effectively spread the load in such a way that each vcpu has a full core available; a messed up one will, when trying to do the same, end up scheduling stuff on siblings, even if there are idle cores available. In this case, things are a bit more tricky. In fact, I've observed the above while looking after the Xen scheduler. In this case, it is the guest (dom0) scheduler that we are looking at, and, e.g., if the load is small enough, Xen's scheduler will fix things up, at least up to a certain extent. It's worth a try anyway, I guess, if you have time, of course. > The first result of the 5 runs was always omitted as it would have to > build up buffer caches etc. The Xen cases were all done in dom0, pinning > of vcpus in the last scenario was done via dom0_vcpus_pin boot parameter > of the hypervisor. > > Here are the results (everything in seconds): > > elapsed user system > bare metal: 100 5770 805 > "random" topology: 283 6740 20700 > "simple" topology: 290 6740 22200 > "real" topology: 185 7800 8040 > > As expected bare metal is the best. Next is "real" topology with pinned > vcpus (expected again - but system time already factor of 10 up!). > I also think that (massively) overloading biases things in favour of pinning. In fact, pinning incurs in less overhead, as there are no scheduling decisions involved, and no migrations of vcpus among pcpus. With the system oversubscribed to to 200%, even in the non-pinning case there shouldn't be much migrations, but certainly there will be some, and they turn out to be pure overhead! In fact, they bring zero benefits, as it's not possible that any of them will put the system in a more advantageous state, performance wise: we're fully loaded and we want to stay fully loaded! > What I didn't expect is: "random" is better than "simple" topology. > Weird indeed! > I > could test some other topologies (e.g. everything on one socket, or even > on one core), but I'm not sure this makes sense. I didn't check the > exact topology result of the "random" case, maybe I'll do that tomorrow > with another measurement. > So, my test box looks like this: cpu_topology : cpu: core socket node 0: 0 1 0 1: 0 1 0 2: 1 1 0 3: 1 1 0 4: 9 1 0 5: 9 1 0 6: 10 1 0 7: 10 1 0 8: 0 0 1 9: 0 0 1 10: 1 0 1 11: 1 0 1 12: 9 0 1 13: 9 0 1 14: 10 0 1 15: 10 0 1 In Dom0, here's what I see _without_ any pinning: root@Zhaman:~# for i in `seq 0 15`;do cat /sys/devices/system/cpu/cpu$i/topology/thread_siblings_list ;done 0-1 0-1 2-3 2-3 4-5 4-5 6-7 6-7 8-9 8-9 10-11 10-11 12-13 12-13 14-15 14-15 root@Zhaman:~# cat /proc/cpuinfo |grep "physical id" physical id : 1 physical id : 1 physical id : 1 physical id : 1 physical id : 1 physical id : 1 physical id : 1 physical id : 1 physical id : 0 physical id : 0 physical id : 0 physical id : 0 physical id : 0 physical id : 0 physical id : 0 physical id : 0 root@Zhaman:~# cat /proc/cpuinfo |grep "core id" core id : 0 core id : 0 core id : 1 core id : 1 core id : 9 core id : 9 core id : 10 core id : 10 core id : 0 core id : 0 core id : 1 core id : 1 core id : 9 core id : 9 core id : 10 core id : 10 root@Zhaman:~# cat /proc/cpuinfo |grep "cpu cores" cpu cores : 4 <same for all cpus> root@Zhaman:~# cat /proc/cpuinfo |grep "siblings" siblings : 8 <same for all cpus> So, basically, as far as Dom0 on my test box is concerned, "random" actually matches the host topology. Sure, without pinning, this looks equally wrong, as Xen's scheduler can well execute, say, vcpu 0 and vcpu 4, which are not siblings, on the same core. But then again, if the load is small, it just won't happen (e.g., if there are only those two busy vcpus, Xen will send them on !siblings core), while if it's too hugh, it won't matter... :-/ Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-28 16:17 ` Dario Faggioli @ 2015-07-28 17:13 ` Dario Faggioli 2015-07-29 6:04 ` Juergen Gross 1 sibling, 0 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-28 17:13 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 4297 bytes --] On Tue, 2015-07-28 at 18:17 +0200, Dario Faggioli wrote: > So, my test box looks like this: > cpu_topology : > cpu: core socket node > 0: 0 1 0 > 1: 0 1 0 > 2: 1 1 0 > 3: 1 1 0 > 4: 9 1 0 > 5: 9 1 0 > 6: 10 1 0 > 7: 10 1 0 > 8: 0 0 1 > 9: 0 0 1 > 10: 1 0 1 > 11: 1 0 1 > 12: 9 0 1 > 13: 9 0 1 > 14: 10 0 1 > 15: 10 0 1 > > In Dom0, here's what I see _without_ any pinning: > And now I've tried with dom0_vcpus_pin: root@Zhaman:~# xl vcpu-list Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) Domain-0 0 0 0 -b- 6.8 0 / all Domain-0 0 1 1 -b- 1.6 1 / all Domain-0 0 2 2 -b- 2.3 2 / all Domain-0 0 3 3 -b- 1.5 3 / all Domain-0 0 4 4 -b- 3.2 4 / all Domain-0 0 5 5 -b- 1.5 5 / all Domain-0 0 6 6 -b- 2.0 6 / all Domain-0 0 7 7 -b- 2.2 7 / all Domain-0 0 8 8 -b- 1.6 8 / all Domain-0 0 9 9 -b- 1.6 9 / all Domain-0 0 10 10 r-- 2.1 10 / all Domain-0 0 11 11 -b- 1.5 11 / all Domain-0 0 12 12 -b- 2.4 12 / all Domain-0 0 13 13 -b- 1.1 13 / all Domain-0 0 14 14 -b- 2.4 14 / all Domain-0 0 15 15 -b- 2.4 15 / all > root@Zhaman:~# for i in `seq 0 15`;do cat /sys/devices/system/cpu/cpu$i/topology/thread_siblings_list ;done > 0-1 > 0-1 > 2-3 > 2-3 > 4-5 > 4-5 > 6-7 > 6-7 > 8-9 > 8-9 > 10-11 > 10-11 > 12-13 > 12-13 > 14-15 > 14-15 > Same result. > root@Zhaman:~# cat /proc/cpuinfo |grep "physical id" > physical id : 1 > physical id : 1 > physical id : 1 > physical id : 1 > physical id : 1 > physical id : 1 > physical id : 1 > physical id : 1 > physical id : 0 > physical id : 0 > physical id : 0 > physical id : 0 > physical id : 0 > physical id : 0 > physical id : 0 > physical id : 0 > Same result. > root@Zhaman:~# cat /proc/cpuinfo |grep "core id" > core id : 0 > core id : 0 > core id : 1 > core id : 1 > core id : 9 > core id : 9 > core id : 10 > core id : 10 > core id : 0 > core id : 0 > core id : 1 > core id : 1 > core id : 9 > core id : 9 > core id : 10 > core id : 10 > Same result. > root@Zhaman:~# cat /proc/cpuinfo |grep "cpu cores" > cpu cores : 4 > <same for all cpus> > Same result. > root@Zhaman:~# cat /proc/cpuinfo |grep "siblings" > siblings : 8 > <same for all cpus> > And same result here as well. So, for Dom0, pinning does not make much difference as far as what topology is guessed, which actually makes sense, considering how pcpus are brought up in Xen, and assuming that Linux does something similar (no, I'm not familiar with that code). It should be making a difference in terms of whether and how much this topology, although matching the host one, mislead the guest scheduler. Actually, I still think it does, it's just harder to identify than we expected, which may be seen as a good thing (it does not look like it is a big deal, after all :-D), or a bad thing (it may start biting us anytime, without us noticing promptly :-/). Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-28 16:17 ` Dario Faggioli 2015-07-28 17:13 ` Dario Faggioli @ 2015-07-29 6:04 ` Juergen Gross 2015-07-29 7:09 ` Dario Faggioli 1 sibling, 1 reply; 95+ messages in thread From: Juergen Gross @ 2015-07-29 6:04 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/28/2015 06:17 PM, Dario Faggioli wrote: > On Tue, 2015-07-28 at 17:11 +0200, Juergen Gross wrote: >> On 07/28/2015 06:29 AM, Juergen Gross wrote: > >>> I'll make some performance tests on a big machine (4 sockets, 60 cores, >>> 120 threads) regarding topology information: >>> >>> - bare metal >>> - "random" topology (like today) >>> - "simple" topology (all vcpus regarded as equal) >>> - "real" topology with all vcpus pinned >>> >>> This should show: >>> >>> - how intrusive would the topology patch(es) be? >>> - what is the performance impact of a "wrong" scheduling data base >> >> On the above box I used a pvops kernel 4.2-rc4 plus a rather small patch >> (see attachment). I did 5 kernel builds in each environment: >> >> make clean >> time make -j 120 >> > Right. If you have time, can you try '-j60' and '-j30' (maybe even -j45 > and -j15, if you've got _a_lot_ of time! :-)). The test machine can do this without me watching, so I've just started the first configuration... > I'm asking this because, with hyperthreading involved, I've sometimes > seen things being the worse when *not* (over)saturating the CPU > capacity. Hmm, oversaturation shouldn't happen here. I've added -j 240 to let it happen. ... > So, basically, as far as Dom0 on my test box is concerned, "random" > actually matches the host topology. Okay, have to check that on my box. I think I'll have another try with a domU. This could be much more "random" than dom0. > Sure, without pinning, this looks equally wrong, as Xen's scheduler can > well execute, say, vcpu 0 and vcpu 4, which are not siblings, on the > same core. But then again, if the load is small, it just won't happen > (e.g., if there are only those two busy vcpus, Xen will send them on > !siblings core), while if it's too hugh, it won't matter... :-/ ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-29 6:04 ` Juergen Gross @ 2015-07-29 7:09 ` Dario Faggioli 0 siblings, 0 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-29 7:09 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 1088 bytes --] On Wed, 2015-07-29 at 08:04 +0200, Juergen Gross wrote: > On 07/28/2015 06:17 PM, Dario Faggioli wrote: > >> On 07/28/2015 06:29 AM, Juergen Gross wrote: > > > >>> I'll make some performance tests on a big machine (4 sockets, 60 cores, > >>> 120 threads) regarding topology information: > >>> > > I'm asking this because, with hyperthreading involved, I've sometimes > > seen things being the worse when *not* (over)saturating the CPU > > capacity. > > Hmm, oversaturation shouldn't happen here. I've added -j 240 to let it > happen. > Oh, sorry... I was doing the math wrong (I was not counting the threads, which is funny, as this is pretty much all about threads! :-O) > I think I'll have another try with a domU. This could be much more > "random" than dom0. > Indeed! Thanks and Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-28 15:11 ` Juergen Gross 2015-07-28 16:17 ` Dario Faggioli @ 2015-07-29 7:44 ` Dario Faggioli 1 sibling, 0 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-29 7:44 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 4528 bytes --] On Tue, 2015-07-28 at 17:11 +0200, Juergen Gross wrote: > On 07/28/2015 06:29 AM, Juergen Gross wrote: > > On 07/27/2015 04:09 PM, Dario Faggioli wrote: > >> On Fri, 2015-07-24 at 18:10 +0200, Juergen Gross wrote: > >>> On 07/24/2015 05:58 PM, Dario Faggioli wrote: > >> > >>>> So, just to check if I'm understanding is correct: you'd like to add an > >>>> abstraction layer, in Linux, like in generic (or, perhaps, scheduling) > >>>> code, to hide the direct interaction with CPUID. > >>>> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd > >>>> check with Xen/match vNUMA/whatever... Is this that you are saying? > >>> > >>> Sort of, yes. > >>> > >>> I just wouldn't add it, as it is already existing (more or less). It > >>> can deal right now with AMD and Intel, we would "just" have to add Xen. > >>> > >> So, having gone through the rest of the thread (so far), and having > >> given a fair amount o thinking to this, I really think that something > >> like this would be a good thing to have in Linux. > >> > >> Of course, it's not that my opinion on where should be in Linux counts > >> that much! :-D Nevertheless, I wanted to make it clear that, while > >> skeptic at the beginning, I now think this is (part of) the way to go, > >> as I said and explained in my reply to George. > > > > I think it's time to obtain some real numbers. > > > > I'll make some performance tests on a big machine (4 sockets, 60 cores, > > 120 threads) regarding topology information: > > > > - bare metal > > - "random" topology (like today) > > - "simple" topology (all vcpus regarded as equal) > > - "real" topology with all vcpus pinned > > > > This should show: > > > > - how intrusive would the topology patch(es) be? > > - what is the performance impact of a "wrong" scheduling data base > > On the above box I used a pvops kernel 4.2-rc4 plus a rather small patch > (see attachment). I did 5 kernel builds in each environment: > > make clean > time make -j 120 > > The first result of the 5 runs was always omitted as it would have to > build up buffer caches etc. The Xen cases were all done in dom0, pinning > of vcpus in the last scenario was done via dom0_vcpus_pin boot parameter > of the hypervisor. > > Here are the results (everything in seconds): > > elapsed user system > bare metal: 100 5770 805 > "random" topology: 283 6740 20700 > "simple" topology: 290 6740 22200 > "real" topology: 185 7800 8040 > > As expected bare metal is the best. Next is "real" topology with pinned > vcpus (expected again - but system time already factor of 10 up!). > What I didn't expect is: "random" is better than "simple" topology. I > could test some other topologies (e.g. everything on one socket, or even > on one core), but I'm not sure this makes sense. I didn't check the > exact topology result of the "random" case, maybe I'll do that tomorrow > with another measurement. > > BTW: the topology hack is working, as each cpu is shown to have a > sibling count of 1 in /proc/cpuinfo. > Hey, just a 'wild' idea, as a possible explanation (or direction for further investigations) on why 'simple' si actually worse than 'random'. Could it be that, by setting up the topology like in 'simple', we create more work for the Linux scheduler, and hence incur in more overhead? Or, looking at thing the other way round, which is it the topology that impose the less overhead, in terms of load balancing work, to the Linux scheduler? I think we should identify and try it as, in the absence of any vNUMA topology being specified, that's what we want, as our flat topology (and with vNUMA, that's what we want within virtual nodes). I don't really know, right now, whether such topology would be 'all cores, no SMT siblings', or 'all siblings of each others', or 'all sockets, no core/SMT siblings'... This requires inspecting the scheduler's and scheduling domain's code, which I can do, but not today, as I won't be working. If you (or anyone) think it's worth and actually have a look, let me/us know your findings. If not, I'll do it myself tomorrow. Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 15:14 ` Juergen Gross 2015-07-24 15:24 ` Juergen Gross @ 2015-07-24 16:05 ` Dario Faggioli 2015-07-28 10:05 ` Wei Liu 1 sibling, 1 reply; 95+ messages in thread From: Dario Faggioli @ 2015-07-24 16:05 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 1818 bytes --] On Fri, 2015-07-24 at 17:14 +0200, Juergen Gross wrote: > On 07/24/2015 04:44 PM, Dario Faggioli wrote: > > Ok. And I already have a question (as I lost track of things a bit). > > What you just said about ACPI tables is certainly true for baremetal and > > HVM guests, but for PV? At the time I was looking into it, together with > > Elena, there were Linux patches being produced for the PV case, which > > makes sense. > > However, ISTR that both Wei and Elena mentioned recently that those > > patches have not been upstreamed in Linux yet... Is that the case? Maybe > > not all, but at least some of them are there? Because if not, I'm not > > sure I see how a PV guest would even see a vNUMA topology (which it > > does). > I checked "bottom up", so when I found the acpi scan stuff I stopped > searching how the kernel obtains numa info. During my search I found no > clue of an pv-numa stuff in the kernel. And a quick "grep -i numa" in > arch/x86/xen and drivers/xen didn't reveal anything. Same for a complete > kernel source search for "vnuma". > BTW, I've also been grepping, and I'm not seeing XENMEM_get_vnumainfo being called anywhere either... Well, no wonder, we're seeing vNUMA setup issues! If I did check for this before, I wouldn't even have tested PV-vNUMA in the first place!! :-O I actually wonder how _some_ of the vNUMA info get through!?!? :-O IAC, all we're saying remains true, of course, and the CPUID issue will still be there, if we don't fix it, when such a patch will be accepted. Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 16:05 ` Dario Faggioli @ 2015-07-28 10:05 ` Wei Liu 2015-07-28 15:17 ` Dario Faggioli 0 siblings, 1 reply; 95+ messages in thread From: Wei Liu @ 2015-07-28 10:05 UTC (permalink / raw) To: Dario Faggioli Cc: Juergen Gross, Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On Fri, Jul 24, 2015 at 06:05:59PM +0200, Dario Faggioli wrote: > On Fri, 2015-07-24 at 17:14 +0200, Juergen Gross wrote: > > On 07/24/2015 04:44 PM, Dario Faggioli wrote: > > > > Ok. And I already have a question (as I lost track of things a bit). > > > What you just said about ACPI tables is certainly true for baremetal and > > > HVM guests, but for PV? At the time I was looking into it, together with > > > Elena, there were Linux patches being produced for the PV case, which > > > makes sense. > > > However, ISTR that both Wei and Elena mentioned recently that those > > > patches have not been upstreamed in Linux yet... Is that the case? Maybe > > > not all, but at least some of them are there? Because if not, I'm not > > > sure I see how a PV guest would even see a vNUMA topology (which it > > > does). > > > I checked "bottom up", so when I found the acpi scan stuff I stopped > > searching how the kernel obtains numa info. During my search I found no > > clue of an pv-numa stuff in the kernel. And a quick "grep -i numa" in > > arch/x86/xen and drivers/xen didn't reveal anything. Same for a complete > > kernel source search for "vnuma". > > > BTW, I've also been grepping, and I'm not seeing XENMEM_get_vnumainfo > being called anywhere either... Well, no wonder, we're seeing vNUMA > setup issues! If I did check for this before, I wouldn't even have > tested PV-vNUMA in the first place!! :-O > > I actually wonder how _some_ of the vNUMA info get through!?!? :-O > Maybe you're using a custom built kernel with my PV vNUMA patch in? I sent that to you once because you wanted to test that. Wei. > IAC, all we're saying remains true, of course, and the CPUID issue will > still be there, if we don't fix it, when such a patch will be accepted. > > Dario > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-28 10:05 ` Wei Liu @ 2015-07-28 15:17 ` Dario Faggioli 0 siblings, 0 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-28 15:17 UTC (permalink / raw) To: Wei Liu Cc: Juergen Gross, Elena Ufimtseva, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 1812 bytes --] On Tue, 2015-07-28 at 11:05 +0100, Wei Liu wrote: > On Fri, Jul 24, 2015 at 06:05:59PM +0200, Dario Faggioli wrote: > > BTW, I've also been grepping, and I'm not seeing XENMEM_get_vnumainfo > > being called anywhere either... Well, no wonder, we're seeing vNUMA > > setup issues! If I did check for this before, I wouldn't even have > > tested PV-vNUMA in the first place!! :-O > > > > I actually wonder how _some_ of the vNUMA info get through!?!? :-O > > > > Maybe you're using a custom built kernel with my PV vNUMA patch in? > Yep, that must be the case... I only recalled about that after sending this email (no, I don't update Dom0/guest kernel for my testbox really often! :-/) > I sent that to you once because you wanted to test that. > And I confirmed it. Booting the guest with Debian's stick 4.0.0, I get this from Xen: (XEN) Memory location of each domain: (XEN) Domain 0 (total: 129914): (XEN) Node 0: 56111 (XEN) Node 1: 73803 (XEN) Domain 2 (total: 262144): (XEN) Node 0: 131072 (XEN) Node 1: 131072 (XEN) 2 vnodes, 4 vcpus, guest physical layout: (XEN) 0: pnode 0, vcpus 0-1 (XEN) 0000000000000000 - 0000000020000000 (XEN) 1: pnode 1, vcpus 2-3 (XEN) 0000000020000000 - 0000000040000000 But the (PV) guest does not actually know anything about vNUMA: root@test:~# numactl --hardware available: 1 nodes (0) node 0 cpus: 0 1 2 3 node 0 size: 993 MB node 0 free: 918 MB node distances: node 0 0: 10 Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-24 14:44 ` Dario Faggioli 2015-07-24 15:14 ` Juergen Gross @ 2015-07-24 20:27 ` Elena Ufimtseva 1 sibling, 0 replies; 95+ messages in thread From: Elena Ufimtseva @ 2015-07-24 20:27 UTC (permalink / raw) To: Dario Faggioli Cc: Juergen Gross, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On Fri, Jul 24, 2015 at 04:44:36PM +0200, Dario Faggioli wrote: > On Fri, 2015-07-24 at 12:28 +0200, Juergen Gross wrote: > > On 07/23/2015 04:07 PM, Dario Faggioli wrote: > > > > FWIW, I was thinking that the kernel were a better place, as Juergen is > > > saying, while now I'm more convinced that tools would be more > > > appropriate, as Boris is saying. > > > > I've collected some information from the linux kernel sources as a base > > for the discussion: > > > That's great, thanks for this! > > > The complete numa information (cpu->node and memory->node relations) is > > taken from the acpi tables (srat, slit for "distances"). > > > Ok. And I already have a question (as I lost track of things a bit). > What you just said about ACPI tables is certainly true for baremetal and > HVM guests, but for PV? At the time I was looking into it, together with > Elena, there were Linux patches being produced for the PV case, which > makes sense. > However, ISTR that both Wei and Elena mentioned recently that those > patches have not been upstreamed in Linux yet... Is that the case? Maybe > not all, but at least some of them are there? Because if not, I'm not > sure I see how a PV guest would even see a vNUMA topology (which it > does). Yes, PV guest should not be able to get any vnuma info. I hope finish with rebasnig patch and post it on Monday. Few things has changed, as well as assignment of cpus to vnodes and the way it was done before (in latest version on linux vnuma patch) does not work anymore. > > Of course, I can go and check, but since you just looked, you may have > it fresh and clear already. :-) > > > The topology information is obtained via: > > - intel: > > + cpuid leaf b with subleafs, leaf 4 > > + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available > > - amd: > > + cpuid leaf 8000001e, leaf 8000001d, leaf 4 > > + msr c001100c > > + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available > > > > The scheduler is aware of: > > - smt siblings (from topology) > > - last-level-cache siblings (from topology) > > - node siblings (from numa information) > > > Right. So, this confirms what we were guessing: we need to "reconcile" > these two sources of information (from the guest point of view). > > Both the 'in kernel' and 'in toolstack' approach should have all the > necessary information to make things match, I think. In fact, in > toolstack, we know what the vNUMA topology is (we're parsing and > actually putting it in place!). In kernel, we know it as we read it from > tables or hypercalls (isn't that so, for PV guest?). > > In fact, I think that it is the topology, i.e., what comes from MSRs, > that needs to adapt, and follow vNUMA, as much as possible. Do we agree > on this? > > IMO, the thing boils down to these: > > 1) from where (kernel vs. toolstack) is it the most easy and effective > to enact the CPUID fiddling? As in, can we do that in toolstack? > (Andrew was not so sure, and Boris found issues, although Jan seems > to think they're no show stopper.) > I'm quite certain that we can do that from inside the kernel, > although, how early would we need to be doing it? Do we have the > vNUMA info already? > > 2) when tweaking the values of CPUID and other MSRs, are there other > vNUMA (and topology in general) constraints and requirements we > should take into account? For instance, do we want, for licensing > reasons, all (or most) of the vcpus to be siblings, rather than full > sockets? Etc. > 2a) if yes, how and where are these constraints specified? > > If looking at 1) only, it still looks to me that doing things within the > kernel would be the way to go. > > When looking at 2), OTOH, toolstacks variants start to be more > appealing. Especially depending on our answer to 2a). In fact, > in case we want to give the user a way to specify this > siblings-vs-cores-vs-sockets information, it IMO would be good to deal > with that in tools, rather than having to involve Xen or Linux! > > > It will especially move tasks from one cpu to another first between smt > > siblings, second between llc siblings, third between node siblings and > > last all cpus. > > > Yep, this part, I knew. > > Maybe, there is room for "fixing" this at this level, hooking up inside > the scheduler code... but I'm shooting in the dark, without having check > whether and how this could be really feasible, should I? > > One thing I don't like about this approach is that it would potentially > solve vNUMA and other scheduling anomalies, but... > > > cpuid instruction is available for user mode as well. > > > ...it would not do any good for other subsystems, and user level code > and apps. > > Dario > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-22 14:09 ` Juergen Gross 2015-07-22 14:44 ` Boris Ostrovsky @ 2015-07-22 14:50 ` Dario Faggioli 2015-07-22 15:32 ` Boris Ostrovsky 1 sibling, 1 reply; 95+ messages in thread From: Dario Faggioli @ 2015-07-22 14:50 UTC (permalink / raw) To: Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 3779 bytes --] On Wed, 2015-07-22 at 16:09 +0200, Juergen Gross wrote: > On 07/22/2015 03:58 PM, Boris Ostrovsky wrote: > > What if I configure a guest to follow HW topology? I.e. I pin VCPUs to > > appropriate cores/threads? With elfnote I am stuck with disabled topology. > > Add an option to do exactly that: follow HW topology (pin vcpus, > configure vnuma)? > I thought about configuring things in such a way that they match the host topology, as Boris is suggesting, too. And in that case, I think arranging for doing so in toolstack, if PV vNUMA is identified (as I think Juergen is suggesting) seems a good approach. However, when I try to do that on my box, manually, but I don't seem to be able to. Here's what I tried. Since I have this host topology: cpu_topology : cpu: core socket node 0: 0 1 0 1: 0 1 0 2: 1 1 0 3: 1 1 0 4: 9 1 0 5: 9 1 0 6: 10 1 0 7: 10 1 0 8: 0 0 1 9: 0 0 1 10: 1 0 1 11: 1 0 1 12: 9 0 1 13: 9 0 1 14: 10 0 1 15: 10 0 1 I configured the guest like this: vcpus = '4' memory = '1024' vnuma = [ [ "pnode=0","size=512","vcpus=0-1","vdistances=10,20" ], [ "pnode=1","size=512","vcpus=2-3","vdistances=20,10" ] ] cpus=["0","1","8","9"] This means vcpus 0 and 1, which are assigned to vnode 0, are pinned to pcpu 0 and 1, which are siblings, per the host topology. Similarly, vcpus 2 and 3, assigned to vnode 1, are assigned to two siblings pcpus on pnode 1. This seems to be honoured: # xl vcpu-list 4 Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) test 4 0 0 -b- 10.9 0 / 0-7 test 4 1 1 -b- 7.6 1 / 0-7 test 4 2 8 -b- 0.1 8 / 8-15 test 4 3 9 -b- 0.1 9 / 8-15 And yet, no joy: # ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" # ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" # ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" # ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" # xl vcpu-list 4 Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) test 4 0 0 r-- 16.4 0 / 0-7 test 4 1 1 r-- 12.5 1 / 0-7 test 4 2 8 -b- 0.2 8 / 8-15 test 4 3 9 -b- 0.1 9 / 8-15 So, what am I doing wrong at "following the hw topology"? > > Besides, this is not necessarily a NUMA-only issue, it's a scheduling > > one (inside the guest) as well. > > Sure. That's what Jan said regarding SUSE's xen-kernel. No toplogy info > (or a trivial one) might be better than the wrong one... > Yep. Exacty. As Boris says, this is a generic scheduling issue, although it's tru that it's only (as far as I can tell) with vNUMA that it bite us so hard... I mean, performance are always going to be inconsistent, but it's only in that case that you basically _loose_ some of the vcpus! :-O Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-22 14:50 ` Dario Faggioli @ 2015-07-22 15:32 ` Boris Ostrovsky 2015-07-22 15:49 ` Dario Faggioli 0 siblings, 1 reply; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-22 15:32 UTC (permalink / raw) To: Dario Faggioli, Juergen Gross Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/22/2015 10:50 AM, Dario Faggioli wrote: > On Wed, 2015-07-22 at 16:09 +0200, Juergen Gross wrote: >> On 07/22/2015 03:58 PM, Boris Ostrovsky wrote: >>> What if I configure a guest to follow HW topology? I.e. I pin VCPUs to >>> appropriate cores/threads? With elfnote I am stuck with disabled topology. >> Add an option to do exactly that: follow HW topology (pin vcpus, >> configure vnuma)? >> > I thought about configuring things in such a way that they match the > host topology, as Boris is suggesting, too. And in that case, I think > arranging for doing so in toolstack, if PV vNUMA is identified (as I > think Juergen is suggesting) seems a good approach. > > However, when I try to do that on my box, manually, but I don't seem to > be able to. > > Here's what I tried. Since I have this host topology: > cpu_topology : > cpu: core socket node > 0: 0 1 0 > 1: 0 1 0 > 2: 1 1 0 > 3: 1 1 0 > 4: 9 1 0 > 5: 9 1 0 > 6: 10 1 0 > 7: 10 1 0 > 8: 0 0 1 > 9: 0 0 1 > 10: 1 0 1 > 11: 1 0 1 > 12: 9 0 1 > 13: 9 0 1 > 14: 10 0 1 > 15: 10 0 1 > > I configured the guest like this: > vcpus = '4' > memory = '1024' > vnuma = [ [ "pnode=0","size=512","vcpus=0-1","vdistances=10,20" ], > [ "pnode=1","size=512","vcpus=2-3","vdistances=20,10" ] ] > cpus=["0","1","8","9"] > > This means vcpus 0 and 1, which are assigned to vnode 0, are pinned to > pcpu 0 and 1, which are siblings, per the host topology. > Similarly, vcpus 2 and 3, assigned to vnode 1, are assigned to two > siblings pcpus on pnode 1. > > This seems to be honoured: > # xl vcpu-list 4 > Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) > test 4 0 0 -b- 10.9 0 / 0-7 > test 4 1 1 -b- 7.6 1 / 0-7 > test 4 2 8 -b- 0.1 8 / 8-15 > test 4 3 9 -b- 0.1 9 / 8-15 > > And yet, no joy: > # ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" > # ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" > # ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" > # ssh root@192.168.1.101 "yes > /dev/null 2>&1 &" > # xl vcpu-list 4 > Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) > test 4 0 0 r-- 16.4 0 / 0-7 > test 4 1 1 r-- 12.5 1 / 0-7 > test 4 2 8 -b- 0.2 8 / 8-15 > test 4 3 9 -b- 0.1 9 / 8-15 > > So, what am I doing wrong at "following the hw topology"? > >>> Besides, this is not necessarily a NUMA-only issue, it's a scheduling >>> one (inside the guest) as well. >> Sure. That's what Jan said regarding SUSE's xen-kernel. No toplogy info >> (or a trivial one) might be better than the wrong one... >> > Yep. Exacty. As Boris says, this is a generic scheduling issue, although > it's tru that it's only (as far as I can tell) with vNUMA that it bite > us so hard... I am not sure that it's only vNUMA. It's just that with vNUMA we can see a warning (on your system) that something goes wrong. In other cases (like scheduling, or sizing objects based on discovered cache sizes) we don't see anything in the log but system/programs are making wrong decisions. (And your results above may well be the example of that) -boris > I mean, performance are always going to be inconsistent, > but it's only in that case that you basically _loose_ some of the > vcpus! :-O > > Dario ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-22 15:32 ` Boris Ostrovsky @ 2015-07-22 15:49 ` Dario Faggioli 2015-07-22 18:10 ` Boris Ostrovsky 0 siblings, 1 reply; 95+ messages in thread From: Dario Faggioli @ 2015-07-22 15:49 UTC (permalink / raw) To: Boris Ostrovsky Cc: Juergen Gross, Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 1830 bytes --] On Wed, 2015-07-22 at 11:32 -0400, Boris Ostrovsky wrote: > On 07/22/2015 10:50 AM, Dario Faggioli wrote: > > Yep. Exacty. As Boris says, this is a generic scheduling issue, although > > it's tru that it's only (as far as I can tell) with vNUMA that it bite > > us so hard... > > I am not sure that it's only vNUMA. It's just that with vNUMA we can see > a warning (on your system) that something goes wrong. In other cases > (like scheduling, or sizing objects based on discovered cache sizes) we > don't see anything in the log but system/programs are making wrong > decisions. > I'm not questioning that the guest scheduler is put in a position where it may make weird and/or wrong choices... Let's put it like this: on that very test box, or any other NUMA box I've worked on, I've never seen performance affected _so_much_ as in the vNUMA case. In fact, of course there are other issues (like the ones you're mentioning, caused by this), but it's only with vNUMA that I see 2 out of 4 vcpus completely lost! :-/ > (And your results above may well be the example of that) > Right. BTW, the example was meant at investigating what you suggested, i.e., to have the guest topology follow the host topology. I tried, but could not achieve it. Do you think I'm doing something wrong? May it be that it's not always doable (ISTR Andrew saying that it is also the pcpu where the vcpus are created that matters)? I'm (re)asking because, if you think that's feasible, I can work on that. I fit's not, well... :-/ Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-22 15:49 ` Dario Faggioli @ 2015-07-22 18:10 ` Boris Ostrovsky 2015-07-23 7:25 ` Jan Beulich 2015-07-23 13:46 ` Dario Faggioli 0 siblings, 2 replies; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-22 18:10 UTC (permalink / raw) To: Dario Faggioli Cc: Juergen Gross, Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel On 07/22/2015 11:49 AM, Dario Faggioli wrote: > On Wed, 2015-07-22 at 11:32 -0400, Boris Ostrovsky wrote: >> On 07/22/2015 10:50 AM, Dario Faggioli wrote: >>> Yep. Exacty. As Boris says, this is a generic scheduling issue, although >>> it's tru that it's only (as far as I can tell) with vNUMA that it bite >>> us so hard... >> I am not sure that it's only vNUMA. It's just that with vNUMA we can see >> a warning (on your system) that something goes wrong. In other cases >> (like scheduling, or sizing objects based on discovered cache sizes) we >> don't see anything in the log but system/programs are making wrong >> decisions. >> > I'm not questioning that the guest scheduler is put in a position where > it may make weird and/or wrong choices... Let's put it like this: on > that very test box, or any other NUMA box I've worked on, I've never > seen performance affected _so_much_ as in the vNUMA case. > > In fact, of course there are other issues (like the ones you're > mentioning, caused by this), but it's only with vNUMA that I see 2 out > of 4 vcpus completely lost! :-/ My guess would be that scheduling domains are not set up properly. And perhaps because your test has lots of IO the load doesn't cross thresholds that are needed to switch to another domain? Does it work the same way if you run a compiled busy loop (i.e. 'main(){while(1);}')? > >> (And your results above may well be the example of that) >> > Right. BTW, the example was meant at investigating what you suggested, > i.e., to have the guest topology follow the host topology. I tried, but > could not achieve it. Do you think I'm doing something wrong? May it be > that it's not always doable (ISTR Andrew saying that it is also the pcpu > where the vcpus are created that matters)? I don't think this is currently doable with what we have for CPUID support in xl syntax. I am pretty sure we need to at least be able to specify all leaf 4's indexes. And we can't. BTW, irrespective of this particular problem, adding support for indexed CPUID leaves would be a good idea. -boris > > I'm (re)asking because, if you think that's feasible, I can work on > that. I fit's not, well... :-/ ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-22 18:10 ` Boris Ostrovsky @ 2015-07-23 7:25 ` Jan Beulich 2015-07-24 16:03 ` Boris Ostrovsky 2015-07-23 13:46 ` Dario Faggioli 1 sibling, 1 reply; 95+ messages in thread From: Jan Beulich @ 2015-07-23 7:25 UTC (permalink / raw) To: Boris Ostrovsky Cc: Elena Ufimtseva, Juergen Gross, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, xen-devel >>> On 22.07.15 at 20:10, <boris.ostrovsky@oracle.com> wrote: > I don't think this is currently doable with what we have for CPUID > support in xl syntax. I am pretty sure we need to at least be able to > specify all leaf 4's indexes. And we can't. > > BTW, irrespective of this particular problem, adding support for indexed > CPUID leaves would be a good idea. So what is it that makes you think this doesn't work already? xc_cpuid_policy() intentionally has "input" as a pointer, allowing not just EAX to be specified. And libxl_cpuid.c also has static const char *input_names[2] = { "leaf", "subleaf" }; Jan ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-23 7:25 ` Jan Beulich @ 2015-07-24 16:03 ` Boris Ostrovsky 0 siblings, 0 replies; 95+ messages in thread From: Boris Ostrovsky @ 2015-07-24 16:03 UTC (permalink / raw) To: Jan Beulich Cc: Elena Ufimtseva, Juergen Gross, Wei Liu, Andrew Cooper, Dario Faggioli, David Vrabel, xen-devel On 07/23/2015 03:25 AM, Jan Beulich wrote: >>>> On 22.07.15 at 20:10, <boris.ostrovsky@oracle.com> wrote: >> I don't think this is currently doable with what we have for CPUID >> support in xl syntax. I am pretty sure we need to at least be able to >> specify all leaf 4's indexes. And we can't. >> >> BTW, irrespective of this particular problem, adding support for indexed >> CPUID leaves would be a good idea. > So what is it that makes you think this doesn't work already? > xc_cpuid_policy() intentionally has "input" as a pointer, > allowing not just EAX to be specified. And libxl_cpuid.c also > has > > static const char *input_names[2] = { "leaf", "subleaf" }; > Right. I wasn't aware of this and until I looked at libxl_cpuid_parse_config_xend() I wouldn't have known what the syntax for specifying subleaves is because we don't describe it in xl man page (at least I can't find it). In which case we may be able to adjust them and see what effect they have. -boris ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-22 18:10 ` Boris Ostrovsky 2015-07-23 7:25 ` Jan Beulich @ 2015-07-23 13:46 ` Dario Faggioli 1 sibling, 0 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-23 13:46 UTC (permalink / raw) To: Boris Ostrovsky Cc: Juergen Gross, Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 1057 bytes --] On Wed, 2015-07-22 at 14:10 -0400, Boris Ostrovsky wrote: > On 07/22/2015 11:49 AM, Dario Faggioli wrote: > > In fact, of course there are other issues (like the ones you're > > mentioning, caused by this), but it's only with vNUMA that I see 2 out > > of 4 vcpus completely lost! :-/ > > My guess would be that scheduling domains are not set up properly. And > perhaps because your test has lots of IO the load doesn't cross > thresholds that are needed to switch to another domain? > I also things sched domains are involved. However, my test has no IO at all, actually. > Does it work the same way if you run a compiled busy loop (i.e. > 'main(){while(1);}')? > Well, I can try, but I think that 'yes > /dev/null' (a bunch of them), is really close to that. Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-16 16:59 ` Boris Ostrovsky 2015-07-17 6:09 ` Jan Beulich @ 2015-07-17 10:17 ` Andrew Cooper 1 sibling, 0 replies; 95+ messages in thread From: Andrew Cooper @ 2015-07-17 10:17 UTC (permalink / raw) To: Boris Ostrovsky, Jan Beulich Cc: Elena Ufimtseva, Wei Liu, Dario Faggioli, David Vrabel, xen-devel On 16/07/15 17:59, Boris Ostrovsky wrote: > On 07/16/2015 12:39 PM, Andrew Cooper wrote: >> On 16/07/15 17:29, Jan Beulich wrote: >>>>>> On 16.07.15 at 17:50, <boris.ostrovsky@oracle.com> wrote: >>>> Can't we set leaf 1's EBX[32:16] to 1? > > (I obviously fat-fingered this --- I meant EBX[23:16]) > >>> I don't think we should partially overwrite the relevant parts of >>> CPUID output - either all or nothing (so that things at least >>> remain consistent). > > Possibly more, but specifically for Dario's problem I think this could > resolve that. > >> Also, there are no masking/override MSRs for that feature leaf on any >> hardware I am aware of, and a PV guest using plain `cpuid` will not >> observe any attempt of Xen's to control the value. > > True, but we already modify other CPUID bits (e.g. we clear > X86_FEATURE_TOPOEXT) so it won't make things much worse. (And again, > for this specific problem CPUID is queried by guest kernel which uses > PV CPUID instruction). The current state of cpuid handling is fragile to say the least. Overriding that leaf might indeed work, if configured from the toolstack level. It can't be clobbered in the hypervisor (like some of the existing cpuid clobbering) as that would break migration. > > And in general (both for PV and HVM) --- is there any reason to expose > CPU topology at all? I can see it being useful if VCPUs are pinned but > if they are not then it can make performance worse. For better or for worse, lots of software running in VMs have licensing restrictions on CPU topology. XenServer has options to configure the visible topology inside the VM, to work around the issue that currently every vcpu looks like a separate socket. ~Andrew ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-16 10:32 PV-vNUMA issue: topology is misinterpreted by the guest Dario Faggioli 2015-07-16 10:47 ` Jan Beulich @ 2015-07-16 15:26 ` Wei Liu 2015-07-27 15:13 ` David Vrabel 2 siblings, 0 replies; 95+ messages in thread From: Wei Liu @ 2015-07-16 15:26 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On Thu, Jul 16, 2015 at 12:32:42PM +0200, Dario Faggioli wrote: > Hey, > > This started on IRC, but it's actually appropriate to have the > conversation here. > > I just discovered an issue with vNUMA, when PV guests are used. In fact, > creating a 4 vCPUs PV guest, and making up things so that all the 4 > vCPUs should be busy, I see this: > > root@Zhaman:~# xl vcpu-list test > Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) > test 4 0 5 r-- 1481.9 all / 0-7 > test 4 1 2 r-- 1479.4 all / 0-7 > test 4 2 15 -b- 7.5 all / 8-15 > test 4 3 10 -b- 1324.8 all / 8-15 > > Going checking inside the guest, confirms that *everything* runs on > vCPUs 0 and 1. However, using schedtool or taskset, I can force tasks to > execute on vCPUs 2 and 3. > > Inspecting the guest's dmesg, I've seen this: > > [ 0.128416] ------------[ cut here ]------------ > [ 0.128416] WARNING: CPU: 2 PID: 0 at ../arch/x86/kernel/smpboot.c:317 topology_sane.isra.2+0x74/0x88() > [ 0.128416] sched: CPU #2's smt-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency. > [ 0.128416] Modules linked in: > [ 0.128416] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.19.0+ #1 > [ 0.128416] 0000000000000009 ffff88001ee3bdd0 ffffffff81657c7b ffffffff810bbd2c > [ 0.128416] ffff88001ee3be20 ffff88001ee3be10 ffffffff81081510 ffff88001ee3bea0 > [ 0.128416] ffffffff8103aa02 ffff88003ea0a001 0000000000000000 ffff88001f20a040 > [ 0.128416] Call Trace: > [ 0.128416] [<ffffffff81657c7b>] dump_stack+0x4f/0x7b > [ 0.128416] [<ffffffff810bbd2c>] ? up+0x39/0x3e > [ 0.128416] [<ffffffff81081510>] warn_slowpath_common+0xa1/0xbb > [ 0.128416] [<ffffffff8103aa02>] ? topology_sane.isra.2+0x74/0x88 > [ 0.128416] [<ffffffff81081570>] warn_slowpath_fmt+0x46/0x48 > [ 0.128416] [<ffffffff8101eeb1>] ? __cpuid.constprop.0+0x15/0x19 > [ 0.128416] [<ffffffff8103aa02>] topology_sane.isra.2+0x74/0x88 > [ 0.128416] [<ffffffff8103ac70>] set_cpu_sibling_map+0x21a/0x444 > [ 0.128416] [<ffffffff81056ac3>] ? numa_add_cpu+0x98/0x9f > [ 0.128416] [<ffffffff8100b8f2>] cpu_bringup+0x63/0xa8 > [ 0.128416] [<ffffffff8100b945>] cpu_bringup_and_idle+0xe/0x1a > [ 0.128416] ---[ end trace 95bff1aef57ee1b1 ]--- > > So, basically, Linux is complaining that we're trying to put two vCPUs, > that looks to be SMT siblings, on different NUMA nodes. And, yes, I > think this is quite disruptive for the Linux's scheduler internal logic. > > The vnuma bits of the guest config are these: > > vnuma = [ [ "pnode=0","size=512","vcpus=0-1","vdistances=10,20" ], > [ "pnode=1","size=512","vcpus=2-3","vdistances=20,10" ] ] > > From inside the guest, the topology looks to be like this: > > root@test:~# numactl --hardware > available: 2 nodes (0-1) > node 0 cpus: 0 1 > node 0 size: 475 MB > node 0 free: 382 MB > node 1 cpus: 2 3 > node 1 size: 495 MB > node 1 free: 475 MB > node distances: > node 0 1 > 0: 10 10 > 1: 20 10 > > root@test:~# cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list > 0-1 > root@test:~# cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list > 0-3 > root@test:~# cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list > 2-3 > root@test:~# cat /sys/devices/system/cpu/cpu2/topology/core_siblings_list > 0-3 > > So the complain during boot seems to be against 'core_siblings' (which > was not what I expected, but perhaps I misremember the meaning of > "core_siblings" VS. "thread_siblings" VS. smt-siblings in Linux; I'll > double check). > > Anyway, is there anything we can do to fix or workaround things? > IIRC Linux already consumes some bits returned by cpuid anyway, is it possible to generate a "dummy" layout in Linux kernel according to vNUMA information? I had this idea long ago but wasn't quite sure if it's dumb or not. Wei. > Regards, > Dario > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-16 10:32 PV-vNUMA issue: topology is misinterpreted by the guest Dario Faggioli 2015-07-16 10:47 ` Jan Beulich 2015-07-16 15:26 ` Wei Liu @ 2015-07-27 15:13 ` David Vrabel 2015-07-27 16:02 ` Dario Faggioli 2 siblings, 1 reply; 95+ messages in thread From: David Vrabel @ 2015-07-27 15:13 UTC (permalink / raw) To: Dario Faggioli, xen-devel Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Jan Beulich, Boris Ostrovsky On 16/07/15 11:32, Dario Faggioli wrote: > > Anyway, is there anything we can do to fix or workaround things? This thread has gotten a bit long... For Linux I would like to see: 1. No support for NUMA in PV guests -- if you want new MM features in a guest use HVM. 2. For HVM guests, use the existing hardware interfaces to present NUMA topology. i.e., CPUID, ACPI tables etc. This will work for both kernel and userspace and both will see the same topology. This also has the advantage that any hypervisor/toolstack work will also be applicable to other guests (e.g., Windows). David ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 15:13 ` David Vrabel @ 2015-07-27 16:02 ` Dario Faggioli 2015-07-27 16:31 ` David Vrabel 0 siblings, 1 reply; 95+ messages in thread From: Dario Faggioli @ 2015-07-27 16:02 UTC (permalink / raw) To: David Vrabel Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 1391 bytes --] On Mon, 2015-07-27 at 16:13 +0100, David Vrabel wrote: > On 16/07/15 11:32, Dario Faggioli wrote: > > > > Anyway, is there anything we can do to fix or workaround things? > > This thread has gotten a bit long... > Yep, indeed... :-( > For Linux I would like to see: > > 1. No support for NUMA in PV guests -- if you want new MM features in a > guest use HVM. > Wow... Really? What about all the code we have in libxl and Xen to deal exactly with that? What about making it possible to configure vNUMA for Dom0? :-O > 2. For HVM guests, use the existing hardware interfaces to present NUMA > topology. i.e., CPUID, ACPI tables etc. This will work for both kernel > and userspace and both will see the same topology. > > This also has the advantage that any hypervisor/toolstack work will also > be applicable to other guests (e.g., Windows). > Yeah, indeed. That's the downside of Juergen's "Linux scheduler approach". But the issue is there, even without taking vNUMA into account, and I think something like that would really help (only for Dom0, and Linux guests, of course). Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 16:02 ` Dario Faggioli @ 2015-07-27 16:31 ` David Vrabel 2015-07-27 16:33 ` Andrew Cooper 0 siblings, 1 reply; 95+ messages in thread From: David Vrabel @ 2015-07-27 16:31 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, Jan Beulich, xen-devel, Boris Ostrovsky On 27/07/15 17:02, Dario Faggioli wrote: > On Mon, 2015-07-27 at 16:13 +0100, David Vrabel wrote: >> On 16/07/15 11:32, Dario Faggioli wrote: >>> >>> Anyway, is there anything we can do to fix or workaround things? >> >> This thread has gotten a bit long... >> > Yep, indeed... :-( > >> For Linux I would like to see: >> >> 1. No support for NUMA in PV guests -- if you want new MM features in a >> guest use HVM. >> > Wow... Really? What about all the code we have in libxl and Xen to deal > exactly with that? What about making it possible to configure vNUMA for > Dom0? I don't think there is any (much?) PV-specific code in Xen/toolstack for this, right? It's common between HVM and PV, yes? I would prefer effort into making a no-dm HVM dom0 work instead because this is a better long-term solution for dom0. That said, if someone does the work for vNUMA in Linux PV guests and it looks sensible and self-contained then I would probably merge it. >> 2. For HVM guests, use the existing hardware interfaces to present NUMA >> topology. i.e., CPUID, ACPI tables etc. This will work for both kernel >> and userspace and both will see the same topology. >> >> This also has the advantage that any hypervisor/toolstack work will also >> be applicable to other guests (e.g., Windows). >> > Yeah, indeed. That's the downside of Juergen's "Linux scheduler > approach". But the issue is there, even without taking vNUMA into > account, and I think something like that would really help (only for > Dom0, and Linux guests, of course). I disagree. Whether we're using vNUMA or not, Xen should still ensure that the guest kernel and userspace see a consistent and correct topology using the native mechanisms. David ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 16:31 ` David Vrabel @ 2015-07-27 16:33 ` Andrew Cooper 2015-07-27 17:42 ` Dario Faggioli 0 siblings, 1 reply; 95+ messages in thread From: Andrew Cooper @ 2015-07-27 16:33 UTC (permalink / raw) To: David Vrabel, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Jan Beulich, xen-devel, Boris Ostrovsky On 27/07/15 17:31, David Vrabel wrote: > >>> 2. For HVM guests, use the existing hardware interfaces to present NUMA >>> topology. i.e., CPUID, ACPI tables etc. This will work for both kernel >>> and userspace and both will see the same topology. >>> >>> This also has the advantage that any hypervisor/toolstack work will also >>> be applicable to other guests (e.g., Windows). >>> >> Yeah, indeed. That's the downside of Juergen's "Linux scheduler >> approach". But the issue is there, even without taking vNUMA into >> account, and I think something like that would really help (only for >> Dom0, and Linux guests, of course). > I disagree. Whether we're using vNUMA or not, Xen should still ensure > that the guest kernel and userspace see a consistent and correct > topology using the native mechanisms. +1 ~Andrew ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 16:33 ` Andrew Cooper @ 2015-07-27 17:42 ` Dario Faggioli 2015-07-27 17:50 ` Konrad Rzeszutek Wilk 2015-07-27 23:19 ` Andrew Cooper 0 siblings, 2 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-27 17:42 UTC (permalink / raw) To: Andrew Cooper Cc: Elena Ufimtseva, Wei Liu, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 4967 bytes --] On Mon, 2015-07-27 at 17:33 +0100, Andrew Cooper wrote: > On 27/07/15 17:31, David Vrabel wrote: > > > >> Yeah, indeed. That's the downside of Juergen's "Linux scheduler > >> approach". But the issue is there, even without taking vNUMA into > >> account, and I think something like that would really help (only for > >> Dom0, and Linux guests, of course). > > I disagree. Whether we're using vNUMA or not, Xen should still ensure > > that the guest kernel and userspace see a consistent and correct > > topology using the native mechanisms. > > +1 > +1 from me as well. In fact, a mechanism for making exactly such thing happen, was what I was after when starting the thread. Then it came up that CPUID needs to be used for at least two different and potentially conflicting purposes, that we want to support both and that, whether and for whatever reason it's used, Linux configures its scheduler after it, potentially resulting in rather pathological setups. It's at that point that some decoupling started to appear interesting... :-P Also, are we really being consistent? If my methodology is correct (which might not be, please, double check, and sorry for that), I'm seeing quite some inconsistency around: HOST: root@Zhaman:~# xl info -n ... cpu_topology : cpu: core socket node 0: 0 1 0 1: 0 1 0 2: 1 1 0 3: 1 1 0 4: 9 1 0 5: 9 1 0 6: 10 1 0 7: 10 1 0 8: 0 0 1 9: 0 0 1 10: 1 0 1 11: 1 0 1 12: 9 0 1 13: 9 0 1 14: 10 0 1 15: 10 0 1 ... root@Zhaman:~# xl vcpu-list test Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) test 2 0 0 r-- 1.5 0 / all test 2 1 1 r-- 0.2 1 / all test 2 2 8 -b- 2.2 8 / all test 2 3 9 -b- 2.0 9 / all GUEST (HVM, 4 vcpus): root@test:~# cpuid|grep CORE_ID (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=1 (APIC synth): PKG_ID=0 CORE_ID=0 SMT_ID=0 (APIC synth): PKG_ID=0 CORE_ID=0 SMT_ID=1 HOST: root@Zhaman:~# xl vcpu-pin 2 all 0 root@Zhaman:~# xl vcpu-list 2 Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) test 2 0 0 -b- 43.7 0 / all test 2 1 0 -b- 38.4 0 / all test 2 2 0 -b- 36.9 0 / all test 2 3 0 -b- 38.8 0 / all GUEST: root@test:~# cpuid|grep CORE_ID (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 HOST: root@Zhaman:~# xl vcpu-pin 2 0 7 root@Zhaman:~# xl vcpu-pin 2 1 7 root@Zhaman:~# xl vcpu-pin 2 2 15 root@Zhaman:~# xl vcpu-pin 2 3 15 root@Zhaman:~# xl vcpu-list 2 Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) test 2 0 7 -b- 44.3 7 / all test 2 1 7 -b- 38.9 7 / all test 2 2 15 -b- 37.3 15 / all test 2 3 15 -b- 39.2 15 / all GUEST: root@test:~# cpuid|grep CORE_ID (APIC synth): PKG_ID=0 CORE_ID=26 SMT_ID=1 (APIC synth): PKG_ID=0 CORE_ID=26 SMT_ID=1 (APIC synth): PKG_ID=0 CORE_ID=10 SMT_ID=1 (APIC synth): PKG_ID=0 CORE_ID=10 SMT_ID=1 So, it looks to me that: 1) any application using CPUID for either licensing or placement/performance optimization will get (potentially) random results; 2) whatever set of values the kernel used, during guest boot, to build up its internal scheduling data structures, has no guarantee of being related to any value returned by CPUID, at a later point. Hence, I think I'm seeing inconsistency between kernel and userspace (and between userspace and itself, over time) already... Am I overlooking something? (I'll provide the same, for a PV guest, tomorrow.) Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 17:42 ` Dario Faggioli @ 2015-07-27 17:50 ` Konrad Rzeszutek Wilk 2015-07-27 23:19 ` Andrew Cooper 1 sibling, 0 replies; 95+ messages in thread From: Konrad Rzeszutek Wilk @ 2015-07-27 17:50 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, Andrew Cooper, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky . snip.. > So, it looks to me that: > 1) any application using CPUID for either licensing or > placement/performance optimization will get (potentially) random > results; Right, that is a bug that Andrew outlined in this leveling document I believe. We just pluck the cpuid results on whatever PCPU we were running when the toolstack constructs it. > 2) whatever set of values the kernel used, during guest boot, to build > up its internal scheduling data structures, has no guarantee of > being related to any value returned by CPUID, at a later point. Or after a migration. > > Hence, I think I'm seeing inconsistency between kernel and userspace > (and between userspace and itself, over time) already... Am I > overlooking something? No. I think that is bad. We should be providing the same data - unless pinning is used or any other mechanism is employed to change the cpuid. By default the cpuid ought to be same for a guest throughout its life (lets ignore PV which is 'special'). Also oddly, what is with the SMT threads? On my HVM guests, if I allocate four CPUs (vcpu=4), I get four cores and each core says it has four SMT threads? This is based on /proc/cpuinfo: [konrad@build-external linux]$ cat /proc/cpuinfo |grep "core id" core id : 0 core id : 1 core id : 2 core id : 3 [konrad@build-external linux]$ cat /proc/cpuinfo |grep "siblings" siblings : 4 siblings : 4 siblings : 4 siblings : 4 ? > > (I'll provide the same, for a PV guest, tomorrow.) > > Regards, > Dario > > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 17:42 ` Dario Faggioli 2015-07-27 17:50 ` Konrad Rzeszutek Wilk @ 2015-07-27 23:19 ` Andrew Cooper 2015-07-28 3:52 ` Juergen Gross 2015-07-28 9:28 ` Dario Faggioli 1 sibling, 2 replies; 95+ messages in thread From: Andrew Cooper @ 2015-07-27 23:19 UTC (permalink / raw) To: Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1: Type: text/plain, Size: 5487 bytes --] On 27/07/2015 18:42, Dario Faggioli wrote: > On Mon, 2015-07-27 at 17:33 +0100, Andrew Cooper wrote: >> On 27/07/15 17:31, David Vrabel wrote: >>> >>>> Yeah, indeed. That's the downside of Juergen's "Linux scheduler >>>> approach". But the issue is there, even without taking vNUMA into >>>> account, and I think something like that would really help (only for >>>> Dom0, and Linux guests, of course). >>> I disagree. Whether we're using vNUMA or not, Xen should still ensure >>> that the guest kernel and userspace see a consistent and correct >>> topology using the native mechanisms. >> >> +1 >> > +1 from me as well. In fact, a mechanism for making exactly such thing > happen, was what I was after when starting the thread. > > Then it came up that CPUID needs to be used for at least two different > and potentially conflicting purposes, that we want to support both and > that, whether and for whatever reason it's used, Linux configures its > scheduler after it, potentially resulting in rather pathological setups. I don't see what the problem is here. Fundamentally, "NUMA optimise" vs "comply with licence" is a user/admin decision at boot time, and we need not cater to both halves at the same time. Supporting either, as chosen by the admin, is worthwhile. > > > It's at that point that some decoupling started to appear > interesting... :-P > > Also, are we really being consistent? If my methodology is correct > (which might not be, please, double check, and sorry for that), I'm > seeing quite some inconsistency around: > > HOST: > root@Zhaman:~# xl info -n > ... > cpu_topology : > cpu: core socket node > 0: 0 1 0 > 1: 0 1 0 > 2: 1 1 0 > 3: 1 1 0 > 4: 9 1 0 > 5: 9 1 0 > 6: 10 1 0 > 7: 10 1 0 > 8: 0 0 1 > 9: 0 0 1 > 10: 1 0 1 > 11: 1 0 1 > 12: 9 0 1 > 13: 9 0 1 > 14: 10 0 1 > 15: 10 0 1 o_O What kind of system results in this layout? Can you dump the ACPI tables and make them available? > > ... > root@Zhaman:~# xl vcpu-list test > Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) > test 2 0 0 r-- 1.5 0 / all > test 2 1 1 r-- 0.2 1 / all > test 2 2 8 -b- 2.2 8 / all > test 2 3 9 -b- 2.0 9 / all > > GUEST (HVM, 4 vcpus): > root@test:~# cpuid|grep CORE_ID > (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 > (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=1 > (APIC synth): PKG_ID=0 CORE_ID=0 SMT_ID=0 > (APIC synth): PKG_ID=0 CORE_ID=0 SMT_ID=1 > > HOST: > root@Zhaman:~# xl vcpu-pin 2 all 0 > root@Zhaman:~# xl vcpu-list 2 > Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) > test 2 0 0 -b- 43.7 0 / all > test 2 1 0 -b- 38.4 0 / all > test 2 2 0 -b- 36.9 0 / all > test 2 3 0 -b- 38.8 0 / all > > GUEST: > root@test:~# cpuid|grep CORE_ID > (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 > (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 > (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 > (APIC synth): PKG_ID=0 CORE_ID=16 SMT_ID=0 > > HOST: > root@Zhaman:~# xl vcpu-pin 2 0 7 > root@Zhaman:~# xl vcpu-pin 2 1 7 > root@Zhaman:~# xl vcpu-pin 2 2 15 > root@Zhaman:~# xl vcpu-pin 2 3 15 > root@Zhaman:~# xl vcpu-list 2 > Name ID VCPU CPU State Time(s) Affinity (Hard / Soft) > test 2 0 7 -b- 44.3 7 / all > test 2 1 7 -b- 38.9 7 / all > test 2 2 15 -b- 37.3 15 / all > test 2 3 15 -b- 39.2 15 / all > > GUEST: > root@test:~# cpuid|grep CORE_ID > (APIC synth): PKG_ID=0 CORE_ID=26 SMT_ID=1 > (APIC synth): PKG_ID=0 CORE_ID=26 SMT_ID=1 > (APIC synth): PKG_ID=0 CORE_ID=10 SMT_ID=1 > (APIC synth): PKG_ID=0 CORE_ID=10 SMT_ID=1 > > So, it looks to me that: > 1) any application using CPUID for either licensing or > placement/performance optimization will get (potentially) random > results; > 2) whatever set of values the kernel used, during guest boot, to build > up its internal scheduling data structures, has no guarantee of > being related to any value returned by CPUID, at a later point. > > Hence, I think I'm seeing inconsistency between kernel and userspace > (and between userspace and itself, over time) already... Am I > overlooking something? All current CPUID values presented to guests are about as reliable as being picked from /dev/urandom. (This isn't strictly true - the feature flags will be in the right ballpark if the VM has not migrated yet). Fixing this (as described in my feature levelling design document) is sufficiently non-trivial that it has been deferred to post feature-levelling work. ~Andrew [-- Attachment #1.2: Type: text/html, Size: 7657 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 23:19 ` Andrew Cooper @ 2015-07-28 3:52 ` Juergen Gross 2015-07-28 9:40 ` Andrew Cooper 2015-07-28 9:28 ` Dario Faggioli 1 sibling, 1 reply; 95+ messages in thread From: Juergen Gross @ 2015-07-28 3:52 UTC (permalink / raw) To: Andrew Cooper, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 07/28/2015 01:19 AM, Andrew Cooper wrote: > On 27/07/2015 18:42, Dario Faggioli wrote: >> On Mon, 2015-07-27 at 17:33 +0100, Andrew Cooper wrote: >>> On 27/07/15 17:31, David Vrabel wrote: >>>> >>>>> Yeah, indeed. >>>>> That's the downside of Juergen's "Linux scheduler >>>>> approach". But the issue is there, even without taking vNUMA into >>>>> account, and I think something like that would really help (only for >>>>> Dom0, and Linux guests, of course). >>>> I disagree. Whether we're using vNUMA or not, Xen should still ensure >>>> that the guest kernel and userspace see a consistent and correct >>>> topology using the native mechanisms. >>> >>> +1 >>> >> +1 from me as well. In fact, a mechanism for making exactly such thing >> happen, was what I was after when starting the thread. >> >> Then it came up that CPUID needs to be used for at least two different >> and potentially conflicting purposes, that we want to support both and >> that, whether and for whatever reason it's used, Linux configures its >> scheduler after it, potentially resulting in rather pathological setups. >> > I don't see what the problem is here. Fundamentally, "NUMA optimise" vs > "comply with licence" is a user/admin decision at boot time, and we need > not cater to both halves at the same time. > > Supporting either, as chosen by the admin, is worthwhile. Wrong assumption again. *It's not only about NUMA*! The choice is: "comply with license" against "sane scheduling". NUMA just makes it more obvious, that the data the guest's scheduling decisions are based on is garbage as soon as you tell the guest there are hyperthreads without pinning the vcpus. Right now the sibling information is more or less random leading eventually to some vcpus thinking they have no sibling at all. As soon as you deliver sibling information based on vcpu number you might end up with a deterministic bad scheduling behaviour. Juergen ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-28 3:52 ` Juergen Gross @ 2015-07-28 9:40 ` Andrew Cooper 0 siblings, 0 replies; 95+ messages in thread From: Andrew Cooper @ 2015-07-28 9:40 UTC (permalink / raw) To: Juergen Gross, Dario Faggioli Cc: Elena Ufimtseva, Wei Liu, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky On 28/07/15 04:52, Juergen Gross wrote: > On 07/28/2015 01:19 AM, Andrew Cooper wrote: >> On 27/07/2015 18:42, Dario Faggioli wrote: >>> On Mon, 2015-07-27 at 17:33 +0100, Andrew Cooper wrote: >>>> On 27/07/15 17:31, David Vrabel wrote: >>>>> >>>>>> Yeah, indeed. >>>>>> That's the downside of Juergen's "Linux scheduler >>>>>> approach". But the issue is there, even without taking vNUMA into >>>>>> account, and I think something like that would really help (only >>>>>> for >>>>>> Dom0, and Linux guests, of course). >>>>> I disagree. Whether we're using vNUMA or not, Xen should still >>>>> ensure >>>>> that the guest kernel and userspace see a consistent and correct >>>>> topology using the native mechanisms. >>>> >>>> +1 >>>> >>> +1 from me as well. In fact, a mechanism for making exactly such thing >>> happen, was what I was after when starting the thread. >>> >>> Then it came up that CPUID needs to be used for at least two different >>> and potentially conflicting purposes, that we want to support both and >>> that, whether and for whatever reason it's used, Linux configures its >>> scheduler after it, potentially resulting in rather pathological >>> setups. >>> >> I don't see what the problem is here. Fundamentally, "NUMA optimise" vs >> "comply with licence" is a user/admin decision at boot time, and we need >> not cater to both halves at the same time. >> >> Supporting either, as chosen by the admin, is worthwhile. > > Wrong assumption again. *It's not only about NUMA*! > The choice is: "comply with license" against "sane scheduling". NUMA > just makes it more obvious, that the data the guest's scheduling > decisions are based on is garbage as soon as you tell the guest there > are hyperthreads without pinning the vcpus. Fine. s/NUMA/scheduling/ But the point still stands. The CPUID information should be consistent with other sources of information. ~Andrew ^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: PV-vNUMA issue: topology is misinterpreted by the guest 2015-07-27 23:19 ` Andrew Cooper 2015-07-28 3:52 ` Juergen Gross @ 2015-07-28 9:28 ` Dario Faggioli 1 sibling, 0 replies; 95+ messages in thread From: Dario Faggioli @ 2015-07-28 9:28 UTC (permalink / raw) To: Andrew Cooper Cc: Elena Ufimtseva, Wei Liu, David Vrabel, Jan Beulich, xen-devel, Boris Ostrovsky [-- Attachment #1.1.1: Type: text/plain, Size: 3627 bytes --] On Tue, 2015-07-28 at 00:19 +0100, Andrew Cooper wrote: > On 27/07/2015 18:42, Dario Faggioli wrote: > > On Mon, 2015-07-27 at 17:33 +0100, Andrew Cooper wrote: >> On > 27/07/15 17:31, David Vrabel wrote: >>> >>>> Yeah, indeed. That's the > downside of Juergen's "Linux scheduler >>>> approach". But the issue > is there, even without taking vNUMA into >>>> account, and I think > something like that would really help (only for >>>> Dom0, and Linux > guests, of course). >>> I disagree. Whether we're using vNUMA or not, > Xen should still ensure >>> that the guest kernel and userspace see a > consistent and correct >>> topology using the native mechanisms. >> >> > +1 >> > +1 from me as well. In fact, a mechanism for making exactly > such thing > happen, was what I was after when starting the thread. > > > Then it came up that CPUID needs to be used for at least two > different > and potentially conflicting purposes, that we want to > support both and > that, whether and for whatever reason it's used, > Linux configures its > scheduler after it, potentially resulting in > rather pathological setups. > > I don't see what the problem is here. Fundamentally, "NUMA optimise" > vs "comply with licence" is a user/admin decision at boot time, and we > need not cater to both halves at the same time. > > Supporting either, as chosen by the admin, is worthwhile. > I agree (with the added consideration that, as Juergen is saying, this is not only related to [v]NUMA). When optimising for placement/NUMA/scheduling/etc, I really don't see any problem at all (once more, that's what I was after when starting the thread!). An issue might be that, when 'optimise for licenses', especially (but not limiting to) if in combination with vNUMA, you risk putting the guest scheduler in a really weird situation, hence the idea of taking care of that in some other way. Anyway, if Juergen manages to give us some numbers, we may have a better picture of whether and how much this matters. > > > > It's at that point that some decoupling started to appear > > interesting... :-P > > Also, are we really being consistent? If my > methodology is correct > (which might not be, please, double check, > and sorry for that), I'm > seeing quite some inconsistency around: > > > HOST: > root@Zhaman:~# xl info -n > ... > cpu_topology : > > cpu: core socket node > 0: 0 1 0 > > 1: 0 1 0 > 2: 1 1 0 > 3: > 1 1 0 > 4: 9 1 0 > 5: 9 > 1 0 > 6: 10 1 0 > 7: 10 1 > 0 > 8: 0 0 1 > 9: 0 0 1 > > 10: 1 0 1 > 11: 1 0 1 > > 12: 9 0 1 > 13: 9 0 1 > > 14: 10 0 1 > 15: 10 0 1 > > o_O > > What kind of system results in this layout? Can you dump the ACPI > tables and make them available? > This is the test box I use for day to day development and testing... I've done an acpidump (while on baremetal), and am attaching the result. Let me know if I should do anything more or different. If I can ask, what is it that looks weird? Thanks and Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.1.2: acpi.log --] [-- Type: text/x-log, Size: 199195 bytes --] APIC @ 0x0000000000000000 0000: 41 50 49 43 6A 02 00 00 01 F6 44 45 4C 4C 20 20 APICj.....DELL 0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20 B10K .....ASL 0020: 61 00 00 00 00 00 E0 FE 01 00 00 00 00 08 01 20 a.............. 0030: 01 00 00 00 00 08 02 22 01 00 00 00 00 08 03 32 .......".......2 0040: 01 00 00 00 00 08 04 34 01 00 00 00 00 08 05 00 .......4........ 0050: 01 00 00 00 00 08 06 02 01 00 00 00 00 08 07 12 ................ 0060: 01 00 00 00 00 08 08 14 01 00 00 00 00 08 09 21 ...............! 0070: 01 00 00 00 00 08 0A 23 01 00 00 00 00 08 0B 33 .......#.......3 0080: 01 00 00 00 00 08 0C 35 01 00 00 00 00 08 0D 01 .......5........ 0090: 01 00 00 00 00 08 0E 03 01 00 00 00 00 08 0F 13 ................ 00A0: 01 00 00 00 00 08 10 15 01 00 00 00 00 08 11 00 ................ 00B0: 00 00 00 00 00 08 12 00 00 00 00 00 00 08 13 00 ................ 00C0: 00 00 00 00 00 08 14 00 00 00 00 00 00 08 15 00 ................ 00D0: 00 00 00 00 00 08 16 00 00 00 00 00 00 08 17 00 ................ 00E0: 00 00 00 00 00 08 18 00 00 00 00 00 00 08 19 00 ................ 00F0: 00 00 00 00 00 08 1A 00 00 00 00 00 00 08 1B 00 ................ 0100: 00 00 00 00 00 08 1C 00 00 00 00 00 00 08 1D 00 ................ 0110: 00 00 00 00 00 08 1E 00 00 00 00 00 00 08 1F 00 ................ 0120: 00 00 00 00 00 08 20 00 00 00 00 00 00 08 21 00 ...... .......!. 0130: 00 00 00 00 00 08 22 00 00 00 00 00 00 08 23 00 ......".......#. 0140: 00 00 00 00 00 08 24 00 00 00 00 00 00 08 25 00 ......$.......%. 0150: 00 00 00 00 00 08 26 00 00 00 00 00 00 08 27 00 ......&.......'. 0160: 00 00 00 00 00 08 28 00 00 00 00 00 00 08 29 00 ......(.......). 0170: 00 00 00 00 00 08 2A 00 00 00 00 00 00 08 2B 00 ......*.......+. 0180: 00 00 00 00 00 08 2C 00 00 00 00 00 00 08 2D 00 ......,.......-. 0190: 00 00 00 00 00 08 2E 00 00 00 00 00 00 08 2F 00 ............../. 01A0: 00 00 00 00 00 08 30 00 00 00 00 00 00 08 31 00 ......0.......1. 01B0: 00 00 00 00 00 08 32 00 00 00 00 00 00 08 33 00 ......2.......3. 01C0: 00 00 00 00 00 08 34 00 00 00 00 00 00 08 35 00 ......4.......5. 01D0: 00 00 00 00 00 08 36 00 00 00 00 00 00 08 37 00 ......6.......7. 01E0: 00 00 00 00 00 08 38 00 00 00 00 00 00 08 39 00 ......8.......9. 01F0: 00 00 00 00 00 08 3A 00 00 00 00 00 00 08 3B 00 ......:.......;. 0200: 00 00 00 00 00 08 3C 00 00 00 00 00 00 08 3D 00 ......<.......=. 0210: 00 00 00 00 00 08 3E 00 00 00 00 00 00 08 3F 00 ......>.......?. 0220: 00 00 00 00 00 08 40 00 00 00 00 00 01 0C 08 00 ......@......... 0230: 00 00 C0 FE 00 00 00 00 01 0C 09 00 00 00 C8 FE ................ 0240: 18 00 00 00 01 0C 0A 00 00 80 C8 FE 30 00 00 00 ............0... 0250: 02 0A 00 00 02 00 00 00 00 00 02 0A 00 09 09 00 ................ 0260: 00 00 0D 00 04 06 FF 0D 00 01 .......... ASF! @ 0x0000000000000000 0000: 41 53 46 21 96 00 00 00 20 84 44 45 4C 4C 20 20 ASF!.... .DELL 0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20 B10K .....ASL 0020: 61 00 00 00 04 00 07 00 00 01 88 01 00 2C 00 00 a............,.. 0030: 00 03 0C 89 04 01 01 05 6F 00 68 08 88 17 00 89 ........o.h..... 0040: 04 04 04 07 6F 00 68 20 88 03 00 89 05 01 01 19 ....o.h ........ 0050: 6F 00 68 20 88 22 00 02 00 18 00 04 04 00 00 02 o.h .".......... 0060: 88 00 01 01 88 00 02 00 88 00 03 03 88 00 04 03 ................ 0070: 00 17 00 22 18 00 00 00 03 0F 01 00 00 00 00 00 ..."............ 0080: 00 00 00 00 00 00 80 00 10 00 FF 00 6E 02 00 00 ............n... 0090: 02 A2 00 00 00 00 ...... BOOT @ 0x0000000000000000 0000: 42 4F 4F 54 28 00 00 00 01 04 44 45 4C 4C 20 20 BOOT(.....DELL 0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20 B10K .....ASL 0020: 61 00 00 00 7A 00 00 00 a...z... DMAR @ 0x0000000000000000 0000: 44 4D 41 52 00 01 00 00 01 5A 44 45 4C 4C 20 20 DMAR.....ZDELL 0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20 B10K .....ASL 0020: 61 00 00 00 27 01 00 00 00 00 00 00 00 00 00 00 a...'........... 0030: 00 00 28 00 00 00 00 00 00 E0 FF DF 00 00 00 00 ..(............. 0040: 02 08 00 00 00 20 07 00 02 08 00 00 00 20 09 00 ..... ....... .. 0050: 03 08 00 00 0A 20 13 00 00 00 20 00 01 00 00 00 ..... .... ..... 0060: 00 00 DC FE 00 00 00 00 03 08 00 00 09 00 13 00 ................ 0070: 03 08 00 00 08 00 1F 07 01 00 58 00 00 00 00 00 ..........X..... 0080: 00 80 E5 DB 00 00 00 00 FF FF E6 DB 00 00 00 00 ................ 0090: 01 08 00 00 00 00 1D 00 01 08 00 00 00 00 1D 01 ................ 00A0: 01 08 00 00 00 00 1D 02 01 08 00 00 00 00 1D 07 ................ 00B0: 01 08 00 00 00 00 1A 00 01 08 00 00 00 00 1A 01 ................ 00C0: 01 08 00 00 00 00 1A 02 01 08 00 00 00 00 1A 07 ................ 00D0: 02 00 18 00 00 00 00 00 02 08 00 00 00 00 03 00 ................ 00E0: 02 08 00 00 00 00 07 00 02 00 18 00 00 00 00 00 ................ 00F0: 02 08 00 00 00 20 07 00 02 08 00 00 00 20 09 00 ..... ....... .. DSDT @ 0x0000000000000000 0000: 44 53 44 54 B9 55 00 00 01 43 44 45 4C 4C 00 00 DSDT.U...CDELL.. 0010: 64 74 5F 65 78 00 00 00 00 10 00 00 49 4E 54 4C dt_ex.......INTL 0020: 24 06 05 20 14 07 44 42 49 4E 00 A3 10 47 05 5C $.. ..DBIN...G.\ 0030: 00 5B 82 1F 2E 5F 53 42 5F 56 42 54 4E 08 5F 48 .[..._SB_VBTN._H 0040: 49 44 0C 41 D0 0C 0C 08 5F 50 52 57 12 05 02 01 ID.A...._PRW.... 0050: 0A 04 5B 80 53 53 54 53 01 0B 28 08 01 5B 81 0B ..[.SSTS..(..[.. 0060: 53 53 54 53 41 54 53 54 53 01 5B 80 53 41 43 54 SSTSATSTS.[.SACT 0070: 01 0B 2A 08 01 5B 81 0D 53 41 43 54 41 00 02 54 ..*..[..SACTA..T 0080: 50 4F 4C 01 10 25 5F 47 50 45 14 1F 5F 4C 30 30 POL..%_GPE.._L00 0090: 00 70 00 54 50 4F 4C 70 01 54 53 54 53 86 5C 2E .p.TPOLp.TSTS.\. 00A0: 5F 53 42 5F 56 42 54 4E 0A 02 10 30 5F 53 42 5F _SB_VBTN...0_SB_ 00B0: 14 2A 5F 49 4E 49 00 A0 23 5B 12 5F 4F 53 49 60 .*_INI..#[._OSI` 00C0: A0 1A 5F 4F 53 49 0D 57 69 6E 64 6F 77 73 20 32 .._OSI.Windows 2 00D0: 30 30 31 00 70 0A 04 4D 53 4F 53 10 4A 26 5C 00 001.p..MSOS.J&\. 00E0: 08 4D 53 4F 53 00 5B 01 4D 54 58 5F 01 14 38 43 .MSOS.[.MTX_..8C 00F0: 4D 52 44 01 5B 23 53 4D 49 4D FF FF 70 68 60 7B MRD.[#SMIM..ph`{ 0100: 60 0A 7F 60 70 60 53 4D 49 44 70 0A 84 53 4D 49 `..`p`SMIDp..SMI 0110: 43 70 0A 85 53 4D 49 43 70 53 4D 49 44 60 5B 27 Cp..SMICpSMID`[' 0120: 53 4D 49 4D A4 60 14 36 43 4D 57 52 02 5B 23 53 SMIM.`.6CMWR.[#S 0130: 4D 49 4D FF FF 70 68 60 7D 60 0A 80 60 70 60 53 MIM..ph`}`..`p`S 0140: 4D 49 44 70 0A 84 53 4D 49 43 70 69 53 4D 49 44 MIDp..SMICpiSMID 0150: 70 0A 85 53 4D 49 43 5B 27 53 4D 49 4D 14 11 47 p..SMIC['SMIM..G 0160: 43 4B 42 00 A4 7B 0A 20 43 4D 52 44 0A 26 00 14 CKB..{. CMRD.&.. 0170: 11 47 43 4D 53 00 A4 7B 0A 40 43 4D 52 44 0A 6F .GCMS..{.@CMRD.o 0180: 00 14 11 47 43 4F 4E 00 A4 7B 0A 04 43 4D 52 44 ...GCON..{..CMRD 0190: 0A 55 00 14 14 47 43 55 43 00 7B 01 43 4D 52 44 .U...GCUC.{.CMRD 01A0: 0A 22 60 A4 7F 01 60 00 14 11 47 43 53 31 00 A4 ."`...`...GCS1.. 01B0: 7B 0A 03 43 4D 52 44 0A 25 00 14 11 47 43 53 32 {..CMRD.%...GCS2 01C0: 00 A4 7B 0A 0C 43 4D 52 44 0A 25 00 14 11 47 43 ..{..CMRD.%...GC 01D0: 46 44 00 A4 7B 0A 18 43 4D 52 44 0A 55 00 14 11 FD..{..CMRD.U... 01E0: 47 43 50 50 00 A4 7B 0A 70 43 4D 52 44 0A 23 00 GCPP..{.pCMRD.#. 01F0: 14 11 47 43 54 50 00 A4 7B 0A 40 43 4D 52 44 0A ..GCTP..{.@CMRD. 0200: 22 00 14 11 47 43 4E 43 00 A4 7B 0A 03 43 4D 52 "...GCNC..{..CMR 0210: 44 0A 59 00 14 4A 10 48 41 43 4B 00 A0 1E 93 4D D.Y..J.HACK....M 0220: 53 4F 53 0A 04 70 43 4D 52 44 0A 6F 60 7D 60 0A SOS..pCMRD.o`}`. 0230: 04 60 43 4D 57 52 0A 6F 60 A4 01 A1 15 70 43 4D .`CMWR.o`....pCM 0240: 52 44 0A 6F 60 7B 60 0A FB 60 43 4D 57 52 0A 6F RD.o`{`..`CMWR.o 0250: 60 72 87 5F 4F 53 5F 01 60 70 60 61 08 42 55 46 `r._OS_.`p`a.BUF 0260: 30 11 02 60 08 42 55 46 31 11 02 60 08 4F 53 4E 0..`.BUF1..`.OSN 0270: 54 01 08 4F 53 39 38 01 70 5F 4F 53 5F 42 55 46 T..OS98.p_OS_BUF 0280: 30 70 0D 4D 69 63 72 6F 73 6F 66 74 20 57 69 6E 0p.Microsoft Win 0290: 64 6F 77 73 20 4E 54 00 42 55 46 31 A2 1F 60 76 dows NT.BUF1..`v 02A0: 60 A0 12 93 83 88 42 55 46 30 60 00 83 88 42 55 `.....BUF0`...BU 02B0: 46 31 60 00 A1 07 70 00 4F 53 4E 54 A0 0E 4F 53 F1`...p.OSNT..OS 02C0: 4E 54 70 0A 03 4D 53 4F 53 A4 01 A1 43 05 70 0D NTp..MSOS...C.p. 02D0: 4D 69 63 72 6F 73 6F 66 74 20 57 69 6E 64 6F 77 Microsoft Window 02E0: 73 00 42 55 46 31 A2 1F 61 76 61 A0 12 93 83 88 s.BUF1..ava..... 02F0: 42 55 46 30 61 00 83 88 42 55 46 31 61 00 A1 07 BUF0a...BUF1a... 0300: 70 00 4F 53 39 38 A0 0D 4F 53 39 38 70 01 4D 53 p.OS98..OS98p.MS 0310: 4F 53 A4 00 A1 0A 70 0A 02 4D 53 4F 53 A4 00 14 OS....p..MSOS... 0320: 26 49 53 4C 49 00 A0 1B 5B 12 5F 4F 53 49 60 A0 &ISLI...[._OSI`. 0330: 0E 5F 4F 53 49 0D 4C 69 6E 75 78 00 A4 01 A1 03 ._OSI.Linux..... 0340: A4 00 A1 03 A4 00 10 05 5F 53 42 5F 5B 80 43 4D ........_SB_[.CM 0350: 53 5F 01 0A 70 0A 02 5B 81 10 43 4D 53 5F 01 43 S_..p..[..CMS_.C 0360: 4D 53 49 08 43 4D 53 44 08 5B 80 53 4D 49 52 01 MSI.CMSD.[.SMIR. 0370: 0A B2 0A 02 5B 81 10 53 4D 49 52 01 53 4D 49 43 ....[..SMIR.SMIC 0380: 08 53 4D 49 44 08 5B 80 53 54 4F 4E 01 0A 84 01 .SMID.[.STON.... 0390: 5B 81 0B 53 54 4F 4E 01 4D 47 49 43 08 5B 80 53 [..STON.MGIC.[.S 03A0: 49 4F 5F 01 0A 2E 0A 02 5B 81 10 53 49 4F 5F 01 IO_.....[..SIO_. 03B0: 53 49 4F 49 08 53 49 4F 44 08 5B 80 50 4D 31 52 SIOI.SIOD.[.PM1R 03C0: 01 0B 00 08 0A 04 5B 81 1A 50 4D 31 52 01 50 4D ......[..PM1R.PM 03D0: 53 31 08 50 4D 53 32 08 50 4D 45 31 08 50 4D 45 S1.PMS2.PME1.PME 03E0: 32 08 5B 80 47 50 53 54 01 0B 20 08 01 5B 81 0B 2.[.GPST.. ..[.. 03F0: 47 50 53 54 01 47 53 54 30 08 5B 80 47 4C 42 43 GPST.GST0.[.GLBC 0400: 01 0B 28 08 0A 06 5B 81 24 47 4C 42 43 01 54 48 ..(...[.$GLBC.TH 0410: 52 50 08 47 4C 42 54 08 45 4F 53 5F 08 4C 49 44 RP.GLBT.EOS_.LID 0420: 50 08 54 48 4D 45 08 52 49 45 4E 08 5B 80 47 50 P.THME.RIEN.[.GP 0430: 45 43 01 0B 42 08 01 5B 81 12 47 50 45 43 01 54 EC..B..[..GPEC.T 0440: 48 50 4C 01 53 57 47 50 01 00 06 5B 80 50 4D 45 HPL.SWGP...[.PME 0450: 53 01 0B 00 0C 01 5B 81 0B 50 4D 45 53 01 47 53 S.....[..PMES.GS 0460: 54 53 08 5B 01 53 4D 49 4D 01 5B 01 53 4D 49 58 TS.[.SMIM.[.SMIX 0470: 01 08 53 58 58 30 11 04 0B 00 01 08 53 58 58 31 ..SXX0......SXX1 0480: 11 03 0A 08 8B 53 58 58 31 00 53 58 58 32 8B 53 .....SXX1.SXX2.S 0490: 58 58 31 0A 04 53 58 58 33 14 2E 53 4D 49 5F 02 XX1..SXX3..SMI_. 04A0: 5B 23 53 4D 49 4D FF FF 70 69 53 4D 49 44 70 68 [#SMIM..piSMIDph 04B0: 53 4D 49 43 70 4D 47 49 43 60 70 53 4D 49 44 61 SMICpMGIC`pSMIDa 04C0: 5B 27 53 4D 49 4D A4 61 14 28 53 4D 49 31 01 5B ['SMIM.a.(SMI1.[ 04D0: 23 53 4D 49 4D FF FF 70 68 53 4D 49 43 70 4D 47 #SMIM..phSMICpMG 04E0: 49 43 60 70 53 4D 49 44 60 5B 27 53 4D 49 4D A4 IC`pSMID`['SMIM. 04F0: 60 14 37 53 4D 49 32 01 5B 23 53 4D 49 4D FF FF `.7SMI2.[#SMIM.. 0500: 70 68 53 4D 49 43 70 4D 47 49 43 60 70 53 4D 49 phSMICpMGIC`pSMI 0510: 43 61 70 53 4D 49 44 60 79 60 0A 08 60 72 61 60 CapSMID`y`..`ra` 0520: 60 5B 27 53 4D 49 4D A4 60 14 23 53 4D 49 34 01 `['SMIM.`.#SMI4. 0530: 72 68 01 60 70 53 4D 49 32 68 62 70 53 4D 49 32 rh.`pSMI2hbpSMI2 0540: 60 61 79 61 0A 10 61 72 61 62 60 A4 60 14 0D 47 `aya..arab`.`..G 0550: 54 4D 52 00 A4 53 4D 49 34 0A 78 14 0D 47 54 4D TMR..SMI4.x..GTM 0560: 4C 00 A4 53 4D 49 34 0A 74 14 0D 47 54 4D 48 00 L..SMI4.t..GTMH. 0570: A4 53 4D 49 34 0A 76 14 0D 47 54 4F 4D 00 A4 53 .SMI4.v..GTOM..S 0580: 4D 49 34 0A 81 14 17 47 55 53 42 00 A0 07 49 53 MI4....GUSB...IS 0590: 4C 49 A4 00 A1 08 A4 53 4D 49 32 0A BA 14 0E 47 LI.....SMI2....G 05A0: 54 50 4D 00 A4 53 4D 49 5F 0A 9B 00 14 14 53 58 TPM..SMI_.....SX 05B0: 31 30 00 5B 23 53 4D 49 58 FF FF 70 00 53 58 58 10.[#SMIX..p.SXX 05C0: 32 14 31 53 58 33 30 01 70 53 58 58 32 60 75 60 2.1SX30.pSXX2`u` 05D0: A0 22 92 94 60 87 53 58 58 30 8C 53 58 58 30 53 ."..`.SXX0.SXX0S 05E0: 58 58 32 53 58 32 30 70 68 53 58 32 30 70 60 53 XX2SX20phSX20p`S 05F0: 58 58 32 14 1B 53 58 33 33 02 A0 14 95 69 87 68 XX2..SX33....i.h 0600: 8C 68 69 53 58 32 30 53 58 33 30 53 58 32 30 14 .hiSX20SX30SX20. 0610: 16 53 58 33 34 02 70 00 60 A2 0C 95 60 69 53 58 .SX34.p.`...`iSX 0620: 33 33 68 60 75 60 14 29 53 58 58 36 02 70 69 53 33h`u`.)SXX6.piS 0630: 4D 49 44 70 68 53 4D 49 43 70 53 4D 49 43 60 A2 MIDphSMICpSMIC`. 0640: 0B 92 93 60 00 70 53 4D 49 43 60 A4 53 4D 49 44 ...`.pSMIC`.SMID 0650: 14 1D 53 58 58 35 02 A0 16 95 69 87 68 8C 68 69 ..SXX5....i.h.hi 0660: 53 58 32 30 53 58 58 36 0A 98 53 58 32 30 14 23 SX20SXX6..SX20.# 0670: 53 58 58 34 00 53 58 58 36 0A 96 00 70 00 60 A2 SXX4.SXX6...p.`. 0680: 12 95 60 53 58 58 32 53 58 58 35 53 58 58 30 60 ..`SXX2SXX5SXX0` 0690: 75 60 14 1F 53 58 58 38 02 A0 18 95 69 87 68 8C u`..SXX8....i.h. 06A0: 68 69 53 58 32 30 70 53 58 58 36 0A 97 00 53 58 hiSX20pSXX6...SX 06B0: 32 30 14 23 53 58 58 37 00 70 00 60 A2 19 95 60 20.#SXX7.p.`...` 06C0: 53 58 58 33 72 53 58 58 32 60 61 53 58 58 38 53 SXX3rSXX2`aSXX8S 06D0: 58 58 30 61 75 60 14 42 04 53 58 31 31 00 53 58 XX0au`.B.SX11.SX 06E0: 58 34 70 53 58 58 36 0A 99 00 53 58 58 33 72 53 X4pSXX6...SXX3rS 06F0: 58 58 32 53 58 58 33 60 A0 1C 95 87 53 58 58 30 XX2SXX3`....SXX0 0700: 60 70 87 53 58 58 30 60 74 60 53 58 58 32 60 70 `p.SXX0`t`SXX2`p 0710: 60 53 58 58 33 53 58 58 37 14 32 53 58 34 30 00 `SXX3SXX7.2SX40. 0720: 70 53 58 58 32 60 75 60 A0 21 92 94 60 87 53 58 pSXX2`u`.!..`.SX 0730: 58 30 8C 53 58 58 30 53 58 58 32 53 58 32 30 70 X0.SXX0SXX2SX20p 0740: 60 53 58 58 32 A4 53 58 32 30 A4 00 14 35 53 58 `SXX2.SX20...5SX 0750: 34 31 00 70 53 58 58 32 60 72 60 0A 02 60 A0 21 41.pSXX2`r`..`.! 0760: 92 94 60 87 53 58 58 30 8B 53 58 58 30 53 58 58 ..`.SXX0.SXX0SXX 0770: 32 53 58 32 31 70 60 53 58 58 32 A4 53 58 32 31 2SX21p`SXX2.SX21 0780: A4 00 14 35 53 58 34 32 00 70 53 58 58 32 60 72 ...5SX42.pSXX2`r 0790: 60 0A 04 60 A0 21 92 94 60 87 53 58 58 30 8A 53 `..`.!..`.SXX0.S 07A0: 58 58 30 53 58 58 32 53 58 32 32 70 60 53 58 58 XX0SXX2SX22p`SXX 07B0: 32 A4 53 58 32 32 A4 00 14 1C 53 58 34 33 02 A0 2.SX22....SX43.. 07C0: 15 95 69 87 68 8C 68 69 53 58 32 30 70 53 58 34 ..i.h.hiSX20pSX4 07D0: 30 53 58 32 30 14 16 53 58 34 34 02 70 00 60 A2 0SX20..SX44.p.`. 07E0: 0C 95 60 69 53 58 34 33 68 60 75 60 14 0C 53 58 ..`iSX43h`u`..SX 07F0: 31 32 00 5B 27 53 4D 49 58 14 15 47 4D 49 30 00 12.['SMIX..GMI0. 0800: 70 53 4D 49 32 0A D4 60 79 60 0A 10 60 A4 60 14 pSMI2..`y`..`.`. 0810: 21 47 4D 58 30 00 70 53 4D 49 32 0A D5 60 79 60 !GMX0.pSMI2..`y` 0820: 0A 10 60 A0 0B 92 93 60 00 7D 60 0B FF FF 60 A4 ..`....`.}`...`. 0830: 60 14 15 47 4D 49 31 00 70 53 4D 49 32 0A D0 60 `..GMI1.pSMI2..` 0840: 79 60 0A 10 60 A4 60 14 21 47 4D 58 31 00 70 53 y`..`.`.!GMX1.pS 0850: 4D 49 32 0A D1 60 79 60 0A 10 60 A0 0B 92 93 60 MI2..`y`..`....` 0860: 00 7D 60 0B FF FF 60 A4 60 14 16 47 49 42 30 00 .}`...`.`..GIB0. 0870: 70 53 4D 49 32 0A D6 60 7B 60 0B FF FF 60 A4 60 pSMI2..`{`...`.` 0880: 14 16 47 49 58 30 00 70 53 4D 49 32 0A D7 60 7B ..GIX0.pSMI2..`{ 0890: 60 0B FF FF 60 A4 60 14 16 47 49 42 31 00 70 53 `...`.`..GIB1.pS 08A0: 4D 49 32 0A D2 60 7B 60 0B FF FF 60 A4 60 14 16 MI2..`{`...`.`.. 08B0: 47 49 58 31 00 70 53 4D 49 32 0A D3 60 7B 60 0B GIX1.pSMI2..`{`. 08C0: FF FF 60 A4 60 10 4A 80 5F 53 42 5F 5B 82 42 80 ..`.`.J._SB_[.B. 08D0: 41 4D 57 30 5B 01 57 4D 49 58 01 08 5F 48 49 44 AMW0[.WMIX.._HID 08E0: 0D 2A 70 6E 70 30 63 31 34 00 08 5F 55 49 44 00 .*pnp0c14.._UID. 08F0: 14 13 53 54 42 59 03 8C 68 69 54 4D 50 5F 70 6A ..STBY..hiTMP_pj 0900: 54 4D 50 5F 14 13 53 54 57 44 03 8B 68 69 54 4D TMP_..STWD..hiTM 0910: 50 5F 70 6A 54 4D 50 5F 14 13 53 54 44 57 03 8A P_pjTMP_..STDW.. 0920: 68 69 54 4D 50 5F 70 6A 54 4D 50 5F 14 18 43 4C hiTMP_pjTMP_..CL 0930: 42 59 01 70 00 60 A2 0E 95 60 87 68 53 54 42 59 BY.p.`...`.hSTBY 0940: 68 60 00 75 60 08 5F 57 44 47 11 48 06 0A 64 BC h`.u`._WDG.H..d. 0950: DC 9D 8D 97 A9 DA 11 B0 12 B6 22 A1 EF 54 92 41 .........."..T.A 0960: 41 01 00 CE 93 05 A8 97 A9 DA 11 B0 12 B6 22 A1 A.............". 0970: EF 54 92 42 41 01 02 94 59 BB 9D 97 A9 DA 11 B0 .T.BA...Y....... 0980: 12 B6 22 A1 EF 54 92 D0 00 01 08 E0 6C 77 A3 88 .."..T......lw.. 0990: 1E DB 11 A9 8B 08 00 20 0C 9A 66 42 43 01 00 21 ....... ..fBC..! 09A0: 12 90 05 66 D5 D1 11 B2 F0 00 A0 C9 06 29 10 4D ...f.........).M 09B0: 4F 01 00 08 49 4E 46 4F 11 03 0A 80 08 45 43 44 O...INFO.....ECD 09C0: 30 00 14 0E 57 45 44 30 01 70 68 45 43 44 30 A4 0...WED0.phECD0. 09D0: 00 14 08 57 43 41 41 01 A4 00 14 48 06 57 51 41 ...WCAA....H.WQA 09E0: 41 01 5B 23 57 4D 49 58 FF FF 43 4C 42 59 49 4E A.[#WMIX..CLBYIN 09F0: 46 4F A0 0B 92 93 68 00 70 49 4E 46 4F 61 A1 3C FO....h.pINFOa.< 0A00: 53 54 44 57 49 4E 46 4F 00 0C 44 45 4C 4C 53 54 STDWINFO..DELLST 0A10: 44 57 49 4E 46 4F 0A 04 0C 20 57 4D 49 53 54 44 DWINFO... WMISTD 0A20: 57 49 4E 46 4F 0A 08 01 53 54 44 57 49 4E 46 4F WINFO...STDWINFO 0A30: 0A 0C 0B 00 10 70 49 4E 46 4F 61 5B 27 57 4D 49 .....pINFOa['WMI 0A40: 58 A4 61 14 08 57 53 41 41 02 A4 69 14 28 57 4D X.a..WSAA..i.(WM 0A50: 42 41 03 8A 6A 0A 28 57 42 55 46 72 57 42 55 46 BA..j.(WBUFrWBUF 0A60: 0A 2C 61 A0 0F 92 94 61 0B 00 10 70 57 4D 49 5F .,a....a...pWMI_ 0A70: 6A 61 60 A4 60 08 57 4D 49 32 11 03 0A 80 08 57 ja`.`.WMI2.....W 0A80: 4D 49 33 00 14 1C 57 4D 49 36 00 5B 23 57 4D 49 MI3...WMI6.[#WMI 0A90: 58 FF FF 70 00 57 4D 49 33 43 4C 42 59 57 4D 49 X..p.WMI3CLBYWMI 0AA0: 32 14 14 57 4D 49 37 00 5B 23 57 4D 49 58 FF FF 2..WMI7.[#WMIX.. 0AB0: 70 00 57 4D 49 33 14 34 57 4D 49 38 01 70 57 4D p.WMI3.4WMI8.pWM 0AC0: 49 33 60 72 60 0A 02 60 A0 22 92 94 60 87 57 4D I3`r`..`."..`.WM 0AD0: 49 32 8B 57 4D 49 32 57 4D 49 33 57 4D 49 39 70 I2.WMI2WMI3WMI9p 0AE0: 68 57 4D 49 39 70 60 57 4D 49 33 14 4A 06 5F 57 hWMI9p`WMI3.J._W 0AF0: 45 44 01 57 4D 49 36 A0 16 91 92 93 68 0A D0 93 ED.WMI6.....h... 0B00: 45 43 44 30 00 57 4D 49 37 A4 57 4D 49 32 A1 3E ECD0.WMI7.WMI2.> 0B10: 53 58 31 30 53 58 33 30 01 53 58 31 31 70 53 58 SX10SX30.SX11pSX 0B20: 34 31 60 A2 20 94 60 00 57 4D 49 38 60 A2 10 94 41`. .`.WMI8`... 0B30: 60 00 57 4D 49 38 53 58 34 31 74 60 01 60 70 53 `.WMI8SX41t`.`pS 0B40: 58 34 31 60 57 4D 49 38 00 53 58 31 32 57 4D 49 X41`WMI8.SX12WMI 0B50: 37 A4 57 4D 49 32 08 57 51 4D 4F 11 44 57 0B 6F 7.WMI2.WQMO.DW.o 0B60: 05 46 4F 4D 42 01 00 00 00 5F 05 00 00 88 1C 00 .FOMB...._...... 0B70: 00 44 53 00 01 1A 7D DA 54 18 D5 8D 00 01 06 18 .DS...}.T....... 0B80: 42 10 0F 10 22 21 04 12 01 A1 C8 2C 0C 86 10 38 B..."!.....,...8 0B90: 2E 84 1C 40 48 1C 14 4A 08 84 FA 13 C8 AF 00 84 ...@H..J........ 0BA0: 0E 05 C8 14 60 50 80 53 04 11 F4 2A C0 A6 00 93 ....`P.S...*.... 0BB0: 02 2C 0A D0 2E C0 B2 00 DD 02 A4 C3 12 91 E0 28 .,.............( 0BC0: 31 E0 28 9D D8 C2 0D 1B BC 50 14 CD 20 4A 82 CA 1.(......P.. J.. 0BD0: 05 F8 46 10 78 B9 02 24 4F 40 9A 05 18 16 60 5D ..F.x..$O@....`] 0BE0: 80 EC 21 50 A9 43 40 C9 19 02 6A 00 AD 4E 40 F8 ..!P.C@...j..N@. 0BF0: 95 4E 09 49 10 CE 58 C5 E3 6B 16 4D CF 49 CE 31 .N.I..X..k.M.I.1 0C00: E4 78 5C E8 41 F0 40 0A 40 58 78 08 45 80 41 49 .x\.A.@.@Xx.E.AI 0C10: 18 0B 75 31 6A D4 48 D9 80 0C 51 DA A8 D1 03 3A ..u1j.H...Q....: 0C20: BF 23 39 BB A3 3B 92 04 46 3D A6 63 2C 6C 46 42 .#9..;..F=.c,lFB 0C30: 8D D1 1C 14 81 C6 0D DA 12 61 35 AE D8 67 66 E1 .........a5..gf. 0C40: C3 12 C6 11 1C 58 82 46 D1 34 C7 B3 0D 91 E0 20 .....X.F.4..... 0C50: 42 63 64 40 C8 F3 B0 05 7A E4 09 EC 1E 51 0A 11 Bcd@....z....Q.. 0C60: 34 DF 13 A9 51 80 36 0C D9 3A 1B 68 A8 B1 1A 43 4...Q.6..:.h...C 0C70: 11 44 84 A0 51 0C 16 21 54 88 FF 7F 94 A8 A7 14 .D..Q..!T....... 0C80: 24 6A 65 20 42 0B 66 04 66 7F 10 24 C6 99 41 87 $je B.f.f..$..A. 0C90: 05 CB 00 91 11 41 A3 61 67 01 0F C7 33 69 7E 62 .....A.ag...3i~b 0CA0: 1A 9C 09 C6 86 90 06 08 89 3A 38 50 02 4B 19 38 .........:8P.K.8 0CB0: B1 3D 2E 8D EF 8C A3 86 38 F5 33 F3 3F C2 5B F0 .=......8.3.?.[. 0CC0: 11 80 8F C1 83 3D 84 80 47 C8 CE 00 06 C4 7B 9F .....=..G.....{. 0CD0: 34 99 8B CF 02 30 86 0F D7 F8 28 34 1E 76 3E 60 4....0....(4.v>` 0CE0: E3 E2 F0 3E 14 9C 70 B1 20 0A 00 21 59 E7 03 F4 ...>..p. ..!Y... 0CF0: AC 8F 2D E0 C3 40 B3 77 08 42 F0 22 E0 A3 83 8F ..-..@.w.B.".... 0D00: 1B 1E F7 F3 06 18 0E 07 1E 8E 4F 1B C0 65 04 5C ..........O..e.\ 0D10: DA 93 C2 04 92 FC 04 90 18 18 D4 81 C0 07 0B B8 ................ 0D20: 92 E0 50 C3 F3 C4 1E 10 FE FF 47 79 22 2F 06 9E ..P.......Gy"/.. 0D30: FE 63 00 8C 03 82 A7 75 52 BE 79 3C 48 78 50 61 .c.....uR.y<HxPa 0D40: 12 F8 94 C0 D0 F8 71 03 AC A3 C6 1F 10 E0 9D 24 ......q........$ 0D50: CE AF CF 01 E8 D0 70 8A 0C E4 35 E0 A4 4F C9 E3 ......p...5..O.. 0D60: 4B E0 33 07 EC BB C1 61 1C 4C 88 08 EF 01 4F 1D K.3....a.L....O. 0D70: BE 6B 3C 0A 04 8A D0 DB 99 83 9E 42 8C 12 ED AC .k<........B.... 0D80: C2 3C 70 44 F1 91 C3 08 EF 1E BE 13 3C 80 B4 36 .<pD........<..6 0D90: 39 E1 06 7A E6 60 D1 CE 2C B2 00 A2 48 A3 41 9D 9..z.`..,...H.A. 0DA0: 11 7C 1A F0 B4 9E 62 7C 94 30 C8 19 1E D8 73 C2 .|....b|.0....s. 0DB0: 63 80 07 CC EE 07 3E 4E F8 5C 80 77 0D A8 19 FA c.....>N.\.w.... 0DC0: B0 01 E7 D0 81 3F 4D E0 0F 16 F8 F1 F8 9A C3 26 .....?M........& 0DD0: 9C C0 F2 07 81 1A 99 A1 3D CB D3 7A 0D F0 69 C7 ........=..z..i. 0DE0: 04 3E 6F F8 FF FF CF F1 78 C0 AF F8 74 41 EE 0A .>o.....x...tA.. 0DF0: 9E AF CF 2E CC C6 78 50 A3 F0 01 07 77 76 F1 11 ......xP....wv.. 0E00: C0 67 17 E0 39 89 67 09 F0 1E 02 7C 22 89 F7 B0 .g..9.g....|"... 0E10: 05 63 C4 78 C8 33 AE 7A 18 BA 08 58 DD 7D 05 75 .c.x.3.z...X.}.u 0E20: F4 02 13 D4 6B 06 EE F4 02 7C 4E 59 F0 FE FF A7 ....k....|NY.... 0E30: 2C E0 7E 55 E0 47 14 30 40 76 76 3A 11 C2 7B C9 ,.~U.G.0@vv:..{. 0E40: 73 88 6F 57 3E 98 04 79 0E 88 F0 94 C5 EF 03 51 s.oW>..y.......Q 0E50: 62 1E 50 A4 28 46 0C F2 84 E5 EB 49 0C 43 07 0B b.P.(F.....I.C.. 0E60: 17 3E C2 53 16 60 F1 92 85 39 65 C1 7C 1B F8 94 .>.S.`...9e.|... 0E70: 05 8E FF FF 29 0B 5C E3 7E CA 02 66 D2 9F 02 3E ....).\.~..f...> 0E80: D5 F8 09 A0 E8 07 0B 0A E3 53 16 E0 4A DE 01 01 .........S..J... 0E90: 34 67 27 DF 16 0C 76 CC BE 64 F8 94 08 86 43 86 4g'...v..d....C. 0EA0: EF 54 87 F2 C8 F1 14 E0 23 16 D8 E3 1C 03 74 5C .T......#.....t\ 0EB0: F1 11 CB FF FF 23 16 C0 8D 03 08 FE D4 01 EB 1E .....#.......... 0EC0: 10 D6 87 0E E0 21 FB 21 A0 33 8C 25 83 C8 C6 B9 .....!.!.3.%.... 0ED0: 86 8E D1 E2 17 AA 9B 42 EC 83 E1 B2 81 04 EA E1 .......B........ 0EE0: 5A 30 85 44 D1 68 34 06 26 30 82 33 88 01 9D 11 Z0.D.h4.&0.3.... 0EF0: 42 87 32 9C 8A F3 10 EA FF 4F 30 D4 8D 89 CE CE B.2......O0..... 0F00: F3 E7 B7 11 9F 08 0C EC 2B C4 5B 06 58 86 E5 C5 ........+.[.X... 0F10: 3D 01 1C E3 49 26 A8 E6 58 83 9A 83 AF 02 6F 64 =...I&..X.....od 0F20: 26 F0 15 0C 6C A7 19 8C BE 3B 01 28 80 7C 14 F0 &...l....;.(.|.. 0F30: 7D F9 6D 80 CD E2 95 D9 68 3E 7F 22 86 8E 12 33 }.m.....h>."...3 0F40: 74 0A E2 A1 3B E8 D0 D1 C7 01 9F AC 70 C1 0E 5F t...;.......p.._ 0F50: D0 26 77 B4 27 E6 59 78 9E B8 B9 83 E9 88 04 63 .&w.'.Yx.......c 0F60: F0 98 C9 83 59 E0 E4 41 F1 FF 9F 3C 4C 78 4C D8 ....Y..A...<LxL. 0F70: C3 21 3D 74 78 64 7C 9C 3E FD 30 EC D3 39 97 A2 .!=txd|.>.0..9.. 0F80: 67 A4 3B C6 33 17 06 D6 23 E7 B0 46 0B 7B C0 CF g.;.3...#..F.{.. 0F90: 21 BE C3 F8 C0 C3 60 7D 7A 01 C7 F1 0B FE 69 00 !.....`}z.....i. 0FA0: 3C 07 10 8F E1 05 84 1F 5F 74 FE A5 42 17 27 79 <......._t..B.'y 0FB0: 30 A8 D3 14 E0 EA F4 06 9E AB 3E EE 2C 85 FB FF 0.........>.,... 0FC0: 9F A5 60 1C 7E 7D 81 F7 5D CA 08 07 F2 2C 05 F6 ..`.~}..]....,.. 0FD0: D8 CF 10 1D 02 7C 96 02 F8 F1 FA F0 79 06 6E E0 .....|......y.n. 0FE0: D3 2F D0 FA FF 9F 42 C0 7F 8C F0 11 07 77 F4 05 ./....B......w.. 0FF0: 6E 07 63 7E C6 C0 1D C7 80 C7 C9 19 77 9A F0 10 n.c~........w... 1000: F8 00 5A 9D 1E 39 F5 9C 12 EE 38 C0 A7 84 1B 00 ..Z..9....8..... 1010: 26 C0 19 13 50 F5 FF 3F 63 02 63 90 08 EF 0A 51 &...P..?c.c....Q 1020: DF 91 0D F1 8C CC 70 DE 1A 1E 5F 9E 91 C1 79 C6 ......p..._...y. 1030: 04 F8 F3 FF 3F D2 63 6F 0A C1 9F 42 80 89 F8 33 ....?.co...B...3 1040: 26 15 7E C6 84 36 17 4F DB 67 4C 80 7B FF FF 33 &.~..6.O.gL.{..3 1050: 26 E0 3B F2 19 13 D0 73 A4 C2 9D 31 C1 30 CE F7 &.;....s...1.0.. 1060: 27 83 3C 8F F8 40 E0 A3 0C 53 68 D3 A7 46 A3 56 '.<..@...Sh..F.V 1070: 0D CA D4 28 D3 A0 56 9F 4A 8D 19 3B 3A 59 C4 5A ...(..V.J..;:Y.Z 1080: 35 58 87 5A AF 40 2C E9 89 21 10 FF FF 45 79 00 5X.Z.@,..!...Ey. 1090: 61 71 4D 80 30 E1 AB 12 88 63 83 50 B1 3A 20 1A aqM.0....c.P.: . 10A0: 19 A2 41 04 E4 10 3E 80 58 24 10 81 13 25 A0 D4 ..A...>.X$...%.. 10B0: 09 08 13 BA 16 81 58 9E 17 10 16 0E 84 4A 32 03 ......X......J2. 10C0: CE F4 81 08 C8 52 5F 21 02 B2 70 10 01 F9 FF 0F .....R_!..p..... 10D0: 5B 01 57 4D 49 53 01 08 57 4D 30 30 00 14 20 57 [.WMIS..WM00.. W 10E0: 4D 30 32 00 53 58 31 30 53 58 33 30 00 53 58 31 M02.SX10SX30.SX1 10F0: 31 70 53 58 34 32 57 4D 30 30 53 58 31 32 14 2C 1pSX42WM00SX12., 1100: 57 4D 30 33 03 5B 80 57 57 50 52 00 68 01 5B 81 WM03.[.WWPR.h.[. 1110: 0B 57 57 50 52 11 4D 45 4D 57 08 8C 69 6A 57 56 .WWPR.MEMW..ijWV 1120: 41 4C 70 57 56 41 4C 4D 45 4D 57 14 2D 57 4D 30 ALpWVALMEMW.-WM0 1130: 35 03 5B 80 57 57 50 52 00 68 0A 04 5B 81 0B 57 5.[.WWPR.h..[..W 1140: 57 50 52 11 4D 57 33 32 20 8A 69 6A 57 56 41 4C WPR.MW32 .ijWVAL 1150: 70 57 56 41 4C 4D 57 33 32 14 32 57 4D 30 34 03 pWVALMW32.2WM04. 1160: 5B 80 57 52 50 52 00 68 01 5B 81 0B 57 52 50 52 [.WRPR.h.[..WRPR 1170: 11 4D 45 4D 52 08 8C 69 6A 57 56 41 4C 70 4D 45 .MEMR..ijWVALpME 1180: 4D 52 57 56 41 4C 70 00 4D 45 4D 52 14 33 57 4D MRWVALp.MEMR.3WM 1190: 30 36 03 5B 80 57 52 50 52 00 68 0A 04 5B 81 0B 06.[.WRPR.h..[.. 11A0: 57 52 50 52 11 4D 52 33 32 20 8A 69 6A 57 56 41 WRPR.MR32 .ijWVA 11B0: 4C 70 4D 52 33 32 57 56 41 4C 70 00 4D 52 33 32 LpMR32WVALp.MR32 11C0: 14 47 05 57 4D 30 37 02 A0 4D 04 92 94 69 0B 00 .G.WM07..M...i.. 11D0: 01 70 57 4D 30 30 60 70 00 61 A2 15 95 61 69 57 .pWM00`p.a...aiW 11E0: 4D 30 35 60 68 61 72 60 0A 04 60 72 61 0A 04 61 M05`har`..`ra..a 11F0: 53 58 58 36 0A 9A 00 70 57 4D 30 30 60 70 00 61 SXX6...pWM00`p.a 1200: A2 15 95 61 69 57 4D 30 36 60 68 61 72 60 0A 04 ...aiWM06`har`.. 1210: 60 72 61 0A 04 61 A4 68 14 28 57 4D 49 5F 02 5B `ra..a.h.(WMI_.[ 1220: 23 57 4D 49 53 FF FF A0 0B 93 57 4D 30 30 00 57 #WMIS.....WM00.W 1230: 4D 30 32 57 4D 30 37 68 69 5B 27 57 4D 49 53 A4 M02WM07hi['WMIS. 1240: 68 10 46 1A 5F 50 52 5F 5B 83 0B 43 50 55 31 01 h.F._PR_[..CPU1. 1250: 10 08 00 00 06 5B 83 0B 43 50 55 32 02 00 00 00 .....[..CPU2.... 1260: 00 00 5B 83 0B 43 50 55 33 03 00 00 00 00 00 5B ..[..CPU3......[ 1270: 83 0B 43 50 55 34 04 00 00 00 00 00 5B 83 0B 43 ..CPU4......[..C 1280: 50 55 35 05 00 00 00 00 00 5B 83 0B 43 50 55 36 PU5......[..CPU6 1290: 06 00 00 00 00 00 5B 83 0B 43 50 55 37 07 00 00 ......[..CPU7... 12A0: 00 00 00 5B 83 0B 43 50 55 38 08 00 00 00 00 00 ...[..CPU8...... 12B0: 5B 83 0B 43 50 55 39 09 00 00 00 00 00 5B 83 0B [..CPU9......[.. 12C0: 43 50 31 30 0A 00 00 00 00 00 5B 83 0B 43 50 31 CP10......[..CP1 12D0: 31 0B 00 00 00 00 00 5B 83 0B 43 50 31 32 0C 00 1......[..CP12.. 12E0: 00 00 00 00 5B 83 0B 43 50 31 33 0D 00 00 00 00 ....[..CP13..... 12F0: 00 5B 83 0B 43 50 31 34 0E 00 00 00 00 00 5B 83 .[..CP14......[. 1300: 0B 43 50 31 35 0F 00 00 00 00 00 5B 83 0B 43 50 .CP15......[..CP 1310: 31 36 10 00 00 00 00 00 5B 83 0B 43 50 31 37 11 16......[..CP17. 1320: 00 00 00 00 00 5B 83 0B 43 50 31 38 12 00 00 00 .....[..CP18.... 1330: 00 00 5B 83 0B 43 50 31 39 13 00 00 00 00 00 5B ..[..CP19......[ 1340: 83 0B 43 50 32 30 14 00 00 00 00 00 5B 83 0B 43 ..CP20......[..C 1350: 50 32 31 15 00 00 00 00 00 5B 83 0B 43 50 32 32 P21......[..CP22 1360: 16 00 00 00 00 00 5B 83 0B 43 50 32 33 17 00 00 ......[..CP23... 1370: 00 00 00 5B 83 0B 43 50 32 34 18 00 00 00 00 00 ...[..CP24...... 1380: 5B 83 0B 43 50 32 35 19 00 00 00 00 00 5B 83 0B [..CP25......[.. 1390: 43 50 32 36 1A 00 00 00 00 00 5B 83 0B 43 50 32 CP26......[..CP2 13A0: 37 1B 00 00 00 00 00 5B 83 0B 43 50 32 38 1C 00 7......[..CP28.. 13B0: 00 00 00 00 5B 83 0B 43 50 32 39 1D 00 00 00 00 ....[..CP29..... 13C0: 00 5B 83 0B 43 50 33 30 1E 00 00 00 00 00 5B 83 .[..CP30......[. 13D0: 0B 43 50 33 31 1F 00 00 00 00 00 5B 83 0B 43 50 .CP31......[..CP 13E0: 33 32 20 00 00 00 00 00 10 15 5C 00 08 53 53 54 32 .......\..SST 13F0: 58 00 14 0B 5F 50 54 53 01 50 53 4B 4D 68 14 29 X..._PTS.PSKMh.) 1400: 2E 5F 47 50 45 5F 4C 30 32 00 86 5C 2E 5F 53 42 ._GPE_L02..\._SB 1410: 5F 41 4D 57 30 0A D0 70 47 53 54 30 60 7B 60 0A _AMW0..pGST0`{`. 1420: 04 60 70 60 47 53 54 30 14 3C 2E 5F 47 50 45 5F .`p`GST0.<._GPE_ 1430: 4C 30 38 00 47 50 4B 4D A0 16 80 93 53 53 54 58 L08.GPKM....SSTX 1440: 01 00 86 5C 2E 5F 53 42 5F 56 42 54 4E 0A 02 86 ...\._SB_VBTN... 1450: 5C 2F 04 5F 53 42 5F 50 43 49 30 49 53 41 5F 4B \/._SB_PCI0ISA_K 1460: 42 44 5F 0A 02 14 49 04 2E 5F 47 50 45 5F 4C 31 BD_...I.._GPE_L1 1470: 38 00 7F 4C 49 44 50 01 4C 49 44 50 7D 50 4D 45 8..LIDP.LIDP}PME 1480: 32 01 50 4D 45 32 A0 28 93 5C 2F 04 5F 53 42 5F 2.PME2.(.\/._SB_ 1490: 50 43 49 30 50 43 49 31 58 4D 45 31 01 86 5C 2F PCI0PCI1XME1..\/ 14A0: 03 5F 53 42 5F 50 43 49 30 50 43 49 31 0A 02 14 ._SB_PCI0PCI1... 14B0: 32 2E 5F 47 50 45 5F 4C 31 42 00 7F 4C 49 44 50 2._GPE_L1B..LIDP 14C0: 0A 08 4C 49 44 50 7D 50 4D 45 32 01 50 4D 45 32 ..LIDP}PME2.PME2 14D0: 86 5C 2F 03 5F 53 42 5F 50 43 49 30 50 43 49 35 .\/._SB_PCI0PCI5 14E0: 0A 02 14 45 07 2E 5F 47 50 45 5F 4C 31 43 00 7F ...E.._GPE_L1C.. 14F0: 4C 49 44 50 0A 08 4C 49 44 50 7D 50 4D 45 32 01 LIDP..LIDP}PME2. 1500: 50 4D 45 32 A0 28 93 5C 2F 04 5F 53 42 5F 50 43 PME2.(.\/._SB_PC 1510: 49 30 50 43 49 32 58 4D 45 32 01 86 5C 2F 03 5F I0PCI2XME2..\/._ 1520: 53 42 5F 50 43 49 30 50 43 49 32 0A 02 A1 2A A0 SB_PCI0PCI2...*. 1530: 28 93 5C 2F 04 5F 53 42 5F 50 43 49 30 50 43 49 (.\/._SB_PCI0PCI 1540: 33 58 4D 45 33 01 86 5C 2F 03 5F 53 42 5F 50 43 3XME3..\/._SB_PC 1550: 49 30 50 43 49 33 0A 02 14 16 2E 5F 47 50 45 5F I0PCI3....._GPE_ 1560: 4C 31 44 00 7F 4C 49 44 50 0A 20 4C 49 44 50 14 L1D..LIDP. LIDP. 1570: 45 07 2E 5F 47 50 45 5F 4C 31 46 00 7F 4C 49 44 E.._GPE_L1F..LID 1580: 50 0A 08 4C 49 44 50 7D 50 4D 45 32 01 50 4D 45 P..LIDP}PME2.PME 1590: 32 A0 28 93 5C 2F 04 5F 53 42 5F 50 43 49 37 50 2.(.\/._SB_PCI7P 15A0: 43 49 42 58 4D 45 42 01 86 5C 2F 03 5F 53 42 5F CIBXMEB..\/._SB_ 15B0: 50 43 49 37 50 43 49 42 0A 02 A1 2A A0 28 93 5C PCI7PCIB...*.(.\ 15C0: 2F 04 5F 53 42 5F 50 43 49 37 50 43 49 41 58 4D /._SB_PCI7PCIAXM 15D0: 45 41 01 86 5C 2F 03 5F 53 42 5F 50 43 49 37 50 EA..\/._SB_PCI7P 15E0: 43 49 41 0A 02 14 43 07 5F 57 41 4B 01 A0 09 93 CIA...C._WAK.... 15F0: 68 0A 04 47 55 53 42 A1 17 A0 15 93 68 0A 03 A0 h..GUSB.....h... 1600: 0F 7B 0A 20 43 4D 52 44 0A 49 00 47 55 53 42 43 .{. CMRD.I.GUSBC 1610: 4D 57 52 0A 49 7B 0A DF 43 4D 52 44 0A 49 00 A0 MWR.I{..CMRD.I.. 1620: 37 7D 7B 50 4D 53 32 01 00 7D 7B 92 48 41 43 4B 7}{PMS2..}{.HACK 1630: 7B 47 4C 42 54 0A 11 00 00 7B 92 48 41 43 4B 7B {GLBT....{.HACK{ 1640: 54 48 52 50 0A 18 00 00 00 00 86 5C 2E 5F 53 42 THRP.......\._SB 1650: 5F 56 42 54 4E 0A 02 A4 00 10 33 5F 53 49 5F 14 _VBTN.....3_SI_. 1660: 2D 5F 53 53 54 01 70 68 53 53 54 58 A0 10 93 68 -_SST.phSSTX...h 1670: 0A 03 7B 4C 45 44 32 0A FD 4C 45 44 32 A0 0F 93 ..{LED2..LED2... 1680: 68 01 7D 4C 45 44 32 0A 02 4C 45 44 32 10 81 79 h.}LED2..LED2..y 1690: 01 5F 53 42 5F 5B 82 43 CC 50 43 49 30 08 5F 48 ._SB_[.C.PCI0._H 16A0: 49 44 0C 41 D0 0A 03 08 5F 55 49 44 0A 04 08 5F ID.A...._UID..._ 16B0: 42 42 4E 00 08 5F 41 44 52 00 08 5F 50 52 57 12 BBN.._ADR.._PRW. 16C0: 06 02 0A 0D 0A 05 14 08 5F 53 31 44 00 A4 01 14 ........_S1D.... 16D0: 14 5F 53 33 44 00 A0 08 48 41 43 4B A4 0A 03 A1 ._S3D...HACK.... 16E0: 04 A4 0A 02 08 50 49 43 30 12 4A 39 40 12 0B 04 .....PIC0.J9@... 16F0: 0B FF FF 00 4C 4E 4B 41 00 12 0B 04 0B FF FF 01 ....LNKA........ 1700: 4C 4E 4B 42 00 12 0C 04 0B FF FF 0A 02 4C 4E 4B LNKB.........LNK 1710: 43 00 12 0C 04 0B FF FF 0A 03 4C 4E 4B 44 00 12 C.........LNKD.. 1720: 0D 04 0C FF FF 01 00 00 4C 4E 4B 41 00 12 0D 04 ........LNKA.... 1730: 0C FF FF 01 00 01 4C 4E 4B 42 00 12 0E 04 0C FF ......LNKB...... 1740: FF 01 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF .....LNKC....... 1750: 01 00 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 02 ....LNKD........ 1760: 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 02 00 01 ..LNKA.......... 1770: 4C 4E 4B 42 00 12 0E 04 0C FF FF 02 00 0A 02 4C LNKB...........L 1780: 4E 4B 43 00 12 0E 04 0C FF FF 02 00 0A 03 4C 4E NKC...........LN 1790: 4B 44 00 12 0D 04 0C FF FF 03 00 00 4C 4E 4B 41 KD..........LNKA 17A0: 00 12 0D 04 0C FF FF 03 00 01 4C 4E 4B 42 00 12 ..........LNKB.. 17B0: 0E 04 0C FF FF 03 00 0A 02 4C 4E 4B 43 00 12 0E .........LNKC... 17C0: 04 0C FF FF 03 00 0A 03 4C 4E 4B 44 00 12 0D 04 ........LNKD.... 17D0: 0C FF FF 04 00 00 4C 4E 4B 41 00 12 0D 04 0C FF ......LNKA...... 17E0: FF 04 00 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 04 ....LNKB........ 17F0: 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF 04 00 ...LNKC......... 1800: 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 05 00 00 ..LNKD.......... 1810: 4C 4E 4B 41 00 12 0D 04 0C FF FF 05 00 01 4C 4E LNKA..........LN 1820: 4B 42 00 12 0E 04 0C FF FF 05 00 0A 02 4C 4E 4B KB...........LNK 1830: 43 00 12 0E 04 0C FF FF 05 00 0A 03 4C 4E 4B 44 C...........LNKD 1840: 00 12 0D 04 0C FF FF 06 00 00 4C 4E 4B 41 00 12 ..........LNKA.. 1850: 0D 04 0C FF FF 06 00 01 4C 4E 4B 42 00 12 0E 04 ........LNKB.... 1860: 0C FF FF 06 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C .......LNKC..... 1870: FF FF 06 00 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF ......LNKD...... 1880: FF 07 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 07 ....LNKA........ 1890: 00 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 07 00 0A ..LNKB.......... 18A0: 02 4C 4E 4B 43 00 12 0E 04 0C FF FF 07 00 0A 03 .LNKC........... 18B0: 4C 4E 4B 44 00 12 0D 04 0C FF FF 08 00 00 4C 4E LNKD..........LN 18C0: 4B 41 00 12 0D 04 0C FF FF 08 00 01 4C 4E 4B 42 KA..........LNKB 18D0: 00 12 0E 04 0C FF FF 08 00 0A 02 4C 4E 4B 43 00 ...........LNKC. 18E0: 12 0E 04 0C FF FF 08 00 0A 03 4C 4E 4B 44 00 12 ..........LNKD.. 18F0: 0D 04 0C FF FF 09 00 00 4C 4E 4B 41 00 12 0D 04 ........LNKA.... 1900: 0C FF FF 09 00 01 4C 4E 4B 42 00 12 0E 04 0C FF ......LNKB...... 1910: FF 09 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF .....LNKC....... 1920: 09 00 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 0A ....LNKD........ 1930: 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 0A 00 01 ..LNKA.......... 1940: 4C 4E 4B 42 00 12 0E 04 0C FF FF 0A 00 0A 02 4C LNKB...........L 1950: 4E 4B 43 00 12 0E 04 0C FF FF 0A 00 0A 03 4C 4E NKC...........LN 1960: 4B 44 00 12 0D 04 0C FF FF 16 00 00 4C 4E 4B 41 KD..........LNKA 1970: 00 12 0D 04 0C FF FF 16 00 01 4C 4E 4B 42 00 12 ..........LNKB.. 1980: 0E 04 0C FF FF 16 00 0A 02 4C 4E 4B 47 00 12 0E .........LNKG... 1990: 04 0C FF FF 16 00 0A 03 4C 4E 4B 44 00 12 0D 04 ........LNKD.... 19A0: 0C FF FF 1A 00 00 4C 4E 4B 41 00 12 0D 04 0C FF ......LNKA...... 19B0: FF 1A 00 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 1A ....LNKB........ 19C0: 00 0A 02 4C 4E 4B 47 00 12 0E 04 0C FF FF 1A 00 ...LNKG......... 19D0: 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 1B 00 00 ..LNKD.......... 19E0: 4C 4E 4B 41 00 12 0D 04 0C FF FF 1C 00 00 4C 4E LNKA..........LN 19F0: 4B 41 00 12 0D 04 0C FF FF 1C 00 01 4C 4E 4B 42 KA..........LNKB 1A00: 00 12 0E 04 0C FF FF 1C 00 0A 02 4C 4E 4B 43 00 ...........LNKC. 1A10: 12 0E 04 0C FF FF 1C 00 0A 03 4C 4E 4B 44 00 12 ..........LNKD.. 1A20: 0D 04 0C FF FF 1D 00 00 4C 4E 4B 48 00 12 0D 04 ........LNKH.... 1A30: 0C FF FF 1D 00 01 4C 4E 4B 42 00 12 0E 04 0C FF ......LNKB...... 1A40: FF 1D 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF .....LNKC....... 1A50: 1D 00 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 1F ....LNKD........ 1A60: 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 1F 00 01 ..LNKA.......... 1A70: 4C 4E 4B 42 00 12 0E 04 0C FF FF 1F 00 0A 02 4C LNKB...........L 1A80: 4E 4B 45 00 08 41 50 49 30 12 4A 31 40 12 09 04 NKE..API0.J1@... 1A90: 0B FF FF 00 00 0A 10 12 09 04 0B FF FF 01 00 0A ................ 1AA0: 11 12 0A 04 0B FF FF 0A 02 00 0A 12 12 0A 04 0B ................ 1AB0: FF FF 0A 03 00 0A 13 12 0B 04 0C FF FF 01 00 00 ................ 1AC0: 00 0A 10 12 0B 04 0C FF FF 01 00 01 00 0A 11 12 ................ 1AD0: 0C 04 0C FF FF 01 00 0A 02 00 0A 12 12 0C 04 0C ................ 1AE0: FF FF 01 00 0A 03 00 0A 13 12 0B 04 0C FF FF 02 ................ 1AF0: 00 00 00 0A 10 12 0B 04 0C FF FF 02 00 01 00 0A ................ 1B00: 11 12 0C 04 0C FF FF 02 00 0A 02 00 0A 12 12 0C ................ 1B10: 04 0C FF FF 02 00 0A 03 00 0A 13 12 0B 04 0C FF ................ 1B20: FF 03 00 00 00 0A 10 12 0B 04 0C FF FF 03 00 01 ................ 1B30: 00 0A 11 12 0C 04 0C FF FF 03 00 0A 02 00 0A 12 ................ 1B40: 12 0C 04 0C FF FF 03 00 0A 03 00 0A 13 12 0B 04 ................ 1B50: 0C FF FF 04 00 00 00 0A 10 12 0B 04 0C FF FF 04 ................ 1B60: 00 01 00 0A 11 12 0C 04 0C FF FF 04 00 0A 02 00 ................ 1B70: 0A 12 12 0C 04 0C FF FF 04 00 0A 03 00 0A 13 12 ................ 1B80: 0B 04 0C FF FF 05 00 00 00 0A 10 12 0B 04 0C FF ................ 1B90: FF 05 00 01 00 0A 11 12 0C 04 0C FF FF 05 00 0A ................ 1BA0: 02 00 0A 12 12 0C 04 0C FF FF 05 00 0A 03 00 0A ................ 1BB0: 13 12 0B 04 0C FF FF 06 00 00 00 0A 10 12 0B 04 ................ 1BC0: 0C FF FF 06 00 01 00 0A 11 12 0C 04 0C FF FF 06 ................ 1BD0: 00 0A 02 00 0A 12 12 0C 04 0C FF FF 06 00 0A 03 ................ 1BE0: 00 0A 13 12 0B 04 0C FF FF 07 00 00 00 0A 10 12 ................ 1BF0: 0B 04 0C FF FF 07 00 01 00 0A 11 12 0C 04 0C FF ................ 1C00: FF 07 00 0A 02 00 0A 12 12 0C 04 0C FF FF 07 00 ................ 1C10: 0A 03 00 0A 13 12 0B 04 0C FF FF 08 00 00 00 0A ................ 1C20: 10 12 0B 04 0C FF FF 08 00 01 00 0A 11 12 0C 04 ................ 1C30: 0C FF FF 08 00 0A 02 00 0A 12 12 0C 04 0C FF FF ................ 1C40: 08 00 0A 03 00 0A 13 12 0B 04 0C FF FF 09 00 00 ................ 1C50: 00 0A 10 12 0B 04 0C FF FF 09 00 01 00 0A 11 12 ................ 1C60: 0C 04 0C FF FF 09 00 0A 02 00 0A 12 12 0C 04 0C ................ 1C70: FF FF 09 00 0A 03 00 0A 13 12 0B 04 0C FF FF 0A ................ 1C80: 00 00 00 0A 10 12 0B 04 0C FF FF 0A 00 01 00 0A ................ 1C90: 11 12 0C 04 0C FF FF 0A 00 0A 02 00 0A 12 12 0C ................ 1CA0: 04 0C FF FF 0A 00 0A 03 00 0A 13 12 0B 04 0C FF ................ 1CB0: FF 16 00 00 00 0A 10 12 0B 04 0C FF FF 16 00 01 ................ 1CC0: 00 0A 11 12 0C 04 0C FF FF 16 00 0A 02 00 0A 12 ................ 1CD0: 12 0C 04 0C FF FF 16 00 0A 03 00 0A 13 12 0B 04 ................ 1CE0: 0C FF FF 1A 00 00 00 0A 10 12 0B 04 0C FF FF 1A ................ 1CF0: 00 01 00 0A 11 12 0C 04 0C FF FF 1A 00 0A 02 00 ................ 1D00: 0A 16 12 0C 04 0C FF FF 1A 00 0A 03 00 0A 13 12 ................ 1D10: 0B 04 0C FF FF 1B 00 00 00 0A 10 12 0B 04 0C FF ................ 1D20: FF 1C 00 00 00 0A 10 12 0B 04 0C FF FF 1C 00 01 ................ 1D30: 00 0A 11 12 0C 04 0C FF FF 1C 00 0A 02 00 0A 12 ................ 1D40: 12 0C 04 0C FF FF 1C 00 0A 03 00 0A 13 12 0B 04 ................ 1D50: 0C FF FF 1D 00 00 00 0A 17 12 0B 04 0C FF FF 1D ................ 1D60: 00 01 00 0A 11 12 0C 04 0C FF FF 1D 00 0A 02 00 ................ 1D70: 0A 12 12 0C 04 0C FF FF 1D 00 0A 03 00 0A 13 12 ................ 1D80: 0B 04 0C FF FF 1F 00 00 00 0A 10 12 0B 04 0C FF ................ 1D90: FF 1F 00 01 00 0A 11 12 0C 04 0C FF FF 1F 00 0A ................ 1DA0: 02 00 0A 14 5B 82 40 13 50 43 49 31 14 08 5F 53 ....[.@.PCI1.._S 1DB0: 31 44 00 A4 01 14 14 5F 53 33 44 00 A0 08 48 41 1D....._S3D...HA 1DC0: 43 4B A4 0A 03 A1 04 A4 0A 02 14 14 5F 53 34 44 CK.........._S4D 1DD0: 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 ...HACK......... 1DE0: 14 5F 53 35 44 00 A0 08 48 41 43 4B A4 0A 03 A1 ._S5D...HACK.... 1DF0: 04 A4 0A 02 14 0F 5F 41 44 52 00 70 0C 00 00 01 ......_ADR.p.... 1E00: 00 60 A4 60 08 5F 55 49 44 0A 20 08 5F 50 52 57 .`.`._UID. ._PRW 1E10: 12 06 02 0A 18 0A 05 5B 80 50 45 52 53 02 0A 94 .......[.PERS... 1E20: 0A 02 5B 81 10 50 45 52 53 01 50 45 52 30 08 50 ..[..PERS.PER0.P 1E30: 45 52 31 08 14 19 58 4D 45 31 00 70 50 45 52 31 ER1...XME1.pPER1 1E40: 60 7B 60 0A 80 60 70 60 50 45 52 31 A4 60 08 50 `{`..`p`PER1.`.P 1E50: 49 43 31 12 34 04 12 0B 04 0B FF FF 00 4C 4E 4B IC1.4........LNK 1E60: 41 00 12 0B 04 0B FF FF 01 4C 4E 4B 42 00 12 0C A........LNKB... 1E70: 04 0B FF FF 0A 02 4C 4E 4B 43 00 12 0C 04 0B FF ......LNKC...... 1E80: FF 0A 03 4C 4E 4B 44 00 08 41 50 49 31 12 2C 04 ...LNKD..API1.,. 1E90: 12 09 04 0B FF FF 00 00 0A 1C 12 09 04 0B FF FF ................ 1EA0: 01 00 0A 28 12 0A 04 0B FF FF 0A 02 00 0A 29 12 ...(..........). 1EB0: 0A 04 0B FF FF 0A 03 00 0A 2F 14 1B 5F 50 52 54 ........./.._PRT 1EC0: 00 70 41 50 49 31 60 A0 0C 92 50 49 43 46 70 50 .pAPI1`...PICFpP 1ED0: 49 43 31 60 A4 60 5B 82 49 13 50 43 49 32 14 08 IC1`.`[.I.PCI2.. 1EE0: 5F 53 31 44 00 A4 01 14 14 5F 53 33 44 00 A0 08 _S1D....._S3D... 1EF0: 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 14 5F 53 HACK.........._S 1F00: 34 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 4D...HACK....... 1F10: 02 14 14 5F 53 35 44 00 A0 08 48 41 43 4B A4 0A ..._S5D...HACK.. 1F20: 03 A1 04 A4 0A 02 14 0F 5F 41 44 52 00 70 0C 00 ........_ADR.p.. 1F30: 00 03 00 60 A4 60 08 5F 55 49 44 0A 21 08 5F 50 ...`.`._UID.!._P 1F40: 52 57 12 06 02 0A 1C 0A 05 5B 80 52 53 54 53 02 RW.......[.RSTS. 1F50: 0A B0 0A 04 5B 81 1A 52 53 54 53 01 50 49 44 30 ....[..RSTS.PID0 1F60: 08 50 49 44 31 08 50 4D 58 53 08 50 4E 55 53 08 .PID1.PMXS.PNUS. 1F70: 14 18 58 4D 45 32 00 70 50 4D 58 53 60 7B 60 01 ..XME2.pPMXS`{`. 1F80: 60 70 60 50 4D 58 53 A4 60 08 50 49 43 32 12 34 `p`PMXS.`.PIC2.4 1F90: 04 12 0B 04 0B FF FF 00 4C 4E 4B 41 00 12 0B 04 ........LNKA.... 1FA0: 0B FF FF 01 4C 4E 4B 42 00 12 0C 04 0B FF FF 0A ....LNKB........ 1FB0: 02 4C 4E 4B 43 00 12 0C 04 0B FF FF 0A 03 4C 4E .LNKC.........LN 1FC0: 4B 44 00 08 41 50 49 32 12 2C 04 12 09 04 0B FF KD..API2.,...... 1FD0: FF 00 00 0A 18 12 09 04 0B FF FF 01 00 0A 22 12 ..............". 1FE0: 0A 04 0B FF FF 0A 02 00 0A 23 12 0A 04 0B FF FF .........#...... 1FF0: 0A 03 00 0A 24 14 1B 5F 50 52 54 00 70 41 50 49 ....$.._PRT.pAPI 2000: 32 60 A0 0C 92 50 49 43 46 70 50 49 43 32 60 A4 2`...PICFpPIC2`. 2010: 60 5B 82 40 14 50 43 49 33 14 08 5F 53 31 44 00 `[.@.PCI3.._S1D. 2020: A4 01 14 14 5F 53 33 44 00 A0 08 48 41 43 4B A4 ...._S3D...HACK. 2030: 0A 03 A1 04 A4 0A 02 14 14 5F 53 34 44 00 A0 08 ........._S4D... 2040: 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 14 5F 53 HACK.........._S 2050: 35 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 5D...HACK....... 2060: 02 14 0F 5F 41 44 52 00 70 0C 00 00 07 00 60 A4 ..._ADR.p.....`. 2070: 60 08 5F 55 49 44 0A 22 08 5F 50 52 57 12 06 02 `._UID."._PRW... 2080: 0A 1C 0A 05 5B 80 52 53 54 53 02 0A B0 0A 04 5B ....[.RSTS.....[ 2090: 81 1A 52 53 54 53 01 50 49 44 30 08 50 49 44 31 ..RSTS.PID0.PID1 20A0: 08 50 4D 58 53 08 50 4E 55 53 08 14 1F 58 4D 45 .PMXS.PNUS...XME 20B0: 33 00 70 50 4D 58 53 60 7B 60 01 60 70 60 50 4D 3.pPMXS`{`.`p`PM 20C0: 58 53 A0 04 60 A4 01 A1 03 A4 00 08 50 49 43 33 XS..`.......PIC3 20D0: 12 34 04 12 0B 04 0B FF FF 00 4C 4E 4B 41 00 12 .4........LNKA.. 20E0: 0B 04 0B FF FF 01 4C 4E 4B 42 00 12 0C 04 0B FF ......LNKB...... 20F0: FF 0A 02 4C 4E 4B 43 00 12 0C 04 0B FF FF 0A 03 ...LNKC......... 2100: 4C 4E 4B 44 00 08 41 50 49 33 12 2C 04 12 09 04 LNKD..API3.,.... 2110: 0B FF FF 00 00 0A 1E 12 09 04 0B FF FF 01 00 0A ................ 2120: 25 12 0A 04 0B FF FF 0A 02 00 0A 27 12 0A 04 0B %..........'.... 2130: FF FF 0A 03 00 0A 26 14 1B 5F 50 52 54 00 70 41 ......&.._PRT.pA 2140: 50 49 33 60 A0 0C 92 50 49 43 46 70 50 49 43 33 PI3`...PICFpPIC3 2150: 60 A4 60 5B 82 4A 0F 50 43 49 35 14 08 5F 53 31 `.`[.J.PCI5.._S1 2160: 44 00 A4 01 14 14 5F 53 33 44 00 A0 08 48 41 43 D....._S3D...HAC 2170: 4B A4 0A 03 A1 04 A4 0A 02 14 14 5F 53 34 44 00 K.........._S4D. 2180: A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 14 ..HACK.......... 2190: 5F 53 35 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04 _S5D...HACK..... 21A0: A4 0A 02 14 0F 5F 41 44 52 00 70 0C 05 00 1C 00 ....._ADR.p..... 21B0: 60 A4 60 08 5F 55 49 44 0A 24 08 5F 50 52 57 12 `.`._UID.$._PRW. 21C0: 06 02 0A 1B 0A 05 5B 80 52 53 54 53 02 0A 60 0A ......[.RSTS..`. 21D0: 04 5B 81 1A 52 53 54 53 01 50 49 44 30 08 50 49 .[..RSTS.PID0.PI 21E0: 44 31 08 50 4D 58 53 08 50 4E 55 53 08 14 1F 58 D1.PMXS.PNUS...X 21F0: 4D 45 35 00 70 50 4D 58 53 60 7B 60 01 60 70 60 ME5.pPMXS`{`.`p` 2200: 50 4D 58 53 A0 04 60 A4 01 A1 03 A4 00 08 50 49 PMXS..`.......PI 2210: 43 35 12 0E 01 12 0B 04 0B FF FF 00 4C 4E 4B 42 C5..........LNKB 2220: 00 08 41 50 49 35 12 0C 01 12 09 04 0B FF FF 00 ..API5.......... 2230: 00 0A 11 14 1B 5F 50 52 54 00 70 41 50 49 35 60 ....._PRT.pAPI5` 2240: A0 0C 92 50 49 43 46 70 50 49 43 35 60 A4 60 5B ...PICFpPIC5`.`[ 2250: 82 49 10 50 43 49 36 14 08 5F 53 31 44 00 A4 01 .I.PCI6.._S1D... 2260: 14 14 5F 53 33 44 00 A0 08 48 41 43 4B A4 0A 03 .._S3D...HACK... 2270: A1 04 A4 0A 02 14 14 5F 53 34 44 00 A0 08 48 41 ......._S4D...HA 2280: 43 4B A4 0A 03 A1 04 A4 0A 02 14 14 5F 53 35 44 CK.........._S5D 2290: 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 ...HACK......... 22A0: 0F 5F 41 44 52 00 70 0C 00 00 1E 00 60 A4 60 08 ._ADR.p.....`.`. 22B0: 5F 55 49 44 0A 25 08 5F 50 52 57 12 06 02 0A 0B _UID.%._PRW..... 22C0: 0A 05 08 50 49 43 36 12 3C 04 12 0D 04 0C FF FF ...PIC6.<....... 22D0: 04 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 04 00 ...LNKA......... 22E0: 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 04 00 0A 02 .LNKB........... 22F0: 4C 4E 4B 43 00 12 0E 04 0C FF FF 04 00 0A 03 4C LNKC...........L 2300: 4E 4B 44 00 08 41 50 49 36 12 34 04 12 0B 04 0C NKD..API6.4..... 2310: FF FF 04 00 00 00 0A 10 12 0B 04 0C FF FF 04 00 ................ 2320: 01 00 0A 11 12 0C 04 0C FF FF 04 00 0A 02 00 0A ................ 2330: 12 12 0C 04 0C FF FF 04 00 0A 03 00 0A 13 14 1B ................ 2340: 5F 50 52 54 00 70 41 50 49 36 60 A0 0C 92 50 49 _PRT.pAPI6`...PI 2350: 43 46 70 50 49 43 36 60 A4 60 5B 82 43 AC 50 43 CFpPIC6`.`[.C.PC 2360: 49 37 08 5F 48 49 44 0C 41 D0 0A 03 08 5F 55 49 I7._HID.A...._UI 2370: 44 0A 26 08 5F 42 42 4E 0A 20 08 5F 41 44 52 00 D.&._BBN. ._ADR. 2380: 14 08 5F 53 31 44 00 A4 01 14 14 5F 53 33 44 00 .._S1D....._S3D. 2390: A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 08 50 ..HACK.........P 23A0: 49 43 37 12 43 2B 30 12 0B 04 0B FF FF 00 4C 4E IC7.C+0.......LN 23B0: 4B 41 00 12 0B 04 0B FF FF 01 4C 4E 4B 42 00 12 KA........LNKB.. 23C0: 0C 04 0B FF FF 0A 02 4C 4E 4B 43 00 12 0C 04 0B .......LNKC..... 23D0: FF FF 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 01 ....LNKD........ 23E0: 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 01 00 01 ..LNKA.......... 23F0: 4C 4E 4B 42 00 12 0E 04 0C FF FF 01 00 0A 02 4C LNKB...........L 2400: 4E 4B 43 00 12 0E 04 0C FF FF 01 00 0A 03 4C 4E NKC...........LN 2410: 4B 44 00 12 0D 04 0C FF FF 02 00 00 4C 4E 4B 41 KD..........LNKA 2420: 00 12 0D 04 0C FF FF 02 00 01 4C 4E 4B 42 00 12 ..........LNKB.. 2430: 0E 04 0C FF FF 02 00 0A 02 4C 4E 4B 43 00 12 0E .........LNKC... 2440: 04 0C FF FF 02 00 0A 03 4C 4E 4B 44 00 12 0D 04 ........LNKD.... 2450: 0C FF FF 03 00 00 4C 4E 4B 41 00 12 0D 04 0C FF ......LNKA...... 2460: FF 03 00 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 03 ....LNKB........ 2470: 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF 03 00 ...LNKC......... 2480: 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 04 00 00 ..LNKD.......... 2490: 4C 4E 4B 41 00 12 0D 04 0C FF FF 04 00 01 4C 4E LNKA..........LN 24A0: 4B 42 00 12 0E 04 0C FF FF 04 00 0A 02 4C 4E 4B KB...........LNK 24B0: 43 00 12 0E 04 0C FF FF 04 00 0A 03 4C 4E 4B 44 C...........LNKD 24C0: 00 12 0D 04 0C FF FF 05 00 00 4C 4E 4B 41 00 12 ..........LNKA.. 24D0: 0D 04 0C FF FF 05 00 01 4C 4E 4B 42 00 12 0E 04 ........LNKB.... 24E0: 0C FF FF 05 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C .......LNKC..... 24F0: FF FF 05 00 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF ......LNKD...... 2500: FF 06 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 06 ....LNKA........ 2510: 00 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 06 00 0A ..LNKB.......... 2520: 02 4C 4E 4B 43 00 12 0E 04 0C FF FF 06 00 0A 03 .LNKC........... 2530: 4C 4E 4B 44 00 12 0D 04 0C FF FF 07 00 00 4C 4E LNKD..........LN 2540: 4B 41 00 12 0D 04 0C FF FF 07 00 01 4C 4E 4B 42 KA..........LNKB 2550: 00 12 0E 04 0C FF FF 07 00 0A 02 4C 4E 4B 43 00 ...........LNKC. 2560: 12 0E 04 0C FF FF 07 00 0A 03 4C 4E 4B 44 00 12 ..........LNKD.. 2570: 0D 04 0C FF FF 08 00 00 4C 4E 4B 41 00 12 0D 04 ........LNKA.... 2580: 0C FF FF 08 00 01 4C 4E 4B 42 00 12 0E 04 0C FF ......LNKB...... 2590: FF 08 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF .....LNKC....... 25A0: 08 00 0A 03 4C 4E 4B 44 00 12 0D 04 0C FF FF 09 ....LNKD........ 25B0: 00 00 4C 4E 4B 41 00 12 0D 04 0C FF FF 09 00 01 ..LNKA.......... 25C0: 4C 4E 4B 42 00 12 0E 04 0C FF FF 09 00 0A 02 4C LNKB...........L 25D0: 4E 4B 43 00 12 0E 04 0C FF FF 09 00 0A 03 4C 4E NKC...........LN 25E0: 4B 44 00 12 0D 04 0C FF FF 0A 00 00 4C 4E 4B 41 KD..........LNKA 25F0: 00 12 0D 04 0C FF FF 0A 00 01 4C 4E 4B 42 00 12 ..........LNKB.. 2600: 0E 04 0C FF FF 0A 00 0A 02 4C 4E 4B 43 00 12 0E .........LNKC... 2610: 04 0C FF FF 0A 00 0A 03 4C 4E 4B 44 00 12 0D 04 ........LNKD.... 2620: 0C FF FF 16 00 00 4C 4E 4B 41 00 12 0D 04 0C FF ......LNKA...... 2630: FF 16 00 01 4C 4E 4B 42 00 12 0E 04 0C FF FF 16 ....LNKB........ 2640: 00 0A 02 4C 4E 4B 43 00 12 0E 04 0C FF FF 16 00 ...LNKC......... 2650: 0A 03 4C 4E 4B 44 00 08 41 50 49 37 12 43 25 30 ..LNKD..API7.C%0 2660: 12 09 04 0B FF FF 00 00 0A 10 12 09 04 0B FF FF ................ 2670: 01 00 0A 11 12 0A 04 0B FF FF 0A 02 00 0A 12 12 ................ 2680: 0A 04 0B FF FF 0A 03 00 0A 13 12 0B 04 0C FF FF ................ 2690: 01 00 00 00 0A 10 12 0B 04 0C FF FF 01 00 01 00 ................ 26A0: 0A 11 12 0C 04 0C FF FF 01 00 0A 02 00 0A 12 12 ................ 26B0: 0C 04 0C FF FF 01 00 0A 03 00 0A 13 12 0B 04 0C ................ 26C0: FF FF 02 00 00 00 0A 10 12 0B 04 0C FF FF 02 00 ................ 26D0: 01 00 0A 11 12 0C 04 0C FF FF 02 00 0A 02 00 0A ................ 26E0: 12 12 0C 04 0C FF FF 02 00 0A 03 00 0A 13 12 0B ................ 26F0: 04 0C FF FF 03 00 00 00 0A 10 12 0B 04 0C FF FF ................ 2700: 03 00 01 00 0A 11 12 0C 04 0C FF FF 03 00 0A 02 ................ 2710: 00 0A 12 12 0C 04 0C FF FF 03 00 0A 03 00 0A 13 ................ 2720: 12 0B 04 0C FF FF 04 00 00 00 0A 10 12 0B 04 0C ................ 2730: FF FF 04 00 01 00 0A 11 12 0C 04 0C FF FF 04 00 ................ 2740: 0A 02 00 0A 12 12 0C 04 0C FF FF 04 00 0A 03 00 ................ 2750: 0A 13 12 0B 04 0C FF FF 05 00 00 00 0A 10 12 0B ................ 2760: 04 0C FF FF 05 00 01 00 0A 11 12 0C 04 0C FF FF ................ 2770: 05 00 0A 02 00 0A 12 12 0C 04 0C FF FF 05 00 0A ................ 2780: 03 00 0A 13 12 0B 04 0C FF FF 06 00 00 00 0A 10 ................ 2790: 12 0B 04 0C FF FF 06 00 01 00 0A 11 12 0C 04 0C ................ 27A0: FF FF 06 00 0A 02 00 0A 12 12 0C 04 0C FF FF 06 ................ 27B0: 00 0A 03 00 0A 13 12 0B 04 0C FF FF 07 00 00 00 ................ 27C0: 0A 10 12 0B 04 0C FF FF 07 00 01 00 0A 11 12 0C ................ 27D0: 04 0C FF FF 07 00 0A 02 00 0A 12 12 0C 04 0C FF ................ 27E0: FF 07 00 0A 03 00 0A 13 12 0B 04 0C FF FF 08 00 ................ 27F0: 00 00 0A 10 12 0B 04 0C FF FF 08 00 01 00 0A 11 ................ 2800: 12 0C 04 0C FF FF 08 00 0A 02 00 0A 12 12 0C 04 ................ 2810: 0C FF FF 08 00 0A 03 00 0A 13 12 0B 04 0C FF FF ................ 2820: 09 00 00 00 0A 10 12 0B 04 0C FF FF 09 00 01 00 ................ 2830: 0A 11 12 0C 04 0C FF FF 09 00 0A 02 00 0A 12 12 ................ 2840: 0C 04 0C FF FF 09 00 0A 03 00 0A 13 12 0B 04 0C ................ 2850: FF FF 0A 00 00 00 0A 10 12 0B 04 0C FF FF 0A 00 ................ 2860: 01 00 0A 11 12 0C 04 0C FF FF 0A 00 0A 02 00 0A ................ 2870: 12 12 0C 04 0C FF FF 0A 00 0A 03 00 0A 13 12 0B ................ 2880: 04 0C FF FF 16 00 00 00 0A 10 12 0B 04 0C FF FF ................ 2890: 16 00 01 00 0A 11 12 0C 04 0C FF FF 16 00 0A 02 ................ 28A0: 00 0A 12 12 0C 04 0C FF FF 16 00 0A 03 00 0A 13 ................ 28B0: 14 1B 5F 50 52 54 00 70 41 50 49 37 60 A0 0C 92 .._PRT.pAPI7`... 28C0: 50 49 43 46 70 50 49 43 37 60 A4 60 14 42 1D 5F PICFpPIC7`.`.B._ 28D0: 43 52 53 00 08 4D 45 4D 53 11 4A 07 0A 76 88 0D CRS..MEMS.J..v.. 28E0: 00 02 0C 00 00 00 20 00 3D 00 00 00 1E 00 88 0D ...... .=....... 28F0: 00 01 00 03 00 00 00 00 00 00 00 00 00 00 87 17 ................ 2900: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ 2910: 00 00 00 00 00 00 00 00 88 0D 00 01 0C 03 00 00 ................ 2920: B0 03 BB 03 00 00 0C 00 88 0D 00 01 0C 03 00 00 ................ 2930: C0 03 DF 03 00 00 20 00 87 17 00 00 0C 03 00 00 ...... ......... 2940: 00 00 00 00 0A 00 FF FF 0B 00 00 00 00 00 00 00 ................ 2950: 02 00 79 00 A0 4E 04 92 93 47 4D 58 31 00 8A 4D ..y..N...GMX1..M 2960: 45 4D 53 0A 2A 4D 50 31 4E 8A 4D 45 4D 53 0A 2E EMS.*MP1N.MEMS.. 2970: 4D 50 31 58 8A 4D 45 4D 53 0A 36 4D 50 31 4C 70 MP1X.MEMS.6MP1Lp 2980: 47 4D 49 31 4D 50 31 4E 70 47 4D 58 31 4D 50 31 GMI1MP1NpGMX1MP1 2990: 58 74 4D 50 31 58 4D 50 31 4E 4D 50 31 4C 75 4D XtMP1XMP1NMP1LuM 29A0: 50 31 4C A0 4E 04 92 93 47 49 58 31 00 8B 4D 45 P1L.N...GIX1..ME 29B0: 4D 53 0A 18 49 50 31 4E 8B 4D 45 4D 53 0A 1A 49 MS..IP1N.MEMS..I 29C0: 50 31 58 8B 4D 45 4D 53 0A 1E 49 50 31 4C 70 47 P1X.MEMS..IP1LpG 29D0: 49 42 31 49 50 31 4E 70 47 49 58 31 49 50 31 58 IB1IP1NpGIX1IP1X 29E0: 74 49 50 31 58 49 50 31 4E 49 50 31 4C 75 49 50 tIP1XIP1NIP1LuIP 29F0: 31 4C 70 43 4D 52 44 0A 74 60 A0 4F 09 95 60 0A 1LpCMRD.t`.O..`. 2A00: 20 8B 4D 45 4D 53 0A 42 56 47 31 4E 8B 4D 45 4D .MEMS.BVG1N.MEM 2A10: 53 0A 44 56 47 31 58 8B 4D 45 4D 53 0A 48 56 47 S.DVG1X.MEMS.HVG 2A20: 31 4C 8B 4D 45 4D 53 0A 52 56 47 32 4E 8B 4D 45 1L.MEMS.RVG2N.ME 2A30: 4D 53 0A 54 56 47 32 58 8B 4D 45 4D 53 0A 58 56 MS.TVG2X.MEMS.XV 2A40: 47 32 4C 8B 4D 45 4D 53 0A 64 56 47 4D 4E 8B 4D G2L.MEMS.dVGMN.M 2A50: 45 4D 53 0A 68 56 47 4D 58 8B 4D 45 4D 53 0A 70 EMS.hVGMX.MEMS.p 2A60: 56 47 4D 4C 70 00 56 47 31 4E 70 00 56 47 31 58 VGMLp.VG1Np.VG1X 2A70: 70 00 56 47 31 4C 70 00 56 47 32 4E 70 00 56 47 p.VG1Lp.VG2Np.VG 2A80: 32 58 70 00 56 47 32 4C 70 00 56 47 4D 4E 70 00 2Xp.VG2Lp.VGMNp. 2A90: 56 47 4D 58 70 00 56 47 4D 4C A4 4D 45 4D 53 5B VGMXp.VGML.MEMS[ 2AA0: 82 4A 0F 50 43 49 38 14 08 5F 53 31 44 00 A4 01 .J.PCI8.._S1D... 2AB0: 14 14 5F 53 33 44 00 A0 08 48 41 43 4B A4 0A 03 .._S3D...HACK... 2AC0: A1 04 A4 0A 02 14 14 5F 53 34 44 00 A0 08 48 41 ......._S4D...HA 2AD0: 43 4B A4 0A 03 A1 04 A4 0A 02 14 14 5F 53 35 44 CK.........._S5D 2AE0: 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 ...HACK......... 2AF0: 0F 5F 41 44 52 00 70 0C 00 00 01 00 60 A4 60 08 ._ADR.p.....`.`. 2B00: 5F 55 49 44 0A 27 08 5F 50 52 57 12 06 02 0A 18 _UID.'._PRW..... 2B10: 0A 05 5B 80 52 53 54 53 02 0A B0 0A 04 5B 81 1A ..[.RSTS.....[.. 2B20: 52 53 54 53 01 50 49 44 30 08 50 49 44 31 08 50 RSTS.PID0.PID1.P 2B30: 4D 58 53 08 50 4E 55 53 08 14 1F 58 4D 45 38 00 MXS.PNUS...XME8. 2B40: 70 50 4D 58 53 60 7B 60 01 60 70 60 50 4D 58 53 pPMXS`{`.`p`PMXS 2B50: A0 04 60 A4 01 A1 03 A4 00 08 50 49 43 38 12 0E ..`.......PIC8.. 2B60: 01 12 0B 04 0B FF FF 00 4C 4E 4B 41 00 08 41 50 ........LNKA..AP 2B70: 49 38 12 0C 01 12 09 04 0B FF FF 00 00 0A 34 14 I8............4. 2B80: 1B 5F 50 52 54 00 70 41 50 49 38 60 A0 0C 92 50 ._PRT.pAPI8`...P 2B90: 49 43 46 70 50 49 43 38 60 A4 60 5B 82 40 14 50 ICFpPIC8`.`[.@.P 2BA0: 43 49 41 14 08 5F 53 31 44 00 A4 01 14 14 5F 53 CIA.._S1D....._S 2BB0: 33 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 3D...HACK....... 2BC0: 02 14 14 5F 53 34 44 00 A0 08 48 41 43 4B A4 0A ..._S4D...HACK.. 2BD0: 03 A1 04 A4 0A 02 14 14 5F 53 35 44 00 A0 08 48 ........_S5D...H 2BE0: 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 0F 5F 41 44 ACK.........._AD 2BF0: 52 00 70 0C 00 00 07 00 60 A4 60 08 5F 55 49 44 R.p.....`.`._UID 2C00: 0A 29 08 5F 50 52 57 12 06 02 0A 1F 0A 05 5B 80 .)._PRW.......[. 2C10: 52 53 54 53 02 0A B0 0A 04 5B 81 1A 52 53 54 53 RSTS.....[..RSTS 2C20: 01 50 49 44 30 08 50 49 44 31 08 50 4D 58 53 08 .PID0.PID1.PMXS. 2C30: 50 4E 55 53 08 14 1F 58 4D 45 41 00 70 50 4D 58 PNUS...XMEA.pPMX 2C40: 53 60 7B 60 01 60 70 60 50 4D 58 53 A0 04 60 A4 S`{`.`p`PMXS..`. 2C50: 01 A1 03 A4 00 08 50 49 43 41 12 34 04 12 0B 04 ......PICA.4.... 2C60: 0B FF FF 00 4C 4E 4B 41 00 12 0B 04 0B FF FF 01 ....LNKA........ 2C70: 4C 4E 4B 42 00 12 0C 04 0B FF FF 0A 02 4C 4E 4B LNKB.........LNK 2C80: 43 00 12 0C 04 0B FF FF 0A 03 4C 4E 4B 44 00 08 C.........LNKD.. 2C90: 41 50 49 41 12 2C 04 12 09 04 0B FF FF 00 00 0A APIA.,.......... 2CA0: 36 12 09 04 0B FF FF 01 00 0A 3D 12 0A 04 0B FF 6.........=..... 2CB0: FF 0A 02 00 0A 3F 12 0A 04 0B FF FF 0A 03 00 0A .....?.......... 2CC0: 3E 14 1B 5F 50 52 54 00 70 41 50 49 41 60 A0 0C >.._PRT.pAPIA`.. 2CD0: 92 50 49 43 46 70 50 49 43 41 60 A4 60 5B 82 40 .PICFpPICA`.`[.@ 2CE0: 14 50 43 49 42 14 08 5F 53 31 44 00 A4 01 14 14 .PCIB.._S1D..... 2CF0: 5F 53 33 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04 _S3D...HACK..... 2D00: A4 0A 02 14 14 5F 53 34 44 00 A0 08 48 41 43 4B ....._S4D...HACK 2D10: A4 0A 03 A1 04 A4 0A 02 14 14 5F 53 35 44 00 A0 .........._S5D.. 2D20: 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 14 0F 5F .HACK.........._ 2D30: 41 44 52 00 70 0C 00 00 09 00 60 A4 60 08 5F 55 ADR.p.....`.`._U 2D40: 49 44 0A 2A 08 5F 50 52 57 12 06 02 0A 1F 0A 05 ID.*._PRW....... 2D50: 5B 80 52 53 54 53 02 0A B0 0A 04 5B 81 1A 52 53 [.RSTS.....[..RS 2D60: 54 53 01 50 49 44 30 08 50 49 44 31 08 50 4D 58 TS.PID0.PID1.PMX 2D70: 53 08 50 4E 55 53 08 14 1F 58 4D 45 42 00 70 50 S.PNUS...XMEB.pP 2D80: 4D 58 53 60 7B 60 01 60 70 60 50 4D 58 53 A0 04 MXS`{`.`p`PMXS.. 2D90: 60 A4 01 A1 03 A4 00 08 50 49 43 42 12 34 04 12 `.......PICB.4.. 2DA0: 0B 04 0B FF FF 00 4C 4E 4B 41 00 12 0B 04 0B FF ......LNKA...... 2DB0: FF 01 4C 4E 4B 42 00 12 0C 04 0B FF FF 0A 02 4C ..LNKB.........L 2DC0: 4E 4B 43 00 12 0C 04 0B FF FF 0A 03 4C 4E 4B 44 NKC.........LNKD 2DD0: 00 08 41 50 49 42 12 2C 04 12 09 04 0B FF FF 00 ..APIB.,........ 2DE0: 00 0A 38 12 09 04 0B FF FF 01 00 0A 42 12 0A 04 ..8.........B... 2DF0: 0B FF FF 0A 02 00 0A 47 12 0A 04 0B FF FF 0A 03 .......G........ 2E00: 00 0A 41 14 1B 5F 50 52 54 00 70 41 50 49 42 60 ..A.._PRT.pAPIB` 2E10: A0 0C 92 50 49 43 46 70 50 49 43 42 60 A4 60 10 ...PICFpPICB`.`. 2E20: 43 24 5C 00 14 4E 04 2E 5F 47 50 45 5F 4C 30 33 C$\..N.._GPE_L03 2E30: 00 A0 16 80 93 53 53 54 58 01 00 86 5C 2E 5F 53 .....SSTX...\._S 2E40: 42 5F 56 42 54 4E 0A 02 5B 22 0A 14 7F 54 48 52 B_VBTN..["...THR 2E50: 50 0A 08 54 48 52 50 7D 50 4D 45 32 01 50 4D 45 P..THRP}PME2.PME 2E60: 32 86 5C 2F 03 5F 53 42 5F 50 43 49 30 55 53 42 2.\/._SB_PCI0USB 2E70: 30 0A 02 14 4E 04 2E 5F 47 50 45 5F 4C 30 34 00 0...N.._GPE_L04. 2E80: A0 16 80 93 53 53 54 58 01 00 86 5C 2E 5F 53 42 ....SSTX...\._SB 2E90: 5F 56 42 54 4E 0A 02 5B 22 0A 14 7F 54 48 52 50 _VBTN..["...THRP 2EA0: 0A 10 54 48 52 50 7D 50 4D 45 32 01 50 4D 45 32 ..THRP}PME2.PME2 2EB0: 86 5C 2F 03 5F 53 42 5F 50 43 49 30 55 53 42 31 .\/._SB_PCI0USB1 2EC0: 0A 02 14 4E 04 2E 5F 47 50 45 5F 4C 30 43 00 A0 ...N.._GPE_L0C.. 2ED0: 16 80 93 53 53 54 58 01 00 86 5C 2E 5F 53 42 5F ...SSTX...\._SB_ 2EE0: 56 42 54 4E 0A 02 5B 22 0A 14 7F 47 4C 42 54 0A VBTN..["...GLBT. 2EF0: 10 47 4C 42 54 7D 50 4D 45 32 01 50 4D 45 32 86 .GLBT}PME2.PME2. 2F00: 5C 2F 03 5F 53 42 5F 50 43 49 30 55 53 42 32 0A \/._SB_PCI0USB2. 2F10: 02 14 4E 04 2E 5F 47 50 45 5F 4C 30 45 00 A0 16 ..N.._GPE_L0E... 2F20: 80 93 53 53 54 58 01 00 86 5C 2E 5F 53 42 5F 56 ..SSTX...\._SB_V 2F30: 42 54 4E 0A 02 5B 22 0A 14 7F 47 4C 42 54 0A 40 BTN..["...GLBT.@ 2F40: 47 4C 42 54 7D 50 4D 45 32 01 50 4D 45 32 86 5C GLBT}PME2.PME2.\ 2F50: 2F 03 5F 53 42 5F 50 43 49 30 55 53 42 33 0A 02 /._SB_PCI0USB3.. 2F60: 14 4E 04 2E 5F 47 50 45 5F 4C 30 35 00 A0 16 80 .N.._GPE_L05.... 2F70: 93 53 53 54 58 01 00 86 5C 2E 5F 53 42 5F 56 42 .SSTX...\._SB_VB 2F80: 54 4E 0A 02 5B 22 0A 14 7F 54 48 52 50 0A 20 54 TN..["...THRP. T 2F90: 48 52 50 7D 50 4D 45 32 01 50 4D 45 32 86 5C 2F HRP}PME2.PME2.\/ 2FA0: 03 5F 53 42 5F 50 43 49 30 55 53 42 34 0A 02 14 ._SB_PCI0USB4... 2FB0: 4E 04 2E 5F 47 50 45 5F 4C 32 30 00 A0 16 80 93 N.._GPE_L20..... 2FC0: 53 53 54 58 01 00 86 5C 2E 5F 53 42 5F 56 42 54 SSTX...\._SB_VBT 2FD0: 4E 0A 02 5B 22 0A 14 7F 54 48 52 50 0A 20 54 48 N..["...THRP. TH 2FE0: 52 50 7D 50 4D 45 32 01 50 4D 45 32 86 5C 2F 03 RP}PME2.PME2.\/. 2FF0: 5F 53 42 5F 50 43 49 30 55 53 42 35 0A 02 14 31 _SB_PCI0USB5...1 3000: 2E 5F 47 50 45 5F 4C 30 44 00 5B 22 0A 14 7F 47 ._GPE_L0D.["...G 3010: 4C 42 54 0A 20 47 4C 42 54 7D 50 4D 45 32 01 50 LBT. GLBT}PME2.P 3020: 4D 45 32 86 5C 2E 5F 53 42 5F 50 43 49 30 0A 02 ME2.\._SB_PCI0.. 3030: 14 32 2E 5F 47 50 45 5F 4C 30 42 00 7F 47 4C 42 .2._GPE_L0B..GLB 3040: 54 0A 08 47 4C 42 54 7D 50 4D 45 32 01 50 4D 45 T..GLBT}PME2.PME 3050: 32 86 5C 2F 03 5F 53 42 5F 50 43 49 30 50 43 49 2.\/._SB_PCI0PCI 3060: 36 0A 02 10 4A 0F 2E 5F 53 42 5F 50 43 49 30 5B 6...J.._SB_PCI0[ 3070: 82 4D 0E 49 53 41 5F 08 5F 41 44 52 0C 00 00 1F .M.ISA_._ADR.... 3080: 00 08 5F 55 49 44 0A 0A 5B 80 50 34 30 43 02 0A .._UID..[.P40C.. 3090: 60 0A 04 5B 80 50 34 31 43 02 0A 68 0A 04 5B 82 `..[.P41C..h..[. 30A0: 4E 0B 4D 42 49 4F 08 5F 48 49 44 0C 41 D0 0C 01 N.MBIO._HID.A... 30B0: 08 5F 55 49 44 0A 0B 14 46 0A 5F 43 52 53 00 08 ._UID...F._CRS.. 30C0: 4D 49 4F 31 11 35 0A 32 47 01 62 00 62 00 01 02 MIO1.5.2G.b.b... 30D0: 47 01 65 00 65 00 01 0B 47 01 E0 00 E0 00 01 10 G.e.e...G....... 30E0: 47 01 00 08 00 08 01 60 47 01 00 0C 00 0C 01 80 G......`G....... 30F0: 47 01 60 08 60 08 01 A0 79 00 08 4D 49 4F 32 11 G.`.`...y..MIO2. 3100: 46 04 0A 42 47 01 60 00 60 00 01 01 47 01 64 00 F..BG.`.`...G.d. 3110: 64 00 01 01 47 01 62 00 62 00 01 02 47 01 65 00 d...G.b.b...G.e. 3120: 65 00 01 0B 47 01 E0 00 E0 00 01 10 47 01 00 08 e...G.......G... 3130: 00 08 01 60 47 01 00 0C 00 0C 01 80 47 01 60 08 ...`G.......G.`. 3140: 60 08 01 A0 79 00 A0 10 7D 47 43 4B 42 47 43 4D `...y...}GCKBGCM 3150: 53 00 A4 4D 49 4F 31 A1 06 A4 4D 49 4F 32 10 41 S..MIO1...MIO2.A 3160: 38 2E 5F 53 42 5F 50 43 49 30 5B 82 45 09 55 53 8._SB_PCI0[.E.US 3170: 42 30 08 5F 41 44 52 0C 00 00 1D 00 08 5F 55 49 B0._ADR......_UI 3180: 44 0A 05 08 5F 50 52 57 12 06 02 0A 03 0A 03 5B D..._PRW.......[ 3190: 80 55 50 43 31 02 0A C1 01 5B 81 0B 55 50 43 31 .UPC1....[..UPC1 31A0: 01 4C 45 47 4B 08 14 28 5F 49 4E 49 00 A0 05 48 .LEGK..(_INI...H 31B0: 41 43 4B A1 17 70 4C 45 47 4B 60 7B 60 0A 60 60 ACK..pLEGK`{`.`` 31C0: 7D 60 0A 20 60 70 60 4C 45 47 4B 47 55 53 42 14 }`. `p`LEGKGUSB. 31D0: 13 5F 53 54 41 00 A0 08 47 43 55 43 A4 0A 0F A1 ._STA...GCUC.... 31E0: 03 A4 00 14 08 5F 53 31 44 00 A4 01 14 14 5F 53 ....._S1D....._S 31F0: 33 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 3D...HACK....... 3200: 02 5B 82 41 09 55 53 42 31 08 5F 41 44 52 0C 01 .[.A.USB1._ADR.. 3210: 00 1D 00 08 5F 55 49 44 0A 06 08 5F 50 52 57 12 ...._UID..._PRW. 3220: 06 02 0A 04 0A 03 5B 80 55 50 43 31 02 0A C1 01 ......[.UPC1.... 3230: 5B 81 0B 55 50 43 31 01 4C 45 47 4B 08 14 24 5F [..UPC1.LEGK..$_ 3240: 49 4E 49 00 A0 05 48 41 43 4B A1 17 70 4C 45 47 INI...HACK..pLEG 3250: 4B 60 7B 60 0A 60 60 7D 60 0A 20 60 70 60 4C 45 K`{`.``}`. `p`LE 3260: 47 4B 14 13 5F 53 54 41 00 A0 08 47 43 55 43 A4 GK.._STA...GCUC. 3270: 0A 0F A1 03 A4 00 14 08 5F 53 31 44 00 A4 01 14 ........_S1D.... 3280: 14 5F 53 33 44 00 A0 08 48 41 43 4B A4 0A 03 A1 ._S3D...HACK.... 3290: 04 A4 0A 02 5B 82 41 09 55 53 42 32 08 5F 41 44 ....[.A.USB2._AD 32A0: 52 0C 02 00 1D 00 08 5F 55 49 44 0A 14 08 5F 50 R......_UID..._P 32B0: 52 57 12 06 02 0A 0C 0A 03 5B 80 55 50 43 31 02 RW.......[.UPC1. 32C0: 0A C1 01 5B 81 0B 55 50 43 31 01 4C 45 47 4B 08 ...[..UPC1.LEGK. 32D0: 14 24 5F 49 4E 49 00 A0 05 48 41 43 4B A1 17 70 .$_INI...HACK..p 32E0: 4C 45 47 4B 60 7B 60 0A 60 60 7D 60 0A 20 60 70 LEGK`{`.``}`. `p 32F0: 60 4C 45 47 4B 14 13 5F 53 54 41 00 A0 08 47 43 `LEGK.._STA...GC 3300: 55 43 A4 0A 0F A1 03 A4 00 14 08 5F 53 31 44 00 UC........._S1D. 3310: A4 01 14 14 5F 53 33 44 00 A0 08 48 41 43 4B A4 ...._S3D...HACK. 3320: 0A 03 A1 04 A4 0A 02 5B 82 41 09 55 53 42 33 08 .......[.A.USB3. 3330: 5F 41 44 52 0C 00 00 1A 00 08 5F 55 49 44 0A 15 _ADR......_UID.. 3340: 08 5F 50 52 57 12 06 02 0A 0E 0A 03 5B 80 55 50 ._PRW.......[.UP 3350: 43 31 02 0A C1 01 5B 81 0B 55 50 43 31 01 4C 45 C1....[..UPC1.LE 3360: 47 4B 08 14 24 5F 49 4E 49 00 A0 05 48 41 43 4B GK..$_INI...HACK 3370: A1 17 70 4C 45 47 4B 60 7B 60 0A 60 60 7D 60 0A ..pLEGK`{`.``}`. 3380: 20 60 70 60 4C 45 47 4B 14 13 5F 53 54 41 00 A0 `p`LEGK.._STA.. 3390: 08 47 43 55 43 A4 0A 0F A1 03 A4 00 14 08 5F 53 .GCUC........._S 33A0: 31 44 00 A4 01 14 14 5F 53 33 44 00 A0 08 48 41 1D....._S3D...HA 33B0: 43 4B A4 0A 03 A1 04 A4 0A 02 5B 82 41 09 55 53 CK........[.A.US 33C0: 42 34 08 5F 41 44 52 0C 01 00 1A 00 08 5F 55 49 B4._ADR......_UI 33D0: 44 0A 16 08 5F 50 52 57 12 06 02 0A 05 0A 03 5B D..._PRW.......[ 33E0: 80 55 50 43 31 02 0A C1 01 5B 81 0B 55 50 43 31 .UPC1....[..UPC1 33F0: 01 4C 45 47 4B 08 14 24 5F 49 4E 49 00 A0 05 48 .LEGK..$_INI...H 3400: 41 43 4B A1 17 70 4C 45 47 4B 60 7B 60 0A 60 60 ACK..pLEGK`{`.`` 3410: 7D 60 0A 20 60 70 60 4C 45 47 4B 14 13 5F 53 54 }`. `p`LEGK.._ST 3420: 41 00 A0 08 47 43 55 43 A4 0A 0F A1 03 A4 00 14 A...GCUC........ 3430: 08 5F 53 31 44 00 A4 01 14 14 5F 53 33 44 00 A0 ._S1D....._S3D.. 3440: 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 5B 82 41 .HACK........[.A 3450: 09 55 53 42 35 08 5F 41 44 52 0C 02 00 1A 00 08 .USB5._ADR...... 3460: 5F 55 49 44 0A 1A 08 5F 50 52 57 12 06 02 0A 20 _UID..._PRW.... 3470: 0A 03 5B 80 55 50 43 31 02 0A C1 01 5B 81 0B 55 ..[.UPC1....[..U 3480: 50 43 31 01 4C 45 47 4B 08 14 24 5F 49 4E 49 00 PC1.LEGK..$_INI. 3490: A0 05 48 41 43 4B A1 17 70 4C 45 47 4B 60 7B 60 ..HACK..pLEGK`{` 34A0: 0A 60 60 7D 60 0A 20 60 70 60 4C 45 47 4B 14 13 .``}`. `p`LEGK.. 34B0: 5F 53 54 41 00 A0 08 47 43 55 43 A4 0A 0F A1 03 _STA...GCUC..... 34C0: A4 00 14 08 5F 53 31 44 00 A4 01 14 14 5F 53 33 ...._S1D....._S3 34D0: 44 00 A0 08 48 41 43 4B A4 0A 03 A1 04 A4 0A 02 D...HACK........ 34E0: 10 16 5C 00 08 50 49 43 46 00 14 0C 5F 50 49 43 ..\..PICF..._PIC 34F0: 01 70 68 50 49 43 46 10 49 3F 2E 5F 53 42 5F 50 .phPICF.I?._SB_P 3500: 43 49 30 14 1B 5F 50 52 54 00 70 41 50 49 30 60 CI0.._PRT.pAPI0` 3510: A0 0C 92 50 49 43 46 70 50 49 43 30 60 A4 60 14 ...PICFpPIC0`.`. 3520: 41 3D 5F 43 52 53 00 5B 80 4F 52 4D 46 00 0C 6C A=_CRS.[.ORMF..l 3530: FF 0F 00 0A 04 5B 81 10 4F 52 4D 46 02 4F 52 47 .....[..ORMF.ORG 3540: 50 10 4F 52 4E 44 10 08 4D 45 4D 50 11 4F 14 0B P.ORND..MEMP.O.. 3550: 4A 01 88 0D 00 02 0C 00 00 00 00 00 1F 00 00 00 J............... 3560: 80 00 47 01 F8 0C F8 0C 01 08 88 0D 00 01 0C 03 ..G............. 3570: 00 00 00 00 AF 03 00 00 B0 03 88 0D 00 01 0C 03 ................ 3580: 00 00 B0 03 BB 03 00 00 0C 00 88 0D 00 01 0C 03 ................ 3590: 00 00 BC 03 BF 03 00 00 04 00 88 0D 00 01 0C 03 ................ 35A0: 00 00 C0 03 DF 03 00 00 20 00 88 0D 00 01 0C 03 ........ ....... 35B0: 00 00 E0 03 F7 0C 00 00 18 09 88 0D 00 01 00 03 ................ 35C0: 00 00 00 00 00 00 00 00 00 00 87 17 00 00 0C 03 ................ 35D0: 00 00 00 00 00 00 0A 00 FF FF 0B 00 00 00 00 00 ................ 35E0: 00 00 02 00 87 17 00 00 0C 03 00 00 00 00 00 80 ................ 35F0: 0C 00 FF FF 0D 00 00 00 00 00 00 80 01 00 87 17 ................ 3600: 00 00 0C 03 00 00 00 00 00 00 0E 00 FF FF 0F 00 ................ 3610: 00 00 00 00 00 00 02 00 87 17 00 00 00 01 00 00 ................ 3620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 3630: 00 00 87 17 00 00 00 01 00 00 00 00 00 00 00 00 ................ 3640: FF FF FF F7 00 00 00 00 00 00 00 00 87 17 00 00 ................ 3650: 00 01 00 00 00 00 00 00 98 FF FF 0F 98 FF 00 00 ................ 3660: 00 00 00 10 00 00 87 17 00 00 08 01 00 00 00 00 ................ 3670: 00 C0 97 FF FF FF 97 FF 00 00 00 00 00 40 00 00 .............@.. 3680: 87 17 00 00 0C 01 00 00 00 00 00 00 D2 FE FF FF ................ 3690: D9 FE 00 00 00 00 00 00 08 00 79 00 8A 4D 45 4D ..........y..MEM 36A0: 50 0A 9C 50 4D 49 4E 8A 4D 45 4D 50 0A A0 50 4D P..PMIN.MEMP..PM 36B0: 41 58 8A 4D 45 4D 50 0A A8 50 4C 45 4E 8A 4D 45 AX.MEMP..PLEN.ME 36C0: 4D 50 0A B6 42 4D 49 4E 8A 4D 45 4D 50 0A C2 42 MP..BMIN.MEMP..B 36D0: 4C 45 4E 79 4F 52 47 50 0A 04 42 4D 49 4E 74 0C LENyORGP..BMINt. 36E0: 00 00 10 00 42 4D 49 4E 42 4C 45 4E 70 0C 00 00 ....BMINBLENp... 36F0: 0C 00 50 4D 49 4E 74 42 4D 49 4E 01 50 4D 41 58 ..PMINtBMIN.PMAX 3700: 74 42 4D 49 4E 50 4D 49 4E 50 4C 45 4E 8A 4D 45 tBMINPMINPLEN.ME 3710: 4D 50 0A EA 4D 45 4D 4D 8A 4D 45 4D 50 0A EE 4D MP..MEMM.MEMP..M 3720: 45 4D 41 8A 4D 45 4D 50 0A F6 4D 45 4D 4C 70 47 EMA.MEMP..MEMLpG 3730: 54 4F 4D 4D 45 4D 4D 74 4D 45 4D 41 4D 45 4D 4D TOMMEMMtMEMAMEMM 3740: 4D 45 4D 4C 75 4D 45 4D 4C 70 47 4D 49 30 60 70 MEMLuMEMLpGMI0`p 3750: 47 4D 49 31 61 A0 0D 95 61 60 A0 08 92 93 61 00 GMI1a...a`....a. 3760: 70 61 60 76 60 70 60 4D 45 4D 41 74 4D 45 4D 41 pa`v`p`MEMAtMEMA 3770: 4D 45 4D 4D 4D 45 4D 4C 75 4D 45 4D 4C 8A 4D 45 MEMMMEMLuMEML.ME 3780: 4D 50 0A D0 4D 49 4E 31 8A 4D 45 4D 50 0A D4 4D MP..MIN1.MEMP..M 3790: 41 58 31 8A 4D 45 4D 50 0A DC 4C 45 4E 31 70 47 AX1.MEMP..LEN1pG 37A0: 4D 58 30 4D 41 58 31 70 47 4D 49 30 4D 49 4E 31 MX0MAX1pGMI0MIN1 37B0: 74 4D 41 58 31 4D 49 4E 31 4C 45 4E 31 75 4C 45 tMAX1MIN1LEN1uLE 37C0: 4E 31 8B 4D 45 4D 50 0A 70 49 52 31 4E 8B 4D 45 N1.MEMP.pIR1N.ME 37D0: 4D 50 0A 72 49 52 31 58 8B 4D 45 4D 50 0A 76 49 MP.rIR1X.MEMP.vI 37E0: 52 31 4C 70 0B 00 0D 49 52 31 4E 70 47 49 58 30 R1Lp...IR1NpGIX0 37F0: 49 52 31 58 74 49 52 31 58 49 52 31 4E 49 52 31 IR1XtIR1XIR1NIR1 3800: 4C 75 49 52 31 4C 70 43 4D 52 44 0A 74 60 A0 4F LuIR1LpCMRD.t`.O 3810: 09 94 60 0A 1F 8B 4D 45 4D 50 0A 30 56 47 31 4E ..`...MEMP.0VG1N 3820: 8B 4D 45 4D 50 0A 32 56 47 31 58 8B 4D 45 4D 50 .MEMP.2VG1X.MEMP 3830: 0A 36 56 47 31 4C 8B 4D 45 4D 50 0A 50 56 47 32 .6VG1L.MEMP.PVG2 3840: 4E 8B 4D 45 4D 50 0A 52 56 47 32 58 8B 4D 45 4D N.MEMP.RVG2X.MEM 3850: 50 0A 56 56 47 32 4C 8B 4D 45 4D 50 0A 82 56 47 P.VVG2L.MEMP..VG 3860: 4D 4E 8B 4D 45 4D 50 0A 86 56 47 4D 58 8B 4D 45 MN.MEMP..VGMX.ME 3870: 4D 50 0A 8E 56 47 4D 4C 70 00 56 47 31 4E 70 00 MP..VGMLp.VG1Np. 3880: 56 47 31 58 70 00 56 47 31 4C 70 00 56 47 32 4E VG1Xp.VG1Lp.VG2N 3890: 70 00 56 47 32 58 70 00 56 47 32 4C 70 00 56 47 p.VG2Xp.VG2Lp.VG 38A0: 4D 4E 70 00 56 47 4D 58 70 00 56 47 4D 4C 8A 4D MNp.VGMXp.VGML.M 38B0: 45 4D 50 0B 04 01 55 4D 49 4E 8A 4D 45 4D 50 0B EMP...UMIN.MEMP. 38C0: 08 01 55 4D 41 58 8A 4D 45 4D 50 0B 10 01 55 4C ..UMAX.MEMP...UL 38D0: 45 4E A0 05 47 43 55 43 A1 13 70 00 55 4D 49 4E EN..GCUC..p.UMIN 38E0: 70 00 55 4D 41 58 70 00 55 4C 45 4E A4 4D 45 4D p.UMAXp.ULEN.MEM 38F0: 50 10 44 05 2F 03 5F 53 42 5F 50 43 49 30 49 53 P.D./._SB_PCI0IS 3900: 41 5F 5B 82 42 04 44 4D 41 5F 08 5F 48 49 44 0C A_[.B.DMA_._HID. 3910: 41 D0 02 00 14 31 5F 43 52 53 00 08 44 4D 41 42 A....1_CRS..DMAB 3920: 11 20 0A 1D 47 01 80 00 80 00 01 20 47 01 00 00 . ..G...... G... 3930: 00 00 01 20 47 01 C0 00 C0 00 01 20 2A 10 12 79 ... G...... *..y 3940: 00 A4 44 4D 41 42 10 43 04 2F 03 5F 53 42 5F 50 ..DMAB.C./._SB_P 3950: 43 49 30 49 53 41 5F 5B 82 31 46 50 55 5F 08 5F CI0ISA_[.1FPU_._ 3960: 48 49 44 0C 41 D0 0C 04 14 21 5F 43 52 53 00 08 HID.A....!_CRS.. 3970: 46 50 55 42 11 10 0A 0D 47 01 F0 00 F0 00 01 10 FPUB....G....... 3980: 22 00 20 79 00 A4 46 50 55 42 10 41 05 2F 03 5F ". y..FPUB.A./._ 3990: 53 42 5F 50 43 49 30 49 53 41 5F 5B 82 3F 50 49 SB_PCI0ISA_[.?PI 39A0: 43 5F 08 5F 48 49 44 0B 41 D0 14 31 5F 43 52 53 C_._HID.A..1_CRS 39B0: 00 08 50 49 43 42 11 20 0A 1D 47 01 20 00 20 00 ..PICB. ..G. . . 39C0: 01 20 47 01 A0 00 A0 00 01 20 47 01 D0 04 D0 04 . G...... G..... 39D0: 01 02 22 04 00 79 00 A4 50 49 43 42 10 3F 2F 03 .."..y..PICB.?/. 39E0: 5F 53 42 5F 50 43 49 30 49 53 41 5F 5B 82 2E 53 _SB_PCI0ISA_[..S 39F0: 50 4B 5F 08 5F 48 49 44 0C 41 D0 08 00 14 1E 5F PK_._HID.A....._ 3A00: 43 52 53 00 08 53 50 4B 42 11 0D 0A 0A 47 01 61 CRS..SPKB....G.a 3A10: 00 61 00 01 01 79 00 A4 53 50 4B 42 10 43 04 2F .a...y..SPKB.C./ 3A20: 03 5F 53 42 5F 50 43 49 30 49 53 41 5F 5B 82 31 ._SB_PCI0ISA_[.1 3A30: 52 54 43 5F 08 5F 48 49 44 0C 41 D0 0B 00 14 21 RTC_._HID.A....! 3A40: 5F 43 52 53 00 08 52 54 43 42 11 10 0A 0D 47 01 _CRS..RTCB....G. 3A50: 70 00 70 00 01 10 22 00 01 79 00 A4 52 54 43 42 p.p..."..y..RTCB 3A60: 10 43 04 2F 03 5F 53 42 5F 50 43 49 30 49 53 41 .C./._SB_PCI0ISA 3A70: 5F 5B 82 31 54 4D 52 5F 08 5F 48 49 44 0C 41 D0 _[.1TMR_._HID.A. 3A80: 01 00 14 21 5F 43 52 53 00 08 54 4D 52 42 11 10 ...!_CRS..TMRB.. 3A90: 0A 0D 47 01 40 00 40 00 01 20 22 01 00 79 00 A4 ..G.@.@.. "..y.. 3AA0: 54 4D 52 42 10 4A 5E 5F 53 42 5F 5B 81 24 2F 03 TMRB.J^_SB_[.$/. 3AB0: 50 43 49 30 49 53 41 5F 50 34 30 43 01 50 52 51 PCI0ISA_P40C.PRQ 3AC0: 30 08 50 52 51 31 08 50 52 51 32 08 50 52 51 33 0.PRQ1.PRQ2.PRQ3 3AD0: 08 5B 81 24 2F 03 50 43 49 30 49 53 41 5F 50 34 .[.$/.PCI0ISA_P4 3AE0: 31 43 01 50 52 51 34 08 50 52 51 35 08 50 52 51 1C.PRQ4.PRQ5.PRQ 3AF0: 36 08 50 52 51 37 08 5B 82 41 0B 4C 4E 4B 41 08 6.PRQ7.[.A.LNKA. 3B00: 5F 48 49 44 0C 41 D0 0C 0F 08 5F 55 49 44 0A 0C _HID.A...._UID.. 3B10: 08 5F 50 52 53 11 09 0A 06 23 F8 9E 18 79 00 14 ._PRS....#...y.. 3B20: 1A 5F 53 54 41 00 70 0A 0B 60 A0 0D 7B 0A 80 50 ._STA.p..`..{..P 3B30: 52 51 30 61 70 0A 09 60 A4 60 14 11 5F 44 49 53 RQ0ap..`.`.._DIS 3B40: 00 7D 50 52 51 30 0A 80 50 52 51 30 14 41 04 5F .}PRQ0..PRQ0.A._ 3B50: 43 52 53 00 08 50 52 52 30 11 09 0A 06 23 10 00 CRS..PRR0....#.. 3B60: 18 79 00 8B 50 52 52 30 01 49 51 52 5F 70 50 52 .y..PRR0.IQR_pPR 3B70: 51 30 60 A0 0C 92 95 60 0A 80 70 00 49 51 52 5F Q0`....`..p.IQR_ 3B80: A1 08 79 01 60 49 51 52 5F A4 50 52 52 30 14 1B ..y.`IQR_.PRR0.. 3B90: 5F 53 52 53 01 8B 68 01 49 51 52 5F 82 49 51 52 _SRS..h.IQR_.IQR 3BA0: 5F 60 76 60 70 60 50 52 51 30 5B 82 41 0B 4C 4E _`v`p`PRQ0[.A.LN 3BB0: 4B 42 08 5F 48 49 44 0C 41 D0 0C 0F 08 5F 55 49 KB._HID.A...._UI 3BC0: 44 0A 0D 08 5F 50 52 53 11 09 0A 06 23 F8 9E 18 D..._PRS....#... 3BD0: 79 00 14 1A 5F 53 54 41 00 70 0A 0B 60 A0 0D 7B y..._STA.p..`..{ 3BE0: 0A 80 50 52 51 31 61 70 0A 09 60 A4 60 14 11 5F ..PRQ1ap..`.`.._ 3BF0: 44 49 53 00 7D 50 52 51 31 0A 80 50 52 51 31 14 DIS.}PRQ1..PRQ1. 3C00: 41 04 5F 43 52 53 00 08 50 52 52 31 11 09 0A 06 A._CRS..PRR1.... 3C10: 23 10 00 18 79 00 8B 50 52 52 31 01 49 51 52 5F #...y..PRR1.IQR_ 3C20: 70 50 52 51 31 60 A0 0C 92 95 60 0A 80 70 00 49 pPRQ1`....`..p.I 3C30: 51 52 5F A1 08 79 01 60 49 51 52 5F A4 50 52 52 QR_..y.`IQR_.PRR 3C40: 31 14 1B 5F 53 52 53 01 8B 68 01 49 51 52 5F 82 1.._SRS..h.IQR_. 3C50: 49 51 52 5F 60 76 60 70 60 50 52 51 31 5B 82 41 IQR_`v`p`PRQ1[.A 3C60: 0B 4C 4E 4B 43 08 5F 48 49 44 0C 41 D0 0C 0F 08 .LNKC._HID.A.... 3C70: 5F 55 49 44 0A 0E 08 5F 50 52 53 11 09 0A 06 23 _UID..._PRS....# 3C80: F8 9E 18 79 00 14 1A 5F 53 54 41 00 70 0A 0B 60 ...y..._STA.p..` 3C90: A0 0D 7B 0A 80 50 52 51 32 61 70 0A 09 60 A4 60 ..{..PRQ2ap..`.` 3CA0: 14 11 5F 44 49 53 00 7D 50 52 51 32 0A 80 50 52 .._DIS.}PRQ2..PR 3CB0: 51 32 14 41 04 5F 43 52 53 00 08 50 52 52 32 11 Q2.A._CRS..PRR2. 3CC0: 09 0A 06 23 10 00 18 79 00 8B 50 52 52 32 01 49 ...#...y..PRR2.I 3CD0: 51 52 5F 70 50 52 51 32 60 A0 0C 92 95 60 0A 80 QR_pPRQ2`....`.. 3CE0: 70 00 49 51 52 5F A1 08 79 01 60 49 51 52 5F A4 p.IQR_..y.`IQR_. 3CF0: 50 52 52 32 14 1B 5F 53 52 53 01 8B 68 01 49 51 PRR2.._SRS..h.IQ 3D00: 52 5F 82 49 51 52 5F 60 76 60 70 60 50 52 51 32 R_.IQR_`v`p`PRQ2 3D10: 5B 82 41 0B 4C 4E 4B 44 08 5F 48 49 44 0C 41 D0 [.A.LNKD._HID.A. 3D20: 0C 0F 08 5F 55 49 44 0A 0F 08 5F 50 52 53 11 09 ..._UID..._PRS.. 3D30: 0A 06 23 F8 9E 18 79 00 14 1A 5F 53 54 41 00 70 ..#...y..._STA.p 3D40: 0A 0B 60 A0 0D 7B 0A 80 50 52 51 33 61 70 0A 09 ..`..{..PRQ3ap.. 3D50: 60 A4 60 14 11 5F 44 49 53 00 7D 50 52 51 33 0A `.`.._DIS.}PRQ3. 3D60: 80 50 52 51 33 14 41 04 5F 43 52 53 00 08 50 52 .PRQ3.A._CRS..PR 3D70: 52 33 11 09 0A 06 23 10 00 18 79 00 8B 50 52 52 R3....#...y..PRR 3D80: 33 01 49 51 52 5F 70 50 52 51 33 60 A0 0C 92 95 3.IQR_pPRQ3`.... 3D90: 60 0A 80 70 00 49 51 52 5F A1 08 79 01 60 49 51 `..p.IQR_..y.`IQ 3DA0: 52 5F A4 50 52 52 33 14 1B 5F 53 52 53 01 8B 68 R_.PRR3.._SRS..h 3DB0: 01 49 51 52 5F 82 49 51 52 5F 60 76 60 70 60 50 .IQR_.IQR_`v`p`P 3DC0: 52 51 33 5B 82 41 0B 4C 4E 4B 45 08 5F 48 49 44 RQ3[.A.LNKE._HID 3DD0: 0C 41 D0 0C 0F 08 5F 55 49 44 0A 10 08 5F 50 52 .A...._UID..._PR 3DE0: 53 11 09 0A 06 23 F8 9E 18 79 00 14 1A 5F 53 54 S....#...y..._ST 3DF0: 41 00 70 0A 0B 60 A0 0D 7B 0A 80 50 52 51 34 61 A.p..`..{..PRQ4a 3E00: 70 0A 09 60 A4 60 14 11 5F 44 49 53 00 7D 50 52 p..`.`.._DIS.}PR 3E10: 51 34 0A 80 50 52 51 34 14 41 04 5F 43 52 53 00 Q4..PRQ4.A._CRS. 3E20: 08 50 52 52 34 11 09 0A 06 23 10 00 18 79 00 8B .PRR4....#...y.. 3E30: 50 52 52 34 01 49 51 52 5F 70 50 52 51 34 60 A0 PRR4.IQR_pPRQ4`. 3E40: 0C 92 95 60 0A 80 70 00 49 51 52 5F A1 08 79 01 ...`..p.IQR_..y. 3E50: 60 49 51 52 5F A4 50 52 52 34 14 1B 5F 53 52 53 `IQR_.PRR4.._SRS 3E60: 01 8B 68 01 49 51 52 5F 82 49 51 52 5F 60 76 60 ..h.IQR_.IQR_`v` 3E70: 70 60 50 52 51 34 5B 82 41 0B 4C 4E 4B 46 08 5F p`PRQ4[.A.LNKF._ 3E80: 48 49 44 0C 41 D0 0C 0F 08 5F 55 49 44 0A 11 08 HID.A...._UID... 3E90: 5F 50 52 53 11 09 0A 06 23 F8 9E 18 79 00 14 1A _PRS....#...y... 3EA0: 5F 53 54 41 00 70 0A 0B 60 A0 0D 7B 0A 80 50 52 _STA.p..`..{..PR 3EB0: 51 35 61 70 0A 09 60 A4 60 14 11 5F 44 49 53 00 Q5ap..`.`.._DIS. 3EC0: 7D 50 52 51 35 0A 80 50 52 51 35 14 41 04 5F 43 }PRQ5..PRQ5.A._C 3ED0: 52 53 00 08 50 52 52 35 11 09 0A 06 23 10 00 18 RS..PRR5....#... 3EE0: 79 00 8B 50 52 52 35 01 49 51 52 5F 70 50 52 51 y..PRR5.IQR_pPRQ 3EF0: 35 60 A0 0C 92 95 60 0A 80 70 00 49 51 52 5F A1 5`....`..p.IQR_. 3F00: 08 79 01 60 49 51 52 5F A4 50 52 52 35 14 1B 5F .y.`IQR_.PRR5.._ 3F10: 53 52 53 01 8B 68 01 49 51 52 5F 82 49 51 52 5F SRS..h.IQR_.IQR_ 3F20: 60 76 60 70 60 50 52 51 35 5B 82 41 0B 4C 4E 4B `v`p`PRQ5[.A.LNK 3F30: 47 08 5F 48 49 44 0C 41 D0 0C 0F 08 5F 55 49 44 G._HID.A...._UID 3F40: 0A 12 08 5F 50 52 53 11 09 0A 06 23 F8 9E 18 79 ..._PRS....#...y 3F50: 00 14 1A 5F 53 54 41 00 70 0A 0B 60 A0 0D 7B 0A ..._STA.p..`..{. 3F60: 80 50 52 51 36 61 70 0A 09 60 A4 60 14 11 5F 44 .PRQ6ap..`.`.._D 3F70: 49 53 00 7D 50 52 51 36 0A 80 50 52 51 36 14 41 IS.}PRQ6..PRQ6.A 3F80: 04 5F 43 52 53 00 08 50 52 52 36 11 09 0A 06 23 ._CRS..PRR6....# 3F90: 10 00 18 79 00 8B 50 52 52 36 01 49 51 52 5F 70 ...y..PRR6.IQR_p 3FA0: 50 52 51 36 60 A0 0C 92 95 60 0A 80 70 00 49 51 PRQ6`....`..p.IQ 3FB0: 52 5F A1 08 79 01 60 49 51 52 5F A4 50 52 52 36 R_..y.`IQR_.PRR6 3FC0: 14 1B 5F 53 52 53 01 8B 68 01 49 51 52 5F 82 49 .._SRS..h.IQR_.I 3FD0: 51 52 5F 60 76 60 70 60 50 52 51 36 5B 82 41 0B QR_`v`p`PRQ6[.A. 3FE0: 4C 4E 4B 48 08 5F 48 49 44 0C 41 D0 0C 0F 08 5F LNKH._HID.A...._ 3FF0: 55 49 44 0A 13 08 5F 50 52 53 11 09 0A 06 23 F8 UID..._PRS....#. 4000: 9E 18 79 00 14 1A 5F 53 54 41 00 70 0A 0B 60 A0 ..y..._STA.p..`. 4010: 0D 7B 0A 80 50 52 51 37 61 70 0A 09 60 A4 60 14 .{..PRQ7ap..`.`. 4020: 11 5F 44 49 53 00 7D 50 52 51 37 0A 80 50 52 51 ._DIS.}PRQ7..PRQ 4030: 37 14 41 04 5F 43 52 53 00 08 50 52 52 37 11 09 7.A._CRS..PRR7.. 4040: 0A 06 23 10 00 18 79 00 8B 50 52 52 37 01 49 51 ..#...y..PRR7.IQ 4050: 52 5F 70 50 52 51 37 60 A0 0C 92 95 60 0A 80 70 R_pPRQ7`....`..p 4060: 00 49 51 52 5F A1 08 79 01 60 49 51 52 5F A4 50 .IQR_..y.`IQR_.P 4070: 52 52 37 14 1B 5F 53 52 53 01 8B 68 01 49 51 52 RR7.._SRS..h.IQR 4080: 5F 82 49 51 52 5F 60 76 60 70 60 50 52 51 37 10 _.IQR_`v`p`PRQ7. 4090: 39 5F 53 42 5F 5B 82 32 48 50 45 54 08 5F 48 49 9_SB_[.2HPET._HI 40A0: 44 0C 41 D0 01 03 14 22 5F 43 52 53 00 08 48 42 D.A...."_CRS..HB 40B0: 41 42 11 11 0A 0E 86 09 00 00 00 00 D0 FE 00 04 AB.............. 40C0: 00 00 79 00 A4 48 42 41 42 10 4A 16 5C 00 5B 80 ..y..HBAB.J.\.[. 40D0: 57 45 4E 58 01 0B 0A 0C 0A 06 5B 81 24 57 45 4E WENX......[.$WEN 40E0: 58 01 57 45 4E 31 08 57 45 4E 32 08 57 45 4E 33 X.WEN1.WEN2.WEN3 40F0: 08 57 45 4E 34 08 57 45 4E 35 08 57 45 4E 36 08 .WEN4.WEN5.WEN6. 4100: 5B 80 57 53 54 58 01 0B 00 0C 0A 0A 5B 81 38 57 [.WSTX......[.8W 4110: 53 54 58 01 57 53 54 30 08 50 4D 45 4A 08 50 45 STX.WST0.PMEJ.PE 4120: 4E 41 08 50 45 4E 4A 08 57 53 54 31 08 57 53 54 NA.PENJ.WST1.WST 4130: 32 08 57 53 54 33 08 57 53 54 34 08 57 53 54 35 2.WST3.WST4.WST5 4140: 08 57 53 54 36 08 5B 80 53 47 50 58 01 0B 4B 0C .WST6.[.SGPX..K. 4150: 0A 06 5B 81 24 53 47 50 58 01 53 47 50 31 08 53 ..[.$SGPX.SGP1.S 4160: 47 50 32 08 53 47 50 33 08 53 47 50 34 08 53 47 GP2.SGP3.SGP4.SG 4170: 50 35 08 53 47 50 36 08 5B 80 4C 45 44 58 01 0B P5.SGP6.[.LEDX.. 4180: 5D 0C 0A 02 5B 81 10 4C 45 44 58 01 4C 45 44 31 ]...[..LEDX.LED1 4190: 08 4C 45 44 32 08 14 4D 06 50 53 4B 4D 01 A0 45 .LED2..M.PSKM..E 41A0: 06 7D 93 68 0A 03 93 68 01 00 70 0A 55 53 49 4F .}.h...h..p.USIO 41B0: 49 70 0A 07 53 49 4F 49 70 0A 07 53 49 4F 44 70 Ip..SIOIp..SIODp 41C0: 0A F0 53 49 4F 49 70 53 49 4F 44 60 7D 60 0A 60 ..SIOIpSIOD`}`.` 41D0: 53 49 4F 44 70 0A AA 53 49 4F 49 70 57 53 54 31 SIODp..SIOIpWST1 41E0: 60 7B 60 0A 18 57 53 54 31 70 47 53 54 53 60 7B `{`..WST1pGSTS`{ 41F0: 60 01 47 53 54 53 70 57 45 4E 31 60 7D 60 0A 18 `.GSTSpWEN1`}`.. 4200: 57 45 4E 31 14 2F 47 50 4B 4D 00 70 57 45 4E 31 WEN1./GPKM.pWEN1 4210: 60 7B 60 0A E7 57 45 4E 31 70 57 53 54 31 60 7B `{`..WEN1pWST1`{ 4220: 60 0A 18 57 53 54 31 70 47 53 54 53 60 7B 60 01 `..WST1pGSTS`{`. 4230: 47 53 54 53 10 41 D7 2F 03 5F 53 42 5F 50 43 49 GSTS.A./._SB_PCI 4240: 30 49 53 41 5F 5B 80 4E 53 49 4F 01 0A 2E 0A 02 0ISA_[.NSIO..... 4250: 5B 81 10 4E 53 49 4F 01 49 4E 44 58 08 44 41 54 [..NSIO.INDX.DAT 4260: 41 08 5B 86 47 05 49 4E 44 58 44 41 54 41 01 00 A.[.G.INDXDATA.. 4270: 10 43 46 47 5F 08 00 20 4C 44 4E 5F 08 00 40 0C .CFG_.. LDN_..@. 4280: 53 49 49 44 08 00 48 07 41 43 54 52 08 00 48 17 SIID..H.ACTR..H. 4290: 49 4F 41 48 08 49 4F 41 4C 08 00 40 07 49 4E 54 IOAH.IOAL..@.INT 42A0: 52 08 00 18 44 4D 43 48 08 00 48 3D 4F 50 54 31 R...DMCH..H=OPT1 42B0: 08 4F 50 54 32 08 4F 50 54 33 08 5B 80 43 4F 4D .OPT2.OPT3.[.COM 42C0: 5F 02 0A 80 01 5B 81 14 43 4F 4D 5F 01 4E 53 43 _....[..COM_.NSC 42D0: 41 03 00 01 4E 53 43 42 03 00 01 5B 80 46 44 50 A...NSCB...[.FDP 42E0: 41 02 0A 81 01 5B 81 14 46 44 50 41 01 4C 50 54 A....[..FDPA.LPT 42F0: 4F 02 00 02 46 44 44 43 01 00 03 5B 01 4D 54 58 O...FDDC...[.MTX 4300: 5F 01 14 18 53 49 4F 44 01 43 53 49 4F 0A 55 70 _...SIOD.CSIO.Up 4310: 68 4C 44 4E 5F 43 53 49 4F 0A AA 14 0C 43 53 49 hLDN_CSIO....CSI 4320: 4F 01 70 68 49 4E 44 58 5B 82 41 2C 46 44 43 5F O.phINDX[.A,FDC_ 4330: 08 5F 48 49 44 0C 41 D0 07 00 14 41 06 5F 53 54 ._HID.A....A._ST 4340: 41 00 08 52 45 54 5F 00 5B 23 4D 54 58 5F FF FF A..RET_.[#MTX_.. 4350: 53 49 4F 44 00 A0 2D 47 43 46 44 43 53 49 4F 0A SIOD..-GCFDCSIO. 4360: 55 A0 12 41 43 54 52 43 53 49 4F 0A AA 70 0A 0F U..ACTRCSIO..p.. 4370: 52 45 54 5F A1 0E 43 53 49 4F 0A AA 70 0A 0D 52 RET_..CSIO..p..R 4380: 45 54 5F A1 0D 43 53 49 4F 0A AA 70 00 52 45 54 ET_..CSIO..p.RET 4390: 5F 5B 27 4D 54 58 5F A4 52 45 54 5F 14 31 5F 44 _['MTX_.RET_.1_D 43A0: 49 53 00 5B 23 4D 54 58 5F FF FF 53 49 4F 44 00 IS.[#MTX_..SIOD. 43B0: 43 53 49 4F 0A 55 70 00 49 4E 54 52 70 00 41 43 CSIO.Up.INTRp.AC 43C0: 54 52 43 53 49 4F 0A AA 5B 27 4D 54 58 5F 14 4A TRCSIO..['MTX_.J 43D0: 11 5F 43 52 53 00 08 46 44 42 30 11 1B 0A 18 47 ._CRS..FDB0....G 43E0: 01 F0 03 F0 03 08 06 47 01 F7 03 F7 03 01 01 22 .......G......." 43F0: 40 00 2A 04 08 79 00 8C 46 44 42 30 0A 02 49 4F @.*..y..FDB0..IO 4400: 4C 30 8C 46 44 42 30 0A 03 49 4F 48 30 8C 46 44 L0.FDB0..IOH0.FD 4410: 42 30 0A 04 49 4F 4C 31 8C 46 44 42 30 0A 05 49 B0..IOL1.FDB0..I 4420: 4F 48 31 8C 46 44 42 30 0A 0A 49 32 4C 30 8C 46 OH1.FDB0..I2L0.F 4430: 44 42 30 0A 0B 49 32 48 30 8C 46 44 42 30 0A 0C DB0..I2H0.FDB0.. 4440: 49 32 4C 31 8C 46 44 42 30 0A 0D 49 32 48 31 8B I2L1.FDB0..I2H1. 4450: 46 44 42 30 0A 11 49 51 52 5F 8C 46 44 42 30 0A FDB0..IQR_.FDB0. 4460: 14 44 41 4D 5F 5B 23 4D 54 58 5F FF FF 53 49 4F .DAM_[#MTX_..SIO 4470: 44 00 43 53 49 4F 0A 55 70 49 4F 41 48 49 4F 48 D.CSIO.UpIOAHIOH 4480: 30 70 49 4F 41 48 49 4F 48 31 70 49 4F 41 4C 49 0pIOAHIOH1pIOALI 4490: 4F 4C 30 70 49 4F 41 4C 49 4F 4C 31 70 49 4F 41 OL0pIOALIOL1pIOA 44A0: 48 49 32 48 30 70 49 4F 41 48 49 32 48 31 72 49 HI2H0pIOAHI2H1rI 44B0: 4F 41 4C 0A 07 49 32 4C 30 72 49 4F 41 4C 0A 07 OAL..I2L0rIOAL.. 44C0: 49 32 4C 31 79 01 49 4E 54 52 49 51 52 5F 79 01 I2L1y.INTRIQR_y. 44D0: 44 4D 43 48 44 41 4D 5F 43 53 49 4F 0A AA 5B 27 DMCHDAM_CSIO..[' 44E0: 4D 54 58 5F A4 46 44 42 30 14 4E 05 5F 50 52 53 MTX_.FDB0.N._PRS 44F0: 00 08 46 44 42 31 11 4C 04 0A 48 30 47 01 F0 03 ..FDB1.L..H0G... 4500: F0 03 08 06 47 01 F7 03 F7 03 01 01 22 40 00 2A ....G......."@.* 4510: 04 08 30 47 01 F0 03 F0 03 08 06 47 01 F7 03 F7 ..0G.......G.... 4520: 03 01 01 22 F8 10 2A 0E 08 30 47 01 70 03 70 03 ..."..*..0G.p.p. 4530: 08 06 47 01 77 03 77 03 01 01 22 F8 10 2A 0E 08 ..G.w.w..."..*.. 4540: 38 79 00 A4 46 44 42 31 14 42 0A 5F 53 52 53 01 8y..FDB1.B._SRS. 4550: 8C 68 0A 02 49 4F 4C 4F 8C 68 0A 03 49 4F 48 49 .h..IOLO.h..IOHI 4560: 8B 68 0A 11 49 51 52 5F 8C 68 0A 14 44 41 4D 5F .h..IQR_.h..DAM_ 4570: 5B 23 4D 54 58 5F FF FF 53 49 4F 44 00 43 53 49 [#MTX_..SIOD.CSI 4580: 4F 0A 55 70 49 4F 4C 4F 49 4F 41 4C 70 49 4F 48 O.UpIOLOIOALpIOH 4590: 49 49 4F 41 48 43 53 49 4F 0A AA A0 0E 93 49 4F IIOAHCSIO.....IO 45A0: 4C 4F 0A 70 70 01 46 44 44 43 A0 0E 93 49 4F 4C LO.pp.FDDC...IOL 45B0: 4F 0A F0 70 00 46 44 44 43 43 53 49 4F 0A 55 82 O..p.FDDCCSIO.U. 45C0: 49 51 52 5F 60 74 60 01 49 4E 54 52 82 44 41 4D IQR_`t`.INTR.DAM 45D0: 5F 60 74 60 01 44 4D 43 48 70 01 41 43 54 52 43 _`t`.DMCHp.ACTRC 45E0: 53 49 4F 0A AA 5B 27 4D 54 58 5F 5B 82 4F 05 4B SIO..['MTX_[.O.K 45F0: 42 44 5F 08 5F 48 49 44 0C 41 D0 03 03 14 24 5F BD_._HID.A....$_ 4600: 53 54 41 00 08 52 45 54 5F 00 70 00 52 45 54 5F STA..RET_.p.RET_ 4610: A0 0C 47 43 4B 42 70 0A 0F 52 45 54 5F A4 52 45 ..GCKBp..RET_.RE 4620: 54 5F 14 29 5F 43 52 53 00 08 4B 42 44 42 11 18 T_.)_CRS..KBDB.. 4630: 0A 15 47 01 60 00 60 00 01 01 47 01 64 00 64 00 ..G.`.`...G.d.d. 4640: 01 01 22 02 00 79 00 A4 4B 42 44 42 5B 82 41 08 .."..y..KBDB[.A. 4650: 4D 4F 55 5F 08 5F 48 49 44 0C 41 D0 0F 13 14 2A MOU_._HID.A....* 4660: 5F 53 54 41 00 08 52 45 54 5F 00 70 00 52 45 54 _STA..RET_.p.RET 4670: 5F A0 12 47 43 4F 4E A0 0C 47 43 4D 53 70 0A 0F _..GCON..GCMSp.. 4680: 52 45 54 5F A4 52 45 54 5F 14 45 04 5F 43 52 53 RET_.RET_.E._CRS 4690: 00 08 4D 4F 55 42 11 08 0A 05 22 00 10 79 00 08 ..MOUB...."..y.. 46A0: 4D 4F 4B 42 11 18 0A 15 47 01 60 00 60 00 01 01 MOKB....G.`.`... 46B0: 47 01 64 00 64 00 01 01 22 00 10 79 00 A0 0A 47 G.d.d..."..y...G 46C0: 43 4B 42 A4 4D 4F 55 42 A1 06 A4 4D 4F 4B 42 5B CKB.MOUB...MOKB[ 46D0: 82 49 40 50 52 54 5F 08 5F 48 49 44 0C 41 D0 04 .I@PRT_._HID.A.. 46E0: 01 14 42 06 5F 53 54 41 00 08 52 45 54 5F 00 5B ..B._STA..RET_.[ 46F0: 23 4D 54 58 5F FF FF 53 49 4F 44 0A 03 A0 2D 47 #MTX_..SIOD...-G 4700: 43 50 50 43 53 49 4F 0A 55 A0 12 41 43 54 52 43 CPPCSIO.U..ACTRC 4710: 53 49 4F 0A AA 70 0A 0F 52 45 54 5F A1 0E 43 53 SIO..p..RET_..CS 4720: 49 4F 0A AA 70 0A 0D 52 45 54 5F A1 0D 43 53 49 IO..p..RET_..CSI 4730: 4F 0A AA 70 00 52 45 54 5F 5B 27 4D 54 58 5F A4 O..p.RET_['MTX_. 4740: 52 45 54 5F 14 39 5F 44 49 53 00 5B 23 4D 54 58 RET_.9_DIS.[#MTX 4750: 5F FF FF 53 49 4F 44 0A 03 43 53 49 4F 0A 55 70 _..SIOD..CSIO.Up 4760: 00 49 4E 54 52 70 0A 04 44 4D 43 48 70 00 41 43 .INTRp..DMCHp.AC 4770: 54 52 43 53 49 4F 0A AA 5B 27 4D 54 58 5F 14 49 TRCSIO..['MTX_.I 4780: 18 5F 43 52 53 00 08 50 54 42 30 11 1B 0A 18 47 ._CRS..PTB0....G 4790: 01 78 03 78 03 08 08 47 01 78 07 78 07 08 08 22 .x.x...G.x.x..." 47A0: 80 00 2A 00 08 79 00 8C 50 54 42 30 0A 02 49 4F ..*..y..PTB0..IO 47B0: 4C 30 8C 50 54 42 30 0A 03 49 4F 48 30 8C 50 54 L0.PTB0..IOH0.PT 47C0: 42 30 0A 04 49 4F 4C 31 8C 50 54 42 30 0A 05 49 B0..IOL1.PTB0..I 47D0: 4F 48 31 8C 50 54 42 30 0A 06 4C 41 4C 4E 8C 50 OH1.PTB0..LALN.P 47E0: 54 42 30 0A 07 4C 4C 45 4E 8C 50 54 42 30 0A 0A TB0..LLEN.PTB0.. 47F0: 49 32 4C 30 8C 50 54 42 30 0A 0B 49 32 48 30 8C I2L0.PTB0..I2H0. 4800: 50 54 42 30 0A 0C 49 32 4C 31 8C 50 54 42 30 0A PTB0..I2L1.PTB0. 4810: 0D 49 32 48 31 8C 50 54 42 30 0A 0E 48 41 4C 4E .I2H1.PTB0..HALN 4820: 8C 50 54 42 30 0A 0F 48 4C 45 4E 8B 50 54 42 30 .PTB0..HLEN.PTB0 4830: 0A 11 49 51 52 5F 8C 50 54 42 30 0A 14 44 41 4D ..IQR_.PTB0..DAM 4840: 5F 5B 23 4D 54 58 5F FF FF 53 49 4F 44 0A 03 43 _[#MTX_..SIOD..C 4850: 53 49 4F 0A 55 70 49 4F 41 48 49 4F 48 30 70 49 SIO.UpIOAHIOH0pI 4860: 4F 41 48 49 4F 48 31 70 49 4F 41 4C 49 4F 4C 30 OAHIOH1pIOALIOL0 4870: 70 49 4F 41 4C 49 4F 4C 31 72 49 4F 41 48 0A 04 pIOALIOL1rIOAH.. 4880: 49 32 48 30 72 49 4F 41 48 0A 04 49 32 48 31 70 I2H0rIOAH..I2H1p 4890: 49 4F 41 4C 49 32 4C 30 70 49 4F 41 4C 49 32 4C IOALI2L0pIOALI2L 48A0: 31 79 01 49 4E 54 52 49 51 52 5F 79 01 44 4D 43 1y.INTRIQR_y.DMC 48B0: 48 44 41 4D 5F A0 24 93 49 4F 4C 30 0A BC 70 0A HDAM_.$.IOL0..p. 48C0: 04 4C 41 4C 4E 70 0A 04 4C 4C 45 4E 70 0A 04 48 .LALNp..LLENp..H 48D0: 41 4C 4E 70 0A 04 48 4C 45 4E A0 0E 93 44 4D 43 ALNp..HLEN...DMC 48E0: 48 0A 04 70 00 44 41 4D 5F A0 0D 93 49 4E 54 52 H..p.DAM_...INTR 48F0: 00 70 00 49 51 52 5F 43 53 49 4F 0A AA 5B 27 4D .p.IQR_CSIO..['M 4900: 54 58 5F A4 50 54 42 30 14 4F 0F 5F 50 52 53 00 TX_.PTB0.O._PRS. 4910: 08 50 54 42 31 11 4D 0E 0A E9 30 47 01 78 03 78 .PTB1.M...0G.x.x 4920: 03 08 08 47 01 78 07 78 07 08 08 22 80 00 2A 00 ...G.x.x..."..*. 4930: 08 30 47 01 78 02 78 02 08 08 47 01 78 06 78 06 .0G.x.x...G.x.x. 4940: 08 08 22 20 00 2A 00 08 30 47 01 78 03 78 03 08 .." .*..0G.x.x.. 4950: 08 47 01 78 07 78 07 08 08 22 F8 10 2A 00 08 30 .G.x.x..."..*..0 4960: 47 01 78 03 78 03 08 08 47 01 78 07 78 07 08 08 G.x.x...G.x.x... 4970: 22 F8 10 2A 0E 08 30 47 01 78 02 78 02 08 08 47 "..*..0G.x.x...G 4980: 01 78 06 78 06 08 08 22 F8 10 2A 00 08 30 47 01 .x.x..."..*..0G. 4990: 78 02 78 02 08 08 47 01 78 06 78 06 08 08 22 F8 x.x...G.x.x...". 49A0: 10 2A 0E 08 30 47 01 78 03 78 03 08 08 47 01 78 .*..0G.x.x...G.x 49B0: 07 78 07 08 08 22 00 00 2A 00 08 30 47 01 78 02 .x..."..*..0G.x. 49C0: 78 02 08 08 47 01 78 06 78 06 08 08 22 00 00 2A x...G.x.x..."..* 49D0: 00 08 30 47 01 BC 03 BC 03 04 04 47 01 BC 07 BC ..0G.......G.... 49E0: 07 04 04 22 00 00 2A 00 08 30 47 01 BC 03 BC 03 ..."..*..0G..... 49F0: 04 04 47 01 BC 07 BC 07 04 04 22 80 00 2A 00 08 ..G......."..*.. 4A00: 38 79 00 A4 50 54 42 31 14 41 0D 5F 53 52 53 01 8y..PTB1.A._SRS. 4A10: 8C 68 0A 02 49 4F 4C 4F 8C 68 0A 03 49 4F 48 49 .h..IOLO.h..IOHI 4A20: 8B 68 0A 11 49 51 52 5F 8C 68 0A 14 44 41 4D 5F .h..IQR_.h..DAM_ 4A30: 5B 23 4D 54 58 5F FF FF 53 49 4F 44 0A 03 43 53 [#MTX_..SIOD..CS 4A40: 49 4F 0A 55 70 49 4F 4C 4F 49 4F 41 4C 70 49 4F IO.UpIOLOIOALpIO 4A50: 48 49 49 4F 41 48 A0 0F 93 49 4F 4C 4F 0A BC 70 HIIOAH...IOLO..p 4A60: 0A 02 4C 50 54 4F A0 26 93 49 4F 4C 4F 0A 78 A0 ..LPTO.&.IOLO.x. 4A70: 0E 93 49 4F 48 49 0A 02 70 01 4C 50 54 4F A0 0E ..IOHI..p.LPTO.. 4A80: 93 49 4F 48 49 0A 03 70 00 4C 50 54 4F A0 0D 93 .IOHI..p.LPTO... 4A90: 49 51 52 5F 00 70 00 49 4E 54 52 A1 0E 82 49 51 IQR_.p.INTR...IQ 4AA0: 52 5F 60 74 60 01 49 4E 54 52 A0 0E 93 44 41 4D R_`t`.INTR...DAM 4AB0: 5F 00 70 0A 04 44 4D 43 48 A1 0E 82 44 41 4D 5F _.p..DMCH...DAM_ 4AC0: 60 74 60 01 44 4D 43 48 70 01 41 43 54 52 43 53 `t`.DMCHp.ACTRCS 4AD0: 49 4F 0A AA 5B 27 4D 54 58 5F 5B 82 42 25 43 4F IO..['MTX_[.B%CO 4AE0: 4D 41 08 5F 48 49 44 0C 41 D0 05 01 08 5F 55 49 MA._HID.A...._UI 4AF0: 44 01 14 42 06 5F 53 54 41 00 08 52 45 54 5F 00 D..B._STA..RET_. 4B00: 5B 23 4D 54 58 5F FF FF 53 49 4F 44 0A 04 A0 2D [#MTX_..SIOD...- 4B10: 47 43 53 31 43 53 49 4F 0A 55 A0 12 41 43 54 52 GCS1CSIO.U..ACTR 4B20: 43 53 49 4F 0A AA 70 0A 0F 52 45 54 5F A1 0E 43 CSIO..p..RET_..C 4B30: 53 49 4F 0A AA 70 0A 0D 52 45 54 5F A1 0D 43 53 SIO..p..RET_..CS 4B40: 49 4F 0A AA 70 00 52 45 54 5F 5B 27 4D 54 58 5F IO..p.RET_['MTX_ 4B50: A4 52 45 54 5F 14 32 5F 44 49 53 00 5B 23 4D 54 .RET_.2_DIS.[#MT 4B60: 58 5F FF FF 53 49 4F 44 0A 04 43 53 49 4F 0A 55 X_..SIOD..CSIO.U 4B70: 70 00 49 4E 54 52 70 00 41 43 54 52 43 53 49 4F p.INTRp.ACTRCSIO 4B80: 0A AA 5B 27 4D 54 58 5F 14 47 0A 5F 43 52 53 00 ..['MTX_.G._CRS. 4B90: 08 43 4D 41 30 11 10 0A 0D 47 01 F8 03 F8 03 08 .CMA0....G...... 4BA0: 08 22 10 00 79 00 8C 43 4D 41 30 0A 02 49 4F 4C ."..y..CMA0..IOL 4BB0: 30 8C 43 4D 41 30 0A 03 49 4F 48 30 8C 43 4D 41 0.CMA0..IOH0.CMA 4BC0: 30 0A 04 49 4F 4C 31 8C 43 4D 41 30 0A 05 49 4F 0..IOL1.CMA0..IO 4BD0: 48 31 8B 43 4D 41 30 0A 09 49 51 52 5F 5B 23 4D H1.CMA0..IQR_[#M 4BE0: 54 58 5F FF FF 53 49 4F 44 0A 04 43 53 49 4F 0A TX_..SIOD..CSIO. 4BF0: 55 70 49 4F 41 4C 49 4F 4C 30 70 49 4F 41 4C 49 UpIOALIOL0pIOALI 4C00: 4F 4C 31 70 49 4F 41 48 49 4F 48 30 70 49 4F 41 OL1pIOAHIOH0pIOA 4C10: 48 49 4F 48 31 79 01 49 4E 54 52 49 51 52 5F 43 HIOH1y.INTRIQR_C 4C20: 53 49 4F 0A AA 5B 27 4D 54 58 5F A4 43 4D 41 30 SIO..['MTX_.CMA0 4C30: 14 48 04 5F 50 52 53 00 08 43 4D 41 31 11 36 0A .H._PRS..CMA1.6. 4C40: 33 30 47 01 F8 03 F8 03 08 08 22 10 00 30 47 01 30G......."..0G. 4C50: E8 03 E8 03 08 08 22 10 00 30 47 01 F8 02 F8 02 ......"..0G..... 4C60: 08 08 22 08 00 30 47 01 E8 02 E8 02 08 08 22 08 .."..0G.......". 4C70: 00 38 79 00 A4 43 4D 41 31 14 44 0B 5F 53 52 53 .8y..CMA1.D._SRS 4C80: 01 8C 68 0A 02 49 4F 4C 4F 8C 68 0A 03 49 4F 48 ..h..IOLO.h..IOH 4C90: 49 8B 68 0A 09 49 51 52 5F 5B 23 4D 54 58 5F FF I.h..IQR_[#MTX_. 4CA0: FF 53 49 4F 44 0A 04 43 53 49 4F 0A 55 70 49 4F .SIOD..CSIO.UpIO 4CB0: 4C 4F 49 4F 41 4C 70 49 4F 48 49 49 4F 41 48 82 LOIOALpIOHIIOAH. 4CC0: 49 51 52 5F 60 74 60 01 49 4E 54 52 A0 27 93 49 IQR_`t`.INTR.'.I 4CD0: 4F 48 49 0A 03 A0 0E 93 49 4F 4C 4F 0A F8 70 00 OHI.....IOLO..p. 4CE0: 4E 53 43 41 A0 0F 93 49 4F 4C 4F 0A E8 70 0A 07 NSCA...IOLO..p.. 4CF0: 4E 53 43 41 A0 27 93 49 4F 48 49 0A 02 A0 0E 93 NSCA.'.IOHI..... 4D00: 49 4F 4C 4F 0A F8 70 01 4E 53 43 41 A0 0F 93 49 IOLO..p.NSCA...I 4D10: 4F 4C 4F 0A E8 70 0A 05 4E 53 43 41 70 01 41 43 OLO..p..NSCAp.AC 4D20: 54 52 43 53 49 4F 0A AA 5B 27 4D 54 58 5F 5B 82 TRCSIO..['MTX_[. 4D30: 46 27 43 4F 4D 42 08 5F 48 49 44 0C 41 D0 05 01 F'COMB._HID.A... 4D40: 08 5F 55 49 44 0A 02 14 42 06 5F 53 54 41 00 08 ._UID...B._STA.. 4D50: 52 45 54 5F 00 5B 23 4D 54 58 5F FF FF 53 49 4F RET_.[#MTX_..SIO 4D60: 44 0A 05 A0 2D 47 43 53 32 43 53 49 4F 0A 55 A0 D...-GCS2CSIO.U. 4D70: 12 41 43 54 52 43 53 49 4F 0A AA 70 0A 0F 52 45 .ACTRCSIO..p..RE 4D80: 54 5F A1 0E 43 53 49 4F 0A AA 70 0A 0D 52 45 54 T_..CSIO..p..RET 4D90: 5F A1 0D 43 53 49 4F 0A AA 70 00 52 45 54 5F 5B _..CSIO..p.RET_[ 4DA0: 27 4D 54 58 5F A4 52 45 54 5F 14 44 04 5F 44 49 'MTX_.RET_.D._DI 4DB0: 53 00 5B 23 4D 54 58 5F FF FF 53 49 4F 44 0A 05 S.[#MTX_..SIOD.. 4DC0: 43 53 49 4F 0A 55 70 00 49 4E 54 52 70 4F 50 54 CSIO.Up.INTRpOPT 4DD0: 31 60 7D 60 0A 20 60 70 60 4F 50 54 31 70 00 41 1`}`. `p`OPT1p.A 4DE0: 43 54 52 43 53 49 4F 0A AA 5B 27 4D 54 58 5F 14 CTRCSIO..['MTX_. 4DF0: 47 0A 5F 43 52 53 00 08 43 4D 42 30 11 10 0A 0D G._CRS..CMB0.... 4E00: 47 01 F8 03 F8 03 08 08 22 10 00 79 00 8C 43 4D G......."..y..CM 4E10: 42 30 0A 02 49 4F 4C 30 8C 43 4D 42 30 0A 03 49 B0..IOL0.CMB0..I 4E20: 4F 48 30 8C 43 4D 42 30 0A 04 49 4F 4C 31 8C 43 OH0.CMB0..IOL1.C 4E30: 4D 42 30 0A 05 49 4F 48 31 8B 43 4D 42 30 0A 09 MB0..IOH1.CMB0.. 4E40: 49 51 52 5F 5B 23 4D 54 58 5F FF FF 53 49 4F 44 IQR_[#MTX_..SIOD 4E50: 0A 05 43 53 49 4F 0A 55 70 49 4F 41 4C 49 4F 4C ..CSIO.UpIOALIOL 4E60: 30 70 49 4F 41 4C 49 4F 4C 31 70 49 4F 41 48 49 0pIOALIOL1pIOAHI 4E70: 4F 48 30 70 49 4F 41 48 49 4F 48 31 79 01 49 4E OH0pIOAHIOH1y.IN 4E80: 54 52 49 51 52 5F 43 53 49 4F 0A AA 5B 27 4D 54 TRIQR_CSIO..['MT 4E90: 58 5F A4 43 4D 42 30 14 48 04 5F 50 52 53 00 08 X_.CMB0.H._PRS.. 4EA0: 43 4D 42 31 11 36 0A 33 30 47 01 F8 02 F8 02 08 CMB1.6.30G...... 4EB0: 08 22 08 00 30 47 01 E8 02 E8 02 08 08 22 08 00 ."..0G.......".. 4EC0: 30 47 01 F8 03 F8 03 08 08 22 10 00 30 47 01 E8 0G......."..0G.. 4ED0: 03 E8 03 08 08 22 10 00 38 79 00 A4 43 4D 42 31 ....."..8y..CMB1 4EE0: 14 45 0C 5F 53 52 53 01 8C 68 0A 02 49 4F 4C 4F .E._SRS..h..IOLO 4EF0: 8C 68 0A 03 49 4F 48 49 8B 68 0A 09 49 51 52 5F .h..IOHI.h..IQR_ 4F00: 5B 23 4D 54 58 5F FF FF 53 49 4F 44 0A 05 43 53 [#MTX_..SIOD..CS 4F10: 49 4F 0A 55 70 49 4F 4C 4F 49 4F 41 4C 70 49 4F IO.UpIOLOIOALpIO 4F20: 48 49 49 4F 41 48 82 49 51 52 5F 60 74 60 01 49 HIIOAH.IQR_`t`.I 4F30: 4E 54 52 A0 27 93 49 4F 48 49 0A 03 A0 0E 93 49 NTR.'.IOHI.....I 4F40: 4F 4C 4F 0A F8 70 00 4E 53 43 42 A0 0F 93 49 4F OLO..p.NSCB...IO 4F50: 4C 4F 0A E8 70 0A 07 4E 53 43 42 A0 27 93 49 4F LO..p..NSCB.'.IO 4F60: 48 49 0A 02 A0 0E 93 49 4F 4C 4F 0A F8 70 01 4E HI.....IOLO..p.N 4F70: 53 43 42 A0 0F 93 49 4F 4C 4F 0A E8 70 0A 05 4E SCB...IOLO..p..N 4F80: 53 43 42 70 01 41 43 54 52 70 4F 50 54 31 60 7B SCBp.ACTRpOPT1`{ 4F90: 60 0A DF 60 70 60 4F 50 54 31 43 53 49 4F 0A AA `..`p`OPT1CSIO.. 4FA0: 5B 27 4D 54 58 5F 10 42 61 2F 03 5F 53 42 5F 50 ['MTX_.Ba/._SB_P 4FB0: 43 49 30 49 53 41 5F 5B 82 48 5A 54 50 4D 5F 08 CI0ISA_[.HZTPM_. 4FC0: 5F 48 49 44 0C 41 D0 0C 31 08 5F 43 49 44 0C 41 _HID.A..1._CID.A 4FD0: D0 0C 31 14 1B 5F 53 54 41 00 A0 07 47 54 50 4D ..1.._STA...GTPM 4FE0: A4 00 A0 08 47 43 54 50 A4 0A 0F A1 03 A4 00 14 ....GCTP........ 4FF0: 33 5F 43 52 53 00 A0 07 47 54 50 4D A4 00 A0 20 3_CRS...GTPM... 5000: 47 43 54 50 A4 11 19 0A 16 86 09 00 01 00 00 D4 GCTP............ 5010: FE 00 50 00 00 47 01 B0 0C B0 0C 08 10 79 00 A1 ..P..G.......y.. 5020: 03 A4 00 14 42 15 5F 44 53 4D 04 A0 43 0F 93 96 ....B._DSM..C... 5030: 68 00 11 13 0A 10 A6 FA DD 3D 1B 36 B4 4E A4 24 h........=.6.N.$ 5040: 8D 10 08 9D 16 53 08 5F 54 5F 30 00 70 99 6A 00 .....S._T_0.p.j. 5050: 5F 54 5F 30 A0 0C 93 5F 54 5F 30 00 A4 11 03 01 _T_0..._T_0..... 5060: 3F A1 4D 0B A0 0D 93 5F 54 5F 30 01 A4 0D 31 2E ?.M...._T_0...1. 5070: 30 00 A1 4C 0A A0 1B 93 5F 54 5F 30 0A 02 70 99 0..L...._T_0..p. 5080: 6B 00 60 7B 60 0A 0F 60 43 4D 57 52 0A 7E 60 A4 k.`{`..`CMWR.~`. 5090: 00 A1 4D 08 A0 1B 93 5F 54 5F 30 0A 03 70 43 4D ..M...._T_0..pCM 50A0: 52 44 0A 7E 60 7B 60 0A 0F 60 A4 50 50 52 32 60 RD.~`{`..`.PPR2` 50B0: A1 4E 06 A0 0B 93 5F 54 5F 30 0A 04 A4 0A 02 A1 .N...._T_0...... 50C0: 4F 05 A0 29 93 5F 54 5F 30 0A 05 70 43 4D 52 44 O..)._T_0..pCMRD 50D0: 0A 7E 60 7A 60 0A 04 60 70 43 4D 52 44 0A 7F 61 .~`z`..`pCMRD..a 50E0: 7B 61 0A 7F 61 A4 50 50 52 33 60 61 A1 32 A0 29 {a..a.PPR3`a.2.) 50F0: 93 5F 54 5F 30 0A 06 08 5F 54 5F 31 0D 00 70 9C ._T_0..._T_1..p. 5100: 6B FF 00 5F 54 5F 31 A0 0C 93 5F 54 5F 31 0D 65 k.._T_1..._T_1.e 5110: 6E 00 A4 00 A1 03 A4 01 A1 06 A4 11 03 01 00 A0 n............... 5120: 41 05 93 96 68 00 11 13 0A 10 ED 54 60 37 13 CC A...h......T`7.. 5130: 75 46 90 1C 47 56 D7 F2 D4 5D 08 5F 54 5F 32 00 uF..GV...]._T_2. 5140: 70 99 6A 00 5F 54 5F 32 A0 0C 93 5F 54 5F 32 00 p.j._T_2..._T_2. 5150: A4 11 03 01 01 A1 1B A0 19 93 5F 54 5F 32 01 70 .........._T_2.p 5160: 53 4D 49 5F 0A BC 7B 99 6B 00 01 00 61 A4 99 61 SMI_..{.k...a..a 5170: 00 A4 11 03 01 00 14 4E 12 50 50 52 32 01 08 5F .......N.PPR2.._ 5180: 54 5F 30 00 70 99 68 00 5F 54 5F 30 A0 0D 93 5F T_0.p.h._T_0..._ 5190: 54 5F 30 00 A4 12 04 02 00 00 A1 44 10 A0 0D 93 T_0........D.... 51A0: 5F 54 5F 30 01 A4 12 04 02 00 01 A1 43 0F A0 0F _T_0........C... 51B0: 93 5F 54 5F 30 0A 02 A4 12 05 02 00 0A 02 A1 40 ._T_0..........@ 51C0: 0E A0 0F 93 5F 54 5F 30 0A 03 A4 12 05 02 00 0A ...._T_0........ 51D0: 03 A1 4D 0C A0 0F 93 5F 54 5F 30 0A 04 A4 12 05 ..M...._T_0..... 51E0: 02 00 0A 04 A1 4A 0B A0 0F 93 5F 54 5F 30 0A 05 .....J...._T_0.. 51F0: A4 12 05 02 00 0A 05 A1 47 0A A0 0F 93 5F 54 5F ........G...._T_ 5200: 30 0A 06 A4 12 05 02 00 0A 06 A1 44 09 A0 0F 93 0..........D.... 5210: 5F 54 5F 30 0A 07 A4 12 05 02 00 0A 07 A1 41 08 _T_0..........A. 5220: A0 0F 93 5F 54 5F 30 0A 08 A4 12 05 02 00 0A 08 ..._T_0......... 5230: A1 4E 06 A0 0F 93 5F 54 5F 30 0A 09 A4 12 05 02 .N...._T_0...... 5240: 00 0A 09 A1 4B 05 A0 0F 93 5F 54 5F 30 0A 0A A4 ....K...._T_0... 5250: 12 05 02 00 0A 0A A1 48 04 A0 0F 93 5F 54 5F 30 .......H...._T_0 5260: 0A 0B A4 12 05 02 00 0A 0B A1 35 A0 0F 93 5F 54 ..........5..._T 5270: 5F 30 0A 0C A4 12 05 02 00 0A 0C A1 23 A0 0F 93 _0..........#... 5280: 5F 54 5F 30 0A 0D A4 12 05 02 00 0A 0D A1 11 A0 _T_0............ 5290: 0F 93 5F 54 5F 30 0A 0E A4 12 05 02 00 0A 0E A4 .._T_0.......... 52A0: 12 04 02 01 00 14 4B 2B 50 50 52 33 02 A0 4C 13 ......K+PPR3..L. 52B0: 93 69 00 08 5F 54 5F 30 00 70 99 68 00 5F 54 5F .i.._T_0.p.h._T_ 52C0: 30 A0 0E 93 5F 54 5F 30 00 A4 12 05 03 00 00 00 0..._T_0........ 52D0: A1 42 11 A0 0E 93 5F 54 5F 30 01 A4 12 05 03 00 .B...._T_0...... 52E0: 01 00 A1 40 10 A0 10 93 5F 54 5F 30 0A 02 A4 12 ...@...._T_0.... 52F0: 06 03 00 0A 02 00 A1 4C 0E A0 10 93 5F 54 5F 30 .......L...._T_0 5300: 0A 03 A4 12 06 03 00 0A 03 00 A1 48 0D A0 10 93 ...........H.... 5310: 5F 54 5F 30 0A 04 A4 12 06 03 00 0A 04 00 A1 44 _T_0...........D 5320: 0C A0 10 93 5F 54 5F 30 0A 05 A4 12 06 03 00 0A ...._T_0........ 5330: 05 00 A1 40 0B A0 10 93 5F 54 5F 30 0A 06 A4 12 ...@...._T_0.... 5340: 06 03 00 0A 06 00 A1 4C 09 A0 10 93 5F 54 5F 30 .......L...._T_0 5350: 0A 07 A4 12 06 03 00 0A 07 00 A1 48 08 A0 10 93 ...........H.... 5360: 5F 54 5F 30 0A 08 A4 12 06 03 00 0A 08 00 A1 44 _T_0...........D 5370: 07 A0 10 93 5F 54 5F 30 0A 09 A4 12 06 03 00 0A ...._T_0........ 5380: 09 00 A1 40 06 A0 10 93 5F 54 5F 30 0A 0A A4 12 ...@...._T_0.... 5390: 06 03 00 0A 0A 00 A1 4C 04 A0 10 93 5F 54 5F 30 .......L...._T_0 53A0: 0A 0B A4 12 06 03 00 0A 0B 00 A1 38 A0 10 93 5F ...........8..._ 53B0: 54 5F 30 0A 0C A4 12 06 03 00 0A 0C 00 A1 25 A0 T_0...........%. 53C0: 10 93 5F 54 5F 30 0A 0D A4 12 06 03 00 0A 0D 00 .._T_0.......... 53D0: A1 12 A0 10 93 5F 54 5F 30 0A 0E A4 12 06 03 00 ....._T_0....... 53E0: 0A 0E 00 A4 12 05 03 01 00 00 A1 46 17 08 5F 54 ...........F.._T 53F0: 5F 31 00 70 99 68 00 5F 54 5F 31 A0 12 93 5F 54 _1.p.h._T_1..._T 5400: 5F 31 00 A4 12 09 03 00 00 0C F0 FF FF FF A1 4B _1.............K 5410: 14 A0 12 93 5F 54 5F 31 01 A4 12 09 03 00 01 0C ...._T_1........ 5420: F0 FF FF FF A1 45 13 A0 14 93 5F 54 5F 31 0A 02 .....E...._T_1.. 5430: A4 12 0A 03 00 0A 02 0C F0 FF FF FF A1 4D 11 A0 .............M.. 5440: 14 93 5F 54 5F 31 0A 03 A4 12 0A 03 00 0A 03 0C .._T_1.......... 5450: F0 FF FF FF A1 45 10 A0 14 93 5F 54 5F 31 0A 04 .....E...._T_1.. 5460: A4 12 0A 03 00 0A 04 0C F0 FF FF FF A1 4D 0E A0 .............M.. 5470: 14 93 5F 54 5F 31 0A 05 A4 12 0A 03 00 0A 05 0C .._T_1.......... 5480: F0 FF FF FF A1 45 0D A0 14 93 5F 54 5F 31 0A 06 .....E...._T_1.. 5490: A4 12 0A 03 00 0A 06 0C F0 FF FF FF A1 4D 0B A0 .............M.. 54A0: 14 93 5F 54 5F 31 0A 07 A4 12 0A 03 00 0A 07 0C .._T_1.......... 54B0: F0 FF FF FF A1 45 0A A0 14 93 5F 54 5F 31 0A 08 .....E...._T_1.. 54C0: A4 12 0A 03 00 0A 08 0C F0 FF FF FF A1 4D 08 A0 .............M.. 54D0: 14 93 5F 54 5F 31 0A 09 A4 12 0A 03 00 0A 09 0C .._T_1.......... 54E0: F0 FF FF FF A1 45 07 A0 14 93 5F 54 5F 31 0A 0A .....E...._T_1.. 54F0: A4 12 0A 03 00 0A 0A 0C F0 FF FF FF A1 4D 05 A0 .............M.. 5500: 14 93 5F 54 5F 31 0A 0B A4 12 0A 03 00 0A 0B 0C .._T_1.......... 5510: F0 FF FF FF A1 45 04 A0 14 93 5F 54 5F 31 0A 0C .....E...._T_1.. 5520: A4 12 0A 03 00 0A 0C 0C F0 FF FF FF A1 2D A0 14 .............-.. 5530: 93 5F 54 5F 31 0A 0D A4 12 0A 03 00 0A 0D 0C F0 ._T_1........... 5540: FF FF FF A1 16 A0 14 93 5F 54 5F 31 0A 0E A4 12 ........_T_1.... 5550: 0A 03 00 0A 0E 0C F0 FF FF FF A4 12 05 03 01 00 ................ 5560: 00 5B 82 46 05 54 43 4D 5F 08 5F 48 49 44 0C 69 .[.F.TCM_._HID.i 5570: 23 01 01 14 1C 5F 53 54 41 00 A0 08 92 47 54 50 #...._STA....GTP 5580: 4D A4 00 A0 08 47 43 54 50 A4 0A 0F A1 03 A4 00 M....GCTP....... 5590: 14 28 5F 43 52 53 00 A0 1D 90 47 54 50 4D 47 43 .(_CRS....GTPMGC 55A0: 54 50 A4 11 11 0A 0E 86 09 00 01 00 00 D4 FE 00 TP.............. 55B0: 50 00 00 79 00 A1 03 A4 00 P..y..... FACP @ 0x0000000000000000 0000: 46 41 43 50 F4 00 00 00 03 23 44 45 4C 4C 20 20 FACP.....#DELL 0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20 B10K .....ASL 0020: 61 00 00 00 00 9C DF DB B5 E9 E9 FF 01 05 09 00 a............... 0030: B2 00 00 00 70 71 00 00 00 08 00 00 00 00 00 00 ....pq.......... 0040: 04 08 00 00 00 00 00 00 50 08 00 00 08 08 00 00 ........P....... 0050: 20 08 00 00 00 00 00 00 04 02 01 04 10 00 00 00 ............... 0060: F4 01 88 13 00 00 00 00 00 00 00 00 00 13 00 00 ................ 0070: A5 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0080: 00 00 00 00 00 9C DF DB 00 00 00 00 B5 E9 E9 FF ................ 0090: 00 00 00 00 01 20 00 01 00 08 00 00 00 00 00 00 ..... .......... 00A0: 00 00 00 00 00 00 00 00 00 00 00 00 01 10 00 01 ................ 00B0: 04 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00C0: 00 00 00 00 01 08 00 01 50 08 00 00 00 00 00 00 ........P....... 00D0: 01 20 00 01 08 08 00 00 00 00 00 00 01 40 00 01 . ...........@.. 00E0: 20 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ............... 00F0: 00 00 00 00 .... FACS @ 0x0000000000000000 0000: 46 41 43 53 40 00 00 00 00 00 00 00 00 00 00 00 FACS@........... 0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ HPET @ 0x0000000000000000 0000: 48 50 45 54 38 00 00 00 01 0E 44 45 4C 4C 20 20 HPET8.....DELL 0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20 B10K .....ASL 0020: 61 00 00 00 01 A3 86 80 00 00 00 00 00 00 D0 FE a............... 0030: 00 00 00 00 00 E8 03 00 ........ MCFG @ 0x0000000000000000 0000: 4D 43 46 47 3C 00 00 00 01 4A 44 45 4C 4C 20 20 MCFG<....JDELL 0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20 B10K .....ASL 0020: 61 00 00 00 00 00 00 00 00 00 00 00 00 00 00 F8 a............... 0030: 00 00 00 00 00 00 00 3F 00 00 00 00 .......?.... SLIC @ 0x0000000000000000 0000: 53 4C 49 43 76 01 00 00 01 85 44 45 4C 4C 20 20 SLICv.....DELL 0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20 B10K .....ASL 0020: 61 00 00 00 00 00 00 00 9C 00 00 00 06 02 00 00 a............... 0030: 00 24 00 00 52 53 41 31 00 04 00 00 01 00 01 00 .$..RSA1........ 0040: 7F F6 C1 05 BE 5C 57 63 A5 8A 68 F3 6E 8F 06 FA .....\Wc..h.n... 0050: AF B4 9F 68 82 23 EC 50 40 5A 73 7F EC E4 07 CB ...h.#.P@Zs..... 0060: DC 25 1A 9C E3 E3 66 11 E0 A5 98 06 C5 80 0A FA .%....f......... 0070: 42 93 86 98 E7 D5 1B D4 D7 3A A4 0B EE E2 7D BE B........:....}. 0080: 5F 5B 15 0C AB D0 21 DE BF E9 B5 6E A4 57 B9 8C _[....!....n.W.. 0090: 0C D2 BA 3A 69 30 76 94 71 A2 64 D7 4C D8 85 BF ...:i0v.q.d.L... 00A0: DF A5 6A C8 DC 45 D5 4D 8C B8 8C 05 2F FC 2E 23 ..j..E.M..../..# 00B0: C4 29 C5 6F 3F 29 6C 6D 57 79 0E B6 75 ED 21 95 .).o?)lmWy..u.!. 00C0: 01 00 00 00 B6 00 00 00 00 00 02 00 44 45 4C 4C ............DELL 00D0: 20 20 42 31 30 4B 20 20 20 00 57 49 4E 44 4F 57 B10K .WINDOW 00E0: 53 20 01 00 02 00 00 00 00 00 00 00 00 00 00 00 S .............. 00F0: 00 00 00 00 00 00 5A 3A A5 CC F1 35 95 0B 73 96 ......Z:...5..s. 0100: 49 3F B2 87 9F 41 2D 3D 56 46 EB 93 67 F2 87 92 I?...A-=VF..g... 0110: F5 C0 6E D7 0B 7C 89 A0 D6 70 63 A2 AF 7C DB 56 ..n..|...pc..|.V 0120: 3A F8 A0 D4 12 C7 8D 27 4F 86 C0 8E A6 DC 72 03 :......'O.....r. 0130: 14 90 03 17 EC 0A A6 E0 5D BE BD 1E 8B 7B E1 0B ........]....{.. 0140: DF 26 44 03 F5 7E B2 0A 07 A2 E9 B7 15 F5 38 8E .&D..~........8. 0150: EA A7 05 CC 84 0F 7C 61 63 ED CF 58 C0 E8 7B DC ......|ac..X..{. 0160: 5C BE 1E 3C 21 9D 4A EC 54 EC 3F 6B 92 00 28 BF \..<!.J.T.?k..(. 0170: E1 F2 32 26 7D E3 ..2&}. TCPA @ 0x0000000000000000 0000: 54 43 50 41 32 00 00 00 01 7F 44 45 4C 4C 20 20 TCPA2.....DELL 0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20 B10K .....ASL 0020: 61 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 a............... 0030: 00 00 .. SRAT @ 0x0000000000000000 0000: 53 52 41 54 70 05 00 00 01 76 44 45 4C 4C 20 20 SRATp....vDELL 0010: 42 31 30 4B 20 20 20 00 15 00 00 00 41 53 4C 20 B10K .....ASL 0020: 61 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 a............... 0030: 00 10 00 20 01 00 00 00 00 00 00 00 00 00 00 00 ... ............ 0040: 00 10 00 22 01 00 00 00 00 00 00 00 00 00 00 00 ..."............ 0050: 00 10 00 32 01 00 00 00 00 00 00 00 00 00 00 00 ...2............ 0060: 00 10 00 34 01 00 00 00 00 00 00 00 00 00 00 00 ...4............ 0070: 00 10 01 00 01 00 00 00 00 00 00 00 00 00 00 00 ................ 0080: 00 10 01 02 01 00 00 00 00 00 00 00 00 00 00 00 ................ 0090: 00 10 01 12 01 00 00 00 00 00 00 00 00 00 00 00 ................ 00A0: 00 10 01 14 01 00 00 00 00 00 00 00 00 00 00 00 ................ 00B0: 00 10 00 21 01 00 00 00 00 00 00 00 00 00 00 00 ...!............ 00C0: 00 10 00 23 01 00 00 00 00 00 00 00 00 00 00 00 ...#............ 00D0: 00 10 00 33 01 00 00 00 00 00 00 00 00 00 00 00 ...3............ 00E0: 00 10 00 35 01 00 00 00 00 00 00 00 00 00 00 00 ...5............ 00F0: 00 10 01 01 01 00 00 00 00 00 00 00 00 00 00 00 ................ 0100: 00 10 01 03 01 00 00 00 00 00 00 00 00 00 00 00 ................ 0110: 00 10 01 13 01 00 00 00 00 00 00 00 00 00 00 00 ................ 0120: 00 10 01 15 01 00 00 00 00 00 00 00 00 00 00 00 ................ 0130: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0140: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0150: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0160: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0170: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0180: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0190: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 01A0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 01B0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 01C0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 01D0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 01E0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 01F0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0200: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0210: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0220: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0230: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0240: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0250: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0260: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0270: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0280: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0290: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 02A0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 02B0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 02C0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 02D0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 02E0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 02F0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0300: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0310: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0320: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0330: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0340: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0350: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0360: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0370: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0380: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0390: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 03A0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 03B0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 03C0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 03D0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 03E0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 03F0: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0400: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0410: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0420: 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0430: 01 28 01 00 00 00 00 00 00 00 00 00 00 00 00 00 .(.............. 0440: 00 00 00 DC 00 00 00 00 00 00 00 00 01 00 00 00 ................ 0450: 00 00 00 00 00 00 00 00 01 28 01 00 00 00 00 00 .........(...... 0460: 00 00 00 00 01 00 00 00 00 00 00 A4 00 00 00 00 ................ 0470: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................ 0480: 01 28 00 00 00 00 00 00 00 00 00 A4 01 00 00 00 .(.............. 0490: 00 00 00 80 01 00 00 00 00 00 00 00 01 00 00 00 ................ 04A0: 00 00 00 00 00 00 00 00 01 28 00 00 00 00 00 00 .........(...... 04B0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 04C0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 04D0: 01 28 00 00 00 00 00 00 00 00 00 00 00 00 00 00 .(.............. 04E0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 04F0: 00 00 00 00 00 00 00 00 01 28 00 00 00 00 00 00 .........(...... 0500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0520: 01 28 00 00 00 00 00 00 00 00 00 00 00 00 00 00 .(.............. 0530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0540: 00 00 00 00 00 00 00 00 01 28 00 00 00 00 00 00 .........(...... 0550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ SSDT @ 0x0000000000000000 0000: 53 53 44 54 9C 00 00 00 01 EF 44 45 4C 4C 00 00 SSDT......DELL.. 0010: 73 74 5F 65 78 00 00 00 00 10 00 00 49 4E 54 4C st_ex.......INTL 0020: 24 06 05 20 14 07 55 53 45 53 00 A3 10 3B 5C 00 $.. ..USES...;\. 0030: 08 5C 5F 53 30 5F 12 06 04 00 00 00 00 08 5C 5F .\_S0_........\_ 0040: 53 33 5F 12 08 04 0A 03 0A 03 00 00 08 5C 5F 53 S3_..........\_S 0050: 34 5F 12 06 04 00 00 00 00 08 5C 5F 53 35 5F 12 4_........\_S5_. 0060: 08 04 0A 02 0A 02 00 00 10 33 5C 2F 04 5F 53 42 .........3\/._SB 0070: 5F 50 43 49 30 49 53 41 5F 4D 4F 55 5F 14 1E 5F _PCI0ISA_MOU_.._ 0080: 50 52 57 00 A0 0D 48 41 43 4B A4 12 06 02 0A 08 PRW...HACK...... 0090: 0A 03 A1 09 A4 12 06 02 0A 08 0A 05 ............ SSDT @ 0x0000000000000000 0000: 53 53 44 54 64 44 00 00 01 A8 49 4E 54 45 4C 00 SSDTdD....INTEL. 0010: 50 50 4D 20 52 43 4D 20 01 00 00 80 49 4E 54 4C PPM RCM ....INTL 0020: 09 11 06 20 10 43 44 5C 2E 5F 50 52 5F 43 50 55 ... .CD\._PR_CPU 0030: 31 08 54 59 50 45 0C 00 00 00 00 08 50 53 45 4E 1.TYPE......PSEN 0040: 0C 01 00 00 00 08 43 53 45 4E 0C 01 00 00 00 08 ......CSEN...... 0050: 4D 57 4F 53 0C 01 00 00 00 08 54 53 45 4E 0C 00 MWOS......TSEN.. 0060: 00 00 00 14 49 07 5F 50 44 43 01 8A 68 0A 08 43 ....I._PDC..h..C 0070: 41 50 41 70 43 41 50 41 54 59 50 45 8A 68 0A 00 APApCAPATYPE.h.. 0080: 52 45 56 53 8A 68 0A 04 53 49 5A 45 70 87 68 60 REVS.h..SIZEp.h` 0090: 70 74 60 0A 08 00 61 5B 13 68 0A 40 77 61 0A 08 pt`...a[.h.@wa.. 00A0: 00 54 45 4D 50 08 53 54 53 30 11 07 0A 04 00 00 .TEMP.STS0...... 00B0: 00 00 73 53 54 53 30 54 45 4D 50 62 5F 4F 53 43 ..sSTS0TEMPb_OSC 00C0: 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70 ......w@.).G...p 00D0: 58 71 39 53 52 45 56 53 53 49 5A 45 62 14 4E 0F Xq9SREVSSIZEb.N. 00E0: 5F 4F 53 43 04 8A 6B 0A 04 43 41 50 41 70 43 41 _OSC..k..CAPApCA 00F0: 50 41 54 59 50 45 8A 6B 0A 00 53 54 53 30 8A 6B PATYPE.k..STS0.k 0100: 0A 04 43 41 50 30 8A 68 0A 00 49 49 44 30 8A 68 ..CAP0.h..IID0.h 0110: 0A 04 49 49 44 31 8A 68 0A 08 49 49 44 32 8A 68 ..IID1.h..IID2.h 0120: 0A 0C 49 49 44 33 08 55 49 44 30 11 13 0A 10 16 ..IID3.UID0..... 0130: A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53 8A .w@.).G...pXq9S. 0140: 68 0A 00 45 49 44 30 8A 68 0A 04 45 49 44 31 8A h..EID0.h..EID1. 0150: 68 0A 08 45 49 44 32 8A 68 0A 0C 45 49 44 33 A0 h..EID2.h..EID3. 0160: 47 03 92 90 90 93 49 49 44 30 45 49 44 30 93 49 G.....IID0EID0.I 0170: 49 44 31 45 49 44 31 90 93 49 49 44 32 45 49 44 ID1EID1..IID2EID 0180: 32 93 49 49 44 33 45 49 44 33 70 0A 06 88 53 54 2.IID3EID3p...ST 0190: 53 30 0A 00 00 A4 6B A0 44 01 92 93 69 0A 01 70 S0....k.D...i..p 01A0: 0A 0A 88 53 54 53 30 0A 00 00 A4 6B A0 48 01 7B ...STS0....k.H.{ 01B0: 53 54 53 30 0A 01 00 7B 43 41 50 30 0B FF 0B 43 STS0...{CAP0...C 01C0: 41 50 30 A4 6B 7B 43 41 50 30 0B FF 0B 43 41 50 AP0.k{CAP0...CAP 01D0: 30 70 43 41 50 30 54 59 50 45 A4 6B 14 4D 11 5F 0pCAP0TYPE.k.M._ 01E0: 50 53 53 00 A0 43 11 50 53 45 4E A4 12 4B 10 08 PSS..C.PSEN..K.. 01F0: 12 20 06 0C 5B 09 00 00 0C 80 38 01 00 0C 0A 00 . ..[.....8..... 0200: 00 00 0C 0A 00 00 00 0C 13 00 00 00 0C 13 00 00 ................ 0210: 00 12 20 06 0C 5A 09 00 00 0C 80 38 01 00 0C 0A .. ..Z.....8.... 0220: 00 00 00 0C 0A 00 00 00 0C 12 00 00 00 0C 12 00 ................ 0230: 00 00 12 20 06 0C D5 08 00 00 0C D0 01 01 00 0C ... ............ 0240: 0A 00 00 00 0C 0A 00 00 00 0C 11 00 00 00 0C 11 ................ 0250: 00 00 00 12 20 06 0C 50 08 00 00 0C D8 D6 00 00 .... ..P........ 0260: 0C 0A 00 00 00 0C 0A 00 00 00 0C 10 00 00 00 0C ................ 0270: 10 00 00 00 12 20 06 0C CB 07 00 00 0C 50 C3 00 ..... .......P.. 0280: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0F 00 00 00 ................ 0290: 0C 0F 00 00 00 12 20 06 0C 46 07 00 00 0C 28 A0 ...... ..F....(. 02A0: 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0E 00 00 ................ 02B0: 00 0C 0E 00 00 00 12 20 06 0C C1 06 00 00 0C D0 ....... ........ 02C0: 84 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0D 00 ................ 02D0: 00 00 0C 0D 00 00 00 12 20 06 0C 3C 06 00 00 0C ........ ..<.... 02E0: 30 75 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C 0u.............. 02F0: 00 00 00 0C 0C 00 00 00 A4 00 14 35 5F 50 43 54 ...........5_PCT 0300: 00 A4 12 4D 02 02 11 14 0A 11 82 0C 00 7F 40 00 ...M..........@. 0310: 00 99 01 00 00 00 00 00 00 79 00 11 14 0A 11 82 .........y...... 0320: 0C 00 7F 10 00 00 98 01 00 00 00 00 00 00 79 00 ..............y. 0330: 14 08 5F 50 50 43 00 A4 00 14 42 03 5F 50 53 44 .._PPC....B._PSD 0340: 00 A0 48 02 7B 54 59 50 45 0C 20 08 00 00 00 A4 ..H.{TYPE. ..... 0350: 12 49 01 01 12 15 05 0A 05 0A 00 0C 00 00 00 00 .I.............. 0360: 0C FE 00 00 00 0C 08 00 00 00 A4 00 14 4B 0F 5F .............K._ 0370: 43 53 54 00 A0 41 0F 43 53 45 4E A0 4D 07 90 4D CST..A.CSEN.M..M 0380: 57 4F 53 7B 54 59 50 45 0B 00 02 00 A4 12 4B 06 WOS{TYPE......K. 0390: 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 ....!........... 03A0: 01 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01 00 .........y...... 03B0: 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00 7F ......!......... 03C0: 01 02 01 10 00 00 00 00 00 00 00 79 00 0A 02 0B ...........y.... 03D0: 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82 0C @.......!....... 03E0: 00 7F 01 02 01 20 00 00 00 00 00 00 00 79 00 0A ..... .......y.. 03F0: 03 0B 60 00 0C 5E 01 00 00 A4 12 4B 06 04 0A 03 ..`..^.....K.... 0400: 12 21 04 11 14 0A 11 82 0C 00 7F 00 00 00 00 00 .!.............. 0410: 00 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03 ......y......... 0420: 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00 ...!............ 0430: 14 08 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C ........y....@.. 0440: F4 01 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08 .....!.......... 0450: 00 00 15 08 00 00 00 00 00 00 79 00 0A 03 0B 60 ..........y....` 0460: 00 0C 5E 01 00 00 A4 00 10 43 44 5C 2E 5F 50 52 ..^......CD\._PR 0470: 5F 43 50 55 32 08 54 59 50 45 0C 00 00 00 00 08 _CPU2.TYPE...... 0480: 50 53 45 4E 0C 01 00 00 00 08 43 53 45 4E 0C 01 PSEN......CSEN.. 0490: 00 00 00 08 4D 57 4F 53 0C 01 00 00 00 08 54 53 ....MWOS......TS 04A0: 45 4E 0C 00 00 00 00 14 49 07 5F 50 44 43 01 8A EN......I._PDC.. 04B0: 68 0A 08 43 41 50 41 70 43 41 50 41 54 59 50 45 h..CAPApCAPATYPE 04C0: 8A 68 0A 00 52 45 56 53 8A 68 0A 04 53 49 5A 45 .h..REVS.h..SIZE 04D0: 70 87 68 60 70 74 60 0A 08 00 61 5B 13 68 0A 40 p.h`pt`...a[.h.@ 04E0: 77 61 0A 08 00 54 45 4D 50 08 53 54 53 30 11 07 wa...TEMP.STS0.. 04F0: 0A 04 00 00 00 00 73 53 54 53 30 54 45 4D 50 62 ......sSTS0TEMPb 0500: 5F 4F 53 43 11 13 0A 10 16 A6 77 40 0C 29 BE 47 _OSC......w@.).G 0510: 9E BD D8 70 58 71 39 53 52 45 56 53 53 49 5A 45 ...pXq9SREVSSIZE 0520: 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A 04 43 41 50 b.N._OSC..k..CAP 0530: 41 70 43 41 50 41 54 59 50 45 8A 6B 0A 00 53 54 ApCAPATYPE.k..ST 0540: 53 30 8A 6B 0A 04 43 41 50 30 8A 68 0A 00 49 49 S0.k..CAP0.h..II 0550: 44 30 8A 68 0A 04 49 49 44 31 8A 68 0A 08 49 49 D0.h..IID1.h..II 0560: 44 32 8A 68 0A 0C 49 49 44 33 08 55 49 44 30 11 D2.h..IID3.UID0. 0570: 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58 .....w@.).G...pX 0580: 71 39 53 8A 68 0A 00 45 49 44 30 8A 68 0A 04 45 q9S.h..EID0.h..E 0590: 49 44 31 8A 68 0A 08 45 49 44 32 8A 68 0A 0C 45 ID1.h..EID2.h..E 05A0: 49 44 33 A0 47 03 92 90 90 93 49 49 44 30 45 49 ID3.G.....IID0EI 05B0: 44 30 93 49 49 44 31 45 49 44 31 90 93 49 49 44 D0.IID1EID1..IID 05C0: 32 45 49 44 32 93 49 49 44 33 45 49 44 33 70 0A 2EID2.IID3EID3p. 05D0: 06 88 53 54 53 30 0A 00 00 A4 6B A0 44 01 92 93 ..STS0....k.D... 05E0: 69 0A 01 70 0A 0A 88 53 54 53 30 0A 00 00 A4 6B i..p...STS0....k 05F0: A0 48 01 7B 53 54 53 30 0A 01 00 7B 43 41 50 30 .H.{STS0...{CAP0 0600: 0B FF 0B 43 41 50 30 A4 6B 7B 43 41 50 30 0B FF ...CAP0.k{CAP0.. 0610: 0B 43 41 50 30 70 43 41 50 30 54 59 50 45 A4 6B .CAP0pCAP0TYPE.k 0620: 14 4D 11 5F 50 53 53 00 A0 43 11 50 53 45 4E A4 .M._PSS..C.PSEN. 0630: 12 4B 10 08 12 20 06 0C 5B 09 00 00 0C 80 38 01 .K... ..[.....8. 0640: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 13 00 00 00 ................ 0650: 0C 13 00 00 00 12 20 06 0C 5A 09 00 00 0C 80 38 ...... ..Z.....8 0660: 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 12 00 00 ................ 0670: 00 0C 12 00 00 00 12 20 06 0C D5 08 00 00 0C D0 ....... ........ 0680: 01 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 11 00 ................ 0690: 00 00 0C 11 00 00 00 12 20 06 0C 50 08 00 00 0C ........ ..P.... 06A0: D8 D6 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 10 ................ 06B0: 00 00 00 0C 10 00 00 00 12 20 06 0C CB 07 00 00 ......... ...... 06C0: 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C .P.............. 06D0: 0F 00 00 00 0C 0F 00 00 00 12 20 06 0C 46 07 00 .......... ..F.. 06E0: 00 0C 28 A0 00 00 0C 0A 00 00 00 0C 0A 00 00 00 ..(............. 06F0: 0C 0E 00 00 00 0C 0E 00 00 00 12 20 06 0C C1 06 ........... .... 0700: 00 00 0C D0 84 00 00 0C 0A 00 00 00 0C 0A 00 00 ................ 0710: 00 0C 0D 00 00 00 0C 0D 00 00 00 12 20 06 0C 3C ............ ..< 0720: 06 00 00 0C 30 75 00 00 0C 0A 00 00 00 0C 0A 00 ....0u.......... 0730: 00 00 0C 0C 00 00 00 0C 0C 00 00 00 A4 00 14 35 ...............5 0740: 5F 50 43 54 00 A4 12 4D 02 02 11 14 0A 11 82 0C _PCT...M........ 0750: 00 7F 40 00 00 99 01 00 00 00 00 00 00 79 00 11 ..@..........y.. 0760: 14 0A 11 82 0C 00 7F 10 00 00 98 01 00 00 00 00 ................ 0770: 00 00 79 00 14 08 5F 50 50 43 00 A4 00 14 42 03 ..y..._PPC....B. 0780: 5F 50 53 44 00 A0 48 02 7B 54 59 50 45 0C 20 08 _PSD..H.{TYPE. . 0790: 00 00 00 A4 12 49 01 01 12 15 05 0A 05 0A 00 0C .....I.......... 07A0: 00 00 00 00 0C FE 00 00 00 0C 08 00 00 00 A4 00 ................ 07B0: 14 4B 0F 5F 43 53 54 00 A0 41 0F 43 53 45 4E A0 .K._CST..A.CSEN. 07C0: 4D 07 90 4D 57 4F 53 7B 54 59 50 45 0B 00 02 00 M..MWOS{TYPE.... 07D0: A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82 0C ..K.....!....... 07E0: 00 7F 01 02 01 00 00 00 00 00 00 00 00 79 00 0A .............y.. 07F0: 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A 11 ..........!..... 0800: 82 0C 00 7F 01 02 01 10 00 00 00 00 00 00 00 79 ...............y 0810: 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11 14 ....@.......!... 0820: 0A 11 82 0C 00 7F 01 02 01 20 00 00 00 00 00 00 ......... ...... 0830: 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 12 4B .y....`..^.....K 0840: 06 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 00 .....!.......... 0850: 00 00 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01 ..........y..... 0860: 00 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00 .......!........ 0870: 01 08 00 00 14 08 00 00 00 00 00 00 79 00 0A 02 ............y... 0880: 0B 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82 .@.......!...... 0890: 0C 00 01 08 00 00 15 08 00 00 00 00 00 00 79 00 ..............y. 08A0: 0A 03 0B 60 00 0C 5E 01 00 00 A4 00 10 43 44 5C ...`..^......CD\ 08B0: 2E 5F 50 52 5F 43 50 55 33 08 54 59 50 45 0C 00 ._PR_CPU3.TYPE.. 08C0: 00 00 00 08 50 53 45 4E 0C 01 00 00 00 08 43 53 ....PSEN......CS 08D0: 45 4E 0C 01 00 00 00 08 4D 57 4F 53 0C 01 00 00 EN......MWOS.... 08E0: 00 08 54 53 45 4E 0C 00 00 00 00 14 49 07 5F 50 ..TSEN......I._P 08F0: 44 43 01 8A 68 0A 08 43 41 50 41 70 43 41 50 41 DC..h..CAPApCAPA 0900: 54 59 50 45 8A 68 0A 00 52 45 56 53 8A 68 0A 04 TYPE.h..REVS.h.. 0910: 53 49 5A 45 70 87 68 60 70 74 60 0A 08 00 61 5B SIZEp.h`pt`...a[ 0920: 13 68 0A 40 77 61 0A 08 00 54 45 4D 50 08 53 54 .h.@wa...TEMP.ST 0930: 53 30 11 07 0A 04 00 00 00 00 73 53 54 53 30 54 S0........sSTS0T 0940: 45 4D 50 62 5F 4F 53 43 11 13 0A 10 16 A6 77 40 EMPb_OSC......w@ 0950: 0C 29 BE 47 9E BD D8 70 58 71 39 53 52 45 56 53 .).G...pXq9SREVS 0960: 53 49 5A 45 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A SIZEb.N._OSC..k. 0970: 04 43 41 50 41 70 43 41 50 41 54 59 50 45 8A 6B .CAPApCAPATYPE.k 0980: 0A 00 53 54 53 30 8A 6B 0A 04 43 41 50 30 8A 68 ..STS0.k..CAP0.h 0990: 0A 00 49 49 44 30 8A 68 0A 04 49 49 44 31 8A 68 ..IID0.h..IID1.h 09A0: 0A 08 49 49 44 32 8A 68 0A 0C 49 49 44 33 08 55 ..IID2.h..IID3.U 09B0: 49 44 30 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E ID0......w@.).G. 09C0: BD D8 70 58 71 39 53 8A 68 0A 00 45 49 44 30 8A ..pXq9S.h..EID0. 09D0: 68 0A 04 45 49 44 31 8A 68 0A 08 45 49 44 32 8A h..EID1.h..EID2. 09E0: 68 0A 0C 45 49 44 33 A0 47 03 92 90 90 93 49 49 h..EID3.G.....II 09F0: 44 30 45 49 44 30 93 49 49 44 31 45 49 44 31 90 D0EID0.IID1EID1. 0A00: 93 49 49 44 32 45 49 44 32 93 49 49 44 33 45 49 .IID2EID2.IID3EI 0A10: 44 33 70 0A 06 88 53 54 53 30 0A 00 00 A4 6B A0 D3p...STS0....k. 0A20: 44 01 92 93 69 0A 01 70 0A 0A 88 53 54 53 30 0A D...i..p...STS0. 0A30: 00 00 A4 6B A0 48 01 7B 53 54 53 30 0A 01 00 7B ...k.H.{STS0...{ 0A40: 43 41 50 30 0B FF 0B 43 41 50 30 A4 6B 7B 43 41 CAP0...CAP0.k{CA 0A50: 50 30 0B FF 0B 43 41 50 30 70 43 41 50 30 54 59 P0...CAP0pCAP0TY 0A60: 50 45 A4 6B 14 4D 11 5F 50 53 53 00 A0 43 11 50 PE.k.M._PSS..C.P 0A70: 53 45 4E A4 12 4B 10 08 12 20 06 0C 5B 09 00 00 SEN..K... ..[... 0A80: 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C ..8............. 0A90: 13 00 00 00 0C 13 00 00 00 12 20 06 0C 5A 09 00 .......... ..Z.. 0AA0: 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00 ...8............ 0AB0: 0C 12 00 00 00 0C 12 00 00 00 12 20 06 0C D5 08 ........... .... 0AC0: 00 00 0C D0 01 01 00 0C 0A 00 00 00 0C 0A 00 00 ................ 0AD0: 00 0C 11 00 00 00 0C 11 00 00 00 12 20 06 0C 50 ............ ..P 0AE0: 08 00 00 0C D8 D6 00 00 0C 0A 00 00 00 0C 0A 00 ................ 0AF0: 00 00 0C 10 00 00 00 0C 10 00 00 00 12 20 06 0C ............. .. 0B00: CB 07 00 00 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A .....P.......... 0B10: 00 00 00 0C 0F 00 00 00 0C 0F 00 00 00 12 20 06 .............. . 0B20: 0C 46 07 00 00 0C 28 A0 00 00 0C 0A 00 00 00 0C .F....(......... 0B30: 0A 00 00 00 0C 0E 00 00 00 0C 0E 00 00 00 12 20 ............... 0B40: 06 0C C1 06 00 00 0C D0 84 00 00 0C 0A 00 00 00 ................ 0B50: 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D 00 00 00 12 ................ 0B60: 20 06 0C 3C 06 00 00 0C 30 75 00 00 0C 0A 00 00 ..<....0u...... 0B70: 00 0C 0A 00 00 00 0C 0C 00 00 00 0C 0C 00 00 00 ................ 0B80: A4 00 14 35 5F 50 43 54 00 A4 12 4D 02 02 11 14 ...5_PCT...M.... 0B90: 0A 11 82 0C 00 7F 40 00 00 99 01 00 00 00 00 00 ......@......... 0BA0: 00 79 00 11 14 0A 11 82 0C 00 7F 10 00 00 98 01 .y.............. 0BB0: 00 00 00 00 00 00 79 00 14 08 5F 50 50 43 00 A4 ......y..._PPC.. 0BC0: 00 14 42 03 5F 50 53 44 00 A0 48 02 7B 54 59 50 ..B._PSD..H.{TYP 0BD0: 45 0C 20 08 00 00 00 A4 12 49 01 01 12 15 05 0A E. ......I...... 0BE0: 05 0A 00 0C 00 00 00 00 0C FE 00 00 00 0C 08 00 ................ 0BF0: 00 00 A4 00 14 4B 0F 5F 43 53 54 00 A0 41 0F 43 .....K._CST..A.C 0C00: 53 45 4E A0 4D 07 90 4D 57 4F 53 7B 54 59 50 45 SEN.M..MWOS{TYPE 0C10: 0B 00 02 00 A4 12 4B 06 04 0A 03 12 21 04 11 14 ......K.....!... 0C20: 0A 11 82 0C 00 7F 01 02 01 00 00 00 00 00 00 00 ................ 0C30: 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21 04 .y............!. 0C40: 11 14 0A 11 82 0C 00 7F 01 02 01 10 00 00 00 00 ................ 0C50: 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00 12 ...y....@....... 0C60: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 20 00 00 !............ .. 0C70: 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00 .....y....`..^.. 0C80: 00 A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82 ...K.....!...... 0C90: 0C 00 7F 00 00 00 00 00 00 00 00 00 00 00 79 00 ..............y. 0CA0: 0A 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A ...........!.... 0CB0: 11 82 0C 00 01 08 00 00 14 08 00 00 00 00 00 00 ................ 0CC0: 79 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11 y....@.......!.. 0CD0: 14 0A 11 82 0C 00 01 08 00 00 15 08 00 00 00 00 ................ 0CE0: 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 00 ..y....`..^..... 0CF0: 10 43 44 5C 2E 5F 50 52 5F 43 50 55 34 08 54 59 .CD\._PR_CPU4.TY 0D00: 50 45 0C 00 00 00 00 08 50 53 45 4E 0C 01 00 00 PE......PSEN.... 0D10: 00 08 43 53 45 4E 0C 01 00 00 00 08 4D 57 4F 53 ..CSEN......MWOS 0D20: 0C 01 00 00 00 08 54 53 45 4E 0C 00 00 00 00 14 ......TSEN...... 0D30: 49 07 5F 50 44 43 01 8A 68 0A 08 43 41 50 41 70 I._PDC..h..CAPAp 0D40: 43 41 50 41 54 59 50 45 8A 68 0A 00 52 45 56 53 CAPATYPE.h..REVS 0D50: 8A 68 0A 04 53 49 5A 45 70 87 68 60 70 74 60 0A .h..SIZEp.h`pt`. 0D60: 08 00 61 5B 13 68 0A 40 77 61 0A 08 00 54 45 4D ..a[.h.@wa...TEM 0D70: 50 08 53 54 53 30 11 07 0A 04 00 00 00 00 73 53 P.STS0........sS 0D80: 54 53 30 54 45 4D 50 62 5F 4F 53 43 11 13 0A 10 TS0TEMPb_OSC.... 0D90: 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53 ..w@.).G...pXq9S 0DA0: 52 45 56 53 53 49 5A 45 62 14 4E 0F 5F 4F 53 43 REVSSIZEb.N._OSC 0DB0: 04 8A 6B 0A 04 43 41 50 41 70 43 41 50 41 54 59 ..k..CAPApCAPATY 0DC0: 50 45 8A 6B 0A 00 53 54 53 30 8A 6B 0A 04 43 41 PE.k..STS0.k..CA 0DD0: 50 30 8A 68 0A 00 49 49 44 30 8A 68 0A 04 49 49 P0.h..IID0.h..II 0DE0: 44 31 8A 68 0A 08 49 49 44 32 8A 68 0A 0C 49 49 D1.h..IID2.h..II 0DF0: 44 33 08 55 49 44 30 11 13 0A 10 16 A6 77 40 0C D3.UID0......w@. 0E00: 29 BE 47 9E BD D8 70 58 71 39 53 8A 68 0A 00 45 ).G...pXq9S.h..E 0E10: 49 44 30 8A 68 0A 04 45 49 44 31 8A 68 0A 08 45 ID0.h..EID1.h..E 0E20: 49 44 32 8A 68 0A 0C 45 49 44 33 A0 47 03 92 90 ID2.h..EID3.G... 0E30: 90 93 49 49 44 30 45 49 44 30 93 49 49 44 31 45 ..IID0EID0.IID1E 0E40: 49 44 31 90 93 49 49 44 32 45 49 44 32 93 49 49 ID1..IID2EID2.II 0E50: 44 33 45 49 44 33 70 0A 06 88 53 54 53 30 0A 00 D3EID3p...STS0.. 0E60: 00 A4 6B A0 44 01 92 93 69 0A 01 70 0A 0A 88 53 ..k.D...i..p...S 0E70: 54 53 30 0A 00 00 A4 6B A0 48 01 7B 53 54 53 30 TS0....k.H.{STS0 0E80: 0A 01 00 7B 43 41 50 30 0B FF 0B 43 41 50 30 A4 ...{CAP0...CAP0. 0E90: 6B 7B 43 41 50 30 0B FF 0B 43 41 50 30 70 43 41 k{CAP0...CAP0pCA 0EA0: 50 30 54 59 50 45 A4 6B 14 4D 11 5F 50 53 53 00 P0TYPE.k.M._PSS. 0EB0: A0 43 11 50 53 45 4E A4 12 4B 10 08 12 20 06 0C .C.PSEN..K... .. 0EC0: 5B 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A [.....8......... 0ED0: 00 00 00 0C 13 00 00 00 0C 13 00 00 00 12 20 06 .............. . 0EE0: 0C 5A 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C .Z.....8........ 0EF0: 0A 00 00 00 0C 12 00 00 00 0C 12 00 00 00 12 20 ............... 0F00: 06 0C D5 08 00 00 0C D0 01 01 00 0C 0A 00 00 00 ................ 0F10: 0C 0A 00 00 00 0C 11 00 00 00 0C 11 00 00 00 12 ................ 0F20: 20 06 0C 50 08 00 00 0C D8 D6 00 00 0C 0A 00 00 ..P............ 0F30: 00 0C 0A 00 00 00 0C 10 00 00 00 0C 10 00 00 00 ................ 0F40: 12 20 06 0C CB 07 00 00 0C 50 C3 00 00 0C 0A 00 . .......P...... 0F50: 00 00 0C 0A 00 00 00 0C 0F 00 00 00 0C 0F 00 00 ................ 0F60: 00 12 20 06 0C 46 07 00 00 0C 28 A0 00 00 0C 0A .. ..F....(..... 0F70: 00 00 00 0C 0A 00 00 00 0C 0E 00 00 00 0C 0E 00 ................ 0F80: 00 00 12 20 06 0C C1 06 00 00 0C D0 84 00 00 0C ... ............ 0F90: 0A 00 00 00 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D ................ 0FA0: 00 00 00 12 20 06 0C 3C 06 00 00 0C 30 75 00 00 .... ..<....0u.. 0FB0: 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C 00 00 00 0C ................ 0FC0: 0C 00 00 00 A4 00 14 35 5F 50 43 54 00 A4 12 4D .......5_PCT...M 0FD0: 02 02 11 14 0A 11 82 0C 00 7F 40 00 00 99 01 00 ..........@..... 0FE0: 00 00 00 00 00 79 00 11 14 0A 11 82 0C 00 7F 10 .....y.......... 0FF0: 00 00 98 01 00 00 00 00 00 00 79 00 14 08 5F 50 ..........y..._P 1000: 50 43 00 A4 00 14 42 03 5F 50 53 44 00 A0 48 02 PC....B._PSD..H. 1010: 7B 54 59 50 45 0C 20 08 00 00 00 A4 12 49 01 01 {TYPE. ......I.. 1020: 12 15 05 0A 05 0A 00 0C 00 00 00 00 0C FE 00 00 ................ 1030: 00 0C 08 00 00 00 A4 00 14 4B 0F 5F 43 53 54 00 .........K._CST. 1040: A0 41 0F 43 53 45 4E A0 4D 07 90 4D 57 4F 53 7B .A.CSEN.M..MWOS{ 1050: 54 59 50 45 0B 00 02 00 A4 12 4B 06 04 0A 03 12 TYPE......K..... 1060: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 00 00 00 !............... 1070: 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03 00 .....y.......... 1080: 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 10 ..!............. 1090: 00 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C F4 .......y....@... 10A0: 01 00 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 ....!........... 10B0: 01 20 00 00 00 00 00 00 00 79 00 0A 03 0B 60 00 . .......y....`. 10C0: 0C 5E 01 00 00 A4 12 4B 06 04 0A 03 12 21 04 11 .^.....K.....!.. 10D0: 14 0A 11 82 0C 00 7F 00 00 00 00 00 00 00 00 00 ................ 10E0: 00 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21 ..y............! 10F0: 04 11 14 0A 11 82 0C 00 01 08 00 00 14 08 00 00 ................ 1100: 00 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00 ....y....@...... 1110: 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00 15 08 .!.............. 1120: 00 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01 ......y....`..^. 1130: 00 00 A4 00 10 43 44 5C 2E 5F 50 52 5F 43 50 55 .....CD\._PR_CPU 1140: 35 08 54 59 50 45 0C 00 00 00 00 08 50 53 45 4E 5.TYPE......PSEN 1150: 0C 01 00 00 00 08 43 53 45 4E 0C 01 00 00 00 08 ......CSEN...... 1160: 4D 57 4F 53 0C 01 00 00 00 08 54 53 45 4E 0C 00 MWOS......TSEN.. 1170: 00 00 00 14 49 07 5F 50 44 43 01 8A 68 0A 08 43 ....I._PDC..h..C 1180: 41 50 41 70 43 41 50 41 54 59 50 45 8A 68 0A 00 APApCAPATYPE.h.. 1190: 52 45 56 53 8A 68 0A 04 53 49 5A 45 70 87 68 60 REVS.h..SIZEp.h` 11A0: 70 74 60 0A 08 00 61 5B 13 68 0A 40 77 61 0A 08 pt`...a[.h.@wa.. 11B0: 00 54 45 4D 50 08 53 54 53 30 11 07 0A 04 00 00 .TEMP.STS0...... 11C0: 00 00 73 53 54 53 30 54 45 4D 50 62 5F 4F 53 43 ..sSTS0TEMPb_OSC 11D0: 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70 ......w@.).G...p 11E0: 58 71 39 53 52 45 56 53 53 49 5A 45 62 14 4E 0F Xq9SREVSSIZEb.N. 11F0: 5F 4F 53 43 04 8A 6B 0A 04 43 41 50 41 70 43 41 _OSC..k..CAPApCA 1200: 50 41 54 59 50 45 8A 6B 0A 00 53 54 53 30 8A 6B PATYPE.k..STS0.k 1210: 0A 04 43 41 50 30 8A 68 0A 00 49 49 44 30 8A 68 ..CAP0.h..IID0.h 1220: 0A 04 49 49 44 31 8A 68 0A 08 49 49 44 32 8A 68 ..IID1.h..IID2.h 1230: 0A 0C 49 49 44 33 08 55 49 44 30 11 13 0A 10 16 ..IID3.UID0..... 1240: A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53 8A .w@.).G...pXq9S. 1250: 68 0A 00 45 49 44 30 8A 68 0A 04 45 49 44 31 8A h..EID0.h..EID1. 1260: 68 0A 08 45 49 44 32 8A 68 0A 0C 45 49 44 33 A0 h..EID2.h..EID3. 1270: 47 03 92 90 90 93 49 49 44 30 45 49 44 30 93 49 G.....IID0EID0.I 1280: 49 44 31 45 49 44 31 90 93 49 49 44 32 45 49 44 ID1EID1..IID2EID 1290: 32 93 49 49 44 33 45 49 44 33 70 0A 06 88 53 54 2.IID3EID3p...ST 12A0: 53 30 0A 00 00 A4 6B A0 44 01 92 93 69 0A 01 70 S0....k.D...i..p 12B0: 0A 0A 88 53 54 53 30 0A 00 00 A4 6B A0 48 01 7B ...STS0....k.H.{ 12C0: 53 54 53 30 0A 01 00 7B 43 41 50 30 0B FF 0B 43 STS0...{CAP0...C 12D0: 41 50 30 A4 6B 7B 43 41 50 30 0B FF 0B 43 41 50 AP0.k{CAP0...CAP 12E0: 30 70 43 41 50 30 54 59 50 45 A4 6B 14 4D 11 5F 0pCAP0TYPE.k.M._ 12F0: 50 53 53 00 A0 43 11 50 53 45 4E A4 12 4B 10 08 PSS..C.PSEN..K.. 1300: 12 20 06 0C 5B 09 00 00 0C 80 38 01 00 0C 0A 00 . ..[.....8..... 1310: 00 00 0C 0A 00 00 00 0C 13 00 00 00 0C 13 00 00 ................ 1320: 00 12 20 06 0C 5A 09 00 00 0C 80 38 01 00 0C 0A .. ..Z.....8.... 1330: 00 00 00 0C 0A 00 00 00 0C 12 00 00 00 0C 12 00 ................ 1340: 00 00 12 20 06 0C D5 08 00 00 0C D0 01 01 00 0C ... ............ 1350: 0A 00 00 00 0C 0A 00 00 00 0C 11 00 00 00 0C 11 ................ 1360: 00 00 00 12 20 06 0C 50 08 00 00 0C D8 D6 00 00 .... ..P........ 1370: 0C 0A 00 00 00 0C 0A 00 00 00 0C 10 00 00 00 0C ................ 1380: 10 00 00 00 12 20 06 0C CB 07 00 00 0C 50 C3 00 ..... .......P.. 1390: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0F 00 00 00 ................ 13A0: 0C 0F 00 00 00 12 20 06 0C 46 07 00 00 0C 28 A0 ...... ..F....(. 13B0: 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0E 00 00 ................ 13C0: 00 0C 0E 00 00 00 12 20 06 0C C1 06 00 00 0C D0 ....... ........ 13D0: 84 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0D 00 ................ 13E0: 00 00 0C 0D 00 00 00 12 20 06 0C 3C 06 00 00 0C ........ ..<.... 13F0: 30 75 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C 0u.............. 1400: 00 00 00 0C 0C 00 00 00 A4 00 14 35 5F 50 43 54 ...........5_PCT 1410: 00 A4 12 4D 02 02 11 14 0A 11 82 0C 00 7F 40 00 ...M..........@. 1420: 00 99 01 00 00 00 00 00 00 79 00 11 14 0A 11 82 .........y...... 1430: 0C 00 7F 10 00 00 98 01 00 00 00 00 00 00 79 00 ..............y. 1440: 14 08 5F 50 50 43 00 A4 00 14 42 03 5F 50 53 44 .._PPC....B._PSD 1450: 00 A0 48 02 7B 54 59 50 45 0C 20 08 00 00 00 A4 ..H.{TYPE. ..... 1460: 12 49 01 01 12 15 05 0A 05 0A 00 0C 01 00 00 00 .I.............. 1470: 0C FE 00 00 00 0C 08 00 00 00 A4 00 14 4B 0F 5F .............K._ 1480: 43 53 54 00 A0 41 0F 43 53 45 4E A0 4D 07 90 4D CST..A.CSEN.M..M 1490: 57 4F 53 7B 54 59 50 45 0B 00 02 00 A4 12 4B 06 WOS{TYPE......K. 14A0: 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 ....!........... 14B0: 01 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01 00 .........y...... 14C0: 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00 7F ......!......... 14D0: 01 02 01 10 00 00 00 00 00 00 00 79 00 0A 02 0B ...........y.... 14E0: 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82 0C @.......!....... 14F0: 00 7F 01 02 01 20 00 00 00 00 00 00 00 79 00 0A ..... .......y.. 1500: 03 0B 60 00 0C 5E 01 00 00 A4 12 4B 06 04 0A 03 ..`..^.....K.... 1510: 12 21 04 11 14 0A 11 82 0C 00 7F 00 00 00 00 00 .!.............. 1520: 00 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03 ......y......... 1530: 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00 ...!............ 1540: 14 08 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C ........y....@.. 1550: F4 01 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08 .....!.......... 1560: 00 00 15 08 00 00 00 00 00 00 79 00 0A 03 0B 60 ..........y....` 1570: 00 0C 5E 01 00 00 A4 00 10 43 44 5C 2E 5F 50 52 ..^......CD\._PR 1580: 5F 43 50 55 36 08 54 59 50 45 0C 00 00 00 00 08 _CPU6.TYPE...... 1590: 50 53 45 4E 0C 01 00 00 00 08 43 53 45 4E 0C 01 PSEN......CSEN.. 15A0: 00 00 00 08 4D 57 4F 53 0C 01 00 00 00 08 54 53 ....MWOS......TS 15B0: 45 4E 0C 00 00 00 00 14 49 07 5F 50 44 43 01 8A EN......I._PDC.. 15C0: 68 0A 08 43 41 50 41 70 43 41 50 41 54 59 50 45 h..CAPApCAPATYPE 15D0: 8A 68 0A 00 52 45 56 53 8A 68 0A 04 53 49 5A 45 .h..REVS.h..SIZE 15E0: 70 87 68 60 70 74 60 0A 08 00 61 5B 13 68 0A 40 p.h`pt`...a[.h.@ 15F0: 77 61 0A 08 00 54 45 4D 50 08 53 54 53 30 11 07 wa...TEMP.STS0.. 1600: 0A 04 00 00 00 00 73 53 54 53 30 54 45 4D 50 62 ......sSTS0TEMPb 1610: 5F 4F 53 43 11 13 0A 10 16 A6 77 40 0C 29 BE 47 _OSC......w@.).G 1620: 9E BD D8 70 58 71 39 53 52 45 56 53 53 49 5A 45 ...pXq9SREVSSIZE 1630: 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A 04 43 41 50 b.N._OSC..k..CAP 1640: 41 70 43 41 50 41 54 59 50 45 8A 6B 0A 00 53 54 ApCAPATYPE.k..ST 1650: 53 30 8A 6B 0A 04 43 41 50 30 8A 68 0A 00 49 49 S0.k..CAP0.h..II 1660: 44 30 8A 68 0A 04 49 49 44 31 8A 68 0A 08 49 49 D0.h..IID1.h..II 1670: 44 32 8A 68 0A 0C 49 49 44 33 08 55 49 44 30 11 D2.h..IID3.UID0. 1680: 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58 .....w@.).G...pX 1690: 71 39 53 8A 68 0A 00 45 49 44 30 8A 68 0A 04 45 q9S.h..EID0.h..E 16A0: 49 44 31 8A 68 0A 08 45 49 44 32 8A 68 0A 0C 45 ID1.h..EID2.h..E 16B0: 49 44 33 A0 47 03 92 90 90 93 49 49 44 30 45 49 ID3.G.....IID0EI 16C0: 44 30 93 49 49 44 31 45 49 44 31 90 93 49 49 44 D0.IID1EID1..IID 16D0: 32 45 49 44 32 93 49 49 44 33 45 49 44 33 70 0A 2EID2.IID3EID3p. 16E0: 06 88 53 54 53 30 0A 00 00 A4 6B A0 44 01 92 93 ..STS0....k.D... 16F0: 69 0A 01 70 0A 0A 88 53 54 53 30 0A 00 00 A4 6B i..p...STS0....k 1700: A0 48 01 7B 53 54 53 30 0A 01 00 7B 43 41 50 30 .H.{STS0...{CAP0 1710: 0B FF 0B 43 41 50 30 A4 6B 7B 43 41 50 30 0B FF ...CAP0.k{CAP0.. 1720: 0B 43 41 50 30 70 43 41 50 30 54 59 50 45 A4 6B .CAP0pCAP0TYPE.k 1730: 14 4D 11 5F 50 53 53 00 A0 43 11 50 53 45 4E A4 .M._PSS..C.PSEN. 1740: 12 4B 10 08 12 20 06 0C 5B 09 00 00 0C 80 38 01 .K... ..[.....8. 1750: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 13 00 00 00 ................ 1760: 0C 13 00 00 00 12 20 06 0C 5A 09 00 00 0C 80 38 ...... ..Z.....8 1770: 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 12 00 00 ................ 1780: 00 0C 12 00 00 00 12 20 06 0C D5 08 00 00 0C D0 ....... ........ 1790: 01 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 11 00 ................ 17A0: 00 00 0C 11 00 00 00 12 20 06 0C 50 08 00 00 0C ........ ..P.... 17B0: D8 D6 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 10 ................ 17C0: 00 00 00 0C 10 00 00 00 12 20 06 0C CB 07 00 00 ......... ...... 17D0: 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C .P.............. 17E0: 0F 00 00 00 0C 0F 00 00 00 12 20 06 0C 46 07 00 .......... ..F.. 17F0: 00 0C 28 A0 00 00 0C 0A 00 00 00 0C 0A 00 00 00 ..(............. 1800: 0C 0E 00 00 00 0C 0E 00 00 00 12 20 06 0C C1 06 ........... .... 1810: 00 00 0C D0 84 00 00 0C 0A 00 00 00 0C 0A 00 00 ................ 1820: 00 0C 0D 00 00 00 0C 0D 00 00 00 12 20 06 0C 3C ............ ..< 1830: 06 00 00 0C 30 75 00 00 0C 0A 00 00 00 0C 0A 00 ....0u.......... 1840: 00 00 0C 0C 00 00 00 0C 0C 00 00 00 A4 00 14 35 ...............5 1850: 5F 50 43 54 00 A4 12 4D 02 02 11 14 0A 11 82 0C _PCT...M........ 1860: 00 7F 40 00 00 99 01 00 00 00 00 00 00 79 00 11 ..@..........y.. 1870: 14 0A 11 82 0C 00 7F 10 00 00 98 01 00 00 00 00 ................ 1880: 00 00 79 00 14 08 5F 50 50 43 00 A4 00 14 42 03 ..y..._PPC....B. 1890: 5F 50 53 44 00 A0 48 02 7B 54 59 50 45 0C 20 08 _PSD..H.{TYPE. . 18A0: 00 00 00 A4 12 49 01 01 12 15 05 0A 05 0A 00 0C .....I.......... 18B0: 01 00 00 00 0C FE 00 00 00 0C 08 00 00 00 A4 00 ................ 18C0: 14 4B 0F 5F 43 53 54 00 A0 41 0F 43 53 45 4E A0 .K._CST..A.CSEN. 18D0: 4D 07 90 4D 57 4F 53 7B 54 59 50 45 0B 00 02 00 M..MWOS{TYPE.... 18E0: A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82 0C ..K.....!....... 18F0: 00 7F 01 02 01 00 00 00 00 00 00 00 00 79 00 0A .............y.. 1900: 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A 11 ..........!..... 1910: 82 0C 00 7F 01 02 01 10 00 00 00 00 00 00 00 79 ...............y 1920: 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11 14 ....@.......!... 1930: 0A 11 82 0C 00 7F 01 02 01 20 00 00 00 00 00 00 ......... ...... 1940: 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 12 4B .y....`..^.....K 1950: 06 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 00 .....!.......... 1960: 00 00 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01 ..........y..... 1970: 00 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00 .......!........ 1980: 01 08 00 00 14 08 00 00 00 00 00 00 79 00 0A 02 ............y... 1990: 0B 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82 .@.......!...... 19A0: 0C 00 01 08 00 00 15 08 00 00 00 00 00 00 79 00 ..............y. 19B0: 0A 03 0B 60 00 0C 5E 01 00 00 A4 00 10 43 44 5C ...`..^......CD\ 19C0: 2E 5F 50 52 5F 43 50 55 37 08 54 59 50 45 0C 00 ._PR_CPU7.TYPE.. 19D0: 00 00 00 08 50 53 45 4E 0C 01 00 00 00 08 43 53 ....PSEN......CS 19E0: 45 4E 0C 01 00 00 00 08 4D 57 4F 53 0C 01 00 00 EN......MWOS.... 19F0: 00 08 54 53 45 4E 0C 00 00 00 00 14 49 07 5F 50 ..TSEN......I._P 1A00: 44 43 01 8A 68 0A 08 43 41 50 41 70 43 41 50 41 DC..h..CAPApCAPA 1A10: 54 59 50 45 8A 68 0A 00 52 45 56 53 8A 68 0A 04 TYPE.h..REVS.h.. 1A20: 53 49 5A 45 70 87 68 60 70 74 60 0A 08 00 61 5B SIZEp.h`pt`...a[ 1A30: 13 68 0A 40 77 61 0A 08 00 54 45 4D 50 08 53 54 .h.@wa...TEMP.ST 1A40: 53 30 11 07 0A 04 00 00 00 00 73 53 54 53 30 54 S0........sSTS0T 1A50: 45 4D 50 62 5F 4F 53 43 11 13 0A 10 16 A6 77 40 EMPb_OSC......w@ 1A60: 0C 29 BE 47 9E BD D8 70 58 71 39 53 52 45 56 53 .).G...pXq9SREVS 1A70: 53 49 5A 45 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A SIZEb.N._OSC..k. 1A80: 04 43 41 50 41 70 43 41 50 41 54 59 50 45 8A 6B .CAPApCAPATYPE.k 1A90: 0A 00 53 54 53 30 8A 6B 0A 04 43 41 50 30 8A 68 ..STS0.k..CAP0.h 1AA0: 0A 00 49 49 44 30 8A 68 0A 04 49 49 44 31 8A 68 ..IID0.h..IID1.h 1AB0: 0A 08 49 49 44 32 8A 68 0A 0C 49 49 44 33 08 55 ..IID2.h..IID3.U 1AC0: 49 44 30 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E ID0......w@.).G. 1AD0: BD D8 70 58 71 39 53 8A 68 0A 00 45 49 44 30 8A ..pXq9S.h..EID0. 1AE0: 68 0A 04 45 49 44 31 8A 68 0A 08 45 49 44 32 8A h..EID1.h..EID2. 1AF0: 68 0A 0C 45 49 44 33 A0 47 03 92 90 90 93 49 49 h..EID3.G.....II 1B00: 44 30 45 49 44 30 93 49 49 44 31 45 49 44 31 90 D0EID0.IID1EID1. 1B10: 93 49 49 44 32 45 49 44 32 93 49 49 44 33 45 49 .IID2EID2.IID3EI 1B20: 44 33 70 0A 06 88 53 54 53 30 0A 00 00 A4 6B A0 D3p...STS0....k. 1B30: 44 01 92 93 69 0A 01 70 0A 0A 88 53 54 53 30 0A D...i..p...STS0. 1B40: 00 00 A4 6B A0 48 01 7B 53 54 53 30 0A 01 00 7B ...k.H.{STS0...{ 1B50: 43 41 50 30 0B FF 0B 43 41 50 30 A4 6B 7B 43 41 CAP0...CAP0.k{CA 1B60: 50 30 0B FF 0B 43 41 50 30 70 43 41 50 30 54 59 P0...CAP0pCAP0TY 1B70: 50 45 A4 6B 14 4D 11 5F 50 53 53 00 A0 43 11 50 PE.k.M._PSS..C.P 1B80: 53 45 4E A4 12 4B 10 08 12 20 06 0C 5B 09 00 00 SEN..K... ..[... 1B90: 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C ..8............. 1BA0: 13 00 00 00 0C 13 00 00 00 12 20 06 0C 5A 09 00 .......... ..Z.. 1BB0: 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00 ...8............ 1BC0: 0C 12 00 00 00 0C 12 00 00 00 12 20 06 0C D5 08 ........... .... 1BD0: 00 00 0C D0 01 01 00 0C 0A 00 00 00 0C 0A 00 00 ................ 1BE0: 00 0C 11 00 00 00 0C 11 00 00 00 12 20 06 0C 50 ............ ..P 1BF0: 08 00 00 0C D8 D6 00 00 0C 0A 00 00 00 0C 0A 00 ................ 1C00: 00 00 0C 10 00 00 00 0C 10 00 00 00 12 20 06 0C ............. .. 1C10: CB 07 00 00 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A .....P.......... 1C20: 00 00 00 0C 0F 00 00 00 0C 0F 00 00 00 12 20 06 .............. . 1C30: 0C 46 07 00 00 0C 28 A0 00 00 0C 0A 00 00 00 0C .F....(......... 1C40: 0A 00 00 00 0C 0E 00 00 00 0C 0E 00 00 00 12 20 ............... 1C50: 06 0C C1 06 00 00 0C D0 84 00 00 0C 0A 00 00 00 ................ 1C60: 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D 00 00 00 12 ................ 1C70: 20 06 0C 3C 06 00 00 0C 30 75 00 00 0C 0A 00 00 ..<....0u...... 1C80: 00 0C 0A 00 00 00 0C 0C 00 00 00 0C 0C 00 00 00 ................ 1C90: A4 00 14 35 5F 50 43 54 00 A4 12 4D 02 02 11 14 ...5_PCT...M.... 1CA0: 0A 11 82 0C 00 7F 40 00 00 99 01 00 00 00 00 00 ......@......... 1CB0: 00 79 00 11 14 0A 11 82 0C 00 7F 10 00 00 98 01 .y.............. 1CC0: 00 00 00 00 00 00 79 00 14 08 5F 50 50 43 00 A4 ......y..._PPC.. 1CD0: 00 14 42 03 5F 50 53 44 00 A0 48 02 7B 54 59 50 ..B._PSD..H.{TYP 1CE0: 45 0C 20 08 00 00 00 A4 12 49 01 01 12 15 05 0A E. ......I...... 1CF0: 05 0A 00 0C 01 00 00 00 0C FE 00 00 00 0C 08 00 ................ 1D00: 00 00 A4 00 14 4B 0F 5F 43 53 54 00 A0 41 0F 43 .....K._CST..A.C 1D10: 53 45 4E A0 4D 07 90 4D 57 4F 53 7B 54 59 50 45 SEN.M..MWOS{TYPE 1D20: 0B 00 02 00 A4 12 4B 06 04 0A 03 12 21 04 11 14 ......K.....!... 1D30: 0A 11 82 0C 00 7F 01 02 01 00 00 00 00 00 00 00 ................ 1D40: 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21 04 .y............!. 1D50: 11 14 0A 11 82 0C 00 7F 01 02 01 10 00 00 00 00 ................ 1D60: 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00 12 ...y....@....... 1D70: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 20 00 00 !............ .. 1D80: 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00 .....y....`..^.. 1D90: 00 A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82 ...K.....!...... 1DA0: 0C 00 7F 00 00 00 00 00 00 00 00 00 00 00 79 00 ..............y. 1DB0: 0A 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A ...........!.... 1DC0: 11 82 0C 00 01 08 00 00 14 08 00 00 00 00 00 00 ................ 1DD0: 79 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11 y....@.......!.. 1DE0: 14 0A 11 82 0C 00 01 08 00 00 15 08 00 00 00 00 ................ 1DF0: 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 00 ..y....`..^..... 1E00: 10 43 44 5C 2E 5F 50 52 5F 43 50 55 38 08 54 59 .CD\._PR_CPU8.TY 1E10: 50 45 0C 00 00 00 00 08 50 53 45 4E 0C 01 00 00 PE......PSEN.... 1E20: 00 08 43 53 45 4E 0C 01 00 00 00 08 4D 57 4F 53 ..CSEN......MWOS 1E30: 0C 01 00 00 00 08 54 53 45 4E 0C 00 00 00 00 14 ......TSEN...... 1E40: 49 07 5F 50 44 43 01 8A 68 0A 08 43 41 50 41 70 I._PDC..h..CAPAp 1E50: 43 41 50 41 54 59 50 45 8A 68 0A 00 52 45 56 53 CAPATYPE.h..REVS 1E60: 8A 68 0A 04 53 49 5A 45 70 87 68 60 70 74 60 0A .h..SIZEp.h`pt`. 1E70: 08 00 61 5B 13 68 0A 40 77 61 0A 08 00 54 45 4D ..a[.h.@wa...TEM 1E80: 50 08 53 54 53 30 11 07 0A 04 00 00 00 00 73 53 P.STS0........sS 1E90: 54 53 30 54 45 4D 50 62 5F 4F 53 43 11 13 0A 10 TS0TEMPb_OSC.... 1EA0: 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53 ..w@.).G...pXq9S 1EB0: 52 45 56 53 53 49 5A 45 62 14 4E 0F 5F 4F 53 43 REVSSIZEb.N._OSC 1EC0: 04 8A 6B 0A 04 43 41 50 41 70 43 41 50 41 54 59 ..k..CAPApCAPATY 1ED0: 50 45 8A 6B 0A 00 53 54 53 30 8A 6B 0A 04 43 41 PE.k..STS0.k..CA 1EE0: 50 30 8A 68 0A 00 49 49 44 30 8A 68 0A 04 49 49 P0.h..IID0.h..II 1EF0: 44 31 8A 68 0A 08 49 49 44 32 8A 68 0A 0C 49 49 D1.h..IID2.h..II 1F00: 44 33 08 55 49 44 30 11 13 0A 10 16 A6 77 40 0C D3.UID0......w@. 1F10: 29 BE 47 9E BD D8 70 58 71 39 53 8A 68 0A 00 45 ).G...pXq9S.h..E 1F20: 49 44 30 8A 68 0A 04 45 49 44 31 8A 68 0A 08 45 ID0.h..EID1.h..E 1F30: 49 44 32 8A 68 0A 0C 45 49 44 33 A0 47 03 92 90 ID2.h..EID3.G... 1F40: 90 93 49 49 44 30 45 49 44 30 93 49 49 44 31 45 ..IID0EID0.IID1E 1F50: 49 44 31 90 93 49 49 44 32 45 49 44 32 93 49 49 ID1..IID2EID2.II 1F60: 44 33 45 49 44 33 70 0A 06 88 53 54 53 30 0A 00 D3EID3p...STS0.. 1F70: 00 A4 6B A0 44 01 92 93 69 0A 01 70 0A 0A 88 53 ..k.D...i..p...S 1F80: 54 53 30 0A 00 00 A4 6B A0 48 01 7B 53 54 53 30 TS0....k.H.{STS0 1F90: 0A 01 00 7B 43 41 50 30 0B FF 0B 43 41 50 30 A4 ...{CAP0...CAP0. 1FA0: 6B 7B 43 41 50 30 0B FF 0B 43 41 50 30 70 43 41 k{CAP0...CAP0pCA 1FB0: 50 30 54 59 50 45 A4 6B 14 4D 11 5F 50 53 53 00 P0TYPE.k.M._PSS. 1FC0: A0 43 11 50 53 45 4E A4 12 4B 10 08 12 20 06 0C .C.PSEN..K... .. 1FD0: 5B 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A [.....8......... 1FE0: 00 00 00 0C 13 00 00 00 0C 13 00 00 00 12 20 06 .............. . 1FF0: 0C 5A 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C .Z.....8........ 2000: 0A 00 00 00 0C 12 00 00 00 0C 12 00 00 00 12 20 ............... 2010: 06 0C D5 08 00 00 0C D0 01 01 00 0C 0A 00 00 00 ................ 2020: 0C 0A 00 00 00 0C 11 00 00 00 0C 11 00 00 00 12 ................ 2030: 20 06 0C 50 08 00 00 0C D8 D6 00 00 0C 0A 00 00 ..P............ 2040: 00 0C 0A 00 00 00 0C 10 00 00 00 0C 10 00 00 00 ................ 2050: 12 20 06 0C CB 07 00 00 0C 50 C3 00 00 0C 0A 00 . .......P...... 2060: 00 00 0C 0A 00 00 00 0C 0F 00 00 00 0C 0F 00 00 ................ 2070: 00 12 20 06 0C 46 07 00 00 0C 28 A0 00 00 0C 0A .. ..F....(..... 2080: 00 00 00 0C 0A 00 00 00 0C 0E 00 00 00 0C 0E 00 ................ 2090: 00 00 12 20 06 0C C1 06 00 00 0C D0 84 00 00 0C ... ............ 20A0: 0A 00 00 00 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D ................ 20B0: 00 00 00 12 20 06 0C 3C 06 00 00 0C 30 75 00 00 .... ..<....0u.. 20C0: 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C 00 00 00 0C ................ 20D0: 0C 00 00 00 A4 00 14 35 5F 50 43 54 00 A4 12 4D .......5_PCT...M 20E0: 02 02 11 14 0A 11 82 0C 00 7F 40 00 00 99 01 00 ..........@..... 20F0: 00 00 00 00 00 79 00 11 14 0A 11 82 0C 00 7F 10 .....y.......... 2100: 00 00 98 01 00 00 00 00 00 00 79 00 14 08 5F 50 ..........y..._P 2110: 50 43 00 A4 00 14 42 03 5F 50 53 44 00 A0 48 02 PC....B._PSD..H. 2120: 7B 54 59 50 45 0C 20 08 00 00 00 A4 12 49 01 01 {TYPE. ......I.. 2130: 12 15 05 0A 05 0A 00 0C 01 00 00 00 0C FE 00 00 ................ 2140: 00 0C 08 00 00 00 A4 00 14 4B 0F 5F 43 53 54 00 .........K._CST. 2150: A0 41 0F 43 53 45 4E A0 4D 07 90 4D 57 4F 53 7B .A.CSEN.M..MWOS{ 2160: 54 59 50 45 0B 00 02 00 A4 12 4B 06 04 0A 03 12 TYPE......K..... 2170: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 00 00 00 !............... 2180: 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03 00 .....y.......... 2190: 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 10 ..!............. 21A0: 00 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C F4 .......y....@... 21B0: 01 00 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 ....!........... 21C0: 01 20 00 00 00 00 00 00 00 79 00 0A 03 0B 60 00 . .......y....`. 21D0: 0C 5E 01 00 00 A4 12 4B 06 04 0A 03 12 21 04 11 .^.....K.....!.. 21E0: 14 0A 11 82 0C 00 7F 00 00 00 00 00 00 00 00 00 ................ 21F0: 00 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21 ..y............! 2200: 04 11 14 0A 11 82 0C 00 01 08 00 00 14 08 00 00 ................ 2210: 00 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00 ....y....@...... 2220: 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00 15 08 .!.............. 2230: 00 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01 ......y....`..^. 2240: 00 00 A4 00 10 43 44 5C 2E 5F 50 52 5F 43 50 55 .....CD\._PR_CPU 2250: 39 08 54 59 50 45 0C 00 00 00 00 08 50 53 45 4E 9.TYPE......PSEN 2260: 0C 01 00 00 00 08 43 53 45 4E 0C 01 00 00 00 08 ......CSEN...... 2270: 4D 57 4F 53 0C 01 00 00 00 08 54 53 45 4E 0C 00 MWOS......TSEN.. 2280: 00 00 00 14 49 07 5F 50 44 43 01 8A 68 0A 08 43 ....I._PDC..h..C 2290: 41 50 41 70 43 41 50 41 54 59 50 45 8A 68 0A 00 APApCAPATYPE.h.. 22A0: 52 45 56 53 8A 68 0A 04 53 49 5A 45 70 87 68 60 REVS.h..SIZEp.h` 22B0: 70 74 60 0A 08 00 61 5B 13 68 0A 40 77 61 0A 08 pt`...a[.h.@wa.. 22C0: 00 54 45 4D 50 08 53 54 53 30 11 07 0A 04 00 00 .TEMP.STS0...... 22D0: 00 00 73 53 54 53 30 54 45 4D 50 62 5F 4F 53 43 ..sSTS0TEMPb_OSC 22E0: 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70 ......w@.).G...p 22F0: 58 71 39 53 52 45 56 53 53 49 5A 45 62 14 4E 0F Xq9SREVSSIZEb.N. 2300: 5F 4F 53 43 04 8A 6B 0A 04 43 41 50 41 70 43 41 _OSC..k..CAPApCA 2310: 50 41 54 59 50 45 8A 6B 0A 00 53 54 53 30 8A 6B PATYPE.k..STS0.k 2320: 0A 04 43 41 50 30 8A 68 0A 00 49 49 44 30 8A 68 ..CAP0.h..IID0.h 2330: 0A 04 49 49 44 31 8A 68 0A 08 49 49 44 32 8A 68 ..IID1.h..IID2.h 2340: 0A 0C 49 49 44 33 08 55 49 44 30 11 13 0A 10 16 ..IID3.UID0..... 2350: A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53 8A .w@.).G...pXq9S. 2360: 68 0A 00 45 49 44 30 8A 68 0A 04 45 49 44 31 8A h..EID0.h..EID1. 2370: 68 0A 08 45 49 44 32 8A 68 0A 0C 45 49 44 33 A0 h..EID2.h..EID3. 2380: 47 03 92 90 90 93 49 49 44 30 45 49 44 30 93 49 G.....IID0EID0.I 2390: 49 44 31 45 49 44 31 90 93 49 49 44 32 45 49 44 ID1EID1..IID2EID 23A0: 32 93 49 49 44 33 45 49 44 33 70 0A 06 88 53 54 2.IID3EID3p...ST 23B0: 53 30 0A 00 00 A4 6B A0 44 01 92 93 69 0A 01 70 S0....k.D...i..p 23C0: 0A 0A 88 53 54 53 30 0A 00 00 A4 6B A0 48 01 7B ...STS0....k.H.{ 23D0: 53 54 53 30 0A 01 00 7B 43 41 50 30 0B FF 0B 43 STS0...{CAP0...C 23E0: 41 50 30 A4 6B 7B 43 41 50 30 0B FF 0B 43 41 50 AP0.k{CAP0...CAP 23F0: 30 70 43 41 50 30 54 59 50 45 A4 6B 14 4D 11 5F 0pCAP0TYPE.k.M._ 2400: 50 53 53 00 A0 43 11 50 53 45 4E A4 12 4B 10 08 PSS..C.PSEN..K.. 2410: 12 20 06 0C 5B 09 00 00 0C 80 38 01 00 0C 0A 00 . ..[.....8..... 2420: 00 00 0C 0A 00 00 00 0C 13 00 00 00 0C 13 00 00 ................ 2430: 00 12 20 06 0C 5A 09 00 00 0C 80 38 01 00 0C 0A .. ..Z.....8.... 2440: 00 00 00 0C 0A 00 00 00 0C 12 00 00 00 0C 12 00 ................ 2450: 00 00 12 20 06 0C D5 08 00 00 0C D0 01 01 00 0C ... ............ 2460: 0A 00 00 00 0C 0A 00 00 00 0C 11 00 00 00 0C 11 ................ 2470: 00 00 00 12 20 06 0C 50 08 00 00 0C D8 D6 00 00 .... ..P........ 2480: 0C 0A 00 00 00 0C 0A 00 00 00 0C 10 00 00 00 0C ................ 2490: 10 00 00 00 12 20 06 0C CB 07 00 00 0C 50 C3 00 ..... .......P.. 24A0: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0F 00 00 00 ................ 24B0: 0C 0F 00 00 00 12 20 06 0C 46 07 00 00 0C 28 A0 ...... ..F....(. 24C0: 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0E 00 00 ................ 24D0: 00 0C 0E 00 00 00 12 20 06 0C C1 06 00 00 0C D0 ....... ........ 24E0: 84 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0D 00 ................ 24F0: 00 00 0C 0D 00 00 00 12 20 06 0C 3C 06 00 00 0C ........ ..<.... 2500: 30 75 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C 0u.............. 2510: 00 00 00 0C 0C 00 00 00 A4 00 14 35 5F 50 43 54 ...........5_PCT 2520: 00 A4 12 4D 02 02 11 14 0A 11 82 0C 00 7F 40 00 ...M..........@. 2530: 00 99 01 00 00 00 00 00 00 79 00 11 14 0A 11 82 .........y...... 2540: 0C 00 7F 10 00 00 98 01 00 00 00 00 00 00 79 00 ..............y. 2550: 14 08 5F 50 50 43 00 A4 00 14 42 03 5F 50 53 44 .._PPC....B._PSD 2560: 00 A0 48 02 7B 54 59 50 45 0C 20 08 00 00 00 A4 ..H.{TYPE. ..... 2570: 12 49 01 01 12 15 05 0A 05 0A 00 0C 00 00 00 00 .I.............. 2580: 0C FE 00 00 00 0C 08 00 00 00 A4 00 14 4B 0F 5F .............K._ 2590: 43 53 54 00 A0 41 0F 43 53 45 4E A0 4D 07 90 4D CST..A.CSEN.M..M 25A0: 57 4F 53 7B 54 59 50 45 0B 00 02 00 A4 12 4B 06 WOS{TYPE......K. 25B0: 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 ....!........... 25C0: 01 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01 00 .........y...... 25D0: 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00 7F ......!......... 25E0: 01 02 01 10 00 00 00 00 00 00 00 79 00 0A 02 0B ...........y.... 25F0: 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82 0C @.......!....... 2600: 00 7F 01 02 01 20 00 00 00 00 00 00 00 79 00 0A ..... .......y.. 2610: 03 0B 60 00 0C 5E 01 00 00 A4 12 4B 06 04 0A 03 ..`..^.....K.... 2620: 12 21 04 11 14 0A 11 82 0C 00 7F 00 00 00 00 00 .!.............. 2630: 00 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03 ......y......... 2640: 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00 ...!............ 2650: 14 08 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C ........y....@.. 2660: F4 01 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08 .....!.......... 2670: 00 00 15 08 00 00 00 00 00 00 79 00 0A 03 0B 60 ..........y....` 2680: 00 0C 5E 01 00 00 A4 00 10 43 44 5C 2E 5F 50 52 ..^......CD\._PR 2690: 5F 43 50 31 30 08 54 59 50 45 0C 00 00 00 00 08 _CP10.TYPE...... 26A0: 50 53 45 4E 0C 01 00 00 00 08 43 53 45 4E 0C 01 PSEN......CSEN.. 26B0: 00 00 00 08 4D 57 4F 53 0C 01 00 00 00 08 54 53 ....MWOS......TS 26C0: 45 4E 0C 00 00 00 00 14 49 07 5F 50 44 43 01 8A EN......I._PDC.. 26D0: 68 0A 08 43 41 50 41 70 43 41 50 41 54 59 50 45 h..CAPApCAPATYPE 26E0: 8A 68 0A 00 52 45 56 53 8A 68 0A 04 53 49 5A 45 .h..REVS.h..SIZE 26F0: 70 87 68 60 70 74 60 0A 08 00 61 5B 13 68 0A 40 p.h`pt`...a[.h.@ 2700: 77 61 0A 08 00 54 45 4D 50 08 53 54 53 30 11 07 wa...TEMP.STS0.. 2710: 0A 04 00 00 00 00 73 53 54 53 30 54 45 4D 50 62 ......sSTS0TEMPb 2720: 5F 4F 53 43 11 13 0A 10 16 A6 77 40 0C 29 BE 47 _OSC......w@.).G 2730: 9E BD D8 70 58 71 39 53 52 45 56 53 53 49 5A 45 ...pXq9SREVSSIZE 2740: 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A 04 43 41 50 b.N._OSC..k..CAP 2750: 41 70 43 41 50 41 54 59 50 45 8A 6B 0A 00 53 54 ApCAPATYPE.k..ST 2760: 53 30 8A 6B 0A 04 43 41 50 30 8A 68 0A 00 49 49 S0.k..CAP0.h..II 2770: 44 30 8A 68 0A 04 49 49 44 31 8A 68 0A 08 49 49 D0.h..IID1.h..II 2780: 44 32 8A 68 0A 0C 49 49 44 33 08 55 49 44 30 11 D2.h..IID3.UID0. 2790: 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58 .....w@.).G...pX 27A0: 71 39 53 8A 68 0A 00 45 49 44 30 8A 68 0A 04 45 q9S.h..EID0.h..E 27B0: 49 44 31 8A 68 0A 08 45 49 44 32 8A 68 0A 0C 45 ID1.h..EID2.h..E 27C0: 49 44 33 A0 47 03 92 90 90 93 49 49 44 30 45 49 ID3.G.....IID0EI 27D0: 44 30 93 49 49 44 31 45 49 44 31 90 93 49 49 44 D0.IID1EID1..IID 27E0: 32 45 49 44 32 93 49 49 44 33 45 49 44 33 70 0A 2EID2.IID3EID3p. 27F0: 06 88 53 54 53 30 0A 00 00 A4 6B A0 44 01 92 93 ..STS0....k.D... 2800: 69 0A 01 70 0A 0A 88 53 54 53 30 0A 00 00 A4 6B i..p...STS0....k 2810: A0 48 01 7B 53 54 53 30 0A 01 00 7B 43 41 50 30 .H.{STS0...{CAP0 2820: 0B FF 0B 43 41 50 30 A4 6B 7B 43 41 50 30 0B FF ...CAP0.k{CAP0.. 2830: 0B 43 41 50 30 70 43 41 50 30 54 59 50 45 A4 6B .CAP0pCAP0TYPE.k 2840: 14 4D 11 5F 50 53 53 00 A0 43 11 50 53 45 4E A4 .M._PSS..C.PSEN. 2850: 12 4B 10 08 12 20 06 0C 5B 09 00 00 0C 80 38 01 .K... ..[.....8. 2860: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 13 00 00 00 ................ 2870: 0C 13 00 00 00 12 20 06 0C 5A 09 00 00 0C 80 38 ...... ..Z.....8 2880: 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 12 00 00 ................ 2890: 00 0C 12 00 00 00 12 20 06 0C D5 08 00 00 0C D0 ....... ........ 28A0: 01 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 11 00 ................ 28B0: 00 00 0C 11 00 00 00 12 20 06 0C 50 08 00 00 0C ........ ..P.... 28C0: D8 D6 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 10 ................ 28D0: 00 00 00 0C 10 00 00 00 12 20 06 0C CB 07 00 00 ......... ...... 28E0: 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C .P.............. 28F0: 0F 00 00 00 0C 0F 00 00 00 12 20 06 0C 46 07 00 .......... ..F.. 2900: 00 0C 28 A0 00 00 0C 0A 00 00 00 0C 0A 00 00 00 ..(............. 2910: 0C 0E 00 00 00 0C 0E 00 00 00 12 20 06 0C C1 06 ........... .... 2920: 00 00 0C D0 84 00 00 0C 0A 00 00 00 0C 0A 00 00 ................ 2930: 00 0C 0D 00 00 00 0C 0D 00 00 00 12 20 06 0C 3C ............ ..< 2940: 06 00 00 0C 30 75 00 00 0C 0A 00 00 00 0C 0A 00 ....0u.......... 2950: 00 00 0C 0C 00 00 00 0C 0C 00 00 00 A4 00 14 35 ...............5 2960: 5F 50 43 54 00 A4 12 4D 02 02 11 14 0A 11 82 0C _PCT...M........ 2970: 00 7F 40 00 00 99 01 00 00 00 00 00 00 79 00 11 ..@..........y.. 2980: 14 0A 11 82 0C 00 7F 10 00 00 98 01 00 00 00 00 ................ 2990: 00 00 79 00 14 08 5F 50 50 43 00 A4 00 14 42 03 ..y..._PPC....B. 29A0: 5F 50 53 44 00 A0 48 02 7B 54 59 50 45 0C 20 08 _PSD..H.{TYPE. . 29B0: 00 00 00 A4 12 49 01 01 12 15 05 0A 05 0A 00 0C .....I.......... 29C0: 00 00 00 00 0C FE 00 00 00 0C 08 00 00 00 A4 00 ................ 29D0: 14 4B 0F 5F 43 53 54 00 A0 41 0F 43 53 45 4E A0 .K._CST..A.CSEN. 29E0: 4D 07 90 4D 57 4F 53 7B 54 59 50 45 0B 00 02 00 M..MWOS{TYPE.... 29F0: A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82 0C ..K.....!....... 2A00: 00 7F 01 02 01 00 00 00 00 00 00 00 00 79 00 0A .............y.. 2A10: 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A 11 ..........!..... 2A20: 82 0C 00 7F 01 02 01 10 00 00 00 00 00 00 00 79 ...............y 2A30: 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11 14 ....@.......!... 2A40: 0A 11 82 0C 00 7F 01 02 01 20 00 00 00 00 00 00 ......... ...... 2A50: 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 12 4B .y....`..^.....K 2A60: 06 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 00 .....!.......... 2A70: 00 00 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01 ..........y..... 2A80: 00 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00 .......!........ 2A90: 01 08 00 00 14 08 00 00 00 00 00 00 79 00 0A 02 ............y... 2AA0: 0B 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82 .@.......!...... 2AB0: 0C 00 01 08 00 00 15 08 00 00 00 00 00 00 79 00 ..............y. 2AC0: 0A 03 0B 60 00 0C 5E 01 00 00 A4 00 10 43 44 5C ...`..^......CD\ 2AD0: 2E 5F 50 52 5F 43 50 31 31 08 54 59 50 45 0C 00 ._PR_CP11.TYPE.. 2AE0: 00 00 00 08 50 53 45 4E 0C 01 00 00 00 08 43 53 ....PSEN......CS 2AF0: 45 4E 0C 01 00 00 00 08 4D 57 4F 53 0C 01 00 00 EN......MWOS.... 2B00: 00 08 54 53 45 4E 0C 00 00 00 00 14 49 07 5F 50 ..TSEN......I._P 2B10: 44 43 01 8A 68 0A 08 43 41 50 41 70 43 41 50 41 DC..h..CAPApCAPA 2B20: 54 59 50 45 8A 68 0A 00 52 45 56 53 8A 68 0A 04 TYPE.h..REVS.h.. 2B30: 53 49 5A 45 70 87 68 60 70 74 60 0A 08 00 61 5B SIZEp.h`pt`...a[ 2B40: 13 68 0A 40 77 61 0A 08 00 54 45 4D 50 08 53 54 .h.@wa...TEMP.ST 2B50: 53 30 11 07 0A 04 00 00 00 00 73 53 54 53 30 54 S0........sSTS0T 2B60: 45 4D 50 62 5F 4F 53 43 11 13 0A 10 16 A6 77 40 EMPb_OSC......w@ 2B70: 0C 29 BE 47 9E BD D8 70 58 71 39 53 52 45 56 53 .).G...pXq9SREVS 2B80: 53 49 5A 45 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A SIZEb.N._OSC..k. 2B90: 04 43 41 50 41 70 43 41 50 41 54 59 50 45 8A 6B .CAPApCAPATYPE.k 2BA0: 0A 00 53 54 53 30 8A 6B 0A 04 43 41 50 30 8A 68 ..STS0.k..CAP0.h 2BB0: 0A 00 49 49 44 30 8A 68 0A 04 49 49 44 31 8A 68 ..IID0.h..IID1.h 2BC0: 0A 08 49 49 44 32 8A 68 0A 0C 49 49 44 33 08 55 ..IID2.h..IID3.U 2BD0: 49 44 30 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E ID0......w@.).G. 2BE0: BD D8 70 58 71 39 53 8A 68 0A 00 45 49 44 30 8A ..pXq9S.h..EID0. 2BF0: 68 0A 04 45 49 44 31 8A 68 0A 08 45 49 44 32 8A h..EID1.h..EID2. 2C00: 68 0A 0C 45 49 44 33 A0 47 03 92 90 90 93 49 49 h..EID3.G.....II 2C10: 44 30 45 49 44 30 93 49 49 44 31 45 49 44 31 90 D0EID0.IID1EID1. 2C20: 93 49 49 44 32 45 49 44 32 93 49 49 44 33 45 49 .IID2EID2.IID3EI 2C30: 44 33 70 0A 06 88 53 54 53 30 0A 00 00 A4 6B A0 D3p...STS0....k. 2C40: 44 01 92 93 69 0A 01 70 0A 0A 88 53 54 53 30 0A D...i..p...STS0. 2C50: 00 00 A4 6B A0 48 01 7B 53 54 53 30 0A 01 00 7B ...k.H.{STS0...{ 2C60: 43 41 50 30 0B FF 0B 43 41 50 30 A4 6B 7B 43 41 CAP0...CAP0.k{CA 2C70: 50 30 0B FF 0B 43 41 50 30 70 43 41 50 30 54 59 P0...CAP0pCAP0TY 2C80: 50 45 A4 6B 14 4D 11 5F 50 53 53 00 A0 43 11 50 PE.k.M._PSS..C.P 2C90: 53 45 4E A4 12 4B 10 08 12 20 06 0C 5B 09 00 00 SEN..K... ..[... 2CA0: 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C ..8............. 2CB0: 13 00 00 00 0C 13 00 00 00 12 20 06 0C 5A 09 00 .......... ..Z.. 2CC0: 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00 ...8............ 2CD0: 0C 12 00 00 00 0C 12 00 00 00 12 20 06 0C D5 08 ........... .... 2CE0: 00 00 0C D0 01 01 00 0C 0A 00 00 00 0C 0A 00 00 ................ 2CF0: 00 0C 11 00 00 00 0C 11 00 00 00 12 20 06 0C 50 ............ ..P 2D00: 08 00 00 0C D8 D6 00 00 0C 0A 00 00 00 0C 0A 00 ................ 2D10: 00 00 0C 10 00 00 00 0C 10 00 00 00 12 20 06 0C ............. .. 2D20: CB 07 00 00 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A .....P.......... 2D30: 00 00 00 0C 0F 00 00 00 0C 0F 00 00 00 12 20 06 .............. . 2D40: 0C 46 07 00 00 0C 28 A0 00 00 0C 0A 00 00 00 0C .F....(......... 2D50: 0A 00 00 00 0C 0E 00 00 00 0C 0E 00 00 00 12 20 ............... 2D60: 06 0C C1 06 00 00 0C D0 84 00 00 0C 0A 00 00 00 ................ 2D70: 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D 00 00 00 12 ................ 2D80: 20 06 0C 3C 06 00 00 0C 30 75 00 00 0C 0A 00 00 ..<....0u...... 2D90: 00 0C 0A 00 00 00 0C 0C 00 00 00 0C 0C 00 00 00 ................ 2DA0: A4 00 14 35 5F 50 43 54 00 A4 12 4D 02 02 11 14 ...5_PCT...M.... 2DB0: 0A 11 82 0C 00 7F 40 00 00 99 01 00 00 00 00 00 ......@......... 2DC0: 00 79 00 11 14 0A 11 82 0C 00 7F 10 00 00 98 01 .y.............. 2DD0: 00 00 00 00 00 00 79 00 14 08 5F 50 50 43 00 A4 ......y..._PPC.. 2DE0: 00 14 42 03 5F 50 53 44 00 A0 48 02 7B 54 59 50 ..B._PSD..H.{TYP 2DF0: 45 0C 20 08 00 00 00 A4 12 49 01 01 12 15 05 0A E. ......I...... 2E00: 05 0A 00 0C 00 00 00 00 0C FE 00 00 00 0C 08 00 ................ 2E10: 00 00 A4 00 14 4B 0F 5F 43 53 54 00 A0 41 0F 43 .....K._CST..A.C 2E20: 53 45 4E A0 4D 07 90 4D 57 4F 53 7B 54 59 50 45 SEN.M..MWOS{TYPE 2E30: 0B 00 02 00 A4 12 4B 06 04 0A 03 12 21 04 11 14 ......K.....!... 2E40: 0A 11 82 0C 00 7F 01 02 01 00 00 00 00 00 00 00 ................ 2E50: 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21 04 .y............!. 2E60: 11 14 0A 11 82 0C 00 7F 01 02 01 10 00 00 00 00 ................ 2E70: 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00 12 ...y....@....... 2E80: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 20 00 00 !............ .. 2E90: 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00 .....y....`..^.. 2EA0: 00 A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82 ...K.....!...... 2EB0: 0C 00 7F 00 00 00 00 00 00 00 00 00 00 00 79 00 ..............y. 2EC0: 0A 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A ...........!.... 2ED0: 11 82 0C 00 01 08 00 00 14 08 00 00 00 00 00 00 ................ 2EE0: 79 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11 y....@.......!.. 2EF0: 14 0A 11 82 0C 00 01 08 00 00 15 08 00 00 00 00 ................ 2F00: 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 00 ..y....`..^..... 2F10: 10 43 44 5C 2E 5F 50 52 5F 43 50 31 32 08 54 59 .CD\._PR_CP12.TY 2F20: 50 45 0C 00 00 00 00 08 50 53 45 4E 0C 01 00 00 PE......PSEN.... 2F30: 00 08 43 53 45 4E 0C 01 00 00 00 08 4D 57 4F 53 ..CSEN......MWOS 2F40: 0C 01 00 00 00 08 54 53 45 4E 0C 00 00 00 00 14 ......TSEN...... 2F50: 49 07 5F 50 44 43 01 8A 68 0A 08 43 41 50 41 70 I._PDC..h..CAPAp 2F60: 43 41 50 41 54 59 50 45 8A 68 0A 00 52 45 56 53 CAPATYPE.h..REVS 2F70: 8A 68 0A 04 53 49 5A 45 70 87 68 60 70 74 60 0A .h..SIZEp.h`pt`. 2F80: 08 00 61 5B 13 68 0A 40 77 61 0A 08 00 54 45 4D ..a[.h.@wa...TEM 2F90: 50 08 53 54 53 30 11 07 0A 04 00 00 00 00 73 53 P.STS0........sS 2FA0: 54 53 30 54 45 4D 50 62 5F 4F 53 43 11 13 0A 10 TS0TEMPb_OSC.... 2FB0: 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53 ..w@.).G...pXq9S 2FC0: 52 45 56 53 53 49 5A 45 62 14 4E 0F 5F 4F 53 43 REVSSIZEb.N._OSC 2FD0: 04 8A 6B 0A 04 43 41 50 41 70 43 41 50 41 54 59 ..k..CAPApCAPATY 2FE0: 50 45 8A 6B 0A 00 53 54 53 30 8A 6B 0A 04 43 41 PE.k..STS0.k..CA 2FF0: 50 30 8A 68 0A 00 49 49 44 30 8A 68 0A 04 49 49 P0.h..IID0.h..II 3000: 44 31 8A 68 0A 08 49 49 44 32 8A 68 0A 0C 49 49 D1.h..IID2.h..II 3010: 44 33 08 55 49 44 30 11 13 0A 10 16 A6 77 40 0C D3.UID0......w@. 3020: 29 BE 47 9E BD D8 70 58 71 39 53 8A 68 0A 00 45 ).G...pXq9S.h..E 3030: 49 44 30 8A 68 0A 04 45 49 44 31 8A 68 0A 08 45 ID0.h..EID1.h..E 3040: 49 44 32 8A 68 0A 0C 45 49 44 33 A0 47 03 92 90 ID2.h..EID3.G... 3050: 90 93 49 49 44 30 45 49 44 30 93 49 49 44 31 45 ..IID0EID0.IID1E 3060: 49 44 31 90 93 49 49 44 32 45 49 44 32 93 49 49 ID1..IID2EID2.II 3070: 44 33 45 49 44 33 70 0A 06 88 53 54 53 30 0A 00 D3EID3p...STS0.. 3080: 00 A4 6B A0 44 01 92 93 69 0A 01 70 0A 0A 88 53 ..k.D...i..p...S 3090: 54 53 30 0A 00 00 A4 6B A0 48 01 7B 53 54 53 30 TS0....k.H.{STS0 30A0: 0A 01 00 7B 43 41 50 30 0B FF 0B 43 41 50 30 A4 ...{CAP0...CAP0. 30B0: 6B 7B 43 41 50 30 0B FF 0B 43 41 50 30 70 43 41 k{CAP0...CAP0pCA 30C0: 50 30 54 59 50 45 A4 6B 14 4D 11 5F 50 53 53 00 P0TYPE.k.M._PSS. 30D0: A0 43 11 50 53 45 4E A4 12 4B 10 08 12 20 06 0C .C.PSEN..K... .. 30E0: 5B 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A [.....8......... 30F0: 00 00 00 0C 13 00 00 00 0C 13 00 00 00 12 20 06 .............. . 3100: 0C 5A 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C .Z.....8........ 3110: 0A 00 00 00 0C 12 00 00 00 0C 12 00 00 00 12 20 ............... 3120: 06 0C D5 08 00 00 0C D0 01 01 00 0C 0A 00 00 00 ................ 3130: 0C 0A 00 00 00 0C 11 00 00 00 0C 11 00 00 00 12 ................ 3140: 20 06 0C 50 08 00 00 0C D8 D6 00 00 0C 0A 00 00 ..P............ 3150: 00 0C 0A 00 00 00 0C 10 00 00 00 0C 10 00 00 00 ................ 3160: 12 20 06 0C CB 07 00 00 0C 50 C3 00 00 0C 0A 00 . .......P...... 3170: 00 00 0C 0A 00 00 00 0C 0F 00 00 00 0C 0F 00 00 ................ 3180: 00 12 20 06 0C 46 07 00 00 0C 28 A0 00 00 0C 0A .. ..F....(..... 3190: 00 00 00 0C 0A 00 00 00 0C 0E 00 00 00 0C 0E 00 ................ 31A0: 00 00 12 20 06 0C C1 06 00 00 0C D0 84 00 00 0C ... ............ 31B0: 0A 00 00 00 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D ................ 31C0: 00 00 00 12 20 06 0C 3C 06 00 00 0C 30 75 00 00 .... ..<....0u.. 31D0: 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C 00 00 00 0C ................ 31E0: 0C 00 00 00 A4 00 14 35 5F 50 43 54 00 A4 12 4D .......5_PCT...M 31F0: 02 02 11 14 0A 11 82 0C 00 7F 40 00 00 99 01 00 ..........@..... 3200: 00 00 00 00 00 79 00 11 14 0A 11 82 0C 00 7F 10 .....y.......... 3210: 00 00 98 01 00 00 00 00 00 00 79 00 14 08 5F 50 ..........y..._P 3220: 50 43 00 A4 00 14 42 03 5F 50 53 44 00 A0 48 02 PC....B._PSD..H. 3230: 7B 54 59 50 45 0C 20 08 00 00 00 A4 12 49 01 01 {TYPE. ......I.. 3240: 12 15 05 0A 05 0A 00 0C 00 00 00 00 0C FE 00 00 ................ 3250: 00 0C 08 00 00 00 A4 00 14 4B 0F 5F 43 53 54 00 .........K._CST. 3260: A0 41 0F 43 53 45 4E A0 4D 07 90 4D 57 4F 53 7B .A.CSEN.M..MWOS{ 3270: 54 59 50 45 0B 00 02 00 A4 12 4B 06 04 0A 03 12 TYPE......K..... 3280: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 00 00 00 !............... 3290: 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03 00 .....y.......... 32A0: 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 10 ..!............. 32B0: 00 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C F4 .......y....@... 32C0: 01 00 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 ....!........... 32D0: 01 20 00 00 00 00 00 00 00 79 00 0A 03 0B 60 00 . .......y....`. 32E0: 0C 5E 01 00 00 A4 12 4B 06 04 0A 03 12 21 04 11 .^.....K.....!.. 32F0: 14 0A 11 82 0C 00 7F 00 00 00 00 00 00 00 00 00 ................ 3300: 00 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21 ..y............! 3310: 04 11 14 0A 11 82 0C 00 01 08 00 00 14 08 00 00 ................ 3320: 00 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00 ....y....@...... 3330: 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00 15 08 .!.............. 3340: 00 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01 ......y....`..^. 3350: 00 00 A4 00 10 43 44 5C 2E 5F 50 52 5F 43 50 31 .....CD\._PR_CP1 3360: 33 08 54 59 50 45 0C 00 00 00 00 08 50 53 45 4E 3.TYPE......PSEN 3370: 0C 01 00 00 00 08 43 53 45 4E 0C 01 00 00 00 08 ......CSEN...... 3380: 4D 57 4F 53 0C 01 00 00 00 08 54 53 45 4E 0C 00 MWOS......TSEN.. 3390: 00 00 00 14 49 07 5F 50 44 43 01 8A 68 0A 08 43 ....I._PDC..h..C 33A0: 41 50 41 70 43 41 50 41 54 59 50 45 8A 68 0A 00 APApCAPATYPE.h.. 33B0: 52 45 56 53 8A 68 0A 04 53 49 5A 45 70 87 68 60 REVS.h..SIZEp.h` 33C0: 70 74 60 0A 08 00 61 5B 13 68 0A 40 77 61 0A 08 pt`...a[.h.@wa.. 33D0: 00 54 45 4D 50 08 53 54 53 30 11 07 0A 04 00 00 .TEMP.STS0...... 33E0: 00 00 73 53 54 53 30 54 45 4D 50 62 5F 4F 53 43 ..sSTS0TEMPb_OSC 33F0: 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70 ......w@.).G...p 3400: 58 71 39 53 52 45 56 53 53 49 5A 45 62 14 4E 0F Xq9SREVSSIZEb.N. 3410: 5F 4F 53 43 04 8A 6B 0A 04 43 41 50 41 70 43 41 _OSC..k..CAPApCA 3420: 50 41 54 59 50 45 8A 6B 0A 00 53 54 53 30 8A 6B PATYPE.k..STS0.k 3430: 0A 04 43 41 50 30 8A 68 0A 00 49 49 44 30 8A 68 ..CAP0.h..IID0.h 3440: 0A 04 49 49 44 31 8A 68 0A 08 49 49 44 32 8A 68 ..IID1.h..IID2.h 3450: 0A 0C 49 49 44 33 08 55 49 44 30 11 13 0A 10 16 ..IID3.UID0..... 3460: A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53 8A .w@.).G...pXq9S. 3470: 68 0A 00 45 49 44 30 8A 68 0A 04 45 49 44 31 8A h..EID0.h..EID1. 3480: 68 0A 08 45 49 44 32 8A 68 0A 0C 45 49 44 33 A0 h..EID2.h..EID3. 3490: 47 03 92 90 90 93 49 49 44 30 45 49 44 30 93 49 G.....IID0EID0.I 34A0: 49 44 31 45 49 44 31 90 93 49 49 44 32 45 49 44 ID1EID1..IID2EID 34B0: 32 93 49 49 44 33 45 49 44 33 70 0A 06 88 53 54 2.IID3EID3p...ST 34C0: 53 30 0A 00 00 A4 6B A0 44 01 92 93 69 0A 01 70 S0....k.D...i..p 34D0: 0A 0A 88 53 54 53 30 0A 00 00 A4 6B A0 48 01 7B ...STS0....k.H.{ 34E0: 53 54 53 30 0A 01 00 7B 43 41 50 30 0B FF 0B 43 STS0...{CAP0...C 34F0: 41 50 30 A4 6B 7B 43 41 50 30 0B FF 0B 43 41 50 AP0.k{CAP0...CAP 3500: 30 70 43 41 50 30 54 59 50 45 A4 6B 14 4D 11 5F 0pCAP0TYPE.k.M._ 3510: 50 53 53 00 A0 43 11 50 53 45 4E A4 12 4B 10 08 PSS..C.PSEN..K.. 3520: 12 20 06 0C 5B 09 00 00 0C 80 38 01 00 0C 0A 00 . ..[.....8..... 3530: 00 00 0C 0A 00 00 00 0C 13 00 00 00 0C 13 00 00 ................ 3540: 00 12 20 06 0C 5A 09 00 00 0C 80 38 01 00 0C 0A .. ..Z.....8.... 3550: 00 00 00 0C 0A 00 00 00 0C 12 00 00 00 0C 12 00 ................ 3560: 00 00 12 20 06 0C D5 08 00 00 0C D0 01 01 00 0C ... ............ 3570: 0A 00 00 00 0C 0A 00 00 00 0C 11 00 00 00 0C 11 ................ 3580: 00 00 00 12 20 06 0C 50 08 00 00 0C D8 D6 00 00 .... ..P........ 3590: 0C 0A 00 00 00 0C 0A 00 00 00 0C 10 00 00 00 0C ................ 35A0: 10 00 00 00 12 20 06 0C CB 07 00 00 0C 50 C3 00 ..... .......P.. 35B0: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0F 00 00 00 ................ 35C0: 0C 0F 00 00 00 12 20 06 0C 46 07 00 00 0C 28 A0 ...... ..F....(. 35D0: 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0E 00 00 ................ 35E0: 00 0C 0E 00 00 00 12 20 06 0C C1 06 00 00 0C D0 ....... ........ 35F0: 84 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0D 00 ................ 3600: 00 00 0C 0D 00 00 00 12 20 06 0C 3C 06 00 00 0C ........ ..<.... 3610: 30 75 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C 0u.............. 3620: 00 00 00 0C 0C 00 00 00 A4 00 14 35 5F 50 43 54 ...........5_PCT 3630: 00 A4 12 4D 02 02 11 14 0A 11 82 0C 00 7F 40 00 ...M..........@. 3640: 00 99 01 00 00 00 00 00 00 79 00 11 14 0A 11 82 .........y...... 3650: 0C 00 7F 10 00 00 98 01 00 00 00 00 00 00 79 00 ..............y. 3660: 14 08 5F 50 50 43 00 A4 00 14 42 03 5F 50 53 44 .._PPC....B._PSD 3670: 00 A0 48 02 7B 54 59 50 45 0C 20 08 00 00 00 A4 ..H.{TYPE. ..... 3680: 12 49 01 01 12 15 05 0A 05 0A 00 0C 01 00 00 00 .I.............. 3690: 0C FE 00 00 00 0C 08 00 00 00 A4 00 14 4B 0F 5F .............K._ 36A0: 43 53 54 00 A0 41 0F 43 53 45 4E A0 4D 07 90 4D CST..A.CSEN.M..M 36B0: 57 4F 53 7B 54 59 50 45 0B 00 02 00 A4 12 4B 06 WOS{TYPE......K. 36C0: 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 ....!........... 36D0: 01 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01 00 .........y...... 36E0: 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00 7F ......!......... 36F0: 01 02 01 10 00 00 00 00 00 00 00 79 00 0A 02 0B ...........y.... 3700: 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82 0C @.......!....... 3710: 00 7F 01 02 01 20 00 00 00 00 00 00 00 79 00 0A ..... .......y.. 3720: 03 0B 60 00 0C 5E 01 00 00 A4 12 4B 06 04 0A 03 ..`..^.....K.... 3730: 12 21 04 11 14 0A 11 82 0C 00 7F 00 00 00 00 00 .!.............. 3740: 00 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03 ......y......... 3750: 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00 ...!............ 3760: 14 08 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C ........y....@.. 3770: F4 01 00 00 12 21 04 11 14 0A 11 82 0C 00 01 08 .....!.......... 3780: 00 00 15 08 00 00 00 00 00 00 79 00 0A 03 0B 60 ..........y....` 3790: 00 0C 5E 01 00 00 A4 00 10 43 44 5C 2E 5F 50 52 ..^......CD\._PR 37A0: 5F 43 50 31 34 08 54 59 50 45 0C 00 00 00 00 08 _CP14.TYPE...... 37B0: 50 53 45 4E 0C 01 00 00 00 08 43 53 45 4E 0C 01 PSEN......CSEN.. 37C0: 00 00 00 08 4D 57 4F 53 0C 01 00 00 00 08 54 53 ....MWOS......TS 37D0: 45 4E 0C 00 00 00 00 14 49 07 5F 50 44 43 01 8A EN......I._PDC.. 37E0: 68 0A 08 43 41 50 41 70 43 41 50 41 54 59 50 45 h..CAPApCAPATYPE 37F0: 8A 68 0A 00 52 45 56 53 8A 68 0A 04 53 49 5A 45 .h..REVS.h..SIZE 3800: 70 87 68 60 70 74 60 0A 08 00 61 5B 13 68 0A 40 p.h`pt`...a[.h.@ 3810: 77 61 0A 08 00 54 45 4D 50 08 53 54 53 30 11 07 wa...TEMP.STS0.. 3820: 0A 04 00 00 00 00 73 53 54 53 30 54 45 4D 50 62 ......sSTS0TEMPb 3830: 5F 4F 53 43 11 13 0A 10 16 A6 77 40 0C 29 BE 47 _OSC......w@.).G 3840: 9E BD D8 70 58 71 39 53 52 45 56 53 53 49 5A 45 ...pXq9SREVSSIZE 3850: 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A 04 43 41 50 b.N._OSC..k..CAP 3860: 41 70 43 41 50 41 54 59 50 45 8A 6B 0A 00 53 54 ApCAPATYPE.k..ST 3870: 53 30 8A 6B 0A 04 43 41 50 30 8A 68 0A 00 49 49 S0.k..CAP0.h..II 3880: 44 30 8A 68 0A 04 49 49 44 31 8A 68 0A 08 49 49 D0.h..IID1.h..II 3890: 44 32 8A 68 0A 0C 49 49 44 33 08 55 49 44 30 11 D2.h..IID3.UID0. 38A0: 13 0A 10 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58 .....w@.).G...pX 38B0: 71 39 53 8A 68 0A 00 45 49 44 30 8A 68 0A 04 45 q9S.h..EID0.h..E 38C0: 49 44 31 8A 68 0A 08 45 49 44 32 8A 68 0A 0C 45 ID1.h..EID2.h..E 38D0: 49 44 33 A0 47 03 92 90 90 93 49 49 44 30 45 49 ID3.G.....IID0EI 38E0: 44 30 93 49 49 44 31 45 49 44 31 90 93 49 49 44 D0.IID1EID1..IID 38F0: 32 45 49 44 32 93 49 49 44 33 45 49 44 33 70 0A 2EID2.IID3EID3p. 3900: 06 88 53 54 53 30 0A 00 00 A4 6B A0 44 01 92 93 ..STS0....k.D... 3910: 69 0A 01 70 0A 0A 88 53 54 53 30 0A 00 00 A4 6B i..p...STS0....k 3920: A0 48 01 7B 53 54 53 30 0A 01 00 7B 43 41 50 30 .H.{STS0...{CAP0 3930: 0B FF 0B 43 41 50 30 A4 6B 7B 43 41 50 30 0B FF ...CAP0.k{CAP0.. 3940: 0B 43 41 50 30 70 43 41 50 30 54 59 50 45 A4 6B .CAP0pCAP0TYPE.k 3950: 14 4D 11 5F 50 53 53 00 A0 43 11 50 53 45 4E A4 .M._PSS..C.PSEN. 3960: 12 4B 10 08 12 20 06 0C 5B 09 00 00 0C 80 38 01 .K... ..[.....8. 3970: 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 13 00 00 00 ................ 3980: 0C 13 00 00 00 12 20 06 0C 5A 09 00 00 0C 80 38 ...... ..Z.....8 3990: 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 12 00 00 ................ 39A0: 00 0C 12 00 00 00 12 20 06 0C D5 08 00 00 0C D0 ....... ........ 39B0: 01 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 11 00 ................ 39C0: 00 00 0C 11 00 00 00 12 20 06 0C 50 08 00 00 0C ........ ..P.... 39D0: D8 D6 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C 10 ................ 39E0: 00 00 00 0C 10 00 00 00 12 20 06 0C CB 07 00 00 ......... ...... 39F0: 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A 00 00 00 0C .P.............. 3A00: 0F 00 00 00 0C 0F 00 00 00 12 20 06 0C 46 07 00 .......... ..F.. 3A10: 00 0C 28 A0 00 00 0C 0A 00 00 00 0C 0A 00 00 00 ..(............. 3A20: 0C 0E 00 00 00 0C 0E 00 00 00 12 20 06 0C C1 06 ........... .... 3A30: 00 00 0C D0 84 00 00 0C 0A 00 00 00 0C 0A 00 00 ................ 3A40: 00 0C 0D 00 00 00 0C 0D 00 00 00 12 20 06 0C 3C ............ ..< 3A50: 06 00 00 0C 30 75 00 00 0C 0A 00 00 00 0C 0A 00 ....0u.......... 3A60: 00 00 0C 0C 00 00 00 0C 0C 00 00 00 A4 00 14 35 ...............5 3A70: 5F 50 43 54 00 A4 12 4D 02 02 11 14 0A 11 82 0C _PCT...M........ 3A80: 00 7F 40 00 00 99 01 00 00 00 00 00 00 79 00 11 ..@..........y.. 3A90: 14 0A 11 82 0C 00 7F 10 00 00 98 01 00 00 00 00 ................ 3AA0: 00 00 79 00 14 08 5F 50 50 43 00 A4 00 14 42 03 ..y..._PPC....B. 3AB0: 5F 50 53 44 00 A0 48 02 7B 54 59 50 45 0C 20 08 _PSD..H.{TYPE. . 3AC0: 00 00 00 A4 12 49 01 01 12 15 05 0A 05 0A 00 0C .....I.......... 3AD0: 01 00 00 00 0C FE 00 00 00 0C 08 00 00 00 A4 00 ................ 3AE0: 14 4B 0F 5F 43 53 54 00 A0 41 0F 43 53 45 4E A0 .K._CST..A.CSEN. 3AF0: 4D 07 90 4D 57 4F 53 7B 54 59 50 45 0B 00 02 00 M..MWOS{TYPE.... 3B00: A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82 0C ..K.....!....... 3B10: 00 7F 01 02 01 00 00 00 00 00 00 00 00 79 00 0A .............y.. 3B20: 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A 11 ..........!..... 3B30: 82 0C 00 7F 01 02 01 10 00 00 00 00 00 00 00 79 ...............y 3B40: 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11 14 ....@.......!... 3B50: 0A 11 82 0C 00 7F 01 02 01 20 00 00 00 00 00 00 ......... ...... 3B60: 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 12 4B .y....`..^.....K 3B70: 06 04 0A 03 12 21 04 11 14 0A 11 82 0C 00 7F 00 .....!.......... 3B80: 00 00 00 00 00 00 00 00 00 00 79 00 0A 01 0B 01 ..........y..... 3B90: 00 0C E8 03 00 00 12 21 04 11 14 0A 11 82 0C 00 .......!........ 3BA0: 01 08 00 00 14 08 00 00 00 00 00 00 79 00 0A 02 ............y... 3BB0: 0B 40 00 0C F4 01 00 00 12 21 04 11 14 0A 11 82 .@.......!...... 3BC0: 0C 00 01 08 00 00 15 08 00 00 00 00 00 00 79 00 ..............y. 3BD0: 0A 03 0B 60 00 0C 5E 01 00 00 A4 00 10 43 44 5C ...`..^......CD\ 3BE0: 2E 5F 50 52 5F 43 50 31 35 08 54 59 50 45 0C 00 ._PR_CP15.TYPE.. 3BF0: 00 00 00 08 50 53 45 4E 0C 01 00 00 00 08 43 53 ....PSEN......CS 3C00: 45 4E 0C 01 00 00 00 08 4D 57 4F 53 0C 01 00 00 EN......MWOS.... 3C10: 00 08 54 53 45 4E 0C 00 00 00 00 14 49 07 5F 50 ..TSEN......I._P 3C20: 44 43 01 8A 68 0A 08 43 41 50 41 70 43 41 50 41 DC..h..CAPApCAPA 3C30: 54 59 50 45 8A 68 0A 00 52 45 56 53 8A 68 0A 04 TYPE.h..REVS.h.. 3C40: 53 49 5A 45 70 87 68 60 70 74 60 0A 08 00 61 5B SIZEp.h`pt`...a[ 3C50: 13 68 0A 40 77 61 0A 08 00 54 45 4D 50 08 53 54 .h.@wa...TEMP.ST 3C60: 53 30 11 07 0A 04 00 00 00 00 73 53 54 53 30 54 S0........sSTS0T 3C70: 45 4D 50 62 5F 4F 53 43 11 13 0A 10 16 A6 77 40 EMPb_OSC......w@ 3C80: 0C 29 BE 47 9E BD D8 70 58 71 39 53 52 45 56 53 .).G...pXq9SREVS 3C90: 53 49 5A 45 62 14 4E 0F 5F 4F 53 43 04 8A 6B 0A SIZEb.N._OSC..k. 3CA0: 04 43 41 50 41 70 43 41 50 41 54 59 50 45 8A 6B .CAPApCAPATYPE.k 3CB0: 0A 00 53 54 53 30 8A 6B 0A 04 43 41 50 30 8A 68 ..STS0.k..CAP0.h 3CC0: 0A 00 49 49 44 30 8A 68 0A 04 49 49 44 31 8A 68 ..IID0.h..IID1.h 3CD0: 0A 08 49 49 44 32 8A 68 0A 0C 49 49 44 33 08 55 ..IID2.h..IID3.U 3CE0: 49 44 30 11 13 0A 10 16 A6 77 40 0C 29 BE 47 9E ID0......w@.).G. 3CF0: BD D8 70 58 71 39 53 8A 68 0A 00 45 49 44 30 8A ..pXq9S.h..EID0. 3D00: 68 0A 04 45 49 44 31 8A 68 0A 08 45 49 44 32 8A h..EID1.h..EID2. 3D10: 68 0A 0C 45 49 44 33 A0 47 03 92 90 90 93 49 49 h..EID3.G.....II 3D20: 44 30 45 49 44 30 93 49 49 44 31 45 49 44 31 90 D0EID0.IID1EID1. 3D30: 93 49 49 44 32 45 49 44 32 93 49 49 44 33 45 49 .IID2EID2.IID3EI 3D40: 44 33 70 0A 06 88 53 54 53 30 0A 00 00 A4 6B A0 D3p...STS0....k. 3D50: 44 01 92 93 69 0A 01 70 0A 0A 88 53 54 53 30 0A D...i..p...STS0. 3D60: 00 00 A4 6B A0 48 01 7B 53 54 53 30 0A 01 00 7B ...k.H.{STS0...{ 3D70: 43 41 50 30 0B FF 0B 43 41 50 30 A4 6B 7B 43 41 CAP0...CAP0.k{CA 3D80: 50 30 0B FF 0B 43 41 50 30 70 43 41 50 30 54 59 P0...CAP0pCAP0TY 3D90: 50 45 A4 6B 14 4D 11 5F 50 53 53 00 A0 43 11 50 PE.k.M._PSS..C.P 3DA0: 53 45 4E A4 12 4B 10 08 12 20 06 0C 5B 09 00 00 SEN..K... ..[... 3DB0: 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00 0C ..8............. 3DC0: 13 00 00 00 0C 13 00 00 00 12 20 06 0C 5A 09 00 .......... ..Z.. 3DD0: 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A 00 00 00 ...8............ 3DE0: 0C 12 00 00 00 0C 12 00 00 00 12 20 06 0C D5 08 ........... .... 3DF0: 00 00 0C D0 01 01 00 0C 0A 00 00 00 0C 0A 00 00 ................ 3E00: 00 0C 11 00 00 00 0C 11 00 00 00 12 20 06 0C 50 ............ ..P 3E10: 08 00 00 0C D8 D6 00 00 0C 0A 00 00 00 0C 0A 00 ................ 3E20: 00 00 0C 10 00 00 00 0C 10 00 00 00 12 20 06 0C ............. .. 3E30: CB 07 00 00 0C 50 C3 00 00 0C 0A 00 00 00 0C 0A .....P.......... 3E40: 00 00 00 0C 0F 00 00 00 0C 0F 00 00 00 12 20 06 .............. . 3E50: 0C 46 07 00 00 0C 28 A0 00 00 0C 0A 00 00 00 0C .F....(......... 3E60: 0A 00 00 00 0C 0E 00 00 00 0C 0E 00 00 00 12 20 ............... 3E70: 06 0C C1 06 00 00 0C D0 84 00 00 0C 0A 00 00 00 ................ 3E80: 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D 00 00 00 12 ................ 3E90: 20 06 0C 3C 06 00 00 0C 30 75 00 00 0C 0A 00 00 ..<....0u...... 3EA0: 00 0C 0A 00 00 00 0C 0C 00 00 00 0C 0C 00 00 00 ................ 3EB0: A4 00 14 35 5F 50 43 54 00 A4 12 4D 02 02 11 14 ...5_PCT...M.... 3EC0: 0A 11 82 0C 00 7F 40 00 00 99 01 00 00 00 00 00 ......@......... 3ED0: 00 79 00 11 14 0A 11 82 0C 00 7F 10 00 00 98 01 .y.............. 3EE0: 00 00 00 00 00 00 79 00 14 08 5F 50 50 43 00 A4 ......y..._PPC.. 3EF0: 00 14 42 03 5F 50 53 44 00 A0 48 02 7B 54 59 50 ..B._PSD..H.{TYP 3F00: 45 0C 20 08 00 00 00 A4 12 49 01 01 12 15 05 0A E. ......I...... 3F10: 05 0A 00 0C 01 00 00 00 0C FE 00 00 00 0C 08 00 ................ 3F20: 00 00 A4 00 14 4B 0F 5F 43 53 54 00 A0 41 0F 43 .....K._CST..A.C 3F30: 53 45 4E A0 4D 07 90 4D 57 4F 53 7B 54 59 50 45 SEN.M..MWOS{TYPE 3F40: 0B 00 02 00 A4 12 4B 06 04 0A 03 12 21 04 11 14 ......K.....!... 3F50: 0A 11 82 0C 00 7F 01 02 01 00 00 00 00 00 00 00 ................ 3F60: 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21 04 .y............!. 3F70: 11 14 0A 11 82 0C 00 7F 01 02 01 10 00 00 00 00 ................ 3F80: 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00 12 ...y....@....... 3F90: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 20 00 00 !............ .. 3FA0: 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00 .....y....`..^.. 3FB0: 00 A4 12 4B 06 04 0A 03 12 21 04 11 14 0A 11 82 ...K.....!...... 3FC0: 0C 00 7F 00 00 00 00 00 00 00 00 00 00 00 79 00 ..............y. 3FD0: 0A 01 0B 01 00 0C E8 03 00 00 12 21 04 11 14 0A ...........!.... 3FE0: 11 82 0C 00 01 08 00 00 14 08 00 00 00 00 00 00 ................ 3FF0: 79 00 0A 02 0B 40 00 0C F4 01 00 00 12 21 04 11 y....@.......!.. 4000: 14 0A 11 82 0C 00 01 08 00 00 15 08 00 00 00 00 ................ 4010: 00 00 79 00 0A 03 0B 60 00 0C 5E 01 00 00 A4 00 ..y....`..^..... 4020: 10 43 44 5C 2E 5F 50 52 5F 43 50 31 36 08 54 59 .CD\._PR_CP16.TY 4030: 50 45 0C 00 00 00 00 08 50 53 45 4E 0C 01 00 00 PE......PSEN.... 4040: 00 08 43 53 45 4E 0C 01 00 00 00 08 4D 57 4F 53 ..CSEN......MWOS 4050: 0C 01 00 00 00 08 54 53 45 4E 0C 00 00 00 00 14 ......TSEN...... 4060: 49 07 5F 50 44 43 01 8A 68 0A 08 43 41 50 41 70 I._PDC..h..CAPAp 4070: 43 41 50 41 54 59 50 45 8A 68 0A 00 52 45 56 53 CAPATYPE.h..REVS 4080: 8A 68 0A 04 53 49 5A 45 70 87 68 60 70 74 60 0A .h..SIZEp.h`pt`. 4090: 08 00 61 5B 13 68 0A 40 77 61 0A 08 00 54 45 4D ..a[.h.@wa...TEM 40A0: 50 08 53 54 53 30 11 07 0A 04 00 00 00 00 73 53 P.STS0........sS 40B0: 54 53 30 54 45 4D 50 62 5F 4F 53 43 11 13 0A 10 TS0TEMPb_OSC.... 40C0: 16 A6 77 40 0C 29 BE 47 9E BD D8 70 58 71 39 53 ..w@.).G...pXq9S 40D0: 52 45 56 53 53 49 5A 45 62 14 4E 0F 5F 4F 53 43 REVSSIZEb.N._OSC 40E0: 04 8A 6B 0A 04 43 41 50 41 70 43 41 50 41 54 59 ..k..CAPApCAPATY 40F0: 50 45 8A 6B 0A 00 53 54 53 30 8A 6B 0A 04 43 41 PE.k..STS0.k..CA 4100: 50 30 8A 68 0A 00 49 49 44 30 8A 68 0A 04 49 49 P0.h..IID0.h..II 4110: 44 31 8A 68 0A 08 49 49 44 32 8A 68 0A 0C 49 49 D1.h..IID2.h..II 4120: 44 33 08 55 49 44 30 11 13 0A 10 16 A6 77 40 0C D3.UID0......w@. 4130: 29 BE 47 9E BD D8 70 58 71 39 53 8A 68 0A 00 45 ).G...pXq9S.h..E 4140: 49 44 30 8A 68 0A 04 45 49 44 31 8A 68 0A 08 45 ID0.h..EID1.h..E 4150: 49 44 32 8A 68 0A 0C 45 49 44 33 A0 47 03 92 90 ID2.h..EID3.G... 4160: 90 93 49 49 44 30 45 49 44 30 93 49 49 44 31 45 ..IID0EID0.IID1E 4170: 49 44 31 90 93 49 49 44 32 45 49 44 32 93 49 49 ID1..IID2EID2.II 4180: 44 33 45 49 44 33 70 0A 06 88 53 54 53 30 0A 00 D3EID3p...STS0.. 4190: 00 A4 6B A0 44 01 92 93 69 0A 01 70 0A 0A 88 53 ..k.D...i..p...S 41A0: 54 53 30 0A 00 00 A4 6B A0 48 01 7B 53 54 53 30 TS0....k.H.{STS0 41B0: 0A 01 00 7B 43 41 50 30 0B FF 0B 43 41 50 30 A4 ...{CAP0...CAP0. 41C0: 6B 7B 43 41 50 30 0B FF 0B 43 41 50 30 70 43 41 k{CAP0...CAP0pCA 41D0: 50 30 54 59 50 45 A4 6B 14 4D 11 5F 50 53 53 00 P0TYPE.k.M._PSS. 41E0: A0 43 11 50 53 45 4E A4 12 4B 10 08 12 20 06 0C .C.PSEN..K... .. 41F0: 5B 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C 0A [.....8......... 4200: 00 00 00 0C 13 00 00 00 0C 13 00 00 00 12 20 06 .............. . 4210: 0C 5A 09 00 00 0C 80 38 01 00 0C 0A 00 00 00 0C .Z.....8........ 4220: 0A 00 00 00 0C 12 00 00 00 0C 12 00 00 00 12 20 ............... 4230: 06 0C D5 08 00 00 0C D0 01 01 00 0C 0A 00 00 00 ................ 4240: 0C 0A 00 00 00 0C 11 00 00 00 0C 11 00 00 00 12 ................ 4250: 20 06 0C 50 08 00 00 0C D8 D6 00 00 0C 0A 00 00 ..P............ 4260: 00 0C 0A 00 00 00 0C 10 00 00 00 0C 10 00 00 00 ................ 4270: 12 20 06 0C CB 07 00 00 0C 50 C3 00 00 0C 0A 00 . .......P...... 4280: 00 00 0C 0A 00 00 00 0C 0F 00 00 00 0C 0F 00 00 ................ 4290: 00 12 20 06 0C 46 07 00 00 0C 28 A0 00 00 0C 0A .. ..F....(..... 42A0: 00 00 00 0C 0A 00 00 00 0C 0E 00 00 00 0C 0E 00 ................ 42B0: 00 00 12 20 06 0C C1 06 00 00 0C D0 84 00 00 0C ... ............ 42C0: 0A 00 00 00 0C 0A 00 00 00 0C 0D 00 00 00 0C 0D ................ 42D0: 00 00 00 12 20 06 0C 3C 06 00 00 0C 30 75 00 00 .... ..<....0u.. 42E0: 0C 0A 00 00 00 0C 0A 00 00 00 0C 0C 00 00 00 0C ................ 42F0: 0C 00 00 00 A4 00 14 35 5F 50 43 54 00 A4 12 4D .......5_PCT...M 4300: 02 02 11 14 0A 11 82 0C 00 7F 40 00 00 99 01 00 ..........@..... 4310: 00 00 00 00 00 79 00 11 14 0A 11 82 0C 00 7F 10 .....y.......... 4320: 00 00 98 01 00 00 00 00 00 00 79 00 14 08 5F 50 ..........y..._P 4330: 50 43 00 A4 00 14 42 03 5F 50 53 44 00 A0 48 02 PC....B._PSD..H. 4340: 7B 54 59 50 45 0C 20 08 00 00 00 A4 12 49 01 01 {TYPE. ......I.. 4350: 12 15 05 0A 05 0A 00 0C 01 00 00 00 0C FE 00 00 ................ 4360: 00 0C 08 00 00 00 A4 00 14 4B 0F 5F 43 53 54 00 .........K._CST. 4370: A0 41 0F 43 53 45 4E A0 4D 07 90 4D 57 4F 53 7B .A.CSEN.M..MWOS{ 4380: 54 59 50 45 0B 00 02 00 A4 12 4B 06 04 0A 03 12 TYPE......K..... 4390: 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 00 00 00 !............... 43A0: 00 00 00 00 00 79 00 0A 01 0B 01 00 0C E8 03 00 .....y.......... 43B0: 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 01 10 ..!............. 43C0: 00 00 00 00 00 00 00 79 00 0A 02 0B 40 00 0C F4 .......y....@... 43D0: 01 00 00 12 21 04 11 14 0A 11 82 0C 00 7F 01 02 ....!........... 43E0: 01 20 00 00 00 00 00 00 00 79 00 0A 03 0B 60 00 . .......y....`. 43F0: 0C 5E 01 00 00 A4 12 4B 06 04 0A 03 12 21 04 11 .^.....K.....!.. 4400: 14 0A 11 82 0C 00 7F 00 00 00 00 00 00 00 00 00 ................ 4410: 00 00 79 00 0A 01 0B 01 00 0C E8 03 00 00 12 21 ..y............! 4420: 04 11 14 0A 11 82 0C 00 01 08 00 00 14 08 00 00 ................ 4430: 00 00 00 00 79 00 0A 02 0B 40 00 0C F4 01 00 00 ....y....@...... 4440: 12 21 04 11 14 0A 11 82 0C 00 01 08 00 00 15 08 .!.............. 4450: 00 00 00 00 00 00 79 00 0A 03 0B 60 00 0C 5E 01 ......y....`..^. 4460: 00 00 A4 00 .... [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 181 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 95+ messages in thread
end of thread, other threads:[~2015-07-29 7:45 UTC | newest] Thread overview: 95+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-07-16 10:32 PV-vNUMA issue: topology is misinterpreted by the guest Dario Faggioli 2015-07-16 10:47 ` Jan Beulich 2015-07-16 10:56 ` Andrew Cooper 2015-07-16 15:25 ` Wei Liu 2015-07-16 15:45 ` Andrew Cooper 2015-07-16 15:50 ` Boris Ostrovsky 2015-07-16 16:29 ` Jan Beulich 2015-07-16 16:39 ` Andrew Cooper 2015-07-16 16:59 ` Boris Ostrovsky 2015-07-17 6:09 ` Jan Beulich 2015-07-17 7:27 ` Dario Faggioli 2015-07-17 7:42 ` Jan Beulich 2015-07-17 8:44 ` Wei Liu 2015-07-17 18:17 ` Boris Ostrovsky 2015-07-20 14:09 ` Dario Faggioli 2015-07-20 14:43 ` Boris Ostrovsky 2015-07-21 20:00 ` Boris Ostrovsky 2015-07-22 13:36 ` Dario Faggioli 2015-07-22 13:50 ` Juergen Gross 2015-07-22 13:58 ` Boris Ostrovsky 2015-07-22 14:09 ` Juergen Gross 2015-07-22 14:44 ` Boris Ostrovsky 2015-07-23 4:43 ` Juergen Gross 2015-07-23 7:28 ` Jan Beulich 2015-07-23 9:42 ` Andrew Cooper 2015-07-23 14:07 ` Dario Faggioli 2015-07-23 14:13 ` Juergen Gross 2015-07-24 10:28 ` Juergen Gross 2015-07-24 14:44 ` Dario Faggioli 2015-07-24 15:14 ` Juergen Gross 2015-07-24 15:24 ` Juergen Gross 2015-07-24 15:58 ` Dario Faggioli 2015-07-24 16:09 ` Konrad Rzeszutek Wilk 2015-07-24 16:14 ` Dario Faggioli 2015-07-24 16:18 ` Juergen Gross 2015-07-24 16:29 ` Konrad Rzeszutek Wilk 2015-07-24 16:39 ` Juergen Gross 2015-07-24 16:44 ` Boris Ostrovsky 2015-07-27 4:35 ` Juergen Gross 2015-07-27 10:43 ` George Dunlap 2015-07-27 10:54 ` Andrew Cooper 2015-07-27 11:13 ` Juergen Gross 2015-07-27 10:54 ` Juergen Gross 2015-07-27 11:11 ` George Dunlap 2015-07-27 12:01 ` Juergen Gross 2015-07-27 12:16 ` Tim Deegan 2015-07-27 13:23 ` Dario Faggioli 2015-07-27 14:02 ` Juergen Gross 2015-07-27 14:02 ` Dario Faggioli 2015-07-27 10:41 ` George Dunlap 2015-07-27 10:49 ` Andrew Cooper 2015-07-27 13:11 ` Dario Faggioli 2015-07-24 16:10 ` Juergen Gross 2015-07-24 16:40 ` Boris Ostrovsky 2015-07-24 16:48 ` Juergen Gross 2015-07-24 17:11 ` Boris Ostrovsky 2015-07-27 13:40 ` Dario Faggioli 2015-07-27 4:24 ` Juergen Gross 2015-07-27 14:09 ` Dario Faggioli 2015-07-27 14:34 ` Boris Ostrovsky 2015-07-27 14:43 ` Juergen Gross 2015-07-27 14:51 ` Boris Ostrovsky 2015-07-27 15:03 ` Juergen Gross 2015-07-27 14:47 ` Juergen Gross 2015-07-27 14:58 ` Dario Faggioli 2015-07-28 4:29 ` Juergen Gross 2015-07-28 15:11 ` Juergen Gross 2015-07-28 16:17 ` Dario Faggioli 2015-07-28 17:13 ` Dario Faggioli 2015-07-29 6:04 ` Juergen Gross 2015-07-29 7:09 ` Dario Faggioli 2015-07-29 7:44 ` Dario Faggioli 2015-07-24 16:05 ` Dario Faggioli 2015-07-28 10:05 ` Wei Liu 2015-07-28 15:17 ` Dario Faggioli 2015-07-24 20:27 ` Elena Ufimtseva 2015-07-22 14:50 ` Dario Faggioli 2015-07-22 15:32 ` Boris Ostrovsky 2015-07-22 15:49 ` Dario Faggioli 2015-07-22 18:10 ` Boris Ostrovsky 2015-07-23 7:25 ` Jan Beulich 2015-07-24 16:03 ` Boris Ostrovsky 2015-07-23 13:46 ` Dario Faggioli 2015-07-17 10:17 ` Andrew Cooper 2015-07-16 15:26 ` Wei Liu 2015-07-27 15:13 ` David Vrabel 2015-07-27 16:02 ` Dario Faggioli 2015-07-27 16:31 ` David Vrabel 2015-07-27 16:33 ` Andrew Cooper 2015-07-27 17:42 ` Dario Faggioli 2015-07-27 17:50 ` Konrad Rzeszutek Wilk 2015-07-27 23:19 ` Andrew Cooper 2015-07-28 3:52 ` Juergen Gross 2015-07-28 9:40 ` Andrew Cooper 2015-07-28 9:28 ` Dario Faggioli
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).