From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752494AbdDGCx1 (ORCPT ); Thu, 6 Apr 2017 22:53:27 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:5399 "EHLO dggrg03-dlp.huawei.com" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1751289AbdDGCxW (ORCPT ); Thu, 6 Apr 2017 22:53:22 -0400 Subject: Re: [PATCH] kvm: pass the virtual SEI syndrome to guest OS To: Laszlo Ersek , Achin Gupta References: <76795e20-2f20-1e54-cfa5-7444f28b18ee@huawei.com> <20170321113428.GC15920@cbox> <58D17AF0.2010802@arm.com> <20170321193933.GB31111@cbox> <58DA3F68.6090901@arm.com> <20170328112328.GA31156@cbox> <20170328115413.GJ23682@e104320-lin> <58DA67BA.8070404@arm.com> <5b7352f4-4965-3ed5-3879-db871797be47@huawei.com> <20170329103658.GQ23682@e104320-lin> <2a427164-9b37-6711-3a56-906634ba7f12@redhat.com> <7c5c8ab7-8fcc-1c98-0bc1-cccb66c4c84d@huawei.com> <6ac1597a-2ed5-36b2-848d-5fd048b16d66@redhat.com> CC: , , , , James Morse , Christoffer Dall , , Marc Zyngier , , , , , , , , , , , , , , , , , , Michael Tsirkin , Igor Mammedov From: gengdongjiu Message-ID: <55546a4b-c33b-37b9-dafe-15ce75bc1b62@huawei.com> Date: Fri, 7 Apr 2017 10:52:10 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 MIME-Version: 1.0 In-Reply-To: <6ac1597a-2ed5-36b2-848d-5fd048b16d66@redhat.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.142.68.147] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020206.58E6FEF4.011F,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: ed8ca8d6f08004e437408708d9a70339 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Laszlo, thanks. On 2017/4/7 2:55, Laszlo Ersek wrote: > On 04/06/17 14:35, gengdongjiu wrote: >> Dear, Laszlo >> Thanks for your detailed explanation. >> >> On 2017/3/29 19:58, Laszlo Ersek wrote: >>> (This ought to be one of the longest address lists I've ever seen :) >>> Thanks for the CC. I'm glad Shannon is already on the CC list. For good >>> measure, I'm adding MST and Igor.) >>> >>> On 03/29/17 12:36, Achin Gupta wrote: >>>> Hi gengdongjiu, >>>> >>>> On Wed, Mar 29, 2017 at 05:36:37PM +0800, gengdongjiu wrote: >>>>> >>>>> Hi Laszlo/Biesheuvel/Qemu developer, >>>>> >>>>> Now I encounter a issue and want to consult with you in ARM64 platform， as described below: >>>>> >>>>> when guest OS happen synchronous or asynchronous abort, kvm needs >>>>> to send the error address to Qemu or UEFI through sigbus to >>>>> dynamically generate APEI table. from my investigation, there are >>>>> two ways: >>>>> >>>>> (1) Qemu get the error address, and generate the APEI table, then >>>>> notify UEFI to know this generation, then inject abort error to >>>>> guest OS, guest OS read the APEI table. >>>>> (2) Qemu get the error address, and let UEFI to generate the APEI >>>>> table, then inject abort error to guest OS, guest OS read the APEI >>>>> table. >>>> >>>> Just being pedantic! I don't think we are talking about creating the APEI table >>>> dynamically here. The issue is: Once KVM has received an error that is destined >>>> for a guest it will raise a SIGBUS to Qemu. Now before Qemu can inject the error >>>> into the guest OS, a CPER (Common Platform Error Record) has to be generated >>>> corresponding to the error source (GHES corresponding to memory subsystem, >>>> processor etc) to allow the guest OS to do anything meaningful with the >>>> error. So who should create the CPER is the question. >>>> >>>> At the EL3/EL2 interface (Secure Firmware and OS/Hypervisor), an error arrives >>>> at EL3 and secure firmware (at EL3 or a lower secure exception level) is >>>> responsible for creating the CPER. ARM is experimenting with using a Standalone >>>> MM EDK2 image in the secure world to do the CPER creation. This will avoid >>>> adding the same code in ARM TF in EL3 (better for security). The error will then >>>> be injected into the OS/Hypervisor (through SEA/SEI/SDEI) through ARM Trusted >>>> Firmware. >>>> >>>> Qemu is essentially fulfilling the role of secure firmware at the EL2/EL1 >>>> interface (as discussed with Christoffer below). So it should generate the CPER >>>> before injecting the error. >>>> >>>> This is corresponds to (1) above apart from notifying UEFI (I am assuming you >>>> mean guest UEFI). At this time, the guest OS already knows where to pick up the >>>> CPER from through the HEST. Qemu has to create the CPER and populate its address >>>> at the address exported in the HEST. Guest UEFI should not be involved in this >>>> flow. Its job was to create the HEST at boot and that has been done by this >>>> stage. >>>> >>>> Qemu folk will be able to add but it looks like support for CPER generation will >>>> need to be added to Qemu. We need to resolve this. >>>> >>>> Do shout if I am missing anything above. >>> >>> After reading this email, the use case looks *very* similar to what >>> we've just done with VMGENID for QEMU 2.9. >>> >>> We have a facility between QEMU and the guest firmware, called "ACPI >>> linker/loader", with which QEMU instructs the firmware to >>> >>> - allocate and download blobs into guest RAM (AcpiNVS type memory) -- >>> ALLOCATE command, >>> >>> - relocate pointers in those blobs, to fields in other (or the same) >>> blobs -- ADD_POINTER command, >>> >>> - set ACPI table checksums -- ADD_CHECKSUM command, >>> >>> - and send GPAs of fields within such blobs back to QEMU -- >>> WRITE_POINTER command. >>> >>> This is how I imagine we can map the facility to the current use case >>> (note that this is the first time I read about HEST / GHES / CPER): >>> >>> etc/acpi/tables etc/hardware_errors >>> ================ ========================================== >>> +-----------+ >>> +--------------+ | address | +-> +--------------+ >>> | HEST + | registers | | | Error Status | >>> + +------------+ | +---------+ | | Data Block 1 | >>> | | GHES | --> | | address | --------+ | +------------+ >>> | | GHES | --> | | address | ------+ | | CPER | >>> | | GHES | --> | | address | ----+ | | | CPER | >>> | | GHES | --> | | address | -+ | | | | CPER | >>> +-+------------+ +-+---------+ | | | +-+------------+ >>> | | | >>> | | +---> +--------------+ >>> | | | Error Status | >>> | | | Data Block 2 | >>> | | | +------------+ >>> | | | | CPER | >>> | | | | CPER | >>> | | +-+------------+ >>> | | >>> | +-----> +--------------+ >>> | | Error Status | >>> | | Data Block 3 | >>> | | +------------+ >>> | | | CPER | >>> | +-+------------+ >>> | >>> +--------> +--------------+ >>> | Error Status | >>> | Data Block 4 | >>> | +------------+ >>> | | CPER | >>> | | CPER | >>> | | CPER | >>> +-+------------+ >>> >>> (1) QEMU generates the HEST ACPI table. This table goes in the current >>> "etc/acpi/tables" fw_cfg blob. Given N error sources, there will be N >>> GHES objects in the HEST. >>> >>> (2) We introduce a new fw_cfg blob called "etc/hardware_errors". QEMU >>> also populates this blob. >>> >>> (2a) Given N error sources, the (unnamed) table of address registers >>> will contain N address registers. >>> >>> (2b) Given N error sources, the "etc/hardwre_errors" fw_cfg blob will >>> also contain N Error Status Data Blocks. >>> >>> I don't know about the sizing (number of CPERs) each Error Status Data >>> Block has to contain, but I understand it is all pre-allocated as far as >>> the OS is concerned, which matches our capabilities well. >> here I have a question. as you comment: " 'etc/hardwre_errors' fw_cfg blob will also contain N Error Status Data Blocks", >> Because the CPER numbers is not fixed, how to assign each "Error Status Data Block" size using one "etc/hardwre_errors" fw_cfg blob. >> when use one etc/hardwre_errors, will the N Error Status Data Block use one continuous buffer? as shown below. if so, maybe it not convenient for each data block size extension. >> I see the bios_linker_loader_alloc will allocate one continuous buffer for a blob(such as VMGENID_GUID_FW_CFG_FILE) >> >> /* Allocate guest memory for the Data fw_cfg blob */ >> bios_linker_loader_alloc(linker, VMGENID_GUID_FW_CFG_FILE, guid, 4096, >> false /* page boundary, high memory */); >> >> >> >> -> +--------------+ >> | HEST + | registers | | Error Status | >> + +------------+ | +---------+ | Data Block | >> | | GHES | --> | | address | --------+-->| +------------+ >> | | GHES | --> | | address | ------+ | | CPER | >> | | GHES | --> | | address | ----+ | | | CPER | >> | | GHES | --> | | address | -+ | | | | CPER | >> +-+------------+ +-+---------+ | | +---> +--------------+ >> | | | | CPER | >> | | | | CPER | >> | +-----> +--------------+ >> | | | CPER | >> +--------> +--------------+ >> | | CPER | >> | | CPER | >> | | CPER | >> +-+------------+ >> >> >> >> so how about we use separate etc/hardwre_errorsN for each Error Status status Block? then >> >> etc/hardwre_errors0 >> etc/hardwre_errors1 >> ................... >> etc/hardwre_errors10 >> (the max N is 10) >> >> >> the N can be one of below values, according to ACPI spec "Table 18-345 Hardware Error Notification Structure" >> 0 – Polled >> 1 – External Interrupt >> 2 – Local Interrupt >> 3 – SCI >> 4 – NMI >> 5 - CMCI >> 6 - MCE >> 7 - GPIO-Signal >> 8 - ARMv8 SEA >> 9 - ARMv8 SEI >> 10 - External Interrupt - GSIV > > I'm unsure if, by "not fixed", you are saying > > the number of CPER entries that fits in Error Status Data Block N is > not *uniform* across 0 <= N <= 10 [1] > > or > > the number of CPER entries that fits in Error Status Data Block N is > not *known* in advance, for all of 0 <= N <= 10 [2] > > Which one is your point? > > If [1], that's no problem; you can simply sum the individual error > status data block sizes in advance, and allocate "etc/hardware_errors" > accordingly, using the total size. > > (Allocating one shared fw_cfg blob for all status data blocks is more > memory efficient, as each ALLOCATE command will allocate whole pages > (rounded up from the actual blob size).) > > If your point is [2], then splitting the error status data blocks to > separate fw_cfg blobs makes no difference: regardless of whether we try > to place all the error status data blocks in a single fw_cfg blob, or in > separate fw_cfg blobs, the individual data block cannot be resized at OS > runtime, so there's no way to make it work. > My Point is [2]. The HEST(Hardware Error Source Table) table format is here: https://wiki.linaro.org/LEG/Engineering/Kernel/RAS/APEITables#Hardware_Error_Source_Table_.28HEST.29 Now I understand your thought. > Thanks, > Laszlo > >> >> >> >> >>> >>> (3) QEMU generates the ACPI linker/loader script for the firmware, as >>> always. >>> >>> (3a) The HEST table is part of "etc/acpi/tables", which the firmware >>> already allocates memory for, and downloads (because QEMU already >>> generates an ALLOCATE linker/loader command for it already). >>> >>> (3b) QEMU will have to create another ALLOCATE command for the >>> "etc/hardware_errors" blob. The firmware allocates memory for this blob, >>> and downloads it. >>> >>> (4) QEMU generates, in the ACPI linker/loader script for the firwmare, N >>> ADD_POINTER commands, which point the GHES."Error Status >>> Address" fields in the HEST table, to the corresponding address >>> registers in the downloaded "etc/hardware_errors" blob. >>> >>> (5) QEMU generates an ADD_CHECKSUM command for the firmware, so that the >>> HEST table is correctly checksummed after executing the N ADD_POINTER >>> commands from (4). >>> >>> (6) QEMU generates N ADD_POINTER commands for the firmware, pointing the >>> address registers (located in guest memory, in the downloaded >>> "etc/hardware_errors" blob) to the respective Error Status Data Blocks. >>> >>> (7) (This is the trick.) For this step, we need a third, write-only >>> fw_cfg blob, called "etc/hardware_errors_addr". Through that blob, the >>> firmware can send back the guest-side allocation addresses to QEMU. >>> >>> Namely, the "etc/hardware_errors_addr" blob contains N 8-byte entries. >>> QEMU generates N WRITE_POINTER commands for the firmware. >>> >>> For error source K (0 <= K < N), QEMU instructs the firmware to >>> calculate the guest address of Error Status Data Block K, from the >>> QEMU-dictated offset within "etc/hardware_errors", and from the >>> guest-determined allocation base address for "etc/hardware_errors". The >>> firmware then writes the calculated address back to fw_cfg file >>> "etc/hardware_errors_addr", at offset K*8, according to the >>> WRITE_POINTER command. >>> >>> This way QEMU will know the GPA of each Error Status Data Block. >>> >>> (In fact this can be simplified to a single WRITE_POINTER command: the >>> address of the "address register table" can be sent back to QEMU as >>> well, which already contains all Error Status Data Block addresses.) >>> >>> (8) When QEMU gets SIGBUS from the kernel -- I hope that's going to come >>> through a signalfd -- QEMU can format the CPER right into guest memory, >>> and then inject whatever interrupt (or assert whatever GPIO line) is >>> necessary for notifying the guest. >>> >>> (9) This notification (in virtual hardware) can either be handled by the >>> guest kernel stand-alone, or else the guest kernel can invoke an ACPI >>> event handler method with it (which would be in the DSDT or one of the >>> SSDTs, also generated by QEMU). The ACPI event handler method could >>> invoke the specific guest kernel driver for errror handling via a >>> Notify() operation. >>> >>> I'm attracted to the above design because: >>> - it would leave the firmware alone after OS boot, and >>> - it would leave the firmware blissfully ignorant about HEST, GHES, >>> CPER, and the like. (That's why QEMU's ACPI linker/loader was invented >>> in the first place.) >>> >>> Thanks >>> Laszlo >>> >>>>> Do you think which modules generates the APEI table is better? UEFI or Qemu? >>>>> >>>>> >>>>> >>>>> >>>>> On 2017/3/28 21:40, James Morse wrote: >>>>>> Hi gengdongjiu, >>>>>> >>>>>> On 28/03/17 13:16, gengdongjiu wrote: >>>>>>> On 2017/3/28 19:54, Achin Gupta wrote: >>>>>>>> On Tue, Mar 28, 2017 at 01:23:28PM +0200, Christoffer Dall wrote: >>>>>>>>> On Tue, Mar 28, 2017 at 11:48:08AM +0100, James Morse wrote: >>>>>>>>>> On the host, part of UEFI is involved to generate the CPER records. >>>>>>>>>> In a guest?, I don't know. >>>>>>>>>> Qemu could generate the records, or drive some other component to do it. >>>>>>>>> >>>>>>>>> I think I am beginning to understand this a bit. Since the guet UEFI >>>>>>>>> instance is specifically built for the machine it runs on, QEMU's virt >>>>>>>>> machine in this case, they could simply agree (by some contract) to >>>>>>>>> place the records at some specific location in memory, and if the guest >>>>>>>>> kernel asks its guest UEFI for that location, things should just work by >>>>>>>>> having logic in QEMU to process error reports and populate guest memory. >>>>>>>>> >>>>>>>>> Is this how others see the world too? >>>>>>>> >>>>>>>> I think so! >>>>>>>> >>>>>>>> AFAIU, the memory where CPERs will reside should be specified in a GHES entry in >>>>>>>> the HEST. Is this not the case with a guest kernel i.e. the guest UEFI creates a >>>>>>>> HEST for the guest Kernel? >>>>>>>> >>>>>>>> If so, then the question is how the guest UEFI finds out where QEMU (acting as >>>>>>>> EL3 firmware) will populate the CPERs. This could either be a contract between >>>>>>>> the two or a guest DXE driver uses the MM_COMMUNICATE call (see [1]) to ask QEMU >>>>>>>> where the memory is. >>>>>>> >>>>>>> whether invoke the guest UEFI will be complex? not see the advantage. it seems x86 Qemu >>>>>>> directly generate the ACPI table, but I am not sure, we are checking the qemu >>>>>> logical. >>>>>>> let Qemu generate CPER record may be clear. >>>>>> >>>>>> At boot UEFI in the guest will need to make sure the areas of memory that may be >>>>>> used for CPER records are reserved. Whether UEFI or Qemu decides where these are >>>>>> needs deciding, (but probably not here)... >>>>>> >>>>>> At runtime, when an error has occurred, I agree it would be simpler (fewer >>>>>> components involved) if Qemu generates the CPER records. But if UEFI made the >>>>>> memory choice above they need to interact and it gets complicated again. The >>>>>> CPER records are defined in the UEFI spec, so I would expect UEFI to contain >>>>>> code to generate/parse them. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> James >>>>>> >>>>>> >>>>>> . >>>>>> >>>>> >>> >>> >>> . >>> >> > > > . > From mboxrd@z Thu Jan 1 00:00:00 1970 From: gengdongjiu Subject: Re: [edk2] [PATCH] kvm: pass the virtual SEI syndrome to guest OS Date: Fri, 7 Apr 2017 10:52:10 +0800 Message-ID: <55546a4b-c33b-37b9-dafe-15ce75bc1b62@huawei.com> References: <76795e20-2f20-1e54-cfa5-7444f28b18ee@huawei.com> <20170321113428.GC15920@cbox> <58D17AF0.2010802@arm.com> <20170321193933.GB31111@cbox> <58DA3F68.6090901@arm.com> <20170328112328.GA31156@cbox> <20170328115413.GJ23682@e104320-lin> <58DA67BA.8070404@arm.com> <5b7352f4-4965-3ed5-3879-db871797be47@huawei.com> <20170329103658.GQ23682@e104320-lin> <2a427164-9b37-6711-3a56-906634ba7f12@redhat.com> <7c5c8ab7-8fcc-1c98-0bc1-cccb66c4c84d@huawei.com> <6ac1597a-2ed5-36b2-848d-5fd048b16d66@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Cc: Michael Tsirkin , kvm@vger.kernel.org, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, qemu-devel@nongnu.org, wuquanming@huawei.com, wangxiongfeng2@huawei.com, Christoffer Dall , suzuki.poulose@arm.com, kvmarm@lists.cs.columbia.edu, Leif.Lindholm@linaro.com, huangshaoyu@huawei.com, vladimir.murzin@arm.com, xiexiuqi@huawei.com, Marc Zyngier , andre.przywara@arm.com, edk2-devel@lists.01.org, nd@arm.com, linux-arm-kernel@lists.infradead.org, ard.biesheuvel@linaro.org, linux-kernel@vger.kernel.org, James Morse , christoffer.dall@linaro.org To: Laszlo Ersek , Achin Gupta Return-path: In-Reply-To: <6ac1597a-2ed5-36b2-848d-5fd048b16d66@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: edk2-devel-bounces@lists.01.org Sender: "edk2-devel" List-Id: kvm.vger.kernel.org SGkgTGFzemxvLAogIHRoYW5rcy4KCk9uIDIwMTcvNC83IDI6NTUsIExhc3psbyBFcnNlayB3cm90 ZToKPiBPbiAwNC8wNi8xNyAxNDozNSwgZ2VuZ2RvbmdqaXUgd3JvdGU6Cj4+IERlYXIsIExhc3ps bwo+PiAgICBUaGFua3MgZm9yIHlvdXIgZGV0YWlsZWQgZXhwbGFuYXRpb24uCj4+Cj4+IE9uIDIw MTcvMy8yOSAxOTo1OCwgTGFzemxvIEVyc2VrIHdyb3RlOgo+Pj4gKFRoaXMgb3VnaHQgdG8gYmUg b25lIG9mIHRoZSBsb25nZXN0IGFkZHJlc3MgbGlzdHMgSSd2ZSBldmVyIHNlZW4gOikKPj4+IFRo YW5rcyBmb3IgdGhlIENDLiBJJ20gZ2xhZCBTaGFubm9uIGlzIGFscmVhZHkgb24gdGhlIENDIGxp c3QuIEZvciBnb29kCj4+PiBtZWFzdXJlLCBJJ20gYWRkaW5nIE1TVCBhbmQgSWdvci4pCj4+Pgo+ Pj4gT24gMDMvMjkvMTcgMTI6MzYsIEFjaGluIEd1cHRhIHdyb3RlOgo+Pj4+IEhpIGdlbmdkb25n aml1LAo+Pj4+Cj4+Pj4gT24gV2VkLCBNYXIgMjksIDIwMTcgYXQgMDU6MzY6MzdQTSArMDgwMCwg Z2VuZ2RvbmdqaXUgd3JvdGU6Cj4+Pj4+Cj4+Pj4+IEhpIExhc3psby9CaWVzaGV1dmVsL1FlbXUg ZGV2ZWxvcGVyLAo+Pj4+Pgo+Pj4+PiAgICBOb3cgSSBlbmNvdW50ZXIgYSBpc3N1ZSBhbmQgd2Fu dCB0byBjb25zdWx0IHdpdGggeW91IGluIEFSTTY0IHBsYXRmb3Jt77yMIGFzIGRlc2NyaWJlZCBi ZWxvdzoKPj4+Pj4KPj4+Pj4gd2hlbiBndWVzdCBPUyBoYXBwZW4gc3luY2hyb25vdXMgb3IgYXN5 bmNocm9ub3VzIGFib3J0LCBrdm0gbmVlZHMKPj4+Pj4gdG8gc2VuZCB0aGUgZXJyb3IgYWRkcmVz cyB0byBRZW11IG9yIFVFRkkgdGhyb3VnaCBzaWdidXMgdG8KPj4+Pj4gZHluYW1pY2FsbHkgZ2Vu ZXJhdGUgQVBFSSB0YWJsZS4gZnJvbSBteSBpbnZlc3RpZ2F0aW9uLCB0aGVyZSBhcmUKPj4+Pj4g dHdvIHdheXM6Cj4+Pj4+Cj4+Pj4+ICgxKSBRZW11IGdldCB0aGUgZXJyb3IgYWRkcmVzcywgYW5k IGdlbmVyYXRlIHRoZSBBUEVJIHRhYmxlLCB0aGVuCj4+Pj4+IG5vdGlmeSBVRUZJIHRvIGtub3cg dGhpcyBnZW5lcmF0aW9uLCB0aGVuIGluamVjdCBhYm9ydCBlcnJvciB0bwo+Pj4+PiBndWVzdCBP UywgZ3Vlc3QgT1MgcmVhZCB0aGUgQVBFSSB0YWJsZS4KPj4+Pj4gKDIpIFFlbXUgZ2V0IHRoZSBl cnJvciBhZGRyZXNzLCBhbmQgbGV0IFVFRkkgdG8gZ2VuZXJhdGUgdGhlIEFQRUkKPj4+Pj4gdGFi bGUsIHRoZW4gaW5qZWN0IGFib3J0IGVycm9yIHRvIGd1ZXN0IE9TLCBndWVzdCBPUyByZWFkIHRo ZSBBUEVJCj4+Pj4+IHRhYmxlLgo+Pj4+Cj4+Pj4gSnVzdCBiZWluZyBwZWRhbnRpYyEgSSBkb24n dCB0aGluayB3ZSBhcmUgdGFsa2luZyBhYm91dCBjcmVhdGluZyB0aGUgQVBFSSB0YWJsZQo+Pj4+ IGR5bmFtaWNhbGx5IGhlcmUuIFRoZSBpc3N1ZSBpczogT25jZSBLVk0gaGFzIHJlY2VpdmVkIGFu IGVycm9yIHRoYXQgaXMgZGVzdGluZWQKPj4+PiBmb3IgYSBndWVzdCBpdCB3aWxsIHJhaXNlIGEg U0lHQlVTIHRvIFFlbXUuIE5vdyBiZWZvcmUgUWVtdSBjYW4gaW5qZWN0IHRoZSBlcnJvcgo+Pj4+ IGludG8gdGhlIGd1ZXN0IE9TLCBhIENQRVIgKENvbW1vbiBQbGF0Zm9ybSBFcnJvciBSZWNvcmQp IGhhcyB0byBiZSBnZW5lcmF0ZWQKPj4+PiBjb3JyZXNwb25kaW5nIHRvIHRoZSBlcnJvciBzb3Vy Y2UgKEdIRVMgY29ycmVzcG9uZGluZyB0byBtZW1vcnkgc3Vic3lzdGVtLAo+Pj4+IHByb2Nlc3Nv ciBldGMpIHRvIGFsbG93IHRoZSBndWVzdCBPUyB0byBkbyBhbnl0aGluZyBtZWFuaW5nZnVsIHdp dGggdGhlCj4+Pj4gZXJyb3IuIFNvIHdobyBzaG91bGQgY3JlYXRlIHRoZSBDUEVSIGlzIHRoZSBx dWVzdGlvbi4KPj4+Pgo+Pj4+IEF0IHRoZSBFTDMvRUwyIGludGVyZmFjZSAoU2VjdXJlIEZpcm13 YXJlIGFuZCBPUy9IeXBlcnZpc29yKSwgYW4gZXJyb3IgYXJyaXZlcwo+Pj4+IGF0IEVMMyBhbmQg c2VjdXJlIGZpcm13YXJlIChhdCBFTDMgb3IgYSBsb3dlciBzZWN1cmUgZXhjZXB0aW9uIGxldmVs KSBpcwo+Pj4+IHJlc3BvbnNpYmxlIGZvciBjcmVhdGluZyB0aGUgQ1BFUi4gQVJNIGlzIGV4cGVy aW1lbnRpbmcgd2l0aCB1c2luZyBhIFN0YW5kYWxvbmUKPj4+PiBNTSBFREsyIGltYWdlIGluIHRo ZSBzZWN1cmUgd29ybGQgdG8gZG8gdGhlIENQRVIgY3JlYXRpb24uIFRoaXMgd2lsbCBhdm9pZAo+ Pj4+IGFkZGluZyB0aGUgc2FtZSBjb2RlIGluIEFSTSBURiBpbiBFTDMgKGJldHRlciBmb3Igc2Vj dXJpdHkpLiBUaGUgZXJyb3Igd2lsbCB0aGVuCj4+Pj4gYmUgaW5qZWN0ZWQgaW50byB0aGUgT1Mv SHlwZXJ2aXNvciAodGhyb3VnaCBTRUEvU0VJL1NERUkpIHRocm91Z2ggQVJNIFRydXN0ZWQKPj4+ PiBGaXJtd2FyZS4KPj4+Pgo+Pj4+IFFlbXUgaXMgZXNzZW50aWFsbHkgZnVsZmlsbGluZyB0aGUg cm9sZSBvZiBzZWN1cmUgZmlybXdhcmUgYXQgdGhlIEVMMi9FTDEKPj4+PiBpbnRlcmZhY2UgKGFz IGRpc2N1c3NlZCB3aXRoIENocmlzdG9mZmVyIGJlbG93KS4gU28gaXQgc2hvdWxkIGdlbmVyYXRl IHRoZSBDUEVSCj4+Pj4gYmVmb3JlIGluamVjdGluZyB0aGUgZXJyb3IuCj4+Pj4KPj4+PiBUaGlz IGlzIGNvcnJlc3BvbmRzIHRvICgxKSBhYm92ZSBhcGFydCBmcm9tIG5vdGlmeWluZyBVRUZJIChJ IGFtIGFzc3VtaW5nIHlvdQo+Pj4+IG1lYW4gZ3Vlc3QgVUVGSSkuIEF0IHRoaXMgdGltZSwgdGhl IGd1ZXN0IE9TIGFscmVhZHkga25vd3Mgd2hlcmUgdG8gcGljayB1cCB0aGUKPj4+PiBDUEVSIGZy b20gdGhyb3VnaCB0aGUgSEVTVC4gUWVtdSBoYXMgdG8gY3JlYXRlIHRoZSBDUEVSIGFuZCBwb3B1 bGF0ZSBpdHMgYWRkcmVzcwo+Pj4+IGF0IHRoZSBhZGRyZXNzIGV4cG9ydGVkIGluIHRoZSBIRVNU LiBHdWVzdCBVRUZJIHNob3VsZCBub3QgYmUgaW52b2x2ZWQgaW4gdGhpcwo+Pj4+IGZsb3cuIEl0 cyBqb2Igd2FzIHRvIGNyZWF0ZSB0aGUgSEVTVCBhdCBib290IGFuZCB0aGF0IGhhcyBiZWVuIGRv bmUgYnkgdGhpcwo+Pj4+IHN0YWdlLgo+Pj4+Cj4+Pj4gUWVtdSBmb2xrIHdpbGwgYmUgYWJsZSB0 byBhZGQgYnV0IGl0IGxvb2tzIGxpa2Ugc3VwcG9ydCBmb3IgQ1BFUiBnZW5lcmF0aW9uIHdpbGwK Pj4+PiBuZWVkIHRvIGJlIGFkZGVkIHRvIFFlbXUuIFdlIG5lZWQgdG8gcmVzb2x2ZSB0aGlzLgo+ Pj4+Cj4+Pj4gRG8gc2hvdXQgaWYgSSBhbSBtaXNzaW5nIGFueXRoaW5nIGFib3ZlLgo+Pj4KPj4+ IEFmdGVyIHJlYWRpbmcgdGhpcyBlbWFpbCwgdGhlIHVzZSBjYXNlIGxvb2tzICp2ZXJ5KiBzaW1p bGFyIHRvIHdoYXQKPj4+IHdlJ3ZlIGp1c3QgZG9uZSB3aXRoIFZNR0VOSUQgZm9yIFFFTVUgMi45 Lgo+Pj4KPj4+IFdlIGhhdmUgYSBmYWNpbGl0eSBiZXR3ZWVuIFFFTVUgYW5kIHRoZSBndWVzdCBm aXJtd2FyZSwgY2FsbGVkICJBQ1BJCj4+PiBsaW5rZXIvbG9hZGVyIiwgd2l0aCB3aGljaCBRRU1V IGluc3RydWN0cyB0aGUgZmlybXdhcmUgdG8KPj4+Cj4+PiAtIGFsbG9jYXRlIGFuZCBkb3dubG9h ZCBibG9icyBpbnRvIGd1ZXN0IFJBTSAoQWNwaU5WUyB0eXBlIG1lbW9yeSkgLS0KPj4+IEFMTE9D QVRFIGNvbW1hbmQsCj4+Pgo+Pj4gLSByZWxvY2F0ZSBwb2ludGVycyBpbiB0aG9zZSBibG9icywg dG8gZmllbGRzIGluIG90aGVyIChvciB0aGUgc2FtZSkKPj4+IGJsb2JzIC0tIEFERF9QT0lOVEVS IGNvbW1hbmQsCj4+Pgo+Pj4gLSBzZXQgQUNQSSB0YWJsZSBjaGVja3N1bXMgLS0gQUREX0NIRUNL U1VNIGNvbW1hbmQsCj4+Pgo+Pj4gLSBhbmQgc2VuZCBHUEFzIG9mIGZpZWxkcyB3aXRoaW4gc3Vj aCBibG9icyBiYWNrIHRvIFFFTVUgLS0KPj4+IFdSSVRFX1BPSU5URVIgY29tbWFuZC4KPj4+Cj4+ PiBUaGlzIGlzIGhvdyBJIGltYWdpbmUgd2UgY2FuIG1hcCB0aGUgZmFjaWxpdHkgdG8gdGhlIGN1 cnJlbnQgdXNlIGNhc2UKPj4+IChub3RlIHRoYXQgdGhpcyBpcyB0aGUgZmlyc3QgdGltZSBJIHJl YWQgYWJvdXQgSEVTVCAvIEdIRVMgLyBDUEVSKToKPj4+Cj4+PiAgICAgZXRjL2FjcGkvdGFibGVz ICAgICAgICAgICAgICAgICBldGMvaGFyZHdhcmVfZXJyb3JzCj4+PiAgICAgPT09PT09PT09PT09 PT09PSAgICAgPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Cj4+PiAg ICAgICAgICAgICAgICAgICAgICAgICAgKy0tLS0tLS0tLS0tKwo+Pj4gICAgICstLS0tLS0tLS0t LS0tLSsgICAgIHwgYWRkcmVzcyAgIHwgICAgICAgICArLT4gKy0tLS0tLS0tLS0tLS0tKwo+Pj4g ICAgIHwgICAgSEVTVCAgICAgICsgICAgIHwgcmVnaXN0ZXJzIHwgICAgICAgICB8ICAgfCBFcnJv ciBTdGF0dXMgfAo+Pj4gICAgICsgKy0tLS0tLS0tLS0tLSsgICAgIHwgKy0tLS0tLS0tLSsgICAg ICAgICB8ICAgfCBEYXRhIEJsb2NrIDEgfAo+Pj4gICAgIHwgfCBHSEVTICAgICAgIHwgLS0+IHwg fCBhZGRyZXNzIHwgLS0tLS0tLS0rICAgfCArLS0tLS0tLS0tLS0tKwo+Pj4gICAgIHwgfCBHSEVT ICAgICAgIHwgLS0+IHwgfCBhZGRyZXNzIHwgLS0tLS0tKyAgICAgfCB8ICBDUEVSICAgICAgfAo+ Pj4gICAgIHwgfCBHSEVTICAgICAgIHwgLS0+IHwgfCBhZGRyZXNzIHwgLS0tLSsgfCAgICAgfCB8 ICBDUEVSICAgICAgfAo+Pj4gICAgIHwgfCBHSEVTICAgICAgIHwgLS0+IHwgfCBhZGRyZXNzIHwg LSsgIHwgfCAgICAgfCB8ICBDUEVSICAgICAgfAo+Pj4gICAgICstKy0tLS0tLS0tLS0tLSsgICAg ICstKy0tLS0tLS0tLSsgIHwgIHwgfCAgICAgKy0rLS0tLS0tLS0tLS0tKwo+Pj4gICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgIHwgfAo+Pj4gICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIHwgIHwgKy0tLT4gKy0tLS0tLS0tLS0tLS0tKwo+Pj4g ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgIHwgICAgICAgfCBFcnJv ciBTdGF0dXMgfAo+Pj4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwg IHwgICAgICAgfCBEYXRhIEJsb2NrIDIgfAo+Pj4gICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHwgIHwgICAgICAgfCArLS0tLS0tLS0tLS0tKwo+Pj4gICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgIHwgICAgICAgfCB8ICBDUEVSICAgICAgfAo+ Pj4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgIHwgICAgICAgfCB8 ICBDUEVSICAgICAgfAo+Pj4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IHwgIHwgICAgICAgKy0rLS0tLS0tLS0tLS0tKwo+Pj4gICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwgIHwKPj4+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICB8ICArLS0tLS0+ICstLS0tLS0tLS0tLS0tLSsKPj4+ICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgIHwgRXJyb3IgU3RhdHVzIHwKPj4+ICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgIHwgRGF0YSBC bG9jayAzIHwKPj4+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAg ICAgICAgIHwgKy0tLS0tLS0tLS0tLSsKPj4+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICB8ICAgICAgICAgIHwgfCAgQ1BFUiAgICAgIHwKPj4+ICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICstKy0tLS0tLS0tLS0tLSsKPj4+ ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8Cj4+PiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKy0tLS0tLS0tPiArLS0tLS0tLS0tLS0tLS0r Cj4+PiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8 IEVycm9yIFN0YXR1cyB8Cj4+PiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICB8IERhdGEgQmxvY2sgNCB8Cj4+PiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICstLS0tLS0tLS0tLS0rCj4+PiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IHwgIENQRVIgICAg ICB8Cj4+PiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB8IHwgIENQRVIgICAgICB8Cj4+PiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICB8IHwgIENQRVIgICAgICB8Cj4+PiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICArLSstLS0tLS0tLS0tLS0rCj4+Pgo+Pj4g KDEpIFFFTVUgZ2VuZXJhdGVzIHRoZSBIRVNUIEFDUEkgdGFibGUuIFRoaXMgdGFibGUgZ29lcyBp biB0aGUgY3VycmVudAo+Pj4gImV0Yy9hY3BpL3RhYmxlcyIgZndfY2ZnIGJsb2IuIEdpdmVuIE4g ZXJyb3Igc291cmNlcywgdGhlcmUgd2lsbCBiZSBOCj4+PiBHSEVTIG9iamVjdHMgaW4gdGhlIEhF U1QuCj4+Pgo+Pj4gKDIpIFdlIGludHJvZHVjZSBhIG5ldyBmd19jZmcgYmxvYiBjYWxsZWQgImV0 Yy9oYXJkd2FyZV9lcnJvcnMiLiBRRU1VCj4+PiBhbHNvIHBvcHVsYXRlcyB0aGlzIGJsb2IuCj4+ Pgo+Pj4gKDJhKSBHaXZlbiBOIGVycm9yIHNvdXJjZXMsIHRoZSAodW5uYW1lZCkgdGFibGUgb2Yg YWRkcmVzcyByZWdpc3RlcnMKPj4+IHdpbGwgY29udGFpbiBOIGFkZHJlc3MgcmVnaXN0ZXJzLgo+ Pj4KPj4+ICgyYikgR2l2ZW4gTiBlcnJvciBzb3VyY2VzLCB0aGUgImV0Yy9oYXJkd3JlX2Vycm9y cyIgZndfY2ZnIGJsb2Igd2lsbAo+Pj4gYWxzbyBjb250YWluIE4gRXJyb3IgU3RhdHVzIERhdGEg QmxvY2tzLgo+Pj4KPj4+IEkgZG9uJ3Qga25vdyBhYm91dCB0aGUgc2l6aW5nIChudW1iZXIgb2Yg Q1BFUnMpIGVhY2ggRXJyb3IgU3RhdHVzIERhdGEKPj4+IEJsb2NrIGhhcyB0byBjb250YWluLCBi dXQgSSB1bmRlcnN0YW5kIGl0IGlzIGFsbCBwcmUtYWxsb2NhdGVkIGFzIGZhciBhcwo+Pj4gdGhl IE9TIGlzIGNvbmNlcm5lZCwgd2hpY2ggbWF0Y2hlcyBvdXIgY2FwYWJpbGl0aWVzIHdlbGwuCj4+ IGhlcmUgSSBoYXZlIGEgcXVlc3Rpb24uIGFzIHlvdSBjb21tZW50OiAiICdldGMvaGFyZHdyZV9l cnJvcnMnIGZ3X2NmZyBibG9iIHdpbGwgYWxzbyBjb250YWluIE4gRXJyb3IgU3RhdHVzIERhdGEg QmxvY2tzIiwKPj4gQmVjYXVzZSB0aGUgQ1BFUiBudW1iZXJzIGlzIG5vdCBmaXhlZCwgaG93IHRv IGFzc2lnbiBlYWNoICJFcnJvciBTdGF0dXMgRGF0YSBCbG9jayIgc2l6ZSB1c2luZyBvbmUgImV0 Yy9oYXJkd3JlX2Vycm9ycyIgZndfY2ZnIGJsb2IuCj4+IHdoZW4gdXNlIG9uZSBldGMvaGFyZHdy ZV9lcnJvcnMsIHdpbGwgdGhlIE4gRXJyb3IgU3RhdHVzIERhdGEgQmxvY2sgdXNlIG9uZSBjb250 aW51b3VzIGJ1ZmZlcj8gYXMgc2hvd24gYmVsb3cuIGlmIHNvLCBtYXliZSBpdCBub3QgY29udmVu aWVudCBmb3IgZWFjaCBkYXRhIGJsb2NrIHNpemUgZXh0ZW5zaW9uLgo+PiBJIHNlZSB0aGUgYmlv c19saW5rZXJfbG9hZGVyX2FsbG9jIHdpbGwgYWxsb2NhdGUgb25lIGNvbnRpbnVvdXMgYnVmZmVy IGZvciBhIGJsb2Ioc3VjaCBhcyBWTUdFTklEX0dVSURfRldfQ0ZHX0ZJTEUpCj4+Cj4+ICAgICAv KiBBbGxvY2F0ZSBndWVzdCBtZW1vcnkgZm9yIHRoZSBEYXRhIGZ3X2NmZyBibG9iICovCj4+ICAg ICBiaW9zX2xpbmtlcl9sb2FkZXJfYWxsb2MobGlua2VyLCBWTUdFTklEX0dVSURfRldfQ0ZHX0ZJ TEUsIGd1aWQsIDQwOTYsCj4+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmFsc2UgLyog cGFnZSBib3VuZGFyeSwgaGlnaCBtZW1vcnkgKi8pOwo+Pgo+Pgo+Pgo+PiAtPiArLS0tLS0tLS0t LS0tLS0rCj4+ICAgICAgfCAgICBIRVNUICAgICAgKyAgICAgfCByZWdpc3RlcnMgfCAgICAgICAg ICAgICB8IEVycm9yIFN0YXR1cyB8Cj4+ICAgICAgKyArLS0tLS0tLS0tLS0tKyAgICAgfCArLS0t LS0tLS0tKyAgICAgICAgICAgICB8IERhdGEgQmxvY2sgIHwKPj4gICAgICB8IHwgR0hFUyAgICAg ICB8IC0tPiB8IHwgYWRkcmVzcyB8IC0tLS0tLS0tKy0tPnwgKy0tLS0tLS0tLS0tLSsKPj4gICAg ICB8IHwgR0hFUyAgICAgICB8IC0tPiB8IHwgYWRkcmVzcyB8IC0tLS0tLSsgICAgIHwgfCAgQ1BF UiAgICAgIHwKPj4gICAgICB8IHwgR0hFUyAgICAgICB8IC0tPiB8IHwgYWRkcmVzcyB8IC0tLS0r IHwgICAgIHwgfCAgQ1BFUiAgICAgIHwKPj4gICAgICB8IHwgR0hFUyAgICAgICB8IC0tPiB8IHwg YWRkcmVzcyB8IC0rICB8IHwgICAgIHwgfCAgQ1BFUiAgICAgIHwKPj4gICAgICArLSstLS0tLS0t LS0tLS0rICAgICArLSstLS0tLS0tLS0rICB8ICB8ICstLS0+ICstLS0tLS0tLS0tLS0tLSsKPj4g ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICB8ICAgICAgIHwgfCAg Q1BFUiAgICAgIHwKPj4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8 ICB8ICAgICAgIHwgfCAgQ1BFUiAgICAgIHwKPj4gICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICB8ICArLS0tLS0+ICstLS0tLS0tLS0tLS0tLSsKPj4gICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgIHwgfCAgQ1BFUiAgICAgIHwK Pj4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICArLS0tLS0tLS0+ICst LS0tLS0tLS0tLS0tLSsKPj4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHwgfCAgQ1BFUiAgICAgIHwKPj4gICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgfCAgQ1BFUiAgICAgIHwKPj4gICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgfCAgQ1BFUiAgICAg IHwKPj4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICstKy0tLS0tLS0tLS0tLSsKPj4KPj4KPj4KPj4gc28gaG93IGFib3V0IHdlIHVzZSBzZXBhcmF0 ZSBldGMvaGFyZHdyZV9lcnJvcnNOIGZvciBlYWNoIEVycm9yIFN0YXR1cyBzdGF0dXMgQmxvY2s/ IHRoZW4KPj4KPj4gZXRjL2hhcmR3cmVfZXJyb3JzMAo+PiBldGMvaGFyZHdyZV9lcnJvcnMxCj4+ IC4uLi4uLi4uLi4uLi4uLi4uLi4KPj4gZXRjL2hhcmR3cmVfZXJyb3JzMTAKPj4gKHRoZSBtYXgg TiBpcyAxMCkKPj4KPj4KPj4gdGhlIE4gY2FuIGJlIG9uZSBvZiBiZWxvdyB2YWx1ZXMsIGFjY29y ZGluZyB0byBBQ1BJIHNwZWMgIlRhYmxlIDE4LTM0NSBIYXJkd2FyZSBFcnJvciBOb3RpZmljYXRp b24gU3RydWN0dXJlIgo+PiAwIOKAkyBQb2xsZWQKPj4gMSDigJMgRXh0ZXJuYWwgSW50ZXJydXB0 Cj4+IDIg4oCTIExvY2FsIEludGVycnVwdAo+PiAzIOKAkyBTQ0kKPj4gNCDigJMgTk1JCj4+IDUg LSBDTUNJCj4+IDYgLSBNQ0UKPj4gNyAtIEdQSU8tU2lnbmFsCj4+IDggLSBBUk12OCBTRUEKPj4g OSAtIEFSTXY4IFNFSQo+PiAxMCAtIEV4dGVybmFsIEludGVycnVwdCAtIEdTSVYKPiAKPiBJJ20g dW5zdXJlIGlmLCBieSAibm90IGZpeGVkIiwgeW91IGFyZSBzYXlpbmcKPiAKPiAgIHRoZSBudW1i ZXIgb2YgQ1BFUiBlbnRyaWVzIHRoYXQgZml0cyBpbiBFcnJvciBTdGF0dXMgRGF0YSBCbG9jayBO IGlzCj4gICBub3QgKnVuaWZvcm0qIGFjcm9zcyAwIDw9IE4gPD0gMTAgWzFdCj4gCj4gb3IKPiAK PiAgIHRoZSBudW1iZXIgb2YgQ1BFUiBlbnRyaWVzIHRoYXQgZml0cyBpbiBFcnJvciBTdGF0dXMg RGF0YSBCbG9jayBOIGlzCj4gICBub3QgKmtub3duKiBpbiBhZHZhbmNlLCBmb3IgYWxsIG9mIDAg PD0gTiA8PSAxMCBbMl0KPiAKPiBXaGljaCBvbmUgaXMgeW91ciBwb2ludD8KPiAKPiBJZiBbMV0s IHRoYXQncyBubyBwcm9ibGVtOyB5b3UgY2FuIHNpbXBseSBzdW0gdGhlIGluZGl2aWR1YWwgZXJy b3IKPiBzdGF0dXMgZGF0YSBibG9jayBzaXplcyBpbiBhZHZhbmNlLCBhbmQgYWxsb2NhdGUgImV0 Yy9oYXJkd2FyZV9lcnJvcnMiCj4gYWNjb3JkaW5nbHksIHVzaW5nIHRoZSB0b3RhbCBzaXplLgo+ IAo+IChBbGxvY2F0aW5nIG9uZSBzaGFyZWQgZndfY2ZnIGJsb2IgZm9yIGFsbCBzdGF0dXMgZGF0 YSBibG9ja3MgaXMgbW9yZQo+IG1lbW9yeSBlZmZpY2llbnQsIGFzIGVhY2ggQUxMT0NBVEUgY29t bWFuZCB3aWxsIGFsbG9jYXRlIHdob2xlIHBhZ2VzCj4gKHJvdW5kZWQgdXAgZnJvbSB0aGUgYWN0 dWFsIGJsb2Igc2l6ZSkuKQo+IAo+IElmIHlvdXIgcG9pbnQgaXMgWzJdLCB0aGVuIHNwbGl0dGlu ZyB0aGUgZXJyb3Igc3RhdHVzIGRhdGEgYmxvY2tzIHRvCj4gc2VwYXJhdGUgZndfY2ZnIGJsb2Jz IG1ha2VzIG5vIGRpZmZlcmVuY2U6IHJlZ2FyZGxlc3Mgb2Ygd2hldGhlciB3ZSB0cnkKPiB0byBw bGFjZSBhbGwgdGhlIGVycm9yIHN0YXR1cyBkYXRhIGJsb2NrcyBpbiBhIHNpbmdsZSBmd19jZmcg YmxvYiwgb3IgaW4KPiBzZXBhcmF0ZSBmd19jZmcgYmxvYnMsIHRoZSBpbmRpdmlkdWFsIGRhdGEg YmxvY2sgY2Fubm90IGJlIHJlc2l6ZWQgYXQgT1MKPiBydW50aW1lLCBzbyB0aGVyZSdzIG5vIHdh eSB0byBtYWtlIGl0IHdvcmsuCj4KTXkgUG9pbnQgaXMgWzJdLiBUaGUgSEVTVChIYXJkd2FyZSBF cnJvciBTb3VyY2UgVGFibGUpIHRhYmxlIGZvcm1hdCBpcyBoZXJlOgpodHRwczovL3dpa2kubGlu YXJvLm9yZy9MRUcvRW5naW5lZXJpbmcvS2VybmVsL1JBUy9BUEVJVGFibGVzI0hhcmR3YXJlX0Vy cm9yX1NvdXJjZV9UYWJsZV8uMjhIRVNULjI5CgpOb3cgSSB1bmRlcnN0YW5kIHlvdXIgdGhvdWdo dC4KCj4gVGhhbmtzLAo+IExhc3psbwo+IAo+Pgo+Pgo+Pgo+Pgo+Pj4KPj4+ICgzKSBRRU1VIGdl bmVyYXRlcyB0aGUgQUNQSSBsaW5rZXIvbG9hZGVyIHNjcmlwdCBmb3IgdGhlIGZpcm13YXJlLCBh cwo+Pj4gYWx3YXlzLgo+Pj4KPj4+ICgzYSkgVGhlIEhFU1QgdGFibGUgaXMgcGFydCBvZiAiZXRj L2FjcGkvdGFibGVzIiwgd2hpY2ggdGhlIGZpcm13YXJlCj4+PiBhbHJlYWR5IGFsbG9jYXRlcyBt ZW1vcnkgZm9yLCBhbmQgZG93bmxvYWRzIChiZWNhdXNlIFFFTVUgYWxyZWFkeQo+Pj4gZ2VuZXJh dGVzIGFuIEFMTE9DQVRFIGxpbmtlci9sb2FkZXIgY29tbWFuZCBmb3IgaXQgYWxyZWFkeSkuCj4+ Pgo+Pj4gKDNiKSBRRU1VIHdpbGwgaGF2ZSB0byBjcmVhdGUgYW5vdGhlciBBTExPQ0FURSBjb21t YW5kIGZvciB0aGUKPj4+ICJldGMvaGFyZHdhcmVfZXJyb3JzIiBibG9iLiBUaGUgZmlybXdhcmUg YWxsb2NhdGVzIG1lbW9yeSBmb3IgdGhpcyBibG9iLAo+Pj4gYW5kIGRvd25sb2FkcyBpdC4KPj4+ Cj4+PiAoNCkgUUVNVSBnZW5lcmF0ZXMsIGluIHRoZSBBQ1BJIGxpbmtlci9sb2FkZXIgc2NyaXB0 IGZvciB0aGUgZmlyd21hcmUsIE4KPj4+IEFERF9QT0lOVEVSIGNvbW1hbmRzLCB3aGljaCBwb2lu dCB0aGUgR0hFUy4iRXJyb3IgU3RhdHVzCj4+PiBBZGRyZXNzIiBmaWVsZHMgaW4gdGhlIEhFU1Qg dGFibGUsIHRvIHRoZSBjb3JyZXNwb25kaW5nIGFkZHJlc3MKPj4+IHJlZ2lzdGVycyBpbiB0aGUg ZG93bmxvYWRlZCAiZXRjL2hhcmR3YXJlX2Vycm9ycyIgYmxvYi4KPj4+Cj4+PiAoNSkgUUVNVSBn ZW5lcmF0ZXMgYW4gQUREX0NIRUNLU1VNIGNvbW1hbmQgZm9yIHRoZSBmaXJtd2FyZSwgc28gdGhh dCB0aGUKPj4+IEhFU1QgdGFibGUgaXMgY29ycmVjdGx5IGNoZWNrc3VtbWVkIGFmdGVyIGV4ZWN1 dGluZyB0aGUgTiBBRERfUE9JTlRFUgo+Pj4gY29tbWFuZHMgZnJvbSAoNCkuCj4+Pgo+Pj4gKDYp IFFFTVUgZ2VuZXJhdGVzIE4gQUREX1BPSU5URVIgY29tbWFuZHMgZm9yIHRoZSBmaXJtd2FyZSwg cG9pbnRpbmcgdGhlCj4+PiBhZGRyZXNzIHJlZ2lzdGVycyAobG9jYXRlZCBpbiBndWVzdCBtZW1v cnksIGluIHRoZSBkb3dubG9hZGVkCj4+PiAiZXRjL2hhcmR3YXJlX2Vycm9ycyIgYmxvYikgdG8g dGhlIHJlc3BlY3RpdmUgRXJyb3IgU3RhdHVzIERhdGEgQmxvY2tzLgo+Pj4KPj4+ICg3KSAoVGhp cyBpcyB0aGUgdHJpY2suKSBGb3IgdGhpcyBzdGVwLCB3ZSBuZWVkIGEgdGhpcmQsIHdyaXRlLW9u bHkKPj4+IGZ3X2NmZyBibG9iLCBjYWxsZWQgImV0Yy9oYXJkd2FyZV9lcnJvcnNfYWRkciIuIFRo cm91Z2ggdGhhdCBibG9iLCB0aGUKPj4+IGZpcm13YXJlIGNhbiBzZW5kIGJhY2sgdGhlIGd1ZXN0 LXNpZGUgYWxsb2NhdGlvbiBhZGRyZXNzZXMgdG8gUUVNVS4KPj4+Cj4+PiBOYW1lbHksIHRoZSAi ZXRjL2hhcmR3YXJlX2Vycm9yc19hZGRyIiBibG9iIGNvbnRhaW5zIE4gOC1ieXRlIGVudHJpZXMu Cj4+PiBRRU1VIGdlbmVyYXRlcyBOIFdSSVRFX1BPSU5URVIgY29tbWFuZHMgZm9yIHRoZSBmaXJt d2FyZS4KPj4+Cj4+PiBGb3IgZXJyb3Igc291cmNlIEsgKDAgPD0gSyA8IE4pLCBRRU1VIGluc3Ry dWN0cyB0aGUgZmlybXdhcmUgdG8KPj4+IGNhbGN1bGF0ZSB0aGUgZ3Vlc3QgYWRkcmVzcyBvZiBF cnJvciBTdGF0dXMgRGF0YSBCbG9jayBLLCBmcm9tIHRoZQo+Pj4gUUVNVS1kaWN0YXRlZCBvZmZz ZXQgd2l0aGluICJldGMvaGFyZHdhcmVfZXJyb3JzIiwgYW5kIGZyb20gdGhlCj4+PiBndWVzdC1k ZXRlcm1pbmVkIGFsbG9jYXRpb24gYmFzZSBhZGRyZXNzIGZvciAiZXRjL2hhcmR3YXJlX2Vycm9y cyIuIFRoZQo+Pj4gZmlybXdhcmUgdGhlbiB3cml0ZXMgdGhlIGNhbGN1bGF0ZWQgYWRkcmVzcyBi YWNrIHRvIGZ3X2NmZyBmaWxlCj4+PiAiZXRjL2hhcmR3YXJlX2Vycm9yc19hZGRyIiwgYXQgb2Zm c2V0IEsqOCwgYWNjb3JkaW5nIHRvIHRoZQo+Pj4gV1JJVEVfUE9JTlRFUiBjb21tYW5kLgo+Pj4K Pj4+IFRoaXMgd2F5IFFFTVUgd2lsbCBrbm93IHRoZSBHUEEgb2YgZWFjaCBFcnJvciBTdGF0dXMg RGF0YSBCbG9jay4KPj4+Cj4+PiAoSW4gZmFjdCB0aGlzIGNhbiBiZSBzaW1wbGlmaWVkIHRvIGEg c2luZ2xlIFdSSVRFX1BPSU5URVIgY29tbWFuZDogdGhlCj4+PiBhZGRyZXNzIG9mIHRoZSAiYWRk cmVzcyByZWdpc3RlciB0YWJsZSIgY2FuIGJlIHNlbnQgYmFjayB0byBRRU1VIGFzCj4+PiB3ZWxs LCB3aGljaCBhbHJlYWR5IGNvbnRhaW5zIGFsbCBFcnJvciBTdGF0dXMgRGF0YSBCbG9jayBhZGRy ZXNzZXMuKQo+Pj4KPj4+ICg4KSBXaGVuIFFFTVUgZ2V0cyBTSUdCVVMgZnJvbSB0aGUga2VybmVs IC0tIEkgaG9wZSB0aGF0J3MgZ29pbmcgdG8gY29tZQo+Pj4gdGhyb3VnaCBhIHNpZ25hbGZkIC0t IFFFTVUgY2FuIGZvcm1hdCB0aGUgQ1BFUiByaWdodCBpbnRvIGd1ZXN0IG1lbW9yeSwKPj4+IGFu ZCB0aGVuIGluamVjdCB3aGF0ZXZlciBpbnRlcnJ1cHQgKG9yIGFzc2VydCB3aGF0ZXZlciBHUElP IGxpbmUpIGlzCj4+PiBuZWNlc3NhcnkgZm9yIG5vdGlmeWluZyB0aGUgZ3Vlc3QuCj4+Pgo+Pj4g KDkpIFRoaXMgbm90aWZpY2F0aW9uIChpbiB2aXJ0dWFsIGhhcmR3YXJlKSBjYW4gZWl0aGVyIGJl IGhhbmRsZWQgYnkgdGhlCj4+PiBndWVzdCBrZXJuZWwgc3RhbmQtYWxvbmUsIG9yIGVsc2UgdGhl IGd1ZXN0IGtlcm5lbCBjYW4gaW52b2tlIGFuIEFDUEkKPj4+IGV2ZW50IGhhbmRsZXIgbWV0aG9k IHdpdGggaXQgKHdoaWNoIHdvdWxkIGJlIGluIHRoZSBEU0RUIG9yIG9uZSBvZiB0aGUKPj4+IFNT RFRzLCBhbHNvIGdlbmVyYXRlZCBieSBRRU1VKS4gVGhlIEFDUEkgZXZlbnQgaGFuZGxlciBtZXRo b2QgY291bGQKPj4+IGludm9rZSB0aGUgc3BlY2lmaWMgZ3Vlc3Qga2VybmVsIGRyaXZlciBmb3Ig ZXJycm9yIGhhbmRsaW5nIHZpYSBhCj4+PiBOb3RpZnkoKSBvcGVyYXRpb24uCj4+Pgo+Pj4gSSdt IGF0dHJhY3RlZCB0byB0aGUgYWJvdmUgZGVzaWduIGJlY2F1c2U6Cj4+PiAtIGl0IHdvdWxkIGxl YXZlIHRoZSBmaXJtd2FyZSBhbG9uZSBhZnRlciBPUyBib290LCBhbmQKPj4+IC0gaXQgd291bGQg bGVhdmUgdGhlIGZpcm13YXJlIGJsaXNzZnVsbHkgaWdub3JhbnQgYWJvdXQgSEVTVCwgR0hFUywK Pj4+IENQRVIsIGFuZCB0aGUgbGlrZS4gKFRoYXQncyB3aHkgUUVNVSdzIEFDUEkgbGlua2VyL2xv YWRlciB3YXMgaW52ZW50ZWQKPj4+IGluIHRoZSBmaXJzdCBwbGFjZS4pCj4+Pgo+Pj4gVGhhbmtz Cj4+PiBMYXN6bG8KPj4+Cj4+Pj4+ICAgIERvIHlvdSB0aGluayB3aGljaCBtb2R1bGVzIGdlbmVy YXRlcyB0aGUgQVBFSSB0YWJsZSBpcyBiZXR0ZXI/IFVFRkkgb3IgUWVtdT8KPj4+Pj4KPj4+Pj4K Pj4+Pj4KPj4+Pj4KPj4+Pj4gT24gMjAxNy8zLzI4IDIxOjQwLCBKYW1lcyBNb3JzZSB3cm90ZToK Pj4+Pj4+IEhpIGdlbmdkb25naml1LAo+Pj4+Pj4KPj4+Pj4+IE9uIDI4LzAzLzE3IDEzOjE2LCBn ZW5nZG9uZ2ppdSB3cm90ZToKPj4+Pj4+PiBPbiAyMDE3LzMvMjggMTk6NTQsIEFjaGluIEd1cHRh IHdyb3RlOgo+Pj4+Pj4+PiBPbiBUdWUsIE1hciAyOCwgMjAxNyBhdCAwMToyMzoyOFBNICswMjAw LCBDaHJpc3RvZmZlciBEYWxsIHdyb3RlOgo+Pj4+Pj4+Pj4gT24gVHVlLCBNYXIgMjgsIDIwMTcg YXQgMTE6NDg6MDhBTSArMDEwMCwgSmFtZXMgTW9yc2Ugd3JvdGU6Cj4+Pj4+Pj4+Pj4gT24gdGhl IGhvc3QsIHBhcnQgb2YgVUVGSSBpcyBpbnZvbHZlZCB0byBnZW5lcmF0ZSB0aGUgQ1BFUiByZWNv cmRzLgo+Pj4+Pj4+Pj4+IEluIGEgZ3Vlc3Q/LCBJIGRvbid0IGtub3cuCj4+Pj4+Pj4+Pj4gUWVt dSBjb3VsZCBnZW5lcmF0ZSB0aGUgcmVjb3Jkcywgb3IgZHJpdmUgc29tZSBvdGhlciBjb21wb25l bnQgdG8gZG8gaXQuCj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4gSSB0aGluayBJIGFtIGJlZ2lubmluZyB0 byB1bmRlcnN0YW5kIHRoaXMgYSBiaXQuICBTaW5jZSB0aGUgZ3VldCBVRUZJCj4+Pj4+Pj4+PiBp bnN0YW5jZSBpcyBzcGVjaWZpY2FsbHkgYnVpbHQgZm9yIHRoZSBtYWNoaW5lIGl0IHJ1bnMgb24s IFFFTVUncyB2aXJ0Cj4+Pj4+Pj4+PiBtYWNoaW5lIGluIHRoaXMgY2FzZSwgdGhleSBjb3VsZCBz aW1wbHkgYWdyZWUgKGJ5IHNvbWUgY29udHJhY3QpIHRvCj4+Pj4+Pj4+PiBwbGFjZSB0aGUgcmVj b3JkcyBhdCBzb21lIHNwZWNpZmljIGxvY2F0aW9uIGluIG1lbW9yeSwgYW5kIGlmIHRoZSBndWVz dAo+Pj4+Pj4+Pj4ga2VybmVsIGFza3MgaXRzIGd1ZXN0IFVFRkkgZm9yIHRoYXQgbG9jYXRpb24s IHRoaW5ncyBzaG91bGQganVzdCB3b3JrIGJ5Cj4+Pj4+Pj4+PiBoYXZpbmcgbG9naWMgaW4gUUVN VSB0byBwcm9jZXNzIGVycm9yIHJlcG9ydHMgYW5kIHBvcHVsYXRlIGd1ZXN0IG1lbW9yeS4KPj4+ Pj4+Pj4+Cj4+Pj4+Pj4+PiBJcyB0aGlzIGhvdyBvdGhlcnMgc2VlIHRoZSB3b3JsZCB0b28/Cj4+ Pj4+Pj4+Cj4+Pj4+Pj4+IEkgdGhpbmsgc28hCj4+Pj4+Pj4+Cj4+Pj4+Pj4+IEFGQUlVLCB0aGUg bWVtb3J5IHdoZXJlIENQRVJzIHdpbGwgcmVzaWRlIHNob3VsZCBiZSBzcGVjaWZpZWQgaW4gYSBH SEVTIGVudHJ5IGluCj4+Pj4+Pj4+IHRoZSBIRVNULiBJcyB0aGlzIG5vdCB0aGUgY2FzZSB3aXRo IGEgZ3Vlc3Qga2VybmVsIGkuZS4gdGhlIGd1ZXN0IFVFRkkgY3JlYXRlcyBhCj4+Pj4+Pj4+IEhF U1QgZm9yIHRoZSBndWVzdCBLZXJuZWw/Cj4+Pj4+Pj4+Cj4+Pj4+Pj4+IElmIHNvLCB0aGVuIHRo ZSBxdWVzdGlvbiBpcyBob3cgdGhlIGd1ZXN0IFVFRkkgZmluZHMgb3V0IHdoZXJlIFFFTVUgKGFj dGluZyBhcwo+Pj4+Pj4+PiBFTDMgZmlybXdhcmUpIHdpbGwgcG9wdWxhdGUgdGhlIENQRVJzLiBU aGlzIGNvdWxkIGVpdGhlciBiZSBhIGNvbnRyYWN0IGJldHdlZW4KPj4+Pj4+Pj4gdGhlIHR3byBv ciBhIGd1ZXN0IERYRSBkcml2ZXIgdXNlcyB0aGUgTU1fQ09NTVVOSUNBVEUgY2FsbCAoc2VlIFsx XSkgdG8gYXNrIFFFTVUKPj4+Pj4+Pj4gd2hlcmUgdGhlIG1lbW9yeSBpcy4KPj4+Pj4+Pgo+Pj4+ Pj4+IHdoZXRoZXIgaW52b2tlIHRoZSBndWVzdCBVRUZJIHdpbGwgYmUgY29tcGxleD8gbm90IHNl ZSB0aGUgYWR2YW50YWdlLiBpdCBzZWVtcyB4ODYgUWVtdQo+Pj4+Pj4+IGRpcmVjdGx5IGdlbmVy YXRlIHRoZSBBQ1BJIHRhYmxlLCBidXQgSSBhbSBub3Qgc3VyZSwgd2UgYXJlIGNoZWNraW5nIHRo ZSBxZW11Cj4+Pj4+PiBsb2dpY2FsLgo+Pj4+Pj4+IGxldCBRZW11IGdlbmVyYXRlIENQRVIgcmVj b3JkIG1heSBiZSBjbGVhci4KPj4+Pj4+Cj4+Pj4+PiBBdCBib290IFVFRkkgaW4gdGhlIGd1ZXN0 IHdpbGwgbmVlZCB0byBtYWtlIHN1cmUgdGhlIGFyZWFzIG9mIG1lbW9yeSB0aGF0IG1heSBiZQo+ Pj4+Pj4gdXNlZCBmb3IgQ1BFUiByZWNvcmRzIGFyZSByZXNlcnZlZC4gV2hldGhlciBVRUZJIG9y IFFlbXUgZGVjaWRlcyB3aGVyZSB0aGVzZSBhcmUKPj4+Pj4+IG5lZWRzIGRlY2lkaW5nLCAoYnV0 IHByb2JhYmx5IG5vdCBoZXJlKS4uLgo+Pj4+Pj4KPj4+Pj4+IEF0IHJ1bnRpbWUsIHdoZW4gYW4g ZXJyb3IgaGFzIG9jY3VycmVkLCBJIGFncmVlIGl0IHdvdWxkIGJlIHNpbXBsZXIgKGZld2VyCj4+ Pj4+PiBjb21wb25lbnRzIGludm9sdmVkKSBpZiBRZW11IGdlbmVyYXRlcyB0aGUgQ1BFUiByZWNv cmRzLiBCdXQgaWYgVUVGSSBtYWRlIHRoZQo+Pj4+Pj4gbWVtb3J5IGNob2ljZSBhYm92ZSB0aGV5 IG5lZWQgdG8gaW50ZXJhY3QgYW5kIGl0IGdldHMgY29tcGxpY2F0ZWQgYWdhaW4uIFRoZQo+Pj4+ Pj4gQ1BFUiByZWNvcmRzIGFyZSBkZWZpbmVkIGluIHRoZSBVRUZJIHNwZWMsIHNvIEkgd291bGQg ZXhwZWN0IFVFRkkgdG8gY29udGFpbgo+Pj4+Pj4gY29kZSB0byBnZW5lcmF0ZS9wYXJzZSB0aGVt Lgo+Pj4+Pj4KPj4+Pj4+Cj4+Pj4+PiBUaGFua3MsCj4+Pj4+Pgo+Pj4+Pj4gSmFtZXMKPj4+Pj4+ Cj4+Pj4+Pgo+Pj4+Pj4gLgo+Pj4+Pj4KPj4+Pj4KPj4+Cj4+Pgo+Pj4gLgo+Pj4KPj4KPiAKPiAK PiAuCj4gCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpl ZGsyLWRldmVsIG1haWxpbmcgbGlzdAplZGsyLWRldmVsQGxpc3RzLjAxLm9yZwpodHRwczovL2xp c3RzLjAxLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2VkazItZGV2ZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34816) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cwK1U-0002rh-LI for qemu-devel@nongnu.org; Thu, 06 Apr 2017 22:53:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cwK1R-0003yg-Es for qemu-devel@nongnu.org; Thu, 06 Apr 2017 22:53:16 -0400 Received: from [45.249.212.189] (port=2903 helo=dggrg03-dlp.huawei.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cwK1P-0003v4-WB for qemu-devel@nongnu.org; Thu, 06 Apr 2017 22:53:13 -0400 References: <76795e20-2f20-1e54-cfa5-7444f28b18ee@huawei.com> <20170321113428.GC15920@cbox> <58D17AF0.2010802@arm.com> <20170321193933.GB31111@cbox> <58DA3F68.6090901@arm.com> <20170328112328.GA31156@cbox> <20170328115413.GJ23682@e104320-lin> <58DA67BA.8070404@arm.com> <5b7352f4-4965-3ed5-3879-db871797be47@huawei.com> <20170329103658.GQ23682@e104320-lin> <2a427164-9b37-6711-3a56-906634ba7f12@redhat.com> <7c5c8ab7-8fcc-1c98-0bc1-cccb66c4c84d@huawei.com> <6ac1597a-2ed5-36b2-848d-5fd048b16d66@redhat.com> From: gengdongjiu Message-ID: <55546a4b-c33b-37b9-dafe-15ce75bc1b62@huawei.com> Date: Fri, 7 Apr 2017 10:52:10 +0800 MIME-Version: 1.0 In-Reply-To: <6ac1597a-2ed5-36b2-848d-5fd048b16d66@redhat.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Subject: Re: [Qemu-devel] [PATCH] kvm: pass the virtual SEI syndrome to guest OS List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Laszlo Ersek , Achin Gupta Cc: ard.biesheuvel@linaro.org, edk2-devel@lists.01.org, qemu-devel@nongnu.org, zhaoshenglong@huawei.com, James Morse , Christoffer Dall , xiexiuqi@huawei.com, Marc Zyngier , catalin.marinas@arm.com, will.deacon@arm.com, christoffer.dall@linaro.org, rkrcmar@redhat.com, suzuki.poulose@arm.com, andre.przywara@arm.com, mark.rutland@arm.com, vladimir.murzin@arm.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, wangxiongfeng2@huawei.com, wuquanming@huawei.com, huangshaoyu@huawei.com, Leif.Lindholm@linaro.comnd@arm.com, Michael Tsirkin , Igor Mammedov Hi Laszlo, thanks. On 2017/4/7 2:55, Laszlo Ersek wrote: > On 04/06/17 14:35, gengdongjiu wrote: >> Dear, Laszlo >> Thanks for your detailed explanation. >> >> On 2017/3/29 19:58, Laszlo Ersek wrote: >>> (This ought to be one of the longest address lists I've ever seen :) >>> Thanks for the CC. I'm glad Shannon is already on the CC list. For good >>> measure, I'm adding MST and Igor.) >>> >>> On 03/29/17 12:36, Achin Gupta wrote: >>>> Hi gengdongjiu, >>>> >>>> On Wed, Mar 29, 2017 at 05:36:37PM +0800, gengdongjiu wrote: >>>>> >>>>> Hi Laszlo/Biesheuvel/Qemu developer, >>>>> >>>>> Now I encounter a issue and want to consult with you in ARM64 platform， as described below: >>>>> >>>>> when guest OS happen synchronous or asynchronous abort, kvm needs >>>>> to send the error address to Qemu or UEFI through sigbus to >>>>> dynamically generate APEI table. from my investigation, there are >>>>> two ways: >>>>> >>>>> (1) Qemu get the error address, and generate the APEI table, then >>>>> notify UEFI to know this generation, then inject abort error to >>>>> guest OS, guest OS read the APEI table. >>>>> (2) Qemu get the error address, and let UEFI to generate the APEI >>>>> table, then inject abort error to guest OS, guest OS read the APEI >>>>> table. >>>> >>>> Just being pedantic! I don't think we are talking about creating the APEI table >>>> dynamically here. The issue is: Once KVM has received an error that is destined >>>> for a guest it will raise a SIGBUS to Qemu. Now before Qemu can inject the error >>>> into the guest OS, a CPER (Common Platform Error Record) has to be generated >>>> corresponding to the error source (GHES corresponding to memory subsystem, >>>> processor etc) to allow the guest OS to do anything meaningful with the >>>> error. So who should create the CPER is the question. >>>> >>>> At the EL3/EL2 interface (Secure Firmware and OS/Hypervisor), an error arrives >>>> at EL3 and secure firmware (at EL3 or a lower secure exception level) is >>>> responsible for creating the CPER. ARM is experimenting with using a Standalone >>>> MM EDK2 image in the secure world to do the CPER creation. This will avoid >>>> adding the same code in ARM TF in EL3 (better for security). The error will then >>>> be injected into the OS/Hypervisor (through SEA/SEI/SDEI) through ARM Trusted >>>> Firmware. >>>> >>>> Qemu is essentially fulfilling the role of secure firmware at the EL2/EL1 >>>> interface (as discussed with Christoffer below). So it should generate the CPER >>>> before injecting the error. >>>> >>>> This is corresponds to (1) above apart from notifying UEFI (I am assuming you >>>> mean guest UEFI). At this time, the guest OS already knows where to pick up the >>>> CPER from through the HEST. Qemu has to create the CPER and populate its address >>>> at the address exported in the HEST. Guest UEFI should not be involved in this >>>> flow. Its job was to create the HEST at boot and that has been done by this >>>> stage. >>>> >>>> Qemu folk will be able to add but it looks like support for CPER generation will >>>> need to be added to Qemu. We need to resolve this. >>>> >>>> Do shout if I am missing anything above. >>> >>> After reading this email, the use case looks *very* similar to what >>> we've just done with VMGENID for QEMU 2.9. >>> >>> We have a facility between QEMU and the guest firmware, called "ACPI >>> linker/loader", with which QEMU instructs the firmware to >>> >>> - allocate and download blobs into guest RAM (AcpiNVS type memory) -- >>> ALLOCATE command, >>> >>> - relocate pointers in those blobs, to fields in other (or the same) >>> blobs -- ADD_POINTER command, >>> >>> - set ACPI table checksums -- ADD_CHECKSUM command, >>> >>> - and send GPAs of fields within such blobs back to QEMU -- >>> WRITE_POINTER command. >>> >>> This is how I imagine we can map the facility to the current use case >>> (note that this is the first time I read about HEST / GHES / CPER): >>> >>> etc/acpi/tables etc/hardware_errors >>> ================ ========================================== >>> +-----------+ >>> +--------------+ | address | +-> +--------------+ >>> | HEST + | registers | | | Error Status | >>> + +------------+ | +---------+ | | Data Block 1 | >>> | | GHES | --> | | address | --------+ | +------------+ >>> | | GHES | --> | | address | ------+ | | CPER | >>> | | GHES | --> | | address | ----+ | | | CPER | >>> | | GHES | --> | | address | -+ | | | | CPER | >>> +-+------------+ +-+---------+ | | | +-+------------+ >>> | | | >>> | | +---> +--------------+ >>> | | | Error Status | >>> | | | Data Block 2 | >>> | | | +------------+ >>> | | | | CPER | >>> | | | | CPER | >>> | | +-+------------+ >>> | | >>> | +-----> +--------------+ >>> | | Error Status | >>> | | Data Block 3 | >>> | | +------------+ >>> | | | CPER | >>> | +-+------------+ >>> | >>> +--------> +--------------+ >>> | Error Status | >>> | Data Block 4 | >>> | +------------+ >>> | | CPER | >>> | | CPER | >>> | | CPER | >>> +-+------------+ >>> >>> (1) QEMU generates the HEST ACPI table. This table goes in the current >>> "etc/acpi/tables" fw_cfg blob. Given N error sources, there will be N >>> GHES objects in the HEST. >>> >>> (2) We introduce a new fw_cfg blob called "etc/hardware_errors". QEMU >>> also populates this blob. >>> >>> (2a) Given N error sources, the (unnamed) table of address registers >>> will contain N address registers. >>> >>> (2b) Given N error sources, the "etc/hardwre_errors" fw_cfg blob will >>> also contain N Error Status Data Blocks. >>> >>> I don't know about the sizing (number of CPERs) each Error Status Data >>> Block has to contain, but I understand it is all pre-allocated as far as >>> the OS is concerned, which matches our capabilities well. >> here I have a question. as you comment: " 'etc/hardwre_errors' fw_cfg blob will also contain N Error Status Data Blocks", >> Because the CPER numbers is not fixed, how to assign each "Error Status Data Block" size using one "etc/hardwre_errors" fw_cfg blob. >> when use one etc/hardwre_errors, will the N Error Status Data Block use one continuous buffer? as shown below. if so, maybe it not convenient for each data block size extension. >> I see the bios_linker_loader_alloc will allocate one continuous buffer for a blob(such as VMGENID_GUID_FW_CFG_FILE) >> >> /* Allocate guest memory for the Data fw_cfg blob */ >> bios_linker_loader_alloc(linker, VMGENID_GUID_FW_CFG_FILE, guid, 4096, >> false /* page boundary, high memory */); >> >> >> >> -> +--------------+ >> | HEST + | registers | | Error Status | >> + +------------+ | +---------+ | Data Block | >> | | GHES | --> | | address | --------+-->| +------------+ >> | | GHES | --> | | address | ------+ | | CPER | >> | | GHES | --> | | address | ----+ | | | CPER | >> | | GHES | --> | | address | -+ | | | | CPER | >> +-+------------+ +-+---------+ | | +---> +--------------+ >> | | | | CPER | >> | | | | CPER | >> | +-----> +--------------+ >> | | | CPER | >> +--------> +--------------+ >> | | CPER | >> | | CPER | >> | | CPER | >> +-+------------+ >> >> >> >> so how about we use separate etc/hardwre_errorsN for each Error Status status Block? then >> >> etc/hardwre_errors0 >> etc/hardwre_errors1 >> ................... >> etc/hardwre_errors10 >> (the max N is 10) >> >> >> the N can be one of below values, according to ACPI spec "Table 18-345 Hardware Error Notification Structure" >> 0 – Polled >> 1 – External Interrupt >> 2 – Local Interrupt >> 3 – SCI >> 4 – NMI >> 5 - CMCI >> 6 - MCE >> 7 - GPIO-Signal >> 8 - ARMv8 SEA >> 9 - ARMv8 SEI >> 10 - External Interrupt - GSIV > > I'm unsure if, by "not fixed", you are saying > > the number of CPER entries that fits in Error Status Data Block N is > not *uniform* across 0 <= N <= 10 [1] > > or > > the number of CPER entries that fits in Error Status Data Block N is > not *known* in advance, for all of 0 <= N <= 10 [2] > > Which one is your point? > > If [1], that's no problem; you can simply sum the individual error > status data block sizes in advance, and allocate "etc/hardware_errors" > accordingly, using the total size. > > (Allocating one shared fw_cfg blob for all status data blocks is more > memory efficient, as each ALLOCATE command will allocate whole pages > (rounded up from the actual blob size).) > > If your point is [2], then splitting the error status data blocks to > separate fw_cfg blobs makes no difference: regardless of whether we try > to place all the error status data blocks in a single fw_cfg blob, or in > separate fw_cfg blobs, the individual data block cannot be resized at OS > runtime, so there's no way to make it work. > My Point is [2]. The HEST(Hardware Error Source Table) table format is here: https://wiki.linaro.org/LEG/Engineering/Kernel/RAS/APEITables#Hardware_Error_Source_Table_.28HEST.29 Now I understand your thought. > Thanks, > Laszlo > >> >> >> >> >>> >>> (3) QEMU generates the ACPI linker/loader script for the firmware, as >>> always. >>> >>> (3a) The HEST table is part of "etc/acpi/tables", which the firmware >>> already allocates memory for, and downloads (because QEMU already >>> generates an ALLOCATE linker/loader command for it already). >>> >>> (3b) QEMU will have to create another ALLOCATE command for the >>> "etc/hardware_errors" blob. The firmware allocates memory for this blob, >>> and downloads it. >>> >>> (4) QEMU generates, in the ACPI linker/loader script for the firwmare, N >>> ADD_POINTER commands, which point the GHES."Error Status >>> Address" fields in the HEST table, to the corresponding address >>> registers in the downloaded "etc/hardware_errors" blob. >>> >>> (5) QEMU generates an ADD_CHECKSUM command for the firmware, so that the >>> HEST table is correctly checksummed after executing the N ADD_POINTER >>> commands from (4). >>> >>> (6) QEMU generates N ADD_POINTER commands for the firmware, pointing the >>> address registers (located in guest memory, in the downloaded >>> "etc/hardware_errors" blob) to the respective Error Status Data Blocks. >>> >>> (7) (This is the trick.) For this step, we need a third, write-only >>> fw_cfg blob, called "etc/hardware_errors_addr". Through that blob, the >>> firmware can send back the guest-side allocation addresses to QEMU. >>> >>> Namely, the "etc/hardware_errors_addr" blob contains N 8-byte entries. >>> QEMU generates N WRITE_POINTER commands for the firmware. >>> >>> For error source K (0 <= K < N), QEMU instructs the firmware to >>> calculate the guest address of Error Status Data Block K, from the >>> QEMU-dictated offset within "etc/hardware_errors", and from the >>> guest-determined allocation base address for "etc/hardware_errors". The >>> firmware then writes the calculated address back to fw_cfg file >>> "etc/hardware_errors_addr", at offset K*8, according to the >>> WRITE_POINTER command. >>> >>> This way QEMU will know the GPA of each Error Status Data Block. >>> >>> (In fact this can be simplified to a single WRITE_POINTER command: the >>> address of the "address register table" can be sent back to QEMU as >>> well, which already contains all Error Status Data Block addresses.) >>> >>> (8) When QEMU gets SIGBUS from the kernel -- I hope that's going to come >>> through a signalfd -- QEMU can format the CPER right into guest memory, >>> and then inject whatever interrupt (or assert whatever GPIO line) is >>> necessary for notifying the guest. >>> >>> (9) This notification (in virtual hardware) can either be handled by the >>> guest kernel stand-alone, or else the guest kernel can invoke an ACPI >>> event handler method with it (which would be in the DSDT or one of the >>> SSDTs, also generated by QEMU). The ACPI event handler method could >>> invoke the specific guest kernel driver for errror handling via a >>> Notify() operation. >>> >>> I'm attracted to the above design because: >>> - it would leave the firmware alone after OS boot, and >>> - it would leave the firmware blissfully ignorant about HEST, GHES, >>> CPER, and the like. (That's why QEMU's ACPI linker/loader was invented >>> in the first place.) >>> >>> Thanks >>> Laszlo >>> >>>>> Do you think which modules generates the APEI table is better? UEFI or Qemu? >>>>> >>>>> >>>>> >>>>> >>>>> On 2017/3/28 21:40, James Morse wrote: >>>>>> Hi gengdongjiu, >>>>>> >>>>>> On 28/03/17 13:16, gengdongjiu wrote: >>>>>>> On 2017/3/28 19:54, Achin Gupta wrote: >>>>>>>> On Tue, Mar 28, 2017 at 01:23:28PM +0200, Christoffer Dall wrote: >>>>>>>>> On Tue, Mar 28, 2017 at 11:48:08AM +0100, James Morse wrote: >>>>>>>>>> On the host, part of UEFI is involved to generate the CPER records. >>>>>>>>>> In a guest?, I don't know. >>>>>>>>>> Qemu could generate the records, or drive some other component to do it. >>>>>>>>> >>>>>>>>> I think I am beginning to understand this a bit. Since the guet UEFI >>>>>>>>> instance is specifically built for the machine it runs on, QEMU's virt >>>>>>>>> machine in this case, they could simply agree (by some contract) to >>>>>>>>> place the records at some specific location in memory, and if the guest >>>>>>>>> kernel asks its guest UEFI for that location, things should just work by >>>>>>>>> having logic in QEMU to process error reports and populate guest memory. >>>>>>>>> >>>>>>>>> Is this how others see the world too? >>>>>>>> >>>>>>>> I think so! >>>>>>>> >>>>>>>> AFAIU, the memory where CPERs will reside should be specified in a GHES entry in >>>>>>>> the HEST. Is this not the case with a guest kernel i.e. the guest UEFI creates a >>>>>>>> HEST for the guest Kernel? >>>>>>>> >>>>>>>> If so, then the question is how the guest UEFI finds out where QEMU (acting as >>>>>>>> EL3 firmware) will populate the CPERs. This could either be a contract between >>>>>>>> the two or a guest DXE driver uses the MM_COMMUNICATE call (see [1]) to ask QEMU >>>>>>>> where the memory is. >>>>>>> >>>>>>> whether invoke the guest UEFI will be complex? not see the advantage. it seems x86 Qemu >>>>>>> directly generate the ACPI table, but I am not sure, we are checking the qemu >>>>>> logical. >>>>>>> let Qemu generate CPER record may be clear. >>>>>> >>>>>> At boot UEFI in the guest will need to make sure the areas of memory that may be >>>>>> used for CPER records are reserved. Whether UEFI or Qemu decides where these are >>>>>> needs deciding, (but probably not here)... >>>>>> >>>>>> At runtime, when an error has occurred, I agree it would be simpler (fewer >>>>>> components involved) if Qemu generates the CPER records. But if UEFI made the >>>>>> memory choice above they need to interact and it gets complicated again. The >>>>>> CPER records are defined in the UEFI spec, so I would expect UEFI to contain >>>>>> code to generate/parse them. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> James >>>>>> >>>>>> >>>>>> . >>>>>> >>>>> >>> >>> >>> . >>> >> > > > . > From mboxrd@z Thu Jan 1 00:00:00 1970 From: gengdongjiu@huawei.com (gengdongjiu) Date: Fri, 7 Apr 2017 10:52:10 +0800 Subject: [PATCH] kvm: pass the virtual SEI syndrome to guest OS In-Reply-To: <6ac1597a-2ed5-36b2-848d-5fd048b16d66@redhat.com> References: <76795e20-2f20-1e54-cfa5-7444f28b18ee@huawei.com> <20170321113428.GC15920@cbox> <58D17AF0.2010802@arm.com> <20170321193933.GB31111@cbox> <58DA3F68.6090901@arm.com> <20170328112328.GA31156@cbox> <20170328115413.GJ23682@e104320-lin> <58DA67BA.8070404@arm.com> <5b7352f4-4965-3ed5-3879-db871797be47@huawei.com> <20170329103658.GQ23682@e104320-lin> <2a427164-9b37-6711-3a56-906634ba7f12@redhat.com> <7c5c8ab7-8fcc-1c98-0bc1-cccb66c4c84d@huawei.com> <6ac1597a-2ed5-36b2-848d-5fd048b16d66@redhat.com> Message-ID: <55546a4b-c33b-37b9-dafe-15ce75bc1b62@huawei.com> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org Hi Laszlo, thanks. On 2017/4/7 2:55, Laszlo Ersek wrote: > On 04/06/17 14:35, gengdongjiu wrote: >> Dear, Laszlo >> Thanks for your detailed explanation. >> >> On 2017/3/29 19:58, Laszlo Ersek wrote: >>> (This ought to be one of the longest address lists I've ever seen :) >>> Thanks for the CC. I'm glad Shannon is already on the CC list. For good >>> measure, I'm adding MST and Igor.) >>> >>> On 03/29/17 12:36, Achin Gupta wrote: >>>> Hi gengdongjiu, >>>> >>>> On Wed, Mar 29, 2017 at 05:36:37PM +0800, gengdongjiu wrote: >>>>> >>>>> Hi Laszlo/Biesheuvel/Qemu developer, >>>>> >>>>> Now I encounter a issue and want to consult with you in ARM64 platform? as described below: >>>>> >>>>> when guest OS happen synchronous or asynchronous abort, kvm needs >>>>> to send the error address to Qemu or UEFI through sigbus to >>>>> dynamically generate APEI table. from my investigation, there are >>>>> two ways: >>>>> >>>>> (1) Qemu get the error address, and generate the APEI table, then >>>>> notify UEFI to know this generation, then inject abort error to >>>>> guest OS, guest OS read the APEI table. >>>>> (2) Qemu get the error address, and let UEFI to generate the APEI >>>>> table, then inject abort error to guest OS, guest OS read the APEI >>>>> table. >>>> >>>> Just being pedantic! I don't think we are talking about creating the APEI table >>>> dynamically here. The issue is: Once KVM has received an error that is destined >>>> for a guest it will raise a SIGBUS to Qemu. Now before Qemu can inject the error >>>> into the guest OS, a CPER (Common Platform Error Record) has to be generated >>>> corresponding to the error source (GHES corresponding to memory subsystem, >>>> processor etc) to allow the guest OS to do anything meaningful with the >>>> error. So who should create the CPER is the question. >>>> >>>> At the EL3/EL2 interface (Secure Firmware and OS/Hypervisor), an error arrives >>>> at EL3 and secure firmware (at EL3 or a lower secure exception level) is >>>> responsible for creating the CPER. ARM is experimenting with using a Standalone >>>> MM EDK2 image in the secure world to do the CPER creation. This will avoid >>>> adding the same code in ARM TF in EL3 (better for security). The error will then >>>> be injected into the OS/Hypervisor (through SEA/SEI/SDEI) through ARM Trusted >>>> Firmware. >>>> >>>> Qemu is essentially fulfilling the role of secure firmware at the EL2/EL1 >>>> interface (as discussed with Christoffer below). So it should generate the CPER >>>> before injecting the error. >>>> >>>> This is corresponds to (1) above apart from notifying UEFI (I am assuming you >>>> mean guest UEFI). At this time, the guest OS already knows where to pick up the >>>> CPER from through the HEST. Qemu has to create the CPER and populate its address >>>> at the address exported in the HEST. Guest UEFI should not be involved in this >>>> flow. Its job was to create the HEST at boot and that has been done by this >>>> stage. >>>> >>>> Qemu folk will be able to add but it looks like support for CPER generation will >>>> need to be added to Qemu. We need to resolve this. >>>> >>>> Do shout if I am missing anything above. >>> >>> After reading this email, the use case looks *very* similar to what >>> we've just done with VMGENID for QEMU 2.9. >>> >>> We have a facility between QEMU and the guest firmware, called "ACPI >>> linker/loader", with which QEMU instructs the firmware to >>> >>> - allocate and download blobs into guest RAM (AcpiNVS type memory) -- >>> ALLOCATE command, >>> >>> - relocate pointers in those blobs, to fields in other (or the same) >>> blobs -- ADD_POINTER command, >>> >>> - set ACPI table checksums -- ADD_CHECKSUM command, >>> >>> - and send GPAs of fields within such blobs back to QEMU -- >>> WRITE_POINTER command. >>> >>> This is how I imagine we can map the facility to the current use case >>> (note that this is the first time I read about HEST / GHES / CPER): >>> >>> etc/acpi/tables etc/hardware_errors >>> ================ ========================================== >>> +-----------+ >>> +--------------+ | address | +-> +--------------+ >>> | HEST + | registers | | | Error Status | >>> + +------------+ | +---------+ | | Data Block 1 | >>> | | GHES | --> | | address | --------+ | +------------+ >>> | | GHES | --> | | address | ------+ | | CPER | >>> | | GHES | --> | | address | ----+ | | | CPER | >>> | | GHES | --> | | address | -+ | | | | CPER | >>> +-+------------+ +-+---------+ | | | +-+------------+ >>> | | | >>> | | +---> +--------------+ >>> | | | Error Status | >>> | | | Data Block 2 | >>> | | | +------------+ >>> | | | | CPER | >>> | | | | CPER | >>> | | +-+------------+ >>> | | >>> | +-----> +--------------+ >>> | | Error Status | >>> | | Data Block 3 | >>> | | +------------+ >>> | | | CPER | >>> | +-+------------+ >>> | >>> +--------> +--------------+ >>> | Error Status | >>> | Data Block 4 | >>> | +------------+ >>> | | CPER | >>> | | CPER | >>> | | CPER | >>> +-+------------+ >>> >>> (1) QEMU generates the HEST ACPI table. This table goes in the current >>> "etc/acpi/tables" fw_cfg blob. Given N error sources, there will be N >>> GHES objects in the HEST. >>> >>> (2) We introduce a new fw_cfg blob called "etc/hardware_errors". QEMU >>> also populates this blob. >>> >>> (2a) Given N error sources, the (unnamed) table of address registers >>> will contain N address registers. >>> >>> (2b) Given N error sources, the "etc/hardwre_errors" fw_cfg blob will >>> also contain N Error Status Data Blocks. >>> >>> I don't know about the sizing (number of CPERs) each Error Status Data >>> Block has to contain, but I understand it is all pre-allocated as far as >>> the OS is concerned, which matches our capabilities well. >> here I have a question. as you comment: " 'etc/hardwre_errors' fw_cfg blob will also contain N Error Status Data Blocks", >> Because the CPER numbers is not fixed, how to assign each "Error Status Data Block" size using one "etc/hardwre_errors" fw_cfg blob. >> when use one etc/hardwre_errors, will the N Error Status Data Block use one continuous buffer? as shown below. if so, maybe it not convenient for each data block size extension. >> I see the bios_linker_loader_alloc will allocate one continuous buffer for a blob(such as VMGENID_GUID_FW_CFG_FILE) >> >> /* Allocate guest memory for the Data fw_cfg blob */ >> bios_linker_loader_alloc(linker, VMGENID_GUID_FW_CFG_FILE, guid, 4096, >> false /* page boundary, high memory */); >> >> >> >> -> +--------------+ >> | HEST + | registers | | Error Status | >> + +------------+ | +---------+ | Data Block | >> | | GHES | --> | | address | --------+-->| +------------+ >> | | GHES | --> | | address | ------+ | | CPER | >> | | GHES | --> | | address | ----+ | | | CPER | >> | | GHES | --> | | address | -+ | | | | CPER | >> +-+------------+ +-+---------+ | | +---> +--------------+ >> | | | | CPER | >> | | | | CPER | >> | +-----> +--------------+ >> | | | CPER | >> +--------> +--------------+ >> | | CPER | >> | | CPER | >> | | CPER | >> +-+------------+ >> >> >> >> so how about we use separate etc/hardwre_errorsN for each Error Status status Block? then >> >> etc/hardwre_errors0 >> etc/hardwre_errors1 >> ................... >> etc/hardwre_errors10 >> (the max N is 10) >> >> >> the N can be one of below values, according to ACPI spec "Table 18-345 Hardware Error Notification Structure" >> 0 ? Polled >> 1 ? External Interrupt >> 2 ? Local Interrupt >> 3 ? SCI >> 4 ? NMI >> 5 - CMCI >> 6 - MCE >> 7 - GPIO-Signal >> 8 - ARMv8 SEA >> 9 - ARMv8 SEI >> 10 - External Interrupt - GSIV > > I'm unsure if, by "not fixed", you are saying > > the number of CPER entries that fits in Error Status Data Block N is > not *uniform* across 0 <= N <= 10 [1] > > or > > the number of CPER entries that fits in Error Status Data Block N is > not *known* in advance, for all of 0 <= N <= 10 [2] > > Which one is your point? > > If [1], that's no problem; you can simply sum the individual error > status data block sizes in advance, and allocate "etc/hardware_errors" > accordingly, using the total size. > > (Allocating one shared fw_cfg blob for all status data blocks is more > memory efficient, as each ALLOCATE command will allocate whole pages > (rounded up from the actual blob size).) > > If your point is [2], then splitting the error status data blocks to > separate fw_cfg blobs makes no difference: regardless of whether we try > to place all the error status data blocks in a single fw_cfg blob, or in > separate fw_cfg blobs, the individual data block cannot be resized at OS > runtime, so there's no way to make it work. > My Point is [2]. The HEST(Hardware Error Source Table) table format is here: https://wiki.linaro.org/LEG/Engineering/Kernel/RAS/APEITables#Hardware_Error_Source_Table_.28HEST.29 Now I understand your thought. > Thanks, > Laszlo > >> >> >> >> >>> >>> (3) QEMU generates the ACPI linker/loader script for the firmware, as >>> always. >>> >>> (3a) The HEST table is part of "etc/acpi/tables", which the firmware >>> already allocates memory for, and downloads (because QEMU already >>> generates an ALLOCATE linker/loader command for it already). >>> >>> (3b) QEMU will have to create another ALLOCATE command for the >>> "etc/hardware_errors" blob. The firmware allocates memory for this blob, >>> and downloads it. >>> >>> (4) QEMU generates, in the ACPI linker/loader script for the firwmare, N >>> ADD_POINTER commands, which point the GHES."Error Status >>> Address" fields in the HEST table, to the corresponding address >>> registers in the downloaded "etc/hardware_errors" blob. >>> >>> (5) QEMU generates an ADD_CHECKSUM command for the firmware, so that the >>> HEST table is correctly checksummed after executing the N ADD_POINTER >>> commands from (4). >>> >>> (6) QEMU generates N ADD_POINTER commands for the firmware, pointing the >>> address registers (located in guest memory, in the downloaded >>> "etc/hardware_errors" blob) to the respective Error Status Data Blocks. >>> >>> (7) (This is the trick.) For this step, we need a third, write-only >>> fw_cfg blob, called "etc/hardware_errors_addr". Through that blob, the >>> firmware can send back the guest-side allocation addresses to QEMU. >>> >>> Namely, the "etc/hardware_errors_addr" blob contains N 8-byte entries. >>> QEMU generates N WRITE_POINTER commands for the firmware. >>> >>> For error source K (0 <= K < N), QEMU instructs the firmware to >>> calculate the guest address of Error Status Data Block K, from the >>> QEMU-dictated offset within "etc/hardware_errors", and from the >>> guest-determined allocation base address for "etc/hardware_errors". The >>> firmware then writes the calculated address back to fw_cfg file >>> "etc/hardware_errors_addr", at offset K*8, according to the >>> WRITE_POINTER command. >>> >>> This way QEMU will know the GPA of each Error Status Data Block. >>> >>> (In fact this can be simplified to a single WRITE_POINTER command: the >>> address of the "address register table" can be sent back to QEMU as >>> well, which already contains all Error Status Data Block addresses.) >>> >>> (8) When QEMU gets SIGBUS from the kernel -- I hope that's going to come >>> through a signalfd -- QEMU can format the CPER right into guest memory, >>> and then inject whatever interrupt (or assert whatever GPIO line) is >>> necessary for notifying the guest. >>> >>> (9) This notification (in virtual hardware) can either be handled by the >>> guest kernel stand-alone, or else the guest kernel can invoke an ACPI >>> event handler method with it (which would be in the DSDT or one of the >>> SSDTs, also generated by QEMU). The ACPI event handler method could >>> invoke the specific guest kernel driver for errror handling via a >>> Notify() operation. >>> >>> I'm attracted to the above design because: >>> - it would leave the firmware alone after OS boot, and >>> - it would leave the firmware blissfully ignorant about HEST, GHES, >>> CPER, and the like. (That's why QEMU's ACPI linker/loader was invented >>> in the first place.) >>> >>> Thanks >>> Laszlo >>> >>>>> Do you think which modules generates the APEI table is better? UEFI or Qemu? >>>>> >>>>> >>>>> >>>>> >>>>> On 2017/3/28 21:40, James Morse wrote: >>>>>> Hi gengdongjiu, >>>>>> >>>>>> On 28/03/17 13:16, gengdongjiu wrote: >>>>>>> On 2017/3/28 19:54, Achin Gupta wrote: >>>>>>>> On Tue, Mar 28, 2017 at 01:23:28PM +0200, Christoffer Dall wrote: >>>>>>>>> On Tue, Mar 28, 2017 at 11:48:08AM +0100, James Morse wrote: >>>>>>>>>> On the host, part of UEFI is involved to generate the CPER records. >>>>>>>>>> In a guest?, I don't know. >>>>>>>>>> Qemu could generate the records, or drive some other component to do it. >>>>>>>>> >>>>>>>>> I think I am beginning to understand this a bit. Since the guet UEFI >>>>>>>>> instance is specifically built for the machine it runs on, QEMU's virt >>>>>>>>> machine in this case, they could simply agree (by some contract) to >>>>>>>>> place the records at some specific location in memory, and if the guest >>>>>>>>> kernel asks its guest UEFI for that location, things should just work by >>>>>>>>> having logic in QEMU to process error reports and populate guest memory. >>>>>>>>> >>>>>>>>> Is this how others see the world too? >>>>>>>> >>>>>>>> I think so! >>>>>>>> >>>>>>>> AFAIU, the memory where CPERs will reside should be specified in a GHES entry in >>>>>>>> the HEST. Is this not the case with a guest kernel i.e. the guest UEFI creates a >>>>>>>> HEST for the guest Kernel? >>>>>>>> >>>>>>>> If so, then the question is how the guest UEFI finds out where QEMU (acting as >>>>>>>> EL3 firmware) will populate the CPERs. This could either be a contract between >>>>>>>> the two or a guest DXE driver uses the MM_COMMUNICATE call (see [1]) to ask QEMU >>>>>>>> where the memory is. >>>>>>> >>>>>>> whether invoke the guest UEFI will be complex? not see the advantage. it seems x86 Qemu >>>>>>> directly generate the ACPI table, but I am not sure, we are checking the qemu >>>>>> logical. >>>>>>> let Qemu generate CPER record may be clear. >>>>>> >>>>>> At boot UEFI in the guest will need to make sure the areas of memory that may be >>>>>> used for CPER records are reserved. Whether UEFI or Qemu decides where these are >>>>>> needs deciding, (but probably not here)... >>>>>> >>>>>> At runtime, when an error has occurred, I agree it would be simpler (fewer >>>>>> components involved) if Qemu generates the CPER records. But if UEFI made the >>>>>> memory choice above they need to interact and it gets complicated again. The >>>>>> CPER records are defined in the UEFI spec, so I would expect UEFI to contain >>>>>> code to generate/parse them. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> James >>>>>> >>>>>> >>>>>> . >>>>>> >>>>> >>> >>> >>> . >>> >> > > > . >