From mboxrd@z Thu Jan 1 00:00:00 1970 From: Evgenii Shatokhin Subject: Re: [BUG] kernel BUG at drivers/block/xen-blkfront.c:1711 Date: Thu, 11 Aug 2016 10:45:07 +0300 Message-ID: <57AC2D03.1010409@virtuozzo.com> References: <57553355.8050302@virtuozzo.com> <1465202552.15816.70.camel@citrix.com> <57835DEF.1090207@virtuozzo.com> <57838B41.5020309@oracle.com> <57877C49.5000503@virtuozzo.com> <57877FCF.1040105@oracle.com> <57AB1F2B.5080403@virtuozzo.com> <57AB22E4.1040208@oracle.com> <57AB4037.7050303@virtuozzo.com> <57ABDE82.6040102@oracle.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------040809000904080307080704" Return-path: In-Reply-To: <57ABDE82.6040102@oracle.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Bob Liu Cc: Juergen Gross , Dario Faggioli , George Dunlap , xen-devel@lists.xen.org, David Vrabel , Konstantin Khorenko , Roger Pau Monne List-Id: xen-devel@lists.xenproject.org --------------040809000904080307080704 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit On 11.08.2016 05:10, Bob Liu wrote: > > On 08/10/2016 10:54 PM, Evgenii Shatokhin wrote: >> On 10.08.2016 15:49, Bob Liu wrote: >>> >>> On 08/10/2016 08:33 PM, Evgenii Shatokhin wrote: >>>> On 14.07.2016 15:04, Bob Liu wrote: >>>>> >>>>> On 07/14/2016 07:49 PM, Evgenii Shatokhin wrote: >>>>>> On 11.07.2016 15:04, Bob Liu wrote: >>>>>>> >>>>>>> >>>>>>> On 07/11/2016 04:50 PM, Evgenii Shatokhin wrote: >>>>>>>> On 06.06.2016 11:42, Dario Faggioli wrote: >>>>>>>>> Just Cc-ing some Linux, block, and Xen on CentOS people... >>>>>>>>> >>>>>>>> >>>>>>>> Ping. >>>>>>>> >>>>>>>> Any suggestions how to debug this or what might cause the problem? >>>>>>>> >>>>>>>> Obviously, we cannot control Xen on the Amazon's servers. But perhaps there is something we can do at the kernel's side, is it? >>>>>>>> >>>>>>>>> On Mon, 2016-06-06 at 11:24 +0300, Evgenii Shatokhin wrote: >>>>>>>>>> (Resending this bug report because the message I sent last week did >>>>>>>>>> not >>>>>>>>>> make it to the mailing list somehow.) >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> One of our users gets kernel panics from time to time when he tries >>>>>>>>>> to >>>>>>>>>> use his Amazon EC2 instance with CentOS7 x64 in it [1]. Kernel panic >>>>>>>>>> happens within minutes from the moment the instance starts. The >>>>>>>>>> problem >>>>>>>>>> does not show up every time, however. >>>>>>>>>> >>>>>>>>>> The user first observed the problem with a custom kernel, but it was >>>>>>>>>> found later that the stock kernel 3.10.0-327.18.2.el7.x86_64 from >>>>>>>>>> CentOS7 was affected as well. >>>>>>> >>>>>>> Please try this patch: >>>>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=7b0767502b5db11cb1f0daef2d01f6d71b1192dc >>>>>>> >>>>>>> Regards, >>>>>>> Bob >>>>>>> >>>>>> >>>>>> Unfortunately, it did not help. The same BUG_ON() in blkfront_setup_indirect() still triggers in our kernel based on RHEL's 3.10.0-327.18.2, where I added the patch. >>>>>> >>>>>> As far as I can see, the patch makes sure the indirect pages are added to the list only if (!info->feature_persistent) holds. I suppose it holds in our case and the pages are added to the list because the triggered BUG_ON() is here: >>>>>> >>>>>> if (!info->feature_persistent && info->max_indirect_segments) { >>>>>> <...> >>>>>> BUG_ON(!list_empty(&info->indirect_pages)); >>>>>> <...> >>>>>> } >>>>>> >>>>> >>>>> That's odd. >>>>> Could you please try to reproduce this issue with a recent upstream kernel? >>>>> >>>>> Thanks, >>>>> Bob >>>> >>>> No luck with the upstream kernel 4.7.0 so far due to unrelated issues (bad initrd, I suppose, so the system does not even boot). >>>> >>>> However, the problem reproduced with the stable upstream kernel 3.14.74. After the system booted the second time with this kernel, that BUG_ON triggered: >>>> kernel BUG at drivers/block/xen-blkfront.c:1701 >>>> >>> >>> Could you please provide more detail on how to reproduce this bug? I'd like to have a test. >>> >>> Thanks! >>> Bob >> >> As the user says, he uses an Amazon EC2 instance. Namely: HVM CentOS7 AMI on a c3.large instance with EBS magnetic storage. >> > > Oh, then it would be difficult to debug this issue. > The xen-blkfront communicates with xen-blkback(in dom0 or driver domain), but that part is a black box when running Amazon EC2. > We can't see the source code of the backend side! Yes, and another problem is, I am still unable to reproduce the issue in my EC2 instance. However, the problem shows up rather often in the user's instance. > > Can this bug be reproduced on your own environment(xen + dom0)? I haven't tried this yet. > >> At least 2 LVM partitions are needed: >> * /, 20-30 Gb should be enough, ext4 >> * /vz, 5-10 Gb should be enough, ext4 >> >> Kernel 3.14.74 I was talking about: https://www.dropbox.com/s/bhus3mubza87z86/kernel-3.14.74-1.test.x86_64.rpm?dl=1 >> >> Not sure if it is relevant, but the user may have installed additional packages from https://download.openvz.org/virtuozzo/releases/7.0-rtm/x86_64/os/ repository. Namely: vzctl, vzmigrate, vzprocps, vztt-lib, vzctcalc, ploop, prlctl, centos-7-x86_64-ez. >> >> After the kernel and the other mentioned packages have been installed, >> the user rebooted the instance to run that kernel 3.14.74. >> >> Then - start the instance, wait 5 minutes, stop the instance, repeat. 2-20 such iterations were usually enough to reproduce the problem. Can be automated with the help of Amazon's API. >> >> BTW, before the BUG_ON triggered this time, there was the following in dmesg. Not sure if it is related but still: >> > > Attach the full dmesg would be better. Well, there is not much in the part the user was able to retrieve besides what I have sent and the BUG_ON() splat. But here it is, anyway. Regards, Evgenii > > Regards, > Bob > >> ---------------------- >> [ 2.835034] scsi0 : ata_piix >> [ 2.840317] scsi1 : ata_piix >> [ 2.842267] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc100 irq 14 >> [ 2.845861] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc108 irq 15 >> [ 2.853840] AVX version of gcm_enc/dec engaged. >> [ 2.859963] xen_netfront: Initialising Xen virtual ethernet driver >> [ 2.867156] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) >> [ 2.885861] blkfront: xvda: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled; >> [ 2.889046] alg: No test for crc32 (crc32-pclmul) >> [ 2.899290] xvda: xvda1 >> [ 2.997751] blkfront: xvdc: flush diskcache: enabled; persistent grants: disabled; indirect descriptors: enabled; >> [ 3.007401] xvdc: unknown partition table >> [ 3.010465] Setting capacity to 31992832 >> [ 3.012922] xvdc: detected capacity change from 0 to 16380329984 >> [ 3.017408] blkfront: xvdd: flush diskcache: enabled; persistent grants: disabled; indirect descriptors: enabled; >> [ 3.023861] xvdd: unknown partition table >> [ 3.026481] Setting capacity to 31992832 >> [ 3.029051] xvdd: detected capacity change from 0 to 16380329984 >> [ 3.033320] blkfront: xvdf: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled; >> [ 3.040712] random: nonblocking pool is initialized >> [ 3.057432] xvdf: unknown partition table >> [ 3.060807] Setting capacity to 41943040 >> [ 3.063194] xvdf: detected capacity change from 0 to 21474836480 >> [ 3.067684] blkfront: xvdb: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled; >> [ 3.076835] xvdb: unknown partition table >> [ 3.079692] Setting capacity to 16777216 >> [ 3.082112] xvdb: detected capacity change from 0 to 8589934592 >> [ 3.086853] vbd vbd-51712: 16 xlvbd_add at /local/domain/0/backend/vbd/9543/51712 >> ---------------------- >> >>> >>>>> >>>>>> So the problem is still out there somewhere, it seems. >>>>>> >>>>>> Regards, >>>>>> Evgenii >>>>>> >>>>>>>>>> >>>>>>>>>> The part of the system log he was able to retrieve is attached. Here >>>>>>>>>> is >>>>>>>>>> the bug info, for convenience: >>>>>>>>>> >>>>>>>>>> ------------------------------------ >>>>>>>>>> [ 2.246912] kernel BUG at drivers/block/xen-blkfront.c:1711! >>>>>>>>>> [ 2.246912] invalid opcode: 0000 [#1] SMP >>>>>>>>>> [ 2.246912] Modules linked in: ata_generic pata_acpi >>>>>>>>>> crct10dif_pclmul >>>>>>>>>> crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel >>>>>>>>>> xen_netfront xen_blkfront(+) aesni_intel lrw ata_piix gf128mul >>>>>>>>>> glue_helper ablk_helper cryptd libata serio_raw floppy sunrpc >>>>>>>>>> dm_mirror >>>>>>>>>> dm_region_hash dm_log dm_mod scsi_transport_iscsi >>>>>>>>>> [ 2.246912] CPU: 1 PID: 50 Comm: xenwatch Not tainted >>>>>>>>>> 3.10.0-327.18.2.el7.x86_64 #1 >>>>>>>>>> [ 2.246912] Hardware name: Xen HVM domU, BIOS 4.2.amazon >>>>>>>>>> 12/07/2015 >>>>>>>>>> [ 2.246912] task: ffff8800e9fcb980 ti: ffff8800e98bc000 task.ti: >>>>>>>>>> ffff8800e98bc000 >>>>>>>>>> [ 2.246912] RIP: 0010:[] [] >>>>>>>>>> blkfront_setup_indirect+0x41f/0x430 [xen_blkfront] >>>>>>>>>> [ 2.246912] RSP: 0018:ffff8800e98bfcd0 EFLAGS: 00010283 >>>>>>>>>> [ 2.246912] RAX: ffff8800353e15c0 RBX: ffff8800e98c52c8 RCX: >>>>>>>>>> 0000000000000020 >>>>>>>>>> [ 2.246912] RDX: ffff8800353e15b0 RSI: ffff8800e98c52b8 RDI: >>>>>>>>>> ffff8800353e15d0 >>>>>>>>>> [ 2.246912] RBP: ffff8800e98bfd20 R08: ffff8800353e15b0 R09: >>>>>>>>>> ffff8800eb403c00 >>>>>>>>>> [ 2.246912] R10: ffffffffa0155532 R11: ffffffffffffffe8 R12: >>>>>>>>>> ffff8800e98c4000 >>>>>>>>>> [ 2.246912] R13: ffff8800e98c52b8 R14: 0000000000000020 R15: >>>>>>>>>> ffff8800353e15c0 >>>>>>>>>> [ 2.246912] FS: 0000000000000000(0000) GS:ffff8800efc20000(0000) >>>>>>>>>> knlGS:0000000000000000 >>>>>>>>>> [ 2.246912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>>>>>> [ 2.246912] CR2: 00007f1b615ef000 CR3: 00000000e2b44000 CR4: >>>>>>>>>> 00000000001406e0 >>>>>>>>>> [ 2.246912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: >>>>>>>>>> 0000000000000000 >>>>>>>>>> [ 2.246912] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: >>>>>>>>>> 0000000000000400 >>>>>>>>>> [ 2.246912] Stack: >>>>>>>>>> [ 2.246912] 0000000000000020 0000000000000001 00000020a0157217 >>>>>>>>>> 00000100e98bfdbc >>>>>>>>>> [ 2.246912] 0000000027efa3ef ffff8800e98bfdbc ffff8800e98ce000 >>>>>>>>>> ffff8800e98c4000 >>>>>>>>>> [ 2.246912] ffff8800e98ce040 0000000000000001 ffff8800e98bfe08 >>>>>>>>>> ffffffffa0155d4c >>>>>>>>>> [ 2.246912] Call Trace: >>>>>>>>>> [ 2.246912] [] blkback_changed+0x4ec/0xfc8 >>>>>>>>>> [xen_blkfront] >>>>>>>>>> [ 2.246912] [] ? xenbus_gather+0x170/0x190 >>>>>>>>>> [ 2.246912] [] ? __slab_free+0x10e/0x277 >>>>>>>>>> [ 2.246912] [] >>>>>>>>>> xenbus_otherend_changed+0xad/0x110 >>>>>>>>>> [ 2.246912] [] ? xenwatch_thread+0x77/0x180 >>>>>>>>>> [ 2.246912] [] backend_changed+0x13/0x20 >>>>>>>>>> [ 2.246912] [] xenwatch_thread+0x66/0x180 >>>>>>>>>> [ 2.246912] [] ? wake_up_atomic_t+0x30/0x30 >>>>>>>>>> [ 2.246912] [] ? >>>>>>>>>> unregister_xenbus_watch+0x1f0/0x1f0 >>>>>>>>>> [ 2.246912] [] kthread+0xcf/0xe0 >>>>>>>>>> [ 2.246912] [] ? >>>>>>>>>> kthread_create_on_node+0x140/0x140 >>>>>>>>>> [ 2.246912] [] ret_from_fork+0x58/0x90 >>>>>>>>>> [ 2.246912] [] ? >>>>>>>>>> kthread_create_on_node+0x140/0x140 >>>>>>>>>> [ 2.246912] Code: e1 48 85 c0 75 ce 49 8d 84 24 40 01 00 00 48 89 >>>>>>>>>> 45 >>>>>>>>>> b8 e9 91 fd ff ff 4c 89 ff e8 8d ae 06 e1 e9 f2 fc ff ff 31 c0 e9 2e >>>>>>>>>> fe >>>>>>>>>> ff ff <0f> 0b e8 9a 57 f2 e0 0f 0b 0f 1f 84 00 00 00 00 00 0f 1f 44 >>>>>>>>>> 00 >>>>>>>>>> [ 2.246912] RIP [] >>>>>>>>>> blkfront_setup_indirect+0x41f/0x430 [xen_blkfront] >>>>>>>>>> [ 2.246912] RSP >>>>>>>>>> [ 2.491574] ---[ end trace 8a9b992812627c71 ]--- >>>>>>>>>> [ 2.495618] Kernel panic - not syncing: Fatal exception >>>>>>>>>> ------------------------------------ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > https://lists.xen.org/xen-devel > --------------040809000904080307080704 Content-Type: text/x-log; name="3.14.74-1.test.x86_64.crash.log" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="3.14.74-1.test.x86_64.crash.log" ICAgICAgICAgU3RhcnRpbmcgQ3JlYXRlIFN0YXRpYyBEZXZpY2UgTm9kZXMgaW4gL2Rldi4u Lg0KWxtbMzJtICBPSyAgG1swbV0gU3RhcnRlZCBDcmVhdGUgU3RhdGljIERldmljZSBOb2Rl cyBpbiAvZGV2Lg0KWxtbMzJtICBPSyAgG1swbV0gU3RhcnRlZCBkcmFjdXQgY21kbGluZSBo b29rLg0KICAgICAgICAgU3RhcnRpbmcgZHJhY3V0IHByZS11ZGV2IGhvb2suLi4NCiAgICAg ICAgIFN0YXJ0aW5nIE9wZW4taVNDU0kuLi4NClsgICAgMi41OTk4MTFdIHNjc2lfdHJhbnNw b3J0X2lzY3NpOiBtb2R1bGUgdmVyaWZpY2F0aW9uIGZhaWxlZDogc2lnbmF0dXJlIGFuZC9v ciAgcmVxdWlyZWQga2V5IG1pc3NpbmcgLSB0YWludGluZyBrZXJuZWwNClsgICAgMi42MDcw NDJdIExvYWRpbmcgaVNDU0kgdHJhbnNwb3J0IGNsYXNzIHYyLjAtODcwLg0KWyAgICAyLjYx MDUwNV0gZGV2aWNlLW1hcHBlcjogdWV2ZW50OiB2ZXJzaW9uIDEuMC4zDQpbICAgIDIuNjEz MzY2XSBkZXZpY2UtbWFwcGVyOiBpb2N0bDogNC4yNy4wLWlvY3RsICgyMDEzLTEwLTMwKSBp bml0aWFsaXNlZDogZG0tZGV2ZWxAcmVkaGF0LmNvbQ0KWxtbMzJtICBPSyAgG1swbV0gU3Rh cnRlZCBPcGVuLWlTQ1NJLg0KWyAgICAyLjYyNzMyN10gc3VucnBjOiBtb2R1bGUgaGFzIGJh ZCB0YWludCwgbm90IGNyZWF0aW5nIHRyYWNlIGV2ZW50cw0KWyAgICAyLjYzMDk2MV0gUlBD OiBSZWdpc3RlcmVkIG5hbWVkIFVOSVggc29ja2V0IHRyYW5zcG9ydCBtb2R1bGUuDQpbICAg IDIuNjM0NDczXSBSUEM6IFJlZ2lzdGVyZWQgdWRwIHRyYW5zcG9ydCBtb2R1bGUuDQpbICAg IDIuNjM3MTg5XSBSUEM6IFJlZ2lzdGVyZWQgdGNwIHRyYW5zcG9ydCBtb2R1bGUuDQpbICAg IDIuNjM5ODc2XSBSUEM6IFJlZ2lzdGVyZWQgdGNwIE5GU3Y0LjEgYmFja2NoYW5uZWwgdHJh bnNwb3J0IG1vZHVsZS4NClsbWzMybSAgT0sgIBtbMG1dIFN0YXJ0ZWQgZHJhY3V0IHByZS11 ZGV2IGhvb2suDQogICAgICAgICBTdGFydGluZyB1ZGV2IEtlcm5lbCBEZXZpY2UgTWFuYWdl ci4uLg0KWxtbMzJtICBPSyAgG1swbV0gU3RhcnRlZCB1ZGV2IEtlcm5lbCBEZXZpY2UgTWFu YWdlci4NCiAgICAgICAgIFN0YXJ0aW5nIGRyYWN1dCBwcmUtdHJpZ2dlciBob29rLi4uDQpb G1szMm0gIE9LICAbWzBtXSBTdGFydGVkIGRyYWN1dCBwcmUtdHJpZ2dlciBob29rLg0KICAg ICAgICAgU3RhcnRpbmcgdWRldiBDb2xkcGx1ZyBhbGwgRGV2aWNlcy4uLg0KICAgICAgICAg TW91bnRpbmcgQ29uZmlndXJhdGlvbiBGaWxlIFN5c3RlbS4uLg0KWxtbMzJtICBPSyAgG1sw bV0gTW91bnRlZCBDb25maWd1cmF0aW9uIEZpbGUgU3lzdGVtLg0KWxtbMzJtICBPSyAgG1sw bV0gU3RhcnRlZCB1ZGV2IENvbGRwbHVnIGFsbCBEZXZpY2VzLg0KICAgICAgICAgU3RhcnRp bmcgZHJhY3V0IGluaXRxdWV1ZSBob29rLi4uDQpbG1szMm0gIE9LICAbWzBtXSBSZWFjaGVk IHRhcmdldCBTeXN0ZW0gSW5pdGlhbGl6YVsgICAgMi44MDE4OTJdIEZEQyAwIGlzIGEgUzgy MDc4Qg0KdGlvbi4NClsbWzMybSAgT0sgIBtbMG1dIExpc3RlbmluZyBvbiBPcGVuLWlTQ1NJ IGlzY3NpdWlvIFNvY2tldC4NClsbWzMybSAgT0sgIBtbMG1dIFJlYWNoZWQgdGFyZ2V0IFNv Y2tldHMuDQpbG1szMm0gIE9LICAbWzBtXSBSZWFjaGVkIHRhcmdldCBCYXNpYyBTeXN0ZW0u DQpbICAgIDIuODM1MDM0XSBzY3NpMCA6IGF0YV9waWl4DQpbICAgIDIuODQwMzE3XSBzY3Np MSA6IGF0YV9waWl4DQpbICAgIDIuODQyMjY3XSBhdGExOiBQQVRBIG1heCBNV0RNQTIgY21k IDB4MWYwIGN0bCAweDNmNiBibWRtYSAweGMxMDAgaXJxIDE0DQpbICAgIDIuODQ1ODYxXSBh dGEyOiBQQVRBIG1heCBNV0RNQTIgY21kIDB4MTcwIGN0bCAweDM3NiBibWRtYSAweGMxMDgg aXJxIDE1DQpbICAgIDIuODUzODQwXSBBVlggdmVyc2lvbiBvZiBnY21fZW5jL2RlYyBlbmdh Z2VkLg0KWyAgICAyLjg1OTk2M10geGVuX25ldGZyb250OiBJbml0aWFsaXNpbmcgWGVuIHZp cnR1YWwgZXRoZXJuZXQgZHJpdmVyDQpbICAgIDIuODY3MTU2XSBhbGc6IE5vIHRlc3QgZm9y IF9fZ2NtLWFlcy1hZXNuaSAoX19kcml2ZXItZ2NtLWFlcy1hZXNuaSkNClsgICAgMi44ODU4 NjFdIGJsa2Zyb250OiB4dmRhOiBiYXJyaWVyIG9yIGZsdXNoOiBkaXNhYmxlZDsgcGVyc2lz dGVudCBncmFudHM6IGRpc2FibGVkOyBpbmRpcmVjdCBkZXNjcmlwdG9yczogZW5hYmxlZDsN ClsgICAgMi44ODkwNDZdIGFsZzogTm8gdGVzdCBmb3IgY3JjMzIgKGNyYzMyLXBjbG11bCkN ClsgICAgMi44OTkyOTBdICB4dmRhOiB4dmRhMQ0KWyAgICAyLjk5Nzc1MV0gYmxrZnJvbnQ6 IHh2ZGM6IGZsdXNoIGRpc2tjYWNoZTogZW5hYmxlZDsgcGVyc2lzdGVudCBncmFudHM6IGRp c2FibGVkOyBpbmRpcmVjdCBkZXNjcmlwdG9yczogZW5hYmxlZDsNClsgICAgMy4wMDc0MDFd ICB4dmRjOiB1bmtub3duIHBhcnRpdGlvbiB0YWJsZQ0KWyAgICAzLjAxMDQ2NV0gU2V0dGlu ZyBjYXBhY2l0eSB0byAzMTk5MjgzMg0KWyAgICAzLjAxMjkyMl0geHZkYzogZGV0ZWN0ZWQg Y2FwYWNpdHkgY2hhbmdlIGZyb20gMCB0byAxNjM4MDMyOTk4NA0KWyAgICAzLjAxNzQwOF0g YmxrZnJvbnQ6IHh2ZGQ6IGZsdXNoIGRpc2tjYWNoZTogZW5hYmxlZDsgcGVyc2lzdGVudCBn cmFudHM6IGRpc2FibGVkOyBpbmRpcmVjdCBkZXNjcmlwdG9yczogZW5hYmxlZDsNClsgICAg My4wMjM4NjFdICB4dmRkOiB1bmtub3duIHBhcnRpdGlvbiB0YWJsZQ0KWyAgICAzLjAyNjQ4 MV0gU2V0dGluZyBjYXBhY2l0eSB0byAzMTk5MjgzMg0KWyAgICAzLjAyOTA1MV0geHZkZDog ZGV0ZWN0ZWQgY2FwYWNpdHkgY2hhbmdlIGZyb20gMCB0byAxNjM4MDMyOTk4NA0KWyAgICAz LjAzMzMyMF0gYmxrZnJvbnQ6IHh2ZGY6IGJhcnJpZXIgb3IgZmx1c2g6IGRpc2FibGVkOyBw ZXJzaXN0ZW50IGdyYW50czogZGlzYWJsZWQ7IGluZGlyZWN0IGRlc2NyaXB0b3JzOiBlbmFi bGVkOw0KWyAgICAzLjA0MDcxMl0gcmFuZG9tOiBub25ibG9ja2luZyBwb29sIGlzIGluaXRp YWxpemVkDQpbICAgIDMuMDU3NDMyXSAgeHZkZjogdW5rbm93biBwYXJ0aXRpb24gdGFibGUN ClsgICAgMy4wNjA4MDddIFNldHRpbmcgY2FwYWNpdHkgdG8gNDE5NDMwNDANClsgICAgMy4w NjMxOTRdIHh2ZGY6IGRldGVjdGVkIGNhcGFjaXR5IGNoYW5nZSBmcm9tIDAgdG8gMjE0NzQ4 MzY0ODANClsgICAgMy4wNjc2ODRdIGJsa2Zyb250OiB4dmRiOiBiYXJyaWVyIG9yIGZsdXNo OiBkaXNhYmxlZDsgcGVyc2lzdGVudCBncmFudHM6IGRpc2FibGVkOyBpbmRpcmVjdCBkZXNj cmlwdG9yczogZW5hYmxlZDsNClsgICAgMy4wNzY4MzVdICB4dmRiOiB1bmtub3duIHBhcnRp dGlvbiB0YWJsZQ0KWyAgICAzLjA3OTY5Ml0gU2V0dGluZyBjYXBhY2l0eSB0byAxNjc3NzIx Ng0KWyAgICAzLjA4MjExMl0geHZkYjogZGV0ZWN0ZWQgY2FwYWNpdHkgY2hhbmdlIGZyb20g MCB0byA4NTg5OTM0NTkyDQpbICAgIDMuMDg2ODUzXSB2YmQgdmJkLTUxNzEyOiAxNiB4bHZi ZF9hZGQgYXQgL2xvY2FsL2RvbWFpbi8wL2JhY2tlbmQvdmJkLzk1NDMvNTE3MTINClsgICAg My4wOTQxNDddIC0tLS0tLS0tLS0tLVsgY3V0IGhlcmUgXS0tLS0tLS0tLS0tLQ0KWyAgICAz LjA5NTA4MV0ga2VybmVsIEJVRyBhdCBkcml2ZXJzL2Jsb2NrL3hlbi1ibGtmcm9udC5jOjE3 MDEhDQpbICAgIDMuMDk1MDgxXSBpbnZhbGlkIG9wY29kZTogMDAwMCBbIzFdIFNNUCANClsg ICAgMy4wOTUwODFdIE1vZHVsZXMgbGlua2VkIGluOiBhdGFfZ2VuZXJpYyhGKSBwYXRhX2Fj cGkoRikgY3JjdDEwZGlmX3BjbG11bChGKSBjcmN0MTBkaWZfY29tbW9uKEYpIGNyYzMyX3Bj bG11bChGKSBjcmMzMmNfaW50ZWwoRikgeGVuX25ldGZyb250KEYpIHhlbl9ibGtmcm9udChG KykgYWVzbmlfaW50ZWwoRikgbHJ3KEYpIGdmMTI4bXVsKEYpIGdsdWVfaGVscGVyKEYpIGFi bGtfaGVscGVyKEYpIGNyeXB0ZChGKSBzZXJpb19yYXcoRikgYXRhX3BpaXgoRikgbGliYXRh KEYpIGZsb3BweShGKSBzdW5ycGMoRikgZG1fbWlycm9yKEYpIGRtX3JlZ2lvbl9oYXNoKEYp IGRtX2xvZyhGKSBkbV9tb2QoRikgc2NzaV90cmFuc3BvcnRfaXNjc2koRikNClsgICAgMy4w OTUwODFdIENQVTogMSBQSUQ6IDQ5IENvbW06IHhlbndhdGNoIFRhaW50ZWQ6IEdGICAgICAg ICAgICAgMy4xNC43NC0xLnRlc3QueDg2XzY0ICMxDQpbICAgIDMuMDk1MDgxXSBIYXJkd2Fy ZSBuYW1lOiBYZW4gSFZNIGRvbVUsIEJJT1MgNC4yLmFtYXpvbiAwNS8xMi8yMDE2DQpbICAg IDMuMDk1MDgxXSB0YXNrOiBmZmZmODgwMGU5ZjdkMTAwIHRpOiBmZmZmODgwMGU5OGFjMDAw IHRhc2sudGk6IGZmZmY4ODAwZTk4YWMwMDANClsgICAgMy4wOTUwODFdIFJJUDogMDAxMDpb PGZmZmZmZmZmYTAxN2Y3ZWY+XSAgWzxmZmZmZmZmZmEwMTdmN2VmPl0gYmxrZnJvbnRfc2V0 dXBfaW5kaXJlY3QrMHg0MWYvMHg0MzAgW3hlbl9ibGtmcm9udF0NClsgICAgMy4wOTUwODFd IFJTUDogMDAxODpmZmZmODgwMGU5OGFkY2Q4ICBFRkxBR1M6IDAwMDEwMjgzDQpbICAgIDMu MDk1MDgxXSBSQVg6IGZmZmY4ODAwMzQ0ZGNhODAgUkJYOiBmZmZmODgwMGU1Njg3MmM4IFJD WDogMDAwMDAwMDAwMDAwMDAyMA0KWyAgICAzLjA5NTA4MV0gUkRYOiBmZmZmODgwMDM0NGRj YTcwIFJTSTogZmZmZjg4MDBlNTY4NzJiOCBSREk6IGZmZmY4ODAwMzQ0ZGNhOTANClsgICAg My4wOTUwODFdIFJCUDogZmZmZjg4MDBlOThhZGQyOCBSMDg6IGZmZmY4ODAwMzQ0ZGNhNzAg UjA5OiBmZmZmODgwMGViNDAzYzAwDQpbICAgIDMuMDk1MDgxXSBSMTA6IGZmZmZmZmZmYTAx N2Y0ZDIgUjExOiBmZmZmZmZmZmZmZmZmZmU4IFIxMjogZmZmZjg4MDBlNTY4NjAwMA0KWyAg ICAzLjA5NTA4MV0gUjEzOiBmZmZmODgwMGU1Njg3MmI4IFIxNDogMDAwMDAwMDAwMDAwMDAy MCBSMTU6IGZmZmY4ODAwMzQ0ZGNhODANClsgICAgMy4wOTUwODFdIEZTOiAgMDAwMDAwMDAw MDAwMDAwMCgwMDAwKSBHUzpmZmZmODgwMGVmYzIwMDAwKDAwMDApIGtubEdTOjAwMDAwMDAw MDAwMDAwMDANClsgICAgMy4wOTUwODFdIENTOiAgMDAxMCBEUzogMDAwMCBFUzogMDAwMCBD UjA6IDAwMDAwMDAwODAwNTAwMzMNClsgICAgMy4wOTUwODFdIENSMjogMDAwMDdmYWRlZTFj Njg5MCBDUjM6IDAwMDAwMDAwMzViNzcwMDAgQ1I0OiAwMDAwMDAwMDAwMTQwNmUwDQpbICAg IDMuMDk1MDgxXSBTdGFjazoNClsgICAgMy4wOTUwODFdICAwMDAwMDAwMDAwMDAwMDIwIDAw MDAwMDAwMDAwMDAwMDEgMDAwMDAwMjBhMDE4MTFmOCAwMDAwMDEwMGU5OGFkZGNjDQpbICAg IDMuMDk1MDgxXSAgMDAwMDAwMDAyOWQzNTUwOSBmZmZmODgwMGU1Njg2MDAwIDAwMDAwMDAw MDAwMDAwMDQgZmZmZjg4MDBlOThhZGRjYw0KWyAgICAzLjA5NTA4MV0gIGZmZmY4ODAwZTk4 NjYwNDAgMDAwMDAwMDAwMDAwMDAwMSBmZmZmODgwMGU5OGFkZTE4IGZmZmZmZmZmYTAxN2Zj ZmUNClsgICAgMy4wOTUwODFdIENhbGwgVHJhY2U6DQpbICAgIDMuMDk1MDgxXSAgWzxmZmZm ZmZmZmEwMTdmY2ZlPl0gYmxrYmFja19jaGFuZ2VkKzB4NGZlLzB4MTI4MCBbeGVuX2Jsa2Zy b250XQ0KWyAgICAzLjA5NTA4MV0gIFs8ZmZmZmZmZmY4MTNhNjRhMD5dID8geGVuYnVzX2dh dGhlcisweDE3MC8weDE5MA0KWyAgICAzLjA5NTA4MV0gIFs8ZmZmZmZmZmY4MTYzMWY0Nj5d ID8gX19zbGFiX2ZyZWUrMHgxMGUvMHgyNzcNClsgICAgMy4wOTUwODFdICBbPGZmZmZmZmZm ODEzYTc0ZmQ+XSB4ZW5idXNfb3RoZXJlbmRfY2hhbmdlZCsweGFkLzB4MTEwDQpbICAgIDMu MDk1MDgxXSAgWzxmZmZmZmZmZjgxM2E2NzE2Pl0gPyB4ZW53YXRjaF90aHJlYWQrMHg2Ni8w eDE2MA0KWyAgICAzLjA5NTA4MV0gIFs8ZmZmZmZmZmY4MTNhNjZiMD5dID8gdW5yZWdpc3Rl cl94ZW5idXNfd2F0Y2grMHgxZjAvMHgxZjANClsgICAgMy4wOTUwODFdICBbPGZmZmZmZmZm ODEzYTk4MTM+XSBiYWNrZW5kX2NoYW5nZWQrMHgxMy8weDIwDQpbICAgIDMuMDk1MDgxXSAg WzxmZmZmZmZmZjgxM2E2NzA1Pl0geGVud2F0Y2hfdGhyZWFkKzB4NTUvMHgxNjANClsgICAg My4wOTUwODFdICBbPGZmZmZmZmZmODEwYzA0NTA+XSA/IGFib3J0X2V4Y2x1c2l2ZV93YWl0 KzB4YjAvMHhiMA0KWyAgICAzLjA5NTA4MV0gIFs8ZmZmZmZmZmY4MTA5YjNjOD5dIGt0aHJl YWQrMHhkOC8weGYwDQpbICAgIDMuMDk1MDgxXSAgWzxmZmZmZmZmZjgxMDliMmYwPl0gPyBr dGhyZWFkX2NyZWF0ZV9vbl9ub2RlKzB4MWEwLzB4MWEwDQpbICAgIDMuMDk1MDgxXSAgWzxm ZmZmZmZmZjgxNjQ2NTU4Pl0gcmV0X2Zyb21fZm9yaysweDU4LzB4OTANClsgICAgMy4wOTUw ODFdICBbPGZmZmZmZmZmODEwOWIyZjA+XSA/IGt0aHJlYWRfY3JlYXRlX29uX25vZGUrMHgx YTAvMHgxYTANClsgICAgMy4wOTUwODFdIENvZGU6IGUxIDQ4IDg1IGMwIDc1IGNlIDQ5IDhk IDg0IDI0IDQwIDAxIDAwIDAwIDQ4IDg5IDQ1IGI4IGU5IDkxIGZkIGZmIGZmIDRjIDg5IGZm IGU4IGJkIGVhIDAzIGUxIGU5IGYyIGZjIGZmIGZmIDMxIGMwIGU5IDJlIGZlIGZmIGZmIDww Zj4gMGIgZTggMWEgNWYgZWYgZTAgMGYgMGIgMGYgMWYgODQgMDAgMDAgMDAgMDAgMDAgMGYg MWYgNDQgMDAgDQpbICAgIDMuMDk1MDgxXSBSSVAgIFs8ZmZmZmZmZmZhMDE3ZjdlZj5dIGJs a2Zyb250X3NldHVwX2luZGlyZWN0KzB4NDFmLzB4NDMwIFt4ZW5fYmxrZnJvbnRdDQpbICAg IDMuMDk1MDgxXSAgUlNQIDxmZmZmODgwMGU5OGFkY2Q4Pg0KWyAgICAzLjI3ODE1Ml0gLS0t WyBlbmQgdHJhY2UgZGRhMTc5M2U3ZGViMGI2NSBdLS0tDQpbICAgIDMuMjgwODI0XSBLZXJu ZWwgcGFuaWMgLSBub3Qgc3luY2luZzogRmF0YWwgZXhjZXB0aW9uDQoK --------------040809000904080307080704 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --------------040809000904080307080704--