From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michal Soltys Subject: Re: [general question] rare silent data corruption when writing data Date: Fri, 8 May 2020 13:10:22 +0200 Message-ID: References: <24244.30530.155404.154787@quad.stoffel.home> <24244.44462.151185.626440@quad.stoffel.home> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <24244.44462.151185.626440@quad.stoffel.home> Content-Language: en-US-large Sender: linux-raid-owner@vger.kernel.org To: John Stoffel Cc: Roger Heflin , Linux RAID , "linux-lvm@redhat.com" List-Id: linux-raid.ids bm90ZTogYXMgc3VnZ2VzdGVkLCBJJ20gYWxzbyBDQ2luZyB0aGlzIHRvIGxpbnV4LWx2bTsg dGhlIGZ1bGwgY29udGV4dCB3aXRoIHJlcGxpZXMgc3RhcnRzIGF0Og0KDQpodHRwczovL3d3 dy5zcGluaWNzLm5ldC9saXN0cy9yYWlkL21zZzY0MzY0Lmh0bWwNClRoZXJlIGlzIGFsc28g dGhlIGluaXRpYWwgcG9zdCBhdCB0aGUgYm90dG9tIGFzIHdlbGwuDQoNCk9uIDUvOC8yMCAy OjU0IEFNLCBKb2huIFN0b2ZmZWwgd3JvdGU6DQo+Pj4+Pj4gIk1pY2hhbCIgPT0gTWljaGFs IFNvbHR5cyA8bXNvbHR5c3BsQHlhbmRleC5wbD4gd3JpdGVzOg0KPiANCj4gTWljaGFsPiBP biAyMC8wNS8wNyAyMzowMSwgSm9obiBTdG9mZmVsIHdyb3RlOg0KPj4+Pj4+Pj4gIlJvZ2Vy IiA9PSBSb2dlciBIZWZsaW4gPHJvZ2VyaGVmbGluQGdtYWlsLmNvbT4gd3JpdGVzOg0KPj4+ DQo+IFJvZ2VyPiBIYXZlIHlvdSB0cmllZCB0aGUgc2FtZSBmaWxlIDJ4IGFuZCB2ZXJpZmll ZCB0aGUgY29ycnVwdGlvbiBpcyBpbiB0aGUNCj4gUm9nZXI+IHNhbWUgcGxhY2VzIGFuZCBs b29rcyB0aGUgc2FtZT8NCj4+Pg0KPj4+IEFyZSB0aGVzZSAxdGIgZmlsZXMgVk1ESyBvciBD T1cgaW1hZ2VzIG9mIFZNcz8gIEhvdyBhcmUgdGhlc2UgZmlsZXMNCj4+PiBtYWRlLiAgQW5k IGRvZXMgaXQgZXZlciBoYXBwZW4gd2l0aCAqc21hbGxlciogZmlsZXM/ICBXaGF0IGFib3V0 IGlmDQo+Pj4geW91IGp1c3QgdXNlIGEgc3BhcnNlIDJ0YiBmaWxlIGFuZCB3cml0ZSBibG9j a3Mgb3V0IHBhc3QgMXRiIHRvIHNlZSBpZg0KPj4+IHRoZXJlJ3MgYSBwcm9ibGVtPw0KPiAN Cj4gTWljaGFsPiBUaGUgVk1zIGFyZSBhbHdheXMgZGlyZWN0bHkgb24gbHZtIHZvbHVtZXMu IChlLmcuDQo+IE1pY2hhbD4gL2Rldi9tYXBwZXIvdmcwLWdpdGxhYikuIFRoZSBndWVzdCAo YnRyZnMgaW5zaWRlIHRoZSBndWVzdCkgZGV0ZWN0ZWQgdGhlDQo+IE1pY2hhbD4gZXJyb3Jz IGFmdGVyIHdlIHJhbiBzY3J1YiBvbiB0aGUgZmlsZXN5c3RlbS4NCj4gDQo+IE1pY2hhbD4g WWVzLCB0aGUgZXJyb3JzIHdlcmUgYWxzbyBmb3VuZCBvbiBzbWFsbCBmaWxlcy4NCj4gDQo+ IFRob3NlIGVycm9ycyBhcmUgaW4gc21hbGwgZmlsZXMgaW5zaWRlIHRoZSBWTSwgd2hpY2gg aXMgcnVubmluZyBidHJmcw0KPiBvbnRvcCBvZiBibG9jayBzdG9yYWdlIHByb3ZpZGVkIGJ5 IHlvdXIgdGhpbi1sdiwgcmlnaHQ/DQo+IA0KDQpZZWEsIHRoZSBzbWFsbCBmaWxlcyB3ZXJl IGluIHRoaXMgY2FzZSBvbiB0aGF0IHRoaW4tbHYuDQoNCldlIGFsc28gZGlzY292ZXJlZCAo eWVzdGVyZGF5KSBmaWxlIGNvcnJ1cHRpb25zIHdpdGggVk0gaG9zdGluZyBnaXRsYWIgcmVn aXN0cnkgLSB0aGlzIG9uZSB3YXMgdXNpbmcgdGhlIHNhbWUgdGhpbi1sdiB1bmRlcm5lYXRo LCBidXQgdGhlIGd1ZXN0IGl0c2VsZiB3YXMgdXNpbmcgZXh0NCAoaW4gdGhpcyBjYXNlLCBk b2NrZXIgc2ltcGx5IHJlcG9ydGVkIGluY29ycmVjdCBzaGEgY2hlY2tzdW0gb24gKHNvIGZh cikgMiBsYXllcnMuDQoNCj4gDQo+IA0KPiBkaXNrcyAtPiBtZCByYWlkNSAtPiBwdiAtPiB2 ZyAtPiBsdi10aGluIC0+IGd1ZXN0IFFDT1cvTFVOIC0+DQo+IGZpbGVzeXN0ZW0gLT4gY29y cnVwdGlvbg0KDQpUaG9zZSBwYXJ0aWN1bGFyIGd1ZXN0cywgeWVhLiBUaGUgaG9zdCBjYXNl IGl0J3MganVzdCB3L28gImd1ZXN0IiBzdGVwLg0KDQpCdXQgKHNvIGZhcikgYWxsIGNvcnJ1 cHRpb24gZW5kZWQgZ29pbmcgdmlhIG9uZSBvZiB0aGUgbHYtdGhpbiBsYXllcnMgKGFuZCB2 aWEgb25lIG9mIG1kIHJhaWRzKS4NCg0KPiANCj4gDQo+IE1pY2hhbD4gU2luY2UgdGhlbiB3 ZSByZWNyZWF0ZWQgdGhlIGlzc3VlIGRpcmVjdGx5IG9uIHRoZSBob3N0LCBqdXN0DQo+IE1p Y2hhbD4gYnkgbWFraW5nIGV4dDQgZmlsZXN5c3RlbSBvbiBzb21lIExWLCB0aGVuIGRvaW5n IHdyaXRlIHdpdGgNCj4gTWljaGFsPiBjaGVja3N1bSwgc3luYywgZHJvcF9jYWNoZXMsIHJl YWQgYW5kIGNoZWNrIGNoZWNrc3VtLiBUaGUNCj4gTWljaGFsPiBlcnJvcnMgYXJlLCBhcyBJ IG1lbnRpb25lZCAtIGFsd2F5cyBhIGZ1bGwgNEtpQiBjaHVua3MgKGFsd2F5cw0KPiBNaWNo YWw+IHNhbWUgY29udGVudCwgYWx3YXlzIHNhbWUgcG9zaXRpb24pLg0KPiANCj4gV2hhdCBw b3NpdGlvbj8gIElzIGl0IGEgNGssIDEuNW0gb3Igc29tZSBvdGhlciBjb25zaXN0ZW50IG9m ZnNldD8gIEFuZA0KPiBob3cgZmFyIGludG8gdGhlIGZpbGU/ICBBbmQgdGhpcyBMViBpcyBh IHBsYWluIExWIG9yIGEgdGhpbi1sdj8gICBJJ20NCj4gcnVubmluZyBhIGRlYmlhbiBib3gg YXQgaG9tZSB3aXRoIFJBSUQxIGFuZCBJIGhhdmVuJ3Qgc2VlbiB0aGlzLCBidXQNCj4gSSdt IG5vdCBuZWFybHkgYXMgY2FyZWZ1bCBhcyB5b3UuICBDYW4geW91IHByb3ZpZGUgdGhlIG91 dHB1dCBvZjoNCj4gDQoNCldoYXQgSSBtZWFudCB0aGF0IGl0IGRvZXNuJ3QgIm1vdmUiIHdo ZW4gdmVyaWZ5aW5nIHRoZSBzYW1lIGZpbGUgKGFrYSBkaWZmZXJlbnQgcmVhZHMgZnJvbSBz YW1lIHRlc3QgZmlsZSkuIEJldHdlZW4gdGhlIHRlc3RzLCB0aGUgZXJyb3JzIGFyZSBvZiBj b3Vyc2UgaW4gZGlmZmVyZW50IHBsYWNlcyAtIGJ1dCBpdCdzIGFsd2F5cyBzb21lIDRLaUIg cGllY2UocykgLSB0aGF0IGxvb2sgbGlrZSBjb3JyZWN0IHBpZWNlcyBiZWxvbmdpbmcgc29t ZXdoZXJlIGVsc2UuDQoNCj4gICAgIC9zYmluL2x2cyAtLXZlcnNpb24NCg0KICBMVk0gdmVy c2lvbjogICAgIDIuMDMuMDIoMikgKDIwMTgtMTItMTgpDQogIExpYnJhcnkgdmVyc2lvbjog MS4wMi4xNTUgKDIwMTgtMTItMTgpDQogIERyaXZlciB2ZXJzaW9uOiAgNC40MS4wDQogIENv bmZpZ3VyYXRpb246ICAgLi9jb25maWd1cmUgLS1idWlsZD14ODZfNjQtbGludXgtZ251IC0t cHJlZml4PS91c3IgLS1pbmNsdWRlZGlyPSR7cHJlZml4fS9pbmNsdWRlIC0tbWFuZGlyPSR7 cHJlZml4fS9zaGFyZS9tYW4gLS1pbmZvZGlyPSR7cHJlZml4fS9zaGFyZS9pbmZvIC0tc3lz Y29uZmRpcj0vZXRjIC0tbG9jYWxzdGF0ZWRpcj0vdmFyIC0tZGlzYWJsZS1zaWxlbnQtcnVs ZXMgLS1saWJkaXI9JHtwcmVmaXh9L2xpYi94ODZfNjQtbGludXgtZ251IC0tbGliZXhlY2Rp cj0ke3ByZWZpeH0vbGliL3g4Nl82NC1saW51eC1nbnUgLS1ydW5zdGF0ZWRpcj0vcnVuIC0t ZGlzYWJsZS1tYWludGFpbmVyLW1vZGUgLS1kaXNhYmxlLWRlcGVuZGVuY3ktdHJhY2tpbmcg LS1leGVjLXByZWZpeD0gLS1iaW5kaXI9L2JpbiAtLWxpYmRpcj0vbGliL3g4Nl82NC1saW51 eC1nbnUgLS1zYmluZGlyPS9zYmluIC0td2l0aC11c3JsaWJkaXI9L3Vzci9saWIveDg2XzY0 LWxpbnV4LWdudSAtLXdpdGgtb3B0aW1pc2F0aW9uPS1PMiAtLXdpdGgtY2FjaGU9aW50ZXJu YWwgLS13aXRoLWRldmljZS11aWQ9MCAtLXdpdGgtZGV2aWNlLWdpZD02IC0td2l0aC1kZXZp Y2UtbW9kZT0wNjYwIC0td2l0aC1kZWZhdWx0LXBpZC1kaXI9L3J1biAtLXdpdGgtZGVmYXVs dC1ydW4tZGlyPS9ydW4vbHZtIC0td2l0aC1kZWZhdWx0LWxvY2tpbmctZGlyPS9ydW4vbG9j ay9sdm0gLS13aXRoLXRoaW49aW50ZXJuYWwgLS13aXRoLXRoaW4tY2hlY2s9L3Vzci9zYmlu L3RoaW5fY2hlY2sgLS13aXRoLXRoaW4tZHVtcD0vdXNyL3NiaW4vdGhpbl9kdW1wIC0td2l0 aC10aGluLXJlcGFpcj0vdXNyL3NiaW4vdGhpbl9yZXBhaXIgLS1lbmFibGUtYXBwbGliIC0t ZW5hYmxlLWJsa2lkX3dpcGluZyAtLWVuYWJsZS1jbWRsaWIgLS1lbmFibGUtZG1ldmVudGQg LS1lbmFibGUtZGJ1cy1zZXJ2aWNlIC0tZW5hYmxlLWx2bWxvY2tkLWRsbSAtLWVuYWJsZS1s dm1sb2NrZC1zYW5sb2NrIC0tZW5hYmxlLWx2bXBvbGxkIC0tZW5hYmxlLW5vdGlmeS1kYnVz IC0tZW5hYmxlLXBrZ2NvbmZpZyAtLWVuYWJsZS1yZWFkbGluZSAtLWVuYWJsZS11ZGV2X3J1 bGVzIC0tZW5hYmxlLXVkZXZfc3luYw0KDQo+IA0KPiB0b28/DQo+IA0KPiBDYW4geW91IHBv c3QgeW91cjoNCj4gDQo+ICAgICAvc2Jpbi9kbXNldHVwIHN0YXR1cw0KPiANCj4gb3V0cHV0 IHRvbz8gIFRoZXJlJ3MgYSBiZXR0ZXIgY29tbWFuZCB0byB1c2UgaGVyZSwgYnV0IEknbSBu b3QgYW4NCj4gZXhwb3J0LiAgWW91IG1pZ2h0IHJlYWxseSB3YW50IHRvIGNvcHkgdGhpcyBv dmVyIHRvIHRoZQ0KPiBsaW51eC1sdm1AcmVkaGF0LmNvbSBtYWlsaW5nIGxpc3QgYXMgd2Vs bC4NCg0KeDIydjAtdHBfc3NkLXRwb29sOiAwIDI1NzcyODUxMjAgdGhpbi1wb29sIDE5IDg4 ODYvNTUyOTYwIDYyOTUzNS84Mzg5NjAgLSBydyBub19kaXNjYXJkX3Bhc3Nkb3duIHF1ZXVl X2lmX25vX3NwYWNlIC0gMTAyNCANCngyMnYwLXRwX3NzZF90ZGF0YTogMCAyMTQ3Njk2NjQw IGxpbmVhciANCngyMnYwLXRwX3NzZF90ZGF0YTogMjE0NzY5NjY0MCA0Mjk1ODg0ODAgbGlu ZWFyIA0KeDIydjAtdHBfc3NkX3RtZXRhX3JpbWFnZV8xOiAwIDQ0MjM2ODAgbGluZWFyIA0K eDIydjAtdHBfc3NkX3RtZXRhOiAwIDQ0MjM2ODAgcmFpZCByYWlkMSAyIEFBIDQ0MjM2ODAv NDQyMzY4MCBpZGxlIDAgMCAtDQp4MjJ2MC1nZXJyaXQtLW5ldzogMCAyNjg2MTU2ODAgdGhp biAyNTU1MTA1MjggMjY4NDU5MDA3DQp4MjJ2MC1idHJmc25vcG9vbDogMCAxMzQ0MzA3MjAg bGluZWFyIA0KeDIydjAtZ2l0bGFiX3Jvb3Q6IDAgNjI5MTQ1NjAwIHRoaW4gNjI4MjkxNTg0 IDYyOTE0NTU5OQ0KeDIydjAtdHBfc3NkX3RtZXRhX3JpbWFnZV8wOiAwIDQ0MjM2ODAgbGlu ZWFyIA0KeDIydjAtbmV4dXNfb2xkX3N0b3JhZ2U6IDAgMTA3Mzc1MDAxNjAgdGhpbiA1MTMw ODE3NTM2IDEwNzM3NTAwMTU5DQp4MjJ2MC1naXRsYWJfcmVnOiAwIDIxNDc2OTY2NDAgdGhp biAxMDcwOTYzNzEyIDIxNDc2OTY2MzkNCngyMnYwLW5leHVzX29sZF9yb290OiAwIDI2ODYx NTY4MCB0aGluIDI1NzY1Nzg1NiAyNjg2MTU2NzkNCngyMnYwLXRwX2JpZ190bWV0YV9yaW1h Z2VfMTogMCA4NjAxNjAwIGxpbmVhciANCngyMnYwLXRwX3NzZF90bWV0YV9ybWV0YV8xOiAw IDI0NTc2MCBsaW5lYXIgDQp4MjJ2MC1taWNyb25fdm9sOiAwIDI2ODYxNTY4MCBsaW5lYXIg DQp4MjJ2MC10cF9iaWdfdG1ldGFfcmltYWdlXzA6IDAgODYwMTYwMCBsaW5lYXIgDQp4MjJ2 MC10cF9zc2RfdG1ldGFfcm1ldGFfMDogMCAyNDU3NjAgbGluZWFyIA0KeDIydjAtZ2Vycml0 LS1yb290OiAwIDI2ODYxNTY4MCB0aGluIDEwMzM4ODE2MCAyNjg0NDM2NDcNCngyMnYwLWJ0 cmZzX3NzZF9saW5lYXI6IDAgMjY4NjE1NjgwIGxpbmVhciANCngyMnYwLWJ0cmZzdGVzdDog MCAyNjg2MTU2ODAgdGhpbiA0MDczNDcyMCAyNjg2MTU2NzkNCngyMnYwLXRwX3NzZDogMCAy NTc3Mjg1MTIwIGxpbmVhciANCngyMnYwLXRwX2JpZzogMCAyMjE2NDYwMjg4MCBsaW5lYXIg DQp4MjJ2MC1uZXh1czNfcm9vdDogMCAxNjc4NTQwODAgdGhpbiAyMTg2MDM1MiAxNjc4NTQw NzkNCngyMnYwLW51c2tuYWNrZXItLXN0YWdpbmc6IDAgMjY4NjE1NjgwIHRoaW4gMjY4MTgy NTI4IDI2ODYxNTY3OQ0KeDIydjAtdG1vYjI6IDAgMTA0ODY1NzkyMCBsaW5lYXIgDQp4MjJ2 MC10cF9iaWctdHBvb2w6IDAgMjIxNjQ2MDI4ODAgdGhpbi1wb29sIDM1IDM1MTUyLzEwNzUy MDAgMzg3MDA3MC83MjE1MDQwIC0gcncgbm9fZGlzY2FyZF9wYXNzZG93biBxdWV1ZV9pZl9u b19zcGFjZSAtIDEwMjQgDQp4MjJ2MC10cF9iaWdfdGRhdGE6IDAgNDI5NTE0NzUyMCBsaW5l YXIgDQp4MjJ2MC10cF9iaWdfdGRhdGE6IDQyOTUxNDc1MjAgMTc4Njk0NTUzNjAgbGluZWFy IA0KeDIydjAtYnRyZnNfc3NkX3Rlc3Q6IDAgMjAxNTIzMjAwIHRoaW4gMTkxODgwMTkyIDIw MTMzNTgwNw0KeDIydjAtbnVzc2tuYWNrZXIyOiAwIDI2ODYxNTY4MCB0aGluIDU4NTczODI0 IDI2ODYxNTY3OQ0KeDIydjAtdG1vYjE6IDAgMTA0ODY1NzkyMCBsaW5lYXIgDQp4MjJ2MC10 cF9iaWdfdG1ldGE6IDAgODYwMTYwMCByYWlkIHJhaWQxIDIgQUEgODYwMTYwMC84NjAxNjAw IGlkbGUgMCAwIC0NCngyMnYwLW51c3NrbmFja2VyMTogMCAyNjg2MTU2ODAgdGhpbiA3NDM3 NjE5MiAyNjg2MTU2NzkNCngyMnYwLXRvdWstLWVsazQ6IDAgODM5MDI0NjQwIGxpbmVhciAN CngyMnYwLWdlcnJpdC0tYmFja3VwOiAwIDI2ODYxNTY4MCB0aGluIDIyODk4OTk1MiAyNjg0 NDM2NDcNCngyMnYwLXRwX2JpZ190bWV0YV9ybWV0YV8xOiAwIDI0NTc2MCBsaW5lYXIgDQp4 MjJ2MC1vcGVudnBuLS1uZXc6IDAgMTM0NDMwNzIwIHRoaW4gMjQxNTIwNjQgNjYyNzIyNTUN CngyMnYwLWs4c2RrcjogMCAyNjg2MTU2ODAgbGluZWFyIA0KeDIydjAtbmV4dXMzX3N0b3Jh Z2U6IDAgMTA3Mzc1MDAxNjAgdGhpbiA0OTc2NjgzMDA4IDEwNzM3NTAwMTU5DQp4MjJ2MC1y b2NrZXQ6IDAgMTY3ODU0MDgwIHRoaW4gMTYzNjAyNDMyIDE2Nzg1NDA3OQ0KeDIydjAtdHBf YmlnX3RtZXRhX3JtZXRhXzA6IDAgMjQ1NzYwIGxpbmVhciANCngyMnYwLXJvZ2VyMjogMCAx MzQ0MzA3MjAgdGhpbiAzMzAxNDc4NCAxMzQ0MzA3MTkNCngyMnYwLWdlcnJpdC0tbmV3LS1i YWNrdXA6IDAgMjY4NjE1NjgwIHRoaW4gNjU1MjU3NiAyNjg0NDM2NDcNCg0KQWxzbyBsdnMg LWEgd2l0aCBzZWdtZW50IHJhbmdlczoNCiAgTFYgICAgICAgICAgICAgICAgICAgICAgVkcg ICAgQXR0ciAgICAgICBMU2l6ZSAgICBQb29sICAgT3JpZ2luICAgICAgRGF0YSUgIE1ldGEl ICBNb3ZlIExvZyBDcHklU3luYyBDb252ZXJ0IExFIFJhbmdlcyAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KICBidHJmc19zc2RfbGluZWFyICAg ICAgICB4MjJ2MCAtd2ktYS0tLS0tIDwxMjguMDlnICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgL2Rldi9tZDEyNToxOTAyMS0y MDExMyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQogIGJ0cmZzX3NzZF90 ZXN0ICAgICAgICAgIHgyMnYwIFZ3aS1hLXQtLS0gICA5Ni4wOWcgdHBfc3NkICAgICAgICAg ICAgIDk1LjIxICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCiAgYnRy ZnNub3Bvb2wgICAgICAgICAgICAgeDIydjAgLXdpLWEtLS0tLSAgIDY0LjEwZyAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC9k ZXYvc2R0MjozNS01ODEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IA0KICBidHJmc3Rlc3QgICAgICAgICAgICAgICB4MjJ2MCBWd2ktYS10LS0tIDwxMjguMDln IHRwX2JpZyAgICAgICAgICAgICAxNS4xNiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgDQogIGdlcnJpdC1iYWNrdXAgICAgICAgICAgIHgyMnYwIFZ3aS1hb3QtLS0g PDEyOC4wOWcgdHBfYmlnICAgICAgICAgICAgIDg1LjI1ICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICANCiAgZ2Vycml0LW5ldyAgICAgICAgICAgICAgeDIydjAgVndp LWEtdC0tLSA8MTI4LjA5ZyB0cF9zc2QgICAgICAgICAgICAgOTUuMTIgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIA0KICBnZXJyaXQtbmV3LWJhY2t1cCAgICAgICB4 MjJ2MCBWd2ktYS10LS0tIDwxMjguMDlnIHRwX2JpZyAgICAgICAgICAgICAyLjQ0ICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQogIGdlcnJpdC1yb290ICAgICAg ICAgICAgIHgyMnYwIFZ3aS1hb3QtLS0gPDEyOC4wOWcgdHBfc3NkICAgICAgICAgICAgIDM4 LjQ5ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCiAgZ2l0bGFiX3Jl ZyAgICAgICAgICAgICAgeDIydjAgVndpLWEtdC0tLSAgICAxLjAwdCB0cF9iaWcgICAgICAg ICAgICAgNDkuODcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KICBn aXRsYWJfcmVnX3NuYXBzaG90ICAgICB4MjJ2MCBWd2ktLS10LS1rICAgIDEuMDB0IHRwX2Jp ZyBnaXRsYWJfcmVnICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgDQogIGdpdGxhYl9yb290ICAgICAgICAgICAgIHgyMnYwIFZ3aS1hLXQtLS0gIDMwMC4w MGcgdHBfc3NkICAgICAgICAgICAgIDk5Ljg2ICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICANCiAgZ2l0bGFiX3Jvb3Rfc25hcHNob3QgICAgeDIydjAgVndpLS0tdC0t ayAgMzAwLjAwZyB0cF9zc2QgZ2l0bGFiX3Jvb3QgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIA0KICBrOHNka3IgICAgICAgICAgICAgICAgICB4MjJ2MCAt d2ktYS0tLS0tIDwxMjguMDlnICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgL2Rldi9tZDEyNjoyMDg5MS0yMTk4MyAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgDQogIFtsdm9sMF9wbXNwYXJlXSAgICAgICAg IHgyMnYwIGV3aS0tLS0tLS0gICAgNC4xMGcgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvZGV2L3NkdDI6MC0zNCAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCiAgbWljcm9uX3ZvbCAgICAg ICAgICAgICAgeDIydjAgLXdpLWEtLS0tLSA8MTI4LjA5ZyAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC9kZXYvc2R0Mjo1ODIt MTY3NCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KICBuZXh1czNf cm9vdCAgICAgICAgICAgICB4MjJ2MCBWd2ktYW90LS0tICA8ODAuMDRnIHRwX3NzZCAgICAg ICAgICAgICAxMy4wMyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQog IG5leHVzM19zdG9yYWdlICAgICAgICAgIHgyMnYwIFZ3aS1hb3QtLS0gICAgNS4wMHQgdHBf YmlnICAgICAgICAgICAgIDQ2LjM1ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICANCiAgbmV4dXNfb2xkX3Jvb3QgICAgICAgICAgeDIydjAgVndpLWEtdC0tLSA8MTI4 LjA5ZyB0cF9zc2QgICAgICAgICAgICAgOTUuOTIgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIA0KICBuZXh1c19vbGRfc3RvcmFnZSAgICAgICB4MjJ2MCBWd2ktYS10 LS0tICAgIDUuMDB0IHRwX2JpZyAgICAgICAgICAgICA0Ny43OCAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgDQogIG51c2tuYWNrZXItc3RhZ2luZyAgICAgIHgyMnYw IFZ3aS1hb3QtLS0gPDEyOC4wOWcgdHBfYmlnICAgICAgICAgICAgIDk5Ljg0ICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCiAgbnVzc2tuYWNrZXIxICAgICAgICAg ICAgeDIydjAgVndpLWFvdC0tLSA8MTI4LjA5ZyB0cF9iaWcgICAgICAgICAgICAgMjcuNjkg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KICBudXNza25hY2tlcjIg ICAgICAgICAgICB4MjJ2MCBWd2ktYW90LS0tIDwxMjguMDlnIHRwX2JpZyAgICAgICAgICAg ICAyMS44MSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQogIG9wZW52 cG4tbmV3ICAgICAgICAgICAgIHgyMnYwIFZ3aS1hb3QtLS0gICA2NC4xMGcgdHBfYmlnICAg ICAgICAgICAgIDE3Ljk3ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAN CiAgcm9ja2V0ICAgICAgICAgICAgICAgICAgeDIydjAgVndpLWFvdC0tLSAgPDgwLjA0ZyB0 cF9zc2QgICAgICAgICAgICAgOTcuNDcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIA0KICByb2dlcjIgICAgICAgICAgICAgICAgICB4MjJ2MCBWd2ktYS10LS0tICAg NjQuMTBnIHRwX3NzZCAgICAgICAgICAgICAyNC41NiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgDQogIHRtb2IxICAgICAgICAgICAgICAgICAgIHgyMnYwIC13aS1h LS0tLS0gPDUwMC4wNGcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAvZGV2L21kMTI1Ojg3MzktMTMwMDUgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICANCiAgdG1vYjIgICAgICAgICAgICAgICAgICAgeDIy djAgLXdpLWEtLS0tLSA8NTAwLjA0ZyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIC9kZXYvbWQxMjU6MTMwMDYtMTcyNzIgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KICB0b3VrLWVsazQgICAgICAgICAg ICAgICB4MjJ2MCAtd2ktYW8tLS0tIDw0MDAuMDhnICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgL2Rldi9tZDEyNjoxNzQ3Ny0y MDg5MCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQogIHRwX2JpZyAgICAg ICAgICAgICAgICAgIHgyMnYwIHR3aS1hb3QtLS0gICAxMC4zMnQgICAgICAgICAgICAgICAg ICAgIDUzLjY0ICAzLjI3ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbdHBfYmlnX3Rk YXRhXTowLTkwMTg3ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCiAgW3Rw X2JpZ190ZGF0YV0gICAgICAgICAgeDIydjAgVHdpLWFvLS0tLSAgIDEwLjMydCAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC9k ZXYvbWQxMjY6MC0xNzQ3NiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IA0KICBbdHBfYmlnX3RkYXRhXSAgICAgICAgICB4MjJ2MCBUd2ktYW8tLS0tICAgMTAuMzJ0 ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgL2Rldi9tZDEyNjoyMTk4NC05NDY5NCAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgDQogIFt0cF9iaWdfdG1ldGFdICAgICAgICAgIHgyMnYwIGV3aS1hb3ItLS0g ICAgNC4xMGcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgMTAw LjAwICAgICAgICAgICBbdHBfYmlnX3RtZXRhX3JpbWFnZV8wXTowLTM0LFt0cF9iaWdfdG1l dGFfcmltYWdlXzFdOjAtMzQNCiAgW3RwX2JpZ190bWV0YV9yaW1hZ2VfMF0geDIydjAgaXdp LWFvci0tLSAgICA0LjEwZyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIC9kZXYvc2RhMzozMC02NCAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIA0KICBbdHBfYmlnX3RtZXRhX3JpbWFnZV8xXSB4 MjJ2MCBpd2ktYW9yLS0tICAgIDQuMTBnICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgL2Rldi9zZGIzOjMwLTY0ICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQogIFt0cF9iaWdfdG1ldGFfcm1l dGFfMF0gIHgyMnYwIGV3aS1hb3ItLS0gIDEyMC4wMG0gICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvZGV2L3NkYTM6MjktMjkg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCiAgW3RwX2JpZ190 bWV0YV9ybWV0YV8xXSAgeDIydjAgZXdpLWFvci0tLSAgMTIwLjAwbSAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC9kZXYvc2Ri MzoyOS0yOSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KICB0 cF9zc2QgICAgICAgICAgICAgICAgICB4MjJ2MCB0d2ktYW90LS0tICAgIDEuMjB0ICAgICAg ICAgICAgICAgICAgICA3NS4wNCAgMS42MSAgICAgICAgICAgICAgICAgICAgICAgICAgICAg W3RwX3NzZF90ZGF0YV06MC0xMDQ4NiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgDQogIFt0cF9zc2RfdGRhdGFdICAgICAgICAgIHgyMnYwIFR3aS1hby0tLS0gICAgMS4y MHQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAvZGV2L21kMTI1OjAtODczOCAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICANCiAgW3RwX3NzZF90ZGF0YV0gICAgICAgICAgeDIydjAgVHdpLWFvLS0t LSAgICAxLjIwdCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIC9kZXYvbWQxMjU6MTcyNzMtMTkwMjAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIA0KICBbdHBfc3NkX3RtZXRhXSAgICAgICAgICB4MjJ2MCBl d2ktYW9yLS0tICAgPDIuMTFnICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIDEwMC4wMCAgICAgICAgICAgW3RwX3NzZF90bWV0YV9yaW1hZ2VfMF06MC0xNyxb dHBfc3NkX3RtZXRhX3JpbWFnZV8xXTowLTE3DQogIFt0cF9zc2RfdG1ldGFfcmltYWdlXzBd IHgyMnYwIGl3aS1hb3ItLS0gICA8Mi4xMWcgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvZGV2L3NkYTM6MTEtMjggICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCiAgW3RwX3NzZF90bWV0YV9y aW1hZ2VfMV0geDIydjAgaXdpLWFvci0tLSAgIDwyLjExZyAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC9kZXYvc2RiMzoxMS0y OCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KICBbdHBfc3Nk X3RtZXRhX3JtZXRhXzBdICB4MjJ2MCBld2ktYW9yLS0tICAxMjAuMDBtICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgL2Rldi9z ZGEzOjEwLTEwICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQog IFt0cF9zc2RfdG1ldGFfcm1ldGFfMV0gIHgyMnYwIGV3aS1hb3ItLS0gIDEyMC4wMG0gICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAvZGV2L3NkYjM6MTAtMTAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICANCg0KDQo+IA0KPj4+IEFyZSB0aGUgTFZzIHNwbGl0IGFjcm9zcyBSQUlENSBQVnMg YnkgYW55IGNoYW5jZT8NCj4gDQo+IE1pY2hhbD4gcmFpZDVzIGFyZSB1c2VkIGFzIFBWcywg YnV0IGEgc2luZ2xlIGxvZ2ljYWwgdm9sdW1lIGFsd2F5cyB1c2VzIG9uZSBvbmx5DQo+IE1p Y2hhbD4gb25lIHBoeXNpY2FsIHZvbHVtZSB1bmRlcm5lYXRoIChpZiB0aGF0J3Mgd2hhdCB5 b3UgbWVhbnQgYnkgc3BsaXQgYWNyb3NzKS4NCj4gDQo+IE9rLCB0aGF0J3Mgd2hhdCBJIHdh cyBhc2tpbmcgYWJvdXQuICBJdCBzaG91bGRuJ3QgbWF0dGVyLi4uIGJ1dCBqdXN0DQo+IHRy eWluZyB0byBjaGFzZSBkb3duIHRoZSBkZXRhaWxzLg0KPiANCj4gDQo+Pj4gSXQncyBub3Qg Y2xlYXIgaWYgeW91IGNhbiByZXBsaWNhdGUgdGhlIHByb2JsZW0gd2l0aG91dCB1c2luZw0K Pj4+IGx2bS10aGluLCBidXQgdGhhdCdzIHdoYXQgSSBzdXNwZWN0IHlvdSBtaWdodCBiZSBo YXZpbmcgcHJvYmxlbXMgd2l0aC4NCj4gDQo+IE1pY2hhbD4gSSdsbCBiZSB0cnlpbmcgdG8g ZG8gdGhhdCwgdGhvdWdoIHRoZSBoZWF2aWVyIHRlc3RzIHdpbGwgaGF2ZSB0byB3YWl0DQo+ IE1pY2hhbD4gdW50aWwgSSBtb3ZlIGFsbCBWTXMgdG8gb3RoZXIgaG9zdHMgKGFzIHRoYXQg aXMvd2FzIG91ciBwcm9kdWN0aW9uIG1hY2huaWUpLg0KPiANCj4gU3VyZSwgbWFrZXMgc2Vu c2UuDQo+IA0KPj4+IENhbiB5b3UgZ2l2ZSB1cyB0aGUgdmVyc2lvbnMgb2YgdGhlIHlvdXIg dG9vbHMsIGFuZCBleGFjdGx5IGhvdyB5b3UNCj4+PiBzZXR1cCB5b3VyIHRlc3QgY2FzZXM/ ICBIb3cgbG9uZyBkb2VzIGl0IHRha2UgdG8gZmluZCB0aGUgcHJvYmxlbT8NCg0KUmVnYXJk aW5nIHRoaXMsIGN1cnJlbnRseToNCg0Ka2VybmVsOiAgNS40LjAtMC5icG8uNC1hbWQ2NCAj MSBTTVAgRGViaWFuIDUuNC4xOS0xfmJwbzEwKzEgKDIwMjAtMDMtMDkpIHg4Nl82NCBHTlUv TGludXggKHdhcyBhbHNvIGhhcHBlbmluZyB3aXRoIDUuMi4wLTAuYnBvLjMtYW1kNjQpDQpM Vk0gdmVyc2lvbjogICAgIDIuMDMuMDIoMikgKDIwMTgtMTItMTgpDQpMaWJyYXJ5IHZlcnNp b246IDEuMDIuMTU1ICgyMDE4LTEyLTE4KQ0KRHJpdmVyIHZlcnNpb246ICA0LjQxLjANCm1k YWRtIC0gdjQuMSAtIDIwMTgtMTAtMDENCg0KPiANCj4gTWljaGFsPiBXaWxsIGdldCBhbGwg dGhlIGRldGFpbHMgdG9tbW9yb3cgKHRoZSBob3N0IGlzIG9uIHVwIHRvIGRhdGUgZGViaWFu DQo+IE1pY2hhbD4gYnVzdGVyLCB0aGUgVk1zIGFyZSBtaXggb2YgYXJjaGxpbnV4ZXMgYW5k IGRlYmlhbnMgKGFuZCB0aGUgaXNzdWUNCj4gTWljaGFsPiBoYXBwZW5lZCBvbiBib3RoKSku DQo+IA0KPiBNaWNoYWw+IEFzIGZvciBob3cgbG9uZywgaXQncyBhIGhpdCBhbmQgbWlzcy4g U29tZXRpbWVzIHdyaXRpbmcgYW5kIHJlYWRpbmcgYmFjaw0KPiBNaWNoYWw+IH4xNmdiIGZp bGUgZmFpbHMgKHRoZSBjaGVrc3VtIHJlYWQgYmFjayBkaWZmZXJzIGZyb20gd2hhdCB3YXMg d3JpdHRlbikNCj4gTWljaGFsPiBhZnRlciAyLTMgdHJpZXMuIFRoYXQncyBvbiB0aGUgaG9z dC4NCj4gDQo+IE1pY2hhbD4gT24gdGhlIGd1ZXN0LCBpdCdzIGJlZW4gKHNvIGZhcikgYSBn dWFyYW50ZWVkIHRoaW5nIHdoZW4gd2Ugd2VyZQ0KPiBNaWNoYWw+IGNyZWF0aW5nIHZlcnkg bGFyZ2UgdGFyIGZpbGUgKDkwMGdiKykuIEFzIGZvciBwYXN0IHR3byB3ZWVrcyB3ZSB3ZXJl DQo+IE1pY2hhbD4gdW5hYmxlIHRvIGNyZWF0ZSB0aGF0IGZpbGUgd2l0aG91dCBlcnJvcnMg ZXZlbiBvbmNlLg0KPiANCj4gT3VjaCEgIFRoYXQncyBub3QgZ29vZC4gIEp1c3QgdG8gY29u ZmlybSwgdGhlc2UgY29ycnVwdGlvbnMgYXJlIGFsbCBpbg0KPiBhIHRoaW4tbHYgYmFzZWQg ZmlsZXN5c3RlbSwgcmlnaHQ/ICAgSSdkIGJlIGludGVyZXN0ZWQgdG8ga25vdyBpZiB5b3UN Cj4gY2FuIGNyZWF0ZSBhbm90aGVyIHBsYWluIExWIGFuZCBjYXVzZSB0aGUgc2FtZSBlcnJv ci4gIFRyeWluZyB0bw0KPiBzaW1wbGlmeSB0aGUgcG90ZW50aWFsIHByb2JsZW1zLg0KDQpJ IGhhdmUgYmVlbiB0cnlpbmcgdG8gLSBidXQgc28gZmFyIGRpZG4ndCBtYW5hZ2UgdG8gcmVw bGljYXRlIHRoaXMgd2l0aDoNCg0KLSBhIHBoeXNpY2FsIHBhcnRpdGlvbg0KLSBmaWxlc3lz dGVtIGRpcmVjdGx5IG9uIGEgcGh5c2ljYWwgcGFydGl0aW9uDQotIGZpbGVzeXN0ZW0gZGly ZWN0bHkgb24gbWRyYWlkDQotIGZpbGVzeXN0ZW0gZGlyZWN0bHkgb24gYSBsaW5lYXIgdm9s dW1lDQoNCk5vdGUgdGhhdCB0aGlzIF9kb2Vzbid0XyBpbXBseSB0aGF0IEkgX2Fsd2F5c18g Z2V0IGVycm9ycyBpZiBsdm0tdGhpbiBpcyBpbiB1c2UgLSBhcyBJIGFsc28gaGFkIGxlbmd0 aHkgcGVyaW9kIG9mIGF0dGVtcHRzIHRvIGNhdXNlIGNvcnJ1cHRpb24gb24gc29tZSB0aGlu IHZvbHVtZSB3L28gYW55IHN1Y2Nlc3NlcyBlaXRoZXIuIEJ1dCB0aGUgb25lcyB0aGF0IGZh aWxlZCBoYWQgdGhvc2UgaW4gY29tbW9uIChzbyBmYXIpOiBtZCAmIGx2bS10aGluIC0gd2l0 aCA0IEtpQiBwaWVjZShzKSBiZWluZyBpbmNvcnJlY3QNCg0KPiANCj4gDQo+Pj4gQ2FuIHlv dSBjb21waWxlIHRoZSBuZXdzdCBrZXJuZWwgYW5kIG5ld2VzdCB0aGluIHRvb2xzIGFuZCB0 cnkgdGhlbQ0KPj4+IG91dD8NCj4gDQo+IE1pY2hhbD4gSSBjYW4sIGJ1dCBhIGJpdCBsYXRl ciAob25jZSB3ZSBtb3ZlIFZNcyBvdXQgb2YgdGhlIGhvc3QpLg0KPiANCj4+Pg0KPj4+IEhv dyBsb25nIGRvZXMgaXQgdGFrZSB0byByZXBsaWNhdGUgdGhlIGNvcnJ1cHRpb24/DQo+Pj4N Cj4gDQo+IE1pY2hhbD4gV2hlbiBpdCBoYXBwZW5zLCBpdCdzIHVzdWFsbHkgZmV3IHRyaWVz IHRyaWVzIG9mIHdyaXRpbmcgYSAxNmdiIGZpbGUNCj4gTWljaGFsPiB3aXRoIHJhbmRvbSBw YXR0ZXJucyBhbmQgcmVhZGluZyBpdCBiYWNrIChkaXJlY3RseSBvbiBob3N0KS4gVGhlDQo+ IE1pY2hhbD4gaXJyaXRhdGluZyB0aGluZyBpcyB0aGF0IGl0IGNhbiBiZSBzb21ld2hhdCBo YXJkIHRvIHJlcHJvZHVjZSAoZS5nLg0KPiBNaWNoYWw+IGFmdGVyIG1hY2hpbmUncyByZWJv b3QpLg0KPiANCj4+PiBTb3JyeSBmb3IgYWxsIHRoZSBxdWVzdGlvbnMsIGJ1dCB1bnRpbCB0 aGVyZSdzIGEgdGVzdCBjYXNlIHdoaWNoIGlzDQo+Pj4gcmVwZWF0YWJsZSwgaXQncyBnb2lu ZyB0byBiZSBoYXJkIHRvIGNoYXNlIHRoaXMgZG93bi4NCj4+Pg0KPj4+IEkgd29uZGVyIGlm IHJ1bm5pbmcgJ2ZpbycgdGVzdHMgd291bGQgYmUgc29tZXRoaW5nIHRvIHRyeT8NCj4+Pg0K Pj4+IEFuZCBhbHNvIGNoYW5naW5nIHlvdXIgUkFJRDUgc2V0dXAgdG8gdXNlIHRoZSBkZWZh dWx0IHN0cmlkZSBhbmQNCj4+PiBzdHJpcGUgd2lkdGhzLCBpbnN0ZWFkIG9mIHRoZSBsYXJn ZSB2YWx1ZXMgeW91J3JlIHVzaW5nLg0KPiANCj4gTWljaGFsPiBUaGUgcmFpZDUgaXMgdXNp bmcgbWRhZG0ncyBkZWZhdWx0cyAod2hpY2ggaXMgNTEyIEtpQiB0aGVzZSBkYXlzIGZvciBh DQo+IE1pY2hhbD4gY2h1bmspLiBMVk0gb24gdG9wIGlzIHVzaW5nIG11Y2ggbG9uZ2VyIGV4 dGVudHMgKGFzIHdlIGRvbid0IHJlYWxseSBuZWVkDQo+IE1pY2hhbD4gNG1iIGdyYW51bGFy aXR5KSBhbmQgdGhlIGx2bS10aGluIGNodW5rcyB3ZXJlIHNldCB0byBtYXRjaCAoYW5kIGFs aWduKQ0KPiBNaWNoYWw+IHRvIHJhaWQncyBzdHJpcGUuDQo+IA0KPj4+DQo+Pj4gR29vZCBs dWNrIQ0KPj4+DQo+IFJvZ2VyPiBJIGhhdmUgbm90IGFzIG9mIHlldCBzZWVuIHdyaXRlIGNv cnJ1cHRpb24gKGV4Y2VwdCB3aGVuIGEgdmVuZG9ycyBkaXNrDQo+IFJvZ2VyPiB3YXMgcmVz ZXR0aW5nIGFuZCBpdCB3YXMgbHlpbmcgYWJvdXQgaGF2aW5nIHdyaXR0ZW4gdGhlIGRhdGEg cHJpb3IgdG8NCj4gUm9nZXI+IHRoZSBjcmFzaCwgdGhlc2Ugd2VyZSBzc2RzLCBpZiB5b3Vy IGRpc2sgd3JpdGUgY2FjaGUgaXMgb24gYW5kIHlvdQ0KPiBSb2dlcj4gaGF2ZSBhIGRpc2sg cmVzZXQgdGhpcyBjYW4gYWxzbyBoYXBwZW4pLCBidXQgaGF2ZSBub3Qgc2VlbiAibG9zdA0K PiBSb2dlcj4gd3JpdGVzIiBvdGhlcndpc2UsIGJ1dCB3b3VsZCBleHBlY3QgdGhlIDIgcmVh ZCBjb3JydXB0aW9uIEkgaGF2ZSBzZWVuDQo+IFJvZ2VyPiB0byBhbHNvIGJlIGFibGUgdG8g Y2F1c2Ugd3JpdGUgaXNzdWVzLiAgU28gZm9yIHRoYXQgbG9vayBmb3Igc2NzaQ0KPiBSb2dl cj4gbm90aWZpY2F0aW9ucyBmb3IgZGlzayByZXNldHMgdGhhdCBzaG91bGQgbm90IGhhcHBl bi4NCj4+Pg0KPiBSb2dlcj4gSSBoYXZlIGhhZCBhICJiYWQiIGNvbnRyb2xsZXIgY2F1c2Ug cmVhZCBjb3JydXB0aW9ucywgdGhvc2UNCj4gUm9nZXI+IGNvcnJ1cHRpb25zIHdvdWxkIG1v dmUgYXJvdW5kLCByZXBsYWNpbmcgdGhlIGNvbnRyb2xsZXIgcmVzb2x2ZWQgaXQsDQo+IFJv Z2VyPiBzbyB0aGVyZSBtYXkgYmUgbGFjayBvZiBlcnJvciBjaGVja2luZyAiaW5zaWRlIiBz b21lIHBhdGhzIGluIHRoZQ0KPiBSb2dlcj4gY2FyZC4gIEx1Y2t5IEkgaGFkIGEgbnVtYmVy IG9mIHRoZXNlIGNvbnRyb2xsZXJzIGFuZCBoYWQgY29sZCBzcGFyZXMNCj4gUm9nZXI+IGZv ciB0aGVtLiAgVGhlIGdpdmUgYXdheSBoZXJlIHdhcyAyIHNlcGFyYXRlIGJ1c2VzIHdpdGgg YWxtb3N0DQo+IFJvZ2VyPiBpZGVudGljYWwgbG9hZCB3aXRoIDYgc2VwYXJhdGUgZGlza3Mg ZWFjaCBhbmQgYWxsMTIgZGlza3Mgb24gMiBidXNlcw0KPiBSb2dlcj4gaGFkIGJldHdlZW4g NDctNTIgc2NzaSBlcnJvcnMsIHdoaWNoIHBvaW50cyB0byB0aGUgb25seSBjb21wb25lbnQN Cj4gUm9nZXI+IHNoYXJlZCAodGhlIGNvbnRyb2xsZXIpLg0KPj4+DQo+IFJvZ2VyPiBUaGUg YmFja3BsYW5lIGFuZCBjYWJsZXMgYXJlIHVubGlrZWx5IGluIGdlbmVyYWwgY2F1c2UgdGhp cywgdGhlcmUgaXMNCj4gUm9nZXI+IHRvbyBtdWNoIGVycm9yIGNoZWNraW5nIGJldHdlZW4g dGhlIGNvbnRyb2xsZXIgYW5kIHRoZSBkaXNrIGZyb20gd2hhdA0KPiBSb2dlcj4gSSBrbm93 Lg0KPj4+DQo+IFJvZ2VyPiBJIGhhdmUgaGFkIHByZS1wY2llIGJ1cyAoUENJLVggYnVzLCAy IHNsb3RzIHNoYXJlZCwgYm90aCBzZXQgdG8gMTMzDQo+IFJvZ2VyPiBjYXVzZSByYW5kb20g cmVhZCBjb3JydXB0aW9ucywgbG93ZXJpbmcgc3BlZWQgdG8gMTAwIGZpeGVkIGl0KSwgdGhp cw0KPiBSb2dlcj4gb25lIHdhcyBkdXBsaWNhdGVkIG9uIG11bHRpcGxlIGlkZW50aWNhbCBw aWVjZXMgb2YgaHcgd2l0aCBhbGwNCj4gUm9nZXI+IGRpZmZlcmVudCBwYXJ0cyBvbiB0aGUg ZHVwbGljYXRpb24gbWFjaGluZS4NCj4+Pg0KPiBSb2dlcj4gSSBoYXZlIGFsc28gc2VlbiBs b3N0IHdyaXRlcyAoZnJvbSBzb2Z0d2FyZSkgYmVjYXVzZSBzb21lb25lIGRpZCBhDQo+IFJv Z2VyPiBzZWVrIHdpdGhvdXQgZG9pbmcgYSBmbHVzaCB3aGljaCBpbiBzb21lIHZlcnNpb25z IG9mIHRoZSBsaWJzIGxvc2VzDQo+IFJvZ2VyPiB0aGUgdW5maWxsZWQgYmxvY2sgd2hlbiB0 aGUgc2VlayBoYXBwZW5zICh0aGlzIGlzIG5vdGVkIGluIHRoZSBtYW4NCj4gUm9nZXI+IHBh Z2UsIGFuZCBJIHNhdyBpdCAyMHllYXJzIGFnbywgaXQgaXMgc3RpbGwgbm90ZWQgaW4gdGhl IG1hbiBwYWdlLCBzbw0KPiBSb2dlcj4gbm8gaWRlYSBpZiBpdCB3YXMgZXZlciBmaXhlZCku ICBTbyBoYXMgbW9yZSB0aGFuIG9uZSBhcHBsaWNhdGlvbiBiZWVuDQo+IFJvZ2VyPiBub3Rl ZCB0byBzZWUgdGhlIGNvcnJ1cHRpb24/DQo+Pj4NCj4gUm9nZXI+IFNvIG9uZSBxdWVzdGlv biwgaGF2ZSB5b3Ugc2VlbiB0aGUgY29ycnVwdGlvbiBpbiBhIHBhdGggdGhhdCB3b3VsZA0K PiBSb2dlcj4gcmVseSBvbiBvbmUgY29udHJvbGxlciwgb3IgYWxsIGNvcnJ1cHRpb25zIHlv dSBoYXZlIHNlZW4gaW52b2x2aW5nDQo+IFJvZ2VyPiBtb3JlIHRoYW4gb25lIGNvbnRyb2xs ZXI/ICBJc29sYXRlIGFuZCB0ZXN0IGVhY2ggY29udHJvbGxlciBpZiB5b3UNCj4gUm9nZXI+ IGNhbiwgb3IgaWYgeW91IGNhbiBhZmZvcmQgdG8gcmVwbGFjZSBpdCBhbmQgc2VlIGlmIGl0 IGNvbnRpbnVlcy4NCj4+Pg0KPj4+DQo+IFJvZ2VyPiBPbiBUaHUsIE1heSA3LCAyMDIwIGF0 IDEyOjMzIFBNIE1pY2hhbCBTb2x0eXMgPG1zb2x0eXNwbEB5YW5kZXgucGw+IHdyb3RlOg0K Pj4+Pj4NCj4+Pj4+IE5vdGU6IHRoaXMgaXMganVzdCBnZW5lcmFsIHF1ZXN0aW9uIC0gaWYg YW55b25lIGV4cGVyaWVuY2VkIHNvbWV0aGluZyBzaW1pbGFyIG9yIGNvdWxkIHN1Z2dlc3Qg aG93IHRvIHBpbnBvaW50IC8gdmVyaWZ5IHRoZSBhY3R1YWwgY2F1c2UuDQo+Pj4+Pg0KPj4+ Pj4gVGhhbmtzIHRvIGJ0cmZzJ3MgY2hlY2tzdW1taW5nIHdlIGRpc2NvdmVyZWQgc29tZXdo YXQgKGV2ZW4gaWYgcXVpdGUgcmFyZSkgbmFzdHkgc2lsZW50IGNvcnJ1cHRpb24gZ29pbmcg b24gb24gb25lIG9mIG91ciBob3N0cy4gT3IgcGVyaGFwcyAiY29ycnVwdGlvbiIgaXMgbm90 IHRoZSBjb3JyZWN0IHdvcmQgLSB0aGUgZmlsZXMgc2ltcGx5IGhhdmUgcHJlY2lzZSA0a2Ig KDEgcGFnZSkgb2YgaW5jb3JyZWN0IGRhdGEuIFRoZSBpbmNvcnJlY3QgcGllY2VzIG9mIGRh dGEgbG9vayBvbiB0aGVpciBvd24gZmluZSAtIGFzIHNvbWV0aGluZyB0aGF0IHdhcyBwcmV2 aW91c2x5IGluIHRoZSBwbGFjZSwgb3Igd3JpdHRlbiBmcm9tIHdyb25nIHNvdXJjZS4NCj4+ Pj4+DQo+Pj4+PiBUaGUgaGFyZHdhcmUgaXMgKGNhbiBwcm92aWRlIG1vcmUgZGV0YWlsZWQg aW5mbyBvZiBjb3Vyc2UpOg0KPj4+Pj4NCj4+Pj4+IC0gU3VwZXJtaWNybyBYOURSNy1MTjRG DQo+Pj4+PiAtIG9uYm9hcmQgTFNJIFNBUzIzMDggY29udHJvbGxlciAoMiBzZmYtODA4NyBj b25uZWN0b3JzLCAxIGNvbm5lY3RlZCB0byBiYWNrcGxhbmUpDQo+Pj4+PiAtIDk2IGdiIHJh bSAoZWNjKQ0KPj4+Pj4gLSAyNCBkaXNrIGJhY2twbGFuZQ0KPj4+Pj4NCj4+Pj4+IC0gMSBh cnJheSBjb25uZWN0ZWQgZGlyZWN0bHkgdG8gbHNpIGNvbnRyb2xsZXIgKDQgZGlza3MsIG1k cmFpZDUsIGludGVybmFsIGJpdG1hcCwgNTEya2IgY2h1bmspDQo+Pj4+PiAtIDEgYXJyYXkg b24gdGhlIGJhY2twbGFuZSAoNCBkaXNrcywgbWRyYWlkNSwgam91cm5hbGVkKQ0KPj4+Pj4g LSBqb3VybmFsIGZvciB0aGUgYWJvdmUgYXJyYXkgaXM6IG1kcmFpZDEsIDIgc3NkIGRpc2tz IChtaWNyb24gNTMwMCBwcm8gZGlza3MpDQo+Pj4+PiAtIDEgYnRyZnMgcmFpZDEgYm9vdCBh cnJheSBvbiBtb3RoZXJib2FyZCdzIHNhdGEgcG9ydHMgKG9sZGVyIGJ1dCBzdGlsbCBmaW5l IGludGVsIHNzZHMgZnJvbSBEQyAzNTAwIHNlcmllcykNCj4+Pj4+DQo+Pj4+PiBSYWlkIDUg YXJyYXlzIGFyZSBpbiBsdm0gdm9sdW1lIGdyb3VwLCBhbmQgdGhlIGxvZ2ljYWwgdm9sdW1l cyBhcmUgdXNlZCBieSBWTXMuIFNvbWUgb2YgdGhlIHZvbHVtZXMgYXJlIGxpbmVhciwgc29t ZSBhcmUgdXNpbmcgdGhpbi1wb29scyAod2l0aCBtZXRhZGF0YSBvbiB0aGUgYWZvcmVtZW50 aW9uZWQgaW50ZWwgc3NkcywgaW4gbWlycm9yZWQgY29uZmlnKS4gTFZNDQo+Pj4+PiB1c2Vz IGxhcmdlIGV4dGVudCBzaXplcyAoMTIwbSkgYW5kIHRoZSBjaHVuay1zaXplIG9mIHRoaW4t cG9vbHMgaXMgc2V0IHRvIDEuNW0gdG8gbWF0Y2ggdW5kZXJseWluZyByYWlkIHN0cmlwZS4g RXZlcnl0aGluZyBpcyBjbGVhbmx5IGFsaWduZWQgYXMgd2VsbC4NCj4+Pj4+DQo+Pj4+PiBX aXRoIGEgZG96ZSBvZiB0ZXN0aW5nIHdlIG1hbmFnZWQgdG8gcm91Z2hseSBydWxlIG91dCB0 aGUgZm9sbG93aW5nIGVsZW1lbnRzIGFzIGJlaW5nIHRoZSBjYXVzZToNCj4+Pj4+DQo+Pj4+ PiAtIHFlbXUva3ZtIChpc3N1ZSBvY2N1cmVkIGRpcmVjdGx5IG9uIGhvc3QpDQo+Pj4+PiAt IGJhY2twbGFuZSAoaXNzdWUgb2NjdXJlZCBvbiBkaXNrcyBkaXJlY3RseSBjb25uZWN0ZWQg dmlhIExTSSdzIDJuZCBjb25uZWN0b3IpDQo+Pj4+PiAtIGNhYmxlIChhcyBhIGFib3ZlLCB0 d28gZGlmZmVyZW50IGNhYmxlcykNCj4+Pj4+IC0gbWVtb3J5ICh1bmxpa2VseSAtIEVDQyBm b3Igb25jZSwgdGhvcm91Z2hseSB0ZXN0ZWQsIG5vIGVycm9ycyBldmVyIHJlcG9ydGVkIHZp YSBlZGFjLXV0aWwgb3IgbWVtdGVzdCkNCj4+Pj4+IC0gbWRhZG0gam91cm5hbGluZyAoaXNz dWUgb2NjdXJlZCBvbiBwbGFpbiBtZHJhaWQgY29uZmlndXJhdGlvbiBhcyB3ZWxsKQ0KPj4+ Pj4gLSBkaXNrcyB0aGVtc2VsdmVzIChpc3N1ZSBvY2N1cmVkIG9uIHR3byBzZXBhcmF0ZSBt ZGFkbSBhcnJheXMpDQo+Pj4+PiAtIGZpbGVzeXN0ZW0gKGlzc3VlIG9jY3VyZWQgb24gYm90 aCBidHJmcyBhbmQgZXh0NCAoY2hlY2tzdW1lZCBtYW51YWxseSkgKQ0KPj4+Pj4NCj4+Pj4+ IFdlIGRpZCBub3QgbWFuYWdlIHRvIHJ1bGUgb3V0ICh0aG91Z2ggc29tZXdoYXQgX2hpZ2hs eV8gdW5saWtlbHkpOg0KPj4+Pj4NCj4+Pj4+IC0gbHZtIHRoaW4gKGlzc3VlIGFsd2F5cyAt IHNvIGZhciAtIG9jY3VyZWQgb24gbHZtIHRoaW4gcG9vbHMpDQo+Pj4+PiAtIG1kcmFpZCAo aXNzdWUgYWx3YXlzIC0gc28gZmFyIC0gb24gbWRyYWlkIG1hbmFnZWQgYXJyYXlzKQ0KPj4+ Pj4gLSBrZXJuZWwgKHRlc3RlZCB3aXRoIC0gaW4gdGhpcyBjYXNlIC0gZGViaWFuJ3MgNS4y IGFuZCA1LjQga2VybmVscywgaGFwcGVuZWQgd2l0aCBib3RoIC0gc28gaXQgd291bGQgaW1w bHkgcmF0aGVyIGFscmVhZHkgbG9uZ3N0YW5kaW5nIGJ1ZyBzb21ld2hlcmUpDQo+Pj4+Pg0K Pj4+Pj4gQW5kIGZpbmFsbHkgLSBzbyBmYXIgLSB0aGUgaXNzdWUgbmV2ZXIgb2NjdXJlZDoN Cj4+Pj4+DQo+Pj4+PiAtIGRpcmVjdGx5IG9uIGEgZGlzaw0KPj4+Pj4gLSBkaXJlY3RseSBv biBtZHJhaWQNCj4+Pj4+IC0gb24gbGluZWFyIGx2bSB2b2x1bWUgb24gdG9wIG9mIG1kcmFp ZA0KPj4+Pj4NCj4+Pj4+IEFzIGZhciBhcyB0aGUgaXNzdWUgZ29lcyBpdCdzOg0KPj4+Pj4N Cj4+Pj4+IC0gYWx3YXlzIGEgNGtiIGNodW5rIHRoYXQgaXMgaW5jb3JyZWN0IC0gaW4gYSB+ MSB0YiBmaWxlIGl0IGNhbiBiZSBmcm9tIGEgZmV3IHRvIGZldyBkb3plbnMgb2Ygc3VjaCBj aHVua3MNCj4+Pj4+IC0gd2UgYWxzbyBmb3VuZCAob3IgcmF0aGVyIGJ0cmZzIHNjcnViIGRp ZCkgYSBmZXcgc21hbGwgZGFtYWdlZCBmaWxlcyBhcyB3ZWxsDQo+Pj4+PiAtIHRoZSBjaHVu a3MgbG9vayBsaWtlIGEgY29ycmVjdCBwaWVjZSBvZiBkaWZmZXJlbnQgb3IgcHJldmlvdXMg ZGF0YQ0KPj4+Pj4NCj4+Pj4+IFRoZSA0a2IgaXMgd2VsbCwgd2VpcmQgPyBEb2Vzbid0IHJl YWxseSBtYXR0ZXIgYW55IGNodW5rL3N0cmlwZXMgc2l6ZXMgYW55d2hlcmUgYWNyb3NzIHRo ZSBzdGFjayAobHZtIC0gMTIwbSBleHRlbnRzLCAxLjVtIGNodW5rcyBvbiB0aGluIHBvb2xz OyBtZHJhaWQgLSBkZWZhdWx0IDUxMmtiIGNodW5rcykuIEl0IGRvZXMgbmljZWx5IGZpdCBh IHBhZ2UgdGhvdWdoIC4uLg0KPj4+Pj4NCj4+Pj4+IEFueXdheSwgaWYgYW55b25lIGhhcyBh bnkgaWRlYXMgb3Igc3VnZ2VzdGlvbnMgd2hhdCBjb3VsZCBiZSBoYXBwZW5pbmcgKHBlcmhh cHMgd2l0aCB0aGlzIHBhcnRpY3VsYXIgbW90aGVyYm9hcmQgb3IgdmVuZG9yKSBvciBob3cg dG8gcGlucG9pbnQgdGhlIGNhdXNlIC0gSSdsbCBiZSBncmF0ZWZ1bCBmb3IgYW55Lg0KPj4+ DQo+IA0KDQo= From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mimecast-mx02.redhat.com (mimecast06.extmail.prod.ext.rdu2.redhat.com [10.11.55.22]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0856110CB26F for ; Fri, 8 May 2020 11:10:30 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [205.139.110.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3BD7E1859163 for ; Fri, 8 May 2020 11:10:30 +0000 (UTC) References: <24244.30530.155404.154787@quad.stoffel.home> <24244.44462.151185.626440@quad.stoffel.home> From: Michal Soltys Message-ID: Date: Fri, 8 May 2020 13:10:22 +0200 MIME-Version: 1.0 In-Reply-To: <24244.44462.151185.626440@quad.stoffel.home> Content-Language: en-US-large Content-Transfer-Encoding: 8bit Subject: Re: [linux-lvm] [general question] rare silent data corruption when writing data Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: Content-Type: text/plain; charset="us-ascii" To: John Stoffel Cc: Linux RAID , Roger Heflin , "linux-lvm@redhat.com" note: as suggested, I'm also CCing this to linux-lvm; the full context with replies starts at: https://www.spinics.net/lists/raid/msg64364.html There is also the initial post at the bottom as well. On 5/8/20 2:54 AM, John Stoffel wrote: >>>>>> "Michal" == Michal Soltys writes: > > Michal> On 20/05/07 23:01, John Stoffel wrote: >>>>>>>> "Roger" == Roger Heflin writes: >>> > Roger> Have you tried the same file 2x and verified the corruption is in the > Roger> same places and looks the same? >>> >>> Are these 1tb files VMDK or COW images of VMs? How are these files >>> made. And does it ever happen with *smaller* files? What about if >>> you just use a sparse 2tb file and write blocks out past 1tb to see if >>> there's a problem? > > Michal> The VMs are always directly on lvm volumes. (e.g. > Michal> /dev/mapper/vg0-gitlab). The guest (btrfs inside the guest) detected the > Michal> errors after we ran scrub on the filesystem. > > Michal> Yes, the errors were also found on small files. > > Those errors are in small files inside the VM, which is running btrfs > ontop of block storage provided by your thin-lv, right? > Yea, the small files were in this case on that thin-lv. We also discovered (yesterday) file corruptions with VM hosting gitlab registry - this one was using the same thin-lv underneath, but the guest itself was using ext4 (in this case, docker simply reported incorrect sha checksum on (so far) 2 layers. > > > disks -> md raid5 -> pv -> vg -> lv-thin -> guest QCOW/LUN -> > filesystem -> corruption Those particular guests, yea. The host case it's just w/o "guest" step. But (so far) all corruption ended going via one of the lv-thin layers (and via one of md raids). > > > Michal> Since then we recreated the issue directly on the host, just > Michal> by making ext4 filesystem on some LV, then doing write with > Michal> checksum, sync, drop_caches, read and check checksum. The > Michal> errors are, as I mentioned - always a full 4KiB chunks (always > Michal> same content, always same position). > > What position? Is it a 4k, 1.5m or some other consistent offset? And > how far into the file? And this LV is a plain LV or a thin-lv? I'm > running a debian box at home with RAID1 and I haven't seen this, but > I'm not nearly as careful as you. Can you provide the output of: > What I meant that it doesn't "move" when verifying the same file (aka different reads from same test file). Between the tests, the errors are of course in different places - but it's always some 4KiB piece(s) - that look like correct pieces belonging somewhere else. > /sbin/lvs --version LVM version: 2.03.02(2) (2018-12-18) Library version: 1.02.155 (2018-12-18) Driver version: 4.41.0 Configuration: ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --exec-prefix= --bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin --with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2 --with-cache=internal --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm --with-thin=internal --with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump --with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping --enable-cmdlib --enable-dmeventd --enable-dbus-service --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-lvmpolld --enable-notify-dbus --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync > > too? > > Can you post your: > > /sbin/dmsetup status > > output too? There's a better command to use here, but I'm not an > export. You might really want to copy this over to the > linux-lvm@redhat.com mailing list as well. x22v0-tp_ssd-tpool: 0 2577285120 thin-pool 19 8886/552960 629535/838960 - rw no_discard_passdown queue_if_no_space - 1024 x22v0-tp_ssd_tdata: 0 2147696640 linear x22v0-tp_ssd_tdata: 2147696640 429588480 linear x22v0-tp_ssd_tmeta_rimage_1: 0 4423680 linear x22v0-tp_ssd_tmeta: 0 4423680 raid raid1 2 AA 4423680/4423680 idle 0 0 - x22v0-gerrit--new: 0 268615680 thin 255510528 268459007 x22v0-btrfsnopool: 0 134430720 linear x22v0-gitlab_root: 0 629145600 thin 628291584 629145599 x22v0-tp_ssd_tmeta_rimage_0: 0 4423680 linear x22v0-nexus_old_storage: 0 10737500160 thin 5130817536 10737500159 x22v0-gitlab_reg: 0 2147696640 thin 1070963712 2147696639 x22v0-nexus_old_root: 0 268615680 thin 257657856 268615679 x22v0-tp_big_tmeta_rimage_1: 0 8601600 linear x22v0-tp_ssd_tmeta_rmeta_1: 0 245760 linear x22v0-micron_vol: 0 268615680 linear x22v0-tp_big_tmeta_rimage_0: 0 8601600 linear x22v0-tp_ssd_tmeta_rmeta_0: 0 245760 linear x22v0-gerrit--root: 0 268615680 thin 103388160 268443647 x22v0-btrfs_ssd_linear: 0 268615680 linear x22v0-btrfstest: 0 268615680 thin 40734720 268615679 x22v0-tp_ssd: 0 2577285120 linear x22v0-tp_big: 0 22164602880 linear x22v0-nexus3_root: 0 167854080 thin 21860352 167854079 x22v0-nusknacker--staging: 0 268615680 thin 268182528 268615679 x22v0-tmob2: 0 1048657920 linear x22v0-tp_big-tpool: 0 22164602880 thin-pool 35 35152/1075200 3870070/7215040 - rw no_discard_passdown queue_if_no_space - 1024 x22v0-tp_big_tdata: 0 4295147520 linear x22v0-tp_big_tdata: 4295147520 17869455360 linear x22v0-btrfs_ssd_test: 0 201523200 thin 191880192 201335807 x22v0-nussknacker2: 0 268615680 thin 58573824 268615679 x22v0-tmob1: 0 1048657920 linear x22v0-tp_big_tmeta: 0 8601600 raid raid1 2 AA 8601600/8601600 idle 0 0 - x22v0-nussknacker1: 0 268615680 thin 74376192 268615679 x22v0-touk--elk4: 0 839024640 linear x22v0-gerrit--backup: 0 268615680 thin 228989952 268443647 x22v0-tp_big_tmeta_rmeta_1: 0 245760 linear x22v0-openvpn--new: 0 134430720 thin 24152064 66272255 x22v0-k8sdkr: 0 268615680 linear x22v0-nexus3_storage: 0 10737500160 thin 4976683008 10737500159 x22v0-rocket: 0 167854080 thin 163602432 167854079 x22v0-tp_big_tmeta_rmeta_0: 0 245760 linear x22v0-roger2: 0 134430720 thin 33014784 134430719 x22v0-gerrit--new--backup: 0 268615680 thin 6552576 268443647 Also lvs -a with segment ranges: LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert LE Ranges btrfs_ssd_linear x22v0 -wi-a----- <128.09g /dev/md125:19021-20113 btrfs_ssd_test x22v0 Vwi-a-t--- 96.09g tp_ssd 95.21 btrfsnopool x22v0 -wi-a----- 64.10g /dev/sdt2:35-581 btrfstest x22v0 Vwi-a-t--- <128.09g tp_big 15.16 gerrit-backup x22v0 Vwi-aot--- <128.09g tp_big 85.25 gerrit-new x22v0 Vwi-a-t--- <128.09g tp_ssd 95.12 gerrit-new-backup x22v0 Vwi-a-t--- <128.09g tp_big 2.44 gerrit-root x22v0 Vwi-aot--- <128.09g tp_ssd 38.49 gitlab_reg x22v0 Vwi-a-t--- 1.00t tp_big 49.87 gitlab_reg_snapshot x22v0 Vwi---t--k 1.00t tp_big gitlab_reg gitlab_root x22v0 Vwi-a-t--- 300.00g tp_ssd 99.86 gitlab_root_snapshot x22v0 Vwi---t--k 300.00g tp_ssd gitlab_root k8sdkr x22v0 -wi-a----- <128.09g /dev/md126:20891-21983 [lvol0_pmspare] x22v0 ewi------- 4.10g /dev/sdt2:0-34 micron_vol x22v0 -wi-a----- <128.09g /dev/sdt2:582-1674 nexus3_root x22v0 Vwi-aot--- <80.04g tp_ssd 13.03 nexus3_storage x22v0 Vwi-aot--- 5.00t tp_big 46.35 nexus_old_root x22v0 Vwi-a-t--- <128.09g tp_ssd 95.92 nexus_old_storage x22v0 Vwi-a-t--- 5.00t tp_big 47.78 nusknacker-staging x22v0 Vwi-aot--- <128.09g tp_big 99.84 nussknacker1 x22v0 Vwi-aot--- <128.09g tp_big 27.69 nussknacker2 x22v0 Vwi-aot--- <128.09g tp_big 21.81 openvpn-new x22v0 Vwi-aot--- 64.10g tp_big 17.97 rocket x22v0 Vwi-aot--- <80.04g tp_ssd 97.47 roger2 x22v0 Vwi-a-t--- 64.10g tp_ssd 24.56 tmob1 x22v0 -wi-a----- <500.04g /dev/md125:8739-13005 tmob2 x22v0 -wi-a----- <500.04g /dev/md125:13006-17272 touk-elk4 x22v0 -wi-ao---- <400.08g /dev/md126:17477-20890 tp_big x22v0 twi-aot--- 10.32t 53.64 3.27 [tp_big_tdata]:0-90187 [tp_big_tdata] x22v0 Twi-ao---- 10.32t /dev/md126:0-17476 [tp_big_tdata] x22v0 Twi-ao---- 10.32t /dev/md126:21984-94694 [tp_big_tmeta] x22v0 ewi-aor--- 4.10g 100.00 [tp_big_tmeta_rimage_0]:0-34,[tp_big_tmeta_rimage_1]:0-34 [tp_big_tmeta_rimage_0] x22v0 iwi-aor--- 4.10g /dev/sda3:30-64 [tp_big_tmeta_rimage_1] x22v0 iwi-aor--- 4.10g /dev/sdb3:30-64 [tp_big_tmeta_rmeta_0] x22v0 ewi-aor--- 120.00m /dev/sda3:29-29 [tp_big_tmeta_rmeta_1] x22v0 ewi-aor--- 120.00m /dev/sdb3:29-29 tp_ssd x22v0 twi-aot--- 1.20t 75.04 1.61 [tp_ssd_tdata]:0-10486 [tp_ssd_tdata] x22v0 Twi-ao---- 1.20t /dev/md125:0-8738 [tp_ssd_tdata] x22v0 Twi-ao---- 1.20t /dev/md125:17273-19020 [tp_ssd_tmeta] x22v0 ewi-aor--- <2.11g 100.00 [tp_ssd_tmeta_rimage_0]:0-17,[tp_ssd_tmeta_rimage_1]:0-17 [tp_ssd_tmeta_rimage_0] x22v0 iwi-aor--- <2.11g /dev/sda3:11-28 [tp_ssd_tmeta_rimage_1] x22v0 iwi-aor--- <2.11g /dev/sdb3:11-28 [tp_ssd_tmeta_rmeta_0] x22v0 ewi-aor--- 120.00m /dev/sda3:10-10 [tp_ssd_tmeta_rmeta_1] x22v0 ewi-aor--- 120.00m /dev/sdb3:10-10 > >>> Are the LVs split across RAID5 PVs by any chance? > > Michal> raid5s are used as PVs, but a single logical volume always uses one only > Michal> one physical volume underneath (if that's what you meant by split across). > > Ok, that's what I was asking about. It shouldn't matter... but just > trying to chase down the details. > > >>> It's not clear if you can replicate the problem without using >>> lvm-thin, but that's what I suspect you might be having problems with. > > Michal> I'll be trying to do that, though the heavier tests will have to wait > Michal> until I move all VMs to other hosts (as that is/was our production machnie). > > Sure, makes sense. > >>> Can you give us the versions of the your tools, and exactly how you >>> setup your test cases? How long does it take to find the problem? Regarding this, currently: kernel: 5.4.0-0.bpo.4-amd64 #1 SMP Debian 5.4.19-1~bpo10+1 (2020-03-09) x86_64 GNU/Linux (was also happening with 5.2.0-0.bpo.3-amd64) LVM version: 2.03.02(2) (2018-12-18) Library version: 1.02.155 (2018-12-18) Driver version: 4.41.0 mdadm - v4.1 - 2018-10-01 > > Michal> Will get all the details tommorow (the host is on up to date debian > Michal> buster, the VMs are mix of archlinuxes and debians (and the issue > Michal> happened on both)). > > Michal> As for how long, it's a hit and miss. Sometimes writing and reading back > Michal> ~16gb file fails (the cheksum read back differs from what was written) > Michal> after 2-3 tries. That's on the host. > > Michal> On the guest, it's been (so far) a guaranteed thing when we were > Michal> creating very large tar file (900gb+). As for past two weeks we were > Michal> unable to create that file without errors even once. > > Ouch! That's not good. Just to confirm, these corruptions are all in > a thin-lv based filesystem, right? I'd be interested to know if you > can create another plain LV and cause the same error. Trying to > simplify the potential problems. I have been trying to - but so far didn't manage to replicate this with: - a physical partition - filesystem directly on a physical partition - filesystem directly on mdraid - filesystem directly on a linear volume Note that this _doesn't_ imply that I _always_ get errors if lvm-thin is in use - as I also had lengthy period of attempts to cause corruption on some thin volume w/o any successes either. But the ones that failed had those in common (so far): md & lvm-thin - with 4 KiB piece(s) being incorrect > > >>> Can you compile the newst kernel and newest thin tools and try them >>> out? > > Michal> I can, but a bit later (once we move VMs out of the host). > >>> >>> How long does it take to replicate the corruption? >>> > > Michal> When it happens, it's usually few tries tries of writing a 16gb file > Michal> with random patterns and reading it back (directly on host). The > Michal> irritating thing is that it can be somewhat hard to reproduce (e.g. > Michal> after machine's reboot). > >>> Sorry for all the questions, but until there's a test case which is >>> repeatable, it's going to be hard to chase this down. >>> >>> I wonder if running 'fio' tests would be something to try? >>> >>> And also changing your RAID5 setup to use the default stride and >>> stripe widths, instead of the large values you're using. > > Michal> The raid5 is using mdadm's defaults (which is 512 KiB these days for a > Michal> chunk). LVM on top is using much longer extents (as we don't really need > Michal> 4mb granularity) and the lvm-thin chunks were set to match (and align) > Michal> to raid's stripe. > >>> >>> Good luck! >>> > Roger> I have not as of yet seen write corruption (except when a vendors disk > Roger> was resetting and it was lying about having written the data prior to > Roger> the crash, these were ssds, if your disk write cache is on and you > Roger> have a disk reset this can also happen), but have not seen "lost > Roger> writes" otherwise, but would expect the 2 read corruption I have seen > Roger> to also be able to cause write issues. So for that look for scsi > Roger> notifications for disk resets that should not happen. >>> > Roger> I have had a "bad" controller cause read corruptions, those > Roger> corruptions would move around, replacing the controller resolved it, > Roger> so there may be lack of error checking "inside" some paths in the > Roger> card. Lucky I had a number of these controllers and had cold spares > Roger> for them. The give away here was 2 separate buses with almost > Roger> identical load with 6 separate disks each and all12 disks on 2 buses > Roger> had between 47-52 scsi errors, which points to the only component > Roger> shared (the controller). >>> > Roger> The backplane and cables are unlikely in general cause this, there is > Roger> too much error checking between the controller and the disk from what > Roger> I know. >>> > Roger> I have had pre-pcie bus (PCI-X bus, 2 slots shared, both set to 133 > Roger> cause random read corruptions, lowering speed to 100 fixed it), this > Roger> one was duplicated on multiple identical pieces of hw with all > Roger> different parts on the duplication machine. >>> > Roger> I have also seen lost writes (from software) because someone did a > Roger> seek without doing a flush which in some versions of the libs loses > Roger> the unfilled block when the seek happens (this is noted in the man > Roger> page, and I saw it 20years ago, it is still noted in the man page, so > Roger> no idea if it was ever fixed). So has more than one application been > Roger> noted to see the corruption? >>> > Roger> So one question, have you seen the corruption in a path that would > Roger> rely on one controller, or all corruptions you have seen involving > Roger> more than one controller? Isolate and test each controller if you > Roger> can, or if you can afford to replace it and see if it continues. >>> >>> > Roger> On Thu, May 7, 2020 at 12:33 PM Michal Soltys wrote: >>>>> >>>>> Note: this is just general question - if anyone experienced something similar or could suggest how to pinpoint / verify the actual cause. >>>>> >>>>> Thanks to btrfs's checksumming we discovered somewhat (even if quite rare) nasty silent corruption going on on one of our hosts. Or perhaps "corruption" is not the correct word - the files simply have precise 4kb (1 page) of incorrect data. The incorrect pieces of data look on their own fine - as something that was previously in the place, or written from wrong source. >>>>> >>>>> The hardware is (can provide more detailed info of course): >>>>> >>>>> - Supermicro X9DR7-LN4F >>>>> - onboard LSI SAS2308 controller (2 sff-8087 connectors, 1 connected to backplane) >>>>> - 96 gb ram (ecc) >>>>> - 24 disk backplane >>>>> >>>>> - 1 array connected directly to lsi controller (4 disks, mdraid5, internal bitmap, 512kb chunk) >>>>> - 1 array on the backplane (4 disks, mdraid5, journaled) >>>>> - journal for the above array is: mdraid1, 2 ssd disks (micron 5300 pro disks) >>>>> - 1 btrfs raid1 boot array on motherboard's sata ports (older but still fine intel ssds from DC 3500 series) >>>>> >>>>> Raid 5 arrays are in lvm volume group, and the logical volumes are used by VMs. Some of the volumes are linear, some are using thin-pools (with metadata on the aforementioned intel ssds, in mirrored config). LVM >>>>> uses large extent sizes (120m) and the chunk-size of thin-pools is set to 1.5m to match underlying raid stripe. Everything is cleanly aligned as well. >>>>> >>>>> With a doze of testing we managed to roughly rule out the following elements as being the cause: >>>>> >>>>> - qemu/kvm (issue occured directly on host) >>>>> - backplane (issue occured on disks directly connected via LSI's 2nd connector) >>>>> - cable (as a above, two different cables) >>>>> - memory (unlikely - ECC for once, thoroughly tested, no errors ever reported via edac-util or memtest) >>>>> - mdadm journaling (issue occured on plain mdraid configuration as well) >>>>> - disks themselves (issue occured on two separate mdadm arrays) >>>>> - filesystem (issue occured on both btrfs and ext4 (checksumed manually) ) >>>>> >>>>> We did not manage to rule out (though somewhat _highly_ unlikely): >>>>> >>>>> - lvm thin (issue always - so far - occured on lvm thin pools) >>>>> - mdraid (issue always - so far - on mdraid managed arrays) >>>>> - kernel (tested with - in this case - debian's 5.2 and 5.4 kernels, happened with both - so it would imply rather already longstanding bug somewhere) >>>>> >>>>> And finally - so far - the issue never occured: >>>>> >>>>> - directly on a disk >>>>> - directly on mdraid >>>>> - on linear lvm volume on top of mdraid >>>>> >>>>> As far as the issue goes it's: >>>>> >>>>> - always a 4kb chunk that is incorrect - in a ~1 tb file it can be from a few to few dozens of such chunks >>>>> - we also found (or rather btrfs scrub did) a few small damaged files as well >>>>> - the chunks look like a correct piece of different or previous data >>>>> >>>>> The 4kb is well, weird ? Doesn't really matter any chunk/stripes sizes anywhere across the stack (lvm - 120m extents, 1.5m chunks on thin pools; mdraid - default 512kb chunks). It does nicely fit a page though ... >>>>> >>>>> Anyway, if anyone has any ideas or suggestions what could be happening (perhaps with this particular motherboard or vendor) or how to pinpoint the cause - I'll be grateful for any. >>> >