From mboxrd@z Thu Jan 1 00:00:00 1970 From: "John Stoffel" Subject: Re: [general question] rare silent data corruption when writing data Date: Fri, 8 May 2020 12:10:21 -0400 Message-ID: <24245.33901.720471.896495@quad.stoffel.home> References: <24244.30530.155404.154787@quad.stoffel.home> <24244.44462.151185.626440@quad.stoffel.home> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: base64 Return-path: In-Reply-To: Sender: linux-raid-owner@vger.kernel.org To: Michal Soltys Cc: John Stoffel , Roger Heflin , Linux RAID , "linux-lvm@redhat.com" , dm-devel@redhat.com List-Id: linux-raid.ids Pj4+Pj4gIk1pY2hhbCIgPT0gTWljaGFsIFNvbHR5cyA8bXNvbHR5c3BsQHlhbmRleC5wbD4gd3Jp dGVzOg0KDQpBbmQgb2YgY291cnNlIGl0IHNob3VsZCBhbHNvIGdvIHRvIGRtLWRldmVsQHJlZGhh dC5jb20sIG15IGZhdWx0IGZvcg0Kbm90IGluY2x1ZGluZyB0aGF0IGFzIHdlbGwuICBJIHN0cm9u Z2x5IHN1c3BlY3QgaXQncyBhbiB0aGluLWx2DQpwcm9ibGVtIHNvbWV3aGVyZSwgYnV0IEkgZG9u J3Qga25vdyBlbm91Z2ggdG8gaGVscCBjaGFzZSBkb3duIHRoZQ0KcHJvYmxlbSBpbiBkZXRhaWwu DQoNCkpvaG4NCg0KDQpNaWNoYWw+IG5vdGU6IGFzIHN1Z2dlc3RlZCwgSSdtIGFsc28gQ0Npbmcg dGhpcyB0byBsaW51eC1sdm07IHRoZSBmdWxsDQpNaWNoYWw+IGNvbnRleHQgd2l0aCByZXBsaWVz IHN0YXJ0cyBhdDoNCk1pY2hhbD4gaHR0cHM6Ly93d3cuc3Bpbmljcy5uZXQvbGlzdHMvcmFpZC9t c2c2NDM2NC5odG1sIFRoZXJlIGlzIGFsc28NCk1pY2hhbD4gdGhlIGluaXRpYWwgcG9zdCBhdCB0 aGUgYm90dG9tIGFzIHdlbGwuDQoNCk1pY2hhbD4gT24gNS84LzIwIDI6NTQgQU0sIEpvaG4gU3Rv ZmZlbCB3cm90ZToNCj4+Pj4+Pj4gIk1pY2hhbCIgPT0gTWljaGFsIFNvbHR5cyA8bXNvbHR5c3Bs QHlhbmRleC5wbD4gd3JpdGVzOg0KPj4gDQpNaWNoYWw+IE9uIDIwLzA1LzA3IDIzOjAxLCBKb2hu IFN0b2ZmZWwgd3JvdGU6DQo+Pj4+Pj4+Pj4gIlJvZ2VyIiA9PSBSb2dlciBIZWZsaW4gPHJvZ2Vy aGVmbGluQGdtYWlsLmNvbT4gd3JpdGVzOg0KPj4+PiANClJvZ2VyPiBIYXZlIHlvdSB0cmllZCB0 aGUgc2FtZSBmaWxlIDJ4IGFuZCB2ZXJpZmllZCB0aGUgY29ycnVwdGlvbiBpcyBpbiB0aGUNClJv Z2VyPiBzYW1lIHBsYWNlcyBhbmQgbG9va3MgdGhlIHNhbWU/DQo+Pj4+IA0KPj4+PiBBcmUgdGhl c2UgMXRiIGZpbGVzIFZNREsgb3IgQ09XIGltYWdlcyBvZiBWTXM/ICBIb3cgYXJlIHRoZXNlIGZp bGVzDQo+Pj4+IG1hZGUuICBBbmQgZG9lcyBpdCBldmVyIGhhcHBlbiB3aXRoICpzbWFsbGVyKiBm aWxlcz8gIFdoYXQgYWJvdXQgaWYNCj4+Pj4geW91IGp1c3QgdXNlIGEgc3BhcnNlIDJ0YiBmaWxl IGFuZCB3cml0ZSBibG9ja3Mgb3V0IHBhc3QgMXRiIHRvIHNlZSBpZg0KPj4+PiB0aGVyZSdzIGEg cHJvYmxlbT8NCj4+IA0KTWljaGFsPiBUaGUgVk1zIGFyZSBhbHdheXMgZGlyZWN0bHkgb24gbHZt IHZvbHVtZXMuIChlLmcuDQpNaWNoYWw+IC9kZXYvbWFwcGVyL3ZnMC1naXRsYWIpLiBUaGUgZ3Vl c3QgKGJ0cmZzIGluc2lkZSB0aGUgZ3Vlc3QpIGRldGVjdGVkIHRoZQ0KTWljaGFsPiBlcnJvcnMg YWZ0ZXIgd2UgcmFuIHNjcnViIG9uIHRoZSBmaWxlc3lzdGVtLg0KPj4gDQpNaWNoYWw+IFllcywg dGhlIGVycm9ycyB3ZXJlIGFsc28gZm91bmQgb24gc21hbGwgZmlsZXMuDQo+PiANCj4+IFRob3Nl IGVycm9ycyBhcmUgaW4gc21hbGwgZmlsZXMgaW5zaWRlIHRoZSBWTSwgd2hpY2ggaXMgcnVubmlu ZyBidHJmcw0KPj4gb250b3Agb2YgYmxvY2sgc3RvcmFnZSBwcm92aWRlZCBieSB5b3VyIHRoaW4t bHYsIHJpZ2h0Pw0KPj4gDQoNCk1pY2hhbD4gWWVhLCB0aGUgc21hbGwgZmlsZXMgd2VyZSBpbiB0 aGlzIGNhc2Ugb24gdGhhdCB0aGluLWx2Lg0KDQpNaWNoYWw+IFdlIGFsc28gZGlzY292ZXJlZCAo eWVzdGVyZGF5KSBmaWxlIGNvcnJ1cHRpb25zIHdpdGggVk0gaG9zdGluZyBnaXRsYWIgcmVnaXN0 cnkgLSB0aGlzIG9uZSB3YXMgdXNpbmcgdGhlIHNhbWUgdGhpbi1sdiB1bmRlcm5lYXRoLCBidXQg dGhlIGd1ZXN0IGl0c2VsZiB3YXMgdXNpbmcgZXh0NCAoaW4gdGhpcyBjYXNlLCBkb2NrZXIgc2lt cGx5IHJlcG9ydGVkIGluY29ycmVjdCBzaGEgY2hlY2tzdW0gb24gKHNvIGZhcikgMiBsYXllcnMu DQoNCj4+IA0KPj4gDQo+PiBkaXNrcyAtPiBtZCByYWlkNSAtPiBwdiAtPiB2ZyAtPiBsdi10aGlu IC0+IGd1ZXN0IFFDT1cvTFVOIC0+DQo+PiBmaWxlc3lzdGVtIC0+IGNvcnJ1cHRpb24NCg0KTWlj aGFsPiBUaG9zZSBwYXJ0aWN1bGFyIGd1ZXN0cywgeWVhLiBUaGUgaG9zdCBjYXNlIGl0J3MganVz dCB3L28gImd1ZXN0IiBzdGVwLg0KDQpNaWNoYWw+IEJ1dCAoc28gZmFyKSBhbGwgY29ycnVwdGlv biBlbmRlZCBnb2luZyB2aWEgb25lIG9mIHRoZSBsdi10aGluIGxheWVycyAoYW5kIHZpYSBvbmUg b2YgbWQgcmFpZHMpLg0KDQo+PiANCj4+IA0KTWljaGFsPiBTaW5jZSB0aGVuIHdlIHJlY3JlYXRl ZCB0aGUgaXNzdWUgZGlyZWN0bHkgb24gdGhlIGhvc3QsIGp1c3QNCk1pY2hhbD4gYnkgbWFraW5n IGV4dDQgZmlsZXN5c3RlbSBvbiBzb21lIExWLCB0aGVuIGRvaW5nIHdyaXRlIHdpdGgNCk1pY2hh bD4gY2hlY2tzdW0sIHN5bmMsIGRyb3BfY2FjaGVzLCByZWFkIGFuZCBjaGVjayBjaGVja3N1bS4g VGhlDQpNaWNoYWw+IGVycm9ycyBhcmUsIGFzIEkgbWVudGlvbmVkIC0gYWx3YXlzIGEgZnVsbCA0 S2lCIGNodW5rcyAoYWx3YXlzDQpNaWNoYWw+IHNhbWUgY29udGVudCwgYWx3YXlzIHNhbWUgcG9z aXRpb24pLg0KPj4gDQo+PiBXaGF0IHBvc2l0aW9uPyAgSXMgaXQgYSA0aywgMS41bSBvciBzb21l IG90aGVyIGNvbnNpc3RlbnQgb2Zmc2V0PyAgQW5kDQo+PiBob3cgZmFyIGludG8gdGhlIGZpbGU/ ICBBbmQgdGhpcyBMViBpcyBhIHBsYWluIExWIG9yIGEgdGhpbi1sdj8gICBJJ20NCj4+IHJ1bm5p bmcgYSBkZWJpYW4gYm94IGF0IGhvbWUgd2l0aCBSQUlEMSBhbmQgSSBoYXZlbid0IHNlZW4gdGhp cywgYnV0DQo+PiBJJ20gbm90IG5lYXJseSBhcyBjYXJlZnVsIGFzIHlvdS4gIENhbiB5b3UgcHJv dmlkZSB0aGUgb3V0cHV0IG9mOg0KPj4gDQoNCk1pY2hhbD4gV2hhdCBJIG1lYW50IHRoYXQgaXQg ZG9lc24ndCAibW92ZSIgd2hlbiB2ZXJpZnlpbmcgdGhlIHNhbWUgZmlsZSAoYWthIGRpZmZlcmVu dCByZWFkcyBmcm9tIHNhbWUgdGVzdCBmaWxlKS4gQmV0d2VlbiB0aGUgdGVzdHMsIHRoZSBlcnJv cnMgYXJlIG9mIGNvdXJzZSBpbiBkaWZmZXJlbnQgcGxhY2VzIC0gYnV0IGl0J3MgYWx3YXlzIHNv bWUgNEtpQiBwaWVjZShzKSAtIHRoYXQgbG9vayBsaWtlIGNvcnJlY3QgcGllY2VzIGJlbG9uZ2lu ZyBzb21ld2hlcmUgZWxzZS4NCg0KPj4gL3NiaW4vbHZzIC0tdmVyc2lvbg0KDQpNaWNoYWw+ICAg TFZNIHZlcnNpb246ICAgICAyLjAzLjAyKDIpICgyMDE4LTEyLTE4KQ0KTWljaGFsPiAgIExpYnJh cnkgdmVyc2lvbjogMS4wMi4xNTUgKDIwMTgtMTItMTgpDQpNaWNoYWw+ICAgRHJpdmVyIHZlcnNp b246ICA0LjQxLjANCk1pY2hhbD4gICBDb25maWd1cmF0aW9uOiAgIC4vY29uZmlndXJlIC0tYnVp bGQ9eDg2XzY0LWxpbnV4LWdudSAtLXByZWZpeD0vdXNyIC0taW5jbHVkZWRpcj0ke3ByZWZpeH0v aW5jbHVkZSAtLW1hbmRpcj0ke3ByZWZpeH0vc2hhcmUvbWFuIC0taW5mb2Rpcj0ke3ByZWZpeH0v c2hhcmUvaW5mbyAtLXN5c2NvbmZkaXI9L2V0YyAtLWxvY2Fsc3RhdGVkaXI9L3ZhciAtLWRpc2Fi bGUtc2lsZW50LXJ1bGVzIC0tbGliZGlyPSR7cHJlZml4fS9saWIveDg2XzY0LWxpbnV4LWdudSAt LWxpYmV4ZWNkaXI9JHtwcmVmaXh9L2xpYi94ODZfNjQtbGludXgtZ251IC0tcnVuc3RhdGVkaXI9 L3J1biAtLWRpc2FibGUtbWFpbnRhaW5lci1tb2RlIC0tZGlzYWJsZS1kZXBlbmRlbmN5LXRyYWNr aW5nIC0tZXhlYy1wcmVmaXg9IC0tYmluZGlyPS9iaW4gLS1saWJkaXI9L2xpYi94ODZfNjQtbGlu dXgtZ251IC0tc2JpbmRpcj0vc2JpbiAtLXdpdGgtdXNybGliZGlyPS91c3IvbGliL3g4Nl82NC1s aW51eC1nbnUgLS13aXRoLW9wdGltaXNhdGlvbj0tTzIgLS13aXRoLWNhY2hlPWludGVybmFsIC0t d2l0aC1kZXZpY2UtdWlkPTAgLS13aXRoLWRldmljZS1naWQ9NiAtLXdpdGgtZGV2aWNlLW1vZGU9 MDY2MCAtLXdpdGgtZGVmYXVsdC1waWQtZGlyPS9ydW4gLS13aXRoLWRlZmF1bHQtcnVuLWRpcj0v cnVuL2x2bSAtLXdpdGgtZGVmYXVsdC1sb2NraW5nLWRpcj0vcnVuL2xvY2svbHZtIC0td2l0aC10 aGluPWludGVybmFsIC0td2l0aC10aGluLWNoZWNrPS91c3Ivc2Jpbi90aGluX2NoZWNrIC0td2l0 aC10aGluLWR1bXA9L3Vzci9zYmluL3RoaW5fZHVtcCAtLXdpdGgtdGhpbi1yZXBhaXI9L3Vzci9z YmluL3RoaW5fcmVwYWlyIC0tZW5hYmxlLWFwcGxpYiAtLWVuYWJsZS1ibGtpZF93aXBpbmcgLS1l bmFibGUtY21kbGliIC0tZW5hYmxlLWRtZXZlbnRkIC0tZW5hYmxlLWRidXMtc2VydmljZSAtLWVu YWJsZS1sdm1sb2NrZC1kbG0gLS1lbmFibGUtbHZtbG9ja2Qtc2FubG9jayAtLWVuYWJsZS1sdm1w b2xsZCAtLWVuYWJsZS1ub3RpZnktZGJ1cyAtLWVuYWJsZS1wa2djb25maWcgLS1lbmFibGUtcmVh ZGxpbmUgLS1lbmFibGUtdWRldl9ydWxlcyAtLWVuYWJsZS11ZGV2X3N5bmMNCg0KPj4gDQo+PiB0 b28/DQo+PiANCj4+IENhbiB5b3UgcG9zdCB5b3VyOg0KPj4gDQo+PiAvc2Jpbi9kbXNldHVwIHN0 YXR1cw0KPj4gDQo+PiBvdXRwdXQgdG9vPyAgVGhlcmUncyBhIGJldHRlciBjb21tYW5kIHRvIHVz ZSBoZXJlLCBidXQgSSdtIG5vdCBhbg0KPj4gZXhwb3J0LiAgWW91IG1pZ2h0IHJlYWxseSB3YW50 IHRvIGNvcHkgdGhpcyBvdmVyIHRvIHRoZQ0KPj4gbGludXgtbHZtQHJlZGhhdC5jb20gbWFpbGlu ZyBsaXN0IGFzIHdlbGwuDQoNCk1pY2hhbD4geDIydjAtdHBfc3NkLXRwb29sOiAwIDI1NzcyODUx MjAgdGhpbi1wb29sIDE5IDg4ODYvNTUyOTYwIDYyOTUzNS84Mzg5NjAgLSBydyBub19kaXNjYXJk X3Bhc3Nkb3duIHF1ZXVlX2lmX25vX3NwYWNlIC0gMTAyNCANCk1pY2hhbD4geDIydjAtdHBfc3Nk X3RkYXRhOiAwIDIxNDc2OTY2NDAgbGluZWFyIA0KTWljaGFsPiB4MjJ2MC10cF9zc2RfdGRhdGE6 IDIxNDc2OTY2NDAgNDI5NTg4NDgwIGxpbmVhciANCk1pY2hhbD4geDIydjAtdHBfc3NkX3RtZXRh X3JpbWFnZV8xOiAwIDQ0MjM2ODAgbGluZWFyIA0KTWljaGFsPiB4MjJ2MC10cF9zc2RfdG1ldGE6 IDAgNDQyMzY4MCByYWlkIHJhaWQxIDIgQUEgNDQyMzY4MC80NDIzNjgwIGlkbGUgMCAwIC0NCk1p Y2hhbD4geDIydjAtZ2Vycml0LS1uZXc6IDAgMjY4NjE1NjgwIHRoaW4gMjU1NTEwNTI4IDI2ODQ1 OTAwNw0KTWljaGFsPiB4MjJ2MC1idHJmc25vcG9vbDogMCAxMzQ0MzA3MjAgbGluZWFyIA0KTWlj aGFsPiB4MjJ2MC1naXRsYWJfcm9vdDogMCA2MjkxNDU2MDAgdGhpbiA2MjgyOTE1ODQgNjI5MTQ1 NTk5DQpNaWNoYWw+IHgyMnYwLXRwX3NzZF90bWV0YV9yaW1hZ2VfMDogMCA0NDIzNjgwIGxpbmVh ciANCk1pY2hhbD4geDIydjAtbmV4dXNfb2xkX3N0b3JhZ2U6IDAgMTA3Mzc1MDAxNjAgdGhpbiA1 MTMwODE3NTM2IDEwNzM3NTAwMTU5DQpNaWNoYWw+IHgyMnYwLWdpdGxhYl9yZWc6IDAgMjE0NzY5 NjY0MCB0aGluIDEwNzA5NjM3MTIgMjE0NzY5NjYzOQ0KTWljaGFsPiB4MjJ2MC1uZXh1c19vbGRf cm9vdDogMCAyNjg2MTU2ODAgdGhpbiAyNTc2NTc4NTYgMjY4NjE1Njc5DQpNaWNoYWw+IHgyMnYw LXRwX2JpZ190bWV0YV9yaW1hZ2VfMTogMCA4NjAxNjAwIGxpbmVhciANCk1pY2hhbD4geDIydjAt dHBfc3NkX3RtZXRhX3JtZXRhXzE6IDAgMjQ1NzYwIGxpbmVhciANCk1pY2hhbD4geDIydjAtbWlj cm9uX3ZvbDogMCAyNjg2MTU2ODAgbGluZWFyIA0KTWljaGFsPiB4MjJ2MC10cF9iaWdfdG1ldGFf cmltYWdlXzA6IDAgODYwMTYwMCBsaW5lYXIgDQpNaWNoYWw+IHgyMnYwLXRwX3NzZF90bWV0YV9y bWV0YV8wOiAwIDI0NTc2MCBsaW5lYXIgDQpNaWNoYWw+IHgyMnYwLWdlcnJpdC0tcm9vdDogMCAy Njg2MTU2ODAgdGhpbiAxMDMzODgxNjAgMjY4NDQzNjQ3DQpNaWNoYWw+IHgyMnYwLWJ0cmZzX3Nz ZF9saW5lYXI6IDAgMjY4NjE1NjgwIGxpbmVhciANCk1pY2hhbD4geDIydjAtYnRyZnN0ZXN0OiAw IDI2ODYxNTY4MCB0aGluIDQwNzM0NzIwIDI2ODYxNTY3OQ0KTWljaGFsPiB4MjJ2MC10cF9zc2Q6 IDAgMjU3NzI4NTEyMCBsaW5lYXIgDQpNaWNoYWw+IHgyMnYwLXRwX2JpZzogMCAyMjE2NDYwMjg4 MCBsaW5lYXIgDQpNaWNoYWw+IHgyMnYwLW5leHVzM19yb290OiAwIDE2Nzg1NDA4MCB0aGluIDIx ODYwMzUyIDE2Nzg1NDA3OQ0KTWljaGFsPiB4MjJ2MC1udXNrbmFja2VyLS1zdGFnaW5nOiAwIDI2 ODYxNTY4MCB0aGluIDI2ODE4MjUyOCAyNjg2MTU2NzkNCk1pY2hhbD4geDIydjAtdG1vYjI6IDAg MTA0ODY1NzkyMCBsaW5lYXIgDQpNaWNoYWw+IHgyMnYwLXRwX2JpZy10cG9vbDogMCAyMjE2NDYw Mjg4MCB0aGluLXBvb2wgMzUgMzUxNTIvMTA3NTIwMCAzODcwMDcwLzcyMTUwNDAgLSBydyBub19k aXNjYXJkX3Bhc3Nkb3duIHF1ZXVlX2lmX25vX3NwYWNlIC0gMTAyNCANCk1pY2hhbD4geDIydjAt dHBfYmlnX3RkYXRhOiAwIDQyOTUxNDc1MjAgbGluZWFyIA0KTWljaGFsPiB4MjJ2MC10cF9iaWdf dGRhdGE6IDQyOTUxNDc1MjAgMTc4Njk0NTUzNjAgbGluZWFyIA0KTWljaGFsPiB4MjJ2MC1idHJm c19zc2RfdGVzdDogMCAyMDE1MjMyMDAgdGhpbiAxOTE4ODAxOTIgMjAxMzM1ODA3DQpNaWNoYWw+ IHgyMnYwLW51c3NrbmFja2VyMjogMCAyNjg2MTU2ODAgdGhpbiA1ODU3MzgyNCAyNjg2MTU2NzkN Ck1pY2hhbD4geDIydjAtdG1vYjE6IDAgMTA0ODY1NzkyMCBsaW5lYXIgDQpNaWNoYWw+IHgyMnYw LXRwX2JpZ190bWV0YTogMCA4NjAxNjAwIHJhaWQgcmFpZDEgMiBBQSA4NjAxNjAwLzg2MDE2MDAg aWRsZSAwIDAgLQ0KTWljaGFsPiB4MjJ2MC1udXNza25hY2tlcjE6IDAgMjY4NjE1NjgwIHRoaW4g NzQzNzYxOTIgMjY4NjE1Njc5DQpNaWNoYWw+IHgyMnYwLXRvdWstLWVsazQ6IDAgODM5MDI0NjQw IGxpbmVhciANCk1pY2hhbD4geDIydjAtZ2Vycml0LS1iYWNrdXA6IDAgMjY4NjE1NjgwIHRoaW4g MjI4OTg5OTUyIDI2ODQ0MzY0Nw0KTWljaGFsPiB4MjJ2MC10cF9iaWdfdG1ldGFfcm1ldGFfMTog MCAyNDU3NjAgbGluZWFyIA0KTWljaGFsPiB4MjJ2MC1vcGVudnBuLS1uZXc6IDAgMTM0NDMwNzIw IHRoaW4gMjQxNTIwNjQgNjYyNzIyNTUNCk1pY2hhbD4geDIydjAtazhzZGtyOiAwIDI2ODYxNTY4 MCBsaW5lYXIgDQpNaWNoYWw+IHgyMnYwLW5leHVzM19zdG9yYWdlOiAwIDEwNzM3NTAwMTYwIHRo aW4gNDk3NjY4MzAwOCAxMDczNzUwMDE1OQ0KTWljaGFsPiB4MjJ2MC1yb2NrZXQ6IDAgMTY3ODU0 MDgwIHRoaW4gMTYzNjAyNDMyIDE2Nzg1NDA3OQ0KTWljaGFsPiB4MjJ2MC10cF9iaWdfdG1ldGFf cm1ldGFfMDogMCAyNDU3NjAgbGluZWFyIA0KTWljaGFsPiB4MjJ2MC1yb2dlcjI6IDAgMTM0NDMw NzIwIHRoaW4gMzMwMTQ3ODQgMTM0NDMwNzE5DQpNaWNoYWw+IHgyMnYwLWdlcnJpdC0tbmV3LS1i YWNrdXA6IDAgMjY4NjE1NjgwIHRoaW4gNjU1MjU3NiAyNjg0NDM2NDcNCg0KTWljaGFsPiBBbHNv IGx2cyAtYSB3aXRoIHNlZ21lbnQgcmFuZ2VzOg0KTWljaGFsPiAgIExWICAgICAgICAgICAgICAg ICAgICAgIFZHICAgIEF0dHIgICAgICAgTFNpemUgICAgUG9vbCAgIE9yaWdpbiAgICAgIERhdGEl ICBNZXRhJSAgTW92ZSBMb2cgQ3B5JVN5bmMgQ29udmVydCBMRSBSYW5nZXMgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCk1pY2hhbD4gICBidHJmc19zc2Rf bGluZWFyICAgICAgICB4MjJ2MCAtd2ktYS0tLS0tIDwxMjguMDlnICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgL2Rldi9tZDEyNToxOTAy MS0yMDExMyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQpNaWNoYWw+ICAgYnRy ZnNfc3NkX3Rlc3QgICAgICAgICAgeDIydjAgVndpLWEtdC0tLSAgIDk2LjA5ZyB0cF9zc2QgICAg ICAgICAgICAgOTUuMjEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KTWljaGFs PiAgIGJ0cmZzbm9wb29sICAgICAgICAgICAgIHgyMnYwIC13aS1hLS0tLS0gICA2NC4xMGcgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAv ZGV2L3NkdDI6MzUtNTgxICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAN Ck1pY2hhbD4gICBidHJmc3Rlc3QgICAgICAgICAgICAgICB4MjJ2MCBWd2ktYS10LS0tIDwxMjgu MDlnIHRwX2JpZyAgICAgICAgICAgICAxNS4xNiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgDQpNaWNoYWw+ICAgZ2Vycml0LWJhY2t1cCAgICAgICAgICAgeDIydjAgVndpLWFvdC0t LSA8MTI4LjA5ZyB0cF9iaWcgICAgICAgICAgICAgODUuMjUgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIA0KTWljaGFsPiAgIGdlcnJpdC1uZXcgICAgICAgICAgICAgIHgyMnYwIFZ3 aS1hLXQtLS0gPDEyOC4wOWcgdHBfc3NkICAgICAgICAgICAgIDk1LjEyICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICANCk1pY2hhbD4gICBnZXJyaXQtbmV3LWJhY2t1cCAgICAgICB4 MjJ2MCBWd2ktYS10LS0tIDwxMjguMDlnIHRwX2JpZyAgICAgICAgICAgICAyLjQ0ICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgDQpNaWNoYWw+ICAgZ2Vycml0LXJvb3QgICAgICAg ICAgICAgeDIydjAgVndpLWFvdC0tLSA8MTI4LjA5ZyB0cF9zc2QgICAgICAgICAgICAgMzguNDkg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KTWljaGFsPiAgIGdpdGxhYl9yZWcg ICAgICAgICAgICAgIHgyMnYwIFZ3aS1hLXQtLS0gICAgMS4wMHQgdHBfYmlnICAgICAgICAgICAg IDQ5Ljg3ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCk1pY2hhbD4gICBnaXRs YWJfcmVnX3NuYXBzaG90ICAgICB4MjJ2MCBWd2ktLS10LS1rICAgIDEuMDB0IHRwX2JpZyBnaXRs YWJfcmVnICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQpNaWNoYWw+ ICAgZ2l0bGFiX3Jvb3QgICAgICAgICAgICAgeDIydjAgVndpLWEtdC0tLSAgMzAwLjAwZyB0cF9z c2QgICAgICAgICAgICAgOTkuODYgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0K TWljaGFsPiAgIGdpdGxhYl9yb290X3NuYXBzaG90ICAgIHgyMnYwIFZ3aS0tLXQtLWsgIDMwMC4w MGcgdHBfc3NkIGdpdGxhYl9yb290ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICANCk1pY2hhbD4gICBrOHNka3IgICAgICAgICAgICAgICAgICB4MjJ2MCAtd2ktYS0tLS0t IDwxMjguMDlnICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgL2Rldi9tZDEyNjoyMDg5MS0yMTk4MyAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgDQpNaWNoYWw+ICAgW2x2b2wwX3Btc3BhcmVdICAgICAgICAgeDIydjAgZXdp LS0tLS0tLSAgICA0LjEwZyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIC9kZXYvc2R0MjowLTM0ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIA0KTWljaGFsPiAgIG1pY3Jvbl92b2wgICAgICAgICAgICAgIHgy MnYwIC13aS1hLS0tLS0gPDEyOC4wOWcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAvZGV2L3NkdDI6NTgyLTE2NzQgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICANCk1pY2hhbD4gICBuZXh1czNfcm9vdCAgICAgICAg ICAgICB4MjJ2MCBWd2ktYW90LS0tICA8ODAuMDRnIHRwX3NzZCAgICAgICAgICAgICAxMy4wMyAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQpNaWNoYWw+ICAgbmV4dXMzX3N0b3Jh Z2UgICAgICAgICAgeDIydjAgVndpLWFvdC0tLSAgICA1LjAwdCB0cF9iaWcgICAgICAgICAgICAg NDYuMzUgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KTWljaGFsPiAgIG5leHVz X29sZF9yb290ICAgICAgICAgIHgyMnYwIFZ3aS1hLXQtLS0gPDEyOC4wOWcgdHBfc3NkICAgICAg ICAgICAgIDk1LjkyICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCk1pY2hhbD4g ICBuZXh1c19vbGRfc3RvcmFnZSAgICAgICB4MjJ2MCBWd2ktYS10LS0tICAgIDUuMDB0IHRwX2Jp ZyAgICAgICAgICAgICA0Ny43OCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQpN aWNoYWw+ICAgbnVza25hY2tlci1zdGFnaW5nICAgICAgeDIydjAgVndpLWFvdC0tLSA8MTI4LjA5 ZyB0cF9iaWcgICAgICAgICAgICAgOTkuODQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIA0KTWljaGFsPiAgIG51c3NrbmFja2VyMSAgICAgICAgICAgIHgyMnYwIFZ3aS1hb3QtLS0g PDEyOC4wOWcgdHBfYmlnICAgICAgICAgICAgIDI3LjY5ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICANCk1pY2hhbD4gICBudXNza25hY2tlcjIgICAgICAgICAgICB4MjJ2MCBWd2kt YW90LS0tIDwxMjguMDlnIHRwX2JpZyAgICAgICAgICAgICAyMS44MSAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgDQpNaWNoYWw+ICAgb3BlbnZwbi1uZXcgICAgICAgICAgICAgeDIy djAgVndpLWFvdC0tLSAgIDY0LjEwZyB0cF9iaWcgICAgICAgICAgICAgMTcuOTcgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIA0KTWljaGFsPiAgIHJvY2tldCAgICAgICAgICAgICAg ICAgIHgyMnYwIFZ3aS1hb3QtLS0gIDw4MC4wNGcgdHBfc3NkICAgICAgICAgICAgIDk3LjQ3ICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCk1pY2hhbD4gICByb2dlcjIgICAgICAg ICAgICAgICAgICB4MjJ2MCBWd2ktYS10LS0tICAgNjQuMTBnIHRwX3NzZCAgICAgICAgICAgICAy NC41NiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQpNaWNoYWw+ICAgdG1vYjEg ICAgICAgICAgICAgICAgICAgeDIydjAgLXdpLWEtLS0tLSA8NTAwLjA0ZyAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC9kZXYvbWQxMjU6 ODczOS0xMzAwNSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KTWljaGFsPiAg IHRtb2IyICAgICAgICAgICAgICAgICAgIHgyMnYwIC13aS1hLS0tLS0gPDUwMC4wNGcgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvZGV2 L21kMTI1OjEzMDA2LTE3MjcyICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCk1p Y2hhbD4gICB0b3VrLWVsazQgICAgICAgICAgICAgICB4MjJ2MCAtd2ktYW8tLS0tIDw0MDAuMDhn ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgL2Rldi9tZDEyNjoxNzQ3Ny0yMDg5MCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgDQpNaWNoYWw+ICAgdHBfYmlnICAgICAgICAgICAgICAgICAgeDIydjAgdHdpLWFvdC0tLSAg IDEwLjMydCAgICAgICAgICAgICAgICAgICAgNTMuNjQgIDMuMjcgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIFt0cF9iaWdfdGRhdGFdOjAtOTAxODcgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIA0KTWljaGFsPiAgIFt0cF9iaWdfdGRhdGFdICAgICAgICAgIHgyMnYwIFR3aS1h by0tLS0gICAxMC4zMnQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAvZGV2L21kMTI2OjAtMTc0NzYgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICANCk1pY2hhbD4gICBbdHBfYmlnX3RkYXRhXSAgICAgICAgICB4MjJ2 MCBUd2ktYW8tLS0tICAgMTAuMzJ0ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgL2Rldi9tZDEyNjoyMTk4NC05NDY5NCAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgDQpNaWNoYWw+ICAgW3RwX2JpZ190bWV0YV0gICAgICAg ICAgeDIydjAgZXdpLWFvci0tLSAgICA0LjEwZyAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAxMDAuMDAgICAgICAgICAgIFt0cF9iaWdfdG1ldGFfcmltYWdlXzBdOjAt MzQsW3RwX2JpZ190bWV0YV9yaW1hZ2VfMV06MC0zNA0KTWljaGFsPiAgIFt0cF9iaWdfdG1ldGFf cmltYWdlXzBdIHgyMnYwIGl3aS1hb3ItLS0gICAgNC4xMGcgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvZGV2L3NkYTM6MzAtNjQgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCk1pY2hhbD4gICBbdHBfYmln X3RtZXRhX3JpbWFnZV8xXSB4MjJ2MCBpd2ktYW9yLS0tICAgIDQuMTBnICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgL2Rldi9zZGIzOjMw LTY0ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQpNaWNoYWw+ICAg W3RwX2JpZ190bWV0YV9ybWV0YV8wXSAgeDIydjAgZXdpLWFvci0tLSAgMTIwLjAwbSAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC9kZXYv c2RhMzoyOS0yOSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KTWlj aGFsPiAgIFt0cF9iaWdfdG1ldGFfcm1ldGFfMV0gIHgyMnYwIGV3aS1hb3ItLS0gIDEyMC4wMG0g ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAvZGV2L3NkYjM6MjktMjkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICANCk1pY2hhbD4gICB0cF9zc2QgICAgICAgICAgICAgICAgICB4MjJ2MCB0d2ktYW90LS0tICAg IDEuMjB0ICAgICAgICAgICAgICAgICAgICA3NS4wNCAgMS42MSAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgW3RwX3NzZF90ZGF0YV06MC0xMDQ4NiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgDQpNaWNoYWw+ICAgW3RwX3NzZF90ZGF0YV0gICAgICAgICAgeDIydjAgVHdpLWFv LS0tLSAgICAxLjIwdCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIC9kZXYvbWQxMjU6MC04NzM4ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIA0KTWljaGFsPiAgIFt0cF9zc2RfdGRhdGFdICAgICAgICAgIHgyMnYw IFR3aS1hby0tLS0gICAgMS4yMHQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAvZGV2L21kMTI1OjE3MjczLTE5MDIwICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICANCk1pY2hhbD4gICBbdHBfc3NkX3RtZXRhXSAgICAgICAg ICB4MjJ2MCBld2ktYW9yLS0tICAgPDIuMTFnICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDEwMC4wMCAgICAgICAgICAgW3RwX3NzZF90bWV0YV9yaW1hZ2VfMF06MC0x NyxbdHBfc3NkX3RtZXRhX3JpbWFnZV8xXTowLTE3DQpNaWNoYWw+ICAgW3RwX3NzZF90bWV0YV9y aW1hZ2VfMF0geDIydjAgaXdpLWFvci0tLSAgIDwyLjExZyAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC9kZXYvc2RhMzoxMS0yOCAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIA0KTWljaGFsPiAgIFt0cF9zc2Rf dG1ldGFfcmltYWdlXzFdIHgyMnYwIGl3aS1hb3ItLS0gICA8Mi4xMWcgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvZGV2L3NkYjM6MTEt MjggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICANCk1pY2hhbD4gICBb dHBfc3NkX3RtZXRhX3JtZXRhXzBdICB4MjJ2MCBld2ktYW9yLS0tICAxMjAuMDBtICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgL2Rldi9z ZGEzOjEwLTEwICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgDQpNaWNo YWw+ICAgW3RwX3NzZF90bWV0YV9ybWV0YV8xXSAgeDIydjAgZXdpLWFvci0tLSAgMTIwLjAwbSAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IC9kZXYvc2RiMzoxMC0xMCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IA0KDQoNCj4+IA0KPj4+PiBBcmUgdGhlIExWcyBzcGxpdCBhY3Jvc3MgUkFJRDUgUFZzIGJ5IGFu eSBjaGFuY2U/DQo+PiANCk1pY2hhbD4gcmFpZDVzIGFyZSB1c2VkIGFzIFBWcywgYnV0IGEgc2lu Z2xlIGxvZ2ljYWwgdm9sdW1lIGFsd2F5cyB1c2VzIG9uZSBvbmx5DQpNaWNoYWw+IG9uZSBwaHlz aWNhbCB2b2x1bWUgdW5kZXJuZWF0aCAoaWYgdGhhdCdzIHdoYXQgeW91IG1lYW50IGJ5IHNwbGl0 IGFjcm9zcykuDQo+PiANCj4+IE9rLCB0aGF0J3Mgd2hhdCBJIHdhcyBhc2tpbmcgYWJvdXQuICBJ dCBzaG91bGRuJ3QgbWF0dGVyLi4uIGJ1dCBqdXN0DQo+PiB0cnlpbmcgdG8gY2hhc2UgZG93biB0 aGUgZGV0YWlscy4NCj4+IA0KPj4gDQo+Pj4+IEl0J3Mgbm90IGNsZWFyIGlmIHlvdSBjYW4gcmVw bGljYXRlIHRoZSBwcm9ibGVtIHdpdGhvdXQgdXNpbmcNCj4+Pj4gbHZtLXRoaW4sIGJ1dCB0aGF0 J3Mgd2hhdCBJIHN1c3BlY3QgeW91IG1pZ2h0IGJlIGhhdmluZyBwcm9ibGVtcyB3aXRoLg0KPj4g DQpNaWNoYWw+IEknbGwgYmUgdHJ5aW5nIHRvIGRvIHRoYXQsIHRob3VnaCB0aGUgaGVhdmllciB0 ZXN0cyB3aWxsIGhhdmUgdG8gd2FpdA0KTWljaGFsPiB1bnRpbCBJIG1vdmUgYWxsIFZNcyB0byBv dGhlciBob3N0cyAoYXMgdGhhdCBpcy93YXMgb3VyIHByb2R1Y3Rpb24gbWFjaG5pZSkuDQo+PiAN Cj4+IFN1cmUsIG1ha2VzIHNlbnNlLg0KPj4gDQo+Pj4+IENhbiB5b3UgZ2l2ZSB1cyB0aGUgdmVy c2lvbnMgb2YgdGhlIHlvdXIgdG9vbHMsIGFuZCBleGFjdGx5IGhvdyB5b3UNCj4+Pj4gc2V0dXAg eW91ciB0ZXN0IGNhc2VzPyAgSG93IGxvbmcgZG9lcyBpdCB0YWtlIHRvIGZpbmQgdGhlIHByb2Js ZW0/DQoNCk1pY2hhbD4gUmVnYXJkaW5nIHRoaXMsIGN1cnJlbnRseToNCg0KTWljaGFsPiBrZXJu ZWw6ICA1LjQuMC0wLmJwby40LWFtZDY0ICMxIFNNUCBEZWJpYW4gNS40LjE5LTF+YnBvMTArMSAo MjAyMC0wMy0wOSkgeDg2XzY0IEdOVS9MaW51eCAod2FzIGFsc28gaGFwcGVuaW5nIHdpdGggNS4y LjAtMC5icG8uMy1hbWQ2NCkNCk1pY2hhbD4gTFZNIHZlcnNpb246ICAgICAyLjAzLjAyKDIpICgy MDE4LTEyLTE4KQ0KTWljaGFsPiBMaWJyYXJ5IHZlcnNpb246IDEuMDIuMTU1ICgyMDE4LTEyLTE4 KQ0KTWljaGFsPiBEcml2ZXIgdmVyc2lvbjogIDQuNDEuMA0KTWljaGFsPiBtZGFkbSAtIHY0LjEg LSAyMDE4LTEwLTAxDQoNCj4+IA0KTWljaGFsPiBXaWxsIGdldCBhbGwgdGhlIGRldGFpbHMgdG9t bW9yb3cgKHRoZSBob3N0IGlzIG9uIHVwIHRvIGRhdGUgZGViaWFuDQpNaWNoYWw+IGJ1c3Rlciwg dGhlIFZNcyBhcmUgbWl4IG9mIGFyY2hsaW51eGVzIGFuZCBkZWJpYW5zIChhbmQgdGhlIGlzc3Vl DQpNaWNoYWw+IGhhcHBlbmVkIG9uIGJvdGgpKS4NCj4+IA0KTWljaGFsPiBBcyBmb3IgaG93IGxv bmcsIGl0J3MgYSBoaXQgYW5kIG1pc3MuIFNvbWV0aW1lcyB3cml0aW5nIGFuZCByZWFkaW5nIGJh Y2sNCk1pY2hhbD4gfjE2Z2IgZmlsZSBmYWlscyAodGhlIGNoZWtzdW0gcmVhZCBiYWNrIGRpZmZl cnMgZnJvbSB3aGF0IHdhcyB3cml0dGVuKQ0KTWljaGFsPiBhZnRlciAyLTMgdHJpZXMuIFRoYXQn cyBvbiB0aGUgaG9zdC4NCj4+IA0KTWljaGFsPiBPbiB0aGUgZ3Vlc3QsIGl0J3MgYmVlbiAoc28g ZmFyKSBhIGd1YXJhbnRlZWQgdGhpbmcgd2hlbiB3ZSB3ZXJlDQpNaWNoYWw+IGNyZWF0aW5nIHZl cnkgbGFyZ2UgdGFyIGZpbGUgKDkwMGdiKykuIEFzIGZvciBwYXN0IHR3byB3ZWVrcyB3ZSB3ZXJl DQpNaWNoYWw+IHVuYWJsZSB0byBjcmVhdGUgdGhhdCBmaWxlIHdpdGhvdXQgZXJyb3JzIGV2ZW4g b25jZS4NCj4+IA0KPj4gT3VjaCEgIFRoYXQncyBub3QgZ29vZC4gIEp1c3QgdG8gY29uZmlybSwg dGhlc2UgY29ycnVwdGlvbnMgYXJlIGFsbCBpbg0KPj4gYSB0aGluLWx2IGJhc2VkIGZpbGVzeXN0 ZW0sIHJpZ2h0PyAgIEknZCBiZSBpbnRlcmVzdGVkIHRvIGtub3cgaWYgeW91DQo+PiBjYW4gY3Jl YXRlIGFub3RoZXIgcGxhaW4gTFYgYW5kIGNhdXNlIHRoZSBzYW1lIGVycm9yLiAgVHJ5aW5nIHRv DQo+PiBzaW1wbGlmeSB0aGUgcG90ZW50aWFsIHByb2JsZW1zLg0KDQpNaWNoYWw+IEkgaGF2ZSBi ZWVuIHRyeWluZyB0byAtIGJ1dCBzbyBmYXIgZGlkbid0IG1hbmFnZSB0byByZXBsaWNhdGUgdGhp cyB3aXRoOg0KDQpNaWNoYWw+IC0gYSBwaHlzaWNhbCBwYXJ0aXRpb24NCk1pY2hhbD4gLSBmaWxl c3lzdGVtIGRpcmVjdGx5IG9uIGEgcGh5c2ljYWwgcGFydGl0aW9uDQpNaWNoYWw+IC0gZmlsZXN5 c3RlbSBkaXJlY3RseSBvbiBtZHJhaWQNCk1pY2hhbD4gLSBmaWxlc3lzdGVtIGRpcmVjdGx5IG9u IGEgbGluZWFyIHZvbHVtZQ0KDQpNaWNoYWw+IE5vdGUgdGhhdCB0aGlzIF9kb2Vzbid0XyBpbXBs eSB0aGF0IEkgX2Fsd2F5c18gZ2V0IGVycm9ycyBpZiBsdm0tdGhpbiBpcyBpbiB1c2UgLSBhcyBJ IGFsc28gaGFkIGxlbmd0aHkgcGVyaW9kIG9mIGF0dGVtcHRzIHRvIGNhdXNlIGNvcnJ1cHRpb24g b24gc29tZSB0aGluIHZvbHVtZSB3L28gYW55IHN1Y2Nlc3NlcyBlaXRoZXIuIEJ1dCB0aGUgb25l cyB0aGF0IGZhaWxlZCBoYWQgdGhvc2UgaW4gY29tbW9uIChzbyBmYXIpOiBtZCAmIGx2bS10aGlu IC0gd2l0aCA0IEtpQiBwaWVjZShzKSBiZWluZyBpbmNvcnJlY3QNCg0KPj4gDQo+PiANCj4+Pj4g Q2FuIHlvdSBjb21waWxlIHRoZSBuZXdzdCBrZXJuZWwgYW5kIG5ld2VzdCB0aGluIHRvb2xzIGFu ZCB0cnkgdGhlbQ0KPj4+PiBvdXQ/DQo+PiANCk1pY2hhbD4gSSBjYW4sIGJ1dCBhIGJpdCBsYXRl ciAob25jZSB3ZSBtb3ZlIFZNcyBvdXQgb2YgdGhlIGhvc3QpLg0KPj4gDQo+Pj4+IA0KPj4+PiBI b3cgbG9uZyBkb2VzIGl0IHRha2UgdG8gcmVwbGljYXRlIHRoZSBjb3JydXB0aW9uPw0KPj4+PiAN Cj4+IA0KTWljaGFsPiBXaGVuIGl0IGhhcHBlbnMsIGl0J3MgdXN1YWxseSBmZXcgdHJpZXMgdHJp ZXMgb2Ygd3JpdGluZyBhIDE2Z2IgZmlsZQ0KTWljaGFsPiB3aXRoIHJhbmRvbSBwYXR0ZXJucyBh bmQgcmVhZGluZyBpdCBiYWNrIChkaXJlY3RseSBvbiBob3N0KS4gVGhlDQpNaWNoYWw+IGlycml0 YXRpbmcgdGhpbmcgaXMgdGhhdCBpdCBjYW4gYmUgc29tZXdoYXQgaGFyZCB0byByZXByb2R1Y2Ug KGUuZy4NCk1pY2hhbD4gYWZ0ZXIgbWFjaGluZSdzIHJlYm9vdCkuDQo+PiANCj4+Pj4gU29ycnkg Zm9yIGFsbCB0aGUgcXVlc3Rpb25zLCBidXQgdW50aWwgdGhlcmUncyBhIHRlc3QgY2FzZSB3aGlj aCBpcw0KPj4+PiByZXBlYXRhYmxlLCBpdCdzIGdvaW5nIHRvIGJlIGhhcmQgdG8gY2hhc2UgdGhp cyBkb3duLg0KPj4+PiANCj4+Pj4gSSB3b25kZXIgaWYgcnVubmluZyAnZmlvJyB0ZXN0cyB3b3Vs ZCBiZSBzb21ldGhpbmcgdG8gdHJ5Pw0KPj4+PiANCj4+Pj4gQW5kIGFsc28gY2hhbmdpbmcgeW91 ciBSQUlENSBzZXR1cCB0byB1c2UgdGhlIGRlZmF1bHQgc3RyaWRlIGFuZA0KPj4+PiBzdHJpcGUg d2lkdGhzLCBpbnN0ZWFkIG9mIHRoZSBsYXJnZSB2YWx1ZXMgeW91J3JlIHVzaW5nLg0KPj4gDQpN aWNoYWw+IFRoZSByYWlkNSBpcyB1c2luZyBtZGFkbSdzIGRlZmF1bHRzICh3aGljaCBpcyA1MTIg S2lCIHRoZXNlIGRheXMgZm9yIGENCk1pY2hhbD4gY2h1bmspLiBMVk0gb24gdG9wIGlzIHVzaW5n IG11Y2ggbG9uZ2VyIGV4dGVudHMgKGFzIHdlIGRvbid0IHJlYWxseSBuZWVkDQpNaWNoYWw+IDRt YiBncmFudWxhcml0eSkgYW5kIHRoZSBsdm0tdGhpbiBjaHVua3Mgd2VyZSBzZXQgdG8gbWF0Y2gg KGFuZCBhbGlnbikNCk1pY2hhbD4gdG8gcmFpZCdzIHN0cmlwZS4NCj4+IA0KPj4+PiANCj4+Pj4g R29vZCBsdWNrIQ0KPj4+PiANClJvZ2VyPiBJIGhhdmUgbm90IGFzIG9mIHlldCBzZWVuIHdyaXRl IGNvcnJ1cHRpb24gKGV4Y2VwdCB3aGVuIGEgdmVuZG9ycyBkaXNrDQpSb2dlcj4gd2FzIHJlc2V0 dGluZyBhbmQgaXQgd2FzIGx5aW5nIGFib3V0IGhhdmluZyB3cml0dGVuIHRoZSBkYXRhIHByaW9y IHRvDQpSb2dlcj4gdGhlIGNyYXNoLCB0aGVzZSB3ZXJlIHNzZHMsIGlmIHlvdXIgZGlzayB3cml0 ZSBjYWNoZSBpcyBvbiBhbmQgeW91DQpSb2dlcj4gaGF2ZSBhIGRpc2sgcmVzZXQgdGhpcyBjYW4g YWxzbyBoYXBwZW4pLCBidXQgaGF2ZSBub3Qgc2VlbiAibG9zdA0KUm9nZXI+IHdyaXRlcyIgb3Ro ZXJ3aXNlLCBidXQgd291bGQgZXhwZWN0IHRoZSAyIHJlYWQgY29ycnVwdGlvbiBJIGhhdmUgc2Vl bg0KUm9nZXI+IHRvIGFsc28gYmUgYWJsZSB0byBjYXVzZSB3cml0ZSBpc3N1ZXMuICBTbyBmb3Ig dGhhdCBsb29rIGZvciBzY3NpDQpSb2dlcj4gbm90aWZpY2F0aW9ucyBmb3IgZGlzayByZXNldHMg dGhhdCBzaG91bGQgbm90IGhhcHBlbi4NCj4+Pj4gDQpSb2dlcj4gSSBoYXZlIGhhZCBhICJiYWQi IGNvbnRyb2xsZXIgY2F1c2UgcmVhZCBjb3JydXB0aW9ucywgdGhvc2UNClJvZ2VyPiBjb3JydXB0 aW9ucyB3b3VsZCBtb3ZlIGFyb3VuZCwgcmVwbGFjaW5nIHRoZSBjb250cm9sbGVyIHJlc29sdmVk IGl0LA0KUm9nZXI+IHNvIHRoZXJlIG1heSBiZSBsYWNrIG9mIGVycm9yIGNoZWNraW5nICJpbnNp ZGUiIHNvbWUgcGF0aHMgaW4gdGhlDQpSb2dlcj4gY2FyZC4gIEx1Y2t5IEkgaGFkIGEgbnVtYmVy IG9mIHRoZXNlIGNvbnRyb2xsZXJzIGFuZCBoYWQgY29sZCBzcGFyZXMNClJvZ2VyPiBmb3IgdGhl bS4gIFRoZSBnaXZlIGF3YXkgaGVyZSB3YXMgMiBzZXBhcmF0ZSBidXNlcyB3aXRoIGFsbW9zdA0K Um9nZXI+IGlkZW50aWNhbCBsb2FkIHdpdGggNiBzZXBhcmF0ZSBkaXNrcyBlYWNoIGFuZCBhbGwx MiBkaXNrcyBvbiAyIGJ1c2VzDQpSb2dlcj4gaGFkIGJldHdlZW4gNDctNTIgc2NzaSBlcnJvcnMs IHdoaWNoIHBvaW50cyB0byB0aGUgb25seSBjb21wb25lbnQNClJvZ2VyPiBzaGFyZWQgKHRoZSBj b250cm9sbGVyKS4NCj4+Pj4gDQpSb2dlcj4gVGhlIGJhY2twbGFuZSBhbmQgY2FibGVzIGFyZSB1 bmxpa2VseSBpbiBnZW5lcmFsIGNhdXNlIHRoaXMsIHRoZXJlIGlzDQpSb2dlcj4gdG9vIG11Y2gg ZXJyb3IgY2hlY2tpbmcgYmV0d2VlbiB0aGUgY29udHJvbGxlciBhbmQgdGhlIGRpc2sgZnJvbSB3 aGF0DQpSb2dlcj4gSSBrbm93Lg0KPj4+PiANClJvZ2VyPiBJIGhhdmUgaGFkIHByZS1wY2llIGJ1 cyAoUENJLVggYnVzLCAyIHNsb3RzIHNoYXJlZCwgYm90aCBzZXQgdG8gMTMzDQpSb2dlcj4gY2F1 c2UgcmFuZG9tIHJlYWQgY29ycnVwdGlvbnMsIGxvd2VyaW5nIHNwZWVkIHRvIDEwMCBmaXhlZCBp dCksIHRoaXMNClJvZ2VyPiBvbmUgd2FzIGR1cGxpY2F0ZWQgb24gbXVsdGlwbGUgaWRlbnRpY2Fs IHBpZWNlcyBvZiBodyB3aXRoIGFsbA0KUm9nZXI+IGRpZmZlcmVudCBwYXJ0cyBvbiB0aGUgZHVw bGljYXRpb24gbWFjaGluZS4NCj4+Pj4gDQpSb2dlcj4gSSBoYXZlIGFsc28gc2VlbiBsb3N0IHdy aXRlcyAoZnJvbSBzb2Z0d2FyZSkgYmVjYXVzZSBzb21lb25lIGRpZCBhDQpSb2dlcj4gc2VlayB3 aXRob3V0IGRvaW5nIGEgZmx1c2ggd2hpY2ggaW4gc29tZSB2ZXJzaW9ucyBvZiB0aGUgbGlicyBs b3Nlcw0KUm9nZXI+IHRoZSB1bmZpbGxlZCBibG9jayB3aGVuIHRoZSBzZWVrIGhhcHBlbnMgKHRo aXMgaXMgbm90ZWQgaW4gdGhlIG1hbg0KUm9nZXI+IHBhZ2UsIGFuZCBJIHNhdyBpdCAyMHllYXJz IGFnbywgaXQgaXMgc3RpbGwgbm90ZWQgaW4gdGhlIG1hbiBwYWdlLCBzbw0KUm9nZXI+IG5vIGlk ZWEgaWYgaXQgd2FzIGV2ZXIgZml4ZWQpLiAgU28gaGFzIG1vcmUgdGhhbiBvbmUgYXBwbGljYXRp b24gYmVlbg0KUm9nZXI+IG5vdGVkIHRvIHNlZSB0aGUgY29ycnVwdGlvbj8NCj4+Pj4gDQpSb2dl cj4gU28gb25lIHF1ZXN0aW9uLCBoYXZlIHlvdSBzZWVuIHRoZSBjb3JydXB0aW9uIGluIGEgcGF0 aCB0aGF0IHdvdWxkDQpSb2dlcj4gcmVseSBvbiBvbmUgY29udHJvbGxlciwgb3IgYWxsIGNvcnJ1 cHRpb25zIHlvdSBoYXZlIHNlZW4gaW52b2x2aW5nDQpSb2dlcj4gbW9yZSB0aGFuIG9uZSBjb250 cm9sbGVyPyAgSXNvbGF0ZSBhbmQgdGVzdCBlYWNoIGNvbnRyb2xsZXIgaWYgeW91DQpSb2dlcj4g Y2FuLCBvciBpZiB5b3UgY2FuIGFmZm9yZCB0byByZXBsYWNlIGl0IGFuZCBzZWUgaWYgaXQgY29u dGludWVzLg0KPj4+PiANCj4+Pj4gDQpSb2dlcj4gT24gVGh1LCBNYXkgNywgMjAyMCBhdCAxMjoz MyBQTSBNaWNoYWwgU29sdHlzIDxtc29sdHlzcGxAeWFuZGV4LnBsPiB3cm90ZToNCj4+Pj4+PiAN Cj4+Pj4+IE5vdGU6IHRoaXMgaXMganVzdCBnZW5lcmFsIHF1ZXN0aW9uIC0gaWYgYW55b25lIGV4 cGVyaWVuY2VkIHNvbWV0aGluZyBzaW1pbGFyIG9yIGNvdWxkIHN1Z2dlc3QgaG93IHRvIHBpbnBv aW50IC8gdmVyaWZ5IHRoZSBhY3R1YWwgY2F1c2UuDQo+Pj4+Pj4gDQo+Pj4+PiBUaGFua3MgdG8g YnRyZnMncyBjaGVja3N1bW1pbmcgd2UgZGlzY292ZXJlZCBzb21ld2hhdCAoZXZlbiBpZiBxdWl0 ZSByYXJlKSBuYXN0eSBzaWxlbnQgY29ycnVwdGlvbiBnb2luZyBvbiBvbiBvbmUgb2Ygb3VyIGhv c3RzLiBPciBwZXJoYXBzICJjb3JydXB0aW9uIiBpcyBub3QgdGhlIGNvcnJlY3Qgd29yZCAtIHRo ZSBmaWxlcyBzaW1wbHkgaGF2ZSBwcmVjaXNlIDRrYiAoMSBwYWdlKSBvZiBpbmNvcnJlY3QgZGF0 YS4gVGhlIGluY29ycmVjdCBwaWVjZXMgb2YgZGF0YSBsb29rIG9uIHRoZWlyIG93biBmaW5lIC0g YXMgc29tZXRoaW5nIHRoYXQgd2FzIHByZXZpb3VzbHkgaW4gdGhlIHBsYWNlLCBvciB3cml0dGVu IGZyb20gd3Jvbmcgc291cmNlLg0KPj4+Pj4+IA0KPj4+Pj4gVGhlIGhhcmR3YXJlIGlzIChjYW4g cHJvdmlkZSBtb3JlIGRldGFpbGVkIGluZm8gb2YgY291cnNlKToNCj4+Pj4+PiANCj4+Pj4+IC0g U3VwZXJtaWNybyBYOURSNy1MTjRGDQo+Pj4+PiAtIG9uYm9hcmQgTFNJIFNBUzIzMDggY29udHJv bGxlciAoMiBzZmYtODA4NyBjb25uZWN0b3JzLCAxIGNvbm5lY3RlZCB0byBiYWNrcGxhbmUpDQo+ Pj4+PiAtIDk2IGdiIHJhbSAoZWNjKQ0KPj4+Pj4gLSAyNCBkaXNrIGJhY2twbGFuZQ0KPj4+Pj4+ IA0KPj4+Pj4gLSAxIGFycmF5IGNvbm5lY3RlZCBkaXJlY3RseSB0byBsc2kgY29udHJvbGxlciAo NCBkaXNrcywgbWRyYWlkNSwgaW50ZXJuYWwgYml0bWFwLCA1MTJrYiBjaHVuaykNCj4+Pj4+IC0g MSBhcnJheSBvbiB0aGUgYmFja3BsYW5lICg0IGRpc2tzLCBtZHJhaWQ1LCBqb3VybmFsZWQpDQo+ Pj4+PiAtIGpvdXJuYWwgZm9yIHRoZSBhYm92ZSBhcnJheSBpczogbWRyYWlkMSwgMiBzc2QgZGlz a3MgKG1pY3JvbiA1MzAwIHBybyBkaXNrcykNCj4+Pj4+IC0gMSBidHJmcyByYWlkMSBib290IGFy cmF5IG9uIG1vdGhlcmJvYXJkJ3Mgc2F0YSBwb3J0cyAob2xkZXIgYnV0IHN0aWxsIGZpbmUgaW50 ZWwgc3NkcyBmcm9tIERDIDM1MDAgc2VyaWVzKQ0KPj4+Pj4+IA0KPj4+Pj4gUmFpZCA1IGFycmF5 cyBhcmUgaW4gbHZtIHZvbHVtZSBncm91cCwgYW5kIHRoZSBsb2dpY2FsIHZvbHVtZXMgYXJlIHVz ZWQgYnkgVk1zLiBTb21lIG9mIHRoZSB2b2x1bWVzIGFyZSBsaW5lYXIsIHNvbWUgYXJlIHVzaW5n IHRoaW4tcG9vbHMgKHdpdGggbWV0YWRhdGEgb24gdGhlIGFmb3JlbWVudGlvbmVkIGludGVsIHNz ZHMsIGluIG1pcnJvcmVkIGNvbmZpZykuIExWTQ0KPj4+Pj4gdXNlcyBsYXJnZSBleHRlbnQgc2l6 ZXMgKDEyMG0pIGFuZCB0aGUgY2h1bmstc2l6ZSBvZiB0aGluLXBvb2xzIGlzIHNldCB0byAxLjVt IHRvIG1hdGNoIHVuZGVybHlpbmcgcmFpZCBzdHJpcGUuIEV2ZXJ5dGhpbmcgaXMgY2xlYW5seSBh bGlnbmVkIGFzIHdlbGwuDQo+Pj4+Pj4gDQo+Pj4+PiBXaXRoIGEgZG96ZSBvZiB0ZXN0aW5nIHdl IG1hbmFnZWQgdG8gcm91Z2hseSBydWxlIG91dCB0aGUgZm9sbG93aW5nIGVsZW1lbnRzIGFzIGJl aW5nIHRoZSBjYXVzZToNCj4+Pj4+PiANCj4+Pj4+IC0gcWVtdS9rdm0gKGlzc3VlIG9jY3VyZWQg ZGlyZWN0bHkgb24gaG9zdCkNCj4+Pj4+IC0gYmFja3BsYW5lIChpc3N1ZSBvY2N1cmVkIG9uIGRp c2tzIGRpcmVjdGx5IGNvbm5lY3RlZCB2aWEgTFNJJ3MgMm5kIGNvbm5lY3RvcikNCj4+Pj4+IC0g Y2FibGUgKGFzIGEgYWJvdmUsIHR3byBkaWZmZXJlbnQgY2FibGVzKQ0KPj4+Pj4gLSBtZW1vcnkg KHVubGlrZWx5IC0gRUNDIGZvciBvbmNlLCB0aG9yb3VnaGx5IHRlc3RlZCwgbm8gZXJyb3JzIGV2 ZXIgcmVwb3J0ZWQgdmlhIGVkYWMtdXRpbCBvciBtZW10ZXN0KQ0KPj4+Pj4gLSBtZGFkbSBqb3Vy bmFsaW5nIChpc3N1ZSBvY2N1cmVkIG9uIHBsYWluIG1kcmFpZCBjb25maWd1cmF0aW9uIGFzIHdl bGwpDQo+Pj4+PiAtIGRpc2tzIHRoZW1zZWx2ZXMgKGlzc3VlIG9jY3VyZWQgb24gdHdvIHNlcGFy YXRlIG1kYWRtIGFycmF5cykNCj4+Pj4+IC0gZmlsZXN5c3RlbSAoaXNzdWUgb2NjdXJlZCBvbiBi b3RoIGJ0cmZzIGFuZCBleHQ0IChjaGVja3N1bWVkIG1hbnVhbGx5KSApDQo+Pj4+Pj4gDQo+Pj4+ PiBXZSBkaWQgbm90IG1hbmFnZSB0byBydWxlIG91dCAodGhvdWdoIHNvbWV3aGF0IF9oaWdobHlf IHVubGlrZWx5KToNCj4+Pj4+PiANCj4+Pj4+IC0gbHZtIHRoaW4gKGlzc3VlIGFsd2F5cyAtIHNv IGZhciAtIG9jY3VyZWQgb24gbHZtIHRoaW4gcG9vbHMpDQo+Pj4+PiAtIG1kcmFpZCAoaXNzdWUg YWx3YXlzIC0gc28gZmFyIC0gb24gbWRyYWlkIG1hbmFnZWQgYXJyYXlzKQ0KPj4+Pj4gLSBrZXJu ZWwgKHRlc3RlZCB3aXRoIC0gaW4gdGhpcyBjYXNlIC0gZGViaWFuJ3MgNS4yIGFuZCA1LjQga2Vy bmVscywgaGFwcGVuZWQgd2l0aCBib3RoIC0gc28gaXQgd291bGQgaW1wbHkgcmF0aGVyIGFscmVh ZHkgbG9uZ3N0YW5kaW5nIGJ1ZyBzb21ld2hlcmUpDQo+Pj4+Pj4gDQo+Pj4+PiBBbmQgZmluYWxs eSAtIHNvIGZhciAtIHRoZSBpc3N1ZSBuZXZlciBvY2N1cmVkOg0KPj4+Pj4+IA0KPj4+Pj4gLSBk aXJlY3RseSBvbiBhIGRpc2sNCj4+Pj4+IC0gZGlyZWN0bHkgb24gbWRyYWlkDQo+Pj4+PiAtIG9u IGxpbmVhciBsdm0gdm9sdW1lIG9uIHRvcCBvZiBtZHJhaWQNCj4+Pj4+PiANCj4+Pj4+IEFzIGZh ciBhcyB0aGUgaXNzdWUgZ29lcyBpdCdzOg0KPj4+Pj4+IA0KPj4+Pj4gLSBhbHdheXMgYSA0a2Ig Y2h1bmsgdGhhdCBpcyBpbmNvcnJlY3QgLSBpbiBhIH4xIHRiIGZpbGUgaXQgY2FuIGJlIGZyb20g YSBmZXcgdG8gZmV3IGRvemVucyBvZiBzdWNoIGNodW5rcw0KPj4+Pj4gLSB3ZSBhbHNvIGZvdW5k IChvciByYXRoZXIgYnRyZnMgc2NydWIgZGlkKSBhIGZldyBzbWFsbCBkYW1hZ2VkIGZpbGVzIGFz IHdlbGwNCj4+Pj4+IC0gdGhlIGNodW5rcyBsb29rIGxpa2UgYSBjb3JyZWN0IHBpZWNlIG9mIGRp ZmZlcmVudCBvciBwcmV2aW91cyBkYXRhDQo+Pj4+Pj4gDQo+Pj4+PiBUaGUgNGtiIGlzIHdlbGws IHdlaXJkID8gRG9lc24ndCByZWFsbHkgbWF0dGVyIGFueSBjaHVuay9zdHJpcGVzIHNpemVzIGFu eXdoZXJlIGFjcm9zcyB0aGUgc3RhY2sgKGx2bSAtIDEyMG0gZXh0ZW50cywgMS41bSBjaHVua3Mg b24gdGhpbiBwb29sczsgbWRyYWlkIC0gZGVmYXVsdCA1MTJrYiBjaHVua3MpLiBJdCBkb2VzIG5p Y2VseSBmaXQgYSBwYWdlIHRob3VnaCAuLi4NCj4+Pj4+PiANCj4+Pj4+IEFueXdheSwgaWYgYW55 b25lIGhhcyBhbnkgaWRlYXMgb3Igc3VnZ2VzdGlvbnMgd2hhdCBjb3VsZCBiZSBoYXBwZW5pbmcg KHBlcmhhcHMgd2l0aCB0aGlzIHBhcnRpY3VsYXIgbW90aGVyYm9hcmQgb3IgdmVuZG9yKSBvciBo b3cgdG8gcGlucG9pbnQgdGhlIGNhdXNlIC0gSSdsbCBiZSBncmF0ZWZ1bCBmb3IgYW55Lg0KPj4+ PiANCj4+IA0K From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mimecast-mx02.redhat.com (mimecast06.extmail.prod.ext.rdu2.redhat.com [10.11.55.22]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 42CB12029F62 for ; Fri, 8 May 2020 16:10:27 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1F6B91859160 for ; Fri, 8 May 2020 16:10:27 +0000 (UTC) MIME-Version: 1.0 Message-ID: <24245.33901.720471.896495@quad.stoffel.home> Date: Fri, 8 May 2020 12:10:21 -0400 From: "John Stoffel" In-Reply-To: References: <24244.30530.155404.154787@quad.stoffel.home> <24244.44462.151185.626440@quad.stoffel.home> Content-Transfer-Encoding: 8bit Subject: Re: [linux-lvm] [general question] rare silent data corruption when writing data Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: Content-Type: text/plain; charset="us-ascii" To: Michal Soltys Cc: Linux RAID , Roger Heflin , dm-devel@redhat.com, "linux-lvm@redhat.com" >>>>> "Michal" == Michal Soltys writes: And of course it should also go to dm-devel@redhat.com, my fault for not including that as well. I strongly suspect it's an thin-lv problem somewhere, but I don't know enough to help chase down the problem in detail. John Michal> note: as suggested, I'm also CCing this to linux-lvm; the full Michal> context with replies starts at: Michal> https://www.spinics.net/lists/raid/msg64364.html There is also Michal> the initial post at the bottom as well. Michal> On 5/8/20 2:54 AM, John Stoffel wrote: >>>>>>> "Michal" == Michal Soltys writes: >> Michal> On 20/05/07 23:01, John Stoffel wrote: >>>>>>>>> "Roger" == Roger Heflin writes: >>>> Roger> Have you tried the same file 2x and verified the corruption is in the Roger> same places and looks the same? >>>> >>>> Are these 1tb files VMDK or COW images of VMs? How are these files >>>> made. And does it ever happen with *smaller* files? What about if >>>> you just use a sparse 2tb file and write blocks out past 1tb to see if >>>> there's a problem? >> Michal> The VMs are always directly on lvm volumes. (e.g. Michal> /dev/mapper/vg0-gitlab). The guest (btrfs inside the guest) detected the Michal> errors after we ran scrub on the filesystem. >> Michal> Yes, the errors were also found on small files. >> >> Those errors are in small files inside the VM, which is running btrfs >> ontop of block storage provided by your thin-lv, right? >> Michal> Yea, the small files were in this case on that thin-lv. Michal> We also discovered (yesterday) file corruptions with VM hosting gitlab registry - this one was using the same thin-lv underneath, but the guest itself was using ext4 (in this case, docker simply reported incorrect sha checksum on (so far) 2 layers. >> >> >> disks -> md raid5 -> pv -> vg -> lv-thin -> guest QCOW/LUN -> >> filesystem -> corruption Michal> Those particular guests, yea. The host case it's just w/o "guest" step. Michal> But (so far) all corruption ended going via one of the lv-thin layers (and via one of md raids). >> >> Michal> Since then we recreated the issue directly on the host, just Michal> by making ext4 filesystem on some LV, then doing write with Michal> checksum, sync, drop_caches, read and check checksum. The Michal> errors are, as I mentioned - always a full 4KiB chunks (always Michal> same content, always same position). >> >> What position? Is it a 4k, 1.5m or some other consistent offset? And >> how far into the file? And this LV is a plain LV or a thin-lv? I'm >> running a debian box at home with RAID1 and I haven't seen this, but >> I'm not nearly as careful as you. Can you provide the output of: >> Michal> What I meant that it doesn't "move" when verifying the same file (aka different reads from same test file). Between the tests, the errors are of course in different places - but it's always some 4KiB piece(s) - that look like correct pieces belonging somewhere else. >> /sbin/lvs --version Michal> LVM version: 2.03.02(2) (2018-12-18) Michal> Library version: 1.02.155 (2018-12-18) Michal> Driver version: 4.41.0 Michal> Configuration: ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --exec-prefix= --bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin --with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2 --with-cache=internal --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm --with-thin=internal --with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump --with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping --enable-cmdlib --enable-dmeventd --enable-dbus-service --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-lvmpolld --enable-notify-dbus --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync >> >> too? >> >> Can you post your: >> >> /sbin/dmsetup status >> >> output too? There's a better command to use here, but I'm not an >> export. You might really want to copy this over to the >> linux-lvm@redhat.com mailing list as well. Michal> x22v0-tp_ssd-tpool: 0 2577285120 thin-pool 19 8886/552960 629535/838960 - rw no_discard_passdown queue_if_no_space - 1024 Michal> x22v0-tp_ssd_tdata: 0 2147696640 linear Michal> x22v0-tp_ssd_tdata: 2147696640 429588480 linear Michal> x22v0-tp_ssd_tmeta_rimage_1: 0 4423680 linear Michal> x22v0-tp_ssd_tmeta: 0 4423680 raid raid1 2 AA 4423680/4423680 idle 0 0 - Michal> x22v0-gerrit--new: 0 268615680 thin 255510528 268459007 Michal> x22v0-btrfsnopool: 0 134430720 linear Michal> x22v0-gitlab_root: 0 629145600 thin 628291584 629145599 Michal> x22v0-tp_ssd_tmeta_rimage_0: 0 4423680 linear Michal> x22v0-nexus_old_storage: 0 10737500160 thin 5130817536 10737500159 Michal> x22v0-gitlab_reg: 0 2147696640 thin 1070963712 2147696639 Michal> x22v0-nexus_old_root: 0 268615680 thin 257657856 268615679 Michal> x22v0-tp_big_tmeta_rimage_1: 0 8601600 linear Michal> x22v0-tp_ssd_tmeta_rmeta_1: 0 245760 linear Michal> x22v0-micron_vol: 0 268615680 linear Michal> x22v0-tp_big_tmeta_rimage_0: 0 8601600 linear Michal> x22v0-tp_ssd_tmeta_rmeta_0: 0 245760 linear Michal> x22v0-gerrit--root: 0 268615680 thin 103388160 268443647 Michal> x22v0-btrfs_ssd_linear: 0 268615680 linear Michal> x22v0-btrfstest: 0 268615680 thin 40734720 268615679 Michal> x22v0-tp_ssd: 0 2577285120 linear Michal> x22v0-tp_big: 0 22164602880 linear Michal> x22v0-nexus3_root: 0 167854080 thin 21860352 167854079 Michal> x22v0-nusknacker--staging: 0 268615680 thin 268182528 268615679 Michal> x22v0-tmob2: 0 1048657920 linear Michal> x22v0-tp_big-tpool: 0 22164602880 thin-pool 35 35152/1075200 3870070/7215040 - rw no_discard_passdown queue_if_no_space - 1024 Michal> x22v0-tp_big_tdata: 0 4295147520 linear Michal> x22v0-tp_big_tdata: 4295147520 17869455360 linear Michal> x22v0-btrfs_ssd_test: 0 201523200 thin 191880192 201335807 Michal> x22v0-nussknacker2: 0 268615680 thin 58573824 268615679 Michal> x22v0-tmob1: 0 1048657920 linear Michal> x22v0-tp_big_tmeta: 0 8601600 raid raid1 2 AA 8601600/8601600 idle 0 0 - Michal> x22v0-nussknacker1: 0 268615680 thin 74376192 268615679 Michal> x22v0-touk--elk4: 0 839024640 linear Michal> x22v0-gerrit--backup: 0 268615680 thin 228989952 268443647 Michal> x22v0-tp_big_tmeta_rmeta_1: 0 245760 linear Michal> x22v0-openvpn--new: 0 134430720 thin 24152064 66272255 Michal> x22v0-k8sdkr: 0 268615680 linear Michal> x22v0-nexus3_storage: 0 10737500160 thin 4976683008 10737500159 Michal> x22v0-rocket: 0 167854080 thin 163602432 167854079 Michal> x22v0-tp_big_tmeta_rmeta_0: 0 245760 linear Michal> x22v0-roger2: 0 134430720 thin 33014784 134430719 Michal> x22v0-gerrit--new--backup: 0 268615680 thin 6552576 268443647 Michal> Also lvs -a with segment ranges: Michal> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert LE Ranges Michal> btrfs_ssd_linear x22v0 -wi-a----- <128.09g /dev/md125:19021-20113 Michal> btrfs_ssd_test x22v0 Vwi-a-t--- 96.09g tp_ssd 95.21 Michal> btrfsnopool x22v0 -wi-a----- 64.10g /dev/sdt2:35-581 Michal> btrfstest x22v0 Vwi-a-t--- <128.09g tp_big 15.16 Michal> gerrit-backup x22v0 Vwi-aot--- <128.09g tp_big 85.25 Michal> gerrit-new x22v0 Vwi-a-t--- <128.09g tp_ssd 95.12 Michal> gerrit-new-backup x22v0 Vwi-a-t--- <128.09g tp_big 2.44 Michal> gerrit-root x22v0 Vwi-aot--- <128.09g tp_ssd 38.49 Michal> gitlab_reg x22v0 Vwi-a-t--- 1.00t tp_big 49.87 Michal> gitlab_reg_snapshot x22v0 Vwi---t--k 1.00t tp_big gitlab_reg Michal> gitlab_root x22v0 Vwi-a-t--- 300.00g tp_ssd 99.86 Michal> gitlab_root_snapshot x22v0 Vwi---t--k 300.00g tp_ssd gitlab_root Michal> k8sdkr x22v0 -wi-a----- <128.09g /dev/md126:20891-21983 Michal> [lvol0_pmspare] x22v0 ewi------- 4.10g /dev/sdt2:0-34 Michal> micron_vol x22v0 -wi-a----- <128.09g /dev/sdt2:582-1674 Michal> nexus3_root x22v0 Vwi-aot--- <80.04g tp_ssd 13.03 Michal> nexus3_storage x22v0 Vwi-aot--- 5.00t tp_big 46.35 Michal> nexus_old_root x22v0 Vwi-a-t--- <128.09g tp_ssd 95.92 Michal> nexus_old_storage x22v0 Vwi-a-t--- 5.00t tp_big 47.78 Michal> nusknacker-staging x22v0 Vwi-aot--- <128.09g tp_big 99.84 Michal> nussknacker1 x22v0 Vwi-aot--- <128.09g tp_big 27.69 Michal> nussknacker2 x22v0 Vwi-aot--- <128.09g tp_big 21.81 Michal> openvpn-new x22v0 Vwi-aot--- 64.10g tp_big 17.97 Michal> rocket x22v0 Vwi-aot--- <80.04g tp_ssd 97.47 Michal> roger2 x22v0 Vwi-a-t--- 64.10g tp_ssd 24.56 Michal> tmob1 x22v0 -wi-a----- <500.04g /dev/md125:8739-13005 Michal> tmob2 x22v0 -wi-a----- <500.04g /dev/md125:13006-17272 Michal> touk-elk4 x22v0 -wi-ao---- <400.08g /dev/md126:17477-20890 Michal> tp_big x22v0 twi-aot--- 10.32t 53.64 3.27 [tp_big_tdata]:0-90187 Michal> [tp_big_tdata] x22v0 Twi-ao---- 10.32t /dev/md126:0-17476 Michal> [tp_big_tdata] x22v0 Twi-ao---- 10.32t /dev/md126:21984-94694 Michal> [tp_big_tmeta] x22v0 ewi-aor--- 4.10g 100.00 [tp_big_tmeta_rimage_0]:0-34,[tp_big_tmeta_rimage_1]:0-34 Michal> [tp_big_tmeta_rimage_0] x22v0 iwi-aor--- 4.10g /dev/sda3:30-64 Michal> [tp_big_tmeta_rimage_1] x22v0 iwi-aor--- 4.10g /dev/sdb3:30-64 Michal> [tp_big_tmeta_rmeta_0] x22v0 ewi-aor--- 120.00m /dev/sda3:29-29 Michal> [tp_big_tmeta_rmeta_1] x22v0 ewi-aor--- 120.00m /dev/sdb3:29-29 Michal> tp_ssd x22v0 twi-aot--- 1.20t 75.04 1.61 [tp_ssd_tdata]:0-10486 Michal> [tp_ssd_tdata] x22v0 Twi-ao---- 1.20t /dev/md125:0-8738 Michal> [tp_ssd_tdata] x22v0 Twi-ao---- 1.20t /dev/md125:17273-19020 Michal> [tp_ssd_tmeta] x22v0 ewi-aor--- <2.11g 100.00 [tp_ssd_tmeta_rimage_0]:0-17,[tp_ssd_tmeta_rimage_1]:0-17 Michal> [tp_ssd_tmeta_rimage_0] x22v0 iwi-aor--- <2.11g /dev/sda3:11-28 Michal> [tp_ssd_tmeta_rimage_1] x22v0 iwi-aor--- <2.11g /dev/sdb3:11-28 Michal> [tp_ssd_tmeta_rmeta_0] x22v0 ewi-aor--- 120.00m /dev/sda3:10-10 Michal> [tp_ssd_tmeta_rmeta_1] x22v0 ewi-aor--- 120.00m /dev/sdb3:10-10 >> >>>> Are the LVs split across RAID5 PVs by any chance? >> Michal> raid5s are used as PVs, but a single logical volume always uses one only Michal> one physical volume underneath (if that's what you meant by split across). >> >> Ok, that's what I was asking about. It shouldn't matter... but just >> trying to chase down the details. >> >> >>>> It's not clear if you can replicate the problem without using >>>> lvm-thin, but that's what I suspect you might be having problems with. >> Michal> I'll be trying to do that, though the heavier tests will have to wait Michal> until I move all VMs to other hosts (as that is/was our production machnie). >> >> Sure, makes sense. >> >>>> Can you give us the versions of the your tools, and exactly how you >>>> setup your test cases? How long does it take to find the problem? Michal> Regarding this, currently: Michal> kernel: 5.4.0-0.bpo.4-amd64 #1 SMP Debian 5.4.19-1~bpo10+1 (2020-03-09) x86_64 GNU/Linux (was also happening with 5.2.0-0.bpo.3-amd64) Michal> LVM version: 2.03.02(2) (2018-12-18) Michal> Library version: 1.02.155 (2018-12-18) Michal> Driver version: 4.41.0 Michal> mdadm - v4.1 - 2018-10-01 >> Michal> Will get all the details tommorow (the host is on up to date debian Michal> buster, the VMs are mix of archlinuxes and debians (and the issue Michal> happened on both)). >> Michal> As for how long, it's a hit and miss. Sometimes writing and reading back Michal> ~16gb file fails (the cheksum read back differs from what was written) Michal> after 2-3 tries. That's on the host. >> Michal> On the guest, it's been (so far) a guaranteed thing when we were Michal> creating very large tar file (900gb+). As for past two weeks we were Michal> unable to create that file without errors even once. >> >> Ouch! That's not good. Just to confirm, these corruptions are all in >> a thin-lv based filesystem, right? I'd be interested to know if you >> can create another plain LV and cause the same error. Trying to >> simplify the potential problems. Michal> I have been trying to - but so far didn't manage to replicate this with: Michal> - a physical partition Michal> - filesystem directly on a physical partition Michal> - filesystem directly on mdraid Michal> - filesystem directly on a linear volume Michal> Note that this _doesn't_ imply that I _always_ get errors if lvm-thin is in use - as I also had lengthy period of attempts to cause corruption on some thin volume w/o any successes either. But the ones that failed had those in common (so far): md & lvm-thin - with 4 KiB piece(s) being incorrect >> >> >>>> Can you compile the newst kernel and newest thin tools and try them >>>> out? >> Michal> I can, but a bit later (once we move VMs out of the host). >> >>>> >>>> How long does it take to replicate the corruption? >>>> >> Michal> When it happens, it's usually few tries tries of writing a 16gb file Michal> with random patterns and reading it back (directly on host). The Michal> irritating thing is that it can be somewhat hard to reproduce (e.g. Michal> after machine's reboot). >> >>>> Sorry for all the questions, but until there's a test case which is >>>> repeatable, it's going to be hard to chase this down. >>>> >>>> I wonder if running 'fio' tests would be something to try? >>>> >>>> And also changing your RAID5 setup to use the default stride and >>>> stripe widths, instead of the large values you're using. >> Michal> The raid5 is using mdadm's defaults (which is 512 KiB these days for a Michal> chunk). LVM on top is using much longer extents (as we don't really need Michal> 4mb granularity) and the lvm-thin chunks were set to match (and align) Michal> to raid's stripe. >> >>>> >>>> Good luck! >>>> Roger> I have not as of yet seen write corruption (except when a vendors disk Roger> was resetting and it was lying about having written the data prior to Roger> the crash, these were ssds, if your disk write cache is on and you Roger> have a disk reset this can also happen), but have not seen "lost Roger> writes" otherwise, but would expect the 2 read corruption I have seen Roger> to also be able to cause write issues. So for that look for scsi Roger> notifications for disk resets that should not happen. >>>> Roger> I have had a "bad" controller cause read corruptions, those Roger> corruptions would move around, replacing the controller resolved it, Roger> so there may be lack of error checking "inside" some paths in the Roger> card. Lucky I had a number of these controllers and had cold spares Roger> for them. The give away here was 2 separate buses with almost Roger> identical load with 6 separate disks each and all12 disks on 2 buses Roger> had between 47-52 scsi errors, which points to the only component Roger> shared (the controller). >>>> Roger> The backplane and cables are unlikely in general cause this, there is Roger> too much error checking between the controller and the disk from what Roger> I know. >>>> Roger> I have had pre-pcie bus (PCI-X bus, 2 slots shared, both set to 133 Roger> cause random read corruptions, lowering speed to 100 fixed it), this Roger> one was duplicated on multiple identical pieces of hw with all Roger> different parts on the duplication machine. >>>> Roger> I have also seen lost writes (from software) because someone did a Roger> seek without doing a flush which in some versions of the libs loses Roger> the unfilled block when the seek happens (this is noted in the man Roger> page, and I saw it 20years ago, it is still noted in the man page, so Roger> no idea if it was ever fixed). So has more than one application been Roger> noted to see the corruption? >>>> Roger> So one question, have you seen the corruption in a path that would Roger> rely on one controller, or all corruptions you have seen involving Roger> more than one controller? Isolate and test each controller if you Roger> can, or if you can afford to replace it and see if it continues. >>>> >>>> Roger> On Thu, May 7, 2020 at 12:33 PM Michal Soltys wrote: >>>>>> >>>>> Note: this is just general question - if anyone experienced something similar or could suggest how to pinpoint / verify the actual cause. >>>>>> >>>>> Thanks to btrfs's checksumming we discovered somewhat (even if quite rare) nasty silent corruption going on on one of our hosts. Or perhaps "corruption" is not the correct word - the files simply have precise 4kb (1 page) of incorrect data. The incorrect pieces of data look on their own fine - as something that was previously in the place, or written from wrong source. >>>>>> >>>>> The hardware is (can provide more detailed info of course): >>>>>> >>>>> - Supermicro X9DR7-LN4F >>>>> - onboard LSI SAS2308 controller (2 sff-8087 connectors, 1 connected to backplane) >>>>> - 96 gb ram (ecc) >>>>> - 24 disk backplane >>>>>> >>>>> - 1 array connected directly to lsi controller (4 disks, mdraid5, internal bitmap, 512kb chunk) >>>>> - 1 array on the backplane (4 disks, mdraid5, journaled) >>>>> - journal for the above array is: mdraid1, 2 ssd disks (micron 5300 pro disks) >>>>> - 1 btrfs raid1 boot array on motherboard's sata ports (older but still fine intel ssds from DC 3500 series) >>>>>> >>>>> Raid 5 arrays are in lvm volume group, and the logical volumes are used by VMs. Some of the volumes are linear, some are using thin-pools (with metadata on the aforementioned intel ssds, in mirrored config). LVM >>>>> uses large extent sizes (120m) and the chunk-size of thin-pools is set to 1.5m to match underlying raid stripe. Everything is cleanly aligned as well. >>>>>> >>>>> With a doze of testing we managed to roughly rule out the following elements as being the cause: >>>>>> >>>>> - qemu/kvm (issue occured directly on host) >>>>> - backplane (issue occured on disks directly connected via LSI's 2nd connector) >>>>> - cable (as a above, two different cables) >>>>> - memory (unlikely - ECC for once, thoroughly tested, no errors ever reported via edac-util or memtest) >>>>> - mdadm journaling (issue occured on plain mdraid configuration as well) >>>>> - disks themselves (issue occured on two separate mdadm arrays) >>>>> - filesystem (issue occured on both btrfs and ext4 (checksumed manually) ) >>>>>> >>>>> We did not manage to rule out (though somewhat _highly_ unlikely): >>>>>> >>>>> - lvm thin (issue always - so far - occured on lvm thin pools) >>>>> - mdraid (issue always - so far - on mdraid managed arrays) >>>>> - kernel (tested with - in this case - debian's 5.2 and 5.4 kernels, happened with both - so it would imply rather already longstanding bug somewhere) >>>>>> >>>>> And finally - so far - the issue never occured: >>>>>> >>>>> - directly on a disk >>>>> - directly on mdraid >>>>> - on linear lvm volume on top of mdraid >>>>>> >>>>> As far as the issue goes it's: >>>>>> >>>>> - always a 4kb chunk that is incorrect - in a ~1 tb file it can be from a few to few dozens of such chunks >>>>> - we also found (or rather btrfs scrub did) a few small damaged files as well >>>>> - the chunks look like a correct piece of different or previous data >>>>>> >>>>> The 4kb is well, weird ? Doesn't really matter any chunk/stripes sizes anywhere across the stack (lvm - 120m extents, 1.5m chunks on thin pools; mdraid - default 512kb chunks). It does nicely fit a page though ... >>>>>> >>>>> Anyway, if anyone has any ideas or suggestions what could be happening (perhaps with this particular motherboard or vendor) or how to pinpoint the cause - I'll be grateful for any. >>>> >>