* A NFS client partial file corruption problem in recent/current kernels @ 2018-09-11 15:59 Chris Siebenmann 2018-09-11 17:12 ` Trond Myklebust 0 siblings, 1 reply; 15+ messages in thread From: Chris Siebenmann @ 2018-09-11 15:59 UTC (permalink / raw) To: linux-nfs; +Cc: cks We've found a readily reproducable situation where the current NFS client code will provide zero bytes instead of actual data at the end of the file (sort of) to user programs. This can result in program failure, or permanent file corruption if the program reading the file writes the bad data back to the file; otherwise, the corruption goes away when the client's cached data is pushed out of memory (or explicitly dropped by dropping the pagecache through /proc/sys/vm/drop_caches). The reproduction steps are: * on a NFS client, open the file read-write and read to the end of the file (possibly just read the end of the file). * hold the file open read-write and wait for the file size to grow. All the bits of these first two steps appear to be required; you must read the end of the file, you must have the file open read-write, and you must hold it open read-write. * on either another NFS client or the NFS server, append data to the file. * now that your program sees the new file size, try to read the new data (from the old end of the file to the new end of the file). Any data from the old end of file up to the next 4 KB boundary will be zero bytes instead of its actual content; after that, it will be the proper new content. I have a demonstration reproduction program here: https://www.cs.toronto.edu/~cks/vendors/linux-nfs/ This issue isn't present in the Ubuntu 16.04 LTS server kernel (labeled as '4.4.0', plus years of Ubuntu changes) and is present in the Ubuntu 18.04 LTS kernel (labeled 4.15.0) and the Fedora 28 4.17.9 and 4.18.5 kernels. It happens on both NFSv3 and NFSv4 mounts (both with 'sec=sys') and the NFS fileserver OS and the filesystem type (on Linux) doesn't appear to matter; we initially saw this against OmniOS NFS servers using ZFS and have reproduced this against Linux NFS servers on ext4, tmpfs, and ZFS (ZFS on Linux) with both Ubuntu 18.04 and Fedora 28 kernels. This bug causes Alpine to fail when accessing your /var/mail inbox over NFS (and you get new mail delivered to it). There are probably other programs affected, although hopefully not many programs hold files open read-write while other programs are appending data. I'd be happy to answer any further questions, but we have limited ability to try different kernels or kernel changes to see if they change the situation (we don't run stock kernels on any machines; they're all vendor-based ones). - cks ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-11 15:59 A NFS client partial file corruption problem in recent/current kernels Chris Siebenmann @ 2018-09-11 17:12 ` Trond Myklebust 2018-09-11 18:02 ` Chris Siebenmann 0 siblings, 1 reply; 15+ messages in thread From: Trond Myklebust @ 2018-09-11 17:12 UTC (permalink / raw) To: cks, linux-nfs T24gVHVlLCAyMDE4LTA5LTExIGF0IDExOjU5IC0wNDAwLCBDaHJpcyBTaWViZW5tYW5uIHdyb3Rl Og0KPiAgV2UndmUgZm91bmQgYSByZWFkaWx5IHJlcHJvZHVjYWJsZSBzaXR1YXRpb24gd2hlcmUg dGhlIGN1cnJlbnQNCj4gTkZTIGNsaWVudCBjb2RlIHdpbGwgcHJvdmlkZSB6ZXJvIGJ5dGVzIGlu c3RlYWQgb2YgYWN0dWFsIGRhdGEgYXQgdGhlDQo+IGVuZCBvZiB0aGUgZmlsZSAoc29ydCBvZikg dG8gdXNlciBwcm9ncmFtcy4gVGhpcyBjYW4gcmVzdWx0IGluDQo+IHByb2dyYW0NCj4gZmFpbHVy ZSwgb3IgcGVybWFuZW50IGZpbGUgY29ycnVwdGlvbiBpZiB0aGUgcHJvZ3JhbSByZWFkaW5nIHRo ZSBmaWxlDQo+IHdyaXRlcyB0aGUgYmFkIGRhdGEgYmFjayB0byB0aGUgZmlsZTsgb3RoZXJ3aXNl LCB0aGUgY29ycnVwdGlvbiBnb2VzDQo+IGF3YXkgd2hlbiB0aGUgY2xpZW50J3MgY2FjaGVkIGRh dGEgaXMgcHVzaGVkIG91dCBvZiBtZW1vcnkgKG9yDQo+IGV4cGxpY2l0bHkNCj4gZHJvcHBlZCBi eSBkcm9wcGluZyB0aGUgcGFnZWNhY2hlIHRocm91Z2ggL3Byb2Mvc3lzL3ZtL2Ryb3BfY2FjaGVz KS4NCj4gDQo+ICBUaGUgcmVwcm9kdWN0aW9uIHN0ZXBzIGFyZToNCj4gDQo+ICogb24gYSBORlMg Y2xpZW50LCBvcGVuIHRoZSBmaWxlIHJlYWQtd3JpdGUgYW5kIHJlYWQgdG8gdGhlIGVuZCBvZg0K PiB0aGUNCj4gICBmaWxlIChwb3NzaWJseSBqdXN0IHJlYWQgdGhlIGVuZCBvZiB0aGUgZmlsZSku DQo+ICogaG9sZCB0aGUgZmlsZSBvcGVuIHJlYWQtd3JpdGUgYW5kIHdhaXQgZm9yIHRoZSBmaWxl IHNpemUgdG8gZ3Jvdy4NCj4gDQo+ICAgQWxsIHRoZSBiaXRzIG9mIHRoZXNlIGZpcnN0IHR3byBz dGVwcyBhcHBlYXIgdG8gYmUgcmVxdWlyZWQ7IHlvdQ0KPiBtdXN0DQo+ICAgcmVhZCB0aGUgZW5k IG9mIHRoZSBmaWxlLCB5b3UgbXVzdCBoYXZlIHRoZSBmaWxlIG9wZW4gcmVhZC13cml0ZSwNCj4g ICBhbmQgeW91IG11c3QgaG9sZCBpdCBvcGVuIHJlYWQtd3JpdGUuDQo+IA0KPiAqIG9uIGVpdGhl ciBhbm90aGVyIE5GUyBjbGllbnQgb3IgdGhlIE5GUyBzZXJ2ZXIsIGFwcGVuZCBkYXRhIHRvIHRo ZQ0KPiAgIGZpbGUuDQo+IA0KPiAqIG5vdyB0aGF0IHlvdXIgcHJvZ3JhbSBzZWVzIHRoZSBuZXcg ZmlsZSBzaXplLCB0cnkgdG8gcmVhZCB0aGUgbmV3DQo+ICAgZGF0YSAoZnJvbSB0aGUgb2xkIGVu ZCBvZiB0aGUgZmlsZSB0byB0aGUgbmV3IGVuZCBvZiB0aGUgZmlsZSkuDQo+ICAgQW55IGRhdGEg ZnJvbSB0aGUgb2xkIGVuZCBvZiBmaWxlIHVwIHRvIHRoZSBuZXh0IDQgS0IgYm91bmRhcnkgd2ls bA0KPiAgIGJlIHplcm8gYnl0ZXMgaW5zdGVhZCBvZiBpdHMgYWN0dWFsIGNvbnRlbnQ7IGFmdGVy IHRoYXQsIGl0IHdpbGwgYmUNCj4gICB0aGUgcHJvcGVyIG5ldyBjb250ZW50Lg0KPiANCj4gSSBo YXZlIGEgZGVtb25zdHJhdGlvbiByZXByb2R1Y3Rpb24gcHJvZ3JhbSBoZXJlOg0KPiAJaHR0cHM6 Ly93d3cuY3MudG9yb250by5lZHUvfmNrcy92ZW5kb3JzL2xpbnV4LW5mcy8NCj4gDQo+IFRoaXMg aXNzdWUgaXNuJ3QgcHJlc2VudCBpbiB0aGUgVWJ1bnR1IDE2LjA0IExUUyBzZXJ2ZXIga2VybmVs DQo+IChsYWJlbGVkDQo+IGFzICc0LjQuMCcsIHBsdXMgeWVhcnMgb2YgVWJ1bnR1IGNoYW5nZXMp IGFuZCBpcyBwcmVzZW50IGluIHRoZQ0KPiBVYnVudHUNCj4gMTguMDQgTFRTIGtlcm5lbCAobGFi ZWxlZCA0LjE1LjApIGFuZCB0aGUgRmVkb3JhIDI4IDQuMTcuOSBhbmQgNC4xOC41DQo+IGtlcm5l bHMuIEl0IGhhcHBlbnMgb24gYm90aCBORlN2MyBhbmQgTkZTdjQgbW91bnRzIChib3RoIHdpdGgN Cj4gJ3NlYz1zeXMnKQ0KPiBhbmQgdGhlIE5GUyBmaWxlc2VydmVyIE9TIGFuZCB0aGUgZmlsZXN5 c3RlbSB0eXBlIChvbiBMaW51eCkgZG9lc24ndA0KPiBhcHBlYXIgdG8gbWF0dGVyOyB3ZSBpbml0 aWFsbHkgc2F3IHRoaXMgYWdhaW5zdCBPbW5pT1MgTkZTIHNlcnZlcnMNCj4gdXNpbmcNCj4gWkZT IGFuZCBoYXZlIHJlcHJvZHVjZWQgdGhpcyBhZ2FpbnN0IExpbnV4IE5GUyBzZXJ2ZXJzIG9uIGV4 dDQsDQo+IHRtcGZzLA0KPiBhbmQgWkZTIChaRlMgb24gTGludXgpIHdpdGggYm90aCBVYnVudHUg MTguMDQgYW5kIEZlZG9yYSAyOCBrZXJuZWxzLg0KPiANCj4gIFRoaXMgYnVnIGNhdXNlcyBBbHBp bmUgdG8gZmFpbCB3aGVuIGFjY2Vzc2luZyB5b3VyIC92YXIvbWFpbCBpbmJveA0KPiBvdmVyIE5G UyAoYW5kIHlvdSBnZXQgbmV3IG1haWwgZGVsaXZlcmVkIHRvIGl0KS4gVGhlcmUgYXJlIHByb2Jh Ymx5DQo+IG90aGVyDQo+IHByb2dyYW1zIGFmZmVjdGVkLCBhbHRob3VnaCBob3BlZnVsbHkgbm90 IG1hbnkgcHJvZ3JhbXMgaG9sZCBmaWxlcw0KPiBvcGVuDQo+IHJlYWQtd3JpdGUgd2hpbGUgb3Ro ZXIgcHJvZ3JhbXMgYXJlIGFwcGVuZGluZyBkYXRhLg0KPiANCj4gIEknZCBiZSBoYXBweSB0byBh bnN3ZXIgYW55IGZ1cnRoZXIgcXVlc3Rpb25zLCBidXQgd2UgaGF2ZSBsaW1pdGVkDQo+IGFiaWxp dHkgdG8gdHJ5IGRpZmZlcmVudCBrZXJuZWxzIG9yIGtlcm5lbCBjaGFuZ2VzIHRvIHNlZSBpZiB0 aGV5DQo+IGNoYW5nZQ0KPiB0aGUgc2l0dWF0aW9uICh3ZSBkb24ndCBydW4gc3RvY2sga2VybmVs cyBvbiBhbnkgbWFjaGluZXM7IHRoZXkncmUNCj4gYWxsDQo+IHZlbmRvci1iYXNlZCBvbmVzKS4N Cj4gDQoNClBsZWFzZSBzZWUgaHR0cDovL25mcy5zb3VyY2Vmb3JnZS5uZXQvI2ZhcV9hOA0KDQoN Ci0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXIsIEhhbW1l cnNwYWNlDQp0cm9uZC5teWtsZWJ1c3RAaGFtbWVyc3BhY2UuY29tDQoNCg0K ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-11 17:12 ` Trond Myklebust @ 2018-09-11 18:02 ` Chris Siebenmann 2018-09-11 20:00 ` Trond Myklebust 0 siblings, 1 reply; 15+ messages in thread From: Chris Siebenmann @ 2018-09-11 18:02 UTC (permalink / raw) To: Trond Myklebust; +Cc: linux-nfs, cks > > We've found a readily reproducable situation where the current > > NFS client code will provide zero bytes instead of actual data at > > the end of the file (sort of) to user programs. This can result > > in program failure, or permanent file corruption if the program > > reading the file writes the bad data back to the file; otherwise, > > the corruption goes away when the client's cached data is pushed out > > of memory (or explicitly dropped by dropping the pagecache through > > /proc/sys/vm/drop_caches). [...] > Please see http://nfs.sourceforge.net/#faq_a8 I don't think this is a close to open consistency issue, or if it is I would argue that it is a clear bug on the Linux NFS client. I have a number of reasons for saying this: - the client clearly sees the new attributes; it knows that the file has been extended from the previous state that it knew of. My demo program specifically waits until user-level fstat() returns a different result, which I believe means that the client kernel has seen a different GETATTR result and so should have purged its cache (based on what the FAQ says). (Unless the FAQ means that the kernel absolutely refuses to guarantee anything about file consistency unless you close and then reopen the file, even if it *knows* that the file has changed on the server, which isn't clear from how the FAQ is currently written.) - the client is fetching some new data from the fileserver (data after the partial 4 KB page at the old end of the file). - the client isn't writing to the file in my demonstration program; it's only opening it in read-write mode and then reading it. Also, this doesn't happen if the client does exactly the same set of operations but has the file open read-only (with it staying open throughout). - this didn't happen in older kernels. In addition, although I didn't mention it in my original email, this happens on a NFS filesystem mounted 'noac'. Pragmatically, Alpine used to work with NFS mounted filesystems where email was appended to them from other machines and it no longer does, and the only difference is the kernel version involved on the client. This breakage is actively dangerous. - cks ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-11 18:02 ` Chris Siebenmann @ 2018-09-11 20:00 ` Trond Myklebust 2018-09-11 20:38 ` Chris Siebenmann 2018-09-11 20:40 ` Chuck Lever 0 siblings, 2 replies; 15+ messages in thread From: Trond Myklebust @ 2018-09-11 20:00 UTC (permalink / raw) To: cks; +Cc: linux-nfs T24gVHVlLCAyMDE4LTA5LTExIGF0IDE0OjAyIC0wNDAwLCBDaHJpcyBTaWViZW5tYW5uIHdyb3Rl Og0KPiA+ID4gIFdlJ3ZlIGZvdW5kIGEgcmVhZGlseSByZXByb2R1Y2FibGUgc2l0dWF0aW9uIHdo ZXJlIHRoZSBjdXJyZW50DQo+ID4gPiBORlMgY2xpZW50IGNvZGUgd2lsbCBwcm92aWRlIHplcm8g Ynl0ZXMgaW5zdGVhZCBvZiBhY3R1YWwgZGF0YSBhdA0KPiA+ID4gdGhlIGVuZCBvZiB0aGUgZmls ZSAoc29ydCBvZikgdG8gdXNlciBwcm9ncmFtcy4gVGhpcyBjYW4gcmVzdWx0DQo+ID4gPiBpbiBw cm9ncmFtIGZhaWx1cmUsIG9yIHBlcm1hbmVudCBmaWxlIGNvcnJ1cHRpb24gaWYgdGhlIHByb2dy YW0NCj4gPiA+IHJlYWRpbmcgdGhlIGZpbGUgd3JpdGVzIHRoZSBiYWQgZGF0YSBiYWNrIHRvIHRo ZSBmaWxlOyBvdGhlcndpc2UsDQo+ID4gPiB0aGUgY29ycnVwdGlvbiBnb2VzIGF3YXkgd2hlbiB0 aGUgY2xpZW50J3MgY2FjaGVkIGRhdGEgaXMgcHVzaGVkDQo+ID4gPiBvdXQNCj4gPiA+IG9mIG1l bW9yeSAob3IgZXhwbGljaXRseSBkcm9wcGVkIGJ5IGRyb3BwaW5nIHRoZSBwYWdlY2FjaGUNCj4g PiA+IHRocm91Z2gNCj4gPiA+IC9wcm9jL3N5cy92bS9kcm9wX2NhY2hlcykuDQo+IA0KPiBbLi4u XQ0KPiA+IFBsZWFzZSBzZWUgaHR0cDovL25mcy5zb3VyY2Vmb3JnZS5uZXQvI2ZhcV9hOA0KPiAN Cj4gIEkgZG9uJ3QgdGhpbmsgdGhpcyBpcyBhIGNsb3NlIHRvIG9wZW4gY29uc2lzdGVuY3kgaXNz dWUsIG9yIGlmIGl0IGlzDQo+IEkgd291bGQgYXJndWUgdGhhdCBpdCBpcyBhIGNsZWFyIGJ1ZyBv biB0aGUgTGludXggTkZTIGNsaWVudC4gSSBoYXZlDQo+IGEgbnVtYmVyIG9mIHJlYXNvbnMgZm9y IHNheWluZyB0aGlzOg0KPiANCj4gLSB0aGUgY2xpZW50IGNsZWFybHkgc2VlcyB0aGUgbmV3IGF0 dHJpYnV0ZXM7IGl0IGtub3dzIHRoYXQgdGhlIGZpbGUNCj4gICBoYXMgYmVlbiBleHRlbmRlZCBm cm9tIHRoZSBwcmV2aW91cyBzdGF0ZSB0aGF0IGl0IGtuZXcgb2YuIE15IGRlbW8NCj4gICBwcm9n cmFtIHNwZWNpZmljYWxseSB3YWl0cyB1bnRpbCB1c2VyLWxldmVsIGZzdGF0KCkgcmV0dXJucyBh DQo+IGRpZmZlcmVudA0KPiAgIHJlc3VsdCwgd2hpY2ggSSBiZWxpZXZlIG1lYW5zIHRoYXQgdGhl IGNsaWVudCBrZXJuZWwgaGFzIHNlZW4gYQ0KPiBkaWZmZXJlbnQNCj4gICBHRVRBVFRSIHJlc3Vs dCBhbmQgc28gc2hvdWxkIGhhdmUgcHVyZ2VkIGl0cyBjYWNoZSAoYmFzZWQgb24gd2hhdA0KPiB0 aGUNCj4gICBGQVEgc2F5cykuDQo+IA0KPiAgIChVbmxlc3MgdGhlIEZBUSBtZWFucyB0aGF0IHRo ZSBrZXJuZWwgYWJzb2x1dGVseSByZWZ1c2VzIHRvDQo+IGd1YXJhbnRlZQ0KPiAgIGFueXRoaW5n IGFib3V0IGZpbGUgY29uc2lzdGVuY3kgdW5sZXNzIHlvdSBjbG9zZSBhbmQgdGhlbiByZW9wZW4N Cj4gdGhlDQo+ICAgZmlsZSwgZXZlbiBpZiBpdCAqa25vd3MqIHRoYXQgdGhlIGZpbGUgaGFzIGNo YW5nZWQgb24gdGhlIHNlcnZlciwNCj4gICB3aGljaCBpc24ndCBjbGVhciBmcm9tIGhvdyB0aGUg RkFRIGlzIGN1cnJlbnRseSB3cml0dGVuLikNCj4gDQo+IC0gdGhlIGNsaWVudCBpcyBmZXRjaGlu ZyBzb21lIG5ldyBkYXRhIGZyb20gdGhlIGZpbGVzZXJ2ZXIgKGRhdGENCj4gYWZ0ZXINCj4gICB0 aGUgcGFydGlhbCA0IEtCIHBhZ2UgYXQgdGhlIG9sZCBlbmQgb2YgdGhlIGZpbGUpLg0KPiANCj4g LSB0aGUgY2xpZW50IGlzbid0IHdyaXRpbmcgdG8gdGhlIGZpbGUgaW4gbXkgZGVtb25zdHJhdGlv biBwcm9ncmFtOw0KPiBpdCdzDQo+ICAgb25seSBvcGVuaW5nIGl0IGluIHJlYWQtd3JpdGUgbW9k ZSBhbmQgdGhlbiByZWFkaW5nIGl0LiBBbHNvLCB0aGlzDQo+ICAgZG9lc24ndCBoYXBwZW4gaWYg dGhlIGNsaWVudCBkb2VzIGV4YWN0bHkgdGhlIHNhbWUgc2V0IG9mDQo+IG9wZXJhdGlvbnMNCj4g ICBidXQgaGFzIHRoZSBmaWxlIG9wZW4gcmVhZC1vbmx5ICh3aXRoIGl0IHN0YXlpbmcgb3BlbiB0 aHJvdWdob3V0KS4NCj4gDQo+IC0gdGhpcyBkaWRuJ3QgaGFwcGVuIGluIG9sZGVyIGtlcm5lbHMu DQo+IA0KPiBJbiBhZGRpdGlvbiwgYWx0aG91Z2ggSSBkaWRuJ3QgbWVudGlvbiBpdCBpbiBteSBv cmlnaW5hbCBlbWFpbCwgdGhpcw0KPiBoYXBwZW5zIG9uIGEgTkZTIGZpbGVzeXN0ZW0gbW91bnRl ZCAnbm9hYycuDQo+IA0KPiBQcmFnbWF0aWNhbGx5LCBBbHBpbmUgdXNlZCB0byB3b3JrIHdpdGgg TkZTIG1vdW50ZWQgZmlsZXN5c3RlbXMgd2hlcmUNCj4gZW1haWwgd2FzIGFwcGVuZGVkIHRvIHRo ZW0gZnJvbSBvdGhlciBtYWNoaW5lcyBhbmQgaXQgbm8gbG9uZ2VyIGRvZXMsDQo+IGFuZCB0aGUg b25seSBkaWZmZXJlbmNlIGlzIHRoZSBrZXJuZWwgdmVyc2lvbiBpbnZvbHZlZCBvbiB0aGUgY2xp ZW50Lg0KPiBUaGlzIGJyZWFrYWdlIGlzIGFjdGl2ZWx5IGRhbmdlcm91cy4NCg0KU3VyZSwgYnV0 IHVubGVzcyB5b3UgYXJlIGxvY2tpbmcgdGhlIGZpbGUsIG9yIHlvdSBhcmUgZXhwbGljaXRseSB1 c2luZw0KT19ESVJFQ1QgdG8gZG8gdW5jYWNoZWQgSS9PLCB0aGVuIHlvdSBhcmUgaW4gdmlvbGF0 aW9uIG9mIHRoZSBjbG9zZS10by0NCm9wZW4gY29uc2lzdGVuY3kgbW9kZWwsIGFuZCB0aGUgY2xp ZW50IGlzIGdvaW5nIHRvIGJlaGF2ZSBhcyB5b3UNCmRlc2NyaWJlIGFib3ZlLiBORlMgdXNlcyBh IGRpc3RyaWJ1dGVkIGZpbGVzeXN0ZW0gbW9kZWwsIG5vdCBhDQpjbHVzdGVyZWQgb25lLg0KDQot LSANClRyb25kIE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWludGFpbmVyLCBIYW1tZXJz cGFjZQ0KdHJvbmQubXlrbGVidXN0QGhhbW1lcnNwYWNlLmNvbQ0KDQoNCg== ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-11 20:00 ` Trond Myklebust @ 2018-09-11 20:38 ` Chris Siebenmann 2018-09-11 20:40 ` Chuck Lever 1 sibling, 0 replies; 15+ messages in thread From: Chris Siebenmann @ 2018-09-11 20:38 UTC (permalink / raw) To: Trond Myklebust; +Cc: linux-nfs, cks > > Pragmatically, Alpine used to work with NFS mounted filesystems where > > email was appended to them from other machines and it no longer does, > > and the only difference is the kernel version involved on the client. > > This breakage is actively dangerous. > > Sure, but unless you are locking the file, or you are explicitly using > O_DIRECT to do uncached I/O, then you are in violation of the close-to- > open consistency model, and the client is going to behave as you > describe above. NFS uses a distributed filesystem model, not a > clustered one. In the close to open consistency model, is it legal and proper to do the following sequence: - open a file read-write - fstat() the file until the reported file size changes - close the file; open it again read-write - read new data from the file If this sequence is legal, then I think there is a bug, because I can make the zero bytes appear even with this sequence. I've updated my reproduction program, in https://www.cs.toronto.edu/~cks/vendors/linux-nfs/ to have a '--reopen' option that does this. If this sequence is not legal and can legally result in corrupted data in the file, then I think there is a potential problem, because it creates a situation where one program (opening the file read-write and holding it open) could cause corruption for another program (which properly opens and closes the file). I can reproduce this with two running instances of my test program. Perhaps this is considered invalid because it is a violation of close to open across the entire client kernel, but if so I feel this is dangerous; it puts all programs reading NFS mounted files at the mercy of everything else on the system, no matter how much they try to do the right thing. They can open it read only and close it while they wait for changes and then reopen it read only afterward, and they will still get corrupted data. - cks ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-11 20:00 ` Trond Myklebust 2018-09-11 20:38 ` Chris Siebenmann @ 2018-09-11 20:40 ` Chuck Lever 2018-09-11 20:47 ` Chris Siebenmann 2018-09-11 20:56 ` Trond Myklebust 1 sibling, 2 replies; 15+ messages in thread From: Chuck Lever @ 2018-09-11 20:40 UTC (permalink / raw) To: Trond Myklebust; +Cc: cks, Linux NFS Mailing List > On Sep 11, 2018, at 4:00 PM, Trond Myklebust <trondmy@hammerspace.com> = wrote: >=20 > On Tue, 2018-09-11 at 14:02 -0400, Chris Siebenmann wrote: >>>> We've found a readily reproducable situation where the current >>>> NFS client code will provide zero bytes instead of actual data at >>>> the end of the file (sort of) to user programs. This can result >>>> in program failure, or permanent file corruption if the program >>>> reading the file writes the bad data back to the file; otherwise, >>>> the corruption goes away when the client's cached data is pushed >>>> out >>>> of memory (or explicitly dropped by dropping the pagecache >>>> through >>>> /proc/sys/vm/drop_caches). >>=20 >> [...] >>> Please see http://nfs.sourceforge.net/#faq_a8 >>=20 >> I don't think this is a close to open consistency issue, or if it is >> I would argue that it is a clear bug on the Linux NFS client. I have >> a number of reasons for saying this: >>=20 >> - the client clearly sees the new attributes; it knows that the file >> has been extended from the previous state that it knew of. My demo >> program specifically waits until user-level fstat() returns a >> different >> result, which I believe means that the client kernel has seen a >> different >> GETATTR result and so should have purged its cache (based on what >> the >> FAQ says). >>=20 >> (Unless the FAQ means that the kernel absolutely refuses to >> guarantee >> anything about file consistency unless you close and then reopen >> the >> file, even if it *knows* that the file has changed on the server, >> which isn't clear from how the FAQ is currently written.) >>=20 >> - the client is fetching some new data from the fileserver (data >> after >> the partial 4 KB page at the old end of the file). >>=20 >> - the client isn't writing to the file in my demonstration program; >> it's >> only opening it in read-write mode and then reading it. Also, this >> doesn't happen if the client does exactly the same set of >> operations >> but has the file open read-only (with it staying open throughout). >>=20 >> - this didn't happen in older kernels. >>=20 >> In addition, although I didn't mention it in my original email, this >> happens on a NFS filesystem mounted 'noac'. >>=20 >> Pragmatically, Alpine used to work with NFS mounted filesystems where >> email was appended to them from other machines and it no longer does, >> and the only difference is the kernel version involved on the client. >> This breakage is actively dangerous. >=20 > Sure, but unless you are locking the file, or you are explicitly using > O_DIRECT to do uncached I/O, then you are in violation of the = close-to- > open consistency model, and the client is going to behave as you > describe above. NFS uses a distributed filesystem model, not a > clustered one. I would expect Alpine to work if "vers=3D3,noac" is in use. -- Chuck Lever ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-11 20:40 ` Chuck Lever @ 2018-09-11 20:47 ` Chris Siebenmann 2018-09-11 21:25 ` Trond Myklebust 2018-09-11 20:56 ` Trond Myklebust 1 sibling, 1 reply; 15+ messages in thread From: Chris Siebenmann @ 2018-09-11 20:47 UTC (permalink / raw) To: Chuck Lever; +Cc: Trond Myklebust, cks, Linux NFS Mailing List > >> Pragmatically, Alpine used to work with NFS mounted filesystems where > >> email was appended to them from other machines and it no longer does, > >> and the only difference is the kernel version involved on the client. > >> This breakage is actively dangerous. > > > > Sure, but unless you are locking the file, or you are explicitly using > > O_DIRECT to do uncached I/O, then you are in violation of the close-to- > > open consistency model, and the client is going to behave as you > > describe above. NFS uses a distributed filesystem model, not a > > clustered one. > > I would expect Alpine to work if "vers=3,noac" is in use. We're setting noac (and vers=3) for our /var/mail NFS mount, but perhaps we've got some additional option that is wrong here and should be changed? According to /proc/mounts, our mount settings for /var/mail are: fs0.cs.toronto.edu:/cs/mail /var/mail nfs rw,sync,nosuid,nodev,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,noac,noacl,proto=tcp,timeo=11,retrans=4,sec=sys,mountaddr=128.100.3.130,mountvers=3,mountport=34630,mountproto=udp,local_lock=none,addr=128.100.3.130 (To an OmniOS fileserver with ZFS.) - cks ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-11 20:47 ` Chris Siebenmann @ 2018-09-11 21:25 ` Trond Myklebust 2018-09-11 21:39 ` Chris Siebenmann 0 siblings, 1 reply; 15+ messages in thread From: Trond Myklebust @ 2018-09-11 21:25 UTC (permalink / raw) To: cks, chuck.lever; +Cc: linux-nfs T24gVHVlLCAyMDE4LTA5LTExIGF0IDE2OjQ3IC0wNDAwLCBDaHJpcyBTaWViZW5tYW5uIHdyb3Rl Og0KPiA+ID4gPiBQcmFnbWF0aWNhbGx5LCBBbHBpbmUgdXNlZCB0byB3b3JrIHdpdGggTkZTIG1v dW50ZWQgZmlsZXN5c3RlbXMNCj4gPiA+ID4gd2hlcmUNCj4gPiA+ID4gZW1haWwgd2FzIGFwcGVu ZGVkIHRvIHRoZW0gZnJvbSBvdGhlciBtYWNoaW5lcyBhbmQgaXQgbm8gbG9uZ2VyDQo+ID4gPiA+ IGRvZXMsDQo+ID4gPiA+IGFuZCB0aGUgb25seSBkaWZmZXJlbmNlIGlzIHRoZSBrZXJuZWwgdmVy c2lvbiBpbnZvbHZlZCBvbiB0aGUNCj4gPiA+ID4gY2xpZW50Lg0KPiA+ID4gPiBUaGlzIGJyZWFr YWdlIGlzIGFjdGl2ZWx5IGRhbmdlcm91cy4NCj4gPiA+IA0KPiA+ID4gU3VyZSwgYnV0IHVubGVz cyB5b3UgYXJlIGxvY2tpbmcgdGhlIGZpbGUsIG9yIHlvdSBhcmUgZXhwbGljaXRseQ0KPiA+ID4g dXNpbmcNCj4gPiA+IE9fRElSRUNUIHRvIGRvIHVuY2FjaGVkIEkvTywgdGhlbiB5b3UgYXJlIGlu IHZpb2xhdGlvbiBvZiB0aGUNCj4gPiA+IGNsb3NlLXRvLQ0KPiA+ID4gb3BlbiBjb25zaXN0ZW5j eSBtb2RlbCwgYW5kIHRoZSBjbGllbnQgaXMgZ29pbmcgdG8gYmVoYXZlIGFzIHlvdQ0KPiA+ID4g ZGVzY3JpYmUgYWJvdmUuIE5GUyB1c2VzIGEgZGlzdHJpYnV0ZWQgZmlsZXN5c3RlbSBtb2RlbCwg bm90IGENCj4gPiA+IGNsdXN0ZXJlZCBvbmUuDQo+ID4gDQo+ID4gSSB3b3VsZCBleHBlY3QgQWxw aW5lIHRvIHdvcmsgaWYgInZlcnM9Myxub2FjIiBpcyBpbiB1c2UuDQo+IA0KPiAgV2UncmUgc2V0 dGluZyBub2FjIChhbmQgdmVycz0zKSBmb3Igb3VyIC92YXIvbWFpbCBORlMgbW91bnQsIGJ1dA0K PiBwZXJoYXBzIHdlJ3ZlIGdvdCBzb21lIGFkZGl0aW9uYWwgb3B0aW9uIHRoYXQgaXMgd3Jvbmcg aGVyZSBhbmQNCj4gc2hvdWxkIGJlIGNoYW5nZWQ/IEFjY29yZGluZyB0byAvcHJvYy9tb3VudHMs IG91ciBtb3VudCBzZXR0aW5ncw0KPiBmb3IgL3Zhci9tYWlsIGFyZToNCj4gDQo+IAlmczAuY3Mu dG9yb250by5lZHU6L2NzL21haWwgL3Zhci9tYWlsIG5mcw0KPiBydyxzeW5jLG5vc3VpZCxub2Rl dixyZWxhdGltZSx2ZXJzPTMscnNpemU9MTA0ODU3Nix3c2l6ZT0xMDQ4NTc2LG5hbWwNCj4gZW49 MjU1LGFjcmVnbWluPTAsYWNyZWdtYXg9MCxhY2Rpcm1pbj0wLGFjZGlybWF4PTAsaGFyZCxub2Fj LG5vYWNsLHByDQo+IG90bz10Y3AsdGltZW89MTEscmV0cmFucz00LHNlYz1zeXMsbW91bnRhZGRy PTEyOC4xMDAuMy4xMzAsbW91bnR2ZXJzPQ0KPiAzLG1vdW50cG9ydD0zNDYzMCxtb3VudHByb3Rv PXVkcCxsb2NhbF9sb2NrPW5vbmUsYWRkcj0xMjguMTAwLjMuMTMwDQo+IA0KDQpUaGlzIGhhcyBu b3RoaW5nIHRvIGRvIHdpdGggbW91bnQgb3B0aW9ucy4gQnVmZmVyZWQgcmVhZHMgb2YgYSBmaWxl DQp0aGF0IGlzIGJlaW5nIHdyaXR0ZW4gdG8gb3ZlciBORlMgd2l0aG91dCB1c2luZyBsb2NraW5n IGlzIGluaGVyZW50bHkNCnVuc2FmZS4gVGhhdCBhbHdheXMgaGFzIGJlZW4gdGhlIGNhc2UuLi4N Cg0KQm90aCB3cml0ZXMgYW5kIHJlYWRzIGNhbiBiZSByZW9yZGVyZWQgYnkgdGhlIFJQQyBsYXll ciBvbiBib3RoIHRoZQ0KY2xpZW50IGFuZCB0aGUgc2VydmVyLCBhbmQgdGhleSBjYW4gYmUgZnVy dGhlciByZW9yZGVyZWQgYnkgdGhlIE5GUw0KbGF5ZXIgb24gdGhlIHNlcnZlci4gSW4gcHJhY3Rp Y2UsIHRoaXMgbWVhbnMgdGhhdCB5b3UgY2FuIGZpbmQgeW91cnNlbGYNCnJlYWRpbmcgcGFydHMg b2YgdGhlIGZpbGUgdGhhdCBoYXZlIG5vdCB5ZXQgY29tcGxldGVkIGJlaW5nIHdyaXR0ZW4gdG8s DQpiZWNhdXNlLCBmb3IgZXhhbXBsZSwgYSB3cml0ZSB0aGF0IGV4dGVuZGVkIHRoZSBmaWxlIGZy b20gb2Zmc2V0IDQwOTYtDQo4MTkxIGNvbXBsZXRlZCBiZWZvcmUgdGhlIHdyaXRlIHRoYXQgd2Fz IHN1cHBvc2VkIHRvIGV4dGVuZCBpdCBmcm9tDQpvZmZzZXQgMC00MDk1IHdhcyBwcm9jZXNzZWQg YnkgdGhlIHNlcnZlci4NCg0KVGhpcyBpcyB0cnVlIHdpdGggb3Igd2l0aG91dCBub2FjLi4uDQoN Ci0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXIsIEhhbW1l cnNwYWNlDQp0cm9uZC5teWtsZWJ1c3RAaGFtbWVyc3BhY2UuY29tDQoNCg0K ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-11 21:25 ` Trond Myklebust @ 2018-09-11 21:39 ` Chris Siebenmann 2018-09-11 22:12 ` Trond Myklebust 0 siblings, 1 reply; 15+ messages in thread From: Chris Siebenmann @ 2018-09-11 21:39 UTC (permalink / raw) To: Trond Myklebust; +Cc: chuck.lever, linux-nfs, cks > This has nothing to do with mount options. Buffered reads of a file > that is being written to over NFS without using locking is inherently > unsafe. That always has been the case... > > Both writes and reads can be reordered by the RPC layer on both the > client and the server, and they can be further reordered by the > NFS layer on the server. In practice, this means that you can find > yourself reading parts of the file that have not yet completed being > written to, because, for example, a write that extended the file from > offset 4096- 8191 completed before the write that was supposed to > extend it from offset 0-4095 was processed by the server. Our issue also happens when the writes are done on the fileserver, though, and they occur even if you allow plenty of time for the writes to settle. I can run my test program in a mode where it explicitly waits for me to tell it to continue, do the appending to the file on the fileserver, 'sync' on the fileserver, wait five minutes, and the NFS client will still see those zero bytes when it tries to read the new data. (To make sure the 'five minutes' bit wasn't hyperbole, I just tested it.) - cks ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-11 21:39 ` Chris Siebenmann @ 2018-09-11 22:12 ` Trond Myklebust 2018-09-11 23:45 ` Chris Siebenmann 0 siblings, 1 reply; 15+ messages in thread From: Trond Myklebust @ 2018-09-11 22:12 UTC (permalink / raw) To: cks; +Cc: linux-nfs, chuck.lever T24gVHVlLCAyMDE4LTA5LTExIGF0IDE3OjM5IC0wNDAwLCBDaHJpcyBTaWViZW5tYW5uIHdyb3Rl Og0KPiA+IFRoaXMgaGFzIG5vdGhpbmcgdG8gZG8gd2l0aCBtb3VudCBvcHRpb25zLiBCdWZmZXJl ZCByZWFkcyBvZiBhIGZpbGUNCj4gPiB0aGF0IGlzIGJlaW5nIHdyaXR0ZW4gdG8gb3ZlciBORlMg d2l0aG91dCB1c2luZyBsb2NraW5nIGlzDQo+ID4gaW5oZXJlbnRseQ0KPiA+IHVuc2FmZS4gVGhh dCBhbHdheXMgaGFzIGJlZW4gdGhlIGNhc2UuLi4NCj4gPiANCj4gPiBCb3RoIHdyaXRlcyBhbmQg cmVhZHMgY2FuIGJlIHJlb3JkZXJlZCBieSB0aGUgUlBDIGxheWVyIG9uIGJvdGggdGhlDQo+ID4g Y2xpZW50IGFuZCB0aGUgc2VydmVyLCBhbmQgdGhleSBjYW4gYmUgZnVydGhlciByZW9yZGVyZWQg YnkgdGhlDQo+ID4gTkZTIGxheWVyIG9uIHRoZSBzZXJ2ZXIuIEluIHByYWN0aWNlLCB0aGlzIG1l YW5zIHRoYXQgeW91IGNhbiBmaW5kDQo+ID4geW91cnNlbGYgcmVhZGluZyBwYXJ0cyBvZiB0aGUg ZmlsZSB0aGF0IGhhdmUgbm90IHlldCBjb21wbGV0ZWQNCj4gPiBiZWluZw0KPiA+IHdyaXR0ZW4g dG8sIGJlY2F1c2UsIGZvciBleGFtcGxlLCBhIHdyaXRlIHRoYXQgZXh0ZW5kZWQgdGhlIGZpbGUN Cj4gPiBmcm9tDQo+ID4gb2Zmc2V0IDQwOTYtIDgxOTEgY29tcGxldGVkIGJlZm9yZSB0aGUgd3Jp dGUgdGhhdCB3YXMgc3VwcG9zZWQgdG8NCj4gPiBleHRlbmQgaXQgZnJvbSBvZmZzZXQgMC00MDk1 IHdhcyBwcm9jZXNzZWQgYnkgdGhlIHNlcnZlci4NCj4gDQo+ICBPdXIgaXNzdWUgYWxzbyBoYXBw ZW5zIHdoZW4gdGhlIHdyaXRlcyBhcmUgZG9uZSBvbiB0aGUgZmlsZXNlcnZlciwNCj4gdGhvdWdo LCBhbmQgdGhleSBvY2N1ciBldmVuIGlmIHlvdSBhbGxvdyBwbGVudHkgb2YgdGltZSBmb3IgdGhl DQo+IHdyaXRlcw0KPiB0byBzZXR0bGUuIEkgY2FuIHJ1biBteSB0ZXN0IHByb2dyYW0gaW4gYSBt b2RlIHdoZXJlIGl0IGV4cGxpY2l0bHkNCj4gd2FpdHMNCj4gZm9yIG1lIHRvIHRlbGwgaXQgdG8g Y29udGludWUsIGRvIHRoZSBhcHBlbmRpbmcgdG8gdGhlIGZpbGUgb24gdGhlDQo+IGZpbGVzZXJ2 ZXIsICdzeW5jJyBvbiB0aGUgZmlsZXNlcnZlciwgd2FpdCBmaXZlIG1pbnV0ZXMsIGFuZCB0aGUg TkZTDQo+IGNsaWVudCB3aWxsIHN0aWxsIHNlZSB0aG9zZSB6ZXJvIGJ5dGVzIHdoZW4gaXQgdHJp ZXMgdG8gcmVhZCB0aGUgbmV3DQo+IGRhdGEuDQo+IA0KPiAoVG8gbWFrZSBzdXJlIHRoZSAnZml2 ZSBtaW51dGVzJyBiaXQgd2Fzbid0IGh5cGVyYm9sZSwgSSBqdXN0IHRlc3RlZA0KPiBpdC4pDQo+ IA0KDQpUaGF0J3MgaGFwcGVuaW5nIGJlY2F1c2Ugd2UncmUgbm90IG9wdGltaXNpbmcgZm9yIHRo ZSBicm9rZW4gY2FzZSwgYW5kDQppbnN0ZWFkIHdlIGFzc3VtZSB0aGF0IHdlIGNhbiBjYWNoZSBk YXRhIGZvciBhcyBsb25nIGFzIHRoZSBmaWxlIGlzDQpvcGVuIGFuZCB1bmxvY2tlZCBhcyBpbmRl ZWQgdGhlIGNsb3NlLXRvLW9wZW4gY2FjaGUgY29uc2lzdGVuY3kgbW9kZWwNCmhhcyBhbHdheXMg c3RhdGVkIHRoYXQgd2UgY2FuIGRvLg0KDQpUcnlpbmcgdG8gaW52YWxpZGF0ZSB0aGUgY2FjaGUg b24gZXZlcnkgdW5leHBlY3RlZCBhdHRyaWJ1dGUgY2hhbmdlDQpsZWFkcyB0byBhIGxvdCBvZiB1 bm5lY2Vzc2FyeSBtaXNzZWQgY2FjaGVkIGhpdHMgd2hlbiBkb2luZyBvcmRpbmFyeQ0KSS9PIGJl Y2F1c2UgaXQgbWVhbnMgd2UgdG9zcyBvdXQgdGhlIGNhY2hlIG9uIGV2ZXJ5IHJlb3JkZXJlZCB3 cml0ZSwNCmV2ZXJ5IGxpbmssIHVubGluaywgcmVuYW1lLCBldGMgb2YgdGhlIGZpbGUuDQpJT1cg aXQgaGFzIGEgbGFyZ2UgcGVyZm9ybWFuY2UgaW1wYWN0IGZvbGxvd2luZyBhIG51bWJlciBvZiBv cGVyYXRpb25zDQp0aGF0IGFyZSBwZXJmZWN0bHkgbGVnYWwgYW5kIGluZGVlZCBjb21tb24gdW5k ZXIgdGhlIGNsb3NlLXRvLW9wZW4NCm1vZGVsIHdoaWxlIGZhaWxpbmcgdG8gZml4IHRoZSBwcm9i bGVtcyBvZiB0aGUgY2xvc2UtdG8tb3BlbiB2aW9sYXRpbmcNCm1vZGVsLg0KDQotLSANClRyb25k IE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWludGFpbmVyLCBIYW1tZXJzcGFjZQ0KdHJv bmQubXlrbGVidXN0QGhhbW1lcnNwYWNlLmNvbQ0KDQoNCg== ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-11 22:12 ` Trond Myklebust @ 2018-09-11 23:45 ` Chris Siebenmann 2018-09-12 2:19 ` Trond Myklebust 0 siblings, 1 reply; 15+ messages in thread From: Chris Siebenmann @ 2018-09-11 23:45 UTC (permalink / raw) To: Trond Myklebust; +Cc: linux-nfs, chuck.lever, cks > > Our issue also happens when the writes are done on the fileserver, > > though, and they occur even if you allow plenty of time for the > > writes to settle. I can run my test program in a mode where it > > explicitly waits for me to tell it to continue, do the appending > > to the file on the fileserver, 'sync' on the fileserver, wait five > > minutes, and the NFS client will still see those zero bytes when it > > tries to read the new data. > > That's happening because we're not optimising for the broken case, and > instead we assume that we can cache data for as long as the file is > open and unlocked as indeed the close-to-open cache consistency model > has always stated that we can do. If I'm understanding all of this right, is what the kernel does more or less like this: when a NFS client program closes a writeable file (descriptor), the kernel flushes any pending writes, does a GETATTR afterward, and declares all current cached pages fully valid 'as of' that GETATTR result. When the file is reopened (in any mode), the kernel GETATTRs the file again; if the GETATTR hasn't changed, the cached pages and their contents remain valid. As a result, if you write to the file from another machine (including the fileserver) before the writeable file is closed, on close the client uses the updated GETATTR from the server but its current cached pages. These cached pages may be out of date, but if so it is because one violated close-to-open; you must always close any writeable file descriptors on machine A before writing to the file on machine B (or obtain and then release locks?). If a client kernel has cached pages this way, is there any simple sequence of system calls on the client that will cause it to discard these cached pages? Or do you need the file's GETATTR to change again, implicitly from another machine? (I assume that changing the file's attributes from the client with the cached pages doesn't cause it to invalidate them, and certainly eg a 'touch' doesn't do it from the client where it does do it from another machine.) - cks ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-11 23:45 ` Chris Siebenmann @ 2018-09-12 2:19 ` Trond Myklebust 2018-09-12 3:03 ` Chris Siebenmann 0 siblings, 1 reply; 15+ messages in thread From: Trond Myklebust @ 2018-09-12 2:19 UTC (permalink / raw) To: cks; +Cc: linux-nfs, chuck.lever T24gVHVlLCAyMDE4LTA5LTExIGF0IDE5OjQ1IC0wNDAwLCBDaHJpcyBTaWViZW5tYW5uIHdyb3Rl Og0KPiA+ID4gIE91ciBpc3N1ZSBhbHNvIGhhcHBlbnMgd2hlbiB0aGUgd3JpdGVzIGFyZSBkb25l IG9uIHRoZQ0KPiA+ID4gZmlsZXNlcnZlciwNCj4gPiA+IHRob3VnaCwgYW5kIHRoZXkgb2NjdXIg ZXZlbiBpZiB5b3UgYWxsb3cgcGxlbnR5IG9mIHRpbWUgZm9yIHRoZQ0KPiA+ID4gd3JpdGVzIHRv IHNldHRsZS4gSSBjYW4gcnVuIG15IHRlc3QgcHJvZ3JhbSBpbiBhIG1vZGUgd2hlcmUgaXQNCj4g PiA+IGV4cGxpY2l0bHkgd2FpdHMgZm9yIG1lIHRvIHRlbGwgaXQgdG8gY29udGludWUsIGRvIHRo ZSBhcHBlbmRpbmcNCj4gPiA+IHRvIHRoZSBmaWxlIG9uIHRoZSBmaWxlc2VydmVyLCAnc3luYycg b24gdGhlIGZpbGVzZXJ2ZXIsIHdhaXQNCj4gPiA+IGZpdmUNCj4gPiA+IG1pbnV0ZXMsIGFuZCB0 aGUgTkZTIGNsaWVudCB3aWxsIHN0aWxsIHNlZSB0aG9zZSB6ZXJvIGJ5dGVzIHdoZW4NCj4gPiA+ IGl0DQo+ID4gPiB0cmllcyB0byByZWFkIHRoZSBuZXcgZGF0YS4NCj4gPiANCj4gPiBUaGF0J3Mg aGFwcGVuaW5nIGJlY2F1c2Ugd2UncmUgbm90IG9wdGltaXNpbmcgZm9yIHRoZSBicm9rZW4gY2Fz ZSwNCj4gPiBhbmQNCj4gPiBpbnN0ZWFkIHdlIGFzc3VtZSB0aGF0IHdlIGNhbiBjYWNoZSBkYXRh IGZvciBhcyBsb25nIGFzIHRoZSBmaWxlIGlzDQo+ID4gb3BlbiBhbmQgdW5sb2NrZWQgYXMgaW5k ZWVkIHRoZSBjbG9zZS10by1vcGVuIGNhY2hlIGNvbnNpc3RlbmN5DQo+ID4gbW9kZWwNCj4gPiBo YXMgYWx3YXlzIHN0YXRlZCB0aGF0IHdlIGNhbiBkby4NCj4gDQo+ICBJZiBJJ20gdW5kZXJzdGFu ZGluZyBhbGwgb2YgdGhpcyByaWdodCwgaXMgd2hhdCB0aGUga2VybmVsIGRvZXMgbW9yZQ0KPiBv ciBsZXNzIGxpa2UgdGhpczogd2hlbiBhIE5GUyBjbGllbnQgcHJvZ3JhbSBjbG9zZXMgYSB3cml0 ZWFibGUgZmlsZQ0KPiAoZGVzY3JpcHRvciksIHRoZSBrZXJuZWwgZmx1c2hlcyBhbnkgcGVuZGlu ZyB3cml0ZXMsIGRvZXMgYSBHRVRBVFRSDQo+IGFmdGVyd2FyZCwgYW5kIGRlY2xhcmVzIGFsbCBj dXJyZW50IGNhY2hlZCBwYWdlcyBmdWxseSB2YWxpZCAnYXMgb2YnDQo+IHRoYXQgR0VUQVRUUiBy ZXN1bHQuIFdoZW4gdGhlIGZpbGUgaXMgcmVvcGVuZWQgKGluIGFueSBtb2RlKSwgdGhlDQo+IGtl cm5lbA0KPiBHRVRBVFRScyB0aGUgZmlsZSBhZ2FpbjsgaWYgdGhlIEdFVEFUVFIgaGFzbid0IGNo YW5nZWQsIHRoZSBjYWNoZWQNCj4gcGFnZXMNCj4gYW5kIHRoZWlyIGNvbnRlbnRzIHJlbWFpbiB2 YWxpZC4NCj4gDQo+ICBBcyBhIHJlc3VsdCwgaWYgeW91IHdyaXRlIHRvIHRoZSBmaWxlIGZyb20g YW5vdGhlciBtYWNoaW5lDQo+IChpbmNsdWRpbmcNCj4gdGhlIGZpbGVzZXJ2ZXIpIGJlZm9yZSB0 aGUgd3JpdGVhYmxlIGZpbGUgaXMgY2xvc2VkLCBvbiBjbG9zZSB0aGUNCj4gY2xpZW50IHVzZXMg dGhlIHVwZGF0ZWQgR0VUQVRUUiBmcm9tIHRoZSBzZXJ2ZXIgYnV0IGl0cyBjdXJyZW50DQo+IGNh Y2hlZA0KPiBwYWdlcy4gVGhlc2UgY2FjaGVkIHBhZ2VzIG1heSBiZSBvdXQgb2YgZGF0ZSwgYnV0 IGlmIHNvIGl0IGlzIGJlY2F1c2UNCj4gb25lIHZpb2xhdGVkIGNsb3NlLXRvLW9wZW47IHlvdSBt dXN0IGFsd2F5cyBjbG9zZSBhbnkgd3JpdGVhYmxlIGZpbGUNCj4gZGVzY3JpcHRvcnMgb24gbWFj aGluZSBBIGJlZm9yZSB3cml0aW5nIHRvIHRoZSBmaWxlIG9uIG1hY2hpbmUgQiAob3INCj4gb2J0 YWluIGFuZCB0aGVuIHJlbGVhc2UgbG9ja3M/KS4NCj4gDQo+ICBJZiBhIGNsaWVudCBrZXJuZWwg aGFzIGNhY2hlZCBwYWdlcyB0aGlzIHdheSwgaXMgdGhlcmUgYW55IHNpbXBsZQ0KPiBzZXF1ZW5j ZSBvZiBzeXN0ZW0gY2FsbHMgb24gdGhlIGNsaWVudCB0aGF0IHdpbGwgY2F1c2UgaXQgdG8gZGlz Y2FyZA0KPiB0aGVzZSBjYWNoZWQgcGFnZXM/IE9yIGRvIHlvdSBuZWVkIHRoZSBmaWxlJ3MgR0VU QVRUUiB0byBjaGFuZ2UNCj4gYWdhaW4sDQo+IGltcGxpY2l0bHkgZnJvbSBhbm90aGVyIG1hY2hp bmU/IChJIGFzc3VtZSB0aGF0IGNoYW5naW5nIHRoZSBmaWxlJ3MNCj4gYXR0cmlidXRlcyBmcm9t IHRoZSBjbGllbnQgd2l0aCB0aGUgY2FjaGVkIHBhZ2VzIGRvZXNuJ3QgY2F1c2UgaXQgdG8NCj4g aW52YWxpZGF0ZSB0aGVtLCBhbmQgY2VydGFpbmx5IGVnIGEgJ3RvdWNoJyBkb2Vzbid0IGRvIGl0 IGZyb20gdGhlDQo+IGNsaWVudCB3aGVyZSBpdCBkb2VzIGRvIGl0IGZyb20gYW5vdGhlciBtYWNo aW5lLikNCg0KVGhlcmUgYXJlIDIgd2F5cyB0byBtYW5pcHVsYXRlIHRoZSBwYWdlIGNhY2hlIGRp cmVjdGx5IG9uIHRoZSBjbGllbnQ6DQogICAxLiBZb3UgY2FuIGNsZWFyIG91dCB0aGUgZW50aXJl IHBhZ2UgY2FjaGUgYXMgdGhlICdyb290JyB1c2VyLCB3aXRoIHRoZQ0KICAgICAgL3Byb2Mvc3lz L3ZtL2Ryb3BfY2FjaGVzIGludGVyZmFjZSAoc2VlICdtYW4gNSBwcm9jJykuDQogICAyLiBBbHRl cm5hdGl2ZWx5LCB5b3UgY2FuIHVzZSBwb3NpeF9mYWR2aXNlKCkgd2l0aCB0aGUNCiAgICAgIFBP U0lYX0ZBRFZfRE9OVE5FRUQgZmxhZyB0byBjbGVhciBvdXQgb25seSB0aGUgcGFnZXMgdGhhdCB5 b3UgdGhpbmsNCiAgICAgIGFyZSBiYWQuIE1ha2Ugc3VyZSB0byBmaXJzdCBmc3luYygpIHNvIHRo YXQgdGhlIHBhZ2VzIGRvbid0IGdldA0KICAgICAgcGlubmVkIGluIG1lbW9yeSBieSB2aXJ0dWUg b2YgYmVpbmcgZGlydHkgKHNlZSAnbWFuIDIgZmFkdmlzZTY0JykuDQoNCllvdSBhbHNvIGhhdmUg dGhlIG9wdGlvbiBvZiB1c2luZyBORlMgaXRzZWxmIHRvIGltcGxpY2l0bHkgY2hhbmdlIHRoZQ0K Y2FjaGU6DQogICAxLiBBcyB5b3Ugc2FpZCBhYm92ZSwgeW91IGNhbiBhbHNvIGNoYW5nZSB0aGUg ZmlsZSBvbiB0aGUgc2VydmVyIHdoaWxlDQogICAgICBpdCBpcyBjbG9zZWQgb24geW91ciBjbGll bnQsIGFuZCB0aGVuIHJlb3BlbiBpdC4NCiAgIDIuIFlvdSBjYW4gcGVyZm9ybSBhbiBPX0RJUkVD VCB3cml0ZSBmcm9tIHRoZSBjbGllbnQgaXRzZWxmLiBCb3RoIHRob3NlDQogICAgICBvcGVyYXRp b25zIHdpbGwgYWxzbyBpbXBseSBhIGNhY2hlIGludmFsaWRhdGlvbi4NCg0KLS0gDQpUcm9uZCBN eWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5lciwgSGFtbWVyc3BhY2UNCnRyb25k Lm15a2xlYnVzdEBoYW1tZXJzcGFjZS5jb20NCg0KDQo= ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-12 2:19 ` Trond Myklebust @ 2018-09-12 3:03 ` Chris Siebenmann 2018-09-12 17:11 ` Trond Myklebust 0 siblings, 1 reply; 15+ messages in thread From: Chris Siebenmann @ 2018-09-12 3:03 UTC (permalink / raw) To: Trond Myklebust; +Cc: linux-nfs, chuck.lever, cks > > If a client kernel has cached pages this way, is there any simple > > sequence of system calls on the client that will cause it to discard > > these cached pages? Or do you need the file's GETATTR to change again, > > implicitly from another machine? (I assume that changing the file's > > attributes from the client with the cached pages doesn't cause it to > > invalidate them, and certainly eg a 'touch' doesn't do it from the > > client where it does do it from another machine.) > > There are 2 ways to manipulate the page cache directly on the client: > 1. You can clear out the entire page cache as the 'root' user, with the > /proc/sys/vm/drop_caches interface (see 'man 5 proc'). > 2. Alternatively, you can use posix_fadvise() with the > POSIX_FADV_DONTNEED flag to clear out only the pages that you think > are bad. Make sure to first fsync() so that the pages don't get > pinned in memory by virtue of being dirty (see 'man 2 fadvise64'). I just did some experiments, and on the Ubuntu 18.04 LTS version of 4.15.0, it appears that flock()'ing the file before re-reading it will cause the kernel to not manifest the problem. I don't seem to have to flock() the file initially when I read it before the change, and it's sufficient to use LOCK_SH instead of LOCK_EX. (And I do have to flock() after the change, otherwise I still see the problem even if I flock() before.) Is this a supported/guaranteed behavior, or is it just lucky coincidence that things currently work this way, much like it was happenstance instead of design that things worked back in the 4.4.x era? It would be very convenient for us if flock() works around this, because it turns out that the only reason Alpine is not flock()'ing files is that it has an ancient 'do not use flock on Linux NFS' piece of code deep inside it that was apparently there to work around a bug that seems to have been fixed a decade or so ago: http://repo.or.cz/alpine.git/blob/HEAD:/imap/src/osdep/unix/flocklnx.c - cks ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-12 3:03 ` Chris Siebenmann @ 2018-09-12 17:11 ` Trond Myklebust 0 siblings, 0 replies; 15+ messages in thread From: Trond Myklebust @ 2018-09-12 17:11 UTC (permalink / raw) To: cks; +Cc: linux-nfs, chuck.lever T24gVHVlLCAyMDE4LTA5LTExIGF0IDIzOjAzIC0wNDAwLCBDaHJpcyBTaWViZW5tYW5uIHdyb3Rl Og0KPiA+ID4gIElmIGEgY2xpZW50IGtlcm5lbCBoYXMgY2FjaGVkIHBhZ2VzIHRoaXMgd2F5LCBp cyB0aGVyZSBhbnkNCj4gPiA+IHNpbXBsZQ0KPiA+ID4gc2VxdWVuY2Ugb2Ygc3lzdGVtIGNhbGxz IG9uIHRoZSBjbGllbnQgdGhhdCB3aWxsIGNhdXNlIGl0IHRvDQo+ID4gPiBkaXNjYXJkDQo+ID4g PiB0aGVzZSBjYWNoZWQgcGFnZXM/IE9yIGRvIHlvdSBuZWVkIHRoZSBmaWxlJ3MgR0VUQVRUUiB0 byBjaGFuZ2UNCj4gPiA+IGFnYWluLA0KPiA+ID4gaW1wbGljaXRseSBmcm9tIGFub3RoZXIgbWFj aGluZT8gKEkgYXNzdW1lIHRoYXQgY2hhbmdpbmcgdGhlDQo+ID4gPiBmaWxlJ3MNCj4gPiA+IGF0 dHJpYnV0ZXMgZnJvbSB0aGUgY2xpZW50IHdpdGggdGhlIGNhY2hlZCBwYWdlcyBkb2Vzbid0IGNh dXNlIGl0DQo+ID4gPiB0bw0KPiA+ID4gaW52YWxpZGF0ZSB0aGVtLCBhbmQgY2VydGFpbmx5IGVn IGEgJ3RvdWNoJyBkb2Vzbid0IGRvIGl0IGZyb20NCj4gPiA+IHRoZQ0KPiA+ID4gY2xpZW50IHdo ZXJlIGl0IGRvZXMgZG8gaXQgZnJvbSBhbm90aGVyIG1hY2hpbmUuKQ0KPiA+IA0KPiA+IFRoZXJl IGFyZSAyIHdheXMgdG8gbWFuaXB1bGF0ZSB0aGUgcGFnZSBjYWNoZSBkaXJlY3RseSBvbiB0aGUN Cj4gPiBjbGllbnQ6DQo+ID4gICAgMS4gWW91IGNhbiBjbGVhciBvdXQgdGhlIGVudGlyZSBwYWdl IGNhY2hlIGFzIHRoZSAncm9vdCcgdXNlciwNCj4gPiB3aXRoIHRoZQ0KPiA+ICAgICAgIC9wcm9j L3N5cy92bS9kcm9wX2NhY2hlcyBpbnRlcmZhY2UgKHNlZSAnbWFuIDUgcHJvYycpLg0KPiA+ICAg IDIuIEFsdGVybmF0aXZlbHksIHlvdSBjYW4gdXNlIHBvc2l4X2ZhZHZpc2UoKSB3aXRoIHRoZQ0K PiA+ICAgICAgIFBPU0lYX0ZBRFZfRE9OVE5FRUQgZmxhZyB0byBjbGVhciBvdXQgb25seSB0aGUg cGFnZXMgdGhhdCB5b3UNCj4gPiB0aGluaw0KPiA+ICAgICAgIGFyZSBiYWQuIE1ha2Ugc3VyZSB0 byBmaXJzdCBmc3luYygpIHNvIHRoYXQgdGhlIHBhZ2VzIGRvbid0DQo+ID4gZ2V0DQo+ID4gICAg ICAgcGlubmVkIGluIG1lbW9yeSBieSB2aXJ0dWUgb2YgYmVpbmcgZGlydHkgKHNlZSAnbWFuIDIN Cj4gPiBmYWR2aXNlNjQnKS4NCj4gDQo+ICBJIGp1c3QgZGlkIHNvbWUgZXhwZXJpbWVudHMsIGFu ZCBvbiB0aGUgVWJ1bnR1IDE4LjA0IExUUyB2ZXJzaW9uIG9mDQo+IDQuMTUuMCwgaXQgYXBwZWFy cyB0aGF0IGZsb2NrKCknaW5nIHRoZSBmaWxlIGJlZm9yZSByZS1yZWFkaW5nIGl0DQo+IHdpbGwN Cj4gY2F1c2UgdGhlIGtlcm5lbCB0byBub3QgbWFuaWZlc3QgdGhlIHByb2JsZW0uIEkgZG9uJ3Qg c2VlbSB0byBoYXZlIHRvDQo+IGZsb2NrKCkgdGhlIGZpbGUgaW5pdGlhbGx5IHdoZW4gSSByZWFk IGl0IGJlZm9yZSB0aGUgY2hhbmdlLCBhbmQgaXQncw0KPiBzdWZmaWNpZW50IHRvIHVzZSBMT0NL X1NIIGluc3RlYWQgb2YgTE9DS19FWC4gKEFuZCBJIGRvIGhhdmUgdG8NCj4gZmxvY2soKQ0KPiBh ZnRlciB0aGUgY2hhbmdlLCBvdGhlcndpc2UgSSBzdGlsbCBzZWUgdGhlIHByb2JsZW0gZXZlbiBp ZiBJIGZsb2NrKCkNCj4gYmVmb3JlLikNCj4gDQo+ICBJcyB0aGlzIGEgc3VwcG9ydGVkL2d1YXJh bnRlZWQgYmVoYXZpb3IsIG9yIGlzIGl0IGp1c3QgbHVja3kNCj4gY29pbmNpZGVuY2UNCj4gdGhh dCB0aGluZ3MgY3VycmVudGx5IHdvcmsgdGhpcyB3YXksIG11Y2ggbGlrZSBpdCB3YXMgaGFwcGVu c3RhbmNlDQo+IGluc3RlYWQgb2YgZGVzaWduIHRoYXQgdGhpbmdzIHdvcmtlZCBiYWNrIGluIHRo ZSA0LjQueCBlcmE/DQo+IA0KPiAgSXQgd291bGQgYmUgdmVyeSBjb252ZW5pZW50IGZvciB1cyBp ZiBmbG9jaygpIHdvcmtzIGFyb3VuZCB0aGlzLA0KPiBiZWNhdXNlIGl0IHR1cm5zIG91dCB0aGF0 IHRoZSBvbmx5IHJlYXNvbiBBbHBpbmUgaXMgbm90IGZsb2NrKCknaW5nDQo+IGZpbGVzIGlzIHRo YXQgaXQgaGFzIGFuIGFuY2llbnQgJ2RvIG5vdCB1c2UgZmxvY2sgb24gTGludXggTkZTJyBwaWVj ZQ0KPiBvZg0KPiBjb2RlIGRlZXAgaW5zaWRlIGl0IHRoYXQgd2FzIGFwcGFyZW50bHkgdGhlcmUg dG8gd29yayBhcm91bmQgYSBidWcNCj4gdGhhdA0KPiBzZWVtcyB0byBoYXZlIGJlZW4gZml4ZWQg YSBkZWNhZGUgb3Igc28gYWdvOg0KPiANCj4gICAgDQo+IGh0dHA6Ly9yZXBvLm9yLmN6L2FscGlu ZS5naXQvYmxvYi9IRUFEOi9pbWFwL3NyYy9vc2RlcC91bml4L2Zsb2NrbG54LmMNCj4gDQoNCk5v dGhpbmcgc2hvdWxkIGJlIHN0b3BwaW5nIHlvdSBmcm9tIHVzaW5nIGVpdGhlciBmbG9jaygpIG9y IFBPU0lYIGxvY2tzDQpvdmVyIE5GUy4gVGhlIG9uZSBkaWZmZXJlbmNlIHRoYXQgeW91IHdpbGwg YmV0d2VlbiBORlMgYW5kIGxvY2FsDQpmaWxlc3lzdGVtcyBpcyB0aGF0IGZsb2NrKCkgbG9ja3Mg Y2FuIGNvbmZsaWN0IHdpdGggUE9TSVggbG9ja3Mgd2hlbg0KcGVyZm9ybWVkIG92ZXIgTkZTLg0K DQpJT1c6IElmIHlvdSBlbmFibGUgZmxvY2soKSBsb2NrcywgdGhlbiBlbnN1cmUgdGhhdCB5b3Ug ZG8gbm90IGVuYWJsZQ0KUE9TSVggbG9ja3MgKGEuay5hLiBmY250bCgpIGxvY2tzLCBhLmsuYS4g bG9ja2YoKSBsb2NrcykuDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50 IG1haW50YWluZXIsIEhhbW1lcnNwYWNlDQp0cm9uZC5teWtsZWJ1c3RAaGFtbWVyc3BhY2UuY29t DQoNCg0K ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: A NFS client partial file corruption problem in recent/current kernels 2018-09-11 20:40 ` Chuck Lever 2018-09-11 20:47 ` Chris Siebenmann @ 2018-09-11 20:56 ` Trond Myklebust 1 sibling, 0 replies; 15+ messages in thread From: Trond Myklebust @ 2018-09-11 20:56 UTC (permalink / raw) To: chuck.lever; +Cc: cks, linux-nfs T24gVHVlLCAyMDE4LTA5LTExIGF0IDE2OjQwIC0wNDAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g PiBPbiBTZXAgMTEsIDIwMTgsIGF0IDQ6MDAgUE0sIFRyb25kIE15a2xlYnVzdCA8DQo+ID4gdHJv bmRteUBoYW1tZXJzcGFjZS5jb20+IHdyb3RlOg0KPiA+IA0KPiA+IE9uIFR1ZSwgMjAxOC0wOS0x MSBhdCAxNDowMiAtMDQwMCwgQ2hyaXMgU2llYmVubWFubiB3cm90ZToNCj4gPiA+ID4gPiBXZSd2 ZSBmb3VuZCBhIHJlYWRpbHkgcmVwcm9kdWNhYmxlIHNpdHVhdGlvbiB3aGVyZSB0aGUNCj4gPiA+ ID4gPiBjdXJyZW50DQo+ID4gPiA+ID4gTkZTIGNsaWVudCBjb2RlIHdpbGwgcHJvdmlkZSB6ZXJv IGJ5dGVzIGluc3RlYWQgb2YgYWN0dWFsDQo+ID4gPiA+ID4gZGF0YSBhdA0KPiA+ID4gPiA+IHRo ZSBlbmQgb2YgdGhlIGZpbGUgKHNvcnQgb2YpIHRvIHVzZXIgcHJvZ3JhbXMuIFRoaXMgY2FuDQo+ ID4gPiA+ID4gcmVzdWx0DQo+ID4gPiA+ID4gaW4gcHJvZ3JhbSBmYWlsdXJlLCBvciBwZXJtYW5l bnQgZmlsZSBjb3JydXB0aW9uIGlmIHRoZQ0KPiA+ID4gPiA+IHByb2dyYW0NCj4gPiA+ID4gPiBy ZWFkaW5nIHRoZSBmaWxlIHdyaXRlcyB0aGUgYmFkIGRhdGEgYmFjayB0byB0aGUgZmlsZTsNCj4g PiA+ID4gPiBvdGhlcndpc2UsDQo+ID4gPiA+ID4gdGhlIGNvcnJ1cHRpb24gZ29lcyBhd2F5IHdo ZW4gdGhlIGNsaWVudCdzIGNhY2hlZCBkYXRhIGlzDQo+ID4gPiA+ID4gcHVzaGVkDQo+ID4gPiA+ ID4gb3V0DQo+ID4gPiA+ID4gb2YgbWVtb3J5IChvciBleHBsaWNpdGx5IGRyb3BwZWQgYnkgZHJv cHBpbmcgdGhlIHBhZ2VjYWNoZQ0KPiA+ID4gPiA+IHRocm91Z2gNCj4gPiA+ID4gPiAvcHJvYy9z eXMvdm0vZHJvcF9jYWNoZXMpLg0KPiA+ID4gDQo+ID4gPiBbLi4uXQ0KPiA+ID4gPiBQbGVhc2Ug c2VlIGh0dHA6Ly9uZnMuc291cmNlZm9yZ2UubmV0LyNmYXFfYTgNCj4gPiA+IA0KPiA+ID4gSSBk b24ndCB0aGluayB0aGlzIGlzIGEgY2xvc2UgdG8gb3BlbiBjb25zaXN0ZW5jeSBpc3N1ZSwgb3Ig aWYgaXQNCj4gPiA+IGlzDQo+ID4gPiBJIHdvdWxkIGFyZ3VlIHRoYXQgaXQgaXMgYSBjbGVhciBi dWcgb24gdGhlIExpbnV4IE5GUyBjbGllbnQuIEkNCj4gPiA+IGhhdmUNCj4gPiA+IGEgbnVtYmVy IG9mIHJlYXNvbnMgZm9yIHNheWluZyB0aGlzOg0KPiA+ID4gDQo+ID4gPiAtIHRoZSBjbGllbnQg Y2xlYXJseSBzZWVzIHRoZSBuZXcgYXR0cmlidXRlczsgaXQga25vd3MgdGhhdCB0aGUNCj4gPiA+ IGZpbGUNCj4gPiA+ICBoYXMgYmVlbiBleHRlbmRlZCBmcm9tIHRoZSBwcmV2aW91cyBzdGF0ZSB0 aGF0IGl0IGtuZXcgb2YuIE15DQo+ID4gPiBkZW1vDQo+ID4gPiAgcHJvZ3JhbSBzcGVjaWZpY2Fs bHkgd2FpdHMgdW50aWwgdXNlci1sZXZlbCBmc3RhdCgpIHJldHVybnMgYQ0KPiA+ID4gZGlmZmVy ZW50DQo+ID4gPiAgcmVzdWx0LCB3aGljaCBJIGJlbGlldmUgbWVhbnMgdGhhdCB0aGUgY2xpZW50 IGtlcm5lbCBoYXMgc2VlbiBhDQo+ID4gPiBkaWZmZXJlbnQNCj4gPiA+ICBHRVRBVFRSIHJlc3Vs dCBhbmQgc28gc2hvdWxkIGhhdmUgcHVyZ2VkIGl0cyBjYWNoZSAoYmFzZWQgb24NCj4gPiA+IHdo YXQNCj4gPiA+IHRoZQ0KPiA+ID4gIEZBUSBzYXlzKS4NCj4gPiA+IA0KPiA+ID4gIChVbmxlc3Mg dGhlIEZBUSBtZWFucyB0aGF0IHRoZSBrZXJuZWwgYWJzb2x1dGVseSByZWZ1c2VzIHRvDQo+ID4g PiBndWFyYW50ZWUNCj4gPiA+ICBhbnl0aGluZyBhYm91dCBmaWxlIGNvbnNpc3RlbmN5IHVubGVz cyB5b3UgY2xvc2UgYW5kIHRoZW4gcmVvcGVuDQo+ID4gPiB0aGUNCj4gPiA+ICBmaWxlLCBldmVu IGlmIGl0ICprbm93cyogdGhhdCB0aGUgZmlsZSBoYXMgY2hhbmdlZCBvbiB0aGUNCj4gPiA+IHNl cnZlciwNCj4gPiA+ICB3aGljaCBpc24ndCBjbGVhciBmcm9tIGhvdyB0aGUgRkFRIGlzIGN1cnJl bnRseSB3cml0dGVuLikNCj4gPiA+IA0KPiA+ID4gLSB0aGUgY2xpZW50IGlzIGZldGNoaW5nIHNv bWUgbmV3IGRhdGEgZnJvbSB0aGUgZmlsZXNlcnZlciAoZGF0YQ0KPiA+ID4gYWZ0ZXINCj4gPiA+ ICB0aGUgcGFydGlhbCA0IEtCIHBhZ2UgYXQgdGhlIG9sZCBlbmQgb2YgdGhlIGZpbGUpLg0KPiA+ ID4gDQo+ID4gPiAtIHRoZSBjbGllbnQgaXNuJ3Qgd3JpdGluZyB0byB0aGUgZmlsZSBpbiBteSBk ZW1vbnN0cmF0aW9uDQo+ID4gPiBwcm9ncmFtOw0KPiA+ID4gaXQncw0KPiA+ID4gIG9ubHkgb3Bl bmluZyBpdCBpbiByZWFkLXdyaXRlIG1vZGUgYW5kIHRoZW4gcmVhZGluZyBpdC4gQWxzbywNCj4g PiA+IHRoaXMNCj4gPiA+ICBkb2Vzbid0IGhhcHBlbiBpZiB0aGUgY2xpZW50IGRvZXMgZXhhY3Rs eSB0aGUgc2FtZSBzZXQgb2YNCj4gPiA+IG9wZXJhdGlvbnMNCj4gPiA+ICBidXQgaGFzIHRoZSBm aWxlIG9wZW4gcmVhZC1vbmx5ICh3aXRoIGl0IHN0YXlpbmcgb3Blbg0KPiA+ID4gdGhyb3VnaG91 dCkuDQo+ID4gPiANCj4gPiA+IC0gdGhpcyBkaWRuJ3QgaGFwcGVuIGluIG9sZGVyIGtlcm5lbHMu DQo+ID4gPiANCj4gPiA+IEluIGFkZGl0aW9uLCBhbHRob3VnaCBJIGRpZG4ndCBtZW50aW9uIGl0 IGluIG15IG9yaWdpbmFsIGVtYWlsLA0KPiA+ID4gdGhpcw0KPiA+ID4gaGFwcGVucyBvbiBhIE5G UyBmaWxlc3lzdGVtIG1vdW50ZWQgJ25vYWMnLg0KPiA+ID4gDQo+ID4gPiBQcmFnbWF0aWNhbGx5 LCBBbHBpbmUgdXNlZCB0byB3b3JrIHdpdGggTkZTIG1vdW50ZWQgZmlsZXN5c3RlbXMNCj4gPiA+ IHdoZXJlDQo+ID4gPiBlbWFpbCB3YXMgYXBwZW5kZWQgdG8gdGhlbSBmcm9tIG90aGVyIG1hY2hp bmVzIGFuZCBpdCBubyBsb25nZXINCj4gPiA+IGRvZXMsDQo+ID4gPiBhbmQgdGhlIG9ubHkgZGlm ZmVyZW5jZSBpcyB0aGUga2VybmVsIHZlcnNpb24gaW52b2x2ZWQgb24gdGhlDQo+ID4gPiBjbGll bnQuDQo+ID4gPiBUaGlzIGJyZWFrYWdlIGlzIGFjdGl2ZWx5IGRhbmdlcm91cy4NCj4gPiANCj4g PiBTdXJlLCBidXQgdW5sZXNzIHlvdSBhcmUgbG9ja2luZyB0aGUgZmlsZSwgb3IgeW91IGFyZSBl eHBsaWNpdGx5DQo+ID4gdXNpbmcNCj4gPiBPX0RJUkVDVCB0byBkbyB1bmNhY2hlZCBJL08sIHRo ZW4geW91IGFyZSBpbiB2aW9sYXRpb24gb2YgdGhlDQo+ID4gY2xvc2UtdG8tDQo+ID4gb3BlbiBj b25zaXN0ZW5jeSBtb2RlbCwgYW5kIHRoZSBjbGllbnQgaXMgZ29pbmcgdG8gYmVoYXZlIGFzIHlv dQ0KPiA+IGRlc2NyaWJlIGFib3ZlLiBORlMgdXNlcyBhIGRpc3RyaWJ1dGVkIGZpbGVzeXN0ZW0g bW9kZWwsIG5vdCBhDQo+ID4gY2x1c3RlcmVkIG9uZS4NCj4gDQo+IEkgd291bGQgZXhwZWN0IEFs cGluZSB0byB3b3JrIGlmICJ2ZXJzPTMsbm9hYyIgaXMgaW4gdXNlLg0KPiANCg0Kbm9hYyBoYXMg bm90aGluZyBhdCBhbGwgdG8gZG8gd2l0aCBkYXRhIGNhY2hlIGNvbnNpc3RlbmN5Lg0KDQotLSAN ClRyb25kIE15a2xlYnVzdA0KQ1RPLCBIYW1tZXJzcGFjZSBJbmMNCjQzMDAgRWwgQ2FtaW5vIFJl YWwsIFN1aXRlIDEwNQ0KTG9zIEFsdG9zLCBDQSA5NDAyMg0Kd3d3LmhhbW1lci5zcGFjZQ0KDQoN Cg== ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2018-09-12 22:16 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-09-11 15:59 A NFS client partial file corruption problem in recent/current kernels Chris Siebenmann 2018-09-11 17:12 ` Trond Myklebust 2018-09-11 18:02 ` Chris Siebenmann 2018-09-11 20:00 ` Trond Myklebust 2018-09-11 20:38 ` Chris Siebenmann 2018-09-11 20:40 ` Chuck Lever 2018-09-11 20:47 ` Chris Siebenmann 2018-09-11 21:25 ` Trond Myklebust 2018-09-11 21:39 ` Chris Siebenmann 2018-09-11 22:12 ` Trond Myklebust 2018-09-11 23:45 ` Chris Siebenmann 2018-09-12 2:19 ` Trond Myklebust 2018-09-12 3:03 ` Chris Siebenmann 2018-09-12 17:11 ` Trond Myklebust 2018-09-11 20:56 ` Trond Myklebust
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.